IT Disaster Recovery: Are You Prepared?

Has your company ever taken the time to estimate the potential economic consequences of a significant data loss?

• By Elizabeth Donley
• Aug 16, 2007

RECOVERING from a disaster is never easy, whether it's a hurricane, tornado, earthquake, fire, flood, or thunderstorm. These events have the potential to cause loss of equipment, power failures, non-functioning communication systems, and missing or otherwise preoccupied personnel. And when your  health and safety (H&S) data are involved, the potential consequences can take on new dimensions, especially when you consider how Web technologies have made it possible for electronic information to become integral to every part of your business. Even without "an act of God," a disruption in your H&S electronic information management system, for whatever reason, can be disastrous.

Down and Out
Have you ever waited in line at a grocery, hardware store, or fast food restaurant during a power outage? Remember the look of panic on the faces of the cashiers? Did everything continue to function efficiently, or did the entire business come to a complete halt?

What would happen if your communication and data management systems were to be unavailable for hours, days, or even weeks? How critical are your data? How long could your company continue to function? What if the failure was limited to your H&S system? Would your core business functions continue uninterrupted? Would this affect your production, cause purchasing delays, result in fines for non-compliance, or put your company at risk for lawsuits or bad publicity? What if the disruption was limited to a specific application? Has your company ever taken the time to estimate the potential economic consequences?

Think this can't happen to you? Consider the devastation of Hurricane Katrina in August 2005. Tens of thousands of businesses and millions of their employees lost basic services--including electricity, water, and telephone--and hundreds of thousands remained without power for more than a month. Unfortunately, this isn't an isolated incident. Just consider the number of times the news media mention power outages or mandatory evacuations (search Google News for "widespread power outages" for a recent sampling). For example, in July 2006, a nine-day power outage in Queens, N.Y., affected more than 100,000 people.

It's Only H&S Data
What's the big deal? It's only H&S data. We'll just wait until the system is back up, right? Suppose a power surge fries your server. The resulting data loss can create more than a minor nuisance. It may take weeks to regenerate the lost data, and some critical information may be lost forever if you weren't diligent about backing up the system frequently. The recovery effort may well require resources that you can't spare and expose your company to fines and other consequences for failing to meet regulatory deadlines.

But this is minor compared to the possible consequences of losing access to your data in an emergency. In many cases, our H&S systems have become our emergency response systems, as well. Whether you are subject to the Emergency Planning and Community Right-to-Act or one of the myriad of Occupational Safety and Health Act requirements, chances are that your facility has combined its H&S functions with its more traditional emergency response programs.

So when an emergency disables your H&S system, it also severely hampers your ability to respond to that same emergency. When a fire disrupts your electrical service, emergency response personnel may no longer have access to the electronic Material Safety Data Sheets (MSDSs) they need to mount an appropriate response. Or when a truck carrying hazardous materials to your facility hits a power pole, you also may lose access to the contact information you need to activate your emergency response team.

The reality is that a comprehensive H&S system can affect all parts of a company. It may contain information required for the purchase of certain chemicals, documentation for waste disposal, material safety information, medical records, emergency response protocols, and contact information for a wide range of issues. It may house the required forms for regulatory reporting or for tracking H&S incidents. And it likely contains the data needed to complete these forms and to file required reports.

With Technology Comes Risk
If your company has a need to share information across facilities or with customers, the potential for problems increases dramatically. Consider the possibility that your Web services are out. At any given time, some portion of the business Web sites in the world are down due to a variety of conditions, including:
• a lightning strike has damaged the server;
• the server software has crashed or the server has experienced a hardware failure;
• the server was hit by a virus or denial of service attack;
• someone failed to renew the domain name or Web services;
• there was an infrastructure failure affecting the electric service, telephone, or broadband access; or
• the system is undergoing routine maintenance.

If you host your own Web presence, several of these conditions are also likely to affect your ability to contact anyone outside of your company because your computer-based e-mail, voice mail, and fax also may be down. And even if you could call someone on your cellphone, the contact information and critical data are all in your computer. And just because you're affected doesn't mean anyone else is or that they know about your disaster, so they still expect you to conduct business as usual. If the problem is limited to a disruption in the connection between your server and the rest of the Internet, you may not even be aware of the situation until you try to visit someone else's Web site.

How to Prepare
The first step in preparing for a disaster is to develop a security plan for all physical locations, including off-site facilities that house your data. The plan should address the daily maintenance of your system, including backup frequency. It should address the physical security of the location, and it should include emergency procedures that consider loss of access to your network, to your Web site, to the Internet, and to other business systems. In short, you need a backup plan for every foreseeable emergency.

When developing the plan, you should begin to think about the way your system operates and alternative approaches that might provide a higher level of security. For example, while most people strive to keep their software and operating system up to date, there is a certain amount of security in avoiding change. The Windows operating system has been around for a long time, now, but there are still companies that use DOS-based programs for certain tasks. For whatever reason, these companies failed to upgrade when the Windows version was released or decided to keep the old system even after the developer went out of business. These folks might not have all of the bells and whistles, but the old DOS program, as unsophisticated and quirky as it may be, still gets the job done. It may be inflexible, but it's generally reliable.

The most sophisticated alternatives often offer the least security. For example, when you choose to host your critical functions with a Web-based service, you may have no control over changes or system upgrades. Then, one day, you log on to the system only to find that some of your commands no longer work--it turns out that the latest upgrade doesn't support some of the features the developer customized for your unique requirements. Or perhaps the server crashed and someone inadvertently installed an older version of your database.

So how do you prepare when your data are housed on someone else's proprietary system? First, make sure you actually own your data. Second, make sure your data are available in a universal format or can be easily converted using desktop software. Third, arrange with the host to allow you to download a copy of all data at frequent intervals. Fourth, have a system for retrieving critical information from your most recent backup. (For example, purchase a copy of the database software that is capable of reading the data format so you can at least perform rudimentary searches.) Finally, test the backup data periodically to make sure they are compatible with your in-house database software. But the most important step is to continue to maintain paper copies of critical information, such as MSDSs, regulatory reports, OSHA incident reports, monitoring data, etc.

Summary
New Web technologies provide new opportunities but also include new risks. In an article in the May 2003 edition of Harvard Business Review, Editor Nicolas G. Carr said, "executives need to shift their attention from IT opportunities to IT risks--from offense to defense." That's probably a bit extreme. A better approach is to look at IT the same way you look at any business proposition. Every decision should be an informed decision. You should weigh the opportunities against the risks in order to select the best option. Then, once you have made your decision, take the necessary steps to minimize and prepare for the risks. This includes preparing for whatever disaster may come your way.