Preparing data centers for disaster

While most Americans spent summer reveling in longer days and (hopefully) better weather, data center teams were preparing for hurricane season. With volatile weather blanketing the country over the summer, the National Oceanic and Atmospheric Administration (NOAA) calculated a 65% chance of seeing an above-normal number of storms this season with a projected 14 to 21 named storms, six to 10 hurricanes, and three to six major hurricanes.

There's no such thing as being too prepared. Forget plans A and B — what about plans C, D, and E? Waiting too long to enact emergency preparedness plans can have disastrous consequences. Two days prior to a Category 3 hurricane making landfall is not the time to discover that someone failed to stock up on batteries, let alone secure the data.

Preparing facilities for hurricane season is critical for the safety of your personnel and your ability to continue business operations. Determining when to begin preparing for a hurricane, keeping aware of a hurricane track’s progression, and identifying those critical activities that are needed before, during, and after a hurricane’s progression help meet this goal. To help with your own emergency preparedness planning, here are some key ideas to consider:

Cone of Uncertainty. Put your emergency plan into action at the first indication one of your facilities falls within the cone of uncertainty. When it comes to emergency preparedness, the more time you have to put your plans into action, the better (we start with a minimum of five days before expected landfall). But, Mother Nature is fickle, and you won’t always have that much warning. That said, assume you will be hit and act accordingly as you count down the time to projected landfall.

• 120 hours prior — Alert your staff to the possibility of a significant, negative weather event. Leadership teams should review the emergency prep checklist to determine specific actions that must be taken in advance of initiating shutdown procedures. Ensure on-site staff have the emergency equipment, food/water supplies, and other necessary equipment they need. Confirm that off-site staff can carry out critical work remotely.
• 96 hours prior — Continue to update staff on developing weather conditions and evacuation procedures should they become necessary. Review business continuity plans to confirm scheduling, tools, and supplies are available. Escalate any deficiencies to executive management and the business continuity and disaster recovery teams.
• 72 hours prior — Identify evacuation zones that may impact employees’ homes and data center locations. Secure offices and begin data replication, ensuring all data is stored in a remote and secure location. Share plans with customers to allay concerns. Open an operations technical bridge between required parties.
• 48 hours prior — Determine which employees need assistance preparing for and possibly evacuating in advance of the storm. (Employees should be encouraged to conduct parallel disaster preparedness planning for their two- and four-legged family members. The last thing anyone wants to see is employees' uncertainty for their families when they may be involved in critical restoration activities.) Identify additional equipment to be relocated. Secure facility and begin shutdown procedures (e.g., back up all data backed up and shut down equipment; cover computers and machinery with water-proof covers; deactivate/disconnect all noncritical, nonessential, and sensitive electrical equipment).
• 24 hours prior — Leadership to discuss final preparations and post-storm recovery plans. Conduct final site walkthrough. Monitor government weather/emergency services alerts and local travel restrictions.
• Reentry — Keep in mind that the DHS Cybersecurity and Infrastructure Security Agency (CISA) plays an important role in disaster response. They have developed the Essential Critical Infrastructure Workers list and also issue access and reentry letters for workers at essential infrastructure locations.
• Post-landfall recovery — Contact employees to determine their safety and status. Determine which employees can return to work and who should remain remote. Ensure facilities are safe from hazards, such as electrical wires, gas leaks, etc. Take photos to document damages and loss.

The Matrix — Emergency prep plans have strong commonalities regardless of geography or event. Neo and Morpheus aside, implementing a risk-rating matrix can help you determine the top risks specific to a given location and inform how investments are made to ensure you have a resilient system. Along these lines, if your company is global, make sure that your plans meet (and preferably exceed) local regulations.

Practice, practice, practice — You should be running practice drills within your organization once a year at minimum. Twice is even better if you want to build a culture of preparedness within the company. You likely have an annual compliance requirement to ensure your business continuity and emergency action plans are kept up-to-date, but it’s also a good idea to review your plan in the wake of a hurricane or other disaster to determine what did and didn’t work well and how those items could be improved upon.

Participate in your own rescue — Take advantage of the many tools and services that exist solely for the purpose of tracking and alerting people to severe weather. The Federal Emergency Management Agency (FEMA), for instance, has a suite of free tools to help maintain local emergency situational awareness. The FEMA app provides alerts in real-time from the National Weather Service, while the Emergency Alert System and Wireless Emergency Alert provide severe weather updates (no sign up required). Consider becoming a member of the National Business Emergency Operations Center (NBEOC), a clearinghouse to enhance information sharing between private and public entities before, during, and after disasters. If you want something tailored to your business needs, you might want to think about contracting with a company that can monitor your employees’ and managers’ whereabouts in a storm and provide them with critical alerts surrounding severe weather and other emergent situations.

Lock the doors — This applies both literally and figuratively as ne’er-do-wells are poised to take advantage of the chaos that follows any disaster. Ensure your facility has adequate protection from looters and other criminals, as well as strong cybersecurity solutions in place that will keep hackers out. Train your staff to be on the alert for phishing emails, especially those that purport to be soliciting aid for victims affected by the storm.

The increased frequency and severity of weather events requires a more comprehensive and measured approach to emergency planning and management. However, solid planning and familiarity with the risks to your facilities, coupled with employee education, will go a long way to minimizing losses and speeding recovery.

Matt Wombacher

Matt Wombacher is director of business continuity and disaster recovery at Cyxtera.