Hopefully, the COVID-19 pandemic will lead CIOs to recognize their important role in Crisis Planning, which goes beyond traditional Disaster Recovery, writes business technology veteran Bob DeRodes.
There is an old basketball saying that goes something like this: If you hang around the hoop long enough, eventually, you’re likely to get a rebound. In my case, if you’ve hung around a profession as long as I have you’re likely to have seen a number of bad things happen.
The COVID-19 pandemic has given me time to reflect back on those events and helped me to realize crucial truths about IT preparedness. CIOs claim we are ready for most any IT disaster, but we’ve proven time and again that we are not equally ready to deal with a major crisis in which IT is left virtually unscathed. Maybe now is the time to acknowledge that there is value in distinguishing a disaster from a crisis, and to broaden our sphere of influence to include planning for a wide variety of crises.
CIOs constantly wrestle with preparing their enterprises for the oft-feared-but-rarely-to-be-executed (let alone funded) disaster recovery (DR). From its inception, DR has meant recovery from a serious disablement of your IT ecosystem for any number of reasons, all lumped together under the label of “disaster”. (Admittedly, in the early days we weren’t using the term ecosystem.)
Disaster Recovery in Earlier Days
For the first couple decades of modern computing, DR was rarely a topic simply because the first processes automated were not deemed critical to the ongoing operation. It was believed we could always fallback and manually produce payroll. DR did not become an area of focus until the core business processes had been automated and enterprises were highly dependent on IT. I would argue that DR wasn’t a serious focus for business leaders, it was more of a blur dumped onto the CIO to figure out. Let’s be honest, at least in the Fortune 50 companies where I had personal experience, initially there was much more disaster than there was recovery! Even as DR planning improved over time, it is pretty clear that our plans were not designed for IT’s response to the various crisis that could cripple our business model; yet not affect the IT operation.
In the boardroom today, we hear less about DR and more about system resiliency, cyber security, and overall risk management. However, as a replacement for DR, resiliency has never resonated with me because resiliency is a goal, while recovery is an action – or better yet, a reaction. You can plan to achieve resiliency, but that’s different than a plan to get you through and out of an IT disaster. While the board practice of risk management is still evolving, it has not yet centered on IT’s response to the various risks. Could comprehensive crisis planning be falling through the gap?
This was the case when I was CIO for The Home Depot and Hurricane Katrina struck in September 2005. Facing major devastation to our stores along the Gulf Coast, we scrambled (an understatement) to get connectivity, devices, new applications and people into the field. Our DR Plan was of little help in this type of crisis. After Katrina faded, we knew we needed to be better prepared for any crisis, and built a formal IT Crisis Plan for reacting to future natural disasters… but we stopped there. Unfortunately, we didn’t plan an IT response to the next major crisis - a total collapse of the housing market some three years later.
by Tim Reed
History clearly shows an IT disaster can create a business crisis, but we have also seen where a business crisis actually creates a subsequent IT disaster, as it did for the airline industry on September 11, 2001. As CEO of Delta Technology and CIO for Delta Airlines on that dreadful day, I learned that lesson the hard way. After the FAA closed all airspace in and out of the USA, the core airline systems began to re-accommodate hundreds of planes and thousands of crew members, passengers, bags, cargo, catering, and maintenance activities to the next best alternative schedule. Since the next best alternative was never available, the systems would try rescheduling again and again. We were quickly running out of addressable storage space and we estimated that in less than 24 hours the core airline system would crash, something that had never happened since its inception. And, because all airlines share the same core technology built by IBM (in the late ‘60’s through the mid-70’s), all airlines were in the same state of shock. Worse yet, in the subsequent chaos we quickly discovered that a system restart was never designed into the real-time portion of the core airline systems. If we could not avoid the crash, we knew we would lose large amounts of data, and we had no idea how long it would take to rebuild and restart the industry. We were in the midst of the worst modern day crisis, and IT was drifting in totally uncharted waters… beyond any IT disaster we ever imagined. In that moment of gut-wrenching pandemonium, we found ourselves without a comprehensive DR plan, and without a plan to provide the IT services the business needed to deal with the actual crisis. Luckily, we had three days before the airline industry would restart again, time enough to get IT support to the crisis teams and to figure out how to restart and help save the airline industry.
Crisis Response Planning
Now, today we are facing the COVID-19 crisis, and who would have ever thought vast amounts of our economy would be shut down and we would simply shelter in place? Undoubtedly, this has become a major crisis; yet there is no associated disaster in the IT ecosystem. Regardless, once again IT is being called upon to respond quickly to the new normal, and our DR plans are of marginal help. CIOs are scrambling to redirect resources, create secure work-at-home environments, distribute laptops and software, adjust workloads, and plan on the fly for whatever may happen next. In that light, I ask you to imagine what could happen if we lost Azure, AWS, or Google cloud capabilities… imagine a major physical attack or a cyber attack on our VPN or Internet infrastructures… just imagine throwing an IT disaster on top of this crisis! Are you ready?
Maybe one of the bright spots that could come from this COVID-19 crisis would be that CIOs recognize they need to play a bigger role in crisis planning, expanding beyond our current disaster recovery horizons. We could become a catalyst to helping business leaders think through more and varied types of crisis, and leading the efforts to build more detailed Crisis Response Plans. These plans would be distinct from our DR Plans, but undoubtedly connected, with unique threads that require separate and unique actions. Either we learn from our history, or we are just standing under the hoop waiting for the ball to fall on our heads.