Healthcare recovering from CrowdStrike outage

Many affected health systems say they’re back up and running, days after a global IT outage caused by a faulty CrowdStrike update that took millions of Microsoft systems offline on July 19. The outage affected industries around the world – not least healthcare, where it caused some providers to revert to pen and paper after losing access to electronic health records and other mission-critical systems.

Getting computers to boot up beyond the blue screen of death took machine-by-machine troubleshooting from exhausted IT workers in many cases, and was also accomplished through large-scale deployments orchestrated by Crowdstrike, Microsoft and others.

As the dust settles – the event is still causing flight cancellations and delays for the airline industry – experts say the global disruption offers an object lesson in technology’s vulnerabilities, and should provoke hard questions about interoperable system design and providers’ operational preparedness during mass outages.

Risks to patient care

Over the weekend Mass General Brigham patient portal users were pleased to see the statement: “Mass General Brigham hospitals are open and seeing patients. All scheduled appointments and procedures will happen as planned on Monday, July 22.”

The critical incident message at the Royal Surrey NHS Foundation Trust health system in England disappeared Saturday morning when the critical incident was stepped down.

While Mass General Brigham remained open during the worldwide IT outage, and provided care to patients with urgent health concerns in the group’s clinics and emergency departments, all previously scheduled nonurgent surgeries, procedures and medical visits were canceled Friday. 

Being back to operational is hopefully a relief to patients like Doreen Richards, who told Channel 5 ABC Boston that she traveled to the city only to have her pre-op visit canceled by the outage, while others who had scheduled surgeries and life-saving therapies were delayed either at the hospital or at home.

Unlike in banking, the accidental outage poses bodily risks to consumer safety.

“It is a tech outage, not a cyberattack, so there is no risk to the safety of your money. Even issues accessing money would be temporary in nature until a fix is deployed,” Greg McBride, chief financial analyst at Bankrate.com told AARP.

Service credits and potential lawsuits

It appears that remuneration from CrowdStrike is likely to be in the form of returned payments for services, according to Business Insider, which took a look at the third-party vendor’s contract terms.

Consumers not treated fairly on returned payments for the loss of services are advised to take it up directly with the companies, such as when a canceled flight cannot be rebooked for a later date and the airline will only issue a credit.

While no major patient suits were quickly filed after Friday’s tech outage, and it can take time, like the lawsuit filed last week for a January 31 cyberattack on Lurie Children’s Hospital – patients have sued over other types of outages.

To hasten a return to operations, Microsoft worked with CrowdStrike and others to address the approximately 8.5 million devices affected.

“Since this event began, we’ve maintained ongoing communication with our customers, CrowdStrike and external developers to collect information and expedite solutions,” the company said in an online statement on Saturday.

“We recognize the disruption this problem has caused for businesses and in the daily routines of many individuals. Our focus is providing customers with technical guidance and support to safely bring disrupted systems back online.” 

Microsoft said it took steps to: 

• Engage with CrowdStrike to automate their work on developing a solution.
• Deploy hundreds of Microsoft engineers and experts to work directly with customers to restore services.
• Collaborate with other cloud providers and stakeholders, including Google Cloud Platform and Amazon Web Services to share awareness on the state of impact we are each seeing across the industry and inform ongoing conversations with CrowdStrike and customers. 
• Post manual remediation documentation and scripts.
• Keep customers informed of the latest status on the incident through the Azure Status Dashboard.

Important to restoring services at healthcare organizations and others on Azure, Microsoft said CrowdStrike helped it develop a scalable patch that would accelerate “a fix for CrowdStrike’s faulty update” customer assets on the cloud.

Too big to fail?

CrowdStrike’s outage may end up being the worst information-technology disaster in history, although major cloud providers have experienced outages in the past. In 2src17, Amazon S3 cloud went down, affecting the functionality of websites and applications across healthcare.

“This incident demonstrates the interconnected nature of our broad ecosystem — global cloud providers, software platforms, security vendors and other software vendors and customers,” Microsoft acknowledged in its statement Saturday.

“It’s also a reminder of how important it is for all of us across the tech ecosystem to prioritize operating with safe deployment and disaster recovery using the mechanisms that exist.”

The event, which did not affect systems that were not using CrowdStrike such as nearly all systems in China, was not unimaginable. 

Modern social systems “have been designed for hyperconnected optimization, not decentralized resilience” and this event should be regarded as a warning, according to The Atlantic.

For healthcare, with its preponderance of third-party providers, it’s another opportunity to test its contingency plans to minimize disruptions to patients’ lives.

Andrea Fox is senior editor of Healthcare IT News.
Email: afox@himss.org
Healthcare IT News is a HIMSS Media publication.

The HIMSS Healthcare Cybersecurity Forum is scheduled to take place October 31-November 1 in Washington, D.C. Learn more and register.