Resilience in Action: Business Continuity and Remediation Post-CrowdStrike

Today, a faulty update to CrowdStrike's security software caused global computer outages, affecting various services.

This incident has indeed been one of the largest IT outages in history, affecting around a billion computers worldwide, primarily corporate ones. Here's an overview of what happened:

The Incident

On July 19, 2024, a routine software update from CrowdStrike, a leading cybersecurity company, caused a massive global IT outage. The update, intended to enhance security, contained a defect that led to widespread disruptions.

Affected Businesses

The outage affected almost every major business sector, including:

• Airlines

  Major airlines like American Airlines, Delta Air Lines, and United Airlines issued ground stops, causing severe delays and cancellations. Airports worldwide experienced long lines as electronic systems failed, and staff had to check in passengers manually. 

• Banks

  Customers in Australia, New Zealand, and other regions reported issues accessing their accounts at major retail banks. 

• Retail

  McDonald’s in Japan closed some stores due to cash register malfunctions, and the British grocery chain Waitrose had to accept only cash.

• Law Enforcement:

  Agencies like the Alaska State Troopers reported issues, with 911 services temporarily not working.

• Media:

  The British broadcaster Sky News was briefly knocked off the air.

The flaw in the CrowdStrike update triggered a widespread occurrence of the infamous Blue Screen of Death (BSOD) on Windows operating systems. This wasn't merely a temporary online service disruption; each impacted computer had to undergo a reboot in fail mode, requiring manual removal of a driver.

Business Continuity Strategies

To ensure business continuity in the face of such incidents, organizations must adopt the following strategies: 

1. Comprehensive Backup Plans: Regularly backing up critical data and systems ensures that organizations can quickly restore operations in the event of an outage. Utilizing cloud-based backup solutions, such as AWS Backup, can centralize and automate the backup process, providing a reliable safety net. 

2. Redundant Systems: Implementing redundant systems and failover mechanisms can minimize downtime. By having backup servers and alternative communication channels in place, businesses can continue operations even if primary systems are compromised. 

3. Incident Response Plans: Developing and regularly updating incident response plans is crucial. These plans should outline the steps to be taken in the event of a cybersecurity incident, including communication protocols, roles and responsibilities, and recovery procedures.

4. Employee Training: Ensuring that employees are trained to recognize and respond to cybersecurity threats can prevent incidents from escalating. Regular training sessions and simulated attack exercises can enhance preparedness.

Here are the 5 key things to include in your ransomware incident response plan!

The Importance of Redundant Systems in Business Continuity

In the wake of this incident, the significance of redundant systems has become more apparent than ever. Redundant systems and failover mechanisms are critical components of a robust business continuity plan. They ensure that businesses can continue operations even if primary systems are compromised, thereby minimizing downtime and mitigating the impact of IT outages. 

What Are Redundant Systems?

Redundant systems refer to backup systems and resources that can take over the functions of primary systems in the event of a failure. These systems are designed to provide continuous availability and reliability, ensuring that business operations are not disrupted. Redundant systems can include:

• Backup Servers: Secondary servers that can take over the workload if the primary servers fail.
• Alternative Communication Channels: Backup communication systems that can be used if the primary communication channels are compromised.
• Data Replication: Copying data to multiple locations to ensure that it is always available, even if one location is affected.
  
  

The Role of Redundant Systems in Minimizing Downtime

Remediation Measures

In the aftermath of the CrowdStrike incident, effective remediation measures are essential to restore normalcy and prevent future occurrences:

1. Swift Identification and Fix Deployment: CrowdStrike’s quick identification of the defect and deployment of a fix was a critical first step. Organizations must have processes in place to rapidly detect and address vulnerabilities.

2. Manual Intervention: The need for manual driver removal highlights the importance of having skilled IT support teams. These teams should be equipped to handle complex recovery tasks and work efficiently under pressure.

3. Patch Management: Applying patches and updates promptly is vital to maintaining system security. Organizations should have a robust patch management process to ensure that all systems are up-to-date and protected against known vulnerabilities.

4. Testing and Validation: Rigorous testing and validation of software updates can prevent similar incidents in the future. Organizations should conduct thorough testing in controlled environments before deploying updates to production systems.
   
   

Expertise in AWS Backup and Restore

At Datalink Networks, we specialize in providing comprehensive AWS backup and restore services. Our expertise can help you quickly recover from any disruptions caused by cybersecurity incidents. Here are some ways we can assist:

1. Centralized Backup Management: Datalink Networks leverages AWS Backup, a fully managed backup service that centralizes and automates the backup of data across AWS services. This includes Amazon, EBS volumes, Amazon RDS databases, Amazon DynamoDB tables, and more. By centralizing backup management, Datalink Networks ensures that all your critical data is protected and easily recoverable. 

2. Efficient Restore Processes: In the event of an incident, Datalink Networks can initiate restore jobs using AWS Backup's StartRestoreJob API operation. This allows for the quick and efficient restoration of impacted resources. Whether it's restoring from warm or cold storage, Datalink Networks ensures minimal downtime and data loss. 

3. Non-Destructive Restores: To protect your existing resources, Datalink Networks performs non-destructive restores. This means that a new resource is created with the backup being restores, ensuring that your original resources remain intact. This approach minimizes the risk of further disruptions during the restore process.

4. Restore Testing: Datalink Networks conducts restore testing to simulate restore experiences and ensure that your organization meets its Restore Time Objective (RTO). This proactive approach helps prepare for future restore needs and ensures that your business can quickly recover from any incidents.

5. Tag Management: During the restore process, Datalink Networks can copy tags from the original backed-up resources to the restored resources. This helps maintain consistency and simplifies resource management after a restore.

Conclusion

The CrowdStrike incident serves as a stark reminder of the potential risks associated with technology and the importance of business continuity and remediation strategies. By adopting comprehensive backup plans, implementing redundant systems, developing incident response plans, and ensuring effective remediation measures, organizations can navigate the challenges posed by cybersecurity incidents and maintain operational resilience. 

Partnering with Datalink Networks

By partnering with Datalink Networks, you gain access to a team of experts dedicated to ensuring your business's resilience in the face of cybersecurity incidents. Our comprehensive AWS backup and restores services provide peace of mind, knowing that your critical data and resources are protected and recoverable, even in the face of unprecedented IT outages. 

For more information on how Datalink Networks can assist with AWS restores, visit our page here!