R3

What the CrowdStrike Outage Taught Us

Why Partnering with an IT MSP and MSSP is Essential in Today’s Complex Business Landscape

It would be an understatement to say that the world is becoming “increasingly digital” – the world is digital. Every business, in every industry, relies on technology for its operations. And in today’s rapidly evolving digital landscape, businesses face increasing complexities and challenges that make it clear no single software solution can be trusted entirely on its own.

In the recent massive IT outage caused by a CrowdStrike update glitch serves as a stark reminder of the importance of having an experienced IT Managed Service Provider (MSP) and Managed Security Service Provider (MSSP) to navigate these complexities. This blog post will explore the necessity of a strategic IT partner, the pitfalls of relying solely on software solutions, and best practices for disaster recovery and preparedness.

The CrowdStrike Outage: A Case Study

On July 19, 2024, a significant IT outage struck businesses worldwide, triggered by a defective update from CrowdStrike, a prominent cybersecurity firm. This glitch left Windows computers unable to start up, causing widespread disruptions across various industries, including airlines, banks, and other businesses. While not caused by a cyberattack, the defect in the update highlighted the vulnerability of even the most robust cybersecurity solutions when not properly managed and monitored.

Details of the Outage:

Machines running Microsoft’s Windows operating system crashed due to a fault in the way a software update was issued by CrowdStrike. This resulted in massive downtime and operational disruptions, affecting flights, banking services, and many other sectors. Without the expertise of an experienced IT team, diagnosing and mitigating the issue could lead to prolonged downtime and substantial losses.

An update pushed out late in the night by Crowdstrike for its Falcon sensor contains a bug that is crashing windows PCs and leaving users with the dreadfully iconic Blue Screen of Death (BSOD). As many in the world are waking up to discover what’s going on teams at Crowdstrike, and all of their customers, are rushing to use a workaround to get people back online. 

Workaround Steps:

  1. Boot Windows into Safe Mode or the Windows Recovery Environment
  2. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
  3. Locate the file matching “C-00000291*.sys”, and delete it.
  4. You can also just delete as well: c:\Windows\System32\drivers\Crowdstrike\C-*
  5. Reboot
  6. If that doesn’t work and you’re drive is locked run this command:
    1. manage-bde.exe -unlock -recoverypassword <<BITLOCKER KEY>> c:

This incident underscores the need for an IT MSP and MSSP partner who can provide strategic oversight, swift resolution, and proactive monitoring to minimize impact.

Kaspersky Software Ban: Another Example

Similarly, the 2024 ban on Kaspersky software by the U.S. government due to concerns over potential espionage exemplifies the risks associated with relying on single-vendor solutions. Businesses using Kaspersky had to quickly pivot to alternative security measures to comply with the ban and protect their data. This situation demonstrated the importance of having a well-rounded IT strategy and the ability to adapt to regulatory changes swiftly.

Software Complexity and Inevitable Complications 

As our digital landscape evolves, software solutions are becoming increasingly complex and sophisticated to meet the growing demands of cybersecurity and business operations. This complexity is reflected in exponentially larger codebases, intricate interdependencies between components, and the integration of advanced technologies like AI and machine learning. While these advancements offer powerful capabilities, they also introduce a higher likelihood of unforeseen issues and vulnerabilities. The recent CrowdStrike incident serves as a reminder that even industry-leading solutions can face challenges due to this complexity. As we move forward, it’s probable that we’ll encounter more situations like this across various software platforms. This underscores the critical importance of robust testing procedures, rapid response capabilities, and the need for businesses to have contingency plans in place. It also highlights the value of partnering with managed service providers who can navigate these complexities and provide swift, expert assistance when issues arise.

The Need for a Custom Solution

These examples illustrate that there is no silver bullet when it comes to IT security and management. Businesses need more than just software; they need an experienced team that develops custom solutions tailored to their unique needs. An effective IT MSP and MSSP partner offers:

  1. Strategic Planning: Developing a comprehensive IT strategy that aligns with your business goals and prepares for potential disruptions.
  2. Customized Solutions: Implementing tailored solutions rather than one-size-fits-all software to address specific business challenges.
  3. Proactive Monitoring: Continuously monitoring systems to detect and resolve issues before they escalate.
  4. Swift Response: Providing rapid response and recovery efforts during outages or attacks to minimize downtime.

Best Practices for Disaster Preparedness

To ensure your business is prepared for situations like the CrowdStrike outage, consider the following best practices:

  1. Regular Backups: Maintain regular backups of critical data and systems to enable quick recovery in the event of an outage.
  2. Incident Response Plan: Develop and regularly update an incident response plan that outlines procedures for various types of disruptions.
  3. Vendor Diversification: Avoid over-reliance on a single vendor by diversifying your IT and cybersecurity solutions.
  4. Employee Training: Educate employees on recognizing potential threats and following best practices for cybersecurity.
  5. Regular Audits: Conduct regular audits of your IT infrastructure to identify vulnerabilities and areas for improvement.

In conclusion, the complexities of today’s business environment necessitate more than just reliance on software solutions. Partnering with an experienced IT MSP and MSSP provides the strategic insight, customized solutions, and rapid response needed to navigate these challenges effectively. By adopting best practices and having a trusted IT partner, businesses can better prepare for and respond to disruptions, ensuring continuity and resilience.

If you’ve found your organization in a situation where you’ve become overly reliant on the reliability of a third-party software then it might be a good time to start a discussion with a member of the R3 team for a free consultation about your tech stack.

Chat with us here. 

What the CrowdStrike Outage Taught Us