CrowdStrike's "Blue Screen of Death" - We don't play that.

In July 2024, CrowdStrike experienced a significant incident caused by a logic error in a routine sensor configuration update, which led to widespread "Blue Screen of Death" (BSOD) errors on Windows PCs globally. This error resulted in affected systems entering recovery boot loops and rendered them inoperative, disrupting operations for major sectors such as banking, travel, and telecommunications. Approximately 8.5 million devices were impacted, causing significant business disruptions and financial losses.

Benelogic and by extension our customers were fortunate NOT to be impacted by this issue.  We developed and follow best practices to try to anticipate and avoid these types of issues. 

Below is our five step approach to help mitigate these types of risks.

1. Test, Test, Test

• Comprehensive Testing: Ensure that every update undergoes extensive testing in a controlled environment that mirrors the production setting as closely as possible. This includes unit tests, integration tests, system tests, and user acceptance tests (UAT).

• Automated Testing: Implement automated testing frameworks to run repetitive tests efficiently, catching potential issues early in the development cycle.

• Stress Testing: Conduct stress tests to understand how the system performs under extreme conditions and ensure it can handle unexpected loads without failing.

• Security Testing: Regularly perform vulnerability assessments and penetration testing to identify and address security weaknesses before updates are deployed.

2. Control Release

• Controlled Rollout: Implement phased deployment strategies such as canary releases or blue-green deployments to roll out updates gradually. This approach minimizes the risk of widespread disruption by first releasing updates to a small subset of users.

• Feature Flags: Use feature flags to enable or disable new features without deploying new code. This allows for safe testing of features in the production environment and quick rollback if necessary.

• Change Management: Establish a robust change management process that includes detailed documentation, peer reviews, and approval workflows to ensure updates are meticulously planned and executed.

3. Stage Rollout

• Staged Deployment: Deploy updates in stages, starting with a small, controlled group of users or a particular geographical region before a full-scale release. This approach helps in identifying and resolving issues in a controlled manner.

• Monitoring and Feedback: Continuously monitor the performance and user feedback during each stage of the rollout. Implement mechanisms to quickly address any issues that arise during the initial stages.

• Incremental Updates: Prefer incremental updates over large, sweeping changes. Smaller updates are easier to manage, test, and rollback if issues occur.

  
  

4. Have Rollback Plans in Place

• Rollback Strategies: Develop and document clear rollback procedures for every update. This should include steps to revert to the previous stable version of the software quickly and efficiently.

• Backup Systems: Ensure that backups of critical systems and data are taken before any update is deployed. This guarantees that systems can be restored to their previous state if the update fails.

• Failover Systems: Implement failover mechanisms to switch to a backup system automatically in the event of an update failure, minimizing downtime and maintaining service continuity.

  
  

5. Use Incidences to Review for Vulnerabilities

• Post-Incident Analysis: Conduct thorough post-incident reviews (PIR) to understand the root cause of any issues that arise. This includes examining logs, error reports, and system behavior to pinpoint vulnerabilities.

• Continuous Improvement: Use the insights gained from incident reviews to improve testing, deployment, and monitoring processes. Implement changes to prevent similar issues in the future.

• Regular Audits: Schedule regular security audits and vulnerability assessments to identify and address potential weaknesses in the system proactively.

• Knowledge Sharing: Document lessons learned from incidents and share them across teams to foster a culture of continuous learning and improvement.

  
  

It may seem like a lot to go through every time we update or patch a program, but we firmly believe this disciplined approach is what safeguards our environment.  These steps create a more resilient and secure update process while minimizing the risk of disruption.