[warp_10]

Multiple failures (or potentially related) between the microsoft 365 failure and the crowdstrike falcon sensor have caused a major internet outage similar to the Live Free or Die Hard cyber attack (just that here it was not intentional or malicious).

https://www.forbes.com/sites/kateofl...at-to-do-next/

Airports, airlines, banks, rails have been reported to have ended with a collateral. Indigo passengers, Changi desks have reported to have issued manual boarding passes to keep the passengers moving.

Just shows how much we have become reliant on these systems and cloud apps that when intended things can be brought to a standstill.

[vvrchandra]

Pl see reply from CrowdStrike ..

[warp_10]

Originally Posted by vvrchandra

Pl see reply from CrowdStrike ..

Crowdstrike will be under a lot of gun on this for sure.

PS: They are down down 15-20% in premarket on this. Short traders must be making merry !

[dgindia]

No matter how comfortable one is with usage of automated processes, everyone must have plan B to continue work in case of unforeseen circumstances.

They should purposefully train people to work with manual system to keep them ready to face any challenging situations.

[volkman10]

Tech disruptions worldwide have hit airlines, banks and businesses, which are scrambling to respond.

Major US carriers, including Delta, United and American Airlines, have had flights grounded, and airlines in Europe and Asia-Pacific region have also seen disruptions. Banks in Australia, New Zealand, South Africa and Britain have been impacted, as have health services in Israel and the UK.

Multiple Asian airlines and airports have been affected by the global IT outage.
One of Europe's largest medical facilities affected by outage.

Link:

[HillMan]

If someone is still facing the issue here are some recommended workaround steps to be followed:

Option 1 - For Workstations
Reboot the Workstations thrice so that it starts working as expected(BSOD)

Option 2 - Workaround for Windows workstations
Boot Windows into Safe Mode or the Windows Recovery Environment
Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
Locate the file matching “C-00000291*.sys”, and delete it.
Boot the host normally.
Note: Bitlocker -encrypted hosts may require a recovery key. Please reach out to your Local IT Admins for recovery key

Option 3 - (if users are experiencing a blue screen(BSOD))
With the workstation powered off, hold down the F5 key and power the workstation on
Hold the F5 key down until you see a message at the bottom of the screen saying it is diagnosing issues
Release the F5 key and let the workstation attempt repairs
When repairs are completed, it will reboot normally

Option 4:
Boot the workstation into Safe Mode and do a System Point restore to the last know good configuration.
When the restore is complete, it will reboot normally


Even though the solution is found. The world woke up with the jolt today to realise how connected we all are and how easily someone can bring down the entire infrastructure to its knees.

[chiekennugget]

https://www.motorsportweek.com/2024/...-hungarian-gp/

Even the Mercedes F1 team is facing heat from the windows outage.

Several IT outages have wreaked havoc across the globe overnight with airlines, banks and shops all impacted alongside Mercedes’ current F1 weekend preparations.

CrowdStrike, which has supplied Mercedes since 2019, revealed that a “defect” with a content update has been behind the disruption that has become headline news.

The disturbance is also thought to have hampered Mercedes’ three engine customers – Aston Martin, McLaren and Williams – but other teams haven’t been hampered.

[amol4184]

Originally Posted by dgindia

They should purposefully train people to work with manual system to keep them ready to face any challenging situations.

Unfortunately there is nothing that can be done manually.
This is akin to trying to start a car without engine in it.

[fhdowntheline]

It is confounding that there is such hyper dependency on one software component and how tightly it is integrated with the Windows platform. It also puts a question mark as to how we are suffering from lack of choice in terms of general computing platforms. Post covid, there was a lot of talk about reducing inter-dependencies. I think businesses need to seriously think about encouraging diverse IT platforms with different configurations for the sake of redundancy. Everything is a binary- MS vs Apple, Android vs IOS, Boeing vs Airbus !!

PS: Crowdstrike literally lived up to its name

[Samurai]

Originally Posted by fhdowntheline

It is confounding that there is such hyper dependency on one software component and how tightly it is integrated with the Windows platform.

Windows is not dependent on it.

Windows PCs in my company don't have CrowdStrike installed on it. It is an optional security software used by many companies. If you don't use it, there is no impact from this bug.

[ajmat]

My company had no issues although some degradation in services happened on some SaaS platforms

[vamsi.vadrevu]

Looking at all these re-affirms my belief that business critical systems should always be on Linux machines with multiple backups. I've read news about some pharmacies in USA not being able to process prescription refills because their billing systems were impacted! It underlines the importance of plan b and business continuity plans. We regularly perform business continuity plans drills in case of total system failure. We also account for 3rd party systems being out of action. I've noticed that some organizations do not even consider the fact that a 3rd party system on which they're dependent on might go down.

Windows as a platform has really become bloated to the point it is insufferably inefficient. With all the cloud hosting of most applications it is time organisations started to move out of windows platform. There would be minimal impact to end user because almost all apps are cloud hosted and end users would just be using a browser to access their apps.

If the airline operations were to be considered as emergency services, they really should have had a backup plan for outages. It is expensive to maintain a second instance of your entire platform but that's the price to pay for zero downtime. Sadly I do not see any organisations doing this. I only know of a few sectors that actually take this seriously. Banking, insurance and defence domains tend to have robust alternative mechanisms in place.

[shivshanker]

Originally Posted by Samurai

Windows is not dependent on it.

Windows PCs in my company don't have CrowdStrike installed on it. It is an optional security software used by many companies. If you don't use it, there is no impact from this bug.

In my humble opinion, (I am old school, still doing Mainframes)

This is a fallout on account of desire for "automation", Integrating DevOps with into CI/CD pipeline with the ultimate desire to cut costs.

An untested patch or module deployment is unheard of in my world.

[AltoLXI]

Failure to include a null check.

[Selective]

Let me share my thoughts:

Crowdstrike builds Endpoint Detection & Response (EDR) product and EDR agents will be deployed on all the laptops, servers, etc. There are many other EDR companies, Crowdstrike is one of the popular vendor. This EDR agent runs in kernel mode which is very sensitive part of OS. Some bad programming in kernel mode device driver can lead to BSOD (Blue Screen Of Death). BSOD is very popular in the context of Windows OS. Though BSOD is not rare, but not that frequent as well. All the changes in device driver MUST go through stringent testing.

Microsoft is not directly involved in this. Though, Microsoft should build the methods to protect the OS/system from BSOD and handle this scenario gracefully. This is long pending. Does that mean Linux is safe? No, just that it doesn’t show Blue Screen Of Death! I guess, Linux handles this gracefully.

The biggest surprise (for me) is, how did this defect escape from the pre-production validation consistently across all the companies??? This raises the question mark on the pre-production process across all the companies. The scale at which this has spread, this is definitely not a corner case. If you look at the banking sector, not many have failed. I know there are many of them who have stringent pre-production testing to catch such defects.