Preliminary post incident review: https://azure.status.microsoft/en-gb/status/history/ Timeline …

“Our protection mechanisms, to validate and block any erroneous deployments, failed due to a software defect which allowed the deployment to bypass safety validations.” Very circular way of saying “the validator didn’t do its job”. This is AFAICT a pretty fundamental root cause of the issue. It’s never good enough to have a validator check the content and hope that finds all the issues. Validators are great and can speed a lot of things up. But because they are independent code paths they will always miss something. For critical services you have to assume the validator will be wrong, and be prepared to contain the damage WHEN it is wrong.

What puzzles me too is the time it took to recognize an outage. Looks like there was no monitoring and no alerts. Which is kinda weird.

>> We began the investigation and started to examine configuration changes within AFD. Troubleshooting has completed Troubleshooting was unable to automatically fix all of the issues found. You can find more details below. >> We initiated the deployment of our ‘last known good’ configuration. System Restore can help fix problems that might be making your computer run slowly or stop responding. System Restore does not affect any of your documents, pictures, or other personal data. Recently installed programs and drivers might be uninstalled. Confirm your restore point Your computer will be restored to the state it was in before the event in the Description field below.