Thread regarding Microsoft Corp. layoffs

So, this is what happened for real…

Microsoft 365 Principal SRE here, and I can provide some context regarding the outage…

At approximately 21:40 UTC, an engineer deployed a routine configuration change to Azure Front Door that inadvertently redirected our authentication traffic to a SharePoint site last updated in 2017. We immediately initiated a rollback, but the rollback pipeline requires MFA approval through Microsoft Authenticator, which is currently waiting on a Conditional Access policy evaluation from the same infrastructure we're trying to restore.

We have a break-glass admin account, but its password is stored in a OneNote notebook on the affected tenant. Our on-call engineer attempted a workaround using PowerShell, but the required module was deprecated last Tuesday, and its replacement requires a P2 license. We then opened a Sev A ticket with Microsoft Premier Support, who responded by asking us to run sfc /scannow and provide a screenshot. The prolonged recovery is primarily due to our change advisory board requiring a Teams meeting to authorize the rollback. The meeting organizer is stuck in the lobby, and the only person authorized to admit him was laid off in June.

Current mitigation involves migrating affected users to Microsoft 365 E5, which should resolve the issue from a revenue perspective. We have identified the root cause as customer misconfiguration. Next update in 30 minutes, assuming someone can get Outlook working.


by
| 14 views | | 7 replies (last 1 day ago) | Reply
Post ID: @OP+1m7f1av5g

7 replies (most recent on top)

LOL. Love this story.

by
| | Reply
Post ID: @148+1m7f1av5g

True story or not I am so glad I am retired and don't have to use Microsoft products for personal or business use anymore.

by
| | Reply
Post ID: @hy+1m7f1av5g

Companies, especially airlines, should move away from using Outlook in any business workflow. With AI, companies should be able to build something up that accomplishes whatever workflow they want. And they should not run it on Azure.

by
| | Reply
Post ID: @dq+1m7f1av5g

@an so what happened for real?

by
| | Reply
Post ID: @ap+1m7f1av5g

@am The post is a BS story. Sad that you didn't catch this.

by
| | Reply
Post ID: @an+1m7f1av5g

Most interesting. Thx for sharing

by
| | Reply
Post ID: @am+1m7f1av5g

I know it's meant to be funny, but I had a customer lock themselves out of a tenant through a not-too-dissimilar chain of events.

by
| | Reply
Post ID: @a1+1m7f1av5g

Post a reply

: