Thread regarding Verizon Communications Inc. layoffs

FAA outage explanation

Hey engineers, did the router overload explanation from the FAA make sense? If thats true, why is this the first time it happened?


by
| 156 views | | 14 replies (last 3 days ago) | Reply
Post ID: @OP+1m7bw0dyh

14 replies (most recent on top)

Are not routers designed for max throughput of 70-80%.along with redundancy? Would not a robust firewall catch a denial of service attack?

by
| | Reply
Post ID: @wq+1m7bw0dyh

This is what happens when you continue to lay off long time experienced engineers replace them with lower paid lack of knowledge engineers and think AI will save the day to try and save a buck! Get ready for more of this to continue!

by
| | Reply
Post ID: @tr+1m7bw0dyh

@dh The dedicated engineer who prevents these problems from occurring.

@fb @df @de The engineers responsible for creating these problems.

by
| | Reply
Post ID: @he+1m7bw0dyh

A simultaneous, nationwide failure of every FAA air traffic control tower, radar facility, and surveillance system on the same day it is astronomically unlikely.

Key points:

Statistical likelihood: If each failure were independent, the probability would be below 0.000001% (less than 1 in 100 million).
Built-in redundancy: The FAA’s architecture is designed so that the loss of any single radar, tower, power source, or communications circuit does not automatically create a safety crisis.
What it would take: A nationwide loss of operational capability would require an extraordinary combination of failures or a major common-cause event, such as a coordinated attack or a catastrophic systemic fault.

Bottom line: While no complex system is completely immune to failure, the layered safeguards in the U.S. air traffic control network make a total, simultaneous outage effectively an impossible scenario.

Now, combine this with issues at Amtrak, and Microsoft ... all in the same day ... yeah, me thinks it was something else...

by
| | Reply
Post ID: @fb+1m7bw0dyh

@OP Relax, all right? My old man is a television repairman, he's got this ultimate set of tools. I can fix it.

by
| | Reply
Post ID: @e5+1m7bw0dyh

@df
I didn't really consider the "overloaded routers" but if the routers were using OSPF protocol there could have been a "link state advertisement" routing storm, which consumes all the available resources and the routers stop passing customer traffic. Sometimes the only way to recover from an OSPF routing storm is to shut all the routers down and then power up in a controlled fashion. That would explain why it took so long to restore service.

by
| | Reply
Post ID: @dh+1m7bw0dyh

Yes, the FAA's explanation is entirely plausible from an engineering standpoint. In fact, these types of network failures happen more often than most people realize, particularly in telecommunications environments where newer equipment is integrated into older, sometimes decades-old infrastructure. Having worked in telecom for decades, I've seen how something as seemingly straightforward as a hardware upgrade, routing change, or configuration issue can create unexpected problems that cascade well beyond the original point of failure. A router generating excessive network traffic and overwhelming other devices is certainly not far-fetched; it's a recognized failure scenario. The bigger concern here isn't necessarily that a piece of equipment malfunctioned, but why a network supporting something as critical as national air traffic control didn't have sufficient safeguards, redundancy, and isolation mechanisms to prevent one localized problem from affecting such a large portion of the system. Actually, I know the answer to this already. It's because our Gov't has put it's own interests before the public's as we've seen for years now. $50Billion is what they're asking for to upgrade the entire FAA infrastructure.

by
| | Reply
Post ID: @df+1m7bw0dyh

To perform an upgrade of physical facilities, switching from copper to fiber, on an active critical system requires deep planning and testing. One would hope that such a critical system has full redundancy including path redundancy. When starting the upgrade they should have upgraded the standby side, then brought that on line. If there was an issue you would immediately fail over to the original primary that has not been upgraded yet. This suggests to me that there was not full redundancy. Also consider that if you do a software upgrade, say on a router, and it hoses up the router, you reload the previous version of software and maybe a data backup so you can restore service quickly. If there are changes made to physical facilities and you have to roll back it will take a lot longer to do. Things that can cause events like this include the people not following the method of procedure to the letter, not stopping when there is an anomaly, insufficient planning or testing. If they were upgrading old gear, like stuff that is decades old the loss of organizational knowledge may have been a factor. Even if you adhere to the strictest change discipline protocol unexpected things can happen. I have no inside knowledge of what happened, but based on my 40 years of experience my guess is that either there was no redundancy, or redundancy failed after something went wrong with the upgrade and it took hours to reinstall the old gear.

by
| | Reply
Post ID: @de+1m7bw0dyh

Incompetency brought down the FAA network. Russia & China looked at the situation and decided we’d sc--w it up all by ourselves so there was no need for a cyber attack

by
| | Reply
Post ID: @db+1m7bw0dyh

Some unknown telecom in Washington DC..was replacing old copper wiring

https://www.bbc.com/news/live/c6z9dwn12037t

by
| | Reply
Post ID: @cr+1m7bw0dyh

Yes, router overloads are caused by mismanaged capacity, failover processes, and transport engineering. I am curious, which other network encounters this issue. Hmm

by
| | Reply
Post ID: @ag+1m7bw0dyh

@ab All lies and excuses that most of the public will believe. All B S.

by
| | Reply
Post ID: @af+1m7bw0dyh

From a telecom tech: “Copper lines DO NOT magically “go out” or get “overloaded”. They are talking about putting new technology on Old Lines and it working? No, that’s not how that works, quite the opposite actually, that new equipment wouldn’t work on old infrastructure, they are lying.”

by
| | Reply
Post ID: @ab+1m7bw0dyh

One things for sure. It definitely wasn’t a cyberattack.

by
| | Reply
Post ID: @a4+1m7bw0dyh

Post a reply

: