#reliability

Can we talk about our failure modes?

Our core product has exactly two states: working, or completely down. There is no middle. No degraded mode, no read-only fallback, no "feature X is unavailable but the rest still works." When something breaks, it doesn't inconvenience customers, it stops their business entirely.

That is a design choice, not an inevitability. And I think we keep making it because the honest answer to "why is it built this way" is "because it's always been built this way."

Every dependency we have is currently a single point of total failure. A hiccup in one subsystem takes down the whole thing. We treat that as an ops problem to be monitored and paged on, when it's actually an architecture problem we've decided not to solve.

Graceful degradation costs something upfront. You have to define what "partial" means for each subsystem, build the fallback paths, decide what's safe to shed under load, and actually test the failure modes. That's real work. But the alternative is what we have now: every incident is a worst-case incident.

I'm not asking for a ground-up rewrite. I'm asking us to tier our failures. Which ones should be invisible to customers, which should be a minor inconvenience, and which are genuinely catastrophic? I'd bet most of what currently triggers a full outage belongs in the "inconvenient" bucket and could be isolated without rebuilding everything.

We accept this as normal. We shouldn't....

Ford is facing a growing nightmare as massive recalls, expensive repairs, and reliability complaints continue spreading across America. Thousands of owners are discovering that some vehicles they trusted are now turning into financial disasters filled with breakdowns, warning lights, and terrifying repair costs. Dealers are quietly struggling to calm angry customers while mechanics warn that certain models may only get worse as they age. Older drivers especially are starting to question whether Ford still builds vehicles tough enough to last long term. And according to insiders, the recall chaos surrounding some of these vehicles could become even more damaging before 2026 is over.

No VP and no resilience/reliability…..

The reasoning behind Accelera pulling out of the industry was - per management the absence of a market.
Strange, regularly you can read about new project starting up and equipment being ordered. 300 MW order placed today for a plant in Spain. Accelera backyard!

Truth is that Accelera pulls out after repeated mismanagement and huge unreliability of the stacks.
Competition takes their time to develop a working product, Accelera chased the 1 and 2 GW projects while they could not deliver a H500 on time, let alone it worked.
Delusional!

Verizon's massive nationwide network outage on Wednesday may have been caused by a failure in just one East Coast state.

Officials believe a network server in New Jersey going down was the likely trigger of the day-long network crash, according to an initial investigation by law enforcement agencies on the East Coast searching for signs of sabotage.

New York State Assembly member Anil Beephan has called on the Federal Communications Commission (FCC) to open a probe into the outage, but no signs of tampering or hacking by cybercriminals have been found so far.

However, James Knight of DigitalWarfare.com told the Daily Mail that cyberwarfare experts are very suspicious about this outage being able to spread across the entire US within minutes.

'True single-point failures shouldn't cascade this way in a properly engineered system, and the silence on exact causes only heightens doubts,' Knight explained.

Knight added Verizon's 'built-in redundancies,' including spread out data centers, constant system tests, and multiple routing paths for signals, should have prevented this kind of long-term and widespread service blackout.

'That said, there are no credible signs or evidence this was cyberwarfare, a cyberattack, or foreign interference,' the cyberwarfare expert noted.

'Everyone I've spoken to is either tight-lipped or suspicious,' he told Daily Mail.

Despite the timing, no groups or nations have claimed responsibility for any kind of potential attack on Verizon's server, which Knight said would have been typical for disruptive actors seeking visibility for a major hack.

The telecommunications giant has yet to provide any details on the exact cause of the mysterious blackout, leaving customers without the ability to make calls or send text messages.

On Thursday, Verizon told Daily Mail that all customers affected by the outage would receive a $20 credit to their account, which they will need to redeem using the myVerizon app.

https://abcnews.go.com/Business/verizon-outage-affecting-175000-customers/story?id=129212330

Verizon outage affecting thousands of customers -- Verizon issued an apology "for the inconvenience.” January 14, 2026, 4:16 PM 🤣🤣🤣

Verizon outage affecting thousands of customers Verizon said it was not immediately clear how long the service would be down. Some Verizon customers were experiencing a service outage on Wednesday afternoon, according to the company.

"We are aware of an issue impacting wireless voice and data services for some customers," Verizon said in a statement to ABC News. "Our engineers are engaged and are working to identify and solve the issue quickly. We understand how important reliable connectivity is and apologize for the inconvenience."

Many Verizon customers said on social media that their phones showed "SOS" in place of network bars.

According to Downdetector, at least 175,000 Verizon customers were affected at one point, but that number has since gone down. Downdetector, a site that tracks outages, said Verizon customers began noticing interrupted service around noon Eastern time.

By 3:30 p.m. Eastern time, the number of Verizon customers affected by the outage was down to less than 60,000, according to Downdetector.

I heard yesterday from a colleague that every January your reliability automatically resets to 100 percent. Is that true?

Long gone are the days when people dreamed of working at Intel. Now we can’t even attract mediocre employees, and the top talent is avoiding us in wide berths. I fear this kind of loss of reputation is irreparable and we’ll be feeling it for a long time to come.

Just saw an ethics alert posted on Inside, and thought it was for sure about good ole LC.

What he has done is far worse than the 3 cases described in the article. What will RB do? MN letting him do whatever he wants does have consequences. Lining his friends pockets while IT reliability goes down. What an excellent choice EB made when she hired this loser.