About 10 years ago we launched 2 new data centers. When JF took over as a CEO, I heard someone in IT saying JF wants everything to go to external cloud.
In light of today’s AWS outage, what do IT people think of JF’s view and the danger of external cloud service?
Are we still using the 2 data centers?
@sv+1k825ed6j That would be aces if there was iterative development built on a solid core but instead there is a jello core with layers of whipped cream and sprinkles on top. As the jello shakes and the whipped cream slides off yet another initiative sprays toppings on the jello, ergo the never ending initiatives. The core is never addressed.
Bad CIO decision to build and manage our own data center. We gave her the golden parachute and still stuck with the big bills and trying to sell it
Two years is a infinite long time. For internal developed projects, the moment those projects are overall, technology refresh projects are on the table. Nothing ends.
The situation is worse than what people leaders report.
The cloud provider(s) and service provider(s) have a honeymoon period of typically two years after that the service degrades significantly and prices increase significantly. We have even had providers essentially holding our data hostage.
Thus the churn from one cloud provider / service provider to another. Since Ford people leaders are also rotating to new positions every couple of years they are never held responsible for their choices and actions. Chasing the cheapest option ends up costing the company more in the long run.
I'm sure they are still be used today, but not sure the criteria. At one time, Ford IT had a directive of 'data center first', which made it ridiculously difficult to get any type of Software as a Service product, even for a trial. They were basically scared of the cloud even though their suppliers had their data (and Ford's) there. Then Ford and Google entering into an agreement which included using Google's cloud platform, smartly called Google Cloud Platform. Magically, GCP became the primary hosting environment because there were usage quotas as part of the agreement. There was a mad rush to migrate everything from other cloud services and many applications from the data centers to GCP. That didn't help if you wanted SaaS service because they use THEIR cloud. Even if it the service was portable to GCP (many were supported in AWS and Azure, but not GPC), it would not work out because the vendor could no longer provide an SLA. As far as Ford's data 'shack', I doubt that there would be enough capacity worth renting out to anyone.
Data is one of the most important assets of the company today. Although the revenue generating power of data does not get discussed in this forum as it used to, nevertheless, data is still a part of the company's business model going forward.
@OP Oh, yeah! Ten years ago, all managers at FMC were throwing the word "cloud", like they do now with "AI", as the "magic buzzword". We had apps in Azure, AWS, GCP and PCF. Every couple of years is a "new" migration to another cloud provider. Whoever is the cheapest, or give us a "discount".
But, yes, we still have those datacenters, and some of the cloud apps were migrated from the cloud back to those datacenters to save money.
Now, any datacenter can fail. Any cloud provider can fail. Even with all the protections and redundancies. And the consequences are vast, due to the huge number of apps concentrated in there. That's why IT wanted to keep the local IT rooms inside every plant, so the outages are local, and not company wide.
However, the mighty dollar strikes again at Ford, and the company has been moving some apps out of the plants, centralizing them without regards to the old adage about "putting all your eggs in one basket". Even worse, the quality of the infrastructure at Ford is going down due to the Ford's mantra of "penny wise, pound foolish". Every project now is about "savings" by replacing the current with an inferior quality product.
But the "savings" don't stop at hardware and software. @dk+1k825ed6j is right. Ford is negotiating the outsourcing of many IT support teams, getting rid of the FTEs and contractors. There is no need to have a high IQ to know the pattern will repeat itself. Ford will pick the cheapest provider, no matter the quality.
Therefore, the cloud providers are looking better and better everyday, at least for Ford, because the local datacenters are going to be in trouble by the end of next year, and worse on the following year.
these MBA and marketing types trying to run technology are like the bro from Bill n' Ted:
https://www.youtube.com/watch?v=fQ9ATejWEgY
"It's kompyuters..."
@bw sounds like someone from Management who believes glossy Vendor presentations.
@dk good summary. BTW, The LL3 was following MA order.
@dk wow..canned both FTEs and contractors at the same time? What teams are these? And more importantly, how's the Service Provider's quality of support so far?
@dk Which 2 teams were canned?
There are two “data centers” in one building located near the WHQ VSC at Hubbard and Mercury. Each “data center” has a separate power source. The “data centers” back each other up. Each “data center” contains mainframes and servers and was sized for future expansion. If the “live” data center fails (or is taken off-line for maintenance) the “backup” data center takes over. Yes dear reader, IT management gambled on this building never suffering from any natural disaster or man-made oops. There is a third data center 20 miles away where some data and apps are backed up to on a cadence.
Despite IT managements claims to the contrary, the Dearborn “data centers” have never been successfully restored from the third data center. One problem is that many applications declare that they don’t need to be restored from the third data center to save blue dollars in their budget. Downstream applications are unaware that upstream apps are not backing up data they require. Another words typical Ford CF. Another problem is the infrastructure silos aren’t properly coordinating backups and activities. Each disaster drill IT management claims success, when it was clearly a failure. Been going on ever since the data centers went live. The ongoing joke is the disaster recovery is to have your resume stored off-site.
The existing Ford data centers have risks; external data centers like AWS have risks.
The impact will depend on what data they move offsite.
On a side note the IT LL3 in charge of infrastructure teams that support the data centers is negotiating out-sourcing the teams to an IT service provider. He has already outsourced two teams this year (and canned all the Ford employees and contractors on those teams).
The original intent when we built the new data center across from regent court was that a large percentage was going to be leased out to Google (or Amazon, I cant remember which one) as a node in their network. I wonder if any of that came to fruition, or is it another 1 of our abandoned ideas. As far as internal vs external, its just a matter of cost. It has nothing to do with uptime availability. The C suite is willing to risk outages if it saves them a nickel. Its just like quality on the manufacturing side..... We'll hope for no recalls, but if we make cr-ppy cars, we'll worry about that down the road. in the mean time, I get a bonus for saving the company money on those pesky and unnecessary quality checks.
Yes we still have data centers. Anyone that worked here would know that.
@bw You definitely worked at Ford and didn't fail up with this GOAT knowledge and CONFIDENCE. Can you please share more about this information. Us mortals have never been in a plant and would love to understand more about this stuff. So like does this mean the servers are external? What does that mean? I don't know much because i'm brown and born here so that means I don't earn it in michigan or something like that. so please share your thoughts on running servers? you sound like you've been running them consistently nonstop for 15 years
@bw No moat = no eating lunch when you have to renegotiate contracts or consider migrating out at the mercy of your vendors
you sound d-mb asf fam and probably will be promoted soon at ford for your unprovoked option delivered with such conviction
internal = cheaper
but u wont get it cuz ur a n00b l0l
@bw "Ha, maintaining an internal data centre with internal employees is just no longer practical."
No basic skills? Could have stopped there
"much cheaper."
Now this makes more sense
Ha, maintaining an internal data centre with internal employees is just no longer practical. Major cloud outages hurt, but they are less frequent than internal and recovered much quicker, and more importantly, much cheaper.
interesting quote
“When that tribal knowledge departs, you're left having to reinvent an awful lot of in-house expertise that didn't want to participate in your RTO games, or play Layoff Roulette yet again this cycle. This doesn't impact your service reliability — until one day it very much does, in spectacular fashion. I suspect that day is today.”
Big AWS OUTAGE today
One thing is for sure, AWS is significantly more capable of maintaining data centers vs Ford.
@a1 you’re right about that! One neighbor retired and the other neighbor cross the street was pushed out.
There aren't any IT people left!