Anyone know what happened????
31 replies (most recent on top)
I was there working with Lisa in Server Ops for 5 years. I looked up to her and Terry. I've never known better management and/or better people in my entire professional IT career. I'm shocked to learn that Lisa was let go. Big mistake for RJ.
From The Top Floor,
Yes, the system went down due to the executions. Unfortunately, the temporary power outage caused the problem. We apologize for this as we also know it temporarily dimmed the lights. We are looking to save money and the lay offs have not been enough. We are saving money on ammunition and using electricity. We have found this to be more profitable for our shareholders and top level executive management. It doesn't matter to us how we get rid of people. But, it does matter if our year end bonus is less.
Sorry for the outage.
Well, she'll be better off now anyway.
Here is her: https://www.youtube.com/watch?v=OQC5ot-UEN4
I attended many meetings and worked with her from outside the department. The Lady should have been let go with the April 2013 layoffs. She was NOT an RJ asset. I think Friday clearly showed that. It just had to come to a head that Bruce could not control.
The Lady chose to deviate from the process in the shadows, during the day at that. She chose to disregard the rules, crashed production and was fired for it. No permission to do it, no change order, just warnings from other groups. Rumor is she played dumb about the work at first too. Nice to throw the tech under the bus. This is one of the few correct calls from management.
We all know that Ferguson and Zolper monitor anyone browsing to this site internally, so make sure you do it from your phones or from home. Even if we have the freedom to say what we want, they'll find some reason to make life difficult.
pretty f'd up if you ask me. the lady who ran a solid department for 18 years. had her shit together. brought our network from a few hundred servers to well over 2000. led the team that migrated hundreds and hundreds of servers to our virtual platform. i can't remember once that her team caused an outage, so it happens once. we live, we learn. i don't feel it was reason enough to fire her. she got the environment to where it is today.
you want to talk messed up. what about what the leadership has done to IT in general. no longer a top place to work. recruiters in the area don't want to put people at rj any longer. people cry to bella because their phone doesn't work like they want and she cracks the whip. managers and employees are scared. who wants to make a change for fear of losing their job or being threatened. why a moratorium? oh, bella and vin are afraid they'll lose their job next so no changes. it's called growth. outages happen. yeah, this was unfortunate, but get over it.
this isn't what TJ built. he handed over the keys, but it was to a bunch of money hungry individuals, not people that care about culture and people. when was the last time you've seen bella, or vin, or andy, or sateesh, walk around and talk to their people. it doesn't happen. they don't give a shit about anything other than a big fat paycheck so they can all retire early. i hate this place now. not happy coming to work. not happy at the direction, and not happy about the leadership.
people, you have the freedom of speech on here, so say what you want and what you feel about this place.
virtual clusters were f-ed up, affected dns connectivity too, that's why LH was axed, the guy who was configuring it was axed too
They let go many of the people that had in depth knowledge and experience with the applications, network, systems etc, and now they are paying the price for all the "history" goes walking out the door.
The trust and loyalty from the IT group to work late nights, weekend, and help out is not there any more, and you can not blame people for that. Training? Market place compensation? ha ha
They caused the network outage? What have they done?
Lisa Huston and and virtual engineering tech were let go.
well, trading is a huge deal, and when its down, the company doesn't make any profits. rj has said for several years they were going to make an example out of someone some day when something major occurs, guess it was time. there have been other outages recently that didnt impact trading and not much was heard. who wants to make a change or upgrade now? probably nobody. it'll be like walking on eggshells around here for fear of making a mistake and losing our jobs. i've always loved my job here, but it's time to start looking. since the leadership change a few years ago, this place isn't the same. lots of good people have left on their own, many were "layed off", and others fired. many good people that got RJ to where it is today, but the second someone higher up doesn't like what they see, out the door. now we see why ubs had it's share of problems and isn't the successful company it once was. people see the company growing, but much of that growth was the purchase of MK a while back. and once we all worked for 24 months to make a successful merge, the cuts started coming. when was the last time RJ IT was in the top 100 places to work? oh yeah, as soon as the new reign of terror came on board. as employees, we don't even get to see the results, but the writing is on the wall. this certainly isn't the company that Tom James built. i know my resume will be online this week.
If it happens again, more will be walked out. This one went to the very top within an hour.
I Yaaaaaaaaaaaawwwwwwwwwwwwnnnnnnnnn, but have no idea the seriousness of the situation. A 3 hour no services for RJ outage with people being walked out is a Yaaaaaaaaaaaawwwwwwwwwwwwnnnnnnnnn situation, so I must be a complete moron or the coffee re-stocker.
Definitely not a shining moment.
Yaaaaaaaaaaaawwwwwwwwwwwwnnnnnnnnn.
Yaaaaaaaaaaaawwwwwwwwwwwwnnnnnnnnn.
I don't know the exact cause, but I do know that Lisa Huston is no longer employed at Raymond James as well as one of her engineers.
how do you explain this to the shareholders. sorry, we made a mistake and didn't react appropriately to keep our systems going. our recovery plan doesn't work like it should. we spend millions on testing and testing year over year and it's never been a complete success. if it worked like its supposed to, could have at least failed some things over to get them working. people like bella and vin should be let go. how much did this outage cost the company??
whats the point in failing over when bcp has never fully worked anyway. at least tim always had a clue what was going on and never had a complete network outage. if we failed over, and things didn't come up, heads would be rolling from the top.
If it wasn't DNS, what was it?
Was not DNS
I guess they learned change is NOT always a good thing. Ok when you dump the ones you don't want. Not so great when key people dump you.
Yes. People were literally just standing around for hours. No email, no SharePoint, no client center, no advisor access,.... How's that new crew now Paul? Can't say Tim had to meet with you on a subject like this before.
Did it affect traders and FAs directly?
2nd HAHAHAHAHAHAHAHAHA
we want a dollar router dos dis we CyberBerkut
HAHAHAHAHAHAHAHAHAHHAHAHAHAHAHAHAHAHAHAHAHA. The Chickens are finally coming home to roost. Great leadership Vin and Bella. Hopefully, one of the 170 people you laid off or someone in your mini layoffs that are running in the shadows lately were not the one person you needed there to resolve it quickly. I imagine your survey's from the FAs might not be that great now Vin. A widely felt 2 1/2 - 3 hours of nothing will not go good with the FAs. Heck, it would be an problem in any company.
More like a massive systems outage for 3 hours. No nothing except in Denver! It will be interesting. I don't recall an outage of this magnitude with Tim in charge. Good job Vin and Bella!!!! How is that BCP working with your leadership?
DNS issues