Newsnews

Sat Feb 9, 2019, 01:06 PM

Wells Fargo, whats really going on?

A little smoke, a fire, shuts down banking across the country? No redundancies?

Millions effected. Direct deposits not working, max $100 withdrawals if you can access your account.

Something about this story isn’t making sense.

Any thoughts? Anyone here a Wells customer that can add something.

4 replies, 243 views

Reply to this thread

Back to top Alert abuse

Always highlight: 10 newest replies | Replies posted after I mark a forum
Replies to this discussion thread
Arrow 4 replies Author Time Post
Reply Wells Fargo, whats really going on? (Original post)
Badsamm Feb 2019 OP
Currentsitguy Feb 2019 #1
Badsamm Feb 2019 #2
Duke Lacrosse Feb 2019 #3
Troll2 Feb 2019 #4

Response to Badsamm (Original post)

Sat Feb 9, 2019, 01:23 PM

1. If what they say is true

The SEC is going to want a word because this is gross negligence.

Reply to this post

Back to top Alert abuse Link here Permalink


Response to Currentsitguy (Reply #1)

Sat Feb 9, 2019, 01:49 PM

2. SEC works for the banks. Any public statement from the SEC will be cleared

By the FED, if not written on FED letter head

Reply to this post

Back to top Alert abuse Link here Permalink


Response to Currentsitguy (Reply #1)

Sat Feb 9, 2019, 02:24 PM

3. It's the Office of the Comptroller of the Currency they should be worried about, not the SEC.

I've been through numerous OCC audits including IT audits (I was a bank IT worker in a previous life), but I've never seen a bank be compelled to demonstrate to the OCC its ability to recover from a computing infrastructure disaster.

Maybe that will change after this incident.

Reply to this post

Back to top Alert abuse Link here Permalink


Response to Badsamm (Original post)

Sat Feb 9, 2019, 02:47 PM

4. Lots of stuff on this reddit

https://www.reddit.com/r/sysadmin/comments/ao4g2y/wells_fargo_is_down_declining_transactions_and_no/

The short story is that the Shoreview, MN data center suffered a total outage after utility work triggered the fire suppression. At best there would be a several hour interval to shift critical apps to another data center, and maybe a day to shift lower priority work.

According to some, the fire suppression systems are very loud and noisy when they go off. With the newest high-density disk drives, the sonic vibrations cause the disks to fail if they are spinning when it goes off. Likely the fire suppression went off before power to the server bays was cut, either because utility power hadn't failed or the servers were still powered from backup. At any rate, this is about the least graceful shutdown of a data center.

Also disaster recover and business continuity arrangements are not thoroughly tested. When they are tested applications belonging to the different verticals in the bank are likely tested separately. According to the reddit, not all DR and BC procedures went well.

This is all made even more complicated by agitated customers placing extreme loads on the system whenever parts come back up. They will likely be pretty well recovered Monday morning. The weekend also gives software developers time to repair databases. reconcile unprocessed transactions, etc. after the hardware and operating systems are restored. But they will likely still be a little shaky running out of a data center other than Shoreview.

Per http://www.fox9.com/news/unclear-if-wells-fargo-system-outage-is-connected-to-minnesota-server-facility-incident

The Lake Johanna Fire Department confirmed an activation of the fire suppression system at the Wells Fargo facility in Shoreview, Minnesota due to construction dust, not a fire.

“There was NOT a fire at the Wells Fargo facility in Shoreview, MN,” the department posted to Facebook. “They did have an activation of a fire suppression system that was triggered due to dust from construction.”

Reply to this post

Back to top Alert abuse Link here Permalink

Newsnews