Newsnews

Thu Feb 7, 2019, 02:33 PM

Wells Fargo System Outage Continues, Bank Blames Power Shutdown After Smoke Detected

Update3: Hours later, the system-wide outage at Wells Fargo continues as customers remain unable to access their online accounts.

The company said in a Thursday tweet: "We’re experiencing system issues due to a power shutdown at one of our facilities, initiated after smoke was detected following routine maintenance. We’re working to restore services as soon as possible."

Apparently Wells Fargo doesn't maintain redundant servers at a separate location in case of such incidents...

Update2: After their first attempt to provide people with account access crashed and burned, Wells Fargo is now simply apologizing and letting people know they can go to a branch or an ATM for their banking needs.

Update: Wells Fargo's homepage now lands on a message which reads "We're experiencing some technical difficulties"

Wells Fargo's online banking and mobile app went down Thursday morning, leaving customers unable to access their accounts and conduct banking needs. It is the second time their system has gone down in less than a week.

https://www.zerohedge.com/news/2019-02-07/wells-fargo-online-banking-goes-down-customers-unable-access-accounts

https://downdetector.com/status/wells-fargo

Map looks ugly. Comments indicate that credit card charges are being refused, so this is not just an online front end system issue. It probably took out systems of record in the backend. Typically you only have one mainframe system running in a single major data center for such purposes. Failover to a second system of record in the alternate data center is not automatic. After 9/11 the Fed required them to be over 200 miles apart, which is beyond the reach of a Parallel Sysplex in multiple data centers.

5 replies, 146 views

Reply to this thread

Back to top Alert abuse

Always highlight: 10 newest replies | Replies posted after I mark a forum
Replies to this discussion thread
Arrow 5 replies Author Time Post
Reply Wells Fargo System Outage Continues, Bank Blames Power Shutdown After Smoke Detected (Original post)
Troll2 Feb 2019 OP
quad489 Feb 2019 #1
foia Feb 2019 #2
Troll2 Feb 2019 #3
Troll2 Feb 2019 #4
Currentsitguy Feb 2019 #5

Response to Troll2 (Original post)

Thu Feb 7, 2019, 02:42 PM

1. Makes one wonder just where WF spends all the money customers give them.........

"Apparently Wells Fargo doesn't maintain redundant servers at a separate location in case of such incidents... "

Reply to this post

Back to top Alert abuse Link here Permalink


Response to quad489 (Reply #1)

Thu Feb 7, 2019, 02:50 PM

2. Having a redundant system at another location should've been done DECADES ago

Especially at a financial services organization where IT is the core technology for their operations.

Reply to this post

Back to top Alert abuse Link here Permalink


Response to quad489 (Reply #1)

Thu Feb 7, 2019, 02:56 PM

3. They probably do, but whole data center disaster recovery plans are multi-hour restoration plans

1, the data bases in the DR data center are not all synchronized and require careful updating from journals.

2. software and hardware that is designed to be used when things go wrong isn't really fully exercised and tested. Therefore, there are more bugs in maintenance, disaster recovery, and business continuity arrangements than in the normally operating systems. This can complicate and delay DR execution.

A classic example is where a company replaced diesels in the basement with turbines on the roof. They left the fuel tanks in the basement, and installed pumps to lift fuel to a holding tank with the new turbogenerators. This all worked fine when tested periodically. Then there was an actual lengthy power outage. The turbines started, emptied the holding tanks, and stopped. The batteries drained and the building shut down. All because the new fuel pumps were powered from commercial power, rather than from the battery plant.

There was also that case where signalling between toll telephone offices ceased because of a single bit error in traffic overload management code...

Reply to this post

Back to top Alert abuse Link here Permalink


Response to Troll2 (Original post)

Thu Feb 7, 2019, 03:06 PM

4. Better call Saul

Reply to this post

Back to top Alert abuse Link here Permalink


Response to Troll2 (Original post)

Thu Feb 7, 2019, 05:32 PM

5. Reminds me of the series Mr. Robot

They took down a huge banking system in that way and disables all of the smoke detection systems at the redundant facilities.

Reply to this post

Back to top Alert abuse Link here Permalink

Newsnews