Father of Ooops!

7 AM June 6, 2004

Simon points to computer-driven woe at the Royal Bank of Canada. They have 155 people working on the problem.

In the olden days, I used to work at a bank. They ran their entire transaction processing load on a brand new IBM 4341. One pension day, something really bad happened, and in the space of two minutes, the computer room filled with people.

They arranged themselves thusly:

  • In the first row, typing at the 3270 console, were the two operators that were rostered to run that day’s jobs.
  • Arranged in a semi-circle behind them were all the other operators, the operations manager, and a few systems programmers. They were being very helpful, pointing at things on the screen and making suggestions.
  • Behind them was a third semi-circle, consisting of the senior systems programmer, and all the department’s managers, including the executive in charge of IT. They also seemed to be making suggestions, but were enjoying it far less than either of the two inner shells.

I know all this because I was visiting some of the other operations staff at the time, and was left to watch the scene unfold through the plate glass windows. Pretty soon, the area I was in filled up too.

I looked around, decided that no good could come from watching a disaster unfold, and made myself scarce.

By alang | # | Comments (1)
(Posted to Tall Tales and Software Development)

Comments

At 14:57, 06 Jun 2004 Rob Meyer wrote:

I've been going through this a lot recently (thankfully nothing serious enough to make the news), but on confrence calls because the team is distributed, which is really a nightmare. You've got 30 people on the call, and some manager, somewhere, keeps dropping in every 30-45 minutes just when you are starting to hit a troubleshooting stride and asks for a status update. Each one thinks that asking for status won't hurt, but by the time the network, application, firewall, load balancer, unix, windows, business owner, and hardware teams are aon the phone, that makes for a lot of managers interested in the status...

(#)

Add Comment




(Not displayed)






(Leave blank line between paragraphs. URLs converted to links. HTML stripped. Indented source code will be formatted with <pre> tags.)




© 2003-2006 Alan Green