North American Network Operators Group

Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical

Re: The power of water

  • From: John Kristoff
  • Date: Wed Oct 23 19:51:46 2002

At 2:03 PM -0400 10/19/02, Sean Donelan wrote:

> Stuff happens to everyone, its how you respond.  Would your company
> have been able to recover as quickly?

Over one weekend I was part of a team of folks involved in moving a
voice/data center for a fairly sizeable regional office across the city
in Atlanta.  We had pretty much everything moved, installed and working
by Saturday night.

Early Sunday morning as we were tidying up, someone called over the
portable radios that it was 'raining in the data center!'  If I remember
correctly, it was a return pipe for the cooling system had come apart at
a poorly soldered connection in the ceiling above the UPS.  Soon enough
a good portion of the floor tiles came down and the underneath the
raised floor a nice pond was forming.  Since it was a Sunday morning, it
took some time before some building engineers could be called in to turn
off building water.  By that time, the 1' raised floor was pretty well
full. We were on an upper floor, maybe 7 or so.  I don't remember actual
amount of water, but there was water draining into the third level deep
parking garage below.

Some servers had been permanantly damaged, but most equipment including
some servers (remember NetFRAMEs?), hubs, routers, the cabling system
and a PBX survived to provide service until the insurance company
provided funds for everything to be replaced.

Having good vendor relationships helped significantly in my experience.
Many were overnighting equipment without requiring a lot of red tape. 
Having access to a bunch of hair dryers was also useful.  :-)