This picture has never been more applicable to me than it was today.  Here’s the story:

One of my fellow DBAs was trying to configure DTC in an existing cluster and he was having some problems.  In an effort to see where the problem lay, I added a DTC resource to the cluster.  I then went to delete it, so he could do it.  I highlighted the DTC resource and clicked the Delete button.  Only I clicked the WRONG Delete button.  And then I didn’t carefully read the “are you sure?” prompt that popped up.  Yes, I’m sure, just do it already.  Well it did.  It deleted everything in the resource group.  Well, everything except the DTC resource, ironically enough.

Oh shiat is right.  My next call was to our Windows admin.  “Hi, um, hypothetically speaking, if I delete everything in a cluster group, can we get that back?”  He didn’t seem nearly as “oh shiat” about it as I was.  Fortunately, the disk was still online, so the first thing we did was copy the backup files to another server.  Just in case.  Once that was done, he right-clicked on the (now nearly-empty) cluster group and selected Add Storage.  He selected the existing mount points and they all moved back into the cluster group.  And to my delight, the SQL Server and SQL Agent resources also re-appeared.  Everything came back online without a hitch.  Whew!

Moral of the story:  Never do anything 3 days before vacation.  Just surf, play Angry Birds, hop on Twitter, whatever.  No work.  Nothing good can come of it.

Leave a comment

Your email address will not be published.