The Disaster Disaster Recovery Plan

Mushroom CloudOnce upon a time, I worked for a company that put out a mandate of updating our disaster recovery plans.  Seeing as how most of the company didn’t have them, it really meant creating disaster recovery plans.  Due to a confluence of events, I happened to be the senior guy in our department who wasn’t a manager, and so it fell to me to craft our plans.

As with any good company, we did have some preparations in place even if we didn’t have a formal plan.  We had an off site backup data center which we could switch over to should a service at the primary site go offline.  And in that site we had an approximate copy of our primary site.  Approximate in that the intention of the backup site was to limp the company along until the primary site could be restored, not in that the backup could become the new primary.  So where the primary site had 3 servers doing a task, the backup site had 1, enough to do the job but not enough to do it without frustration.

So there began a series of meetings between departments as we began updating and creating our recovery plans, budgets were outlined and all the ducks were put in a row.  In one of these meetings, I noticed a flaw in one of the other department’s plans.  I brought it up, but since I’d only been there a year and was dealing with people who’d been with the company for 5, 10, even 25 years, I was ignored.  I was told, “You don’t know what you’re talking about.”  To that end, I decided to be sure I was right.  I began researching the other plans and looking for flaws.

My own department’s plan was simple: money.  We needed to spend the money to duplicate our server functions.  I called the companies we licensed software from and got the okay and licenses to maintain a fail-over site without spending any more money on licenses.  I wrote up hardware orders so that where we had 3 servers doing a task at the primary location we would have 3 servers waiting to take over if the primary failed.  My plan done, I had plenty of time to look at other people’s work.  So I did, and I made a nuisance of myself, sending emails and showing up for meetings I wasn’t invited to in an effort to actually make our company’s disaster recovery capable of recovery.  Eventually, I got told to stop.  The message was clear.  I was to focus on my own plan and leave everyone else alone.

I dove back into my disaster recovery plan.  You see, because of the flaws in the other plans, my original plan wouldn’t work.  It had to be redone.  I went to my boss and made one request.  On presentation day, I wanted to go last.

The day came to show off our new plans.  I sat in the back and waited through each department.  One by one they went to the podium and showed charts and laid out plans that illustrated they were well on their way to being ready, each department patting themselves on the back.  Finally, it was my turn.  I got up and started handing out my plan.  It was very short.  A cover sheet and then just two pieces of paper beneath that.  As I made my way through the room people began muttering to each other.  I got to the podium and said, “If you will turn to page one of the packet I’ve handed out you will clearly see the full extent of my disaster recovery plan.”

It was a copy of my resume.

“If the primary data center were to go offline, I would, in reaction to this disaster, begin sending out copies of my resume in an effort to find another job, because I certainly wouldn’t want to work here anymore.”  I could see my boss turning red with rage.  I could also see the managers for other departments shooting dirty looks at me.  Then I opened up my PowerPoint presentation.  I quickly showed the single page of my real disaster recovery plan: buy servers, install software, use extra license keys I’d already obtained.  Then I showed how my plan would still fail due to a flaw in Department X’s plan.  Then I showed that without fixing another flaw in Department Y’s plan, Departments A, B and C would fail.  And then I showed how Department M had overlooked a critical piece of hardware for which there was no backup and rendered everyone moot because the only working mainframe terminal in the backup site would be the one hooked directly to the mainframe.  Their plan actually had them unhooking a piece of equipment, loading it on a truck, and driving it nearly two thousand miles to the backup site, rather than actually purchasing a duplicate – probably because it was extremely expensive.  ”So, as you can clearly see, my only reasonable course of action – since I was instructed not to involve myself in the affairs of other departments – is to find another job.”

The fallout from that meeting was huge.  First, I got yelled at.  Then, I got apologized to as they discovered I was right.  Eventually, new plans were drawn up and big money was spent, but our recovery plan was actually capable of recovering from disaster.  To date, that company has not had a disaster from which recovery was needed, but that’s not the point.  The point is that each and every department concerned themselves only with their own particular areas and no one had been assigned the task of looking at the places where they relied on another department.  Each one was happy to be able to say “We have a back up for our functions” and didn’t bother to examine if slot where their tab was supposed to insert was being covered, they just assumed it was someone else’s responsibility and it would be handled.

Since then, I’ve always tried to make sure I keep an eye on the big picture when I do things.  And I try to be open to suggestions and/or criticism from others on the off-chance that I’ve missed something big because I’m too close to it.  Outside that, there is no point to this story other than I just like to share it.

Amazon vs Lord of the Rings Online

At the last possible moment… okay, not the last moment, but close… Saturday, I decided the wife and I would play Lord of the Rings Online: Shadows of Angmar. So, I threw a copy into the Amazon shopping cart, changed the quantity to two and placed my order. The reason I started by saying it was a close call was that the game released on Tuesday, and you had to order it before then as a pre-order to get the super cool Founder’s bonus stuff, most importantly the $9.99 a month rate. Since the normal rate would be $14.99, $5 a month times 2 is $120 a year savings.

If we end up playing for two years, I might kick myself for not taking the Lifetime subscription, but then again, if I paid $200 and then canceled after six month I’d kick myself. Damned if you do…

Back to my point though… we ordered our two copies of the game and then I went to the digital download section to claim our pre-order key… wait. Key? Singular? Shouldn’t I have two keys?

Why yes, yes I should.

So I call Amazon Customer Service… or rather, I go to the web page, enter my phone number and click the button to have their help desk (helpfully located in India) call me. The woman is nice enough, at least the broken formal English she is reading from her CS manual is nice enough. After many unsuccessful attempts to explain how the pre-order, account registration and all that is supposed to work, and trying to point out that I ordered two copies of the game but only got one key… to give a quick example, it went sorta like this:

Me: “I ordered two copies of the game, only got one pre-order key.”
Her: “Order shows one item.”
Me: “With a quantity of two.”
Her: “Not two, just one item on order.”
Me: “There is one item, with a quantity of two.”
Her: “Sir, your order has only one item. Digital downloads are given one per item.”
Me: “The item cost $50, my invoice is for $100 because I bought two.”
Her: “But there is only one line item.”
Me: “With a quantity of two.”

Finally, she grasps the concept… one item, quantity of two… and determines that she is not capable of resolving my issue. She says that I should have ordered the items at separate lines, then forwards my problem to another department, says they will email me the resolution, and hangs up.

Now, there are many things I am not, but one thing I am is a Web Programmer. You would think, if Amazon has an issue with providing digital content on multiple quantity single line items someone over there might be able to trap a flag and issue a warning to the user, or even not allow multiple quantities for items with digital content. A nice little pop up that says, “This item includes a digital download and product key, please add multiples to the cart separately.”

In any event, we are now waiting to hear from Amazon. They owe us a pre-order key, or they owe us $5 a month. Let’s see how long this takes to resolve…

Update: As bad as the first call was, my follow up 48 hours later was good. The woman was pleasant, contacted the department needed, got us all on the phone, got the issue reviewed and resolved, and she apologized for it taking a second call to get the work done. Apparently the first woman hadn’t actually forwarded my issue to the other department. All is good now.

Corporate Security

Its one of those things that on one hand I can understand, but on the other hand its really overstepping: Browsing the Internet from work and security. Now, I completely agree… block porn sites and other stuff that people should not be doing at work. But I’m a programmer, and there are a number of times that I come across the need to do something that I don’t know how to do. I check with my co-workers, and none of them know either… so I head to the internet. However, where I’m working now, they have a content review department. They log the urls of sites that everyone visits and they audit it for content. The more often you visit, the sooner the audit. Anything they deem “not good” they block. Many of the best programming sites on the net have ads to support paying for the site. The companies that pay the most for ads are those “scam-like” sites where their promise you “a FREE iPod*” if you punch the monkey. The review department bans any site with those ads. So, the more useful I find a website, the sooner it gets banned.

What?

I mean, I could understand if they reviewed the content, then banned it because its not work appropriate. But banning a site because of the people who buy ads on the site? Huh?

Whatever happened to monitoring people’s usage, then warning or firing them when they do something wrong? If someone wants to visit porn at work, you should let them, once, then fire them. But if they use the Internet in order to do their job better, why the hell are you trying to stop them? Why?