Tuesday, September 27, 2005

Disaster Recovery

One common comment I have heard since Katrina has been "well, they didn't teach you about this at seminary, did they?".
No, they didn't.
But, as someone who spent 20+ years in the banking world, in Data Processing, I should have been better prepared.
As a mainframe systems programmer, we were required not just to have published disaster plans, but to test them. At least twice a year we would take backup tapes of all our systems and fly to an off site center, load the entire systems (this is WAY WAY bigger than PC network stuff, folks), get it up and running, run a day's work for the bank, establish communications links to branches in Mississippi, etc. It was HARD stuff but very important. We utilized two offsite vaults and backups were sent to each every day - in case the "disaster" affected one vault as well as our main location.
To make it more realistic, at times the auditors would come in a declare a "surprise" disaster test. Part of the surprise was to mark certain staff members as MIA - so they could not participate in the test, nor were they allowed to communicate with those who were. We had 4 folks on my staff, and two would be exempt, really testing how well we documented our plan and cross trained our staff.
So here comes Katrina, and you would think I would have had a better plan. Now I must give credit to our altar guild, as they had a written plan that they did execute - three seperate large buckets were loaded with altar sets - chalices, patens, linens, bread, wine, BCP, Bible - all that we needed to "do church" from one container. They were then sent to three different evacuees to take off site. All survived, so we have all our chalices, etc.
As to the church records, we loaded them to a parishioner's house who had built to withstand 150+MPH winds, and had a "safe room" made of extra reinforced concrete and steel. In it we took our Parish Register, our Services Record, our charter (when we became a parish), checks, financial records, backups of our computers. It was all wiped out by storm surge. All....gone.
Thankfully my secretary, at my suggestion, took a copy of our church database records with her, so we do have a backup to restore from once we get a computer.
I left WAY too much stuff in my office, stuff that is not replaceble. I did get all my personal stoles, but left albs (mine and my wife's), ordination certificates, some pictures, all my books. Idiot. I should know better. And I should have simply taken the desktop PCs (sans monitors and printers) with me, it would have saved us tons of time (and money).

Bottom line - you ALL need a plan and you need to TEST it too. Write it up, declare some people MIA, and try it out. You need redundant backups and you need safe places for stuff. Lots of places. Lots of stuff.
If Seabury (or any other seminary) wants me to teach a class - PLEASE let me know. I can, by example, tell how not to do it - and maybe a little of how to.

1 comment:

St. Casserole said...

You are correct. We need to have workable disaster plans.

I'm so sorry about your losses but who could anticipate that your church records would be damaged.

My records show that the first mention of Mississippi being in Katrina's path appeared at 8:44am. on 8-26. The storm began near the Bahamas, not off the Coast of Africa, so we had less time to prepare.

I'm back on the Coast. Can you drink coffee on Wednesday?