December 21, 2009 – 1:48 pm
Well, my turn to make a stupid mistake… Well actually, I took a risk, and it backfired.
I am hosting my virtual servers on a QNAP TS419P iSCSI NAS, which has disk redundancy, etc… I needed to reconfigure the array, so I moved all my VMs locally to one of the ESX servers, and I made a backup on the 2nd ESX box (to play it safe). I proceeded to reconfigure the array.
A few days passed and I sort of left things as they were (everything was running so…) Since I am studying for a bunch of Microsoft exams, I figured I should build a HyperV server. Since I have only two physical machines, I decided to use one of the ESX servers and reconfigured it to HyperV. While I was having fun playing with HyperV, setting up new servers etc, I overlooked all my servers that were hosted on my now single ESX box with no backups. Guess what….
Hard disk failure on Sunday.
I lost both my domain controllers, the mail server, this webserver hosting 2 sites and some tests boxes. I have backup jobs running to export the mail server’s configuration daily so I was able to recover that easily, and for some strange reason I backed up the website folders last friday. I overlooked the MySQL database however and had no more content for the blog.
I was able (by a miracle) to recover the web server from the failed disk (it booted just long enough for me to copy the server from the failed disk’s datastore to the new instance of ESX I rebuilt, so I managed to save the database!
Lesson learned: Never leave anything important running with no redundancy
Posted in Technical | No Comments »