Site's Down! A Leavins Software Server Outage Tale

What Happened?

Around 7:00 (all times Central) this morning, I took the server running Leavins-Software.com offline to perform a server update. I planned for it to be a one-step process: update the server's operating system from Debian 6 to Debian 7. Just a few commands and everything should work right, correct?

Well, I had to replace the server and redo the SSL certificate that makes secure access possible.

The operating system upgrade required upgrading the server software I use. And during this setup problems, I hit some serious problems.

First, some incompatibilities between my home computer and the server prevented a necessary PostgreSQL upgrade. It took me a while, but I resolved that issue. Only to run into a memory issue. The Leavins-Software.com server had very modest specifications, and it ran out of memory during a necessary cluster update. The system was left in an 'in-between' state where I could no longer run the outdated database software and I could not run the newest copy.

I decided to switch hosts and create a new server. If you're reading this blog, it's working! Server setup was finished around 12:35 (noon), meaning that Leavins-Software.com was offline for over five hours. I know that mine is a low-traffic site (especially on Sundays), but I still want people to know about the outage.

No data was lost. I do nightly database backups and regularly save this backup on-site in multiple locations.