A massive data center outage knocked out my website for a day or so. Read on for more of the gory details.
The only good news about the downtime is that the problem was not on my server, so the risk of data loss was very low. The fact that many other websites were affected was not much of a consolation.
I plan to have a hot backup site ready to take over in case something like this happens again. I spent months setting up the new website and had 100% uptime during the entire testing period. Being down for a day or so just a week after launch was definitely unexpected. I’ll be much better prepared next time.
It’d be easy to blame the fine folks that run the data center for the long downtime, but I don’t walk in their shoes. I’m sure they did their best to restore service to all the websites affected as soon as possible. We were in communication throughout the incident and it was clear that they didn’t need additional pressure from me. They were on it.
Nevertheless, I apologize for the downtime. I take my technical work very seriously. My philosophy is always “the buck stops here”. A perfect example is RCadvisor’s model airplane design calculator. Every time I update the calculator I spend lots and lots of time making sure it works. More often than not, I find a problem that is caused by a bug in software that I did not write but instead depend on. Then I typically end up banging my head against a wall for a couple of hours just trying to figure out a way to include the functionality that I want while bypassing the bug that I have no control over.
I have a zero bug tolerance policy. There are no bugs that I know about in the calculator. Zero. Zip. Nada. When I learn of a bug (either through my testing or a website member email) I either fix it right away, find a workaround so that I can still deliver the functionality that I want, or (as a very last resort) disable the feature until I can include it and make it reliable. I make no exceptions to this philosophy.
Back to the Regularly Scheduled Program
In case you haven’t noticed, I’ve been posting on average once a day on a variety of topics of interest to model airplane enthusiasts. The litmus test for whether something is appropriate is simple. I ask myself if, as a model airplane fan, I’d enjoy reading the article. My goal is having lots of high quality relevant contents.
Let me know what you think of my posts. I’ve gotten a few responses to my poll. If you haven’t filled it out yet (it only takes about a minute), then please do so now.