Early this afternoon I pushed out a code change to Warhorn that broke the site for several hours.
It’s common for me to push changes at any time of day or night, because it’s generally a very simple and risk-free procedure. However, sometimes a code change has to be accompanied by a change to the database that sits beneath the website. Today was one of those times.
Website and database changes have to be coordinated carefully. If one is performed but not the other, you get error pages with weird monkeys on them instead of, say, event schedule pages. Well, I didn’t coordinate carefully enough. In fact, I didn’t coordinate at all. While eating lunch, I pushed the website change. But then I got sucked back into my day job and completely forgot all about the database change 😦
By the time I opened my personal laptop around 11:30pm to read the New York Times evening briefing, I saw 26 open tickets in the Warhorn support queue as well as public and private messages on Facebook and Twitter. It took all of two minutes to execute the database change, and the website was back in business.
I’ve been building websites for over 25 years. I’m a Vice President of Technology at a major financial services company that you’ve definitely heard of. And today I made a rookie mistake.
I apologize for all the frustration I caused today. I’ll do my best to keep this from happening again. I can’t promise it won’t, because I’m only human, but I’ll do my best.