15. March, 2003 - Ugh

Flaws Put Open Source on Hot Seat talks about the security hole that I spent most of the past 20 hours fixing. It was a simple patch that should have just dropped in, but when I applied it to the source for my OS, I suddenly lost the ability to compile sshd (which I use to connect to the server). Things rapidly went downhill from there, and the upshot was that a simple update that I started at 8pm yesterday took until almost 6pm today to get applied and get the server running correctly again.

Problems along the way? Flaky CVS servers meant that when I was trying to first fetch the patches, and then later fetch an entire new source-tree for my OS, I was getting partial updates that wouldn’t build. I tried to reinstall the OS from CD, but that left me vulnerable to the security hole, and I still couldn’t recompile. The I upgraded my OS, and got things partly working, but because the new version of the OS has a chrooted web-server, tons of my web-stuff stopped working. Also, the default install of the web-server didn’t include PHP so I had to recompile to get that into the mix. And in PHP, the htmlspecialchars function changed behavior, so any place I had a form that accepted text from the user changed how it behaved suddenly. It took another hour to find that and go back and add in the extra parameter I needed so it would behave the way I expected.

All of these problems were compounded by the fact that I tried to apply the update on a Friday evening, and didn’t leave enough time to get things done correctly. Oh well, at least it was a weekend when I had a number of my customer’s sites dead in the water (and then only for a while this afternoon). In all, a pretty unsatisfying experience, though. I think the answer is that I need to set up a second server again so I can apply updates to that first and verify that they actually work before updating the production server. Yeah, it’s extra hassle, but it’s got to be better than the software hell I’ve been dealing with yesterday and today.

I've got some serious server hose-age going on at the moment. The websites are all fine, and I got mail working again (kinda), but if you were experiencing problems last night, I'm aware of them. And I get to spend my weekend trying to make the server work right again. Ugh.

Copyright 2009, Dave Polaschek. Last updated on Mon, 15 Feb 2010 13:53:34.