Flaws Put Open Source on Hot Seat talks about the security hole that I spent most of the past 20 hours fixing. It was a simple patch that should have just dropped in, but when I applied it to the source for my OS, I suddenly lost the ability to compile sshd (which I use to connect to the server). Things rapidly went downhill from there, and the upshot was that a simple update that I started at 8pm yesterday took until almost 6pm today to get applied and get the server running correctly again.
Problems along the way? Flaky CVS servers meant that when I was trying to first fetch the patches, and then later fetch an entire new source-tree for my OS, I was getting partial updates that wouldn’t build. I tried to reinstall the OS from CD, but that left me vulnerable to the security hole, and I still couldn’t recompile. The I upgraded my OS, and got things partly working, but because the new version of the OS has a chrooted web-server, tons of my web-stuff stopped working. Also, the default install of the web-server didn’t include PHP so I had to recompile to get that into the mix. And in PHP, the htmlspecialchars function changed behavior, so any place I had a form that accepted text from the user changed how it behaved suddenly. It took another hour to find that and go back and add in the extra parameter I needed so it would behave the way I expected.
All of these problems were compounded by the fact that I tried to apply the update on a Friday evening, and didn’t leave enough time to get things done correctly. Oh well, at least it was a weekend when I had a number of my customer’s sites dead in the water (and then only for a while this afternoon). In all, a pretty unsatisfying experience, though. I think the answer is that I need to set up a second server again so I can apply updates to that first and verify that they actually work before updating the production server. Yeah, it’s extra hassle, but it’s got to be better than the software hell I’ve been dealing with yesterday and today.
I've got some serious server hose-age going on at the moment. The websites are all fine, and I got mail working again (kinda), but if you were experiencing problems last night, I'm aware of them. And I get to spend my weekend trying to make the server work right again. Ugh.
- You know that FBI plane I mentioned yesterday that was mistaken for a terrorist? Well, the FBI has a fleet of aircraft tracking terrorist suspects totalling 80 planes, not just the one in Bloomington, IN that got caught. [strib]
- Four guys are Bowling to Vegas. A road trip that involves bowling and drinking from Chicago to Vegas. Sounds like a good plan to me. Definitely better than spending my weekend working on the server.
- USDA to keep some food recall data from US consumers [warning: popups] because we really don’t need to know which grocery stores, restaurants and butchers are selling tainted meat. Ulch! How about just letting me sue when I get sick and putting enough information on the package so I can tell who produced it? No way. That’d be way too simple. [metafilter]
- Natalie Merchant, No Strings Attached. She’s said to heck with her label, and is releasing her next album herself. She only needs to sell 50,000 copies to break even, so she’s pretty sure she doesn’t need a label.
Why subject myself and the work that I do to that kind of environment when it really doesn’t matter any more?
[boing boing]