Cisco PIX 501 dropping connections

Update, April 15

I finally talked to a senior tech yesterday. He called in response to my call to the CEO’s office, and spent about an hour on the phone with me. After seeing the problem happen, he realized that it was caused by what I’d call a bug in the software my PIX was running, and upgraded the software in it for me. All better now.

The specific bug is that when a VPN connection goes up or down, connections that aren’t routed through the VPN have to be checked against the VPN’s network policies. And that bounces those connections. Version 6.3(3) of the PIX software caches the network policies, and non-VPN connections don’t get bounced when the VPN status changes (as long as the policies of the VPN don’t require a change in the route for the connection). Yeah, that’s maybe more technical than most people want, but now someone else trying to figure out the problem with their PIX can find this via google and say Oh! I just need to upgrade the software. rather than having to waste hours of time over the course of months trying to get Tech Support to figure out the problem.

Update, April 14

In the continuing Tech Support saga, it’s no wonder I’m having so many problems with Cisco Systems Technical Assistance Center. It’s really run by Convergys, and the Indian accents of everyone who’s worked with me are explained nicely by the fact that they’ve moved most of those jobs to India. Further, the woman I was working with has quit, so I’ve got to explain the problem all over again to new techs. I guess her quitting isn’t surprising, either, since India call centers face employee exodus.

In fact, I was starting to think that Installing Linux on a Dead Badger would be easier than getting through to someone who actually cared at Cisco, but then one final call to Cisco headquarters got me through to the executive assistant to John T Chambers (the CEO & President), where I left a message explaining the situation. I got a call back a couple hours later, and got to spend fifteen minutes explaining what the product was (are you sure it’s a Cisco product?), who Convergys is, and generally cluing in the person at Cisco to why this was all a problem. Since she’d never heard of the product, it’ll take her a while to figure out who should deal with this problem, but maybe there’s some hope of progress.

I’ve got a Cisco PIX 501 that I use to VPN to a client’s network. It seemed like a Really Cool Thing out of the box. Plug it in, configure it via its built-in web-server, and you’re up and running, with no added VPN software needed on client machines. That’s a good thing, as I have multiple clients I work for, and installing more than one VPN configuration is a pain in the ass.

The problems started when I started seeing dropped connections when not connecting via the VPN. I’d be working on something on my server, with a shell open on the server (often doing something like a tail -f on some various logfile), and after a while, the connection would just mysteriously drop.

I didn’t think a whole lot about it at the time. I was busy getting ready to move into the house I’d bought, and I figured I’d diagnose it once I got moved.

Well, I moved. I managed to reconfigure the PIX once via the web-interface, and everything seemed back to normal. I was still seeing dropped connections, but I was settling in, and not all that worried about it.

Finally, in February, I got the time to take a look at the problem. The first problem was that the web-server on the PIX had stopped responding to requests. The second was that I was still dropping connections. Further complicating matters was the fact that I couldn’t find the serial cable to connect to the diagnostic port on the thing.

In any case, I send an email off to Cisco, asking for some help. I got a call initially, but without the serial cable in hand, there was little that could be done. A few weeks later, I found the cable, and got some of the information the tech there wanted, and we determined that I needed to set up a server to catch the syslog information from the PIX.

I finally got that set up. One of the things I noticed was that because my server was getting its IP address via DHCP, I was having trouble. The problem was that the PIX would give out a different IP address each time the DHCP lease would expire, so if you had a connection open, your machine’s IP address would change, and that would whack the connection. Okay, so I’ll quit using DHCP. That seemed to help, but I was soon seeing dropped connections again. Looking at the syslog information showed that the PIX was giving no useful information at all when it was dropping connections.

So I’m now using static IPs on everything on the “secure side” of my home network. But I’m still seeing dropped connections, and the PIX doesn’t seem to want to tell me anything useful about why it’s dropping those conenctions. I also have to walk down to the basement with a serial cable and laptop in hand in order to do anything with it, since the web interface still isn’t working.

I dunno. I guess it’s still a handy box to have, but I’ve gone from whole-hearted endorsement to thinking that it’s useful if you absolutely need it, but with a few caveats. The box is aimed at the consumer market, and if I were a normal consumer, I’m pretty sure I would have returned it already.

Copyright 2009, Dave Polaschek. Last updated on Mon, 15 Feb 2010 14:08:46.