About Dave’s Picks (the site)

I use a bunch of different pieces of technology to put Dave’s Picks together every week. I also use it as a test-bed for new things I’m learning. Sometimes this will make things appear a little rough around the edges as I’m figuring something new out, but generally I try to keep everything happy and HTML 4.0 compliant.

This is an attempt to describe all of those bits of technology, plus give some reasons for why I chose them. But first I need to give a little history, since that will provide some background for the decisions I made. I’ll continue adding to this file as I change things, so it’ll be an ongoing history.

Finally, rather than being the usual most-recent-first order, this is in straight chronological order. It makes more sense for someone who’s reading it for the first time that way, and I think that’s important.

The early history

I started Dave’s Picks on Friday 21. November, 1997. Well, I actually started it before then, but that’s the oldest entry that survives. I was at WAM!NET at the time, and found that a lot of folks in the office were missing out on news that was relevant to our business. I was sending around links to news stories via email. Rather than clogging email boxes, I decided to start publishing those links to the web every Friday.

I’d been using Frontier to build web stuff for a while, and the ability to have a scripts menu appear in my browser that could fetch the URL and title of a web-page was pretty cool. Plus, Frontier has an outliner, which I used heavily to get the structure I wanted to sort the links out every week.

When I was laid-off from WAM!NET in January of 1999 and decided to start my own business, Dave’s Picks provided a diversion while I was getting the business up and running (you wouldn’t believe how much time was spent waiting on other people because I didn’t know how long things took). It also gave me a chance to keep current with the latest changes in HTML.

Moving to davespicks.com

I’d been running Dave’s Picks over on my personal site up to this point, but in November, 1999, I bought the davespicks.com domain name, and started running pre-release versions of Manila. I spent 2000 trying out cool ideas using Manila and generally being happy with the product, though its mysterious crashes got on my nerves.

Leaving Manila

This went on until March, 2001, when I’d finally had enough. The same bugs that I’d reported late in 1999 were still there in 2001. They would crash my server hard, and as near as I could tell, nobody was looking into the problems. They certainly weren’t getting solved.

So I bought my first x86 machine in nearly 15 years. It’s an AMD box, built by General NanoSystems. I wanted a bit of security (another thing lacking in Manila), so I installed OpenBSD 2.8 and a slew of patches, got Apache up and running, installed PHP and got that working. Then, because that was working too well, I decided to upgrade to OpenBSD 2.9, the latest Apache, PHP and MySQL and had everything completely broken for a while. Finally got back to a clean 2.9 build and documented how I needed to build all the various pieces, and that’s what’s serving these pages.

The twice-weekly updates of Dave’s Picks were still generated using Frontier in March of 2001. Many of the other pages are built using BBEdit, and use Apache Server Side Includes or PHP to supply things like the navigation on the right (I used to have navigation both on left and right, but decided I liked things better with just one navigation bar -- you can’t get to a specific month in the archives with a single-click anymore, but there’s a search now, so you don’t need to, either).

The search technology is a kludged-up version of DGS Search that I’ve modified so it generates relatively clean HTML 4 results, rather than using the font tags that the original did.

My best.com site goes away

In December, 2001, prompted by Verio killing the server at best.com that had held those web pages for me since 1995, I wrote a bunch of software to wrap every page at Dave’s Picks a template to give the site a consistent look (including the hundreds of pages I migrated from best.com). This software’s written mostly in PHP, and is based on the template class within PHPLib. It also uses mod_rewrite to keep the URLs looking pretty (though if you get something wrong in mod_rewrite, it’s quite possible to have a server task chew up incredible amounts of memory, which can make the server pretty sluggish. Beware the infinite loop.

Rather than having to re-render all the pages in Frontier just to change the look, I can edit one template file by hand and everything changes. I’ve also got a better chance of cleaning up the code so I can make the transition from HTML 4.01 Transitional to XHTML 1.0 one of these days. There’s little chance of Frontier 5.0 ever generating XHTML for me.

Of course in changing to using PHP templates, I broke the search I had running when I did that, but I expect to have it working soon. And with the newfound flexibility, I can maybe drop Frontier entirely. I’ll be happy to not have it crashing my desktop Mac for me.

There’s also the sitemap. Since I was mucking with everything, I decided I’d add some intelligent caching to the sitemap. Now when you ask for the sitemap, it checks to see if the existing map is "current enough", and if it is, the cached version gets returned, rather than worrying about building a new one each time. Quicker response is good, especially since I display the sitemap when a requested page isn’t found.

Leaving Frontier entirely

The most recent changes (30. December, 2001) are the Suggest a Pick script and changes to the CSS and HTML to throw a background behind the date lines in the picks to make them stand out better.

The Suggest a Pick script pops a form for you, takes the information, formats it up just like in the final output version and writes it to a file only I can access. I’m currently manually moving things from that file into the main page, but the plan is that I’ll eventually write more scripts to automate this process. The bonus is that I can collect the various items until I’ve got enough for a theme. Having a script on my server also means that I’m no longer using Frontier 5.0 for the day-to-day work of keeping Dave’s Picks up to date. Everything new is added via the web, and I can do the editing either via BBEdit’s FTP functionality, using vi from a shell on my server, or any of a number of other ways.

The idea behind highlighting the date line for each entry is that eventually each date will be a separate file (or will at least look that way from the outside). Putting in the extra information now will allow me to break the old archives into pieces automatically, rather than having to do it by hand.

I make per-day archives (and more) [3. January, 2002]

Error reading .writing/software20020103.html

Better inclusion & permalinks [5. January, 2002]

Error reading .writing/software20020105.html

Including just the titles for some days [7. January, 2002]

Error reading .writing/software20020107.html

Redirecting pages I’ve moved [24. February, 2002]

Error reading .writing/software20020224.html

Showing how folks are getting here [4. March, 2002]

Error reading .writing/software20020304.html

More about the last 10 referrers [10. April, 2002]

Error reading .writing/programming/last10refs.html

Improving broken URL handling [4. May, 2002]

I describe how I improved handling incoming requests with an extra period or greater-than at the end of the URL in writing/programming/progexample1.html.

 

Multiple Categories [23. October, 2004]

There have been multiple changes I didn’t document, but I’ve updated the Suggest a Pick script so that multiple categories are supported per item. This is part of a larger plan to do the following:

  1. Give each month its own directory in the archives (working for 1997 - other years to come)
  2. Do some serious mod_rewrite hacking so none of the old links break
  3. Make per-month archive listings more automagic (currently I build them by hand)
  4. Make each item a separate file, so permalinks won’t have those #s in them
  5. Update the display code so you see a permalink for every entry
  6. Make per-day archive listings automagically, rather than manually
  7. Write code to automatically flag broken links (404s) and update 301 links
  8. Add icons for each of the categories so you can see things more easily
  9. Make per-topic archives available automagically
  10. Finally hook up comments

I’ve got a lot of work left to get all that done, but one of the first things I wanted to do was have multiple categories for a single posting. That lets me categorize things better as I enter them, and capturing that data is important to me. The rest of it is all display stuff, and I can worry about that as time goes on. But one of these days I’d still like to get Dave’s Picks being a real content management system with a lot less manual intervention on my part. I’d like to think this is the first step along the way.

 

Per-month Archives [23. November, 2004]

I finially finished switching over the archives so each month has a directory. Thanks to Bill for some help writing the perl script that did much of the work. The to-do list is now:

  1. Give each month its own directory in the archives (done)
  2. Do some serious mod_rewrite hacking so none of the old links break(done, as far as I know)
  3. Make per-month archive listings more automagic (currently I build them by hand)
  4. Make each item a separate file, so permalinks won’t have those #s in them
  5. Update the display code so you see a permalink for every entry
  6. Make per-day archive listings automagically, rather than manually
  7. Write code to automatically flag broken links (404s) and update 301 links
  8. Add icons for each of the categories so you can see things more easily
  9. Make per-topic archives available automagically
  10. Finally hook up comments
Copyright 2009, Dave Polaschek. Last updated on Mon, 15 Feb 2010 13:47:50.