October 29, 2009

Craigslist and Yahoo Pipes

by Nick Staroba

When I moved last spring I had a garage sale and one of the things I sold off to save storage space was my 75 gallon aquarium. I’ve been lugging the empty thing around for years because I’ve been moving so much and it was time for it to go. Well, now that the wife and I are finally in an apartment and an area that we like, I wanted to set up a smaller aquarium but I didn’t want to buy one new.

So I began browsing Craigslist and after a while started realizing what a pain it was going to be to try to surf through that every day because there were two different areas where I was willing to go to get what I wanted—Detroit and Ann Arbor—and those areas are split into two different sites on Craigslist. I noticed an RSS link at the bottom of all the pages including the search pages and I thought about subscribing to those. But then I remembered Yahoo Pipes.

I thought about how I would create my own Pipe to handle some filtering from both the AA and Detroit sites at once but I decided to do a search first. That’s where I found the “Craigslist Free Search” Pipe. Amazingly it was already set up for Michigan:

“Searches Flint, Ann Arbor, Saginaw and Lansing for FREE ads. Features include limiting the search to the 20 most recent, sorting by most recent across the 4 cities searched, prefixing cities with city codes for easy identification. Additionally, you can get only items with images by adding &hasPic=1 to feed URL.”

I cloned the Pipe and started customizing it. I replaced the searches to look in my areas and then I changed the RSS feeds to the more general “For Sale” category instead of the “Free” section. I then added two filters between the sort and the pipe output modules.

The first filter checks the item.title and item.description for my search terms (aquarium, fish, reptile and tank). The second filter narrows it down a bit further by scanning the URL of each item. Craigslist uses three-letter codes for different categories in their URLs so I set the filter to allow the categories that I wanted to see. For example, http://detroit.craigslist.com/gms/ is the category for garage sales so I added “gms” as a criterion to allow. I didn’t want to see any posts about “gas tanks” so I left out “pts” which is the auto parts category. I still see a few posts that aren’t what I’m looking for, but this helps a lot.

After I finished customizing the Pipe, I subscribed to the RSS feed in Thunderbird and now I get regular updates on my search terms pushed right to me. I know there are a number of ways to set this up and I wouldn’t say that original Pipe or mine are the best, but I’ve found some good deals without much effort and I wanted to share. If anyone has some tips on making this better, I'm listening.

info@stoutsystems.com
877.663.0877
© Copyright 1995-2019 - STOUT SYSTEMS DEVELOPMENT INC. - All Rights Reserved
envelopephone-handsetlaptop linkedin facebook pinterest youtube rss twitter instagram facebook-blank rss-blank linkedin-blank pinterest youtube twitter instagram