Piping Data into Proto | February 16, 2007
by Byron Binkley
Searching and browsing Craigslist using Proto and Yahoo! Pipes:
Here’s a YouTube video walking through the application and the Yahoo! Pipes tool:
(There’s a higher resolution version of the video on our website.)
Like Jeb, when I saw Yahoo! Pipes I had to play with it. Unlike him I couldn’t resist the blog title pun. Having read a little and heard a lot about Pipes, I was itching to find out how it fits into the mashup space. In Yahoo!’s introduction they say pipes is to “remix feeds and create new data mashups in a visual programming environment.” Sounds good to me.
After poking around some of the pipes that other people built, I got a feel for the basics and then took a crack at building something new. It was easy to remix feeds into new feeds, but there wasn’t any way to consume the cool new feed you build in the Pipes environment. I think they fit into the same space of mashup enabling as Dapper: mashing data into feeds that can then be consumed by other apps or mashups.
I decided to build a pipe to serve up data to consume with Proto. Proto rocks the data manipulation, cleaning, aggregation, and well – a lot of the analytical stuff you wouldn’t expect to see in Pipes. My choice really came finding that the “Location Extractor” processor pulled addresses out of feeds and then added geocoding to the output. What feed should we add locations to? Craigslist apartment listings, naturally.
Here’s the pipe I built that can take some REST parameter inputs and then output a geocoded feed with only the Craigslist apartment listings that it thinks have solid addresses in them.
Since every pipe publishes a REST API, I was able to access it using the REST API Query module in Proto. Once I wrapped up the pipe as a Proto component, I was free to put it to work to gather a bunch of Craigslist listings all at once (which you couldn’t do with pipes), and then browse, store, and process those listings locally.
I decided to make an apartment hunting tool from start to finish. You can download the application from our gallery if you want to play around with it. (You’ll also need to get Proto to run it.) Overall this might be my favorite Proto tool to date, and it’s thanks, in part, to Yahoo! for piping in the data.
In lieu of a blow-by-blow description of what it does, watch the videos above, or check out these additional screenshots:
A screen to manage listings, filter out some of them, and see incoming emails from brokers you’re working with:
See only your top picks on a map with the real monthly rent that takes into account broker fees, other monthly costs, and the number of roommates you will have:
Finally, automatically generate a summary email with a custom map in Outlook:
