ReadWriteWeb

Earlier this week, The Wall Street Journal posted an article entitled "'Scrapers' Dig Deep for Data on Web". While the article highlights some important issues surrounding the murky and potentially shady business of Web crawling, it fails to provide a comprehensive story on the uses of Web crawling. In other words, by focusing on one or two companies with spotty business practices, it casts the entire practice of data collection from the Web as something to be feared.

Wired has an awesome top story today on the world of startups utilizing scraped data from big companies to offer new layers of value for their own users. It's a roughly objective piece that I highly recommend reading but it was also inspiration for me to finally record a screencast on the subject (see below).

I love RSS, probably more than anything on the web. If you're not familiar with the concept, see my very old definition of RSS and my almost-as-old post on teaching people about RSS.

Not every page on the web publishes an RSS feed, though. Thus the need for these wonderful screen scraping tools. I've written about a variety of tools you can use to create a feed for a site or page that doesn't have one. Sometimes, though, you've got to pull out the big guns. In those cases, it's time for Dapper.