Comments for The Glory, Bliss and How-to of Screen Scraping for RSS

The Glory, Bliss and How-to of Screen Scraping for RSS

2008-01-01T04:57:24Z

Wired has an awesome top story today on the world of startups utilizing scraped data from big companies to offer new layers of value for their own users. It's a roughly objective piece that I highly recommend reading but it was also inspiration for me to finally record a screencast on the subject (see below).

I love RSS, probably more than anything on the web. If you're not familiar with the concept, see my very old definition of RSS and my almost-as-old post on teaching people about RSS.

Not every page on the web publishes an RSS feed, though. Thus the need for these wonderful screen scraping tools. I've written about a variety of tools you can use to create a feed for a site or page that doesn't have one. Sometimes, though, you've got to pull out the big guns. In those cases, it's time for Dapper.

]]>

Dapper is a company founded in Israel, now venture backed and was named in the aforementioned Wired article. It is the sweetness.

Dapper will let you pull data from almost any web page and get it in a wide variety of outputs, including RSS, email, iCal, a Google Gadget, CSV and Google Maps. Is that incredible or what?

Let's let the video do that talking. I have an awful cold (it's almost better, Mom!) so please excuse the very rough voice. I made the following screencast using JingProject, setting up an RSS feed of search results in Del.icio.us for articles tagged from ReadWriteWeb.

Clicking on the image below will open up another window so you can view the 4 minute video full screen.

If you're as excited about Dapper as I am, you should check out DapperCamp, a two day free conference all about Dapper coming up in early February in San Francisco. IBM and Mindtouch are sponsoring the event and Mitch Kapor is keynoting it. It looks like it's going to be a lot of fun.

Take that, Wired Mag ambivalence! Really, though, you should read that Wired article - it's a good one that discusses some issues that are going to be very big once more people figure out how exciting data portability is.

Comment from Ben Greenberg on 2008-01-01

2008-01-01T08:38:04Z

Just watched the screencast. That was way cool.

Comment from Dr Nic on 2008-01-01

2008-01-01T09:28:42Z

Very good screencast. Great enthusiasm.

Comment from G on 2008-01-01

2008-01-01T13:44:47Z

Dapper is perhaps the best scraping tool available today - Extremely user-friendly, especially for us non-programmer types - registration to first Dapp in less than 5 minutes. I also appreciate their customer service - certainly stands apart when it comes to an early stage company.

Well deserved coverage!

Comment from Jansie Blom on 2008-01-01

2008-01-01T13:56:12Z

This is a great post. my first exposure to scraping. looking good.

Comment from Neville Hobson on 2008-01-01

2008-01-01T14:07:11Z

Great screencast, Marshall. Explains it all very nicely.

Comment from David Scott Lewis on 2008-01-01

2008-01-01T14:09:13Z

I tried it and didn't like it. I still have it as a Firefox add-on (DapperFox), but it's disabled.

I can't see what all the excitement is about. I have other ways for doing all (or, at least most of) the things that Dapper can do.

Biggest downside: Seemed to seriously impact Firefox performance.

Comment from RSS Fan on 2008-01-01

2008-01-01T15:24:05Z

Hi All,

Same as you, I love RSS. To help non-geek people, I created http://milliondollarchannel.com, which intends to provide alternative list of feeds than technorati etc.

Does it makes sense? I plan add more feeds into it. Which feeds do you think are worth to list?

Thanks!

Comment from ITrush on 2008-01-01

2008-01-01T15:26:34Z

This is cool, thanks for the screencast, now I know where to go. :)

Nhick
http://www.itrush.com

Comment from Deepak on 2008-01-01

2008-01-01T18:49:34Z

Marshall

Great screencast. It's kind of sad that not more people are aware of the power of services like Dapper, which IMO, can really really increase their productivity.

Comment from Ymo on 2008-01-01

2008-01-01T21:44:39Z

I have been using http://www.feedyes.com for scraping websites a long time. Creates a valid rss feed. Works for every website, with every browser, and is reliable. Only thing: feeds are updated once every hour, and not realtime.

Comment from Chris Saad on 2008-01-01

2008-01-01T22:18:56Z

Funny you mention Dapper in relation to DataPortability Marshall - I had recently posted a post to the GraphSync project (sponsored by the DataPortability workgroup) about using Dapper and others for the cause.

Here it is here:
http://groups.google.com/group/graphsync/browse_thread/thread/81c69e0aa786a3df

Comment from Charles David Livingston on 2008-01-01

2008-01-01T22:33:11Z

Marshall

A very interesting tool and a very user-friendly screencast. Many thanks.

Best wishes for 2008!

Charles

P.S. I also find Jing to be a very handy free screencast tool too.

Comment from billwatson on 2008-01-02

2008-01-02T15:07:37Z

I have used dapper to get some good rss feeds during my experimenting time with http://semantisize.com

Dapper seems to work great so far.

Comment from Alex on 2008-01-02

2008-01-02T15:34:50Z

Orchestr8's AlchemyPoint is another solution that offers advanced screen-scraping capability. A screencast is available here. Another note: AlchemyPoint can be utilized "in-the-cloud" or installed locally for behind-the-firewall screen scraping.

Comment from Jason on 2008-01-02

2008-01-02T16:54:52Z

Dapper is a great tool... at first, it can be a bit confusing, but quickly becomes easy to understand. I love it... Great post, very good information on Dapper.

Comment from Ben Tremblay on 2008-01-02

2008-01-02T19:06:41Z

The ListPic example from Wired serves as a great example ... I think we can draw something concerning "fair use" from this. After all, we aren't talking about someone sharing links with friends and the occasional reader:

"43,000 visitors a day and thousands of dollars a month in Google AdSense revenue."

I've been beavering away at the distinction between "marketing" and, say, sales or coding or Q&A or other more obviously production-related aspects of rolling something out.

IP arises from effort, ok? So buddy created a web-scraping site ... and a good one ... well-coded enough to deliver the goods with that many hits, and (I can only guess) well designed enough to attract a following. That deserves something.
But he was poaching ... picking up his stock from someone else's loading dock, re-packaging and then selling.

Metaphorically, he had an atractive store in a good location ... but the stuff folk were selecting from his shelves, where did it come from?

I've been in "stealth mode" for a good long while *Toooo long!* for a single simple reason: I can't find a way of sharing cream and cheese without some marketing-type taking not only my cow but more ... and I don't intend on giving away the farm.

Comment from jb on 2008-01-02

2008-01-03T01:29:42Z

dont forget about open kaPow. Its an amazing product and openKapow is free.

http://kapowtech.com/products_data_collection.html?google&keyword=dataextraction&gclid=CI7s0tHU2JACFRE_OAodjQgbZQ

Comment from Megan Keane on 2008-01-03

2008-01-03T16:57:41Z

Thanks, Marshall, great information here and I can't wait to play around with Dapper further. I've shared this tool with others on TechSoup: http://tinyurl.com/282jo6

Comment from Mike Doeff on 2008-01-03

2008-01-03T23:42:39Z

Great tutorial. Do you know if Dapper supports the ability to create an RSS feed with enclosures - e.g. photos, MP3's, etc.?

Comment from Good Credit on 2008-01-04

2008-01-05T01:31:51Z

Last week I applied for a credit card. The website was cool, absolutely what I was looking for! Visit if you need good service. They have credit cards for bad credit score. Please tell me if you like it

http://cardits-immidetly.cn/discover-interest-credit-lower.html >discover credit card lower interest rate

R56ma87de