How To: Backup And Search All Your Friends' Tweets In Google Reader
I just set up an automatic backup of all 3000 of my friends' Twitter messages and became able to search through their Twitter history two years into the past with just five minutes of easy clicking. Only two things are required: Dave Winer's new Twitter OPML tool and a Google Reader account.
Twitter's search engine only goes back about a week and a half. Sometimes you want to retrieve a message you saw, or get a feeling for what your circle of friends said about something, from longer ago than that. We wrote yesterday about 10 Ways To Archive Your Tweets. The next step is to archive the Tweets of everyone else you find of interest, and make them searchable.
Last week RSS forefather Dave Winer wrote and posted a little tool for pulling any Twitter user's friends list out of Twitter and saving it as an OPML file. It's part of his broader open real-time messaging project called RSS Cloud.
OPML stands for Outline Processor Markup Language and in this case it's just a bundle of RSS feeds than can be moved around in bulk. It's a beautiful idea that has a lot more potential than has been realized, but you'll see how it comes in handy here.
It's all about pulling down an OPML file of your Twitter friends' feeds and slapping that file into Google Reader. Then it's archived and searchable. It's very easy to do.
How to Make it Happen
It couldn't be much simpler. Just put your Twitter username into this link, instead of mine, and load it up in your browser: http://tw.opml.org/get?user=marshallk&folder=1
It may take just a minute, but the end result will be an OPML file. You can either go up to your browser's File menu and select "save as" or you can View Source and copy and paste the source of the page into a text document. Save it with a memorable name and either .xml or .opml as the file type. It's really quite easy.
Now if you want to put this puppy into Google Reader just log in, click on "manage subscriptions" and find the import/export button. Import that file into Google Reader and you're ready to rock and roll!
Update: The combination of interest from this post and the limitations of Twitter's API has caused a temporary challenge for this service and Winer says he's had to limit friend extractions to 1,000 per user for now. Give it a go, but bookmark it for later when there's less of a rush and it can be less of a proof of concept.
What Can You Do With This?
Do you want to read Tweets through Google Reader? Probably not. But do you want to archive, retrieve and search that way? It works remarkably well! Especially because of the social nature of Google Reader. If anyone you're following has had their RSS feed read by anyone in Google Reader, ever, the system will have an archive of their tweets that goes far beyond what's immediately available in their RSS feed right now. Thus my ability to pull up tweets from two years ago in a search.
You may want to create a separate Google Reader account for this (I did) so the Tweets don't clutter up other feeds you like to read.
There are certainly other little things you can do with easy OPML files of Twitter followers as well. You can't import them directly into Twitter clients (yet) but you can share and trade them into an RSS reader as a preview mechanism before deciding to subscribe in Twitter proper. (My favorite tech analysts on Twitter, informative LGBT activists on Twitter, etc. in OPML format - collect 'em all!)
These are the kinds of things that make simple protocols for dynamic information delivery, like RSS and OPML, so much fun.
You could create and share OPML files of your favorite twitter users concerning a particular topic. You could build an OPML file of a group of twitter users and have their RSS feeds automatically displayed on a page on your website.
You could assemble different groups of people into different OPML files, saved in different folders in your Google Reader account, and then limit searches to one folder or another in order to get a sampling of what various groups of people have to say about a topic. That would be hot!
The possibilities are endless, but the most basic use cases of archiving and search are already worth doing. We'd love to know how you can imagine using tools like this together.
Caveats
It's not clear how quickly Google Reader is updating its record of tweets and it is definitely missing a lot. We searched for some specific twitter messages from several months ago from people whose Tweets are being subscribed to in Google Reader, but that it didn't retrieve. I hope that won't happen with user feeds I've subscribed to myself now, but take it with a grain of salt. Google Reader isn't really a terribly serious product, anyway. It could come in quite handy, but there is a larger value here in getting your hands on an OPML file of your Twitter friends' messages.
How can you imagine putting an OPML file like this to use?
You can find ReadWriteWeb on Twitter, as well as the entire RWW Team. Please follow: Marshall Kirkpatrick, Bernard Lunn, Alex Iskold, Sarah Perez, Frederic Lardinois, Jolie Odell, Dana Oshiro, Steven Walling and Lidija Davis.
Share
Facebook
Comments
Subscribe to comments for this post OR Subscribe to comments for all ReadWriteWeb posts
I could see this as a valuable "listening" and monitoring tool, particularly on the feeds related to keywords. Great tip, thanks!
The main problem with Google Reader is that you dont get all the entries in a feed downloaded, only the most recent 20 or so. You may have downloaded 3000 tweeets but those are spread across 1000+ friends. This means your search queries are limited to only the past couple days or so, not a true search through teh complete twitter history of you and your friends.
What you really need is an effective way to simply download your tweets incrementally and store them locally, preferably in XML. You can then regenerate an RSS feed from them or transform them via XSLT to a dedicated webpage independent of twitter as you desire.
I wrote a post a while back about various backup strategies for twitter (and also did the math for how much disk space a billion tweets woudl require - the answer may surprise you). I've updated that post with some n ewer solutions and will add links to your recent posts too, Marshall.
BTW, downloading is the ONLy solution for another reason - twitter only keeps 3200 of your tweets archived. Once you exceed that many, you start losing tweets irrevocably.
On Sunday at StomperNet Live 8 Dan Hollings talked about archiving tweets and it was something I hadn't thought of. He was personally archiving all #SNL8 tweets for posterity. I then just thought of it for business reasons, like saving testimonial tweets, but using it for friends too is a good idea - a lot changes in a week and half.
Thanks for the post,
Fran.
I just tried doing what Marshall suggested and got just 20 tweets or less per person show up in GR.
Thanks for your blog post Aziz, I tweeted it - good backup info there - and although you say none of the options are perfect I'll try them all on for size. :)
Fran.
Fran, did you scroll down to the bottom of the list and then keep scrolling? It's also possible that the people you tested have not been subscribed to by RSS in Google Reader before, so there's no archive saved of their Tweets there yet.
Marshall,
It would be nice and useful to have a "retwit" button here on the post page, no ? ;-)
Good call, just added that.
Marshall, thats the problem - you can only retrieve 20 entries at a time for a given feed, you can of course keep scrolling but if you are importing thousands of tweets then it will take an hour of tedious scrolling and refreshing to load them all in. this isnt really a backup solution, just a search-recent-tweets solution. True backup strategies need to be local, not in the cloud.
Not sure what the problem is with others, but I can go back more than 2 years for all of the accounts ive subscribed to in GR. I'd scrolled back 2,000 of Scobles tweets before I got bored.
and dont forget you can now sync Google Reader to NetNewsWire. A much better tool for searching/navigating tens of thousands of items.
Despite the limitations, this is a really cool "how to." Hopefully, those behind Google Reader will recognize the potential for this kind of Twitter integration into their increasingly social product.
Genius.
Posted by: mrshl
|
August 13, 2009 8:03 AM
I've been trying to find a way to archive my posts for side project. The easiest way to do this is to use RSS and add a count. Example below;
http://twitter.com/statuses/user_timeline/15271202.rss?count=1099
At the end of the RSS add ?count=(how many posts do you want)
then you view source, and save the file to your local machine.
How To: Backup And Search All Your Friends' Tweets In Google Reader http://bit.ly/GU8ne [from http://twitter.com/marshallk/statuses/3266421928]
Posted by: Marshall Kirkpatrick
|
August 14, 2009 7:24 AM
Would that work with Yahoo! Pipes?
Things can get pretty interesting there...
I've subscribed as mentioned in the article. It works great. Searching to older tweets or from one of your followers is easy. Only thing was, that some of my followers had a little archive and some go back into "earlier days". I don't know how that is possible.
Google Reader has folders. Mine was just there 'add to Twitter/subscriptions folder' so I did then I'll just keep that folder minimized when going through reader (I read by my own categories not "all"). Couldn't be easier. No separate account needed. So new to twitter took MAYBE a minute. New to twitter people need to do this asap.
OPML deserves so much more attention. We've used Dave Winer's app to generate and run the OPML feed bundle through an automatic tagging system. Here's the Jay Rosen @ Twitter Term Cloud
that shows the top 100 terms. You can click on each term to see the relevant tweets.
And as a bonus, the RWW Term Cloud.
If you have the MashLogic add-on you can "subscribe" to this and see links from these terms on any web page.
If you are trying to get more than 20 entries from Google Reader, append the n=99
For Example :
All the 17 first items items from xkcd.com main feed that are not read can be found on the url : http://www.google.com/reader/atom/feed/http://xkcd.com/rss.xml?n=17&ck=1169900000&xt=user/-/state/com.google/read
Check out here for more information :
http://code.google.com/p/pyrfeed/wiki/GoogleReaderAPI
I use Twitter in Google Reader using greasemonkey script http://userscripts.org/scripts/show/49958 The script also allows you to create OMPL files of friends/followers for importing into google reader.
Regarding this:
"http://twitter.com/statuses/user_timeline/15271202.rss?count=1099
At the end of the RSS add ?count=(how many posts do you want)
then you view source, and save the file to your local machine."
This wont work! Because as per twitter api documentation only the last 200 entries are allowed with count parameter (unless you use the page parameter)
great articles
there is an OPML addon for firefox that allows users to import/export bookmarks in the OPML format.
I wont include the link to the addon but a google search will find it.
I only realised after reading this that Google Reader actually archived tweets - I have tweets going back 18 months.