10 Ways to Archive Your Tweets
Did you know that your tweets have an expiration date on them? While they never really disappear from your own Twitter stream, they become unsearchable in only a matter of days. At first, Twitter held onto your tweets for around a month, but as the service grew more popular, this "date limit" has dramatically shortened. According to Twitter's search documentation, the current date limit on the search index is "around 1.5 weeks but is dynamic and subject to shrink as the number of tweets per day continues to grow."
What that means is something tweeted prior to a week and a half ago can never be retrieved via search.twitter.com. That's bad for users and it's definitely bad for data-mining. Unless Twitter corrects this issue on its own, we have to find another solution for archiving tweets ourselves. Here are 10 ways to do so.
One of the unfortunate side effects of the FriendFeed acquisition is the very real possibility that the company will eventually shut down its servers. There are many reasons why this is upsetting - the site's users now have to figure out how to extract everything from their natively posted content to their comment streams - or lose them forever. However, one of the most disappointing losses will be losing FriendFeed's search feature. Since the service functioned as an aggregator of the social web, most users piped their tweets into FriendFeed, making the site a searchable archive of tweets which were still available no matter how old they were - quite unlike Twitter's own search. But if FriendFeed is going to disappear, we need to consider some alternatives.
1. The Archivist: A Desktop Tool for Archiving Searches
The Archivist is a Windows desktop software application built by members of Microsoft's Mix Online team. With this program, you can create Twitter searches which will then be archived to your PC so they can be data-mined by you at a later date. Recently, the program was updated so that it can be minimized to the system tray - especially helpful for when you want to track a Twitter search over a long period of time. They also added a data visualization feature which calculates who's tweeting the most about your topic.
2. Twapper Keeper: Archive Tweets Based on Hashtags
Twapper Keeper is an online tool which archives tweets based on a given hashtag. Once you set up a query, Twapper Keeper will periodically scan Twitter for that tag and then archive the tweets it finds on its own servers. Tweets are scanned approximately every 5 minutes but that can vary based on the velocity of the incoming tweets. Once archived, you can then organize the tweets into categories of your choosing which show up on the right-hand side of the archived page.
3. Twitter Tools: Archive Tweets in WordPress
Twitter Tools is a WordPress blog plugin which integrates your blog and Twitter account. Once installed and configured, the plugin can be used to both Twitter links to your blog and to create posts which contain your recent tweets. While this is handy for the WordPress blog owner, keep in mind that post after post of "Today's Tweets" isn't all that appealing to blog readers. You may want to create a separate blog for this if you intend to use WordPress as your own personal Twitter archive.
4. Twistory: Tweets in Your Calendar
Twistory is a service which lets you add your Twitter backlog feed to your favorite calendar application. The service lets you subscribe to any user's Twitter RSS feed which can then be integrated into a calendaring application like Google Calendar, Outlook, Thunderbird, or any other application that supports the iCal format. Tweets are added immediately to the calendar upon posting and the service can even import almost all the way back to your first tweet ever.
5. SweetCron, AmpliFeeder, or Storytlr: A Lifestream of Tweets
We reviewed SweetCron back nearly a year ago when it first launched. Essentially, this self-hosted lifestreaming application lets you create your own customizable version of a FriendFeed-like service, but one that's hosted on your own server. The software is installed on a server with PHP and MySQL running on it and then must be configured with the social media sites you want to aggregate. Of course, one of the sites you can pull in is Twitter.
AmpliFeeder works the same way and includes a number of themes to choose from. This service can also generate XML files which can be used to backup all your social streams' data, too. Recently, Amplifeeder launched their hosted version of the service (get started here), but unfortunately, it relies on FriendFeed to import your lifestream. And who knows how long that will be around.
Storytlr is a third option for creating a lifestream at your own custom URL. However, it's doesn't just function as an aggregator - it also lets you post your own, unique content too. (Our review).
6. Twinbox: Tweets in Your Outlook
TwInbox is an Outlook plugin which lets you receive your friends' updates directly in your Outlook inbox. With this plugin activated, you can search, archive, and group your tweets the same way you manage your email. You can also update your Twitter status from Outlook or retrieve tweets based on keyword searches. Of course, if you're in a corporate environment, your I.T. admin may enforce mailbox size limits which means you'll have to archive your "tweets" folder more often than you'd like to keep your PST/OST to a manageable size.
7. RSS Feeds
One of the simplest ways to archive tweets is to simply grab the RSS feed for your tweet stream of choice and add it to your preferred RSS reader. If you use Google Reader, for example, the service's "infinite scrolling" feature lets you go back to the very first item ever tracked for that subscription. Plus, Reader has a search box at the top for searching for keywords or phrases in your feeds which will also help you rediscover older tweets. (Update! See our post "How to Backup and Search All Your Friends Tweets in Google Reader" for how-to instructions on using RSS as a backup method.)
8. Tweetake, TweetDumpr, Tweetscan, BackupMyTweets, TweetBackup: Twitter Backup Tools
Tweetake is a third-party service which lets you back up your Twitter account including tweets, DMs, followers, friends, and favorites. However, Tweetake is limited to backing up only the first 1000 contacts and there's no way to re-import the data into Twitter at the moment. It's mainly used as a personal archive of data which you can store for safe-keeping. TweetDumpr is a similar service but only focuses on backing up the tweets themselves from public timelines. Tweetscan also lets you download your Twitter message archive including replies to a CSV file. BackupMyTweets and TweetBackup are yet two more Twitter backup services which lets you backup your tweets daily.
9. Archive Tweets with Evernote
The popular note-taking application Evernote offers a feature which lets you send your public Twitter messages and private DMs right into Evernote where they can be searched and stored permanently. Unfortunately, in order for this service to work, you have to put "@myEN" into the body of any public tweet, making it less-than-ideal for archiving your every missive. Instead, this is a good way to archive select tweets you wouldn't want to lose.
10. Geek Tools Let You Archive in XML, PDF, HTML, TXT...or even with Python
RSS guru Dave Winer released a tool earlier this year which archives Twitter posts using the OPML Editor and optionally synchronizes with a structure on Amazon S3. Alternately, there's this simple Python script for archiving tweets. Sourceforge also hosts an app with lets you backup up tweets of different users as XML, HTML, PDF, or TXT. However, it can only perform backups of 3200 tweets at a time. Each subsequent backup will append the additional tweets to the current existing archive.
Note: To create this list, we asked Twitter for help. Thanks to @webdesignfanboy, @leemathews, @christhilk, @buerstinghaus, @nroy, @wesley83, and @markwoodhams for your responses.
Share
Facebook
Comments
Subscribe to comments for this post OR Subscribe to comments for all ReadWriteWeb posts
I use Twilert to scan for hashtags and send me an email summary each day. Twilert.com
I also take screen shots of Tweets I a) want to save or b) pass on to others who are not on Twitter.
Plus you can mark certain Tweets as Favorites to keep them as long as you want.
@cnansen: Thanks for the tip re: Twilert!
I think the most poupular service for archiving tweets is http://TweetBackup.com
I t would be a piety if Friendfeed is shut down. It is a terrific aggregator. I use it with all my blogs.
A sample of #3 can be found here: http://twitter.waynejohn.com.
I finally decided to just create a subdomain called Twitter and dump everything into Wordpress there. It doesn't effect my primary blog, and I'm archiving everything, everyday.
Love it!
Great post!! Thank you for such helpful information to keep track of my tweets. ((huggs)) @dhatfield
Sarah, I don't know if you were reading our mind but we just updated our backup service at Tweet Scan today! It now calls the Twitter api to allow you to download your friends, followers, tweets and more and combines that with results from our database of 200 million tweets.
If people tweet the tag #dlmydata they can get theirs for free while supplies last ;)
https://www.tweetscan.com/data.php
You forgot to mention http://www.tweets2mail.com :)
in version 2 of the twitter api there are plans to be able to get all your tweets out of twitter.
here are the details
http://apiwiki.twitter.com/V2-Roadmap#Users
Nice list, but there's also http://ootweet.com — It allows you to save some tweets (and vote, share & favourite them).
Very helpful post, thanks! Does anyone know if there are similar options for backing up FriendFeed posts, in particular, the contents/history of a private FriendFeed room?
It seems that Twitter Tools and Wordpress would be a nice automatic, hands-off approach to archiving that I will be trying out. Good tip on not making the "archive blog" publicly available!
I dunno... This sound like archiving my hankies, in case I want to find something that I snoozed half a year ago.
Micro - Yup! I've yet to find a valid reason for archiving my tweets.
or you could also tag your twits with @fleetthoughts and it will be cataloged at http://fleetingthoghts.com/
Can't Google just archive and make searchable each and every tweet then if Twitter can't manage it? I'm sure they'd be up to it and this is where a great part of Twitter's value lies for me being able to mine my own and everybody's knowledge in the realtime social web.
I love using Twitter.... it's so much fun. I think Harvey Dent got a raw deal, but who played dent, that is the real question.
This is a terrific post - chocked full of really helpful info on this subject. Thanks for sharing. Learned about this from Guy Kawasaki (through a tweet link of course).
@Sarah, I have been using http://mytweetopics.com to archive my tweets. It does a lot more than just archiving - It lets me track my topics based on the #tags used in my tweets and also gives me RSS / Embeddable widget for each of the topics am tweeting about! Highly recommended !
Great post.It's so funny I never thought about backing up my tweets, I thought they would remain searchable forever.Thank you.
Informative artilce. Readers can get new infromation here which not tell to twitter users. Thank you for the information.
The Archivist is an excellent program that could be used for backing up one's tweets, but it can be used for any kind of twitter search. It searches every 5 minutes and adds what it finds to the file on the desktop on which it is running. It searches for history so it will go back as far as Twitter permits, and if down -- twitter does go down from time to time -- it will go back to the point at which it stopped. It produces an xml file and will export a txt file that is ready to read into excel. It is great for searching as well as for backing up one's own tweets. And they are helpful persons.
Archiving tweets - thanks for an excellent summary! I've been thinking more about searching for tweets about specific topics rather than just archiving my own tweets. Does the Archivist also save data into a database? This kind of archive should be very useful if you happen to look for info about a topic some time after the real-time discussion has taken place... I didn't realize up till now that Twitter search only provides results less than a week and a half old.
They should set the tweet expiry to 1 second. That way we wouldn't be subjected to this banal crap.
@DrorOrbach you should try http://www.mytweetopics.com . It is sorting all my tweets into topic buckets and looks really cool. I had the same request - to search within my tweets - and they said it would be launched soon! Waiting for that feature!
Sorry that was fleetingthought.com and don't forget the @fleetthoughts
Fat fingers
fleetingthoght.com a thought without the U!
Don't forget Lifestream Backup who is doing it too with alot of other service too
Want to backup just any type of data to S3? Try CloudBerry Backup. It is powered by Amazon S3 reliable and cost efficient storage. If you want to take part in beta sign up on the website http://cloudberrydrive.com What safer place to keep your files than Amazon's servers?
Check another amazing twitter 2 pdf app Tweetbook http://tweetbook.in/
You forgot to mention http://tweetbook.in, which generates a .pdf ebook (+ a xml backup file) of all your tweets (replies included), and also proposes to share your ebook through scribd, see http://tweetbook.in/v/18485601
That is really awesome information, I think I'll try out the Twistory and see what I think.
I would also like to point out that www.lifestreambackup.com backs up friends and followers too, not just tweets.
Sarah - thx for the coverage and overview of TwapperKeeper.com.
If anyone has any questions / enhancement requests for our service, please do not hesitate to contact me @jobrieniii or jobrien@ob3solutions.com
v/r,
John
Founder of TwapperKeeper (http://twapperkeeper.com)
"A weekend science project for a friend that has grown into a monster" :-)
great artilce. Readers can get new infromation from here. Thank for your share.
wowww...nice share..
Thank you for such helpful information to keep track of my tweets now..
regards
TOP News
http://blogtopshare.blogspot.com
Has anyone used or had experience with Twitterslurp, which was used at the PDF09 conference?
There's also my Tweet Tweet plugin at http://ocaoimh.ie/tweet-tweet/
It archives your tweets plus the tweets of everyone you follow. I've been using it since July last year.
As a bonus, it also sends sms notifications to Irish O2 or Vodafone users using their free web texts. Meteor's there too but they redesigned their site last week and I haven't got around to fixing it yet.
My guess is that doing a Twitter Search for tweets more than a few days old will become possible, but will require a paid subscription to Twitter.
Is it possible that people are already storing the Twitter public timeline (i.e. all non-direct tweets) to their own databases? If so, they could compete with Twitter for making that searchable.
Twitter's response could be to make the public timeline lossy to non-subscribers.
Actually I don't know Twitter will remove all my tweets from their search database. Thanks for listing these ways.
My twitter account is plugged to friendfeed. Would you say Friendfeed archive everything?
Anyone used Twittermail, http://www.twittermail.com/?
If you're using Supr, http://su.pr/ your tweets are likely to stay for 1 month in their archive.
Thanks for the info. I took the WordPress plugin Twitter Tools and made a automatic backup WordPress blog for myself.
I put some of my friends on subdomains so their Tweets will be safe in WordPress databases as well.
I'm open for helping others.
http://safetweet.net
Your WordPress Blog backup of your Twitter Tweets would be
http://username.safetweet.net
Easy to search and ordered by day and month.
I suggest using multiple services for redundancy.
very good.
please come to our wibesite:http://www.ztseals.com
It's a shame that http://printyourtwitter.com is not in this list! It is free & used a lot and now has a new filtering option and possibility to search keywords.
TwapperKeeper is now offering direct APIs to the archives...
http://twapperkeeper.com/api.php
Let us know if you have interest in using and what else you would like to see!
v/r,
John O'Brien
http://twapperkeeper.com
Founder of TwapperKeeper
Thanks! This is good to know. But we should have seen it coming. Because every day there are thousands coming on Twitter. Most of them is to let you know about their business like mine. http://www.bronziwebgiftstore.com/
Thanks for sharing these services.
1 2 Next