Did you know that your tweets have an expiration date on them? While they never really disappear from your own Twitter stream, they become unsearchable in only a matter of days. At first, Twitter held onto your tweets for around a month, but as the service grew more popular, this "date limit" has dramatically shortened. According to Twitter's search documentation, the current date limit on the search index is "around 1.5 weeks but is dynamic and subject to shrink as the number of tweets per day continues to grow."
What that means is something tweeted prior to a week and a half ago can never be retrieved via search.twitter.com. That's bad for users and it's definitely bad for data-mining. Unless Twitter corrects this issue on its own, we have to find another solution for archiving tweets ourselves. Here are 10 ways to do so.
]]> One of the unfortunate side effects of the FriendFeed acquisition is the very real possibility that the company will eventually shut down its servers. There are many reasons why this is upsetting - the site's users now have to figure out how to extract everything from their natively posted content to their comment streams - or lose them forever. However, one of the most disappointing losses will be losing FriendFeed's search feature. Since the service functioned as an aggregator of the social web, most users piped their tweets into FriendFeed, making the site a searchable archive of tweets which were still available no matter how old they were - quite unlike Twitter's own search. But if FriendFeed is going to disappear, we need to consider some alternatives.1. The Archivist: A Desktop Tool for Archiving Searches
The Archivist is a Windows desktop software application built by members of Microsoft's Mix Online team. With this program, you can create Twitter searches which will then be archived to your PC so they can be data-mined by you at a later date. Recently, the program was updated so that it can be minimized to the system tray - especially helpful for when you want to track a Twitter search over a long period of time. They also added a data visualization feature which calculates who's tweeting the most about your topic.
2. Twapper Keeper: Archive Tweets Based on Hashtags
Twapper Keeper is an online tool which archives tweets based on a given hashtag. Once you set up a query, Twapper Keeper will periodically scan Twitter for that tag and then archive the tweets it finds on its own servers. Tweets are scanned approximately every 5 minutes but that can vary based on the velocity of the incoming tweets. Once archived, you can then organize the tweets into categories of your choosing which show up on the right-hand side of the archived page.
3. Twitter Tools: Archive Tweets in WordPress
Twitter Tools is a WordPress blog plugin which integrates your blog and Twitter account. Once installed and configured, the plugin can be used to both Twitter links to your blog and to create posts which contain your recent tweets. While this is handy for the WordPress blog owner, keep in mind that post after post of "Today's Tweets" isn't all that appealing to blog readers. You may want to create a separate blog for this if you intend to use WordPress as your own personal Twitter archive.
4. Twistory: Tweets in Your Calendar
Twistory is a service which lets you add your Twitter backlog feed to your favorite calendar application. The service lets you subscribe to any user's Twitter RSS feed which can then be integrated into a calendaring application like Google Calendar, Outlook, Thunderbird, or any other application that supports the iCal format. Tweets are added immediately to the calendar upon posting and the service can even import almost all the way back to your first tweet ever.
5. SweetCron, AmpliFeeder, or Storytlr: A Lifestream of Tweets
We reviewed SweetCron back nearly a year ago when it first launched. Essentially, this self-hosted lifestreaming application lets you create your own customizable version of a FriendFeed-like service, but one that's hosted on your own server. The software is installed on a server with PHP and MySQL running on it and then must be configured with the social media sites you want to aggregate. Of course, one of the sites you can pull in is Twitter.
AmpliFeeder works the same way and includes a number of themes to choose from. This service can also generate XML files which can be used to backup all your social streams' data, too. Recently, Amplifeeder launched their hosted version of the service (get started here), but unfortunately, it relies on FriendFeed to import your lifestream. And who knows how long that will be around.
Storytlr is a third option for creating a lifestream at your own custom URL. However, it's doesn't just function as an aggregator - it also lets you post your own, unique content too. (Our review).
6. Twinbox: Tweets in Your Outlook
TwInbox is an Outlook plugin which lets you receive your friends' updates directly in your Outlook inbox. With this plugin activated, you can search, archive, and group your tweets the same way you manage your email. You can also update your Twitter status from Outlook or retrieve tweets based on keyword searches. Of course, if you're in a corporate environment, your I.T. admin may enforce mailbox size limits which means you'll have to archive your "tweets" folder more often than you'd like to keep your PST/OST to a manageable size.
7. RSS Feeds
One of the simplest ways to archive tweets is to simply grab the RSS feed for your tweet stream of choice and add it to your preferred RSS reader. If you use Google Reader, for example, the service's "infinite scrolling" feature lets you go back to the very first item ever tracked for that subscription. Plus, Reader has a search box at the top for searching for keywords or phrases in your feeds which will also help you rediscover older tweets. (Update! See our post "How to Backup and Search All Your Friends Tweets in Google Reader" for how-to instructions on using RSS as a backup method.)
8. Tweetake, TweetDumpr, Tweetscan, BackupMyTweets, TweetBackup: Twitter Backup Tools
Tweetake is a third-party service which lets you back up your Twitter account including tweets, DMs, followers, friends, and favorites. However, Tweetake is limited to backing up only the first 1000 contacts and there's no way to re-import the data into Twitter at the moment. It's mainly used as a personal archive of data which you can store for safe-keeping. TweetDumpr is a similar service but only focuses on backing up the tweets themselves from public timelines. Tweetscan also lets you download your Twitter message archive including replies to a CSV file. BackupMyTweets and TweetBackup are yet two more Twitter backup services which lets you backup your tweets daily.
9. Archive Tweets with Evernote
The popular note-taking application Evernote offers a feature which lets you send your public Twitter messages and private DMs right into Evernote where they can be searched and stored permanently. Unfortunately, in order for this service to work, you have to put "@myEN" into the body of any public tweet, making it less-than-ideal for archiving your every missive. Instead, this is a good way to archive select tweets you wouldn't want to lose.
10. Geek Tools Let You Archive in XML, PDF, HTML, TXT...or even with Python
RSS guru Dave Winer released a tool earlier this year which archives Twitter posts using the OPML Editor and optionally synchronizes with a structure on Amazon S3. Alternately, there's this simple Python script for archiving tweets. Sourceforge also hosts an app with lets you backup up tweets of different users as XML, HTML, PDF, or TXT. However, it can only perform backups of 3200 tweets at a time. Each subsequent backup will append the additional tweets to the current existing archive.
Note: To create this list, we asked Twitter for help. Thanks to @webdesignfanboy, @leemathews, @christhilk, @buerstinghaus, @nroy, @wesley83, and @markwoodhams for your responses.
]]>