Comments for Is Twitter Strangling its Famous API?

Is Twitter Strangling its Famous API?

2009-01-21T21:20:50Z

The most extreme developers may find themselves left out in the cold.

Twitter watchers know that a large part of the service's use comes through its Application Programming Interface (API) and that's been a big part of what helped the young service grow. Now that the company has Britney Spears, CNN and Barack Obama among its ranks of users, though, developers seeking to push the limits of that API may soon find themselves no longer welcome.

]]> Last night Twitter announced on its developer email list that API calls from a single IP will be limited to 20,000 per hour. Desktop clients and services seeking only status updates won't be affected, only services pinging Twitter over and over again for information like users' friends lists or other relatively unsupported data types will likely run up against the limit.

No More "I Dislike You Too" Lists

What does that mean? For now it means that a handful of services that notify users who has unfollowed them will be effectively non-viable. That's far from our only concern about the issue, however.

Readers may remember a service called Qwitter, which emailed you every time someone unfollowed you on Twitter and told you what your last message posted was before that happened. On October 25th, a 15 year old UK blogger named "Joel D." stopped following me after I tweeted about being tired in ceramics class, for example. I don't hold a grudge but if you've got something against ceramics, Joel, then I say good riddance to you!

Seriously, though, a veil of paranoia and petty bitterness lifted from all of Twitter-dom just before Christmas when Qwitter finally gave up the ghost due to scalability issues on its end. (Apparently some people are still getting messages.) Update: Qwitter's founder dropped by to tell us that the service is now back up. So head on over, masochists of Twitter.

Now a service called SocialToo has raised the alarm about the forthcoming limit and says that a similar feature it offers will disappear as well.

What The Developers Say

SocialToo complains that Twitter should let it pay for heavy access to the API if it won't allow it for free, but Twitter apparently isn't interested. Third party developers also say that if the API was made more efficient, they could get the information they needed with less wear and tear on Twitter.

These developers who are unhappy with the new policy complain that it will unfairly limit their companies' growth, that it pulls the rug out from under supporters of the Twitter ecosystem and that it raises the possibility of Twitter reproducing the services they've worked hard to develop.

We're less concerned about unfollowing notifications in particular than we are with the ability for developers to push the limits of accessing this incredible store of data to create unforeseen and otherwise impossible innovations.

What does Twitter have to say about this? We asked them.

What Twitter Says

We pinged Twitter headquarters for a comment and this is what Alex Payne, API Lead at Twitter, had to say.

We picked the 20,000 requests per hour number precisely because it effects the fewest applications (less than ten of the hosts we see in our hourly report of high-traffic consumers of our site). In most cases, these larger Twitter applications make requests from multiple IPs. Since each of their IPs gets its own 20k/hour allotment of
requests, the developers behind these big Twitter API projects shouldn't have to lift a finger.

We're constantly upgrading our API technologies and educating developers about offerings we already have that they may not know about, including the Search API, Data Mining Feed, and the upcoming "firehose" of all public tweets. We prioritize this work based on what's going to have the most benefit for the broadest reach of applications.

The fact that 100% Twitter-powered companies like StockTwits are getting funding and expanding in popularity suggests that the Twitter API is meeting the needs of successful, growing businesses today.

At first we bought that explanation, but the more we thought about it the more doubt crept into our minds. That statement reads like a PR agent had a heavy hand in writing it.

Is Payne really saying that big companies will be fine, data that the company selectively exposes (tweets vs. profile data) will remain available and it's only a handful of developers that will be impacted? What if in that handful of developers there are people who are working on building a game changing service that we as users will be really excited about? We love Twitter but our loyalties lie with the developer community that builds on top of its service first.

Maybe we're wrong, though, and Payne's response is reasonable. Maybe the handful of developers who want to ping Twitter to death are just loser freaks with too much time on their hands, not enough business development skills and nowhere near the style of Twitter's new friends in Hollywood and on Madison Avenue! Really, though, maybe Twitter isn't putting future innovation at risk in order to take the easy way out on scalability challenges. What do you think?

Comment from John Sutton on 2009-01-21

2009-01-21T21:53:27Z

Odd, I got a Qwitter message this evening - obviously not completely expired yet.

Comment from Four20 on 2009-01-21

2009-01-21T21:57:09Z

uh oh, Marshall Kirkpatrick is tackling those who do us wrong on the internets. I smell another apology coming :p

Comment from Nik Smit on 2009-01-21

2009-01-21T22:12:37Z

There is a solution to this : gnipcentral.com (we use them for Retaggr's feed).

Right now its not a 100% solution, but once the new firehouse mentioned in the article is done, it will be. Includes facilities for heavy users to pay.

Comment from chris on 2009-01-21

2009-01-21T22:15:05Z

I wonder what this means for services like gnip. We're currently using gnip for our app, but seeing as we're not including followers, etc..., we might be in the clear. It is a wonder, though, why Twitter hasn't approached their issue from the api. Maybe it's harder than it sounds.

Comment from Jesse Stay on 2009-01-21

2009-01-21T22:18:26Z

Excellent post, Marshall. John - just wait. I don't think Qwitter has enough users yet, and the limit doesn't go into place until this weekend anyway. When they grow large enough, they'll hit the exact same limit we're running into. True, we could always expand to multiple servers, but what's the difference if we're handling it just fine on one IP? So now we're expected to game and hack the system to achieve what Twitter wants? This doesn't put us out of business entirely - we just throttle our requests, which means fewer users can have access to the service. In the end it's the users that get hurt by this, not the developers. It's unfortunate.

Comment from Jesse Stay on 2009-01-21

2009-01-21T22:19:37Z

Nik, re: Gnip, I don't think Gnip supports follower/friend info, does it? Of course I may be wrong and I'd love to know if they do.

Comment from Steve S. on 2009-01-21

2009-01-21T22:28:06Z

And this limit introduces a possible revenue stream: Developers that want access to more requests/hr may be able to pay for it in the future.

Comment from Gavin on 2009-01-21

2009-01-21T22:32:42Z

@JesseStay if you are complaining about breaking up your twitter activities onto 2 different machines, I don't see you're argument. Either your scared cause you don't know how to parallelize your app or you think it's too expensive which its not, so not sure what your beef is.

Comment from Mihai Secasiu on 2009-01-21

2009-01-21T22:34:52Z

@Jesse Stay you don't need multiple servers, you can assign multiple ips on a single server.

Comment from Alex Williams on 2009-01-21

2009-01-21T22:52:55Z

Is Qwitter back? Could its zombie rebirth be at all possibly related to the tweak in the Twitter API? I just put up a post on the topic. Other folks are seeing Qwitter messages, too.

Comment from Allison Reynolds on 2009-01-21

2009-01-21T23:04:40Z

Fascinating. I always had leasing access to the Twitter API as one of the ways Twitter could monetise without selling advertising.

Maybe short sighted of them to knock back the opportunity of making a few bucks to help them out.

Comment from Steffan Antonas on 2009-01-21

2009-01-21T23:11:56Z

Of course Twitter is putting future innovation at risk in order to take on scalability challenges. Any time you put constraints on developers and code you limit what is possible. The question is what's the risk-to-reward ratio for Twitter re their developer community. The important thing is that Twitter isn't limiting access to data, rather they're limiting pings. Twitter is betting that forcing developers to be creative with how they display/maniplulate/use data is going to eliminate the hogs and make the scalability situation better for the community - users and developers alike. If they've got to trim some fat by forcing developers to be creative so that everyone can be better off, so be it. We're all sick of seeing the fail whale.

Comment from Steffan Antonas on 2009-01-21

2009-01-21T23:15:10Z

That said, what's Twitter's argument for not letting hogs pay for what they consume over 20K/hr? Seems like a good idea - if the price was right.

Comment from Paul Campbell on 2009-01-21

2009-01-21T23:17:15Z

Marshall,

Paul here. I revived Qwitter during some downtime last October It was originally a side project between my two business partners in Contrast (http://www.contrast.ie)

Qwitter died in early December. The original code that we used could barely handle 100 users. I spent the weekend (just 2 days) working on code that handled several thousand. By December, user numbers had far exceeded that.

It was frustrating to led it slide, but company needs must, and though we were lucky to get hosting sponsored ( by http://www.hosting365.ie ).

Last week I figured out a technique that would help us scale the service up to many thousands of users, so I put that in place over the weekend, and wrapped up development on it, ironically enough, this afternoon. We're sending about a 1000 Qwitter emails per hour.

A 20,000 request per hour limit definitely won't take us out of the game, and Qwitter should be good, at least technically, for the foreseeable future!

Comment from Rashmi Ranjan Padhy on 2009-01-21

2009-01-21T23:32:22Z

I am still not sure how this is going to impact my application.

It was only a day before I got the whitelisting for my upcoming application. I am yet to launch it to public so I can not measure on the right amount required.

Twitter would need the apps to grow and in return twitter would also grow faster. I seriously think twitter is just starting and it will explode in the coming days, but with limiting the apps, it might create a barrier for twitter's growth.

Hopefully, they will take decision before we hit the wall and it starts impacting our application.

Comment from Eric Marcoullier on 2009-01-21

2009-01-21T23:43:13Z

Twitter continually has a tightrope to walk -- serve the developers vs serve the consumer directly. Given unlimited resources and an infinite timeline they would obviously satisfy both, but as is they are constantly in triage.

We do our share of bitching when things don't work for us -- their XMPP firehose frequently has more than 90 minutes of lag which impacts every single one of our users and we'd like nothing more than for them to drop everything to fix this problem. But it's important to recognize that it's rarely a good idea to cater to the needs of the edge cases at the expense of the general users.

Just my $0.02

@chris -- Gnip is fine as Twitter's sending us a dedicated firehose of public tweets.

@JesseStay -- Yeah, we're not a solution for you in the short term. We're going to be rolling out distributed polling moving forward, but it will be a while before we release *authenticated* distributed polling.

Comment from Thorsten Claus on 2009-01-21

2009-01-22T00:30:12Z

Let's take this as a great opportunity and push by Twitter for more cloud applications: how about having the next Qwitter client running on my desktop machine or mobile handset (hello Android), helping Qwitter to circumvent the limit of 20k requests? Or rent cloud computing power across the globe to have an ever changing IP address. Hello Amazon, after offering a *fixed* IP address for a fee, how about offering an *hourly changing* IP address for a fee? I see lots of opportunities to really push cloud computing to new frontiers...

Comment from Todd on 2009-01-21

2009-01-22T01:09:47Z

The Quitter mascot haz a sad.

*tear*

Comment from Don Draper on 2009-01-21

2009-01-22T01:25:58Z

Perhaps the Twitter crew has figured out what their burn rate is and is now concerned that perhaps their resources aren't unlimited?

Developers that can't get the data they need from the API are likely to just spider the pages themselves rather than going through the API.

Comment from DC Crowley on 2009-01-21

2009-01-22T06:37:25Z

Twitter is switching back into 3rd gear imo instead of going full out. They pretend it will affect few. But it is blocking it's own potential. Other companies like SocialToo and Seesmic may suffer from these choices. But as I mentioned in my post if media like the BBC, CNN, etc are catching on to twitter and they are, this will limit their eventual use. For live TV programs twitter is an amazing back channel.

Comment from william on 2009-01-21

2009-01-22T07:38:07Z

The Elephant in the room

If you are an Open Source Developer or Content creator (If you are an active member of twitter you are a content creator) you should not use Twitter period.

By using Twitter you are essentially raising the value of an ungrateful closed source company and supporting the illusion that their commodity application and their team have more worth and power than the community of developers and content creators/members.

Twitter is a closed source content silos that does not allow you to control the content that you create.
Twitter does not put the content developers or content creator/members at the top of their pyramids when thinking about revenue models, they put twitter first.

Twitter is and will continue to cripple competitive innovation by changing and throttling their api. It is the natural process for a company that is built on "proprietary code."

If you are a developer you may be able to make some money by creating applications for Twitter ; but I do not believe that Twitter will ever allow your application to eat into their user base or their revenue. Because Twitter is a closed source company that have the ability to literally cut you off by changing the code/api or by using their proprietary knowledge to build an application that you can not possibly compete with. As a developer understand that when you build and extend Twitters propitiatory platforms that you undermine the longevity of the Open Internet.

Content owners and Developers do not help closed source companies in their goal of creating another closed source content trap that will extract hundreds of Millions of dollars from their member and developer communities and give nothing back in return.

Comment from Joel Hughes on 2009-01-22

2009-01-22T10:47:48Z

20,000 requests does seem very low, but then again the majority of apps would struggle to fill this many. For example http://www.twitter.com/worldometer only updates 48 times a day at most, and a popular one like http://secrettweet.com/ is probably still in the low thousands (if that).

The desktop apps would be on a per-install basis surly as each has it's own authenticated session? Or am I missing the point?

Comment from Dawn on 2009-01-22

2009-01-22T15:26:19Z

What alarms me about this is that it shows a willingness to change the rules in the middle of the game. If applications make money while Twitter is still dry, are they going to pull the rug out from under them?

Comment from Den on 2009-01-22

2009-01-22T18:34:57Z

As seems to me 20K requests per hour is even more than the past limit. We've developed Twitter 411 service based on the much more stronger restrictions. See http://t411.linkstore.ru

Comment from Paul Neto on 2009-01-23

2009-01-23T18:20:12Z

not sure we are to assume that someone unfollows due to a single particular tweet.