ReadWriteWeb

Last fall, Twitter announced a partnership with Gnip, making the latter company the only commercial provider of the Twitter activity stream. And although the "firehose" metaphor has been beaten to death, says Gnip CEO Jud Valeski, it still holds true.

Valeski spoke today at Gluecon about the challenges of handling the firehose - what it means to process high volume, real-time data streams and to be able to do so "in a consistent and predictable manner."

Like a prism to a ray of sunlight, stream-hacking startup Mediasift CEO Nick Halstead took the stage today with Twitter's Ryan Sarver at the Data 2.0 conference to announce Twitter's second data resales channel partnership. Halstead's service will allow customers to parse the full Twitter fire hose along any of the 40 fields of data hidden inside every Tweet, with the addition of augmented data layers from services including Klout (influence metrics), PeerIndex (influence), Qwerly (linked social media accounts) and Lexalytics (text and sentiment analysis). Storage, post-processing and historical snapshots will also be available.

The price? Dirt cheap. Halstead told me after the announcement that customers would be able to apply as many as 10,000 keyword filters to the fire hose for as little as 30 cents an hour. The most computationally expensive filtering Mediasift will offer won't be priced above $8k per year. (Pricing approximate but indicative, Halstead says.) What does this mean? It means that far more developers than ever before will now have a stable, officially approved and very affordable way to access highly targeted slices of data. Twitter just found a way to hand developers an Amazon River's worth of golden tinker-toys, each with more than 40 points of contact, at commodity prices.

Last week at Strata, Gnip released a new set of features for its social-stream processing platform. Called Power Track, the new layer allows customers to set up complex search queries and receive a stream of all the Twitter messages that match the criteria. Unlike existing ways of filtering the firehose, there are no limits on how many keywords or results you can receive. However, the part of the offering with the most long-term significance is the pricing.

On top of the standard $2,000 a month to rent a Gnip collector, founder Jud Valeski told me it will cost 10 cents for every thousand Twitter messages delivered. Though the split of the revenue between the two companies wasn't disclosed, he told me Twitter intends to standardize this price for any similar offerings in the future from other sellers of their data. This sounds like a big step in Twitter's journey to find a sustainable business model.

Twitter announced today a new partnership with social data streaming service Gnip at the Defrag Conference outside of Denver: Gnip will offer 50% of all the messages posted to Twitter for $360,000 per year, or 5% of all messages for $60,000 per year. Pricing is not yet on the Gnip site, but was disclosed in an interview with ReadWriteWeb.

Customers will only be allowed to analyze the messages, not display them, and resale of the content itself will remain prohibited. The two companies emphasized that this is the first time a structured, reliable arrangement has been available for the many customers interested in purchasing a large quantity of streaming Tweets. Sale of the 100% full firehose will remain in the hands of Twitter itself. The full firehose contains approximately 1,000 Tweets every second.

Today's announcement at Defrag that Gnip will be the only commercial providers of Twitter's activity stream raises a lot of questions, so I sat down with Chris Hogue, Rob Johnson and Jud Valeski from the Gnip team to get some answers from a technical perspective.

The first thing I wanted to know was the nuts-and-bolts of accessing the stream using Gnip. One fundamental advantage that Gnip offers is that they have redundant firehose trunk streams coming into independent Amazon Availability Zones, which they're then able to syndicate across the internal network to every machine. External developers get access to this by renting a customized Gnip machine in this cluster, and writing a receiving application that listens to the streaming HTTP connection it's given from this master stream. That sounds a lot like the current way you access the firehose, but there are some key distinctions.

When so many conferences feature CEOs rehashing their past successes, FailCon does exactly the opposite. The event asks successful founders, investors and developers to discuss their past blunders and what they've learned from them. While this may seem like a series of sob stories, the result is actually a list of practical tips on how to reduce risk, manage teams and recover from adversity. In today's afternoon sessions several panelists shared their war stories and set the stage for lessons.

Real-time computing is not new. This is the third generation of real-time:

• First generation: was done on a single processor, usually for process control in military systems.

• Second generation: within a Local Area Network, usually for a financial trading room.

• Third generation: applied across the whole Web/Internet, what we call the real-time Web.

In each generation a stack has emerged, and secure messaging has been key to that stack. The names change and the scale of the prize and challenges certainly changes, but the basic issue remains the same: delivering messages reliably and quickly. In this post, we trace the steps from the second generation to the third generation to see how the real-time Web might play out.

Full feeds of data are exciting, but sometimes you need a little something special.

Gnip, the Boulder, Colorado startup aiming to act as a clearinghouse for user activity updates from around the web, announced a partnership today with Canadian firm PostRank, to offer additional versions of Gnip-delivered data feeds, filtered by popularity. Gnip could already deliver anyone a big bucket of user data like photos from Flickr, submissions from Digg or slide shows from SlideShare - but now this partnership will allow customers the option of receiving only those items that were most commented on, linked-to, tweeted about, etc. It's wonky, but it's a whole lot of fun.

Can being "present in the now" be packaged and sold as a service? A number of companies believe that it can be and are aiming to offer a "real-time" layer of functionality to consumer websites and businesses interested in this growing trend online.

On one hand it's just a speed up the infrastructure play, but the impact of real time information delivery on a user's experience of a website can be profound. The latest entrant into this market of white label real time service layers is called Notify.me.

RSS and syndication are the veins that the new social web flows through. Countless products and services have been built on top of RSS in the past few years but there are always a few that stand above the rest.

As part of this year's Top 10 Products series, we offer below the Top 10 RSS and Syndication Products of 2008. These are the feed tools we and the people we know use day in and day out - we love them, we hate them, we wouldn't want to work without them.