The price? Dirt cheap. Halstead told me after the announcement that customers would be able to apply as many as 10,000 keyword filters to the fire hose for as little as 30 cents an hour. The most computationally expensive filtering Mediasift will offer won't be priced above $8k per year. (Pricing approximate but indicative, Halstead says.) What does this mean? It means that far more developers than ever before will now have a stable, officially approved and very affordable way to access highly targeted slices of data. Twitter just found a way to hand developers an Amazon River's worth of golden tinker-toys, each with more than 40 points of contact, at commodity prices.

You can make a filtering rule, make it public or private, share and comment on other peoples' rules and more.

While Twitter's partnership with bulk data reseller Gnip (announced in November) offered half the fire hose in bulk for a whopping $360k or 5% of the fire hose for $60k per year - Mediasift prices and use cases will be very different. Pricing will be modeled like Amazon Cloud Computing and each function's cost will be spelled out as the user requests it. Geo-filtering is expensive, keyword filtering is cheap - for example. Keyword filtering is done, stable and available now. Storage of the data for post-processing and snapshots of historical data are described as in alpha stages.

Below: This is what a Tweet looks like. Every little message has more than 40 different fields. Mediasift customers will be able to filter the full Twitter fire hose by any of those fields or by data from additional 3rd party services.

"Twitter is moving up the value chain by offering the high-level information that developers want," said ReadWriteWeb contributor and leading social data hacker Pete Warden about the announcement. "Rather than selling commodity information for further processing, this partnership offers a narrow but deep slice."

This is a bet on a future wherein greater value is built by widespread, low-cost access to social data on the part of many and diverse developers. It's not just raw data either, it's really rich. It's the opposite of what the Gnip announcement signalled and what many people have feared - that Twitter would horde its river of data and sell it just to high bidders.

More on this in the coming days. I'm very excited to start hacking on it all with ReadWriteWeb's team of developers.

Customers will only be allowed to analyze the messages, not display them, and resale of the content itself will remain prohibited. The two companies emphasized that this is the first time a structured, reliable arrangement has been available for the many customers interested in purchasing a large quantity of streaming Tweets. Sale of the 100% full firehose will remain in the hands of Twitter itself. The full firehose contains approximately 1,000 Tweets every second.

Gnip CEO Jud Valeski (right) says that there has long been a black market for this kind of data. "The grey area around Twitter data access allowed that black market to grow," he says. Won't someone fill the demand for the new level of data at 10% Gnip's price on the black market? "Someone will fill that demand," Valeski said, "but Gnip is guaranteed to deliver 50% indefinitely and legitimately."

Developers using Twitter's "Spritzer" feed, which delivers a random selection of 2% of all Tweets, will continue to be able to do so for free. The larger "Garden Hose" will no longer be made available and Twitter's Ryan Sarver says the company hasn't decided yet how to transition that informal program.

This could be just the beginning of something very big. "I wonder if the glory days of open data sharing are coming to an end," says Steve Rubel, SVP, Director of Insights for Edelman Digital.

"All these platforms, pick any one of them, they all realize that the data they have is extremely valuable to everyone from API partners to marketers to in some cases even ordinary consumers. I think all these companies could see that there's more money in data services than there could be for them in advertising. No one is going to have more data than the platforms themselves - for Twitter that could be a bigger money maker than advertising."

That's all well and good for Twitter's bottom line, but will it be a net positive for the larger developer community and users of their software? Or is this a case of Twitter and Gnip selling a potential public good at a price out of range of the people most likely to create radical new innovations out of it? If instead this monetization of Twitter data makes everyday developer access more financially viable for the long term, then perhaps it will be a good thing.

• First generation: was done on a single processor, usually for process control in military systems.

• Second generation: within a Local Area Network, usually for a financial trading room.

• Third generation: applied across the whole Web/Internet, what we call the real-time Web.

In each generation a stack has emerged, and secure messaging has been key to that stack. The names change and the scale of the prize and challenges certainly changes, but the basic issue remains the same: delivering messages reliably and quickly. In this post, we trace the steps from the second generation to the third generation to see how the real-time Web might play out.

Generation 2's Winner: The Teknekron Information Bus Story

I worked with the technology's second generation. My company took on the challenge of delivering market prices from multiple sources to hundreds of traders in a trading room, enabling new apps based on real-time calculation. The solution had to scale to maybe 2,000 traders, which is laughably small by the standards of the real-time Web. But you did not want to face a roomful of furious traders if your system was down for three minutes or delayed prices by 15 seconds. The reliability and latency requirements were very stringent.

The company that emerged as leader, Teknekron, coined the term "Information Bus." It did very well by coining the term, because it established Teknekron as a market leader; it followed the "get mindshare first, and then go for market share" strategy. Vivek Ranadive, the visionary founder and CEO, describes in his 1999 book "The Power Of Now" how the term came to him. He was schooled as a hardware engineer and was appalled by how much more chaotic (i.e. error- and delay-prone) software projects were compared to hardware projects. Looking at why, he realized that the concept of a "bus," which is the device that all hardware components connect to, could be applied to software.

Ranadive understood that real-time, event-driven systems would not be limited to financial trading, and he executed brilliantly on that insight. He sold Teknekron to Reuters, the company's main rival in trading systems, but retained the right to use the technology outside of financial markets. That offshoot became TIBCO (The Information Bus Company), which at the time of writing is a publicly traded company valued at $1.6 billion, with annual revenue of over $600 million.

But TIBCO did not leverage its position to dominate the third generation. Another company tried that and blew a lot of money in the attempt.

KnowNow: Lessons of a Blow-Out

KnowNow worked through $50 million in funding before throwing in the towel in July 2008. It was going to update the message bus to the Web using RSS. It was still an enterprise play.

It had a strong management team, top-tier VC (Kleiner Perkins), and strong technology. Why did it fail? Three big reasons stand out:

• Generation 3 is a tougher technical challenge. Generation 2's state of technology was perfect for local area networks using Ethernet, where the user device was always connected and where we measured users by the thousands. With generation 3, we are dealing with millions of users who are offline some of the time, and the network bandwidth available for messages is not under their control. In other words, it is one big bear of a technical challenge!
• Companies are reluctant to invest millions of dollars in closed software when open standards clearly always win on the Web. Companies can instead experiment very cheaply using consumer-centric RSS tools and open source.
• KnowHow was early. The hype on real-time enterprise did not last long enough, and it hit the trough of disillusionment.

Contenders for the Real-Time Web Message Bus

We have probably missed a few, so please tell us about them in the comments. These are very different types of solutions, but they are working towards a similar objective. First, we will list them and then attempt a bit of categorization:

• Gnip: an independent venture whose monetization model relies on it being a hub that different sites can connect to.
• Tornado (FriendFeed's Python-based Web server technology, which was open sourced by Facebook): this uses PubSubHubbub.
• RSS Cloud: promoted by Dave Winer, with WordPress as a marquee partner.
• PubSubHubbub: promoted by Google (and FriendFeed/Facebook), with SixApart as a marquee partner.
• XMPP The technology with which IM clients interoperate. Being used by Yammer, Present.ly, and Drop.io.
• Twitter.
• Facebook.
• TIB: an enterprise solution from TIBCO.
• MQ: an enterprise solution from IBM.
• Sonic: an enterprise solution from Progress Software.

Here are the categories they fall into:

1. Enterprise
   TIB, MQ, and Sonic all fall into this category, and there are more. They will find it hard to make the transition to the real-time Web for two reasons. The first is technical. Connecting millions of users over a low bandwidth network via HTTP is very different from connecting a few thousand users over a corporate network. The second is commercial: they monetize by selling licensing fees to enterprises. But that is not the primary way in which the real-time Web will be monetized. Still, these are big profitable companies with the right tech chops, so they cannot be dismissed.
2. Open source and open standards
   Tornado, RSS Cloud, PubSubHubbub, and XMPP fall into this category. They have differing purposes but also a lot of overlap. XMPP is low level and not affiliated with any big company. RSS Cloud and PubSubHubbub are most similar to each other. Tornado is a Web server. But they all face the issue that an open standard takes a long time to evolve and consolidate into a winner. Before that happens, we will likely see more entrants, making the choice even more confusing for developers. So, developers may hedge their bets and go for commercial/de facto standards.
3. Commercial hub
   Gnip is the purest example of this. It aims to get traction by being simpler to implement than the open-source/open-standards offerings. It is a bold strategy from a strong team of entrepreneurs and investors. For something as big as this - basic plumbing for the Web - it feels like a bit too much reliance on one company.
4. Traffic plays
   Twitter and Facebook are the major players here. They might become de facto standards because the biggest issue is traffic. Developers will build where there is traffic as long as two conditions are met:
   • It works technically (i.e. latency at scale). Twitter has some credibility issues here.
   • It is open and transparent - i.e. anyone can connect without fear of the game changing on them. Facebook has some credibility issues here.

The Technical Challenges

First, we have to address the question of "What does real time mean?"

In each generation, the gurus of the previous generation get sniffy about the definition of real time ("What do you mean one second is real time? We measure in micro-seconds!").

It depends on the usage case and technical possibilities. Real time within a single processor is obviously faster than real time over a LAN, which is faster than real time over HTTP for the whole Web.

Real time, practically speaking, means "orders of magnitude faster than how data has been delivered in the past, and faster than most people think they need today." So, RSS Cloud is faster than RSS, and Twitter search is faster than Google search, and IM is faster than email.

In generation 2, the thinking evolved through three approaches:

1. Polling. It soon became obvious that this would not scale.
2. Broadcast, which put too much load on the client.
3. Multicast, which became PubSub.

Most current Web HTTP-based real-time systems are still in the polling phase (i.e. subscribers poll publishers to see if they have an update), which clearly won't scale. The newer approaches fundamentally enable a publisher to say, "I will tell you, the subscriber, when I have an update, so you don't need to poll me." It's a geek's way of saying "Don't call us. We'll call you."

For a good technical primer on how this works and a discussion of the differences between RSS Cloud and PubSubHubbub (mercifully shortened to PuSH), check out this post.

Sticking My Neck Out to Guess the Winner

Twitter. Why? It has the traffic and is open and transparent enough to win confidence. But two big problems remain:

1. It has to reveal its business model. Only then will partners feel confident that they understand its strategic direction.
2. It has to prove it can get good latency at scale. Its early history of fail whales makes people justifiably skeptical.

The winner will likely be the platform that hosts the killer app. Most platforms get traction through a killer app. In the second generation of real time, that killer app was market data for financial traders. What will it be in the third generation? We will explore this in a future post.

What to Name This Generation?

"Message/Information Bus" worked for generation 2. It took a key concept from generation 1 - the hardware bus - and applied it to a local area network.

That does not work so well for generation 3, the real-time Web. For now, we talk about a Web-wide real-time message bus because there is no better alternative.

The new term may have something to do with "status," because the status update is a central concept in social media. Perhaps something like "StatusFabric" or "StatusNet" or "StatusWeb" will emerge.

Gnip, the Boulder, Colorado startup aiming to act as a clearinghouse for user activity updates from around the web, announced a partnership today with Canadian firm PostRank, to offer additional versions of Gnip-delivered data feeds, filtered by popularity. Gnip could already deliver anyone a big bucket of user data like photos from Flickr, submissions from Digg or slide shows from SlideShare - but now this partnership will allow customers the option of receiving only those items that were most commented on, linked-to, tweeted about, etc. It's wonky, but it's a whole lot of fun.

We use Postrank around the ReadWriteWeb team often: to make sure we don't miss big stories on niche blogs, to display the most recent break-out hits on other blogs we write about and to power the Community Management Aggregator that automatically delivers the hottest posts from community management experts to customers of our Guide to Online Community Management.

On one hand it's just a speed up the infrastructure play, but the impact of real time information delivery on a user's experience of a website can be profound. The latest entrant into this market of white label real time service layers is called Notify.me.

Making websites real time is the hottest trend online this season. From Facebook to Google, Twitter, Digg and countless little innovative startups, it seems like everyone is either doing it or talking about it. (See our Introduction to the Real Time Web for background.) Might some sites choose to use an outside service that specializes in real time infrastructure, instead of building their own in-house? That's what Notify.me is betting on.

Notify.me is a San Diego based startup made of tech industry veterans, some working on the company on the side, others full time. The company has taken no funding, has no revenue and no one working there is being paid. The executive team is made up of engineers from companies like MP3.com, the Health Care division of SAIC, Napster, DekiWiki and Yahoo. It's a pretty hot crew to get together with no pay to take a long shot at productizing a technology like real time.

This isn't just another fly-by-night "instant alerts" service, though casual observers may have thought as much over the several months that the free consumer version of Notify.me has been available. (We count ourselves among those casual observers, in fact!)

Using the free consumer service, anyone can set up alerts to be delivered by IM, SMS, email or to an Adobe AIR app whenever an RSS feed updates. That's nice and hopefully Notify.me's service will work better than the alternatives do these days, but RSS to IM/SMS alerts are nothing new.

Where it gets really exciting is in the two APIs the company is working on.

What's New: The APIs

A REST API is available for publishers now and C# and Perl libraries should be available in two weeks. That API allows publishers to define particular events on their site and then offer real time alerts to readers when those events occur. You might want real time notification when someone leaves a comment in reply to yours, or when a site publishes news concerning a particular topic, or when a new event listing is published so you can buy tickets right away. The possibilities are endless and fun to imagine.

The second API in the works is an Actionscript and XMPP Client API that allows developers to build interfaces for audiences to consume real time alerts through. That API has specs in draft form now but the company says it expects little change to occur before a final release.

What does that mean? It means you could add real time notification consumption to apps on the web, desktop or iPhone (using the new Push Notification Service in the next iPhone OS release).

Put those two APIs together and you've got publishing and reading apps going in real time. Hello real time web!

Can They Sell Real Time to Businesses?

The Notify.me team has immediate designs on business customers. Talks have begun with companies in the medical, financial and software fields. Doctors could use real time updates to track patient updates, including allergies and drug conflicts as they are discovered, prior to prescribing medication. Medical practices could push lab results to physicians instead of waiting on a chart pull request.

Would medical software companies use a hosted 3rd party API as real time infrastructure? Notify.me says they have consulted with HIPAA experts who believe that as long as the company transmits notification of an event and not personal medical information, they should be legally compliant.

Figuring out rules for determining what kinds of information gets delivered will be one challenge that Notify.me will have to tackle with customers. As Sameer Patel, another entrepreneur in this market, points out: "What's absolutely necessary in the B2B space though is smart aggregation before push comes into play. Failing this, its going to be a fire hose that will quickly alienate the end user."

Gnip is another service offering similar kinds of functionality, but for different markets. Gnip head Eric Marcoullier had this to say about Notify.me's B2B prospects:

"Good for them. Further validation that slinging realtime data around has value. I bet they'll find good money there. We've even considered some of those use cases in the past, but have shied away because of the liability that's associated. The Gnip team works really hard to make sure that the platform is always running (with 99 point nine something uptime since launch) but if data gets held up for an hour, nobody's life depends on it. I'm psyched someone else is diving into the mission-critical data delivery while we work on business-critical data."

Indeed, reliable scalability will be Notify.me's biggest challenge. That's something the company has focused on since the start. Proving their case and building a name for themselves as a popular consumer notification service is a business strategy that quite a few other Web 2.0 type startups have done well with.

Can that strategy work with real time notifications, though? We suspect that business customers may be more interested in integrating real time functionality than all but a few power user consumers will be, so if you like the consumer service of Notify.me you'd better use it now, before the more viable business market takes precedence in the company's day to day decision making to the detriment of free accounts.

In the meantime, we expect that someone will succeed in bringing a real time service layer to the websites we use and work with every day. Real time is just too compelling for the paradigm to go back into the genie's bottle. It could be Notify.me that finds that success.

As part of this year's Top 10 Products series, we offer below the Top 10 RSS and Syndication Products of 2008. These are the feed tools we and the people we know use day in and day out - we love them, we hate them, we wouldn't want to work without them.

1. Top 10 Semantic Web Products of 2008
2. Top 10 International Products of 2008
3. Top 10 Consumer Web Apps of 2008

About the Selections

These aren't all new products from 2008. They are the products in the RSS and syndication world that we think made the biggest impact or were the most useful.

To be honest, this was not a particularly good year for innovation in the RSS space. Too many of the products listed below are incumbents, several of which drove us crazy this year. They remain on the list, however, because they are incredibly useful and nothing topped them.

Some honorable mentions are deserved as well. We talked to many people who like RSS magazine-style start page Feedly, though we found it overly constrictive and don't feel that it's made a big market splash yet. We also found the Associated Press's AP Member Marketplace very interesting. Had we gotten a chance to get to know it better, it could very well have been on this list. Finally, we love African social media aggregator Afrigator - it's a great way to learn about what's happening all over the continent and it's a great use of RSS. We named it one of the Top 10 International Products of 2008 but we think it deserves an honorable mention in this category as well.

And Now the RWW Top 10 RSS and Syndication Products of 2008

Postrank

Formerly known as AideRSS, Postrank is simply the most useful RSS related application we've seen in a long time. Plug in any RSS feed and Postrank will rate each item in the feed on a scale of 1 to 10, by number of comments, inbound links, saves in Delicious, etc. You can then subscribe to a filtered feed of just the 10% most popular items in that feed.

We use Postrank all the time, in all kinds of contexts: from monitoring break-out stories in niche markets we don't follow closely, to finding out about the bread and butter of new blogs we discover to running search feeds through Postrank to surface hot conversations on any topic.

Postrank has been around for about a year and a half, but we write about it over and over again.

This year Postrank opened an API, made a bunch of deals with other companies, improved its service, raised a round of funding and just generally rocked.

FriendFeed

Social "life streaming" service FriendFeed is making syndication a more social activity than anything else has yet. The service aggregates your activity data from all around the web, lets your friends comment on it and shows you the activities of all your friends' friends when someone you know comments on something and exposes it to their network.

If RSS readers will change your life and work through their awesome usefulness, FriendFeed is a service that makes syndication fun. It's one of the first places we go on the web every morning.

We interviewed the ex-Googlers who founded FriendFeed last February and that interview is still the best place to learn how the service works under the hood.

If you'd like to connect with the ReadWriteWeb crew on FriendFeed (and we hope you will) we've posted a tour of our FriendFeed profile pages here. Please join us also in the ReadWriteWeb FriendFeed Room.

Gnip

Gnip is a social media ping server, a service that other services ask for user data updates from all around the web. There's nothing here for users, but almost every developer we talk to these days who is aggregating content in order to add value to it (and that is the name of the game) has Gnip on its radar. The company aims to make aggregation more timely, scalable and efficient than it is today.

We wrote about Gnip at length when the service launched in July.

Snackr

Snackr is a simple little RSS ticker built in Adobe AIR. Its frenetic and unstopping delivery of news is too much for many people, but the rest of us love it. It's where our eyes wander during page loads and other down times. Many of the stories you read here at ReadWriteWeb were based on things we first caught wind of through Snackr.

Snackr was built in-house at Adobe by Flex team member Narciso Jaramillo. We reviewed it in May and have been using it ever since.

Google Reader

Google Reader is the market leader in full featured RSS readers, having pulled ahead of the troubled Bloglines in recent months. This year Google Reader has made their sharing feature much more transparent, added the ability to translate any feed into a number of different languages and recently redesigned.

It hasn't been a super exciting year for the product, and there are still basic problems like very infrequent caching of rare feeds, but Google Reader's incredible dominance in the field makes it a required part of this list.

Google Reader RSS Subscriber Count Greasemonkey Script

One of the simplest little changes we've made to our browsers lately is the addition of this greasemonkey script that shows the number of readers in Google Reader that any page's RSS feed has. You can usually multiply that number by 2 to 4 times for an estimate of how many total readers a feed has across all readers, but either way it's a great little indication of a site's popularity.

The script was written by an anonymous user named "uncv" and we'd like to thank them. We love what they've done! This was one of the 7 coolest browser tweaks from the last month that we wrote about earlier this week. It's already won a permanent place in our hearts!

Dapper

Dapper.net is a point and click interface for data extraction - a nice way to say scraping an RSS feed. We continue to depend on Dapper for all kinds of research, we're always finding new ways to use it around here. We love it.

Unfortunately, some sites don't like us to have access to links back to them available in our RSS readers (like Facebook, for example) and that really upsets us. In many cases those feeds that we created ourselves are the only way we'd be drawn back to a site, so it's their loss as much as ours.

Dapper has been around since 2006, but they recently launched a semantic ad platform that we included in our list of the top 10 semantic web products of 2008.

Twitterfeed

Love it or hate it, Twitterfeed has made a big impact on the web in 2008. It's the service people use to publish an RSS feed right into Twitter.

Some people argue that twitter is all about conversation and that publishing an RSS feed there is grating and inappropriate. We like getting our local newspaper story links on Twitter, though, and everything from disaster monitoring to traffic conditions are now available via Twitterfeed.

Feedburner

Google's RSS publishing service Feedburner hurt our ability to break news first, can't be used in many corporate environments because it gets blocked in China and only made 6 posts all year to its company blog, none since May. That's compared to 28 posts in 2007. Apparently once you get your Google money there's not much point in communicating with the people who depend on you every day.

Why would we call Feedburner one of the top 10 RSS products on the year then? Because despite how frustrating it can be, the service is still so incredibly useful that we don't know what we'd do without it. Not just for publishing and analytics for ReadWriteWeb feeds - from numbers to email delivery to FeedFlare links, Feedburner will work magic easily on any feed you work with. I've got 68 different feeds in my account and I'll probably publish several more before the year is up.

Pipes

Yahoo! Pipes is another RSS based service that is really frustrating, hasn't innovated substantially in the last year - but is still so powerfully useful that it deserves a spot as one of the top products in this market.

Splicing and filtering RSS feeds is the simplest thing to do with Pipes, but there's much more you can do with it as well. It's great for us pseudo-geeks, we can work all kinds of magic with it. We've used Pipes throughout the year to do things that we (ok I) don't have the technical chops to do otherwise. For that I thank the Pipes team a whole lot.

Those Were Our Favorites This Year - How About You?

Did we miss anyone you think should have been on this list? We hope you'll share your favorites in comments below. What RSS and syndication products impacted you the most in 2008?

]]> Discuss]]> http://www.readwriteweb.com/archives/top_10_rsssyndication_products_of_2008.php http://www.readwriteweb.com/archives/top_10_rsssyndication_products_of_2008.php 2008 in Review Thu, 11 Dec 2008 15:30:30 -0800 Marshall Kirkpatrick There's no more riding through the transit station on roller blades - the rental shops aren't keeping them in good enough repair. That could be an analogy for a decision announced today by Gnip, a startup aiming to become the ultimate ping server for social media.

XMPP/Jabber, the Open Source real-time communication protocol popularized by Instant Messaging that many have hoped would serve as foundation for a real-time web of the future, has become too much trouble to support and will no longer be a supported protocol at Gnip. More than just one protocol, it's a story of long tail developer communities and the ambitious startups forced to make resource decisions.

]]> Gnip founder Eric Marcoullier told us that the company really wants to support XMPP but that it's taking up an inordinate amount of support time, in many cases just because data consumers are using Google Talk or Jabber.org servers and are being throttled.

Marcoullier says there are simply no open source Jabber servers that are capable of the kind of robustness that a social media ping server requires. If a major vendor came to Gnip and said they wanted data streamed to them exclusively in XMPP, the company would continue the practice, but the long tail of tiny consumers that want their data that way is taking up too much resources. The company's top priorities are data delivery and maximizing the number of publishers participating in their program.

Our top priority as industry watchers is to cheer for radical innovation online, something that is in some cases more likely to come from the marginal developers making up the Long Tail that Gnip is herein limiting its support for.

Does This Matter?

This move may be less significant than it seems, though, too. The real-time communication experience is still available via basic Restful push, Marcoullier says, but for now XMPP in particular will be turned off this week.

When Gnip launched, we thought that XMPP in particular and protocol transformation in general were key to its value proposition. Months later, the company says that there is data normalization occurring but no protocol transformations are being performed.

Gnip now has a total of five developers, most recently adding enterprise data specialist Michael Barinek onto the team. The company has significant momentum even if today's announcement feels like it's losing some sexy sheen.

We hope that Gnip can continue to advance its goals of making the social web faster and more scalable. While we're disappointed that this hyped protocol is giving them too much trouble, we hope that a future web of real-time communication can come to us promptly none the less.

]]> Discuss]]> http://www.readwriteweb.com/archives/gnip_says_xmpp_ecosytem_too_ha.php http://www.readwriteweb.com/archives/gnip_says_xmpp_ecosytem_too_ha.php News Mon, 03 Nov 2008 14:39:03 -0800 Marshall Kirkpatrick RSS is the backbone for most things Web 2.0 but these days, it's not always fast enough. Politeness limits ping times to every 15 minutes at best in most cases, string a couple of applications together and information will sometimes not arrive where you're waiting for it for up to an hour.

A number of people are trying to speed up the feeds but today sees the first public mention of a new effort lead by the guys at popular lifestreaming service FriendFeed. FriendFeed is working on an open source add-on to RSS and Atom that will make it easier to discover when a feed has been updated. This could be a big deal.

]]> What it Is

The FriendFeed crew is working on something called a Simple Update Protocol. It was first reported on by venture publication The Deal and subsequently by Venture Beat, leading us to believe the PR push is an effort to for the company to raise some more money.

In response to our questions, FriendFeed co-founder Paul Buchheit told us the following:

• RWW: How will it work?

  Buchheit: SUP is just a very simple extension to RSS and Atom that makes it easier to discover when a feed has been updated.



• RWW: Where is this relative to XMPP?

  Buchheit: It's unrelated to XMPP.



• RWW: Gnip? (See our coverage of Gnip, a startup that appears to be aiming to do what SUP will do and more.)

  Buchheit: We're talking with several companies about supporting SUP, but aren't ready to announce anything.



• RWW: Open source?

  Buchheit: Yes, absolutely Open Source.

Not a whole lot of information is available about SUP, but we hope the above helps. We're real excited to see what FriendFeed has under its hat. The company has done more interesting things with popular use of RSS than anyone else has in awhile.

The Simple Update Protocol is due to be released next month. We look forward to checking it out. Soggy feeds put a damper on our day far too often. Update: FriendFeed has posted details about this on their blog.

Interested in FriendFeed? Take a tour of the RWW writers' activities on the site here. See also our months-old podcast interview with the founders of FriendFeed, still one of the best sources of in depth information available about this important service.

]]> Discuss]]> http://www.readwriteweb.com/archives/speed_up_rss_friendfeeds_going.php http://www.readwriteweb.com/archives/speed_up_rss_friendfeeds_going.php RSS & Feeds Wed, 27 Aug 2008 13:04:51 -0800 Marshall Kirkpatrick It's time to review the week that was on ReadWriteWeb. On the product side we looked at Adobe's announcement of searchable Flash, checked in with online TV service Hulu, reviewed a couple of innovative new web apps (Gnip and Identi.ca) and reviewed Firefox's recent world record. On the trends side, we analyzed Microsoft's acquisition of semantic search company Powerset, looked into the latest Yahoo stats, asked if email is in danger, and reported on a new Mobile Web standards initiative.

]]> Sponsored by:

Web Products

Adobe Makes Flash Searchable

For years the big problem with Flash-based websites is that they could not be properly indexed by search engines. Flash websites have been favored by marketers and advertisers for a long time, because of the ability to create rich, interactive Web experiences. However for most other businesses, particularly those with a lot of information on their website (let's face it, that's everyone except marketers and advertisers), Flash has been nearly an automatic 'no' for website development. That may be about to change.

Hulu To Earn Up to $90M In First Year....But It's Not A Success Story Yet!

To the average user, Hulu.com, the free web site that offers high-quality streams of TV shows and movies in the U.S., looks like a runaway success: the selection of available content is more than decent, Hulu's Collections make browsing related videos easy, HD videos have been made available, embed codes are provided for re-posting the videos on the web, and the site gets a good amount of traffic, too. In fact, Hulu's CEO reported in March that 5 million visitors watched videos on the site during the past 30 days while the service was still in beta, and that number has been increasing ever since.

Gnip: Grand Central Station for the Social Web

Ping, ping, ping! That's the sound made day and night by the new social media technologies rapidly proliferating around the web... and the machines are getting tired. Polling for updates to user data streams, wishing they spoke the same language and dreaming they knew which accounts belonged to the same people across different services. Sounds like a great opportunity for an infrastructure provider, doesn't it? Enter the sexiest infrastructure provider we've seen in a long time: Gnip. Venture funded and built by exited MyBlogLog co-founder Eric Marcoullier, Gnip wants to serve as the grand central station and universal translation service for the new social web.

Identi.ca: May A Million Twitters Bloom

Identi.ca is a new microblogging service that launched this week- but it's not just another also-ran. The service is an Open Source, CreativeCommons framework for a distributed network of federated microblogging services. If you've become interested in the paradigm changing model of communication popularized by Twitter but have been frustrated by Twitter's frequent down time or other shortcomings - then Identi.ca could be for you.

It's Official: Firefox Downloads Set Guinness World Record

We already knew that Mozilla had a record breaking day on June 17th when Firefox 3 was downloaded close to 8 million times, despite the download site not working for at least part of the morning. Now, Mozilla has announced that Firefox 3 has indeed made it into the Guinness Book of World Records with 8,002,530 downloads. Mozilla had set itself a goal of only 5 million downloads.

See also: Mozilla Releases Weave 0.2: Filling in for Browser Sync

SEE MORE WEB PRODUCTS COVERAGE IN OUR PRODUCTS CATEGORY

Web Trends

Does Microsoft + Powerset Beat Google?

What can the plan be with Microsoft's purchase of hot startup Powerset? The 3-year old company, founded by Dr Barney Pell, recently launched a semantic search experience for Wikipedia. It is doubtful that Microsoft bought the company just to enhance Live Search. Possibly the plan is to replicate the Wikipedia solution, then incorporate Powerset into Internet Explorer. In this post we look at what the thinking behind the acquisition might be.

See also: Microsoft Releases Interop Docs: Is This What Data Portability Looks Like?

Yahoo Would be Just Fine Without Search

Hitwise Intelligence took an interesting look at the breakdown of Yahoo's properties this week. They come to the conclusion that, even if Yahoo sells off its search division, Yahoo's other properties probably wouldn't be too affected by this, as they get most of their traffic from Google's search anyway. Only Yahoo Image Search, Games, Maps, and News get most of their traffic from Yahoo Search.

Is Email In Danger?

Human history is one of progressive improvement in communication. From the 20th century mail was a fundamental form of communication. The invention of electronic mail (email) changed two things. It became cheap to send mail, and delivery was instant. Email became favored for both corporate and personal communication. But email faces increasing competition. Chat, text messages, Twitter, social networks and even lifestreaming tools are chipping away at email usage. In this post we take a look at what's happening and assess if email is in danger.

Mobile Web To Get Standards

A group of mobile operators have just unveiled a new initiative they're calling "BONDI" whose goal is to encourage development of new mobile web applications while not compromising customers' security. BONDI was created by members of the OMTP (Open Mobile Terminal Platform), an industry group that includes participants from all parts of the mobile world and whose members include operators like AT&T, Hutchison 3G, Orange, Telecom Italia, Telefónica, Telenor, T-Mobile and Vodafone.

SEE MORE WEB TRENDS COVERAGE IN OUR TRENDS CATEGORY

That's a wrap for another week! Enjoy your weekend everyone.

]]> Discuss]]> http://www.readwriteweb.com/archives/weekly_wrapup_30_june-4_july_2008.php http://www.readwriteweb.com/archives/weekly_wrapup_30_june-4_july_2008.php Weekly Wrap-ups Sat, 05 Jul 2008 05:00:00 -0800 Richard MacManus Ping, ping, ping! That's the sound made day and night by the new social media technologies rapidly proliferating around the web... and the machines are getting tired. Polling for updates to user data streams, wishing they spoke the same language and dreaming they knew which accounts belonged to the same people across different services.

Sounds like a great opportunity for an infrastructure provider, doesn't it? Enter the sexiest infrastructure provider we've seen in a long time: Gnip. Venture funded and built by exited MyBlogLog co-founder Eric Marcoullier, Gnip wants to serve as the grand central station and universal translation service for the new social web.

]]> What Gnip Does Now

The primary service that Gnip offers at launch today is to capture user data updates from any web application and then serve up the very latest information to anyone else who requests it. Your application doesn't have to ping Flickr, YouTube, etc. etc. every few minutes and ask "have any of our users done anything on your individual service?" Now with Gnip, Flickr (a launch partner in fact) can report user data updates to Gnip, which can then pass that data along to consuming parties, along with data from all the other social media services of interest.

It's about scalability and decreasing latency to near zero. It sounds like a great idea.

What Gnip Says it Will Do Within 90 Days

The sexiest features are still in the works. Here are just a few plans that the company has disclosed so far.

Protocol switching You want to communicate instantly with an application using the IM protocol XMPP/Jabber but it only publishes and consumes RSS feeds? Gnip will stand in the middle and translate so each end of the transaction can send and receive data in the format it wants. Hot!

Standardized metadata Different services publish user data with different titles for the various fields sometimes. That makes it hard for the robots to know what exactly is being said. Gnip is working with visionary developer Chris Messina to create a meta data standardization process, for social bookmarking activities in particular. Gnip will consume feeds from user bookmarks on any service, then make publicly available those same feeds appended with another version of the same data but in a standardized format. That's a really big deal because it makes interoperability possible.

Identity discovery Right now it's hard for services and for users to tie multiple social media accounts together from around the web. Gnip will let users provide usernames or emails and then check to see where else those identifiers are being used. That's a solid sounding idea.

All of this will be free, none of it will be public-facing. Application developers will tie into Gnip and there may be premium services available eventually, like translation of data into a particular vendor's proprietary template.

Is This Too Centralized?

Gnip's Eric Marcoullier acknowledges that the centralization here is worth questioning. Two primary concerns come up: scalability and privacy.

As far as scalability is concerned, that's the name of the game here. If Gnip can't scale with fantastic uptime then there's no service. The company has been working with Pivotal Labs, the strike force hired to fix Twitter, since Gnip started work.

When we asked Marcoullier about privacy, he emphasized that Gnip is only working with publicly available data right now. The company might venture into user authentication and private data, but that's "a whole other can of worms," he said.

No doubt Google is already indexing most of the information that Gnip will be transmitting, but we can't help but think that Gnip will be in a uniquely powerful position to do some mining of anonymized user data and social graphs. That could lead to very big money but it could also raise some concerns on the parts of users. Gnip says they aren't worried about monetization right now, they just want to build out their service and add value to the market place.

That may be the case, but we think that Gnip has a whole lot of potential to deliver huge value to the applications leveraging it, to the backers financing it and ultimately to all the users of the emerging class of social web applications. We love this kind of stuff.

]]> Discuss]]> http://www.readwriteweb.com/archives/gnip_grand_central_station.php http://www.readwriteweb.com/archives/gnip_grand_central_station.php Data Portability Tue, 01 Jul 2008 09:02:05 -0800 Marshall Kirkpatrick