The new Seesmic Desktop - which went through a series of public beta tests - is a major update from the old version. As the company's founder and CEO Loic Le Meur told us earlier today, Twitter is still a major focus of the app - with native support for Twitter's streaming API, for example, and lots of Twitter-focused plugins in its library. You could also just use the app as a Facebook, LinkedIn, MySpace, Ning or Formspring client, however, and never even touch Twitter at all.

Seesmic Marketplace

At launch, the new Seesmic Marketplace will feature over 40 plugins. Over the last few months, the company's developer partners created plugins for a wide range of service, including ecommerce site Zappos, real-time search engines OneRiot and Topsy, as well as a Google Reader plugin and support for YouTube, Last.fm and Seesmic's own Ping.fm.

For now, all the plugins in the marketplace are available for free. Starting next year, Le Meur told us, Seesmic will also offer an e-commerce solution and allow developers to charge for their plugins.

Using Seesmic Desktop

We got a chance to test the final version Seesmic Desktop 2 ahead of today's launch. While the first beta versions had a number of quirks, this final version feels very polished and fast. Adding plugins is as easy as browsing the marketplace from your browser and hitting "install."

The only minor annoyance with installing plugins is that you have to restart the application before the plugins can be used. It is also a shame that the Twitter plugin does not show conversations, though it does offer virtually every other Twitter feature you would expect to find in a modern desktop client.

The layout of the app, with side tabs for 'searches,' 'userlists' and 'accounts' still harkens back to Seesmic Desktop's Twitter legacy. Once you install a lot of plugins, this static and Twitter-centric list of tabs quickly feels limiting. This won't be a problem for long, though, Le Meur told us, as the team is already working on a more flexible tab layout. Thanks to this, you will soon be able to keep all your news-related Twitter lists and plugins in one tab and all your personal Twitter lists and your Facebook stream in another, for example.

Overall, these are just minor issues. After using Seesmic for a while, the application's potential to become a major desktop hub for real-time services quickly becomes clear and hopefully, more developers will soon offer plugins for their services as well.

All of these scenarios require users to have a participatory role in publishing and sharing. This works well for some apps that require active engagement such as broadcasting your Twitter status and where you'll be later. But an entirely new class of compelling scenarios really shine when you use a technology called geofencing to leverage background location and push location updates up to public and private clouds.

Place Geofences

Place-based geofences use a stream of continuously pushed location to identify when a user enters or exits a place or static zone. A place may be the local bar or even the state of California. Place and zone data sources are infinite - from restaurants and other POIs to boundaries such as a county, city, zip code, neighborhood, your home, school, or workplace.

If you need more ideas for zone data types, check out UrbanMapping's geodata catalog, which lists dozens of geodata source types. With places loaded into a geofencing service, you can build Smartphone apps to trigger when a user enters or leaves the static area that you care about. Your app could send an email, text or multimedia alert to the user ("Billy has arrived safely at school!") or trigger an action such as updating a database.

Imagine a geofence that automatically checks you in to one of your favorite venues, without having to even interact with the Foursquare mobile client. Or when you leave, it checks you out so your friends aren't wondering where you are.

Dynamic Geofences

Anyone reading this post has heard of, or probably even used, a mobile app that delivers real-time and right-time information to your smartphone app: weather alerts, Amber Alerts, traffic alerts, emergency notifications. With dynamic-feed proximity-based geofencing and geofencing APIs, it's possible to deliver these dynamic data sources just-in-time as mobile users move within geographic proximity of continuously shifting, moving-target data streams.

The interesting nuance here is that the geofence location is relatively static (a traffic accident doesn't move much), but the geofence time window is extremely dynamic (accident locations appear and disappear). This gets very interesting when you incorporate rapidly moving real-time data, like Twitter feeds. You could imagine real-time location-aware celebrity sighting notifications where I would receive an alert that Lionel Messi (the best soccer player in the world, who lead my beloved Argentina in the World Cup) was spotted and then geo-tagged and tweeted about around the corner from me.

Also, think about all the "expiring assets" - movie tickets just before showtime, pizza slices post-lunch rush - these could be fed to nearby subscribers, offering discounts. Or maybe an exclusive Gilt sale that lasts a couple of hours.

Peer-to-Peer Geofences

With peer-to-peer geofencing, both the geofence location and the time window are regularly shifting - and the goal is to detect "collision" (nearness) of two very dynamic points as, for example, when we want to detect that two friends are near each other. This is a more advanced feature of a geofencing services aPI because it involves a real time comparison of many streams of location data both for location and time collision (it's much less interesting to detect that your friend was here a few days ago).

What's Taking So Long?

The easy part is coming up with these use cases. But Android, Blackberry and WinMo have had background process capability for some time - why haven't we seen an explosion of these kinds of apps on those platforms?

It turns out that managing background process - especially using location and geospatial calculations - can be very complex. Naïve implementations can be a massive drain on battery life of a phone (leaving the GPS chip on would drain the battery in a few hours). Geospatial calculations can be complex, as is rapid computation of massive amounts of location data.

Producing a scalable, commercial-grade solution is a difficult problem. iPhone attempted to solve some of these issues (like battery drain), the solution is far from ideal. For instance, in our early testing, the iPhone background location API was not reliable out-of-the-box for geofencing. We have a long way to go. But the opportunity is huge and obvious, so I expect we will see much innovation in this area in the coming months as the industry moves into Location 2.0.

Photo by Yaroslav B.

As Envolve's marketing materials point out, "websites are lonely places." While we can go to Twitter and Facebook to discuss a site, there is often very little real-time interaction between users on the site itself (except, of course, when site owners use a real-time commenting system like JS-Kit's Echo).

What makes Envolve so interesting, is that it allows you to create numerous topic chat rooms on your site. This, for example, allows you to create one room to answer customer service questions and another that allows users to discuss a specific product or a new feature you just launched. By default, all users remain anonymous, though they can opt to use their real name or even create an Envolve account. The service also features private chats and the ability to block abusive users. Admins can also permanently ban any user.

Pricing

Installing the service is as easy as copying and pasting a few lines of code into your website's HTML template. The company also offers a plugin for Wordpress sites. For small sites, Envolve offers a free plan that supports up to 5 simultaneous users. Paid plans start at $9 per month for up to 10 connected visitors and go up to $99 per month with support for up to 200 concurrent visitors. All of the paid and free accounts also feature real-time visitor statistics and the ability to customize the color of the chat rooms.

The site lets you track everything from daily spill positions to the locations of ships responding to the crisis. State and non-governmental organizations are also collecting and mapping real-time information. In some instances the efforts include citizen-generated data from iPhone apps and photos mapped on sites like Flickr.

LA Earth (Louisiana Earth) is a Google Earth Enterprise Server operated by the state of Louisiana that provides daily oil spill trajectories, closures, and many other maps as a Google Earth layer (kml). (Download the Google Earth Client/Plugin to use the date.)

Crisis and Crowd-sourced Mapping

There are several ways that that Gulf locals are mapping and reporting spill related incidents. The Oil Spill Crisis map was created buy the Louisiana Bucket Brigade and students at Tulane University. The map shows hundreds of reports sent through SMS messages, Twitter and the Internet. The map was built using open-source crisis mapping software developed by Ushahidi.

The sensors built into smartphones are collecting a rich stream of spill-related data for mapping. The iPhone app Oil Spill Response (iTunes download) allows you to file mobile reports on wildlife, oiled shorelines and other types of spill-related damage. Oil Reporter was created by Crisis Commons and is available for both iPhones and Android smart-phones. It geotags photos, files reports, and provides information on how to contact authorities or volunteer.

Beyond the information collected by the government and oil industry, people are documenting their own stories. Communities are forming on photo sharing sites like Flickr where location-tagged images can be visualized using Flickr Map.

The Real-Time Web is a set of technologies that impacts almost every service, activity and application on the Web. Come to the summit to understand how it impacts you, your business and your next development.

Why We Use the Unconference Format

Watch live video from ReadWriteWeb Mobile Summit 2010 on Justin.tv

"[The Mobile Summit] was the best best best event I've been to in years, and I mean it. To be frank, I almost left ten minutes after I arrived, thinking that I was completely out of place. Turns out I was exactly in the right place all along. I left feeling completely energized about media, innovation, women in tech, new social models, and have so many new ideas about what I want to do next and what I want to get involved in. What I really didn't expect was to walk away with a pretty clear picture of what the future of mobile media (and social media) looks like. Now I'm staring at invites to keynote at a couple of tech events this year and hoping they're a fraction of the win that was RWWMS." - Violet Blue, sex educator and author

An unconference is a free-flowing, organically generated series of group discussions agreed on and led by attendees.

To get a feel for how thoughtful discussion happens at these events, check out the comments by Ted Morgan, Chris Saad and Ben Metcalfe in this video by Evelyn Rusli from TechCrunch:

We were one of the first news outlets to analyze the Real-Time Web and we've since written extensively about it. We also published a premium report on the subject last year, which featured interviews with 50 companies, developers and executives building or leveraging real-time Web technology.

If you're a company in the Real-Time Web market, you may be interested in helping to sponsor this event. Please contact our COO Sean Ammirati for more information.

The ReadWriteWeb team is excited about our first New York event on June 11th, so we look forward to seeing you there!

Every tweet is a drop, every blog post is a drop, every check-in is a drop that falls into the ocean. This ocean is almost constantly under a tropical storm in some places, like Twitter or Facebook.

When you're a search engine, you obviously have an exhaustivity requirement. You can't really skip on indexing the Indian Ocean. Google sends its bo(a)ts all over the ocean where it's raining to update its index. However, the ocean is growing so fast that it will eventually become harder and harder to stay exhaustive.

Unfortunately, not only the ocean is growing, but it's also raining more, which means that if a bo(a)t is away from a zone for too long, when it will be back it will have changed tremendously. That's what happens when you see results in a search engine that are 1- or 2-years old, or even older. They're not wrong, they're just often inaccurate, but rank well.

It's a real technical problem for search engines to know where to send their bo(a)ts, and at the right time! And when Google says they're going to feed their search index with PubSubHubbub data, that's what they're trying to do: save a little bit on the boats.

I strongly disagree with John Battelle when he says this is not a huge deal. My take is that he sees this only as a great technical and infrastructure opportunity for Google, not so much as an immediate benefit for the end user. I strongly disagree - and so do you. You disagreed when you typed "earthquake" into Twitter Search, or even "hudson crash", or "Mickael Jackson". At that point, you knew that Google wasn't able to provide you with the information you were looking for, and this is a massive loss for Google.

Google will have a hard time getting this brain share back. The first thing it needs to do is to actually have results that date back from the minute when people look for these things.

You may argue that if you search 10 times a day on Google, you go maybe once a week to Twitter search. I'm the same, no worries. Yet, I know that Twitter is much better than Google at contextualization. When I do a search on Google, I expect to find the absolute truth. If I look for earthquake, I'm looking at facts about earthquakes: pictures or maybe historical data. If I look for earthquake on Twitter, I'm looking for context; I want what is being said about earthquakes now (and here!).

As a matter of facts, Google always had a lot of issues about context because they know so little about the people who search there (or maybe they know a lot, but don't want to scare us). Adding PubSubHubbub is a way for them to be able to take the "time dimension" back. They many never have the conversations that Twitter has, but they will have a much bigger ocean of data than Twitter's sea of Tweets

Photo by Pam Roth.

Google already indexes a whole lot of content very quickly, will a real-time indexing system make a big difference? There are differences of opinion on the matter and we'd like to know what you think. Search analyst Danny Sullivan told us on Wednesday that he thought it could be "the next chapter" for Google. John Battelle said this morning: "In short, it's a new way for Google to get (more) real time signals. But honestly, not a huge deal. I don't think. Correct me if I'm wrong..." What do you think, readers?

As we wrote on Wednesday:

PuSH is much more computationally efficient for Google but [Google's Brett] Slatkin says that even more important is the impact of such a move for small publishers. Right now many small sites get visited by Google maybe once a week. With a PuSH system in place, they would be able to get their content to Google automatically right away.

A richer, faster, more efficient internet would be good for everyone, but the benefits in search wouldn't be limited to Google, either. The PubSubHubbub is an open protocol and the feeds would be as visible to Yahoo and Bing as they would be to Google.

Readers Who Think This is Big

Sharon Kavanagh says:

This all sounds fantastic for the small guy as I have just created my first ever website which is for a reunion. The site will only be live for a short period as the date is May15th 2010 for the event and yet, it will probably take Google till then before my site is indexed and hence the peple I am trying to reach will never find it.

Scott Holodak says:

Previously you had to wait for spiders to crawl around the web to find changes on your site. Pages are crawled over again and again just to see if anything has changed. It's a pretty inefficient process. Now the spiders are going to be fat and lazy because you are going to deliver your changes directly to them.

No Big Deal

Reader comments arguing this is not a big deal.

"Scott" says:

A properly designed website already "pushes" to (more accurately: gets "pulled" by) search engines and the frequency of indexing by search engines is determined by the popularity of the website.

This information doesn't seem too new to me.

Bruce Wayne says:

Pushing unstructured content in real time can only mean the non relevant results will make it into the search results faster. To me this is another google hocus pocus distraction away from the the fact that search as it is today has hit a wall....millions of pages on unstructured data created exclusively to game the system....and now these pages of non relevant content can be pushed into the search stream in real time....

What Do You Think?

I think there is something fundamentally different about a web that Google's index subscribes to in real time vs. a web that Google has to plow through with a spider looking for new content. I'm still wrapping my head around it, but there's something about the PuSH method that feels like it would make the Google index a living, breathing phenomenon.

What do you think?

What this means, essentially, is that you no longer need to wait for your news reader to ping your blog every so often to find out if there are any updates - you'll find out in real time.

Just like the adoption of RSSCloud last fall, there is no need to opt-in or install a plugin for a blog hosted on Wordpress.com to become PuSH enabled - it's already active. For Wordpress blogs hosted separately, a PuSH plugin, PuSHPress is now available for download.

This is yet another big step in our progression to a real-time Web. Last month, Google Reader went real time by consuming PuSH feeds, meaning they show up on the news site almost immediately after being published to the originating site. In conjunction, this means that any Wordpress.com hosted blogs, as well as any PuSH enabled blogs running Wordpress, will be immediately available on Google Reader and any other reader set to work with PuSH.

This also means that, if you want to be on the razors edge of what's happening on the Web, you can also receive chat notifications of PuSH enabled blogs. RSS readers can be so last year when you can get a chat notification the instant a piece of content is published.

For a further explanation of PubSubHubbub, read Marshall Kirkpatrick's article from last year's Real Time Web Summit.

What Ellerdale is now doing with Twitter's 50 million tweets per day is definitely interesting - the service uses an intelligent data-parsing engine to analyze the context of tweets and the links they contain and combines that with other data sources like RSS feeds and Wikipedia to create a real-time search engine and trends tracker that provides more than just a list of tweets - it provides an understanding of the world's conversations.

You can click through on any of the topics or sub-topics to learn more about what's being discussed. Although Ellerdale's best feature is its ability to highlight these sorts of trends, you can also use it to search the real-time web for your own keywords. Here, unlike Twitter's own search engine, Ellerdale won't just return a simple list of tweets in response to a query.

Instead, any topical page on Ellerdale returns an incredible amount of data. There are summaries provided from sources like Wikipedia, Freebase (an online semantic database), New York Times' people search and more. Related topics, in the form of thumbnail images, are listed above the live-updating message stream on every topic's main page. To the right, a graph charts that keyword's popularity over time and you can manipulate this to show you data from the past hour, day, week or month. Also to the right is a list of top articles from around the web, ranked by how many times they've been mentioned on Twitter. That article list can even be subscribed to via an included RSS feed.

And let's not forget the main dish - the live-updating stream of tweets. The message stream shows who tweeted what, when and what Twitter client they used to do so, which is the same information you would see on Twitter.com. However, where Twitter's own homepage and search results pages stay put until you refresh them, this message stream moves in real-time as tweets come in. If it goes too fast for you (something that's a real possibility when you watch a currently hot trend), you can pause the stream with a click of a button.

For data hounds, search results like these are tantalizing to say the least. And this engine is now just one of many that has access to Twitter's entire stream of tweets. The other new Twitter partners are also search and discovery services, including Collecta, Kosmix, Scoopler, twazzup, CrowdEye, and Chainn Search, all of which parse the Twitter data feed in their own way. Is any one better than another? That's hard to say. Each has their own niche, site design and unique features which allow them to appeal to select groups of searchers. Ellerdale is interesting because of its semantic capabilities, but it's not the only one to offer those. Kosmix, for example, has been developing their semantic-based news portal for the last three years.

The best part about all these new partnerships is that we're about to see an entirely new way to search the web emerge. For quick real-time results, there will always be the major search engines and their more basic lists of tweets, but for true data analysis, we now have incredible new options like Ellerdale and all the others.

Take real-time, for instance. The next generation of search, aggregation, notification and findability services are being developed using real-time technologies that enable users and machines to receive real-time updates. In a recent post, Robert Scoble said he would be better off curating news than actually attending the Apple launch! What? If you aren't thinking about how real time, along with social networks, mobile and location-based services fits in your marketing plan, you're missing an opportunity.

In a recent post, ReadWriteWeb's Marshall Kirkpatrick asked "Why is Google afraid of Facebook?" The answer is because social networking sites have become a key link in the search and information sharing value chain. You would have to be hiding out in a dark hole not to understand social media and the effect it has had on marketing the past couple of years - but surpass search? Oh, right, now I get it: These sites are an important information source for everyone. Importantly, friends' recommendations are key.

Mobile is Better

Google's VP of product development recently stated that, "with all the capabilities these phones that are coming out have - like GPS, cameras - we think there is the potential to actually make this mobile Web better than the PC Web." That is a profound statement for marketing managers. A mobile phone experience better than the web? If you haven't bought yourself a smart phone like iPhone or Android, we suggest you go out and grab one. Mobile applications are proliferating like rabbits. What would be better than to be first to market and offer your customers an exceptional product experience while on the go.

Perfect product placement

Location-based services mean the ability to market right outside your front door is happening now. ReadWriteWeb's Frederic Lardinois reported in June 2009 that 1 in 3 smart phone owners use location based services. Take this simple example. You're in Vail, you just finished 8 hours on the mountain and now you're looking for the perfect apres ski location. You're walking down the Mall, you take your iPhone out of your pocket and ta-da! Buy one-get-one-free margaritas at Las Margaritas. You're standing right outside. Perfect product placement. And now you can talk about the restaurant and broadcast it immediately to all your friends.

If you aren't listening to the conversation, you better start. There are numerous listening applications available to get you started in your pursuit to join the conversation and get a handle on positive as well as negative feedback on your product or brand. A simple saved search in Twitter can go along way.

All these trends have a profound impact on how we market to our website guests at ReadWriteWeb. Not only do we have to understand search engine optimization, but the opportunities offered by social media marketing, the new capabilities and possibilities offered by mobile, geolocation, augmented reality and real-time notification and information sharing. One seems to becoming just as important as the next.

If you don't understand these technology trends as a marketer, you better get out while the getting is good. Enabled by technology, 2010 is already a watershed year for new ways to reach your customers.

Want to know more about the real-time Web? Read ReadWriteWeb's report, The Real-Time Web and its Future.

Photo by Clix.

Real-time search appears to have been rolled out very recently, with no mention, on this page. In addition to search results continuously updated ala Facebook's newsfeed ("3 new results") there's also a frequently-updated list of "trending topics" on the search page.

Proper nouns will likely be of interest to searchers watching YouTube comments. This could be a popular addition to the toolkits of social media watchers everywhere.

What's the benefit of serving those results up in real time? For certain search queries you don't want to wait around to find out there's new results.

What could be next? Presuming this feature is as real as it looks and goes live to the public soon, we'd love to see YouTube support something like the Salmon comment aggregation protocol and publish updates for this and other GData feeds through in a real-time syndication format.

Thanks to Tikva Morowati for the tip. Tikva is the Community Platform Director for KGBWeb, a stealth startup made up of ex-Googlers and others in New York City that will likely make a splash among web-watchers later this year.

Flash-based P2P Video

For TinyChat, the ability to route these videos around its servers means reduced bandwidth costs. Once this feature becomes part of the default TinyChat setup, only calls with more than 2 participants will have to go through the company's servers and as TinyChat's founder Dan Blake told us earlier today, the experience of switching between the P2P chat and the server-based version should be completely seamless.

Unlike other P2P solutions, TinyChat is able to leverage a plugin that virtually all users already have on their machines and users don't have to download another plugin. All a user needs is a webcam and a microphone. Currently, when you are chatting on the P2P server, your privacy is also protected, as the service simply won't allow a third user to listen in or join the call.

Many of the recent real-time search engines are based on Twitter. They use the URLs enclosed in tweets to discover and rank new and popular pages. In this post, we'll take a look at the quantitative structure of the underlying foundation, to determine the feasibility and limits of this approach. We'll also look at how to overcome these limitations by using the implicit Web.

But it also shows another fact: that those people who contribute publish an even smaller fraction of the information they know.

Both of these factors account for the huge difference in efficiency between implicit and explicit voting. Explicit voting, as the name implies, requires users to actively express interest in a page; for example, by tweeting a link. Implicit voting requires no deliberate action on the part of the user; a simple visit to a Web page would count as a vote.

A Quick Calculation

Twitter now has 44.5 million users and delivers about 20,000 tweets per minute. If every second tweet contained a URL, that would be 10,000 URLs shared per minute.

According to Nielsen, the number of visited Web pages per person per month is 1,591.

Twitter's 44.5 million users visit 1.6 million Web pages per minute and explicitly vote for only 10,000 per minute. That is to say, implicit voting and discovery generates 160 times more attention-getting data than explicit voting.

This means that 280,000 implicit votes could provide as much information as 44.5 million explicit votes. Put another way, as many Web pages are implicitly discovered during one day as there are Web pages explicitly discovered during half a year.

This dramatically shows the limits of Web searches based solely on explicit votes and mentions, searches whose potential could be leveraged by using the implicit Web.

Beyond the Mainstream

This becomes even more important if we look beyond mainstream topics and the English language. Then it becomes simply impossible to achieve the critical mass of explicit votes needed to have statistically significant attention-based ranking or popularity-based discovery.

Time and Votes Are Precious

Time is also a crucial factor, especially with real-time search. We want to be able to discover new pages as soon as possible. And we want to assess almost instantly how popular those new pages are. If we fail to reliably rank a page quickly, it will get buried in the noise. But the goals of speed and votes conflict with the fact that the number of votes a page gets is inversely proportional to the time it took to be viewed.

Again a much higher frequency of implicit votes would help.

Relevance vs. Equality

We could also improve on explicit votes. But we should not treat them as being equal because they are not. We trust some of them more than others, and our interests overlap with some more than others, for the very same reason that we follow some people and not others. This helps us get more value and meaning out of that very first vote.

FAROO is moving in this direction by combining real-time search with a peer-to-peer infrastructure.

A Holistic Approach

Discovering topical, fresh, and novel information has always been an important aspect of search. But the perception of what "recent" is has changed dramatically with the popularity of services such as Twitter, and it has led to the emergence of real-time search engines.

Real-time search shouldn't be a silo, but rather should be part of a unified and distributed approach to Web search.

The era of purely document-centered search is over. The equally important roles of user and conversation, both as targets of search and as contributors to discovery and ranking, should be reflected in the infrastructure.

A Distributed Infrastructure

As long as both source and recipient of information are distributed, then the natural design of search is distributed, too. P2P offers an efficient alternative to the ubiquitous concentration and centralization of search we find today.

A peer-to-peer client allows every visited Web page to be implicitly discovered and ranked according to attention received. This is important, because the majority of pages in a real-time search are in the long tail. They appear once or not at all in the Twitter stream and can't be discovered or ranked through explicit votes.

With real-time search, the amount of indexed data is limited, because only recent documents (those that have gained a lot of attention and a high reputation) are accounted for in the index. This allows for a centralized infrastructure at a moderate cost. But as soon as search moves beyond the short head of real-time search and aims to fully index the long tail of the entire Web, then a distributed peer-to-peer architecture provides a huge cost advantage.

• First generation: was done on a single processor, usually for process control in military systems.

• Second generation: within a Local Area Network, usually for a financial trading room.

• Third generation: applied across the whole Web/Internet, what we call the real-time Web.

In each generation a stack has emerged, and secure messaging has been key to that stack. The names change and the scale of the prize and challenges certainly changes, but the basic issue remains the same: delivering messages reliably and quickly. In this post, we trace the steps from the second generation to the third generation to see how the real-time Web might play out.

Generation 2's Winner: The Teknekron Information Bus Story

I worked with the technology's second generation. My company took on the challenge of delivering market prices from multiple sources to hundreds of traders in a trading room, enabling new apps based on real-time calculation. The solution had to scale to maybe 2,000 traders, which is laughably small by the standards of the real-time Web. But you did not want to face a roomful of furious traders if your system was down for three minutes or delayed prices by 15 seconds. The reliability and latency requirements were very stringent.

The company that emerged as leader, Teknekron, coined the term "Information Bus." It did very well by coining the term, because it established Teknekron as a market leader; it followed the "get mindshare first, and then go for market share" strategy. Vivek Ranadive, the visionary founder and CEO, describes in his 1999 book "The Power Of Now" how the term came to him. He was schooled as a hardware engineer and was appalled by how much more chaotic (i.e. error- and delay-prone) software projects were compared to hardware projects. Looking at why, he realized that the concept of a "bus," which is the device that all hardware components connect to, could be applied to software.

Ranadive understood that real-time, event-driven systems would not be limited to financial trading, and he executed brilliantly on that insight. He sold Teknekron to Reuters, the company's main rival in trading systems, but retained the right to use the technology outside of financial markets. That offshoot became TIBCO (The Information Bus Company), which at the time of writing is a publicly traded company valued at $1.6 billion, with annual revenue of over $600 million.

But TIBCO did not leverage its position to dominate the third generation. Another company tried that and blew a lot of money in the attempt.

KnowNow: Lessons of a Blow-Out

KnowNow worked through $50 million in funding before throwing in the towel in July 2008. It was going to update the message bus to the Web using RSS. It was still an enterprise play.

It had a strong management team, top-tier VC (Kleiner Perkins), and strong technology. Why did it fail? Three big reasons stand out:

• Generation 3 is a tougher technical challenge. Generation 2's state of technology was perfect for local area networks using Ethernet, where the user device was always connected and where we measured users by the thousands. With generation 3, we are dealing with millions of users who are offline some of the time, and the network bandwidth available for messages is not under their control. In other words, it is one big bear of a technical challenge!
• Companies are reluctant to invest millions of dollars in closed software when open standards clearly always win on the Web. Companies can instead experiment very cheaply using consumer-centric RSS tools and open source.
• KnowHow was early. The hype on real-time enterprise did not last long enough, and it hit the trough of disillusionment.

Contenders for the Real-Time Web Message Bus

We have probably missed a few, so please tell us about them in the comments. These are very different types of solutions, but they are working towards a similar objective. First, we will list them and then attempt a bit of categorization:

• Gnip: an independent venture whose monetization model relies on it being a hub that different sites can connect to.
• Tornado (FriendFeed's Python-based Web server technology, which was open sourced by Facebook): this uses PubSubHubbub.
• RSS Cloud: promoted by Dave Winer, with WordPress as a marquee partner.
• PubSubHubbub: promoted by Google (and FriendFeed/Facebook), with SixApart as a marquee partner.
• XMPP The technology with which IM clients interoperate. Being used by Yammer, Present.ly, and Drop.io.
• Twitter.
• Facebook.
• TIB: an enterprise solution from TIBCO.
• MQ: an enterprise solution from IBM.
• Sonic: an enterprise solution from Progress Software.

Here are the categories they fall into:

1. Enterprise
   TIB, MQ, and Sonic all fall into this category, and there are more. They will find it hard to make the transition to the real-time Web for two reasons. The first is technical. Connecting millions of users over a low bandwidth network via HTTP is very different from connecting a few thousand users over a corporate network. The second is commercial: they monetize by selling licensing fees to enterprises. But that is not the primary way in which the real-time Web will be monetized. Still, these are big profitable companies with the right tech chops, so they cannot be dismissed.
2. Open source and open standards
   Tornado, RSS Cloud, PubSubHubbub, and XMPP fall into this category. They have differing purposes but also a lot of overlap. XMPP is low level and not affiliated with any big company. RSS Cloud and PubSubHubbub are most similar to each other. Tornado is a Web server. But they all face the issue that an open standard takes a long time to evolve and consolidate into a winner. Before that happens, we will likely see more entrants, making the choice even more confusing for developers. So, developers may hedge their bets and go for commercial/de facto standards.
3. Commercial hub
   Gnip is the purest example of this. It aims to get traction by being simpler to implement than the open-source/open-standards offerings. It is a bold strategy from a strong team of entrepreneurs and investors. For something as big as this - basic plumbing for the Web - it feels like a bit too much reliance on one company.
4. Traffic plays
   Twitter and Facebook are the major players here. They might become de facto standards because the biggest issue is traffic. Developers will build where there is traffic as long as two conditions are met:
   • It works technically (i.e. latency at scale). Twitter has some credibility issues here.
   • It is open and transparent - i.e. anyone can connect without fear of the game changing on them. Facebook has some credibility issues here.

The Technical Challenges

First, we have to address the question of "What does real time mean?"

In each generation, the gurus of the previous generation get sniffy about the definition of real time ("What do you mean one second is real time? We measure in micro-seconds!").

It depends on the usage case and technical possibilities. Real time within a single processor is obviously faster than real time over a LAN, which is faster than real time over HTTP for the whole Web.

Real time, practically speaking, means "orders of magnitude faster than how data has been delivered in the past, and faster than most people think they need today." So, RSS Cloud is faster than RSS, and Twitter search is faster than Google search, and IM is faster than email.

In generation 2, the thinking evolved through three approaches:

1. Polling. It soon became obvious that this would not scale.
2. Broadcast, which put too much load on the client.
3. Multicast, which became PubSub.

Most current Web HTTP-based real-time systems are still in the polling phase (i.e. subscribers poll publishers to see if they have an update), which clearly won't scale. The newer approaches fundamentally enable a publisher to say, "I will tell you, the subscriber, when I have an update, so you don't need to poll me." It's a geek's way of saying "Don't call us. We'll call you."

For a good technical primer on how this works and a discussion of the differences between RSS Cloud and PubSubHubbub (mercifully shortened to PuSH), check out this post.

Sticking My Neck Out to Guess the Winner

Twitter. Why? It has the traffic and is open and transparent enough to win confidence. But two big problems remain:

1. It has to reveal its business model. Only then will partners feel confident that they understand its strategic direction.
2. It has to prove it can get good latency at scale. Its early history of fail whales makes people justifiably skeptical.

The winner will likely be the platform that hosts the killer app. Most platforms get traction through a killer app. In the second generation of real time, that killer app was market data for financial traders. What will it be in the third generation? We will explore this in a future post.

What to Name This Generation?

"Message/Information Bus" worked for generation 2. It took a key concept from generation 1 - the hardware bus - and applied it to a local area network.

That does not work so well for generation 3, the real-time Web. For now, we talk about a Web-wide real-time message bus because there is no better alternative.

The new term may have something to do with "status," because the status update is a central concept in social media. Perhaps something like "StatusFabric" or "StatusNet" or "StatusWeb" will emerge.

We love to debunk myths and prick bubbles as much as the next set of pundits, but we think the real-time Web is for real. Financial traders have lived in a real-time world for a while, but only within the confines of the trading floor. When they left work, they entered a batch world. Most other people work in a batch world. That is changing. We are all entering the real-time world of the trader. Some of us are getting there faster, but we are all heading there. And relax, there is an "Off" button!

One reason you see so much news about Twitter is that it is a news business, and journalists love to write about their own business!

Calling Twitter a news business may be a bit limiting. It is more than that. But it certainly is affecting the news business dramatically:

• As a source of stories. One of the millions of just-in-time stringers twittering away may witness a revolution at any second;
• As a backchannel for other media, replacing the dial-in line used for TV and radio;
• As a traffic source;
• As a source to check stories. Traditional journalists can tap into their Twitter network faster than using their Rolodex or speed dial.

Twitter makes it much easier for journalists to tap into real-time news sources. Mastering Twitter is part of new journalism school.

Traders have always worked in a real-time news business. Information services such as Bloomberg and Reuters compete to deliver news a few seconds faster, and that time difference is important to traders.

Okay, so traders and journalists live in a real-time world. What about the rest of us?

Advertising Exchanges

A friend who runs a financial trading systems business told me that Google was constantly trying to poach his best engineers. Why? Because those employees were very good at real-time engineering, and advertising is also a real-time market.

Advertising markets are just like financial markets: available space is matched by available demand. Today, we have two totally different worlds:

1. Very personal: the "Let's do lunch" style of selling, with long-term commitment. This is a batch world.
2. Totally automated: no messy humans. Machines negotiate with machines in real time. This is the world of automated ad networks and exchanges, typically for sales of either remnant or long-tail inventory.

These worlds get more interesting as they move closer together and we see the gray areas in between where humans make quick judgment calls, inserting themselves into the real-time flow to, for example, approve creative or simply optimize (choosing one advertiser or publisher over another).

Buying and Selling Digital Goods

Anything that can be sold in digital form is becoming part of this trend towards real time. We are simply matching supply and demand. Information (or code or images or songs) that was not worth very much yesterday is suddenly very valuable. Or the opposite: its value suddenly drops. No matter what the digital artifact -- writing, spreadsheet numbers, code, design, images, music -- matching supply and demand is critical to realizing value.

You may be selling a digital artifact that costs money to create, but the marginal cost is zero, so you are quite happy to optimize supply and demand in real time and take whatever price the market offers at the time.

Or you may want real-time markets to help manage your spare work capacity. This is what ventures such as Turn Here focus on.

Real-Time Supply Networks for Physical Products

Dell revolutionized the PC industry by using a real-time supply chain to eliminate costs. Inventory risk is the biggest pain point for most businesses that trade physical goods. The real-time Web can have a similarly revolutionary impact on Main Street for businesses that no longer need multi-million dollar supply-chain systems. We are already seeing straws in the wind of this massive change with very simple uses of Titter, such as:

• The bakery that tells its patrons/followers that "The bread is hot and fresh."
• The Korean BBQ truck in LA that has 39,000 followers who want to know when it is going to roll into their neighborhood.

One can imagine this for anything that is either fresh or scarce: when your local farmer has freshly laid eggs, for example. These tools can be used to sell inventory, matching actual customer demand with bargain offers in real time.

Relaxed Concentration

Human beings naturally operate in real time. Anyone who has children knows about the "real-time interruption machine."

We also need our batch time, our quiet time for reflection and creation. We need to know when to switch off... literally switch off the devices that constantly ping us. Filtering tools and techniques have become critical. As Clay Shirky famously remarked: "It's not information overload. It's filter failure." But we also need to master the ability to deal with a lot of real-time information in a mode of relaxed concentration. In other words, we need to study how great traders work.

Photo credit: artemuestra.