Many of the recent real-time search engines are based on Twitter. They use the URLs enclosed in tweets to discover and rank new and popular pages. In this post, we'll take a look at the quantitative structure of the underlying foundation, to determine the feasibility and limits of this approach. We'll also look at how to overcome these limitations by using the implicit Web.

But it also shows another fact: that those people who contribute publish an even smaller fraction of the information they know.

Both of these factors account for the huge difference in efficiency between implicit and explicit voting. Explicit voting, as the name implies, requires users to actively express interest in a page; for example, by tweeting a link. Implicit voting requires no deliberate action on the part of the user; a simple visit to a Web page would count as a vote.

A Quick Calculation

Twitter now has 44.5 million users and delivers about 20,000 tweets per minute. If every second tweet contained a URL, that would be 10,000 URLs shared per minute.

According to Nielsen, the number of visited Web pages per person per month is 1,591.

Twitter's 44.5 million users visit 1.6 million Web pages per minute and explicitly vote for only 10,000 per minute. That is to say, implicit voting and discovery generates 160 times more attention-getting data than explicit voting.

This means that 280,000 implicit votes could provide as much information as 44.5 million explicit votes. Put another way, as many Web pages are implicitly discovered during one day as there are Web pages explicitly discovered during half a year.

This dramatically shows the limits of Web searches based solely on explicit votes and mentions, searches whose potential could be leveraged by using the implicit Web.

Beyond the Mainstream

This becomes even more important if we look beyond mainstream topics and the English language. Then it becomes simply impossible to achieve the critical mass of explicit votes needed to have statistically significant attention-based ranking or popularity-based discovery.

Time and Votes Are Precious

Time is also a crucial factor, especially with real-time search. We want to be able to discover new pages as soon as possible. And we want to assess almost instantly how popular those new pages are. If we fail to reliably rank a page quickly, it will get buried in the noise. But the goals of speed and votes conflict with the fact that the number of votes a page gets is inversely proportional to the time it took to be viewed.

Again a much higher frequency of implicit votes would help.

Relevance vs. Equality

We could also improve on explicit votes. But we should not treat them as being equal because they are not. We trust some of them more than others, and our interests overlap with some more than others, for the very same reason that we follow some people and not others. This helps us get more value and meaning out of that very first vote.

FAROO is moving in this direction by combining real-time search with a peer-to-peer infrastructure.

A Holistic Approach

Discovering topical, fresh, and novel information has always been an important aspect of search. But the perception of what "recent" is has changed dramatically with the popularity of services such as Twitter, and it has led to the emergence of real-time search engines.

Real-time search shouldn't be a silo, but rather should be part of a unified and distributed approach to Web search.

The era of purely document-centered search is over. The equally important roles of user and conversation, both as targets of search and as contributors to discovery and ranking, should be reflected in the infrastructure.

A Distributed Infrastructure

As long as both source and recipient of information are distributed, then the natural design of search is distributed, too. P2P offers an efficient alternative to the ubiquitous concentration and centralization of search we find today.

A peer-to-peer client allows every visited Web page to be implicitly discovered and ranked according to attention received. This is important, because the majority of pages in a real-time search are in the long tail. They appear once or not at all in the Twitter stream and can't be discovered or ranked through explicit votes.

With real-time search, the amount of indexed data is limited, because only recent documents (those that have gained a lot of attention and a high reputation) are accounted for in the index. This allows for a centralized infrastructure at a moderate cost. But as soon as search moves beyond the short head of real-time search and aims to fully index the long tail of the entire Web, then a distributed peer-to-peer architecture provides a huge cost advantage.

Become a Fan

One feature that makes Tweetmi unique is that it gives users the option to become 'fans' of a certain topic. While this is definitely an interesting concept, users actually have to send out a tweet about the fact that they are now fans of 'RWW' or 'Follow Friday,' which somehwat limits the usefulness of this feature.

Another feature we liked is that the application can show you a list of all the Twitter users who tweeted a popular link. Like most Twitter search engines, Tweemi displays a list of the most popular links about a topic in a sidebar.

Given that there are already numerous Twitter search engines and more comprehensive real-time search engines like OneRiot on the market, Tweetmi will probably have a bit of a struggle to attract a dedicated user base. It is however, a perfectly capable Twitter search engine that offers all the typical features you would expect and definitely worth a try.

OneRiot's Tobias Peggs said the new capability is "a key feature. They are the first of these type of desktop applications to offer real time web search results in addition to the usual Twitter conversation search. This means Nambu users don't just keep track of conversations, but they also get to find related fresh content (news, blogs, videos, etc) as it emerges on the real-time web."

"OneRiot's real-time web search engine is the best way to find fresh, socially-relevant content from across the web," said Eric Woodward, CEO of Nambu. "Now our users can track realtime conversations on any topic, and discover new related content at the same time. It's a great combination."

In addition to OneRiot real-time web search, Nambu now offers Twitter and FriendFeed search to index conversations and links. The search offering is rounded out with Yahoo! News, which provides feeds from AP, Reuters, and other outlets.

"With the increase of noise and spam in real-time conversation search," said Peggs, "it's good for the users to also have real-time content search. Now you can track a term ... see what people are saying, and also discover emerging content. It's a nicely rounded experience for the user."

Here's an example of what those results would look like; OneRiot results are highlighted:

Clearly, adding search - and adding persistent searches in the sidebar especially - is a way for Twitter to get users to come back to their homepages regularly. For Twitter's most active users, the current homepage simply doesn't offer enough of the features they are used to from their desktop clients. However, if Twitter wants to monetize its service through advertising on its site (and the small ads that started to appear on the site in the last few weeks make us believe that this is the way Twitter is planning to go), it will have to give users a reason to come back to its site.

Twitter has also clearly recognized the real value of its service, besides offering a way for users to communicate, is in its real-time search.

As of now, Twitter is only testing this new homepage with a select number of users. The company will test the current iteration of the homepage with these users and then roll these features out to all users "as soon as possible" - assuming, of course, that the Twitter team doesn't decide to go back to the drawing board again.

RSS feeds, while extremely useful, can't really provide a real-time experience as your feed reader or other RSS enabled program has to actually ping the feed at regular intervals. Instead of getting information pushed out to, you have to actively pull the information in - and, for practical reasons, most feed readers like Google Reader only poll feeds a few times an hour.

Thanks to protocols like XMPP and SUP, however, it is becoming easier for developers to pass along updates to their users almost immediately.

Notifixious

Notifixious's Superfeeder wants to bridge the world of RSS feeds and the real-time web.

Notifixious's mission is to provide (almost) real-time updates when a blog or news organization posts a new story. You simply tell Notifixious which sites to monitor and whenever it finds an update, it will send out an alert to your IM account or mobile phone.

Currently, Notifixious can only get 'real' real-time updates through XMPP from Identica, Seesmic, LiveJournal, and Sixapart's TypePad and Vox blogging communities. Notifixious also monitors public ping servers, though these can be unreliable at times.

FriendFeed Real-Time

We are big fans of FriendFeed here at ReadWriteWeb (you can find - and join - our FF room here). One of the neatest features of FriendFeed is its ability to show you a real-time stream of your friend's updates. You can actually put those real-time updates into a browser sidebar if you use Firefox. Of course, this stream, like a lot of real-time applications, can often move faster than you can read if you follow a lot of FriendFeed users.

Monitter

The search function that Twitter acquired from Summize is probably the most commonly used real-time search engine on the Internet right now. Sadly, the results page doesn't automatically update as new updates come in.

A service that does just that is Monitter (our review). Monitter lets you simultaneously search for three keywords and it automatically updates whenever a new post with the keyword appears.

The Real-Time Web at Work: Google Docs, Zoho, MindMeister

Of course, the real-time web isn't just about consuming information - Google Docs, Zoho, MindMeister and many others allow you to collaborate and edit documents with your colleagues in real-time. As you make changes to the document, those changes will immediately be reflected on your fellow users' screens as well.

Starting Next Week: Your Facebook Homepage

You can't actually use this today, but earlier this week, Facebook announced that it will enable real-time updates of its users' news feeds next week. Currently, Facebook only updates the feed a few times per hour. Once this new version of the news feed goes live, it will probably be one of the largest implementations of real-time status updates on the Internet.

CC-licensed logo image used courtesy of Flickr user RBerteig

The Surchur website isn't only a search engine with an empty box awaiting your queries. It also serves as a dashboard-style homepage where the latest web trends are tracked on an ongoing basis. When you first visit Surchur, the main page features trending topics on Yahoo Buzz, CNN, Google Trends, Twitter Search, and Technorati Popular. It's at-a-glance information about the conversations and news which are hot right now.

Links at the top of the page let you also click through to see the "Weekly Hots" and "Daily Hots," which track topics that have been popular in the recent past.

But the new Surchur isn't just a dashboard-style mashup of trend lists - it's also a search engine. If you're looking for all the news on a particular topic, you can enter your query into the box provided and then click "surch."  The engine delves into social media sites, pictures, blogs, news, video, and product searches. The only things it doesn't return are the static pages of the traditional web, as Google would.

Performing a search directly from the search box delivers you to a dashboard where each section (social web, pictures, blogs, etc.) displays along with modules that feature the latest information from sites relevant in that niche. For example, the social web section displays modules that show the latest from Twitter, Digg, and Delicious. The Images section features flickr, Photobucket, and Pixsy. Blog Search shows Google Blog Search, Technorati, and IceRocket. You get the idea. However, you can delve into any one of these areas to see even more from those searches than what's displayed on the initial results page.

Also handy is the new "surchmeter" which gives the topic a ranking on both blogs and Twitter based on its current "hotness." For example, a search for the TV show "American Idol" (which aired last night in the U.S.) returned a blog ranking of "8" and a Twitter score of "9." Unfortunately, a search for the "Oscars" returned a "9" and "8" respectively, even though that event is long-gone in real-time web terms - it aired Sunday night. That system seems to need a bit of tweaking if it wants to really rank what's hot right now as opposed to what was hot a few days ago.

In addition, Surchur provides a plethora of RSS feeds. Any searches on the site as well as the trending topics and previously hot items can be tracked via RSS. Every section and subsection you visit has a feed, so it's easy to subscribe to just the news you want to follow.

Why We Need This

There has been a lot of talk lately about how Twitter Search could become the next "Google killer," but we think that may be taking it a bit too far. Clearly, the real-time web is an important vertical to follow, but it's only a slice of the pie. The real impetus behind this ongoing debate is likely due to our deep-seated frustration with the fact that Google has not integrated a real-time web search vertical into their offerings. Instead, we're having to turn to other search engines - like search.twitter.com or FriendFeed search - to know what's happening now. Still, we have no doubt that a company as smart as Google is working on this problem, even if they're not telling us about it just yet. 

However, Google's delay to launch a real-time search vertical leaves the door wide open for other startups to deliver what's being missed. The speed with which news spreads today is moving us further away from a searchable web of ranked static pages and closer towards a web where news is posted and spread as fast as it can be typed.

Does Surchur Deliver?

The most important question when analyzing these potential new real-time engines is simply this: Are they fast enough? Sadly, the answer in Surchur's case is "No." A true real-time web engine wouldn't still be displaying results for things that happened last night on their dashboard this morning. Also, Surchur's dashboard for Twitter doesn't even match up with the trending topics found on the search.twitter.com homepage. That points to a delay between what is happening right now and when Surchur's web service is made aware of it. And that, essentially, kills our hopes that Surchur could be the new real-time engine of our dreams. 

But if Surchur can close that gap - or if any other startup could do the same - the end result could be the next big thing in search engines. Until then, Surchur remains an innovative concept but one whose delivery isn't quite fast enough to meet our needs.

The concept behind Zcapes is somewhat reminiscent of the mobile social network Brightkite's feature called "placestreams." With Brightkite, users can post text and images surrounding a particular place or event directly from their mobile phone. Zcapes is very similar in concept - except instead of being a social network whose primary goal is connecting you to other users, a Zcape page is just the placestream, nothing more. However, Zcape pages aren't limited to a physical place in the real world or an event like a conference or concert. They can also be created for an activity, like "watching the Oscars" or a group - like your coworkers, for example. The pages you create can be set to public or private, as you choose.

If you want to tap into conversations surrounding a particular keyword, the service could easily do that. However, it's not limited to web chatter alone. In addition to tracking the real-time web of Twitter updates and Flickr uploads, the service can also track RSS feeds while letting you integrate messaging options, text boxes, images, polls, RSVPs, maps of a particular location, and other features that a traditional blog might have right into your Zcape page. (You can see Zcape in action by checking out this one for ReadWriteWeb: http://rww.zcapes.com or this one for SmartMobs, which recently introduced the service to us.)

How To Share Your Creation

Once you have created a Zcape of your own, you can share it via email, Twitter, or even QR code. Upon creation, the service provides an image of a QR barcode which you could easily stick somewhere in the real world - especially if you had one of these portable, ink-free photo/sticker printers. Make sure to save the image right way, though, because once we left that final page of the creation process, we had trouble finding the sharing options again.

To interact with the Zcapes created by others, you'll need to sign up for a free account. You can then "Favorite" Zcapes by clicking on "Love this!" which is found at the bottom of all Zcape pages. The pages themselves are designed to be viewed from a mobile phone, not a desktop/laptop PC, as you can see from the image below, captured on a laptop.

Where Does Zcapes Fit In?

To understand where exactly a product like Zcapes fits in, you can refer to the following diagram, which will either simplify the concept for you or confuse you, depending on how well you interpret diagrams!:

Zcapes was founded by Raimo van der Klein, Claire Boonstra and Maarten Lens-FitzGerald, who are mobile service architects from a strategic creative consultancy called SPRXmobile based in Amsterdam. Together with their technical partner Triple IT, they built this service which launched last week at the Mobile World Congress in Barcelona, Spain.

For more information about Internet-connected "things", see the following articles: "5 Companies Building an Internet of Things" and "The Next Node on the Net: Your Car!"

One of the queries I used on Google was "Gmail outage," but "Gmail down" is how most people report problems. You can see past problems reported on blogs and media sites. We have covered the subject before on ReadWriteWeb, and ideally we don't want to focus on failures of Gmail or the cloud.

Twitter Search Scores

Searching "Gmail down" on Twitter gave me much more useful data. And you can see the data in real time (using the layman's version of the query). When I searched, I saw results that were minutes old. But I looked further and got past 36 pages before I stopped.

Twitter is simply a better medium in which to report that "Gmail is down." It takes just a few seconds, and it is clearly not worth going through the trouble of blogging about it.

Twitter search does not have any built-in latency. Google has to index a page before you can search it. A little-known page that has just been updated will get missed. Twitter gives its results to you raw (not filtered by popularity) and immediately.

What was even more powerful was getting replies from people I don't know who had the same problem, or variations of the problem. It was like an instant uber-forum.

Is This an Isolated Case?

Was this just a fluke, an isolated case? Most of the "Twitter-is-useful" stories have centered on the social networking angle: people you know telling you where to get a great cup of coffee in a new city, for example. What was interesting about my usage was that the results did not come from my contacts. It was just like using Google search, but better.

Of course, this does not mean that Twitter search will replace Google. But this is the first time that I have used an alternative to Google for a general search term and found the alternative to be substantively, immediately better. That seems significant.

Is This Where Twitter's Elusive Revenue Lurks?

Most of the commentary on our Help Twitter Find A Revenue Model post focuses on the social aspects of Twitter and uses those as the basis of a revenue model. Those models, though, mostly feel like an intrusion on Twitter's primary function.

But is there potential in search?