Written by Alex Iskold and edited by Richard MacManus
In
an article in the January 1st 2007 issue of NYTimes, reporter Miguel Helft writes
about the race in Silicon Valley to beat Google. Certainly the future of search has been
much talked about lately. Last year Read/WriteWeb had a number of big posts on this
topic, including Emre's Search 2.0 and
my post about vertical
search. We have also profiled many search players, including Retrevo, Hakia, Quintura, Pluggd, Microsoft
Live Search, Snap and ChaCha.
Since we have been following the battle closely, we are excited to see the coverage in
New York Times - which signals that the search space has heated up enough to be worthy of
attention by a tech-savvy mainstream audience.
The reporter Miguel Helft works through the main issues of the race. He explains that a huge amount of venture money has been poured into the search space - not necessarily to beat Google, but perhaps to get little slices of the gargantuan advertising market. However, Helft concludes that even though many companies are trying to catch up, he says it's not clear whether they can. Google is not a monopoly like Microsoft, he states, but still it seems unfeasible to unseat to Google at this point.
Obviously, we all have the same feeling about Google's dominant position - but why? To understand, we need to segment Google's competitors and their approaches. In this post we will use Emre's categorization to see why we all intuitively feel that Google is going to remain the king of search.
There are two classical business cases for a new company: a better way or a cheaper way. In the case of the search market, cheaper simply does not apply. So the companies in the current search space are racing towards different shades of 'better'. In his post, Emre outlined three major approaches to outdoing Google:
Google's automated algorithm, based on
the original PageRank idea, is a phenomenon. The contenders to overthrow Google are
deploying two radically different strategies: 1) natural language processing/AI
techniques and 2) power of people.
In the AI camp, Hakia and Powerset are building technologies that aim to deliver better search results by better understanding what you mean. Both companies have received a lot of attention in the press lately and we also reviewed Hakia here recently. While it is clear that there are benefits in understanding natural language, what is not clear is how much difference it makes in the quality of the results.
Based on what we have seen so far, it is difficult to see how these companies can beat Google. Firstly, being able to enter the query using natural language is already allowed by Google, so this is not a competitive difference. It must then be the actual results that are vastly better. Now that is really difficult to imagine. Somewhat better maybe, but vastly different? Unlikely.
People powered search is interesting, since it brings to the game what computers lack the most - common sense. del.icio.us is not typically thought of as a search engine, yet it is probably the best example of people power in action. The great thing about del.icio.us is that we get high-quality filtered results. The not-so-great thing is that the results are limited to what most people have found popular. Arguably this is the intent, but in some cases it will also be the limitation.
Another problem is that del.icio.us has only been used so far by techies, so the results in non-technical areas (medicine for example) are not going to be great. A different approach is taken by ChaCha, which literally has people help you find things. It is interesting idea, however in our review in December we pointed out a few issues - including high monthly burn rate (since search guides have to be paid) as well as potential scalability problems.
While it is clear that solutions like del.icio.us are superior to static web directories, it seems that the search algorithm is going to be difficult to beat by power of people. The speed and completeness of the Google algorithm is going to be particularly out of reach when it comes to people powered solutions.
Google's motto is simplicity. This idea
runs through their entire product stack and is closely guarded by the company. This is
why many competitors are racing to present their search results in a better way.
Snap and Microsoft Image Search are showing Previews to improve the user experience. Previews are interesting and useful. I particularly like the Microsoft Image Search UI, which is a big improvement in how image search is presented. Unfortunately both of these technologies have to improve on relevancy and even quantity of results. Also what works well for images, does not work as well for search results - since it is difficult to determine if it's worth drilling in, purely based on the preview.
Next we look at Clustering solutions, that essentially attempt to do what vertical search engines do - but across the board and using the entire result set. Clusty is very impressive, because it manages to add a lot of value without changing how we think about search results. Along with the search results, the tab with clusters allows the user to refine the query - to basically get to the answer in no more than 2 steps.
The second engine in this category is SearchMash, which is a Google property and apparently their playground. What is striking about SearchMash is that its ascetic look - so typical of Google - is actually creating a richer way of viewing the search results. It's done using Ajax and has a mode that displays nothing but the headlines of the results. I found this mode to be surprisingly useful. There are also knobs for slicing the results along verticals like Blogs, News and Images - which are very straightforward. Both of these sites (Clusty and SearchMash) seem very useful and capable of reaching mainstream acceptance.
In the last sub-category of better UI, we find Quintura and Kartoo applying innovative Visualization technologies to search. Quintura deploys a graph/tag-cloud like approach to present search result clusters. In a way, the idea is very similar to Clusty - just done in a different way. Kartoo deploys a graph/heatmap-like approach to do exactly the same cluster partitioning. The problem with both of these approaches is that they are too complex, in my view, to make it to mainstream. Even though I have a background in graph visualizations, I do not find either one obvious or intuitive. It is unlikely that in this incarnation, visual search would be widely accessible.
A lot of these UI improvements seem neat and interesting, but none of them is going to be able to beat Google. The main reason is that it is straightforward to replicate them. Of course legally there can't be exact clones, but in this case close enough will also be good enough. None of these technologies has barrier to entries for Google, so if any of them gets enough market share Google is simply going to roll out an upgrade. Note that their toying at SearchMesh.com can already hurt some current vertical search engine providers.
There has been a lot written about vertical search and for a good reason. These technologies aim to unseat Google by having both better technology and better UI and focusing on a specific vertical. There are a lot of players going after Blogs, Classifieds, Electronics, Health, Jobs and Travel engines - because of both popularity and monetization opportunities.
Consider this simple example. If you wanted to find blog posts about web 2.0, would you use plain Google or Google Blog search? Clearly even Google itself recognizes the utility of having specialized, focused search - if not for every vertical, at least for some like Blogs and Music. So there is no doubt that vertical search makes sense and it is here to stay. The key question is how much of the Search market can vertical search engines bite off?
Actually, it could be a lot. The categories that we named are all money generating/short-pass to transaction kind of categories, which are highly sought after by advertisers. So what exactly does that mean? Given that Google already has specialized search for Blogs and Music, it seems like it is going to be just a matter of time before Google ventures into other categories. Depending on how soon this happens, it might mean a lot of trouble for current players (think Google Blog Search surpassing Technorati). Only those who have managed to build strong brand awareness (like Technorati) will survive.
So overall, even though there is a lot of activity in the space, it seems like Google will remain the search king for the foreseeable future. Various approaches will have different degrees of success in seizing bits of the market, but to make a serious dent will require time, flawless execution, big marketing dollars and, of course, a better technology. This is not a trivial combination of things.
Given that the first three are difficult points, but doable, we still need to understand what better technology means. In my mind it is the one that delivers relevant results faster. From that point of view, another promising contender (mentioned by Emre) is personalized search. With this technology, search results are going to be organized not by PageRank but by your personal interests. It is likely that a combination of a vertical search and personalized search is going to deliver fundamentally better results than Google, so that might have a chance. However, as we pointed out with the other technologies. Google is not going to sleep through this.
Let us know what you think and tell us what alternative search engine is your favorite. Also vote for the search technology you think stands the best chance of beating Google, in our poll:
Listed below are links to blogs that reference this entry: The Race to Beat Google.
TrackBack URL for this entry: http://www.readwriteweb.com/cgi-bin/mt/mt-tb.cgi/2926
Nineteen of the best sites for search engines or search engine information. Know of another site that should be listed here? Leave your suggestion at the bottom of this page.1. Google - So popular the term “Googling” has entered the vernacu... Read More
Disclosure: Eurekster is a sponsor of Read/WriteWeb Eurekster has quietly rolled out an interesting upgrade to their social search service, Swicki. Now users can contribute their own search answers to a swicki, if they feel they have expertise in the... Read More
Read/Write Web writes about NLS and the NYT article:Based on what we have seen so far, it is difficult to see how these companies can beat Google. Firstly, being able to enter the query using natural language is already allowed Read More
The Read/WriteWeb takes a comprehensive look at search in: The Race to Beat Google . Regular contributor Alex Iskold categorizes the contenders into three groups: better technology, better UI and vertical/semantic search. In other news: Google recently... Read More
Written by Alex Iskold Overview of clustering and Clusty search engine Earlier this week we have written about The Race to beat Google. In that article we discussed various approaches that startups are taking trying to unseat the web giant.... Read More
Comments
Subscribe to comments for this post OR Subscribe to comments for all Read/WriteWeb posts
I was very impressed when I discovered AskX (http://www.askx.com) a few weeks ago. It's similar to SearchMash but with a more refined UI. I think Ask (with AskX but also Ask City) could become a good candidate to gain market share over - well at least - Yahoo, Microsoft, and why not Google.
Posted by: Nicolas | January 2, 2007 2:51 AMRecently I Stumbled upon YubNub (www.yubnub.org) and Sugarcodes (www.sugarcodes.com). Why not use one of these? Then you have best of all worlds.
Posted by: Max | January 2, 2007 3:17 AMAlex, it's a very good review. However, Quintura (http://www.quintura.com) is a technology company and not a user interface (UI) company. Quintura applies its proprietary Neural Networking technology to provide a visual and more intuitive search UI. In contrast to regular search engines, Quintura indexes relationships between words and builds an index on-the-fly. This technology is patented and can not be easily replicated by anyone market player. Keep tuned to our developments. Did you check Quintura for Kids out at http://kids.quintura.com? There are more innovations out of the Quintura technology to be out soon!
Posted by: Yakov | January 2, 2007 4:54 AMThe only way Google is going to be beaten is when the next great paradigm shift in computing comes along. IBM wasn't beaten by another hardware vendor, it was beaten by Microsoft's operating system. Microsoft wasn't beaten by a another operating system, it's being beaten by the web. It follows then that Google isn't going to be beaten by another search product, it will be beaten by something else.
It stands to reason that if any of these other things prove viable, Google will incorporate it into their own product. Just like Microsoft never plays the innovator with Windows - when you're the leader, you just have to be "good enough" and copy the best ideas from competitors.
The question we should be asking is: what's the next paradigm going to be?
Posted by: Eric | January 2, 2007 8:37 AMBeating Google on their own game - the search engine - seems to me like mission impossible…they have invested billions so far in improving their technologies and data centers. By knowing some of the tiny start-ups outlined above, it seems to me pretty unserious and unprofessional to compare them to Google (perhaps it is PR). I believe that, if ever there is a chance for a company, web site or product to be able to compete with Google on the search engine’s arena, that hypothetical site should be taking a whole lot different approach, i.e. NO search engine, No indexing, regardless it is semantic based or people powered… forget about it..
The web site that would potentially, NOT beat, but undermine Google would look like anything else BUT not search engine in the traditional aspect…It should help the people find web information on demand with an ease, no toolbar needed, No specific site visited, no java scripts, no other annoying tricks as well as should be helpful to the little guys on the web manage their sites respectively. It could be a form of symbiosis or a hybrid of new breed, but without being a search engine…this hypothetical site should as well be search engine independent to be successful…
None of the above sites match that profile, so we keep on looking ahead for the site that will potentially lessen the Google’s importance on the web, not beat…
Posted by: web 2.0 innovations | January 2, 2007 8:49 AMAwesome article, Alex!
That is a very nice categorization - technology, UI and vertical search. I wonder if you should you break out "improved specification of search input" separately from the rest of search technology, e.g. natural language input, implicit behavior, parametric data, search-by-example, etc.?
My feeling is that, in the end, Google will get beaten by some sort of disruptive change - e.g. say, an Information Discovery engine that monitors for items of interest in the background and sends a periodic report, or some other completely new way of thinking about search.
Posted by: Nitink | January 2, 2007 9:08 AMUsing people power to beat search spam in Google is particularly effective. Wink (http://wink.com) let's you block any spam result you see and make it disappear immediately. Wink also incorporates tags from across the web - not just del.icio.us - to provide people ranked results on top of Google. Passionate people not paid-people make the best guides IMHO. -m
Posted by: Matthew Stotts | January 2, 2007 10:37 AMI fully agree with Eric.
I also believe that every day that is passing by google is getting stronger not weaker. With thousands of web pages cropping up every day, the only way to find meanigful information out of this jungle is through a vote of confidence from human beings.
It might not seem obvious at the first shot but I see the birth of social networking when google was born. The 'page rank algorithm' is nothing but a social phenomenon where if RRW puts a link to Maxthon then it's an edorsment from RRW of Maxthon.
The author suggested 'people powered search' as a dfferent category. I beg to differ. The way I see is that this 'people powered search' is already built into google's page ranking algorithm. In a blog a user puts a link to YouTube and google sees that a vote of confidence. Delicious serves a differen need. Delicious will provide me popular CSS or RoR articles but when it comes to search there is not much breadth.
In a way google is a mixture of both people power (page rank) + good automation (weightage etc) and with each passing day they are learning and perfecting the art.
I see only one way google could be challenged. That will be a mathematical genius coming out with a better page ranking algorithm. There is no getting around to that. I do believe it's hard to find a combination of such a guy/gal and then finding a VC who will back him/her up given that they will be up against Google. I am not even considering the patent issue here.
Posted by: Neeraj Kumar | January 2, 2007 11:21 AM"Better Technology"
I am curious why you think those are better technologies. Different they certainly are, but better? Each tries to accomplish its own different goal apparently. And as they evolve over time, that's free R and D for Google.
Google has incredible algorithms. Take for example the "Did you mean?" feature : this thing can accomplish a lot in many unrelated areas. Also, one of the strengths of Google algorithms is the fact they are tweaked daily, apparently (weights). I don't think any of the "challengers" listed above have the budget to afford that.
Posted by: Stephane Rodriguez | January 2, 2007 11:32 AM
Posted by: Stephane Rodriguez | January 2, 2007 11:37 AMAs for some improvements to beat Google in their core search area, returning English from non-English pages with English queries is one among many ways to create those "wow effects".
I agree with some of the skeptics here. Good UI or not, to unseat Google you need serious publicity and a database backend that rivals Google's for its reliability and speed. The good UI stuff is simple to conceptualize, but to deploy on a large scale? That's tricky. The core of Google is not only that they've taken very good marginal steps in the directions listed above but also that they have that amazing backend.
Posted by: TopTenSources - Social Networks | January 2, 2007 11:52 AM@Stephane Rodriguez
We do not mean that these are better technologies, we are just saying that the companies are trying to build better technologies (as oppose to different UI or vertical search).
Alex
Posted by: Alex Iskold | January 2, 2007 11:57 AM@Neraaj
Google's page rank algorithm is certainly implicitely based on people, like anything online really. It is worth while to distinguish it from search based on services like del.icio.us, where people are doing both discovery and tagging. In Google's case the machine is doing that.
Alex
Posted by: Alex Iskold | January 2, 2007 11:58 AMThe problem with unseating Google is that they are so entrenched, someone must not only be better, but they must have enough money or generate enough word-of-mouth to make people switch, or at least make people consider them as a second option.
One of the problems with the human-powered search engines is that it's impossible to cover every topic/keyword phrase with humans. To go head-to-head with Google based on that criteria is impossible.
At Bessed, we're building a human-powered search site that can complement what Google and the two or three other major engines are offering. Just as Wikipedia is a complementary, go-to source for people who want information vetted by humans, we intend to be a go-to source for people that want search results vetted by humans. We can't be a search engine replacement, but we can be a second choice that is markedly different than what Google offers. The problem with many of Google's competitors is that they don't offer anything particularly different, and only occasionally something better. (At this point the same could be said for Bessed, but we're only three months old.)
Posted by: Adam Jusko | January 2, 2007 1:16 PMIt might be worthwhile to look backward as well as forward for some perspective.
Back in the day, we had Altavista. It was absolutely tops. Then it went to paid rankings and its popularity tanked.
Maybe ultimately Google's worst enemy will prove to be itself -- if they ever forget their core business.
Posted by: Leo Klein | January 2, 2007 1:44 PM>Back in the day, we had Altavista. It was absolutely tops.
I've seen "[i]The Rise & Fall of XYZ Search Engine[/i]" a couple of times now. I'll have to say that there is an increasingly uneasy feeling about Google similar to Altavista when it first started losing traction. There are, of course, many notable differences between then and now, but the short list would be [1] more money, [2] bigger brand, and [3] better management.
By now, most of you have probably read Skrenta's "[url=http://www.skrenta.com/2007/01/winnertakeall_google_and_the_t.html]Winner-Take-All[/url]" blog post. He touches upon the real Achilles' Heel of any search engine, [b]zero switching costs[/b]. What Skrenta outlines as a Google strength could also turn against it. We saw exactly that scenario with AV. ...And, back in the day, it was unthinkable that AV could be dislodged.
Posted by: rcjordan | January 2, 2007 2:56 PMIf you ask me Google will always be king (well queen, yahoo has always been number one (http://googleresearch.blogspot.com/. So google will still be king, hopefully forever!
Posted by: Dillon | January 2, 2007 3:03 PMWTF, thats not what I said!
Posted by: Dillon | January 2, 2007 3:03 PMGotta give the shout out to DoHowTo for sheer originality. Search for something and you get free how-to videos only. Would be great to partner with another site like snap.com or wink.com and only show results when there's a positive match.
Posted by: Nathan H | January 2, 2007 3:18 PMGoogle was spread primarily by word of mouth.
Posted by: Trevor Nohcud | January 2, 2007 3:38 PMNo marketing campaign or even money to market their product whose competitor was Yahoo at the time.
It is to be noted that inertia and habit carries a lot of weight . Yahoo still has not a small market share.
Google knows that with a better product their market share can be overshone by technologically better products that can come as they did out of nowhere. However they command a greater presence and usefull product than Yahoo ever was.
What is rather surprising as well is that the google products that are constantly being introduced it is not the
ones that google would bet on that do well - it is the little pet projects that were somehow sheltered along.
It is a similar story as to google's success.
One never knows in the game of life or business.
Anyone tried out oihoi.com? It lets you search a bunch of sites using just one page. Seems interesting.
Posted by: Max | January 2, 2007 3:57 PMHaki looks good, also I was invited by ChaCha to join the search editor squad.
Posted by: Tymanis Television | January 2, 2007 4:03 PMAsk.Com is growing fastest of the search engines according to many business news pages.
Posted by: intersito | January 2, 2007 4:54 PMThere is a web search engine we have been working on for sometime now. It is called YuFind and it displays Tags when searching. http://www.yufind.com
Posted by: Gsurface | January 2, 2007 4:58 PM@Gsurface
This is a cool idea! I can immediately think of a few improvements:
Posted by: Alex Iskold | January 2, 2007 5:42 PM- Display search tag cloud on the right to get instant clustering
- Get rid of the google ads on top - they spoil it, you can put them back once you get a ton of fans
- Tags need to be cleaner, you've got some noise still
- Add links to tag searches on del.icio.us, flickr, etc.
Alex
I just completed the "Top 100 Search Engines for 2006" with the Top 10, of course, and the #1 Search Engine for 2006 - IMHO. This Excel file also has the Top 10 to watch in 2007,
and a few other things.
If you would like a copy, just email me at:
Charles@CharlesKnightSEO.com
Charles Knight
Posted by: Charles Knight | January 2, 2007 6:44 PMCharles Knight SEO
Do you know VaZiGo.com? A search engine really dangerous for Google... But only in french for this moment. http://www.vazigo.com
Posted by: Bertrand | January 2, 2007 7:38 PMI do not have enough knowledge about the other guys to comment objectively or even take the poll and this probabely tells the whole story.
Clearly, because of all the success Google has had many players will come hither, but Googs is ahead...way ahead and has cash...lots of cash.....
Posted by: Adrian Keys | January 2, 2007 8:21 PMYou say of del.icio.us:
"The not-so-great thing is that the results are limited to what most people have found popular. "
That's not really true, since del.icio.us has a search engine; you can search for any obscure topic. (Admittedly, the search isn't as fast as Google.)
In my experience, tag combinations also offer good results (often better than Google's) -- by going to something like:
http://del.icio.us/tag/search+google
http://del.icio.us/tag/unicode+japanese
Etc. You'll get just what you expect.
I'm not suggesting that del.icio.us replaces Google, but it does do some jobs better.
Posted by: Patrick Hall | January 2, 2007 8:54 PMAll good things must come to an end, trust me when I say this, Google may be king now, but theres just too much money in that market and alot of competition on the rise.
Posted by: Free Arcade Games | January 2, 2007 10:14 PMAskX and Structured Format searches like Vast and Edgio will be the key standouts against google...
Manual Trackback
Posted by: Chris Saad | January 2, 2007 10:15 PMhttp://www.touchstonelive.com/blog/2007/01/google-vs-world.html
Thats impressive stuff.
Posted by: Caribbean | January 3, 2007 7:11 AMInteresting post and discussion...
However, there's not much talk about how users can efficiently persist the results of their searches (be it URLs, HTML snippets from websites that seem relevant, images etc.) irrespective of which search engine they used.
Often, searching for something is more of a continuous process rather than a one-shot thing. Hence, users would like to continue the search process from where they left off the last time. In other cases, users may have searched for something a little while ago and just want to see the results again. Persisting the old search results (that were in many cases compiled after sorting through a maze of relevant/irrelevant sites) is thus quite valuable to users. And persisting them by simply adding links to favorites is probably not the most ideal solution.
At ZCubes, this is one of the pain that we're trying to address. The platform allows the user to quickly save URLs, any HTML content etc into a "tagged" experience that can be accessed quickly in the future.
Posted by: Parag Mathur | January 3, 2007 8:22 AMSir,
Richard MacManus did a story on our product, www.ePrecis.com , in October, 2005.
However Google shut us down on that project, and we have spent the last year doing a bigger and better version of the product, that does not scrape Google.
However in mid year I lost all my email addresses, and now I can't find the email or phone number for Richard.
We would like to bring him in on a preliminary review of the new product, which is now scheduled for release on February 14.
Can you send me his email address or phone number????
Thanks
Posted by: Ward Johnson | January 3, 2007 2:47 PMWard Johnson
ePrecis.com
952-921-9368 x115
I think all of them innovate in some way and all of them are good. But I think it's nearly impossible for them to beat Google in its own game. Powerset sounds very promising but I never had the chance to give it a try. I believe personalized search can be a big hit, even I sometimes prefer del.icio.us results over Google ones.. I think Google somehow should find a way to implement power of masses into their core search mechanism. And also I like SearchMash UI, the others are too exaggerated.
Posted by: Emre Sokullu | January 3, 2007 3:21 PMI'd be interested in your thoughts about Nstein Technologies. They've developed a search engine using semantics analysis software. Their clients already include big media such as Time Magazine and the BBC website.
Is it possible that this kind of software could vastly improve the Google algorithm or provide very good competition if big media took it over and built a Google competitor?
I wrote a brief intro about them on the Teleporting Tattler blog.
Posted by: Teleporting Tattler | January 3, 2007 4:09 PMAll of these AJAX technologies are interesting - but there is one important void
Google's success may be its ability to let the SERPs dominate,and be the center of attention - as opposed to the STYLE - and do so simply and quickly
Posted by: Search Engines WEB | January 3, 2007 10:15 PMBarking.Up.Wrong.Tree.
Google is an advertising company. Figure out how to do advertising better than they do and their search expertise is for naught.
Microsoft sells software. Google sells advertising. Sell it better, faster, cheaper than Google and you'll be a billionaire too!
Posted by: Robin Harris | January 4, 2007 6:41 AMAt Lijit we create a vertical search engine centered around a person, their content, and the people in their social network. Basically it's a personal vertical search. ( " rel="nofollow">See my post about it)
I would argue that people-powered algorithms are most powerful (and spam proof) when you have some connection to the people doing the powering: your social network. This goes back to my orignal master's thesis, Outfoxed.
-stan
Posted by: Stan James | January 11, 2007 8:21 AMCTO Lijit, Networks
Before you can beat Google they either buy your business or patent or chop of your head.
Google came in the right moment like Microsoft when they introduced Windows 95. To beat Google is "almost" impossible!
Lets see how far Wikipedias new search engine will bring it.
My tip: They will collaborate with Google! Wanna bet on it!
Posted by: anonymous email | January 17, 2007 5:29 AMThe search engine "VaZiGo" (made by Carl Guillemette) make fear to Google... For only search in french...
Posted by: Luc Béland | January 19, 2007 1:15 PM