Search 2.0 vs Traditional Search
Written by Ebrahim Ezzy and edited by Richard MacManus. Ebrahim is lead developer and co-founder of Qelix Technologies, the company behind a search 2.0 contender called Qube. This 2-part series of posts is adapted from Ebrahim's research material in developing Qube. [update Part 2 is here]
Let's start be defining what we mean by "search 2.0" vs traditional search.
Traditional Search (TSE):
Traditional search engines are based on information retrieval technologies. They implement operations such as boolean queries, proximity searches, text relevance and link analysis.
Examples: Google, Yahoo, MSN, Ask
Search 2.0 (S-2.0):
What I'm calling Search 2.0 are actually third generation search technologies. To explain the generations:
- First-generation search ranked sites based on page content - examples are early yahoo.com and Alta Vista.
- Second-generation relies on link analysis for ranking - so they take the structure of the Web into account. Examples are Google and Overture.
- Third-generation search technologies are designed to combine the scalability of existing internet search engines with new and improved relevancy models; they bring into the equation user preferences, collaboration, collective intelligence, a rich user experience, and many other specialized capabilities that make information more productive.
Examples: Swicki, Rollyo, Clusty, Wink, Lexxe
The Search 2.0 Companies (Pt 1)
Search is a multi-billion dollar market and a lot of startups want to be 'the next Google' So lets take a look at what current hot technologies are shaping the future of search.
Swicki
Swicki is a community-driven search engine that allows users
to create deep, focused searches on a specific niche. Search results from a Swicki are
more focused than a TSE and can learn and adapt automatically, based on the search
behavior of the community.
Key Feature: Pattern recognition and Adaptive filtering
How it is useful compared to TSE?: Sometimes, looking for specific information in huge web indexes is so mystifying that users feel lost. Services like Swicki promise to accelerate the evolution of Search, by providing hyper-contextual (to use Mike Arrington's term) search results.
Rollyo
Both Rollyo and Swicki pursue a similar goal: community
powered, theme-based search. Rollyo allows users to create and publish their own personal
search engines, based on websites they decide to include in their 'SearchRoll'.
SearchRoll doesn't replace a TSE, it's just a great way to search your favorite things in
your favorite places.
Key Feature: Community-driven Search
How is it useful compared to TSE?: It narrows your search down to only a few trusted sources. A welcome retreat from the current in-your-face information chaos of the web.
Clusty
As the name suggests, Clusty is a clustering engine that groups similar items
together - organizing search results into folders. It goes beyond simple search and
combines the power of clustering with meta-search (i.e. a search of other searches), to
provide a productive and flexible search experience. As well as producing organic web
results, Clusty also enables searching of shopping information, yellow pages data, news,
blog posts and images.
Key Feature: Result Clustering
How it is useful compared to TSE?: The competition has shifted from crawling the web and returning search results, to adding value to the information that has been retrieved. Clusty has a few advantages over Google:
1) You don't have to come up with your own categories or subjects in order to narrow, or refine, the search.
2) You don't have to rely on Google's perceived emphasis on links.
3) You don't have to guess the keyword, to get to that perfect page you need. Navigate the clusters and sub-clusters, just as you would use eBay, to find that one specific treasure you've been hunting for.
Wink
Using the power of social networking, Wink enables users to tag their favorite results, block
irrelevant spam and display the best sites - as hand-picked by other users.
Key Feature: Collaborative Search
How is it useful compared to TSE?: Humans can recognize spam better than any automated filter. Social Search battles search manipulation (i.e. Black Hat SEO) by allowing users to block spam directly. However, one issue is that this system can be easily gamed. But if improved, Wink can deliver a leap in value to Web searchers.
Lexxe
Lexxe
does what TSE's already do, but more efficiently. Lexxe is designed to extract short
answers on-the-fly, instead of finding the page on which the answer might be located. It
emphasizes the processing of language rather than symbols - using the level of words and
the meanings associated with them.
Key Feature: Linguistic Search
How is it useful compared to TSE?: Although they claim to be "50% more accurate and relevant than any other search engine, including google", I'm not convinced. However, they do have mechanisms in place to determine fairly accurate answers for short questions, compared to Google. For example: Who was Louis-Nicholas Vauquelin? Compare Google's answer to Lexxe's.
Summary
That wraps up Part 1 of our look into Search 2.0. In the next installment we'll be looking at other notable contenders like Jookster, Gravee, PreFound and Ebrahim's own company Qube. We'll also address questions such as:
How is traditional search evolving to Search 2.0? Can Search 2.0 replace Traditional Search, ever?
Update: Part 2 of this series is available now, with more profiles plus an analysis of how traditional search is evolving towards social search.
Share2 TrackBacks
TrackBack URL for this entry: http://www.readwriteweb.com/cgi-bin/mt/mt-tb.cgi/2697
Comments
Subscribe to comments for this post OR Subscribe to comments for all ReadWriteWeb posts
This is notably the best short summary in a most readable and understandable form for discussion of new technologies I've seen.
Very commendable.
I wish more technology articles could be so clear and concise...as well as user software and hardware manuals.
You might want to check out TicTap Web Search.
http://yws.tictap.com
It combines both Rollyo and Clusty's capabilities.
While these engines seem very exciting because they're new technology and all, they also seem like a lot of work. Personally, I like to throw in "Java Swing SDK" in my Google search bar, let 'er rip, and choose the first three entries. I think that, for now at least, all these extra capabilities -- communities (I can't even think of how this would work!), folders (too much work), -- would work for the end user, rather than against them. And what I mean is that all this new Search 2.0 shininess just seems to slow me down. I want some kind of listing within 2 seconds of me hitting the enter key. That's it. I want to enter a few key words and get my results. I think once internet search becomes as detailed as the 2.0 stuff, we need to find a better analogy, or Google needs to get more servers. Maybe I just sound like an old net veteran, but I don't see anything wrong with current search technologies. I think the people that want all this Search 2.0 nonsense are the people having trouble coming up with a comprehensible list of phrases to describe what they're talking about. I mean, it's really not that hard. I think that like, say, the Desktop on an OS, Internet Search has access analogies that work and don't work, and I say folders, communities, and digging for things in clusters isn't it. And there's my Dvorak rant for the day.
If you want to see clustering/categorization applied to Google, msn and other search engines, you may want to check out the free search client, CQ web:
http://www.q-phrase.com/
It sifts through the actual content of you search results to build an index of the most statistically relevant topics - each one which can be explored by "digging in" to citations found from your search results.
It's in beta now for Windows and Mac.
Haha, I'll sure compare those answers for Louis-nicholas Vauquelin. Google has a bunch of pages on him, and I can quickly garner he was a late 18th century chemist who had to do with discovery of the element chromium.
Lexxe has the following:
"Sorry, Lexxe has just experienced Internet connection problem. Please try a few minutes later. Thank you for your cooperation."
Part of being a good search is... uptime! (and speed).
Intriguing that you should mention Clusty... it's based on the Vivisimo search engine, which I've been using for years as an alternative to Google. It's much more useful when searching for technical information (say, development documentation) since the categorization allows you to find relevant results much more quickly.
Hi Richard,
Just to test out my theory that everything with a 2.0 designation is actually social, are you talking about "social search"
Check out my relate rant:
Thoughts?
Karl
Great article ! Just the right length and punchy
My first time here but ill come back.
You have missed out one of the major search engines that is
peforming clustering etc..
Exalead is a French company producing a clustering search engine, which allows for selecting or rejecting categories
Visit their website at http://www.exalead.com/
I know this completely redundant but... Really nice article... but i have to agree with Sherrod. I just don't see this as helpful or productive. I fail to see how any community input could help my search. This gives me the same feeling as myspace which has got to be the pinnacle of lame-ness.
Exalead is buggy and overcluttered -- the homepage gets stuck in loading limbo
TicTap is really nice , the option video is pretty cool , i meed reaaaalllly nice !
You forgot about Dumbfind! http://www.dumbfind.com
And we just launched a social bookmarking site called Searchles at http://www.searchles.com
We are busy combining the two into a social search engine.
Now that's Search 2.0!
YAAAAAAAAAAAAAAAAAAAAAAAAAAWWWWNN!!!!!!!!!!
Search 2.0? Good grief. The "next Google" is probably going to be Google, or it's going to be the "next Netscape", which is nothing.
These are all examples of "better mousetraps", which people don't really care about. What comes next is something that is out there now in the margins, or something no one has thought of yet, and is just going to catch fire while everyone is looking the other way.
Though I found all the sites mentioned here (and there are others of course) very interesting and "fun", I don't think they approach the main problem described in the third point of Search 2.0 in an effective way:
"Third-generation search technologies are designed to combine the scalability of existing internet search engines with new and improved relevancy models"
"Relevancy model" for me (and I believe for many others) in the context of web search means first and foremost getting the answer for what I'm looking for as accurately as possible, as fast as possible. There is no doubt there is room for community input, but all the bells and whistles some of the "web 2.0" companies add on can't hide the fact that sometimes (and even most of the time) the returned results are either not sufficient or still hidden within layers and layers of information. Clustering is a step in the right direction because it gives more meaning to the results, but still when the search is general the clusters can’t help the user in focusing their search.
I’m still waiting to see if there will be a service out there that will conceptually combine the powers of www.wikipedia.org (breadth), www.google.com (size) , www.digg.com (community), www.metaglossary.com (focus) and www.rollyo.com (personalization).
also, PreFound now uses Eurekster (swicki) technology...
I think Google is vulnerable right now but I haven't seen anything (yet) that will knock it off it's throne. G's algorithm is to dependent on trust based issues right now, things like age of document. It's hard to find anything current. I think that's where there's room for someone to take a dent out of Google, by showing results that are relevant, but yet more timely.
My foundd.com adds to this conversation -- please check it out. It is a vertical search engine aggregator (think: Bloglines for search) charged with online bookmarking features.
Its novel features include dual-browser operation and a drag-and-drop target for fast bookmarking. Microformat fans will be especially keen on the drag-and-drop technique.
Cheers- Sid
Watson by Intellext adds a whole another dimension to this - proactive search. It works alongside the user, understands the context of what you are doing (reading a blog post, writing an email) and proactively delivers results to your desktop in a sidebar.
Plus it acts as a customizable search aggregator - you choose all the sources Watson searches, and you can have it tap into just about anywhere: Google, Yahoo, and other search engines; Technorati and other blog search engines; MySpace, LinkedIn and other networking sites; eBay and Amazon for shopping deals; desktop search applications; company intranets...you get the idea.
Check it out at www.intellext.com
Cheers,
Leigh from Intellext
In agreement with Sherrod and shi Jun ning, Search 2.0 just doesn't seem as fast or productive.
On the other hand, add "... yet" to the end of my comment above. I'm sure someone at one of these engines will eventually get their minds around the interface tweaks/simplifications necessary to make them usable and useful.
As for the "community" ideas behind web 2.0 - they could potentially return better results than an algorithmic TSA, but they're all going to be prone to mediocrity once contributions flow in from anyone not a part of the technorati and from spammers.
I've written previously about some of the problems associated with the nonlinearity in large communities and how it can end up producing a lot of garbage (like flames and trolls in old NNTP groups): http://www.moogle1.com/2006/01/why-search-and-aggregation-work-best-in-small-communities/ and the same applies to Web 2.0 communities. A good example of problems in a large, web 2.0 community is Wikipedia: with an increasingly larger community of contributors they see more and more turbulence in both the quality and accuracy of content.
The same end-result applies to any search engine dependent on human judgement to measure context. Context not only changes quickly, it's tied to a wide range of things like the reader's environment and existing scope of knowledge. Unfortuantely that means that algorithmic TSA's can probably add a measurement of "positional-click-rate" or "multiple visits, same site selected" to their algorithms and get more consistent results than Search 2.0 communities can provide.
... that also means that TSA's are going to win this round of the search wars.
Sherrod,
Don't be quick to dismiss these as ways to cash in on the "2.0 buzzword bandwagon"! I actually find that Clusty is good at something that Google isn't. Specifically, ever try searching for something where your search term has many meanings? For example, "apple" could refer to the computer, the fruit, or the Beatles' record company. "record" could mean "a world record", an audio recording, a 12" vinyl platter, the act of "recording" something, ect.
To exemplify my point, take a look at the differences between the results of a Google search for "record" vs. a Clusty search for "record". Clusty's results are much, much, much better:
* http://www.google.com/search?hl=en&q=record [google.com]
* http://clusty.com/search?query=record [clusty.com]
What about Raw Sugar? http://www.rawsugar.com
I agree with #15's take on the importance of relevency model. Also, I agree that MetaGlossary.com is a pretty "novel" newcomer to the web 2.0 market.
> First-generation search ranked sites based on page content
Sure, Search 1.0
> Second-generation relies on link analysis for ranking - so they take the structure of the Web into account.
Yep, Search 2.0
> Third-generation search technologies are designed to combine the scalability of existing internet search engines with new and improved relevancy models
Well, I don't think the first and second gen of search engines thought they weren't doing this, so....
> they bring into the equation user preferences, collaboration, collective intelligence, a rich user experience, and many other specialized capabilities that make information more productive.
This is kind of weak. A rich user experience? It makes it sound like the second gen search engines are somehow neglecting this when in reality, they have tried some things but may also have found that a typical searcher wants a simple, easy to use, non-participatory experience. So let's not do third gen on that.
The third gen has commonly been considered the combination of personal data -- either refining results because of your own past history or that of others. Lump it all into social search, and that's your third gen. And since it's third gen, please call it Search 3.0 if you must and don't force it into a Web 2.0 world just to try and mesh some Web 2.0 companies into a Search 2.0 framework.
Folks you've named all pretty well fit in that because they're bringing in social aspect. Plus, they fit in the sense that a lot of web 2.0 seems to be simply being a start up with high hopes :)
Well, all but Clusty. Clusty is simply Vivisimo, better looking, but clustering technology that second gen has played with and experimented with for literally years.
And let's go back to the so called second gen, because they are third gen, as well. Google's Personalized Search is third gen and rocks, absolutely rocks, to improve relevancy.
Yahoo's My Web is a year-old push to be all third gen with tagging and saving and so on. And it's a big lesson in that for general search, so far people don't really see to want to be that much of a community, that into sharing. They just seem to want to search, at least the masses. But if you want to do sharing and the like, My Web totally deserves to be in the third gen round-up. Better shove Yahoo Answers in there, as well.
Fantastic comments everyone! Danny, good points and thanks for the extra suggestions. Just one thing... I don't think it's worth getting too excited about the term 'search 2.0'. It's basically just a catchy tie-in with the whole "2.0" trend right now. As you say, the social software and personalization aspects of the current Web era - and how those things and more are part of these third-gen search engines.
Dropped Richard an email too, but for everybody's benefit you may want to have a look at Qunu ( http://www.qunu.com ), which searches for people, not documents. We're still in an alpha phase (but already survived two diggs and one /.), so we really appreciate your feedback!
Great summary of the next generation search engines. Although, search 2.0 seems catchy and following the ever growing fad around web2.0 but I think the whole concept of social networking and users driving the relevancy as opposed to a search engine is a concept nobody can dismiss. Initially I dismissed many of the web2.0 driven websites,but the power of users contribution in making something better is very real. I agree that the third generation search engines discussed here are definitely better than google search.
It's worth noting in your Lexxe question answering example that the name appears to be misspelled. Lexxe does well with the misspelling, but if you spell it "Louis Nicolas Vauquelin" (no H in the Nicolas) then Google shows a OneBox result from Wikipedia.
I think it's worth pointing out that one of the Search 2.0 companies listed (Rollyo) is actually powered by (as you term it) a "TSE" (Yahoo) via backend web services APIs. As Rollyo says on their about page: "Rollyo puts the power of Yahoo! Search in your hands, by giving you the tools to create your own personal search engines - with no programming required. All you have to do is pick the sites you want to search, and we'll create a custom search engine for you." Of course, we're proud to be powering Rollyo, just thought that was noteworthy.
Broadmining should also be mentioned(http://broadmining.com). It provides concept asscoication and categorization. It can provide direct and accurate answers for search like:
"Tom Hanks movies"
"Lakers players"
The first hurdle facing these new search engines with all the cute names will be getting noticed in the first place. I Googled "Search 2.0": novel indie search engines didn't start showing up until page 5. Most of these guys didn't show up at all, before my patience gave out. I and most users like me, outside the inner circle, may never know the wonders we're missing.
"what you going to do with the novelty has gone?" - Ian Curtis, Joy Division
Last weekend in Miami South Beach on 8th Ave eating, I was approach by several people asking about my 'Fooky' t-shirt, knowing it was a search site. Keep this in mind regarding the uncritical mass marketing strategy of search 2.0 blog promoting versus talking to real world people looking for a better way to find web sites and have their web sites be found.
Sorry - I find all of above search 2.0 services trivial because they focus on the same fundemental path Yahoo and Google took back in 90s and that is to focus their service more on the people searching than the people who have web sites. Everybody and their mother have a web site - I realize this talking to real people. They want their site to be found more than they want to search for something. The fact there is a demand for a sham industry called Search Engine Optimization or Search Engine Marketing is evident enough of a need to find better solutions for web site owners in the realm of search services. Until someone gets it that it's a lot of people who own web sites and want to be found and willing to pay for solutions other than pay-per-click(sham), don't expect a search revolution too soon, especially by these comical search 2.0 acts you just profiled that focus on cuteness.
I was estastic when several called what I was doing "boring" because Google is boring meaning I'm doing it right. Web services should be boring - people usage is what makes it exciting. Ebay is boring also as well as Craiglist. Google is attractive because it is archaic and brute. Google operates in a brute force manner by indexing a huge amount of pages and extracting keywords, hence the early bragging of billions of page indexes. People made Google hot because people did their own research and investigated the search results for relevancy. That is the plain simple service of search. Keep in mind AltaVista and others fell off as they clutter the simple service of search with graphic ads and other nonsense. And these Search 2.0 repeating those same mistakes with their novelty. All this social, rolling up, clustering or semantic or whatever is simply gimmicks.
Look at message #31, the individual typed something into Google and did their own research - all Google did was returned documents and the individual in message #31 did the work to determine relevancy to themselves.
I'm the happiest search enterprenuer on earth to see you guys define search 2.0 as cuteness when I see next generation search as artifical intelligence, multi-platform and service-orientated architecture.
#32,
"Artificial Intelligence: A branch of computer science that studies how to endow computers with capabilities of human intelligence"
So you mean, AI can outperform the collective human intelligence that Search2.0 services adopt? Interesting!
Anyway, I think some Search2.0 technologies already rely on artificial intelligence (clusty, clustering; lexe, natural language processing), are multi-platform and also predominantly service-orientated.
Everything, except the last paragraph which too was meaningless, didn't look like self-advertisement of your engine (fooky?).
Hey Ed, where did you get the idea that search 2.0 was about cuteness? Have a read of Part 2 before you go making judgements like that.
Congrats. You have given me a very good link and constructive approach to look at 'what you see is what you get.'
Searching Is Polarized; Will The Five Laws Get a New Revised Version: Every Search Engine Its Searcher [ http://lit2542006.blogspot.com/2006/06/searching-is-polarized-will-five-laws.html ]
Keep up the good work. You have inspired all of us. I am citing your work in my post.
Thanks again, Mohamed
Parts of traditional search technology and experience are being sliced, isolated, re-shaped, made more explicit, given distinct user interfaces, and above all, suggested to users in different contexts than previously. The bulk of the "innovation" occuring [in Search 2.0 companies], in true Web 2.0 spirit, comes in the form of mashups of ideas first used under the hood of traditional search engines.
From http://www.animasblog.com/2006/08/03/search-20-goliath-kills-david/
Hello, Ebrahim!
Thanx for the great job you do for searchers. My posts owe yours a lot. I describe Search 2.0 for RuNet. You might check Exalead/Kartoo types for your reviews. No doubt this Search 2.0 will overtake.