As our network blog AltSearchEngines reported this morning, the long-awaited and much hyped natural language processing search engine Powerset launched this morning. Kind of. For now, the search service only uses Wikipedia and Freebase as source material for answers to your query. So it's not really fair to compare it to Google yet, but this is a search engine, and that means it will always be held to the gold standard set by the market leader.
Comparing the two is tricky, since Google searches the entire web and Powerset only processes two sites. The admittedly not very scientific method that we came up with was to compare a handful of searches on Powerset, to the results for the same query on Google restricted to "site:wikipedia.org."
Powerset does some interesting things with general queries, such as displaying "Factz," which is an ontology showing various concepts related to your query and how they relate to one another, or "Dossiers," which are a summary of key information about your query. Sometimes it yields some odd results (such as this query for "ants" for which the key finding is that ants are "a fictional race from the video game Crash Twinsanity.") However, the real promise of NLP search engines, in our opinion, is that users will be able to make search queries using natural language -- or in other words, by asking a question. So we chose a few questions at random -- things we knew Wikipedia would have answers for -- and threw them at both Powerset and Google.
Powerset's answer for this query was curious. The number one result comes from the Wikipedia entry for dental floss and highlights this line: "It was around this time, however, that Dr. Charles C. Bass developed nylon floss." Charles Bass, however, is not the correct answer. Earlier in the same article is this line, "Levi Spear Parmly, a dentist from New Orleans, is credited with inventing the first form of dental floss." Why didn't Powerset find it? It's second results, which comes from a Wikipedia entry on scientific achievements from the year 1815, correctly highlights Parmly as the inventor.
Google performed poorly for this query. The same 1815 article is identified in the sixth spot on the results, with the sentence mentioning Levi Spear Parmly highlighted, but the first few results aren't even close. Even though that's not as impressive as Powerset's results, both would require a user to click through to the article to verify the answer (because Powerset returned two different answers), and is scrolling to the 6th spot really that taxing? Taxing enough to make you switch to a new search engine? Interestingly, this query set loose on all of Google does quite well, returning the correct answer in a link to a trivia site in the first result.
Not surprisingly, both Google and Powerset nail this one. Both point to the Wikipedia entry on Paris, France in the number one spot with the sentence, "Paris is the capital of France" highlighted.
This is a fundamentally more challenging query, because there are a large number of cities and towns called "Paris" in the world. And not surprisingly, neither search engine gives what we would call a "perfect" result.
Both return the article on Paris, France first. On Google, that's followed but a handful of other articles about the city and one about Paris, Tennessee. On Powerset, the second article is about Paris Hilton -- um? -- followed by one about Paris, Texas, and in fourth place the most helpful article it could have returned, the disambiguation page on Wikipedia for Paris. (Oddly, with the question mark, the query returned "Paris, Missouri" from Freebase, and without the question mark it returned "Paris, Texas.")
On Google at large, the results focus almost exclusively on Paris, France.
It would seem that both search engines generally understand that "where is Paris" means that Paris is a place (though upon reflection, perhaps we could have been searching for the location of Paris Hilton...), but neither recognize very well that it could mean any number of different places.
Both Powerset and Google correctly call up the article about this fictional character in their first spot, but Google actually does a better job of highlighting who he is. Compare:
Google has the name of both shows in which the character appears in their excerpt, while Powerset's excerpt is made up of information about the series' that only someone who already knew the character would understand (without clicking through to read the full article) -- and it doesn't differentiate between the two -- before the ellipses the excerpt is talking about "Friends" and after it is talk about "Joey."
Google at large also finds the Wikipedia article first with the same excerpt -- it also finds clips of the show on YouTube, and the actor's (Matt LeBlanc) IMDB entry, as well the official site for the spin-off "Joey."
This was really just a very quick and informal test, and we barely put Powerset through its paces. But our first snap impressions are that Powerset doesn't do a markedly better job of finding answers than Google for most queries. Some might argue that we didn't play to Powerset's strengths and frame our queries properly, or search for things obscure enough to notice any differentiation. But the promise of natural language search is that people don't have to learn how to search -- they can just ask questions as they normally would. We also can't expect that everything they're going to look for will be obscure and hard to find via traditional search engines -- more often than not, they probably won't be.
Powerset will have an immense uphill battle to make any sort of dent in the search market. Google controls 67% of searches in the US, and the top 4 search engines make up about 98% of searches. If Google remains "good enough," Powerset will have a hard time convincing people to switch. It will be easier to make a judgment about the company's future as a real Google competitor once it is crawling more than two sites, however.
What do you think about Powerset? Impressed? Not impressed? Let us know in the comments below.
Comments
Subscribe to comments for this post OR Subscribe to comments for all Read/WriteWeb posts
Got to say, if Powerset keeps improving, it might be a challenge for other search engines... but Google? Don't think so
Posted by: MrCooker | May 12, 2008 3:04 PM
Try searching for "inventor of dental floss" and you have the answer straight in the first hit on google
Posted by: paqpaq | May 12, 2008 3:09 PM
Agreed.It shouldn't be compared with Google in anyway.
Posted by: TechPluto | May 12, 2008 3:47 PM
Powerset is a research project, pretending to be a consumer project.
If someone really wants to beat Google, they need to come up with a better ad system, not a better search engine.
Posted by: Hashim Warren | May 12, 2008 4:19 PM
Though Powerset shines on many natural language (NL) queries, even if a user never typed in a question, we improve the overall user experience. On the search results page, we aggregate Factz about a topic. On the enhanced Wikipedia pages, we provide an automatic summarization of the article ("show facts") and tools to help users navigate through the content. For example, see the France page for an excellent summary of Factz in French history.
I'm glad that you tried a few queries to test out Powerset: that's really the best way to learn about the system. And restricting a keyword engine to Wikipedia is the right methodology. But, quality can't be judged from a small sampling of queries.
Powerset uses a similar testing methodology to the major search engines based on many, many blind judgments of relevance. These tests show that, when restricted to Wikipedia, Powerset does as well or better than the other major players on both keyword and NL queries.
Keep testin' us out!
{mark} powerset product manager
Posted by: Mark Johnson | May 12, 2008 4:25 PM
didnt Ask.com have something like "what is" "where is" crap??? and nobody used it?
"why would anyone type the whole sentence for searching when keywords work just fine?"
--COP, May 12, 2008
(yea I might quoted next year with this when powerset overtakes google)
Posted by: COP | May 12, 2008 4:43 PM
I have tried PowerSet myself. Be honest, it is not very much impressive at present. If Google Search can generally outperform PowerSet even at its primarily claimed territory (Wikipedia resources), can somebody give me a reason that I need to be care of this PowerSet site in the future?
Somehow I am a little bit confused how PowerSet have raised more than 10 million round-A money.
Yihong
Posted by: Yihong Ding | May 12, 2008 4:45 PM
One of the big draw backs of Powerset is that updating constantly changing web pages and parsing the new takes a really long time. Not good for everchanging current and novel information
Posted by: jojo | May 12, 2008 7:13 PM
Josh,
I would encourage you to get from PowerSet and from Google a test set of queries (maybe 100) that you mash together and create your own "golden set". Or create your own like you did. Then get 3 people (that don't know much about search) to just run the queries and rate them 1-3. Then just use that data to evaluate. Slightly more scientific.
BTW, if you want to test UpTake vs. PowerSet on travel related terms, go ahead. :) Just said this to get Mark Johnson and Barney Pell's goat! Seriously, congrats to them for launching...its tough to get any innovative product out the door!
Posted by: Elliott Ng | May 12, 2008 7:17 PM
Elliott,
Your argument is probably sound in academic institutes. But it is not sound at all for company products that are looking for profit.
One simple reason: no ordinary user will do such a test!
As an ordinary user, I just input one or at most two queries of my interest and see results. If the results are not good, I will leave this site and probably for ever.
You can say that this is unfair (and I agree with it). But it is the real world. Will you compare two web sites by your way in real life? So Josh's test is completely sound from a technology blogger's point of view because we are standing on the ground of ordinary people, but not ivory tower scientists.
PowerSet must improve itself, at least either on the search results or on the interface design, so that it may amaze first-time users. Otherwise, I am pessimistic to the future of PowerSet.
Yihong
Posted by: Yihong Ding | May 12, 2008 10:00 PM
power set is better then google
Posted by: Ajay | May 12, 2008 10:09 PM
AskMeNow gets everyone one of your test questions with answers that are better than Google and Powerset.
Who invented dental floss?
What is the capital of France?
Where is Paris?
Who is Joey Tribbiani
Here's some more that AskMeNow gets while Powerset and Google are stumped:
who is the current president of the Unites States
who are the band members of U2?
Why is Obama popular?
Posted by: Miron | May 12, 2008 10:38 PM
Have a look at couple of queries ..
1) what is not a success?
the first entry was
Success may mean:
a level of social status ...
When a sentence is analysed from NLP point of view
is it that difficult to find a 'Not' intention in query?
If you have kind of concept net, does it have any negative links or any other way to knowing such things?
If you intend to grab attention of people
few such tricks might help .....
Statistically this query may not come as favourite
This is right but when u say NLP ....
is it same as Not NLP?
2)
Try : where is microsoft
and then : where is microsoft?
The second query fetches the Fact out but not first.
If a ? makes a sentence question in NLP what is a RegEx?
I just feel this is engine is nice but too early to release may be ..when there are whole lot of people trying
find a better search engine than google it should answer more interesting questions.
And there is not No 2 in search engine space :(
Posted by: mandar.joshi.myopenid.com
|
May 13, 2008 12:42 AM
I tested Powerset and Trueknowledge s. engines, and Trueknowledge is more powerfull in NLP field, gives better answers.
Another point is, that where are AI apps in Web20?
I'm excited which search engine fits better to users Google keyword or Powerset/Trueknowledge NLP.
Posted by: jadezoole | May 13, 2008 2:19 AM
whoooops, and there go 10 millions...
Posted by: Eray | May 13, 2008 3:07 AM
powerset is a pretty useless business model right now.
Cribbing Wikipedia and freebase is not a business but a leeching business.
It will only take wikipedia/freebase to block powerset crawling them for their business to collapse.
(see recent wired article how crawling business models are doomed to failure)
So certainly not what I expected.
Infact it wouldn't take any decent php programmer to replicate powerset as it stands by just leeching wikipedia and freebase.
For one I wish these people that manage to get all this VC capitial actually tried to make a difference and push boundries.
This is not AI or Natural Language or nothing particularly clever or useful to users.
Right now it's got as much intellience as a 1980s text adventure game.
For how much money they have to start off with, I'd be embarrassed.
Posted by: Haniff Din | May 13, 2008 5:05 AM
I have some serious doubts about NLP based system in terms of speed, but good on PowerSet for brining their NLP product to the market.
Posted by: Falafulu Fisi | May 13, 2008 5:06 AM
LOL. Lot of people, college students, adults don't like powerset. Only very few uneducated newbies like powerset.
I think Google is one of best search engine in the world. What can I say. I don't use powerset.
Posted by: Jim | May 13, 2008 8:24 AM
http://www.powerset.com/explore/pset?q=powerset+founded+what+year%3F
I didn't get the year. :/ Is this really NLP? Nope, I don't think so. Powerset is greatest vaporware. Lack of engineering talent. It's waste of money.
Posted by: Fake Stuff | May 13, 2008 8:42 AM
- Fake Stuff
who are you?
Posted by: ubitz | May 14, 2008 12:11 AM
http://www.google.com/search?num=100&hl=en&safe=off&client=firefox-a&rls=org.mozilla%3Aen-US%3Aofficial&hs=9hq&q=dental+floss+was+invented+site%3Aen.wikipedia.org&btnG=Search
http://www.powerset.com/explore/pset?q=dental+floss+was+invented&x=0&y=0
"You can't always get what you want
You can't always get what you want
You can't always get what you want
But if you try sometimes you might find
You get what you need"
Posted by: ubitz | May 14, 2008 12:27 AM
France's capital:
http://www.powerset.com/explore/pset?q=france%27s+capital&x=0&y=0
You can't always get what you want
You can't always get what you want
You can't always get what you want
But if you try sometimes well you just might find
You get what you need
Oh baby, yeah, yeah!
Posted by: ubitz | May 14, 2008 12:32 AM
Paris is located
http://www.powerset.com/explore/pset?q=paris+is+located&x=0&y=0
You can't always get what you want, no!
You can't always get what you want (tell ya baby)
You can't always get what you want (no)
But if you try sometimes you just might find
You get what you need
Oh yes! Woo!
Posted by: ubitz | May 14, 2008 12:34 AM
Joey Tribbiani
http://www.powerset.com/explore/pset?q=Joey+Tribbiani&x=0&y=0
"entia non sunt multiplicanda praeter necessitatem"
Posted by: ubitz | May 14, 2008 12:43 AM
I was involved in the Powerset beta and did not find it useful in the least. I can get far more relevant answers from Google - and if they wanted to add that fact header they could do it in a blink.
I can see natural language search working well over cell phones when using voice for input. But then again, if I could Google using voice I'd probably do that.
Bottom line, search really isn't that tough - This is just another Ask Jeeves.
Vertical search will trump this. You go into a world atlas search and search for Paris + Location. Go into a shopping search and type "Sony Bravia". Go into a Wikipedia search to search Wikipedia.
This is a feature of a search engine, not a search engine itself. I bet Wikipedia is right happy with another Wikipedia vertical search.
Posted by: MattS | May 22, 2008 7:57 AM
What about AskWiki (http://www.askwiki.com)?
It was left out of the comparison, but nails perfectly all the tested questions, including the one on dental floss.
Perhaps Powerset is just the most-hyped around, but not necessarily the best in its field.
Nico
Posted by: Nico | May 30, 2008 12:20 AM
I agree that Powerset should not be compared to Google at this point but as they claimed that they want to be the next Google, it is logical and fair to do so. Google can count on the vast redundancy of the web to counter the lower precision of keyword based search.
In any case, we (Expert System) have developed AskWiki which is based on our semantic technology Cogito. Considering that we invested very limited time and money compared to Powerset in developing the early beta of Askwiki and that we had the harder objective of providing only 1 answer (vs a list of answers like Powerset) because the application had to run on mobile phones, I think that Askwiki even in this early stage has already shown what could be the power of a mature semantic NLP technology in Vertical Search.
Luca (Expert System)
http://www.expertsystem.net/blog/
Posted by: Luca | May 30, 2008 4:39 AM