Comments for Retrospective: My Day Without Google Fails to Impress

Retrospective: My Day Without Google Fails to Impress

2007-06-13T18:21:38Z

Yesterday was AltSearchEngine's Day Without Google, which challenged users to drop the big 5 search engines (Google, Yahoo!, MSN/Live, Ask, and AOL) for an entire day, and instead use an alternative search engine. Taking a cue from Richard MacManus, who said he would be checking out Hakia, I decided to go with a natural language/meaning-based search engine, too: Lexxe (pronounced "Leksi").

Suffice it to say, while I got by, I missed Google and Yahoo!. Lexxe says it is powered by natural language processing and urges searchers to ask questions in plain English. It then attempts to give you an answer based on what it determines are the top results. Unfortunately, more often than not, the answer is gave just wasn't right, and the results it gave weren't much stronger than Google's.

]]> As an example, last night I caught part of a fascinating documentary about Israel's 1967 war with Egypt, Jordan, and Syria. Later, I couldn't remember the name of Israel's prime minister at the time, so I fired up Lexxe and asked: Who was Israel's prime minister during the 1967 war? Lexxe suggested that it was Yitzhak Rabin -- I knew that wasn't right. The second result, however, mentioned Levi Eshkol -- which is, it turns out, the correct answer. (Oddly enough, when I tried 'Who was the prime minister of Israel during the 1967 war?' as my query, I got a message saying I should stick to queries of under 10 words, and a suggested answer of "Feisty," but the first results highlighted the sentence: "Levi Eshkol, the Prime Minister of Israel during the 1967 war ..." which exactly answered my question.)

So the results, when I studied them a bit, weren't terrible, but were they really enough to make me switch from Google? When I tried my first query in Google this morning, the first result I got was Wikipedia's Six-Day War entry, which would lead me to Eshkol. And the fourth result highlighted text on the search results page mentioning Eshkol as prime minister. In general, though, I wouldn't ask Google a direct question, I'd likely start with something like: israel's prime minister 1967 war, which interestingly provides basically the same results as the question.

After being reminded by my Google query that the 1967 was also known as the Six-Day War, I tried to give Lexxe one last test, repeating my first query, but replacing "1967 war" with "Six-Day War." The answer was unintelligible, but the very first result it gave was titled "Levi Eshkol" and displayed a snippet talking about him being PM during the war. Google's results for the same query were not so good, with the first mention of Eshkol in slot 5, and a snippet that wasn't very clear as to who he was. Finally, a win for Lexxe!

So what does that mean? To me, it shows that meaning-based and natural language search isn't really ready for prime time. The answers Lexxe suggested were almost never right, and though the results could eventually lead to the answer, they rarely did so any quicker than Google -- and often required tinkering with my query, the same as other search engines, so didn't save me much time. (I also tried the above queries at Hakia, but with an even less positive result: it suggested Golda Meir and David Ben-Gurion, the prime ministers before and after Eshkol, for my original question).

It would seem that Google understands natural language queries just as well as these other search engines, right now. In some cases, Google actually understands natural language better (for example, I can ask "What is 3 + 5?" and get 8, Lexxe doesn't understand what I mean). One of the most hyped natural language search engines is the yet-to-be released Powerset. On Monday, TechCrunch posted the first public screenshot of a Powerset results page. The screenshot showed the query "politicians who died in office" (without quotes) and Michael Arrington was impressed:

But for "politicians who died in office" the results on Google won‚Äôt be as good. Context is required: Google has only six results for the query in quotes, and without quotes it loses its meaning and the results aren‚Äôt useful (notice the Powerset blog is the fourth result). The Powerset results are relevant and useful.

But as many TechCrunch commenters pointed out, the Powerset results aren't really that helpful, just providing a seemingly random list of sites about various politicians who died in office. With or without quotes, however, Google points users to a site called The Political Graveyard that lists every US politician that died in office with the first result. So it would seem that Google's results are still better.

Conclusion

So my day without Google demonstrated to me just how far alternative search engines (at least those of the natural language set -- which is one of the most hyped variety as potential "Google-killers") have to go to catch up with Google and the other big 5 search engines. How was your day without Google? Share your experiences in the comments below.

Comment from Peter Cooper on 2007-06-13

2007-06-13T19:41:24Z

A search engine is not always the best place to start, though. For example, when I read your query, I just thought I'd go to en.wikipedia.org/wiki/Israel .. and bam, the answer's there. There are lots of little tricks like this to track things down without relying on inaccurate search engines or poor query systems. For example, if I think.. "What was that really cool Wordpress theme I saw the other day?".. instead of hitting Google, I'll try del.icio.us/popular/wordpress and hope that a bunch of other people found it interesting too. I imagine I should probably make a list of all these shortcuts I use sometime..! Google et al can be great, but there are often much better ways to solve specific queries (sadly, though, most users just use Google for everything :()

Comment from Charles Knight on 2007-06-13

2007-06-13T19:48:44Z

Josh,

Putting aside the convenience factor, if you had been allowed to choose five of the alts, which do you think would give you the best results - Google, or the combined features/results of the five? How about ten?

Comment from Josh Catone on 2007-06-13

2007-06-13T19:54:04Z

Point well taken, Peter -- I often start with Wikipedia, or IMDB, or some other specialized site. But I think most people start with Google... and the fact that Google points me toward Wikipedia gives it a lot of points. :) (And I promise to stop overusing the word 'point' :D)

I would venture that many R/WW readers probably attack the web the way you do Peter, using a variety of tools to find the information than need and deciding where to start on a case-by-case basis.

But, as you say, I doubt that's typical of the majority of web users -- at least not yet. And if Google continues to improve (or any of these other search engines, include Lexxe, Powerset, and Hakia), that may never be how the majority of people attack web searching. :)

Comment from Josh Catone on 2007-06-13

2007-06-13T20:04:58Z

@Charles #2: I get what you're saying. And that's really how I search on a day to day basis (mostly I use Google or Yahoo!, but often times I turn to alt search engines when I think it will get me better results -- especially within specific verticals). But the convenience factor is hard to ignore. When Google works for almost all searches, it will be hard to get people to switch to a search engine that works better, but for only a handful of searches.

Comment from Marta Strickland on 2007-06-13

2007-06-13T20:22:27Z

I decided to try myself, but got bored very quickly. For one, Lexxe seemed to provide decent results when Google did, and poor results when Google also failed me. So no improvement. The only difference really is that Lexxe seems to run a billion times slower.

Then, I started getting a little more creative, and Lexxe started giving me some interesting results. Apparently, God exists, so does Santa Claus, the Mayans got it right, and I‚Äôm already dead. So I guess I won't be coming into work tomorrow. I wonder what other weird things Lexxe believes.

Comment from Michael Clarke on 2007-06-13

2007-06-13T21:18:29Z

I lasted about ten minutes. A day without wikipedia, though - that I wouldn't like to imagine.

Comment from Dan Grossman on 2007-06-13

2007-06-13T22:10:49Z

I'm not hardcore enough to switch to alternative search engines I've never heard of. I did, however, switch the search default in Firefox at home to Ask.com. I've been using it as my primary search engine for about a week now and I'm very impressed. I love the interface, I'm getting the results I need in the first page (much moreso than my week-long trial of Live.com), and often I find the unified results better than Google's new universal results.

For example, when I wanted to make sure I had the week of Father's Day right last week, I typed "Father's Day" (sans quotes) into both Google and Ask.com. Finding the date through Google meant clicking through to a result and reading the page, hoping it had the date on it. At Ask.com, the first thing on the result page is the date, description, and links to gifts, recipes, quotes (great for card writing) and eCards. It knew it was a holiday, and that context information gave Ask.com the ability to provide what most searchers are looking for immediately.

Comment from Hong Liang Qiao on 2007-06-13

2007-06-14T01:05:56Z

I have just re-run Josh Catone's test on both Lexxe and Google. My finding was:

Test Questions:
Q1. Who was Israel's prime minister during the 1967 war?
Q2. Who was the prime minister of Israel during the 1967 war?
Q3. Who was Israel's prime minister during the Six-Day war?

Answer Key:
"Levi Eshkol".

Top 10 Snippets Results:
Google Q1 (score=2/10, no.6, no.8)
Google Q2 (score=2/10, no.5, no.10)
Google Q3 (score=2/10, no.6, no.7)

Lexxe Q1 (score=3/10, no.2, no.5, no.6)
Lexxe Q2 (score=1/3, no.1)
Lexxe Q3 (score=1/3, no.1)

I am not going to analyse the results for you here. The results speak for themselves. I have saved the screenshots of the above tests as proof.

Obtaining results from the snippets is more efficient than clicking open and reading the webpage to find an answer. By saying "the first result I got was Wikipedia's Six-Day War entry, which would lead me to Eshkol", Josh Catone is not very professional in judging the results.

Many people claimed that they were yet to find out a query that a Natural Language search engine can find good results, while Google can't. Let me take this opportunity to show you just a few, e.g. "a large cup of chocolate milk", "key member of the ring", and "cheap hotels near Washington", etc. Please compare the results between Google and Lexxe and find out for yourself.

I also tried the question in another form (change "Israel's" to "Israeli"):

Q4. Who was israeli prime minister during the 1967 war?

and Lexxe's answer was "eshkol -", spot on!

Snippets scoring of Q4, showed Lexxe was far better than Google:
Google Q4 (score3/10, no.5, no. 8, no.10)
Lexxe Q4 (score=7/8, no.1-6, no.8)

Lexxe still needs to improve its Short Question Answering ability. But many errors were resulted from short of enough information about certain topics out there on the net or a choice of wording in the questions. We are trying some relaxed methods on sparse data in order to improve Lexxe's Question Answering ability.

Currently, Lexxe is developing its Beta version. Hopefully, we will be able to show users a much better search engine than our Alpha version in the near future.

I encourage users to judge Natural Language search engines with more tests before jumping into any conclusion.

Hong Liang Qiao, Founder and CEO of Lexxe

Comment from Josh Catone on 2007-06-13

2007-06-14T01:22:27Z

@Hong #8: I already went over my results in the post, but just quickly... the test for "Who was israeli prime minister during the 1967 war?" gives the result: "Answer: 1967 war yitzhak rabin israel's chief of staff in 1967 in le monde 2/28/68" ...

The snippets results matter less, because they still require a click-thru for verification and context -- if I don't know who the prime minister is, just seeing his name in a snippet, out of context, won't tell me anything. So what I really need is the proper answer straight away, or better results, for the search engine to give me an advantage over Google. The results, in general, were not any better than Google (neither were they usually worse). I will stand by that.

As for "Who was israeli prime minister during the 1967 war?" You're right, it's spot on. And that's impressive, but having to tweak your query multiple times to hit on something that Lexxe likes is, again, not an improvement over Google. To me that indicates that, as I said, the technology is not ready for prime time and needs some work before it can understand what the user is after now matter how precisely it is phrased. (This is why it is alpha, no?)

I'm not sure why you would say I was unprofessional for saying that "the first result I got was Wikipedia's Six-Day War entry, which would lead me to Eshkol."

That's a statement of fact. That Wikipedia page mentions Eshkol 6 or 7 times and has a section called "Key People Involved" in the war that lists "Levi Eshkol, Prime Minister of Israel"

The top result for that query from Lexxe was this page: http://www.cactus48.com/1967war.html ... it never mentions Levi Eshkol.

Comment from Hong Liang Qiao on 2007-06-13

2007-06-14T01:48:50Z

Regarding Comment #9: Sorry Josh, I didn't explain that clearly enough. All I meant was clicking on a link to a webpage to find out the result is not as good as getting it from the snippets, if you can. It involves an extra step before finding the information one wants, from the point of view of search efficiency. When we do laboratory test, this is a very important factor to judge the quality of search relevance.

Comment from Will on 2007-06-13

2007-06-14T03:35:35Z

Nice post. In reference to having to tweak searches multiple times, I suspect that is the consequence of you being relatively low on the learning curve for the specific search engine. After using it for a while you'll pick up some usage strategies/idioms and start feeding information in forms more likely for the algorithm to pick up.

That said... I still agree with you that these probably do not yet represent an improvement over Google.

Comment from Ramon on 2007-06-14

2007-06-14T07:24:26Z

@ Peter: #1
The way you search/browse the web is just like the way I do.
I use Google for common questions and other sites (like IMDb, Wikipedia etc) for specific subjects.

To make this way of searching easier we developed www.sugarcodes.com

btw, I used Exalead instead of Google and it is a very nice ASE.

Comment from Dit on 2007-06-14

2007-06-14T11:00:56Z

It's interesting to see that by now, Google's first page of results for the string "politicians who died in office" presents links referencing the screen shot published by Powerset, therefore overshadowing links to pages that return the actual information requested in the query. Google's belief that more popular pages mean better search (or "higher quality) results fails in this case.

Comment from Zuula on 2007-06-14

2007-06-15T01:06:12Z

Thanks for writing up your "Day Without Google" experiences, Josh. As someone who works at one of Charles Knight's "Alternative Search Engines," I can assure you that it's incredibly valuable to hear the sort of detailed feedback you've provided -- even when it's not exactly the kind of news you want to hear.

That said, I would urge you not to give up so quickly on the Alts. One of the unfortunate aspects of the "Day Without Google" is that, in his "call to action", Charles explicitly suggested that people avoid the Alts that are meta engines. This is unfortunate not only because it probably led people to avoid trying out the Alt where I work (Zuula), but also because it gave the mistaken impression that most of the non-meta Alts are somehow truly independent of the "5 Majors".

The reality, of course, is that many, many of the Alts depend on Majors such as Yahoo to provide the search results which the Alts manipulate or present in ways different than the Majors.

If, instead, the experiment had included meta engines such as Zuula, your experience might have been different ... and, I would argue, a lot more interesting.

At Zuula, for example, you still could have had access to results from your favorite Major such as Google or Yahoo. But you also could have tested web search results from Exalead, Gigablast, or even Alexa. And those results would have always been right at your fingertips when a Major didn't quite do the trick.

(And let's not forget the ten or eleven blog search engines Zuula gives you access to, or the variety of image search engines.)

So ... don't be so quick to give up on the Alts. If you keep experimenting with them, I wouldn't be surprised if you find something you really like.

All the best,

Boris Simkovich, Zuula LLC