Google is like a young mammoth, already very strong but still growing. Healthy quarter results and rising expectations in the online advertising space are the biggest factors for Google to keep its pace in NASDAQ. But now let's think outside the square and try to figure out a Google killer scenario. You may know that I am obsessed with open source (e.g. my projects openhuman and simplekde), so my proposition will be open source based - and I'll call it Google@Home.
First let me define what my concept of Google@Home is. Briefly, Google@Home is an open source, distributed clone of Google. We already have many open source search engine projects - Apache Lucene (which is composed of Nutch and Hadoop distributed file system sub-projects) being the most credible one. So this Google@Home concept can be based on one of those open source search engines. Of course it will have a long way to go before reaching Google's utility and reach. But more importantly, Google@Home will be a distributed, decentralized system. What this means is that our desktop computers' idle time will become a part of this new search engine's computational power. In effect this allows it to compete with Google's beefy data centers. This is not a new concept either, SETI@Home and Folding@Home are 2 well known scientific projects that use the same grid computing idea in their cores. Indeed Google itself is the biggest supporter of Stanford University based Folding@Home, by dedicating the resources of their toolbars to this project.
]]> Comparison to WikiasariThe distributed nature of the engine is what makes it different from Wikipedia co-founder Jimmy Wales' Wikiasari project, which is an open source wiki-inspired search engine. While Wikiasari's power may come from Wikipedia, its weakest chain is too much human dependency; the power of masses worked well in the open, community driven encyclopedia project, Wikipedia. But vandalism has still been present - albeit at a manageable level. I'm not sure if this can work so well in search engines though.
Well the concept is clear, but you may wonder about the motivation behind it - why would anyone, an organization or a loosely formed group of people, unite around such a project; and why would people dedicate their computer's' idle time to this? Here are some reasons:
Who would create an open source Google clone?
Perhaps, Google itself. Or Google competitors such as Ask or Yahoo. Also it might be something that P2P kings Niklas Zennstrom and Janus Friisk are up to - besides their Joost project. Everything is possible, but in my opinion the most plausible option would be a joint attack by direct competitors. Indeed perhaps the best fit would be the classic "closed source" company Microsoft!! This could be a mirror response to Google, who up till now has leveraged most of its PR towards Microsoft's 'evil' closed source approach (i.e. the subtle 'do no evil' mantra of Google). Stranger things have happened.
Another idea, this Google@Home project can make more use of power of masses in its core - Google is still reluctant to use the direct power of masses idea in its search. Yahoo, on the other hand, with their new unified Social Search Unit seems more ambitious in this arena. As a total underdog, Google@Home would be more open to such innovations and could probably profit from these new paradigms.
How could you support this type of search engine with a complementary distributed and open source ad network? Baris Karadogan has more about this in his blog. (I met him at a conference last week and it turned out that surprisingly we hatched and blogged about these similar concepts at the same time!)
Yes. this is my 'Google killer' scenario. There are many open questions though - some of them are:
Let us know what you think, and also your 'Google killer' scenarios too!
Disclosure: Emre Sokullu now works for Hakia, as a Search Evangelist. He started at Hakia in March 2007.
]]>http://lists.wikia.com/pipermail/search-l/2007-May/000418.html
http://lists.wikia.com/pipermail/search-l/2007-May/000351.html
]]>Oh course you "cache" crawled content. That's essential for creating a searchable index.
I think Craven at #25 put it well. He mentioned the same thing I mentioned in my original comment above.
You also mention PageRank in #27. PageRank is but *one* component of the scoring algorithm. An *old* one, too, predating all those PhDs hired by Google since the PageRank algo was published. As far spam, I don't think defeating spam is easy. How many spam emails did you get today? I got at least 500. You may not see as much spam in SERPs as in your inbox because of the nature of SERP/queries/ranking and Inbox/time/freshness/sorting, but you know it's there, eating Google's resources.
]]>I just wanted to make sure that it's perfectly clear that Jimmy (Jimbo) Wales' (yet-to-be-named) Wikia Search project is already discussing in some detail along these lines, (primarily on the search-l discussion list).
The distributed computing thread there has been active for some time.
]]>Back in 2003 LookSmart (through Grub http://searchenginewatch.com/showPage.html?page=2177021) once used user's unused computer resources in a grid to spider sites, but it did little more than compress the pages for the indexing process. This has been tried and it has failed.
That doesn't mean that it's not possible (I've been dreaming about it myself for a long time and it's been one of my dream projects) but there are inherent problems with it.
Firstly the biggest problem search engines face is "search engine spam". "Social" components to a search engine are weaknesses in this fight for search relevancy.
Furthermore the "social" components have yet to show benefits that can scale. For example, the rating of sites works well in limited scope (like a digg, a delicious or a stumbleupon) but do a piss poor job with web-wide searching. Link-based algorithms still dominate the quest for search relevancy and nothing else has proven to do as well just yet (not saying it won't just that it's been tried thousands of times without success, indicating a degree of difficulty).
But most importantly, people underestimate the effect that the latency of p2p has on usability. Nobody wants searches to take as long as they do on p2p platforms, where the query is passed to peers in sequenced handshakes and rare results come back minutes (or more) later.
Look, I want to beat google as much as the next guy. Both because I dislike the power they have over the web as well as my inherent interests in building leading technologies.
But the p2p platform isn't the solution. Open source is. I've followed nutch and lucene since their inceptions with hopes to build decent search engines out of them but they aren't ready yet. But there's hope:
The biggest reason I haven't put my passion for search and my skill with algorithms to use in the quest for a better search engine are due to problems with foundational issues that bore me (distributed filesystems etc) as well as scale issues that impede me (one example is the bandwidth needed to spider the web and keep it fresh).
The hope comes from next-generation services, the likes of which companies like Amazon are currently realizing. For example they offer their "300 terabyte" index as a service to build search engines upon. They also provide cloud computing as a service as well as a storage service.
The evolution of services like those will drive overhead down for problems of the scale of web search and more algo guys like me who think they can tweak their way to better relevancy (and competition for google) will have at it.
I summarize by saying that standardizing protocols (it'd be nice if the index api's protocol was a open standard so you could switch from alexa to another provider if needed) you'd avoid lock in that might discourage developers (I'm not ready to invest a boatload of time on someone else's search service just yet) and by making open source foundations for search with easy abilities to tweak the ranking algos you will see more engines out there.
p2p can only help with some of the spidering and indexing, serving the serps needs to be done from centralized servers hosting the indexes for speed reasons. With future services out there ready to help small developers scale what's really missing is something like lucene but with easy ability to customize the indexing and ranking algos.
Give me that and the maturation of cloud-computing as a service and I'll give you a very relevant search engine.
P.S.
Here's a very basic primer on making a search engine:
Why Writing Your Own Search Engine is Hard
http://www.acmqueue.com/modules.php?name=Content&pa=showpage&pid=143
@Otis, crawling is not cheap, consider 10MB sitemaps that webmasters include in their web directory; just for one site, parsing 10MB of sitemap and crawling these pages, extracting data out is a tremendous job and P2P can come to help here. I agree with your latency point, that's the problem that should be tackled, but at least, this can be solved with a semi P2P approach - a Wikipedia, P2P hybrid.
]]>Google is constantly defending itself against search engine optimizers who use dubious techniques to get their (clients) websites higher in the search results. Google manages to some extent by keeping their PageRank algorithm secret. This is one of the few examples where I think security through obscurity works.
A distributed search engine has two problems: first, someone has to manage some kind of ranking algorithm - but who? And second, by placing part of a web index database on my computer, it places me in a position to manipulate the data.
Without these problems solved, the distributed search engine is bound to fail because of manipulation by search engine optimizers.
I haven't come up with an idea to solve these problems. I'm very curious if they can be solved.
]]>@Sagar, you have to be patient, this is not an easy job and will definitely take time!
@Zeno, the good thing is if this was an open source system, you could create this add-on by yourself and let everyone use it; or perhaps you could fork the project too :-)
]]>We already have many open source search engine projects - Apache Lucene (which is composed of Nutch and Hadoop distributed file system sub-projects) being the most credible one. So this Google@Home concept can be based on one of those open source search engines.
I follow Chas's sentiments on this. Lucene has its place, but Google was successful primarily because they developed a great algorithm (which is beginning to get stale, granted) and then developed an architecture designed to use that algorithm at its full potential. Google is an optimized grid running a grid algorithm. Creating a grid and then running a generic algorithm would be ineffective and a poor design.
Building great things takes pain (if only the architects of more modern e-mail systems realized this before SMTP became so prevalent) and Google's founders took the initial pain of developing a reasonably risky, unproven algorithm into a system which then became a business. They weren't look for full results on day one, month one, or even year one, but started with the principles they thought were right and then built the whole system up from there.
Starting with something like Lucene almost means throwing away ideals and many new ideas and concepts merely to get a time bonus. Torvalds, Brin and Page didn't do it that way with their projects, and I'm not sure an open source "Google killer" could do it that way either.
Chas is right in that we need to redefine the problem, and the potential solution, although I am not entirely convinced that redefining it must mean it would change significantly. There are a lot of poor two-bit search engines out there that seem to live in the smugness of being different.. and that's another mistake to avoid.
]]>Yes, it is a good idea, but it will stay a good idea until there is some code out there.
There is a lot of search engine code published with a free license, and some experiments on distributed crawling, but I have not seen anything that scales up to the size of Google. And it is not a novel idea out there, it is just very hard to climb up to the scale of Google.
]]>Thanks for informing about these projects but unfortunately we don't have anything people can really **use** - we obviously lack such a project.
]]>- distributed search engine has been done.... what was the name... Gruby? Looksmart bought the small team + sw many years ago, obviously did nothing with it.
- I know there is one active search engine like that somewhere in UK. Can't rememebr the name now, obviously nothing spectacular.
- Jimbo's SE project might be distributed after all - check the list archives for May.
- Comparing Lucene to Google makes no sense. Lucene is a low-level library. It's up to the application to make clever use of it.
]]>Now, for the algorithms. Needless to say, Lucene is ages away from Google - and there's no reason to believe that making it distributed would change that. MS and Yahoo! both have many very smart people working for them, and I'm sure they can equal Google at some point (maybe they have, at least at some point in time - but I don't search there so I wouldn't know ;-) ) - it's safe to assume the search algorithms used by these giants are pretty dynamically evolving. Would such a (momentary) advantage make users switch? Not likely, because Google is good enough. It has to take really a radical advantage to make people switch. Personally, I haven't seen this disruptive search technology or product yet.
A minor point is also that open source development is, well, open. If there's such a great idea of a researcher working on this Mega-Lucene, Google can implement it too - there's nothing stopping them, because the GPL doesn't apply here (the software runs on Google servers), let alone other Open Source licenses. And patents probably wouldn't be used to protect this hypothetical researcher's ideas - they're not really compatible with the open source way of thinking.
(Note the difference in this aspect between this case and the case of Windows vs. Linux).
So IMHO - no, this is really far from being a "Google killer". A "Google [search] killer" needs to at least redefine the search problem as we know it ("I give you some keywords, you give me links") - and it still, of course, wouldn't kill Google :-)
You may have a better chance in conjuring up a "Google ads killer" - that's where the real killer potential is.
]]>