Comments for Search Agents: Will Startups or MIT Win Out?

Search Agents: Will Startups or MIT Win Out?

2007-07-19T10:20:38Z

Written by Guest Writer Steve Spalding

The idea of "Search" is a powerful concept. Whether it's guide based like Mahalo, algorithmically powered like Google or driven by something as close to home as natural language, like Powerset. Where there is information, there is a need to be able to easily sift through it. So far, we have focused our attention on brute force solutions to the problem of search. If an engine produces bad results, well then develop a better engine. However another path that might provide a solution to this problem is a search engine that can do our work for us - a search agent. Web services that can learn from our choices, to make future searches more accurate.

]]> Let’s take a closer look at a few search agents that are tackling search from the perspective of the searcher, using our choices to improve their results and driving the idea of search in compelling new directions.

Swamii

Swamii is a topic based search agent. You begin your journey with Swamii by entering a topic. This could be anything from "swim suits" to "Steven Spielberg". After you enter this hook, Swamii conducts a search as usual. The interesting bit is that the next time you return to Swamii, it will present you with everything new it has found about your subject of interest, without forcing you to re-enter these terms. Your results are broken down into broad categories (News, Blogs, Shopping) and some more interesting ones like Peer to Peer and Television.

Unfortunately, as you add to your list of interests the results can become unwieldy - which makes it falter a bit as a general purpose search engine. All in all, this is a great tool for tracking ideas as they progress through time. As a search agent it shines, reducing the effort it would normally take to research similar ideas manually and sift through results that you have already seen.

Rollyo

Rollyo is another beta product that tackles the task of moving from search engine to search agent. This product is more of a meta-search engine than Swamii. You customize your search based on the sources you wish to search within. For example, if you only want results from the Washington Post and Read/WriteWeb, you add those sources to your SearchRoll. All future searches are conducted by searching within these sources. SearchRolls can be customized, and there are a number of them that come pre-made for you. These custom search engines can also be saved for future use.

What Rollyo lacks that separates it from a true search agent is autonomy. While the engine "learns" from your behavior, it forces you to bookmark sites that you are interested in before they can be used in your SearchRoll. This site is great if you have a very specific set of sites that you know will contain the information that you are looking for; and you want to filter out the rest. It's not so great if your main interest is to have a repetitive search conducted for you over time.

Swicki

Eurekster's Swicki competes admirably as a search agent. Once again, you start off by creating your own search engine. You select the topics, you choose the sites you wish to include and you pick among common search terms that you believe would be of interest to your users. After that, your new Swicki takes on a life of its own. Your users can vote results up and down, and you have the ability to completely delete entries from the result page. Ideally, in a short amount of time this will create a "perfect" results page, geared towards your users.

The major flaw with Swicki is that it suffers from the problem that all community based web services do: it only works optimally if you have enough users. Otherwise, it more closely resembles a "guide-based" search engine than an intelligent search agent. A Swicki without community is just a slick personalized search page. However for the most part it works perfectly, you can write your own search results and subscribe to an RSS feed which will deliver pertinent results to you.

[Ed: disclosure that Eurekster is a R/WW sponsor - there is a swicki in our sidebar]

A Glimpse Into The Future

To complete this review of software agents, a look into the future seems appropriate. MIT's Media Lab is spearheading a series of projects that may take search into a realm that the current offerings only touch upon. Three projects of particular interest are Creo, Miro and Adeo. Together, they form a matched set in semantic search.

Creo is a programming tool that enables you to teach your computer how to autonomously interact with Web sites based on a demonstration. Miro, the search module, will allow you to make goal-based, semantic searches. For example if you want to find a cure for the common cold, it will assist you in making searches that would most likely lead you towards a decent cough suppressant. Adeo helps to streamline web interactions, minimizing the inputs and outputs to only the most pertinent details.

Miro in action

Used together these MIT tools could allow you to automatically do goal-based searches with a minimum amount of steps. You can see these tools in action here.

Conclusion

Whether guided search, semantic search or algorithmic search wins the day is a question for history, but in the meantime it will be the simplest solution that will draw the most users. If any engine is to get a foothold into the hotly contested search industry, it will have to combine accuracy and relevance with ease of use. Autonomous search through agents may be one key to that puzzle.

Ed: Thanks Steve for this excellent post. On the same theme of search agents, check out Read/WriteWeb's review of Allth.at earlier this year.

Agents pic: MrEddy

Comment from Steve S on 2007-07-19

2007-07-19T13:30:56Z

Thanks for the opportunity! Fantastic blog as always Richard.

Comment from Amrit Hallan - Content Blog on 2007-07-19

2007-07-19T14:06:34Z

Thanks for these reviews Steve. Accept for Mahalo I'd never heard of these search software.

Comment from Nick Schmidt on 2007-07-19

2007-07-19T14:12:09Z

Great article. It is always interesting what new companies are thinking.

For instance wikipedia.org is thinking about developing a better search engine too. I think they have had over 4million volunteers to help out too.. If I remember correctly from a Fast Company Magazine article a few months back.
Right now, wikipedia, it is still pretty efficient.

I can also see search engine develop for specific topics. Like one only for music, video, jobs, real estate, and etc..

Comment from Dmitri on 2007-07-19

2007-07-19T14:30:15Z

Interesting post, although one can argue that there is much more variety among the alternative search engine approaches. Even when the approaches are similar (say, NLP-based), the overall search paradigm may be very different. Summarizing results in a form of a digest (sensebot.net) is one example that is not covered by neither Google nor MIT nor others.

Comment from Phil Butler on 2007-07-19

2007-07-19T15:49:46Z

A true semantic search like hakia or Powerset should be able to resolve the long tail query - Wikia-Search on the other hand involves utilizing subjective data from human beings - though human interaction may be a great "filtering" capability - my talks with hakia have convinced me that the indexing system will have to be as strict and universal as language itself.

As for sensebot and other "mood" engines - these are the most simplistic forms to create and they are correspondingly limited. A combination of algorithmic - natural language engine has the greatest chance and most impact in search capability.

Apparently, the MIT approach is essentially that of hakia and Powerset all be it on a localized scale. Teaching machines to communicate with machines is a big part of the equation. If we think about it simply - we are just asking the machines to resolve a different language - or more aptly "more of it" - kind of like a kid growing up in a way. From small words and then phrases to more complex combinations and etc.

Great post Steve! The discourse on this subject drives the human engine of change and excellence. :)

Comment from David on 2007-07-19

2007-07-19T15:51:57Z

Does the Miro portion of the MIT labs search service have anything to do with the Miro video service launched yesterday (also out of the same area)?...

Comment from Steve S on 2007-07-19

2007-07-19T18:05:29Z

I'm not certain David, I'll take a look at that and get back to you.

Comment from Nat on 2007-07-19

2007-07-19T23:05:44Z

This is an interesting problem

Comment from Alex on 2007-07-19

2007-07-20T04:46:01Z

Interesting times. New engine InfoCream.com shows promise. ChaCha is interesting, stats look good on compete.com but not on Alexa. Not the best for technical questions to their guides. Mahalo seems slow moving but interesting. Wikia results don't look good yet, but I am sure they will radically improve. I think there is another level of algorithm coming as well. By 2012 I think you will be able to by this months websites worldwide indexed in a usb stick at Starbucks.

Comment from Yihong Ding on 2007-07-20

2007-07-20T17:06:50Z

Great post Steve! I totally agree with you that the movement from search engines to search agents is an interesting, and even crucial, transition on web search. In short, this transition is from centralized web search strategy to distributed web search strategy.

As people, such as Phil Butler in his comments above, pointed out, the capability of solving the Long Tail theory must be a critical property of semantic search tools. If it is only about web search, long tail means to have everyone search for the realms of himself and then share their results to each other. Apparently it seems that this job could be done by traditional centralized web search strategy.

But when the search is about semantics, things become very different. Meanings of web content are so rich and various that it is very unlikely effectively processed by any single centralized engine. In contrast, a better solution might be that we decompose a giant engine into many small agent and allow each agent dealing with a particular realm of semantics. Nevertheless, a bolder thought is to turn the entire web into a search web, i.e., every web node itself become a search agent. And then these web nodes are implicitly and automatically connected based on their search semantics.

Hakia and Powerset are excellent movement towards the dream of semantic search. But they are not real solutions. Google-style strategy can solve the web search problem at present. But it could not solve the web search in the future. At ancient time, we have priests. At modern age, we have teachers. Priests knew everything (in assumption) and our ancestors worshiped them as if they were Gods. Teachers know only partial of knowledge and they are coordinators of interest groups. Within these interest groups, we can share our knowledge even bypass the teachers. Google is treated as if it is a god. And now Hakia and Powerset want to be other gods. It is fine because we know none of them are real God (but probably some fake god may be smarter than the others). But web evolution (as comparing to the human evolution) will tell us that electing gods from no-gods can only solve problems in short terms. In a long term, a really practical solution of spread knowledge is abandoning gods but starting a general education system and producing teachers. This is the real solution about semantic search.

In summary, I am favor to the practice of search agents. The importance of Google-style web search engines will be decreased, though many readers might still not believe so yet.

--- Yihong

Comment from Michael on 2007-07-22

2007-07-22T15:47:28Z

Its unlikely that any truly semantic search or an autonomous intelligent software agent company will operate in the US and some non-us countries as the scalable indexing, autonomous intelligent software agent, image feature extraction, mediation between ontologies and other foundation patents are already issued to or licensed by US based Jarg Corporation.