Can anyone name a few companies that were thought to have taken over an industry and then got old, slow, and too big to move quickly?

I was going to list some but I figured you didn't want this page to scroll down for a mile.

Not that I think Ask will be the one, just that there will be one.

"What is the purpose/goal for a USER to use a search engine?"

- and probably there are multiple, only sometimes overlapping answers

Adding metadata MAY assist in the solution to some people's use cases - i.e. I might want to find "a popular forum on a given topic" - a site with low traffic is unlikely to be what I'm looking for, even if they "rank" highly due to some SEO factors (perhaps they were an active forum for the first season of Lost but haven't stayed active, a blog that has moved to a new domain, etc)

Search has a couple of key parts:

- indexing (and the highly related act of crawling) - both for the completeness of the index and for the frequency of updates

- mapping a QUERY to the index

- deciding in what order & manner to display results ("order" itself may/may not fully apply - though today we tend to assume a ranked list for search results that's not the only possibility)

- choosing what to show about each search result and what ELSE to show around the search results (paid for content, multiple mediums of content, estimates of other/related searches, possible refinements to the search etc)

- in some contexts choosing what & how to LINK within the search results (Google's "cached" link, on Google mobile search linking to a mobile proxy version of the site etc) Potentially at least a search engine might leverage something like the Wayback machine - or even just the engine's own changelogs from the indexing process to offer visability into the changes & rate of change of a given site

I would guess there are many opportunities for Google and for competitors to continue to innovate in helping people answer questions, discover/recover resources, and "search" for all the multitude of reasons we search.

Shannon

So while I agree with you about the overall vulnerability of Google, I don't agree with Doc Searls' rant at all.

"There are so many ways search can be improved... There's so many ways to granulate search and make it conditional and do a much better job."

I think a chapter in "Getting Real" serves as a good riposte to that (excerpted at length below).

Finally, to respond more practically to Mr. Searls, I think that in the future Google will continue to expand the Google Custom Search product to precisely enable "the many ways to granulate search and make it conditional".

---
http://gettingreal.37signals.com/ch04_Make_Opinionated_Software.php

"Some people argue software should be agnostic. They say it's arrogant for developers to limit features or ignore feature requests. They say software should always be as flexible as possible.

We think that's bullshit. The best software has a vision. The best software takes sides. When someone uses software, they're not just looking for features, they're looking for an approach. They're looking for a vision. Decide what your vision is and run with it.

And remember, if they don't like your vision there are plenty of other visions out there for people. Don't go chasing people you'll never make happy."

I could have thrown a dart at any chapter in Getting Real and it would have been just as relevant to exposing the weakness in DS's rant.

While technologists may complain about the wasted potential of Google's search, many users struggle to find useful data on the internet at all, and do not concern themselves with, for example, search metadata. Google's most recent set of innovations - folding their search engines into one - is an example of Google's brilliance in subtlety. Prior to that, searching for a video required going to another site, but they encompassed video searching in a way that did not disrupt the user experience for the majority of viewers, but made it simpler (and, of course, drew more users to their page).

It seems to me that people do not want Google to make dramatic "improvements". It's something that works. I suspect that a big innovation is going to become necessary in the next few years. Google's search is still largely built around a page-centric model of the internet, and that model is, and already has been, shifting (i.e. the loss of the page-view metric). For the most part, AJAX and the like has not disrupted that model significantly enough to require changes to search, but the combination of the (I imagine) inevitable introduction of a more semantic web along with increasing reliance on asynchronous pages will require a reworking of search. It's not here yet, though, and that's why Google's simple model is still king.

Maybe if we acknowledge that its really, really hard to get back the results we want on the first try, that the way to develop a better search experience is how you engage with the results to refine that search. It is vitally important to have good results no question, but how do you move beyond that? Add words to the search term? Add boolean operators?

Some sites are looking at a visual interface on top of search results (like Quintura or Silobreaker) in order to allow easier, more intuitive refining of results to quickly move beyond the first results.

Jeffrey's thoughts above are actually right on the mark. How can we drill down into a topic of interest (personally that's where I believe linked data shines, or where one can use an initial search query to search del.icio.us, etc).

Let's use a life science analogy. One can use one sequence to search a sequence database, one can use a profile to search a sequence database, or one can use a profile to search a database of profiles. That last layer is where search should end up. It doesn't necessarily have to be smart, but give you the capability of building upon your existing queries.

No, adding Compete to Ask will not help them compete.

Ask is all about features and goofy add-ons to search. None, have helped by searching experience whatsoever.

Admittingly, as a developer, pagerank is frustrating, and it hard to make the most popular engine of search show your stuff, because there is so room to put in your clutter.

But, that's just it. All these add-ons are clutter, and slow down the search experience. If I want traffic data, I'll goto Alexa or Compete or whatever. If I'm looking up information on foot-care products, I have no concern at-all how popular the site I'm going to is.

The most important thing is relevant information. You will need dramatic paradigm shift in order compete with Google, and Google will likely be ready when the time comes.

See also:
... Xerox became complacent. As its patents expired, it should have expected exactly what it got: ferocious competition and inroads from the Japanese and ...

See also:
...The US auto's had the fat of the land for so long that they became complacent. Complacent with their designs, quality and financing. They turn their heads and now cars like Honda, Toyota, VW, Nissan, etc., which had been around for quite a bit of years and worked their way up, are crawling up Ford's ass...

There are dozens of innovative alternative search engines poised to show Google that if you fail to study history, you are doomed to repeat it...

"Google hasn't changed search in 7 or 8 years, they are fat and happy."

I'll disagree with this. We disclosed in the NY Times piece last year that we usually launch 5+ changes *per week* just in search quality. That's a lot of changes compared to (say) even last year. We often go back and rewrite fundamental parts of our infrastructure.

Because most of the changes are under-the-hood, it's easy to come away with the impression that Google isn't changing quickly. That's a misconception. In my experience, we are iterating and improving every week in our search quality. Just because the changes aren't as visible as (say) adding Compete stats to a binocular pop-over doesn't mean that Google isn't working to improve our search results.

That said, I'm glad when other search engines try new things and different approaches. The competition keeps Google on its toes and is ultimately good for users.

In our industry people are creating duplicate sites with slight variations like wildfire. One of our competitors has 8-10 sites for the same product. They come out with a new one every quarter and it's ridiculous. Google places so much weight on keywords in the url and isn't policing this practice very well. Unfortunately their results are cluttered just like Ask.com's. I'm hoping they fix this in the near future because I prefer to use Google.

The same problem occurs with PPC. Companies are setting up multiple accounts and dominating them as well. We do pretty well with both but we only run one site. However, we used to have a broader range of competitors around us. On one Google search I perform there are 3 companies representing all of the Google top 10 results. We have one of the 10 listings so you have two companies with 9 sites appearing in the top 10 results. All for one product. It needs to be fixed or the problem will get much much worse.

There's no doubt either that Google is complacent when it comes to search. There are too many inconsistencies in the search experience (effect of capitalisation for example, or the way in which search syntax differs according to the search), and despite the advent of universal search there is still too much it doesn't do well. For example, geographical search, proximity searching, narrowing searches, phonetic search, and date search are just a few of the things that Google does badly, if at all. Other search engines compete and exceed Google in many areas, but none of them have, nor will come close to breaking the Google domination.

Why? Because Google is easy, don't need to think and it's acceptable in its results. And for many people 'good enough' is good enough. In order to do better searching searchers need to work harder, and most of them won't. Ultimately we end up with the search machine the majority are happy with, and that's still Google.

(is the URL for this article supposed to say how_VULVERABLE_is_google by the way? ;) )

Instead, let's not forget that Google entered an originally entered an already saturated and highly-competitive search engine market - just a year or two shy of when the .bomb bubble blew many start-ups out of the water.

And as much as their success was built on the PageRank algorithm, what also contributed greatly to their use was how they re-invented the human-user interface to the search engine process.

That is, they made it simple, fun, and easy.

This is why while other search engines failed to combat Google's growth, as they added feature after feature to their product - only to mire the user down in a bog of complexity and confusion.

Whereas Google continued to grow, and continues to this day to grow with a pattern of re-inventing and simplifying the workflow and/or processes involved in any given task is where Google has earned its loyalty.

Complexity kills software, as much in ease of use by consumers as its maintenance by coders. The fool's gold that features are what make a product or service #1 has been the bane of many startups.

Me, I'd prefer to see more work in the direction of making the browser the desktop operating system, while exposing their API's so in a sense I can have all the benefits of a well-funded SOA so I can provide my users with the simplest and most innovative solutions to solve their daily problems.