During my visit to Hewlett Packard Labs earlier this month, I spoke to Parthasarathy Ranganathan - a Distinguished Technologist at HP Labs - about this large influx of data onto the Web.

281 Exabytes of Online Data in 2009

In her presentation at PARC, intriguingly entitled "The Physics of Data," Mayer noted that there have been three big changes to Internet data in recent times:

1. Speed (real-time data);
2. Scale ("unprecedented processing power");
3. Sensors ("new kinds of data").

Mayer went on to say that there were 5 exabytes of data online in 2002, which had risen to 281 exabytes in 2009. That's a growth rate of 56 times over seven years. Partly, she said, this has been the result of people uploading more data. Mayer said that the average person uploaded 15 times more data in 2009 than they did just three years ago.

A Sensor Revolution

Mayer talked about "a sensor revolution," including data from mobile phones. She remarked that "today's phones are almost like people," in that they have senses such as eyes (a camera), ears (a microphone) and skin (a touch screen).

HP's Ranganathan used the term "ubiquitous nanosensors," that can have multiple dimensions per sensor:

• Vibration
• Tilt
• Rotation
• Navigation
• Sound
• Air flow
• Light
• Temperature
• Biological
• Chemical
• Humidity
• Pressure
• Location

Ranganathan noted that there will soon be millions of sensors working in real time, with data sampled every second. He said there'll be lots of different applications for this data, including retail, defense, traffic, seismic, oil, wildlife, weather and climate modeling.

Exascale Web

HP sees its role as providing the computing platform required to deal with this massive influx of data and the complexity of processing it in real-time. Google clearly sees itself as a provider of exascale Web services.

We don't know yet which computing or Internet companies will be most successful over the next 5-10 years, but one thing is for sure. They'll have to know how to process and make sense of massive quantities of data flowing through the Web - and do it in real-time.

Photo credit: nasa1fan/MSFC

Now we're told of a whole new batch of far-out search innovations that are in the works, in an interview with Google's vice-president of search products and user experience Marissa Mayer in today's UK Telegraph. What are they? Translated search, social/personalized search and intuitive search. Here's what Mayer has to say about these three projects.

That sounds like a great way to move beyond an internet dominated entirely by English, or to help English-only speakers cope with an internet dominated by content in other languages. It could help open up monetization to more content and it could greatly expand all our horizons. To think that a significant portion of the world's information is inaccessible because of something as almost-solvable as language differences seems like a real shame.

Related: Google Announces New Translator Gadget for Website Owners

Social Search and Personalization

"Although we search the web right now, what we really want to do is search it as each individual user sees the web. We want Google to be the most accurate reference tool which allows people to search the web and each have an individual experience," Mayer told the Telegraph.

The Telegraph's Emma Barnett identifies social network friend connections as a key part of this. "Right now Google can only include the updates and information from these networks if the users' privacy settings are 'public'," the reporter writes. "According to Mayer - the ideal will be to get access to your friend's updates in search."

Mayer: "Understanding the social network structure and the permission rules around social networks status updates when they are not public - will really empower us in terms of search."

Understanding the structure and permission is one thing but getting access to Facebook's social graph would be another. Have you noticed that Google doesn't leverage Facebook Connect anywhere in any of its products? So far the company's experiments with social search have been impressive if small in scope. Now that Facebook is opening up, if Google can connect with it then hundreds of millions of people could have social search placed front and center in their search experience.

But if connecting Google and your Facebook social graph was a simple matter, it probably would have already happened. Microsoft, meanwhile, is a big investor in Facebook and may seek to do something similar with Bing.

See also: Google Search Gets Personal: Social Search Launches in Google Labs

Intuitive Search

Recommendation technology is something we've written about extensively here and Mayer seems to be telling the Telegraph that recommendation is going to be a big part of Google's future. Intuitive search sounds pretty far-out, but Barnett writes that it may be closer than we think.

Barnett: "The ultimate prize for Mayer is intuitive search. She wants Google to be capable of presenting information to users before they even know what they're looking for. Amazingly she doesn't think her team are that far away from achieving what she calls the 'omnivorous' search engine -i.e. one which is able to take a user's total context - where they are, what they were just reading, which direction their mobile phone is pointed and so on."

Mayer: "You could have some information waiting for you when you turn on your computer or some relevant URLs forming part of your browser background (presumably if you use Chrome - Google's browser) or on your side wiki".

Our take: This sounds cool but shouldn't be too surprising. Last week Google demoed a mobile search product that automatically recommends categories you might want to search for and gives you a way to find nearby restaurants, etc. with a single click.

Search engines have long struggled with the limitations of human users and their abilities to explain what they want. Search queries are maddeningly short and compared to many of the other signals we emit implicitly - like location, click-stream history and more - explicit search queries are relatively rare.

The future of search may very well be in semi or unprompted recommendations based largely on our implicit behavioral data. That sounds like the movie Minority Report, but it's also the direction that search companies are moving in. You didn't think Google was going to leave a Minority Report type future untouched, did you?

Check out Barnett's full coverage in the Telegraph at 'An omnivorous Google is coming'.

While Mayer expects the search market to continue to grow, she also thinks that a lot of additional growth can come from introducing new ways of searching the web. Translation and personalization are also a major issue for Google.

Asked about SearchWiki - which Arrington considers a failure - Mayer said that Google wants to morph the user experience a bit, but didn't go into any details.

Regarding the Google Goggles and Google's current dependency on text to power its search, Mayer noted that the application looks at more than just location data and image recognition algorithms. Speech recognition, however, is still easier to do for Google than image recognition.

Mobile Search

Talking about mobile searches, Mayer said that the number of mobile searches doubled last year. Mobile searches make up slightly more than 5% of all of the search queries that Google processes.

Chrome and the ChromeOS

With regards to Chrome, Mayer noted that Google wants to focus on the user experience with features like the new tab page. She described the ChromeOS as an anti-operating system. In total, Google sees "tens of millions of Chrome users," though characteristically, Mayer did not go into any details.

Google and the News Media

Google wants to increase users' engagement with news. According to Mayer, if we were to reinvent the news today, it would look very different from what we know today. She cited Google Living Stories as an experiment that tries to reinvent the news for the 21st century.

Currently, readers tend to come to articles from Google and only read one article. To increase engagement, Mayer wants to create more personalized services. In addition, she also thinks that newspapers can do a better job at keeping users on their sites. Why, for example, do most sites not offer links to related articles?

The Future of News

Mayer's vision of the future of news is a personalized stream of news that is portable. The personalization would take into account stories that your friends read, location and a knowledge of the topics a user is interested in. Asked about Rupert Murdoch, Mayer noted that Google partnered with MySpace to aggregate real-time status updates from MySpace users. She hopes that Murdoch will not pull all of his content out of Google.

Surprisingly, Mayer didn't completely deny that Google would be willing to pay publishers for their content.

Music Search

Mayer said that she was happy with Google Music as a start, especially because it includes song lyrics. Mayer sidestepped any discussion about the future of Google's Music search feature.

Google and Social Networks

Asked about Google Social Search, Mayer noted that search can help social networks by helping users to find experts in their circle of friends. Mayer noted that users are more likely to trust their friends when it comes to certain queries (snow conditions, for example). The perfect search engine would also be able to crawl private updates that a user is credentialed to see.

Mayer also noted that Google might be able to help to create an authority ranking system for real-time updates from services like Twitter and Facebook.

Google Wave

Arrington asked Mayer if users need to be trained better to understand Wave or if Google plans to tweak the experience. Permanent URLs are one of the features that Google plans to add. The fact that Google Wave doesn't have critical mass yet is also hindering the experience. Some teams at Google are currently using Wave for their internal communication. Mayer did not make any announcement regarding the future of Wave.

The event, which we live blogged earlier this morning, is the second Searchology event; the first one that was held two years ago introduced Universal Search. Today, Google announced Search Options, Google Squared, Rich Snippets and a whimsical but brilliant new Android application named Sky Map.

Google is providing a live webcast of the event, named Searchology, which is scheduled to begin at 10 PDT and conclude at 11.30 PDT after product demos.

Update: We had a chance to speak with Marissa Mayer about the new search features and have posted a three minute video of her describing the products.

Update: See our analysis, Google Search Evolves - But Has Google Finally Lost its Core Focus?

Live blogging starts here

Speaker line up today:

• Udi Manber, VB Engineering
• Patrick Riley, Software Engineer
• Scott Huffman Engineering Director
• Marissa Mayer, VP, Search Products and User Experience
• Nundu Janakiram, Associate Product Manager
• Alex Komoroske, Associate Product Manager
• Kavi Goel, Product Manager
• John Taylor, Software Engineer

Udi Manber kicks the event off saying "our job is to do rocket science that will be taken for granted." But, he adds, "there is still a lot of work to do."

He continues talking about growth throughout the centuries. "In the 20th century," he explains, the dream was to conquer nature; I think the 21st centure will be about understanding people."

"We have made a lot of progress, and you'll see some more today. But, the most promising advance is that we are starting to 'understand.'"

"We have very high confidence that this is what you're looking for. It's actually hard to do, but it looks easy to you."

"Search has to be lightening fast, relevant comprehensive fresh, but the main point is that even that is not enough."

With Google focused on doing the magic so we get instant, relevant and useful results, does this mean we are at the beginning of the real-time Web?

He ends by juggling three eggs. Then saying "I wanted to highlight that things are not always what they seem," he throws the eggs, and they bounce off the stage.

Semantic search?

Patrick Riley begins by saying "it's all about user intent." By tweaking the algorithm, and operating at scale, Google can update the search results page so that it offers results for not only the query, but for what Google thinks the query could be should what you typed into the search box be incorrect.

"We know that everything we do on the search results page, and we really care about every pixel on that page," Riley says.

"We liked the 'did you mean' link so did not want to take it off." Instead they expanded it.

Mobile search

Scott Huffman talks about mobile search: "There are a few things that make mobile search interesting," he begins. "Mobile search is growing fast -- faster than search for the PC."

"Another thing that makes mobile search interesting is the challenge of devices," he adds. "There are hundreds of them, with widely varying capabilities. Search is generally difficult to use... The third thing inherently interesting about mobile search is location."

"Our dream," he said, "is that people use mobile search every day because it is":

• Complete: all of Google is on my device; easy access; "one click" ability to dig deeper;
• Easy: effortless to search and get answers.
• Local: knows where I am and acts accordingly

One thing that isn't out yet but should be in the next few weeks is the ability to share your desktop environment with your mobile environment.

Universal Search and a Brand New Search Feature

Marissa Mayer begins with an anecdotal story about a friend of hers who needed to tie a bow tie and was convinced that by searching on Google he would be greeted with incomprehensible information. As it turned out, he told Mayer, the results were brilliant; he got videos, diagrams, and a lot of explanations.

"This is Universal Search," Mayer said, "which we introduced a couple of years ago."

Universal search began with images, maps, books, news, and video. Mayer explained that over the past two years, Google has added products and blog search.

"Universal Search now runs in 174 countries," said Mayer, and it's getting better.

"The Universal Search experience is displayed in a 'bento box' environment," Mayer said.

Google also now has "location-less" Universal Search. That is, you don't need to include your location; Google works that out by either your profile or your cookie information.

In November, Google introduced SearchWiki: the ability to add, annotate, and remove results to "really make the page your own."

But with Search Wiki and Universal Search, there is more media and more interaction. Google wants to help you find more and do more with it.

The hard, unsolved problems in search are:

• Finding the most recent info
• Expressing what you want just one type of result
• Accessing which results are best
• Knowing what you want
• Being tied to keywords can be limiting

"We need a slice-and-dice mechanism," Mayer explained, something that will help you find a particular type of result.

Search Options

As a result, Google today is launching "Search Options," which, according to Mayer, is going live as she speaks. With Search Options, the results page gives you more options and allows you to show and hide options. It ultimately provides more useful results and a better search experience.

Google Squared

Another product Google is launching and will be available in Google Labs later this month is called "Google Squared," which offers data extraction (using sentiment analysis) for values and facts and that includes rich information. Could this be similar to what we can expect from Wolfram Alpha?

Rich Snippets

A third product announced today is "Rich Snippets," which will show extra metadata in results -- things like user reviews and the cost of products. According to Mayer, "It's a way for us to enrich our results page and help users make a more informed decision about what is relevant to them."

New Android App

The last announcement was of a new Android application that looks at stars. Known as "Sky Map," it was built by John Taylor in his 20% time and will be available on the Android Market later today.

Much like other apps, it allows you to pan, zoom in, and zoom out. What is special about this, asks Mayer? The G1 phone has built-in GPS, so it knows where in the world you are and can produce maps specific to you. Using the accelerometer technology, you can point the device to the stars and see the night sky beyond. Of course (having been announced at Searchology), the app also allows you to search for a star -- say, Orion -- and then it shows you where to point your device to see that star in your night sky.

Questions and Answers

Question: Will Google start selling semantic keywords?

Marissa Mayer: No plans yet to sell keywords differently. It's about the data analysis, as opposed to building in the notion of semantics.

Question: Are products launched in all languages?

Udi Manber: Products are extremely internationalized. Everything we do applies all over the world. We have active projects in probably more than 70 languages, and we try to launch at the same time.

More About Google Squared

Mayer: The technology behind it is "totally amazing." "Google Squared blew me away even though I get jaded over time." "We are looking for structures on the Web that seem to apply facts and then corroborate the evidence by looking at other pointers. It's an incredible amount of work."

Wolfram vs. Google

Manber: "I think I proved that things are not always what they seem, and you need to look at corroborating evidence. It's not enough to find info; you have to corroborate in many places."

Regarding Wolfram, "Sergey and I had a demo early on but have been asked to be quiet."

Mayer: "We at Google are optimistic about the Web. When you see the power of search and the Internet, hopefully we'll get some signals about which information we can trust. This will help improve the data."

Q&A (cont'd)

Question: When will Google squared be available?

Mayer: Later this month.