Halevy, who heads the "Deep Web" search initiative at Google, described the "Shallow Web" as containing about 5 million web pages while the "Deep Web" is estimated to be 500 times the size. This hidden web is currently being indexed in part by Google's automated systems that submit queries to various databases, retrieving the content found for indexing. In addition to that aspect of the Deep Web - dubbed "vertical searching" - Halevy also referenced two other types of Deep Web Search: semantic search and product search.

Google wants to also be able to retrieve the data found in structured tables on the web, said Halevy, citing a table on a page listing the U.S. presidents as an example. There are 14 billion such tables on the web, and, after filtering, about 154 million of them are interesting enough to be worth indexing.

Can Google Dig into the Deep Web?

The question that remains is whether or not Google's current search engine technology is going to be adept at doing all the different types of Deep Web indexing or if they will need to come up with something new. As of now, Google uses the Big Table database and MapReduce framework for everything search related, notes Alex Esterkin, Chief Architect at Infobright, Inc., a company delivering open source data warehousing solutions. During the talk, Halevy listed a number of analytical database application challenges that Google is currently dealing with: schema auto-complete, synonym discovery, creating entity lists, association between instances and aspects, and data level synonyms discovery. These challenges are addressed by Infobright's technology, said Esterkin, but "Google will have to solve these problems the hard way."

Also mentioned during the speech was how Google plans to organize "aspects" of search queries. The company wants to be able to separate exploratory queries (e.g., "Vietnam travel") from ones where a user is in search of a particular fact ("Vietnam population"). The former query should deliver information about visa requirements, weather and tour packages, etc. In a way, this is like what the search service offered by Kosmix is doing. But Google wants to go further, said Halevy. "Kosmix will give you an 'aspect,' but it's attached to an information source. In our case, all the aspects might be just Web search results, but we'd organize them differently."

Yahoo Working on Similar Structured Data Retrieval

The challenges facing Google today are also being addressed by their nearest competitor in search, Yahoo. In December, Yahoo announced that they were taking their SearchMonkey technology in-house to automate the extraction of structured information from large classes of web sites. The results of that in-house extraction technique will allow Yahoo to augment their Yahoo Search results with key information returned alongside the URLs.

In this aspect of web search, it's clear that no single company has yet to dominate. However, even if a non-Google company surges ahead, it may not be enough to get people to switch engines. Today, "Google" has become synonymous with web search, just like "Kleenex" is a tissue, "Band-Aid" is an adhesive bandage, and "Xerox" is a way to make photocopies. Once that psychological mark has been made into our collective psyches and the habit formed, people tend to stick with what they know, regardless of who does it better. That's something that's a bit troublesome - if better search technology for indexing the Deep Web comes into existence outside of Google, the world may not end up using it until such point Google either duplicates or acquires the invention.

Still, it's far too soon to write Google off yet. They clearly have a lead when it comes to search and that came from hard work, incredibly smart people, and innovative technical achievements. No doubt they can figure out this Deep Web thing, too. (We hope).

If it had been available then, I would have used the website GreenReportcard, just launched today by the Sustainable Endowment Institute, in my college hunt.

The site is very usable and includes the option of posting personalized comparisons to social networking sites. The service could certainly be more read-write based but it is a great example of a large organization making effective online use of its database.

Users can line up any number of schools to compare, though with a list of 300 schools tracked you're quite likely to find some gaps. Above are comparisons of the school I wanted to go to during that first Nirvana era (Macalester College, I wanted to study cultural anthropology) and the school I eventually graduated from (the University of Oregon, in political science).

I knew that the University of Oregon had a good reputation for recycling, green building, etc. but hadn't thought about its lack of transparency with its endowment as an ecological issue. A huge portion of the school's money comes from Nike founder Phil Knight, who's allegedly helping fund an Orwellian super-soldier research program that combines the jewel encrusted football facilities, nanotech research and a hyper-corporate urban planning regime. That's the word on the street, not information available from GreenReportCard, unfortunately. Apparently, Macallester College's money from the Readers Digest Foundation is used in a similarly non-transparent way. Who knew?

Hopefully the next generation of college students can find out this kind of information ahead of time using this new website. Don't forget, kids, to listen to Nevermind while perusing it. You don't have to listen to Emo - and you don't have to pick your colleges uninformed.

The problem with spreadsheets that their system is trying to address has to do with those who need to derive data from multiple spreadsheets (two or more). Although it's easy enough to perform sorts, build macros, and create formulas within one spreadsheet, when needing to compare values in multiple spreadsheets the process becomes more difficult.

The company's app, The Semantic Discovery System for Excel, or just SDS for short, will look for similar columns or rows between the sheets and then "semantically" connects them. They don't appear to just be throwing that term around either - the app uses the same W3C Semantic Web technologies (RDF, OWL, SPARQL) to help you capture "meaning, intelligence, and knowledge" from the data saved in your spreadsheets.

Do We Need Semantic Desktop Apps?

Does SDS solve a business problem that is not yet being addressed through current technologies? In my experience, the short answer to this question is "no." (But wait, there's more...)

Typically, when a business has need of comparing and analyzing large amounts of data, the solution is to turn to a database product that can then be queried and from which custom reports can be pulled. And a business doesn't need to spend a lot of money on a robust solution to do so - even a smaller business can create a database by using inexpensive desktop software.

However, the difference between using a database technology and "semantically connecting" some spreadsheets comes down to for whom this product is being built. In the past, databases and other business intelligence apps were built as if the creators knew that the only person using them would be an I.T. guy or gal. SDS, instead, aims to satisfy the needs of the non-technical end user.

Is this another example of tech populism at work? It certainly looks like it. Yet, in this case their market is small - a non-technical user who's also a power user with Excel? There's usually some overlap there. Not to mention, by the time you've achieved "power user" status, you've often also figured out how to do more complicated things in Excel...like, say, formulas that work across spreadsheets, for example - the very pain points this app is trying to address.

Still, it's an interesting concept to think of taking the semantic web capabilities and integrating them into everyday programs to add a layer of intelligence to these programs as well. Done correctly, it could improve the capabilities of our favorite software apps without making the programs overly complex, which is what typically happens when you add more features.

What do you think? Is the Semantic Desktop (that is, semantically-enabled desktop apps) right around the corner? Or is this product and those like it too niche to find an audience? Let us know what you think in the comments.

WorldWideScience is a science portal developed and maintained by the Office of Scientific and Technical Information (OSTI), an element of the Office of Science within the U.S. Department of Energy. The WorldWideScience Alliance, a partnership consisting of participating member countries provides the governance structure for the WorldWideScience.org portal.

When it debuted back in June 2007, it linked to 12 databases from 10 countries. Today, the portal links to 32 national, scientific databases and portals from 44 different countries.

WorldWideScience Homepage

How To Use WWS

To use the portal, you just enter a search term, as you would with any search engine and click "search." An advanced search feature lets you specify more details like title, author, or year, and lets you specify which databases to query.

Unlike Google, where results are ranked based on an algorithm that essentially displays items by popularity, WorldWideScience provides only authoritative scientific information by relevance - a ranking that is noted by the number of stars next to the result. The higher the number of stars, the more relevant the result.

Another difference between WWS and other search engines is that WorldWideScience's results are retrieved in real time. So, as you search and results come in, you may see a box appear with a "include these results" button. Clicking this will update the list with the latest information.

On your search results page, there are several features that make finding the answers you need easy to do. On the left, are "clusters," which let you narrow down a broad subject by specifying topics or dates. On the right, a snippet from Wikipedia provides a quick definition and link to an article about the subject you queried. Below that, a "EurekAlert!" section provides links to relevant articles from EurekAlert!, an online, global news service operated by AAAS, the science society. EurekAlert! is like a PR news wire for scientific research, providing a central place through which universities, medical centers, journals, government agencies, corporations and other organizations can bring their news to the media.

WorldWideScience Search Results

The WorldWideScience portal is a great resource for anyone looking for the most current findings from fields such as technology, energy, medicine, agriculture, environment, and more. You don't have to be a student, professor, or researcher to enjoy the richness of the data provided, either, as WWS has been designed to be easy enough for anyone to use. You can try it out for yourself here: worldwidescience.org

Jamie has an interesting title: Minister of Information, and his primary responsibility is to seed Freebase with information and ensure the quality of the data. According to Jamie, Freebase is "open shared database of the world's knowledge." This sounds the same as Wikipedia, but it is really quite different, because at the heart of Freebase are the ideas of semantics and openness via API.

Unlike Wikipedia, which is a free form database, Freebase is structured, where concepts and relationships are interlinked into a gigantic network or graph. Another important difference is that Freebase is all about its API. Any information contained inside the database is accessible and can be retrieved via queries. In addition, the data in Freebase is under a Creative Commons license - meaning that is readily exportable and useful by others.

When it comes to defining the meanings of things, Freebase is focused on community, with collective editing, attribution, and collaboratively built semantics. This last point is quite crucial - the founders of Freebase believe that meaning has to emerge from the collaboration between users. As such, Freebase is one of the first experiments of web-scale social contracts. The site is really focused on the notion that information is not encumbered by licenses and is free to use.

What is in Freebase Today?

Data comes into Freebase from many sources: Wikipedia, Flickr, the US Department of Commerce, Music Brainz, the USGS, SFMOMA, the US Exchange Commission, Chef Moz, and many other places. Right now the information is mostly about people and places, but the system is engineered to have a wide range of data types. As an example of "People" information, there is a lot of information in Freebase about artists along with their artwork and place in history. More esoteric types of information you might find in the database include airplanes, french cheese, tropical storms in the 90s, oil companies, and candies.

Freebase also contains lots of other kinds of data and has:

• 3.4 Million Subjects
• 750K People
• 450K Locations
• 50K Companies
• 40K Movies
• ... Over 1K Data Types with over 3K Properties

Data Representation in Freebase

While Freebase certainly has long way to go before it can claim completeness of information, its core idea of object representation and linking seems very solid. Each object in Freebase is unique. As more information comes into the system about an object, more links are created about it in the system. It is particularly interesting how Freebase establishes object identity and decides that two concepts (or subjects) are the same.

The diagram above illustrates the idea. When a new source of information is added to Freebase, it is parsed into entities and facts. The new information is then cleaned up and is merged with the existing system. But the merge only occurs if the system determines that the two bits of information are really about the same subject (in this case Leonardo Da Vinci). This is a powerful approach which allows Freebase to grow the knowledge around individual subjects. What is also interesting is that Freebase allows human editing to reconcile situations when the system is unable to automatically link the two concepts together.

Each permanent object in the system has a GUID - a unique identifier, something like this: #9202a8c040000064..... The identifier can be used to refer to the object via URL and via queries. In addition to the GUID, there are other ways to refer to the object, for example, http://www.freebase.com/view/en/leonardo_da_vinci. Beyond that, there are even other aliases, for example, you can refer to a public company by its stock ticker symbol. But regardless of the reference, the key point is that you end up with the same, unique node in the system.

Freebase also has the ability to create new domains and types that describe new concepts, for example, science fiction movies. There is a way to attach new data types to the existing domains, and then these types can be shared and used by other users. The idea is that you can model things with the fine grained resolution that you need and then you can invite people to help you refine and evolve your models. An example is the motorcycle community, which evolved out of an effort led by one guy and who was then joined by others, and has since been promoted to the top level. The community process is about merging private types to build common models.

What Can You Do With Freebase?

Freebase is not a formal system, it is not a reasoning engine, it is just a knowledge repository, a database. To query Freebase you use the Metaweb Query Language (MQL), which is based on JSON. The language is meant to be very simple and it is actually very interesting as well. The idea is that you fill out a tree which represents a partial graph with pieces that you know and then the system basically fills in all the slots that you left blank and delivers back all possible subgraphs.

For example, say you are watching a movie and you can't tell what it is. You know that the movie stars Patrick Swayze and an actress who was also in "Tank Girl." So you create a movie query and express all these facts, using JSON-style syntax. And when you run the query you get back that the actress is Lory Petty and the movie is "Point Break" and you also get links to IMDB. So the query and the results have the same structure and to find matches you simply traverse the set of results that is returned.

Building on this example, Freebase is really meant for complex inferencing queries, the sorts of questions that Google has no way of answering using its statistical frequency algorithms. For example, what US senators took money from a foreign entity? Turns out that both Barak Obama and Hillary Clinton received donations from UBS AG, based in Switzerland. That is a complex inferencing query that needs to be expressed in a query language before it can be answered and so questions of this nature are outside of the reach of any search engine -- and Wikipedia too, for that matter.

Resources

There is quite a lot of activity going on around Freebase today. Many enthusiasts are building small proof of concept applications showcasing what can be done in the future with this powerful database. You can stay on top of the cutting edge stuff coming both from the Freebase team and community at: http://download.freebase.com and http://research.freebase.com

Responding to user feedback, the blist team focused on making their application more stabile and faster. Specifically, in two areas where it was too slow - horizontal scrolling and "blist-in-a-blist" (table inside a cell). Prior to the update, those were two of the slowest areas, but now, each has received a noticeable performance boost.

Additionally, two of blist's social features were also updated - Discovery and the Dashboard. In the Discovery section, there is now a leaderboard showing the most social blist users - that is, who is doing the most sharing of their databases. The goal here seems to be to encourage people to publicly share their data with other blist users instead of making their databases private. On the blist Dashboard, there were some minor improvements - you'll now see more events and you can even scroll through the past 100. The dashboard's UI was also tweaked to enhance the flyouts which now give you a quick overview of the user.

blist flyout

However, one of the best improvements in terms of usability is native Excel import. Prior to the update, the only way to import Excel files was to first save them as a .CSV. Now Excel docs can be imported even if they are in .XLS format.

We love to see what's arguably one of the more useful web applications still doing well. As an alternative to complicated software like Access or Zoho's database, which requires some knowledge of SQL, blist makes database creation nearly as easy as making a spreadsheet. And anything that makes our lives easier is a good thing.

Certainly those things are contributing factors to your digitally encoded self, however, this recent EMC-sponsored study discovered that your digital footprint includes far more than just the data related to individual actions.

Out of the 281,000,000,000 GB digital universe, each person's contribution is about 45 GB, and out of that 45 GB, only about half of the digital footprint would be related to these "active" individual actions - taking pictures, making VoIP calls, uploading videos, downloading content, etc.

Awareness of those sorts of self-created data trails has been steadily increasing according to a recent PEW Internet report (Dec. 2007), with nearly half of all internet users (47%) having searched for information about themselves, up 22% from 2002.

But this new research shows that we need to be aware of much more than just online mentions. What we need to concern ourselves with now, is the other half of our digital footprint. This "ambient content," the research team concluded, comprises of passive contributions, something termed as your "digital shadow."

Your shadow includes things like images of you on a surveillance camera, your bank records, your retail and airline purchase records, your telephone records, your medical database entries, copies of hospital scans, information about your web searches, general backup data, information about credit card purchases, etc.

John Gantz, Chief Research Officer and Senior Vice President of IDC explains the digital shadow as simply "information about you," but what's surprising about this shadow, he explains, is that "for the first time your digital shadow is larger than the digital information you actively create about yourself."

While for you this means being aware of the numerous places your information is stored to protect yourself from identity theft, for businesses, especially enterprise IT organizations that gather this information, it means a tremendous responsibility for the security, privacy protection, reliability and legal compliance of this information.

"Society is already feeling the early effects of the world’s digital information explosion. Organizations need to plan for the limitless opportunities to use information in new ways and for the challenges of information governance," said Joe Tucci, EMC Chairman, President and CEO. "As people’s digital footprints continue growing, so too will the responsibility of organizations for the privacy, protection, availability and reliability of that information. The burden is on IT departments within organizations to address the risks and compliance rules around information misuse, data leakage and safeguarding against security breaches."

If you're interested in the current size of your own digital footprint, you can download a copy of the Personal Digital Footprint Calculator. This tool walks you through a questionnaire that calculates your impact based on the responses to questions about your computer usage, email usage, digital camera/camcorder usage, web downloading habits, potential surveillance areas, and geographical information, among other things. The questions do make you think about your online activities, but they may be hard to answer if you're not really aware of your online activities or good at coming up with averages for things like "number of emails sent per week," for example.

Digital Footprint Calculator

However, if you take the time to fill out the Digital Footprint Calculator correctly, you'll be presented with your current "daily digital footprint," in megabytes. You can then click "Start Ticker" to launch your own personal ticker that increments over time according to your digital information creation. You can even upload this, along with the .swf file, to your own web site and share your results with others.

Example Ticker (taking wizard defaults)

Having a digital shadow is not necessarily a bad thing, the study points out, as it's what allows Amazon to make recommendations for you or display your "trustworthiness" as a seller on eBay, the downside is that, in many cases, erasing that shadow is still difficult or impossible: think about the Facebook user rebellion that took place when it was discovered how difficult/impossible it was to remove your profile from the service.

But there are other examples of where people have even less choice in the matter, like government-mandated traffic light cameras or citywide surveillance. And of course, your safety is at the mercy of credit card companies and the like - if they aren't taking security seriously, your digital shadow can be snatched away from you while an identity thief goes on a a rampage with your good name.

In the long run, it will be up to businesses to adapt to these changes and protect their customer's data. Those that don't will pay as their clients take their business to safer, more protective businesses elsewhere. And for us, just being aware of our impact on the digital universe is a good place to start.

Blist Dashboard

Blist has also added features that increase the usability of their online tool, including the following:

Email Blist to Myself/Share Blist: The new "email blist" feature lets you email a blist database to yourself with one click of a button. The blist can be sent as either an HTML table or in plain text. From the email you receive, you can either print the blist or forward it on to colleagues and friends. Or to share a blist with others directly, you only have to click on the "Share" button and enter in the email address of the person you want to share the blist with.

Shared blists pane: As a recipient of a blist shared via email, you can easily keep tabs on all the blists that had been previously shared with you from your "Shared blists" pane. Similar  to Google Reader's Shared Items, you can think of your shared blists as databases shared by your friends, as opposed to shared RSS news stories.

Starter blist: For new users, a "starter blist" appears after you have created your new Blist account. This guide shows you the different datatypes you can use on the service and helps you to understand how blists are more than just spreadsheets.

CSV Import/Export: Just as it's important to make creating new, online databases as easy as possible, it's also necessary to make importing data into them and extracting data from them easy as well. So in addition to creating databases from scratch, Blist now allows for an import and export of data in CSV format. To access the CSV export feature, you just have to click on the "Export" option from the Blist's "File" menu, one of Blist's familiarly named menus which recall the naming convention of Microsoft Office software programs (pre-Office 2007's ribbon feature). The other menus include "Edit," "View," "Tools," "Columns," "Rows," and "Cells."

Change Column Type: A feature which helps make modifying a blist simpler is the new feature which lets you quickly change a column's datatype from one type to another. You can change numbers to text, text to date, or even convert ones and zeroes to a True/False column, among other things.

Charts: The latest version of Blist now offers a charting option letting you create a variety of different charts, like bar graphs and pie charts, which helps you better visualize your data.

Discovery & Search: A feature called "Blist Discovery" lets you search through the Blist community to find database templates or information in public databases relating to your search terms. Full text search within blists now returns rows where that word or piece of text exists, where before, Blist would only search tags.

Although Blist's simplicity makes it seem like you're not dealing with anything more complicated than an online spreadsheet, the Blist UI is actually hiding a complex relational database backend. And unlike Zoho's online database, using Blist doesn't require you to know SQL to enjoy all it has to offer, which makes Blist great for users who need more than an Excel spreadsheet, but who don't want to delve into the complexities of a database software application like Access. Playing with Blist, it's hard not to think that this SaaS offering would be the next great acquisition for Google to snatch up and integrate into their online office suite, Google Docs. Until then, web office users should keep Blist in mind for the times when they need a simple database creation tool.

But now, students, journalists, and everyone else can access data straight from the source - the actual statistics published by the UN.

Using UN Data

Underneath the search box on the homepage are a few popular searches, based on what users are searching for the most. You can either click on those or enter in your own keywords to begin.

For example, to see data about a country, you just enter in the country's name and click "Search." The search result will take you to a landing page where some general information about the country is provided, like population, GDP, life expectancy, etc. You can click the link to "view full profile" to get the full details of that country's statistics.

To the left, is a box where you can apply filters to your search by specifying that you want to include or exclude certain information from your search.

Below the country's profile are all the search results for that country. Each result has a "Download" link to download the data, but there is also a handy "Preview" link which will open a small window displaying the data, so you don't have to navigate away from the page you're on to see if that data is what you were looking for. Another link, "Explore," allows you to delve into to related data sources around that topic.

You can do more than search for data on specific countries or regions, though. You can also search for data about global statistics, like greenhouse gas emissions or global solar production statistics (well, that's what I looked up).

What's great about the UN Data site is not just the ability to access this huge database of statistics (currently 55 million+ records), but the well-designed way that such massive amounts of data have been provided to the public. Anyone can use this database - school kids, techies, even mom and dad. This is one site that is definitely worth a look.

Customizable home pages can be created with configurable dashboard widgets for any purpose or can be assigned to specific members of your team. Built-in customizable reporting tools can be used to create charts, graphs, and other reports that can also be displayed on these homepages.

If you build an app with a LongJump database, you can can choose to publish it to LongJump's Application Catalog so other users can try it, review it, and subscribe to it. LongJump will even work with you if you want to make your apps available for a subscription fee, the pricing of which is up to you.

Your LongJump database is not only easy to set up and use, it is also likely to be more secure, reliable, scalable, and flexible than anything you could afford to build yourself. For $19.95/month, LongJump provides customers with an SLA guaranteeing 99.999% server and network uptime on systems that are certified for SAS 70 Type II, an industry standard for data protection compliance. If you need to add new capacity, LongJump can do this within 24 hours, while load-balancing incoming traffic.

LongJump is another example of the trend toward open platforms on the web, as it also provides a REST-based XML API for developers. This means LongJump developers have more options than just using the provided user interface to manage their data. Instead, they could forgo that interface altogether to use LongJump just as a data store; they could build a web-based front end to their apps or deliver the data in the LongJump database via a web service.

A low-cost cloud database that can be designed and used by non-programmers is a great resource on its own; the gallery of subscription applications and tools for developers make it even better and sure to be adopted by many businesses in the future.