Glue's synaptic web-esque technology is based on a user's browsing across common sites such as Amazon, Wikipedia and YouTube, and those visits and any interactions (comments, "likes," etc.) feeding back to automatically create a taste profile and a web of affinity with other users and recommendations of other items or content across about a dozen categories, including music, books and movies. So, can this be done without violating users' privacy or - worse yet - frustrating and boring them into attrition?

AdaptiveBlue VP Fraser Kelton thinks both aims can be accomplished. In a phone interview this morning, he shared that Glue has a three-tiered set of privacy controls to ensure that items are shared only when a user wants them to be. As the privacy/permissions inverse of Facebook Beacon, an ill-fated system for socially powered product recommendation, the Glue system is completely opt-in for users and opt-out for publishers.

Glue's all-new user profiles are rolling out today at GetGlue.com.

"Nobody's confortable sharing all of their stream," said Kelton. "Financial sites, adult content - we don't want any of that stuff shared." In addition to having a set list of sites from which to pull browsing and interaction data, Kelton said, "The secondary privacy controls can prevent visits to a website from being shared unless the user has a specific interaction on that site. You can also opt to have a private profile so only your friends can see the items you interact with." Finally, Kelton told us that users can delete or edit their own shared content on an item-by-item basis.

Our other concern was whether Glue would be a semantic web geek-fest or whether the average end user would also be able to use, enjoy, and return to the site. Kelton said the site went over well with non-techie beta testers, many of whom started out using the GetGlue.com website and eventually also installed the add-on for an even richer experience.

One of the factors we think makes Glue so much fun for end users, from geeks to complete Luddites, is its Foursquare-like system of achievement and rewards. "We're big fans of game interaction driving the user experience," said Kelton, who also revealed that the product team considered Xbox 360 as inspiration for the Glue system. Users can earn stickers on their profiles based on how many items they browse or interact with, how many users discover content through them, and how much of an expert they are on a particular topic.

Glue's game-like system of badges or stickers as rewards for user interactions and discovery makes for addictive browsing.

Each item indexed by Glue can have a guru, and only one guru at a time. The result is friendly competition, increased interactions and a very sticky website that's compelling and entertaining for even the least technical end user.

The site also allows for content to be pushed to a user's existing social networks, such as Facebook and Twitter. Here's a demo video showing more about how the site works:

As previously mentioned, the user experience of Glue - both the site and the revamped add-on - is one of synapses and nodes, not back-and-forth browsing and search for content. As data and content from all around the web is associated with films, books, music, and other items, users are able to quickly scan and consume related content without further navigation. This type of curated experience is something the AdaptiveBlue team calls "shortcut search."

As a social network, GetGlue.com is a hub; but the way it curates content makes it a series of spokes, as well.

"You don't have to go to Rotten Tomatoes or YouTube. We know there's a subset of information users want, and we're bringing it in," said Kelton, who is very excited about Glue's position on distributed social networking and the semantic web. "Jeremiah [Owyang] uses the term 'social colonization.' I also think we're implementing what Charlene Li talks about when she uses the phrase, 'The network is all around us.'"

This time, we'll focus on the Web of services, which makes services accessible to and processable for machines. These Webs all have a semantic architecture in common and follow basic Web principles, such as being decentralized, modular, simple, addressable via URIs, and built for machines.

The effort to standardize such things as service-oriented architectures (SOA) and Web services has taken years, but still we have no clear definition of what constitutes a service at a conceptual level. The interface, which is the format of what goes in and out of the service, is often described formally, but what the service is actually doing, semantically speaking, is not. While there are a number of different approaches to semantically describing Web services, such as OWL-S, WSMO and WDSL-S, none so far has managed to break out of its academic confines.

Today, there are already all kinds of services with different levels of complexity, and their number is expected to grow exponentially. The services follow different standards, and a lot of them are proprietary, uni-directional and designed to be used by humans to mash up something new. Editorial catalogs such as ProgrammableWeb and search engines for Web services such as seekda are designed for humans who are searching for a particular service for that reason. For tasks that are unsolvable for machines, there are even Web services such as Amazon's Mechanical Turk, which have humans in the back end answering tricky queries.

The problem with all of this is that each of the tens of thousands of services is accessible but not findable by a machine without a machine-understandable description. Thus, every service nowadays has to be wired to a machine by hand. So, what would machines be capable of if services were annotated with semantic descriptions?

• Service discovery
  Given an index of Web services, a machine charged with finding the right service for a particular problem could choose one among those that have been indexed.
• Contracting and execution
  Once a service has been selected, a machine could look up its terms and decide on contracting and execution details. How often would the service be needed? And what would be the cheapest contract then?
• Billing or revenue sharing
  Depending on the autonomy of the machine, one could imagine something like an Autonomous Agent, which automatically makes the best deal with the service provider on such things as billing or revenue sharing for service usage.
• Replacement on failure, based on experience
  Of course, the machine would be able to replace a failing service with an equivalent one. It could also rate a service and publish it.
• Service orchestration
  A machine could, given enough intelligence, split a task into sub-tasks and then discover, contract and orchestrate services to solve these sub-tasks. And after the sub-tasks have been addressed, the main task would be solved. Such orchestration could involve the parallelization of tasks, for speeding up or redundancy purposes, or chaining services (whereby the output of one service is inputted into the next).

Research projects such as TripCom, SUPER, SHAPE and SOA4All are dealing with these ideas and scenarios.

Future scenarios are limited only by our imagination: machines could autonomously pursue goals on behalf of their master user or company, according to a specified level of freedom. These agents could solve increasingly complex problems and be granted increasingly more autonomy (finally ending up as Skynet).

In the next and final post in this series, we will discuss how all of these scenarios could become a reality with the arrival of all three Webs: a revolution in the ability of machines to access, process and apply information.

Do you also count the Web of services as a third Web? Where do you see its limits?

(Photo by zorro-art.)

We've been given screencasts of the new version of this knowledge management application - screencasts of both the consumer- and developer-facing facets of the site. Take a look, and let us know if the new Twine lives up to expectations. This new version, we are told, will be live by the end of the year.

Developers and other techies can check out this screencast exploring Twine's collaboratively authored ontologies:

The Twine folks see the new version as a realization of Tim Berners-Lee's vision of the semantic web. So what do ReadWriteWeb readers think; is the new Twine worth the wait? Does it live up to the hype? Leave your expert comments below.

The latest critique is a scathing post by Semantic Web consultant Greg Boutin, entitled Twine in Freefall?. Boutin argues that Twine's traffic has taken a dive recently. We followed up with Twine founder Nova Spivack for his response. He admits that traffic has declined "20-25%," but says that Twine is focusing on an all-new version of its product.

The premise of Greg Boutin's post is that Twine had earlier in the year trumpeted passing Delicious and Friendfeed in traffic - ReadWriteWeb was the first to cover the Delicious trend, back in March. However now Boutin points to statistics from Compete, Alexa and Quancast showing a marked drop in traffic. Compete was the source we used in our March post, so below we've pasted comparison charts from then and now:

Compete chart from March '09 showing that Twine was trending upwards, while Delicious growth appeared to have tapered off.

Compete chart showing that Twine did indeed pass Delicious in March '09, however over June-July Twine's traffic has plummeted but Delicious held steady.

Twine: Yes Traffic Has Dropped, But We're Focusing on Version 2...

ReadWriteWeb questioned Twine about these statistics and the company admits that "our internal data shows us down 20-25%."

Twine appears to put some of this down to problems with version 1 of its product. The company told us that it is putting all of its focus and marketing efforts into a brand new version (more on that below). Therefore the drop in traffic is something Twine and its investors are comfortable with, for now.

Nova Spivack also told us that Twine had indexing problems with Google over the summer. If this was the case, that may be a big reason for the decline in Compete. Spivack explained that "we have about 500K pages that should be indexed. They [Google] are only indexing 140K pages, but we're basically not worrying about it, since T2 [version 2, see below for details] will change the game and the way we deal with Google anyway. . ."

Twine 2.0: Make or Break

Twine says that it is busy working on a new version of its product, which is why it hasn't been active on the PR front in the last few months. The company is hoping the new version gets its momentum back.

Nova Spivack elaborated on the new version to ReadWriteWeb, which we'll quote in its entirety because it illuminates Spivack and company's marketing approach:

"In the last 9 months we have made a breakthrough with the new version of Twine that changes the economics of vertical search and navigation on the Web. This new technology enables Twine to provide Web-scale faceted navigation and search across numerous vertical search categories. We are able to index structured data (like recipes, products, reviews, or any kind of database driven or XML content) with search-engine performance and scalability. This is a huge leap beyond what we were able to deliver in the first version of Twine."

"As a result of this breakthrough, we have made a strategic decision to focus all of our resources on bringing Twine version 2 (T2) to market by the end of the year. Version 1 of Twine will remain online until we are able to cutover to version 2. We are doing no further work on version 1 and no marketing for it, either. We are of course still supporting it from a technical and user perspective, however. But all our focus is on T2 moving forward."

Twine also told us that it has signed deals with nearly a dozen major content providers and brands to integrate the new search capabilities into their online services. See our March post for more context about the new version.

So what do we think about this latest twist to the Twine saga? With language like "breakthrough," "changes the economics" and "huge leap beyond," once again Twine is hyping itself up. While there have always been signs that Twine is at least partly delivering on its clear promise, the fact is that Twine continues to struggle to deliver a product the market wants. This accounts for its inconsistent growth and much of the criticism of usability which Twine has endured.

We continue to cheer for Twine, but it does seem that version 2 is make or break for the company.

Two such "divining" projects include CoPhIR (Content-based Photo Image Retrieval) Test-Collection and IBM's MUFIN (Multi-Feature Indexing Network). These projects tie into SAPIR's back end by extracting data from the Flickr archive and indexing features such as scalable color, color structure, color layout, shape edges and texture.

As shown in the video of Madrid's Plaza de España, SAPIR identifies matching media in the same way that humans derive intrinsic value from visual and sensory clues. Users can also choose to combine search terms with additional text to further drill down in search results. As is the case with regular search, if you already know the city where your image was taken, you're one step closer to finding your result. Additionally, SAPIR also has the ability to index sound and video files.

While the catalogue of media is still very limited, theoretically we may one day be able to search for almost anything using this technique. If Ashton Kutcher wears a pair of sunglasses we like, we can scan the image and search for the storefronts stocking them. If we're looking for the name of a town square, we can find it in the tags of similar images. And finally, if we're looking to self-diagnose we can compare photos of ourselves against jaundice or malaria patients.

The advantage of this tool is that we may one day have a chance to collect up the disparate bits in the digital ether and identify them as useful points of information. To test SAPIR in its early research stage, visit the homepage. You may also want to test out MUFIN to compare results.

The difference, claims Eqentia, is that "with Evri or OpenCalais, the onus is on the programmer." Eqentia says that with its product, "the content is already semanticized and all you have to do is to place it on your portal while preserving your SEO." The other two companies may disagree with that, but let's take a closer look at Eqentia.

At its heart, Eqentia is an aggregation platform. It promotes itself as "an aggregator of context, not just content." The way it does this is to add context in the navigation. Each portal has its own taxonomy, which Mougayar described as "a bit like a hierarchical tagging structure." He said that "we basically wrap any content with a semantic wrapper."

How it Works

Under the hood, Eqentia does "content harvesting" from social media sites such as Twitter, blogs and more. Currently Eqentia is getting content from over 13,000 feeds, collecting an estimated 65,000 articles daily.

Eqentia told us that it's indexed 20 million articles so far. The largest topic currently is Outsourcing, with 90,000 articles. Other topics include: Cloud Computing: 60,000; Supply Chain Management: 40,000; Twitter: 20,000; Social Media: 11,000.

Eqentia then does "text mining and filtering" and the results are run through an "Aggregation Engine" (which has rules for sources and filters). Finally there is what Eqentia calls "Semantics Management" - including entity extractions, taxonomy definition, controlled vocabulary.

What The User Gets

Eqentia is starting off with a focus on "professional" content topics. It will target business and technology content, ignoring more mainstream topics like current affairs, sports, entertainment.

Eqentia is launching with 3 products:

1) Out-of-the-box portals. These will give general users free access to topic streams (of which there are 12 at launch, with more coming). There will be email options, widgets and RSS feeds available.

2) Personalized portal. These can be private or public. [note: this is what ReadWriteWeb has signed up for]

3) Enterprise. A SaaS platform that can be customized. A stated use case is for large companies to "disseminate organized news intelligence for their employees across distinct groups or market segments."

Conclusion: Tough Competition, But Important Market

The proof will be in the pudding as to how Eqentia compares to OpenCalais and Evri. We've been very impressed with both OpenCalais and Evri in our previous coverage, so Eqentia has high standards to live up to. In particular Eqentia is going to have to nail the User Experience, because it is relying on its interface a lot to give value to the user.

Finally, Mougayar noted to us that "if web 2.0/social media rewarded the socially savvy user, the semantic web/web 3.0 will reward the research oriented user." It's a nice marketing line, but we are apt to agree that products like Eqentia, OpenCalais and Evri are bringing much needed smarts to the oceans of content in the Web.

NetBase employs the same principals across a variety of enterprise tools, but healthBase is its first foray into consumer-facing products. While the company is used to powering corporate, federal and market research, healthBase allows NetBase to show off its content intelligence tool in a way that gives us insight into our selves and our bodies.

Because NetBase is not reliant on manual annotation or custom taxonomies, the system is also very scalable. It took roughly 2 days to produce all of the data in healthBase - a feat that would never be possible by a combination human and machine system.

"It's important for us to address real issues with semantic technologies outside of a lab," Says Tellefsen. To try healthBase visit healthbase.netbase.com

At first glance Dygest looks like the work of one more computer scientist who's trained a robot to hunt for hot links. There are a lot of those now. This one uses natural language processing algorithms instead of tracking links between sources, though, and the summaries from multiple sources are enjoyable to read. Demo sites have been set up for technology, environmental, film, Chicago-area and San Francisco area news.

The digests aren't perfect but they are surpassingly coherent. The site isn't particularly eye catching at all, but as a proof of concept it's very interesting. You can either get a broad understanding of a story by reading highlighted paragraphs from various sources, or pick one of those sources to dive deep into its particular coverage. The opportunity to sample different writing styles, angles and key points makes Dygest fun to read casually and for in-depth exploration of the news.

The team behind Dygest says they think the service would be well suited for mobile consumption, but white labeling the technology is high on their list of ideas as well. OneSpot comes to mind as an immediate competitor, but there are others.

Dygest has pre-seeded each of its demo sites with a few hundred sources, but the possibilities the analysis engine presents are many.

At first we thought Evri's API would simply rank things as positive or negative, much like the Twitter tracker twendz does today, highlighting positive, negative, and neutral items. However, the sentiment API does so much more, allowing you deeper insight into the "who's," and "what's," and "why's" associated with the particular expression or feeling.

To be more specific, according to the announcement, Evri lets you:

• Find the percentage of positive and negative expressions of sentiment made by an entity, or about an entity. For example, find out what percentage of things being written about the iPhone are positive and which percent are negative.
• Discover who is criticizing and who is praising a particular person, place or thing. For example, see who is criticizing and praising Microsoft right now.
• Read what praisers and critics are saying about an entity. For example, see what the GOP are saying about the Democrats.
• Discover who or what your favorite entity is bashing and why. For example, see who Lance Armstrong is complaining about.
• Discover who or what your favorite entity is praising and why. For example, see who the World Health Organization is commending and why.

When unleashed upon the web as a whole, this could unearth a veritable goldmine of information. Just thinking of how many different ways it could be used is enough to blow anyone's mind. Of course, marketers will be the first to jump on board, looking for practical ways to track the feelings about their companies, clients, and brands and why they're changing, but an engine that understands sentiment could do so much more than just this. It can literally take the pulse of the web the way we take the pulse of Twitter using apps like the above-mentioned twendz to rank trends as positive or negative.

Demo: The "Vibology Meter"

To demonstrate what Evri can do, the company created a widget called the "Vibology Meter." (Sadly, no link is provided). The widget not only ranks the good or bad "vibes" about a particular entity (in the example, Barack Obama), but also explores topics associated with that entity and whether or not the primary entity feels positively or negatively towards them. For example, the widget shows Obama is negative towards the GOP and Rush Limbaugh but feels positive about Michele Obama. (Well, that's good!)

When you click on any one of the associated topics (or click on "anything" to see all topics of either positive or negative slant), you're then presented with a sidebar of information. Here, snippets from articles found on the web display along with a title, link, and timestamp.

Of course, this is just a simple example of the Evri API in action. We're sure the developers out there can think up even better ideas than this.

Challenges Ahead

The challenge now for Evri is to keep expanding its index in order to track more sources to rank. At the moment, the engine doesn't track a large slice of the web the way a typical search engine like Google does - in fact they don't even claim to be a search engine...despite what that "Go to" box on their homepage would have you believe. Instead, Evri looks specifically at the people, places, and things on the web and maps the connections between them.

To determine these connections - and now, the associated sentiments as well - Evri pulls from a limited number of "highly regarded" sources. That means you'll definitely see a site like CNN used to rank a person like Obama, but the myriad of tiny politico blogs will be ignored. That's actually a shame, since delving into this "long tail" of the web could give a better overall picture of how all people really feel, not just the sentiments expressed on high-profile sites written by top bloggers and journalists. Still, we know indexing and parsing this long tail is something that's much easier said than done.

In the end, what Evri's doing, even on this smaller scale, is definitely interesting. We hope to see the new API put to good use in the near future.

Below we briefly review two of these services, Apture and Zemanta's Balloons. Is this kind of product really worth using? Once you add a pop-up of someone's LinkedIn profile next to their name as you type it, you may never want to go back to not having a tool like this at your disposal.

Apture

We started by testing out Apture's WordPress plug-in (on my personal blog) and were very happy with the results. It takes just a few minutes to install, and learning to use it is quite intuitive. We wrote an extensive review of Apture in February.

Pros:

• It's beautiful. From the admin section to the pop-up windows, design has been emphasized at Apture.


• Lots of user control. The amount of control users have over what's included in their pop-ups is amazing. You can choose between assets with a few clicks, or you can pick out start and end timestamps in an embedded YouTube video. The list of options is big and keeps getting bigger, as evidenced by the recent addition of really nice LinkedIn and Twitter profile options.


• You can now include multiple tabs in one link, making it easy to pack a lot of information inside.


• The user experience is solid, and the product is pretty well baked.

Cons:

• Apture is proprietary software offered by one company, unlike Zemanta's standards-based offering, which was built as part of a consortium of developer- and community-minded companies.


• Sometimes it hangs on the UI. We found one bug that the company has since fixed, but pop-up loading is sometimes slower than we'd like.

Zemanta

Zemanta is a feature-rich service for bloggers and has a great API that developers can use to automatically discover keywords in bodies of text in lots of different scenarios. You should check it out. It's quite easy to use. Last week the company released a feature that competes with Apture, called Balloons. Balloons is now automatically included in the blogging plug-in from Zemanta, which is very easy to get started with.

To be frank, we would recommend installing the core Zemanta plug-in for the rest of its features but using Apture for info pop-ups instead. The way the two products are administered is very different; Zemanta detects key concepts in the text of your post and suggests Balloon links you can add with a click. You're limited to adding just those handful of Balloons; you can't link up just any text you want.

Pros:

• Zemanta is open source and standards-based. It feels good to use.


• Zemanta works with the rest of the tech community and has some awesome tools for supporting non-profit organizations. Did we mention that it feels good to use Zemanta?


• The auto-detection of key concepts -- just click on the buttons and they're linked to resources -- makes Zemanta a little bit faster to use than Apture. It takes fewer clicks.

Cons:

• This tool isn't nearly as pretty. In fact, the pop-ups are almost the opposite of pretty.


• You have far less control over the sources of information you can include. Zemanta's Balloons is tied to the ambitious CommonTags standards effort and apparently does not include anything outside the world of standards. That's noble but limiting.


• Most of the links Zemanta inserts are to FreeBase, which is like a machine-readable version of Wikipedia but also a noble, well-funded mess. Thus the pop-ups you get from Zemanta are quite hit and miss.


• There are Amazon affiliate ads in the Zemanta product; Apture's business strategy appears to be to serve bloggers for free and ad-free and charge big publishers to white-label the service. Zemanta's Amazon ads might get on your nerves.


• This is a very early product, having just launched last week. We hope it is further developed.

That's our experience so far with these tools. If you've tried either or both, we'd love to know about your experience as well.

Siri has been preparing for this for some time now. While the product is still more promise than substance, I at least got to look at some real-life iPhone demos in San Jose.

What Users Can Expect (U.S. Ones at Least...)

Last October it raised $8.5 million and in November ReadWriteWeb named it as a Semantic App to watch, purely based on the promise of the app. Siri was spun out of SRI International and its core technology is based on the ambitious CALO artificial intelligence project. Siri is aiming to be a "personalized assistant that learns."

I asked Kittlaus and Gruber what specifically is in store for users of Siri? They told me that Siri will start as a mobile application, but it will become available over IM, email and other communication platforms too. Kittlaus added that you will be able to "access your assistant in the manner you choose." The iPhone app will be the first one launched, at the end of summer, however the pair were button-lipped about which device would be next.

Siri also sees a future for their product in smart devices, for example VCRs - where they could replace the instruction manual. The overall message is that Siri will be "a systems play in everyday systems." Later in the interview, Kittlaus and Gruber described Siri to me as an "intelligent interface".

Siri wants to enable a "person-centric world," rather than the device-centric one we have today. In other words, they want Siri to serve the user and act as an intermediary between the person and the device. In the demos I saw, you could type or speak instructions into an iPhone - e.g. "What is a good movie about to start near my current location?" - and the product would deliver a textual and graphical answer.

Siri vs Chandler

I asked if there are any similarities between Siri and another 'Personal Assistant' product, the Mitch Kapor inspired Chandler - an open source project which re-defined the phrase 'scope creep' for the web 2.0 era (it finally reached version 1.0 status in August 2008, after more than 7 years development).

Kittlaus and Gruber replied that Siri will be very task-focused and will be much more pragmatic than Chandler. They claimed that Siri will "focus on domains we understand," such as movies and restaurants. They also noted that the time and place for a personal assistant product is now, due to rise of the Mobile Internet. So Chandler suffered from bad timing too.

How Will Siri Make Money?

As noted above, one business model for Siri will be partnerships with companies that make personal appliances such as VCRs. What other ways will Siri make money, I asked?

Kittlaus and Gruber told me that they will have affiliate agreements on transactions, for example movie tickets. What's more, their model will be CPA based - where advertisers define what they want Siri to do and pay for the outcomes. An example might be Netflix paying for every new customer.

Browser Technology to the Fore

We spoke a bit about the problems of designing Siri for many different devices. Siri is looking for browser technology to come to their rescue, pointing to WebKit's progress in the mobile browsing world. Siri is also looking for upcoming browsers to integrate functionality such as speech, which Siri will be able to leverage; inputting commands into Siri by voice is expected to be a big use case. As a lot of Siri's processing is done in 'the cloud,' and browser technology will be an important factor in the success or otherwise of Siri.

I think this is a pretty good bet by Siri, given all of the competition and advances in the browser space right now - check out what IBM is doing in a browser-related project called Blue Spruce, as just one example.

Mobile a Big Part of Siri's Vision

The demographic that Siri is targeting initially, 18-35 urban and social, are early adopters of the iPhone and mobile in general. This user base is also a multi-tasking one. That again plays to Siri's goal to deliver personalized information in context, which these days usually means on a mobile device.

The proof - as always - will be in the pudding when Siri is launched at end of the summer to U.S. users. We think it's a great shame that international users won't be able to enjoy this product too, until probably 2011. In any case, you can see Tom Gruber give a demo of Siri in this SemTech video (slides here):

In Part 2 we discuss: how previously reticent search engines like Google and Yahoo have begun to participate in the Semantic Web in 2009, user interfaces for browsing and using data, what Tim Berners-Lee thinks of new computational engine Wolfram Alpha, how e-commerce vendors are moving into the Linked Data world, and finally how the Internet of Things intersects with the Semantic Web.

RWW: You've been talking about the Semantic Web for many years now. Generally the view is that Semantic Web is great in theory, but we're still not seeing a large number of commercial web apps that use RDF (we've seen a number of scientific or academic ones). However we have begun to see some traction with RDFa (embedding RDF metadata into XHTML Web content), for example Google's Rich Snippets and Yahoo's SearchMonkey. Has the takeup of RDFa taken you by surprise?

TBL: Not really, but the takeup by the search engines is interesting. In a way I was happy to see that, it was a milestone for those things to come out of the search engines. The search engines had typically not been keen on the Semantic Web - maybe you could argue that their business is making order out of chaos, and they're actually happy with the chaos. And if you provide them with the order, they don't immediately see the use of it.

Also I think there was misunderstanding in the search engine industry that the Semantic Web meant metadata, and metadata meant keywords, and keywords don't work because people lie. Because traditionally in information retrieval systems, keywords haven't proven up to the task of finding stuff on the Web. One of the reasons is that people lie, the other is that they can't be bothered to enter keywords. So keywords have gotten a bad reputation, then metadata in general was tarred with this 'keywords don't work' brush. Because a lot of Semantic Web data included metadata, then people thought that with Semantic Web data -- again, that people will lie and won't have the time to produce it.

Google rich snippets example; image credit: Matt Cutts

Now I think there's a realization that when you're putting data online, that people are motivated NOT to lie. For example when your band is going to produce its next album, or when your band is going to play next downtown, you're motivated to put that information up there on the Semantic Web. There's an awful lot of cases when actually data is really important to people; and it's on the web anyway. So I think it's great that some of the search engine companies are starting to read RDFa.

Does this mean that they [search engines] will start to absorb the whole RDF data model? If they do, then they will be able to start pulling all of the linked data cloud in.

Will they know what to do with it? Because when it's data in a very organized form, I think some people have been misunderstanding the Semantic Web as being something that tries to make a better search engine - i.e. when you type something into a little box. But of course the great thing about the Semantic Web is that you can query it, you can ask a complicated query of the Semantic Web, like a SQL query (we call it a SPARQL query), and that's such a different thing to be able to do. It really doesn't compare to a search engine.

You've got search for text phrases on one side (which is a useful tool) and querying of the data on the other. I think that those things will connect together a lot.

So I think people will search using a search text engine, and find a webpage. On the front of the webpage they'll find a link to some data, then they'll browse with a data browser, then they'll find a pattern which is really interesting, then they'll make their data system go and find all the things which are like that pattern (which is actually doing a query, but they'll not realize it), then they'll be in data mode with tables and doing statistical analysis, and in that statistical analysis they'll find an interesting object which has a home page, and they'll click on that, and go to a homepage and be back on the Web again.

So the web of linked data and the web of documents actually connect in both directions, with links.

User Interfaces for Semantic Content

RWW: At the recent SemTech conference, Tom Tague of Thomson Reuters' Calais project suggested that user interfaces for semantic content are key in getting more take-up. With that in mind, I wonder if you've seen some great interfaces or designs for semantic applications in recent months - if so which ones and why did they impress you?

TBL: I think that whole area is very exciting at the moment. The only piece of hacking I've done over the past few years has been on a thing called the Tabulator [a data browser and editor], which is addressing exactly that. Partly because I wanted to be able to look at this data. And now there are lots of different ways that people need to be able to look at data. You need to be able to browse through it piece by piece, exploring the world of data. You need to be able to look for patterns of particular things that have happened. Because this is data, we need to be able to use all of the power that traditionally we've used for data. When I've pulled in my chosen data set, using a query, I want to be able to do [things like] maps, graphs, analysis, and statistical stuff.

W3C Tabulator, a data browser/editor; Image credit: wiwiss.fu-berlin.de

So when you talk about user interfaces for this, it's really very very broad. Yes I think it's important. There's also the distinction we can make between the generic interfaces and the specific interfaces.

There will always be specific interfaces; for example if you're looking at calendar data, there's nothing else like a calendar that understands weeks, months and years. If you're looking at a genome, it's good to have a genetics-specific user interface.

However you also need to be able to connect that data, through generic interfaces. So if my genome data was taken during an experiment which happened over a particular period, I need to be able to look at that in the calendar - so I can connect the genetics to the calendar.

So one of the things I hope to see is domain-specific things for various different domains, and the generic user interfaces. And hopefully the generic interfaces will be able to tie together all of the domains.

Next Page: Wolfram Alpha; e-Commerce and Linked Data

Wolfram Alpha and Natural Language Interfaces

RWW: An interesting new product was launched this year called Wolfram|Alpha, described as a 'computational knowledge engine.' It's kind of a mix between Google (search) and Wikipedia (knowledge), and its key attribute is that enables you to compute something. The founders think that 'computing' things on the fly is something we're going to see a lot of in future. What's your take on Wolfram|Alpha?

TBL: There are two parts to that sort of technology. One of them is a sort of stilted natural language interface. We've seen those sort of natural language queries for years. Boris Katz [from W3C] created a system called START [a software system designed to answer questions that are posed to it in natural language]. I think with the Semantic Web out there, those sorts of interfaces are going to become important, very valuable, because people will be able to ask more complicated things. The search engine has traditionally been limited to just a phrase, but some of the search engines are now starting to realize that if they put data behind them and have computation engines, then you can ask things like 'what's this many pounds in dollars?' and so on. So yes, those interfaces will become important.

Conversational interfaces have always been a really interesting avenue. We've had voice browser work in W3C, that has been an interesting alternative avenue. It's possible that as compute power goes up, we'll see a prolifieration of machines capable of doing voice. It'll move from the mainframe to being able to run on a laptop or your phone. As that happens, we'll get actual voice recognition and pattern natural language at the front end. That will perhaps be an important part of the Semantic Web.

We talked before about what a great challenge the Semantic Web is going to be from a user interface point of view. Conversational interfaces are going to be part of [solving] that. Of course it's also going to be really valuable to have compositional interfaces - for the visually impaired and so on.

Wolfram|Alpha is also a large curated database of data sets. Obviously I'm interested in the big data set which is out there, which is Linked Data. This everybody can connect to. I don't really know a lot about the internals of Wolfram|Alpha's data set. I don't know whether they're likely to put any of it out on the web as Linked Data - that might be an interesting addition. I imagine that quite a lot of it may have come from the web of Linked Data.

e-Commerce and Linked Data

RWW: There have been reports recently that both Google and Yahoo will be supporting the Good Relations ontology and linked data for e-commerce. Companies such as Best Buy are already putting out product information in RDFa. What would be your advice to e-commerce vendors right now, to help them transition to this world of structured data on the Web. The same question could be asked across many verticals, but e-commerce seems like one area which has some momentum right now. Would you advise them just to put out their data as Linked Data?

TBL: Yup! Certainly this year is the year to do it. I've been advising governments to do it and when you look at an enterprise, you find that a lot of the issues are the same. But when you put your data from government or enterprise out there, make sure you don't disturb existing ecosystems. Don't threaten those systems, because you've spent years building them up.

Maybe there's an analogy with when the Web first started and the first bookshops went online. They were more or less a flyer, saying 'hey we have a great bookshop at 23 Main St, come on down!'. Let's say that a person named Joe owned one of these early online bookshops. If somebody had suggested to Joe that he should put his catalog online, Joe would've felt that that was very proprietary data. And he'd be worried that other bookshops would see where he was weak, so they'd be able to advertise themselves as filling that niche he's weak in.

But when his competitors Fred and Albert put their catalogs online, then Joe can check which books people are browsing at Fred and Albert's websites. So Joe would [finally] be pursuaded to put his book catalog up online. But he doesn't put up the prices... until Albert and/or Fred does. And even if catalog and pricing is up there, nobody puts their stock levels online. And there was a period of time when nobody [i.e. online booksellers] had their stock levels up. But people got fed up with ordering stuff that wasn't in stock. So the first book shop to actually tell you about stock levels suddenly was then unbelievably attractive to its customers.

So there's this syndrome of progressive competitive disclosure. This happens when people realize that if you're going to do business with somebody, if you're going to have your partners up and down the supply chain, really it's useful to check the data web - and life goes much more quickly and open.

Best Buy may be what starts the ball rolling [among e-commerce vendors]. Now if I want to look out for what [products are] available, I can write a program to see what there is. If somebody wants to compete with Best Buy, to my program they'll be invisible unless they can get their data up in RDF. Doesn't matter whether they use RDFa or RDF XML, as long as it maps in a standard fashion to the RDF model, then they will be visible.

Next Page: Internet of Things; Conclusion

The Internet of Things

RWW: I'm fascinated by how the Internet is becoming more and more integrated into the real world. For example the Internet of Things, where everyday objects become Internet connected via sensors. Have you been following this trend closely too, and if so what impact do you think this will have on the Web in say 5 years time?

TBL: It connects very much with Semantic Web [and] with linked data. With Linked Data you've got the ability to give a thing a URI. So I can give a URI to my phone, and I can say that's my phone in Linked Data. And also the company that made it can give a URI to the model of the phone. They can also put online all the specs of the phone, and then I can make a link to say that my phone is an example of that product. So now any system which is dealing with me and has access to that data will be able to figure out the sorts of things I can do with my phone, which actually is really valuable. Especially if the phone breaks.

The Semantic Web has already given URIs to things, and to types of things. When the things themselves have an RFID chip in them, then I think it's a very exciting world. One can take that RFID chip, go to the Internet and find out the data about the thing. Whether we'll be able to do that, whether the manufacturers will be open enough to allow me to turn data about the identifier of the thing into data about the thing, is yet to be seen. But it's a very exciting idea.

Pachube, an example of the Internet of Things (see ReadWriteWeb profile)

Similarly, I'd like to be able to scan a barcode and get back nutritional information about what's in - for example - a can of food. But we don't have that yet. To get that sort of thing, which is very powerful, we need to build look-up systems, which allow you to translate an RFID code or a barcode into an HTTP address.

The Semantic Web is a web of things, conceptually. Tying an actual thing down to a part of the web is the last link - the last mile. Give the thing a notion of its own identity in the web.

Conclusion

RWW: The over-riding message in both Part 1 and 2 of our interview with Tim Berners-Lee, is for companies and organizations to make their data available online. Preferably as Linked Data, which uses a subset of Semantic Web technologies. But Berners-Lee noted, in Part 1 of our interview, that he'd even be happy with the data in CSV (comma separated values) format.

It's clear that we've seen a lot of progress in linked data already in 2009. In upcoming posts on ReadWriteWeb, we'll continue to track this trend and explain how organizations can contribute their data.

This was my first meeting with the Web's creator, whose work and philosophy was a direct inspiration for me when I launched ReadWriteWeb back in 2003.¹

Note: the interview will be published in two parts, with Part 1 today on the topic of Linked Data. Part 2 will explore other topics and will run tomorrow.

UPDATE: Part 2 of this interview is now available.

How Linked Data Relates to The Semantic Web

RWW: Earlier this year you gave an inspiring talk at TED about Linked Data. You described Linked Data as a sea change akin to the invention of the WWW itself - i.e. we've gone from a Web of documents to a Web of data. Can you please explain though how Linked Data relates to the Semantic Web, is it a subset of it?

TBL: They fit in completely, in that the linked data actually uses a small slice of all the various technologies that people have put together and standardized for the Semantic Web.

We started off with the Semantic Web roadmap, which had lots of languages that we wanted to create. [However] the community as a whole got a bit distracted from the idea that actually the most important piece is the interoperability of the data. The fact that things are identified with URIs is the key thing.

The Semantic Web and Linked Data connect because when we've got this web of linked data, there are already lots of technologies which exist to do fancy things with it. But it's time now to concentrate on getting the web of linked data out there.

Web inventor Tim Berners-Lee and ReadWriteWeb founder Richard MacManus

How Linked Data Has Evolved via Grassroots

RWW: Linked Data has had a lot of grassroots support, which you mentioned in your TED speech. This is something Semantic Web technologies, such as RDF, have struggled to get over the years. Has the W3C been pushing the more bottom-up Linked Data world, because of the frustration over lack of take-up of top-down Semantic Web?

TBL: A lot of the initial RDF and OWL projects came out of the academic world; and some of them were projects to show what you could do in a closed world. And the files were zipped up and left on a disc. While they were interesting projects, and while the systems were useful systems, the semantic web community maybe missed the point of the 'web' bit and focused too much on the 'semantic'. However the work that's been done in the Semantic Web, the standards, was really valuable. It's relatively recently for example that SPARQL [an RDF query language] has been developed.

Somebody drew an analogy the other day: can you imagine trying to promote a world of databases without SQL? Even though it's not an interoperable protocol, it's just a query language. So similarly, all that's been put into RDF, rdfs and OWL is very valuable to the linked data community.

The Linked Data community tend to use a subset of that [Semantic Web technologies], of OWL for example. But they certainly use SPARQL. So you could argue that really it wasn't ready to be deployed widely.

Linked Data started as a very informal Design Issues note that I put in; it was a grassroots movement from very early on. So yes W3C has been emphasizing the importance of Linked Data. It's been the Semantic Web Interest Group of course, and various [other Semantic Web] activities, which has been pushing it. But also Linked Data has been seized on - a group of people for example put together DBpedia.³ That wasn't commissioned, that was that they just thought it would be a really cool idea.

Graph of Linked Data sets on the Web, as at March 2009

Linked Data and Governments

RWW: In a recent Design Issues note, you urge governments to put their data online as Linked Data (although you'd also be happy for governments to just make available the raw data - presumably so that others can then structure it). What do you realistically expect, for example, the U.S. or U.K. governments to do over the next year? And in the near future, do you foresee different governments interconnecting their Linked Data sets?

TBL: One can't generalize, governments are (like most big organizations) fascinatingly diverse inside them. So you'll find that there are places inside governments where you get a champion who gets linked data and who's just written a script and produced some linked data. So in the UK government for example, you'll find there's RDFa [in the code of its website] for civil service jobs. So if somebody wants to make a database of all the jobs, they can do that very easily.

There are other cases where the easiest thing for somebody to do is to just put data up in whatever form it's available. Comma separated values (CSV) files are remarkably popular. They're exported sometimes from spreadsheets. It's remarkable how much information is in spreadsheets. Or sometimes pulled out of a database and then put up on the web. It's not as good, not as useful to the community, as if Linked Data had been put up there and linked. But the first step of actually putting the data out there is the one that nobody else can do.

Data.gov, a catalog of public data, was launched in May by the U.S. government

The way to go is for government departments to go the extra step and convert [their data] into Linked Data. One of the nice things about Linked Data, when they have a pile of it, is that they could run a SPARQL server on it. SPARQL servers are a commodity product, a solution for all of the people who say 'but actually I wanted to have XML.' A SPARQL server will generate an XML file [and] allow somebody to write out, effectively, a URL for the XML file.

In fact, I don't see why SPARQL servers shouldn't provide CSV files, something which as far as I know isn't in the standards. But I'd recommend it, certainly in government context, because CSV files are what people have and what people want.

So the message [for government] is to use RDF. Linked Data is the backplane, it's the thing that you connect to in both directions. As a [web] producer your job is to make sure that you produce Linked Data one way or another. And as a consumer, there are lots of ways to consume that data once it's out there as Linked Data.

In Part 2 of this interview we discuss: how previously reticent search engines like Google and Yahoo have begun to participate in the Semantic Web in 2009, user interfaces for browsing and using data, what Tim Berners-Lee thinks of new computational engine Wolfram Alpha, how e-commerce vendors are moving into the Linked Data world, and finally how the Internet of Things intersects with the Semantic Web. Read Part 2 here.

Footnotes:

1. The very first sentence written on this blog, on 20 April, 2003, was: "The World Wide Web in 2003 is beginning to fulfill the hopes that Tim Berners-Lee had for it over 10 years ago when he created it."

2. For more on read/write browsers, you can read another early RWW post entitled What became of the Browser/Editor.

3. DBpedia is a community project to extract structured information from Wikipedia; see ReadWriteWeb's profile of this and similar resources.

Today, Faviki is releasing an upgrade to their service which will give you even better control over the tagging process, making bookmarking even easier than before. They're also announcing support for OpenID.

The biggest upgrade today is Faviki's enhanced tagging interface. In the past, Faviki struggled with some of the tag suggestions pulled out of Wikipedia because they were too long and too hard to enter for practical use. Plus, users wanted to use tags of their own creation, not the tag suggestions.

For example, if someone is tagging an article about the soccer player "Filippo Inzaghi," they may want to tag it by the player's nickname "Pippo." Before, this was not possible. But now, if Faviki doesn't understand a tag, it will pull in possible matches and ask you "What exactly do you mean by ______?" After you pick your selection, Faviki will remember your choice.

This is an important change for the service because it means users can tag web pages any which way they want, but they're still linked to the structured data on the back-end. That way, when someone searches through Faviki's community tags, all the web pages for that particular item or concept will appear, even if people tagged them using their own personal keywords.

Beyond Wikipedia

Another change in Faviki's service is the ability to define new tags. Prior to today, the service was limited to searching Wikipedia for tag suggestions, but now it has the whole web at its fingertips. If a tag is entered which doesn't match anything from Wikipedia, Faviki will search Google for relevant URLs and then ask if the links presented represent the same tag. As multiple users go through this process, Faviki learns what URLs best represent that concept and adds the new tags created by the users to its database.

API, OpenID, and More

Faviki has also just launched a Save/Edit API that provides a way to save and edit bookmarks from other applications. In addition, they've introduced support for OpenID. Other new features arriving today include a smarter autocomplete list, the ability to convert tags, spam control, the ability to export/backup your bookmarks, and a new tag description tooltip.

The only issue we have with Faviki is the same one we had before: there's still no import function available. That means you'll have to leave your extensive bookmark collection behind if you want to use this service. We suppose that it could be difficult to properly tag and match all of our old bookmarks, but without this feature, Faviki doesn't have the best shot at attracting the heaviest users of social bookmarking services.

2. Cuil

3. Hakia

4. Worio

5. Ubiquity

Ubiquity for Firefox from Aza Raskin on Vimeo.

6. Semanti

If you've got more examples of semantic data collection tools, list them in the comments below.