We've been given screencasts of the new version of this knowledge management application - screencasts of both the consumer- and developer-facing facets of the site. Take a look, and let us know if the new Twine lives up to expectations. This new version, we are told, will be live by the end of the year.

Developers and other techies can check out this screencast exploring Twine's collaboratively authored ontologies:

The Twine folks see the new version as a realization of Tim Berners-Lee's vision of the semantic web. So what do ReadWriteWeb readers think; is the new Twine worth the wait? Does it live up to the hype? Leave your expert comments below.

The latest critique is a scathing post by Semantic Web consultant Greg Boutin, entitled Twine in Freefall?. Boutin argues that Twine's traffic has taken a dive recently. We followed up with Twine founder Nova Spivack for his response. He admits that traffic has declined "20-25%," but says that Twine is focusing on an all-new version of its product.

The premise of Greg Boutin's post is that Twine had earlier in the year trumpeted passing Delicious and Friendfeed in traffic - ReadWriteWeb was the first to cover the Delicious trend, back in March. However now Boutin points to statistics from Compete, Alexa and Quancast showing a marked drop in traffic. Compete was the source we used in our March post, so below we've pasted comparison charts from then and now:

Compete chart from March '09 showing that Twine was trending upwards, while Delicious growth appeared to have tapered off.

Compete chart showing that Twine did indeed pass Delicious in March '09, however over June-July Twine's traffic has plummeted but Delicious held steady.

Twine: Yes Traffic Has Dropped, But We're Focusing on Version 2...

ReadWriteWeb questioned Twine about these statistics and the company admits that "our internal data shows us down 20-25%."

Twine appears to put some of this down to problems with version 1 of its product. The company told us that it is putting all of its focus and marketing efforts into a brand new version (more on that below). Therefore the drop in traffic is something Twine and its investors are comfortable with, for now.

Nova Spivack also told us that Twine had indexing problems with Google over the summer. If this was the case, that may be a big reason for the decline in Compete. Spivack explained that "we have about 500K pages that should be indexed. They [Google] are only indexing 140K pages, but we're basically not worrying about it, since T2 [version 2, see below for details] will change the game and the way we deal with Google anyway. . ."

Twine 2.0: Make or Break

Twine says that it is busy working on a new version of its product, which is why it hasn't been active on the PR front in the last few months. The company is hoping the new version gets its momentum back.

Nova Spivack elaborated on the new version to ReadWriteWeb, which we'll quote in its entirety because it illuminates Spivack and company's marketing approach:

"In the last 9 months we have made a breakthrough with the new version of Twine that changes the economics of vertical search and navigation on the Web. This new technology enables Twine to provide Web-scale faceted navigation and search across numerous vertical search categories. We are able to index structured data (like recipes, products, reviews, or any kind of database driven or XML content) with search-engine performance and scalability. This is a huge leap beyond what we were able to deliver in the first version of Twine."

"As a result of this breakthrough, we have made a strategic decision to focus all of our resources on bringing Twine version 2 (T2) to market by the end of the year. Version 1 of Twine will remain online until we are able to cutover to version 2. We are doing no further work on version 1 and no marketing for it, either. We are of course still supporting it from a technical and user perspective, however. But all our focus is on T2 moving forward."

Twine also told us that it has signed deals with nearly a dozen major content providers and brands to integrate the new search capabilities into their online services. See our March post for more context about the new version.

So what do we think about this latest twist to the Twine saga? With language like "breakthrough," "changes the economics" and "huge leap beyond," once again Twine is hyping itself up. While there have always been signs that Twine is at least partly delivering on its clear promise, the fact is that Twine continues to struggle to deliver a product the market wants. This accounts for its inconsistent growth and much of the criticism of usability which Twine has endured.

We continue to cheer for Twine, but it does seem that version 2 is make or break for the company.

What blogging was to publishing, a simple tool that made far more people able to participate, Twine's new ontology writing and hosting service could be to the act of teaching machines about new topics.

Twine Could Surpass Delicious in a Matter of Months

Twine's public product lets people bookmark items like web pages and videos into topical collections. The service then analyzes the contents of all the bookmarks to identify the key concepts, people, places and other information automatically. It's like tagging in Delicious but automated and, in theory, more thorough than any human being would be in assigning tags.

Compete.com says Delicious gets about 2 million unique visitors a month and has stopped growing. Twine just passed 1 million uniques and is growing fast. Spivack said that 40% of that traffic comes from Google, and sure enough those Twine pages look awfully juicy from a spider's perspective. Spivack expects Twine to hit 2 million uniques in a matter of months and that looks like a credible claim to us.

The number of saved items is far greater in Delicious than in Twine - about 150 million vs. 3 million. Spivack says though that the company will soon turn back on its system that crawls all the links on bookmarked pages. Those linked-to pages will be automatically bookmarked and analyzed too, quickly expanding Twine's total archives.

So by this summer, Twine could be bigger and more visited than Delicious. We wrote a scathing review of the Twine user experience when the long-awaited service began to launch last year. The site has changed a lot since then and we're excited about the company's plans for the future. We are still concerned about the company's ability to make its interfaces really usable -- but if they can, then look out, internet.

Twine and the Semantic Web

The semantic web is a paradigm that adds standardized, structured markup to web content so that savvy applications can comprehend the key topics of any web page. Publishers can do that when they publish, or services like Twine can create the semantic markup from the outside. The automatic tagging Twine does is actually semantic markup.

For example, you can't ask Google today to show you all the book reviews around the web that were written by friends of yours who live in New York - but semantic search engines could make such a query trivial and use that information as the ground level for building more sophisticated features on top. It's a form of standardized metadata. It turns free text into data that can be mashed up.

Semantics Plus Ontology Equals Meaning

Spivack says that his existing product, Twine, is just one of a number of applications that only extract key concepts (people, places, key terms) out of a web page. Placing those concepts in context is the next step.

Twine can tell you that a web page is about goat cheese, for example, but it doesn't yet know how to infer that the page is also about a dairy product - the larger category that is not explicitly stated in the article. An ontology is that context, be it a dairy ontology, a cheese ontology or a new node in the existing accepted ontology of food.

Those new ontologies can be created using Spivack's simple, open source authoring tool and then hosted on his open source community site for ontologies. It's open source authoring like Wordpress and code hosting and discussion like Sourceforge.

Either Twine or a third party will then combine the extracted "entities" (people, places, key terms) with an appropriate ontology and that company's "inference engine" to build a full picture of what a web page is about and where it stands in relation to everything else.

Busting Out of the Tech Ghetto

The limited number of ontologies that have been authored to date are largely centered on technology topics. An easy ontology authoring tool could change that radically. A standardized, accessible ontology can shine a light on a whole new part of the world. Once that topic has been illuminated for the eyes of a semantics reading machine, web developers can build services that intelligently make use of the new information.

Spivack says that heavy-duty ontologies that require computationally intensive logic navigation will still need to be built using heavy-duty desktop apps. But web applications that just need data served up smartly will work well with the kinds of ontologies that can be written with Spivack's new authoring tool.

Ready for the whole, diverse internet to be contextually understandable by web applications? Ready to contribute to the creation of those contextual explanations yourself? Keep your eye on Nova Spivack because that's what he's aiming to make happen.

In a recent interview Tim Berners-Lee pointed out that the infrastructure to power the Semantic Web is already here. ReadWriteWeb's founder, Richard MacManus, even picked it to be the number one trend in 2008. And rightly so. Not only are the bits of infrastructure now in place, but we are also seeing startups and larger corporations working hard to deliver end user value on top of this sophisticated set of technologies.

The Semantic Web means many things to different people, because there are a lot of pieces to it. To some, the Semantic Web is the web of data, where information is represented in RDF and OWL. Some people replace RDF with Microformats. Others think that the Semantic Web is about web services, while for many it is about artificial intelligence - computer programs solving complex optimization problems that are out of our reach. And business people always redefine the problem in terms of end user value, saying that whatever it is, it needs to have simple and tangible applications for consumers and enterprises.

The disagreement is not accidental, because the technology and concepts are broad. Much is possible and much is to be imagined.

1. Bottom-Up and Top-Down

We have written a lot about the different approaches to the Semantic Web - the classic bottom-up approach and the new top-down one. The bottom-up approach is focused on annotating information in pages, using RDF, so that it is machine readable. The top-down approach is focused on leveraging information in existing web pages, as is, to derive meaning automatically. Both approaches are making good progress.

A big win for the bottom-up approach was recent announcement from Yahoo! that their search engine is going to support RDF and microformats. This is a win-win-win for publishers, for Yahoo!, and for customers - publishers now have an incentive to annotate information because Yahoo! Search will be taking advantage of it, and users will then see better, more precise results.

Another recent win for the bottom-up approach was the announcement of the Semantify web service from Dapper (previous coverage). This offering will enable publishers to add semantic annotations to existing web pages. The more tools like Semantify that pop up, the easier it will be for publishers to annotate pages. Automatic annotation tools combined with the incentive to annotate the pages is going to make the bottom-up approach more compelling.

But even if the tools and incentive exist, to make the bottom-up approach widespread is difficult. Today, the magic of Google is that it can understand information as is, without asking people to fully comply with W3C standards of SEO optimization techniques. Similarly, top-down semantic tools are focused on dealing with imperfections in existing information. Among them are the natural language processing tools that do entity extraction - such as the Calais and TextWise APIs that recognize people, companies, places, etc. in documents; vertical search engines, like ZoomInfo and Spock, which mine the web for people; technologies like Dapper and BlueOrganizer, which recognize objects in web pages; and Yahoo! Shortcuts, Snap and SmartLinks, which recognize objects in text and links.

[Disclosure: Alex Iskold is founder and CEO of AdaptiveBlue, which makes BlueOrganizer and SmartLinks.]

Top-down technologies are racing forward despite imperfect information. And, of course, they benefit from the bottom-up annotations as well. The more annotations there are, the more precise top-down technologies will get - because they will be able to take advantage of structured information as well.

2. Annotation Technologies: RDF, Microformats, and Meta Headers

Within the bottom-up approach to annotation of data, there are several choices for annotation. They are not equally powerful, and in fact each approach is a trade off between simplicity and completeness. The most comprehensive approach is RDF - a powerful, graph-based language for declaring things, and attributes and relationships between things. In a simplistic way, one can think of RDF as the language that allows expressing truths like: Alex IS human (type expression), Alex HAS a brain (attribute expression), and Alex IS the father of Alice, Lilly, and Sofia (relationship expression). RDF is powerful, but because it is highly recursive, precise, and mathematically sound, it is also complex.

At present, most use of RDF is for interoperability. For example, the medical community uses RDF to describe genomic databases. Because the information is normalized, the databases that were previously silos can now be queried together and correlated. In general, in addition to semantic soundness, the major benefit of RDF is interoperability and standardization, particularly for enterprises, as we will discuss below.

Microformats offer a simpler approach by adding semantics to existing HTML documents using specific CSS styles. The metadata is compact and is embedded inside the actual HTML. Popular microformats are hCard, which describes personal and company contact information, hReview, which adds meta information to review pages, and hCalendar, which is used to describe events.

Microformats are gaining popularity because of their simplicity, but they are still quite limiting. There is no way to describe type hierarchies, which the classic semantic community would say is critical. The other issue is that microformats are somewhat cryptic, because the focus is to keep the annotations to a minimum. This, in turn, brings up another question of whether embedding metadata into the view (HTML) is a good idea. The question is: what happens if the underlying data changes when someone makes a copy of the HTML document? Nevertheless, despite these issues, microformats are gaining popularity because they are simple. Microformats are currently used by Flickr, Eventful, and LinkedIn; and many other companies are looking to adopt microformats, particularly because of the recent Yahoo! announcement.

An even simpler approach is to put meta data into the meta headers. This approach has been around for a while and it is a shame that it has not been widely adopted. As an example, the New York Times recently launched extended annotations for its news pages. The benefit of this approach is that it works great for pages that are focused on a topic or a thing. For example, a news page can be described with a set of keywords, geo location, date, time, people, and categories. Another example would be for book pages. O'Reilly.com has been putting book information into the meta headers, describing the author, ISBN, and category of the book.

Despite the fact that all these approaches are different, they are also somewhat complementary; and each of them is helpful. The more annotations there are in web pages, the more standards are implemented, and the more discoverable and powerful the information becomes.

3. Consumer and Enterprise

Yet another dimension of the conversation about the Semantic Web is the focus on consumer and enterprise applications. In the consumer arena we have been looking for a Killer App - something that delivers tangible and simple consumer value. People simply do not care that a product is built on the Semantic Web; all they are looking for is utility and usefulness.

Up until recently, the challenge has been that the Semantic Web focused on rather academic issues - like annotating information to make it machine-readable. The promise was that once the information is annotated and the web becomes one big giant RDF database, then exciting consumer applications would come. The skeptics, however, have been pointing out that first there needs to be a compelling use case.

Some consumer applications based on the Semantic Web: generic and vertical search, contextual shortcuts and previews, personal information management systems, semantic browsing tools. All of these applications are in their early days and have a long way to go before being truly compelling for the average web user. Still, even if these applications succeed, consumers will not be interested in knowing about the underlying technology - so there is really no marketing play for the Semantic Web in the consumer space.

Enterprises are a different story for a couple of reasons. First, enterprises are much more used to techno speak. To them utilizing semantic technologies translates into being intelligent and that, in turn, is good marketing. 'Our products are better and smarter because we use the Semantic Web' sounds like a good value proposition for the enterprise.

But even above the marketing speak, RDF solves a problem of data interoperability and standards. This "Tower of Babel" situation has been in existence since the early days of software. Forget semantics; just a standard protocol, a standard way to pass around information between two programs, is hugely valuable in the enterprise.

RDF offers a way to communicate using XML-based language, which on top of it has sound mathematical elements to enable semantics. This sounds great, and even the complexity of RDF is not going to stop enterprises from using it. However, there is another problem that might stop it - scalability. Unlike relational databases, which have been around for ages and have been optimized and tuned, XML-based databases are still not widespread. In general, the problem is in the scale and querying capabilities. Like object-oriented database technologies of the late '90s, XML-based databases hold a lot of promise, but we have yet to see them in action in a big way.

4. Semantic APIs

With the rise of Semantic Web applications, we are also seeing the rise of Semantic APIs. In general, these web services take as an input unstructured information and find entities and relationships. One way to think of these services is mini natural language processing tools, which are only concerned with a subset of the language.

The first example is the Open Calais API from Reuters that we have covered in two articles here and here. This service accepts raw text and returns information about people, places, and companies found in the document. The output not only returns the list of found matches, but also specifies places in the document where the information is found. Behind Calais is a powerful natural language processing technology developed by Clear Forest (now owned by Reuters), which relies on algorithms and databases to extract entities out of text. According to Reuters, Calais is extensible, and it is just a matter of time before new entities will be added.

Another example is the SemanticHacker API from TextWise, which is offering a one million dollar prize for the best commercial semantic web application developed on top of it. This API classifies information in documents into categories called semantic signatures. Given a document, it outputs entities or topics that the document is about. It is kind of like Calais, but also delivers a topical hierarchy, where the actual objects are leafs.

Another semantic API is offered by Dapper - a web service which facilitates the extraction of structure from unstructured HTML pages. Dapper works by enabling users to define attributes of an object based on the bits of the page. For example, a book publisher might define where the information about author, ISBN and number of pages is on a typical book page and the Dapper application would then create a recognizer for any page on the publisher site and enable access to it via REST API.

While this seems backwards from an engineering point of view, Dapper's technology is remarkably useful in the real world. In a typical scenario, for websites that do not have clean APIs to access their information, even non-technical people can build an API in minutes with Dapper. This is a powerful way of quickly turning websites into web services.

5. Search Technologies

Perhaps the first significant blow to the Semantic Web has been the inability thus far to improve search. The premise that a semantic understanding of pages leads to vastly better search has yet to be validated. The two main contenders, Hakia and PowerSet, have made some progress, but not enough. The problem is that Google's algorithm, which is based on statistical analysis, deals just fine with semantic entities like people, cities, and companies. When asked What is the capital of France? Google returns a good enough answer.

There is a growing realization that marginal improvement in search might not be enough to beat Google or to declare search the killer app for the Semantic Web. Likely, understanding semantics is helpful but not sufficient to build a better search engine. A combination of semantics, innovative presentation, and memory of who the user is, will be necessary to power the next generation search experience.

Alternative approaches also attempt to overlay semantics on top of the search results. Even Google ventures into verticals by partitioning the results into different categories. The consumer can then decide which type of answer they are interested in.

Yet search is a game that is far from won and a lot of semantic companies are really trying to raise the bar. There may be another twist to the whole search play - contextual technologies, as well as semantic databases, could lead to qualitatively better results. And so we turn to these next.

6. Contextual Technologies

We are seeing an increasing number of contextual tools entering the consumer market. Contextual navigation does not just improve search, but rather shortcuts it. Applications like Snap or Yahoo! Shortcuts, and SmartLinks "understand" the objects inside text and links and bring relevant information right into the user's context. The result is that the user does not need to search at all.

Thinking about this more deeply, one realizes that contextual tools leverage semantics in a much more interesting way. Instead of trying to parse what a user types into the search box, contextual technologies rely on analyzing the content. So the meaning is derived in a much more precise way - or rather, there is less guessing. The contextual tools then offer the users relevant choices, each of which leads to a correct result. This is fundamentally different from trying to pull the right results from a myriad of possible choices resulting from a web search.

We are also seeing an increasing number of contextual technologies make their way into the browser. Top-down semantic technologies need to work without publishers doing anything; and so to infer context, contextual technologies integrate into the browser. Firefox's recommended extensions page features a number of contextual browsing solutions - Interclue, ThumbStrips, Cooliris, and BlueOrganizer (from my own company).

The common theme among these tools is the recognition of information and the creation of specific micro contexts for the users to interact with that information.

7. Semantic Databases

Semantic databases are another breed of semantic applications focused on annotating web information to be more structured. Twine, a product of Radar Networks and currently in private beta, focuses on building a personal knowledge base. Twine works by absorbing unstructured content in various forms and building a personal database of people, companies, things, locations, etc. The content is sent to Twine via a bookmarklet, via email, or manually. The technology needs to evolve more, but one can see how such databases can be useful once the kinks are worked out. One of the very powerful applications that could be built on top of Twine, for example, is personalized search - a way to filter the results of any search engine based on a particular individual.

It is worth noting that Radar Networks has spent a lot of time getting the infrastructure right. The underlying representation is RDF and is ready to be consumed by other semantic web services. But a big chunk of the core algorithms, the ones that are dealing with entity extraction, are being commoditized by Semantic Web APIs. Reuters offers this as an API call, for example, and so moving forward, Twine won't need to be concerned with how to do that.

Another big player in the semantic databases space is a company called Metaweb, which created Freebase. In its present form, Freebase is just a fancier and more structured version of Wikipedia - with RDF inside and less information in total. The overall goal of Freebase, however, is to build a Wikipedia equivalent of the world's information. Such a database would be enormously powerful because it could be queried exactly - much like relational databases. So once again the promise is to build much better search.

But the problem is, how can Freebase keep up with the world? Google indexes the Internet daily and grows together with the web. Freebase currently allows editing of information by individuals and has bootstrapped by taking in parts of Wikipedia and other databases, but in order to scale this approach, it needs to perfect the art of continuously taking in unstructured information from the world, parsing it, and updating its database.

The problem of keeping up with the world is common to all database approaches, which are effectively silos. In the case of Twine, there needs to be continuous influx of user data, and in the case of Freebase there needs to be influx of data from the web. These problems are far from trivial and need to be solved successfully in order for the databases to be useful.

Conclusion

With any new technology it is important to define and classify things. The Semantic Web is offering an exciting promise: improved information discoverability, automation of complex searches, and innovative web browsing. Yet the Semantic Web means different things to different people. Indeed, its definitions in the enterprise and consumer spaces are different, and there are different means to a common end - top-down vs. bottom-up and microformats vs. RDF. In addition to these patterns, we are observing the rise of semantic APIs and contextual browsing tools. All of these are in their early days but hold a big promise to fundamentally change the way we interact with information on the web.

What do you think about Semantic Web Patterns? What trends are you seeing and which applications are you waiting for? And if you work with semantic technologies in the enterprise, please share your experiences with us in the comments below.

However Twine has just launched publicly, confident that it is ready for prime time. I spoke with Twine founder and Semantic Web proponent Nova Spivack today to find out what's changed, who's been using Twine up till now, and where the service is headed in the future.

When I met Nova Spivack one year ago in San Francisco, Twine was still in a private beta. Spivack described Twine to us at that point as a "knowledge networking" application with aspects of social networking, wikis, blogging, knowledge management systems. All built with Semantic Web technologies. It sounded exciting, potentially revolutionary.

The problem though, as we found out when we actually got our hands on the product, was that the theory and hype hadn't translated into a usable app. Marshall identified major shortcomings in usability and performance. For example he explained that the service "doesn't consistently grab summary text or tags for pages you save in Twine, it doesn't recognize article authors as relevant people and it often captures summary information about the domain you're on instead of a particular page's content."

In a word, Twine felt "half baked" in March '07 according to our review of the product.

That Was Then, This is Now...

Unsurprisingly, in our interview today Nova Spivack was at pains to say that both usability and speed of the system have been improved for the 1.0 launch. Spivack told us that the main focus of this release is usability. Twine claims it has implemented "major performance fixes", which Spivack said has resulted in "very dramatic speed-ups". He explained that Twine has implemented caching on a lot of features, except for some of the more personal unique ones such as 'My Items' and general search.

Twine is also being re-marketed now as an "interest network" and not just a social network or bookmarking system. Part of the 1.0 launch is a new feature called an "interest feed". Spivack told us that "interest tracking" can start from importing your bookmarks (from e.g. delicious) and users (from e.g. Yahoo Mail). From there Twine's recommendation algorithm will find more items and people of interest - it apparently looks at your social graph, which "twines" you join (i.e. groups of topical interest), and other semantic data that Twine can surface. From there Twine automatically creates "semantic tags" and mines data to expand your 'interest network'.

Spivack said that in the next 2-3 weeks Twine will release "next gen crawling and mining", which will allow the system to index "pockets of the web" for you. As an example, Twine will index all the links and data in the first page of a website that you add to the system.

The Numbers & Demographics

How has Twine performed in its 1-year closed beta? Here are some interesting details about usage so far:

We were told that Twine has had 500,000 unique visitors in its closed beta, of which 50,000 are currently "active". I asked how they define active: in this case it means a user who visits the Twine site at least once per month.

There are currently 20,000 'twines', with 1 million pieces of content having been added to the system. 50% of the twines are private - Spivack said that many of those belong to companies ranging from small businesses to large corporations. Of the public twines, they range from people who use them for bookmarking and hobbies, to people who use twines as a kind of blog (e.g. Nova himself does this), to "cool hunters" (kind of like BoingBoing).

The most popular twines reflect the early adopter audience: cool stuff, semantic web, politics, web industry news, etc.

The average age of a Twine user is 30 yrs, and they tend to be young professionals with medium-to-high income and education. They've used the product for both professional and personal reasons. 50% of users come from outside the US, but the service is primarily english language. Spivack told us that currently they provide basic level support for other languages, but this will be enhanced over time.

User engagement is currently at 12 minutes per session, which Twine says is "trending upward" (it started out as 6 mins). By comparison we were told that Digg is 2.5 mins, StumbleUpon 5.5 mins and Facebook 15 minutes - all according to Compete.com data.

What's Next?

In terms of Twine's roadmap, in 2008 their focus is on usability - which Spivack says is "currently 80% there". In 2009 they will focus more on "surfacing the semantics", meaning improving recommendations, search, and adding support for more kinds of data (e.g. currently users can add YouTube videos, but not all video sites are supported).

Spivack said that in '09 users will be able to bring RSS into Twine (creating "a Reader on steriods"). Also users will be able to import emails. Twine hopes to data mine all of this.

And in 2009 an API is coming to Twine.

Monetizing Twine - Beacon-Like System Coming

Perhaps most intriguingly, Twine is planning to implement a new type of monetization system in 2009. Spivack had a big claim for this: "Twine will be for marketing what google is for advertising" (!). He said it will be the semantic equivalent of Google's Adwords, but for marketers.

The system he described sounded similar to Facebook's Beacon, in that it will insert marketing recommendations into the core content. Essentially marketers will be able to post things into Twine, targeted to users interests. Twine will make recommendations in users interest feeds and some of this will be sponsored. Spivack says Twine has some patents around new metrics for this - they'll be able to see (in aggregate only) what people are doing with their content.

From the description, this sounds very similar to Facebook's controversial Beacon, which was panned for infringing on users privacy. However Spivack claims it is "quite different from Beacon". He also said it will be a CPA not CPM model.

It remains to be seen how this system will work and whether users will have reason to be up in arms. But given that social networks are so hard to monetize and CPM is under pressure right now from the economic situation, the Web industry is in need of innovative solutions. So this will be something to watch closely.

So, Will You Use Twine?

I have been an irregular user of Twine since joining the beta earlier this year. I've saved some bookmarks into Twine as a substitute for Delicious, but I tend to go back to Delicious for most of my bookmarks. For my purposes, Delicious is simpler. One issue that Twine has is that it tries to do an awful lot, which has the risk of confusing users.

Nova Spivack is pretty direct when talking about competing with other products. About Delicious he claimed that the ROI of putting things in Twine is better than Delicious - and it will get higher over time. At various stages in the interview he also said that Twine "can do better than google reader" (as an RSS Reader) and it is better than StumbleUpon, Digg, Facebook.

Nova Spivack can talk a very good talk - and I admire his passion for the Semantic Web and vision for his product. But for Twine to succeed, it needs to do the core things well. Behind all the talk about Semantic Web, beating Adwords and being better than Delicious, Google Reader and so on... is a product that basically is a knowledge management application. Can Twine find enough mainstream users interested in that core functionality? Only time will tell.

I for one will be giving it another go, if only because if Twine does fulfil its hype and becomes the first mainstream Semantic Web app - then it would be embarassing if I missed out on it.

"Qitera is a web service empowering you to build and access your personal knowledge (the geeks call it “knowledge graph”). So you can organize, remix and search all the data dealing with the companies, business partners, friends or projects you track in a more productive way. Additionally, we let you share your wisdom with your peers and publish to blogs, websites and cell phones."

There is little mention of Qitera on Technorati, Google, or other sources. I did however find a slideshow on Slideshare, which featured this graphic illustrating its open standards support:

Below is the full slideshow. Let us know in the comments if you've seen Qitera in action - and if so what did you think? Meantime I've applied for a beta pass to check it out.

In a recent interview Tim Berners-Lee pointed out that the infrastructure to power the Semantic Web is already here. ReadWriteWeb's founder, Richard MacManus, even picked it to be the number one trend in 2008. And rightly so. Not only are the bits of infrastructure now in place, but we are also seeing startups and larger corporations working hard to deliver end user value on top of this sophisticated set of technologies.

The disagreement is not accidental, because the technology and concepts are broad. Much is possible and much is to be imagined.

1. Bottom-Up and Top-Down

We have written a lot about the different approaches to the Semantic Web - the classic bottom-up approach and the new top-down one. The bottom-up approach is focused on annotating information in pages, using RDF, so that it is machine readable. The top-down approach is focused on leveraging information in existing web pages, as-is, to derive meaning automatically. Both approaches are making good progress.

A big win for the bottom-up approach was recent announcement from Yahoo! that their search engine is going to support RDF and microformats. This is a win-win-win for publishers, for Yahoo!, and for customers - publishers now have an incentive to annotate information because Yahoo! Search will be taking advantage of it, and users will then see better, more precise results.

Another recent win for the bottom-up approach was the announcement of the Semantify web service from Dapper (previous coverage). This offering will enable publishers to add semantic annotations to existing web pages. The more tools like Semantify that pop up, the easier it will be for publishers to annotate pages. Automatic annotation tools combined with the incentive to annotate the pages is going to make the bottom-up approach more compelling.

But even if the tools and incentive exists, to make the bottom-up approach widespread is difficult. Today, the magic of Google is that it can understand information as is, without asking people to fully comply with W3C standards of SEO optimization techniques. Similarly, top-down semantic tools are focused on dealing with imperfections in existing information. Among them are the natural language processing tools that do entity extraction - such as the Calais and TextWise APIs that recognize people, companies, places, etc. in documents; vertical search engines, like ZoomInfo and Spock, which mine the web for people; technologies like Dapper and BlueOrganizer, which recognize objects in web pages; and Yahoo! Shortcuts, Snap and SmartLinks, which recognize objects in text and links.

[Disclosure: Alex Iskold is founder and CEO of AdaptiveBlue, which makes BlueOrganizer and SmartLinks.]

Top-down technologies are racing forward despite imperfect information. And, of course, they benefit from the bottom-up annotations as well. The more annotations there are, the more precise top-down technologies will get - because they will be able to take advantage of structured information as well.

2. Annotation Technologies: RDF, Microformats, and Meta Headers

Within the bottom-up approach to annotation of data, there are several choices for annotation. They are not equally powerful, and in fact each approach is a tradeoff between simplicity and completeness. The most comprehensive approach is RDF - a powerful, graph-based language for declaring things, and attributes and relationships between things. In a simplistic way, one can think of RDF as the language that allows expressing truths like: Alex IS human (type expression), Alex HAS a brain (attribute expression), and Alex IS the father of Alice, Lilly, and Sofia (relationship expression). RDF is powerful, but because it is highly recursive, precise, and mathematically sound, it is also complex.

At present, most use of RDF is for interoperability. For example, the medical community uses RDF to describe genomic databases. Because the information is normalized, the databases that were previously silos can now be queried together and correlated. In general, in addition to semantic soundness, the major benefit of RDF is interoperability and standardization, particularly for enterprises, as we will discuss below.

Microformats offer a simpler approach by adding semantics to existing HTML documents using specific CSS styles. The metadata is compact and is embedded inside the actual HTML. Popular microformats are hCard, which describes personal and company contact information, hReview, which adds meta information to review pages, and hCalendar, which is used to describe events.

Microformats are gaining popularity because of their simplicity, but they are still quite limiting. There is no way to described type hierarchies, which the classic semantic community would say is critical. The other issue is that microformats are somewhat cryptic, because the focus is to keep the annotations to a minimum. This, in turn, brings up another question of whether embedding metadata into the view (HTML) is a good idea. The question is: what happens if the underlying data changes when someone makes a copy of the HTML document? Nevertheless, despite these issues, microformats are gaining popularity because they are simple. Microformats are currently used by Flickr, Eventful, and LinkedIn; and many other companies are looking to adopt microformats, particularly because of the recent Yahoo! announcement.

An even simpler approach is to put meta data into the meta headers. This approach has been around for a while and it is a shame that it has not been widely adopted. As an example, the New York Times recently launched extended annotations for its news pages. The benefit of this approach is that it works great for pages that are focused on a topic or a thing. For example, a news page can be described with a set of keywords, geo location, date, time, people, and categories. Another example would be for book pages. O'Reilly.com has been putting book information into the meta headers, describing the author, ISBN, and category of the book.

Despite the fact that all these approaches are different, they are also somewhat complimentary; and each of them is helpful. The more annotations there are in web pages, the more standards are implemented, and the more discoverable and powerful the information becomes.

3. Consumer and Enterprise

Yet another dimension of the conversation about the Semantic Web is the focus on consumer and enterprise applications. In the consumer arena we have been looking for a Killer App - something that delivers tangible and simple consumer value. People simply do not care that a product is built on the Semantic Web, all they are looking for is utility and usefulness.

Up until recently, the challenge has been that the Semantic Web is focused on rather academic issues - like annotating information to make it machine readable. The promise was that once the information is annotated and the web becomes one big giant RDF database, then exciting consumer applications will come. The skeptics, however, have been pointing out that first there needs to be a compelling use case.

Some consumer applications based on the Semantic Web: generic and vertical search, contextual shortcuts and previews, personal information management systems, semantic browsing tools. All of these applications are in their early days and have a long way to go before being truly compelling for the average web user. Still, even if these applications succeed, consumers will not be interested in knowing about the underlying technology - so there is really no marketing play for the Semantic Web in the consumer space.

Enterprises are a different story for a couple of reasons. First, enterprises are much more used to techno speak. To them utilizing semantic technologies translates into being intelligent and that, in turn, is good marketing. 'Our products are better and smarter because we use the Semantic Web' sounds like a good value proposition for the enterprise.

But even above the marketing speak, RDF solves a problem of data interoperability and standards. This "Tower of Babel" situation has been in existence since the early days of software. Forget semantics; just a standard protocol, a standard way to pass around information between two programs, is hugely valuable in the enterprise.

RDF offers a way to communicate using XML-based language, which on top of it has sound mathematical elements to enable semantics. This sounds great, and even the complexity of RDF is not going to stop enterprises from using it. However, there is another problem that might stop it - scalability. Unlike relational databases, which have been around for ages and have been optimized and tuned, XML-based databases are still not widespread. In general, the problem is in the scale and querying capabilities. Like object-oriented database technologies of the late nineties, XML-based databases hold a lot of promise, but we are yet to see them in action in a big way.

4. Semantic APIs

With the rise of Semantic Web applications, we are also seeing the rise of Semantic APIs. In general, these web services take as an input unstructured information and find entities and relationships. One way to think of these services is mini natural language processing tools, which are only concerned with a subset of the language.

The first example is the Open Calais API from Reuters that we have covered in two articles here and here. This service accepts raw text and returns information about people, places, and companies found in the document. The output not only returns the list of found matches, but also specifies places in the document where the information is found. Behind Calais is a powerful natural language processing technology developed by Clear Forest (now owned by Reuters), which relies on algorithms and databases to extract entities out of text. According to Reuters, Calais is extensible, and it is just a matter of time before new entities will be added.

Another example is the SemanticHacker API from TextWise, which is offering a one million dollar prize for the best commercial semantic web application developed on top of it. This API classifies information in documents into categories called semantic signatures. Given a document, it outputs entities or topics that the document is about. It is kind of like Calais, but also delivers a topical hierarchy, where the actual objects are leafs.

Another semantic API is offered by Dapper - a web service which facilitates the extraction of structure from unstructured HTML pages. Dapper works by enabling users to define attributes of an object based on the bits of the page. For example, a book publisher might define where the information about author, isbn and number of pages is on a typical book page and the Dapper application would then create a recognizer for any page on the publisher site and enable access to it via REST API.

While this seems backwards from an engineering point of view, Dapper's technology is remarkably useful in the real world. In a typical scenario, for web sites that do not have clean APIs to access their information, even non-technical people can build an API in minutes with Dapper. This is a powerful way of quickly turning web sites into web services.

5. Search Technologies

Perhaps the first significant blow to the Semantic Web has been the inability thus far to improve search. The premise that semantical understanding of pages leads to vastly better search has yet to be validated. The two main contenders, Hakia and PowerSet, have made some progress, but not enough. The problem is that Google's algorithm, which is based on statistical analysis, deals just fine with semantic entities like people, cities, and companies. When asked What is the capital of France? Google returns a good enough answer.

There is a growing realization that marginal improvement in search might not be enough to beat Google, and to declare search the killer app for the Semantic Web. Likely, understanding semantics is helpful but not sufficient to build a better search engine. A combination of semantics, innovative presentation, and memory of who the user is, will be necessary to power the next generation search experience.

Alternative approaches also attempt to overlay semantics on top of the search results. Even Google ventures into verticals by partitioning the results into different categories. The consumer can then decide which type of answer they are interested in.

Yet search is a game that is far from won and a lot of semantic companies are really trying to raise the bar. There may be another twist to the whole search play - contextual technologies, as well as semantic databases, could lead to qualitatively better results. And so we turn to these next.

6. Contextual Technologies

We are seeing an increasing number of contextual tools entering the consumer market. Contextual navigation does not just improve search, but rather shortcuts it. Applications like Snap or Yahoo! Shortcuts or SmartLinks "understand" the objects inside text and links and bring relevant information right into the user's context. The result is that the user does not need to search at all.

Thinking about this more deeply, one realizes that contextual tools leverage semantics in a much more interesting way. Instead of trying to parse what a user types into the search box, contextual technologies rely on analyzing the content. So the meaning is derived in a much more precise way - or rather, there is less guessing. The contextual tools then offer the users relevant choices, each of which leads to a correct result. This is fundamentally different from trying to pull the right results from a myriad of possible choices resulting from a web search.

We are also seeing an increasing number of contextual technologies make their way into the browser. Top-down semantic technologies need to work without publishers doing anything; and so to infer context, contextual technologies integrate into the browser. Firefox's recommended extensions page features a number of contextual browsing solutions - Interclue, ThumbStrips, Cooliris, and BlueOrganizer (from my own company).

The common theme among these tools is the recognition of information and the creation of specific micro contexts for the users to interact with that information.

7. Semantic Databases

Semantic databases are another breed of semantic applications focused on annotating web information to be more structured. Twine, a product of Radar Networks and currently in private beta, focuses on building a personal knowledge base. Twine works by absorbing unstructured content in various forms and building a personal database of people, companies, things, locations, etc. The content is sent to Twine via bookmarklet or via email or manually. The technology needs to evolve more, but one can see how such databases can be useful once the kinks are worked out. One of the very powerful applications that could be built on top of Twine, for example, is personalized search - a way to filter the results of any search engine based on a particular individual.

It is worth noting that Radar Networks has spent a lot of time getting the infrastructure right. The underlying representation is RDF and is ready to be consumed by other semantic web services. But a big chunk of the core algorithms, the ones that are dealing with entity extraction, are being commoditized by Semantic Web APIs. Reuters offers this as an API call, for example, and so moving forward, Twine won't need to be concerned with how to do that.

Another big player in the semantic databases space is a company called Metaweb, which created Freebase. In its present form, Freebase is just a fancier and more structured version of Wikipedia - with RDF inside and less information in total. The overall goal of Freebase, however, is to build a Wikipedia equivalent of the world's information. Such a database would be enormously powerful because it could be queried exactly - much like relational databases. So once again the promise is to build much better search.

But the problem is, how can Freebase keep up with the world? Google indexes the Internet daily and grows together with the web. Freebase currently allows editing of information by individuals and has bootstrapped by taking in parts of Wikipedia and other databases, but in order to scale this approach, it needs to perfect the art of continuously taking in unstructured information from the world, parsing it, and updating its database.

The problem of keeping up with the world is common to all database approaches, which are effectively silos. In the case of Twine, there needs to be continuous influx of user data, and in the case of Freebase there needs to be influx of data from the web. These problems are far from trivial and need to be solved successfully in order for the databases to be useful.

Conclusion

With any new technology it is important to define and classify things. The Semantic Web is offering an exciting promise: improved information discoverability, automation of complex searches, and innovative web browsing. Yet the Semantic Web means different things to different people. Indeed, its definition in the enterprise and consumer spaces is different, and there are different means to a common end - top-down vs. bottom up and microformats vs. RDF. In addition to these patterns, we are observing the rise of semantic APIs and contextual browsing tools. All of these are in their early days, but hold a big promise to fundamentally change the way we interact with information on the web.

What do you think about Semantic Web Patterns? What trends are you seeing and which applications are you waiting for? And if you work with semantic technologies in the enterprise, please share your experiences with us in the comments below.

Congratulations David, you've won a $30 Amazon voucher - courtesy of our competition sponsors AdaptiveBlue and their Netflix Queue Widget.

Update: David, please contact editor at readwriteweb.com with your real email address :-)

Update 2: Twine founder Nova Spivack has written a response to our post on his blog.

Here is David's full comment:

"Marshall, Marshall, Marshall. You're a great writer, generally right on top of things. Matter of fact, of the Web 2.0 A-list bloggers, you've demonstrated the most knowledge of the Semantic Web. You even get -- which few do -- that recommenders are an important part of a comprehensive solution.

This being said, you're way off on Twine. First, you've been on Twine only since March 7th. Second, you have only three connections, Rafe, Nova (their CEO) and yours truly. As a result, you haven't had a chance to explore Twine for social knowledge sharing. Third, you've subscribed to only three Twines. That's it!! You did choose two of the most active Twines, but you didn't explore how Twine can be used beyond geekdom. In contrast to what Gabe has tried to do, Digg, et al, Twine has a lot of value when exploring broader topics. See the China, Public Policy, and Futures Twines as three examples. Honestly, you need to get a lot more engaged before you can really comment. At this point, you've had minimal engagement. And, as you know, I'm the MOST engaged private beta tester, so I believe I'm in a good position to make an objective observation. (Remember, I was a VP-level analyst at META. So I have the street cred for making objective observations on emerging technologies; I'm not just some blogger or the like.)

In fact, here in Qingdao (China) we're starting to use Twine as a combination social network + social bookmarking site + threaded online discussion group + wiki. We have two groups that are slowly getting engaged with Twine. One is a local ecumenical Christian fellowship for expats, the other (which is much more active at this point) is for the de facto Chamber of Commerce for Westerners living in Qingdao. Although we haven't rolled out Twine for the general membership, the "Chamber" Executive Committee has adopted Twine as its key communications and knowledge sharing tool. Matter of fact, the head (yes, THE head) of one of the largest U.S. operations in China is getting addicted to Twine ... as is his wife!! We've even had the 17 year old adopted daughter of one of our local Kiwis start a Twine which already has among the most members. Started four or so days ago, it's already among the top dozen or so Twines. (It's the Connecting People Together Twine.) And I've even created a TUG (Twine Users Group) for Qingdao which I'll officially launch after we've had a chance to roll out Twine to at least a few hundred fellow expats ... and we'll likely follow with a TUG in Shanghai, too.

To be very frank, I was personally concerned about rolling out Twine to media types such as yourself. It's not ready for prime time, but nobody ever said it was. It's in PRIVATE beta. Hand holding helps ... and I've done a lot of hand holding. But I'm also dealing with a crowd that has NO IDEA what social bookmarking is, what a wiki is. Yet, I/we have demonstrated that Twine has a great deal of utility for newbies; it's not just for semweb geeks.

Unfortunately, you entered Twine just when the UI changed. This was bad timing. The new UI should have been rolled out to the existing private beta testers BEFORE (and NOT concurrently) with letting in selected media personalities. I concur that the UI needs some work, as does the daily e-mail Digest. Hey, it's a work in progress.

But, Marshall, the core tech underneath Twine's hood is what really matters, is the power (and competitive advantage) for Twine. The UI can be improved. I've already seen a lot of improvement in the UI; it will get a lot easier to use.

Without being modest, I can tell you that there are two power users of Twine: Myself and Hrafn Thorisson. (If you check the Explore page, you'll see this.) Both of us are extremely active in two private Twines, the Product Community and Evangelism Twines. We also have a lot of direct discourse with Nova -- "direct" as is by private e-mail. I can tell you first hand that Nova and his team (Peter, Scott, Chris, Jim, James) are highly responsive to our suggestions. Don't be too concerned: When someone needs to piss on Twine, we do the pissing. But they clean up the mess, make Twine better. And they respond in fairly short order, too. Hey, not everything can be changed overnight. Changes, however, do happen on a fairly frequent basis, more so than in any other beta I've been part of.

Marshall, don't give up on Twine. Just watch it, pledge to write another review in July and then another in December. You'll become a true believer over time. BTW, feel free to contact me if you want to toss around any ideas, need uber user feedback. My contact info is on my Twine profile page; don't use the e-mail address that I provide for my R/WW comments. Thanks!!"

If that's the case, it will probably be too soon. Twine has some major shortcomings that I think are going to drastically hinder the service's adoption. Perhaps unsurprisingly, those shortcomings come down to usability and performance. Hopefully these problems will be resolved, but it isn't going to be easy.

The Basic Idea

Twine looks at content and parses it automatically for the names of people, places, organizations and other subject tags. Users are then able to navigate between related content, view recommended content and connect with recommended people with related interests.

The semantic analysis is faster and smarter than full text search. Making content online machine readable cuts time and thought out of discovery of related information, letting users focus on higher levels of engagement. It's a great idea and I hope Twine can overcome the issues I'm seeing with it.

Problem: It Doesn't Work Very Well

The biggest problem with Twine right now may be that it doesn't work as well as it should. It doesn't consistently grab summary text or tags for pages you save in Twine, it doesn't recognize article authors as relevant people and it often captures summary information about the domain you're on instead of a particular page's content.

Twine founder Nova Spivack saw that I was saving pages that weren't coming in with summary information and commented on one of my items that the page at issue was irregularly formatted. That's why Twine wasn't able to analyze it, he said. That is a major problem; most of the web is made up of ugly, non-standard pages. Fundamental to the value proposition of a top-down semantic analysis tool should be the ability to discover meaning from unstructured data. Many of the other problems Twine faces will be challenging but do seem solvable. This one could be a deal breaker.

Serious researchers will also be frustrated with the lack of support for authenticated (password protected) pages and the absence of RSS feeds -though feeds may come as soon as the app is public.

Problem: It's Poorly Organized

Twine has bitten off a whole lot to chew on. It's an impressive service for the most part. Unfortunately, full-featured social bookmarking is information-dense enough that adding all the semantic features and recommendations from Twine turns information architecture and User Experience into huge challenges.

Twine's user experience is confusing. It's hard to keep track of all the levels and types of information available, site navigation is dizzying and my use of the service happened in spite of the interface.

There are a lot of little things Twine could do to help, like defaulting the saved item path to the same category I saved the previous item in.

There are a variety of different approaches already explored in the social bookmarking market. Del.icio.us is simple and does what it says it does, nothing more nothing less. Ma.gnolia does a little bit more, looks great and is relatively self-explanatory. Furl.net was probably better technology than either Del.icio.us or Ma.gnolia but the user experience makes you want to punch some one and the service has withered accordingly. Twine needs to blow this category out of the water but it doesn't.

I'm sure with some practice I could learn to use Twine more easily, but that's not an ideal first experience. I don't feel compelled to keep trying, other than because of my interest in the semantic technology. There's no visualization, just flat interlinked pages, the only zing to the product today is the recommendation feature.

I would use Twine for recommendation alone, but the value of that feature is minimal until the service finds a large number of users. As it stands, that's not likely to occur. When it comes to collective organization and discovery of content - nothing is as important as network effect.

Twine's in closed beta right now - but it's been in the oven for a long time, has substantial investor backing and is highly anticipated. Despite all that support, it still feels half baked. I hate to say that because everyone says the trouble with the semantic web is that products never come to market - but I don't think Twine is ready. I don't know if it ever will be. Someone else may have to be the first mainstream semantic web app - or maybe no one will be. Semantics may be best suited to the back end. I hope Twine, or someone, can bring something like this to market that I want to use.

You can join the long line of people requesting beta access and make your own decision about Twine probably later this month. For a more positive review, see Rafe Needleman's write up at Webware.

Super-sleuth Dan Primack over at PE Hub dug up the early news about the investment and Chris Morrison at Venture Beat says Velocity Interactive Group and a number of other investors are putting in money in the $15 to $20 million range.

For a look at the breadth of Twine coverage across blogs specializing in semantic web technology, check out this search across our Semantic Web custom search engine, part of the RWW Toolkit for the Top Issues of 2008. See also Bernard Lunn's post here this week titled, 11 Things to Know About the Semantic Web.

The following is Robert Scoble's 10 minute highlight tape of an interview he did with Twine founder Nova Spivack in December.

In this post, we analyze several existing and potential applications of semantic technologies and look for the killer app.

Natual Language Understanding

Since the beginning, the Semantic Web has been associated with Artificial Intelligence. The idea of representing information in structured form so that computers can "understand it" and then solve complex problems was one of the keystones of the Semantic Web vision. The problem is that representing billions of existing web documents as RDF is a rather daunting, if not impossible task. An alternative would be to "teach" computers natural language. If an application could read the page the way we read it and interpret what it says, the annotations would not be necessary.

Natural language processing has been the Holy Grail of AI for awhile now. However, it is a very difficult problem, because humans are born with the innate ability to understand language and we learn it not in a vacuum, but in the context of life. Certainly if we could replicate that with computers, it would be amazing and it would be the killer app. The problem is that this is not on the horizon. The Semantic Web technologies of today are not able to represent natural language in its entirety, and this is not really even their goal. Even if we could represent each page completely, there is still the matter of interpreting structure into semantics, which is the magic that our brain does so well and so easily.

Genie In The Bottle

Related to natural language understanding, is another idea that is not on the horizon. John Markoff called it "the perfect vacation." I call it the "Genie in the Bottle" to illustrate the impossibility of this. There is a misunderstanding about the Semantic Web which is floating around, which equates the Semantic Web with ability to solve really hard problems. It is simply not true.

For example, if you go to a new travel agency and ask them to book the perfect vacation for you, the travel agent will not be able to do it, because she does not know you. In order to find the perfect vacation there needs to be constraints: where you've been before, who you are going with, what you like to do, what is your budget, etc. Finding the "perfect" vacation is not a one shot deal, it is a process, which leverages iteration and memory.

True, with the Semantic Web the information is structured, but it does not mean that the computer can necessarily solve complex problems. These are two completely different things. Just because you have a map, does not mean that you know the best way to get from point A to point B. Having a map is necessary, but it is not sufficient, you need the algorithm to find the best path. There is a big difference between asking what is the capital of France and what is the cheapest airfair today to fly from New York to Paris. And the even harder question is: Where should I go on vacation next? Computers are not going to give us an instant, perfect answer to this question anytime soon, if ever. Again, this would be the killer app, it is just not likely to happen.

Semantic Knowledge Databases

So what is realistic and possible today? The first in the list of growing applications are Semantic Knowledge Databases. The two examples that we will look at here are Freebase and Twine. While Freebase is focusing on building essentially a semantic equivalent of Wikipedia, and Twine is focused on a personal semantic database, both are databases, both focus on knowledge management, and both are Wikipedia-like. The advantage of these databases over Wikipedia is that they represent information in a structured way and support queries. To understand the difference, take a look at the Alicia Keys page on Freebase and on Wikipedia. At first glance they are very similar, but Freebase "knows" that Alicia Keys is a blues singer and it then knows other blues singers. For Wikipedia, blues is just another page, not a music genre. So Freebase can potentially answer a question of listing all blues singers, while Wikipedia can not.

This is certainly interesting but the question is will people care? Can the end consumer tell the difference? Unlikely. Today Wikipedia contains definitive references on a vast number of topics. Like Google, it is easy to search and find relevant information, and as a result, people are not likely to be in need of a better Wikipedia. With Twine the situation might prove to be different, because personal knowledge management is an important problem. The first question is: Are their enough people who want to be efficient in managing personal knowledge? I think the answer is increasingly likely to be "yes." And the second question is: Does knowing the semantics of knowledge help you build the best application? At the very least Twine has to beat del.icio.us bookmarks and ideally needs to do for personal knowledge management what Highrise is doing for CRM.

But beyond the execution, there is still another problem. For a semantic knowledge base to be the killer app it needs to ignite imagination and capture people's hearts and minds. This is not likely to happen. We appreciate libraries, we can not live without them, but we take them for granted. Knowledge has been commoditized thanks to Google, Wikipedia, and the blogosphere, and is perceived as abundant and unexciting. For this reason Semantic Databases are not likely to be the killer apps -- but they might become a stepping stone towards one.

Semantic Search

An early candidate for the killer app in the semantic web category was search. First Hakia and more recently Powerset marketed the idea that a semantic search engine, one that is based on the understanding of natural language, can beat Google. On top of having the pressure to deliver qualitatively better results, Semantic Search companies also have to, at least approximately, solve the problem of natural language understanding, which as we discussed earlier is a very difficult one.

Where things stand right now, it does not look like search is the killer app for semantics. The understanding of natural language does not seem to give you a noticeable edge in getting better search results. At least in the comparisons that we have performed earlier there is no major difference. The statistical algorithm deployed by Google is precise and good enough, which is why it has been the clear leader in web search for the past 8 years. To unseat Google will require more than incremental improvement in search, it will likely take a paradigm shift and the creation of a different web experience. Below, we discuss how "discovery" could possibly take a bite out of the pie, but as of now Google's algorithm remains good and strong.

Social Graph

After Tim Bernes-Lee posted his thoughts on the Social Graph, a discussion began on the web in which people wondered if the Social Graph is in fact the Semantic Web. This, however, is a gross misinterpretation of the post. The Social Graph is not the Semantic Web, nor is it the killer app of the Semantic Web. They are just two separate concepts. The confusion comes from the fact that they both are Mathematical Graphs or a Network. The underlying structure of both consists of nodes connected by links. Many things in the nature and society are networks, so it is not surprising that meaning and people fall into this category.

If anything, it is more correct to say that the Social Graph is a subset of the giant, all encompasing Semantic Web. Knowing how people are connected is important in order to solve the perfect vacation problem. After all, a perfect vacation should be taken together with perfect friends, right? But jokes aside, the Social Graph is an interesting and important trend for 2008, however, it is not really related to Semantic Web.

Shortcuts

Increasingly, we are seeing a new breed of Semantic Applications, which we generalize as shortcuts. This category includes SnapShots from Snap, BlueOrganizer and SmartLinks from AdaptiveBlue, Shortcuts from Yahoo!, and In-text search from Lingospot. What is common between all these technologies is that they leverage the simple semantics of the content to deliver additional information. In the case of Snap and AdaptiveBlue, the semantics is defined by the URL, while Yahoo! and Lingospot perform text analysis.

Regardless of the method, all of these technologies deliver related information via Ajax popups. That is, they leverage semantics to pull the information from the web. This is essentially discovery or reverse search. When the user is looking at a book there is a preview with a brief description and the cover image, when the user encounters a stock symbol he is presented with a stock chart, analysis and additional links to the company, when the user is looking at a music album there is a play button, and when the user encounters a movie there is an ability to watch the trailer in place. The shortcuts remove the need to search, instead, the related content from the web comes right into the page.

Today's shortcut technologies are simple and still in their infancy, but they are among the most successful examples of semantic applications. However, we can not call them the killer app for several reasons.

First, people perceive them as advertising, which is not the point. Snap certainly made an early push into ads, but this is not a representation of what these technologies will look like in the future. Second, in their current implementation, all of these technologies are utilities. For the same reason that people are not going to get emotional about personal knowledge management, they will not be emotional about shortcuts. Shortcuts will also be taken for granted.

Yet, shortcuts hold the most promise. With a few more iterations these technologies are going to get slicker and more precise. They will leverage content and micro-context to reduce the amount of search. They will become more personalized based on user behavior. And once this happens it will be a big deal.

Full Disclosure: Alex Iskold is the founder and CEO of AdaptiveBlue.

Conclusion

We are still waiting for the killer app for Semantic Web, something that can get viral and turn semantics into a marketing term. Problems like natural language understanding still remain difficult to solve, and the solutions do not appear to be on our horizon right now. It also appears that a semantic search engine, at least based on the ones we have seen to date, does not have a substantial advantage over Google. We are seeing the rise of early Semantic Knowledge Databases, but while we expect them to get better and more interesting, they are more likely to be the stepping stones to the killer app, rather than the app itself.

In the mean time, we are seeing the rise of shortcut technologies, which leverage the basic semantics of the content, like URL and simple context analysis, to deliver relevant information, links, and media directly into the page. While still very early, these technologies hold the most promise because they are simple and useful. We expect that the next generation of these technologies in conjunction with personalization will deliver an interesting alternative to search -- contextual discovery. We will discuss this alternative in more detail in a future post.

Now tell us what you think the killer app for Semantic Web will be? Which of these technologies do you think is the most promising?