This time, we'll focus on the Web of services, which makes services accessible to and processable for machines. These Webs all have a semantic architecture in common and follow basic Web principles, such as being decentralized, modular, simple, addressable via URIs, and built for machines.

The effort to standardize such things as service-oriented architectures (SOA) and Web services has taken years, but still we have no clear definition of what constitutes a service at a conceptual level. The interface, which is the format of what goes in and out of the service, is often described formally, but what the service is actually doing, semantically speaking, is not. While there are a number of different approaches to semantically describing Web services, such as OWL-S, WSMO and WDSL-S, none so far has managed to break out of its academic confines.

Today, there are already all kinds of services with different levels of complexity, and their number is expected to grow exponentially. The services follow different standards, and a lot of them are proprietary, uni-directional and designed to be used by humans to mash up something new. Editorial catalogs such as ProgrammableWeb and search engines for Web services such as seekda are designed for humans who are searching for a particular service for that reason. For tasks that are unsolvable for machines, there are even Web services such as Amazon's Mechanical Turk, which have humans in the back end answering tricky queries.

The problem with all of this is that each of the tens of thousands of services is accessible but not findable by a machine without a machine-understandable description. Thus, every service nowadays has to be wired to a machine by hand. So, what would machines be capable of if services were annotated with semantic descriptions?

• Service discovery
  Given an index of Web services, a machine charged with finding the right service for a particular problem could choose one among those that have been indexed.
• Contracting and execution
  Once a service has been selected, a machine could look up its terms and decide on contracting and execution details. How often would the service be needed? And what would be the cheapest contract then?
• Billing or revenue sharing
  Depending on the autonomy of the machine, one could imagine something like an Autonomous Agent, which automatically makes the best deal with the service provider on such things as billing or revenue sharing for service usage.
• Replacement on failure, based on experience
  Of course, the machine would be able to replace a failing service with an equivalent one. It could also rate a service and publish it.
• Service orchestration
  A machine could, given enough intelligence, split a task into sub-tasks and then discover, contract and orchestrate services to solve these sub-tasks. And after the sub-tasks have been addressed, the main task would be solved. Such orchestration could involve the parallelization of tasks, for speeding up or redundancy purposes, or chaining services (whereby the output of one service is inputted into the next).

Research projects such as TripCom, SUPER, SHAPE and SOA4All are dealing with these ideas and scenarios.

Future scenarios are limited only by our imagination: machines could autonomously pursue goals on behalf of their master user or company, according to a specified level of freedom. These agents could solve increasingly complex problems and be granted increasingly more autonomy (finally ending up as Skynet).

In the next and final post in this series, we will discuss how all of these scenarios could become a reality with the arrival of all three Webs: a revolution in the ability of machines to access, process and apply information.

Do you also count the Web of services as a third Web? Where do you see its limits?

(Photo by zorro-art.)

• First generation: was done on a single processor, usually for process control in military systems.

• Second generation: within a Local Area Network, usually for a financial trading room.

• Third generation: applied across the whole Web/Internet, what we call the real-time Web.

In each generation a stack has emerged, and secure messaging has been key to that stack. The names change and the scale of the prize and challenges certainly changes, but the basic issue remains the same: delivering messages reliably and quickly. In this post, we trace the steps from the second generation to the third generation to see how the real-time Web might play out.

Generation 2's Winner: The Teknekron Information Bus Story

I worked with the technology's second generation. My company took on the challenge of delivering market prices from multiple sources to hundreds of traders in a trading room, enabling new apps based on real-time calculation. The solution had to scale to maybe 2,000 traders, which is laughably small by the standards of the real-time Web. But you did not want to face a roomful of furious traders if your system was down for three minutes or delayed prices by 15 seconds. The reliability and latency requirements were very stringent.

The company that emerged as leader, Teknekron, coined the term "Information Bus." It did very well by coining the term, because it established Teknekron as a market leader; it followed the "get mindshare first, and then go for market share" strategy. Vivek Ranadive, the visionary founder and CEO, describes in his 1999 book "The Power Of Now" how the term came to him. He was schooled as a hardware engineer and was appalled by how much more chaotic (i.e. error- and delay-prone) software projects were compared to hardware projects. Looking at why, he realized that the concept of a "bus," which is the device that all hardware components connect to, could be applied to software.

Ranadive understood that real-time, event-driven systems would not be limited to financial trading, and he executed brilliantly on that insight. He sold Teknekron to Reuters, the company's main rival in trading systems, but retained the right to use the technology outside of financial markets. That offshoot became TIBCO (The Information Bus Company), which at the time of writing is a publicly traded company valued at $1.6 billion, with annual revenue of over $600 million.

But TIBCO did not leverage its position to dominate the third generation. Another company tried that and blew a lot of money in the attempt.

KnowNow: Lessons of a Blow-Out

KnowNow worked through $50 million in funding before throwing in the towel in July 2008. It was going to update the message bus to the Web using RSS. It was still an enterprise play.

It had a strong management team, top-tier VC (Kleiner Perkins), and strong technology. Why did it fail? Three big reasons stand out:

• Generation 3 is a tougher technical challenge. Generation 2's state of technology was perfect for local area networks using Ethernet, where the user device was always connected and where we measured users by the thousands. With generation 3, we are dealing with millions of users who are offline some of the time, and the network bandwidth available for messages is not under their control. In other words, it is one big bear of a technical challenge!
• Companies are reluctant to invest millions of dollars in closed software when open standards clearly always win on the Web. Companies can instead experiment very cheaply using consumer-centric RSS tools and open source.
• KnowHow was early. The hype on real-time enterprise did not last long enough, and it hit the trough of disillusionment.

Contenders for the Real-Time Web Message Bus

We have probably missed a few, so please tell us about them in the comments. These are very different types of solutions, but they are working towards a similar objective. First, we will list them and then attempt a bit of categorization:

• Gnip: an independent venture whose monetization model relies on it being a hub that different sites can connect to.
• Tornado (FriendFeed's Python-based Web server technology, which was open sourced by Facebook): this uses PubSubHubbub.
• RSS Cloud: promoted by Dave Winer, with WordPress as a marquee partner.
• PubSubHubbub: promoted by Google (and FriendFeed/Facebook), with SixApart as a marquee partner.
• XMPP The technology with which IM clients interoperate. Being used by Yammer, Present.ly, and Drop.io.
• Twitter.
• Facebook.
• TIB: an enterprise solution from TIBCO.
• MQ: an enterprise solution from IBM.
• Sonic: an enterprise solution from Progress Software.

Here are the categories they fall into:

1. Enterprise
   TIB, MQ, and Sonic all fall into this category, and there are more. They will find it hard to make the transition to the real-time Web for two reasons. The first is technical. Connecting millions of users over a low bandwidth network via HTTP is very different from connecting a few thousand users over a corporate network. The second is commercial: they monetize by selling licensing fees to enterprises. But that is not the primary way in which the real-time Web will be monetized. Still, these are big profitable companies with the right tech chops, so they cannot be dismissed.
2. Open source and open standards
   Tornado, RSS Cloud, PubSubHubbub, and XMPP fall into this category. They have differing purposes but also a lot of overlap. XMPP is low level and not affiliated with any big company. RSS Cloud and PubSubHubbub are most similar to each other. Tornado is a Web server. But they all face the issue that an open standard takes a long time to evolve and consolidate into a winner. Before that happens, we will likely see more entrants, making the choice even more confusing for developers. So, developers may hedge their bets and go for commercial/de facto standards.
3. Commercial hub
   Gnip is the purest example of this. It aims to get traction by being simpler to implement than the open-source/open-standards offerings. It is a bold strategy from a strong team of entrepreneurs and investors. For something as big as this - basic plumbing for the Web - it feels like a bit too much reliance on one company.
4. Traffic plays
   Twitter and Facebook are the major players here. They might become de facto standards because the biggest issue is traffic. Developers will build where there is traffic as long as two conditions are met:
   • It works technically (i.e. latency at scale). Twitter has some credibility issues here.
   • It is open and transparent - i.e. anyone can connect without fear of the game changing on them. Facebook has some credibility issues here.

The Technical Challenges

First, we have to address the question of "What does real time mean?"

In each generation, the gurus of the previous generation get sniffy about the definition of real time ("What do you mean one second is real time? We measure in micro-seconds!").

It depends on the usage case and technical possibilities. Real time within a single processor is obviously faster than real time over a LAN, which is faster than real time over HTTP for the whole Web.

Real time, practically speaking, means "orders of magnitude faster than how data has been delivered in the past, and faster than most people think they need today." So, RSS Cloud is faster than RSS, and Twitter search is faster than Google search, and IM is faster than email.

In generation 2, the thinking evolved through three approaches:

1. Polling. It soon became obvious that this would not scale.
2. Broadcast, which put too much load on the client.
3. Multicast, which became PubSub.

Most current Web HTTP-based real-time systems are still in the polling phase (i.e. subscribers poll publishers to see if they have an update), which clearly won't scale. The newer approaches fundamentally enable a publisher to say, "I will tell you, the subscriber, when I have an update, so you don't need to poll me." It's a geek's way of saying "Don't call us. We'll call you."

For a good technical primer on how this works and a discussion of the differences between RSS Cloud and PubSubHubbub (mercifully shortened to PuSH), check out this post.

Sticking My Neck Out to Guess the Winner

Twitter. Why? It has the traffic and is open and transparent enough to win confidence. But two big problems remain:

1. It has to reveal its business model. Only then will partners feel confident that they understand its strategic direction.
2. It has to prove it can get good latency at scale. Its early history of fail whales makes people justifiably skeptical.

The winner will likely be the platform that hosts the killer app. Most platforms get traction through a killer app. In the second generation of real time, that killer app was market data for financial traders. What will it be in the third generation? We will explore this in a future post.

What to Name This Generation?

"Message/Information Bus" worked for generation 2. It took a key concept from generation 1 - the hardware bus - and applied it to a local area network.

That does not work so well for generation 3, the real-time Web. For now, we talk about a Web-wide real-time message bus because there is no better alternative.

The new term may have something to do with "status," because the status update is a central concept in social media. Perhaps something like "StatusFabric" or "StatusNet" or "StatusWeb" will emerge.

Called Portable Contacts, the technical spec offers a standard, interoperable way for social networks to serve up your friends lists to anyone you give permission to access them. This should allow application developers to innovate on top of your social connections much more efficiently.

we're seeing major Internet companies making contacts APIs available, such as Google's GData Contacts API, Yahoo's Address Book API, and Microsoft's Live Contacts API (with more to come). Not surprisingly though, each of these APIs is unique and proprietary. We believe this creates the ideal conditions for developing a common, open spec that everyone can benefit from.

Why is This Important?

The social web works best when it's truly social. New applications that use social sharing can be much more useful when new users can port in their existing network of friends and see who they know is already using a site. That's much better than starting cold.

These types of standardized approaches to passing that data are secure (that's good) and allow developers to write code once to use all the supporting sources of data. You've heard the old illustration about railroads? When all the railroads in the US accepted a standard size of rail, all the trains were able to travel much farther than ever before. That's where we're headed with all this information on the web. When we give it standard methods of transport, it can go further and do more than ever before.

That's a pretty big deal and it's fantastic that Google has moved to support the Portable Contacts standard. Hopefully sometime soon everyone will and then we'll wonder what took the web so long to enable social interoperability.

Though Dreamweaver catches a lot of flack from cynics, many people still use it and the design software is said to be much improved in recent years. Zeldman's new tool is worth checking out.

You can read all about the software and wish Zeldman a happy birthday in his blog post from this morning.

Zeldman is the author of the internationally celebrated book Designing With Web Standards. We love standards based development and design here at ReadWriteWeb because increased machine readability creates a playing field more favorable for innovation.

That's only going to be more true in the future, so if you'd like to have a say in how it all goes down - now's the time to get involved. The OpenID Foundation is one of the leading organizations in the new standards world and it's having its first ever election of community board members this month. Nominations close Monday and the voting begins on Wednesday.

Individuals will have to pay a $25 Foundation membership fee in order to vote, but this author just paid his and is looking forward to pulling the virtual voter's lever. Nominees so far are listed below.

What Are the Issues?

OpenID usability, getting major players to respect incoming OpenID and not just authenticate their own users elsewhere with OpenID, the personal data payload that travels with OpenID and many other difficult questions remain unanswered, despite all the progress the Foundation and other organizations have made in the last year.

A year ago this week we wrote a post saying that OpenID was in serious trouble. One year later, the situation seems to have improved quite a lot. That's thanks not just to the work of the OpenID Foundation, but they deserve a large part of the credit.

The protocol is far from out of the woods, though, and so this election is going to be an important one.

Who's Been Nominated?

So far twelve people have been nominated. Once you register as a Foundation member, you can see the nominees and their position statements. More nominations will likely occur before this weekend is over. Seven of the following twelve total number of people nominated by Monday will get positions on the board. Here's who's been nominated so far.

Johannes Ernst - founder and CEO of startup Netmesh
David Recordon - is from SixApart and is one of the most publicly visible members of the OpenID community
Mike Kirkwood - CEO of iPhone-centric medical patient data service Polka
Eric Sachs - Product Manager at Google
Snorri Giorgetti - OpenID Foundation's European Representative
Eran Hammer-Lahav - Open Web Evangelist at Yahoo! and OAuth lover
Allen Tom - Architect, Yahoo! Membership
Scott Kveton - Current OpenID Foundation Chair and VP Open Platforms at Vidoop
Nat Sakimura - Identity tech wonk from Japan
Brian Kissel - CEO of JanRain, makers of MyOpenID.com
John Bradley - OpenID security wonk
Martin Atkins - an OpenSocial and identity developer

Which seven of those people do you want driving the future of the OpenID Foundation? Register as a member, read their policy statements and you can have your hopes for this important technology paradigm recognized.

We're big fans of web standards here at ReadWriteWeb and we'll tell you why.

This year's been a big one for web standards; the President Elect just enabled users to login to comment on his website using the standard authentication protocol OpenID, for example.

So get your blue beanie or similar hat on and make yourself a photo. Your friends will wonder why you and others are wearing them online today and when they ask either you or Google - they'll end up thinking about the importance of web standards as a result.

For more info, visit Jeffrey Zeldman's blog and see if there's a group relevant to you participating, like the South African Web Standards and Accessibility Group, by whom we were reminded of today's event.

It's a great little idea that could help breath new life into the wiki community. We would love to see the extension become a standard part of Firefox.

The group says that the Universal Edit Button "will be a convenience to web surfers who are already inclined to contribute, and an invitation to those who have yet to discover the thrill of building a common resource."

"As this kind of public editing becomes more commonplace," they say, "the button may become regarded as a badge of honor. It may serve as an incentive to encourage companies and site developers to add publicly-editable components to their sites, in order to have the UEB displayed for their sites. We hope that this button catalyzes the acceleration of the editable web, and helps accelerate society's trend toward building valued common resources."

We do wonder how many people notice the RSS icon in the browser toolbar, and thus how many people will notice the Universal Edit Button - but we love the idea. The fact is, the world is full of people who don't even know the difference between the address bar and the search bar of their browsers.

Installing this extension is a no-brainer though and could help any of us remember to edit the pages we knew we could but perhaps didn't think about. Seeing all these wiki providers come together to build a common standard is particularly inspiring.

It works something like this: Let's assume your lifestream aggregator supports idAuth. From within your aggregator, you specific a unique identifier that you'll use around the web (such as OpenID or email address -- it is important to note that while idAuth has low-level support for identity systems such as OpenID, it is using them only as an identifier, not for authentication). You'll also specify some keys for use, such as "blog comments" or "readwriteweb.com blog comments" or "photos." These details are then set in a cookie.

When I add data to a service -- which would also support idAuth -- it searches for an idAuth cookie and then looks for an appropriate key. For example, ReadWriteWeb would search for a "readwriteweb.com" key or a "blog comments" key, Flickr might search for a "photos" key. Once it finds the right key, it packages the data you've entered and pushes it back to your lifestream aggregator (whose information is included in the idAuth cookie) in XML format, which the aggregator compares to your cookie to make sure the keys match and the data is valid. You can think of this as something akin to the trackbacks that blogs use to notify one another of links, with a layer of identity verification.

It might seem that something like idAuth wouldn't be necessary for Flickr -- whose stream you verified as yours when you added it to your aggregator -- but the idea here is that your lifestream aggregation service can collect data you create from anywhere on the web and verify that it was indeed you that created it. And you don't have to add a million feeds into your aggregator (nor do they have to bake in support for a million different services), to get it done. That would be supremely useful for something like blog comments, which are very fragmented.

Brady hopes that moving forward he can gain the support of some current lifestream aggregators, then start creating libraries for popular languages and plugins for popular blog clients. His entire proposal, which goes much more into depth about the technical specifics than this post, can be downloaded on his blog in PDF, Word, and OpenOffice formats.

We asked last month how vulnerable Google is in search and the leveraging of standards-based structured data may be the most obvious approach to improving on the search industry's current best practices. As Tim Berners-Lee said just weeks ago the time for the semantic web is now.

Here's one example of what that could mean: Today, a web service might work very hard to scour the internet to discover all the book reviews written on various sites, by friends of mine, who live in Europe. That would be so hard that no one would probably try it. The suite of technologies Yahoo! is moving to support will make such searches trivial. Once publishers start including things like hReview, FOAF and geoRSS in their content then Yahoo!, and other sites leveraging Yahoo! search results, will be able to ask easily what it is we want to do with those book reviews. Say hello to a new level of innovation.

This has been really geeky stuff for a long time, with little market traction and a whole lot of promises from academic research and outlying innovators. That will now change.

The basic idea behind Semantic Web technology is that by signaling what kind of content you are publishing on an item-by-item or field-by-field basis, publishers can help make the meaning of their text readable by machines. If machines are able to determine the meaning of the content on a page, then our human brains don't have to waste time determining, for example, which search results go beyond containing our keywords and actually mean what we are looking for.

Publishers will now be able to clearly designate content on a page as related to other particular content, as business card type information, as a calendar event, a review or as many other types of content. It will make Yahoo! a lot smarter and should shake up the world of Search Engine Optimization and web publishing, a lot.

Who Does the Markup?

Many observers of the Semantic Web, including us at times, have argued that it's unrealistic to expect web publishers to markup their own content and that a more realistic path to market for technologies based on semantics is to build applications that can parse the semantics out of other peoples' content from outside.

In my interview with Mark Zuckerberg last week, for example, the Facebook CEO expressed disinterest in participating in the Semantic Web. I didn't publish it in the interview, but he indicated such a move would be up to a third party site organizing information via the Facebook Platform if it was going to happen at all. He will probably change his tune now, as adding hCard support to Facebook public profiles will now be a no-brainer. Other publishers will be faced with similar questions.

Semantic web markup will quickly become standard practice though for all CMS/publishing systems and we'll wonder what we ever did without it or why it seemed so hard.

Google Will Soon Follow

This move by Yahoo! will likely be followed up by Google, it's just too much opportunity for any search engine to pass up. Semantic markup is like a content-level site map, something all the search engines have agreed on a standard for already. Semantic web technology is next. There will be big job opportunities, more than there are for SEO in the short term, for people who can help publishers implement Semantic Web markup retroactively and into the future.

The Semantic Web was one of a handful of topics that we identified as key themes for the coming year in our RWW Toolkit for 2008. Check that toolkit out for resources you can use to follow this important topic as it unfolds.

The first two apps to engage with Fire Eagle is Dopplr, the super-hip social-travel app, and Danger Day, a service for updating your location on Twitter. Others are ramping up quickly, though Fire Eagle is still invite-only. We've got invite URLs posted at the end of this post, knock yourselves out. The Yahoo! Group for developers interested in Fire Eagle is here.

First, Yahoo! put privacy right out front. Many people want their data to be portable from service to service and many people want that to include their location data from mobile or other interfaces. I personally don't want my location broadcast automatically, at all, to anyone thank you very much. Fire Eagle has privacy and user control of data written all over it.

Users have the option to hide themselves with a single click, they can click to purge all their data from the Fire Eagle databases, the service even lets you select how often you'd like to receive an email reminding you that it is tracking your location as asking you to confirm that you want tracking to continue. By default you're emailed once a month for consent to be reconfirmed! Hello trust building measures! It's almost enough to make me interested in exposing my location, selectively.

Second, the way Yahoo! is developing its Platform is great. It's offering API kits in five different programming languages, it's got user authorization protocols already available for web, desktop and mobile apps and it's using the open standards community built oAuth to facilitate faster, more secure mashups. We wrote about oAuth's launch here and Google is also using it extensively in OpenSocial. This aint no cry-baby do it my way or I'm taking my ball and going home framework like the Facebook platform. This is leveraging universal open standards.

Standards based platform plus strong privacy equals the best scenario I can imagine for a location tracking service. We'll see what kinds of innovative applications get built on top of it.