But none of them are as drop-dead good-looking - or as customizable - as AmpliFeeder, a free, open-source distributed social activity aggregator. The only major drawback: It's the kind of web app that needs to be installed on a server. But a hosted version is in the works, and the screen shots prove it's so worth the effort.

Perhaps best of all, it'll automagically import any of the services you link to through FriendFeed, making your new site setup time about 30 seconds:

Creator Jon Paul Davies has uploaded several other interesting and useful videos on using AmpliFeeder.

Certainly, the best features of the product for the end user are its slew of gorgeous interfaces. The themes differ not just in color/fonts/ridiculous design doodads; they mix up the information design itself.

For example, if the user prefers straight-up streams of data, there are several sexy options such as this:

For those who like their data with a little more segregation between services, there are themes such as these:

And then, a couple themes go all-out on the visualization:

Best of all, there's a custom CSS function that graphic designer-type users can use to style themes to their hearts' content.

Anyone can comment on posted content as comments "live" on the AmpliFeeder site. Items can be hidden or deleted. On the back end, a graph report shows what percentage of content comes from which services. AmpliFeeder also has its own microblog function; posts appear on the AmpliFeeder page and are pushed to the linked services. And AmpliFeeder can also generate a nice, data-portable XML file for users to backup all their social stream's data; XML files can also be used to restore data.

Burton Group analyst Mike Gotta wrote back in the mists of time (May 2007), "The term [lifestream] actually goes back to at least 1997, when Eric Freeman and David Gelernter saw it "as a network-centric replacement for the desktop metaphor. As their project page (last updated in 2000) at Yale put it: 'A lifestream is a time-ordered stream of documents that functions as a diary of your electronic life; every document you create and every document other people send you is stored in your lifestream.'"

Since then, lifestreaming has become the must-have method for communicating with one's public. Look at Modernista; look at Skittles. Better yet, look at what independent designers and other creatives are doing with the medium. And all this time, aside from complicated and costly proprietary solutions, most lifestreaming sites have displayed unbearably ugly UIs.

Kudos to Davies for making a functional lifestream aggregator that looks like a real website. In fact, we imagine that since the current state of the web has given rise to more and more personal and enterprise/corporate sites of the lifestreaming persuasion, Davies' creation comes at a perfect time for designers and webmasters alike.

UPDATE: For our super-smart commenters, here's what we wrote last year about Sweetcron. Yup, I'm new here.

"In the quest to make the Web more structured, Yahoo Local now supports the hCalendar, hCard, and hReview microformats on almost all business listings, search results, events, and reviews, said Yahoo's Andy Baio (below) during a workshop on decentralizing data at Supernova. [...]"

Am now in the 'Engaged Markets' workshop at Supernova and will blog that soon.

Update: I wrote up some thoughts regarding the microformat news on my ZDNet blog, in a post entitled Yahoo and Microsoft support microformats - what chance Google?.

BBC TV Listings competition

Backstage held a TV Listings competition in July 2005. The idea was for contestants to rethink how to present BBC TV and Radio listings, using the TV-Anytime data format for 7 day listing data and other relevant RSS feeds. As described on the Backstage blog:

"The way people watch television - and choose what they are going to watch - is changing. More channels, new delivery platforms, new ways to consume television programming...

For the first backstage.bbc.co.uk competition, we are offering you the opportunity to innovate and build prototypes that demonstrate new ways of exploring the BBC TV schedule."

In October 2005 the winners were announced. An application called mightyv, developed by Leon Brocard and Leo Lapworth, was the ultimate winner. Mightyv is a personalized tv scheduling service, which also enables users to track their friends schedules. It features extensive search, which can be subscribed to with RSS feeds. It also has many options for viewing tv and radio listings, customizing channel lists, tags, ratings, recommendations. The BBC calls this "findability".

Mightyv isn't restricted to BBC data - it also uses data from an independent website called bleb.org, for non-BBC channel data. The developers note that the bleb.org data isn't "as rich in meta data (genre/group/series) as the BBC so you will see a marked difference in the search results because of this." So obviously the more meta data you can provide in your RSS feeds and APIs, the better it is for developers and users!

Mightyv: step by step

What makes much of mightyv possible is the metadata rich '7 Day Listing Data' RSS feed. It's a daily snapshot of BBC TV and radio 7-day listing information and is in the TV-Anytime data format (specifications developed by the TV-Anytime Forum, an industry body).  

As it states on the main page, the data is originally sourced from the BBC's scheduling system but undergoes some manipulations in order to provide TV-Anytime features. The BBC eventually provides it to developers in XML format.

The following naming structure is used for the XML files: YYYYMMDDCCCC_TT.xml, where:

Let's look at the main XML files, as they relate to Mightyv.

ProgramInformationTable

The program I'm going to use for my example is a BBC World Service radio program called 'Go Digital'. It sounds like something readers of Read/WriteWeb would be interested in! For this example, I'm going to use the episode of Go Digital that was broadcast on 21 February 2006 at 20:05.

We find out more about Go Digital in an XML file labeled 20060221BBCWrld_pi. The 'pi' stands for 'ProgramInformationTable' and it contains the metadata of the program - title, synopsis, whether it's subtitled, etc:

The key piece of data that we're interested in for now is the unique program ID on line 1: 'crid://bbc.co.uk/1736500958' (note: the BBC design policy is to give each piece of content a unique identifier). Here's how it's described in the BBC help documentation:

"Each programme is allocated a CRID (Content Reference IDentifier), that represents unique programme content (repeated material would generally have the same CRID). All the tables are linked together using the CRIDs. At the moment the CRIDs are not guaranteed to be unique for all time."

Returning to the ProgramInformationTable. The program title and synopsis are listed on lines 3-4. Genre data is listed on lines 5-22, followed by other metadata including the webpage for the 'Go Digital' program - listed on line 28 under the 'MediaURI' tag as http://news.bbc.co.uk/1/hi/technology/1478157.stm. That webpage has links to the radio show in MP3 format and other relevant information. Screenshot:

ProgramLocationTable

But we're still looking for one vital piece of data - the time and date of the program. Nowhere in the 'ProgramInformationTable' XML file is that data to be found. It is in fact in another XML file, named 20060221BBCWrld_pl, which is the ProgramLocationTable. This file describes where and when the content can be found, which of course is the data that drives tv listings.

Line 40 references the unique program id: 'crid://bbc.co.uk/1736500958'. Then on line 43 is the information we're after:

<PublishedStartTime>2006-02-21T20:05:00Z</PublishedStartTime>

Deciphered, the above piece of code means: the Go Digital program will screen on 21 February 2006 at 22:05.

ContentReferencingTable

The ContentReferencingTable is a little more complicated. It provides the precise location(s) where an item of content can be found, which in general means it lists instances of a program. It can also be used to group programs, for example into series or collections.

In our Go Digital example above, here is an extract from the ContentReferencingTable, 20060221BBCWrld_cr:

Line 50 has details of the episode broadcast on 21 February 2006 at 22:05:

GroupInformationTable

The final table in our example contains descriptions of groups. Like programs, groups are also identified by a CRID (Content Reference IDentifier). Indeed the structure and content of the GroupInformationTable is very similar to the ProgramInformationTable, in that it provides the metadata for the groups. Here's an example from a file called groups_gr:

This extract describes a group containing the Go Digital radio program.

There are other XML files provided by the BBC, but our purpose here is just to gain an understanding of the main content components of the Mightyv mashup.

Mightyv display

The end result for the user is that 'Go Digital' displays on the mightyv tv listings webpage, scheduled for 22:05 on 21 February 2006. Below is a screenshot, showing also the Go Digital Synopsis metadata (from ProgramInformationTable) in a pop-up:

(nb: the numbers down the side represent hours, not date)

Caveats and Issues

It's important to note that the Mightyv application is in one sense only as good as the data it gets, which is mainly from the BBC. Backstage is careful to note some caveats and issues with the TV-Anytime data they provide. For example they state:

"This data is generated for experimental purposes only. There are some known issues with repeated programmes occasionally not being allocated the same CRIDs - these issues are being looked at and improved. Accuracy is striven for, but not currently guaranteed! Any feedback on any aspects of this data would be most appreciated."

Once again this is an illustration of who holds the balance of power in a mashup - it's the data owner.

Mightyv functionality

So the TV-Anytime data from the BBC allows Mightyv to provide a fairly straight-forward tv listings service, without having to do too much tinkering under the hood. Of course, the main value in a mashup is in developing something unique and compelling on top of the basic data. So let's take a look and see why the BBC chose Mightyv as the winner of its backstage developer competition.

If the user clicks on the blue underlined 'Go Digital' link on the tv listings page, they are sent to another mightyv page which has further options (if logged in):

Options include rating the program, tagging it, adding it to your schedule. So there are some neat 'web 2.0' features there. Admittedly it has a ways to go to be truly compelling to users, but you can see the potential in Mightyv. Especially when you consider that personalizing your TV-watching schedule is becoming increasingly important, as television integrates with the Web.

For those of you not familiar with Structured Blogging, it's an initiative that gives bloggers the tools to create and syndicate structured information - such as reviews and events. Kind of like what edgeio is doing commercially. Indeed I've always seen edgeio and SB as being highly complementary. I anticipate that a whole raft of other aggregators will emerge over the next few years, to collect structured data over the Web. Imagine an edgeio for reviews, or an open events aggregator. It's all about the edge, baby!

Structured Blogging is also about providing tools for using microformats, so you don't have to be a geek to use them ;-). SB.org currently offers two plugins for the popular blogging platforms Movable Type and Wordpress. In the future, we hope to see SB integrated into hosted platforms like Typepad and wordpress.com.

I'd recommend looking at the new Roadmap if you're at all interested in where Structured Blogging is going. Marc Canter largely created the roadmap, with input from the PubSub and BBM teams. Here's a snippet from it:

"In Part 2 of the Structured Blogging project, we want to enable people to DO things with microformats. Structured Blogging is all about providing end-users with solutions, enabling them to use microformats and microcontent. We're in the era of the 'Live Web' (or Web 2.0), which is about content applications and services that utilize the Web platform. But to make the Live Web work, we need compatible schemas and APIs to mesh all our apps and services together. There are a lot of missing links currently in the Live Web, which is where Structured Blogging can help."

Finally, if you're a blogger wondering: what's in it for me? Check out the new Benefits page. It's a great starting point.

As with the vast majority of feeds or APIs from big companies, the BBC’s come with certain restrictions. For example, backstage.bbc.co.uk is specifically for non-commercial use:

“backstage.bbc.co.uk is for individual developers and designers to build things using BBC content and anyone who has an idea for how to use BBC content in new ways. It is not for big corporates to play around with. backstage.bbc.co.uk is for non-commercial use by the little people.”

It’s also worth noting that the reason BBC was able to release some of their content to external developers is because of their government remit to "build public value".

Nevertheless, backstage.bbc.co.uk is a pointer to how big companies will open up their data in future. So it’s a great case study for what I'm calling Microcontent Design.

Low barriers to adoption and RSS

The first lesson from backstage.bbc.co.uk is to follow existing standards and make it was easy as possible for developers to use their content. The main method that backstage.bbc.co.uk uses to expose its data is RSS feeds. Here is the selection of RSS feeds available at time of writing:

• News Feeds
• Sport Feeds
• BBC Community Feeds
• Travel Feeds
• Weather Feeds
• Entertainment Feeds
• Radio and Music feeds
• TV Channel Feeds

Note though that all of the RSS feeds have only excerpts, not full text. So this is another restriction to backstage.bbc.co.uk. Nevertheless it’s a significant body of content that has been made available, because there are around 500 unique pieces of news posted each and every day from its network of over 5000 journalists.

One of the more interesting mashups that has been developed so far using backstage data is a BBC News/Wikipedia mashup, which mixes BBC news with links to Wikipedia articles. Its developer Stef Magdalinski described it as “a more open [BBC] News Online”. The two main features are that it “regexes out” capitalised phrases and acronyms, tests them against a database of wikipedia topic titles, and if the phrase is a topic in wikipedia then it’s turned into a hyperlink. It also “uses the technorati API to add a sidebar of links to blogs referencing the story”, enabling users to see who else in the blogosphere is talking about the story.

APIs

The BBC knows it's important to release a wide set of APIs that reflect the range of content on bbc.co.uk. At time of writing they have plans for two APIs: Search (BBC Search, best links etc) and Postcoder (Query by geo-location data).

The Postcoder API actually brings to light an interesting fact about the BBC and their content. The geo-location data for the Postcoder API is provided by a third party, the Royal Mail. The problem is if the BBC provides co-ordinates of postcodes, then that would undercut commercial services the Royal Mail has with other parties. This is an issue in Britain, because geo-location is owned by various companies in the UK – whereas in other countries that data is freely available to the public. So the BBC licences the geo-location data and at the time of writing was unable to resyndicate it to third parties via backstage.

Data ownership affects the BBC too

Just as mashup developers are ultimately at the whim of API providers, it’s worth noting that sometimes API providers aren’t the ultimate data owner. In the case of backstage.bbc.co.uk, its inability to license geo-location data has meant delays in rolling out their Postcoder API.

And it’s not only APIs that are affected. A lot of the data on backstage, including RSS feeds like Travel data, is actually provided by third parties. The BBC has to license that data, in a way that lets them resyndicate it via backstage. Indeed a lot of public-facing companies like BBC and Google often don't own the data they provide. 

So ultimately, a lot of times the data owner needs to be convinced of the value of opening up their data via APIs and web services.

BBC backstage Project Leader Ben Metcalfe explained to me how they’ve been able to get data from third parties by outlining the benefits to them of opening data up to backstage. He tells them it will create innovation around their data. Plus with Intellectual Property rights staying with the data owners, they are free to contact a successful mashup producer directly. For example, Ben said that he’s pretty sure TeleAtlas (which owns some of the data used in Google Maps) is keeping tabs on what people are doing with Google Maps.

Ben also told me this is the reason why almost all backstage data is provided to developers on the condition it is used in a non-commercial manner. It's so the actual data owner does not miss out on commercial contracts directly.

As Ben noted:

“A lot of this stems from the fact we are a media organization and not a technology company. Media, in the 21st centuary is about aggregating and buying in content and that's the case with the BBC. We buy in more and more and do in-house less and less. And that's reflected in our APIs - or feeds at least, for the time being.”

So even though the aim is for BBC to create low barriers for adoption, this isn’t as easy as it sounds! A lot of work has to be done to convince data owners of the value of releasing their data into the Web 2.0 world, outside their control.

Second pic: Ben Metcalfe

To be continued...

...microchunking your content, taking advantage of open standards, employing microformats, letting users subscribe to all kinds of RSS feeds, freeing your content via APIs and other means, designing for re-use of information, monetizing it, and more.

Of those things, RSS, XHTML, APIs and microformats are key building blocks.

In the comments to my first post, several people noted that microformats and Structured Blogging are too hard for normal people to use - and that is true currently. But the goal of Structured Blogging is to provide easy-to-use tools for using microformats. Currently there are just two plugins available that achieve this (for Wordpress and Movable Type), but over time there will be more mainstream tools released - such as integration with hosted blog platforms like Typepad and 'instant blogging' web forms such as what edgeio recently introduced. Also some browsers are integrating microformats into their product.

Another concern raised was about the number of microformats that will be released - will a proliferation of microformats hamper uptake? For this I have to defer to Marc Canter, who went as far as to create a 'law' to address this. He calls it Canter's Law #1 and it basically says: support all formats and don't take sides, because the user doesn't care about your geeky format wars. As Marc put it :

"No human cares about what format is supported. Only us. Flickr proved that they could be completely format agnostic and provide a compelling experience to all."

Fred Simmons pointed out in his comment that we need structured microcontent in order to better filter and find the quality microcontent - and keep spam at bay. This to me has always been the corollary of microcontent: having good filters to sort it. It's why I declared 2006 to be The Year of the Filter. Currently tools like Memeorandum, Topix, TailRank, Rojo and Findory are doing a good job of tackling this issue.

Thomas Bate left a long comment, noting:

"This discussion needs to extend a bit further to include REALLY structured blogging via enclosures containing not just audio and video, but images, maps and detailed tabular structured data as well. [...] REAL business involves VERY structured data with no room for ambiguity...detailed product specifications, prices, serial numbers for instruction manuals, etc."

It's an intriguing analysis from Thomas, because he says that "Container design is far from dead, but it HAS moved away from the server-side web page as container." So in Thomas' view, although content has been freed from its website container - there are new containers to take its place in the form of "portable web feed enclosures". He calls it 'datacasting'. Interesting point and I will explore that some more...

Software developer Joel Hoard at Browserless Web wrote an interesting post in response to mine. I especially liked this point:

"My primary problem with structured blogging is that it’s a very powerful concept that doesn’t provide a lot of immediate value to individual bloggers. It’s what in VC terms is a “vitamin product” (as opposed to a “painkiller”) in that while it’s very beneficial, it doesn’t solve an immediate need."

That's so true and is one of the big challenges of the Structured Blogging initiative - to show users where the value is.

Sean McGrath, CTO of Propylon, wrote:

"The future of structured content in my opinion is content tunnelled inside human readable content. *Not* machine readable content that can be converted/published for human readability."

Sean thinks ODF and XHTML will play a big part in this new microcontent world and he has a theory about "Hi XML and Lo XML", which developers should read.

Finally we get to Ryan King's post. Despite the inflammatory title ("Richard MacManus gets microformats wrong"), what it boils down to is that Ryan thinks HTML is more important in the grand scheme of things than XML and its dialects. Said Ryan:

"XML has come no where near matching HTML in terms of distribution and interoperability."

Well firstly I don't see the point of an XML vs HTML us-vs-them conversation - that's format wars and I'm not interested in those. The Structured Blogging initiative actually spits out microformats in both HTML and XML (RSS etc) formats. Plus XML itself has wormed its way into HTML, via XHTML, so the two (xml and html) complement each other well.

Secondly I’m not saying HTML still isn’t really important, perhaps still moreso than XML. However I do think the future of data formats on the Web is XML and its dialects - especially RSS/Atom. As a delivery method, RSS has already proven itself and it will continue to be extended into a more general purpose content format on the Web - look at the developments around Atom for example, or at the things Microsoft is doing with RSS.

OK, that wraps up the responses to my first article. Next up: Microcontent Design in action at the BBC.

Photo: Dion Hinchcliffe (of Marc Canter and the microformats BoF at MIX 06)

"...content will be more important than its container in this next phase.

That's a big shift for old media to come to grips with. Killer apps, such as search, RSS and video-capture software such as Tivo -- to name just a few -- have begun to unlock content from any vessel we try to put it in.

Who needs to bookmark and surf a bunch of Web sites anymore, when you can search or monitor several RSS "feeds" much more efficiently?"

When Associated Press CEO Tom Curley spoke those words in a November 2004 keynote speech to the Online News Association Conference, he also struck at the heart of a paradigm shift in web design - from designing for the page to designing for microcontent. Put another way: when a Web ‘site’, or 'container' to use Curley's lexicon, is no longer necessarily how users will experience your content – what does that mean for web designers? It essentially means taking a microcontent view of design. 

Photo: venegas

As I’ll outline in this series, microcontent design involves: microchunking your content, taking advantage of open standards, employing microformats, letting users subscribe to all kinds of RSS feeds, freeing your content via APIs and other means, designing for re-use of information, monetizing it, and more.

Data sources and formats

"The Semantic Web is just the application of weblike design to data; it will be many more decades before we will be able to say we have really implemented the Web idea in the full, if ever we can."
Tim Berners-Lee, October 2004

While Sir Tim Berners-Lee was referring to the grand notion of the Semantic Web in the above quote, in many ways his vision of applying “weblike design to data” is already being implemented in the form of technologies like RSS, APIs, XML.

XML has largely lived up to its promise of being the data format of choice for the Web 2.0 era. And by far the most widely deployed format is RSS 2.0, which is a loosely structured XML dialect. Sir Tim Berners-Lee would probably prefer that RDF, a much more rigorously structured form of XML, were used instead. But that’s another story! 

Microsoft bullish for RSS, Google for Atom

Microsoft and Yahoo are two big Internet companies putting their weight behind RSS 2.0, as I've documented at length over the last couple of years. But there are also a lot of advocates for Atom, an alternative RSS format that is said to be more extensible. Indeed at the Microsoft Mix '06 event yesterday, Google employee Patrick Chanezon (an Adwords evangelist) said in an interview that Google is "very bullish" on Atom. Patrick said:

"Instead of taking Atom as the rich content model for feeds at the implementation layer, you [Microsoft] took RSS 2.0 - which obliges you to do all kind of translations.  [...] I really think this [Atom] is the future of syndication. At Google we're very bullish for Atom. [...] As Gates said in his speech, feeds usage will skyrocket in the next few years - but Atom is a much more solid format for that kind of growth."

The Microsoft interviewer retorted that RSS has the same "good enough" attribute that drove the adoption of MP3.

Pic: kathy kawasaki

Either way you look at it, RSS (including Atom) and XML are the de-facto formats for data in the Web 2.0 world. If you release your data in those formats, that’s step one in the data design process.

Representing data and designing for re-use of information

Step two is standard ways of representing data, to enable people (and machines) to find and consume it. In an era where a veritable glut of media is available online, from both professional and amateur content sources, it’s become very important to make sure your data is easy to find and use.

Structured Blogging and microformats are two relatively geeky topics at this point in their evolution, but they are significant developments in terms of representing data.

Structured Blogging

Structured Blogging is an initiative launched in December 2005 by small RSS-driven companies PubSub and Broadband Mechanics (disclaimer: I work for the latter). Structured Blogging is a set of formats and plugins that enable blogs to publish different kinds of information - like events, reviews and classified ads - in a 'structured' format, so that aggregators can pick up the data from all over the Web.

It’s that ‘re-use’ of blog content via aggregation that will be where the real value lies in Structured Blogging. As of writing there are no Structured Blogging aggregators available, but a hint at the value that it could provide in future is the independent company edgeio – which was launched in February 2006. Sellers can get their data listed on edgeio’s website, simply by posting an item to sell on their own weblog and tagging it “listings”. Buyers are able to search and find goods and services at the edgeio website. How it works is that edgeio aggregates goods and services data by scanning over 25 million RSS feeds, looking for the tag "listings".

This is a great example of how data you publish on your own weblog, or at a specialist service such as the jobs listing site SimplyHired.com, can be ‘re-used’ by a service like edgeio – simply because of the way the data is marked up. Whether you use Structured Blogging markup, or simple tagging that edgeio will recognize and pick up, either way you are in a very real sense designing your microcontent for re-use.

Microformats

Microformats is the generic name given to any format that builds on XML (X)HTML to provide additional metadata about web objects. This is the definition on the microformats.org website:

"Designed for humans first and machines second, microformats are a set of simple, open data formats built upon existing and widely adopted standards."

A good example of a microformat is hReview, a format that provides a common markup for reviews (of products, services, etc). Check out Phil Pearson's NZ Coffee Review site for an example of hReview in action. Also it's great to see Microsoft embracing microformats, as announced at Mix '06.

It’s important to note that microformats and the Structured Blogging initiative are both open standards and complement one another. The Structured Blogging toolset outputs reviews in the hReview format, for example. So essentially Structured Blogging provides tools to publish structured content, which formats it nicely for users and marks it up with microformats.

To be continued...

Update: microformats actually build on (X)HTML, not XML as I originally wrote in the first sentence of the last section. Thanks Ryan King for correcting me and Phil Pearson for confirming it.