A small startup company called InfoChimps released for sale yesterday three very large sets of data extracted from 500 million Twitter messages. Included in the offering are the senders and recipients of 1 billion @ messages, Retweets and Favorites. We wrote in-depth about the release late last night. This morning we interviewed Iain Dodsworth, creator of the most popular Twitter client, TweetDeck, about the value he might find in that data and the direction he's aiming to take TweetDeck in the future.

Dodsworth: Straight off the bat - an archive of tweets could form the basis of a profiler and that's very interesting. Sentiment analysis (which I am ALL over) requires that kind of base corpus.

RWW: InfoChimps isn't releasing full text yet, but they would do a custom slice if you wanted it.

Dodsworth: It's the historical element that a large number of services are missing and where they will fall flat - analysis based on the last few hundreds tweets is almost pointless.

RWW: I'm curious what "a profiler" might mean to you and what this data could help make possible in those terms.

Dodsworth: For me a true profiler would be akin to the holy grail - we would analyse who a person converses with, who RTs them the most, essentially all interactions. Then we would track activity metrics (how many tweets sent, replies) and then we would analyse language patterns (usage of certain words) to ascertain how they express themselves and pinpoint sentiment. Off the top of my head this could lead to elements of intention prediction and I'm steering TweetDeck to have this kind of very very basic Artificial Intelligence at its heart.

I'm currently researching intent predicition inside high frequency trading systems and it's fascinating and could directly relate to TweetDeck and social media systems/services in general.

[Dodsworth's background is in developing for financial services, at places like Prudential Financial and PricewaterhouseCoopers.]

RWW: What would intention prediction look like in this context? On twitter?

Dodsworth: At its most basic if TweetDeck could predict what the user was probably about to require next, based on current activity, then it could start to collate that data in the background - cross twitter/facebook/linkedin data for example. I'm looking at it right now from a cross-service data gathering perspective where our servers do the gathering and hopefully get around the issues of API limits for example.

This is based on future functionality we're mapping out now which is a lot more complex than looking at someone's profile or seeing how many RTs one of your tweets has.

I'm thinking the scope is full social graph rather than just twitter/facebook.

RWW: I guess I'm having a hard time imagining "what the user was probably about to require next, based on current activity, then TweetDeck could start to collate that data in the background - cross twitter/facebook/linkedin data for example" might look like. Like, if I'm looking at a person's profile, I'd probably like to see their LinkedIn data?

Dodsworth: Good example...or see how a certain person you're tweeting with right now stacks up against "similar" people you've spoken to - a box could pop up mid-conversation and give you a tonne of metrics on this person. How full of [crap] are they? Are they a social media guru? Would you be wise to tell this person anything sensitive? Based on previous language patterns, is the person you're tweeting with right now probably lying? A bit out there but possible in theory.

Since her unplanned departure from an L.A.-based startup in 2008, Berrent has traveled through eight countries, documenting and publicizing the struggles of those in developing areas through her blog posts, tweets, images, videos, and her own presence at events at home and abroad. From post-Katrina New Orleans to a trash dump in Manila to a monastery in Burma, read on for her story of trying to achieve social good through social media.

I am a strong believer in the idea that the things you do online are meant to facilitate your offline interactions. People are so fast to click a button, and that can be great. Retweeting, forwarding, and Facebook walls are great engagements. But what's more difficult is the donate button. That's the big hurdle and disconnect. I'm trying to provide these inspirational opportunities in timeboxed campaigns. Social media is slowly catching on, but there's a lot of noise. Standing out is hard; it's important to have an offline component.

Berrent was visibly disturbed by what she witnessed at this Manila trash dump, where she saw shoeless children running through piles of debris.

RWW: Tell me about your experiences with Kiva borrowers. What kinds of people and enterprises have you seen? In your opinion, does microlending have a measurable impact on struggling local economies?

Kiva is really unique. It has a lot of power users - more than any nonprofit I've ever seen. One man has made a thousand loans. It's individual stories, and people really connect. You get updates on that person, and people say it's their favorite email of the month. As a microlending company, Kiva is one spoke in the larger wheel of microfinance. On a global scale, it has a very big impact.

Typically, when you go to a village or province, certain industries are prevalent. In a fishing community, maybe the borrower bought a fishnet or a fishing boat. In an area with a lot of bamboo, it's going to be crafts. I worked in eleven branch offices. I met over 40 different female borrowers individually and over 250 in my time there.

I can see that the money Kiva provides makes a difference. Microfinance is a very slow process, and there are gems and sparks of people who break through the poverty cycle. When you see villages changing, it's really something. It's like watching grass grow, but it's really beautiful grass.

This woman is a pig farmer and a recipient of funds from a Kiva-affiliated organization.

RWW: Now you're working on a seven-day, seven-city tour to raise awareness and funds for malaria prevention through bed nets. Where did this idea come from?

It's a city-by-city competition on who can raise the most money for malaria nets, but also an opportunity for anyone to donate who wants to get involved. The tour starts this Saturday night in New York City and continues for the next seven days in Miami, New Orleans, Chicago, Seattle, San Francisco, and ends in Los Angeles on Friday...

I'd just finished Kiva training, and I was going to the Philippines for three months. And all I could think was, "When I come back, I'm going to be thirty." I've honed in a lot on my direction - using the Internet to help people. And what if I could use this opportunity to give back, involving people in different parts of the country - something really ambitious?

I wanted it to be about saving lives. I wanted to say, "I saved this many lives on my birthday." I've done a lot of work in HIV and AIDS; I looked into that and polio and malaria, and that's what stuck with me. The campaign has no administrative fees. One hundred percent of the funds go to malaria... in rural northern Ghana. Providing malaria nets will really be a part of saving lives there.

Berrent met this monk in Burma and spent the afternoon pagoda-hopping with him.

RWW: What needs or gaps do you see in philanthropic efforts online?

I think it's not having a strategy to begin with, not knowing the tools in your toolbox before you start. There's a lot to be said for jumping in and having fun, but nonprofits don't have the resources to play around online. They think it's about getting interns and getting followers and fans without figuring out why a medium is important and how to make it successful for them.

RWW: What's one surprise - good or bad - that you've come across since you started working with Kiva? What did you not expect from this experience, and what did you learn?

I learned that it's much more complicated than the website makes it seem. There's an entire division devoted to foreign exchange currency. The operational cost analysis, the challenges of technology in the developing world, the processes of remittance - it's incredibly complex. There are regional specialists. On the site, you can make a loan in five clicks, but a lot of machinery comes together to make it that way.

RWW: What's next for you? Is there more globe-trotting in your immediate future? How do you think the web will continue to be part of your life and career?

One of the best parts of this past year has been that I've gone through long periods where I didn't have Internet access. That's brought me a heightened and renewed sense of my purpose in the world and my authentic desire to make the world a better place. I'd like to be able to continue to support campaigns - even for-profit ventures - that I believe in, and I think social business is a wonderful intersection of the two.

I want to explore avenues with online and offline components, while continuing to blog and tell stories I'm passionate about.

Follow Berrent's next adventures on Twitter or at her blog.

And all this is just the tip of the iceburg that is Sloane Berrent's fascinating story. For a fuller look at her travels and timeline, check out this list of her nine favorite posts on her blog, The Causemopolitan, covering humanitarianism, her work in New Orleans, the phenomenon of serendipity in international travel, and much more.

Many thanks to Sloane Berrent for the use of her videos and images as well as for sharing her story with us and our readers.

Tim O'Reilly is a publisher of technical books, the organizer of a series of conferences on diverse topics, an investor in web startup companies and smart electrical grid technologies. He's credited with shepharding the term Web 2.0 into public consciousness and he regularly uses his extensive influence to call on technologists to "do something worthy," especially in the face of ecological and political crisis. Now he's brokering meetings of Obama administration officials and bleeding-edge geeks.

O'Reilly is talking to people, but he's helping people talk to eachother as well. He's introducing officials like Vivek Kundra, the new CIO of the Federal government, and Federal CTO Aneesh Chopra to ground-breaking hackers like geek rennaisance man Chris Messina and YCombinator founder Paul Graham. He's bringing together geospatial visionaries and the government officials that provide them the GPS data they work with.

"What I've learned from all these conversations," O'Reilly says,"is about government as a platform. It's not just social media use by government, or government using wikis. No, it's something more profound. How do you think like a platform provider? We've moved our government from a lean vehicle for collective action, and over the last 200 years it has become so strong that it's now 40% of GDP. I want to go back to the original vision of the role of government: a convener of things that we as individuals and companies can't do alone. Standard setting, pilot programs; government providing enabling technologies for citizens to serve themselves.

"This morning we did a call with the White House and some geohackers, talking about what's wrong with government geodata now and how could it be fixed. The government people said we need to translate this into real projects that will appeal to politicians. 'If you fix this kind of geodata then we'll be able to provide this service - street safety, education attainment, public policy objectives,' was what they wanted to hear from the hackers. It's really about social innovation, building better tools for us as a nation to use technology to focus on real problems."

Healthcare, education and innovation policy are the three sectors O'Reilly says have the most momentum when it comes to government as platform.

"The old model," O'Reilly argues, "said we'll build services ourselves or we'll make deals with a few prefered providers that we'll then offer to our customers. This is very similar to what we saw recently in the cell phone market. Rather than providing all the apps themselves, Apple provided a platform and said to developers 'go build on it.' That's where I think the government is trying to go. Instead of offering a website, here's an API [application programming interface]. Can we spark innovation against what we're doing? It's not about picking a provider or partner and then your conduit to the private sector is them, instead its about evangelizing your platform so far more people develop on top of it."

"There are absolutely other companies coming to Washington and saying otherwise, to stick with old model," he says,"but there's an opportunity for government to say if people want to build services on this then we need the data we make public to be granular and timely. We should not be publishing updates once a month. Real time, local, responsive to users - those are new thinking for government. It's just like the 90's when government was discovering websites, now they are discovering web services and we're saying this is what they need to look like."

That conversation will become very public when O'Reilly hosts the Gov 2.0 conference next month. The lineup of geeks and people from the government is already intruiging and O'Reilley has said that some of the holes in the schedule are placeholders for very high-profile speakers who haven't yet sent final confirmation.

"Vivek [Kundra, US CIO] says he wants to make working for government sexy," O'Reilly says. "It's a huge part of our economy and there's a lot of opportunity for entreprenuers. Why are we letting beltway bandits get away with overchanging government to do work? We're missing opportunities to get our best thinking into government planning."

Making work for the government sexy is going to be a very big challenge. If there's a person and a paradigm that just might be able to do it, though, Tim O'Reilly and this vision of "government as platform" might be the right combination.

Photo: O'Reilly at Web 2.0 Expo Berlin, by Adam Tinworth

In Part 2 we discuss: how previously reticent search engines like Google and Yahoo have begun to participate in the Semantic Web in 2009, user interfaces for browsing and using data, what Tim Berners-Lee thinks of new computational engine Wolfram Alpha, how e-commerce vendors are moving into the Linked Data world, and finally how the Internet of Things intersects with the Semantic Web.

RWW: You've been talking about the Semantic Web for many years now. Generally the view is that Semantic Web is great in theory, but we're still not seeing a large number of commercial web apps that use RDF (we've seen a number of scientific or academic ones). However we have begun to see some traction with RDFa (embedding RDF metadata into XHTML Web content), for example Google's Rich Snippets and Yahoo's SearchMonkey. Has the takeup of RDFa taken you by surprise?

TBL: Not really, but the takeup by the search engines is interesting. In a way I was happy to see that, it was a milestone for those things to come out of the search engines. The search engines had typically not been keen on the Semantic Web - maybe you could argue that their business is making order out of chaos, and they're actually happy with the chaos. And if you provide them with the order, they don't immediately see the use of it.

Also I think there was misunderstanding in the search engine industry that the Semantic Web meant metadata, and metadata meant keywords, and keywords don't work because people lie. Because traditionally in information retrieval systems, keywords haven't proven up to the task of finding stuff on the Web. One of the reasons is that people lie, the other is that they can't be bothered to enter keywords. So keywords have gotten a bad reputation, then metadata in general was tarred with this 'keywords don't work' brush. Because a lot of Semantic Web data included metadata, then people thought that with Semantic Web data -- again, that people will lie and won't have the time to produce it.

Google rich snippets example; image credit: Matt Cutts

Now I think there's a realization that when you're putting data online, that people are motivated NOT to lie. For example when your band is going to produce its next album, or when your band is going to play next downtown, you're motivated to put that information up there on the Semantic Web. There's an awful lot of cases when actually data is really important to people; and it's on the web anyway. So I think it's great that some of the search engine companies are starting to read RDFa.

Does this mean that they [search engines] will start to absorb the whole RDF data model? If they do, then they will be able to start pulling all of the linked data cloud in.

Will they know what to do with it? Because when it's data in a very organized form, I think some people have been misunderstanding the Semantic Web as being something that tries to make a better search engine - i.e. when you type something into a little box. But of course the great thing about the Semantic Web is that you can query it, you can ask a complicated query of the Semantic Web, like a SQL query (we call it a SPARQL query), and that's such a different thing to be able to do. It really doesn't compare to a search engine.

You've got search for text phrases on one side (which is a useful tool) and querying of the data on the other. I think that those things will connect together a lot.

So I think people will search using a search text engine, and find a webpage. On the front of the webpage they'll find a link to some data, then they'll browse with a data browser, then they'll find a pattern which is really interesting, then they'll make their data system go and find all the things which are like that pattern (which is actually doing a query, but they'll not realize it), then they'll be in data mode with tables and doing statistical analysis, and in that statistical analysis they'll find an interesting object which has a home page, and they'll click on that, and go to a homepage and be back on the Web again.

So the web of linked data and the web of documents actually connect in both directions, with links.

User Interfaces for Semantic Content

RWW: At the recent SemTech conference, Tom Tague of Thomson Reuters' Calais project suggested that user interfaces for semantic content are key in getting more take-up. With that in mind, I wonder if you've seen some great interfaces or designs for semantic applications in recent months - if so which ones and why did they impress you?

TBL: I think that whole area is very exciting at the moment. The only piece of hacking I've done over the past few years has been on a thing called the Tabulator [a data browser and editor], which is addressing exactly that. Partly because I wanted to be able to look at this data. And now there are lots of different ways that people need to be able to look at data. You need to be able to browse through it piece by piece, exploring the world of data. You need to be able to look for patterns of particular things that have happened. Because this is data, we need to be able to use all of the power that traditionally we've used for data. When I've pulled in my chosen data set, using a query, I want to be able to do [things like] maps, graphs, analysis, and statistical stuff.

W3C Tabulator, a data browser/editor; Image credit: wiwiss.fu-berlin.de

So when you talk about user interfaces for this, it's really very very broad. Yes I think it's important. There's also the distinction we can make between the generic interfaces and the specific interfaces.

There will always be specific interfaces; for example if you're looking at calendar data, there's nothing else like a calendar that understands weeks, months and years. If you're looking at a genome, it's good to have a genetics-specific user interface.

However you also need to be able to connect that data, through generic interfaces. So if my genome data was taken during an experiment which happened over a particular period, I need to be able to look at that in the calendar - so I can connect the genetics to the calendar.

So one of the things I hope to see is domain-specific things for various different domains, and the generic user interfaces. And hopefully the generic interfaces will be able to tie together all of the domains.

Next Page: Wolfram Alpha; e-Commerce and Linked Data

Wolfram Alpha and Natural Language Interfaces

RWW: An interesting new product was launched this year called Wolfram|Alpha, described as a 'computational knowledge engine.' It's kind of a mix between Google (search) and Wikipedia (knowledge), and its key attribute is that enables you to compute something. The founders think that 'computing' things on the fly is something we're going to see a lot of in future. What's your take on Wolfram|Alpha?

TBL: There are two parts to that sort of technology. One of them is a sort of stilted natural language interface. We've seen those sort of natural language queries for years. Boris Katz [from W3C] created a system called START [a software system designed to answer questions that are posed to it in natural language]. I think with the Semantic Web out there, those sorts of interfaces are going to become important, very valuable, because people will be able to ask more complicated things. The search engine has traditionally been limited to just a phrase, but some of the search engines are now starting to realize that if they put data behind them and have computation engines, then you can ask things like 'what's this many pounds in dollars?' and so on. So yes, those interfaces will become important.

Conversational interfaces have always been a really interesting avenue. We've had voice browser work in W3C, that has been an interesting alternative avenue. It's possible that as compute power goes up, we'll see a prolifieration of machines capable of doing voice. It'll move from the mainframe to being able to run on a laptop or your phone. As that happens, we'll get actual voice recognition and pattern natural language at the front end. That will perhaps be an important part of the Semantic Web.

We talked before about what a great challenge the Semantic Web is going to be from a user interface point of view. Conversational interfaces are going to be part of [solving] that. Of course it's also going to be really valuable to have compositional interfaces - for the visually impaired and so on.

Wolfram|Alpha is also a large curated database of data sets. Obviously I'm interested in the big data set which is out there, which is Linked Data. This everybody can connect to. I don't really know a lot about the internals of Wolfram|Alpha's data set. I don't know whether they're likely to put any of it out on the web as Linked Data - that might be an interesting addition. I imagine that quite a lot of it may have come from the web of Linked Data.

e-Commerce and Linked Data

RWW: There have been reports recently that both Google and Yahoo will be supporting the Good Relations ontology and linked data for e-commerce. Companies such as Best Buy are already putting out product information in RDFa. What would be your advice to e-commerce vendors right now, to help them transition to this world of structured data on the Web. The same question could be asked across many verticals, but e-commerce seems like one area which has some momentum right now. Would you advise them just to put out their data as Linked Data?

TBL: Yup! Certainly this year is the year to do it. I've been advising governments to do it and when you look at an enterprise, you find that a lot of the issues are the same. But when you put your data from government or enterprise out there, make sure you don't disturb existing ecosystems. Don't threaten those systems, because you've spent years building them up.

Maybe there's an analogy with when the Web first started and the first bookshops went online. They were more or less a flyer, saying 'hey we have a great bookshop at 23 Main St, come on down!'. Let's say that a person named Joe owned one of these early online bookshops. If somebody had suggested to Joe that he should put his catalog online, Joe would've felt that that was very proprietary data. And he'd be worried that other bookshops would see where he was weak, so they'd be able to advertise themselves as filling that niche he's weak in.

But when his competitors Fred and Albert put their catalogs online, then Joe can check which books people are browsing at Fred and Albert's websites. So Joe would [finally] be pursuaded to put his book catalog up online. But he doesn't put up the prices... until Albert and/or Fred does. And even if catalog and pricing is up there, nobody puts their stock levels online. And there was a period of time when nobody [i.e. online booksellers] had their stock levels up. But people got fed up with ordering stuff that wasn't in stock. So the first book shop to actually tell you about stock levels suddenly was then unbelievably attractive to its customers.

So there's this syndrome of progressive competitive disclosure. This happens when people realize that if you're going to do business with somebody, if you're going to have your partners up and down the supply chain, really it's useful to check the data web - and life goes much more quickly and open.

Best Buy may be what starts the ball rolling [among e-commerce vendors]. Now if I want to look out for what [products are] available, I can write a program to see what there is. If somebody wants to compete with Best Buy, to my program they'll be invisible unless they can get their data up in RDF. Doesn't matter whether they use RDFa or RDF XML, as long as it maps in a standard fashion to the RDF model, then they will be visible.

Next Page: Internet of Things; Conclusion

The Internet of Things

RWW: I'm fascinated by how the Internet is becoming more and more integrated into the real world. For example the Internet of Things, where everyday objects become Internet connected via sensors. Have you been following this trend closely too, and if so what impact do you think this will have on the Web in say 5 years time?

TBL: It connects very much with Semantic Web [and] with linked data. With Linked Data you've got the ability to give a thing a URI. So I can give a URI to my phone, and I can say that's my phone in Linked Data. And also the company that made it can give a URI to the model of the phone. They can also put online all the specs of the phone, and then I can make a link to say that my phone is an example of that product. So now any system which is dealing with me and has access to that data will be able to figure out the sorts of things I can do with my phone, which actually is really valuable. Especially if the phone breaks.

The Semantic Web has already given URIs to things, and to types of things. When the things themselves have an RFID chip in them, then I think it's a very exciting world. One can take that RFID chip, go to the Internet and find out the data about the thing. Whether we'll be able to do that, whether the manufacturers will be open enough to allow me to turn data about the identifier of the thing into data about the thing, is yet to be seen. But it's a very exciting idea.

Pachube, an example of the Internet of Things (see ReadWriteWeb profile)

Similarly, I'd like to be able to scan a barcode and get back nutritional information about what's in - for example - a can of food. But we don't have that yet. To get that sort of thing, which is very powerful, we need to build look-up systems, which allow you to translate an RFID code or a barcode into an HTTP address.

The Semantic Web is a web of things, conceptually. Tying an actual thing down to a part of the web is the last link - the last mile. Give the thing a notion of its own identity in the web.

Conclusion

RWW: The over-riding message in both Part 1 and 2 of our interview with Tim Berners-Lee, is for companies and organizations to make their data available online. Preferably as Linked Data, which uses a subset of Semantic Web technologies. But Berners-Lee noted, in Part 1 of our interview, that he'd even be happy with the data in CSV (comma separated values) format.

It's clear that we've seen a lot of progress in linked data already in 2009. In upcoming posts on ReadWriteWeb, we'll continue to track this trend and explain how organizations can contribute their data.

This was my first meeting with the Web's creator, whose work and philosophy was a direct inspiration for me when I launched ReadWriteWeb back in 2003.¹

Note: the interview will be published in two parts, with Part 1 today on the topic of Linked Data. Part 2 will explore other topics and will run tomorrow.

UPDATE: Part 2 of this interview is now available.

How Linked Data Relates to The Semantic Web

RWW: Earlier this year you gave an inspiring talk at TED about Linked Data. You described Linked Data as a sea change akin to the invention of the WWW itself - i.e. we've gone from a Web of documents to a Web of data. Can you please explain though how Linked Data relates to the Semantic Web, is it a subset of it?

TBL: They fit in completely, in that the linked data actually uses a small slice of all the various technologies that people have put together and standardized for the Semantic Web.

We started off with the Semantic Web roadmap, which had lots of languages that we wanted to create. [However] the community as a whole got a bit distracted from the idea that actually the most important piece is the interoperability of the data. The fact that things are identified with URIs is the key thing.

The Semantic Web and Linked Data connect because when we've got this web of linked data, there are already lots of technologies which exist to do fancy things with it. But it's time now to concentrate on getting the web of linked data out there.

Web inventor Tim Berners-Lee and ReadWriteWeb founder Richard MacManus

How Linked Data Has Evolved via Grassroots

RWW: Linked Data has had a lot of grassroots support, which you mentioned in your TED speech. This is something Semantic Web technologies, such as RDF, have struggled to get over the years. Has the W3C been pushing the more bottom-up Linked Data world, because of the frustration over lack of take-up of top-down Semantic Web?

TBL: A lot of the initial RDF and OWL projects came out of the academic world; and some of them were projects to show what you could do in a closed world. And the files were zipped up and left on a disc. While they were interesting projects, and while the systems were useful systems, the semantic web community maybe missed the point of the 'web' bit and focused too much on the 'semantic'. However the work that's been done in the Semantic Web, the standards, was really valuable. It's relatively recently for example that SPARQL [an RDF query language] has been developed.

Somebody drew an analogy the other day: can you imagine trying to promote a world of databases without SQL? Even though it's not an interoperable protocol, it's just a query language. So similarly, all that's been put into RDF, rdfs and OWL is very valuable to the linked data community.

The Linked Data community tend to use a subset of that [Semantic Web technologies], of OWL for example. But they certainly use SPARQL. So you could argue that really it wasn't ready to be deployed widely.

Linked Data started as a very informal Design Issues note that I put in; it was a grassroots movement from very early on. So yes W3C has been emphasizing the importance of Linked Data. It's been the Semantic Web Interest Group of course, and various [other Semantic Web] activities, which has been pushing it. But also Linked Data has been seized on - a group of people for example put together DBpedia.³ That wasn't commissioned, that was that they just thought it would be a really cool idea.

Graph of Linked Data sets on the Web, as at March 2009

Linked Data and Governments

RWW: In a recent Design Issues note, you urge governments to put their data online as Linked Data (although you'd also be happy for governments to just make available the raw data - presumably so that others can then structure it). What do you realistically expect, for example, the U.S. or U.K. governments to do over the next year? And in the near future, do you foresee different governments interconnecting their Linked Data sets?

TBL: One can't generalize, governments are (like most big organizations) fascinatingly diverse inside them. So you'll find that there are places inside governments where you get a champion who gets linked data and who's just written a script and produced some linked data. So in the UK government for example, you'll find there's RDFa [in the code of its website] for civil service jobs. So if somebody wants to make a database of all the jobs, they can do that very easily.

There are other cases where the easiest thing for somebody to do is to just put data up in whatever form it's available. Comma separated values (CSV) files are remarkably popular. They're exported sometimes from spreadsheets. It's remarkable how much information is in spreadsheets. Or sometimes pulled out of a database and then put up on the web. It's not as good, not as useful to the community, as if Linked Data had been put up there and linked. But the first step of actually putting the data out there is the one that nobody else can do.

Data.gov, a catalog of public data, was launched in May by the U.S. government

The way to go is for government departments to go the extra step and convert [their data] into Linked Data. One of the nice things about Linked Data, when they have a pile of it, is that they could run a SPARQL server on it. SPARQL servers are a commodity product, a solution for all of the people who say 'but actually I wanted to have XML.' A SPARQL server will generate an XML file [and] allow somebody to write out, effectively, a URL for the XML file.

In fact, I don't see why SPARQL servers shouldn't provide CSV files, something which as far as I know isn't in the standards. But I'd recommend it, certainly in government context, because CSV files are what people have and what people want.

So the message [for government] is to use RDF. Linked Data is the backplane, it's the thing that you connect to in both directions. As a [web] producer your job is to make sure that you produce Linked Data one way or another. And as a consumer, there are lots of ways to consume that data once it's out there as Linked Data.

In Part 2 of this interview we discuss: how previously reticent search engines like Google and Yahoo have begun to participate in the Semantic Web in 2009, user interfaces for browsing and using data, what Tim Berners-Lee thinks of new computational engine Wolfram Alpha, how e-commerce vendors are moving into the Linked Data world, and finally how the Internet of Things intersects with the Semantic Web. Read Part 2 here.

Footnotes:

1. The very first sentence written on this blog, on 20 April, 2003, was: "The World Wide Web in 2003 is beginning to fulfill the hopes that Tim Berners-Lee had for it over 10 years ago when he created it."

2. For more on read/write browsers, you can read another early RWW post entitled What became of the Browser/Editor.

3. DBpedia is a community project to extract structured information from Wikipedia; see ReadWriteWeb's profile of this and similar resources.

This is historically important stuff. "The transition from one-way to two-way media is not something that newspapers are used to doing," Ingram told us. "It's a big change."

"The earliest version of community we had was comments on news stories," Ingram told us. "For anyone who runs a blog, you take that for granted; but for us, that was a big step. We were the first newspaper to do that in 2005. It crept up for us; there weren't that many people commenting. Now we're getting five, six, seven thousand comments a day. On good or bad days we can get up to ten thousand comments. [We're sure he gets help dealing with all of those!]

"I like to call that community 1.0 or 1.5, because they all just sit in a big heap at the bottom of the story. It's like a petri dish of a community; it's little micro-organisms that could become community. You see people who reply to each other, good and bad commenters who return, people who assist each other. One thing I want to encourage more is writers responding to comments and using comments as a resource. That's commenting 2.0, I think.

"Community is great because it makes people feel good, democratizes the process, but also delivers value. One of our writers wrote a story, and the comments pointed out that she only talked to one guy about one aspect of the story. She said 'I read the comments and thought F*!@ you. I wrote a story. Go write your own.' But then she admitted it was true, phoned someone else, and updated the story. For me, that's a gigantic win for us and for readers as well. That's where the feedback should be. "

"I've also seen a noticeable change in tone in comments and other interactive forums, like Coveritlive.com. As soon as someone from the paper steps in and makes a comment, the whole tone changes. If you just give people a blank wall and a spray paint can, you get a predictable outcome. But as soon as anyone says we should stick to the topic or knock off the personal attacks, it has a noticeable effect.

"Comments are the base level of interaction. I've been thinking of other ways to enhance that. We've got live blog, a wiki project, and hopefully we've got groups and forums around a particular issue.

"One of the biggest things we need to do is identify and encourage members of the community who are thoughtful, intelligent, and produce comments of value -- encouraging them to contribute more, elevating what they do and suppressing some of the noise. I'm hoping our new Web publishing system that lets people vote on comments will help with that. I'm trying to think of more ways to use the volunteer fire department principle. Identify key members, ask them to contribute more, and incentivize them. Making their comments look different, giving them a title, giving them different tools. There's no way we can moderate all these comments every day, and the only way to do it is take advantage of our community. I think a task or a goal helps a community gel."

Does the Globe use TwItter? It sure does. "I have been using it as a way to connect with people and push out features," Ingram told us. "You can pull Twitter feeds into Coveritlive.com. We did an Oscar one, an Obama visit, covered a shooting in the subway. I was looking for people commenting on Twitter on those topics, pulling in what people say. I've retweeted, approved users, or approved with hashtags. There is a surprising number of everyday people on Twitter; the Mayor of Toronto is on it. But something like that for raw information delivery is always going to be valuable. You may be touching only 1% or .1% of the population, but they are reaching ten times that many people."

Ingram's closing thoughts on the changing media landscape: "Sometimes you do things, like the policy wiki we set up to get people's input on serious issues, the first issue we got a lot of input on and the second one we got a lot less input on. It's the ghost-town phenomenon. Or they are talking about what you want them to talk about but someplace else. You can build a cool night club and tell people about it, but if people don't want to come, if they want to go to an empty warehouse, then that's what they are going to do. As a big media entity, we used to have the audience; now you have to win over an audience to pay attention to you. I don't know how to solve that one either."

Mathew Ingram is an active participant in conversations on Twitter about international media and technology; you can connect with him at @mathewi. Learn more about The ReadWriteWeb Guide to Online Community Management via this link.

Question: How would you describe the current state of early-stage VC in general?

Clavier: Early-stage technology investments have traditionally been made by a mix of business angels, which invest their own money, and venture capital firms, which invest from funds they raise. Over the past four years, a few seed-stage firms have become prominent players in early-stage investing: First Round Capital, Maples Investments, Baseline Ventures, True Ventures, Alsop Louie, KPG Ventures, and my firm, SoftTech VC. In the current environment, all of these firms are actively investing and so are "professional" business angels. More casual angels have disappeared from the market since October 2008, when the public markets started unraveling.

Hornik: I think it is a very tough time for venture capital in general. The public markets are closed up tight as a drum. But more importantly, the financing environment is very tough. It is particularly hard to assume that a company you finance today will be able to remain financed in the future. So venture capitalists are being very careful about what companies they fund, with a particularly careful eye on capital efficiency.

Question: What would you say about the state of VC investing specifically in the digital content or digital content technology space? Do VCs look on this space favorably right now?

Clavier: VCs continue to invest broadly in digital content, technology, services, etc. The bar has just gone much higher on what companies have to demonstrate to show themselves worthy of an investment: traction, revenues, market potential, etc.

Hornik: There is no question a number of exciting companies in the digital media space will have no problem getting financed. But many others will not be able to find backers. There is a general sense that online advertising is slowing down. Because a large number of the digital media opportunities are monetized with advertising, many of the companies coming up for funding will face a fairly skeptical venture community and have a tough time getting funded.

Question: What have been your most notable deals in this space since you started investing?

Clavier: I started investing in the consumer Internet space five years ago, at the very beginning of Web 2.0, and since then have closed more than 50 deals. Five of them got acquired: Truveo, Userplane, MyBlogLog, Kaboodle, and Maya's Mom. In my angel portfolio, I have companies like Mint, Kongregate, and Buzznet. More recently, I invested in Tapulous, maker of the #1 game on the iPhone, as well as Circle of Moms, GetSatisfaction, and SocialMedia. All of these companies have millions of users, but it is still the beginning for them.

Hornik: The partners at August Capital were the earliest investors in such companies as Microsoft, Sun, Compaq, Intuit, Symantec, Seagate, Skype, and many others. I have had the good fortune to invest in such exciting digital media companies as Evite, Tickle, Six Apart, VideoEgg, etc.

Question: Have you done any digital content deals in the past six months?

Clavier: I typically don't talk about recent deals. Funding announcements are most often done when companies launch. In the past six months, I have invested in Outright, Foodzie, and TextDigger, and I am also about to close two deals in digital content infrastructure.

Hornik: We have invested in a couple but, unfortunately, we aren't talking about either publicly. Stay tuned.

Question: Is now a good time to be a VC investor?

Clavier: I personally consider the current environment a great time for investing. Opportunities I see tend to be stronger; there are stronger talent pools around companies; and the general focus is on building sustainable businesses, with revenues being part of the short-term plan, as opposed to an afterthought.

Hornik: I think it is a great time to be a VC at August Capital. We have had the good fortune to invest successfully in a number of really interesting companies in past down economies. We believe that great entrepreneurs are undaunted by the challenging economy. And a number of things make it easy to build a company in these difficult times: plentiful talent, cheaper rents, less competition, etc.

Question: What is the mindset of your investing partners right now, including other VC firms you often invest with? What percentage of them are positive?

Clavier: We are all busy looking at deals and investing right now. It is, however, fair to say that the pace has slowed down a little bit compared to last year.

Hornik: I certainly think that plenty of venture investors feel quite daunted by the market conditions. Not only is it harder to raise money for their portfolio companies, it is harder for them to raise money for their own firms. But plenty of folks out there have seen these up-and-down cycles before and remain enthusiastic about venture investing. I personally remain quite optimistic about the future of venture investing; so long as there is technical innovation, there will be great opportunities in venture capital.

Question: Is VC investing more difficult in the current environment?

Clavier: I would not say it is more or less difficult than it was in the past. The big question for us is, what type of companies are likely to be successful in these challenging times. There is a flight to quality in terms of management teams and a bigger focus on short-term revenue. But otherwise, it is pretty much the same.

Hornik: It is definitely more difficult. If capital is the lubricant of markets, then we are facing some pretty serious challenges. But there will be opportunity to succeed despite the markets. And those firms that have a long history of success will be able to weather the storm far more easily.

Question: What types of digital content solutions or tools are you interested in funding? What's on your wish list?

Clavier: I don't maintain a wish-list of tools or companies, to be honest. My investment strategy is sector-based, within the realm of consumer Internet. That includes social media, gaming, search and discovery, monetization and ad networks, and consumer and cloud infrastructure. I am currently looking at mobile deals as well, on new platforms like the iPhone and Android.

Hornik: I don't really have a wish list. I'm always looking for smart, talented entrepreneurs who can tell me what is interesting. The entrepreneurs know way more than the venture community, so I have always followed their lead.

Question: Has deal flow slowed down for first-round financing? And can we assume that requests for follow-on financings are way up? On which are you spending most of your time these days?

Clavier: Deal flow has slowed down a little bit, but I find what comes my way is of higher quality. I have closed six follow-on rounds in the portfolio in the last six months: companies I had seed-funded that received capital from new investors in a subsequent round, and I see that as very positive. It was a lot of hard work; it took more time than in the past; and valuations were in line with market realities. But these deals got done. I have allocated 75% of my time to my existing portfolio, and 25% to looking at new deals.

Question: Any further thoughts on the subject?

Clavier: We're in the most challenging economic and financing environment of the last decade. I have been an investor for nine years, but I'm certain that we'll see fantastic companies emerge from these difficult times. And I am excited to be involved in the early-stage community that will help build them.

Hornik: I joined the venture business in June of 2000, which was a challenging time in its own right. But I found some very interesting companies to invest in then. And I'm sure I will find some interesting companies to invest in over the next couple of years as well. I'm looking forward to it.

Graeme Thickins is an independent writer, consultant, and blogger based in Minneapolis, MN, and San Clemente, CA. His main blog is www.Tech-Surf-Blog.com, where he recently posted some 23 interviews he conducted at the DEMO '09 conference in early March.

Chris Beard and I first discussed what Mozilla is doing to keep Firefox, its flagship product, competitive in the latest generation of the 'browser wars'. Google, whose headquarters are just up the road from Mozilla and who I also visited on the same day, upped the ante in the browser industry in September last year when it launched a brand new browser called Chrome. Not only that, but Google went out of its way to claim that Chrome represents the next generation of browsers, because (according to Google) it is much better than existing browsers at managing the increasingly sophisticated web apps we see on the Web nowadays.

Beard noticeably bristled at the suggestion that Chrome performs better with heavy duty web apps. He noted that Firefox is also working hard to make highly interactive web apps run smoothly. Regarding Google's claim that Chrome's isolated tab processes mean a more stable browser, Beard replied that Firefox too is very stable and that it doesn't crash much these days. And to be fair, in this author's experience the latest production versions of Firefox have indeed performed much better than they used to. I still get the odd browser crash though.

What's the Vision for Firefox?

But arguments about browser stability aren't going to differentiate the two browsers, Firefox and Chrome, in the eyes of the general public. So I asked Chris Beard to explain Mozilla's vision for the future of Firefox. Beard replied that the vision for Firefox is to help users navigate and manage an increasingly complex world. Beard likened this concept to intelligent agents; and he also used the term 'trusted assistant'. Beard told me that the browser will be "tied to services" - he mentioned the current activity happening in the Linked Data and Semantic Web communities.

Add-ons are a huge part of the current Firefox experience and Chris Beard said that some of those add-ons will become more integrated into the core browser. While that isn't a new trend, I noted that it sounds similar to what Flock has done. Flock is a browser built on the Mozilla platform that integrated many social web elements into the browsing experience (Flickr, YouTube, etc). I suggested that Firefox may want to offer bundles of add-ons, so that users don't have to go hunting around for various individual add-ons. Beard said that yes, this is in the works. He said that users will be able to create add-on "lists" and offer them as a single click to other users - much like Amazon's wish lists. However he noted that there are usability issues to overcome, because some add-ons aren't necessarily compatible with others. He said that currently Firefox has around 8000 add-ons and that we can expect this bundling feature to come out in the next couple of months.

As for other upcoming changes to Firefox, Beard told me that many aspects of the current Firefox experience could be in the cloud - for example bookmarks and the "Awesome Bar" (Mozilla's term for its adaptive learning URL bar). Beard said that portability of the user experience is important in this era of the Web and so they'll be looking to offer certain functionality and data in the cloud.

Another part of Mozilla's strategy for Firefox going forward is to integrate aspects from some of its associated products, such as Ubiquity (an experimental Firefox add-on that gives your browser a context sensitive command-line - see ReadWriteWeb's most recent write-up) and its sync product Weave (our write-up). Beard told me that all Mozilla products are designed to be extended, but this may include making them part of the core Firefox browser. Ubiquity, for example, may end up being baked into Firefox in the future.

In my next post, we'll explore Mozilla's strategy for Fennec (its new mobile browser) and we'll look at recent developments in other Mozilla products such as Ubiquity and Weave.

Hon's vision of the future is sci-fi influenced, cross-platform and web-native. He mocks the "urban games" of online hipsters but believes there will soon be a layer of "Harry Potter ether" that we can dip in and out of while we're walking to work.

Making Books a Different Animal

The Telling Stories project transformed the work of six UK book authors into six different web experiences. Hon said the authors were mainstream writers whose reactions ranged from indifferent to bemused when they were first approached. After participating, all six are now enthusiastic to do more on the web, he said.

Hon's favorite of the six parts was a mystery thriller written about the streets of London that his company transformed into a Google Maps overlay; the map marker became a flying first-person narrator for the bird's-eye readers. Book chapters unfolded as map annotations.

Another author's book was serialized into 140 character abridged lines and delivered over months to followers on Twitter.

The whole Telling Stories project has been applauded as a great example of book publisher Penguin boldly stepping into a new medium. Hon says the authors were assured that visiting emissaries from the internet had not come to destroy them.

The project has brought the authors creative opportunity and substantial exposure. Personalized social serendipity service StumbleUpon has brought in half of the traffic to Telling Stories, Hon says; sometimes up to 10,000 people will Stumble on to the site on a seemingly random day.

Those visitors are encouraged to jump media and buy the full dead-tree version of the web-ified stories. Hon says though that he thinks the division between media types will become much less clear in the near future.

The Future of Stories

This CEO and I didn't talk much about monetization - emergent forms of creativity shaking up the old are more exciting. We didn't tackle the debasement of literature by Twitter because Twitter's awesome potential is more interesting.

We talked just hours after the iPhone OS 3.0 announcement was made and Hon was excited that the new Bluetooth connectivity could mean a vastly improved interface for glucose monitors, for example. He said that developments like this could be the stepping stones toward a future of ubiquitous computing.

"Soon people will realize that there is no 'mobile internet' - there is only the Internet," he says. "And stories are everywhere." Hon says web content today is like the early days of TV, when all anyone could think to do was broadcast actors from the theater in the new medium. But new types of media enable fundamentally new types of content and experiences.

For example, we're just beginning to learn how to leverage the web's social connections, Hon says. He points to the first iteration of "urban games" as something rudimentary that won't last: groups of people organizing online to meet in person dressed, let's say, as Pac-man characters, running through city streets and posting videos of their adventures on YouTube. "Those games ask people to get up and do something they don't really want to do," Hon says.

Instead, he believes that the future of interactive story telling will be pervasive - it will be available throughout your typical day. Walking to work, even while at work.

"I have no idea what we can produce in this medium," he said, "but I think it's going to be like turning the whole world into Disney Land."

Just remember, Dan, how much free time you said you discovered when you quit playing World of Warcraft. Turning the whole world into Disney Land is nothing to take lightly. That said, I'll see you when we meet up in the Harry Potter ether. I won't be surprised if you and your team help build it.

With the number of Internet-enabled devices particularly mobile phones soaring, very few IP addresses remain vacant, and with only about 20 per cent of the world connected to the Net, that's a problem. And consumers, if you think this doesn't affect you, think again. That latest gadget you bought - is it IPv6 compatible?

To fully understand IPv6 we need to take a look at TCP/IP and this means a quick trip back in time.

It all started way back in 1969, when the Advanced Research Projects Agency Network (ARPANET) was using a transmission protocol known as the Network Control Protocol(NCP) to transmit data across networks. Protocols, if you think of them as languages, are needed so that networks and computers can talk to one another.

Expensive, cumbersome and slow, NCP was found to be limiting and in 1973, the U.S. Defense Advanced Research Projects Agency (DARPA) initiated a research program, known as the Internetting project, to develop a better communication protocol.

The networks which emerged from this research became the basis for what we know as the Internet, and the protocols developed during this time became known as the TCP/IP Protocol Suite.

At its most basic level, the IP part ensured packets were routed to the right place by providing unique identifying numbers to all hosts connecting to the network, and the TCP part managed the transfer of that data.

On January 1, 1983 NCP was deemed obsolete when the ARPANET switched over to the new TCP/IP protocol suite, and as a result, marked this date as the official birth date [for some] of the Internet.

Getting to V1 from V6

According to the Living Internet, after Vint Cerf and Robert Kahn designed TCP/IP, DARPA contracted with three sites to develop operational versions: BBN, Stanford and the University College London, and four increasingly better versions of TCP/IP were developed: TCPv1, TCPv2, which then split into TCPv3 and IPv3. Stability finally arrived with TCPv4 and IPv4; the standard protocol we know and use today.

IPv4 uses 32-bit addresses, which limits the address space to 4,294,967,296 (232) possible unique addresses. But, as some of these are reserved for specific purposes, it reduces the total number available.

IPv6 with its 128 bit addresses increases the number of potential unique addresses to 3.4e+38 (a little bit more than 340 trillion, trillion, trillion). Additionally, it is designed to rectify issues found with IPv4 such as data security.

IPv6 is expected to slowly replace IPv4, with the two protocol systems expected to run simultaneously for many years.

But, what happened to IPv5?

Typically, the most often asked question when talking about IPv4 and IPv6 is what happened to IPv5? IPv5 was known as an experimental streaming audio/video protocol. According to Raffi Krikorian, a protocol named ST, the Internet Stream Protocol was created in the late 1970's and two decades later revised to become ST2, at which point it was implemented in commercial projects by IBM, NeXT, Apple and Sun. ST and ST2 were already given that magical "5" notes Krikorian. Given it had little to do with the fundamental structure of IP addressing, IPv5 is not commonly recognized.

We're running out of IP addresses

While the establishment of a single networking protocol was an important step toward maintaining order in the then new internetworked world, no one could have guessed the growth of the Internet, nor the number of IP addresses required to cover the ever growing demand.

"My only defense is that decision was made in 1977, at a time when it was uncertain if the Internet would work," Cerf said recently, adding that a "128-bit address space seemed excessive back then."

Watch our video below to get Cerf's take on IPv6 - and why switching over is so important.

Recorded at SMX West 2009 by ReadWriteWeb
Vint Cerf image: Vint Cerf playing Spacewar on PDP-1
Credit: Flickr Joi

McCarthy tends the gears of Techmeme, makes sure the content on the site remains of high quality and helps ensure the inclusion of new and important voices. It sounds like an awesome job and one that has probably never existed before - a half woman, half robot, news gathering machine. How can you get your blog on Techmeme? What's in the future for the site? We asked Megan in the following interview.

Marshall Kirkpatrick: What do you do all day? I imagine you standing next to one of the most awesome news discovery machines available, tending it, making sure it keeps running smoothly, and looking out beyond its reaches to feed it things it hasn't gotten to yet itself. Is that an accurate picture?

Megan McCarthy: That is fairly accurate, actually. I make sure that the news on Techmeme represents an accurate, current, and full overview of what's happening in technology right now. So, that's trimming back stories that aren't relevant, adding in viewpoints that ought to be heard, etc.

Marshall: Can you tell us a little bit about your personal background?

Megan: My personal background is a little varied. Prior to [writing for] Valleywag, I bounced around a few different jobs and places and never really found a niche. I lived in Hawaii for a few years, had various office drone jobs and other gigs to pay the bills (Nanny, bartender, coffee server). But I loved following technology and reading about what was happening in silicon valley - and I've been a news junkie since I was young.

News Selection and Twitter Tips on Techmeme

Marshall: So, did your coming on board "break" the "objectivity" of the site?

Megan: Techmeme is biased and has been so for a while. If you read Gabe's post announcing the addition of an editor, he makes that point.

What do you think, though? What changes have you noticed since I joined?

Marshall: I have noticed no changes to story selection, perhaps less wonky stuff. I've always considered Techmeme a very reliable source of news and I think you're doing a good job continuing that tradition - but there were certainly some people who grumbled about the human touch being formally introduced, an editor.

Megan: I think some of those people might grumble about anything.

Marshall: How can new bloggers get indexed on Techmeme?

Megan: We just introduced a program where people can tip relevant posts to us through Twitter. Anyone can tip any post they think is relevant to us.

Marshall: How is the new Twitter tips program working out? I see a lot of stories go up with thanks to Twitter, quite a lot - is it changing the face of the site substantially? Changing the content?

I see a handful of people getting thanks over and again, I imagine there's limited participation so far but how does the algorithm determine whose tips to accept and whose not to?

Also, a lot of people are sending tips regarding their own stories - is that ok? Even mainstream media outlets.

Megan: I don't think it's changing the content overall. Many of the stories that are tipped are ones which are worthy of a Techmeme headline. Not everything that gets tipped to us gets on the site. There are two situations that I can think of where the tip program does affect the content: It can help surface breaking stories faster, and if there are two similar stories from different outlets and someone cares enough to tip a certain one, that will probably effect which one ends up as a headline on Techmeme.

As for people tipping their own stories... personally I'm not completely opposed to it. If a writer has a breaking story that he or she wants to let us know right away, that's a good way to do it. But, they should keep in mind that their twitter handle will be credited with tipping us to the story. If "Thanks: Marshall" showed up next to every Techmeme headline you get, people might put two and two together and think that you really like your work.

To my knowledge, the identity of the person tipping the story has no effect on whether or not it will show up on the page. It's about the post itself.

Marshall: Well, if shame and loads of people saying "you're an f*ing jackass" was sufficient deterrent to anti-social behavior in social media, then...[indecipherable, record of this part of the conversation lost forever.]

Megan: Ha. Is he though?

Marshall: Oh I'm sure he is. ANYWAY. Is accuracy taken into account on Techmeme?

Megan: Accuracy is absolutely taken into account on Techmeme. That's one of my goals, anyway. If there's a post which has a lot of buzz around it, which turns out not to be true...

Marshall: What does that look like? Are you like "Steve Jobs is NOT out at Apple, I don't believe those reports! Story...gone!"

Megan: Or, a story that says "Steve Jobs NOT out at Apple" gets published next to the earlier, erroneous rumor.

Marshall: Then you yank the false story?

Megan: Either yank it or surround it with stories pointing out *why* it's false. Sometimes the false rumor becomes a story itself and yanking it can be jarring. We want our readers to be able to visit the site and know what's going on in technology - to know what people are talking about. The earlier rumor would probably be replaced as the top story by one with the correct information, but yanking it without giving our readers full context of the overall arc might be a bit jarring.

Marshall: You have to be reading a lot of these stories in great detail. What time does your work day start and end?

Megan: I start around 7:30ish and end later than that. News never stops!

The Future of Techmeme and Other Aggregators

Marshall: So, everyone wants to be an aggregator these days. All the young kids are like "mommy, I'm going to grow up to find recommended stories for an online news publisher."

What kinds of things do you foresee becoming points of leverage for content aggregators and news discovers in the future?

Megan: I think a reliable real-time web is going to have the greatest impact on aggregation services. I'd love to be able to see stories from sites as they're published, without a lag.

I hope that quality, accurate, and speedy stories get rewarded by receiving more attention - and that new voices are discovered and make the media chorus sound fuller and stronger.

You were asking me about my electric sheep dreams.

Marshall: Are you a cyborg?

Megan: Depends on my mood.

Marshall: At least between 7am and 7pm?

Megan: That sounds about right. This is super-nerdy, but reading an overwhelming amount of news is something that I rather enjoy doing.

Thanks to Megan McCarthy and Techmeme for doing this interview and doing the things they do each day - help us find the hottest conversation in technology. We appreciate it. You can find Megan on Twitter as well. Photo at top by Scott Beale

We had the opportunity to speak with Satnam Alag, author of the recently published Collective Intelligence in Action, about what makes for a good recommendation system, where the technology is heading, and why Netflix is finding it so hard to improve its own system.

ReadWriteWeb: In our recent post about Netflix, we identified four main approaches to recommendations: Personalized recommendation: based on prior behavior of the user; Social recommendation: based on prior behavior of similar users; Item recommendation: based on the item itself; And a combination of all three. Do you agree with the four approaches we laid out in our article?

Satnam: Those four categories are pretty comprehensive. I present an alternate classification of recommendation systems in my book. I lay out two fundamental approaches. The first approach, item-based analysis, determines items that are related to a particular item. When a user likes a particular item, related ones are recommended. The second approach, user-based analysis, first determines users who are similar to that user.

Further, there are two main approaches to finding similar items and similar users. For the first, content-based analysis, content associated with the item, especially text, is used to compute similarity. In the second, the collaborative approach, actions such as ratings, bookmarking, and so forth are used to find similar items. For the second, user-based analysis, a number of approaches have been taken, including ones based on profile information, user actions, and lists of the user's friends or contacts. Of course, you can combine any these item/user and content/collaborative approaches to build a recommendation system.

The dimensions of the particular item and user space are helpful in deciding whether to use an item-based or user-based approach. Typically, an item-based approach is used to bootstrap one's application when the number of users is small. As the user base grows, the item-based approach is augmented by a user-based approach.

ReadWriteWeb: Other than Amazon and Netflix, which Internet companies have most impressed you in their implementation of recommendation systems?

Satnam: Other than Amazon and Nextflix, Google News' personalization is my personal favorite. Google News is a good example of building a scalable recommendation system for a large number of users (several million unique visitors per month) and a large number of items (several million new stories every two months), with constant item churn. This is different from Amazon's, whose rate of item churn is much lower. Google decided to use collaborative filtering for its recommendation system mainly because of its access to the data of its large user base and because this same approach could be applied to other applications, countries, and languages. A content-based recommendation system perhaps could have worked just as well, but may have required language- or location-specific tweaking. Google also wanted to leverage the same collaborative filtering technology to be able to recommend images, videos, and music, for which it's more difficult to analyze the underlying content.

Among start-ups, my personal favorite is the one we are developing at my current company, NextBio. It's not available yet but should be next month. The key point about this particular recommendation engine is its strong use of an ontology, similar in concept to tags, to develop a common vocabulary for items and users. The system then makes use of profile information and user interactions, both short- and long-term, to provide recommendations. The system leverages both item- and user-based approaches.

ReadWriteWeb: What commercial opportunities do you forsee with recommendation systems over the next few years?

Satnam: A good personalized recommendation system can mean the difference between a successful and a failed website. Given that most applications now invite users to interact and to leverage user-generated content, new content is being generated at a phenomenal rate. Showing the right content to the right user at the right time is key to creating a sticky application. I would be surprised if most successful websites did not leverage recommendation systems to provide personalized experiences to their users.

ReadWriteWeb: Your book includes a discussion of collaborative filtering. Can you tell us a bit about how this fits into the overall picture of recommendation systems?

Satnam: In recent years, an increasing amount of user interaction has provided applications with a large amount of information that can be converted into intelligence. This interaction may be in the form of ratings, blog entries, item tagging, user connections, or shared items of interest. This has led to the problem of information overload. What we need is a system that can recommend items based on the user's interests and interactions. This is where personalization and recommendation engines come in.

In my book, I take a holistic view of adding intelligence to one's application, a recommendation engine being one way to do it. The book focuses on both content-based and collaborative approaches to building recommendation systems. It focuses on capturing relevant information about the user, information from both within and outside one's application, and converting it into recommendations. One of the things you mentioned in your write-up on recommendation systems is that you would like to apply such a system to your website to recommend things to users. Someone reading my book should be able to create such a system using the techniques I demonstrate.

Next Page: Satnam's thoughts on the Netflix Prize and whether the 10% mark will ever be reached.

ReadWriteWeb: Netflix is offering $1 million to the team that can improve its recommendation algorithm by 10%. It's been over 2 years now, with the leading company at 9.63%. There is some skepticism, though, that 10% will be reached anytime soon, because now the contestants are making only incremental progress. Do you expect the 10% mark to be reached soon?

Satnam: Netflix's recommendation engine, Cinematch, uses an item-to-item algorithm (similar to Amazon's) with a number of heuristics. Given that Netflix' recommendation system has been very successful in the real world, it is pretty impressive that teams have been able to improve on it by as much as 9.63%. Of course, the Netflix competition doesn't take into account speed of implementation or the scalability of the approach. It simply focuses on the quality of recommendations in terms of closing the gap between user rating and predicted rating. So, it isn't clear whether Netflix will be able to leverage all of the innovation coming out of this competition. Also, the Netflix data doesn't contain much information to allow for a content-based approach; it's for this reason that teams are focusing on collaborative-based techniques.

The challenges to reaching the 10% mark are:

Skewed data: The data set for the competition consists of more than 100 million anonymous movie ratings, using a scale of one to five stars, made by 480,000 users for 17,770 movies. Note that the user-item data set for this problem is sparsely populated, with nearly 99% of user-item entries being zero. The distribution of movies per user is skewed. The median number of ratings per user is 93. About 10% of users rated 16 or fewer movies, while 25% of users rated 36 or fewer. Two users rated as many as 17,000 movies. Similarly, the ratings per movie are also skewed: almost half the user base rated one popular movie (Miss Congeniality); about 25% of movies had 190 or fewer ratings; and a handful of movies were rated fewer than 10 times.

The approach: The winning team, BellKor, spent more than 2,000 combined hours poring over data to find the winning solution. The winning solution was a linear combination of 107 sets of predictions. Many of the algorithms involved either the nearest-neighbor method (k-NN) or latent factor models, such as SVD/factorization and Restricted Boltzmann Machines (RBMs).

The winning solution uses k-NN to predict the rating for a user, using both the Pearson-r correlation and cosine methods to compute the similarities, with corrections to remove item-specific and user-specific biases. Latent semantic models are also widely used in the winning solution.

The BellKor team found it important to use a variety of models that compensated for each other's shortcomings. No one model alone could have gotten the BellKor team to the top of the competition. The combined set of models achieved an improvement of 8.43% over Cinematch, while the best model -- a hybrid of k-NN applied to output from RBMs -- improved the result by 6.43%. The biggest improvement by LSI methods was 5.1%, with the best pure k-NN model scoring below that. (K for the k-NN methods was in the range of 20 to 50.) The BellKor team also applied a number of heuristics to further improve the results.

The BellKor team demonstrates a number of guidelines for building a winning solution to this kind of competition:

• Combining complementary models helps improve the overall solution. Note that a linear combination of three models, one each for k-NN, LSI, and RBM, would have yielded fairly good results, an improvement of 7.58%.
• A principled approach is needed to optimize the solution.
• The key to winning is building models that can accurately predict when there is sufficient data, without over-applying in the absence of adequate data.

The final solution will be along the same lines, combining multiple models with heuristics. Contestants will probably reach the magic 10% mark in the next year or two.

ReadWriteWeb: Some people think the 10% mark can't be reached with algorithms alone, but that the "human" element is required. For example, ClerkDogs is a service that hires actual former video-store clerks to "create a database that is much richer and deeper than the collaborative filtering engines." It's a similar approach to that of Pandora, which has 50 employees who listen to and tag songs. How far do you think algorithms can go in making recommendations?

Satnam: Recommendation systems are not perfect. A number of elements go into making successful ones, including approach, the speed of computing results, heuristics, the exploration and exploitation of coefficients, and so on. But it has been shown in the real world that the more personalized you can make recommendations, the higher the click-through rate, the stickier the application, and the lower the bounce rate.

Using humans to form a rich database for recommendations may work for small applications, but it would probably be too expensive to scale. I don't see them competing against each other, human versus machine. Even with human/expert recommendations, one first needs to find a human/expert with tastes similar to those of the user, especially if you want to go after the long tail.

Derek Slater now runs public policy for Google, but before that he worked with the Berkman Center and the Electronic Frontier Foundation.

• Google's philosophy that copyright is supposed to benefit the public.
• Support for rights holders and ways to get them high Google rankings.
• Google's history, starting in a garage and how their innovation thrived.
• Copyright issues and Comcast.

An Interivew with Google Policy Analyst Derek Slater from alex williams on Vimeo.

In this post we explore last.fm's feature set, how it compares to MySpace Music, and what we can expect to see from last.fm in the near future. Richard Jones also discusses how last.fm has managed to avoid the legal difficulties that have plagued Pandora.

RJ: Well, we liked the Last.fm/YouTube mashup Tim Bormans made so much that we hired him! Internally we've been working on multi-tag search which is available at our Playground, which allows you to search for music using multiple genres (folk + rock + gabba, etc); also on Playground we've been tweaking our Musical Soulmates app, which you could possibly consider the prototype for a future Last.fm dating service ;)

And of course, scrobbling continues to be an almost default setting for new music services now - from Hype Machine to Muxtape (RIP) to Blip.fm to the new version of VLC (which has 4 million downloads already since launch a couple of weeks ago), it seems like everything has to integrate scrobbling now. Great for us obviously, and great for our recommendations which will continue to improve as more people scrobble. People are scrobbling at a rate of 800 million times a month currently.

RWW: In terms of features, Pandora is similar to last.fm, in that both services have great recommendations and allow the user to discover new music. Both are streaming music services, yet it just seems to be Pandora - of all the 4 major services we've discussed so far - that has been having legislation issues. Can you clarify for our readers how last.fm has managed to avoid those sticky issues, when Pandora hasn't? I think many people are confused about that.

Last.fm is about more than just online radio. We've got millions of tracks available free-on-demand as well, and beyond that there's a massive social network element to the site. We also offer videos, the biggest events listings on the web (personalised to your taste), and our own audio and video content under the Last.fm/Presents banner. So the point is, online radio is only one of the things we do, so the legislation affects us only in one particular area of the Last.fm experience. As online radio is pretty much the single focus for Pandora, it understandably hits them harder.

The wider issue here, of course, is that royalty rates are high, and the debate around this needs to continue so we can reach a mutually beneficial and economically workable resolution. We don't want to see legitimate online broadcasters stifled by this - it's not good for music fans, artists or the wider music industry.

RWW: Lastly, one of the most interesting aspects of online music is its ability for new artists to be discovered. It's something MySpace has done well in the past, but we get the sense the perfect solution hasn't been found yet. As our own Marshall Kirkpatrick asked recently:

"How about a service that scans my iTunes library and my online listening history, determines my genres of interest and then never plays music from artists I've already listened to. Or makes sure to play some that I haven't."

Does last.fm have a feature like that coming up? ;-)

RJ: Do you think Myspace has done this well in the past? As you can tell from my reply to your first question [see Part 1], I would argue that Myspace has been a rather difficult site to navigate unless you're after popular stuff - and it's possibly going to be more of the same on Myspace Music, as the major labels jostle for frontpage real estate and push more indie/obscure music off the page.

Our recommendation system is being constantly refined to give music fans the best music discovery service on the web. I think we've got that covered. What's equally important is that these artists being discovered, if they're Long Tail or DIY, get the same kind of licensing and royalty breaks that more established artists get, which is why our Artist Royalty Program exists.

We've been doing this for 6 years, as I said, which is why it's kind of funny to be talking about this now because of Myspace Music. They're just catching up to free-on-demand after we pioneered the model almost a year ago. Now they've got to figure out how to make it easy to discover music that suits your taste (sharing playlists is one thing, but how do you find that music to share in the first place?), which we've been doing since 2002. After that, maybe they'll start paying unsigned artists. I would hope all this will come to Myspace Music at some point in the future - but it's happening on Last.fm now.

See also: Interview With Last.fm Founder Richard Jones: Part 1, The Competition and Part 2, Business Models