A small startup company called InfoChimps released for sale yesterday three very large sets of data extracted from 500 million Twitter messages. Included in the offering are the senders and recipients of 1 billion @ messages, Retweets and Favorites. We wrote in-depth about the release late last night. This morning we interviewed Iain Dodsworth, creator of the most popular Twitter client, TweetDeck, about the value he might find in that data and the direction he's aiming to take TweetDeck in the future.

Dodsworth: Straight off the bat - an archive of tweets could form the basis of a profiler and that's very interesting. Sentiment analysis (which I am ALL over) requires that kind of base corpus.

RWW: InfoChimps isn't releasing full text yet, but they would do a custom slice if you wanted it.

Dodsworth: It's the historical element that a large number of services are missing and where they will fall flat - analysis based on the last few hundreds tweets is almost pointless.

RWW: I'm curious what "a profiler" might mean to you and what this data could help make possible in those terms.

Dodsworth: For me a true profiler would be akin to the holy grail - we would analyse who a person converses with, who RTs them the most, essentially all interactions. Then we would track activity metrics (how many tweets sent, replies) and then we would analyse language patterns (usage of certain words) to ascertain how they express themselves and pinpoint sentiment. Off the top of my head this could lead to elements of intention prediction and I'm steering TweetDeck to have this kind of very very basic Artificial Intelligence at its heart.

I'm currently researching intent predicition inside high frequency trading systems and it's fascinating and could directly relate to TweetDeck and social media systems/services in general.

[Dodsworth's background is in developing for financial services, at places like Prudential Financial and PricewaterhouseCoopers.]

RWW: What would intention prediction look like in this context? On twitter?

Dodsworth: At its most basic if TweetDeck could predict what the user was probably about to require next, based on current activity, then it could start to collate that data in the background - cross twitter/facebook/linkedin data for example. I'm looking at it right now from a cross-service data gathering perspective where our servers do the gathering and hopefully get around the issues of API limits for example.

This is based on future functionality we're mapping out now which is a lot more complex than looking at someone's profile or seeing how many RTs one of your tweets has.

I'm thinking the scope is full social graph rather than just twitter/facebook.

RWW: I guess I'm having a hard time imagining "what the user was probably about to require next, based on current activity, then TweetDeck could start to collate that data in the background - cross twitter/facebook/linkedin data for example" might look like. Like, if I'm looking at a person's profile, I'd probably like to see their LinkedIn data?

Dodsworth: Good example...or see how a certain person you're tweeting with right now stacks up against "similar" people you've spoken to - a box could pop up mid-conversation and give you a tonne of metrics on this person. How full of [crap] are they? Are they a social media guru? Would you be wise to tell this person anything sensitive? Based on previous language patterns, is the person you're tweeting with right now probably lying? A bit out there but possible in theory.

"What we want is to see people use this to build web apps," Infochimps co-founder Flip Kromer told us today. "You take this data, mash it up with any other very large corpus of data with timestamps - and you've got a web app."

This is particular, extracted data though - not the full text of Tweets. "We're trying to be careful," Kromer says, "we are not yet exposing the contents of tweets." And this data isn't cheap if you want the numbers broken out by the hour instead of the month.

This is a very big move because most developers struggle to get access to a large quantity of data from Twitter.

Here's what InfoChimps is putting on sale:

1. Hashtags, links and smiley emoticons used across Twitter on an hour-by-hour basis.


2. @ messages, RT and favorites and who they came from: 1 billion relations, making what the company calls a "conversation metric."


3. A useful if less exciting set of data that will help developers map user ID numbers from search.twitter over to the different ID numbers used in the primary Twitter API. These systems were never merged and it can require a lot of API calls to merge user data.

The company believes it is capturing about 10% of the total data on Twitter right now, but Kromer says that he believes he can ramp that up to 30%.

Data as a Pot of Gold

InfoChimps is a bulk data marketplace with more than 5000 data sets in its catalog so far. The vast majority are free and were added by the company's own staff, but not all. The decades-old polling firm Zogby International, for example, is selling some Iraqi polling data through InfoChimps. Cross-reference that polling data with publicly available data about civilian casualties in Iraq and you can see some interesting patterns, InfoChimps' PR rep Josh Dilworth told us. (Dilworth is known as the most data-savvy PR guy in the Web 2.0 world and also represents Wolfram Alpha and Twine.)

The company hopes that it can sell the data derived from sitting on the Twitter API as a demonstration of the value that this and other data sets have. InfoChimps says it can help companies monetize data that they'd otherwise be paying to serve up through repeated API calls, if at all.

From sentiment analysis (not yet an option with the current InfoChimps data set) to social graph discovery (definitely an option), we've written extensively here before about the impacts that social data could have on business, social and political policies in the future.

John Zogby, founder of polling firm Zogby International, spoke to us at length (in a separate phone interview several months ago) about the value of using online social networks to measure public opinion. "We've been particularly known for innovating and polling new technologies," he said.

"83% of all households are online today and 92% of likely voters, so with online polling we are today about where the country was with telephone penetration when telephone surveys started. Social networking is not as representative as online access [in general] yet, but I'm comfortable with caveats: that you can do a random sampling, so long as you claim that's what your universe is, as long as you don't extrapolate to all Americans, etc. It has tremendous, tremendous value.

"I know that the landline era is coming to an end - not today or tomorrow but we've got to find new and different ways of doing our work. It's the same kind of crossroads as the '70s, when we moved away from the door-to-door and mail-in results to the landlines.

"Online, frankly just like telephone, doesn't have the minority population, but for market surveys you may be looking for a different kind of consumer.

"We know that the landline phone is pushing us away; we know that we can't use the cell phone in the same way; and we know that we've got to reinvent this industry [of measuring public opinion]. What's happening are simultaneous new technologies and at the same time growing penetration of these new technologies. We're riding a bucking bronco."

Use Cases

The conversation metric data that InfoChimps is selling is the most exciting to me. Imagine a third-party app using historical social-conversation data to filter Twitter or other messages based on the strongest social connections that I or other people have. Imagine, for example, social Q&A service Aardvark combining the Twitter Lists API with this InfoChimps data set for a scenario like this: "You have a question about stock options? How would you like us to find a person who knows about that, is regularly conversed-with by people on Robert Scoble's Twitter list of Venture Capitalists and is available right now?" That sounds pretty great to me.

The possible applications are many. "I see Twitter as a data acquisition device for what people talk about and how they relate to each other," InfoChimps' Kromer says.

Right now InfoChimps is selling the hashtag and link dataset for $8,000 and the social metric data set for $9,500. Eventually the company will likely move to a subscription model.

How They Got the Data

How did InfoChimps get the data? The company hits the Twitter Developer API 20,000 times an hour (the standard for developers) but takes big swaths of data each time it does. "I have a priority queue," Kromer told us.

"I can set a search term, and for each search term I can get 1500 tweets per API call. If I get 1500 tweets at a time, then the number of wasted tweets at the end of a series of searches is the smallest. If I'm searching for a term and get less than 1500 results back, then I forecast how long it will take to fill that number of results back up to the maximum and move it down the priority queue accordingly. On the lowest priority I have searches for RT or http. There will always be 1500 results for that. It's only API calls that limit me. As is, it's like a fisherman setting nets: what matters is that dinner is tasty."

Does that sound so hard? Worth thousands of dollars? Here's what Kromer says:

"It's not magic. If you talk to people who use Hadoop and do social networking analysis, this is underwhelming. You take 30 million users, 1 billion links, adorn each link with info at the end of the link and acrue it with the person at the head of the link. That breaks conventional databases; the plumbing is hard. The math is easy but when you do it a billion times, it starts to get interesting. You have to be careful and clever. We plan to do stuff that is structural - a clustering co-efficient true pagerank."

Ultimately it's about specialization and data as a service. "The people we need to come in and connect this info with human beings," Kromer says, "aren't the people who should be wasting their time on the math. And the guys who are good at doing these things should not be building Web apps."

But Can They Get Away With It?

There's some question whether Twitter will allow InfoChimps to sell data based on Twitter data. Kromer says he'd much rather resell the data on a commission than have to do all the work he's done to set up the extraction system. But it was a year ago that InfoChimps caught the eye of people who love data: by releasing a large collection of scraped Twitter data.

The InfoChimps blog post for that read: "Big huge thanks to twitter.com: they have given us permission to share this freely. Please go build tools with this data that make both twitter.com and yourself rich and famous: then more corporations will free their data."

But then Twitter founder Evan Williams asked InfoChimps to take those data sets down until a Terms of Service for them could be figured out. That never happened, and communication between the two companies hasn't progressed very far over the last year.

InfoChimps does not have Twitter's permission to do what it did today, but Kromer says Twitter hasn't contacted them either. No one from Twitter headquarters has responded to our request for comment yet.

"We talked to our lawyer about this a lot," Kromer told us, "we are on absolutely solid ground with regards to copyright, user privacy and use of the API. This is clearly for the benefit of their community."

That's nice that Kromer feels so assured, but his attitude seems a little unrealistic.

We asked technology journalist Robert Scoble what he thought of the dilemma, and his opinion is pretty clear. "If Twitter wants to be a platform, they have to behave like a platform," he said. "Don't be king-makers. Let the marketplace choose the winners. If they are going to say nobody should study the data because we're going to sell that, that's not being a platform. Twitter tries to pick the winners and it pisses me off. They admit that they are king-makers. All that does is make everyone vote against them and hope a competitor comes around."

Perhaps time will tell. But these are very early days in what looks to be an era of widespread innovation built on top of social data analysis.

We've been critical of the sparse offering from Data.gov but Sunlight's plan looks like it could surpass what even the most ambitious government program is likely capable of.

Because of politics and scale there's only so much the government is going to be able to do. There are legal hurdles and boundaries the government can't cross that we can. For instance: there's no legislative or judicial branch data inside Data.gov and while Data.gov links off to state data catalogs, entries aren't in the same place or format as the rest of the catalog. Community documentation and collaboration are virtual impossibilities because of the regulations that impact the way Government interacts with people on the web.

We think we can add value on top of things like Data.gov and the municipal data catalogs by autonomously bringing them into one system, manually curating and adding other data sources and providing features that, well, Government just can't do.

Johnson says the site, which is being developed with publicly visible open source code, will allow users to contribute sets of data, documentation to go along with data and links to places around the web that are using any set of data cataloged. It sounds like a ProgrammableWeb type site for public data.

In addition to government information, Sunlight will also be including derivative data sets, like information from the campaign contribution tracker OpenSecrets. That's something it's very hard to imagine a government website including.

Can Sunlight build a one-stop-shopping destination for public data, and will people make use of that? Time will tell, but it sounds like a very important project.

This year's Knight News Challenge winners were just announced this afternoon, nine winning projects will receive over $2 million total to try and change the way the news world works. All of us should benefit from the results.

None more so than DocumentCloud. The winning project's description continues: "DocumentCloud will provide an online database of documents contributed by a consortium of news organizations, watchdog groups and bloggers, and shared with the public at large. Users will be able to search by topic, agency or location. Reporters will benefit from the wisdom of the crowd, which will be able to collaboratively examine large document sets."

Examination of large document sets and the systematic creation of shared cross-institutional memory with public access online? In concept at least, it doesn't get much hotter than that. It reminds us of Infochimps, a similar cloud for large sets of data, with programmatic access and user feedback on data quality. DocumentCloud isn't built yet, it appears, and InfoChimps is going to have a radical relaunch with more complete functionality soon. (We hope not to burst with excitement first.)

These kinds of juicy public banks of information are a big part of what the future of content and application functionality will be built with. Right now we're stumbling around online in the dark, with no illumination on all the data and documents around us except for cold-start full text or metadata search of what a handful of search engines can find spread out across disparate locations. Hopefully these upstart projects can pull it all off and all of us will be able to do much more informed research in the future. For updates on DocumentCloud, see the project's Twitter account.

The eight other winners of today's Knight Challenge include Salon.com co-founder Scott Rosenberg's proposal to build a neutral site for the public to discuss reporting errors with journalists, Aaron Presnall's proposal to create an easy-to-use data-visualization tool set, Katrin Verclas's plan to build a mobile media toolkit to turn anyone in the world into an instant mobile reporter and artist John Ewing's virtual street corners - a project that puts full size audio and video displays for real-time conversation between people on two different street corners in different locations. Information about all the winners is here.

Disclosure: The New York Times syndicates ReadWriteWeb.

Green tech writer, Katie Fehrenbacher, has written an important article arguing that utility companies don't get it, are afraid of the costs, and are thus unlikely to offer the kind of "real time" data delivery that could serve as a foundation for eye-opening innovation like we've seen from the networked world of the Internet.

Fehrenbacher's article last week, titled Why the Smart Grid Won't Have the Innovations of the Internet Any Time Soon, began like this:

Many people (myself included) have painted a picture of how the consumer piece of the smart grid could develop into a real-time, two-way communication network that looks a lot like the Internet. In that world, consumers would be able to see variable pricing change in real time, while smart meters and energy management devices read and visualize energy consumption data every second, leading to changes in consumer behavior. The ultimate vision of that landscape is that real-time energy data unleashes innovations and applications that we haven't yet thought of, which will deliver substantial behavior changes.

Well, that's the outcome for which entrepreneurs and innovators are hoping. The reality is that the consumer piece of the smart grid will look very different for many years to come.

Fehrenbacher goes on to detail the state of the art in smart grids, reporting that private tech firms, including Google, are frustrated with utility company delays of up to 24 hours in eventual return of consumption data to consumers. She draws a comparison with real time GIS location data, which telecom companies initially objected was too expensive to deliver and not really needed by consumers. When the killer app of turn-by-turn driving directions was invented, that debate was put to rest and the real time geolocation data infrastructure was born.

Some of the applications built on real time utility data will serve consumers, others will analyze data in aggregate and serve policy and other decision makers. Tim O'Reilly has invested in a company called AMEE, for example, that has discovered that the energy fluctuations of home appliances are so unique that they can tell what make and model of refrigerator you have by the way it acts when the motor turns on. Then it can suggest a more energy efficient appliance.

Remember how Net Neutrality proponents argue that if internet companies don't give cheap, equal bandwidth access to small innovators, then the team of people in a garage working on the next YouTube won't be able to afford to get it out the door? In this case the innovators coming up with the next killer app built on real time utility data won't be able to bring a service to market because utilities are simply unwilling to invest in making the real time data available to build on. (Fehrenbacher also points out that when it comes to data for analysis beyond consumer use, there are also some security concerns to consider. Not all data should be made available, she says.)

What's the killer app for smart utility grids? Fehrenbacher says she doesn't know and that's the point - we can't even imagine what kinds of cool and useful applications will be developed on that platform once it's available. The lack of a killer app leads to less support for the building of the platform, though, a catch-22 we can relate to from discussions of our calls to open up aggregate activity data from social networks for analysis.

We've written about three possible models of value for the real time web and we suspect some of the same principles could apply in delivery of real time utility data: ambient awareness, automated actions and monitoring for emerging trends. Those concepts almost make more sense in power than they do on the web.

We followed up with Fehrenbacher and asked her how the real time GIS killer apps got built if there wasn't an enabling platform, driven by demand for killer apps, yet. This is what she said.

"It was mandated that phone companies have to have location in phones to find 911 callers. So some companies did it by GPS and some did it by triangulation and some services had real time and some didn't. And then the phone companies wanted to recoup the investment they had to make for 911 location. The companies that had real time location found that they could better recoup the investment with the killer app of real time driving. The other phone companies that didn't have that, I think eventually added that on, but were late and didn't recoup investment as fast...It's [similar] with smart grids, because [those companies and utilities] have to make these investments too, by both upcoming regulation and current regulation, so why not build it out fully so they can create something where they can get more value? But they're not thinking about it as a platform - they're thinking about it as controlling the current electric system."

For more discussion of how smart grids may or may not soon become a platform enabling web-like innovation, see Fehrenbacher's full post at Earth2Tech.

Illustration at top of page: Smart Grid technology in my living room, from GE's Smart Grid Augmented Reality campaign.

Berners-Lee, a man whose invention of the web has had a greater impact on humanity than all but a handful of inventions over the last 50 years, is now one of the world's leading advocates not just for government data on the web but for free public access to raw bulk data that anyone can process for analysis and mashups. While the new Obama administration has made big promises about open government, it may now quickly find itself falling behind the UK.

The Obama administration has had a mixed record so far in terms of opening up government data. The launch of Data.gov was widely celebrated, though we found it too limited an offering. The Democrat controlled Senate finally opened up its voting record in machine readable XML format last month (that's great), but the nominee for the first federal CTO position faced zero questions about data transparency in his congressional hearings (that's bad). The White House said yesterday that it has clearly heard the public call for more open APIs (Application Programming Interfaces filled with open data) - so things certainly won't be standing still on this side of the Atlantic either. US CIO, Vivek Kundra, wrote an excellent blog post on the challenges and opportunities of open government data on Monday.

Having Berners-Lee on the team should be a huge boost for the UK government efforts, though. Most recently Berners-Lee made headlines for (trying to) lead the TED conference of global thought leaders in a chant of "Raw Data Now!" That conference was the same one where Bill Gates opened a jar full of mosquitoes into the crowd like a mad man, so that the elite group could get some feeling of the fear associated with malaria like so many millions of other people feel.

In other words, Berners-Lee is a hard core advocate of open data. It's hard to imagine a more high profile move for a government to take than to announce that he's coming on board to help the effort.

The user interface is inflexible, the data is odd looking and it's hard to imagine using Squared regularly. It's a great idea but we'll see where it goes.

All the other queries we tried were similarly "almost helpful." The dog breed example is actually unusually good. Sorting by a particular column isn't possible, when I define a content type you don't get to see it unless I share it with you, and the user experience is an off mix of intriguing and maddening. The description fields would benefit from borrowing the first few lines of a Wikipedia article on a topic.

It is very impressive that when you request a square for a concept Google is unfamiliar with, you're prompted to offer up to five examples and then it goes out and builds the data set for you! Unfortunately, when I tried to explain to Squared who some examples of "tech bloggers" were it brought back a terrible picture of me and said that CNet's Caroline McCarthy is sixty four years old. I'm pretty sure that's not true.

We're as excited as anyone about the future of creating structured data from the sea of information online, but Google Squared isn't very inspiring so far. We've been looking forward to it since interviewing Marissa Mayer, VP of Search Products and User Experience at Google, about Squared. When the day comes that you can slap a .xml or .csv to the end of one of these Squared URLs and pull out data programatically, that will be impressive.

Here's our review of Wolfram Alpha, which we said was likely to be a good service for engineers but not for anyone else. Hopefully it's still early days for all of these kinds of tools.

It's a fun site to use. Basketball, football and NASCAR are supported initially - hopefully baseball and hockey will be next. There's OpenID integration, the image selection is really nice and it's just great. It's still a little rough around the edges but given that the service just launched today - we're impressed. This is the kind of democratized data visualization that any field could benefit from with enough open data and a good user interface.

The Adobe Flash charts that StatPlot produces are even nicer, but they aren't easy to scale down to the size I needed.

The whole StatSheet franchise is an interesting one to watch. See TechCrunch's recent coverage of the company's tussle with Twitter over innovation on that platform.

Bring on the huge data sets and easy charting interfaces! We'd love to see a little of that action over at Data.gov, for example. Heck, let's see these kinds of options put on top of the forthcoming ClearSpring API tracking hundreds of millions of peoples' sharing activities online.

As StatSheet said on its blog today: "There are other services (Swivel, iCharts.net, Many-Eyes) that allow you to upload data and create a variety of visualizations, but these all suffer from the same issue. The average sports fan does not have access to quality sports stats to upload. With StatPlot, you don't need to bring your own data because you can use the expansive StatSheet database!"

It's not just sports fans that could use a hand with data sets. We all could. Thanks for leading the way in truly democratizing data visualization, StatSheet.

That's a whole lot of information. ClearSpring sees not just what you're sharing, but nearly everything you're doing on the pages its products are on. (AddThis is on ReadWriteWeb, for example.) So what on earth is it going to do with all that data? Like they said in Spiderman, "with great power, comes great responsibility." We asked ClearSpring's CEO about these super hero-like responsibilities and his thoughts are below. You can decide for yourself whether he can be trusted, but the work the company is doing is very cool.

In February we wrote about ClearSpring competitor ShareThis and its venture backed plans to become a major development platform with all the data it sees by facilitating content sharing. What are ClearSpring's development plans and how will it treat user privacy? We asked ClearSpring CEO, Hooman Radfar, and this is what he said.

To summarize: Radfar says ClearSpring is analyzing the content on all the web pages it sees, as well as user intent in sharing content, and categorizing that information. And will soon make APIs available that allow developers to build trend analysis and recommendation services on top of at least some of the data. The company is also taking privacy seriously, Radfar says. As nerds about aggregate data analysis and APIs, we think this all hits the sweet spot. It's kind of creepy too, though.

Here's our interview.

ReadWriteWeb: So, you see half the internet now. What's that old phrase? "With great power comes great responsibility?"

Hooman: Yea - weird right? It's honestly a bit surreal.

ReadWriteWeb: What are you going to do with all that info?

Hooman: Well, there are two parts to leveraging our reach. First, now that we have so many folks using AddThis, we want to bring other service providers into the mix. The next twitter, etc. [Support for more sharing services.]

The next part - which you alluded to - is USING the data. Using the data is a longer term project. We are starting to classify it into categories and apply it to problems to understand how it can enhance the AddThis experience for users via new products, or enhancements to existing products.

ReadWriteWeb: What kind of categories?

Hooman: So we are looking at intent-based information, as well as behavioral. We profile users of the AddThis/clearspring platforms and attribute intent or behavioral info to the profiles.
Within each of those categories, we are contextualizing them vertically. So, for example, within intent, we are looking at subsections like auto, shopping, etc. It's an insane amount of information.

Here are some use cases. First, we have a social distribution business where we distribute branded widgets for advertisers. We can leverage the data to optimize distribution by targeting users with a higher proclivity for sharing and the content type.
That's a low-hanging fruit opportunity.

Some of the other opportunities that we are planning on pursuing are more product focused. For instance, we see a large sample of shares into Facebook and Twitter, in real-time. Every second we are seeing shares.

We have built a platform to expose that information and built recommendation applications a la Digg, that show real-time recommendations based on user activities. The idea is that we could leverage the data to launch user applications that will enhance the 'sharing' ecosystem by providing more recommendations, we are helping publishers get more traffic and users get better content.

This summer we plan to expose our internal platform and capabilites. To put it in perspective we see orders of magnitude more than some of the most popular recommendation sites.

ReadWriteWeb: What will that look like? Will it be limited to data concerning sharing activities?

Hooman: Still working that out to be honest. We probably will expose more - but we have to be careful, per your spiderman quote. We are a small company with a whole lot of info. To put it in context, if we were a publisher only Google, Yahoo and Microsoft have more reach - and we are getting closer to Microsoft. We keep it simple - as long as we can sleep at night then we are probably on the right track. We have already started talking to privacy experts.

So products we have built on our data internally include trends (think google), recommendations (think digg).

ReadWriteWeb: Can I get a raw data dump to play Hadoop or whatever with?

Hooman: We have not done anything like that yet. We want to be sensitive to Personally Identifiable Information. That being said, we are going to expose rather large data sets soon. We want to be sensitive, but we can see trending topics a la twitter.

ReadWriteWeb: What large data sets will you expose?

Hooman: Sets we are looking at are...trends across the ecosystem from a topic perspective, so you can find if a 'term' is rising and falling. Exposing the actual sharing data so someone could create a 'shares around the world' mashup with google maps. And we might do some categorization and have an API to let people actually see trends within categories like auto, etc.

ReadWriteWeb: Are you analyzing the content on the pages being shared? Or just the messages that get posted along with the sharing?

Hooman: Both. That's where it gets powerful.

ReadWriteWeb: How are you analyzing the content?

Hooman: Contextualization. We have access to pages via javascript so we can categorize, not unlike Lijit or Google. We are partnering to do that.

ReadWriteWeb: With whom?

Hooman: Can't say yet.

ReadWriteWeb: Oh come on now!

Hooman: I wish I could. I don't know if I am allowed.

ReadWriteWeb: If you don't know if you're allowed then you have not been told to stay quiet about it. That's a key detail.

Hooman: Ha ha ha. I know.

How Does it Work?

According to Sensaris,

"Whether used by pedestrians or cyclists, the sensors provide noise and ozone levels in real time over the Internet and web based visualization (mashups). Our Senspod technology forms the basis for innovative citizen centric services. We look forward to helping other cities use it for their own community action and urban life quality improvement plans."

We think this is potentially a wonderful tool that will empower citizens to become involved and take back some control over their quality of life. Hopefully it will force cities to pay more attention to air and noise pollution problems in less affluent neighborhoods and take steps to improve conditions.

On one hand it's just a speed up the infrastructure play, but the impact of real time information delivery on a user's experience of a website can be profound. The latest entrant into this market of white label real time service layers is called Notify.me.

Making websites real time is the hottest trend online this season. From Facebook to Google, Twitter, Digg and countless little innovative startups, it seems like everyone is either doing it or talking about it. (See our Introduction to the Real Time Web for background.) Might some sites choose to use an outside service that specializes in real time infrastructure, instead of building their own in-house? That's what Notify.me is betting on.

Notify.me is a San Diego based startup made of tech industry veterans, some working on the company on the side, others full time. The company has taken no funding, has no revenue and no one working there is being paid. The executive team is made up of engineers from companies like MP3.com, the Health Care division of SAIC, Napster, DekiWiki and Yahoo. It's a pretty hot crew to get together with no pay to take a long shot at productizing a technology like real time.

This isn't just another fly-by-night "instant alerts" service, though casual observers may have thought as much over the several months that the free consumer version of Notify.me has been available. (We count ourselves among those casual observers, in fact!)

Using the free consumer service, anyone can set up alerts to be delivered by IM, SMS, email or to an Adobe AIR app whenever an RSS feed updates. That's nice and hopefully Notify.me's service will work better than the alternatives do these days, but RSS to IM/SMS alerts are nothing new.

Where it gets really exciting is in the two APIs the company is working on.

What's New: The APIs

A REST API is available for publishers now and C# and Perl libraries should be available in two weeks. That API allows publishers to define particular events on their site and then offer real time alerts to readers when those events occur. You might want real time notification when someone leaves a comment in reply to yours, or when a site publishes news concerning a particular topic, or when a new event listing is published so you can buy tickets right away. The possibilities are endless and fun to imagine.

The second API in the works is an Actionscript and XMPP Client API that allows developers to build interfaces for audiences to consume real time alerts through. That API has specs in draft form now but the company says it expects little change to occur before a final release.

What does that mean? It means you could add real time notification consumption to apps on the web, desktop or iPhone (using the new Push Notification Service in the next iPhone OS release).

Put those two APIs together and you've got publishing and reading apps going in real time. Hello real time web!

Can They Sell Real Time to Businesses?

The Notify.me team has immediate designs on business customers. Talks have begun with companies in the medical, financial and software fields. Doctors could use real time updates to track patient updates, including allergies and drug conflicts as they are discovered, prior to prescribing medication. Medical practices could push lab results to physicians instead of waiting on a chart pull request.

Would medical software companies use a hosted 3rd party API as real time infrastructure? Notify.me says they have consulted with HIPAA experts who believe that as long as the company transmits notification of an event and not personal medical information, they should be legally compliant.

Figuring out rules for determining what kinds of information gets delivered will be one challenge that Notify.me will have to tackle with customers. As Sameer Patel, another entrepreneur in this market, points out: "What's absolutely necessary in the B2B space though is smart aggregation before push comes into play. Failing this, its going to be a fire hose that will quickly alienate the end user."

Gnip is another service offering similar kinds of functionality, but for different markets. Gnip head Eric Marcoullier had this to say about Notify.me's B2B prospects:

"Good for them. Further validation that slinging realtime data around has value. I bet they'll find good money there. We've even considered some of those use cases in the past, but have shied away because of the liability that's associated. The Gnip team works really hard to make sure that the platform is always running (with 99 point nine something uptime since launch) but if data gets held up for an hour, nobody's life depends on it. I'm psyched someone else is diving into the mission-critical data delivery while we work on business-critical data."

Indeed, reliable scalability will be Notify.me's biggest challenge. That's something the company has focused on since the start. Proving their case and building a name for themselves as a popular consumer notification service is a business strategy that quite a few other Web 2.0 type startups have done well with.

Can that strategy work with real time notifications, though? We suspect that business customers may be more interested in integrating real time functionality than all but a few power user consumers will be, so if you like the consumer service of Notify.me you'd better use it now, before the more viable business market takes precedence in the company's day to day decision making to the detriment of free accounts.

In the meantime, we expect that someone will succeed in bringing a real time service layer to the websites we use and work with every day. Real time is just too compelling for the paradigm to go back into the genie's bottle. It could be Notify.me that finds that success.

The key idea behind the Web is that pages are connected through hypertext links. Google changed the world and made money beyond anyone's wildest dreams by analyzing those connections between pages. It was a simple proposition, at its core: the more a page is linked to, the more authoritative it is. The web isn't just pages anymore, though. Now the web also includes people as a fundamental factor to take into consideration.

We don't want to argue that Bit.ly is the next Google, but the technology it's brought to market could be very important in the indexing of the social web. Bit.ly shortens links so they are easier to share, like TinyURL. The service creates a redirect from a short Bit.ly link out to a longer link on any web page. Allong the way the service analyzes the page being linked to, pulls out the key concepts discussed on that page, and then provides real-time statistics about where the link is being shared and how many people are clicking on it.

Today Bit.ly quietly became the new default link shortening service for Twitter. Neither company would tell us much about the transition but Twitter did indicate that Bit.ly was chosen primarily because of its superior reliability. TinyURL is notorious for downtime, which is a real killer when it comes to redirecting services. Link redirection services have enough ugly warts already that going down is just not acceptable. [Note, if you're interested in the business side of things - Twitter and Bit.ly also have some common investors, BetaWorks. BetaWorks is also a Tweetdeck investor. The group's stock in Twitter is very small though and can't explain the whole partnership.]

What Bit.ly Could Make Possible

Once Bit.ly has been put to enough use, and today's news will likely be a big part of that happening, you'll be able to ask it questions like: within the last hour, what are the five hottest web pages about President Obama's budget? What social networks are sharing links to my web page the most today? What are ornithologists on Twitter most interested in this week?

The columns and rows here are semantic key terms on pages shared, method of sharing used (Facebook, Twitter, email, etc.), number of click-throughs, time and person who created the original shortcut. There's a whole lot you can do when you have that kind of information about a link. Bit.ly says its API isn't quite there yet, but it's close.

Scaling Bit.ly

As you can imagine, scaling is a serious issue. Last week Bit.ly decoded short links for 50 million clicks, only about half of which were from Twitter - about 10% were shared on Facebook. That was up from 15 million just 5 weeks ago. Becoming the default URL shortener for Twitter should send those numbers through the roof. Real time metrics and latency on this kind of scale take time and money to get a handle on. The company says it has five layers of redundancy built already to make sure that Bit.ly doesn't go down.

A mind blowing number of links will now be sent by Twitter users, through Bit.ly and back into Twitter again. Twitter may or may not be planning on taking advantage of all of this information. The company tends to keep things pretty simple, acting as a platform for other people to innovate on top of.

We fully expect to see Bit.ly take advantage of this increased flow. And we expect to see many of the countless other startups building value on top of data gleaned from Twitter to do so as well, from Tweetmeme to Tweepz to Twazzup. Maybe even some startups that will have less silly names! This is serious stuff.

Publishing the metrics of sharing on the social web is something that is very fairly compared to indexing the pages of the web and analyzing the links between them. We may not find the next Google in Bit.ly or the services built on top of it, but something very important is afoot.

On May 1st South Carolina's Senator, Jim DeMint, officially asked the Senate Rules Committee to make the data available and just four days later the feed is here. Not everyone is happy about about the information being made publicly available like this, however.

John Wonderlich, policy director for the Sunlight Foundation, told Politico that the reason he's been given for the lack of XML feeds is this: "the secretary of the Senate has cited a general standing policy ... that they're not supposed to present votes in a comparative format, that senators have the right to present their votes however they want to...it's pretty bad."

Dave Lundy, acting executive director of the Chicago-based Better Government Association, told Politico again that: "It's a strategy to make information hard to find and hard to digest and hard to analyze...Call me a cynic, but I don't ... think [government entities] deserve the benefit of the doubt. We have ample experience to know that people try to hide information, even in plain sight."

Apparently, those problems were washed away this week by the tides of open technology. The Washington Post has offered something similar to what's now available for some time, but there's something to be said for what we hope will be a big, fat, official pipe of data.

We learned of the news this morning when New York Times technologist, Derek Willis, celebrated mention of the news by Rob Pierson, who yesterday began a new job leading new media initiatives for the House Democratic Caucus. The Sunlight Foundation said last week that neither the House nor the senate "maintain any reasonable database of lawmaker votes." The House of Representatives does release their votes in structured format, though.

Willis points out that the new Senate data feeds aren't perfect; the absence of Bioguide ID information linking Senators' names to their online profiles creates an unnecessary additional step for developers, for example.

It's exciting news none-the-less. "It's good to see high profile senators from both parties behind this," says John Musser, founder of the web's leading mashup and API directory, Programmable Web. "Those first steps are often the hardest. That is, just getting understanding of the value, getting buy-in and then having the data accessible in a developer friendly format. The next logical step is to wrap it in an API; having the XML is closer to having an RSS feed, there's not a lot of developer control of what data to retrieve. An API typically gives much more control over what data gets retrieved. Like 'give me all roll call votes for January 2009', versus 'here's the last 20 roll call votes.' Or all roll call votes by a specific senator, etc."

Musser says that he's seeing a broad movement towards increased access to government data. That work is being done by both official sources like this new Senate feed and the data-centric Recovery.org and by outside organizations like the Sunlight Foundation and the New York Times, work Musser is tracking closely.

What's left to open up? Check out, for example, this list of the 8 most desirable but unavailable government data sets, per Willis from the NYT. As of today, one of those can be checked off the list.

Perhaps nothing like this should be a surprise coming from a company that built the leading webmail product online and still calls it Beta five years later.

Who's worn the virtual cap and gown already? Twenty seven projects are listed as graduates, including the YouTube APIs, Maps APIs and (thank goodness) Contacts API. We've been wondering why more developers aren't using the Contacts API but instead keep asking us for our precious Google passwords. It's been out for a whole year now. Now there can be no excuse! The Contacts API is real, it's solid and Google is committed to supporting it.

Surprises among the 18 projects still deemed too green for prime time? Gears, Finance Data APIs, Social Graph API and the Feedburner APIs.

Google APIs are an incredible resource and today's announcement should further their adoption. We'll be monitoring the Code Labs page for the experimental availability of the inevitable Google Brain Implant API, but until that day comes we feel almost unconditionally excited about the code projects the Google teams are working on.

In a world of online fraud and flaky customer service, the prospect of having Amazon handle payment collection for apps sounds great to us.

A number of companies have been beta testing Amazon DevPay, from cloud monitoring service Hyperic to photo sales app SmugMug.

DevPay charges 30 cents per transaction plus %3 of costs to customers beyond the cost of the Amazon Web Services charged to developers.