How does an Internet junkie, news organization, or political operative monitor rapidly evolving real-time events, from the crucial details to the bigger picture? More importantly, how can a data stream be turned into real-time action, reaching the people who need it, when they need it, and in a form they can easily digest?

Overview

The proliferation of real-time search engines and trend monitors (sometimes referred to as "listening platforms") has thus far done little to address problems of this scale and importance. This is because they fail to provide context -- i.e. show how a new piece of information is relevant to what we've seen before and where it fits in the space of possibilities and relationships.

For instance, if you are a programming director at CNN trying to discriminate between significant news and Internet memes, simply knowing that #iranelection is a trending topic doesn't tell you its relation to other topics or which communities are driving it -- both critical factors.

One promising area is data-oriented user interfaces: data and algorithmic analysis in the back-end and direct visualization and navigation in the front-end. This the next stage of social information, slicing and dicing, mixing and matching, interpreting and analyzing, completely on demand. In this new landscape, the data is the interface.

It's not just about sitting back and looking at pretty pictures. It is about setting aside stale UI metaphors and getting as bare-to-the-bone a human interface as possible for computation. The recently launched Wolfram-Alpha applies this principle to structured data. (Disclosure: I was a member of the core Wolfram-Alpha team and may continue to consult with Wolfram Research.)

Real-time data streaming offers similar possibilities and opportunities. In this vein, let's outline some basic ideas and methods for giving context to the streams.

The Computational History of #iranelection

At the most abstract level, history and computation are the same thing: the evolution of systems over time. Twitter has several remarkable properties that allow us to finally leverage this correspondence in tangible ways. The simplicity of its data, the openness of its system, and its extreme time resolution make it possible for us to detect atoms of history, those moments when something is triggered and society is reconfigured ever so slightly.

Look at the bandwidth plots below. They represent the relative volume of different Iran-related phrases on Twitter over time. Their most striking characteristic is how discrete and spikey they are: a tell-tale sign of an organic computational system.

The first pair of terms compares the bandwidth of "Ahmadinejad" and "Khamenei" mentions, respectively. The evolution of the uprising at the very highest level of social abstraction is shown with remarkable clarity: moving from a dispute over the election process involving Ahmadinejad (shown in pink) to a dispute over authority involving the supreme leader Khamenei (shown in red).

Not only do we get the gist of the evolution, we also see its details and relationships to other social sub-structures. For example, looking at the second plot, we see a co-relation between mentions of the Basij militia and the reports of deaths; and that initial uptick in Khamenei mentions corresponds to the uptick in Basij mentions, foreshadowing the later crackdown.

This idea of computational history applies to events that Twitter not only reported but shaped and hosted as well. A plot further down below compares the Twitter-centric discussion of #cnnfail to the distribution of Twitter proxy IPs that allowed information to continue to flow out of Iran. Is it a coincidence that these two terms merge smoothly together? And what about the big spike in mentions of proxy distribution coinciding with the first reports of violence?

Computing with Social Structures

Simply tracking the volume of various phrases gives us a sense of what is happening on the street, literally and figuratively. But that signal is but a shadow of a far more complex and intricate reality, an interwoven web of individuals and actions.

Twitter allows these social structures to become data structures by means of the "RT" convention. And this in turn allows us to perform extremely powerful computations on the social structures that underlie the flow of information.

Network layout algorithms are a familiar, powerful, and fascinating example. They self-organize in your computer to reveal self-organization in the real world. And that is exactly the kind of tool we need to test our hypothesis about #cnnfail.

The plot below shows the network of people who re-tweeted mentions of IP proxies, with those who had tweeted earlier about #cnnfail highlighted. We see not only significant overlap among the people involved but also a considerable structure in the relationships between them. We have captured a real community at the moment of its birth.

Remember this as you look at the next plot below. Here, we see the re-tweet network that formed around the top five Iranian tweets. Its structure shows a very different phenomenon, capturing the emergence not of a community but of an elite. Despite massive interest, or perhaps because of it, most people did not discover more than one of the top Iranians. The network simply grew faster than the information could naturally propagate. But a small inner circle did succeed in identifying core sources of information.

The final plot below shows yet another community structure, as well as a new algorithmic technique. This plot does not show the emergence of a new community but rather shows the appropriation of a new topic by mature political factions. This re-tweet network has formed around Iranian tweets that mention Obama. Using graph theory, we can computationally extract the sub-communities and then use that information to color the network. The large blue mass on the right is the conservative Twittersphere, while the other structures are a less-organized collection of mainstream or progressive news outlets.

Algorithms and Social Change

Will future Presidents express strategic goals in terms of Twitter graph theory? That is almost a certainty.

The purpose of these computations is two-fold: first, to contextualize information from across time and space in terms that are accessible to humans; and secondly, to distill abstract ideas into actionable form.

Twitter is a platform for achieving both of these purposes in human affairs: detecting networks of information propagation and erecting new networks to reshape emerging social computations. This is the core of Twitter's social and business value. If we were to play that age-old game of "Guess the business model," we'd look here first.

How to algorithmically discover and deploy novel social structures is perhaps the billion, or trillion, dollar question. With Twitter, the data and API are in place. And if the history of computation is any guide, once programming a system becomes possible, progressing from a hack to an application to a platform is only a matter of time.

Guest author: Kovas Boguta is a co-founder of Infoharmoni, a stealth startup building computable knowledge interfaces for real-time data sets. He just returned from last week's Personal Democracy Forum, where discussion about the Iranian uprising took center-stage.

SMS service, which political dissidents had used to spread messages and organize protests, has been restricted since June 11, causing many Iranians to use Twitter, Facebook, YouTube, and other social sites to broadcast and communicate.

Iran's broken digital communication infrastructure caused many Iranians to turn to services such as Twitter, using proxies to work around government restrictions for web use. Twitter became so integral to Iranians' communication, particularly with the wider global community, that the U.S. State Department asked Twitter to postpone scheduled maintenance which would have occurred in the immediate aftermath of the election and resultant protests. Other services rushed to add Persian translation features.

Hopefully, the unblocking of text messaging in Iran is a sign that communication channels are returning to normal. So, does this mean that everyone's new favorite color, "Solidarity Green," will begin to fade away from social web avatars sometime soon? Once the country and its government emerge from crisis mode, what news will come from Iran, and what will the Internet have to say about it?

It can't be assumed that all the proxies were created to support Iranian protesters, but they were probably all shared for that purpose. Renesys reports that the rate of proxy creation appears to have slumped dramatically over the last few days and newly shared proxies are now being added quickly to the official list of filtered destinations online. This could have consequences for the political movement's global visibility and underlines how important it is for everyday people to create proxies and share them with friends overseas ahead of time.

Renesys offers the following explanation of how proxies work:

A proxy server is a simple bit of software that you run on your computer. It effectively lets you share your computer with anonymous strangers as a "repeater" for content that they aren't allowed to fetch themselves. For example, an Iranian web browser might be manually configured to use your computer (identified by an IP address and a port number) as a Web proxy. When your anonymous friend reads twitter.com, or posts a tweet, the request goes via your computer, instead of to Twitter's web server directly. Except for a little delay, and the fact that your friend gets to see what the uncensored Internet looks like from New York or London or São Paolo instead of Tabriz or Qom, surfing through a proxy is pretty much like surfing without one.

The United States may have offered up more proxies than anywhere else, but the effort has truly been global. "The USA and Western Europe were well-represented, but so were China, India, Russia, Romania, Bulgaria, Vietnam," the firm wrote today. "87 countries in all, a pretty impressive breadth of representation, considering the relatively small size of this sample."

Unfortunately the construction of Iran's sophisticated censorship system was a global effort as well; the Wall St. Journal reported this weekend that Siemens and Nokia sold the Iranian government the technology that's being used to track down and silence protesters.

Renesys notes that public sharing of the proxies is making it too easy for the Iranian government to find and control them. The company suggests that it's far better to create proxies and start sharing them with people living under repressive governments ahead of major conflict - "don't wait until the tanks are in the streets to figure this out, because by that point, you may have already lost the proxy war."

If you would like to think ahead and set up a proxy for sharing with friends who could need it, Glype.com is one service you might begin with.

Is it doing any good? Do our efforts, symbolic or concrete, have any impact? We don't really know. Critics will point out that, for the most part, people are jumping on a bandwagon. Very few of us have any real grasp of Iran's political or social complexities; we have no idea who the leading opposition candidate is or what kind of record he has (and most of us would recoil if we found out). All true.

What we do have, though, is a shared belief in human dignity, and a shared recognition that this is a chance for that cause to take a big step forward. Green avatars are, in a sense, a way of signaling as much to each other: "Yeah, I care about this, too." Perhaps we do this to show solidarity as much with each other as with the people of Iran.

More Noise to Signal.

As Twitter became the star platform for Iranians to convey breaking news from on the ground, Facebook has been quiet by comparison. Now, by releasing an edition in the official language of Iran (also colloquially called Farsi), its attempting to capture some of the revolutionary self-expression that has thus far passed it by.

Whatever the language, Facebook pales in comparison to other social mediums currently in use for speedy transmission of events. To Iranians, Facebook is likely to be neither as familiar a publishing medium as blogs, nor relatively easy to access in the face of crackdowns (like Twitter is).

Persian Facebook might be viable for something with a longer-term vision, with groups and causes being better for cohesive political organizing than an amalgamation of hashtagged tweets. But for now, Facebook is decidedly ill-suited to helping Iranians to speak to the world.

Is Persian Necessary At All?

With one of the largest and most active blogospheres in the world, Iranians are not hurting for tools to express themselves politically and culturally online. They're just hurting for free access to them, which Facebook may not be able to provide considering it was blocked shortly before the election, and may even remain so.

As far as Twitter and other platforms go, the vast majority of those tweeting from within Iran are doing so in English. Adding a private, Persian-language social network to the mix isn't likely to aid those with the aim of communicating with the rest of the globe.

Twitter decided yesterday to delay a scheduled downtime for maintenance until the middle of the night, Iran time. CNN reports that the decision was made at the suggestion of the State Department, but a large number of users suggested it as well. Incidentally, open source Twitter alternative Identi.ca, which has the potential to be much more useful due to its decentralized nature, followed Twitter's lead and also made major changes at a time convenient for Iranians.

Some critics have argued that news of US government collaboration with Twitter will offer the Iranian government more leverage in calling them insurgent agents of the US. The CNN report was confirmed this afternoon by Reuters. BoingBoing has a good write up on how to help, not hurt, the efforts in Iran.

We've been following news from Iraq via this excellent aggregator built by Twitter search engine Twazzup, by watching the StopAhmadi account in particular and by reading Robert Fisk's excellent coverage for context.

Twitter cynics take note - this service is clearly not just for posting about what people ate for lunch anymore. We wish the Twitter team luck in fighting to keep the lines of communication open.

In the aftermath of the election, during which a (some say statistically improbable landslide) victory for Mahmoud Ahmadinejad was announced, the social media buzz grew into a roar as a meme began to circulate: Where Is My Vote. A website, several Facebook pages, and now thousands of tweets have ensured few social web users have not yet heard of the controversy and the Iranian government's response of censorship.

"In coordination with Twitter," Stone wrote, "our network host had planned this upgrade for tonight. However, our network partners at NTT America recognize the role Twitter is currently playing as an important communication tool in Iran... Our partners are taking a huge risk, not just for Twitter, but also the other services they support worldwide--we commend them for being flexible in what is essentially an inflexible situation. We chose NTT America Enterprise Hosting Services early last year specifically because of their impeccable history of reliability and global perspective."

Citizens Had Taken to Twitter to Report and Find News

Although, as noted above, sources cannot be verified at this time, the vast majority of tweets we have seen tell stories of violent military reactions to protesters:

Many more tweets are linking to news coverage and photo or video evidence of what is happening on the ground during these protests.

Iranian Censorship of the Social Web

The Wall Street Journal reported this afternoon that many inside the country have resorted to overseas proxies to continue to use the websites. However, as proxies appeared on the web, they were gradually blocked. Many are now calling for users to stop publicly announcing proxies and use private channels such as email instead.

Blocked Hashtags, Changed Locations, & Other Misinformation

Still more non-Iranian users are changing their Twitter profile locations and time zones to reflect that they are in Iran. These users are often also posting green-tinted versions of their avatars. Some say this is a show of solidarity; other users insist this action will somehow "screw with the government's head."

As one Twitter user said and as several others would likely agree, "Most of the people on Twitter are [expletive deleted] retarded."

Changing one's avatar and location would likely do little to affect the Iranian government; these actions are simply a sign to a user's followers that he or she has at least a cursory interest in Iran current events.

As the conflict continues, Twitter users are reminded that the best, most useful information to share is that which is most verifiable and hence most likely to be accurate. Retweeting biased or exaggerated accounts damages any cause by feeding a hype cycle and drawing attention away from the heart of the matter. At the very least, users should attempt to search for and verify information before passing it on.

The western world's most feared government is shaking with insurrection in the streets after a contested election and the leading name in news, CNN, is shockingly absent from the story. Twitter, meanwhile, is how Iranians are communicating with the outside world. It's the best place to follow events going on in that country and CNN's failure to engage with the story is one of the hottest topics of conversation there.

Hours after Iranian police began clashing with tens of thousands of people in the street, the top story on CNN.com remains peoples' confusion about the switch from analog TV signals.

Update: See Tom's Tech Blog on Sunday morning for a well written critique of our coverage of these events. By Sunday morning the conflict in Iran was the top story on CNN.com, though, so the extenuating circumstances that blog post brings up don't seem to explain what took so long in the heat of the action.

Political blog aggregator Memeorandum is all Iran right now and is a great place to get in-depth information. The BBC is covering the story well, we found this video posted to YouTube. It's being passed around Twitter. These photos on Flickr are good, too. This video from LiveLeak is quite moving, if likely to cause motion-sickness. Twitter search engine Twazzup has created a great aggregator page for real-time multi-media updates from and about Iran. Andrew Sullivan writes well about the extensive use of Twitter by Iranians in the uprising. There's a collection of Twitter accounts and other media from Iranians over at Reddit. This in a country where the government recently debated applying the death penalty for subversive blogging.

Barack Obama has a good excuse for not engaging substantially with the protests in the streets - if he condemns the incumbent's claim of victory then negotiations around nuclear weapons will be much more difficult. What's CNN's excuse?

Twenty years ago CNN's coverage of Tienanmen Square made its reputation. If in twenty more years it has become consensus that real-time, online, crowdsourced media is the best place to keep up with current events, this incident could be an important part of that history unfolding.

The presidential elections in Iran are being held on June 12, 2009 and will be the country's 10th such election. There are 3 leading candidates right now. Incumbant Mahmoud Ahmadinejad is running against Iran Reform Movement candidate and former Prime Minister Mousavi. Former Speaker of the Majlis (Parliament) Mehdi Karroubi, another Reformist candidate, also intends to run.

As All Facebook's Nick O'Neill puts it, "unfortunately for the Iranians, they have a regime in place that doesn't support freedom of speech and would prefer to run a dictatorship". Those of us who enjoy such freedoms should do our part to make sure that censorship like this never happens to us. We wish our brothers and sisters in Iran the best of luck.

Facebook Response:

When we first learned of this story we immediately contacted Facebook to see what they had to say. They were gracious to respond swiftly with this comment from a Facebook spokesperson:

"We are disappointed to learn of reports that users in Iran may not have access to
Facebook, especially at a time when voters are turning to the Internet as a source
of information about election candidates and their positions. We are investigating
these reports.

We believe that people around the world should be able to use Facebook to
communicate and share information with their friends, family and coworkers. It is
always a shame when a countries' cultural and political concerns lead to limits
being placed on the opportunity for sharing and expression that the Internet
provides."

No cause of death is yet known, but the Committee says torture of bloggers is common in Iran and they are usually placed in close proximity to the most dangerous criminals in any facility. Misayafi was sentenced in December to 30 months in prison "for insulting Islamic Republic Leaders." The man said he was a cultural blogger, not a political one, and only wrote a few satirical articles that got him into trouble.

We've reported here on a number of bloggers imprisoned in Iran and in Egypt for documenting government abuses or just writing critical words about governments that demand total compliance. In the middle of last year we wrote about Iran's parliament debating legislation that would add the death penalty to the list of possible punishments for using blogs to challenge government authority.

It is a timeless battle all around the world between freedom, art and self expression on one side and authority, expediency and abuse on the other. The rise of the web has made that battle different, though. Blogs give a voice to the previously voiceless, and the historical and moral importance of efforts to save those new voices from arrest, torture and death cannot be overstated.

We would love to see the Obama administration, which has made extensive use of online social media, publicly and explicitly condemn this death at the Iranian government's hands. We'd be surprised if that happened.

Social media is powerful and changing the world; we don't expect that this will be the last person to lose their life over it. Omid Reza Misayafi, brave Iranian blogger, may you rest in peace. May all those imprisoned for blogging in Iran, and around the world, be set free.

For ongoing coverage of this and all-too similar situations around the world, see The Committee to Protect Bloggers and associated organizations linked to on their site.

It's reminiscent of the recent film Persepolis and helps put "social media" in a much more serious context than people often presume it goes on in.

The short video was created by Aaron Chiesa, Hendy Sukarya, Lisa Temes and Toru Kageyama and written by Kate Tremills.

To learn more about the struggles, victories and defeats of bloggers around the world, check out The Committee to Protect Bloggers, Global Voices Online and Reporters Without Borders.

The French Press Agency reported on the bill yesterday and according to The Committee to Protect Bloggers, the BBC's The World radio show will offer a more in depth report in the coming days.

Blogging is wildly popular in Iran, where a new generation of young people frequently challenge the old, hyper-conservative religious government. The Committe to Protect Bloggers says that Iran is "among the worst offenders in terms of harassing, arresting and imprisoning bloggers, as well as students." You can see the group's extensive coverage of Iranian cyber-censorship and harrasment of bloggers here. The Iranian government has blocked access to Facebook, Yahoo! and Flickr, among other sites.

We at ReadWriteWeb condemn the application of the death penalty to bloggers as itself an abhorent crime. Cultural relativism has its place, but this isn't it. We want to offer our support to the new generation of Iranian young people struggling for freedom online and elsewhere, in any way we can, short of a US invasion of the country.