Reflecting on the human condition was once a popular past-time. A lot of people used to read poetry as you may have heard. It may not be the Internet's fault that we're becoming less introspective - in fact the huge amount of activity data we're sharing online offers incredible opportunities for reflection, and for learning more about ourselves. It seems quite likely that we're going to miss those opportunities because our software is focused entirely on doing (and advertising) instead of on helping us think as much as it could. Of course that's much harder to do.

Burton Group analyst Mike Gotta wrote a blog post two years ago that articulated both the opportunities and some of the challenges to building meaningful value on top of our streams of aggregated data.

Stream processing systems (and associated analytical components) will become a critical underpinning for much of what is talked about in terms of workstreaming, lifestreaming, attention streams, collective intelligence and so on. Discovering patterns across people, interactions, information, activities and social networks and assessing those relationships is difficult enough. It becomes even more challenging when you also want the results to be communicated in a manner that is contextual, relevant and sensitive to attention (and confidentiality) needs.

Two years later activity streams are far, far more widespread than they were when Gotta envisioned analytics built on top of them. The analytics remain almost nowhere, though. That side of things just hasn't been meaningfully developed. Millions of people are farming for Facebook game makers, who are raking in millions of dollars, but all the social interaction that goes on in this increasingly social web remains otherwise under-utilized.

I want software that will tell me: "On Wednesdays you tend to post messages a lot in the morning, despite the fact that you have a lot of meetings. You post a lot about your health, too. Is work making you feel unhealthy?" Instead we get software like LuckyCal; it's cool but, instead it says to users "I see you're going to Denver next week, can I give you an affiliate link to buy tickets to a concert by one of the artists in your iTunes library?" That's a limited view of life and the world.

When ground-breaking service FriendFeed redesigned its site recently, it didn't build out more analytics than it initially offered each user about who connected most with their content, where their content was coming from, etc. Instead, FriendFeed shut that feature down. The API is still open and so little startups launch projects like FeedStats, but really - that's so limited it's nothing to get too excited about.

Twitter is a great, wide open platform of social data. Content, connections, time and user biographies can all be cross referenced there. The company hasn't allowed for really monster big data extraction for analysis, though.

Facebook is the most closed of all the social eco-systems, but they claim they are opening up. Firefox creator and now Facebook employee Blake Ross recently called our critique of Facebook's lack of openness dishonest but did offer this worthwhile explanation:

I believe it is disingenuous to summarize Facebook as 'fundamentally closed' because we have yet to build an API that would primarily be of interest to researchers and marketing companies. We've opened all of the information that users have granted permission to open, and that most developers have asked for.

Those are telling words. Marketers (and maybe a handful of researchers, though I'd argue that those are pretty important) are the only people who want Facebook activity stream data. Users don't want it. Developers don't, Ross claims. That's sad.

The patterns of activity in that data offer a unique opportunity to learn about ourselves - individually, in groups and as a society. Unfortunately, that opportunity may not be taken advantage of. A better title of this post might be If Web 3.0 Is Poetry, Will Anyone But Marketers Read It? The gleam of contextual advertising has shined so bright that targeted advertising is thought of as gold spun out of the straw of context. Context is being treated as otherwise worthless fodder for the creation of advertising. But the stuff of our lives isn't just a pathway to market to us.

Chris Messina is one of the leaders in the movement to create standards for Activity Stream data, so it can flow from site to site and be processed in interesting ways. It's the processing part that is most "3.0-like." Messina is optimistic. "I've made an assumption that somewhat richer feeds of what people are doing will lead to more aggregate analysis," he says.

"Blake may have a point, but I think the problem is that no one has access to the kind of data we're talking about at scale yet besides folks like Facebook or Twitter. Even FriendFeed is somewhat hamstrung with whatever comes out of these services and they have to try to pick out what's going on, in a somewhat arbitrary manner. Meanwhile, Myspace is trying to be as open as possible to maintain their position in the marketplace, and no one seems to want their data.

"I think there's a Facebook-sized opportunity to address the areas that you mentioned... to do something more subtle and intuitive based on these streams. We're really far off from it happening because the technology is so primitive still - but i do think that connecting the what that someone did with the why that motivated them will become a huge area of academic research. And yes, marketers will lead the way, and probably get a lot of it wrong. But then someone else (like Apple) will come along and synthesize all this data, and help people make better decisions by looking at everything that everyone else has done in the same situation before them...and then we'll start to see it pay off."

It may "pay off" - but if that just means in commercial terms we'll all be the poorer for it. Cool commercial apps sound great, but if that's all the further this goes that will be a real tragedy. There are technical challenges for sure, but hopefully developers will aim for the sky.

Odd "Fail" pic at top by Flickr user Nimbu.