The need to keep people from being exploited on account of violations of their privacy is clear, well-known, intuitive and amply articulated by highly capable people. The up-side of making use of peoples' data is far less so. The two concerns are closely tied together. That's something Bill Gates is likely very aware of, if his comments 10 years ago are any indication.

Bill Gates got this about the last era of computing, the first instances of e-commerce and the web. He wrote a famous company-wide memo ten years ago this month all about the importance of what a controversial hardware-based security paradigm called Trusted Computing.

"If we don't do this, people simply won't be willing -- or able -- to take advantage of all the other great work we do. Trustworthy Computing is the highest priority for all the work we are doing. We must lead the industry to a whole new level of Trustworthiness in computing."

Regarding Privacy in particular, the Gates memo put some things in ways we can relate to today, but other things seem antiquated.

"Users should be in control of how their data is used. Policies for information use should be clear to the user. Users should be in control of when and if they receive information to make best use of their time. It should be easy for users to specify appropriate use of their information including controlling the use of email they send."

Users should be in control of when and if they receive information to make best use of their time! Can you imagine that? Info overload as privacy violation. It makes sense, yet it seems hopelessly antiquated too.

"In the past, we've made our software and services more compelling for users by adding new features and functionality, and by making our platform richly extensible," he wrote.

"We've done a terrific job at that, but all those great features won't matter unless customers trust our software.

"So now, when we face a choice between adding features and resolving security issues, we need to choose security. Our products should emphasize security right out of the box, and we must constantly refine and improve that security as threats evolve."

Here's how the International Data Privacy Day organization puts it today.

"In this networked world, in which we are thoroughly digitized, with our identities, locations, actions, purchases, associations, movements, and histories stored as so many bits and bytes, we have to ask - who is collecting all of this data - what are they doing with it - with whom are they sharing it? Most of all, individuals are asking 'How can I protect my information from being misused?' These are reasonable questions to ask - we should all want to know the answers.

"Data Privacy Day promotes awareness about the many ways personal information is collected, stored, used, and shared, and education about privacy practices that will enable individuals to protect their personal information.

Robert Siciliano, an Online Security Evangelist at McAfee, paints a much more negative picture in a blog post yesterday - probably even about the companies participating in International Data Privacy Day. McAfee is owned by the primary sponsor of the event, though, Intel. Siciliano speaks for many people when he says:

"Lately, it seems that barely a day goes by when we don't learn about a major Internet presence taking steps to further erode users' privacy. The companies with access to our data are tracking us in ways that make Big Brother look like a sweet little baby sister.

"Typically when we hear an outcry about privacy violations, these perceived violations involve some apparently omnipotent corporation recording the websites we visit, the applications we download, the social networks we join, the mobile phones we carry, the text messages we send and receive, the places we go, the people we're with, the things we like and dislike, and so on.

"How do they do this? By offering us free stuff to consume online and infrastructure for the online communities that tie us together. We gobble up their technologies, download their programs, use their services, and mindlessly click 'I Agree' to terms and conditions we haven't bothered to read."

It's a cynical perspective that refers to all the glory of the Interwebs as simply free stuff to consume with mindless clicks.

I think I prefer the description Gates might have offered. The global computer is now rich with features and opportunities, but those will be put at risk if people don't trust the network. Please, Mr. Zuckerberg, don't spoil this opportunity.

The move is being widely condemned in the press as a violation of privacy but if Facebook would do this right, it could be a huge win for everyone. Facebook could be the biggest, most dynamic census of human opinion and interaction in history. Unfortunately, failure to talk prominently about privacy protections, failure to make this opt-in (or even opt out!) and the inclusion of private messages are all things that put at risk any remaining shreds of trust in Facebook that could have served as the foundation of a new era of social self-awareness.

We, ok I, have long argued here at ReadWriteWeb that aggregate analysis of Facebook data is an idea with world-changing potential. The analogy from history that I think of is about Real estate Redlining. Back in the middle of the last century, when US Census data and housing mortgage loan data were both made available for computer analysis and cross referencing for the first time, early data scientists were able to prove a pattern of racial discrimination by banks against people of color who wanted to buy houses in certain neighborhoods. The data illuminated the problem and made it undeniable, thus leading to legislation to prohibit such discrimination.

I believe that there are probably patterns of interaction and communication of comparable historic importance that could be illuminated by effective analysis of Facebook user data. Good news and bad news could no doubt be found there, if critical thinking eyes could take a look.

"Assuming you had permission, you could use a semantic tool to investigate what issues the users are discussing, what weight those issues have in relation to everything else they are saying and get some insights into the relationships between those issues," writes systemic innovation researcher Haydn Shaughnessy in a comment on Forbes privacy writer Kashmir Hill's coverage of the Politico deal. "As far as I can see people use sentiment analysis because it is low overhead; the quickest, cheapest way to reflect something of the viewpoints, however fallible the technique. Properly mined though you could really understand what those demographics care about."

Several years ago I had the privilege to sit with Mark Zuckerberg and make this argument to him, but it doesn't feel like the company has seized the world-changing opportunity in front of it.

Facebook does regularly analyzes its own data of course. And sometimes it publishes what it finds. For example, two years ago the company cross referenced the body of its users' names with US Census data that tied last names and ethnicity. Facebook's conclusion was that the site used to be disproportionately made up of White people - but now it's as ethnically diverse as the rest of America. Good news!

But why do we only hear the good news? That millions of people are talking about Republican Presidential candidates might be considered bad news, but the new deal remains a very limited instance of Facebook treating its user data like the platform that it could be.

It could be just a sign of what's to come, though. "This is especially interesting in terms of the business relationships--who's allowed to analyze Facebook data across all users?" asks Nathan Gilliatt, principal at research firm Social Target and co-founder of AnalyticsCamp. "To my knowledge, they haven't let other companies analyze user data beyond publicly shared stuff and what people can access with their own accounts' authorization. This says to me that Facebook understands the value of that data. It will be interesting to see what else they do with it."

I've been told that Facebook used to let tech giant HP informally hack at their data years ago, back when the site was small and the world's tech privacy lawyers were as yet unaroused. That kind of arrangement would have been unheard of for the past several years, though. Two years ago, social graph hacker Pete Warden pulled down Facebook data from hundreds of millions of users, analyzing it for interesting connections before planning on releasing it to the academic research community. Facebook's response was assertive and came from the legal department. Warden decided not to give the data to researchers after all. (Disclosure: I am writing this post from Warden's couch.)

"Like a lot of Facebook's studies, this collaboration with Politico is fascinating research, it's just a real shame they can't make the data publicly available, largely due to privacy concerns" bemoans Warden. "Without reproducability, it loses a lot of its scientific impact. With a traditional opinion poll, anyone with enough money can call up a similar number of people and test a survey's conclusions. That's not the case with Facebook data."

"Everyone is going 'gaga' over the potential for Facebook," says Kaliya Hamlin, Executive Director of a trade and advocacy group called the Personal Data Ecosystem Consortium.

"The potential exists only because they have this massive lead (monopoly) so it seems like they should be the ones to do this.

"Yes we should be doing deeper sentiment analysis of peoples' real opinions. But in a way that they are choosing to participate - so that the entities that aggregate such information are trusted and accountable.

"If I had my own personal data store/service and I chose to share say my music listening habits with a ratings service like Neilson - voluntarily join a panel. I have full trust and confidence that they are not going to turn on me and do something else with my data - it will just go in a pool.

"Next thing you know Facebook is going to be selling to the candidate the ability to access people who make positive or negative comments in private messages. Where does it end? How are they accountable and how do we have choice?"

Not everyone is as concerned about this from a privacy perspective. "There are many things in the online world that give me willies for Fourth-Amendment-like reasons," says Curt Monash of data analyst firm Monash Research. "This isn't one of them, because the data collectors and users aren't proposing to even come close to singling out individual people for surveillance."

Monash's primary concern is in the quality of the data. "There's a limit as to how useful this can be," he says. "Online polls and similar popularity contests are rife with what amounts to ballot box stuffing. This will be just another example. It is regrettable that you can now stuff an online ballot box by spamming your friends in private conversation."

It doesn't just have to be about messages, though. Social connections, Likes and more all offer a lot of potential for analysis, if it's done appropriately.

"We need trust and accountability frameworks that work for people to allow analysis AND not allow creepiness," says Hamlin.

Two years ago social news site Reddit began giving its users an option to "donate your data to science" by opting in to have activity data made available for download. Massive programming Question and Answer site StackOverflow has long made available periodic dumps of its users' data for analysis. "You never know what's going to come out of it," StackOverflow co-founder Joel Spolsky says about analysis of aggregate user data.

The unknown potential is indicitive not just of how valuable Facebook data is, but potentially of the relationship between data and knowledge generally in the emerging data-rich world.

That's the thesis of author David Weinberger's new book, Too Big to Know. "It's not simply that there are too many brickfacts [datapoints] and not enough edifice-theories," he writes. "Rather, the creation of data galaxies has led us to science that sometimes is too rich and complex for reduction into theories. As science has gotten too big to know, we've adopted different ideas about what it means to know at all."

The world's largest social network, rich with far more signal than any of us could wrap our heads around, could help illuminate emergent qualities of the human experience that are only visible on the network level.

Please don't mess up our chance to learn those things, Mr. Zuckerberg.

The service is powered by Wappwolf, an online “action store” that features a set of Web actions that can process files. For example, it has ready made actions to encrypt and decrypt files, extract text from PDFs, convert documents to PDF, generate QR codes and manipulate images.

The Dropbox Automator works by connecting to your Dropbox account and then defining actions based on which folder you place files into. For example, I connected my Dropbox account and created a folder called Appwolf. Then I defined actions to convert files placed into that folder into PDFs.

You can also do things like upload files to Slideshare, sign PDFs, scrape PDFs to text files and even translate files automatically using Bing Translator. It looks like much of Automator’s functionality just comes from tapping into Web-based services.

You can also automatically upload photos to Facebook or Flickr, add a bug (stamp) to a photo, resize or rotate photos and much more.

A Few Glitches

I found that the service isn’t entirely glitch free. It says that it can covert HTML files to PDF, which it does… but it just converts the text to PDF, so the tags are presented in the document instead of used for formatting. It might be that you need the header information before the service (conf2pdf) properly recognizes the file as HTML instead of plain text.

When Dropbox Automator zips files, it uses a format that doesn’t seem to be recognized on Mac OS X as a zip file. At least not by the Archive Utility that comes with OS X Lion.

It does convert OpenDocument Format (ODF) files OK, when it actually converts them. Of two ODF files I placed in the Appwolf directory, only one was converted. The other was placed in the processed folder that Dropbox Automator creates, but no PDF ever materialized.

But it’s a brand new service and I suspect they’re still shaking the bugs out. The service, at least for now, is free. How will they make their money? It’s unclear, but some of the actions you set up for files may cost money. So it’s possible that the developers will add premium services or charge a fee to other services for connecting users. If it catches on, I do hope that they start providing paid accounts so users can support the service.

How do they do it? With a combination of math, science, creative thinking and long hours of hard work. As democratized publishing, network connected devices and the instrumentation of everyday life combine to create a great blue ocean of big data all around us, the latest Mu Sigma funding is a valuable opportunity to get a taste of how one emerging leader in that market combines technology, math and art to engage with this big trend. Not everyone agrees that outsourcing Big Data work like this is the solution, though.

Mu Sigma says it exists to "enable businesses to institutionalize data-driven decision making." Its 1300 employees in Chicago and Bangalore help clients with marketing, supply chain and risk analytics. The firm says it "is arguably the world's largest pure-play decision sciences and analytics services company."

Employee reviews of the company on website GlassDoor paint a picture of hard-driving young employees working grueling hours for low pay, but learning a lot at a young company.

The seven year old firm helps clients with things like customer segmentation and purchase likelihood analysis in marketing, fraud detection and severity and statistical analysis for FDA trials in risk analysis and supply chain work like trend plotting, due date quoting, expedition optimization, location allocation "decisioning", etc. All based on data.

How can Mu Sigma compete in each of those tasks with other firms that specialize in one or the other? That's unclear, but the company has developed momentum based on its broad approach. Mu Sigma says that it's profitable, though the company declined to provide any specific financial numbers.

Not everyone believes that solutions like Mu Sigma are the answer to Big Data problems and opportunities. "I'm skeptical of the idea of end to end 'analytics outsourcing' right now," says Peter Skomoroch, of DataWrangling.com.

"There is value in having external experts embedded with internal teams to help with big data, but to compete companies will also need to build up in-house talent.

"It is tough to find good data people, and even more difficult to find ones with business sense and domain knowledge. Insight and creativity are not likely to be commoditized any time soon. The competitive advantage in this space will go to companies that build up unique datasets and build teams that know how to leverage them. Most game changing analytics is going to come from a small set of talented individuals, not an army of contractors."

In-house data scientists are incredibly hard to find, though. Cathy O'Neil, data scientist at ad startup Intent Media, says this is in part because "It is far less sexy to try to honestly find the confidence interval of a prediction than it is to model behavior."

"Data scientists are considered magical when they forecast behavior that was hitherto unknown, and they are considered total downers when they tell their CEO, 'hey there's just not enough data to start that business you want to start,' or 'hey this data is actually really fat-tailed and our confidence intervals suck.'

"In other words, it's something like what the head of risk management had to face at a big bank taking risks in 2007. There's a responsibility to warn people that too much confidence in the models is bad, but then there's the political reality of the situation, where you just want to be liked and you don't actually have the power to stop the relevant decisions anyway."

Perhaps given that reality, outside big data firm Mu Sigma is clearly a company with some economic wind in its sails. Deborah Gage at the Wall St. Journal's Venture Wire provides a good look at the company's fast growth and interesting training program in her coverage this morning.

Mu Sigma and Innovation

Reading previous coverage of the company's work elsewhere, one name keep coming up: Zubin Dowlaty, Vice President and head of innovation and development at Mu Sigma.

Dowalty spent the 1990's doing statistical modeling at UPS. Then he joined the publicly traded InterContinental Hotels Group, where he was first the Director of Analytics in Consumer Insight and then the VP of Decision Sciences. He was featured prominently in a 2008 New York Times story about corporations using Prediction Markets to surface cost-saving and other ideas from inside their companies. Dowalty was photographed for the story wearing a wizard's cap and holding a magical looking walking staff in his hands.

He built an elaborate system to invite the hotel company's employees to submit and vote on ideas, win rewards if theirs were selected and to surface via crowdsourcing strategic initiatives the company could act on. "We wanted to tap the creative class that may not be able to voice their ideas," Dowlaty told the Times.

Once at Mu Sigma, Dowlaty has become one of the company's most visible public figures. His statements, as the head of innovation and development at a firm so focused on innovation, are noteworthy.

In a January 2011 article from The Data Warehousing Institute on the rise of the data scientist, Dowlaty articulates the role of art and of science in big data.

"I'm not a big fan of the spaghetti method. It makes me nervous when people run a lot of analytic techniques just to get the answer they want, instead of being objective. Doing this job properly requires the rigor of a scientist. The scientist can see things that other people cannot see."

As a standalone statement, that doesn't sound particularly creative. It is important, though. "The 'spaghetti method,'" cautions Josh Wills, Chief Data Scientist at Cloudera, "frantically searching for a technique that gives you the answer you want (or potentially, the answer that someone higher up in the org wants), as opposed to using the scientific method. This is a big problem in the industry, and the theory is that using an external firm mitigates that habit to some extent. Being a good data scientist often means telling powerful people stuff that they don't want to hear."

Other statements from Dowlaty help put that sentiment about rigor in creative context. Mu Sigma itself uses a variety of different analytic models to tackle all the problems they engage with.

Dowlaty told Revolution Analytics, whose R statistics software Mu Sigma makes use of:

"We like to diversify our models...We have a portfolio of about 10 models that we'll run to assess the stability of the coefficient and the predictive capability of that particular model. By running all the models, you can see which ones are the best predictors."

Revolution says of Dowlaty's use of R at Mu Sigma, "The benefit of an 'ensemble' approach is that when new analytic techniques emerge, they can be brought into the mix without causing disruption. This makes the R especially valuable to Dowlaty, since the R software library evolves continually as members of the worldwide R community contribute new packages and programs."

In fact, both rigor and flexibility are key to the paradigm Dowlaty advocates. "The trend is toward a multi-disciplinary approach to extracting value from data," he told The Data Warehousing Institute early this year. "It's not just about math anymore. You also need technology skills, but what ultimately separates the analyst from the scientist is the dimension of artistic creativity. It's the soft skills that make the big difference."

That combination of skills is what enables the firm to tackle the incredibly complex work they do. Dhiraj Rajaram, Mu Sigma's CEO and the man who founded the company in 2004, spoke at the 2010 Predictive Analytics World conference on a panel with Mu Sigma customer Walmart.

Walmart Financial Services, which named Mu Sigma its Supplier of the Year in 2011, works with the big data company to analyze and optimize the marketing of its financial products.

Decision Management analyst James Taylor blogged the following summary of the conference presentation about the collaboration between Walmart Financial Services and Mu Sigma. This sounds like very complicated work.

"WFS uses transaction life analysis around run rate and growth, price / mix analysis, financial returns and qualitative analysis of the creative. Marketing Mix modeling, optimization, lets them see the effect of individual campaigns (there's a lot of Walmart stuff going on in the market), account for seasonality and manage at the store level. The idea is to make sure the marketing investment is optimized, targeted and repeatable.

"Marketing mix optimization uses weekly sales, store traits and demographics, event information and macro-economic data to see how effective specific events were and what was the contribution to the overall effect. What was the value or contribution of each element, did they cannibalize each other, did they resonate in specific areas etc."

That sounds like a potent combination of math, science and creative thinking. It's probably more a picture of the sector than of one company alone. Forrester analyst James Kobielus specializes in big data and says he's done one briefing with Mu Sigma but didn't detect any particular unique flavor to the firm's work relative to others in the sector. Mu Sigma hasn't yet responded to our request for comment on this article.

Perhaps this company is typical of the sector and the questions to ask about it are more general.

"My caveat with services like MuSigma is that they can analyze your data, but they can't change your business," says Cloudera's Wills.

"You are free to ignore what they tell you, and it is often the case that the answers they can give you are limited by your business practices and the data that you currently collect. The advantage of having an in-house data scientist, especially one with some programming skill, is that they can develop systems that collect better data so that they can come up with better answers.

It's an incredibly feature-rich new type of social network profile. It's a re-imagination of what a profile can be. It makes me want to use Facebook more, to share more data with Facebook so that it can be preserved and displayed so nicely, years into the future. While other Facebook features have pushed users into posting publicly by default, or posted their activities from other places they didn't understand would become part of the public record, I think Timeline is a genuine value add to incentivize users to share more. I think it's great.

It's one thing to see this data all in a News Feed as Facebook has long showed it, it's fundamentally different to see Yourself and Others presented like a work of art in this new Timeline layout.

By highlighting the content you've published that has received the most social engagement, in the form of comments and Likes, your Facebook Timeline takes its best shot at presenting your Best Self to the world. The mundane updates are hidden in the background and the highlights of your life, if you posted about them on Facebook, are programmatically discoverable and now displayed in an attractive page layout.

It doesn't work perfectly, my Timeline says that I married my wife 3 times on 3 different dates, but generally speaking it works really well. It looks great on m.facebook.com too.

The Facebook Timeline represents the Instrumentation of Your Life, making things measurable and then building on top of those measurements. It's a big deal in the world of social software.

That Facebook launched such a bold new implementation of every user's data about themselves just months after getting slapped with a 20 year privacy audit requirement from the US government is bold.

As Not Seen on Twitter

Meanwhile, over on Twitter, that competing social network can't remember what you did two weeks ago. It does remember, it just won't let you remember. Historical content on Twitter is severely limited.

The company has said officially that's because Twitter is all about the here and now, it's real-time. Unofficially it's said though that the root of the problem was in a series of database creation decisions that were made years ago. It would now be super expensive to change that.

There is something about Twitter that's more conversational, more News focused and less conducive culturally to something like Timeline.

For the vast majority of its users, I'd also guess that Twitter accounts post fewer messages and get fewer responses that can be measured to determine highlights than is the case on Facebook.

Facebook also has a lot of structured data in the user's profile and changes to that become events, which social activity swarms around and which then become notable points in your life. You changed your marital status? That's probably going to get a lot of discussion. There is no equivalent on Twitter. Were Twitter to highlight your biggest tweets, they would likely be the wittiest quips you've made over the years, not the real life events.

Twitter is working on convincing people that tweets are great for reading, that it's largely a reading experience. Facebook, on the other hand, has always wanted you to share, share, share.

Many of us are doing things outside of Facebook, though. A lot of that is being shared back into our Newsfeed, but not all of it. I am very impressed with what Facebook has done, but I wish there was some more effective competition out there. There are various startups who have tried to do this, though none anywhere near as well as Facebook's hired and acquired team of world-beating design pros.

I joined Facebook 5 years ago this Fall, according to my Timeline. It's cool to see all that history presented so nicely and it makes me want to put more content into Facebook so I can see it later. I imagine that's the point.

Engag.io, which gets its name from being the place for your online engagement input and output, is like an inbox for all your conversations on Twitter, Facebook, Google Plus, Foursquare and blog comments. It's an inbox with analytics. It's built by the team behind content curation company Eqentia. Eqentia is ambitious but a little too complicated; Engag.io is very simple and the value of it will be immediately obvious to many people.

In order to get started with Engagio, you have to authenticate with different services you use around the web. Fortunately, this has become super easy to do and very secure with just a few clicks. The open authentication standards that have been developed over recent years make mashups like Engagio really easy to implement and that's awesome.

You can log in with your accounts on Twitter, Facebook, Foursquare, Google Plus, Disqus, Hacker News and Tumblr. Then Engagio will watch for comments posted to and from you on any of those services and give you one unified inbox to track the conversations inside of.

"We believe that having a universal Conversation Inbox could become a daily time saver," says the Engagio blog. "It will save you time because you don't have to check the multiple source sites where you have placed your comments. And you can for example focus on replies first, before you get to other commenting."

That's pure gold, right there - but a few days later I'd already forgotten who said it to me, where to find it, etc. Enter Engagio Comment Search and boom! All my problems are solved.

It's a great idea and I've been returning to the site daily to try and stay engaged with people who took the time to respond to me around the web. It's mostly Twitter conversations and some Google Plus threads in my experience, but I hope that Engagio will help me be all the more...in touch with conversations in other places too.

The analytics part of the service could really use some UI work, but the idea is that Engagio will show you who you're interacting with the most. You might be surprised who some of your top responders are - and those are people you should probably engage with all the more. Or at least know, if you're going to be as social as you might want to be in the social media.

The Engagio team could use someone to sit down with them and go through some real-life commenting experiences because I think the user flow could really be improved. Site founder William Mougayar is a commenting machine, he posts comments all the time everywhere, but I suspect his experiences are different from the way other people would want to use a service like this.

Super blogger and tech investor Fred Wilson, a man who gets more and more intelligent comments in response to his online activity than probably anyone else you'll ever meet, has been a cheerleader for Engag.io. Wilson says he urged Mougayar to "make it like gmail for social conversations." Gmail is deceptively simple though and Engagio will take more work to get close to that level of usefulness. As FAKE GRIMLOCK put it, "IS MVP. UGLY OK FOR NOW." A minimum viable product it is, but one that I think many people will want to see developed further.

That this is a tool designed to make the living social graph more transparent and sticky is exciting. I absolutely love the idea. Several users have pointed out that a mobile interface would suit real user behavior especially well and I agree with that.

A small number of people can jump in and kick the tires now, using the code "rwwengage" at Engag.io.

ifttt allows anyone to set up a chain of conditional actions between a wide variety of web services, like "If I post a photo to Flickr, save it to my Dropbox." The company calls these "recipes." We wrote about the service when it launched to the public in September. Microsoft's Scott Hanselman also wrote up a nice review of the service and says "this is going to be huge." ifttt isn't just a single service, though, and it isn't even just an amalgamation of multiple services strung-together; it's a great example of a whole paradigm of DIY mashups. As Blogger and WordPress were to self-publishing and YouTube was to video publishing, so ifttt could be to working with interlinked web applications for everyday people. Can this startup herald a new era of lay hackers? The UI is good, the only question is whether there's really enough demand for such a service.

Here's how the startup introduced itself at launch:

"We began with the theory that as our digital tools became more domain specific and easier to use, there would be vast amounts of creative potential in how any two tools might be used in tandem. We knew that with this immense potential came a problem of equal proportions. There just aren't enough developers and designers in the world to craft all these connections. A million developers at a million laptops wouldn't even make a dent. So we set out to build an incredibly simple tool that anyone could use to define creative, event-driven tasks that fit the pattern 'if this then that.'"

If there's anyone who can pull something like this off, having experience at IDEO is great background from which to give it a shot.

The most popular ifttt recipes are here; co-founder Linden Tibbets's are here.

Is this something that a whole lot of people are going to be interested in and go to the trouble to do? I know I am and I wouldn't be surprised to see that you are, RWW readers, but it will be interesting to see how this becomes a business. Either way, it's great to see one of the web's most interesting investors back something so focused on generating creative use of online tools.

The honeymoon has not been sweet.

Publishers did not have the tools to create fully functional magazines from the very start. Sure, they were nice looking, but that was about it. Over the last two years, though, publishers and developers have created dynamic tools that allow the news media to create apps that do not just meet user expectations, but go beyond them.

What can these services offer now that was more difficult to do when publishers initially came around to the idea of tablet magazines and newspapers? Quite a bit, actually. Let's look at what Urban Airship can do for publishers looking to put magazine apps in Apple's iOS Newsstand.

• Newsstand library support: configures newsstand apps for auto-renewable subscriptions and single issue purchases through Apple's in-app purchasing toolset.
• Content delivery: Hosts and delivers content available to users on any iOS device.
• Push notifications: This is a big one and perhaps one of the main items missing from original tablet apps. The ability to create easy push notifications enables publishers to put news right under the noses of users, increasing engagement and page views and alerting readers of new issues. The evolution of push notifications for mobile and tablet apps will be a critical piece in how publishers utilize the devices.
• Audience segmentation: Who is reading what? Where do you want to send certain readers? Urban Airship has "tags and aliases" features to associate mobile users with publishers CRM and personalize the experience.
• Free seed content: Users hate to download a news app and then automatically be forced to subscribe before getting content. Publishers need to sweeten the deal a little bit and give users free content, such as back issues, listed next to subscription services.
• Reports: How are users interacting with the news app? Analytics is the heart of understanding mobile users and there are a variety of companies out there, from Urban Airship, Localytics, Flurry, Apsalar and others that can help administrators reach the core goals of engagement and behavioral analysis.

What it comes down to is that publishers need to start looking at tablet apps like a developer would look at any other app, like a game or a utility. The goal is to gain reach, engage users, analyze behavior and monetize it. The Guardian and Future Publishing use Urban Airship and large publishers employ Flurry, Apsalar and Localytics for engagement and advertising insights.

There are also design tools that publishers can turn to for creating news apps that have evolved in the last year or so. Mag+ recently announced a new suite of publishing tools and pricing tiers for the smallest publisher to the largest magazine. Designed by editors and designers for editors and designers, it provides three tiers of publishing tools. There are also solutions from companies like Daylife, which launched real-time media management in June.

Apps are a data driven ecosystem, tied to the cloud, in the pockets of users on the go. News publishers need to understand the tools that can help them make the jump for the decaying old media standard to flexible and responsive new media technology companies.

The Learning Resource Metadata Initiative (LRMI), a project co-led by the Association of Educational Publishers and Creative Commons, today took the next step towards submitting its specification to Schema.org, the collaboration between Google, Yahoo and Bing that maps out 100 different types of content online in a standardized format.

Active participants working to figure out how to construct LRMI and how to integrate it into Schema.org include people from small non-profits like open curriculum community Curriki, corporate education technology giant Pearson, international information standards group Dublin Core and intellectual property law group Creative Commons, among others.

Participants debate on the official mailing list over new terminology, balancing concerns like coherence with Schema.org, ease of input by people who will enter metadata to go with resources being published online and specificity gained or lost by the way that metadata fields are named and framed.

While some semantic technologies are able to assert categorization from the top down, whether content publishers participate or not, it seems likely that the kind of data that will be communicated in LRMI will require informed participation by the producers of the content themselves. Requiring participation in categorization could pose a challenge to hopes the spec will gain meaningful adoption.

The LRMI effort doesn't seem well-known yet outside its own ranks, either; the official website has almost no inbound links indexed by Google yet and none of the education technology blogs we track here at ReadWriteWeb have mentioned LRMI yet. The project was just announced last month though and in the education market, a month isn't a very long time.

LRMI isn't alone though, either. Nathan Angell, a Board Director at the collaborative open education software community Sakai Foundation and a Product Manager at rSmart, calls LRMI "another welcome intervention in growing list of data specifications for education."

"These days we have access to an unbelievable number of learning resources--both open and proprietary--but it's still hard to find the right ones, quality resources, suited to your needs, when you need them.

"For example, in the Sakai community, we have built a new platform--the Open Academic Environment--that helps people create and tag learning materials, and most importantly, share them openly by default.

"With the LRMI specification, we can help people tag their materials with exactly the right information that will make them easy for others to find and use...and even better, we can augment the suggested content widgets we already have in place to discover resources in the moment that match the very specific needs of a particular educator or student."

Angell, who isn't associated with LRMI in particular, sees data specifications like this as potential game changers. Those suggested content widgets are really shorthand for computation that can begin at a higher level of abstraction if the hard work of content categorization and description has already been done in a standardized way. That means education technology providers, search engines and others don't have to invest time and energy into understanding educational resources online - they can begin with a pre-existing understanding of that content and then offer higher-level features and services on top of already-organized information.

"LRMI helps set the stage for the hive mind that will help our children's children learn faster and better than we ever thought possible," Angel says. "In comparison, school today will look like drawing pictures in the dirt with a stick."

Those who hadn't memorized Metro's schedules had to employ its persnickety Trip Planner, a clunky Web form that not only won't let you click on a map to specify your location but also chokes on cities, states, Zip codes and even commas if you add them to a street address. Meanwhile, other U.S. cities had enjoyed transit directions from sites like Google Maps since at least 2005. But not DC.

Start lobbying to open a conversation. Greater Greater Washington editor David Alpert quickly had hundreds of signatures on a petition protesting the decision. That persuaded Metro to detail its objections: fear of losing $68,000 in yearly ad revenue from the Trip Planner page, a wish to be paid for its data, and the legalese around data sharing. That then widened the discussion from an argument over APIs to one over the proper use of taxpayer dollars.

"David came along and started this campaign," said Christopher Zimmerman, an Arlington County Board member who chaired Metro's board in 2008. "It wouldn't have happened without the public pressure." Alpert could also lobby Metro from closer in after joining its Riders' Advisory Council in January of 2009.

Flipping the debate from potential profits to actual expenses. Gordon Linton, a former head of the Federal Transit Administration who served alongside Zimmerman on the Metro board, focused on the opportunity cost of giving data to other sites. He said the agency had given away resources - for example, free parking for car-sharing services - that it later realized could yield income.

"While we were raising fares and cutting service if we were investing staff time and energy for a product that would reap some financial benefit for those who would use it and sell it, then we in turn should get some money back," Linton said.

Zimmerman took the opposite argument: the ease of upgrading one aspect of the Metro experience. "We were having a lot of difficulties," he said. "If some of these things don't cost us anything or don't cost us a lot [to fix], we should do them right away."

Alpert suggested this debate encouraged Metro staffers to rethink things. "Staff may have felt they were under orders from the board to maximize revenue. Zimmerman gave them permission not to worry about that."

Oh, and Google never had any interest in paying for a schedule feed. Wrote spokeswoman Anne Espiritu: "We do not pay agencies for their data."

A change in leadership can help. All of this effort got sidetracked on June 22, 2009 when two Red Line trains collided and killed nine passengers. Things were set back further when general manager John Catoe unexpectedly resigned in early 2010 and his interim replacement Richard Sarles had to focus on safety upgrades.

But Sarles' earlier employer, NJ Transit, had provided rail schedules to Google back in 2008. He wanted to follow suit here, said Metro spokesman Dan Stessel.

Metro and Google signed a data-sharing agreement in July of 2010, once Google had dropped earlier demands for an indemnification clause. Metro directions showed up on Microsoft's Bing Maps in September 2010; they arrived on Google in May after a final shove from Sarles following his appointment as WMATA's full-time general manager in January.

(But even now, you must return to Metro's sites or a few third-party apps for bus and train arrival predictions. Google only announced a standard format for that data, GTFS-realtime, in August, and Metro is still weighing support for that. And some regional bus systems that Metro's Trip Planner includes have yet to provide their own GTFS data to mapping sites.)

What about the original financial arguments? We may never know how the math worked out: without detailed surveys, you can't draw a line from clicking on maps to boarding trains. But Stessel, who didn't provide a dollar cost for the work involved, suggested that the rationale merchants invoke to invest in intangibles like store or site designs works for public transit too: "Our primary motivation is improving the customer experience." Zimmerman had a more philosophical justification for how a government agency should act: "Putting information in the public domain is part of what we do."

Sometimes, being open isn't easy. You could say that there were many stops on the route that Metro took.

That's what Twitter is. Inside every unborn tweet you can find infinite potential - someone will be in a place, with social context and they will say something, anything, and give that potential a form. They will say something and it will be instantly available to anyone in the world who's subscribed. Each tweet has more than 30 fields of metadata under the hood; the value populating each of those fields makes up the unique patterns of holes in the metal cover that lets the light out from inside. A company launched today that lets you control a robot that drills holes in the metal covers trapping the infinite potential of the sun inside.

"One could spend months mining Twitter using @DataSift," said Paul M. Watson, CEO CTO of stream curation startup Storyful, today on Twitter. "Great balance of usability & power in the CSDL." (Curated Stream Definition Language)

Were one to look into the screaming firehose of hundreds of millions of tweets on Twitter and call out to the robot gods of data sifting, "Give me the tweets by self-proclaimed South Africans, living in Ireland, with positive sentiment and that have been retweeted by data-loving tech investor Roger Ehrenberg!" Were one to dip into that river in search of a sliver like that, which may or may not exist (it does, Watson's fits the bill), then the freshly launched startup Datasift would be the tool one would use to do so. It would cost you pennies, too.

Years in the making, Datasift launched today as the second licensed reseller of tweets. The startup doesn't just resell tweets like yesterday's news, though.

Datasift lets anyone parse the full firehose of Twitter messages with its simple Curated Stream Definition Language and see the resulting flow of messages that fit the criteria described.

You wouldn't likely ask for all the tweets from self-proclaimed South Africans living in Ireland, but you might ask for all the tweets posted by women living in a particular state in the US and using any of a list of keywords. Market research, political monitoring, news reporting, there are all kinds of use cases. Whether the set of would-be customers intersects substantially with the set of people curious enough to think of the right questions to ask the data is a big question.

Anyone can say anything on Twitter, and with Datasift you can ask whether anything is being said. 80%+ of retweets of @justinbieber are from Females, the company says.

Datasift just opened to the public today, so despite its best efforts there are still some kinks that need to be worked out. While the team attended its launch party, the preview rendering functionality stopped working for a little while. The pricing, while explained a number of different ways, still needs more clarification and development, founder Nick Halstead said today on Twitter.

The query and filtering tool isn't as easy to use as the company would like it to be, either. "It is decidedly not easy to use," says recent University of Washington Masters of Science in Information Management graduate and former ReadWriteWeb researcher Emily Cunningham. "I find their UI clumsy. Creating streams is confusing, not intuitive at all, hard to understand."

But the potential here is huge. It's a simple proposition, too: Twitter is now an incredibly rich source of information about all kinds of topics. "I learn about most things through Twitter," DataSift's new CEO Rob Bailey told Vator.tv this week. "I spend more time on that platform than I do on any cable channel . . . I've learned so much more about events like the occupy movement that cable news just wasn't covering . . . and I was able to see that the RIM audience was getting more and more frustrated leading up to the outages they experienced. It's such a powerful news source."

Those are words that many of us can relate to, but Bailey now leads a team of engineers building a tool that aims to make it relatively simple for anyone to create filters to capture the messages about the events that we all catch wind of on Twitter.

Anyone who's willing to pay for some tweets, that is, even at a low low price. Datasift is intended for businesses users.

Can Datasift build a business serving a broad range of business users interested in juggling those spheres of light, filtering by patterns and in turn using that information to create new levels of value? Time will tell. It's a very ambitious undertaking.

Disclosure: The author leads a stealth startup that works in the social media data space as well, more likely a Datasift customer than a competitor though.

The Foundation is an organization dedicated to leveraging the falling costs of crawling and storage for the benefit of "individuals, academic groups, small start-ups, big companies, governments and nonprofits." It's lead by Gilad Elbaz, the forefather of Google AdSense and the CEO of data platform startup Factual. Joining Elbaz on the Foundation board is internet public domain champion Carl Malamud and semantic web serial entrepreneur Nova Spivack. Director Lisa Green came to the Foundation by way of Creative Commons.

"Common Crawl is a Web Scale crawl, and as such, each version of our crawl contains billions of documents from the various sites that we are successfully able to crawl. This dataset can be tens of terabytes in size, making transfer of the crawl to interested third parties costly and impractical. In addition to this, performing data processing operations on a dataset this large requires parallel processing techniques, and a potentially large computer cluster.

"Luckily for us, Amazon's EC2/S3 cloud computing infrastructure provides us with both a theoretically unlimited storage capacity coupled with localized access to an elastic compute cloud."

The organization was formed three years ago, just now started talking about itself publicly and believes that free access to all this information could lead to "a new wave of innovation, education and research."

Open Web Advocate James Walker agrees: "An openly accessible archive of the web - that's not owned and controlled by Google - levels the playing field pretty significantly for research and innovation."

The potential for syndicated Tweets is big, but hopefully Twitter won't go after everyone else in the world who puts Tweets on other websites as a part of their business. The company doesn't seem to be welcoming interested parties to license those rights either. We've asked Twitter for comment on the prospect of enforcement of the prohibition against unofficial resyndication of Tweets (who said this stuff was free as the wind?) but haven't heard back from the company yet. (Update: Twitter's comment below.)

Below: No unauthorized Tweets on TV! An example from licensee Mass Relevance's website.

Twitter as a developer platform has traveled a rocky road, lots of highs and lots of lows. Enabling serious business use of Tweets is going to be an important next step. Hopefully that will happen in a way that's accessible to small developers and allows them to create the fabulous things a broad ecosystem can produce better than one consisting of a limited set of high-end companies.

A number of the leaders of Twitter's partnership team came from Current.TV, where in the last Presidential election there was some great work done showing live Tweets with live video of Presidential debates. Will use cases like that have to pay up in the future? Will it be a price that won't stifle experimentation? For what it's worth, were we all posting messages on a distributed, open source, microblogging protocol this probably wouldn't be happening.

Writing on the Twitter developers' blog, Twitter's Jason Costa wrote today, "Expect to see additional partnerships of this kind as we look for new ways to help everyone get the best out of Twitter."

ORLY? That doesn't sound like "click here to buy a license." That sounds like Twitter is going to drive this themselves, the company is looking for new ways to help everyone get the best out of Twitter. Isn't the lesson of a platform that no single company will ever be able to produce as much innovation as a larger ecosystem of independent developers?

Am I the only one feeling uncomfortable about this?

Update: In response to this article and an email inquiry, Twitter's Jodi Olsen offered the following comment. "We're not disclosing terms of the deal, but this is about growth--not a revenue play for us. Our goal with this partnership, which we expect to be the first of many, is to empower the ecosystem and help media grow."

The idea is great, though: calendar data as the basis of new features. People say that your telephone contact list is your ultimate social network - and they say that your email inbox is a potent platform for potential products and services to be developed on top of. What about your calendar, though? The new Google Calendar feature is just the tip of the iceberg - so much more could be made possible if the data latent in our calendars were made programmatically available and turned into a platform for development. Like what? A few ideas are below, I'd love to learn about any other ideas you've got, too.

Imagine...

• When did you last have a meeting scheduled with the same person you're scheduling something with now? Could meeting notes or other assets be linked?


• Who else might you want to invite - like Gmail's who else to email recommendation feature.


• "Would you like to schedule a Google Hangout at this time?"


• Data visualization of historical patterns of your schedule. Did you know that you tend to schedule far fewer meetings on afternoons later in the week than in the mornings early in the week? I'd like to see a visualization of my contacts' patterns of busy and free so I know when a good time to suggest a meeting might be.


• Are you about to schedule something that will conflict with an event on a public events calendar and that you're likely to be interested in? That would be awesome to be notified about. ReadWriteWeb staff hacker Tyler Gillies has built a Plancast layer on Geoloqi -it triggers a push notification to peoples' phones whenever they are in a part of town where a public event with the category "nerdout" has been planned. That's pretty awesome and paints a picture of what the future might look like.

John Girard, a Portland entrepreneur, says on Google+ that he wants to see calendar events and location integrated. "Who is in the office and who isn't; who will dial in; who is traveling and unavailable etc.," he writes. "One of my big issues with Tungle has been that it's still not good at helping me to schedule in person meetings when I am traveling. The temporal is only one aspect of meetings - geography / location (whether virtual or actual) is almost completely neglected as of yet."

Some people just want Google Calendar to function like other apps do. "I was hoping for a public events offering from Google," says gold spinner at Gravity.com Robbie Coleman. "Tired of evite and not all of my friends/family have (or want) Facebook accounts."

Time is a factor that can be cross referenced with almost anything else to return interesting new data points. Ought our calendars not do more for us than sit on our computers and our phones, all alone and boring? I think they ought to. What more would you like to see your calendar do?

Sentiment analysis, user ranking, data from sensors and potentially much more can all be taken into account in requesting data from the API. Connectors have been built for "all sorts of databases including PostgreSQL, MySQL, Oracle, HBase, and MongoDB as well as an even newer types of databases and APIs like Google Fusion Tables." Awesome.

That sounds fabulous to me. When physical place and the real-time social web come together in the form of streaming data APIs, the possibilities for augmenting time, place, civil society and the meaning of the web are substantial.