Jennifer Zeszut was the founder and CEO of Scout Labs, a social media monitoring service acquired by social CRM company Lithium in 2010. ScoutLabs does sentiment analysis, among other things, and Zeszut spoke at the O'Reilly Strata Summit on Big Data this week about the things her company has done that she believes point toward the future of this red hot tech trend.

Here's a summary of her advice about sentiment analysis, for companies that build it and customers that buy it. My take-away here? It's not just technology, it's also a lot about muscle and brains.

• Harness the wisdom of crowds - scale and let your algorithm learn. Zeszut says that once ScoutLabs got enough scale, across hundreds of companies with hundreds of users at many of them, it was common for one user to manually reclassify the sentiment on an item that was relevant to other users as well. Zeszut says that making sure her algorithm learned from that was key.
• Don't stop at positive, negative or neutral sentiment. How about noticing people at a moment of indecision? Imagine wishes, caveats, comparisons and preferences. "You should push your vendors for more than just positive, negative and neutral," Zeszut says.
• Architect for flexibility: if you built a sentiment engine 10 years ago - it won't work today. If you don't build for flexibility now - it will not work in 5 years. Several speakers throughout the day discussed the balance between art and science in working with big data.

"Thank you," Zeszut concluded, "to the Googles and Twitters of the world for giving us metadata, demographic data, geodata. With each new metadata we can filter with easily, we get be more nuanced in what we offer and be more expansive about how we build the algorithms."

Photo by Pinar Özger for O'Reilly.

The web moves fast enough that we may as well start looking at what comes next. Easy to use and affordable tools like Radian6 and ScoutLabs that track blog and twitter mentions are a given - but what kinds of crazy innovations can we hope for in the future?

Here's what we're thinking that might look like. We base these predictions in part on a conversation with Chris Golda of social media discussion search engine Backtype. Backtype is used under the hood at Radian6 and the company just announced a new partnership with Filterbox today. For an in-depth look at some of these kinds of services, see Social Media Monitoring Grudge Match: Radian6 vs. Scout Labs by Jason Falls. In house we use FriendFeed search a lot.

None of our predictions for the future are terribly shocking; to some degree they are just further developments of the same core value propositions these services already focus on. Hopefully this discussion will help spur some original thinking on your part, readers, and you can throw some suggestions for the future of the industry out in comments.

More Sources, Especially Facebook

The most obvious place for social media monitoring tools to go next is into Facebook. Twitter is the primary target for analysis right now because it's wide open and programmatically accessible. It's also 10% the size of Facebook. We're not sure whether Facebook is going to open up further or if monitoring tools are going to find a way to get around the fundamentally closed nature of the site to at least sample the conversations going on there. Can you imagine a monitoring company paying a wildly popular Facebook user to open up access to their newsfeed to monitoring? We can; it's a surprise that it's not already happening.

There's also a lot of multimedia to monitor, including television and radio content that is increasingly easy to find online. We expect that speech-to-text search services like Everyzing will find a place in the social media monitoring technology toolkit in the near future.

Source Analysis

Backtype's Chris Golda says that his company is working on more sophisticated analysis of participants in discovered conversations. If Robert Scoble says something bad about your company in comments, that's probably going to go further than if someone with a relatively small circle of connections does.

This is something that all the monitoring companies do now but it's relatively crude. We can imagine a much more sophisticated analysis. For every person online - primary circles of friends, semantic analysis of areas of interest and personal background information are all readily available in tools like Mailana, Twazzup, Calais (disclosure: RWW sponsor) and Headup. Imagine that information served up in an interface like Apture, but for every person and conversation on the web.

This is the kind of thing that data portability makes easier - when people travel around the web with their profiles and data with them, they are easier to get to know in a hurry. That's true for both the technologies that would serve up personalized content and the companies that would monitor what we do.

Conversation Discovery

If it's all about "joining the conversation," one of the biggest challenges for organizations jumping into social media is finding and prioritizing all the conversations available. Golda says that Backtype is working on becoming "like Compete for communities." Postrank is doing something like this already; it's discovering the most "engaging" blogs on a wide variety of topics. It's still ridiculously difficult to identify the most important sites of conversation on a given topic though. That's a problem ripe for solving.

We'd also like to see some real-time threshold monitoring. Let me know the moment that conversation about me or a given topic hits a certain level of intensity.

Those are some of the things we expect to see come to the world of social media monitoring in the near future. What else do you expect - or what would you really like to see?