How do they do it? With a combination of math, science, creative thinking and long hours of hard work. As democratized publishing, network connected devices and the instrumentation of everyday life combine to create a great blue ocean of big data all around us, the latest Mu Sigma funding is a valuable opportunity to get a taste of how one emerging leader in that market combines technology, math and art to engage with this big trend. Not everyone agrees that outsourcing Big Data work like this is the solution, though.

Mu Sigma says it exists to "enable businesses to institutionalize data-driven decision making." Its 1300 employees in Chicago and Bangalore help clients with marketing, supply chain and risk analytics. The firm says it "is arguably the world's largest pure-play decision sciences and analytics services company."

Employee reviews of the company on website GlassDoor paint a picture of hard-driving young employees working grueling hours for low pay, but learning a lot at a young company.

The seven year old firm helps clients with things like customer segmentation and purchase likelihood analysis in marketing, fraud detection and severity and statistical analysis for FDA trials in risk analysis and supply chain work like trend plotting, due date quoting, expedition optimization, location allocation "decisioning", etc. All based on data.

How can Mu Sigma compete in each of those tasks with other firms that specialize in one or the other? That's unclear, but the company has developed momentum based on its broad approach. Mu Sigma says that it's profitable, though the company declined to provide any specific financial numbers.

Not everyone believes that solutions like Mu Sigma are the answer to Big Data problems and opportunities. "I'm skeptical of the idea of end to end 'analytics outsourcing' right now," says Peter Skomoroch, of DataWrangling.com.

"There is value in having external experts embedded with internal teams to help with big data, but to compete companies will also need to build up in-house talent.

"It is tough to find good data people, and even more difficult to find ones with business sense and domain knowledge. Insight and creativity are not likely to be commoditized any time soon. The competitive advantage in this space will go to companies that build up unique datasets and build teams that know how to leverage them. Most game changing analytics is going to come from a small set of talented individuals, not an army of contractors."

In-house data scientists are incredibly hard to find, though. Cathy O'Neil, data scientist at ad startup Intent Media, says this is in part because "It is far less sexy to try to honestly find the confidence interval of a prediction than it is to model behavior."

"Data scientists are considered magical when they forecast behavior that was hitherto unknown, and they are considered total downers when they tell their CEO, 'hey there's just not enough data to start that business you want to start,' or 'hey this data is actually really fat-tailed and our confidence intervals suck.'

"In other words, it's something like what the head of risk management had to face at a big bank taking risks in 2007. There's a responsibility to warn people that too much confidence in the models is bad, but then there's the political reality of the situation, where you just want to be liked and you don't actually have the power to stop the relevant decisions anyway."

Perhaps given that reality, outside big data firm Mu Sigma is clearly a company with some economic wind in its sails. Deborah Gage at the Wall St. Journal's Venture Wire provides a good look at the company's fast growth and interesting training program in her coverage this morning.

Mu Sigma and Innovation

Reading previous coverage of the company's work elsewhere, one name keep coming up: Zubin Dowlaty, Vice President and head of innovation and development at Mu Sigma.

Dowalty spent the 1990's doing statistical modeling at UPS. Then he joined the publicly traded InterContinental Hotels Group, where he was first the Director of Analytics in Consumer Insight and then the VP of Decision Sciences. He was featured prominently in a 2008 New York Times story about corporations using Prediction Markets to surface cost-saving and other ideas from inside their companies. Dowalty was photographed for the story wearing a wizard's cap and holding a magical looking walking staff in his hands.

He built an elaborate system to invite the hotel company's employees to submit and vote on ideas, win rewards if theirs were selected and to surface via crowdsourcing strategic initiatives the company could act on. "We wanted to tap the creative class that may not be able to voice their ideas," Dowlaty told the Times.

Once at Mu Sigma, Dowlaty has become one of the company's most visible public figures. His statements, as the head of innovation and development at a firm so focused on innovation, are noteworthy.

In a January 2011 article from The Data Warehousing Institute on the rise of the data scientist, Dowlaty articulates the role of art and of science in big data.

"I'm not a big fan of the spaghetti method. It makes me nervous when people run a lot of analytic techniques just to get the answer they want, instead of being objective. Doing this job properly requires the rigor of a scientist. The scientist can see things that other people cannot see."

As a standalone statement, that doesn't sound particularly creative. It is important, though. "The 'spaghetti method,'" cautions Josh Wills, Chief Data Scientist at Cloudera, "frantically searching for a technique that gives you the answer you want (or potentially, the answer that someone higher up in the org wants), as opposed to using the scientific method. This is a big problem in the industry, and the theory is that using an external firm mitigates that habit to some extent. Being a good data scientist often means telling powerful people stuff that they don't want to hear."

Other statements from Dowlaty help put that sentiment about rigor in creative context. Mu Sigma itself uses a variety of different analytic models to tackle all the problems they engage with.

Dowlaty told Revolution Analytics, whose R statistics software Mu Sigma makes use of:

"We like to diversify our models...We have a portfolio of about 10 models that we'll run to assess the stability of the coefficient and the predictive capability of that particular model. By running all the models, you can see which ones are the best predictors."

Revolution says of Dowlaty's use of R at Mu Sigma, "The benefit of an 'ensemble' approach is that when new analytic techniques emerge, they can be brought into the mix without causing disruption. This makes the R especially valuable to Dowlaty, since the R software library evolves continually as members of the worldwide R community contribute new packages and programs."

In fact, both rigor and flexibility are key to the paradigm Dowlaty advocates. "The trend is toward a multi-disciplinary approach to extracting value from data," he told The Data Warehousing Institute early this year. "It's not just about math anymore. You also need technology skills, but what ultimately separates the analyst from the scientist is the dimension of artistic creativity. It's the soft skills that make the big difference."

That combination of skills is what enables the firm to tackle the incredibly complex work they do. Dhiraj Rajaram, Mu Sigma's CEO and the man who founded the company in 2004, spoke at the 2010 Predictive Analytics World conference on a panel with Mu Sigma customer Walmart.

Walmart Financial Services, which named Mu Sigma its Supplier of the Year in 2011, works with the big data company to analyze and optimize the marketing of its financial products.

Decision Management analyst James Taylor blogged the following summary of the conference presentation about the collaboration between Walmart Financial Services and Mu Sigma. This sounds like very complicated work.

"WFS uses transaction life analysis around run rate and growth, price / mix analysis, financial returns and qualitative analysis of the creative. Marketing Mix modeling, optimization, lets them see the effect of individual campaigns (there's a lot of Walmart stuff going on in the market), account for seasonality and manage at the store level. The idea is to make sure the marketing investment is optimized, targeted and repeatable.

"Marketing mix optimization uses weekly sales, store traits and demographics, event information and macro-economic data to see how effective specific events were and what was the contribution to the overall effect. What was the value or contribution of each element, did they cannibalize each other, did they resonate in specific areas etc."

That sounds like a potent combination of math, science and creative thinking. It's probably more a picture of the sector than of one company alone. Forrester analyst James Kobielus specializes in big data and says he's done one briefing with Mu Sigma but didn't detect any particular unique flavor to the firm's work relative to others in the sector. Mu Sigma hasn't yet responded to our request for comment on this article.

Perhaps this company is typical of the sector and the questions to ask about it are more general.

"My caveat with services like MuSigma is that they can analyze your data, but they can't change your business," says Cloudera's Wills.

"You are free to ignore what they tell you, and it is often the case that the answers they can give you are limited by your business practices and the data that you currently collect. The advantage of having an in-house data scientist, especially one with some programming skill, is that they can develop systems that collect better data so that they can come up with better answers.

Interestingly, while student email is often outsourced, faculty and staff email is generally hosted in-house because of concerns over confidentiality. Only 2.3% of all schools outsource these systems to commercial vendors.

Schools, for the most part, aren't able to keep up with the speed of innovation on the web anyway, and the fact that many college-run email systems have fallen far behind the innovation curve has driven a lot of students to just forward their school email to a commercial account anyway.

Given the cost pressures that schools are under right now, the choice for these colleges is to either spend a lot of money on providing costly email systems that most students hardly ever look at, or to outsource them to a commercial vendor, or even to Google, which will happily offer these services for free.

Next Step: Get Rid of It

The logical next step, then, is to simply stop providing .edu email addresses to students - and a number of schools are actually considering this move. Last month, at The Chronicle of Higher Education's Technology Forum, Steven Zink of the University of Nevada in Reno announced that his campus plans to stop providing students with a college email system altogether.

Most colleges will probably continue to provide students with an official .edu email address, but this will just be used for forwarding mail to another account - something most students prefer over using their college email systems anyway.

In many ways, this makes a lot of sense. Schools won't give up email as their preferred way of communicating with students anytime soon, but the days when colleges provided the most important on-ramp to email and the Internet for their students are long over.

In their shutdown announcement, Google suggested Lively users capture their work by taking videos and screenshots, and thanked their users adding: "We've learned a lot about how users interact in rich social environments." Is this all Lively was about? An experiment in user behavior?

Image: Digiteen Action Project

Teacher Vicki Davis, in a blog post on the Dream Team site said that the class had contacted Livelyzens (other Lively users) and found that there are classrooms around the world using the tool. "On a Skype call between my class and some Livelyzen's yesterday, we learned that one Livelyzen has built a translator for multiple languages to allow avatars to communicate and speak in their native language! So cool!"

The American Education System Needs Your Help

In an attempt to have their voices heard, the Digiteen Dream Team created a blog where they have been listing their goals, along with suggestions on how Google could turn Lively around. You have to commend them on their efforts.

The student led protest is planned for this Wednesday 2.15 - 3.00 p.m. (EST). These are the ways you can help:

1. Create an account on Lively
2. Create a room and host a protest. Let the Dream Team know, and they'll post about it
3. Visit the protest room on Wednesday and show your support
4. Sign the Lively petition
5. Write a letter to Google about the use of Lively in education
6. Pass the word on; promote the protest

With students around the world counting on virtual worlds, the economy in the sorry state it is in, and schools across the United States working on minimal funding, we need to find a way to provide a safe online environment for students to work in.

"My students have a dream to create 3D worlds for teaching digital citizenship - they are going to pursue this dream and I'm going to help them. We will not stop - but if we have to start over we want it to be the right place that is accessible to as many students as possible," Davis said recently on her post.

With the holiday season fast approaching, let's hope Santa has a few goodies in his bag - or at the very least, a trick up his sleeve.

Recessions change consumer behavior, drive weak businesses and models to the wall and enable new businesses and models to thrive. This recession is likely to have a greater impact on Web ventures than past recessions for two reasons:

This means that the stakes are high for a lot of people. During the Dot Com era, a few visionaries were surrounded by financial opportunists who convinced public market investors to become VCs. After the crash, engineers and real entrepreneurs reclaimed the Net and celebrated that as Web 2.0. That sense of being an elite community, where everybody knows anybody who matters and knows what they are doing, is about to be obliterated by the Main Street Web.

There are two consolations. First, we can finally let go and stop trying to follow every feed from everybody. Second, the Main Street Web will have sustainable business models that allow millions of people to make a good living and a few people to get rich beyond the dreams of avarice.

The Dot Com era handed the power to individual consumers to research prices. That was a massive and irrevocable shift in the balance of power between buyer and seller. The Web 2.0 era, social media, extends that power. We can now collectively research prices and, even more important, we can take action collectively. In a recession, when everybody gets real about saving money and making additional income, we will use this power. Social media taught everybody that we had the power - we could write as well as read, create as well as consume, take action to change the world as well as observe changes in the world.

In the new web era, we will use that power to make a living

That is why I call this new era the Main Street Web. This is a nod to Geoffrey Moore’s Crossing The Chasm, the point in the adoption cycle when technology goes mainstream. The Main Street Web is about people who don’t care about technology or media, they just use it. Above all it is about really simple business models that work in the physical world as well as online world. The Main Street Web will empower small business and level the playing field with big business.

The Main Street Web is all about revenue, fixing the one big weakness of Web 2.0

This is not just about revenue for the founders/investors who own the site, but for the millions of ordinary people who use it. This will be the “show me the money era“. The way for Main Street Web ventures to make money is to help other people to make money. This is when the digital sharecroppers revolt and new entrepreneurs will come forward to give them a fair shake. In a recession everybody needs revenue. Small businesses need revenue. Digital free agents need revenue. The only people who don’t need revenue are employees with a solid paycheck from a big company, and as layoffs hit the news they will also be doing a bit of moonlighting and looking at options.

Boring technology, exciting business

Clay Shirky vividly describes, in Here Comes Everybody, how technology gets socially interesting at the same time as it gets technically boring. The basic tools of Web 2.0, such as forums, blogs, Skype, SMS, RSS and social networking have now passed the early adopter phase and crossed into the mainstream. Ordinary people are using social media to change their world.

Shirky focuses on social change and does it brilliantly. He does not focus as much on the lower levels of Maslov’s hierarchy of needs. Specifically, level 2, safety. In a recession, people focus on getting more income and reducing costs.

People cannot be duped into selling products for free

Web 2.0 was a non-commercial wave. A few people made $ millions selling their social media start-ups to Google, Yahoo, Microsoft, eBay and Amazon. But everybody else was told “here is a great new way to have fun and it is all free”. The few attempts to monetize were weak and embarrassing, because entrepreneurs and investors, desperate to justify nose-bleed valuations, seized on the only source of large amounts of cash, the good old-fashioned “massconomy” Fortune 500.

We went, in a heart-beat, from “this stuff will change the world” to “yippee, this is new way to sell mass produced products to consumers online”. That is the Dot Com era on steroids vision that made Facebook’s Beacon a massive embarrassment. “Coca Cola wants to be your friend” is not the next phase of the web!

If digital creations are free, how do digital free agents make money?

This era also taught us that all digital products (code, data, writing, video, photos, music and so on) tend towards “free to air”, because the Internet is a giant free copy machine. There is still room to sell subscriptions to large companies as they will use these services to replace the layers of legacy products and the massive maintenance and management teams dedicated to serving them. There are still huge enterprise budgets up for grabs in legacy replacement by SAAS (Enterprise 2.0, if you like). But for consumers and small businesses, the monetization of digital products will be through advertising or e-commerce.

There is plenty of advertising $$$ that wants to go online. The gap between 20% of time spent online versus 5% of ad $$$ spent online is still the big wave driving all this. The problem is not availability of budget, it is effective ways to spend it.

Most of that advertising will not be for more free digital products - that would create a shell game. (Yes, in the final days of Web 2.0 there is a lot of that shell game going on). So eventually we have to have some physical products that users crave/need/want.

“Green e-commerce” and rising gas prices

This is where it gets interesting and a bit more complex. Digital products are simple. The whole process of creating craving and fulfilling craving takes place online. Physical products have to travel in the real world, using trucks and planes and they need to be handled by lots of people. With rising gas prices, trucker strikes and increased awareness of eco damage from carbon emissions, the e-commerce 1.0 model looks seriously flawed. The Internet is brilliant at surfacing the long tail. You can find that wonderful extra virgin olive oil from a small producer in Italy. But ordering a single bottle, picked and packed just for you by the same people who grow the olives, sent on a transatlantic jet and finally delivered 24 hours later just in time for your dinner party? Some people can afford to do that, but it is no longer hip. The rest of us cannot afford to.

E-commerce 1.0 is the unprofitable part of Amazon (all those expensive warehouses with inventory). It failed dramatically at Webvan. It was always a problematic model. High gas prices and eco awareness just dealt the coup de grace. A new “green e-commerce” is needed. Orders need to be aggregated and delivered through a local hub and spoke supply chain.

Mom & Pop get hip

Luckily we already have a local hub and spoke supply chain. They are called independent retailers, the ones sprouting up again in the Main Street cracks left by the big box chains. Those big box retail chains, with their standardized supply chain processes, create long delays between spotting a trend and meeting it and limit their ability to to customize and localize. The independent shops - the once derided Mom & Pops - have the agility and direct, real customer relationships to prosper with a bit of help from new online infrastructure.

Independent retailers thrive by aggregating long tail demand at the local level

This has to be about long tail products, what ordinary people call “exotic” or “special” or “unique”. This can mean hand-made, which is what Etsy is doing. It can mean local food from sustainable agriculture farms. Or custom motorbikes, fancy cashmere sweaters and other cool stuff that mass affluence has taught us to want and expect. This is about the emerging global micro-brands.

Independent retailers make a living by offering products that you cannot find in the big box chains. Most have adapted to the waves of Wall Mart, CVS and other bland megastores by selling what the big guys don’t have. Sure they have to offer some basic staples as well, but consumers go to the independents because of the fun of finding something different and special. And having a human connection with somebody who knows your name or at least your face. You cannot - thank goodness - replicate that, automate it, or turn it into a platform.

It is a marriage made in heaven. The Internet is very good at long tail. Independent retailers need long tail products to differentiate.

The Internet is clearly good at surfacing long tail products. With a little extra technology, the Internet could connect some dots to deliver tremendous value (read “extra income”) to the 3 key players in this emerging Main Street Web - the independent retailers, the digital product creators and the global micro brands.

The elusive revenue model for social networking?

This is where social networking sites may find an enduring revenue model. Possibly. They have already found what does not work - helping friends to sell massconomy products to their friends in a glorified online Amway scheme. What would be much more in keeping with the peer ethos would be to enable friends to get together to buy cooperatively. General Motors really, really wants to spend more ad $$$ online because the young car buyer is getting so hard to reach in other media. Say 1,000 Facebook users, all of whom wanted a new car, agreed on a specific model so they could bulk order and get a discount? GM would be happy, after getting over the enraged calls from the dealers. They would find ways to keep the dealers happy, the buyers have to pick the cars up somewhere local.

That sounds good, but the problem for social networking sites is that this is not about friends. If I want that discount on a new car, I don’t really care who my fellow buyers are. They don’t need to be friends. In fact it would be pure coincidence that my needs happened to coincide with the needs of my friends.

So this is about content more than about friends

But the content has to be structured to enable this to happen. Which sounds like a reasonable mission for all that cool Semantic Web stuff brewing in start-ups. This is where that implicit web and social browsing technology seems to be headed.

Get local (even “hyperlocal”)

It also has to be local. Remember, it has to get delivered in a way that makes sense for carbon/gas cost.

What on earth does “hyperlocal” mean? Is this “hype about local”. Or “what techies call local to make it sound cool?” “Local” is good enough.

Local, as a tech/media opportunity, has tended to mean local advertising. Remember Microsoft Sidewalk? over $100m spent to take on the $100 billion local advertising market. This was one of many attempts. The problem is that Craigslist took the air out of that market. Traditionalists continue to use Yellow Pages and local newspapers, but they are fighting a losing battle, while Craigslist moves rapidly into the mainstream and Google has a shot at a big chunk.

Everybody else will have to work a bit harder to deliver value to local businesses. Selling ads alone won’t cut it. Small businesses are used to a simple world of orders and margins. How many of these will you buy at what price? This has to be e-commerce and not just ads. But it has to be green e-commerce, aggregating long tail demand at the local level.

Now that enough people are online, the chance of finding other people locally who are interested in that product becomes a bit more reasonable. It needs some smart software to surface demand at that level of specificity - for that long tail product in that area and wanting it now - but given the scale of the prize, the technology will come.

Enter the digital content creators

This is where the digital content creators come in. They find the cool stuff, write about it, take pictures, interview the workers who make it, review it, create videos to explain it. They can do this on behalf of the global micro brands, being paid to create marketing materials directly for Giovanni’s Olive Oil. This will happen through free agent webs and exchanges. Or they can do this as independent micro-media, writing/videoing about the subject they are passionate about in a commercially independent way (an expert on Olive Oil who may write a negative review of Giovanni’s Olive Oil)

Adding some structure to the content provided by the digital content creators is where semantic web technologies can help. What has been described as “SEO 2.0” means that the long tail media expert tags the content semantically, so that Giovanni and his competitors can more easily find which long tail sites to advertise on. Which means that the provider of the toll booth - Google primarily - may not be able to charge quite so much. Yahoo seems to be focused on this opportunity.

Get real about ROI and monetization - its called revenue and orders

The next step is to make the monetization much more direct. CPM is a hold-over from traditional media. Re-reading the very entertaining Burn Rate, it was weird to see the Web cognoscenti in 1995 saying that CPM was not going to last, knowing that in 2008 it is very much alive and well. CPC is better, but most small businesses don’t have the engine to monetize clicks. Cost Per Action sounds better, but what Giovanni really needs is Cost Per Revenue. Or to put it more simply, orders.

More importantly he needs orders in a reasonable volume for delivery to a specific retailer.

Which means that the olive oil expert with his rich media blog has to also have an API that syndicates that data to all the places on the web where people who might be interested in olive oil hang out and also ensures that he makes some money when orders get placed.

That is one scenario for how this could play out. The goal of aggregating long tail demand at the local level is not simple to achieve. So this scenario may not be what happens. But I am convinced that emerging phase of the Web, what I call the Main Street Web, will be about:

1. Simple revenue models, orders not clicks, suitable for small businesses and individuals.
2. Leveraging and empowering independent retailers to compete with big box stores.
3. Enabling global micro brands to grow and take bigger share from the massconomy.
4. Reducing costs by cutting out physical intermediaries such as distributors.
5. Reduced cost for seller, lower prices for buyer and more income for digital content creators from new technology that reduces the toll booth fee from the platform operator (known today as Google).

Small business - the neglected middle child

Small business has been the neglected middle child - neither big companies with big budgets nor billions of individual consumers. Entrepreneurs don’t know whether to treat small business as a scaled down enterprise or a scaled up consumer. That will change. Small business is on a major upswing.

Before the Internet and during the Dot Com era, entrepreneurs spent a lot of time courting big companies, as that was the quickest way to get a return on R&D. That style of enterprise selling went out of fashion in the Web 2.0 era for good reason. It is very hard to build sustainable advantage selling to large companies. This is the lesson learned by anybody who sells to Wallmart. You become a commodity vendor and the big company is always looking to replace you with a cheaper alternative and you have very little leverage when that time comes. Better to have 10,000 small clients than 10 big ones.

So we moved to the opposite extreme and sold to consumers. In a consumer recession thats a problem.

Small business is due for a comeback, powered by the Internet

In 1955, Fortune 500 companies accounted for around 1/3 of GDP in America. By 2000, that share had risen to 2/3. To put it another way, the share of the economy controlled by large companies doubled from 33% to 66%.

Imagine a world where the Fortune 500 share of GDP went back to 1/3 and small businesses got back the 1/3 they lost in the last 50 years. That maybe about to happen for 4 reasons:

1. Coase’s theorem and the Internet. Coase’s law could become as famous in tech/media as Moore’s law. The main point is simple. Large firms exist because transaction costs make it cheaper to organize within the firm. The Internet obliterates transaction costs, fundamentally threatening the single rationale of economies of scale.The glue holding together big companies was the fact that external transactions cost too much. The Internet changed that.
2. Mass Affluence and the hunt for “special”. Big companies are great at mass production. Small companies are better at meeting the wealthy demand for niche, special, exotic and customized.
3. The decline of Fortune 500 as employer of choice. There used to be a trade off. You could the take security of a big employer and face a bit of boredom or choose the wild, scary ride of a start-up, the risks of small business or the uncertainties of the free agent life. Waves of reengineering, downsizing and other restructuring translated to workers as a simple message; “don’t rely on us for your paycheck any more”. Profit follows talent.
4. The availability of capital. Yes we are in a bit of a funding squeeze right now but that is cyclical. The longer term trend is much deeper pools of capital, packaged through many intermediaries hunting for the spark of innovation

Dancing with gorillas

Looked at from this context, the online platforms that have the toll booth are critical to the health of the economy and the ability of millions of people to make a reasonable living. That is not only a phenomenal opportunity for the big platforms (the “gorillas”) but also an awesome responsibility. Healthy competition is critical. We cannot afford for one company to have the same power that Microsoft had in the PC era.

The final post in this series, “Dancing with Gorillas”, looks at opportunities for entrepreneurs in the emerging Main Street Web in a world dominated by a few big companies such as Google, Microsoft, Yahoo, eBay and Amazon.

Image credit: bettlebrox

Part 3: Dancing With Gorillas: The New Web Era