Poll: Are Pilots a Waste of Time?

Written by Steven Walling / August 31, 2009 10:53 AM / 13 Comments

Late last week Socialtext's Michael Idinopulos wrote a post with some interesting advice for anyone looking to start a social software implementation in the enterprise: skip the pilot. His argument was that since the new breed of enterprise 2.0 tools are about human interaction, something which changes dramatically at scale, then small pilots were not a useful barometer of future success or failure.

Idinopulos admitted that pilots are great for traditional IT, which revolves around a set of actions that do not change much whether it's 10 or 10,000 people (think billing or adding leads in a CRM). We agree that in any kind of collaboration, the shift from 10 to 10,0000 causes dramatic change. But that leaves an open question: do you still use pilots for your wikis, blogs, and other social software implementations, or are they a waste of time?

In one sense it's a little strange for Idinopulos to question the need for pilots, since Socialtext's switch to freemium was pitched to us with pilots as one important use case. It makes us wonder whether the "free 50" version has seen less use as a pilot than the company expected.

But whether you're looking at a big social platform like Socialtext, or something much simpler, whether or not you spend time and energy on a pilot is an important issue. We generally feel that the premise Idinopulos presents is correct; interaction among workers in a tiny subset of your organization isn't a fair test.

But where does that leave us? Are pilots a necessary evil to support purchasing decisions, or a waste of time that gives an inaccurate picture of how enterprise 2.0 really plays out? We'd like to hear whether you love or loathe pilots in the enterprise.

Photo by totalAldo

Comments

Subscribe to comments for this post OR Subscribe to comments for all ReadWriteEnterprise posts

Here's the thing about pilots: they work well because they are culturally and functionally useful. One of the most important benefits of running a pilot is that it gives you an opportunity to discover and fix any technical issues before a tool is made available to thousands or tens of thousands of people. Fixing those problems while they're small and you're working with a pilot group that's empathetic makes it much easier to refine the system for a smooth rollout and adoption.

Ann All wrote a great article that explores why her organization's wiki hasn't taken off, and she cited technical problems as a major factor. A pilot would have helped catch and fix those problems early.

Posted by: Stewart Mader | August 31, 2009 12:18 PM

Thanks chiming in Stewart. I know you have a lot of experience with pilots, so I appreciate the comment.

Posted by: stevenwalling.com | August 31, 2009 12:32 PM

Stewart is right in that a pilot can discover small technical issues and allow you to adjust but there's a more direct psychological issue at play.

The vast majority of people don't like change. They don't want to use new tools and have no interest in it because they've been able to do the job before the tool came around and suddenly, here's yet another thing to learn.

Pilots can break this down.

By deploying a tool and training on it within a single group/project/department, you're working with a small group and can work with them to show/convince them of the value. If it's successful and you roll it out further, you have a sample of real live data *and* potentially a few champions who's lives/work were made better. Now instead of you trying to teach everyone, you have other users doing it.

Evangelism... virality... word of mouth marketing... this isn't a new subject, just applied a bit differently.

Posted by: D. Keith Casey, Jr. | August 31, 2009 1:42 PM

Far mor important for enterprise adoption of social tools is identifying those few people with the inherent skills of facilitation who can engage others without being control freaks and letting them loose on the organization.

Posted by: Kris Olsen | September 1, 2009 5:11 AM

Hi Steven. In the Council, we're finding very specifically targeted pilots help seed content and inspire participation. It's more of an iterative process with a lot of feedback loops built in. For some of these large deployments, it's essential to get buy-in with a carefully phased approach.

@Marshall- hope that html worked! :-)

Posted by: Susan Scrupski | September 1, 2009 3:55 PM

My organization is running a "pilot" of some social media tools; they're being wonderfully well received, to extent that the signup link has been shared across continents !! In fact that's how I got onto them myself, and I've become a bit of a local evangelist.

At the moment the tools are being hosted via the vendor, and there's not been too much talk about how they handled scaling issues. It's obvious from conversations with the implementation team that they were surprised, even overwhelmed a bit, by the uptake.

However, the suite is still perceived in some quarters (including some levels of management) as a pilot. This has lead to the situation where the ONLY thing I hear people say when asked why they aren't engaged with the tools is along the lines "It's a pilot and I'm not going to invest my effort in a throwaway system".

In short, we may have PLANNED a pilot, but we ended up with a full implementation.

Posted by: martin english | September 1, 2009 7:23 PM

This discussion reminds me of a true personal story from my McKinsey youth. As a young consultant, I once prepared a document recommending that a client pass over investment A in favor of investment B.

"Why are you recommending A over B?" asked the partner at about 1 a.m. over a plate of stale takeout sushi.

"Because the client will never go for A. It'll never fly politically" I answered truthfully.

"But what's the right answer from a business standpoint?" he replied.

I thought for a while and finally admitted the truth: "Investment B."

"So go rewrite the deck. Recommend Investment B. If that's politically impossible, the client can always fall back on A. Always start with the right answer. Otherwise you're just telling people what they want to hear."

End of story, beginning of rant.

The common thread through many of these very thoughtful comments is that pilots are politically necessary, that you need them to generate buy-in and organizational support.

I fully agree that a successful pilot is politically expedient. You'll get no argument from me there. But that's a point about expedience. From a *political* standpoint, pilots are useful.

My point--and I still stand by it--is that from a *business* standpoint pilots are sub-optimal. Small-scale pilots do not predict social software adoption or business value patterns at full-scale deployment. (I won't revisit my reasons; I've already laid them out in a couple of blog posts here http://michaeli.typepad.com).

So if you're pinning your Enterprise 2.0 hopes on a small-scale pilot, you're making the wrong *business* decision in order to make an unobjectionable *political* decision.

Look, I'm a pragmatist. I've been a McKinsey consultant, a line executive, and a nonprofit board member. I know my way around the shark tank, and I know that you have to compromise to get things done. So I'm not saying that you should never do the politically expedient thing, or that you should never run small pilots.

What I am saying is precisely what the McKinsey partner told me back in the day: Start with the right answer and *then* massage the politics. Shoot for full-scale deployment. You can always fall back on small-scale pilots if you need to, but that shouldn't be your ingoing position.

If you stick to your guns you may get a lot further than you think. Heck, you might even transform your company.

Posted by: Michael Idinopulos | September 4, 2009 2:09 PM

Michael, thanks for the response. It wasn't too ranty. :)

I hear you on the points about politics and culture. But what about Stewart's suggestion that pilots let you "discover and fix any technical issues before a tool is made available to thousands or tens of thousands of people"?

That sounds pretty pragmatic to me. Technical hurdles are way less challenging if you're talking about a deployment for 10 people instead of 10,000.

Posted by: stevenwalling.com | September 4, 2009 2:39 PM

That's the part about pilots that's inarguable in my experience. Politics can go one way or the other, but technical problems can derail even the best-laid plans.

I can agree that a small pilot might not be representative of large-scale adoption, but that's simply a question of how you structure the pilot participants. If you only involve hard-core early adopters, then of course it won't be representative. However, if you construct a pilot that involves people from several key areas of the organization (HR, product/project groups, technical writers, customer support/service, etc.) and those people understand going into the pilot that they're getting a chance to set up their uses before everyone else in exchange for their help and patience in finding and fixing technical problems, you'll get a group that will solidly support the project once it's opened up for large-scale use.

During the pilot phase, if you find & fix a problem with the assistance of that compact group of helpful people, you'll be in much better shape than if you have 10,000 angry people beating down your door while you try to fix some problem that has brought down the whole system.

This is tangible stuff, and it directly affects the success or failure of the project. I see it all the time.

Posted by: Stewart Mader | September 4, 2009 3:12 PM

I agree with Stewart. As a program lead for our internal E2.0 pilot I can say with confidently pilots are important culturally and functionally.

Pilots allow you test the waters. For us, we had tremendous experience with collaboration tools in the past. We knew the short comings of these tools. Our pilot had several critical success factors to prove out (security, features, adoption, business terms, would it scale, could it perform, and so on). So that was all the functional part.

But, and more importantly, when we're talking about E2.0 tools - or social collaboration - there's some 'cultural' sell as well. Pilots represent a 'smaller' chunk of work or investment. They are easier to sell to management. And if they don't work, you adjust.

If they do work, Pilot results are HUGELY important leverage for the next stage - the full business case. What better way to make a full business case by having experience, performance metrics, success stories and user anecdotes to back up your assertions?

One thing I think we did well with our pilot was that we didn't limit the pilot scope. We tested the tool in full scale. How would it be adopted? Would it take off virally? Would it meet our user's needs? And how would we know if it would stand up to real use if we didn't try it in real scale?

We piloted our communication plans and our advocate plan. We were grateful when it went viral because we knew we could really look at scaling and performance and all the technical criteria you hope will work. Martin (who commented above) was one of key advocates that came on board. His energy was representative of the energy we got across the globe. Folks our core team had never met 'got it' and brought others on board. Yup - folks like Martin would not have had the opportunity to help us make the business case for a full term deployment if we didn't run a pilot.

Hi Martin - I'm glad we met in C3 :)

Posted by: Claire Flanagan | September 4, 2009 4:19 PM

Stewart, I'm with you on the technical stuff. At McKinsey we referred to that as a "technical pilot" to distinguish it from an adoption/value pilot. The axe I'm grinding is about adoption and business value.

Posted by: Michael Idinopulos | September 4, 2009 8:52 PM

Oh, I think it’s fairly harmless really… although one of my colleagues did say he felt it was like 20 years of women’s lib had never happened!

I do think it’s a shame there’s just the one hunky pilot though (you know how I feel about the fact there are always two on board…) and one extra source of testosterone in the ad would have been lovely!!

It’s a great tune and a funny ad. I think Virgin is generally tongue in cheek about itself, and I like that. And at the moment, if it encourages more people to fly, well, that can only be a good thing!

Posted by: thé yerba mate | December 22, 2009 9:16 PM