Lessons from the Ant Colony: Overcoming the Biases of Web 2.0
Operating as a collective, an ant colony can achieve remarkable things, complete tasks, and solve problems that would be unimaginable for a single ant. Colonies are responsible for building elaborate nests, waging battles, and creating efficient highway systems to food sources. The collective intelligence of an ant colony can serve as inspiration to help us solve complex human problems. Businesses in particular are finding innovative ways to apply these lessons from nature, from routing trucks to managing plane congestion on the tarmac... to making Internet search more accurate.
The theory of swarm intelligence (or collective intelligence) relates to how the simple actions of individuals can come together to produce the sophisticated behavior of the collective. Deborah Gordon, a biologist at Stanford University who has spent decades studying harvester ants in the Arizona desert, summed up the concept this way: "Ants aren't smart. Ant colonies are."
Take foraging as an example. Whenever an ant finds food and carries it back to the nest, that ant leaves a chemical trail (pheromones) along the way. Other ants sniff the chemical trail and follow it toward that same food source. As more ants find the food and carry it back to the nest, the path gets a stronger chemical dose and, in turn, becomes more attractive to fellow foragers. Individually, these ants are following a simple set of rules and acting on local information: follow the pheromone clues and bring food back to the nest. However, the colony as a collective is behaving in quite a complex way: creating a sophisticated highway system that leads to the best food sources.
Collective Intelligence and the Web
So, what do ants and chemical trails have to do with the web? For starters, lessons from colony behavior can be applied to enhance the way we search for information, products, and solutions.
The Internet puts an unprecedented range of goods and information right at our fingertips. And while we now have the ability to find what we want, when we want, many are finding that more isn't necessarily better. As Barry Schwartz explains in "The Paradox of Choice: Why More is Less," too many options actually cause more psychological distress than good.
Think about it. How many times have you abandoned a search after hitting page 7 (or maybe 2) because you couldn't find what you were looking for, and then ended up doing multiple searches with different search terms? The sheer enormity of results often makes searches exhaustive, tedious, and overwhelming; we're forced to wade through pages of results before finding the product or link we want.
Of course, the answer is not to limit the available products or retrieved results. After all, access to this rich "long tail" of goods has been a key driver behind the success of many online retailers. Instead, websites try to minimize this search exhaustion by predicting what a person really wants and putting these products up front. It's a great idea in theory, but not so easy to implement in the real world.
The Limits of Web 2.0, and the "Squeaky Wheel" Syndrome
With a strong foundation in collaboration, community, and user participation, the Web 2.0 movement seemed to solve this dilemma by factoring in the contributions of users to narrow down choices. Eager community participants can make their voices and opinions known through user reviews, recommendations, ratings, tagging, and more. Websites have tapped into these crowd-sourcing techniques to determine the relevance of search results. While these methods may help tame the tangle of options, they suffer from one major problem: bias.
Traditional crowd-sourcing demands active participation from its members. The problem here is that not everyone contributes. Only certain types of individuals are likely to make an effort, and they are driven by various motives, from a mere hope to be noticed in a crowd to an altruistic desire to help others to a need to rant about a negative experience. In short, only a subset of the population (the squeaky wheels) will participate, significantly limiting the sample pool and possibly skewing the results with personal bias and inaccuracy.
But what if there was a way to sidestep these biases and gather a perfect representation of consumer attitudes by tapping into the opinion of every single person who visited a site or conducted a search?
Back to the ant colony...
The Next Phase of Social Search: the Super-Community
Watching a trail of ants march toward crumbs of food, it's hard to imagine that ants aren't aware of their actions. But according to studies on swarm intelligence, what appears to be intelligent behavior actually results from nothing more than the complex interaction of simple actions.
Likewise, websites can tap into the implicit wisdom of the community to more accurately predict the most popular and relevant results of any given search. There's a wealth of information in the everyday online activity and behavior of website visitors: every successful or failed search, every page visited, every purchase or abandoned cart represents valuable information. These natural behaviors and actions reflect the true and unbiased opinion of the community as a whole.
By listening to these implicit actions, website owners can gain new insight into the preferences of the silent majority; by leveraging the data, they can optimize results for future searchers. Just like ants that leave a chemical trail each time they bring food back to the nest, we leave real-time feedback each time we visit a page or select (or ignore) a result.
With each search, we unknowingly participate in a cooperative design that improves the search experience for all searchers to follow. Simple, self-guided actions -- entering keywords and selecting results -- drive the greater common good. And as more people participate, both the chemical trail and the overall system grow stronger.
This new participatory strategy gives greater power to the super-community, in which the collective intelligence of all site visitors is harnessed to create a better search and shopping experience for everyone. With each search, the community carves out a faster, more efficient pathway to desired information and products, no different than the trail of pheromones leading to food sources. And like the ants, web searchers act as a collective team (whether they know it or not), yet another example of the whole being greater than the sum of its parts.
Scott Brave is a founder and CTO of Baynote, Inc. Prior to Baynote, he was a postdoctoral scholar at Stanford University and served as lab manager for the CHIMe (Communication between Humans and Interactive Media) Lab. Scott is an inventor of six patents and co-author of over 25 publications in the areas of human-computer interaction and artificial intelligence. Scott is also an Editor of the "International Journal of Human-Computer Studies" (Amsterdam: Elsevier) and co-author of "Wired for speech: How voice activates and advances the human-computer relationship" (Cambridge, MA: MIT Press). Scott received his Ph.D. in Human-Computer Interaction, and B.S. in Computer Systems Engineering from Stanford University, and his Master's from the MIT Media Lab.
(Photo by Il conte di Luna.)
Facebook
Comments
Subscribe to comments for this post OR Subscribe to comments for all ReadWriteWeb posts
Thank You..
Intriguing insights on Supercommunities. What can we expect as "superempowered individuals" connect to each other in web-enabled Supercommunities? John Robb has written about the growing risks (see http://tinyurl.com/superempowerment) that centralized societies face as a result of technological trends leading to "super-empowerment" of malevolent individuals and small groups. Do you see paths for Supercommunities of super-empowered individuals to evolve, who can help create resilient communities and relationships based on soft rather than hard power?
Mark Frazier
Openworld
@openworld (twitter)
"And like the ants, web searchers act as a collective team (whether they know it or not), yet another example of the whole being greater than the sum of its parts."
Very thoughtful point here.
The ant metaphore can bring the web to something very powerful, since, as you say, ant's behaviour is not pure intelligence, but "complex interaction of simple actions". And, hey, great, "websites" too aren't intelligent. So, if we make each site(or websearcher) participate in one simple action to the complex goal of providing the good result for any given search, then we'll use the "super-community" power to its full potential.
Sounds great in theory, but, what do we do next?
I think that what we're talking about is something quite "enthousiasming"(?). Let's have a dream:
I search for "term1 term2" on my favorite search engine.
A couple of links clicked later, I land on a page that fits exactly with what I wanted.
Here, I can click on a small button somewhere on my browser saying "I found what I was looking for when searching for term1 and term2". Isn't that sort of a "chemical track" towards something that may wind up being useful for some other NetAnt looking for (more or less) the same thing I did?
Spam, ad biases, jerks, ...
... Hmmpf...
Ok, we need a plan...
But, I'm in!
Let's build this Super NetAnt Community.
How can I help?
great post.... thanks...
I want to make ma website a community type website...
it's a biotech pharma related blogging website..
I want to make ma website a community type website...
it's a biotech pharma related blogging website..
http://biotechbytes.com
Please help
nice info.......
We heartily inviting to my blog http://funevil.blogspot.com/ Ready to Accept Pain of FUN
Using the ant colony as a metaphor for an online community is a tour de force; it gives me new insight when I compare the actions of the colony to social networking activity. As an administrator of a community, I can track the pathways of activity and build up the popular routes (activity choices) in increments. In terms of social media, most-used and shared community content can be reinforced, and a better community will emerge over time. Thank you for using the concept of a colony to help me think of social networking administration in a different way.
This looks like an enlargement of the Google algorithm.
Google revolutionised search by being the first to use semantic information (the choices webmasters made when linking to a site) to improve search relevance.
The ant trail metaphor (and it's an excellent metaphor) enlarges the source pool of semantic information to all users.
Google sends out bots to gather its information, and this is where something new is needed - how does ant trail data flow to where it will help the next ant? Site managers use clickstream analysis to improve their own sites, but how could this data benefit the wider community?
Could all sites automatically encode ants' movements in machine-readable format to help subsequent users? I wasn't going to talk about semantic web here, just semantic information, but wouldn't that be the way forward?
What about the privacy implications of this (cf how some governments here in the EU forbid the use of Google Analytics on publicly run sites).
Ask a lot of questions, don't I? ;-)
Hi!
What an interesting post! I'm spreading the word online about the book Sustainability 2.0, where Ernesto van Peborgh and the Odiseo consultancy firm discuss topics of Web 2.0, sustainability and collective intelligence.
Actually, we have a whole chapter devoted to an issue so relevant that is probably not only the present but the future of media and sustainable practices.
I'll re-post this on our blog later.
Thanks!
I've felt this way since Bill Gates wrote about the digital nervous system. The internet is like a digital pheromone - it's a communicator of alphaero ideas that will evolve over time - it is a living organism.
Nice article but its bordering on plagarism
http://www.opendemocracy.net/arts/white_ant_4359.jsp
Scott, this is my first comment at RRW. I don't know you, but I like the article. The principles you shared are worth revisiting and your insights made it an enjoyable "read." Looking forward to more like this. All the best, Brian
Thanks for all of the great comments! Let me respond to a few of them...
Zackatoustra, actually what you're suggesting is one of our key technologies at Baynote (www.baynote.com) called Social Search, part of our "Collective Intelligence Platform". It works very similar to how you described except we don't require explicit feedback from users to indicate usefulness. We instead infer usefulness from users’ natural patterns of behavior (e.g. when viewing a page).
matthew, good point on privacy. We can't do much if we can't collect any usage data. But, one nice thing about the collective intelligence approach is that no one user is singled out. In fact, there's no need to store any data at the individual level, personal or otherwise.
Eric, thanks for the pointer to "The Soul of the White Ant." I'll have to check that one out. There's quite a lot written on ant colonies and collective intelligence. My favorite book on the topic is "Emergence" by Steven Johnson. Also check out early work from EO Wilson and Deborah Gordon.
Enterprise search vendor has talked about using "intelligent ant" technology for about three years.
See: http://www.autonomy.com/content/News/Releases/2006/1107.en.html
Indeed, the ant-colony metaphor for the collective intelligence that fuels social search goes back many years, and has been blogged about well before Autonomy's PR - for example, see:
http://blogs.zdnet.com/BTL/?p=3181
Interesting read!
I have read a lot of very interesting articles on Ants lately. I find it amazing how Ants can release Pheromones for their tracks, it is crazy. I suppose we probably all do it but dogs and skunks do it naturally.
Very interesting article, it's amazing how much we have to learn from nature. From improved swimming techniques with dolphins, to community efforts with the ants, nature has always been a step ahead of us. We're always hungry to learn more, we do extensive lab researches, which is a good thing, but sometime it might be also good to stop, and just look at what surrounds us. Very good article, all this is great food for though!