In September 2006 the News Feed was a radical idea; thousands of Facebook users revolted against the idea that all their friends would be shown every photo they uploaded, when their relationship status changed and other information as soon as it was available. Today we live in a different world. Almost everything is social and the new challenge is tackling information overload. That's what Facebook just did today and it's going to be very important for the future.

Everyone's trying to solve this problem. There are inbox filtering services like ReMail, Threadsy and the experimental new Mozilla Raindrop. There are column filters in stream readers like Tweetdeck and Seesmic. Google Reader yesterday introduced a "magic" filter view for the most popular items across the whole network. FriendFeed, a small but innovative social aggregator started by one of the creators of GMail and acquired by Facebook for $50 million this summer, offers a "best of day" view of any stream of updates you're looking at.

That FriendFeed view is the closest thing to the new Facebook News Feed, but a Facebook spokesperson told us that the two products are unrelated.

Everyone's trying to tackle information overload. Step one, get more people sharing information. Step two, figure out how to create a personalized, high-value view of all that information by surfacing the most important updates for each user. Step three, profit!

How It Works

The new News Feed view is based on an algorithm that scores every update coming in through what's now called the Live Feed. That scoring is based on the number of "likes" and comments an item has received and how much you personally have interacted with the update's author in the past.

A related algorithm was used in the past to create the "highlights" section on the right-hand side of the Facebook home page. That section was getting too little interaction and didn't include things like important status updates, the company says. If your sister posted a status update saying that she's pregnant, a Facebook spokesperson told us today, that wouldn't show up in the old highlights view. It should show up in your News Feed now.

So three big changes: 1. The new Live Feed is linked-to at the top of the page and shows a number of new items since your last visit. 2. Highlights plus hot status updates are now the default, the new News Feed. 3. Birthdays and other important events have taken the place of the old Highlights section; they are of particular interest to users and will now be easier to see.

What It Means

Facebook says that after viewing your new News Feed, you can go check out the raw Live Stream of all the most recent updates from your contacts. That's the opposite of the way FriendFeed did it and neither strategy should be taken for granted. Decisions like this impact a major method of communication for hundreds of millions of people around the world.

By showing the News Feed highlights as the default view, Facebook will probably encourage users to pay more attention to, interact with more and grow closer to the people they already have a history of interacting with and the events that are already popular. Weak social connections and your personal long-tail of content are less prioritized in this view.

The inclusion of a user's past behavior as a criteria for hotness is key, though. It's not just a popularity contest. Your News Feed is your little universe and popularity is defined in relative terms.

That, again, is a particular strategy. The new Google Reader Popular View, for example, appears to evaluate popularity across all users in total.

What It Could Mean In the Future

Someday social networking is going to be like the telephone. Today you can't send messages from Facebook to people on MySpace or LinkedIn but that isn't going to last forever. Just as you can call someone who uses T-Mobile from your Sprint phone, someday sharing and messaging between online social networks will be a given.

How will social networks retain users then? Why stick with Facebook when some smaller service offers a decentralized social networking service outside of Facebook's control but still tied into your friends on Facebook and elsewhere?

These services will someday have to compete on user experience, when they no longer have your social connections locked-in. The service that does the best job filtering up the most important information you have coming your way will likely be the service you stick with. That's going to be a key area of competition between social networks.

How well will Facebook do at filtering the Live Stream of content? We're about to find out and it's going to make a big difference in how we experience the web. That will only be more true as more and more people begin publishing content.

There's been a lot of emphasis on the live stream of real-time web content, but Facebook now joins many other services in recognizing that the best value is sometimes built by combining real time and slower assets.

The lesson here is that while we're meant to "embrace the chaos" of audience feedback, it's best to look at the forest (or overall reactions) rather than the individual (and sometimes spiteful) trees.

While competitor Infegy displays relevant web chatter, UberVU goes one step further by offering developers a chance to create their own mash up and filtering tools. The ContextVoice API's new search functionality allows developers to measure public reactions within specified time frames. Explains the company, "A conversation with 10 reactions in the last minute may be hotter than one with 1500 reactions distributed over a month. But the hot conversation matters more, because that's the one that has the attention and momentum."

Suggested mashups include social media dashboards to measure outreach, memetrackers to get the lowdown on entire industries, community dialogue tools to pull comments back onto your site and comment tracking for stock trading purposes. To check out the search API visit the ContextVoice developer page.

Now, Google is working to fix that with new filtering options.

Still, I have to admit that I'm with Tam Vo over at VentureBeat. Even with these new options, the filter that would provide the most value is still conspicuously absent. I remain hopeful that Google Image Search takes a page from Flickr's book and adds the ability to search by Creative Commons licensing.

Until filtering by license information becomes available, Image Search remains a valuable tool for finding images - and with the new filters you have a much better chance of finding what you're seeking - but they're not images you're likely able to use, legally.

Given the current economy, there may be fewer applications and services to try next year. Whatever will we early adopters do? We love to flit from service to service, trying the latest shiny new thing, endlessly discussing whether or not it will stick, whether it will "cross the chasm." Without the endless barrage of new services being released one after another, in 2009 we may find ourselves having to more deeply embrace the ones we have left. More importantly, we'll finally have the time to figure out how we can really integrate them (or not) into our daily lives.

As we discover how to better manage the social media apps we added to our daily workflow during 2008, we may end up turning a more critical eye towards any newcomers in 2009. Enriched with a better understanding that doesn't come just from being enamored of "shininess," but from experiences that grew over time, we may question the new arrivals in ways we never did before. What value does this bring me?, we'll ask. Is this really doing anything new?

Thankfully, the answer to that last one will likely be "yes," as the funding possibilities for straight up clones of popular services will probably be dialed back in 2009.

What We Want in 2009: Help Us Manage Social Media Better

For the entrepreneurs still looking to get our attention with the latest social media toys, their pitch may no longer be "come try this, it's new," but instead, "come try this, it helps." Because if there's anything we learned from 2008, it's that social media overload is not sustainable.

Over the course of the past year, we found ourselves drawn to the apps, services, and features that helped us better organize the madness that is information overload. We added our friends to lists in both FriendFeed and Tweet Deck, we categorized our RSS feeds and even cleared out some for good, we de-friended the strangers we had collected on Facebook, we synced our social network friend lists, and we found ways to multi-post to our preferred networks. Yes, we became more efficient..but there's still so much room for improvement.

Our Social Media Wish List

Perhaps next year, we'll see more apps that help us better organize, if not filter, the information we deal with every day. We have some thoughts about what we would like to see and we hope that 2009 will bring these ideas to fruition.

• Google Reader add-ins and/or Greasemonkey scripts:We want Labs for Google Reader! It seems Google is more interested in revamping the Reader UI than giving us any real tools to deal with our RSS overload. If they won't help, then someone else should. We would love to see tools that let us view our feeds based on our attention data, without having to manually reorganize the feeds ourselves. We also want duplicates marked as read - if we read a friend's shared item from a feed we subscribe to, why do we have to see it again as we plow through our unread feeds? Finally, we need tools that let us better filter our subscriptions to reduce noise. Why can't we click a button to hide all the posts where someone has spliced in their delicious links or Twitter updates, for example?
• Auto-categorization tools: We tried to emulate Robert Scoble and what did we end up with? Only several thousand friends whose updates fly by at the speed of light. We tried to organize them into lists, but do you know how long that takes?! What would we would like to see are tools that organize people for you. Is it really so hard? The tools could parse our friends' Twitter profiles, for example, to categorize people based on location, business, or company. All the local people could be in one list. Everyone whose profile says "SEO" in another. Anyone in the top 50 or 100 users (based on followers/friends) in a third list called "noteworthy." Just because we want to customize and personalize our lists doesn't mean we couldn't use a little help getting started with the task.
• More Friend Synchronization tools: We want to friend you - really we do - but it's hard because you're here and there and everywhere. To make matters worse, you don't even use the same username on Digg as you do on Twitter. How will we ever find you? What we want is a tool that allows us to friend people, with one click on all the networks we possibly can, according to our preferences. It should also be able to delve into our social graph and sync up the friends we have already added.
• Friend List Sanitizers: OK, we followed/friended you, but we don't know why. We don't know you, we don't have any friends in common, in fact, we think you might have requested our friendship by mistake. So why are you still in our Facebook friends list? We need tools that help us clean up our lists to remove the accidental "stranger friendings" left over from our MySpace days. Even better, the tool could compare our Facebook list to our FriendFeed or Twitter friends to see if we know you elsewhere in order to determine whether to retain or remove the friendship.

These are just a few social media tools we would like to see developed in 2009. What are yours?

PostRank scores every item in your (or any) RSS feed, by number of comments, inbound links, saves in Delicious, mentions on Twitter, votes on Digg, etc. It then offers a filtered view or feed of the most relatively popular posts in that feed. The new top posts widget offers powerful new functionality, can be customized and installed in less than a few minutes and looks really hot.

Some publishers might hesitate to let users easily subscribe to such a filtered feed from their site - but those are often people who wouldn't subscribe at all if you didn't give them such a personalized option.

We've embedded the widget below - give it a try and you'll see what a powerful experience it offers.

The new widget also comes with a WordPress plug-in that will display each post's Postrank score in your WordPress dashboard. That's pretty hot.

Using this widget out of the box is really easy and it should fit nicely on your blog's sidebar. Unfortunately changing the size of the widget to put it anywhere else is a real pain - you can see all the white space above. It made us quite angry, in fact! The company said there was a bug that should be fixed promptly though, so hopefully all problems will be solved. There really are too few customization options. It would also be nice to be able to hover over the post rank numbers and see a popup of criteria for that score as well, as you can on the main site.

That minor frustration aside, we're very impressed with this new widget's speed and functionality. We expect to see it on a lot of blog sidebars around the web soon.

We decided to follow up on and reprint that post here on a late Friday afternoon. We're sure many of readers either didn't see it at the time or hadn't yet discovered ReadWriteWeb. Not everyone who did read it agreed with our conclusions, so after the post below we've added some of our favorite pro and con comments from the original, plus a cool personal story from a member of the RWW community. What do you think? Does online noise play a meaningful role in your life?

Blogs, RSS, IM, Twitter and FriendFeed - the number of sources of sources of information online can feel like it's multiplying exponentially every day. It's easy, natural even, to feel overwhelmed. Especially when we are more familiar with the tightly controlled editorial policies of mainstream media.

The social media space is noisy, though. There are many times when filtering that noise effectively makes a lot of sense (some tools discussed below) - but there are also many times when noise is just what we need.

Experiments in Noise Control

There are many ways you can roughly cut down on the noise in your information stream. More emerge all the time and this is a very valuable direction for services to be exploring. We don't want to argue that noise is always good, it's clearly important to spend some time without it every day.

The most recent entry into the noise filtering scene is probably FriendFeed's new "best of" feature. Late last night FriendFeed rolled out the ability to view just the items most popular with your friends on the service for the last day, week or month. It's something many people have been hoping for and there's no doubt it will prove useful. If you're not using FriendFeed yet, you can check it out and add me as a friend if you like here.

Other services that are good for filtering out noise are del.icio.us popular for a particular tag, AideRSS and Google Reader's overly friendly shared items from friends feature. We'd love to read about your favorite noise filtering tools in comments below.

On the Beauty of Noise

Filtering isn't everything it's cracked up to be, though, and you wouldn't want to live in a fully filtered world all the time. Social media noise is an essential part of learning and living on the web. Hear are some reasons why.

Unexpected opportunities.

Some people call it "serendipity," others call it "passive and opportunistic information acquisition." (Erdelez, see below.) The less limited the boundaries of your scope of view are, the more likely you may be to find things you didn't even think to look for.

Scanning quickly over large quantities of roughly relevant information can turn up invaluable resources, opportunities, context and contacts that you can passively process or opportunistically leverage at will.

Future Needs

It's one thing to find something you didn't know you needed right now, it's a whole other skill to be able to recall information that seemed marginally useful at best in the past at a time in the future when the need for it arises. Who can't remember doing that before?

The ability to recall passively collected information that was gathered purposelessly in the past and put it to use in the future is a particularly powerful form of intelligence. A person with a substantial reservoir of generally relevant information is a great person to have on any team.

Maximizing Recall

Some people worry that being exposed to too much information will lead to not remembering very much of it. Scientists say that's not necessarily the case, though. Sanda Erdelez, for example, wrote the following in her study Information Encountering: It's More Than Just Bumping into Information

A majority of participants in my information encountering study, when asked about their past experiences of "bumping into information," were familiar with the notion of accidental discovery of information and could recall these experiences clearly

We may be afraid that we won't remember key information that rushed past us in a river of news, but Erdelez argues that when prompted about a particular incident of accidental discovery our memories are better than we might think.

We would argue here in fact that the more total information our minds are exposed to, the more particular items we'll be able to recall in the future. One useful strategy may be to spend some time going through a large amount of information just a touch more quickly than we're comfortable with.

General Knowledge

Beyond simple recall of particular information in the past, internalized noise can be just as useful in the formation of wisdom and perspective as introspection, thoughtfulness and other forms of attentiveness can be. Spend some time skimming, it'll make you a better person. You'll meet new people, learn new things - don't worry, it's fun.

Personal Growth

Serendipitous search in the offline world is believed to be one of the ways our understanding of the world expands. David Pescovitz at BoingBoing writes about Swedish ethnologist Erik Ottoson's PhD thesis titled Seeking One's Own: On Encounters Between Individuals and Objects:

"Ideals of what is beautiful, useful and reasonable," Ottoson argues "materialize in conjunction with the experience of what is available and what is absent or out of reach."

That's more than just a beautiful reason you should read BoingBoing, it's an interesting understanding of the way that swimming through noise helps us become who we are.

Conclusion

Quiet time, time off-line, deep thoughts and long books are all beautiful things - essential to a healthy intellectual, psychological and social life. We argue, though, that the opposite of all those things - online social media noise, is also a great opportunity that deserves to have its worth recognized at a time in history when many of us are struggling to deal with it.

So take some time for yourself when you can, find a nice place to sit with a cup of tea and blow through a few hundred items in your RSS reader. If you can relax into it, it'll help you remember some of the reasons why you love the internet.

Creative Commons photos, Christmas 2007 series, by Flickr user Kevin Dooley.

Following up on this post

We write enough here everyday and read enough around the web that sometimes looking back at a post from earlier in the same year can feel like we're visiting another planet. This post, though, still feels pretty familiar.

A few things have changed, for sure. Hutch Carpenter, the blogger who made the chart in this posts about different ways to relate to noise, got a job at enterprise social bookmarking startup Connectbeam - in part, he says in announcing his new position, because of his use of FriendFeed! That's pretty heart warming.

In announcing his new job, Hutch wrote the following:

FriendFeed opened my eyes to the possibilities of knowledge as the basis of relationships. The ways in which content from a variety of sources is a powerful, addictive basis for learning, conversations and collaboration. How activity streams are compelling reads. I've been active on FriendFeed since March, and it shocks me how much I know about web 2.0 and technology in general versus last year. I've still got much to learn, and FriendFeed will continue to be a good source for that.

Diverse Reactions from Readers

Hutch's is a pretty happy ending to a story about noise, but not everyone who read our original post agreed with it.

One dissenter summarized a number of peoples' positions well when he wrote: "The web is about ME first and then comes the noise driven by the hype and the false version of truth that is popularity." That commenter, who went by the name "directeur," is building a startup based on this belief of his called FeedEgo.com. It's a personal relevance based feed reader and it's worth checking out, even if we do disagree with its creator about relevance vs. noise.

Some commenters said that balance was really what's most important. "Portland Broker" for example, wrote that "As with most everything, I think it's a matter of balance. Noise is everywhere; sometimes it's serendipitous, often it's not. A world that is overly filtered is lacking, just as one that is not filtered at all."

How can you as a person online create that kind of balance? Iconoclastic tech/culture blogger Stetoscope suggested the following: " I think what makes noise unbearable is the guilty feeling we have to not read everything. But if we takes some times to dive in the noise, without feeling guilty of what we have missed, it is just a positive habit." We like that advice and it sounds like it could work well with some of the tips we shared in a post last Spring titled Seven Tips for Making the Most of Your RSS Reader.

What do you think? Is social media noise good for you? How has it been treating you lately? If you believe in the need for balance, what are your favorite ways to create it?

Just recently, we polled the Twitter audience about their love (or not) of FriendFeed by asking the following question: "If you could only answer YES or NO, how would you answer this question: "Do You Love FriendFeed?" The reason for posing the question this way to not allow for qualified responses like "well, the service has potential, but at the moment it...." or anything of that manner.

In the end, the responses were decidedly mixed, and surprisingly, a lot of NO's turned up. At final count on Twitter it was 16 NO's to 10 YES's. (Of course, on FriendFeed, the ratio was a bit different...and, as is typical on FriendFeed, a conversation ensued.) While most FriendFeed users agree that the service is great for sharing content and starting conversations, a good many will also admit that FriendFeed hasn't yet hit the sweet spot when it comes to combating info overload.

Growing or Fading?

So where does that leave FriendFeed now? On the one hand, you have people like Steve Rubel claiming that he now has over 5000 people following him on Friendfeed - 60% of what he has on Twitter. That certainly seems to show promise for the FriendFeed service. Even with all of Twitter's issues, the service is bordering on mainstream, having already been used for presidential debates, MTV awards shows, and for tweeting news from the Mars Rover. For FriendFeed to even come close to rivaling Twitter numbers, there must be something there.

However, on the other hand, you have the king of early adopters himself, Robert Scoble, sharing a post in Google Reader entitled "Why Have I Been Neglecting FriendFeed?" by Kyle Lacy. In the post, Lacy cites information overload, burnout, and increased work responsibilities among other things, as reasons for his neglect. But what's really interesting is the comment Scoble left when sharing the feed:

Wait! Stop the presses! Robert Scoble tired of FriendFeed?! If Scoble is the canary in the coal mine of social media, what does this mean for the rest of us? (Note: he appears to have gotten over this).

Still, we wonder - is a FriendFeed burnout on the horizon? Or is it only a matter of FriendFeed adding a feature or two to skyrocket it to uber-success?

Now that we've taken the poll of a small Twitter (and FriendFeed!) audience, we thought it would be good to take the pulse of a wider audience that includes our decided readers here on RWW. We hope that you'll not only answer the poll, but share your overall thoughts in the comments - be them here or on FriendFeed. We support both.

PostRank looks at every item that comes through an RSS feed and scores it on a scale of 1 through 10 based on the number of comments it's received, inbound links, saves to del.cio.us, times it's been Tweeted and Dugg. The excitement comes in when the service delivers a filtered feed of just the 15% "most popular" items in that feed. It's a great way to pay casual attention to prolific feeds when you just want to see its own highlights.

Today the company is rolling out a slew of performance enhancements and new metrics including clickthroughs from its extensions, bookmarks in Ma.gnolia and mentions on microblogging service Pownce.

The company also rolled out a dedicated page for its very handy Google Reader extension - GReader users should check this one out.

We use AideRSS here at RWW every day and can't say enough about this simple but powerfully useful tool. We've written about it numerous times, including in the following particularly popular posts:

• Want That Post to Go Popular? Here's the Best and Worst Times to Post It


• What's Next on the Web: A ReadWriteWeb Toolkit for 2008


• How to Find the Weirdest Stuff on the Internet


• Comparing Six Ways to Find Top Blogs in Any Niche


• Ten Common Objections to Social Media Adoption and How You Can Respond


• Seven Tips for Making the Most of Your Feed Reader


• Why Gen Y is Going to Change the Web


• Five Ways You Can Fall in Love With Tagging Again


• An Ode to RSS On RSS Appreciation Day


• Why Online "Noise" is Good for You

It's true, we love AideRSS. It's just so incredibly useful we can't get over it. We wish the algorithm for determining popularity was more transparent and we hope that today's performance enhancements make a big difference - but we love it none the less. We'd love to see the folks at AideRSS connect with the good people at Gnip, a social media pinging service plus that we wrote about here.

The prospect of AideRSS's PostRank being rolled into other applications around the web is an exciting one. In what contexts would you like to see just the most popular items in an RSS feed?

As Corvida noted earlier, the next step for social media should be filtering, which makes the timing of The Filter's launch perfect. As web users and social media addicts become inundated with choices, there's now more need than ever for noise-reduction tools as opposed to just more aggregators.

In fact, Peter Gabriel's own reasons for creating this service echo the complaints of the information overloaded netizen. He says, "the first freedom the internet brought was the possibility of access to any content, at any time, or anywhere. Now that many of us are drowning in choice, we need good tools to help us make smart decisions."

The Filter Homepage

How The Filter Works

Some systems make recommendations based on your actions and history where others leverage the power of the crowd to find the best content, but The Filter combines both methods and uses data learned in one area to augment the other.

When you first sign up at The Filter, you begin by stepping through a brief profiling wizard in order for the site to establish an initial set of recommendations. As you begin to use the site, you can continue to personalize your recommendations in a way that's very much reminiscent of Amazon's "Recommended for You" section. On Amazon, items are rated with starts but The Filter uses a + / - sliding scale instead. However, like Amazon, you can mark items you own so they won't be recommended to you again while also helping the system get to know you better.

Getting to Know You

A second part to discovering your personalized tastes comes from "The Filter" which you download. This tool is a plugin application that works with either Windows Media Player, Winamp, and, of course, iTunes. Unlike other music player plugins like iLike, this piece of The Filter's system is not designed for social sharing, but actively collects data from your computer and sends it back to The Filter's servers. Where iLike uses the data it collects to help you discover friends of similar taste, The Filter solely uses that data for the purpose of improving recommendations.

However, that's not to say there isn't a friends element to The Filter - you can add friends and socialize with them via onsite mail and a very Facebook-esque  "Wall." However, besides inviting new friends via email, there doesn't really seem to be any good way to find new friends whose interests mesh with yours. This is one area where social music services like iLike and Last.fm have The Filter beat. For example, when you're viewing an artist's page, you're recommended more items like that artist, but not the profiles of other users who also like the artist, so it's hard to know where to begin with the friending process.

Another feature designed to improve recommendations is a profile import tool. To give The Filter a jumpstart, you can import your profiles from Last.fm and Flixster into the service to improve your recommendations quicker than if you had to start from scratch.

Is It Worthwhile?

What's most interesting about The Filter is the way that it combines your manually rated items, your buying history, your playing history, and your friends' likes (assuming you can find some) to provide an overall recommendation service. Its ability to stretch beyond just music to include videos, TV, and movies is also unique.

Yet it still feels somewhat lacking when compared to Last.fm or Flixster because, despite the social element it purports to have, it's difficult to locate other users to befriend on the service. Last.fm makes that easy - displaying other listeners when you go to play a song and offering numerous Groups where users can bond around a particular genre or artist. Flixster also has a tab on its homepage to help you "Meet People."

Additionally, the site was slow (although that could be launch day jitters) and occasionally buggy. For example, somehow clicking into the Genres section logged me out. The site's player was good - offering songs I enjoyed - but then again, Last.fm's radio is good too and they also offer software for scrobbling your tracks from your music player back to their site.

The Filter is still an interesting experiment, though. Instead of focusing on just one genre of entertainment, it has taken several different recommendation techniques and mashed them up to provide one overall aggregation and recommendation service for many different types of media.

If they can work out the kinks on the site and enhance the socializing aspect, The Filter could have a shot since it does have some unique features that make it interesting. For example, it offers entertainment news you can subscribe to and it recommends not just web videos, but also movies that go with an artist or band, and vice versa. And since you can continue to use Last.fm and Flixster (and hopefully other services in the future) to enhance The Filter's recommendations, then it's not really necessary to switch entirely. Instead, you can simply use The Filter more like an aggregator and filter for all the activity you do out on the social, entertainment-focused web.

An RSS reader like illumio could have a real shot at marketshare among the digerati if not for a few issues. The app, more of a competitor to the Newsgator desktop reader product line than to online readers, provides automatic filtering of your news feeds. Unlike technologies like AideRSS, which filters by popularity, illumio personalizes your feed reading experience by determining what's important to you and then displaying those top articles in a newspaper layout.

Illumio is not new, but its latest incarnation - illumio version 3.0 - was only released a couple of weeks ago. With this version, the app, although downloadable software, now launches within a web browser window. The UI has also been completely redesigned so articles are easier to read and navigation is simplified.

illumio

However, what's really interesting about illumio is not the fact that you can read your RSS in a newspaper layout - it's the built-in filtering technologies the software employs to do so. In order to determine your interests, illumio scans the files on your computer to discover your interests and expertise. Not to worry, though - this information is never shared with others, nor does it ever leave your PC, according to the company. The privacy-conscious set might find that a little bit disconcerting, but you have to admit - that's certainly a unique approach to uncovering someone's interests in order to personalize their news.

Using illumio

When configuring the software for the first time, you must initially specify some default interests, but after completing the configuration you can remove any unwanted feeds and upload your own OPML file. (It's too bad you can't just start with an OPML upload, though.)

Once you're up-and-running, your feeds are displayed in a newspaper layout that features a tag cloud of topics on the right and your feed groups on the left. You can rate articles with star ratings to further train illumio as to what you like.

There's also a tab at the top of the newspaper called "Questions," because, if it wasn't enough that illumio was a filtered RSS reader of sorts, it's also trying to be a Q&A service, too. Here you can ask questions and respond to those posed by others in the community. While that might be useful in a business environment where team members review feeds together and then discuss as a group, it's hard to see exactly how this would really benefit a typical user.

Recommendation Alone Doesn't Ensure Success

Unfortunately for illumio, their great strides in improving recommendation technologies are going to be overlooked by the community they wish to engage for two major reasons: 1) it's downloadable software, 2) it's Windows only.

While they are working on a Mac version (sign up here to be notified of its release), the fact that illumio is a software download is going to be a huge turn-off for many users. Those at illumio maintain that the reason for it being a download is due to privacy concerns - since it scours your hard drive to assist with its recommendation and filtering services, you wouldn't necessarily want that private data stored online. That being said, most users are looking for filtering and recommendation to occur within their web apps, so illumio doesn't have a chance at converting anyone beyond the already small niche of desktop reader fans.

Instead, Try Newsgator Online

By partnering with a company called SenseArray, NewsGator Online is now offering RSS feed recommendations to its users. These recommendations come from the data NewsGator had been collecting en masse from their users as well as from an individual's actions - like a thumbs up or thumbs down - that were performed in their desktop reader (either FeedDemon for Windows and NetNewsWire for the Mac).

This latest news comes on the heels of last month's announcement about Newsgator's incorporation of our favorite filtering service, AideRSS. While both of these technologies are currently only available in Newsgator's online reader, there are plans to make them available in the desktop readers as well.

Additionally, according to a blog post on Venture Chronicles, the company has also been working with mSpoke to provide a categorization capability to their products that will soon offer Wikipedia-style indexing of content.

Newsgator Online, image courtesy of Jeff Nolan

Who Will Win?

With illumio's commitment to being a downloadable product only, Newsgator has the advantage, but even it still faces opposition from the still popular online reader provided by Bloglines as well as the increasingly-popular Google Reader.

Although users are demanding products that provide filtering, it's yet to be seen whether or not they will ditch their currently preferred online readers just to have access to these tools. If anything, Google Reader is one of the worst when it comes to filtering - in fact, its social feature that lets friends share stories means that you are likely to read the same story over and over again. Yet, it is growing in popularity despite its lack of filtering. (That is, unless you just read your friends' shares, which could be see as a type of human fitltering for your RSS.)

However, it is nice to see some forward movement in the recommendation and filtering space, even if it's not available in all readers yet. The fact that it's out there will mean other web apps and desktop readers will need to start offering similar technologies in order to stay competitive.

Since we published our first look at AideRSS last July, their PostRank algorithm has changed a lot. At launch, PostRank included information from comments, Digg, del.icio.us, Technorati, IceRocket, and Bloglines -- now the latter three have been replaced with Twitter links and Google blog search conversations. Some of those changes likely had to do with API restrictions, some likely with just general tweaking to make the algorithm perform better.

Because AideRSS calculates PostRank against only that blog's past performance, the ranking is a fair representation of that blog's best work. For example, a PR 10 post on ReadWriteWeb would require different interaction metrics than a post on a small personal blog. PostRank would be easy to cheat -- you could comment a million times on your post, get your friends to Digg it, tweet it, add it it del.icio.us -- but since the service isn't measuring you against other blogs, there's really no incentive to cheat it.

AideRSS also announced support for OpenID.

With so many different platforms to aggregate, noise levels are surging. An underlying issue in the level of noise is that some of these services were not made to interact with one another. Users of social aggregation tools should understand that what you may consider noise is actually a side-effect of using a social aggregation platform. Users should also note when you may be confusing aggregation with importation.

Though importation may be a necessary step within aggregation, there is a difference between the two. Importation is usually more selective and limited to the inclusion of select items of other services. This may include common specifics such as names, ages, and locations. However, with aggregation a service attempts to conglomerate key features and entire contents of other services. This makes aggregation seemingly more flexible due to it's ability to encompass a wider scope of content.

Using Platforms for Different Purposes

The services that are aggregated are usually used for entirely different purposes. For example, some platforms are used to keep in touch with others such as family, friends, or business contacts. On the other hand, you have services that are used only for the purpose of finding more content and conversations pertaining to certain contents.

When you pull in an account from a platform that is completely unrelated to to the usage of another, you will inevitably create a small amount of noise. However, with social aggregation platforms it's hardly ever just one account. This can increase the noise level to an irritating high for other members of these services, including those within your personal network.

Services Cater To Various Audiences

Aggregated services are not only used for entirely different purposes, but also cater to different audiences. Consequently, who you may be catering to is dependent upon why you may be using the service. While some articles or content submitted to services may overlap, this is only because there are overlapping interests for the different audiences on these platforms.

How does this affect noise levels? If you're using a service to promote content, you may be considered noisy to those that are looking for conversations. If you're using a platform to keep in touch, then those looking for content and in depth conversation surrounding particular content would need a way to block out idle chatter.

Filters Are The Future Of Social Media

Filters are rapidly becoming a pertinent issue for developers of social media services. As a result, social aggregation platforms are in the perfect position to lead the pack. While this is no easy task and one that cannot be solved in its entirety, it would help resolve another issue social media users are facing: courtesy.

Instead of being able to freely add whatever service you wish, some users like myself are taking into account what others may consider noise on certain services as a courtesy to members. In essence, you are becoming our own filter. You may refrain from important other services for fear of being labeled as "noisy". With better filtering options, users can use these services to their fullest extent without becoming a nuisance to others or missing the benefits of aggregating all of their accounts.

Fellow venture capitalist Brad Feld voices similar concerns, in a post entitled, "I Need A News Feed For My News Feeds." The solution for each of them lies in the creation of some sort of universal feed dashboard that manages your social activity feeds and determines which items require action and which are of interest.

For consultant Jevon MacDonald, who thinks that lifestream aggregators are starting to become "noise aggregators," the solution to the problem lies in the development of filters that learn what you want to read. "If I give someone's del.icio.us bookmarks a thumbs down every time I see it, then you should stop showing it to me. If I give a thumbs down on ever single del.icio.us bookmark I see, then make sure you never show me one again," he writes.

Interestingly enough, it was Facebook that really pushed this whole activity stream idea into the limelight in the first place, and it is Facebook who seems to be taking an early lead in developing tools to filter them. The Facebook News Feed is already filtered algorithmically, and Facebook offers a couple of tools to help users tailor the filters to their interests (including the thumbs up/down method that MacDonald espouses).

According to Marshall Kirkpatrick, the concept of the News Feed has been a more important contribution to the social media space by Facebook than their vaunted platform. It could be that Facebook will also take the lead in tackling how to cope with the information overload that has resulted from the numerous activity streams we're now tracking on various social services.

To be fair, lifestreaming and lifestream aggregation is in its infancy. The Facebook News Feed only appeared about a year an a half ago, Twitter only gained real attention about a year ago, and FriendFeed and similar services are even newer. However, dealing with information overload is clearly a problem that these services will need to figure out how to address -- whichever does it best will likely be a big winner.

Feed Rinse is a best of class RSS filtering application. It offers filtering by keyword, tag, author, title, etc. It supports regular expressions, has a built in profanity filter, and lets you upload your OPML file for easy importing of your RSS feeds (it can also export to OPML). And since we first covered it in April of 2006 it has gone completely free.

FilterMyRSS is a no-nonsense keyword filter that filters posts out by keyword. By that we mean that the service tracks posts for the keywords you want to filter and removes posts where it finds matches. It can filter by description, title, or category and offers some advanced XML options.

When we first reviewed 2or3things' Blastfeed in late 2006 it appeared to be shaping up as a good consumer filtering alternative to Feed Rinse. But since, the company has shifted gears and now offers Blastfeed as an enterprise filtering solution. Blastfeed's keyword filters can also be used to remix and republish blog feeds filtered for specific content -- for example, a fan site for a specific band could create a news feed from multiple general music sites that publishes only stories about that specific band.

Feed Sifter is an almost painfully simple RSS filter that filters in by keyword. Or, in other words, it watches for the keywords you enter and pushes stories to you that match those keywords. It can search for single keywords, or return stories that only match sets of keywords.

ZapTXT is a keyword filter (of the in variety) that returns results via email, instant messenger, or mobile phone. It is designed for people who prefer to consume RSS feeds via non-traditional methods (i.e., not via an RSS reader). ZapTXT also powers email, IM, and SMS alerts for sites like TheStreet.com.

For do-it-yourselfers, Yahoo! Pipes offers an easy way to create filtered RSS feeds. The RSS remixing application makes it easy to create simple filters. Just define your feed using the "Fetch Feed" module, connect it to a "Filter" module, which can filter either in or out by title, description, category, author, or date, and then connect it out to the Pipe Output. It isn't as comprehensive as Feed Rinse, but you do perhaps have more control. The example pipe below would filter our feed and return only posts that talk about "Facebook" in the title.

Building a recommendation engine is a complex endeavor, which we discussed here a year ago. But in addition to being a technical challenge, there are also fundamental psychological questions: do people want recommendations and if so, then when are they open to them? Perhaps an even bigger question is: what happens when the user receives one or more bad recommendations? How tolerant will they be?

Genetics of Recommendation Engines

All recommendation engines are trying to solve the following problem: given a set of ratings for a particular user, along with those of the whole user base, come up with new items that this user will like. There are many algorithms that can be applied to the problem, but all of them focus on three elements: personal, social and fundamental:

• Personalized recommendation - recommend things based on the individual's past behavior
• Social recommendation - recommend things based on the past behavior of similar users
• Item recommendation - recommend things based on the item itself
• A combination of the three approaches above

A social recommendation is also known as collaborative filtering - people who liked X also like Y. For example, people who liked Lord of The Rings are likely to enjoy Eragon and The Chronicles of Narnia. The problem with this approach is that peoples tastes do not in reality fall into simple categories. If two people share the same taste in fantasy movies, it does not mean that they will also both like dramas or mysteries. A good way to think about this problem comes from genetics. Many times we meet people who have features that we recognize and have seen in others. For example, eyes might look familiar, or lips, but it is a totally different person.

The other kind of recommendation is an item-based recommendation. The best example of this system is the Pandora music recommendation service. It works by ranking each musical piece by more than 400 different characteristic - musical genes. It then automatically matches the pieces based on these characteristics. There are challenges with tuning the algorithm to work well, but it is also challenging to apply it to other verticals. For movies, for example, you'd need to come up with ranking each movie along many scales, starting from director, cast, plot; and then obscure things like musical score, locations, light, camera work, etc. It certainly can be done, but this is complicated.

The Guy In The Garage

The complexity of the recommendation problem is due to its vast space of possibilities. Much like it's hard to figure out which exact gene is responsible for a particular human trait, it is hard to figure out which bits of the movie or music make us rate it as 5 stars. Reverse engineering human thinking is hard. Which is exactly why one of the contestants highlighted in the Wired article is relying on a very different trick to make his algorithm work.

Nicknamed Guy In The Garage, Gavin Potter from London is relying on human inertia. Apparently, the rating of the movie depends on the ratings of previous movies that we just saw. For example, if you watch three movies in a row and rate them with 4 stars, and then watch the next one which is slightly better, you will rate it 5. Conversely, if you rate three movies in a row with 1 star, then the same movie that you would otherwise rate as 5 would only get 4 stars from you.

Just when you think that this is not true, you will discover that this algorithm now sits in the 5th place and still is making progress, while other algorithms are spinning. Enhancing formulas with a bit of human psychology is a really good idea and this is where we turn next.

Replacing Recommendations with Filters

How many times has this happened to you: a friend recommended you a movie or a restaurant, so you went there all excited - but ended up disappointed? A lot! It is obvious that hype sets the bar high, increasing the chances of a miss. In math speak, this kind of miss is known as a false positive. Consider now what would happen if instead of recommending a movie, a friend tells that you are not going to like certain movie, so do not bother renting it.

What bad can come of that? Not much, because likely you are not going to watch it. But even if you do and you like it, you are not going to be experience negative feelings. This example demonstrates the difference between our reaction to a false negative and a false positive. False positives upset us, but false negatives do not. The idea of respinning recommendations as filters is about leveraging this phenomenon.

When Netflix makes recommendations, it sets itself up for a sure failure. Sooner rather than later it is going to miss and recommend you a movie that you are not going to like. What if instead of doing that, it would show you new releases and have a button: filter the ones I am not going to like. The algorithm is the same, but perception is different.

Filters in Real-Time Culture

And this idea becomes increasingly important and powerful in the age of real-time news. We are increasingly oriented towards continuously filtering new information. We do this with our RSS Readers everyday. We think of the world in terms of streams of news, where things of the past are not relevant. We do not need recommendations, because we are already over subscribed. We need noise filters. An algorithm that says: 'hey, you are definitely not going to like that' and hide it.

If the machines can do the work of aggressively throwing information out for us, then we can deal with the rest on our own. Borrowing from the spam box in emails, if all the tools around us had a button that said 'filter this for me', and maybe even had a mode where such a filter is on by default, we'd all to get more things done.

Conclusion

Building a perfect recommendation engine is a very complex task. Regardless of the method, collaborative filtering or inherent properties of things - recommendations are an unforgiving business, where false positives quickly turn users off. Perhaps applying psychology to the problem can make people appreciate what these complex algorithms are doing. If instead of recommending things, machines would filter things we definitely won't like, we might be more forgiving and understanding.

Now tell us please about your experiences with recommendation engines. Were there ones that worked really well? Would you be open to filtering instead of recommendation? Besides movies and news, where would you like to have these filters?

See also our follow-up post 10 Recommended Recommendation Engines.