The Future of Search: Social Relevancy Rank
FriendFeed has recently launched a search feature, and so Facebook search must be coming soon.
Real-time Web search (of streams of activities) is a hot topic right now. Everyone, including Google and Microsoft, recognizes the value of using trusted contacts as filters. What was once called social search is now called real-time search, but this time it will really happen. First, it will be applied to streams and then to the Web in general.
What we are about to get is a Social Relevancy Rank. Whenever you search streams of activity, the results will be ordered not chronologically but by how relevant each is to you based on your social graph. That is, people who matter more to you will bubble up. How does this work? Well, there will be a formula, just as there is a formula for Page Rank.
Solution 101: Rank by Friends and People You Follow
Here is an idea so obvious that it is surprising Twitter has not implemented it already: front-load search results with people you follow. When you search for, say, "Wilco" on Twitter today, the results are in the chronological order. That is not really relevant because you do not know who most of these people are. But if instead you could see people you follow, the search results would be much more useful.
This is not possible on Twitter today, but it already works great on FriendFeed. There, results are filtered or ranked based your social graph. This is not difficult for FriendFeed to do because, on the one hand, it knows who you care about and, on the other, it applies its advanced feed search technology to your social graph:
This sounds awesome, but there is a problem. "Wilco" works well as a query because the band has just released a new album, but many other queries would return no results. Simply put, your friends on Facebook and people you follow on Twitter can't possibly have an opinion on every topic you may be interested in. This is a problem of sparse data: trusted opinions are scarce.
Small Worlds and Taste Neighbors
To solve the problem of sparse data, we need more data... obviously. One possible solution is to incorporate other sources that you trust (i.e. broaden your social graph). As a next step, search results could rank people you may not be directly following but who are being followed by people you follow. Or in Facebook-speak, friends of friends. You could argue that you are not familiar with their opinions and so cannot yet trust them, but given the small world phenomenon, their contributions are often just as valuable.
Another step could be to include people with similar tastes, so-called taste neighbors. This approach is common among vertical social networks such as Last.fm, Flixster, and Goodreads. These networks have ideas about which people, other than your friends, are like you. However, this is a costly calculation and takes time. In order for Twitter to do something like this, it would have to compare people based on links or perform semantic analyses of tweets over time. Yet even though this is a difficult problem, it will be solved in time.
The Influencers and the Crowd
Aside from using the "second degree" of your social graph or taste neighbors, a Social Relevancy Rank could front-load influencers. In the absence of any other metric, someone who is followed by hundreds of thousands of users is likely more relevant to you than someone you don't know at all. Using number of followers as a weight might be a good way to order the rest of the activity stream.
In general, combing through countless tweets from strangers is not terribly useful anyway. Just as people have stopped looking at anything beyond the first page of results on Google, sifting through pages of tweets in chronological order gets tedious quickly. What needs to be incorporated into the Social Relevancy Rank is the aggregate sentiment of the crowd: a score that tells you yay or nay and gives you an opportunity to drill into more results if you choose.
The Quest for the Perfect Filter
There is no such thing as a perfect formula. Even Page Rank isn't perfect. Yet we all use it and find it useful. Much as Page Rank has been adapted and tuned to search the web, Social Relevancy Rank will evolve over time to help us make sense of endless streams of activity. This ranking will have a profound impact on how we tap into our friends' opinions.
It will change the face of general Web searches in time, too. Today, results are automatically ranked by relevancy and freshness. Once Social Relevancy Rank is factored in, search results will be re-ordered based on social relevancy.
And now, as always, please tell us what you think? What would you expect from a search engine with Social Relevancy Rank built in?
Share
Facebook
Comments
Subscribe to comments for this post OR Subscribe to comments for all ReadWriteWeb posts
Alex,
Interesting ideas but I have to disagree that social search is now synonymous with realtime search. I get results to many queries (stocks, scores, weather) that are the epitome of realtime without social qualifiers.
When people search in a recovery modality (exemplified above) they want the answer. Period. There is nothing about social connections that makes delivering that answer any more precise. At the moment (and in rare occasions) social connections can deliver that answer faster but I think that is a temporary advantage.
When a searcher is in discovery mode things are different. There is a window where it is possible to develop learning that places authority around people to particular subjects in much the same way Google does with domains. However, I’m not sure that once authority is established that segmenting, filtering and ranking based on proximity to the node of “me” becomes all that valuable. In fact, it is the nature of discovery goals that they are less dependent on trusted sources than those of recovery.
For me Social Search has always been about discovering people. That is what makes it unique from other search modes and where its (large) value resides. It is also where Social Relevancy Rank is likely most valuable. LinkedIn is the clearest & best example of a Social Search Engine that I know of and not coincidentally makes great use of the ideas you have captured here.
Cheers,
Jonathan
Hey Jonathan,
You are right, I collapsed real-time into social.
The view of real-time that you hold, to me is really just news. I am not sure how it is different from what blogsearch or google news or any other news search has been? That is, if its coming from newspapers or bloggers its news and it needs to be fresh, but when it comes from people on Twitter or Facebook its social.
When people pass links they effectively vote/filter information much like search engine does. So when you look at this process in aggregate it really does look like what search engine is doing - voting things up.
Alex
Another interesting area is the idea of improving on Page Rank by switching from analyzing the structure of links (which after all reflect interests of content creators) to analyzing the structure of behaviors (instead of using a link from Page A to Page B, which PageRank does, use the probability of a user's transitioning from Page A to Page B). This reflects user's intentions, and user's assessment of the utility of links, and is a much better approach than PageRank. The problem, of course, is getting this data. But there are several good approaches to this, and not all of them involve violating user's privacy!
Microsoft is trying something like this with their research project called BrowseRank (http://news.cnet.com/8301-1023_3-9999038-93.html), although they weren't the first to come up with the concepts in detail (http://appft1.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&p=1&u=%2Fnetahtml%2FPTO%2Fsearch-bool.html&r=0&f=S&l=50&TERM1=Galvin%2C+Brian&FIELD1=IN&co1=AND&TERM2=behavioral&FIELD2=&d=PG01)...
Once again when looking about useful information the secret is selection and trust (i.e. filtered information). That's probably why I value so much what is in my Delicious Network.
nda
I found a very interesting solution which allows you to combine the typical search behavior of most people (which is typing in keywords at Google) with the ability to define a specific set of people doing research on the same topic (for example an ad hoc research team OR every employee of a certain company OR a small group of people from different companies working together on a project... you name it): www.qitera.com. Qitera integrates with Google which means that it displays search results of those associated researchers as the first Google results and offers additional services like alerts.
(Disclaimer: Qitera sponsored one of my Competitive Intelligence Seminars - BUT apart from that I am really excited by the solution.)
the opening statement implying that facebook get's all of its innovative ideas from friendfeed is pretty funny alex - but also sadly true though the fb folks don't come close to the simplistic functionality of ff, just say'n :)
Posted by: mike "glemak" dunn
|
July 17, 2009 7:14 AM
Linkedin sorts some search results by closeness of a connection and has done so for quote some time.
this concept of social raking seems interesting
Alex,
A very interesting and timely post - you raise a lot of key points. I think your Solution 101 will get us a long way and there's certainly a perfect storm brewing that should enable us to deliver this kind of functionality.
However, we need to keep an eye on what comes next once generic use of Friends and Followers, or even Friends of Friends, no longer scales due to ever increasing amounts of information being available. In fact there are cases already where this doesn't work well enough, particularly outside of taste domains.
What I found in my PhD research in this area was that people make sophisticated decisions about who to ask for information, and this varies according to the nature of the task. For example, there's no guarantee that you'll ask the same person to recommend both movies and financial products, however well you know them: these two examples depend to varying degrees on taste overlap and the expertise of the information source. It's this dynamic we'll need to understand in greater detail.
If you'll forgive the element of self-promotion then there's a full discussion of this in my thesis: http://tomheath.com/thesis
Cheers, Tom.
Posted by: http://openid.tomheath.com/me
|
July 17, 2009 9:43 AM
The One-Riot white paper you Tweeted ( http://bit.ly/WhitePaper ) provides a nice framework to extend search beyond its informational confines to include browse and real-time search. However, discovery within a social framework warrants it's own framework.
I would expand Johnathan's premise about social search beyond merely discovering people to include discovering people, their passions, their opinions, their expertise, their assistance, etc. Jud Valeski at Gnip posted a rather elegant framework for the interplay of self-expression and discovery that pivots on connectedness in a social environment ( http://bit.ly/connectedness ). Assisting online discovery with criteria from connectedness and the aggregated experience of my social graph not only serves my "logos", but, my "pathos" and "eros" too.
Hi Alex,
I think an interesting fourth possibility is refining the Influencer Rank concept you outlined so it's topic-specific instead of a measure of generic popularity. I touched on this concept briefly in my PageRank for People article a month ago, and the more I think about it the more it seems like the most likely solution for general-topic searches. The challenge is finding a social search algorithm that can provide meaningful results for all types of queries. If I'm searching a topic that my social graph doesn't already follow, I still want to get returned meaningful results and I suspect some form of universal Influencer Rank score is required to achieve this.
Great post - I really think this cross-breeding of Search and Social Graph is going to give us the next big web/nav innovation.
Cheers,
/-Marshall
Alex, great post that really defines the issues clearly. I agree with Tom (#9) that "people make sophisticated decisions about who to ask for information, and this varies according to the nature of the task". That is true based on my personal observations (not on external research). What is interesting to me about Twitter - compared to FB or LI - is that it is about WHAT you know and not WHO you know. I find myself looking at somebody who tweets something interesting on a topic that I am researching and then I look for who they are following on that same subject. What I am looking for is Subject Relevance more than Social Relevance. I want to follow somebody who is an expert in that subject. Today I make that judgment intuitively, but as the volume scales it would be great to get some filtering based on how other people have rated this person ON THIS SUBJECT. Bernard
The site that could really benefit from a "social relevancy" filter would be Yelp.
If I could see restaurant ratings based solely on "friends & following", it would dramatically I would actually use it again.
As a question to the community: How would this affect search results for brands and companies that have essentially put time and energy into building their own relevancy, whether through some system of pay-to-play or overall presence. Would it be a total reordering of the search results based on the system that's been described? Not that I think this would necessarily be a bad thing by any means, particularly if it ultimately benefits users, but I'm curious to understand how this would work if implemented, as certainly corporations would have other thoughts (as they almost always do). Perhaps I'm missing something fundamental here, but I'd welcome your reactions.
On a side note (though it may be obvious), it seems that this system would most benefit the growing community of online creators - bloggers, designers, thought leaders - who are actively engaging with a variety of online platforms.
From tupperware parties and water cooler conversations to the next online revolution. Good stuff.
Thanks,
MC
Depends on the level of openness of the service enabled by APIs, the application of this system may come much quicker and may be developed to become superior service.
There is a window where it is possible to develop learning that places authority around people to particular subjects in much the same way Google does with domains.
Very good blog. You make some very good points
Check out the first real time search engine demonstration here:
http://blog.wowd.com/wowd_blog/2009/07/short-wowd-demo-screencast.html
I'm not sure if this is the future, but it is very cool.
One limitation of this approach is its focus on finding information already found in your social sphere. While this is useful for known-item, precision-oriented searches, it is less useful for exploratory search when you're trying to find something novel or make a connection that hasn't been made before. See my post Which future of search? for more details.
That's probably why I value so much what is in my Delicious Network.
Well... still too static. The main question to ask right now is what kind of purpose search really fills for a person. Secondly we have to ask ourselves whether or not there is a better way to meet that purpose.
Thinking static search such as the one proposed above, we are still only considering linked pages and people, forgetting the impact that linked data will induce on the way we use the web. In fact, the web will not be anything like the web in the future. It will be something different, and thus it becomes very difficult to know whether or not search in the form of question and answer will take place in the web.
Think about it. How did you use the web before the search engines?
I believe the future web is no host for influencers or naturals or social medias or serps. I believe the future web understands us individually and then translates our wishes from how we use the web, putting us in contact with the best possible person or database to talk to in order to solve our situation in a way that suits us the best.
I don't think we will search the web, I think the web will search us, answering our questions even before we ask them.
//Jesper
There are influence sites that rate whether a given Twitterer seems to know more about a subject than others do. The algorithms take into account the quality, not quantity, of the followers.
Although you could score tweets with links higher than those without, you definitely don't want search results to reflect votes on how politically correct information is (RTs often reflect this) and lots of people with huge Twitter followings are charlatans who used steroid software to get that "audience" which is itself largely made up of spammers.
Even for non-charlatans on social media, having a huge following may mean you don't have a day job and don't ultimately know what you are talking about on a variety if not majority of subjects. Furthermore, RT'd tweets are often the shorter ones about Twitterers with short names or they are jokes...making them suspect as to relevance for a serious researcher.
I delete Twitter spam followers constantly, which has halved what my follower count could have been. I don't want get-rich-quick or see-my-cam spammers unfollowing and refollowing all the time so this is why I block (anyone who tries to tell others how to get more followers gets blocked).
Like the influence metering sits, search algorithms should reward people for having a better concentration of real followers who might read any given tweet.
So, sure, if algorithms can identify that such and such a person has 750 followers few of whom score as spammers + this person seems to be constantly commenting with a given keyword combo...then this person should be considered more of an expert on the subject than @APlusk with 2,404,049 followers who just mentioned the keyword for the first time.
Reference: #9, above: Since following doesn't scale, you can't pull any relevancy from it.
Ah – leave it to the technologists to dicker over the approaches and miss the main point.
What’s missing in these techno-sym models is the human element of trust. We must build in the trust layer into all these processes if it is to have any value. Add the filter of trust to the Social Relevancy Rank (which can have definable algorithmic attributes) and the complex picture becomes clearer.
Trust is the glue that holds societies together – that goes for the virtual world too. Real time or not.
Judy Shapiro
Sr. VP, Paltalk
So, we're actually back to building Collaborative Filtering's Mentor Pool from which to select one's most likely recommendations.
What a concept! Why don't computer scientist bother studying history?
TV
This is a very good article, very well written and easy to understand, took a lot of work. kudos http://www.ClubDistrict.com Nightlife Simplified is something I came across, that will be big.
nice article...
but der is more on this...
follow this..
http://u.voizle.com/socialrank
hope u found it helpful
There's an app http://chatterfly.appaturelabs.com that restricts search to friends, would be nice to see as a native Twitter feature.
Alex,
thanks for the great post and the important issues you raised here. I totally agree with you that social ranking will build the base for future search. In this context, I would like to share with you and your readers a service that goes in the same direction.
The service is called PLEM and is available at http://eiche.informatik.rwth-aachen.de:3333/PLEM/
The main aim of PLEM is to harness the collective intelligence to rank and recommend learning elements. The idea is quite simple, each filtering action on a learning element from the Web (e.g. comment, link, save, like, rate, vote, view, share) counts as one "vote" for that learning element. The aggregation of the "votes" would then reflect the popularity of that element.
You can find more information about this service on my blog at http://mohamedaminechatti.blogspot.com/2009/07/plem-mashup-driven-long-tail-aggregator.html
Looking forward to receiving your valuable feedback...
Social Rank concept should be implemented in real time search results. I have started to recognized the importance as I started to see lot of real time hoax. Read http://www.gseo.net/blog/post/241 to know my views on Social Rank.