Maybe I'm not quite understanding the term "Personalized Clustering". I've got a script [1] that scans through about 1000 RSS feeds from my subscriptions, extracts all the links, then groups and sorts the links by number of entries containing the links. This gives me a pretty decent overview of the day's memes from the perspective of people I watch.

Now, this script is simpleminded and doesn't pull in any other factors like "blog authority" or popularity amongst sites not included in my subscription list, but it's been pretty useful as my first daily stop.

[1]: a">a">http://decafbad.com/blog/2006/01/03/sharing-attention-while-reading-feeds">a script

I believe that there are less hardware intensive methods of personalization and cluster that can be developed without a huge resource investment. There have been some intriguing developments in data mining and data warehousing recently that give me hope that the massive amounts of data necessary to develop a personalized meme tracker aren't too far away. I think the biggest issue is that people look at the challenges from an established perspective and say "Eh...that's too difficult." If more people develop applications, it will inspire others to come along.

(It's early, I apologize for any typos or spelling mistakes. And I'm not sure how to add newlines in this forum.)

As for the problems you cite, you're absolutely right - link data isn't good enough for personalized feeds, so while we try to use it when it's there, the bulk of the clustering is done using text analysis. And because you can cluster any arbitrary set of RSS feeds, we do the clustering ON-DEMAND, so efficiency is a big deal (and we manage - for most views it happens on my laptop in a 1 or 2 seconds - feels more or less like loading any other web page).

Ari's doing text analysis in Flock. That definitely helps, but my guess is not enough. On a scale of 1-10 in "personal memetracker" usefulness, link analysis gives 2 points, state-of-the-art text analysis gives another 2 points, but that's only 4/10 when most people demand higher. Yes, I just made up those numbers, but it's one way to illustrate my point.

If I'm wrong, and "personal memetrackers" make the cover of Time Magazine, well, I'll just release my desktop client.

1. Pushing the calculation load onto the client
2. Using contextual analysis to overcome the lack of links problem

I look forward to trying it out as that approach seems to make a lot more sense, but I believe that a destination site such as memeorandum can live alongside having relevance in your desktop aggregator - two different things. Ari it seems you guys can pull this off well and create a decent aggregator (with 147 feeds in my bloglines, I usually skip whole sections of posts because I can't keep up).

Contextual analysis is needed in the current crop of memetrackers, look at TailRank today and you will see that the same story appears in a few seperate clusters, and both Megite and memeorandum sometimes get the same problem though some of them solve it better than others. Relying just on links isn't even working well for the bigger stories.

More information here: http://www.google.com/support/bin/answer.py?answer=26651&topic=1593

Am I missing something subtle here?

I would guess these people don't have more then passing knowledge of database systems. I would say what is being proposed here is trivial.

I create a personalized view for each and every member on my site on each and every pageview. I peak out at 1 million pageviews/hour so what i'm doing is several orders of magnitude more complex then anything proposed here.

One reason some of the memetrackers have entered the market so fast is that they have a somewhat trivial problem. All they have to do is bake one static HTML page and serve the same page to all their readers. This is a somewhat easy problem. TailRank's architecture is a bit more complicated than this obviously and when memetrackers start to be used by more and more people TailRank will be able to scale.

Scalable search is hard but for the most part Google solved it.

Maybe TailRank is the Google of memetrackers. God I hope so :)

Anyway. Wanted to follow up with a comment but I'm pending a blog post on this soon. Just swamped with work right now to write the post :-/

Onward!

Kevin

Nik: I hope we can make a good aggregator, but don't get your hopes up too high, we still have a lot of work to do (even I still rely on bloglines...at the very least we need to finish adding in read/unread which we have code for but won't quite be ready for this release). You're definitely right that there is room for both Memeorandum-style and this personalized style to coexist: I'd say the Memeorandum style is more accessible (you don't need to go through and pick your own feeds to read, you just get a good view of what's going on as a whole) but for some people there is that need to read every post on a certain set of feeds you like which is where I think the personalized ones fit in.

Aapo: ugg, I hate this personalized search thing. Useful personalized search would be if you're searching for something that you saw once before and it says "here's that page you were looking at last week about this". Personalized search that a lot of people love to talk about is "you're a tech person, so when you search for Windows you want Microsoft". Dammit, people interested in technology still need to get their windows washed every once in a while. Personalization is not the solution to query disambiguation and I wish people would stop fixating on this. Yahoo has the right idea with it's query refinement suggestions: "Also try: jaguar cars, jaguar animal, jaguar parts, jaguar pictures".

It really depends on how you define "hard".

1. This looks like a 1 person company, what are the chances that person has the nessary skills to make a kick ass database?

2. Hard in most online applications is rarely a expotential problem. Search\personalization has already been proven to be solved in linar time.

3. Is it "hard" to allow personalization and still make any kind of profit/revenue?

I think the reason we don't see a ton of personalized news trackers is because there is no money in it. I make that comment because I see no monitization on Memeorandum. If there is money in it, who cares about spending 10 grand and buying a bunch of servers?

You didn't describe a specific personalization method that could easily solve the problem.

It's not going completely live until Monday, but nothing beats being topical :)

Allow a user to define a filter containing blogs and subjects they don't care about. Then you can continue to expand on those filters. I get the feeling this is something you have to throw a lot of possibilities out there and then see what works best.