Many people have worried that the inaccessibility of historical Twitter search results might mean that the messages weren't being saved at all. Company co-founder Biz Stone told us otherwise by email today, though. Twitter is in fact saving all the tweets. You just can't access them through search "right now."

Scrubbed geolocation data after two weeks and no way to access historical information at all? That sounded like a pretty bum deal for a world-changing new communication platform. So we emailed to ask.

This was the reply we received from Biz Stone: "We definitely save all the tweets although you're right in noting that our search focuses more on newer content right now. And yes, the plan is to drop the coordinates after 14 days."

There you go. Now that Twitter is putting $100 million more into the bank, it would be great to see some of those resources dedicated to making information retrieval on the service a first class function. The future would be thankful.

Twitter has offered two explanations for the change. First, that very few users were choosing to receive these kinds of messages anyway and that it was confusing. Then, this morning, the company put up a blog post saying simply that "there were serious technical reasons why that setting had to go or be entirely rebuilt--it wouldn't have lasted long even if we thought it was the best thing ever." So what's the story? Here's our best guess.

First, it is clear that most users have not chosen to receive public replies sent by their friends to people they themselves aren't following. Non-early-adopters in particular are quite likely to consider these kinds of messages noise. Many of us early adopters believed such messages were a part of the magic of Twitter; it's a great way to discover people your friends find interesting. Information overload can be dealt with by forming groups in a third party twitter client like Tweetdeck or Seesmic Desktop. Getting over our antiquated sense of guilt and obligation concerning reading every message we receive would help too. (We've argued that online noise is good for you.)

The fact is though that most Twitter users now probably only receive replies sent to people they know. That's been the default setting for several months and we assume the system has thus been architected for this use case.

Scalability

What does that mean? Probably this: when you publish a Tweet that begins with @username, the computer that all your user files lives on in Twitter HQ probably already knows not only who your friends are, but also who their friends are. Everyone who is your friend but is not friends with the person you are replying to won't even have the message sent to them. Making that determination locally and limiting the tweets broadcast to your friends is probably much more efficient for Twitter than sending that message to all your friends.

Even updating the "friends of friends" files for all your friends every time you make a new friend - is more efficient than sending out replies to all your friends who aren't following the recipient.

Sending Tweets from one user to another is traffic-expensive and if most people don't want replies directed at people they don't know, then none of us are going to get that kind of message.

Most people don't want a noisy public conversation? Most people don't want serendipitous introduction to new people? That sounds like most people don't want a key part of what makes Twitter most magical. That magic is expensive and if millions of Oprah followers don't even want it, then a few thousand @marshallk followers don't get to have it either.

Who can blame them? Everyone wants Twitter to scale. Maybe it has to be neutered in order to do so. Architecting the social graph can't be easy. We can be sad about the decisions that the company makes in order to try to do it, but that probably isn't going to change things.

Instead of strolling through the social graph to discover new users, Twitter users will likely be given a new set of recommendation features pointing them to new people to follow. It may be a "people you follow also follow these people" type of feature, hopefully it won't just be more recommendations based on who Twitter's established leaders like. Or perhaps those of us who want all messages published by our friends can pay for a premium account. I'm ready to do that right now.

Thanks to the super-smart Alex Iskold, who we trust a lot and who helped talk us through this technical speculation.

Update: Hours after we put up this post, Twitter has reversed this policy. The story just keeps getting more interesting!

Even the app's developer agrees that the app crossed the line, overstepping Facebook's much celebrated privacy controls. We're still disappointed though, and we wish that this rich source of data could be opened up for developers and users to build value on top of. What kind of publishing system doesn't offer an RSS feed? A fundamentally closed one.

The argument is that the News Feed RSS made it too easy to violate privacy conditions put on some users' Facebook data.

Facebook hasn't responded yet to our request for an official comment, but Facebook software engineer, Ari Steinberg, explained in an unofficial comment on the Activity Streams discussion group, "We're certainly not opposed to enabling you to export your own content (in fact, we're always trying to work on ways to make that easier), but exporting all your friends' content to a totally public place without their permission isn't cool."

But it was really useful. Sometimes privacy is at odds with innovation, and while we would never want to say that privacy is illegitimate - we're not happy to see it shut down major potential avenues for innovation either.

If the app published an authenticated feed (meaning you had to log in to view it) and if the apps around the web had better support for authenticated feeds, then the story would probably be different. That's not where we're at, though; even the very popular Google Reader can't handle password protected RSS feeds.

App developer, Teck Chia, says privacy issues and copyright violation in the use of the phrase "News Feed" were both cited by Facebook when the app was shut down. Chia understands the privacy concern but hopes to be able to find a solution shortly. One option may be to publish only a user's own items in a feed, perhaps folding in the updates of friends who have added the app as well and specifically opted-in, and perhaps sending items through the feed that say merely that "Your friend John updated his staus, click here to log in to Facebook and read it."

We're not sure how useful those options sound. It's not surprising but it is a real disappointment that Facebook shut the app down. The wall that keeps Facebook user data in and private by default feels too contrary to the fundamental nature of the internet for it to last. In a poll we performed last week, 40% of our readers said they wanted Facebook to open their data either a little or a lot. We're sure the percentage of all Facebook users who feel that way would be smaller, but a closed pocket of the web seems to us to be something that will be worn away in time.

There may not be an RSS feed for your Facebook News Feed today, but it sure seems like only a matter of time until there is.

We're big fans of web standards here at ReadWriteWeb and we'll tell you why.

This year's been a big one for web standards; the President Elect just enabled users to login to comment on his website using the standard authentication protocol OpenID, for example.

So get your blue beanie or similar hat on and make yourself a photo. Your friends will wonder why you and others are wearing them online today and when they ask either you or Google - they'll end up thinking about the importance of web standards as a result.

For more info, visit Jeffrey Zeldman's blog and see if there's a group relevant to you participating, like the South African Web Standards and Accessibility Group, by whom we were reminded of today's event.

During this transition, it's not "old with the old and in with the new" - several technologies from the prior period of innovation are still important. These include the following:

• Service-oriented Architecture (SOA) - middleware that enables new component apps
• Business Process Management (BPM) - user-driven automation of manual tasks
• Mobile - beyond laptops to cell phones and PDAs
• X Internet - RFID and sensors at the edge the net connects physical objects to the internet

New Trends CIO Should Care About

Keeping in mind that the above trends are not going away anytime soon, CIOs still need to be aware of the upcoming trends that will define the future of enterprise IT. In the "IT everywhere" wave, business technology (BT) is the driving factor. The control of this technology is being shifted away from IT and is increasingly under the control of the business organizations and the users themselves. Simply put, BT is the future of IT. When looking ahead to the future, Forrester recommends CIOs keep the following trends in focus:

1. Technology Populism: Web 2.0 and social networking in the enterprise. Workers are provisioning their own tools, especially when IT can't provide. IT had best look towards integrating Enterprise 2.0 into their organization - if they don't, the end users will simply go find their own apps to use. The risks of ignoring this trend include compromised security, comprised privacy, and poor control of intellectual property. (We discussed this concept in more detail here).
2. The Information Workplace: The information workplace is a term describing a next-gen platform that consists of numerous parts such as unified communications, portals, enterprise content management apps, office productivity apps, collaborative technologies, business intelligence, data warehousing, and more. However, the information workplace isn't about each of these technologies individually, but how they all seamlessly come together as a whole. Today's information workplace is role-based, individualized, and thanks to the Web 2.0 invasion, it's also often "social" and "quick," as Web 2.0 tools tend to be.
3. Dynamic Business Applications: These are component apps that target certain roles but change easily. Over the next five years, IT's goal is to develop enterprise software that adapts to the business and that's capable of evolving as the business grows. These apps are designed with a focus on the people who use them, but are also highly adaptable as the business changes over time.
4. Digital Business Architecture: This includes SOA, unified communications, and virtual computing among other things as a top-level conceptual model for planning the future of both technology and architecture. A digital business architecture means the design of your business is accurately reflected in your technology.
5. IT Ecosystems: By 2012, there will be a shift in the dominant form of IT delivery from buyers self-integrating technology to having outside providers assemble and manage it. Those with the strongest delivery capabilities will lead the way. This trend will also include a shift away from software investments based on ownership to those based on subscription as well as an increase in new IP sourced from open communities.
6. Enterprise Master Data Management: MDM focuses on delivering trusted data throughout the enterprise. Today the focus is on addressing cross-application data use and management while also considering MDM's multiyear and multiphase business capabilities. In 2008, information and knowledge management professional will work on overcoming the organizational, process, and business case challenges to bringing this data to the enterprise.

I believe I have finally seen the light as to how the Internet Topic Exchange (ITE) could be made simpler to use and more viral at once, taking one cue from my colleague Stephen's recent ridiculously easy shibboleth strategy for putting together conference aggregators and another from Joshua Shachter's now-defunct reversible.org

1. Right now submitting a post to a channel requires people to either fill in a form on the channel page of their choice or send a TrackBack ping to the channel. Both are simple, but still harder than they should. How about adding a third option: simply link to the channel in your post. To make it easier, the top of each channel page could even provide some standard boilerplate chunk of the requisite HTML.

2. In order for this to work the ITE needs to watch the participating weblogs. It already watches for weblog updates, so just let people register their blog or feed once and let the ITE pick out the posts that link to it as they appear thereafter.

Note that this new option automagically generates visibility for the channels on participating blogs with every post submitted; this is one of the key elements that were missing for effective word-of-mouth propagation of awareness of the Exchange. I'm kicking myself for having taken so long to find a simpler way.

Of course, along with ease of use comes more spam. I've been thinking about this too - more later.

My Reply

Here's what I posted in Seb's comments (tided up a bit):

This is pretty much what I was getting at a month or so ago, only I wanted Topic Exchange and K-Collector to play together more. Ref: Proposed Solution for ENT Topic-Sharing Community and the follow-ups here and here. And this is what I ended up with - K-Collector picks up my posts via my ENT-enabled RSS feed. And my weblog automatically pings Topic Exchange every time I post, via my use of MT categories (which match up with Topic Exchange names).

You propose people include the TE link in their post, which is similar to what I do - only I include the link on a reference tacked onto the end of the post (so it doesn't actually form part of the post itself). This way works for me, because it means I don't have to manually enter the link - MT does it automagically for me. The downside of my method is that I use pre-defined categories, which does limit the topic selection somewhat. But I balanced this by knowing that my topics/categories match now with *both* Topic Exchange and K-Collector, which was what I wanted.

Anywho, good luck with your proposal. If it helps more people adopt TE then it's a good thing.

<ent:cloud ent:href="http://topicexchange.com/topics">
<ent:topic ent:id="<$MTEntryCategory dirify="1"$>" ent:href="http://topicexchange.com/t/<$MTEntryCategory dirify="1"$>/"><$MTEntryCategory dirify="1"$></ent:topic>
</ent:cloud>
<ent:cloud ent:href="http://w4.evectors.it/itEntDirectory/topicRoll.opml">
<ent:topic ent:classification="what" ent:href="http://w4.evectors.it/itEntDirectory/topic?topic= <$MTEntryCategory dirify="1"$>" ent:id="<$MTEntryCategory dirify="1"$>"><$MTEntryCategory$></ent:topic>
</ent:cloud>

The KC and TE references are pretty similar, except that KC doesn't need the dirify="1" bit for the topic name. Dirify in this context just means the topic value is made lower-case and if it's a double word an underscore is added between the two words.

Each post references R/WW, KC & TC

At the end of each individual post, you'll now notice there are 3 links: the first one takes you to the Read/Write Web archive for the topic I've specified for the post. The next two links take you to the respective KC and TE pages for that topic. This is all done automatically using the <MTCategory> tag (oh, did I mention I'm using Category now instead of Keyword?).

There is one downside: sometimes TE and KC won't have the same topic name as me. In the case of this post here, I'm using the topic name "Movable Type" - which is fine because both TE and KC have a topic by that name. But a lot of times the topics don't synch. But that's a whole other issue, which I can't fix by myself.

So that's me done with topics for a little while. I'm going to relax a bit this week and write up some laid-back R/WW posts.

What I will do is continue with my own experiments with Movable Type. I already have the TE cloud reference in my RSS template, so I'll add the KC cloud reference too. Then as I write posts I'll add topics using my chosen MT field (at this stage 'Keyword', but I'll probably change to 'Category').

Note that I could download the MT client Matt's created, but I'd actually rather play around myself with MT and see how both KC and TE pick up my posts.

Likewise, I've asked Phil Pearson if Topic Exchange can pick up my ENT data from my RSS feed automatically. If I can get both KC and TE aggregating my ENT data, without me having to ping either one, I'll be a happy man :-)

Also I should perhaps clarify my goals with all this. In the short-term I want to:

1) Set-up MT so that I can add topic data to both KC and TE.

2) Set-up my own internal topic navigation, which ideally I'd like to synch with KC and TE (in terms of topic names).

For now I'll leave the hard part aside - i.e. synching topic data between KC and TE. Matt's done a lot of work on this in the past, using XFML and XTM and so forth, and so he can tell you it's not a trivial task. So let's call that a long-term goal.

I guess my remaining goal for the short-term is to try and convince Phil to get TE to automatically aggregate my ENT data from my RSS feed. But being a humble user, I've no idea how difficult a request this is - i.e. it's easy for me to ask the question, it may be a lot harder for Phil to do the work ;-) So I don't want to press the issue...

Also, I want to investigate Blogdigger some more to see how I can tie in my topic navigation experiments with Bloggdigger's categorisation system. Greg - any ideas for that?

Here's some of my feedback (copied from the comments - I must get these enabled inline...):

Overnight while pondering my post (which I regard as just a 'starter for 10' btw, not a final solution by any means), I did conclude that KC essentially already does what I describe - polls registered RSS feeds with ENT in them and aggregates them. It would be great if TE also had that functionality.

It's the client ping that I think is unnecessary and possibly holding back community uptake - with TE the ping is a manual process for the blogger, and with KC you need to install an add-on tool to enable the pinging. Both require too much manual effort for the blogger (IMHO of course). eg Bloglines does all its aggregation automatically (every hour I think), with no pinging required from the blogger.

Although Andrew I take your point about bandwidth utilization. But if Bloglines (and Blogdigger) can do it, why not KC and TE?

It has to start somewhere and where better than from a humble user's perspective - i.e. me. So here's what I've done so far to get the ball rolling and I want to challenge others to build on this (or if its wrongheaded, suggest other ways to merge KC and TE's functionality).

When I was using Radio Userland, I had my site hooked up to K-Collector using their add-on tool for Radio Userland. When I moved to Movable Type a couple of weeks ago, I had to cease my connection to K-Collector because as far as I know it's a Radio Userland-only tool (although there was talk of an MT add-on being developed?). So tonight I did some tinkering with my blog, to 

1) try and re-enable community topic sharing for my weblog; and 

2) to resume my project to get an internal topic navigation system running (I had done some initial work with XSLT at the beginning of the year, but that fell by the wayside...).

Now, there is one major thing that Topic Exchange and K-Collector have in common: they both use ENT (Easy News Topics), which is an extension of RSS. Paulo and Matt created ENT as a simple form of topic mapping - and recently they've been talking of upgrading it. When I used Radio, whenever I published a post I also selected some topics from the K-Collector Radio add-on tool. This added the topics to my RSS feed, like so:

<ent:cloud ent:href="http://w4.evectors.it/itEntDirectory/topicRoll.opml">
<ent:topic ent:classification="what" ent:href="http://w4.evectors.it/itEntDirectory/topic?topic=strategy" ent:id="strategy">Strategy</ent:topic> 
</ent:cloud>

As I understand it, the K-Collector website then aggregated my post to whatever KC topic was specified in the ENT tags in my RSS file ("Strategy" in the example above).

Because I'm using Movable Type now, I needed to find another way to specify topics. The great thing about MT is that it has spare fields where you can basically add whatever you like. So I decided to use the previously unused "Keyword" field to hold all my Topic data. This will serve two purposes: 1) I can hopefully get Topic Exchange and K-Collector to aggregate my posts based on the topic data I enter using this field (more on that in a minute); and 2) I will use it also for my own internal topic navigation. So this allows me to match my own topic navigation to the community topics (in TE and KC).

The next thing I did was alter my RSS template. Firstly I declared the ENT namespace in the <rss> element, like so:

<rss version="2.0" xmlns:ent="http://www.purl.org/NET/ENT/1.0/">

 Next, I added the ENT tags. Now for this I decided to start off using Topic Exchange, because it happens to be a perfect fit with the <$MTEntryKeywords$> tag which I'm using for my topics. If you look at the Topic Exchange RSS feeds for each channel, it makes it easy to swap in my MTEntryKeyword. e.g. this is from the TE "Technology" topic:

<ent:cloud ent:href="http://topicexchange.com/topics">
<ent:topic ent:id="technology" ent:href="http://topicexchange.com/t/technology/"> technology</ent:topic> 
</ent:cloud>

Because all the TE URL's are in the format "http://topicexchange.com/t/" and the id perfectly matches the topic name, it's easy for me to swap in my MTEntryKeyword. Here's what I added to my RSS template in MT:

<ent:cloud ent:href="http://topicexchange.com/topics">
<ent:topic ent:id="<$MTEntryKeywords$>" ent:href="http://topicexchange.com/t/<$MTEntryKeywords$>/"> <$MTEntryKeywords$></ent:topic> 
</ent:cloud>

I added this just above the ending <item> tag.

NB: it may be just as easy to do this with KC, I haven't checked that yet. However KC has the additonal component of the ent:classification tag, so I decided to tackle TE first as it didn't have that extra tag.

So that's my RSS file ENT-enabled. Now whenever I add a new entry to my weblog, I simply enter the Topic Exchange topic name into my "Keywords" field. When published, this adds it to my RSS file in the ENT tags specified above - which at the moment map exactly to Topic Exchange's format. But...I still have to manually send a ping to TE under the current TE system.

So here's my suggestion (and I apologise if this has been suggested before - I'm sure it has!). What if we, the users, could register our RSS files with Topic Exchange and K-Collector, and both of those services then regularly poll all registered RSS files - say hourly - and pick up any new posts that have ENT tags in them? So in effect Topic Exchange and K-Collector act like RSS Aggregators (e.g. Bloglines) and poll users RSS files, rather than the users having to manually ping the TE and KC websites. It makes more sense for the Aggregator to do the work, rather than the person/blogger.

The beauty of this idea is that we - the users - can use ENT as our 'Topic Central'. That is, we specify our topics in the RSS - and that's the end of our part of the job. It's then up to the TE and KC aggregators to poll registered RSS files and collect all the new ENT tagged data. If e.g. both TE and KC have a topic named "technology", then great they won't have to do any work to add them to their respective clouds. There is a bit of work of course when the topic names and/or ids don't match up - but that's solvable surely? e.g. some form of topic mapping between TE and KC.

So what do you all think? I'd really like to see some action to merge the functionality of TE and KC together - for the good of the community (rah rah!).

Seb Paquet recently re-opened the conversation on topic-mapping in blogs, and Rogers Cadenhead and Dave Winer have been talking about it, so it's on peoples minds. I'd like to suggest the following...

We need to meld the best features of Topic Exchange and K-Collector.

Topic Exchange and K-Collector each has its strengths. K-Collector has a great add-on for Radio Userland, which allows you to easily select relevant topics and add them to the community server. Topic Exchange requires you to send a trackback ping to its server for each topic, which makes it more open and extensible but also more effort for the blogger. Topic Exchange has a strong user community - it's like an open source project. K-Collector seems to be aiming at a corporate market and so that's where their development focus is I think. Nothing wrong with that per se, but it does mean I'm more emotionally attached to Topic Exchange these days.

So the two development efforts can learn from each other. For example, it'd be great if Topic Exchange can automate its ping process a bit more (I'll try and think more about this, so I can offer some potential solutions), and K-Collector can keep the blogosphere in the loop regarding its continued development (e.g. more updates, especially to the mailing list).

I wonder if it's worthwhile merging Topic Exchange and K-Collector? There is so much talent in each project, but perhaps topic mapping has its best chance of gaining mainstream acceptance if we work under one umbrella "project". What do you reckon, am I getting all hippie about this or is a combined project a viable solution?

Cross-posted to Topic Exchange and K-Collector

My ideal is to do the XSL transformation on the server-side, rather than the client (browser) side. The reason for this is that due to the proliferation of different browsers on the Web, it'll be a nightmare to second-guess how all of them will process the XSL transformation. Whereas with a server-side transformation, I know how my server will handle the task. Basically it comes down to this: it's one less thing for the user's browser to do when reading my site. Why get the client to process the XSL transform if I can do it on my own turf (my server)?

But having just said all that, I currently don't have the correct server configuration to do the XSLT processing. I'm used to working with IIS at work, so I was able to come up with a nifty ASP solution to transform my XML to HTML. But my weblog runs on Apache, so ASP can't be used (there may be a plug-in somewhere to get around this, but I wouldn't bet on it being an easy implementation). Far better that I do it in a language Apache understands, and the obvious one is PHP. So I've investigated using PHP to do the transformation and this will probably be my long-term solution. However it requires me to install two things on my server, which I've yet to do - Sablotron and Expat. These things will enable me to do XSLT transformations on my Apache server using PHP. There are other options too: Java/Cocoon and Perl/AxKit, to name a couple I found while searching. However I know very little about those options. If there are any XSLT experts out there who can advise me on the best method to transform XSLT server-side, I'd appreciate it.

My short-term solution is to do the XSL transformations on the client-side, using Javascript. And yes I know I just talked myself out of doing this a couple of paragraphs up. But I really want to see how my XML transforms look now and a Javascript is the quickest way. Besides it doesn't hurt to experiment with both client-side and server-side, to see for myself the differences.

Here is a test page I've done: it's an HTML page with Javascript (c/o W3Schools) that uses an XSL file to transform a selected section of my XML file into HTML. A caveat: it currently only works with Internet Explorer. I haven't been able to track down a a cross-platform version that will work in Mozilla and Firebird etc. Note that there is also a way to transform straight XML-to-XSL-to-HTML in modern browsers (eg IE6) without using Javascript. However to do that I'd need multiple xml files (or else do some tricky things to bundle multiple XSL files into 1 XSL file). As the purpose of my topic nav is to have a single XML file to update, and bearing in mind this is a short-term solution, I decided to use Javascript to do the job.

Hey, what are you trying to achieve with all this XSLT processing?

That's a good question; allow me to explain. I recently converted my taxonomy to a flatter hierarchy, with a maximum of 3 levels. In line with this, I also decided I only want to categorise each weblog post into one category. This may seem to go against the grain of the latest in weblog taxonomy trends (see Jon Udell's Dynamic Categories post), but there is a method to my madness. I hope. 

I was browsing through an introductory book on Wittgenstein, as you do, and I read that his major philosophical work called the Tractatus is ordered using a decimal numbering system. He lays out his arguments like so:

1 -> 2 -> 3 (first level)
1.1 -> 1.2 -> 1.3 (subordinate to first level)
1.11 -> 1.12 -> 1.13 (subordinate to second level)

My understanding, based on my limited reading of Wittgenstein, is that he structured the Tractatus using seven main theses. For each theses, he drilled down and analysed it using the above numbering system. Not to sound pompous or anything, but this is similar to what I do with my weblog. I have a dozen or so topics that I regularly write on and it's tempting to think of these as theses. They're probably more like themes than theses, but hey it's just one letter difference :-) My recurring themes are things like Universal Canvas and Microcontent.

To make a long story short, I discovered Wittgenstein's numbering system wasn't suitable for my weblog taxonomy. However I did end up with a manifesto of 12 themes that generally revolve around the subject of the Two-Way Web (it's not restrictive though). I've categorised each of my posts into one of those 12 themes. There is one further level below that, so that I can bundle things together if need be - e.g. my collection of posts about Nanowrimo 2003 is categorised as Top > Writing > Nanowrimo 2003.

To bring this post full circle, currently I'm using Radio Userland's OPML-to-HTML service to produce the above taxonomy. As I mentioned, I've got a draft XML-to-HTML version going that uses client-side XSLT. Here is my list of 12 main categories (a plain html file for now) and here is a list of all my weblog posts categorised using this taxonomy (this one uses XSL).

In future I will add extra bits of data to the XML file (e.g. dates, maybe even the content of the posts). This is another advantage of using XML over OPML. I'll also eventually introduce some dynamic categorisation, a la Udell. All of this XML exploration may be leading me inexorably towards a tool like Syncato, which stores all its content in XML.

In summary I think my thematic taxonomy will help me keep my weblog writing on topic. And from a readers perspective, you will be able to explore any one of '12 paths to Two-Way Web enlightenment' ;-)

Meme 1) The current trend for tech blog re-designs to have a minimalist, lotsa-white-space look that places emphasis on the content. Dave Winer probably started this trend with his re-design, but I've seen it elsewhere before him (e.g. Peter Lindberg and Erik Benson). And now Robert Scoble and Marc Canter. Mark Pilgrim's site is also bathed in white space nowadays. Hey I guess my site is pretty minimalist too. Maybe it's just a tech blog thing?

Meme 2) Jason Kottke's comment today: "Nothing takes the fun and personality out of writing like metadata." Jason points out that blogs have lots of extra design bits in them to help people organise and link together information, but it distracts from the main content.

What's the connection? Maybe it's that some of us bloggers are trying to push extraneous pieces of visual design out of our weblogs. And what's this a trend towards? The usurpation of websites by RSS perhaps. I'm beginning to sound like Steve Gillmor or Dave Winer. What I mean is: does web design, in the visual / graphical sense of the phrase, really matter anymore? Does ontology / taxonomy of a physical website mean much nowadays? If the majority of people read a weblog via an RSS Aggregator - and that's not the case yet, but it's heading that way - then does Web Design or Taxonomy matter a hill of beans? Why bother putting in all these design flourishes and metadata if our readers don't see/use it?

I'll give you an example. I use k-collector to categorise each weblog post I write into topics. But those topics can't be seen via RSS Aggregators (at least not in the one I use - let me know if you do see them). Another example: trackbacks aren't visible in the RSS Aggregator. A link to comments is available in my Aggregator, but there is no context - ie I don't know how many people have commented on a post, I have to click the link to open it up in my web browser.

And some people still don't provide the whole of their text in their RSS feeds. Movable Type people are the biggest offenders (if that's the right word), but only because it is the default behavior to include only excerpts in their RSS feeds. Doing this may be The Last Bastion of Web Design, because it's forcing us readers to go out of our RSS Aggregators and visit their websites. It's noticeable that most of the people who I've categorised as "Designers" in my Bloglines RSS Aggregator exhibit this "click to see" behavior. Can't blame them, they've got nice pretty sites and they want people to view them.

Mind you I've noticed in my own referrer logs that about half of my visitors (to my actual site) get here via a search engine. So that alone is probably a good case for me to continue to provide a nice design and a helpful taxonomy. Plus of course you want to make a good impression generally speaking with your web presense. It's like you don't want people to see your house when it's messy and has things strewn all around the lounge. You want to vacuum the place and have your furniture arranged in an orderly fashion before visitors call. So design and taxonomy has its place, even in our increasingly RSS-ified world.

But RSS (and/or Atom) is the Future. How long before we can represent our content's taxonomy/ontology in our RSS feeds? I mentioned this in a previous post and Dave Winer commented: "I plan to make my aggregator work with categories." That's definitely a good start. What are other aggregator developers planning to do in this regard?

And how long before we can cram all those bits of metadata that Jason mentions into our RSS feeds? That wasn't Jason's point of course, he was saying all that metadata necessarily de-emphasizes the main content. I agree with that sentiment, but I have to admit also that I'm addicted to those little bits of metadata. I like reading comments, clicking on the trackbacks, seeing the referrers, etc. It all adds to the community aspect of weblogs. And if we can cram all that community into our RSS or Atom feeds, then all the better.

Front Page > Personal Network Architecture > Group Ware > Collaboration Ware > Wiki For Collaboration Ware > Summarizing Is Necessary

I don't think it is a hierarchy of categories. Bill explains it as "the chain of generations of pages whose creation led to the current page."

Secondly Bill notes that a key question to building an ontology is asking yourself: what's the point? This is something I was pondering last night when I went to sleep (dreams being one way I think through technical things...sad as that makes me sound!). I was also thinking about why we put so much effort into organising our weblog sites, when the majority of our readers read our content via an RSS Aggregator - which doesn't care about the content structure. How long before some bright spark creates an RSS Aggregator that does take into account each publisher's content ontology? Or maybe the question should be asked the other way round: how long before site developers figure out how to create an RSS feed(s) that represents its home site's ontology?

Andrew also makes a good point: "The ontologies are nice, but they shouldnít require oodles of work to set up, maintain, and categorize things into."

Amen to that. This is the drawback to using XTM topic maps - it's going to require a lot of work to set it up. Same could be said of RDF. Hmm, thinking more...

1. FTrain - Paul Ford: Trust me to start with the most complex ;-) Paul Ford's site is graphically striking and he's one of the few bloggers to have implemented a Semantic Web-like structure. Not to mention his writing is mind-blowing. But to the design. I don't claim to fully understand it yet, but basically it seems every piece of content is connected to other content according to various types of relationships. Here's how Paul describes it:

"Ftrain is a hierarchy. Any given page has one or more of parent, children, and sibling pages, and every page lives somewhere in the hierarchy."

Further down that page, he states:

"Ftrain is this complicated because it has over 1000 separate nodes, all of them connected to one another in some way, with something like 700,000 words between them, and all extensible."

I decided to start at a recent Ftrain article, A Response to Clay Shirky's "The Semantic Web, Syllogism, and Worldview", and browse from there. If you scroll down to the end of the article, you'll find some navigation and external links. First there is a "Links Related To" table, which has 1 external link. Below that there is a statement of the hierarchy:

"This is A Response to Clay Shirky's "The Semantic Web, Syllogism, and Worldview", a technical essay by Paul Ford, published Monday, November 10, 2003. It is part of Theory, which is part of Ftrain.com."

By this I understood I am at the third level down from the homepage. Breadcrumb-like: Home > Theory > A Response to Clay etc. But the "technical essay" bit threw me. I clicked on that and discovered this article had been cross-posted to another category: Home > Taxanomy > Things > Ways of Communicating > Forms of Expression > Essays > Technical Essays.

So I back-button back to the Clay essay. The next thing after the hierarchy statement is a list of 6 links under the heading 'Related'. 3 of these links are also attached to 'Technical Essays'. The other 3 aren't immediately obvious relations.

Below this is a group of links called "Navigate by Hierarchy", which is two other links from the category 'Theory'. Finally there is a "Navigate by Time" option.

So all up, a pretty complex navigational structure and who knows how it is done under the hood. The Ftrain Sitekit provides some clues - we find out the site is built using XML technologies and XSLT.

2. Erik Benson: Erik bases his site around the concept of nodes, which are grouped into categories, which are placed in a section. So it is a hierarchy, like Paul Ford's site. Hmm, already I'm sensing a pattern in the ontologies I admire - they're hierarchies. To be honest, I hadn't clicked that Ftrain and Erik Benson's sites were hierarchical (grouped by categories) until now.

I took Erik's most recent article, The future has already happened, as my starting point to check out the ontology. Under the title, the following breadcrumb displays:

"home > thing > idea"

I clicked on "idea" and it took me to that category page, which displayed an alphabetical list of all the other nodes in the "idea" category. There's also a chronological navigation, under Weblog Archive. The article above is listed as:

"home > weblog > general > archive for Dec 2003"

Another feature is the "Related Nodes" functionality. I'm not quite sure how this works yet, I'll have to come back to it.

3. Dave Winer's Scripting News. As most people are aware, Dave has recently switched to a category-based design. Right at the top of his homepage is the following breadcrumb:

"Top > Dave's World > Weblog Archive > 2003 > December > 12"

This is chronological, but Dave is also categorising each of his weblog entries. To find the category listing, you have to go to the search drop-down box and click "All Cats". Then you will see a long list of all Dave's categories. For my research purposes I clicked on "Politics / Money" and got a page which displayed all the posts in that category.

4. Bill Seitz's Wikilog. Bill's site is a cross between a Wiki and a weblog. When I first saw it a few months ago I was blown away by it. As I've been following it I've gotten to like the way all content on one topic is grouped together on a single topic page. So no matter if two entries on the same topic were written a year apart, both entries end up on the same page. This has huge benefits in terms of linking and relating ideas together. Bill calls it his "thinking space".

Usually with Bill's site, I track his RSS feed of headings in my RSS Aggregator (Bloglines). When I see a heading that looks interesting, I click on it. For example, a recent one was SummarizingIsNecessary. If you scroll to the bottom of this page, you'll see a list of "Backlinks". According to Bill, backlinks are:

"...a list of all pages referring to the current page. This is useful for finding "related" information. (This is the Two Way Links feature available in pre-World Wide Web Hyper Text environments, that people like Ted Nelson have been complaining about since the Web came about.)"

Summary of Part 1:

I'm sure I haven't done justice to the sites I've analysed so far. They are all pretty complex and very well-developed. But there are some patterns emerging for my purposes: they all in some way use the concept of "nodes", 3 of the 4 use hierarchical categories, all but Dave's have a "related links" feature.

This is just a start. There are other sites whose ontology/taxonomy I admire. Andrew Chen, Mark Pilgrim, Phil Pearson - to name just a few. But I'll be here all night if I write about them now. Maybe tomorrow. For now, I'll think more about the "category vs topics" dilemma that I'm stuck on currently. I'm very keen on having a topic-based navigation, which has the benefit of a bottom-up "flat" structure of content - and I was thinking of using XTM topic mapping to achieve "related link" functionality. However given that hierarchical categories are being used to great affect by 3 of the 4 people listed above, maybe I'll change tack. Hmm.