There may be millions of webpages around the Internet that display a widget of recent Tweets, but when the Library of Congress does it, I think that's notable because of the organization's prominence, its public role and its complicated relationship with transparency. "One of our goals with the Twitter account is to provide timely alerts about legislative developments and, with this change, now we can do that directly from the homepage," explained Andrew Weber, Senior Legal Information Analyst at the Library of Congress this morning, in a post on the Library's blog.

There's something a little funny about seeing a Twitter widget on the front page of the THOMAS site, but it sure does get the job done. It's not Facebook, Google Plus or the open-source, federated service Status dot net (wouldn't that be nice to see). No, it's a microblogging service now extensively owned by a sprawling Russian investment group that was used. I guess you've got to go where the people are. Hey, I love Twitter as much as the next person.

The Library of Congress, believe it or not, has not in all cases always been so interested in transparent communication with its constituents. In early 2009, back before Wikileaks was famous for leaking diplomatic cables, the organization leaked what it said was a billion dollars worth of research reports from the Library's Congressional Research Service. Public release of those documents had allegedly been opposed on the grounds that it could cause "impairment of [congressional] member communication with constituents."

At press time, @ThomasDotGov has 5,508 followers on Twitter, now that I added one by following it myself. The account was created just three months ago.

The songs can all be listened to for free online, but they cannot be downloaded.

The National Jukebox launches with recordings made by the Victor Talking Machine Company between 1901 and 1925 and contains a rich history of American music, including Fanny Brice singing the original "My Man" and Theodore Roosevelt giving a speech on "The Farmer and the Businessman.

More content will be added to the National Jukebox over time, says the Library of Congress, with additional Victor recordings and other music from Sony-owned labels. The new site today follows the Library's announcement last month unveiling a Music Consortium Treasures site that gave researchers access to various historical music manuscripts.

I'm sure the majority of readers here have at least sent a personal letter to friends or family in their lifetime. However, the same cannot be said about my generation. I've sent tens of thousands of emails, Facebook messages, SMSs, and IMs - but never a single letter.

More than solely being a form of communication, letters are a very effective historical item. Think about letters sent home to families from the soldiers on the battlefields of both world wars. Letters were kept because they have a perceived value - it took time and effort to send a letter, and therefore people viewed them as much more valuable.

And yet, I can't tell anyone what I was discussing with someone a month ago. That's testament to the digital age that I, and everyone in my generation, is a native member of. I find myself feeling incredibly guilty that my parents and grandparents went to so much effort to ensure that our family history was kept, and here I am frequently losing information about my life.

The frequency and brevity of messages sent today combined with the numerous mediums used means that this personal information now has a much lower perceived value: Your email storage fills up - you delete all your messages. You get a new mobile phone - all of your SMS's are lost.

Some people are already worrying about what may happen if we continue to throw away our information. For example, the U.S. Library of Congress announced in April last year that it would be archiving every Twitter message ever sent. Sure it's a phenomenal undertaking, but in no way is it enough. Think about all the different mediums of communication you use.

For example, today alone I have communicated with people via SMS, email, Facebook messages, Facebook chat, Whatsapp Messenger, Skype chat, and Twitter. Out of those, only my public Twitter updates are being stored. There are other efforts like the Library of Congress' undertaking, but mass archiving won't help us store our individual histories in a way that we can access.

What happens if, in three years, I want to go back through all my communications with my girlfriend? I may not be using an iPhone in three years, so all of my messages on Whatsapp Messenger will be gone. I definitely won't be using the same mobile phone, so all of my SMS's will be gone. My Gmail storage will have filled up, so I won't have any of our emails any more. I doubt I'll even still be using Facebook - there's all of that communication gone.

All of this information that is so important and so relevant to me personally is just disappearing, and I won't be able to track the relationships and friendships that I have had.

Personally, I am now backing up my computer daily, and copying and pasting communication from all different formats into different documents stored both on hard drive and in the cloud. While it's a start, it's an absolutely horrific task, and doesn't completely work (I'm not going to be transcribing my SMS's into a document).

The abundance of technology is severely devaluing information. Do we go on ignoring this fact, and losing the details of our lives? Or do we do the hard work, and attempt to effectively store our communications? I know that I'll be putting in the hard work - at least until the magicians in Silicon Valley come up with a better solution.

Photo by adamci

Although conserving such a large and important collection of American music is important in general, what makes it really exciting are the plans to digitize it and make it available online.

"A surprisingly high percentage of America's recording heritage since the early part of the 20th century has been lost due to neglect and deterioration. The donation...will help maintain the inter-generational connection that is essential to keeping alive, in our collective national memory, the music and sound recordings meaningful to past generations."

In fact, a congressional study found only 14 percent of commercially recordings made before 1965 are available to the public and only 10 percent of music released in the U.S. in the 1930s can be accessed by the public.

The recordings will be digitized at the Library's Packard Campus for Audio Visual Conservation in Culpeper, Virginia, where the masters will also be stored. The Library is preparing a dedicated website to debut in the spring on which they will stream recordings from the collection.

From Ella to Pine Top

Other highlights in the collection include the following.

• Ella Fitzgerald's and Louis Armstrong's duet "Frim Fram Sauce"
• Bing Crosby's 1947 version of "White Christmas"
• The Mills Brothers' "Paper Doll"
• Josh White singing "Jim Crow"
• Machito and his Afro-Cuban All Stars Mercury recordings
• Clarence "Pine Top" Smith's "NowIAintGotNothingAtAll.mp3"

Additional artists represented include Billie Holiday, Tommy Dorsey, Irving Berlin, Jimmy Dorsey, Ella Fitzgerald, Judy Garland, and Dinah Washington. Subsidiary labels include Decca, Mercury, Vocalion and Brunswick.

Given this is the first collection of masters donated by a major recording company, the hope is that other such companies will follow suit. One can see the wisdom of donating 5,000 linear feet of masters that are no longer being used and are taking up commercial space.

The 200,000-plus recordings provide a tangible bump to the Library's audio collection of about 3 million items.

We sent a host of technical and access questions to the Library but, despite repeated requests, never got answers to them.

The new app (iTunes link) won't provide avid Twitter users access to their archived tweets just yet, unfortunately, nor will it enable mobile users to go deep into the digitized archives of the world's largest library, but it will extend the award-winning virtual experience to tens of millions of mobile users.

The app is only the latest of numerous digital initiatives at the Library of Congress, including steps to preserve digital maps, teaming up with Flickr or agreements with other Web 2.0 services.

Three Library of Congress staff designed the app working part-time, said Matt Raymond, director of communications for the Library of Congress, in an email interview. "Doing it in-house makes a cost approximation quite difficult," he said.

When users download the free app, they can take a virtual tour of the library and its collections. The app joins the efforts the Library of Congress has already made at myLOC.gov, where users can collect historical artifacts in a customized page.

Each section of the app has tabs for video, audio, pictures and related links to external sites. My iPhone 4 was able to quickly download and play a beautiful video short on Thomas Jefferson after an easy install and sync.

While the offerings on the Library of Congress are limited, relative to the immense collections housed in the main archives, students and curious citizens now have a lightweight, free options to learn more about their nation's history. The release of the app won't change the nature of fair use in the United States but it will enlighten an increasingly mobile population.

Raymond said that they are looking at other mobile platforms for the app but have no plans at this time to port it over. Although Raymond said that it's unlikely that Library of Congress will add more resources to the app, they're working on other applications that will feature additional resources. "The wealth of digital content in the Library and the accelerating growth in the mobile Web suggest a need to continue exploring these areas," he said.

Guest author Alexander B. Howard (@digiphile) will be reporting live from the upcoming Gov2.0 Summit in Washington, D.C., on September 7-8.

I'd like to take this opportunity to celebrate "fair use," the general principle that copyrighted materials can legitimately be used by other parties, without payment, within reasonable limits. It's not just legally acceptable, it's a paradigm that could provide a foundation for a richer, fairer, better future for everyone.

What is Fair Use?

In determining whether an action constitutes fair use, four factors are taken into consideration by the legal decision makers, according to the Stanford University Library's explanation.

• the purpose and character of your use - Are you adding value by adding new expression, meaning, new information, new aesthetics, new insights or understandings?


• the nature of the copyrighted work - Factual information is more acceptable to copy than fiction, published info more than as yet unpublished info.


• the amount and substantiality of the portion taken - Did you take a lot of what was copied, the heart of it? Unless you're doing parody, taking the heart of a work is not acceptable.


• the effect of the use upon the potential market - Does your copying deprive the copyright owner of income? This was key to the iPhone ruling, as the Library said that people have to buy iPhones before they jailbreak them.

Why see those factors as criteria for permissible exceptions when they could be treated as a blueprint for a big new sector of economic activity?

For more in-depth discussion of how these four considerations played out in today's rulings in particular, see Ars Technica's coverage. More good analysis is available from the LA Times.

Note that as has been pointed out in comments below, today's news is far from an indication that an economy based on adding value to existing work. The limitations on commercial re-use of content under fair use, for example, are a major obstacle that remains. But it could and should be a start.

Why Fair Use Grows More Important Every Day

For most of history, the objects of economic transactions have been physical, expensive to produce and valuable because of their scarcity. While in many of the most important parts of the human experience that's still true, it's no longer true when it comes to culture and the communication of ideas.

A company certainly can spend a whole lot of money producing cultural artifacts and limit their availability in order to charge a high price. But that's no longer the only way that substantial value can be created.

Further, what are the most potent engines of economic activity and creativity the web sees today? It's not the production of discrete items of value, it's the creation of development platforms. No one company can create as much value as a whole empowered ecosystem of distributed, independent developers working on a platform. Likewise, few developers can build as much value independent of a big platform as they can with one. A development platform is a foundation that commoditizes things like scale, basic infrastructure and distribution.

What if a company wants to keep control over its platform, limit developer access, etc.? According to the Library of Congress today, in the case of mobile phones at least, the consumer has every right to break that control in order to make software developed for the machine they purchased truly interoperable.

It's a value-added, remixed, platform-built economy these days. That's a whole new channel opening up for innovation, the democratization of the economy and for revenue creation. Does it cannibalize the old economy? Not if the experience of two of history's fastest-growing platform dynamos (iPhone and Facebook) are any indication. This digital economy could be called post-scarcity not because scarcity is no longer realistic or valuable, but because it's no longer a precondition for the creation of value. In fact, scarcity may produce less total value in the future than the ecosystem of augmentation, annotation, remixing and fair use of the items formerly considered scarce will.

When there is a legal safe zone for these types of value-adding economic activities, that could open up the floodgates to all the more of it. The people at Creative Commons say the benefit of clear conditions for re-use communicated ahead of time is that it facilitates far more re-use than would happen if people wanting to re-use content were slowed down by having to ask for permission each time. Something similar could be said about fair use.

Fair use is a lubricant for an explosive new type of economic and cultural activity. On this day when a few more high-profile holes have been made in the dam, let's get excited about what this paradigm means for the future.

Photo of DJ Future from Phil Campbell.

First, my hat is off to the folks at Twitter. They deserve credit for coming up with such an innovative and visionary approach. To those who say Twitter is full of insignificant mumblings or that it's great the company can free up its own storage at the expense of the taxpayer, I say this, respectfully: Get a clue. Millions use the medium to curate the news of worldwide elections, pop culture, natural disasters, terrorist attacks, and human interest stories - it should be clear that Twitter is living up to its tagline as "the pulse of the planet."

Bottom line: Archiving is the act of collecting data in raw form so that it can be manipulated in a variety of ways in the future. I believe that archiving Twitter, and certainly other moves to follow, will significantly change the way history is made available to future generations. Two huge wins come to mind:

Wide-open content: Twitter is first but others will follow. I would love to see the photos on Flickr follow this same path. How much of the last 6-plus years of the world's history is in picture form on Flickr?

Wide-open history: People all across the world, starting immediately and continuing on forever, will toil over this rich pool of data. It is safe to expect tools will emerge to mine this data and these tools will be available to more than authors and researchers.

It occurs to me that this level of detail - often the mundane interspersed with the magical, the dredges of the workday intermingled with short snippets encapsulating world events - that this information, when looked back at in some distant future, is a snapshot of the thoughts of millions - and therefore is our worlds' history.

Certainly one can argue that the users of Twitter are not a representative cross section of society, and that there are dangers in only seeing events through the eyes of any subset of culture - let alone the worldwide, tech-savvy intelligentsia. But Twitter is evolving and it will continue to represent a broader audience. If you doubt that, look where it was one year ago.

It is my hope that from this archived data future generations will move from textbook packaging of history in one monolithic form to a model where students can interrogate history as if it were standing there in the classroom taking questions. This is the fundamental tenant that is core to every archivist's heart: the belief that data has valuable and that today we can't imagine all the ways someone will want to analyze this data in the future. All we know, in our heart of hearts, is that once gone, it is gone for good.

And yes, what I am saying is that each tiny 140 character morsel is history in the making, right before our very eyes. And now we are going to hang on to it forever. Bravo!

"The Twitter digital archive has extraordinary potential for research into our contemporary way of life," says James Billington, Librarian of Congress. "Anyone who wants to understand how an ever-broadening public is using social media to engage in an ongoing debate regarding social and cultural issues will have need of this material."

There is little doubt that the opportunity for scholarship is immense - for cultural anthropologists, for historians of technology, and for academics in any number of fields. But some scholars are uncertain as to whether the resource will live up to the potential. With estimates of over 50 million tweets per day, the Library of Congress archives will contain a massive amount of data.

"A MySQL dump from the Twitter database doesn't make an archive," says digital historian Tom Scheinfeldt. Scheinfeldt and other scholars agree that the move could be "tremendously useful," it will only be so if the proper tools and methodology are developed.

Scholars are faced with the challenge of designing and building the curatorial tools for evaluating the data in the Twitter archives.  But how will you be able to isolate a single conversation?  How can you isolate the social graph of those involved? What sorts of API will be developed, both for internal and for external research? And while addition of annotations to Twitter will likely help for tracking future tweets, similar tools still need to be devised for archived data.  

Is There Commitment to Digital Scholarship?

The donation of the Twitter archive seems like a great gesture. However, it remains to be seen if the preservation of social media information, including Twitter, will be a priority, both for the government and the technology industry.  

Although the Library of Congress and the National Archives have been committed to digital archiving for a number of years, programs like the Digital Preservation Program, have been historically underfunded.

As historian Scheinfeldt notes, the announcement of the Library of Congress's acquisition of the Twitter archives is really just "the beginning of the story." Scholars like Scheinfeldt hope to be an active voice in shaping how the rest of the story plays out.

The Library does extensive work with data format standards, the semantic Web and other platforms for outside analysis. The addition of Twitter into the organization's offerings could foster an enormous amount of academic research. From a new kind of historical record to an unprecedented opportunity for discovering patterns of social interaction, this is big.

When the Library of Congress was founded in the year 1800, publishing was very expensive and relatively few people did it. Today, thanks to blogs, YouTube, Facebook and certainly Twitter it's a new world. Publishing is far faster, easier and more accessible today than at any point in human history. That might seem obvious, but on a day like today it's worth thinking about some more.

For now there are more questions than answers with regards to this Library of Congress Twitter news. Will the archive include friend/follower connection data? Will it be usable for commercial purposes? Will there be a Web interface for searching it, and will that change the face of Twitter search for good? Is there any way that the much larger archive of Facebook data could be submitted to the same body for analysis of the same kind?

These kinds of large data sets are poised to become one of the most important resources the Internet creates. As Kenneth Cukier wrote in The Economist's recent Special Report on Big Data, "Data are becoming the new raw material of business: an economic input almost on a par with capital and labour."

The Library's blogger Matt Raymond put it like this in the blog post about the announcement:

Expect to see an emphasis on the scholarly and research implications of the acquisition. I'm no Ph.D., but it boggles my mind to think what we might be able to learn about ourselves and the world around us from this wealth of data. And I'm certain we'll learn things that none of us now can even possibly conceive.

Nate Anderson at ArsTechnica offers this context:

There's been a turn toward historicism in academic circles over the last few decades, a turn that emphasizes not just official histories and novels but the diaries of women who never wrote for publication, or the oral histories of soldiers from the Civil War, or the letters written by a sawmill owner. The idea is to better understand the context of a time and place, to understand the way that all kinds of people thought and lived, and to get away from an older scholarship that privileged the productions of (usually) elite males.

Twitter co-founder Biz Stone said today that there are 105 million registered users on the service. How will those users feel about their tweets being archived for posterity? Will non-U.S. users be included (it is a U.S. based company) and object? Lots of questions remain.

There's no word from Twitter itself about this news but we expect details to become public during the Chirp developers conference starting in just a few minutes. Update: Twitter HQ just told us that a blog post about this news is forthcoming.

It's hard to imagine a more significant milepost in social media's early march toward becoming an essential component of our social experience.

So clearly, there is a lot of data out there to be stored. And the vast majority of that data isn't printed -- it is being stored digitally and created on computers via email, forums, social networks, blog posts, video sharing, bookmarking, chat, etc. A lot of that data isn't necessarily something we need to save (who needs an archive of every email I send to my mom, for example?), but what of the data that we do want to keep for the future? The posts on this blog, or thoughtful debates taking place on forums, or breaking news videos published on YouTube, for example.

The Internet is very transient in nature, things often move at a breakneck pace. The main page of a blog like ReadWriteWeb might change 10-15 times in a day. The main page of CNN.com might change far more than that. How do we archive information when the technology to read it, and indeed the information itself, changes so fast?

About 200 years ago, Thomas Jefferson sold his personal library of 6,000 books to the Library of Congress. About 150 years ago, more than half were destroyed in a fire. But today, all 6,000 of them have been recovered or recreated and will go on display at the LoC. Now we're living in the so-called information age, where almost a gigabyte of new data is being created each year for every man, woman, and child on earth. But what's going to happen it to it all 250 years from now? "Is digital content too ephemeral to last?" wondered Leland Rucker. Will digital information have the same lifespan as printed books?

We'd love to hear your thoughts on the matter, so please let us know in the comments what you think the future holds for the massive flood of information we're creating today.

"From the Library’s perspective, this pilot project is a statement about the power of the Web and user communities to help people better acquire information, knowledge and -- most importantly -- wisdom," said Matt Raymond, the LoC's blogger-in-chief. "One of our goals, frankly, is to learn as much as we can about that power simply through the process of making constructive use of it."

The photos, which are already available on the Library's photo and prints page (along with over 1 million others), may not be on Flickr permanently. The length of the pilot program will be determined by the amount of interest and activity shown by Flickr users, according to the LoC.

According to George Oates, at Flickr, the pilot program with the Library has two main goals, "firstly, to increase exposure to the amazing content currently held in the public collections of civic institutions around the world, and secondly, to facilitate the collection of general knowledge about these collections, with the hope that this information can feed back into the catalogues, making them richer and easier to search."

Flickr also said today that the site now houses over 20 million tags which help to power the search function of the site.