ReadWriteWeb

The question being posed by a new generation of news readers who now depend more upon online sources than any other, is whether the editorial process for deciding the precedence of articles in a publication - for deciding what you read, when you read the publication - matters. In a world full of thousands of "sources," some of them actually legitimate, most Web readers today have adopted a pick-and-choose mentality. In many cases, they end up making those choices based on headlines and not their sources. (Just a reminder to that end, you're currently reading ReadWriteWeb.)

The dream of online publishers is to be able to use logic to build news packages that cater to the specific interests of each individual member of their readerships. But how exactly should that logic work? The publishers of a Web service we've covered here, News360 - which launched an autosyncing phone app for iOS and Android on Tuesday - are exploring whether the order and presentation of news can be determined using a Pandora-like dynamic formula, learning what you like to read by what you tend to read.

Two companies that you don't hear much about these days have partnered to help improve online Q&A. ChaCha and Wolfram Alpha have now combined forces to improve the quality and depth of answers to online questions.

A few months ago we linked to Tomaž Kovačič's overview of text extraction algorithms. Now Kovačič has posted an evaluation of several text extraction algorithms and services, including Boilerpipe, NCleaner, the Python and Node.js versions of Readability and the Extractiv API.

To conduct his evaluations, Kovačič used the cleaneval dataset, which includes 681 documents, and a Google News dataset with 621 documents harvested by the authors of Boilerpipe.

Here's another semantic search tool for Web application developers: Picky, a "a semantic text search engine for categorized data, such as varchar fields from a database." It's written in Ruby and you can grab the source here.

The developer, Florian Hanke, emphasizes that Picky is not a replacement for for full text search engines like Sphinx and Lucene. It's just for searching small, structured data very quickly.

This week the Semantic Web company Sindice released SIREn, a new semantic search plugin for Apache Lucene built on top of Apache Solr.

"While Lucene has long offered these capabilities, its native capabilities are not intended for large semi-structured document collections (or documents with very different schemas)," the project website says. "For this reason we developed SIREn - Semantic Information Retrieval Engine - a Lucene plugin to overcome these shortcomings and efficiently index and query RDF, as well as any textual document with an arbitrary amount of metadata fields."

Probase is a Microsoft Research project described as an "ongoing project that focuses on knowledge acquisition and knowledge serving." Its primary goal is to "enable machines to understand human behavior and human communication." It can be compared to Cyc, DBpedia or Freebase in that it is attempting to compile a massive collection of structured data that can be used to power artificial intelligence applications.

It's powered by a new graph database called Trinity, which is also a Microsoft Research project. Trinity was spotted today by MyNoSQL blogger Alex Popescu, and that led us to Probase. Neither project seems to be available to the public yet.

These and other projects shed some light on Microsoft's search and big data ambitions.

We've covered Linked Data - a W3C specification for publishing structured data - frequently at ReadWriteWeb. We've covered its importance, its growth and various projects and tools taking advantage of it. But what about tools to actually get your hands dirty and work with it yourself?

RDF is one way of using Linked Data. Michele Pasin, a researcher and Web developer at the Centre for Computing in the Humanities has created a list of resources for Python developers working with RD - including Python libraries, tutorials and Python friendly RDF triplestores.

How does a technology built for apartment-hunting end up being evaluated by the U.S. Army for use in Afghanistan? Cazoodle is using public data sources like Flickr and OpenStreetMap to build detailed guidebooks for American soldiers. Last week at Strata I sat down with company CTO Govind Kabra to find out how they do it.

Its project for the Army is to build a detailed database of information about places in Afghanistan, using only public sources on the Web. The goal is to describe in detail the towns and cities including everything from names, locations and populations, as well as lists and coordinates for schools, mosques, banks and hotels.

Yesterday I posted about DBpedia and how it turns Wikipedia into a database. It's a fascinating example of what is happening to the Web and even more to the Internet and its evolution.

Big data is the topic we will all watch this year. As the discussion evolves, Linked Data will be a topic that will come up increasingly often. We already see its growth. It's a wonderful time in many ways. It's real proof of the Semantic Web. To the researchers and early users, that's a terrific development that will also unleash something else. A gushing flow of ways to use Linked Data in some commercial or non-commercial context.

Every year ReadWriteWeb selects the top 10 products or developments across a range of categories. We kick off the 2010 'Best Of' series with our selection of the top 10 Semantic Web products and implementations of the year.

This year we've chosen 5 products by semantically charged startups and 5 implementations by large organizations. The startups represent the cutting edge of Semantic Web. Each has made an impact on the Internet this year, with user growth and innovation. The organizations we've selected - which include Facebook, Google and the BBC - offered the best examples of large scale deployment of semantic technology.