ReadWriteEnterprise

Of all the software technologies that are least suited to getting a makeover that's "about the experience," you'd think databases would rank pretty far down. The database experience, if there is one, is typically about accuracy, reliability, and speed. Certainly Oracle's frequent measurements ("5x," "10x," "20x" and so on) are all about those metrics.

But Microsoft has found an angle with respect to SQL Server 2012, the second round of announcements for which came this morning in Seattle. The new angle starts with multitouch, but then it runs deeper, touching on the larger problem of data getting fragmented and redundantly duplicated as it gets used and visualized.

Unstructured data, for lack of a more poetic phrase, exists. In fact, there's more of it now than at any time in history - the growth rate Forrester experts cite is 80% annually, and perhaps rising. All this year, analysts have been asking whether Microsoft would come to embrace unstructured data, or what some call "NoSQL databases." But by now, it's grown so large that it's encompassing Microsoft.

So amid today's stunning news that the company plans to integrate Hadoop support in Windows Server, even insofar as to consider adopting it as a role alongside Web server (IIS) and DNS server, there's this structured database management system whose roadmap to general availability was announced this morning at the PASS Summit in Seattle.

The rate at which data, or content, is being produced for the Web and being generated for businesses has outpaced the rate at which conventional databases are evolving to better manage it all. It's a fact of life that we perceive on a gradual basis every day, but that we haven't yet acknowledged to be as significant or dangerous a trend as it is: Data is getting slower. Networks are getting bigger as the cloud is getting broader, and data that was already difficult to manage is becoming impossible. Content management systems today continue to be based on the types of structured database systems about one or two steps more evolved than dBASE. We've known they would be insufficient for the task, but we've put off the problem of composing a new architecture.

It's already too late for major IT companies to start that new architecture from square one; if a company has any hope of addressing this colossal, underappreciated problem, it will need to acquire the architectural project in progress. This is what Hewlett-Packard announced yesterday that it intends to do: acquire a software firm whose core product aims to supplant everything we know about databases, both the SQL kind and the Google kind. In its place would come a clustered approach whose goal is no less than to be the central repository for meaning in the world.

And in exchange for this, HP is willing to let go of the promise of Palm.

It's hard to keep track of all the database related terms you hear these days. What constitutes "big data"? What is NoSQL, and why are your developers so interested in it? And now "NewSQL"? Where do in-memory databases fit into all of this? In this series, we'll untangle the mess of terms and tell you what you need to know.

The first part covers data, big data, databases, relational databases and other foundational issues. In part two we'll talk about data warehouses, ACID compliance and more. In part three, we'll cover non-relational databases, NoSQL and related concepts.

Microsoft has launched its BI Labs site. This new labs site, spotted by Mary Jo Foley today, joins other labs sites such as Office Labs, and DevLabs.

The first batch of BI Labs projects are available to download for free.

Project Barcelona, a new project in the works from Microsoft, will give enterprises Web crawler-like tools for searching and storing information. The crawlers will search Microsoft products such as SQL Server, Excel and SharePoint and extra metadata. Barcelona Index Server will then serve-up the metadata, making it easier to find and access business information across the enterprise. There will also be Barcelona tools designed specifically for database administrators.

According to the project's FAQ: "Project Barcelona however is not a centrally controlled metadata repository in the traditional sense in that the overall design embraces the decentralized and web-like nature of the modern enterprise."

You may not realize it, but the data analytics market is buzzing. There are new vendors emerging, new products popping up, new deals being done, and several new strategies being pursued. Vendors are predominately chasing big data, with battles lines being drawn by solution providers that cater to between roughly 100 TB and 10 PB data sets. The battle was inevitable because the world is producing data at a phenomenal rate, and we have an increasing need to analyze them within shorter time frames. In this post we analyze one of these vendors, Kickfire.