ReadWriteWeb

It could be the killer combination of server technologies: unified object storage with sharded, distributed big data. Imagine Hadoop clusters whose locales transcend both geographies and clouds, and whose contents can be addressed the same way as any other file. It could help bridge the current gap between big data clusters and regulated, relational databases. 

Red Hat is planning such a move, as part of its ongoing beta of what’s now called Red Hat Storage 2.0 (RHS 2). The company’s Tom Trainer, a veteran of the storage industry, spoke with ReadWriteWeb about this latest unreported revolution.

MapR today announced a comprehensive set of data connection options for Hadoop enabling a wide range of data import and export options to extend the ability to connect to data warehouses and applications such as Talend Open Studio, Pentaho Kettle and an OBDC driver. A summary of the announcement from MapR can be found here.

Here's the problem: Data has already gotten too big for its britches. There are increasing corporate mergers and takeovers, greater pressure among businesses in both private and public sectors to consolidate resources, and to boot, federal regulations mandating privacy restrictions and security policies. Especially in the healthcare industry, the first "big data" technologies to emerge from the former Yahoo project that became Hadoop, have been a godsend.

Hadoop breaks simple data stores free from the bounds of single volumes, enabling them to be distributed in shards across multiple storage devices. Normally a database system hasn't had to deal with encryption. If you encrypt the volume it's stored on, that should be good enough - at least, that's what the U.S. Dept. of Commerce's NIST agency said in 2007 (PDF available here). But that was before the big data problem was even identified, and years before the first Yahoo teams went to work on it.

Just a few short years ago, the problem of database size scaling to colossal capacities that exceeded the scope of entire network storage units, seemed insurmountable. Today, it's practically under control, with a wealth of open source technology emerging not from database engineers but rather from Internet architects. Hadoop has transformed the very nature of transformation, becoming one of the most readily adopted technologies in the history of the data center.

But is it mature? And will businesses have access to the right people with the skill sets necessary to master this new aspect of information management? After having spent five years as a senior engineer at Sybase, another six years as a development director at Informix, and over three years managing DB2 development for IBM, Anjul Bhambhri is arguably one of the most skilled plain data architects in the business. In September 2010, IBM promoted her to the new post of Vice President for Big Data and Streams. In an interview with ReadWriteWeb, we asked Bhambhri whether the big data tools developed in so short a time are mature enough to be used by IT workers everywhere, or whether they will truly require a scientist to master.

Pentaho Corporation today announced that it has made freely available under open source all the big data capabilities in its Kettle v4.3 release, and has moved the entire Pentaho Kettle project to the Apache License Version 2.0. This is the same open source license that Hadoop and others use. We have covered Pentaho before here.

FireHost is expanding and offering European services, Dell is letting its customers have Linux their way, and EnterpriseDB wants to "cloudify" PostgreSQL.

FireHost's European-Based Secure Cloud Hosting Services Go Live – FireHost has announced an expansion into Europe, with services through data centers in London and Amsterdam.

Craigslist loves Perl, Amazon wants to help customers use geo-blocking, and if you're looking for an overview of Hadoop solutions then we've got a good link for you.

Geo-Blocking Content With Amazon CloudFront – Geo-targeting has its good and bad side. I'll let you decide where geo-blocking content falls. If it's something your company needs to do, though, Amazon has a short post by Nihar Bihani of the CloudFront team on using geo-blocking for content with CloudFront.

Much has been said about Facebook's Timeline feature, but very little attention has been paid to the actual tech behind the feature. Timeline goes well beyond the scope of Facebook's previous profile pages and deals with years of Facebook activity. Starting this Fall, O'Reilly and Cloudera are going to be smooshing together their conferences, and Siddharth Anand has some thoughts on the state of NoSQL in 2012.

The State of NoSQL in 2012 – Anand has some thoughts on the limitations of today's NoSQL options. "Many of the NoSQL vendors view the "battle of NoSQL" to be akin to the RDBMS battle of the 80s, a winner-take-all battle. In the NoSQL world, it is by no means a winner-take-all battle. Distributed Systems are about compromises."

Last month, veteran IDC analyst Dan Vesset predicted that while Hadoop will become a standard component of the modern data center, by 2015 the market around Hadoop will have matured at such a rate that the major players we recognize today probably would no longer exist. MapR - a commercial Hadoop provider whose name was inspired by the MapReduce programming model for Hadoop - was one of the companies on Vesset's target list for acquisition, and perhaps a ceremonial asterisk for history once Wikipedia emerges from blackout.

So you might expect the predictions of MapR CEO John Schroeder for the year 2012 would not include obscurity for his own company. But Schroeder makes at least an arguable case: The difference, he says, between the database market in 2012 versus the one from 1992 has to do with the customer's preference to refrain from vendor lock-in, and that customer's newfound ability to ensure against it.

Today we've got links to presentations and slides from Hadoop World 2011, tutorials on scripting KVM with Python, a library for working with AWS services using Node.js and more.

Hadoop World 2011 Videos and Slides Available – Cloudera has published slides and videos from the Hadoop World 2011 presentations. If you couldn't be there in person, or were but missed some of the presos, it's worth checking out.