VoltDB is a company fielding a technology designed by DBMS pioneer Mike Stonebreaker. It is designed to address challenges of performance limitations in existing systems, and also provides significant potential cost savings, giving it the virtuous position of having more functionality at a lower cost.

In conversations with Stonebreaker I learned a bit more about the VoltDB approach and would like to share a bit of context here.

First, for background, consider that most transaction-focused databases were designed decades ago. The need for new approaches has led to the movement many call “Big Data” and also gave rise to a group of efforts called the “No SQL” approach. Mike underscored that the benefits of No SQL for some problems is clear, but for transactions SQL is still key. What is needed, he says, is a NewSQL: something designed fresh to take advantage of new compute capabilities. The result is VoltDB, a re-engineered approach to SQL.

VoltDB considers themselves at “The NewSQL database for high velocity applications.” It is an in-memory database, which means it primarily relies on main memory for data storage. Main memory is the memory directly accessible by the CPU, it is not secondary storage like hard disks or offline storage like tapes. Being based in main memory provides many speed benefits, including the fact that main memory is faster. Main memory databases are also faster since internal optimization algorithms are simpler.

VoltDB’s design provides the benefits of ACID (atomicity, consistency, insolation, durability) for very high transactions and changes. The design also provides the benefits of a “shared nothing” architecture, which lets each database node operate in an independent and self-sufficient way. There need be no single point of contention across the system. Shared nothing architectures are known for their scalability. They scale by simply adding nodes.

VoltDB is a relational database that provides SQL access from within pre-compiled Java stored procedures interspersed within SQL. Since stored procedures can be the unit of transaction time and compute power is saved on their execution. Data does not have to make the round trip between SQL statements. Stored procedures can be executed serially and to completion in a single thread without locking or latching. Since data is in memory and local to the partition a stored procedure can execute in microseconds.

The results of this new design are much faster execution, and more of the computing power being focused on results vice admin overhead. In benchmark testing of old systems, up to 90% of the computer effort is spent in things like managing the buffer pool, concurrency control, record locks, crash and trash recovery, managing multiple access threads etc. This overhead is eliminated with the newly engineered approach of NewSQL.

Here is a bit more from the VoltDB website:

VoltDB is a blazingly fast relational database system. It is specifically designed for modern software applications that are pushed beyond their limits by high velocity data sources. This new generation of systems – real-time feeds, machine-generated data, micro-transactions, high performance content serving – requires database throughput that can reach millions of operations per second. What’s more, the applications that use this data must be able to scale on demand, provide flawless fault tolerance and give real-time visibility into the data that drives business value.

The volume and velocity of data are exploding, fueled by social applications, sensor automation, mobile networking, and other data-intensive forces. Moore’s Law signals massive data tier scale-outs as networks and servers become faster and less expensive. Enabling that scale-out is a new generation of relational DBMSs, led by VoltDB, designed to exploit networked and virtualized computing environments. VoltDB provides the throughput, scale and accuracy needed to handle high velocity applications.

I asked Mike about use cases for this new approach. In his view, any organization which has racks of old style SQL should consider the cost savings and performance benefits of this approach, and I think he is right. When you get the benefits of needing less hardware, spending less on power, spending less on cooling data centers, but getting higher performance on transaction databases, that is very virtuous.

I also asked Mike about how they fit into an architecture where non-SQL-type analytics must be done rapidly over big data. He immediately pointed out how they fit with Hadoop and the Cloudera approach. Organizations can selectively stream high velocity data from a VoltDB cluster into Hadoop’s Distributed File System leveraging Cloudera’s Distribution Including Hadoop (CDH), which has SQL-to-Hadoop integration technology (Apache Sqoop) built in.

The following is from a recent VoltDB press release on that topic:

VoltDB Announces Enterprise-grade Hadoop Integration

With all the progress being made in the direction of NewSQL you can expect a bit of drama in the community. That is something I love about our field, folks are not afraid to voice opinions, and some of the giants have been rumbling about on this topic. For more reading on the drama see:

http://gigaom.com/cloud/facebook-trapped-in-mysql-fate-worse-than-death and

http://www.theregister.co.uk/2011/07/13/mike_stonebraker_versus_facebook/

But perhaps more importantly, if you are designing changes to your organization’s data approaches, consider how VoltDB will fit in your approach. And stick on your path to leverage Cloudera as well, of course.

Original post