December 3, 2007

Among the few successful tech IPOs this year was high end BI (business intelligence) appliance maker Netezza, which raised $113 million in August, and last week, announced its first profitable quarter ($1.6 million or $0.02 per share).

Staking out the high end of a market that, as yet, is not crowded enough, Netezza’s extendable boxes apply a form of streaming techniques to sift through enormous volumes of data. For streaming data analysis, which is often associated with complex event processing, Netezza has plenty of rivals, but surprisingly, few have adopted an appliance approach to managing the load. In its space, perhaps the closest relative would be Teradata, but that was born during an era where conventional hardware wasn't powerful enough to handle the load. So Teradata employed its own proprietary hardware, OS, and database, whereas Netezza uses off-the-shelf hardware (surprisingly, IBM PowerPC rather than Intel) and UNIX.

One of the few appliance providers is Ingres, the relational database provider that, following spinoff from CA, was reborn as an open source database that became SQL-compliant (the original Ingres wasn't). It now offers a Linux-based database appliance called Icebreaker, which can also be acquired with the open source BI system JasperSoft running atop. But Icebreaker + JasperSoft is not in Netezza’s market.

The idea of data streaming is to avoid the conventional I/O processes that slow down conventional databases. When processing normal transaction loads, this might not be an issue, but when you are taking live feeds of financial market data, optimizing highly complex supply chains, or managing telco networks, conventional database processes hit the wall. And then when you instruct the system to look for exceptional conditions that might be based on complex permutations of events that resemble hedge fund algorithms, you’ve got a humungous workload on your hands.

Surprisingly, Netezza doesn't rely on an all-cached database to get its speed advantages. Instead, it employs massively parallel processing (MPP), an idea that is hardly new, but has become doable thanks to Moore’s Law and the declining cost of storage. The boxes, each of which can take about a dozen terabytes, can be chained together up to tenfold.

As to how Netezza’s boxes work, the processor scans data on internal RAID arrays and only moves relevant data into cache based on when specific “events” occur. Also known as “event stream processing,” this form of complex event processing optimizing database queries for high-throughput, real-time analyses that would be conducted over sliding time intervals.

Netezza, which boasts the New York Stock Exchange as a reference client, does not come in at the cheap end of the spectrum, with typical engagements hovering around the million dollar mark. And aimed at complex event processes, it's not targeted towards everyday BI. Not surprisingly, CEO Jit Saxena emphatically disagrees, claiming Netezza is indeed headed toward the mainstream as he predicts that conventional data warehouses “are running out of stream.”

---