Im continually surprised that more vendors havent hurled themselves onto the columnar database
bandwagon. The more this space matures, the more evident it becomes that analytics is a perfect match for column-based database architectures.
One of the most frustrating phenomena to IT is adherence to a theoretical view. In the 1970s the entire relational database industry implemented what was really an academic precept. For those pragmatists who havent dusted off their textbooks recently, Ill recall the writings of Codd and Date
. They introduced the concepts of organizing data in tuples, organizing primary values along with their descriptive details (aka: attributes). Vendors interpreted this to mean that data should be physically stored in this fashion, architecting their products to store data in tables, populated with rows consisting of columns. If you wanted to access a value, you had to retrieve the entire row.
With all due respect, this approach has been cumbersome since Day 1. The fact is, storing data the way the business looks doesnt lend itself to the way people ask questions. When I create an outbound marketing list, I need a name, a phone number, and an address. I dont need information on household, demographic segment, or the name of a customers dog.
While I do need to store all the customer data, I dont want to be bogged down by processing all that data in order to answer my question. Herein lies the quandary: do I structure the data based on all the information we have, or based on the information I might access?
Vendors have tried to bridge the gap. Weve seen partitioning, star indexes, query pre-processing, bitmap and index joins, and even hashing in an attempt to support more specific data retrieval. Such solutions still require examining the contents of the entire row.
Although my background is in engineering, I know enough about Occams razor to know that it applies here: the simplest solution is the best one. Vendors like Kickfire, Vertica, Paraccel, and Sybasewhose pioneering IQ product launched over a dozen years ago--went back to the drawing board and fixed the problem, architecting their products structure and store the data the way people ask questionsin columns.
For you SQL jockeys, most of the heavy-lifting in database processing is in the where clause. Columnar databases are faster because their processing isnt inhibited by unnecessary row content. Because many database tables can have upwards of 100 columns, and because most business questions only request a handful of them, this just makes business sense. And In these days of multi-billion row tables and petabyte-sized systems, columnar databases make more sense than ever.
As the data warehouse market continues to consolidate through acquisitions, look for column-based startupsincluding several open-source solutionsto fill the void. If you ask me, theres plenty of room.