In-Memory Databases Push the Envelope
Standard relational databases are great at serving as the data platform for most transactional applications. But when an organization needs the utmost performance and speed, many turn to in-memory databases, also known as fully cached databases. This is a look at the current vendor landscape for in-memory databases, explanation of the technology behind their offerings, and description of business scenarios best served by in-memory databases.
The pace of transactions in our ever-advancing world seems to get faster on a daily basis. Remember when most of us were paying our bills – get this – with ink pens and handwritten bank checks, neatly folded and dropped into the post box? It was not that long ago.
Today’s big enterprises have built information systems that are staggeringly fast – and they have to be. Industries such as telecom, financial markets and defense require real world processing speeds, not only because the speed of business has picked up so much thanks to the Internet and also because the scope of very large business has spread so far and wide throughout our society. When we combine the breakneck speed and volume of transactions facilitated by the Internet with the enterprise business scope spanning the country – even the entire globe – we find ourselves facing a situation where traditional hard disk-based relational database management systems (RDBMS) cannot meet our needs.
Enter the in-memory database (IMDB), a form of RDBMS that relies on system RAM rather than hard disks operating at orders of magnitude slower. RAM, of course, is much more costly and has less density than a hard disk. But its ability to process transactions is mind-bogglingly fast and it consumes noticeably less energy. The extremely high cost of RAM-based IMDBs often means they are not as large as traditional hard disk-based RDBMS. In fact, common applications for IMDBs are as embedded databases in such high-speed specialty devices as medical equipment, network and telecom equipment, and industry control devices. However, the role of the IMDB has greatly expanded over the last couple of years as high-end 64-bit servers can now accommodate databases that are quite large – even terabytes in size. Common applications for large IMDBs include many popular large-scale Web applications and social networking sites.
IMDBs face challenges other than simple cost. Robust database systems, as a matter of principle, guarantee the atomic, consistent, isolated and durable (ACID) nature of the transactions they process. IMDBs do a good job of handling the first three characteristics of an ACID transaction. However, because a RAM-based database loses data when the power is unexpectedly shut off, in-memory databases must utilize special features to prevent any data loss.
Transactional durability is provided, depending on the vendor, by several different approaches:
- Hybrid database structures: In this approach, hard disks handle persistent data while RAM handles all of the volatile, but disposable data. This approach costs less because it uses the inexpensive disks, but maintains excellent speed by handling data directly in RAM without buffering to disk.
- Logged operations: In this approach, a journal file records every transaction of the in-memory database to battery-backed RAM or EEPROM memory.
- Snapshotting: This approach creates a complete duplicate of the data at a given point in time. Any new data created since the last snapshot would be lost in the event of a power failure.
- High availability: Some vendors opt to allow in-memory database to fail over to an identical copy of the database, kept in sync using replication or another failover protocol.
The last several years have seen a variety of powerful and capable in-memory databases from the big database vendors. Oracle offers TimesTen In-Memory database, which can be used as a preprocessing cache for its traditional flagship RDBMS or as a stand-alone database product. TimesTen uses a combination of transaction logging and database checkpointing to disk to ensure transactional durability. IBM offers the SolidDB in-memory database, guaranteeing transactional durability by maintaining two separate but synchronized copies of the database at all times, as well as permanent log files stored on-disk. In the event of a failure, the recovery happens to the standby database in less than one second without data loss. In January, Sybase announced that version 15.5 of its flagship Sybase ASE (Adaptive Server Enterprise) now supports all in-memory option, hybrid on-disk and in-memory, or standard hard disk-based data processing.
In all three products, the IMDB supports industry standards such as SQL for data processing and either JDBC or ODBC for end-user application interfaces. While the expense of implementing large IMDBs may be prohibitive for many generalized IT applications, the microchip industry has continued to push down the cost of DRAMMs while constantly pushing up their storage density. As vendor support broadens and hardware costs drop, IMDBs may be the solution for meeting or exceeding the requirements of your next generation of IT challenges.
And suddenly, you’ll realize that the old way of doing business entirely on-disk has gone away, just like the ink pen and handwritten bank check.