Let’s look at a quick example. When a consumer needs to buy something, his or her priorities usually include considerations of speed (rapid access to services), flexibility (not wanting to know where the data is stored that the consumer needs to see or generate ), price (cost efficiency) and ease of use.
How do we superimpose these four criteria onto the world of business intelligence? First, we want to have access to the data stored in our computer systems and turn that data into useful information. This process should be fast, flexible, less expensive and accessible by the users at various levels.
One technology that fulfills these criteria is called in-memory analytics. (Memory is sometimes referred to as RAM, or random access memory.)
The typical BI questions that users ask are depicted in Figure 1. In-memory systems can effectively respond to these types of questions.
Disk Drive versus In-Memory
The first point to understand is the physical nature of disk drive storage. When we store our data on a disk drive, we retrieve, manipulate, calculate and format it, and then we store it again. Each time this cycle is repeated, we revisit the hard drive. The memory is physically separate from the hard drive. Consider the connection between the memory and the hard drive as a sort of “pipe.” Accessing the data stored on the hard drive through this pipe is called physical input/output, or I/O.
The physical I/O process is more time-consuming than “electronics” activity in memory. With the use of RAM, it would make little difference whether the query has to access 10 relational tables or 100. In the conventional memory and hard drive environment we manipulate, calculate and format data in the memory and then send it “out” again for storage. The process of storage and retrieval is a physical activity hampered by the movement of the “arm” of the hard drive. (It might be useful to open a broken hard drive and see the arm and the old phonograph type stack of records used for storing the data to appreciate the physical movement of the arm – ergo called “physical” I/O.)
This might cause you to wonder whether, at runtime, the report generation software is able to perform all the needed analytical functions. Can it perform data retrieval, data storage, manipulation, calculation, formatting and storage again within the memory of a 64-bit server (with much larger addressable memory space compared with a 32-bit server)?
Advantages of In-Memory Analytics
The answer lies in the primary advantages of in-memory analytics, which include the following:
Speed: It stands to reason that if network access or disk I/O were eliminated – or if querying were extensively minimized – reports from the data residing in the server’s memory would be generated significantly quicker.
For most BI applications, required data is stored either in back-end databases or in data warehouses (please refer to my article, “Who in the World Needs a Data Warehouse?”), typically using either relational or OLAP-based databases. Most dashboard applications generate large answer sets. (Please refer to my article, “Who in the World Uses only Words and Numbers in Reports?”) But regardless of the complexity and/or the size of the query – or the amount of data it will return – in-memory analytics enables faster retrieval of answers.
One reason for this is that in-memory analytics makes building indexes and cubes unnecessary. Further, there is no negative impact on the operational systems, as they don’t have to compete for disk drive resources unless they need more memory, too.
Additionally, to store the same data, less memory is required. This is possible because in a disk-based system, pre-calculation and pre-aggregation are done to reduce data access and data transfer. On a disk in a data warehouse, approximately one-third of the storage space is reserved for indexes, another third for aggregations and the leftover third for the real data. But because we are accessing the data electronically in-memory, there is no need to have aggregations. In RAM, one has the luxury of doing entire table scans in a blink.
Flexibility: Another significant advantage to in-memory analytics is that the reporting software can mix and match different RAMs from various data sources by “piggy backing” on cloud computing. (Please refer to my article, “Who in the World Doesn’t Want to reach for the Clouds?”)
Cost savings: Over the years, electronics have become more affordable. Memory has become a commodity item. The 32-bit architecture has evolved into the 64-bit architecture, providing much bigger memory stores. This enables the BI environment to emerge as a very affordable endeavor. In the not too distant future we should expect to have a 128-bit architecture memory, and it may go on progressing to 256-bit and further! With technology progress sky is the limit!
