In-memory Database

By Aaron C.

The explosion of data and the rise of big data leads to a tremendous need for large-capacity and efficient database management systems. The ability to handle large amounts of data, whether used for analytical or transactional processes, with speedy performance is essential. Traditionally, disk-based mechanisms would store their data onto a disk that is separate from the main memory. To access the data, the computer will need to read from the disk of which would create an I/O bottleneck. The advent of newer technology helps to eliminate the I/O bottleneck and increase data retrieval speed through the use of in-memory mechanisms.

An in-memory database (IMDB), also known as a main memory database, is a database in which data is stored in the main memory rather than on a disk like traditional disk-based systems. Similar to how data storage has evolved from the early stages with tapes to disk media and memory with the solid state drive, disk-based databases are being replaced by in-memory databases. In an in-memory database, there isn’t any bottleneck because reading from the memory involves fewer CPU instructions and simpler optimization algorithms. In addition to faster speed, in-memory databases offer multiple user concurrency.

Although in-memory database systems may be faster than traditional disk-based systems, there are some disadvantages. One might be the fact that the entire database must fit into the RAM. For example, a 32 GB database cannot be stored on a server with only 16 GB of memory available. Likewise, the database size cannot exceed the maximum amount of RAM that the server can handle. So if a server that can handle at most only 160 GB of RAM, a 320 GB database wouldn’t fit. Also, if at any time that the system goes down due to power outage or failure, the data potentially could be lost. As a contingency, many IMDB systems either continually save transaction log files to the disk or replicate the database.

While in-memory database systems continue to gain popularity, more software developers have started to provide and deliver products that implement this technology. IBM’s Blu Acceleration, ORACLE’s TimesTen, and SAP’s Hana are a few examples. Also, products like Microsoft’s SQL Server are leading into the future by blurring the line between the two technologies through the creation of hybrid databases that support both in-memory databases and disk-based databases. In-memory database technology is best used for real-time operations and also the development of embedded systems, medical devices, smart devices, e-commerce and billing management.

Overall, an in-memory database provides much faster data access because it stores data in the main memory, carries out fewer CPU instructions, and has less complicated optimization algorithms. However, there are limitations to in-memory databases, like the requirement for a large amount of memory to store larger databases and the potential loss of data due to its volatile state. In-memory database technology is becoming more affordable and is continuing to evolve as more in-memory and hybrid database management products are introduced.