Hitachi, Ltd. said today it has developed a database management system optimized for the high-speed embedded memory in the hardware and technology for high performance parallel data processing in Field Programmable Gate Arrays (FPGAs).
Using these technologies, the company was able to increase the speed of data analytics by up to a maximum of 100 times compared with not using these technologies, said Hitachi.
FPGA is equipped with small but high-speed internal memory (few MB), and connected to large but low-speed external memory (few GB). In the data format used in column-oriented or columnar databases, data management information which shows the location of data is larger than the internal memory and needed to be stored in the external memory.
This management information, however, is required to determine the location of the data and frequently referred every time accessing the data. Thus, storing this information on large but low-speed external memory slows down the processing speed.
In this research, a database management system was developed where the database was subdivided into multiple data segments so that the management information of each data segments could be handled by the FPGA internal memory, stored in the flash storage, and processed within the FPGA by each data segment. This database management system enables high-speed processing, said Hitachi.
Parallel data processing is widely adopted to conduct high-speed processing. In column-oriented or columnar database, however, this is difficult as the processing of one column must finish before the next column can be processed.
To overcome this, a column processing method was developed to enable a set number of columns to be processed in turn. Parallel data processing was realized using this method together with a data filter circuit to select the data for analytics, and an aggregation circuit to group the data and calculate values such as total or average, to realize parallel data processing.
The two technologies developed were combined with “Pentaho Business Analytics,” a business analytics software developed by Pentaho Corporation (a Hitachi Group company) to visualize business analytics results, and with flash storage for data storage, to create a prototype real-time data analytics system.
The prototype will contribute to realizing self-service data analytics enabling employees in the field to easily and quickly execute data analytics on massive business data, said Hitachi.
Hitachi plans to exhibit these technologies at the Flash Memory Summit 2016, to be held from 9-11 August 2016 in Santa Clara, California, USA.
