Home > Spotlights

Collection: Techniques and applications of IIoT time series database management system

By Tsinghua University wuzhenwic.org Updated: 2021-11-17

The Industrial Internet of Things (IIoT) connects machine equipment, control systems, information systems and business processes, uses massive data for analysis and decision-making. It is the infrastructure of intelligent manufacturing and affects the entire industrial value chain. The Industrial Internet of Things machines and equipment generate massive amounts of industrial Internet of Things time series data (each time series is a set of data points stored in the order of timestamps), which is the scale and value subject of industrial big data.

Time Series DBMS is a system software that provides optimized storage and query services for time series. It is an important type of database system, including representative products such as GE Predix, InfluxDB, OpenTSDB, KairosDB, and TimescaleDB in the United States. The database trend graph published by the authoritative database ranking website DB-Engines shows that the time series database management system is the area with the fastest increase in attention. However, the "Edge-Cloud" complex application requirements of the Industrial Internet of Things further test the three capabilities of fast data acquisition, real-time insight discovery, and efficient edge processing. The existing time series database products cannot meet the above-mentioned new requirements. The challenges are mainly reflected in three aspects: file storage structure, metadata management, and robust processing technology.

示意图1-场景_副本.png

In response to the above challenges, the project team focuses on the key technologies of the Industrial IoT Time Series Database Management System and Application, carry out systematic technological innovation in accordance with the two main lines of "one vertical and one horizontal".

On the vertical main line, the project closely follows the "sequence (total order) + set (disorder)" dual characteristics of the time series data of the Industrial Internet of Things, carry out research in accordance with the three-layer abstract theoretical system of data, and obtain the results in the physical layer, logical layer, and application layer. Innovative results:

○At the physical layer, an adaptive storage technology for time series data has been introduced. The compact column storage file format TsFile and copy optimization methods have been innovated to implement the high compression ratio storage and high-speed reading and writing of industrial IoT time series data on physical disks;

○In the logic layer, the automatic metadata identification technology has been introduced. The edge-cloud two-stage metadata identification method and the sequence segment-level working status label identification method have been innovated, which solves the problem of the representation and understanding of the time series data of the Industrial Internet of Things;

示意图2-时序数据介绍_副本.jpg

○ At the application layer, a highly robust processing technology for time series data has been introduced. An out-of-order tolerant time series data receiving and processing technology and industrial Internet of Things low-quality data cleaning technology have been innovated, which is a breakthrough that improve quality and efficiency from low-quality data.

示意图5-关键技术3_副本.jpg

On the horizontal main line, fucus on the "edge-cloud" distributed computing platform of the Industrial Internet of Things, the project team has innovated an integrated edge-cloud data collaboration architecture that decouples storage, query, and processing. By integrating the aforementioned technological innovations, the project team independently develops the Industrial IoT Time Series Database System software "Tsinghua Dataway IoTDB", which provides the write performance of more than 32 million points per second with a single machine, supports management of nearly 100 million sequences, and supports cluster horizontal expansion on the cloud side. The open-source version of IoTDB was the only Apache global top-level project in the field of Industrial IoT Time Series Data Management in Apache community, deeply integrated with industrial IoT and big data systems such as PLC4X, Flink, Spark, Hadoop, etc., to jointly build an industrial IoT open-source data software ecosystem.

示意图7-典型用户_副本.jpg

Compared with time series database systems, key-value databases, and relational database transformation systems represented by InfluxDB/GE Predix, OpenTSDB/KairosDB, and TimescaleDB, IoTDB has obvious advantages in data management/processing and data query/architecture/ecosystem. The third-party report of China Software Test Center shows that IoTDB is ahead of the best-performing InfluxDB among similar products in disk space cost, write performance, and query performance; another third-party CNAS test report shows that IoTDB is 4 to 11 times better than the US GE Predix in writing latency, data scan and aggregate query area.

Dr. C. Mohan, one of the founders of database transaction recovery technology and a member of the American National Academy of Engineering, commented that "IoTDB is the first database project in a Chinese university that meets the top international standards. " Evaluation by Liao Xiangke, a member of the Chinese Academy of Engineering, "The data storage and query analysis technologies have been innovated. It is the core basic software of the Industrial Internet of Things". The achievement won the first prize of Beijing Science and Technology Progress Award in 2020.

At present, IoTDB has been applied in many fields such as metallurgical engineering, petrochemical engineering, aircraft manufacturing, nuclear power, wind power, smart power plants and transportation, with thousands of business users from China, Germany, Australia, the United States, India, etc. In China, IoTDB has effectively supported the implementation and upgrading of industrial Internet in leading companies such as Chengdu Aircraft Industrial (Group), Qingdao Sifang Locomotive & Rolling Stock, China State Shipbuilding, State Grid Corporation of China, China Tobacco, Goldwind, Datang Power, Lenovo, Business-intelligence of Oriental Nations (Dongfang Guoxin), and Changan Automobile. Internationally, ArcelorMittal, the world largest steel production company, tried IoTDB to replace the HBase+Spark data management system. Besides, Pragmatic Industries, funded by the German Federal Ministry for Economic Affairs and Energy, used IoTDB to provide strong support for the real-time data collection and processing of German BMW engine block manufacturing.


ORGANIZED BY
Cyberspace Administration of China
People's Government of Zhejiang Province
CO-ORGANIZED BY
United Nations Department of Economic and Social Affairs
International Telecommunication Union
World Intellectual Property Organization
GSMA
HOSTED BY
Secretariat of World Internet Conference (Preparatory Office)
Cyberspace Administration of Zhejiang Province
Economy and Information Technology Department of Zhejiang Province
Tongxiang Municipal People's Government
National Internet Emergency Center
CONTACT US
Tel: 0086-571-85311391(For Conference) 0086-571-85800770-213(For Exhibition)
Fax: 0086-571-85195207
Email: service@wicwuzhen.cn
QQ: 2092919312

Copyright © World Internet Conference. All rights Reserved
Presented by China Daily. 京ICP备13028878号-23

Copyright © World Internet Conference. All rights Reserved Presented by China Daily. 京ICP备13028878号-23