With the rapid development of the Internet of Things (IoT), the IT environment has become increasingly complex, and the demand for data storage and management has risen accordingly. To enhance the efficiency of information and data transmission in the IoT context, it is essential to rationally apply IoT big data storage and management technologies, promote information sharing, and highlight the practical value of information data. Therefore, based on the fundamental concepts of IoT big data, this article analyzes the challenges faced in IoT big data storage and management. At the same time, it clarifies the application scenarios and key technical aspects of IoT big data storage and management technologies, aiming to improve the quality of IoT data management and meet the storage requirements of IoT data in the new era.

Introduction

The Internet of Things is a network-centric system that integrates a vast array of sensing devices and the internet. Big data from the IoT is a critical component for the operation of this network center. To effectively perceive and utilize relevant data, it is essential to employ IoT big data storage and management technologies for persistent data storage, as well as real-time data retrieval and categorized processing. However, to establish an effective data ecosystem, it is also necessary to master the key technical aspects of IoT big data storage and management, and to refine the technical management framework for IoT big data, with a focus on data applications and services.

Basic Concepts of IoT Big Data

At its core, the Internet of Things is a data-networking hub centered on “things.” As China’s sensing devices and network technologies have matured, the IoT has become a crucial source of data and information across various sectors. In contrast to the traditional internet, which centers on human beings as the primary hub for data interaction—with data applications largely focused on file transfer, data sharing, video-on-demand, and social networking—the IoT places “things” at its heart. Through the interconnectedness of diverse devices, data services have permeated every aspect of life, with data sources now covering areas such as target tracking and positioning, location sharing, smart city development, and urban security management. Smartphones, automotive sensors, televisions—these are all examples of data sources within the IoT. Since 2013, China’s IoT big data volume has already exceeded 1.4 ZB, and it is projected that by 2030, the number of IoT sensing devices will surpass 7 trillion, providing users with comprehensive data services across the board.

Data storage and management are core technologies for big data in the Internet of Things. When deploying IoT services, it is necessary to employ database technologies, data analytics techniques, and data retrieval methods. Starting from fundamental logical concepts as well as software and hardware perspectives, a high-performance data center network system must be established to provide users with a dedicated environment for data storage and management. Throughout this process, the system will need to handle massive amounts of data processing tasks while simultaneously supporting data retrieval, storage, and analysis. Therefore, big data storage and management technologies must be applied to meet the data service requirements of IoT applications.

Challenges in IoT Big Data Storage and Management

In recent years, the scale of the Internet of Things has continued to expand, and the volume of data in IoT data systems has been increasing rapidly. In IoT application scenarios, the amount of data recorded has already exceeded billions, with data storage reaching the PB level and the data ingestion rate accelerating. Various sensing devices use physical transmission technologies to detect data and store sample data files. However, due to the large number of sensors, when specifically capturing relevant data, the scale of data generated by sensor nodes becomes substantial, thereby increasing the difficulty of data collection, processing, and perception.

Moreover, the sheer volume of data itself poses significant challenges to data storage and management. For instance, in IoT application services, image acquisition and object recognition require distributed data systems. However, when storing data, the Hadoop distributed system achieves a write throughput of only 15 MB/s when writing 10 KB image files. Data retrieval typically follows an offline batch retrieval mode, and during data processing, the object detection speed is limited to 10 frames per second. Given these limitations, the efficiency of data storage and processing for massive datasets is relatively low, making it difficult to meet the stringent data management requirements of specialized fields such as power supply, logistics services, and urban security. Therefore, it is essential to conduct in-depth research into the application scenarios of IoT big data storage and management technologies and to develop more comprehensive data technology solutions.

Application Scenarios of IoT Big Data Storage and Management Technologies

(1) Distributed Data Storage

A distributed database, built on data storage technology, establishes an HBase distributed database that stores IoT data using unstructured and semi-structured data models. This type of database boasts distinct data characteristics and strong security features, making it a scalable database that supports various data access interfaces and offers high flexibility in data storage. In IoT application services, distributed databases can meet diverse data access requirements across multiple scenarios; by selecting specific data orientation formats tailored to particular scenarios, they help enhance the efficiency of database applications. Compared to traditional databases, the HBase distributed database features a simple data model and straightforward data transmission and storage processes. Its technical principle involves converting IoT data into specific strings before storing the data information, which significantly increases the difficulty of external decryption and thereby ensures the security of data storage.

(2) Distributed Database Query

Distributed data querying is an application of scalable data storage and management technologies that, combined with specific data structures, enables comprehensive planning of data information to meet the needs of browsing and querying across multiple data ports. In the IoT data ecosystem, distributed data querying builds upon the categorization of information resources within databases, leverages the database’s distributed capabilities, and, in conjunction with appropriate data structures, provides information services to users.

The technical principle behind distributed database query services lies in leveraging the horizontal scalability of IoT big data technologies to establish a distributed computing framework that enables parallel execution of large-scale, distributed data query tasks. In data retrieval, this approach reduces I/O input ports typically associated with aggregated queries and optimizes data query algorithms through data classification and compression. For ultra-large-scale distributed data queries, linear or near-linear algorithms can be applied to analyze data characteristics within statistical analysis models. By using aggregation algorithms to scan data, highly efficient querying of IoT big data can be achieved.

(3) Cloud Database Retrieval

In IoT big data storage and management, cloud database retrieval leverages cloud computing and virtualization technologies to enhance the storage capabilities of traditional databases and improve the service level for querying IoT information data. Cloud database retrieval can reduce various issues in data computation and statistical analysis, minimize resource consumption during data storage and management, and simultaneously meet the scalability requirements of diverse hardware and software platforms, enabling users to access and utilize databases remotely.

During this period, cloud computing has emerged as a key enabling technology for cloud databases. It can aggregate data resources when storing massive IoT data, match data information, establish management mechanisms for huge volumes of data, and optimize the allocation of data resources. Moreover, in the context of the IoT, data storage requirements vary across different domains. Cloud database retrieval systems can meet these diverse data storage needs by providing users with tailored data services through remote cloud-based services. For instance, given the limited storage capacity of smartphones and computers, cloud databases offer users a secure and reliable platform for data storage and management. By comprehensively managing data resources, these systems not only satisfy users’ data retrieval demands but also enhance the efficiency of IoT data storage.

(3) NoSQL Databases

NoSQL databases are an essential component of big data storage and management technologies for the Internet of Things. Based on NoSQL databases, data storage and management adopt non-relational data models, enabling IoT data services to take on entity models, text models, and parallel models, thereby adapting to diverse data application scenarios and enhancing the effectiveness of data analysis. However, when applying NoSQL database technologies in specific contexts, it is also important to consider the information resource storage and management requirements of the database itself, flexibly addressing data query and retrieval tasks in special scenarios, and building diversified NoSQL databases. For IoT data services involving large volumes of data processing, it is advisable to rationally select column-family databases such as HadoopDB, reenPLum, BigTable, and Dynamo, as well as key-value databases, to overcome the limitations inherent in traditional NoSQL database solutions.

Practical Techniques for IoT Big Data Storage and Management

(1) Technical Solution

1. Build a massive distributed file system

To address the challenge of storing massive volumes of files in IoT data services, we should leverage the big-data environment of the IoT to develop a large-scale distributed file system—such as an efficient storage system for massive small files—named “Sensor FS (Sensor Files Storage).” This system can enhance the write performance of massive data resources and optimize the technical approach to data storage.

1) Design a “write cache service” that, when storing data, first writes massive amounts of file information into a dataset cache module. After initially caching the data in memory, the system employs a clustered write service to enhance data throughput efficiency and reduce communication costs across individual data nodes.

2) Upgrade the “Cluster Write” service: After aggregating small file resources from various sensors in the IoT, these files are merged into larger files and then written into the storage module within the database.

3) Add a bottom-layer storage module. The system can integrate the DMFS distributed memory file system on top of the underlying storage in HDFS (Hadoop Distributed File System). This system takes on the functions of aggregating and categorizing sensor data, as well as caching data writes, when handling massive data volumes, thereby optimizing the database’s throughput performance for data files. During system operation, the system can divide the massive small files within sensor data into multiple datasets and persistently store them in HDFS (Hadoop Distributed File System) after writing.

4) Optimize system deployment. During system deployment, the DMFS (Data Migration File System) provides “write throughput” services for data in the distributed file system via a “top-level” mode. When designing the system, there is no need to modify the HDFS source code, making data management more convenient and ensuring that it remains unaffected by updates to the distributed file system version. During DMFS installation, it can be deployed on any server, the HDFS (Hadoop Distributed File System) master node, or a data storage node, and connected to the data network.

5) Control memory overhead. After applying the DMFS system to enhance the write-cache efficiency of IoT big data, we leverage the access correlations among sensors within the system to aggregate and categorize data resources and summarize data processing results. Once highly correlated sensor data are aggregated into large files, they are written into the HDFS system, thereby breaking through the throughput bottleneck traditionally encountered when storing massive volumes of small files. This approach also controls the volume of raw data stored, reducing memory overhead in the data storage module.

2. Establish a fast retrieval system for massive key-value data.

In IoT big data storage and management, data indexing is the core of IoT data services. However, when the data storage and management platform updates index performance, the retrieval efficiency during data writes to storage tends to be low. To address this issue, a high-speed retrieval system for massive key-value data can be established. By optimizing IoT big data storage and management functions based on data inflow rates and indexing update requirements, we can enhance overall system performance.

1) Enhance the adaptive capability of system index updates and appropriately configure the parallelism of index models.

2) Add a radix tree data structure space and, based on data combinations, adjust the method of representing the data structure.

3) The high-volume key-value data fast retrieval system is divided into multiple data processing workflows, including data writing, data querying, and key-value data storage. Data querying is further subdivided into dimensions such as data writing, data sorting, and the construction of a data indexing model. Based on the data-writing module, keywords can be used to locate key information within the data source. After the data is written, stored, and indexed, the data indexing model is applied to search for the keywords, retrieve the corresponding data, take a snapshot of the data results, and then feed the results back to the user.

(2) Data Storage and Management Strategy

1. Clearly define data characteristics

In the IoT data ecosystem, data sources are diverse, and data storage and management should be tailored to data characteristics, with flexible utilization of data storage resources. Specifically, IoT data exhibit features such as massive volume, real-time nature, structured format, and limited periodicity. Therefore, when managing data storage, it is essential to meet the specific data management requirements associated with these different characteristics.

1) IoT sensors collect various types of data in real time and upload it to the cloud, generating a large volume of information daily. Data storage should take into account the classification, statistical analysis, and write throughput rate of massive datasets.

2) IoT data needs to be transmitted and stored in real time. When aggregating and categorizing data, the data collection frequency should be increased to efficiently record critical data.

3) With regard to the structured characteristics of data, data retrieval and query patterns should be strictly defined in accordance with the application scenarios of IoT data and the data generation cycle, thereby meeting the data storage and management requirements of different fields.

4) IoT data exhibits limited periodicity. After data is collected, its characteristics and storage requirements should be analyzed based on the data source, and the data should be categorized and written into the appropriate data repositories.

2. Conduct an in-depth analysis of data storage requirements.

IoT data originates from sensor devices across various fields. However, since sensors themselves have certain limitations, data collection and storage are constrained by these sensor devices, thereby affecting the effectiveness of IoT data services. Therefore, when applying IoT big data storage and management technologies, it is also essential to take into account the practical data storage needs of the new era, and to establish data interfaces that facilitate data integration, focusing on the data sensing and information transmission objectives of sensors. At the same time, we should actively adopt cloud computing and computer networking technologies to enhance data transmission efficiency.

3. Strengthen data classification management

Since IoT data sources vary, the data formats obtained after data collection differ accordingly. Generally, data can be categorized into structured data and unstructured data. When storing and managing data, we can classify and manage data resources based on their respective formats. For structured data, we can establish a unified data model and use “relational databases” for data storage. As for unstructured data, we can employ distributed file systems and “non-relational databases” for storage and management.

4. Optimize the data storage model design

When designing an IoT big data storage and management model, it is important to emphasize both data security and the efficiency of data utilization.

1) The functional modules—such as the “data storage layer,” “data service layer,” and “data application layer”—can be designed in conjunction with IoT data service content.

2) The model design should meet the requirements for storing both structured and unstructured data in the massive datasets of the Internet of Things. For example, traditional IoT data storage systems have shortcomings in terms of unstructured data read/write operations, database functionality expansion, and data transformation, making them unable to satisfy the data storage demands arising from the explosive growth of data in the new era. Therefore, it is necessary to establish a distributed data storage model, leveraging distributed databases such as NoSQL and building computer clusters to enhance data storage capacity and storage efficiency.

Conclusion

In summary, to effectively control and perceive data resources in the era of the Internet of Things, it is necessary to establish a distributed database based on IoT big data storage and management technologies, and to refine technical solutions for data analysis, data processing, data retrieval, and data storage, thereby meeting the storage requirements of IoT big data and promoting standardization in data storage and management. When applying IoT big data storage and management technologies in practice, we should also take into account the actual needs of data governance in the new era and develop efficient data file storage and management systems, thus providing support for the sustainable development of the IoT industry.

Featured in Digital Design, Issue 13, 2024

Due to space limitations, the footnotes have been omitted. For the complete version, please visit ShuiBiao.com for free access.

Source: SanChuan Wisdom

Authors: Tang Xiong, Zheng Zhiqiang

Editor: Li Jingshuai

First Instance: Zhou Qi

Second Instance: Zhan Zhijie, Cai Jinhui