Abstract

Currently, the Internet of Things (IoT) is being widely adopted and promoted across various industries. It connects physical objects in the real world to the internet via information-sensing devices, enabling them to exchange data—thus achieving "object-to-object communication"—and thereby realizing intelligent identification and management. This paper, based on a unified meter-reading platform system for smart water meters, addresses the performance bottlenecks encountered by the application service platform within the IoT platform architecture when dealing with large-scale device connections. To overcome these challenges, we propose a solution that leverages MongoDB as the data storage system and RabbitMQ as the message middleware. Specifically, we utilize MongoDB’s native sharding and replica cluster architecture to store both uplink and downlink data from devices. We have also established a document-based storage model for device data in the database, which significantly enhances the concurrent performance and data persistence efficiency of the application service system. Furthermore, by employing RabbitMQ message queues, we decouple functions such as protocol access, protocol parsing, and data push after protocol parsing, thereby improving the system’s real-time data processing capabilities.

Introduction

With the rapid development of today’s internet, the number of smart device terminals is increasing at an astonishing rate, giving rise to IoT protocols that enable the interconnection of all things. As a crucial bridge connecting users and terminal devices, application service platform systems must be designed with architectures that offer high performance, high concurrency, and high scalability in order to handle the massive volume of smart device terminals and the vast array of diverse types of sensor data they generate. This paper first provides a brief introduction to document-oriented database MongoDB and message middleware RabbitMQ. Then, based on real-world scenarios, we propose an application service system architecture built upon the NoSQL database MongoDB and the message middleware RabbitMQ. Finally, we conduct tests on the new architectural design to validate the feasibility and effectiveness of this proposed solution.

Main text

1 Technical Background

1.1 Document-oriented NoSQL database MongoDB

From the perspective of data storage structure, databases can be classified into relational databases (which organize data using two-dimensional tables) and non-relational databases. Non-relational databases are often also referred to as NoSQL (Not Only SQL) databases. From a data model standpoint, they can further be divided into four categories: key-value databases, column-family databases, document databases, and graph databases. In document databases, the term "document" refers to a collection of loosely structured key-value pairs within a document, rather than documents or tables in the conventional sense. Document databases treat the document as a single entity and do not break it down into multiple key-value pairs.

As a quintessential example of a document-oriented database, MongoDB combines features and functionalities from both relational and non-relational databases. Because it doesn't require a pre-defined table structure, its data schema is highly flexible and loosely structured, and it supports complex structures such as embedded documents. Therefore, MongoDB is particularly well-suited for storing the massive volumes of semi-structured and unstructured heterogeneous data generated by the large number of IoT platform’s terminal devices.

1.2 RabbitMQ Message Middleware

As an essential component in distributed systems, message queue middleware primarily addresses issues such as application coupling, asynchronous messaging, and traffic surge smoothing. It plays a crucial role in enabling high performance, high availability, scalability, and eventual consistency architectures—making it an indispensable middleware for large-scale distributed systems.

RabbitMQ is an open-source AMQP implementation written in the Erlang language. It is cross-platform and supports clients in multiple programming languages. Its key features include message-oriented architecture, routing, and queuing, while also ensuring high reliability in message delivery. RabbitMQ supports various message interaction models, such as point-to-point transmission, publish-subscribe, routing, and wildcard-based messaging. Thanks to the graphical management interface provided by RabbitMQ, developers can easily identify and address issues during development. The following sections introduce several important concepts in RabbitMQ.

1) Exchange

The message producer sends the message to the Exchange, which then forwards the message to the corresponding queue based on the matching routing rules—specifically, the Routing Key.

2) Routing Key

The Routing Key must be used in conjunction with the Exchange Type and Binding Key. When sending a message to an Exchange, you specify a Routing Key to determine which specific Queue the message will be forwarded to.

3) Queue

A queue used for storing messages—message consumers only need to listen to the queue to process and consume messages.

4) Exchange Type

The Exchange Type specifies the matching rules between the Routing Key and the Binding. The commonly used Exchange Types in RabbitMQ include fanout, direct, topic, and headers. The relationships among the Exchange, Routing Key, Queue, and other components are illustrated in Figure 1.

Figure 1: Diagram of the Relationship Among Exchange, Routing Key, and Queue

2. Feasibility Study

Currently, IoT meter-reading systems exhibit the following two characteristics. First, the number of terminal devices is enormous. Compared to typical internet applications, which typically have tens of thousands or hundreds of thousands of users, IoT systems often involve millions or even tens of millions of terminals. Such a massive number of terminals places even higher demands on the system’s ability to handle high concurrency and deliver high performance. Second, the data generated by these devices is highly diverse and varies in structure. A single terminal can collect multiple types of sensor data, and the structures of these different sensor data cannot be standardized in advance. As a result, device data exhibits characteristics such as multi-source heterogeneity and loose structural organization.

Through analysis of the existing system, we identified two key areas that do not align with the aforementioned characteristics. First, in large-scale device-access scenarios, the device-access layer is difficult to scale, leading to performance bottlenecks. Second, the traditional relational database SQL Server, with its fixed table structures and field attributes, cannot effectively accommodate heterogeneous and loosely structured device data. Moreover, the strict database transaction constraints limit read/write efficiency, and the database deployment architecture results in single-point data storage, thereby compromising data disaster recovery capabilities. Third, pushing data via timers fails to meet users’ requirements for real-time data updates. To address these issues, we propose an application service platform solution based on MongoDB database and message middleware RabbitMQ. This solution reconfigures the original integration approach of the device-access cluster, changes the data persistence method, and modifies the data-pushing mechanism. In this solution, the higher-performance Netty framework is employed to aggregate message data reported by devices—distributed across various channels—into RabbitMQ. Within the application server cluster, parsing service nodes consume the reported messages from RabbitMQ in a distributed manner, achieving centralized data aggregation in a single pass. After completing the corresponding processing, the aggregated data is persistently stored in MongoDB using a sharding strategy.

Figure 2: Architecture Diagram of the Application Service System

3. Design and Implementation of the Application Service System

Based on the solution proposed in this paper, the architecture of the application service platform is shown in the gray area of Figure 2. This solution employs a message middleware cluster to connect the access cluster with the application service system. The superior concurrent performance of both components will enhance the efficiency of data flow. Meanwhile, all data is uniformly collected into RabbitMQ for buffering, which further improves the concurrency performance of the access layer. Leveraging the flexibility of document data structures in MongoDB’s collections, semi-structured data can be stored directly, thereby avoiding cumbersome data structure conversions. Additionally, MongoDB’s replica-based replication mechanism provides disaster recovery capabilities for data storage. Furthermore, MongoDB’s features—such as the aggregate() method and support for the Map-Reduce computing model—enable it to meet the business requirements of big data analytics and high-concurrency operations on massive datasets.

This solution primarily focuses on three aspects: data format, storage mode, and operations. The data format adopts MongoDB’s built-in BSON storage format, which leverages MongoDB’s flexible data structure to facilitate efficient data storage. For data storage, we choose the MongoDB database and deploy it using a 3-shard, 1-replica configuration. Based on local testing of this cluster setup, MongoDB can achieve a throughput of up to 17,500 operations per second under balanced read-write scenarios. Building on this foundation, we employ a combined approach of RabbitMQ and MongoDB for the device access system, aiming to reduce the database workload during large-scale data writes. First, device data is aggregated into a RabbitMQ buffer, thereby avoiding direct pressure on the database. Second, leveraging RabbitMQ’s message queue features, in a producer-consumer pattern, each node in the server cluster acts as a consumer to process and persistently store the incoming data.

4 System Performance Testing

This document describes the setup and deployment of a system based on a MongoDB cluster with three shards and one replica, combined with a message middleware. The system was built and deployed to meet actual requirements for response performance metrics: 100 million devices should be able to come online discretely within 8 hours with a 100% success rate, and responses should be delivered within 3 seconds. The system’s operation is illustrated in Figure 4.

Using the JMeter testing tool, we simulated 500 virtual device terminals concurrently accessing the cloud platform. The system reached a concurrency level of 500 within one minute and maintained this level for 10 consecutive minutes. During this period, the transaction failure rate was only 0.03%, indicating that the system had reached its performance peak. According to the analysis report of the test results, under the test environment’s 500 concurrent users, the average response time was 150 milliseconds, easily meeting the system’s requirement of a 3-second response time. The TPS (transactions per second) was around 1,800, fully satisfying the system’s business needs. Moreover, the CPU utilization of the platform’s server nodes stabilized at approximately 70%, and memory usage remained within a reasonable range. Overall, the platform’s throughput showed a steady trend, demonstrating excellent performance stability.

Figure 3: System Operation Performance Diagram

5 Summary

This paper identifies two common characteristics inherent in IoT platforms and analyzes three shortcomings of existing systems. Based on these findings, we propose a system solution for meter-reading applications that leverages the NoSQL database MongoDB and the message middleware RabbitMQ. This solution redefines the integration approach between the access platform and the application service platform, changes the data flow and persistence mechanisms, and effectively addresses the severe coupling issue among multiple modules. Experimental results demonstrate that the proposed solution is both effective and feasible. It enhances the platform’s concurrent performance and data real-time capabilities, reduces the difficulty of subsequent development and maintenance, and provides better support for business expansion.

The built-in, visual monitoring page provided by the RabbitMQ cluster service makes it easier and more convenient to monitor and maintain the platform’s message cluster. In the future, by integrating the Grafana + Prometheus monitoring interface system, we’ll be able to more conveniently track and view all performance metrics of the entire system.

Due to space limitations, the footnotes have been omitted. For the complete version, please download and view it on ShuiBiao.com.

Source: SanChuan Wisdom

Author: Zhu Zhiyun

Editor: Li Jingshuai

First Instance: Zhou Qi

Second Instance: Zhan Zhijie, Huang Zhenwei