Background
Project Background
The clearing and settlement system of a provincial high-speed center has been unable to solve related problems such as database design and selection. In the process of the construction of a provincial high-speed project, the database construction scheme has always been a difficult point.
The construction of this project is under pressure from two sides:
1) The time is short and the task is heavy: in terms of policy, a provincial traffic science institute ordered a provincial high-speed highway to realize the design of the network toll billing, lane data and the existing clearing and settlement system as soon as possible, and do a good job in rectification and cooperation, and strive to complete the rectification of the province's network toll collection and clearing and settlement system within four months;
2) Large amount of data and difficult selection: Since May 2019, the province has gradually abolished the provincial toll stations of expressways and gradually promoted ETC tolls, the mileage of expressways in a province has exceeded 10,000 kilometers, and the total number of ETC users issued in the province has exceeded 6 million, the utilization rate of passenger cars and ETC has exceeded 60%, and the utilization rate of ETC in trucks has exceeded 60%.
Construction difficulties
The big data platform where this clearing project is located has a large amount of business support, and the billing and other settlement tasks supported by it are very arduous. Among them, the difficulties of the project are mainly in the following three aspects:
1) Distributed database clusters with stable performance are required for large business data volumes. On the one hand, the database needs to support read/write splitting, otherwise there will be data backlog and low query efficiency, and the Oracle database cannot support read/write splitting.
2) The provincial center clearing and settlement system does not use the data caching mechanism before the data storage, and the program directly connects to the database operation, resulting in a large number of database connections, and the program directly connects to the database to write the table, resulting in inter-process lock waiting, resulting in low efficiency of flowing data storage.
3) The provincial center database of the clearing and settlement system is not effectively combined and planned, resulting in the data uploaded to the provincial center of the clearing system at each toll station can not be centralized, and can only be stored in multiple databases, and the data can not be accessed and counted efficiently and quickly, which affects the timely completion of the normal clearing and settlement business.
ANTDB database solution
The database of the clearing and settlement system adopts the domestic database antdb developed by AsiaInfo Technology to realize the comprehensive independent and controllable database of the high-speed core business system. The clearing and settlement system is responsible for processing and storing the data issued by the billing system, such as the flow of the billing, the gantry transaction flow, and the accounting information issued by the department, and produces a variety of clearing and settlement data system, with hundreds of millions of daily processing data, and all kinds of transaction processing have strong timeliness.
Figure 1: Architecture diagram of the deployment of antdb and a provincial clearing and settlement center.
Native Distributed Capability:The clearing and settlement business has the characteristics of short-term ultra-high concurrency, and the antdb database is equipped with an in-memory computing engine to provide extreme data processing performance, SQL-based data access services, and pluggable expansion services. The native distributed design is completely transparent to applications, taking into account both performance and scalability. In addition, ANTDB has super scale up capabilities, and there are no restrictions on the number of CPUs, memory, and connections. Users can expand the capacity without affecting the user's online business, and the database can be expanded horizontally with the rapid growth of the business. antdb can efficiently support the ultra-high concurrency business of the clearing and settlement system, and ensure that the data processing such as transaction flow is "not leaked, and one is good".
HTAP's ultra-high computing power:The clearing and settlement business includes scenarios such as ** transactions, data analysis, and report output, and the antdb database can handle HTAP scenarios, that is, transaction and real-time analysis fusion scenarios. The antdb database has real-time and strong consistent distributed transaction control capabilities to ensure zero data loss and transaction consistency, and ensure the accuracy of split data and report data. In distributed scenarios, it provides standardized data access control, consistent backup and recovery of global data, and strict data access control and anti-skew design to ensure the safety and reliability of business data. antDB supports remote disaster recovery, has high availability, and can automatically eliminate faulty nodes in the event of a failure, ensuring continuous service availability and ensuring that the database system reaches 99999% service level; ANTDB has a dynamic and efficient SQL execution engine, provides heterogeneous indexes, and has a built-in Reduce data processing engine, which can communicate between data nodes and improve data processing efficiency, especially for complex query scenarios (such as Union, Join, etc.), which plays an important role in real-time data analysis and report generation of the clearing and settlement system.
Figure 2: Distributed AntDB product architecture.
The clearing and settlement system uses the ANTDB distributed architecture database, and the data nodes are deployed with one primary and two standby data nodes to ensure high availability, and the unique read/write splitting feature of ANTDB is enabled to improve data processing performance. After the launch of ANTDB, the efficiency of the ETC splitting business of the system is increased by 90%, the efficiency of ETC clearing and bookkeeping is increased by 60%, and the report generation time is shortened by nearly 90%, which greatly improves the processing efficiency of various businesses of the clearing and settlement system.
The schematic diagram of the antdb deployment architecture is as follows:
Figure 3: Schematic diagram of the ANTDB deployment architecture.
**Expansion, ultra-high storage capacity:At present, the amount of data processed by antDB in the clearing and settlement system reaches hundreds of millions of pieces every day, and the data volume is rapidly surging with business growth, the pressure on the database host IO is gradually increasing, and the resources are running at full capacity for a long time, which will lead to a decrease in business operation efficiency.
Figure 4: Server load pressure before antdb auto scaling.
The problem of business operation efficiency is that the rapid growth of business volume leads to serious performance consumption of database hosts, so it is necessary to reduce the running pressure of production databases. Based on resource situation and cost considerations, the database team proposed two sets of alternative solutions based on the elastic scalability and flexibility of the ANTDB distributed architecture
The first is to expand the antdb database and add more database hosts to share the operating pressure.
The second is to add another set of database clusters, which are specially used to store historical data, and the production library can store data for nearly 6 months, and the historical data of more than 6 months can be stored in the new historical database cluster, so as to reduce the pressure on the production database.
Finally, according to the comprehensive consideration of the demand cost and the actual resources on site, the second option is selected. Based on the logic of scheme 2, in order to maximize the use of resources, the data nodes of the production database are adjusted from one primary and two standby to one primary and one standby, and the asynchronous slave database in the original production library is removed from the shelves as a historical database.
The following figure shows the adjusted DB cluster.
Figure 5: Schematic diagram of the adjusted ANTDB database.
The cluster adjustment of the production database and the launch of the historical database fully demonstrate the elastic scalability and flexibility of ANTDB, which can be adjusted according to the needs and business development.
Figure 6: Server load pressure after antdb auto scaling.