About the Author:
Liu Yu,He is the basic technology architect of Tiancheng Financial Innovation Center, with 9 years of IT experience, 9 years of database development and operation experience, and 4 large-scale marketing activities. At present, he focuses on basic technologies such as containers and distributed database technology.
*When troubleshooting Redis performance issues, it is a common practice to find out from Redis for optimization.
Redis Slowlog is designed as a first-in-first-out queue structure in memory, and in the case of a large number of slow logs, old commands will be flushed out of the slowlog and cannot be tracked.
This topic describes how to use the modified RSBEAT to dump the slowlog to Elasticsearch for analysis.
1. Introduction to Redis Slowlog
Slowlog core configuration parameters and commands.
slowlog-log-slower-than：The threshold of slowlog, only the query execution time is greater than slowlog-log-slower-than, the unit is set to microseconds, and the default is 10,000 microseconds, that is, 10ms.
slowlog-max-len：Indicates the maximum number of slow queries, and the default value is 128. When the slowlog exceeds the maximum value, the oldest slowlog is deleted.
slowlog get n：To obtain the specified number of records, you can run the slowlog get n command.
Introduction to rsbeat
RSBEAT** is the official Beats platform, which integrates a variety of single-purpose data collectors. They send data from hundreds, thousands, or thousands of machines and systems to Logstash or Elasticsearch.
Official website: elasticco/guide/en/beats/libbeat/current/community-beats.html
This time, we used three products: Elasticsearch, Kibana, and RSBEAT. Elasticsearch is used to store parsed Redis slowlogs, Kibana is used for graphical analysis, and RSBEAT is used to collect Redis slowlogs.
The collection principle of rsbeat is as follows:
2. RSBEAT transformation
Because the community's rsbeat only supports ipport retrieval. Redis Sentinel and Redis Cluster are not supported, and in order to meet our needs, RSBake has been partially modified here. The transformation points are as follows:
1. Added support for Sentinel and Redis Cluster
The original configuration of rsbeat is as follows:
The Redis parameter is configured to the list of Redis instances in the form of ipport, which we transform to the form of ipportname
For Redis Sentinel, MyRedis uses the Master Name registered in Sentinel to retrieve it.
For Redis Cluster, you can use a custom cluster name or the cluster name in CMDB to retrieve MyRedis. Generally, there are many nodes in a Redis cluster, so you can avoid manually logging in to multiple nodes to obtain slow queries, thereby improving efficiency.
Revamp rsbeatGo ** is as follows, adding an ipportname string handler:
In the PoolList map structure:
2. Record the role information of the Redis instance
If you want to filter the role of Redis, you need to record the role information. In rebeat, add the role command after slowlog get and slowlog reset to obtain the role information.
in func (bt *rsbeat) redisc(beatname string, init bool, c redisconn, ipport string) function.
and beats
Then modify the poolinit function and add the slowerlen parameter:
3. Slowlog-max-len is configurable
Modify slowlog-max-len in configIn the go file, add the slowerlen attribute to the config structure, and the default value:
3. Use Kibana to search
1. Search by master name or Redis cluster name
2. Use master name and role search:
Add a filter for role:
Then retrieve master name:
3. Search in the dashboard:
Fourth, summary
The Elastic Stack is backed by a strong open source community. Through simple transformation and tossing, the rsbeat function can be expanded to better meet the needs of more flexible query and retrieval such as redis sentinel and redis cluster. This article explains the content of the transformation, hoping to provide you with help and ideas.