MySQL combat optimization expert 09 production experience: how to deploy a monitoring system for a database in a production environment?

1、 The database of production environment can't run naked ！

We have explained the knowledge related to the pressure measurement of the database to you before , Presumably, you just need to use the knowledge we explain , In your own company , Even if the DBA group team 、QA The team can't provide you with professional database pressure testing technical support , But after you get a database in your hand , In fact, you can also use various tools And orders , It is very good to complete the pressure test of the database on a machine .

You should know very well , What kind of configuration is a machine , After deploying a database , utilize sysbench How many tables and data volumes have been constructed , then How many threads are simulated during pressure testing , When the hardware load of the machine is within an acceptable range , Database QPS and TPS How high can the pressure be measured .

At this time, you can roughly understand how much your database can bear at most during peak periods QPS and TPS 了 .

But after the pressure test , Does everyone just want to start developing your Java System ？ Connect your system directly to MySQL Go up and carry out various CRUD Of SQL language sentence ？ Then he began to write all kinds of Java Code and SQL sentence , After writing, find QA To test , Then deploy to the online production environment , Then everything is big Lucky , Regardless of the database ？

This practice may be the same for many companies and teams at present , But if you just do this, it's absolutely impossible . Because in fact, we need to improve the online system Perfect monitoring , Not just for you Java The system monitors , You have to monitor your database , Including yes CPU、 Memory 、 The Internet 、 disk IO、 Slow search Inquiry 、QPS、TPS Monitoring of .

Because if you don't do any monitoring on your database , Then maybe your database CPU The load is already very high , Or disk IO The limit has been reached , You don't even know Avenue , As a result, you still run your Java System , One day, your database may suddenly hang up, and you don't react ！

So today we will take you step by step to build a visual monitoring platform for the production environment database , We will be based on Prometheus+Grafana To build .

Of course, in the company , If you want to build a unified visual monitoring platform for the database , This job is often DBA The team is responsible for , But anyway , Here we are You should also have a certain understanding of the technology of visual monitoring of this database .

2、 A brief introduction Prometheus and Grafana What is it?

Let's give you a brief introduction Prometheus and Grafana What are the two systems .

Simply speaking ,Prometheus In fact, it is a monitoring data acquisition and storage system , He can use the monitoring data acquisition component （ such as mysql_exporter） From you designated MySQL Collect the monitoring data he needs from the database , Then he has a time series database , He will put the collected monitoring data into his own time sequence According to the library , In fact, the essence is to store it in disk files .

We collected MySQL The monitoring data is not enough , Now we need to look at some reports composed of these data , So you need to use Grafana 了 ,Grafana It is a visual monitoring data display system , He can turn Prometheus Collected a large number of MySQL The monitoring data is displayed in various exquisite newspapers surface , Let us see intuitively MySQL Monitoring of .

In fact, it is not only the database monitoring that can be used Prometheus+Grafana The combination of , For all kinds of things you have developed Java System 、 Middleware system , You can use this Set of combinations to carry out visual monitoring , Nothing more than letting Prometheus To collect your monitoring data , And then use Grafana It's just a report .

3、 Installation and startup Prometheus

I told you before , Let's prepare one by ourselves linux machine , If you are windows laptop , You can install one by yourself linux virtual machine . We are based on a linux Machine to deploy Prometheus and Grafana, as for MySQL Installation , This is very simple , It is easy for everyone to search on the Internet .

First of all, you need to download 3 Compressed packages , Link below ：

http://cactifans.hi-www.com/prometheus/

You can download the following two compressed packages , here prometheus It is used to deploy the monitoring system itself , then node_exporter It's for collecting MySQL Count According to the machine where the library is located CPU、 Memory 、 The Internet 、 Disk and other monitoring data ：

prometheus-2.1.0.linux-amd64.tar.gz     node_exporter-0.15.2.linux-amd64.tar.gz  

Then you can download the third compressed package through the following link ：mysqld_exporter-0.10.0.linux-amd64.tar.gz, This mysqld_exporter Just It's for collecting MySQL Some monitoring data of the database itself , such as SQL performance 、 Number of connections ：

https://github.com/prometheus/mysqld_exporter/releases/download/v0.10.0/mysqld_exporter-0.10.0.linux-amd64.tar.gz

Then you need to unzip the above packages , Just follow my order below ：

mkdir /data mkdir /root 

tar xvf prometheus-2.1.0.linux-amd64.tar -C /data

tar xf node_exporter-0.15.2.linux-amd64.tar -C /root

tar xf mysqld_exporter-0.10.0.linux-amd64.tar.gz -C /root 

cd /data

mv prometheus-2.1.0.linux-amd64/ prometheus

cd /prometheus

vi prometheus.yml, I'm going to modify prometheus Configuration file for , Actually, it's mainly in scrape_configs Now add a large section of customized configuration , because Because he needs to collect MySQL The database itself and MySQL Monitoring data of the machine ：

scrape_configs:

 - file_sd_configs:

   -files:

     - host.yml

   job_name: Host
   metrics_path: /metrics
   relabel_configs:

   - source_labels: [__address__]      regex: (.*)

     target_label: instance
     replacement: $1

   - source_labels: [__address__]      regex: (.*)

     target_label: __address__
     replacement: $1:9100

 - file_sd_configs:

   - files:

     - mysql.yml

   job_name: MySQL
   metrics_path: /metrics
   relabel_configs:

   - source_labels: [__address__]      regex: (.*)

     target_label: instance
     replacement: $1



   - source_labels: [__address__]      regex: (.*)

     target_label: __address__
     replacement: $1:9104

 - job_name: prometheus    static_configs:

   - targets:

     - localhost: 9090

After the above configuration file is written , You can start it Prometheus 了 , But let's look at the configuration information above several times , Because I'm not very square when writing articles then , They are all directly knocked out by hand , There may be a few configuration errors , If you find something wrong with the configuration file , Tell me in the backstage comment area in time .

Then you have to /data/prometheus Directory , To execute the startup command ：

/data/prometheus/prometheus --storage.tsdb.retention=30d &, there 30d It means that your monitoring data is retained 30 Days of . Start it up after , It can be accessed in the browser 9090 Check the port number prometheus The home page of .

Because we deploy and install Prometheus and Grafana There are many processes , So it is divided into two articles , In the second article updated synchronously today , Will put the rest Of Grafana The installation and deployment process of , And the process of monitoring configuration and acquisition .