Opengauss database log management guide

In this paper, openGauss Database log related contents and management methods , understand openGauss The contents of log management in the database , And daily management and maintenance of the database 、 Operation of problem location and database recovery .

Environmental statement

• Networking instructions
  The experimental environment is openGauss Database management system , Installed in Huawei cloud openEuler Elastic servers ECS On .

• Introduction of equipment
  In order to meet the experimental needs of database principle and practice course , The following configurations are recommended for each set of experimental environment software version ：

  Equipment name	Model of equipment	Software version
  database	openGauss	openGauss 1.1.0
  operating system	openEuler	openEuler 20.3LTS

• overview
  

1. Log management

1.1 Experiment introduction

1.1.1 About this experiment
In the actual database management work , Log with small amount of data can be in accordance with the Experimental Manual , Use cat see . But it should be noted that , There may be a large amount of logs in the production environment , The log file capacity is often GB About level , In order to improve the effectiveness of reading logs , It is generally recommended to use more、tail、grep Wait for the command to view the log , Therefore, it is also necessary to learn the basic usage of these commands . in addition , There are many nodes in the production environment （ Thousands of production nodes ）、 When the log type is complex , Labor has been unable to meet the actual production needs , In this case , Generally, scripts are used for automatic analysis or third-party software is used for real-time monitoring of logs 、 analysis 、 The alarm .
This experiment mainly describes openGauss The contents of log management in the database , And daily management and maintenance of the database 、 Operation of problem location and database recovery .

1.1.2 The experiment purpose

• master openGauss The contents of log management in the database ;
• Master the daily management and maintenance of the database 、 Problem location .

1.1.3 system log
openGauss Runtime database nodes and openGauss The logs generated during installation and deployment are collectively referred to as system logs . If openGauss Failure while running , These system logs can be used to locate the cause of the failure in time , Make the recovery plan according to the log content openGauss Methods .
explain ：

• The log path is being installed openGauss By XML In file gaussdbLogPath Parameter assignment , If this parameter value is not specified , The default path is /var/log/gaussdb.

• The data file path is being installed openGauss By XML In file dataNode Parameter assignment , If this parameter value is not specified , The default path is /gaussdb/data/data_dn.

1.1.3.1 Runtime log

step 1 Switch to omm user , Operating system users omm Log in to the database master node .

su - omm

step 2 The running log of the database node is placed in “$gaussdbLogPath/ user name /pg_log / user name /pg_log” in , The user name in the current scenario is ”omm”, Switch to pg_log Folder , And display the contents of the folder .

cd /var/log/gaussdb/omm/pg_log  ## The actual path is in $gaussdbLogPath Subject to ls

The contents of the file are shown below :

dn_6001

The database node folder is ”dn_6001”, Take your actual database node name first .

step 3 Switch to “pg_log” In the database node folder under the folder , view log file .

cd dn_6001ls

The list of log files is as follows ：

postgresql-2020-07-21_170715.log  postgresql-2020-07-23_145207.logpostgresql-2020-07-21_174440.log  postgresql-2020-07-23_160709.logpostgresql-2020-07-22_102526.log  postgresql-2020-07-23_162357.logpostgresql-2020-07-23_140520.log  postgresql-2020-07-24_103226.logpostgresql-2020-07-23_142946.log  postgresql-2020-07-25_000000.log

step 4 Because the log size may exceed 16 MB, So in the process of simple query , You can check the size of the log first （ among l yes L Lower case of ）.

ll -h

The list is as follows ：

total 56M-rw------- 1 omm dbgrp  56K Jul 21 17:38 postgresql-2020-07-21_170715.log-rw------- 1 omm dbgrp 3.5M Jul 21 20:51 postgresql-2020-07-21_174440.log-rw------- 1 omm dbgrp  12M Jul 22 21:10 postgresql-2020-07-22_102526.log-rw------- 1 omm dbgrp  46K Jul 23 14:06 postgresql-2020-07-23_140520.log-rw------- 1 omm dbgrp  83K Jul 23 14:32 postgresql-2020-07-23_142946.log-rw------- 1 omm dbgrp 1.1M Jul 23 15:45 postgresql-2020-07-23_145207.log-rw------- 1 omm dbgrp 373K Jul 23 16:23 postgresql-2020-07-23_160709.log-rw------- 1 omm dbgrp 5.3M Jul 23 21:03 postgresql-2020-07-23_162357.log-rw------- 1 omm dbgrp  15M Jul 24 23:59 postgresql-2020-07-24_103226.log-rw------- 1 omm dbgrp  19M Jul 25 16:52 postgresql-2020-07-25_000000.log

step 5 You can view logs with less content .

cat postgresql-2020-07-23_142946.log

The content of the log is ：

2020-07-23 14:29:46.457 5f192e5a.1 [unknown] 281466519832976 [unknown] 0 dn_6001 00000 0 [BACKEND] LOG:  start create thread!2020-07-23 14:29:46.457 5f192e5a.1 [unknown] 281466519832976 [unknown] 0 dn_6001 00000 0 [BACKEND] LOG:  create thread end!2020-07-23 14:29:46.458 5f192e5a.10000 [unknown] 281466446432848 dn_6001 0 dn_6001 00000 0 [BACKEND] LOG:  reaper backend started.2020-07-23 14:29:46.458 5f192e5a.10000 [unknown] 281466463275600 dn_6001 0 dn_6001 00000 0 [BACKEND] LOG:  [Alarm Module]alarm checker started.2020-07-23 14:29:46.459 [MOT] <TID:21137/-----> <SID:-----/-----> [INFO]     <JitExec>            Using native LLVM version 7.0.02020-07-23 14:29:46.459 [MOT] <TID:21137/-----> <SID:-----/-----> [INFO]     <Configuration>      Configuring total memory for relative memory values to: 2048 MB2020-07-23 14:29:46.459 [MOT] <TID:21137/-----> <SID:-----/-----> [INFO]     <System>             Startup: Loading configuration from /gaussdb/data/db1/mot.conf2020-07-23 14:29:46.459 [MOT] <TID:21137/-----> <SID:-----/-----> [INFO]     <Configuration>      Loaded max_mot_global_memory: 80% from total = 1638 MB2020-07-23 14:29:46.459 [MOT] <TID:21137/-----> <SID:-----/-----> [INFO]     <Configuration>      Loaded max_mot_local_memory: 15% from total = 307 MB2020-07-23 14:29:46.459 [MOT] <TID:21137/-----> <SID:-----/-----> [INFO]     <Configuration>      Loading max_connections from envelope into MOTEngine: 50002020-07-23 14:29:46.459 [MOT] <TID:21137/-----> <SID:-----/-----> [INFO]     <Configuration>      Configuring asynchronous redo-log handler due to synchronous_commit=off2020-07-23 14:29:46.460 [MOT] <TID:21137/-----> <SID:-----/-----> [WARNING]  <Configuration>      Adjusting MOT memory limits: global = 102 MB, local = 26 MB, session large store = 0 MB, total = 128 MB2020-07-23 14:29:46.460 [MOT] <TID:21137/-----> <SID:-----/-----> [INFO]     <Memory>             Global Memory Limit is configured to: 0 MB --> 102 MB2020-07-23 14:29:46.460 [MOT] <TID:21137/-----> <SID:-----/-----> [INFO]     <Memory>             Local Memory Limit is configured to: 0 MB --> 26 MB2020-07-23 14:29:46.460 [MOT] <TID:21137/-----> <SID:-----/-----> [INFO]     <Memory>             Session Memory Limit is configured to: 0 KB --> 0 KB (maximum 5000 sessions)2020-07-23 14:29:46.460 [MOT] <TID:21137/-----> <SID:-----/-----> [INFO]     <Memory>             Configured automatic chunk allocation policy to 'LOCAL' on single node machine

1.1.3.2 Log during installation and uninstallation
step 1 Switch to omm user , Operating system users omm Log in to the database master node .

su - omm

step 2 openGauss The logs generated during installation and uninstallation are placed in “$gaussdbLogPath/ user name /om” Under the table of contents , The user name in the current scenario is ”omm”, Switch to om Folder , And display the contents of the folder .

cd /var/log/gaussdb/omm/om ## The actual path is in $gaussdbLogPath Subject to ll -h

The contents of the folder are shown below ：

total 160K-rw------- 1 omm dbgrp 9.1K Jul 21 17:07 gs_install-2020-07-21_170455.log-rw------- 1 omm dbgrp  68K Jul 24 10:32 gs_local-2020-07-21_160847.log-rw------- 1 omm dbgrp  35K Jul 24 10:32 gs_om-2020-07-21_174438.log-rw------- 1 omm dbgrp  32K Jul 21 16:36 gs_preinstall-2020-07-21_160843.log

step 3 see “gs_install-2020-07-21_170455.log” journal , understand openGauss Installation .

cat gs_install-2020-07-21_170455.log

The contents of the log are as follows ：

[2020-07-21 17:04:55.967536][27138][gs_install][DEBUG]:The /opt/huawei/wisequery/omm_mppdb/install_step/install_step.dat does not exits.[2020-07-21 17:04:55.967641][27138][gs_install][DEBUG]:gs_install execution takes 7 steps in total[2020-07-21 17:04:55.967709][27138][gs_install][LOG][Step1]:Parsing the configuration file.[2020-07-21 17:04:55.970059][27138][gs_install][DEBUG]:Instance information of cluster:ClusterName=dbCluster,AppPath=/opt/gaussdb/app,LogPath=/var/log/gaussdb,ClusterType=single-instHostName=db1,backIps=['192.168.0.58']InstanceId=10001,MirrorId=-3,Host=db1,Port=0,DataDir=cm_agent,XlogDir=,SsdDir=,InstanceType=-1,Role=5,ListenIps=['192.168.0.58'],HaIps=[]InstanceId=6001,MirrorId=1,Host=db1,Port=26000,DataDir=/gaussdb/data/db1,XlogDir=,SsdDir=,InstanceType=0,Role=4,ListenIps=['192.168.0.58'],HaIps=['192.168.0.58'],azName=AZ1.[2020-07-21 17:04:56.022578][27138][gs_install][DEBUG][Step1]:Successfully parsed the configuration file.[2020-07-21 17:04:56.067808][27138][gs_install][LOG][Step2]:Check preinstall on every node.……alarm=/opt/huawei/snas/bin/snas_cm_cmd  --time_out=300 [2020-07-21 17:07:16.476528][27138][gs_install][LOG]:Successfully started cluster.[2020-07-21 17:07:16.476605][27138][gs_install][DEBUG][Step7]:Successfully started the cluster.[2020-07-21 17:07:16.487388][27138][gs_install][LOG]:Successfully installed application.[2020-07-21 17:07:16.487469][27138][gs_install][LOG]:end deploy..

1.1.4 The operation log
Operation log refers to when the database administrator uses tools to operate the database and the tools are openGauss Log generated when calling . If openGauss failure , These log information can be used to track what users have done to the database , Reproduce the fault scenario .
1.1.4.1 Operation steps
step 1 Switch to omm user , Operating system users omm Log in to the database master node .

su - omm

step 2 Default in “ G A U S S L O G / b i n ” Objective record Next , Its in GAUSSLOG/bin” Under the table of contents , among GAUSSLOG/bin” Objective record Next , Its in GAUSSLOG The default is “/var/log/gaussdb/ user name ”, The user name in the current scenario is ”omm”.

cd /var/log/gaussdb/omm/binls

It is shown as follows ：

gs_ctl  gs_guc  gs_initdb  gs_obs

step 3 With gs_guc Operation as an example , Get into gs_guc Folder , And view the properties of the file .

cd gs_gucll -h

The list of documents is shown below ：

total 20K-rw------- 1 omm dbgrp 17K Jul 23 17:20 gs_guc-2020-07-21_170713-current.log

step 4 View the operation log file

cat gs_guc-2020-07-21_170713-current.log

The log shows ：

[2020-07-21 17:07:13]expected instance path: [/gaussdb/data/db1/postgresql.conf]gs_guc set: ssl=on: [/gaussdb/data/db1/postgresql.conf]gs_guc set: ssl_cert_file='server.crt': [/gaussdb/data/db1/postgresql.conf]gs_guc set: ssl_key_file='server.key': [/gaussdb/data/db1/postgresql.conf]gs_guc set: ssl_ca_file='cacert.pem': [/gaussdb/data/db1/postgresql.conf]Total instances: 1. Failed instances: 0.Success to perform gs_guc!……[2020-07-23 17:20:20]Begin to perform gs_guc for all datanodes.[2020-07-23 17:20:20][2020-07-23 17:20:20]expected instance path: [/gaussdb/data/db1/pg_hba.conf]Notice: the above configuration uses cluster internal communication, your configuration may affect the cluster internal communication.gs_guc sethba: host all  dim 192.168.0.96/24  sha256: [/gaussdb/data/db1/pg_hba.conf]Total instances: 1. Failed instances: 0.Success to perform gs_guc!Total instances: 1. Failed instances: 0.Success to perform gs_guc!Total instances: 1. Failed instances: 0.Success to perform gs_guc!

1.1.5 The audit log
When the audit function is enabled, a large number of audit logs will be generated , Take up disk space . Users can set audit log maintenance policies according to the size of disk space .
1.1.5.1 Prerequisite

• Users must have audit authority .
• omm Users connect to the database .

step 1 Operating system users omm Log in to the database master node .

su - omm

step 2 start-up openGauss Database services

gs_om -t start

step 3 Use the following command to connect to the database .

gsql -d postgres -p 26000 -r

postgres The name of the database to be connected ,26000 Is the port number of the database master node .

1.1.5.2 Background information
Please refer to the following table for the configuration parameters related to the audit log and their meanings .

Configuration item	meaning	The default value is
audit_directory	Storage directory of audit files .	/var/log/gaussdb/ user name /pg_audit
audit_resource_policy	Saving strategy of audit log .	on（ Indicates the use of space configuration policies ）
audit_space_limit	The total amount of disk space occupied by audit files .	1 GB
audit_file_remain_time	Minimum saving time of audit log file .	90 God
audit_file_remain_threshold	The maximum number of audit documents under the audit directory .	1048576

explain ： If you use gs_om Tool deployment openGauss, Then the audit log path is “/var/log/gaussdb/ user name /pg_audit”.

• The audit log deletion command provides for the database sql function pg_delete_audit, Its prototype is ：
  pg_delete_audit(timestamp startime,timestamp endtime)
• The parameter startime and endtime Respectively represents the start time and end time of the audit record .

At present, there are two ways to record audit content ： Record to the table in the database 、 It was recorded that OS In file . The advantages and disadvantages of these two methods are compared as shown in the following table .

The way	advantage	shortcoming
Record in table	There is no need for users to maintain audit logs .	Because the table is the object of the database , If a database user has certain permissions , You can access the audit table . If the user illegally operates the audit table , The accuracy of audit records is difficult to be guaranteed .
It was recorded that OS In file	To compare safety , Even if an account can access the database , But there may not be visits OS Permissions for this file .	Users are required to maintain audit logs .

From the perspective of database security ,openGauss Use record to OS File the audit results , Ensure the reliability of the audit results .

1.1.5.3 Select the log maintenance method to maintain .

1.1.5.3.1 Set to automatically delete audit logs
When the disk space occupied by audit files or the number of audit files exceeds the specified maximum , The system will delete the oldest audit file , And record the deletion information of the audit file in the audit log .
notes ： The default value of disk space occupied by audit files is 1024MB, Users can reset parameters according to the size of disk space .

step 1 Configure the amount of disk space occupied by the audit file （audit_space_limit）, View the configured parameters .

postgres=# SHOW audit_space_limit;  audit_space_limit -------------------  1GB (1 row)

If the display result is not 1 GB（1024 MB）, perform “\q” Command to exit the database .

postgres=# \q

step 2 It is recommended to execute the following command to set the default value 1024 MB.

gs_guc reload -N all -I all -c "audit_space_limit=1024MB"

The result is ：

"audit_space_limit=1024MB";Begin to perform gs_guc for all datanodes.Total instances: 1. Failed instances: 0.Success to perform gs_guc!

step 3 Configure the maximum number of audit files （audit_file_remain_threshold）. Connect to database , View the configured parameters .
Connect to database ：

gsql -d postgres -p 26000 -r

Check the parameter of the number of audit files ：

postgres=# SHOW audit_file_remain_threshold; audit_file_remain_threshold ----------------------------- 1048576(1 row)

If the display result is not 1048576, perform “\q” Command to exit the database .

postgres=# \q

step 4 It is recommended to execute the following command to set the default value 1048576.

gs_guc reload -N all -I all -c "audit_file_remain_threshold=1048576"

Is shown as ：

Begin to perform gs_guc for all datanodes.Total instances: 1. Failed instances: 0.Success to perform gs_guc!

1.1.5.3.2 Manually back up audit files
When the disk space occupied by the audit file or the number of audit files exceeds the value specified in the configuration file , The system will automatically delete older audit files , Therefore, it is recommended that users save important audit logs periodically .

step 1 Connect to database , Use show Command to get the directory where the audit file is located （audit_directory）.
Connect to database ：

gsql -d postgres -p 26000 -r

Obtain the directory where the audit files are located ：

postgres=# SHOW audit_directory;            audit_directory            --------------------------------------- /var/log/gaussdb/omm/pg_audit/dn_6001(1 row)

Exit database

postgres=# \q

step 2 Copy the whole audit directory and save it .

cp -r /var/log/gaussdb/omm/pg_audit/dn_6001 /var/log/gaussdb/omm/pg_audit/dn_6001_bak

Check whether the backup was successful ：

cd /var/log/gaussdb/omm/pg_audit/ll -htotal 8.0Kdrwx------ 2 omm dbgrp 4.0K Sep 14 16:52 dn_6001drwx------ 2 omm dbgrp 4.0K Sep 15 11:43 dn_6001_bak

1.1.6 WAL journal
Pre written logs WAL（Write Ahead Log, Also known as Xlog） If you want to modify the data file , These modifications can only be made after they have been recorded in the log file , That is, after the log records describing these changes are refreshed to permanent memory . When the system crashes , have access to WAL Log pair openGauss Perform recovery operations .
1.1.6.1 Operation steps
step 1 Operating system users omm Log in to the database master node .

su - omm

step 2 Take a database node as an example , Default in “/gaussdb/data/data_dn/pg_xlog” Under the table of contents . among “/gaussdb/data/data_dn” representative openGauss The data directory of the node , The current situation is “/gaussdb/data/db1/”.

cd /gaussdb/data/lsdb1

Switch to db1 Folder .

cd db1ls

The contents in the folder are as follows ：

base                pg_ctl.lock       pg_notify     postgresql.confcacert.pem          pg_errorinfo      pg_replslot   postgresql.conf.bakgaussdb.state       pg_hba.conf       pg_serial     postgresql.conf.lockglobal              pg_hba.conf.bak   pg_snapshots  postmaster.optsgswlm_userinfo.cfg  pg_hba.conf.lock  pg_stat_tmp   postmaster.pidmot.conf            pg_ident.conf     pg_tblspc     server.crtpg_clog             pg_llog           pg_twophase   server.keypg_copydir          pg_location       PG_VERSION    server.key.cipherpg_csnlog           pg_multixact      pg_xlog       server.key.rand

step 3 Switch to pg_xlog Folder , see WAL Log files .

cd pg_xlogls

The list of log files is as follows ：

00001000000010000004E  000000010000000100000066  00000001000000010000007E00000001000000010000004F  000000010000000100000067  00000001000000010000007F000000010000000100000050  000000010000000100000068  ……000000010000000100000063  00000001000000010000007B  archive_status000000010000000100000064  00000001000000010000007C000000010000000100000065  00000001000000010000007D

1.1.7 Performance log
The performance log mainly focuses on the access performance of external resources . Performance log refers to the log that the database system detects the running state of physical resources when running , Performance detection when accessing external resources , Include disk 、OBS And other external resources . In case of performance problems , You can use the performance log to locate the cause of the problem in time , It can greatly improve the efficiency of problem solving .
1.1.7.1 Operation steps
step 1 Operating system users omm Log in to the database master node .

su - omm

step 2 The performance log directory of the database node is “$GAUSSLOG/gs_profile” In the corresponding directory , among $GAUSSLOG The default is “/var/log/gaussdb/ user name ”, The user name in the current scenario is ”omm”. Switch to folder , View the information of the file list .

cd /var/log/gaussdb/omm/gs_profile/lsdn_6001

Switch to dn_6001 Folder .

cd dn_6001ls –h

The contents of the performance log are shown below :

total 4.0K-rw------- 1 omm dbgrp  0 Jul 21 17:07 postgresql-2020-07-21_170715.prf-rw------- 1 omm dbgrp  0 Jul 21 17:44 postgresql-2020-07-21_174440.prf-rw------- 1 omm dbgrp  0 Jul 22 10:25 postgresql-2020-07-22_102526.prf-rw------- 1 omm dbgrp  0 Jul 23 14:05 postgresql-2020-07-23_140520.prf-rw------- 1 omm dbgrp  0 Jul 23 14:29 postgresql-2020-07-23_142946.prf-rw------- 1 omm dbgrp  0 Jul 23 14:52 postgresql-2020-07-23_145207.prf-rw------- 1 omm dbgrp  0 Jul 23 16:07 postgresql-2020-07-23_160709.prf-rw------- 1 omm dbgrp  0 Jul 23 16:23 postgresql-2020-07-23_162357.prf-rw------- 1 omm dbgrp 48 Jul 25 00:00 postgresql-2020-07-24_103226.prf-rw------- 1 omm dbgrp  0 Jul 25 00:00 postgresql-2020-07-25_000000.prf

explain ：

• The performance log mainly monitors three resource accesses ： disk 、OBS、Hadoop.openGauss I won't support it OBS、Hadoop, So only the monitoring information of disk access .
• The access information of disk monitoring is mainly in disk files IO Make statistics when reading and writing . Such as reading files when copying files IO, normal SQL Access during execution OS Table file pread system call .
• Configuration of performance log collection ：logging_collector Whether to collect logs ;plog_merge_age How often can I gather logs once , Unit millisecond .logging_collector Parameter is on, And plog_merge_age Greater than 0, And the host is in normal operation , The recovery process does not collect performance .
• Through the tool gs_log Export the file to view .

Here we are ,openGauss The database log management guidance ends .