Year SQL audit platform

Yearning-SQL Audit platform

Two research platforms

archeryd-SQL Audit query platform

https://archerydms.com/

yearning-SQL Audit query platform

http://yearning.io/

Yearning function

Yearning It has many practical functions . help DBA Improve work automation , Standardization .

Yearning It has the following functions :

  automation SQL Statement audit , But for SQL Automatically detect and execute 
 DDL/DML After the statement is executed, the rollback statement is automatically generated 
  to examine / Inquire about   Audit function 
  Support LDAP Sign in / Nailing and email message push 
  Support custom audit workflow 
  Support fine-grained permission allocation 

Grammar support

Yearning Current compatibility 99% Of Mysql standard SQL grammar . However, due to the limitation of audit logic and other factors , Yes, some SQL Statements do not guarantee correct feedback The known unsupported statement types are :

 Complex query statements ( Multiple tables , Multifunction , Not necessarily . Not all complex statements do not support )
  Foreign key related statements 
  Cross library is not supported DML Rollback of statement 
  stored procedure / trigger 
 Only represents the current syntax support progress , Not as a degree of completion in the future .

The main function

SQL  Inquire about 
 Query the work order 
 export 
 Automatic completion , Smart tips 
 Query statement audit 
 Desensitization of query results 
SQL  to examine 
 Process chemical sheet 
SQL Sentence syntax detection 
 Detect according to rules SQL Statement compliance 
 Automatic generation DDL/DML Rollback statement 
 Historical audit records 
 push 
E-mail  Work order push 
 nailing  webhook  Robot work order push 
 User rights and management 
 Role partition 
 Fine grained permissions based on users 
 register 
 other 
todoList
LDAP  Sign in 
 Dynamic audit rule configuration 
 Customize the approval level 
AutoTask  Automatic execution 

Yearning Community

 I encountered some confusion in the use and deployment ？ Welcome to join Yearning Community (QQ Group ) 363323798

Operation document

https://next.yearning.io/guide/config/source.html

matters needing attention

Yearning Do not rely on any third party SQL Audit tool as audit engine , Internal audit has been implemented by itself / Rollback related logic .

Rely only on Mysql database .

mysql Version must be 5.7 And above (8.0 And above, please sql_mode Set to null ) It has been installed by itself in advance and establish Yearning library , Character set should be UTF8mb4 ( only Yearning what is needed mysql edition )

Yearning be based on 1080p Resolution development Support only 1080p And above

For the problem of setting page configuration overlap, please confirm your resolution and whether you have zoomed in

Please use Chrome The latest version ( barring 360 Wait for other magic versions )

establish mysql database , Set character set

Method 1 ：MySQL command

# utf8 
CREATE DATABASE  `wordpress` DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;

# utf8mb4
CREATE DATABASE  `Yearning` DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

Get into mysql, Input show variables like “%char%”; View the current database code ：

Method 2 ： modify /etc/my.cnf file

［client］
default-character-set = utf8mb4

［mysqld］
character-set-server=utf8mb4
collation-server=utf8mb4_unicode_ci

［mysql］
default-character-set = utf8mb4

then , restart Mysql service ：

`/etc/init.d/mysqld stop /etc/init.d/mysqld start`

Yearning download

Yearning-go Provide binary download package

 Download address  https://github.com/cookieY/Yearning/releases

Please select the latest version stay Assets Choose from Yearning-x.x.x.linux-amd64.zip Package for download

If you need to modify the code or compile it yourself, please move to the secondary development page

Yearning Directory structure

1. Fill in the configuration file

cat conf.toml
[Mysql]
Db = "Yearning"
Host = "127.0.0.1"
Port = "3306"
Password = "xxxx"
User = "root"

[General]   # Database encryption and decryption key, It can only be changed once .
SecretKey = "dbcjqheupqjsuwsm"

About SecretKey

SecretKey  yes  token/ Database password encryption / Declassified  salt.

 It is recommended that all users install  Yearning  Before will  SecretKey  change ( There will be a security risk of not changing )

** Format :  Both upper and lower case letters ,  The length must be  16  position   If the length is not 16 Bit will result in failure to create a new data source **

** Particular attention :**

** this  key  Can only be changed during initial installation ! It can't be changed again later ! If it is changed again, the previously stored data source password cannot be decrypted , Finally, the relevant data source information cannot be obtained **

Use the help

./Yearning --help

2. Initialization and installation

./Yearning install

To install again , Please put yearning Delete all tables in the library , Otherwise, repeated execution is invalid

Successful initialization !

 Successful initialization !
 user name : admin
 password :Yearning_admin
 Please pass ./Yearning run  function , Default address :http://<host>:8000

3. Start the service

The default startup

./Yearning run

Parameter startup

./Yearning run --push "172.27.80.35" -port "8000"

[email protected]:/usr/local/Yearning# ./Yearning run --port "8008" # Modify the port 

Open the browser http://127.0.0.1:8000

Default account / password ：admin/Yearning_admin

docker The installation adopts local mysql database

Yearning  Is based on [Inception](https://so.csdn.net/so/search?q=Inception&spm=1001.2101.3001.7020) Of web visualization SQL Audit platform , It only provides visual interactive pages and does not have sql Ability to audit . So it must match Inception Use it together . Inception  Install the reference  [MySQL  Audit tools  Inception](https://blog.csdn.net/kk185800961/article/details/84863216).

Create database

CREATE DATABASE Yearning CHARACTER SET utf8mb4 COLLATE utf8mb4_bin;
grant all on Yearning.* to 'yearning'@'%' identified by 'sMHcsff32wUv38';

MySQL Table introduction in ：

mysql> show tables;
+----------------------------+
| Tables_in_yearning         |
+----------------------------+
| core_accounts              |      #  Save user information 
| core_auto_tasks            |      #  Automatic task 
| core_data_sources          |      #  data source 
| core_global_configurations |      #  management -- Set details 
| core_graineds              |      #  User permissions 
| core_query_orders          |      #  Query audit records 
| core_query_records         |      #  Query the detailed records next to the audit 
| core_role_groups           |      #  Permission group 
| core_rollbacks             |      #  Roll back sql
| core_sql_orders            |      #  Work order audit records 
| core_sql_records           |      #  Detailed records next to work order audit 
| core_workflow_details      |		#  My work order 
| core_workflow_tpls         |		#  The process template 
+----------------------------+
13 rows in set (0.01 sec)

start-up

Mode one

docker run -d   --name yearning   --restart always   -p 8000:8000   -e MYSQL_USER=yearning   -e MYSQL_ADDR=192.168.2.202:3306   -e MYSQL_PASSWORD=sMHcsff32wUv38   -e MYSQL_DB=Yearning   -e TZ="Asia/Shanghai"   test/yearning:latest

Mode two

docker run -d \
  --name yearning \
  --restart always \
  -p 8000:8000 \
  -e MYSQL_USER=yearning \
  -e MYSQL_ADDR=xxxxx:3306 \
  -e MYSQL_PASSWORD=sMHcsff32wUv38 \
  -e MYSQL_DB=Yearning \
  -e TZ="Asia/Shanghai" \
  test/yearning:latest

Default password ：admin/Yearning_admin

images Mirror image

Completion of construction

docker-compose build yearningSQL Audit platform

preparation

1. Close the firewall and selinux

2. Install well docker

install docker-compose

[[email protected] ~]# sudo curl -L "https://github.com/docker/compose/releases/download/1.27.3/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
[[email protected] ~]# sudo chmod +x /usr/local/bin/docker-compose
[[email protected] ~]# docker-compose version
docker-compose version 1.27.1, build 509cfb99
docker-py version: 4.3.1
CPython version: 3.7.7
OpenSSL version: OpenSSL 1.1.0l  10 Sep 2019

Create directory

mkdir -p /data/docker/mysql/3306/data/db
mkdir -p /data/docker/var/run/
mkdir -p /data/docker/mysql/3306/data/conf
touch /data/docker/var/run/mysql-3306.sock

To write docker-compose.yaml file

[email protected]:/yearning# docker images |grep yearning
test/yearning                   latest    3bd14792c8f1   5 months ago    29.4MB

version: '3.5'

services:
  mysql:
    image: mysql:5.7
    container_name: yearning-mysql
    volumes:
      - /data/docker/mysql/3306/data/db:/var/lib/mysql/
      - /data/docker/var/run/mysql-3306.sock:/var/run/mysql.sock
      - /data/docker/mysql/3306/data/conf:/etc/mysql/conf.d
    ports:
      - 3306:3306
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: 123456
      MYSQL_DATABASE: yearning
      MYSQL_USER: yearning
      MYSQL_PASSWORD: 123456
      TZ: Asia/Shanghai
    command:
      - --character-set-server=utf8mb4
      - --collation-server=utf8mb4_unicode_ci

  yearning:
    image: test/yearning:latest
    container_name: yearning-web
    environment:
      MYSQL_USER: yearning
      MYSQL_PASSWORD: 123456
      MYSQL_ADDR: mysql
      MYSQL_DB: yearning
    ports:
      - 8000:8000
    depends_on:
      - mysql
    restart: always

start-up

docker-compose up -d

visit

 account number ：admin
 password ：Yearning_admin

3、 ... and 、 Basic use

First of all, of course, is the login interface , The default login account password has been prompted during initialization ： user name : admin password :Yearning_admin.Yearning Recommended Chrome browser .

Service update

Yearning Adopt automatic table structure to synchronize There is no need to manually update the table structure . just Stop the original service and restart after replacing the installation package that will do

In some special upgrade cases ( Disruptive change ) You need to synchronize data manually . If there is no prompt for destructive upgrade in the version update announcement, ignore the following commands !

./Yearning migrate

After entering, it is the home page

The current number of users can be seen on the homepage 、 data source 、 Number of work orders 、 Number of queries 、 Personal information 、 Work order submission statistics 、 High frequency library 、 And work order trends .

Historical work orders

In my work order interface, you can see historical work orders , Execution status , Execution information , Whether to back up, etc ..

DML to examine

DML relevant SQL Submit audit ,SQL Syntax highlighting / Automatic completion

Database management

add to / edit / Delete data source

User permissions

Basic settings and advanced settings

Audit rules

Yearning Use secondary / Multi level audit mode , The relevant use process can be changed according to the actual needs , The executor role can only be specified after multi-level approval is enabled ( Open please go to the settings page ), If you need to change multi-level approval to second level approval , Please make sure that all multi-level approved work orders have been confirmed for execution . Otherwise, if the work order is not executed, it will not be retrieved . When multi-level approval is closed, the system will not automatically reset the role as the user of the executor , Please reset the corresponding user role

Secondary audit process ：

• 1. The user submits the unit to the corresponding work order according to his / her own permission (DDL,DML) Submit work order
• 2. After receiving the message, the administrator will review the work order request and execute it on the review work order page / rejected Corresponding work order
• 3. The execution record will be recorded under the administrator user

Multi level approval process ：

• 1. The user submits the unit to the corresponding work order according to his / her own permission (DDL,DML) Submit work order ,
• 2. After receiving the message, the administrator reviews the work order request on the review work order page and agrees / rejected Corresponding work order and select corresponding executor （ The executor must be a user whose role is executor ）
• 3. After the executor receives the work order perform / Reject the work order
• 4. The execution record will be recorded under the executor user

Set up SQL Detection rules

Usage flow

Open the management - user , First, create the submitter and reviewer users

Open the management - database , Add the database information to operate

Open the management - Permission group , Check database

Open the management - The process template

Open the management - Audit rules , Set audit conditions according to enterprise requirements

After setting, you need to authorize the user database

3、 ... and , User submits work order

The user logs in with the user name and password assigned by the administrator

Work order submission

View the submitted work order and notify the administrator to review

The management staff user receives the work order and executes

You can view the tables in the database , built .

yearning Relevant error reporting and troubleshooting

1- screening yearning-SQL Audit questions 


 Have a problem 
 Question 1 ：docker-compose Starting up yearning, visit web The default account cannot be logged in admin
 resolvent ： see docker Containers  logs journal , The error report is not configured correctly mysql database 

 Question two ：docker-compose Starting up yearning, visit web The default account cannot be logged in admin
 resolvent ：docker-compose Starting up mysql, Access cannot be connected , The solution is to temporarily change to local mysql database 




1-debian10 Install non docker Version database 
2- Authorize local mysql-yearning library , User permissions 

 Question why ：
 problem -1：mysql Start up , land mysql Report errors 
mysql: error while loading shared libraries: libncurses.so.5: cannot open shared object file: No suc
 resolvent ： Finding related documents is missing , You can't rely on soft connections , Thorough solution   sudo apt install libncurses5


 problem -2：docker Starting up yearning, land web The account number did not report an error , Find out yearning The version is a little low , Lack of permission group management ,
 resolvent ： Replace yearning Related image , You can't get , Take the group to find yearning Relevant latest image , It has been replaced to v2.8 Around the version 

 problem -3： Replace yearning-v2.8 Version image , Login appears web The account prompt account password is incorrect 
 resolvent ： Check the related mysql surface , The relevant account permission field is not inserted , The last reason   establish yearning library   It's case sensitive Yearning

Default account number on land admin
resolvent ：docker-compose Starting up mysql, Access cannot be connected , The solution is to temporarily change to local mysql database

1-debian10 Install non docker Version database
2- Authorize local mysql-yearning library , User permissions

Question why ：
problem -1：mysql Start up , land mysql Report errors
mysql: error while loading shared libraries: libncurses.so.5: cannot open shared object file: No suc
resolvent ： Finding related documents is missing , You can't rely on soft connections , Thorough solution sudo apt install libncurses5

problem -2：docker Starting up yearning, land web The account number did not report an error , Find out yearning The version is a little low , Lack of permission group management ,
resolvent ： Replace yearning Related image , You can't get , Take the group to find yearning Relevant latest image , It has been replaced to v2.8 Around the version

problem -3： Replace yearning-v2.8 Version image , Login appears web The account prompt account password is incorrect
resolvent ： Check the related mysql surface , The relevant account permission field is not inserted , The last reason establish yearning library It's case sensitive Yearning