Small project on log traffic monitoring and early warning | environment foundation 2

Mysql： Facilitate data persistence

MySQL The content of ：

1. install
   1. edition ：5.7.34 Binary installation
      1. Business Edition charge    Community Edition free
      2. MySQL Support distributed and clustered Such as ： Ali tencent Baidu Meituan Xiaomi and others are used MySQL
   2. Script installation   Installation package mysql Official website
2. Building tables and databases

Library in mysql It is equivalent to a folder , It's a place to store data

Building database ：create database Library name ;

Check the library ：use Library name ;

# Create table

create table nginxlog (

id  int primary key auto_increment,

dt  datetime not null,

prov int ,

isp  int,

bd  float（ bandwidth ）

) CHARSET=utf8;

surface ： A file for storing data , It can be understood as one that only MySQL Readable documents , The data is stored in this file line by line

See the table ：show tables;

Field ： A field is a column

primary key： Do primary key （ Each row of this column cannot be the same ）

auto_increment： Self increasing

datetime： The date will be recorded Hours, minutes and seconds

not null： This column cannot be empty

CHARSET=utf8 Set the character set to utf8

desc indicate ; Look at the structure of this table

show create table Table name ： Look at the initial statement that created this table

drop table Table name ; Delete table

drop database Library name ; Delete Library （ Including all the tables inside ）

select * from Table name ; Query all the data in this table （* Represents all fields You can also select one or more fields Separated by commas ）

insert into indicate （ One or more fields ） values（ Each value is separated by a comma , The string is enclosed by a colon ）,（）,（）; Insert multiple data （ The front and back correspond one by one ）

varchar（ Set the maximum length ）： Variable length string  

row That's ok

record Record That is, line by line

/data/mysql: There are two structures in the data directory

.frm Storage table structure

.ibd Storing indexes and real data

Delete data ：delete from Table name where Conditions （ Such as id=2）;

Modifying data ：update Table name set Modified value （ Such as isp=3） where Conditions （ Such as id=3）;

1. Additions and deletions
2. Pymysql Library usage

Mysql Configuration file for ：/etc/my.cnf

Data directory ：/data/mysql

Mysql Two mainstream versions ：5.7.34（ Stable ）8.0.27（ new function unstable ）

Python Of pymysql library ：python In the operation MySQL The library of （ Additions and deletions ）

‘[email protected]’: Only allow login on this computer Cannot log on remotely

Create users and authorizations

These two users are not one user （ One is 0 Users of network segment There is also a user connected to any network segment ）

first sc User priority of is lower than （ Because the user below is more accurate Therefore, the priority is higher ）

Remote login will automatically connect to the higher priority

grant all on *.* to 'sc'@'%';

effect ： Giving ‘sc’@‘%’ All permissions of this user （ In addition to the addition, deletion and modification of the search ） Include add delete change check

grant It is an authorized order

all On behalf of all authorities , Include add delete change check , But not including grant jurisdiction

on *.*： On all libraries , Operate on all tables （ first * On behalf of Ku the second * Represents all tables ）

to ‘sc’@‘%’ To this user

grant You can also create users while authorizing ：

show warnings; Is to view warnings

show processlist; See who logged in remotely

select version(); Check the version of the database

import pymysql

db=pymysql.connect(host='192.168.0.36',user='sc',password='Sanchuang123#',database='mysql')  # Create an object You can also leave the library unspecified （ It's specified here mysql library , After that, there is no need for use Switch Libraries ）

cursor=db.cursor() # Create a cursor

Database cursor ： Just a temporary cache space in the database , Cursors help execute sql Object of statement

You name .execute(“sql sentence ”)： The number returned is the number of rows of the result

You name .fetchone() Get data one by one （ There will be tour marks to which line you are admitted ）

You name .fetchmany()

You name .fetchall()  Get all the data

Close the connection （ The cursor name disappears ）：

Connection name .close()

Mysql Commands are not case sensitive Table name, function name, password, etc. are case sensitive

establish hejin In the database nginxlog surface

Mysql There's a now function Therefore, when transferring values, you can now（）

Database transactions ： A group of （ Or a ） Operations make up a transaction

A transaction consists of two results ： All or nothing Or failure

commit Submit ： This means that all operations in this transaction have been successfully completed （ Data is written from memory to disk   At first, the data will only be stored in memory buffer in , Then drop the plate to disk Inside Keep forever Persistence ）

rollback Roll back （ revoke ）： Everything that has just been done is undone , Rollback to the original state （ An operation failed All three operations are undone ）

The operation of creating tables and databases will automatically commit and rollback There's no need for us to intervene

And the operation of moving data （insert update delete ） You can use commit and rollback

try:

   # perform sql sentence

   cursor.execute(sql)

   # Commit to database execution

   db.commit()

except:

   # Roll back if an error occurs

   db.rollback()

As long as the primary keys are different, they are different records , If the primary key is set to increase automatically , If it is not specified, it will be added automatically 1

Automatically give the offset to kafka preservation

Judge url get Whether the request is successful 200 On behalf of success

Splitting tables is to reduce storage Convenient access

# For sub table , Our script should be modified accordingly .

# Insert data into nginxlog Before the table

#1、 obtain prov_index and isp_index Data in table

#2、 Determine what to insert prov and isp In the data not in the table

#3、 If prov_index and isp_index Data exists , Get the corresponding id, Store it again

#4、 If prov_index and isp_index The data doesn't exist , Just insert the new data into these two tables , And then get id Field , Store in the data table nginxlog

flush privileges; Refresh the permissions

copy on right： Realistic reproduction