Getting started with saltstack

One 、SaltStack Introduce

1.1 Automatic operation and maintenance tools

As an operator , A large part of the work is on business configuration management, state maintenance and version release , When the business scenario and the company scale reach a certain scale , It will be extremely difficult to do these tasks manually , At this time, we will need to use some automatic operation and maintenance tools to achieve the purpose of batch management .

Common automatic operation and maintenance tools include ：

• puppet
• ansible
• saltstack

These three are similar tools , Can be used to improve the efficiency of operation and maintenance management , But they have their own advantages , At present, the mainstream automatic operation and maintenance tool is ansible and saltstack. among ansible No need to install client , This is its biggest advantage , and saltstack You need to install the client tools , similar zabbix Of agent. In terms of application scenarios ,ansible Commonly used in small businesses , and saltstack It is often used in medium and large enterprises , because ansible Cannot execute in parallel and saltstack It can be done in parallel . But whatever its characteristics , Are essentially of the same kind , Therefore, you only need to master one to be easily competent for operation and maintenance work .

Can be SaltStack Understood as the magic pen Ma Liang's pen ！

1.2 saltstack Characteristics

• be based on python Developed C/S Architecture configuration management tools
• Bottom use ZeroMQ Message queue pub/sub Means of communication
• Use SSL Certification management shall be carried out in the way of certificate issuance , Transmission uses AES encryption

1.3 saltstack Service Architecture

stay saltstack The server side of the architecture is called Master, The client is called Minion.

stay Master and Minion Both ends run in daemon mode , Always listen to the... Defined in the configuration file ret_port( Accept minion request ) and publish_port( Release the news ) The port of .

When Minion The runtime will automatically connect to the... Defined in the configuration file Master Address ret_port Port for connection authentication .

saltstack Except for the traditional C/S Out of architecture , In fact, there is another kind called masterless The architecture of , It does not require a separate installation master The server , Just install... On each machine Minion End , Then adopt the mode that the local machine is only responsible for the service of the local configuration management mechanism .

Two 、SaltStack Four functions and four operation modes

SaltStack There are four functions , Namely ：

• Remote execution
• Configuration Management / State management
• Cloud management (cloud)
• Event driven

SaltStack Batch management can be realized through remote execution , And achieve the purpose of realizing some functions by describing the State .

SaltStack Four operation modes ：

• local Run locally （masterless No network mode ）
• Master/Minion The traditional way （ This one is used by default ）
• Syndic Distributed
• Salt ssh（agentless）

3、 ... and 、SaltStack Component is introduced

Components	function
Salt Master	Used to send commands and configurations to the server running on the managed system Salt minion
Salt Minions	from Salt master Receive commands and configuration
Execution Modules	A temporary command executed from the command line against one or more managed systems . Yes … Useful ： 1. Real-time monitoring , Status and inventory 2. One time commands and scripts 3. Deploy critical updates
Formulas (States)	Declarative or imperative representation of system configuration
Grains Grains	Is static information about the underlying managed system , Including the operating system , Memory and many other system properties .
Pillar	User defined variables . These security variables are defined and stored in Salt Master in , Then use the target “ Distribute ” To one or more Minion. Pillar Data storage such as ports , File path , Values such as configuration parameters and passwords .
Top File	take Formulas (States) and Salt Pillar Data and Salt minions matching .
Runners	stay Salt master Modules executed on , Used to perform support tasks .Salt runners Report job status , Connection status , From the outside API Reading data , Query connected Salt minions etc. .
Returners	take Salt minions The returned data is sent to another system , Like databases .Salt Returners Can be in Salt minion or Salt master Up operation .
Reactor	stay SaltStack Trigger reactions when an event occurs in the environment .
Salt Cloud / Salt Virt	In the cloud provider / The hypervisor provides the system , And immediately put it under management .
Salt SSH	In the absence of Salt minion On the system SSH function Salt command .

Four 、SaltStack install

Environmental statement ：

Host name	Host type	IP	Application to install
master	Control machine	192.168.91.135	salt-master salt-minion
minion	Controlled machine	192.168.91.137	salt-minion

official yum source address

4.1 Install... On the controller saltstack Master control software

//  Close the firewall and selinux
[[email protected] ~]# systemctl disable --now firewalld
Removed /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
[[email protected] ~]# vi /etc/selinux/config 
SELINUX=disabled
[[email protected] ~]# setenforce 0

//  To configure yum Source 
[[email protected] ~]# cd /etc/yum.repos.d/
[[email protected] yum.repos.d]# ls
CentOS-Stream-AppStream.repo
CentOS-Stream-BaseOS.repo
CentOS-Stream-Debuginfo.repo
CentOS-Stream-Extras.repo
CentOS-Stream-HighAvailability.repo
CentOS-Stream-Media.repo
CentOS-Stream-PowerTools.repo
CentOS-Stream-RealTime.repo
[[email protected] yum.repos.d]# rpm --import https://repo.saltproject.io/py3/redhat/8/x86_64/latest/SALTSTACK-GPG-KEY.pub

[[email protected] yum.repos.d]# curl -fsSL https://repo.saltproject.io/py3/redhat/8/x86_64/latest.repo | sudo tee /etc/yum.repos.d/salt.repo
[salt-latest-repo]
name=Salt repo for RHEL/CentOS 8 PY3
baseurl=https://repo.saltproject.io/py3/redhat/8/x86_64/latest
skip_if_unavailable=True
failovermethod=priority
enabled=1
enabled_metadata=1
gpgcheck=1
gpgkey=https://repo.saltproject.io/py3/redhat/8/x86_64/latest/SALTSTACK-GPG-KEY.pub

//  Clean cache 
[[email protected] yum.repos.d]# yum clean all
Failed to set locale, defaulting to C.UTF-8
21 files removed
[[email protected] yum.repos.d]# ls
CentOS-Stream-AppStream.repo
CentOS-Stream-BaseOS.repo
CentOS-Stream-Debuginfo.repo
CentOS-Stream-Extras.repo
CentOS-Stream-HighAvailability.repo
CentOS-Stream-Media.repo
CentOS-Stream-PowerTools.repo
CentOS-Stream-RealTime.repo
salt.repo

//  install saltstack Main control end 
[[email protected] yum.repos.d]# cd
[[email protected] ~]# dnf -y install salt-master salt-minion
 The installation process is a little bit ....

//  Modify the configuration file of the master 
[[email protected] ~]# vim /etc/salt/master
#master: salt
master: 192.168.91.135   //  stay #master Add this line after , Point to the control host IP Address 


//  Start the main control terminal salt-master and salt-master, And set the power on self start 
[[email protected] ~]# systemctl start salt-master
[[email protected] ~]# systemctl start salt-minion

[[email protected] ~]# systemctl enable --now salt-master
Created symlink /etc/systemd/system/multi-user.target.wants/salt-master.service → /usr/lib/systemd/system/salt-master.service.
[[email protected] ~]# systemctl enable --now salt-minion
Created symlink /etc/systemd/system/multi-user.target.wants/salt-master.service → /usr/lib/systemd/system/salt-minion.service.

//  Check for 4405、4406 Port number , Verify successful startup 
[[email protected] ~]# ss -antl
State  Recv-Q Send-Q Local Address:Port   Peer Address:Port Process                                                     
LISTEN 0      128          0.0.0.0:22          0.0.0.0:*                                                                
LISTEN 0      128          0.0.0.0:4505        0.0.0.0:*                                                                
LISTEN 0      128          0.0.0.0:4506        0.0.0.0:*                                                                
LISTEN 0      128             [::]:22             [::]:*    

4.2 Install... On the controlled machine salt-minion client

//  Close the firewall and selinux
[[email protected] ~]# systemctl disable --now firewalld
Removed /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
[[email protected] ~]# vi /etc/selinux/config 
SELINUX=disabled
[[email protected] ~]# setenforce 0

//  Pass on yun Source to the controlled machine 
[[email protected] ~]# scp /etc/yum.repos.d/salt.repo 192.168.91.137:/etc/yum.repos.d/
The authenticity of host '192.168.91.137 (192.168.91.137)' can't be established. ECDSA key fingerprint is SHA256:CHl+F/ZU1pC0SQ7a5/IlsoVQoRXN/TMa3ikGYM1hsoA. Are you sure you want to continue connecting (yes/no/[fingerprint])? yes Warning: Permanently added '192.168.91.137' (ECDSA) to the list of known hosts. [email protected]'s password: 
salt.repo                 100%  292    95.2KB/s   00:00  

[[email protected] ~]# cd /etc/yum.repos.d/
[[email protected] yum.repos.d]# ls
CentOS-Stream-AppStream.repo
CentOS-Stream-BaseOS.repo
CentOS-Stream-Debuginfo.repo
CentOS-Stream-Extras.repo
CentOS-Stream-HighAvailability.repo
CentOS-Stream-Media.repo
CentOS-Stream-PowerTools.repo
CentOS-Stream-RealTime.repo
salt.repo

[[email protected] yum.repos.d]# cd
[[email protected] ~]# dnf -y install salt-minion
 The installation process is a little bit ....

//  Modify the configuration file of the controlled end 
[[email protected] ~]# vim /etc/salt/master
#master: salt
master: 192.168.91.135  //  stay #master Add this line after , Point to the control host IP Address 


//  Start the controlled end salt-minion And set the power on self start 
[[email protected] ~]# systemctl start --now salt-minion
[[email protected] ~]# systemctl enable --now salt-minion
Created symlink /etc/systemd/system/multi-user.target.wants/salt-minion.service → /usr/lib/systemd/system/salt-minion.service.

4.3 saltstack The configuration file

saltstack The configuration file of is in /etc/salt Catalog

saltstack Profile description ：

The configuration file	explain
/etc/salt/master	Main control end ( Control terminal ) The configuration file
/etc/salt/minion	Controlled end configuration file

The configuration file /etc/salt/master The default configuration works well , Therefore, there is no need to modify this configuration file .

The configuration file /etc/salt/minion Common configuration parameters

• master： Set the settings of the master terminal IP
• id： Set the unique identifier of the controlled end , It can be ip It can also be a host name or a meaningful word

In daily use , Often need to adjust or modify Master The configuration file ,SaltStack Most configurations already specify default values , Just modify it according to your actual needs . The following parameters are important

• max_open_files： According to the Master take Minion Adjust the quantity properly
• timeout： According to the Master and Minion Adjust the network condition properly
• auto_accept and autosign_file： On a large scale Minion Automatic visa can be set when
• master_tops And all with external Parameters at the beginning ： These parameters are SaltStack Relevant configuration parameters for integration with external systems

5、 ... and 、SaltStack Authentication mechanism

saltstack The main control end depends on openssl Certificate to authenticate and communicate with the controlled end host , After the controlled end starts, it will send a public key certificate file to the master end , Use... At the main control end salt-key Command to manage certificates .

salt-minion And salt-master The certification process ：

• minion On the first start , Will be in /etc/salt/pki/minion/ Automatically generate a pair of keys , Then send the public key to master
• master received minion After the public key of , adopt salt-key The command accepts the public key . here master Of /etc/salt/pki/master/minions The directory will be stored in minion id Named public key , then master You can be right minion Send control command

salt-key Common options

Parameter name	effect
-L	List all public key information
-a minion	Accept designation minion Waiting for certification key
-A	Accept all minion Waiting for certification key
-r minion	Refuse to specify minion Waiting for certification key
-R	Reject all minion Waiting for certification key
-f minion	According to specified key Fingerprint information
-F	Show all key Fingerprint information
-d minion	Delete the specified minion Of key
-D	Delete all minion Of key
-y	Automatic answer yes

Automatically generate public key

[[email protected] ~]# cd /etc/salt
[[email protected] salt]# dnf -y install tree
[[email protected] salt]# tree pki/
pki/
|-- master
`-- minion |-- minion.pem // |-- minion.pub //  These two lines are a pair of keys generated automatically  `-- minion_master.pub

[[email protected] salt]# tree pki/
pki/
|-- master
|   |-- master.pem
|   |-- master.pub
|   |-- minions	//  Accepted, right here 
|   |   `-- minion //  This is acceptance minion The public key of the host  | |-- minions_autosign | |-- minions_denied //  Rejection is here  | |-- minions_pre //  Key acceptance is here  | | `-- master
|   `-- minions_rejected // -r  Designate to refuse to come here  `-- minion
    |-- minion.pem
    `-- minion.pub 7 directories, 6 files //  Check the public keys on both sides and compare  [[email protected] salt]# cat pki/master/minions/minion  -----BEGIN PUBLIC KEY----- MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAo/aG5BHskiYf7sM0WHEo yMlP5maj5VuIwPE1ZbN0rrbK6T6WKjUuLBJiezEtsGGi7MwQJ/v7GqiyucSoyGX1 f9tBpoPy86US8iAdTXCtc6BXc0wzrcQWmH2BCQ0I+WrmKfhX4Q9OHJyqaJOM8eNn s+uNjObMs2P/A0Ucc0hHS6l74KaslMqWpo/iQ8fEN6VUO1QwKQ/jcJpvlH9/SirV obIcdDqwtQtLc3Oo1A6s/ejZ2tb4fI4i3Y3bpeM8sjP5CbABiTTXTdycsTnKmMfD LCvcZb0WCxWQ2snJjwTxJk12hi5tTObraQYf/5Qwl3S3iVfOBztq7e0vbBqUvA5E IQIDAQAB -----END PUBLIC KEY----- [[email protected] salt]# cat pki/minion/minion.pub  -----BEGIN PUBLIC KEY----- MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAo/aG5BHskiYf7sM0WHEo yMlP5maj5VuIwPE1ZbN0rrbK6T6WKjUuLBJiezEtsGGi7MwQJ/v7GqiyucSoyGX1 f9tBpoPy86US8iAdTXCtc6BXc0wzrcQWmH2BCQ0I+WrmKfhX4Q9OHJyqaJOM8eNn s+uNjObMs2P/A0Ucc0hHS6l74KaslMqWpo/iQ8fEN6VUO1QwKQ/jcJpvlH9/SirV obIcdDqwtQtLc3Oo1A6s/ejZ2tb4fI4i3Y3bpeM8sjP5CbABiTTXTdycsTnKmMfD LCvcZb0WCxWQ2snJjwTxJk12hi5tTObraQYf/5Qwl3S3iVfOBztq7e0vbBqUvA5E IQIDAQAB //  It is found that the public key is the same  -----END PUBLIC KEY----- //  Use -r  Specify reject  master  Then we'll get to minions_rejected It's inside  [[email protected] salt]# salt-key -r master The following keys are going to be rejected: Unaccepted Keys: master Proceed? [n/Y] Y Key for minion master rejected. [[email protected] salt]# tree pki pki |-- master | |-- master.pem | |-- master.pub | |-- minions | | `-- minion
|   |-- minions_autosign
|   |-- minions_denied
|   |-- minions_pre
|   `-- minions_rejected | `-- master
`-- minion |-- minion.pem `-- minion.pub

7 directories, 6 files

//  If you want to continue to accept master, Just move back 
[[email protected] salt]# mv pki/master/minions_rejected/master pki/master/minions_pre/
[[email protected] salt]# tree pki
pki
|-- master
|   |-- master.pem
|   |-- master.pub
|   |-- minions
|   |   `-- minion | |-- minions_autosign | |-- minions_denied | |-- minions_pre | | `-- master
|   `-- minions_rejected `-- minion
    |-- minion.pem
    `-- minion.pub

7 directories, 6 files

Some parameter examples

//  View all public keys key
[[email protected] ~]# salt-key -L
Accepted Keys:		//  Accepted key
Denied Keys:		//  refuse key
Unaccepted Keys:	//  Not accepted key
master
minion
Rejected Keys:		//  Rejected key

//  Accept the appointed minion The host key
[[email protected] ~]# salt-key -a minion
The following keys are going to be accepted:
Unaccepted Keys:
minion
Proceed? [n/Y] Y
Key for minion minion accepted.
[[email protected] salt]# salt-key -L
Accepted Keys:
minion
Denied Keys:
Unaccepted Keys:
master
Rejected Keys:

//  control minion host 
[[email protected] ~]# salt '*' test.ping
minion:
    True   //  success 
    
//  Accept all hosts key
[[email protected] ~]# salt-key -yA
The following keys are going to be accepted:
Unaccepted Keys:
master
Key for minion master accepted.
[[email protected] ~]# salt-key -L
Accepted Keys:
master
minion
Denied Keys:
Unaccepted Keys:
Rejected Keys:

Certificate movement

[[email protected] salt]# tree pki/
pki/
|-- master
|   |-- master.pem
|   |-- master.pub
|   |-- minions
|   |   |-- master
|   |   `-- minion | |-- minions_autosign | |-- minions_denied | |-- minions_pre | `-- minions_rejected
`-- minion |-- minion.pem |-- minion.pub `-- minion_master.pub

//  Because the transfer file is rejected key Under the directory 
[[email protected] salt]# mv /etc/salt/pki/master/minions/minion /etc/salt/pki/master/minions_rejected/
[[email protected] salt]# tree pki/
pki/
|-- master
|   |-- master.pem
|   |-- master.pub
|   |-- minions
|   |   `-- master | |-- minions_autosign | |-- minions_denied | |-- minions_pre | `-- minions_rejected
|       `-- minion `-- minion
    |-- minion.pem
    |-- minion.pub
    `-- minion_master.pub 7 directories, 7 files //  So I can't ping through  [[email protected] salt]# salt "minion" test.ping No minions matched the target. No command was sent, no jid was assigned. ERROR: No return received //  Just move the file back ping through  [[email protected] salt]# mv /etc/salt/pki/master/minions_rejected/minion /etc/salt/pki/master/minions/ [[email protected] salt]# tree pki/pki/ |-- master | |-- master.pem | |-- master.pub | |-- minions | | |-- master | | `-- minion
|   |-- minions_autosign
|   |-- minions_denied
|   |-- minions_pre
|   `-- minions_rejected `-- minion
    |-- minion.pem
    |-- minion.pub
    `-- minion_master.pub

[[email protected] salt]# salt "minion" test.ping
minion:
    True

modify id, Appoint IP visit

//  List all public keys 
[[email protected] ~]# salt-key -L
Accepted Keys:
Denied Keys:
Unaccepted Keys:
master
minion
Rejected Keys:

//  Specify the delete minion host 
[[email protected] ~]# salt-key -yd minion
The following keys are going to be deleted:
Accepted Keys:
minion
Key for minion minion deleted.

[[email protected] ~]# salt-key -L
Accepted Keys:
master
Denied Keys:
Unaccepted Keys:
Rejected Keys:

//  Appoint master Host Authentication 
[[email protected] ~]# salt-key -ya "master"
The following keys are going to be accepted:
Unaccepted Keys:
master
Key for minion master accepted

[[email protected] ~]# salt-key -L
Accepted Keys:
master
Denied Keys:
Unaccepted Keys:
Rejected Keys:

//  Delete the key of the controlled machine 
[[email protected] ~]# cd /etc/salt/pki/minion/
[[email protected] minion]# ls
minion.pem  minion.pub  minion_master.pub
[[email protected] minion]# rm -f *
[[email protected] minion]# ls

//  modify id
[[email protected] minion]# vim /etc/salt/minion
#id:
id: 192.168.91.137	//  Add the of the controlled machine IP Address 

//  Restart 
[[email protected] minion]# systemctl restart --now salt-minion
//  Automatically generate Certificate 
[[email protected] minion]# ls
minion.pem  minion.pub


//  see ip Whether the named host exists 
[[email protected] ~]# salt-key -L
Accepted Keys:
master
Denied Keys:
Unaccepted Keys:
192.168.91.137
Rejected Keys:

//  Authenticate the host 
[[email protected] ~]# salt-key -ya 192.168.91.137
The following keys are going to be accepted:
Unaccepted Keys:
192.168.91.137
Key for minion 192.168.91.137 accepted.
//  Check the certification 
[[email protected] ~]# salt-key -L
Accepted Keys:
192.168.91.137
master
Denied Keys:
Unaccepted Keys:
Rejected Keys:

// ping All hosts 
[[email protected] ~]# salt "*" test.ping
master:
    True
192.168.91.137:
    True

//  The master controller certificate is automatically transmitted to the controlled machine 
[[email protected] minion]# ls
minion.pem  minion.pub  minion_master.pub

6、 ... and 、SaltStack Remote execution

//  Test the specified controlled end minion Whether the host is alive or not 
[[email protected] ~]# salt "minion" test.ping
minion:
    True
[[email protected] ~]# salt "minion" cmd.run hostname
minion:
    minion

//  Test whether all controlled end hosts survive 
[[email protected] ~]# salt "*" test.ping
minion:
    True
master:
    True

7、 ... and 、salt Command to use

 grammar ：salt [options] '<target>' <function> [arguments]

frequently-used options

Parameter name	effect
–version	see saltstack Version number of
–versions-report	see saltstack And the version number of the dependent package
-h	View help information
-c CONFIG_DIR	Specify the profile directory ( The default is /etc/salt/)
-t TIMEOUT	Specify the timeout period ( The default is 5s)
–async	Asynchronous execution
-v	verbose Pattern , Show the execution process in detail
–username=USERNAME	Specify the external authentication user name
–password=PASSWORD	Specify the external authentication password
–log-file=LOG_FILE	Specify logging file

Commonly used target Parameters

Parameter name	effect
-E	Regular matching
-L	The list matches
-S	CIDR Matching network segment
-G	grains matching
–grain-pcre	grains Add regular match
-N	Group matching
-R	Range match
-C	Comprehensive match （ Specify multiple matches ）
-I	pillar Values match

//  Regular matching 
[[email protected] ~]# salt -E 'min*' test.ping
minion:
    True

//  The list matches 
[[email protected] ~]# salt -L master,minion test.ping
master:
    True
minion:
    True

//  Segment matching 
[[email protected] ~]# salt -S '192.168.91.0/24' test.ping
master:
    True
minion:
    True

//  Match according to the system 
[[email protected] ~]# salt -G 'os:CentOS Stream' test.pingmaster:
    True
minion:
    True


//  Group matching 
[[email protected] ~]# vim /etc/salt/master
# as the main master config file).
default_include: master.d/*.conf	//  uncomment 

//  Create corresponding directories and files 
[[email protected] ~]# mkdir -p /etc/salt/master.d/
[[email protected] ~]# touch /etc/salt/master.d/nodegroups.conf

 Add new groups and members （ Host after group , The host name should be written salt-key Accepted host name . No need to reboot master）
[[email protected] ~]# vim /etc/salt/master.d/nodegroups.conf
nodegroups:
    group1: 'master'

[[email protected] ~]# salt -N 'group1' 'test.ping'
master:
    True


//  Combination match 
[[email protected] ~]# salt -C '[email protected],minion or [email protected]/24' test.ping
minion:
    True
master:
    True
#[email protected]:CentOS Stream or [email protected] It's a composite combination , Support use and and or Associate multiple conditions 

//  Universal module use  -- cmd.run
 Four hosts are used here 

[[email protected] ~]# salt-key -L
Accepted Keys:
192.168.91.137
master
node2
node3
Denied Keys:
Unaccepted Keys:
Rejected Keys:
[[email protected] ~]# salt "*" test.ping
node2:
    True
master:
    True
192.168.91.137:
    True
node3:
    True

//  The universal module depends on the running time 
[[email protected] ~]# salt "*" cmd.run "uptime"
node2:
     14:02:17 up  3:08,  2 users,  load average: 0.14, 0.09, 0.03
192.168.91.137:
     14:02:17 up  4:24,  2 users,  load average: 0.18, 0.20, 0.09
master:
     14:02:17 up  4:24,  2 users,  load average: 0.24, 0.08, 0.02
node3:
     14:02:17 up  3:01,  2 users,  load average: 0.03, 0.05, 0.01

//  The universal module looks at the date 
[[email protected] ~]# salt "*" cmd.run "date"
node2:
    Tue Nov  2 14:04:40 CST 2021
node3:
    Tue Nov  2 14:04:40 CST 2021
192.168.91.137:
    Tue Nov  2 14:04:40 CST 2021
master:
    Tue Nov  2 14:04:40 CST 2021