Bash shell learning notes (V)

Course objectives

• Master the meaning of common regular expression metacharacters
• master sed Delete row
• master sed Print line operation of
• master sed Add line operation
• master sed Modification and replacement of

Regular expressions

Regular expression Introduction

Regular expressions （Regular Expression、regex or regexp, Abbreviation for RE）, Also translated as normal notation 、 Conventional representation , It's a character pattern , Used to match a specified character during a lookup .

Almost all development languages support regular expressions , Later learning python There are regular expressions in language .

linux The main commands that support regular expressions in grep, sed, awk

Regular one

expression	explain	Example
[]	Choose one of the characters in brackets	[abc][0-9][a-z]
[^]	Does not match any characters in parentheses ( In parentheses ^ Number is " Not ", It's not the beginning )	[^abc] It means not a, No b, No c
^ []	Start with any single character in parentheses ( there ^ It means the beginning )	^ [abc]: With a or b or c start
^ [^]	Do not start with any single character in parentheses
^	The beginning of a row	^root
$	End of line	bash$
^$	Blank line

Example : Prepare a document

# vim 1.txt
boot
boat
rat
rot
root
Root
brot.

 Find yes rat or rot Lines of characters 
# grep r[oa]t 1.txt 
rat
rot
brot.

 View non r The beginning of a character , but 2-4 The characters are oot The line of 
# grep ^[^r]oot 1.txt
boot
Root

 Find a non capitalized letter with a o Character connected lines 
# grep '[^A-Z]o' 1.txt
boot
boat
rot
root				
Root			#  This can also be found , Because the first 2-3 Characters meet the requirements 
brot.

 Search not to r and b Beginning line 
# grep ^[^rb] 1.txt
Root

 Looking to rot Begin with rot The line at the end ( That is, there are only rot Three characters )
# grep ^rot$ 1.txt
rot

 lookup . Characters at the end of the sign , Need to escape and quote ( A special , Because I will talk about . No. is a special metacharacter )
# grep "\."$ 1.txt
brot.

problem : Use grep Output /etc/vsftpd/vsftpd.conf Configuration in configuration file ( Remove comments and blank lines )

 Tips : grep -v Take the opposite 
 Don't forget grep -E Extension mode can be implemented or 
 Such as : grep -E "root|ftp" /etc/passwd

# yum install vsftpd -y


# grep -v ^# /etc/vsftpd/vsftpd.conf |grep -v ^$
# grep -v '^#|^$' /etc/vsftpd/vsftpd.conf

problem : Use grep Output /etc/samba/smb.conf Configuration in configuration file ( Remove comments and blank lines )

# yum install samba -y

# grep -v -E '#|^;|^$' /etc/samba/smb.conf |grep [a-z]
 notes :
samba Configuration file is a wonderful configuration file , The format is different in different versions , So different versions will have different answers . Please be flexible 

grep Add keyword color display （centos7 There are colors by default )

 stay /etc/bashrc Add a sentence in the last blank 
# vim /etc/bashrc
alias grep='grep --color=auto'

# source /etc/bashrc

Regular two

expression	function
[[:alnum:]]	Upper and lower case letters and numbers
[[:alpha:]]	Alphabetic character ( Include upper and lower case letters )
[[:blank:]]	Spaces and tabs
[[:digit:]] or [0-9] ( Commonly used )	Numbers
[[:lower:]] or [a-z] ( Commonly used )	Lowercase letters
[[:upper:]] or [A-Z] ** ( Commonly used **)	Capital
[[:punct:]]	Punctuation
[[:space:]]	Include line breaks , All the blanks in the carriage return

 Find lines that don't start with uppercase letters 
# grep '^[^[:upper:]]' 1.txt #  This negative expression is very special ,^ The symbol is between two brackets ( Understanding can )
# grep '^[^A-Z]' 1.txt 
# grep -v '^[A-Z]' 1.txt 

 Find the line with the number 
# grep '[0-9]' 1.txt
# grep [[:digit:]] 1.txt

 Find a line with a number and a letter 
# grep -E '[0-9][a-Z]|[a-Z][0-9]' grep.txt # grep -E Is the extended mode , middle | The symbols " perhaps "

problem : Excuse me, Chinese description grep [^a-z] 1.txt What to look for ?

 Find lines that are not all lowercase 

Example : Enter a character , Judge whether the input is capital letters, lowercase letters or numbers , Or else?

 Method 1 :
#!/bin/bash

read -n 1 -p "input a char:" char			# -n 1 For input 1 Enter automatically after characters 
echo										# echo On behalf of the line 

case "$char" in							#  Use case The statement of $char Process multi branch judgment 
	[[:upper:]] )						#  Regular expressions match uppercase letters ,( Be careful : Here we use [A-Z] It doesn't match )
		echo " Capital "
		;;
	[[:lower:]] )						#  Regular expressions match lowercase letters ,( Be careful : Here we use [a-z] It doesn't match )
		echo " Lowercase letters "
		;;
	[[:digit:]] )						#  Regular expressions match numbers 
		echo " Numbers "
		;;
	* )
		echo " Other "
esac

 Method 2 :
#!/bin/bash

read -n 1 -p "input a char:" char			# -n 1 For input 1 Enter automatically after characters 
echo										# echo On behalf of the line 

#  Use if Multiple branches for judgment 
if [[ $char =~ [A-Z] ]];then			# [[ ]] The judgment conditions are written in two brackets , It uses =~ Match regular expression 
	echo " Capital "
elif [[ $char =~ [a-z] ]];then
	echo " Lowercase letters "
elif [[ $char =~ [0-9] ]];then
	echo " Numbers "
else
	echo " Other "
fi

practice : read -s Enter a password , Determine the complexity of this password .( The length is greater than or equal to 8 position , And there are three types of characters, upper and lower case letters and numbers )

Regular three

A term is used to explain ：

Metacharacters : Those that have in regular expressions Special characters with special meaning , Such as : spot (.) star (*) question mark (?) etc.

Leading characters ： The character before the metacharacter abc * aooo .

Be careful : Metacharacters need escape if they want to express the character itself , Such as . No. just wants to match "." The number itself needs to be used \.

character	Character description	Example
*	The leading character appears 0 Several or more times in a row	ab* abbbb
.	Except for line breaks , Any single character	ab. ab8 abu
.*	Characters of any length	ab.* abdfdfdf
{n}	Leading characters appear consecutively n Time （ Need to cooperate with grep -E or egrep Use )	[0-9]{3}
{n,}	The leading character appears at least n Time （ Need to cooperate with grep -E or egrep Use )	[a-z]{4,}
{n,m}	Leading characters appear consecutively n To m Time （ Need to cooperate with grep -E or egrep Use )	o{2,4}
+	The leading character appears 1 Times or more （ Need to cooperate with grep -E or egrep Use )	[0-9]+
?	The leading character appears 0 Time or 1 Time （ Need to cooperate with grep -E or egrep Use )	go?
( )	Group character

Sample text :

# vim 2.txt
ggle
gogle
google
gooogle
gagle
gaagle
gaaagle
abcgef
abcdef
goagle
aagoog
wrqsg

problem : Let's see which characters can be found below , Understand memory through results

# grep g. 2.txt
# grep g* 2.txt #  The result is strange 
# grep g.g 2.txt
# grep g*g 2.txt
# grep go.g 2.txt
# grep go*g 2.txt
# grep go.*g 2.txt
# grep -E go{2}g 2.txt
# grep -E 'go{1,2}g' 2.txt #  Need quotation marks , Either single or double citation is OK 
# grep -E 'go{1,}g' 2.txt #  Need quotation marks , Either single or double citation is OK 
# grep -E go+g 2.txt
# grep -E go?g 2.txt

Example : Find out eth0 NIC IP, Broadcast address , Subnet mask

# ifconfig eth0 | grep -E -o '([0-9]{1,3}\.){3}[0-9]{1,3}'
10.1.1.11
10.1.1.255
255.255.255.0
 analysis :
[0-9]{
    1,3}\.  	    representative 3 Number after number . Number 
([0-9]{
    1,3}\.){
    3}   Make a group with parentheses in front , Back {
    3} Represents repetition 3 Time 
[0-9]{
    1,3}		    There is no need to . Number 

Example : Match email address

# echo "[email protected]" | grep -E '^[0-9a-zA-Z][email protected][a-z0-9]+\.[a-z]+$'
 analysis :
^[0-9a-zA-Z]+@      	 representative @ The symbol is preceded by 1 Start with one or more characters ( Upper and lower case letters or numbers )
@[a-z0-9]+\.[a-z]+$	 	 representative @ Symbols and . Number ( Be careful . The number should be escaped ) There is 1 Characters or more ( Small letters or numbers )
						. There is 1 Characters or more ( Lowercase letters ) ending 

perl Built in regular ( expand )

Perl Built in regular ( Need to use grep -P To match ), This matching method is used in python Also have . But it is not recommended to remember , What you have learned above is enough .

\d       Match the Numbers   [0-9]
\w       Match alphanumeric underscores [a-zA-Z0-9_]
\s       Match spaces 、 tabs 、 Page identifier [\t\r\n]

sed

sed Introduce

# man sed
sed  - stream editor for filtering and trans-forming text

Windows Under the editor :

linux Under the editor :

• vi or vim
• gedit
• emacs etc.

• First sed Save the row currently being processed in a temporary buffer （ Also known as pattern space ）, Then process the rows in the temporary buffer , When you're done, send the line to the screen .
• sed Put each line in a temporary buffer , For this copy Editing , So it won't change the original file . Of course, you can also choose to modify the source file , need sed -i Files for operation

Study sed The key is to find out , It's a flow Editor , The common functions of the editor are :

• Delete That's ok
• Print That's ok
• increase That's ok
• Replace ( modify )

sed Parameters 
-e	 Make multiple edits , That is, apply multiple sed Use 
-n	 Cancel the default output 
-r   Using extended regular expressions 
-i inplace, Edit in place （ Modify source file ）

sed operation 
d   Delete row 
p   Print row 
a   Add a line after 
i   Add a line ahead 
s   Replace, modify 

Delete row operation

d(delete) On behalf of the delete

Match lines with numbers

Specify to delete the 2 That's ok

# head -5 /etc/passwd |cat -n |sed 2d

 Variable references require double quotes 
a=2					
# head -5 /etc/passwd |cat -n |sed "$a"d 

Delete the first 2 Go to the first place 3 That's ok , The comma in the middle indicates the range

# head -5 /etc/passwd |cat -n |sed 2,3d

Delete the first 1 Xing He 5 That's ok , Semicolon in the middle , Represents a separate operation

 error , Need quotation marks 
# head -5 /etc/passwd |cat -n |sed 1d;5d
 correct 
# head -5 /etc/passwd |cat -n |sed '1d;5d'

Delete the first 1,2,4 That's ok , -e Parameters connect different operations

head -5 /etc/passwd |cat -n |sed -e '2d;4d' -e '1d'

problem : What does the following operation represent

# head -n 5 /etc/passwd |cat -n |sed '1d;3d;5d'

# head -n 5 /etc/passwd |cat -n |sed '1,3d;5d'

Use regular matching lines

If you don't know the line number , But know a word or related character in the line , We can use regular expressions to match

Delete match oo The line of

# head -n 5 /etc/passwd |cat -n |sed '/oo/d'

Delete with root Beginning line

# head -n 5 /etc/passwd |sed '/^root/d'

Delete with bash The line at the end

# head -n 5 /etc/passwd |sed '/bash$/d'

Any other regular expression can match

practice :( Be careful : -i Parameters will directly manipulate the source file , So please don't add -i test , test OK Then add -i Parameters )

sed -i Delete /etc/vsftpd/vsftpd.conf All comments and blank lines in

# sed -i '/^#/d;/^$/d' /etc/vsftpd/vsftpd.conf

sed -i Delete /etc/samba/smb.conf All comments and blank lines in

# sed -i '/#/d;/;/d;/^$/d;/^\t$/d' /etc/samba/smb.conf

Print line operation

Print row ( Deleted antonym )

p(print) For printing

Match lines with numbers

Print page 1 That's ok

# head -5 /etc/passwd |cat -n | sed 1p
     1  root:x:0:0:root:/root:/bin/bash				#  Will be in the original 5 Print the second line on the basis of 1 That's ok 
     1  root:x:0:0:root:/root:/bin/bash
     2  bin:x:1:1:bin:/bin:/sbin/nologin
     3  daemon:x:2:2:daemon:/sbin:/sbin/nologin
     4  adm:x:3:4:adm:/var/adm:/sbin/nologin
     5  lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin

# head -5 /etc/passwd |cat -n | sed -n 1p #  The right way to add one -n Parameters 
     1  root:x:0:0:root:/root:/bin/bash

Print page 1 Xing He 4 That's ok

# head -5 /etc/passwd |sed -ne '1p;4p'

Print 1-4 That's ok

# head -5 /etc/passwd |sed -ne '1,4p'

Use regular matching lines

Printed with root Keyword line

# head -5 /etc/passwd |sed -n '/root/p'

Print to root Beginning line

# head -5 /etc/passwd |sed -n '/^root/p'

find var/log/secure Log in successfully ssh Information

 Method 1:
# sed -n '/Accepted/p' /var/log/secure
 Method 2:
# grep Accepted /var/log/secure
 Method 3:(awk I haven't learned yet , So let's see )
# awk '$0~"Accepted" {print $0}' /var/log/secure

Add row operation

a(append) On behalf of the next line

i(insert) Represents the line inserted before

Prepare a document

# cat 1.txt
11111
22222
44444
55555

In the 2 Add after the line 33333 This business

# sed -i '2a33333' 1.txt
# cat 1.txt
11111
22222
33333
44444
55555

In the 1 Row insertion 00000 This business

# sed -i '1i00000' 1.txt
# cat 1.txt
00000
11111
22222
33333
44444
55555

It can also be used. Regular Match lines , It's in 4 Add ccccc This business

# sed -i '/^4/accccc' 1.txt 

# cat 1.txt
00000
11111
22222
33333
44444
ccccc
55555

Modify the replacement operation

sed The modified replacement format is the same as vi The format of modification and replacement in is the same

Match lines with numbers

Replace the... In each line 1 Matching characters

# head -5 /etc/passwd |sed 's/:/===/'
root===x:0:0:root:/root:/bin/bash
bin===x:1:1:bin:/bin:/sbin/nologin
daemon===x:2:2:daemon:/sbin:/sbin/nologin
adm===x:3:4:adm:/var/adm:/sbin/nologin
lp===x:4:7:lp:/var/spool/lpd:/sbin/nologin

Replace all

# head -5 /etc/passwd |sed 's/:/===/g'
root===x===0===0===root===/root===/bin/bash
bin===x===1===1===bin===/bin===/sbin/nologin
daemon===x===2===2===daemon===/sbin===/sbin/nologin
adm===x===3===4===adm===/var/adm===/sbin/nologin
lp===x===4===7===lp===/var/spool/lpd===/sbin/nologin

Replace 2-4 That's ok

# head -5 /etc/passwd |sed '2,4s/:/===/g'
root:x:0:0:root:/root:/bin/bash
bin===x===1===1===bin===/bin===/sbin/nologin
daemon===x===2===2===daemon===/sbin===/sbin/nologin
adm===x===3===4===adm===/var/adm===/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin

Replace 2 Row sum 4 That's ok

# head -5 /etc/passwd |sed '2s/:/===/g;4s/:/===/g'
root:x:0:0:root:/root:/bin/bash
bin===x===1===1===bin===/bin===/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm===x===3===4===adm===/var/adm===/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin

Replacement section 2 OK, No 1 And the first 3 Matching characters

 Note that the following number is 2, Replaced 1 individual , Replace the rest with 2 One is the original number 3 individual 
# head -5 /etc/passwd |sed '2s/:/===/;2s/:/===/2'
root:x:0:0:root:/root:/bin/bash
bin===x:1===1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin

Use regular matching lines

Replace with daemon The first line

# head -5 /etc/passwd |sed '/^daemon/s/:/===/g'
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon===x===2===2===daemon===/sbin===/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin

& The sign is in sed The replacement represents the character that has been replaced before

 Understand the difference between the following two sentences :
s/[0-9]/ &/
s/[0-9]/ [0-9]/

practice : Use the script to modify the hostname ( The assumption is changed to a.b.com)

# hostname a.b.com
# sed -i '/HOSTNAME=/s/server.cluster.com/a.b.com/' /etc/sysconfig/network
# echo "10.1.1.11 a.b.com" >> /etc/hosts

practice : modify /etc/selinx/config The configuration file , take selinux close

# sed -i '/SELINUX=/s/enforcing/disabled/' /etc/selinux/config

sed Domain operation ( expand )

take () String between ( Generally regular expressions ) Defined as a group , And save the matching expression to a region （ A regular expression can save at most 9 individual ）, They use \1 To \9 To express . And then replace it

Be careful : sed The format of domain operation is replacement and modification .

Example : hold hello,world.sed become world,sed.hello( Be careful : 1 Commas 1 Point number )

 Method 1:
# echo "hello,world.sed" | sed 's/\(.*\),\(.*\)\.\(.*\)/\2,\3.\1/'
world,sed.hello
 This method \ Too many symbols , It is recommended to use -r Extended mode , You don't have to add it \ Escape parentheses 

 Method 2:
# echo "hello,world.sed" | sed -r 's/(.*),(.*)\.(.*)/\2,\3.\1/'
world,sed.hello

 Method 3:
# echo "hello,world.sed" | sed -r 's/(.....)(.)(.....)(.)(...)/\3\2\5\4\1/'
world,sed.hello

 Method 4:
# echo "hello,world.sed" | sed -r 's/(.{5})(.)(.{5})(.)(.{3})/\3\2\5\4\1/'
world,sed.hello

Example : With /etc/passwd Before document 5 Behavior example , Do the following

Delete the first character of each line

# head -5 /etc/passwd |cut -c2-
# head -5 /etc/passwd |sed -r 's/(.)(.*)/\2/'
# head -5 /etc/passwd |sed -r 's/.//1'
# head -5 /etc/passwd |sed -r 's/^.//'

Delete the ninth character of each line

# head -5 /etc/passwd |cut -c1-8,10-
# head -5 /etc/passwd |sed -r 's/(.{8})(.)(.*)/\1\3/'
# head -5 /etc/passwd |sed -r 's/.//9'

Delete the last 5 Characters

# head -5 /etc/passwd |rev |cut -c1-4,6- |rev
# head -5 /etc/passwd |sed -r 's/(.*)(.)(....)/\1\3/'

Put the... Of each line 5 Characters and the 8 Characters are interchanged , And delete paragraph 10 Characters

# head -5 /etc/passwd | sed -r 's/(....)(.)(..)(.)(.)(.)(.*)/\1\4\3\2\5\7/'

Practice after class

Write a simple script to initialize the system

1, Modify hostname （ That is, you get IP by 10.1.1.11, Then the host name is changed to server11.cluster.com)

2, Configure available local yum

3, Turn off the firewall and selinux

4, Installation configuration vsftpd, Anonymous users are not allowed to log in , After configuration, start the service and set it to start automatically

5, You can continue to expand freely on this basis