Shell Programming Notes

Shell Programming Notes

1. Basic overview

Shell Script , Is the use Shell The function of command interpretation , Parsing a plain text file , And then perform these functions , It can also be said that Shell A script is a collection of commands .

Shell It can be used directly in win/Unix/Linux above , And a large number of functions inside the system can be called to explain and execute the program , If you master Shell Script , It can make it easier for us to operate computers , It also saves a lot of time .

Shell Not exactly which program , Is a general term for a class of programs , As long as these programs can call the interface of the operating system according to the user's requirements , You can call it Shell Program .

linux Up to now , There are many shell Program , One of the software is called bash.

1.1 Basic commands

1.1.2 Find out who logged in ：who command

# Get the information of all users currently logged in to the system 
# When a user logs in to coworker Xiong ,UNIX The system will assign a unique identification number to the user's terminal or network equipment , The number is tty Number 
who

1.1.3 Echo characters ：echo command

echo The command will be printed on the terminal （ Or echo ） Everything you enter in the line , And echo Will compress extra white space characters between words , Extra white space characters are usually ignored .

echo this is test

echo ?		# Print out a one letter file 
echo a? 	# Print out to a The first two letters 
echo ??		# Print out a two letter file 
echo ??*  	# Print out all documents with more than two letters 
echo *		# Replace... With file name , List all files 

1.2 Working with files

UNIX The system recognizes only three basic types of files ： Ordinary documents 、 Catalog files and special files .

1.2.3 Count the number of words in the file ：wc command

wc Command can get the number of lines in the file 、 Number of words and characters . You need to take the file name to be counted as the parameter of this command .

//  command 
wc [ Parameters ] [ file name ]
    
//  Command options （ Parameters ）
l  Count the number of lines the file contains 
c  Count the number of characters contained in the file 
w  Count the number of words contained in the file 

1.3.1 Home directory and pathname

By convention ,【…】 Point to the directory above the current directory , Also known as the parent directory .

Another common way of writing is the single dot 【.】, He always refers to the current directory .

1.4.1 asterisk

Shell There's a powerful feature ： File name replacement .Shell Will automatically change the mode 【*】 Replace with all file names that can be matched in the current directory .

1.4.2 Match a single character

asterisk （*） Can match zero or more characters , in other words ,x Stars can match files x, Can also match x1、x2、xabc etc. . question mark （？） Only a single character can be matched .

Dashes （-） Specify the logical range of a character . for example ,[0-9] Can match characters 0~9. When specifying the character range , The only restriction is that the first character in the alphabet must precede the last character .

1.5.1 Spaces in file names

There are two solutions ： Use a backslash to escape all spaces , Or put the entire file name in quotation marks , Give Way Shell Instruction this is a word that contains spaces , Not multiple words .

example ：

cat "my test document"

cat "my\test\document"

1.6 The standard input / Output and I/O Redirect

1.6.1 Standard input and standard output

UNIX The command can use a file or the output of the previous command as its input , You can also send its output to another command or other program . This concept is extremely important .

Ctrl+d, Indicates that data input is completed .

1.6.2 Output redirection

Transfer the command output sent to standard output to a file , This is called output redirection .

If you will 【>file】 After the command that can write output to standard output , Then the output of the command will be written to the file 【file】 in ：

# send who The output of is written to a file users in 
who > users

Be careful ： This will not have any output , Because the output has been from the default standard output device （ terminal ) Redirected to the specified file .

If the output of the command is redirected to a file , The contents of this file already exist , Then the existing content will be rewritten .

When using characters 【>>】 Indicates another type of output redirection . This set of characters causes the standard output of the command to be appended to the existing contents of the specified file , The previous contents of the file are not lost , The new output is simply added to the tail .

example ：

# take file1 The content of is added to file2 after 
cat file1 >> file2

1.6.3 Input redirection

Just as the output of a command can be redirected to a file , Files can also be redirected into the input of the command . Less than no. 【<】 Is used as the input redirection symbol . Only those instructions that receive input from standard input can use this method to redirect files to their input .

1.7 The Conduit

1.7.1 The Conduit

UNIX Be able to put two commands “ Connect ” together , This connection is called a pipe , It can take the output of one command directly as the input of another command .

Pipes use characters 【|】 Express , Placed between two commands . for example , stay 【who】 and 【wc -l】 Create a pipe between , You can enter ：

who | wc -l

Pipes can be created between any two programs , The premise is that the first program will write the output to standard output , The second program reads the input from the standard input .

1.7.1 filter

Input can be received from standard input , Process the input data , Then write the results to the standard output . In general, it is used to modify the output of other programs in the pipeline .

cat、sort Can be used as a filter .

ls、who、date、cd、pwd、echo、rm、mv、 And cp Input is not read from standard input , Not a filter .

1.9.1 Enter multiple commands on one line

If you enter multiple commands on one line , Just use a semicolon as a separator between commands .

1.9.3 ps command

ps Command can give information about the processes running in the system .

If cooperate 【-f】 Options ,ps More process information will be printed , Include parent process ID（PPID）、 Process start time （STIME） And other command parameters .

2. What is? Shell

2.1 Kernel and Utilities

UNIX The system is logically divided into two different parts ： Kernel and Utilities （Utility）, Generally speaking , All access must be via Shell,Shell Is a utility program , It is loaded into memory and executed as part of the login process .

2.3 stay Shell Enter command in

It is important to realize that Shell It's a program , There are no privileges in the system .

2.4 Shell Responsibility for

（1） Program execution

（2） Variable and file description replacement

（3）I/O Redirect

（4） The Conduit

（5） environmental control

（6） Interpretive programming language

3. Standing tools

3.1 Regular expressions

Regular expressions provide a convenient way 、 Specify the pattern to be matched in a consistent way .

Shell Think 【？】 Can match any single character , And in regular expressions （regular expression, Usually abbreviated to regex） Use the point number （.） To achieve the same effect .

3.1.1 Match any character ： Order number （.）

 Regular expressions 
r.	# Can match r And any single character 

 Regular expressions 
.x.	# Can match any two characters surrounded by x, The two characters do not have to be the same 

 Regular expressions ： Search function 
/ ... /	

notes ： Editor ed Is an old-fashioned line editor , Always exist and Linux in

ed Command regular expression test ：

ed temp.txt	# Use ed The editor opens the file , Then the terminal will print out the total number of characters in the file 

1,$p		# Print out all lines ,【p】 For the print command ,【$】 Is the range qualifier ,
    		# The most basic range qualifier is 1,$, This means that from the first line to the last line of the file 
    
/ ... /		# Means to find a space surrounded by 3 Characters 
/			# Means to repeat the last query （ Continue to search ）  

1,$s/p.o/XXX/g	# Will all p.o Modified into XXX,【s】 To replace the command ,【g】 Means to perform a global replacement 

3.1.2 Match the beginning of the line ： De character （^）

De character ^ As the first character of a regular expression , To match the beginning of the line

/the/	# Expression lookup the
/^the/	# Indicates search to the Beginning line 

1,$s/^/>>/		# Insert... At the beginning of each line >>
1,$s/^/    /	# Insert... At the beginning of each line 4 A space 

3.1.3 Match the end of the line ： Dollar symbol （$）

The dot can match any character , So regular expressions 【$】 Matches any character at the end of the line （ Including point numbers ）.

How to match point numbers ？ generally speaking , If you want to match any character that has a special meaning for a regular expression , You can precede this character with a backslash （\） Come and go except for its special meaning .

\.$		# Match lines ending with a dot 
    
^\.		# Match lines that begin with a dot 
     	# Use backslashes as normal characters , You can use two consecutive backslashes 【\\】
    
/\.$/		# Search for lines ending with a dot  
    
1,$s/..$//	# Delete the last two characters of each line 
    
^$		# Match blank line 
^ $		# Match a line consisting of a single space 
    

3.1.4 Match character group ：【…】

[tT]he	# Match lowercase or uppercase t, Then the characters the
/[tT]he/	# lookup the or The

1,$s/aeiouAEIOU//g	# All the vowels 

# You can use a character range within square brackets to indicate , Specific writers use connectors （-） To separate the start and end characters 
[123456789]	 or 	[0-9]

[A-Z]	# Match capital letters 
[A-Za-z]	# Match upper or lower case letters 
/^[A-Z]/	# Find the line starting with a capital letter 

stay ed Editor , image *、.、[…]、$ and ^ This kind of regular expression sequence only makes sense in the search string , There is no special meaning in the replacement string .

If the caret （^） Is the first character after the left bracket , Then the meaning of the matched content is the opposite （ by comparison ,Shell Use ！ To achieve the same effect ）

[^A-Z]	# Match any character except uppercase letters 
[^A-Za-z]	# Match any non alphabetic character 

3.1.5 Matches zero or more characters ： asterisk （*）

+ Can be used to match an expression that appears one or more times before it , Therefore, the following two expressions are not different in function

XX*	# Can match to X And zero or more consecutive X
XXX*	# Can match at least two consecutive X

X+

1,$s/ */ /	# Change multiple spaces to a single space 

.*	# It is often used to specify zero or more arbitrary characters , remember , Regular expressions match the longest string that matches the pattern , therefore , The whole line of text is always matched 

 Regular expressions 
1,$s/e.*e/+++/	# Place the first in a line e To the last e All characters between （ Including the beginning and the end e , ） Replace with +++

[A-Za-z][A-Za-z]*	# Matches alphabetic characters followed by zero or more alphabetic characters 

[]a-z]	# Can match right bracket or lowercase letter 

3.1.6 Match a fixed number of subpatterns ：\{…\}

General to specify exactly how many characters to match ：

\{
    min,max\}	#min Specifies the minimum number of occurrences of the regular expression to be matched ,max Specifies the maximum number of occurrences .
			# And pay attention , Curly braces must be escaped with a backslash 

X\{
    1,10\}	# matching 1 To 10 A continuous X

Several special cases ：

\{
    10\}	# There is only one number in curly brackets , Indicates the number of times this regular expression must match 

.\{
    10\}	# Able to match 10 Any character 

1,$s/^.\{
    10\}//	# Delete the first... Of each line 10 Characters 
1,$s/.\{
    5\}$//	# After deleting each line 5 Characters , Be careful , No 5 A character , Unable to meet matching criteria , No changes 

# If a single number in curly braces is followed by a comma , Indicates the number of times that the previous regular expression should match at least 
+\{
    5,\}	# Can at least match 5 Consecutive plus signs ,

3.1.7 Save the matched characters ：\(…\)

To reference a character that has been matched by a regular expression , You can put this part of the regular expression in parentheses that are escaped by backslashes . These captured characters are stored in variables called registers predefined by the regular expression parser , Its number is from 1 To 9.

# It can match the first character in the text line and save it in the register 1 in 
^\(.\)	

# To retrieve a character stored in a register , Use  \n （ among n It's from 1 To 9 The number of ）
^\(.\)\1 # First match the first character in the text line and save it in the register 1 in , Then the match is saved in the register 1 The content in （\1 designated ）
		 # The final effect of this regular expression is to be able to match the first two identical characters in a line .

^\(.\).*\1$	# Can match the first character of the line （^.） And end of line characters （\1$） All the same lines . All characters between the first and last characters have .* matching 

Continuous \(…\) The matching contents can be allocated to subsequent multiple registers . example ：

# The first... In the text line 3 Characters will be stored in the register 1 in , Next 3 Characters will be stored in the register 2 in 
^\(...\)\(...\)	
# Can match to a length of 12 Character string , Among them 1~3 Characters and the 10~12 It's like two characters , The first 4~6 Characters and the 7~9 It's like two characters 
^\(...\)\(...\)\2\1

stay ed When using the replace command in , Registers can also be referenced as part of the replacement string .

1,$s/\(.*\)	\(.*\)/\2 \1/	# Swap two fields 

3.2 cut

This command extracts from a data file or command output （ Cut out ） Various fields are very convenient .

cut -cchars file

among ,“chars” Specifies what you want from file Which characters are extracted in each line of the （ According to location ） contain ：
（1） Single digit ; Such as -c5, You can extract the... From each line of input 5 Characters

（2） Comma separated list of numbers ; Such as ,-c1,13,50, You can extract the second 1 individual 、 The first 13 individual 、 The first 50 Characters

（3） A range of numbers separated by connectors ; Such as -c20-50, You can extract the second 20~50 Characters

cut -c5- data	# Extract all characters from the custom position to the end of the line , And write the results to the standard output 

If not specified file,cut Will read the input from the standard input , This means that you can put cut Use as a filter .

You can use cut Extract characters at different positions from text lines , as follows ：

who | cut -c1-8,18-	# To extract the first 1~8 Characters （ user name ）, Then extract the second 18 Characters to the end of the line ( The login time )

#  command 
cut [ Parameters ] [ file name ]
    
#  Command options （ Parameters ）, Solve the problem of non fixed format data files 
-d  Specifies the field separator 
-f  Specify the fields to be extracted 

cut -d: -f1,6 /etc/passwd	# Indicates that the extracted data is in “：” Separate the first field of the row from the 6 A field 

If you are not using -d Option with cut Command to extract fields from a file , The tab character is used as the default field separator , Not spaces .

If the fields are tab delimited , have access to cut Of -f Options ：

cut -f1 phonebook	# Extract the first field in the file 

expand ： tabs

The tab character is equivalent to... On the keyboard Tab key （Q Left side ,Caps lock On top of ）
Use... In format control output “\t” Express
It is equivalent to entering the train at printf of use “\n” Express

Usually used to align output

How to know if a field has spaces or is tab delimited ？

（1） Trial and error method

（2） command ：

sed -n 1 file

If the fields are tab delimited , Show in tab position ：\t

3.3 paste

paste The effect of the command and cut contrary ; It is not a split line , Instead, merge rows . The general format of the order is ：

paste files

from files The corresponding line in each specified file is “ Paste ” Or merge together , Form a line , Then write to the standard file （ The two are separated by tabs ）. Connector “-” Can be used in files in , Specify the input source as standard input .

#  command 
paste [ Parameters ] [ File name before column ] [ File name after column ]
    
#  Command options （ Parameters ）
-d  Specify a single separator , This separator is used to separate all pasted fields 
-s  Appoint paste Only lines can be pasted from the same file , Regardless of other documents , If a single file is specified , The effect is to merge all the lines in the file into one 	 rise , They are separated by tabs （ Or by -d The separator specified by the ）

 example ：
paste -d'+' names address numbers	# It is always safest to put the delimiter in single quotation marks 

paste -s names	# Paste names All lines in the file 

ls | paste -d' ' -s -	# Paste ls Command input , Use spaces as separators