Shell_ 01_ data processing

Data processing

Redirect / The Conduit

Redirect

File descriptor （File Descriptor）：Linux A positive integer representing each file object

1） A single process can have at most 9 File descriptors ;

2）Linux Default hold 0、1 and 2 File descriptor （ Users can customize 6~8）;

Temporary redirection

Output redirection ： Save the execution result of the command to a specific file or device

1）Linux The default command in is output to the terminal after execution

2） Command format ： Instructions Redirect Path to file

standard output （standard output）stdout： The code is 1

>	>>
Cover the output Overwrite the contents of the original file	Additional output Continue to add at the end of the existing file content

1） It can also be expressed as 1> or 1>>, By default “1” Omit

2） If the specified file does not exist , The system will create this file

Such as ： Use ls -la Inquire about “/” Post import ls.txt file

Standard error output （standard error output）stderr： The code is 2

2>	2>>
Cover the output Overwrite the contents of the original file	Additional output Continue to add at the end of the existing file content

// Numbers 2 Don't omit （ Used to distinguish standard output ）

Such as ： Inquire about /home All of them .bashrc file , And output to list1 and list2 file

	stdout	stderr
difference	After the command is executed successfully Correct information returned	After the command fails Error message returned

//stdout and stderr You cannot use the output redirection symbol of the other party

// If the data of successful and failed operation are written in the same file , Special syntax is required “&>”

Such as ： Inquire about /home All of them .bashrc File and enter the results into list file

// If two pieces of data are written to one at the same time , Special syntax should be used , If not “&>” It will lead to cross write or data loss

/dev/null：Linux The recycle bin （ Black holes ）

1） equipment /dev/null Can digest any information directed to the device

Input redirection ： Input data on the specified file or device

The standard input （standard input）stdin： The code is 0

<	<< Delimiter
File content replaces keyboard input	Stop inputting when you encounter a delimiter

Such as ： Send using input redirection mail to mwl user

tee command ： Two way redirection

Command format ：tee Options File path

1） Send data streams to terminals and files respectively （ By default, the file is written in the form of overwrite ）

Options	meaning
-a	Write files in append mode
-i	Ignore interrupt signal

Such as ： Output $PATH Address the variable and save it to /root/test/test2.txt file

Permanent redirection

exec command ： Realization Shell Permanent redirection of fragments in the script

Command format ：exec Redirect format

1） stay exec The command specifies the output information after resetting , Save to exec Designated location ;

// stay exec Before the command is specified , Then save according to the default output （ Or follow the previous exec Specify save ）

Such as ： adopt exec Command to achieve permanent output redirection

1） To write test19.sh Script files

2） call test19.sh Script files

Such as ： adopt exec The command implements permanent input redirection

1） To write test20.sh Script files

2） call test20.sh Script files

Custom redirection

exec command ： Custom output / Input redirection

Command format ：exec Numeric descriptor and redirection

1） File descriptors can store redirection attributes of other file descriptors

Such as ： Create descriptor 3 Output redirection for

1） To write test21.sh Script files ;

2） call test21.sh Script files

// Custom output redirection is not specified , Output to by default stdout（ Or the last permanent redirect ）

Such as ： adopt exec Command to record the attributes of other file descriptors

1） To write test22.sh Script files

2） call test22.sh Script files

mktemp command ： Create a document in the current directory

Command format ：mktemp Options Document prefix name .XXXXXX

Options	meaning
-t	stay /tmp Create documents under the directory
-d	Create directory （ Create file by default ）

1） The created document is not affected by umask Limit , Only the owner owns rw jurisdiction ;

2） The system will randomly fill “XXXXXX”, To achieve the uniqueness of creating documents ;

Such as ： adopt mktemp Command create file

The Conduit

** The Conduit ：** be used for “ Filtering treatment ”、“ Special treatment ”、“ Extended processing ”

1） Symbolic form “|”

2） Pipes cannot be used alone , Must be used with instructions （ The essence is auxiliary ）

// Pipelines can be understood as dividing lines . The output in front of the pipeline is the input of the following instructions

Such as ： The query root directory contains “y” The document name of the letter

Such as ： Count the total number of documents in a directory

xargs command （x arguments）： Filters that pass parameters to other commands

Command format ：xargs Options command

1） No command , The default command is echo

Options	meaning
-n	Execute the command with several parameters
-p	Execute each command and ask whether to execute
-e	Set end character
-0	Restore special characters to general characters

Such as ： adopt id Command query /etc/passwd Information of the first three users in the file

1）xargs The newline character and space character in the output information through the pipeline will be replaced by space character

// Many commands do not support pipelines , But through xargs Provides command parameters to realize multiple commands in a single line

format

file

join command ： Specify the same field to connect according to the two files

Command format ：join Options File path 1 File path 2

Options	meaning
-t	Specify the delimiter of the data
-i	Ignore case
-1	Specified file 1 Field of
-2	Specified file 2 Field of

1） The default is the second of two files 1 Compare fields

// Field ： Data divided by separator in each row

Such as ： Will file /etc/passwd And documents /etc/group Integrate

// because GID stay /etc/passwd pass the civil examinations 4 Fields , stay /etc/group The third field in

paste command ： Merge multiple files in a queue

Command format ：paste file 1 file N

1） Whether the contents of the document are the same or not , Will be merged according to the number of rows

split command ： In size / The number of lines divides a file into multiple data

Command format ：split Options File path Prefix

Options	meaning
-b	Split data by file size
-l	Split the file by the number of lines

1） After the file is split , Its name is ： Prefix aa、 Prefix ab、 Prefix ac And so on

// If there is no prefix name , Default to aa、ab、ac Name and so on

2） After segmentation, the original file will be saved

Such as ： take /root/etc/passwd File by 10 Split rows and save to /root/test Under the folder

Format

tr command ： Modify the specified characters in the data

Command format 1：tr Options character // Modify character

Options	meaning
-d	Delete keywords in the data
-s	Delete consecutive repeated characters to , And keep one

Command format 2：tr Original character Target character // Replace character

Such as ： Use tr adjustment echo Output

expand command ： Put the tabs in the data （Tab） Convert to space

Command format ：expand Options Document path

Options	meaning
-t	Define a Tab How many blank characters does the key equal （ The default is 8 Characters ）

1）unexpand The effect of the command is the opposite （ The usage is the same ）

col command ： Adjust input / Output content

Command format ：col Options

Options	meaning
-x	Replace the skip character with multiple space characters （Tab）
-b	Delete all control characters （ Include RLF and HRLF）

uniq command ： Show / Ignore duplicate rows in data （ Default ignore ）

Command format ：uniq Options

Options	meaning
-i	Ignore case
-c	Displays the number of repetitions
-d	Show only recurring rows and columns

1） Generally speaking, it is related to sort Command in combination with

Output

grep command ： Find the data containing keywords in the data and output

Command format ：grep Options key word Document path

1）grep After the keyword is found, the data is intercepted in behavioral units for output ;

2）egrep The order is grep Command upgrade , It supports POSIX Extended regular expression ;

Options	meaning
-a	Find data in the form of binary file as text file
-c	Calculate the number of rows with keywords
-i	Ignore case differences
-n	Output line number
-v	Find data without keywords
-E	Support extended expression

sort command ： Sort the data in the file and output

Command format ：sort Options File path

Options	meaning
-f	Ignore case （ By default, uppercase letters are placed first ）
-b	Ignore the space at the beginning of the line
-n	Sort numbers in numerical form （ By default, numbers are sorted as characters ）
-M	Sort by month
-r	Reverse sorting
-t	Specify the delimiter of the data （ The separator must be ‘’ Cover up ）
-k	Specify which interval to sort

1） The default is ASCII Code value comparison for sorting , And the sorted characters are related to the coding of language family

2）-t and -k It makes sense to use them together （ similar cut Of -d and -f）

echo command ： Output the extracted value of the specified string or variable to the terminal

Command format ：echo String or ${ Variable }

1） If the string contains spaces , It needs to be enclosed in double quotation marks ;

2） If the string contains single quotation marks or double quotation marks , Use different quotation marks

// When using single quotation marks , Use double quotes （ conversely , Use single quotation marks ）

Such as ： Output Hello,World and SHELL Variable content to terminal

Such as ： Output the contents with quotation marks to the terminal

printf command ： Format and output the result to stdout

Command format ：printf ‘ Output format ’ Print the content

Format	meaning
%s	String format output
%i	Output in integer format
%f	Floating point format output
%%	Output %

1）printf Not a pipeline command

2） There are some special escape characters （ Keyboard can't input ） Use envoy format instead

Special format	meaning
\a	Output warning sound
\b	Backspace
\f	Clear the screen
\n	Output a new line
\r	The next line （ Enter key ）
\t	level Tab Key
\v	vertical Tab Key

Such as ： Format conversion output /root/test/test1.txt The contents of the document

command	difference
printf	Ksh Built in commands , regular expression Don't wrap
print	Bash Built in commands Word wrap