Multithread decompression of tar

Hello everyone , I am nine in the morning and twelve in the evening , At present, I am doing operation and maintenance related work . Blogging is to accumulate , I hope you can make progress together ！
My home page ： Nine in the morning and twelve in the evening

About tar

We usually operate on the server , Often used to tar command , This is a Unix And the class Unix Compression packaging tools on the system , You can combine multiple files into one file , The file suffix after packing is also “tar”. It can be used by any user .
meanwhile , It has multiple versions with different compression rates , Such as tar.xz and tar.gz, The former has a higher compression ratio , But there may be compatibility issues .
If you've used tar, Must have this feeling ： When you are decompressing or compressing a very large file , It's going to be slow ！ This is because tar Decompression is performed by a single thread , If you are decompressing , Use top command , I often see a certain cpu Soar to 100%.

Today we will introduce ,tar Multithreading operation of .

Multithreaded installation

yum -y install pigz

pigz Commands come from English phrases ”parallel implementation of gzip“ Abbreviation , Its function is to extract files for multithreading . Unlike other decompression commands pigz The command supports multi-threaded parallel processing , Year on year gzip It can be fast 60% above , Of course CPU The consumption will also be higher .

# Compressed files 
tar --use-compress-program=pigz -cvpf  app.tar.gz  app

# Unzip the file 
tar --use-compress-program=pigz -xvpf     app.tar.gz

--use-compress-program=pigz That is to say, it represents pigz Tools , What is the effect , Let's look forward to

Results test

We can write a script , Calculation and use pigz And unused time .

#!/bin/bash

echo "starttime: `date +"%Y-%m-%d %H:%M:%S"`" > tar.txt
sleep 5 # Fill in our compression command here 
echo "endtime: `date +"%Y-%m-%d %H:%M:%S"`" >> tar.txt

start=`awk '{print $2,$3}' tar.txt | sed -n '1p'` 
end=`awk '{print $2,$3}' tar.txt | sed -n '2p' `
echo $start
echo $end

starttime=$(date -d "$start" +%s) # Convert to system time 
endtime=$(date -d "$end" +%s)
runtime=$(($endtime-$starttime))  # Calculate program run time 
echo $runtime

Single threaded compression

take sleep 5 Replace with tar -cvpf app.tar.gz app/ After execution , The output is 26s

Multithreaded compression

take sleep 5 Replace with tar --use-compress-program=pigz -cvpf app.tar.gz app After execution , The output is 15s

According to the above test results , You can see that the time is shortened 11s. Decompress , My friends can test it by themselves .

It's not easy to code words , I hope you can support one wave for three times . If there is any problem, you can point out , Thank you. ！