Chapter 8 Intermediate Shell Tools II

1.1 Sorting Your Output

Use the sort utility. You can sort one or more files by putting the file names on the command line.

[[email protected] sample]$ sort a.txt b.txt c.txt d.txt
It can be handly to have your output in sorted order, and handier still not have to add sorting code to every program you write.
Use the sort utility
With no filenames on the command,sort will read from standard input you can pipe the output from a previous command into sort.
You can sort one or more files by putting the file names on the command line
[[email protected] sample]$ ls -ltr
total 16
-rw-rw-r-- 1 maxwell maxwell  21 Jul 28 16:09 a.txt
-rw-rw-r-- 1 maxwell maxwell  77 Jul 28 16:10 b.txt
-rw-rw-r-- 1 maxwell maxwell 127 Jul 28 16:10 c.txt
-rw-rw-r-- 1 maxwell maxwell 129 Jul 28 16:12 d.txt
[[email protected] sample]$ 

With no filenames on the command, sort will read from standard input so you can pipe the output from a previous command into sort.

[[email protected] sample]$ somecommands | sort

• sort -r 

to reverse the order of the sort.

• sort -f

to "fold" lower- and uppercase characters together.

• sort --ignore-case

[[email protected] sample]$ sort -r a.txt
Use the sort utility
[[email protected] sample]$ sort -r a.txt b.txt c.txt d.txt
You can sort one or more files by putting the file names on the command line
With no filenames on the command,sort will read from standard input you can pipe the output from a previous command into sort.
Use the sort utility
It can be handly to have your output in sorted order, and handier still not have to add sorting code to every program you write.
[[email protected] sample]$ sort -f a.txt b.txt c.txt d.txt
It can be handly to have your output in sorted order, and handier still not have to add sorting code to every program you write.
Use the sort utility
With no filenames on the command,sort will read from standard input you can pipe the output from a previous command into sort.
You can sort one or more files by putting the file names on the command line
[[email protected] sample]$ 
[[email protected] sample]$ sort --ignore-case a.txt b.txt c.txt d.txt
It can be handly to have your output in sorted order, and handier still not have to add sorting code to every program you write.
Use the sort utility
With no filenames on the command,sort will read from standard input you can pipe the output from a previous command into sort.
You can sort one or more files by putting the file names on the command line
[[email protected] sample]$ 

1.2 Sorting Numbers

[[email protected] sample]$ sort somedata
2
200
21
250
[[email protected] sample]$ sort -n somedata
2
21
200
250
[[email protected] sample]$

cut -d':' -f7 /etc/passwd isolates the shell from the /etc/passwd file. Then we have to do an initial sort so that uniq will work. uniq -c counts consecutive, duplicate lines, which is why we need the pre-sort. Then sort -rn gives us a reverse, numerical sort, with the most popular shell at the top.

[[email protected] sample]$ cut -d':' -f7 /etc/passwd | sort | uniq -c | sort -rn
     27 /sbin/nologin
      4 /bin/bash
      1 /sbin/shutdown
      1 /sbin/halt
      1 /bin/sync
[[email protected] sample]$ cut -d':' -f7 /etc/passwd | sort -u
/bin/bash
/bin/sync
/sbin/halt
/sbin/nologin
/sbin/shutdown
[[email protected] sample]$ 

1.3 Sorting IP Addresses

To sort the entire address as you would expect(POSIX syntax):

[[email protected] sample]$ sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n ipaddr.list
10.0.0.2
10.0.0.5
10.0.0.20
118.178.233.224
192.168.0.1
192.168.0.2
192.168.0.4
192.168.0.12
[[email protected] sample]$ 

The -t option indicates the character to use as a separator between fields (in our case, a period) so that we can also specify which fields to sort first.

In this case, -k 1,1n means “start sorting at the beginning of field one (1) and (,) stop sorting at the end of field one (1) and do a numerical sort (n).

[[email protected] sample]$ sort -t. -k4n ipaddr.list
192.168.0.1
10.0.0.2
192.168.0.2
192.168.0.4
10.0.0.5
192.168.0.12
10.0.0.20
118.178.233.224
[[email protected] sample]$

1.4 Cutting Out Parts of Your Output

Use the cut command with the -c option to take particular columns: Note that our example 'ps' command only works with certain systems

[[email protected] sample]$ ps -l | cut -c12-15
   P
2004
2005
2005
[[email protected] sample]$ ps -elf | cut -c58-
IME TTY          TIME CMD
n28 ?        00:00:45 /usr/lib/systemd/systemd --system --deserialize 21
n28 ?        00:00:01 [kthreadd]
n28 ?        00:00:00 [rcu_gp]
n28 ?        00:00:00 [rcu_par_gp]
n28 ?        00:00:00 [kworker/0:0H-kblockd]
n28 ?        00:00:00 [mm_percpu_wq]
n28 ?        00:00:07 [ksoftirqd/0]
n28 ?        00:14:11 [rcu_sched]
n28 ?        00:00:00 [migration/0]
n28 ?        00:00:01 [watchdog/0]
n28 ?        00:00:00 [cpuhp/0]
n28 ?        00:00:00 [cpuhp/1]
n28 ?        00:00:01 [watchdog/1]

Using cut to print out fields rather than columns is possible, though more limited than other choices such as awk. The default delimiter between fields is the Tab character, but you can specify a different delimiter with the -d option. Here is an example of a cut command using fields:

[[email protected] sample]$ cut -d'#' -f2 < ipaddr.list
10.0.0.2
118.178.233.224
192.168.0.2
192.168.0.1
192.168.0.4
10.0.0.5
192.168.0.12
10.0.0.20

[[email protected] sample]$ cut -d'[' -f2 delimited_data | cut -d']' -f1
l1
l2
l3
[[email protected] sample]$ cat delimited_data
Line [l1].
Line [l2].
Line [l3].
[[email protected] sample]$ 

1.5 Removing Duplicate Lines

somesequence | sort -u

 If you aren't running sort, just pipe the output into uniq.

$ somesequence > myfile
$ uniq myfile

1.6 Compressing Files

$ tar cf tarball_name.tar directory_of_files
$ gzip tarball_name.tar

[[email protected] sample]$ ls -ltr
total 28
-rw-rw-r-- 1 maxwell maxwell  21 Jul 28 16:09 a.txt
-rw-rw-r-- 1 maxwell maxwell  77 Jul 28 16:10 b.txt
-rw-rw-r-- 1 maxwell maxwell 127 Jul 28 16:10 c.txt
-rw-rw-r-- 1 maxwell maxwell 129 Jul 28 16:12 d.txt
-rw-rw-r-- 1 maxwell maxwell  13 Jul 28 16:22 somedata
-rw-rw-r-- 1 maxwell maxwell  93 Jul 28 16:32 ipaddr.list
-rw-rw-r-- 1 maxwell maxwell  33 Jul 28 16:45 delimited_data
[[email protected] sample]$ tar cf tarball_somedata.tar somedata
[[email protected] sample]$ ls -ltr
total 40
-rw-rw-r-- 1 maxwell maxwell    21 Jul 28 16:09 a.txt
-rw-rw-r-- 1 maxwell maxwell    77 Jul 28 16:10 b.txt
-rw-rw-r-- 1 maxwell maxwell   127 Jul 28 16:10 c.txt
-rw-rw-r-- 1 maxwell maxwell   129 Jul 28 16:12 d.txt
-rw-rw-r-- 1 maxwell maxwell    13 Jul 28 16:22 somedata
-rw-rw-r-- 1 maxwell maxwell    93 Jul 28 16:32 ipaddr.list
-rw-rw-r-- 1 maxwell maxwell    33 Jul 28 16:45 delimited_data
-rw-rw-r-- 1 maxwell maxwell 10240 Jul 29 08:12 tarball_somedata.tar
[[email protected] sample]$ gzip tarball_somedata.tar
[[email protected] sample]$ ls -ltr
total 32
-rw-rw-r-- 1 maxwell maxwell  21 Jul 28 16:09 a.txt
-rw-rw-r-- 1 maxwell maxwell  77 Jul 28 16:10 b.txt
-rw-rw-r-- 1 maxwell maxwell 127 Jul 28 16:10 c.txt
-rw-rw-r-- 1 maxwell maxwell 129 Jul 28 16:12 d.txt
-rw-rw-r-- 1 maxwell maxwell  13 Jul 28 16:22 somedata
-rw-rw-r-- 1 maxwell maxwell  93 Jul 28 16:32 ipaddr.list
-rw-rw-r-- 1 maxwell maxwell  33 Jul 28 16:45 delimited_data
-rw-rw-r-- 1 maxwell maxwell 157 Jul 29 08:12 tarball_somedata.tar.gz
[[email protected] sample]$ 

1.7 Uncompressing Files

 

 1.8 Checking a tar Archive for Unique Directories

Use an awk script to parse off the directory names from the tar archieve's table of contents, then use sort -u to leave you with just the unique directory names:

tar tf some.tar | awk -F/ '{print $1}' | sort -u

1.9 Translating Characters

Use the tr command to translate one character to another.

[[email protected] sample]$ tr ";" ',' <before.txt >after.txt
[[email protected] sample]$ ls -ltr
total 40
-rw-rw-r-- 1 maxwell maxwell  21 Jul 28 16:09 a.txt
-rw-rw-r-- 1 maxwell maxwell  77 Jul 28 16:10 b.txt
-rw-rw-r-- 1 maxwell maxwell 127 Jul 28 16:10 c.txt
-rw-rw-r-- 1 maxwell maxwell 129 Jul 28 16:12 d.txt
-rw-rw-r-- 1 maxwell maxwell  13 Jul 28 16:22 somedata
-rw-rw-r-- 1 maxwell maxwell  93 Jul 28 16:32 ipaddr.list
-rw-rw-r-- 1 maxwell maxwell  33 Jul 28 16:45 delimited_data
-rw-rw-r-- 1 maxwell maxwell  45 Jul 29 08:15 tarball_somedata.tar.gz
-rw-rw-r-- 1 maxwell maxwell 342 Jul 30 20:32 before.txt
-rw-rw-r-- 1 maxwell maxwell 342 Jul 30 20:33 after.txt
[[email protected] sample]$ cat after.txt
In its simplest form, a tr command replaces occurrences of the first (and only) character of, the first argument with the first (and only) character of the second argument.
In the example solution, we redirected input from the file named before and sent the
output into the file named after and we translated all occurrences of a semicolon,
[[email protected] sample]$ cat before.txt
In its simplest form; a tr command replaces occurrences of the first (and only) character of; the first argument with the first (and only) character of the second argument.
In the example solution; we redirected input from the file named before and sent the
output into the file named after and we translated all occurrences of a semicolon;
[[email protected] sample]$ 

The tr command can do more that one translation at a time by putting the several characters to be translated in the first argument and their corresponding resultant characters in the second argument.

 tr ';:.!?' ',' <other.punct >commas.all

1.10 Converting Uppercase to Lowercase

You can translate all uppercase characters(A-Z) to lowercase(a-z) using the tr command and specifying a range of characters.as in:

tr 'A-Z' 'a-z' <be.fore >af.ter

There is also special syntax in tr for specifying this sort of range for upper- and lowercase conversions:

tr '[:upper:]' '[:lower:]' <be.fore >af.ter

Here’s a very simplistic encoding of a text message using a simple substitution cypher that offsets each character by 13 places (i.e., ROT13). An interesting characteristic of ROT13 is that the same process is used to both encipher and decipher the text:

[[email protected] tmp]$ cat /tmp/joke
Q: Why did the chicken cross the road?
A: To get to the other side.
[[email protected] tmp]$ tr 'A-Za-z' 'N-ZA-Mn-za-m' < /tmp/joke
D: Jul qvq gur puvpxra pebff gur ebnq?
N: Gb trg gb gur bgure fvqr.
[[email protected] tmp]$ tr 'A-Za-z' 'N-ZA-Mn-za-m' < /tmp/joke | tr 'A-Za-z' 'N-ZA-Mn-za-m'
Q: Why did the chicken cross the road?
A: To get to the other side.
[[email protected] tmp]$ 

1.11 Converting DOS Files to Linux Format

Use the -d option on tr to delete the character(s) in the supplied list. For example, to delete all DOS carriage returns (\r), use the command:

tr -d '\r' <file.dos >file.txt

 1.13 Counting Lines,Words,or Characters in a File

[[email protected] sample]$ wc before.txt
  3  58 342 before.txt
[[email protected] sample]$ # Line only
[[email protected] sample]$ wc -l before.txt
3 before.txt
[[email protected] sample]$ # Words only
[[email protected] sample]$ wc -w before.txt
58 before.txt
[[email protected] sample]$ # Characters only
[[email protected] sample]$ wc -c before.txt
342 before.txt
[[email protected] sample]$ ls -l before.txt
-rw-rw-r-- 1 maxwell maxwell 

 1.14 Rewrapping Paragraphs

Use the fmt command, optionally with a goal and maximum line length:

$ fmt mangled_text
$ fmt 55 60 mangled_text