Simple Text Processing

Introduction

In this section, we will introduce tr, col, join, and paste and will still review the pipeline to familiarize ourselves with these commands.

Skills Graph

tr

The command tr can be used to delete word(s) in a piece of text or to convert it.

Format

tr [option]...SET1 [SET2]

The Use of tr

Examples

Delete all of the 'o', 'l' and 'h' characters in "hello labex":

echo 'hello labex' | tr -d 'olh'

Delete duplicated char 'l' in 'hello':

echo 'hello' | tr -s 'l'

Show the input text in uppercase or lowercase. For example, the following commands show in uppercase:

cat /etc/passwd | tr '[:lower:]' '[:upper:]'
## or cat /etc/passwd | tr '[a-z]' '[A-Z]'

Output

labex:project/ $ echo 'hello labex' | tr -d 'olh'
e abex
labex:project/ $ echo 'hello' | tr -s 'l'
helo
labex:project/ $ cat /etc/passwd | tr '[:lower:]' '[:upper:]'
...

For more uses of tr, use tr --help or man tr to fetch further information.

col

col can replace the Tab character with several space characters or reverse the operation.

Format

col [option]

Use Cases for col

Example

To view invisible characters in /etc/protocols: We can see a lot of ^I in the file. This is Tab transformed into visible characters.

cat -A /etc/protocols

Use col -x to convert tabs in /etc/protocols to spaces, then use cat to see the file content, and we will find that ^I is gone.

cat /etc/protocols | col -x | cat -A

Output

labex:project/ $ cat /etc/protocols | col -x | cat -A
## Internet (IP) protocols$
#$
## Updated from http://www.iana.org/assignments/protocol-numbers and other$
## sources.$
## New protocols will be added on request if they have been officially$
## assigned by IANA and are not historical.$
...

join

The join utility performs an "equality join'' on the specified files and writes the result to the standard output. The "join field'' is the field in each file by which the files are compared.

Format

join [option]... file1 file2

Parameters (Options) of join

Example

## Create two files
echo '1 hello' > file1
echo '1 labex' > file2
join file1 file2

Output

labex:project/ $ echo '1 hello' > file1
echo '1 labex' > file2
join file1 file2
1 hello labex

paste

paste is similar to join. It simply merges multiple files without comparison.

Format

paste [option] file...

Parameters (Options) of paste

Example

echo hello > file1
echo labex > file2
echo labex.io > file3

paste -d ':' file1 file2 file3
echo
paste -s file1 file2 file3

Output

hello:labex:labex.io

hello
labex
labex.io

Time for Fun

Please use the following command to install Space Invaders:

sudo apt-get install ninvaders -y
ninvaders

Summary

In the "File Packing and Compression" section, we mentioned that some special characters in text files on Windows/DOS and Linux/UNIX are not the same, such as the line break: It is CR+LF (\r\n) on Windows and LF (\n) on Linux/UNIX. We can use the cat -A text to see the invisible special characters contained in the text.