Data Stream Redirection

Introduction

You may not be familiar with the concept of redirection, but you should have seen > or >> operators in previous labs and know that they are the standard output to a file or are added to a file. This is redirection. The actual output to the standard output is redirected to a file. Since the standard output (/dev/stdout) is a file, there is no problem when we redirect the output to another file.

Skills Graph

Stream Redirection Tools

Let's take a brief look at the two redirection operations we used earlier:

echo 'hello labex' > redirect
echo 'www.labex.io' >> redirect
cat redirect

You can also try echo 'hello labex' < redirect. The difference between < and >, >> is the direction of redirect. >, >> means the direction is left to right and < means right to left.

Simple Use of Redirection

Before we learn more about Linux redirection, we need to know some basic concepts. First, as we have mentioned earlier, Linux provides three special devices for terminal input and output, which are stdin (standard input, corresponding to your input at the terminal), stdout (standard output, corresponding to the output of the terminal), and stderr (standard error output, corresponding to the output of the terminal for error messages).

We can use these file descriptors like this:

By default, the system uses the standard input as the command input and the standard output as the command output.

We redirect the output of cat to a file:

mkdir Documents
touch Documents/test.c
cat > Documents/test.c << EOF
#include <stdio.h>

int main()
{
    printf("hello world\n");
    return 0;
}

EOF

We put a file as a command input and the standard output as a command output:

cat Documents/test.c

stderr Redirection

We have seen how to redirect the standard output to a file, which is convenient. Another practical operation is to redirect the standard error. The standard output and the standard error are directed to the screen display. So, the output of the command we often see usually includes the standard output and standard error results, such as the following:

Use the cat command to read two files at the same time, one of which exists and the other does not exist:

cat Documents/test.c hello.c

The command not only outputs the contents of the first file but also shows an error message at the end of the listing.
Then, we redirect the output to a file. However, according to our previous experience, we may not see any output:

cat Documents/test.c hello.c somefile

labex:project/ $ cat Documents/test.c hello.c
#include <stdio.h>

int main()
{
    printf("hello world\n");
    return 0;
}
cat: hello.c: No such file or directory
labex:project/ $ cat Documents/test.c hello.c somefile
#include <stdio.h>

int main()
{
    printf("hello world\n");
    return 0;
}
cat: hello.c: No such file or directory
cat: somefile: No such file or directory

Unfortunately, although we have redirected the output to the file, there is still an error message in the terminal listing. Sometimes, we want to hide some errors or warnings. What should we do in such cases? We can use file descriptors to achieve that.

Redirect the standard error to the error.log, and then redirect the standard output to the terminant:

cat cat Documents/test.c hello.c somefile > /dev/tty 2> error.log
echo
cat error.log

labex:project/ $ cat Documents/test.c hello.c somefile > /dev/tty 2>error.log
#include <stdio.h>

int main()
{
    printf("hello world\n");
    return 0;
}

tee

Sometimes, in addition to redirecting the output to a file, we must print the information at the terminal. Then you can use the tee command:

echo 'hello labex' | tee hello

labex:project/ $ echo 'hello labex' | tee hello
hello labex
labex:project/ $ cat hello
hello labex

Permanent Redirection

Redirect operations seen above were temporary, meaning the redirection was only valid for the current command. We can use exec to implement a permanent redirect:

## Open a sub-shell
zsh

## Use exec to redirect the standard output to a file
exec 1> somefile

## The output of the command you execute later will be redirected to the file until you exit the current shell or cancel the exec's redirect (later, we will tell you how to operate it)
ls
exit
cat somefile

labex:project/ $ zsh
labex:project/ $ exec 1> somefile
labex:project/ $ ls
labex:project/ $ exit
labex:project/ $ cat somefile
Documents
error.log
hello
redirect
someefile
somefile

Creating a File Descriptor

There are 9 file descriptors in the shell. We previously used the 0, 1, and 2 file descriptors. We can also use the file descriptors 3 to 8, but they are not open by default. You can use the following command to view the file descriptors that are open in the current shell process:

cd /dev/fd/
ls -Al

exec can also create a new file descriptor:

exec 3> somefile1
cd /dev/fd/
ls -Al
cd -
## Note that there should be no space between and &. If there is a space, the result will be wrong.
echo "this is test" >&3
cat somefile1

labex:project/ $ exec 3> somefile1
labex:project/ $ cd /dev/fd/
ls -Al
cd -
total 0
lrwx------ 1 labex labex 64 Feb 20 23:16 0 - > /dev/pts/0
lrwx------ 1 labex labex 64 Feb 20 23:16 1 - > /dev/pts/0
...
l-wx------ 1 labex labex 64 Feb 20 23:16 3 - > /home/labex/project/somefile1
labex:project/ $ echo "this is test" >&3
cat somefile1
this is test

If the file descriptor is no longer needed, you can turn off the "3" file descriptor using the following actions:

exec 3>&-
cd /dev/fd
ls -Al
cd -

Dropping the Output of a Command

There is a device file called "black hole" in Linux so that that data may be swallowed.

On UNIX-like systems, /dev/null is a device file typically used to drop unwanted output streams or as empty files for input streams, usually done by redirecting. Reading it will get an EOF.

We can use /dev/null to mask command output:

cat Documents/test.c nefile 1 > /dev/null

Thus the output cannot be obtained. But the error messages will still be printed.

labex:project/ $ cat Documents/test.c nefile 1> /dev/null
cat: nefile: No such file or directory

Using xargs to Split the Parameter List

xargs is a common command for UNIX and UNIX-like operating systems. Its role is to convert the parameter list into small pieces of segmentation passed to other commands to avoid the problem that the parameter list is too long.

This command is helpful, especially when dealing with commands that produce many outputs, such as find, locate and grep. Please refer to man for detailed information.

cut -d: -f1 < /etc/passwd | sort | xargs echo

labex:project/ $ cut -d: -f1 < /etc/passwd | sort | xargs echo
_apt avahi backup bin colord daemon games gnats irc labex list lp mail man messagebus mongodb mysql news nobody proxy pulse redis root rtkit saned sshd sync sys systemd-network systemd-resolve systemd-timesync tcpdump usbmux uucp www-data
labex:project/ $

The above command is used to sort the /etc/passwd file with the first field, which is spilted by : and use echo to generate a list.

Summary

You may be familiar with the concept of redirection, and you know that the standard output to a file or are added to a file. This is redirection. The actual output to the standard output is redirected to a file. Since the standard output (/dev/stdout) is a file, there is no problem when we redirect the output to another file.