Linux Stream Editing

Introduction

Welcome to the Linux Stream Editing lab. In this lab, you will learn how to use sed (Stream Editor), a powerful command-line utility for parsing and transforming text. sed is used to perform basic text transformations on an input stream (a file or input from a pipeline).

By the end of this lab, you will be able to:

Perform basic text substitutions with sed
Edit files in-place using sed
Apply global substitutions across entire files
Use advanced text processing techniques

The skills you learn will help you efficiently manipulate files and text streams, which is essential for various system administration and text processing tasks in Linux.

Basic sed Usage and Text Substitution

In this step, we'll learn the basics of sed and how to perform simple text substitutions. The sed command is a powerful text processing tool that allows you to search, find and replace, insert, and delete text in files without opening them in an editor.

First, let's verify that sed is installed on our system by running:

which sed

You should see output similar to:

/usr/bin/sed

This confirms that sed is available. If for some reason it's not installed, you can install it with:

sudo apt-get update
sudo apt-get install -y sed

Now, let's create a text file to work with. We'll create a file named sample.txt in the current directory:

echo "Linux is a powerful operating system." > ~/project/sample.txt
echo "Many users prefer Linux for servers." >> ~/project/sample.txt
echo "Linux has strong security features." >> ~/project/sample.txt

Let's examine the content of our new file:

cat ~/project/sample.txt

You should see:

Linux is a powerful operating system.
Many users prefer Linux for servers.
Linux has strong security features.

Now, let's use sed to perform a basic substitution. The following command will replace the first occurrence of "Linux" with "Ubuntu" on each line:

sed 's/Linux/Ubuntu/' ~/project/sample.txt

You should see:

Ubuntu is a powerful operating system.
Many users prefer Ubuntu for servers.
Ubuntu has strong security features.

Notice that sed printed the modified text to the standard output (your terminal), but the original file remains unchanged. You can verify this by running:

cat ~/project/sample.txt

The output should still show "Linux" and not "Ubuntu":

Linux is a powerful operating system.
Many users prefer Linux for servers.
Linux has strong security features.

The basic syntax for sed substitution is:

sed 's/pattern/replacement/' filename

Where:

s is the substitution command
pattern is the text you want to replace
replacement is the new text
filename is the input file

Let's try another example. This time, we'll replace "powerful" with "versatile":

sed 's/powerful/versatile/' ~/project/sample.txt

This should output:

Linux is a versatile operating system.
Many users prefer Linux for servers.
Linux has strong security features.

In-place Editing and Global Substitution

In the previous step, we learned how to use sed for basic text substitution, but the changes were only displayed on the screen, not saved to the file. In this step, we'll learn how to:

Edit files in-place with the -i option
Perform global substitutions using the g flag

In-place Editing

To make permanent changes to a file using sed, we use the -i option. This option modifies the file directly instead of just printing the output to the terminal.

Let's modify our sample.txt file by replacing "Linux" with "Ubuntu":

sed -i 's/Linux/Ubuntu/' ~/project/sample.txt

Unlike the previous step, this command doesn't produce any visible output. Let's check the content of the file to see if it changed:

cat ~/project/sample.txt

You should now see:

Ubuntu is a powerful operating system.
Many users prefer Ubuntu for servers.
Ubuntu has strong security features.

The changes have been saved to the file! This is the power of in-place editing with sed.

Global Substitution

By default, sed only replaces the first occurrence of the pattern on each line. To replace all occurrences, we use the g (global) flag at the end of the substitution command.

Let's create a new file with repeated text:

echo "The red car stopped at the red light near the red building." > ~/project/colors.txt

Now, let's use sed without the global flag to replace "red" with "blue":

sed 's/red/blue/' ~/project/colors.txt

You should see:

The blue car stopped at the red light near the red building.

Notice that only the first occurrence of "red" was replaced with "blue". Now, let's use the global flag:

sed 's/red/blue/g' ~/project/colors.txt

The output should be:

The blue car stopped at the blue light near the blue building.

All occurrences of "red" have been replaced with "blue"!

Let's apply this to our sample.txt file. First, let's add more occurrences of "Ubuntu":

echo "Ubuntu is great. I use Ubuntu daily for my work with Ubuntu tools." >> ~/project/sample.txt

Now, let's replace all occurrences of "Ubuntu" with "Linux" using the global flag and in-place editing:

sed -i 's/Ubuntu/Linux/g' ~/project/sample.txt

Let's verify the changes:

cat ~/project/sample.txt

You should see:

Linux is a powerful operating system.
Many users prefer Linux for servers.
Linux has strong security features.
Linux is great. I use Linux daily for my work with Linux tools.

All occurrences of "Ubuntu" have been replaced with "Linux" throughout the file.

Creating a Backup Before In-place Editing

When using in-place editing, it's often a good practice to create a backup of the original file. You can do this by providing an extension to the -i option:

sed -i.bak 's/Linux/Ubuntu/g' ~/project/sample.txt

This command will:

Create a backup of sample.txt as sample.txt.bak
Replace all occurrences of "Linux" with "Ubuntu" in sample.txt

Let's verify both files:

cat ~/project/sample.txt
cat ~/project/sample.txt.bak

The first command should show all "Linux" replaced with "Ubuntu", while the backup file should still contain "Linux".

Advanced sed Commands and Pattern Matching

Now that we've mastered basic substitution and in-place editing with sed, let's explore some more advanced features:

Using different delimiters
Using address ranges to target specific lines
Combining multiple commands

Using Different Delimiters

While we've been using the forward slash / as the delimiter in our substitution commands, sed allows us to use any character as a delimiter. This is especially useful when the pattern or replacement text contains slashes.

Let's create a file with file paths:

echo "/usr/local/bin is in the PATH" > ~/project/paths.txt
echo "My config is in /etc/myapp/config.json" >> ~/project/paths.txt

If we want to replace /usr/local/bin with /opt/bin, using slashes would be confusing:

sed 's/\/usr\/local\/bin/\/opt\/bin/' ~/project/paths.txt

Instead, we can use a different delimiter, such as #:

sed 's#/usr/local/bin#/opt/bin#' ~/project/paths.txt

This is much more readable! The output should be:

/opt/bin is in the PATH
My config is in /etc/myapp/config.json

Other common delimiters include |, :, and _.

Addressing - Targeting Specific Lines

sed allows us to specify which lines to apply the substitution to. This is done by prefixing the command with an address.

Let's create a new file with numbered lines:

echo "Line 1: This is the first line." > ~/project/numbered.txt
echo "Line 2: This is the second line." >> ~/project/numbered.txt
echo "Line 3: This is the third line." >> ~/project/numbered.txt
echo "Line 4: This is the fourth line." >> ~/project/numbered.txt
echo "Line 5: This is the fifth line." >> ~/project/numbered.txt

To replace "line" with "row" only on line 3:

sed '3 s/line/row/' ~/project/numbered.txt

The output should be:

Line 1: This is the first line.
Line 2: This is the second line.
Line 3: This is the third row.
Line 4: This is the fourth line.
Line 5: This is the fifth line.

We can also specify a range of lines. To replace "line" with "row" on lines 2 through 4:

sed '2,4 s/line/row/' ~/project/numbered.txt

The output should be:

Line 1: This is the first line.
Line 2: This is the second row.
Line 3: This is the third row.
Line 4: This is the fourth row.
Line 5: This is the fifth line.

Another useful feature is the ability to match lines based on a pattern. For example, to replace "line" with "row" only on lines that contain "third" or "fourth":

sed '/\(third\|fourth\)/ s/line/row/' ~/project/numbered.txt

The output should be:

Line 1: This is the first line.
Line 2: This is the second line.
Line 3: This is the third row.
Line 4: This is the fourth row.
Line 5: This is the fifth line.

Combining Multiple Commands

We can combine multiple sed commands using the -e option or by separating commands with semicolons.

Let's replace "first" with "1st", "second" with "2nd", and "third" with "3rd" in a single command:

sed -e 's/first/1st/' -e 's/second/2nd/' -e 's/third/3rd/' ~/project/numbered.txt

Alternatively, we can use semicolons:

sed 's/first/1st/; s/second/2nd/; s/third/3rd/' ~/project/numbered.txt

Both commands should produce the same output:

Line 1: This is the 1st line.
Line 2: This is the 2nd line.
Line 3: This is the 3rd line.
Line 4: This is the fourth line.
Line 5: This is the fifth line.

Let's now make these changes permanent:

sed -i 's/first/1st/; s/second/2nd/; s/third/3rd/' ~/project/numbered.txt

And verify the changes:

cat ~/project/numbered.txt

You should see the updated text with ordinal numbers.

Working with sed Scripts

In the previous steps, we've been running sed commands directly from the command line. However, for more complex operations, it's often more convenient to create a sed script file. This allows us to:

Organize multiple sed commands
Add comments to explain what each command does
Reuse the same operations on different files

Creating a sed Script

Let's create a simple sed script that performs several text transformations:

cat > ~/project/transform.sed << 'EOF'
## Replace "Linux" with "Ubuntu"
s/Linux/Ubuntu/g

## Replace "user" with "developer"
s/user/developer/g

## Replace "system" with "platform"
s/system/platform/g

## Convert "is" to uppercase
s/is/IS/g
EOF

This script performs four substitutions:

Replaces all occurrences of "Linux" with "Ubuntu"
Replaces all occurrences of "user" with "developer"
Replaces all occurrences of "system" with "platform"
Converts all occurrences of "is" to uppercase "IS"

Let's create a new file to test our script:

cat > ~/project/test.txt << 'EOF'
Linux is a powerful operating system.
The user interface is very customizable.
Many Linux users prefer the command-line interface.
The system resources are efficiently managed.
EOF

Running a sed Script

To run our sed script on the test file, we use the -f option:

sed -f ~/project/transform.sed ~/project/test.txt

You should see:

Ubuntu IS a powerful operating platform.
The developer interface IS very customizable.
Many Ubuntu developers prefer the command-line interface.
The platform resources are efficiently managed.

All the substitutions from our script have been applied!

In-place Editing with a Script

Just like with command-line sed commands, we can use the -i option to perform in-place editing with a script:

sed -i -f ~/project/transform.sed ~/project/test.txt

Let's verify the changes:

cat ~/project/test.txt

The file should now contain the transformed text.

Creating a Backup with a Script

And again, we can create a backup before making changes:

## First, let's restore the original content
cat > ~/project/test.txt << 'EOF'
Linux is a powerful operating system.
The user interface is very customizable.
Many Linux users prefer the command-line interface.
The system resources are efficiently managed.
EOF

## Now run sed with a backup
sed -i.bak -f ~/project/transform.sed ~/project/test.txt

Let's check both the modified file and the backup:

cat ~/project/test.txt
cat ~/project/test.txt.bak

The first command should show the transformed text, while the backup file should contain the original text.

Applying a sed Script to Multiple Files

One of the advantages of using a sed script is that we can easily apply the same transformations to multiple files.

Let's create a few more test files:

## Create test1.txt
cat > ~/project/test1.txt << 'EOF'
Linux offers excellent system performance.
Many users appreciate its stability.
EOF

## Create test2.txt
cat > ~/project/test2.txt << 'EOF'
The Linux community provides great support.
New users can find helpful resources online.
EOF

## Create test3.txt
cat > ~/project/test3.txt << 'EOF'
The system updates are well-managed in Linux.
Users can customize their experience.
EOF

Now, let's apply our sed script to all these files at once:

sed -i -f ~/project/transform.sed ~/project/test1.txt ~/project/test2.txt ~/project/test3.txt

Let's verify the changes:

cat ~/project/test1.txt
cat ~/project/test2.txt
cat ~/project/test3.txt

All three files should show the transformed text according to our script.

This demonstrates the power of sed scripts for batch processing multiple files with the same set of transformations.

Summary

In this lab, you have learned how to use sed, the Stream Editor, to manipulate text files in Linux. You have gained practical experience with several important aspects of sed:

Basic text substitution using the s command
In-place editing with the -i option
Global substitution with the g flag
Creating backups before editing with the -i.bak option
Using different delimiters for substitution patterns
Targeting specific lines with addresses and ranges
Creating and running sed scripts for more complex operations
Applying transformations to multiple files at once

These skills are essential for efficient text processing in Linux environments. The ability to manipulate text files without opening them in an editor is particularly valuable for system administrators, developers, and data analysts who often need to process large volumes of text data or make systematic changes across multiple files.

As you continue to work with Linux, you'll find that sed is an indispensable tool in your text processing toolkit, alongside other utilities like grep, awk, and cut.