How to perform global substitution with sed in Linux

Introduction

Sed, the "stream editor," is a versatile command-line tool in the Linux/Unix ecosystem that specializes in text manipulation. In this tutorial, you will explore the fundamental concepts of sed, its key features, and how it can be leveraged to master text manipulation tasks, from simple find-and-replace to complex pattern-matching and transformation workflows.

Skills Graph

Introduction to sed: Text Manipulation Mastery

Sed, short for "stream editor," is a powerful command-line tool in the Linux/Unix ecosystem that specializes in text manipulation. It allows you to perform a wide range of operations on text files, from simple find-and-replace tasks to complex pattern-matching and transformation workflows.

In this section, we will explore the fundamental concepts of sed, its key features, and how it can be leveraged to master text manipulation tasks.

Understanding sed

Sed is a non-interactive, line-oriented text editor that reads input, applies one or more editing commands to it, and writes the results to the output. It operates on a line-by-line basis, making it highly efficient for processing large text files.

Key Features of sed

1. Global Substitution: Sed excels at performing global search-and-replace operations, allowing you to replace patterns across an entire file or set of files.
2. Regular Expressions: Sed utilizes powerful regular expressions to match and manipulate text patterns, enabling advanced text processing capabilities.
3. Scripting and Automation: Sed scripts can be created and executed to automate repetitive text manipulation tasks, improving efficiency and productivity.

Sed Syntax and Basic Commands

Sed follows a specific syntax for its commands, typically in the format: sed [options] 'command' file(s). Some of the most commonly used sed commands include:

• s/pattern/replacement/: Performs a global substitution of the specified pattern with the replacement text.
• d: Deletes the current line.
• p: Prints the current line.
• i: Inserts text before the current line.
• a: Appends text after the current line.

Practical Examples

Let's explore some practical examples of using sed for text manipulation tasks:

1. Global Substitution: Replace all occurrences of "old_string" with "new_string" in a file:

   sed 's/old_string/new_string/g' input_file.txt

2. Deleting Lines: Remove all lines containing the pattern "unwanted_text" from a file:

   sed '/unwanted_text/d' input_file.txt

3. Inserting Text: Insert a new line before the first line of a file:

   sed '1i\This is a new line' input_file.txt

By mastering the fundamentals of sed, you can unlock the power of text manipulation and streamline various tasks in your Linux/Unix environment.

Advanced sed Substitution Techniques

While the basic substitution command s/pattern/replacement/ is a powerful tool, sed offers advanced substitution techniques that can significantly expand your text manipulation capabilities. In this section, we will explore some of these advanced techniques and their practical applications.

Capturing and Referencing Matched Patterns

Sed allows you to capture parts of the matched pattern using parentheses () and reference them in the replacement string using the special & and \1, \2, etc. markers.

## Replace the first word in each line with "REPLACED"
sed 's/\w+/REPLACED/' input_file.txt

## Capitalize the first letter of each line
sed 's/^./\U&/' input_file.txt

## Swap the first two words in each line
sed 's/(\w+) (\w+)/\2 \1/' input_file.txt

Multi-Line Substitutions

Sed can also perform substitutions that span multiple lines using the newline character \n in the pattern and replacement strings.

## Replace the first line with the last line
sed '1{/.*/{h;d};$G' input_file.txt

## Reverse the order of lines in a file
sed '1!G;h;$!d' input_file.txt

Conditional Substitutions

Sed supports conditional substitutions, allowing you to apply different replacement patterns based on specific conditions.

## Replace "foo" with "bar" only if the line contains "baz"
sed '/baz/s/foo/bar/g' input_file.txt

## Delete lines that contain both "foo" and "bar"
sed '/foo/!d;/bar/!d' input_file.txt

Extended Regular Expressions

Sed can utilize extended regular expressions (ERE) to perform more complex pattern matching and substitutions.

## Replace all occurrences of "foo", "bar", or "baz" with "replaced"
sed 's/foo\|bar\|baz/replaced/g' input_file.txt

## Extract all email addresses from a file
sed -n '/\w\+@\w\+\.\w\+/p' input_file.txt

By mastering these advanced sed substitution techniques, you can unlock the full potential of text manipulation and streamline your workflows in the Linux/Unix environment.

Automating sed Workflows for Efficiency

While sed is a powerful tool for text manipulation, its true potential lies in its ability to be automated and integrated into larger workflows. In this section, we will explore how to leverage sed scripting and shell scripts to streamline your text processing tasks and boost productivity.

Creating sed Scripts

Sed commands can be stored in a file and executed as a script, allowing you to automate complex text manipulation tasks. This approach is particularly useful when you need to perform the same operations on multiple files or on a regular basis.

## Example sed script: replace.sed
s/old_string/new_string/g
s/foo/bar/g
d

To run the script:

sed -f replace.sed input_file.txt

Integrating sed with Shell Scripts

Sed can be seamlessly integrated into shell scripts, enabling you to create powerful text processing pipelines. By combining sed with other shell utilities, you can automate complex workflows and reduce manual effort.

#!/bin/bash

## Extract email addresses from a file
sed -n '/\w\+@\w\+\.\w\+/p' input_file.txt | sort | uniq > email_list.txt

## Replace all occurrences of "foo" with "bar" in multiple files
for file in *.txt; do
    sed 's/foo/bar/g' "$file" > "${file%.txt}_modified.txt"
done

Advanced Sed Scripting Techniques

Sed scripts can be further enhanced with features like conditional logic, loops, and variables to create more sophisticated and flexible text processing workflows.

## Example: Conditional sed script
/pattern1/ {
    s/old/new/
    a\
    This line will be appended
}
/pattern2/ {
    d
}

By automating your sed workflows, you can streamline repetitive text manipulation tasks, improve efficiency, and enhance your productivity in the Linux/Unix environment.

Summary

This tutorial has provided a comprehensive introduction to the sed text manipulation tool in Linux. You have learned about the key features of sed, including global substitution, regular expressions, and scripting capabilities. By exploring practical examples, you now have the knowledge to apply sed for a wide range of text processing tasks, from simple search-and-replace to advanced text transformations. Mastering sed will empower you to streamline your text manipulation workflows and improve your productivity in the Linux environment.