Python Regular Expressions

Introduction

A regular expression, or regex, is a powerful tool for text processing. It's a sequence of characters that defines a search pattern, allowing you to find, match, and manage text with incredible flexibility. In Python, regular expression functionality is handled by the re module.

In this lab, you will learn the fundamentals of using regular expressions in Python. We will start with the basics of matching patterns and gradually move to more advanced topics like grouping, using quantifiers for repetition, defining character classes, and leveraging powerful methods and flags to handle complex matching scenarios. By the end, you will be comfortable using regex to perform sophisticated string operations.

Introduction to Regex Matching

In this step, you will learn the basic workflow for using regular expressions in Python. The process typically involves four main steps:

1. Import the re module.
2. Create a Regex object using re.compile(). Using a raw string (r"...") is recommended to prevent backslashes from being interpreted as escape characters.
3. Use the Regex object's search() method to find a pattern in a string. This returns a Match object if the pattern is found.
4. Use the Match object's group() method to get the actual matched text.

First, let's look at some common regex symbols that you will be using.

Now, let's put this into practice. We will write a script to find a phone number in a string. The pattern for a U.S. phone number is XXX-XXX-XXXX, where X is a digit. In regex, \d represents a digit.

Note: If you are not familiar with using WebIDE (VS Code like), please learn it first at this lab.

Open the main.py file from the file explorer on the left and add the following code.

## Import re module for regular expression operations
import re

## re.compile(): create regex pattern object (use raw string r'' to avoid escaping)
## The pattern \d\d\d-\d\d\d-\d\d\d\d matches three digits, a hyphen, three digits, a hyphen, and four digits.
phone_num_regex = re.compile(r'\d\d\d-\d\d\d-\d\d\d\d')

## The search() method scans the string, looking for the first location where this regex produces a match.
mo = phone_num_regex.search('My number is 415-555-4242.')

## The group() method returns the string of the matched text.
print(f'Phone number found: {mo.group()}')

To run the script, open a terminal and execute the following command:

python3 main.py

You should see the matched phone number printed to the console.

Phone number found: 415-555-4242

This basic workflow is the foundation for all regex operations in Python.

Grouping and Alternation

In this step, you'll learn how to isolate specific parts of your matched text using groups and how to match one of several possible patterns using alternation.

Grouping with Parentheses ()

Parentheses () are used to create groups within a regular expression. This is useful when you want to extract parts of a match, like the area code from a phone number. You can access individual groups with group(n), where group(1) is the first group, group(2) is the second, and so on. group(0) or group() returns the entire matched string.

Let's modify main.py to separate the area code from the rest of the phone number.

import re

## Parentheses create groups: group(1) for the first group, group(2) for the second.
## The first group (\d\d\d) captures the area code.
## The second group (\d\d\d-\d\d\d\d) captures the main number.
phone_num_regex = re.compile(r'(\d\d\d)-(\d\d\d-\d\d\d\d)')
mo = phone_num_regex.search('My number is 415-555-4242.')

## To retrieve all groups at once, use the groups() method, which returns a tuple.
area_code, main_number = mo.groups()

print(f"Area Code: {area_code}")
print(f"Main Number: {main_number}")
print(f"Full Match: {mo.group(0)}")

Run the script:

python3 main.py

The output will show the separated parts of the number:

Area Code: 415
Main Number: 555-4242
Full Match: 415-555-4242

Alternation with the Pipe |

The pipe character | acts like an "OR" operator. It allows you to match one of several expressions. For example, Batman|Tina Fey will match either "Batman" or "Tina Fey".

You can also use the pipe within a group to match one of several possible patterns as part of your regex. Let's try matching different "Bat-" related terms. Update your main.py with the following code:

import re

## The pipe | matches one of several patterns within a group.
## Here, it matches 'man', 'mobile', 'copter', or 'bat'.
bat_regex = re.compile(r'Bat(man|mobile|copter|bat)')
mo = bat_regex.search('Batmobile lost a wheel')

print(f"Full match: {mo.group()}")
print(f"Group 1: {mo.group(1)}")

Run the script again:

python3 main.py

You will see that the regex matched "Batmobile", and the first group captured "mobile".

Full match: Batmobile
Group 1: mobile

Repetition with Quantifiers

In this step, we'll explore quantifiers, which are special characters that let you specify how many times a preceding group should repeat.

Optional Matching with ?

The question mark ? makes the preceding group optional, meaning it will match zero or one time. This is perfect for patterns that may or may not appear. For example, we can match both "Batman" and "Batwoman".

Update main.py with this code:

import re

## The (wo)? group is optional. It will match 'wo' if it exists, or match nothing if it doesn't.
bat_regex = re.compile(r'Bat(wo)?man')

mo1 = bat_regex.search('The Adventures of Batman')
print(mo1.group())

mo2 = bat_regex.search('The Adventures of Batwoman')
print(mo2.group())

Run the script:

python3 main.py

The output shows that both strings were matched successfully:

Batman
Batwoman

Zero or More with *

The asterisk * (or star) matches the preceding group zero or more times. This is useful when a pattern can repeat any number of times, including not at all.

import re

## The (wo)* group can appear zero or more times.
bat_regex = re.compile(r'Bat(wo)*man')

mo1 = bat_regex.search('The Adventures of Batman') ## Zero 'wo'
print(mo1.group())

mo2 = bat_regex.search('The Adventures of Batwowowowoman') ## Four 'wo's
print(mo2.group())

One or More with +

The plus sign + matches the preceding group one or more times. Unlike the star, the group must appear at least once.

import re

## The (wo)+ group must appear at least once.
bat_regex = re.compile(r'Bat(wo)+man')

mo1 = bat_regex.search('The Adventures of Batwoman') ## One 'wo'
print(mo1.group())

mo2 = bat_regex.search('The Adventures of Batman') ## Zero 'wo's
print(mo2 is None) ## This will print True because no match is found.

Specific Repetitions with {}

Curly brackets {} allow you to specify an exact number of repetitions, or a range.

• {3} matches exactly three times.
• {3,5} matches between three and five times.
• {3,} matches three or more times.
• {,5} matches up to five times.

import re

## (Ha){3} matches 'Ha' repeated exactly 3 times.
ha_regex = re.compile(r'(Ha){3}')
mo1 = ha_regex.search('HaHaHa')
print(mo1.group())

Greedy and Non-Greedy Matching

By default, Python's regex is "greedy," meaning it matches the longest possible string. To make it "non-greedy" (match the shortest possible string), add a question mark ? after the quantifier.

Let's see the difference. Clear main.py and add the following:

import re

## Greedy match: {3,5} matches the longest possible string (5 'Ha's).
greedy_ha_regex = re.compile(r'(Ha){3,5}')
mo1 = greedy_ha_regex.search('HaHaHaHaHa')
print(f"Greedy match: {mo1.group()}")

## Non-greedy match: {3,5}? matches the shortest possible string (3 'Ha's).
non_greedy_ha_regex = re.compile(r'(Ha){3,5}?')
mo2 = non_greedy_ha_regex.search('HaHaHaHaHa')
print(f"Non-greedy match: {mo2.group()}")

Run the script:

python3 main.py

The output clearly shows the difference:

Greedy match: HaHaHaHaHa
Non-greedy match: HaHaHa

Character Classes Anchors and Wildcards

In this step, you'll learn more advanced techniques, including finding all matches, creating custom character sets, anchoring matches to the start or end of a string, and using wildcards.

The findall() Method

While search() returns only the first match, the findall() method returns a list of all non-overlapping matches in the string. If the regex has groups, findall() returns a list of tuples, where each tuple contains the strings for each group.

Update main.py with this example:

import re

## If the regex has no groups, findall() returns a list of strings.
phone_num_regex = re.compile(r'\d\d\d-\d\d\d-\d\d\d\d')
matches = phone_num_regex.findall('Cell: 415-555-9999 Work: 212-555-0000')
print(matches)

Run the script to see the output:

python3 main.py

['415-555-9999', '212-555-0000']

Making Your Own Character Classes []

Square brackets [] let you define a custom character class that will match any single character within it. For example, [aeiou] matches any lowercase vowel. You can also specify a range, like [a-z] for any lowercase letter or [0-9] for any digit.

A caret ^ as the first character inside the brackets creates a negative character class, which matches any character not in the class.

Let's find all vowels in a string. Update main.py:

import re

## [aeiouAEIOU] matches any vowel, lowercase or uppercase.
vowel_regex = re.compile(r'[aeiouAEIOU]')
vowels = vowel_regex.findall('Robocop eats baby food. BABY FOOD.')
print(vowels)

Anchors: Caret ^ and Dollar Sign $

Anchors are used to assert a position in the string.

• The caret ^ matches the beginning of the string.
• The dollar sign $ matches the end of the string.

When used together (^...$), they enforce that the entire string must match the pattern.

import re

## ^Hello matches strings that start with 'Hello'.
begins_with_hello = re.compile(r'^Hello')
match1 = begins_with_hello.search('Hello world!')
print(match1 is not None) ## True

## \d+$ matches strings that end with one or more digits.
ends_with_number = re.compile(r'\d+$')
match2 = ends_with_number.search('Your number is 42')
print(match2.group()) ## '42'

## ^\d+$ matches strings that consist only of digits.
whole_string_is_num = re.compile(r'^\d+$')
match3 = whole_string_is_num.search('1234567890')
print(match3 is not None) ## True

The Wildcard Character .

The dot . (or wildcard) character matches any single character except a newline. It's a useful shorthand for "any character".

Let's find all three-letter words ending in "at". Update main.py:

import re

## The . character matches any character (except newline).
## So, .at matches 'cat', 'hat', 'sat', etc.
at_regex = re.compile(r'.at')
matches = at_regex.findall('The cat in the hat sat on the flat mat.')
print(matches)

Run the script:

python3 main.py

The output will be a list of all matched words:

['cat', 'hat', 'sat', 'lat', 'mat']

Advanced Methods and Flags

In this final step, we'll cover some powerful methods and flags that modify the behavior of your regular expressions, making them even more versatile.

Matching Everything with Dot-Star .*

The combination .* is a common pattern that matches any character (.) repeated zero or more times (*). It's used to match any sequence of characters. By default, it's greedy and will match as much text as possible.

import re

## The pattern (.*) captures everything between 'First Name: ' and ' Last Name: '.
name_regex = re.compile(r'First Name: (.*) Last Name: (.*)')
mo = name_regex.search('First Name: Al Last Name: Sweigart')
print(f"First Name: {mo.group(1)}")
print(f"Last Name: {mo.group(2)}")

To make it non-greedy, use .*?. This will match the shortest possible string.

Matching Newlines with re.DOTALL

By default, the dot . character does not match newline characters (\n). To change this, you can pass re.DOTALL as a second argument to re.compile(). This flag allows . to match all characters, including newlines.

import re

## With re.DOTALL, '.*' will match across multiple lines.
newline_regex = re.compile('.*', re.DOTALL)
mo = newline_regex.search('Serve the public trust.\nProtect the innocent.\nUphold the law.')
print(mo.group())

Case-Insensitive Matching with re.IGNORECASE

To make your regex ignore case (e.g., match "cat", "Cat", and "CAT"), pass re.IGNORECASE or its shorthand re.I as the second argument to re.compile().

import re

## re.I makes the pattern 'robocop' match 'Robocop', 'ROBOCOP', etc.
robocop = re.compile(r'robocop', re.I)
match = robocop.search('Robocop is part man, part machine, all cop.')
print(match.group())

Substituting Strings with the sub() Method

The sub() method finds all occurrences of a pattern and replaces them with a specified string. It takes two arguments: the replacement string and the string to search.

Let's replace agent names with "CENSORED". Update main.py:

import re

## The pattern 'Agent \w+' matches 'Agent ' followed by one or more word characters.
names_regex = re.compile(r'Agent \w+')
result = names_regex.sub('CENSORED', 'Agent Alice gave the secret documents to Agent Bob.')
print(result)

Run the script:

python3 main.py

The output will show the substituted string:

CENSORED gave the secret documents to CENSORED.

Managing Complex Regexes with re.VERBOSE

When regex patterns become complex, they can be hard to read. By passing re.VERBOSE to re.compile(), you can write your regex across multiple lines and add comments to explain different parts. Whitespace and comments (#) within the string are ignored.

This complex regex: phone_regex = re.compile(r'((\d{3}|\(\d{3}\))?(\s|-|\.)?\d{3}(\s|-|\.)\d{4}(\s*(ext|x|ext.)\s*\d{2,5})?)')

Becomes this readable version with re.VERBOSE:

import re

phone_regex = re.compile(r'''(
    (\d{3} | \(\d{3}\))?          ## area code (optional)
    (\s | - | \.)?                ## separator (optional)
    \d{3}                         ## first 3 digits
    (\s | - | \.)                 ## separator
    \d{4}                         ## last 4 digits
    (\s*(ext|x|ext.)\s*\d{2,5})?  ## extension (optional)
    )''', re.VERBOSE)

mo = phone_regex.search("My phone number is 415-555-4242 ext. 55")
if mo:
    print(f"Found: {mo.group()}")

This makes your code much more maintainable.

Summary

In this lab, you have mastered the fundamentals of regular expressions in Python using the re module. You learned the complete workflow from compiling a pattern with re.compile() to finding matches with search() and retrieving text with group().

You explored key concepts including:

• Grouping with () to capture specific parts of a match.
• Alternation with | to match one of several patterns.
• Quantifiers like ?, *, +, and {} to control repetition.
• Greedy vs. Non-greedy matching to control the length of matches.
• Advanced tools like findall() to get all matches, character classes [] for custom sets, and anchors ^ and $ to pin matches to string boundaries.
• Powerful flags such as re.DOTALL for multiline matching, re.IGNORECASE for case-insensitivity, and re.VERBOSE for writing clean, commented regexes.
• String substitution with the sub() method.

With these skills, you are now equipped to handle a wide range of text processing tasks, from simple data validation to complex data extraction and manipulation.