Text Tokenization with Python

Introduction

In this project, you will learn how to implement a text tokenization system using Python. Text tokenization is a fundamental task in natural language processing, where a given text is broken down into smaller units called tokens. These tokens can represent words, numbers, punctuation, or other meaningful elements in the text. The ability to tokenize text is essential for many applications, such as lexical analysis in compilers, sentiment analysis, and text classification.

👀 Preview

## text = 'total = 1 + 2 * 3'

tokens = [Token(type='NAME', value='total'), Token(type='WS', value=' '), Token(type='EQ', value='='), Token(type='WS', value=' '), Token(type='NUM', value='1'), Token(type='WS', value=' '), Token(type='ADD', value='+'), Token(type='WS', value=' '), Token(type='NUM', value='2'), Token(type='WS', value=' '), Token(type='MUL', value='*'), Token(type='WS', value=' '), Token(type='NUM', value='3')]

🎯 Tasks

In this project, you will learn:

• How to define a Token class to represent the tokens in the text
• How to implement a generate_tokens function that takes an input text and generates a stream of tokens
• How to test the tokenization process with a sample text

🏆 Achievements

After completing this project, you will be able to:

• Understand the concept of text tokenization and its importance in natural language processing
• Implement a basic text tokenization system using Python
• Customize the tokenization process by defining different token types and their corresponding regular expressions
• Test and debug the tokenization system with various input texts

Defining the Token Class

In this step, you will learn how to define the Token class, which will represent the tokens in the text tokenization process.

1. Open the /home/labex/project/texttokenizer.py file in a text editor.

2. At the beginning of the file, import the namedtuple function from the collections module:

   from collections import namedtuple
   

3. Define the Token class as a named tuple with two attributes: type and value.

   Token = namedtuple("Token", ["type", "value"])
   

Implementing the generate_tokens Function

In this step, you will implement the generate_tokens function, which will take the input text and generate a stream of tokens.

1. In the texttokenizer.py file, define the generate_tokens function:

   def generate_tokens(text):
       ## Define token types and corresponding regular expressions
       token_specification = {
           "NAME": r"[a-zA-Z_][a-zA-Z_0-9]*",
           "NUM": r"\d+",
           "ADD": r"\+",
           "SUB": r"-",
           "MUL": r"\*",
           "DIV": r"/",
           "EQ": r"=",
           "WS": r"\s+",
       }
   
       ## Combine regular expressions into one large regular expression
       regex = "|".join("(?P<%s>%s)" % pair for pair in token_specification.items())
   
       scanner = re.finditer(regex, text)
       for m in scanner:
           type = m.lastgroup
           value = m.group()
           yield Token(type, value)
   

2. The generate_tokens function first defines a dictionary token_specification that maps the token types to their corresponding regular expressions.

3. It then combines all the regular expressions into a single large regular expression using the | operator.

4. The function then uses the re.finditer function to find all the matches in the input text and yields a Token object for each match, with the token type and value.

Testing the Tokenization

In this step, you will test the generate_tokens function by parsing a sample text.

1. At the end of the texttokenizer.py file, add the following code:

   if __name__ == "__main__":
       text = "total = 1 + 2 * 3"
       tokens = list(generate_tokens(text))
       print(tokens)
   

2. Save the texttokenizer.py file.

3. Run the texttokenizer.py script from the /home/labex/project directory:

   python texttokenizer.py
   

4. The output should be a list of Token objects representing the tokens in the input text:

   [Token(type='NAME', value='total'), Token(type='WS', value=' '), Token(type='EQ', value='='), Token(type='WS', value=' '), Token(type='NUM', value='1'), Token(type='WS', value=' '), Token(type='ADD', value='+'), Token(type='WS', value=' '), Token(type='NUM', value='2'), Token(type='WS', value=' '), Token(type='MUL', value='*'), Token(type='WS', value=' '), Token(type='NUM', value='3')]
   

Congratulations! You have successfully implemented the generate_tokens function and tested it with a sample text. In the next step, you will learn how to use the generate_tokens function to tokenize a larger text.

Summary

Congratulations! You have completed this project. You can practice more labs in LabEx to improve your skills.