One Cut Into Two

Introduction

In this project, you will learn how to implement a subword tokenizer, which is a crucial step in natural language processing tasks. Tokenization is the process of breaking down a string of text into smaller units, called tokens, which can be individual words, characters, or subwords. This project focuses on subword-level tokenization, which is commonly used in English and other Latin-based languages.

👀 Preview

['I', 'studied', 'in', 'LabEx', 'for', '1', '0', 'days', 'and', 'completed', 'the', '[UNK]', '[UNK]', 'course', '.']

🎯 Tasks

In this project, you will learn:

• How to implement a subword tokenizer function that performs character-level tokenization using the greedy longest-match-first algorithm
• How to test the subword tokenizer with a provided example and analyze the output
• How to understand the tokenization algorithm and its implementation

🏆 Achievements

After completing this project, you will be able to:

• Understand the importance of tokenization in natural language processing tasks
• Implement a core component of a natural language processing pipeline
• Differentiate between character-level and subword-level tokenization
• Apply the greedy longest-match-first algorithm to tokenize text into subwords

Skills Graph

Understand the Tokenization Process

In this step, you will learn about the tokenization process and its importance in natural language processing tasks.

Tokenization is the process of breaking down a string of text into smaller units, called tokens. These tokens can be individual words, characters, or subwords, depending on the specific tokenization method used.

In natural language processing tasks, most machine learning models do not support string data directly. In order for the models to learn effectively, string data must be numericalized, which is a process known as Tokenization. Tokenization is also the preparation for numericalization, and numericalization requires a mapping, which is provided by a mapping table.

Character-level tokenization is a method that divides strings based on the smallest symbols in a language and is commonly used in English tokenization.

Subword-level tokenization is commonly used in English and other Latin-based languages and is an improvement over word-level tokenization.

Implement the Subword Tokenizer

In this step, you will implement a subword tokenizer function that performs character-level tokenization for English words using the greedy longest-match-first algorithm. This function also removes all symbols, including spaces, from the input string.

Open the subword_tokenizer.py file in your code editor. This file contains a skeleton of the subword_tokenizer() function. Your task is to fill in the missing parts of the function.

The function should have the following requirements:

1. The function should take a string as input. This function receives a string containing English characters, numbers, and punctuation marks. The string has been provided to you, and you cannot change the content of the string.
2. The function should return the tokenization result as a list.

Here's the complete code for the subword_tokenizer() function:

import re

def subword_tokenizer(text, vocab) -> list:
    """
    Tokenizes input text into subwords based on a provided vocabulary.

    Args:
    - text (str): Input text to be tokenized.
    - vocab (list): Vocabulary list containing subwords.

    Returns:
    - list: List of subword tokens.
    """

    def is_in_vocab(word) -> bool:
        """
        Checks if a given word is in the vocabulary.

        Args:
        - word (str): Word to check.

        Returns:
        - bool: True if the word is in the vocabulary, False otherwise.
        """
        return word in vocab

    def find_longest_match(word) -> tuple:
        """
        Finds the longest matching subword for a given word in the vocabulary.

        Args:
        - word (str): Word to find a match for.

        Returns:
        - tuple: A tuple containing the longest matching subword and the remaining part of the word.
        """
        for i in range(len(word), 0, -1):
            subword = word[:i]
            if is_in_vocab(subword):
                return subword, word[i:]
        return "[UNK]", ""

    tokens = []
    ## Remove non-alphanumeric characters and split the text into words
    words = re.findall(r"\b\w+\b", text)

    for word in words:
        while word:
            subword, remaining = find_longest_match(word)
            tokens.append(subword)
            word = remaining

    return tokens


if __name__ == "__main__":
    ## Example usage:
    vocab = [
        "I",
        "studied",
        "in",
        "LabEx",
        "for",
        "1",
        "0",
        "days",
        "and",
        "completed",
        "the",
        "course",
    ]
    text = "I studied in LabEx for 10 days and completed the TensorFlow Serving course."

    tokenization_result = subword_tokenizer(text, vocab)
    print(tokenization_result)

Test the Subword Tokenizer

In this step, you will test the subword_tokenizer() function with the provided example and verify the output.

Run the subword_tokenizer.py script in your terminal:

python3 subword_tokenizer.py

The output should be similar to the following:

['I', 'studied', 'in', 'LabEx', 'for', '1', '0', 'days', 'and', 'completed', 'the', '[UNK]', '[UNK]', 'course', '.']

Observe the output and make sure that the tokenization process is working as expected. The function should tokenize the input text into a list of subwords, where the unknown words are represented by the [UNK] token.

Understand the Tokenization Algorithm

In this step, you will dive deeper into the implementation of the subword_tokenizer() function and understand the tokenization algorithm.

The subword_tokenizer() function uses a greedy longest-match-first algorithm to tokenize the input text. Here's how the algorithm works:

1. The function first removes all non-alphanumeric characters from the input text and splits the text into individual words.
2. For each word, the function tries to find the longest matching subword in the provided vocabulary.
3. If a subword is found in the vocabulary, it is added to the list of tokens. If no subword is found, the [UNK] token is added to the list.
4. The process continues until all words in the input text have been tokenized.

The is_in_vocab() function is a helper function that checks if a given word is present in the provided vocabulary.

The find_longest_match() function is the core of the tokenization algorithm. It iterates through the word, starting from the longest possible subword, and checks if the current subword is in the vocabulary. If a match is found, it returns the subword and the remaining part of the word. If no match is found, it returns the [UNK] token and an empty string.

Understanding the tokenization algorithm will help you to further improve the subword tokenizer or adapt it to different use cases.

Summary

Congratulations! You have completed this project. You can practice more labs in LabEx to improve your skills.