Play with Your Text Data

Introduction

Python is a powerful and versatile programming language widely used for data analysis and statistical computing. Its rich ecosystem includes libraries specifically designed for text analysis and natural language processing, making it an excellent choice for working with textual data.

In this challenge, we will leverage Python's capabilities to perform text-based statistical analyses on a collection of text files. Let's explore how we can extract meaningful insights from text data using Python.

Word Total Count

You will find several text files located in the home/labex/files folder.

Your task is to write a Python script, named word_count.py, that reads all of these text files and calculates the total number of words across all of them.

It's important to note that punctuations are not considered words. For example, in the "java" file, there are 111 words.

Requirements

• The script should print the total word count to the console when executed.

High-Frequency Words

Having successfully counted the total words, your next task is to identify the top 3 most frequent words across all the text files. You need to write a Python script, top_3_high_frequencies.py, to achieve this. The script should then print these top 3 words along with their frequencies in descending order of frequency to the console.

For example, the output should look like this:

python top_3_high_frequencies.py

## print word and frequency in console
word1 20
word2 15
word3 13

Requirements

• The script should print the top 3 words and their counts to the console when executed.
• Word counting is case-sensitive, meaning "Word" and "word" are treated as distinct words.
• Punctuations are not considered part of words and should be excluded from the count.

Words Line Up in Order

Now, let's consider the order of words within each file. What if we wanted to collect the first word from each file, then the second word from each file, and so on?

Your task is to write a Python script, step3_code.py, that takes the n-th word from each input file and writes them into a new file named output/n. Here, 'n' represents the word position (starting from 1). The output files should be created in the /home/labex/project/output/ directory.

For example, if we consider the first words of each file, the content of output/1 should be:

## output/1, start count with 1.
CentOS Java A Python Ubuntu

Similarly, for the 100th words (if they exist), the content of output/100 should be:

## output/100, the 100-th file, only java, linux and program have 100-th word.
applications and the

Requirements

• The output folder should be located at /home/labex/project/.
• The order in which files are read does not matter; only the word order within each file is important.
• Punctuations are not considered part of words and should be excluded.

Summary

In this Python challenge, you've learned how to use Python for basic text data analysis. You've practiced counting total words, identifying top frequency words, and extracting words based on their position in multiple text files, writing the results to separate output files. By completing this challenge, you've gained valuable skills for working with text data in Python, enhancing your ability to perform text-based statistical analyses. These skills form a foundation for more advanced text processing tasks in the future.