ASCII Art Animation with OpenCV

Introduction

In this project, we will use OpenCV to process images and videos to create ASCII art animations.

• OpenCV compilation
• Image and video processing using OpenCV
• Principles of image to ASCII art conversion
• Daemon threads
• Cursor positioning and escape encoding

👀 Preview

🎯 Tasks

In this project, you will learn:

• How to convert images and videos into ASCII art animations using OpenCV.
• How to play ASCII art animations in the terminal using cursor positioning and escape encoding.
• How to export and load ASCII art animation data.

🏆 Achievements

After completing this project, you will be able to:

• Use OpenCV to process images and videos.
• Convert images into ASCII art.
• Play ASCII art animations.
• Export and load ASCII art animation data.

Creating File

Everyone should understand that a video is actually a series of images, so the basic principle of converting a video into ASCII animation is to convert images into ASCII art.

Here's a simple explanation of the principle behind converting images into ASCII art: First, the image is converted into a grayscale image, where each pixel only contains brightness information (represented by the values 0-255). Then, we create a limited character set, where each character corresponds to a range of brightness values. We can then represent each pixel with the corresponding character based on this correspondence and the brightness information of the pixel, thus creating ASCII art.

For ASCII animation to be meaningful, it needs to be playable. The most crude and simplistic way to do this is to open the ASCII animation text file in a text editor and repeatedly press the PageDown key. However, this approach is really too simple and crude, and not at all elegant.

Instead, we can play the ASCII animation in the terminal by outputting one frame at a time. However, this approach has a major downside: during playback, you will notice that the scrollbar on the right side of the terminal becomes smaller and smaller (if it exists); after playback, if you scroll up in the terminal, all you see is the previously outputted ASCII art, and all the command history before playback is pushed out. A solution to this problem will be provided later in this project.

Create a CLIPlayVideo.py file in the ~/project directory.

cd ~/project
touch CLIPlayVideo.py

Then, Install the opencv-python modules:

sudo pip install opencv-python

Creating the CharFrame class

To avoid forgetting, import the necessary packages:

import sys
import os
import time
import threading
import termios
import tty
import cv2
import pyprind

The last module, pyprind, provides a progress bar for display. Since converting a video to a character animation is a time-consuming process, we need a progress bar to view the progress and approximate remaining time, providing a more intuitive understanding of the program's status. Install it as follows:

sudo pip3 install pyprind

In this project, in addition to converting video files into character animations and playing them, we also added the functionality of converting image files into character art. Therefore, in our program design, we have three classes: CharFrame, I2Char, and V2Char, with the latter two classes inheriting from the first class.

The CharFrame class:

class CharFrame:

    ascii_char = "$@B%8&WM#*oahkbdpqwmZO0QLCJUYXzcvunxrjft/\|()1{}[]?-_+~<>i!lI;:,\"^`'. "

    ## Map pixels to characters
    def pixelToChar(self, luminance):
        return self.ascii_char[int(luminance/256*len(self.ascii_char))]

    ## Convert a regular frame to an ASCII character frame
    def convert(self, img, limitSize=-1, fill=False, wrap=False):
        if limitSize != -1 and (img.shape[0] > limitSize[1] or img.shape[1] > limitSize[0]):
            img = cv2.resize(img, limitSize, interpolation=cv2.INTER_AREA)
        ascii_frame = ''
        blank = ''
        if fill:
            blank += ' '*(limitSize[0]-img.shape[1])
        if wrap:
            blank += '\n'
        for i in range(img.shape[0]):
            for j in range(img.shape[1]):
                ascii_frame += self.pixelToChar(img[i,j])
            ascii_frame += blank
        return ascii_frame

The ascii_char attribute can be adjusted according to the video you want to convert.

The pixelToChar() method takes a single parameter, luminance, which receives the brightness information of the pixel. It should be noted that the expression in the return statement of the method uses the value 256. Although the brightness range of the pixel is 0~255, changing 256 to 255 in this expression may cause an IndexError exception.

The convert() method has one positional parameter and three optional parameters. The img parameter receives an object of type numpy.ndarray, which is the object returned by OpenCV when opening an image. Similarly, the frames from the video obtained later using OpenCV will also be of this type. The limitSize parameter accepts a tuple that represents the maximum width and height of the image. The fill parameter indicates whether to fill the width of the image to the maximum width with spaces, and the wrap parameter indicates whether to add a line break at the end of each row.

img.shape returns a tuple containing the number of rows (height), columns (width), and color channels of the image. If the image is grayscale, it does not include the number of color channels.

The cv2.resize() function is used to resize the image. The first parameter is the numpy.ndarray object, and the second is the desired width and height of the resized image. The interpolation parameter specifies the interpolation method. Several interpolation methods are available, as explained on the OpenCV official website:

Preferable interpolation methods are cv2.INTER_AREA for shrinking and cv2.INTER_CUBIC (slow) & cv2.INTER_LINEAR for zooming. By default, the interpolation method used is cv2.INTER_LINEAR for all resizing purposes.

img[i,j] returns a list of the BGR values of the pixel at the i-th row and j-th column of the image, or the brightness value of the corresponding pixel for grayscale images.

Creating I2Char class

Here is the I2Char class that inherits from CharFrame:

class I2Char(CharFrame):

    result = None

    def __init__(self, path, limitSize=-1, fill=False, wrap=False):
        self.genCharImage(path, limitSize, fill, wrap)

    def genCharImage(self, path, limitSize=-1, fill=False, wrap=False):
        img = cv2.imread(path, cv2.IMREAD_GRAYSCALE)
        if img is None:
            return
        self.result = self.convert(img, limitSize, fill, wrap)

    def show(self, stream = 2):
        if self.result is None:
            return
        if stream == 1 and os.isatty(sys.stdout.fileno()):
            self.streamOut = sys.stdout.write
            self.streamFlush = sys.stdout.flush
        elif stream == 2 and os.isatty(sys.stderr.fileno()):
            self.streamOut = sys.stderr.write
            self.streamFlush = sys.stderr.flush
        elif hasattr(stream, 'write'):
            self.streamOut = stream.write
            self.streamFlush = stream.flush
        self.streamOut(self.result)
        self.streamFlush()
        self.streamOut('\n')

cv2.imread() reads an image and returns a numpy.ndarray object. The first parameter is the path to the image to be opened, and the second parameter specifies the way the image is opened and can have the following three values:

• cv2.IMREAD_COLOR: Loads a color image, ignoring the alpha channel.
• cv2.IMREAD_GRAYSCALE: Loads the image in grayscale mode.
• cv2.IMREAD_UNCHANGED: Loads the image with alpha channel included.

The show() method accepts a parameter indicating which output stream to use. 1 represents standard output sys.stdout, and 2 represents standard error output sys.stderr. sys.stdout.fileno() and sys.stderr.fileno() respectively return the file descriptor for standard output and standard error output. os.isatty(fd) returns a boolean value indicating whether the file descriptor fd is open and connected to a tty device.

Create the V2Char class

Then our focus is on the V2Char class, which inherits from the CharFrame class.

First, let's think about our class. One of its attributes is charVideo, which is a list used to store all the data for the character animation.

Then we have two main methods: one is the genCharVideo() method, which converts a video file into a character animation, and the other is the play() method, which plays the character animation.

Additionally, since converting from video to character animation is a time-consuming process, we can export the character animation data from charVideo to facilitate future playback. This means we need export and load methods, namely the export() method and the load() method.

The class also needs an initialization method that takes the file path to be read as a parameter. If the file is a exported txt file, it will call the load() method to load the data into the charVideo attribute. Otherwise, it will be treated as a video file and will call the genCharVideo() method to convert the video into a character animation and store it in the charVideo attribute.

class V2Char(CharFrame):

    def __init__(self, path):
        if path.endswith('txt'):
            self.load(path)
        else:
            self.genCharVideo(path)

Next is the genCharVideo() method:

def genCharVideo(self, filepath):
    self.charVideo = []
    cap = cv2.VideoCapture(filepath) ## Use the `cv2.VideoCapture()` method to read the video file
    self.timeInterval = round(1/cap.get(5), 3) ## Get the frame rate of the video
    nf = int(cap.get(7)) ## Get the total number of frames in the video
    print('Generate char video, please wait...')
    for i in pyprind.prog_bar(range(nf)):
        rawFrame = cv2.cvtColor(cap.read()[1], cv2.COLOR_BGR2GRAY)
        frame = self.convert(rawFrame, os.get_terminal_size(), fill=True) ## Convert the raw frame into a character frame
        self.charVideo.append(frame)
    cap.release() ## Release the resource

The cv2.VideoCapture() method is used to read the video file, and the returned object is assigned to cap.

Using the cap.get() method, we can obtain the video's properties, such as cap.get(3) and cap.get(4) which return the width and height of the video, and cap.get(5) which returns the frame rate of the video. cap.get(7) returns the total number of frames in the video.

timeInterval stores the playback time interval, used to make the frame rate of the character animation same as the original video.

pyprind.prog_bar() is a generator that outputs a progress bar in the terminal when iterating.

cap.read() reads the next frame of the video. It returns a tuple with two elements. The first element is a boolean value indicating whether the frame was read correctly, and the second element is a numpy.ndarray containing the frame's data.

cv2.cvtColor() is used to convert the color space of the image. The first parameter is the image object, and the second parameter indicates the conversion type. There are over 150 color space conversions in OpenCV. Here, we use the color to gray conversion cv2.COLOR_BGR2GRAY.

os.get_terminal_size() returns the current terminal's column count (width) and row count (height). We set the fill parameter to True and did not set the wrap parameter, so it defaults to False. In the terminal, if the printed characters exceed the width of one line, the terminal will automatically wrap the display.

Finally, don't forget to release the resource with cap.release().

Next is the play() method. As mentioned earlier, in order to prevent the terminal from being filled with useless characters after playing the character animation, we can use cursor positioning escape codes.

We can do this: after outputting each frame, move the cursor to the beginning of the playback, and the next frame will be output from this position and automatically overwrite the previous content. Repeat this process until the playback is complete, then clear the last frame that was output, so that the terminal will not be filled with character art.

Here's a series of cursor positioning escape codes (some terminals may not support some escape codes), taken from The Linux Command Line:

There is another problem, how to stop the playback halfway? Of course, you can press Ctrl + C, but in this way, the program cannot do any cleanup work and the terminal will be filled with a bunch of useless characters.

We designed it in this way: when the character animation starts playing, it starts a daemon thread that waits for user input. Once it receives input, it stops playing the character animation.

There are two things to note here:

1. Do not use the input() method to receive character input.
2. Cannot use normal threads.

For the first point, if you want to stop the playback by pressing any character, then you should not use input(), otherwise you either have to press Enter to stop playback, or press another character and then press Enter to stop playback. In short, using input() is not comfortable. The best solution is to use something similar to the getchar() method in the C language. However, Python does not provide a similar method, and an alternative solution will be provided in the code later.

For the second point, we need to understand that if any derived thread is still running, the main thread will not exit unless the derived thread is set as a daemon thread. So if we use a normal thread and the user does not stop it halfway, it will continue to run until the animation is finished, and then it will run forever until the user enters any character. If we can manually kill this derived thread when the playback is complete, it is not a problem. However, Python does not provide a method to kill threads. Therefore, we can only set this derived thread as a daemon thread. When the main thread exits, the program will only have a daemon thread running, and the daemon thread will be killed automatically when the program exits.

The complete code for the V2Char class is as follows:

class V2Char(CharFrame):

    charVideo = []
    timeInterval = 0.033

    def __init__(self, path):
        if path.endswith('txt'):
            self.load(path)
        else:
            self.genCharVideo(path)

    def genCharVideo(self, filepath):
        self.charVideo = []
        cap = cv2.VideoCapture(filepath)
        self.timeInterval = round(1/cap.get(5), 3)
        nf = int(cap.get(7))
        print('Generate char video, please wait...')
        for i in pyprind.prog_bar(range(nf)):
            rawFrame = cv2.cvtColor(cap.read()[1], cv2.COLOR_BGR2GRAY)
            frame = self.convert(rawFrame, os.get_terminal_size(), fill=True)
            self.charVideo.append(frame)
        cap.release()

    def export(self, filepath):
        if not self.charVideo:
            return
        with open(filepath,'w') as f:
            for frame in self.charVideo:
                f.write(frame + '\n') ## Add a newline character to separate each frame

    def load(self, filepath):
        self.charVideo = []
        for i in  open(filepath): ## Each line is a frame
            self.charVideo.append(i[:-1])

    def play(self, stream = 1):
        ## Bug:
        ## Cursor positioning escape codes are incompatible with Windows
        if not self.charVideo:
            return
        if stream == 1 and os.isatty(sys.stdout.fileno()): ## If it's a standard output and its file descriptor refers to a terminal
            self.streamOut = sys.stdout.write
            self.streamFlush = sys.stdout.flush
        elif stream == 2 and os.isatty(sys.stderr.fileno()): ## If it's a standard error output and its file descriptor refers to a terminal
            self.streamOut = sys.stderr.write
            self.streamFlush = sys.stderr.flush
        elif hasattr(stream, 'write'): ## If it has a write attribute
            self.streamOut = stream.write
            self.streamFlush = stream.flush

        old_settings = None
        breakflag = None
        fd = sys.stdin.fileno() ## Get the file descriptor of the standard input

        def getChar():
            nonlocal breakflag
            nonlocal old_settings
            old_settings = termios.tcgetattr(fd) ## Save the attributes of the standard input
            tty.setraw(sys.stdin.fileno()) ## Set the standard input to raw mode
            ch = sys.stdin.read(1) ## Read a character
            breakflag = True if ch else False

        ## Create a thread
        getchar = threading.Thread(target=getChar)
        getchar.daemon = True ## Set it as a daemon thread
        getchar.start() ## Start the daemon thread
        rows = len(self.charVideo[0])//os.get_terminal_size()[0] ## Number of rows in the output character art
        for frame in self.charVideo:
            if breakflag is True: ## Exit the loop if input is received
                break
            self.streamOut(frame)
            self.streamFlush()
            time.sleep(self.timeInterval)
            self.streamOut('\033[{}A\r'.format(rows-1)) ## Move up `rows-1` lines to the start
        termios.tcsetattr(fd, termios.TCSADRAIN, old_settings) ## Restore the standard input to its original attributes
        self.streamOut('\033[{}B\033[K'.format(rows-1)) ## Move down `rows-1` lines to the last line and clear it
        for i in range(rows-1): ## Clear all lines of the last frame from the second last line onwards
            self.streamOut('\033[1A')
            self.streamOut('\r\033[K')
        info = 'User interrupt!\n' if breakflag else 'Finished!\n'
        self.streamOut(info)

Testing and Running

Type the following code below the previous code, and this code can be used as a script file to convert video to ASCII art.

if __name__ == '__main__':
    import argparse
    ## Set command line arguments
    parser = argparse.ArgumentParser()
    parser.add_argument('file',
                        help='Video file or charvideo file')
    parser.add_argument('-e', '--export', nargs='?', const='charvideo.txt',
                        help='Export charvideo file')
    ## Get arguments
    args = parser.parse_args()
    v2char = V2Char(args.file)
    if args.export:
        v2char.export(args.export)
    v2char.play()

Enter the following command to convert the file ~/project/BadApple.mp4 to ASCII animation, export it as a file, and play the converted ASCII animation (the encoding process may take a few minutes, please be patient).

cd ~/project
python3 CLIPlayVideo.py BadApple.mp4 -e

Afterwards, to play it again without converting it again, you can directly read the exported ASCII art file. Suppose the file is charvideo.txt:

python3 CLIPlayVideo.py charvideo.txt

Summary

This project uses OpenCV to process images and videos. The operations of OpenCV mentioned here should only be the beginning for everyone. If you want to delve deeper, you should look at the official documentation more. By using a daemon thread, it is believed that everyone can understand what a daemon thread is and how it is different from a regular thread. Finally, we also learned about cursor positioning and escape codes. Although this may not be very useful, it is still an interesting thing that can be used to do many things. You can play around with it more.