Introduction
A retailer receives order records and wants to turn them into useful information. Before calculating sales reports, you need a way to represent rows, describe their columns, and ask the processing tool to execute work. You will start with three records small enough to check by eye.
Big data describes data workloads whose size, arrival rate, or processing needs can outgrow a convenient single-machine approach. Apache Spark is a data-processing engine that can divide work across machines in a cluster. PySpark is its Python interface. A DataFrame represents rows with named, typed columns and supports operations such as filtering and aggregation. Spark processes data; a separate file or storage system supplies persistent input and output. This course begins with tiny examples so you can reason about correctness before scale.
You will create a SparkSession, display a DataFrame, inspect its schema and row count, and add one record before running again. The prepared environment uses local[2]: Spark runs on this one machine with two execution threads. This teaches real Spark APIs, while a distributed deployment is a separate topic. You need basic Python variables, lists, and function calls; the Python path can help with those prerequisites. No pandas, Hadoop, Hive, Java, installation, or earlier lab files are required. Use the Terminal.
Run a Complete Small DataFrame Script
In this step, you will run a complete script that turns three Python records into a Spark table. Start in your workspace:
cd /home/labex/project
A SparkSession is the entry point for creating DataFrames and accessing Spark's structured-data APIs. The builder sets an application name, selects local execution, and creates the session. local[2] permits two execution threads on this machine; it does not create two cluster machines. setLogLevel reduces later informational logs so the learning output is easier to find.
In the Python source, from pyspark.sql import SparkSession makes that installed class available to your program. A dot accesses a named member, and parentheses call a method: df.show() asks this DataFrame to display its rows. The builder chains several calls; the surrounding parentheses let that expression span lines. You can read each line as one setting before getOrCreate() obtains the session.
The orders list contains three tuples, parenthesized groups of values. Each tuple supplies one row in the same order as the declared columns. The schema string names each column and its type: text identifiers and products, followed by a whole-number quantity. createDataFrame uses those records and that schema. show asks Spark to display rows, and stop releases the session when finished.
Save the complete script. The shell's > writes the named file; the quoted PYTHON marker begins a here-document that preserves the enclosed Python text literally. Copy through the final PYTHON line:
cat > /home/labex/project/first_dataframe.py <<'PYTHON'
from pyspark.sql import SparkSession
spark = (
SparkSession.builder
.appName("FirstOrderTable")
.master("local[2]")
.getOrCreate()
)
spark.sparkContext.setLogLevel("ERROR")
orders = [
("s1", "notebook", 2),
("s2", "pen", 1),
("s3", "folder", 3),
]
columns = "order_id string, product string, quantity int"
df = spark.createDataFrame(orders, schema=columns)
df.show()
spark.stop()
PYTHON
The prepared bigdata-python command selects the Python environment with PySpark installed. Run the file as a normal Python program, saving its ordinary output locally:
bigdata-python /home/labex/project/first_dataframe.py > /home/labex/project/output/first-run.txt
cat /home/labex/project/output/first-run.txt
Expect a table with columns order_id, product, and quantity, containing s1/notebook/2, s2/pen/1, and s3/folder/3. Startup warnings or progress messages may still appear in the terminal; they are separate from the saved table. No row order is promised without an explicit sort, so identify records by their order IDs.
Your Python program is the driver: it creates the session and asks Spark to perform work. In a cluster, executor processes run tasks over pieces of data. Here the execution happens locally on the same machine. You have a working program without needing to configure a cluster.
Inspect the Schema and Count Rows
In this step, you will distinguish what a column can contain from how many records the table has. A schema lists column names, data types, and nullability. An integer column represents whole numbers suitable for quantity calculations; text has different operations. Nullable means the field permits a missing value, not that every row is missing it.
Add printSchema to inspect that description and count to calculate the number of rows. Here is the complete updated file so there is no hidden interactive state:
cat > /home/labex/project/first_dataframe.py <<'PYTHON'
from pyspark.sql import SparkSession
spark = (
SparkSession.builder
.appName("FirstOrderTable")
.master("local[2]")
.getOrCreate()
)
spark.sparkContext.setLogLevel("ERROR")
orders = [
("s1", "notebook", 2),
("s2", "pen", 1),
("s3", "folder", 3),
]
columns = "order_id string, product string, quantity int"
df = spark.createDataFrame(orders, schema=columns)
df.show()
df.printSchema()
print("Row count:", df.count())
spark.stop()
PYTHON
bigdata-python /home/labex/project/first_dataframe.py > /home/labex/project/output/schema-and-count.txt
cat /home/labex/project/output/schema-and-count.txt
The table should still contain three records. Below it, look for:
root
|-- order_id: string (nullable = true)
|-- product: string (nullable = true)
|-- quantity: integer (nullable = true)
Row count: 3
printSchema describes the DataFrame. show and count are actions that ask Spark to produce a visible result or value. Spark can defer much processing until an action requests an answer; later labs will explain that lazy execution with transformations and plans. A count of 3 means three order records, not the sum of quantities, which would answer a different business question.
Each invocation starts a new Python process and creates its own SparkSession. The file explicitly creates df; it does not rely on an interactive shell having created spark or df earlier.
Predict and Test One Additional Record
In this step, you will change the data while keeping its shape. Predict what adding s4/notebook/1 changes: the number of order rows becomes four, the schema stays the same, and the original three records remain. The new row is one order even though notebook already appears in another order.
Replace the script with this version, which adds just that record:
cat > /home/labex/project/first_dataframe.py <<'PYTHON'
from pyspark.sql import SparkSession
spark = (
SparkSession.builder
.appName("FirstOrderTable")
.master("local[2]")
.getOrCreate()
)
spark.sparkContext.setLogLevel("ERROR")
orders = [
("s1", "notebook", 2),
("s2", "pen", 1),
("s3", "folder", 3),
("s4", "notebook", 1),
]
columns = "order_id string, product string, quantity int"
df = spark.createDataFrame(orders, schema=columns)
df.show()
df.printSchema()
print("Row count:", df.count())
spark.stop()
PYTHON
Run the file again in a fresh process:
bigdata-python /home/labex/project/first_dataframe.py > /home/labex/project/output/expanded-run.txt
cat /home/labex/project/output/expanded-run.txt
Find all four order IDs and Row count: 4. The schema should still describe the same three column names and types. Repeating a product name does not automatically group orders or remove duplicates; those operations must be requested explicitly in later lessons.
The earlier first-run.txt and schema-and-count.txt remain as evidence of your three-row version. The executable file now describes the four-row version, so another independent run can reproduce that result. You have completed a small change-predict-run-check cycle, which will help you judge larger data workflows rather than simply trusting that a program exited successfully.
Summary
You ran a real PySpark script with its own SparkSession, created a typed DataFrame, and checked its displayed records, schema, and row count. Adding one record changed the row count while preserving the schema. Next, you will move from records written inside Python to a CSV file with an explicit schema.
