Introduction
The retailer's order records now arrive in a CSV file instead of being written directly in your Python list. A file stores text such as 2 and 10.00; a calculation needs to know whether those values mean text, a whole-number quantity, or a decimal price. Reading the right characters is only part of reading the data correctly.
You will compare Spark's default string loading, automatic type inference, and an explicit schema. Then you will apply the same schema to a second file with the same field order. The business data is deliberately valid here so you can focus on the contract between columns and values. A later lab handles missing and invalid input explicitly.
The environment supplies PySpark and two independent local CSV fixtures. You should understand the first lab's SparkSession, DataFrame display, schema, and row count, along with basic Python. No pandas, Hadoop, Hive, installation, or earlier VM state is required. Use the Terminal.
Read the Header and Observe String Columns
In this step, you will connect CSV field positions to the table Spark creates. Start by inspecting the actual file:
cd /home/labex/project
cat /home/labex/project/data/orders.csv
The first line names seven columns; the next six lines are orders. Each row represents one order with one product type. The file includes paid, pending, and refunded statuses; this lab reads them all without applying a sales rule.
spark.read creates a DataFrame reader. option("header", True) tells that reader to treat the first CSV line as column names, rather than as an order record. csv(input_path) names the local file. Without type inference or a supplied schema, the CSV reader produces string columns.
Save and run this complete program. The quoted PYTHON here-document preserves the enclosed source text, and > writes the local file. bigdata-python uses the prepared PySpark environment:
cat > /home/labex/project/read_orders.py <<'PYTHON'
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("ReadOrderCSV").master("local[2]").getOrCreate()
spark.sparkContext.setLogLevel("ERROR")
input_path = "/home/labex/project/data/orders.csv"
df = spark.read.option("header", True).csv(input_path)
df.printSchema()
df.orderBy("order_id").show(truncate=False)
print("Rows:", df.count())
spark.stop()
PYTHON
bigdata-python /home/labex/project/read_orders.py > /home/labex/project/output/string-orders.txt
cat /home/labex/project/output/string-orders.txt
Expect six rows and seven string columns, including quantity and unit_price. The displayed 2 may look numeric, but printSchema identifies its actual string type. orderBy("order_id") sorts the displayed rows by identifier for predictable inspection; sorting is separate from assigning types.
header=True prevented the header from becoming a seventh record. It did not tell Spark that prices should be exact decimals or that dates should support date operations.
Compare Inference with an Explicit Data Contract
In this step, you will declare what each CSV field means. inferSchema=True asks Spark to examine values and infer types. That can be convenient, but the result depends on the observed data; it is not a durable agreement about prices or future files. In this fixture, inference identifies unit_price as a double, a floating-point number.
Use an explicit schema for the intended contract. StructType is the ordered collection of fields. Each StructField contains a name, a type, and True to permit missing values. StringType holds identifiers and status text; IntegerType holds quantities; DecimalType(10, 2) holds exact prices with ten total digits and two fractional digits; DateType holds calendar dates. Nullable fields allow missing values but do not make missing business information acceptable automatically.
Replace the program with this version. It prints the inferred type description for comparison, then reads the same file with your explicit schema:
cat > /home/labex/project/read_orders.py <<'PYTHON'
from pyspark.sql.types import StructType, StructField, StringType, IntegerType, DecimalType, DateType
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("ReadOrderCSV").master("local[2]").getOrCreate()
spark.sparkContext.setLogLevel("ERROR")
order_schema = StructType([
StructField("order_id", StringType(), True),
StructField("customer_id", StringType(), True),
StructField("product_id", StringType(), True),
StructField("quantity", IntegerType(), True),
StructField("unit_price", DecimalType(10, 2), True),
StructField("order_status", StringType(), True),
StructField("order_date", DateType(), True),
])
input_path = "/home/labex/project/data/orders.csv"
inferred = spark.read.option("header", True).option("inferSchema", True).csv(input_path)
print("Inferred schema:", inferred.schema.simpleString())
df = (
spark.read
.option("header", True)
.option("dateFormat", "yyyy-MM-dd")
.schema(order_schema)
.csv(input_path)
)
print("Explicit schema:")
df.printSchema()
df.orderBy("order_id").show(truncate=False)
print("Rows:", df.count())
spark.stop()
PYTHON
bigdata-python /home/labex/project/read_orders.py > /home/labex/project/output/typed-orders.txt
cat /home/labex/project/output/typed-orders.txt
Find the explicit schema below the inferred summary. quantity should be integer, unit_price decimal(10,2), and order_date date. All six records should still be present. The explicit dateFormat option describes the input dates as year-month-day, matching values such as 2026-10-14.
The schema's field order matches this CSV's field order. A supplied schema is not a general instruction to rearrange any incoming file by header name. For this known-order delivery, the contract makes the intended columns and types clear. A schema also does not replace a data-quality policy; later you will learn how to handle invalid values safely.
Apply the Same Schema to a New Delivery
In this step, you will test the contract against a second valid file. Inspect it first:
cat /home/labex/project/data/new-orders.csv
It has the same seven fields in the same order, with two new orders. Predict the changes: two business rows instead of six, while quantity, price, and date keep their declared types.
The following complete version keeps the explicit schema and changes the input path. The inference comparison is no longer needed for this normal read:
cat > /home/labex/project/read_orders.py <<'PYTHON'
from pyspark.sql.types import StructType, StructField, StringType, IntegerType, DecimalType, DateType
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("ReadOrderCSV").master("local[2]").getOrCreate()
spark.sparkContext.setLogLevel("ERROR")
order_schema = StructType([
StructField("order_id", StringType(), True),
StructField("customer_id", StringType(), True),
StructField("product_id", StringType(), True),
StructField("quantity", IntegerType(), True),
StructField("unit_price", DecimalType(10, 2), True),
StructField("order_status", StringType(), True),
StructField("order_date", DateType(), True),
])
input_path = "/home/labex/project/data/new-orders.csv"
df = (
spark.read
.option("header", True)
.option("dateFormat", "yyyy-MM-dd")
.schema(order_schema)
.csv(input_path)
)
print("Explicit schema:")
df.printSchema()
df.orderBy("order_id").show(truncate=False)
print("Rows:", df.count())
spark.stop()
PYTHON
bigdata-python /home/labex/project/read_orders.py > /home/labex/project/output/new-typed-orders.txt
cat /home/labex/project/output/new-typed-orders.txt
Expect o7 with quantity 2, price 12.50, paid status, and date 2026-10-16; o8 with quantity 3, price 4.00, pending status, and date 2026-10-17. The count should be 2. Check the same integer, decimal, and date types in printSchema.
The original six-order file remains unchanged. A reusable schema lets a new delivery enter the same processing logic without redefining what a quantity or price means. Next, you will use these typed columns to select orders and calculate each order's amount.
Summary
You read a CSV header correctly, compared string loading and inference, and declared an explicit contract for identifiers, quantities, decimal prices, and dates. The same schema correctly interpreted a second delivery. Next, typed column expressions will let you filter rows and calculate order amounts.
