Introduction
The retailer already has a correct, typed set of paid-order results. Now another application needs to read those results later, select one date, and trust that a repeated export has not silently duplicated orders. Saving data is part of the processing contract, not just the final line of a script.
You will write Parquet with directory partitioning by order_date. The dataset root will contain date-named subdirectories, and Spark will reconstruct the date column when reading that root. These storage directories are different from the computation partitions used to execute work; the next lab examines execution separately.
This lab independently prepares four paid orders with decimal amounts in a local Parquet input. You should know DataFrame reads, filters, and the minimal overwrite-Parquet workflow. No Hive catalog, HDFS, SQL prerequisite, installation, or earlier output is required. Use the Terminal.
Write a Date-Partitioned Dataset and Read It Back
In this step, you will organize the prepared results into date directories. The input dataset under data/prepared-orders contains four orders, with the same typed quantity, decimal price, date, and decimal order_amount used earlier. Parquet is binary; inspect it through Spark, rather than expecting cat to display CSV text.
Start in your workspace:
cd /home/labex/project
partitionBy("order_date") instructs the writer to put rows into directories named by date. mode("overwrite") replaces the existing derived dataset at the specified output path. Keep that output path separate from input_path, which must remain intact.
Save and execute the complete program:
cat > /home/labex/project/write_sales_data.py <<'PYTHON'
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("WriteSalesData").master("local[2]").getOrCreate()
spark.sparkContext.setLogLevel("ERROR")
input_path = "/home/labex/project/data/prepared-orders"
output_path = "/home/labex/project/output/orders-by-date"
prepared = spark.read.parquet(input_path)
prepared.write.mode("overwrite").partitionBy("order_date").parquet(output_path)
print("Main read-back:")
reloaded = spark.read.parquet(output_path)
reloaded.orderBy("order_id").show(truncate=False)
reloaded.printSchema()
print("Main rows:", reloaded.count())
spark.stop()
PYTHON
bigdata-python /home/labex/project/write_sales_data.py > /home/labex/project/output/partitioned-read-back.txt
cat /home/labex/project/output/partitioned-read-back.txt
Expect four reconstructed records: o1 and o2 on October 14, and o4 and o6 on October 15, with amounts 20.00, 25.00, 50.00, and 30.00. Main rows should be 4. Check that quantity is integer, price and amount are decimal, and order_date is date.
A partition column can appear in a different position after reading because Spark reconstructs it from directory names rather than storing it as an ordinary field in each partition file. Column names and values establish its identity; use an explicit select when a downstream consumer requires a specific column order. Display order also needs orderBy.
Inspect Directories and Select One Date
In this step, you will connect the directory layout to the reconstructed date column. List the output recursively; -R includes the date subdirectories and their files:
ls -R /home/labex/project/output/orders-by-date
Expect order_date=2026-10-14 and order_date=2026-10-15 directories. Part filenames, their number, and hidden metadata can vary; do not hardcode one part file as the dataset. The meaningful root is orders-by-date.
Reading that root lets Spark discover the partition values. This file-layout discovery does not require registering a Hive partition or table. The following independent script reads the root, compares its DateType column with a valid ISO date string, and saves the selected day's records separately:
cat > /home/labex/project/inspect_sales_data.py <<'PYTHON'
from pyspark.sql import SparkSession
from pyspark.sql.functions import col
spark = SparkSession.builder.appName("InspectDateDataset").master("local[2]").getOrCreate()
spark.sparkContext.setLogLevel("ERROR")
root = "/home/labex/project/output/orders-by-date"
df = spark.read.parquet(root)
selected = df.filter(col("order_date") == "2026-10-14")
selected.orderBy("order_id").show(truncate=False)
selected.write.mode("overwrite").parquet("/home/labex/project/output/selected-day")
spark.stop()
PYTHON
bigdata-python /home/labex/project/inspect_sales_data.py > /home/labex/project/output/selected-date.txt
cat /home/labex/project/output/selected-date.txt
Expect only o1 and o2, with their October 14 dates and amounts 20.00 and 25.00. The other date still exists in the main dataset. A filtered read changes the returned records; it does not delete unselected directories.
You inspected the storage organization and the actual query result. On this tiny sample, elapsed time does not establish that one format or layout is faster. The purpose is a correct, discoverable dataset that later readers can use.
Compare Repeated Overwrite with Append Safely
In this step, you will preserve the correct main dataset and experiment in a separate directory. Predict the disposable counts: writing the same four rows with overwrite twice leaves four rows each time; appending a full copy afterward produces eight rows. Those extra rows are duplicate exports, not new orders.
Run this complete version. The main path uses overwrite. Only write-mode-demo receives the deliberate append:
cat > /home/labex/project/write_sales_data.py <<'PYTHON'
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("WriteSalesData").master("local[2]").getOrCreate()
spark.sparkContext.setLogLevel("ERROR")
input_path = "/home/labex/project/data/prepared-orders"
output_path = "/home/labex/project/output/orders-by-date"
prepared = spark.read.parquet(input_path)
prepared.write.mode("overwrite").partitionBy("order_date").parquet(output_path)
print("Main read-back:")
reloaded = spark.read.parquet(output_path)
reloaded.orderBy("order_id").show(truncate=False)
reloaded.printSchema()
print("Main rows:", reloaded.count())
practice_path = "/home/labex/project/output/write-mode-demo"
prepared.write.mode("overwrite").partitionBy("order_date").parquet(practice_path)
print("First overwrite rows:", spark.read.parquet(practice_path).count())
prepared.write.mode("overwrite").partitionBy("order_date").parquet(practice_path)
print("Second overwrite rows:", spark.read.parquet(practice_path).count())
prepared.write.mode("append").partitionBy("order_date").parquet(practice_path)
print("After append rows:", spark.read.parquet(practice_path).count())
spark.stop()
PYTHON
bigdata-python /home/labex/project/write_sales_data.py > /home/labex/project/output/write-mode-comparison.txt
cat /home/labex/project/output/write-mode-comparison.txt
Look for Main rows: 4, First overwrite rows: 4, Second overwrite rows: 4, and After append rows: 8. The disposable dataset now intentionally contains two copies of each order, showing why appending a complete snapshot is inappropriate for the main export.
Run the saved program again to check the whole workflow's repeatability:
bigdata-python /home/labex/project/write_sales_data.py > /home/labex/project/output/write-mode-comparison.txt
cat /home/labex/project/output/write-mode-comparison.txt
The same counts should appear. Each complete run rebuilds the main dataset and resets the disposable experiment before its intentional append, so it does not grow to twelve or sixteen rows. Appending is a distinct policy for genuinely new, nonoverlapping data; overwrite is the chosen policy for this complete derived snapshot.
The original prepared-orders dataset remains available. Date directories describe stored values, while Spark's runtime computation partitions describe how a job divides work. You will connect that second meaning of partition to plans and actual execution next.
Summary
You wrote a date-partitioned Parquet dataset, reconstructed its typed records from the root, and selected one date without removing others. Repeated overwrite preserved the main snapshot, while a disposable append demonstrated duplicate copies. Next, execution plans and real events will explain computation partitions, jobs, stages, and tasks.
