Introduction
A useful batch application should run from a saved program with explicit inputs and outputs, not depend on a DataFrame or temporary view left in an earlier session. You will submit a prepared daily-sales program in separate processes and check both its saved results and its execution evidence.
spark-submit is Spark's application launcher. You will give it the execution master, the Python program, and the program's input/output arguments. The script deliberately lets that external master setting control execution. Its reporting logic repeats the typed read, paid filter, decimal amount, and grouping already learned; argument parsing is supplied so you can focus on submission and safe reruns.
This independent lab supplies two valid CSV deliveries and the complete program. You will inspect logs and exit codes, change the input path, and distinguish missing input from an existing output conflict. No remote cluster, packaging, scheduler, Hive, or previous process state is required. Use the Terminal.
Submit a Prepared Program with Explicit Arguments
In this step, you will run the first delivery through a standalone application. Inspect the source and the supplied program:
cd /home/labex/project
cat /home/labex/project/data/orders.csv
cat /home/labex/project/sales_batch.py
The parser already accepts input_path, output_path, and optional --overwrite. You do not need to implement argparse. The first positional path is always input and the second output. The SparkSession builder does not call master, so the launcher's --master setting controls execution.
Read the command from left to right: spark-submit options precede the Python filename; arguments after that filename go to the application. Quote local[2] so the shell treats its brackets literally. > saves ordinary output, and 2>&1 sends error output to the same log:
spark-submit --master 'local[2]' /home/labex/project/sales_batch.py /home/labex/project/data/orders.csv /home/labex/project/output/run-1 > /home/labex/project/output/run-1-job.log 2>&1
Immediately capture the exit status before running any other terminal command. $? is the previous command's status: zero means success, and nonzero means failure. printf writes that value and a line break to a local status file:
printf '%s\n' "$?" > /home/labex/project/output/run-1-exit-status.txt
cat /home/labex/project/output/run-1-exit-status.txt
Expect 0. Read the end of the driver's log:
tail -n 30 /home/labex/project/output/run-1-job.log
Find Master: local[2], the actual application ID, the selected input/output paths, and the report's dates. The ID varies. The report should show October 14 / 2 / 45.00 and October 15 / 2 / 80.00. A zero exit code plus a correct actual Parquet result is stronger evidence than one reassuring log line.
The program creates and stops its own session. It did not reuse the preceding lesson's views or Python variables.
Change the Input in a New Process
In this step, you will prove that the program derives its result from the supplied path. Inspect the second delivery:
cat /home/labex/project/data/new-orders.csv
It has two paid orders on October 16, with amounts 30.00 and 25.00, plus one pending order. Predict one report row: count 2 and revenue 55.00. Keep the first output and submit this new file to a separate destination:
spark-submit --master 'local[2]' /home/labex/project/sales_batch.py /home/labex/project/data/new-orders.csv /home/labex/project/output/run-2 > /home/labex/project/output/run-2-job.log 2>&1
printf '%s\n' "$?" > /home/labex/project/output/run-2-exit-status.txt
cat /home/labex/project/output/run-2-exit-status.txt
tail -n 30 /home/labex/project/output/run-2-job.log
Expect status 0 and the predicted October 16 result. Its actual application ID differs from the first process. The program loaded the new path and created its own session; a leftover view was not required.
Both run-1 and run-2 remain available, making the relationship between each input and result inspectable. Do not change the source file merely to make a remembered answer appear.
Diagnose Missing Input and Existing Output Separately
In this step, you will capture two distinct path failures without deleting data. First submit a nonexistent input. The next command is expected to fail.
spark-submit --master 'local[2]' /home/labex/project/sales_batch.py /home/labex/project/data/missing-orders.csv /home/labex/project/output/unused-missing-input > /home/labex/project/output/missing-input.log 2>&1
Capture its status immediately, before inspecting the log:
printf '%s\n' "$?" > /home/labex/project/output/missing-input-exit-status.txt
cat /home/labex/project/output/missing-input-exit-status.txt
tail -n 25 /home/labex/project/output/missing-input.log
Expect a nonzero status and a path-not-found diagnostic naming missing-orders.csv. Correcting the input path addresses this problem; changing the master or recreating a cluster does not.
The script's default errorifexists mode also protects an existing destination. Try the known input against the already completed run-1 path. The next command is expected to fail.
spark-submit --master 'local[2]' /home/labex/project/sales_batch.py /home/labex/project/data/orders.csv /home/labex/project/output/run-1 > /home/labex/project/output/existing-output.log 2>&1
printf '%s\n' "$?" > /home/labex/project/output/existing-output-exit-status.txt
cat /home/labex/project/output/existing-output-exit-status.txt
tail -n 25 /home/labex/project/output/existing-output.log
Expect a nonzero status and an output-already-exists diagnostic. This differs from a missing input, and the successful earlier report should remain intact.
Make a preservation-first repair using the correct source and a new output directory:
spark-submit --master 'local[2]' /home/labex/project/sales_batch.py /home/labex/project/data/orders.csv /home/labex/project/output/run-3 > /home/labex/project/output/run-3-job.log 2>&1
printf '%s\n' "$?" > /home/labex/project/output/run-3-exit-status.txt
cat /home/labex/project/output/run-3-exit-status.txt
tail -n 25 /home/labex/project/output/run-3-job.log
Expect success and the original two daily totals. A new destination retains both earlier deliveries and the failure evidence, without removing raw files or broad output trees.
Rebuild an Intended Derived Snapshot Explicitly
In this step, you will choose replacement deliberately for a known complete snapshot. The optional --overwrite argument is already implemented: it changes only this program's destination write mode to overwrite. It does not alter the input, other result paths, or Spark's external master.
Use it when your intention is to rebuild the whole specified derived result from the same source. Here run-1 is that known snapshot; the other runs and raw inputs remain separate:
spark-submit --master 'local[2]' /home/labex/project/sales_batch.py /home/labex/project/data/orders.csv /home/labex/project/output/run-1 --overwrite > /home/labex/project/output/overwrite-job.log 2>&1
printf '%s\n' "$?" > /home/labex/project/output/overwrite-exit-status.txt
cat /home/labex/project/output/overwrite-exit-status.txt
tail -n 30 /home/labex/project/output/overwrite-job.log
Expect status 0, Write mode: overwrite, and the same two report rows. This new process replaces the snapshot instead of adding duplicate dates. Its application ID differs from the first run, and its log has its own filename.
You have practiced two valid responses to an existing output: choose a new path when retaining both versions matters, or explicitly rebuild a known derived snapshot when replacement is intended. Input-path errors still require a correct input; overwrite is not a repair for missing data. These submission and rerun habits prepare you for the final reusable sales pipeline challenge.
Summary
You launched a parameterized Spark application in independent processes, captured actual status and logs, and verified results for different inputs. You distinguished missing input from an existing output conflict and chose either a new destination or explicit snapshot replacement. The final challenge now combines these submission habits with cleaning, enrichment, aggregation, and partitioned output.
