Introduction
Small Linux commands become much more useful when the shell coordinates them. A command reports success or failure through an exit status; sequence operators decide what runs next; a pipe connects one command's standard output to another command's standard input.
You will build these ideas one layer at a time, then use grep, cut, wc, sort, and uniq to transform a small service-event dataset. Later text-processing labs will deepen these filters, so the focus here is how data and decisions move between commands.
Observe Command Exit Status
In this step, you will connect visible command behavior with the numeric status the shell records.
An exit status of 0 means success; a nonzero value means the command could not complete its requested operation. Run a successful file test and immediately print $?, which holds the most recent status:
cd /home/labex/project/pipeline-lab
test -f events.csv
echo $?
The output is 0. Now test a missing file:
test -f missing.csv
echo $?
The status is nonzero. Read $? immediately because every later command replaces it.
Create a stable observation using an if statement. You will study full shell scripting later; here it simply converts the status into text:
if test -f events.csv; then echo "events.csv is ready"; else echo "events.csv is missing"; fi > status-result.txt
cat status-result.txt
Control Which Command Runs Next
In this step, you will compare unconditional and conditional command operators.
A semicolon runs the next command regardless of the previous result:
false; echo "semicolon continues"
&& runs the right command only after success:
test -f events.csv && echo "input found"
test -f missing.csv && echo "you should not see this"
|| runs the right command only after failure:
test -f missing.csv || echo "input missing"
These operators can express a small success/failure decision. Copy the dataset only if it exists, otherwise print an error:
test -f events.csv && cp events.csv working.csv || echo "Copy failed"
Verify that the guarded copy matches its source:
cmp events.csv working.csv && echo "guarded copy matches" > sequence-result.txt
cat sequence-result.txt
For complex logic, prefer a readable if statement; long &&/|| chains can become ambiguous.
Send Output Through a Pipeline
In this step, you will connect commands with | and observe how each stage narrows a stream.
The left command writes standard output into the pipe; the right command reads that data as standard input. Select only ERROR records:
cd /home/labex/project/pipeline-lab
cat events.csv | grep ',ERROR,'
grep can read a file directly, so this shorter form has the same result:
grep ',ERROR,' events.csv
Add wc -l to count matching lines:
grep ',ERROR,' events.csv | wc -l
Save both the filtered stream and count. tee duplicates its standard input: it writes one copy to error-events.csv and passes the other copy onward through standard output. The final wc -l therefore counts exactly the lines that were saved:
grep ',ERROR,' events.csv | tee error-events.csv | wc -l > error-count.txt
cat error-events.csv
cat error-count.txt
Inspect both files after the pipeline. error-events.csv contains the matching records, while error-count.txt contains their count.
Extract Fields with Cut
In this step, you will treat each CSV line as delimiter-separated fields.
cut -d, -f1 uses comma as the delimiter and prints field 1, the service name:
cd /home/labex/project/pipeline-lab
cut -d, -f1 events.csv
Print the service and severity fields together:
cut -d, -f1,2 events.csv
Combine filtering and extraction to list only services that produced warnings:
grep ',WARN,' events.csv | cut -d, -f1
Save this intermediate result:
grep ',WARN,' events.csv | cut -d, -f1 > warning-services.txt
cat warning-services.txt
cut works well for simple records whose delimiter never appears inside a field. More complex formats may need awk or a format-aware tool.
Sort and Count Repeated Values
In this step, you will build a complete frequency pipeline and understand why sorting comes before uniq.
uniq combines only adjacent equal lines. First extract service names, then sort them so identical values are neighbors:
cd /home/labex/project/pipeline-lab
cut -d, -f1 events.csv | sort
Add uniq -c to count each run of equal names:
cut -d, -f1 events.csv | sort | uniq -c
Sort the counts numerically in descending order. sort -k1,1nr uses the first field as a numeric, reverse-order key:
cut -d, -f1 events.csv | sort | uniq -c | sort -k1,1nr
Save the final report:
cut -d, -f1 events.csv | sort | uniq -c | sort -k1,1nr > service-frequency.txt
cat service-frequency.txt
Read this pipeline left to right: extract → sort → count adjacent duplicates → rank counts.
Summary
You used exit status as the shell's success signal, controlled execution with ;, &&, and ||, and built pipelines with clear stages. You then filtered records, extracted fields, counted lines, sorted values, and counted duplicates—exactly the vocabulary needed for the next data-pipeline Challenge.
