Text-Fu · Lesson 14

uniq (Unique)

Learn how to collapse, count, or filter adjacent groups of equal lines with uniq.

The uniq command compares each input line with the preceding line. It can collapse, count, or select groups of adjacent equal lines, but it does not search the entire file for separated duplicates.

Collapsing Adjacent Duplicate Lines

Suppose reading.txt contains grouped values:

book
book
paper
paper
article
article
magazine

Run uniq with no filtering option to print one representative line from each adjacent group:

$ uniq reading.txt
book
paper
article
magazine

The input file remains unchanged because the result goes to stdout.

What does uniq reading.txt do by default?

Counting Adjacent Groups

Use -c to prefix each output group with its number of consecutive input lines:

$ uniq -c reading.txt
      2 book
      2 paper
      2 article
      1 magazine

These are run lengths, not global totals unless all equal lines have first been made adjacent.

What does the count from uniq -c represent?

Selecting Unique or Repeated Groups

Use -u to print only groups containing exactly one line:

$ uniq -u reading.txt
magazine

Use -d to print one representative line from each adjacent group containing more than one line:

$ uniq -d reading.txt
book
paper
article

GNU uniq -D prints every line from repeated groups, whereas lowercase -d prints each repeated group's value once.

Which command prints only adjacent groups that occur exactly once?

Which command prints one line for each adjacent group that appears more than once?

Grouping Separated Duplicates

If equal lines are separated, they form different groups:

book
paper
book
paper
article
magazine
article

Running uniq on this file will produce a surprising result:

$ uniq reading.txt
book
paper
book
paper
article
magazine
article

No lines are collapsed because neighboring values differ. Sort first when changing the order is acceptable and you want equal complete lines grouped together:

$ sort reading.txt | uniq
article
book
magazine
paper

Use a consistent locale and comparison policy across both steps. sort -u reading.txt can also sort and retain one line per equal sort key in a single command.

Equal lines are scattered through reading.txt, and output order may change. Which pipeline produces one sorted copy of each distinct complete line?

uniq reads stdin when no input file is named, which is why it fits naturally after sort. GNU options such as -i can ignore case, while -f, -s, and -w can skip or limit comparison regions; use them only when equality should be defined by part of each line.

To practice grouping, counting, and filtering duplicates, try these hands-on labs:

Linux uniq Command: Duplicate Filtering - Learn how to use the Linux uniq command in combination with sort to identify, filter, and analyze duplicate lines in text files.
Linux sort Command: Text Sorting - Practice using the sort command to organize lines of text files, a crucial step before using uniq effectively.
Word Count and Sorting - Learn the essential Linux text processing tools wc (word count) and sort in this hands-on challenge. Learn to count lines, words, and characters, find frequent patterns, and sort data efficiently for various text analysis tasks.

Lesson complete

You finished uniq (Unique)

You can now analyze adjacent groups of equal lines with uniq.

Collapse each adjacent duplicate group to one line.
Count consecutive occurrences with -c.
Select singleton groups with -u.
Select repeated groups with -d or GNU -D.
Sort first when separated duplicates must be grouped.

Keep your learning progress

Create a free account to save this lesson and continue learning on any device.

Create a free account

Next Lesson

Back to Text-Fu