How to use uniq command to count duplicate lines?

Using the `uniq` Command to Count Duplicate Lines

The uniq command in Linux is a powerful tool that can be used to identify and count duplicate lines in a file or the output of a command. Here's how you can use it to count the number of duplicate lines:

Basic Usage

The basic syntax for using uniq to count duplicate lines is:

uniq -c [file]

This will output the count of each unique line, followed by the line itself. For example, let's say we have a file called data.txt with the following contents:

apple
banana
apple
cherry
banana

Running uniq -c data.txt would give us the following output:

   2 apple
   2 banana
   1 cherry

This tells us that the line "apple" appears 2 times, the line "banana" appears 2 times, and the line "cherry" appears 1 time.

Sorting the Output

To get a more organized output, you can first sort the file before running uniq -c. This will group all the duplicate lines together, making it easier to see the counts. For example:

sort data.txt | uniq -c

This would give us the following output:

   2 apple
   2 banana
   1 cherry

Counting Unique Lines

If you only want to see the unique lines (without the count), you can use the uniq command without the -c option:

uniq [file]

This will output each unique line, without the count.

Mermaid Diagram

Here's a Mermaid diagram that explains the core concept of using uniq to count duplicate lines:

graph TD A[Input File] --> B[Sort File] B --> C[Run uniq -c] C --> D[Output: Unique Lines with Count] C --> E[Output: Unique Lines (without count)]

Real-World Example

Imagine you have a log file that records the activities of your website visitors. You want to know how many unique visitors you had. You can use the uniq command to count the number of unique IP addresses in the log file.

For example, let's say the log file access.log contains the following lines:

192.168.1.100
192.168.1.101
192.168.1.100
192.168.1.102
192.168.1.101

Running sort access.log | uniq -c would give you the following output:

   2 192.168.1.100
   2 192.168.1.101
   1 192.168.1.102

This tells you that you had 3 unique visitors, with IP addresses 192.168.1.100, 192.168.1.101, and 192.168.1.102.

In summary, the uniq command is a versatile tool that can be used to count the number of duplicate lines in a file or the output of a command. By combining it with the sort command, you can get a more organized and informative output, which can be useful in a variety of real-world scenarios.