Uniq

Yevhenii Tomberg

What does uniq command on Linux?

The uniq command in Linux is a command-line utility that reports or filters out the repeated lines in a file. In simple words, uniq is the tool that helps to detect the adjacent duplicate lines and also deletes the duplicate lines. uniq filters out the adjacent matching lines from the input file(that is required as an argument) and writes the filtered data to the output file.

Syntax of uniq command

uniq has following syntax: uniq [OPTION] [INPUT[OUTPUT]]

Examples of using uniq

We'll walk through several use cases. Some involve only uniq, and others rely on additional commands.

Without any option

We have a file named tomberg, which contains some data. Before using the uniq command with this file, we should sort it. In the example, I have tried the uniq command with the original file, but it only prints the output as it is, much like a cat output. In the next example, we take output from a sort command and pipe it with uniq command. This helps us understand the behavior of the uniq command:

With -c, --count option

The uniq command prints that count as a prefix with the line. The below example tells us that the first line is repeated once, the second line three times, and the third line three times:

With -d, --repeated option

The -d option prints only lines that are repeated. It discards non-duplicate lines. Therefore, line djsldkfjklsdjf has been discarded in the below example:

In the below example, I’ve used the -c option to cross-check whether the -d option is only printing the repeated lines or not:

With -D, --all-repeated option

The -D option prints repeated lines and discards the non-duplicate lines. In the below example, the uniq command prints all duplicate lines only and discards non-duplicate lines:

With -u, --unique option

Opposite of the above option, the -u option prints unique lines i.e., non-duplicate lines. Therefore, in the below example, it prints djsldkfjklsdjf as an output:

With -i, --ignore-case option

Using the -i option, we can ignore the case sensitivity of characters. Below I’ve given an output of the uniq command with and without the -i option to compare:

With -f, --skip-fields=N

Sometimes we need to skip some fields to filter duplicate lines. This is possible using the -f option. In the following example, we’re skipping the first field (first column) to compare the duplicate lines from the second field. I’ve given both examples, with and without the -f option, for a better understanding of the option’s behavior:

With -s, --skip-char=N option

Just like the field, we can skip characters as well by using the -s option. Please keep in mind that the uniq command prints only the first duplicate line and discards other duplicate lines. Therefore 33aa and 55bb have been discarded. Here is the example:

With -w, --check-chars=N option

Just like skipping characters, we can consider characters as well using the -w option, such as in the example:

With --version option

Use the --version option to check the version of the uniq command.

Sources

1. How to use the uniq command to process lists in Linux (RedHat article)
2. uniq Command in LINUX with examples (Geek For Geeks article)