Linux uniq Guide

Complete beginner-friendly guide to uniq on Linux, covering Arch Linux, CachyOS, and other distributions including removing duplicates, finding unique lines, and data deduplication.

---

Table of Contents

1. Understanding uniq
2. uniq Basics
3. Duplicate Removal
4. Counting Duplicates
5. Troubleshooting

---

Understanding uniq

What is uniq?

uniq removes duplicate lines.

Uses:

• Remove duplicates: Eliminate duplicate lines
• Count duplicates: Count occurrences
• Data cleaning: Clean up data
• Text processing: Process text files

Note: Requires sorted input for best results.

---

uniq Basics

Remove Duplicates

Basic usage:

# Remove duplicates (requires sorted input)
sort file.txt | uniq

# Or
uniq file.txt

Save Output

Save result:

# Save unique lines
sort file.txt | uniq > unique.txt

# Or
uniq file.txt > unique.txt

---

Duplicate Removal

Show Only Duplicates

Find duplicates:

# Show only duplicates
sort file.txt | uniq -d

# -d: Duplicates only

Show Only Unique

Unique lines:

# Show only unique
sort file.txt | uniq -u

# -u: Unique only

---

Counting Duplicates

Count Occurrences

Count duplicates:

# Count occurrences
sort file.txt | uniq -c

# Shows count before each line

Format Output

Formatted count:

# Count and format
sort file.txt | uniq -c | sort -rn

# Sorted by count

---

Troubleshooting

uniq Not Working

Sort first:

# uniq requires sorted input
sort file.txt | uniq

# Or use sort -u
sort -u file.txt

---

Summary

This guide covered uniq usage, duplicate removal, and data deduplication for Arch Linux, CachyOS, and other distributions.

---

Next Steps

• sort Guide - Text sorting
• sed and awk Guide - Text processing
• uniq Documentation: man uniq

---

This guide covers Arch Linux, CachyOS, and other Linux distributions. For distribution-specific details, refer to your distribution's documentation.