Remove Duplicate Lines

Paste a list, such as emails, URLs, keywords or log lines, and every repeated line is removed. The first occurrence stays where it was, so the original order is kept. You can also find the lines that repeat, or list only the ones that appear once.

How to remove duplicate lines

1. Paste your list with one item per line, or open a .txt or .csv file.
2. Choose the options: Ignore case treats “Apple” and “apple” as the same, and Trim spaces ignores leading and trailing spaces.
3. Copy or download the deduplicated list. The summary shows how many lines were removed, and the list below shows which lines repeated and how often.

Options explained

• Remove duplicates keeps one copy of every line. Use Keep last occurrence when later entries are more up to date.
• Only lines that appear once drops every line that has a duplicate, including its first copy. It's handy for finding items that are in one list but not another: paste both lists together.
• Only duplicated lines lists each repeated line once, so you can review what was duplicated.

Doing it in a spreadsheet or terminal

• Excel: Data → Remove Duplicates. Google Sheets: =UNIQUE(A:A). Both are case-insensitive in practice, unlike the default here.
• Terminal, keeping order: awk '!seen[$0]++' file.txt. Sorted: sort -u file.txt. Note that uniq alone only removes adjacent duplicates.

Related tools

• Sort Lines Alphabetically
• Text Cleaner — Remove Extra Spaces & Empty Lines
• Diff Checker
• Word Counter