✨Text Cleanup
Remove Duplicate Lines
Filter out repeated lines to get a unique list of items.
Explanation
When merging multiple sources, duplicates are common. This pattern keeps only the first occurrence of each unique line.
Examples
Duplicate Emails
Input
test@me.com hello@me.com test@me.com
Output
test@me.com hello@me.com
Code Examples
JavaScript (Set)
const unique = [...new Set(input.split('\n'))].join('\n'); Python
# Keeps order in Python 3.7+
output = "\n".join(dict.fromkeys(input_text.splitlines()))Try it Now
💡 Tips
- Case sensitivity matters: "App" vs "app"
- Trim lines before deduplicating for best results
- Useful for cleaning up mailing lists or logs
⚠️ Common Pitfalls
- Sets in some languages don't preserve original order