Dedupe — Data Deduplication Reference#

Quick-reference skill for deduplication strategies, algorithms, and data quality patterns.

When to Use#

• Removing duplicate rows from datasets or databases
• Deduplicating files in storage systems
• Implementing fuzzy matching for near-duplicate detection
• Choosing between exact and probabilistic dedup methods
• Building ETL pipelines with deduplication stages

Commands#

intro#

bash

scripts/script.sh intro

Overview of deduplication — types, strategies, and tradeoffs.

exact#

bash

scripts/script.sh exact

Exact deduplication — hash-based, key-based, and sorting approaches.

fuzzy#

bash

scripts/script.sh fuzzy

Fuzzy deduplication — similarity measures, blocking, and record linkage.

files#

bash

scripts/script.sh files

File-level deduplication — fdupes, jdupes, rdfind, and storage dedup.

algorithms#

bash

scripts/script.sh algorithms

Dedup algorithms — bloom filters, HyperLogLog, MinHash, SimHash.

sql#

bash

scripts/script.sh sql

SQL deduplication patterns — ROW_NUMBER, DISTINCT, GROUP BY strategies.

cli#

bash

scripts/script.sh cli

Command-line dedup tools — sort, uniq, awk, and stream processing.

checklist#

bash

scripts/script.sh checklist

Deduplication quality checklist and validation steps.

help#

bash

scripts/script.sh help

version#

bash

scripts/script.sh version

Configuration#

VariableDescriptionDEDUPE_DIRData directory (default: ~/.dedupe/)

---

Powered by BytesAgain | bytesagain.com | hello@bytesagain.com