VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
DSA for AI/ML · Module 11: Graphs

Lesson 11.15 of 16 in this module Unrolled Pro

Group Near-Duplicate Records with Union-Find

Merge records that refer to the same entity, where 'same' spreads transitively (A shares an email with B, B shares a phone with C). From comparing every pair of records and searching the resulting graph (O(n²)), to indexing records by each key value and uniting records that share one (O(n · keys · α)), with normalisation so trivial formatting differences still match. Then near-duplicate documents with MinHash and locality-sensitive hashing, guarding against clusters that snowball, and new records arriving over time.

about 10 min 3 Python examples 2 quick checks

Problem. You have user records, each with a name, a set of emails and a set of phone numbers. Two records belong to the same person if they share any email or phone, and "same person" is transitive. Group the records and return each group.

text
r0: Ana     ana@x.io                 +1 555 0100
r1: A. Diaz ana.d@y.com              +1-555-0100       (shares a phone with r0, after normalising)
r2: Ana D   ana.d@y.com                                (shares an email with r1)
r3: Bo      bo@z.net                 555 0199
r4: Bo K    BO@Z.NET                                   (same email, different case)

groups: {r0, r1, r2}, {r3, r4}

Records are nodes and a shared key is an edge; the groups are connected components (lesson 11.8). The two real problems are avoiding n² comparisons, and deciding when two keys are "the same" (normalisation, and for free text, near-duplicates).

Why it matters. Deduplication is one of the highest-value steps in data preparation. Duplicate users inflate metrics and leak across train/test splits; duplicate and near-duplicate documents make up a large share of web-scale pretraining corpora, and removing them improves models and reduces memorisation. The tools are exactly these: exact keys with union-find for structured records, MinHash with LSH for text (Change A).


1. Step 1: Clarify

That's the preview. Modules 00–01 are free; this lesson is part of Unrolled Pro.
Staff AI & Pro Tier Lesson

Unlock "Group Near-Duplicate Records with Union-Find"

The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.

Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied
View Plans & Unlock All Lessons

From ₹167/mo (~$2/mo) · Cancel anytime · Instant activation

100% Risk-Free: 14-day no-questions-asked money-back guarantee