Cleaning pipeline applied to ai4bharat/sangraha. Deterministic run; all figures measured.
The pipeline runs ten cleaning strategies in a fixed order.
| # | Strategy | What it does |
|---|---|---|
| 1 | Extraction / OCR | Repairs PDF-OCR damage: detached vowel-signs, line-break hyphenation. |
| 2 | Normalization | Script-aware Unicode canonicalization; strips noise, keeps functional joiners; HTML unescape. |
| 3 | Ghost-tag unification | Rewrites mixed literal speaker markers to one special-token format. |
| 4 | Quality filtering | Script-aware heuristics + educational-value gate; per-script thresholds. |
| 5 | Line boilerplate | Removes template lines recurring across many documents. |
| 6 | Deduplication | Near-duplicate: char-shingles → MinHash-128 → LSH 8×16, cluster re-verified. |
| 7 | Language ID | Unique-script + Assamese/Bengali + Perso-Arabic ur/snd/kas + Ol Chiki. |
| 8 | PII removal | Regex (ASCII-digit) for emails/phones/IPs + conservative name masking. |
| 9 | Decontamination | Fingerprint eval sets (transliteration-expanded); remove overlaps; canaries. |
| 10 | Manifest | Content-hash IDs, per-shard provenance, Indic-aware token count. |
Source. huggingface.co/datasets/ai4bharat/sangraha — built by AI4Bharat, released as part of IndicLLMSuite. Config used: verified.
What it is. Sangraha is one of the largest open pre-training corpora for Indian languages — native web and document text across 22 Indian languages plus English. It ships in three parts: verified (human-authored text from vetted sources), unverified (broader web crawl), and synthetic (transliterated / translated). This work uses verified.
What it's for. Pre-training and continual pre-training of Indic and multilingual LLMs, where native-origin text — not translated data — is the scarce, high-value ingredient.
Why chosen. The verified config is precisely the native-origin Indic supply an India-first model treats as its binding constraint. It sits in the 10–100M range, ships with no cross-language deduplication, and carries real folder mislabels — so every cleaning stage has genuine work, unlike a pre-cleaned English dump.
Validated on a 6,575-doc sample (real Telugu + defect probes) and executed at 10,000,000-document scale.
| Strategy | Why | Result |
|---|---|---|
| Extraction/OCR | OCR splits Indic syllables and hyphenates line ends | 1,070 fixes · 60 docs |
| Normalization | Variant/stray characters waste tokenizer vocabulary | 18,162 chars · 3,894 docs |
| Ghost-tag | Literal turn markers clash with real special tokens | 600 markers · 4→1 |
| Quality | English rules mis-score good Indic text | 400 dropped · 6,171 saved |
| Boilerplate | Site templates repeat and add no signal | 278 lines removed |
| Deduplication | Near-dups waste compute and cause memorization | 1,000 removed · 865 clusters |
| Language ID | Folder labels are unreliable | 154 mislabels flagged |
| PII | Remove personal data without over-masking Indic | 1,000 IDs masked |
| Decontamination | Prevent eval leakage, including cross-script | 8 docs · 1 canary |
| Manifest | Reproducibility + correct Indic token accounting | deterministic · 2.5× fix |
Quality, concretely: an English-tuned filter would have wrongly dropped 6,171 good Indic documents; the script-aware filter keeps them.
A standard English-first cleaner lowercases, strips diacritics, and applies Latin-tuned filters — each of which silently deletes or corrupts Indic text. Every stage here is the script-aware version.
| Step | Generic cleaner | This pipeline |
|---|---|---|
| Normalization | lowercase, strip accents, plain NFC | keeps matras/joiners; per-script nukta, chillu, split-vowel rules |
| Quality filter | English stop-words & word length | per-script thresholds — kept 6,171 Indic docs a generic filter drops |
| Language ID | Latin-centric; collapses shared scripts | splits Assamese/Bengali, ur/snd/kas; marks Devanagari tail unresolvable, not guessed |
| PII digits | Unicode \d masks native digits | ASCII-only — Telugu/Devanagari numbers preserved |
| Decontamination | exact string match | transliteration-expanded (catches cross-script leaks) |
| Tokens | English byte ratio | Indic-aware — corrects a 2.5× undercount |
| Stage | Docs remaining |
|---|---|
| Raw | 6,575 |
| OCR · normalize · ghost-tag (no doc removal) | 6,575 |
| Quality filter | 6,175 |
| Boilerplate | 6,175 |
| Deduplication | 5,175 |
| Decontamination | 5,166 |
At scale — run on 10,000,000 docs (mean 560 B/doc, measured), projected to the full config:
| Corpus | Documents | Survivors | Tokens (Indic-aware) |
|---|---|---|---|
| 10M run | 10,000,000 | 7.71M (77.1%) | — |
| materialized | 15.1M | 11.65M | 4.08B |
| estimated | 86.9M | 67.05M | 23.5B |
seed=42, content-hash IDs), offline on AI4Bharat IndicNLP + lingua.
Code: pipeline.py, indic_lid2.py, indic_normalize.py, clean_extract.py, scale_run.py.