Indic data cleaning — report

Cleaning pipeline applied to ai4bharat/sangraha. Deterministic run; all figures measured.

Strategies — 10

The pipeline runs ten cleaning strategies in a fixed order.

#StrategyWhat it does
1Extraction / OCRRepairs PDF-OCR damage: detached vowel-signs, line-break hyphenation.
2NormalizationScript-aware Unicode canonicalization; strips noise, keeps functional joiners; HTML unescape.
3Ghost-tag unificationRewrites mixed literal speaker markers to one special-token format.
4Quality filteringScript-aware heuristics + educational-value gate; per-script thresholds.
5Line boilerplateRemoves template lines recurring across many documents.
6DeduplicationNear-duplicate: char-shingles → MinHash-128 → LSH 8×16, cluster re-verified.
7Language IDUnique-script + Assamese/Bengali + Perso-Arabic ur/snd/kas + Ol Chiki.
8PII removalRegex (ASCII-digit) for emails/phones/IPs + conservative name masking.
9DecontaminationFingerprint eval sets (transliteration-expanded); remove overlaps; canaries.
10ManifestContent-hash IDs, per-shard provenance, Indic-aware token count.

Dataset

Source. huggingface.co/datasets/ai4bharat/sangraha — built by AI4Bharat, released as part of IndicLLMSuite. Config used: verified.

What it is. Sangraha is one of the largest open pre-training corpora for Indian languages — native web and document text across 22 Indian languages plus English. It ships in three parts: verified (human-authored text from vetted sources), unverified (broader web crawl), and synthetic (transliterated / translated). This work uses verified.

What it's for. Pre-training and continual pre-training of Indic and multilingual LLMs, where native-origin text — not translated data — is the scarce, high-value ingredient.

Why chosen. The verified config is precisely the native-origin Indic supply an India-first model treats as its binding constraint. It sits in the 10–100M range, ships with no cross-language deduplication, and carries real folder mislabels — so every cleaning stage has genuine work, unlike a pre-cleaned English dump.

15.1M
docs (est. 86.9M)
33.9 GB
raw parquet
22
languages · ~10 scripts
CC-BY-4.0
licence

Validated on a 6,575-doc sample (real Telugu + defect probes) and executed at 10,000,000-document scale.

What was cleaned, and why

StrategyWhyResult
Extraction/OCROCR splits Indic syllables and hyphenates line ends1,070 fixes · 60 docs
NormalizationVariant/stray characters waste tokenizer vocabulary18,162 chars · 3,894 docs
Ghost-tagLiteral turn markers clash with real special tokens600 markers · 4→1
QualityEnglish rules mis-score good Indic text400 dropped · 6,171 saved
BoilerplateSite templates repeat and add no signal278 lines removed
DeduplicationNear-dups waste compute and cause memorization1,000 removed · 865 clusters
Language IDFolder labels are unreliable154 mislabels flagged
PIIRemove personal data without over-masking Indic1,000 IDs masked
DecontaminationPrevent eval leakage, including cross-script8 docs · 1 canary
ManifestReproducibility + correct Indic token accountingdeterministic · 2.5× fix

Quality, concretely: an English-tuned filter would have wrongly dropped 6,171 good Indic documents; the script-aware filter keeps them.

Other concerns cleaned up

Why this isn't a generic pipeline

A standard English-first cleaner lowercases, strips diacritics, and applies Latin-tuned filters — each of which silently deletes or corrupts Indic text. Every stage here is the script-aware version.

StepGeneric cleanerThis pipeline
Normalizationlowercase, strip accents, plain NFCkeeps matras/joiners; per-script nukta, chillu, split-vowel rules
Quality filterEnglish stop-words & word lengthper-script thresholds — kept 6,171 Indic docs a generic filter drops
Language IDLatin-centric; collapses shared scriptssplits Assamese/Bengali, ur/snd/kas; marks Devanagari tail unresolvable, not guessed
PII digitsUnicode \d masks native digitsASCII-only — Telugu/Devanagari numbers preserved
Decontaminationexact string matchtransliteration-expanded (catches cross-script leaks)
TokensEnglish byte ratioIndic-aware — corrects a 2.5× undercount

Final statistics

6,575 → 5,166
docs survived (78.6%)
82.3%
bytes retained
3.24M
tokens (Indic-aware)
deterministic
byte-identical re-run
StageDocs remaining
Raw6,575
OCR · normalize · ghost-tag (no doc removal)6,575
Quality filter6,175
Boilerplate6,175
Deduplication5,175
Decontamination5,166

At scale — run on 10,000,000 docs (mean 560 B/doc, measured), projected to the full config:

CorpusDocumentsSurvivorsTokens (Indic-aware)
10M run10,000,0007.71M (77.1%)—
materialized15.1M11.65M4.08B
estimated86.9M67.05M23.5B
Deterministic (seed=42, content-hash IDs), offline on AI4Bharat IndicNLP + lingua. Code: pipeline.py, indic_lid2.py, indic_normalize.py, clean_extract.py, scale_run.py.