Scaling Indic Pretraining introduces a massive 2.8 trillion token dataset and an open data generation pipeline for South Asian languages. Researchers bridge a major data scarcity gap in low-resource linguistic models. The release provides crucial resources for developers building multilingual AI applications and regional language models.
Opening Kapyn…