ReScraper: Unified Scraping and Cleaning of Web Data for Effective LLM Pretraining
https://github.com/cxcscmu/ReScraper
NanoGPT Slowrun: 10x Data Efficiency with Infinite Compute
https://qlabs.sh/10x
Structured Wikipedia
https://huggingface.co/datasets/wikimedia/structured-wikipedia
Always Learning, Always Mixing: Efficient and Simple Data Mixing All The Time
https://arxiv.org/abs/2605.15220
Chroma Context-1 Data Generation
https://github.com/chroma-core/context-1-data-gen
The Synthetic Data Playbook
https://huggingface.co/spaces/HuggingFaceFW/finephrase