Useful Pretrain-Datasets - a Leon-Leee Collection

Leon-Leee 's Collections

awesome-zh-corpus

GPT-4 generated datasets

Useful Pretrain-Datasets

Code, Math, and Reasoning related Instruct datasets

High Quality Instruct Collections

Code Benchmarks

Useful Pretrain-Datasets

updated Mar 12, 2025

pretrain-datasets with (maybe) good quality

Zyphra/Zyda

Viewer • Updated Jun 19, 2024 • 4.35B • 3.59k • 83

Note English only
HuggingFaceFW/fineweb

Viewer • Updated Jul 11, 2025 • 52.5B • 164k • 2.67k

Note English only
HuggingFaceFW/fineweb-edu

Viewer • Updated Jul 11, 2025 • 3.5B • 226k • 961

Note English only
m-a-p/Matrix

Viewer • Updated Feb 25, 2025 • 6.43B • 1.85k • 173

Note zh-en bilingual
bigcode/the-stack-v2

Viewer • Updated Apr 23, 2024 • 5.45B • 7.43k • 474

Note code
bigcode/the-stack-v2-train-full-ids

Viewer • Updated Jun 6, 2024 • 60.5M • 431 • 58

Note code
togethercomputer/RedPajama-Data-V2

Updated Nov 21, 2024 • 2.84k • 396
allenai/c4

Viewer • Updated Jan 9, 2024 • 10.4B • 464k • 527
EleutherAI/pile

Updated May 3, 2023 • 1.92k • 487

Note english
Skywork/SkyPile-150B

Viewer • Updated Dec 7, 2023 • 1.76M • 7.01k • 399

Note chinese
Tele-AI/TeleChat-PTD

Updated Mar 20, 2024 • 296 • 174

Note chinese
wenge-research/yayi2_pretrain_data

Viewer • Updated Dec 29, 2023 • 1.68M • 555 • 59

Note chinese and english
wikimedia/wikipedia

Viewer • Updated Jan 9, 2024 • 61.6M • 73.1k • 1.14k

Note multi-lingual
liwu/MNBVC

Updated Feb 12, 2023 • 4

Note chinese
oscar-corpus/oscar

Updated Sep 4, 2025 • 713 • 204

Note multi-lingual
oscar-corpus/colossal-oscar-1.0

Preview • Updated Oct 26, 2023 • 50 • 36

Note multi-lingual
uonlp/CulturaX

Viewer • Updated Dec 16, 2024 • 7.18B • 38.1k • 590

Note multi-lingual
JosephusCheung/GuanacoDataset

Viewer • Updated Apr 15, 2024 • 1.17M • 201 • 516
allenai/MADLAD-400

Updated Sep 9, 2024 • 213k • 156

Note multi-lingual
Leon-Leee/unofficial-pyedu

Viewer • Updated Mar 12, 2025 • 7.68M • 56 • 3