Tried and tested mixes for strong pretraining
AI & ML interests
We release large pre-training datasets to accelerate open LLM development. Part of the Hugging Face Science team (hf.co/science)
Recent Activity
Papers
FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language
The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
-
HuggingFaceFW/finepdfs
Viewer β’ Updated β’ 476M β’ 37.4k β’ 817 -
HuggingFaceFW/finepdfs-edu
Viewer β’ Updated β’ 49.5M β’ 7.85k β’ 78 -
HuggingFaceFW/ocr-annotations
Viewer β’ Updated β’ 1.62k β’ 201 β’ 17 -
HuggingFaceFW/finepdfs_lang_classification
Viewer β’ Updated β’ 3.08M β’ 4.56k β’ 4
-
FineWeb: decanting the web for the finest text data at scale
π·1.29kExplore the FineWeb dataset and its creation process
-
HuggingFaceFW/fineweb
Viewer β’ Updated β’ 52.5B β’ 202k β’ 2.66k -
HuggingFaceFW/fineweb-edu
Viewer β’ Updated β’ 3.5B β’ 282k β’ 948 -
HuggingFaceFW/fineweb-edu-score-2
Viewer β’ Updated β’ 13.9B β’ 8.24k β’ 84
1.8B models trained on 350BT to compare different pretraining datasets
-
HuggingFaceFW/ablation-model-fineweb-edu
Text Generation β’ 2B β’ Updated β’ 477 β’ 19 -
HuggingFaceFW/ablation-model-fineweb-v1
Text Generation β’ 2B β’ Updated β’ 37 β’ 14 -
HuggingFaceFW/ablation-model-refinedweb
Text Generation β’ 2B β’ Updated β’ 4 β’ 3 -
HuggingFaceFW/ablation-model-c4
Text Generation β’ 2B β’ Updated β’ 1 β’ 4
-
FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language
Paper β’ 2506.20920 β’ Published β’ 77 -
HuggingFaceFW/fineweb-2
Viewer β’ Updated β’ 4.48B β’ 79.6k β’ 753 -
Scaling FineWeb to 1000+ languages: Step 1: finding signal in 100s of evaluation tasks
π88Evaluate multilingual models using FineTasks
FineWeb-Edu datasets, classifier and ablation model
-
HuggingFaceFW/fineweb-edu
Viewer β’ Updated β’ 3.5B β’ 282k β’ 948 -
HuggingFaceFW/fineweb-edu-score-2
Viewer β’ Updated β’ 13.9B β’ 8.24k β’ 84 -
HuggingFaceFW/fineweb-edu-classifier
Text Classification β’ 0.1B β’ Updated β’ 4.81k β’ β’ 206 -
HuggingFaceFW/ablation-model-fineweb-edu
Text Generation β’ 2B β’ Updated β’ 477 β’ 19
Ablation models trained for our data experiments.
-
HuggingFaceFW/ablation-exp-textext-warc_trafilatura-28BT
Text Generation β’ 2B β’ Updated β’ 1 β’ 1 -
HuggingFaceFW/ablation-exp-textext-wet-28BT
Text Generation β’ 2B β’ Updated β’ 1 -
HuggingFaceFW/ablation-exp-fw-base_filtering-350BT
Text Generation β’ 2B β’ Updated β’ 1 -
HuggingFaceFW/ablation-exp-dedup-global_minhash-350BT
Text Generation β’ 2B β’ Updated β’ 1
Tried and tested mixes for strong pretraining
-
HuggingFaceFW/finepdfs
Viewer β’ Updated β’ 476M β’ 37.4k β’ 817 -
HuggingFaceFW/finepdfs-edu
Viewer β’ Updated β’ 49.5M β’ 7.85k β’ 78 -
HuggingFaceFW/ocr-annotations
Viewer β’ Updated β’ 1.62k β’ 201 β’ 17 -
HuggingFaceFW/finepdfs_lang_classification
Viewer β’ Updated β’ 3.08M β’ 4.56k β’ 4
-
FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language
Paper β’ 2506.20920 β’ Published β’ 77 -
HuggingFaceFW/fineweb-2
Viewer β’ Updated β’ 4.48B β’ 79.6k β’ 753 -
Scaling FineWeb to 1000+ languages: Step 1: finding signal in 100s of evaluation tasks
π88Evaluate multilingual models using FineTasks
-
FineWeb: decanting the web for the finest text data at scale
π·1.29kExplore the FineWeb dataset and its creation process
-
HuggingFaceFW/fineweb
Viewer β’ Updated β’ 52.5B β’ 202k β’ 2.66k -
HuggingFaceFW/fineweb-edu
Viewer β’ Updated β’ 3.5B β’ 282k β’ 948 -
HuggingFaceFW/fineweb-edu-score-2
Viewer β’ Updated β’ 13.9B β’ 8.24k β’ 84
FineWeb-Edu datasets, classifier and ablation model
-
HuggingFaceFW/fineweb-edu
Viewer β’ Updated β’ 3.5B β’ 282k β’ 948 -
HuggingFaceFW/fineweb-edu-score-2
Viewer β’ Updated β’ 13.9B β’ 8.24k β’ 84 -
HuggingFaceFW/fineweb-edu-classifier
Text Classification β’ 0.1B β’ Updated β’ 4.81k β’ β’ 206 -
HuggingFaceFW/ablation-model-fineweb-edu
Text Generation β’ 2B β’ Updated β’ 477 β’ 19
1.8B models trained on 350BT to compare different pretraining datasets
-
HuggingFaceFW/ablation-model-fineweb-edu
Text Generation β’ 2B β’ Updated β’ 477 β’ 19 -
HuggingFaceFW/ablation-model-fineweb-v1
Text Generation β’ 2B β’ Updated β’ 37 β’ 14 -
HuggingFaceFW/ablation-model-refinedweb
Text Generation β’ 2B β’ Updated β’ 4 β’ 3 -
HuggingFaceFW/ablation-model-c4
Text Generation β’ 2B β’ Updated β’ 1 β’ 4
Ablation models trained for our data experiments.
-
HuggingFaceFW/ablation-exp-textext-warc_trafilatura-28BT
Text Generation β’ 2B β’ Updated β’ 1 β’ 1 -
HuggingFaceFW/ablation-exp-textext-wet-28BT
Text Generation β’ 2B β’ Updated β’ 1 -
HuggingFaceFW/ablation-exp-fw-base_filtering-350BT
Text Generation β’ 2B β’ Updated β’ 1 -
HuggingFaceFW/ablation-exp-dedup-global_minhash-350BT
Text Generation β’ 2B β’ Updated β’ 1