When people ask what language models are trained on, the honest short answer is: mostly the web. But raw web pages are full of navigation menus, spam, duplicated text and low-quality content. Turning a web crawl into good training data is a major engineering problem — and, as Hugging Face’s FineWeb project showed in 2024, filtering choices can matter as much as model size.

Common Crawl: the raw material

Common Crawl is a nonprofit organization that has crawled the web since 2008 and publishes its archives for free. Each crawl contains billions of web pages, and the full archive amounts to petabytes of data. It is the main public source of web text used to train language models, from early models such as GPT-3 (which used a filtered Common Crawl as its largest data component) to many open models today.

Raw Common Crawl data comes as HTML or pre-extracted text, in hundreds of languages, with no quality guarantees.

The pipeline: from crawl to dataset

Building a training set typically involves these steps:

  1. Text extraction: strip HTML and keep the main content of each page.
  2. Language identification: keep pages in the target languages.
  3. Quality filtering: remove pages that look like spam, boilerplate or machine-generated junk, using rules (for example the ratio of punctuation or repeated lines) and sometimes classifiers.
  4. Deduplication: remove repeated documents and near-duplicates, which are very common on the web.
  5. Safety and privacy filtering: remove adult content, and personal information such as email addresses.

What FineWeb did

The FineWeb team built a dataset of about 15 trillion tokens from 96 Common Crawl snapshots, and — unusually — documented and tested every design decision. For each choice, they trained small language models on the alternatives and compared performance on a set of benchmarks.

Findings included:

  • Text extraction matters: extracting text from raw HTML with a dedicated tool gave better results than using Common Crawl’s pre-extracted text.
  • Deduplication across all snapshots was not always better: deduplicating each crawl separately worked better than aggressive global deduplication, which removed too much useful data.
  • Combining filters inspired by previous datasets with new heuristic filters improved results.

FineWeb-trained models outperformed models trained on other open web datasets at the time in the team’s evaluations.

FineWeb-Edu: letting a model choose the data

The team also released FineWeb-Edu. They used a large language model to rate a sample of pages for educational value, trained a small classifier on those ratings, and applied it to the whole dataset, keeping only pages judged educational. The result — about 1.3 trillion tokens in its main version — produced large gains on knowledge and reasoning benchmarks such as MMLU and ARC compared with the full dataset.

This illustrates a broader trend: model-based filtering, where AI systems help select the data for the next generation of AI.

Why openness matters

Most frontier labs do not disclose the details of their training data. Open datasets such as FineWeb let researchers study how data affects models, reproduce results and build competitive open models. They also make it easier to debate issues such as consent, copyright and representation of different languages and communities.

Open questions

  • Rights and consent: website owners increasingly use robots.txt and terms of service to restrict AI crawling.
  • Language balance: English dominates; multilingual versions extend the approach to other languages.
  • Data exhaustion: high-quality public text is finite, pushing labs toward synthetic data and other sources.

Key takeaways

  • Common Crawl is a free, petabyte-scale archive of the public web and the main source of open training text.
  • Turning it into training data requires extraction, language identification, quality filtering, deduplication and safety filtering.
  • FineWeb built about 15 trillion tokens of filtered web text and tested each processing choice by training small models.
  • FineWeb-Edu used an AI classifier to keep educational pages, sharply improving knowledge and reasoning benchmarks.
  • Open, documented datasets let researchers study and reproduce how data shapes model quality.

Frequently asked questions

Is Common Crawl the same as “the whole internet”?

No. It is a large sample of the public web, shaped by crawling decisions, robots.txt rules and which sites link to each other. Large parts of the internet, such as content behind logins, are not included.

What does deduplication do?

It removes repeated or nearly identical documents. Without it, models see the same text many times, which wastes compute and can increase memorization of specific passages.

Can I use FineWeb myself?

Yes. FineWeb and FineWeb-Edu are available on Hugging Face under the ODC-By license, subject to Common Crawl’s terms of use. Smaller samples are available for experimentation.