Common Crawl
commoncrawl.org
Free, open repository of petabytes of web crawl data, updated monthly — the raw material behind many large language models.
Why it is useful
Building a large-scale web dataset from scratch is prohibitively expensive for almost anyone. Common Crawl's free, regularly updated corpus is the foundation many NLP and LLM projects are trained or evaluated on.
More in Research & Data
Other research & data you might not have heard of.
- FRED Economic DataFederal Reserve Bank of St. Louis database of over 800,000 economic time series from hundreds of sources.
- Climate TraceIndependent emissions tracking initiative using satellite data and machine learning to estimate greenhouse gas emissions from individual facilities worldwide.
- AI2 PaperfinderAI research search agent from the Allen Institute for AI, searching over 100 million abstracts with a transparent, step-by-step query process.
- OpenStreetMapA free, editable map of the world built by volunteers, whose open data powers a large number of apps and services.
- Our World in Data ExplorersInteractive data explorers on topics from energy to pandemics, built on curated global datasets.
- Semantic ScholarAI-powered academic search engine with citation graphs, influence metrics, and paper summaries.