AI · Training Compute

AI Training Compute

The biggest training run ever done, doubling roughly every six months.

• 4.2× growth per year since 2013
• 360,000,000× from 2013 to GPT-6 Astra
• The record holders, not the famous names
• 1 PF-day = 8.64e19 FLOP

The compute frontier: the largest training run anyone had done at each moment, a running maximum over Epoch AI’s database of notable models. Where several records fell within a few months of each other, the chart shows the one that stood, since at this scale they would land on the same pixel. One petaflop-day is 8.64e19 floating-point operations, roughly what a machine doing a quadrillion operations per second gets through in a day. Between 2013 and 2026 the frontier grew 4.2-fold per year, a doubling every 5.8 months, finishing 360 million times higher than it started. A few curves on this site double faster, none over anything like thirteen and a half years.

Grok 4’s record from July 2025 stood for fourteen months, until Epoch put OpenAI’s GPT-6 Astra, released in September 2026, at about 9.4e26 FLOP, nearly twice as much. Epoch estimates that figure from the size of the training cluster OpenAI and Nvidia have described, not from a compute figure OpenAI published, and gives it a range of roughly 5e26 to 2e27; in October it revised the figure down from 1e27. Most frontier models released in between still have no estimate, so the flat stretch is not evidence that scaling paused. AlexNet, the Transformer and BERT are missing for another reason: none of them was ever the biggest training run. The original Transformer paper used about 0.086 petaflop-days in 2017, when the record already stood at 77, roughly 890 times higher; it is famous for an idea, not for its scale. When AlexNet won ImageNet in 2012 the all-time compute record still belonged to a Google language model from 2007, and it went unbeaten until DistBelief in 2013, where this curve begins. Pick models by fame rather than by size and the chart lies: the Transformer would appear as a step upward while sitting three orders of magnitude below the frontier of its own year.

Source: Epoch AI, Notable AI Models

Download the data: ai_compute.csv · ai_compute.json

All curves · The chart is drawn in your browser and needs JavaScript. Every number it plots is in the table above.