What Is AI Model Training? Costs, Methods and Failures

• Meta’s 405-billion-parameter Llama 3 trained on about 15.6 trillion tokens using roughly 3.8×10²⁵ FLOP in 2024, still the largest publicly verified training compute, per Meta’s 2024 Llama 3 paper.
• DeepSeek-V3’s 2024 pre-training took 2.788 million H800 GPU-hours, about $5.576 million at an assumed $2 per GPU-hour rental rate (DeepSeek technical report, December 2024).
• Frontier training compute grew 4 to 5× per year through May 2024, and the amortized cost of the largest runs grew 2.4× per year from 2016 (Epoch AI, 2024).
• Human raters preferred a 1.3-billion-parameter model fine-tuned on human feedback over the 175-billion-parameter GPT-3 (OpenAI’s InstructGPT study, 2022).
• About 300 trillion tokens of quality-adjusted public human text exist, and models could exhaust them between 2026 and 2032 (Villalobos and colleagues at Epoch AI, 2024 revision).

What Is Model Training and Why Every AI Model Depends on It

Model training is the process of adjusting a model’s numerical parameters until its outputs match the examples it is shown. The model starts with weights set at random, makes a prediction, gets scored on how wrong it was, and shifts its weights a little in the direction that would have shrunk the error. Repeat that a few million times and you have a trained model.

That raises the obvious question: what is a machine learning model in the first place? Two things, bolted together.

• Architecture: the fixed structure that decides how inputs flow to outputs. A transformer, a decision tree, a logistic regression. A person designs it, and it stays the same during training.
• Weights (also called parameters): the numbers inside that structure. A 405-billion-parameter model has 405 billion of them. Before training they’re noise. After training they hold everything the model “knows.”

The product you ship is the second thing. When a lab releases a model, it is releasing a file of weights plus a description of the architecture that reads them. The architecture alone is an empty container. That’s why no useful AI model exists without a training phase: skip it and the weights are random, so the outputs are random too.

Training is also the only phase in which the weights change. Once training stops, the weights are frozen and the model moves to inference, the word for running the model on new inputs to get answers. Every chat message you send to an assistant is inference. The training happened months earlier, once, at enormous expense, and the result was copied onto the servers that answer you.

One consequence trips up newcomers. A model cannot learn from your conversation with it. Nothing you type during inference touches the weights. If a product appears to remember you, that memory lives in a database or in the prompt, somewhere outside the model. Changing what the model itself knows means going back and training again.

How Model Training Works: Data, Loss Functions, and Gradient Descent

Model training works by running examples through the model, measuring the error with a loss function, and using gradient descent to nudge every weight in the direction that lowers that error. The same loop sits under a hundred-row spreadsheet model and a trillion-token language model. Only the scale changes.

Here is the loop, one pass at a time:

1. Initialization. Weights and biases start at small random values near zero.
2. Forward pass. A batch of examples goes in and predictions come out.
3. Loss. A loss function scores the gap between the predictions and the right answers as a single number. Lower is better.
4. Backpropagation. The model computes the gradient of the loss for every weight, which says how much a tiny change to each one would raise or lower the loss.
5. Update. An optimizer moves each weight a small step against its gradient.

Then it repeats. When the loss stops falling, the run has reached convergence, and that’s where you stop.

Gradient descent is “a mathematical technique that iteratively finds the weights and bias” that give the model its lowest loss. (Google Machine Learning Crash Course)

The optimizer doing the nudging is, in most deep learning today, Adam. Kingma and Ba introduced Adam in 2014 with running estimates of the gradient’s moments and three suggested defaults: a step size of 0.001, β1 of 0.9 and β2 of 0.999. Those three numbers, plus the batch size and the number of iterations, are hyperparameters. Nobody learns them. The practitioner sets them, and the model only learns the weights.

The learning rate is the one that bites. Too large a step and the loss bounces around or blows up. Too small and the run crawls for days and stalls short of a good answer. What I’ve found in practice is that the Adam defaults reliably get you a run that trains, and almost never the best run. The step size a paper used on its data is a starting point for yours, and the hours go into schedules and batch sizes after that.

For a modern large language model, that single loop runs several times in a row with different data and different loss signals. The pipeline looks like this:

• Pre-training: the model learns to predict the next token across a corpus of trillions of tokens. DeepSeek-V3’s 2024 technical report describes 14.8 trillion tokens for its pre-training run.
• Supervised fine-tuning: the pre-trained model is trained again on human-written demonstrations of the behaviour you want, as OpenAI did with GPT-3 in its 2022 InstructGPT work.
• Preference training: the model is trained on rankings of its own outputs. InstructGPT used human rankings with reinforcement learning from human feedback (RLHF). Anthropic’s 2022 Constitutional AI method replaced human harm labels with AI feedback guided by a written set of principles, in a critique-and-revise phase followed by reinforcement learning from AI feedback.
• Reinforcement learning for reasoning: DeepSeek-R1, published in 2025, showed reasoning strategies emerging from reinforcement learning on tasks with verifiable rewards. Its R1-Zero variant did this with no human-written reasoning demonstrations at all.

Each stage reuses the weights the previous one left behind. Gradient descent is the engine in every one of them.

What Model Training Is Used For in Real Products

Model training is used to turn an empty architecture into a product capability: a chat assistant that predicts the next word, a classifier that flags a fraudulent transaction, a forecaster that estimates next week’s demand. Each capability maps to a specific training stage, and the stage is what you are paying for.

The InstructGPT result deserves a second look. In OpenAI’s 2022 InstructGPT study, human raters preferred a 1.3-billion-parameter model fine-tuned on human feedback over the original 175-billion-parameter GPT-3. The model got more than 100 times smaller and people liked its answers more. That’s the clearest proof I know that the later training stages, the cheap ones, decide whether a model is pleasant to use.

Harmlessness followed the same pattern. Anthropic’s Constitutional AI paper from 2022 showed a model could be steered away from harmful outputs using a written set of principles and AI-generated feedback, which cuts the human labelling bill that RLHF normally carries.

Reasoning is the newest outcome. Before DeepSeek-R1 in 2025, step-by-step reasoning was something you coaxed out with prompts. R1 showed it could be trained in directly, by rewarding correct answers on problems with checkable solutions.

And then there’s the unglamorous majority. Most of the models running inside a bank, a hospital or a logistics operation are custom supervised models trained on labelled rows: approve or decline, churn or stay, ship Tuesday or ship Thursday. No tokens, no preferences. Same loop.

What Does Model Training Actually Cost?

Model training costs anywhere from a few GPU-days to tens of millions of dollars in rented compute, and the honest answer depends on which bill you count. The frontier figures get the headlines. The small ones describe most real projects.

Start with the largest verified number. Meta’s 2024 Llama 3 paper reports that its 405-billion-parameter model trained on about 15.6 trillion tokens using roughly 3.8×10²⁵ floating-point operations, almost 50 times the compute of the biggest Llama 2 model. DeepSeek’s December 2024 technical report puts V3’s pre-training at 2.788 million H800 GPU-hours and prices that at about $5.576 million. Read the second figure carefully. It assumes a $2 per GPU-hour rental rate, so it describes what the final run would cost to rent. It leaves out the chips DeepSeek owns, the staff, and every experiment that failed before the run that worked.

That gap is the whole problem with the phrase “training cost”. It can mean at least three things:

• Final-run rental: GPU-hours of the last successful run at a market rate. DeepSeek’s $5.576 million is this.
• Amortized hardware plus energy: the share of purchased chips and electricity the run consumed. Epoch AI measures this.
• Total programme spend: hardware, energy, research staff and every abandoned run. Nobody publishes this.

Mix them and you get nonsense comparisons, which is exactly what happened to the DeepSeek number in 2025.

The trend lines are older than I’d like, but they’re the latest measured. Epoch AI’s June 2024 cost analysis puts the amortized hardware-plus-energy cost of frontier runs on a 2.4× per year climb since 2016 and projects the biggest runs passing $1 billion by 2027. Epoch’s May 2024 estimate for training compute growth is 4 to 5× per year. For models like GPT-4 and Gemini Ultra, Epoch’s 2024 breakdown puts hardware at 47 to 67% of cost, R&D staff at 29 to 49%, and energy at just 2 to 6%.

Energy is small per run and large in aggregate. The IEA’s April 2025 Energy and AI report projects global data-centre electricity demand of about 945 TWh by 2030, roughly double, though that figure lumps training and inference together.

Compute is now a legal quantity too. Under the EU AI Act, the European Commission’s guidance for general-purpose model providers presumes systemic risk for any model trained above 10²⁵ FLOP, a line the Commission may move. Cross it and the training bill comes with obligations attached.

Pre-training vs Fine-tuning vs Training From Scratch: How to Choose

Fine-tune an existing pre-trained model unless you have a reason that survives a budget review; pre-train your own foundation model only if you are a lab, and train a task model from scratch only for classic supervised work on your own labelled data. Those three paths differ by orders of magnitude in data, compute and skill.

Fine-tuning is cheap because of transfer learning: the pre-trained weights already encode grammar, facts and patterns, so fine-tuning moves them a short distance instead of building them from noise. OpenAI’s 2022 InstructGPT work is the clean example. It took GPT-3’s existing weights, fine-tuned on human demonstrations, then on human rankings, and never ran pre-training again.

Within fine-tuning, the choice follows the data you hold:

• Supervised fine-tuning: you have examples of the exact output you want.
• Preference training: you have rankings or judgments, and you want the model to prefer the better answer. RLHF is the original method, with lighter variants such as direct preference optimization now common.
• RL on verifiable rewards: you have tasks with checkable answers, code tests or maths solutions, and you want reasoning. DeepSeek-R1 showed this working in 2025.

If you are weighing a fine-tuning project for an LLM, the data question settles the method before the compute question comes up.

From-scratch pre-training runs on scaling laws, and the literature changed its mind once. Kaplan and colleagues at OpenAI found in 2020 that loss falls as a power law in model size, data and compute across more than seven orders of magnitude, and they advised spending compute on very large models fed modest data. DeepMind’s 2022 Chinchilla paper trained over 400 models and reached the opposite split: parameters and tokens should grow together, and the big models of the day were undertrained. Its 70-billion-parameter Chinchilla matched the 280-billion-parameter Gopher’s compute budget with about four times the data and beat it.

Chinchilla is the standard reference now. Practice has moved past it in one direction. Llama 3’s roughly 15 trillion tokens, applied to models far smaller than a compute-optimal rule would call for, reads to me as deliberate over-training: spend extra compute once so the model you serve forever is smaller. Neither paper says that outright. The numbers do.

Model Training Compared With Inference, RAG, and Prompt Engineering

Model training is the only one of these four that changes a model’s weights; inference, retrieval-augmented generation and prompt engineering all leave the weights frozen and change what goes in. So the first question on any AI project is whether the problem lives in the weights at all.

A short rule of thumb I use with clients:

1. If the answer exists in a document, retrieve it.
2. If the model can do the task but in the wrong shape, prompt it.
3. If the model can’t do the task, or the prompt has grown to pages, train it.

The most expensive mistake I see in enterprise work is fine-tuning to inject facts. A team fine-tunes a model on this quarter’s product catalogue, ships it, and the catalogue changes in six weeks. The weights can’t follow. A retrieval pipeline would have pulled the current page at query time, and the training run was wasted.

The reverse mistake is rarer but real. Teams stack prompt after prompt to force a model into a strict output schema or a specialised vocabulary, then pay for those tokens on every request. A few thousand examples and an afternoon of fine-tuning would have moved that behaviour into the weights for good.

Training and RAG also combine well. Fine-tune for the behaviour you want on every call. Retrieve for the facts that change. Prompt for everything that’s left.

Why Model Training Fails: Overfitting, Data Leakage, and Unstable Loss

Model training fails in five recognisable ways: the model memorises its training set, the test data leaks into training, the learning rate is wrong so the loss diverges or stalls, someone corrupts the data on purpose, or the training data was itself written by a model. Each one leaves a signal in the loss curve or the evaluation scores, and each has a fix.

Overfitting is the one every course teaches. The model gets so good at the examples it has seen that it stops generalising to examples it hasn’t. The defence is a validation set the model never trains on, and the signal is unmistakable once you’ve seen it: training loss still improving, validation loss already climbing.

Leakage is the one I’d bet on in any new team. A customer appears in both the training and the test split because the split was random instead of by customer. A feature was computed using data from after the prediction date. The held-out score comes back at 99%, everyone celebrates, and the model falls apart on its first week of real traffic. Any held-out score that looks too good is leakage until proven otherwise.

Learning rate problems show up in the first few hundred steps. A loss that jumps around or blows up means the step is too big. A loss that barely moves means it’s too small. Either way, the fix costs one restart.

Poisoning is a security problem rather than a statistics problem. NIST’s March 2025 adversarial machine learning taxonomy, AI 100-2 E2025, classes poisoning (corrupting the training data) alongside evasion and privacy attacks and widens its coverage of generative AI. If your training data comes from the open web or from users, assume someone can plant examples in it.

Model collapse is the newest failure. Shumailov and colleagues reported in Nature in July 2024 that models trained recursively on output from earlier models degrade with each generation, and the rarer parts of the original distribution vanish first. A quick synthetic-data shortcut can quietly erase the edge cases you most needed.

Who Should Train Their Own Model and Who Should Not

Most teams should never train a model from scratch, many should fine-tune one, and only a handful of labs should pre-train. The deciding factors are the data you hold, the behaviour you need, and whether anyone on staff has run a training job before.

Solo developers and small teams. Fine-tune a small open model when a prompt can’t get the behaviour you want. Otherwise skip training entirely. A few hundred examples and a rented GPU for an afternoon is the realistic budget, and a maintained training pipeline is a second job.

Enterprises with proprietary data and compliance needs. Fine-tune, and pre-train almost never. Hospitals, banks, insurers and logistics operators hold exactly the labelled rows that make supervised training pay off, and a fine-tuning run sits so far below the EU AI Act’s compute threshold that systemic-risk obligations don’t come into it. The hard part is usually data access and audit trails rather than the run itself.

Frontier labs. Pre-train. The Stanford HAI 2026 AI Index reports that industry produced over 90% of notable frontier models in 2025, and that U.S. private AI investment reached $285.9 billion in 2025. That investment figure covers far more than training runs, but it shows the scale of balance sheet that pre-training now demands.

Signals that justify a training run:

• The model fails the task even with a well-built prompt and retrieved context.
• You hold thousands of labelled examples of the exact output you want.
• Prompt length or per-call cost has become the bottleneck.
• The behaviour must be identical on every call, down to format and vocabulary.

Signals that point to prompting or retrieval instead:

• The missing piece is a fact or a document.
• The underlying data changes weekly.
• You have fewer than a hundred clean examples.
• Nobody will own the pipeline after launch.

Two from the first list and none from the second, and training is probably the right call.

How Model Training Changed in 2026: Synthetic Data, RL, and Smaller Models

As of October 2026, model training has shifted in four ways: human text is running short so labs train on synthetic and multimodal data, reinforcement learning on verifiable rewards has become a standard stage, small models are deliberately over-trained, and the biggest runs are hitting limits on duration and power. A fifth change is quieter. The frontier labs say far less about their runs than they did.

The data ceiling is real. Villalobos and colleagues at Epoch AI estimated in their 2024 revision that about 300 trillion tokens of quality-adjusted public human text exist, with a 90% range of 100 trillion to 1,000 trillion, and projected that models could exhaust it between 2026 and 2032. The 2026 end of that window is now. Epoch’s August 2024 analysis widens the picture to multimodal data, estimating 400 trillion to 20 quadrillion effective tokens by 2030 once images, video and audio count. Both are 2024 projections and remain the latest published estimates. The turn to synthetic data follows from them, with one caveat: the model-collapse result from Shumailov and colleagues in 2024 shows that training on model output without human data in the mix erodes exactly the rare examples that make a model useful.

RL on verifiable rewards went mainstream. DeepSeek-R1’s reinforcement learning on checkable tasks, peer-reviewed in Nature in 2025, moved reasoning from a prompting trick to a training stage. Any task with an automatic checker, code tests or maths answers above all, is now a candidate for this treatment.

Smaller models, trained longer. Labs spend extra compute once so the model they serve every day stays small. Llama 3’s roughly 15 trillion tokens in 2024, applied to models well below a compute-optimal size, set the pattern.

The runs themselves are hitting limits. Epoch’s data on run duration shows frontier training times growing about 1.4× per year since 2020, against a median longest viable run of about 8.6 months. Epoch places the crossover, where the trend meets the ceiling, around May 2027, with a 90% range of August 2025 to September 2029. Its August 2024 scaling study concludes that runs of about 2×10²⁹ FLOP are likely feasible by 2030, with power the first constraint to bind, then chip manufacturing, data and latency.

Disclosure has dried up. Parameter counts, dataset sizes and run durations for the leading closed models are now mostly estimated by outside analysts. The largest publicly verified training-compute figure is still Meta’s 2024 Llama 3 run at 3.8×10²⁵ FLOP. Larger runs have surely happened since. Their numbers are inferred from chip orders and power contracts, and any 2026 compute figure you read should be treated as an estimate until a lab or a peer-reviewed source confirms it.

Frequently Asked Questions About Model Training

The questions people actually type into a search box about model training have short answers, and most of them turn on scale: how big the model is, how much data you have, and whether you’re starting from a pre-trained checkpoint or from random weights.

How long does it take to train an AI model?

Anywhere from minutes to months. A classifier on a few thousand labelled rows trains in minutes on a laptop, and fine-tuning a small open language model on a few thousand examples takes hours on a rented GPU. Pre-training a frontier model is the far end: Epoch AI’s data puts frontier run durations on a 1.4× per year climb since 2020, against a median longest viable run of about 8.6 months. The 2017 Transformer paper’s state-of-the-art translation result took about 3.5 days on eight GPUs, which is a useful middle marker.

How much data do you need to train an AI model?

It depends entirely on where you start. Fine-tuning a pre-trained model works with hundreds to tens of thousands of examples because the weights already hold the general knowledge. Training a task model from scratch on tabular data needs thousands of clean labelled rows. Pre-training a language model from scratch needs trillions of tokens, and DeepMind’s 2022 Chinchilla work showed that data should grow in step with model size rather than lag behind it.

Can I train an AI model on my laptop?

Yes, for small models. A logistic regression, a gradient-boosted tree or a tiny neural network on a few megabytes of data trains fine on a laptop CPU. Fine-tuning a language model of a few billion parameters is possible on a laptop with a capable GPU but slow, and anything larger means renting cloud GPUs. Nobody pre-trains on a laptop.

What is the difference between training and fine-tuning?

Fine-tuning is training that starts from an already-trained model instead of random weights. Both run the same gradient descent loop and both change the weights. The difference is the starting point, which is why fine-tuning needs a fraction of the data and compute: OpenAI’s 2022 InstructGPT run reused GPT-3’s weights and never repeated pre-training.

What is an epoch in model training?

An epoch is one full pass of the training loop over the entire training dataset. A run of ten epochs has shown the model every example ten times. Small supervised models often train for many epochs, while large language models typically see most of their tokens only once because the corpus is so large.

Do you need a GPU to train a model?

Only for neural networks of meaningful size. Classic supervised models (trees, linear models, small networks) train on a CPU in reasonable time. Anything with millions of parameters or more benefits from a GPU, and anything with billions requires several. DeepSeek-V3’s 2024 pre-training used 2.788 million H800 GPU-hours, which gives a sense of the top end.

Is it legal to train on public data?

The EU AI Act doesn’t ban training on public data. It attaches obligations to the result: the European Commission presumes systemic risk for any general-purpose model trained above 10²⁵ FLOP, a threshold the Commission can adjust. Copyright and data-protection rules for the data itself vary by jurisdiction and by dataset, and that question belongs with a lawyer rather than an ML engineer.

Where to Start With Your First Training Run

Start your first model training run small, with a held-out validation set and the default optimizer settings, and only scale up once the held-out numbers justify it. The order matters more than the tooling.

1. Pick a small open model that already does something close to your task.
2. Split your data before you touch it: a validation set the model never sees, split by time or by entity rather than at random.
3. Fine-tune with Adam at its 2014 defaults (step size 0.001, β1 0.9, β2 0.999) and a modest batch size.
4. Watch the loss curve from the first hundred steps. A loss that spikes means the step is too big. A loss that barely moves means it’s too small.
5. Evaluate on the held-out set. A score that looks too good is leakage until you’ve proved otherwise.
6. Scale the data, the model or the compute only if step five earned it.

Most first runs fail at step two or step five, and both failures are cheap to fix on a small model and expensive on a large one. That’s the whole argument for starting small.

If you’d rather have someone who has shipped production fine-tuning runs sit beside your team for the first one, AlphaCorp AI does exactly that.