Models

What it is about

When you type a question into a chatbot, a model reads it as a sequence of tokens and calculates, one token at a time, which piece of text should come next. All its knowledge sits in billions of parameters, numbers that were set during training and stay fixed while it answers.

The model layer sits between the data it learns from and the agents and apps built on top of it. It decides much of what AI can do, what it costs and whom you depend on. It is also where the fiercest debates run, about openness and about how far today's methods can go.

What a model is: weights, parameters, tokens

A model is a very large collection of numbers arranged as a neural network. Each artificial neuron holds a number. It takes a weighted sum of the neurons in the layer before, adds a small offset and passes the result on. The weights and offsets together are the parameters; once a model is trained, people usually call them its weights. Learning means adjusting all these numbers until the outputs are good.

Scale separates a textbook example from a chatbot. The small network in Grant Sanderson's 3Blue1Brown lesson, which recognises handwritten digits, has 13,002 parameters. OpenAI's GPT-3 had 175 billion in 2020. On disk the weights are very concrete: a model with 70 billion parameters takes about 140 GB at two bytes per number, which Andrej Karpathy describes as a parameter file plus a small program that runs it.

Language models read and write tokens, pieces of words. OpenAI's rule of thumb for English is about four characters, or three quarters of a word, per token. How text splits depends on the language, and other languages often need more tokens for the same content, so the same message can cost more. The context window is how many tokens a model can take in at once, input and answer together. DeepSeek V4 lists 1 million tokens (vendor data), by OpenAI's rule of thumb about 750,000 English words.

• Architecture: how the numbers are arranged. Almost all of today's language models use the transformer, explained in the next section.
• Foundation model: a foundation model is trained on broad data and then adapted to many tasks. Its strengths and flaws pass on to every product built on it.
• Inference: inference means running a trained model to get an answer. The weights stay fixed; only the input changes.
• Model and system: the EU AI Act treats the general-purpose AI model as a building block and the AI system as the model plus software and interface.

One token at a time: GPT-3 is described by its makers as an autoregressive model, so every new token is predicted from everything before it. A long answer means the model runs once for each token it writes.

The transformer and attention in plain words

Older language models read a sentence word by word and carried a running summary forward. In 2017 eight Google researchers replaced this with the transformer, a design built only on attention. Attention lets every token look at every other token in the input at once and weigh which ones matter for its meaning. In the sentence ‘The bank raised its rates’, attention lets ‘its’ draw on ‘bank’.

The design suits GPUs, because all these comparisons can run in parallel. The original paper set a new best score for English to German translation and trained an English to French model in 3.5 days on eight GPUs. That parallelism is a large part of why the architecture could be scaled up to today's models.

Attention has a price. Each token is compared with every other one, so long inputs get expensive, and models use positions in a long text unevenly. A Stanford study found accuracy was usually highest when the relevant information sat at the start or the end of the input and dropped markedly in the middle, even for models built for long context. The study dates from 2023; whether 2026 models show the effect as strongly has not been verified.

Many new open models change attention to save memory. Sebastian Raschka's comparison of ten releases from early 2026 finds hybrid and linear attention and compressed attention variants as recurring tricks, alongside mixture of experts. He argues that data quality and training recipes probably matter more than architecture alone.

How a model learns: pretraining and post-training

Training starts with random numbers. In pretraining the model reads huge amounts of text and predicts the next token, again and again. A loss function turns each prediction into a single number that says how wrong it was. Gradient descent then nudges every parameter a little in the direction that lowers this number. Repeated over trillions of tokens, this builds general knowledge of language and the world.

A pretrained model continues text well and follows instructions poorly. OpenAI's InstructGPT paper stated it in 2022: making models bigger does not by itself make them better at following what a user intends. Post-training shapes the behaviour in several steps.

1. 01Supervised fine-tuning (SFT). People write model answers to example requests, and the model is fine-tuned on them.
2. 02Human preferences (RLHF). People rank several answers, and the model is trained further with reinforcement learning from human feedback. In InstructGPT, a model with 1.3 billion parameters trained this way was preferred over the 175 billion parameter GPT-3, though it still made simple mistakes.
3. 03Verifiable rewards (RLVR). A program checks the answer: is the maths result right, do the code tests pass. The Allen Institute for AI named this reinforcement learning with verifiable rewards in 2024. Andrej Karpathy calls it the major new training stage of 2025, one that drew compute away from pretraining.
4. 04Reasoning through RL. DeepSeek reported in 2025, peer-reviewed in Nature, that pure reinforcement learning can teach a model to reason without human-written examples of reasoning. Self-checking and switching strategies emerged during training.
5. 05Testing and release. Labs evaluate and red-team the model and publish a model card or system card. These describe intended use and known risks, written by the vendor itself.

Our reading of the sources: maths and code improved fastest in 2025 and 2026 because answers there can be checked automatically, which is exactly where RLVR works.

Scaling laws, reasoning models and thinking time

In 2020 Kaplan and colleagues at OpenAI found that a language model's error falls smoothly and predictably as model size, data and compute grow, with trends spanning more than seven orders of magnitude. These scaling laws turned model building into an investment plan. In 2022 DeepMind corrected the recipe: size and training data should grow together, doubling the tokens with every doubling of size. Its 70 billion parameter Chinchilla, trained with the same compute as the 280 billion parameter Gopher on four times the data, beat Gopher and even larger models.

Spending follows the laws. Training compute for frontier models has grown about fivefold a year since 2020, training cost about 3.5-fold (Epoch AI, February 2026). Many recent open-weight models train on about 20 times more data per parameter than Chinchilla's optimum (Epoch AI, 2025). The leading labs no longer disclose parameter counts or data sizes; the reported counts have hovered near one trillion for three years, which the Stanford AI Index says probably understates real growth.

Since September 2024 a second lever exists: test-time compute. Reasoning models such as OpenAI's o1 write a long internal chain of thought before they answer. On the AIME 2024 maths competition, GPT-4o solved about 12% of the problems; o1 solved 74% with one attempt and 83% with a majority vote over 64 attempts (vendor data). Researchers at UC Berkeley and Google DeepMind showed that on some problems, extra thinking at answer time beats a 14 times larger model at equal compute.

For you, thinking is a setting. Providers sell effort levels or thinking modes; DeepSeek V4, for example, ships with a thinking and a non-thinking mode. More thinking means more output tokens, so each request costs more and takes longer. A reasoning model is a language model trained further with RL, which spends more tokens before answering.

Scaling laws predict a model's error on text. Specific business skills are a different matter: a 2023 Stanford paper showed that many seemingly sudden jumps in ability come from the choice of measure, and vanish with smoother metrics.

Which kinds of models exist

Chat models are one family among several. Google DeepMind's model page alone lists models for text, images, video, music, speech, embeddings, robotics and world models (October 2026). Six types matter most for builders.

• Mixture of experts: a mixture-of-experts model holds many sub-networks, and a router activates only a few for each token. DeepSeek V4-Pro has 1.6 trillion parameters, of which 49 billion work per token. Per token it computes like a much smaller model, yet all its weights still need memory.
• Multimodal: multimodal models link text with images, audio and video. A key ancestor is OpenAI's CLIP from 2021, trained to match 400 million internet images with their captions.
• Diffusion: most image generators are diffusion models. They learn to remove noise step by step and create a picture by turning pure noise into an image. Researchers at LMU Munich and Heidelberg made this affordable in 2021 by working in a compressed space and steering it with text.
• Embeddings: embedding models turn text into lists of numbers in which closeness means similar meaning. They power semantic search and RAG. OpenAI's small embedding model uses 1,536 numbers per text (vendor data). You choose it separately from the chat model.
• Small models: small language models with a few billion parameters run on a laptop or phone. Microsoft reported its Phi-3-mini (3.8 billion parameters, 2024) as comparable to GPT-3.5 on benchmarks, largely thanks to filtered and synthetic training data (vendor claim).
• World models: a world model predicts how an environment changes when someone acts in it. Google DeepMind lists Genie 3 in this category; Fei-Fei Li's World Labs and Yann LeCun's AMI Labs build their companies on the idea.

How models get smaller and cheaper

Two techniques move capability from large models into small ones. In distillation a small model learns to imitate a large one. Geoffrey Hinton and colleagues at Google described the method in 2015, and DeepSeek reports that the reasoning patterns of its large model can be distilled into smaller models. Quantization stores each weight with fewer bits. The GPTQ method from IST Austria and ETH Zurich compressed models with 175 billion parameters to 3 or 4 bits with negligible loss of accuracy, so that one GPU could run them.

A simple calculation shows the effect: the 140 GB file of a 70 billion parameter model shrinks to about a quarter at 4 bits. That is why one model family comes in cheap, fast variants, and why open-weight models can run on your own hardware as local AI. The quality loss is usually small and never zero, so test on your own task.

Prices fall accordingly. Reaching a fixed level of performance became 9 to 900 times cheaper per year between 2022 and early 2025, depending on the task, according to Epoch AI; the fastest falls were the most recent, so they may not last. Within one provider, prices today span a factor of 100: OpenAI charges $0.50 to $50 per million output tokens, depending on model size (October 2026, vendor).

Open and closed models, the US and China

‘Open’ covers three separate questions. Open weights means the trained parameters can be downloaded. Open source AI under the Open Source Initiative's definition of October 2024 also requires detailed information on the training data and the complete code to train and run the model. The licence then decides what you may do: Meta's Llama 3 licence, for example, requires companies with more than 700 million monthly users to ask Meta for a licence and binds every user to an acceptable-use policy.

Very few models meet the full definition. Of 102 notable models released in 2025, 47 came with API access only, and 4 shipped open-source training code (Stanford AI Index 2026). Disclosure is shrinking overall: the average score in Stanford's Foundation Model Transparency Index fell from 58 to 40 out of 100 between 2024 and 2025. IBM scored highest with 95, xAI and Midjourney lowest with 14 each, and training data is the least disclosed area. This lack of transparency makes independent checks harder.

How far behind are open models? Epoch AI measures the gap in months: since January 2026 the best open-weight models trailed the closed frontier by four months on average, 8 points on its capability index (May 2026). Epoch counts every downloadable model as open, from unrestricted to non-commercial licences. On cyber tasks the lag is longer, 6 to 7 months (Epoch, October 2026); the UK AI Security Institute measures 4 to 7 months. On the Chatbot Arena, where people vote between two anonymous answers, the best closed model reached 1,503 points and the best open one 1,454 in 2026 (AI Index). Months and points are different units and cannot be converted.

The US and China comparison also depends on definitions. The AI Index counts a notable model for a country when at least one author of its paper works at an institution there, so one model can count twice. By this rule, 59 notable models of 2025 had US authors, 35 Chinese and 8 South Korean; a widely shared summary gives lower numbers (50, 30 and 5), and we use the report itself. On the Arena the best US model led the best Chinese one by 1,503 to 1,464 points, the 39 points often quoted as 2.7%. Most of the strongest open-weight models now come from Chinese labs such as DeepSeek, Alibaba (Qwen), Moonshot (Kimi) and Z.ai (GLM), as the listings of Epoch AI, Artificial Analysis and Sebastian Raschka show; no source gives a share.

Where models fail: hallucinations and benchmarks

A model predicts tokens from its weights and looks nothing up, so it can produce fluent, wrong statements, called hallucinations. OpenAI researchers explain them as ordinary classification errors: when a model cannot tell some true statements from false ones, statistics alone produce falsehoods. They persist because most tests score ‘I don't know’ as zero, which rewards guessing. In OpenAI's SimpleQA test, one model that abstained on 52% of questions made errors on 26%, while an older model that almost never abstained erred on 75% (vendor data).

How often models hallucinate depends on the test. The AI Index 2026 reports 1.8% to 5.4% for the top 15 models when they summarise a given text, and 22% to 94% for 26 models on a knowledge test. Each model also has a knowledge cutoff; vendors describe knowledge as most extensive and reliable up to that date, so coverage thins before it. Grounding answers in your own documents, allowing the model to say it does not know and checking its output all lower the risk.

Benchmarks are standard tests for comparing models, and they are fragile. Test items leak into training data, called contamination: a Yale team found that ChatGPT and GPT-4 could guess a hidden wrong answer option in the MMLU test in 52% and 57% of cases. Leaderboards can be gamed: the Leaderboard Illusion study (Cohere Labs with universities, 2025) found that some providers test many private variants on the Chatbot Arena and publish only the best, Meta reportedly 27 before Llama 4. Google and OpenAI received an estimated 19.2% and 20.4% of all Arena data, 83 open-weight models together 29.7%.

Tests also wear out fast. On OSWorld, which measures computer use, the best score rose from about 5% to 66.3% within a year, against 72.35% for humans (AI Index 2026). Capability is jagged: the International AI Safety Report 2026 notes that leading systems can master hard tasks and fail at simpler ones, and that models increasingly tell test situations from real use. Language matters too. GPT-5 scored 99.8% on a Slovenian commonsense test in the standard language and 88.6% in dialect, and a 2024 benchmark for African languages found the best open model at about 63% of the best closed one.

Launch tables are vendor claims, and each lab picks the benchmarks it leads. Use independent, dated indices to build a shortlist, then decide with evals on your own tasks.

The big debate: scale further or build differently

Researchers disagree about where the next gains will come from. Leading figures hold both positions, several of them with commercial interests, and the question is open. Here are the arguments side by side.

• Scale and RL work: scaling laws and the reasoning recipe of 2025 carried most recent progress. Karpathy describes RL on checkable tasks as the big new training stage of 2025, the DeepSeek team showed that RL alone can teach reasoning, and the International AI Safety Report 2026 attributes large gains in maths, coding and science partly to more compute at answer time. Rich Sutton's essay The Bitter Lesson (2019) is often cited for this camp: over 70 years, general methods that scale with computation have won.
• Text-only models are not enough: Yann LeCun dismisses the bet that scaling language models leads to human-level intelligence and works on world models that predict the next state of the world from an action (Brown University lecture, April 2026, as reported). Fei-Fei Li argues that language models are strong with words and weak at judging space, distance and size. Sutton himself says language models imitate what people say and lack a goal, so his lesson may favour systems that learn from experience.
• Interests: LeCun is executive chairman of AMI Labs in Paris, which raised a reported $1.03 billion seed round in March 2026, and Li co-founded World Labs; both build the approach they argue for. Sara Hooker, a critic of compute thresholds and leaderboards, now builds her own company, Adaption Labs. OpenAI, Google, Anthropic and DeepSeek argue for the path they sell, too.
• Common ground: Karpathy, the safety report and Melanie Mitchell, from different camps, all describe uneven, jagged capability and a gap between test scores and real use.

How to choose a model

There is no neutral standard for choosing a model. The checklist below is Aulory's synthesis of the sources on this page. Our reading behind it: capability moves quickly into smaller, cheaper models, so a lasting advantage sits in your data, your tests and your workflow.

1. 01Test on your tasks. Collect 20 to 100 real examples from your business and run two or three candidates on them. Use public rankings only to build the shortlist; your evals decide.
2. 02Compare cost per task. Measure cost per finished task at the effort level you need, retries included. Reasoning models write many more tokens, so price per token misleads. Independent benchmarkers such as Artificial Analysis report the cost of a full test run.
3. 03Check speed. For routing, extraction and classification, the fastest and cheapest tier is often enough; mind the latency your users accept. Model routing sends each request to the smallest model that can handle it.
4. 04Plan the context. Estimate how much material each request needs, count tokens in every language you serve, and put key material at the start or the end of the context window.
5. 05Decide on control. A closed API is quick to start with. Open weights on your own or regional infrastructure, through self-hosting, give you control over your data. Read the licence either way.
6. 06Add fresh knowledge. Compare the knowledge cutoff with your need for current information and add search or RAG where needed.
7. 07Read the cards. The model or system card lists known limits and risk ratings. It is the vendor's own assessment, so treat it as a starting point.
8. 08Re-test every release. Three leading labs shipped new frontier models between early September and mid-October 2026 alone. Keep your test set, switch when the numbers say so, and avoid features only one model can deliver, to limit vendor lock-in.

What the layer is made of

Key figures, with source and date

• about 5x a yeargrowth of training compute for frontier models since 2020. Training cost grows about 3.5x a year.Epoch AI · 5 Feb 2026 · neutral
• 9x to 900xcheaper per year to reach a fixed level of performance, depending on the task (data to early 2025). The fastest falls were the most recent and may not last.Epoch AI · 12 Mar 2025 · neutral
• 59 / 35 / 8notable models of 2025 with authors in the US, China and South Korea. A model counts for every country where one of its authors works.Stanford AI Index 2026 (Epoch AI data) · Apr 2026 · academic
• 4 monthsaverage lag of the best open-weight models behind the closed frontier since January 2026, 8 points on Epoch's capability index.Epoch AI · 29 May 2026 · neutral
• 4 of 102notable models of 2025 shipped open-source training code. 47 came with API access only.Stanford AI Index 2026 · Apr 2026 · academic
• from 58 to 40fell the average transparency score of major model developers out of 100, from 2024 to 2025. Training data is disclosed least.Foundation Model Transparency Index, Stanford · 11 Dec 2025 · academic
• 1,503 vs 1,464Arena points of the best US and the best Chinese model in 2026, a lead of 39 points or about 2.7%.Stanford AI Index 2026 · Apr 2026 · academic
• 22% to 94%hallucination rates of 26 models on a knowledge test, against 1.8% to 5.4% for the top 15 models when summarising a given text. The number depends on the test.Stanford AI Index 2026 · Apr 2026 · academic
• 1,112generative AI services were filed with China's internet regulator by 31 August 2026. Filings count services, many built on the same base models.remio, citing a CAC notice · 15 Sep 2026 · secondary source

Regions compared

Frontier models are built mainly in the US and China. Europe, South Korea, the Gulf, India and Latin America build mostly smaller models for their own languages, from 270 million to about 105 billion parameters in the examples we found. Rules for models differ just as much: the EU regulates models directly, China requires filings for public services, the US relies on its states, and the UK tests models without a dedicated law.

Figures follow different definitions and are not directly comparable. Where our research found nothing reliable, the table says so.

Sources: Stanford AI Index 2026, chapters 1, 2 and 8 (April 2026), European Commission (GPAI guidelines, Code of Practice, status 25 Oct 2026), EU AI Act Art. 51, Governor of California (29 Sep 2025), CAC (13 Jul 2023), remio citing CAC (15 Sep 2026), UK AI Security Institute, Mistral, LG AI Research, TII Falcon blog, OpenSourceForU (Mar 2026), Radio Universidad de Chile (10 Feb 2026), IrokoBench (2024). Model families in the US row from the labs' own pages, status 25 Oct 2026. Several figures come from secondary reports of the original.

Examples

• Closed families: GPT (OpenAI), Claude (Anthropic), Gemini (Google).
• Open families: Llama (Meta), Mistral, DeepSeek, Qwen (Alibaba), Gemma (Google).
• In April 2026, DeepSeek released a model with 1.6 trillion parameters, of which only 49 billion work per token (mixture of experts).
• A model with 70 billion parameters is about 140 GB in size (Andrej Karpathy).

Why it matters

The model layer sets three things you feel directly: what your product can do, what each task costs and whom you depend on. Prices for the same performance fall fast, open models trail the closed frontier by months, and new releases arrive every few weeks. A choice that was right in spring can be expensive by autumn.

It also sets your risks. Hallucinations follow from how models are trained and graded, benchmarks can mislead, and disclosure by the big labs is falling. Rules differ by region: if you build on a model in the EU, the US or China, its provider faces different duties, and so may you.

Common misconceptions

• More parameters mean a better model.

  DeepMind's 70 billion parameter Chinchilla beat models with 280 to 530 billion, and a tuned 1.3 billion parameter model was preferred over the 175 billion parameter GPT-3 (DeepMind 2022; OpenAI 2022).

• Open weights are open source.

  The OSI definition also requires information on training data and the full code. Only 4 of 102 notable models of 2025 shipped open-source training code (OSI 2024; AI Index 2026).

• Open weights mean free use for anything.

  Licences can set conditions. Llama 3 requires companies with more than 700 million monthly users to ask Meta for a licence and binds all users to an acceptable-use policy.

• Hallucinations are a bug the next version will fix.

  They follow from statistics in training and from tests that reward guessing (OpenAI, 2025). Rates vary widely by model and test.

• The benchmark leader is the best model for my business.

  Test items leak into training data, providers test private variants and publish the best, and capability is jagged (Yale 2023; Leaderboard Illusion 2025; International AI Safety Report 2026).

• A context window of 1 million tokens means the model reads everything equally well.

  Information in the middle of a long input was used markedly less reliably than at the start or end (Stanford, 2023).

• Price per token is what a task costs.

  Reasoning models write far more tokens per task, so independent benchmarkers measure the cost of a full run (Artificial Analysis).

Perspectives from leading experts

Ten perspectives from practice, research, policy and critical scholarship, from optimists about scaling to its sharpest critics. Each summary paraphrases a published work, linked below it. Vendors are marked as such, including researchers who now lead companies.

• Andrej Karpathy

  AI researcher and educator, author of widely used introductions to language models · independent analysis

  2025 added a new training stage, reinforcement learning on automatically checkable tasks such as maths and code, which delivered much capability per dollar. He pictures language models as summoned ghosts more than growing animals, with jagged skills: brilliant near checkable domains and easy to trick elsewhere. Because benchmarks are checkable too, labs can train towards them.

  2025 LLM Year in Review · 19 Dec 2025

• DeepSeek-AI team

  Lab behind the open-weight DeepSeek models, China · vendor

  Reasoning can be developed through pure reinforcement learning, without human-written examples of reasoning. Self-reflection, verification and changes of strategy emerged during training. The reasoning patterns of the large model can be distilled into smaller models.

  DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (arXiv; Nature 2025) · 22 Jan 2025

• Yoshua Bengio and the International AI Safety Report

  Turing Award laureate, chair of a report by over 100 experts backed by more than 30 countries and organisations · policy research

  Recent gains came partly from spending more compute at answer time, with large jumps in maths, coding and science, while leading systems can still fail at simpler tasks. The report names an evaluation gap between test results and real-world usefulness and risk. Policy faces an evidence dilemma: capabilities move fast, evidence on risks arrives slowly, and acting too early or too late both carry costs.

  International AI Safety Report 2026, executive summary · 3 Feb 2026

• Fei-Fei Li

  Computer scientist and co-founder of World Labs, which builds world models · vendor and academic

  Language models are strong with words and ungrounded in the physical world; multimodal models still judge distance, orientation and size close to chance. She calls for world models that generate consistent worlds, take many kinds of input and predict the next state from actions. Her company World Labs builds such a model.

  From Words to Worlds: Spatial Intelligence is AI's Next Frontier · 10 Nov 2025

• Yann LeCun

  NYU professor and executive chairman of AMI Labs, Paris, France and USA · vendor and academic

  Current AI is helpless in the physical world, and language-only systems merely seem smart; he rejects the bet that scaling language models leads to human-level intelligence. He proposes world models that predict the next state of the world given an action, trained on images, video, audio and scientific data. Human-level AI is, in his view, much harder than most expect.

  Lecture at Brown University, as reported by Brown University · 1 Apr 2026

• Richard Sutton

  Pioneer of reinforcement learning, author of The Bitter Lesson · academic

  Language models imitate what people would say and do not predict what will happen in the world, so they lack ground truth and a goal, which he sees as the essence of intelligence. He leaves open whether they fit his Bitter Lesson, since they scale with compute and still rest heavily on human-written data. He expects systems that learn from experience to take over.

  Interview on the Dwarkesh Podcast · 26 Sep 2025

• Emily M. Bender, Timnit Gebru and co-authors

  Authors of the Stochastic Parrots paper; Bender is a linguist · critical research

  Ever larger language models carry costs: energy and emissions borne by people least likely to benefit, high entry costs that concentrate research and neglect many languages, and scraped data that encodes dominant views. A language model stitches word sequences together by probability without grounding in meaning, and people mistake its fluency for an accountable speaker.

  On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? (ACM FAccT) · Mar 2021

• Melanie Mitchell

  Computer scientist and author of the newsletter AI: A Guide for Thinking Humans · academic

  Earlier AI systems often learned surface shortcuts, and experts are split on whether language models build real world models or rely on approximate retrieval from their training data. The term world model has no agreed definition. On a scale from look-up table to simulator, she reads the evidence as map-like, with little sign of detailed causal simulation.

  LLMs and World Models, Part 1 · 13 Feb 2025

• Sara Hooker

  Co-founder of Adaption Labs, formerly head of Cohere Labs · vendor and academic

  Thresholds of training compute, such as the EU's 10^25 and the then US line of 10^26 FLOP, are a short-sighted proxy for risk. Smaller models increasingly beat larger ones through better data and methods, and techniques at answer time change risk without changing training compute. She proposes dynamic thresholds and an index built from several risk measures.

  On the Limitations of Compute Thresholds as a Governance Strategy · 8 Jul 2024

• David Ifeoluwa Adelani and co-authors

  Researchers behind IrokoBench, a benchmark for African languages · academic

  Language models show a significant performance gap between high-resource languages such as English and French and low-resource African languages. The best open model tested reached only about 63% of the best closed model. Translating the tests into English first narrowed the gap for large English-centric models.

  IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models · 5 Jun 2024

Terms for this layer

• ModelThe trained file of numbers that turns input into output. On its own it can't browse, remember or act.
• ParametersThe billions of numbers inside a model, set during training. Everything the model knows is stored in them.
• Neural NetworkA layered structure of simple calculations, loosely inspired by the brain, that learns patterns from data.
• TokenThe small chunk of text a model reads and writes. Price, speed and memory limits are all counted in tokens.
• Context WindowThe maximum amount of text a model can take in at once. When it is full, something has to go.
• TransformerThe neural network design behind today's language models, built to weigh how every part of a text relates to every other part.
• AttentionThe mechanism inside a transformer that lets every word piece weigh how much every other piece matters for its meaning.
• Foundation ModelA large, general-purpose model trained on broad data that many different products and tasks are built on top of.
• InferenceRunning a finished model to get an answer. What you pay for every time you send a message.
• General-Purpose AI ModelThe EU AI Act's term for broadly capable models, like large language models, that can be used for many different tasks.
• AI SystemThe EU AI Act's core legal term: a machine-based system that infers from its input how to produce outputs such as content, recommendations or decisions.
• GPUA graphics processor: a chip that runs thousands of simple calculations at once, which made it the workhorse of AI.
• TrainingThe one-time, very expensive process that sets a model's parameters by showing it vast amounts of data.
• Pre-TrainingThe first and most expensive phase of training, in which a model learns general patterns from huge, broad data.
• Gradient DescentThe method by which a model learns: nudge every parameter a little in the direction that makes its errors smaller, and repeat.
• Post-TrainingEverything that happens after pre-training to turn a raw base model into a helpful, safer assistant.
• Fine-TuningTraining an existing model further on your own examples to change its behaviour. Rarely the first step.
• RLHFReinforcement learning from human feedback: people rank a model's answers, and the model is trained to give the kind of answers people prefer.
• RLVRReinforcement learning with verifiable rewards: a program checks whether the model's answer is correct, and the model is rewarded only for answers that pass.
• Reinforcement LearningA way of training in which a system learns by trial and error which actions earn the most reward over time.
• Red TeamingDeliberately attacking your own AI system, the way an adversary would, to find weaknesses before others do.
• Model CardA short document that comes with an AI model and states what it is for, how it was tested and where it falls short.
• ComputeThe computing power, and the chips and software behind it, that AI needs to be trained and to run.
• Scaling LawsThe observation that a model's error falls predictably as model size, data and computing power grow.
• Test-Time ComputeComputing power spent while answering, so a model can think longer or try several solutions, instead of only during training.
• Reasoning ModelA model that generates intermediate working steps before its final answer, trading time and tokens for better results on hard problems.
• Chain of ThoughtGetting a model to work through a problem step by step before answering, which tends to improve results on multi-step tasks.
• Mixture of ExpertsA model built from many specialised sub-networks, of which only a few are switched on for each input.
• MultimodalA model that works with more than text: images, audio, video, documents.
• Diffusion ModelA generative model that creates images and other media by starting from noise and removing it step by step.
• EmbeddingsText turned into lists of numbers that capture meaning. The basis of semantic search and RAG.
• Retrieval-Augmented GenerationLooking up relevant documents first and handing them to the model, so it answers from your sources.
• Small Language ModelA compact language model that runs fast and cheaply, often on a laptop or phone, at the cost of some capability.
• World ModelAn AI model that predicts how an environment changes when someone acts in it, so it can simulate and plan.
• Large Language ModelA model trained on huge amounts of text to predict the next piece of text. The engine behind ChatGPT, Claude, Gemini and Mistral.
• DistillationTraining a smaller model to imitate a larger one, so you get much of the quality at a fraction of the cost.
• QuantizationShrinking a model by storing its numbers with less precision, so it runs on smaller, cheaper hardware.
• Open-Weight ModelA model whose trained parameters are published, so anyone can download and run it.
• Local AIRunning a model on your own laptop or server. Nothing leaves the machine, but smaller models fit.
• Open-Source AIAI released with the freedom to use, study, change and share it, which requires more than published weights.
• TransparencyBeing open that AI is involved, what it does and where its limits are.
• HallucinationWhen AI states something false with full confidence, because it produces plausible text, not checked facts.
• Knowledge CutoffThe date after which a model learned nothing new. Anything later must be given to it.
• GroundingTying a model's answer to specific, trusted sources, so it works from facts you provided instead of memory.
• BenchmarkA standard test used to compare models. Useful for orientation, but no substitute for testing on your own tasks.
• Benchmark ContaminationWhen test questions end up in a model's training data, so its benchmark score overstates its ability.
• Jagged CapabilityThe uneven skill profile of AI models: excellent at some hard tasks, unreliable at some easy ones.
• EvalsRepeatable tests that show whether your AI setup actually works, and whether a change made it better or worse.
• Cost per TaskWhat one finished piece of work really costs with AI: tokens, tools, retries and the human time to check it.
• LatencyThe waiting time between sending a request and getting the answer. In AI, speed is a feature.
• Model RoutingSending each request to the model that fits it best, small and cheap for easy jobs, large for hard ones.
• Self-HostingRunning AI models or tools on infrastructure you control, instead of using someone else's service.
• Vendor Lock-inWhen switching to another AI provider becomes so costly or painful that you effectively can't.

Sources

1. Attention Is All You Need, Vaswani et al., Google, 12 Jun 2017
2. But what is a neural network?, Grant Sanderson, 3Blue1Brown, 5 Oct 2017
3. Language Models are Few-Shot Learners (GPT-3), Brown et al., OpenAI (vendor), 28 May 2020
4. Intro to Large Language Models, summary, Andrej Karpathy, via Simon Willison, 23 Nov 2023
5. What are tokens and how to count them, OpenAI Help Center (vendor), undated
6. DeepSeek-V4 release, DeepSeek (vendor), 24 Apr 2026
7. Lost in the Middle: How Language Models Use Long Contexts, Liu et al., Stanford, 6 Jul 2023
8. A Dream of Spring for Open-Weight LLMs, Sebastian Raschka, Ahead of AI, 25 Feb 2026
9. Training language models to follow instructions with human feedback (InstructGPT), Ouyang et al., OpenAI (vendor), 4 Mar 2022
10. Tulu 3: Pushing Frontiers in Open Language Model Post-Training, Lambert et al., Allen Institute for AI, 22 Nov 2024
11. Scaling Laws for Neural Language Models, Kaplan et al., OpenAI (vendor), 23 Jan 2020
12. Training Compute-Optimal Large Language Models (Chinchilla), Hoffmann et al., DeepMind (vendor), 29 Mar 2022
13. Learning to reason with LLMs, OpenAI (vendor), 12 Sep 2024
14. Scaling LLM Test-Time Compute Optimally, Snell et al., UC Berkeley and Google DeepMind, 6 Aug 2024
15. Are Emergent Abilities of Large Language Models a Mirage?, Schaeffer, Miranda, Koyejo, Stanford, 28 Apr 2023
16. Models overview, Google DeepMind (vendor), 25 Oct 2026
17. Learning Transferable Visual Models From Natural Language Supervision (CLIP), Radford et al., OpenAI (vendor), 26 Feb 2021
18. High-Resolution Image Synthesis with Latent Diffusion Models, Rombach et al., LMU Munich and Heidelberg University, 20 Dec 2021
19. Embeddings guide, OpenAI (vendor), 25 Oct 2026
20. Phi-3 Technical Report, Abdin et al., Microsoft (vendor), 22 Apr 2024
21. Distilling the Knowledge in a Neural Network, Hinton, Vinyals, Dean, Google, 9 Mar 2015
22. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers, Frantar et al., IST Austria and ETH Zurich, 31 Oct 2022
23. LLM inference price trends, Epoch AI, 12 Mar 2025
24. Introducing GPT-6.1 Sol, OpenAI (vendor), Oct 2026
25. The Open Source AI Definition 1.0, Open Source Initiative, 28 Oct 2024
26. Meta Llama 3 Community License, Meta (vendor), 18 Apr 2024
27. AI Index Report 2026, Chapter 1: Research and Development, Stanford HAI, Apr 2026
28. AI Index Report 2026: Technical Performance, Stanford HAI, Apr 2026
29. AI Index Report 2026, Chapter 3: Responsible AI, Stanford HAI, Apr 2026
30. AI Index Report 2026, Chapter 8: Policy and Governance, Stanford HAI, Apr 2026
31. The 2025 Foundation Model Transparency Index, Wan, Bommasani, Liang et al., Stanford CRFM, 11 Dec 2025
32. Trends in AI, Epoch AI, 5 Feb 2026
33. Open-weight models lag closed frontier by about four months, Epoch AI, 29 May 2026
34. Open models topic page, Epoch AI, 16 Oct 2026
35. AI models leaderboard, Intelligence Index v4.3.2, Artificial Analysis (independent, commercial), 25 Oct 2026
36. Frontier AI Trends Report, UK AI Security Institute, 2025
37. Why language models hallucinate, OpenAI (vendor), 5 Sep 2025
38. Transparency Hub, Anthropic (vendor), 25 Oct 2026
39. Investigating Data Contamination in Modern Benchmarks for Large Language Models, Deng et al., Yale, 16 Nov 2023
40. The Leaderboard Illusion, Singh et al., Cohere Labs with universities, 29 Apr 2025
41. The Bitter Lesson, Richard Sutton, 13 Mar 2019
42. Yann LeCun's AMI Labs raises $1.03 billion, The Next Web, 10 Mar 2026
43. EU AI Act, Article 51: classification of GPAI models with systemic risk, European Union, 2024
44. Guidelines on obligations for providers of general-purpose AI models, European Commission, 18 Jul 2025
45. General-Purpose AI Code of Practice, signatories, European Commission, 25 Oct 2026
46. Governor Newsom signs SB 53, Office of the Governor of California, 29 Sep 2025
47. Interim Measures for the Management of Generative AI Services, Cyberspace Administration of China, 13 Jul 2023
48. CAC generative AI filings hit 1,112, remio, citing CAC, 15 Sep 2026
49. Mistral news, Mistral AI (vendor), 25 Oct 2026
50. EXAONE, LG AI Research (vendor), 25 Oct 2026
51. Falcon blog, Technology Innovation Institute (vendor), 6 Oct 2026
52. Sarvam releases 30B and 105B LLMs under Apache 2.0, OpenSourceForU, Mar 2026
53. Gobierno lanza Latam-GPT, Radio Universidad de Chile, 10 Feb 2026
54. AI Security Institute, UK Department for Science, Innovation and Technology, 25 Oct 2026