Chips

What it is about

A neural network is mostly one operation repeated billions of times: multiply two numbers and add the result. Grouped together, these are matrix multiplications, huge tables of numbers multiplied with each other. A normal processor, the CPU, works through them with a few powerful cores, one after another. A GPU or another AI chip has thousands of simpler units that do them at the same time.

That is why the chip layer decides how fast and how expensive AI is, both for training a model and for inference, the moment a model answers you. It also explains why a few companies and countries have so much power over AI: the most advanced chips need machines, materials and know-how that exist in very few places.

Why AI needs special chips

Computer architects John Hennessy and David Patterson describe the shift as the rise of domain-specific architectures: chips built for one kind of work. They beat general processors for three main reasons. They run many simple calculations in parallel, they keep data in memory right next to the compute, and they calculate with fewer digits when that is precise enough.

Google's first AI chip, the TPU, shows the effect. On Google's own inference workloads it was 29 times faster than a general-purpose processor and more than 80 times more energy efficient, as Hennessy and Patterson summarise.

• CPU: the central processor: a few strong, flexible cores for step-by-step logic. It handles almost any program and coordinates the AI chips.
• GPU: the graphics processor: thousands of simpler cores plus special matrix units. Built for graphics, today the flexible standard for AI.
• ASIC: an application-specific chip for one purpose, such as Google's TPU or Amazon's Trainium. Very efficient, usually rented inside its maker's cloud.
• NPU: the AI unit inside phones and laptops. It runs small models directly on your device, so your data stays there.

Precision in short: FP32, FP16, BF16, FP8 and FP4 describe how many bits a number uses. Fewer bits mean more calculations per chip, less memory per number and bigger models in the same memory. On Google's current TPU, halving the precision exactly doubles peak speed, according to its datasheet.

The memory wall: why memory matters as much as compute

Modern AI chips can calculate much faster than they can fetch data. Hennessy and Patterson put it in energy terms: reading a block from a small on-chip cache costs about 200 times the energy of a simple addition. Moving the data is the expensive part.

That is why HBM, high-bandwidth memory stacked right next to the chip, appears in every chip announcement. Google's datasheet for its current TPU implies that the chip needs roughly 625 calculations for every byte it loads from memory to stay busy. Work with fewer calculations per byte leaves the compute waiting.

In practice this mostly affects answering. When a model writes its answer word by word for one person, it has to stream its whole set of parameters from memory for each step, while doing little maths. Serving many users at once (batching) reuses each loaded number and lowers the cost per token, at the price of slightly longer waiting times. This is standard knowledge in chip architecture.

From chip to cluster

Today the unit of AI compute is a whole rack or pod. Several chip pieces (chiplets) and memory stacks are combined through advanced packaging, dozens of chips are connected into one fast group, and thousands of these groups are networked in a data centre.

1. 01Die and chiplets. Google's current TPU consists of two chiplets, each with its own compute cores and 96 GiB of HBM.
2. 02Package. TSMC's CoWoS packaging places compute chips and HBM stacks side by side on a silicon base up to 3.3 times the size a single exposure can print.
3. 03Rack. NVIDIA's current flagship combines 72 GPUs and 36 CPUs in one rack, linked by NVLink so they act like one machine.
4. 04Pod and cluster. A Google TPU pod links up to 9,216 chips. Inside the pod each chip talks to its neighbours at 1,200 GB/s; to the wider data-centre network it has only about 12.5 GB/s. Keeping work inside the fast group is worth a lot.

How a chip is made

A leading AI chip passes through several countries and a handful of companies, each close to a monopoly in its step. The order below is the standard textbook sequence; the figures come from the sources named.

1. 01Design. Engineers design the chip in software (EDA tools, mainly from Synopsys, Cadence and Siemens), often licensing building blocks such as Arm cores.
2. 02Wafer. Ultra-pure silicon is grown into a crystal and sliced into 300 mm wafers. China produced 93.5% of the world's polysilicon in 2024, mostly solar grade.
3. 03Lithography. The finest layers are printed with extreme ultraviolet light (EUV) of 13.5 nm. It is created by hitting tin droplets with lasers, 50,000 times a second. Only ASML in the Netherlands makes these EUV lithography machines.
4. 04Fabrication. In the fab, the foundry, hundreds of steps build transistors and metal layers. TSMC earned 72.5% of global foundry revenue in Q2 2026. A leading-edge fab costs about $35 to 43 billion over ten years, construction and operation together (BCG).
5. 05Memory. HBM comes from three makers: SK hynix (50%), Samsung (33%) and Micron (18%) in Q2 2026, so about 83% from South Korean companies.
6. 06Packaging and test. Chips and memory are tested, cut apart and packaged together. In 2025 this advanced packaging, together with memory, was the main bottleneck for AI chips (Epoch AI).

The label 3 nm describes no measurable width on the chip. Since the 1990s these names have been marketing; experts propose density measures to replace them (IEEE Spectrum).

Where chip progress comes from today

For decades, smaller transistors made chips both faster and more efficient. That free ride is over: Dennard scaling, where smaller transistors also used less power, faded between 2007 and 2012, and by 2018 there was a roughly 15-fold gap between Moore's prediction and reality (Hennessy and Patterson).

Progress now comes from design. NVIDIA's chief scientist Bill Dally reported that AI inference on a single GPU got about 1,000 times faster in ten years, of which only about 2.5 times came from smaller transistors; most came from simpler number formats and smarter instructions (vendor statement). Independent data from Epoch AI shows leading AI hardware becoming about 40% more energy efficient each year.

More efficiency can even mean more consumption. Sasha Luccioni, Emma Strubell and Kate Crawford argue that efficiency gains in AI can increase total use, a rebound effect known as the Jevons paradox.

Which kinds of AI chips exist

For founders, five families matter. You use most of them as a cloud service.

• Data-centre GPUs: NVIDIA and AMD. The flexible default, sold to clouds and large buyers as whole racks.
• Custom chips of the clouds: Google TPU, Amazon Trainium, Meta MTIA, Microsoft Maia. Google holds about a quarter of the world's AI compute, mostly on its own TPUs (Epoch AI).
• Wafer-scale chips: Cerebras builds one chip from a whole wafer, with 4 trillion transistors and memory on the chip (vendor data).
• Inference specialists: Groq builds chips for fast answers. In December 2025 it licensed its technology to NVIDIA, non-exclusively, and stays independent.
• On-device NPUs: in phones and laptops. They run small language models and enable local AI without sending data to a cloud.

Where the raw materials come from

The material chain of a chip runs across continents: much is mined in Africa and South America, refined in China, turned into chips in Taiwan and South Korea, and designed in the United States. Each step is concentrated in few countries, and many of these critical minerals are hard to replace.

The human cost sits at the start. Researchers Fabio Andrés Díaz Pabón and Azza Mustafa Babikir Ahmed (University of Cape Town) describe miners in the DR Congo, some of them children, working cobalt mines without protection. Congolese writer Didier Cimalamungo Bahizire calls for verifiable supply-chain transparency and a fair share for local communities.

• Gallium: about 99% of primary production in China (USGS 2026).
• Polysilicon: 93.5% made in China in 2024 (Bernreuter Research).
• Rare earths and tungsten: China mines about 69% of rare earths and 79% of tungsten (USGS 2026).
• Cobalt and tantalum: the DR Congo mines about 73% of cobalt and about 52% of tantalum; Rwanda and Nigeria add about 16% each, Brazil about 8% (USGS 2026).
• Export controls: China has paused its export ban on gallium, germanium and antimony to the US until 27 November 2026, and a package of rare-earth and graphite controls until 10 November 2026. Check the current status.

The footprint of chip making

Making chips emits about 185 to 190 million tonnes of CO2 equivalent a year, projected to reach 247 million tonnes by 2030 (imec, TechInsights). A single wafer at the 2 nm class carries about 1,600 kg of CO2 equivalent, and chips account for about 30% of an AI data centre's carbon footprint as embodied emissions (imec).

Fabs also need a lot of water and power. TSMC reports a new contract for 136,000 cubic metres of desalinated seawater a day and a renewable share of 20.1% in 2025 (company report). Many fabs sit on carbon-heavy grids in Taiwan, China and South Korea, which raises the footprint of every chip. Globally, only 22.3% of e-waste was documented as recycled in 2022 (UN Global E-waste Monitor).

What this means for your decisions

You pay for AI chips with every API bill. The chip generation, the number format and how well the provider fills its chips all flow into your cost per task. Our reading of the sources: renting through APIs or the cloud fits most young companies, because hardware gets about 40% more efficient each year and a new flagship arrives yearly.

Often less hardware is enough. Smaller or compressed models, for example through quantization, often run on smaller GPUs, laptops or phones. Song Han's MIT team showed that 4-bit models can stay accurate when the few critical weights are protected. And if data must stay in-house, on-device chips or self-hosting are real options for small models.

What the layer is made of

Key figures, with source and date

• $1.5 to 1.66 tnforecast for the global chip market in 2026, roughly double 2025. The jump comes mostly from exploding memory prices.WSTS · 6 Aug 2026 · industry association
• about $795 bnglobal chip market in 2025. AI processors made up more than $200 bn, and NVIDIA alone 15.8% of all chip sales.Gartner · 12 Jan 2026 · market research
• 72.5%of global foundry revenue went to TSMC in Q2 2026. Companies headquartered in Taiwan held about 80%, in China about 10%.TrendForce · 9 Sep 2026 · market research
• over 15 millionAI chips counted as NVIDIA H100 equivalents are installed worldwide, needing more than 10 GW of power. Google owns about a quarter of this compute.Epoch AI · 7 Apr 2026 · neutral
• every 7 monthsthe compute in the world's largest single AI data centre doubles.Epoch AI · 11 Jun 2026 · neutral
• about 40%more energy efficient each year is leading AI hardware.Epoch AI · 23 Oct 2024 · neutral
• over 90%of packaging and memory for AI chips went to four companies in 2025, but only about 12% of the most advanced logic chips.Epoch AI · 12 Mar 2026 · neutral
• 185 to 190 MtCO2 equivalent a year from chip manufacturing worldwide.imec · 19 Aug 2025 · neutral

Regions compared

No region can build an advanced AI chip alone, as historian Chris Miller stresses. Each holds a different link of the chain, and almost every government now subsidises its part. Subsidy figures use different definitions and are not directly comparable.

Sources: SIA/BCG (8 May 2024), GAO (Aug 2026), TrendForce (9 Sep 2026), Counterpoint via Edaily (3 Sep 2026), US International Trade Administration (Nov 2025), METI via Adnkronos (11 Apr 2026), Yicai (27 May 2024), AENEAS (3 Jun 2026), ThePrint (15 Jul 2026), USGS (Feb 2026), ODI. Several figures come from secondary reports of the original.

Examples

• NVIDIA sells whole server racks: 72 GPUs and 36 CPUs, connected like one single computer.
• Google, Amazon, Meta and Microsoft build their own AI chips, often with Broadcom or Marvell as design partners.
• In China, Huawei Ascend is the main alternative, held back mostly by a lack of memory.

Why it matters

The chip layer explains three things you feel directly: why AI capacity is scarce, why prices fall so fast for the same performance, and why geopolitics reaches into your product. The bottlenecks sit one behind the other, from Congolese mines to Dutch lithography machines, Taiwanese fabs and Korean memory. A disruption at any link travels all the way to the price of an API call.

It also explains where responsibility lies. Chips carry a material footprint, from mining to water to e-waste, that customers, regulators and employees increasingly ask about.

Common misconceptions

• 3 nm means something on the chip is 3 nm wide.

  Node names are marketing labels; real features are much larger (IEEE Spectrum, 2020).

• More TFLOPS means faster AI.

  Memory often decides. Moving data costs far more energy than calculating, and many workloads wait for memory (Hennessy and Patterson).

• Vendor speed figures are comparable.

  They depend on the number format and on tricks such as sparsity. The same NVIDIA rack is listed at 288 or 3,600 PFLOPS, depending on the format.

• More efficient chips will lower AI's energy use.

  Efficiency gains can raise total use, a rebound effect (Luccioni, Strubell and Crawford, 2025).

• Only NVIDIA can run AI.

  Google runs pods of up to 9,216 of its own TPUs and owns about a quarter of the world's AI compute; Amazon, Meta and Microsoft build their own chips too.

• Chips are digital, so they are clean.

  Making them emits about 185 to 190 million tonnes of CO2 equivalent a year and depends on mining with serious human costs (imec; University of Cape Town researchers).

Perspectives from leading experts

Ten perspectives from research, industry, policy and critical scholarship. Each summary paraphrases a published work, linked below it. Vendors are marked as such.

• John Hennessy and David Patterson

  Turing Award laureates, authors of the standard textbooks on computer architecture, USA · academic

  General processors now gain only a few percent a year. The way forward is chips built for one domain, with simple parallelism, memory controlled by software and lower precision. Google's TPU is their prime example.

  A New Golden Age for Computer Architecture (CACM) · Feb 2019

• Vivienne Sze

  Professor at MIT, efficient AI hardware, USA · academic

  Running neural networks widely depends on hardware that saves energy and raises throughput without losing accuracy. She asks to judge chips on accuracy, energy, throughput, latency and cost together.

  Efficient Processing of Deep Neural Networks: A Tutorial and Survey · Mar 2017

• Song Han

  Associate professor at MIT, efficient deep learning, born in China · academic

  Only a small share of a language model's weights causes most of the error when it is compressed. Protecting those makes 4-bit models practical and more than three times faster on desktop and mobile GPUs. His MIT course on efficient AI is free online.

  AWQ: Activation-aware Weight Quantization (MLSys 2024 Best Paper) · Jun 2023

• Lisa Su

  CEO of AMD, born in Taiwan, USA · vendor

  Energy efficiency is the industry's central challenge. Without system-level gains, a top supercomputer could soon need about 500 MW, roughly a nuclear power plant. Her levers: chiplets, 3D stacking, specialised computing and lower precision.

  ISSCC 2023 keynote, reported by IEEE Spectrum · 21 Feb 2023

• Mark Liu and H.-S. Philip Wong

  Then chairman of TSMC and Stanford professor, Taiwan and USA · vendor and academic

  AI will need GPUs with about a trillion transistors within a decade. The path there runs through 3D integration, advanced packaging that combines compute dies with many HBM stacks, and silicon photonics.

  How We'll Reach a 1 Trillion Transistor GPU (IEEE Spectrum) · 28 Mar 2024

• Chris Miller

  Historian at Tufts University, author of Chip War, USA · policy research

  No country can build advanced chips alone; even the US depends on Europe, Japan, Korea, Taiwan and China. Governments are redrawing supply chains with tariffs and export controls. Access to chips is today's question, trust will be tomorrow's.

  Chips and chokepoints, lecture at Carnegie Mellon University · 11 Mar 2026

• Girish Sastry, Lennart Heim and co-authors

  Researchers on compute governance, Europe and USA · policy research

  Compute is a better lever for governing AI than data or algorithms: it is visible, can be restricted and measured, and its supply chain is highly concentrated. Badly designed rules could harm privacy and the economy.

  Computing Power and the Governance of Artificial Intelligence · 14 Feb 2024

• Kate Crawford

  Professor at USC and researcher at Microsoft Research, born in Australia · critical research

  AI's energy and water costs are rising and are mostly kept as corporate secrets. Industry should measure and report them, researchers should design leaner systems, and lawmakers should require disclosure.

  Generative AI's environmental costs are soaring and mostly secret (Nature) · 20 Feb 2024

• Sasha Luccioni

  Researcher on the energy and climate cost of AI, Canada · critical research

  Measuring 88 models showed that generating an image uses more than 1,000 times the energy of classifying a text. Large general models used for simple tasks cost far more than small task-specific ones, so model choice is a hardware decision.

  Power Hungry Processing (ACM FAccT 2024) · Jun 2024

• Fabio Andrés Díaz Pabón and Azza Mustafa Babikir Ahmed

  Researchers at the University of Cape Town, South Africa · critical research

  AI's supply chain begins in mines. In the DR Congo, miners, including children, work in dangerous cobalt mines without protection, a part of AI that is rarely acknowledged. They call for recognising this labour and supporting worker-led movements.

  How AI exploits (D+C Development and Cooperation) · 23 Apr 2025

Terms for this layer

• GPUA graphics processor: a chip that runs thousands of simple calculations at once, which made it the workhorse of AI.
• AI ChipA processor built for the calculations AI needs, such as a GPU or a custom chip, trading flexibility for speed and efficiency.
• NPUThe AI unit inside phones and laptops: a small chip block that runs AI models directly on your device.
• ASICA chip designed for one specific job. In AI, Google's TPU and Amazon's Trainium are the best-known examples.
• CPUThe central processor of every computer: a few strong, flexible cores that work through tasks one step after another.
• HBMVery fast memory stacked right next to an AI chip, so its processors never wait for data. In 2025, it was a key bottleneck.
• FoundryA company that manufactures chips designed by others. The most advanced AI chips come from very few of them.
• ComputeThe computing power, and the chips and software behind it, that AI needs to be trained and to run.
• TrainingThe one-time, very expensive process that sets a model's parameters by showing it vast amounts of data.
• InferenceRunning a finished model to get an answer. What you pay for every time you send a message.
• ParametersThe billions of numbers inside a model, set during training. Everything the model knows is stored in them.
• TokenThe small chunk of text a model reads and writes. Price, speed and memory limits are all counted in tokens.
• QuantizationShrinking a model by storing its numbers with less precision, so it runs on smaller, cheaper hardware.
• Small Language ModelA compact language model that runs fast and cheaply, often on a laptop or phone, at the cost of some capability.
• Local AIRunning a model on your own laptop or server. Nothing leaves the machine, but smaller models fit.
• Data CentreA building full of servers, storage and network equipment, with the power and cooling that AI runs on.
• HyperscalerOne of the few tech giants that run data centres at enormous scale and rent out computing power, storage and AI services.
• Scaling LawsThe observation that a model's error falls predictably as model size, data and computing power grow.
• Cost per TaskWhat one finished piece of work really costs with AI: tokens, tools, retries and the human time to check it.
• Self-HostingRunning AI models or tools on infrastructure you control, instead of using someone else's service.
• Vendor Lock-inWhen switching to another AI provider becomes so costly or painful that you effectively can't.
• EUV LithographyThe machine process that prints the finest structures of modern chips with extreme ultraviolet light. Only ASML builds these machines.
• Advanced PackagingCombining several chips and memory stacks into one package, side by side or on top of each other. Today one of the main bottlenecks for AI chips.
• Critical MineralsRaw materials that chips, power grids and batteries cannot do without and that come from very few countries, such as gallium, cobalt or rare earths.

Sources

1. A New Golden Age for Computer Architecture, Hennessy, Patterson, CACM, Feb 2019
2. Heeding Huang's Law: Hot Chips 2023 keynote by Bill Dally, NVIDIA (vendor), 29 Sep 2023
3. Can AI Scaling Continue Through 2030?, Epoch AI, 20 Aug 2024
4. TPU7x (Ironwood) documentation, Google Cloud (vendor), Oct 2026
5. Vera Rubin NVL72, NVIDIA (vendor), 21 Oct 2026
6. CoWoS technology, TSMC (vendor), 2026
7. Light and lasers in lithography, ASML (vendor), 2026
8. A better way to measure progress in semiconductors, Samuel K. Moore, IEEE Spectrum, 21 Jul 2020
9. Navigating the costly economics of chip making, BCG, 28 Sep 2023
10. Global semiconductor market in Q2 2026, WSTS, 6 Aug 2026
11. Worldwide semiconductor revenue 2025, Gartner, 12 Jan 2026
12. Foundry revenue ranking Q2 2026, TrendForce, 9 Sep 2026
13. HBM market share Q2 2026, Counterpoint via Edaily, 3 Sep 2026
14. Emerging Resilience in the Semiconductor Supply Chain, SIA and BCG, 8 May 2024
15. CHIPS Act implementation, US Government Accountability Office, Aug 2026
16. Japan semiconductors, US International Trade Administration, 20 Nov 2025
17. Chips Act 2.0 proposal, AENEAS, 3 Jun 2026
18. Big Fund III, Yicai Global, 27 May 2024
19. Additional aid for Rapidus, Adnkronos, 11 Apr 2026
20. Semicon Mission 2.0, ThePrint, 15 Jul 2026
21. South Korea chip fund, The Standard, 10 Aug 2026
22. Export of advanced chips to the UAE and Saudi Arabia, Jordan News, 21 Nov 2025
23. Mineral Commodity Summaries 2026: gallium, USGS, Feb 2026
24. Mineral Commodity Summaries 2026: tantalum, USGS, Feb 2026
25. China produces over 90% of global polysilicon, Solarserver citing Bernreuter Research, 25 Nov 2025
26. China suspends export controls on key raw materials, CIRS Group, Nov 2025
27. Critical minerals, critical moment: Africa's role in the AI revolution, ODI, 2025
28. Nothing in AI is immaterial, Didier Cimalamungo Bahizire SJ, SJES, 14 Aug 2026
29. How can we reduce the environmental impact of chip manufacturing, imec, 19 Aug 2025
30. 2025 Sustainability Report, TSMC (vendor), 2026
31. Global E-waste Monitor 2024, UNITAR and ITU, 20 Mar 2024
32. From Efficiency Gains to Rebound Effects, Luccioni, Strubell, Crawford, 27 Jan 2025
33. Groq and NVIDIA licensing agreement, Groq (vendor), 24 Dec 2025
34. Wafer Scale Engine, Cerebras (vendor), 2026