Every AI model ultimately runs on silicon. The chips involved have become strategic assets, the subject of trade restrictions and the reason one company, NVIDIA, became among the most valuable in the world. Here is what the main types do and why they differ.
What AI computation looks like
Neural networks spend most of their time doing one thing: multiplying large matrices of numbers and adding the results. A single answer from a language model involves billions of these multiply-and-add operations. Chips that perform many of them in parallel — and move data fast enough to keep busy — win.
CPUs: the generalists
Central processing units run operating systems and everyday software. They have a few powerful cores optimized for complex, sequential tasks. They can run small AI models, but they are inefficient at massive parallel arithmetic.
GPUs: parallel workhorses
Graphics processing units were built to calculate millions of pixels at once for video games. That same parallelism suits neural networks. In 2012, the AlexNet image recognition model was trained on two consumer NVIDIA GPUs, helping launch the deep learning era.
Modern data-center GPUs add specialized tensor cores for matrix math, support low-precision number formats that speed up AI, and use high-bandwidth memory (HBM) stacked next to the processor. They remain the most flexible option: almost any AI model or research idea can run on them.
TPUs: Google’s custom accelerators
Google’s Tensor Processing Units, announced in 2016, are application-specific chips designed for neural network workloads. They are organized around large matrix-multiplication units and connected into “pods” of thousands of chips with dedicated high-speed links. Google trains and serves its Gemini models on TPUs and rents them through Google Cloud. Amazon (Trainium, Inferentia), Microsoft (Maia) and Meta (MTIA) have similar custom chips.
NPUs: AI in your pocket
Neural processing units are small, power-efficient AI accelerators built into phones, laptops and other devices. They handle tasks such as camera processing, voice recognition, background blur and small language models locally, without draining the battery. Their performance is often quoted in TOPS (trillions of operations per second). See our explainer on NPUs and AI PCs.
Memory and interconnects matter as much as compute
For large models, raw arithmetic is often not the bottleneck:
- Memory capacity decides whether a model fits on a chip.
- Memory bandwidth decides how fast weights can be read — critical when generating text token by token.
- Interconnects link chips together; training frontier models requires thousands of accelerators communicating constantly.
This is why AI hardware announcements emphasize HBM capacity, bandwidth and networking, not only peak operations per second.
Which chip runs what
| Chip | Where | Typical AI work |
|---|---|---|
| CPU | Everywhere | Small models, data processing |
| GPU | PCs, servers, cloud | Training and serving most models |
| TPU and custom ASICs | Hyperscale clouds | Large-scale training and inference for specific platforms |
| NPU | Phones, laptops, devices | Efficient on-device inference |
Common misconceptions
- More TOPS means a better AI chip. TOPS figures use different precisions and rarely reflect memory limits. Real workloads depend on memory, bandwidth and software.
- GPUs are only for gaming. Data-center GPUs are specialized AI machines with large high-bandwidth memory and fast interconnects.
- Custom chips will quickly replace GPUs. Custom accelerators excel at their owners’ workloads, but GPUs keep a large flexibility and software advantage.
Choosing in practice
Most developers never choose chips directly: they choose a cloud service or device, and the hardware follows.
Key takeaways
- AI mostly means massive matrix multiplication, which rewards parallel chips.
- GPUs are the flexible default for training and serving AI models.
- TPUs and other custom accelerators are optimized for their owners’ large-scale workloads.
- NPUs bring efficient AI to phones and laptops.
- Memory capacity, bandwidth and interconnects often limit performance more than raw compute.
Frequently asked questions
Can I buy a TPU?
Google’s data-center TPUs are rented through Google Cloud rather than sold. Google has offered small Edge TPU devices for embedded use.
Why is HBM so important?
High-bandwidth memory feeds the chip with data quickly. Generating text requires reading model weights repeatedly, so bandwidth directly affects speed.
Do I need an NPU to use AI on my laptop?
No. Cloud assistants run on remote servers, and local models can run on CPUs or GPUs. An NPU makes certain on-device AI features faster and more power-efficient.