AI Vocabulary · 02 AI Infrastructure
What is NPU?
The AI unit inside phones and laptops: a small chip block that runs AI models directly on your device.
An NPU, neural processing unit, is a part of a phone or laptop chip built for AI calculations. It runs small models with little power, so features like live translation, photo editing or a local assistant work without sending data to a cloud.
NPUs are far smaller than the data centre chips that train large models. They are made for inference, the moment a model answers, and they shine with small language models that have been compressed through quantization. MIT researcher Song Han's work shows how 4-bit models can run accurately on laptops and phones.
Why it matters for you
On-device AI keeps data on the device and works offline. For sensitive information, such as health data or confidential documents, that is often the safest setup.
Own it
When you choose laptops or phones for your team, check whether the NPU can run the models you need. For privacy-sensitive tasks, local AI on an NPU can replace a cloud service.
Where it comes from
How compressed models run on small devices is shown in AWQ: Activation-aware Weight Quantization by Ji Lin, Song Han and co-authors (MIT, MLSys 2024 Best Paper).
Related
Your AI Learning Hub
Learn the whole AI Vocabulary, step by step, and build with it. Build it. Own it.