Skip to main content
Zubnet AI › Learn › Wiki › NPU
Infrastructure

NPU

Also known as: Neural Processing Unit, AI Accelerator
A chip, or part of a chip, built specifically to run neural networks efficiently, found in phones, laptops and other devices for on-device AI.

Why it matters

NPUs are what make always-on AI features practical on battery-powered devices: face unlock, camera processing, live captions, background blur in video calls, voice typing, on-device translation. Running these on the CPU or GPU would drain the battery and heat up the device.

They became a marketing headline in 2024, when Microsoft introduced Copilot+ PCs, a category that requires an NPU rated at 40 TOPS or more. That pushed every laptop chip maker to ship faster NPUs, and "AI PC" became a label on most new machines.

Deep Dive

A neural processing unit is a processor designed for one job: the matrix arithmetic of neural network inference. Where a CPU is a generalist and a GPU is a powerful parallel engine, an NPU trades flexibility for efficiency. It does a narrow set of operations, usually on low-precision integers, using very little power.

Most NPUs are not separate chips. They're a block inside a system-on-chip, next to the CPU and GPU. Apple's Neural Engine has been in iPhones since the A11 in 2017 and is in every Apple Silicon Mac. Qualcomm's Snapdragon chips have the Hexagon NPU, Intel's Core Ultra processors include one branded AI Boost, and AMD ships its own in Ryzen AI chips.

What TOPS Means

NPU performance is advertised in TOPS, trillions of operations per second. Snapdragon X Elite launched at 45 TOPS, Intel's Lunar Lake at 48 and AMD's Ryzen AI 300 series at 50. Treat these numbers with caution. TOPS is a theoretical peak, usually measured with 8-bit integer math, and it says nothing about memory bandwidth, supported operations or software quality, which often matter more in practice. Two NPUs with the same TOPS can perform very differently on a real model.

What NPUs Are Good At, and Not

NPUs excel at small to medium models with fixed shapes that run continuously: vision models, speech recognition, noise suppression, small language models. They usually require the model to be quantized to 8-bit integers and converted to a vendor format.

They've been much less useful for large language models. LLMs are limited by memory capacity and bandwidth more than by raw arithmetic, they work with variable sequence lengths, and many NPUs lack support for operations they need. This is why popular local LLM tools such as Ollama, LM Studio and llama.cpp mostly run on the GPU or CPU and ignore the NPU. The exceptions are models that the platform vendor has specifically adapted, such as Microsoft's Phi Silica on Copilot+ PCs, Google's Gemini Nano on Android and the on-device models in Apple Intelligence.

A Fragmented Software Stack

Each vendor has its own toolchain: Core ML for Apple, QNN for Qualcomm, OpenVINO for Intel, Ryzen AI software for AMD and LiteRT (formerly TensorFlow Lite) on Android. Cross-platform layers such as ONNX Runtime and Windows ML try to hide the differences, with mixed results. For developers, this fragmentation is the main obstacle. A model that runs on one NPU often needs separate conversion and testing for another, and if an operation isn't supported, it may quietly fall back to the CPU, where performance collapses.

The Wider Family of AI Accelerators

"AI accelerator" is the broader term for any hardware specialized for neural networks. In data centers that includes Google's TPUs, Amazon's Trainium, Microsoft's Maia, Meta's MTIA and chips from companies such as Cerebras and Groq. Those are vastly larger and aimed at training or serving big models, but the principle is the same: give up generality to gain efficiency on one workload.

Should You Care When Buying?

For running LLMs locally, look at memory and the GPU first: how much RAM or VRAM the machine has, and how fast it is. For battery life and built-in features such as live captions or webcam effects, the NPU matters. If you build apps, target the NPU only when you need sustained, low-power inference for a small model, and plan for a GPU or CPU fallback.

← All Terms
ESC