Introduction to NPUs

A CPU is built to do almost anything, one instruction at a time, which makes it flexible but not particularly fast at any one task. A neural network's prediction step is mostly one repeated operation done millions of times over: multiplying and adding numbers in large grids (matrix multiplication). A Neural Processing Unit is a chip purpose-built to do exactly that operation, in parallel, and not much else. It cannot run a web browser or a spreadsheet efficiently -- that is not its job. What it does extremely well is run an already-trained neural network's predictions (inference) using far less power than a general-purpose CPU, or even a GPU, would need for the same task.