Memory-bandwidth-bound generation
Writing each new token means reading the model's active weights out of memory, so on local hardware the generation speed follows memory bandwidth (GB/s) far more than CPU or GPU core counts. Kai cites Apple's machine learning research for this: twice the bandwidth gives about twice the tokens per second, and the Neural Engine or extra cores don't help.
Numbers from Kai's Mac mini breakdown (Sept 2026): the M6 reads at 153 GB/s and the older M5 Pro at 307 GB/s, which is why buying guides favour the M5 Pro. The 24 GB M6 configuration runs at 170 GB/s, against 153 for a 16 GB model upgraded to the same memory.
Memory size is a different limit: it decides whether a model loads at all. Prefill is outside the rule (time-to-first-token). Sparse models ease the bandwidth limit because only the active experts are read per token (mixture-of-experts).
tina-huang (Sept 2026) shows the same rule at the low end: a Raspberry Pi 5 and an iPhone 16 both have 8 GB, but the iPhone runs models faster because its memory bus is wider and it adds a GPU and an NPU. In their kitchen analogy the bus is the conveyor belt from the prep counter (RAM) to the cooks. See capacity-vs-bandwidth.
Related: model-parameters, open-weights.