Capacity vs bandwidth
Two separate limits on local AI hardware: capacity (RAM) decides whether a model fits at all, while memory bandwidth and the number of processors (GPU, NPU) decide how fast it runs.
tina-huang's cases:
- A Raspberry Pi 5 and an iPhone 16 both have 8 GB, but the iPhone is faster and can generate images, because its memory bus is wider and it has a GPU and an NPU; the Pi has only a CPU.
- The AMD Ryzen AI Halo is slow, especially for image, video and music, because its bandwidth is small, though its 128 GB (about 96 usable) holds Llama 3.3 70B, GPT-OSS 120B and several models loaded at once.
- A rented RTX 4090 generates fast but has only 24 GB.
- Only the 8x H100 (640 GB) has both. In their words, everything below it is "forcing you to choose between like capacity and bandwidth".
Image, music and video generation are the compute-heavy end and need many GPU cores on top of memory.
Related: memory-bandwidth-bound-generation (why bandwidth sets token speed), ram-to-parameters-rule (sizing for capacity).
Source: report
Linked from
Memory-bandwidth-bound generationRAM-to-parameters ruleTina Huang