Local AI Speed Is Decided by Memory Bandwidth — Four Machines Measured

本ページは広告（アフィリエイトプログラム）を含みます。詳しくはプライバシーポリシーをご覧ください。

When you are thinking about running local AI on a new PC, Mac or mini PC, the thing you most want to know is: how fast will it actually be? You can work that out from the spec sheet before you buy, with reasonable accuracy. The number that matters is not CPU speed or core count but memory bandwidth — how quickly data can be read from and written to memory. I measured four machines here to check it. (Measured on real hardware, June 2026)

Why memory bandwidth decides it

Every time the AI produces one character — one token, strictly — it re-reads the model weights, tens of gigabytes of data, from memory. How fast memory can be read translates directly into how fast text comes out. The width of the pipe into memory matters more than raw compute; the technical term for the state is “memory-bound." Turn that around and it means that if you know the memory bandwidth, you can roughly predict the speed.

Four machines: divide speed by bandwidth and you get nearly the same number

I ran the same model (8 billion parameters — a small model with 8 billion internal values) on four machines of very different character, and divided each generation speed by that machine’s published memory bandwidth. The devices have nothing in common, and yet the results line up remarkably well.

Machine Memory bandwidth Generation speed Speed ÷ bandwidth
RTX 3090 (discrete GPU) 936 GB/s 130 tok/s 0.139
RTX 3060 (discrete GPU) 360 GB/s 61 tok/s 0.169
Mini PC (128GB unified memory) 256 GB/s 40 tok/s 0.155
Mac mini M4 (base) 120 GB/s 19 tok/s 0.161

The right-hand column lands between 0.14 and 0.17 in every case. Discrete GPU, the mini PC’s unified memory (a large pool shared by CPU and GPU), or a Mac — regardless of type, speed tracks memory bandwidth almost proportionally.

Using it: estimate your machine’s speed before you buy

From that proportion comes a rough rule. For a small model, generation speed is about “bandwidth × 0.15" tokens per second. Find the bandwidth figure on the spec sheet and multiply by 0.15.

Machine (bandwidth) Predicted speed
Mac mini M4 / 120 GB/s about 18 tok/s (measured 19, near enough)
MacBook Pro M5 base / 154 GB/s about 24 tok/s
Mini PC (128GB unified) / 256 GB/s about 40 tok/s
M5 Max / 614 GB/s about 90 tok/s

It is a coarse guide, but it is enough to tell whether a machine will feel fine in conversation or leave you waiting. People read at roughly 10 tokens per second, so anything above that can be treated as usable for back-and-forth.

Three conditions worth knowing

The rule comes with caveats. For honesty’s sake, here they are one by one.

  • This is about the speed of writing. The speed of reading — feeding in a long document to be summarised — is governed by compute and behaves differently.
  • Compare at the same compression (quantisation). Shrinking a model to 4-bit rather than 8-bit changes how much data is read per pass, and the coefficient shifts with it. Compare like with like.
  • The bandwidth figures are theoretical. Usable bandwidth is around 80–90% of the theoretical figure depending on the machine. The mini PC here measured about 215 against a theoretical 256.

One more: Macs have a dedicated high-speed engine and can come out faster than this rule suggests. That is a gap worth checking by actually running something.

Conclusion: the choice is bandwidth or capacity

Local AI speed is mostly decided by memory bandwidth. Hold on to that one point and choosing hardware gets much simpler. If you want speed above all, a discrete GPU with wide bandwidth. If you want to load the largest models you can, a mini PC or Mac with a big pool of unified memory. Bandwidth and capacity trade against each other, and which you prioritise decides the answer. Next time: supposing you can load a large model, how clever is it really, and how close does it get to the latest cloud models? I measured that too.

Sources

Benchmarks,Local AI