Local LLM Load Times Measured on 3 Machines: The Culprit Is the SSD — and the Memory Wall

2026年7月16日

When people talk about local LLM (large language model — a ChatGPT-style text-generation AI running on your own computer) performance, the conversation usually revolves around “tokens per second." In everyday use, though, what bothers you more is often the wait before generation starts. Big models in particular can keep you waiting a surprisingly long time before they even begin. I wanted to pin down where that wait actually comes from, so I measured the full end-to-end waiting time on three machines I have at hand. (Hands-on measurements on 3 machines, as of July 2026)

What I measured: not generation speed, but every second you wait

I used four yardsticks, all of which can be extracted from a single request to the model.

• First load time (cold). Starting from a state where the model is not yet in memory — like right after boot — how long until it is ready to use. This is the time to read the model from disk.
• Second load time (warm). How long a reload takes after the model has been loaded once. This usually comes back from the memory cache (a mechanism that keeps recently read data in a faster place).
• Time to first token (TTFT). The wait you actually feel: from sending a question to seeing the first character of the reply.
• Generation speed (decode). How fast the text is written out — the familiar “tokens per second."

The three machines: a desktop with dedicated GPUs (GeForce RTX 3090 24GB + GeForce RTX 3060 12GB, 62GB system RAM), the GMKtec EVO-X2 (hereafter EVO-X2), a mini PC with 122GB of unified memory, and a compact Apple Mac mini (M4, 24GB). All runs used the same conditions (temperature 0, identical prompt), with model sizes ranging from 8B (about 5GB) to 235B parameters (about 86GB).

Finding #1: startup time scales with model size

First, cold load times on the dedicated-GPU desktop.

Warm loads are near-instant for every model. They come back from cache, so size barely matters. The problem is the first load. Up to a point, the numbers line up neatly: load time grows in proportion to data size. Fitting a line from 8B through 70B gives load time ≈ about 1.6 s (fixed startup cost) + data size ÷ about 4 GB/s — and that “4 GB per second" is exactly the real-world read speed of this machine’s SSD. Most of the first-load wait is simply the time to read the model off the SSD. The culprit behind slow startup turned out to be SSD read speed.

Finding #2: once the model doesn’t fit in RAM, times explode

The bottom two rows of the table, however, break far away from that formula. The 120B model took 88 s against a predicted 18 s; the 235B took 107 s against a predicted 23 s — four to five times slower. The boundary is unmistakable. This desktop has 62GB of system RAM, and the blow-up happens exactly with the 65GB and 86GB models that exceed it.

When the whole model can’t sit in RAM, part of it spills over while loading, and the process is no longer a simple SSD read. Beyond this point, a second wall — separate from SSD speed — dominated the waiting time.

On a high-memory machine, the same giant models start 4× faster

Next I put exactly the same models on the EVO-X2, with its 122GB of unified memory. The hypothesis to check: with double the memory, the 120B and 235B models should stay on the “fits in RAM" side.

It played out as predicted. On the EVO-X2, both 120B and 235B fit inside the 122GB of memory, so they never hit that second wall. All six models fell back onto a single straight line — the plain proportional relationship set by SSD read speed. The result: first loads 4.1× faster for 120B and 3.9× faster for 235B. For the mid-size 70B the desktop was slightly quicker (a difference tied to dedicated-GPU startup), but the bigger the model, the more memory capacity mattered.

And it isn’t only about the wait. Generation speed also pulls apart on large models: gpt-oss 120B ran at 36.7 tokens/s on the EVO-X2 versus 17.5 on the desktop, and the 235B at 19.6 versus 6.0. Being able to hold the whole model in memory, without spilling, paid off in both loading and generation. The value of large unified memory is not just that “big models fit" — it’s that when they do, first loads don’t fall off a cliff and generation stays fast. Measuring it made that plain.

▼ The EVO-X2 tested in this article (check prices on Amazon / Rakuten)

GMKtec EVO-X2 (Ryzen AI Max+ 395 / 128GB / 2TB)

¥523,499 (as of 2026-06-18 楽天)

Amazon楽天市場Yahoo!GMKtec公式

From the second time on, it’s instant — only the “first time in a while" is slow

Everything so far has been about the first load, and there is a silver lining. Once a model has been read in, every subsequent load takes under a second on every model, because the memory cache does its job. The only real waits are the first run of the day, or a reload after the model has been evicted from memory during a long idle. Put the other way around: just keeping the model resident (configuring it to stay in memory for a while) removes most of the day-to-day waiting. If you use giant models, an operating habit that avoids frequent reloads translates directly into perceived speed.

Conclusion: choosing by waiting time

Turning the measurements into buying advice gives this picture. If you mainly run small-to-mid models (sizes that fit comfortably in memory), even cold loads take a few seconds, and a dedicated-GPU desktop is straightforwardly fast and pleasant. If, on the other hand, you plan to run 100B–200B-class giants daily, memory capacity decides your waiting time. The moment a model exceeds system RAM, first loads jump from tens of seconds to nearly two minutes. A mini PC with large unified memory kept even those giants to around 30 seconds on first load, with generation speed intact. If you want to run big models without the wait, memory capacity is the first number to look at.

Method, and some honest caveats

Each condition was measured five times and I report the median. For cold runs, the model’s files were evicted from the memory cache before every measurement, forcing a re-read from the SSD (on the two Linux machines, using per-file cache eviction). A few things deserve honest disclosure:

• One model (a high-precision Gemma-family quantization, 34GB) was an extreme outlier — 32 s cold, far off the line for its size. I suspect a quirk in how it spills to the CPU side, but I haven’t fully chased down the cause. Read the trends in the tables as excluding this one.
• The Mac mini has 24GB of memory, so only the small models were measured there. Its cache-eviction mechanism also differs from Linux, so the Mac’s “cold" numbers are reload-based approximations and not strictly on the same footing as the other two machines. Treat them as reference values.
• I also intended to test how time-to-first-token grows with prompt length, but I couldn’t make the inputs long enough for solid numbers this time. That’s homework for a future article.

Questioning the yardstick itself is part of what I consider real measurement. These numbers come from my three machines; different SSDs and memory configurations will give different values. Take this as a report on the tendency — first-load waiting is set by your SSD and your memory capacity — rather than as universal figures.

*Test environment: desktop (GeForce RTX 3090 24GB + GeForce RTX 3060 12GB / 62GB system RAM / NVMe SSD), GMKtec EVO-X2 (AMD Ryzen AI Max+ 395 / 122GB unified memory), Apple Mac mini (M4 / 24GB). Inference engine: Ollama. Conditions: temperature 0, identical prompt, median of 5 runs each. As of July 2026.