Gemma 4 Speed by Quantization: Same Model, Several-Fold Difference (and the Quant That Won’t Fit)

2026年7月8日

Even with the exact same large language model, the single choice of how you apply “quantization" (a form of compression) can change generation speed by several times over, and in some cases can determine whether the model even fits on your graphics card at all. Here I take Google’s lightweight Gemma 4 model and, focusing on measurements I actually ran on my own RTX 3090, lay out how speed shifts across quantization variants.

Quantization is a technique that deliberately coarsens the numeric precision inside a model to shrink its size. It’s a similar idea to lowering a photo’s image quality to make the file lighter: in exchange for giving up a little accuracy, you cut down on memory use and processing load. Labels like “Q4" and “Q8" indicate the degree of that coarseness, and the smaller the number, the more has been trimmed away.

Test conditions

To keep the attribution of the numbers clear, I’ve separated how each machine was measured.

Generation speed is expressed as “how many tokens can be produced per second (tok/s)." A token is roughly a word or a fragment of a word, and the larger the number, the faster text comes out.

Results from changing quantization on the RTX 3090

These are the measured values from running different quantizations and sizes of Gemma 4 on the same 3090.

What stood out clearly was that even at the same 26B, the “MoE a4b, QAT" version ran roughly 1.4x faster than the standard version, while the 31B Q8 version exceeded 24GB of capacity and instead became extremely slow.

MoE (mixture of experts) is a scheme that runs only the necessary portion of the model each time rather than the whole thing, so it runs lighter than its apparent size suggests. QAT (quantization-aware training, where quantization is baked in and the model is retrained) is a version tuned so that accuracy holds up even when coarsened. The overlap of these two, the “a4b, QAT" build, ran the most efficiently on the 3090.

The 31B Q8 version, on the other hand, swells to about 33GB to preserve accuracy, which doesn’t fit in 24GB of VRAM. The overflow gets pushed out beyond the GPU, and speed dropped all the way to 8 tok/s. Quantization isn’t the simple story of “coarser is always faster"; keeping it to a size that actually fits is the precondition for speed.

How other machines fared

Here I also line up data measured on the same model in different environments. Because the conditions differ from my 3090, please treat these as reference values labeled with the machine name.

Comparing the same “26B, MoE a4b, QAT," the data showed 33 tok/s on the mac-mini against 110 tok/s on the 3090. Between a dGPU (a separate, dedicated graphics card) and a mini PC that folds memory into the CPU, the difference in memory bandwidth (how fast data moves to and from memory) shows up directly as a difference in speed. The EVO-X2’s strength is the capacity to hold large models, and the data positions it as conceding raw speed to the 3090.

Rules of thumb for choosing a quantization

Here’s what I can say within the bounds of these measurements.

Check first and foremost whether it fits in VRAM. Pick a quantization that doesn’t fit, and no matter how high the accuracy, speed drops beyond the point of being usable.

At the same size, favor the MoE and QAT versions. On the 3090, “a4b, QAT" came out clearly faster than the standard version.

There’s no need to swing all the way to a coarse quantization like Q4 for speed’s sake alone. If it fits, a version that preserves accuracy can still deliver plenty of speed.

When this setup isn’t a good fit

Measuring and comparing multiple quantizations on a single machine burns time and storage on downloading and swapping models. For someone who just wants to pick one variant and keep using it, it’s overkill.

The numbers are strictly for the fixed conditions of context length 4096 and 512 generated tokens. Since the tendencies change once your use case shifts, such as long-input prompts or image generation, read them on the assumption that you’ll re-measure under conditions close to your own usage.

Conclusion: Quantization isn’t “coarser is faster." First choose a size that fits in VRAM, then take an efficient version like MoE or QAT on top of that. In these 3090 measurements, that was the combination that best balanced speed and accuracy.