I Set It, So Why Is Nothing Happening? — The KV Cache Compression That Was Being Ignored
Running an LLM (a large language model, the thing behind text-handling AI) locally, you sometimes find it slows down sharply once you feed it something long. I looked into that before and traced the cause: the KV cache (the area that remembers what has been read) does not fit in the graphics card’s memory and spills over to the CPU side.
So compressing that KV cache ought to fix it. Ollama (a common way to run LLMs locally) has a setting for exactly this. I tried it, and none of the numbers moved at all. The setting was not taking effect.
This is as of July 2026.
The previous article in this series:
https://kimono-oyaji.com/en/arc-b580-context-length-spill-en...- 1. The goal: does compression let you handle long text?
- 2. The experiment: compression on or off, everything else equal
- 3. The result: all three gave the same numbers
- 4. Why it did not take effect, from the log
- 5. A by-product: the log confirmed what the spill is
- 6. Something else: past a point it stops falling
- 7. In summary: the conclusion is that I could not test it
- 8. Hardware used
- 9. Sources
The goal: does compression let you handle long text?
What set this off was a post from someone abroad: a single 12GB graphics card, a context of 240,000 tokens, and 77 tokens per second.
That is nothing like what I measure. On the same 12GB card, stretching the context to 32,768 tokens dropped me to 9 tokens per second. An order of magnitude apart.
Where is the difference? The post mentioned KV cache compression. I had never tried it once. That looked like the likely cause, so I set out to check.
The experiment: compression on or off, everything else equal
One variable at a time. KV cache precision at three settings (default f16, compressed to 8-bit, compressed to 4-bit), each measured across a range of context lengths.
| Test machine | Desktop PC (Ryzen 9 3950X / Intel Arc B580 12GB) |
| Model | qwen3:14b |
| Variable | KV cache precision (default / 8-bit / 4-bit) x six context lengths |
| Method | Three runs per condition, first discarded, median taken |
| Other | Configured to use only the Arc B580; the RTX 3090 in the same machine was not used at all |
The result: all three gave the same numbers
The result was rather deflating.
| Context length | Default (f16) | 8-bit | 4-bit | Load on the CPU |
|---|---|---|---|---|
| 2,048 | 36.05 | 36.08 | 36.16 | 0% |
| 8,192 | 28.16 | 28.27 | 27.97 | 7% |
| 16,384 | 12.31 | 12.41 | 12.43 | 18% |
| 32,768 | 9.09 | 9.20 | 9.13 | 33% |
| 65,536 | 7.86 | 8.14 | 8.18 | 37% |
| 131,072 | 8.03 | 7.75 | 8.21 | 37% |
The figures are tokens per second. All three conditions landed on essentially the same value at every context length, well inside run-to-run variation.
More conclusive is the rightmost column, the load on the CPU. Not one percent of difference across the three. If the KV cache were really being compressed, it would need less room and less of it would spill to the CPU. Nothing moving at all is evidence that no compression is happening.
Why it did not take effect, from the log
Opening the log of the program Ollama launches internally (llama-server) gave the answer.
K (f16): 3200.00 MiB, V (f16): 3200.00 MiB
Compression was requested, and yet it says f16. The default.
I checked the launch command too. The options that set KV cache precision are not being passed at all.
Setting compression through an environment variable, in other words, never reached the program that actually runs. No error, no warning. From where you set it, there is no way to tell whether it took.
What saved me here was having a second indicator, the CPU load, that did not move by even one percent. Watching speed alone, I would have missed it.
A by-product: the log confirmed what the spill is
The same log had a line backing up my earlier reasoning.
Of 6.4GB of KV cache, 2.4GB is sitting on the CPU side. I had written before that the KV cache was the main thing spilling over; now it is confirmed in a log rather than inferred.
Something else: past a point it stops falling
This time I pushed the context to 131,072 tokens. Previously I had only measured to 32,768.
Looking at the results, it stops falling somewhere past 32,768. It flattens out at around 8 tokens per second and stays there at 131,072. The CPU load also stops at 37%.
I had written that “the longer the context, the slower it gets." More precisely, it bottoms out around 32,768 and then holds level. Still slow, but it does not degrade without limit as you stretch it.
In summary: the conclusion is that I could not test it
- Whether compressing the KV cache lets you handle long contexts: no answer
- Because the compression setting never reached the actual program (Ollama 0.30.5, with no error and no warning)
- I caught it because the CPU load did not move by one percent. Watching speed alone would have missed it
- As a by-product, the log confirmed the KV cache is what spills over (2.4GB of 6.4GB on the CPU side)
- Context bottoms out around 32,768 and holds level after that
Next I will bypass Ollama and run llama.cpp directly. There the precision can be set explicitly, so the effect of compression itself should finally be measurable. I will write it up when I have results.
Hardware used
As an Amazon Associate we earn from qualifying purchases.
Sources
Checked 29 July 2026. Behaviour changes with versions.