Does a New llama.cpp Feature Speed Up AI Models Too Big for VRAM? Measuring GLM-5.3-Flash on an RTX 5060 Ti (16GB VRAM)
When you run a large model that does not fit in your GPU, a Mixture of Experts (MoE) model lets you keep the weights of its “experts" in RAM and compute them on the CPU. In MoE models, only a few of the “experts" inside the model work on each input. In llama.cpp this is the -cmoe (–cpu-moe) option. In October 2026, llama.cpp gained an “MoE cache" (–moe-cache-mib) that copies the most-used experts from RAM into VRAM and keeps them there. I measured on a real machine how much the writing speed changes when part of the VRAM is given to this cache.
As of 28 October 2026.
What I measured
I ran GLM-5.3-Flash (an MoE model, 93GB) with -cmoe on an RTX 5060 Ti (16GB VRAM) connected by OCuLink to a mini PC (EVO-X2). I changed the MoE cache size and measured the writing speed [tok/s], the input processing time [ms] and the VRAM usage [MiB]. The state without the cache is the baseline, and I look at the difference when the cache is on.
The writing speed is the number of tokens generated per second [tok/s] (a token is a unit into which text is split). Larger means faster writing.
I did not evaluate the quality of the answers (how good the text is).
Purpose
I check three things. First, what writing speed [tok/s] results when the cache size is changed to 2,200 / 4,400 / 6,600 / 8,800 MiB. Second, how much the VRAM grows by the amount given to the cache. Third, whether the input processing time changes with the cache.
Method
What I used
| Item | Detail |
|---|---|
| Machine | EVO-X2 (mini PC). Main memory is about 122GiB |
| GPU | NVIDIA GeForce RTX 5060 Ti (16GB VRAM). Connected by OCuLink (PCIe Gen4 x4). Measured with only this GPU visible (CUDA_VISIBLE_DEVICES=1) |
| Software | llama.cpp b11946 (official CUDA 12.8 build). I used llama-server |
| Model | GLM-5.3-Flash (GGUF distributed by unsloth, quantization UD-IQ1_S, split into 3 files, about 93GB in total) |
| Launch options | -ngl 99 -cmoe -fa on -c 4096 -t 16 –no-warmup. With the cache, the size [MiB] is given to –moe-cache-mib |
-cmoe keeps the expert weights in RAM and computes them on the CPU. –moe-cache-mib sets how much of them [MiB] is copied into VRAM. b11946 includes a change in how the cache swaps entries, from LRU (discard the entry unused for the longest time) to a method called 50/50 SLRU.
The model weights differed in total bytes from the ones used in an article on different conditions published on 23 October (93,077,724,536 bytes in the 23 October article, 93,087,245,088 bytes as shown by the distributor this time, a difference of about 9.5MB). The distributor may have updated the files. I confirmed that the weights used this time match the hash value (sha256) shown by the distributor.
How I measured
| Item | Setting |
|---|---|
| Input and output | 473 input tokens (the first sentence changed every time to avoid reusing the previous computation), 128 generated tokens, temperature 0 |
| Number of runs | 6 runs each. The first run is treated separately because it is measured right after launch. Of the remaining 5, the median of the 3 left after dropping the fastest and the slowest |
| Writing speed [tok/s] | predicted_per_second in the timings returned by llama-server |
| Input processing time [ms] | prompt_ms in the same timings (the time to process the 473 input tokens) |
| VRAM [MiB] | Maximum usage reported by nvidia-smi (this GPU only) |
| Transfer record | During the measurement with a 4,400 MiB cache, I recorded the PCIe receive rate [MB/s] and the GPU utilization [%] of the GPU 15 times with nvidia-smi dmon |
Results
| –moe-cache-mib [MiB] | Writing speed, median of the middle 3 [tok/s] | Ratio to baseline | First-run writing speed [tok/s] | Input processing time, median of the middle 3 [ms] | Max VRAM [MiB] |
|---|---|---|---|---|---|
| None (baseline) | 17.846 | 1.00 | 17.354 | 11,128 | 8,487 |
| 2,200 | 2.774 | about 0.16 | 2.768 | 11,701 | 10,663 |
| 4,400 | 3.943 | about 0.22 | 3.953 | 11,342 | 12,863 |
| 6,600 | 4.428 | about 0.25 | 4.438 | 10,969 | 15,063 |
| 8,800 | Failed at launch because GPU memory (the CUDA0 buffer) could not be allocated (OOM). Could not be measured | ||||
Without the cache, the writing speed was 17.846 tok/s. With the cache, it was 2.774 tok/s at 2,200 MiB, 3.943 tok/s at 4,400 MiB and 4.428 tok/s at 6,600 MiB, all slower than the baseline. The larger the cache, the faster it tends to be, but it recovered only to about a quarter of the baseline.
I re-measured 4,400 MiB to be sure (3 runs, median). The result was 3.955 tok/s, the same level as the first measurement.
The input processing time was about 11,000 ms (about 11 seconds) under all four conditions, and no difference due to the cache was visible. Under these conditions, the input processing (473 tokens) was not affected by the cache.
Even without the cache, the VRAM usage reached 8,487 MiB at most. This is probably because, even with -cmoe, the weights other than the experts, such as attention (the layers that read the context), and the computation buffers are placed on the GPU. With the cache, the VRAM grew by an amount close to the specified size (about 2,180 MiB more with 2,200 MiB specified, about 4,380 MiB more with 4,400 MiB, about 6,580 MiB more with 6,600 MiB). With 8,800 MiB, adding it to 8,487 MiB exceeds the 16GB of VRAM, so the launch probably failed for that reason. I have not confirmed the exact cause of the failure.
Discussion
Why was it slower under these conditions?
I cannot pin down the cause from this measurement alone. What I could observe is the following. During the measurement with a 4,400 MiB cache, the PCIe receive rate of the GPU was 4,293 to 7,761 MB/s over 15 records. The link was Gen4 x4, and the GPU utilization was 94 to 100%. The theoretical one-way limit of PCIe Gen4 x4 is about 7.9GB/s. Many records came close to this limit (12 of the 15 were at or above 6,280 MB/s).
From this observation, the transfer of experts that are not in the cache from RAM to VRAM may be capped by the OCuLink link. In the baseline (no cache), the CPU reads the experts directly from memory to compute, so this transfer does not occur. However, I have not confirmed the share of requests served from the cache (the hit rate), because it does not appear in the log. I have not confirmed whether the transfer is the cause.
If the cap on transfer is the main cause, the difference should shrink with a connection with more bandwidth (PCIe x16). I have not checked this.
Cache size and writing speed
As the cache grew, the writing speed rose as 2.774 → 3.943 → 4.428 tok/s. The increase was about 1.2 tok/s from 2,200 MiB to 4,400 MiB, and about 0.5 tok/s from 4,400 MiB to 6,600 MiB. With only these three points, I cannot tell what happens when the size grows further (8,800 MiB failed).
Room in VRAM
Even on a 16GB GPU, 8,487 MiB is used without the cache, so the largest amount I confirmed could be given to the cache under these conditions was 6,600 MiB (max VRAM 15,063 MiB). I did not measure the values in between. On a GPU with less VRAM, the amount that can be given is even smaller.
What I did not check this time
I did not check the following in this measurement.
The connection was only OCuLink (PCIe Gen4 x4). There is no result from a machine with the GPU plugged directly into PCIe x16.
The model was only GLM-5.3-Flash at UD-IQ1_S (a quantization of about 1.56 bits). I did not measure other MoE models or other quantizations. I did not separate whether the small quantization affected the result.
The GPU was only the RTX 5060 Ti. I did not measure the behavior of the RTX 3060 (12GB VRAM) in the same machine, or of the integrated GPU.
I did not check the cache hit rate. For the launch failure at 8,800 MiB, I only looked at the end of the log (the memory allocation failure) and did not investigate the cause in detail.
The input was a single kind, 473 tokens in and 128 tokens out. I did not measure behavior with a long input or a long output. I did not evaluate the quality of the answers.
Summary
| What I looked at | Result (RTX 5060 Ti 16GB, OCuLink, GLM-5.3-Flash UD-IQ1_S, -cmoe) |
|---|---|
| Writing speed [tok/s] | No cache 17.846 / 2,200 MiB 2.774 / 4,400 MiB 3.943 / 6,600 MiB 4.428 |
| Input processing time [ms] | About 11,000 ms. No difference visible with or without the cache |
| Max VRAM [MiB] | No cache 8,487 / 2,200 MiB 10,663 / 4,400 MiB 12,863 / 6,600 MiB 15,063. Launch failed at 8,800 MiB |
| Transfer observation (4,400 MiB) | PCIe receive 4,293 to 7,761 MB/s, Gen4 x4, GPU utilization 94 to 100%. The transfer may be capping the speed (hit rate not confirmed) |
Under these conditions, turning on the MoE cache lowered the writing speed to about 0.16 to 0.25 times that without the cache. The input processing time did not change, and the VRAM grew by an amount close to the specified size (about 2,180 to 6,580 MiB).
When you try it on your own machine
The numbers here speak only for these conditions. The table below lines up what goes up and what limits come with turning the cache on.
| What to look at | Result this time |
|---|---|
| Writing speed | Under these conditions it went down by the amount of cache added (17.846 → 2.774 to 4.428 tok/s). The larger the size, the smaller the drop |
| Input processing time | No change (about 11,000 ms) |
| VRAM | Grows by the specified size. 8,487 MiB is used even without the cache |
| Connection bandwidth | On PCIe Gen4 x4, the receive rate rose close to the limit. A machine with x16 was not measured |
To try it on your own machine, measure once with -cmoe alone, then add –moe-cache-mib and compare the writing speed with the same input. A different connection (x4 or x16) or a different model may reverse the result.
Reference: figures reported by the proposer (different conditions from the main tables; I have not reproduced them)
The proposal for this feature (llama.cpp Pull Request #29887) contains speeds reported by the proposer. Because the conditions differ from mine, I do not line them up with the values in my tables, and they are not values I reproduced.
| Item | Reported by the proposer | Difference from my measurement |
|---|---|---|
| GPU | RTX 4090 and RTX 5090 | Mine is an RTX 5060 Ti (16GB VRAM) |
| Connection | PCIe 4.0 x16 | Mine is OCuLink (PCIe Gen4 x4) |
| Model | Qwen3.8-Flash-Next (Q4_0) | Mine is GLM-5.3-Flash (UD-IQ1_S) |
| Writing speed | 25.0 → 39.4 tok/s (about 1.57 times, 4090, 6.4GB cache) / 30.8 → 54.5 tok/s (about 1.77 times, 5090) | Mine went down with the cache |
These values are copied as written in the PR. Because the conditions differ this much, it cannot be said that they contradict my results.