How Fast Is Nemotron 3.5 Bolt, Compressed for 24GB of VRAM, on an RTX 3090? Measured with llama.cpp v0.7.0

October 26, 2026

“Nemotron 3.5 Bolt 74B-A9B," published by NVIDIA (74 billion parameters in total, 9 billion used at a time), is an image- and video-capable model built to fit in 24GB of VRAM. I ran this model with llama.cpp v0.7.0 on an Ubuntu machine with an RTX 3090 (24GB of VRAM) and measured its speed. This is a model that the older llama.cpp and ollama could not load. Will the newer version load it?

This reflects the state of things as of October 17, 2026.

Purpose: what this article measured

This article measured the following three things.

• Whether it loads: whether v0.7.0 can load a model that the older llama.cpp (b10941 and b11192) and ollama 0.30.5 could not
• Input-processing speed (pp512): how many tokens per second it processes when fed a 512-token input (the speed of reading)
• Generation speed (tg128): how many tokens per second come out when it writes 128 tokens of text (the speed of writing)

I also recorded the usage of the GPU’s memory (VRAM). I did not measure the quality or accuracy of the answers, the behavior with image input, or the behavior with long inputs. Details are in “Limitations: what I did not measure."

The goal is that, after reading, you have a rule of thumb: “a file of about 20.5GiB (gibibytes) fits entirely, all layers, on an RTX 3090 with 24GB of VRAM and writes at about 60 tokens per second."

Background: terms explained

Several technical terms follow in a row, so let me sort them out first.

• Token: the unit an AI uses when processing text. It does not match the character count; a word or a fragment of a word counts as one token.
• tok/s (tokens per second): the number of tokens processed per second. Larger is faster, and all speeds in this article use this unit.
• llama.cpp: software for running AI locally. It is separate software from ollama, and I used v0.7.0 (build 620d222) this time.
• llama-bench: a measurement tool that comes with llama.cpp. It measures the processing of a 512-token input (pp512) and the generation of 128 tokens (tg128).
• Vulkan: one of the mechanisms (APIs) for using a GPU. I ran llama.cpp through Vulkan this time.
• VRAM: the dedicated memory on a GPU. If the whole model fits here, the GPU alone can run it.
• Quantization: a method of making a model’s numbers coarser to shrink the file. The IQ2_XXS used here is a format that holds each value in about 2.06 bits.
• MoE (Mixture of Experts): a design that holds many small parts called “experts" inside the model and uses only some of them for each token. Even if the total parameter count is large, the computation per token is only for the active parameters.
• ollama: another piece of software for running AI locally. It is separate from llama.cpp, so I do not compare speed numbers in the same table.

What Nemotron 3.5 Bolt is: history and design trivia

Before the measurement, let me sort out where this model came from. This part was checked against public information from Hugging Face (a site that distributes AI models) and NVIDIA, and the sources are collected at the end of the article.

The Nemotron lineage and where Bolt sits

The publisher is NVIDIA. The Release Date in the README is October 15, 2026. The repository creation dates on Hugging Face are October 7 for the 24GB version and its GGUF version and October 9 for the BF16 version, earlier than the release date. I could not confirm why the creation dates are earlier (whether there was a private period).

Nemotron is the name of a series of NVIDIA models. The origin of the name is not written in the materials I checked, so it is unconfirmed. The origin of “Bolt" is also not explained in the README or the technical blog, so I could not confirm it.

Following the creation dates on Hugging Face, the series has progressed as follows.

Model	Repository creation date	Note
Nemotron 3 Super 120B-A12B	March 10, 2026	This blog has an article measuring it with ollama
Nemotron 3 Ultra 550B-A55B	June 3, 2026	Has LatentMoE and an MTP layer
Nemotron 3.5 Lightning 30B-A3B	August 1, 2026	A mix of Mamba-2, MoE, and Attention
Nemotron 3.5 Super VL 120B-A12B	October 12, 2026	The parent model of Bolt
Nemotron 3.5 Bolt 74B-A9B	October 7, 2026 (24GB version)	A derivative that compresses Super VL

The Model Design field in the README says it was “compressed from Nemotron 3.5 Super VL 120B-A12B." Bolt is not a newly pretrained model but a derivative that makes Super VL smaller. The article from Artificial Analysis (AA, a site that compares AI models from a third-party standpoint) calls Bolt “the first model in NVIDIA’s new Bolt family." This is a third-party statement, and I could not confirm whether NVIDIA itself declares it a family.

On the license, the previous generation, Nemotron 3 Super, used the “NVIDIA Nemotron Open Model License," whereas the 3.5 generation uses OpenMDW License 1.1. The README says it can be used both commercially and non-commercially. I only checked a summary of the license text and did not read the full text.

How to read the name: 74B-A9B and 24GB

“74B-A9B" means 74B (74 billion) parameters in total, of which 9B (9 billion) are used per token. The share used is about 12%. The “24GB" in the name means the distributed files were designed to fit in 24GB of memory. The README says “native 2-bit quantization to fit in 24GB of memory."

How it was compressed: a different number of experts per layer

According to the README, the compression was done in five stages. The first stage is Star Elastic, a compression method devised by NVIDIA. Using the router (the part that chooses which experts to use) as a clue, it searches while varying the number of experts and the intermediate width for each layer. The target was “under 24GB at batch size 1 with a 128K-token context."

Counting in the configuration file (config.json), the parent Super VL has 512 experts in every MoE layer and uses 22 per token. In Bolt, the number of experts differs from one of the 40 MoE layers to another: 240 in 12 layers, 304 in 6 layers, 368 in 9 layers, 432 in 10 layers, and 496 in 3 layers. The number used per token is reduced to 9. This tally is a value this blog counted from the config.

The layer structure is the same 88 layers as the parent: 40 Mamba-2, 40 MoE, and 8 Attention. MTP (a layer that predicts ahead at the same time) is not in Bolt’s configuration, while the parent has it.

This design, “a different number of experts per layer," was the cause of the loading error we will see later.

Quantization and speculative decoding

In the last stage of compression, the experts are quantized to around 2 bits. According to the README, all experts (both the routed ones and the shared one) are IQ2_XXS (2.0625 bits per value). To recover accuracy it uses QAD (quantization-aware distillation, a distillation that builds quantization into the training), and the tool is written as ModelOpt.

The distribution also includes a drafter for speculative decoding (a method in which a small model guesses the continuation ahead of time and the main model checks it, making generation faster). Its name is LiLiCorr, and the origin is unconfirmed. The README says it uses a method called DFlash, not MTP. I did not use the drafter in this measurement.

The performance NVIDIA shows, and AA’s value

The Benchmarks table on the model card carries NVIDIA’s own evaluation. The values for the IQ2 version of Bolt are below. The comparison targets are other models in BF16 (not quantized), such as Qwen3.6-35B-A3B.

Metric	Bolt IQ2	Qwen3.6-35B-A3B	Gemma4 26B-A4B	Lightning 30B-A3B
GPQA Diamond	86.43	83.40	79.61	75.57
Terminal-Bench 2.1	50.98	44.38	37.22	23.46
HLE (without tools)	25.30	19.56	17.42	10.47

The values in the table are NVIDIA’s own evaluations, not third-party verification. There are also rows where Bolt is not above the others.

The AA Intelligence Index for Bolt was 21.4 (the value printed on a bar in a graph in the AA article; Intelligence Index v4.3.2). In the same article, the parent Super VL is 26, and Ultra, from the same Nemotron 3 series, is 23. AA writes that Bolt’s output is about 82k (82 thousand) tokens per task, more than Super VL’s roughly 46k. All of these are numbers published by AA, not measurements by this blog.

Method: how I measured

The machine and software I measured

The machine is a PC running Ubuntu, and I used its RTX 3090 (24GB of VRAM; the capacity is per spec). The software is llama.cpp v0.7.0 (build 620d222), which I built from source on this machine. I used the GPU through Vulkan and set the environment variable GGML_VK_VISIBLE_DEVICES so that only the RTX 3090 is visible.

Measurement conditions

Item	Details
Model	Nemotron-3.5-Bolt-74B-A9B-24GB.gguf, the official one from nvidia (22,018,326,528 bytes = 20.50GiB. IQ2_XXS, 74.04B)
Parts for images	No mmproj (text only)
Command	llama-bench -ngl 99 -p 512 -n 128 -r 6
-ngl 99	A setting that places all layers on the GPU
Runs	6
Aggregation	Sort the 6 values from smallest to largest, drop the maximum and minimum, and use the median of the remaining 4 points (the spread is the difference between the maximum and minimum of those 4 points)

I tried the same procedure with the older versions too. On October 16, 2026, I tried loading with llama.cpp builds b10941 and b11192 (both Vulkan), and also with ollama 0.30.5. On October 17, I measured again with v0.7.0.

Why I do not put ollama numbers in the same table

ollama is separate software from llama.cpp, with a different loading implementation and a different way of measuring. The numbers this time are llama-bench values, so I do not compare them in the same table with the numbers from this blog’s articles measured with ollama.

Results

The older versions could not load it

With b10941 and b11192, the following errors appeared partway through loading.

• llama.cpp b10941 and b11192: check_tensor_dims: tensor 'blk.1.ffn_gate_inp.weight' has wrong shape; expected 4096,496 got 4096,240
• ollama 0.30.5: a type error because expert_used_count is an array (a different reason from llama.cpp’s)

The first error is a mismatch: “the first MoE layer in the file has 240 experts, but the reader assumes 496." The design with a different number of experts per layer is the Bolt feature we saw in the previous section. The older versions assume the same number in every layer, so they could not load it. With v0.7.0, the same file loaded. I did not check the source, so I do not know what changed in v0.7.0.

Speed

Run	Input processing pp512 [tok/s]	Generation tg128 [tok/s]
1	1188.76	59.57
2	1190.35	59.70
3	1193.26	59.75
4	1195.41	59.82
5	1195.74	59.98
6	1197.60	60.30
Median	1194.34 (spread 5.39)	59.79 (spread 0.28)

For input processing, the 6 runs fell within 1188.76 to 1197.60 tok/s, and the median was 1194.34 tok/s. At that rate, a 512-token input takes about 0.43 seconds to read. Generation ranged from 59.57 to 60.30 tok/s, and the median was 59.79 tok/s. At that rate, 128 tokens take about 2.1 seconds to write. The values in the table are sorted from smallest to largest, not in the order they were measured.

VRAM

The VRAM usage seen with nvidia-smi peaked at about 21,400MiB (about 20.9GiB) during the measurement. At idle it was 450MiB. With the setting that places all layers on the GPU (-ngl 99), this is close to the file size of 20.5GiB. It fit within 24GB.

During the measurement, the temperature peaked at 72°C and the power consumption at about 328W. However, the sampling was sparse, and it is not a continuous record.

Discussion: what the results suggest

The generation speed of about 60 tokens per second is a value from one condition measured 6 times. I did not measure it with long inputs or with several questions used at once.

On how the model fit, there are two things I can say as fact. First, the 20.5GiB file ran with all layers on the GPU. Second, VRAM use peaked at about 20.9GiB, within the range of 24GB. The “24GB" in the model name can be read as having worked as the design target intended. I did not measure how VRAM grows when the context length is increased, so I cannot say whether there is headroom in the 24GB.

Here an explanation comes to mind: “only 9B are used at a time, so it can write fast." The share used is about 12% (9B ÷ 74B), so the computation per token is small. However, this is a guess, and I did not verify it. Unless models with different active shares are lined up under the same conditions, the reason cannot be pinned down.

The measurement of the predecessor, Nemotron 3 Super, is in another article, and another model run on the same RTX 3090 is in the Qwen3.8 27B article. Because the software and machine conditions differ, I do not line up the speed numbers for comparison.

The reason the older versions could not load it can be read from the error message as “a mismatch in the number of experts." I could not confirm why v0.7.0 could load it.

Limitations: what I did not measure

• Answer quality and accuracy: I did not measure how well the text is written, the rate of correct answers, or how often it hallucinates. Quality is unmeasured. Speed and the quality of answers are separate matters.
• Image and video input: this model is said to accept images and video too, but I ran it with text only, without attaching mmproj. Image input is untested.
• Values for a single question only: these are values from one llama-bench condition (512 input tokens, 128 generated tokens) measured 6 times. I did not measure behavior with long inputs, long generation, or several questions used at once.
• Speculative decoding: I did not use the bundled drafter (LiLiCorr). The speed with it is unmeasured.
• Temperature and power consumption: the 72°C maximum and about 328W are reference values from sparse sampling. They are not a continuous record.
• Comparison with ollama numbers: ollama is separate software. I did not compare against numbers measured with ollama in the same table.
• Differences in quantization: only the official IQ2_XXS. I did not measure the differences from other quantized versions.
• Day of measurement: a single day, October 17, 2026, this one machine, and one version of v0.7.0 only. On another version or another day, the values may move.
• The Intelligence Index of 21.4: a value published by Artificial Analysis, printed on a bar (Intelligence Index v4.3.2). This blog did not verify it. The table on the model card is also NVIDIA’s own evaluation.

Summary: results of running Nemotron 3.5 Bolt on an RTX 3090

• The IQ2_XXS (20.5GiB) of Nemotron 3.5 Bolt 74B-A9B ran with all layers on the RTX 3090 using llama.cpp v0.7.0.
• The older llama.cpp (b10941 and b11192) and ollama 0.30.5 could not load it. The llama.cpp error can be read as caused by the number of experts differing from layer to layer.
• Input processing was 1194.34 tok/s (pp512) and generation was 59.79 tok/s (tg128). Both are medians of 6 runs.
• VRAM use peaked at about 21.4GB, within the range of 24GB.

If you want to try it, first update llama.cpp to v0.7.0 or later and measure the values on your own machine with llama-bench. The values here are only a record of a single run on one machine.

This article did not measure quality or image input. I will keep following them in this blog’s local LLM articles.

References

• Nemotron 3.5 Bolt 24GB GGUF (README): https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF
• Nemotron 3.5 Bolt technical blog (NVIDIA): https://research.nvidia.com/labs/nemotron/Nemotron-3.5-Bolt-24GB/
• Nemotron 3.5 Bolt repository information (creation date): https://huggingface.co/api/models/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF
• Nemotron 3.5 Bolt BF16 (used to recount the configuration file): https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-BF16
• Nemotron 3.5 Super VL: https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16
• OpenMDW License 1.1: https://github.com/OpenMDW/OpenMDW/blob/main/1.1/LICENSE.OpenMDW-1.1
• Artificial Analysis article (Nemotron 3.5 Super and Bolt): https://artificialanalysis.ai/articles/nemotron-3-5-super-and-bolt
• The information above was checked on October 17, 2026.