Nemotron 3 Super Keeps Most of Its Weights in RAM: How Many Tokens per Second Do Two GPUs Give?

2026年10月25日

I ran NVIDIA’s large model “Nemotron 3 Super" (120 billion parameters in total, 12 billion used at a time) on the mini PC “EVO-X2" (a mini PC with a Ryzen AI Max+ 395 and 128GB of memory, of which the OS sees about 122GiB) and measured how fast the answers come out. Two cards are attached to this mini PC: an RTX 5060 Ti (16GB VRAM) and an RTX 3060 (12GB VRAM, per spec). Can a file that is still 87GB after quantization run on 28GB of VRAM in total?

This reflects the state of things as of October 16, 2026.

Purpose: what this article measured

This article measured the following two things.

• Generation speed: I repeated the same question and measured how many tokens per second the answer text comes out at (the speed of writing, not of reading).
• Where the model landed: how the model’s weights were split between the two GPUs and the main memory (host RAM), checked in the ollama server log.

There are also things I did not measure. The quality and accuracy of the answers, behavior with long inputs, simultaneous use by several people, and changing the number of GPUs are outside the scope of this article. Details are in “Limitations: what I did not measure."

The goal is that, after reading, you have a rule of thumb: “an 87GB model does not fit in 28GB of VRAM, but it runs with most of its weights placed in main memory, and it writes at around 20 tokens per second."

Background: terms explained

Several technical terms follow in a row, so let me sort them out first.

• Token: the unit an AI uses when processing text. It does not match the character count; a word or a fragment of a word counts as one token.
• tok/s (tokens per second): the number of tokens processed per second. Larger is faster, and all speeds in this article use this unit.
• VRAM: the dedicated memory on a GPU. If the whole model fits here, the GPU alone can run it. What does not fit can be placed in main memory (host RAM) and used from there.
• Quantization: a method of making a model’s numbers coarser to shrink the file. The Q4_K_M used here is a format shrunk to roughly 4 bits, and the file is 87GB.
• MoE (Mixture of Experts): a design that holds many small parts called “experts" inside the model and uses only some of them for each token. Even if the total parameter count is large, the computation per token is only for the active parameters.
• Mamba-2: a type of layer that works differently from a Transformer (the usual attention mechanism). Its memory (KV cache) is said to grow less as the text gets longer.
• ollama: software for running AI locally. I used version 0.32.15 here.
• Offload: assigning layers (the stages that make up a model) to the GPU side. “89 of 89 layers offloaded" means the GPU side is in charge of the computation for all layers.

What Nemotron 3 Super is: history and design trivia

Before the measurement, let me sort out where this model came from. Knowing the background of the numbers changes how you read the results. This part was checked against public information on Hugging Face (a site that distributes AI models), and the sources are collected at the end of the article.

The Nemotron lineage published by NVIDIA

Nemotron 3 Super is published by NVIDIA. Its repository on Hugging Face was created on March 10, 2026. The Release Date in the README is March 11, one day off. The creation date of a repository does not always match the release date, so treat it as a rough guide to the release timing.

Nemotron is the name of a series of NVIDIA models. The origin of the name is not written in the materials I checked, so it is unconfirmed.

Following the creation dates on Hugging Face, the series has progressed as follows.

Model	Repository creation date	Note
Nemotron-4-340B-Instruct	June 13, 2024	The README says it was also used to synthesize training data for this model
Llama-3.1-Nemotron-70B-Instruct-HF	October 12, 2024	The name contains Llama-3.1 (I have not checked whether it is based on it)
Nemotron-H-8B / 56B (Base)	March 19 and April 8, 2025	A series with H in the name (this model’s config.json has model_type nemotron_h; I have not checked how the two are related)
NVIDIA-Nemotron-3-Nano-30B-A3B	December 4, 2025	30B in total, 3.5B active (README)
NVIDIA-Nemotron-3-Super-120B-A12B	March 10, 2026	The model in this article

In the Nemotron 3 generation, Nano came in December 2025 and Super in March 2026. The differences from Nano that the README lists are that Super has an MTP layer and was trained in NVFP4.

How to read the name: 120B-A12B

In the model name “120B-A12B," the 120B means 120 billion parameters in total, and the A12B means 12B (12 billion) of them are used per token. The share used is about 10%. This notation is common for Qwen and others too. Its predecessor, Nemotron 3 Nano, is 30B-A3B, and the README gives its active size as 3.5B.

What matters for this measurement is that even though only 12B is used, the weights that must be kept on hand cover all 120B.

The internal design: Mamba-2, MoE, and a few attention layers

The README describes Super as “a LatentMoE hybrid that alternates Mamba-2 and MoE layers, with Attention layers inserted here and there." I have not checked the details of how LatentMoE works. The configuration file has an item moe_latent_size=1024.

The configuration file (config.json) has a string of symbols that shows the order of the layers. Counting the symbols gives 88 layers in total: 40 are M, 40 are E, and 8 are *. Reading M as Mamba-2, E as MoE, and * as Attention matches the README’s description. I could not find an official legend for these symbols, so this reading is this blog’s own.

The same configuration file lists 512 experts, 22 of them used per token, and 1 shared expert that everyone uses. Regular attention is in only 8 of the 88 layers, and the number of KV heads is 2. This reads as a design in which memory grows less as the text gets longer, but I did not measure that effect this time.

For context length, the README says “up to 1 million tokens," while the maximum position in the configuration file is 262,144 tokens. I could not confirm the reason for the mismatch. The README also mentions that it has a Multi-Token Prediction (MTP) layer, which predicts ahead at the same time, and that it was trained in 4-bit NVFP4. The license is the NVIDIA Nemotron Open Model License, and I have not checked the terms.

On the Intelligence Index from Artificial Analysis (AA, a site that compares AI models from a third-party standpoint), Nemotron 3 Super scored 13 (the value printed on the bar in a graph in an X post; Intelligence Index v4.3.2). This is a number published by AA, not a measurement by this blog.

Method: how I measured

The machine I measured

The machine is the EVO-X2. It is a mini PC with a Ryzen AI Max+ 395, and of its 128GB of memory, the OS sees about 122GiB. An RTX 5060 Ti (16GB VRAM) and an RTX 3060 (12GB VRAM, per spec) are attached to it. It also has an integrated Radeon 8060S GPU, but it was not used this time (checked in the server log).

Measurement conditions

Item	Details
Model	nemotron-3-super:120b-a12b (Q4_K_M, 87GB file)
Software	ollama 0.32.15
Memory handling	use_mmap:false (a setting that loads the file into real memory up front, instead of reading only the parts it needs)
Thinking mode	Not specified (left at the default)
Generation length	num_predict 256 (stop at a maximum of 256 tokens)
Runs	The same question 6 times. The first run includes loading the model, so I excluded it and used the remaining 5
Aggregation	Sort the 5 runs from smallest to largest and use the median of the middle 3

Where the model landed was checked in the ollama server log (obtained with journalctl).

Why I do not use the input-processing numbers

When the same question is repeated, ollama reuses the cache for the beginning part, and the input-processing speed comes out much larger than it really is. Because the input-processing numbers are overstated in this measurement, I do not include them in the article. What I include is the generation speed and where the model landed.

There is one more caution. The GPU numbers in ollama and in nvidia-smi were reversed. In the ollama log, CUDA0 is the RTX 5060 Ti and CUDA1 is the RTX 3060, but in nvidia-smi the RTX 3060 is number 0 in PCI order. In the text, I refer to the GPUs by name, not by number.

Results

Generation speed

Run	Generation [tok/s]
2nd	20.31
3rd	20.32
4th	20.33
5th	20.35
6th	20.38
Median of the middle 3	20.33 (spread 0.03)

The values of the 5 runs ranged from 20.31 to 20.38 tok/s and were almost the same. The median of the middle three points (20.32, 20.33, and 20.35 tok/s) is 20.33 tok/s, with a spread of 0.03 tok/s. At that rate, 256 tokens take about 12.6 seconds to write.

Where the model landed

In ollama ps (the list of models ollama has loaded), the model size (size) was 91.4GB, of which the part on VRAM (size_vram) was 24.5GB. This differs from the breakdown in the server log that I look at later (17,545MiB for the two GPUs together, about 18.4GB). I have not confirmed why the numbers differ between ollama’s display and the log breakdown. In the server log, the placement was as follows.

Location	Model area [MiB]
RTX 5060 Ti (16GB VRAM)	9,997
RTX 3060 (12GB VRAM)	7,548
Host RAM (CUDA_Host)	63,493
Integrated GPU (Radeon 8060S)	Not used

In the layer assignment, 89 of 89 layers were offloaded to the GPU side. The model area on the two GPUs added together is 17,545MiB and the host RAM holds 63,493MiB, for a total of 81,038MiB. By calculation, about 78% of the weights (63,493 ÷ 81,038) were placed in host RAM.

Discussion: how 87GB ran on 28GB of VRAM

The speed of about 20 tokens per second is the value when only one question was sent at a time. I did not measure what happens when several questions are handled at once.

The way the model landed this time was “most of the weights are in host RAM, and the GPUs hold a part of them." The 87GB file does not fit in the 28GB of total VRAM. Even so, all 89 layers were assigned to the GPU side, and it ran. It seems ollama decided where to put each layer and placed what did not fit in VRAM into host RAM.

Here an explanation comes to mind: “MoE does little computation per token, so it runs fast even with weights in main memory." Only about 10% (12B ÷ 120B) is used per token, so fewer weights are read. This is a guess, and I did not verify it this time. If the explanation is right, speed should drop as the share of experts used increases.

The GPU assignment is a fact confirmed in the log. The integrated Radeon 8060S was not used, and the model ran on the two NVIDIA cards and host RAM. This is probably because ollama detected the NVIDIA GPUs and chose them, but I did not confirm why they were chosen.

Limitations: what I did not measure

• Answer quality and accuracy: I did not measure how well the text is written, the rate of correct answers, or how often it hallucinates. Speed and the quality of answers are separate matters.
• Long inputs: I did not measure with varying input lengths. How it behaves with the context length said to go up to 1 million tokens is unconfirmed.
• Input-processing speed: I did not include it because the cache makes it overstated.
• Simultaneous use: I sent questions one at a time and did not measure several at once.
• Changing the number of GPUs: only the two-GPU configuration. I did not measure one GPU only, no GPU, or the integrated GPU only.
• Switching thinking mode and MTP: thinking mode was left at the unspecified default and was not switched. Whether ollama used MTP is unconfirmed.
• Differences in quantization: only Q4_K_M. I did not measure the differences from other formats.
• Power consumption and temperature: not recorded.
• Number of measurements: one condition, with 5 of 6 runs used, on a single measurement day, October 16, 2026. On another day or with a different version of ollama, the values may move.
• The Intelligence Index of 13: a number published by Artificial Analysis, not verified by this blog.

Summary: results of running Nemotron 3 Super on the EVO-X2

• The Q4_K_M (87GB) of Nemotron 3 Super (120B total, 12B active) ran on two cards, an RTX 5060 Ti and an RTX 3060 (28GB of VRAM in total), plus host RAM.
• The generation speed was 20.33 tok/s as the median of the middle 3 of 5 runs (spread 0.03 tok/s, up to 256 tokens generated).
• For the placement, in the model area the RTX 5060 Ti held 9,997MiB, the RTX 3060 held 7,548MiB, and host RAM held 63,493MiB. About 78% of the weights were in host RAM, and all 89 layers were assigned to the GPU side.
• The integrated Radeon 8060S was not used.

If you want to try it, first send just one question and run ollama ps on your own machine. The gap between size and size_vram is a rough guide to the part placed in host RAM.

This article did not measure quality or behavior with long inputs. I will keep following, in this blog’s local LLM articles, how to run large MoE models on machines with limited memory.

References

• Nemotron 3 Super model card (README and config.json): https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16
• Nemotron 3 Super repository information (creation date and license notation): https://huggingface.co/api/models/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16
• NVIDIA Nemotron Open Model License: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/
• Nemotron 3 Nano 30B-A3B: https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16
• Nemotron-4-340B-Instruct: https://huggingface.co/nvidia/Nemotron-4-340B-Instruct / Llama-3.1-Nemotron-70B-Instruct-HF: https://huggingface.co/nvidia/Llama-3.1-Nemotron-70B-Instruct-HF
• Nemotron-H-8B-Base-8K: https://huggingface.co/nvidia/Nemotron-H-8B-Base-8K / Nemotron-H-56B-Base-8K: https://huggingface.co/nvidia/Nemotron-H-56B-Base-8K
• Llama-3_1-Nemotron-Ultra-253B-v1: https://huggingface.co/nvidia/Llama-3_1-Nemotron-Ultra-253B-v1 / NVIDIA-Nemotron-Nano-9B-v2: https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-9B-v2
• Artificial Analysis X post (Intelligence Index graph): https://x.com/artificialanlys/status/2110760441240752399
• The Hugging Face information above was retrieved on October 16, 2026.