What Is Strata? How a 12GB GPU Runs a 125B Model: Mechanism, Speed, and Supported Setups

October 14, 2026

A model too big to fit in GPU memory (VRAM) runs on a 12GB graphics card. That is how Strata is being introduced, and it has become a talking point. What is actually going on, and is it something you can use on your own PC?

In this article I read Strata’s official explanation (the README on GitHub and the documents bundled with it) and sorted out how it works, how fast it is, what it can do, how it became famous, how far it can be customized, which operating systems it supports, and what comes next. I have already run Strata and measured its speed in my earlier Qwen3.8-Flash-Next article, but I did not explain the mechanism there.

This reflects the state of things as of October 12, 2026. Strata is updated almost every day.

Method: what I looked into and where the sources come from

I looked into three things: what Strata does, how fast it is said to be, and what you should know before using it.

• Official information: I read the README of Niko1221/Strata on GitHub, the bundled documents (HOW_IT_WORKS, MODELS, INSTALL, SECURITY, COMMUNITY_BENCHMARKS), the release history, and the Issues.
• Public reaction: I collected the dates from Hacker News (an international tech news site) and from news and explainer articles, as far as I could confirm them.
• My own measurements: I quote the values from my earlier article as they are. I did not run any new measurements this time.

I checked the numbers in the README and the bundled documents against the originals. The author’s paper (Strata-Paper.pdf) and the multi-GPU explanation (MULTI_GPU.md) are not reflected in this article.

What is Strata?

Strata is a local inference engine built only for one large model, Qwen3.8-Flash-Next. Qwen3.8-Flash-Next is a Mixture-of-Experts (MoE) model with 125B (125 billion) parameters in total, and only about 6B of them work when it writes a single token. An MoE model uses only a few of its many “experts" at a time.

It is not a general-purpose inference engine. That is the big difference from llama.cpp and Ollama.

What is the trick? Splitting experts and lookup tables by where they live

According to the README, the model has 24,576 experts, and 10 of them are used to write one token. Strata places them in locations of different speeds.

When reading long text, it also overlaps work: it streams the next layer’s experts to the GPU while computing the current layer. On top of that, the README says speculative decoding makes it 1.6 to 1.8 times faster. This uses the model’s built-in look-ahead layers (MTP) to draft up to 3 tokens and then verify them together.

One note: in my earlier article I wrote the n-gram table as 26.8GB, but that was in GiB. 26.8GiB is about 28.8GB, the same size as the README’s 28.8GB.

How fast is it?

The author’s published writing speeds, on an RTX 5070 (12GB, Ryzen 5 7600, 64GB RAM) and an RX 9070 XT (16GB), are below. The conditions are 4K tokens of output and 32K tokens of input. The README also has rows for IQ3_XXS and a Coder edition, which I left out here.

• Third-party records: COMMUNITY_BENCHMARKS lists 16 measurements (September 30 to October 10, 2026). Some beat the author’s 94, such as 179.4 tok/s on an RTX 5090 (4K output, IQ2_XS). Others fall below it, such as 18.1 to 23.2 tok/s on two Quadro RTX 4000 cards (73K tokens of input). The GPU, version, quantization, and input length are not aligned, so I do not rank them.
• My measurement: on an RTX 5060 Ti 16GB it was 70.7 tok/s (the median of the middle 3 of 5 runs).
• A note on conditions: one developer’s article says the figure of about 100 tok/s applies to code and structured output, where the look-ahead guesses are often right. The same article reports about 22 tok/s for ordinary prose and about 82 tok/s for code on a Strix Halo machine. This is a third-party article, and I have not verified it.

What can it do?

How did it become famous?

The Hacker News score is the value I checked through its API on October 12. I could not confirm who the author is, or any mentions by well-known people.

There are skeptical voices too. One news report quotes a developer and argues that it is unclear how much of the code was written by AI. On Hacker News, a post said that Strata and llama.cpp differ in error on an image task. Both are third-party claims, and I could not find materials to reproduce them.

Can you customize it?

• Startup options: –calibrate tunes it automatically to your PC, –pcie-frac changes how experts are split between GPU and CPU, there is a setting for how often the expert cache is refreshed, a context length setting (8K to 256K), and –vram-elastic lets it coexist with other GPU apps.
• Reasoning effort is changed on the chat screen, and the number of parallel requests in the settings file (or with –parallel in setup).
• Your own model: if it is a GGUF of the same family, you can use your own file with –gguf-dir (–inspect checks whether it is usable). Models of other architectures cannot be used.
• The code is MIT licensed, and 398 pull requests are open.

Which operating systems and GPUs does it run on?

What happens next?

It is updated almost daily. The notes for v0.1.43 (2026-10-11) say that on AMD GPUs, with the cache limited to the equivalent of 16GB, the writing speed of IQ3_XXS improved by up to 38%. I could not find a roadmap document, and I could not confirm any statement from the author about future plans.

In Issue #770, the author answered that a dense 27B model is out of scope and told the asker to use llama.cpp. A question about supporting other MoE models has no answer. llama.cpp itself added support for Qwen3.8-Flash-Next on 2026-08-27 (#27742).

What to know before using it

• It works by having a startup script download a prebuilt engine and run it. setup.py checks the engine against the SHA-256 that GitHub publishes. I could not confirm any code signing.
• By default it listens only on your own PC (127.0.0.1). To open it to the outside you need to pass –host 0.0.0.0, and it warns you if there is no API key. The Docker version listens externally by default.
• The author writes in SECURITY that it has not yet had an external audit.
• The documentation says your PC may freeze for 1 to 3 minutes on the first launch.
• Models live outside the repository and carry their own licenses. Derived models that are not on the menu are not covered by the documentation.

Discussion: what is impressive, and what is limited?

Three things can be said from the published facts.

• It made an MoE model that does not fit in VRAM fast enough to use on a 12GB GPU, by keeping all experts in RAM, putting the frequently used ones in VRAM, and adding the CPU to the computation.
• But it is specific to Qwen3.8-Flash-Next, and the RAM it needs is large, 32 to 64GB.
• Third-party records include cases both above and below the author’s published values, and there is not yet a comparison under matched conditions.

The buzz comes from both the novelty of the mechanism and the way the numbers are presented. This investigation does not judge which is stronger.

What I could not find out

• The contents of the author’s paper and of the multi-GPU document are not reflected in this article.
• I could not confirm code signing, reaction on Reddit or X, or mentions by well-known people.
• I could not find a future roadmap.
• What I verified myself is only the operation and speed in my earlier article. I did not measure quality (how smart the model is).

Summary: Strata is an engine for running a huge MoE model on a 12GB graphics card

Strata is an engine dedicated to Qwen3.8-Flash-Next. It keeps all experts in RAM and puts the frequently used ones in VRAM, so it can run a model that does not fit in VRAM. The author’s published figure is 94 tok/s on an RTX 5070, while 16 third-party records range from 18 to 179 tok/s, depending heavily on conditions.

If you want to try it, first check that your PC has at least 32GB of RAM (64GB for every size) and at least 12GB of VRAM. Because it runs a prebuilt executable, read INSTALL and SECURITY in the README before you decide.

References

• Niko1221/Strata (GitHub)
• HOW_IT_WORKS
• MODELS
• INSTALL
• SECURITY
• COMMUNITY_BENCHMARKS
• Release history
• Hacker News post
• My earlier article: running Qwen3.8-Flash-Next at home