Models Installed With Ollama Won’t Load in llama.cpp? The Seven That Failed, and Re-measuring From Hugging Face

October 12, 2026

12 October 2026

Models installed with Ollama are stored as a single large file (a blob). Its contents are GGUF, so llama.cpp’s llama-bench should have been able to read it as is. But 6 models I tried to measure in September and 1 added in October, 7 in all, stop with “failed to load model". There is plenty of free space and the files are not corrupted. What is different? On 3 October I checked the cause and re-measured the same models in a different container.

This reflects information as of October 2026.

How I measured: reading Ollama blobs directly with llama-bench

The machine was a GMKtec EVO-X2 (Ryzen AI Max+ 395, 128GB memory), using the integrated Radeon 8060S. The software was llama-bench from the official llama.cpp build b11192. I passed the blobs that Ollama saved (files starting with sha256- under places like /usr/share/ollama) as the model, as they are, and measured reading speed (pp512) and writing speed (tg128) five times each. I sorted the runs, dropped the highest and lowest, and took the median of the middle three. The measurements were made on 3 October 2026.

The 7 that would not load, and the 14 that did

Of 22 models, 14 passed all five runs, 1 (qwen3-235b-q2) stopped loading after the first run, and 7 stopped with “failed to load model" from the start.

The 7 that failed vary in file size and family. The 18GB gemma4:31b and the 80GB Nemotron 3 Super gave the same error. It is not a capacity problem.

Here are the numbers for the side that loaded: the 14 that passed five runs on the same day under the same conditions. tg128 is tokens per second when writing 128 tokens, and pp512 is tokens per second when reading 512 tokens.

llama3.3:70b at 5.3 tok/s is slower than a person reads, and the three MoE models (30-35B) run at 70-90 tok/s, as fast as 4B-class models.

Reading the head of a GGUF shows what is inside

A GGUF file carries a “header" at the start with the model’s type (architecture), number of layers and so on. I read the head of two of the models that failed with Python.

In both, the text body and the part that reads images (the vision encoder) lived together in one file. In mainline llama.cpp the image part is kept in a separate file called mmproj. For its own engine, Ollama distributes a form that bundles the body and the image part into one. Mainline llama.cpp does not know this combined header, so it stops loading.

A little background. In 2025 Ollama began moving from using llama.cpp as is to its own engine written in Go. The structures of Gemma, Qwen, Llama and others are implemented directly in the engine, and the image handling is inside the same engine. The multimodal models Ollama’s official library distributes (Gemma 4, the Qwen3.5 family and others) embed the vision weights into the body GGUF on the premise of this own engine. An Ollama GitHub issue (ollama/ollama #14730, “unknown model architecture: qwen35moe when loading imported GGUF with mmproj") also explains that the official qwen3.5:35b holds qwen35moe.vision weights in the body GGUF, so images work without a separate file. The source for the move to its own engine is the official Ollama blog post “Ollama’s new engine for multimodal models" (2025).

Of the 7 that failed, I checked the file header on only two: gemma4:31b and qwen3.6:27b. The other five (gpt-oss, Nemotron 3 Super, Qwen3 Coder Next, Qwen3.6 35B-A3B, GLM 4.7 Flash) gave the same error, but I did not read their headers. This article does not assert that the cause is the same. In this blog’s 24 September record, with an older build (b10941), the gpt-oss blob gave “format name gptoss not supported" and Nemotron gave “weight shapes do not match". The reason may differ from the bundled vision, and I have not checked whether it is the same on b11192.

The 14 that loaded were text-only models, or ones distributed in a form without the vision part. Even for the same “GGUF", the conventions inside differ a little by engine.

GGUF files distributed on Hugging Face load fine

I swapped in GGUF files distributed on Hugging Face (unsloth, ggml-org) for the same models and measured them with the same llama-bench. All four loaded and passed five runs. As a rule of thumb, the writing speed works out to roughly 10-60 characters of Japanese text per second.

The machine was the integrated GPU (Radeon 8060S) of a GMKtec EVO-X2, with llama.cpp b11192; each is the median of the middle three of five runs with the highest and lowest dropped. For the remaining three (Qwen3 Coder Next, Nemotron 3 Super, GLM 4.7 Flash), I did not fetch Hugging Face versions that day.

The speed numbers change with the container

gpt-oss 120B measured 37.2 tok/s on Ollama on 24 September (five runs, middle three; in this blog’s MiMo article). On 3 October with llama.cpp b11192 it measured 55.2 tok/s. On the same machine and the same MXFP4 format, that is a 1.5 times difference. But the Ollama file is 60.9GiB and the Hugging Face one 59.0GiB, so the distributed files themselves differ. In terms of text, it is a change from about 40 characters a second to about 60. The cause is probably the version of the engine inside Ollama or its default settings (context length, batch), but I have not separated which factor is bigger.

For other models I also have values measured on both Ollama and llama.cpp. The measurement days differ, so they are reference values, and the pattern of difference is not uniform.

The three dense models differed by about 5% in this comparison. The July values were measured on a different day with different settings, so this is no basis to say there is no difference. Only gpt-oss 120B differed greatly. gpt-oss is distributed in MXFP4, a 4-bit format, and its speed probably depends on how llama.cpp handles calculation in this format. I have not separated which factor is at work.

On this blog, the speed tables keep a column for “which engine, which build it was measured with". When lining up numbers across containers, you need to read with that column matched.

What I Haven’t Confirmed in This Article

• Whether there is a way to export the same 7 models from Ollama into a llama.cpp-readable form (I have not tried)
• For 5 of the 7 that would not load, I did not read the file headers (it is unconfirmed whether the cause is the same as for the two)
• The breakdown of the speed difference between Ollama and llama.cpp (37.2 vs 55.2 tok/s)
• Speeds of Qwen3 Coder Next, Nemotron 3 Super and GLM 4.7 Flash on their Hugging Face versions
• The same comparison on an NVIDIA graphics card

Summary: even the “same GGUF" follows the container’s own conventions

For the two models whose headers I checked, the reason Ollama blobs would not load in llama.cpp was neither capacity nor version, but a custom form that houses the image part together. For the other five the error is the same, but I have not confirmed the cause. If you measure with llama.cpp, using the official GGUF on Hugging Face is the shortcut. Speed numbers change with the engine, so I always attach the engine and build to a table.

A word on finding the Hugging Face version. Search Hugging Face for “model name GGUF", and if the publisher is one of unsloth, bartowski, ggml-org or the developer’s official account, it is likely a distribution that mainline llama.cpp can read. Q4_K_M in the file name is the standard 4-bit quantization; take a separate mmproj file only when you want to use images. Conversely, to use a Hugging Face GGUF in Ollama, Ollama has an entry point where you specify it in the form hf.co/publisher/repository:quantization.

As a next step, if you want to use a model installed with Ollama in another tool, first check whether a GGUF of the same model exists on Hugging Face. How to choose a model is covered in the by-use summaries.

Which local LLM to install first (2026)

What runs by memory size (VRAM)

Hardware used for the tests

As an Amazon Associate we earn from qualifying purchases.