How Do You Actually Choose a Local LLM Model? Five Axes I Found by Running a Dozen of Them — Tuning Series, Part 1

September 12, 2026September 9, 2026

The first thing that stumped me when I started with local AI was which model to pick. Open the model list in Ollama, the app for running generative AI, and you find hundreds of models lined up with different names and different sizes. There is no way to tell which one will run on your computer, or which one you should actually use.

So I actually ran a dozen-plus models on my own machines and measured their speed and how smart they are. This article records the axes for choosing a model that came out of that, together with the measured numbers. Instead of staring at a list of names, what should you actually look at to find the one model that fits your machine?

By “model" here, I mean the file that is the AI itself — the easiest way to picture it is the “smart" part of something like ChatGPT, packaged as a file and placed on your own computer.

These measurements were taken in July 2026. The lineup of models changes fast, so please read the numbers as a snapshot from that point in time.

The Hardware I Used

I used two machines for these measurements: my everyday desktop PC, and a mini PC with a large amount of memory. The two differ in how large a model actually fits.

The desktop’s RTX 3090 is a product with 24GB of dedicated memory (VRAM) built into the GPU itself. The mini PC, the EVO-X2, has no memory dedicated to the GPU; instead, its 128GB of memory is shared between the CPU and GPU, which suits it to carrying larger models.

The Five Things I Look at When Choosing a Model

After running a dozen-plus models, I found the things that actually matter when choosing sort into five. Go through them in order, and you can narrow things down to the one model that fits your machine.

Let’s go through each of these one at a time, with measured numbers.

Axis ①: Decide by Purpose (Being Smart Doesn’t Scale With Size)

It’s easy to assume bigger models are smarter, but once I actually measured it, things weren’t that simple.

I gave the same test to 12 models and scored them on two things: “reasoning power" (8 reasoning questions) and “coding ability" (the pass rate on 7 coding tasks). Since the score depends only on the test result, it comes out the same no matter which machine you run the model on.

phi4-mini stood out. At 2.5GB, the smallest of the group, it matched the 20GB-class models on reasoning at 7/8. Its coding ability, though, stayed at 4/7 — this is where the larger models pulled ahead.

If what you mainly want is research or help drafting text, a small model like phi4-mini is enough. If you want it to write code, you need a larger model such as Ornith-35B or gemma4:31b. Deciding your purpose first keeps you from picking a model bigger than you actually need.

Axis ②: Size and VRAM (Whether It Fits Matters Most)

A model runs at its true speed when it fits entirely inside VRAM, the GPU’s memory. When part of it doesn’t fit and spills over, that overflow gets handed off to the CPU’s memory instead, and speed drops sharply.

To see how much this matters, here’s the same gemma4:31b loaded onto the desktop’s RTX 3090 (24GB) at different levels of compression (quantization, explained below).

On paper, the Q8 version is 22.8GB, which looks like it should fit inside 24GB. In practice, though, the working memory used during inference gets added on top, it overflows, and speed dropped to 7.95 tok/s — less than a third of the 24.82 tok/s the Q4 version managed once it actually fit. It seems safer to assume that “the size number is only a little smaller than your VRAM" still means it will overflow.

Bigger Models Suit a Machine With More Memory

On a machine with plenty of memory, on the other hand, even a large model runs without overflowing. Comparing the two machines on gpt-oss:120b, a 120B-parameter (120 billion parameter) model, made the difference obvious.

Even with the exact same model, whether it fits changed speed by close to a factor of two. The starting point for “which model to choose" is picking from what actually fits inside your GPU’s VRAM (or memory).

Axis ③: Adjusting Size With Quantization

Quantization is a technique for compressing a model to shrink its file. It’s similar to saving a photo as a JPEG and trading a little image quality for a smaller file — you give up a small amount of intelligence in exchange for a better chance of fitting inside VRAM.

These are the three types you’ll commonly see on Ollama.

• Q4 … standard compression. A good balance of size and intelligence; picking this first rarely goes wrong
• Q8 … a version with almost no compression. Smarter, but roughly twice the size and prone to overflowing
• QAT … a version retrained with compression in mind from the start. Stays small while holding onto quality more easily

The gemma4:31b table above shows exactly what quantization does. Q8 has the edge on intelligence, but its larger size overflowed VRAM and made it slower instead. QAT came in small at 19.2GB and was also the fastest at 36.95 tok/s — among the compressed versions, it was the easiest one to work with.

When in doubt, pick Q4, then look for a QAT version if you have VRAM to spare — that order should serve you fine.

Axis ④: With MoE, Bigger Doesn’t Mean Slower

There are broadly two ways a model can be built. One is called dense, where the whole thing gets used every time. The other is MoE (Mixture of Experts) — a design that holds many internal “experts" and uses only some of them for any given question.

With MoE, even though the model as a whole is large, only part of it actually runs, so it moves fast for its size. Comparing two models of roughly equal intelligence on the EVO-X2 made the difference clear.

Both scored 7/8 and 7/7 on intelligence, yet on speed the MoE model, qwen3.6:35b-a3b, hit 55.3 tok/s against the dense gemma4:31b’s 10.11 tok/s — a gap of more than five times. If a name carries an “a" plus a number, like “a3b" or “a4b," you can read that as MoE, with that number being roughly the amount used per turn. If you want both speed and intelligence, MoE is a strong option.

qwen3.6:35b-a3b (MoE) | 55.3
gemma4:31b (dense) | 10.11

Axis ⑤: Speed Is Decided by the GPU’s Bandwidth

Even the same model runs at a different speed depending on which GPU it’s running on. Generating text is capped by how much data can move to and from memory (memory bandwidth), so a GPU with wider bandwidth runs faster.

Here are three differently sized models measured on two GPUs.

Between the higher-end RTX 3090 and the mainstream RTX 3060, the same model showed close to a twofold difference. What matters here is that the speed ranking doesn’t change across models. Choosing a GPU roughly sets the ceiling on speed, and choosing a model size within that ceiling is what decides the actual generation speed.

What I Did Not Measure

What I measured is limited to what I ran on my own hardware. Even for the same model, the numbers move if you change the quantization type or the context length. The intelligence comparison is against tasks I put together myself, and the ranking could change with a different set of tasks. I can’t say anything about models that don’t appear here.

In Summary: Narrow It Down in This Order

After running a dozen-plus models, I found that thinking about model choice in this order keeps you from getting lost.

Speed itself won’t change much unless you swap the GPU, but choosing in this order gets you to the one model that runs most comfortably on the machine you already have.

It turned out not to be a matter of just installing the biggest model available. Searching for the one that fits your own machine is what makes words like VRAM and quantization click, with a real feel behind them.

Next time, keeping the same model I chose here, I’ll look at measured tricks for tuning the settings to run it a bit faster.

Hardware Used for This Testing

Here’s the hardware used for this round of measurements.

Desktop GPU used (higher-end): NVIDIA GeForce RTX 3090

Desktop GPU used (mainstream): NVIDIA GeForce RTX 3060

Mini PC used for testing larger models: GMKtec EVO-X2

If you’re just getting started with local AI, here’s a related article.

Local AI — Complete Guide

Run AI on your own PC — local LLMs and image generation. Start here, pick a model, choose your G ...

 https://kimono-oyaji.com/local-ai-guide-en/

Here’s an article that measured the difference quantization makes.

Gemma 4 Quantization Compared: Q4 vs QAT vs Q8 (Speed & Quality)

Gemma 4 ships several quantization methods (ways of shrinking a model so it needs less memory) f ...

 https://kimono-oyaji.com/gemma4-quantization-compared/