From GPT-2 to Kimi K3: What Actually Changed in Seven Years? A Beginner-Friendly Walkthrough

2026年8月5日

GPT-2 came out in 2019 with 124 million parameters. Kimi K3, released in July 2026, has 2.8 trillion. You could fit 22,580 GPT-2s inside it. Seven years, that much bigger — but is size all that changed?

An article walking through that question by following the implementation code was published on X recently and drew a lot of attention: “From GPT2 to Kimi3, Explained" by ali (@waterloo_intern). It is a rewarding read, but it is built around code and mathematics, and it assumes a fair amount of background.

This article takes the same thread and unpacks it in a form a beginner can follow. Alongside it, I put in numbers I have actually measured on my own machines, so you can see how the reasoning shows up in real hardware.

As of July 2026.

About the original article

The original is organized around a single thread: what did each generation struggle with, and what did it add in response? That framing made it easy to follow, so I keep it here.

Where it started: how plain GPT-2 was

GPT-2 simply predicted text left to right. Turn words into numbers, push them through 12 processing blocks, and produce the word most likely to come next. That was it.

There was waste in how it generated. Every time it produced one word, it recomputed the entire text from the beginning. To produce character 100 it redid the work for the previous 99.

What happened when the recomputation stopped? The KV cache

The fix was simple: remember what you already computed and reuse it. That is the KV cache. The amount of computation dropped dramatically.

But a different problem appeared. The amount you have to remember grows as the text gets longer. And the speed of reading that remembered material back became the new bottleneck. The arithmetic got fast; memory could not keep up.

That bottleneck showed up on my own hardware

I have measured this. On a 12GB graphics card, generation speed while varying the context length:

One model slowed down as the context grew, ending at a quarter of its starting speed: what it had to remember no longer fit in VRAM and spilled onto the CPU. The other held the same speed to the end, within the same 12GB. How efficiently a model remembers differs from model to model, and that difference is what you feel.

Everything that follows makes more sense if you read it as a fight with “how much do we have to remember". I wrote up the detailed measurements in a separate article on context length and speed.

Can we fix how much we remember? The idea behind Linear Attention

The next direction was to stop the remembered amount from growing at all.

The conventional approach lines up past information on a shelf, one item at a time; a longer text means a longer shelf. Linear Attention instead writes information over the top of itself on a single board of fixed size. The board never changes size, so no matter how long the text, the amount remembered stays constant.

Making the order of the arithmetic interchangeable is what allows this. There is a cost, though: overwriting is coarser than the original approach. Speed traded against precision.

What goes wrong when you overwrite? Information collides

A fixed-size board eventually fills up. Keep writing on a full board and the new content mixes with the old until neither can be read.

The original article quotes a paper that named this problem. Paraphrased: if you keep adding to a finite memory forever, you will hit the limit; the model itself should get to choose what to keep and what to erase.

Enter DeltaNet. The idea is familiar: instead of adding on top, erase the old content first and then write the new. Like rewriting one column of a blackboard. That eased the problem of memories blurring together.

Isn’t rewriting one at a time slow? Making it parallel

“Erase then write" has a weakness. Each rewrite changes the state of the board, so it has to be done in order. GPUs are built to do enormous numbers of things at once, and queueing up stops them working at full strength.

The fix was to cut the text into chunks of a reasonable size. Inside a chunk, compute everything at once as before; hand the board over only between chunks. That cuts the number of times anything has to wait in line.

The author of the original article writes, honestly, that this part took them seven hours to understand. It does seem to be the hardest passage.

Can it forget? The idea of decay

Even with rewriting, something was still missing: forgetting in bulk.

DeltaNet can replace an individual entry, but when the topic changes completely it cannot flush what came before. Only individual rewrites are available.

So a mechanism was added that thins the whole board slightly as it goes. Old information fades naturally, making room for the new. Kimi Linear went further and made the rate of fading adjustable per entry — keep what matters, forget the rest quickly.

Here is the whole progression in one table.

Every stage exists to fill a gap left by the one before it. That is the original article’s argument — it did not just get bigger — and this table is the evidence for it.

What did Kimi K3 put in? All of the above

Which brings us to Kimi K3. It combines everything so far, and then adds more.

The interesting part is that it does not commit to one approach. In each group of four layers, three remember using the fixed-size board and the remaining one uses the conventional method to look at the whole context. That group repeats 23 times.

Whatever would not fit on the board gets picked up by the conventional layer. The roles are divided. On top of that, every 12 layers there is a mechanism that goes back to look at what earlier layers produced.

Choosing experts, and the power draw I measured

The other eye-catching number in Kimi K3 is the count of experts. It holds 898 of them and wakes only 18 for any given token (2 always work; 16 are chosen from the remaining 896). Because it never wakes everyone, it runs light for its size.

I have measured how that “only the relevant staff work" structure actually plays out. Power draw for models of different types on the same graphics card:

The type that wakes only who it needs was both faster and lower-power. Not waking everyone means reading less from memory. That mechanism is presumably what lets Kimi K3 carry 898 experts at all.

As an Amazon Associate we earn from qualifying purchases.

The details are in my article on power-limiting a GPU for local LLMs.

So what actually changed?

The original article’s conclusion is clear: size is not the only thing that changed.

Once you are packing information into a limited space, that space will run out, and you need a mechanism for deciding what to throw away. Seven years of changes read as a history of refining how to throw things away. Addition alone breaks, so it learned to rewrite; rewriting alone was not enough, so it learned to forget; forgetting was too blunt, so it learned to control the rate per entry.

Laid alongside the numbers I have measured, none of this is somebody else’s problem. Speed falling as the context grows, and the wake-only-who-you-need type being frugal with power, both come out of the same fight with capacity. What is happening at the research frontier and what you feel on a PC at home are connected by the same reasoning — that was the most interesting part of this for me.

This article unpacks the original for beginners. If the mathematics or the implementation interests you, please read the original. For technical claims, checking the papers and official documentation is recommended.

Hardware used

The graphics card used for the power comparison.