Can a Local Model Reproduce Jev, the Fast-Decision AI? — Rebuilding the Mechanism on My Own GPU

2026年9月24日

A generative AI that doesn’t write. TypeSafe AI’s Jev returns options and probabilities instead of text, in 70 to 500 milliseconds.

It’s a hosted service, though, so you can’t put it on your own machine. How far can the same shape be built on a local GPU?

Purpose — How Much Faster Is It If the Model Doesn’t Write?

When you run a local LLM, most of the waiting is the model writing prose. But if all you want is “critical, watch, or ignore," you don’t need prose at all.

Jev drops that part. The official page puts it plainly:

Jev gives up string generation

In exchange for not writing, it returns a typed answer with probabilities — and claims 40x to 200x the speed.

Three things to check here. Does the same speed appear locally? Does the answer change? And can the probabilities be recovered too?

First, What the Word “Probability" Means Here

This article is full of numbers like “84% confidence." Read them the wrong way and they mean the opposite of what you think, so here is what they are.

Say you show the model a log line and ask “is this critical? answer yes or no." It answers “yes" and attaches 84% confidence.

That 84% means the weight on the word “yes" inside the model was 0.84 and “no" was 0.16. When a generative AI picks its next word, it assigns every candidate a weight — how much it wants to say that word. This is just the ratio between two of them. It’s a real computed number, not something made up.

What it is not is a claim that the answer is right 84% of the time. The weight means “how much I want to say this," and it was never trained to mean “how often I turn out to be right." A model that is confidently wrong reports 84% and misses anyway.

So is that 84% worth anything? You can’t know without measuring. You have to count how often it was actually right when it said 84%. That match-up is called calibration. It is what this article set out to check.

What about Jev’s side? The official documentation says:

Confidence is a statistic computed from the probability distribution the answer already gives you.

And that distribution spreads across the options you declared — not across the vocabulary. The docs also give thresholds: below 0.5 route to a human, 0.5 to 0.9 proceed with care, above 0.9 act autonomously.

So my probabilities and Jev’s are different objects by construction. The same “84%" is a number placed directly on the options in one case, and a stand-in built from word weights in the other.

Method — Two Ways to Ask the Same Question

Hardware and Software

Item
Hardware	GeForce RTX 5060 Ti 16GB (RTX 3060 12GB added for the 26B only)
Runtime	llama.cpp b10941 (HTTP server)
Arguments used	grammar / n_predict / n_probs / json_schema — all stock features

No new software. Everything runs on what llama.cpp already ships.

The Questions

Twenty server log lines, each to be sorted into critical, watch, or ignore. Ground truth is 6 critical, 6 watch, 8 ignore. The speed comparison alone used 6 questions.

The Two Approaches Compared

	A. Answer in prose	B. Just pick one
What it gets	The log, the question, and “explain your reasoning"	The log and the question
Tokens allowed	Up to 160	8
What it may emit	Anything	The three option words only (grammar-constrained)

Four Ways to Recover the Probabilities

Approach	What it does
Read off the three-way	Ask once, pick the options out of the top-N word probabilities
Split into yes/no	Ask “is it critical?" once per option, then normalize the yes-probabilities
Raise the temperature and count	Draw 8 times at temperature 0.8, treat the counts as probabilities
The official definition	Pin the model to one option, have it write that string, add up how likely it was, compare across options

How It Was Measured

Five runs per question. The first is discarded (it includes load time); the median of the rest is reported. Every time a model was swapped, the server was queried to confirm which model was actually loaded.

Metric	Meaning
Correct	How many of the 20 were right
Per question	Median time
Calibration gap	Distance between stated confidence and actual hit rate, averaged across bands by weight. Closer to zero is more honest

Results — The Speed Caught Up

Roughly 20x Faster

Model	A. Prose	B. Pick one	Ratio
Gemma 4 E4B	2,963.5 ms	146.6 ms	20.2x
Gemma 4 12B	3,851.1 ms	198.2 ms	19.4x

Three seconds of waiting becomes 0.15. Jev’s published figure is 70 to 500 milliseconds, so on speed alone this is the same bracket.

All Six Answers Matched

Gemma 4 E4B
A. Prose	3/6 correct
B. Pick one	3/6 correct
A and B agreed	6/6

Taking prose away didn’t change the model’s judgment. The only thing that changed was the wait. So far, exactly as intended.

But the Probabilities Don’t Come Out

This is where it stalled. Asking n_probs for the top-N word probabilities returned this:

Token	Probability
緊急 (critical)	0.4995
**	0.2738
[	0.1075
newline	0.0320

Everything below first place is markup. And the other two options never appeared, even in the top 400. n_probs returns the top N of a 150,000-word vocabulary, not the probabilities of your options.

Splitting Into Yes and No Lines Them Up

Approach	All options recovered
Three-way (Japanese)	0/6
Three-way (English)	2/6
Split into yes/no	6/6

“Yes" and “no" are each one token, so they always land near the top. Switching to English options didn’t change accuracy — still 2/6 — so the misses weren’t a Japanese problem.

Above 70% It Hits, Below It Misses

Stated confidence	Questions	Actually right
90% and up (avg 94.4%)	4	100%
80–90% (avg 84.1%)	5	80%
70–80% (avg 76.5%)	5	80%
50–70% (avg 56.2%)	5	0%
Under 50% (avg 50.0%)	1	0%

Gemma 4 E4B. Split at 70%: above it, 12 of 14 were correct; below it, all 6 were wrong. Whatever the exact numbers are worth, it works for deciding what a human should look at.

Results — Then the Model Changes Everything

All of the above is one model. Running the same 20 questions on everything else on hand broke the premise.

Gemma From the Bottom to the Top

Model	Size	Correct	Per question	Can confidence separate auto from human?
Gemma 4 E2B	2.7GB	11/20	220.6 ms	Yes (says 80–90%, hits 83.3%)
Gemma 4 E4B	4.3GB	12/20	337.0 ms	Yes (90%+ → 100% hit)
Gemma 4 12B (QAT)	6.5GB	9/20	468.7 ms	No — bunches at the low end
Gemma 4 12B (non-QAT)	6.8GB	10/20	469.8 ms	No
Gemma 4 26B-A4B	14GB	12/20	415.2 ms	No — all 20 sit under 50%

The 26B tied the E4B at 12/20 while being more than three times the size. The 12B is a trough, and swapping the quantization method (QAT versus not) didn’t move it.

Other Vendors’ Models

Model	Size	Correct	Per question
MiniCPM5 2B	1.5GB	11/20	95.6 ms
Spark-X2.5 4B	2.5GB	12/20	212.4 ms
Ornith-1.5 9B	5.4GB	17/20	404.5 ms

Ornith-1.5 9B took it at 17/20. MiniCPM5 2B was fastest at 96 ms, but all 20 questions came back under 50% confidence, so no cut point separates auto from human.

Can It Say a Normal Log Is Normal?

The eight “ignore" questions	Correct
Gemma 4 E4B	1/8
Ornith-1.5 9B	6/8

Gemma 4 E4B promoted a clean backup, twelve nodes healthy for 24 hours, and a successful certificate renewal — all to “watch." Same questions, same phrasing, and this much daylight between the two.

What Moves When You Change How You Read the Probability

Approach	Correct	Calibration gap	Per question
① Yes/no, once each	17/20	0.241	409 ms
② Two phrasings, averaged	18/20	0.519 (worse)	792 ms
③ Temperature 0.8, drawn 8 times	15/20	0.138 (best)	1,194 ms
④ The official definition	14/20	0.140	442 ms

Ornith-1.5 9B. The most accurate (①) and the most honest (③④) are in different places.

④ builds the probability the way the official definition does. The shape of its hits differs from ①:

Ground truth	Ornith-1.5 9B	Gemma 4 E4B
critical	6/6	6/6
watch	0/6	3/6
ignore	8/8	5/8
Total	14/20	14/20

What both share is “critical" — six out of six each. How they fall apart differs: Ornith pushed all six “watch" lines up to “critical."

Where to Cut, and How Much Runs Itself

Cut at	Handled automatically	Accuracy of those	Sent to a human
0.5	17	88%	3
0.6	10	100%	10
0.7	4	100%	16
0.8	0	—	20

Ornith-1.5 9B, 20 questions. Cut at 0.6 and half the queue handles itself, with all ten correct. A human looks at the other half.

More Options, Proportionally More Time

Options	Per question	Relative to 3
3	402.5 ms	1.0
5	640.7 ms	1.6
8	998.1 ms	2.5

Accuracy held at 9 of 10 throughout. Only the time went up.

Declaring a Type Returns Several Judgments at Once

	Time	Result
Four questions at once, with a declared type	762.8 ms	Type held on all five runs; identical answers
Same four questions in prose, no type	5,497.9 ms	Emitted headings and never answered

Using json_schema. That works out to about 190 ms per question.

Discussion — The Bottleneck Is Which Model You Pick

Those Three Seconds Were Typing, Not Thinking

Twenty times faster with all six answers unchanged means most of that wait was never deliberation. It was transcription. When the reader is a program rather than a person, none of it is needed.

If it really had been thinking time, cutting generation short would have coarsened the answers. It didn’t, so this reading holds. Waiting time itself is something I measured separately for time-to-first-output.

Whether Confidence Is Usable Depends on How the Model Is Built

This was the surprise. Across the Gemma line, only E2B and E4B could separate auto from human. The 12B and 26B bunch at the low end, and no cut point splits them.

E2B and E4B are built differently. Per Google’s model card, the E stands for effective, and each layer carries its own small table that is only consulted on lookup:

PLE gives each decoder layer its own small embedding for every token. These embedding tables are large but are only used for quick lookups, which is why the effective parameter count is much smaller than the total.

E2B is 2.3B effective (5.1B total); E4B is 4.5B effective (8B total). Those two, and only those two, returned confidence spread across bands. That said, this comes from five models measured once each — the architecture link is an observation, not something I established.

Going Bigger Doesn’t Make It More Accurate

Within Gemma, the 26B (14GB) and the E4B (4.3GB) tied. The 12B lands below even the E2B. Tripling the footprint returns nothing on this job.

And the 12B’s weakness isn’t the quantization: QAT scored 9/20, non-QAT 10/20 — the same level.

The best of everything measured was a 9B, Ornith at 17/20. Sorting by size does not sort by accuracy. You have to pick on measured numbers, not on size. Choosing a model in general is something I covered in a piece that ran a dozen of them and boiled it down to five axes.

“Can’t Call Normal Normal" Is a Model Quirk

Gemma 4 E4B promoted seven of eight normal lines. You add this to reduce alerts, and the alerts don’t go down. It errs toward safety rather than missing danger, so the harm is limited — but it won’t make things quieter.

Ornith managed 6/8, though. This is a per-model trait, not a fact about generative AI. Check it on the model you actually plan to use.

Accuracy and Honesty Move Independently

Changing how you read the probability sent accuracy and calibration in opposite directions. Two phrasings raise the hit count but make the probabilities dishonest; raising the temperature and counting makes them honest but costs accuracy.

The counting approach being honest makes sense. A read-off weight was never trained to mean “how often I’m right," but the count of what actually came out in 8 draws is reality itself.

If these were the same quantity, they couldn’t point opposite ways. “The tool that hits" and “the tool that is honest" are separate choices.

Where the Gap With the Official Service Is Widest

Scaling with the option count is the weak point here: 402 ms at three options, 998 ms at eight. Extrapolated, 255 options would run past 30 seconds.

The official docs state that up to 255 options are supported, and that questions are evaluated together in one pass. A handful of buckets is a fair fight. Dozens or hundreds is not.

A Note for Building This in Japanese

Option	Tokens	If you stop at one token
緊急 (critical)	2	comes out
無視 (ignore)	2	comes out
様子見 (watch)	3	truncates to 様

English write-ups say “stop at one token," which works because English option words fit in one. Copy that in Japanese and only the long option silently disappears. Count your longest option first.

What This Measurement Can’t Tell You

• Faster doesn’t mean more accurate. This technique cuts waiting; it doesn’t make anything smarter
• The 20 questions are ones I wrote. They are not production logs, and I decided the correct answers myself. Some may be ones reasonable people would split on
• Each model was measured once. Re-measuring could reorder them
• Some confidence bands hold only four to six questions. That is not enough to state a hit rate precisely
• The official definition was measured on two models. “Critical hits 6/6" lined up on both, but two is two
• 255 options were never actually tried — that figure is extrapolated from 3 through 8
• The 12B’s prose accuracy couldn’t be measured: its preamble ran past the 160-token budget before it answered
• I have not run Jev itself. The weights aren’t published and it can’t be hosted locally. The comparison is against published figures only
• Several open implementations of the same idea exist, but I have not run any of them. Everything here uses stock llama.cpp features

Future Work — Can the Calibration Gap Be Corrected Afterward?

Next to check	Why
Whether confidence can be corrected after the fact	Ornith is so understated it never produces a 0.8. If the shape of the gap is known, post-processing may fix it
Expanding to 100 questions	Bands currently hold four to six questions, which makes the hit rates coarse
The link to the E-series architecture	Only E2B and E4B produced usable confidence. Does the same happen on other vendors’ models built this way?
Throughput when requests run in parallel	llama.cpp holds several slots. Batching may absorb the cost of more options
Trying it on real logs	These 20 questions are mine. Production logs will split opinion far more

Conclusion — The Speed Caught Up; the Choice Was the Model

Asking the model to pick instead of write made it about 20x faster on a local GPU — 2,963 ms down to 146 — and all six answers stayed the same. Most of that wait was transcription.

The probabilities came out too, once the question was split into yes and no. Above 70% confidence, 12 of 14 were right; below it, all 6 were wrong. Cut at 0.6 and 10 of the 20 handle themselves, all ten correct.

But which model you pick changed everything. The largest (26B, 14GB) tied the 4.3GB E4B at 12/20, and the best of the lot was a 9B at 17/20. Only E2B and E4B could separate auto from human at all.

So what limits this approach is neither the speed nor the mechanism — it is picking the model. Sorting by size doesn’t sort by accuracy, so you have to measure on your own work. And past a few dozen options, asking once per option stops being competitive.

No new tooling is required. If llama.cpp is already running, a couple of extra arguments is the whole setup. Start by putting 20 of your own log lines through it and laying confidence next to hit rate. Where you can cut depends on your model and your work, and nobody can hand you that number.