Open Judge Models Tie Jev on 8 of 11 Tasks, With Some Asterisks
We ran benchmarks for several eval tasks, prompt injection detection, PII, moderation, but also routing, tool call choosing etc, 11 in total, Jev vs 15 open models up to 28B. We controlled the most we could for out of distribution so they can't "cheat".
Open models tie Jev on 8 of the 11 tasks, beat it on PII (Eikos-27B 95.2% vs 90.8%) and tool routing (Shisa DE-1 81.3% vs 78.3%), and lose on off-topic detection (Kev-9B 89.6% vs Jev 93.4%). On the average, Eikos-27B gets 78.5% and Jev 78.4%, but that's a 27B model, you need a proper GPU like an H100 to run it.
We run LangWatch Instant Evals on Jev so we really care about which one is best. I think small judge models are here to stay, and evals is our thing, so we'll keep benchmarking every new one that comes out and run your evals on whatever is winning. For now it's still Jev, it's the only one that scored on all 11 tasks, and it's tied or ahead on 9 of them.
Per-task charts and the full method are in the first comment.
The comparisons are accurately reported and match LangWatch's own benchmark page (released September 25, about ten days after Jev's launch). But the two averages are out of date, and several qualifiers from the page itself didn't make it into the post.
Headline and counts: confirmed. Across 15 open models up to 28B on 11 tasks, the page reports ties with Jev on 8, open-model leads on 2, and a trail on off-topic. Jev is the only model with a score on all 11 tasks. Your "tied or ahead on 9" checks out: 8 ties plus 1 win.
The three per-task numbers are exact. PII (catch rate at 5% false alarms) is Eikos-27B 95.2% vs Jev 90.8%. Tool routing is Shisa DE-1 81.3% vs 78.3%. Off-topic is Kev-9B 89.6% vs Jev 93.4%.
The averages are stale. The post says 78.5% vs 78.4%. The current page shows Eikos-27B at 80.0% across the 8 tasks it could be ranked on, against 79.9% for Jev on those same 8 tasks. The 0.1 gap is the same, but both figures are about 1.5 points higher now. An independent GitHub issue quotes the same current numbers. Jev's own 11-task average is 81.9%, and the page warns against comparing one row's average to another, since each model is averaged over its own tasks.
The two open-model wins rest on datasets with documented problems. For PII, the negatives carry redaction stand-ins that a bag-of-words classifier separates perfectly. Without those sources the best open model becomes Kev-9B at 91.6%, and the page says the Jev-vs-open gap on the one clean source is a fraction of the headline gap. For tool routing, roughly a fifth of one source is mislabeled by the task's own definition, and most of its items offer a single tool. On the cleaner multiple-tool half, the top open models score 97 to 98%. LangWatch's companion page shows Jev at 96.8% on what appears to be that same subset, so the 3-point lead shrinks to about a point or less. Jev's off-topic lead has a similar artifact, but it holds without that source (91.9% vs 87.3%).
"They can't cheat" is stronger than the method supports. Models known to be trained on a task's data are grayed out and never ranked, but that relies on what authors disclosed. Jev's training data is undisclosed, and the page says no declaration is not proof of no exposure. So Jev being the only model ranked on all 11 partly reflects disclosure rules. For example
"Tie" means the test can't separate them. With 300 to 1,000 items per task and intervals several points wide, that is not the same as equal.
"You need an H100" overstates it. The 27B models ran on an H100 in this benchmark. That's what was measured, not a stated requirement. A quantized 27B can fit a 24 GB card, though I found no benchmark of that here.
"Caught up" applies to 4B to 28B models, not tiny ones. LangWatch's earlier page reported that Jev beats every open model under 1B.
Stake and reproducibility. The post discloses that Instant Evals runs on Jev, and the page says so too. But the harness and decision thresholds aren't public, the runs came from a working tree with uncommitted changes, and labels haven't been human-rechecked.