AnyJev: Bring Calibrated, Typed Decisions to Any Open LLM You Already Run
Researchers just open-sourced AnyJev, a Python library that extracts typed choices, yes/no judgments, scores, and probabilities from causal LLMs.
Beyond its raw baseline, AnyJev offers three levels depending on how much labeled data and calibration you need:
- L0 requires no labels. It rotates the answer options to reduce position bias, but its probabilities remain uncalibrated.
- L1 uses 100-500 labeled examples per question to calibrate those probabilities through temperature scaling. It improves confidence estimates without changing the answer ranking.
- L2 uses 100-300 labels to fit a closed-form head on an intermediate hidden state. It leaves model weights untouched and can stop inference before the full forward pass completes.
The main constraint is model access. AnyJev requires access to logits or hidden states, so it is built for open models. L2 heads are also specific to a model and question.
It is compatible with several open causal LLMs.
Find the repo in the replies
Researchers have open-sourced AnyJev, a Python library that extracts typed choices, yes/no judgments, and scores directly from any open causal LLM's logits or hidden states — no fine-tuning required by default. It's an independent project, not affiliated with or built from TypeSafe's proprietary Jev model.
It offers three levels of trust depending on available labeled data: a label-free baseline (L0) that rotates answer options to cancel out position bias, but leaves probabilities uncalibrated; a calibrated tier (L1) using 100-500 labeled examples with temperature scaling to fix confidence estimates without changing which answer wins; and a lightweight-head tier using 100-300 labels to fit a small linear classifier on an intermediate hidden state — leaving the base model's weights untouched, and allowing inference to stop early rather than running the full forward pass.
Its published benchmark makes a strong case for why calibration matters more than raw accuracy alone: on one classification task, moving from raw logits to a calibrated readout only shifted accuracy by 6 points, but roughly 7x more of the traffic became confident enough to safely automate without human review.
It requires access to a model's internals, so it works with open models via Hugging Face/vLLM, and its most-tested benchmarks are currently on the Qwen model family.