optillm: Trade Compute for Accuracy Without Touching Your Model Weights

Your LLM can reason better without any fine-tuning!
optillm is an OpenAI API-compatible proxy that implements 20+ optimization techniques to improve LLM accuracy on reasoning tasks without training or fine-tuning.
The concept: Instead of one API call, optillm makes multiple calls using different techniques and combines the results. You're trading compute for accuracy - more API calls, higher cost, slower response, but better results.
How it works: optillm sits between your OpenAI client and the LLM API. You control which technique by prepending a slug to the model name. With Mixture of Agents, optillm makes 3 parallel API calls with different approaches, synthesizes them, and returns the best answer.
The tradeoff: A query that takes 1 API call and 2 seconds now takes 4 calls and 5 seconds. Token cost goes up 4x. But accuracy jumps significantly on reasoning tasks.
Results show the gains. Mixture of Agents using gpt-4o-mini matches GPT-4 on Arena-Hard-Auto. PlanSearch achieves 20% higher pass@5 on LiveCodeBench.
Available techniques:
• Mixture of Agents: Multiple models critique each other
• Monte Carlo Tree Search: Explores decision trees
• PlanSearch: Searches candidate plans before executing
• Best of N: Generates multiple responses, picks best
• Chain-of-Thought with Reflection: Structured thinking and output
• Self-Consistency: Multiple reasoning paths
Works with 100+ models via LiteLLM. You can combine techniques in pipelines or run them parallel.
The insight: Spend more computation at query time to get better results without training. Works for benchmarks, offline tasks, critical queries. Not for real-time production.

optillm is real, and the technique list and mechanics are described with strong accuracy — several phrases in your post are near-exact matches to the project's own README. A couple of specific benchmark numbers I couldn't independently confirm, and one framing point ("not for real-time production") is worth a small nuance.

Core description — confirmed almost verbatim. The project's own README states: OptiLLM is an OpenAI API-compatible optimizing inference proxy that implements 20+ state-of-the-art techniques to dramatically improve LLM accuracy and performance on reasoning tasks — without requiring any model training or fine-tuning. That's essentially your opening line word for word.

"Prepend a slug to the model name" — confirmed exactly. The docs state directly: you can control the technique you use for optimization by prepending the slug to the model name.

Every technique in your list is confirmed as a real, implemented feature, several with near-identical wording:

• Mixture of Agents (moa) — combines responses from multiple critiques.
• PlanSearch — implements a search algorithm over candidate plans for solving a problem in natural language.
• Best of N and MCTS — both directly referenced in the project's own tagline: 20+ Optimization Techniques: From simple best-of-N to advanced MCTS and planning.
• Chain-of-Thought with Reflection — implements chain-of-thought reasoning with <thinking>, <reflection> and <output> sections.
• Self-Consistency — confirmed, listed as an advanced self-consistency method.

"MoA makes 3 parallel calls" — confirmed by the project's own example log output, which shows exactly three POST requests firing when the moa approach is invoked.

Two things I couldn't independently confirm this session: the specific "matches GPT-4 on Arena-Hard-Auto" and "20% higher pass@5 on LiveCodeBench" figures. These are the kind of numbers optillm's own benchmark documentation reports (I found a related PlanSearch pass@1/5/10 table in the same format), but I wasn't able to pull the exact baseline comparison to verify the specific deltas cited. Worth a quick check against the current README's benchmark section before repeating them as hard numbers. Similarly, "100+ models via LiteLLM" is plausible given the project's broad OpenAI-compatible design, but I didn't find that exact figure stated directly.

One framing point worth softening slightly: "Not for real-time production." The project's own marketing explicitly claims Production Ready: Used in production by companies and researchers worldwide. That's not strictly a contradiction of your point — "production" doesn't necessarily mean latency-sensitive, real-time chat UX, and your distinction (batch/offline/critical-query use vs. real-time chat) is a reasonable and common one in practice — but a reader skimming both claims side by side could read them as conflicting. Worth being explicit that it's real-time latency-sensitive use cases specifically that don't fit, not production generally.

Nothing here is fabricated — the architecture and mechanics are accurately and precisely described; the softness is entirely in two specific benchmark figures and one framing nuance.