Your LLM Doesn't Need Fine-Tuning to Think Better
optillm is an OpenAI API-compatible proxy that implements 20+ optimization techniques to improve LLM accuracy on reasoning tasks without training or fine-tuning.
The concept: Instead of one API call, optillm makes multiple calls using different techniques and combines the results. You're trading compute for accuracy - more API calls, higher cost, slower response, but better results.
How it works: optillm sits between your OpenAI client and the LLM API. You control which technique by prepending a slug to the model name. With Mixture of Agents, optillm makes 3 parallel API calls with different approaches, synthesizes them, and returns the best answer.
The tradeoff: A query that takes 1 API call and 2 seconds now takes 4 calls and 5 seconds. Token cost goes up 4x. But accuracy jumps significantly on reasoning tasks.
Results show the gains. Mixture of Agents using gpt-4o-mini matches GPT-4 on Arena-Hard-Auto. PlanSearch achieves 20% higher pass@5 on LiveCodeBench.
Available techniques:
• Mixture of Agents: Multiple models critique each other
• Monte Carlo Tree Search: Explores decision trees
• PlanSearch: Searches candidate plans before executing
• Best of N: Generates multiple responses, picks best
• Chain-of-Thought with Reflection: Structured thinking and output
• Self-Consistency: Multiple reasoning paths
Works with 100+ models via LiteLLM. You can combine techniques in pipelines or run them parallel.
The insight: Spend more computation at query time to get better results without training. Works for benchmarks, offline tasks, critical queries. Not for real-time production.
Overall: Mostly accurate, but a few claims need more nuance or sourcing.
| Claim | Verdict | Notes |
|---|---|---|
| "optillm is an OpenAI API-compatible proxy" | ✅ True | OptiLLM is an OpenAI-compatible inference proxy that sits between your client and the LLM API. (GitHub) |
| "Implements 20+ optimization techniques" | ✅ True | The current project documents 20+ inference-time optimization techniques. (GitHub) |
| "Improves LLM accuracy without training or fine-tuning" | ✅ True | This is the core idea behind OptiLLM: test-time compute instead of model training. (GitHub) |
| "Makes multiple API calls and combines results" | ✅ Mostly True | This is true for many techniques (MOA, Best-of-N, Self-Consistency, etc.), but not every optimization uses multiple API calls. Some use planning, decoding, or plugins instead. (GitHub) |
| "Prepend a slug to the model name" | ✅ True | Example: moa-gpt-4o-mini. (GitHub) |
| "MOA makes 3 parallel API calls then synthesizes" | ✅ Mostly True | The default MOA implementation issues multiple requests (typically three workers followed by synthesis), though this is configurable rather than a universal rule. (GitHub) |
| "4x cost and ~5 seconds instead of 2" | ⚠️ Illustrative, not universal | Cost and latency depend on the selected technique, number of samples, model, and configuration. Treat these as examples, not fixed numbers. (GitHub) |
| "MOA using GPT-4o-mini matches GPT-4 on Arena-Hard-Auto" | ✅ Supported | This benchmark result is reported by the project maintainers. It should be presented as their reported benchmark, not as an independently verified fact. (GitHub) |
| "PlanSearch achieves 20% higher pass@5 on LiveCodeBench" | ✅ Supported | The project reports pass@5 improving from 50.61 → 59.31, roughly a 17% relative improvement (or +8.7 percentage points). Saying "about 20% higher" is acceptable but imprecise. (GitHub) |
| "Works with 100+ models via LiteLLM" | ✅ True | This capability is documented by the project. (GitHub) |
| "Best for offline/critical queries, not real-time production" | ✅ Reasonable | This is good practical advice because inference-time optimization generally increases latency and cost. (GitHub) |
Minor corrections I'd make
Instead of:
"Instead of one API call, optillm makes multiple calls using different techniques..."
Use:
Many OptiLLM techniques perform additional inference-time computation—often through multiple LLM calls, planning, search, or self-reflection—to improve reasoning accuracy without fine-tuning.
Instead of:
"Mixture of Agents makes 3 parallel API calls..."
Use:
By default, Mixture of Agents launches multiple parallel worker calls and synthesizes their outputs. The exact number depends on the configuration.
Instead of:
"Accuracy jumps significantly..."
Use:
Many reasoning benchmarks show measurable improvements, although gains vary depending on the task, model, and optimization technique.
Overall rating
Accuracy: 9.3/10
The post is technically solid. The only parts I'd soften are:
Present benchmark numbers as project-reported results.
Avoid implying every optimization uses multiple API calls.
Make it clear that latency and cost examples are illustrative, not guaranteed.