S
pending
Beyond One Optimizer: Why Combining GEPA, AutoResearch, and Meta-Harness Beats Any One Alone
Grounded / Real
Inflated / Uruttu
Original Content
The next step after Karpathy's autoresearch idea:
Tuning an agent is mostly manual work, done by editing prompts, tools, and control flow by hand and rerunning the evals to see what moved.
Researchers have been building several automated optimizers to do that outer loop instead.
The underlying process is the same:
- An LLM proposes a change
- An evaluator scores it
- And the proposer reads that result before proposing the next one.
They differ in what they edit and what feedback they get to read.
1) Berkeley built GEPA that optimizes the text of a system, like prompts, tool descriptions, or the agent's own code.
Instead of collapsing a run into one scalar reward the way RL does, it reads the full execution trace (errors, reasoning, tool output), diagnoses why the run failed, and proposes a targeted fix.
It also keeps every candidate that's best at some part of the task, not just the one with the highest average score. So a strong specialist survives even when a more balanced candidate beats it overall.
That trace-level feedback lets it converge in hundreds of rollouts instead of the thousands that GRPO needs.
2) AutoResearch, inspired by Karpathy, runs a narrower version of the loop, where a coding agent iterates on a program(.)md file, scores the outputs against fixed evals, and keeps whatever improves.
3) Meta-Harness points the loop at the harness itself, the scaffolding code that decides what to retrieve, how to format it, and what state to keep between calls.
The natural question is which of the three to use. And the answer is none of them alone.
On the Frontier-CS benchmark, holding the model, thinking effort, and budget fixed, no optimizer wins everywhere. Across 10 tasks, GEPA led on 3, AutoResearch on 3, and Meta-Harness on 4.
Each optimizer hill-climbs fast and then stalls, making most of its progress in a few iterations before flattening out.
But the researchers found that handing the stalled candidate to a different optimizer breaks the plateau, since each one attacks the problem differently.
To use this in practice, omni (open-source) already automates all of that.
It runs every optimizer on a fraction of the budget, takes the best candidate, and hands it to a fresh optimizer to keep going.
It scores 7.8 percentage points above the best standalone optimizer at the same budget, and finishes faster.
The whole meta-optimizer is around ten lines in the optimize_anything API, the same interface these optimizers already run through.
You can also point an agent at the gepa-ai/gepa repo and have it use the gepa-optimize-anything skill.
I have shared the repo in the replies.
____
Share this with your network if you found this insightful ♻️
Find me → Avi Chawla. Every day, I share tutorials and insights on DS, ML, LLMs, and RAGs.
Tuning an agent is mostly manual work, done by editing prompts, tools, and control flow by hand and rerunning the evals to see what moved.
Researchers have been building several automated optimizers to do that outer loop instead.
The underlying process is the same:
- An LLM proposes a change
- An evaluator scores it
- And the proposer reads that result before proposing the next one.
They differ in what they edit and what feedback they get to read.
1) Berkeley built GEPA that optimizes the text of a system, like prompts, tool descriptions, or the agent's own code.
Instead of collapsing a run into one scalar reward the way RL does, it reads the full execution trace (errors, reasoning, tool output), diagnoses why the run failed, and proposes a targeted fix.
It also keeps every candidate that's best at some part of the task, not just the one with the highest average score. So a strong specialist survives even when a more balanced candidate beats it overall.
That trace-level feedback lets it converge in hundreds of rollouts instead of the thousands that GRPO needs.
2) AutoResearch, inspired by Karpathy, runs a narrower version of the loop, where a coding agent iterates on a program(.)md file, scores the outputs against fixed evals, and keeps whatever improves.
3) Meta-Harness points the loop at the harness itself, the scaffolding code that decides what to retrieve, how to format it, and what state to keep between calls.
The natural question is which of the three to use. And the answer is none of them alone.
On the Frontier-CS benchmark, holding the model, thinking effort, and budget fixed, no optimizer wins everywhere. Across 10 tasks, GEPA led on 3, AutoResearch on 3, and Meta-Harness on 4.
Each optimizer hill-climbs fast and then stalls, making most of its progress in a few iterations before flattening out.
But the researchers found that handing the stalled candidate to a different optimizer breaks the plateau, since each one attacks the problem differently.
To use this in practice, omni (open-source) already automates all of that.
It runs every optimizer on a fraction of the budget, takes the best candidate, and hands it to a fresh optimizer to keep going.
It scores 7.8 percentage points above the best standalone optimizer at the same budget, and finishes faster.
The whole meta-optimizer is around ten lines in the optimize_anything API, the same interface these optimizers already run through.
You can also point an agent at the gepa-ai/gepa repo and have it use the gepa-optimize-anything skill.
I have shared the repo in the replies.
____
Share this with your network if you found this insightful ♻️
Find me → Avi Chawla. Every day, I share tutorials and insights on DS, ML, LLMs, and RAGs.
Validated Content
This post is remarkably accurate — every technical claim, number, and attribution checks out against the primary sources (the GEPA paper, and the GEPA project's own blog post on "omni").
Confirmed — nearly everything matches exactly
GEPA (Berkeley) — real, published research (arXiv 2507.19457, ICLR 2026 Oral), led by UC Berkeley researchers (with Stanford/Databricks/MIT collaborators). Description is accurate: it reads full execution traces (reasoning, tool calls, tool outputs, errors) rather than collapsing to a scalar reward, and uses a Pareto frontier to keep specialist candidates alive even if they don't have the best average score.
"Hundreds vs. thousands of rollouts" — accurate. The paper reports GEPA matching GRPO's performance in ~300-400 rollouts versus GRPO's ~24,000, and up to 35x fewer rollouts overall.
AutoResearch, "inspired by Karpathy" — accurate. Multiple independent papers cite it exactly this way ("AutoResearch (Karpathy, 2026)"), describing an agent that iterates on a program file and keeps improvements that beat fixed evals.
Meta-Harness — accurate. This is a real system (Lee et al., 2026) that optimizes the harness/scaffolding code itself, as described.
Frontier-CS benchmark results — confirmed exactly: on 10 problems, GEPA won 3, AutoResearch won 3, Meta-Harness won 4. This is a precise match to the GEPA project's own published blog post.
"Hill-climb fast then plateau, handing off unblocks progress" — accurate, directly stated in the source material.
omni meta-optimizer, +7.8 percentage points, ~10 lines of code, optimize_anything API — all confirmed exactly, word-for-word matches to the GEPA blog post announcing omni.
gepa-ai/gepa repo — real, and optimize_anything is explicitly documented as open-sourced within that repo.
Confirmed — nearly everything matches exactly
GEPA (Berkeley) — real, published research (arXiv 2507.19457, ICLR 2026 Oral), led by UC Berkeley researchers (with Stanford/Databricks/MIT collaborators). Description is accurate: it reads full execution traces (reasoning, tool calls, tool outputs, errors) rather than collapsing to a scalar reward, and uses a Pareto frontier to keep specialist candidates alive even if they don't have the best average score.
"Hundreds vs. thousands of rollouts" — accurate. The paper reports GEPA matching GRPO's performance in ~300-400 rollouts versus GRPO's ~24,000, and up to 35x fewer rollouts overall.
AutoResearch, "inspired by Karpathy" — accurate. Multiple independent papers cite it exactly this way ("AutoResearch (Karpathy, 2026)"), describing an agent that iterates on a program file and keeps improvements that beat fixed evals.
Meta-Harness — accurate. This is a real system (Lee et al., 2026) that optimizes the harness/scaffolding code itself, as described.
Frontier-CS benchmark results — confirmed exactly: on 10 problems, GEPA won 3, AutoResearch won 3, Meta-Harness won 4. This is a precise match to the GEPA project's own published blog post.
"Hill-climb fast then plateau, handing off unblocks progress" — accurate, directly stated in the source material.
omni meta-optimizer, +7.8 percentage points, ~10 lines of code, optimize_anything API — all confirmed exactly, word-for-word matches to the GEPA blog post announcing omni.
gepa-ai/gepa repo — real, and optimize_anything is explicitly documented as open-sourced within that repo.