MemoHarness: Teaching Agent Harnesses to Learn From Their Own Mistakes

Agent harnesses that learn from experience! MemoHarness is a new framework that makes agent harnesses adaptive rather than static. First, what's a harness. It's the external control layer that wraps a base LLM and turns it into a functional agent - managing context, tools, orchestration, memory, decoding, and output handling. Things like your CLAUDE. md, tool configurations, system prompts, memory strategies. Most agents use one fixed harness for every task regardless of what the task actually is. That's the problem MemoHarness addresses. A single static harness optimized for code generation isn't necessarily the right configuration for shell commands or analytical reasoning. But no one rebuilds the harness per task - it's too expensive and there's no principled way to do it. MemoHarness decomposes the harness into six editable control dimensions and treats execution history as a learning signal. After each run, it stores two types of knowledge in a dual-layer experience bank: per-case diagnoses of what went wrong in specific runs, and distilled global patterns extracted across many runs. For each new task, it retrieves relevant past experience and adapts the harness accordingly - no test-time labels, no additional feedback loop, no search required. The cost concern is real but addressed: when retrieved experience is cacheable, the additional context stays cost-competitive with fixed harnesses. Evaluated across shell-agent, code-generation, and analytical-reasoning benchmarks, MemoHarness improves over fixed harnesses and shows transfer to unseen task suites and base models. The broader point: most automatic improvement methods optimize prompts, pipelines, or workflows. The harness itself - the layer that controls how the agent processes everything - is usually left static. This paper makes the case that execution experience is a practical substrate for building harnesses that adapt. I've shared the link to the paper in the replies!

This is a real, verified academic paper — and the post's summary tracks the paper closely.

Confirmed accurate:

• MemoHarness is a real paper — "MemoHarness: Agent Harnesses That Learn from Experience" (arXiv:2607.14159), with accompanying code at github.com/HowieHwong/MemoHarness.
• The definition of "harness" — the external control layer wrapping a base LLM, managing context, tools, orchestration, memory, decoding, and output handling — is taken essentially verbatim from the paper's own abstract.
• The core problem statement — that deployed agents usually reuse a single global/static harness for all cases regardless of task, and that most automatic improvement methods target narrower things (prompts, pipelines, workflows) rather than the harness itself — matches the paper's abstract and introduction directly.
• Six editable control dimensions — confirmed; the paper explicitly decomposes the harness along six control surfaces: context, tool, generation, orchestration, memory, and output.
• Dual-layer experience bank — confirmed; the paper describes storing both per-case execution entries (diagnoses of specific runs) and distilled global patterns.
• No test-time labels, feedback, or additional search — confirmed; this is the paper's own stated design constraint for test-time adaptation.
• Two-phase design (training-time search + test-time case adaptation) — matches the GitHub repo's description of "Phase A" (training-time search) and "Phase B" (test-time adaptation via retrieval).
• Evaluation on shell-agent, code-generation, and analytical-reasoning benchmarks, with improvement over fixed harnesses and selective transfer to unseen suites/base models — confirmed directly from the paper's abstract, including the specific word "selective" (the post says "shows transfer," slightly simplifying "selective transfer" — a minor softening, not an inaccuracy).
• Cost/cacheability claim — confirmed; the paper states additional context "can remain cost-competitive" when retrieved experience is cacheable, matching the post's framing closely.

Minor nuance:

• The paper's own phrasing is "shows selective transfer to unseen suites and base models" — the post drops the word "selective," which slightly overstates the generality of the transfer result. The paper's more cautious phrasing suggests the improvement doesn't uniformly transfer across all conditions, only in some cases.

Verdict: This is an unusually faithful summary of a real, specific research paper — nearly every claim, including fairly granular details (six dimensions, dual-layer bank, no test-time labels, cost-competitiveness caveat), is directly traceable to the paper's own abstract and repo description.