The LLM Is the Smallest Part of the Agent

A harnessed LLM agent. And why LLM is a very small part of the agent? Most people picture this as a model with tools bolted on. The real architecture inverts that relationship. The model itself is deliberately thin. Intelligence gets pushed outward, and the harness composes it at runtime. Three dimensions orbit the harness core: 𝗠𝗲𝗺𝗼𝗿𝘆 holds state the model shouldn't carry in weights or context. Working context, semantic knowledge, episodic experience, and personalized memory each have their own lifecycle. 𝗦𝗸𝗶𝗹𝗹𝘀 hold procedural knowledge. Operational procedures, decision heuristics, and normative constraints specialize the general model per task. 𝗣𝗿𝗼𝘁𝗼𝗰𝗼𝗹𝘀 hold the interaction contracts. Agent-to-user, agent-to-agent, and agent-to-tools are three distinct surfaces with their own failure modes. Between the core and these modules sit the mediators: sandboxing, observability, compression, evaluation, approval loops, and sub-agent orchestration. They govern how the harness reaches out and how state flows back in. The useful question this framing unlocks: for any new capability, where should it live? Stable knowledge goes to memory, learned playbooks go to skills, communication contracts go to protocols, loop governance goes to the mediators. Harness design becomes a question of what to externalize, and how to mediate it. I'm building a minimal agent harness from scratch. Didactic, easy to read, no magic. Open-sourcing it soon. Stay tuned.

Most people picture an agent as a model with tools bolted on. A more useful framing inverts that: the model stays deliberately thin, and capability gets pushed outward into a harness that composes it at runtime.

Three dimensions orbit the harness core:

Memory holds state the model shouldn't carry in weights or context — working context, semantic knowledge, episodic experience, and personalized memory, each with its own lifecycle.

Skills hold procedural knowledge — operational procedures, decision heuristics, and normative constraints that specialize a general model per task.

Protocols hold interaction contracts — agent-to-user, agent-to-agent, and agent-to-tools are distinct surfaces, each with its own failure modes.

Between the core and these modules sit the mediators: sandboxing, observability, compression, evaluation, approval loops, and sub-agent orchestration. They govern how the harness reaches out and how state flows back in.

The useful design question this unlocks: for any new capability, where should it live? Stable knowledge goes to memory, learned playbooks go to skills, communication contracts go to protocols, loop governance goes to the mediators