All Playbooks
Field Manual6 min readUpdated November 8, 2026

What You’re Actually Buying When You Buy an AI Agent

The model gets the attention, but the software around it decides what an agent can reach, when it stops and how you find out what it did. Here is what that layer is, and eight questions to ask about it.

Published 8 November 2026. Sources checked on this date.

In April 2026, the founder of a car-rental software startup, PocketOS, reported that a coding agent had deleted the company’s production database and its volume-level backups in about nine seconds. The agent, Cursor running Claude Opus 4.6, had hit a credential problem in a staging environment and gone looking for a token. It found one in an unrelated file. That token had been created to manage custom domains, but according to the founder it was “scoped for any operation”. The Register’s account

The data was recovered, and the parties disagree about where the blame lies. The hosting company, Railway, said it honours authenticated delete requests, and that the call reached an old endpoint without its delayed-delete protection, which it has since patched. Most of this rests on the founder’s and Railway’s own statements. The incident record

Notice what the argument was about. Nobody was debating how clever the model was. They were debating what it could reach, whether a destructive action needed confirmation, and where the backups lived. That surrounding layer has a name: the harness.

What a harness is

LangChain’s shorthand is Agent = Model + Harness: the harness is all the code, configuration and execution logic that isn’t the model. LangChain’s anatomy of a harness

Think of the model as a capable new hire. The harness is the job description, the system logins, the checklists, the manager who signs things off and the record of what got done. In practice, it includes:

  • Instructions: what the agent is told about its job and its rules.
  • Tools and permissions: which systems it can use, and whether it can only read them or also change them.
  • Memory and context: what it sees on each step and what carries over between sessions.
  • Checks: tests, validation rules and second reviewers.
  • Stops: budgets, step limits, human approvals and escalation paths.
  • Logs: a record of each action, so someone can reconstruct what happened.

Birgitta Böckeler, writing on Martin Fowler’s site, offers a useful split. Guides steer the agent before it acts, while sensors observe the result so it can correct itself. Some of these controls are ordinary deterministic software, such as tests and linters. Others use AI judgment, which is slower and less predictable. Harness engineering, explained

Why everyone is suddenly talking about it

Much of the interest comes from the labs’ own reports. In February, OpenAI described building an internal product with no hand-written code: roughly a million lines and 1,500 merged pull requests in five months, starting with three engineers. It says early progress was slow because the environment was underspecified, not because the model was incapable. It also warns that the result depends on that repository’s structure and may not generalise. OpenAI’s report

Anthropic described similar problems on long-running work. Its agents tried to do too much at once, ran out of context mid-feature and declared work finished too early. The fixes were unglamorous: a progress file, a feature list, one feature per session and end-to-end testing. Anthropic’s write-up

The most quoted number comes from SWE-Bench Mobile, a benchmark built from a production iOS codebase. Across 22 agent and model combinations, the same model showed up to a 6× performance gap depending on the agent around it. Often missed is the other finding: even the best setup completed only 12% of tasks. The SWE-Bench Mobile paper

That 6× figure is often credited to a Stanford study. The Stanford-led Meta-Harness paper does repeat it, but cites the benchmark above as the source. Its own contribution is automating the search for better harnesses, with strong coding results, though it tuned and tested on the same 89 tasks, as its authors acknowledge. The Meta-Harness paper

A smaller study makes the point from another angle. In a planning agent built for a Battleship-style game, a declarative planning layer added 24.1 percentage points to the win rate without making a single model call. One game proves little about your business, but it’s a reminder that plain software can do much of the heavy lifting. The planning-agent study

A harness is not a free upgrade

More harness doesn’t automatically mean better. In one Anthropic experiment, a full multi-agent harness ran for six hours and cost $200, against 20 minutes and $9 for the model working alone. That was one prompt, not a general rate, but it shows that coordination has a price. Anthropic’s harness experiment

Parts of a harness also go stale. Anthropic dropped context resets and later other scaffolding once newer models no longer needed them, arguing that each component encodes an assumption about what the model can’t do alone. LangChain likewise expects models to absorb some harness functions, while expecting harness work to stay useful. Anthropic on removing components LangChain’s view

So treat the harness as something to maintain, not a one-off build. When the model changes, re-test what the harness is doing and remove what no longer earns its cost.

What it looks like in practice

Illustrative only: this is a hypothetical policy for an agent that chases unpaid invoices, not a real product’s configuration.

{
  "agent": "invoice-reminders",
  "tools": {
    "accounting.read_invoices": "allow",
    "email.draft_reminder": "allow",
    "email.send_reminder": "require_approval",
    "accounting.delete_invoice": "deny"
  },
  "limits": {
    "max_steps_per_run": 20,
    "max_cost_per_run_usd": 2
  },
  "escalate_to_human": [
    "amount_over_5000",
    "customer_disputes_invoice"
  ],
  "log": "every_tool_call"
}

The important part is where these rules live. An instruction in a prompt is a request the model can misread or ignore. A rule enforced by the system holds whatever the model decides. Even then, the credential must match the rule. In the PocketOS account, the danger came from a token that could do far more than its stated purpose. The Register’s account

Eight questions to ask before you buy or build

  • What can it access, and with which credentials? Ask for the scope of each one. A shared admin key is a red flag.
  • Which actions need a person, and where is that enforced? “We told it to ask first” is weaker than a block in the software.
  • How is “done” verified? The agent’s own report isn’t evidence. Anthropic saw features marked complete too early.
  • What happens when it goes wrong? Ask about retries, rollback and whether backups sit apart from what they protect.
  • Can I see what it did? You should be able to replay any run: inputs, tool calls, outputs and approvals.
  • How was it tested on work like ours? Ask for results on representative cases, not a vendor benchmark. The best SWE-Bench Mobile setup managed 12%.
  • What does a completed task cost? Include retries, review time and failed runs, not just the price per token.
  • What happens when the model changes? Ask whether it can be swapped, who re-runs the tests and who removes scaffolding that is no longer needed.

The practical takeaway

When someone pitches you an agent, the model name tells you little about how it will behave in your business. Models can usually be swapped. The access rules, checks and records are what you’ll live with day to day.

You also don’t need a sophisticated harness on day one. A narrow, read-only agent with a few hard limits and a good log is a sensible first version, and you can widen its permissions as it earns them.

If you want to test one workflow properly before committing to anything bigger, that’s what our AI automation pilot is for.