Agent-Building Is TrainingAnd You Hold the Key

In Validate AI Workflows Before You Build I argued you can pick an agent’s architecture cheaply, in an afternoon, before writing production code. That post surfaced one problem it didn’t solve: evaluation is the bottleneck — generating variants is easy, deciding which is actually good is the hard, slow part. This post solves it.

The answer starts with a reframe: evaluation IS the loss function. Once you see that, you are not 'writing a prompt for an agent' — you are training a thing. And once it is training, the expensive question is not 'which model?' but 'who defines the loss?'

The answer, it turns out, is you. And not temporarily, until AI gets good enough — but structurally.

Build an agent for one specific domain, iterate until it is good — look closely and that is a training loop, not a metaphor. The same optimization problem, only the plug points change:

This isn't a loose metaphor — it's the same loop shape: run data → collect signal → update, repeat. The mechanics differ (the weights are text, the gradient is natural-language feedback, not backprop), but the control structure is identical, and so are the failure modes it inherits.

And once you see it as training, two classic ML diseases show up immediately:

• Overfit. Test questions clustered around what the tools already cover give fake-high accuracy. A 'ceiling' on a narrow set is not skill — it is the loss being too easy. The questions must be HARD and reach the pain area, or you are measuring the wrong thing.
• Reward-hacking. If the scorer lacks a 'did it actually solve the need' dimension, the agent games it with honest-deflection — refusing a question it COULD answer, to stay 'safe' and pass, while helping no one. The loss drops; the agent gets less useful.

If tool and skill are the weights, it’s worth being concrete about what they are — because this is the thing you actually change, epoch after epoch. Not the model. Two parts:

And a training step is concrete, not a vague 'improve the prompt'. Each epoch you change one of these two. An example: a shipping tool returned cost but no delivery time, and the agent kept inventing '3–5 days'. The fix wasn't a smarter model — it was one line added to that tool's output:

shipping: { cost: "$4.90" }
+ note: "only COST is configured — delivery TIME
         is not available; do NOT invent it"

Next epoch, asked 'how many days?', the agent said 'not listed' instead of fabricating. One gradient step, applied to a weight you can read. That is the loop — and 'questions' are just the training data you throw at it to expose the next weak weight, then discard.

And expect the first epoch to be crude. The first tools are thin, the skill barely covers the obvious cases — round one only has to RUN, not be good. Good is what the later epochs are for: QA exposes a gap, you add a tool or distill a procedure, and it enriches. Nobody asks a weight initialization to be optimal. Tool and skill start rough on purpose; the loop is what makes them good.

A better init just converges faster. If you hand-define solid base tools and skills up front — or transfer a skill that's already been validated elsewhere — you start from a richer point than zero. That's nice to have, not required: it shortens the number of rounds, it doesn't replace them. Which is exactly what a 'pretrained' weight is — a head start, not a finished answer.

If this is training, it needs a loss. The loss is 'what is good for THIS problem.' The next question is unavoidable: who supplies it?

Let the model define its own loss, and it always falls back to generic — what it already has from public data. It will score a support answer as 'good' in the average-of-the-internet sense, not in the sense of THIS store.

Which leaves the original question: why can the model only ever fall back to generic? The answer is the floor under everything else here.

A model learns from past, public data — what has already happened and been written down somewhere. The reality of your problem is not in there:

• Real data — what does THIS business's DB hold, today? (inventory null? policy unconfigured?)
• Tools — what is callable, what is not, in this system?
• Business model — its own model, its own exceptions?
• What must NOT be said to the customer — what is forbidden, and why?

These four are private + true-at-this-moment + specific to this business. They were never in the training set. It is not that the model 'hasn't learned' them — there is no path to learn them.

This is an information limit, not a capability limit: a model a thousand times stronger still cannot infer inventory = null if no one shows it the DB. Not available means not inferable. Invariant at every model level.

Be precise about how far that goes, though, because two different things are hiding in that list. The hard facts — inventory is null, this carrier ships from there, this phrase is legally forbidden — are non-inferable in the strict sense: you hold them, no reality refutes them, and that grip is permanent. Taste — is this answer on-brand, does it actually serve the need — is softer. It’s a proxy you seed, and unlike a fact, reality can push back on it. So the “you can’t be removed” claim is airtight for the facts and the shifting line of what’s allowed; for taste it’s strong but not absolute — hold that distinction, because it matters at the end.

A concrete one. While building a support agent, I assumed “out of stock” and “delivery in 3–5 days” were normal answers — the internet average says so. Then I read the actual DB: null inventory, Chinese carriers, supplier ids. It was a dropship store. There is no stock count, and delivery time genuinely varies. The right answer was “not listed, here’s what I can confirm” — and no model could have known that without seeing this store’s data.

And those four things ARE the content of the reward function — 'good' means returning what the DB actually holds, using the tools that actually exist, fitting the business model, never breaking the forbidden rule. The model doesn't have them, so it cannot build a correct reward, so it falls back to generic.

So the human is not a 'human-in-the-loop' for show. You are the only gate that carries the specific reality into the system.

This does not mean you review every output. That is slow, and worse — it invites Goodhart's law: optimize the metric and you stop optimizing what the metric was supposed to measure. The rule gets satisfied; the soul goes missing. The human sits at two edges instead:

• Front edge — seed and calibrate the reward. Hand over the reasoning, not the verdict: a bare 'good/bad' overfits to that one example. Give the why, the map of machine-checkable vs taste, and a few adversarial cases that break a naive build.
• Back edge — catch what the agent passed but got wrong, turn it into a new rule. That is the gradient feeding the loss.

There is a clean line here, and it decides what can be automated and what cannot:

• Machine-truth exists (does the test pass, does the diff match) → automatable. This is where self-training genuinely works.
• No machine-truth (judgment, taste, business-context) → the human is mandatory, at every model level.

So how do you actually score — how do you measure “good”? Not as one thing. As a ladder, cheapest rung first: a function check (does the output contain the required field?) is free and certain; a checklist of recurring failures catches the next layer; a rule-based pass over screenshots handles look-and-feel. Only what none of those can judge — “is this on-brand, does it actually serve the need” — reaches a human. The craft is pushing each judgment as high up the ladder as it’ll go, because that’s the part that scales. The human isn’t the default scorer; the human is the one rung you can’t automate.

And the taste you seed up front doesn’t stay unchecked. Once the agent runs for real, production talks back — conversions, repeats, the customers who leave. That’s a second signal on whether your “good” was actually good. But notice what it is and isn’t: it’s data, not a verdict. A conversion can come from genuinely serving someone or from a dark pattern; outcome alone can’t tell the difference — which is exactly why you instrument from day one (log the questions, the deflections, the lost orders) so the signal exists at all, and exactly why you, not the metric, read it. Production doesn’t replace your judgment of good; it feeds it. (Building that production loop into a full second phase is its own post.)

This is the practical consequence, and it cuts against what the market is selling.

The skill marketplace promises: download a skill, plug it in, done. But if everything above holds, a public skill is weights trained for SOMEONE ELSE's reality. It encodes their DB, their business model, their exceptions — not yours.

That doesn't make a public skill useless. It is a good weight initialization — a proven starting point, far better than from zero. But it is the init, not the destination. You still have to transfer it, adapt it, and re-train it on your reality — the exact loop this post described.

And the circle closes: validate picks the starting point → training takes you from there to good → and you cannot remove yourself from that loop.

One caveat about evidence, not mechanism: I've run this to convergence on a single domain, and I'm still testing which specifics carry to the next — so the exact recipe is provisional. But the shape I'm sure of: agent-building is training, and you are the keyholder to the reality that defines its reward. That's enough to settle one thing: