The model never does the math

There is a demo everyone in this field has now seen. Someone asks a system a question in plain language, it thinks for a moment, and it produces a number. The number is correct. The room is impressed. And nobody in the room can tell you why it was correct, or what would have happened if it hadn’t been.

That gap is the whole problem. In most domains worth automating, a number carries consequences — somebody prices a trade on it, stamps a drawing with it, files it with a regulator. The person accountable for that decision cannot use a number whose provenance is “the model said so,” no matter how good the model’s track record is. Accuracy is not the missing property. Attribution is.

The fix is not a better model. It is a smaller job.

The boundary

The rule I design against is short enough to put on a wall: the model decides what to do; anything with a correct answer comes from code that computes it.

An agent reads intent, picks a course of action, calls things, and explains the result. It does not add, compare, aggregate, convert units, or decide whether a constraint was satisfied. Those go to a tool — a solver, a query, an existing service that some team has owned for years — and the tool’s output is what lands in the answer, unaltered.

This is more restrictive than it sounds, because it rules out a class of things that feel harmless. If the tool returns three numbers and the answer needs their sum, the sum is a fourth tool call, not a sentence the model composes. The moment a model is permitted to do arithmetic “just this once, it’s simple,” you have lost the property you were building for: that every quantity in the output can be traced to something that cannot be wrong about it.

The part that makes it interesting

Stated that way it sounds like a cage, and most writing about AI guardrails stops there — a list of things the model isn’t allowed to touch. But the constraint is worth more than the safety it buys, because of what it permits.

Consider a search. Some catalogue of options, a set of constraints, and a question: which one works? The model is excellent at the search — at reading a vague human request, deciding what to try, ordering the candidates sensibly, knowing when to give up. It is not qualified to decide whether any given candidate actually passes. But it doesn’t have to be, because each candidate is checked by the thing that owns that judgement.

So the model can roam. It can try a hundred options, follow a hunch, backtrack, pick up something the human never would have considered — and none of that is dangerous, because the search is the model’s job and the verdict belongs to the solver. A wrong hunch costs a tool call. It cannot cost you a wrong answer.

That inversion is what I find most people miss. The boundary is usually presented as the price of deploying AI in a serious domain. It is closer to the opposite: it is the thing that lets you hand a language model genuine latitude, because you have arranged for its mistakes to be cheap.

Why this beats “reduce hallucinations”

Most of the effort in this field goes into making a probabilistic system less likely to be wrong. Better prompts, better retrieval, better models, an evaluation suite to watch the error rate drift. All of it is worth doing. None of it changes the shape of the risk, which is that the system can be confidently wrong about a thing nobody can check, and the error rate only tells you how often that happened in the past.

Removing the opportunity is different from reducing the probability. A number that came from a solver is not “unlikely to be wrong” — it is wrong only if the solver is wrong, which is a bug with a stack trace and an owner.

I had this taught to me by my own site. There is a terminal on this page that answers questions about my work, and early on it confidently said a product of mine had never been acquired, while another page one click away said it had. It wasn’t hallucinating. The document it retrieved described the handover without ever writing the sentence. Retrieval answers from what you wrote down, not from what you meant — and I have come to like that failure mode, because when the system is wrong there is a file to blame and a diff that fixes it.

That is the same property, one layer up. Citations do for claims what tools do for numbers: they move the answer’s authority out of the model and into something you can inspect.

The rule has a third form, at the rendering layer, where I keep it absolute:

for (const segment of segments) {
  if (segment.bold) {
    const strong = document.createElement('strong');
    strong.textContent = segment.text;
    el.appendChild(strong);
  } else {
    el.appendChild(document.createTextNode(segment.text));
  }
}

Every line of model output on that page is built from text nodes. Nothing the model produces is ever parsed as markup, because a machine whose words I don’t control is writing into a page I sign. Same instinct as the arithmetic: decide in advance which powers the model simply does not have, and you stop having to reason about whether today’s output abused them.

What it costs

Three things, and they are real.

The product is narrower. There are questions the system declines because no tool owns the answer, and “I can’t compute that” is a worse demo than a confident guess. You are trading impressiveness for defensibility, which is an easy trade in a regulated domain and a genuinely hard one in a consumer product.

Every capability is plumbing. A model can be told about a new kind of calculation in a sentence; giving it a tool means a typed schema, a service that owns the logic, error handling for the case where the tool refuses, and a decision about what the agent says when it comes back empty. Capability arrives at the speed of engineering, not the speed of prompting.

And somebody has to hold the line. The pressure to let the model handle one small computation — just this field, just this once, the tool call isn’t worth it — never arrives as a bad idea. It arrives as a reasonable shortcut on a deadline, and it is the only way this design actually fails.

What you get for it is a system whose answers survive being questioned. In the domains I care about, that is the only kind worth shipping.