The LLM call you can delete

Last week my agent system silently archived a card pre-authorization alert for $1,066. Two days later it held a $251 alert from the same bank in front of me, flagged as a material charge I needed to see. My policy does not distinguish those two notices. Both are routine card activity. The model invented a materiality test I never asked for, and it applied that test in opposite directions two days apart.

The frustrating part is that the correct rule was already written down. My communications system keeps a file of routing overrides. Routine card activity from that bank is informational: file it, archive it, create no work. The skill that classifies my messages told the model to load that file and apply it. I had written policy as data, then hired a language model as its interpreter.

That is an expensive interpreter with a wandering mind. So I measured what it would cost to fire it.

Replaying the decisions

My classification pipeline logs every decision it makes, and it logs my corrections. That turns out to be everything an evaluation needs. I replayed all six routing rules against 631 logged decisions from the pipeline’s shadow period and scored them against the final, corrected outcomes.

The rules matched 13 decisions. On those 13, the rules disagreed with my corrected decisions zero times. The model disagreed three times. All three of its misses were on decisions I had corrected by hand, which means the rules sided with me and the model sided with its own invented judgment.

That inverts the usual way people validate this kind of replacement. The standard approach trains or writes a substitute, then measures agreement with the model it replaces. But where corrections exist, agreement with the model is the wrong target. My corrections mark exactly the places where the model got my policy wrong. A replacement that faithfully reproduces the model reproduces those errors. A replacement that encodes the corrected policy will score worse on agreement while being better at the actual job.

The honest number

Thirteen decisions with zero errors sounds like a win. It certifies very little. With that sample size, the exact statistical ceiling says only that the true error rate is below 20.6 percent. Nobody should deploy on that sentence.

So the deployment is staged. The rules now run as code in front of the model, and the model still reviews every matched decision and records whether it agrees. Corrections and disagreements accumulate as evidence. At 60 covered decisions the ceiling drops below 5 percent. Until then, the system refuses to claim more than the data supports. A validation method that cannot say “not yet” is a rationalization method.

The bigger number is the ceiling above this. Nearly three quarters of my email classification decisions end the same way: no reply, no task, file the message. Six hand-written rules cover eight percent of that traffic today. The rest is sitting there, one honest replay away from being ordinary software.

What shipped

The rules in my own stack are now executed by a small deterministic program, not by the model reading a JSON file. Every email decision records the exact inputs the rules need, so future evaluations replay exactly instead of reconstructing.

The evaluation method went upstream. TRACER is an open-source library that replaces LLM classification calls with small trained models behind a statistical gate. Its gate had two of the problems my replay surfaced: a pooled threshold that lets a high-volume class hide a failing minority class, and a small-sample path that quietly picked the worst passing answer. I submitted a patch with per-class calibration and an explicit refusal when a class lacks the evidence to certify: PR #80.

And the method is now a short paper on when an LLM call can be replaced by code, and how to prove it with the traces you already have: “When Can You Delete an LLM Call?” (preprint PDF).

This is the other half of the verification work I wrote about in the obligations essay. Contracts catch an agent claiming an outcome the world does not show. Deterministic offload removes the variance before it happens, for the decisions that were never judgment calls to begin with. The models keep the judgment. The code keeps the bookkeeping.