Drift ·
A model should never be what decides something happened
Only what explains it afterwards. The difference is the whole architecture.
The honest problem with agent telemetry is volume. A fleet of agents generates more events in an hour than anyone will read in a week, and the obvious move is to put a language model in front of it and ask what matters. That instinct is right about the difficulty and wrong about the remedy.
It is wrong for a reason that only shows up under attack. A model asked to read evidence can be addressed by that evidence. If the input is prose — logs, summaries, tool responses, the output of another agent — then the input is also an instruction channel, and you have added a second model that can be talked out of reporting. The thing you brought in to audit inherits the failure mode it was auditing.
The way out is not to avoid models. It is to be exact about which job they do.
Detection has to be deterministic. A distance in a learned distribution cannot be prompt-injected. A byte count cannot be argued with. A rule matching a destination hash has no opinion about whether it should fire. This is also, conveniently, the part that actually scales: a baseline turns ten million events into forty deviations before anything reads anything at all. Statistics are not a compromise made because models are risky. They are the only thing that works at that volume.
The model's job is the last mile. Once deterministic machinery has isolated forty events, a model can turn them into a sentence somebody acts on at two in the morning. That input is small enough that a human could check the work by hand — which is the real test of whether you reduced the problem or just moved it. If you are handing a model ten million events and asking it to find the interesting one, you have not done detection. You have outsourced it to something that cannot be held to an answer.
And the model must not be able to change the verdict. It explains what already happened. It cannot un-flag it. The record exists before the model sees anything and is chained so that an altered entry is arithmetic rather than opinion. Its narration is commentary on evidence, never the evidence itself.
There is a stricter version of this worth taking seriously, and it costs less than it sounds. The explanation itself can be generated from the same primitives as the detection — byte counts, hostnames, tool names — rather than by a model. You lose some fluency. You gain the guarantee that the sentence handed to an on-call engineer cannot contain a consequence the evidence does not support, which is precisely the failure you were trying to catch.
None of this is an argument against using models in security. It is an argument about where they sit. Put one at the point of decision and its mistakes become your findings. Put one after the decision and its mistakes are bad prose, which is a survivable class of error.
The short version: a model should never be what decides something happened, only what explains it afterwards.