Synthèse
The easiest fix for inconsistent LLM output is to force temperature=0, which helps reduce randomness from the generation. But two of the three models in our SAGE pipeline don’t support that setting at all, so we had to solve our nondeterminism issue architecturally. At first we explored an alternative to plain-text comparison. Instead of checking if the new summary’s text was identical to the original, we considered checking if the new summary’s meaning was close enough to the original. It sound
The easiest fix for inconsistent LLM output is to force temperature=0, which helps reduce randomness from the generation. But two of the three models in our SAGE pipeline don’t support that setting at all, so we had to solve our nondeterminism issue architecturally. At first we explored an alternative to plain-text comparison. Instead of checking if the new summary’s text was identical to the original, we considered checking if the new summary’s meaning was close enough to the original. It sounded good on paper, but ultimately we rejected the idea because a genuinely important security change (like adding a new endpoint that’s reachable from outside the trust boundary) might only shift the similarity score a small amount, and would be indistinguishable from ordinary LLM-rephrasing noise.
For a security tool, silently missing that kind of change is worse than being too cautious. Instead, we narrowed what SAGE v2 considers a change. Rather than comparing everything within a directory's SCAFFOLDING. md file, we only track the SHA256 hashes of a small, fixed set of structural facts, — things like the files in the directory, their sensitivity ratings, and trust-boundary designations. If any of those hashes shift, we treat it as a real change worth propagating throughout the scaffolding.
The written analysis generated for humans is intentionally left out of the comparison because it's the piece most likely to come back worded differently from one scan to the next without any meaningful changes. We made one other deliberate choice: The LLM never gets to decide what counts as a change, either. The hashes themselves are computed by our Go harness, which is completely deterministic. The LLM no longer needs to infer what has changed, so we’re not asking the model to grade its own homework. Long story short, we let the structure decide, not the prose.
A new file in a directory is a real signal; a reworded summary of the same code is just noise.