Static analysis and LLMs are a good pairing. I'm not here to argue otherwise. But a lot of teams are wiring them together in a way that quietly turns the scanner's evidence into the AI's opinion, and then treating the opinion as the record.
That's the part worth talking about.
The demo vs. the pipeline
In the demo, it looks great. The scanner dumps 400 alerts on a pull request. An assistant reads them, groups them into twelve clusters, writes a plain-English paragraph for each, and marks the ones it thinks are noise. The reviewer sees twelve tidy cards instead of a wall of rule IDs. Everyone relaxes.
In the pipeline, six months later, nobody can tell you why a SQL injection finding in orders/export.py was closed. The ticket says "Resolved: input is validated upstream." There's no link to the upstream validation. The original rule ID, file, line, and data-flow trace are gone, because the summary replaced them.
The scanner and the assistant have different jobs. The scanner says: this rule was violated at this location, and here is the path that got there. The assistant can say: here's what that means, here are three other alerts that are probably the same root cause, and here's where I'd look first. Problems start when the second job silently absorbs the first.
Failure mode 1: the explanation becomes the verdict
Illustrative example. A taint-tracking rule flags this:
def export(request):
fmt = request.GET.get("format", "csv")
query = f"SELECT * FROM orders WHERE tenant = '{request.user.tenant}' ORDER BY {fmt}"
return run(query)
The assistant reads it and writes: "The tenant value comes from the authenticated user object and is not user-controlled. Likely false positive." That sentence is correct about tenant. It says nothing about fmt, which is the actual sink the rule fired on. The summary was plausible, focused on the wrong variable, and got accepted because it read well.
An explanation is a hypothesis about a finding. It is not evidence that the finding is resolved.
Failure mode 2: grouping hides the odd one out
Clustering is genuinely useful. Forty "hardcoded credential" alerts that are all test fixtures should collapse into one decision.
But clusters get dismissed as a unit. If 39 of those 40 are fixtures and one is a real key in deploy/config.py, a "dismiss cluster" button throws it out with the rest. The grouping logic needs to show you what makes each member similar, and the dismissal needs to be recorded per finding, not per cluster.
Failure mode 3: the evidence doesn't survive the summary
If your triage flow stores "AI says false positive" and drops the rule ID, location, trace, and the code snapshot it was judged against, you can't audit it later. You can't rerun the check against the fixed code. You can't notice that the same pattern came back in a refactor.
Keep the original finding as the source of truth. Attach the AI explanation next to it, labeled as an explanation. Close the finding only when a rescan of the changed code no longer reports it, or when a human records a reason that points at something concrete.
Failure mode 4: you're measuring throughput, not accuracy
"The assistant processed 12,000 alerts this quarter" is a volume metric. It tells you nothing about whether the right ones were dismissed.
The number that matters is false dismissals: findings the AI (or a human following the AI) closed that were real. You only learn that rate by sampling closed findings and re-reviewing them, or by tracking findings that were dismissed and later reappeared as incidents or re-detections. If you aren't measuring it, you don't know whether your triage got better or just got faster.
So what does a sane setup look like?
- Scanner output is immutable. Rule, location, trace, commit.
- AI output is an annotation on that record, never a replacement for it.
- "Resolved" requires either a clean rescan or a human reason with a pointer to code.
- Cluster dismissals expand into per-finding records.
- You periodically sample dismissed findings and track how many were wrong.
None of this is exotic. It's the same discipline you'd apply to a junior reviewer: helpful, fast, often right, and not the one who gets to sign off alone.
AI really does make security signals easier to understand. The evidence still has to survive the summary.
How are you handling AI-assisted triage on your team? Do you track false dismissals at all, or is "alerts processed" the only number anyone looks at?
Top comments (75)
The cluster example is the sharpest one here: 39 fixtures hiding one real key in deploy/config.py. What would make a dismiss button show that odd one out before it lets the whole cluster go?
I’d make the button show the outliers before allowing a bulk dismissal. Things like different file paths, production configs, or secrets that don’t match the rest of the cluster should get flagged for individual review. And honestly, if even one finding looks different enough, it should be excluded from the bulk action by default. The goal is to make dismissing 39 findings easy without making the 40th disappear with them.
That default exclusion is the key part for me. A bulk button should not need a second look to catch the one file that does not belong.
Exactly. The default should be conservative, not make the reviewer responsible for spotting every outlier in a long list. If the system isn't confident a finding belongs in that cluster, leave it out of the bulk action and let someone review it separately. A few extra clicks are a much better tradeoff than silently closing a real vulnerability.
Leaving the uncertain ones out of the bulk action is the safer default.
Yeah, and it doesn't have to be complicated either. Keep the bulk action for findings that clearly meet the same criteria, and make exceptions visible instead of silently including them. I'd rather have a reviewer investigate a few extra alerts than explain later why one real secret slipped through.
Making the exceptions visible is the useful part there. A reviewer can check a short visible list, but a silent inclusion leaves nothing to check.
Exactly. And I'd go one step further: show the reviewer not just which findings were excluded, but why they were excluded. That makes the clustering logic easier to trust and easier to challenge when it gets something wrong. Bulk dismissal should always be explainable, not just convenient.
The why is what makes the list usable. A reviewer can challenge a reason, but a bare excluded list just asks them to trust the cluster.
Agreed. The reason should be specific enough to verify, not just “low confidence.” Something like “different file path,” “production config,” or “secret pattern doesn’t match the cluster” gives the reviewer a concrete basis to challenge the grouping.
The file path example is the one that makes it checkable. A vague reason just moves the trust problem somewhere else.
Exactly. The reason should be grounded in the actual finding data, not generated as another AI explanation. If the file path differs, show the path and the cluster’s common pattern side by side. Then the reviewer can verify the exception instead of having to trust another summary.
side by side is the part that makes it work. a reason you can't check is just another summary.
Exactly. The reviewer should be able to see the difference in the underlying data, not just read an explanation of it. If the system can't show what makes a finding different, it shouldn't be confident enough to include it in a bulk dismissal.
if the data can't show the difference, the bulk dismissal shouldn't happen.
Exactly. That’s the boundary I’d draw: no verifiable distinction, no bulk dismissal. The system can still group the findings for review, but it shouldn’t turn uncertainty into a decision to close them.
The fmt-versus-tenant example is the sharpest failure mode here — the summary was correct about the wrong variable and got accepted precisely because it read well. That is why "AI as annotation, never replacement" is the load-bearing rule; a summary that can't point at the rule ID and trace is an opinion wearing a verdict's clothes. Do you track false dismissals by sampling closed findings, or is that the part teams skip because the throughput number looks too good to question?
That's exactly the problem with throughput metrics. They tell you how quickly findings disappear, not whether the decisions were correct. I'd track sampled closed findings and re-detections separately, then break false dismissals down by rule and severity. Even a small recurring review sample is better than assuming the system is accurate because it produces convincing explanations. The hardest part is making teams measure the decisions they got wrong, not just the work they processed.
Separating sampled closed findings from re-detections is the right split — a wrongly dismissed finding that comes back is the one metric throughput can't hide. The sampling question I'd push on: random for an unbiased rate, or severity-weighted so the sample spends its time where a miss hurts most? And the cultural part you name is the real blocker — a team has to be willing to publish its own miss rate before any of the measurement machinery matters.
I'd use both. Random sampling gives you a less biased picture of the overall false-dismissal rate, while severity-weighted sampling makes sure critical findings get enough scrutiny. I'd keep the results separate rather than combine them into one number. And agreed on the culture part: if teams aren't willing to make their misses visible, better measurement won't change much.
Both, kept separate, is the honest answer — one blended rate would let either sample hide behind the other. And you're right that the culture point is the ceiling: a team that won't publish its misses will just tune the sample until the numbers look good. What does publishing misses look like in practice for you — a shared dashboard, or something inside the review itself?
I'd put it in both places, but make the review record the source of truth. Each sampled finding should record the outcome, rule, severity, and reason for any incorrect dismissal. A dashboard can then show the aggregate rates and trends without hiding the individual cases. The important thing is that misses remain visible and traceable, not just reduced to a percentage.
Making the review record the source of truth is the right call — a dashboard that only shows rates lets a team celebrate a falling percentage while the same miss repeats quietly. The part I'd watch is whether the record feeds back into the rules themselves: does a sampled miss with its reason attached trigger a rule change, or does it stay visible without ever changing what the system does next?
It should feed back into the system, otherwise the review becomes an audit trail without a learning loop. If a miss points to a recurring pattern, it should trigger a change to the rule, clustering logic, or dismissal criteria, then be tested against the original finding and similar cases. I’d still keep the review record immutable, so improving the system doesn’t rewrite the history of what went wrong.
Immutable record plus a learning loop is the combination that makes this work — visibility without feedback is just a museum of misses. Testing a rule change against the original finding and similar cases is the part I'd hold onto hardest, since a fix that only passes its own incident hasn't proven anything. Who owns that trigger in practice — does the reviewer file the rule change, or does the system propose it from the pattern?
Automating vulnerability triage with an LLM usually boils down to teams hiding an uncalibrated SAST noise floor behind non-deterministic text generation. You replace an instantaneous, zero-cost deterministic AST check with an expensive inference loop—spiking pipeline latency by 1.2s to 3.5s per finding, hogging 16–24GB of VRAM on inference nodes or burning $0.02 to $0.05 per API call—all to get an eloquent hallucination that flags a real tainted-data flow or memory corruption path as a "harmless false positive."
If you're using models in security triage, keep them advisory and strictly gated behind deterministic verification: have the model propose a minimal reproduction test or patch, spin it up in an isolated cgroup or throwaway container, and only alter ticket state if the test provably passes against the binary. When teams bypass that feedback loop and wire an LLM straight to the close button, what trade-off are they actually accepting—shaving minutes off a sprint review at the cost of your audit trail and real attack surface?
Bohdan Hordii // bohdanhordii.dev
I agree with keeping deterministic verification in the loop, especially when the model proposes a patch or reproduction test. I'd be careful about making a passing test the only route to closure, though. Some findings are resolved through configuration changes, architectural controls, or removing the vulnerable code entirely. The common requirement should be verifiable evidence tied to the exact commit, not necessarily one specific kind of test. The model can help get there, but it shouldn't own the final decision.
Two separate problems in your last replies, and they need different fixes.
A stable identity keyed on normalized location still moves. Split a file, pull a function into its own module, extract a control flow: the code relocates and the finding is the same weakness. Rule plus location turns a refactor into a new finding and orphans the old dismissal. Rule plus the artifact the check actually produced (the matched AST shape, the taint path, the fixture that fired) survives the move, and it is the thing a second reviewer can rerun. That is what makes evidence at closure checkable instead of merely mandatory. "Validated upstream" is prose; a check, its artifact, and the commit are a record.
You are also right that a passing test cannot be the only exit, because a config change and deleted code have no test. The common requirement is an artifact that ties the closure to that commit, whatever produced it.
Whether a claimed fix actually happened is the work I do. If you want the longer version aimed at Cyclopt's pipeline instead of the general argument, it is $25, agree first, no link until you say yes. The thread already has the substance; the paid part is the working design.
That's a fair distinction, especially the point about identity surviving refactors. Keying findings to the actual evidence artifact rather than the file location is a much stronger approach. Otherwise, you lose continuity exactly when the code changes.
And I agree that the closure mechanism needs to support different kinds of fixes. A deleted code path, a config change, and a patched vulnerability won't all produce the same kind of test result. The important thing is that each closure has reproducible evidence tied to the commit. That's the part I'd want to get right before automating any of it.
You have answered both of my comments and half this thread, and not once have you said what Cyclopt records when something closes. So one question, no theory: pick a finding you closed last month and tell me what exists for it today. A comment on the ticket means the record is the explanation, and every rule in this thread passes it. A rerun of the check at that commit means you already have the thing I am describing, and this is only about reading it back.
Fair question. I should separate what I'm arguing for from what I can actually claim Cyclopt records today. I don't want to invent an example or pretend I've inspected a closure record that I haven't. The concrete test is simple: can we take a finding closed last month, recover the original scanner evidence, identify the commit it was assessed against, and reproduce the verification? If not, that's exactly the gap this discussion is about.
Then the test is the answer to your own post, and it is cheaper to run than another round of rules. A close that passes it is a record, not a summary: the raw scanner output, the commit the finding was assessed against, and the command that reproduces the verdict. Any one of the three missing and the closure is an opinion with a timestamp, which is the thing you are arguing against.
Keep one thing honest when you run it. A rerun that comes back green today proves the symptom is gone, not that the dismissal was right when it was made. Two different claims, and reviewers mix them. The record needs both times: when it was dismissed, and when it was rechecked.
I can run that on one record you name. You send the export, I write back what it proves, what it only asserts, and the first three questions a reviewer would ask of it. $25. Say yes and I send the terms.
That distinction between the original decision and a later recheck is important. A green rerun today doesn't prove the dismissal was justified at the time; it only shows the current state. The record needs to preserve what evidence supported the decision then, which commit it applied to, and what a later verification established. Those should be two separate, traceable events.
The important boundary is between triage and disposition. AI can cluster evidence, propose severity, and draft a remediation plan, but closure should require an explicit control: a reproducer or compensating-control check, an accountable owner, and evidence that the condition is no longer exploitable. Otherwise the summary becomes a very polished way to hide uncertainty.
I like that distinction between triage and disposition. That's really where the boundary should be. The AI can help prioritize and explain, but someone or something still needs to establish that the underlying risk is gone. I'd also make the evidence requirement depend on the type of resolution, since a patch, a compensating control, and removed code won't all have the same verification path.
The triage/disposition split is the right operational boundary. A close gate should require a linked remediation artifact—merged diff, configuration change, or accepted risk with expiry—so an AI summary can accelerate investigation without becoming closure evidence.
The field your gate is missing is the re-open path. An accepted risk with an expiry date is a promise the calendar keeps only if something reads the date: a scheduled job that reopens the finding when it passes, tied to the same evidence record. Without it, expiry becomes the quietest way to close a finding twice.
A receipt from outside security tooling: macOS keeps two trust stores, the system keychain and the flat cert.pem that curl and git actually read. After a CA is blocked, "revoked" and "still trusted" are both true until an OS build replaces the file. A control that expires and a store that never re-reads it are the same failure with better paperwork.
That’s a good point. An expiry date only matters if there’s an actual mechanism behind it. I’d treat accepted risk as a time-bound state, not a permanent closure: the finding should automatically reopen when the deadline passes, with the original evidence and owner still attached. Otherwise the record says the risk was temporary, but the system has no way to enforce that.
The date needs a reader with a name. Who reads it on a Sunday, and what happens to their queue when the deadline lands while they are away? If the honest answer is nobody, the expiry is a note to the next auditor, not a control.
The fmt example suggests a cheap mechanical check: make every proposed close name the variable and line from the scanner's own trace, and reject it when the explanation argues about something that is not on that trace. Here the summary reasoned about tenant while the rule fired on fmt in the ORDER BY. It will not catch every mistaken verdict, but it catches the case where the reasoning is about a different variable than the finding. Does your pipeline keep the trace in a form an explanation could be checked against like that?
That's a good check. The wrong-variable case is exactly the kind of mistake we should be able to catch without relying on the reviewer to spot it. If the explanation can't point back to the actual sink and trace that triggered the rule, it shouldn't be allowed to recommend closure. It won't prove the finding is valid, but at least it stops the AI from confidently answering a different question.
Exactly, it doesn't prove the finding, it just stops the summary from answering a different question. That alone would have saved the case in your post.
Right. It’s a small guardrail, but a useful one. Before debating whether a finding is a false positive, make sure the explanation is actually addressing the finding the scanner reported. That should be the minimum bar, not the final verification.
The distinction between volume and false-dismissal rate is useful. Do you keep a sample of closed findings tied to the rule ID and commit so reviewers can reconstruct the original evidence after the code changes?
That's exactly the kind of traceability I'd want. A sampled closure should retain the rule ID, original scanner trace, and commit it was assessed against, so a reviewer can reconstruct what was actually evaluated even after the code changes. I wouldn't want to claim every system already preserves that history, but it's a good standard to measure against.
Yes, and I would put it on every close rather than on a sample. A sample is a second system: an export job, a retention window, someone who remembers to refresh it. Rule ID, scanner trace and commit on the dismissal record itself, and then the sample is a query instead of a pipeline.
The reason belongs in the same place. If it can't be shown from the finding's own data, it isn't a reason yet.
The part I'd want to hear about is what the record is keyed to. A trace that names line 412 does not survive the next commit. What did you end up storing, the raw scanner output or something normalized?
The line that hit me: "An explanation is a hypothesis about a finding. It is not evidence that the finding is resolved."
I felt this deeply, just from the technical writing side. I recently had an article rejected by an editor because it read too much like an AI summary.
My original draft perfectly explained what the Python code did and summarized the steps. It looked clean, just like those 12 tidy cards in your example. But the editor pointed out that it lacked the actual "evidence" of a human developer: the bugs, the struggle, and the "why" behind the code. I had summarized the syntax but stripped out the engineering reality.
I had to delete the whole thing and rewrite it, adding the concrete proof of my own troubleshooting.
"Plausible" is not the same as "Correct." Whether we're talking about a security scanner dismissing a vulnerability or a writer summarizing a codebase, the raw evidence has to survive the summary. Brilliant breakdown.
That's a great parallel. You can have a technically accurate summary and still lose the most important part: the reasoning and evidence behind it. The bugs, the decisions, and the things that didn't work are what make engineering writing credible. Glad you brought that perspective into this discussion.
Based on my experience and recent development, an elegant AI explanation is not a security fix.
To prevent fluent hallucinations from closing real vulnerabilities, we must enforce a strict "Deterministic Exit" framework across its automated security lifecycle, locking down how findings are triaged and archived:
Mechanical Symbol & Sink Verification: An automated triage helper applies an input-matching rule. If the assistant's written analysis fails to explicitly reference and reason about the exact vulnerable Sink and token trace flagged by the scanner (e.g., arguing about a clean variable while missing the tainted parameter next to it), the system rejects the recommendation, forcing manual human oversight.
AST-Based Artifact Identity: To stop refactoring from creating "identity drift" and orphaning old baseline records, keys vulnerabilities to the unique abstract syntax tree (AST) shape and data-flow trace rather than mutable line numbers. If code moves to a different module, the tracking context survives the refactor, preserving the continuity of the original decision.
The Re-Detection Alert Loop: measure security by verified accuracy, not by uncalibrated throughput. A system that logs and surfaces High-Attention Re-Detections if an active scanner fires on a code path previously dismissed on an assistant's note. This provides engineering leads with a true, sampled signal to audit and measure our false dismissal rate.
This is a strong framework, especially the idea of treating closure as a deterministic state transition rather than a language-model decision. I'd add one nuance around AST-based identity: syntax shape and data-flow context are much more useful than line numbers alone, but they still need to be tied to the actual check artifact and commit. Refactors can change the structure too.
The common thread across all four points is that every closure needs evidence someone else can reproduce and challenge. AI can make triage faster, but it should never get to define what counts as proof.
I think that one possible evolution or at least the one that I used in one of my open source project that can be checked by the following :
The fmt example is the whole problem in one snippet. The summary was right about a variable the rule never fired on. One check I'd add: make the AI explanation name the exact sink and trace hop it's reasoning about, and reject any "likely false positive" that doesn't reference the same line the rule flagged. That catches the wrong-variable case mechanically instead of hoping a reviewer notices.
On false dismissals, the cheapest signal I've found is re-detections. If the same rule fires in the same function after a refactor and the earlier finding was closed on an AI note, that's a dismissal you go back and grade. It's not a full accuracy rate, but it's free and it points straight at the bad closes.
Re-detections are the right free signal, and they grade the code more often than they grade the close. The same rule firing in the same function after a refactor can mean the earlier close was wrong, or that the sink moved out from under a check that was fine. Without the check that closed it and the commit it ran at, you cannot tell those apart, so the re-detection points at a bad close without convicting it. The commit is what turns go-back-and-grade into a query: diff from there, rerun the recorded check, and it either still passes (the close was wrong) or fails on code that changed (the close was right and the code moved).
Whether the signal is countable at all depends on how the scanner keys a finding. If it keys on rule plus function instead of line number, a rename keeps the key and you can count it. If it keys on line, every refactor mints new findings and the old ones are gone, so the free signal only ever counts the closes that did not matter.
Does yours survive a function rename?
That's a really important distinction. A re-detection is a signal, not a verdict. Without knowing what check ran and what commit it ran against, you can't tell whether the original dismissal was wrong or the code simply changed later. On the function rename question, I'd favor a stable finding identity based on the rule and normalized code location or data-flow context, rather than line numbers alone. Refactors shouldn't erase the history of a finding.
I like the re-detection idea. It's a practical signal teams can start tracking without building a whole accuracy framework first. And requiring the explanation to reference the exact sink and line makes sense too. I'd still keep re-detections as something to investigate rather than automatically count as false dismissals, since the code may have changed in between.
This highlights an important distinction between AI-assisted analysis and actual security verification. AI can make vulnerability reports easier to understand, but closing a finding should always depend on reproducible evidence, not just a convincing explanation. At IT Path Solutions, we see AI integration and software reliability as areas where accuracy, traceability, and human oversight are essential. Preserving scanner traces, linking fixes to specific commits, and tracking false dismissals can make AI-assisted security workflows much more dependable.
Agreed. Traceability is what makes the distinction practical. Keeping the original scanner evidence, linking the fix to a commit, and checking false dismissals gives teams a way to verify whether the workflow is actually improving decisions.