I recently watched Claude Security find several pieces of an attack yet still miss the actual vulnerability.
The individual findings were there and some were even cross-referenced. But the scanner reported them separately, assigned each one its own severity, and stopped. It never performed the final step: determining what an attacker could accomplish when those conditions were combined.
That final step was left to the human, which raises an uncomfortable question. If the human still has to assemble the evidence, understand the application, and determine the real risk, what exactly made the scanner intelligent?
This came out of a bakeoff we ran between DryRun Security and Anthropic’s Claude Security product. We tested both tools against the same commits in three applications and independently verified the findings against the source code. I expected the tools to disagree about bugs. What was more interesting was how differently they represented the risks they did identify.
Claude often treated related weaknesses as separate findings. DryRun reconciled them into a single finding organized around the attacker outcome. That difference matters because defenders do not manage lines of suspicious code. We manage what attackers can accomplish.
We automated the cheapest part of code review
Long before LLMs, there were two noticeably different kinds of secure code review reports.
The first came from a scanner. It found recognizable patterns and low-hanging fruit, listed each result separately, and attached a severity based on the information available to the rule. The scanner might find useful things, but it did not know the application in any meaningful sense. It did not reconcile overlapping results, decide which findings were credible, understand how controls elsewhere changed the risk, or connect several weaknesses into an attack. The human receiving the report had to do that work.
The second came from a consultant or experienced application security practitioner. A good reviewer did more than run tools and repackage the output. They learned the application. They investigated whether findings were real, looked at architecture and trust boundaries, followed data through the system, considered the controls already in place, and searched for risks specific to how that application worked. They suppressed observations that did not deserve the client’s attention and connected seemingly minor conditions when they created a more serious outcome together.
At DryRun, my cofounder and I never wanted to use AI to produce a more articulate version of the first report. We wanted automated code review to move closer to the second.
That distinction influences everything we build. We intentionally omit findings that an organization considers ultra-low priority, noisy, or nitpicky. We learn those preferences from feedback in pull and merge requests, classifications and explanations in the product, repository guidance such as CLAUDE.md and AGENTS.md, git history, and other application context. We also preserve collateral from the review: what the system learned about the application, what it considered, and the context that may be useful in a later review or another agentic system.
Fewer findings can mean the system did more work, not less. If the tool validated, reconciled, prioritized, and discarded ten useless observations before they became tickets, it saved the customer from doing that work. I would argue that a smaller backlog containing the risks that matter is a better outcome.
Vulnerability chaining is one of the clearest examples of this philosophy.
Three Lows do not add up to a High
To be precise, three Low-severity vulnerabilities do not mathematically add up to a High. Severity is not arithmetic. Three unrelated Lows are still three Lows.
But several individually minor weaknesses can provide the necessary conditions for a much more serious attack. A predictable resource name may be Low. An overly broad read permission may be Low. Sensitive information entering application logs may be another Low. If those logs are written to the predictably named resource and the permission allows an attacker to retrieve them, the actual finding is not a collection of configuration nits. It is unauthorized access to sensitive data.
The atomic observations remain important because they are the evidence supporting the conclusion. They should not become three or four independent tickets that force someone else to reconstruct the attack. They should sit beneath one unified finding describing the attacker’s starting position, the conditions that make exploitation possible, the resulting impact, and the remediation points that would break the chain.
This is broader than conventional reachability analysis. Reachability asks whether execution can arrive at a vulnerable component or code path. Runtime verification asks whether a finding can be exercised under tested conditions. Both are valuable prioritization tools. Composition asks a different question: given the vulnerabilities, controls, architecture, data flows, and deployment conditions we have established, what can an attacker accomplish by combining them?
That is the question the human has historically been paid to answer.
Claude found the pieces. DryRun found the outcomes.
The bakeoff gave us several concrete examples.
In the first application, a deliberately vulnerable Django task manager, Claude found stored cross-site scripting, DOM-based cross-site scripting, and insecure cookie configuration. It reported the XSS findings and the readable-cookie weakness as separate Mediums. DryRun connected the executable injection paths with readable JWT cookies and recognized the resulting attacker outcome: persistent session theft. That was a High-risk finding. Claude had not necessarily failed to detect the individual problems. It failed to finish the security analysis.
In a second application, a Next.js and Firebase voice-cloning service, rate-limit counters lived in a Firestore document that the rate-limited user could modify. The server trusted those counters, and the limits protected calls to two paid upstream APIs. Claude reported the relevant weaknesses separately as Mediums and Lows. DryRun connected the client-writable state, the trusting limiter, and the paid services into one High-risk outcome: an attacker could bypass the controls and create unbounded paid API usage.
The individual implementation mistakes mattered because of what they enabled together. Reporting them independently obscured the reason the team should care.
The third application was an OSS Java Spring application (and I will not name the project until it has been secured). In this application, an empty last-name search returned every owner. Records used sequential identifiers. Owner lookups were not scoped. Templates exposed addresses and phone numbers. The application was published through a NodePort. Together, those conditions created a path for harvesting every owner’s personal information from the network. Claude mentioned the absence of authentication while bounding the severity of other findings, but it never filed the assembled vulnerability at all. DryRun did.
These were not three variations of the same type of chain. One combined executable injection and session behavior. Another combined attacker-controlled state, business logic, and financial consequence. The third combined enumeration, broken access control, sensitive data, and deployment exposure. The common failure was the unit of analysis. Claude evaluated defects. DryRun evaluated attacker outcomes.
Across the root causes found by both tools in the bakeoff, DryRun assigned a higher severity ten times and Claude assigned a higher severity zero times. Six of those ten differences came from DryRun rating the chain while Claude rated one of its individual links. In several cases, Claude’s report contained enough information to hint at the relationship. The information was present. The judgment was not.
Judgment does not mean making things up
There is an obvious danger in asking an agent to build attack chains: it can become very good at telling plausible stories. An imaginative attack narrative is not evidence, and adding more agents does not make an unsupported assertion true.
Our approach is to separate confidence in the atomic findings from confidence in the composition. An individual vulnerability must meet a high confidence threshold before we promote it to a finding. The system then makes a separate judgment about whether those validated findings and the surrounding application facts satisfy the necessary conditions for the asserted attacker outcome.
If the composition is not supported with sufficient confidence, the system should not manufacture a High-severity vulnerability. The atomic findings remain, and the summary can explain the possible chain, identify the uncertain condition, and state why the system did not assert it. That gives a practitioner a precise hypothesis to investigate without poisoning the backlog with speculation.
Runtime verification can strengthen that evidence, and I am a fan of it when it is useful and safe. But successfully exploiting an issue is not the only form of legitimate proof. Sometimes the code, configuration, permissions, and data flow establish the behavior with high confidence. Some attacks are computationally difficult, dangerous, or impractical to reproduce. A failed runtime attempt can also mean the test environment was incomplete rather than the vulnerability was impossible to exploit.
We should use runtime verification as a valuable source of evidence, not turn it into a silver bullet or an excuse to ignore what we can already establish about the system.
The same discipline applies to suppression. DryRun does not discard a low-value observation the moment it appears. It remains available during reconciliation so the system can determine whether it contributes to a more serious chain. Only after that analysis do we decide that it adds no value and does not deserve to become someone else’s ticket. Suppression should be a presentation decision made after analysis, not an instruction for the system to become ignorant.
If the output did not change, what did AI actually change?
The new generation of agentic AppSec tools can explore repositories, read across files, retain context, invoke tools, test hypotheses, and produce remarkably fluent explanations. Yet many of them are still organized around the same assumptions as the scanners that came before them. One vulnerability goes in. One finding comes out. The system repeats that process until it has built a backlog, then declares success because the backlog has better prose, greater categories of findings, and fewer obvious false positives. That is not nearly enough.
Traditional scanners had an excuse for stopping at detection. They were built from rules, patterns, and limited program analysis. They could not understand an application in the same way an experienced reviewer could, so validation, contextualization, composition, and prioritization remained human work.
Agentic tools do not get to claim intelligence and retain the same excuse. If a system can reason about the application, it should make choices about which observations matter, which weaknesses reinforce one another, which controls change the outcome, and what deserves a developer’s attention. Those judgments will not always be perfect, but refusing to make them guarantees that the human continues to perform the most expensive part of the review.
There is a difference between having experience in security and having operated an AppSec program where you personally carried responsibility for application risk. When you have done the latter, you learn quickly that finding more bugs is not inherently considered “success”. Every finding creates work, consumes credibility, competes for developer attention, and changes how the organization perceives security. A backlog can grow while the organization becomes no more secure.
You stop asking only, “How many vulnerabilities did we find?” You start asking, “Where are we actually exposed, and what can an attacker accomplish?”
This is why new agentic AppSec companies need to be more opinionated. The market does not need another company reproducing SAST, SCA, DAST, IaC scanning, and secrets detection because agents made those products easier to build. We should not use the existing scanner as the specification for the future. We should decide what an intelligent security review ought to accomplish and build that, even when the result no longer resembles the tools that came before it.
An agentic scanner that refuses to exercise security judgment is not AI-native. It is a traditional SAST with an LLM operating it.
We have the opportunity to build better
Chaining vulnerabilities into attacker outcomes is only one example, in one tooling category, of what becomes possible when we stop treating AI as a new engine for an old practice. The larger opportunity is to reinspect every assumption inherited from traditional application security tooling (as well as programs.. consider this a teaser).
Some of those assumptions will survive. Others existed because the old technology could not do anything better. We should revisit the problems that were previously too contextual, judgment-heavy, or expensive to solve. We should examine where software development is moving and design security systems for that future, not rebuild the workflow practitioners already present.
At DryRun, we did not set out to create the same scanner report with more findings and better-written remediation advice with our DeepScan and Pull/Merge-Request product lines. We wanted the system to absorb more of the work an experienced reviewer performs: learning the application, evaluating conditions, preserving context, suppressing noise, connecting evidence, and telling the organization what matters. Vulnerability chaining is part of that, and our implementation will continue to improve, but the philosophy is more important than any one feature.
We finally have enough intelligence in our systems to make automated security review (and more) look less like scanner output and more like security work. That opportunity is much bigger than finding more bugs.
Don’t rebuild what already exists with moar-er AI. Build the capabilities the old technology never made possible.
