By clicking “Accept All Cookies”, you agree to the storing of cookies on your device to enhance site navigation, analyze site usage, and assist in our marketing efforts. View our Privacy Policy for more information.
ToolAccuracy of FindingsDetects Non-Pattern-Based Issues?Coverage of SAST FindingsSpeed of ScanningUsability & Dev Experience
DryRun SecurityVery high – caught multiple critical issues missed by othersYes – context-based analysis, logic flaws & SSRFBroad coverage of standard vulns, logic flaws, and extendableNear real-time PR feedback
Snyk CodeHigh on well-known patterns (SQLi, XSS), but misses other categoriesLimited – AI-based, focuses on recognized vulnerabilitiesGood coverage of standard vulns; may miss SSRF or advanced auth logic issuesFast, often near PR speedDecent GitHub integration, but rules are a black box
GitHub Advanced Security (CodeQL)Very high precision for known queries, low false positivesPartial – strong dataflow for known issues, needs custom queriesGood for SQLi and XSS but logic flaws require advanced CodeQL experience.Moderate to slow (GitHub Action based)Requires CodeQL expertise for custom logic
SemgrepMedium, but there is a good community for adding rulesPrimarily pattern-based with limited dataflowDecent coverage with the right rules, can still miss advanced logic or SSRFFast scansHas custom rules, but dev teams must maintain them
SonarQubeLow – misses serious issues in our testingLimited – mostly pattern-based, code quality orientedBasic coverage for standard vulns, many hotspots require manual reviewModerate, usually in CIDashboard-based approach, can pass “quality gate” despite real vulns
Vulnerability ClassSnyk (partial)GitHub (CodeQL) (partial)SemgrepSonarQubeDryRun Security
SQL Injection
*
Cross-Site Scripting (XSS)
SSRF
Auth Flaw / IDOR
User Enumeration
Hardcoded Token
ToolAccuracy of FindingsDetects Non-Pattern-Based Issues?Coverage of C# VulnerabilitiesScan SpeedDeveloper Experience
DryRun Security
Very high – caught all critical flaws missed by others
Yes – context-based analysis finds logic errors, auth flaws, etc.
Broad coverage of OWASP Top 10 vulns plus business logic issuesNear real-time (PR comment within seconds)Clear single PR comment with detailed insights; no config or custom scripts needed
Snyk CodeHigh on known patterns (SQLi, XSS), but misses logic/flow bugsLimited – focuses on recognizable vulnerability patterns
Good for standard vulns; may miss SSRF or auth logic issues 
Fast (integrates into PR checks)Decent GitHub integration, but rules are a black box (no easy customization)
GitHub Advanced Security (CodeQL)Low - missed everything except SQL InjectionMostly pattern-basedLow – only discovered SQL InjectionSlowest of all but finished in 1 minuteConcise annotation with a suggested fix and optional auto-remedation
SemgrepMedium – finds common issues with community rules, some missesPrimarily pattern-based, limited data flow analysis
Decent coverage with the right rules; misses advanced logic flaws 
Very fast (runs as lightweight CI)Custom rules possible, but require maintenance and security expertise
SonarQube
Low – missed serious issues in our testing
Mostly pattern-based (code quality focus)Basic coverage for known vulns; many issues flagged as “hotspots” require manual review Moderate (runs in CI/CD pipeline)Results in dashboard; risk of false sense of security if quality gate passes despite vulnerabilities
Vulnerability ClassSnyk CodeGitHub Advanced Security (CodeQL)SemgrepSonarQubeDryRun Security
SQL Injection (SQLi)
Cross-Site Scripting (XSS)
Server-Side Request Forgery (SSRF)
Auth Logic/IDOR
User Enumeration
Hardcoded Credentials
VulnerabilityDryRun SecuritySemgrepGitHub CodeQLSonarQubeSnyk Code
1. Remote Code Execution via Unsafe Deserialization
2. Code Injection via eval() Usage
3. SQL Injection in a Raw Database Query
4. Weak Encryption (AES ECB Mode)
5. Broken Access Control / Logic Flaw in Authentication
Total Found5/53/51/51/50/5
VulnerabilityDryRun SecuritySnykCodeQLSonarQubeSemgrep
Server-Side Request Forgery (SSRF)
(Hotspot)
Cross-Site Scripting (XSS)
SQL Injection (SQLi)
IDOR / Broken Access Control
Invalid Token Validation Logic
Broken Email Verification Logic
DimensionWhy It Matters
Surface
Entry points & data sources highlight tainted flows early.
Language
Code idioms reveal hidden sinks and framework quirks.
Intent
What is the purpose of the code being changed/added?
Design
Robustness and resilience of changing code.
Environment
Libraries, build flags, and infra metadata flag, infrastructure (IaC) all give clues around the risks in changing code.
KPIPattern-Based SASTDryRun CSA
Mean Time to Regex
3–8 hrs per noisy finding set
Not required
Mean Time to Context
N/A
< 1 min
False-Positive Rate
50–85 %< 5 %
Logic-Flaw Detection
< 5 %
90%+
Severity
CriticalHigh
Location
utils/authorization.py :L118
utils/authorization.py :L49 & L82 & L164
Issue
JWT Algorithm Confusion Attack:
jwt.decode() selects the algorithm from unverified JWT headers.
Insecure OIDC Endpoint Communication:
urllib.request.urlopen called without explicit TLS/CA handling.
Impact
Complete auth bypass (switch RS256→HS256, forge tokens with public key as HMAC secret).
Susceptible to MITM if default SSL behavior is weakened or cert store compromised.
Remediation
Replace the dynamic algorithm selection with a fixed, expected algorithm list. Change line 118 from algorithms=[unverified_header.get('alg', 'RS256')] to algorithms=['RS256'] to only accept RS256 tokens. Add algorithm validation before token verification to ensure the header algorithm matches expected values.
Create a secure SSL context using ssl.create_default_context() with proper certificate verification. Configure explicit timeout values for all HTTP requests to prevent hanging connections. Add explicit SSL/TLS configuration by creating an HTTPSHandler with the secure SSL context. Implement proper error handling specifically for SSL certificate validation failures.
Key Insight
This vulnerability arises from trusting an unverified portion of the JWT to determine the verification method itself
This vulnerability stems from a lack of explicit secure communication practices, leaving the application reliant on potentially weak default behaviors.
AI in AppSec
December 4, 2025

If Your Code Changes, Your Rules Should, Too

Determinism had its moment. It powered the early days of software security and gave us a language to talk about findings, patterns, and rules. It helped us believe we could capture risk inside a tidy set of if-this-then-that conditions. And for a long time, that belief carried us. But the world we’re building software in today doesn’t operate that way. 

Codebases shift constantly, people write code in unpredictable ways, and also there’s these AI co-pilots and coding assistants. What’s emerging is not a deterministic ecosystem but an even more probabilistic one. And that shift, one a lot of teams feel but haven’t fully named, is reshaping how we have to think about security.

A conversation with an AppSec pro recently captured this perfectly. 

They said, almost off-hand, “Your code changes, so it only makes sense that your rules change too.” 

It was one of those lines that stops you for a moment. They weren’t trying to make a grand statement; they were just reflecting what they saw in their own environment. But that simple observation gets at the heart of why deterministic security testing doesn’t keep up anymore. 

Deterministic tools assume that risk can be fully expressed through fixed patterns, prewritten rules, and known heuristics. You match a signature, you find a problem; you don’t match it, you move on. That model worked well enough when the universe of code vulnerabilities was relatively stable and when developers tended to repeat the same mistakes in the same ways.

But humans are not deterministic beings. No two developers think exactly alike. Even the same developer rarely thinks the same way across two different days. We write code based on memory, context switching, interpretation, intuition, and sometimes sheer momentum. 

We misunderstand APIs; we bend logic around deadlines; we copy old patterns into new situations where they don’t quite fit. In other words, we behave like probabilistic systems. So our errors are probabilistic too. They’re weird, novel, half-formed, unexpected, and in many cases they don’t map cleanly to predefined vulnerability patterns. Yet traditional security tools are limited to precisely those patterns. They can only catch what they already know.

What’s changed in the last few years is that our tooling began to mirror our own unpredictability. 

The active set of languages in use has doubled over the last decade. AI pair programmers like Copilot, Claude, and Gemini generate code not through deterministic processes but through probability distributions. These models produce output based on what seems likely or contextually appropriate, not what is logically guaranteed. 

Combine that with humans and you get a pipeline where neither the author nor the assistant is producing code that maps cleanly to deterministic rules. It’s still code; it still compiles and runs; but the pathways that lead to it are fundamentally unpredictable. If the inputs to your system are probabilistic, you can’t rely on deterministic rules to validate them.

If you’re still doubting, take the thought experiment of bug bounties or breaches. We run scanners, we find things, but despite these efforts, bug bounties and security breaches remain a persistent problem.. My point is that we’ve intrinsically known that static patterns and signatures don’t work, but AI has and the increase of code velocity and volume is making this even more clear.  

This isn’t a new idea in distributed systems. Mark Burgess, the creator of CFEngine and the person behind Promise Theory, has argued for decades that determinism at scale is largely an illusion. In high-performance systems, he points out, there is always variability: network jitter, contention, timing drift, competing workloads, latency, and a whole host of factors that make the system behave less like a machine with predictable states and more like an ecosystem managing fluctuating conditions. 

Computers “keep their promises,” as Burgess often says, by aiming for outcomes within acceptable certainty, not by guaranteeing perfect determinism. His work reminds us that many of the assumptions we make about computers being rigidly predictable are, at best, convenient fictional accounts we assure ourselves with.

Operators and engineers in SRE or DevOps usually understand this intuitively because they live in environments where variability is obvious. Highly distributed systems don’t behave the same way twice, and anyone who has operated one knows you don’t secure reliability with static rules; you secure it with adaptive, context-aware approaches. 

Security, though, has largely remained anchored in deterministic thinking. We kept believing we could enumerate vulnerabilities into rule sets and use those rule sets to define the boundaries of risk. But as software has shifted toward AI-assisted development, microservice architectures, and rapid iteration, those boundaries became porous.

The reality is that today’s software is built by a hybrid process. Humans write part of it, AI writes part of it, and sometimes the two collaborate so tightly that it’s hard to say where one influence ends and the other begins. Both contributors operate with probability. 

The result is code that doesn’t follow predictable patterns and therefore cannot be effectively governed by tools that only understand those patterns. When customers come to us saying they’re finding issues that pattern-based tools never even thought to look for, that’s not an accident. It’s a sign that their development process has outgrown determinism.

What teams really want, whether they articulate it or not, is a security model that adapts as quickly as the code itself. 

If the structure of the code changes, the understanding of risk should change too. If the behavior of the system shifts because an AI assistant refactored a few functions, the threat model should shift accordingly. Deterministic tools can’t do that. They rely on frozen logic. 

A probabilistic system, on the other hand, evaluates the likelihood that a given code change introduces exploitability based on a wide range of contextual signals. It doesn’t ask, “Does this match a known vulnerability rule?” It asks, “Given everything we know about how this system behaves, does this look exploitable now?”

That shift is subtle but transformative. It’s how we catch vulnerabilities that don’t match classical patterns. It’s how we see partial SSRF setups, logic flaws, misused APIs, or emergent risks created by AI-generated snippets. And most importantly, it means our model evolves with the codebase. There’s no static rule set waiting for a quarterly vendor update. There’s no assumption that the past contains all the answers for the future.

This is why more teams are moving toward AI-native, probabilistic security. They’re not doing it because it’s trendy. They’re doing it because their development process already made the shift without asking. Once you realize you’re building probabilistic systems—whether through humans, AI, or a blend of both—the only logical conclusion is that your security testing must also be probabilistic.

So yes: if your code changes, your rules should too. That’s not a slogan. That’s simply how modern software works. The teams who embrace that reality are the ones who are going to secure the systems the rest of the world depends on.