I have a pet peeve.
Recently, I ’ve watched the term agent harness become shorthand for increasingly thin layers around increasingly capable models. You have an agent framework like Crew, LangChain, or whatever. You give it tools, skills, context, and a good system prompt. Maybe you add sub-agents, a supervisor, or a graph. Then a really good model does something impressive, and we point at the whole thing and talk about the sophistication of the “harness.”
The problem is that, in many of these systems, the model is still doing most of the heavy lifting. It is deciding what matters, how to approach the problem, which tools to use, what evidence to trust, whether it has enough information, and ultimately whether its own answer is good enough. The things surrounding it may make the model more capable, but capability is not the same thing as orchestration.
There is nothing inherently wrong with that architecture. Harnesses are incredibly useful. For personal projects, experiments, small teams, and plenty of internal workflows, a good model with the right tools and context may be exactly what you should build. But a harness is not necessarily an orchestration layer, and when we start talking about production security systems operating across thousands of developers, enormous repositories, different languages and frameworks, wildly different architectures, organizational risk tolerances, geographic restrictions, provider failures, token budgets, and actual security consequences, that distinction matters.
After three and a half years of building AI systems for application security, there is one architectural rule I’ve become fairly uncompromising about:
If you depend on a model for your performance, you’re cooked.
Better Models Are Hiding Bad Architecture
That statement probably sounds strange coming from someone who has spent the last several years building a software security company known for its agentic/AI prowess. So yes, models are incredible. We benchmark them constantly. We’ve used different model families and providers for different jobs, and sometimes a new model materially improves something we’re doing. Sometimes a much cheaper model turns out to be better at a particular task than the expensive frontier model (and to be clear, we don’t use frontier models as our benchmarks show this is unnecessary). But yes, I want models to keep getting better.
However, there is an enormous difference between benefiting from model performance and depending on model performance. A properly orchestrated agentic system should depend on the model for interpretation while depending on the surrounding system for resilience and accuracy. The model absolutely contributes to accuracy, but accuracy as a property of the product cannot be whatever accuracy the underlying model happens to deliver today.
I think about this in terms of model blast radius. If I change a model, I expect the portions of my system that deliberately use probabilistic reasoning to behave differently. Models have different strengths, weaknesses, reasoning characteristics, costs, latency profiles, and failure modes. What I don’t want is for a model change to fundamentally redefine the behavior of the entire application. Better models should improve my system; they should not be my system.
If upgrading the underlying model is your primary strategy for improving your product, you have effectively outsourced your product roadmap to the model provider. Worse, modern models are now good enough to make that arrangement look much more robust than it actually is. Modern models are so good that you can build the first 80% astonishingly quickly. That’s arguably the problem: they hide architectural weakness.
We Learned This When Models… Stunk
When we started building DryRun Security, models weren’t remotely as capable as they are today. In retrospect, that constraint was one of the best things that could have happened to us because weak models punished architectural laziness immediately.
One of our earliest problems sounds almost laughably basic today: reliably understanding the composition of an application. We needed to understand frameworks, components, application boundaries, and other basic facts before we could meaningfully reason about their security implications. Initially, we allowed models to do more of this work. They got it wrong often enough that it became a real problem.
The important lesson wasn’t simply that LLMs hallucinate. The problem was what happened after the hallucination. If a probabilistic operation incorrectly determines which framework an application uses, and another probabilistic operation reasons about authentication based on that result, and another reasons about authorization, and another reasons about exploitability then you haven’t created one opportunity for failure. You have created cascading uncertainty. Each individual inference can look perfectly reasonable while the entire chain is operating from a false premise.
That forced us to start asking a different question: why are we spending tokens or even using probability on something we can know, deterministically?
Today, DryRun deliberately establishes repository facts, application boundaries, structure, metadata, and other context before asking models to interpret their security significance. Our architecture explicitly separates software understanding from software reasoning because they are different computational problems. We use deterministic systems where they are exceptionally good at establishing objective facts and reserve semantic reasoning for the places where interpretation actually adds value.
That eventually became a broader engineering doctrine for us: don’t spend probability cycles on facts you can know. Or, in the language of our architecture, spend model uncertainty only where deterministic structure cannot satisfactorily resolve the question. This doesn’t make the resulting system less agentic. It makes the agentic parts considerably more useful.
Tool Access Is Not Orchestration
This is usually where someone raises the obvious objection: “Fine, but my harness has deterministic tools. My agent can call Opengrep. It has an AST. I used LSP via MCP. It queries databases and searches the repository. Isn’t that exactly what you’re describing?”
It’s a fair question, but tool access is not orchestration. An LSP is a fantastic way to give an agent deterministic information about a codebase. We use that kind of capability ourselves. But telling me that your agent can use an LSP tells me almost nothing about the sophistication of your system. The more interesting questions are how it knows when to use it, what it is instructed to look for, what context existed before it started navigating, what caused that investigation to happen, how you prevent it from wandering, how you determine whether it navigated correctly, and what happens when the tool fails or returns something technically valid but contextually misleading.
More fundamentally, I want to know who is driving. Does the model use the tool correctly because the model happens to natively understand how to do that today, or has the system been engineered to guide it through the task? What gets preserved after the operation? What gets evaluated? What happens when the agent reaches the wrong conclusion despite having the right tools?
The same problem appears in architectures that amount to running a traditional scanner and then asking an LLM to validate its findings. Congratulations, you have technically combined deterministic and probabilistic analysis, but the deterministic scanner has already defined your search space. If it never nominated the interesting vulnerability, your probabilistic validator never gets the opportunity to reason about it. This is particularly limiting in application security, where some of the most important flaws are important precisely because of business logic, trust boundaries, authorization semantics, or interactions distributed across an application.
At DryRun, some of our vulnerability pipelines move through deterministic candidate selection, semantic applicability and reachability reasoning, deeper bounded agentic verification, and deterministic and semantic controls afterward. I’m not suggesting that exact pipeline is universally correct. The point is that the computational technique changes depending on the question being answered. When somebody tells me their harness contains deterministic tools, I’m less interested in whether determinism exists somewhere in the architecture than I am in a much harder question: where does control live?
Don’t Just Evaluate Outputs. Evaluate the Conditions That Produced Them.
This may be the single most important lesson we’ve learned building these systems.
Everyone wants to evaluate the final answer. Did we correctly identify the SQL injection? Did we find the authorization flaw? What are our precision and recall? Obviously those things matter, but if the answer is wrong, I need to know why it was wrong. Otherwise I don’t actually know what to improve.
Was the wrong context assembled? Was an important signal missing? Did retrieval fail? Was the prompt wrong? Did the knowledge base provide misleading information? Did a tool return something unexpected? Did our middleware steer the agent incorrectly? Did an earlier probabilistic decision poison a later one? Did a deterministic filter remove something it shouldn’t have? Or was the model itself simply wrong?
Don’t just evaluate outputs. Evaluate the conditions that produced them.
Those conditions are the system. A model can even be confidently wrong because you gave it a reason to be confident. That is why adding another “judge agent” isn’t necessarily a solution. If the failure occurred upstream and your judge receives the same poisoned evidence, you may now have two models confidently agreeing with each other.
This is also why I get immediately suspicious when I see an agent architecture containing models, agents, skills, tools, context, and memory but nothing about evaluation or observability. If I change a prompt, retrieval strategy, model, provider, middleware rule, or tool, I need to know what happened to the system. “We ran it a few times and it seemed better” is not sufficient once you are building something people depend on.
DryRun continuously evaluates major stages throughout our analysis pipelines, and our observability allows us to reconstruct routing, evidence, refinement, suppression, and final decisions. We maintain evaluation data because model behavior can change as prompts, models, codebases, and usage patterns evolve. That isn’t operational garnish around the AI. It is part of the reliability architecture. More importantly, it gives us the evidence necessary to identify weaknesses and systematically improve them. You cannot engineer around weaknesses you cannot measure.
The First 80% Is Deceptively Easy
Earlier this year, a company came to us after trying to build this themselves. I won’t identify them, but their experience matters because, months later, our sales team is increasingly hearing versions of the same story.
They initially put one engineer on the problem. The engineer built something quickly, developers were impressed, and compared with their existing SAST experience it felt dramatically better. Then a couple of weeks passed. Developers started complaining about noise, accuracy wasn’t where it needed to be, and the team struggled to improve it.
Their next decision was completely rational: use a better frontier model. Initially, things seemed better again, but before long the complaints returned. So they added another engineer to work on reducing false positives. At that point they had two engineers spending meaningful time on an internal security product, inference costs were climbing, and they still couldn’t cross the gap between “wow! this works!” and “we can depend on this.”
Eventually they stepped back and looked at what closing that gap was actually going to require. That gap is where I think people badly underestimate these systems.
You can build 80% of this yourself. That’s precisely why build-vs-buy is so deceptive. The first 80% might be shockingly fast. The last 20% can take way longer than the first 80%, and the last 20% is the product.
That remaining work is not one clever prompt. It is evaluation and observability, specialized pipelines, deterministic context construction, probabilistic reasoning, routing, steering, middleware, memory, feedback loops, classification, suppression, provider selection, fallbacks, retries, queues, cost controls, failure recovery, infrastructure observability, developer experience, and hundreds of smaller engineering decisions you don’t know you need until production teaches you.
A prototype demonstrates capability. Production exposes conditions.
Scale Doesn’t Mean Requests Per Second
When I say these architectures fall apart at scale, I don’t mean you forgot to put Kubernetes underneath them. Scale in an agentic security system is multidimensional.
I may need to review a microservice whose authorization semantics actually depend on a centralized authentication or authorization service somewhere else in an SOA. Tomorrow it might be an iOS application. The next repository might contain a Web3 CLI used to transact on-chain. One customer may have 100 developers, while another has thousands distributed around the world with different availability requirements, geographic restrictions, government sanctions, regulatory constraints, and internal security policies.
Then add repository size, monorepos, languages, frameworks, application architectures, finding volume, organizational risk tolerance, latency expectations, security consequences, provider behavior, and infrastructure failures. Even the organization itself becomes part of the environment: how developers work, how security teams make risk decisions, what they consider actionable, and how previous findings were triaged all influence whether an ostensibly “correct” result is actually useful.
And none of those variables are static. Models change. Providers change. Applications change. Frameworks change. Organizations change. Attackers change. Production scale is volume, variance, time, and organizational context all interacting at once.
A thin harness can push an enormous amount of that complexity onto the model and hope that the model reconstructs enough of the world correctly on every run. A mature orchestration system instead tries to make important context explicit, durable, measurable, and reusable. DryRun, for example, preserves repository understanding and human security decisions so future analysis can build on what the system has already learned rather than repeatedly reconstructing it from scratch.
That distinction becomes more important with every dimension you add.
Your Token Budget Is an Architecture Problem
Something else interesting has started showing up repeatedly in conversations we’re having with companies: token budgets. (And CFOs are noticing)
For the last few years, the answer to an AI system that wasn’t quite good enough could often be some variation of giving it more context, letting it reason longer, adding another pass, adding another agent, or simply using the better model. That becomes a much harder answer when somebody upstairs says security needs to provide the same functionality with fewer tokens.
Inference cost is not merely the advertised price per million tokens. Poor orchestration burns money through unnecessary inference, bloated context, agents wandering through repositories, repeated work, failed tool calls, retries, misused tools, expensive models answering cheap questions, and probabilistic reasoning being used for things traditional software could answer faster and more reliably. Inaccurate results themselves have a cost because somebody ultimately has to deal with them.
This is where orchestration stops merely being a reliability problem and becomes an economic one. You need to know which model actually needs to answer a particular question, whether a model needs to answer it at all, whether candidates can be eliminated before inference, whether prior understanding can be reused, when an investigation should terminate, whether a cheaper model performs a particular operation just as well, and how much of the available context actually belongs in the window.
At DryRun, deterministic context construction intentionally constrains the search space before semantic reasoning begins, improving reliability while also reducing unnecessary token consumption and latency. Good orchestration doesn’t merely make AI more accurate. It lets you spend intelligence—and money—where intelligence is actually required.
The Model Is Not Your Moat
This is especially uncomfortable in security because the attacker has access to excellent models too. If your strategy for staying ahead of attackers is waiting for the same frontier models they can access to become smarter, you aren’t staying ahead of them. You’re staying current.
So when someone asks me what makes our “magical solution” different from everybody else’s “magical solution,” my answer is that there is no magical solution. If the question is really asking which magical model we wrapped, it has already misunderstood where the engineering is.
What makes your product better than another company wrapping the same model? What happens when they get access to that model tomorrow? What happens when an open model becomes nearly as capable for your particular task at a fraction of the price? What did you actually build?
For us, the answer increasingly lives in the surrounding system: specialized orchestration, repository intelligence, context construction, security memory, deterministic and probabilistic signals, human-feedback integration, continuous evaluation, and the mechanisms that determine what models see, what they are asked, how they are steered, and how their conclusions are challenged. The model is an extraordinarily powerful component of that system. It isn’t the system.
Drive the Darn Thing
If you’re building one of these systems and recognizing some of your architecture in what I’m criticizing, I wouldn’t start by ripping everything apart. I’d start by getting serious about evals and observability. Before adding more capability, make sure you can tell what happens when you change something and diagnose why the system failed.
Then start decomposing your probabilism. Look at every place you’ve asked a model to make a decision and ask whether that decision genuinely requires interpretation. If it doesn’t, stop spending probability there. At the same time, start thinking much more deliberately about who is driving the workflow. Don’t assume that because a model knows how to use a tool, it knows when, why, and how that tool should be used as part of your system.
Perhaps the most useful mental model I can offer after doing this for several years is to treat the model like an incredibly capable intern with massive ADHD, memory problems, unlimited enthusiasm, and absolutely no discernment. That intern may be capable of doing work that would have seemed impossible a few years ago, but I’m not handing them a million-line enterprise repository, access to a bunch of tools, and an instruction to “go find security problems.”
I’m keeping them on task. I’m giving them narrowly scoped work and the memories and context they actually need. I’m telling them what evidence matters. I’m providing the appropriate tools and steering how they use them. I’m establishing checkpoints, checking their work, preserving what they learn, and making sure that a mistake in one task doesn’t silently become the premise for five more.
Most importantly, when that intern repeatedly struggles with something, I don’t just yell the same prompt louder. I figure out why.
That is where evaluation becomes particularly valuable. Once you can see where the gaps actually are, you can backfill them with better context, different techniques, deterministic logic, specialized knowledge, new tools, better steering, another model, or whatever the problem actually requires.
That is orchestration.
Harnesses Aren’t Throwaway Garbage
Despite the title and despite how irritated I clearly am I don’t think agent harnesses are “lesser than”. Sometimes they are exactly what you should build. If you’re automating something for yourself, experimenting with an idea, building an internal tool for a small team, or working on something where occasional failure isn’t particularly consequential, a sophisticated model surrounded by some context, skills, and tools may be more than enough. That accessibility is one of the genuinely amazing things about where AI is right now.
I just have a problem with confusing the term “harness” with engineering a production orchestration system.
Eventually the question stops being how good is the model? and becomes how good is the system you’ve engineered around it?
A properly orchestrated agentic system should depend on models for interpretation while depending on the system for resilience and accuracy. Better models should make it better, but worse models should have a bounded blast radius. Provider failures shouldn’t destroy it. Token constraints shouldn’t neuter it. A wrong answer should be diagnosable. Known facts shouldn’t repeatedly become probabilistic guesses. Past knowledge shouldn’t disappear at the end of an inference call. And improving the product shouldn’t depend primarily on waiting for somebody else’s model to get smarter.
So, no. I don’t have a magical solution.
There is no magical solution.
That’s the entire point.
Better models aren’t enough. You need a system that understands your code, verifies what’s actually exploitable, and gives developers useful answers without burying them in noise.
See how DryRun turns powerful models into dependable application security.
.jpg)