Two years ago, jailbreaking an LLM was a hobbyist's party trick. You'd get it to say something it wasn't supposed to, screenshot it, post it on Twitter, everyone laughs. Today, those same techniques are release-blocking bugs. Dedicated QA engineers run automated adversarial suites like Garak and PyRIT before anything ships, and a single prompt injection that slips through can cascade into a deleted database row or a hallucinated financial transaction. That's the shift Daniel wants us to map. He's asking for a full tour of QA methodologies for agentic AI — what breaks from traditional testing, what's established, what's emerging, and what a sane minimum stack actually looks like for a small team. And he wants us to anchor it on that transformation of adversarial prompting from curiosity to standard discipline.
The adversarial piece is the right anchor, because it's the clearest example of the QA mindset adapting to a probabilistic system. But let's start with the thing that makes all of this hard in the first place. There are really two axes here. Workflow QA — testing multi-step orchestrations, tool calls, state management, the plumbing. And agent QA — testing the LLM's reasoning, its safety, the quality of what it actually says and decides. Traditional QA assumes determinism. You write an assertion, you run it a thousand times, you get the same result every time. Agentic AI breaks that completely. Same prompt, same model, same temperature — you can get different valid outputs. So the pass-fail binary doesn't work anymore.
The unit test that says "output must equal this exact string" is dead on arrival.
Right. And it's not just assertions. Regression suites become probabilistic — a test that passed yesterday can fail today because of a model update or even just prompt drift, where a minor change in how you phrase the system prompt shifts behavior in ways you didn't anticipate. And code coverage — the old metric of "have we exercised every branch" — is meaningless when the "code" is a neural network with no branches to cover. You can't instrument a transformer's attention heads and say "we hit eighty percent of the neurons."
So what actually survives? Because I refuse to believe the entire discipline of software testing just evaporates.
It doesn't. A few things carry over cleanly. Input-output schema validation — you can still assert that the agent's output is valid JSON, that it has the right fields, that types match. Latency and cost monitoring — those are deterministic metrics, you can set thresholds and alert on them. Integration testing of tool call interfaces — when the agent decides to call an API, you can verify it called the right endpoint with parameters that match the schema. And canary deployments — the practice of routing a small percentage of traffic to a new version before full rollout — that maps almost one-to-one from traditional services.
The schema validation one is interesting because it's structural, not semantic. You're not judging whether the answer is good, just whether it's well-formed.
And that distinction — structural versus semantic — is the fault line that runs through this whole space. Everything on the structural side is easier. Everything on the semantic side requires new tools. So let's walk through the established methods, starting with the simplest.
Golden datasets.
Golden datasets. You curate a set of input-output pairs that define acceptable behavior. Maybe a hundred examples, hand-annotated. You run the agent against them, compare outputs, flag deviations. It catches regression in output quality — if your model update suddenly starts giving worse answers to the same questions, you'll see it. The cost is moderate — hours of human annotation per hundred examples — and it belongs in pre-release and CI.
What does it miss?
Novel edge cases. Multi-turn reasoning failures where the first response is fine but the third goes off the rails. And it's only as good as your coverage — if your golden set doesn't include a scenario, you're blind to it.
So it's a safety net with holes you can see.
Which is why most teams layer it with LLM-as-judge. You take a stronger model — GPT-four, Claude three point five — and have it evaluate your agent's outputs against a rubric. Studies show eighty to ninety percent agreement with human evaluators. It catches nuanced quality issues that schema validation misses. Is the tone appropriate? Did the agent actually answer the question or just sound confident while dodging it?
The confident dodging is a whole failure mode unto itself.
It is. But LLM-as-judge has its own problems. The judge model can exhibit biases — it tends to prefer longer outputs, more confident-sounding outputs, outputs that match its own style. And it misses subtle factual errors that the judge model also makes. If both your agent and your judge share the same blind spot about, say, a specific historical date, the error sails through. Cost is per-evaluation — you're making API calls for every test — and it belongs in CI and pre-release.
And rubric scoring is the structured version of that.
You define criteria — correctness, helpfulness, safety — and score on a Likert scale. One to five, something like that. It catches systematic quality degradation. If your helpfulness scores trend down over three releases, you know something's wrong. The setup cost is moderate, but once the rubric exists, it's cheap to run. The miss is task-specific failure pattern that your rubric doesn't capture. If your rubric doesn't have a criterion for "didn't hallucinate a legal precedent," and your agent is doing legal work, you're in trouble.
That brings us to trajectory evaluation, which I think is the most underappreciated method in the stack.
Completely. Most teams evaluate the final output. Did the agent get the right answer? But trajectory evaluation looks at the entire reasoning chain — every intermediate step, every tool call, every piece of reasoning the model generated along the way. It catches cases where the agent got the right answer for the wrong reason.
Give me the example.
An agent is calculating a tip on a restaurant bill. The prompt gives it the subtotal and the tax rate. Instead of using the tax rate to compute the pre-tax amount and then applying the tip percentage, the agent just guesses a number. But it guesses correctly. Final output is right. Trajectory evaluation catches that the reasoning was nonsense — the agent hallucinated a calculation step. If the numbers had been slightly different, it would have been wrong.
So you're testing the process, not just the product.
And that matters enormously for agentic systems, because a wrong reasoning step in step two can cascade into a disaster in step seven. The cost is higher — you're evaluating more tokens, and the evaluation itself is more complex — but it belongs in pre-release and post-release monitoring. The miss is cases where a plausible trajectory leads to a wrong final output. The trajectory looks reasonable, the evaluator signs off, but the answer is wrong.
Plausible but wrong is the scariest failure pattern in this whole space.
It really is. Which connects to replay testing. You capture production traces — real conversations, real tool calls, real user interactions — and you replay them through new agent versions. Does the new version handle the same scenario the same way? It catches regressions in specific user scenarios. A customer support agent that previously handled refund requests correctly now hallucinates a policy change after a model update. You catch that before it hits users.
The cost there is storage plus replay compute.
Moderate. And it belongs in CI and pre-release. The miss is novel scenarios not in your replay corpus. If your user base suddenly starts asking about a new product category, your replay tests won't cover it.
So replay testing is backward-looking by design.
Yes. And canary deployments are forward-looking. You route one to five percent of traffic to the new agent version, monitor for anomalies, and roll back if quality metrics degrade. It catches real-world failures at low blast radius — only a fraction of users see the bad behavior. The miss is failures that require high traffic volume to manifest. If the bug only triggers on one in ten thousand requests, your five percent canary might not see it before you go to full rollout. Infrastructure cost is moderate, and this belongs in production rollout.
That covers the standard toolkit. But there's a whole other category that's emerged from the security world, and this is where Daniel's anchor comes in. Adversarial prompting has gone from hacker curiosity to release-blocker. Walk me through what that actually looks like as a QA discipline.
It looks a lot like penetration testing, but for language models. The core question is: can someone trick the agent into ignoring its instructions? Prompt injection testing probes for exactly that. You craft inputs designed to override the system prompt — "ignore all previous instructions and do X" — and see if the agent complies. Jailbreak suites like the Gandalf game from Lakera test whether the agent can be coerced into revealing sensitive information or performing disallowed actions.
Gandalf is the one where you try to get a password out of the model, and it gets harder at each level.
Right. It's gamified, but the techniques it surfaces are real. And then there are the automated adversarial generators. Garak is an open-source LLM vulnerability scanner — it runs a battery of probes against your model, testing for prompt injection, jailbreaking, encoding-based attacks, all of it. PyRIT is Microsoft's Python Risk Identification Tool — it automates red-teaming for generative AI systems. You feed it a scenario, it generates adversarial inputs, evaluates responses, and produces a risk report.
So instead of a human red-teamer spending three days crafting clever prompts, you run a tool that generates thousands of variations and scores the results.
And that's the shift Daniel's talking about. It's not a hobbyist in a basement anymore. It's a CI pipeline step. You run Garak before release. If the vulnerability score crosses a threshold, the release is blocked. That's a QA methodology.
What do these tools actually catch, and what do they miss?
They catch safety and security vulnerabilities — prompt injection, jailbreaking, data exfiltration attempts, harmful output generation. They miss subtle reasoning failures and benign quality issues. Garak won't tell you if your agent is giving mediocre financial advice. It'll tell you if your agent can be tricked into telling someone how to build a bomb. The cost is moderate to high — you need adversarial expertise to configure the tools properly and interpret results — and it belongs in pre-release and periodic security audits.
There's a related practice that I think is under-discussed: chaos-engineering-style fault injection into tool calls.
This is one of my favorites. The idea is you deliberately break the tools the agent depends on. The weather API returns a five hundred error. The database query times out. The search engine returns garbage data. And you watch what the agent does. Does it crash? Does it hallucinate a weather report? Does it retry gracefully? Does it tell the user something went wrong?
The graceful degradation test.
And most agents fail it spectacularly in early versions. They'll confidently report that it's seventy-two degrees and sunny because the model filled in the blank when the API didn't respond. Catches brittle error handling and cascading failures. Misses issues that only manifest with valid tool responses. Cost is moderate — you need infrastructure to inject faults — and it belongs in pre-release and staging.
I want to push on something. You said most agents fail this. Is that anecdotal or is there data on it?
Mostly anecdotal from teams I've talked to, but the pattern is consistent enough that I'd bet on it. The default behavior of an LLM when a tool call fails is to confabulate — it fills the gap with plausible-sounding information because that's what it's trained to do. You have to explicitly train or prompt it to say "I couldn't get that information, let me try again."
That's the fundamental tension — the thing is a prediction engine, and "I don't know" is a low-probability prediction in most contexts.
Right. Which brings us to the properly emerging stuff. Simulation environments and synthetic user agents. The idea is you create simulated users — often using another LLM — that interact with your agent in realistic multi-turn scenarios. They have goals, personalities, they get frustrated, they change topics. You run hundreds of these simulations and evaluate how your agent handles them.
So you're generating synthetic test cases at scale, with realistic conversational dynamics.
It catches multi-turn reasoning failures and edge cases in conversation flow that static eval sets miss. The miss is that simulated users don't match the real-world distribution of user behavior. Real users do things no simulator would think of. The cost is high — simulation infrastructure plus compute for both the agent and the simulated users — and it belongs in pre-release and continuous training.
Conference-talk material or actually being used?
The big labs are using it. OpenAI and Anthropic both use synthetic user simulations for safety testing. For smaller teams, it's mostly aspirational — the infrastructure overhead is significant. I'd put it in the "emerging but not yet standard" bucket.
What about self-critiquing setups — the agent evaluates its own outputs?
This one gets talked about a lot and used very little in production. The idea is elegant — have the agent critique its own outputs, or have a separate agent evaluate the primary agent. Catches obvious errors and inconsistencies. Misses subtle errors the critiquing agent also makes, which is the same problem as LLM-as-judge but worse because you're using the same model or a similar one. The circularity problem is real. Cost is moderate — additional inference — but the reliability concerns keep it out of most production pipelines. Conference-talk material.
Property-based testing, on the other hand, seems genuinely useful and under-adopted.
Hugely under-adopted. The idea comes from the lm-evaluation-harness and similar frameworks. Instead of asserting a specific output, you assert properties that must hold regardless of the output. "The output must be valid JSON." "The total must equal the sum of line items." "The agent must not call the delete endpoint without confirmation." "The response must not contain personally identifiable information."
So you're testing invariants, not values.
And invariants are testable even when the system is non-deterministic. That's the insight. It catches structural and safety violations. Misses semantic quality issues — your agent could return perfectly structured, perfectly safe, perfectly useless answers. Cost is low to moderate, and it belongs in CI and pre-release. This should be in everyone's stack.
The last one on Daniel's list is continuous online evaluation from production traces. Monitoring production behavior, sampling traces, scoring them against quality metrics.
This is the holy grail and also the hardest to do well. You're sampling real user interactions, running them through evaluators, tracking quality scores over time. It catches real-world degradation and drift — if your agent's helpfulness score drops five points after a model update, you know immediately. The miss is issues that haven't occurred yet — it's reactive by nature. Cost is moderate — monitoring infrastructure plus evaluation compute — and it belongs in production.
Most teams I've talked to aspire to this and very few have it running.
The sampling and scoring pipeline is straightforward. The hard part is defining quality metrics that actually correlate with user satisfaction and business outcomes. A helpfulness score of four point two out of five — what does that actually mean for retention or revenue? Most teams can't answer that.
So let's get opinionated. Of everything we've covered, what's becoming standard and what's conference-talk material?
Standards: LLM-as-judge, trajectory evaluation, replay testing, and adversarial prompt injection testing using Garak or PyRIT. Those four are widely adopted in production systems shipping today. Canary and shadow deployments are standard practice — that's just good ops, it predates LLMs. Property-based testing is emerging fast and I think it'll be standard within a year. Simulation environments and synthetic users are being used by the big labs but not by most teams. Self-critiquing agents are conference-talk material — elegant on stage, unreliable in production. Chaos-engineering-style fault injection for tool calls is discussed more than practiced, but the teams that do it swear by it. Continuous online evaluation is aspirational for most.
Give me the minimum stack. Small team, shipping an agentic product, can't afford a dedicated QA army. What do they actually need?
Six things. One: input-output schema validation in CI. Structural correctness is cheap to test and catches a surprising amount. Two: a golden dataset of fifty to a hundred curated examples with LLM-as-judge evaluation in CI. You need a semantic quality baseline. Three: replay testing against recorded production traces. As soon as you have real users, capture their interactions and replay them. Four: adversarial prompt injection testing using an automated suite — Garak is free and open-source, there's no excuse not to run it. Five: canary deployment with automated rollback on quality metric degradation. If your scores dip, the canary rolls back automatically — no human in the loop. Six: property-based tests for critical safety invariants. Define the things your agent must never do, and test those as invariants.
That's a list a team could actually implement in a sprint or two.
The order matters. Start with schema validation and property-based tests — those are deterministic, you can write them today. Add the golden dataset and LLM-as-judge next. Adversarial testing once you have something that works. Replay testing once you have production traffic. Canary deployment as part of your release process. Each layer catches different failure pattern, and they compound.
The thing I keep coming back to is the meta-QA problem. As agents become more autonomous and multi-step, we're building systems where one agent evaluates another agent, which is itself being evaluated by a third system. Who tests the testers?
That's the open question I'd leave listeners with. We're moving toward a world where every production agent has a shadow evaluator agent that continuously scores its behavior and triggers retraining or rollback. QA becomes a real-time control loop rather than a pre-release gate. But the evaluator itself can drift, can have biases, can miss things. And now you need an evaluator for the evaluator. The recursion doesn't have an obvious bottom.
The QA infinite regress.
We don't have a good answer for it yet. The pragmatic approach is to use a stronger model as the evaluator and hope the gap stays wide enough. But that's a hope, not a solution.
Before we wrap — the misconception I want to name is the idea that you can just use unit tests for agentic systems. I still hear this from teams coming from traditional software. "We'll write assertions for the outputs." Deterministic assertions break because the same prompt can produce different valid outputs. You can't assert equality on a probabilistic system.
The corollary misconception is that LLM-as-judge is a perfect evaluator. It has known biases — it prefers longer outputs, more confident-sounding outputs — and it misses subtle errors the judge model also makes. It's a useful tool, not a gold standard.
Daniel's prompt pushed us to map the whole terrain, and I think the shape that emerges is clear. The old QA paradigm was a gate — pass these tests, ship. The new paradigm is a layered monitoring system — structural checks, semantic evaluation, adversarial probing, production observation. You're not certifying correctness. You're managing risk across multiple dimensions, continuously.
The adversarial piece that Daniel anchored on is the perfect illustration of that shift. Two years ago, prompt injection was a curiosity. Today, it's a CI pipeline step with dedicated open-source tooling. That's what maturation looks like in this space — a threat vector becomes a test suite.
If you're shipping an agentic product, start with the six-item stack we laid out. Add methods as you encounter specific failure pattern. And if you learn something that isn't in the literature yet, share it — this discipline is being built in public, and the teams doing the work are the ones writing the playbook.
Thanks to our producer Hilbert Flumingtop for keeping us on track. This has been My Weird Prompts. Find us at my weird prompts dot com, and if you've got a QA horror story or a method we missed, email the show at show at my weird prompts dot com. We'll be back soon.
