I was reading through some enterprise risk disclosures this morning, and it is a completely different world than it was even eighteen months ago. It used to be that companies were worried about their employees asking a chatbot how to hide a body or something equally ridiculous, but now the fear is much more structural. It is about the plumbing. It is about the very foundation of how these systems interact with the data they are supposed to protect.
It has to be. When seventy-two percent of the S and P five hundred companies are listing artificial intelligence as a material risk in their annual disclosures, you know the era of just playing around with prompts is over. We have moved from the curiosity phase into the infrastructure phase, and that shift is exactly what Daniel's prompt is pushing us to look at today. In early twenty twenty-four, that number was only twelve percent. Think about that jump. In two years, AI has gone from a shiny new toy to a primary concern for the board of directors.
Today's prompt from Daniel is about the rise of AI security tools, specifically focusing on PII protection, data loss prevention, and the emergence of the AI Gateway as the mandatory control plane for what he calls Agentic AI. He wants us to dig into how these tools actually function as middleware before a prompt even touches a model API. It is a timely one, especially coming off the back of the NVIDIA GTC conference last week, which ran from March sixteenth to the eighteenth.
I am glad he brought up the middleware aspect because that is where the real innovation is happening right now. My name is Herman Poppleberry, and I have been obsessed with this specific transition for months. We are seeing the birth of what people are calling the AI firewall, but it is much more sophisticated than a traditional network firewall. In the old days, you just blocked a port or an IP address. Now, you have to inspect the intent of a sentence in real-time. We are talking about a world where the perimeter has completely dissolved.
The term Agentic AI really changes the stakes here, doesn't it? We aren't just talking about a text box on a website anymore. We are talking about autonomous systems that have the keys to the castle. They can browse internal files, they can hit database endpoints, and they can execute code. If one of those agents decides to "helpfully" export a list of customer social security numbers to an external model for analysis, you have a catastrophe on your hands.
The average cost of a data breach involving these kinds of vulnerabilities has hit four point eight eight million dollars. That is a number that gets the attention of a board of directors very quickly. And if you look at the recent updates from NVIDIA, specifically their Agent Toolkit with NemoClaw and OpenShell, they are essentially trying to build a lead-lined room for these agents to live in. We are moving away from model-native security toward external middleware because the models themselves just aren't designed to be security guards.
I want to get into the technical "how" of this middleware. If I am an enterprise developer and I am sitting between a user and, say, a Claude or a Gemini model, what am I actually doing in that middleware layer? Daniel mentioned Named Entity Recognition for PII redaction. How does that look in practice?
Think of the middleware as a high-speed inspection station on a toll road. When a prompt comes in from a user or an autonomous agent, it doesn't go straight to the model. It hits a proxy layer first. This is what companies like Lasso Security and Lakera are building. The first thing that happens is a scan for PII, or personally identifiable information. In twenty twenty-six, we aren't just using simple regular expressions to find patterns like credit card numbers. We are using specialized, lightweight transformer models that perform Named Entity Recognition.
So the middleware itself is running a smaller AI model just to check the input for the larger AI model?
That is the only way to do it with the necessary accuracy. These NER models are trained to identify the context of a word. If I say "The river is near the bank," it knows that is a geographic feature. If I say "Send this to the Bank of America," it flags that as a sensitive entity. It can identify names, addresses, health information, and even API keys that might be buried in a block of code. Once identified, the middleware redacts or masks that data. It might replace a name with a placeholder like "USER_NAME_ONE" before the model ever sees it. This is the "Man-in-the-Middle" architecture Daniel mentioned.
That seems straightforward for simple text, but Daniel mentioned Data Loss Prevention or DLP. That feels like a much harder problem. If I am a developer and I am trying to prevent source code leakage or intellectual property theft, how does a middleware layer stop that without being incredibly intrusive or slowing everything down to a crawl?
This is where the latency trade-off becomes a real engineering challenge. For DLP, the gateway is comparing the prompt against a set of corporate policies. For example, a policy might state "Do not allow the export of any files ending in dot P-Y or dot J-S to external endpoints." The middleware has to parse the prompt, identify if it contains code snippets, and then run a similarity check against known internal repositories.
Wait, is it actually checking the code in real-time against the company's entire GitHub history? That sounds like it would add seconds of latency.
Not the entire history, but they use vector embeddings of sensitive internal documents and codebases. The middleware takes the user's prompt, turns it into a mathematical representation, and does a quick distance calculation against the "forbidden" vectors. If the prompt is too similar to a sensitive internal project, the gateway blocks it. This is what CrowdStrike is doing now with their Falcon AI Detection and Response. They announced natively supporting NVIDIA NeMo Guardrails version zero point twenty point zero just a few days ago, on March nineteenth. It gives them over seventy-five built-in rules to sanitize these inputs instantly.
I'm interested in the "intent" part of this. Daniel mentioned that the industry is moving toward "Intent Security." It is one thing to catch a social security number, but it is another thing entirely to catch a user who is trying to be sneaky. We've talked about prompt injection before, but in an agentic world, isn't the injection risk much higher because the agent has more power?
It is significantly higher. This is why the OWASP Top Ten for Large Language Model Applications was just updated for twenty twenty-six on March twentieth. Prompt injection is still number one, but "Sensitive Information Disclosure" jumped to number two. The reason is RAG, or Retrieval-Augmented Generation. When you have a system that is automatically pulling data from internal databases to answer a question, the risk isn't just what the user says, it's what the system "helpfully" retrieves.
So the middleware has to work on the output side too?
It has to. It's a two-way street. The gateway inspects the prompt going in, but it also inspects the model's response coming back. If the model hallucinations lead it to reveal a password or if the RAG system pulls a piece of data the user shouldn't see, the middleware catches it on the way out. It's a "Man-in-the-Middle" in the best possible sense. It is the only thing standing between a useful answer and a major compliance violation.
You mentioned the NVIDIA Agent Toolkit earlier. Those names, NemoClaw and OpenShell, sound very specific. What are they actually doing that is different from just a standard API wrapper?
They are designed to be secure runtimes. Think of it like a sandbox for an AI agent. When an agent wants to perform an action, like searching a file system or making a web request, it has to do it through the "Claw" or the "Shell." These runtimes have programmable privacy guardrails baked in. They don't just rely on the model being "good." They enforce hard limits at the operating system level. If the agent tries to execute a command that isn't on the pre-approved list, the runtime just says no. It is a fundamental shift from asking the AI to be safe to forcing the environment to be safe.
That reminds me of the JetPatch announcement from March twentieth. They talked about an "Enterprise Control Plane" with a kill-switch. That sounds a bit dramatic, like something out of a sci-fi movie where you have to pull the plug on the rogue AI. Is it really that extreme?
In an enterprise context, a kill-switch is just good governance. If an autonomous agent starts a loop where it is rapidly querying a database and attempting to move data to an external S-three bucket, you don't want to wait for a human to notice. The JetPatch system monitors the "reasoning traces" of these agents. If it detects a pattern that looks like data exfiltration or unauthorized privilege escalation, it hard-stops the process immediately. It is an automated circuit breaker for AI. It's about preventing the "Agentic Secret Gap" we discussed in episode ten seventy—that space where the agent starts doing things the developer never intended.
I think one of the biggest "aha moments" in Daniel's prompt is the shift away from system prompts as a security measure. For a long time, the advice was just to tell the model "Do not reveal your instructions" or "Do not share customer data" in the system message. Why is that finally being recognized as a failure in twenty twenty-six?
Because it's like putting a "Do Not Enter" sign on a door that doesn't have a lock. If the secret is in the prompt, it is effectively public. There is no such thing as a "hidden" instruction once a sophisticated attacker starts using jailbreak techniques. OWASP is very clear about this now in their twenty twenty-six update. If you are relying on the model's own "personality" or "instructions" to keep data safe, you have already lost. You are asking the model to be both the prisoner and the prison guard. We touched on this in episode twelve seventeen, but the consensus has really hardened since then.
So what is the alternative? If we can't trust the system prompt, where does the security live?
It lives in what is called Context-Based Access Control, or CBAC. This is the next evolution of identity and access management. Instead of just checking who the user is, the system checks the context of the request. The middleware layer looks at the data being requested, the role of the user, the history of the conversation, and the current state of the network. It makes a real-time decision about whether to allow the data flow. The model never even gets the sensitive data unless the middleware has already cleared the context.
That feels like a much more robust architecture, but it also sounds like a nightmare to manage. You are basically adding a whole new layer of infrastructure that needs its own rules, its own logs, and its own maintenance. For a smaller company, is this even feasible?
It is becoming a requirement because of the regulatory landscape. The EU AI Act has that big August second deadline coming up later this year. One of the key requirements is what they call "Technical Truth." You can't just have a document in a drawer saying you protect PII. You have to be able to produce real-time logs showing every time a piece of sensitive data was redacted or blocked. You need an audit trail for the AI's "thought process."
"Technical Truth" is a great phrase. It moves the conversation from "we promise we are being careful" to "here is the cryptographic proof that we blocked this specific leak." I can see why the AI Gateway is becoming the "Nginx" of the model era. You wouldn't put a web server on the public internet without a reverse proxy, and you shouldn't put an LLM in your enterprise without a security gateway.
We actually touched on this a bit in episode eight hundred forty-one when we talked about LiteLLM and AI Gateways, but the complexity has grown so much since then. Back then, we were just talking about load balancing and cost tracking. Now, we are talking about inspecting reasoning traces and tool calls. If an agent is using a tool to access a database, the gateway has to understand the schema of that database to know if the query is safe. It has to be able to look at a SQL query generated by an AI and say, "Wait, that's joining the payroll table with the public employee directory, block that."
It's that "Agentic Secret Gap" again. There is this space between what the developer thinks the agent is doing and what the agent is actually doing when it's "reasoning" through a task. The middleware is the only thing that can bridge that gap by providing a transparent window into those tool calls.
And that is why these specialized players like Lasso and Prompt Security are gaining so much traction. They aren't just looking at the text; they are looking at the "intent." If an agent says "I'm going to look up the customer's history to help them with their billing issue," that sounds fine. But if the "history" it looks up includes their full credit card numbers and the agent then attempts to "help" by sending that data to a third-party translation API, the intent has shifted from helpful to hazardous. The gateway catches that shift in real-time.
I'm curious about the Lakera acquisition by Check Point. That happened last year, but it feels like it was a harbinger of this trend. They were famous for their "P-Leak" defense. For those who might not remember that specific term, what exactly is a P-Leak attack?
P-Leak is short for Prompt Leak. It is a specific type of injection where the attacker tries to get the model to spit out its original instructions or any data that was provided in the context window. Lakera Guard was one of the first middleware layers that could detect the "signature" of a P-Leak attack in real-time. By moving that defense out of the model and into the middleware, they made it much harder to bypass. You can't "convince" a middleware proxy to ignore its code the same way you can convince an LLM to ignore its instructions.
It's the difference between a psychological barrier and a physical one. You can talk your way past a person, but you can't talk your way past a locked gate. I think that's the fundamental shift Daniel is highlighting here. We are moving from "AI Ethics" which feels very soft and suggestion-based, to "AI Security" which is hard-coded and enforceable.
The market growth numbers Daniel included are staggering. Seventy-two percent of the S and P five hundred listing this as a risk is one thing, but the prediction that over fifty percent of enterprises will be using dedicated AI security platforms by twenty twenty-eight is the real story. In early twenty twenty-five, that was less than ten percent. We are in the middle of a massive infrastructure build-out. By the end of this year, if you don't have a dedicated AISP, you're going to be an outlier.
So, if you are a developer listening to this and you are building an agentic workflow right now, what is the immediate takeaway? Is it "don't build until you have a gateway," or is there a way to start safely?
My advice is to audit your stack immediately. If your application is talking directly to an LLM API without any intermediate inspection layer, you are effectively running a web server without a firewall. You need to look at implementing an open-source gateway like NeMo Guardrails at the very least. It gives you a way to define "Colang" scripts—that's the language NVIDIA uses for these guardrails—to specify exactly what the model is allowed to talk about and what it must avoid.
And don't forget the August second deadline for the EU AI Act. Even if you aren't based in Europe, if you have customers there, the "Technical Truth" requirement is going to hit you. You need to start logging your PII redaction events now so you have a baseline when the regulators come knocking. You can't build an audit trail retroactively.
It is also worth revisiting the "No Training" promise. A lot of companies in twenty twenty-four felt safe because OpenAI or Anthropic promised they wouldn't train on enterprise data. But as we discussed in episode twelve thirty-five, "No Training" is not the same thing as "No Disclosure." The data still leaves your perimeter. It still sits on a third-party server. If that server is compromised, or if the model itself is tricked into revealing that data to another user of the same system, the "No Training" promise doesn't help you.
It is like handling digital plutonium, which is a callback to our "Digital Plutonium" episode twelve thirty-four. You can't just put it in a box and hope it stays there. You need active, real-time monitoring of the radiation levels. In this case, the radiation is the sensitive data leaking out of your network. The middleware is your Geiger counter.
I find it fascinating that we've come back to the "Man-in-the-Middle" as a solution. Usually, in security, a Man-in-the-Middle is the villain. But in the world of Agentic AI, the Man-in-the-Middle is the hero. It is the only entity that has the full picture—the user's intent, the model's response, and the corporate policy. It's the only place where you can actually enforce a "kill-switch" like JetPatch is doing.
What really struck me from Daniel's prompt was the cost of failure. Two thousand legal claims related to "insufficient AI guardrails" expected by the end of twenty twenty-six. That is a lot of work for lawyers and a lot of pain for companies. It feels like we are going to see a "Great Cleanup" where companies have to go back and retrofit all those experimental bots they built in twenty twenty-four and twenty twenty-five with this new middleware.
They have to. The "move fast and break things" approach doesn't work when "breaking things" means leaking the entire payroll database. The shift toward "Intent Security" and AI Gateways is the sign that the industry is finally maturing. We are treating AI like the powerful, dangerous, and incredibly useful tool that it is, rather than just a fancy toy. If you're a developer, your next sprint should probably include a security audit of your tool calls.
I wonder if we will eventually see these guardrails baked directly into the silicon. If NVIDIA is pushing NeMo Guardrails so hard, it wouldn't surprise me if future H-three hundred or whatever comes next has dedicated "safety cores" that handle PII redaction at the hardware level.
That is a very distinct possibility. If you can offload the NER and DLP checks to dedicated hardware, you solve the latency problem. You get the security of a gateway with the speed of a direct connection. But until then, the middleware layer is the most important part of your AI strategy that you probably aren't spending enough time on. We are building the dams today to prevent the floods of tomorrow.
It's a lot to take in, but it's clearly the direction the wind is blowing. If you are still trying to secure your AI with a long, rambling system prompt, you are basically trying to stop a flood with a "Please Don't Rain" sign. It's time to get serious about the plumbing.
It's time to build the dam. Use the tools that are available. Whether it's the open-source libraries from NVIDIA or the enterprise platforms from Lasso and Lakera, the infrastructure is there. There is no excuse for a data breach in twenty twenty-six caused by a simple prompt injection or a "helpful" agent leaking a database schema.
Well, I think we've thoroughly explored the "why" and the "how" of this shift. It's a complex topic, but Daniel always has a knack for pointing us toward the exact spot where the friction is happening in the industry. This move toward prompt governance is the defining challenge of the year.
He really does. This one felt like a natural follow-up to our deep dive on system instructions in episode twelve seventeen. If you want to understand why those instructions are failing, go back and listen to that one, then come back and look at these gateway tools. It all connects into a larger picture of enterprise maturity.
Before we wrap up, I want to give a big thanks to our producer Hilbert Flumingtop for keeping everything running smoothly behind the scenes.
And a huge thanks to Modal for providing the GPU credits that power the generation of this show. Their serverless infrastructure is exactly the kind of modern stack that makes this level of AI exploration possible.
This has been My Weird Prompts. If you are finding these deep dives helpful, we would love it if you could leave a quick review on your favorite podcast app. It really does help other people find the show and join the conversation.
We will be back soon with another prompt from Daniel. Until then, stay curious and keep an eye on your gateways.
Goodbye, everyone.
Take care.
