Daniel sent us this one — and I think it's the question a lot of engineers are quietly asking themselves right now. Agentic AI is clearly becoming a meta-skill, the thing you build your career around, not just a tool you pick up. But within that, we're seeing specializations emerge that barely existed two years ago. He's asking what the major skill silos and specific functions actually look like — and specifically, what the DevOps engineer of agentic AI would map to as a concrete job. What's the day-to-day, what's the training, what does the role actually do when it's not just vibes and API keys.
The timing on this question is perfect. In March, OpenAI dropped Agent SDK version two point three with native multi-agent orchestration and a built-in eval framework. Anthropic's Model Context Protocol — MCP — hit production scale in Q1 with over ten thousand production deployments. And Google released the A2A protocol, Agent-to-Agent, standardizing inter-agent communication in March as well. The field has bifurcated. It's not "AI tinkerer" anymore — it's distinct engineering disciplines with real salary bands and certification paths forming around them.
The gold rush metaphor applies, but the pickaxes aren't what people think. The pickaxes are these specialized roles.
And I want to frame this properly before we dive into the specifics. Agentic AI is becoming a -skill the way software engineering became a -skill in the late nineties. You used to just be a programmer — and then the field splintered into frontend, backend, DevOps, SRE, data engineering. The same thing is happening here, except faster.
The Latent Space engineering survey from February put some numbers behind this. They found that a poorly designed orchestration layer can increase your API costs by a factor of ten. Not ten percent — ten times. Redundant calls, inefficient agent routing, agents calling other agents that call the same agent again. That's the kind of thing that turns a cool prototype into a thirty-thousand-dollar monthly bill nobody saw coming.
That's exactly where the first major silo comes in. I think of it as three primary axes of specialization. The first is Architecture and Orchestration — the DevOps analog. The second is Evaluation and Safety Engineering — the QA and SRE of agentic AI. And the third is Interaction Design and Prompt Systems Engineering — the UX layer, but for agent behavior rather than visual interfaces.
Let's start with the one Daniel specifically asked about. The DevOps engineer of agentic AI. What does that actually look like as a job?
The title that's emerging is "Agent Infrastructure Engineer." And I've been tracking job postings on this — Indeed's data from April showed that postings for this role have grown three hundred forty percent year over year. These are real positions with real requirements, not speculative future-casting.
Three hundred forty percent year over year. That's not a trend, that's a stampede.
And the day-to-day is fascinating because it pulls from distributed systems in ways that traditional DevOps doesn't quite prepare you for. Let me walk through what this person actually does.
First, you're designing multi-agent topologies. Star patterns, mesh patterns, hierarchical patterns. You're deciding how agents communicate, how state propagates across agent boundaries, how you handle agent handoffs when one agent completes a subtask and needs to pass context to another. This isn't abstract architecture diagram stuff — this is live production routing where a mistake means your loan processing pipeline drops applications on the floor.
The patterns have names now. The supervisor agent pattern from LangGraph, for instance.
LangChain's 2026 State of Agentic AI report found that the supervisor agent pattern is used in over sixty percent of production multi-agent systems. That's a single pattern dominating the field — which means if you're an Agent Infrastructure Engineer, you'd better understand how supervisor topologies work, where they break, and what the alternatives are.
What does the supervisor pattern actually do?
You have a central orchestrator agent that receives the initial task, decomposes it, delegates to specialized sub-agents, and synthesizes their outputs. It's the manager, not the worker. The problem is, if the supervisor agent hallucinates a delegation — sends a contract review to the image classification agent — the whole pipeline can silently produce garbage. So the infrastructure engineer has to implement routing guards, validation layers, and circuit breaker patterns specifically designed for LLM calls.
Circuit breakers for LLMs. That's not a phrase I expected to hear five years ago.
It's essential. An LLM call can fail in ways a database query never would. It can return a valid response that's semantically wrong. It can exceed context windows mid-call. It can hallucinate a tool that doesn't exist and try to invoke it. The circuit breaker pattern — borrowed from microservices architecture — wraps each agent call and trips when the failure rate exceeds a threshold, redirecting to a fallback or a human-in-the-loop queue.
You're managing failure modes that are probabilistic rather than deterministic. That's the core shift.
And that connects to another major responsibility — building observability stacks. Tools like LangSmith version three and Arize Phoenix are becoming standard, but the infrastructure engineer has to instrument everything. You're tracking token consumption per agent, latency at each handoff point, success rates by agent type, cost per completed task. You're building dashboards that show you, in real time, whether your agent pipeline is healthy or silently degrading.
Let me give you a concrete example that I think makes this real. Imagine a fintech company processing fifty thousand loan applications a day through an agent pipeline. What does the orchestration layer actually handle?
This is a great case. The orchestration layer receives a loan application — PDFs, bank statements, tax returns. It has a routing agent whose sole job is to classify the document type and decide which specialized sub-agent to invoke. The income verification agent gets the tax returns, the identity verification agent gets the ID documents, the fraud detection agent scans everything for inconsistencies. Each handoff is a potential failure point. The orchestration layer has to implement retry logic with exponential backoff — if the income verification agent times out, you don't just fail the application, you retry with increasing delays. You need a dead-letter queue for tasks that exhaust their retries. You need idempotency guarantees so that a retried task doesn't double-process the same document.
The cost implications of getting this wrong are brutal.
If your routing agent misclassifies a document and sends it to the wrong sub-agent, that sub-agent might make three or four tool calls trying to process something it can't understand. Each tool call is an API cost. Multiply that across fifty thousand applications and you've just burned through your monthly budget in a week. The Latent Space survey documented cases where companies saw their API costs drop by sixty to seventy percent just by implementing proper agent routing and caching.
What's the training requirement for this role? Because I think the misconception is that you just need to be good at prompt engineering and you're set.
That misconception is exactly what's going to leave a lot of people behind. The Agent Infrastructure Engineer role requires distributed systems knowledge — specifically, understanding the CAP theorem as it applies to agent state. Consistency, availability, partition tolerance — when you have multiple agents running across different services, how do you maintain coherent state? You need experience with event-driven architectures — Kafka, Redis Streams, message queues. You need proficiency in at least one agent framework at the protocol level, not just the API level. LangGraph, CrewAI, or the new Microsoft AutoGen version two. You need to understand the framework's internal routing, its state management model, its failure modes.
Protocol level, not API level. That's the distinction.
That's the line between someone who can build a demo and someone who can run production. The API level is "I call this function and get a response." The protocol level is "I understand how this framework serializes agent state, how it handles concurrent invocations, what its memory model looks like, and where it's going to break under load.
There's a case study I want to mention because it illustrates exactly this. A healthcare startup built an agent pipeline for processing patient intake forms. They used synchronous agent calls — one agent calls the next, waits for a response, then continues. Worked fine in testing with ten concurrent users. When they went to production with hundreds of concurrent patients, the whole thing collapsed. Agents were timing out waiting for responses from other agents that were themselves waiting for responses. The fix required rebuilding the entire orchestration layer with RabbitMQ, adding asynchronous message passing, and implementing a dead-letter queue for failed agent invocations.
That rebuild probably cost them months and hundreds of thousands of dollars that proper architecture would have avoided. The synchronous-to-asynchronous shift is one of those things that separates prototype from production, and it's exactly what an Agent Infrastructure Engineer is hired to get right from the start.
That's the Architecture and Orchestration silo. But there's another specialization that I think is arguably even more critical, and it's the one most people get wrong. Let's talk about evaluation and safety engineering.
This is the one I find most intellectually interesting, because it forces you to rethink everything you know about testing. In traditional software, you test for known failure modes. You write a unit test that asserts a specific output for a specific input. In agent evaluation, you're testing for emergent failure modes — behaviors you couldn't have predicted.
The non-determinism is the fundamental challenge. The same input can produce different outputs, and both might be correct — or both might be wrong in different ways.
So the role emerging here is "Agent Safety Engineer" or "AI Evaluation Engineer." And this person builds evaluation suites that test not just output quality, but agent behavior chains. Does the agent recover gracefully from a hallucinated tool call? Does it correctly handle ambiguous user intent? Can it detect and reject prompt injection attempts? These are behavioral questions, not output comparison questions.
Let me give an example that makes this concrete. A legal document review agent was processing contracts — standard stuff. Then the underlying model got updated, and the agent started rejecting valid contracts. Not because the contracts changed, but because the model's behavior shifted in subtle ways around edge cases in contract language. The eval suite caught it because it included adversarial test cases specifically designed for those edge cases.
That's the key concept — agent drift. When a model update changes agent behavior in ways that aren't immediately obvious. An Agent Safety Engineer builds monitoring systems that track drift over time. You maintain a "safety scorecard" across agent versions — success rate on known-good test cases, consistency with a reference knowledge base, adherence to policy constraints. When the scorecard dips, you investigate.
How do you actually measure drift?
One is semantic similarity — you compare agent responses to a known-good knowledge base and flag responses that diverge beyond a threshold. Another is behavioral regression testing — you maintain a suite of scenarios with expected behaviors, not expected outputs. "When the user asks for a refund they're not entitled to, the agent should politely decline and explain why." That's a behavioral expectation. You test it with dozens of phrasings, adversarial variations, edge cases. If the agent starts granting those refunds after a model update, you've caught drift.
The training requirements for this role are really different from the Architecture silo.
This role draws from testing methodology — specifically property-based testing from the Haskell and Erlang world, applied to agent behavior. You're defining properties that should hold for any agent response — "the agent never reveals system prompts," "the agent never processes obviously malicious instructions," "the agent always cites sources when making factual claims." Then you're generating thousands of test cases that probe those properties.
You also need statistical methods that most software testers never touch.
A-B testing with significance thresholds. Inter-rater reliability metrics for human evaluation — when you have humans reviewing agent outputs, you need to measure whether your reviewers agree with each other, because if they don't, your eval data is noise. You need familiarity with adversarial ML techniques — understanding how prompt injection works, how jailbreaking works, how to red-team an agent system systematically rather than just poking at it.
There's a sub-specialty emerging within this silo too — the Agent Observability Engineer.
This person builds the dashboards and alerting systems specifically for agent behavior in production. They track cost-per-task metrics, success rate by agent type, latency distributions across the agent chain, and crucially, they build the audit trail infrastructure. Every agent decision, every tool call, every handoff — logged, timestamped, attributable.
The regulatory pressure is going to make this role non-optional. The EU AI Act's requirements for human oversight of high-risk AI systems take full effect in August. That translates directly to technical requirements for agent logging, audit trails, and the ability to reconstruct exactly what an agent did and why.
This is where I think the Evaluation and Safety silo becomes the highest-paid specialization in the field. When agentic AI moves into regulated industries — finance, healthcare, legal — the compliance requirements don't go away just because the system is non-deterministic. You still need to prove to a regulator that your loan approval agent isn't discriminating, that your medical triage agent isn't making dangerous recommendations, that your contract review agent isn't introducing liability. Someone has to build the systems that prove that. That someone is the Agent Safety Engineer.
There's a comparison I want to draw. In traditional software testing, you know what a bug looks like. The application crashes, the output is wrong, the database gets corrupted. In agent evaluation, a "bug" might be a subtle shift in tone that makes the agent slightly more likely to approve borderline cases. It might be a new failure mode where the agent handles a situation correctly ninety-five percent of the time but catastrophically the other five percent. Traditional QA methodology doesn't catch that.
That's why the testing frameworks for this are so different. DeepEval, LangFuse, these are purpose-built for evaluating non-deterministic systems. They support things like "consistency checks" where you compare agent responses to a knowledge base using semantic similarity thresholds. They support "trajectory evaluation" where you test not just the final output but the sequence of decisions the agent made to get there.
I have a case study that illustrates this. A customer support agent was fine-tuned on new data — recent customer interactions — to improve its response quality. After fine-tuning, it started hallucinating refund policies. It would confidently tell customers they were entitled to full refunds for products that had a strict no-refund policy. The eval pipeline caught it because it included a consistency check that compared agent responses to the known-good policy knowledge base. The semantic similarity score dropped below the threshold, the alert fired, and the model update was rolled back before it reached production.
That's the kind of failure that would have been invisible to traditional monitoring. The agent wasn't crashing. Its responses were coherent and confident. They were just wrong in a very specific, very damaging way.
We've covered Architecture and Orchestration, and Evaluation and Safety. What about the third silo — Interaction Design and Prompt Systems Engineering?
This is the one that's most adjacent to what people think of as "prompt engineering," but it's much more systematic. It's about designing the behavioral interface of an agent — how it communicates, how it handles ambiguity, how it expresses uncertainty, how it recovers from misunderstandings. It's UX design, but the medium is language rather than pixels.
The glockenspiel of corporate approachability.
I'm sorry?
You know when a company designs their chatbot to be "friendly" and it ends up sounding like it's trying to sell you essential oils? That's bad interaction design. The Prompt Systems Engineer is the person who prevents that. They design the conversational architecture — when the agent should be direct versus empathetic, how it should escalate to a human, what tone it should use for different user emotional states. And they encode that in prompt systems, not one-off prompts. Modular prompt components that compose together to produce consistent agent behavior.
This role requires a really unusual skill combination. You need enough linguistic sophistication to understand how language choices shape user trust and behavior. You need enough technical skill to implement prompt systems programmatically — templating, conditional prompt assembly, dynamic context injection. And you need enough evaluation methodology to test whether your interaction design is actually working — are users completing tasks? Are they frustrated? Are they trusting the agent appropriately rather than over-trusting or under-trusting?
The over-trust problem is fascinating. Users either treat the agent like it's infallible and follow bad advice, or they treat it like it's useless and ignore good advice. Designing the interaction to calibrate trust appropriately — that's a genuine engineering challenge, not just vibes.
It connects back to the Evaluation silo, because you need metrics for trust calibration. You need to measure whether users are accepting agent recommendations at appropriate rates given the agent's actual accuracy. That's a research problem as much as an engineering problem.
Let me pull us back to the practical question. If someone's listening and thinking "okay, I need to pick a direction," how do they decide?
I'd say start by auditing your current skill set against the three silos. If you're strong on distributed systems, event-driven architectures, infrastructure — go Architecture and Orchestration. If you're strong on testing methodology, statistical analysis, safety and compliance — go Evaluation and Safety. If you're strong on UX, language, human-computer interaction — go Interaction Design.
The market is already rewarding depth over breadth. That three hundred forty percent growth in Agent Infrastructure Engineer postings isn't an accident. Companies are realizing that the generalist "AI engineer" who can do a bit of everything is less valuable than the specialist who can build the orchestration layer that makes everything else work.
The other thing I'd say is — learn the protocols, not just the frameworks. MCP, the Model Context Protocol, is becoming the standard for how agents connect to tools and data sources. A2A, Google's Agent-to-Agent protocol, standardizes how agents communicate with each other. And there's an emerging OpenTelemetry for AI standard that will define how agent observability works across platforms. These protocols will outlast any single framework. If you build your expertise around the protocols, you're building on infrastructure that'll be around in five years.
Frameworks come and go — LangChain was the default two years ago, now it's one option among many. But the protocol layer is stickier.
I want to address the misconception that you need a PhD in machine learning to work on agentic AI. You don't. The most in-demand skills right now are in systems engineering, testing methodology, and interaction design — not model training. The people building the most impressive production agent systems are often coming from backend engineering and DevOps backgrounds, not ML research.
The model training piece is increasingly commoditized anyway. You're not training models from scratch for agent systems — you're orchestrating existing models, evaluating their behavior, and designing the interaction layer. The frontier is in how you compose these things, not in how you build the underlying models.
That's the shift that I think a lot of people haven't internalized yet. The agentic AI revolution isn't about replacing engineers — it's about creating entirely new categories of engineering. These are roles that didn't exist three years ago and are now commanding senior-level salaries because the demand is so far ahead of the supply.
Let me ask you a forward-looking question. Do you think we'll see "Agent Architect" emerge as a distinct role, separate from both software architects and AI engineers? Or will the field consolidate into a single "Agent Engineer" role that encompasses all three silos?
I think we're going to see both — fragmentation and consolidation — but at different levels. At large companies, absolutely, you'll see dedicated Agent Architects who design multi-agent system topologies the way solutions architects design microservice architectures today. At startups, you'll see the "Agent Engineer" who wears all three hats. But even at startups, I think the Evaluation and Safety function will split off fastest, because the regulatory pressure is coming from outside the company. You can't skip compliance just because you're small.
The EU AI Act in August is a forcing function. If you're deploying high-risk AI systems in Europe, you need documented human oversight, you need audit trails, you need to demonstrate that you've tested for safety. That's not optional, and it's not something you can bolt on at the end. It has to be built into the evaluation infrastructure from the start.
That's going to create a whole sub-specialty of "Agent Auditors" — people who verify compliance, who can look at an agent system and say "this meets the regulatory requirements for human oversight, this doesn't." And separately, "Agent Trainers" who fine-tune agent behavior post-deployment based on production data and eval results.
The Evaluation silo itself splits. Auditors on one side, trainers on the other, and the safety engineers building the infrastructure that both of them depend on.
And I think the Architecture silo will split too — into orchestration specialists who design agent topologies, and infrastructure specialists who build the underlying platform that agents run on. The field is nowhere near done differentiating.
Let's make this actionable. If someone listening wants to build a portfolio that demonstrates one of these specializations, what should they actually build?
For Architecture and Orchestration — build a multi-agent system with proper observability. Something that processes documents, routes them to specialized agents, handles failures gracefully, and exposes metrics. Share the architecture diagram. Write about the failure modes you encountered and how you solved them. That's the kind of artifact that gets you hired.
For Evaluation and Safety?
Build an open-source eval suite for a common agent framework. Pick LangGraph or CrewAI, build a set of test cases that probe for common failure modes — hallucination, prompt injection, policy violations — and publish the results with your methodology. Bonus points if you include adversarial test cases. That demonstrates exactly the skill set that companies are desperate for.
For Interaction Design?
Build an agent with a carefully designed interaction model. Document your design decisions — why the agent communicates the way it does, how it handles ambiguity, how it expresses uncertainty. Run user tests and publish the results. Show that you can design agent behavior systematically, not just write clever prompts.
One more thing I want to flag. The Indeed data from April showed three hundred forty percent growth in Agent Infrastructure Engineer postings. That's a window. It won't stay that wide open forever. The people who establish themselves as experts in a specific silo now — in 2026 — are going to be the ones writing the job descriptions in 2028.
The window is closing faster than people think. The generalist "AI engineer" role is already starting to feel like the "webmaster" role of the late nineties — it made sense when the field was new and nobody knew what the specializations would be. But now the specializations are clear, and the market is rewarding people who go deep.
The question isn't whether you'll adapt to agentic AI. It's which direction you'll specialize.
That's where we want to hear from listeners. What specializations are you seeing in your work? Are you hiring for these roles? What skills are you finding hardest to recruit for? Send your thoughts to prompts at myweirdprompts dot com, or tag us on X at myweirdprompts.
We're genuinely curious — the field is moving so fast that the people building these systems right now have the best view of what's actually happening on the ground.
Now: Hilbert's daily fun fact.
Hilbert: In 1937, Japanese physicist Yoshio Nishina proposed that muon-catalysed fusion — where a muon replaces an electron in a hydrogen molecule, shrinking the atomic radius and allowing fusion at room temperature — could solve humanity's energy needs. The theory was mainstream for nearly a decade until experiments revealed that each muon could only catalyse about a hundred fusion reactions before decaying, making the process energetically worthless. Nishina later pivoted to studying cosmic rays on the Kuril Islands, where he spent three years measuring muon flux at different altitudes and concluded, quote, "nature is not an engineer.
Nature is not an engineer.
actually a pretty good summary of the whole muon fusion problem.
This has been My Weird Prompts. I'm Herman Poppleberry.
I'm Corn. If you enjoyed this episode, share it with someone who's trying to figure out where they fit in the agentic AI landscape. We're at myweirdprompts dot com. Until next time.
