Daniel's been building an aggregated MCP setup, one gateway fronting a bunch of remote servers, and he keeps hitting the same wall. The moment a tool needs an actual binary, a real image file, the whole thing goes sideways. He's got a local image he wants a remote generation server to work on, but MCP is JSON-RPC, so there's no obvious way to hand over the bytes. His workaround is staging the file to object storage and passing a URL. It works. But it feels like routing around the protocol, and every binary-touching tool ends up needing its own middleware standing beside it. So his question is fourfold: what's the right way to get binary payloads to a remote server in an aggregated setup, is the stage-and-pass-a-URL pattern the intended answer or a workaround for a gap, does the transport underneath actually change anything, and is there something the protocol should be doing here that it isn't.
This is one of those questions where the answer starts with what the spec can and cannot express. Because the awkwardness Daniel's feeling isn't a configuration problem. It's a design question about what MCP is actually for.
Which is what, exactly?
Tool invocation. MCP is a protocol for describing tools and invoking them. The entire data model is JSON-RPC messages, and every payload has to be representable as a JSON value. Raw bytes are not a JSON value. There's no binary type in the schema vocabulary. The tools page in the current spec revision, that's the July twenty-eighth one, defines tool inputs as JSON Schema objects. Strings, numbers, booleans, arrays, objects. That's the full vocabulary.
So the protocol has no word for "here is a picture."
Correct. And it's not an oversight. JSON itself has no binary type. JSON-RPC inherits that. So the question of "how do I move bytes" immediately becomes "how do I represent bytes as something JSON can carry." And there are exactly three answers in practice.
Three. Base64, URL reference, and... something else.
The something else is the working group. But in the current spec, two patterns. Base64 inline, which encodes the bytes as a string. It works, it's legal JSON, and it breaks down fast. Base64 adds about thirty-three percent overhead, so a four megabyte file becomes five and a half megabytes of string. And then that string has to live in the model's context window if you're passing it through an LLM, which burns tokens at a rate that makes it impractical for anything beyond a small icon. Daniel's own take on this was right, by the way, for small audio clips inline is fine. For anything bigger, it stops being fine quickly.
And the practical limit people hit is around four megabytes in MCP contexts.
Four megabytes is the number that comes up constantly. It's not a spec limit, it's a practical one, but it's real. So base64 is the answer for tiny things and a trap for everything else. Which leaves the URL pattern.
Stage the file to object storage, pass a URL string.
And here's the thing Daniel's intuition is getting wrong. He says it feels like routing around the protocol. But it's not a workaround. It's the only pattern the current spec can express. A URL is a JSON string. JSON strings are legal tool arguments. So the stage-to-bucket-and-pass-a-URL pattern isn't a hack on top of MCP. It's the maximum of what MCP can say natively.
That's a subtle distinction. It's not that the protocol permits it. It's that the protocol has nothing else to offer.
If you want to move an image to a remote tool through MCP today, the bytes never enter the protocol. The protocol carries a pointer. The pointer is a string. The string is legal. Everything else happens outside MCP, in whatever storage layer you've stood up.
Which is where the middleware tax comes from.
Right. Daniel says every binary-touching tool needs middleware standing beside it, and that's structural. Because the gateway can't help. The protocol gives the gateway nothing to intercept. If a tool needs a binary, the tool's own implementation has to know how to fetch a URL, authenticate against the bucket, handle expiry, handle cleanup. The gateway is blind to all of it. It just sees a string argument.
So the gateway's whole value proposition, which is centralizing and routing tool calls, evaporates the moment a tool needs a file.
It doesn't evaporate, but it stops adding value. The gateway still routes the JSON-RPC call. But the actual payload movement happens in a parallel universe the gateway can't see, can't log, can't audit, can't rate-limit. And every server that wants binary input has to re-solve the same problems independently.
Which is the opposite of what an aggregation layer is supposed to do.
Yes. And this is where the transport question comes in, because Daniel asked whether stdio versus streamable HTTP changes anything. And the answer is genuinely no. Not today.
That surprised me. I'd assumed HTTP would have some advantage.
It doesn't, because both transports carry the same JSON-RPC messages. Stdio is a local pipe, streamable HTTP is a remote POST, but the message body is identical. The streamable HTTP spec, same July twenty-eighth revision, defines message framing. It tells you how to delimit JSON-RPC messages over HTTP. It says nothing about attaching raw bytes. There's no multipart support, no side channel, no binary frame. So whether you're on stdio or HTTP, the payload problem is identical. The only thing HTTP gives you is a natural place to host a separate upload endpoint, but that endpoint would be outside the MCP protocol entirely.
So the transport is orthogonal.
Completely orthogonal today. The bytes can't ride on either transport. The URL pattern is the only option on both. Which is worth saying plainly, because I've seen people assume that switching from stdio to streamable HTTP unlocks binary support. It doesn't. It just changes where the JSON-RPC messages go.
So that's the current state. The protocol can't express binary payloads, the transport doesn't matter, and the URL pattern is the only legal move. Now the more interesting question is whether that's where the protocol should stay.
And there's a signal that the protocol's authors think it shouldn't. There's a File Uploads Working Group chartered in the MCP community index. Active effort. Which tells you the maintainers consider binary payloads an open problem, not a settled one.
What does the charter actually say?
It's a charter, so it's a statement of intent rather than a design. But the existence of it is the important part. The people who own the spec have looked at this and said, yes, this is a gap worth closing. The question is what shape the fix takes, and that's wide open.
What are the options?
Three main directions. One, a new content type in the JSON-RPC envelope. You'd define a way to embed binary data in the message itself, probably still base64 but with explicit binary semantics, so the gateway can recognize it and handle it differently. Two, a multipart extension to streamable HTTP. That's the HTTP-native approach. You'd send the JSON-RPC message as one part and the binary payload as another part. It's how web forms have handled file uploads for decades. It works, but it's HTTP-only. Three, formalize the URL reference pattern. Give it lifecycle semantics. Expiry, cleanup, size limits. Make it a first-class part of the spec instead of a convention people reinvent.
That third one is interesting. It's saying, the pattern Daniel's already using isn't a workaround, it's the design, and we should bless it.
And there's a real argument for that. URLs are cacheable. A URL to an object in a bucket can be fetched by multiple servers, can be re-fetched if the first attempt fails, can be logged, can be audited. Bytes embedded in a JSON-RPC envelope are none of those things. They're ephemeral, they're single-use, they're invisible to every layer except the one that decodes them.
So the URL pattern has properties that an envelope-based approach would lose.
It does. And that's the tension the working group is going to have to resolve. Do you optimize for the common case, which is small files that could live inline, or do you optimize for the hard case, which is large files that need lifecycle management? Because those pull in different directions.
The aggregation problem is the sharp edge here.
It is. Because a gateway fronting many servers can't know which tools need binary input without schema introspection. The current spec gives it no way to advertise that capability. A tool's schema says it takes a string. The gateway can't tell whether that string is a URL the remote server will fetch, or a username, or a freeform text field. So the gateway can't route binary-capable tools differently, can't enforce size limits, can't apply different auth policies. It's blind to the distinction because the schema doesn't encode it.
So even if the working group lands a fix, the aggregation layer has a whole second problem: how do you advertise binary capability in a way the gateway can act on.
Right. And that's a gateway problem, not a transport problem. The transport just moves messages. The gateway has to understand what the messages mean. And right now, the messages don't mean anything about binary payloads. They just say "string."
Let me push on the transport question one more time, because I think there's a future dimension Daniel's asking about without quite saying it.
Go ahead.
If the working group lands a multipart extension, that only works over HTTP. Stdio deployments would be left with the URL pattern forever. So the transport question, which is orthogonal today, becomes non-orthogonal the moment the spec blesses a particular mechanism.
And it's a real fork in the road. If the working group chooses multipart, they're implicitly saying that binary payloads are an HTTP concern. If they choose the formalized URL pattern, they're saying binary payloads are an application-layer concern, and the protocol just needs to give them a name and some rules. Those are very different philosophies about what MCP should own.
And stdio is not going away. Local MCP servers are a huge part of the ecosystem. So a fix that leaves stdio out in the cold is a fix that splits the protocol.
Which is why I suspect, and this is pure speculation, that the URL pattern has more staying power than people think. It's transport-agnostic. It works over stdio, over HTTP, over anything else that carries JSON-RPC. It's the only pattern that doesn't require the transport to grow a new capability.
The deeper insight here is that MCP is a protocol for tool invocation, not data transfer. And the working group's existence suggests the community is deciding that distinction is no longer tenable.
That's the second-order question. Not "how do I move bytes today," but "what should the protocol own, and what should it leave to the application layer." Because you could argue that binary payloads are an application-layer concern. The protocol says "invoke this tool with these arguments." What the tool does with a URL argument is the tool's business. The protocol doesn't need to know.
But the counterargument is that the protocol is the only place where the gateway can see what's happening. If binary handling lives entirely in application code, the gateway can never help.
Right. And that's the aggregation problem in a nutshell. The gateway is the one place where you could centralize auth, cleanup, size limits, audit logging. But the protocol gives it no vocabulary to do so. So every binary-touching tool re-implements the same logic, and the gateway sits there watching strings go by.
What do other protocols do? You mentioned multipart for HTTP forms. What about gRPC?
gRPC has streaming channels. It's a binary protocol from the ground up, so bytes are first-class. You can stream a file as a sequence of chunks over a gRPC channel, and the framework handles framing, backpressure, all of it. But gRPC made that choice at the protocol level. MCP chose JSON-RPC, which is a text protocol. And text protocols are always going to be awkward about bytes.
HTTP itself was a text protocol that grew multipart to handle file uploads.
And that's the precedent the multipart camp is pointing to. HTTP was designed for hypertext, and file uploads were bolted on later. It worked. But it took years, and it's still awkward in places. MCP is in that same position now, except it has the benefit of seeing how HTTP solved it.
Let me walk through Daniel's concrete scenario, because I think it makes the tradeoffs vivid. He's got a local image. He wants a remote image generation server to work on it. What actually happens today?
He stages the image to object storage. That gives him a URL. The URL is a string. He invokes the remote tool through the gateway, and the tool arguments include that URL string. The remote server receives the JSON-RPC call, sees the URL, fetches the image from the bucket, does its generation work, and returns a result. The gateway never sees the image. It saw a string go by, and maybe a result come back.
And the middleware tax is everything around that.
The staging logic. The bucket auth. The cleanup job that deletes the staged image after some TTL. The error handling for when the remote server can't reach the bucket. All of that is custom code that has to live somewhere, and it has to be replicated for every tool that needs binary input. The gateway can't do any of it, because the gateway doesn't know the string is a URL to a binary.
And if the remote server is on the other side of a network boundary, the image makes two network hops. Once from Daniel's machine to the bucket, once from the bucket to the remote server.
Three if you count the JSON-RPC call itself. And each hop has its own failure modes. The bucket could be down. The remote server could have different network egress rules than Daniel's machine. The URL could expire before the remote server fetches it. It's a distributed system problem, and the protocol gives you no help with any of it.
So the question of "is this the intended answer" has a weird answer. It's not intended. The protocol never intended anything about binary payloads. It's simply the only answer the protocol can express. Intention implies a design decision. There was no design decision. There was a gap, and the URL pattern is what fills it.
That's the precise way to put it. The URL pattern is not blessed by the spec. It's not condemned either. It's just the thing that happens when you have a JSON string and a remote server that needs bytes. The spec is silent, and silence is not the same as intention.
Daniel says he's never seen this addressed properly. I think that's because the people building MCP systems have all independently discovered the URL pattern and moved on. It's not documented as a pattern because it's not a pattern. It's just what you do.
And the working group is the first sign that the silence is ending. The charter is a signal of intent. But it's a charter, not a shipped feature. Nothing has landed yet. The design space is wide open. Anyone building a gateway today has to assume the URL pattern is what they're supporting, because it's the only thing that exists.
Which brings us to the practical question for gateway builders. What do you do right now?
Right now, you support the URL pattern well. That means your gateway should at least be able to recognize when a string argument looks like a URL, and log it differently, and maybe apply different rate limits. You can't do that at the protocol level, but you can do it heuristically. It's brittle. It breaks the moment someone passes a URL that doesn't look like a URL to your heuristics. But it's better than nothing.
And you should be watching the working group.
Because whatever they land is going to change what gateways need to support. If it's the formalized URL pattern, gateways get a clean way to advertise and route binary capability. If it's multipart, gateways get a whole new transport concern, and stdio deployments get left behind.
Let me ask the question Daniel didn't quite ask. Is the URL pattern actually bad?
It's not bad. It's got real advantages. Cacheability. Auditability. Resumability. You can put a URL in a log and a human can read it. You can't do that with a base64 blob. The URL pattern is the only approach that gives you any operational visibility into what's moving.
So the working group might land on formalizing the URL pattern and call it a day.
They might. And there's a decent argument that they should. The URL pattern is transport-agnostic, it's simple, it's already widely deployed, and it has operational properties that an envelope-based approach would lose. The main thing it's missing is lifecycle semantics. Expiry, cleanup, size limits. Those are conventions today. They could be spec text tomorrow.
The counterargument is that formalizing the URL pattern means every MCP deployment needs access to object storage. That's a real operational burden.
It is. And that's the argument for an envelope-based approach. If the bytes ride in the message, you don't need a bucket. You don't need cleanup jobs. You don't need a second network hop. The bytes go where the message goes. For small files, that's strictly simpler.
So the design space is a tradeoff between operational burden and operational visibility.
That's a good way to frame it. Envelope-based is less infrastructure but less visibility. URL-based is more infrastructure but more visibility. And the working group has to pick a direction, or support both, which is its own kind of complexity.
What would supporting both look like?
A spec that says, small payloads can ride inline, large payloads must use the URL pattern, and here are the size thresholds and lifecycle rules. That's probably where this ends up, honestly. It mirrors what HTTP did. Small forms inline, large files via multipart.
But multipart is HTTP-only. So the inline-or-URL split would work over stdio, but the inline-or-multipart split wouldn't.
Right. Which is another argument for the URL pattern as the universal answer. It's the only answer that doesn't fork the protocol by transport.
Hilbert: The URL is the carrier.
Go on.
Hilbert: I ran a one-hour photo lab in the late nineties. Strip mall in Waterbury. The whole business was moving pictures between machines that couldn't talk to each other. Camera gave you film. Enlarger wanted a negative. Scanner wanted a print. Nothing spoke the same language. So you put the negative in a carrier. The carrier had a number. You wrote the number on the order envelope. The enlarger operator pulled the carrier by number. Nobody handed anybody a raw negative. You handed them a location.
The staging area was the point.
Hilbert: The staging area was the whole job. The carrier was a URL. It had a number, it had a slot on the rack, it had an owner. The negative didn't move through the air. It sat in the carrier until someone needed it. Then it went back.
That's the lifecycle semantics the working group is missing. The carrier had a place. It had a checkout process. It had a return process.
Hilbert: If you lost the carrier, you lost the job. So you didn't lose the carrier. You kept a log. You knew which carrier was where. That log was the only reason the place ran.
The protocol has no log.
Hilbert: The protocol has no rack. No carrier. No log. It has a string. A string could be anything. My carrier was a thing. It had weight. It had a number written on it in marker.
You'd formalize the URL pattern.
Hilbert: Give it a number. Give it a slot. Give it a rule for how long it sits there before you throw it out. That's all a carrier is. You don't need to teach the pipe to carry film. The pipe never carried film. The carrier carried film. The pipe carried the number.
That's the cleanest argument for the URL pattern I've heard yet. The protocol shouldn't carry bytes. It should carry references to bytes, and it should formalize what a reference means.
Hilbert: The light table's still in my basement. The carrier rack's still on it. I should throw it out. I don't.
The point about the pipe carrying the number, that's exactly the distinction. The protocol is the pipe. It moves JSON-RPC messages. The bytes are the film. They were never going to go through the pipe. The question is whether the protocol names the carrier or pretends the carrier doesn't exist.
Hilbert: Right now it pretends. That's why every shop builds its own carrier. Different numbers. Different racks. Different rules. Same job.
The middleware tax is the cost of every shop building its own carrier.
Hilbert: Then the gateway's just a guy at the counter who doesn't know which carrier is which. He sees a number go by. He doesn't know if it's a picture or a phone number.
That's the schema introspection problem. The gateway can't tell a URL from a username.
Hilbert: My counter guy could tell. He had a rack. The rack was the schema. Number on the envelope, slot on the rack. You knew what you were holding.
The working group is essentially deciding whether to build a rack.
Hilbert: Or keep letting everyone build their own. Which is fine if you're one shop. It's not fine if you're forty shops and the guy at the counter can't tell what's coming through.
That's the aggregation problem in one sentence.
Hilbert: I should get rid of that light table.
What we're left with is a protocol that can't express binary payloads, a URL pattern that works but isn't blessed, and a working group that's deciding whether to bless the pattern or replace it with something transport-specific. The open question is which way they'll go.
The stakes are higher than they look. If they pick multipart, stdio deployments are frozen out of first-class binary support forever. If they pick the formalized URL pattern, every MCP deployment needs object storage, but the gateway can finally see what's moving. If they pick both, the spec gets more complex but the ecosystem gets a clean migration path.
The misconception people have is that the URL staging pattern is a hack. It's not. It's the only thing the current spec can express, and it has real advantages that an envelope-based approach would lose. The hack is the absence of lifecycle semantics, and that's what the working group is positioned to fix.
The second misconception is that switching transports unlocks binary support. It doesn't. Not today. Stdio and streamable HTTP are equally blind to bytes. The transport question only becomes real when the spec decides to make it real.
What I'm watching for is whether the working group formalizes the carrier before someone builds a de facto standard that the spec has to chase.
That's the race. The URL pattern is already the de facto standard. Everyone's using it. The question is whether the spec catches up and names it, or whether the community splinters into fifty different carrier designs and the spec has to pick a winner retroactively.
The gateway layer is where that splintering hurts most, because the gateway is the one place that could centralize all of this if the protocol gave it the vocabulary.
The forward-looking question isn't "how do I move bytes." It's "how do I advertise and route binary capability through an aggregation layer." That's a gateway problem. The transport is a sideshow. The spec is the bottleneck.
Thanks to Hilbert Flumingtop for producing, as always.
This has been My Weird Prompts. If you want to dig into the spec yourself, the working group charter is public in the MCP community index. Send us your questions at show at my weird prompts dot com.
We'll be back soon.
