Daniel's been on what he calls an episode generation binge, and the site's been paying for it in redundant builds. His question is whether there's a professional way to stage this whole production pipeline so that capture stays ad-hoc but processing becomes deliberate, batched, and cheap.
And the thing is, he's already tried the obvious fix and it bit him. He built a polling queue, and the polling itself produced the project's only surprise bill. So the real question underneath all of this is what does a queue look like when you're not allowed to keep a process alive.
Right. He wants to keep the episode creation form exactly as it is, send prompts whenever they occur to him, but have the system hold them in a staging pattern. Two queues, he says. One for generation, one for deployment. Each stage always a batch. And he's asking what the most professional version of that looks like, plus what the cost benefits are, plus what product offerings he should be watching now that providers are starting to think about batch inference as a first-class thing.
And he flagged Modal specifically. No native batch scheduler there, as far as he can tell. So he's asking what the landscape looks like if your long-run AI workload is heading in this direction.
The question underneath it all is what a queue is when your process is disposable. Let's start there.
A serverless pipeline has no long-lived process. There's no worker sitting in memory with a channel of pending jobs. So the word queue has to mean something else. It means durable state plus a trigger that fires at the right moment. The state lives somewhere that survives between invocations, and the trigger is a scheduled event, not an in-memory poller.
And Daniel's actually asking for two different things that he's calling queues. The generation queue is prompts waiting to become episodes. The deployment queue is finished episodes waiting to go live. They have different cadences and different failure modes, and treating them as the same kind of thing is how you end up with a polling loop that bills you for doing nothing.
The ad-hoc form is actually good design. Capture should be immediate and frictionless. But capture and processing need to be decoupled. The form should write to durable storage and return instantly. It should never trigger work inline. That's the whole trick. The form feels instant, but behind it, nothing has happened yet except a row appearing in a table.
And the cost dimension is not cosmetic. Batching is the difference between paying for five cold starts and paying for one warm batch. When you're running serverless inference on GPU time, that gap is real money. So the architecture question and the economics question are the same question.
Let's walk through why the obvious thing falls apart. Daniel's site rebuilds on a repository push or a deploy hook. Every push is an independent event. The build system has no concept of coalescing five events into one build. So if five prompts arrive in ten minutes, that's five builds, each rebuilding the same site with one more episode in the database.
And the work is about eighty percent redundant. The site isn't meaningfully different after the first build except for one new page. Everything else gets rebuilt identically five times. Five builds at five minutes each is twenty-five minutes of build time to produce what one build could have produced at the end of the batch.
The polling queue failed for a different reason. A serverless function that polls a database table for a queue number runs on a timer, forever, whether or not there's work. That's the canonical serverless anti-pattern. You're paying for idle time at a per-invocation rate, and the bill scales with wall-clock time, not with work actually done.
Let's put a number on it. A function that checks a table every minute costs forty-three thousand two hundred invocations per month to do nothing. That's before it processes a single item. And if the table is empty, which it is most of the time, every one of those invocations is pure waste. Daniel said that was the only surprise bill the project ever had. That's not a coincidence. That's what polling costs.
The fix isn't a better polling loop. It's not polling at all. The correct primitive is scheduling. A cron-triggered function wakes on a fixed cadence, drains whatever has accumulated in the queue since the last tick, and exits. No idle cost between runs. No per-poll charge. The batch size is naturally bounded by how many items showed up since the last run.
And this is where the two-queue architecture becomes clear. Queue A is generation. The form submission writes a row with status pending. A scheduled job picks up all pending rows, runs the inference batch, and writes episodes with status draft. Queue B is deployment. A separate scheduled job, or a manual approval step, picks up all approved episodes and triggers exactly one build.
Each stage is a batch. Each stage is idempotent. Each stage can be re-run safely if something crashes halfway. That's the professional pattern. It's not one queue, it's two queues with a state transition between them.
The human-in-the-loop insertion point is almost embarrassingly simple once you see it. Because Queue A writes drafts and Queue B only ships approved items, the review step is just a status transition. A column changes from draft to approved. No new infrastructure. No new queue. The holding pattern Daniel wants is literally a column value in a database.
And that's the inversion that makes serverless feel weird until it clicks. In a traditional system, the queue lives in the process. A worker holds jobs in memory and works through them. In serverless, the queue lives in the database, and the process is disposable. The function wakes, reads the queue, does the work, writes results, and dies. The queue persists because the database persists.
Daniel's polling attempt felt natural because it tried to make a disposable process behave like a persistent one. He wanted the function to watch the queue, which is what a worker does. But a serverless function that watches is a function that runs forever, and forever is billed per second.
Let's run the concrete example. Five prompts submitted between nine and nine-thirty in the morning. Under the old model, five builds, roughly twenty-five minutes of build time, five cold starts. Under the staged model, one generation batch at the next cron tick, five drafts written, one review pass, one deployment batch, one build. Same output, one-fifth the compute.
And the review pass is the part that used to be a bolt-on. Now it's a first-class stage. Daniel said he wants to move toward more human-in-the-loop review. This architecture doesn't just accommodate that, it makes it the natural shape of the pipeline. The human is between the two queues, and the pipeline waits for them.
There's a subtlety with idempotency here. A scheduled batch job that crashes halfway must be safe to re-run. If the generation job picks up five pending rows, processes three, and dies, the next run needs to pick up the remaining two without duplicating the first three. That means status columns, unique constraints, and a claim-then-process pattern. You mark rows as in-progress before you start generating, and if the function dies, those rows can be re-claimed by the next run.
The old in-memory worker gave you those guarantees for free because the process just kept running. If it crashed, you lost the whole queue and started over. In serverless, the database gives you durability, but you have to build the claim semantics yourself. It's a trade, and it's one of the few places where serverless asks more of the developer.
I think it's worth naming the thing that makes this feel harder than it is. Most people think a queue requires a long-running worker or a polling loop. In serverless, a status column plus a scheduled batch job is the queue. That's the whole pattern. Once you see it, the polling loop feels like trying to build a campfire out of matches instead of just using the stove.
The other misconception is that batching is about tidiness. It's not. It's an economic decision. The batch discount is the point. If you're already willing to wait for a cron tick, you're already in batch territory, and you should be collecting the discount.
That's the architecture. Now let's talk about what it costs, and what the provider landscape looks like if you're heading this direction.
The big knock-on effect is that batching changes the economics of inference, not just builds. The major providers now offer batch APIs with roughly fifty percent discounts on inference in exchange for relaxed latency. That's a perfect fit for a pipeline that is explicitly not time-critical. Daniel said this podcast production is never time-critical. He's already living in the batch world. He just hasn't been collecting the discount.
The discount is the product. The latency is the price. If you're willing to wait for a cron tick, you're already paying the latency. So the question becomes why wouldn't you take the fifty percent off.
The batch discount reframes when should this run from a latency question to a scheduling question. If you're generating five episodes in a batch, you don't need any of them in the next ten seconds. You need them whenever the batch finishes. That's exactly the trade the batch APIs are offering.
The provider landscape is bifurcating. Some providers offer explicit batch endpoints with turnaround windows measured in hours. You submit a file of requests, they process it when they have spare capacity, and you get results back later. Others offer scheduled or cron-triggered serverless functions where you bring your own batching logic. You write the queue table, you write the scheduled function, and you call the regular endpoint from inside it.
Modal sits closer to the second camp. No native batch scheduler, as Daniel suspected, but scheduled functions plus your own queue table gets you there. The provider gives you the cron trigger and the compute. You bring the queue semantics. It's more work than an explicit batch endpoint, but it's also more flexible. You're not constrained by someone else's turnaround window.
The queue table itself is the durable state that serverless otherwise lacks. That's the inversion again. In a non-serverless world, the queue lives in the process. In serverless, the queue lives in the database, and the process is disposable. The polling attempt felt natural because it tried to make a disposable process behave like a persistent one. The scheduled batch job is the correct pattern because it accepts disposability and puts the state where state belongs.
And once you accept that, the staged model stops being a compromise forced by serverless. It's arguably the better architecture for this workload, because it makes the human review step a first-class part of the pipeline rather than a bolt-on. Daniel said he wants to move toward more human-in-the-loop review. This architecture doesn't just accommodate that. It makes the human the natural gate between the two queues.
The form stays ad-hoc. That's the part people get wrong. They think staging means you have to change how you capture input. You don't. The form writes to durable storage and returns. Staging happens downstream. The person sending prompts never knows or cares whether the next batch runs in five minutes or five hours.
And the cost benefits compound. The build savings are real, but they're small compared to the inference savings. If you're running serverless GPU time on Modal, and you can shift from five separate inference runs to one batched run, you're saving on cold starts, on idle time, and potentially on the inference itself if you're using a batch-discounted endpoint. The build savings are the cherry. The inference savings are the sundae.
There's a subtlety with the batch discount that's worth being honest about. The fifty percent figure is the headline number, but it comes with trade-offs. The turnaround window might be hours, not minutes. The endpoint might have lower rate limits. If your pipeline is truly not time-critical, none of that matters. But if you have a day where you want an episode out the door in an hour, the batch endpoint might not be the right tool for that job.
That's where the two-queue design earns its keep. The generation queue can be slow and cheap. The deployment queue can be fast and manual. You can run generation through a batch-discounted endpoint and still trigger a build the moment you approve the drafts. The latency is absorbed at the stage where latency doesn't matter, and preserved at the stage where it does.
The idempotency point from earlier gets more important when you're using batch endpoints. If you submit a batch of five prompts to a provider and the request times out, did it process zero of them or all five? You need to be able to check and re-submit without duplicating. That means the queue table needs to track submission status, not just processing status. Claim then process, and record the claim before you fire the batch request.
The provider landscape is worth a quick survey. The explicit batch endpoint camp includes the big model providers. They've all shipped something like this in the last couple of years. The scheduled function camp includes the serverless compute platforms. They give you cron triggers and you build the queue yourself. And there's a third camp emerging, which is the orchestration platforms that sit on top and manage the queue for you. They're not quite batch endpoints and not quite raw compute. They're the middle layer.
For someone like Daniel, who's already on Modal and already has a queue table, the scheduled function path is probably the right one. He doesn't need to move providers. He needs to add a cron trigger and a status column. The explicit batch endpoints are more interesting if he's doing high-volume inference where the fifty percent discount really adds up.
And the queue table itself is the durable state that serverless otherwise lacks. That's the inversion one more time. The database isn't just where the episodes live. It's where the work lives. The process is a disposable thing that wakes, reads the work, does it, and dies. The queue is the database, and the database is the queue.
I keep coming back to the polling bill. Forty-three thousand two hundred invocations a month to do nothing. That's the number that should be tattooed on every serverless developer's forearm. The moment you write a function that checks a table on a timer, you've made a decision about cost that has nothing to do with the work you're doing.
And the fix is so simple it feels like cheating. Instead of checking every minute, check every ten minutes. Or every hour. The cadence is a business decision, not a technical one. If Daniel's pipeline is never time-critical, he could run the generation batch once an hour and nobody would notice except his bill.
The deployment batch is different. That one should probably stay manual, or at least human-gated. The whole point of the review step is that a person looks at the drafts and says yes. The deployment job should fire when the person says yes, not on a timer. That's the one place where event-driven is still the right model.
That's the nuance that makes the two-queue design more than just batching. Generation is cadence-driven. Deployment is approval-driven. They're different triggers because they're different stages with different failure pattern. A generation batch that runs late is fine. A deployment that runs before the human approves it is a problem.
The fully automated pipeline that Daniel mentioned is the trap. It feels efficient because it removes the human, but it removes the human from the wrong place. The human should be between the stages, not before the first one. The form can stay automated. The review should be deliberate.
There's a broader implication here that I think is worth naming. As more providers ship explicit batch and scheduled offerings, the default pipeline shape is going to shift from event-driven to cadence-driven. The old model was react to every event immediately. The new model is collect events, then process them on a rhythm. That shift has design consequences well beyond this podcast.
It changes how you think about latency. If your pipeline is cadence-driven, you stop asking how fast can this run and start asking how long am I willing to wait. Those are different questions, and they lead to different architectures.
It changes how you think about failure. An event-driven pipeline fails loudly and immediately. A cadence-driven pipeline fails quietly and later. You need different monitoring, different alerting, different expectations about what normal looks like.
The queue table becomes the dashboard. If you want to know how the pipeline is doing, you look at how many rows are pending, how many are in progress, how many are approved. That's the whole system, in one table.
Hilbert: I worked a summer at a commercial laundry in the late nineties. My job was running the industrial washers. The rule was you never ran a washer with one bag of linen. You waited until you had a full load, because the machine cost the same to run either way. One bag or ten bags, same water, same soap, same electricity. So you let the bags pile up until the machine was full, and then you ran it.
Hilbert: The whole episode is just that rule with more steps. You don't run the machine for one bag. You don't run the build for one episode. You wait until you have a pile, and then you run it once.
Hilbert: But I want to correct a word. You keep saying queue. The laundry never had a queue. It had a pile. A queue implies order and fairness. First in, first out. A pile just means you grab whatever's on top when the machine is free. The status-column design you're describing is a pile. That's fine. Piles are fine. But calling it a queue is the kind of thing that makes people build polling loops.
The pile distinction is good. A queue has order. A pile has membership. And the status column is really just membership with a state attached. You're not promising to process these in order. You're promising to process them eventually.
The word queue carries assumptions about fairness that the architecture doesn't need. If five prompts arrive in a batch, does it matter which one gets generated first? No. They all end up in the same build. The pile is the honest description.
Hilbert: I ran a half-load once. Nineteen ninety-four. I wanted to get home early, so I threw one bag of sheets in the washer and started it. The manager found out and docked my pay for the wasted cycle. Forty dollars. I've never forgiven him. His name was Ray Kowalski. I bring this up about once a year.
Hilbert: Forty dollars in nineteen ninety-four. That's what a half-load cost. One bag of sheets. The machine didn't care. It ran the same cycle. But Ray Kowalski cared, because he was paying for the water and the soap and the electricity, and he wasn't going to pay for a half-load.
The forty dollars is the cold start. That's the fixed cost of running the machine regardless of how much work it does. The batch discount is just Ray Kowalski's rule applied to inference. Don't pay the fixed cost for one bag. Wait until you have a full load.
The surprise bill from polling is the inverse of Ray Kowalski's rule. Ray was paying for the machine to run when it had work. The polling function was paying for the machine to run when it had nothing. It's the half-load in reverse. You're not wasting a full cycle on one bag. You're wasting a full cycle on zero bags.
Hilbert: The pile is the right word. You don't need a queue. You need a pile and a schedule. The schedule says run the machine when the pile is big enough or when it's been long enough. The pile says what's waiting. That's all.
The pile also has a natural batching property that a queue doesn't. A queue processes one at a time. A pile processes whatever's there when the machine runs. The batch size is whatever accumulated since the last run. That's exactly the cron-triggered model.
The human review step is just someone looking at the pile and saying which of these are good enough to ship. Ray Kowalski would have done that by feel. He'd look at the pile and know which bags were hotel sheets and which were restaurant napkins, and he'd run them in different machines. The human-in-the-loop is the manager looking at the pile.
Hilbert: Ray Kowalski was a cheap man, but he wasn't wrong. The machine cost the same either way. You don't run it for one bag.
The misconception people carry into this is that a queue requires a long-running worker. It doesn't. It requires a table and a timer. The table holds the pile. The timer runs the machine. The worker is disposable.
The other misconception is that batching is about neatness. It's about not paying Ray Kowalski's forty dollars over and over. The batch discount is the point. The pile is the mechanism. The status column is the holding pattern.
The open question I'm left with is whether the batch discounts keep widening. If they do, the fully automated pipeline becomes the expensive choice rather than the convenient one. The event-driven model starts to look like running a half-load every time a prompt arrives.
The deeper question is how much of the serverless awkwardness here is just the absence of a durable queue primitive, and how much is a genuine architectural mismatch. If the providers shipped a pile primitive, would we still be building status columns by hand?
I suspect the queue table is going to remain the answer for a while, because it's the one piece of the system that the provider can't abstract away. The queue is your data. The provider can't own it without owning your workflow.
Thanks to Hilbert Flumingtop for producing. This has been My Weird Prompts, the human-AI collaboration podcast. If you enjoy the show, leave a review wherever you listen. We'll be back soon.
