Daniel sent us this one, and it's a genuinely practical question. He wants to dig into object detection APIs: what they are, what you'd actually use them for, and how to wire them into an automated annotation workflow. The core setup is a two-stage pipeline: first you call an API to get bounding box coordinates and confidence scores for detected objects, then you use something like PIL or Pillow to draw those annotations programmatically onto the image. And the central question he's circling is whether general-purpose multimodal vision models like Gemini can reliably spit out structured bounding box output, or whether dedicated tools like AWS Rekognition, Google Vision API, YOLO-based solutions, or Roboflow are just meaningfully better at that specific job. He also wants the cost picture: pricing models, what's cloud-only versus self-hosted, and which of these you can run locally, including open-source YOLO variants, Grounding DINO, things available on Hugging Face.
There's a lot to unpack there. And by the way, today's episode is powered by Claude Sonnet four point six, which feels appropriate for an episode about what vision models can and can't do.
The friendly AI down the road, writing our lines for us. Let's think about where this actually shows up before we get into the API weeds, because the use cases are everywhere. A warehouse retailer running automated inventory checks, a camera scanning shelves and flagging which slots are empty, which products are misplaced. That is object detection doing real work in a real workflow. Not a demo, not a research paper. Actual operational infrastructure.
That retail inventory case is a good one to anchor on because it captures both what makes these systems impressive and where they fall apart under pressure. You're not just asking "is there a cereal box in this image." You're asking "which cereal box, where exactly is its bounding box, how confident are you, and can you give me that output in a consistent structured format so my downstream annotation pipeline doesn't break." Those are four different things, and not every tool handles all four equally well.
Which is basically Daniel's question in a nutshell. The gap between "this model can see things" and "this model reliably returns machine-readable coordinates in a consistent schema" turns out to be surprisingly wide.
Wider than most people expect when they first start building these workflows. And that gap is exactly what makes the choice of tool non-obvious—especially when terms like "object detection API" get thrown around without much clarity.
Right, and that's why it's worth defining what's actually happening under the hood. "Object detection API" is one of those phrases people use before they've thought about what the two words after "object" are doing.
At the most basic level, an object detection API takes an image as input and returns structured data describing what it found and where. Not just labels, but spatial information. The bounding box is typically four values: x and y coordinates for the top-left corner, plus width and height, or sometimes two corner points depending on the format. And alongside that you get a confidence score, usually a float between zero and one, telling you how certain the model is about that detection.
The confidence score matters a lot in practice, because you're almost always setting a threshold. Anything below, say, zero point seven gets filtered out before it hits your annotation step.
And that threshold decision is actually consequential. Set it too high and you miss real objects. Too low and your annotation pipeline is drawing boxes around shadows and label edges. The API gives you the raw output; your workflow logic decides what to do with it. And there's a subtler version of this problem that bites people: the threshold that works well in your test environment often doesn't transfer cleanly to production images. You tune it on a clean, well-lit dataset, and then your production camera has slightly different exposure settings and suddenly you're either flooding the pipeline with false positives or missing detections you were counting on.
The threshold isn't a one-time decision. It's something you're revisiting as your input distribution shifts.
It's more of a dial you keep your hand on than a setting you configure once. And that ongoing calibration work is part of the real operational cost of running one of these systems that doesn't show up in any pricing table.
Which brings us to the second stage. You've got your bounding box coordinates, you've got your confidence scores. Now you're handing those to something like PIL, Pillow, or OpenCV to actually render the annotations onto the image, or to write them out to a file format your labeling tool understands, like Pascal VOC XML or COCO JSON.
That second stage is mostly deterministic. PIL draws a rectangle where you tell it to draw a rectangle. The variability, the unreliability, the part that actually breaks workflows, that all lives in stage one. In the API call itself. Which is why the choice of detection tool is the decision that matters.
Everything downstream is just plumbing.
Though it's worth saying the plumbing has its own gotchas. COCO JSON and Pascal VOC XML represent bounding boxes differently. COCO uses x, y, width, height from the top-left corner. Pascal VOC uses xmin, ymin, xmax, ymax as two corner points. If you're mixing tools in a pipeline and one upstream component gives you COCO format and your downstream labeling tool expects VOC, you get silently wrong annotations. The boxes end up in the right ballpark but offset in ways that are hard to spot visually until you're staring at a misaligned label and wondering why.
The kind of bug that takes two hours to find and thirty seconds to fix.
The absolute worst kind. The plumbing question is actually where Gemini gets interesting, because people hear "multimodal vision model" and assume it works like a dedicated detection API. It doesn't.
What's the actual difference in how you'd call it?
With a dedicated tool like AWS Rekognition or Google Vision API, you send the image, you get back a JSON object with a well-defined schema. Bounding box, label, confidence, every time. The structure is part of the contract. With Gemini, you're prompting a language model, and the output is whatever the model decides to generate. You can ask it to return JSON, you can describe the schema you want, but there's no hard guarantee. The model might return slightly different field names across calls, might wrap the coordinates in prose, might decide to be helpful in ways your parser wasn't expecting.
You're essentially trusting the model's interpretation of your prompt rather than hitting a typed endpoint.
And in January of this year, Google did push an update specifically improving Gemini's bounding box detection capabilities, which brought it meaningfully closer to what dedicated tools do. So this is a moving target. But even with that update, the consistency of structured output is still not on the same level as Rekognition or a YOLO inference endpoint.
When you say not on the same level, are we talking about occasional hiccups or fundamental unreliability?
It's somewhere in between, and it depends heavily on your prompt engineering. If you're careful, if you specify the exact JSON schema, if you use function calling or structured output modes where available, you can get Gemini to return coordinates reliably most of the time. But "most of the time" is a problem when you're running a batch annotation job over fifty thousand retail inventory images and one in twenty returns malformed output.
Because that malformed one breaks the pipeline.
It breaks the pipeline, or you have to build error handling and retry logic that you wouldn't need with a dedicated tool. And that overhead is real engineering cost that doesn't show up in the API pricing comparison. I've seen teams budget a week for integration and end up spending three weeks just on the validation and retry layer because they underestimated how often the model would return something structurally unexpected.
At that point you're paying for engineering hours, not API calls.
Which is the hidden cost that makes the "Gemini is flexible and relatively cheap per call" argument less clean than it looks on paper. There's also a precision question separate from the schema consistency question. Even when Gemini returns well-formed bounding box coordinates, how tight are those boxes?
That's the deeper issue, right? Because a box that's in the right general area but loose at the edges is still a labeling error.
That's the deeper issue. Dedicated tools like YOLO are trained specifically for pixel-level localization. The architecture is optimized for that. YOLOv8, for example, uses a single-stage detection head that's explicitly regressing bounding box coordinates as part of its training objective. Gemini is reasoning about spatial position, which is a different cognitive operation. It tends to do well on coarse localization, getting the general region right, but the boxes can be looser, less precise at the edges.
Which matters enormously in retail inventory, because you might be trying to distinguish between two products sitting right next to each other on a shelf. A loose bounding box that bleeds into the neighboring item is a labeling error.
And YOLO on that same task, running locally with a model fine-tuned on retail shelf imagery, can achieve sub-pixel precision at inference speeds that are ten to a hundred times faster than a Gemini API call for a batch job. The tradeoff is that YOLO is fixed-class. You train it on your category set, and that's what it finds. Gemini can generalize to things it's never been explicitly trained to detect.
Gemini is the tool you reach for when you don't know what you're looking for yet.
That's a clean way to put it. Zero-shot detection, open vocabulary, situations where the category list isn't fixed. Gemini's spatial reasoning in those contexts is impressive. There was work out of the robotics side showing ninety-three percent accuracy on instrument reading tasks using Gemini's agentic vision, which is remarkable for an open-ended detection problem. But for a well-defined production annotation workflow where you know your classes and you need consistent structured output at scale, a dedicated tool wins on almost every dimension.
Which is a less exciting answer than "one model rules everything," but it's the honest one—and honestly, the cost implications make it even messier.
Because the economics don't always point in the same direction as the performance comparison, and that's where things get really complicated.
Walk me through the cloud options first.
AWS Rekognition for object detection is currently sitting at about a tenth of a cent per image. One dollar per thousand images. That sounds cheap until you're running fifty thousand images a day, at which point you're looking at fifty dollars daily just for the detection calls, before you factor in any storage or compute around it.
Which is still not outrageous in absolute terms, but it compounds.
It compounds, and it's unpredictable. That's the thing about pay-as-you-go pricing. Your cost scales linearly with volume, so a spike in batch jobs hits your bill immediately. Google's Vision API runs on a similar model through Cloud Run, somewhere in the range of fifteen cents to just over a dollar per hour depending on what compute tier you're provisioning. The per-image math ends up roughly comparable to Rekognition at moderate volumes, but the billing surface is different because you're paying for compute time rather than per inference.
You're essentially choosing between paying per image or paying for uptime.
Right, and neither is obviously better. Per-image pricing is more predictable for sporadic workloads. Compute-time pricing rewards you if you can batch efficiently and keep utilization high. Roboflow takes a different approach entirely: credit-based pricing starting around forty-nine dollars a month, which gives you a fixed operational cost. Easier to budget, but you're paying that floor even in quiet months.
The classic SaaS versus consumption trade-off. And there's a vendor lock-in dimension to this too, isn't there? If you build your whole annotation pipeline against Rekognition's specific JSON schema and then AWS changes their pricing tier, migrating is not a small job.
That's a real consideration that people underweight when they're just trying to get something working. The schema you build your downstream tooling around becomes load-bearing infrastructure. Switching detection providers later means rewriting your parsing layer, your validation logic, your error handling. It's not insurmountable, but it's not free either. Which is part of why some teams prefer the open-source local route from the start, even when the cloud API would be cheaper in the short term—you own the interface.
Now what about running this locally? Because Daniel specifically asked about YOLO variants, Grounding DINO, Hugging Face. What does that actually look like?
The ultralytics package, which is the main library for YOLOv8, YOLO11, and now YOLO26n, is a pip install away. You pull the model weights, you run inference locally, and your marginal cost per image is essentially zero once you've got the hardware. That's the appeal. A YOLOv8 nano model runs on a CPU if you're not in a hurry, or on a modest GPU if you are. On a Raspberry Pi you're looking at something like ten to fifteen frames per second with the nano variant, which is usable for a lot of real-world edge deployment scenarios.
That Raspberry Pi number is actually wild to me. Like, that's a thirty-five dollar piece of hardware doing real-time detection.
It's one of those things that still surprises me when I say it out loud. The nano model is tiny—around three million parameters, which is almost nothing by modern standards—but it's been distilled and optimized specifically for this task, so the efficiency is remarkable. You're not getting the accuracy of a larger model, but for a fixed category set in a controlled environment, the nano variant is often more than sufficient. It's a good reminder that "bigger model" isn't always the right answer when your problem is well-defined.
The "local is always cheaper" assumption breaks down where exactly?
Hardware acquisition and engineering time. A GPU instance that can run YOLO at production throughput costs real money to buy or rent. And you're now responsible for model versioning, infrastructure maintenance, and the fine-tuning work if your category set isn't covered by the pretrained weights. The cloud API charges you for the inference but absorbs all of that operational overhead.
Grounding DINO is a different beast though. That's not a YOLO-style fixed-class detector.
No, Grounding DINO is a transformer-based open-vocabulary model. You pass it a text prompt alongside the image, something like "cereal box, price tag, empty shelf slot," and it detects those categories without needing to have been fine-tuned on them. It's available on Hugging Face, and for a small-scale project it's powerful. The catch is inference speed. It's significantly heavier than a YOLO nano model, so if you're running it on CPU you're looking at several seconds per image rather than milliseconds.
How much heavier are we talking? Like, order of magnitude?
On a CPU, a YOLO nano model might process an image in under a hundred milliseconds. Grounding DINO on the same hardware is more like three to eight seconds depending on image size and how many text categories you're passing in. On a good GPU that gap narrows considerably, but it never closes entirely. The transformer attention mechanism is doing fundamentally more work to align your text prompt with image regions, and that computation has to happen somewhere. There's also RF-DETR in the same transformer family, which has been showing strong benchmark numbers recently and is worth keeping an eye on for anyone who needs open-vocabulary detection at better throughput than Grounding DINO.
Which is fine for a research project or a low-volume annotation task, but not for anything with real throughput requirements.
And that's where the hybrid approach becomes interesting. You could use Grounding DINO to handle the novel or ambiguous categories, the things you haven't trained a YOLO model on yet, and then route your known high-volume categories through a fine-tuned YOLO endpoint that returns consistent structured output at speed. You're not picking one tool for the whole job.
The tools aren't actually in competition. They're covering different parts of the problem space.
Which is probably the more useful mental model for anyone actually building one of these annotation pipelines. The question isn't "YOLO or Gemini," it's "what does my category set look like, what's my volume, and where does the precision requirement actually live." Once you've got those answers, the next step is mapping out the decision tree.
Right, so what does that decision tree look like? What are the key questions someone should be asking themselves as they start building one of these workflows?
Volume and category stability. Those two things narrow it down faster than anything else. If you know your classes, you have a fixed label set, and you're processing more than a few thousand images, you're almost certainly better off with a dedicated tool. Fine-tune a YOLO model, get your structured output as a typed endpoint, and don't think about it again. If your category set is shifting, if you're exploring, if someone hands you a new dataset and asks "what's in here," that's where Gemini or Grounding DINO earns its place.
Budget is the second filter. AWS Rekognition at a tenth of a cent per image is accessible for small projects. You can annotate ten thousand images for ten dollars. But if you're scaling to fifty thousand images a day consistently, that's fifty dollars daily, and at that point the math on running YOLO locally on a rented GPU instance starts to look attractive. The crossover point is somewhere around twenty to thirty thousand images per day depending on your GPU costs, and it shifts if you're already running infrastructure for other things.
The "local is always cheaper" instinct isn't wrong, it's just premature.
It's premature and it ignores the engineering cost. The pip install is easy. The fine-tuning pipeline, the model versioning, the monitoring for drift when your production images start looking different from your training data, that's where the real cost lives.
The drift monitoring point is underrated. You can have a perfectly tuned model at launch and then six months later the lighting in your warehouse changes, or a supplier redesigns their packaging, and your detection accuracy quietly degrades without anyone noticing until something downstream breaks.
That's the insidious thing about drift—it's usually gradual enough that no single inference looks obviously wrong. You don't get an error, you just get slowly worse annotations, and if no one is tracking precision and recall over time you might not catch it for weeks. Cloud APIs handle this invisibly on their end because the model gets updated, but that's also a risk: the model gets updated and your carefully tuned threshold is suddenly wrong in the other direction.
You're trading one monitoring problem for a different one.
There's no version of this that doesn't require some ongoing attention. For someone wanting to dig deeper, where are they going?
The ultralytics documentation is excellent for YOLO. The Roboflow blog has solid practical walkthroughs on annotation pipelines. And Hugging Face is the right starting point for Grounding DINO, RF-DETR, anything in the open-source transformer space. Start there before assuming you need a cloud API at all.
Yeah, and what's wild is how much this space is still evolving. Gemini's January update alone already shifted what multimodal models can do for bounding box detection. Where do you think that trajectory is headed in two years?
That's the open question. If multimodal models get to the point where structured output is as reliable as a typed API endpoint, and bounding box precision closes the gap with YOLO-style architectures, then the case for dedicated tools gets much narrower. You'd essentially have one model that handles the full detection-to-annotation pipeline without the routing logic.
Though I'd argue the speed gap is harder to close than the precision gap. Physics doesn't really care how smart your model is.
That's fair. A single-stage detection head running on purpose-built inference hardware is always going to have an advantage over a general-purpose transformer doing spatial reasoning. But the interesting implication isn't "which wins," it's what happens to annotation workflows when the tooling gets good enough that the two-stage pipeline collapses into one call. You'd be passing an image in and getting a fully annotated output back, schema and all.
Which would be remarkable for small teams and solo developers who currently have to stitch all of this together themselves.
That's Daniel's world, isn't it. The person building an open-source annotation tool who doesn't want to manage three different API contracts and a fine-tuning pipeline. The simpler that workflow gets, the more accessible serious computer vision becomes.
There's a version of that future where the hard parts of this conversation—the threshold tuning, the schema validation, the drift monitoring—just become someone else's problem by default. Whether that's a good thing probably depends on how much you enjoy debugging pipelines at midnight.
I know people who would say that's the whole job and they'd miss it.
I know people who would not miss it even slightly.
We'll be watching that one closely. Thanks to Hilbert Flumingtop for producing this episode, and to Modal for keeping the lights on and the GPUs warm. This has been My Weird Prompts. If you've got a moment, a review on Spotify helps other people find the show.
Until next time.
