Alright, so Daniel sent us a technical one this week. Here's what he wrote: he wants us to dig into building AI pipelines for real-time breaking news coverage, using the Iran-Israel war as a case study. His core question is this — even day-trailing summaries go stale fast when a conflict is evolving by the hour. So what are the actual tools and approaches for ingesting up-to-the-minute information into an AI system? He specifically wants us to cover Perplexity Sonar, Groq, direct RSS feed ingestion, and news APIs, and to get into the real subtleties of each approach, not just the surface-level pitch. Good prompt. Let's get into it.
This is one of those problems that sounds straightforward until you actually try to solve it, and then it reveals itself to be genuinely hard in interesting ways. Because the naive assumption is — search exists, right? Just have your AI search the internet. Done. But breaking news stress-tests every assumption in that model.
And the Iran-Israel war is almost a perfect adversarial example for this. The situation has been changing multiple times per day. Just this morning, a US naval blockade of Iranian ports went into effect. The USS Frank E. Petersen Jr. and USS Michael Murphy are conducting mine-clearing operations. Trump confirmed it. That's a development from this morning — any system with even a six-hour-old index could be operating on completely wrong assumptions about the state of the conflict.
And it's not just the blockade. The Islamabad ceasefire talks collapsed over the weekend. The Iranian foreign minister posted on X that they were, quote, inches away from an Islamabad memorandum of understanding before hitting what he called maximalism and shifting goalposts. The IDF chief of staff has now instructed forces to prepare for renewed hostilities. Hezbollah announced it won't abide by any Lebanon-Israel agreements. This is a situation where being six hours behind isn't just inconvenient — it's the difference between your system believing there's an active peace process and your system knowing there isn't one.
So let's establish the actual problem space before we get into tooling. Because I think there are at least three distinct failure modes for AI systems covering something like this, and they're worth separating out.
Go for it.
First, there's the training cutoff problem. The conflict started February twenty-eighth. That's after most major LLM training cutoffs. So the base model simply has no knowledge of this war existing. Second, there's the index lag problem — even if you're using retrieval-augmented generation, your index might be hours old, which for this story means it's wrong. And third — and this one is fascinating — there's what I'd call the blackout problem. Iran has been under a national internet blackout for thirty-eight days now. That's the longest national internet shutdown on record, surpassing Sudan's thirty-seven day blackout in twenty-nineteen. So the most important information — what's actually happening inside Iran — is precisely what no search pipeline can retrieve.
That third one is the one that keeps me up at night as a systems design problem. The training cutoff issue and the index lag issue are both solvable with engineering. The blackout problem is a data availability problem, not a retrieval problem. No API in the world can give you ground truth from inside a country that has severed its own internet connection. What you get instead is Iranian state media output, which is regime-curated, satellite imagery analysis, diaspora sources, and leaked communications. Your pipeline needs to be epistemically aware of this — it needs to know what it can't know.
Which is a remarkably hard thing to build. Okay, so with that framing in place — let's talk tools. And by the way, today's script is courtesy of Claude Sonnet four point six, which feels appropriate given we're talking about AI systems processing information in real time.
Ha. Meta. Alright, let's start with Perplexity Sonar, because it's probably the most discussed option in this space right now, and also the most misunderstood.
What's the misunderstanding?
Most people treat Perplexity as a single thing — you send it a query, it searches the web, you get an answer. But they actually have four distinct APIs that are architecturally quite different. The Sonar API is what most people think of — it returns an AI-synthesized answer plus citations from the live web. The Search API returns raw ranked web results with no LLM synthesis. The Agent API lets you use third-party models like Claude or GPT-4 with Perplexity's search tools. And then there's an Embeddings API for semantic search and RAG pipelines. For a breaking news application, the choice between Sonar and Search is actually a fundamental architectural decision.
Walk me through that decision. Because on the surface, getting a synthesized answer sounds better — less work for you.
The synthesis is convenient but it introduces a layer of abstraction you can't fully inspect. If Perplexity's Sonar API tells you "ceasefire talks are ongoing," you don't know exactly which sources it weighted, how it resolved conflicting reports, or whether its index actually has the last two hours of coverage. For a news pipeline where accuracy is paramount and where you might be making downstream decisions based on the output, raw results plus your own synthesis is often safer. You control the synthesis step, you can see exactly what sources you're working with, and you can implement your own conflict resolution logic when AP and Reuters disagree.
What does the Search API actually give you in terms of configurability for something like this?
Quite a bit. You can set max results up to twenty per search. You can filter to specific domains — so you could restrict retrieval to apnews.com, reuters.com, timesofisrael.com, bbc.com. That's essentially building a curated editorial desk into your retrieval layer. You can filter by language, by country for regional results. And here's a feature that's underappreciated for breaking news — you can run up to five queries in a single request. So for a complex story like Iran-Israel, you could simultaneously query for the blockade status, the ceasefire talks, IDF mobilization, and Hezbollah's position in one API call. That matters when you're trying to build a complete picture quickly.
What's the pricing look like?
For the Search API, it's five dollars per thousand requests at low context, scaling to twelve dollars at high context. So a typical query runs you somewhere between half a cent and just over a cent. If you're running a pipeline that queries every five minutes around the clock, you're looking at roughly one dollar forty to six dollars per day depending on your context settings. That's genuinely affordable for most applications.
But here's my concern with Perplexity for this use case specifically. You mentioned index freshness. And from what I can tell, Perplexity doesn't actually publish their crawl frequency. So for the US naval blockade announcement this morning — you genuinely don't know if Perplexity's index has it yet.
That's the critical subtlety, and it's a real gap in the tooling landscape. Perplexity's index freshness is opaque. They don't publish an SLA for how quickly new content gets indexed. For a story that broke this morning, you might be getting it, you might not. There's no way to know from the outside. And for a breaking news application, that opacity is a significant architectural risk.
So let's talk about Groq, because the pitch there is different. The pitch is speed.
Speed is the headline, but the architecture is interesting. Groq built custom chips they call Language Processing Units — LPUs — specifically optimized for LLM inference. The result is inference speeds that are genuinely in a different category. Their fastest model right now is running at a thousand tokens per second. To put that in context, standard GPU inference on a comparable model runs somewhere between fifty and a hundred tokens per second. So Groq is ten to twenty times faster.
What does that actually mean for a news pipeline in practice?
It means you can process a two-thousand-token news summary in about two seconds. Which enables architectures that would be too slow on standard inference. You could build a triage layer that runs every few minutes, scoring new articles for relevance in near-real-time, and the latency doesn't become a bottleneck. At a thousand tokens per second, the inference is no longer the slow part of your pipeline.
And Groq has a built-in web search capability too?
They do, through what they call Compound systems — Compound and Compound-mini. The web search is actually powered by Tavily under the hood, which is an important detail. Groq's speed is their own, but the freshness of the search results depends on Tavily's crawl frequency, not Groq's infrastructure. So you get Groq's speed with Tavily's index.
Which has the same opacity problem as Perplexity.
Exactly the same problem. Neither Perplexity nor Tavily publishes exact crawl frequency. This is actually a systemic gap in the current landscape — two of the most popular options for AI-powered web search both have opaque freshness guarantees. For most applications that's fine. For breaking news where you're trying to track something that changed this morning, it's a real limitation.
What I find interesting about Groq's Compound system is that it exposes the reasoning trace. You can actually see what search queries the model ran internally and what it found.
That's a significant debugging advantage. When your pipeline misses a breaking development, you can audit exactly why — what queries it ran, what results came back, why it didn't surface the relevant information. For a production news pipeline, that observability is worth a lot. It's the difference between knowing your system failed and knowing why it failed.
And the pricing on Groq is remarkably cheap. Their fastest model is seven and a half cents per million input tokens, thirty cents per million output tokens. For a news triage pipeline, you're talking fractions of a cent per article.
The combination of that speed and that price point is why Groq makes sense as the triage layer in a multi-tier architecture. You don't need the highest-quality model to answer "is this article about the Iran blockade relevant to my monitoring topic?" You need a fast, cheap model that gets that right ninety-five percent of the time. Groq's models at those speeds and prices are well-suited for that role.
Okay, so we've got two search-API-based approaches that are fast and convenient but have opaque index freshness. What's the alternative for genuine low-latency coverage?
Direct RSS ingestion. And I know that sounds almost anachronistically simple given everything we've been discussing, but RSS is genuinely the lowest-latency option available. Articles appear in RSS feeds within minutes of publication. There are no API costs beyond your own infrastructure. And you're getting data directly from authoritative sources — AP, Reuters, BBC, Times of Israel — rather than through an intermediary's index that you don't control.
So why doesn't everyone just do RSS?
Because it requires significantly more engineering. The raw RSS feed gives you headlines and summaries, not full article text. So you need a second step to fetch the full content from each URL, which adds latency and may hit paywalls. Deduplication is a serious problem — the same story will appear across dozens of feeds, and without deduplication your LLM context gets flooded with near-identical content that eats your context window and degrades synthesis quality. And published timestamps in RSS are notoriously unreliable — many feeds have incorrect or missing timestamps, so you have to track what you've already processed by URL rather than by time.
What's the right polling frequency?
That's a genuine engineering judgment call. Poll too infrequently — say every hour — and you miss breaking developments. The US blockade announcement this morning would have sat in your feed for up to an hour before your pipeline saw it. Poll too frequently — every thirty seconds — and you risk being rate-limited or IP-blocked by the source. Two to three minutes is probably the sweet spot for a breaking news application. But at that frequency, a fifteen-minute polling cycle across twenty RSS feeds might yield fifty to a hundred new articles. You need a fast triage layer before that hits your main model — which is where Groq at a thousand tokens per second becomes useful again.
So these things are complementary, not competing.
That's the key insight. No single tool wins on all three dimensions of speed, freshness, and synthesis quality. The right architecture combines them. RSS gives you freshness measured in minutes. Groq gives you the speed to triage that firehose of articles cheaply and quickly. Perplexity gives you synthesis quality for on-demand queries. You're building a pipeline, not choosing a single tool.
Let's talk about the dedicated news APIs, because there's a whole category of tools here that I think gets underappreciated. GDELT in particular.
GDELT is fascinating and also genuinely difficult to use. The Global Database of Events, Language, and Tone has been running since nineteen seventy-nine in terms of historical coverage, and the modern version updates every fifteen minutes. That fifteen-minute update cycle is the only option in this space with a published SLA for freshness. Perplexity doesn't publish theirs, Tavily doesn't publish theirs, but GDELT explicitly commits to fifteen-minute updates. And it's free.
Free is a remarkable price point for something that updates every fifteen minutes and covers a hundred-plus countries.
The catch is the learning curve. GDELT's data model is complex — it uses CAMEO event codes for categorizing geopolitical events, it has its own query syntax, the documentation is scattered across multiple sites, and making sense of the raw output requires real data processing expertise. But for a sophisticated pipeline, you can query GDELT for all articles mentioning Iran and the Strait of Hormuz published in the last hour, and you'll get back a structured response with geolocation data, sentiment scores, and related entity information. That's powerful for building an event graph of a complex conflict.
What about on the commercial side? Because there's been some significant movement in this space recently.
Bloomberg launched something quite interesting in early March — customizable real-time news feeds designed specifically for systematic workflows. The framing is different from everything else we've discussed. Instead of querying a search index, you subscribe to a feed for specific entities. You could subscribe to news about the Strait of Hormuz as a topic, or the IRGC, or specific ships like the USS Frank E. Petersen Jr. You get a structured feed of everything relevant to that entity in real time, with sentiment scores and what Bloomberg calls Market Moving News indicators — an estimated probability that a given story will move markets in the short term.
That's a fundamentally different mental model. It's less like a search engine and more like a financial data feed.
Which is exactly where it comes from. Bloomberg built this for systematic trading workflows where you need to know the moment something happens to a specific company or security. But the same model applies to geopolitical monitoring. If you've subscribed to a feed for "Strait of Hormuz," you get the blockade announcement the moment Bloomberg's reporters file it. The latency is as close to real-time as the reporting itself allows.
The pricing is presumably not GDELT-like.
Enterprise pricing. No public numbers. It's a different category of customer. But it points to where the architecture is going — entity-centric, structured, subscription-based, rather than query-centric and search-based.
There's another development in this space that I want to flag, which is NewsAPI.ai launching an MCP server in April. Because Model Context Protocol integration changes the retrieval paradigm in an interesting way.
It does. The traditional RAG paradigm is: user asks a question, system searches for relevant documents, documents get stuffed into context, LLM synthesizes an answer. MCP enables something different — the LLM can directly invoke a structured news query as a tool call, get back enriched and deduplicated results with entity tagging, and incorporate that into its reasoning. Instead of asking a search engine "what's happening with the Iran blockade?" and getting back web pages that need parsing, you're asking a structured news database and getting back organized, entity-tagged articles. For a breaking news pipeline, that's architecturally cleaner because the deduplication and entity extraction happens in the data layer rather than having to be implemented in your pipeline.
Let me push on something here, because I think there's a tension in all of this that we haven't fully addressed. You've laid out this multi-tier architecture — RSS for freshness, Groq for triage, Perplexity or NewsAPI for synthesis. But building and maintaining that is non-trivial engineering. What's the realistic build-versus-buy calculus for someone who actually wants to do this?
The cost structure is pretty clear when you lay it out. A Perplexity Search API pipeline querying every five minutes around the clock costs somewhere between one and a half and six dollars per day. GDELT is free but requires maybe a week of engineering to get a working pipeline. NewsAPI.ai starts at ninety dollars a month and gives you full article text, entity recognition, and event clustering out of the box. Bloomberg is enterprise pricing but gives you the most structured real-time data available. The question is really what your engineering capacity is and what your accuracy requirements are.
And for the Iran-Israel case specifically, what does the accuracy requirement actually demand?
For a use case where you're briefing someone on the current state of the conflict — someone making decisions based on that briefing — I'd argue you need the multi-tier approach. The blockade announcement this morning is the kind of development that changes everything about the strategic picture. A system that doesn't have that information isn't giving you a briefing on the current conflict — it's giving you a briefing on yesterday's conflict. And in a situation where ceasefire talks collapsed over the weekend, where the IDF is preparing for renewed hostilities, where Hezbollah has announced it won't honor any Lebanon agreements — yesterday's briefing is actively misleading.
Let me bring back the blackout problem, because I think it deserves more attention than it usually gets in these discussions. Iran has been under a national internet shutdown for thirty-eight days. That's not just a data gap — it's a systematic bias in everything any pipeline can retrieve about the conflict.
Right. Every search API, every RSS feed, every news database — they're all drawing from the same limited pool of information about conditions inside Iran. You're getting Iranian state media, which is what the regime wants you to see. You're getting reporting from foreign correspondents who are either outside the country or operating under severe restrictions. You're getting satellite imagery analysis. What you're not getting is independent reporting from inside Iran about civilian conditions, about actual military movements, about what the population is experiencing. Iran's internet blackout means the information asymmetry in this conflict is enormous.
And a well-designed pipeline should surface that uncertainty rather than paper over it.
That's the meta-problem. Building a system that knows what it doesn't know. When your pipeline retrieves information about conditions inside Iran, it should be flagging that the source pool is severely limited due to the internet blackout, that Iranian state media is the primary available source for internal conditions, and that this represents a significant uncertainty about ground truth. Most pipelines don't do this — they just retrieve what's available and synthesize it without flagging the epistemological limitations.
There's a parallel here to the index freshness opacity problem. In both cases, the failure mode is a system that presents information with more confidence than is warranted. The index might be stale, but the system doesn't tell you that. The source pool is severely limited by a blackout, but the system doesn't tell you that. You get an answer that sounds authoritative but carries hidden uncertainty.
And for breaking news specifically, that hidden uncertainty can be more dangerous than no answer at all. If your system confidently tells you that ceasefire talks are ongoing in Islamabad when they actually collapsed two days ago, that's worse than saying "I'm uncertain about the current status of the talks." The Islamabad talks collapse is a great example — Iranian FM Araghchi's post on X about being inches away from an agreement before hitting what he called shifting goalposts and a blockade — that's a significant diplomatic development that a stale index would completely miss.
Let's talk about domain filtering as a practical tool, because I think this is one of the most underappreciated features in both the Perplexity Search API and Groq's Compound system.
Domain filtering is essentially building editorial judgment into your retrieval layer. You can allowlist specific domains — AP, Reuters, BBC, Times of Israel, Al Jazeera — and your pipeline will only retrieve from those sources. The practical effect is that you're not surfacing random blogs or low-quality aggregators when you query about the Iran blockade. You're getting primary reporting from organizations with actual reporters on the ground or at least with editorial standards.
The Perplexity Search API lets you filter up to twenty domains. Groq's Compound system supports wildcards — you could include all dot gov and dot mil domains if you're building a government-facing application.
And this becomes a form of automated source credibility management. Instead of having to evaluate the credibility of each retrieved source at synthesis time, you've pre-selected your trusted source pool at the retrieval layer. For a breaking news application where you're going to be synthesizing dozens of articles, that's a significant quality improvement for relatively little engineering effort.
One thing I want to flag about the multi-query feature in Perplexity's Search API — up to five queries per request — is that for a complex conflict like Iran-Israel, the ability to simultaneously query multiple dimensions of the story is genuinely valuable. You're not just asking "what's happening in Iran?" You might be asking about the blockade status, the IDF mobilization, the Hezbollah position, the economic impacts on the Strait of Hormuz, and the diplomatic situation simultaneously. Getting all five in one API call rather than five sequential calls meaningfully reduces your pipeline latency.
And the economic impacts are worth mentioning because the secondary effects of this conflict are cascading globally in ways that a single-query approach might miss. The Strait of Hormuz closure has created energy market disruptions. There are fluoride shortages hitting US water utilities because a significant portion of the fluoride supply chain runs through the region. China has been gaining clean tech advantages as Western energy markets scramble. A pipeline that's only asking about the military situation is missing half the story.
Alright, let's try to give people something practical to take away here. If you're building a real-time news ingestion pipeline for breaking news coverage today, what does the architecture actually look like?
I'd think about it in three tiers. Tier one is your real-time triage layer. You're polling ten to twenty curated RSS feeds every two to three minutes. When new articles come in, you run them through a fast Groq model — the GPT OSS twenty billion parameter model at a thousand tokens per second — for relevance scoring. Is this article about the topics I'm monitoring? Is it from a primary source or an aggregator? Flag the high-relevance articles for deeper processing. This whole tier costs almost nothing and gives you sub-five-minute latency on new developments.
Tier two is enrichment?
Right. For the flagged articles, you fetch full text from the URLs. You run them through NewsAPI.ai or GDELT for entity extraction, sentiment analysis, and related article clustering. You're building a structured event graph — who are the actors, what happened, where, when, and what's the assessed significance. This is where you also flag the epistemic limitations — if the primary sources for a development are Iranian state media, that gets noted in the event graph.
And tier three is on-demand synthesis.
When a user query comes in — "what's the current status of the Iran blockade?" — you use Perplexity Sonar Pro or Groq Compound for synthesis, but you inject the tier one and tier two context into the system prompt. So the synthesis model has access to the most recent articles your triage layer flagged, enriched with entity and sentiment data, before it even starts searching the web. That means even if Perplexity's own index is a few hours behind, your injected context has the information from your RSS polling from twenty minutes ago.
The injected context compensates for the index lag.
That's the key architectural insight. You're not relying on any single tool's freshness guarantee. You're using your own RSS polling for freshness, your own enrichment layer for structure, and the synthesis APIs for language generation and any additional retrieval they can contribute. The weaknesses of each tool get compensated by the strengths of the others.
What about cost at scale? If you're running this continuously for a major breaking story?
For a serious operation monitoring a major conflict continuously, you're looking at — RSS polling is essentially free beyond infrastructure. GDELT is free. Groq triage at those token prices is probably two to five dollars a day even at high volume. Perplexity Search API for enrichment queries, maybe three to eight dollars a day. NewsAPI.ai at ninety dollars a month adds three dollars a day. So you're in the range of ten to twenty dollars a day for a serious continuous monitoring pipeline. That's genuinely accessible for most organizations that would have a legitimate need for this capability.
Compare that to what a human news monitoring operation costs, and the economics are striking.
Dramatically different order of magnitude. And the pipeline doesn't sleep, doesn't miss the three AM development, doesn't have to read through fifty duplicate articles to find the one that has the new information. The combination of speed, coverage, and cost is genuinely transformative for news intelligence operations.
I want to come back to something you said earlier about epistemic awareness, because I think it's the hardest part of this problem and the part that gets the least attention. You can build the fastest, freshest, best-integrated pipeline in the world, and if it doesn't know what it doesn't know, it's going to produce confident wrong answers at exactly the moments when you need accurate uncertain answers.
The Iran internet blackout is the clearest example, but it's a specific instance of a general problem. Any breaking news situation has information that's unavailable — because it hasn't been reported yet, because it's behind a paywall, because it's in a language your pipeline doesn't handle well, because the source is being actively suppressed. A well-designed pipeline needs to model its own coverage gaps. When it synthesizes an answer, it should be able to say — here's what I know from AP and Reuters, here's what I know from Israeli sources, here's the significant gap in my knowledge about conditions inside Iran due to the internet blackout, and here's my confidence level given those gaps.
That's a much harder engineering problem than the retrieval problem.
It is. The retrieval problem is largely solved — you can get fresh, high-quality information from authoritative sources with reasonable latency. The epistemic modeling problem — having the system reason about the quality and completeness of its own knowledge — is an active research area. For production pipelines today, the practical approach is to build explicit uncertainty flags into your data model. Tag every piece of information with its source, its source's credibility, its geographic provenance, and any known limitations on the source pool. Then surface those flags when synthesizing.
Alright. I think we've covered a lot of ground here. What are the two or three things you'd want someone to walk away with?
First: there is no single tool that wins on freshness, speed, synthesis quality, and cost simultaneously. The right approach for serious breaking news coverage is a multi-tier pipeline that uses different tools for what each does best. RSS for freshness, fast cheap inference for triage, structured news APIs for enrichment, and synthesis APIs for language generation.
Second?
Index freshness opacity is a real and underappreciated problem. Neither Perplexity nor Tavily publishes their crawl frequency SLA. GDELT is the only major option with a published fifteen-minute update commitment. For applications where you need to know how fresh your data actually is, that matters enormously. Build your own RSS polling layer if freshness guarantees matter to your use case.
And third?
Design for epistemic uncertainty from the start. The hardest part of building a real-time news pipeline isn't the retrieval — it's building a system that accurately represents the limits of its own knowledge. Iran's thirty-eight-day internet blackout is a perfect illustration of why this matters. The information that's most important is sometimes the information that's least available, and a system that doesn't surface that gap is actively misleading.
The blockade that started this morning, the collapsed ceasefire talks, the IDF mobilization, Hezbollah's announcement — all of that happened in the last seventy-two hours. Any system that doesn't have a real-time ingestion pipeline is operating on a fundamentally different understanding of this conflict than the one that actually exists. That's the stakes.
And it's only going to get more important as more consequential decisions get made with AI assistance. The gap between a well-designed real-time pipeline and a stale one isn't a technical footnote — it's the difference between situational awareness and confident ignorance.
Thanks as always to our producer Hilbert Flumingtop for putting this together. Big thanks to Modal for providing the GPU credits that keep this show running. This has been My Weird Prompts. If you want to find us, search for My Weird Prompts on Telegram to get notified when new episodes drop. Take care.
Poppleberry out.
