Your backup philosophy is solid — if it's not automated, it's not happening. But here's the thing Daniel's running into right now. He's shooting casual tech videos on his phone, uploading raw to a YouTube channel called Daniel's Tech World, and he's fine using YouTube's compressed version as his backup file. The NAS isn't set up yet in the new apartment. So the question is, how do you build a pipeline that detects a new upload on a specific channel, downloads it automatically, pushes it to Wasabi, and optionally pulls it onto a NAS? And then, where do you actually run this thing?
The trigger detection is the hard part. The download-and-upload pipeline? That's straightforward. But knowing that a new video exists on a specific channel, in a way that's reliable and doesn't cost you anything — that's the bottleneck that determines the rest of the architecture. And Daniel's right to frame this as a worker problem, because fundamentally you're asking, what is the cheapest, most reliable always-listening thing that can fire off a download job?
Before we go further though — is this backup, or is this archival? Because those are different things. Backup implies you'd restore from it. Archival implies you just want it somewhere safe.
It's backup with a lower SLA. Daniel's explicit about this — these are second-priority videos, shot on a phone, no editing. He's not keeping the raw files. If YouTube's compression is acceptable as the canonical copy, then the goal is getting that compressed file off YouTube's servers and onto storage he controls. The automation is the insurance policy, not the quality guarantee.
Fair. So let's walk the pipeline. Step one, trigger detection. How do you know a new video went live on Daniel's Tech World?
YouTube has an RSS feed for every channel. The URL format is youtube dot com slash feeds slash videos dot xml, with a query parameter for the channel ID. This endpoint has been stable since twenty ten. It's undocumented — YouTube doesn't advertise it — but it works, and it's been working for fifteen years. Every time a channel publishes a video, that feed updates within a few minutes.
And this is free? No API key?
Completely free, no authentication, no quota. You just fetch the XML. The feed includes the video ID, title, publish date, and a link. You parse it, extract the video ID, compare it against a list of IDs you've already processed, and if it's new, you fire the download.
So the polling approach is just a cron job hitting that RSS feed every fifteen minutes. What's the catch?
The lag. The feed can be five to fifteen minutes behind the actual publish time. For a backup pipeline, that's nothing. Nobody needs their backup the millisecond the video goes live. The bigger catch is that if your cron job fails silently — script crashes, machine goes down, network blip — you miss the window. You need idempotency and a persistent record of what's been processed.
And the alternative to polling?
YouTube's Data API version three has a search dot list endpoint where you filter by channel ID. That's the official, documented way. But it costs a hundred quota units per request, and the free tier gives you ten thousand units a day. That's a hundred requests per day. If you poll every fifteen minutes, you burn through your entire quota in twenty-five hours. You'd need to poll much less frequently, or pay for more quota. The RSS feed has no limits.
So the RSS feed wins on cost and simplicity. What about webhooks? Daniel mentioned wanting to be notified rather than polling.
YouTube does not offer native webhooks for new uploads. Period. The closest thing was PubSubHubbub — now called WebSub — where you could subscribe to the RSS feed and get pushed updates. But most feed readers and providers dropped support in twenty twenty-three and twenty twenty-four because it was unreliable. The subscriptions would expire silently. You'd think you're getting notifications and you're not.
So the thing that sounds modern and event-driven is actually the flakiest option.
Polling the RSS feed is boring, but boring is what you want for backup infrastructure. It's deterministic. You can see when it last ran. You can test it. You can alert on it.
Alright, so detection is solved — cron job, RSS feed, fifteen-minute interval, compare against a SQLite database of processed video IDs. Now the download step. What does that actually look like?
yt-dlp. It's the active fork of youtube-dl, and as of mid twenty twenty-six it's handling YouTube's latest signature cipher changes. The key flags you want: dash f for format selection, dash dash output for naming convention, dash dash no-playlist so it doesn't try to grab an entire channel if you accidentally point it at a playlist URL, and dash dash limit-rate to avoid YouTube throttling your IP.
Give me the command.
yt-dlp dash f, and then in quotes, bestvideo, square bracket height less than or equal to ten eighty, close square bracket, plus bestaudio slash best, square bracket height less than or equal to ten eighty. Then dash dash output with a naming template — something like percent channel slash percent upload date underscore percent title dot percent ext. Dash dash no-playlist. Dash dash limit-rate five M. And then the video URL.
And that naming convention gives you a clean directory structure per channel with the date baked into the filename.
Right. And that matters when you're doing multi-channel backups with different policies. You can route based on the channel name in the path.
What breaks with yt-dlp?
YouTube changes their player code frequently. yt-dlp has to reverse-engineer the signature cipher each time. The maintainers are fast — usually updates within a day or two — but there are windows where downloads fail. You need retry logic. Exponential backoff. If a download fails, wait five minutes, try again. If it fails again, wait fifteen. After three failures, log it and move on.
And the upload step?
rclone or the AWS CLI to Wasabi. Wasabi is S3-compatible and their pricing is five ninety-nine per terabyte per month with no egress fees for the first three times your stored amount. For backup, that's ideal — you're uploading a lot, downloading rarely. rclone has a dash dash progress flag and a dash dash checksum flag that verifies the file landed intact. The command is basically rclone copy, the local file path, the Wasabi remote, and the bucket path.
And then the NAS pull?
If the NAS is on the same network as wherever this script runs, you can do an rsync or an rclone copy directly from the Wasabi bucket to the NAS mount point. If the NAS is remote, you pull over SSH. Synology has Cloud Sync with native S3 support — you point it at your Wasabi bucket and it syncs. But that's a separate step from the core pipeline. The core pipeline is detect, download, upload. The NAS pull can be its own scheduled job.
So we've established the detection and the pipeline. Now the question Daniel actually asked — where do you run this thing? Home server or serverless?
Let me kill the serverless idea right now. Cloudflare Workers — the free tier gives you a hundred thousand requests a day, which sounds great, but the CPU time limit is ten milliseconds per request. You can get that waived to thirty seconds on paid plans. Thirty seconds. A ten-minute video download takes minutes, not seconds. It's an I/O-bound, long-running task. Workers are designed for lightweight request-response, not for sitting on a socket waiting for bytes.
AWS Lambda?
Fifteen-minute timeout. One gigabyte of ephemeral storage. A thirty-minute four-K video download can easily exceed both. Google Cloud Functions is nine minutes. These platforms are not designed for video downloads. You're fighting the architecture the entire time.
So serverless is the wrong tool for this job.
It's the wrong tool. The right tool is a small always-on machine — either a home server or a five-dollar-a-month VPS. A DigitalOcean droplet or a Hetzner CX twenty-two gives you a full Linux environment, no timeout limits, persistent storage, and you can run whatever you want. For five bucks a month you get a machine that can handle this pipeline for dozens of channels.
And the home server argument?
If you already have a machine running twenty-four seven — a Raspberry Pi four, an old laptop, a NUC — the marginal cost is zero beyond electricity. The downside is reliability. Your home internet goes down, the backup doesn't run. Your power blips, the backup doesn't run. A VPS in a data center has five-nines uptime. For backup infrastructure, I'd argue the five dollars a month is worth it.
What about the orchestration layer? Daniel asked about an infrastructural management platform for running multiple workers.
Start simple. A single bash script that curls the RSS feed, parses it with a Python script, runs yt-dlp, runs rclone, and updates a SQLite database. That script runs from cron every fifteen minutes. That's it. That handles one to five channels without breaking a sweat.
But Daniel wants different backup behavior per channel. Different buckets, different format preferences, maybe different schedules.
Then you move from a single script to one config file per channel — YAML or JSON — and one cron job per channel. Each cron job points at its config, which specifies the channel ID, the target Wasabi bucket, the format flags, the NAS path. It's still simple. You don't need an orchestration platform until you have ten plus channels or complex failure handling requirements.
Alright, but let's say you do want a platform. What's out there?
Huginn is the one I'd point to. It's an open-source, self-hosted agent system written in Ruby. You deploy it via Docker. It has a built-in YouTube channel agent that polls the RSS feed for you — you give it a channel ID and it emits events when new videos appear. Then you chain that to a shell command agent that runs your yt-dlp and rclone script. The whole thing lives in a Docker Compose file.
And Huginn handles the idempotency?
It tracks which videos it's seen. Each event has a GUID based on the video ID. If the agent restarts, it picks up where it left off. It's not perfect — you still want your own processed-ID database as a safety net — but it handles the basics.
What about n8n?
n8n is more visual, node-based. It has YouTube trigger nodes. It's good if you want a drag-and-drop interface. But for this specific pipeline, Huginn's agent model maps more naturally — each channel is an agent, each download step is an agent, and they're loosely coupled. n8n tends toward monolithic workflows.
And the heavy hitters — Airflow, Prefect?
Overkill. Airflow is for data engineering pipelines with complex dependencies and backfilling requirements. You don't need a Directed Acyclic Graph to download YouTube videos. Prefect is lighter but still designed for data workflows, not simple cron replacement. You'd spend more time configuring the orchestrator than building the actual pipeline.
So the recommendation is: start with a cron job and a Python script. If you outgrow that, Huginn in Docker. Don't touch Airflow.
Don't touch Airflow for this. Save it for when you're orchestrating a hundred data sources with retry windows and SLAs. This is a backup script.
Let's talk about resilience. What happens when things fail?
Three failure modes. One, the RSS feed doesn't respond. Your script should have a timeout — thirty seconds — and if it can't reach the feed, it logs and exits. Next cron run picks it up. Two, yt-dlp fails because YouTube changed something. Retry with exponential backoff — five minutes, fifteen minutes, then give up and alert. Three, the Wasabi upload fails. rclone has retry built in with the dash dash retries flag. Set it to three.
And how do you know if the whole thing has been silently failing for a week?
Health checks. Cronitor or healthchecks dot io. Your script pings a URL at the end of each successful run. If that URL doesn't get pinged for thirty minutes, you get an alert. It's a dead man's switch for cron jobs. Free tier on healthchecks dot io gives you enough checks for a handful of pipelines.
That's the part most people skip. They build the automation and then discover it broke three months ago.
Every automation eventually breaks. The question is whether you find out from an alert or from data loss.
Let's get concrete about the multi-channel setup. Daniel has his main YouTube channel with edited content and this new Daniel's Tech World channel with raw phone videos. Different backup policies.
So two config files. The main channel config might specify higher quality format flags, a different Wasabi bucket, and a NAS sync step. The tech channel config uses the compressed version, a separate bucket, and maybe no NAS sync at all — just Wasabi. Two cron jobs, two configs, same script.
And if he adds a third channel?
Copy the config, change the channel ID and bucket, add a cron job. The script doesn't change.
What about the Docker Compose approach? You mentioned a scheduled container.
You build a single Docker image with yt-dlp, rclone, Python, and your script. The Docker Compose file defines a service for that image, mounting your config directory and a data directory for the SQLite database. Then you either run cron inside the container, or you use a separate cron container that docker execs into the yt-dlp container on a schedule.
The cron sidecar pattern.
Yeah. It's clean. The cron container just triggers, the worker container does the work. If you're using Huginn, it's even simpler — Huginn is the cron, and the shell command agent runs the script inside the same Docker network.
I want to circle back to something. The RSS feed approach — you said it's been stable since twenty ten. But it's undocumented. What happens if YouTube kills it?
Then you fall back to the Data API. The quota costs money if you poll frequently, but for a backup pipeline, you could poll every hour instead of every fifteen minutes and stay within the free tier. The trade-off is you might wait up to an hour for a backup to trigger. For second-priority videos, that's acceptable.
So the RSS feed is a bet on YouTube not breaking a fifteen-year-old endpoint. It's a reasonable bet, but you should have the API fallback scripted and ready.
Write the script to try the RSS feed first, and if it fails three consecutive times, flip to the API. That's a weekend project, not a rewrite.
What about the legal angle? yt-dlp has faced DMCA takedowns. The RIAA and others have gone after these tools. Are we recommending something that stops working next month?
yt-dlp exists in a legal gray area. The tool itself is just a downloader — it doesn't circumvent DRM, it doesn't decrypt encrypted streams. It downloads what YouTube serves to a browser. The DMCA takedowns have targeted the repository hosting, not the tool's legality per se. The code keeps moving — new forks, new mirrors. As of July twenty twenty-six, it's actively maintained and working.
But downloading your own content from YouTube — is that even against the terms of service?
YouTube's terms of service prohibit downloading content unless a download button is provided by YouTube itself. But here's the thing — Google Takeout lets you export your own YouTube videos. It's an official, supported method. The catch is it's not real-time. You request an export, you get a zip file hours or days later. For backup, that's not automation.
So the official method exists but doesn't solve the automation problem.
Right. And that's the tension. You own your content. You uploaded it. You should be able to retrieve it programmatically. But YouTube's API doesn't support that, and their terms prohibit the workaround. Practically speaking, individuals downloading their own videos for backup are not the target of enforcement. The legal risk is near zero for personal use.
That's a reasonable assessment. Alright, let's steelman the objection. Someone listening says, this is all too complicated. Just use Google Takeout once a month and call it a day. What do you say?
I'd say that's a valid approach if you're comfortable with a thirty-day backup window. If you upload a video on day one and your channel gets terminated or hacked on day twenty-nine, you lose that video. The automated pipeline gives you a fifteen-minute backup window. The difference is whether you're protecting against hardware failure or platform risk.
And platform risk is real. YouTube channels get terminated by mistake. Accounts get hacked. Google's support is... let's say, not known for rapid human intervention.
The phrase you're looking for is "automated enforcement with limited appeal paths."
That's the diplomatic version, yes.
Look, the pipeline we're describing isn't complicated. It's a cron job, a Python script, yt-dlp, and rclone. If you can set up a VPS, you can set this up in an afternoon. The complexity isn't in the tools — it's in thinking through the failure pattern and building the resilience in from the start.
That's where most people stop. They get the happy path working and declare victory.
The happy path is maybe twenty percent of the work. The other eighty percent is what happens when the RSS feed is slow, yt-dlp breaks, the Wasabi upload times out, the disk fills up, the cron daemon silently dies. That's the actual engineering.
Let's give the concrete recommendation. If Daniel wants to build this tomorrow, what's the path of least resistance?
Step one, spin up a five-dollar Hetzner or DigitalOcean droplet. Step two, install yt-dlp and rclone. Step three, write a Python script that fetches the RSS feed, parses it with feedparser, checks a SQLite database for processed video IDs, downloads new videos with yt-dlp, uploads to Wasabi with rclone, and updates the database. Step four, add it to crontab for every fifteen minutes. Step five, set up a healthchecks dot io ping at the end of the script. That's it. That's the whole thing.
And for multi-channel?
One config file per channel in a configs directory. The script loops over configs. Or, if you want isolation, one cron job per channel pointing at the same script with a different config flag. Both work.
What about the NAS pull?
Separate cron job on the NAS itself. rclone sync from the Wasabi bucket to the local NAS path, running every hour or every day depending on how urgent the local copy is. Don't couple it to the upload pipeline — let them be independent.
That decoupling is smart. If the NAS is off or unreachable, the upload to Wasabi still succeeds.
Each step should be able to fail independently without taking down the rest of the pipeline.
One more thing — what about the notification that a backup completed? Daniel didn't ask for it, but it seems like the kind of thing you'd want.
The health check ping tells you the script ran. For per-video notifications, you could add a simple webhook — a Discord message, a Telegram bot, an email. But honestly, for a backup pipeline, silence is success. You only want to hear about it when it fails.
The no-news-is-good-news model.
The only model that scales. If you get a notification for every successful backup, you'll train yourself to ignore them within a week.
Alright. Let's land this. The architecture is RSS polling on a cheap VPS, yt-dlp for download, rclone to Wasabi, optional NAS sync as a separate job. Start with a cron job and a script. Add Huginn if you outgrow it. Serverless is the wrong tool. The RSS feed is the right trigger. And build in retry logic and health checks from day one.
Daniel, if you build this, share the config. The community benefits from battle-tested scripts, and the devil really is in the details — the exact yt-dlp flags, the rclone retry settings, the SQLite schema. Those are the things that take an afternoon of trial and error to get right.
The open question I'm left with is sustainability. yt-dlp is a cat-and-mouse game with YouTube's engineering team. Google could decide tomorrow to break the RSS feed or change the signature cipher in a way that takes weeks to reverse. The pipeline works today, but it requires maintenance. The question is whether that maintenance burden is lower than the alternative — manually downloading from Google Takeout every month and accepting the backup gap.
I think for the kind of videos Daniel's describing — casual, shot on a phone, second priority — the maintenance burden is worth it. You set it up once, you check on it when the health check pings you that something's wrong, and you fix it. That's maybe a few hours a year. Versus remembering to do manual backups, which humans are terrible at.
That's the core of Daniel's philosophy. If it's not automated, it's not happening. The automation doesn't have to be perfect. It just has to be better than the human it's replacing.
The human it's replacing is Future Daniel, who will definitely forget to do the backup.
Future Daniel is the least reliable person in this equation.
Present Daniel knows that. That's why he's asking the question.
Thanks to our producer Hilbert Flumingtop for making this possible.
This has been My Weird Prompts. If you build this pipeline, email the show at show at my weird prompts dot com — we'd love to see your configs and hear what broke in production.
We'll be back soon.
