So Daniel writes in with a server rescue story that I have to say, reading it, I felt a kind of slow-building dread. It starts with what he thought was a failed power distribution unit, and then it cascaded. One component after another, each diagnosis leading to the next failure. By the time the dust settled, he had replaced literally every single part in the machine. Not one original component remains. He calls it his Ship of Theseus server, which is accurate and also a little haunting.
That is a rough repair arc. I've been there. You fix one thing, that exposes the next thing, and before you know it you're three weekends in and the only original part left is a case screw you're emotionally attached to.
And then he got to the storage. Four SSDs, about seven years old, one had already failed, the other three he judged not worth keeping. All four went in the bin. He replaced them with a single two terabyte NVMe drive. Fast, big improvement, but now he's right back where he started. One drive, no redundancy. He says it was a money decision, and I respect that. A proper RAID array with enough usable space plus redundancy was priced beyond what he could spend, so he bought capacity and accepted the risk.
That is a completely rational choice. RAID is not a backup, and sometimes the budget just says buy the big drive and cross your fingers. But here's where it gets interesting. He's not asking us to build him a RAID array. He wants something cruder, and in his words, more useful.
Right. He's got an NVMe boot and data drive, but the motherboard has several unused SATA ports. He knows mixing drive types and interfaces in a single array is generally discouraged, and he doesn't want a software-defined array anyway. What he wants is to buy a two terabyte SATA SSD, keep it permanently installed, and have some periodic background process maintain it as a bootable standby copy of the whole system. Then if the NVMe drive dies catastrophically, his recovery is physical. Pull the dead drive, or just change the boot order, and come up on the standby. Recovery by hand, not by array.
I love this. This is the kind of practical thinking that comes from someone who's actually been burned. He's not trying to build a high-availability cluster in his home office. He just wants to know that when the drive dies at eleven PM on a Tuesday, he can be back up in five minutes instead of spending the next three days reinstalling everything.
His actual questions are: does this pattern have a real name and established tooling behind it, or would he be inventing a fragile thing from scratch? What is it actually called, what are people using to do it properly, and how would he set it up on a Linux box so the standby is genuinely bootable and not just a pile of copied files that fails the one time he needs it?
Those are excellent questions. And the answer is yes, this pattern has a name, it has mature tooling, and we can absolutely build it. But first, Corn, I have to say, the Ship of Theseus detail is what got me. He replaced every single part. At what point does it stop being the server that failed and start being a completely new server that just happens to occupy the same physical space?
I think the answer is when you've replaced the power supply, the motherboard, the CPU, the RAM, and all the storage, you are no longer repairing a server. You are performing a server exorcism. The old server is gone. You have built a new one inside the ghost of the old one.
And now he wants to make sure the ghost doesn't come back. Let's talk about what this pattern actually is.
Before we do, I want to acknowledge something. He said he probably should have let the server die. That is a deeply relatable sentence. There is a point in every cascading hardware failure where you look at the pile of parts and the receipts and think, I could have bought a new machine for this. But you're in too deep, and you keep going. I respect the honesty.
And now he's making a smart move by thinking about recovery before the next failure happens. So let's get into it. What Daniel is describing has a name, and the name is bootable mirror, or sometimes hot spare by rsync, or more formally, a periodically synchronized standby disk. It's not RAID, it's not a clone in the dd sense, and it's not a backup in the traditional backup-and-restore sense. It's a running spare that lives inside the same machine.
And the key phrase there is periodically synchronized. This is not real-time replication. It's a snapshot that's maybe a few hours old, maybe a day old. You accept that you might lose the most recent changes, but you gain simplicity and you don't pay the performance penalty of synchronous mirroring.
The established tooling for this on Linux is rsync, combined with some careful handling of bootloader configuration, fstab entries, and a few other details we'll walk through. There are also purpose-built tools for this, and I want to name a few. There's a project called rsnapshot that wraps rsync with snapshot management. There's borgbackup, which is more of a deduplicating backup tool but can serve a similar purpose. And there's a lesser-known but very relevant tool called systemback, which was originally an Ubuntu project for creating live system images and restore points. But for what Daniel wants, which is a straightforward, bootable, identical copy on a second internal drive, I think the right answer is a well-configured rsync script plus some boot-time preparation.
So he's not inventing a fragile thing from scratch. This is a known pattern with known tooling. That's the good news. What's the bad news?
The bad news is that making a drive bootable is not just about copying files. There are several things that can go wrong, and if you miss any of them, the standby drive won't boot when you need it, and you'll find that out at the worst possible moment. We need to handle partition layout, filesystem UUIDs, the bootloader, the initramfs, and the fstab. Miss any one of those and you've got a pile of copied files that fails the one time you need it, exactly as Daniel feared.
So let's walk through this properly. I want to start with the honest assessment. Is this a good project? Should he just buy an external USB drive and run a backup tool like every normal person?
That's a fair question. An external USB drive with something like Veeam Agent for Linux, or even just Deja Dup, would give him backups. He could restore from backup after a failure. The downside is recovery time. Restoring a two terabyte system from a USB external drive, even a fast one, is going to take hours. And he'd need separate boot media, a live USB stick, to do the restore. What Daniel wants is five-minute recovery. Pull the dead drive, change the boot order, and he's back up. That's a different value proposition.
And there's a middle ground too. He could buy a two terabyte SATA SSD, put it in an external USB enclosure, and do the same rsync approach but with the drive external. That would protect against NVMe failure but also against some kinds of motherboard or power supply failure, since the backup drive is physically separate. But it would be slower to sync, and it wouldn't be as clean for the boot-swap recovery he's describing.
Right. The internal SATA approach means the standby is always there, always ready, and the sync is fast because it's all on internal SATA bandwidth. For a two terabyte drive, even a full rsync after the initial copy is going to be reasonably quick because rsync only copies changed files. I think for his use case, internal SATA is the right call. Let's spec out the parts.
Parts list. He already has the NVMe drive, so the only new hardware is the standby drive itself. What are we recommending?
He wants a two terabyte SATA SSD. I'm going to recommend the Samsung 870 EVO, two terabyte model. Part number MZ-77E2T0B. It's a proven drive, good endurance, good performance, and it's widely available. Roughly a hundred and thirty dollars, or about four hundred and eighty shekels. You can get it on Amazon with shipping to Israel, or from KSP locally for a bit more but faster delivery. The crucial alternative is the Crucial MX500, also two terabytes, model CT2000MX500SSD1. Slightly cheaper, around a hundred and ten dollars, about four hundred shekels. Both are solid. Avoid the cheap no-name SATA SSDs on AliExpress for this application. You want something with a DRAM cache and a known controller. The standby drive is your insurance policy, don't cheap out on the insurance.
And he'll need a SATA data cable and a SATA power cable. Does his power supply have a spare SATA power connector?
Almost certainly yes. Any power supply from the last fifteen years has multiple SATA power connectors. If he's somehow out of SATA power connectors, he can get a Molex to SATA adapter for a few shekels, but I'd be surprised if he needs one. The SATA data cable is more likely to be missing. He should check his motherboard box or his cable drawer. If he needs one, any standard SATA three cable works. They're about fifteen shekels at any computer shop. KSP stocks them, so does Ivory. Don't overthink the SATA cable, but also don't buy the absolute cheapest one. A decent latching SATA cable from a brand like StarTech or Cable Matters is worth the extra few shekels.
So total hardware cost is roughly a hundred and thirty dollars, about four hundred and eighty shekels, assuming he already has the cable. If he needs the cable, add five dollars, about twenty shekels. That's the entire hardware budget. This is not an expensive project.
It really isn't. And the payoff is enormous. Now let's talk about what he needs to have on hand tools-wise. A Phillips head screwdriver to mount the drive. Maybe a flashlight if his case interior is dark. That's it. No soldering, no multimeter, no special tools. This is a screwdriver-level project.
I appreciate a screwdriver-level project. Let's get into the build sequence. This is where the real work is. How do we make this standby drive bootable?
I'm going to walk through this step by step. The goal is to end up with a SATA SSD that is a byte-for-byte bootable copy of the NVMe drive, minus the parts that need to be different, like UUIDs and bootloader configuration. Here's the sequence.
Before you start, I want to flag something. He's going to be doing this on a live running system. That means he needs to be careful about what he copies and when. Files that are open and being written to during the copy can end up in an inconsistent state on the standby drive.
Great point. For a home server, the risk is manageable if he's not running a database or something with heavy write activity during the sync. But we'll address that in the script. Let's lay out the steps.
Step one. Physically install the SATA SSD. Power down the machine, mount the drive in a drive bay, connect the SATA data cable to the drive and to a SATA port on the motherboard, connect the SATA power cable. Boot back up.
Step two. Verify the system sees the new drive. He can run lsblk or fdisk minus l. The new drive will probably show up as something like dev slash sda or dev slash sdb, depending on what else is in the system. He needs to note the device name carefully. We do not want to accidentally overwrite the wrong drive.
Absolutely critical. Triple-check the device name. I recommend running lsblk minus o NAME, SIZE, MODEL, SERIAL to see the drive model and serial number, and make absolutely sure you know which device is your new two terabyte SATA SSD. If you get this wrong, you will destroy data. This is the step where you slow down and double-check everything.
Step three. Partition the new drive. He needs to create a partition table that matches the NVMe drive's layout. He can check the NVMe layout with fdisk minus l dev slash nvme zero n one. If the NVMe has an EFI system partition and a root partition, the SATA SSD needs the same layout. Use fdisk or parted to create matching partitions on the SATA drive.
And the sizes don't need to be identical, but the SATA partitions need to be at least as large as the data on the corresponding NVMe partitions. For the EFI partition, which is typically around five hundred megabytes, just make it the same size. For the root partition, make it fill the rest of the drive. Format the EFI partition as FAT thirty-two, and the root partition as the same filesystem type he's using on the NVMe, probably ext4 or maybe XFS or Btrfs. Use mkfs dot fat minus F thirty-two for the EFI partition, and mkfs dot ext4 or equivalent for the root partition.
Step four. This is important. He needs to handle UUIDs. Every filesystem has a unique UUID, and if both drives have the same UUID mounted at the same time, the system will get very confused. So after formatting, the new partitions will have their own UUIDs, which is what we want. We'll use those new UUIDs in the standby's fstab and bootloader configuration.
Right. And we'll capture those UUIDs now. Run blkid on the new partitions and write down the UUIDs. You'll need them later. Step five. Mount the new root partition somewhere, like mnt slash standby. Mount the new EFI partition at mnt slash standby slash boot slash efi, or wherever your EFI partition normally mounts.
Step six. Now we do the actual copy. This is the rsync step. The command is something like rsync minus aAXv, with exclusions for dev, proc, sys, run, mnt, media, and tmp, copying from the root of the running system to mnt slash standby. The minus A flag preserves ACLs, minus X preserves extended attributes, minus a is archive mode which preserves permissions, ownership, timestamps, and does recursive copy. The exclusions are critical because dev, proc, sys, and run are virtual filesystems created by the kernel at boot. You don't copy them, you create empty mount points for them on the target.
Let me give the exact rsync command. rsync minus aAXv slash mnt slash standby minus minus exclude equals dev minus minus exclude equals proc minus minus exclude equals sys minus minus exclude equals run minus minus exclude equals mnt minus minus exclude equals media minus minus exclude equals tmp minus minus exclude equals lost plus found. That trailing slash on the source is important. It means copy the contents of slash, not the directory itself.
And how long will this initial copy take? Two terabytes over SATA.
SATA three is six gigabits per second theoretical, real-world more like four to five hundred megabytes per second for a good SSD. Two terabytes at four hundred megabytes per second is about five thousand seconds, which is roughly eighty-five minutes. In practice, with filesystem overhead and small files, I'd budget two to three hours for the initial sync. Subsequent syncs will be much faster because rsync only copies changed files.
Step seven. After the copy completes, he needs to create the empty mount points on the standby that he excluded from the copy. So mkdir minus p mnt slash standby slash dev, mnt slash standby slash proc, and so on for sys, run, mnt, media, tmp.
Step eight. Now we fix the fstab on the standby drive. Edit mnt slash standby slash etc slash fstab. Replace the UUIDs with the UUIDs of the new SATA partitions that we captured earlier with blkid. This is the step that makes the standby bootable on its own. If you leave the old UUIDs in the fstab, the standby will try to mount the NVMe partitions at boot, which defeats the whole purpose.
And if the NVMe is dead, those UUIDs won't exist, and the boot will fail or drop to an emergency shell. So this step is not optional. Change the UUIDs.
Step nine. Now the bootloader. This depends on whether he's using GRUB or systemd-boot. If he's using GRUB, he needs to install GRUB to the SATA drive's boot sector and update the GRUB configuration on the standby. The commands are grub-install minus minus target equals x86 sixty-four-efi minus minus efi-directory equals boot slash efi minus minus bootloader-id equals standby-grub minus minus recheck dev slash sda, replacing dev slash sda with the actual SATA device. Then he needs to chroot into the standby and run update-grub, or on non-Debian systems, grub-mkconfig minus o boot slash grub slash grub dot cfg.
That chroot step is where people get tangled up. Walk through it.
Sure. After the rsync and fstab fix, he binds the virtual filesystems into the standby mount so that the chroot environment has access to device nodes and process information. The commands are mount minus minus bind dev mnt slash standby slash dev, mount minus minus bind proc mnt slash standby slash proc, mount minus minus bind sys mnt slash standby slash sys. Then chroot mnt slash standby. Now he's running inside the standby system's environment. From there he runs grub-install and update-grub. When done, exit the chroot and unmount the binds with umount mnt slash standby slash dev and so on.
And if he's using systemd-boot instead of GRUB?
systemd-boot is simpler in some ways. He needs to copy the kernel and initramfs from the NVMe's boot partition to the SATA's boot partition, and create or update the loader entries in the EFI partition pointing to the correct root UUID. The loader entries live in boot slash efi slash loader slash entries. He'd create a new dot conf file there with the correct root equals UUID equals whatever for the SATA root partition. Then he'd run bootctl install minus minus esp-path equals boot slash efi to install systemd-boot to the SATA drive's EFI partition. Same chroot approach applies.
So either way, the bootloader needs to know about the new root UUID. That's the common thread.
Step ten. Before he unmounts everything, he should verify the initramfs on the standby includes the drivers needed to boot from SATA. On most modern distributions, the initramfs is generic enough that this isn't a problem, but if he's using a custom initramfs or a very minimal one, he might need to regenerate it inside the chroot with update-initramfs minus u or dracut minus minus force. This is one of those failure modes I mentioned earlier. If the initramfs doesn't have the SATA controller driver, the standby won't boot.
So the full list of things that can go wrong: wrong UUIDs in fstab, bootloader not installed to the standby drive, bootloader configuration pointing to the wrong root partition, missing kernel or initramfs on the standby boot partition, and initramfs lacking the SATA driver. That's five failure pattern.
A sixth one, and this one is subtle. If he's using full disk encryption with LUKS on the NVMe, the standby drive needs its own separate LUKS setup. You can't just rsync an encrypted volume and expect it to work. If he's using LUKS, this whole process gets more complicated, and honestly, it probably moves beyond the scope of a simple rsync script. But he didn't mention encryption, so I'm assuming unencrypted ext4 or similar.
Reasonable assumption for a home server that lives inside his house. Let's move to the automation part. He wants a periodic background process to keep the standby updated.
Right. Step eleven is the sync script and the cron job or systemd timer. The script is essentially the rsync command from earlier, minus the exclusions that don't need repeating, plus a few safety checks. I'll outline it. First, the script should check that the standby drive is mounted. If it's not mounted, exit with an error. Second, it should check that the standby is actually the standby and not something else, maybe by checking the UUID of the mounted partition. Third, it runs the rsync. Fourth, it touches a file on the standby like dot last-sync with the current date so he can see when the last sync happened.
And how often should this run? Daily? Hourly?
For a home server, daily is probably fine. He can set it to run at three AM via cron. If he's doing work where losing a day of changes would be painful, he could run it every six hours or even every hour. The sync is incremental, so after the first run, each sync only copies files that changed. For a typical home server, that's probably a few hundred megabytes at most, and it'll complete in seconds to minutes. Running it hourly is not unreasonable.
Let's talk about the script in more detail. What about files that are open and being written to during the sync? You mentioned this earlier.
For most files on a home server, this isn't a catastrophic problem. If he's editing a document while the sync runs, the standby might get a partially-written copy. But the next sync will fix it. The bigger concern is databases. If he's running something like PostgreSQL or MySQL, or even SQLite databases with active writes, the rsync copy of the database file could be inconsistent and unusable. The fix is to either shut down the database service before the sync, or use database-specific dump tools to create a consistent snapshot, or accept that the standby's database files might need to be restored from a proper backup if the main drive fails. For a home server running a media server and some Docker containers, shutting down services for two minutes at three AM is probably fine. He can add systemctl stop commands before the rsync and systemctl start commands after.
The script stops critical services, runs rsync, starts the services, logs the result. That's a clean approach.
Clean and reliable. He can also add a pre-sync hook that checks disk health on both drives using smartctl. If the NVMe is showing signs of failure, the script can email him or send a notification. smartctl minus H dev slash nvme zero gives a simple pass or fail health status. Catching a failing drive before it dies completely is the dream scenario.
Let's put a pin in the monitoring part and come back to it. I want to make sure we cover testing. How does he verify the standby actually boots before he needs it?
This is the most important step and the one most people skip. Step twelve. Once everything is set up, he needs to do a test boot from the standby drive. Power down the machine, disconnect the NVMe drive entirely, or just change the boot order in the BIOS to boot from the SATA drive. Boot up and verify everything works. Check that services start, check that data is accessible, check that the filesystem mounted is actually the SATA drive and not somehow the NVMe. Run df minus h and verify the mount points show the SATA drive's partitions.
Disconnecting the NVMe is the safer test because it removes any ambiguity. If the system boots with the NVMe physically removed, he knows the standby is fully independent.
Agreed. Do the physical removal test at least once. After he's verified it works, he can reconnect the NVMe and set the boot order back to NVMe first. Going forward, he might want to do a test boot from the standby every few months just to make sure nothing has drifted. It takes five minutes and it's the only way to have real confidence.
What about the scenario where the NVMe fails and he boots from the standby? What's his path back to normal operations?
Good question. Once he's running on the standby, he's in a degraded state. He should replace the dead NVMe with a new one, then reverse the process. Partition the new NVMe, rsync from the now-running SATA system back to the NVMe, fix the UUIDs and bootloader on the NVMe, and switch back. The script he wrote for NVMe-to-SATA sync can be adapted for SATA-to-NVMe sync with just the paths changed. He might even write a second script for that direction and keep it ready.
He's not just building a standby, he's building a two-way synchronization capability. That's more useful than a one-shot clone.
Once he has the pattern down, it's reusable for any future drive replacement. Now let's talk about alternatives. I mentioned rsnapshot earlier. rsnapshot uses rsync under the hood but adds rotating snapshots using hard links, so he could keep daily, weekly, and monthly snapshots on the standby drive, not just the latest sync. That gives him a time dimension. If he accidentally deletes a file and doesn't notice for three days, a simple mirror won't help because the deletion would have been synced. But with rsnapshot's rotating snapshots, he could go back to last week's copy.
That's a meaningful upgrade. How much more complex is rsnapshot versus a plain rsync script?
rsnapshot is config-file driven. You install it with your package manager, edit etc slash rsnapshot dot conf, set the snapshot root to a directory on the standby drive, configure the backup intervals, and point it at the source directories. It handles the hard link rotation automatically. The tradeoff is that rsnapshot isn't designed to produce a bootable system image. It backs up files, not bootloaders and partition tables. So he'd still need to handle the bootloader installation and fstab UUIDs separately, and then point rsnapshot at the mounted filesystem. It's a hybrid approach. More flexible for file recovery, but more work to set up for full system bootability.
For his stated goal of a bootable standby, the plain rsync script is actually the cleaner path. rsnapshot is better if he also wants versioned file backups.
Right. He could also combine both. Use the rsync script for the bootable standby, and run rsnapshot targeting a separate directory on the same standby drive for versioned backups of his important data. With a two terabyte drive, he's got space for both. The standby system might take up a few hundred gigabytes, leaving plenty of room for rsnapshot snapshots.
What about borgbackup? You mentioned that earlier.
borg is fantastic for deduplicated, encrypted, compressed backups. It's not designed for creating a bootable standby. You'd use borg to back up your data to the standby drive, and then if the NVMe fails, you'd reinstall the OS from scratch and restore data from borg. That's a recovery time measured in hours, not minutes. Different use case.
The recommendation is clear. Plain rsync script, daily cron job, careful UUID and bootloader handling, and a physical test boot to verify. That's the build.
Let me give the complete parts and tools summary, and then the approximate total cost. Hardware: one Samsung 870 EVO two terabyte SATA SSD, model MZ-77E2T0B, roughly a hundred and thirty dollars or four hundred and eighty shekels, from Amazon or KSP. Or the Crucial MX500 two terabyte, model CT2000MX500SSD1, roughly a hundred and ten dollars or four hundred shekels. One SATA data cable if needed, about five dollars or twenty shekels, from KSP or Ivory. Tools: Phillips head screwdriver, flashlight optional. Software: rsync, which is pre-installed on essentially every Linux distribution. GRUB or systemd-boot, also pre-installed. A text editor for editing fstab and writing the sync script.
Total cost: roughly a hundred and fifteen to a hundred and thirty-five dollars, about four hundred twenty to five hundred shekels, depending on which drive he chooses and whether he needs a cable. That's the entire project budget.
For a five-minute recovery path on a server he's clearly invested a lot of time in, that's a bargain.
Let's talk about the monitoring piece you mentioned. smartctl for drive health. Should he set up automated monitoring, or just check manually?
Automated is better, because people forget to check manually. He can add a smartctl check to the sync script and have it send an email or a notification if the drive reports anything other than PASS. On most Linux systems, he can use the mail command or something like ntfy for push notifications. Even a simple cron job that runs smartctl minus H on both drives weekly and emails him if something changes is better than nothing. SSDs don't usually give a lot of warning before they fail, but SMART can catch some failure pattern, like excessive wear or reallocated sectors.
One more thing I want to flag. He should make sure the SATA SSD's firmware is up to date before he starts relying on it. Samsung provides their Magician software for firmware updates, though it's primarily Windows. On Linux, he can use fwupd or check the Samsung website for a bootable ISO. Crucial has their Storage Executive tool. It's a small step that's easy to skip.
Good call. Firmware bugs in SSDs are rare but when they hit, they can cause data loss or drive disappearance. Worth the five minutes to check.
To recap the build sequence one more time. Install the drive. Verify the device name. Partition and format to match the NVMe layout. Capture the new UUIDs. Mount the partitions. Rsync the system, excluding virtual filesystems. Create empty mount points. Fix fstab with the new UUIDs. Install and configure the bootloader. Regenerate initramfs if needed. Unmount everything. Do a test boot with the NVMe disconnected. Write the sync script with service stop and start hooks. Set up a cron job. Add SMART monitoring. Test boot from the standby quarterly.
That's the list. It sounds like a lot, but each step is straightforward, and the whole process, excluding the initial rsync, is maybe an hour of actual work. The rsync runs while he does other things.
The payoff is that when the NVMe eventually fails, and all drives eventually fail, he's back online in the time it takes to reboot. That's a good feeling at eleven PM on a Tuesday.
It really is. I've been in that situation, staring at a dead drive and knowing I have a standby ready to go. The relief is real. Daniel, you've already been through the hardware wringer with this machine. This project is your insurance policy against the next round. It's cheap, it's well-understood, and the tooling is mature. You're not inventing anything fragile. You're implementing a pattern that sysadmins have used for decades.
If you want to get fancy later, you can layer rsnapshot on top for versioned file backups, or set up the two-way sync script so you can flip back and forth between drives at will. But start with the basic bootable standby. Get that working, test it, and sleep better.
One last tip. Document what you did. Write down the partition layout, the UUIDs, the rsync command, the bootloader configuration. Put that document somewhere not on the server. If the server dies and you're panicking, you don't want to be trying to remember whether you used GRUB or systemd-boot. A simple text file in your notes app or a printed page taped inside the case can save you a lot of stress.
That's good advice for any project, and almost nobody does it. Be the person who does it.
Total cost, roughly a hundred and fifteen to a hundred and thirty-five dollars, about four hundred twenty to five hundred shekels. Send us a note when it's working, Daniel. We want to hear that the test boot succeeded and the standby came up clean.
Thanks to our producer Hilbert Flumingtop for keeping the show running. This has been My Weird Prompts. Find us at myweirdprompts dot com, or search for My Weird Prompts on Spotify. We'll be back next time with another build.
Happy syncing.
