Her https://tinkerbetter.tube/w/ffhBwuXDg7ZuPPFcqR93Bd made me learn a new way of looking at data. There was some tricks I havent done before. She has such good videos.

Agree. It’s neat for file transfers and simple one-shot backups, but if you’re looking for a proper backup solution then other tools/services have advanced virtually every aspect of backups so much it pretty much always makes sense to use one of those instead.

I have it add a backup suffix based on the date. It moves changed and deleted files to another directory adding the date to the filename.

It can also do hard-link copied so that you can have multiple full directory trees to avoid all that duplication.

No file deltas or compression, but it does mean that you can access the backups directly.

If you want a “time travel” feature, your only option is to duplicate data.

Not true. Look at the --link-dest flag. Encryption, sure, rsync can’t do that, but incremental backups work fine and compression is better handled at the filesystem level anyway IMO.

Most Unix commands will show a short list of the most-helpful flags if you use --help or -h.

tldr and atuin have been my main way of remembering complex but frequent flag combinations

rsync -avzhP gang unite! I knew someone would have posted my standard flags. I used them enough that my brain moved them from RAM to ROM at this point…

sed can do a bunch of things, but I overwhelmingly use it for a single operation in a pipeline: the s// operation. I think that that’s worth knowing.

sed 's/foo/bar/'

will replace all the first text in each line matching the regex “foo” with “bar”.

That’ll already handle a lot of cases, but a few other helpful sub-uses:

sed 's/foo/bar/g'

will replace all text matching regex “foo” with “bar”, even if there are more than one per line

sed 's/\([0-9a-f]*\)/0x\1/g

will take the text inside the backslash-escaped parens and put that matched text back in the replacement text, where one has ‘\1’. In the above example, that’s finding all hexadecimal strings and prefixing them with ‘0x’

If you want to match a literal “/”, the easiest way to do it is to just use a different separator; if you use something other than a “/” as separator after the “s”, sed will expect that later in the expression too, like this:

sed 's%/%SLASH%g

will replace all instances of a “/” in the text with “SLASH”.

+1 for rdiff-backup. Been using it for 20 years or so, and I love it.

Having a synced copy elsewhere is not an adequate backup and snapshots are pretty important. I recently had RAM go bad and my most recent backups had corrupt data, but having previous snapshots saved the day.

Beware rdiff-backup. It certainly does turn rsync (not a backup program) into a backup program.

However, I used rdiff-backup in the past and it can be a bit problematic. If I remember correctly, every “snapshot” you keep in rdiff-backup uses as many inodes as the thing you are backing up. (Because every “file” in the snapshot is either a file or a hard link to an identical version of that file in another snapshot.) So this can be a problem if you store many snapshots of many files.

But it does make rsync a backup solution; a snapshot or a redundant copy is very useful, but it’s not a backup.

(OTOH, rsync is still wonderful for large transfers.)

I don’t know if there’s a term for them, but Bacula (and I think AMANDA might fall into this camp, but I haven’t looked at it in ages) are oriented more towards…"institutional” backup. Like, there’s a dedicated backup server, maybe dedicated offline media like tapes, the backup server needs to drive the backup, etc).

There are some things that rsnapshot, rdiff-backup, duplicity, and so forth won’t do.

• At least some of them (rdiff-backup, for one) won’t dedup files with different names. If a file is unchanged, it won’t use extra storage, but it won’t identify different identical files at different locations. This usually isn’t all that important for a single host, other than maybe if you rename files, but if you’re backing up many different hosts, as in an institutional setting, they likely files in common. They aren’t intended to back up multiple hosts to a single, shared repository.

• Pull-only. I think that it might be possible to run some of the above three in “pull” mode, where the backup server connects and gets the backup, but where they don’t have the ability to write to the backup server. This may be desirable if you’re concerned about a host being compromised, but not the backup server, since it means that an attacker can’t go dick with your backups. Think of those cybercriminals who encrypt data at a company and wipe other copies and then demand a ransom for an unlock key. But the “institutional” backup systems are going to be aimed at having the backup server drive all this, and have the backup server have access to log into the individual hosts and pull the backups over.

• Dedup for non-identical files. Note that restic can do this. While files might not be identical, they might share some common elements, and one might want to try to take advantage of that in backup storage.

• rdiff-backup and rsnapshot don’t do encryption (though duplicity does). If one intends to use storage not under one’s physical control (e.g. “cloud backup"), this might be a concern.

• No “full” backups. Some backup programs follow a scheme where one periodically does a backup that stores a full copy of the data, and then stores “incremental” backups from the last full backup. All rsnapshot, rdiff-backup, and duplicity are always-incremental, and are aimed at storing their backups on a single destination filesystem. A split between “full” and “incremental” is probably something you want if you’re using, say, tape storage and having backups that span multiple tapes, since it controls how many pieces of media you have to dig up to perform a restore.

• I don’t know how Bacula or AMANDA handle it, if at all, but if you have a DBMS like PostgreSQL or MySQL or the like, it may be constantly receiving writes. This means that you can’t get an atomic snapshot of the database, which is critical if you want to be reliably backing up the storage. I don’t know what the convention is here, but I’d guess either using filesystem-level atomic snapshot support (e.g. btrfs) or requiring the backup system to be aware of the DBMS and instructing it to suspend modification while it does the backup. rsnapshot, rdiff-backup, and duplicity aren’t going to do anything like that.

I’d agree that using the more-heavyweight, “institutional” backup programs can make sense for some use cases, like if you’re backing up many workstations or something.

.

rclone does support other protocols besides S3. You can also selfhost your own S3 storage.

rsync can sometimes look similarly scary! I very clearly remember triple-checking what it’s doing.

rclone works amazingly well if you have hundreds of folders or thousands of files and you can’t be bothered to babysit a portal.

It has been remarkably useful! I keep trying to tell people about it but apparently I am just their main use case or something.

I would have loved it when I was using Samba to share files on my local network decades ago. It’s like a Swiss Army knife!

Ah, I didn’t know of this. This should be in the linked article! Because it’s one of the ways to turn rsync into a real backup! (I didn’t know this flag- I thought this was the main point of rdiff-backup.)

The hard links aren’t between the source and backup, they’re between Friday’s backup and Saturday’s backup

For a home setup that seems fine. But I can understand why you wouldn’t want this for a whole enterprise.

That would only matter if it’s lots of small files, right? And after the initial sync, you’d have very few files, no?

Rsync is designed for incremental syncs, which is exactly what you want in a backup solution. If your multithreaded alternative doesn’t do a diff, rsync will win on larger data sets that don’t have rapid changes.

On low end CPUs you can max out the CPU before maxing out network—if you want to get fancy, you can use rsync over an unencrypted remote shell like rsh, but I would only do this if the computers were directly connected to each other by one Ethernet cable.

It’s just a one-off transfer, I’m not planning to stop the transfer, and it’s my media library, so nothing should change, but I figured something resumable is a good idea for a transfer that’s going to take 12+ hours, in case there’s an unplanned stop.

Yes but imagine… 18 seconds.

With xargs everything is multithreaded.

I dunno.

I am using it to keep a real time copy of documents on an offsite server.

Feels like a backup to me.

It took me like 10 min to setup rsnapshot (installing, and writing systemd unit /timer files) on my servers.

man rsync

Blogs can have ads.

Yup, just configure a snapshot policy and you can recover deleted and modified files going back as long as you choose. And it is probably more space efficient than both/restic too.

If I connect to the same server via my own VPN I don’t have the disconnections, so I’m thinking it’s tailscale cutting connections after too much traffic. But connecting via tailscale is so much more convenient 😢

Yeah but then it’s not really a good backup!

How does that get sent over rsync though? Wouldn’t you need snapshots on the remote destination server?

Why not just use a backup utility instead?

If you want to use it for backups, there are other solutions, so you still don’t need to use it or know it. You can use something else. That’s my only point. 🤷‍♂️

And “really bad” is all relative. If you are only backing up your home drive with documents or whatever, copying a few unnecessary gigabytes over a LAN connection isn’t too bad at all. But scp isn’t what you should be using for backups anyway. I only used rsync for file transfer…