PSA: Don't use nextcloud's auto upload on the android app as a backup
from skiguy0123@lemmy.ca to selfhosted@lemmy.world on 07 Dec 18:35
https://lemmy.ca/post/56459587
from skiguy0123@lemmy.ca to selfhosted@lemmy.world on 07 Dec 18:35
https://lemmy.ca/post/56459587
I recently noticed that my nextcloud instance was missing photos. I have the android app set to automatically upload my photos. When I need to clear up space on my phone, I make a separate backup (because I’m a paranoid SOB and hard drives are relatively cheap). I noticed that nextcloud auto upload missed about 10% of the photos. I’m not going to bash the nextcloud devs, as I recognize that I am using a free product and am owed nothing, but I’m making this post so others are aware of this risk. Apparently I’m not alone help.nextcloud.com/t/…/14
threaded - newest
Can you reproduce it?
I don’t have the time, or quite frankly the inclination, to try, when I can just switch to immich and/or syncthing (currently evaluating my options right now). I will say we noticed photos missing from my wife’s upload as well, so that’s two android clients and one server. That plus the complaints in the link I provided are all the evidence I need to bail.
Ok
Just wanted to watch out for you cause I ran into this with Immich. It doesn’t chunk uploads. So if you haveva big video, it will never upload because anytime it gets interrupted, it has to start over.
I’ve had very occasional issues with it not uploading new photos in a timely manner in the past. I haven’t had any issues in a long time, but I have gotten into the habit of explicitly opening the app, clicking “Uploads” and hitting refresh and making sure everything has been uploaded.
I’m not really sure what causes it, though if I had to guess Android is putting the app to sleep in the background so it may have something to do with power saving settings. I’ve switched to the F-Droid version of the app and manually disabled the appropriate power settings as a just-in-case, though that may have nothing to do with anything.
Yeah. Unfortunately a known problem, even if the cause isn’t always clear.
This is a know issue that has a very long issue on github, it started for me when Google changed the rules on full file access. But the latest version of the app has been rock solid.
I ended up downloading all my photos to my laptop and doing a sync from there
What’s the link to the issue? I’d like to follow its progress. Thank you!
your in luck, as I couldnt find the issue when I posted.
But there is an update to this 👉 github.com/nextcloud/android/issues/15822
Thank you! It seems the ongoing 3.35 version is reimplementing the auto-upload logic, so this is promising. It is still in RC stage, but hopefully will be stable soon.
Weird coincidence that the exact reason for this thread is addressed in github.com/nextcloud/android/…/rc-3.35.0-01. RC2 was just released yesterday.
Yes and no, there have been issues with the android app for a long time. Some of it was Google and some the way it would detect files, it just happens that I posted the GitHub issue.
i just have a simple shell script that uses rsync to upload my files to a server and have a termux shortcut on the homescreen, i just press it once in a few days
I do something similar using only
rsync. I’ve had nothing but headaches whenever I used automated file syncing programs. The bare bones aspect of just using rsync has made it a much more consistent experience.I found using automated file syncing programs have too much complexity under the hood that just seems to lead to more time troubleshooting issues.
Do you have any need to backup things on a mobile phone? What do you use if so? I’m trying to make everything simpler and I already use rsync on my desktop.
There’s a few things I backup from my phone.
I have an Android phone so I use Termux as a terminal emulator. I use ssh and passwordless keys to make transfers simpler and quicker.
Although this is closer to a backup process and not like SyncThing where it’s syncing a folder between two devices. I don’t believe rsync is capable of acting like SyncThing but I’m tempted to dig into rsync more and see if I can put something basic together one day.
Hoping you hit it with Borg backup before uploading anywhere!
this is my first time hearing about borg, is there anything it does that can improve my current setup? from a quick look it seems like it creates its own backup file while rsync works with raw directory structure.
I mainly use it to deduplicate, and for version history. Rsync - >Borg - > rclone to backblaze b2 bucket
I have the same issue with Immich on android. It pretty much never uploads files until I manually open the app; then the app refuses to acknowledge it has uploaded those new files until it’s closed and re-opened :( (power saving is set to un-restricted in android, and background data usage is allowed. I’ve been through troubleshooting very thoroughly, it just doesn’t work)
FolderSync has been the only reliable (non-root) backup solution I’ve used. It’s set to monitor my image folders for changes and upload any new files as soon as they’re created; this works ~85% of the time. Then, It’s also set with a few schedules to check for changes every 3hrs, backing up everything on the phone the app can access; this catches anything the on-change/on-creation file detection misses, while also backing up more data than just my images. I have yet to see that fail after ~3 years.
Foldersync is not open source though.
Can attest that Folder Sync is excellent. I use it all day (in the background) for two-way sync (notes) and backup of photos videos etc
Though a small PSA on setting up:
I once set up a new share on a new phone with two-way sync, and the app decided to sync the (newer) empty directory to the server (i.e. delete everything) instead of pulling the files from the server to the phone.
Easy fix: Restore notes from backup (step 0: have backups in the first place), then do an initial 1-way sync from server to phone, then change the sync job to two-way.
I’ve been using Syncthing for years and it’s been almost flawless with only rare file sync errors that are clearly shown in the UI. Was going to switch to Nextcloud for everything. Looks like I’ll be sticking with Syncthing for the foreseeable future.
Thanks for posting this.
I came from Nextcloud to syncthing, you’re in the right place.
Syncthing for android is a mess. I need to redo a folder or two every couple weeks because of some unresolvable sync errors or ghost files…
That’s bizarre. Did you try clearing up the hidden folders
.stfolderand.stversionsand starting from scratch? I’m not doubting that you experience what you experience, I just never had it and I’m curious how you can even get to such a state. Using “syncthing fork” from F-Droid and Syncthing-v2.0.12 on desktop.That is what I usually end up having to do. The worst problem I’ve had so far was one directory endlessly reappearing even after deleting it from all devices manually. I looked into it and found out it was because android syncthing can’t handle some characters in the directory names, but of course it doesn’t tell you this. Just renaming the directory duplicated the data with the old name, so I had to setup everything from scratch like you said.
When I was running a mesh topology I often had the same issue. Switching to a star topology fixed pretty much everything.
Yeah I’m running mesh too. I need to try that, thanks.
Can’t wait for proper Linux phones to be more viable so that we have more control over shit like this.
Honest question: what would you do if there was a proper Linux phone to not have this problem? Fork and maintain your own nextcloud client?
There are so maby different solutions for this on Linux. Syncing folders and backups in general is an old problem with many, many approaches
Use Nextcloud’s desktop client? Or the cli client with cron/systemd timers. Or any old webdav client.
Use the desktop client on a phone?
Yes, that’s the whole point of Linux on a phone. The ability to easily use existing software. The app already has small screens so using it on a phone should be possible without any trouble.
Interesting, I’ve never had this problem and have been using the auto upload for many years. I only use it for the photo taking, though. Any time I go through a phone cleanup and delete photos and videos, I’ll look at the folders I backup to on nc and actually go through them carefully to make sure everything was uploaded.
My install comes from f-droid so not sure if that’s something to do with it (unless yours is too).
Mine was also f-droid, but my wife’s wasn’t. Experienced issue on both :(
I thought with this for years. It’s unreliable and buggy on Android and iPhone. I caved and paid for some photo sync app and it’s been super stable.
That or folder sync on Android. Then feed into immich or photosphere.
I spent many nights running diff and comparing sources and destinations and md5sums and so on
Is that the proprietary FolderSync or is there a FOSS solution for syncing folders ?
I used FolderSync with OneDrive (in the past) a’d it worked ok, not shitting on it, I’m just looking for a FOSS equivalent with Nextcloud
You could use Syncthing for local folder syncing between devices. It’s been really reliable for me.
Thanks but the goal is to sync with the “cloud”, for backup in case of fire or something.
I have hundreds (thousand ?) of albums I need to backup. I can reencode them cause 98% are on CD, but if I loose both my computers and my CDs, I’m done :/
I only use cloud backup for music and the few photos I take with my phone so I don’t really need real-time syncing.
I sync to TrueNAS scale with photosync and then I sync scale with back blaze b2
Would a safety deposit box at a bank be an appropriate option for your off-site backups?
Syncthing or Resilio Sync for photo/file backup from phone. Both work amazingly well.
Sun thing for me was absolutely terrible. If I recall correctly, huge pain in the ass because I vlan off my wifi and had to mess with policies for discovery. I found the app would often freeze and lock up, glitch, etc. And this was on two different phones.
i stopped using it for a while because of this bug. this is the entire purpose of using something like nextcloud. syncthing is much worse, so…