RaspBerry with YunoHost randomly crashing
from ISOmorph@feddit.org to selfhosted@lemmy.world on 13 Jul 18:29
https://feddit.org/post/32603021

Hello,

I need some help troubleshooting. As per the title, I’m using a Raspberry Pi 3 Model B Plus Rev 1.3 with Debian bookworm kernel 6.12.93+rpt-rpi-v8 to host a local YunoHost server (12.1.40.1 stable) with the following apps:

Sadly, the server randomly crashes and can only be rebooted trough a hardware reset (pulling the plug).

Here are the last 50 lines of journalctl before the last crash: privatebin.net/?dbca869581c95573#83bk7W2CEPbeBGAX…

Here are the last 50 lines of journalctl before the crash before that: privatebin.net/?f7bdf2fd48ab12f7#9ZPh68rJ1p8JdzyL…

I do not see anything troubling. The older journalctl read outs are equally bland. Do you have any advice on how to proceed?

#selfhosted

threaded - newest

51dusty@lemmy.world on 13 Jul 19:04 next collapse

In my experience random reboot problems are usually related to the sd card. standard logging setups kill the card faster than you’d think. dd the existing install or rebuild the server.

time to test your disaster recovery strategy!

GeneralDingus@lemmy.cafe on 13 Jul 19:09 next collapse

I went with Alpine Linux for hosting because of this. I like up how my light setup runs completely in RAM, saving the SD card for a while.

undefined@lemmy.hogru.ch on 14 Jul 05:31 collapse

I really tried to use a “usb” install for a long time but could never get kernel upgrades (and some other packages) to work correctly.

These days I do a “sys” install but mount /var/log and some other directories to RAM.

ISOmorph@feddit.org on 13 Jul 19:55 collapse

Thank you and u/irmadlad for the suggestion. I ordered a new SD card and will just clone the current one. I’ll probably get to it this weekend and let you know if it worked out.

irmadlad@lemmy.world on 13 Jul 20:41 collapse

Fingers crossed.

ISOmorph@feddit.org on 19 Jul 18:06 collapse

So I have the new SD card active and haven’t had any service interruptions in the last 48h. I’m gonna optimistically assume this solved the problem. Thanks again.

irmadlad@lemmy.world on 19 Jul 18:53 collapse

That’s awesome news! Happy to be of what little service I can. I’m hoping the new SD card does the trick.

irmadlad@lemmy.world on 13 Jul 19:10 next collapse

I’m an expert at nothing, but I’m going to guess and watch the thread: Power/Under voltage or SD corruption. I’m favoring the latter.

Reason being, the logs are pretty normal. I don’t see anything that would portend a crash. So it could be that when the RPi crashes requiring you to pull the plug, it hasn’t had a chance to write that to the logs. I know, back in my RPi days, I had a lot of issues with SD corruption.

51dusty@lemmy.world on 13 Jul 21:07 collapse

agreed.

voltage problems would most likely result in odd errors in syslog… random controller grumbling messages, resets, etc.

deviceshelf@lemmy.world on 18 Jul 13:28 collapse

The bland logs are themselves a data point. If the journal just stops mid-line with nothing unusual before it, that points at a hard hang or a power cut rather than something userspace did, because a kernel oops normally leaves a trace behind. There’s also a catch-22 in the SD theory: a card that’s failing can’t reliably write the log that would prove it’s failing.

Two things worth doing before the card swap, both cheap. Check vcgencmd get_throttled: bit 16 stays set if undervoltage happened at any point since boot, so you get an answer on power-versus-card without having to catch the crash live. And switch on the hardware watchdog, bcm2835_wdt plus RuntimeWatchdogSec in systemd, so the Pi resets itself instead of waiting for someone to pull the plug. That doesn’t fix the cause, but it stops every crash from becoming an outage that lasts until you’re back home.

ISOmorph@feddit.org on 18 Jul 18:05 collapse

I wasn’t aware of a hardware watchdog, thank you very much for that. It’s now turned on and the new SD card has been active since yesterday without hiccups.