HDD randomly unmounting
from dontblink@feddit.it to selfhosted@lemmy.world on 13 Nov 2024 08:54
https://feddit.it/post/12374951

Hi!

First of all sorry if this is the wrong place to ask, if it is, please point me to a better suited channel!

Anyway I’ve got this old 2TB HDD attached to a rpi 4b, it worked flawlessly until now, the last few days it started disconnecting randomly…

If i reboot it mounts back again.

This is the df output:

/dev/sdb1       1.8T  535G  1.2T  31% /mnt/2tb

And this is sudo dmesg | grep sdb (the device is sdb ofc).

[   14.970908] sd 1:0:0:0: [sdb] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
[   14.978857] sd 1:0:0:0: [sdb] 4096-byte physical blocks
[   14.984484] sd 1:0:0:0: [sdb] Write Protect is off
[   14.989382] sd 1:0:0:0: [sdb] Mode Sense: 43 00 00 00
[   14.989684] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   15.044802] sd 1:0:0:0: [sdb] Preferred minimum I/O size 4096 bytes
[   15.051196] sd 1:0:0:0: [sdb] Optimal transfer size 33553920 bytes not a multiple of preferred minimum block size (4096 bytes)
[   15.065585]  sdb: sdb1
[   15.068403] sd 1:0:0:0: [sdb] Attached SCSI disk
[   22.631983] EXT4-fs (sdb1): recovery complete
[   22.660922] EXT4-fs (sdb1): mounted filesystem with ordered data mode. Quota mode: none.

The device has an external power supply of its own, so it’s not a power issue… This setup worked for a couple of years.

I cannot see anything wrong here, pheraps is the HDD which is going bad?

#selfhosted

threaded - newest

AceBonobo@lemmy.world on 13 Nov 2024 09:34 next collapse

Maybe the power supply is dying? Do you move it often? Or could the USB cable be degrading?

seaQueue@lemmy.world on 13 Nov 2024 09:43 next collapse

Don’t just look at sdb hits in the log. Open up that entire session in journalctl kernel mode (journalctl -k -bN where N is the session number in session history) and find the context surrounding the drive dropping and reconnecting.

You’ll probably find that something caused a USB bus reset or a similar event before the drive dropped and reconnected. if you find nothing like that try switching power supplies for the HDD and/or switching USB ports until you can move the drive to a different USB root port. Use lsusb -t and swap ports until the drive is attached beneath a different root port. You might have a neighboring USB device attached to the bus that’s causing issues for other devices attached to the same root port (it happens, USB devices or drivers sometimes behave badly.)

Always look at the context of the event when you’re troubleshooting a failure like this, don’t just drill down on the device messages. Most of the time the real cause of the issue preceded the symptom by a bit of time.

hendrik@palaver.p3x.de on 13 Nov 2024 10:59 next collapse

Very good answer. I've also spent some time analyzing some red herrings when it was something else like a bad cable or connector. And by the way, you can use the same keys in journalctl as in the usual pager (less(?)) so hit / and search for 'unmount', 'disconnect', etc. And then scroll through the log and find out what led to the situation.

dontblink@feddit.it on 22 Nov 2024 10:16 collapse

Thank you so much for taking the time to answer!

I’m not sure how to get the N from session history, nor how to check my session history…

but this might be some relevant output I’ve found with journalctl -k -b

Nov 21 16:08:18 rpi kernel: usb 2-2.1-port2: cannot reset (err = -110)
Nov 21 16:08:19 rpi kernel: usb 2-2.1-port2: cannot reset (err = -110)
Nov 21 16:08:19 rpi kernel: usb 2-2.1-port2: Cannot enable. Maybe the USB cable is bad?

Nov 21 16:41:57 rpi kernel: I/O error, dev sdb, sector 2466347032 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 2
Nov 21 16:41:57 rpi kernel: EXT4-fs warning (device sdb1): ext4_dx_find_entry:1796: inode #75497968: lblock 42: comm apache2: error -5 reading directory block
Nov 21 16:41:57 rpi kernel: EXT4-fs error (device sdb1): ext4_journal_check_start:83: comm apache2: Detected aborted journal
Nov 21 16:41:57 rpi kernel: Buffer I/O error on dev sdb1, logical block 0, lost sync page write
Nov 21 16:41:57 rpi kernel: EXT4-fs (sdb1): I/O error while writing superblock
Nov 21 16:41:57 rpi kernel: EXT4-fs (sdb1): Remounting filesystem read-only

The output is from yesterday, when the device stopped working correctly.

I’m not familiar with linux kernel, but I can see there is definitely something wrong…

The HDD (old) is attached to a USB hub (new), I tried switching port of the hub but the same issue happened again, if I try to mount it with sudo mount /mnt/2tb, it says it is already mounted:

mount: /mnt/2tb: /dev/sdb1 already mounted on /mnt/2tb.
       dmesg(1) may have more information after failed mount system call.

sudo dmesg | grep sdb gives back:

[147776.801028] I/O error, dev sdb, sector 77904 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 2
[147776.815452] EXT4-fs warning (device sdb1): htree_dirblock_to_tree:1083: inode #2: lblock 0: comm ls: error -5 reading directory block
[147796.731734] sdb1: Can't mount, would change RO state
seaQueue@lemmy.world on 26 Nov 20:32 collapse

I’m not sure how to get the N from session history, nor how to check my session history…

journalctl --list-boots will list all sessions stored in the journal.

The output is from yesterday, when the device stopped working correctly.

I’m not familiar with linux kernel, but I can see there is definitely something wrong…

The HDD (old) is attached to a USB hub (new), I tried switching port of the hub but the same issue happened again, if I try to mount it with sudo mount /mnt/2tb, it says it is already mounted:

Those messages tell you what’s happening, there’s an unrecoverable error on the USB bus connecting the hard drive which is causing filesystem errors when writes fail. Diagnose that, lose the hub first and directly connect the drive to the pi, then try replacing the cable that attaches the drive if the error still occurs. I’d also check with people in the rpi community in case there are any known issues with USB on your model. There may be some pi specific USB firmware things you can do to increase reliability.

You can also try disabling UASP for the drive in case BOT transfer somehow stabilizes the connection. You’ll lose performance but that helps with some USB storage bridges.

Some USB storage bridges are just unreliable under Linux and crash under load, your last option is to buy another drive enclosure that’s tested and known to work correctly. I went through like 5 USB/NVMe enclosures looking for one that worked properly, that whole space is a compatibility mess.

BCsven@lemmy.ca on 13 Nov 2024 15:26 next collapse

USB Cable connection, power failing? Is drive set to power down on idle and then falling off the radar?

possiblylinux127@lemmy.zip on 13 Nov 2024 20:40 next collapse

Please post a full dmesg and a full list of specs

stonkage@aussie.zone on 13 Nov 2024 21:35 next collapse

What does your fstab say?

AceSLS@ani.social on 14 Nov 2024 14:21 collapse

Sounds like the HDD is dying, maybe check it’s S.M.A.R.T. status? Most drives have statistics for errors and such