Oh f❌k it, my NAS crashed in the middle of nightly backups because presumably that + Nextcloud uploads were too much.

I hate when the front LCD comes on. Because it's 40 characters out of a kpanic.

Even better, My mounts are over NFS. If I had to shut a container down right now it would hang until the NAS came back, potentially even needing a forceful host power-off from the iDRAC.

Even better: the one iSCSI initiator (backup) needs me to take the entire backup frontend offline to unmount and remount.

Even BETTER: I have to power cycle the NAS now. I legit created a script to send the SNMP commands to toggle its outlet group remotely on the UPS, just for this. I hate this f❌king NAS.

# fsck -C /dev/sdb1
fsck.ext2: No such device or address while trying to open /dev/sdb1
Possibly non-existent or swap device?

# iscsiadm -m session -P 3
iSCSI Connection State: TRANSPORT WAIT
iSCSI Session State: FREE
Internal iscsid Session State: REOPEN
Attached scsi disk sdb State: transport-offline

The superblock could not be read or does not describe a valid ext2/ext3/ext4 filesystem.

False alarm, needed to re-log the iSCSI connection, even though it said it came back up just fine.

NAS started a disk resync (md resync) so I guess I'll just have unbearably slow NAS ops for the next.... 13 days.

Follow

Seems it's all clean. Time to retry the backups.

· · Web · 0 · 0 · 0
Sign in to participate in the conversation
Tek's Mastodon

The social network of the future: No ads, no corporate surveillance, ethical design, and decentralization! Own your data with Mastodon!