I think I know why was the server misbehaving. One of boot SSDs is failing.
SMART overall-health self-assessment test result: FAILED!
Drive failure expected in less than 24 hours. SAVE ALL DATA.
No failed Attributes found.
Follow
If you are booting from a Linux RAID1 array, can you just power it off, replace the drive and it boots up and starts rebuilding?
@matrix in theory it should, it just depends on implementation
giving it a try should be fine as long as you have atomic writes
giving it a try should be fine as long as you have atomic writes
@matrix worst case you lose a few writes but if it's critical writes they will be attempted multiple times usually
@caskd I think you need to add the drive to the array
@matrix if it is software raid, yes
@matrix not sure about booting from it but once you boot with a degraded array you add the new device and it should start rebuilding
@matrix It's almost dead but I guess it kind of works occasinally for long enough to pass the self-assessment test
@matrix https://raid.wiki.kernel.org/index.php/Replacing_a_failed_drive
assuming you're using mdadm
@matrix
NO!
You need to at least add the new disk to the array.
And now it passed a test. Wtf