Disk crashing NSA325? How to find out & fix

This forum is for topics dealing with problems with software specifically in the ARMv5 repo.

Disk crashing NSA325? How to find out & fix

Postby AvanOsch » Wed May 01, 2019 11:56 pm

After many years, I think one of the disks in my NSA325 is crashing.
This is output from "dmesg", showing I/O Errors:
$this->bbcode_second_pass_code('', '[ 34.438768] EXT4-fs (sda1): mounting ext2 file system using the ext4 subsystem
[ 34.711275] EXT4-fs (sda1): warning: mounting unchecked fs, running e2fsck is recommended
[ 34.792416] EXT4-fs (sda1): mounted filesystem without journal. Opts: (null)
[ 40.164842] mv643xx_eth_port mv643xx_eth_port.0 eth0: link down
[ 40.230568] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 40.471920] 8021q: 802.1Q VLAN Support v1.8
[ 43.221589] mv643xx_eth_port mv643xx_eth_port.0 eth0: link up, 1000 Mb/s, full duplex, flow control disabled
[ 43.231583] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 52.312645] NFSD: starting 90-second grace period (net c09161d0)
[ 67.397177] ata1.00: qc timeout (cmd 0x47)
[ 67.401319] ata1.00: READ LOG DMA EXT failed, trying unqueued
[ 67.407101] ata1: failed to read log page 10h (errno=-5)
[ 67.412448] ata1.00: exception Emask 0x1 SAct 0xa SErr 0x0 action 0x6 frozen
[ 67.419522] ata1.00: edma_err_cause=02000084 pp_flags=00000003, dev error, EDMA self-disable
[ 67.428007] ata1.00: failed command: READ FPDMA QUEUED
[ 67.433193] ata1.00: cmd 60/08:08:00:99:84/00:00:ac:00:00/40 tag 1 ncq 4096 in
res 41/40:18:00:cd:09/40:00:00:00:00/40 Emask 0x9 (media error)
[ 67.448226] ata1.00: status: { DRDY ERR }
[ 67.452293] ata1.00: error: { UNC }
[ 67.455799] ata1.00: failed command: READ FPDMA QUEUED
[ 67.460969] ata1.00: cmd 60/20:18:00:cd:09/00:00:00:00:00/40 tag 3 ncq 16384 in
res 41/40:18:00:cd:09/40:00:00:00:00/40 Emask 0x9 (media error)
[ 67.476339] ata1.00: status: { DRDY ERR }
[ 67.480362] ata1.00: error: { UNC }
[ 67.483882] ata1: hard resetting link
[ 67.991812] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl F300)
[ 68.071839] ata1.00: configured for UDMA/133
[ 68.076167] sd 0:0:0:0: [sda] tag#1 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
[ 68.084401] sd 0:0:0:0: [sda] tag#1 Sense Key : 0x3 [current] [descriptor]
[ 68.091391] sd 0:0:0:0: [sda] tag#1 ASC=0x11 ASCQ=0x4
[ 68.096570] sd 0:0:0:0: [sda] tag#1 CDB: opcode=0x28 28 00 ac 84 99 00 00 00 08 00
[ 68.104184] blk_update_request: I/O error, dev sda, sector 2894371072
[ 68.110816] sd 0:0:0:0: [sda] tag#3 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
[ 68.119049] sd 0:0:0:0: [sda] tag#3 Sense Key : 0x3 [current] [descriptor]
[ 68.126063] sd 0:0:0:0: [sda] tag#3 ASC=0x11 ASCQ=0x4
[ 68.131226] sd 0:0:0:0: [sda] tag#3 CDB: opcode=0x28 28 00 00 09 cd 00 00 00 20 00
[ 68.138843] blk_update_request: I/O error, dev sda, sector 642304
[ 68.145078] ata1: EH complete
[ 72.421873] ata1.00: exception Emask 0x0 SAct 0xc00000 SErr 0x0 action 0x6
[ 72.428781] ata1.00: edma_err_cause=02000084 pp_flags=00000003, dev error, EDMA self-disable
[ 72.437277] ata1.00: failed command: READ FPDMA QUEUED
[ 72.442459] ata1.00: cmd 60/b0:b0:58:ac:b5/00:00:5a:00:00/40 tag 22 ncq 90112 in
res 41/40:b8:00:cd:09/40:00:00:00:00/40 Emask 0x9 (media error)
[ 72.457663] ata1.00: status: { DRDY ERR }
[ 72.461688] ata1.00: error: { UNC }
[ 72.465195] ata1.00: failed command: READ FPDMA QUEUED
[ 72.470365] ata1.00: cmd 60/08:b8:00:cd:09/00:00:00:00:00/40 tag 23 ncq 4096 in
res 41/40:08:00:cd:09/00:00:00:00:00/00 Emask 0x409 (media error) <F>
[ 72.486259] ata1.00: status: { DRDY ERR }
[ 72.490281] ata1.00: error: { UNC }
[ 72.493801] ata1: hard resetting link
[ 73.001816] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl F300)
[ 73.101837] ata1.00: configured for UDMA/133
[ 73.106181] sd 0:0:0:0: [sda] tag#22 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
[ 73.114498] sd 0:0:0:0: [sda] tag#22 Sense Key : 0x3 [current] [descriptor]
[ 73.121576] sd 0:0:0:0: [sda] tag#22 ASC=0x11 ASCQ=0x4
[ 73.126876] sd 0:0:0:0: [sda] tag#22 CDB: opcode=0x28 28 00 5a b5 ac 58 00 00 b0 00
[ 73.134582] blk_update_request: I/O error, dev sda, sector 1521855576
[ 73.141192] sd 0:0:0:0: [sda] tag#23 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
[ 73.149520] sd 0:0:0:0: [sda] tag#23 Sense Key : 0x3 [current] [descriptor]
[ 73.156608] sd 0:0:0:0: [sda] tag#23 ASC=0x11 ASCQ=0x4
[ 73.161884] sd 0:0:0:0: [sda] tag#23 CDB: opcode=0x28 28 00 00 09 cd 00 00 00 08 00
[ 73.169569] blk_update_request: I/O error, dev sda, sector 642304
[ 73.177080] ata1: EH complete
[ 321.641792] EXT4-fs (sda2): error count since last fsck: 14
[ 321.647400] EXT4-fs (sda2): initial error at time 1556589755: ext4_journal_check_start:56
[ 321.655621] EXT4-fs (sda2): last error at time 1556659779: ext4_do_update_inode:4697: inode 115527604: block 460854827
[86820.199968] EXT4-fs (sda2): error count since last fsck: 14
[86820.205577] EXT4-fs (sda2): initial error at time 1556589755: ext4_journal_check_start:56
[86820.213798] EXT4-fs (sda2): last error at time 1556659779: ext4_do_update_inode:4697: inode 115527604: block 460854827
')

Because I've been running this server for years without problems, I totally forgot how the system was setup.
I vaguely remember the two disks are setup to share space as one volume.
Luckily, I can still access my data (read-only), but I have not been able to run "e2fsck" as recommended. Seems the two disks are always automatically mounted on boot, and I am unable to unmount.

Any tips on how I can diagnose the problem, find out what disk is failing, and possibly how to recover from this?
AvanOsch
 
Posts: 12
Joined: Thu Sep 05, 2013 12:57 am

Re: Disk crashing NSA325? How to find out & fix

Postby summers » Thu May 02, 2019 9:00 am

Probably safest is to pull the HDD from the nsa325, plug them into a desk top computer, and run fsck from there.

If this isn't possible, you can run fsck on a mounted file system, just force it. If the file system is mount read only, shouldn't cause many problems.

I'm a tad worried you seem to be getting major errors, and it doesn't correct it. Almost suggests a problem else where. So check the connections to the HDD, maybe just a loose connection ...

Oh yes, if you check /etc/fstab it should say how your HDD are mounted, e.g. how you merged the disks ...
summers
 
Posts: 984
Joined: Sat Sep 06, 2014 12:56 pm

Re: Disk crashing NSA325? How to find out & fix

Postby AvanOsch » Thu May 02, 2019 12:09 pm

$this->bbcode_second_pass_quote('summers', 'O')h yes, if you check /etc/fstab it should say how your HDD are mounted, e.g. how you merged the disks ...

Thanks Summers, but that's the thing... /mnt/disk1 and /mnt/disk2 are automatically mounted, and not listed in fstab...
I already tried disabling everything in fstab (except / and /boot), but that still didn't work.

I'll open her up, and see if I can spot something wrong.
I also have an external HDD dock, so I'll try diagnosing the disks that way.

Thanks again for your reply!
AvanOsch
 
Posts: 12
Joined: Thu Sep 05, 2013 12:57 am

Re: Disk crashing NSA325? How to find out & fix

Postby summers » Thu May 02, 2019 1:03 pm

$this->bbcode_second_pass_code('', '/proc/mounts') should show all the live mounts as well.
summers
 
Posts: 984
Joined: Sat Sep 06, 2014 12:56 pm

Re: Disk crashing NSA325? How to find out & fix

Postby AvanOsch » Thu May 02, 2019 1:55 pm

/proc/mounts only shows sda1 mounted on /boot (a 26MB partition of sda).
Using findmnt, I got the following output:
$this->bbcode_second_pass_code('', '[albert@Spock /]$ findmnt
TARGET SOURCE FSTYPE OPTIONS
/ /dev/sda2 ext4 rw,relatime,errors=remount-ro,data=ordered
├─/dev devtmpfs devtmpfs rw,relatime,size=254564k,nr_inodes=63641,mode=755
│ ├─/dev/shm tmpfs tmpfs rw,nosuid,nodev
│ ├─/dev/pts devpts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
│ └─/dev/mqueue mqueue mqueue rw,nosuid,nodev,noexec,relatime
├─/sys sysfs sysfs rw,nosuid,nodev,noexec,relatime
│ ├─/sys/kernel/security securityfs securityfs rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/cgroup tmpfs tmpfs ro,nosuid,nodev,noexec,mode=755
│ │ ├─/sys/fs/cgroup/systemd cgroup cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd
│ │ ├─/sys/fs/cgroup/cpu,cpuacct cgroup cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct
│ │ ├─/sys/fs/cgroup/blkio cgroup cgroup rw,nosuid,nodev,noexec,relatime,blkio
│ │ ├─/sys/fs/cgroup/cpuset cgroup cgroup rw,nosuid,nodev,noexec,relatime,cpuset
│ │ ├─/sys/fs/cgroup/memory cgroup cgroup rw,nosuid,nodev,noexec,relatime,memory
│ │ ├─/sys/fs/cgroup/perf_event cgroup cgroup rw,nosuid,nodev,noexec,relatime,perf_event
│ │ ├─/sys/fs/cgroup/net_cls,net_prio cgroup cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio
│ │ ├─/sys/fs/cgroup/devices cgroup cgroup rw,nosuid,nodev,noexec,relatime,devices
│ │ └─/sys/fs/cgroup/freezer cgroup cgroup rw,nosuid,nodev,noexec,relatime,freezer
│ ├─/sys/kernel/debug debugfs debugfs rw,nosuid,nodev,noexec,relatime
│ └─/sys/fs/fuse/connections fusectl fusectl rw,nosuid,nodev,noexec,relatime
├─/proc proc proc rw,nosuid,nodev,noexec,relatime
│ ├─/proc/fs/nfsd nfsd nfsd rw,relatime
│ └─/proc/sys/fs/binfmt_misc systemd-1 autofs rw,relatime,fd=45,pgrp=1,timeout=0,minproto=5,maxproto=5,direct
├─/run tmpfs tmpfs rw,nosuid,nodev,mode=755
├─/tmp tmpfs tmpfs rw,nosuid,nodev
├─/var/lib/mysqltmp tmpfs tmpfs rw,noatime,size=102400k,mode=750,uid=89,gid=89
├─/home/albert/Data /dev/sda2[/mnt/disk2] ext4 rw,relatime,errors=remount-ro,data=ordered
├─/boot /dev/sda1 ext2 rw,relatime,block_validity,barrier,user_xattr,acl,stripe=4
├─/mnt/disk1/Programming/www /dev/sda2[/srv/http] ext4 rw,relatime,errors=remount-ro,data=ordered
├─/srv/nfs/Logic /dev/sda2[/mnt/disk1] ext4 rw,relatime,errors=remount-ro,data=ordered
├─/srv/nfs/Data /dev/sda2[/mnt/disk2] ext4 rw,relatime,errors=remount-ro,data=ordered
├─/home/albert/Logic /dev/sda2[/mnt/disk1] ext4 rw,relatime,errors=remount-ro,data=ordered
└─/var/lib/nfs/rpc_pipefs sunrpc rpc_pipefs rw,relatime
')
Note the weird "SOURCE" of /srv/nfs/Logic, and /srv/nfs/Data (both /dev/sda2, but one on /mnt/disk1, the other /mnt/disk2)...
"Data" and "Logic" are the main NFS shares I have been using.
Also, I still can't seem to find where sdb1 is mounted.
AvanOsch
 
Posts: 12
Joined: Thu Sep 05, 2013 12:57 am


Return to ARMv5

Who is online

Users browsing this forum: No registered users and 2 guests