aarch64: Kernel instabilities

[df8oe]

Hi to all,
I have run linux vanilla kernel since over a year up to August 22. I started a bigger project on my rpi at this time. I was unsure this project leads to instablities, so I switched to linux-rpi aarch 64 and instablities vanished. Some days later they started again (LAN bug...). After this was fixed they were rarely present. Starting with 5.15.65-2 they popped up minutes later. It seems to me there are different changes in drivers and kernel which are more or less counterproductive and may lead to these instabilities. I want to check if I can find an older kernel which does not show these at all. So I need a list at which time a kernel version was released. So I want to step down to check when instablities start.

Also I want to ask if it is possibl (and if how) I can step from linux-rpi back to vanilla, because I am sure to know the latest kernel which was rock-solid.

[muddI]

thats interesting.

I left the 5.15.kernel after patchlevel 39 and swiched to 5.18, 5.19.
I noticed some more probs with i2c, but I am not sure where they come from.
Want to check this and downgrade the kernel too. Could you tell us which kernel version you think was the last stable for you?

for a list of "vanilla" kernels release I simply use
https://cdn.kernel.org/pub/linux/kernel/v5.x/incr/

[graysky]

There was an ethernet bug introduced in .61 which was fixed in .64. Without logs we have nothing to go on to help you. If you want to just downgrade until you find that a stable kernel you can use http://tardis.tiny-vps.com/aarm/packages/l/linux-rpi/

Better to find out the source of the problem and use an updated kernel. If 5.15.65-2 is not stable, try 5.19.7, see: viewtopic.php?f=3&t=16144

[df8oe]

I want to test new kernel - but I cannot find any keyserver from where I can import your public key...
$this->bbcode_second_pass_code('', 'pacman-key -r 5EE46C4C --keyserver hkp://pgp.mit.edu
gpg: Empfangen vom Schlüsselserver fehlgeschlagen: Datei oder Verzeichnis nicht gefunden
==> FEHLER: Entfernter Schlüssel konnte nicht korrekt vom Schlüssel-Server abgerufen werden.
')


Forget it. I found it on keyserver.ubuntu.com.

Now I start to test new kernel. Booting was successfully.

Instabilities of 15 kernel was related to USB I guess. I have attached an USB RAID controller and system crashes often when big amount of data was moved from/to the raid. The crash always was deep and leads to set the root directory mounted ro so I do not have any logs at this moment. I have attached a monitor to hdmi so I can see there was a kernel panic and then system goes crazy. I will report what is the behaviour of 19 kernel.

[df8oe]

First observations with 5.19.7-2-rpi-ARCH:

This kernel works fine until now. Yesterday I copied ~500.000 files from and to USB-RAID and the night run 3 rsync processes to my rootservers - everything finished successfully. I will report some days later. @graysky: Many thanks for providing 5.19 kernel with rpi specific support!

[df8oe]

OK - last night it crashed again during a rsync process. But I have exported something from the logs (journal) - may be it points where to look??
$this->bbcode_second_pass_code('', 'Sep 11 02:53:14 raspserver kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s
Sep 11 02:53:14 raspserver kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x2a 2a 00 03 c0 08 00 00 00 08 00
Sep 11 02:53:14 raspserver kernel: EXT4-fs error: 22 callbacks suppressed
Sep 11 02:53:14 raspserver kernel: EXT4-fs error (device sda4): __ext4_find_entry:1658: inode #109961253: comm rsync: reading directory lblock 0
Sep 11 02:53:15 raspserver kernel: EXT4-fs error (device sda4): __ext4_find_entry:1658: inode #109961253: comm rsync: reading directory lblock 0
Sep 11 02:53:15 raspserver kernel: EXT4-fs error (device sda4): __ext4_find_entry:1658: inode #109961253: comm rsync: reading directory lblock 0
Sep 11 02:53:15 raspserver kernel: EXT4-fs error (device sda4): __ext4_find_entry:1658: inode #109961253: comm rsync: reading directory lblock 0
Sep 11 02:53:15 raspserver kernel: EXT4-fs error (device sda4): __ext4_find_entry:1658: inode #109961253: comm rsync: reading directory lblock 0
Sep 11 02:53:15 raspserver kernel: EXT4-fs error (device sda4): __ext4_find_entry:1658: inode #109961253: comm rsync: reading directory lblock 0
Sep 11 02:53:15 raspserver kernel: blk_print_req_error: 54 callbacks suppressed
Sep 11 02:53:16 raspserver kernel: I/O error, dev sda, sector 62916608 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
Sep 11 02:53:16 raspserver kernel: I/O error, dev sda, sector 3587059592 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
Sep 11 02:53:16 raspserver kernel: EXT4-fs error (device sda4): __ext4_find_entry:1658: inode #109961253: comm rsync: reading directory lblock 0
Sep 11 02:53:16 raspserver kernel: I/O error, dev sda, sector 62916608 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
Sep 11 02:53:16 raspserver kernel: I/O error, dev sda, sector 3587059592 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
Sep 11 02:53:16 raspserver kernel: EXT4-fs error (device sda4): __ext4_find_entry:1658: inode #109961253: comm rsync: reading directory lblock 0
Sep 11 02:53:16 raspserver kernel: I/O error, dev sda, sector 62916608 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
Sep 11 02:53:16 raspserver kernel: I/O error, dev sda, sector 3587059592 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
Sep 11 02:53:16 raspserver kernel: EXT4-fs error (device sda4): __ext4_find_entry:1658: inode #109961253: comm rsync: reading directory lblock 0
Sep 11 02:53:16 raspserver kernel: I/O error, dev sda, sector 62916608 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
Sep 11 02:53:16 raspserver kernel: I/O error, dev sda, sector 3587059592 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
Sep 11 02:53:16 raspserver kernel: EXT4-fs error (device sda4): __ext4_find_entry:1658: inode #109961253: comm rsync: reading directory lblock 0
Sep 11 02:53:16 raspserver kernel: I/O error, dev sda, sector 62916608 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
Sep 11 02:53:16 raspserver kernel: I/O error, dev sda, sector 3587059592 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
Sep 11 02:53:16 raspserver kernel: buffer_io_error: 22 callbacks suppressed
Sep 11 02:53:16 raspserver kernel: Buffer I/O error on dev sda4, logical block 0, lost sync page write
Sep 11 02:53:16 raspserver kernel: EXT4-fs: 22 callbacks suppressed
Sep 11 02:53:16 raspserver kernel: EXT4-fs (sda4): I/O error while writing superblock
Sep 11 02:53:16 raspserver kernel: Buffer I/O error on dev sda4, logical block 0, lost sync page write
Sep 11 02:53:16 raspserver kernel: EXT4-fs (sda4): I/O error while writing superblock
Sep 11 02:53:16 raspserver kernel: Buffer I/O error on dev sda4, logical block 0, lost sync page write
Sep 11 02:53:16 raspserver kernel: EXT4-fs (sda4): I/O error while writing superblock
Sep 11 02:53:16 raspserver kernel: Buffer I/O error on dev sda4, logical block 0, lost sync page write
Sep 11 02:53:16 raspserver kernel: EXT4-fs (sda4): I/O error while writing superblock
Sep 11 02:53:17 raspserver kernel: Buffer I/O error on dev sda4, logical block 0, lost sync page write
Sep 11 02:53:17 raspserver kernel: EXT4-fs (sda4): I/O error while writing superblock
Sep 11 02:53:17 raspserver kernel: Buffer I/O error on dev sda4, logical block 0, lost sync page write
Sep 11 02:53:17 raspserver kernel: EXT4-fs (sda4): I/O error while writing superblock
Sep 11 02:53:17 raspserver kernel: Buffer I/O error on dev sda4, logical block 0, lost sync page write
Sep 11 02:53:17 raspserver kernel: EXT4-fs (sda4): I/O error while writing superblock
Sep 11 02:53:17 raspserver kernel: Buffer I/O error on dev sda4, logical block 0, lost sync page write
Sep 11 02:53:17 raspserver kernel: EXT4-fs (sda4): I/O error while writing superblock
Sep 11 02:53:17 raspserver grafana-server[485]: logger=ngalert t=2022-09-11T02:53:17.724366377+02:00 level=error msg="scheduler failed to update alert rules" err="failed to get alert rules: database disk image is malformed"
Sep 11 02:53:17 raspserver kernel: Buffer I/O error on dev sda4, logical block 0, lost sync page write
Sep 11 02:53:17 raspserver kernel: EXT4-fs (sda4): I/O error while writing superblock
Sep 11 02:53:18 raspserver kernel: Buffer I/O error on dev sda4, logical block 0, lost sync page write
Sep 11 02:53:18 raspserver kernel: EXT4-fs (sda4): I/O error while writing superblock
Sep 11 02:53:18 raspserver kernel: scsi_io_completion_action: 54 callbacks suppressed
Sep 11 02:53:18 raspserver kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s
Sep 11 02:53:18 raspserver kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 d5 ce 2f 88 00 00 08 00
Sep 11 02:53:18 raspserver kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s
Sep 11 02:53:18 raspserver kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x2a 2a 00 03 c0 08 00 00 00 08 00
Sep 11 02:53:18 raspserver kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s
Sep 11 02:53:18 raspserver kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 d5 ce 2f 88 00 00 08 00
Sep 11 02:53:18 raspserver kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s
Sep 11 02:53:18 raspserver kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x2a 2a 00 03 c0 08 00 00 00 08 00
Sep 11 02:53:19 raspserver kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s
Sep 11 02:53:19 raspserver kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 d5 ce 2f 88 00 00 08 00
Sep 11 02:53:19 raspserver kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s
Sep 11 02:53:19 raspserver kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x2a 2a 00 03 c0 08 00 00 00 08 00
Sep 11 02:53:19 raspserver kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s
Sep 11 02:53:19 raspserver kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 d5 ce 2f 88 00 00 08 00
Sep 11 02:53:19 raspserver kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s
Sep 11 02:53:19 raspserver kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x2a 2a 00 03 c0 08 00 00 00 08 00
Sep 11 02:53:19 raspserver kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s
Sep 11 02:53:19 raspserver kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 d5 ce 2f 88 00 00 08 00
Sep 11 02:53:19 raspserver kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s
Sep 11 02:53:19 raspserver kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x2a 2a 00 03 c0 08 00 00 00 08 00
Sep 11 02:53:19 raspserver kernel: EXT4-fs error: 22 callbacks suppressed
Sep 11 02:53:19 raspserver kernel: EXT4-fs error (device sda4): __ext4_find_entry:1658: inode #109961253: comm rsync: reading directory lblock 0
Sep 11 02:53:20 raspserver kernel: EXT4-fs error (device sda4): __ext4_find_entry:1658: inode #109961253: comm rsync: reading directory lblock 0
Sep 11 02:53:20 raspserver kernel: EXT4-fs error (device sda4): __ext4_find_entry:1658: inode #109961253: comm rsync: reading directory lblock 0
Sep 11 02:53:20 raspserver kernel: EXT4-fs error (device sda4): __ext4_find_entry:1658: inode #109961253: comm rsync: reading directory lblock 0
Sep 11 02:53:20 raspserver kernel: EXT4-fs error (device sda4): __ext4_find_entry:1658: inode #109961253: comm rsync: reading directory lblock 0
Sep 11 02:53:20 raspserver kernel: EXT4-fs error (device sda4): __ext4_find_entry:1658: inode #109961253: comm rsync: reading directory lblock 0
Sep 11 02:53:20 raspserver kernel: blk_print_req_error: 54 callbacks suppressed
Sep 11 02:53:20 raspserver kernel: I/O error, dev sda, sector 62916608 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
Sep 11 02:53:20 raspserver kernel: I/O error, dev sda, sector 3587059592 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
Sep 11 02:53:20 raspserver kernel: EXT4-fs error (device sda4): __ext4_find_entry:1658: inode #109961253: comm rsync: reading directory lblock 0
Sep 11 02:53:20 raspserver kernel: I/O error, dev sda, sector 62916608 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
Sep 11 02:53:21 raspserver kernel: I/O error, dev sda, sector 3587059592 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
Sep 11 02:53:21 raspserver kernel: EXT4-fs error (device sda4): __ext4_find_entry:1658: inode #109961253: comm rsync: reading directory lblock 0
Sep 11 02:53:21 raspserver kernel: I/O error, dev sda, sector 62916608 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
Sep 11 02:53:21 raspserver kernel: I/O error, dev sda, sector 3587059592 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
Sep 11 02:53:14 raspserver kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s
Sep 11 02:53:14 raspserver kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x2a 2a 00 03 c0 08 00 00 00 08 00
Sep 11 02:53:14 raspserver kernel: EXT4-fs error: 22 callbacks suppressed
Sep 11 02:53:14 raspserver kernel: EXT4-fs error (device sda4): __ext4_find_entry:1658: inode #109961253: comm rsync: reading directory lblock 0
Sep 11 02:53:15 raspserver kernel: EXT4-fs error (device sda4): __ext4_find_entry:1658: inode #109961253: comm rsync: reading directory lblock 0
Sep 11 02:53:15 raspserver kernel: EXT4-fs error (device sda4): __ext4_find_entry:1658: inode #109961253: comm rsync: reading directory lblock 0
Sep 11 02:53:15 raspserver kernel: EXT4-fs error (device sda4): __ext4_find_entry:1658: inode #109961253: comm rsync: reading directory lblock 0
Sep 11 02:53:15 raspserver kernel: EXT4-fs error (device sda4): __ext4_find_entry:1658: inode #109961253: comm rsync: reading directory lblock 0
Sep 11 02:53:15 raspserver kernel: EXT4-fs error (device sda4): __ext4_find_entry:1658: inode #109961253: comm rsync: reading directory lblock 0
Sep 11 02:53:15 raspserver kernel: blk_print_req_error: 54 callbacks suppressed
Sep 11 02:53:16 raspserver kernel: I/O error, dev sda, sector 62916608 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
Sep 11 02:53:16 raspserver kernel: I/O error, dev sda, sector 3587059592 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
Sep 11 02:53:16 raspserver kernel: EXT4-fs error (device sda4): __ext4_find_entry:1658: inode #109961253: comm rsync: reading directory lblock 0
Sep 11 02:53:16 raspserver kernel: I/O error, dev sda, sector 62916608 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
Sep 11 02:53:16 raspserver kernel: I/O error, dev sda, sector 3587059592 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
Sep 11 02:53:16 raspserver kernel: EXT4-fs error (device sda4): __ext4_find_entry:1658: inode #109961253: comm rsync: reading directory lblock 0
Sep 11 02:53:16 raspserver kernel: I/O error, dev sda, sector 62916608 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
Sep 11 02:53:16 raspserver kernel: I/O error, dev sda, sector 3587059592 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
Sep 11 02:53:16 raspserver kernel: EXT4-fs error (device sda4): __ext4_find_entry:1658: inode #109961253: comm rsync: reading directory lblock 0
Sep 11 02:53:16 raspserver kernel: I/O error, dev sda, sector 62916608 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
Sep 11 02:53:16 raspserver kernel: I/O error, dev sda, sector 3587059592 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
Sep 11 02:53:16 raspserver kernel: EXT4-fs error (device sda4): __ext4_find_entry:1658: inode #109961253: comm rsync: reading directory lblock 0
Sep 11 02:53:16 raspserver kernel: I/O error, dev sda, sector 62916608 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
Sep 11 02:53:16 raspserver kernel: I/O error, dev sda, sector 3587059592 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
Sep 11 02:53:16 raspserver kernel: buffer_io_error: 22 callbacks suppressed
Sep 11 02:53:16 raspserver kernel: Buffer I/O error on dev sda4, logical block 0, lost sync page write
Sep 11 02:53:16 raspserver kernel: EXT4-fs: 22 callbacks suppressed
Sep 11 02:53:16 raspserver kernel: EXT4-fs (sda4): I/O error while writing superblock
Sep 11 02:53:16 raspserver kernel: Buffer I/O error on dev sda4, logical block 0, lost sync page write
Sep 11 02:53:16 raspserver kernel: EXT4-fs (sda4): I/O error while writing superblock
Sep 11 02:53:16 raspserver kernel: Buffer I/O error on dev sda4, logical block 0, lost sync page write
Sep 11 02:53:16 raspserver kernel: EXT4-fs (sda4): I/O error while writing superblock
Sep 11 02:53:16 raspserver kernel: Buffer I/O error on dev sda4, logical block 0, lost sync page write
Sep 11 02:53:16 raspserver kernel: EXT4-fs (sda4): I/O error while writing superblock
Sep 11 02:53:17 raspserver kernel: Buffer I/O error on dev sda4, logical block 0, lost sync page write
Sep 11 02:53:17 raspserver kernel: EXT4-fs (sda4): I/O error while writing superblock
Sep 11 02:53:17 raspserver kernel: Buffer I/O error on dev sda4, logical block 0, lost sync page write
Sep 11 02:53:17 raspserver kernel: EXT4-fs (sda4): I/O error while writing superblock
Sep 11 02:53:17 raspserver kernel: Buffer I/O error on dev sda4, logical block 0, lost sync page write
Sep 11 02:53:17 raspserver kernel: EXT4-fs (sda4): I/O error while writing superblock
Sep 11 02:53:17 raspserver kernel: Buffer I/O error on dev sda4, logical block 0, lost sync page write
Sep 11 02:53:17 raspserver kernel: EXT4-fs (sda4): I/O error while writing superblock
Sep 11 02:53:17 raspserver grafana-server[485]: logger=ngalert t=2022-09-11T02:53:17.724366377+02:00 level=error msg="scheduler failed to update alert rules" err="failed to get alert rules: database disk image is malformed"
Sep 11 02:53:17 raspserver kernel: Buffer I/O error on dev sda4, logical block 0, lost sync page write
Sep 11 02:53:17 raspserver kernel: EXT4-fs (sda4): I/O error while writing superblock
Sep 11 02:53:18 raspserver kernel: Buffer I/O error on dev sda4, logical block 0, lost sync page write
Sep 11 02:53:18 raspserver kernel: EXT4-fs (sda4): I/O error while writing superblock
Sep 11 02:53:18 raspserver kernel: scsi_io_completion_action: 54 callbacks suppressed
Sep 11 02:53:18 raspserver kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s
Sep 11 02:53:18 raspserver kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 d5 ce 2f 88 00 00 08 00
Sep 11 02:53:18 raspserver kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s
Sep 11 02:53:18 raspserver kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x2a 2a 00 03 c0 08 00 00 00 08 00
Sep 11 02:53:18 raspserver kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s
Sep 11 02:53:18 raspserver kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 d5 ce 2f 88 00 00 08 00
Sep 11 02:53:18 raspserver kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x2a 2a 00 03 c0 08 00 00 00 08 00
Sep 11 02:53:18 raspserver kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s
Sep 11 02:53:18 raspserver kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 d5 ce 2f 88 00 00 08 00
Sep 11 02:53:18 raspserver kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s
Sep 11 02:53:18 raspserver kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x2a 2a 00 03 c0 08 00 00 00 08 00
Sep 11 02:53:19 raspserver kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s
Sep 11 02:53:19 raspserver kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 d5 ce 2f 88 00 00 08 00
Sep 11 02:53:19 raspserver kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s
Sep 11 02:53:19 raspserver kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x2a 2a 00 03 c0 08 00 00 00 08 00
Sep 11 02:53:19 raspserver kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s
Sep 11 02:53:19 raspserver kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 d5 ce 2f 88 00 00 08 00
Sep 11 02:53:19 raspserver kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s
Sep 11 02:53:19 raspserver kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x2a 2a 00 03 c0 08 00 00 00 08 00
Sep 11 02:53:19 raspserver kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s
Sep 11 02:53:19 raspserver kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 d5 ce 2f 88 00 00 08 00
Sep 11 02:53:19 raspserver kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s
Sep 11 02:53:19 raspserver kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x2a 2a 00 03 c0 08 00 00 00 08 00
Sep 11 02:53:19 raspserver kernel: EXT4-fs error: 22 callbacks suppressed
Sep 11 02:53:19 raspserver kernel: EXT4-fs error (device sda4): __ext4_find_entry:1658: inode #109961253: comm rsync: reading directory lblock 0
Sep 11 02:53:20 raspserver kernel: EXT4-fs error (device sda4): __ext4_find_entry:1658: inode #109961253: comm rsync: reading directory lblock 0
Sep 11 02:53:20 raspserver kernel: EXT4-fs error (device sda4): __ext4_find_entry:1658: inode #109961253: comm rsync: reading directory lblock 0
Sep 11 02:53:20 raspserver kernel: EXT4-fs error (device sda4): __ext4_find_entry:1658: inode #109961253: comm rsync: reading directory lblock 0
Sep 11 02:53:20 raspserver kernel: EXT4-fs error (device sda4): __ext4_find_entry:1658: inode #109961253: comm rsync: reading directory lblock 0
Sep 11 02:53:20 raspserver kernel: EXT4-fs error (device sda4): __ext4_find_entry:1658: inode #109961253: comm rsync: reading directory lblock 0
Sep 11 02:53:20 raspserver kernel: blk_print_req_error: 54 callbacks suppressed
Sep 11 02:53:20 raspserver kernel: I/O error, dev sda, sector 62916608 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
Sep 11 02:53:20 raspserver kernel: I/O error, dev sda, sector 3587059592 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
Sep 11 02:53:20 raspserver kernel: EXT4-fs error (device sda4): __ext4_find_entry:1658: inode #109961253: comm rsync: reading directory lblock 0
')I think there is a driver issue with USB. Something with USB goes wrong and after that the filesystems on the USB RAID cannot be accessed. This never occured before ~August 12th. I want to identify the kernel version when it starts but that is not trivial for me. I can look what vanilla kernel was the last which worked stable - but I do not know which rpi kernel (5.15 and 5.19) corresponds to the upstream...

[df8oe]

I think I have identified the root cause. Looking at the logs sometimes pops up this warning:
$this->bbcode_second_pass_code('', 'kernel: WARN::dwc_otg_hcd_urb_dequeue:638: Timed out waiting for FSM NP transfer to complete on x')
where x is a number from 1 to 4 in my case.

Throwing this to google leads exactly to the problem I have:
https://www.google.de/search?q=kernel%3A+WARN%3A%3Adwc_otg_hcd_urb_dequeue%3A638%3A+Timed+out+waiting+for+FSM+NP+transfer+to+complete&source=hp

And that tells me that it is a problem which is not related to USB in general but to a problem that is only present if you have specific USB hardware. The time the problem rises I attached a serial to USB adaptor

[lategoodbye]

Thanks this is helpful. Please provide the output of:

$this->bbcode_second_pass_code('', 'sudo lsusb -v')

Could you please tell what is connected to the serial side of the adapter?

Is it a Raspberry Pi 3 B or a Pi 3 B+?

[df8oe]

lsusb -v is 19kb long. I provide lsusb:$this->bbcode_second_pass_code('', 'Bus 001 Device 008: ID 174c:5106 ASMedia Technology Inc. ASM1051 SATA 3Gb/s bridge
Bus 001 Device 005: ID 0bda:2838 Realtek Semiconductor Corp. RTL2838 DVB-T
Bus 001 Device 007: ID 1a86:7523 QinHeng Electronics CH340 serial converter
Bus 001 Device 006: ID 0403:6001 Future Technology Devices International, Ltd FT232 Serial (UART) IC
Bus 001 Device 004: ID 0a05:7211 Unknown Manufacturer hub
Bus 001 Device 003: ID 0424:ec00 Microchip Technology, Inc. (formerly SMSC) SMSC9512/9514 Fast Ethernet Adapter
Bus 001 Device 002: ID 0424:9514 Microchip Technology, Inc. (formerly SMSC) SMC9514 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
')The serial adapters are connected to RS232 solar hybrid inverter and the second via RS485 to a 5kwh LiFEPO4-battery bms. there is a cron job running which pulls data from these two devices every minute. But the crashes only take place when big data transfers are running to the USB RAID device. This can be locally in my own network or via rsync to my external servers (backups during the night). I have run the rsyncs for yeas without any problems. The problems popped up since I added the two serial adapters and polled data every minute. But not reproduceable. There are kernels which crash within half an hour and others which are running for days before the crash happens. I have rpi 3b+ . This is another critical message that I have in the log: $this->bbcode_second_pass_code('', 'Sep 11 11:09:12 raspserver kernel: ERROR::handle_hc_chhltd_intr_dma:2212: handle_hc_chhltd_intr_dma: Channel 0, DMA Mode -- ChHltd set, but reason for halting is unknown, hcint 0x00000002, intsts 0x06600021')I have seperated journal that is written now to RAID and not to ramdisk. If the crash occurs sometimes sd card is mounted ro and ramdisk is scattered, too so that "nothing is working". Possibly I can get further logs before / short after the crash with this extended logging...

[graysky]

Have you searched on the linux-rpi github issues: https://github.com/raspberrypi/linux/issues

If not there considering opening an issue. Since both 5.19 and 5.15 give the error, recommend you boot to the official supported version which is in the 5.15 tree and report the issue with that not with 5.19. You can mention that you get the same behavior under both kernels.