High iowait on a single core

[dictummortuum]

Can I please get some help with my high iowait on a single core? I have followed these threads, but I couldn't find anything related:
make_clickable_callback(MAGIC_URL_FULL, '
', 'https://bbs.archlinux.org/viewtopic.php?id=287343', '', ' class="postlink"')make_clickable_callback(MAGIC_URL_FULL, '
', 'https://bbs.archlinux.org/viewtopic.php?id=287658', '', ' class="postlink"')make_clickable_callback(MAGIC_URL_FULL, '
', 'https://bugzilla.kernel.org/show_bug.cgi?id=217965', '', ' class="postlink"')

What basically happened is that I restarted the server because of an update. After the update (which included a kernel bump to linux-rpi-6.12.22-1, I started getting high iowait on a single core (which changes if I reboot). I have since downgraded back to 6.6.59.

I tried investigating via top, iostat, iotop, pidstat, etc, however I couldn't find a process that waits on the disk. I tried stopping some services, but that didn't make any difference. I was seeing kthread as having a little bit high iowait on pidstat though.

Some info:

[url=https://ibb.co/ks6hPfMF][img]https://i.ibb.co/ks6hPfMF/2025-04-23-133434-1268x485-scrot.png[/img][/url]

[code]System:
Kernel: 6.6.59-2-rpi arch: aarch64 bits: 64 compiler: N/A clocksource: arch_sys_counter
parameters: coherent_pool=1M 8250.nr_uarts=0 snd_bcm2835.enable_headphones=0
cgroup_disable=memory numa_policy=interleave snd_bcm2835.enable_hdmi=0
smsc95xx.macaddr=XX:XX:XX:XX:XX:XX vc_mem.mem_base=0x3eb00000 vc_mem.mem_size=0x3ff00000
root=PARTUUID=69bbced6-02 rw rootwait console=ttyS0,115200 console=tty1 selinux=0
plymouth.enable=0 smsc95xx.turbo_mode=N dwc_otg.lpm_enable=0 kgdboc=ttyS0,115200
cgroup_enable=memory
Console: N/A Distro: Arch Linux ARM
Machine:
Type: ARM System: Raspberry Pi 4 Model B Rev 1.4 details: N/A rev: d03114 Battery:
Message: No system battery data found. Is one present?
Memory:
System RAM: total: 8 GiB available: 7.62 GiB used: 1.73 GiB (22.8%) igpu: 76 MiB
RAM Report: smbios: No SMBIOS data for dmidecode to process
PCI Slots:
Message: No ARM data found for this feature.
CPU:
Info: model: ARMv8 v8l variant: cortex-a72 bits: 64 type: MCP arch: v8l family: 8 model-id: 0
stepping: 3
Topology: cpus: 1x cores: 4 smt: cache: L1: 320 KiB desc: d-4x32 KiB! i-4x48 KiB
L2: 1024 KiB desc: 1x1024 KiB
Speed (MHz): avg: 1500 min/max: 600/1800 scaling: driver: cpufreq-dt governor: schedutil cores:
1: 1500 2: 1500 3: 1500 4: 1500 bogomips: 432
Features: asimd cpuid crc32 evtstrm fp
Vulnerabilities:
Type: gather_data_sampling status: Not affected
Type: itlb_multihit status: Not affected
Type: l1tf status: Not affected
Type: mds status: Not affected
Type: meltdown status: Not affected
Type: mmio_stale_data status: Not affected
Type: reg_file_data_sampling status: Not affected
Type: retbleed status: Not affected
Type: spec_rstack_overflow status: Not affected
Type: spec_store_bypass status: Vulnerable
Type: spectre_v1 mitigation: __user pointer sanitization
Type: spectre_v2 status: Vulnerable
Type: srbds status: Not affected
Type: tsx_async_abort status: Not affected
Graphics:
Device-1: bcm2711-hdmi0 driver: vc4_hdmi v: N/A bus-ID: N/A chip-ID: brcm:fef00700 class-ID: hdmi
Device-2: bcm2711-hdmi1 driver: vc4_hdmi v: N/A bus-ID: N/A chip-ID: brcm:fef05700
class-ID: hdmi
Device-3: bcm2711-vc5 driver: vc4_drm v: N/A bus-ID: N/A chip-ID: brcm:gpu class-ID: gpu
Display: unspecified server: N/A driver: N/A tty: 80x40
API: N/A Message: No API data available in console. Headless machine?
Audio:
Device-1: bcm2711-hdmi0 driver: vc4_hdmi bus-ID: N/A chip-ID: brcm:fef00700 class-ID: hdmi
Device-2: bcm2711-hdmi1 driver: vc4_hdmi bus-ID: N/A chip-ID: brcm:fef05700 class-ID: hdmi
API: ALSA v: k6.6.59-2-rpi status: kernel-api tools: N/A
Server-1: JACK v: 1.9.22 status: off tools: N/A
Network:
Device-1: bcm2835-mmc driver: mmc_bcm2835 v: N/A port: N/A bus-ID: N/A chip-ID: brcm:fe300000
class-ID: mmcnr
IF: wlan0 state: down mac:
Device-2: bcm2711-genet-v5 driver: bcmgenet v: N/A port: N/A bus-ID: N/A chip-ID: brcm:fd580000
class-ID: ethernet
IF: end0 state: up speed: 1000 Mbps duplex: full mac:
IP v4: type: dynamic scope: global
IP v6: virtual: proto kernel_ll scope: link
IF-ID-1: br-4edbf5775554 state: down mac:
IP v4: scope: global broadcast:
IF-ID-2: br-5f47a6ca28a9 state: down mac:
IP v4: scope: global broadcast:
IF-ID-3: br-6950c48226da state: up speed: 10000 Mbps duplex: unknown mac:
IP v4: scope: global broadcast:
IP v6: virtual: proto kernel_ll scope: link
IF-ID-4: br-9d20ab664bf8 state: down mac:
IP v4: scope: global broadcast:
IF-ID-5: br-b17848ae88a0 state: down mac:
IP v4: scope: global broadcast:
IF-ID-6: br-cf6c47fe5d07 state: down mac:
IP v4: scope: global broadcast:
IF-ID-7: br-d036090619dd state: up speed: 10000 Mbps duplex: unknown mac:
IP v4: scope: global broadcast:
IP v6: virtual: proto kernel_ll scope: link
IF-ID-8: br-d3234ad267c0 state: up speed: 10000 Mbps duplex: unknown mac:
IP v4: scope: global broadcast:
IP v6: virtual: proto kernel_ll scope: link
IF-ID-9: br-d747488a22a3 state: up speed: 10000 Mbps duplex: unknown mac:
IP v4: scope: global broadcast:
IP v6: virtual: proto kernel_ll scope: link
IF-ID-10: br-f722e1981fea state: up speed: 10000 Mbps duplex: unknown mac:
IP v4: scope: global broadcast:
IP v6: virtual: proto kernel_ll scope: link
IF-ID-11: br-ff207bf69f14 state: up speed: 10000 Mbps duplex: unknown mac:
IP v4: scope: global broadcast:
IP v6: virtual: proto kernel_ll scope: link
IF-ID-12: docker0 state: down mac:
IP v4: scope: global broadcast:
IF-ID-13: veth14d912e state: up speed: 10000 Mbps duplex: full mac:
IF-ID-14: veth6b1f42e state: up speed: 10000 Mbps duplex: full mac:
IF-ID-15: veth7c19ddb state: up speed: 10000 Mbps duplex: full mac:
IF-ID-16: veth87a7ba2 state: up speed: 10000 Mbps duplex: full mac:
IF-ID-17: veth94bd624 state: up speed: 10000 Mbps duplex: full mac:
IF-ID-18: vethf7e1185 state: up speed: 10000 Mbps duplex: full mac:
IF-ID-19: zt0 state: unknown speed: 10000 Mbps duplex: full mac:
IP v4: scope: global broadcast:
IP v6: virtual: proto kernel_ll scope: link
WAN IP:
Bluetooth:
Device-1: pl011 driver: uart_pl011 bus-ID: N/A chip-ID: arm:fe201000 class-ID: serial
Report: rfkill ID: hci0 rfk-id: 0 state: down bt-service: not found rfk-block: hardware: no
software: no address: see --recommends
Device-2: pl011 driver: N/A bus-ID: N/A chip-ID: arm:serial0 class-ID: serial
Report: This feature requires one of these tools: hciconfig/bt-adapter
Logical:
Message: No logical block device data found.
RAID:
Message: No RAID data found.
Drives:
Local Storage: total: 223.57 GiB used: 147.12 GiB (65.8%)
ID-1: /dev/sda maj-min: 8:0 vendor: Kingston model: SA400S37240G family: Driven SSDs
size: 223.57 GiB block-size: physical: 512 B logical: 512 B type: USB rev: 3.1 spd: 5 Gb/s
lanes: 1 mode: 3.2 gen-1x1 sata: 3.2 speed: 6.0 Gb/s tech: SSD fw-rev: S3700100 temp: 39 C scheme: MBR
SMART: yes state: enabled health: PASSED on: 3y 356d 5h cycles: 233 read: 1269 GiB
written: 6907 GiB
Message: No optical or floppy data found.
Partition:
ID-1: / raw-size: 50 GiB size: 48.91 GiB (97.83%) used: 34.96 GiB (71.5%) fs: ext4
block-size: 4096 B dev: /dev/sda2 maj-min: 8:2 label: N/A
uuid: 285bad0b-bbe4-4c02-8da6-ffabf21b60a5
ID-2: /boot raw-size: 256 MiB size: 255.7 MiB (99.88%) used: 58.9 MiB (23.0%) fs: vfat
block-size: 512 B dev: /dev/sda1 maj-min: 8:1 label: N/A uuid: 7B6C-9C4E
ID-3: /data raw-size: 173.29 GiB size: 169.51 GiB (97.82%) used: 112.11 GiB (66.1%) fs: ext4
block-size: 4096 B dev: /dev/sda3 maj-min: 8:3 label: N/A
uuid: 261dd0aa-0500-49b4-ab57-638ef59e0156
ID-4: /data/nextcloud raw-size: N/A size: 3.48 TiB used: 2.78 TiB (79.8%) fs: nfs
remote: synology.dictummortuum.com:/volume1/nextcloud
Swap:
Alert: No swap data was found.
Unmounted:
Message: No unmounted partitions found.
USB:
Hub-1: 1-0:1 info: hi-speed hub with single TT ports: 1 rev: 2.0 speed: 480 Mb/s (57.2 MiB/s)
lanes: 1 mode: 2.0 chip-ID: 1d6b:0002 class-ID: 0900
Hub-2: 1-1:2 info: VIA Labs Hub ports: 4 rev: 2.1 speed: 480 Mb/s (57.2 MiB/s) lanes: 1
mode: 2.0 power: 100mA chip-ID: 2109:3431 class-ID: 0900
Hub-3: 2-0:1 info: super-speed hub ports: 4 rev: 3.0 speed: 5 Gb/s (596.0 MiB/s) lanes: 1
mode: 3.2 gen-1x1 chip-ID: 1d6b:0003 class-ID: 0900
Device-1: 2-1:2 info: ASMedia Ugreen Storage Device type: mass storage driver: uas
interfaces: 1 rev: 3.1 speed: 5 Gb/s (596.0 MiB/s) lanes: 1 mode: 3.2 gen-1x1 chip-ID: 174c:235c
class-ID: 0806 Sensors:
System Temperatures: cpu: 50.6 C mobo: N/A
Fan Speeds (rpm): N/A
Repos:
Packages: pm: pacman pkgs: 680 libs: 171 tools: yay
Active pacman repo servers in: /etc/pacman.conf
1: http: //pacman.dictummortuum.com/$arch
Active pacman repo servers in: /etc/pacman.d/mirrorlist
1: http: //mirror.archlinuxarm.org/$arch/$repo
2: http: //de.mirror.archlinuxarm.org/$arch/$repo
3: http: //de4.mirror.archlinuxarm.org/$arch/$repo
Processes:
CPU top: 5 of 290
1: cpu: 300% command: ps pid: 44649 mem: 3.62 MiB (0.0%)
2: cpu: 16.6% command: -zsh pid: 44594 mem: 6.98 MiB (0.0%)
3: cpu: 5.2% command: sshd-session: pid: 44591 mem: 11.0 MiB (0.1%)
4: cpu: 3.8% command: htop pid: 35131 mem: 6.38 MiB (0.0%)
5: cpu: 1.6% command: -zsh pid: 44617 mem: 5.07 MiB (0.0%)
Memory top: 5 of 290
1: mem: 328.4 MiB (4.2%) command: gitea pid: 874 cpu: 0.6%
2: mem: 283.2 MiB (3.6%) command: mariadbd pid: 34337 cpu: 0.6%
3: mem: 199.7 MiB (2.5%) command: prometheus pid: 680 cpu: 1.2%
4: mem: 170.8 MiB (2.1%) command: java pid: 8120 cpu: 0.5%
5: mem: 155.8 MiB (1.9%) command: grafana pid: 875 cpu: 0.5%
Info:
Processes: 290 Uptime: 3h 31m Init: systemd v: 257 default: graphical tool: systemctl Compilers:
gcc: 14.2.1 Client: shell wrapper v: 5.2.37-release inxi: 3.3.31[/code]

[code]
❯ sudo iostat
Linux 6.6.59-2-rpi (sol) 04/23/2025 _aarch64_ (4 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
3.22 0.01 1.92 23.82 0.00 71.03

Device tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd
sda 26.78 307.84 158.57 5573.44 4194752 2160733 75947076
[/code]

[code]
❯ dmesg | grep -iE 'sda|usb|uas|error|reset'

[ 0.658047] usbcore: registered new interface driver usbfs
[ 0.658066] usbcore: registered new interface driver hub
[ 0.658086] usbcore: registered new device driver usb
[ 1.035865] usbcore: registered new device driver r8152-cfgselector
[ 1.035885] usbcore: registered new interface driver r8152
[ 1.035901] usbcore: registered new interface driver lan78xx
[ 1.035915] usbcore: registered new interface driver smsc95xx
[ 1.127295] xhci_hcd 0000:01:00.0: new USB bus registered, assigned bus number 1
[ 1.128539] xhci_hcd 0000:01:00.0: new USB bus registered, assigned bus number 2
[ 1.128550] xhci_hcd 0000:01:00.0: Host supports USB 3.0 SuperSpeed
[ 1.128672] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.06
[ 1.128682] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 1.128690] usb usb1: Product: xHCI Host Controller
[ 1.128696] usb usb1: Manufacturer: Linux 6.6.59-2-rpi xhci-hcd
[ 1.128702] usb usb1: SerialNumber: 0000:01:00.0
[ 1.128950] hub 1-0:1.0: USB hub found
[ 1.129194] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 6.06
[ 1.129204] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 1.129212] usb usb2: Product: xHCI Host Controller
[ 1.129218] usb usb2: Manufacturer: Linux 6.6.59-2-rpi xhci-hcd
[ 1.129223] usb usb2: SerialNumber: 0000:01:00.0
[ 1.129390] hub 2-0:1.0: USB hub found
[ 1.130102] usbcore: registered new interface driver uas
[ 1.130132] usbcore: registered new interface driver usb-storage
[ 1.135970] usbcore: registered new interface driver usbhid
[ 1.135977] usbhid: USB HID core driver
[ 1.383514] usb 1-1: new high-speed USB device number 2 using xhci_hcd
[ 1.534151] usb 1-1: New USB device found, idVendor=2109, idProduct=3431, bcdDevice= 4.21
[ 1.534183] usb 1-1: New USB device strings: Mfr=0, Product=1, SerialNumber=0
[ 1.534192] usb 1-1: Product: USB2.0 Hub
[ 1.535916] hub 1-1:1.0: USB hub found
[ 1.659703] usb 2-1: new SuperSpeed USB device number 2 using xhci_hcd
[ 1.680791] usb 2-1: New USB device found, idVendor=174c, idProduct=235c, bcdDevice= 1.00
[ 1.680803] usb 2-1: New USB device strings: Mfr=2, Product=3, SerialNumber=1
[ 1.680811] usb 2-1: Product: Ugreen Storage Device
[ 1.680817] usb 2-1: Manufacturer: Ugreen
[ 1.680822] usb 2-1: SerialNumber: 20D11E804DBA
[ 1.693833] scsi host0: uas
[ 1.697532] sd 0:0:0:0: [sda] 468862128 512-byte logical blocks: (240 GB/224 GiB)
[ 1.697718] sd 0:0:0:0: [sda] Write Protect is off
[ 1.697730] sd 0:0:0:0: [sda] Mode Sense: 43 00 00 00
[ 1.697925] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 1.748139] sd 0:0:0:0: [sda] Preferred minimum I/O size 512 bytes
[ 1.748168] sd 0:0:0:0: [sda] Optimal transfer size 33553920 bytes
[ 1.750777] sda: sda1 sda2 sda3
[ 1.751135] sd 0:0:0:0: [sda] Attached SCSI disk
[ 2.000067] EXT4-fs (sda2): mounted filesystem 285bad0b-bbe4-4c02-8da6-ffabf21b60a5 r/w with ordered data mode. Quota mode: none.
[ 3.072004] EXT4-fs (sda2): re-mounted 285bad0b-bbe4-4c02-8da6-ffabf21b60a5 r/w. Quota mode: none.
[ 4.218693] usbcore: registered new interface driver brcmfmac
[ 4.543481] EXT4-fs (sda3): mounted filesystem 261dd0aa-0500-49b4-ab57-638ef59e0156 r/w with ordered data mode. Quota mode: none.
[ 2840.616943] EXT4-fs (sda3): re-mounted 261dd0aa-0500-49b4-ab57-638ef59e0156 r/w. Quota mode: none.
[ 3033.758466] EXT4-fs (sda2): re-mounted 285bad0b-bbe4-4c02-8da6-ffabf21b60a5 r/w. Quota mode: none.
[12677.723451] block sda: the capability attribute has been deprecated.
[/code]

[code]
❯ uname -a
Linux sol 6.6.59-2-rpi #1 SMP PREEMPT Sat Nov 2 06:14:11 MDT 2024 aarch64 GNU/Linux
[/code]

[code]
01:45:55 PM 0 22 0.00 0.01 0.00 0.02 0.01 1 ksoftirqd/1
01:45:55 PM 0 32 0.00 0.01 0.00 0.02 0.01 3 ksoftirqd/3
01:45:55 PM 33 850 0.06 0.04 0.00 0.02 0.09 1 nginx
01:45:55 PM 943 671 0.05 0.07 0.00 0.02 0.11 2 sssd_pam
01:45:55 PM 0 295 0.05 0.06 0.00 0.02 0.11 3 systemd-journal
01:45:55 PM 0 16 0.00 0.02 0.00 0.05 0.02 0 ksoftirqd/0
01:45:55 PM 970 505 0.41 0.23 0.00 0.05 0.63 0 redis-server
01:45:55 PM 0 17 0.02 0.09 0.00 0.12 0.11 3 rcu_preempt
01:45:55 PM UID PID %usr %system %guest %wait %CPU CPU Command
[/code]