odroid-c2 fails proper boot after systemd update

[mcloaked]

This evening I updated my odroid-c2 (running stock kernel) and after rebooting following the update to systemd 236.0 I get a failure to start the systemd-sysctl service:

$this->bbcode_second_pass_code('', '
[root@odroid-c2 ~]# systemctl status systemd-sysctl
* systemd-sysctl.service - Apply Kernel Variables
Loaded: loaded (/usr/lib/systemd/system/systemd-sysctl.service; static; vendor preset: disabled)
Active: failed (Result: exit-code) since Thu 2017-12-21 13:33:56 UTC; 5h 16min ago
Docs: man:systemd-sysctl.service(8)
man:sysctl.d(5)
Process: 203 ExecStart=/usr/lib/systemd/systemd-sysctl (code=exited, status=1/FAILURE)
Main PID: 203 (code=exited, status=1/FAILURE)

Dec 21 13:33:56 odroid-c2 systemd-sysctl[203]: Couldn't write '1' to 'net/ipv4/conf/all/rp_filter': Input/output er
Dec 21 13:33:56 odroid-c2 systemd-sysctl[203]: Couldn't write '0' to 'net/ipv4/conf/all/accept_source_route': Input
Dec 21 13:33:56 odroid-c2 systemd-sysctl[203]: Couldn't write '1' to 'net/ipv4/conf/all/promote_secondaries': Input
Dec 21 13:33:56 odroid-c2 systemd-sysctl[203]: Couldn't write 'fq_codel' to 'net/core/default_qdisc': Input/output
Dec 21 13:33:56 odroid-c2 systemd-sysctl[203]: Couldn't write '1' to 'fs/protected_hardlinks': Input/output error
Dec 21 13:33:56 odroid-c2 systemd-sysctl[203]: Couldn't write '1' to 'fs/protected_symlinks': Input/output error
Dec 21 13:33:56 odroid-c2 systemd-sysctl[203]: Couldn't write '100000' to 'fs/inotify/max_user_watches': Input/outp
Dec 21 13:33:56 odroid-c2 systemd[1]: systemd-sysctl.service: Main process exited, code=exited, status=1/FAILURE
Dec 21 13:33:56 odroid-c2 systemd[1]: systemd-sysctl.service: Failed with result 'exit-code'.
Dec 21 13:33:56 odroid-c2 systemd[1]: Failed to start Apply Kernel Variables.
...skipping...
* systemd-sysctl.service - Apply Kernel Variables
Loaded: loaded (/usr/lib/systemd/system/systemd-sysctl.service; static; vendor preset: disabled)
Active: failed (Result: exit-code) since Thu 2017-12-21 13:33:56 UTC; 5h 16min ago
Docs: man:systemd-sysctl.service(8)
man:sysctl.d(5)
Process: 203 ExecStart=/usr/lib/systemd/systemd-sysctl (code=exited, status=1/FAILURE)
Main PID: 203 (code=exited, status=1/FAILURE)
')

Does anyone know how to work around this problem? The system does boot and has the sshd server running so I can at least ssh in and presumably make changes but I have no idea what changes will sort this out?

Any help appreciated.

Edit: I downgraded systemd:

$this->bbcode_second_pass_code('', '
[root@odroid-c2 ~]# pacman -U /var/cache/pacman/pkg/libsystemd-235.38-4-aarch64.pkg.tar.xz /var/cache/pacman/pkg/systemd-235.38-4-aarch64.pkg.tar.xz /var/cache/pacman/pkg/systemd-sysvcompat-235.38-4-aarch64.pkg.tar.xz
loading packages...
warning: downgrading package libsystemd (236.0-2 => 235.38-4)
warning: downgrading package systemd (236.0-2 => 235.38-4)
warning: downgrading package systemd-sysvcompat (236.0-2 => 235.38-4)
')

The system rebooted but the timesyncd service fails now:

$this->bbcode_second_pass_code('', '
[root@odroid-c2 ~]# systemctl status systemd-timesyncd
* systemd-timesyncd.service - Network Time Synchronization
Loaded: loaded (/usr/lib/systemd/system/systemd-timesyncd.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2017-12-13 13:42:04 UTC; 18s ago
Docs: man:systemd-timesyncd.service(8)
Process: 313 ExecStart=/usr/lib/systemd/systemd-timesyncd (code=exited, status=238/STATE_DIRECTORY)
Main PID: 313 (code=exited, status=238/STATE_DIRECTORY)

Dec 13 13:42:04 odroid-c2 systemd[1]: systemd-timesyncd.service: Failed with result 'exit-code'.
Dec 13 13:42:04 odroid-c2 systemd[1]: Failed to start Network Time Synchronization.
Dec 13 13:42:04 odroid-c2 systemd[1]: systemd-timesyncd.service: Service has no hold-off time, scheduling restart.
Dec 13 13:42:04 odroid-c2 systemd[1]: systemd-timesyncd.service: Scheduled restart job, restart counter is at 5.
Dec 13 13:42:04 odroid-c2 systemd[1]: Stopped Network Time Synchronization.
Dec 13 13:42:04 odroid-c2 systemd[1]: systemd-timesyncd.service: Start request repeated too quickly.
Dec 13 13:42:04 odroid-c2 systemd[1]: systemd-timesyncd.service: Failed with result 'exit-code'.
Dec 13 13:42:04 odroid-c2 systemd[1]: Failed to start Network Time Synchronization.
')

I can manually correct the system clock with hwclock hctosys but the timesyncd daemon still won't start without failure.

Edit: I can load the parameters manually, after updating back to systemd 236.0 with:

$this->bbcode_second_pass_code('', '
[root@odroid-c2 ~]# sysctl --system
* Applying /usr/lib/sysctl.d/50-coredump.conf ...
kernel.core_pattern = |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %e
* Applying /usr/lib/sysctl.d/50-default.conf ...
kernel.sysrq = 16
kernel.core_uses_pid = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.all.promote_secondaries = 1
net.core.default_qdisc = fq_codel
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
* Applying /etc/sysctl.d/90-inotify.conf ...
fs.inotify.max_user_watches = 100000
')

So I guess there is a code bug or config bug somewhere?

[mcloaked]

On the affected system I don't know if this is pointing to something that gives an idea where to look for a solution:

$this->bbcode_second_pass_code('', '
[root@odroid-c2 ~]# journalctl _PID=301 | grep systemd
Dec 13 13:42:04 odroid-c2 systemd[301]: systemd-timesyncd.service: Failed to set up special execution directory in /var/lib: Too many levels of symbolic links
Dec 13 13:42:04 odroid-c2 systemd[301]: systemd-timesyncd.service: Failed at step STATE_DIRECTORY spawning /usr/lib/systemd/systemd-timesyncd: Too many levels of symbolic links
')