promtail journal not wokring aarch64 / rpi4

[angellusmortis]

I am getting the following error when I trying to use a `journal` scrape job for `promtail`. Is everything built correctly for `promtail` or is this possibly an upstream issue with aarch64?

Cross-posted Github issue to promtail in case it is upstream: https://github.com/grafana/loki/issues/3522

$this->bbcode_second_pass_code('', '
Mar 20 15:29:16 alarm promtail[243566]: level=info ts=2021-03-20T19:29:16.163114189Z caller=main.go:110 msg="Starting Promtail" version="(version=2.1.0, branch=, revision=)"
Mar 20 15:29:17 alarm promtail[243566]: unexpected fault address 0x6c62767862ea9f
Mar 20 15:29:17 alarm promtail[243566]: fatal error: fault
Mar 20 15:29:17 alarm promtail[243566]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x6c62767862ea9f pc=0xaaaaaec746d0]
Mar 20 15:29:17 alarm promtail[243566]: goroutine 225 [running]:
Mar 20 15:29:17 alarm promtail[243566]: runtime.throw(0xaaaaaffa7175, 0x5)
Mar 20 15:29:17 alarm promtail[243566]: runtime/panic.go:1117 +0x54 fp=0x40008f38d0 sp=0x40008f38a0 pc=0xaaaaae848b24
Mar 20 15:29:17 alarm promtail[243566]: runtime.sigpanic()
Mar 20 15:29:17 alarm promtail[243566]: runtime/signal_unix.go:741 +0x230 fp=0x40008f3910 sp=0x40008f38d0 pc=0xaaaaae8603a0
Mar 20 15:29:17 alarm promtail[243566]: github.com/golang/snappy.encodeBlock(0x4000f80003, 0x6b614, 0x6b614, 0x4000db8000, 0x10000, 0x5c08b, 0x400089c138)
Mar 20 15:29:17 alarm promtail[243566]: github.com/golang/snappy@v0.0.2/encode_arm64.s:666 +0x360 fp=0x40008fb9b0 sp=0x40008f3920 pc=0xaaaaaec746d0
Mar 20 15:29:17 alarm promtail[243566]: github.com/golang/snappy.Encode(0x4000f80000, 0x6b617, 0x6b617, 0x4000dc8000, 0x4c08b, 0x4c08b, 0x0, 0xaaaaaea2075c, 0x40009a48f0)
Mar 20 15:29:17 alarm promtail[243566]: github.com/golang/snappy@v0.0.2/encode.go:39 +0x17c fp=0x40008fba00 sp=0x40008fb9b0 pc=0xaaaaaec7336c
Mar 20 15:29:17 alarm promtail[243566]: github.com/grafana/loki/pkg/promtail/client.(*batch).encode(0x400095e3c0, 0x1, 0x10, 0x4c, 0x4000af6730, 0x40009a48f0, 0x400007dbd8)
Mar 20 15:29:17 alarm promtail[243566]: github.com/grafana/loki/pkg/promtail/client/batch.go:79 +0x94 fp=0x40008fba60 sp=0x40008fba00 pc=0xaaaaaefaecc4
Mar 20 15:29:17 alarm promtail[243566]: github.com/grafana/loki/pkg/promtail/client.(*client).sendBatch(0x400044c900, 0x0, 0x0, 0x400095e3c0)
Mar 20 15:29:17 alarm promtail[243566]: github.com/grafana/loki/pkg/promtail/client/client.go:241 +0x30 fp=0x40008fbc90 sp=0x40008fba60 pc=0xaaaaaefafe40
Mar 20 15:29:17 alarm promtail[243566]: github.com/grafana/loki/pkg/promtail/client.(*client).run(0x400044c900)
Mar 20 15:29:17 alarm promtail[243566]: github.com/grafana/loki/pkg/promtail/client/client.go:233 +0x46c fp=0x40008fbfd0 sp=0x40008fbc90 pc=0xaaaaaefafdac
Mar 20 15:29:17 alarm promtail[243566]: runtime.goexit()
Mar 20 15:29:17 alarm promtail[243566]: runtime/asm_arm64.s:1130 +0x4 fp=0x40008fbfd0 sp=0x40008fbfd0 pc=0xaaaaae87fda4
Mar 20 15:29:17 alarm promtail[243566]: created by github.com/grafana/loki/pkg/promtail/client.New
Mar 20 15:29:17 alarm promtail[243566]: github.com/grafana/loki/pkg/promtail/client/client.go:170 +0x320
Mar 20 15:29:17 alarm promtail[243566]: goroutine 1 [chan receive]:
Mar 20 15:29:17 alarm promtail[243566]: github.com/grafana/loki/pkg/promtail/server.(*noopServer).Run(0x40009bc2b8, 0x6, 0x400098c3c0)
Mar 20 15:29:17 alarm promtail[243566]: github.com/grafana/loki/pkg/promtail/server/server.go:260 +0x8c
Mar 20 15:29:17 alarm promtail[243566]: github.com/grafana/loki/pkg/promtail.(*Promtail).Run(0x40003569b0, 0x4000998b00, 0x4)
Mar 20 15:29:17 alarm promtail[243566]: github.com/grafana/loki/pkg/promtail/promtail.go:98 +0xe8
Mar 20 15:29:17 alarm promtail[243566]: main.main()
Mar 20 15:29:17 alarm promtail[243566]: github.com/grafana/loki/cmd/promtail/main.go:113 +0x48c
Mar 20 15:29:17 alarm promtail[243566]: goroutine 19 [chan receive]:
Mar 20 15:29:17 alarm promtail[243566]: k8s.io/klog.(*loggingT).flushDaemon(0xaaaab1a35920)
Mar 20 15:29:17 alarm promtail[243566]: k8s.io/klog@v1.0.0/klog.go:1010 +0x70
Mar 20 15:29:17 alarm promtail[243566]: created by k8s.io/klog.init.0
Mar 20 15:29:17 alarm promtail[243566]: k8s.io/klog@v1.0.0/klog.go:411 +0x11c
Mar 20 15:29:17 alarm promtail[243566]: goroutine 69 [select]:
Mar 20 15:29:17 alarm promtail[243566]: go.opencensus.io/stats/view.(*worker).start(0x400049e100)
Mar 20 15:29:17 alarm promtail[243566]: go.opencensus.io@v0.22.4/stats/view/worker.go:276 +0x98
Mar 20 15:29:17 alarm promtail[243566]: created by go.opencensus.io/stats/view.init.0
Mar 20 15:29:17 alarm promtail[243566]: go.opencensus.io@v0.22.4/stats/view/worker.go:34 +0x70
Mar 20 15:29:17 alarm promtail[243566]: goroutine 70 [chan receive]:
Mar 20 15:29:17 alarm promtail[243566]: k8s.io/klog/v2.(*loggingT).flushDaemon(0xaaaab1a35a00)
Mar 20 15:29:17 alarm promtail[243566]: k8s.io/klog/v2@v2.3.0/klog.go:1131 +0x70
Mar 20 15:29:17 alarm promtail[243566]: created by k8s.io/klog/v2.init.0
Mar 20 15:29:17 alarm promtail[243566]: k8s.io/klog/v2@v2.3.0/klog.go:416 +0x11c
Mar 20 15:29:17 alarm promtail[243566]: goroutine 148 [chan receive]:
Mar 20 15:29:17 alarm promtail[243566]: go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0x4000111728)
Mar 20 15:29:17 alarm promtail[243566]: go.etcd.io/etcd@v0.5.0-alpha.5.0.20200520232829-54ba9589114f/pkg/logutil/merge_logger.go:173 +0x2dc
Mar 20 15:29:17 alarm promtail[243566]: created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger
Mar 20 15:29:17 alarm promtail[243566]: go.etcd.io/etcd@v0.5.0-alpha.5.0.20200520232829-54ba9589114f/pkg/logutil/merge_logger.go:91 +0x7c
Mar 20 15:29:17 alarm promtail[243566]: goroutine 187 [chan receive]:
Mar 20 15:29:17 alarm promtail[243566]: go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0x400000e840)
Mar 20 15:29:17 alarm promtail[243566]: go.etcd.io/etcd@v0.5.0-alpha.5.0.20200520232829-54ba9589114f/pkg/logutil/merge_logger.go:173 +0x2dc
Mar 20 15:29:17 alarm promtail[243566]: created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger
Mar 20 15:29:17 alarm promtail[243566]: go.etcd.io/etcd@v0.5.0-alpha.5.0.20200520232829-54ba9589114f/pkg/logutil/merge_logger.go:91 +0x7c
Mar 20 15:29:17 alarm promtail[243566]: goroutine 190 [chan receive]:
Mar 20 15:29:17 alarm promtail[243566]: go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0x400000eab0)
Mar 20 15:29:17 alarm promtail[243566]: go.etcd.io/etcd@v0.5.0-alpha.5.0.20200520232829-54ba9589114f/pkg/logutil/merge_logger.go:173 +0x2dc
Mar 20 15:29:17 alarm promtail[243566]: created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger
Mar 20 15:29:17 alarm promtail[243566]: go.etcd.io/etcd@v0.5.0-alpha.5.0.20200520232829-54ba9589114f/pkg/logutil/merge_logger.go:91 +0x7c
Mar 20 15:29:17 alarm promtail[243566]: goroutine 258 [select]:
Mar 20 15:29:17 alarm promtail[243566]: github.com/grafana/loki/pkg/promtail/positions.(*positions).run(0x4000356a00)
Mar 20 15:29:17 alarm promtail[243566]: github.com/grafana/loki/pkg/promtail/positions/positions.go:153 +0xbc
Mar 20 15:29:17 alarm promtail[243566]: created by github.com/grafana/loki/pkg/promtail/positions.New
Mar 20 15:29:17 alarm promtail[243566]: github.com/grafana/loki/pkg/promtail/positions/positions.go:95 +0x144
Mar 20 15:29:17 alarm promtail[243566]: goroutine 263 [select]:
Mar 20 15:29:17 alarm promtail[243566]: github.com/coreos/go-systemd/sdjournal.(*JournalReader).Follow(0x40009bc288, 0x4000984300, 0xaaaab0b1aaf8, 0xaaaab1a64c00, 0x0, 0x0)
Mar 20 15:29:17 alarm promtail[243566]: github.com/coreos/go-systemd@v0.0.0-20191104093116-d3cd4ed1dbcf/sdjournal/read.go:238 +0x130
Mar 20 15:29:17 alarm promtail[243566]: github.com/grafana/loki/pkg/promtail/targets/journal.journalTargetWithReader.func1(0x40001ae400, 0x4000984300)
Mar 20 15:29:17 alarm promtail[243566]: github.com/grafana/loki/pkg/promtail/targets/journal/journaltarget.go:184 +0x54
Mar 20 15:29:17 alarm promtail[243566]: created by github.com/grafana/loki/pkg/promtail/targets/journal.journalTargetWithReader
Mar 20 15:29:17 alarm promtail[243566]: github.com/grafana/loki/pkg/promtail/targets/journal/journaltarget.go:182 +0x310
Mar 20 15:29:17 alarm promtail[243566]: goroutine 275 [syscall]:
Mar 20 15:29:17 alarm promtail[243566]: os/signal.signal_recv(0x0)
Mar 20 15:29:17 alarm promtail[243566]: runtime/sigqueue.go:168 +0xcc
Mar 20 15:29:17 alarm promtail[243566]: os/signal.loop()
Mar 20 15:29:17 alarm promtail[243566]: os/signal/signal_unix.go:23 +0x20
Mar 20 15:29:17 alarm promtail[243566]: created by os/signal.Notify.func1.1
Mar 20 15:29:17 alarm promtail[243566]: os/signal/signal.go:151 +0x44
Mar 20 15:29:17 alarm promtail[243566]: goroutine 203 [syscall]:
Mar 20 15:29:17 alarm promtail[243566]: github.com/coreos/go-systemd/sdjournal._Cfunc_my_sd_journal_wait(0xffff5af0eb80, 0xaaaadb8b6200, 0x186a0, 0x0)
Mar 20 15:29:17 alarm promtail[243566]: _cgo_gotypes.go:554 +0x44
Mar 20 15:29:17 alarm promtail[243566]: github.com/coreos/go-systemd/sdjournal.(*Journal).Wait.func1(0xffff5af0eb80, 0x40001b47d0, 0x186a0, 0x0)
Mar 20 15:29:17 alarm promtail[243566]: github.com/coreos/go-systemd@v0.0.0-20191104093116-d3cd4ed1dbcf/sdjournal/journal.go:1014 +0x80
Mar 20 15:29:17 alarm promtail[243566]: github.com/coreos/go-systemd/sdjournal.(*Journal).Wait(0x40001b47d0, 0x5f5e100, 0x0)
Mar 20 15:29:17 alarm promtail[243566]: github.com/coreos/go-systemd@v0.0.0-20191104093116-d3cd4ed1dbcf/sdjournal/journal.go:1014 +0xa8
Mar 20 15:29:17 alarm promtail[243566]: github.com/coreos/go-systemd/sdjournal.(*JournalReader).Follow.func1(0x40009bc288, 0x4000982150, 0x40001b4830)
Mar 20 15:29:17 alarm promtail[243566]: github.com/coreos/go-systemd@v0.0.0-20191104093116-d3cd4ed1dbcf/sdjournal/read.go:233 +0x38
Mar 20 15:29:17 alarm promtail[243566]: created by github.com/coreos/go-systemd/sdjournal.(*JournalReader).Follow
Mar 20 15:29:17 alarm promtail[243566]: github.com/coreos/go-systemd@v0.0.0-20191104093116-d3cd4ed1dbcf/sdjournal/read.go:232 +0xe8
Mar 20 15:29:17 alarm systemd[1]: promtail.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
')

[angellusmortis]

Ugh. Sorry about this, it is _not_ an issue with archlinuxarm. It is an issue with Manjaro ARM. I have multiple RPIs and the older ones run Manjaro ARM since archlinuxarm did not support aarach64 yet for RPI4 when I originally set them up. My archlinuxarm RPI works, my Manjaro ARM RPI does not.

EDIT: Also sorry for the double post, this should have been edited into the OP.