syzbot: ci-openbsd unable to start syz-managers

[a-nogikh]

For some reason, the smoke test fails on freshly built OpenBSD kernels.

2024/12/09 04:30:18 [ERROR] ci-openbsd: ci-openbsd-setuid: manager exited unexpectedly: exit status 1
2024/12/09 04:30:37 [ERROR] ci-openbsd: ci-openbsd-multicore: manager exited unexpectedly: exit status 1
2024/12/09 04:30:53 [ERROR] ci-openbsd: ci-openbsd-main: manager exited unexpectedly: exit status 1
2024/12/09 04:31:59 ci-openbsd: ci-openbsd-main: poll: 1b6b588e81a400a8190da30b39ed84c2b707f01c
2024/12/09 04:34:10 ci-openbsd: ci-openbsd-multicore: testing image...
2024/12/09 04:35:16 ci-openbsd: syzkaller: poll: 9ac0fdc66500475f1914254ef369b32d51adbff9 (sys/linux/tests: add a seed that unrolls syz_kvm_vgic_v3_setup())
2024/12/09 04:37:46 [ERROR] ci-openbsd: ci-openbsd-multicore: image testing failed w/o kernel bug
2024/12/09 04:37:46 ci-openbsd: ci-openbsd-multicore: failed to run ["/syzkaller/syzkaller/current/bin/syz-manager" "-config" "/syzkaller/managers/multicore/latest.tmp/workdir/manager.cfg" "-mode=smoke-test"]: exit status 1

Cc @blackgnezdo

[blackgnezdo]

This vnd image mount has been temperamental in the past. Since all the tests pass, I don't expect this to happen in steady state. Could you restart the openbsd ci-bot machine so we can rule out any weird state accretion?

[a-nogikh]

It did not change the outcome

2024/12/11 07:54:24 ci-openbsd: ci-openbsd-setuid: latest image is on 1511e544c2133eb425911df1402f18e15bb5ab0c
2024/12/11 07:54:24 ci-openbsd: ci-openbsd-job: job loop started
2024/12/11 07:54:24 ci-openbsd: ci-openbsd-main: latest image is on 1511e544c2133eb425911df1402f18e15bb5ab0c
2024/12/11 07:54:24 ci-openbsd: ci-openbsd-multicore: latest image is on 1511e544c2133eb425911df1402f18e15bb5ab0c
2024/12/11 07:56:08 ci-openbsd: ci-openbsd-main: poll: 131e173293c3c7a5e17595e519cbd4c9bf30d51c
2024/12/11 07:56:08 ci-openbsd: ci-openbsd-main: building kernel...
2024/12/11 07:56:15 ci-openbsd: ci-openbsd-multicore: poll: 131e173293c3c7a5e17595e519cbd4c9bf30d51c
2024/12/11 07:56:15 ci-openbsd: ci-openbsd-setuid: poll: 131e173293c3c7a5e17595e519cbd4c9bf30d51c
2024/12/11 08:00:32 ci-openbsd: ci-openbsd-main: testing image...
2024/12/11 08:04:08 [ERROR] ci-openbsd: ci-openbsd-main: image testing failed w/o kernel bug
2024/12/11 08:04:09 ci-openbsd: ci-openbsd-main: failed to run ["/syzkaller/syzkaller/current/bin/syz-manager" "-config" "/syzkaller/managers/main/latest.tmp/workdir/manager.cfg" "-mode=smoke-test"]: exit status 1
2024/12/11 08:04:09 ci-openbsd: ci-openbsd-multicore: building kernel...
2024/12/11 08:08:27 ci-openbsd: ci-openbsd-multicore: testing image...
2024/12/11 08:09:31 [ERROR] ci-openbsd: ci-openbsd-main: manager exited unexpectedly: exit status 1
2024/12/11 08:11:55 [ERROR] ci-openbsd: ci-openbsd-multicore: image testing failed w/o kernel bug
2024/12/11 08:11:55 ci-openbsd: ci-openbsd-multicore: failed to run ["/syzkaller/syzkaller/current/bin/syz-manager" "-config" "/syzkaller/managers/multicore/latest.tmp/workdir/manager.cfg" "-mode=smoke-test"]: exit status 1
2024/12/11 08:11:55 ci-openbsd: ci-openbsd-setuid: building kernel...

The error says that, for some reason, running ./bin/syz-manager -config config.cfg -mode=smoke-test no longer works.

Manager logs are as follows:

2024/12/11 08:09:08 ci-openbsd-main: wait instance to boot: ci-openbsd-main-2 (10.128.0.230)
2024/12/11 08:09:10 ci-openbsd-main: wait instance to boot: ci-openbsd-main-3 (10.128.0.13)
2024/12/11 08:09:11 ci-openbsd-main: wait instance to boot: ci-openbsd-main-9 (10.128.1.146)
2024/12/11 08:09:13 ci-openbsd-main: wait instance to boot: ci-openbsd-main-6 (10.128.1.183)
2024/12/11 08:09:14 ci-openbsd-main: wait instance to boot: ci-openbsd-main-7 (10.128.0.247)
2024/12/11 08:09:15 ci-openbsd-main: wait instance to boot: ci-openbsd-main-4 (10.128.0.221)
2024/12/11 08:09:16 ci-openbsd-main: wait instance to boot: ci-openbsd-main-1 (10.128.0.6)
2024/12/11 08:09:17 ci-openbsd-main: wait instance to boot: ci-openbsd-main-5 (10.128.1.130)
2024/12/11 08:09:26 ci-openbsd-main: runner 8 connected
2024/12/11 08:09:27 ci-openbsd-main: runner 0 connected
2024/12/11 08:09:29 ci-openbsd-main: runner 3 connected
2024/12/11 08:09:30 ci-openbsd-main: runner 9 connected
2024/12/11 08:09:31 ci-openbsd-main: machine check:
BinFmtMisc              : enabled
Comparisons             : ioctl init trace write failed.  (errno 22: Invalid argument). . process exited with status 67.
Coverage                : ioctl init trace write failed.  (errno 22: Invalid argument). . process exited with status 67.
DelayKcovMmap           : ioctl init trace write failed.  (errno 22: Invalid argument). . process exited with status 67.
DevlinkPCI              : enabled
ExtraCoverage           : ioctl init trace write failed.  (errno 22: Invalid argument). . process exited with status 67.
Fault                   : enabled
KCSAN                   : enabled
LRWPANEmulation         : enabled
Leak                    : enabled
NetDevices              : enabled
NetInjection            : enabled
NicVF                   : enabled
SandboxAndroid          : unknown sandbox type.  (errno 9: Bad file descriptor). . process exited with status 67.
SandboxNamespace        : unknown sandbox type.  (errno 9: Bad file descriptor). . process exited with status 67.
SandboxNone             : enabled
SandboxSetuid           : enabled
Swap                    : enabled
USBEmulation            : enabled
VhciInjection           : enabled
WifiEmulation           : enabled
syscalls                : 389/396

2024/12/11 08:09:31 [FATAL] ci-openbsd-main: check failed: coverage is not supported: ioctl init trace write failed.  (errno 22: Invalid argument). . process exited with status 67.

[blackgnezdo]

This is a bit on the terse side to be debugged easily:

Comparisons             : ioctl init trace write failed.  (errno 22: Invalid argument). . process exited with status 67.

My guess would be that we changed the kernel ABI such that kcov related ioctls compiled on ~7.6 release don't work on -current kernels. The remedy would be to reimage the OpenBSD ci-bot (the usual). It seems like a safe thing to do anyway.

[a-nogikh]

I've started the dashboard/config/openbsd/recreate.sh script, let's see if that helps :)

[a-nogikh]

Didn't help, unfortunately

2024/12/12 01:58:38 wait instance to boot: ci-openbsd-main-test-2 (10.128.1.163)
2024/12/12 01:58:43 wait instance to boot: ci-openbsd-main-test-1 (10.128.10.49)
2024/12/12 01:58:51 wait instance to boot: ci-openbsd-main-test-0 (10.128.10.38)
2024/12/12 01:59:11 machine check:
BinFmtMisc              : enabled
Comparisons             : ioctl init trace write failed.  (errno 22: Invalid argument). . process exited with status 67.
Coverage                : ioctl init trace write failed.  (errno 22: Invalid argument). . process exited with status 67.
DelayKcovMmap           : ioctl init trace write failed.  (errno 22: Invalid argument). . process exited with status 67.
DevlinkPCI              : enabled
ExtraCoverage           : ioctl init trace write failed.  (errno 22: Invalid argument). . process exited with status 67.
Fault                   : enabled
KCSAN                   : enabled
LRWPANEmulation         : enabled
Leak                    : enabled
NetDevices              : enabled
NetInjection            : enabled
NicVF                   : enabled
SandboxAndroid          : unknown sandbox type.  (errno 9: Bad file descriptor). . process exited with status 67.
SandboxNamespace        : unknown sandbox type.  (errno 9: Bad file descriptor). . process exited with status 67.
SandboxNone             : enabled
SandboxSetuid           : enabled
Swap                    : enabled
USBEmulation            : enabled
VhciInjection           : enabled
WifiEmulation           : enabled
syscalls                : 389/396

2024/12/12 01:59:11 [FATAL] check failed: coverage is not supported: ioctl init trace write failed.  (errno 22: Invalid argument). . process exited with status 67.

[a-nogikh]

And now there's also https://syzkaller.appspot.com/bug?extid=94080caf3163d1964354