Unable to build custom kernel module in 2605.9.0

[mindw]

Description

Build of a custom kernel module fails on 2605.9.0 - the same process is successful on 2605.8.0/

Impact

Production env is dependant on the module in question - blocked from upgrading.

Environment and steps to reproduce

1. Set-up: spin up Flatcar EC2 instance in AWS using ami-01005e70742a799fe
2. Task: build wireguard custom kernel module 1.0.20201112 for Flatcar 2605.9
3. Action(s): [ sequence of actions that triggered the bug, see example below ]

[2020-12-09T00:27:08.532Z] �[0m�[0mnull_resource.run_install_script (remote-exec): + export WIREGUARD_VERSION=1.0.20201112
[2020-12-09T00:27:08.532Z] �[0m�[0mnull_resource.run_install_script (remote-exec): + WIREGUARD_VERSION=1.0.20201112
[2020-12-09T00:27:08.532Z] �[0m�[0mnull_resource.run_install_script (remote-exec): + modules=/opt/modules
[2020-12-09T00:27:08.532Z] �[0m�[0mnull_resource.run_install_script (remote-exec): + mkdir -p /opt/modules /opt/modules.wd
[2020-12-09T00:27:08.532Z] �[0m�[0mnull_resource.run_install_script (remote-exec): + mount -o lowerdir=/lib/modules,upperdir=/opt/modules,workdir=/opt/modules.wd -t overlay overlay /lib/modules
[2020-12-09T00:27:08.532Z] �[0m�[0mnull_resource.run_install_script (remote-exec): + systemctl enable lib-modules.mount
[2020-12-09T00:27:08.532Z] �[0m�[0mnull_resource.run_install_script (remote-exec): Created symlink /etc/systemd/system/local-fs.target.wants/lib-modules.mount → /etc/systemd/system/lib-modules.mount.
[2020-12-09T00:27:08.791Z] �[0m�[0mnull_resource.run_install_script (remote-exec): + . /usr/share/flatcar/release
[2020-12-09T00:27:08.791Z] �[0m�[0mnull_resource.run_install_script (remote-exec): ++ FLATCAR_RELEASE_VERSION=2605.9.0
[2020-12-09T00:27:08.791Z] �[0m�[0mnull_resource.run_install_script (remote-exec): ++ FLATCAR_RELEASE_BOARD=amd64-usr
[2020-12-09T00:27:08.791Z] �[0m�[0mnull_resource.run_install_script (remote-exec): ++ FLATCAR_RELEASE_APPID='{e96281a6-d1af-4bde-9a0a-97b76e56dc57}'
[2020-12-09T00:27:08.791Z] �[0m�[0mnull_resource.run_install_script (remote-exec): + . /usr/share/flatcar/update.conf
[2020-12-09T00:27:08.791Z] �[0m�[0mnull_resource.run_install_script (remote-exec): ++ SERVER=https://public.update.flatcar-linux.net/v1/update/
[2020-12-09T00:27:08.791Z] �[0m�[0mnull_resource.run_install_script (remote-exec): ++ GROUP=stable
[2020-12-09T00:27:08.791Z] �[0m�[0mnull_resource.run_install_script (remote-exec): + url=https://stable.release.flatcar-linux.net/amd64-usr/2605.9.0/flatcar_developer_container.bin.bz2
[2020-12-09T00:27:08.791Z] �[0m�[0mnull_resource.run_install_script (remote-exec): + curl -L https://stable.release.flatcar-linux.net/amd64-usr/2605.9.0/flatcar_developer_container.bin.bz2
[2020-12-09T00:27:08.791Z] �[0m�[0mnull_resource.run_install_script (remote-exec): + bzip2 -d
...
[2020-12-09T00:29:30.219Z] �[0m�[0mnull_resource.run_install_script (remote-exec): + src_dir=/home/core/coreos_src
[2020-12-09T00:29:30.219Z] �[0m�[0mnull_resource.run_install_script (remote-exec): ++ pwd
[2020-12-09T00:29:30.219Z] �[0m�[0mnull_resource.run_install_script (remote-exec): + var_tmp_dir=/home/core/coreos_var_tmp
[2020-12-09T00:29:30.219Z] �[0m�[0mnull_resource.run_install_script (remote-exec): + mkdir -p /home/core/coreos_src /home/core/coreos_var_tmp
[2020-12-09T00:29:30.219Z] �[0m�[0mnull_resource.run_install_script (remote-exec): + sed -i s/_wireguard_version_/1.0.20201112/ /tmp/installer.sh
[2020-12-09T00:29:30.219Z] �[0m�[0mnull_resource.run_install_script (remote-exec): + systemd-nspawn --bind=/lib/modules --bind=/tmp --overlay=/home/core/coreos_src::/usr/src --overlay=/home/core/coreos_var_tmp::/var/tmp --image=flatcar_developer_container.bin /tmp/installer.sh
...
[2020-12-09T00:29:30.219Z] �[0m�[0mnull_resource.run_install_script (remote-exec): + src_dir=/home/core/coreos_src
[2020-12-09T00:29:30.219Z] �[0m�[0mnull_resource.run_install_script (remote-exec): ++ pwd
[2020-12-09T00:29:30.219Z] �[0m�[0mnull_resource.run_install_script (remote-exec): + var_tmp_dir=/home/core/coreos_var_tmp
[2020-12-09T00:29:30.219Z] �[0m�[0mnull_resource.run_install_script (remote-exec): + mkdir -p /home/core/coreos_src /home/core/coreos_var_tmp
[2020-12-09T00:29:30.219Z] �[0m�[0mnull_resource.run_install_script (remote-exec): + sed -i s/_wireguard_version_/1.0.20201112/ /tmp/installer.sh
[2020-12-09T00:29:30.219Z] �[0m�[0mnull_resource.run_install_script (remote-exec): + systemd-nspawn --bind=/lib/modules --bind=/tmp --overlay=/home/core/coreos_src::/usr/src --overlay=/home/core/coreos_var_tmp::/var/tmp --image=flatcar_developer_container.bin /tmp/installer.sh
...
2020-12-09T00:30:04.691Z] �[0m�[1mnull_resource.run_install_script: Still creating... (3m30s elapsed)�[0m�[0m
[2020-12-09T00:30:05.257Z] �[0m�[0mnull_resource.run_install_script (remote-exec): >>> Git clone in /var/lib/portage/portage-stable successful
[2020-12-09T00:30:05.257Z] �[0m�[0mnull_resource.run_install_script (remote-exec): Warning: No revision found for https://github.com/flatcar-linux/coreos-overlay.git
[2020-12-09T00:30:05.257Z] �[0m�[0mnull_resource.run_install_script (remote-exec): >>> Cloning repository 'coreos' from 'https://github.com/flatcar-linux/coreos-overlay.git'...
[2020-12-09T00:30:05.257Z] �[0m�[0mnull_resource.run_install_script (remote-exec): >>> Starting git clone in /var/lib/portage/coreos-overlay
[2020-12-09T00:30:05.257Z] �[0m�[0mnull_resource.run_install_script (remote-exec): Cloning into '/var/lib/portage/coreos-overlay'...
...
[2020-12-09T00:30:11.924Z] �[0m�[0mnull_resource.run_install_script (remote-exec): >>> Git clone in /var/lib/portage/coreos-overlay successful

4. Error: [describe the error that was triggered]

02:30:12  null_resource.run_install_script (remote-exec): Performing Global Updates
02:30:12  null_resource.run_install_script (remote-exec): (Could take a couple of minutes if you have a lot of binary packages.)
02:30:13  
02:30:13  
02:30:13  null_resource.run_install_script (remote-exec): + emerge -gKv coreos-sources
02:30:13  null_resource.run_install_script (remote-exec): 
02:30:13  
02:30:13  
02:30:13  null_resource.run_install_script (remote-exec): !!! Error fetching binhost package info from 'https://storage.googleapis.com/flatcar-jenkins/developer/boards/amd64-usr/2605.9.0/pkgs'
02:30:13  null_resource.run_install_script (remote-exec): !!! HTTP Error 404: Not Found
02:30:13  
02:30:13  
02:30:13  
02:30:13  null_resource.run_install_script (remote-exec): !!! Error fetching binhost package info from 'https://storage.googleapis.com/flatcar-jenkins/developer/boards/amd64-usr/2605.9.0/toolchain'
02:30:13  null_resource.run_install_script (remote-exec): !!! HTTP Error 404: Not Found
02:30:13  

Expected behavior
build is successful

Additional information
Since we're building wireguard, the issue isn't relevant to beta and alpha build where it is part of the kernel.

Thank you for the great work on Flatcar and otherwise :)

[margamanterola]

Hey there! Thanks for the report! The fix for it has been merged and will be present in the next patch release for each channel.

In the meantime, if you need to build the module, you can modify the /etc/portage/make.conf file in the container to point to https://storage.googleapis.com/flatcar-jenkins/boards/amd64/2605.9.0/{pkgs,toolchain} instead of the developer addresses.

[mindw]

Unfortunately this fix re-opens a can of worms - the build fails on what seems a regressions of
flatcar-archive/coreos-overlay#19

[2020-12-17T00:33:08.184Z] null_resource.run_install_script (remote-exec):   CC [M]  /wireguard-linux-compat-1.0.20201112/src/crypto/zinc/chacha20/chacha20.o
[2020-12-17T00:33:09.118Z] null_resource.run_install_script (remote-exec):   PERLASM /wireguard-linux-compat-1.0.20201112/src/crypto/zinc/chacha20/chacha20-x86_64.S
[2020-12-17T00:33:09.376Z] null_resource.run_install_script (remote-exec): /bin/sh: line 1: 30227 Segmentation fault      (core dumped) perl /wireguard-linux-compat-1.0.20201112/src/crypto/zinc/chacha20/chacha20-x86_64.pl > /wireguard-linux-compat-1.0.20201112/src/crypto/zinc/chacha20/chacha20-x86_64.S
[2020-12-17T00:33:09.376Z] null_resource.run_install_script (remote-exec): make[2]: *** [/wireguard-linux-compat-1.0.20201112/src/crypto/Kbuild.include:43: /wireguard-linux-compat-1.0.20201112/src/crypto/zinc/chacha20/chacha20-x86_64.S] Error 139

Adding emerge perl allows building wireguard but with ranings

[margamanterola]

Huh, that's interesting, but I think it's unrelated to this bug (which was due to a bad URL in the dev container) and something perl specific. Could you please open a new issue about this? In that new issue, please include whether you're cross compiling or not (the old bug you pointed to was about cross compiling).