Add container-based build workflow and improve build tooling

[ynezz]

This PR introduces a new container-based build workflow and several improvements to the build system:

Major Changes

• Added a new Makefile for streamlined container-based builds
• Created tools/build_firmware.sh script to standardize build steps
• Updated CI to use the new build script
• Added QA checks for shell scripts using reviewdog
• Fixed various build-related issues
• Mitigate certain supply chain attacks using sha256sums for external dependencies

New Build Workflow Features

• Simple make-based commands for building firmware
• Support for both Docker and Podman
• Flexible firmware manifest selection
• Shared build steps between local and CI environments

Technical Improvements

• Fixed permission issues with extension symlinks under rootless podman
• Improved error reporting in build_project.py
• Added shellcheck and shfmt QA checks
• Standardized build steps between local and CI environments

Usage Example

Build all firmware

make

Build specific manifest

make build_firmware MANIFESTS=manifests/nabucasa/yellow_bootloader.yaml

Documentation

Added comprehensive documentation in README.md covering:

• Prerequisites installation
• Build workflow usage
• Available make targets
• Customization options

Fixes

• Permission issues with rootless podman
• Git commit ID error reporting
• Build step standardization

[ynezz]

Just fixed the input device is not a TTY when running make all in GitLab CI within Docker In Docker GitLab runner

docker run --rm -it --user 0:0 -v /builds/prpl-foundation/mirrors/silabs-firmware-builder:/build -v /builds/prpl-foundation/mirrors/silabs-firmware-builder/outputs:/outputs -v /builds/prpl-foundation/mirrors/silabs-firmware-builder/build_dir:/build_dir silabs-firmware-builder \
	bash -c " \
		build_firmware.sh \
		--build-dir /build_dir \
		--output-dir /outputs \
		--manifest manifests/nabucasa/skyconnect_zigbee_ncp.yaml --manifest manifests/nabucasa/yellow_bootloader.yaml --manifest manifests/nabucasa/skyconnect_openthread_rcp.yaml --manifest manifests/nabucasa/yellow_openthread_rcp.yaml --manifest manifests/nabucasa/yellow_zigbee_ncp.yaml --manifest manifests/nabucasa/zwave_stick.yaml --manifest manifests/nabucasa/skyconnect_bootloader.yaml --manifest manifests/prpl_foundation/wnc_freedom_openthread_rcp.yaml --manifest manifests/prpl_foundation/wnc_freedom_bootloader.yaml \
	"
the input device is not a TTY

[ynezz]

just improved the UX by providing No container engine found, please install docker or podman when neither podman or docker is installed and make was run

[puddly]

Thanks for the PR! I will give it a thorough review in January.

[puddly]

I've had some time to peek at this PR and I like the core idea. The checksum changes to the Dockerfile are something that I'd merge right now and the addition of linting for the shell scripts is useful once we add those in-tree.

The Makefile + shell script combination in my eyes introduce two layers of duplication and separation for something that can be directly done. The shell script also mutates the global system environment and I would like to keep those steps explicit.

As an alternative, what do you think about introducing a thin shell script as an ENTRYPOINT to the Docker container? The shell script would then perform the required git and slc signature trust operations, keeping the container stateless, and then just directly run the Python script.

This would then let you build things directly:

docker run --rm -v "$PWD":/build --output-dir artifacts --manifest manifests/nabucasa/yellow_zigbee_ncp.yaml --output gbl

If building multiple images at once is a bottleneck, this would be very easy to add to the Python script to speed things up.

[ynezz]

The Makefile + shell script combination in my eyes introduce two layers of duplication and separation for something that can be directly done.

I just wanted to make it more reusable (DRY), thus being able to do make all everywhere, including my workstation and CI https://gitlab.com/prpl-foundation/mirrors/silabs-firmware-builder/-/blob/4efbe05b39870846753977ffd39d519d3afe4359/.gitlab/build.yml#L19

The shell script also mutates the global system environment

If you mean git config --global --add safe.directory "$work_dir", then IMO that call just moved into the common place, so it can be reused and its always run in the scratch container (--rm), so "stateless" unless I miss something.

silabs-firmware-builder/Makefile

Lines 23 to 29 in c20b59a

	$(CONTAINER_ENGINE) \
	run --rm \
	--user $(CONTAINER_USER_GROUP) \
	-v $(TOPDIR):/build$(VOLUME_OPTS) \
	-v $(TOPDIR)/outputs:/outputs$(VOLUME_OPTS) \
	-v $(TOPDIR)/build_dir:/build_dir$(VOLUME_OPTS) \
	$(CONTAINER_NAME)

As an alternative, what do you think about introducing a thin shell script as an ENTRYPOINT to the Docker container?

Could be an option, yes, I'll look into this.