Builds are not reproducible #3
Comments
|
Interesting, The compressed man pages remain different when disabling Next move will be to check what can be done to help |
|
Setting the environment variable Setting the variable to the commit timestamp of the last commit of this repository (git log -1 --format=%ct) should be reasonable IMHO. Vanagon then needs to pass that variable down to the build container. Then, tweaking vanagon to also use the value of root@desktop-fln40kq …/voxpupuli/OpenVoxProject/openvox-agent # md5sum output/deb/debian12/openvox8*/openvox-agent_8.11.0.2.g0ea7f7d4f-1+debian12_amd64.deb
0b91f77ee5b991e2e1f99f0a1758886d output/deb/debian12/openvox8-first/openvox-agent_8.11.0.2.g0ea7f7d4f-1+debian12_amd64.deb
0b91f77ee5b991e2e1f99f0a1758886d output/deb/debian12/openvox8-second/openvox-agent_8.11.0.2.g0ea7f7d4f-1+debian12_amd64.debHowever I am not satisfied with disabling |
[...]
For some reason, I don't see these non-reproducibility issues anymore today. I guess I messed-up something on my side 🤷. After rebasing on top of master, I see some non-reproducibility regression and will look at it. |
|
PSA: I think we are ready for broader testing of Debian packages reproducible builds! (rpm have not been worked on yet) I pushed a commit into my fork of the repo to test if builds are reproducible. It change a few references to point to the last commit to open PR linked above. If you can test: This will build Debian 12 packages in the romain@desktop-fln40kq …/OpenVoxProject/openvox-agent % find output -type f | sort | xargs md5sum
d41a48957a3dccc482ff312e9ee26b6d output/deb/debian10/openvox8/openvox-agent_8.11.0.8.g67c82b198-1+debian10_amd64.deb
a91c6c3c4b811aea513bace3c8ad7169 output/deb/debian10/openvox8/openvox-agent-dbgsym_8.11.0.8.g67c82b198-1+debian10_amd64.deb
af2e626ba1451cba74391aea7d187729 output/deb/debian11/openvox8/openvox-agent_8.11.0.8.g67c82b198-1+debian11_amd64.deb
9499fff69a4fa3a9b79346a131ac9e61 output/deb/debian11/openvox8/openvox-agent-dbgsym_8.11.0.8.g67c82b198-1+debian11_amd64.deb
1ef91a5ec51f081f7442af1ef26591ad output/deb/debian12/openvox8/openvox-agent_8.11.0.8.g67c82b198-1+debian12_amd64.deb
2560af6c4c645f0a9d0e84b3a4c083c4 output/deb/debian12/openvox8/openvox-agent-dbgsym_8.11.0.8.g67c82b198-1+debian12_amd64.debIf they match, cool, please add a 👍 reaction to this message, otherwise add a 👎 reaction and see if your different checksum is stable across multiple builds on your system. If they change, keep a copy of the generated files and provide a link to them so that we can inspect them. Thank you 😍 ! |
|
The debian10 and debian11 generated files match, however the debian12 openvox-agent file does not. I get the following md5 hash for it (reproducably*): Here's a link to the openvox-agent file on my end: https://1drv.ms/u/s!Ai2oMivOdzMWhqJpHsh29wLIlR7LLg?e=DC0ZHg Note that so far, to reproduce it, I've rerun the rake task. I have also rebooted, which has of course deleted /tmp including the build dir, and after that I still get This is on a freshly created Debian 12.9 system on Azure These are the images brought into docker: |
That's interesting! I compared your package and mine, and they differ by a line in --- mine/control/md5sums 2025-01-23 15:23:29.000000000 -1000
+++ yours/control/md5sums 2025-01-23 15:23:29.000000000 -1000
@@ -8568,7 +8568,6 @@
5b561a90362b8eb9127c792c3f5902e0 opt/puppetlabs/puppet/ssl/ct_log_list.cnf
5b561a90362b8eb9127c792c3f5902e0 opt/puppetlabs/puppet/ssl/ct_log_list.cnf.dist
2b369e6aac60538d66f7fc8a85a4f68f opt/puppetlabs/puppet/ssl/misc/CA.pl
-70e16994fa00b4f75c132ad08d1ca21e opt/puppetlabs/puppet/ssl/misc/tsget
70e16994fa00b4f75c132ad08d1ca21e opt/puppetlabs/puppet/ssl/misc/tsget.pl
953fe829df1b6f2b1b39c7742ab6d0b8 opt/puppetlabs/puppet/ssl/openssl.cnf
953fe829df1b6f2b1b39c7742ab6d0b8 opt/puppetlabs/puppet/ssl/openssl.cnf.distIn my archive |
|
I built using a newly created Debian 12 (12.9) VM on a gen 2 x64 Azure vm (image publisher: debian, image offers: debian-12), docker.io package resulting in docker version 20.10.24+dfsg1, debian packaged ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) and bundler. ... Bizarre, I'll do some more testing I have also tried running the build with ruby 3.4.1 via rbenv and ruby-build on a separate VM (still Debian 12 on Azure) and the results are no different |
|
I've also tried building on my macbook air M3 with podman and get the same md5 for the debian 12 openvox-agent deb of I am not very familiar with the openssl build system but it looks like |
|
Hey! I managed to obtain your checksum on one of my machines using Vagrant to spin-up a Debian 12 node: # To generate packages and see their md5sum:
# vagrant up --provision
Vagrant.configure("2") do |config|
config.vm.box = "debian/bookworm64"
config.vm.provider "virtualbox" do |vb|
vb.memory = "2048"
end
config.vm.provision "shell", inline: <<-SHELL
apt-get update
apt-get install -y git ruby-bundler build-essential ruby-dev libyaml-dev docker.io
cd /tmp
git clone https://github.com/OpenVoxProject/openvox-agent # my clone does not have the required tags
cd openvox-agent
git remote add smortex https://github.com/smortex/openvox-agent
git fetch smortex
git checkout 67c82b19818d19ccac631d7ea5dc5d840f66b7ae # You must be on that exact commit!
bundle config path /tmp/do-not-mess-up-with-system
bundle install
bundle exec rake 'vox:build[openvox-agent,debian-12-amd64]' # Adjust here for a different target
md5sum /tmp/openvox-agent/output/deb/debian12/openvox8/openvox-agent-dbgsym_8.11.0.8.g67c82b198-1+debian12_amd64.deb
SHELL
endThis allowed be to see that the symlink vs. regular file issue appear during debuild (on this exact line: before we have no difference, after we do have a difference). By overriding actions in the rule file, I bisected to the I added some debugging when running
|
This is a meta-issue to track work on improving openvox-agent packages build reproducibility.
Rationale
Puppetlabs packages relied on internal resources to build packages, so it was not possible for the community to fully audit the build process.
Now that we have our own tooling for building packages and anybody can check how packages are built, we can go further and make it possible to check that packages which have been built with these tools have not been altered by ensuring builds are reproducible.
How to check for reproducibility
Basically you build the package twice and check if the generated artifacts are different.
For examples, for Debian packages:
The generated files in
output/deb/debian12/openvox8-firstandoutput/deb/debian12/openvox8-secondcan be further checked, e.g.:Scope
For now, I (@smortex) work on the Debian 12 packages only. Issues are likely to be similar on different targets but you are encouraged to test your platform and help with reproducibility issues resolution.
Current issues
WIP / Related PRs
Make builds reproducible vanagon#2
opt/puppetlabs/puppet/lib/ruby/vendor_gems/cache/puppet-resource_api-1.9.0.gem(current date affect build)usr/share/doc/openvox-agent/changelog.Debian.gz(current date used in the generated changelog entry)Files changed by(it did not failed the second day I worked on this. I probably messed something on my side)dh_strip_nondeterminismin a non reproducible wayopt/puppetlabs/puppet/lib/ruby/3.2.0/rdoc/generator/template/darkfish/images/macFFBgHack.png(timestamp in image)opt/puppetlabs/puppet/lib/ruby/gems/3.2.0/gems/concurrent-ruby-1.2.3/lib/concurrent-ruby/concurrent/concurrent_ruby.jar(needs further inspection)opt/puppetlabs/puppet/lib/ruby/vendor_gems/gems/highline-3.1.0/site/images/logo.png(a few bytes different in headers, more inspection needed)opt/puppetlabs/puppet/lib/ruby/vendor_gems/gems/locale-2.1.4/lib/locale/data/languages.tab.gz(gzip timestamp in header)opt/puppetlabs/puppet/lib/ruby/vendor_gems/gems/locale-2.1.4/lib/locale/data/regions.tab.gz(gzip timestamp in header)opt/puppetlabs/puppet/share/man/man8/facter.8.gz(gzip timestamp in header)opt/puppetlabs/puppet/share/man/man8/puppet*.8.gz(gzip timestamp in header)The text was updated successfully, but these errors were encountered: