Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tests __pycache__ data in pypi sdist tarball makes packaging unreproducible #72

Closed
dvzrv opened this issue Jul 15, 2020 · 2 comments · Fixed by #73
Closed

tests __pycache__ data in pypi sdist tarball makes packaging unreproducible #72

dvzrv opened this issue Jul 15, 2020 · 2 comments · Fixed by #73

Comments

@dvzrv
Copy link
Contributor

dvzrv commented Jul 15, 2020

Hi! When rebuilding the package for 1.0.4 on Arch Linux I ran our reproducible builds tooling against the package.
Unfortunately the __pacache__ data found in the tarball makes the package unreproducible.

The diffoscope output:

--- python-django-appconf-1.0.4-2-any.pkg.tar.zst
+++ build/python-django-appconf-1.0.4-2-any.pkg.tar.zst
├── python-django-appconf-1.0.4-2-any.pkg.tar
│ ├── file list
│ │ @@ -1,9 +1,9 @@
│ │  -rw-r--r--   0 root         (0) root         (0)     5267 2020-07-15 20:27:00.000000 .BUILDINFO
│ │ --rw-r--r--   0 root         (0) root         (0)     1638 2020-07-15 20:27:00.000000 .MTREE
│ │ +-rw-r--r--   0 root         (0) root         (0)     1637 2020-07-15 20:27:00.000000 .MTREE
│ │  -rw-r--r--   0 root         (0) root         (0)      521 2020-07-15 20:27:00.000000 .PKGINFO
│ │  drwxr-xr-x   0 root         (0) root         (0)        0 2020-07-15 20:27:00.000000 usr/
│ │  drwxr-xr-x   0 root         (0) root         (0)        0 2020-07-15 20:27:00.000000 usr/lib/
│ │  drwxr-xr-x   0 root         (0) root         (0)        0 2020-07-15 20:27:00.000000 usr/lib/python3.8/
│ │  drwxr-xr-x   0 root         (0) root         (0)        0 2020-07-15 20:27:00.000000 usr/lib/python3.8/site-packages/
│ │  drwxr-xr-x   0 root         (0) root         (0)        0 2020-07-15 20:27:00.000000 usr/lib/python3.8/site-packages/appconf/
│ │  -rw-r--r--   0 root         (0) root         (0)       57 2020-07-15 20:27:00.000000 usr/lib/python3.8/site-packages/appconf/__init__.py
│ │ @@ -14,15 +14,15 @@
│ │  -rw-r--r--   0 root         (0) root         (0)     4714 2020-07-15 20:27:00.000000 usr/lib/python3.8/site-packages/appconf/__pycache__/base.cpython-38.pyc
│ │  -rw-r--r--   0 root         (0) root         (0)      722 2020-07-15 20:27:00.000000 usr/lib/python3.8/site-packages/appconf/__pycache__/utils.cpython-38.opt-1.pyc
│ │  -rw-r--r--   0 root         (0) root         (0)      722 2020-07-15 20:27:00.000000 usr/lib/python3.8/site-packages/appconf/__pycache__/utils.cpython-38.pyc
│ │  -rw-r--r--   0 root         (0) root         (0)     5181 2020-07-15 20:27:00.000000 usr/lib/python3.8/site-packages/appconf/base.py
│ │  -rw-r--r--   0 root         (0) root         (0)      861 2020-07-15 20:27:00.000000 usr/lib/python3.8/site-packages/appconf/utils.py
│ │  drwxr-xr-x   0 root         (0) root         (0)        0 2020-07-15 20:27:00.000000 usr/lib/python3.8/site-packages/django_appconf-1.0.4-py3.8.egg-info/
│ │  -rw-r--r--   0 root         (0) root         (0)     5998 2020-07-15 20:27:00.000000 usr/lib/python3.8/site-packages/django_appconf-1.0.4-py3.8.egg-info/PKG-INFO
│ │ --rw-r--r--   0 root         (0) root         (0)      834 2020-07-15 20:27:00.000000 usr/lib/python3.8/site-packages/django_appconf-1.0.4-py3.8.egg-info/SOURCES.txt
│ │ +-rw-r--r--   0 root         (0) root         (0)      666 2020-07-15 20:27:00.000000 usr/lib/python3.8/site-packages/django_appconf-1.0.4-py3.8.egg-info/SOURCES.txt
│ │  -rw-r--r--   0 root         (0) root         (0)        1 2020-07-15 20:27:00.000000 usr/lib/python3.8/site-packages/django_appconf-1.0.4-py3.8.egg-info/dependency_links.txt
│ │  -rw-r--r--   0 root         (0) root         (0)        7 2020-07-15 20:27:00.000000 usr/lib/python3.8/site-packages/django_appconf-1.0.4-py3.8.egg-info/requires.txt
│ │  -rw-r--r--   0 root         (0) root         (0)        8 2020-07-15 20:27:00.000000 usr/lib/python3.8/site-packages/django_appconf-1.0.4-py3.8.egg-info/top_level.txt
│ │  drwxr-xr-x   0 root         (0) root         (0)        0 2020-07-15 20:27:00.000000 usr/share/
│ │  drwxr-xr-x   0 root         (0) root         (0)        0 2020-07-15 20:27:00.000000 usr/share/doc/
│ │  drwxr-xr-x   0 root         (0) root         (0)        0 2020-07-15 20:27:00.000000 usr/share/doc/python-django-appconf/
│ │  -rw-r--r--   0 root         (0) root         (0)       91 2020-07-15 20:27:00.000000 usr/share/doc/python-django-appconf/AUTHORS
│ ├── .MTREE
│ │ ├── .MTREE-content
│ │ │ @@ -1,11 +1,11 @@
│ │ │  #mtree
│ │ │  /set type=file uid=0 gid=0 mode=644
│ │ │  ./.BUILDINFO time=1594844820.0 size=5267 md5digest=6299c5cad295b5a554fc329a23172992 sha256digest=3fb781c7b5f62a369060c7b8b0b65d5518fa5fbd8ef1bac999fbd6ae765de614
│ │ │ -./.PKGINFO time=1594844820.0 size=521 md5digest=4437e18c8a24b63e722e0f1884e05579 sha256digest=e3a3a3f5dca8a5c8a757bdbb2105673605265b1f572abcb03c2fb0cf5cab3aff
│ │ │ +./.PKGINFO time=1594844820.0 size=521 md5digest=15f505c0ba63e11cd28db270c77ebf82 sha256digest=cbf99e1cfeb6bf2fb50d5312754bc3a9e1948408d543da580ca51162268b36f5
│ │ │  /set mode=755
│ │ │  ./usr time=1594844820.0 type=dir
│ │ │  ./usr/lib time=1594844820.0 type=dir
│ │ │  ./usr/lib/python3.8 time=1594844820.0 type=dir
│ │ │  ./usr/lib/python3.8/site-packages time=1594844820.0 type=dir
│ │ │  /set mode=644
│ │ │  ./usr/lib/python3.8/site-packages/appconf time=1594844820.0 mode=755 type=dir
│ │ │ @@ -17,15 +17,15 @@
│ │ │  ./usr/lib/python3.8/site-packages/appconf/__pycache__/__init__.cpython-38.pyc time=1594844820.0 size=205 md5digest=f5bcab9e6c3236cfaee0627ba7c3e9a4 sha256digest=9c9ec95e9c9c92fc32314a42a1f9d58dd95dc6a29e8861282a329bd9fd87f9b6
│ │ │  ./usr/lib/python3.8/site-packages/appconf/__pycache__/base.cpython-38.opt-1.pyc time=1594844820.0 size=4714 md5digest=bf47d7a975cfbaa8d687538ff1f22d9c sha256digest=31d4eb93a4c59c9bd79cd051b5d35c8a5c54bf5d08c00e7f4f96f4e537f55bc3
│ │ │  ./usr/lib/python3.8/site-packages/appconf/__pycache__/base.cpython-38.pyc time=1594844820.0 size=4714 md5digest=ffd61114730ce4ef49c2cdccbb19f49a sha256digest=69e495b19ac4513333c3b6f5441802a16c0301133f169780d9832f362f655b3c
│ │ │  ./usr/lib/python3.8/site-packages/appconf/__pycache__/utils.cpython-38.opt-1.pyc time=1594844820.0 size=722 md5digest=59a810d0963b9e62af32fe6c218f810a sha256digest=643c5b82200caf79278a8fcb4cbb8c19188e09f7942879be5335054efc2242e1
│ │ │  ./usr/lib/python3.8/site-packages/appconf/__pycache__/utils.cpython-38.pyc time=1594844820.0 size=722 md5digest=7a6a2652ce03e713e55236aa75cf3c9e sha256digest=6cf0c7929555b5746410e73c3999ea357fdd7652a820e43ecf1508e43d41b1b3
│ │ │  ./usr/lib/python3.8/site-packages/django_appconf-1.0.4-py3.8.egg-info time=1594844820.0 mode=755 type=dir
│ │ │  ./usr/lib/python3.8/site-packages/django_appconf-1.0.4-py3.8.egg-info/PKG-INFO time=1594844820.0 size=5998 md5digest=abc213792ad4b89405c5035cc3f5ab94 sha256digest=51fdc7a8bcca9dedf67bd4f0e3a6e6781e648256ab734a0994df353e66d66c3f
│ │ │ -./usr/lib/python3.8/site-packages/django_appconf-1.0.4-py3.8.egg-info/SOURCES.txt time=1594844820.0 size=834 md5digest=caa3c4e2faa327037e47c15f51b151fd sha256digest=0c288ae70d348cec450bb5a160a96a0d5deb441b2f473d3f4b56bc166ad61677
│ │ │ +./usr/lib/python3.8/site-packages/django_appconf-1.0.4-py3.8.egg-info/SOURCES.txt time=1594844820.0 size=666 md5digest=50179acd1ea1c44f1168b73acebcfd6c sha256digest=cdfbed1aec98df9f190aea521c4cf4332d87b39c30da3a881556810b497f5a1c
│ │ │  ./usr/lib/python3.8/site-packages/django_appconf-1.0.4-py3.8.egg-info/dependency_links.txt time=1594844820.0 size=1 md5digest=68b329da9893e34099c7d8ad5cb9c940 sha256digest=01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b
│ │ │  ./usr/lib/python3.8/site-packages/django_appconf-1.0.4-py3.8.egg-info/requires.txt time=1594844820.0 size=7 md5digest=bea59528d92a016338600bf69118c5ab sha256digest=57f8288a383db5f3b6d28c7fee8b3a09c9cfbe605abdbc6ee3a2e926234bc230
│ │ │  ./usr/lib/python3.8/site-packages/django_appconf-1.0.4-py3.8.egg-info/top_level.txt time=1594844820.0 size=8 md5digest=c3aa368fedd37101d3d4a373e0d97c05 sha256digest=6cbc70de4b67de3eba0de6c567b866ac511c519489cc7cff72784bd7a5a737b9
│ │ │  /set mode=755
│ │ │  ./usr/share time=1594844820.0 type=dir
│ │ │  ./usr/share/doc time=1594844820.0 type=dir
│ │ │  /set mode=644
│ ├── .PKGINFO
│ │ @@ -3,15 +3,15 @@
│ │  pkgname = python-django-appconf
│ │  pkgbase = python-django-appconf
│ │  pkgver = 1.0.4-2
│ │  pkgdesc = An app to handle configuration defaults of packaged Django apps gracefully
│ │  url = https://github.com/django-compressor/django-appconf
│ │  builddate = 1594844820
│ │  packager = David Runge <[email protected]>
│ │ -size = 29714
│ │ +size = 29546
│ │  arch = any
│ │  license = BSD
│ │  depend = python-django
│ │  depend = python-six
│ │  makedepend = python-setuptools
│ │  checkdepend = python-coverage
│ │  checkdepend = python-pytest
│ ├── usr/lib/python3.8/site-packages/django_appconf-1.0.4-py3.8.egg-info/SOURCES.txt
│ │ @@ -21,14 +21,10 @@
│ │  docs/usage.rst
│ │  tests/__init__.py
│ │  tests/models.py
│ │  tests/settings.py
│ │  tests/test_settings.py
│ │  tests/tests.py
│ │  tests/__pycache__/__init__.cpython-37.pyc
│ │ -tests/__pycache__/__init__.cpython-38.pyc
│ │  tests/__pycache__/models.cpython-37.pyc
│ │ -tests/__pycache__/models.cpython-38.pyc
│ │  tests/__pycache__/test_settings.cpython-37.pyc
│ │ -tests/__pycache__/test_settings.cpython-38.pyc
│ │ -tests/__pycache__/tests.cpython-37.pyc
│ │ -tests/__pycache__/tests.cpython-38.pyc
│ │ +tests/__pycache__/tests.cpython-37.pyc

The package only becomes reproducible if I remove the tests/__pycache__ directory before build and after testing.
@carltongibson
Copy link
Contributor

Ok, yes. These should be pruned in the manifest.

@carltongibson
Copy link
Contributor

Looks like you added the tests in #55. Fancy pruning the cache files and checking it's how you want/need it?

We'll do a point release declaring 3.1 support so good timing.

dvzrv added a commit to dvzrv/django-appconf that referenced this issue Jul 16, 2020
@dvzrv
Recursively excluding __pycache__
a5d425a 
MANIFEST.in:
Recursively exclude __pycache__ directories.
They break reproducibility and are test/build artifacts of the
build/test/release machine, which should never end up in a sdist.

Closes django-compressor#72
@dvzrv dvzrv mentioned this issue Jul 16, 2020
Merged
carltongibson pushed a commit that referenced this issue Aug 22, 2021
@dvzrv
Recursively excluding __pycache__ (#73)
46c56b6 
MANIFEST.in:
Recursively exclude __pycache__ directories.
They break reproducibility and are test/build artifacts of the
build/test/release machine, which should never end up in a sdist.

Closes #72
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Recursively excluding __pycache__ dvzrv/django-appconf
2 participants
@carltongibson @dvzrv