test failures due to simultaneous tests trying to download AWS files

[grahamgower]

I think I was missing some of the latest annotations files in my _test_cache folder, and running the tests consistently produced the error below, even after removing the file in question.

(venv) t490:stdpopsim $ mv _test_cache _test_cache.bak
(venv) t490:stdpopsim $ pytest -n 8 -x tests/
============================================================= test session starts ==============================================================
platform linux -- Python 3.10.7, pytest-7.2.0, pluggy-1.0.0
rootdir: /home/grg/src/stdpopsim, configfile: setup.cfg
plugins: cov-4.0.0, xdist-3.0.2, forked-1.4.0
gw0 [2168] / gw1 [2168] / gw2 [2168] / gw3 [2168] / gw4 [2168] / gw5 [2168] / gw6 [2168] / gw7 [2168]
........................................................................................................................................ [  6%]
........................................................................................................................................ [ 12%]
......................................................................................................................................... [ 18%]
........................................................................................................................................ [ 25%]
........................................................................................................................................ [ 31%]
........................................................................................................................................ [ 37%]
........................................................................................................................................ [ 43%]
........................................................................................................................................ [ 50%]
........................................................................................................................................ [ 56%]
........................................................................................................................................ [ 62%]
........................................................................................................................................ [ 69%]
........................................................................................................................................ [ 75%]
........................................................................................................................................ [ 81%]
........................................................................................................................................ [ 87%]
..............................................E
==================================================================== ERRORS ====================================================================
_______________________________________________ ERROR at setup of TestAnnotation.test_cache_dirs _______________________________________________
[gw4] linux -- Python 3.10.7 /home/grg/src/stdpopsim/venv/bin/python

    def setup_module():
        destination = pathlib.Path("_test_cache/tarballs")
        for an in stdpopsim.all_annotations():
            key = an.id
            local_file = destination / (key + ".tar.gz")
            logging.info(f"key {key} local_file {local_file}")
            if not local_file.exists():
                cache_dir = local_file.parent
                cache_dir.mkdir(exist_ok=True, parents=True)
                print("Downloading", an.intervals_url)
                utils.download(an.intervals_url, local_file)
            # This assertion could fail if we update a file on AWS,
            # or a developer creates a new annotation with the wrong checksum
            # (in the latter case, this should at least be caught during CI tests).
>           assert utils.sha256(local_file) == an.intervals_sha256, (
                f"SHA256 for {local_file} doesn't match the SHA256 for "
                f"{an.id}. If you didn't add this SHA256 yourself, "
                f"try deleting {local_file} and restarting the tests."
            )
E           AssertionError: SHA256 for _test_cache/tarballs/FlyBase_BDGP6.32.51_CDS.tar.gz doesn't match the SHA256 for FlyBase_BDGP6.32.51_CDS. If you didn't add this SHA256 yourself, try deleting _test_cache/tarballs/FlyBase_BDGP6.32.51_CDS.tar.gz and restarting the tests.
E           assert 'f5fa9b830dc0...695420e225fac' == 'b993c8fc997e...6866a6a0def15'
E             - b993c8fc997e1c7ecdad626b7eeceae724cc0e0e477d8ab2f186866a6a0def15
E             + f5fa9b830dc0cf14d0b7e053997a7db627e6030d33d3678a667695420e225fac

tests/test_annotations.py:42: AssertionError
------------------------------------------------------------ Captured stdout setup -------------------------------------------------------------
Downloading https://stdpopsim.s3-us-west-2.amazonaws.com/annotations/AraTha/araport_11_exons.tar.gz
Downloading https://stdpopsim.s3-us-west-2.amazonaws.com/annotations/AraTha/araport_11_CDS.tar.gz
Downloading https://stdpopsim.s3-us-west-2.amazonaws.com/annotations/DroMel/FlyBase_BDGP6.32.51_exons.tar.gz
=============================================================== warnings summary ===============================================================
venv/lib/python3.10/site-packages/fsspec/spec.py:7
venv/lib/python3.10/site-packages/fsspec/spec.py:7
venv/lib/python3.10/site-packages/fsspec/spec.py:7
venv/lib/python3.10/site-packages/fsspec/spec.py:7
venv/lib/python3.10/site-packages/fsspec/spec.py:7
venv/lib/python3.10/site-packages/fsspec/spec.py:7
venv/lib/python3.10/site-packages/fsspec/spec.py:7
venv/lib/python3.10/site-packages/fsspec/spec.py:7
  /home/grg/src/stdpopsim/venv/lib/python3.10/site-packages/fsspec/spec.py:7: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
    from distutils.version import LooseVersion

tests/test_dfes.py: 5 warnings
tests/test_masking.py: 2 warnings
tests/test_slim_engine.py: 4 warnings
  /home/grg/src/stdpopsim/venv/lib/python3.10/site-packages/msprime/ancestry.py:831: TimeUnitsMismatchWarning: The initial_state has time_units=ticks but time is measured in generations in msprime. This may lead to significant discrepancies between the timescales. If you wish to suppress this warning, you can use, e.g., warnings.simplefilter('ignore', msprime.TimeUnitsMismatchWarning)
    warnings.warn(message, TimeUnitsMismatchWarning)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================================================== short test summary info ============================================================
ERROR tests/test_annotations.py::TestAnnotation::test_cache_dirs - AssertionError: SHA256 for _test_cache/tarballs/FlyBase_BDGP6.32.51_CDS.tar.gz doesn't match the SHA256 for FlyBase_BDGP6.32.51_CDS. If you...
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! xdist.dsession.Interrupted: stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
================================================= 1951 passed, 19 warnings, 1 error in 28.73s ==================================================

After running the tests with a single thread (pytest -n 0 -x tests/), the _test_cache was populated successfully. My guess is that multiple tests are running simultaneously that trigger a download of the same file - at least if the file doesn't exist in the cache. Downloads from AWS us-west servers aren't fast (at least not for me, right now, #1414), which may contribute to an increased chance of encountering this problem.