Accommodations for running tests in containers that don't match the h…

…ost OS (#1834)

* Set object store URL in tests using an env var

The S3 endpoint URL used in the unit tests was hard-coded to localhost.
When executing the tests in a container, a minio sibling container can't
be accessed via localhost. This allows setting the endpoint URL via an
environment variable, and adds 'BANDERSNATCH_*' to the environment
variable passthrough list for Tox.

...noticing that Tox was the reason the environment variable was being
ignored took a very long time.

* Fix line endings for storage plugin sample file test

Previously the test_get_hash test used shutil.copy to duplicate a sample file
from the project directory into a temp directory for the test.

If the project folder is cloned on Windows then bind-mounted into
a Linux container, with the test run in the container, the file
would have Windows line endings while the test would expect unix
line endings based on sys.platform within the container.

Instead of copying the file, this change writes the file from a string
value using Python's universal newlines support, so the written file
will have the same line ending as the platform executing the test.

This change trips mypy errors in the pre-commit checks. The errors show
when running mypy in the console or via pre-commit, but the MyPy plugin
for VS Code *doesn't* and will make the *opposite complaint* if you try
to change the file to fix the errors shown in pre-commit. I don't know
what to do about that.

* Set some pytest file discovery settings explicitly in pytest.ini

Pytest would fail to discover any test files when run in a container
with the project folder mounted. Adding these helped, and doesn't
change anything when tests are run 'normally' as far as I can tell.

I know what each does, but I don't know why the problem was happening
in the container specifically, or why these fix it. This is a "web search
the error message" -> "find reasonable option on StackOverflow" fix.

pytest.ini

@@ -2,7 +2,11 @@
 log_cli_level = DEBUG
 log_level = DEBUG
 asyncio_mode=strict
+
+testpaths = src
+python_files = test.py test_*.py
 norecursedirs = src/bandersnatch_docker_compose
+pythonpath = .
 
 markers =
     s3: mark tests that require an S3/MinIO bucket

src/bandersnatch/tests/conftest.py

@@ -196,13 +196,14 @@ def reset_configuration_cache() -> Iterator[None]:
 def s3_mock(reset_configuration_cache: None) -> S3Path:
     if os.environ.get("os") != "ubuntu-latest" and os.environ.get("CI"):
         pytest.skip("Skip s3 test on non-posix server in github action")
+    endpoint = os.environ.get("BANDERSNATCH_S3_ENDPOINT_URL", "http://localhost:9000")
     register_configuration_parameter(
         PureS3Path("/"),
         resource=boto3.resource(
             "s3",
             aws_access_key_id="minioadmin",
             aws_secret_access_key="minioadmin",
-            endpoint_url="http://localhost:9000",
+            endpoint_url=endpoint,
         ),
     )
     new_bucket = S3Path("/test-bucket")

src/bandersnatch/tests/plugins/test_storage_plugins.py

@@ -30,7 +30,7 @@
 if TYPE_CHECKING:
     import swiftclient
 
-
+SAMPLE_FILE_CONTENT = "I am a sample!\n"
 BASE_SAMPLE_FILE = os.path.join(
     os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "sample"
 )
@@ -412,7 +412,9 @@ def setUp_backEnd(self) -> None:
             target_sample_file = f"{self.container}/{target_sample_file}"
         assert self.tempdir
         self.sample_file = os.path.join(self.tempdir.name, target_sample_file)
-        shutil.copy(BASE_SAMPLE_FILE, self.sample_file)
+        with open(self.sample_file, mode="w") as sample_file:
+            sample_file.write(SAMPLE_FILE_CONTENT)
+
         if self.backend == "swift":
             self.mirror_path = Path(mirror_path)
         else:

tox.ini

@@ -15,6 +15,7 @@ deps =
 passenv =
     os
     CI
+    BANDERSNATCH_*
 
 [testenv:doc_build]
 basepython=python3