System tests on k8s #793
System tests on k8s #793
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #793 +/- ##
=======================================
Coverage 86.93% 86.94%
=======================================
Files 102 102
Lines 6967 6971 +4
=======================================
+ Hits 6057 6061 +4
Misses 910 910
|
rtuck99
force-pushed
the
633_system_tests_on_k8s
branch
2 times, most recently
from
6ed7f61 to
b02c969
Compare
| @@ -165,7 +166,7 @@ legacy_tox_ini = """ | |||
| [tox] | |||
| skipsdist=True | |||
|
|
|||
| [testenv:{pre-commit,type-checking,tests,docs}] | |||
| [testenv:{pre-commit,type-checking,tests,docs,systemtests}] | |||
There was a problem hiding this comment.
Nit: We've been calling it system-test in other places, not really bothered but would be nice to be consistent
There was a problem hiding this comment.
Unfortunately, dashes have a special meaning within tox:
https://tox.wiki/en/latest/user_guide.html
Test environment names can consist of alphanumeric characters and dashes; for example: py311-django42. The name will be split on dashes into multiple factors, meaning py311-django42 will be split into two factors: py311 and django42.
There was a problem hiding this comment.
Hmm, should we also change type-checking and pre-commit then?
| @@ -0,0 +1,20 @@ | |||
| System Tests | |||
There was a problem hiding this comment.
If there are any system tests which we permanently want to leave in the option to locally run them, we should explain how to run them here
There was a problem hiding this comment.
I think long term the plan would be that rather than run locally directly against the ispyb dev database + expeye staging, we would run them locally against the test instances running in podman. Then we could get rid of the various ST_xxx config options.
This is what I'm currently doing, however there are a few minor wrinkles with podman that I have run into.
It's somewhat complicated:
- There are 3 or 4 different ways of running compose files:
- podman with podman-compose (what I'm currently using)
- podman with docker-compose plugin (I think technically this is now the more actively developed solution)
- docker with docker-compose standalone (Now deprecated, I didn't try this)
- docker with docker-compose plugin (current docker solution)
- I don't know if podman with docker-compose standalone works, I didn't try
However I seem to run into slightly different problems depending which of the above I'm using. Note that a lot of these issues are also dependent on how you are running podman/docker and whether they are running with root privileges. So it's doubly complicated as if I ran them on my laptop or at home instead of my workstation I'd probably get different results.
- podman with podman-compose - I get caching issues with images not getting always getting pulled from gcr.io, you can work around by deleting them but this is annoying. However with the compose file I have (using the default bridging network), it doesn't expose the services to the host so you can't run the tests locally in the debugger. Unfortunately if you try host networking, then you run into other problems in that the containers can't talk to each other bc no hostnames.
- podman with docker-cli - I ran into docker strangeness as it seems to use the docker local repository and also authentication. But also ran into similar but slightly different problems with networking. I think a lot of this is possibly due to not having root therefore it doesn't set up routing properly.
- docker I didn't try but assume similar problems w/o root.
Bottom line is, for ideal local dev scenario, the test containers need to be routable from the outside so that you can run individual python test modules locally against the test service instances. The services also need to be routable to each other.
If you want to run the test container locally as well it needs to be able to reach the outside in order to pull from GH/pypi
The solution I have at the moment is to run in podman with the test instance via the bridging network but able to mount a local RO git repo, this is reasonable but to use it you have to shell in to the container.
It is definitely possible to get what I want with docker as I have pretty much exactly this setup on my home server, but there I have root and also the server is configured as a router. </end of rant>
| @pytest.fixture(autouse=True) | ||
| def use_dev_ispyb_unless_overridden_by_environment(): | ||
| ispyb_config_path = os.environ.get( | ||
| "ISPYB_CONFIG_PATH", CONST.SIM.DEV_ISPYB_DATABASE_CFG |
There was a problem hiding this comment.
Where is ISPYB_CONFIG_PATH set? I can't find a corresponding name in the Dockerfiles
There was a problem hiding this comment.
We don't override it for the containerised system tests - the test credentials are mounted to the default place.
… was not working reliably
Move from podman to podman-compose
Helmcharts for deployment to k8s cluster Updates to modify secrets handling
Allow fixtures to be configurable to match the test db
rtuck99
force-pushed
the
633_system_tests_on_k8s
branch
from
b02c969 to
40cc1c3
Compare
This addresses
The changes here mainly consist of changes to the unit tests to allow the ISPyB endpoint to be configured via the
ISPYB_CONFIG_PATHduring tests, which was previously overridden.Also there are changes to allow sample ID, visit and suchlike to be configured as an interim so that it is still possible to run the system tests individually locally against the
ispyb_hyperion_dev.cfgdatabase.The actual system test infrastructure can be found in the associated GitLab merge request