Really unstable tests in the openSUSE Build Service

[mcepl]

We have plenty of troubles with running the test suite in the openSUSE Build System (see https://build.opensuse.org/package/show/devel:languages:python/python-aiohttp for the current situation), many (even majority) of runs ends up bizarre internal errors like:

[   47s] + PYTHONPATH=/home/abuild/rpmbuild/BUILDROOT/python-aiohttp-3.9.1-120.1.x86_64/usr/lib64/python3.9/site-packages
[   47s] + PYTHONDONTWRITEBYTECODE=1
[   47s] + pytest-3.9 --ignore=_build.python39 --ignore=_build.python310 --ignore=_build.python311 -v -n 4 tests -k 'not (test_aiohttp_request_coroutine or test_mark_formdata_as_processe
d or test_aiohttp_plugin_async or test_secure_https_proxy_absolute_path or test_client_session_timeout_zero or test_requote_redirect_url_default or test_https_proxy_unsupported_tls_in_tl
s or test_circular_imports test_import_time )'
[   52s] ============================= test session starts ==============================
[   52s] platform linux -- Python 3.9.18, pytest-7.4.3, pluggy-1.3.0 -- /usr/bin/python3.9
[   52s] cachedir: .pytest_cache
[   52s] rootdir: /home/abuild/rpmbuild/BUILD/aiohttp-3.9.1
[   52s] configfile: setup.cfg
[   52s] plugins: time-machine-2.13.0, timeout-2.2.0, mock-3.12.0, xdist-3.4.0
[   52s] created: 4/4 workers
[   52s] 4 workers [2807 items]
[   52s] 
[   52s] scheduling tests via LoadScheduling
[   52s] INTERNALERROR> Traceback (most recent call last):
[   52s] INTERNALERROR>   File "/usr/lib/python3.9/site-packages/_pytest/main.py", line 271, in wrap_session
[   52s] INTERNALERROR>     session.exitstatus = doit(config, session) or 0
[   52s] INTERNALERROR>   File "/usr/lib/python3.9/site-packages/_pytest/main.py", line 325, in _main
[   52s] INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
[   52s] INTERNALERROR>   File "/usr/lib/python3.9/site-packages/pluggy/_hooks.py", line 493, in __call__
[   52s] INTERNALERROR>     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
[   52s] INTERNALERROR>   File "/usr/lib/python3.9/site-packages/pluggy/_manager.py", line 115, in _hookexec
[   52s] INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
[   52s] INTERNALERROR>   File "/usr/lib/python3.9/site-packages/pluggy/_callers.py", line 152, in _multicall
[   52s] INTERNALERROR>     return outcome.get_result()
[   52s] INTERNALERROR>   File "/usr/lib/python3.9/site-packages/pluggy/_result.py", line 114, in get_result
[   52s] INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
[   52s] INTERNALERROR>   File "/usr/lib/python3.9/site-packages/pluggy/_callers.py", line 77, in _multicall
[   52s] INTERNALERROR>     res = hook_impl.function(*args)
[   52s] INTERNALERROR>   File "/usr/lib/python3.9/site-packages/xdist/dsession.py", line 123, in pytest_runtestloop
[   52s] INTERNALERROR>     self.loop_once()
[   52s] INTERNALERROR>   File "/usr/lib/python3.9/site-packages/xdist/dsession.py", line 148, in loop_once
[   52s] INTERNALERROR>     call(**kwargs)
[   52s] INTERNALERROR>   File "/usr/lib/python3.9/site-packages/xdist/dsession.py", line 187, in worker_workerfinished
[   52s] INTERNALERROR>     assert not crashitem, (crashitem, node)
[   52s] INTERNALERROR> AssertionError: ('tests/test_base_protocol.py::test_loop', <WorkerController gw3>)
[   52s] INTERNALERROR> assert not 'tests/test_base_protocol.py::test_loop'
[   52s] 
[   52s] ============================ no tests ran in 4.89s =============================

or

[   62s] + PYTHONPATH=/home/abuild/rpmbuild/BUILDROOT/python-aiohttp-3.9.1-120.1.i386/usr/lib/python3.9/site-packages
[   62s] + PYTHONDONTWRITEBYTECODE=1
[   62s] + pytest-3.9 --ignore=_build.python39 --ignore=_build.python310 --ignore=_build.python311 -v -n 4 tests -k 'not (test_aiohttp_request_coroutine or test_mark_formdata_as_processed or test_aiohttp_plugin_async or test_secure_https_proxy_absolute_path or test_client_session_timeout_zero or test_requote_redirect_url_default or test_https_proxy_unsupported_tls_in_tls or test_circular_imports test_import_time )'
[   68s] ============================= test session starts ==============================
[   68s] platform linux -- Python 3.9.18, pytest-7.4.3, pluggy-1.3.0 -- /usr/bin/python3.9
[   68s] cachedir: .pytest_cache
[   68s] rootdir: /home/abuild/rpmbuild/BUILD/aiohttp-3.9.1
[   68s] configfile: setup.cfg
[   68s] plugins: timeout-2.2.0, mock-3.12.0, xdist-3.4.0, time-machine-2.13.0
[   68s] created: 4/4 workers
[   68s] 4 workers [2807 items]
[   68s] 
[   68s] scheduling tests via LoadScheduling
[   68s] INTERNALERROR> Traceback (most recent call last):
[   68s] INTERNALERROR>   File "/usr/lib/python3.9/site-packages/_pytest/main.py", line 271, in wrap_session
[   68s] INTERNALERROR>     session.exitstatus = doit(config, session) or 0
[   68s] INTERNALERROR>   File "/usr/lib/python3.9/site-packages/_pytest/main.py", line 325, in _main
[   68s] INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
[   68s] INTERNALERROR>   File "/usr/lib/python3.9/site-packages/pluggy/_hooks.py", line 493, in __call__
[   68s] INTERNALERROR>     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
[   68s] INTERNALERROR>   File "/usr/lib/python3.9/site-packages/pluggy/_manager.py", line 115, in _hookexec
[   68s] INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
[   68s] INTERNALERROR>   File "/usr/lib/python3.9/site-packages/pluggy/_callers.py", line 152, in _multicall
[   68s] INTERNALERROR>     return outcome.get_result()
[   68s] INTERNALERROR>   File "/usr/lib/python3.9/site-packages/pluggy/_result.py", line 114, in get_result
[   68s] INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
[   68s] INTERNALERROR>   File "/usr/lib/python3.9/site-packages/pluggy/_callers.py", line 77, in _multicall
[   68s] INTERNALERROR>     res = hook_impl.function(*args)
[   68s] INTERNALERROR>   File "/usr/lib/python3.9/site-packages/xdist/dsession.py", line 123, in pytest_runtestloop
[   68s] INTERNALERROR>     self.loop_once()
[   68s] INTERNALERROR>   File "/usr/lib/python3.9/site-packages/xdist/dsession.py", line 148, in loop_once
[   68s] INTERNALERROR>     call(**kwargs)
[   68s] INTERNALERROR>   File "/usr/lib/python3.9/site-packages/xdist/dsession.py", line 273, in worker_collectionfinish
[   68s] INTERNALERROR>     self.sched.schedule()
[   68s] INTERNALERROR>   File "/usr/lib/python3.9/site-packages/xdist/scheduler/load.py", line 278, in schedule
[   68s] INTERNALERROR>     self._send_tests(node, node_chunksize)
[   68s] INTERNALERROR>   File "/usr/lib/python3.9/site-packages/xdist/scheduler/load.py", line 290, in _send_tests
[   68s] INTERNALERROR>     node.send_runtest_some(tests_per_node)
[   68s] INTERNALERROR>   File "/usr/lib/python3.9/site-packages/xdist/workermanage.py", line 298, in send_runtest_some
[   68s] INTERNALERROR>     self.sendcommand("runtests", indices=indices)
[   68s] INTERNALERROR>   File "/usr/lib/python3.9/site-packages/xdist/workermanage.py", line 317, in sendcommand
[   68s] INTERNALERROR>     self.channel.send((name, kwargs))
[   68s] INTERNALERROR>   File "/usr/lib/python3.9/site-packages/execnet/gateway_base.py", line 810, in send
[   68s] INTERNALERROR>     raise OSError(f"cannot send to {self!r}")
[   68s] INTERNALERROR> OSError: cannot send to <Channel id=3 closed>
[   68s] 
[   68s] ============================ no tests ran in 5.98s ========================

(presented errors are always different).

Do you have any idea what could be wrong? Do you have special requirements for the tests to be run (huge amount of CPU, disk space, memory)?

Complete build log from the first error

[Dreamsorcerer]

Looks weird, but I'd try running against the 3.9 branch. We've had issues with time-machine and have now reverted back to freezegun, so double check if that makes a difference.

Otherwise, I've never seen errors like that before and can't see how they even relate to our code...

[webknjaz]

@mcepl I see you use pytest-xdist. Up until today it didn't support asyncio. See pytest-dev/pytest-xdist#620