PYTEST_ADDOPTS env var not respected for test debug configuration with debugpy

[alenzo-arch]

Type: Bug

Behaviour

Expected vs. Actual

When using pytest-cov options are present in settings.json python.testing.pytestArgs, I would expect to be able to disable pytest-cov during test debugging when using the documenations suggested workaround of setting "env": {"PYTEST_ADDOPTS": "--no-cov" } in the test configuration with purpose "debug-test". This still works when specifying "type": "python" in the launch configuration, but when specifying "type": "debugpy" as suggested by the deprication warning, it does not.

Actual behaviour when "type": "debugpy" and "env": {"PYTEST_ADDOPTS": "--no-cov" } is set in launch configuration and "--cov" is included in python.testing.pytestArgs in settings.json, the test debugger does not stop on breakpoints. If I remove "--cov" form python.testing.pytestArgs or set "type": "python" in the launch config, breakpoints work as expected. It appears that the environment variables are being properly set in the test environment.

Steps to reproduce:

1. add the follwing to settings.json "python.testing.pytestArgs": ["--cov", "tests/"]
2. set the following test debug configuration in launch.json
   {
   "name": "Python: Debug Tests",
   // "type": "python", // Works when type is set to python
   "type": "debugpy",
   "request": "launch",
   "program": "${file}",
   "purpose": [
   "debug-test"
   ],
   "env": {
   "PYTEST_ADDOPTS": "--no-cov"
   },
   "console": "integratedTerminal",
   "justMyCode": true
   },
3. set a breakpoint in a pytest function
4. run the test function via the debug panel, debug indiviual test button
5. observe that breakpoints are not stopped at
6. remove --cov from settings.json
7. run the test function via the debug panel, debug indiviual test button
8. observe that breakpoints work as expected
9. modify launch.json config to use "type": python
10. add --cov back to settings.json pytest args
11. run the test function via the debug panel, debug indiviual test button
12. observer that breakpoints work as expected

Diagnostic data

• Python version (& distribution if applicable, e.g. Anaconda): 3.11.4
• Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Venv
• Value of the python.languageServer setting: Default

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

XXX

User Settings

languageServer: "Pylance"

testing
• pytestArgs: "<placeholder>"
• pytestEnabled: true

experiments
• optInto: ["pythonTestAdapter"]

Extension version: 2023.25.10292213
VS Code version: Code - Insiders 1.86.0-insider (05047486b6df5eb8d44b2ecd70ea3bdf775fd937, 2024-01-31T03:29:26.801Z)
OS version: Linux x64 6.5.0-15-generic snap
Modes:

System Info

Item	Value
CPUs	13th Gen Intel(R) Core(TM) i7-13700 (24 x 5100)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: disabled_off direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_graphite: disabled_off video_decode: enabled video_encode: disabled_software vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: disabled_off
Load (avg)	0, 0, 0
Memory (System)	30.97GB (20.27GB free)
Process Argv	--no-sandbox --force-user-env --unity-launch --crash-reporter-id 3aa60ae4-37a9-4e56-8adf-187f0d7e643f
Screen Reader	no
VM	0%
DESKTOP_SESSION	ubuntu
XDG_CURRENT_DESKTOP	Unity
XDG_SESSION_DESKTOP	ubuntu
XDG_SESSION_TYPE	x11

A/B Experiments

vsliv368:30146709
vspor879:30202332
vspor708:30202333
vspor363:30204092
vstes516:30244333
vscod805cf:30301675
vsaa593:30376534
py29gd2263:30784851
c4g48928:30535728
962ge761:30841072
pythongtdpath:30726887
welcomedialog:30812478
pythonidxpt:30768918
pythonnoceb:30776497
asynctok:30898717
dsvsc013:30777762
dsvsc014:30777825
dsvsc015:30821418
pythontestfixt:30866404
pythonregdiag2:30926734
pyreplss1:30879911
pythonmypyd1:30859725
pythoncet0:30859736
pythontbext0:30879054
accentitlementst:30870582
dsvsc016:30879898
dsvsc017:30880771
dsvsc018:30880772
d34g3935:30953408
b5d27386:30943795
fegfb526:30952798
7j2b6412:30951517
bg6jg535:30946824
ccp1r6:30950905

[eleanorjboyd]

Hello! Could you include your logs from the python output channel? For your logs, can you first set your log level to trace via theDeveloper: set log level command in the command palette? I am specifically looking to see if we can find where the command is called to debug and see the args passed there.

@paulacamargo25 any other thoughts / suggestions on what to try to debug this?

[alenzo-arch]

Sorry for the late reply. I will work on this and add it to the issue. Thank you for following up on this.

[eleanorjboyd]

Hi! So debugging and coverage don't work together (more on that here #19322). Seems like you are adding coverage in the test args then attempting to remove it in the debug launch config environment? So it seems like the pytest arg is being respected and not the env var correct? I think this is how pytest does precedence and therefore would be the expected outcome.

[alenzo-arch]

Hi @eleanorjboyd so long story short, it appears this has been resolved in a recent update. I have modified my lauch.json to use debugpy again and I am able to get it to stop at breakpoints using the env var

I'm now on
Python extension: v2024.5.11021008 (pre-release)

Version: 1.89.0-insider
Commit: a2194e8c49790e2139c139a0d986e815ebd43e5b
Date: 2024-04-16T05:50:26.167Z
Electron: 28.2.8
ElectronBuildId: 27744544
Chromium: 120.0.6099.291
Node.js: 18.18.2
V8: 12.0.267.19-electron.0
OS: Linux x64 6.5.0-27-generic snap

Using launch config:

{
            "name": "Python: Debug Tests",
            "type": "debugpy", // This used to not work
            // "type": "python",
            "request": "launch",
            "program": "${file}",
            "purpose": [
                "debug-test"
            ],
            "env": {
                "PYTEST_ADDOPTS": "--no-cov",
            },
            "console": "integratedTerminal",
            "justMyCode": true
        },

And settings.json

...
"python.testing.pytestArgs": [
        "--cov",
        "--cov-report=html",
        "-vvv",
        "test",
]
...

I can't comment on how pytest does precedence of args vs env vars. Quick search didn't make it clear for me.

However, the VSCode documentation recommends the setup that I have (using the PYTEST_ADDOPTS env var) to disable coverage during debugging but allow coverage when running with the built in test runner. I had been using this for quite a while with

From VSCode docs:

You can also configure pytest using a pytest.ini file as described on pytest Configuration.

Note If you have the pytest-cov coverage module installed, VS Code doesn't stop at breakpoints while debugging because pytest-cov is using the same technique to access the source code being run. To prevent this behavior, include --no-cov in pytestArgs when debugging tests, for example by adding "env": {"PYTEST_ADDOPTS": "--no-cov"} to your debug configuration. (See Debug Tests above about how to set up that launch configuration.) (For more information, see Debuggers and PyCharm in the pytest-cov documentation.)

[eleanorjboyd]

Hi @alenzo-arch, glad this is working! Generally the rule should be Command-line arguments take precedence over environment variables so if you run into a situation where that is not the case please let me know and I can look into it further.

We do know that coverage and debugging doesn't work so we remove coverage args from the user args when calling debugging so it is likely having to do with that since we had an issue a few weeks ago with the arg parsing not working as desired.

Generally this problem will be helped by our proposal to allow custom arguments per run type so hopefully we will get working on that soon and that could be useful to you. Thanks!