Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Xfail flaky ontobee tests, unless running daily tests #1423

Merged
merged 2 commits into from
Mar 25, 2024
Merged

Xfail flaky ontobee tests, unless running daily tests #1423

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
408f795
Xfail flaky ontobee tests, unless running daily tests
jwodder Mar 22, 2024
6e33659
Put xfail marker in `mark_xfail_ontobee` variable
jwodder Mar 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,10 @@ jobs:
echo HOME=/tmp/nfsmount/home >> "$GITHUB_ENV"
echo DANDI_DEVEL_INSTRUMENT_REQUESTS_SUPERLEN=1 >> "$GITHUB_ENV"

- name: Use scheduled test configuration
if: github.event_name == 'schedule'
run: echo PYTEST_ADDOPTS=--schedule >> "$GITHUB_ENV"

- name: Run all tests
if: matrix.mode != 'dandi-api'
run: |
Expand Down
6 changes: 6 additions & 0 deletions dandi/conftest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,12 @@ def pytest_addoption(parser: Parser) -> None:
default=False,
help="Only run tests of the new Django Dandi API",
)
parser.addoption(
"--scheduled",
action="store_true",
default=False,
help="Use configuration for a scheduled daily test run",
)


def pytest_collection_modifyitems(items: list[Item], config: Config) -> None:
Expand Down
19 changes: 19 additions & 0 deletions dandi/tests/test_metadata.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@
from dateutil.tz import tzutc
from pydantic import ByteSize
import pytest
import requests
from semantic_version import Version

from .fixtures import SampleDandiset
Expand Down Expand Up @@ -234,6 +235,12 @@ def test_timedelta2duration(td: timedelta, duration: str) -> None:
assert timedelta2duration(td) == duration


@pytest.mark.xfail(
condition="not config.getoption('--scheduled')",
reason="Flaky ontobee site",
strict=False,
raises=requests.RequestException,
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you please place this into @mark.xfail_ontobee similarly to @mark.skipif_no_network to avoid duplication/clutter?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jwodder ping on this one so we could bring our tests back into greenland

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yarikoptic The custom mark machinery only seems to support custom skipif decorators, not xfail. I think creating a custom mark.xfail_ontobee would be more trouble than it's worth.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, ok

What about just binding decorator into a helper "variable" eg @mark_xfail_ontobee or that wouldn't work?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems to work, but now the tests are failing due to #1424.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, since seems to be unrelated -- let's proceed. Thank you @jwodder

@mark.skipif_no_network
@pytest.mark.parametrize(
"filename, metadata",
Expand Down Expand Up @@ -459,6 +466,12 @@ def test_time_extract_gest() -> None:
)


@pytest.mark.xfail(
condition="not config.getoption('--scheduled')",
reason="Flaky ontobee site",
strict=False,
raises=requests.RequestException,
)
@mark.skipif_no_network
@pytest.mark.obolibrary
@pytest.mark.parametrize(
Expand Down Expand Up @@ -489,6 +502,12 @@ def test_parseobourl(url, value):
assert parse_purlobourl(url) == value


@pytest.mark.xfail(
condition="not config.getoption('--scheduled')",
reason="Flaky ontobee site",
strict=False,
raises=requests.RequestException,
)
@pytest.mark.obolibrary
@mark.skipif_no_network
def test_species():
Expand Down