Skip to content

Commit

Permalink
[9.18] chg: ci: Set stricter limits for respdiff testing
Browse files Browse the repository at this point in the history
Adjust the limit of maximum disagreements in respdiff results based on
recent pipeline results.

The respdiff and respdiff:asan seem to have almost identical results,
typically around 0.07 % of differences with ocassional spikes up to
around 0.11 %. Similar results are for respdiff:tsan, perhaps with more
common spikes with values up to around 0.12 %. Set the limit to 0.15 %
to allow for some tolerance due to network conditions, time of day etc.

The respdiff:third-party has a slightly higher disagreements average,
with typical values being around 0.12 %. Set the limit to 0.2 %.

Exceeding either of those values should be quite clear indication that
some resolution behaviour has changed, since the values appear to be
very stable within the newly configured limits.

Backport of MR !9950

Merge branch 'backport-nicki/ci-respdiff-limits-9.18' into 'bind-9.18'

See merge request isc-projects/bind9!9990
  • Loading branch information
nicki-krizek committed Jan 23, 2025
2 parents 2c8d9c4 + 1c1a7aa commit 46be0e9
Showing 1 changed file with 4 additions and 4 deletions.
8 changes: 4 additions & 4 deletions .gitlab-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1514,7 +1514,7 @@ respdiff:
variables:
CC: gcc
CFLAGS: "${CFLAGS_COMMON} -Og"
MAX_DISAGREEMENTS_PERCENTAGE: "0.5"
MAX_DISAGREEMENTS_PERCENTAGE: "0.15"
script:
- bash respdiff.sh -s named -q "${PWD}/100k_mixed.txt" -c 3 -w "${PWD}/rspworkdir" "${CI_PROJECT_DIR}" "/usr/local/respdiff-reference-bind/sbin/named"

Expand All @@ -1527,7 +1527,7 @@ respdiff:asan:
CFLAGS: "${CFLAGS_COMMON} -Og -fsanitize=address,undefined"
LDFLAGS: "-fsanitize=address,undefined"
EXTRA_CONFIGURE: "--disable-dnsrps --without-jemalloc"
MAX_DISAGREEMENTS_PERCENTAGE: "0.5"
MAX_DISAGREEMENTS_PERCENTAGE: "0.15"
script:
- bash respdiff.sh -s named -q "${PWD}/100k_mixed.txt" -c 3 -w "${PWD}/rspworkdir" "${CI_PROJECT_DIR}" "/usr/local/respdiff-reference-bind/sbin/named"

Expand All @@ -1542,7 +1542,7 @@ respdiff:tsan:
OPENSSL_CFLAGS: -I/opt/tsan/include
OPENSSL_LIBS: -L/opt/tsan/lib -lssl -lcrypto
EXTRA_CONFIGURE: "--disable-dnsrps --enable-pthread-rwlock --without-jemalloc PKG_CONFIG_PATH=/opt/tsan/lib/pkgconfig"
MAX_DISAGREEMENTS_PERCENTAGE: "0.5"
MAX_DISAGREEMENTS_PERCENTAGE: "0.15"
TSAN_OPTIONS: "${TSAN_OPTIONS_DEBIAN}"
script:
- bash respdiff.sh -s named -q "${PWD}/100k_mixed.txt" -c 3 -w "${PWD}/rspworkdir" "${CI_PROJECT_DIR}" "/usr/local/respdiff-reference-bind/sbin/named"
Expand All @@ -1557,7 +1557,7 @@ respdiff-third-party:
variables:
CC: gcc
CFLAGS: "${CFLAGS_COMMON} -Og"
MAX_DISAGREEMENTS_PERCENTAGE: "0.5"
MAX_DISAGREEMENTS_PERCENTAGE: "0.2"
script:
- bash respdiff.sh -s third_party -q "${PWD}/100k_mixed.txt" -c 1 -w "${PWD}/rspworkdir" "${CI_PROJECT_DIR}"

Expand Down

0 comments on commit 46be0e9

Please sign in to comment.