Use stagemole with e2e tests

[niklasberglund]

Using stagemole for e2e tests by default. Can optionally specify to use prod. Artefacts will be uploaded if testing with stagemole.

Also added very basic error handling for 429 rate limited error responses(very basic since we probably will migrate to ktor) and moved pulling of outputs from run-instrumented-tests.sh to the workflow, so that artefacts are uploaded even if the run is cancelled or tests step time out. Fixes DROID-1635

---

This change is 

[linear]

DROID-1635 Pull test outputs even when cancelled/reaching timeout

DROID-1561 Use stagemole in self-hosted e2e tests

Copilot reviewed 1 out of 4 changed files in this pull request and generated no comments.

Files not reviewed (3)

• android/scripts/pull-test-output.sh: Language not supported
• android/scripts/run-instrumented-tests.sh: Language not supported
• android/test/e2e/src/main/kotlin/net/mullvad/mullvadvpn/test/e2e/misc/SimpleMullvadHttpClient.kt: Language not supported

[Pururun]

Reviewed 4 of 4 files at r1, all commit messages.
Reviewable status: complete! all files reviewed, all discussions resolved

[albin-mullvad]

Reviewed 3 of 4 files at r1, all commit messages.
Reviewable status: all files reviewed, 7 unresolved discussions (waiting on @niklasberglund)

---

android/scripts/run-instrumented-tests.sh line 112 at r1 (raw file):

        echo "Error: The 'e2e' test type with billing flavor 'play' require infra flavor 'stagemole'."
        exit 1
    elif [[ $BILLING_FLAVOR == "oss" && $INFRA_FLAVOR != "prod" ]]; then

Should this block really be removed? Didn't we say to keep using the play build? 🤔

---

.github/workflows/android-app.yml line 51 at r1 (raw file):

      e2e_tests_infra_flavor:
        description: >
          Infra environment to run e2e tests on(prod/stagemole).

nit: add space on (prod/stagemole)

Code quote:

on(prod/stagemole).

---

.github/workflows/android-app.yml line 351 at r1 (raw file):

      - name: Build stagemole app
        uses: burrunan/gradle-cache-action@v1
        if: github.event.inputs.e2e_test_repeat != '0'

Can we check e2e_tests_infra_flavor instead?

Code quote:

github.event.inputs.e2e_test_repeat != '0'

---

.github/workflows/android-app.yml line 494 at r1 (raw file):

      - name: Upload instrumentation report (${{ matrix.test-type }})
        uses: actions/upload-artifact@v4
        if: always() && matrix.test-repeat != 0 && github.event.inputs.e2e_tests_infra_flavor == 'stagemole'

Do we need this check? These tests doesn't include anything sensitive since it's only used for isolated tests atm (unless we decide to merge instrumented-tests & instrumented-e2e-tests

Code quote:

 && github.event.inputs.e2e_tests_infra_flavor == 'stagemole'

---

.github/workflows/android-app.yml line 501 at r1 (raw file):

          retention-days: 7

  instrumented-e2e-tests:

Could it make sense to merge this one with instrumented-tests now that the difference between the two is less than before? I believe the main thing to consider is that we would only want to set some of the env variables for the e2e tests (such as PARTNER_AUTH).

Code quote:

instrumented-e2e-tests

---

android/scripts/pull-test-output.sh line 35 at r1 (raw file):

LOCAL_LOGCAT_FILE_PATH="$REPORT_DIR/logcat.txt"
LOCAL_SCREENSHOT_PATH="$REPORT_DIR/screenshots"
LOCAL_TEST_ARRACHMENTS_PATH="$REPORT_DIR/test-attachments"

Would be really nice if we could just have a single on-device path for all this content such as /sdcard/test-report that we just pull recursively into $REPORT_DIR instead of a bunch of paths that we now have to keep track of in two different scripts. But probably a bit out-of-scope for this PR...

Code quote:

DEVICE_SCREENSHOT_PATH="/sdcard/Pictures/mullvad-$TEST_TYPE"
DEVICE_TEST_ATTACHMENTS_PATH="/sdcard/Download/test-attachments"
LOCAL_LOGCAT_FILE_PATH="$REPORT_DIR/logcat.txt"
LOCAL_SCREENSHOT_PATH="$REPORT_DIR/screenshots"
LOCAL_TEST_ARRACHMENTS_PATH="$REPORT_DIR/test-attachments"

---

android/scripts/pull-test-output.sh line 38 at r1 (raw file):

echo "Collecting report and produced test attachments..."
adb logcat -d > "$LOCAL_LOGCAT_FILE_PATH"

The other script that runs the tests should probably be responsible for dumping the logcat content into a file on the device. This script can then be responsible for downloading that file.

Code quote:

adb logcat -d > "$LOCAL_LOGCAT_FILE_PATH"

[niklasberglund]

Reviewable status: 1 of 8 files reviewed, 7 unresolved discussions (waiting on @albin-mullvad and @Pururun)

---

.github/workflows/android-app.yml line 51 at r1 (raw file):

Previously, albin-mullvad wrote…

nit: add space on (prod/stagemole)

Added space. Also added to mockapi_test_repeat and e2e_test_repeat descriptions

---

.github/workflows/android-app.yml line 351 at r1 (raw file):

Previously, albin-mullvad wrote…

Can we check e2e_tests_infra_flavor instead?

Now checking e2e_test_repeat, run_firebase_tests and e2e_tests_infra_flavor

---

.github/workflows/android-app.yml line 494 at r1 (raw file):

Previously, albin-mullvad wrote…

Do we need this check? These tests doesn't include anything sensitive since it's only used for isolated tests atm (unless we decide to merge instrumented-tests & instrumented-e2e-tests

Removed the stagemole check

---

.github/workflows/android-app.yml line 501 at r1 (raw file):

Previously, albin-mullvad wrote…

Could it make sense to merge this one with instrumented-tests now that the difference between the two is less than before? I believe the main thing to consider is that we would only want to set some of the env variables for the e2e tests (such as PARTNER_AUTH).

Created triage issue for this. We had an offline talk about keeping it outside of this PR.

---

android/scripts/pull-test-output.sh line 35 at r1 (raw file):

Previously, albin-mullvad wrote…

Would be really nice if we could just have a single on-device path for all this content such as /sdcard/test-report that we just pull recursively into $REPORT_DIR instead of a bunch of paths that we now have to keep track of in two different scripts. But probably a bit out-of-scope for this PR...

Now all files are under /sdcard/Downloads/test-outputs

---

android/scripts/pull-test-output.sh line 38 at r1 (raw file):

Previously, albin-mullvad wrote…

The other script that runs the tests should probably be responsible for dumping the logcat content into a file on the device. This script can then be responsible for downloading that file.

Dumping the content in the other file now

---

android/scripts/run-instrumented-tests.sh line 112 at r1 (raw file):

Previously, albin-mullvad wrote…

Should this block really be removed? Didn't we say to keep using the play build? 🤔

Brought this back

[niklasberglund]

Reviewable status: 1 of 8 files reviewed, 8 unresolved discussions (waiting on @albin-mullvad and @Pururun)

---

.github/workflows/android-app.yml line 560 at r2 (raw file):

          INFRA_FLAVOR: ${{ github.event.inputs.e2e_tests_infra_flavor }}
          PARTNER_AUTH: ${{ github.event.inputs.e2e_tests_infra_flavor == 'stagemole' && secrets.STAGEMOLE_PARTNER_AUTH || '' }}
          VALID_TEST_ACCOUNT_NUMBER: ${{ github.event.inputs.e2e_tests_infra_flavor == 'prod' && secrets.ANDROID_PROD_TEST_ACCOUNT || '' }}

The two lines above are too long and should be split into multiple lines, working on splitting them without breaking the code

[niklasberglund]

Reviewable status: 1 of 8 files reviewed, 9 unresolved discussions (waiting on @albin-mullvad and @Pururun)

---

android/test/common/src/main/kotlin/net/mullvad/mullvadvpn/test/common/rule/CaptureScreenshotOnFailedTestRule.kt line 20 at r2 (raw file):

import java.time.OffsetDateTime
import java.time.temporal.ChronoUnit
import net.mullvad.mullvadvpn.test.common.misc.Attachment

Only using Attachment for the outputs path right now. Should probably also use it for storing screenshots though?

[albin-mullvad]

Reviewed 4 of 7 files at r2, all commit messages.
Reviewable status: 4 of 8 files reviewed, 6 unresolved discussions (waiting on @niklasberglund and @Pururun)

---

android/scripts/pull-test-output.sh line 5 at r3 (raw file):

set -eu

TEST_DEVICE_OUTPUTS_DIR="${TEST_DEVICE_OUTPUTS_DIR:-/sdcard/Download/test-outputs/attachments}" # Must match the path where e2e tests output their attachments

Should we not skip /attachments here?

Code quote:

/attachments

---

android/scripts/pull-test-output.sh line 5 at r3 (raw file):

set -eu

TEST_DEVICE_OUTPUTS_DIR="${TEST_DEVICE_OUTPUTS_DIR:-/sdcard/Download/test-outputs/attachments}" # Must match the path where e2e tests output their attachments

nit: move comment to the line above

Code quote:

# Must match the path where e2e tests output their attachments

---

android/test/common/src/main/kotlin/net/mullvad/mullvadvpn/test/common/misc/Attachment.kt line 10 at r3 (raw file):

object Attachment {
    val DIRECTORY_PATH = "${Environment.DIRECTORY_DOWNLOADS}/test-outputs/attachments"

Can we define this test-output path in a single place?

Code quote:

${Environment.DIRECTORY_DOWNLOADS}/test-outputs

---

android/test/common/src/main/kotlin/net/mullvad/mullvadvpn/test/common/misc/CaptureScreenRecordingsExtension.kt line 58 at r3 (raw file):

    companion object {
        val OUTPUT_DIRECTORY =
            "${Environment.getExternalStorageDirectory().path}/Download/test-outputs/attachments/video"

Can we define this test-output path in a single place?

Code quote:

${Environment.getExternalStorageDirectory().path}/Download/test-outputs/

[Rawa]

Reviewed 2 of 2 files at r4, all commit messages.
Reviewable status: 6 of 8 files reviewed, 8 unresolved discussions (waiting on @albin-mullvad, @niklasberglund, and @Pururun)

---

.github/workflows/android-app.yml line 491 at r4 (raw file):

      - name: Pull test report
        if: always() && matrix.test-repeat != 0 && github.event.inputs.e2e_tests_infra_flavor == 'stagemole'

Is there a reason for not uploading test report if flavor was prod? We don't have test reports if it is prod?

---

.github/workflows/android-app.yml line 494 at r4 (raw file):

        shell: bash -ieo pipefail {0}
        env:
          TEST_DEVICE_OUTPUTS_DIR: '/sdcard/Download/test-outputs'

TEST_DEVICE_OUTPUTS_DIR: '/sdcard/Download/test-outputs'
We define the same variable in 4 places, can we set it in one common place so we don't repeat?

[Rawa]

Reviewed 1 of 4 files at r1, 2 of 7 files at r2.
Reviewable status: 7 of 8 files reviewed, 8 unresolved discussions (waiting on @niklasberglund)

[Rawa]

Reviewed 1 of 7 files at r2.
Reviewable status: all files reviewed, 8 unresolved discussions (waiting on @niklasberglund)

[Rawa]

Reviewed 1 of 7 files at r2.
Reviewable status: all files reviewed, 8 unresolved discussions (waiting on @niklasberglund)

[niklasberglund]

Reviewable status: 3 of 8 files reviewed, 5 unresolved discussions (waiting on @albin-mullvad and @Rawa)

---

.github/workflows/android-app.yml line 491 at r4 (raw file):

Previously, Rawa (David Göransson) wrote…

Is there a reason for not uploading test report if flavor was prod? We don't have test reports if it is prod?

Yes correct, before we didn't upload any test reports ever but now we will upload when tests run against staging environment since then account number etc is OK to "leak".

---

.github/workflows/android-app.yml line 494 at r4 (raw file):

Previously, Rawa (David Göransson) wrote…

TEST_DEVICE_OUTPUTS_DIR: '/sdcard/Download/test-outputs'
We define the same variable in 4 places, can we set it in one common place so we don't repeat?

Good point. Now setting it in TEST_OUTPUTS_PATH on workflow-level which is passed to steps env's TEST_DEVICE_OUTPUTS_DIR. I'm not sure but maybe it is possible to set TEST_OUTPUTS_PATH on workflow level directly and not set it per step but I'm thinking it would make the code harder to follow.

---

android/scripts/pull-test-output.sh line 5 at r3 (raw file):

Previously, albin-mullvad wrote…

Should we not skip /attachments here?

Yes. Removed

---

android/scripts/pull-test-output.sh line 5 at r3 (raw file):

Previously, albin-mullvad wrote…

nit: move comment to the line above

Moved

---

android/test/common/src/main/kotlin/net/mullvad/mullvadvpn/test/common/misc/Attachment.kt line 10 at r3 (raw file):

Previously, albin-mullvad wrote…

Can we define this test-output path in a single place?

Defined in Attachment now under DIRECTORY_NAME.

---

android/test/common/src/main/kotlin/net/mullvad/mullvadvpn/test/common/misc/CaptureScreenRecordingsExtension.kt line 58 at r3 (raw file):

Previously, albin-mullvad wrote…

Can we define this test-output path in a single place?

Defined in Attachment now under DIRECTORY_NAME.

---

android/scripts/run-instrumented-tests.sh line 239 at r5 (raw file):

    -e runnerBuilder de.mannodermaus.junit5.AndroidJUnit5Builder \
    ${OPTIONAL_TEST_ARGUMENTS:-""} \
    androidx.test.orchestrator/androidx.test.orchestrator.AndroidTestOrchestrator"

CI runs are currently failing because of invalid syntax here I believe. Haven't found the cause, but maybe $INSTRUMENTATION_COMMAND or $GRADLE_ENVIRONMENT_VARIABLES is misformatted?