Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update docs for running tests #12987

Merged
merged 5 commits into from
Jan 7, 2025
Merged
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 43 additions & 20 deletions docs/contributing/running-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,14 @@

Open Telemetry Auto Instrumentation's minimal supported version is java 8.
All jar files that we produce, unless noted otherwise, have bytecode
compatible with java 8 runtime. Our test suite is executed against
compatibility with the java 8 runtime. Our test suite is executed against
java 8, all LTS versions and the latest non-LTS version.

Some libraries that we auto-instrument may have higher minimal requirements.
In this case we compile and test corresponding auto-instrumentation with
higher java version as required by library. The resulting classes will have
higher bytecode level, but as it matches library's java version, no runtime
problem arise.
In these cases, we compile and test the corresponding auto-instrumentation with
higher java versions as required by the libraries. The resulting classes will
have a higher bytecode level, but since it will match the library's java version,
no runtime problems arise.

## Instrumentation tests

Expand All @@ -22,10 +22,11 @@ instrumented library.

### Executing tests with specific java version

We run all tests on Java 11 by default, along with Java 8 and 15. To run on the later, set the
`testJavaVersion` Gradle property to the desired major version, e.g., `./gradlew test -PtestJavaVersion=8`,
`./gradlew test -PtestJavaVersion=15`. If you don't have a JDK of these versions
installed, Gradle will automatically download it for you.
We run all tests on `Java 21` by default, along with Java 8, 11, 17, and 23. To run on
a specific version, set the `testJavaVersion` gradle property to the desired major
version, e.g., `./gradlew test -PtestJavaVersion=8`, `./gradlew test -PtestJavaVersion=23`.
If you don't have a JDK of these versions installed, Gradle will automatically download
it for you.

### Executing tests against the latest versions of libraries under instrumentation

Expand All @@ -36,7 +37,7 @@ To run these tests locally, add `-PtestLatestDeps=true` to your existing `gradle

### Executing single test

Executing `./gradlew :instrumentation:<INSTRUMENTATION_NAME>:test --tests <GROOVY TEST FILE NAME>` will run only the selected test.
Executing `./gradlew :instrumentation:<INSTRUMENTATION_NAME>:test --tests <TEST FILE NAME>` will run only the selected test.

### How to prevent linting and formatting warnings from failing tests

Expand All @@ -52,7 +53,7 @@ The `dev` flag will ignore warnings in tests.

## Smoke tests

The smoke tests are not run as part of a global `test` task run since they take a long time and are
The smoke tests are not run as part of a global `test` task since they take a long time and are
not relevant for most contributions. Explicitly specify `:smoke-tests:test` to run them.

If you need to run a specific smoke test suite:
Expand All @@ -73,11 +74,10 @@ If you want to run a specific smoke test:
./gradlew :smoke-tests:test --tests '*SpringBootSmokeTest*'
```

## Smoke OpenTelemetry starter tests
## OpenTelemetry starter smoke tests

Smoke tests for the [OpenTelemetry Spring starter](../../instrumentation/spring/starters/spring-boot-starter/README.md).

You can execute the tests in a JVM (`./gradlew smoke-tests-otel-starter:test`) or as Spring native tests (`./gradlew smoke-tests-otel-starter:nativeTest`).
Smoke tests for the [OpenTelemetry Spring starter](../../instrumentation/spring/starters/spring-boot-starter/README.md)
can be executed in a JVM (`./gradlew smoke-tests-otel-starter:test`) or as Spring Native tests (`./gradlew smoke-tests-otel-starter:nativeTest`).

## GraalVM native test

Expand All @@ -100,22 +100,45 @@ old containers, images and volumes using `docker system prune --volumes`.
For some container environments, such as rootless Podman or a remotely hosted Docker,
testcontainers may need additional configuration to successfully run the tests.
The following environment variables can be used for configuration:
- `TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE` - The location of the Docker socket on the host. Default is `/var/run/docker.sock`
- `TESTCONTAINERS_HOST_OVERRIDE` - The hostname used for container-to-container communication. Default Docker is `localhost`, but rootless Podman uses `host.containers.internal`
- `TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE` - The location of the Docker socket on the host. Default is `/var/run/docker.sock`
- `TESTCONTAINERS_HOST_OVERRIDE` - The hostname used for container-to-container communication. Default Docker is `localhost`, but rootless Podman uses `host.containers.internal`

# Troubleshooting CI Test Failures

CI test logs are pretty big around 75MB. To make it easier to troubleshoot test failures, you can download the raw logs from the CI job and then look for
`Publishing build scan...` in the logs. Copy the URL from the logs and open it in a browser. This will give you a nice view of the test execution breakdown.
CI test logs are pretty large, sometimes exceeding 75MB. To make it easier to troubleshoot test failures, you can download or
view the raw logs from the CI job and then look for `Publishing build scan...` in the logs. There may be several occurrences
of this phrase, so look for the one that follows "BUILD FAILED".

Example:

```
2025-01-01T05:06:24.8710392Z BUILD FAILED in 15m 4s
2025-01-01T05:06:24.8710682Z
2025-01-01T05:06:24.8713216Z 1121 actionable tasks: 1103 executed, 18 up-to-date
2025-01-01T05:06:25.2671952Z
2025-01-01T05:06:25.2672609Z Publishing build scan...
2025-01-01T05:06:25.4674291Z https://gradle.com/s/u4vbxhlidd5ka
```

Copy the gradle.com URL and open it in a browser. This will give you a nice view of the test execution breakdown.
jaydeluca marked this conversation as resolved.
Show resolved Hide resolved

## How to download the raw logs

1. Click on the `Details` link in one of the failing CI jobs under `Some checks were not successful` section of your PR.
2. Click on one of the failed tests in the left panel.
3. Click on the `Setting` gear in the top right corner of the logs panel.
4. Right click on 'View raw logs' and then 'Save link as' to save the page as a text file locally.
5. Once the file is downloaded, open it in a text editor and search for `Publishing build scan...` to find the URL.
5. Once the file is downloaded, open it in a text editor and search for `Publishing build scan...` that follows an occurrence of "BUILD FAILED" to find the URL.
6. Open the URL in a browser to view the test execution breakdown. It might prompt you to "Activate your Build Scan" the very 1st time, you can use your own email address and activate it via email.

Unfortunately, the Build Scan service hosted via Develocity has an allowed size limits of the free build scans. Once you exceed the limit, then you won't be able to view the scan anymore.
Then you can just use the raw logs to search for "FAILED" or "Task failed with an exception" to identify the failing tests.

# Using breakpoints in instrumentation

For instrumentation that has been migrated to use the [invokedynamic based instrumentation mechanism](https://github.com/open-telemetry/opentelemetry-java-instrumentation/issues/8999),
you can leverage breakpoints and standard debugging strategies by adding `-PtestIndy=true` to the gradle command:

```
./gradlew -PtestIndy=true :instrumentation:<INSTRUMENTATION_NAME>:test
```
jaydeluca marked this conversation as resolved.
Show resolved Hide resolved
Loading