-
Notifications
You must be signed in to change notification settings - Fork 3.6k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'master' into Demogorgon314/fix-non-persistent-system-to…
…pic-schema-compatibility
- Loading branch information
Showing
787 changed files
with
33,698 additions
and
16,347 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -84,6 +84,9 @@ github: | |
branch-2.11: {} | ||
branch-3.0: {} | ||
branch-3.1: {} | ||
branch-3.2: {} | ||
branch-3.3: {} | ||
branch-4.0: {} | ||
|
||
notifications: | ||
commits: [email protected] | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -25,9 +25,9 @@ on: | |
- branch-* | ||
- pulsar-* | ||
schedule: | ||
# scheduled job with JDK 17 | ||
- cron: '0 12 * * *' | ||
# scheduled job with JDK 21 | ||
- cron: '0 12 * * *' | ||
# scheduled job with JDK 17 | ||
# if cron expression is changed, make sure to update the expression in jdk_major_version step in preconditions job | ||
- cron: '0 6 * * *' | ||
workflow_dispatch: | ||
|
@@ -44,7 +44,7 @@ on: | |
options: | ||
- '17' | ||
- '21' | ||
default: '17' | ||
default: '21' | ||
trace_test_resource_cleanup: | ||
description: 'Collect thread & heap information before exiting a test JVM. When set to "on", thread dump and heap histogram will be collected. When set to "full", a heap dump will also be collected.' | ||
required: true | ||
|
@@ -65,7 +65,7 @@ concurrency: | |
cancel-in-progress: true | ||
|
||
env: | ||
MAVEN_OPTS: -Xss1500k -Xmx1500m -Daether.connector.http.reuseConnections=false -Daether.connector.requestTimeout=60000 -Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false -Dmaven.wagon.http.retryHandler.class=standard -Dmaven.wagon.http.retryHandler.count=3 -Dmaven.wagon.http.retryHandler.requestSentEnabled=true -Dmaven.wagon.http.serviceUnavailableRetryStrategy.class=standard -Dmaven.wagon.rto=60000 | ||
MAVEN_OPTS: -Xss1500k -Xmx2048m -XX:+UnlockDiagnosticVMOptions -XX:GCLockerRetryAllocationCount=100 -Daether.connector.http.reuseConnections=false -Daether.connector.requestTimeout=60000 -Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false -Dmaven.wagon.http.retryHandler.class=standard -Dmaven.wagon.http.retryHandler.count=3 -Dmaven.wagon.http.retryHandler.requestSentEnabled=true -Dmaven.wagon.http.serviceUnavailableRetryStrategy.class=standard -Dmaven.wagon.rto=60000 | ||
# defines the retention period for the intermediate build artifacts needed for rerunning a failed build job | ||
# it's possible to rerun individual failed jobs when the build artifacts are available | ||
# if the artifacts have already been expired, the complete workflow can be rerun by closing and reopening the PR or by rebasing the PR | ||
|
@@ -95,13 +95,13 @@ jobs: | |
- name: Select JDK major version | ||
id: jdk_major_version | ||
run: | | ||
# use JDK 21 for the scheduled build with cron expression '0 6 * * *' | ||
# use JDK 17 for the scheduled build with cron expression '0 6 * * *' | ||
if [[ "${{ github.event_name == 'schedule' && github.event.schedule == '0 6 * * *' && 'true' || 'false' }}" == "true" ]]; then | ||
echo "jdk_major_version=21" >> $GITHUB_OUTPUT | ||
echo "jdk_major_version=17" >> $GITHUB_OUTPUT | ||
exit 0 | ||
fi | ||
# use JDK 17 for build unless overridden with workflow_dispatch input | ||
echo "jdk_major_version=${{ github.event_name == 'workflow_dispatch' && github.event.inputs.jdk_major_version || '17'}}" >> $GITHUB_OUTPUT | ||
# use JDK 21 for build unless overridden with workflow_dispatch input | ||
echo "jdk_major_version=${{ github.event_name == 'workflow_dispatch' && github.event.inputs.jdk_major_version || '21'}}" >> $GITHUB_OUTPUT | ||
- name: checkout | ||
if: ${{ github.event_name == 'pull_request' }} | ||
|
@@ -147,7 +147,7 @@ jobs: | |
name: Build and License check | ||
env: | ||
JOB_NAME: Build and License check | ||
DEVELOCITY_ACCESS_KEY: ${{ secrets.GE_ACCESS_TOKEN }} | ||
DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }} | ||
CI_JDK_MAJOR_VERSION: ${{ needs.preconditions.outputs.jdk_major_version }} | ||
runs-on: ubuntu-22.04 | ||
timeout-minutes: 60 | ||
|
@@ -224,7 +224,7 @@ jobs: | |
env: | ||
JOB_NAME: CI - Unit - ${{ matrix.name }} | ||
COLLECT_COVERAGE: "${{ needs.preconditions.outputs.collect_coverage }}" | ||
DEVELOCITY_ACCESS_KEY: ${{ secrets.GE_ACCESS_TOKEN }} | ||
DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }} | ||
CI_JDK_MAJOR_VERSION: ${{ needs.preconditions.outputs.jdk_major_version }} | ||
TRACE_TEST_RESOURCE_CLEANUP: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.trace_test_resource_cleanup || 'off' }} | ||
TRACE_TEST_RESOURCE_CLEANUP_DIR: ${{ github.workspace }}/target/trace-test-resource-cleanup | ||
|
@@ -472,7 +472,7 @@ jobs: | |
- linux/amd64 | ||
- linux/arm64 | ||
env: | ||
DEVELOCITY_ACCESS_KEY: ${{ secrets.GE_ACCESS_TOKEN }} | ||
DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }} | ||
CI_JDK_MAJOR_VERSION: ${{ needs.preconditions.outputs.jdk_major_version }} | ||
IMAGE_JDK_MAJOR_VERSION: ${{ needs.preconditions.outputs.jdk_major_version }} | ||
steps: | ||
|
@@ -550,7 +550,7 @@ jobs: | |
env: | ||
JOB_NAME: CI - Integration - ${{ matrix.name }} | ||
PULSAR_TEST_IMAGE_NAME: apachepulsar/java-test-image:latest | ||
DEVELOCITY_ACCESS_KEY: ${{ secrets.GE_ACCESS_TOKEN }} | ||
DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }} | ||
CI_JDK_MAJOR_VERSION: ${{ needs.preconditions.outputs.jdk_major_version }} | ||
strategy: | ||
fail-fast: false | ||
|
@@ -603,6 +603,9 @@ jobs: | |
- name: Metrics | ||
group: METRICS | ||
|
||
- name: Upgrade | ||
group: UPGRADE | ||
|
||
steps: | ||
- name: checkout | ||
uses: actions/checkout@v4 | ||
|
@@ -828,7 +831,7 @@ jobs: | |
needs: ['preconditions', 'build-and-license-check'] | ||
if: ${{ needs.preconditions.outputs.docs_only != 'true' }} | ||
env: | ||
DEVELOCITY_ACCESS_KEY: ${{ secrets.GE_ACCESS_TOKEN }} | ||
DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }} | ||
CI_JDK_MAJOR_VERSION: ${{ needs.preconditions.outputs.jdk_major_version }} | ||
IMAGE_JDK_MAJOR_VERSION: ${{ needs.preconditions.outputs.jdk_major_version }} | ||
steps: | ||
|
@@ -890,8 +893,10 @@ jobs: | |
run: src/check-binary-license.sh ./distribution/server/target/apache-pulsar-*-bin.tar.gz && src/check-binary-license.sh ./distribution/shell/target/apache-pulsar-shell-*-bin.tar.gz | ||
|
||
- name: Run Trivy container scan | ||
uses: aquasecurity/trivy-action@master | ||
id: trivy_scan | ||
uses: aquasecurity/[email protected] | ||
if: ${{ github.repository == 'apache/pulsar' && github.event_name != 'pull_request' }} | ||
continue-on-error: true | ||
with: | ||
image-ref: "apachepulsar/pulsar:latest" | ||
scanners: vuln | ||
|
@@ -902,7 +907,8 @@ jobs: | |
|
||
- name: Upload Trivy scan results to GitHub Security tab | ||
uses: github/codeql-action/upload-sarif@v3 | ||
if: ${{ github.repository == 'apache/pulsar' && github.event_name != 'pull_request' }} | ||
if: ${{ steps.trivy_scan.outcome == 'success' && github.repository == 'apache/pulsar' && github.event_name != 'pull_request' }} | ||
continue-on-error: true | ||
with: | ||
sarif_file: 'trivy-results.sarif' | ||
|
||
|
@@ -951,7 +957,7 @@ jobs: | |
env: | ||
JOB_NAME: CI - System - ${{ matrix.name }} | ||
PULSAR_TEST_IMAGE_NAME: apachepulsar/pulsar-test-latest-version:latest | ||
DEVELOCITY_ACCESS_KEY: ${{ secrets.GE_ACCESS_TOKEN }} | ||
DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }} | ||
CI_JDK_MAJOR_VERSION: ${{ needs.preconditions.outputs.jdk_major_version }} | ||
strategy: | ||
fail-fast: false | ||
|
@@ -1181,7 +1187,7 @@ jobs: | |
env: | ||
JOB_NAME: CI Flaky - System - ${{ matrix.name }} | ||
PULSAR_TEST_IMAGE_NAME: apachepulsar/pulsar-test-latest-version:latest | ||
DEVELOCITY_ACCESS_KEY: ${{ secrets.GE_ACCESS_TOKEN }} | ||
DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }} | ||
CI_JDK_MAJOR_VERSION: ${{ needs.preconditions.outputs.jdk_major_version }} | ||
strategy: | ||
fail-fast: false | ||
|
@@ -1324,7 +1330,7 @@ jobs: | |
needs: ['preconditions', 'integration-tests'] | ||
if: ${{ needs.preconditions.outputs.docs_only != 'true' }} | ||
env: | ||
DEVELOCITY_ACCESS_KEY: ${{ secrets.GE_ACCESS_TOKEN }} | ||
DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }} | ||
CI_JDK_MAJOR_VERSION: ${{ needs.preconditions.outputs.jdk_major_version }} | ||
steps: | ||
- name: checkout | ||
|
@@ -1364,7 +1370,7 @@ jobs: | |
contents: read | ||
security-events: write | ||
env: | ||
DEVELOCITY_ACCESS_KEY: ${{ secrets.GE_ACCESS_TOKEN }} | ||
DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }} | ||
CI_JDK_MAJOR_VERSION: ${{ needs.preconditions.outputs.jdk_major_version }} | ||
CODEQL_LANGUAGE: java-kotlin | ||
steps: | ||
|
@@ -1425,7 +1431,7 @@ jobs: | |
needs: [ 'preconditions', 'integration-tests' ] | ||
if: ${{ needs.preconditions.outputs.need_owasp == 'true' }} | ||
env: | ||
DEVELOCITY_ACCESS_KEY: ${{ secrets.GE_ACCESS_TOKEN }} | ||
DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }} | ||
CI_JDK_MAJOR_VERSION: ${{ needs.preconditions.outputs.jdk_major_version }} | ||
NIST_NVD_API_KEY: ${{ secrets.NIST_NVD_API_KEY }} | ||
steps: | ||
|
@@ -1488,15 +1494,21 @@ jobs: | |
restore-keys: | | ||
owasp-dependency-check-data- | ||
- name: Log warning when skipped | ||
if: ${{ !steps.restore-owasp-dependency-check-data.outputs.cache-matched-key }} | ||
run: | | ||
echo "::warning::OWASP Dependency Check was skipped since the OWASP Dependency check data wasn't found in the cache. Run ci-owasp-dependency-check.yaml workflow to update the cache." | ||
# Projects dependent on flume, hdfs, and hbase currently excluded from the scan. | ||
- name: trigger dependency check | ||
if: ${{ steps.restore-owasp-dependency-check-data.outputs.cache-matched-key }} | ||
run: | | ||
mvn -B -ntp verify -PskipDocker,skip-all,owasp-dependency-check -Dcheckstyle.skip=true -DskipTests \ | ||
-pl '!distribution/server,!distribution/io,!distribution/offloaders,!tiered-storage/file-system,!pulsar-io/flume,!pulsar-io/hbase,!pulsar-io/hdfs2,!pulsar-io/hdfs3,!pulsar-io/docs,!pulsar-io/jdbc/openmldb' | ||
-pl '!distribution/server,!distribution/io,!distribution/offloaders,!tiered-storage/file-system,!pulsar-io/flume,!pulsar-io/hbase,!pulsar-io/hdfs3,!pulsar-io/docs,!pulsar-io/jdbc/openmldb' | ||
- name: Upload report | ||
uses: actions/upload-artifact@v4 | ||
if: ${{ cancelled() || failure() }} | ||
if: ${{ steps.restore-owasp-dependency-check-data.outputs.cache-matched-key && (cancelled() || failure()) }} | ||
continue-on-error: true | ||
with: | ||
name: dependency report | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -99,3 +99,5 @@ test-reports/ | |
.mvn/.gradle-enterprise/ | ||
# Gradle Develocity | ||
.mvn/.develocity/ | ||
.vscode | ||
effective-pom.xml |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.