Instability of artifact-caching-proxy on AWS

[darinpope]

Service(s)

Artifact-caching-proxy

Summary

The following build (which has been kept):

https://ci.jenkins.io/job/Tools/job/bom/view/change-requests/job/PR-4471/16/pipeline-console/

encountered an ACP failure in the pct-parallel-test-executor-plugin-2.479.x stage:

00:33:51  Caused by: org.apache.maven.project.DependencyResolutionException: Could not resolve dependencies for project org.jenkins-ci.plugins:parallel-test-executor:hpi:999999-SNAPSHOT
00:33:51  dependency: org.jenkins-ci.main:remoting:jar:3261.v9c670a_4748a_9 (provided)
00:33:51  	Could not transfer artifact org.jenkins-ci.main:remoting:jar:3261.v9c670a_4748a_9 from/to aws-eks-internal (http://artifact-caching-proxy.artifact-caching-proxy.svc.cluster.local:8080/): status code: 403, reason phrase: Forbidden (403)
00:33:51  
00:33:51      at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve (DefaultProjectDependenciesResolver.java:191)
00:33:51      at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies (LifecycleDependencyResolver.java:224)
00:33:51      at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies (LifecycleDependencyResolver.java:136)
00:33:51      at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved (MojoExecutor.java:355)
00:33:51      at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:313)
...

Reproduction steps

No response

[github-actions]

Take a look at these similar issues to see if there isn't already a response to your problem:

1. 85% Instability of artifact-caching-proxy #4442

[darinpope]

We had another issue in build #19 with another 403:

18:39:54  Caused by: org.eclipse.aether.transfer.ArtifactTransferException: Could not transfer artifact io.jenkins.plugins:jaxb:hpi:2.3.9-1 from/to aws-eks-internal (http://artifact-caching-proxy.artifact-caching-proxy.svc.cluster.local:8080/): status code: 403, reason phrase: Forbidden (403)
18:39:54      at org.eclipse.aether.connector.basic.ArtifactTransportListener.transferFailed (ArtifactTransportListener.java:44)
18:39:54      at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run (BasicRepositoryConnector.java:417)
18:39:54      at org.eclipse.aether.util.concurrency.RunnableErrorForwarder.lambda$wrap$0 (RunnableErrorForwarder.java:66)
18:39:54      at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1136)
18:39:54      at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:635)
18:39:54      at java.lang.Thread.run (Thread.java:840)
18:39:54  Caused by: org.apache.http.client.HttpResponseException: status code: 403, reason phrase: Forbidden (403)
18:39:54      at org.eclipse.aether.transport.http.HttpTransporter.handleStatus (HttpTransporter.java:620)
18:39:54      at org.eclipse.aether.transport.http.HttpTransporter.execute (HttpTransporter.java:489)
18:39:54      at org.eclipse.aether.transport.http.HttpTransporter.implGet (HttpTransporter.java:436)
18:39:54      at org.eclipse.aether.spi.connector.transport.AbstractTransporter.get (AbstractTransporter.java:64)
18:39:54      at org.eclipse.aether.connector.basic.BasicRepositoryConnector$GetTaskRunner.runTask (BasicRepositoryConnector.java:482)
18:39:54      at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run (BasicRepositoryConnector.java:414)
18:39:54      at org.eclipse.aether.util.concurrency.RunnableErrorForwarder.lambda$wrap$0 (RunnableErrorForwarder.java:66)
18:39:54      at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1136)
18:39:54      at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:635)
18:39:54      at java.lang.Thread.run (Thread.java:840)

[dduportal]

This problem should be solved by #4525

[darinpope]

here's another one that just occurred (2025-02-24T18:36:45Z) in a normal PR merge into bom:

12:36:45  [ERROR] Failed to execute goal org.apache.maven.plugins:maven-dependency-plugin:3.8.1:go-offline (default-cli) on project sample: org.eclipse.aether.resolution.DependencyResolutionException: The following artifacts could not be resolved: com.coravy.hudson.plugins.github:github:jar:1.42.0 (absent): Could not transfer artifact com.coravy.hudson.plugins.github:github:jar:1.42.0 from/to aws-eks-internal (http://artifact-caching-proxy.artifact-caching-proxy.svc.cluster.local:8080/): status code: 500, reason phrase: Internal Server Error (500) -> [Help 1]
12:36:45  [ERROR] 
12:36:45  [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
12:36:45  [ERROR] Re-run Maven using the -X switch to enable full debug logging.

This PR will probably be gone by the time you see this.

I've seen a couple of other like this (just a normal PR) outside of the Tuesday or Friday "big" builds.

[smerle33]

I got the log for this one:

here is the order for the logs from https://github.com/jenkins-infra/helm-charts/blob/c5796c50fe96d3d29a143cc5b15030994f3c704c/charts/artifact-caching-proxy/templates/nginx-default-configmap.yaml#L25-L28

$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent $request_time "$http_referer" "$http_user_agent" "$http_x_forwarded_for" "$upstream_addr" "$upstream_status" "$upstream_response_time"

10.0.148.32 - - [24/Feb/2025:18:36:45 +0000] "GET /com/coravy/hudson/plugins/github/github/1.42.0/github-1.42.0.jar HTTP/1.1" 500 314 5.201 "-" "Apache-Maven/3.9.9 (Java 21.0.6; Linux 6.1.127-135.201.amzn2023.x86_64)" "-" "34.225.228.28:443 : 16.15.192.199:443" "302 : 500" "0.312 : 4.889"

so upstream_status = 302 : 500

we also have one later :

10.0.147.157 - - [24/Feb/2025:23:28:19 +0000] "GET /org/eclipse/jgit/org.eclipse.jgit.ssh.apache/7.0.0.202409031743-r/org.eclipse.jgit.ssh.apache-7.0.0.202409031743-r.jar HTTP/1.1" 500 326 5.126 "-" "Apache-Maven/3.9.9 (Java 21.0.6; Linux 6.1.127-135.201.amzn2023.x86_64)" "-" "44.209.124.32:443 : 16.15.178.75:443" "302 : 500" "0.063 : 5.063"

also with upstream_status = 302 : 500

so those errors are coming from the upstream jfrog replying with 500

On 1 day of logs, we got 2 errors (those 2) for 2,38M of requests ...