Reduce Artifactory storage and bandwidth use

[MarkEWaite]

Service(s)

Artifactory

Summary

Jenkins artifact storage use has increased to greater than 10 TB and bandwidth use has recently increased significantly, almost to the bandwidth use prior to the work done in:

• Remove jcenter, and oss.sonatype.org-releases repositories from public virtual repository; reconfigure Atlassian remote repositories #3842

JFrog sponsors open source without charging the open source project. They have asked us to reduce our storage to less than their 5 TB open source sponsorship threshold. We also want to reduce our bandwidth use to be much closer to the bandwidth we were using in early 2024.

Reproduction steps

1. Review the storage use and reduce storage use to less than 5 TB
2. Review the bandwidth use and reduce to be much closer to the January 2024 usage

[dduportal]

Let's start with a snapshot of metrics provided by JFrog so we can watch the effect of any change.

• Outbound Bandwidth (e.g. amount of data downloaded) can be seen in JFrog's portal:

• Storage (e.g. amount of data stored on disk) can be found in Artifactory administration:

=> We are clearly pointing fingers to the Atlassian Mirror `` for both storage and outbound bandwidth as the top-level item to work on
=> Cleaning up incrementals repository is useful BUT should be secondary

[timja]

I mean the atlassian one should be fairly easy to resolve, just swap it to includes only.

[dduportal]

Update: we've granted access to @darinpope to Artifactory so he can drive this topic:

• Ref. https://groups.google.com/g/jenkinsci-dev/c/6_X0ABdLAgE/m/3XwzSn1xEAAJ for the decision process
• I've created him a secondary (non LDAP) account darinpope-admin with a permission scheme named darinpope-helpdesk-4533 for easier tracking (and reverting in the future)
  • He already has accessed the account and set up a strong password
  • @darinpope do you mind enabling 2FA on this account (since it is not an LDAP)?
  • Despite the account name (darinpope-admin) , he does not have administration permission: the permission scheme allows him to Manage only 5 repositories: all *atlassian* mirrors and the incrementals. That should be sufficient enough: we'll see if he need more permissions.

[timja]

I wonder if we still need the jcenter-cache as well?

[dduportal]

I mean the atlassian one should be fairly easy to resolve, just swap it to includes only.

Let's now track work for each subject on associated issues:

• Work related to atlassian mirror: [repo.jenkins-ci.org] Atlassian Mirror Repository: Artifactory outdated maven-metadata.xml for public/com/github/jnr/jnr-posix/ #4472
• Work related to incrementals: [INFRA-2821] Remove unused incrementals #2386

=> there might be subsqiuent follows up issue.

[dduportal]

I wonder if we still need the jcenter-cache as well?

Yes, we do. It has artifacts which have been deleted from remotes as far as I remember. Worth checking this subject but let's not get carried away: if @darinpope is able to solve the atlassian and incremental, we'll see immediate results to share with JFrog

[darinpope]

@dduportal I've dug around and cannot find a way to implement MFA on my admin account. Is that another switch you need to set on my account?

[dduportal]

@dduportal I've dug around and cannot find a way to implement MFA on my admin account. Is that another switch you need to set on my account?

My mistake: MFA is only available on the JFrog portal, not on Artifactory. Sorry for wasting your time on this one.

Ref. #4472 (comment):

@dduportal with my current permissions with the admin user, I don't have access to look at the configuratio page for repos.

@darinpope I've set your account as admin, you should have access now

[darinpope]

applied com/atlassian/**/* as an include on atlassian-maven-public at 10:15am Central/1615 UTC on 2025-02-11

Now waiting to see if the repo cleans itself up or not. If not by this evening my time, I'll start purging.

[darinpope]

Well, that was fast. It appears that it's already cleaned up.

but not really. It appears it still exists on disk:

so now to figure out how/if the HDD gets cleaned up

[darinpope]

It might be because of "Store Artifacts Locally":

I want to wait on unchecking that box until we meet with JFrog on 2025-02-12.

[MarkEWaite]

Thanks very much! I've started jobs on ci.jenkins.io that I expect will require Atlassian libraries. In case others want to check those jobs, here they are with embeddable build status links:

[timja]

I think what dduportal normally does is move all artifacts out to a separate archived repository that isn't mirrored.

Then once we are confident we can delete?

We can likely just repopulate the atlassian one by running builds like Mark has.

[dduportal]

I think what dduportal normally does is move all artifacts out to a separate archived repository that isn't mirrored.

Then once we are confident we can delete?

Usually, we let the Artifactory's internal Garbage Collector to kick. But we never really checked artifacts were really deleted (unless exceptional cases with less than 10 occurrences) from disk as the storage always has been advertised by JFrog as "not an issue" until nowadays (focus was on bandwidth and builds safety).
Additionally, there are artifacts only present on mirror of upstream which are now gone so we used to be really careful.

The "Store Artifacts Locally" checkbox (to unselect) seems like a good idea for Atlassian public specifically.

We can likely just repopulate the atlassian one by running builds like Mark has.

+1 with @timja : it make sense for this specific repository. @darinpope I believe you can go for this

[darinpope]

We implemented the new artifactory-pkg-maven-cache today (2025-02-13) into public and remove the old atlassian entry around 1545 UTC.

At ~1910 UTC, I changed the include pattern from com/atlassian/**/* to just com/atlassian/** to see if that will make the pull through work as we expect.

[darinpope]

I left the pattern in place, but the root cause why things weren't showing up is we had not updated the User Management->Users->anonymous user and change the Anything and Any Remote permissions to remove atlassian-maven-public and add atlassian-pkg-maven-public. This is similar to the add/remove we did with the public Virtual repository.

Prior to this change, I was getting a 401 when trying to build jira-plugin locally. After the change, everything works as expected.

[darinpope]

added another include of io/atlassian/** at 2318 UTC (5:18p Central) on 2025-02-13 to atlassian-pkg-maven-public so that jira-plugin builds

[darinpope]

After watching atlassian-pkg-maven-public-cache, it's continuing to grow:

Looking back at the previous cache, the com/atlassian path alone is over 4TB !!!!

so...the basic includes of:

• com/atlassian/**
• io/atlassian/**

are obviously not going to cut it.

Right now (2025-02-14 10:50a Central/1650UTC), I'm going to (a) delete the cache and (b) replace the "generic" includes from above and instead apply the following includes:

• com/atlassian/bitbucket/server/**
• com/atlassian/confluence/**
• com/atlassian/connect/**
• com/atlassian/crowd/**
• com/atlassian/fugue/**
• com/atlassian/httpclient/**
• com/atlassian/jira/**
• com/atlassian/maven/plugins/**
• com/atlassian/plugins/**
• com/atlassian/sal/**
• io/atlassian/fugue/**
• io/atlassian/util/concurrent/**

These were pulled from github.com/jenkinsci with the search org:jenkinsci path:pom.xml ".atlassian.". I then grabbed all the active groupId values.

With that said, I know that in jira-plugin there are a few dependency paths that are not explicitly listed above:

• com/atlassian/event/
• com/atlassian/platform/
• com/atlassian/pom/

so there is a chance we may need to add more includes after rerunning the plugin builds that Mark did earlier. Once I get everything in place, I'll do another local jira-plugin build to see what happens and I'll report back.

[timja]

Looking back at the previous cache, the com/atlassian path alone is over 4TB !!!!

I assume thats a build up of over 10 years+. Clearing it down should help sort it.

Your approach above should work but its more maintenance.

i.e. whats the cache size now after being cleared out and some builds ran?

[darinpope]

ok, that was fast. So I also need to include the "missing" 3 includes. That means any plugin using any dependencies from atlassian-pkg-maven-public will need to have more includes added to the config.

Since I didn't state it super-clearly...com/atlassian in the old repo was ~4TB. We don't want that to happen again, so we need to crank down tighter on the includes.

[darinpope]

@timja as noted in the image above, it's over 200GB with lots of "weird" paths (activeobject, adf, alternator, etc) that none of the plugins use.

[timja]

They’ll likely be transitive dependencies but you can see if you can avoid them

[dduportal]

Great work, it was a task that we would have had to run in any cases.

I wonder if their artifacts we need from atlassian cannot be retrieved from another of their Maven repos (e.g. another than maven-public in packages.atlassian.com) a bit like our releases Maven repository in repo.jenkins-ci.org: that would help only getting the required artifacts.

I also guess we can exclude snapshots from the new atlassian-pk-maven-public and only mirror releases, WDYT?

[timja]

There’s at least one package that’s needed that’s not in their releases repository but yeah most of them are in releases. (We hit it when the caching proxy was added)

[darinpope]

I've checked the following plugins locally:

• jira
• jira-ext
• jira-integration
• atlassian-jira-software-cloud
• JiraTestResultReporter

and they all built fine with the list of includes I've added.

Test process:

• clone and cd into the repo
• rm -rf $HOME/.m2/repository/com/atlassian $HOME/.m2/repository/io/atlassian (so that I always get fresh for those paths)
• mvn clean verify

There are a couple more I need to look into, but this was the list that Mark did above (minus jira-trigger; need to setup gradle).

[darinpope]

@MarkEWaite if you want to rerun your same list from above, it'll probably be good to exercise IRL to see what happens.

[darinpope]

I checked "Disable Artifact Resolution in Repository" on atlassian-maven-public at 2025-02-14 11:51a Central/1751 UTC in preparation for deleting that repository. It's already removed from virtual public, just like the repo1 repositories, so it should be fine.

[darinpope]

Here's the current numbers on atlassian-maven-public-cache:

The 6163461 file count hasn't changed in a while (within the last day), so I know that it's not updating anymore.

If no one objects, I'll delete that repo once I get back from lunch. Based on what I'm seeing from the deletion of the repo1 repositories (which started yesterday 2025-02-13), it could take quite a few days (maybe week(s)) for it to completely delete.

[darinpope]

atlassian-maven-public delete submitted at 2025-02-14 1:15p Central/ 19:15UTC.

Now we wait...

[dduportal]

Update: Jfrog confirmed they saw a huge and visible decrease in storage. We now are under the 5Tb threshold.

A quick check on the JFrog portal shows the same metrics on the storage.

We also see a huge decrease in outbound bandwidth which is a good thing (as JFrog also pays for this):

• Storage over the past 30 days:

• Bandwidth on the *artifactory* repository over the past 30 days:

• Bandwidth (total) over the past 30 days:

[yonarbel]

Adding an updated usage status