Make older OpenSearch releases (1.1, 1.2 and etc) available to fetch with latest capability

[amitgalitz]

Is your feature request related to a problem? Please describe

With the addition of #1569. We can now support latest keyword when fetching distribution builds.
This could potentially be a win for bwc testing if we can backfill so we can fetch the correct latest build for older distribution such as Opensearch 1.1.

Describe the solution you'd like

Right now this link downloads latest distribution build for job-scheduler-1.3.1.0:
https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.3.1/latest/linux/x64/builds/opensearch/plugins/opensearch-job-scheduler-1.3.1.0.zip

If I try out the following link for js-1.2.0, no build is found:
https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.2.0/latest/linux/x64/builds/opensearch/plugins/opensearch-job-scheduler-1.2.0.0.zip

Solution:
Using above link for 1.2.0 will download appropriate distribution build

This could be used for example in AD BWC testing where we currently need to keep older JS and AD zips in order to test bwc.

[amitgalitz]

@tianleh @peterzhuamazon @dblock any thoughts on this and using these downloads for bwc testing?

[tianleh]

Option 1: we manually run a build for 1.2.0 and this will add index.json with latest build number.
Option 2: we manually find the existing max build number and manually create a file index.json with the max number.

My understanding is that we cannot run new builds for released versions. Seems Option 2 is more feasible.

cc @bbarani @seraphjiang

[kavilla]

Option 2!

[peterzhuamazon]

Option 2.

[tianleh]

Thanks all for the input! I will pair with infra team to perform the manual update as backfill.

[dblock]

@amitgalitz I don't recommend using latest for bcw because 1) it may not be the actual released version 2) those bits are not signed. There may be builds that are newer than what was released. I would implement something that looks for a .zip on opensearch.org first, then falls back to latest for the current version instead of this proposal.

[amitgalitz]

@amitgalitz I don't recommend using latest for bcw because 1) it may not be the actual released version 2) those bits are not signed. There may be builds that are newer than what was released. I would implement something that looks for a .zip on opensearch.org first, then falls back to latest for the current version instead of this proposal.

In order to get the correct zip from opensearch.org we would need to have the exact build number stored for those version which shouldn't be too difficult I assume? We don't have another location where we can fetch AD or JS zip for example for 1.1. For current version AD will transition to using latest (opensearch-project/anomaly-detection#475) for zips we currently constantly need to update but wanted to remove the zips we have for bwc and implement an easily changeable matrix for bwc versions.

[dblock]

@amitgalitz There's only 1 release on artifacts.opensearch.org per version. The issue rn is that we don't publish ZIPs for plugins, which is #1823. I recommend implementing that, but you'd still have the backfill problem.

[amitgalitz]

@dblock I agree that we should be publishing plugin zips to artifcats.opensearch.org and maven and I can help with that issue also or at least communicate with @prudhvigodithi who seems to be assigned to it. I still think however there should be more discussion on backfilling older releases. I know there is a possibility that there may be newer builds but I was wondering what is the likelihood of this or if it can be checked.

[bbarani]

@amitgalitz We stared publishing the location of the zips in the manifest of the distribution starting 1.1.0 release and you should be able to use it to parse and download the zips used for specific versions.

Example: https://ci.opensearch.org/ci/dbc/builds/1.1.0/405/plugins/opensearch-sql-1.1.0.0.zip is present inside manifest of https://artifacts.opensearch.org/releases/bundle/opensearch/1.1.0/opensearch-1.1.0-linux-x64.tar.gz

Using /latest (at this point in time) is not going to provide you the exact build version of the zip used for a specific distribution as that might point to the latest build but not necessarily the one used for RC candidate / production distribution. We are working on solving that issue in #1823.

Once the above issue is completed, we will look in to Option 2.

[gaiksaya]

With 2.x line moving forward and 1.x running in maintenance mode, do we still need this issue @amitgalitz @dblock ?

[peterzhuamazon]

With 2.x line moving forward and 1.x running in maintenance mode, do we still need this issue @amitgalitz @dblock ?

Also to note that older version before 1.3 have different folder structures, so it is not easy to capable of doing this, especially those are not being tracked at the moment and this issue was opened in 2022.

[dblock]

IMO it's still a valid request, but I personally don't have a scenario where I need latest for a 1.x build, so wouldn't have a heartburn over the ask being closed.

[amitgalitz]

I am also okay with closing this request, even though I think option 2 can still be implemented. I am okay with just utilizing the zip location from the manifest.