[RFC] Decoupling OpenSearch Distribution, Server and Plugin versioning

[pgtgrly]

Executive Summary

This RFC proposes decoupling the versioning of OpenSearch Server, plugins, and distributions to address challenges in the current tightly coupled system. The proposal suggests implementing independent versioning for plugins starting from OpenSearch 3.0.0, using SemVer for plugin versions and SemVer range support for compatibility with OpenSearch Server versions. Key changes include plugins maintaining their own version numbers, defining compatibility for entire major versions of OpenSearch Server, and adopting a four-part versioning system for distributions. This approach aims to increase flexibility in plugin development and releases, reduce unnecessary version bumps, simplify backporting of features and bug fixes, and provide clearer tracking of changes specific to individual components. The proposed implementation involves new branching and release strategies for plugins, updated CICD processes, and revised distribution versioning and release notes structure, ultimately streamlining the development and release process for the entire OpenSearch ecosystem.

Glossary

• OpenSearch server Version: The release version of OpenSearch server without any plugins.
• Plugin Version: The release version of plugins, this is independent of OpenSearch server version.
• Default Plugins: Plugins which are bundled in with standard OpenSearch release
• Core repository Plugins: Plugins which are present in the core OpenSearch repository
• Managed Plugins: Plugins developed and released by OpenSearch Project, contains both default and core repository plugins

Motivation

This document aims to highlight the need for decoupling versioning across OpenSearch Server, Plugins and distributions.

Background

Before diving into the problem statement, let us set up the background on a few topics

OpenSearch Versioning

OpenSearch follows SemVer versioning. According to SemVer convention, given a version number MAJOR.MINOR.PATCH, increment the:

1. MAJOR version when you make incompatible API changes
2. MINOR version when you add functionality in a backward compatible manner
3. PATCH version when you make backward compatible bug fixes

OpenSearch Plugin versions

Properties of an OpenSearch Plugin is defined in plugin-descriptor.properties file. For the sake of this discussion we will focus on two fields within the file; version and opensearch.version.
version defines the release version of a plugin, it can follow any versioning convention that the maintainer decides.
opensearch.version defines the opensearch server version that the plugin is compatible with. When OpenSearch server loads a plugin, it validates if the server version matches the opensearch.version in the properties file.
Previously, for a plugin the opensearch.version and server version had to be an exact match for the plugin to load.

Now, plugins support SemVer range support for opensearch.version. Which means that a plugin can be compiled for an entire major version. for example, A plugin released for OpenSearch server 3.0.0 can be compatible with all the minor and patch releases up-til the next major version release (ie 4.0.0)

OpenSearch Distributions

OpenSearch Server is currently released in two distributions:

• Default distribution: Standard distribution that comes prebundled with a default setof plugins. This distribution is readily available with Linux [available across x86-64 and ARM64 as tarbell & on package managers (deb, rpm, yum)], Docker [x86-64 and ARM64], FreeBSD [x86-64 and ARM64], Arch Linux [x86-64] (btw I use Arch :P) and Windows [x86-64]
• Minimal Distribution: This distribution contains OpenSearch Server without any plugins. It is only readily available with Linux [x86-64 and ARM64 as tarbell].

Plugin Installation

Simply put, OpenSearch plugins can be installed by putting the compiled plugin zip in the plugins directory and restarting the cluster nodes.
However this is automated by the opensearch plugin cli which does some prevalidations. This CLI can take links from maven or any zip.
Additionally it can also install plugins maintained by OpenSearch org by name. (These plugins are hosted in the core repository). The build zip are hosted in “https://artifacts.opensearch.org/releases/plugins/..."

Current State of OpenSearch Server, Plugins and Distribution versioning

If a critical bug fix is required for the ML Commons plugin:

1. The ML Commons plugin version would be incremented to 2.4.1.
2. Despite potentially having no significant changes, the OpenSearch Server version must also be bumped to 2.4.1 to maintain compatibility.
3. All other managed plugins, even those without changes, need to be updated to version 2.4.1 to ensure compatibility with the server.
4. A new distribution incorporating all these version changes would need to be released and named 2.4.1.

Backporting Plugin fixes

Any critical plugin fixes need to be backported to plugin releases corresponding to all OpenSearch branches.

Previous discussion

This issue (opensearch-project/OpenSearch-Dashboards#5877) previously started this discussion in OpenSearch Dashboards repo. This was a precursor to Dashboards maintaining a separate versioning from OpenSearch Server (opensearch-project/OpenSearch-Dashboards#9192). This document is focused on plugins for OpenSearch server only

Assumptions

This RFC assumes that no breaking changes will be introduced in minor version release for OpenSearch Server.

Problem Statement

The current versioning strategy for OpenSearch, which tightly couples the versions of the OpenSearch Distribution, plugins, and server, presents several challenges:

1. Version Interdependence: The strong tie between OpenSearch Distribution version, plugin versions, and OpenSearch server version makes it difficult to track changes specific to individual components. This interdependence often leads to unnecessary version bumps across the entire ecosystem.
2. Forced Server Version Updates: OpenSearch server versions are frequently bumped due to plugin releases, even when there are no significant changes to the server itself. This can lead to confusion about the actual changes in each release.
3. Redundant Plugin Builds: Plugins need to be rebuilt and released for every patch version, even if they contain no changes. This results in unnecessary development and release efforts.
4. Limited Plugin Versioning Flexibility: Plugin versioning is directly tied to the OpenSearch distribution version, limiting the ability to manage plugin-specific version progressions independently.
5. Release Synchronization Bottlenecks: Plugins are forced to wait for each other before a release can be made, potentially delaying critical updates or fixes for individual components.
6. Increased Complexity in Distribution Management: The need to update all components simultaneously complicates the process of creating and managing OpenSearch distributions.
7. Plugin functionality coupling with OpenSearch server version: Given that plugin versioning is coupled with OpenSearch server versions, new functionality is available in newer OpenSearch server versions. To backport it to older minor version releases an new patch version needs to be introduced.

These issues can collectively lead to inefficient development cycles, potential delays in releasing critical updates, and difficulties in maintaining clear version histories for individual components. A more flexible and decoupled versioning approach is needed to address these challenges and improve the overall development and release process for OpenSearch and its ecosystem.

Proposed Solution

We propose maintaining an independent versioning for plugins from 3.0.0.
With the release of OpenSearch 3.0.0, all the plugins will be moved to version 3.0.0 (To avoid confusion with previous plugin versions).

Post that, the plugin version will be kept independent of the server version, instead it will depend on the features of a plugin. The plugin versioning would be implemented using SemVer. [example]. The release notes would be maintained as such.

The opensearch.version for the plugin will use the SemVer range support (^3.0.0) to ensure compatibility across the entire major version for OpenSearch Server.

We can also maintain a compatibility matrix for the plugin version and server version.

For Example:

Plugin Version	Compatible OpenSearch Version
"3.x"	OpenSearch 3.x
"4.x"	OpenSearch 3.x , OpenSearch 4.x
"5.x"	OpenSearch 4.x
"6.x"	OpenSearch 5.x

Q. How will Plugin branching work?
A. The main branch would always track the active development and whenever a release is made a new branch would be created for the release as an archive.

Q. How will Plugin zip be released for different OpenSearch Versions?
A. Since a plugin zip can be compatible with a major version, CICD will compile and release one zip per major version.
The naming for the released artifact can be <plugin-name>-<plugin-version>-<opensearch-server-version>.zip.
example: analysis-sindarin-3.4.2-OS_3.x.zip and analysis-sindarin-3.4.2-OS_4.x.zip

Q. How will we ensure that a plugin is functioning across all OpenSearch Server minor versions in a OpenSearch Server major version?

The CICD process will run plugin integration tests across all the semver ranges mentioned in the build gradle. This can either be in form of a list maintained and populated by the maintainers or be extracted by some central source via network call (Maybe get versions from Maven?)

Q. How will features be back ported across OpenSearch Versions?
No need to backport the features across OpenSearch Versions as the plugin versioning is independent of OpenSearch version.

**Q. What if a Plugin is not compatible across all OpenSearch Server minor versions in a OpenSearch Server major version?
**
A. As per definition of SemVer, the changes across minor should be backward compatible. However if we forsee that not being the case, we can modify the SemVer range support in plugins to have lower and upper bounds.

Q. How will OpenSearch standard distribution be impacted by this?

The distribution versioning will need to change to accommodate independent plugin versions. We propose using a four-part version number for distributions: Major.Minor.Patch.Release, where:

• The first three parts (Major.Minor.Patch) match the OpenSearch server version
• The fourth part (Release) tracks distribution-specific releases that may only include plugin updates

Here's an example sequence of releases:

1. Initial Distribution Release (3.2.3.0)
   ├── OpenSearch Server 3.2.3
   ├── Plugin A v3.1.0
   └── Plugin B v4.5.1

2. Plugin Update Release (3.2.3.1)
   ├── OpenSearch Server 3.2.3 (unchanged)
   ├── Plugin A v3.1.1 (updated)
   └── Plugin B v4.5.1 (unchanged)

3. New Server Version Release (3.3.0.0)
   ├── OpenSearch Server 3.3.0 (new version)
   ├── Plugin A v3.1.1 (unchanged)
   └── Plugin B v4.5.1 (unchanged)

Each distribution release will maintain its own release notes, documenting which components were updated and linking to their respective detailed release notes.

FAQs

Will populate it as per discussion on the thread

Supporting References

• [RFC] Plugins Version Decoupling OpenSearch-Dashboards#5877

[reta]

@pgtgrly thanks for this proposal

For the plugins that are outside the OpenSearch core repository, this is a solved problem with semver, as you mentioned. From the other side, I think it is going to be exceptionally difficult to manage (and release) the components within the same monolithic repository independently.

Also, it RFC does not capture the complexity of the OpenSearch managed plugins, which are further separated into regular plugins and modules (mandatory preinstalled plugins that cannot be removed).

Q. How managed Plugin zip be installed for same OpenSearch Versions?

At the moment, with CLI tooling the managed plugins are installed by name (from known location) with the same version that matches the distribution version. How it is going to work if managed plugins are released independently?

Q. How one should know a new Plugin version is available?

As of today, the users do not expect new versions for managed plugins (and I suspect for any plugins at all) that could be dropped independently of OpenSearch release cycle. How such notifications could be delivered?

[rursprung]

thanks for this proposal! i think it'd be great if this could be achieved with 3.0.0.

i think the RFC should also explicitly state which combinations are officially supported: i.e. if a user has a problem with a server+plugin version combination which isn't in an official distribution does that user still get support or is the first order of business "just update to a version combination which we released as a distribution and report back if it's still not working"? admittedly this could already happen now (since technically plugins can use semver ranges for their server dependency) but i have never seen a plugin use the semver ranges so it's not really happening in 2.x.

Q. What if a Plugin is not compatible across all OpenSearch Server minor versions in a OpenSearch Server major version?
A. As per definition of SemVer, the changes across minor should be backward compatible. However if we forsee that not being the case, we can modify the SemVer range support in plugins to have lower and upper bounds.

IMHO this needs to be supported. say if OpenSearch 3.1.0 introduces a new API which a plugin wants to use then that plugin must have the possibility to state that it's compatible with OpenSearch >= 3.1.0. this is also how package managers (cargo, npm, etc.) work. after all, that's a big part of the fun of SemVer: you get to specify quite exactly what you're depending on in a way a package manager can calculate a valid tree for you.

Q. How one should know a new Plugin version is available?

As of today, the users do not expect new versions for managed plugins (and I suspect for any plugins at all) that could be dropped independently of OpenSearch release cycle. How such notifications could be delivered?

it might make sense to extend the opensearch-plugin CLI and add an update command - if you run it without any arguments it gives a list of available updates, if you run it with -a it'll update all and if you run it with a list of plugin names it'll update those (and of course only those from that list which are (a) installed and (b) have available updates). for core plugins this might work OOTB since the CLI already now knows where to look for them. for other GitHub hosted plugins it might be enough to extend the plugin.properties to list the GH repo URL and the CLI can check the releases there? for plugins hosted elsewhere it'd either need some separate release tracking file or the CLI could just always list them as "you have to check them manually, the author provided this URL for you to check: <...>"

this might be a bit orthogonal to this, but did you consider releasing the server and (core) plugins on package managers (apt, yum, winget, homebrew, etc.) rather than just the distributions? in that case you could apt install opensearch-core opensearch-plugin-security opensearch-plugin-analysis-phonenumber and you'd get the server with the listed plugins and you let apt deal with updates. then the distribution on the package manager could also just be a meta-package which references the core + relevant plugins (and, thanks to semver, the package manager always picks the correct versions). though you'd probably still need the archive files for downloads for platforms where this isn't available.

also, if i recall correctly, there was an RFC somewhere about versioning different APIs in the core used by plugins separately so that plugins could specify their dependency on those APIs rather than specifying a dependency on the whole server. IIRC @dblock was involved with that RFC? though i can't seem to find it anymore... i found #1632 and #1422 which are probably related, though.

[shikharj05]

Thanks for the proposal @pgtgrly !

quick one- Can you also review opensearch-project/.github#251 and align with branching strategy proposed in the other proposal? cc: @andrross

@pgtgrly what are you thoughts on management of plugins dependent on each other? For e.g. If a plugin relies on functionality provided by another plugin, how will the compatibility and versioning of those dependencies be handled?

The CICD process will run plugin integration tests across all the semver ranges mentioned in the build gradle. This can either be in form of a list maintained and populated by the maintainers or be extracted by some central source via network call (Maybe get versions from Maven?)

What would this look like, have you tried a POC ? Maybe the RFC should cover changes required in build/CICD processes for plugin owners.

This document is focused on plugins for OpenSearch server only

I am not 100% sure we can leave out dashboards out of this RFC discussion - plugins that offer a frontend plugin introducing a new feature mostly require tight coupling with the same minimum version of backend plugin. If we don't want to include OSD & plugins completely, should this RFC at least cover potential impact or what the integration between frontend + backend plugin looks like post this change?

While this covers build aspects, to answer more questions around availability/notification of newer version, installation, etc - maybe you can consider and write what user experience looks like today v/s what it would look like post this change. That should help gain insights into tooling, documentation, etc updates required to support this change. Similarly, plugin owner experience can be compared.

Can you also add what would release and default distribution look like? For e.g. today if the bundle is on vX, it moves to vY together. Post decoupling, if the plugin runs an independent release, will the bundle be updated as vX' or the bundle be not updated at all?

[seraphjiang]

Love the initiative and it is aligned with how we vision the future decoupling for OpenSearch Dashboards and its plugin in build, Distribution

opensearch-project/OpenSearch-Dashboards#9192

[cwperks]

In this model, it leaves it up to the plugin's discretion when a release needs to take place. Will there be continuous integration amongst the latest released plugin versions to ensure they integrate together for plugins in the default distribution?

How will release testing work when a plugin is requesting a release? With the current release model, all plugins are integrated together at release time in the Release Candidate and tests across all plugins are performed.

For the compatibility matrix, do you propose that it matches major version only or can it go down to minor version? i.e. The plugin version 2.0.0 is compatible with OpenSearch 2.0-2.5, then plugin version 2.1 is compatible with 2.6-2.10, etc.

With the extensions project one of the goals was version decoupling a separating out the release process between opensearch and an extension

[smacrakis]

What if Plugin A, version 3.1 requires Plugin B, version 2.6 or better?
Is there some way to express dependencies like this?

[krishna-ggk]

Nice initiative and thanks for the detailed proposal @pgtgrly ! Also nice to see similar thinking (OSD#9192) in OSD .

Lots of good questions from everyone.

Overall I see good coverage on OpenSearch and plugins perspective in the RFC and this is a large change. This idea of decoupling release in the world of 50+ plugins helps in transforming to good scaleable model. Like @cwperks pointed out, it was one of extensions' core idea.

However taking a step back, I think we need to draw the consequence and experience of users in detail first. In the RFC, we address plugin/build/distribution problems in general, but the impact is also largely to users/consumers. Directionally it is similar to external non-OpenSearch plugins, however the convenience of simpler installation of OpenSearch with plugins is lost.

1. With 50+ plugins being part of OpenSearch monolith today, the matrix of feature set across plugins can become confusing. This can get worse if the plugins depend on different version of plugins. Any thoughts on how this can be simplified?
2. Overall it sounds like we need to go all the way down to separation of plugins into separate artifacts as @rursprung commented which would then offer. Can we expand on comparisons of different approaches (four part versioning vs separate plugin artifact distribution vs any other ideas)?

[rursprung]

The plugin version 2.0.0 is compatible with OpenSearch 2.0-2.5, then plugin version 2.1 is compatible with 2.6-2.10, etc.

i don't think that this is a valid constellation: if plugin version 2.0.0 is compatible with core 2.0 then semver dictates that it is compatible with any core >=2.0,<3.0. so you could use plugin 2.0.0 with core 2.99 just fine. but it's correct that plugin 2.1 could only be compatible with core >=2.6 (again leading to >=2.6,<3.0). otherwise core would've broken semver if the plugin 2.0 suddenly became incompatible with core 2.6.

[pgtgrly]

Hey folks! thanks for prompt feedback on this RFC, I am really grateful for your comments and quesions.

I think we need to add two more terms to the glossary

• Core Repository Plugins - Managed Plugins in the OpenSearch core repository (eg analysis-nori)
• Default Plugins - Managed Plugins that are bundled with standard OpenSearch distribution (eg security plugin).

Currently none of the plugins in core repository are default plugins.

@reta

RFC does not capture the complexity of the OpenSearch managed plugins, which are further separated into regular plugins and modules

By Modules, do we mean sub projects such as opensearch-common, classloader etc? This RFC is focused on regular plugins which are added to OpenSearch Server via the plugin interface.

Q. How managed Plugin zip be installed for same OpenSearch Versions?

This might require a bit of design and implementation change on cli and release process end.
For example if I want to install a managed plugin by name it downloads the plugin file from
https://artifacts.opensearch.org/releases/plugins/<PLUGIN-NAME>/<OPENSEARCH-SERVER-VERSION>/<PLUGIN-NAME>-<OPENSEARCH-SERVER-VERSION>.zip example: https://artifacts.opensearch.org/releases/plugins/discovery-ec2/1.3.0/discovery-ec2-1.3.0.zip.. Here the Opensearch version an plugin version is the same.

One solution can be The artifact download URL can be restructured to https://artifacts.opensearch.org/releases/plugins/<OPENSEARCH-SERVER-VERSION>/<PLUGIN-NAME>/latest this can link to the latest plugin version that is compatible with the Opensearch server.

Q. How one should know a new Plugin version is available? As of today, the users do not expect new versions for managed plugins (and I suspect for any plugins at all) that could be dropped independently of OpenSearch release cycle. How such notifications could be delivered?

For the default plugins this can be done by packaging them in a new release, for the other managed plugins this can be done by adding an update command as suggested by @rursprung .

@rursprung

i think the RFC should also explicitly state which combinations are officially supported.

I think we should support Plugin versions that are included in distribution release. Prehaps individual plugins can add on LTS

say if OpenSearch 3.1.0 introduces a new API which a plugin wants to use then that plugin must have the possibility to state that it's compatible with OpenSearch >= 3.1.0.

We can have two versions 1.4.0 >3.0.0 and 1.5.0 >3.1.0. Since it is SemVer, both will support uptill 4.0.0

but did you consider releasing the server and (core) plugins on package managers

This is a good proposal and can leverage the dependency management they provide out of the box. That can be picked as a follow up to this

@shikharj05

what are you thoughts on management of plugins dependent on each other?

Excellent question, the way the SemVer dependency management notation in plugins have been implemented, can be expanded to define multiple dependencies. Currently it only supports opensearch. We can work on implementing such multi dependency management in plugin loader. @smacrakis I think this functionailty is what allows us to determine dependencies between multiple plugins.

Can you also review opensearch-project/.github#251 and align with branching strategy proposed in the other proposal?

Will do so and update here.

What would this look like, have you tried a POC ? Maybe the RFC should cover changes required in build/CICD processes for plugin owners.

This should be feasible, but we will need to look into the scalablity. @peterzhuamazon any thoughts on this?
The purpose of this RFC is to determine if decoupling is an approach that should be done moving forward. Once we get consensus for that and determine the scope of changes, we can move to the design phase.

I am not 100% sure we can leave out dashboards out of this RFC discussion

As @seraphjiang called out, Dashboards are planned to be decoupled from Opensearch.

maybe you can consider and write what user experience looks like today v/s what it would look like post this change.

This will be a great section to add. I will add this to RFC (by EOD 30 January IST). Will consider customers who are: consuming distributed versions and customers who will install minimal distribution and install plugins separately.

Can you also add what would release and default distribution look like? For e.g. today if the bundle is on vX, it moves to vY together. Post decoupling, if the plugin runs an independent release, will the bundle be updated as vX' or the bundle be not updated at all?

The releases of default distribution will be the same. Have covered it in the question Q. How will OpenSearch standard distribution be impacted by this?

@cwperks

In this model, it leaves it up to the plugin's discretion when a release needs to take place. Will there be continuous integration amongst the latest released plugin versions to ensure they integrate together for plugins in the default distribution?

Yes we need to have a CI system that will run plugin integration tests for all the OpenSearch versions that the SemVer range supports.

How will release testing work when a plugin is requesting a release? With the current release model, all plugins are integrated together at release time in the Release Candidate and tests across all plugins are performed.

This should not impact the release of the standard distribution. Just the versioning name change. It will be upon release team to decide when a release needs to be made and that release candidate would be tested.

For the compatibility matrix, do you propose that it matches major version only or can it go down to minor version? i.e. The plugin version 2.0.0 is compatible with OpenSearch 2.0-2.5, then plugin version 2.1 is compatible with 2.6-2.10, etc.

As @rursprung pointed out. SemVer ensures backward compatibility across minor version so a posstible matrix can be

Plugin version 1.4 - OpenSearch version ^2.5.0 , ^3.0.0
Plugin version 1.5 - OpenSearch version ^3.1.0, ^4,0,0

With the #1422 one of the goals was version decoupling a separating out the release process between opensearch and an extension

Agreed, However Until plugin functionality is migrated into extension (or core OpenSeach) we can use this approach for decoupling releases. What will standard distribution look like once we migrate them to extensions?

@krishna-ggk

With 50+ plugins being part of OpenSearch monolith today, the matrix of feature set across plugins can become confusing. This can get worse if the plugins depend on different version of plugins. Any thoughts on how this can be simplified?

We currently have 21 default plugins that come bundled as part of the standard distribution . In addition to this we have an additional 22 plugins that are offered by OpenSearch organisation. The feature set consistency of the standard plugin release can be maintained via the standard distribution release. The rest of managed plugins are installed by users as per their requirements, so they can look into the changeset and install the plugin version as per their needs.

Can we expand on comparisons of different approaches (four part versioning vs separate plugin artifact distribution vs any other ideas)?

I think standard distributions should still be offered regardless of separate plugin artifact distribution. We currently publish the managed plugins (except those in core repo) to maven, artifacts.opensearch.org and ci.opensearch.org.
Deploting OpenSearch Standard distribution as a meta package with separate plugin and core packages to be installed via package manager would be a good approach to tackle dependency management. We can explore that route further once we decide the degree of decoupling we need.

As suggested by you, we can start by just decoupling plugin version from OpenSearch Server version and keep the OpenSearch Server and Distribution version coupled. For someone consuming the standard distribution of the plugin, there will be no noticable changes.

Note: I am working on creating another RFC to move the core repository plugins outside the core repo. However that does not need to be coupled with 3.0 release. I will publish that RFC soon (aiming for EOD 30 January IST ) and ping the link here.

Side note: we cannot browse the directory in artifacts.opensearch.org , Maybe we should make it browsable?

[rursprung]

Side note: we cannot browse the directory in artifacts.opensearch.org , Maybe we should make it browsable?

yes, please! currently it's unusable since (a) it's undocumented (needs opensearch-project/documentation-website#8763) and (b) not browsable (thus finding URLs is just guesswork)

[peterzhuamazon]

A few quick points from my side:

1. https://artifacts.opensearch.org/releases/plugins/ only serves core plugins, as we other plugins are already bundled. Uses can still use the manifest.yml file in the bundle to find the zip location and install accordingly.
2. As for branching strategy, see RFC: [PROPOSAL] Simplify OpenSearch distribution branching strategy .github#251.
3. It is possible that we need a compatibility matrix that is well-maintained over time if we ever de-couple the versions, as new plugins might start from 1.0.0 and engine/core is already 3.0.0 or above. Or some plugins might have major breaking changes mid-gen.
4. We already have auto workflows to help plugin repos bump their versions if they have no changes for the release.
5. If a plugin haven't had any changes for a long period of time, would it still be feasible to bundle it in the final distribution release?
6. The current build scripts and pipelines might require a overhaul in order to support this.

These are just initial thoughts and we would need more discussion on this.

cc: @andrross @getsaurabh02

Thanks.

[reta]

Thanks @pgtgrly

By Modules, do we mean sub projects such as opensearch-common, classloader etc? This RFC is focused on regular plugins which are added to OpenSearch Server via the plugin interface.

No, modules are 100% plugins which:

• are installed by default (fe, transport-netty4)
• cannot be removed (nonetheless those are still plugins)

[dblock]

Decoupling the distribution from its plugins is an important and ambitious idea. This was a stated goal of the Extensions Project as @rursprung calls out correctly and it's still an option to revive that. We also intended to do this migration entirely incrementally as proposed in this RFC, but we chose to create a new SDK (new interface in the middle between core and plugins), rather than using the existing plugins interface, to solve a number of other challenges (transitive dependencies, isolation, remote-ability).

The RFC could call out the key tenets:

1. Most users want to be able to download and install a single, feature-rich distribution of OpenSearch that includes many plugins.
2. Plugin developers want to be able to ship new versions of plugins whenever they want independently of the distribution in (1).

The proposed incremental path should work, don't make any changes to the current distribution except that it packages binary plugins, as opposed to rebuilding them from source. This is already supported by declarative semver in plugins/core and the tooling in https://github.com/opensearch-project/opensearch-build.

I would not do this all at once for all plugins, just ship some plugins independently. Despite semver features existing for many versions, there are currently none AFAIK. Start with the simplest, job scheduler. I expect you're going to run into all the problems we tried to solve with extensions like transitive dependencies that will create jar hell. Make sure it works well, and package those plugins in the 3.x distribution instead of building them from source.

Then I'd migrate plugins 1-by-1 over to (1) over the duration of 3.0. This is least disruptive to users and most developer-friendly to plugin authors.

PS: Note that you still have the option of reusing https://github.com/opensearch-project/opensearch-sdk-java, downscope Extensions a bit and make that the plugin v2 interface. A plugin that depends on core instead of a lightweight SDK is still very painful to work with.