Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CASSANDRA-17750: Add docs for dependency management changes #170

Open
wants to merge 13 commits into
base: trunk
Choose a base branch
from
Open

CASSANDRA-17750: Add docs for dependency management changes #170

Changes from 7 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
97d1340
Add docs for dependency management changes in CASSANDRA-17750
aratno Sep 1, 2022
dd7af1c
4.1 -> 4.2
aratno Sep 1, 2022
1b81c58
Expand documentation
aratno Sep 1, 2022
23f2f82
Rm outdated lib steps
aratno Sep 1, 2022
0411c2c
Separate pre-4.2 and post-4.2 documentation per feedback
aratno Sep 1, 2022
ed4d65a
Update intro
aratno Sep 6, 2022
62c7546
Updates based on Claude Warren's feedback
aratno Oct 31, 2023
1591f8e
Fix typo
aratno Nov 1, 2023
c1211af
Apache -> ASF
aratno Nov 2, 2023
1889d09
4.2 -> 5.0
aratno Nov 2, 2023
ed2c1a5
Clarify release process
aratno Nov 3, 2023
8a6bd37
Clarify lib/ and resolver-dist-lib, per PR feedback
aratno Nov 6, 2023
fd4d6f7
Fix link
aratno Nov 6, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
97 changes: 65 additions & 32 deletions site-content/source/modules/ROOT/pages/development/dependencies.adoc
View file
Open in desktop
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The POM FIle Types section does not make sense where it is now. As it is at the same level as the 4.2 and pre-4.2 sections it would appear to apply to both, however, it mentions files that are only in the pre-4.2 code. I think that a section for the POMs found in each section would be cleaner and easier to understand.

Original file line number Diff line number Diff line change
Expand Up @@ -2,50 +2,83 @@

== Dependency Management

Managing libraries for Cassandra is a bit less straight forward compared
to other projects, as the build process is based on ant, maven and
manually managed jars. Make sure to follow the steps below carefully and
pay attention to any emerging issues in the `ci` and reported related
issues on Jira/ML, in case of any project dependency changes.
New dependencies should not be included without community consensus first being
obtained via a `[DISCUSS]` thread on the [email protected] mailing list.

As Cassandra is an Apache product, all included libraries must follow
aratno marked this conversation as resolved.
Show resolved Hide resolved
Apache's https://www.apache.org/legal/resolved.html[software license
requirements].

=== Required steps to add or update libraries

* Add or replace jar file in `lib` directory
* Add or update `lib/license` files
* Update dependencies in `build.xml`
** Add to `parent-pom` with correct version
** Add to `all-pom` if simple Cassandra dependency (see below)

=== POM file types

* *parent-pom* - contains all dependencies with the respective version.
All other poms will refer to the artifacts with specified versions
listed here.
* *build-deps-pom(-sources)* + *coverage-deps-pom* - used by `ant build`
compile target. Listed dependenices will be resolved and copied to
`build/lib/{jar,sources}` by executing the
`maven-ant-tasks-retrieve-build` target. This should contain libraries
that are required for build tools (grammar, docs, instrumentation), but
are not shipped as part of the Cassandra distribution.
* *test-deps-pom* - refered by `maven-ant-tasks-retrieve-test` to
retrieve and save dependencies to `build/test/lib`. Exclusively used
during JUnit test execution.
* *all-pom* - pom for
https://mvnrepository.com/artifact/org.apache.cassandra/cassandra-all[cassandra-all.jar]
that can be installed or deployed to public maven repos via
`ant publish`
Cassandra uses the Ant build system and Maven POMs for dependency specification.
In Cassandra 4.2 the format of POMs was moved from within the `build.xml` file to separate POM template files that are processed by Ant.
aratno marked this conversation as resolved.
Show resolved Hide resolved
In both pre-4.2 and post-4.2 Cassandra, there are several POMs that dependencies can be included in:

* *parent-pom*
- Contains all dependencies with the respective version. All other poms
will refer to the artifacts with specified versions listed here.
- Since Cassandra 4.2, the `parent-pom` template is `.build/parent-pom-template.xml`.
* *build-deps-pom(-sources)* + *coverage-deps-pom*
- used by the `ant build` target. Listed dependencies will be resolved and
copied to `build/lib/{jar,sources}` by executing the
`maven-ant-tasks-retrieve-build` target. This should contain libraries that are
required for build tools (grammar, docs, instrumentation), but are not
shipped as part of the Cassandra distribution.
- Since Cassandra 4.0, `coverage-deps-pom` has been removed and the
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add an item about lib/ and resolver-dist-lib ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you share more about what you mean? I'll admit I haven't used resolver-dist-lib directly so I'm not the right person to document its behavior for new contributors.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When you ant artifacts this will depend on the resolver-dist-lib target. This propagates the lib/ folder. This folder contains the jar files that are actually distributed in a release (e.g. the tarball, the deb and rpm packages). As opposed to build/lib/jars which contains more, e.g. build libs, provided scope, optional deps, etc.

We have to keep a closer eye on what gets generated into the lib/ folder, according to ASF distribution policy: https://www.apache.org/legal/resolved.html

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added more details here

`build-deps-pom` template is `.build/cassandra-build-deps-template.xml`.
* *all-pom*
- POM for https://mvnrepository.com/artifact/org.apache.cassandra/cassandra-all[cassandra-all.jar]
that can be installed or deployed to public maven repos via `ant publish`.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no you can't. nothing can be pushed public without a vote and three PMC +1s

(what ant publish does it stage it the ASF nexus, where we can then vote on it. but no one AFAIK needs to do this because we only do it as part of our release process.)

when a vote is finalised the staging repo in ASF nexus is "published" and gets pushed out to other public repos.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was an existing part of the documentation, but I just made some clarifications - let me know if this is suitable for you

- Since Cassandra 4.2, the `all-pom` template is `.build/cassandra-deps-template.xml`.
* *test-deps-pom*
- Referenced by `maven-ant-tasks-retrieve-test` to retrieve and save
dependencies to `build/test/lib`. Exclusively used during JUnit test
execution.
- Since Cassandra 3.0, `test-deps-pom` has been removed.

The `ant write-poms` target produces valid POM files in the `build/` directory.

=== Dependency management before Cassandra 4.2

To update dependencies, update the parent and child POM definitions in
`build.xml`. The parent POM should include the `dependency` tag with `groupId`,
`artifactId`, `version`, and optional `scope` fields. The child POM(s) should
include the `dependency` tag with `groupId` and `artifactId`. See the
https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Dependency_Management[Maven docs]
for a complete reference on how to reference dependencies across parent and
child POMs.

Here is
https://github.com/apache/cassandra/commit/4b3f07fc74089151efeff7a8fdfa9c414a1f0d6a#diff-766797f233c18114f9499750cf1ffbf3829aeea50283850619c01bd173132021[an example]
of a commit that changes dependency versions pre-4.2.

=== Dependency management in Cassandra 4.2 and later

In Cassandra 4.2 and later, dependencies are managed in Maven POM templates in
`.build/\*-template.xml`. These templates are processed into valid Maven POMs
and copied to `build/\*.pom` the `ant write-poms` task.
aratno marked this conversation as resolved.
Show resolved Hide resolved

For new dependencies, add to `parent-pom-template` and
`cassandra-deps-template`, and optionally `cassandra-build-deps-template` if
the dependency is required for build only. See
https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Dependency_Management[the Maven docs]
on how to reference dependencies in the parent POM from the child POMs.

For dependency versions that need to be available in `build.xml` and
`parent-pom-template`, specify the version as a property in `build.xml`, add it
to the `ant write-poms` target, then add the property to `parent-pom-template`
with the value of the template substitution.

Here is
https://github.com/apache/cassandra/commit/b61bd93e574503aff8c29f0efefbe9879d3b32eb[an example]
of a commit that changes dependency versions since 4.2.

=== Troubleshooting and conflict resolution

Here are some useful commands that may help you out resolving conflicts.

* `ant realclean` - gets rid of the build directory, including build
artifacts.
* `mvn dependency:tree -f build/apache-cassandra-*-SNAPSHOT.pom -Dverbose -Dincludes=org.slf4j`
* `mvn dependency:tree -f build/apache-cassandra-\*-SNAPSHOT.pom -Dverbose -Dincludes=org.slf4j`
- shows transitive dependency tree for artifacts, e.g. org.slf4j. In
case the command above fails due to a missing parent pom file, try
running `ant mvn-install`.
Expand Down