TDP 1.1 BOM

[PACordonnier]

BOM as in Bill of Materials (e.g: https://issues.apache.org/jira/browse/BIGTOP-3909)

What's included in this proposed TDP 1.1 update

• Zookeeper is now included
• Components Upgrade to their latest patch version if released
• Components rebased to a patch version, more recent, when they weren't based on a tag. Upgrade the first TDP digit in this case.
• When applicable, reference the tag where TDP forked the apache repo instead of a branch (which is subject to change). When the fork was done on a inactive branch, keep the branch. (Question: should a commit reference needs to be used ?)

Result:

• Zookeeper: now included. Uses 3.4.14 tag as a fork (latest 3.4.X). Repo is on alliage-io and needs to be migrated to TOSIT-IO.
• Hadoop: 3.1.1 -> 3.1.4. Upstream branch is inactive. Tag is latest commit of the branch.
• Hive 3: 3.1.3-1.0 -> 3.1.3-2.0. Fork was based on a unreleased branch. Now uses the released tag. TDP branch should be rebased
• Hive 2: 2.3.9-1.0 -> 2.3.9-2.0. Fork was based on a unreleased branch. Now uses the released tag. TDP branch should be rebased
• Hive 1: No changes required. Released tag is not the HEAD of upstream branch. No activity since 2019. TDP uses head of upstream branch. Keep it as is or ask Apache to release 1.2.3.
• Tez: 0.9.1-1.0 -> 0.9.2-1.0. Upstream branch is inactive. Tag is latest commit of the branch.
• Ranger: Tag is older than head of branch. 2 fix are in HEAD and unreleased (RANGER-3377, RANGER-3378). TDP already backported a lot of stuff. May backports these two fix and bump TDP digit.
• Solr: Nothing to do
• Hbase: 2.1.10 was never release by apache (only in RC). Latest RC is older than Head of upstream. TDP forked latest commit of branch which was not updated since 2021. We may ask for a release or keep it as is.
• Phoenix: TDP forked latest tag so keep it as is. We may update if 5.1.4 is released
• Phoenix Query Server : TDP forked latest tag so keep it as is. We may update if 6.0.1 is released
• Hbase Connectors : Upstream branch is inactive. Tag is latest commit of the branch. Nothing to do.
• Hbase Operator tools: Nothing to do, no branch anyway.

[PACordonnier]

Question:

• Should we upgrade the dependency version of Hadoop and/or zookeeper in the build of dependent project (e.g: Update to hadoop 3.1.4-1.0 and zookeeper 3.4.14-1.0 in Hbase) ?

[rpignolet]

We should update hadoop to 3.1.4-1.0 for dependent project.

[gonzaloetjo]

We should update hadoop to 3.1.4-1.0 for dependent project.

And tez to 0.9.2-1.0

[PACordonnier]

Forgot to talk about spark and Knox in the PR comment.

Spark2: I just linked the latest commit of branch 2.3 (which is spark 2.3.5, but actually is just 2.3.4 + 1 commit for build so its ok)
Spark3: 3.2.2 -> 3.2.4
Knox: Nothing to do, only updates the link to the commit(tag) instead of the branch

[PACordonnier]

Still missing bumping the TDP version for all the subproject as they are probably affected by the bump of the Hadoop and zookeeper version number change

[PACordonnier]

I have too much issues with tez 0.9.2 having breaking changes with 0.9.1. For now I focus on 0.9.1. It may be just the matter of a few backports, but may create a lof of issues down the line

[rpignolet]

Stack 1.1 with Hadoop 3.1 released