Support for Distribution path mapping with AQL #861

YardenEdery · 2023-11-14T10:21:08Z

All tests passed. If this feature is not already covered by the tests, I added new tests.
All static analysis checks passed.
This pull request is on the dev branch.
I used gofmt for formatting the code before submitting the pull request.

Added support of "mappings" property to file spec in order to have the ability to control where to distribute the artifacts that are being added dynamically when using AQL in the release bundle that is being created.

When looking for docker artifacts using AQL, the release bundle also includes, in the distribution server backend, all the layers for a manifest.json and all of the manifest.json files and their layers if its a list.manifest.json

From now on the "mappings" will take affect on those none specified added manifests and layers and will allow the users a native way of creating the file spec for distribution commands.

FYI This couldn't be achieved using "target" for the list.manifest.json example.

github-actions · 2023-11-14T10:21:22Z

CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅

YardenEdery · 2023-11-14T10:30:37Z

I have read the CLA Document and I hereby sign the CLA

Yarden-Edery · 2023-11-14T10:41:33Z

I have read the CLA Document and I hereby sign the CLA

Yarden-Edery · 2023-11-14T10:41:44Z

recheck

github-actions · 2023-11-20T06:41:21Z

🐸 JFrog Frogbot

eyalbe4 · 2023-11-22T15:40:47Z

Thanks for creating this PR @YardenEdery!
Can you please add full description for this PR?

eyalbe4 · 2023-11-22T16:04:15Z

tests/distribution_test.go

+	// Make sure <RtTargetRepo>/b.out does exist in Artifactory
+	searchParams := artifactoryServices.NewSearchParams()
+	searchParams.Pattern = getRtTargetRepo() + "b.out"
+	reader, err := testsSearchService.Search(searchParams)
+	assert.NoError(t, err)
+	readerCloseAndAssert(t, reader)
+	length, err := reader.Length()
+	assert.NoError(t, err)
+	assert.Equal(t, 1, length)


The above code lines seem to be duplicated multiple times in this file. Let's move into a function and reuse the code.

eyalbe4 · 2023-11-22T16:04:47Z

tests/distribution_test.go

+	// Distribute release bundle
+	distributeBundleParams := distribution.NewDistributeReleaseBundleParams(bundleName, bundleVersion)
+	distributeBundleParams.DistributionRules = []*distribution.DistributionCommonParams{{SiteName: "*"}}
+	testsBundleDistributeService.Sync = true
+	// On distribution with path mapping, the target repository cannot be auto-created
+	testsBundleDistributeService.AutoCreateRepo = false
+	testsBundleDistributeService.DistributeParams = distributeBundleParams
+	err = testsBundleDistributeService.Distribute()
+	assert.NoError(t, err)


The above code lines seem to be duplicated multiple times in this file. Let's move into a function and reuse the code.

yahavi · 2023-11-23T06:37:10Z

tests/distribution_test.go

+	createBundleParams.SpecFiles = []*utils.CommonParams{
+		{
+			Aql: utils.Aql{
+				ItemsFind: "{\"$or\":[{\"$and\":[{\"repo\":{\"$match\": \"" + strings.TrimSuffix(getRtTargetRepo(), "/") + "\"},\"name\":{\"$match\":\"b.in\"}}]}]}",


Suggested change

ItemsFind: "{\"$or\":[{\"$and\":[{\"repo\":{\"$match\": \"" + strings.TrimSuffix(getRtTargetRepo(), "/") + "\"},\"name\":{\"$match\":\"b.in\"}}]}]}",

ItemsFind: "{\"$or\":[{\"$and\":[{\"repo\":{\"$match\":\"" + strings.TrimSuffix(getRtTargetRepo(), "/") + "\"},\"name\":{\"$match\":\"b.in\"}}]}]}",

yahavi · 2023-11-23T06:37:22Z

tests/distribution_test.go

+	createBundleParams.SpecFiles = []*utils.CommonParams{
+		{
+			Aql: utils.Aql{
+				ItemsFind: "{\"$or\":[{\"$and\":[{\"repo\":{\"$match\": \"" + strings.TrimSuffix(getRtTargetRepo(), "/") + "\"},\"name\":{\"$match\":\"*.in\"}}]}]}",


Suggested change

ItemsFind: "{\"$or\":[{\"$and\":[{\"repo\":{\"$match\": \"" + strings.TrimSuffix(getRtTargetRepo(), "/") + "\"},\"name\":{\"$match\":\"*.in\"}}]}]}",

ItemsFind: "{\"$or\":[{\"$and\":[{\"repo\":{\"$match\":\"" + strings.TrimSuffix(getRtTargetRepo(), "/") + "\"},\"name\":{\"$match\":\"*.in\"}}]}]}",

yahavi · 2023-11-23T06:40:59Z

tests/distribution_test.go

+
+	createBundleParams.SignImmediately = true
+	summary, err := testsBundleCreateService.CreateReleaseBundle(createBundleParams)
+	if !assert.NoError(t, err) {


For better readability and shorter code, let's not use "ifs" in tests. The "assert" and/or "require" alone should help us to eliminate them.

yahavi

Well done @Yarden-Edery!
Thanks.

YardenEdery had a problem deploying to frogbot November 14, 2023 10:21 — with GitHub Actions Failure

YardenEdery had a problem deploying to frogbot November 14, 2023 11:03 — with GitHub Actions Failure

YardenEdery had a problem deploying to frogbot November 14, 2023 13:26 — with GitHub Actions Failure

YardenEdery had a problem deploying to frogbot November 15, 2023 11:16 — with GitHub Actions Failure

YardenEdery had a problem deploying to frogbot November 15, 2023 11:20 — with GitHub Actions Failure

YardenEdery had a problem deploying to frogbot November 15, 2023 11:25 — with GitHub Actions Failure

YardenEdery had a problem deploying to frogbot November 15, 2023 11:30 — with GitHub Actions Failure

YardenEdery had a problem deploying to frogbot November 15, 2023 11:39 — with GitHub Actions Failure

YardenEdery changed the base branch from master to dev November 15, 2023 15:42

YardenEdery had a problem deploying to frogbot November 15, 2023 16:02 — with GitHub Actions Failure

yahavi added improvement Automatically generated release notes safe to test Approve running integration tests on a pull request labels Nov 15, 2023

github-actions bot removed the safe to test Approve running integration tests on a pull request label Nov 15, 2023

YardenEdery had a problem deploying to frogbot November 19, 2023 16:18 — with GitHub Actions Failure

Added path mappings support for bundle creation and distribution

c7934b9

YardenEdery temporarily deployed to frogbot November 19, 2023 16:25 — with GitHub Actions Inactive

yahavi added the safe to test Approve running integration tests on a pull request label Nov 19, 2023

github-actions bot removed the safe to test Approve running integration tests on a pull request label Nov 19, 2023

eyalbe4 added new feature Automatically generated release notes and removed improvement Automatically generated release notes labels Nov 22, 2023

eyalbe4 changed the title ~~Add distribution path mappings~~ Distribution path mapping support Nov 22, 2023

eyalbe4 self-requested a review November 22, 2023 15:36

eyalbe4 added the safe to test Approve running integration tests on a pull request label Nov 22, 2023

github-actions bot removed the safe to test Approve running integration tests on a pull request label Nov 22, 2023

Merge branch 'dev' into add-distribution-path-mappings

16b115a

eyalbe4 had a problem deploying to frogbot November 22, 2023 15:41 — with GitHub Actions Failure

eyalbe4 requested changes Nov 22, 2023

View reviewed changes

eyalbe4 added the safe to test Approve running integration tests on a pull request label Nov 22, 2023

github-actions bot removed the safe to test Approve running integration tests on a pull request label Nov 22, 2023

eyalbe4 changed the title ~~Distribution path mapping support~~ Support for Distribution path mapping with AQL Nov 22, 2023

yahavi requested changes Nov 23, 2023

View reviewed changes

YardenEdery had a problem deploying to frogbot November 23, 2023 09:20 — with GitHub Actions Failure

YardenEdery had a problem deploying to frogbot November 23, 2023 10:05 — with GitHub Actions Failure

Added path mappings support for bundle creation and distribution

aed86c7

YardenEdery had a problem deploying to frogbot November 26, 2023 11:51 — with GitHub Actions Failure

Added path mappings support for bundle creation and distribution

39b3825

YardenEdery had a problem deploying to frogbot November 26, 2023 11:52 — with GitHub Actions Failure

YardenEdery requested a review from yahavi November 26, 2023 12:12

yahavi added the safe to test Approve running integration tests on a pull request label Nov 26, 2023

github-actions bot removed the safe to test Approve running integration tests on a pull request label Nov 26, 2023

yahavi approved these changes Nov 26, 2023

View reviewed changes

yahavi merged commit 14887b8 into jfrog:dev Nov 26, 2023
18 of 19 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for Distribution path mapping with AQL #861

Support for Distribution path mapping with AQL #861

YardenEdery commented Nov 14, 2023 •

edited

Loading

github-actions bot commented Nov 14, 2023 •

edited

Loading

YardenEdery commented Nov 14, 2023

Yarden-Edery commented Nov 14, 2023

Yarden-Edery commented Nov 14, 2023

github-actions bot commented Nov 20, 2023

eyalbe4 commented Nov 22, 2023

eyalbe4 Nov 22, 2023

eyalbe4 Nov 22, 2023

yahavi Nov 23, 2023

yahavi Nov 23, 2023

yahavi Nov 23, 2023

yahavi left a comment

	ItemsFind: "{\"$or\":[{\"$and\":[{\"repo\":{\"$match\": \"" + strings.TrimSuffix(getRtTargetRepo(), "/") + "\"},\"name\":{\"$match\":\"b.in\"}}]}]}",
	ItemsFind: "{\"$or\":[{\"$and\":[{\"repo\":{\"$match\":\"" + strings.TrimSuffix(getRtTargetRepo(), "/") + "\"},\"name\":{\"$match\":\"b.in\"}}]}]}",

Support for Distribution path mapping with AQL #861

Support for Distribution path mapping with AQL #861

Conversation

YardenEdery commented Nov 14, 2023 • edited Loading

github-actions bot commented Nov 14, 2023 • edited Loading

YardenEdery commented Nov 14, 2023

Yarden-Edery commented Nov 14, 2023

Yarden-Edery commented Nov 14, 2023

github-actions bot commented Nov 20, 2023

eyalbe4 commented Nov 22, 2023

eyalbe4 Nov 22, 2023

Choose a reason for hiding this comment

eyalbe4 Nov 22, 2023

Choose a reason for hiding this comment

yahavi Nov 23, 2023

Choose a reason for hiding this comment

yahavi Nov 23, 2023

Choose a reason for hiding this comment

yahavi Nov 23, 2023

Choose a reason for hiding this comment

yahavi left a comment

Choose a reason for hiding this comment

YardenEdery commented Nov 14, 2023 •

edited

Loading

github-actions bot commented Nov 14, 2023 •

edited

Loading