Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add DeploymentTemplate controller for GitOps support #8001

Merged

Conversation

willdavsmith
Copy link
Contributor

@willdavsmith willdavsmith commented Oct 17, 2024

Description

  • Add rad bicep generate-kubernetes-manifest CLI
  • Add DeploymentTemplate controller
  • Add DeploymentResource controller

Type of change

  • This pull request adds or changes features of Radius and has an approved issue (issue link required).

#6689

Contributor checklist

Please verify that the PR meets the following requirements, where applicable:

  • An overview of proposed schema changes is included in a linked GitHub issue.
  • A design document PR is created in the design-notes repository, if new APIs are being introduced.
  • If applicable, design document has been reviewed and approved by Radius maintainers/approvers.
  • A PR for the samples repository is created, if existing samples are affected by the changes in this PR.
  • A PR for the documentation repository is created, if the changes in this PR affect the documentation or any user facing updates are made.
  • A PR for the recipes repository is created, if existing recipes are affected by the changes in this PR.

@@ -527,7 +527,6 @@ func Test_Run(t *testing.T) {
})

t.Run("Deployment with missing parameters", func(t *testing.T) {
//t.Skip()
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removing commented out t.Skip()

@willdavsmith willdavsmith changed the title [WIP/ Do not review] Flux support Add DeploymentTemplate controller Nov 1, 2024
@willdavsmith willdavsmith marked this pull request as ready for review November 1, 2024 20:14
@willdavsmith willdavsmith requested review from a team as code owners November 1, 2024 20:14
@radius-functional-tests
Copy link

radius-functional-tests bot commented Nov 1, 2024

Radius functional test overview

🔍 Go to test action run

Name Value
Repository willdavsmith/radius
Commit ref a774840
Unique ID func56e23e2717
Image tag pr-func56e23e2717
Click here to see the list of tools in the current test run
  • gotestsum 1.12.0
  • KinD: v0.20.0
  • Dapr: 1.12.0
  • Azure KeyVault CSI driver: 1.4.2
  • Azure Workload identity webhook: 1.3.0
  • Bicep recipe location ghcr.io/radius-project/dev/test/testrecipes/test-bicep-recipes/<name>:pr-func56e23e2717
  • Terraform recipe location http://tf-module-server.radius-test-tf-module-server.svc.cluster.local/<name>.zip (in cluster)
  • applications-rp test image location: ghcr.io/radius-project/dev/applications-rp:pr-func56e23e2717
  • controller test image location: ghcr.io/radius-project/dev/controller:pr-func56e23e2717
  • ucp test image location: ghcr.io/radius-project/dev/ucpd:pr-func56e23e2717
  • deployment-engine test image location: ghcr.io/radius-project/deployment-engine:latest

Test Status

⌛ Building Radius and pushing container images for functional tests...
✅ Container images build succeeded
⌛ Publishing Bicep Recipes for functional tests...
✅ Recipe publishing succeeded
⌛ Starting corerp-cloud functional tests...
⌛ Starting ucp-cloud functional tests...
✅ ucp-cloud functional tests succeeded
✅ corerp-cloud functional tests succeeded

Copy link
Member

@brooke-hamilton brooke-hamilton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran make generate build test and got 8 failures.

=== FAIL: pkg/cli/cmd/bicep/generatekubernetesmanifest Test_Run/Create_basic_DeploymentTemplate (0.00s)
=== FAIL: pkg/cli/cmd/bicep/generatekubernetesmanifest Test_Run/Create_DeploymentTemplate_with_template_content (0.00s)
=== FAIL: pkg/cli/cmd/bicep/generatekubernetesmanifest Test_Run/Create_DeploymentTemplate_with_Azure_scope (0.00s)
=== FAIL: pkg/cli/cmd/bicep/generatekubernetesmanifest Test_Run/Create_DeploymentTemplate_with_AWS_scope (0.00s)
=== FAIL: pkg/cli/cmd/bicep/generatekubernetesmanifest Test_Run (0.00s)
=== FAIL: pkg/controller/reconciler Test_DeploymentResourceReconciler_Basic (10.01s)
=== FAIL: pkg/controller/reconciler Test_DeploymentTemplateReconciler_Basic (0.01s)
=== FAIL: pkg/controller/reconciler Test_DeploymentTemplateReconciler_FailureRecovery (0.00s)

Here's one example:

=== FAIL: pkg/cli/cmd/bicep/generatekubernetesmanifest Test_Run/Create_DeploymentTemplate_with_Azure_scope (0.00s)
Creating DeploymentTemplate YAML file
    generatekubernetesmanifest_test.go:396: 
                Error Trace:    /workspaces/radius/pkg/cli/cmd/bicep/generatekubernetesmanifest/generatekubernetesmanifest_test.go:396
                Error:          Not equal: 
                                expected: "apiVersion: radapp.io/v1alpha3\nkind: DeploymentTemplate\nmetadata:\n  name: azure.yaml\n  namespace: radius-system\nspec:\n  parameters: '{}'\n  providerConfig: '{\"radius\":{\"type\":\"radius\",\"value\":{\"scope\":\"/planes/radius/local/resourceGroups/test-resource-group\"}},\"az\":{\"type\":\"azure\",\"value\":{\"scope\":\"/subscriptions/test-subId/resourceGroups/test-rg\"}},\"deployments\":{\"type\":\"Microsoft.Resources\",\"value\":{\"scope\":\"/planes/radius/local/resourceGroups/test-resource-group\"}}}'\n  template: '{}'\n"
                                actual  : "apiVersion: radapp.io/v1alpha3\nkind: DeploymentTemplate\nmetadata:\n  name: azure.yaml\n  namespace: radius-system\nspec:\n  parameters: '{}'\n  providerConfig: '{\"radius\":{\"type\":\"radius\",\"value\":{\"scope\":\"/planes/radius/local/resourceGroups/test-resource-group\"}},\"az\":{\"type\":\"azure\",\"value\":{\"scope\":\"/subscriptions/test-subId/resourceGroups/test-rg\"}},\"deployments\":{\"type\":\"Microsoft.Resources\",\"value\":{\"scope\":\"/planes/radius/local/resourceGroups/test-resource-group\"}}}'\n  repository: azure.yaml\n  template: '{}'\n"
                            
                                Diff:
                                --- Expected
                                +++ Actual
                                @@ -8,2 +8,3 @@
                                   providerConfig: '{"radius":{"type":"radius","value":{"scope":"/planes/radius/local/resourceGroups/test-resource-group"}},"az":{"type":"azure","value":{"scope":"/subscriptions/test-subId/resourceGroups/test-rg"}},"deployments":{"type":"Microsoft.Resources","value":{"scope":"/planes/radius/local/resourceGroups/test-resource-group"}}}'
                                +  repository: azure.yaml
                                   template: '{}'
                Test:           Test_Run/Create_DeploymentTemplate_with_Azure_scope
    --- FAIL: Test_Run/Create_DeploymentTemplate_with_Azure_scope (0.00s)

Copy link
Member

@brooke-hamilton brooke-hamilton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Running make generate results in these files being changed.

deploy/Chart/crds/radius/radapp.io_deploymentresources.yaml
deploy/Chart/crds/radius/radapp.io_deploymenttemplates.yaml
deploy/Chart/crds/radius/radapp.io_recipes.yaml
deploy/Chart/crds/ucpd/ucp.dev_queuemessages.yaml
deploy/Chart/crds/ucpd/ucp.dev_resources.yaml

For example, controller-gen.kubebuilder.io/version: v0.16.4 changes to controller-gen.kubebuilder.io/version: v0.16.0

Signed-off-by: willdavsmith <[email protected]>
Signed-off-by: willdavsmith <[email protected]>
Signed-off-by: willdavsmith <[email protected]>
Signed-off-by: willdavsmith <[email protected]>
Signed-off-by: willdavsmith <[email protected]>
Signed-off-by: willdavsmith <[email protected]>
Signed-off-by: willdavsmith <[email protected]>
Signed-off-by: willdavsmith <[email protected]>
Signed-off-by: willdavsmith <[email protected]>
Signed-off-by: willdavsmith <[email protected]>
Signed-off-by: willdavsmith <[email protected]>
Signed-off-by: willdavsmith <[email protected]>
Signed-off-by: willdavsmith <[email protected]>
@@ -72,7 +72,7 @@ type BicepOptions struct {

type bicepDriver struct {
ArmClientOptions *arm.ClientOptions
DeploymentClient *clients.ResourceDeploymentsClient
DeploymentClient clients.ResourceDeploymentsClient
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Swapping to use interface instead of struct

@@ -175,3 +188,57 @@ func (client *ResourceDeploymentsClient) createOrUpdateCreateRequest(ctx context
req.Raw().Header["Accept"] = []string{"application/json"}
return req, runtime.MarshalAsJSON(req, parameters)
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had to add some functionality to the ResourceDeploymentsClient for async processing. Today we just call PollUntilDone() in the CLI deployment and bicep driver.

waitForDeploymentTemplateStateDeleted(t, k8sClient, namespacedName)
}

func Test_DeploymentTemplateReconciler_OutputResources(t *testing.T) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

return runtime.NewPollerFromResumeToken[ClientCreateOrUpdateResponse](resumeToken, *client.pipeline, nil)
}

func (client *ResourceDeploymentsClientImpl) Delete(ctx context.Context, resourceID, apiVersion string) (*runtime.Poller[ClientDeleteResponse], error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pls add comment for function.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added

connection sdk.Connection
}

func NewDeploymentClient(connection sdk.Connection) *DeploymentClientImpl {
Copy link
Contributor

@lakshmimsft lakshmimsft Jan 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comments needed for exported type/functions

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

@@ -42,9 +39,6 @@ type DeploymentResourceStatus struct {

// Phrase indicates the current status of the Deployment Resource.
Phrase DeploymentResourcePhrase `json:"phrase,omitempty"`

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was this unused before?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't tell exactly what this is referring to but if it's the ProviderConfig, yes it's unused now that I'm using the DeploymentsClient instead of the RadiusClient.

Signed-off-by: willdavsmith <[email protected]>
@radius-functional-tests
Copy link

radius-functional-tests bot commented Jan 27, 2025

Radius functional test overview

🔍 Go to test action run

Name Value
Repository willdavsmith/radius
Commit ref cfa5f66
Unique ID func84b31ab145
Image tag pr-func84b31ab145
Click here to see the list of tools in the current test run
  • gotestsum 1.12.0
  • KinD: v0.20.0
  • Dapr:
  • Azure KeyVault CSI driver: 1.4.2
  • Azure Workload identity webhook: 1.3.0
  • Bicep recipe location ghcr.io/radius-project/dev/test/testrecipes/test-bicep-recipes/<name>:pr-func84b31ab145
  • Terraform recipe location http://tf-module-server.radius-test-tf-module-server.svc.cluster.local/<name>.zip (in cluster)
  • applications-rp test image location: ghcr.io/radius-project/dev/applications-rp:pr-func84b31ab145
  • dynamic-rp test image location: ghcr.io/radius-project/dev/dynamic-rp:pr-func84b31ab145
  • controller test image location: ghcr.io/radius-project/dev/controller:pr-func84b31ab145
  • ucp test image location: ghcr.io/radius-project/dev/ucpd:pr-func84b31ab145
  • deployment-engine test image location: ghcr.io/radius-project/deployment-engine:latest

Test Status

⌛ Building Radius and pushing container images for functional tests...
✅ Container images build succeeded
⌛ Publishing Bicep Recipes for functional tests...
✅ Recipe publishing succeeded
⌛ Starting ucp-cloud functional tests...
⌛ Starting corerp-cloud functional tests...
✅ ucp-cloud functional tests succeeded
✅ corerp-cloud functional tests succeeded


// ResourceDeploymentsClient is an interface for interacting
// with UCP Deployments.
type ResourceDeploymentsClient interface {
Copy link
Contributor

@sk593 sk593 Jan 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a little confused by the naming choice. We already have a ResourceDeploymentsClient used in the SDK clients package:

type ResourceDeploymentsClient interface {

Based on the code, it seems like these are both deployment clients that serve UCP (although the one linked above serves Azure and UCP) so why are we redefining the interface? If both `ResourceDeploymentsClient are meant to serve different purposes or serve different deployments (Azure/UCP vs deployment resources/templates), I think we should rename this for clarity.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They do the same thing - I think we could consolidate them. I think I would rather handle this together with other changes later to move the mocks outside of reconciler package though.

var _ DeploymentClient = (*DeploymentClientImpl)(nil)

// CreateOrUpdate creates or updates a deployment.
func (rdc *ResourceDeploymentsClientImpl) CreateOrUpdate(ctx context.Context, parameters sdkclients.Deployment, resourceID, apiVersion string) (Poller[sdkclients.ClientCreateOrUpdateResponse], error) {
Copy link
Contributor

@sk593 sk593 Jan 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is rdc *ResourceDeploymentsClientImpl the correct value? Should it not be DeploymentClientImpl?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is right. ResourceDeploymentsClientImpl is the type that has all of the actual functions, DeploymentClientImpl just has an accessor (ResourceDeployments())

var _ ResourceDeploymentsClient = (*ResourceDeploymentsClientImpl)(nil)

// ResourceDeploymentsClientImpl is an implementation of ResourceDeploymentsClient.
type ResourceDeploymentsClientImpl struct {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain why we're choosing to wrap the ResourceDeploymentsClient from sdkclients and also defined a new ResourceDeploymentsClient in this file? I'm not sure I see the 2 distinct use cases here

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They're the same - we can merge them. For now, I wanted to keep the code in the reconciler package to match the RadiusClient. We can consolidate after

Signed-off-by: willdavsmith <[email protected]>
@@ -44,7 +44,7 @@ type RecipeControllerConfig struct {
ConfigLoader configloader.ConfigurationLoader

// DeploymentEngineClient is the client for interacting with the deployment engine.
DeploymentEngineClient *clients.ResourceDeploymentsClient
DeploymentEngineClient clients.ResourceDeploymentsClient
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is struct -> interface, shouldn't affect functionality

@radius-functional-tests
Copy link

radius-functional-tests bot commented Jan 28, 2025

Radius functional test overview

🔍 Go to test action run

Name Value
Repository willdavsmith/radius
Commit ref 4101258
Unique ID funcd908053173
Image tag pr-funcd908053173
Click here to see the list of tools in the current test run
  • gotestsum 1.12.0
  • KinD: v0.20.0
  • Dapr:
  • Azure KeyVault CSI driver: 1.4.2
  • Azure Workload identity webhook: 1.3.0
  • Bicep recipe location ghcr.io/radius-project/dev/test/testrecipes/test-bicep-recipes/<name>:pr-funcd908053173
  • Terraform recipe location http://tf-module-server.radius-test-tf-module-server.svc.cluster.local/<name>.zip (in cluster)
  • applications-rp test image location: ghcr.io/radius-project/dev/applications-rp:pr-funcd908053173
  • dynamic-rp test image location: ghcr.io/radius-project/dev/dynamic-rp:pr-funcd908053173
  • controller test image location: ghcr.io/radius-project/dev/controller:pr-funcd908053173
  • ucp test image location: ghcr.io/radius-project/dev/ucpd:pr-funcd908053173
  • deployment-engine test image location: ghcr.io/radius-project/deployment-engine:latest

Test Status

⌛ Building Radius and pushing container images for functional tests...
✅ Container images build succeeded
⌛ Publishing Bicep Recipes for functional tests...
✅ Recipe publishing succeeded
⌛ Starting ucp-cloud functional tests...
⌛ Starting corerp-cloud functional tests...
✅ ucp-cloud functional tests succeeded
✅ corerp-cloud functional tests succeeded

Signed-off-by: willdavsmith <[email protected]>
@radius-functional-tests
Copy link

radius-functional-tests bot commented Jan 28, 2025

Radius functional test overview

🔍 Go to test action run

Name Value
Repository willdavsmith/radius
Commit ref be4c051
Unique ID func9d668a1917
Image tag pr-func9d668a1917
Click here to see the list of tools in the current test run
  • gotestsum 1.12.0
  • KinD: v0.20.0
  • Dapr:
  • Azure KeyVault CSI driver: 1.4.2
  • Azure Workload identity webhook: 1.3.0
  • Bicep recipe location ghcr.io/radius-project/dev/test/testrecipes/test-bicep-recipes/<name>:pr-func9d668a1917
  • Terraform recipe location http://tf-module-server.radius-test-tf-module-server.svc.cluster.local/<name>.zip (in cluster)
  • applications-rp test image location: ghcr.io/radius-project/dev/applications-rp:pr-func9d668a1917
  • dynamic-rp test image location: ghcr.io/radius-project/dev/dynamic-rp:pr-func9d668a1917
  • controller test image location: ghcr.io/radius-project/dev/controller:pr-func9d668a1917
  • ucp test image location: ghcr.io/radius-project/dev/ucpd:pr-func9d668a1917
  • deployment-engine test image location: ghcr.io/radius-project/deployment-engine:latest

Test Status

⌛ Building Radius and pushing container images for functional tests...
✅ Container images build succeeded
⌛ Publishing Bicep Recipes for functional tests...
✅ Recipe publishing succeeded
⌛ Starting ucp-cloud functional tests...
⌛ Starting corerp-cloud functional tests...
✅ ucp-cloud functional tests succeeded
✅ corerp-cloud functional tests succeeded

Copy link
Contributor

@sk593 sk593 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@willdavsmith willdavsmith merged commit 835abb0 into radius-project:main Jan 28, 2025
29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants