-
Notifications
You must be signed in to change notification settings - Fork 5
feat: add endpoint to create cluster #41
feat: add endpoint to create cluster #41
Conversation
- endpoint access is limited to new `toolchain-operator` SA token - cluster service validates that the given cluster to create is valid - using cluster service mock to test the controller, to produce errors. fixes fabric8-services#22 Signed-off-by: Xavier Coulon <[email protected]>
@dipak-pawar @alexeykazakov the PR uses a mock of |
@xcoulon we will also need to create credentials for the operator SA and add them to Auth configuration/secret. We will need an issue for that. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah.. I think we don't really benefit from that service mock. We have to use mocks when services mare remote calls. But the cluster service talks to DB only. IMO regular integration tests is better here.
cluster/service/cluster_service.go
Outdated
// the `scheme` or `host` parts. Returns `true` otherwise. | ||
func validURL(urlStr string) (url.URL, bool) { | ||
u, err := url.Parse(urlStr) | ||
if err != nil || u.Scheme == "" || u.Host == "" { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn't we wrap the original error? Instead of returning bool I would return error
and if it's not nil then use wrap it in errInvalidURLMsg. Otherwise we can lose important error details from url.Parse().
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok, let me see that
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
addressed in 59cdcb7
cluster/service/cluster_service.go
Outdated
return *u, true | ||
} | ||
|
||
func forgeURL(baseURL url.URL, subdomain string) (string, error) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can use https://github.com/fabric8-services/fabric8-cluster/blob/master/cluster/cluster.go#L14 instead.
See an example of usage - https://github.com/fabric8-services/fabric8-cluster/blob/master/configuration/configuration.go#L191
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ah right, I forgot about this function. Thanks for the pointer!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
addressed in 59cdcb7
cluster/service/cluster_service.go
Outdated
return errors.NewBadParameterErrorFromString(fmt.Sprintf(errEmptyFieldMsg, "service-account-username")) | ||
} | ||
if clustr.TokenProviderID == "" { | ||
return errors.NewBadParameterErrorFromString(fmt.Sprintf(errEmptyFieldMsg, "token-provider-id")) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We actually could generate TokenProviderID if it's empty. TokenProviderID is used only by Auth service to associate user tokens obtained from the provider/cluster with that provider. We use provider ID because there are other token providers like GitHub.
So, TokenProviderID is basically some unique ID which will store in Auth external token DB when creating a token entry (so, we know that token came from that provider/cluster).
I would maybe set TokenProviderID (if empty) to the same value as ClusterID in our cluster repo, to minimize confusion we can have later on if we keep clusterID and TokenProviderID different.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok, let me do that (in which case, the ID will be generated by the app, not the DB, but that's a minor side-effect)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
addressed in 5467634
// when | ||
err := s.Application.ClusterService().CreateOrSaveCluster(context.Background(), &clustr) | ||
// then | ||
require.Error(t, err) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Check error message and type. You can use https://github.com/fabric8-services/fabric8-common/blob/b10e057d860d730661b2440dc1326c7d82606acc/test/error.go#L11 for that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
addressed in 545d475
// then | ||
require.Error(t, err) | ||
|
||
}) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Check all possible errors you can get (wrong URL, missing required params, etc)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is going to be a duplicate of the validation tests that I did in https://github.com/fabric8-services/fabric8-cluster/pull/41/files#diff-e9581e303b0b0335c4bed88053faff86R20. Here, I just wanted to make sure that in case of a validation error, the service returns an error
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You don't know if the validation is actually called. I would rather test the whole service which will test the internal validation function indirectly then relaying of testing implementation details of the service.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Your validate() function may work perfectly and all tests will pass but if this function is not used properly in the service or not used at all then the service won't work as expected but your tests won't catch it.
We have to make sure we test all exported functions very well with all possible outputs and usecases. Testing internal functions and implementation details is optional. Sometimes we have to test them because it's very tricky to test the exposed functions fully, so, it's better to test at least some the implementation parts (== not exposed functions) instead of the API (== exposed functions) than nothing. But it's not the case.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fair enough, I'll refactor the tests, because there's no point in having duplicate tests here, then
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
addressed in e54ba34
err := s.Application.ClusterService().CreateOrSaveCluster(context.Background(), &clustr) | ||
// then | ||
require.NoError(t, err) | ||
assert.NotNil(t, clustr.ClusterID) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Check the cluster was actually created. Check all the fields.
Also check if optional params are honored (consoleURL, etc are constructed as expected).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
checking that optional fields are honored is already done in https://github.com/fabric8-services/fabric8-cluster/pull/41/files#diff-e9581e303b0b0335c4bed88053faff86R36, but ok, let me check that the data actually exists with a repository call
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same as #41 (comment)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
addressed in e54ba34
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
all fields checked in 5467634
clusterSvc := c.app.ClusterService() | ||
err := clusterSvc.CreateOrSaveCluster(ctx, &clustr) | ||
if err != nil { | ||
return app.JSONErrorResponse(ctx, err) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Log the error.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, good point ;)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
addressed in 0de0dbd
a.Description("Add a cluster configuration") | ||
a.Response(d.Created) | ||
a.Response(d.Unauthorized, JSONAPIErrors) | ||
a.Response(d.InternalServerError, JSONAPIErrors) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add 400 Bad Request. And also tests for that :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
good point, indeed!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done in 7ba3711
a.Description("Add a cluster configuration") | ||
a.Response(d.Created) | ||
a.Response(d.Unauthorized, JSONAPIErrors) | ||
a.Response(d.InternalServerError, JSONAPIErrors) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
missing a.Response(d.BadRequest, JSONAPIErrors)
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, as @alexeykazakov also pointed out. Let me fix that ;)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done in 7ba3711
design/clusters.go
Outdated
@@ -13,6 +13,36 @@ var clusterList = JSONList( | |||
nil, | |||
nil) | |||
|
|||
// singleCluster represents a single cluster object |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
copypaste?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
probably...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
addressed in 59369a0
design/clusters.go
Outdated
// singleCluster represents a single cluster object | ||
var createCluster = JSONSingle( | ||
"Cluster", | ||
"Holds the response to a single cluster request", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it's not response, holds the data attributes to create cluster
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok, let me fix that
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
addressed in 59369a0
"app-dns", "type", | ||
"service-account-token", "service-account-username", "token-provider-id", "auth-client-id", "auth-client-secret", | ||
"auth-client-default-scope") | ||
}) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IMPO, we should user -
-> _
e.g. auth-client-secret
-> auth_client_secret
, as we are already at initial stage and let's have uniform way, so everywhere we'll use _
. If you agree on it I will update existing code.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure what is better to use _
or -
but I fully agree that we should be consistent.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I thought we agreed on -
last week ? See the whole content of https://github.com/fabric8-services/fabric8-cluster/blob/79d13b3f4773af3e71bbbeb8eebf5c6b631a044d/design/clusters.go
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, we agreed on -
. I know I have preference _
over -
. Just wanted to bring alexey's attention
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@dipak-pawar yes, consistency across the API design is what matters the most to me too (I have a personal preference for -
over _
but it's a personal taste, of course).
Now, if we decide to change the other endpoints' attributes pattern from -
to _
, we'll also need to redeploy all the other services with an updated client lib at the same time, otherwise all requests will fail. So, for the sake of simplicity, I suggest we carry on with -
to avoid troubles with the exists deployments. Ok for you?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Guys, just checked and we are very bad in consistency :) We use _
, -
, camelCase
all over the place. I mean in all our services. Not just here.
I do not have any preferences but let's pick something and try to stick with that. BTW is there any guidelines, good practices, etc, or it's just pure taste thing?
cluster/service/cluster_service.go
Outdated
|
||
// validate checks if all data in the given cluster is valid, and fills the missing/optional URLs using the `APIURL` | ||
func validate(clustr *repository.Cluster) error { | ||
if clustr.Name == "" { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is no need to validate empty params if you are mentioning it as required in design
, because goa will handle it e.g.
{"errors":[{"code":"bad_request","detail":"[zzwRTz/z] 400 invalid_request: attribute \"name\" of request is missing and required; parent: request","id":"CwNtrPHC","status":"400","title":"Bad Request"}]}
It's same for all below empty validation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
are you sure about that ? you're showing a JSON-API response, but I'm not sure Goa returns such a thing by default. Anyways, I'll double-check that
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, I am sure about it. Maybe we can check by trimming fields, to catch by some weired request with non-empty space, like " "
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
good point for trimming spaces from the fields, indeed. But what about the response format?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
addressed in 133b68b
@alexeykazakov ok, so I will remove the minimock parts and just rely on the actual service when testing the endpoint. |
@alexeykazakov ok, I just opened fabric8-services/fabric8-auth#723 for that matter. We'll need to fill an HK for SD teams, too. |
Signed-off-by: Xavier Coulon <[email protected]>
Signed-off-by: Xavier Coulon <[email protected]>
Signed-off-by: Xavier Coulon <[email protected]>
… the new 'auth.ToolChainOperator' const Signed-off-by: Xavier Coulon <[email protected]>
Codecov Report
@@ Coverage Diff @@
## master #41 +/- ##
==========================================
+ Coverage 73.5% 75.27% +1.76%
==========================================
Files 15 15
Lines 872 991 +119
==========================================
+ Hits 641 746 +105
- Misses 192 197 +5
- Partials 39 48 +9
Continue to review full report at Codecov.
|
Signed-off-by: Xavier Coulon <[email protected]>
Signed-off-by: Xavier Coulon <[email protected]>
Signed-off-by: Xavier Coulon <[email protected]>
@@ -78,4 +78,4 @@ tool/ | |||
migration/sqlbindata.go | |||
configuration/confbindata.go | |||
wit/witservice | |||
account/tenant | |||
account/tenant |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add test/generated
if we are using mini mock, if not then let's forget
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
are you asking me to restore the entry in .gitignore
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
no, forget it as we are not generating any mock for tests.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok
Signed-off-by: Xavier Coulon <[email protected]>
cluster/service/cluster_service.go
Outdated
@@ -147,21 +161,15 @@ func validate(clustr *repository.Cluster) error { | |||
|
|||
// validateURL validates the URL: return `false` if the given url could not be parsed or if it is missing |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Need to fix the comment. It returns error now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed in 790c3b0
@@ -145,7 +144,7 @@ func TestValidation(t *testing.T) { | |||
// then | |||
require.Error(t, err) | |||
require.IsType(t, errors.BadParameterError{}, err) | |||
assert.Equal(t, fmt.Sprintf(errInvalidURLMsg, "API", c.URL), err.(errors.BadParameterError).Error()) | |||
assert.Equal(t, "'API' URL '' is invalid: missing scheme or host", err.(errors.BadParameterError).Error()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why still not using https://github.com/fabric8-services/fabric8-common/blob/b10e057d860d730661b2440dc1326c7d82606acc/test/error.go#L11 here and in other places? You could use just one statement instead of three asserts/requires (error not nil, type, mesage). See an example of usage here - https://github.com/fabric8-services/fabric8-common/blob/b10e057d860d730661b2440dc1326c7d82606acc/auth/service_blackbox_test.go#L70
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok, let me do that
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
addressed in af4ee21
@xcoulon another important thing.. We should make sure we are consistent in storing URLs with trailing slashes or without them in our DB to avoid situations when we have "https://api.somecluster.com/" for some clusters and "https://api.anothercluster.com" for others. It will complicate our select queries. Let's make sure we add or remove all trailing slashes in all URLs in the cluster when storing them in DB. I guess it's better to do in cluster repo to make sure we don't miss a thing. |
…ovided by request also, check all fields in cluster upon creation by service Signed-off-by: Xavier Coulon <[email protected]>
Signed-off-by: Xavier Coulon <[email protected]>
Signed-off-by: Xavier Coulon <[email protected]>
what about URLs such as |
cluster/service/cluster_service.go
Outdated
} | ||
if strings.TrimSpace(clustr.TokenProviderID) == "" { | ||
// generated a value based on the ID of this cluster, so it's easier to track | ||
clustr.ClusterID = uuid.NewV4() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think we can set ClusterID here. CreateOrSave() will try to update the cluster if it exists. And we can't change its ID. It will fail to update then. I guess we have to load the cluster by API URL here and if it exists then use this ID?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
good point. Let me run a test to verify what happens when an existing cluster is updated, but no TokenProviderID
value is given
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
addressed in 638ecb7
@xcoulon there are two things. What we store in our DB and what we accept and return in our REST API. Let's say we always add trailing slashes when storing in DB. Then we know if some client wants to get a cluster for some URL we just have to add the trailing slash to that URL and execute a select. |
sure, that makes sense! Let me address this with a few tests to cover all case :) |
Signed-off-by: Xavier Coulon <[email protected]>
@@ -34,12 +34,12 @@ type Cluster struct { | |||
LoggingURL string | |||
// Application host name used by the cluster | |||
AppDNS string | |||
// Service Account token | |||
SaToken string | |||
// Encrypted Service Account token |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We don't know, it's encrypted or not by looking at this field. we need to look at SATokenEncrypted
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
addressed in eb0aa0a
Signed-off-by: Xavier Coulon <[email protected]>
@alexeykazakov addressed in fbe2a7f |
Signed-off-by: Xavier Coulon <[email protected]>
Signed-off-by: Xavier Coulon <[email protected]>
Signed-off-by: Xavier Coulon <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great work @xcoulon!
Just a few minor comments.
@@ -170,7 +170,12 @@ func (m *GormClusterRepository) CreateOrSave(ctx context.Context, c *Cluster) er | |||
obj, err := m.LoadClusterByURL(ctx, c.URL) | |||
if err != nil { | |||
if ok, _ := errors.IsNotFoundError(err); ok { | |||
return m.Create(ctx, c) | |||
err = m.Create(ctx, c) | |||
log.Info(ctx, map[string]interface{}{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well, if err!=nil then the cluster has not been created. If you want to log successful creation then use logging in Create() instead?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
good point!
and since the Create
method already logs a message upon successful creation or error, I'm going to remove it here ;)
return fmt.Errorf("missing scheme or host") | ||
} | ||
// make sure that the URL ends with a slash in all cases | ||
*urlStr = httpsupport.AddTrailingSlashToURL(*urlStr) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
validate()
is a bit confusing name for this function now. It's not just validation anymore. Shouldn't we call it validateAndNormalize()
or something?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sound good. let me rename this func to validateAndNormalize
} | ||
|
||
func newCreateClusterPayload() app.CreateClustersPayload { | ||
random := uuid.NewV4().String() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why to use the same value for all fields? Isn't it safer to use random and different field values in the test? Then we can differ them in the tests.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok, let me (partially) revert that one
Signed-off-by: Xavier Coulon <[email protected]>
toolchain-operator
SA tokenfixes #22
Signed-off-by: Xavier Coulon [email protected]