Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Move cluster assignment from status to spec #643

Open
wants to merge 20 commits into
base: release-v1.31.0
Choose a base branch
from
Open

[WIP] Move cluster assignment from status to spec #643

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
87a1201
using 1.32.0 tag
Aug 15, 2023
7715679
Added stanze def. for clusterscheduling.
dmatch01 Aug 16, 2023
bcd4127
Added and fixed example.
dmatch01 Aug 16, 2023
f8c7031
Added api field for targetClusterName
dmatch01 Aug 22, 2023
85ab98b
Merge pull request #2 from cwiklik/mcad-v131-add-cluster-status-field
dmatch01 Aug 22, 2023
0afc21f
added support for setting targetClusterName in status
Aug 24, 2023
0cbb7fb
added support to set target cluster id and delete appwrapper when no …
Aug 24, 2023
9e1821a
fixed assignment for targetClusterName and modified cleanup()
Aug 24, 2023
40b121f
modified helm related resources to support new option externalDispatch
Aug 24, 2023
7d97f3e
fixed wrong assignment of targetClusterName
Aug 25, 2023
fd0d436
more fixes to support setting targetClusterName
Aug 25, 2023
0976f11
Merge pull request #3 from cwiklik/set-targetClusterName-in-spec
cwiklik Aug 26, 2023
e972e07
fixes supporting status
Aug 31, 2023
9f67d0f
add completion status to every item in a pipeline
Sep 6, 2023
0bbda5a
simplified implementation of gvk in completion status for each item i…
Sep 7, 2023
c795e2a
redone gvk. Modified dispatcher so that it handled job completion status
Sep 8, 2023
bc0966d
add new status-check test suite
Sep 17, 2023
f3aea37
moved cluster assignment to spec
Sep 18, 2023
c192f7e
added clusterDecision to appwrappers crd
Sep 19, 2023
4d7ab51
added TODO
Sep 24, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions cmd/kar-controllers/app/options/options.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@ type ServerOption struct {
QuotaRestURL string
HealthProbeListenAddr string
DispatchResourceReservationTimeout int64
ExternalDispatch bool // if true, will use external plugin to dispatch workloads
}

// NewServerOption creates a new CMServer with a default config.
Expand Down Expand Up @@ -83,6 +84,8 @@ func (s *ServerOption) AddFlags(fs *flag.FlagSet) {
fs.IntVar(&s.SecurePort, "secure-port", 6443, "The port on which to serve secured, authenticated access for metrics.")
fs.StringVar(&s.HealthProbeListenAddr, "healthProbeListenAddr", ":8081", "Listen address for health probes. Defaults to ':8081'")
fs.Int64Var(&s.DispatchResourceReservationTimeout, "dispatchResourceReservationTimeout", s.DispatchResourceReservationTimeout, "Resource reservation timeout for pods to be created once AppWrapper is dispatched, in millisecond. Defaults to '300000', 5 minutes")
fs.BoolVar(&s.ExternalDispatch,"externalDispatch", s.ExternalDispatch,"Use external workload dispatch plugin. Default is false.")

flag.Parse()
klog.V(4).Infof("[AddFlags] Controller configuration: %#v", s)
}
Expand Down Expand Up @@ -147,6 +150,12 @@ func (s *ServerOption) loadDefaultsFromEnvVars() {
s.DispatchResourceReservationTimeout = to
}
}
externalDispatch, envVarExists := os.LookupEnv("EXTERNAL_DISPATCH")
s.ExternalDispatch = false
if envVarExists && strings.EqualFold(externalDispatch, "true") {
s.ExternalDispatch = true
}

}

func (s *ServerOption) CheckOptionOrDie() {
Expand Down
131 changes: 128 additions & 3 deletions config/crd/bases/mcad.ibm.com_appwrappers.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,10 +62,10 @@ spec:
completionstatus:
description: Optional field that drives completion status of appwrapper.
This field within an item of an appwrapper determines the full state of appwrapper.
The completionstatus field contains a list of conditions that make the associate item considered
The completionstatus field contains a list of conditions that make the associate item considered
completed, for instance :- completion conditions could be "Complete" or "Failed".
The associated item's level .status.conditions[].type field is monitored for any one of these conditions. Once all items with this
option is set and the conditionstatus is met the entire appwrapper state will be changed to one of the valid appwrapper completion state. Note :- this is an AND
The associated item's level .status.conditions[].type field is monitored for any one of these conditions. Once all items with this
option is set and the conditionstatus is met the entire appwrapper state will be changed to one of the valid appwrapper completion state. Note :- this is an AND
operation for all items where this option is set. See the list of appwrapper states for a list of valid complete states.
type: string

Expand Down Expand Up @@ -299,6 +299,98 @@ spec:
type: integer
default: 0
type: object
clusterScheduling:
description: Stanza to define cluster allocation restrictions.
Used only in multiCluster mode.
properties:
clusters:
description: Specifies cluster IDs for allocation
restrictions. Used only in
multiCluster mode.
items:
properties:
name:
description: Cluster ID name. Used only in
multiCluster mode.
type: string
type: object
type: array
clusterSelector:
description: (UNSUPPORTED FIELD) A label selector is a label
query over a set of cluster objects. The result of
matchLabels and matchExpressions are ANDed. An empty
label selector matches all objects. A null label selector
matches no objects.
properties:
matchExpressions:
description: matchExpressions is a list of label selector requirements.
The requirements are ANDed.
items:
description: A label selector requirement is a selector that
contains values, a key, and an operator that relates the key
and values.
properties:
key:
description: key is the label key that the selector applies
to.
type: string
operator:
description: operator represents a key's relationship to
a set of values. Valid operators are In, NotIn, Exists
and DoesNotExist.
type: string
values:
description: values is an array of string values. If the
operator is In or NotIn, the values array must be non-empty.
If the operator is Exists or DoesNotExist, the values
array must be empty. This array is replaced during a strategic
merge patch.
items:
type: string
type: array
required:
- key
- operator
type: object
type: array
matchLabels:
additionalProperties:
type: string
description: matchLabels is a map of {key,value} pairs. A single
{key,value} in the matchLabels map is equivalent to an element
of matchExpressions, whose key field is "key", the operator
is "In", and the values array contains only "value". The requirements
are ANDed.
type: object
type: object
policyResult:
properties:
policySource:
items:
properties:
lastUpdateMicroTime:
description: The latest time this condition was updated.
format: date-time
type: string
message:
description: A human readable message indicating details
about the cluster decision.
type: string
name:
description: ID/Name of the policy decision maker. Most
often this will be MCAD but design can support alternatives
type: string
type: object
type: array
targetCluster:
properties:
name:
type: string
required:
- name
type: object
type: object
type: object
dispatchDuration:
description: Wall clock duration time of appwrapper in seconds
properties:
Expand Down Expand Up @@ -822,6 +914,39 @@ spec:
(is this different from the MinAvailable from JobStatus)
format: int32
type: integer
itemCompletionStatus:
properties:
genericItems:
items:
properties:
condition:
type: string
gvk:
description: Group Version Kind
properties:
Group:
type: string
Version:
type: string
Kind:
type: string
required:
- Group
- Version
- Kind
type: object
name:
type: string
namespace:
type: string
required:
- condition
- gvk
- name
- namespace
type: object
type: array
type: object
type: object
required:
- spec
Expand Down
92 changes: 92 additions & 0 deletions config/crd/bases/mcad.ibm.com_schedulingspecs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,98 @@ spec:
type: integer
default: 0
type: object
clusterScheduling:
description: Stanza to define cluster allocation restrictions.
Used only in multiCluster mode.
properties:
clusters:
description: Specifies cluster IDs for allocation
restrictions. Used only in
multiCluster mode.
items:
properties:
name:
description: Cluster ID name. Used only in
multiCluster mode.
type: string
type: object
type: array
clusterSelector:
description: (UNSUPPORTED FIELD) A label selector is a label
query over a set of cluster objects. The result of
matchLabels and matchExpressions are ANDed. An empty
label selector matches all objects. A null label selector
matches no objects.
properties:
matchExpressions:
description: matchExpressions is a list of label selector requirements.
The requirements are ANDed.
items:
description: A label selector requirement is a selector that
contains values, a key, and an operator that relates the key
and values.
properties:
key:
description: key is the label key that the selector applies
to.
type: string
operator:
description: operator represents a key's relationship to
a set of values. Valid operators are In, NotIn, Exists
and DoesNotExist.
type: string
values:
description: values is an array of string values. If the
operator is In or NotIn, the values array must be non-empty.
If the operator is Exists or DoesNotExist, the values
array must be empty. This array is replaced during a strategic
merge patch.
items:
type: string
type: array
required:
- key
- operator
type: object
type: array
matchLabels:
additionalProperties:
type: string
description: matchLabels is a map of {key,value} pairs. A single
{key,value} in the matchLabels map is equivalent to an element
of matchExpressions, whose key field is "key", the operator
is "In", and the values array contains only "value". The requirements
are ANDed.
type: object
type: object
policyResult:
properties:
policySource:
items:
properties:
lastUpdateMicroTime:
description: The latest time this condition was updated.
format: date-time
type: string
message:
description: A human readable message indicating details
about the cluster decision.
type: string
name:
description: ID/Name of the policy decision maker. Most
often this will be MCAD but design can support alternatives
type: string
type: object
type: array
targetCluster:
properties:
name:
type: string
required:
- name
type: object
type: object
type: object
dispatchDuration:
description: Wall clock duration time of appwrapper in seconds
properties:
Expand Down
Loading