Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update driver & launcher image handling #11533

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from
Draft

update driver & launcher image handling #11533

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2b49643
update driver & launcher image handling
HumairAK Jan 16, 2025
d55b4fa
test overlay
HumairAK Jan 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/resources/manifests/tekton/kustomization.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,3 +108,7 @@ patches:
resources:
requests:
storage: 5Gi


patchesStrategicMerge:
- overlays/apiserver-env.yaml
14 changes: 14 additions & 0 deletions .github/resources/manifests/tekton/overlays/apiserver-env.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: ml-pipeline
spec:
template:
spec:
containers:
- name: ml-pipeline-api-server
env:
- name: V2_DRIVER_IMAGE
value: gcr.io/ml-pipeline/kfp-driver@sha256:dc8b56a2eb071f30409828a8884d621092e68385af11a6c06aa9e9fbcfbb19de
- name: V2_LAUNCHER_IMAGE
value: gcr.io/ml-pipeline/kfp-launcher@sha256:50151a8615c8d6907aa627902dce50a2619fd231f25d1e5c2a72737a2ea4001e
4 changes: 2 additions & 2 deletions backend/src/v2/compiler/argocompiler/container.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,9 +30,9 @@ import (
const (
volumeNameKFPLauncher = "kfp-launcher"
volumeNameCABundle = "ca-bundle"
DefaultLauncherImage = "ghcr.io/kubeflow/kfp-launcher:2.4.0"
DefaultLauncherImage = "ghcr.io/kubeflow/kfp-launcher:latest"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please add a comment above mentioning that the latest tag will be replaced by with a pinned version at release time?

Copy link
Collaborator Author

@HumairAK HumairAK Jan 30, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chensun the goal is to pin the version from the manifests, seen here

Apologies for the large diff on the yaml, the only change is what is highlighted above in the link, the rest is just fixing indenting. The idea is to keep this latest in code (even for releases) and update the version in the manifests so we can build all these images (including api server, etc.) all at once during releases. Let me know what you think about this. Having versions embedded in code, requires us to always build driver/launcher, then commiting code changes first, before initiating the rest of the image builds afterwards.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If there is concern about having latest in code, in a release version, we can also make the driver/launcher env vars a hard requirement, and throw an error when it is not provided, and simply not have a default hardcoded tag at all

LauncherImageEnvVar = "V2_LAUNCHER_IMAGE"
DefaultDriverImage = "ghcr.io/kubeflow/kfp-driver:2.4.0"
DefaultDriverImage = "ghcr.io/kubeflow/kfp-driver:latest"
DriverImageEnvVar = "V2_DRIVER_IMAGE"
DefaultDriverCommand = "driver"
DriverCommandEnvVar = "V2_DRIVER_COMMAND"
Expand Down
312 changes: 158 additions & 154 deletions manifests/kustomize/base/pipeline/ml-pipeline-apiserver-deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,158 +16,162 @@ spec:
cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
spec:
containers:
- env:
- name: LOG_LEVEL
value: "info"
- name: AUTO_UPDATE_PIPELINE_DEFAULT_VERSION
valueFrom:
configMapKeyRef:
name: pipeline-install-config
key: autoUpdatePipelineDefaultVersion
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: OBJECTSTORECONFIG_SECURE
value: "false"
- name: OBJECTSTORECONFIG_BUCKETNAME
valueFrom:
configMapKeyRef:
name: pipeline-install-config
key: bucketName
# relic variables
- name: DBCONFIG_USER
valueFrom:
secretKeyRef:
name: mysql-secret
key: username
- name: DBCONFIG_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-secret
key: password
- name: DBCONFIG_DBNAME
valueFrom:
configMapKeyRef:
name: pipeline-install-config
key: pipelineDb
- name: DBCONFIG_HOST
valueFrom:
configMapKeyRef:
name: pipeline-install-config
key: dbHost
- name: DBCONFIG_PORT
valueFrom:
configMapKeyRef:
name: pipeline-install-config
key: dbPort
# end of relic variables
- name: DBCONFIG_CONMAXLIFETIME
valueFrom:
configMapKeyRef:
name: pipeline-install-config
key: ConMaxLifeTime
- name: DB_DRIVER_NAME
valueFrom:
configMapKeyRef:
name: pipeline-install-config
key: dbType
# MySQL Config
- name: DBCONFIG_MYSQLCONFIG_USER
valueFrom:
secretKeyRef:
name: mysql-secret
key: username
- name: DBCONFIG_MYSQLCONFIG_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-secret
key: password
- name: DBCONFIG_MYSQLCONFIG_DBNAME
valueFrom:
configMapKeyRef:
name: pipeline-install-config
key: pipelineDb
- name: DBCONFIG_MYSQLCONFIG_HOST
valueFrom:
configMapKeyRef:
name: pipeline-install-config
key: mysqlHost
- name: DBCONFIG_MYSQLCONFIG_PORT
valueFrom:
configMapKeyRef:
name: pipeline-install-config
key: mysqlPort
# end of MySQL Config
- name: OBJECTSTORECONFIG_ACCESSKEY
valueFrom:
secretKeyRef:
name: mlpipeline-minio-artifact
key: accesskey
- name: OBJECTSTORECONFIG_SECRETACCESSKEY
valueFrom:
secretKeyRef:
name: mlpipeline-minio-artifact
key: secretkey
image: ghcr.io/kubeflow/kfp-api-server:dummy
imagePullPolicy: IfNotPresent
name: ml-pipeline-api-server
ports:
- name: http
containerPort: 8888
- name: grpc
containerPort: 8887
readinessProbe:
exec:
command:
- wget
- -q # quiet
- -S # show server response
- -O
- "-" # Redirect output to stdout
- http://localhost:8888/apis/v1beta1/healthz
initialDelaySeconds: 3
periodSeconds: 5
timeoutSeconds: 2
livenessProbe:
exec:
command:
- wget
- -q # quiet
- -S # show server response
- -O
- "-" # Redirect output to stdout
- http://localhost:8888/apis/v1beta1/healthz
initialDelaySeconds: 3
periodSeconds: 5
timeoutSeconds: 2
# This startup probe provides up to a 60 second grace window before the
# liveness probe takes over to accomodate the occasional database
# migration.
startupProbe:
exec:
command:
- wget
- -q # quiet
- -S # show server response
- -O
- "-" # Redirect output to stdout
- http://localhost:8888/apis/v1beta1/healthz
failureThreshold: 12
periodSeconds: 5
timeoutSeconds: 2
securityContext:
allowPrivilegeEscalation: false
seccompProfile:
type: RuntimeDefault
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 0
capabilities:
drop:
- ALL
resources:
requests:
cpu: 250m
memory: 500Mi
- env:
- name: LOG_LEVEL
value: "info"
- name: AUTO_UPDATE_PIPELINE_DEFAULT_VERSION
valueFrom:
configMapKeyRef:
name: pipeline-install-config
key: autoUpdatePipelineDefaultVersion
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: OBJECTSTORECONFIG_SECURE
value: "false"
- name: OBJECTSTORECONFIG_BUCKETNAME
valueFrom:
configMapKeyRef:
name: pipeline-install-config
key: bucketName
# relic variables
- name: DBCONFIG_USER
valueFrom:
secretKeyRef:
name: mysql-secret
key: username
- name: DBCONFIG_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-secret
key: password
- name: DBCONFIG_DBNAME
valueFrom:
configMapKeyRef:
name: pipeline-install-config
key: pipelineDb
- name: DBCONFIG_HOST
valueFrom:
configMapKeyRef:
name: pipeline-install-config
key: dbHost
- name: DBCONFIG_PORT
valueFrom:
configMapKeyRef:
name: pipeline-install-config
key: dbPort
# end of relic variables
- name: DBCONFIG_CONMAXLIFETIME
valueFrom:
configMapKeyRef:
name: pipeline-install-config
key: ConMaxLifeTime
- name: DB_DRIVER_NAME
valueFrom:
configMapKeyRef:
name: pipeline-install-config
key: dbType
# MySQL Config
- name: DBCONFIG_MYSQLCONFIG_USER
valueFrom:
secretKeyRef:
name: mysql-secret
key: username
- name: DBCONFIG_MYSQLCONFIG_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-secret
key: password
- name: DBCONFIG_MYSQLCONFIG_DBNAME
valueFrom:
configMapKeyRef:
name: pipeline-install-config
key: pipelineDb
- name: DBCONFIG_MYSQLCONFIG_HOST
valueFrom:
configMapKeyRef:
name: pipeline-install-config
key: mysqlHost
- name: DBCONFIG_MYSQLCONFIG_PORT
valueFrom:
configMapKeyRef:
name: pipeline-install-config
key: mysqlPort
# end of MySQL Config
- name: OBJECTSTORECONFIG_ACCESSKEY
valueFrom:
secretKeyRef:
name: mlpipeline-minio-artifact
key: accesskey
- name: OBJECTSTORECONFIG_SECRETACCESSKEY
valueFrom:
secretKeyRef:
name: mlpipeline-minio-artifact
key: secretkey
- name: V2_DRIVER_IMAGE
value: ghcr.io/kubeflow/kfp-driver:2.4.0
- name: V2_LAUNCHER_IMAGE
value: ghcr.io/kubeflow/kfp-launcher:2.4.0
image: ghcr.io/kubeflow/kfp-api-server:dummy
imagePullPolicy: IfNotPresent
name: ml-pipeline-api-server
ports:
- name: http
containerPort: 8888
- name: grpc
containerPort: 8887
readinessProbe:
exec:
command:
- wget
- -q # quiet
- -S # show server response
- -O
- "-" # Redirect output to stdout
- http://localhost:8888/apis/v1beta1/healthz
initialDelaySeconds: 3
periodSeconds: 5
timeoutSeconds: 2
livenessProbe:
exec:
command:
- wget
- -q # quiet
- -S # show server response
- -O
- "-" # Redirect output to stdout
- http://localhost:8888/apis/v1beta1/healthz
initialDelaySeconds: 3
periodSeconds: 5
timeoutSeconds: 2
# This startup probe provides up to a 60 second grace window before the
# liveness probe takes over to accomodate the occasional database
# migration.
startupProbe:
exec:
command:
- wget
- -q # quiet
- -S # show server response
- -O
- "-" # Redirect output to stdout
- http://localhost:8888/apis/v1beta1/healthz
failureThreshold: 12
periodSeconds: 5
timeoutSeconds: 2
securityContext:
allowPrivilegeEscalation: false
seccompProfile:
type: RuntimeDefault
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 0
capabilities:
drop:
- ALL
resources:
requests:
cpu: 250m
memory: 500Mi
serviceAccountName: ml-pipeline
11 changes: 11 additions & 0 deletions manifests/kustomize/hack/release.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,3 +40,14 @@ do
done

yq w -i "${MANIFEST_DIR}/base/installs/generic/pipeline-install-config.yaml" data.appVersion "$TAG_NAME"

## Driver & Launcher images are added as environment variables
API_SERVER_MANIFEST="${MANIFEST_DIR}/base/pipeline/ml-pipeline-apiserver-deployment.yaml"

yq w -i ${API_SERVER_MANIFEST} \
"spec.template.spec.containers.(name==ml-pipeline-api-server).env.(name==V2_LAUNCHER_IMAGE).value" \
"ghcr.io/kubeflow/kfp-launcher:${TAG_NAME}"

yq w -i ${API_SERVER_MANIFEST} \
"spec.template.spec.containers.(name==ml-pipeline-api-server).env.(name==V2_DRIVER_IMAGE).value" \
"ghcr.io/kubeflow/kfp-driver:${TAG_NAME}"
Loading