Add Tekton-backed Kubeflow Pipelines

[anishasthana]

Closes operate-first/support#435

Signed-off-by: Anish Asthana [email protected]

[anishasthana]

Need to figure out the implications of the required SCC, probably move some of the contents of this PR around to different folders too.

[HumairAK]

/hold

[HumairAK]

Okay I'm going through the manifests. So off the bat, all the cluster-scope manifests should go to the cluster-scope directory here: https://github.com/operate-first/apps/tree/master/cluster-scope

See the readme docs in that link on how to structure it.

You might find this tool usefule for this https://github.com/larsks/halberd Remember you have to pull these into the smaug overlay.

Having said that, there are a lot of questionable manifests here ... in my opinion, at least for what it seems to be asking. And I will try to go through them.

[HumairAK]

This PR is installing csi-s3:

This is a Container Storage Interface (CSI) for S3 (or S3 compatible) storage. This can dynamically allocate buckets and mount them via a fuse mount into any container.

I don't think we want to just install a new CSI that can mount s3 storage w/o a separate discussion on why we should add it...

Do we need this? Which component needs it? Can we use the s3 we already have?

I see that this all results in a new storage class kfp-csi-s3 but...there are no pvs/pvcs that use it, leading me to believe it's not being utilized.

[HumairAK]

@anishasthana I also noticed in your demo deployed on the dev cluster that the cache-deployer and cache-server are not running successfully either. Is this something we need? Maybe we can try taking these out to reduce some of this clutter.

I skimmed through the manifests/configfiles/secrets and I also don't see any reference to the cache-deployer service. (could be I just missed it)

[HumairAK]

So there's a minio deployment also included, we could probably trim this down and just use an OBC to get our bucket. The following deployments seem to require the s3 creds:

apiVersion: apps/v1
kind: Deployment
metadata:
    name: ml-pipeline-ui
    namespace: kubeflow
---

apiVersion: apps/v1
kind: Deployment
metadata:
    name: ml-pipeline
    namespace: kubeflow

Both of these require s3 creds from a secret, we can sub our own s3 for these.
The secret they are currently mounting is: mlpipeline-minio-artifact
The deployment ml-pipeline also mounts a configmap called kfp-tekton-config which seems to take in the s3 endpoint.

[anishasthana]

We disabled creds-init by setting the disable-creds-init field in the feature-flag configmap in the openshift-pipelines namespace to True. This fixed another error.
Context: shipwright-io/build#679 (comment)

[anishasthana]

track_artifacts: "false" in the kfp-tekton config map

[HumairAK]

/lgtm
/approve

[sesheta]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: HumairAK

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [HumairAK]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment