Skip to content

Latest commit

 

History

History
76 lines (58 loc) · 3.77 KB

persistence-agent-dev-instructions.md

File metadata and controls

76 lines (58 loc) · 3.77 KB

Persistence Agent

Prerequisites - you already have full-fledged Kubeflow installation (see here)

How to run persistent-agent locally (as in-cluster) ?

  • Get locally the Kube API certificate & token and persistenceagent-sa-token needed to Persistence Agent to communicate with Kubeflow Pipeline API
# copy in-cluster service account at /var/run/secrets/kubernetes.io/serviceaccount to local dev

PA_POD=$(kubectl get pods -n kubeflow -l app=ml-pipeline-persistenceagent -o jsonpath='{.items[0].metadata.name}')
kubectl exec -ti $PA_POD -n kubeflow -- cat /var/run/secrets/kubernetes.io/serviceaccount/ca.crt > /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
kubectl exec -ti $PA_POD -n kubeflow -- cat /var/run/secrets/kubernetes.io/serviceaccount/token > /var/run/secrets/kubernetes.io/serviceaccount/token
sudo mv $HOME/ca.crt /var/run/secrets/kubernetes.io/serviceaccount
sudo mv $HOME/token /var/run/secrets/kubernetes.io/serviceaccount

# copy in-cluster persistent-agent service account /var/run/secrets/tokens/ to local dev. 
# This SA has an expiration time and become invalid in specific period. Also if the pod is deleted, a new Persistent agent service account is created
# Better to increase the expiration time by editing ` kubectl edit deploy ml-pipeline-persistenceagent -n kubeflow ` -> `volumes/ projected/ sources/  serviceAccountToken / expirationSeconds to a greated number.
# Applicable once https://github.com/kubeflow/pipelines/pull/9699 got merged
sudo mkdir -p /var/run/secrets/kubeflow/tokens/
kubectl exec -ti $PA_POD  -n kubeflow -- cat /var/run/secrets/kubeflow/tokens/persistenceagent-sa-token > $HOME/persistenceagent-sa-token
sudo mv $HOME/persistenceagent-sa-token /var/run/secrets/kubeflow/tokens/persistenceagent-sa-token

  • NOTE: if you would like to run both KubeflowAPI server and Persistence Agent locally they conflict reading /var/run/secrets/kubernetes.io/serviceaccount

  • PersistenceAgent.run.xml

<component name="ProjectRunConfigurationManager">
    <configuration default="false" name="PersistentAgent" type="GoApplicationRunConfiguration" factoryName="Go Application">
        <module name="pipelines" />
        <working_directory value="$PROJECT_DIR$" />
        <useCustomBuildTags value="true" />
        <parameters value="--kubeconfig=$USER_HOME$/.kube/config --master=https://127.0.0.1:39385 --mlPipelineAPIServerName=localhost --mlPipelineServiceHttpPort=8888 --namespace=kubeflow-user-example-com" />
        <envs>
            <env name="KUBEFLOW_USERID_HEADER" value="kubeflow-userid" />
            <env name="KUBEFLOW_USERID_PREFIX" value="" />
            <env name="KUBERNETES_SERVICE_HOST" value="172.18.0.2" />
            <env name="KUBERNETES_SERVICE_PORT" value="6443" />
            <env name="MULTIUSER" value="True" />
        </envs>
        <kind value="DIRECTORY" />
        <directory value="$PROJECT_DIR$/backend/src/agent/persistence" />
        <filePath value="$PROJECT_DIR$/backend/src/agent/persistence/main.go" />
        <method v="2" />
    </configuration>
</component>

Hints:

--master=https://127.0.0.1:<PORT> comes from kubectl config view -cluser-

KUBERNETES_SERVICE_HOST could be taken from the output kubectl get nodes -o wide INTERNAL-IP column

Build Image

docker build -t docker.io/<DOCKER_USER>/persistenceagent:<tag> -f backend/Dockerfile.persistenceagent .
docker push <DOCKER_USER>/persistenceagent:<tag>
kubectl edit deployment.apps/ml-pipeline-persistenceagent -o yaml -n kubeflow

Other

kubectl api-resources -o wide
kubectl exec -it -n kubeflow  ml-pipeline-persistenceagent-6cb87cf64c-svsg2 -- /bin/sh

Original image: image: gcr.io/ml-pipeline/persistenceagent:1.8.1