[K8s] Custom Image Support for Kubernetes Instances #2729

landscapepainter · 2023-10-23T01:29:54Z

This PR resolves #2599

When users opt to provide their custom images for setting up the k8s instance, instead of relying on Skypilot's default image built with Dockerfile_k8s, certain assumptions and adjustments are made to ensure smooth operation:

Base Distribution: We expect users to provide images that are based on Debian-based distributions.
User Privilege: It's crucial for users to configure the default user's privilege correctly within their custom image by either setting the default user to have root privilege or installing sudo.
Dependency Installation & SSH Setup: Instead of baking these directly into the image, we've shifted the responsibility of installing dependencies and setting up SSH to the ray manifest. These processes are triggered after the pods are active and are managed within node_provider.py under the create_node() method.
SSH user update: We have default ssh_user of sky set in kubernetes-ray.yml.j2. However, as users' may have different user name, we need to update the ssh_user in the cluster yaml and the SSHCommandRunner used within NodeProvider. This is each done from node_provider.py/create_node() and node_provider.py/get_command_runner().

By adopting these measures, we aim to offer greater flexibility for users bringing their own images while ensuring consistent system behavior.

Update: This PR supports public registries only, and private registries may be supported in near future. Until then, there will be a doc on how to setup for the private registry by the user.

Tested (run the relevant ones):

Code formatting: bash format.sh
Any manual or new tests for this PR (please specify below)
All smoke tests: pytest tests/test_smoke.py
Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
Backward compatibility tests: bash tests/backward_comaptibility_tests.sh

Tested Base Images:

continuumio/miniconda3:22.11.1
nvidia/cuda:12.2.0-devel-ubuntu20.04
pytorch/pytorch:latest
pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime
tensorflow/tensorflow:latest
tensorflow/tensorflow:nightly-gpu-jupyter
ubuntu:latest
gpuci/miniforge-cuda:11.4-devel-ubuntu18.04

Tested Registries:

AWS ECR public
GCP Artifact Registry public
Docker Registry public

Early failovers:

AWS ECR private handled with error message
Docker private registry handled with error message with early failure
GCP Registry failover when user's not configured for the GCP account
GCP Registry failover when user's configured GCP account does not have IAM authorization
Attempting to pull from non-existent repository
Attempting to pull from non-existent tag name with correct repository

landscapepainter · 2023-11-22T23:03:45Z

Adding curl to be installed in setup_commands at kubernetes-ray.yaml.j2 to address #2799.

…o k8s_custom_image # Conflicts: # Dockerfile_k8s # sky/backends/backend_utils.py # sky/clouds/kubernetes.py

… k8s_custom_image

romilbhardwaj · 2024-01-21T10:38:35Z

Made a few fixes and added pod_override ability to ~/sky/config.yaml. This allows users to specify custom pod spec overrides, such as custom labels, imagePullSecrets, imagePullPolicy etc as required through the sky config yaml. Helps a lot by allowing users to configure pods to their environments.

Quick guide for anyone interested in trying out:
To use a custom image, use the image_id: docker:<your-image-tag> field under resources field in your YAML. Here's an example:

resources:
  cloud: kubernetes
  accelerators: T4:1
  image_id: docker:638892464027.dkr.ecr.us-west-2.amazonaws.com/romilpvtrepo:latest # Or nvidia/cuda:11.8.0-devel-ubuntu18.04

If your custom image is hosted on a private registry, you'll need to setup the Kubernetes secret to access it. Once the secret is setup, add the following to your ~/.sky/config.yaml:

kubernetes:
  pod_override:
    spec:
      imagePullSecrets:
        - name: <your-secret-name>

This will automatically add the secret to the pod spec when SkyPilot creates the pod.

Michaelvll · 2024-01-24T07:38:46Z

sky/templates/kubernetes-ray.yml.j2

    (pip3 list | grep skypilot && [ "$(cat {{sky_remote_path}}/current_sky_wheel_hash)" == "{{sky_wheel_hash}}" ]) || (pip3 uninstall skypilot -y; pip3 install "$(echo {{sky_remote_path}}/{{sky_wheel_hash}}/skypilot-{{sky_version}}*.whl)[remote]" && echo "{{sky_wheel_hash}}" > {{sky_remote_path}}/current_sky_wheel_hash || exit 1);
    sudo bash -c 'rm -rf /etc/security/limits.d; echo "* soft nofile 1048576" >> /etc/security/limits.conf; echo "* hard nofile 1048576" >> /etc/security/limits.conf';
    sudo grep -e '^DefaultTasksMax' /etc/systemd/system.conf || (sudo bash -c 'echo "DefaultTasksMax=infinity" >> /etc/systemd/system.conf'); sudo systemctl set-property user-$(id -u $(whoami)).slice TasksMax=infinity; sudo systemctl daemon-reload;
    mkdir -p ~/.ssh; (grep -Pzo -q "Host \*\n  StrictHostKeyChecking no" ~/.ssh/config) || printf "Host *\n  StrictHostKeyChecking no\n" >> ~/.ssh/config;
+    pip3 install ray==2.4.0 aiohttp opencensus prometheus_client aiohttp_cors kubernetes==28.1.0;


Ray installation sould go before skypilot installation and needs some patch to make it work. Reference:

skypilot/sky/templates/aws-ray.yml.j2

Line 163 in efdd07d

(pip3 list | grep "ray " | grep {{ray_version}} 2>&1 > /dev/null || pip3 install --exists-action w -U ray[default]=={{ray_version}});

skypilot/sky/templates/aws-ray.yml.j2

Line 170 in efdd07d

python3 -c "from sky.skylet.ray_patches import patch; patch()" || exit 1;

sky/skylet/providers/kubernetes/node_provider.py

Michaelvll · 2024-01-24T18:07:44Z

sky/skylet/providers/kubernetes/node_provider.py

+                    f'Setting up SSH in pod.')
+        self._setup_ssh_in_pods(new_nodes)
+        logger.info(config.log_prefix +
+                    f'Setting up environment variables in pod.')
        self._set_env_vars_in_pods(new_nodes)


Do we need to setup envs for jump pod?

The envs are necessary for pods to interact with GPUs, so it's only necessary for the instance pods, not for jump pod.

sky/skylet/providers/kubernetes/node_provider.py

…r/skypilot into k8s_custom_image

Michaelvll · 2024-01-30T17:57:56Z

Thanks @landscapepainter for submitting this PR! Since the changes has been adopted into #3019 and merged into master, probably we should close this PR. : )

Michaelvll

Thanks for adding the support for k8s custom image @landscapepainter! Most of the parts of this PR has been adopted in #3019, but we should have got this PR in first before that PR to avoid losing the commit history. Let's get this PR in to keep the commit history.

Note for people looking at this PR: the node_provider is deprecated for k8s support and all the awesome changes/commits from @landscapepainter has been applied to #3019. We merged this PR for tracking the commit history. : )

landscapepainter added 4 commits October 8, 2023 23:45

testing images

079caff

nit

0912fab

update

2f08ea2

custom image support

115840d

landscapepainter marked this pull request as draft October 23, 2023 01:30

landscapepainter added 15 commits October 28, 2023 04:12

check root privilege and installation of sudo

18efb2b

Update ssh_user with user from custom image

ff5ec12

support custom image from task yaml

e5603ed

nit

faaa157

nit

5ff053d

nit

bb67a82

check if default user can run 'sudo'

5a15234

nit

656aa3a

nit

00f3db1

nit

ea66277

updated to support 'docker:' for k8s custom image

13298ca

fix conda init issue

8892352

nit

5410dc6

install rsync from create_node()

2e01767

format

acc039a

landscapepainter marked this pull request as ready for review November 6, 2023 06:30

landscapepainter added 3 commits November 21, 2023 06:47

Merge branch 'master' into k8s_custom_image

f7ca4cf

factor out function for running cmd on pods

beeab4e

nit

d9f2573

landscapepainter added 2 commits November 22, 2023 23:04

add 'curl' in setup_commands

9d7232b

update Dockerfile_k8s

eca405c

romilbhardwaj mentioned this pull request Dec 6, 2023

[k8s] change the container images for k8s #2846

Closed

lint

d800011

landscapepainter requested a review from romilbhardwaj December 11, 2023 06:18

landscapepainter and others added 6 commits January 20, 2024 00:24

nit

cbce4cf

Merge branch 'master' of https://github.com/skypilot-org/skypilot int…

7e5fc60

…o k8s_custom_image # Conflicts: # Dockerfile_k8s # sky/backends/backend_utils.py # sky/clouds/kubernetes.py

Merge remote-tracking branch 'landscapepainter/k8s_custom_image' into…

b7c69bb

… k8s_custom_image

fixes

3e82711

Add support for pod_override in config

b6037b2

lint

8e8aa6b

romilbhardwaj added 4 commits January 21, 2024 11:01

logging and apt fixes

16a1775

apt fixes

f38402b

multinode fixes

2545ac1

multinode fixes

efdd07d

Michaelvll reviewed Jan 24, 2024

View reviewed changes

sky/skylet/providers/kubernetes/node_provider.py Show resolved Hide resolved

Michaelvll reviewed Jan 24, 2024

View reviewed changes

sky/skylet/providers/kubernetes/node_provider.py Show resolved Hide resolved

Michaelvll reviewed Jan 24, 2024

View reviewed changes

sky/skylet/providers/kubernetes/node_provider.py Show resolved Hide resolved

landscapepainter added 7 commits January 30, 2024 17:01

Merge branch 'master' into k8s_custom_image

e3acc67

Merge branch 'k8s_custom_image' of https://github.com/landscapepainte…

3d0ad19

…r/skypilot into k8s_custom_image

nit

2101751

nit

1474598

nit

0eed02e

nit

fe394ee

nit

3a40507

Michaelvll closed this Jan 31, 2024

landscapepainter reopened this Jan 31, 2024

rebase k8s ray yaml

2b842b9

Michaelvll approved these changes Jan 31, 2024

View reviewed changes

landscapepainter merged commit b6ccf1c into skypilot-org:master Jan 31, 2024
19 checks passed

romilbhardwaj mentioned this pull request Feb 19, 2024

[k8s] Add curl to SkyPilot Kubernetes base image #2799

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[K8s] Custom Image Support for Kubernetes Instances #2729

[K8s] Custom Image Support for Kubernetes Instances #2729

landscapepainter commented Oct 23, 2023 •

edited

Loading

landscapepainter commented Nov 22, 2023

romilbhardwaj commented Jan 21, 2024

Michaelvll Jan 24, 2024

Michaelvll Jan 24, 2024

landscapepainter Jan 26, 2024 •

edited

Loading

Michaelvll commented Jan 30, 2024

Michaelvll left a comment

[K8s] Custom Image Support for Kubernetes Instances #2729

[K8s] Custom Image Support for Kubernetes Instances #2729

Conversation

landscapepainter commented Oct 23, 2023 • edited Loading

landscapepainter commented Nov 22, 2023

romilbhardwaj commented Jan 21, 2024

Michaelvll Jan 24, 2024

Choose a reason for hiding this comment

Michaelvll Jan 24, 2024

Choose a reason for hiding this comment

landscapepainter Jan 26, 2024 • edited Loading

Choose a reason for hiding this comment

Michaelvll commented Jan 30, 2024

Michaelvll left a comment

Choose a reason for hiding this comment

landscapepainter commented Oct 23, 2023 •

edited

Loading

landscapepainter Jan 26, 2024 •

edited

Loading