Merge branch 'master' of github.com:skypilot-org/skypilot into update…

…-docs-dependency

.github/workflows/format.yml

@@ -33,14 +33,12 @@ jobs:
     - name: Running yapf
       run: |
         yapf --diff --recursive ./ --exclude 'sky/skylet/ray_patches/**' \
-            --exclude 'sky/skylet/providers/aws/**' \
             --exclude 'sky/skylet/providers/gcp/**' \
             --exclude 'sky/skylet/providers/azure/**' \
             --exclude 'sky/skylet/providers/ibm/**'
     - name: Running black
       run: |
-        black --diff --check sky/skylet/providers/aws/ \
-            sky/skylet/providers/gcp/ \
+        black --diff --check sky/skylet/providers/gcp/ \
             sky/skylet/providers/azure/ \
             sky/skylet/providers/ibm/
     - name: Running isort for black formatted files
@@ -50,7 +48,6 @@ jobs:
     - name: Running isort for yapf formatted files
       run: |
         isort --diff --check ./ --sg 'sky/skylet/ray_patches/**' \
-            --sg 'sky/skylet/providers/aws/**' \
             --sg 'sky/skylet/providers/gcp/**' \
             --sg 'sky/skylet/providers/azure/**' \
             --sg 'sky/skylet/providers/ibm/**'

README.md

@@ -55,16 +55,16 @@ SkyPilot **cuts your cloud costs**:
 
 SkyPilot supports your existing GPU, TPU, and CPU workloads, with no code changes.
 
-Install with pip:
+Install with pip (we recommend the nightly build for the latest features or [from source](https://skypilot.readthedocs.io/en/latest/getting-started/installation.html)):
 ```bash
-pip install "skypilot[aws,gcp,azure,ibm,oci,scp,lambda,kubernetes]"  # choose your clouds
+pip install "skypilot-nightly[aws,gcp,azure,oci,lambda,runpod,ibm,scp,kubernetes]"  # choose your clouds
 ```
-To get the latest features/updates, install [from source](https://skypilot.readthedocs.io/en/latest/getting-started/installation.html) or the nightly build:
+To get the last release, use:
 ```bash
-pip install -U "skypilot-nightly[aws,gcp,azure,ibm,oci,scp,lambda,kubernetes]"  # choose your clouds
+pip install -U "skypilot[aws,gcp,azure,oci,lambda,runpod,ibm,scp,kubernetes]"  # choose your clouds
 ```
 
-Current supported providers (AWS, Azure, GCP, Lambda Cloud, IBM, Samsung, OCI, Cloudflare, any Kubernetes cluster):
+Current supported providers (AWS, Azure, GCP, OCI, Lambda Cloud, RunPod, IBM, Samsung, Cloudflare, any Kubernetes cluster):
 <p align="center">
   <picture>
     <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/skypilot-org/skypilot/master/docs/source/images/cloud-logos-dark.png">

docs/source/getting-started/installation.rst

@@ -26,11 +26,12 @@ Install SkyPilot using pip:
           pip install "skypilot-nightly[aws]"
           pip install "skypilot-nightly[gcp]"
           pip install "skypilot-nightly[azure]"
-          pip install "skypilot-nightly[kubernetes]"
-          pip install "skypilot-nightly[lambda]"
           pip install "skypilot-nightly[oci]"
+          pip install "skypilot-nightly[lambda]"
+          pip install "skypilot-nightly[runpod]"
           pip install "skypilot-nightly[ibm]"
           pip install "skypilot-nightly[scp]"
+          pip install "skypilot-nightly[kubernetes]"
           pip install "skypilot-nightly[all]"
 
     .. tab-item:: Latest Release
@@ -48,11 +49,12 @@ Install SkyPilot using pip:
           pip install "skypilot[aws]"
           pip install "skypilot[gcp]"
           pip install "skypilot[azure]"
-          pip install "skypilot[kubernetes]"
-          pip install "skypilot[lambda]"
           pip install "skypilot[oci]"
+          pip install "skypilot[lambda]"
+          pip install "skypilot[runpod]"
           pip install "skypilot[ibm]"
           pip install "skypilot[scp]"
+          pip install "skypilot[kubernetes]"
           pip install "skypilot[all]"
 
     .. tab-item:: From Source
@@ -73,11 +75,12 @@ Install SkyPilot using pip:
           pip install -e ".[aws]"
           pip install -e ".[gcp]"
           pip install -e ".[azure]"
-          pip install -e ".[kubernetes]"
-          pip install -e ".[lambda]"
           pip install -e ".[oci]"
+          pip install -e ".[lambda]"
+          pip install -e ".[runpod]"
           pip install -e ".[ibm]"
           pip install -e ".[scp]"
+          pip install -e ".[kubernetes]"
           pip install -e ".[all]"
 
 To use more than one cloud, combine the pip extras:
@@ -126,12 +129,13 @@ This will produce a summary like:
     AWS: enabled
     GCP: enabled
     Azure: enabled
-    Kubernetes: enabled
-    Lambda: enabled
     OCI: enabled
+    Lambda: enabled
+    RunPod: enabled
     IBM: enabled
     SCP: enabled
     Cloudflare (for R2 object store): enabled
+    Kubernetes: enabled
 
   SkyPilot will use only the enabled clouds to run tasks. To change this, configure cloud credentials, and run sky check.
 
@@ -149,8 +153,8 @@ section :ref:`below <cloud-account-setup>`.
 Cloud account setup
 -------------------
 
-SkyPilot currently supports these cloud providers: AWS, GCP, Azure, IBM, OCI,
-SCP, Lambda Cloud, and Cloudflare (for R2 object store).
+SkyPilot currently supports these cloud providers: AWS, GCP, Azure, OCI, Lambda Cloud, RunPod,
+IBM, SCP, and Cloudflare (for R2 object store).
 
 If you already have cloud access set up on your local machine, run ``sky check`` to :ref:`verify that SkyPilot can properly access your enabled clouds<verify-cloud-access>`.
 
@@ -218,6 +222,29 @@ Azure
 
 Hint: run ``az account subscription list`` to get a list of subscription IDs under your account.
 
+
+Oracle Cloud Infrastructure (OCI)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To access Oracle Cloud Infrastructure (OCI), setup the credentials by following `this guide <https://docs.oracle.com/en-us/iaas/Content/API/Concepts/apisigningkey.htm>`__. After completing the steps in the guide, the :code:`~/.oci` folder should contain the following files:
+
+.. code-block:: text
+
+  ~/.oci/config
+  ~/.oci/oci_api_key.pem
+
+The :code:`~/.oci/config` file should contain the following fields:
+
+.. code-block:: text
+
+  [DEFAULT]
+  user=ocid1.user.oc1..aaaaaaaa
+  fingerprint=aa:bb:cc:dd:ee:ff:gg:hh:ii:jj:kk:ll:mm:nn:oo:pp
+  tenancy=ocid1.tenancy.oc1..aaaaaaaa
+  region=us-sanjose-1
+  key_file=~/.oci/oci_api_key.pem
+
+
 Lambda Cloud
 ~~~~~~~~~~~~~~~~~~
 
@@ -228,10 +255,22 @@ Lambda Cloud
   mkdir -p ~/.lambda_cloud
   echo "api_key = <your_api_key_here>" > ~/.lambda_cloud/lambda_keys
 
+
+RunPod
+~~~~~~~~~~
+
+`RunPod <https://runpod.io/>`__ is a specialized AI cloud provider that offers low-cost GPUs. To configure RunPod access, go to the `Settings <https://www.runpod.io/console/user/settings>`_ page on your RunPod console and generate an **API key**. Then, run:
+
+.. code-block:: shell
+  
+  pip install "runpod>=1.5.1"
+  runpod config
+
+
 IBM
 ~~~~~~~~~
 
-To access IBM's VPC service, store the following fields in ``~/.ibm/credentials.yaml``:
+To access `IBM's VPC service <https://www.ibm.com/cloud/vpc>`__, store the following fields in ``~/.ibm/credentials.yaml``:
 
 .. code-block:: text
 
@@ -263,26 +302,28 @@ Finally, install `rclone <https://rclone.org/>`_ via: ``curl https://rclone.org/
 .. note::
   :code:`sky check` does not reflect IBM COS's enabled status. :code:`IBM: enabled` only guarantees that IBM VM instances are enabled.
 
-Oracle Cloud Infrastructure (OCI)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-To access Oracle Cloud Infrastructure (OCI), setup the credentials by following `this guide <https://docs.oracle.com/en-us/iaas/Content/API/Concepts/apisigningkey.htm>`__. After completing the steps in the guide, the :code:`~/.oci` folder should contain the following files:
 
-.. code-block:: text
+Samsung Cloud Platform (SCP)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-  ~/.oci/config
-  ~/.oci/oci_api_key.pem
+Samsung Cloud Platform(SCP) provides cloud services optimized for enterprise customers. You can learn more about SCP `here <https://cloud.samsungsds.com/>`__.
 
-The :code:`~/.oci/config` file should contain the following fields:
+To configure SCP access, you need access keys and the ID of the project your tasks will run. Go to the `Access Key Management <https://cloud.samsungsds.com/console/#/common/access-key-manage/list?popup=true>`_ page on your SCP console to generate the access keys, and the Project Overview page for the project ID. Then, add them to :code:`~/.scp/scp_credential` by running:
 
-.. code-block:: text
+.. code-block:: shell
+
+  # Create directory if required
+  mkdir -p ~/.scp
+  # Add the lines for "access_key", "secret_key", and "project_id" to scp_credential file
+  echo "access_key = <your_access_key>" >> ~/.scp/scp_credential
+  echo "secret_key = <your_secret_key>" >> ~/.scp/scp_credential
+  echo "project_id = <your_project_id>" >> ~/.scp/scp_credential
+
+.. note::
+
+  Multi-node clusters are currently not supported on SCP.
 
-  [DEFAULT]
-  user=ocid1.user.oc1..aaaaaaaa
-  fingerprint=aa:bb:cc:dd:ee:ff:gg:hh:ii:jj:kk:ll:mm:nn:oo:pp
-  tenancy=ocid1.tenancy.oc1..aaaaaaaa
-  region=us-sanjose-1
-  key_file=~/.oci/oci_api_key.pem
 
 Cloudflare R2
 ~~~~~~~~~~~~~~~~~~
@@ -318,26 +359,6 @@ Next, get your `Account ID <https://developers.cloudflare.com/fundamentals/get-s
   Support for R2 is in beta. Please report and issues on `Github <https://github.com/skypilot-org/skypilot/issues>`_ or reach out to us on `Slack <http://slack.skypilot.co/>`_.
 
 
-Samsung Cloud Platform (SCP)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Samsung Cloud Platform(SCP) provides cloud services optimized for enterprise customers. You can learn more about SCP `here <https://cloud.samsungsds.com/>`__.
-
-To configure SCP access, you need access keys and the ID of the project your tasks will run. Go to the `Access Key Management <https://cloud.samsungsds.com/console/#/common/access-key-manage/list?popup=true>`_ page on your SCP console to generate the access keys, and the Project Overview page for the project ID. Then, add them to :code:`~/.scp/scp_credential` by running:
-
-.. code-block:: shell
-
-  # Create directory if required
-  mkdir -p ~/.scp
-  # Add the lines for "access_key", "secret_key", and "project_id" to scp_credential file
-  echo "access_key = <your_access_key>" >> ~/.scp/scp_credential
-  echo "secret_key = <your_secret_key>" >> ~/.scp/scp_credential
-  echo "project_id = <your_project_id>" >> ~/.scp/scp_credential
-
-.. note::
-
-  Multi-node clusters are currently not supported on SCP.
-
 Kubernetes
 ~~~~~~~~~~
 

docs/source/index.rst

@@ -122,6 +122,7 @@ Documentation
 
    serving/sky-serve
    serving/service-yaml-spec
+   serving/autoscaling
 
 .. toctree::
    :maxdepth: 1

docs/source/reference/kubernetes/index.rst

@@ -4,10 +4,9 @@ Running on Kubernetes
 =============================
 
 .. note::
-    Kubernetes support is in alpha preview and under active development.
-    There may be rough edges and features may change without notice.
-    Please report any `bugs <https://github.com/skypilot-org/skypilot/issues>`_ and
-    `reach out to us <http://slack.skypilot.co>`_ for feature requests.
+    Kubernetes support is under active development. `Please share your feedback <https://forms.gle/KmAtyNhEysiw2ZCR7>`_
+    or `directly reach out to the development team <http://slack.skypilot.co>`_
+    for feature requests and more.
 
 SkyPilot tasks can be run on your private on-prem or cloud Kubernetes clusters.
 The Kubernetes cluster gets added to the list of "clouds" in SkyPilot and SkyPilot

docs/source/serving/autoscaling.rst

@@ -0,0 +1,121 @@
+.. _serve-autoscaling:
+
+Autoscaling
+===========
+
+SkyServe provides out-of-the-box autoscaling for your services.
+
+Fixed Replicas
+--------------
+
+In a service YAML, the number of replicas to launch is specified in the ``service`` section's ``replicas`` field:
+
+.. code-block:: yaml
+    :emphasize-lines: 3
+
+    service:
+      readiness_probe: /
+      replicas: 2
+
+    # ...
+
+In this case, SkyServe will launch 2 replicas of your service. However, this deployment is fixed and cannot adjust to dynamic traffic.
+SkyServe provides autoscaling to help you scale your service up and down based on traffic, as shown below.
+
+Enabling Autoscaling
+--------------------
+
+Here is a minimal example to enable autoscaling for your service:
+
+.. code-block:: yaml
+    :emphasize-lines: 3-6
+
+    service:
+      readiness_probe: /
+      replica_policy:
+        min_replicas: 2
+        max_replicas: 10
+        target_qps_per_replica: 2.5
+
+    # ...
+
+In this example, SkyServe will:
+
+- Initially, launch 2 replicas of your service (``min_replicas``)
+- Scale up gradually if the traffic is high, up to 10 replicas (``max_replicas``)
+- Scale down gradually if the traffic is low, with a minimum of 2 replicas (``min_replicas``)
+
+The replica count will always be in the range
+:code:`[min_replicas, max_replicas]`.
+
+Autoscaling is performed based on the QPS (Queries Per Second) of your service.
+SkyServe will scale your service so that, ultimately, each replica receives
+approximately :code:`target_qps_per_replica` queries per second.
+This value can be a float; for example:
+
+- If the QPS is higher than 2.5 per replica, SkyServe will launch more replicas (but no more than 10 replicas)
+- If the QPS is lower than 2.5 per replica, SkyServe will scale down the replicas (but no less than 2 replicas)
+
+Specifically, the current target number of replicas is calculated as:
+
+.. code-block:: python
+
+    current_target_replicas = ceil(current_qps / target_qps_per_replica)
+    final_target_replicas = min(max_replicas, max(min_replicas, current_target_replicas))
+
+.. tip::
+
+    :code:`replicas` is a shortcut for :code:`replica_policy.min_replicas`. These two fields cannot be specified at the same time.
+
+.. tip::
+
+    :code:`target_qps_per_replica` can be any positive floating point number. If processing one request takes two seconds in one replica, we can use :code:`target_qps_per_replica=0.5`.
+
+Scaling Delay
+-------------
+
+SkyServe will not scale up or down immediately. Instead, SkyServe will only
+upscale or downscale your service if the QPS of your service is higher or lower
+than the target QPS for a period of time.  This is to avoid scaling up and down
+too aggressively.
+
+The default scaling delay is 300s for upscale and 1200s for downscale. You can
+change the scaling delay by specifying the :code:`upscale_delay_seconds` and
+:code:`downscale_delay_seconds` fields:
+
+.. code-block:: yaml
+    :emphasize-lines: 7-8
+
+    service:
+      readiness_probe: /
+      replica_policy:
+        min_replicas: 2
+        max_replicas: 10
+        target_qps_per_replica: 3
+        upscale_delay_seconds: 300
+        downscale_delay_seconds: 1200
+
+    # ...
+
+If you want more aggressive scaling, set those values to a lower number and vice versa.
+
+Scale-to-Zero
+===============
+
+SkyServe supports scale-to-zero.
+
+If your service might experience long periods of time with no traffic, consider using :code:`min_replicas: 0`:
+
+.. code-block:: yaml
+    :emphasize-lines: 4
+
+    service:
+      readiness_probe: /
+      replica_policy:
+        min_replicas: 0
+        max_replicas: 3
+        target_qps_per_replica: 6.3
+
+    # ...
+
+The service will scale down all replicas when there is no traffic to the system and will save costs on idle replicas. When upscaling from zero, the upscale delay will be ignored in order to bring up the service faster.

docs/source/serving/service-yaml-spec.rst

@@ -1,7 +1,7 @@
 .. _service-yaml-spec:
 
 Service YAML
-==========================
+============
 
 SkyServe provides an intuitive YAML interface to specify a service. It is highly similar to the :ref:`SkyPilot task YAML <yaml-spec>`: with an additional service section in your original task YAML, you could change it to a service YAML.