Adding some initial docs content and diagrams #129
Conversation
k8s-ci-robot
added
the
cncf-cla: yes
k8s-ci-robot
added
the
size/M
|
/lgtm |
k8s-ci-robot
added
the
lgtm
| Gateway API has [more than 25 | ||
| implementations](https://gateway-api.sigs.k8s.io/implementations/). As this | ||
| pattern stabilizes, we expect a wide set of these implementations to support |
There was a problem hiding this comment.
not clear what pattern this refers to, perhaps we should say the word pattern in the composable layer section intro so that the reader can connect the dots
| ## Composable Layers | ||
|
|
||
| This project aims to develop an ecosystem of implementations that are fully |
There was a problem hiding this comment.
What do we mean by "ecosystem of implementations"? This project aims to "define specifications to enable a compatible ecosystem for extending the Gateway API with custom endpoint selection algorithms"?
There was a problem hiding this comment.
+1 on ^ description from @ahg-g
|
|
||
| As part of this project, we're building an initial reference extension that is | ||
| focused on routing to LoRA workloads. Over time, we hope to see a wide variety |
There was a problem hiding this comment.
I prefer not to give the impression that this is centered on LoRA, in reality LoRA is just one of multiple criteria for selection.
| [vLLM](https://github.com/vllm-project/vllm) and | ||
| [Triton](https://github.com/triton-inference-server/server), and will be open to | ||
| other integrations as they are requested. |
|
Part of #72 |
k8s-ci-robot
removed
the
lgtm
|
New changes are detected. LGTM label has been removed. |
k8s-ci-robot
added
size/L
k8s-ci-robot
added
size/XL
| processor: | ||
| ignoreTypes: | ||
| - "(InferencePool|InferenceModel)List$" | ||
| # RE2 regular expressions describing type fields that should be excluded from the generated documentation. |
|
|
||
| render: | ||
| # Version of Kubernetes to use when generating links to Kubernetes API documentation. | ||
| kubernetesVersion: 1.31 |
There was a problem hiding this comment.
Can this value be an environment variable? I ask b/c it would be nice to have all version dependencies centrally located, e.g. Makefile.
|
|
||
| * InferencePool is supported as a backend type | ||
| * Implementations forward requests to the configured extension for an | ||
| InferencePool using the protocol specified by this project |
There was a problem hiding this comment.
using the protocol specified by this project
Can you clarify? Are you referencing to protocol as xRoute types supported by the implementation?
|
|
||
| The overall resource model focuses on 2 new inference-focused | ||
| [personas](/concepts/roles-and-personas) and corresponding resources that |
There was a problem hiding this comment.
Note that roles-and-personas.md is in a TODO state.
| replace a Kubernetes Service with an InferencePool. This resource has some | ||
| similarities to Service (a way to select Pods and specify a port), but has some | ||
| unique capabilities. With InferenceModel, you can configure a routing extension |
There was a problem hiding this comment.
but has some unique capabilities.
It would be helpful to talk about what the unique capabilities are or provide an example of a unique capability.
With InferenceModel, you can configure a routing extension as well as inference-specific routing optimizations.
Why is InferenceModel being highlighted here? Are you trying to describe how an InferencePool combined with an InferenceModel provides the unique capabilities described above?
| with that model. This resource enables you to configure the relative criticality |
There was a problem hiding this comment.
s/configuration associated with that model./ its associated configuration./
| ## Composable Layers | ||
|
|
||
| This project aims to develop an ecosystem of implementations that are fully |
There was a problem hiding this comment.
+1 on ^ description from @ahg-g
| metrics probing may happen asynchronously, depending on the extension. | ||
|
|
||
| 4. The extension will instruct the Gateway which endpoint should be routed to. |
There was a problem hiding this comment.
s/which endpoint should be routed to./which endpoint the request should be routed to./
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: danehans, robscott The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Initial WIP attempt of some content for our docs.