Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Piotrm api server #30

Closed
wants to merge 52 commits into from
Closed
Changes from 1 commit
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
9a67e25
updated with skeleton / straw man structure
nnshah1 Jan 15, 2025
44304db
updated
nnshah1 Jan 15, 2025
f4a1ef0
updated
nnshah1 Jan 15, 2025
62b20c5
moving under operators
nnshah1 Jan 15, 2025
7dd83c1
updated skeleton
nnshah1 Jan 16, 2025
aafe096
Merge branch 'main' into nnshah1-hello-world
nnshah1 Jan 16, 2025
cad9d3f
updated with minimal example - not yet tested
nnshah1 Jan 16, 2025
41abd50
Merge branch 'nnshah1-hello-world' of https://github.com/triton-infer…
nnshah1 Jan 16, 2025
97c752d
updated to use deployment
nnshah1 Jan 16, 2025
b0f3464
updating with single_file separate from main
nnshah1 Jan 16, 2025
991358b
updating
nnshah1 Jan 16, 2025
bd57e10
docs: Add descriptions for example and main folder
piotrm-nvidia Jan 16, 2025
1109f65
updating to add operator to operators folder for us in 'deploy'
nnshah1 Jan 16, 2025
d16811c
Merge branch 'nnshah1-hello-world' of https://github.com/triton-infer…
nnshah1 Jan 16, 2025
88de51c
updates to remove operator from deploy
nnshah1 Jan 16, 2025
a7face5
incremental updates
nnshah1 Jan 16, 2025
c52b3ad
removing temporary files
nnshah1 Jan 16, 2025
ff72717
putting more common handling into deployment
nnshah1 Jan 16, 2025
3611828
updating default for single file
nnshah1 Jan 16, 2025
a01fd7b
Merge branch 'main' into nnshah1-hello-world
nnshah1 Jan 16, 2025
958f5f8
updating with deploy app
nnshah1 Jan 16, 2025
c4e6fed
updated with signal handler for exit
nnshah1 Jan 16, 2025
00aedbc
adding parser
nnshah1 Jan 16, 2025
c2dc190
removing gitkeep
nnshah1 Jan 16, 2025
3a74bcb
adding in initial client - still testing
nnshah1 Jan 17, 2025
e7bd20f
updated with working client
nnshah1 Jan 17, 2025
369259f
updated to remove version
nnshah1 Jan 17, 2025
9f56b11
removing version from tests
nnshah1 Jan 17, 2025
dadd9f0
adding basic exit code detection to help with sanity tests
nnshah1 Jan 17, 2025
2da91d7
adding sanity test
nnshah1 Jan 17, 2025
f2673f8
delete unnecessary file
nnshah1 Jan 17, 2025
348b578
adding copyright
nnshah1 Jan 17, 2025
5748a15
removing unnecessary files
nnshah1 Jan 17, 2025
f62b18e
Merge branch 'main' into nnshah1-hello-world
nnshah1 Jan 17, 2025
26fe1d8
updates for static type checks
nnshah1 Jan 17, 2025
9421700
adding sanity test to pre-merge until we have additional triggers ena…
nnshah1 Jan 17, 2025
7298502
reduce number of requests for sanity test
nnshah1 Jan 17, 2025
aac2881
Merge branch 'main' into nnshah1-hello-world
nnshah1 Jan 17, 2025
444a82a
removing version from RemoteOperator calls
nnshah1 Jan 17, 2025
671ce6a
adding copyright
nnshah1 Jan 17, 2025
2d0f1db
updating copyright
nnshah1 Jan 17, 2025
8f2173c
Add API server draft
piotrm-nvidia Jan 17, 2025
0b5b814
Add tests for API server
piotrm-nvidia Jan 17, 2025
45e2e46
Add more changes
piotrm-nvidia Jan 17, 2025
b10479e
Add server thread
piotrm-nvidia Jan 21, 2025
fac9da6
Adjust sleep to block server
piotrm-nvidia Jan 21, 2025
58a97e9
Add README.md to API server
piotrm-nvidia Jan 22, 2025
32c50d5
Adjustr import for Worker config
piotrm-nvidia Jan 22, 2025
f5ba2ce
Wrapp connector logic with LLMEngine
piotrm-nvidia Jan 22, 2025
3a79d11
Add parameters parsing
piotrm-nvidia Jan 22, 2025
9d7e8bd
Fix uvicorn harness in operator
piotrm-nvidia Jan 22, 2025
48b6a83
Shift FastAPI endpoint outside operator
piotrm-nvidia Jan 22, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
updating copyright
nnshah1 committed Jan 17, 2025
commit 2d0f1db06b43aafafc228780a829d4fa655f62cd
Original file line number Diff line number Diff line change
@@ -1,28 +1,17 @@
# Copyright 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# * Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
# * Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
# * Neither the name of NVIDIA CORPORATION nor the names of its
# contributors may be used to endorse or promote products derived
# from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
# SPDX-FileCopyrightText: Copyright (c) 2024-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at

# http://www.apache.org/licenses/LICENSE-2.0

# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

import asyncio
import gc
Original file line number Diff line number Diff line change
@@ -1,28 +1,17 @@
# Copyright 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# * Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
# * Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
# * Neither the name of NVIDIA CORPORATION nor the names of its
# contributors may be used to endorse or promote products derived
# from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
# SPDX-FileCopyrightText: Copyright (c) 2024-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at

# http://www.apache.org/licenses/LICENSE-2.0

# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

name: "mock_disaggregated_serving"
backend: "python"