setup.sh script multiple model support using HF download #61

tstescoTT · 2024-12-20T04:09:28Z

change log

add multiple model support using persistent_volume/model_envs/*.env
setup using Hugging Face huggingface-cli to download models: llama model install script support for llama CLI and huggingface hub #14
add model setup for llama 3.x
address Initial vLLM setup fails due to missing HuggingFace permissions #37
address Docker run support for HF_TOKEN authentication using env var pass in #23

…ace input sizes

…tion

…seq_lengths and output_seq_lengths directly args to test_api_call_threaded_full_queue() to allow for varied isl and osl within batch

…e and prefill

…_client_cli.py

…d utils/batch_processor.py with configs in utils/prompt_configs.py and utils/prompt_generation.py for prompt generation

…t.py

…llm online benchmark script

…/prompt_client_online_benchmark.py

…e locust default test length to 10 minutes

…t_of logprobs

…ugging Face authorization and model download, add llama 3.3 70B instruct

…ile dir in persistent dir

… 3.2 11b vision instruct, llama 3.3 70b instruct

…setup

…issues for troubleshooting, add support for llama 3.2 1B / 3B

tstescoTT · 2025-01-14T05:10:29Z

I'll similarly fix the history and reopen this PR once #59 can be merged.

tstescoTT added 30 commits December 19, 2024 22:25

add print_prompts cli arg

7d68656

remove redundant stop token from vLLM example api calls

8d78d64

add capture_trace.py util to pre-prompt vllm server to capture all tr…

3108bc0

…ace input sizes

adding utils/startup_utils.py to refine handling of startup in automa…

ea3d75d

…tion

adding force_max_tokens as option to call_inference_api(), add input_…

cc1d17a

…seq_lengths and output_seq_lengths directly args to test_api_call_threaded_full_queue() to allow for varied isl and osl within batch

faster mock model prefill

059d513

make it not send stop tokens by default and speed up mock model decod…

48d17de

…e and prefill

adding token count verification for vllm open ai api server to prompt…

fead1aa

…_client_cli.py

add max-log-len to limit logging of prompts to avoid clutter in logs

5a80551

add InferenceServerContext to startup_utils.py, improve wait_for_healthy

d845f08

add all_responses to utils/prompt_client_cli.py not using globals

632ac83

adding new utils/prompt_client_cli.py using utils/prompt_client.py an…

f563e32

…d utils/batch_processor.py with configs in utils/prompt_configs.py and utils/prompt_generation.py for prompt generation

fix health endpoint

2467c74

add vllm_model to EnvironmentConfig instead of BatchConfig

af5e8dc

refactor utils/capture_traces.py with new prompt_client

60c7ab2

fix utils imports

10993a2

fix BatchConfig usage

20ccdf4

add benchmarking/online_benchmark_prompt_client.py using prompt_clien…

eab7e76

…t.py

add benchmarking/online_benchmark_prompt_client.py using prompt_clien…

90acdf6

…t.py

add benchmarking, evals, and tests dirs to Dockerfile

ec486ad

update patchfile and benchmarking README.md with commands

c58d7b3

update Docker IMAGE_VERSION to v0.0.3

fe4f96d

improve doc

f3d815a

update benchmark_serving.patch

8246a72

add tt_model_runner.py patch for best_of

765c4be

update benchmarking/benchmark_serving.patch

b93370d

use CACHE_ROOT for vllm_online_benchmark_results dir

5e07baa

adding timestamped online benchmark run result directory, rps=1 for v…

d0e0b0f

…llm online benchmark script

update benchmark output file naming convention

5db2523

rename benchmarking/online_benchmark_prompt_client.py to benchmarking…

5ab742c

…/prompt_client_online_benchmark.py

tstescoTT added 13 commits December 31, 2024 20:55

update sweeps context lengths

ffaabd6

model id as header not in table

4602ff3

add better formatting in benchmark_summary.py, update iso/osl sweeps

594b9a1

add better markdown formatting, add saving display .csv

2ce6fe7

update sweep isl/osl

6be324f

update sweep isl/osl

f558876

add metadata to markdown summary

b4260d3

add ignore_eos=True to locust requests to use min/max tokens, increas…

89958d9

…e locust default test length to 10 minutes

update for llama 3.1 70B v0 testing

126c588

adding evals changes from tstesco/llama-evals

aef6a94

adding TT_METAL_COMMIT_SHA_OR_TAG=v0.54.0-rc2

5ab1816

update README commit tags

0e5b67a

adding vllm benchmarking patch to stop sending unsupported params bes…

0c48a9f

…t_of logprobs

tstescoTT force-pushed the tstesco/benchmark-v0 branch from 26c8a6c to 0c48a9f Compare December 31, 2024 20:55

tstescoTT added 10 commits December 31, 2024 21:21

move vllm-tt-metal-llama3-70b/setup.sh -> setup.sh, add support for H…

471c90b

…ugging Face authorization and model download, add llama 3.3 70B instruct

add llama 3.2 refs

ec43450

WIP make setup.sh run from repo root, add fixed model impl dir, env f…

d17e46e

…ile dir in persistent dir

adding setup.sh support for multiple models, adding support for llama…

895ff8d

… 3.2 11b vision instruct, llama 3.3 70b instruct

update .env file location in documentation

49ee14f

remove MODEL_IMPL_ROOT_DIR and add note about MODEL_NAME

1675f54

move setup_tt_metal_cache into setup_weights to use load_env scope

d1bffe0

better logging and handling of {PERSISTENT_VOLUME}/model_weights dir …

97c90cf

…setup

adding error message when huggingface-cli download fails with common …

6df6c7c

…issues for troubleshooting, add support for llama 3.2 1B / 3B

update README for llama 3.1 70B v0 drop commits

6beaa66

tstescoTT force-pushed the tstesco/setup-script-llama32 branch from 3b0e8bc to 6beaa66 Compare December 31, 2024 21:21

tstescoTT force-pushed the tstesco/benchmark-v0 branch 2 times, most recently from 5d5b4ca to 4ae94dd Compare January 14, 2025 05:04

tstescoTT closed this Jan 14, 2025

tstescoTT deleted the tstesco/setup-script-llama32 branch January 31, 2025 21:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

setup.sh script multiple model support using HF download #61

setup.sh script multiple model support using HF download #61

tstescoTT commented Dec 20, 2024

tstescoTT commented Jan 14, 2025

setup.sh script multiple model support using HF download #61

setup.sh script multiple model support using HF download #61

Conversation

tstescoTT commented Dec 20, 2024

change log

tstescoTT commented Jan 14, 2025