Fixes for Llama 3.x support, adding image input tooling, tt-studio environment variable handling #83
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…vironment variable handling (#82) * Add benchmark summary support handling for vllm benchmark script, add documentation example * Rename client side scripts batch_size options to max_concurrent to indicate client side concurrent request limits * Stop-gap handling of MESH_DEVICE for Llama 3.x models * Add image input support for image-text-to-text models in client scripts and tools - Added support for image input in trace capturing - Added new parameters for image width and height - Implemented handling of both text-only and image+text trace captures * Modified setup script improvements: - Improved environment variable handling and persistence storage integration - Added IMPL_ID field (set to "tt-metal" for all current models) - Introduced MODEL_VERSION and MODEL_ID variables for better versioning - Enhanced HF token validation with character count check * Fixed the vLLM model registration logic: - Added missing ModelRegistry.register_model call for TTLlamaForCausalLM for legacy implementation models * Updated benchmark path handling to use $HOME environment variable instead of hardcoded /home/user path * Added support for a new model "DeepSeek-R1-Distill-Llama-70B" in the model setup configurations
ppetrovicTT
approved these changes
Jan 31, 2025
| echo " llama-3-70b" | ||
| echo " llama-3-8b-instruct" | ||
| echo " llama-3-8b" | ||
| echo " DeepSeek-R1-Distill-Llama-70B" |
There was a problem hiding this comment.
i'm up for this, as it will align id with metal's expectations! (model_config.py, L137)
but we need we need to change the names in setup_model_environment as well then.
i've done that locally, will push soon. you can the usage chunk if you wish?
| @@ -74,6 +75,7 @@ get_hf_env_vars() { | |||
| echo "HF_TOKEN environment variable is not set. Please set it before running the script." | |||
| read -r -s -p "Enter your HF_TOKEN: " input_hf_token | |||
| echo | |||
| echo "entered HF_TOKEN contains: ${#input_hf_token} characters, expected 37." | |||
There was a problem hiding this comment.
we can ping HF API to check if the token is valid
i have that in my change, will follow up!
There was a problem hiding this comment.
Nice, can you open a separate PR on that or share the code you're using? I had a TODO to also check repo access for the target model and message to user on 404.
* fix casing * clean up HF setup venv before repacking
* update refs to /home/container_app_user/ for container home usage
* update documentation
|
Moved this PR to #88 to use the proposed RC git workflow. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
change log