Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Switch to TRT-LLM LLM (High Level) API over trtllm-build CLI workflow #87

Merged
merged 14 commits into from
Dec 9, 2024

Conversation

rmccorm4
Copy link
Collaborator

@rmccorm4 rmccorm4 commented Oct 25, 2024

Changes:

  • Switch to TRT-LLM LLM (High Level) API over trt-llm build workflow
    • Remove unnecessary helper scripts
    • Expands model support
    • Simplifies engine generation and model download process
  • Refer to framework model support sections in README
  • Add some better checks and error messages around --source arg
  • Add 'local:' source prefix for parity
  • Bump version to 0.1.0dev so it's easier to have 'patch' versions such as 0.1.1, 0.1.2, etc.
  • Disable genai-perf (triton profile) based tests due to bug in genai-perf r24.09:

Note:

  • Disable genai-perf (triton profile) based tests due to bug in genai-perf r24.09:
$ triton profile -m gpt2
triton - INFO - Running: 'genai-perf profile -m gpt2'
2024-10-25 12:40 [INFO] genai_perf.parser:83 - Profiling these models: gpt2
Traceback (most recent call last):
  File "/home/rmccormick/.local/lib/python3.10/site-packages/genai_perf/main.py", line 198, in run
    generate_inputs(config_options)
  File "/home/rmccormick/.local/lib/python3.10/site-packages/genai_perf/main.py", line 125, in generate_inputs
    inputs.create_inputs()
  File "/home/rmccormick/.local/lib/python3.10/site-packages/genai_perf/inputs/inputs.py", line 57, in create_inputs
    json_in_pa_format = self._convert_generic_json_to_output_format(
  File "/home/rmccormick/.local/lib/python3.10/site-packages/genai_perf/inputs/inputs.py", line 77, in _convert_generic_json_to_output_format
    return converter.convert(generic_dataset, self.config)
  File "/home/rmccormick/.local/lib/python3.10/site-packages/genai_perf/inputs/converters/tensorrtllm_converter.py", line 52, in convert
    pa_json = self._populate_trtllm_output_json(
  File "/home/rmccormick/.local/lib/python3.10/site-packages/genai_perf/inputs/converters/tensorrtllm_converter.py", line 96, in _populate_trtllm_output_json
    pa_json = self._add_optional_tags_to_trtllm_json(
  File "/home/rmccormick/.local/lib/python3.10/site-packages/genai_perf/inputs/converters/tensorrtllm_converter.py", line 125, in _add_optional_tags_to_trtllm_json
    if config.add_stream:
AttributeError: 'str' object has no attribute 'add_stream'

…el support sections in README, add some better checks and error messages around --source arg, add 'local:' source prefix for parity, bump version to 0.1.0dev so it's easier to have 'patch' versions
… ensure server is cleaned properly between tests
…dd support for MODEL_SOURCE in local testing
Copy link
Contributor

@KrishnanPrash KrishnanPrash left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall change LGTM. Will clarify order of operations and upcoming releases offline.

@rmccorm4 rmccorm4 requested a review from krishung5 December 7, 2024 01:01
@rmccorm4 rmccorm4 merged commit 2e055fa into main Dec 9, 2024
4 checks passed
@KrishnanPrash KrishnanPrash deleted the rmccormick-trtllm-hlapi branch December 16, 2024 18:15
@KrishnanPrash KrishnanPrash restored the rmccormick-trtllm-hlapi branch December 16, 2024 18:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants