fix: Fixing CI for TRTLLM HLAPI

[KrishnanPrash]

This PR fixes the CI failures that have been observed when using the TRTLLM HL API. The fixes include:

• Mainly, for the ScopedTritonServer, it relies on using subprocess and it's relevant functions instead of using psutils and having to manually keep track of the process id.
• Additionally, resolving an issue that was observed when stopping triton instance in CI. Here is the error messaged being displayed:

Received signal=15, frame=<frame at 0x79d701106f80, file '/mnt/triton_cli/src/triton_cli/server/server_local.py', line 148, code logs>. Shutting down...
reentrant call inside <_io.BufferedReader name=3>
triton - ERROR - Failed to start server, unknown error.

The root cause for this was after process.terminate() was called for the underlying tritonserver process, it is followed by a process.communicate(timeout=...) which triggers a file I/O operation. Hence, to resolve this we replace process.communicate() with process.wait() to give the process time to clean up with making an I/O call.

[rmccorm4]

LGTM if pipeline passes. Please follow-up on the other PR from my branch into main that this is merging into afterwards

[krishung5]

LGTM. qq - does this also fix the GPU OOM issue or is it a separate problem?

[rmccorm4]

LGTM. qq - does this also fix the GPU OOM issue or is it a separate problem?

My PR which launches the HL API conversion as a subprocess seems to definitively fix the OOM issue and guarantee memory cleanup. Results were muddied for a bit because it turned out the gitlab mirror got out of sync and was running old code each time.

[KrishnanPrash]

@krishung5 That fix was made as a part of Ryan's branch [Link]. IIUC, the solution was building the trtllm engine as a separate child process. Here is the relevant code snippet:

# Run TRT-LLM build in a separate process to make sure it definitely
# cleans up any GPU memory used when done.
p = multiprocessing.Process(
         target=self.__build_trtllm_engine, args=(huggingface_id, engines_path)
)
p.start()
p.join()