Optimize multiple concurrent LLM calls

[NicolaZomer]

Hi everyone!

I would like to know if there is an efficient way to optimize multiple LLM calls. Let's say I need to make 10 independent requests to the same LLM, instantiated with llama-cpp-python. Is there a more efficient way then doing it sequentially? Can we manage the workload, or parallelize it, or do you any other strategies that might help?

Thank you!

[Felipe-Amdocs]

I guess you might be thinking on batch. As far as I know, this PR will bring it to the project:
#951

[NicolaZomer]

Thank you! And yes, this is what I was looking for. Hope to see it done soon!