Skip to content

Is this the correct LLM inference setup #1491

Answered by CISC
taoofstefan asked this question in Q&A
Discussion options

You must be logged in to vote

You won't get any function call responses from this model, mainly because it does not have a function calling capable chat template (and you're using chat_format="llama-2" anyway, which will ignore the chat template), if you want to use the tools parameter you have to use a model with the correct chat template (like this one) or a function calling chat format that the model supports.

Additionally you are doing a couple of things wrong:

  • it should be tools=tools, you are passing a single function instead, that won't work
  • assistant_response is set wrong it should be set to result["choices"][0]["text"]
  • tool_choice="auto" actually does nothing with regular chat templates (which might be what …

Replies: 1 comment 1 reply

Comment options

You must be logged in to vote
1 reply
@taoofstefan
Comment options

Answer selected by taoofstefan
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
Q&A
Labels
None yet
2 participants