How can I invoke a tool and continue chatting without calling addToolResult using useChat?

[MarkLyck]

I have a few client tools I want to invoke, which renders a component with a form, action or button the user needs to interact with. The tools work fine if the user completes the component and I run the addToolResult function.

However, if the user decides not to complete the task, e.g. if the AI ran the wrong tool, or they just changed their mind on what task they want to do, if they try to chat with the AI without completing the tool it throws an error:

Error [AI_MessageConversionError]: ToolInvocation must have a result: {"state":"call","toolCallId":"call_8vYD2hg9BKKySEj6yso07EfD","toolName":"myToolName","args":{}}

How can I allow tools to be invoked without requiring a result?

I have tried adding a result either in the tool execute function, or when the component renders. But in both cases, the AI starts blabbering in the chat claiming the task is done, which I don't want. The AI shouldn't respond unless the user completes the tool.

I'm also okay if there's a solution that makes the AI not respond at all. As long as the chat can be continued without the tool having submitted a result yet.

It's a bad user experience that they must complete the tool before being able to interact with the chat again.

[brennanmceachran]

Seems like the tools job is to actually display the form, not submit the form. I'd swap the tools purpose to be something like display{{Contact}}Form and describe it's purpose is to display the form for the user.

Then, you can immediately return a success ToolInvocation.

If you need the chat thread to be aware of the data the form is submitting, you can add the form data to the thread with append when the user submits the displayed form.

[MarkLyck]

@brennanmceachran Thanks for the response, that is indeed the case and I did try this. However the AI will still respond to the "success", often with incorrect information.

e.g. a user will ask the AI to do a task, the AI will display the component for the user to complete the task, then before the user has used the form, the AI will mistakenly claim that the task was completed successfully.

Ideally I don't want the AI to say or respond with anything when a tool is invoked.

[brennanmceachran]

I did a quick test to show how I was thinking about it.
Code:

Results:

Basically I'd consider the return from execute to almost contain a prompt for how the llm should interpret the result. I view the toolname, description, all params, and all responses to effectively be prompts / additional instructions.

[MarkLyck]

That works if the user is asking to "show x form".

In my case, I want the users to be able to ask the AI for a task e.g. "Enable x feature". Some tools might require some user input to do that in my case. So the AI should call a tool that just displays a component inside the chat that lets the user do the task themselves.

But when phrased in this way, I find that the AI often will just claim that the task was complete, and say .e.g "I've enabled x feature for you, is there anything else I can help with?" even though the feature was not enabled.

This sometimes still happen even if the tool is named displayXForm and has display X form in the description, and the return statement is "X form displayed."

I could try adding more instructions in the initial prompt to not claim a task for a tool with "display" in the name was completed. But it still feels iffy... It's not very deterministic, it might still screw it up sometimes.

What is the reasoning behind not allowing further chatting if a tool has been called but doesn't have a result yet? Or forcing a tool to send a result?

Surely I can't be the only person to use useChat who wants to show a tool that should be optional for the user to complete before continuing the chat. The AI could call the wrong tool for example, and the user should be able to clarify their question better.

[brennanmceachran]

What is the reasoning behind not allowing further chatting if a tool has been called but doesn't have a result yet? Or forcing a tool to send a result?

It's a strict limitation in the LLMs. OpenAI, for example, will error directly if a result is not returned before generating the next message. Tool calls are sort of pre-requisites functions for the ai to respond.

You could get around this by filtering/editing the message history prior to sending to LLMs. But that's not a great way around the problem, imo.

I could try adding more instructions in the initial prompt to not claim a task for a tool with "display" in the name was completed. But it still feels iffy... It's not very deterministic, it might still screw it up sometimes.
I'd definitely do this. LLMs just aren't deterministic. Informing the llm in the system prompt of the tools available to it has been shown with benchmarks to signifigantly improve it's ability to call the correct tool in the correct moment.

With the above, I'd still do something like below if youre having issues

featureXEnableConfirmationForm: tool({
            description: [
              'Displays an enable/disable UI to the user.',
              'The form will display details and the user will optionally enable/disable the feature on their own.',
            ].join('\n'),
            parameters: z.object({}),
            execute: async () => {
              return 'Feature X enable/disable form displayed -- user interaction required in confirmation form for further action.';
            },
          }),

[MarkLyck]

@brennanmceachran Thank you for the quick responses, that makes sense!