Fix error after second start_chat() for StatefulLLMPipeline #1684
Conversation
sbalandi
requested review from
Wovchena and
ilya-lavrenov
and removed request for
Wovchena
| if (have_state) { | ||
| m_model_runner.reset_state(); | ||
| m_model_runner.get_tensor("attention_mask").set_shape({1, 0}); | ||
| } |
There was a problem hiding this comment.
I think the request to add a test is valid. You can take the idea from #1674 (comment)
There was a problem hiding this comment.
I think we should also cover a case when user call next start_chat w/o stop_chat for current one.
IMO, it should automatically stop current chat
There was a problem hiding this comment.
cases with system_massage, with double start_chat and with several chats one after another are added.
If i understand right tests work now for PA backend, should it be added run for SDPA too ?
There was a problem hiding this comment.
If i understand right tests work now for PA backend, should it be added run for SDPA too ?
they will be added as part of CVS-159925
For now, you can check locally that tests are passing on SPDA branch by changing default backend here
ilya-lavrenov
added
bug
|
does a current PR fix #1663 ? |
The reason looks like point which Vladimir mention above. If execution is interrupted and generation() does not complete correctly, this error will occur. I moved the resets to the beginning, this fixes similar behavior for phi-3. |
sbalandi
force-pushed
the
phi3
branch
2 times, most recently
from
77c2688 to
ac2474c
Compare
There are other pipelines that need that change. (VLM, at least.) Can you submit a ticket to address it or simply fix in the next PR |
added to VLM |
Ticket: CVS-161902