How do I run Llama(...) in a non-blocking way?

[madprops]

When loading a model the UI will freeze until the process is done. I tried using a thread but I don't know how to interrupt/cancel a load. How do you do this?

[woheller69]

see e.g. here https://github.com/woheller69/LLAMA_TK_CHAT/blob/main/LLAMA_TK_GUI.py

[madprops]

Hi, thanks for the reply. So if I'm not mistaken what you do is raise an exception right? self.inference_thread.raise_exception()

[madprops]

Copilot doesn't really like this approach, says it's unsafe and OS dependent
ctypes.pythonapi.PyThreadState_SetAsyncExc(ctypes.c_long(thread_id), ctypes.py_object(SystemExit))

[woheller69]

I don't like it either...

[woheller69]

inference is running in a separate thread, so it does not block the UI. And when I want to stop inference I raise an exception.

Not ideal, but obviously there is no other way at the moment
See #733