Android App Llama-v3-2-3B-Chat Quantized returns garbled text

[zhehui-chen]

Follow the guideline to build the ChatApp with Llama-v3-2-3B-Chat Quantized. The QNN version I used is 2.28.2.

I successfully run the ChatApp on my android device (OnePlus 13 with snapdragon 8Elite).

However, while chatting with the app, it always returns me with garbled text like the following.

Does anyone have any idea about this problem?

[franklyd]

I have faced a similar issue. I found the generation quality downgraded a lot, comparing to run it with onnx-runtime in 4 bit.

[gustavla]

Hi @zhehui-chen and @franklyd,

Sorry to hear you are seeing poor results through the app.

We know that the app has issues on some consumer devices (especially on Android 14 or earlier). There are two underlying features needed in the Android "metabuild". This is why in the app README we say it only works on Android 15. If you can provide us with exactly what devices, what Android version, and ideally the exact Android build (should be in the settings), that would be really helpful so that we can investigate further why it's not working. Especially if it's on Android 15, where we expect it to work. Thanks!

[franklyd]

Thanks! Actually, I was running genie-t2t-run on Android device (gen 3) directly.
The instruction for the LLM was to generate some json output, but I observe much poorer quality comparing to onnx q4, especially it cannot follow the instruction to output in json.