Time to consider on-device GPU accelerate?

[sorasoras]

option 1,
opencl backend by qualcomm
https://www.qualcomm.com/developer/blog/2024/11/introducing-new-opn-cl-gpu-backend-llama-cpp-for-qualcomm-adreno-gpu
https://github.com/CodeLinaro/llama.cpp
The backend has been tested with many premium devices powered by Snapdragon SOCs:

Laptops running Windows 11 with Snapdragon X Elite and Snapdragon X Plus chips
Android smartphones powered by Snapdragon 8 Gen 1, 2, 3, and the latest Snapdragon 8 Elite
it would work faster on qualcomm device which is the most used brand these day.

option 2

https://github.com/mlc-ai/mlc-llm

should work with most SOC

it would help a lot for running larger model like 7B or even 14B on android devices

[shubham0204]

Adding OpenCL support for devices with Adreno GPUs looks doable. Would you be interested in testing an unreleased APK that contains OpenCL acceleration? I do not have a physical device that has any one of the required CPUs from the Snapdragon 8 family. I can attach the APK in this thread once I have an APK ready for testing.

[sorasoras]

Adding OpenCL support for devices with Adreno GPUs looks doable. Would you be interested in testing an unreleased APK that contains OpenCL acceleration? I do not have a physical device that has any one of the required CPUs from the Snapdragon 8 family. I can attach the APK in this thread once I have an APK ready for testing.

Sure, I have 8G2 device, it should work.