Performance on Steam deck

[FrostyMisa]

First I want to thanks all who made Koboldcpp ❤️. I do some test with the latest version with Vulcan support and... It's great! Generation time is not much faster (but still great), but the prompt processing at the beginning is now lightning fast!

5 times faster with Vulcan (offload all layers to GPU). With this setup it doesn't use CPU, so it's keep the deck more on cooler temperatures, so fan doesn't scream.

4 times faster with Vulcan (0 layers offload to GPU). With this setup it use CPU, but only 1/3 compared to previous version of Koboldcpp with OpenBLAS.

If I know how to open the Koboldcpp console, I will write more accurate data here, but I don't know how and is running the process in background even I select bring Koboldcpp in foreground.

[LostRuins]

Terminal can be viewed by launching koboldcpp via command line from a shell (like bash)

[FrostyMisa]

Thanks for tip. So here is the results.
7B_Q5_K_M GGUF model:
Vulcan, offload all layers.

ContextLimit: 365/2048, Processing:3.68s (29.4ms/T), Generation:45.86s (191.1ms/T), Total:49.54s (206.4ms/T = 4.84T/s)
ContextLimit: 596/2048, Processing:3.46s (203.2ms/T), Generation:46.84s (195.2ms/T), Total:50.29s (209.6ms/T = 4.77T/s)
ContextLimit: 822/2048, Processing:3.44s (202.1ms/T), Generation:47.76s (199.0ms/T), Total:51.20s (213.3ms/T = 4.69T/s)
ContextLimit: 1065/2048, Processing:3.38s (187.7ms/T), Generation:48.77s (203.2ms/T), Total:52.15s (217.3ms/T = 4.60T/s)

Vulcan, 0 layers offloaded.

ContextLimit: 365/2048, Processing:5.96s (47.6ms/T), Generation:60.10s (250.4ms/T), Total:66.06s (275.2ms/T = 3.63T/s)
ContextLimit: 596/2048, Processing:2.86s (178.8ms/T), Generation:61.03s (254.3ms/T), Total:63.89s (266.2ms/T = 3.76T/s)
ContextLimit: 829/2048, Processing:2.93s (183.2ms/T), Generation:61.84s (257.7ms/T), Total:64.77s (269.9ms/T = 3.71T/s)
ContextLimit: 1076/2048, Processing:3.16s (186.0ms/T), Generation:62.67s (261.1ms/T), Total:65.83s (274.3ms/T = 3.65T/s)

CPU only.

ContextLimit: 365/2048, Processing:21.95s (175.6ms/T), Generation:60.38s (251.6ms/T), Total:82.33s (343.0ms/T = 2.92T/s)
ContextLimit: 621/2048, Processing:3.11s (182.7ms/T), Generation:61.29s (255.4ms/T), Total:64.39s (268.3ms/T = 3.73T/s)
ContextLimit: 846/2048, Processing:3.11s (183.0ms/T), Generation:62.22s (259.2ms/T), Total:65.33s (272.2ms/T = 3.67T/s)
ContextLimit: 1085/2048, Processing:3.35s (185.9ms/T), Generation:62.97s (262.4ms/T), Total:66.31s (276.3ms/T = 3.62T/s)

13B_Q5_K_M GGUF model:
Vulcan, 0 layers offloaded (cannot offload all, because of GPU memory limit).

ContextLimit: 269/2048, Processing:16.06s (122.6ms/T), Generation:62.17s (450.5ms/T), Total:78.23s (566.9ms/T = 1.76T/s)
ContextLimit: 404/2048, Processing:6.23s (311.5ms/T), Generation:53.35s (460.0ms/T), Total:59.58s (513.7ms/T = 1.95T/s)
ContextLimit: 547/2048, Processing:6.28s (313.9ms/T), Generation:57.62s (464.7ms/T), Total:63.90s (515.3ms/T = 1.94T/s)
ContextLimit: 677/2048, Processing:6.64s (316.4ms/T), Generation:52.00s (472.7ms/T), Total:58.65s (533.2ms/T = 1.88T/s)

The 7B model is fast, when all layers offloaded to GpU, doesn't use CPU and keep the fan almost quiet. Vulcan and 0 layers offloaded use CPU little bit, fan is noticeable. Only CPU is have very noticeable fan noise.