LKL latency/throughput

[speedingdaemon]

I was reading this awesome paper written by Jerry : https://netdevconf.org/1.2/papers/jerry_chu.pdf
It mentions that the performance of lkl in user space did not beat that of host OS. Is that true still?
Does anyone know/has done some more performance benchmarking to find out more such latency /throughput numbers?
Curious to know what to expect with a TCP proxy service based off of lkl.

[liuyuan10]

I don't think there is much change in performance since then. If the number of TCP connections are small, then the performance of LKL is OK (still a bit slower than host). But if the number is big (I'm talking about hundreds), due to the single threading nature of it, there is a big gap vs host.

…

On Tue, Jul 11, 2017 at 6:35 AM, speedingdaemon ***@***.***> wrote: I was reading this awesome paper written by Jerry : https://netdevconf.org/1.2/papers/jerry_chu.pdf It mentions that the performance of lkl in user space did not beat that of host OS. Is that true still? Does anyone know/has done some more performance benchmarking to find out more such latency /throughput numbers? Curious to know what to expect with a TCP proxy service based off of lkl. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#357>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEVfUkVmvFhsManibfIlIDE-XgEy0VRvks5sM3ongaJpZM4OUPoQ> .

[speedingdaemon]

How would you guys compare performance of LKL vs MTCP?

[linhua55]

@speedingdaemon
Thanks to @thehajime , I integrated LKL into rinetd proxy in order to use BBR in OpenVZ VPS. You can refer to
https://github.com/linhua55/lkl_study
https://github.com/linhua55/rinetd

It uses raw socket backend, but modified some code. As OpenVZ’s venet0 network interface is a Cooked interface. Its raw packet has no MAC layer(14 bytes). It can’t use AF_PACKET/SOCK_RAW, can only use AF_PACKET/SOCK_DGRAM.

But it has much CPU usage. Recently, I want to use TPACKET_V2 (packet mmap) to achieve Zero Copy to reduce some CPU.

[speedingdaemon]

@linhua55 @thehajime @tavip
I have an async server running on my blade.
Here is what I see:

1. Server running on host kernel as a user process (doing regular socket calls, but I did not set the incoming FD to NONBLOCKING) - 400 connections per second - did this experiment only because of the issue mentioned in Does LKL support non-blocking sockets? #360
2. Server running on host kernel as a user process (doing regular socket calls, but I set the incoming FD to NONBLOCKING) - 820 connections per second
3. Server running on host as a user process but is linked to LKL (doing LKL socket calls, but I did not set the incoming FD to NONBLOCKING, have another issue - Does LKL support non-blocking sockets? #360 ) - 369 connections per second (within 90% of host for experiment number 1 above)
4. Server running on host as a user process, but is using mTCP stack instead of LKL/Host - 40k connections per second

Do these numbers sound reasonable? I would have thought that LKL-based server should have been much faster than it really turned out to be... If the throughput found above in experiment number 3 is true, then why would anyone use LKL for HFT applications?

I am really really hoping that there is some init stuff that I am missing while using LKL. Wish someone could help.

[tavip]

I expect LKL to perform a bit worse that host, so the host / LKL ratio seems right.

I am not sure why do you think that LKL would be useful for HFT applications.

[speedingdaemon]

Oh. Sorry. My bad. I thought that I read somewhere.
@tavip
How about applications that need high performance/throughput?
What is the use-case for LKL?
I would have assumed that user-mode networking stack should have had much higher throughput...

[speedingdaemon]

Does this, the output of perf, look fine to folks?

 2.04%  myserver  [kernel.kallsyms]         [k] packet_recvmsg
 1.90%  myserver  [kernel.kallsyms]         [k] _raw_spin_lock_irqsave
 1.85%  myserver  [kernel.kallsyms]         [k] copy_user_enhanced_fast_string
 1.63%  myserver  [kernel.kallsyms]         [k] skb_release_head_state
 1.47%  myserver  liblkl.so                 [.] memset
 1.44%  myserver  liblkl.so                 [.] raid6_int8_xor_syndrome
 1.41%  myserver  liblkl.so                 [.] raid6_int8_gen_syndrome
 1.39%  myserver  liblkl.so                 [.] raid6_int4_xor_syndrome
 1.35%  myserver  liblkl.so                 [.] raid6_int2_xor_syndrome
 1.34%  myserver  liblkl.so                 [.] raid6_int4_gen_syndrome
 1.34%  myserver  liblkl.so                 [.] raid6_int2_gen_syndrome
 1.24%  myserver  liblkl.so                 [.] raid6_int1_xor_syndrome
 1.08%  myserver  liblkl.so                 [.] raid6_int1_gen_syndrome
 1.06%  myserver  liblkl.so                 [.] do_csum
 1.05%  myserver  liblkl.so                 [.] arch_local_irq_restore

Why do I see references to kernel.kallsyms at the top 4? I thought that with the use of LKL, I shouldn't have seen kernel.kallsyms stuff at the top.

Also, what is all this raid stuff? Is it for the 8 lkl_netdev_raw_create() calls I made in myserver?
Any way I can get rid of these things at the top to increase throughput?

[thehajime]

Why do I see references to kernel.kallsyms at the top 4? I thought that with the use of LKL, I shouldn't have seen kernel.kallsyms stuff at the top.

the raw socket (AF_PACKET socket) backend uses system calls (of host kernel) where you may frequently uses those functions when sending/receiving packets. it can be amortized by several techniques (bulk processing, segmentation offloading e.g.) but need modification to the backend itself.

Also, what is all this raid stuff? Is it for the 8 lkl_netdev_raw_create() calls I made in myserver?
Any way I can get rid of these things at the top to increase throughput?

See #301

This is due to the benchmark code of btrfs (I guess) in the boot phase so you won't see better performance/throughput even if you disabled the feature.

[speedingdaemon]

@thehajime

I don't mind modifying the backend. Can you please point me to the places that need to be modified?

Also, I saw code in lkl_netdev_tap_init() that takes an input, offload, to offload certain functionality.
Why don't we have something similar for lkl_netdev_raw_create()?

[dakami]

I spent some time poking at this. lt might be faster to use a pthread spinlock instead of a mutex, but there's a dependency right now on having a recursive mutex which I haven't been able to entangle.

At the extreme, it may be possible to run lkl on top of Xenomai, which does in fact support recursive mutexes.

It would be nice if this particular design decision was optional, as there are quite a few threading libraries to explore that don't implement this feature.

[tavip]

I have been working on moving the threading stuff out to host ops which removes a lot of dependencies (including mutexes), see:

https://github.com/tavip/linux/commits/the-expanse

The main problem I am still facing are the latency optimizations (direct irqs and syscalls), for which I could not find a model yet that allows us to move it to host ops.

[dakami]

I tried removing the recursive flag from master and kernel booting just froze. So something's certainly using it. I did see you were trying to let the host declare a jump instruction.

Where do the latency optimizations live?

In my mental model, syscalls just become function calls, and IRQs are just blocking tasks that interrupted the host in a way we caught (userfaultfd, signal) or nonblocking tasks in a list that can be serviced at the host's discretion (perhaps between functions via -finstrument-functions). But I haven't quite grokked what needs to be locked at this layer.