Unable to handle page fault with program redis

[easonycliu]

I first tested UKL with the program hello and every thing worked well. Then I tried to test with redis but failed because there is a page fault after launching the redis.

[   10.860705] In PID 1 and in_user is 1
[   10.860705] Going to create UKL here.
[   10.861700] Run /init as init process
[   10.862389] BUG: unable to handle page fault for address: ffffffff5277897c
[   10.863212] #PF: supervisor read access in user mode
[   10.863910] #PF: error_code(0x0000) - not-present page
[   10.864621] IDT: 0xfffffe0000000000 (limit=0xfff) GDT: 0xfffffe0000001000 (limit=0x7f)
[   10.865651] LDTR: NULL
[   10.865973] TR: 0x40 -- base=0xfffffe0000003000 limit=0x4087
[   10.866719] PGD 2a11067 P4D 2a11067 PUD 0 
[   10.867435] Oops: 0000 [#1] SMP
[   10.867864] CPU: 0 PID: 1 Comm: UKL Not tainted 5.14.0-rc7 #4
[   10.868519] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
[   10.869890] RIP: 00c3:ukl___strlen_avx2+0x1d/0x180
[   10.870438] Code: Unable to access opcode bytes at RIP 0xffffffff81e46393.
[   10.871252] RSP: 0018:00007fff1ac690e8 EFLAGS: 00010287
[   10.871824] RAX: 000000000000097c RBX: ffffffff5277897c RCX: 0000000000000000
[   10.872740] RDX: ffffffff5277897c RSI: ffffffff8285cd50 RDI: ffffffff5277897c
[   10.873654] RBP: ffffffff52778973 R08: 00007fff1ac69148 R09: 0000000000000030
[   10.874545] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff83317440
[   10.875394] R13: ffffffff8285cd74 R14: 0000000083280e41 R15: 0000000000000000
[   10.876328] FS:  0000000002200000 GS:  0000000000000000
[   10.876905] CR2: ffffffff5277897c
[   10.877440] ---[ end trace f74d7d8c4bb605ed ]---
[   10.878116] RIP: 00c3:ukl___strlen_avx2+0x1d/0x180
[   10.878667] RSP: 0018:00007fff1ac690e8 EFLAGS: 00010287
[   10.879374] RAX: 000000000000097c RBX: ffffffff5277897c RCX: 0000000000000000
[   10.880211] RDX: ffffffff5277897c RSI: ffffffff8285cd50 RDI: ffffffff5277897c
[   10.881154] RBP: ffffffff52778973 R08: 00007fff1ac69148 R09: 0000000000000030
[   10.882014] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff83317440
[   10.882921] R13: ffffffff8285cd74 R14: 0000000083280e41 R15: 0000000000000000
[   10.883719] FS:  0000000002200000(0000) GS:ffff88813bc00000(0000) knlGS:0000000000000000
[   10.884597] CS:  00c3 DS: 0000 ES: 0000 CR0: 0000000080050033
[   10.885389] CR2: ffffffff5277897c CR3: 0000000002a0f001 CR4: 0000000000070ef0
[   10.886287] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   10.887250] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   10.888252] UKL exiting

My gcc version is 9.4.0 and I added -no-pie when compiling redis.

[rwmjones]

@khers

[khers]

What commit (and from which repo) did you use to create this kernel? gcc-9.4 is a little old, what distro did you use to build this?

I have tested on Debian 11 & 12 and Ubuntu 22.04 but have not tested on older.

[easonycliu]

$ git log
commit 1ce534f22c0894c6d8607368faf3a945d9a7dcf6 (HEAD -> main, origin/main, origin/HEAD)
Author: Eric B. Munson <[email protected]>
Date:   Fri May 5 16:26:08 2023 +0000

    Update saveconfig enable fanotify
    
    Glibc-bypass complains with this system call disabled. It costs nothing
    to enable it so turn it on and avoid the build failures.
    
    Signed-off-by: Eric B. Munson <[email protected]>

commit 1d5722db203654cba4e5ba463bc81ac3917edd3d
Author: Eric B. Munson <[email protected]>
Date:   Fri May 5 16:06:41 2023 +0000

    Update glibc to 2.37
    
    We had to add a number of syscall overrides to our ukl specialization.

Do I need to fetch the newest commit?

Currently I build it on Ubuntu 20.04 with gcc-9. I can switch to Ubuntu 22.04 and there are gcc-10, gcc-11 and gcc-12 to choose from, which one do you recommend to use?

[khers]

You are free to try Ubuntu 20.04, I just have not but you will need the latest commits for ukl and for linux.

[easonycliu]

I have updated ukl and linux and now I can see the redis logo but it still triggers a page fault. Good news is I can see the call trace now.

+ /UKL
71:C 20 Oct 2023 05:27:06.849 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
71:C 20 Oct 2023 05:27:06.850 # Redis version=6.2.2, bits=64, commit=71a07850, modified=0, pid=71, just started
71:C 20 Oct 2023 05:27:06.851 # Configuration loaded
71:M 20 Oct 2023 05:27:06.852 * Increased maximum number of open files to 10032 (it was originally set to 1024).
71:M 20 Oct 2023 05:27:06.853 * monotonic clock: POSIX clock_gettime
                _._                                                  
           _.-``__ ''-._                                             
      _.-``    `.  `_.  ''-._           Redis 6.2.2 (71a07850/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._                                  
 (    '      ,       .-`  | `,    )     Running in standalone mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
 |    `-._   `._    /     _.-'    |     PID: 71
  `-._    `-._  `-./  _.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |           https://redis.io       
  `-._    `-._`-.__.-'_.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |                                  
  `-._    `-._`-.__.-'_.-'    _.-'                                   
      `-._    `-.__.-'    _.-'                                       
          `-._        _.-'                                           
              `-.__.-'                                               

71:M 20 Oct 2023 05:27:06.864 # Server initialized
71:M 20 Oct 2023 05:27:06.864 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fi.
71:M 20 Oct 2023 05:27:06.867 * Ready to accept connections
[    7.009312] Setting bypass limit to 10
[    7.009612] BUG: scheduling while atomic: UKL/71/0x00000002
[    7.009614] CPU: 0 PID: 71 Comm: UKL Tainted: G S                 6.3.0+ #1
[    7.009616] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
[    7.009617] Call Trace:
[    7.009618]  <TASK>
[    7.009619]  dump_stack_lvl+0x33/0x50
[    7.009625]  __schedule_bug+0x44/0x50
[    7.009627]  __schedule+0x6c5/0x800
[    7.009629]  schedule+0x52/0xa0
[    7.009631]  exit_to_user_mode_prepare+0xa7/0x110
[    7.009633]  irqentry_exit_to_user_mode+0x5/0x10
[    7.009635]  asm_sysvec_apic_timer_interrupt+0x16/0x20
[    7.009637] RIP: 00c3:console_flush_all+0x1da/0x3d0
[    7.009638] Code: Unable to access opcode bytes at 0xffffffff810c92b0.
[    7.009639] RSP: 0018:00007fff2609b620 EFLAGS: 00000246
[    7.009640] RAX: 0000000000000000 RBX: 00007fff2609b6a7 RCX: 0000000000000000
[    7.009641] RDX: 0000000000000001 RSI: 0000000000000046 RDI: ffffffff83c2e9e0
[    7.009642] RBP: 00007fff2609b6a8 R08: 3031206f74207469 R09: 6d696c2073736170
[    7.009642] R10: 6c20737361707962 R11: 20676e6974746553 R12: 0000000000000000
[    7.009643] R13: ffffffff831cdb40 R14: 0000000000000000 R15: 0000000000000001
[    7.009644]  </TASK>
[    7.018558] BUG: unable to handle page fault for address: 00007fff26099d98
[    7.018561] #PF: supervisor write access in user mode
[    7.018562] #PF: error_code(0x0002) - not-present page
[    7.018569] IDT: 0xfffffe0000000000 (limit=0xfff) GDT: 0xfffffe0000001000 (limit=0x7f)
[    7.018574] LDTR: NULL
[    7.018577] TR: 0x40 -- base=0xfffffe0000003000 limit=0x4087
[    7.018579] PGD 101d76067 P4D 101d76067 PUD 101d6e067 PMD 101d84067 PTE 0
[    7.018581] Oops: 0002 [#1] PREEMPT SMP
[    7.018584] CPU: 0 PID: 71 Comm: UKL Tainted: G S      W          6.3.0+ #1
[    7.018585] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
[    7.018586] RIP: 00c3:ukl___vfprintf_internal+0x28/0x3110
[    7.018591] Code: Unable to access opcode bytes at 0xffffffff8212dcae.
[    7.018591] RSP: 0018:00007fff26099da0 EFLAGS: 00010206
[    7.018592] RAX: ffffffffffe00070 RBX: 00007fff2609a3d0 RCX: 0000000000000002
[    7.018593] RDX: 00007fff2609a458 RSI: ffffffff82d94bba RDI: 00007fff2609a2e0
[    7.018593] RBP: 00007fff2609a2d0 R08: 0000000000000000 R09: 00000000ffffffff
[    7.018594] R10: 0000000000000001 R11: 2ce33e6c02ce33e7 R12: 00007fff2609a2e0
[    7.018594] R13: ffffffff82d94bba R14: 00007fff2609a458 R15: 0000000000000002
[    7.018595] FS:  0000000000e00000 GS:  0000000000000000
[    7.018596] CR2: 00007fff26099d98
[    7.028300] ---[ end trace 0000000000000000 ]---
[    7.028302] RIP: 00c3:ukl___vfprintf_internal+0x28/0x3110
[    7.028306] RSP: 0018:00007fff26099da0 EFLAGS: 00010206
[    7.028307] RAX: ffffffffffe00070 RBX: 00007fff2609a3d0 RCX: 0000000000000002
[    7.028308] RDX: 00007fff2609a458 RSI: ffffffff82d94bba RDI: 00007fff2609a2e0
[    7.028308] RBP: 00007fff2609a2d0 R08: 0000000000000000 R09: 00000000ffffffff
[    7.028309] R10: 0000000000000001 R11: 2ce33e6c02ce33e7 R12: 00007fff2609a2e0
[    7.028310] R13: ffffffff82d94bba R14: 00007fff2609a458 R15: 0000000000000002
[    7.028312] FS:  0000000000e00000(0000) GS:ffff88813bc00000(0000) knlGS:0000000000000000
[    7.028313] CS:  00c3 DS: 0000 ES: 0000 CR0: 0000000080050033
[    7.028314] CR2: 00007fff26099d98 CR3: 0000000101d87006 CR4: 0000000000070ef0
[    7.028315] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[    7.028315] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[    7.028316] Kernel panic - not syncing: Fatal exception in interrupt
[    7.036543] Kernel Offset: disabled
[    7.036853] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---

[easonycliu]

Sometimes redis can boot successfully and keep working for a while.
Test cli

$ ./redis-cli -h 192.168.122.128
192.168.122.128:6379> ping
PONG
192.168.122.128:6379> set hello world
OK
192.168.122.128:6379> get hello
"world"
192.168.122.128:6379> exit

Test benchmark

$ ./redis-benchmark -h 192.168.122.128 -p 6379 -n 100000 -c 50 -t set
====== SET ======                                                   
  100000 requests completed in 2.79 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1
  host configuration "save": 3600 1 300 100 60 10000
  host configuration "appendonly": no
  multi-thread: no

Latency by percentile distribution:
0.000% <= 0.231 milliseconds (cumulative count 1)
50.000% <= 1.239 milliseconds (cumulative count 50705)
75.000% <= 1.631 milliseconds (cumulative count 75033)
87.500% <= 1.719 milliseconds (cumulative count 88424)
93.750% <= 1.783 milliseconds (cumulative count 93962)
96.875% <= 2.079 milliseconds (cumulative count 96969)
98.438% <= 2.431 milliseconds (cumulative count 98445)
99.219% <= 2.559 milliseconds (cumulative count 99234)
99.609% <= 2.671 milliseconds (cumulative count 99620)
99.805% <= 2.735 milliseconds (cumulative count 99805)
99.902% <= 2.823 milliseconds (cumulative count 99905)
99.951% <= 3.263 milliseconds (cumulative count 99952)
99.976% <= 3.799 milliseconds (cumulative count 99976)
99.988% <= 4.263 milliseconds (cumulative count 99988)
99.994% <= 4.487 milliseconds (cumulative count 99994)
99.997% <= 4.927 milliseconds (cumulative count 99997)
99.998% <= 5.111 milliseconds (cumulative count 99999)
99.999% <= 5.159 milliseconds (cumulative count 100000)
100.000% <= 5.159 milliseconds (cumulative count 100000)

Cumulative distribution of latencies:
0.000% <= 0.103 milliseconds (cumulative count 0)
0.002% <= 0.303 milliseconds (cumulative count 2)
0.011% <= 0.407 milliseconds (cumulative count 11)
0.020% <= 0.503 milliseconds (cumulative count 20)
0.040% <= 0.607 milliseconds (cumulative count 40)
0.059% <= 0.703 milliseconds (cumulative count 59)
0.077% <= 0.807 milliseconds (cumulative count 77)
0.104% <= 0.903 milliseconds (cumulative count 104)
0.169% <= 1.007 milliseconds (cumulative count 169)
1.647% <= 1.103 milliseconds (cumulative count 1647)
34.259% <= 1.207 milliseconds (cumulative count 34259)
61.481% <= 1.303 milliseconds (cumulative count 61481)
67.891% <= 1.407 milliseconds (cumulative count 67891)
70.352% <= 1.503 milliseconds (cumulative count 70352)
72.423% <= 1.607 milliseconds (cumulative count 72423)
84.284% <= 1.703 milliseconds (cumulative count 84284)
94.567% <= 1.807 milliseconds (cumulative count 94567)
95.885% <= 1.903 milliseconds (cumulative count 95885)
96.475% <= 2.007 milliseconds (cumulative count 96475)
97.122% <= 2.103 milliseconds (cumulative count 97122)
99.936% <= 3.103 milliseconds (cumulative count 99936)
99.985% <= 4.103 milliseconds (cumulative count 99985)
99.998% <= 5.103 milliseconds (cumulative count 99998)
100.000% <= 6.103 milliseconds (cumulative count 100000)

Summary:
  throughput summary: 35880.88 requests per second
  latency summary (msec):
          avg       min       p50       p95       p99       max
        1.388     0.224     1.239     1.831     2.511     5.159

But minutes later, redis occurred page faults again.

71:M 20 Oct 2023 05:38:23.194 # Background saving terminated by signal 9
71:M 20 Oct 2023 05:38:29.017 * 10000 changes in 60 seconds. Saving...
71:M 20 Oct 2023 05:38:29.020 * Background saving started by pid 76
*** stack smashing detected ***: terminated
[   73.166744] BUG: unable to handle page fault for address: 0000000000000003
[   73.167501] #PF: supervisor instruction fetch in user mode
[   73.168012] #PF: error_code(0x0010) - not-present page
[   73.168509] IDT: 0xfffffe0000000000 (limit=0xfff) GDT: 0xfffffe0000001000 (limit=0x7f)
[   73.169227] LDTR: NULL
[   73.169455] TR: 0x40 -- base=0xfffffe0000003000 limit=0x4087
[   73.169966] PGD 101df5067 P4D 101df5067 PUD 101ddd067 PMD 0 
[   73.170573] Oops: 0010 [#2] PREEMPT SMP
[   73.170991] CPU: 0 PID: 76 Comm: UKL Tainted: G S    D            6.3.0+ #1
[   73.171705] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
[   73.172872] RIP: 00c3:0x3
[   73.173204] Code: Unable to access opcode bytes at 0xffffffffffffffd9.
[   73.173750] RSP: 0018:00007ffd58fabb08 EFLAGS: 000514d2
[   73.174255] RAX: 0000000000000000 RBX: 0000000000a00000 RCX: 0000000000000027
[   73.174917] RDX: 00007ffd58fabb00 RSI: 00007ffd58fabc30 RDI: 0000000000000006
[   73.175510] RBP: 00007ffd58fac510 R08: 0000000000000000 R09: 0000000000000000
[   73.176104] R10: 0000000000000008 R11: 0000000000000002 R12: 0000000000001000
[   73.176693] R13: 00007ffd58fac410 R14: 0000000000000020 R15: 00007fa770ee9000
[   73.177289] FS:  0000000000a00000 GS:  0000000000000000
[   73.177751] CR2: 0000000000000003
[   73.178039] ---[ end trace 0000000000000000 ]---
[   73.178397] RIP: 00c3:0x3
[   73.178608] RSP: 0018:00007ffd58fabb08 EFLAGS: 000514d2
[   73.179012] RAX: 0000000000000000 RBX: 0000000000a00000 RCX: 0000000000000027
[   73.179570] RDX: 00007ffd58fabb00 RSI: 00007ffd58fabc30 RDI: 0000000000000006
[   73.180119] RBP: 00007ffd58fac510 R08: 0000000000000000 R09: 0000000000000000
[   73.180668] R10: 0000000000000008 R11: 0000000000000002 R12: 0000000000001000
[   73.181216] R13: 00007ffd58fac410 R14: 0000000000000020 R15: 00007fa770ee9000
[   73.181774] FS:  0000000000a00000(0000) GS:ffff88813bc00000(0000) knlGS:0000000000000000
[   73.182426] CS:  00c3 DS: 0000 ES: 0000 CR0: 0000000080050033
[   73.182931] CR2: 0000000000000003 CR3: 0000000101dd4004 CR4: 0000000000070ef0
[   73.183531] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   73.184181] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   73.184812] note: UKL[76] exited with irqs disabled
71:M 20 Oct 2023 05:38:29.123 # Background saving terminated by signal 9
71:M 20 Oct 2023 05:38:35.044 * 10000 changes in 60 seconds. Saving...
71:M 20 Oct 2023 05:38:35.047 * Background saving started by pid 77
*** stack smashing detected ***: terminated
[   79.193591] BUG: unable to handle page fault for address: 0000000000000003
[   79.194132] #PF: supervisor instruction fetch in user mode
[   79.194688] #PF: error_code(0x0010) - not-present page
[   79.195154] IDT: 0xfffffe0000000000 (limit=0xfff) GDT: 0xfffffe0000001000 (limit=0x7f)
[   79.195813] LDTR: NULL
[   79.196021] TR: 0x40 -- base=0xfffffe0000003000 limit=0x4087
[   79.196496] PGD 101de5067 P4D 101de5067 PUD 101de0067 PMD 0 
[   79.196971] Oops: 0010 [#3] PREEMPT SMP
[   79.197297] CPU: 0 PID: 77 Comm: UKL Tainted: G S    D            6.3.0+ #1
[   79.197876] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
[   79.198851] RIP: 00c3:0x3
[   79.199078] Code: Unable to access opcode bytes at 0xffffffffffffffd9.
[   79.199664] RSP: 0018:00007ffd58fabb08 EFLAGS: 000514d2
[   79.200113] RAX: 0000000000000000 RBX: 0000000000a00000 RCX: 0000000000000027
[   79.200706] RDX: 00007ffd58fabb00 RSI: 00007ffd58fabc30 RDI: 0000000000000006
[   79.201298] RBP: 00007ffd58fac510 R08: 0000000000000000 R09: 0000000000000000
[   79.201891] R10: 0000000000000008 R11: 0000000000000002 R12: 0000000000001000
[   79.202497] R13: 00007ffd58fac410 R14: 0000000000000020 R15: 00007fa770ee9000
[   79.203091] FS:  0000000000a00000 GS:  0000000000000000
[   79.203530] CR2: 0000000000000003
[   79.203814] ---[ end trace 0000000000000000 ]---
[   79.204203] RIP: 00c3:0x3
[   79.204459] RSP: 0018:00007ffd58fabb08 EFLAGS: 000514d2
[   79.204914] RAX: 0000000000000000 RBX: 0000000000a00000 RCX: 0000000000000027
[   79.205509] RDX: 00007ffd58fabb00 RSI: 00007ffd58fabc30 RDI: 0000000000000006
[   79.206128] RBP: 00007ffd58fac510 R08: 0000000000000000 R09: 0000000000000000
[   79.206721] R10: 0000000000000008 R11: 0000000000000002 R12: 0000000000001000
[   79.207331] R13: 00007ffd58fac410 R14: 0000000000000020 R15: 00007fa770ee9000
[   79.207926] FS:  0000000000a00000(0000) GS:ffff88813bc00000(0000) knlGS:0000000000000000
[   79.208706] CS:  00c3 DS: 0000 ES: 0000 CR0: 0000000080050033
[   79.209247] CR2: 0000000000000003 CR3: 0000000101dd4005 CR4: 0000000000070ef0
[   79.209926] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   79.210535] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   79.211127] note: UKL[77] exited with irqs disabled

[khers]

The page faults listed here are from secondary Redis threads that are used for replication or writing to storage and should not effect the results (we are looking into them but I don't have an ETA)

I have also seen the scheduling while atomic failure before, but only on QEMU and only occasionally so I have not spent much time trying to track it down yet. I do most of my testing directly on hardware.

[easonycliu]

I started testing on memcached. And I got the first error

+ /UKL
can't run as root without the -u switch

So changed the username from NULL to "root" in memcached.c main

Then I met page fault again

+ /UKL
[    6.455298] BUG: unable to handle page fault for address: 0000000000000000
[    6.456913] #PF: supervisor write access in user mode
[    6.458091] #PF: error_code(0x0002) - not-present page
[    6.458601] IDT: 0xfffffe0000000000 (limit=0xfff) GDT: 0xfffffe0000001000 (limit=0x7f)
[    6.458957] LDTR: NULL
[    6.459061] TR: 0x40 -- base=0xfffffe0000003000 limit=0x4087
[    6.459304] PGD 101ded067 P4D 101ded067 PUD 101df2067 PMD 0 
[    6.459547] Oops: 0002 [#1] PREEMPT SMP
[    6.459713] CPU: 0 PID: 71 Comm: UKL Tainted: G S                 6.3.0+ #6
[    6.460016] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014
[    6.460370] RIP: 00c3:ukl_fini+0x0/0x80
[    6.460546] Code: Unable to access opcode bytes at 0xffffffff82400b56.
[    6.460831] RSP: 0018:00007ffe029b4a78 EFLAGS: 00010206
[    6.461064] RAX: 0000000000000000 RBX: ffffffff820f7f20 RCX: 0000000000ff0910
[    6.461376] RDX: 000000000000001c RSI: 0000000000000001 RDI: 0000000000ff15c0
[    6.461690] RBP: ffffffff838ea7c4 R08: 0000000000000000 R09: ffffffff82ae31bd
[    6.462006] R10: 0000000000000000 R11: ffffffff827e2b20 R12: 0000000000000000
[    6.462322] R13: 00007ffe029b50d0 R14: 0000000000000000 R15: 0000000000000002
[    6.462637] FS:  0000000000e00000 GS:  0000000000000000
[    6.462868] CR2: 0000000000000000
[    6.463024] ---[ end trace 0000000000000000 ]---
[    6.463233] RIP: 00c3:ukl_fini+0x0/0x80
[    6.463407] RSP: 0018:00007ffe029b4a78 EFLAGS: 00010206
[    6.463643] RAX: 0000000000000000 RBX: ffffffff820f7f20 RCX: 0000000000ff0910
[    6.463959] RDX: 000000000000001c RSI: 0000000000000001 RDI: 0000000000ff15c0
[    6.464273] RBP: ffffffff838ea7c4 R08: 0000000000000000 R09: ffffffff82ae31bd
[    6.464584] R10: 0000000000000000 R11: ffffffff827e2b20 R12: 0000000000000000
[    6.464892] R13: 00007ffe029b50d0 R14: 0000000000000000 R15: 0000000000000002
[    6.465206] FS:  0000000000e00000(0000) GS:ffff88813bc00000(0000) knlGS:0000000000000000
[    6.465568] CS:  00c3 DS: 0000 ES: 0000 CR0: 0000000080050033
[    6.465824] CR2: 0000000000000000 CR3: 0000000101de2006 CR4: 0000000000470ef0

I tried to debug and guess this is the stack trace
at main
at server_sockets
at server_socket
at getaddrinfo
at gaiconf_init
at fini