Reliability of the zombie_handled test

[sysarch-repo]

Team,
I am using CNF testsuite 1.3.3 and have doubts regarding the reliability of the zombie_handling test that is part of the cert essential group. I ask comments and support in shedding light on the topics below.
Thanks!

1. failed zombie handling test makes the subsequent sig term handling test fail due to orphaned processes (zombies) left behind. Here the log of sig_term handling test hitting (IMO) a zombie process left behind after the failed zombie_handled test:

I, [2025-01-01 22:31:17 UTC #11785]  INFO -- cnf-testsuite: KubectlClient.exec command: kubectl exec -n cnf-testsuite -ti cluster-tools-dkvzh -- bash -c 'sleep 10 && kill 33069 && sleep 5 && kill -9 33069’
bash: line 1: kill: (33069) - No such process
command terminated with exit code 1

This seems to be a design flaw and may justify a bug ticket. -> done as per #2207

2. Consider two application (a) simple app1 and (b) production app2. Both are tested for zombie handling using exec into the container and there starting and terminating a script that continuously forks child processes, e.g.:

#!/bin/sh
# Parent process
while true; do
  # Fork a child process in the background
  (sleep 1) &
  # Do not call wait
  sleep 2
done

2a. The simple app1 shows zombie processes after the script is terminated in the app1 container:

top (script terminated):
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
    1     0 1000     S    1207m   4%   1   0% /usr/local/bin/app1
   28     0 1000     S     1672   0%   6   0% sh
   49    28 1000     R     1600   0%  10   0% top
   47     1 1000     Z        0   0%  14   0% [sleep] <- zombie
   46     1 1000     Z        0   0%   9   0% [sleep] <- zombie

Sig term handling details:
top:

  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
    1     0 1000     S    1207m   4%   5   0% /usr/local/bin/app1
   27     0 1000     S     1672   0%  14   0% sh
   33    27 1000     R     1600   0%   9   0% top
~ $ strace -p 1
strace: Process 1 attached
futex(0xbeae40, FUTEX_WAIT_PRIVATE, 0, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=0, si_uid=0} ---
futex(0xc4b700, FUTEX_WAKE_PRIVATE, 1)  = 1
rt_sigreturn({mask=~[HUP INT QUIT ILL TRAP ABRT BUS FPE KILL SEGV TERM STKFLT CHLD STOP URG PROF SYS RTMIN RT_1 RT_2]}) = ?
command terminated with exit code 137

The app1 application passes both zombie_handling and sigterm_test in both cgroup v1 and v2 environments (here cgroup v2):

Microservice Tests
🎬 Testing: [specialized_init_system]
pod/dns-dserver-bfc5679b-t9c64 has container 'dserver' with /usr/local/bin/app1 as init process
pod/dns-drouter-0 has container 'drouter' with /usr/local/bin/app1 as init process
✖️  🏆FAILED: [specialized_init_system] Containers do not use specialized init systems (ভ_ভ) ރ 🚀
🎬 Testing: [single_process_type]
✔️  🏆PASSED: [single_process_type] Only one process type used ⚖👀
🎬 Testing: [zombie_handled]
✔️  🏆PASSED: [zombie_handled] Zombie handled ⚖👀
🎬 Testing: [sig_term_handled]
✔️  🏆PASSED: [sig_term_handled] Sig Term handled ⚖👀
Microservice results: 3 of 4 tests passed

The discrepancy is that the manual test using the script is showing zombie process while the CNTI test will pass. Note, when using specialized init system, no zombie process is seen:

top (script running)
 PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
    7     1 1000     S    1207m   4%  12   0% /usr/local/bin/app1
   28     0 1000     S     1672   0%   2   0% sh
   36    28 1000     S     1604   0%   2   0% {script.sh} /bin/sh ./script.sh
   39    28 1000     R     1600   0%   2   0% top
   44    36 1000     S     1588   0%   5   0% sleep 1
   45    36 1000     S     1588   0%   4   0% sleep 2
    1     0 1000     S      224   0%  12   0% /usr/local/bin/dumb-init -- /usr/local/bin/app1

top (script terminated):
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
    7     1 1000     S    1207m   4%  12   0% /usr/local/bin/app1
   28     0 1000     S     1672   0%   4   0% sh
   92    28 1000     R     1600   0%  15   0% top
    1     0 1000     S      224   0%  12   0% /usr/local/bin/dumb-init -- /usr/local/bin/app1

There arises a question why the zombie_handling test is not detecting the zombie processes, unless the test using the script is not representative and cannot be used as reference. Thoughts?

2b. The production app2 shows zombie processes after the script is terminated in the app2 container:

top (script terminated):
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
      1 prod-us+  20   0 9051044   1.3g  26308 S   0.0   4.1   0:53.24 app2
   2004 prod-us+  20   0    4972   4448   3288 S   0.0   0.0   0:00.01 sh 
   2056 prod-us+  20   0       0      0      0 Z   0.0   0.0   0:00.00 sleep <- zombie
   2057 prod-us+  20   0       0      0      0 Z   0.0   0.0   0:00.00 sleep <- zombie
   2058 prod-us+  20   0    7412   3592   2992 R   0.0   0.0   0:00.00 top 

Sig term handling details:

strace: Process 1 attached
futex(0x1d049878, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=0, si_uid=0} ---
rt_sigprocmask(SIG_UNBLOCK, [ILL TRAP BUS FPE SEGV], NULL, 8) = 0
futex(0x1d021fc0, FUTEX_WAKE_PRIVATE, 1) = 1
rt_sigreturn({mask=[QUIT]})             = 202
futex(0x1d049878, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY) = 0
futex(0x1d049828, FUTEX_WAKE_PRIVATE, 1) = 0
rt_sigprocmask(SIG_BLOCK, ~[], [QUIT], 8) = 0
clone3({flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, child_tid=0x7fd9fedfe910, parent_tid=0x7fd9fedfe910, exit_signal=0, stack=0x7fd9fecff000, stack_size=0xfeec0, tls=0x7fd9fedfe640} => {parent_tid=[508]}, 88) = 508
rt_sigprocmask(SIG_SETMASK, [QUIT], NULL, 8) = 0
futex(0x1d38d638, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x1d38d5e8, FUTEX_WAKE_PRIVATE, 1) = 1
rt_sigprocmask(SIG_BLOCK, ~[], [QUIT], 8) = 0
clone3({flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, child_tid=0x7fdb64ade910, parent_tid=0x7fdb64ade910, exit_signal=0, stack=0x7fdb649df000, stack_size=0xfeec0, tls=0x7fdb64ade640} => {parent_tid=[509]}, 88) = 509
rt_sigprocmask(SIG_SETMASK, [QUIT], NULL, 8) = 0
futex(0x1db8c828, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x1db8c7d8, FUTEX_WAKE_PRIVATE, 1) = 1
rt_sigprocmask(SIG_BLOCK, ~[], [QUIT], 8) = 0
clone3({flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, child_tid=0x7fdb649de910, parent_tid=0x7fdb649de910, exit_signal=0, stack=0x7fdb648df000, stack_size=0xfeec0, tls=0x7fdb649de640} => {parent_tid=[510]}, 88) = 510
rt_sigprocmask(SIG_SETMASK, [QUIT], NULL, 8) = 0
futex(0x1d044cd8, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x1e59df48, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x1e59def8, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x1d04987c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY) = 0
futex(0x1d049828, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x1d049878, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY) = 0
futex(0x1d049828, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x1d046148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x1d0460f8, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x1d33c620, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x1d33cc50, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x1d33cc00, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x1d02614c, FUTEX_WAKE_PRIVATE, 2147483647) = 1
futex(0x1d0260f8, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x1d021fc0, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x1d044f48, FUTEX_WAKE_PRIVATE, 2147483647) = 1
futex(0x1d044ef8, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x1d044f4c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY) = ?
command terminated with exit code 137

The app2 application passes both zombie_handling and sigterm_test in cgroup v1 environment and fails in v2 environments:

Test of app2 with cgroup v1:
Microservice Tests
🎬 Testing: [specialized_init_system]
pod/dnstest-cnti-0 has container 'dnstest-cnti' with /opt/comp/sys/bin/app2as init process
✖️  🏆FAILED: [specialized_init_system] Containers do not use specialized init systems (ভ_ভ) ރ 🚀
🎬 Testing: [single_process_type]
✔️  🏆PASSED: [single_process_type] Only one process type used ⚖👀
🎬 Testing: [zombie_handled]
✔️  🏆PASSED: [zombie_handled] Zombie handled ⚖👀
🎬 Testing: [sig_term_handled]
✔️  🏆PASSED: [sig_term_handled] Sig Term handled ⚖👀
Microservice results: 3 of 4 tests passed

Test of app2 with cgroup v2:
Microservice Tests
🎬 Testing: [specialized_init_system]
pod/dnstest-cnti-0 has container 'dnstest-cnti' with /opt/comp/sys/bin/app2as init process
✖️  🏆FAILED: [specialized_init_system] Containers do not use specialized init systems (ভ_ভ) ރ 🚀
🎬 Testing: [single_process_type]
✔️  🏆PASSED: [single_process_type] Only one process type used ⚖👀
🎬 Testing: [zombie_handled]
Process sleep has a state of Z (zombie)
✖️  🏆FAILED: [zombie_handled] Zombie not handled ⚖👀
🎬 Testing: [sig_term_handled]
Pod: dnstest-cnti-0, Container: dnstest-cnti, Result: failed
✖️  🏆FAILED: [sig_term_handled] Sig Term not handled ⚖👀
Microservice results: 1 of 4 tests passed

The discrepancy is that the manual test using the script is showing zombie processes while the CNTI test will pass in cgroup v1 environments. Note, when using specialized init system, no zombie process is seen and the test will pass:

top (script running):
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
      7 prod-us+  20   0 9054080   1.3g  26224 S   2.0   4.1   0:27.91 app2
      1 prod-us+  20   0     228      8      0 S   0.0   0.0   0:00.00 dumb-init
    699 prod-us+  20   0    4972   4400   3232 S   0.0   0.0   0:00.01 sh
    764 prod-us+  20   0    4324   3544   3040 S   0.0   0.0   0:00.00 script.sh
    769 prod-us+  20   0    7412   3656   3052 R   0.0   0.0   0:00.00 top
    773 prod-us+  20   0    4936   1408   1232 S   0.0   0.0   0:00.00 sleep
    774 prod-us+  20   0    4936   1320   1144 S   0.0   0.0   0:00.00 sleep 

top (after script termination):
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
      7 prod-us+  20   0 9054080   1.3g  26224 S  26.7   4.1   0:28.56 app2
      1 prod-us+  20   0     228      8      0 S   0.0   0.0   0:00.00 dumb-init
    699 prod-us+  20   0    4972   4400   3232 S   0.0   0.0   0:00.01 sh
    891 prod-us+  20   0    7412   3560   2956 R   0.0   0.0   0:00.00 top

There arises again a question why the zombie_handling test is not detecting the zombie processes, unless the test using the script is not representative and cannot be used as reference. Further, why are there different results seen depending on the cgroup version? I only see that app2 has child processes to manage while app1 is only having one parent process. Thoughts?

Note, this is not about coming to the conclusion that using specialized init system solves the problem. The question here is how reliable the zombie_handling test is and/or what is wrong with the assumptions / conclusions posted here.

I am staying tuned for any comments or follow-up thoughts. Thank you.