💾 Archived View for thrig.me › blog › 2024 › 02 › 14 › re-zombies.gmi captured on 2024-08-18 at 18:57:28. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2024-03-21)
-=-=-=-=-=-=-
I had over a 1000 PIDs on one of my servers that were <defunct> zombies children of a weechat process someone was running. I didn't want to kill their weechat, but I still wanted the zombies gone. So I sent the weechat a SIGCHLD just in case that'd trigger a cleanup wait() in it. It did not.
gemini://gemini.thebackupbox.net/~epoch/blog/zombies
There's a "signal (SIGCHLD, SIG_DFL);" line somewhere in src/plugins/ruby/weechat-ruby.c that sets the default action for SIGCHLD. Elsewhere there's a signal_init function that sets up actions for various signals, none of which are SIGCHLD. The default for SIGCHLD is to discard the signal. So unless there's signal code hiding somewhere else that I missed, the weechat default is the default for SIGCHLD. Unless of course the parent of weechat setup some funky signal handling that was inherited, though if the ruby code is ever reached the handling for SIGCHLD should get set back to the default.
As for the waiting part in weechat, the usual form is waitpid(2) with WHNOHANG, sometimes for a specific process and sometimes for any process (-1). In theory "sys_waitpid" is supposed to prevent children from becoming zombies, but that may not be being reached, or that function may only check up to a certain number of processes, so bad luck if there are more children than that call is told to check for, and the waitpid -1 (any process) misses them, such as due to a failed waitpid aborting the loop over the given number of forks? Probably you'd need to debug weechat to see exactly what the signal states are and what waitpid code is being reached. I would suspect that something in sys_wait is going awry?
One idea if the problem continues would be to call "signal(SIGCHLD, SIG_IGN)" in the process; this should prevent zombies from accumulating on what passes for unix these days. However, if the ruby code is ever gone through the handler will get set back to the default state. Also the signal handler would need to be setup before any zombies accumulate; one way would be to use a weechat wrapper that sets up signal handling and then execs weechat. (Or to block the signal in the parent exec wrapper, if you're really serious.)
Now into the weeds!
Portability may vary here especially if you are on some ancient or non-standard unix. All my testing was done on OpenBSD 7.4. You may want to do zombie testing on a sacrificial system for when things get out of hand. (Or not, if you want practice debugging process problems on a critical system.)
Here we want to see what signal related calls are made for the default case (discard the signal) and when there is a SIGCHLD handler setup. Note that signals may be temporarily blocked or masked in complicated processes, and again that signal handling can be inherited.
void handle_child(int unused) { printf("SIGCHLD %d\n", getpid()); }
...
int status;
pid_t pid, ret;
// TWEAK either the default or with a handler
signal(SIGCHLD, SIG_DFL);
//signal(SIGCHLD, handle_child);
pid = fork();
if (pid < 0) err(1, "fork");
if (pid == 0) {
sleep(1);
_exit(42); // make the exit status more distinctive
} else {
sleep(5);
raise(SIGCHLD);
errno = 0;
ret = waitpid(WAIT_ANY, &status, WNOHANG);
printf("WAITPID %d ret=%d status=%d errno=%d\n", pid, ret,
status, errno);
}
The default case shows a sigaction call for SIGCHLD and not much else going on signal-wise.
$ make child && ktrace -id ./child
cc -O2 -pipe -o child child.c
WAITPID 1641 ret=1641 status=10752 errno=0
$ kdump | grep -i sig
52654 child CALL sigaction(SIGCHLD,0x760f6473d960,0x760f6473d950)
52654 child STRU struct sigaction { handler=SIG_DFL, mask=0<>, flags=0x2<SA_RESTART> }
52654 child STRU struct sigaction { handler=SIG_DFL, mask=0<>, flags=0<> }
52654 child RET sigaction 0
52654 child CALL thrkill(0,SIGCHLD,0)
"thrkill" is what the "raise" becomes; right after that the waitpid code is reached, as far as system calls go.
...
52654 child CALL thrkill(0,SIGCHLD,0)
52654 child RET thrkill 0
52654 child CALL kbind(0x760f6473d8c8,24,0x13c471bb53311f31)
52654 child RET kbind 0
52654 child CALL kbind(0x760f6473d8c8,24,0x13c471bb53311f31)
52654 child RET kbind 0
52654 child CALL wait4(WAIT_ANY,0x760f6473d99c,0x1<WNOHANG>,0)
...
If instead a SIGCHLD handler is installed, there is more signal activity. Note that the handler is called twice; why might that be?
$ make child && ktrace -id ./child
cc -O2 -pipe -o child child.c
SIGCHLD 83265
SIGCHLD 83265
WAITPID 84656 ret=84656 status=10752 errno=0
$ kdump | grep -i sig
83265 child CALL sigaction(SIGCHLD,0x7bee9112dfd0,0x7bee9112dfc0)
83265 child STRU struct sigaction { handler=0x67107344d20, mask=0<>, flags=0x2<SA_RESTART> }
83265 child STRU struct sigaction { handler=SIG_DFL, mask=0<>, flags=0<> }
83265 child RET sigaction 0
83265 child PSIG SIGCHLD caught handler=0x67107344d20 mask=0<> status=SIG 0 pid=0 uid=0<"root">
"SIGCHLD 83265
83265 child CALL sigreturn(0x7bee9112dad0)
83265 child RET sigreturn JUSTRETURN
83265 child CALL thrkill(0,SIGCHLD,0)
83265 child PSIG SIGCHLD caught handler=0x67107344d20 mask=0<> status=SIG 0 pid=0 uid=0<"root">
"SIGCHLD 83265
83265 child CALL sigreturn(0x7bee9112db10)
83265 child RET sigreturn JUSTRETURN
Ignoring SIGCHLD should not result in zombie accumulation, at least on modern unix systems that follow the SUSv3 specification. What happens when the parent crashes (e.g. with abort(3)) before calling waitpid without the SIG_IGN setup?
int status;
pid_t pid, ret;
// TWEAK with this commented out the child should become a zombie
// for about 10 seconds
signal(SIGCHLD, SIG_IGN);
pid = fork();
if (pid < 0) err(1, "fork");
if (pid == 0) {
sleep(1);
_exit(42); // make the exit status more distinctive
} else {
sleep(10); // allow time to run process tools elsewhere
errno = 0;
ret = waitpid(WAIT_ANY, &status, WNOHANG);
printf("WAITPID %d ret=%d status=%d errno=%d\n", pid, ret,
status, errno);
}
With the signal line commented out, a ps(1) should briefly show the zombie.
$ ps axo pid,state | fgrep Z
32673 Z+
Elsewhere, the waitpid call eventually goes through, which should remove the zombie. Note that the exit status word is a 16-bit value, not what the shell mangles into the $? variable.
$ make child && ./child
cc -O2 -pipe -o child child.c
WAITPID 32673 ret=32673 status=10752 errno=0
$ perl -e 'printf "%016b\n", shift' 10752
0010101000000000
$ perl -e 'printf "%08b\n", shift >> 8' 10752
00101010
$ perl -e 'printf "%d\n", shift >> 8' 10752
42
With the SIGCHLD set to SIG_IGN, waitpid should get a ECHILD error, which here means that the process has no children to be waited on, as the kernel already has shuffled the child off its mortal coil.
$ make child && ./child
cc -O2 -pipe -o child child.c
WAITPID 91430 ret=-1 status=0 errno=10
$ errno 10
ECHILD 10 No child processes
This is mostly to show that a parent process can influence the signal handling of a child. This means that parents may need to take care to not pass on anything inappropriate to a child (e.g. SIGPIPE handling) and child processes may need to take care to setup the correct signal environment in certain cases.
int status;
pid_t pid, ret, another_pid;
// TWEAK SIG_IGN is probably not enough
sigset_t block;
sigemptyset(&block);
sigaddset(&block, SIGCHLD);
sigprocmask(SIG_BLOCK, &block, NULL);
//signal(SIGCHLD, SIG_IGN);
pid = fork();
if (pid < 0) err(1, "fork");
if (pid == 0) {
signal(SIGCHLD, handle_child);
another_pid = fork();
if (another_pid < 0) err(1, "fork");
if (another_pid == 0) {
sleep(1);
_exit(42); // make the exit status more distinctive
} else {
sleep(5);
errno = 0;
ret = waitpid(WAIT_ANY, &status, WNOHANG);
printf("WAITPID %d ret=%d status=%d errno=%d\n",
another_pid, ret, status, errno);
}
} else {
wait(NULL);
}
In this case SIG_IGN is not enough (the signal(3) call in the child resets it) but the parent blocking the signal does prevent the child's handle_child function from being called, as the "SIGCHLD <pid>" line does not appear:
$ make child && ./child
cc -O2 -pipe -o child child.c
WAITPID 95271 ret=95271 status=10752 errno=0
If the parent simply ignores SIGCHLD then that ignore will be passed down to clients and may inappropriately change their behavior. Better code might reset the signal handling just before the exec of the new process, something like:
signal(SIGCHLD, SIG_IGN);
pid_t pid = fork();
if (pid == 0) {
signal(SIGCHLD, SIG_DFL);
// any other preparation
// exec new program here
...
err(1, "exec failed??");
}
// parent stuff
...