2014-12-26 Emacs Wiki Migration

I’m not happy. Today, multiple Apache restarts. Grepping syslog for restarts:

Dec 26 07:38:32 localhost monit[29022]: 'apache' trying to restart
Dec 26 11:38:53 localhost monit[29022]: 'apache' trying to restart
Dec 26 14:47:16 localhost monit[29022]: 'apache' trying to restart
Dec 26 16:49:46 localhost monit[29022]: 'apache' trying to restart
Dec 26 17:26:06 localhost monit[29022]: 'apache' trying to restart
Dec 26 18:06:29 localhost monit[29022]: 'apache' trying to restart
Dec 26 19:48:54 localhost monit[29022]: 'apache' trying to restart

https://alexschroeder.ch/pics/15494392223_95ee7bf147_o.png

I’m going to further reduce the number of threads.

<IfModule mpm_worker_module>
    StartServers          1
    ServerLimit           2
    ThreadsPerChild       5
    MaxClients           10
    MinSpareThreads      10
    MaxSpareThreads      20
    MaxRequestsPerChild  10000
</IfModule>

I also installed *lsof* and wrote a little plugin for munin. What I really want to see is the number of open files, or something like that. I remember checking that when we had load problems on Mirabel without a matching number of processes actually /doing/ anything. Just in case the processes are all waiting for file access.

Mirabel

I’m not sure how well this is going to work, but this is my first stab at the *files* plugin. 🙂

#!/bin/sh
#
# Plugin to monitor the number of open files in the system.
#
# Parameters:
#
# 	config   (required)
# 	autoconf (optional - used by munin-config)
#
# Magic markers (Used by munin-config and some installation scripts.
# Optional):
#
#%# family=contrib
#%# capabilities=autoconf

# PROC=${0##*/files_}
PORTS="80 443"

if [ "$1" = "autoconf" ]; then
	if [ -x /usr/bin/lsof ]; then
		echo yes
		exit 0
	else
		echo "no (no lsof)"
		exit 0
	fi
fi

if [ "$1" = "config" ]; then
	echo graph_title "open files"
	echo graph_vlabel "number of files"
	echo graph_info "This graph shows the number of open files by port (80 is http, 443 is https)."
	echo graph_category "apache"
	for P in $PORTS; do
	    echo "$P.label port $P"
	done
	exit 0
fi

for P in $PORTS; do
    N=$(lsof -i :$P | tail -n +2 | wc -l)
    echo "$P.value $N"
done

☯

All I know is that load is climbing right now. Close to 6. Time for bed! Grrrrr.

☯

A few hours later, stats seem to be worse:

Dec 26 22:47:16 localhost monit[29022]: 'apache' trying to restart
Dec 26 23:23:34 localhost monit[29022]: 'apache' trying to restart
Dec 26 23:58:01 localhost monit[29022]: 'apache' trying to restart
Dec 27 00:10:19 localhost monit[29022]: 'apache' trying to restart
Dec 27 00:38:40 localhost monit[29022]: 'apache' trying to restart
Dec 27 01:06:58 localhost monit[29022]: 'apache' trying to restart
Dec 27 01:17:15 localhost monit[29022]: 'apache' trying to restart
Dec 27 03:51:40 localhost monit[29022]: 'apache' trying to restart
Dec 27 04:23:59 localhost monit[29022]: 'apache' trying to restart
Dec 27 04:42:21 localhost monit[29022]: 'apache' trying to restart
Dec 27 05:30:47 localhost monit[29022]: 'apache' trying to restart
Dec 27 06:59:09 localhost monit[29022]: 'apache' trying to restart
Dec 27 08:05:33 localhost monit[29022]: 'apache' trying to restart

Let’s hope that things will improve with prefork!

​#Emacs ​#Wikis ​#Oddmuse ​#mod perl ​#Apache ​#devops ​#Administration