(Continued from here.)
Memory problems continue. The VPS control panel says “Memory Usage: 98%. 1.96 GB of 2 GB Used / 38.52 MB Free” for example. The numbers change all the time. I thought perhaps reducing the number of Apache processes would help.
I changed `/etc/apache2/apache2.conf` to say:
<IfModule mpm_prefork_module>
StartServers 2
MinSpareServers 2
MaxSpareServers 4
MaxClients 150
MaxRequestsPerChild 10000
</IfModule>
The numbers used to be 5, 5, 10, I think. When I use `ps aux|grep sbin/apache` I see four processes, now. So, did things improve? No. 😭
When I’m running `top`, I can watch as one of the apache processes starts with around 8% of memory and grows and grows and within about ten seconds, %MEM approaches 100% and then the process disappears. Killed, I presume. As far as I can tell, this would happen even if I had just one process running.
top - 12:15:57 up 11:13, 1 user, load average: 2.71, 3.33, 3.16
Tasks: 25 total, 1 running, 21 sleeping, 2 stopped, 1 zombie
%Cpu(s): 57.1 us, 25.5 sy, 0.0 ni, 16.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.5 st
KiB Mem: 2097152 total, 2088764 used, 8388 free, 0 buffers
KiB Swap: 524288 total, 496028 used, 28260 free, 80 cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1550 www-data 20 0 2589m 1.9g 1704 S 89.7 97.0 0:06.34 /usr/sbin/apach
1583 www-data 20 0 485m 18m 1704 S 8.6 0.9 0:00.26 /usr/sbin/apach
1 root 20 0 10608 144 40 S 0.0 0.0 0:00.64 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd/2828
3 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khelper/2828
1612 www-data 20 0 0 0 0 Z 0.0 0.0 0:00.14 wiki.pl
1790 messageb 20 0 29764 244 4 S 0.0 0.0 0:00.00 dbus-daemon
How can this be? It looks like a *memory leak* to me! I need to figure out how to debug this for my mod_perl setup. Perhaps I can start by setting MaxRequestsPerChild to a really low number? This morning, this parameter wasn’t set at all. Then I read Tuning an Apache server in 5 minutes and set it to 10000. I think I’ll set it to 100 and see how it goes.
Tuning an Apache server in 5 minutes
top - 12:27:09 up 11:25, 1 user, load average: 3.48, 2.99, 2.99 Tasks: 24 total, 1 running, 21 sleeping, 2 stopped, 0 zombie %Cpu(s): 35.2 us, 14.4 sy, 0.0 ni, 50.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 2097152 total, 2087168 used, 9984 free, 0 buffers KiB Swap: 524288 total, 509824 used, 14464 free, 72 cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4547 www-data 20 0 2527m 1.9g 1708 S 66.8 96.9 0:04.95 /usr/sbin/apach 4575 www-data 20 0 420m 17m 1708 S 5.7 0.9 0:00.17 /usr/sbin/apach 4363 alex 20 0 23124 724 596 R 0.3 0.0 0:00.05 top
Did things improve? No. 😳
OK, I rewrote the initialization of the wiki. I had an idea...
top - 12:57:43 up 11:55, 1 user, load average: 9.53, 5.91, 3.88 Tasks: 30 total, 2 running, 26 sleeping, 2 stopped, 0 zombie %Cpu(s): 98.7 us, 0.0 sy, 1.3 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 2097152 total, 470128 used, 1627024 free, 0 buffers KiB Swap: 524288 total, 404140 used, 120148 free, 106960 cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 11517 www-data 20 0 580m 120m 4640 S 49.2 5.9 4:08.00 /usr/sbin/apach 11547 www-data 20 0 567m 108m 4532 S 49.2 5.3 1:48.72 /usr/sbin/apach
Wow, memory consumption is really low, now. But load problems are back. 😥
Since memory consumption is low, I’m trying to use more processes again.
<IfModule mpm_prefork_module>
StartServers 2
MinSpareServers 2
MaxSpareServers 10
MaxClients 150
MaxRequestsPerChild 10000
</IfModule>
The output of `top` seems to indicate that the number of processes has no effect on %CPU. Basically I have two CPUs available, and each is running an apache process... There’s not much else to be gained unless these processes spend their time waiting for I/O. We’ll see.
An hour later:
top - 14:44:16 up 13:42, 1 user, load average: 1.97, 1.92, 1.97 Tasks: 25 total, 1 running, 22 sleeping, 2 stopped, 0 zombie %Cpu(s): 40.5 us, 14.1 sy, 0.0 ni, 45.2 id, 0.0 wa, 0.0 hi, 0.0 si, 0.2 st KiB Mem: 2097152 total, 1457144 used, 640008 free, 0 buffers KiB Swap: 524288 total, 416852 used, 107436 free, 4400 cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 14628 www-data 20 0 1858m 1.3g 3956 S 87.2 65.6 0:02.78 /usr/sbin/apach 14656 www-data 20 0 445m 43m 3468 S 10.0 2.1 0:00.30 /usr/sbin/apach 16857 www-data 20 0 939m 1532 140 S 0.0 0.1 0:00.00 /usr/sbin/apach 21618 www-data 20 0 494m 2936 132 S 0.0 0.1 0:00.00 /usr/sbin/apach 21619 www-data 20 0 494m 2952 132 S 0.0 0.1 0:00.00 /usr/sbin/apach
Load seems to be under control. Memory, not so much. It keeps fluctuating. I guess whatever I thought I had fixed, wasn’t actually helping? 😞
Another fix, another moment of hope.
top - 15:00:48 up 13:58, 1 user, load average: 1.14, 1.10, 1.42
Tasks: 25 total, 1 running, 22 sleeping, 2 stopped, 0 zombie
%Cpu(s): 91.2 us, 1.8 sy, 0.0 ni, 6.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.2 st
KiB Mem: 2097152 total, 331668 used, 1765484 free, 0 buffers
KiB Swap: 524288 total, 404936 used, 119352 free, 85024 cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16085 www-data 20 0 568m 101m 4588 S 75.2 5.0 5:32.24 /usr/sbin/apach
17497 www-data 20 0 574m 52m 4276 S 43.9 2.6 0:03.15 /usr/sbin/apach
16114 www-data 20 0 527m 52m 4832 S 0.3 2.6 0:31.14 /usr/sbin/apach
1 root 20 0 10608 196 60 S 0.0 0.0 0:00.81 init
☯
Gaaaah. It `dpkg-query -l apache2` indicates that I’m not using the prefork server but the worker! All those times I thought I was changing the parameters, I guess I wasn’t.
alex@kallobombus:~$ dpkg-query -l apache2* Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-=================-=============-=============-======================================== ii apache2 2.2.22-13+deb amd64 Apache HTTP Server metapackage un apache2-common <none> (no description available) un apache2-doc <none> (no description available) un apache2-mpm <none> (no description available) un apache2-mpm-event <none> (no description available) un apache2-mpm-itk <none> (no description available) un apache2-mpm-prefo <none> (no description available) ii apache2-mpm-worke 2.2.22-13+deb amd64 Apache HTTP Server - high speed threaded un apache2-suexec <none> (no description available) un apache2-suexec-cu <none> (no description available) ii apache2-utils 2.2.22-13+deb amd64 utility programs for webservers ii apache2.2-bin 2.2.22-13+deb amd64 Apache HTTP Server common binary files ii apache2.2-common 2.2.22-13+deb amd64 Apache HTTP Server common files
Given that currently load is hovering between 45 and 50, what should I do? I changed MaxRequestsPerChild from 0 to 10000.
<IfModule mpm_worker_module>
StartServers 2
MinSpareThreads 25
MaxSpareThreads 75
ThreadLimit 64
ThreadsPerChild 25
MaxClients 150
MaxRequestsPerChild 10000
</IfModule>
When I restarted apache2 just now, I noticed how load has dropped below 10. Weird! Why was it so high?
☯
Out of curiosity, I wanted to know whether it is indeed emacswiki.org that is smashing my server. And I guess it is. But other sites, such as my homepage, aren’t doing too badly, either. Perhaps I need to mod_perl them, too!
bin/site-comparison < /var/log/apache2/access.log
Site hits bandw. hits% interv. status code distrib.
www.emacswiki.org 65092 10K 74% 1.0s 200 (61%), 301 (19%), 500 (7%)
www.alexschroeder.ch 7877 30K 9% 8.3s 200 (90%), 304 (3%), 404 (1%),
www.communitywiki.org 7531 15K 8% 8.7s 200 (81%), 301 (8%), 404 (4%),
www.oddmuse.org 3389 21K 3% 19.4s 200 (69%), 301 (12%), 404 (8%)
www.campaignwiki.org 3302 11K 3% 19.9s 200 (78%), 404 (14%), 302 (5%)
www.orientalisch.info 136 4K 0% 481.8s 200 (73%), 301 (14%), 304 (8%)
www.arabisch-lernen.org 60 22K 0% 1102.9s 200 (90%), 404 (5%), 304 (3%),
☯
I’ve been reading a German blog post about Apache tuning and decided to add a ServerLimit:
a German blog post about Apache tuning
<IfModule mpm_worker_module>
StartServers 2
ServerLimit 3
MinSpareThreads 25
MaxSpareThreads 75
ThreadLimit 64
ThreadsPerChild 50
MaxClients 150
MaxRequestsPerChild 10000
</IfModule>
Less processes would seem to be the way to reduce load? I’m not sure...
Right now, my homepage is incredibly slow and Emacs Wiki is—sometimes—very fast. That’s cool, I guess. mod_perl really seems to help. Perhaps it’s time to start investigating all the remaining Emacs Wiki issues?
As I’m looking at the output of `ps`, I noticed something else that’s weird: There are several apache processes which don’t do anything (apparently) and they were started a long time ago. The last apache2 restart I ran was around 23:00. I guess I’ll send them a `kill` signal. Let’s see where that takes us.
root@kallobombus:/home/alex# ps -u www-data ao pid,ppid,%cpu,%mem,vsz,rss,stat,start,time,cmd PID PPID %CPU %MEM VSZ RSS STAT STARTED TIME CMD 16857 1 0.0 0.0 962520 1536 S 04:20:18 00:00:00 /home/alex/emacsw -k start 21618 1 0.0 0.1 506716 2940 S 02:19:01 00:00:00 /usr/sbin/apache2 -k start 21619 1 0.0 0.1 506724 2956 S 02:19:01 00:00:00 /usr/sbin/apache2 -k start 22110 22109 0.0 0.2 21340 4312 Ss 21:42:45 00:00:00 -bash 26046 22110 0.0 0.0 33584 1872 S 22:23:58 00:00:00 sudo su 26051 26046 0.0 0.0 37112 1368 S 22:24:08 00:00:00 su 26052 26051 0.0 0.1 20488 3484 S 22:24:09 00:00:00 bash 28681 28668 78.5 5.8 973236 123176 Sl 23:00:34 00:04:36 /usr/sbin/apache2 -k start 29401 1 0.0 0.1 55720 2216 S 16:37:12 00:00:01 /usr/bin/perl /home/alex/alexschroeder.ch/wiki.pl 29423 1 0.0 0.0 486056 192 S 16:37:46 00:00:00 /home/alex/emacsw -k start 30429 26052 0.0 0.0 16796 1264 R+ 23:06:26 00:00:00 ps -u www-data ao pid,ppid,%cpu,%mem,vsz,rss,stat,sta
I wonder why load is so high. A disk latency issue?
(Continued tomorrow.)
#Apache #mod\ perl #Oddmuse #Administration