Gaah! 😟
So, I still think this is a problem with the hosting provider. Here’s what I think is evidence:
https://alexschroeder.ch/pics/15509223034_8ee49a110d_o.png
What could be causing this regular forks if not other virtual machines hogging CPU at regular intervals, or other virtual machines hogging disk access, causing CGI scripts to pile up? I wonder...
☯
Things I care about: Diagnosing Disk I/O issues: swapping, high IO wait, congestion.
Diagnosing Disk I/O issues: swapping, high IO wait, congestion
Let’s see what *iotop -ao* says right now:
Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND 11774 be/4 www-data 36.00 K 20.00 K 0.00 % 0.02 % [/usr/sbin/apach] 11801 be/4 www-data 0.00 B 12.00 K 0.00 % 0.00 % apache2 -k start 11810 be/4 www-data 0.00 B 4.00 K 0.00 % 0.00 % apache2 -k start 29022 be/4 root 0.00 B 32.00 K 0.00 % 0.00 % monit -c /etc/monit/monitrc 20368 be/4 root 0.00 B 60.00 K 0.00 % 0.00 % rsyslogd -c5
I don’t think we’re using swap at all (munin shows us using less than 2G with 4G available). But who knows? I used `top`, pressed `f` to show the list of fields, and I used `d` (display) and `s` (sort) on `SWAP`.
top - 22:14:00 up 7 days, 21:11, 2 users, load average: 0.51, 0.31, 0.22 Tasks: 24 total, 1 running, 23 sleeping, 0 stopped, 0 zombie %Cpu(s): 3.6 us, 0.6 sy, 0.0 ni, 95.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 4194304 total, 894976 used, 3299328 free, 0 buffers KiB Swap: 1048576 total, 5524 used, 1043052 free, 785880 cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND SWAP 15874 root 20 0 71192 5224 1620 S 0.0 0.1 2:11.36 fail2ban-server 2296 27076 root 20 0 72564 1092 708 S 0.0 0.0 0:16.47 sendmail-mta 1384 21949 root 20 0 49892 684 568 S 0.0 0.0 0:02.23 sshd 492 29022 root 20 0 42984 2688 2136 S 0.0 0.1 0:48.28 monit 284 20367 root 20 0 49860 4504 712 S 0.0 0.1 0:24.56 rsyslogd 236 19427 root 30 10 12252 788 784 S 0.0 0.0 0:00.00 rsync 160 15876 root 20 0 22260 896 672 S 0.0 0.0 0:08.86 gam_server 136 22815 root 20 0 20368 496 364 S 0.0 0.0 0:02.03 cron 120
I don’t think that’s suspicious. Those tasks probably ended up in swap because they have been running for so long?
☯
I installed the apache_vhosts plugin for munin. I *think* there will be no surprises with emacswiki.org taking the brunt with a factor of 10:1. Right now, however, it seems to be misconfigured somehow, not recording any emacswiki.org traffic and missing most of the other traffic. Weird!
☯
Ah, apache_vhosts munin plugin problem fixed. All my sites were using a `CustomLog` except for alexschroeder.ch:80 and campaignwiki.org:80—these just redirect to the HTTPS version of the site. Thus, these sites were being counted by the apache_vhosts plugin. But that is now fixed, and we have a graph!
https://alexschroeder.ch/pics/15960447830_a102ae6df7_o.png
https://alexschroeder.ch/pics/15528022193_78df3fc37d_o.png
I’m not quite sure how to compare this with the following log file analysis where I simply group access by site:
root@kallobombus:/var/log/apache2# perl /home/alex/bin/site-comparison < access.log
Site hits bandw. hits% interv. status code distrib.
www.emacswiki.org 104499 6K 47% 2.1s 301 (29%), 501 (26%), 200 (21%
www.alexschroeder.ch 67396 31K 30% 3.2s 200 (94%), 304 (2%), 301 (1%),
www.communitywiki.org 24574 17K 11% 8.9s 200 (80%), 301 (7%), 404 (3%),
www.campaignwiki.org 13902 13K 6% 15.6s 200 (74%), 404 (17%), 302 (4%)
www.oddmuse.org 9498 23K 4% 22.9s 200 (66%), 404 (14%), 301 (12%
www.arabisch-lernen.org 845 7K 0% 256.4s 200 (94%), 404 (3%), 403 (1%),
www.orientalisch.info 404 4K 0% 536.5s 200 (80%), 301 (14%), 404 (2%)
Ah, of course. The access.log is can be up to seven days. Thus:
root@kallobombus:/var/log/apache2# tail -n 10000 access.log | /home/alex/bin/site-comparison
Site hits bandw. hits% interv. status code distrib.
www.alexschroeder.ch 6430 24K 64% 1.9s 200 (88%), 301 (8%), 404 (1%),
www.emacswiki.org 1187 12K 11% 10.4s 301 (38%), 200 (36%), 404 (19%
www.communitywiki.org 1150 14K 11% 10.7s 200 (79%), 301 (7%), 404 (5%),
www.campaignwiki.org 604 9K 6% 20.3s 200 (58%), 301 (22%), 404 (12%
www.oddmuse.org 415 18K 4% 29.6s 200 (64%), 404 (18%), 301 (10%
www.arabisch-lernen.org 202 5K 2% 37.4s 404 (90%), 200 (9%), 501 (0%)
www.orientalisch.info 12 4K 0% 1010.2s 200 (91%), 304 (8%)
​#Emacs ​#Wikis ​#Oddmuse ​#mod perl ​#Apache ​#devops ​#Administration