I just noticed the following in my Munin:
What had happened between 11:25 and 11:35? I looked at `/var/log/syslog` and found the following:
Jun 8 11:30:25 sibirocobombus kernel: [5341224.320404] python3 invoked oom-killer: gfp_mask=0x24201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=0, order=0, oom_score_adj=0 Jun 8 11:30:26 sibirocobombus kernel: [5341224.320407] python3 cpuset=/ mems_allowed=0 Jun 8 11:30:26 sibirocobombus kernel: [5341224.320414] CPU: 0 PID: 6857 Comm: python3 Not tainted 4.9.0-3-amd64 #1 Debian 4.9.30-2+deb9u5 Jun 8 11:30:26 sibirocobombus kernel: [5341224.320415] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_083030-gandalf 04/01/2014 Jun 8 11:30:26 sibirocobombus kernel: [5341224.320418] 0000000000000000 ffffffffb9f285b4 ffffb9ccc195fc38 ffff8f1a486d1080 Jun 8 11:30:26 sibirocobombus kernel: [5341224.320421] ffffffffb9dfe020 0000000000000000 0000000000000000 0000000000000001 Jun 8 11:30:26 sibirocobombus kernel: [5341224.320424] ffffffffb9d844e7 0000004294fa8a80 ffffffffc0454695 0000000000000001 Jun 8 11:30:26 sibirocobombus kernel: [5341224.320427] Call Trace: Jun 8 11:30:26 sibirocobombus kernel: [5341224.320436] [<ffffffffb9f285b4>] ? dump_stack+0x5c/0x78 Jun 8 11:30:26 sibirocobombus kernel: [5341224.320439] [<ffffffffb9dfe020>] ? dump_header+0x78/0x1fd Jun 8 11:30:26 sibirocobombus kernel: [5341224.320443] [<ffffffffb9d844e7>] ? get_page_from_freelist+0x3f7/0xb40 Jun 8 11:30:26 sibirocobombus kernel: [5341224.320449] [<ffffffffc0454695>] ? virtballoon_oom_notify+0x25/0x70 [virtio_balloon] Jun 8 11:30:26 sibirocobombus kernel: [5341224.320453] [<ffffffffb9d8047a>] ? oom_kill_process+0x21a/0x3e0 Jun 8 11:30:26 sibirocobombus kernel: [5341224.320455] [<ffffffffb9d800fd>] ? oom_badness+0xed/0x170 Jun 8 11:30:26 sibirocobombus kernel: [5341224.320456] [<ffffffffb9d80911>] ? out_of_memory+0x111/0x470 Jun 8 11:30:26 sibirocobombus kernel: [5341224.320458] [<ffffffffb9d85b4f>] ? __alloc_pages_slowpath+0xb7f/0xbc0 Jun 8 11:30:26 sibirocobombus kernel: [5341224.320460] [<ffffffffb9d85d8e>] ? __alloc_pages_nodemask+0x1fe/0x260 Jun 8 11:30:26 sibirocobombus kernel: [5341224.320465] [<ffffffffb9dd5ec1>] ? alloc_pages_current+0x91/0x140 Jun 8 11:30:26 sibirocobombus kernel: [5341224.320467] [<ffffffffb9d7e940>] ? filemap_fault+0x300/0x5a0 Jun 8 11:30:26 sibirocobombus kernel: [5341224.324016] [<ffffffffc0383791>] ? ext4_filemap_fault+0x31/0x50 [ext4] Jun 8 11:30:26 sibirocobombus kernel: [5341224.324016] [<ffffffffb9daeef1>] ? __do_fault+0x81/0x170 Jun 8 11:30:26 sibirocobombus kernel: [5341224.324016] [<ffffffffb9db3671>] ? handle_mm_fault+0xdd1/0x1350 Jun 8 11:30:26 sibirocobombus kernel: [5341224.324016] [<ffffffffb9c24701>] ? __switch_to+0x2c1/0x6c0 Jun 8 11:30:26 sibirocobombus kernel: [5341224.324016] [<ffffffffb9c5fd84>] ? __do_page_fault+0x2a4/0x510 Jun 8 11:30:26 sibirocobombus kernel: [5341224.324016] [<ffffffffba207788>] ? async_page_fault+0x28/0x30 Jun 8 11:30:26 sibirocobombus kernel: [5341224.402865] Mem-Info: Jun 8 11:30:26 sibirocobombus kernel: [5341224.402875] active_anon:573808 inactive_anon:143778 isolated_anon:0 Jun 8 11:30:26 sibirocobombus kernel: [5341224.402875] active_file:186 inactive_file:96 isolated_file:37 Jun 8 11:30:26 sibirocobombus kernel: [5341224.402875] unevictable:0 dirty:0 writeback:0 unstable:0 Jun 8 11:30:26 sibirocobombus kernel: [5341224.402875] slab_reclaimable:3009 slab_unreclaimable:11993 Jun 8 11:30:26 sibirocobombus kernel: [5341224.402875] mapped:1216 shmem:8885 pagetables:12225 bounce:0 Jun 8 11:30:26 sibirocobombus kernel: [5341224.402875] free:14221 free_pcp:152 free_cma:0 Jun 8 11:30:26 sibirocobombus kernel: [5341224.402883] Node 0 active_anon:2295232kB inactive_anon:575112kB active_file:744kB inactive_file:384kB unevictable:0kB isolated(anon):0kB isolated(file):148kB mapped:4864kB dirty:0kB writeback:0kB shmem:35540kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB pages_scanned:2335 all_unreclaimable? yes Jun 8 11:30:26 sibirocobombus kernel: [5341224.402884] Node 0 DMA free:12124kB min:232kB low:288kB high:344kB active_anon:3688kB inactive_anon:68kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:12kB kernel_stack:0kB pagetables:4kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB Jun 8 11:30:26 sibirocobombus kernel: [5341224.402890] lowmem_reserve[]: 0 2975 2975 2975 2975 Jun 8 11:30:26 sibirocobombus kernel: [5341224.402894] Node 0 DMA32 free:44760kB min:44820kB low:56024kB high:67228kB active_anon:2291544kB inactive_anon:575044kB active_file:744kB inactive_file:384kB unevictable:0kB writepending:0kB present:3129216kB managed:3068860kB mlocked:0kB slab_reclaimable:12036kB slab_unreclaimable:47960kB kernel_stack:8352kB pagetables:48896kB bounce:0kB free_pcp:608kB local_pcp:608kB free_cma:0kB Jun 8 11:30:26 sibirocobombus kernel: [5341224.402899] lowmem_reserve[]: 0 0 0 0 0 Jun 8 11:30:26 sibirocobombus kernel: [5341224.402902] Node 0 DMA: 3*4kB (UH) 4*8kB (UMH) 1*16kB (H) 3*32kB (UH) 5*64kB (UMH) 3*128kB (UH) 2*256kB (UM) 1*512kB (U) 2*1024kB (UH) 2*2048kB (UM) 1*4096kB (E) = 12124kB Jun 8 11:30:26 sibirocobombus kernel: [5341224.402915] Node 0 DMA32: 6682*4kB (UMEH) 2054*8kB (UEH) 4*16kB (H) 2*32kB (H) 1*64kB (H) 1*128kB (H) 1*256kB (H) 2*512kB (H) 0*1024kB 0*2048kB 0*4096kB = 44760kB Jun 8 11:30:26 sibirocobombus kernel: [5341224.402927] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Jun 8 11:30:26 sibirocobombus kernel: [5341224.402928] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jun 8 11:30:26 sibirocobombus kernel: [5341224.402929] 14918 total pagecache pages Jun 8 11:30:26 sibirocobombus kernel: [5341224.402931] 5715 pages in swap cache Jun 8 11:30:26 sibirocobombus kernel: [5341224.402932] Swap cache stats: add 1066964, delete 1061249, find 206625603/206789816 Jun 8 11:30:26 sibirocobombus kernel: [5341224.402933] Free swap = 0kB Jun 8 11:30:26 sibirocobombus kernel: [5341224.402934] Total swap = 1022972kB Jun 8 11:30:26 sibirocobombus kernel: [5341224.402934] 786302 pages RAM Jun 8 11:30:26 sibirocobombus kernel: [5341224.402935] 0 pages HighMem/MovableOnly Jun 8 11:30:26 sibirocobombus kernel: [5341224.402936] 15110 pages reserved Jun 8 11:30:26 sibirocobombus kernel: [5341224.402937] 0 pages hwpoisoned Jun 8 11:30:26 sibirocobombus kernel: [5341224.402938] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Jun 8 11:30:26 sibirocobombus kernel: [5341224.402954] [ 188] 0 188 15502 780 27 3 77 0 systemd-journal Jun 8 11:30:26 sibirocobombus kernel: [5341224.402958] [ 376] 0 376 11625 55 26 3 86 0 systemd-logind Jun 8 11:30:26 sibirocobombus kernel: [5341224.402961] [ 381] 0 381 7416 47 20 3 28 0 cron Jun 8 11:30:26 sibirocobombus kernel: [5341224.402963] [ 387] 0 387 62529 478 30 3 77 0 rsyslogd Jun 8 11:30:26 sibirocobombus kernel: [5341224.402966] [ 388] 0 388 3152 9 12 3 37 0 rsync Jun 8 11:30:26 sibirocobombus kernel: [5341224.402969] [ 393] 0 393 1054 0 8 3 35 0 acpid Jun 8 11:30:26 sibirocobombus kernel: [5341224.402972] [ 394] 105 394 11283 60 27 3 78 -900 dbus-daemon Jun 8 11:30:26 sibirocobombus kernel: [5341224.402974] [ 450] 0 450 3634 0 12 3 38 0 agetty Jun 8 11:30:26 sibirocobombus kernel: [5341224.402976] [ 458] 0 458 28698 227 26 3 59 0 monit Jun 8 11:30:26 sibirocobombus kernel: [5341224.402979] [ 464] 0 464 17486 31 37 3 164 -1000 sshd Jun 8 11:30:26 sibirocobombus kernel: [5341224.402982] [ 614] 0 614 12949 927 30 3 1385 0 munin-node Jun 8 11:30:26 sibirocobombus kernel: [5341224.402984] [ 823] 107 823 27941 18 50 3 365 0 exim4 Jun 8 11:30:26 sibirocobombus kernel: [5341224.402987] [ 826] 0 826 73633 1101 44 3 1522 0 fail2ban-server Jun 8 11:30:26 sibirocobombus kernel: [5341224.402990] [ 1059] 1000 1059 27474 376 55 3 6919 0 /home/alex/camp Jun 8 11:30:26 sibirocobombus kernel: [5341224.402994] [22985] 1000 22985 74242 2 130 3 23696 0 /home/alex/farm Jun 8 11:30:26 sibirocobombus kernel: [5341224.402997] [23038] 1000 23038 72562 1 126 3 22595 0 /home/alex/farm Jun 8 11:30:26 sibirocobombus kernel: [5341224.403000] [22174] 1000 22174 75475 1 135 3 25314 0 /home/alex/farm Jun 8 11:30:26 sibirocobombus kernel: [5341224.403003] [10368] 100 10368 31821 17 32 3 116 0 systemd-timesyn Jun 8 11:30:26 sibirocobombus kernel: [5341224.403005] [10648] 0 10648 11308 11 24 3 92 -1000 systemd-udevd Jun 8 11:30:26 sibirocobombus kernel: [5341224.403008] [12215] 0 12215 26437 794 56 3 47 0 apache2 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403011] [15822] 1000 15822 77511 1 135 3 26522 0 /home/alex/farm Jun 8 11:30:26 sibirocobombus kernel: [5341224.403014] [ 5025] 1000 5025 76563 2636 134 4 22902 0 /home/alex/farm Jun 8 11:30:26 sibirocobombus kernel: [5341224.403017] [ 5955] 1000 5955 79129 2718 139 4 25356 0 /home/alex/farm Jun 8 11:30:26 sibirocobombus kernel: [5341224.403020] [ 6554] 1000 6554 87614 3839 156 4 32240 0 /home/alex/farm Jun 8 11:30:26 sibirocobombus kernel: [5341224.403022] [ 6845] 1000 6845 87597 7559 156 4 28512 0 /home/alex/farm Jun 8 11:30:26 sibirocobombus kernel: [5341224.403026] [ 6930] 1000 6930 81413 6399 144 4 23554 0 /home/alex/farm Jun 8 11:30:26 sibirocobombus kernel: [5341224.403029] [ 7143] 1000 7143 87659 7600 156 4 28482 0 /home/alex/farm Jun 8 11:30:26 sibirocobombus kernel: [5341224.403031] [ 7564] 1000 7564 75261 1351 132 4 22884 0 /home/alex/farm Jun 8 11:30:26 sibirocobombus kernel: [5341224.403034] [ 7610] 1000 7610 78205 4910 138 4 21774 0 /home/alex/farm Jun 8 11:30:26 sibirocobombus kernel: [5341224.403037] [ 8548] 1000 8548 73994 624 129 3 23285 0 /home/alex/farm Jun 8 11:30:26 sibirocobombus kernel: [5341224.403039] [ 8960] 1000 8960 74356 658 131 3 22646 0 /home/alex/farm Jun 8 11:30:26 sibirocobombus kernel: [5341224.403043] [ 9075] 1000 9075 72920 686 127 3 22198 0 /home/alex/farm Jun 8 11:30:26 sibirocobombus kernel: [5341224.403045] [ 9174] 1000 9174 72909 635 127 3 22213 0 /home/alex/farm Jun 8 11:30:26 sibirocobombus kernel: [5341224.403048] [10473] 1000 10473 73076 11977 126 3 11083 0 /home/alex/farm Jun 8 11:30:26 sibirocobombus kernel: [5341224.403052] [10295] 1000 10295 76489 21414 134 4 4042 0 /home/alex/farm Jun 8 11:30:26 sibirocobombus kernel: [5341224.403054] [10296] 1000 10296 78132 24544 137 4 2560 0 /home/alex/farm Jun 8 11:30:26 sibirocobombus kernel: [5341224.403057] [10300] 1000 10300 78133 24292 138 4 2812 0 /home/alex/farm Jun 8 11:30:26 sibirocobombus kernel: [5341224.403060] [10436] 1000 10436 76499 20586 134 4 4887 0 /home/alex/farm Jun 8 11:30:26 sibirocobombus kernel: [5341224.403062] [18588] 1000 18588 75233 24172 129 3 0 0 /home/alex/farm Jun 8 11:30:26 sibirocobombus kernel: [5341224.403065] [14367] 1000 14367 96991 45941 170 3 0 0 /home/alex/farm Jun 8 11:30:26 sibirocobombus kernel: [5341224.403067] [ 3040] 1000 3040 27474 3870 54 3 3562 0 /home/alex/camp Jun 8 11:30:26 sibirocobombus kernel: [5341224.403069] [ 4378] 1000 4378 69698 18695 125 4 0 0 /home/alex/farm Jun 8 11:30:26 sibirocobombus kernel: [5341224.403072] [15022] 65534 15022 12598 2534 27 4 0 0 python Jun 8 11:30:26 sibirocobombus kernel: [5341224.403074] [16776] 1000 16776 76163 25107 136 4 0 0 /home/alex/farm Jun 8 11:30:26 sibirocobombus kernel: [5341224.403076] [13061] 1000 13061 79048 28907 143 4 0 0 /home/alex/farm Jun 8 11:30:26 sibirocobombus kernel: [5341224.403079] [ 2928] 1000 2928 72276 22294 123 4 0 0 /home/alex/farm Jun 8 11:30:26 sibirocobombus kernel: [5341224.403082] [ 3415] 33 3415 26311 733 53 3 47 0 apache2 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403084] [ 3526] 33 3526 216362 3392 100 4 46 0 apache2 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403086] [ 3661] 1000 3661 20782 4802 43 3 0 0 perl Jun 8 11:30:26 sibirocobombus kernel: [5341224.403089] [ 4308] 1000 4308 17017 3773 37 3 0 0 perl Jun 8 11:30:26 sibirocobombus kernel: [5341224.403091] [23738] 1000 23738 72298 22266 125 4 0 0 /home/alex/farm Jun 8 11:30:26 sibirocobombus kernel: [5341224.403094] [ 5556] 33 5556 214419 1979 95 4 46 0 apache2 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403096] [ 5588] 33 5588 214589 2148 95 4 46 0 apache2 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403099] [ 6431] 33 6431 35599 106 22 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403101] [ 6432] 33 6432 17320 5307 40 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403103] [ 6437] 33 6437 35599 107 22 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403105] [ 6438] 33 6438 17090 5076 38 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403107] [ 6462] 33 6462 35599 106 21 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403110] [ 6464] 33 6464 214782 1522 95 4 47 0 apache2 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403112] [ 6493] 33 6493 12016 4372 30 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403114] [ 6495] 33 6495 35599 106 23 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403117] [ 6500] 33 6500 35599 106 23 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403119] [ 6507] 33 6507 12016 4378 28 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403121] [ 6508] 33 6508 35599 106 22 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403123] [ 6509] 33 6509 35599 106 20 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403126] [ 6513] 33 6513 12016 4371 29 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403128] [ 6517] 33 6517 215340 2016 97 4 46 0 apache2 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403131] [ 6521] 33 6521 35599 107 21 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403133] [ 6524] 33 6524 11994 4306 29 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403135] [ 6555] 33 6555 12016 4339 29 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403137] [ 6556] 33 6556 35599 106 22 4 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403139] [ 6562] 33 6562 35599 106 20 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403142] [ 6568] 33 6568 12019 4338 25 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403144] [ 6570] 33 6570 35599 107 22 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403146] [ 6574] 33 6574 12017 4337 29 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403148] [ 6575] 33 6575 11917 4264 29 4 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403151] [ 6591] 33 6591 11917 4236 29 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403153] [ 6599] 33 6599 35599 106 21 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403155] [ 6600] 33 6600 35599 107 22 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403157] [ 6602] 33 6602 215282 1961 97 4 47 0 apache2 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403160] [ 6603] 33 6603 35599 107 21 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403162] [ 6611] 33 6611 35599 106 20 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403164] [ 6634] 33 6634 11833 4170 27 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403166] [ 6637] 33 6637 11821 4380 28 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403169] [ 6643] 33 6643 35599 106 21 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403171] [ 6645] 33 6645 11856 4174 28 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403173] [ 6649] 33 6649 11833 4168 27 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403176] [ 6663] 33 6663 11853 4235 29 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403178] [ 6672] 33 6672 35599 106 22 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403180] [ 6678] 33 6678 11854 4170 28 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403183] [ 6690] 33 6690 35599 106 23 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403185] [ 6693] 33 6693 35599 106 24 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403187] [ 6696] 33 6696 35599 107 21 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403190] [ 6698] 33 6698 11692 4062 28 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403192] [ 6703] 33 6703 11660 4040 28 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403194] [ 6709] 33 6709 11596 3984 27 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403196] [ 6722] 33 6722 35599 106 22 4 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403199] [ 6727] 33 6727 35599 107 21 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403201] [ 6730] 33 6730 11560 3906 28 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403204] [ 6734] 33 6734 11560 3906 28 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403206] [ 6744] 33 6744 35599 107 23 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403208] [ 6750] 33 6750 11204 3558 25 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403210] [ 6751] 33 6751 35599 106 23 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403212] [ 6752] 33 6752 35599 106 21 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403214] [ 6753] 33 6753 35599 106 22 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403217] [ 6756] 33 6756 11305 3638 28 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403219] [ 6758] 33 6758 35599 106 21 4 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403221] [ 6760] 33 6760 11564 3903 28 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403223] [ 6763] 33 6763 11304 3650 28 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403225] [ 6764] 33 6764 35599 107 20 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403227] [ 6768] 33 6768 35599 106 21 4 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403238] [ 6769] 33 6769 35599 107 23 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403240] [ 6773] 33 6773 11139 3464 26 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403242] [ 6776] 33 6776 35599 105 23 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403244] [ 6778] 33 6778 11172 3530 27 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403246] [ 6779] 33 6779 11109 3498 28 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403248] [ 6781] 33 6781 35599 106 25 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403251] [ 6787] 33 6787 35599 106 22 4 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403254] [ 6788] 33 6788 35599 106 22 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403256] [ 6792] 33 6792 11139 3463 28 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403258] [ 6793] 33 6793 11108 3463 26 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403261] [ 6796] 33 6796 35599 106 23 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403263] [ 6797] 33 6797 11139 3464 27 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403265] [ 6802] 33 6802 11108 3464 25 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403267] [ 6803] 33 6803 35599 107 22 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403269] [ 6804] 33 6804 35599 105 22 4 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403271] [ 6805] 33 6805 11140 3464 26 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403273] [ 6815] 33 6815 35599 106 22 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403275] [ 6821] 33 6821 35599 105 21 4 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403278] [ 6822] 33 6822 35599 106 22 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403281] [ 6823] 33 6823 11139 3475 27 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403284] [ 6826] 33 6826 11044 3458 26 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403287] [ 6829] 33 6829 11075 3455 26 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403290] [ 6833] 33 6833 35599 105 22 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403294] [ 6839] 33 6839 35599 107 22 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403296] [ 6842] 33 6842 11075 3456 28 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403298] [ 6843] 33 6843 11075 3399 26 4 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403301] [ 6849] 33 6849 35599 107 20 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403303] [ 6850] 33 6850 35599 105 22 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403305] [ 6851] 33 6851 11139 3465 27 4 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403307] [ 6853] 33 6853 11075 3457 25 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403310] [ 6857] 33 6857 11075 3457 25 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403312] [ 6862] 33 6862 35599 106 21 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403314] [ 6872] 33 6872 35599 107 23 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403316] [ 6876] 33 6876 11044 3457 25 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403319] [ 6877] 33 6877 11075 3427 25 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403321] [ 6881] 33 6881 35599 106 21 4 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403324] [ 6882] 33 6882 35599 106 23 4 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403326] [ 6884] 33 6884 11075 3399 28 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403328] [ 6890] 33 6890 35599 105 21 4 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403331] [ 6891] 33 6891 35599 105 22 4 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403333] [ 6895] 33 6895 11011 3352 25 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403335] [ 6898] 33 6898 35599 107 22 4 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403337] [ 6900] 33 6900 35599 106 21 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403340] [ 6901] 33 6901 35599 107 22 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403342] [ 6902] 33 6902 11142 3467 27 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403345] [ 6903] 33 6903 11075 3399 27 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403347] [ 6905] 33 6905 11075 3400 25 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403349] [ 6907] 33 6907 10980 3397 27 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403352] [ 6911] 33 6911 35599 106 23 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403354] [ 6912] 33 6912 35599 107 22 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403356] [ 6918] 33 6918 11015 3401 25 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403359] [ 6920] 33 6920 11011 3349 27 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403361] [ 6925] 33 6925 35599 106 22 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403363] [ 6926] 33 6926 35599 107 20 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403365] [ 6933] 33 6933 10980 3351 27 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403368] [ 6935] 33 6935 11014 3351 26 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403369] [ 6940] 33 6940 11011 3351 25 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403372] [ 6942] 33 6942 35599 106 22 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403374] [ 6943] 33 6943 35599 106 21 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403376] [ 6944] 33 6944 35599 105 21 4 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403379] [ 6952] 33 6952 35599 106 22 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403381] [ 6953] 33 6953 35599 107 23 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403383] [ 6959] 33 6959 10980 3352 25 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403386] [ 6960] 33 6960 10768 3138 25 4 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403388] [ 6964] 33 6964 11075 3427 25 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403391] [ 6968] 33 6968 35599 105 21 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403393] [ 6969] 33 6969 35599 106 22 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403396] [ 6970] 33 6970 35599 106 23 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403398] [ 6972] 33 6972 11012 3351 27 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403400] [ 6975] 33 6975 10768 3132 27 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403402] [ 6977] 33 6977 35599 107 22 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403405] [ 6984] 33 6984 35599 106 22 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403407] [ 6985] 33 6985 10912 3254 26 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403409] [ 6989] 33 6989 10767 3087 26 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403411] [ 6990] 33 6990 10771 3091 27 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403414] [ 6992] 33 6992 35599 107 22 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403417] [ 6993] 33 6993 35599 107 24 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403419] [ 6994] 33 6994 35599 106 23 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403421] [ 7000] 33 7000 10767 3131 25 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403423] [ 7004] 33 7004 10814 3155 27 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403426] [ 7008] 33 7008 35599 107 21 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403428] [ 7009] 33 7009 10814 3150 27 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403431] [ 7014] 33 7014 10768 3087 26 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403433] [ 7019] 33 7019 10751 3087 27 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403435] [ 7023] 33 7023 10915 3245 27 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403437] [ 7024] 33 7024 10984 3353 26 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403439] [ 7025] 33 7025 35599 106 22 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403442] [ 7026] 33 7026 35599 107 21 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403445] [ 7027] 33 7027 35599 105 22 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403447] [ 7028] 33 7028 35599 106 22 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403449] [ 7029] 33 7029 35599 107 22 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403451] [ 7030] 33 7030 35599 106 23 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403454] [ 7031] 33 7031 35599 106 24 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403456] [ 7041] 33 7041 10735 3102 25 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403458] [ 7043] 33 7043 10768 3139 26 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403460] [ 7046] 33 7046 10687 3081 26 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403463] [ 7052] 33 7052 10688 3081 26 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403466] [ 7059] 33 7059 35599 107 20 4 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403469] [ 7060] 33 7060 35599 107 21 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403472] [ 7066] 33 7066 10671 3086 25 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403475] [ 7069] 33 7069 10735 3086 25 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403479] [ 7070] 33 7070 10703 3071 25 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403482] [ 7072] 33 7072 35599 107 23 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403485] [ 7073] 33 7073 35599 107 23 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403487] [ 7082] 33 7082 10674 3075 24 4 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403490] [ 7083] 33 7083 10704 3022 27 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403492] [ 7087] 33 7087 35599 107 21 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403494] [ 7088] 33 7088 35599 107 22 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403497] [ 7089] 33 7089 35599 106 21 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403499] [ 7090] 33 7090 10691 3023 26 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403501] [ 7098] 33 7098 10704 3081 26 4 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403503] [ 7099] 33 7099 10687 3054 24 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403505] [ 7105] 33 7105 35599 107 21 4 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403508] [ 7106] 33 7106 35599 106 22 4 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403510] [ 7107] 33 7107 10688 3048 26 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403513] [ 7114] 33 7114 10639 3007 26 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403515] [ 7116] 33 7116 35599 107 22 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403517] [ 7117] 33 7117 35599 106 23 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403519] [ 7124] 33 7124 10623 2954 25 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403521] [ 7125] 33 7125 10690 3058 25 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403523] [ 7126] 33 7126 10688 3059 27 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403525] [ 7130] 33 7130 10704 3022 25 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403527] [ 7134] 33 7134 35599 106 20 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403530] [ 7135] 33 7135 35599 107 22 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403532] [ 7140] 33 7140 10704 3021 27 4 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403535] [ 7145] 33 7145 35599 106 20 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403537] [ 7146] 33 7146 35599 106 22 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403539] [ 7152] 33 7152 10607 3006 24 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403541] [ 7156] 33 7156 35599 107 23 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403544] [ 7157] 33 7157 35599 107 21 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403546] [ 7160] 33 7160 10687 3020 24 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403547] [ 7165] 1000 7165 21621 6821 46 3 0 0 perl Jun 8 11:30:26 sibirocobombus kernel: [5341224.403550] [ 7171] 33 7171 10703 3042 26 4 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403552] [ 7172] 33 7172 35599 106 24 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403555] [ 7173] 33 7173 35599 107 20 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403557] [ 7180] 33 7180 10607 3006 25 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403559] [ 7184] 33 7184 35599 105 23 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403561] [ 7185] 33 7185 35599 106 24 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403564] [ 7186] 33 7186 35599 106 23 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403566] [ 7188] 0 7188 12206 96 29 3 9 0 cron Jun 8 11:30:26 sibirocobombus kernel: [5341224.403569] [ 7195] 33 7195 10469 2825 26 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403571] [ 7198] 33 7198 10639 2954 27 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403574] [ 7199] 33 7199 10557 2885 26 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403576] [ 7200] 33 7200 35599 106 23 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403579] [ 7201] 33 7201 35599 105 20 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403582] [ 7205] 33 7205 10595 2958 27 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403584] [ 7207] 33 7207 35599 106 22 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403586] [ 7208] 33 7208 35599 107 23 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403589] [ 7209] 108 7209 1072 18 8 3 0 0 sh Jun 8 11:30:26 sibirocobombus kernel: [5341224.403591] [ 7210] 33 7210 10525 2880 24 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403593] [ 7212] 33 7212 10590 2952 25 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403595] [ 7214] 33 7214 10557 2888 26 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403597] [ 7215] 108 7215 1072 18 8 3 0 0 munin-cron Jun 8 11:30:26 sibirocobombus kernel: [5341224.403599] [ 7220] 33 7220 10571 2888 24 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403601] [ 7225] 108 7225 10477 2549 24 3 0 0 munin-update Jun 8 11:30:26 sibirocobombus kernel: [5341224.403604] [ 7226] 33 7226 10571 2940 26 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403605] [ 7227] 33 7227 35599 106 22 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403607] [ 7228] 33 7228 35599 105 23 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403609] [ 7229] 33 7229 35599 107 22 4 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403612] [ 7231] 33 7231 10557 2889 26 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403615] [ 7240] 33 7240 10433 2758 26 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403617] [ 7246] 33 7246 35599 106 21 4 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403620] [ 7255] 33 7255 10227 2558 25 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403622] [ 7265] 33 7265 10460 2757 25 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403624] [ 7268] 33 7268 10229 2564 24 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403626] [ 7275] 33 7275 35599 105 22 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403629] [ 7276] 33 7276 35599 106 22 4 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403631] [ 7277] 33 7277 10227 2587 22 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403633] [ 7289] 33 7289 10227 2558 24 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403634] [ 7291] 33 7291 10227 2563 26 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403637] [ 7299] 33 7299 35599 107 21 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403639] [ 7300] 33 7300 10032 2364 23 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403641] [ 7302] 33 7302 35599 106 23 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403643] [ 7306] 33 7306 9830 2201 22 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403646] [ 7311] 1000 7311 73390 22892 129 4 0 0 /home/alex/farm Jun 8 11:30:26 sibirocobombus kernel: [5341224.403648] [ 7313] 1000 7313 77513 25432 136 4 0 0 /home/alex/farm Jun 8 11:30:26 sibirocobombus kernel: [5341224.403650] [ 7314] 1000 7314 79566 28908 141 4 0 0 /home/alex/farm Jun 8 11:30:26 sibirocobombus kernel: [5341224.403652] [ 7315] 1000 7315 76681 25114 135 4 0 0 /home/alex/farm Jun 8 11:30:26 sibirocobombus kernel: [5341224.403654] [ 7316] 1000 7316 79566 28909 141 4 0 0 /home/alex/farm Jun 8 11:30:26 sibirocobombus kernel: [5341224.403657] [ 7317] 33 7317 35599 107 22 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403659] [ 7318] 33 7318 8943 1308 21 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403662] [ 7320] 33 7320 35599 106 23 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403664] [ 7324] 33 7324 8940 1302 22 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403666] [ 7329] 1000 7329 72816 22268 128 4 0 0 /home/alex/farm Jun 8 11:30:26 sibirocobombus kernel: [5341224.403669] [ 7330] 1000 7330 72794 22296 128 4 0 0 /home/alex/farm Jun 8 11:30:26 sibirocobombus kernel: [5341224.403671] [ 7331] 1000 7331 18462 4678 39 3 0 0 perl Jun 8 11:30:26 sibirocobombus kernel: [5341224.403673] [ 7332] 1000 7332 76681 25109 135 4 0 0 /home/alex/farm Jun 8 11:30:26 sibirocobombus kernel: [5341224.403676] [ 7333] 1000 7333 18428 4665 39 3 0 0 perl Jun 8 11:30:26 sibirocobombus kernel: [5341224.403678] [ 7334] 1000 7334 72816 22268 128 4 0 0 /home/alex/farm Jun 8 11:30:26 sibirocobombus kernel: [5341224.403680] [ 7335] 33 7335 35599 106 22 3 0 0 cgit.cgi Jun 8 11:30:26 sibirocobombus kernel: [5341224.403682] [ 7336] 1000 7336 76163 25107 130 4 0 0 /home/alex/farm Jun 8 11:30:26 sibirocobombus kernel: [5341224.403684] [ 7337] 33 7337 5848 283 16 3 0 0 python3 Jun 8 11:30:26 sibirocobombus kernel: [5341224.403686] Out of memory: Kill process 14367 (/home/alex/farm) score 44 or sacrifice child Jun 8 11:30:26 sibirocobombus kernel: [5341224.406828] Killed process 14367 (/home/alex/farm) total-vm:387964kB, anon-rss:183760kB, file-rss:4kB, shmem-rss:0kB
Grepping for “Out of memory” gives me the following:
Jun 8 11:31:10 sibirocobombus kernel: [5341254.428514] Out of memory: Kill process 7143 (/home/alex/farm) score 35 or sacrifice child Jun 8 11:31:13 sibirocobombus kernel: [5341254.487951] Out of memory: Kill process 6554 (/home/alex/farm) score 35 or sacrifice child Jun 8 11:31:22 sibirocobombus kernel: [5341281.999273] Out of memory: Kill process 6845 (/home/alex/farm) score 35 or sacrifice child Jun 8 11:31:40 sibirocobombus kernel: [5341298.779134] Out of memory: Kill process 6930 (/home/alex/farm) score 29 or sacrifice child Jun 8 11:31:44 sibirocobombus kernel: [5341303.361810] Out of memory: Kill process 7316 (/home/alex/farm) score 28 or sacrifice child Jun 8 11:31:47 sibirocobombus kernel: [5341306.283764] Out of memory: Kill process 13061 (/home/alex/farm) score 28 or sacrifice child Jun 8 11:31:51 sibirocobombus kernel: [5341310.597420] Out of memory: Kill process 5955 (/home/alex/farm) score 27 or sacrifice child Jun 8 11:31:58 sibirocobombus kernel: [5341317.593756] Out of memory: Kill process 10300 (/home/alex/farm) score 26 or sacrifice child Jun 8 11:32:03 sibirocobombus kernel: [5341322.250094] Out of memory: Kill process 10296 (/home/alex/farm) score 26 or sacrifice child Jun 8 11:32:10 sibirocobombus kernel: [5341328.457769] Out of memory: Kill process 7610 (/home/alex/farm) score 26 or sacrifice child Jun 8 11:32:10 sibirocobombus kernel: [5341328.803566] Out of memory: Kill process 15822 (/home/alex/farm) score 25 or sacrifice child Jun 8 11:32:10 sibirocobombus kernel: [5341329.535406] Out of memory: Kill process 5025 (/home/alex/farm) score 25 or sacrifice child Jun 8 11:32:36 sibirocobombus kernel: [5341354.585875] Out of memory: Kill process 10436 (/home/alex/farm) score 24 or sacrifice child Jun 8 11:32:46 sibirocobombus kernel: [5341364.953989] Out of memory: Kill process 10295 (/home/alex/farm) score 24 or sacrifice child Jun 8 11:33:08 sibirocobombus kernel: [5341386.209522] Out of memory: Kill process 7344 (/home/alex/farm) score 25 or sacrifice child Jun 8 11:33:10 sibirocobombus kernel: [5341389.336958] Out of memory: Kill process 7336 (/home/alex/farm) score 24 or sacrifice child Jun 8 11:33:15 sibirocobombus kernel: [5341394.201015] Out of memory: Kill process 22174 (/home/alex/farm) score 24 or sacrifice child
OK, so something was definitely hammering the site. Some idiot web spider, perhaps? I mean, mostly my Toadfarm processes were killed. These are the ones serving my wikis. But when I look at the syslog, I see that there is a surprising number of python3 and cgit processes in the list. I just installed cgit yesterday as part of my move away from GitHub. Did somebody try to spider these?
I definitely need to do the following:
1. ban the idiot
2. make sure cgit only lists to one hostname
3. make sure `robots.txt` prevents spiders from indexing cgit output
Then again, there seems to be no obvious idiot, here. Just small numbers of hits in these minutes.
grep "2018:11:[23]" /var/log/apache2/access.log | /home/alex/bin/leech-detector | head
But here’s an interesting finding:
cat /var/log/apache2/access.log | grep cgit | /home/alex/bin/time-grouping
08/Jun/2018:14 2 0%
08/Jun/2018:11 186 91%
08/Jun/2018:09 1 0%
08/Jun/2018:08 9 4%
08/Jun/2018:07 6 2%
At the hour I’m interested in, I got 186 hits for cgit. A visual inspection showed a lot of Mastodon instances!
cat /var/log/apache2/access.log | grep "cgit.*Mastodon" | wc -l 177
I had tooted something about self-hosting git and provided an example URL, and every instance that got the toot probably tried to verify that the link worked, or tried to fetch a preview window, or something. And since cgit installed as CGI script is slow, that caused the problem!
CGI scripts are optimized for not running at all: if you don’t call them, they don’t use any CPU cycles nor memory, unlike Toadfarm and similar setups.
This also shows that editing `robots.txt` won’t help:
cat /var/log/apache2/access.log | grep "robots.*Mastodon" | wc -l 0
So... some sort of User Agent test which will block access? Let’s edit `/etc/apache2/conf-enabled/cgit.conf` and add a rewrite rule:
ScriptAlias /cgit/ "/usr/lib/cgit/cgit.cgi/"
RedirectMatch ^/cgit$ /cgit/
Alias /cgit-css "/usr/share/cgit/"
<Directory "/usr/lib/cgit/">
AllowOverride None
Options ExecCGI FollowSymlinks
Require all granted
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} Mastodon
RewriteRule .* - [redirect=403,last]
</Directory>
Or, to make links in the `README.md` of the about section work (if your links to directories end with a slash, in any case):
ScriptAlias /cgit/ "/usr/lib/cgit/cgit.cgi/"
RedirectMatch ^/cgit$ /cgit/
Alias /cgit-css "/usr/share/cgit/"
<Directory "/usr/lib/cgit/">
AllowOverride None
Options ExecCGI FollowSymlinks
Require all granted
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} Mastodon
RewriteRule .* - [redirect=403,last]
RewriteRule /(.*)/about/(..*)/$ /cgit/$1/tree/$2 [redirect,last]
RewriteRule /(.*)/about/(..*)$ /cgit/$1/plain/$2 [redirect,last]
</Directory>
#Administration
(Please contact me if you want to remove your comment.)
⁂
This has been reported https://github.com/tootsuite/mastodon/issues/4486 but it is not obvious how to address it.
https://github.com/tootsuite/mastodon/issues/4486
– Stéphane Bortzmeyer 2018-06-08 13:54 UTC
---
Yeah, I think I will have to protect cgit from Mastodon or I’ll never be able to link to cgit. 😄
– Alex Schroeder 2018-06-08 15:17 UTC
---
FYI, this post shows up as “out of memory” in your RSS feed.
– Andreas Gohr 2018-06-08 18:37 UTC
---
Yeah, makes sense. 😄
– Alex Schroeder 2018-06-08 20:47 UTC