2016-05-31 Monit strangeness

I’m using Toadfarm on two machines. I can start the farm using my own username, and monitor it using Monit, but I can’t start or stop the farm using Monit – even though I use “as uid” and “as gid”:

check process toadfarm with pidfile /home/kensanata/farm/farm.pid
    start program = "/home/kensanata/farm/farm start" as uid "kensanata" and gid "kensanata"
    stop program = "/home/kensanata/farm/farm stop"
    ...

Weird.

At first, I thought that this actually worked on the older machine, but as it turns out, it never worked. I wonder why.

I rewrote it as follows:

check process toadfarm with pidfile /home/kensanata/farm/farm.pid
    start program = "/bin/bash -c '/home/kensanata/farm/farm start &> /home/kensanata/farm/start.log'" as uid kensanata
    stop program = "/bin/bash -c '/home/kensanata/farm/farm stop &> /home/kensanata/farm/stop.log'" as uid kensanata
    ...

The log file revealed that the `hypnotoad` executable was not on the path. So I added the following to my farm file:

$ENV{PATH} .= ":/usr/local/bin";

Now it will start – but not serve! I’m assuming this has to do with permissions somehow?

I think I’m giving up...

kensanata@po6:~/farm$ cat start.log
Server available at http://www.emacswiki.org:8081
farm (17832) started.

Testing:

alex@Megabombus:~$ curl http://www.emacswiki.org:8081/wiki
curl: (52) Empty reply from server

It takes a bit longer than 15s... Looks like an Oddmuse timeout?

I give up. I even added a call to `id`:

kensanata@po6:~/farm$ cat start.log
uid=1001(kensanata) gid=1001(kensanata) groups=1001(kensanata),0(root)
Server available at http://www.emacswiki.org:8081
farm (18294) started.

It looks good, but it doesn’t work.

I tried to make sure that I’m not using any relative paths. Log file, script, everything is getting absolute paths. And I’ve made some progress. I’m finally seeing some Mojolicious logs:

kensanata@po6:~/farm$ cat farm.log
[Wed Jun  1 00:14:25 2016] [info] Mounting emacs with conditions
[Wed Jun  1 00:14:25 2016] [info] Loading plugin Toadfarm::Plugin::AccessLog
[Wed Jun  1 00:14:26 2016] [info] Mounting emacs with conditions
[Wed Jun  1 00:14:26 2016] [info] Loading plugin Toadfarm::Plugin::AccessLog
[Wed Jun  1 00:14:26 2016] [info] Listening at "http://www.emacswiki.org:8081"
[Wed Jun  1 00:14:26 2016] [info] Manager 19291 started
[Wed Jun  1 00:14:26 2016] [debug] Worker 19292 started
[Wed Jun  1 00:14:26 2016] [debug] Worker 19293 started
[Wed Jun  1 00:14:26 2016] [info] Creating process id file "/home/kensanata/farm/farm.pid"
[Wed Jun  1 00:14:26 2016] [debug] Worker 19295 started
[Wed Jun  1 00:14:26 2016] [debug] Worker 19294 started
[Wed Jun  1 00:14:36 2016] [debug] GET "/wiki"
[Wed Jun  1 00:14:36 2016] [debug] Routing to application "Mojolicious::Lite"
[Wed Jun  1 00:14:36 2016] [debug] Routing to a callback
[Wed Jun  1 00:14:36 2016] [debug] [CGI:CODE(0x2740360):19299] START /home/kensanata/farm/wiki.pl
[Wed Jun  1 00:14:36 2016] [error] STDERR: : Permission denied at /usr/local/share/perl/5.18.2/Mojolicious/Plugin/CGI.pm line 62.
...

I wonder what this is.

The code in question:

 open STDERR, $STDERR[0], $STDERR[1] or die "STDERR: @$stderr: $!";

Ah! There was an error logfile with not absolute path... Good, fixed, works!

So now, this is how Monit starts the Toadfarm:

kensanata@po6:~/farm$ sudo cat /etc/monit/conf.d/toadfarm.conf
check process toadfarm with pidfile /home/kensanata/farm/farm.pid
    start program = "/home/kensanata/farm/farm start" as uid www-data and gid www-data
    stop program = "/home/kensanata/farm/farm stop" as uid www-data and gid www-data
    if failed host www.emacswiki.org port 8081 type tcp protocol http
      and request "/wiki" for 5 cycles then restart
    if totalmem > 500 MB for 5 cycles then restart
    if 3 restarts within 15 cycles then timeout

This defines the Toadfarm:

kensanata@po6:~/farm$ cat farm
#! /usr/bin/perl
use Toadfarm -init;

$ENV{PATH} .= ":/usr/local/bin";

logging {
  combined => 1,
  file     => "/home/kensanata/farm/farm.log",
  level    => "debug",
};

my $farm = '/home/kensanata/farm';

mount "$farm/emacs.pl" => {
  "Host" => qr{www.emacswiki.org:8081$},
  mount_point => '/wiki',
};

plugin "Toadfarm::Plugin::AccessLog";

start ["http://www.emacswiki.org:8081"], {proxy => 1}; # needs to be at the last line

(Must change “debug” back to “warn”!)

The wiki itself:

kensanata@po6:~/farm$ cat emacs.pl
#! /usr/bin/env perl
use Mojolicious::Lite;

plugin CGI => {
  support_semicolon_in_query_string => 1,
};

plugin CGI => {
  route => '/',
  script => '/home/kensanata/farm/wiki.pl',
  run => \&OddMuse::DoWikiRequest,
  before => sub {
      my $c = shift;
      $OddMuse::RunCGI = 0;
      $OddMuse::DataDir = '/home/kensanata/emacswiki';
      require '/home/kensanata/farm/wiki.pl' unless defined &OddMuse::DoWikiRequest;
  },
  env => {},
  errlog => '/home/kensanata/farm/emacs.log', # path to where STDERR from cgi script goes
};

app->start;

Note all the absolute paths. 🙂

This server is now also running a Toadfarm started by Monit. Yay!

Don’t forget permissions: key files and directories need to belong to `www-data.kensanata` (my user’s group so that I can also edit these files).

1. the farm directory needs to be writable by `www-data` so that the pid file can be written; without it, Monit can start the toadfarm but will still claim it failed (pid file is missing) and stopping it will also fail (use `ps fax` to figure out which process to `sudo kill`)

2. the farm log needs to be writable by `www-data` so that the toadfarm can log

3. the application log file needs to be writable by `www-data` so that the wiki itself can log

To wit:

kensanata@po6:~$ ls -la farm
total 1368
drwxrwxr-x  2 www-data  kensanata    4096 Jun  2 17:00 .
drwxr-xr-x 12 kensanata kensanata    4096 Jun  1 00:24 ..
-rw-r--r--  1 www-data  kensanata   27935 Jun  2 16:34 emacs.log
-rw-rw-r--  1 kensanata kensanata     549 Jun  1 00:22 emacs.pl
-rw-rw-r--  1 kensanata kensanata     528 Jun  1 00:12 emacs.pl~
-rwxrwxr-x  1 kensanata kensanata     432 Jun  2 16:32 farm
-rwxrwxr-x  1 kensanata kensanata     433 Jun  1 00:13 farm~
-rw-r--r--  1 www-data  kensanata 1002606 Jun  2 16:34 farm.log
-rw-r--r--  1 www-data  www-data        5 Jun  2 17:00 farm.pid
-rw-rw-r--  1 kensanata kensanata     322 May 29 12:01 Makefile
-rw-rw-r--  1 kensanata kensanata     384 May 29 11:59 Makefile~
-rwxrwxr-x  1 kensanata kensanata  159959 Jun  1 00:10 wiki.pl
-rwxrwxr-x  1 kensanata kensanata  159988 Jun  1 00:07 wiki.pl~

And why is that? Because I want the process to have few permissions elsewhere, and the existing wiki is being run as a CGI script by Apache using `www-data` which is why the data directory has the same ownership – owned by user `www-data` and writable by a group I belong to.

​#Administration ​#Monit