Sibirocobombus Setup
I’m paying the systemd tax, twice. I had a Debian Wheezy (7) machine that I wanted to upgrade to Debian Jessie (8). It ran in an OpenVZ environment, ie. the kernel is shared between instances. Sadly, Jessie uses systemd which means I need a new kernel. OpenVZ prevents me from installing a new kernel. I tried it and ended with a machine that was offline. I could connect via serial console and nothing else. But like an idiot, I once again used OpenVZ. And now as I am about to upgrade to Debin Stretch (9) I am faced with the same prospect. This time, however, it is going to be a KVM.
I tried it
So now I’m setting up a new machine. I’m copying stuff over, step by step. Needless to say, I’m not happy.
From the web site:
- reinstall operating system
- change root password
Connect as root via ssh:
- `cat /etc/debian_version` # check that we have the correct version
- `apt update`
- `apt install dialog`
- `apt upgrade -y`
- `dpkg-reconfigure locales` was not necessary because I liked the default of `en_GB.UTF-8`
- `dpkg-reconfigure tzdata` and pick Europe/Zurich
- `adduser alex`
- `usermod -a -G sudo alex`
On the old machine, I installed `apt-clone` and ran `apt-clone clone sibirocobombus` which game me the file `sibirocobombus.apt-clone.tar.gz`. When I tried to restore this on the new machine, I was unable to run the `restore-new-distro` command because there seemed to be no appropriate destination distro shortcut. When I used `restore` it seemed to take forever and I aborted it. Now I have a broken setup and I’m angry.
Don’t do this:
- `scp -P 882 alex@192.71.233.105:sibirocobombus.apt-clone.tar.gz .`
- `apt install apt-clone`
- `apt-clone restore sibirocobombus.apt-clone.tar.gz`
To undo the damage:
- `mv /etc/apt/sources.list.apt-clone /etc/apt/sources.list`
The manual alternative also doesn’t work. On the old system, you can export a list of packages:
- `dpkg --get-selections > packages.list`
- `apt-mark showauto > package.states.auto.list`
- `apt-mark showhold > package.states.hold.list`
- `apt-mark showmanual > package.states.manual.list`
Copy them to the new system and try to import them:
- `scp -P 882 alex@192.71.233.105:*.list .`
- `dpkg --set-selections < packages.list`
I’m getting an error for practically every single package. So now I’m trying to do it the old way.
- `apt-get install -y emacs rsync sudo less ssh w3m git apache2 munin monit make telnet checksecurity lockfile-progs bsd-mailx mutt cron-apt fail2ban strace bzip2 unzip dialog makepatch man info subversion git python-pygments colordiff diffutils curl gcc libgd-dev hunspell-an hunspell-ar hunspell-be hunspell-br hunspell-ko hunspell-en-us hunspell-fr hunspell-fr-modern hunspell-gl-es hunspell-kk hunspell-ml hunspell-ru hunspell-se hunspell-sv-se hunspell-de-at hunspell-de-ch hunspell-de-de hunspell-da hunspell-en-ca hunspell-hu hunspell-ne hunspell-ro hunspell-sr hunspell-vi hunspell-uz hunspell-eu-es espeak lame libwww-perl spf-tools-perl sasl2-bin spamassassin swaks libnet-ssleay-perl exim4-daemon-heavy procmail libdb-dev python3-pip` # don’t bother creating keys for tripwire, we will be deinstalling it
- `apt-get remove -y tripwire logcheck` # because this one is annoying
- `apt-get autoremove -y`
- `rsync --rsh="ssh -p 882" --archive --compress --verbose 192.71.233.105:/etc/monit/conf.d/ /etc/monit/conf.d` # no changes to `/etc/monit/monitrc`
- `rsync --rsh="ssh -p 882" --archive --compress --verbose 192.71.233.105:/etc/ssl/localcerts/ /etc/ssl/localcerts` # required for monit.pem
- `rsync --rsh="ssh -p 882" --archive --compress --verbose 192.71.233.105:/etc/munin/plugin-conf.d/ /etc/munin/plugin-conf.d`
- `rsync --rsh="ssh -p 882" --archive --compress --verbose 192.71.233.105:/etc/munin/plugins/ /etc/munin/plugins`
- `rsync --rsh="ssh -p 882" --archive --compress --verbose 192.71.233.105:/etc/munin/munin-htpasswd /etc/munin/`
- `rm /etc/munin/plugins/if_* /etc/munin/plugins/http_loadtime`
- `rsync --rsh="ssh -p 882" --archive --compress --verbose 192.71.233.105:/etc/rsyncd.conf /etc/`
- `rsync --rsh="ssh -p 882" --archive --compress --verbose 192.71.233.105:/etc/logrotate.d/alex-websites /etc/logrotate.d`
- `echo 178.209.50.237 sibirocobombus communitywiki.org emacswiki.org www.emacswiki.org campaignwiki.org rpg.alexschroeder.ch korero.org arabisch-lernen.org alexschroeder.ch oddmuse.org >> /etc/hosts`
- `for f in hardening.conf letsencrypt.conf security.conf max-uri.conf perl5.conf servername.conf log.conf; do rsync --rsh="ssh -p 882" --archive --compress --verbose 192.71.233.105:/etc/apache2/conf-available/$f /etc/apache2/conf-available; done`
- `for f in hardening.conf letsencrypt.conf security.conf max-uri.conf perl5.conf servername.conf log.conf; do a2enconf $f; done`
- `for f in other-vhosts-access-log.conf serve-cgi-bin.conf localized-error-pages.conf charset.conf; do a2disconf $d; done`
Here’s the essential info: `for f in hardening.conf letsencrypt.conf security.conf max-uri.conf perl5.conf servername.conf log.conf; do echo $f; printf %s "$f"|tr -c '-' '[-*]'; echo; grep -v '^#' $f | grep -v '^
; echo; done`
hardening.conf
--------------
SSLProtocol All -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
SSLHonorCipherOrder On
SSLCipherSuite HIGH:!RSA:!MD5:!DSS
letsencrypt.conf
----------------
Alias /.well-known/acme-challenge /var/www/letsencrypt.sh/
<Directory /var/www/letsencrypt.sh/>
Options None
AllowOverride None
Order allow,deny
Allow from all
</Directory>
security.conf
-------------
ServerTokens Prod
ServerSignature Off
TraceEnable Off
Header set X-Frame-Options: "sameorigin"
Header set Strict-Transport-Security "max-age=15768000; includeSubDomains"
max-uri.conf
------------
LimitRequestLine 32000
LimitRequestFieldSize 32000
perl5.conf
----------
SetEnv PERL5LIB /home/alex/perl5/lib/perl5
servername.conf
---------------
ServerName sibiricobombus
log.conf
--------
CustomLog ${APACHE_LOG_DIR}/access.log vhost_combined
- `for f in cgid expires headers info proxy_http proxy_wstunnel rewrite ssl; do a2enmod $f; done`
- set hostname in `/etc/munin/munin.conf`
- at this point I realise that I might have just copied all of `/etc/munin` over to the new system
- `rsync --rsh="ssh -p 882" --archive --compress --verbose 192.71.233.105:/etc/letsencrypt.sh/ /etc/letsencrypt.sh`
- `rsync --rsh="ssh -p 882" --archive --compress --verbose 192.71.233.105:/etc/apache2/sites-available/ /etc/apache2/sites-available`
- `for f in 100-alexschroeder.ch.conf 500-arabisch-lernen.org.conf 500-campaignwiki.org.conf 500-communitywiki.org.conf 500-emacswiki.org.conf 500-korero.org.conf 500-oddmuse.org.conf 500-orientalisch.info.conf; do a2ensite $f; done`
- `a2dissite 000-default.conf`
- `perl -pi -e 's/#Port 22/Port 882/g' /etc/ssh/sshd_config`
- `echo AllowUsers alex root >> /etc/ssh/sshd_config`
- `echo MaxAuthTries 3 >> /etc/ssh/sshd_config` (because of PAM)
- `systemctl restart apache2`
- `systemctl restart munin-node`
- `systemctl restart monit`
- `systemctl restart sshd`
PAM
Watch out! Now that `monit` is monitoring the services, it will kill the SSH service if it can’t connect to it on port 882. I also had to comment the section about `/etc/ssh/ssh_host_dsa_key` in `/etc/monit/conf.d/openssh-server.conf`. I need to investigate where this comes from.
Verify that all the services are up and running:
I did not add reverse DNS entry on the web site. I don’t remember why this was necessary. Probably it was required back when I tried to run a mail server.
I thought I was going to need a copy of `/usr/local` but it was unnecessary. There is nothing in there.
I am using Perl via Perlbrew! It’s part of my home directory, so nothing else is required.
Connect as alex via ssh and copy stuff from the old server.
- `rsync --rsh="ssh -p 882" --archive --compress --progress --rsh="ssh -p 882" 192.71.233.105: .`
Copy the cronjobs from the old server. Use `EDITOR=emacs crontab -e` to edit the file using Emacs. Note that I’ve uncommented all the jobs because they should either run on the old server (kallobombus) or on the new server (sibirocobombus), but not on both!
MAILTO=kensanata@gmail.com
1. 02 5 * * * /home/alex/bin/maintain-campaignwiki
1. 47 4,16 * * * /home/alex/bin/backup
1. 28 4 * * * /home/alex/bin/subscriptions
Don’t forget `/etc/cron.weekly/letsencrypt.sh`:
!/bin/sh
exec /etc/letsencrypt.sh/letsencrypt.sh -c
Make it executable.
This requires the WELLKNOWN directory. Then run it once.
- `mkdir /var/www/letsencrypt.sh/`
- `chown www-data.www-data /var/www/letsencrypt.sh/`
- `sh /etc/cron.weekly/letsencrypt`
What follows are the various DNS records.
emacswiki.org:
@ 10800 IN A 178.209.50.237
@ 10800 IN AAAA 2a02:418:6a04:178:209:50:237:1
@ 10800 IN SOA ns1.gandi.net. hostmaster.gandi.net. 1520800289 10800 3600 604800 10800
@ 10800 IN TXT "keybase-site-verification=A89bZB-zWTXsS00OTCMKTc9FILAujxFssBg-jHOHuRs"
www 10800 IN A 178.209.50.237
oddmuse.org:
@ 10800 IN A 178.209.50.237
@ 10800 IN AAAA 2a02:418:6a04:178:209:50:237:1
@ 10800 IN SOA ns1.gandi.net. hostmaster.gandi.net. 1520800453 10800 3600 604800 10800
@ 10800 IN TXT "keybase-site-verification=A89bZB-zWTXsS00OTCMKTc9FILAujxFssBg-jHOHuRs"
alexine 10800 IN A 94.23.219.181
www 10800 IN A 178.209.50.237
alexschroeder.ch:
@ 10800 IN A 178.209.50.237
@ 10800 IN AAAA 2a02:418:6a04:178:209:50:237:1
@ 10800 IN SOA ns1.gandi.net. hostmaster.gandi.net. 1520800528 10800 3600 604800 10800
@ 10800 IN TXT "keybase-site-verification=A89bZB-zWTXsS00OTCMKTc9FILAujxFssBg-jHOHuRs"
rpg 10800 IN A 178.209.50.237
www 10800 IN A 178.209.50.237
communitywiki.org:
@ 10800 IN A 178.209.50.237
@ 10800 IN AAAA 2a02:418:6a04:178:209:50:237:1
@ 10800 IN SOA ns1.gandi.net. hostmaster.gandi.net. 1520800356 10800 3600 604800 10800
@ 10800 IN TXT "keybase-site-verification=A89bZB-zWTXsS00OTCMKTc9FILAujxFssBg-jHOHuRs"
www 10800 IN A 178.209.50.237
orientalisch.info (includes mail redirection):
@ 10800 IN A 178.209.50.237
@ 10800 IN AAAA 2a02:418:6a04:178:209:50:237:1
@ 10800 IN MX 10 spool.mail.gandi.net.
@ 10800 IN MX 50 fb.mail.gandi.net.
@ 10800 IN SOA ns1.gandi.net. hostmaster.gandi.net. 1520792978 10800 3600 604800 10800
@ 10800 IN TXT "keybase-site-verification=A89bZB-zWTXsS00OTCMKTc9FILAujxFssBg-jHOHuRs"
@ 10800 IN TXT "v=spf1 include:_mailcust.gandi.net ?all"
flying-carpet.ch (parked domain):
- 10800 IN CNAME webredir.vip.gandi.net.
- .blog 10800 IN CNAME webredir.vip.gandi.net.
- .imap 10800 IN CNAME webredir.vip.gandi.net.
- .pop 10800 IN CNAME webredir.vip.gandi.net.
- .smtp 10800 IN CNAME webredir.vip.gandi.net.
- .webmail 10800 IN CNAME webredir.vip.gandi.net.
- .www 10800 IN CNAME webredir.vip.gandi.net.
@ 10800 IN A 217.70.184.38
@ 10800 IN MX 10 spool.mail.gandi.net.
@ 10800 IN MX 50 fb.mail.gandi.net.
@ 10800 IN SOA ns1.gandi.net. hostmaster.gandi.net. 1520793004 10800 3600 604800 10800
blog 10800 IN CNAME blogs.vip.gandi.net.
imap 10800 IN CNAME access.mail.gandi.net.
pop 10800 IN CNAME access.mail.gandi.net.
smtp 10800 IN CNAME relay.mail.gandi.net.
webmail 10800 IN CNAME agent.mail.gandi.net.
www 10800 IN CNAME webredir.vip.gandi.net.
campaignwiki.org:
@ 10800 IN A 178.209.50.237
@ 10800 IN AAAA 2a02:418:6a04:178:209:50:237:1
@ 10800 IN SOA ns1.gandi.net. hostmaster.gandi.net. 1520800412 10800 3600 604800 10800
@ 10800 IN TXT "keybase-site-verification=A89bZB-zWTXsS00OTCMKTc9FILAujxFssBg-jHOHuRs"
www 10800 IN A 178.209.50.237
arabisch-lernen.org:
@ 10800 IN A 178.209.50.237
@ 10800 IN AAAA 2a02:418:6a04:178:209:50:237:1
@ 10800 IN SOA ns1.gandi.net. hostmaster.gandi.net. 1520800579 10800 3600 604800 10800
@ 10800 IN TXT "keybase-site-verification=A89bZB-zWTXsS00OTCMKTc9FILAujxFssBg-jHOHuRs"
www 10800 IN A 178.209.50.237
If you want, copy the latest stuff.
- `rsync --rsh="ssh -p 882" --archive --compress --progress --rsh="ssh -p 882" 192.71.233.105: .`
Lock the old sites:
- `emacs */config`
- make sure these all say `$EditAllowed = 0;` somehere
- `for f in */config; do grep EditAllowed $f; done`
Write a note on the important old sites:
- Visit alexschroeder.ch and emacswiki.org and leave a note like the following: “Once again, a new server is in order. The current OpenVZ system doesn’t allow me to dist-upgrade the server from Debian 8 to Debian 9, I think. While the domain name changes propagate through the sytem, editing has been disabled on the old site. If you see this notice, you’re still on the old site. As my domain name service provider (Gandi) also decided to switch to a new system and I only just learned about it, this change might take up to 24h to propagate. I’m sorry! If you have any questions, feel free to contact me via mail. – Alex Schroeder”
- Post a note on the Google+ profiles for Emacs Wiki, Oddmuse and Campaign Wiki.
- Connect via SSH to the new sites and use w3m to compose a *different* note for alexschroeder.ch and emacswiki.org. As soon as people get the new DNS info, they will see the new note, and the sites will be writable again. This should work because of the line we added `/etc/hosts`.
- I checked my SSL setup at the SSL Labs site and got an A+ for `alexschroeder.ch`.
- verified that the Old School Hex Mapper still works
- verified that the M20 Hard Core Character Generation still works
- verified that Mark's site still works
- verified Munin setup
- verified Monit setup
- verified that this works: `keybase id kensanata`
- verified that korero.org still works
- verified that local mail delivery works: I had to edit `/etc/aliases` and change the alias for root from “debian” to “alex”. Then take a look at the exim cheatsheet: `exim -bp` to look at the queue, `exim -qff` to flush the queue, including frozen messages
- replace `letsencrypt.sh` with dehydrated, see Sibirocobombus Dehydrated
Mark's site
korero.org
exim cheatsheet
dehydrated
Sibirocobombus Dehydrated
Todo:
CAA
easy