going-flying.com gemini git repository
bdb16dc32a358643c01a2fa84f581ff84edc9e4e - Matthew Ernisse - 1623692629
new post
diff --git a/users/mernisse/articles/24.gmi b/users/mernisse/articles/24.gmi
new file mode 100644
index 0000000..87ce85b
--- /dev/null
+++ b/users/mernisse/articles/24.gmi
@@ -0,0 +1,373 @@
+---
+Title: Sysadmining: DNS Four, Automation
+Date: 2021-06-14 13:30
+
+In the previous three DNS posts I covered DNS from a fairly high level
+and then focused in on some key elements in BIND, the de-facto DNS server
+package. DNS is such an important part of inter-networking that it is bound
+to be a bit complex and given that complexity leads towards errors when
+humans are involved it is a pretty solid use-case for automation.
+
+I mainly use Puppet to automate the state of my systems. I find it to be
+much more powerful than other tools and it allows me to specify a desired
+state instead of a set of tasks. It is also one of the most mature
+automation tools out there and is well supported on the platforms I use.
+
+In general I automate three key pieces of my DNS infrastructure (other than
+making sure the required packages are installed and up-to-date, of course).
+1) Zonefile generation
+2) TLSA and DMARC records
+3) DNSSEC key rotation and signing
+
+Zonefile generation is easy, they are all generated from templates and
+concatenated together. This way I can specify records that may change
+in YAML using Hiera (Puppet's hierarchical key/value store), or resolve
+them via custom functions (as I do with the TLSA records for example) and
+have static blocks that are the same for all of my domains (like the NS
+and MX record sections).
+
+My Puppet master runs various jobs that generate and rotate cryptographic
+keys. I wrote some custom plugins for LetsEncrypt certbot that lets
+Puppet manage the certificates and keys. Since they are available to
+Puppet I can generate the TLSA records that are used for DANE (a method to
+assure a client that the certificates they are getting from a TLS server
+are the correct ones). I provide the fingerprints used by the keys
+in the format the Apache (my webserver of choice) and BIND want using
+Facter (the Puppet host fact engine). The following Ruby code provides
+the custom facts to Facter.
+
+```
+# Generate the fingerprint of the SSL cert for things like
+# Certificate pinning and TLSA records.
+require 'fileutils'
+require 'openssl'
+
+Facter.add("ssl_pubkey_fingerprint") do
+ confine :kernel => "Linux"
+
+ setcode do
+ cert = "/etc/ssl/certs/ssl.ub3rgeek.net_cert.pem"
+ fingerprints = {}
+
+ if File.exist?(cert)
+ fd = File.read(cert)
+ s_cert = OpenSSL::X509::Certificate.new(fd)
+ p_key = s_cert.public_key().to_der()
+ dgst = OpenSSL::Digest::SHA256.new(p_key)
+ fingerprints['ssl.ub3rgeek.net'] = dgst.base64digest()
+ end
+
+ cert = "/etc/ssl/certs/going-flying.com_cert.pem"
+ if File.exist?(cert)
+ fd = File.read(cert)
+ s_cert = OpenSSL::X509::Certificate.new(fd)
+ p_key = s_cert.public_key().to_der()
+ dgst = OpenSSL::Digest::SHA256.new(p_key)
+ fingerprints['www.going-flying.com'] = dgst.base64digest()
+ end
+
+ fingerprints
+ end
+end
+
+Facter.add("ssl_pubkey_hexdigest") do
+ confine :kernel => "Linux"
+
+ setcode do
+ cert = "/etc/ssl/certs/ssl.ub3rgeek.net_cert.pem"
+ digests = {}
+ if File.exist?(cert)
+ fd = File.read(cert)
+ s_cert = OpenSSL::X509::Certificate.new(fd)
+ p_key = s_cert.public_key().to_der()
+ dgst = OpenSSL::Digest::SHA256.new(p_key)
+ digests['ssl.ub3rgeek.net'] = dgst.to_s()
+ end
+
+ cert = "/etc/ssl/certs/going-flying.com_cert.pem"
+ if File.exist?(cert)
+ fd = File.read(cert)
+ s_cert = OpenSSL::X509::Certificate.new(fd)
+ p_key = s_cert.public_key().to_der()
+ dgst = OpenSSL::Digest::SHA256.new(p_key)
+ digests['www.going-flying.com'] = dgst.to_s()
+ end
+
+ digests
+ end
+end
+```
+
+Finally, the most complex automated task is DNSSEC. DNSSEC is different
+from DNS over TLS (or DNS over HTTPS). DNSSEC does not seek to provide any
+privacy, rather it provides a chain of trust back to the root of the global
+DNS system so your resolver can verify that the answer it is getting is in
+fact the correct answer. DNSSEC is a huge topic but in general it allows
+zone administrators to cryptographically sign the records in their zones
+and any zones that they delegate to, so in going-flying.com you can trace
+the signatures from going-flying, to com to '.' (the root of DNS) and know
+for sure that someone isn't spoofing the answer.
+
+I have a script that is called by puppet any time the DNS configuration
+changes (which is at least monthly due to various cryptographic key
+rotations and things like LetsEncrypt).
+
+```
+#!/bin/sh
+# dnssec-signzones (c) 2017-2018 Matthew J. Ernisse <matt@going-flying.com>
+#
+# Manage DNSSEC for zones, including rotating keys as needed.
+
+set -e
+PATH=$PATH:/sbin:/usr/sbin
+
+# Paths to BIND files
+KEYDIR="/etc/bind/keys"
+STATEDIR="/var/tmp"
+ZONEDIR="/var/cache/bind"
+
+# Timeouts
+DEACTIVATE_TIMEOUT="+3d"
+DELETE_TIMEOUT="+5d"
+PREPUBLISH_INTERVAL="+2d"
+REKEY_AGE="30" # days
+
+# delete any keys that are past their delete-by
+delete_old_keys()
+{
+ local dtime=0
+ local now=$(date +"%s")
+
+ for file in $(find ${KEYDIR} -name \*.key); do
+ if grep -q 'key-signing' "$file"; then
+ continue
+ fi
+
+ dtime=$(sed -ne 's/^; Delete:\s*[0-9]*\s*(\(.*\))/\1/p' \
+ "$file")
+
+ if [ -z "$dtime" ]; then
+ continue
+ fi
+
+ dtime=$(date -d "${dtime}" +"%s")
+ if [ "$dtime" -le "$now" ]; then
+ echo "Deleted expired key: $file"
+ rm -- "$file"
+ rm -- "$(echo "$file" | sed -e 's/\.key$/.private/')"
+ fi
+ done
+}
+
+find_domains()
+{
+ for domain in $(ls -1 ${KEYDIR} | \
+ sed -ne 's/^K\([-\._a-z0-9]*\)\.\+.*/\1/p' | sort -u); do
+ echo $domain
+ done
+}
+
+# return the latest (currently used) DNSSEC keyfile
+get_last_key()
+{
+ if [ -z $1 ]; then
+ echo "usage: get_last_key domain"
+ return 127
+ fi
+
+ local domain fn m mtime=0
+ domain="$1"
+
+ for file in $(find ${KEYDIR} -name K${domain}\*.key); do
+ if grep -q 'key-signing' "$file"; then
+ continue
+ fi
+
+ keyage=$(sed -ne \
+ 's/^; Activate:\s\{1,\}[0-9]\{1,\}\s\{1,\}(\(.*\))$/\1/p' \
+ "$file")
+ keyage=$(date -d "$keyage" +%s)
+
+ if [ -z "$keyage" ]; then
+ echo "get_last_key() failed to read activation time"
+ return 1
+ fi
+
+ if [ "$keyage" -gt "$mtime" ]; then
+ mtime="$keyage"
+ fn="$file"
+ fi
+ done
+
+ echo "$(basename $fn)"
+}
+
+need_new_key()
+{
+ if [ -z $1 ]; then
+ echo "usage: need_new_key domain"
+ return 127
+ fi
+
+ local domain last_key keyage now
+ domain="$1"
+ now=$(date +%s)
+
+ last_key=$(get_last_key "$domain")
+ keyage=$(sed -ne \
+ 's/^; Activate:\s\{1,\}[0-9]\{1,\}\s\{1,\}(\(.*\))$/\1/p' \
+ "$KEYDIR/$last_key")
+ keyage=$(date -d "$keyage" +%s)
+
+ if [ $(( $now - $keyage )) -ge $(( ${REKEY_AGE} * 86400 )) ]; then
+ echo "key ${last_key} needs rotating"
+ return 0
+ fi
+
+ return 1
+}
+
+rotate_keys()
+{
+ if [ -z $1 ]; then
+ echo "usage: rotate_keys domain"
+ return 127
+ fi
+
+ domain="$1"
+ last_key=$(get_last_key "$domain")
+
+ # Logically this should go the other way, generate a new and then
+ # if successful expire the old but dnssec-keygen refuses to generate
+ # a successor key if there is no expire time on the current key..
+ echo "Setting expire times on ${last_key}"
+ if ! dnssec-settime -K "$KEYDIR" -I "$DEACTIVATE_TIMEOUT" \
+ -D "$DELETE_TIMEOUT" "$last_key"; then
+ exit 1
+ fi
+
+ echo "Generating new key..."
+ if ! dnssec-keygen -K "$KEYDIR" -S "$last_key" \
+ -i "$PREPUBLISH_INTERVAL" -q; then
+ exit 1
+ fi
+
+}
+
+sign_domain()
+{
+ if [ -z $1 ]; then
+ echo "usage: sign_domain domain"
+ return 127
+ fi
+
+ domain="$1"
+
+ for view in "public" "private"; do
+ zonefile="${view}-${domain}.hosts"
+
+ if [ -f "$ZONEDIR/$zonefile" ]; then
+ sign_zone "$domain" "$zonefile"
+ fi
+ done
+}
+
+sign_zone()
+{
+ if [ -z "$1" ] || [ -z "$2" ]; then
+ echo "usage: sign_zone domain zonefile"
+ return 127
+ fi
+
+ domain="$1"
+ zonefile="$2"
+
+ echo "signing $zonefile for $domain"
+ dnssec-signzone -S -K "$KEYDIR" -o "$domain" \
+ "$ZONEDIR/$zonefile"
+}
+
+# update the zone's serial -- this is no longer handled by puppet.
+update_serial()
+{
+ if [ -z $1 ]; then
+ echo "usage: update_serial domain"
+ return 127
+ fi
+ local new_serial old_rev old_rev_stripped old_serial=0
+ local zonefile
+
+ if [ -f "${STATEDIR}/${domain}.serial" ]; then
+ old_serial=$(< "${STATEDIR}/${domain}.serial")
+ old_rev=$(echo "$old_serial" | sed -ne \
+ 's/^[0-9]\{,8\}\([0-9]\{,2\}\)$/\1/p')
+ old_serial=$(echo "$old_serial" | sed -ne \
+ 's/^\([0-9]\{,8\}\).*/\1/p')
+
+ # Bash will think we're in octal if we feed it a number
+ # with a leading zero.
+ old_rev_stripped=$(echo "$old_rev" | sed -ne \
+ 's/^0\([0-9]\)/\1/p')
+ if [ -n "$old_rev_stripped" ]; then
+ old_rev="$old_rev_stripped"
+ fi
+ fi
+
+ new_serial="$(date +%Y%m%d)"
+
+ if [ "$new_serial" -eq "$old_serial" ]; then
+ if [ $(( $old_rev + 1 )) -lt 10 ]; then
+ new_serial="${new_serial}0$(( $old_rev + 1 ))"
+ else
+ new_serial="${new_serial}$(( $old_rev + 1 ))"
+ fi
+ else
+ new_serial="${new_serial}00"
+ fi
+
+ echo "$new_serial" > "${STATEDIR}/${domain}.serial"
+
+ for view in "public" "private"; do
+ zonefile="${ZONEDIR}/${view}-${domain}.hosts"
+ sed -e "s/##SERIAL_NUMBER##/${new_serial}/" \
+ "${zonefile}.puppet" > "$zonefile"
+ done
+}
+
+for domain in $(find_domains); do
+ if need_new_key "$domain"; then
+ rotate_keys "$domain"
+ fi
+
+ delete_old_keys
+ update_serial "$domain"
+ sign_domain "$domain"
+done
+```
+
+The script is complex because it needs to do several things but the for
+loop at the bottom tells the story of what is happening. The primary
+job of the script is to manage the DNSSEC keys, rotating them and
+expiring old ones, so since this script is run every time Puppet every
+time any of the DNS zonefiles are changed the first thing we do is to
+check to see if we actually need to rotate the keys. Once we do that
+we simply expire any old keys, update the serial numbers on the zones
+and then sign them. I can't imagine this script working verbatim for
+anyone else, it assumes a lot of things that are probably only true in
+my setup, but it should get anyone looking to automate DNSSEC with BIND
+well on their way.
+
+DNS is one of the most important services provided on the Internet. It
+is used every time you type a name instead of an IP address. It is used
+for service discovery and configuration, e-mail SPAM detection, malware
+prevention and is the core method CDNs use to steer you
+to their closest edge location to optimize delivery of everything from
+JPEGs to critical software patches. Being comfortable with it gives a
+systems / network administrator tremendous control over network
+traffic and performance, it is an extremely powerful tool in the
+security administrator's portfolio, being able to short circuit command
+and control server connections and prevent malicious payloads from being
+downloaded, and any support people as trouble in DNS land is the root cause
+of so many end-user problems. I highly recommend _DNS and BIND_, by
+Cricket Liu and Paul Albitz, published by O'Reilly Media as further
+reading.