How to account systemd services bandwidth usage on NixOS

Author: Solène
Date: 20 July 2022
Tags: nixos bandwidth monitoring

Introduction

Did you ever wonder how many bytes a system service is daily receiving from the network? Thanks to systemd, we can easily account this.

This guide targets NixOS, but the idea could be applied on any Linux system using systemd.

NixOS project website

In this article, we will focus on the nix-daemon service.

Setup

We will enable the attribute IPAccounting on the systemd service nix-daemon, this will make systemd to account bytes and packets that received and sent by the service. However, when the service is stopped, the counters are reset to zero and the information logged into the systemd journal.

In order to efficiently gather the network information over time into a database, we will run a script just before the service stops using the preStop service hook.

The script checks the existence of a sqlite database /var/lib/service-accounting/nix-daemon.sqlite, creates it if required, and then inserts the received bytes information of the nix-daemon service about to stop. The script uses the service attribute InvocationID and the current day to ensure that a tuple won't be recorded more than once, because if we restart the service multiple times a day, we need to distinguish all the nix-daemon instances.

Here is the code snippet to add to your `/etc/nixos/configuration.nix` file before running `nixos-rebuild test` to apply the changes.

  systemd.services.nix-daemon = {
      serviceConfig.IPAccounting = "true";
      path = with pkgs; [ sqlite busybox systemd ];
      preStop = ''
#!/bin/sh

SERVICE="nix-daemon"
DEST="/var/lib/service-accounting"
DATABASE="$DEST/$SERVICE.sqlite"

mkdir -p "$DEST"

# check if database exists
if ! dd if="$DATABASE" count=15 bs=1 2>/dev/null | grep -Ea "^SQLite format.[0-9]$" >/dev/null
then
cat <<EOF | sqlite3 "$DATABASE"
CREATE TABLE IF NOT EXISTS accounting (
        id TEXT PRIMARY KEY,
        bytes INTEGER NOT NULL,
        day DATE NOT NULL
);
EOF
fi

BYTES="$(systemctl show "$SERVICE.service" -P IPIngressBytes | grep -oE "^[0-9]+$")"
INSTANCE="'$(systemctl show "$SERVICE.service" -P InvocationID | grep -oE "^[a-f0-9]{32}$")'"

cat <<EOF | sqlite3 "$DATABASE"
INSERT OR REPLACE INTO accounting (id, bytes, day) VALUES ($INSTANCE, $BYTES, date('now'));
EOF
     '';
  };

If you want to apply this to another service, the script has a single variable SERVICE that has to be updated.

Display the information from the database

You can use the following command to display the bandwidth usage of the nix-daemon service with a day-by-date report:

$ echo "SELECT day, sum(bytes)/1024/1024 AS Megabytes FROM accounting group by day" | sqlite3 -header -column /var/lib/service-accounting/nix-daemon.sqlite
day         Megabytes
----------  ---------
2022-07-17  173
2022-07-19  3018
2022-07-20  84

Please note this command requires the sqlite package to be installed in your environment.

Enhancement

I have some ideas to improve the setup:

The script could be improved to support multiple services within the database by using a new field
The command to display data could be improved and turned into a system package to make it easier to use
Provide an SQL query for monthly summary

Conclusion

Systemd services are very flexible and powerful thanks to the hooks provided to run script at the right time. While I was interested into network usage accounting, it's also possible to achieve a similar result with CPU usage and I/O accesses.