I'm still playing with monitoring programs, and I've been remembered about Netdata. What an improvement over the last 8 years!
This tutorial explains how to get Netdata installed on NixOS, and how to register your node in Netdata cloud.
This program is a simple service to run on a computer, it will automatically gather a ton of metrics and make them easily available over the local TCP port 19999. You just need to run Netdata and nothing else, and you will have every metrics you can imagine from your computer, and some explanations for each of them!
That's pretty cool because Netdata is very efficient, it draws nearly no CPU while gathering a few thousands metrics every few seconds, and is memory efficient and can be constrained to a dozen of megabytes.
While you can export its metrics to something like graphite or Prometheus, you lose the nice display which is absolutely a blast compare to Grafana (in my opinion).
Update: as pointed out by a reader (thanks!), it's possible to connect Netdata instances to only one used for viewing metrics. I'll investigate this soon.
Netdata documentation about streaming.
Netdata also added some machine learning anomaly detection, it's simple and doesn't use many resources or require a GPU, it only builds statistical models to be able to report if some metrics have an unusual trend. It takes some time to gather enough data, and after a few days it's starting to work.
As usual, it's simple, add this to your NixOS configuration and reconfigure the system.
services.netdata = {
enable = true;
config = {
global = {
# uncomment to reduce memory to 32 MB
#"page cache size" = 32;
# update interval
"update every" = 15;
};
ml = {
# enable machine learning
"enabled" = "yes";
};
};
};
You should have Netdata dashboard available on http://localhost:19999 .
Here is a simple configuration on NixOS to connect a headless node without persistency to send all on a main Netdata server storing data but also displaying them.
You need to generate an UUID with uuidgen, replace UUID in the text with the result. It can be per system or shared by multiple Netdata instances.
My networks are 10.42.42.0/24 and 10.43.43.0/24, I'll allow everything matching 10.* on the receiver, I don't open port 19999 on a public interface.
services.netdata.enable = true;
services.netdata.config = {
global = {
"default memory mode" = "none"; # can be used to disable local data storage
};
};
services.netdata.configDir = {
"stream.conf" = pkgs.writeText "stream.conf" ''
[stream]
enabled = yes
destination = 10.42.42.42:19999
api key = UUID
[UUID]
enabled = yes
'';
};
networking.firewall.allowedTCPPorts = [19999];
services.netdata.enable = true;
services.netdata.configDir = {
"stream.conf" = pkgs.writeText "stream.conf" ''
[UUID]
enabled = yes
default history = 3600
default memory mode = dbengine
health enabled by default = auto
allow from = 10.*
'';
};
Netdata company started a "cloud" offer that is free, but they plan to keep it free but also propose more services for paying subscribers. The free plan is just a convenience to see metrics from multiple nodes at the same place, they don't store any metrics apart metadata (server name, OS version, kernel, etc..), when you look at your metrics, they just relay from your server to your web browser without storing the data.
The free cloud plan offers a correlating feature, but I still didn't have the opportunity to try it, and also email alerting when an alarm is triggered.
Netdata cloud data privacy information
The official way to connect a Netdata agent to the Netdata cloud is to use a script downloaded on the internet and run it with some parameter.
I strongly dislike this method as I'm not a huge fan of downloading script to run as root that are not provided by my system.
When you want to add a new node, you will be given a long command line and a token, keep that token somewhere. NixOS Netdata package offers a script named `netdata-claim.sh` (which seems to be part of Netdata source code) that will generate a pair of RSA keys, and look for the token in a file.
Once you got the token, we will claim it to associate it to a node:
1. create /var/lib/netdata/cloud.d/token and write the token in it
2. run nix-shell -p netdata --run "netdata-claim.sh" as root
3. your node should be registered in Netdata cloud
Netdata is really a wonderful tool, ideally I'd like it to replace all the Grafana + storage + agent stack, but it doesn't provide persistent centralized storage compatible with its dashboard. I'm going to experiment with their Netdata cloud service, I'm not sure if it would add value for me, and while they have a very correct data privacy policy, I prefer to self-host everything.