💾 Archived View for dctrud.randomroad.net › gemlog › 20170415-graphite-grafana.gmi captured on 2022-04-28 at 17:21:29. Gemini links have been rewritten to link to archived content

View Raw

More Information

⬅️ Previous capture (2020-09-24)

🚧 View Differences

-=-=-=-=-=-=-

2017-04-15 - Graphite, Grafana with gunicorn and NGINX

At work I have wanted to implement system monitoring using graphite and grafana. Eventually I will use this particularly to monitor our Lustre storage system with either collectl or collectd shipping Lustre and general host stats to graphite. To try things out I have installed the stack onto my Kimsufi server.

The graphite installation documentation uses apache mod_fcgi, but I'm running various other python projects with gunicorn and then an NGINX or Apache front-end proxy, so will use gunicorn to run the graphite-web app, and NGINX to proxy it and grafana over https.

Installation on CentOS 7

I run CentOS 7 on my Kimsufi 2c server, and the procedure to install graphite, grafana, gunicorn, nginx is as follows:

EPEL repository

yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

Graphite Dependencies & Graphite Packages

# Distribution packages
yum install -y httpd net-snmp perl python-devel git gcc-c++ pycairo mod_wsgi libffi-devel
yum install -y python-pip node npm
# Python package via pip
pip install django
pip install django-tagging
pip install pytz  
pip install Twisted==16.4.1
# Installing graphite from master, as the stable release is quite old at the moment
export PYTHONPATH="/opt/graphite/lib/:/opt/graphite/webapp/"
pip install --no-binary=:all: https://github.com/graphite-project/whisper/tarball/master
pip install --no-binary=:all: https://github.com/graphite-project/carbon/tarball/master
pip install --no-binary=:all: https://github.com/graphite-project/graphite-web/tarball/master

Configure Graphite

Copy the example configuration into live locations

sudo cp /opt/graphite/conf/storage-schemas.conf.example /opt/graphite/conf/storage-schemas.conf
sudo cp /opt/graphite/conf/storage-aggregation.conf.example /opt/graphite/conf/storage-aggregation.conf  
sudo cp /opt/graphite/conf/graphTemplates.conf.example /opt/graphite/conf/graphTemplates.conf 
sudo cp /opt/graphite/conf/graphite.wsgi.example /opt/graphite/conf/graphite.wsgi  
sudo cp /opt/graphite/webapp/graphite/local_settings.py.example /opt/graphite/webapp/graphite/local_settings.py  
sudo cp /opt/graphite/conf/carbon.conf.example /opt/graphite/conf/carbon.conf

Configure the storage schema:

vi /opt/graphite/conf/storage-schemas.conf

Add the default retention:

[default]
pattern = .*
retentions = 12s:4h, 2m:3d, 5m:8d, 13m:32d, 1h:1y

Add a user account for graphite:

useradd -d /opt/graphite graphite
chown graphite /opt/graphite -R

Configure the carbon daemon, and start it

vi /opt/graphite/conf/storage-schemas.conf

Make these edits:

USER = graphite

Create system.d unit file /etc/systemd/system/carbon.service

[Unit]
Description = Carbon Metrics store

[Service]
Type = forking
GuessMainPID = false
PIDFile = /opt/graphite/storage/carbon-cache-a.pid
ExecStart = /opt/graphite/bin/carbon-cache.py start

[Install]
WantedBy = multi-user.target

sytemctl daemon-reload
systemctl start carbon
systemctl status carbon
systemctl enable carbon

Edit /opt/graphite/webapp/graphite/local_settings.py

SECRET_KEY = '' # Get a random hash to put here
DEBUG = False

Install nginx, gunicorn:

yum install nginx
pip install gunicorn

Setup webapp:

cp /opt/graphite/conf/graphite.wsgi.example /opt/graphite/webapp/wsgi.py
cd /opt/graphite/webapp/graphite
sudo -u graphite python manage.py migrate

Create /etc/systemd/system/graphite-web.service:

[Unit]

[Service]
Environment=PYTHONPATH="/opt/graphite/lib/:/opt/graphite/webapp/"
WorkingDirectory=/opt/graphite/webapp
ExecStart=/usr/bin/gunicorn -u graphite -g graphite -b 127.0.0.1:8080 --log-file=/opt/graphite/storage/log/webapp/gunicorn.log wsgi:application
Restart=on-failure
User=graphite
Group=graphite
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s TERM $MAINPID
PrivateTmp=true

[Install]
WantedBy=multi-user.target

systemctl daemon-reload
systemctl start graphite-web
systemctl status graphite-web
systemctl enable graphite-web

NGINX proxy for graphite-web

Setup a self-signed certificat for nginx right now

cd /etc/nginx
sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/nginx-selfsigned.key -out /etc/ssl/certs/nginx-selfsigned.crt

Create /etc/nginx/conf.d/graphite.conf

server {
    listen 80;
    return 301 https://$host$request_uri;
}

server {

    listen 443;
    server_name graphite.example.com;

    ssl_certificate           /etc/nginx/nginx-selfsigned.crt;
    ssl_certificate_key       /etc/nginx/nginx-selfsigned.key;

    ssl on;
    ssl_session_cache  builtin:1000  shared:SSL:10m;
    ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
    ssl_prefer_server_ciphers on;

    location /graphite/static/ {
        alias /opt/graphite/webapp/content/;
    }

    location /graphite {

      proxy_set_header        Host $host;
      proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header        X-Forwarded-Proto $scheme;

      # Fix the “It appears that your reverse proxy set up is broken" error.
      proxy_pass          http://127.0.0.1:8080/graphite;
      proxy_read_timeout  90;

      proxy_redirect      http://127.0.0.1:8080/graphite https://graphite.example.com/graphite;
    }
  }

Start and open firewall

firewall-cmd --add-service=http
firewall-cmd --add-service=https
firewall-cmd --runtime-to-permanent

systemctl start nginx
systemctl enable nginx

Try collectl locally to sanity check the install

Will use collectl running locally on the same server to make sure that data is getting into graphite.

yum install collectl
# Test it
collectl --export graphite,127.0.0.1
# Configure in /etc/collectl.conf
DaemonCommands = -smcdn --export=graphite,127.0.0.1
# Start
systemctl start collectl
systemctl enable collectl

Now browse to ~https://localhost/graphite~ and check that data is making it into graphite, and the graphite-web interface is working OK.

Grafana is a nicer dashboard front-end than the default graphite-web, so we will use that by default.

yum install https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana-4.2.0-1.x86_64.rpm
systemctl daemon-reload
systemctl start grafana-server
systemctl status grafana-server
systemctl enable grafana-server

Add nginx proxy setup to /etc/nginx/conf.d/graphite.conf

    location / {

      proxy_set_header        Host $host;
      proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header        X-Forwarded-Proto $scheme;

      # Fix the “It appears that your reverse proxy set up is broken" error.
      proxy_pass          http://127.0.0.1:3000/;
      proxy_read_timeout  90;

    }

systemctl restart nginx

Configure Grafana

Gemlog Index

Home