Menu Close

Monitoring

The monitoring wiki seems outdated; especially the described grafana templates don’t work. There is a project going for updated templates.

I’ve chosen to also install node-exporter, it has a good basis template which gives you information about your server resources with a nice server selector, which I combined with a polkadot data selection.

Setup prometheus

There is no apt repository so you have to install binaries manually

sudo useradd --no-create-home --shell /usr/sbin/nologin prometheus
sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus
sudo chown -R prometheus:prometheus /etc/prometheus
sudo chown -R prometheus:prometheus /var/lib/prometheus
cd /tmp
wget https://github.com/prometheus/prometheus/releases/download/v2.35.0/prometheus-2.35.0.linux-amd64.tar.gz
tar xfz prometheus-2.35.0.linux-amd64.tar.gz
cd prometheus-2.35.0.linux-amd64
sudo cp ./prometheus /usr/local/bin/
sudo cp ./promtool /usr/local/bin/
sudo cp -r ./consoles /etc/prometheus
sudo cp -r ./console_libraries /etc/prometheus
sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool
sudo chown -R prometheus:prometheus /etc/prometheus/consoles
sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries
sudo vi /etc/prometheus/prometheus.yml

/etc/prometheus/prometheus.yml

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  # - "first.rules"
  # - "second.rules"

scrape_configs:
  - job_name: "prometheus"
    scrape_interval: 5s
    static_configs:
      - targets: ["00.stakeworld.nl:9090"]
  - job_name: 'node_exporter'
    scrape_interval: 5s
    static_configs:
      - targets: ['00.stakeworld.nl:9100']
        labels: 
           name: "00.stakeworld.nl"
      - targets: ['01.stakeworld.nl:9100']
        labels: 
           name: "01.stakeworld.nl"
  - job_name: "validators"
    scrape_interval: 5s
    static_configs:
      - targets: ["00.stakeworld.nl:9601"]
        labels: 
           name: "STAKEWORLD/01"
      - targets: ["00.stakeworld.nl:9602"]
        labels: 
           name: "STAKEWORLD/02"
      - targets: ["01.stakeworld.nl:9611"]
        labels: 
           name: "STAKEWORLD/11"
      - targets: ["01.stakeworld.nl:9612"]
        labels: 
           name: "STAKEWORLD/12"

/etc/systemd/system/prometheus.service

[Unit]
  Description=Prometheus Monitoring
  Wants=network-online.target
  After=network-online.target

[Service]
  User=prometheus
  Group=prometheus
  Type=simple
  ExecStart=/usr/local/bin/prometheus \
  --config.file /etc/prometheus/prometheus.yml \
  --storage.tsdb.path /var/lib/prometheus/ \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries
  ExecReload=/bin/kill -HUP $MAINPID

[Install]
  WantedBy=multi-user.target
systemctl daemon-reload 
systemctl start prometheus.service

Setup node-exporter

cd /tmp
wget https://github.com/prometheus/node_exporter/releases/download/v1.3.1/node_exporter-1.3.1.linux-amd64.tar.gz
tar xfz node_exporter-1.3.1.linux-amd64.tar.gz
cd node_exporter-1.3.1.linux-amd64
sudo cp ./node_exporter /usr/local/bin

/etc/systemd/system/node-export.service

[Unit]
  Description=Prometheus Node Export
  Wants=network-online.target
  After=network-online.target

[Service]
  User=prometheus
  Group=prometheus
  Type=simple
  ExecStart=/usr/local/bin/node_exporter
  ExecReload=/bin/kill -HUP $MAINPID

[Install]
  WantedBy=multi-user.target
systemctl daemon-reload 
systemctl start node-export.service
systemctl enable node-export.service

Setup grafana

The wiki uses binaries but there is a apt repository available. To install the latest grafana server

sudo apt-get install -y apt-transport-https
sudo apt-get install -y software-properties-common wget
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
sudo apt-get update
sudo apt-get install grafana
sudo systemctl daemon-reload
sudo systemctl start grafana-server
sudo systemctl status grafana-server
sudo systemctl enable grafana-server.service

If you are running behind a proxy adjust grafana config:

/etc/grafana/grafana.ini
[server]
domain = stakeworld.nl
root_url = %(protocol)s://%(domain)s:%(http_port)s/grafana/
serve_from_sub_path = true

Grafana dashboard json

My config: stakeworld.json

Leave a Reply

Your email address will not be published.