grafana Crypto Stats using Grafana, InfluxDB, Denarius Daemon

[buzzkillb]

How to pull stats from a coin daemon and throw them into influxdb, and then use Grafana to create a pretty graph.

Setup a VM, VPS, raspberry pi, whatever to play on this. Anything breaks its pretty easy to delete stuff. Using ubuntu/debian for the example.

The idea is relatively simple. Ask the daemon for information, throw that into influxdb and then have Grafana pull from that database.

Install denarius daemon
Install python denarius rpc
https://github.com/buzzkillb/python-denariusrpc
Install grafana
https://grafana.com/docs/grafana/latest/installation/debian/
Install influxdb
https://docs.influxdata.com/influxdb/v1.7/introduction/installation/

One this is all done we need a test to see if python is working with the daemon. Switch in rpc_user and rpc_password that's inside of denarius.conf

test.py

from denariusrpc.authproxy import AuthServiceProxy, JSONRPCException

# rpc_user and rpc_password are set in the denarius.conf file
rpc_connection = AuthServiceProxy("http://%s:%[email protected]:32369"%(rpc_user, rpc_password))
best_block_hash = rpc_connection.getbestblockhash()
print(rpc_connection.getblock(best_block_hash))

# batch support : print timestamps of blocks 0 to 99 in 2 RPC round-trips:
commands = [ [ "getblockhash", height] for height in range(100) ]
block_hashes = rpc_connection.batch_(commands)
blocks = rpc_connection.batch_([ [ "getblock", h ] for h in block_hashes ])
block_times = [ block["time"] for block in blocks ]
print(block_times)

test run

python test.py

Lets double check another test run to get current Denarius price from Coinvex.

testcoinvex.py

from denariusrpc.authproxy import AuthServiceProxy, JSONRPCException

import requests, urllib, json

#southexchange
#coinvex
coinvex_url = requests.get('https://coinvex.org/api/v1/public/getlastmarketdata')
coinvex_data = json.loads(coinvex_url.text)
coinvex_price = coinvex_data['result']['coins']
for r in coinvex_price:
    if r["name"] == "Denarius":
        coinvex_last = float(r["price"])
        print format(coinvex_last, '0.8f')

test run

python testcoinvex.py

Works? The next post will show how to take this data and put into influxdb.

[buzzkillb]

Create a new database ininfluxdb. Run influx to get into the command prompt.

influx

create database

CREATE DATABASE mydb

use database

USE mydb

from a second command line, lets make a sample.py

sample.py

from denariusrpc.authproxy import AuthServiceProxy, JSONRPCException

import time
import sys
import datetime
import urllib
import json
from influxdb import InfluxDBClient

# rpc_user and rpc_password are set in the denarius.conf file
rpc_connection = AuthServiceProxy("http://%s:%[email protected]:32369"%("rpc_user", "rpc_password"))

#test
blocktest = rpc_connection.getblockcount()
print(blocktest)

# Configure InfluxDB connection variables
host = "127.0.0.1" # My Ubuntu NUC
port = 8086 # default port
user = "admin" # the user/password created for the pi, with write access
password = "admin" 
dbname = "mydb" # the database we created earlier
interval = 60 # Sample period in seconds

# Create the InfluxDB client object
client = InfluxDBClient(host, port, user, password, dbname)

# think of measurement as a SQL table, it's not...but...
measurement = "measurement"
# location will be used as a grouping tag later
blockchain = "denarius"

blockcount = rpc_connection.getblockcount()
block = rpc_connection.getblockbynumber(blockcount)
grafanatime = block['time'] * 1000000000
hash = block['hash']
size = block['size']
height = block['height']
version = block['version']
merkleroot = block['merkleroot']
mint = int(block['mint'])
timed = block['time']
nonce = block['nonce']
bits = block['bits']
difficulty = float(block['difficulty'])
blocktrust = block['blocktrust']
chaintrust = block['chaintrust']
chainwork = block['chainwork']
previousblockhash = block['previousblockhash']
#nextblockhash = block['nextblockhash']
flags = block['flags']
proofhash = block['proofhash']
entropybit = block['entropybit']
modifier = block['modifier']
modifierchecksum = block['modifierchecksum']

data = [
  {
      "measurement": measurement,
          "tags": {
              "blockchain": blockchain,
          },
          "time": grafanatime,
          "fields": {
             #"block" : i,
              "hash" : hash,
              "size" : size,
	      "height" : height,
	      "version" : version,
	      "merkleroot" : merkleroot,
              "mint" : mint,
	      "time" : timed,
	      "nonce" : nonce,
	      "bits" : bits,
	      "difficulty" : difficulty,
	      "blocktrust" : blocktrust,
	      "chaintrust" : chaintrust,
	      "chainwork" : chainwork,
#	      "nextblockhash" : nextblockhash,
	      "flags" : flags,
	      "proofhash" : proofhash,
	      "entropybit" : entropybit,
	      "modifier" : modifier,
	      "modifierchecksum" : modifierchecksum
              }
          }
        ]
    # Send the JSON data to InfluxDB
print(difficulty)
client.write_points(data)

If you run python sample.py you should see data appear in your influxdb. Run below a few times to throw data in.

python sample.py

In the other command line running influx already, make sure you are using the correct database and lets see if the data is inside.

use mydb
SELECT * FROM /.*/ LIMIT 10

You should see output, now we need to connect influxdb to grafana.

[buzzkillb]

Login to your localhost grafana.

http://192.168.1.337:3000/

on the left click the gear -> data sources

add data source

search for influxdb

use settings like this

save and test

Go back to the grafana homepage and click + and add query

Your query would look something like this. Remember to select your mydb database, my screenshot is using another name for the DB.

Visualization

General

Click Save at the top right

Should see something like this on your homepage.

[buzzkillb]

Lets make sure to run that python file over and over and over on every new block.

Go into denarius.conf and add this line. I have my files in a influx directory.

blocknotify=/home/USERNAME/influx/block.sh

Lets create block.sh in the influx directory and throw our sample.py in there

#!/bin/bash
python /home/USERNAME/influx/sample.py

make block.sh executable

chmod +x block.sh

stop and restart daemon to pick this change up

denariusd stop
denariusd

If it all went right your grafana page should start updating the block count.

[buzzkillb]

reserved

[buzzkillb]

rerserved

[buzzkillb]

To send data into influxdb using multiple variables from a bash script.

curl -i -XPOST 'http://localhost:8086/write?db=stat' --data-binary 'crawler,blockchain=denarius address="'"$ipaddress"'",svcs="'"$svcs"'",version="'"$version"'",version2="'"$version2"'",percent_7d="'"$percent_7d"'",country_code="'"$country_code"'",latitude="'"$latitude"'",longitude="'"$longitude"'"'

Which through some use of bash script grabbing a local dns seeder crawler, can now post to a world map panel.

how the query panel looks, I was playing with the options in the SELECT. GROUP BY is very important to select your tag.

And the Map Data Options

[buzzkillb]

Now that I have the site on Docker, adding some sample docker run commands.

Grafana

docker run -d \
-p 3000:3000 \
--name=grafana \
--user $ID \
--volume "$PWD/data:/var/lib/grafana" \
-e "GF_INSTALL_PLUGINS=grafana-worldmap-panel" \
-e "GF_USERS_VIEWERS_CAN_EDIT=false" \
-e "GF_USERS_EDITORS_CAN_ADMIN=false" \
-e "GF_USERS_ALLOW_SIGN_UP=false" \
-e "GF_USERS_ALLOW_ORG_CREATE=false" \
-e "GF_AUTH_DISABLE_LOGIN_FORM=false" \
-e "GF_AUTH_ANONYMOUS_ENABLED=true" \
-e "GF_AUTH_ANONYMOUS_ORG_ROLE=Viewer" \
-e "GF_ANALYTICS_GOOGLE_ANALYTICS_UA_ID=UA-157676508-1" \
-e "GF_SERVER_DOMAIN=denarius.pro" \
grafana/grafana

I used this for setting up. https://grafana.com/docs/grafana/latest/installation/configure-docker/

This below line to put the grafana shared volume in /home/username/data and have the container start with current user id.

mkdir data # creates a folder for your data
ID=$(id -u) # saves your user id in the ID variable

To pick and choose environment variables, find a stock grafana.ini and for an example go to [users] and then you will find the USERS options in the docker run commands

InfluxDB

--name="influxdb" \
-p 8086:8086 \
-v /home/USERNAME/influxdb:/var/lib/influxdb \
influxdb -config /etc/influxdb/influxdb.conf

To manipulate influxdb inside the container hop into the container using

docker exec -it influxdb /bin/bash

then go into influx cli

influx

and do your typical commands like

show databases

Practice making some edits on both and doing

docker stop grafana
docker stop influxdb
docker start grafana
docker start influxdb
docker stop grafana
docker rm grafana
rerun the full grafana run command

And see persistence will stick.

My assumption is upgrading will just be

docker stop grafana
docker rm grafana
docker pull grafana/grafana
then the full docker run command

Next to get a docker compose together putting it all together in one command.

[buzzkillb]

Slowly editing the github to show how to set this up, but will do a separate guide on blockforums for how to grab coingecko API first, then how to grab daemon info, and display like https://denarius.pro/d/jouEBGkGk/coingecko is showing. https://github.com/buzzkillb/snakes-on-a-chain