Influxdb troubleshooting using python - ksator/junos_monitoring_with_healthbot GitHub Wiki

InfluxDB is an open source time series database written in GO.
Healthbot uses InfluxDB (docker container).
The data collected by Healthbot from network devices is stored in InfluxDB.

In the below demo, we will use Python to configure healthbot. healthbot will collect data from the network and save the data collected in Influxdb. We will then use python to query the database (this can be useful for troubleshooting purpose)

Configure healthbot to collect some data from network devices

edit the file python_input.yml to indicate healtbot ip address and credentials

$ vi  python_input.yml

Run this command to configure healthbot with the details described in the file python_input.yml

$ python ./configure_healthbot.py

so, healthbot is now configured with
- devices vMX1 to vMX7
- The rule check-bgp-state-using-netconf.
  - This rule uses an iagent sensor with the table and view bgp_sessions_state.yml
- The rule check-bgp-state-using-snmp.
  - This rule uses an SNMP sensor.
- The playbook bgp-using-netconf
  - This playbook uses the rule check-bgp-state-using-netconf
- The playbook bgp-using-snmp
  - This playbook uses the rule check-bgp-state-using-snmp
- The device group Group1 is configured with
  - devices vMX1 to vMX7
  - an instance of the playbook bgp-using-netconf
- The device group Group2 is configured with
  - devices vMX3 and vMX4
  - an instance of the playbook bgp-using-snmp

Demo requirements

install the influxdb python library.
This python library is a client for interacting with InfluxDB.

$ pip install influxdb

Verify

$ pip list | grep influx

you can now interact with InfluxDB using Python

Read InfluxDB data using Python

run this command to start a python interactive session

$ python

connect to InfluxDB using Python.

8086 is the InfluxDB port.

>>> from influxdb import InfluxDBClient
>>> influx_client = InfluxDBClient('100.123.35.0',8086)

list the databases

>>> influx_client.query('show databases')
ResultSet({'(u'databases', None)': [{u'name': u'_internal'}, {u'name': u'Group1_vMX1'}, {u'name': u'Group1_vMX2'}, {u'name': u'Group1_vMX3'}, {u'name': u'Group1_vMX4'}, {u'name': u'Group1_vMX5'}, {u'name': u'Group1_vMX6'}, {u'name': u'Group1_vMX7'}, {u'name': u'Group2_vMX3'}, {u'name': u'Group2_vMX4'}]})

There is one database per device group/device pair.

>>> gp=influx_client.query('show databases').get_points()
>>> for item in gp:
...     print item
...
{u'name': u'_internal'}
{u'name': u'Group1_vMX1'}
{u'name': u'Group1_vMX2'}
{u'name': u'Group1_vMX3'}
{u'name': u'Group1_vMX4'}
{u'name': u'Group1_vMX5'}
{u'name': u'Group1_vMX6'}
{u'name': u'Group1_vMX7'}
{u'name': u'Group2_vMX3'}
{u'name': u'Group2_vMX4'}
>>>

list measurements for a database

>>> influx_client.query('show measurements', database='Group1_vMX1')
ResultSet({'(u'measurements', None)': [{u'name': u'bgp/check-bgp-state-using-netconf'}, {u'name': u'bgp/check-bgp-state-using-netconf/neighborship'}]})
>>>

bgp is a healthbot topic.
check-bgp-state-using-netconf is a healthbot rule.
neighborship is a trigger for the rule check-bgp-state-using-netconf

query data from a particular measurement and database

>>> influx_client.query('select * from "bgp/check-bgp-state-using-netconf"  order by desc limit 10 ', database='Group1_vMX1')
ResultSet({'(u'bgp/check-bgp-state-using-netconf', None)': [{u'_playbook-name': u'bgp-using-netconf', u'hostname': u'100.123.1.0', u'peer-address': u'11.0.0.104+53942', u'state': u'Established', u'time': u'2018-11-22T16:04:16.833170463Z', u'_instance-id': u'["bgp_using_netconf_intance_1"]'}, {u'_playbook-name': u'bgp-using-netconf', u'hostname': u'100.123.1.0', u'peer-address': u'11.0.0.103+55504', u'state': u'Established', u'time': u'2018-11-22T16:04:16.833170463Z', u'_instance-id': u'["bgp_using_netconf_intance_1"]'}, {u'_playbook-name': u'bgp-using-netconf', u'hostname': u'100.123.1.0', u'peer-address': u'11.0.0.102+51849', u'state': u'Established', u'time': u'2018-11-22T16:04:16.833170463Z', u'_instance-id': u'["bgp_using_netconf_intance_1"]'}, {u'_playbook-name': u'bgp-using-netconf', u'hostname': u'100.123.1.0', u'peer-address': u'11.0.0.104+53942', u'state': u'Established', u'time': u'2018-11-22T16:04:00.797173422Z', u'_instance-id': u'["bgp_using_netconf_intance_1"]'}, {u'_playbook-name': u'bgp-using-netconf', u'hostname': u'100.123.1.0', u'peer-address': u'11.0.0.103+55504', u'state': u'Established', u'time': u'2018-11-22T16:04:00.797173422Z', u'_instance-id': u'["bgp_using_netconf_intance_1"]'}, {u'_playbook-name': u'bgp-using-netconf', u'hostname': u'100.123.1.0', u'peer-address': u'11.0.0.102+51849', u'state': u'Established', u'time': u'2018-11-22T16:04:00.797173422Z', u'_instance-id': u'["bgp_using_netconf_intance_1"]'}, {u'_playbook-name': u'bgp-using-netconf', u'hostname': u'100.123.1.0', u'peer-address': u'11.0.0.104+53942', u'state': u'Established', u'time': u'2018-11-22T16:03:44.864965953Z', u'_instance-id': u'["bgp_using_netconf_intance_1"]'}, {u'_playbook-name': u'bgp-using-netconf', u'hostname': u'100.123.1.0', u'peer-address': u'11.0.0.103+55504', u'state': u'Established', u'time': u'2018-11-22T16:03:44.864965953Z', u'_instance-id': u'["bgp_using_netconf_intance_1"]'}, {u'_playbook-name': u'bgp-using-netconf', u'hostname': u'100.123.1.0', u'peer-address': u'11.0.0.102+51849', u'state': u'Established', u'time': u'2018-11-22T16:03:44.864965953Z', u'_instance-id': u'["bgp_using_netconf_intance_1"]'}, {u'_playbook-name': u'bgp-using-netconf', u'hostname': u'100.123.1.0', u'peer-address': u'11.0.0.104+53942', u'state': u'Established', u'time': u'2018-11-22T16:03:27.826324718Z', u'_instance-id': u'["bgp_using_netconf_intance_1"]'}]})
>>>

>>> r = influx_client.query('select * from "bgp/check-bgp-state-using-netconf"  order by desc limit 10 ', database='Group1_vMX1')

>>> for item in r.raw['series'][0]['values']:
...    print item
...
[u'2018-11-22T16:05:05.855021302Z', u'["bgp_using_netconf_intance_1"]', u'bgp-using-netconf', u'100.123.1.0', u'11.0.0.104+53942', u'Established']
[u'2018-11-22T16:05:05.855021302Z', u'["bgp_using_netconf_intance_1"]', u'bgp-using-netconf', u'100.123.1.0', u'11.0.0.103+55504', u'Established']
[u'2018-11-22T16:05:05.855021302Z', u'["bgp_using_netconf_intance_1"]', u'bgp-using-netconf', u'100.123.1.0', u'11.0.0.102+51849', u'Established']
[u'2018-11-22T16:04:49.820096971Z', u'["bgp_using_netconf_intance_1"]', u'bgp-using-netconf', u'100.123.1.0', u'11.0.0.104+53942', u'Established']
[u'2018-11-22T16:04:49.820096971Z', u'["bgp_using_netconf_intance_1"]', u'bgp-using-netconf', u'100.123.1.0', u'11.0.0.103+55504', u'Established']
[u'2018-11-22T16:04:49.820096971Z', u'["bgp_using_netconf_intance_1"]', u'bgp-using-netconf', u'100.123.1.0', u'11.0.0.102+51849', u'Established']
[u'2018-11-22T16:04:33.776654272Z', u'["bgp_using_netconf_intance_1"]', u'bgp-using-netconf', u'100.123.1.0', u'11.0.0.104+53942', u'Established']
[u'2018-11-22T16:04:33.776654272Z', u'["bgp_using_netconf_intance_1"]', u'bgp-using-netconf', u'100.123.1.0', u'11.0.0.103+55504', u'Established']
[u'2018-11-22T16:04:33.776654272Z', u'["bgp_using_netconf_intance_1"]', u'bgp-using-netconf', u'100.123.1.0', u'11.0.0.102+51849', u'Established']
[u'2018-11-22T16:04:16.833170463Z', u'["bgp_using_netconf_intance_1"]', u'bgp-using-netconf', u'100.123.1.0', u'11.0.0.104+53942', u'Established']
>>>

>>> for item in r.raw['series'][0]['columns']:
...    print item
...
time
_instance-id
_playbook-name
hostname
peer-address
state
>>>

>>> gp = influx_client.query('select * from "bgp/check-bgp-state-using-netconf"  order by desc limit 10 ', database='Group1_vMX1').get_points()
>>> for item in gp:
...     print item
...
{u'_playbook-name': u'bgp-using-netconf', u'hostname': u'100.123.1.0', u'peer-address': u'11.0.0.104+53942', u'state': u'Established', u'time': u'2018-11-22T16:08:53.774693019Z', u'_instance-id': u'["bgp_using_netconf_intance_1"]'}
{u'_playbook-name': u'bgp-using-netconf', u'hostname': u'100.123.1.0', u'peer-address': u'11.0.0.103+55504', u'state': u'Established', u'time': u'2018-11-22T16:08:53.774693019Z', u'_instance-id': u'["bgp_using_netconf_intance_1"]'}
{u'_playbook-name': u'bgp-using-netconf', u'hostname': u'100.123.1.0', u'peer-address': u'11.0.0.102+51849', u'state': u'Established', u'time': u'2018-11-22T16:08:53.774693019Z', u'_instance-id': u'["bgp_using_netconf_intance_1"]'}
{u'_playbook-name': u'bgp-using-netconf', u'hostname': u'100.123.1.0', u'peer-address': u'11.0.0.104+53942', u'state': u'Established', u'time': u'2018-11-22T16:08:37.830969538Z', u'_instance-id': u'["bgp_using_netconf_intance_1"]'}
{u'_playbook-name': u'bgp-using-netconf', u'hostname': u'100.123.1.0', u'peer-address': u'11.0.0.103+55504', u'state': u'Established', u'time': u'2018-11-22T16:08:37.830969538Z', u'_instance-id': u'["bgp_using_netconf_intance_1"]'}
{u'_playbook-name': u'bgp-using-netconf', u'hostname': u'100.123.1.0', u'peer-address': u'11.0.0.102+51849', u'state': u'Established', u'time': u'2018-11-22T16:08:37.830969538Z', u'_instance-id': u'["bgp_using_netconf_intance_1"]'}
{u'_playbook-name': u'bgp-using-netconf', u'hostname': u'100.123.1.0', u'peer-address': u'11.0.0.104+53942', u'state': u'Established', u'time': u'2018-11-22T16:08:21.799565899Z', u'_instance-id': u'["bgp_using_netconf_intance_1"]'}
{u'_playbook-name': u'bgp-using-netconf', u'hostname': u'100.123.1.0', u'peer-address': u'11.0.0.103+55504', u'state': u'Established', u'time': u'2018-11-22T16:08:21.799565899Z', u'_instance-id': u'["bgp_using_netconf_intance_1"]'}
{u'_playbook-name': u'bgp-using-netconf', u'hostname': u'100.123.1.0', u'peer-address': u'11.0.0.102+51849', u'state': u'Established', u'time': u'2018-11-22T16:08:21.799565899Z', u'_instance-id': u'["bgp_using_netconf_intance_1"]'}
{u'_playbook-name': u'bgp-using-netconf', u'hostname': u'100.123.1.0', u'peer-address': u'11.0.0.104+53942', u'state': u'Established', u'time': u'2018-11-22T16:08:04.761004836Z', u'_instance-id': u'["bgp_using_netconf_intance_1"]'}
>>>

Print the sessions state

>>> gp = influx_client.query('select * from "bgp/check-bgp-state-using-netconf"  order by desc limit 10 ', database='Group1_vMX1').get_points()
>>> for item in gp:
...     print item['state']
...
Established
Established
Established
Established
Established
Established
Established
Established
Established
Established
>>>

>>> r=influx_client.query('select * from "bgp/check-bgp-state-using-netconf/neighborship"  order by desc limit 10 ', database='Group1_vMX1')

>>> for item in r.raw['series'][0]['values']:
...    print item
...
[u'2018-11-22T16:11:25Z', u'["bgp_using_netconf_intance_1"]', u'bgp-using-netconf', u'green', u'100.123.1.0', u'BGP neighbor 11.0.0.104+53942 is in Established state', u'11.0.0.104+53942', True, u'Established']
[u'2018-11-22T16:11:25Z', u'["bgp_using_netconf_intance_1"]', u'bgp-using-netconf', u'green', u'100.123.1.0', u'BGP neighbor 11.0.0.103+55504 is in Established state', u'11.0.0.103+55504', True, u'Established']
[u'2018-11-22T16:11:25Z', u'["bgp_using_netconf_intance_1"]', u'bgp-using-netconf', u'green', u'100.123.1.0', u'BGP neighbor 11.0.0.102+51849 is in Established state', u'11.0.0.102+51849', True, u'Established']
[u'2018-11-22T16:10:55Z', u'["bgp_using_netconf_intance_1"]', u'bgp-using-netconf', u'green', u'100.123.1.0', u'BGP neighbor 11.0.0.104+53942 is in Established state', u'11.0.0.104+53942', True, u'Established']
[u'2018-11-22T16:10:55Z', u'["bgp_using_netconf_intance_1"]', u'bgp-using-netconf', u'green', u'100.123.1.0', u'BGP neighbor 11.0.0.103+55504 is in Established state', u'11.0.0.103+55504', True, u'Established']
[u'2018-11-22T16:10:55Z', u'["bgp_using_netconf_intance_1"]', u'bgp-using-netconf', u'green', u'100.123.1.0', u'BGP neighbor 11.0.0.102+51849 is in Established state', u'11.0.0.102+51849', True, u'Established']
[u'2018-11-22T16:10:25Z', u'["bgp_using_netconf_intance_1"]', u'bgp-using-netconf', u'green', u'100.123.1.0', u'BGP neighbor 11.0.0.104+53942 is in Established state', u'11.0.0.104+53942', True, u'Established']
[u'2018-11-22T16:10:25Z', u'["bgp_using_netconf_intance_1"]', u'bgp-using-netconf', u'green', u'100.123.1.0', u'BGP neighbor 11.0.0.103+55504 is in Established state', u'11.0.0.103+55504', True, u'Established']
[u'2018-11-22T16:10:25Z', u'["bgp_using_netconf_intance_1"]', u'bgp-using-netconf', u'green', u'100.123.1.0', u'BGP neighbor 11.0.0.102+51849 is in Established state', u'11.0.0.102+51849', True, u'Established']
[u'2018-11-22T16:09:55Z', u'["bgp_using_netconf_intance_1"]', u'bgp-using-netconf', u'green', u'100.123.1.0', u'BGP neighbor 11.0.0.104+53942 is in Established state', u'11.0.0.104+53942', True, u'Established']

>>> for item in r.raw['series'][0]['columns']:
...    print item
...
time
_instance-id
_playbook-name
color
hostname
message
peer-address
session_is_established_when_1
state

>>> gp=influx_client.query('select * from "bgp/check-bgp-state-using-netconf/neighborship"  order by desc limit 10 ', database='Group1_vMX1').get_points()
>>> for item in gp:
...     print item['message']
...
BGP neighbor 11.0.0.104+53942 is in Established state
BGP neighbor 11.0.0.103+55504 is in Established state
BGP neighbor 11.0.0.102+51849 is in Established state
BGP neighbor 11.0.0.104+53942 is in Established state
BGP neighbor 11.0.0.103+55504 is in Established state
BGP neighbor 11.0.0.102+51849 is in Established state
BGP neighbor 11.0.0.104+53942 is in Established state
BGP neighbor 11.0.0.103+55504 is in Established state
BGP neighbor 11.0.0.102+51849 is in Established state
BGP neighbor 11.0.0.104+53942 is in Established state
>>>