GraphiteQuickNotes - henk52/knowledgesharing GitHub Wiki

Table of Contents

Introduction

references

First Questions

  1. How do I install the app?
  2. How do I provide data to it?
  3. Why aren't there any data points?
  4. How do I show data for the past 24 hours
  5. How do I login?
  6. How do I customize the view?
  7. How do I extract data and analyze that data?
  8. How do I provide large amounts of data efficient?

Installation

Installing manually

Installing on RHEL 7

Installing on CentOS 7

  1. remove all references to EPEL repo(2018-02-27 the epel version fails, there should be a fix on the way)
  2. Add 'opstools' to the CentOS repo file
    • [CentOS-opstools]
    • name=CentOS-opstools
    • baseurl=http://dm/mirrors/CentOS/$releasever/opstools/$basearch/
    • gpgcheck=0
    • gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-$releasever
  3. yum install -y graphite-web python-carbon
  4. vi /etc/carbon/storage-schemas.conf
    • Add 'default'
  5. systemctl enable carbon-cache
  6. systemctl start carbon-cache
  7. systemctl status carbon-cache
  8. vi /etc/graphite-web/local_settings.py
    • TIME_ZONE = 'CET'
    • SECRET_KEY = 'UNSAFE_DEFAULT'
  9. PYTHONPATH=/usr/share/graphite/webapp django-admin syncdb --settings=graphite.settings
  10. echo > /etc/httpd/conf.d/welcome.conf
  11. vi /etc/httpd/conf.d/graphite-web.conf
  12. sudo chown apache:apache /var/lib/graphite-web/graphite.db
  13. touch /var/lib/graphite-web/index
  14. sudo systemctl enable httpd
  15. sudo systemctl start httpd
  16. sudo systemctl status httpd
  17. sudo firewall-cmd --permanent --add-service=http
  18. sudo firewall-cmd --permanent --add-port=2003-2004/tcp
  19. sudo firewall-cmd --reload
[default]
pattern = .*
retentions = 10s:4h, 2m:3d, 5m:8d, 13m:32d, 1h:1y
or: retentions = 10s:24h, 1m:14d, 5m:32d, 1h:1y

Troubleshooting the Centos 7 installation

ImportError: cannot import name generic

PYTHONPATH=/usr/share/graphite/webapp django-admin syncdb --settings=graphite.settings
Traceback (most recent call last):
  File "/usr/bin/django-admin", line 5, in <module>
    management.execute_from_command_line()
  File "/usr/lib/python2.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python2.7/site-packages/django/core/management/__init__.py", line 338, in execute
    django.setup()
  File "/usr/lib/python2.7/site-packages/django/__init__.py", line 27, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/usr/lib/python2.7/site-packages/django/apps/registry.py", line 108, in populate
    app_config.import_models()
  File "/usr/lib/python2.7/site-packages/django/apps/config.py", line 202, in import_models
    self.models_module = import_module(models_module_name)
  File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/usr/lib/python2.7/site-packages/graphite/events/models.py", line 6, in <module>
    from tagging.managers import ModelTaggedItemManager
  File "/usr/lib/python2.7/site-packages/tagging/managers.py", line 8, in <module>
    from tagging.models import Tag, TaggedItem
  File "/usr/lib/python2.7/site-packages/tagging/models.py", line 10, in <module>
    from django.contrib.contenttypes import generic
ImportError: cannot import name generic

Tweeking carbon

  • MAX_CREATES_PER_MINUTE = 70
  * /etc/carbon/carbon.conf

collectd

Installing collectd

  1. sudo yum install -y collectd collectd-utils collectd-disk collectd-sensors
  2. sudo vi /etc/collectd.conf
  3. sudo setsebool collectd_tcp_network_connect on
  4. sudo systemctl enable collectd
  5. sudo systemctl start collectd
  6. sudo systemctl status collectd
<Plugin write_graphite>
  <Node "graphing">
    Host "GRAPHITE_HOST_IP"
    Port "2003"
    Protocol "tcp"
    ReconnectInterval 0
    LogSendErrors true
    Prefix "collectd."
#    Postfix "collectd"
    StoreRates true
#    AlwaysAppendDS false
    EscapeCharacter "_"
#    SeparateInstances false
#    PreserveSeparator false
#    DropDuplicateFields false
  </Node>
</Plugin>
<Plugin disk>
        Disk "/^[hs]d[a-f][0-9]?$/"
        IgnoreSelected false
#       UseBSDName false
#       UdevNameAttr "DEVNAME"
</Plugin>

Troubleshooting collectd

write_graphite plugin: Connecting to localhost:2003 via tcp failed. The last error was: failed to connect to remote host: Permission denied

See: https://stackoverflow.com/questions/34787016/collectd-permission-denied-to-connect-to-remote-host

Fix: setsebool collectd_tcp_network_connect on

write_graphite plugin: Connecting to localhost:2003 via tcp failed. The last error was: failed to connect to remote host: Permission denied

Installing Grafana

Installing grafana on Ubuntu

Installing grafana on ubuntu via rpm

  1. wget https://dl.grafana.com/oss/release/grafana_5.4.3_amd64.deb
  2. sudo dpkg -i grafana_5.4.3_amd64.deb
  3. sudo apt install ./grafana_5.4.3_amd64.deb
  4. sudo vi /etc/grafana/grafana.ini
  5. sudo systemctl start grafana-server

Puppet dwerder-graphite

  1. Create the machine instance
  2. connect to the machine
  3. puppet module install dwerder-graphite
  4. update the machine
    1. apt-get update
    2. apt-get upgrade
  5. puppet apply tst.pp
    • Content of tst.pp:
      • class { 'graphite': }

trying out installing via std RPMs

  1. sudo python /usr/lib/python2.7/site-packages/graphite/manage.py syncdb
  2. python /usr/share/graphite/graphite-web.wsgi

Troubleshooting

Starting

# https://doauto.wordpress.com/2013/06/22/how-to-change-a-file-using-puppet/
file_line { 'whitelist':
  path  => '/opt/graphite/webapp/graphite/whitelist/urls.py',
  line  => 'from django.conf.urls import *',
  match => '^from django\.conf\.urls\.defaults import \*',
}
file_line { 'graphite':
  path  => '/opt/graphite/webapp/graphite/urls.py',
  line  => 'from django.conf.urls import *',
  match => '^from django\.conf\.urls\.defaults import \*',
}
file_line { 'render':
  path  => '/opt/graphite/webapp/graphite/render/urls.py',
  line  => 'from django.conf.urls import *',
  match => '^from django\.conf\.urls\.defaults import \*',
}
file_line { 'account':
  path  => '/opt/graphite/webapp/graphite/account/urls.py',
  line  => 'from django.conf.urls import *',
  match => '^from django\.conf\.urls\.defaults import \*',
}
file_line { 'browser':
  path  => '/opt/graphite/webapp/graphite/browser/urls.py',
  line  => 'from django.conf.urls import *',
  match => '^from django\.conf\.urls\.defaults import \*',
}
file_line { 'version':
  path  => '/opt/graphite/webapp/graphite/version/urls.py',
  line  => 'from django.conf.urls import *',
  match => '^from django\.conf\.urls\.defaults import \*',
}
file_line { 'dashboard':
  path  => '/opt/graphite/webapp/graphite/dashboard/urls.py',
  line  => 'from django.conf.urls import *',
  match => '^from django\.conf\.urls\.defaults import \*',
}
file_line { 'composer':
  path  => '/opt/graphite/webapp/graphite/composer/urls.py',
  line  => 'from django.conf.urls import *',
  match => '^from django\.conf\.urls\.defaults import \*',
}
file_line { 'events':
  path  => '/opt/graphite/webapp/graphite/events/urls.py',
  line  => 'from django.conf.urls import *',
  match => '^from django\.conf\.urls\.defaults import \*',
}
file_line { 'metrics':
  path  => '/opt/graphite/webapp/graphite/metrics/urls.py',
  line  => 'from django.conf.urls import *',
  match => '^from django\.conf\.urls\.defaults import \*',
}
file_line { 'graphlot':
  path  => '/opt/graphite/webapp/graphite/graphlot/urls.py',
  line  => 'from django.conf.urls import *',
  match => '^from django\.conf\.urls\.defaults import \*',
}
file_line { 'cli':
  path  => '/opt/graphite/webapp/graphite/cli/urls.py',
  line  => 'from django.conf.urls import *',
  match => '^from django\.conf\.urls\.defaults import \*',
}
 sudo systemctl restart httpd

ImportError: No module named defaults

Fix: See: https://github.com/graphite-project/graphite-web/commit/fc3f018544c19b90cc63797d18970a4cc27ef2ad#diff-e383725a971fca0685db19bfe7c65b32

Delete the '.defaults' from 'from django.conf.urls.defaults import *'

 vi /opt/graphite/webapp/graphite/whitelist/urls.py
 vi /opt/graphite/webapp/graphite/urls.py
 vi /opt/graphite/webapp/graphite/render/urls.py
 vi /opt/graphite/webapp/graphite/account/urls.py
 vi /opt/graphite/webapp/graphite/browser/urls.py
 vi /opt/graphite/webapp/graphite/version/urls.py
 vi /opt/graphite/webapp/graphite/dashboard/urls.py
 vi /opt/graphite/webapp/graphite/composer/urls.py
 vi /opt/graphite/webapp/graphite/events/urls.py
 vi /opt/graphite/webapp/graphite/metrics/urls.py
 vi /opt/graphite/webapp/graphite/graphlot/urls.py
 vi /opt/graphite/webapp/graphite/cli/urls.py
 service apache2 restart

tail -f /opt/graphite/storage/error.log

 File "/opt/graphite/webapp/graphite/urls.py", line 15, in <module>
    from django.conf.urls.defaults import *
 ImportError: No module named defaults

OSError: [Errno] No such file or directory

Seems to not be a problem, running the httpd it works without issue.

  • /var/log/httpd/graphite-web-error.log
 python /usr/share/graphite/graphite-web.wsgi
Traceback (most recent call last):
  File "/usr/share/graphite/graphite-web.wsgi", line 22, in <module>
    import graphite.metrics.search  # noqa
  File "/usr/lib/python2.7/site-packages/graphite/metrics/search.py", line 111, in <module>
    searcher = IndexSearcher(settings.INDEX_FILE)
  File "/usr/lib/python2.7/site-packages/graphite/metrics/search.py", line 17, in __init__
    retcode = subprocess.call(build_index_path)
  File "/usr/lib/python2.7/subprocess.py", line 522, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1335, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

OperationalError: attempt to write a readonly database

The quick fix is to restart the httpd service, that takes care of it.

troubleshooting data problems

No data

https://theholyjava.wordpress.com/2014/05/05/graphite-shows-metrics-but-no-data-troubleshooting/

or it could be a problem of e.g. using a db with 5 second resolution and your data only being sent every 16 seconds. Fix: Send your data 4 times with a 0,5,10,15 second timestamp delta, then it will show up.

'Search' tab in graphite is not active

Must be owned by the HTTPD e.g.

 chown apache:apache /var/lib/graphite-web/index

Missing metrics

(This is not about missing datapoints, but about the metric entry missing)

The metrics are sent but not created.

⚠️ **GitHub.com Fallback** ⚠️