Grafana notification set up - secbuzzer/DetectionEdge GitHub Wiki
Grafana notification set up
In the structure of log collection, the dashboard of Grafana can offer a visual display for users, reducing operational difficulties.
Following the instruction below:
- Log in Grafana
- Cascading data sources
- 2.1 Source 1：metricbeat
- 2.2 Source 2：suricata
- Import dashboard
- 3.1 Import「Monitoring system resources 」dashboard
- 3.2 Import「Intrusion detection alert」dashboard
- Alert set up
- 4.1 Set up 「Monitoring system resources」alert
- 4.2 Set up「Intrusion detection alert」alert
- Set up notifications
- 5.1 Set up a receiver
- 5.2 Set up alert notifications
1. Log in Grafana
Default Dashboard Login Page to 「http://your_ip:13000」, Default login name is admin, password admin, then click「Log In」
If use default login name "admin" and not change default password "admin", it comes up with a notice of「Change Password」. Once click "Skip", users can jump to next step, however, users can also change password in the system.
Home page may show different depends on different versions of Grafana. As can be seen below, it's a home page by version of Grafana v.6.2.5. The bottom right side is installed plugings slightly different.
2. Cascading data sources
Click on the left side menu item 「Configuration」>「Data Sources」page, click 「Add data source」, then add new data source.
Click on Data source menu item, select「Elasticsearch」.
2.1 Sources 1：metricbeat
Add a new data source named 「metricbeat-*」, then enter the information blow:
- Fill 「metricbeat-*」in Name field.
- Fill 「http://your_ip:19200」in URL field.
- In the block of Elasticsearch details, fill 「metricbeat-*」in Index name field.
- Fill 「@timestamp」in Time field name.
- Select Version「7.0+」
Complete and click「Save & Test」. If it appears「Index OK. Time field name OK.」, which means it's just done completely.
2.2 Source 2：suricata
Add one more new source data named 「suricata-*」, then enter the information below：
- Fill 「suricata-*」 in name field.
- Fill 「http://your_ip:19200」in the field of URL.
- In the block of Elasticsearch details，fill 「suricata-*」 in Index name field.
- Fill 「log_time」 in Time field name.
- Select Version「7.0+」
Complete and click「Save & Test」. If it appears「Index OK. Time field name OK.」, which means it's just done completely.
3. Import dashboard
Click on the left side menu item 「Create」>「Import」page, the dashboard can import from GitHub 「SecBuzzerESM」>「grafana_dashboards」.
3.1 Import「System resource monitoring」dashboard
From the page of「Import」, click the button on the top right corner「Upload .json file」, select「Monitoring system resources.json」（the file locates in 「SecBuzzerESM」>「grafana_dashboards」），click the button of 「Import」.
It imported completely as can be seen in the dashboard of「Monitoring system resources」. The data source of dashboard cascading with 「metricbeat-」. If not, please note data source consist with「metricbeat-」from chapter 2 set up instruction.
3.2 Import「Intrusion detection alert」dashboard
From the page of「Import」, click the button on the top right corner 「Upload .json file」, select 「Intrusion detection alert.json」file(the file locates in 「SecBuzzerESM」>「grafana_dashboards」), then click the button of 「Import」.
It imported completely as can be seen in the dashboard of「Monitoring system resources」. The data source of dashboard cascading with 「suricata-」. If not, kindly note data source consist with「suricata-」from chapter 2 set up instruction.
4. Alert Set up
4.1 Set up 「Monitoring system resources」alert
From the dashboard of「Monitoring system resources」, click a chart of「CPU utilization」, open a drag down menu and select 「Edit」
Enter an edit page, click 「Alert」on the left menu to alert edit page, there's a default setup 「alert for the CPU utilization over 10%」
Select on the right button「Test Rule」to test the current state of this alarm trigger citeria.
Return to the dashboard of「Monitoring system resources 」and click 'save', after refreshing the dashboard, it appears 3 unknow status in 「Alert List」: CPU utilization, Disk utilization, Memory utilization:
After one minute, 「Alert List」shows CPU utilization, Disk utilization, Memory utilization in low level, green color stands for normal status, red color(ALERTING) for abnormal.
In the dashboard of 「Monitoring system resources」, the default threshold values relate to CPU utilization, Disk utilization, Memory utilization, users can edit accordingly:
- CPU utilization：Alert while CPU utilization over 10%.
- Disk utilization：Alert while disk utilization over 50%.
- Memory utilization：Alert while memory utilization over 90%.
4.2 Set up「Intrusion detection alert」
Move to the dashboard of「Intrusion detection alert, click a chart of「CPU utilization, open a drag down menu and select 「Edit」.
Enter an edit page, click 「Alert」on the left menu to alert edit page, there's a default setup alert for the number over 2000」.
Return to the dashboard of「Intrusion detection alert 」and click save, after refreshing the dashboard, it appears an unknow status in 「Alert List」: Intrusion detection alert.
After one minute, 「Alert List」shows Intrusion detection alert in low level, green color stands for normal status, red color(ALERTING) for abnormal.
In the dashboard of 「Intrusion detection alert」, the default threshold value relates to Intrusion detection alert, users can edit accordingly:
- 「Intrusion detection alert」triggered: while the number over 2000.
5. Set up notifications
5.1 Set up a receiver
From left side menu 「Alerting」>「Notification channels」, select「Add channel」and add up a new receiver.
Enter the name of receiver in a filed of name, and email in a field of Address, click 「Save」 在 Name
5.2 Set up alert notifications
Set up a receiver in every alert chart, for instance, 「CPU utilization」 of 「System resource monitoring」. Enter edit page, users can find in Notifications section on the Alert tab then send to receiver name and click save.
Recommend set up the alert notifications of the following charts：
- 「CPU utilization」 of 「Monitoring system resources」
- 「Disk utilization」of 「Monitoring system resources」
- 「Memory utilization」of 「Monitoring system resources」
- 「Trend Line Alerts」of 「Intrusion detection alert」
Congratulations! You have successfully completed「Grafana alert notifications set up」.