http://docs.graylog.org/en/2.1/pages/installation/os/centos.html

http://www.3mc2.com/the-turorail-install-graylog2-under-centos7-linux.html

https://testerhome.com/topics/3026

1)服务启动： [root@localhost ~]# systemctl start mongod [root@localhost ~]# systemctl start elasticsearch [root@localhost ~]# systemctl start graylog-server 2) 开机自启动 [root@localhost ~]# systemctl enable mongod [root@localhost ~]# systemctl enable elasticsearch [root@localhost ~]# systemctl enable graylog-server 3)防火墙配置 [root@localhost ~]# firewall-cmd --zone=public --add-port=9000/tcp --permanent success [root@localhost ~]# firewall-cmd --zone=public --add-port=12900/tcp --permanent success [root@localhost ~]# firewall-cmd --reload success

1. Elasticsearch 2.x (2.1.0 or later) 1.1.下载tar包，解压。 1.2.编辑修改主机地址：config/elasticsearch.yml。 1.3.Run bin/elasticsearch 1.4.curl http://ip:9200/ 1.5.插件 https://my.oschina.net/imi/blog/754168 plugin install license plugin install marvel-agent 1.6.kibana 1.6.1.kibana plugin --install elasticsearch/marvel/latest 1.6.2.编辑elasticsearch-2.4.0\config\elasticsearch.yml，在最后一行添加： marvel.agent.exporters: id1: type: http host: ["http://127.0.0.1:9200"] ，启动elasticsearch. 1.6.3.编辑kibana\config\kibana.yaml，配置 elasticsearch.url: "http://localhost:9200" ，然后启动kibana即可。 1.6.4.访问http://127.0.0.1:5601/app/marvel点击你的节点，即可看到统计界面。

MongoDB Installing MongoDB on CentOS should follow the tutorial for RHEL and CentOS from the MongoDB documentation. First add the repository file /etc/yum.repos.d/mongodb-org-3.2.repo with the following contents:

[mongodb-org-3.2] name=MongoDB Repository baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/3.2/x86_64/ gpgcheck=1 enabled=1 gpgkey=https://www.mongodb.org/static/pgp/server-3.2.asc After that, install the latest release of MongoDB with sudo yum install mongodb-org.

Additionally, run these last steps to start MongoDB during the operating system’s boot and start it right away:

$ sudo chkconfig --add mongod $ sudo systemctl daemon-reload $ sudo systemctl enable mongod.service $ sudo systemctl start mongod.service Elasticsearch Graylog 2.0.0 and higher requires Elasticsearch 2.x, so we took the installation instructions from the Elasticsearch installation guide.

First install the Elastic GPG key with rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch then add the repository file /etc/yum.repos.d/elasticsearch.repo with the following contents:

[elasticsearch-2.x] name=Elasticsearch repository for 2.x packages baseurl=https://packages.elastic.co/elasticsearch/2.x/centos gpgcheck=1 gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch enabled=1 followed by the installation of the latest release with sudo yum install elasticsearch.

Make sure to modify the Elasticsearch configuration file (/etc/elasticsearch/elasticsearch.yml) and set the cluster name to graylog:

cluster.name: graylog After you have modified the configuration, you can start Elasticsearch:

$ sudo chkconfig --add elasticsearch $ sudo systemctl daemon-reload $ sudo systemctl enable elasticsearch.service $ sudo systemctl restart elasticsearch.service Graylog Now install the Graylog repository configuration and Graylog itself with the following commands:

$ sudo rpm -Uvh https://packages.graylog2.org/repo/packages/graylog-2.1-repository_latest.rpm $ sudo yum install graylog-server Follow the instructions in your /etc/graylog/server/server.conf and add password_secret and root_password_sha2. These settings are mandatory and without them, Graylog will not start!

You need to use the following command to create your root_password_sha2:

echo -n yourpassword | sha256sum To be able to connect to Graylog you should set rest_listen_uri and web_listen_uri to the public host name or a public IP address of the machine you can connect to. More information about these settings can be found in Configuring the web interface.

Note

If you’re operating a single-node setup and would like to use HTTPS for the Graylog web interface and the Graylog REST API, it’s possible to use NGINX or Apache as a reverse proxy. The last step is to enable Graylog during the operating system’s startup:

$ sudo chkconfig --add graylog-server $ sudo systemctl daemon-reload $ sudo systemctl enable graylog-server.service $ sudo systemctl start graylog-server.service

1.安装 https://my.oschina.net/imi/blog/754168

pwgen -N 1 -s 96

graylog 测试

1，TCP报文日志

# echo date | nc 192.168.1.135 33333

任何安装nc的Linux机器上执行：

echo date | nc 10.0.0.11 5555 浏览器的http://10.0.0.11:9000/登录后首页 ，点击第三行绿色搜索按钮，看到一条新消息：

Timestamp Source Message 2015-05-22 08:49:15.280 10.0.0.157 2015年 05月 22日 星期五 16:48:28 CST 说明安装已成功！！

4.2 进入 System > Inputs > Inputs in Cluster > GELF HTTP | Launch new input 取名"http 12201" 完成创建 任何安装curl的Linux机器上执行：

curl -XPOST http://10.0.0.11:12201/gelf -p0 -d '{"short_message":"Hello there", "host":"example.org", "facility":"test", "_foo":"bar"}' 浏览器的http://10.0.0.11:9000/登录后首页 ，点击第三行绿色搜索按钮，看到一条新消息：

Timestamp Source Message 2015-05-22 08:49:15.280 10.0.0.157 Hello there 说明GELF HTTP Input设置成功！！

发送日志到Graylog服务器

使用http协议发送：

http://docs.graylog.org/en/1.1/pages/sending_data.html#gelf-via-http

curl -XPOST http://graylog.example.org:12202/gelf -p0 -d '{"short_message":"Hello there", "host":"example.org", "facility":"test", "_foo":"bar"}' 使用tcp协议发送

http://docs.graylog.org/en/1.1/pages/sending_data.html#raw-plaintext-inputs

echo "hello, graylog" | nc graylog.example.org 5555 结合inotifywait收集nginx日志

gather-nginx-log.sh

#!/bin/bash app=nginx node=$HOSTNAME log_file=/var/log/nginx/nginx.log graylog_server_ip=10.0.0.11 graylog_server_port=12201

while inotifywait -e modify $log_file; do last_size=cat ${app}.size curr_size=stat -c%s $log_file echo $curr_size > ${app}.size count=echo "$curr_size-$last_size" | bc python read_log.py $log_file ${last_size} $count | sed 's/"/\\"/g' > ${app}.new_lines while read line do if echo "$line" | grep "^20[0-9][0-9]-[0-1][0-9]-[0-3][0-9]" > /dev/null; then seconds=echo "$line" | cut -d ' ' -f 6 spend_ms=echo "${seconds}*1000/1" | bc http_status=echo "$line" | cut -d ' ' -f 2 echo "http_status -- $http_status" prefix_number=${http_status:0:1} if [ "$prefix_number" == "5" ]; then level=3 #ERROR elif [ "$prefix_number" == "4" ]; then level=4 #WARNING elif [ "$prefix_number" == "3" ]; then level=5 #NOTICE elif [ "$prefix_number" == "2" ]; then level=6 #INFO elif [ "$prefix_number" == "1" ]; then level=7 #DEBUG fi echo "level -- $level" curl -XPOST http://${graylog_server_ip}:${graylog_server_port}/gelf -p0 -d "{"short_mess sage":"$line", "host":"${app}", "level":${level}, "_node":"${node}", "_spend_msecs":$ {spend_ms}, "_http_status":${http_status}}" echo "gathered -- $line" fi done < ${app}.new_lines done
read_log.py

#!/usr/bin/python #coding=utf-8 import sys import os

if len(sys.argv) < 4: print "Usage: %s /path/of/log/file print_from count" % (sys.argv[0]) print "Example: %s /var/log/syslog 90000 100" % (sys.argv[0]) sys.exit(1)

filename = sys.argv[1] if (not os.path.isfile(filename)): print "%s not existing!!!" % (filename) sys.exit(1)

filesize = os.path.getsize(filename)

position = int(sys.argv[2]) if (filesize < position): print "log file may cut by logrotate.d, print log from begin!" % (position,filesize) position = 0

count = int(sys.argv[3]) fo = open(filename, "r")

fo.seek(position, 0) content = fo.read(count) print content.strip()

Close opened file

fo.close() 5秒一次收集iotop日志，找出高速读写磁盘的进程

#!/bin/bash app=iotop node=$HOSTNAME graylog_server_ip=10.0.0.11 graylog_server_port=12201

while true; do sudo /usr/sbin/iotop -b -o -t -k -q -n2 | sed 's/"/\\"/g' > /dev/shm/graylog_client.${app}.new_lines while read line; do if echo "$line" | grep "^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]" > /dev/null; then read -a WORDS <<< $line epoch_seconds=date --date="${WORDS[0]}" +%s.%N pid=${WORDS[1]} read_float_kps=${WORDS[4]} read_int_kps=${read_float_kps%.} write_float_kps=${WORDS[6]} write_int_kps=${write_float_kps%.}

        command=${WORDS[12]}
        if [ "$command" == "bash" ] && (( ${#WORDS[*]} > 13 )); then
            pname=${WORDS[13]}
        elif [ "$command" == "java" ] && (( ${#WORDS[*]} > 13 )); then
            arg0=${WORDS[13]} 
            pname=${arg0#*=}
        else
            pname=$command
        fi

        curl --connect-timeout 1 -s -XPOST http://${graylog_server_ip}:${graylog_server_port}/gelf -p0 -d "{\"timestamp\":$epoch_seconds, \"short_message\":\"${line::200}\", \"full_message\":\"$line\", \"host\":\"${app}\", \"_node\":\"${node}\", \"_pid\":${pid}, \"_read_kps\":${read_int_kps}, \"_write_kps\":${write_int_kps}, \"_pname\":\"${pname}\"}"
    fi 
done < /dev/shm/graylog_client.${app}.new_lines
sleep 4 

done 收集android app日志

device.env

export device=4b13c85c export app=com.tencent.mm export filter="( I/ServerAsyncTask2(| W/| E/)"

export graylog_server_ip=10.0.0.11 export graylog_server_port=12201 adblog.sh

#!/bin/bash . ./device.env adb -s $device logcat -v time *:I | tee -a adb.log gather-androidapp-log.sh

#!/bin/bash . ./device.env log_file=./adb.log node=$device

if [ ! -f $log_file ]; then echo $log_file not exist!! echo 0 > ${app}.size exit 1 fi

if [ ! -f ${app}.size ]; then curr_size=stat -c%s $log_file echo $curr_size > ${app}.size fi while inotifywait -qe modify $log_file > /dev/null; do last_size=cat ${app}.size curr_size=stat -c%s $log_file echo $curr_size > ${app}.size pids=./getpids.py $app $device if [ "$pids" == "" ]; then continue fi count=echo "$curr_size-$last_size" | bc python read_log.py $log_file ${last_size} $count | grep "$pids" | sed 's/"/\\"/g' | sed 's/\t/　　　　/g' > ${app}.new_lines #echo "${app}.new_lines lines: wc -l ${app}.new_lines" while read line do if echo "$line" | grep "$filter" > /dev/null; then priority=${line:19:1} if [ "$priority" == "F" ]; then level=1 #ALERT elif [ "$priority" == "E" ]; then level=3 #ERROR elif [ "$priority" == "W" ]; then level=4 #WARNING elif [ "$priority" == "I" ]; then level=6 #INFO fi #echo "level -- $level" curl -XPOST http://${graylog_server_ip}:${graylog_server_port}/gelf -p0 -d "{"short_message":"$line", "host":"${app}", "level":${level}, "_node":"${node}"}" echo "GATHERED -- $line" #else #echo "ignored -- $line" fi done < ${app}.new_lines done get_pids.py

#!/usr/bin/python import sys import os import commands

if name == "main": if len(sys.argv) != 3: print sys.argv[0]+" packageName device" sys.exit() device = sys.argv[2] cmd = "adb -s "+device+" shell ps | grep "+sys.argv[1]+" | cut -c11-15" output = commands.getoutput(cmd) if output == "": sys.exit() originpids = output.split("\n") strippids = map((lambda pid: int(pid,10)), originpids) pids = map((lambda pid: "%5d" %pid), strippids) pattern = "(("+")|(".join(pids)+"))" print pattern

log4j https://marketplace.graylog.org/addons/a3c8bbdf-6b47-43fe-8298-f259db85fc59

日志采集方案1：日志文件->(kafka)->Graylog。

1.应用使用log4j将异常及其他信息，输出到日志文件。

2.nxlog等监听并提交到kafka，再由graylog由kafka取回（首选）。或者nxlog等监听并直接提交日志到graylog。

日志采集方案2：log4j->kafka/Graylog。

1.应用使用log4j kafka appender将异常及其他信息，直接输出到kafka，由graylog从kafka取回（首选）。

2.或者使用gelf插件直接提交日志到graylog。

注意：

log4j可直接输出到kafka，但kafka故障时，log4j会等待超时，影响应用响应。使用异步方式，会丢失数据。

生产环境和测试环境应使用相同方案。

问题： ####elasticsearch [2016-11-24 14:03:54,625][WARN ][discovery.zen ] [elasticsearch-node-1] failed to validate incoming join request from node [{graylog-101c3f0e-4d20-4b1a-b985-ae36cfa39754} {MWjq9_SXSrS0me3-4lsSmA} {192.168.252.129} {192.168.252.129:9350} {client=true, data=false, master=false}]

###graylog 2016-11-24T14:02:45.358+08:00 INFO [zen] [graylog-101c3f0e-4d20-4b1a-b985-ae36cfa39754] failed to send join request to master [{elasticsearch-node-1}{iBn7tmvUQeiVEWIZIb0Xwg}{192.168.252.129}{192.168.252.129:9300}{master=true}], reason [RemoteTransportException[elasticsearch-node-1][192.168.252.129:9300]internal:discovery/zen/join; nested: IllegalStateException[failure when sending a validation request to node]; nested: RemoteTransportException[graylog-101c3f0e-4d20-4b1a-b985-ae36cfa39754][192.168.252.129:9350]internal:discovery/zen/join/validate; nested: IllegalArgumentException[No custom metadata prototype registered for type [licenses], node like missing plugins]; ]

解决：https://github.com/Graylog2/graylog2-server/issues/2286

cp -R /usr/share/elasticsearch/plugins /usr/share/graylog-server/els.plugins

cd /usr/share/graylog-server/els.plugins

cd license

rm license*.jar

wget https://maven.elasticsearch.org/public-releases/org/elasticsearch/plugin/license/2.3.5/license-2.3.5.jar

wget https://maven.elasticsearch.org/releases/org/elasticsearch/license-plugin-api/2.3.5/license-plugin-api-2.3.5.jar

wget https://maven.elasticsearch.org/releases/org/elasticsearch/license-core/2.3.5/license-core-2.3.5.jar

vi plugin-descriptor.properties

version=2.3.5 elasticsearch.version=2.3.5

cd marvel-agent

rm marvel-agent-*.jar

wget https://maven.elasticsearch.org/public-releases/org/elasticsearch/plugin/marvel-agent/2.3.5/marvel-agent-2.3.5.jar

vi plugin-descriptor.properties

version=2.3.5 elasticsearch.version=2.3.5

vi /etc/graylog/server/server.conf

uncomment elasticsearch_config_file = /etc/graylog/server/elasticsearch.yml

cp /etc/elasticsearch/elasticsearch.yml /etc/graylog/server/elasticsearch.yml

vi elasticsearch.yml

change the following:

node.name: graylog-nodexx path.plugins: /usr/share/graylog-server/els.plugins node.master: false node.data: false bootstrap.mlockall: false

chmod 755 /etc/graylog/server/elasticsearch.yml

restart graylog-server and watch the logfile

问题： org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized]

解决：

I got the setup to run with the following additions -

/etc/elasticsearch/elasticsearch.yml

cluster.name: graylog2 node.master: true node.data: true bootstrap.mlockall: true ES_HEAP_SIZE: 8192 (@ 16GB memory) discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicasts.hosts: [“127.0.0.1”, "SERVER IP"]

/etc/graylog2.conf

elasticsearch_discovery_zen_ping_multicast_enabled = false elasticsearch_discovery_zen_ping_unicast_hosts = IP_ARR:9300

#vi /etc/security/limits.conf

allow user 'elasticsearch' mlockall

elasticsearch soft memlock unlimited elasticsearch hard memlock unlimited

http://192.168.252.129:9200/_cat/nodes

graylog各种日志采集方式总结和验证。log4j2、 java代码、kafka、nginx。 log4j可能存在占用连接等待问题（graylog故障）。 kafka更为理想。

rsyslog

[root@centos7-21 rsyslog.d]# pwd /etc/rsyslog.d [root@centos7-21 rsyslog.d]# ll total 8 -rw-r--r-- 1 root root 161 Nov 23 2016 graylog.conf -rw-r--r-- 1 root root 49 May 26 05:22 listen.conf [root@centos7-21 rsyslog.d]# more listen.conf

测试graylog

任何安装nc的Linux机器上执行： echo date | nc 192.168.252.129 33336 浏览器的http://IP_adress:9000/登录后首页 ，点击第三行绿色搜索按钮，看到一条新消息： Timestamp Source Message 2016-04-10 08:49:15.280 59.70.156.6 2016年 04月 10日 星期日 16:48:28 CST

curl -XPOST http://IP_adress/gelf -p0 -d '{"short_message":"Hello there", "host":"example.org", "facility":"test", "_foo":"bar"}'

多行处理

http://docs.graylog.org/en/2.1/pages/collector.html

使用 graylog-collector

inputs { graylog-server-logs { type = "file" path = "/var/log/graylog-server/server.log" content-splitter = "PATTERN" content-splitter-pattern = "^\d{4}-\d{2}-\d{2}T" // Make sure to escape the \ character! } }

使用 logstash-gelf

http://grokbase.com/t/gg/graylog2/13bf6rgq4f/graylog2-rsyslog-and-apache-tomcat

添加gelf支持

logstash-plugin install logstash-input-gelf

logstash启动（实用配置文件）

./bin/logstash -f ./conf/access-log.conf

test logstash

To test your Logstash installation, run the most basic Logstash pipeline. For example:

cd logstash-5.5.2 bin/logstash -e 'input { stdin { } } output { stdout {} }'

收集stacktrace log-------------------------------------------------------------

input { file { type => "nginx_access" path => ["/var/log/nginx/access.log"]

            codec => multiline {
                    pattern => "^\s"
                    what => "previous"
            }
    }
     file {
            type => "tomcat7_catalina"
            path => ["/opt/tomcat-7.0.70/logs/catalina.out"]
             codec => multiline {
                    pattern => "^\s"
                    what => "previous"
            }
    }

}

output { gelf { host => "192.168.192.168" port => 33336 } } ~

测试

echo date | nc localhost 33333

配置为： tcp_plaintext_33333 Raw/Plaintext TCP