Log Aggregation - wuyichen24/spring-microservices-in-action GitHub Wiki

Overview

Log aggregation basics
Send log to Splunk (Local)
Send log to Papertrail (Cloud)
References

Log aggregation basics

There are 2 main log aggregation solutions:

Local: Send log to the local log server.
Cloud: Send log to the cloud log server.

Send log to Splunk (Local)

Step 1: Set up a data input point

Step 1.1: Login the Splunk web portal

Step 1.2: Click "Add Data" under "Explore Splunk Enterprise" dashboard

Step 1.3: Choose "Monitor" under "Or get data in with the following methods" section

Step 1.4: Select Source

On the left side, select "TCP / UDP".
For choosing TCP or UDP, if you want to send the log by syslog traffic, use "UDP". You can also send the log by TCP traffic for better transmission reliability.
Set the port number.

Step 1.5: Input Settings

Source type
- Choose "Select".
- Select Source Type: Choosing "Application" -> "catalina".
App context
- By default, you can choose "Search and Reporting (search)".
- If you have created your own app, you can select the app you created.
Host
- Splunk can track the log sender by DNS (hostname) or IP (address).
Index
- Choose "Default".

Step 1.6: Review

Step 1.7: Done

Step 2: Let Spring application send the log to Splunk

Option 1: Use UDP traffic

If you set the type of the data input point as "UDP" in Splunk, you need to change the logback-spring.xml file under the resources directory like this:

logback-spring.xml

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <appender name="SYSLOG" class="ch.qos.logback.classic.net.SyslogAppender">
        <syslogHost>127.0.0.1</syslogHost>                                      <!-- Splunk server is running in local  -->
        <facility>SYSLOG</facility>
        <port>50000</port>                                                      <!-- Use the port number when you set up the UDP data input point  -->
        <suffixPattern> [%thread] %logger %msg</suffixPattern>
    </appender>
    <root level="INFO">
        <appender-ref ref="SYSLOG" />
    </root>
</configuration>

Option 2: Use TCP traffic

(Need to add)

Step 3: Verification

After you start the Spring application with the updated logback-spring.xml, you should see the log coming into the Splunk (with the app you chose).