Data Ingestion In Splunk - PushpaPersonnel/Splunk-Administrator GitHub Wiki

Data Integrations/Ingestion:- 2 types agent - UF, HF and software program(API Script) agent less - HEC(8088), Monitoring TCP UDP ports-514(Syslog), Splunk App for stream(addon's like dB connect(8863), snow, aws addon)

Hec - token based mechanism allows apps to send data via http or https ports any tool or script can send http post request with log data and token to https://:8088/services/collector gui-->settings-->data inputs-->HEC-->new token-->provide index-->submit-->copy auto generated token

  1. enable HEC
  2. Verify HEC is listening in server.conf , HEC must be enabled [httpserver] disabled=0 port = 8088

Testurl - curl -k https://:8088/service/collector \ -H "authorization: Splunk <HEC_token>"\ -d '{"event":"test event from curl"}'

Syslog:- universal standard for message logging across Linux systems, load balancers, network devices like firewall, routers, switches, etc.. in real time these devices generate logs like login attempts, network drop, configuration changes, and security events. syslog source-->syslog receiver(HF/UF-rsyslog/syslog-ng)514-tcp/udp-->Splunk Indexer

  1. setup syslog receiver -choose HF/UF and create syslog listening port by configure 514 in rsyslog/syslog-ng -rsyslog method udp-514 module(load ="imudp, input(type=imuddp, port=514) -tcp-514 module(load ="imtcp, input(type=imtcp, port=514) -save logs - $template remotelogs,"/var/log/syslog/%hostname%/%programname%.log .?RemoteLogs -restart rsyslog - sudo systemctl restart rsyslog
  2. configure splunk to monitor syslog directory /opt/splunk/etc/system/local/inputs.conf [monitor:///var/log/syslog/] index = syslog, sourcetype = syslog, recursive = true restart the splunk
  3. point devices to send logs configure network device to send syslog to the receiver(pointing to hf on port 514 tcp/udp) logger -p local0.info "Test syslog message" login to receiver ip, logging trap informational
  4. validate in splunk index= syslog sourcetype=syslog |stats count by host,source

DB Connect:- it is a app used to connect to db's like oracle, MySQL,mssql,postgresql, etc.. with to ingest DB data into splunk, run sql queries from splunk and write back data into database used to analyze patterns and anomalies in splunk source Database-->HF(addon placed)-->Splunk

  1. Install DB connect app(HF UI-->Manage apps-->install app from file/browser more apps) -install:Splunk DB Connect -Appname :splunk_app_db_connect
  2. Install java JDK java 8 or 11 version installed on splunk server and then set the javahome environment variable: export javahome=/path/to/java
  3. Install JDBC driver for your database ex:- MySQL - MySQL jdbc driver(mysql_connector-java.jar) place it under opt/splunk/etc/apps/splunk_app_db_connect/drivers/
  4. configure db connect in splunk UI apps-->DB Connect create new identity Name, username, password create connection: connection name,database type,host,port-3306,database,indentity,driver, test connection
  5. create inputs to ingest data into splunk types of inputs: -Rising column(best for incremental data) -Batch (fetches full result every time) -Tail(like tailing a log table) ex:-raising column table:transactions raising coloumn:transaction_id interval:every 5 minutes target Index:sales_index sql for input - SELECT transaction_id, customer_id,amount,status,created_at FROM transactions where $rising_coloumn$&gt;? order by $rising_colomn$ ASC

API Script:- We have to use authorized API credentials like AWS or Azure AD generated, based on the requirement, via api script these credentials everytime it will authenticate in between and receives the data to indexer

AWS Add On:- used integrate the data from AWS such as cloudwatch logs, metrics, cloudtrail events,etc

  1. install addon from manage app in HF/UF or searchhead based on architecture
  2. create IAM role or user and provide access to the required services for these users and get the credentials
  3. configure inputs in splunk
  • Go to Settings > Data Inputs > AWS Add-on
  • Choose the service (e.g., CloudWatch Logs) and set parameters like region, log group, polling interval.
  1. Validate index=aws sourcetype="aws:cloudwatchlogs"

Snow Add On:-

  1. Install spplunk addon for service now from manage app in HF/UF or searchhead based on architecture
  2. configure service now account in
  1. Setup Data inputs
  • Navigate to Inputs > Create New Input.
    • Choose the SNOW table (e.g., incident, cmdb_ci, change_request).
    • Set polling interval and filters (e.g., only open incidents).

SNMP:- useful for monitoring network devices, infrastructure health, and alerting on anomalies. Method 1: Using snmptrapd + File Monitoring This is a common and resilient approach:

  1. Install SNMP Tools On your Splunk Heavy Forwarder or a dedicated Linux box: sudo yum install net-snmp net-snmp-utils
  2. Configure snmptrapd Edit /etc/snmp/snmptrapd.conf: authCommunity log public Start the service: sudo systemctl start snmptrapd
  3. Log SNMP Traps to a File Configure snmptrapd to write traps to a log file: OPTIONS="-Lf /var/log/snmptrapd.log -p /var/run/snmptrapd.pid"
  4. Monitor the Log File in Splunk On your Heavy Forwarder: /opt/splunk/etc/syatem/local/inputs.conf [monitor:///var/log/snmptrapd.log] index = network sourcetype = snmp:trap Method 2: Splunk SNMP Modular Input App Use the SNMP Modular Input for polling SNMP data directly. Features:
  • Supports SNMP v1, v2c, and v3
  • Can poll OIDs or listen for traps
  • Custom MIB support Setup:
  • Install the app on your Heavy Forwarder.
  • Go to Settings > Data Inputs > SNMP.
  • Choose Trap Listener or Polling Mode.
  • Configure community string, port (usually 162), and sourcetype.
⚠️ **GitHub.com Fallback** ⚠️