batch server - SeoJeongHan/managed_log GitHub Wiki

Batch server

Get Log File

  1. SSH Public Key 생성
  2. SSH Public Key 등록 - get logFile from remote server
  3. batch job 실행 (/home1/irteam/scripts/get_log_file.sh)
rsync -avz --exclude-from '/home1/irteam/logs/config/pattern.txt' 'irteam@remoteServerHostName:/dir/*/file.log' '/local/dir/'$DATE

Filebeat, Logstash

  1. Filebeat 실행
  • config에 등록된 경로에 input file & output Logstash
  1. Logstash 실행
  • filebeat로 부터 받은 row Data input
  • row data filter
  • output DB table insert
Log Data exam - one row
* YYYY-MM-DD 16:41:45 [text] INFO  query - exam_text .. 중략 .. A=A, B=B
Log Data info
- Date, A, B => Parse Log Data
  1. Logstash error log
  • /localDir/logs/logstash-plan.log

crontab batch job

  • DB table에 insert 된 data를 가지고 job 실행
  • java -jar -Dspring.profiles.active=dev | beta | real /xxx/xxx/deploy/xxx/batch-0.0.1-SNAPSHOT.jar --spring.batch.job.names=xxxJob