Tues November 7th 2023 Notes - Hsanokklis/2023-2024-Tech-journal GitHub Wiki

ELK Stack

• A collection of 3 open source products, maintained by elastic co
  • Elasticsearch
  • Logstash
  • Kibana
  • Beats
• Called Elastic Stack because it has beats now not just the ELK
• Allows us to work with unstructured or semi-structure data
  • logs (not normally structured)
• Structure data
  • spreadsheets
  • have formatting

We will learn how to work with unstructured data!

Elasticsearch

• allows you to search the data

Logstash

• Feeds it into search index so that it can be searched

Kibana

• gives us a graphical overlay so you can interact with data in a more user friendly way

Beats

• agents that can be installed on systems around the network
• collect data and forward it into the stack
• build logs and collect data in a more lightweight and efficient way

Why is ELK Popular

Biggest competitor is SPLUNK

• Its open source
• ELK is downloaded more then SPLUNK because splunk is very expensive(and not open-source)

Why Log management

• Systems are complex - need to aggregate logs (put them all in one place)

Why Centralized Log Management

• Aggregation
• Processing
• Storage
• Analysis

Visualization

Beats ---> Logstash ---> elasticsearch < ----- Kibana

Usage Examples

Financial Services

E-Commerce

Healthcare Analytics

IoT data Analytics

Media and Entertainment

NoSQL

• No structured Query
• a way to refer to databases that work with unstructured data

Problem with the model is storage Except that its not because disk storage have dropped The use of NoSQL databases and Elastic and Splunk have gone up