ELK Stack Research: Beats - lydia-wu/cadence GitHub Wiki

Alloway, Hunter; Last Edited: 2021-09-06

Meeting Notes for the Week of: September 6

ELK Stack: Beats ELK is popular because it helps analyze data from a plethora of distributed, noisy, hard to find data “Beats are lightweight agents that are installed on edge hosts to collect different types of data for forwarding into the stack.” Organizations need to comply with standards and mitigate performance issues Engineers require multiple sources of data coming from applications and infrastructure This system enables quick identification of problems and normalicies within received data

Beats Beats are defined as the medium to which log data and analytics is moved around multiple servers to streamline metrics/feedback from a software or application. Field-formatting, data collection service pulling data from cloud-based applications( for example a google cloud storage server). It is the process that collects the data for: aggregation, indexing and storage and finally analysis and visualization Subfields include: Filebeat, Metricbeat, Packetbeat, Auditbeat, Heartbeat and Winlogbeat

Filebeat: “Collects and ships log files.” It intakes and sends log files from a server and slows down read/write speeds should the intake of data exceed the rate that it can send data to different portions of a server.

Packetbeat: Think of it like a large-scale version of Task Manager, logging traffic and other data being sent to and from a server. It decodes and records the amount of data being transferred per transaction. (I believe transaction here is referring to an instance of a file-transfer).

Metricbeat: The medium used to transfer data between servers and manages platform analytics

Winlogs: Collection of “Windows Event Logs” from Windows-based applications.

Data Pipeline is composed of three stages: Aggregation, storage and processing Data Shipper: The item that ships information “the beat", sends information in the form of a log file to Elasticsearch or Logstash to perform whatever operation is requested. It is a medium that sends data away for processing and development. The data shipper works with beats in order to communicate with application to software engineer.

ASP.NET VIA Microsoft Store Offers a web-based application management platform with processing requests in C# and F# Browser sends a web-form to a web-server and receives a response of a fully functional, markup page/HTML of a website For the purposes of our endeavor, it does not appear to be a useful application unless used for publishing of findings to a webpage or blog