log colector metric - ghdrako/doc_snipets GitHub Wiki

• Vector - - A high-performance observability data pipeline.
  • https://vector.dev/
  • https://github.com/vectordotdev/vector

[sources.kafka_logs]
type = "kafka"
bootstrap_servers = "kafka:9092"
topics = ["api_logs"]

[transforms.parse_json]
type = "remap"
inputs = ["kafka_logs"]
source = '''
. = parse_json!(.message)
.timestamp = to_timestamp!(.timestamp)
'''

[sinks.clickhouse]
type = "clickhouse"
inputs = ["parse_json"]
endpoint = "http://clickhouse:8123"
database = "logs"
table = "requests"
compression = "lz4"
batch.max_events = 1000
batch.timeout_secs = 1

To oznacza:

• czyta logi z Kafki,

• parsuje JSON,

• łączy po 1000 rekordów lub co 1 sekundę,

• wysyła do ClickHouse.

• Fluent Bit / Fluentd – (CNCF project, też open source)

• Logstash – część Elastic Stack (open source, cięższy)

• Telegraf – dla metryk (InfluxData) *OpenTelemetry Collector – neutralny, CNCF