Logging - jbrucker/home-log GitHub Wiki

Some recommended practices for logging of containerized applications are:

• Log to standard output (stdout) and let the container or orchestrator (Kubernetes) manage logs.
• In a container, keep the logging configuration simple. One logger.
• Uvicorn logs to stdout by default. If using Uvicorn, align the application's logs with Uvicorn's logs.
  • There is a command line option to request logging (--access-logs)?
• Use structured logging, i.e. JSON
• Let Docker, Kubernetes, or Systemd handle log collection.
• Use a log collector and aggregator.

docker logs <container>

• Log aggregation system from Grafana Labs.
• Stores logs efficiently by indexing only labels (not full log text).
• Offers fast search and low storage cost.
• Integrates well with Prometheus and Grafana.
• Typically paired with Promtail, but also accepts logs from Fluent Bit and others.
• Role: log storage + query engine (ElasticSearch alternative optimized for logs).

• Lightweight log shipper designed specifically for Loki.
• Reads logs from:
  • Docker containers (stdout/stderr)
  • Files (e.g., nginx access/error logs)
  • Journald
• Pushes them to Loki.
• Simple, reliable, minimal resource usage.

• CNCF log processor and forwarder. "General purpose log pipeline engine."
• Very flexible with hundreds of plugins.
• Can route logs to many destinations:
  • Loki
  • Elasticsearch
  • Kafka
  • S3
  • Cloud logging services
• Often used in Kubernetes or multi-pipeline data environments.
• Better suited for central/on-node log transformation and routing than Promtail.

• Part of the Elastic (ELK) stack.
• Feature-rich ingestion and transformation pipeline.
• Designed for complex parsing, enrichment, and routing.
• Most commonly outputs logs to Elasticsearch.
• Pros:
  • Enormous processing capability.
• Cons:
  • Heavy compared to Promtail or Fluent Bit.
  • Usually unnecessary unless using Elasticsearch or complex log rules.

• Lightweight log forwarder from the Fluentd ecosystem.
• Written in C and designed for performance (Kubernetes, edge nodes).
• Supports routing logs to:
  • Loki
  • Elasticsearch
  • Kafka
  • Fluentd
  • Cloud targets
• Some filtering and parsing capabilities; more than Promtail but less than Fluentd.

• Lightweight shipper from the Elastic Beats family.
• Reads local logs and sends them to:
  • Elasticsearch (most common)
  • Logstash
  • Kafka
• Provides metadata for Docker and Kubernetes logs.
• Less flexible for heavy processing, usually paired with Logstash or Elasticsearch.
• Filebeat is to Elasticsearch what Promtail is to Loki.

A distributed, scalable search and analytics engine commonly used for:

• Log storage and querying
• Full-text search
• Analytics dashboards
• Application search features
• Original code is open-source and free to self-host. Current version is commercial & requires paid license.
• Often used with Logstash and Kibana for visualization (the "ELK Stack").

• Single agent that collects:
  • Logs
  • Metrics
  • APM traces
• Sends to Datadog’s SaaS backend.
• Very easy to install with Docker autodiscovery.
• Proprietary and paid beyond trial/free limits.

• Promtail best for Loki users.
• Fluent Bit best universal lightweight shipper supporting many outputs.
• Filebeat best for Elastic Stack users.
• Datadog Agent best when using the Datadog SaaS platform.

All can sit on a Docker host and ship container logs to a central destination.

• Fluentd - more lightweight and widely used in Kubernetes
• Logstash - best in Elastic-based enterprise pipelines with heavy transformations.

• Loki
• Elasticsearch - commercial with "source available" version that can be self-hosted for free
• OpenSearch - open-source version of Elasticsearch
• Splunk (commercial)
• Datadog logs (commercial)
• AWS CloudWatch (commercial)

The requirements for this application are

1. Collect logs from containerized apps. Containers may be running on different servers.
   • FastAPI web service (stdout logs)
   • Nginx proxy for FastAPI app. Has access/error logs.
   • Postgres database logs (if available)
2. Centralize the logs
3. Minimal operational overhead
4. (Desirable) Integrate with existing monitoring that is using Prometheus

Best Overall Choice: Loki + Promtail

Reasons:

• Low storage cost.
• Lightweight.
• Integrates well with Prometheus and Grafana.
• Install and configuration are simple in Docker or Docker Compose.
• Perfect for remote, containerized environments.
• Grafana can be used as interface to both metrics (Prometheus) and logs (Loki)

This is also the most common modern open-source stack for observability in Docker/Kubernetes setups.

A good alternative is Fluent Bit with Loki

• more flexible and universal shipper
• more routing/filtering ability than Promtail
• good if logs may later go to multiple destinations (S3, Kafka, etc.)