Logging Research and Recommendations - JU-DEV-Bootcamps/ERAS GitHub Wiki

User Story #24

1. Basics of Logging

  1. What is Logging?
    Logging is the act of recording application events (like errors, warnings, info messages) for debugging, analysis, and auditing

  2. Why Logging Matters

    • Debugging & Troubleshooting: Quickly locate the root cause of production issues
    • Monitoring & Observability: Measure and track performance, user behavior, etc
    • Auditing & Compliance: Keep records of actions to meet regulatory requirements
    • Security: Spot suspicious or malicious activity

  3. Log Levels

    • Trace: Most detailed logs, helpful for diagnosing intricate problems
    • Debug: General diagnostic logs for development
    • Information: Highlight normal application behavior (e.g., successful operations)
    • Warning: Something unexpected happened but the app can continue
    • Error: Issues that cause functionality to fail in a recoverable way
    • Critical (Fatal): Unrecoverable errors that require immediate attention

2. Common .NET Logging Approaches

2.1 Built-In Microsoft.Extensions.Logging

  • Pros
    • Included natively with ASP.NET Core.
    • Minimal setup required
    • Easy to integrate with existing ILogger<T> usage
  • Cons
    • Advanced scenarios (structured logging, multiple sinks) require extra configuration
    • Not as feature-rich for complex or high-volume needs as some dedicated frameworks

2.2 Serilog

  • Pros
    • Great support for structured logging (JSON output)
    • Plenty of “sinks” (destinations) such as file, console, AWS CloudWatch, etc
    • Active community and ecosystem of extensions
  • Cons
    • Some overhead in configuration (must install and configure NuGet packages)
    • Slight learning curve if you haven’t used structured logging before

2.3 NLog

  • Pros
    • Mature, widely adopted in the .NET ecosystem
    • Flexible configuration via XML or JSON
    • Supports multiple targets (file, database, console, etc.)
  • Cons
    • Configuration can get complex for advanced setups
    • Less “plug-and-play” structured logging experience compared to Serilog

2.4 Log4Net

  • Pros
    • Very mature (one of the oldest .NET logging frameworks)
    • Stable and widely used historically
  • Cons
    • Generally considered more “legacy” compared to newer frameworks like Serilog or NLog
    • Configuration is often verbose

2.5 Cloud Logging Solutions

  • Examples: AWS CloudWatch, Azure Application Insights, Datadog, Splunk
  • Pros
    • Centralized storage and analysis of logs
    • Built-in dashboards, alerting, metrics
    • Scalability for large volumes of logs
  • Cons
    • Additional cost
    • Vendor lock-in or more complex setups
    • Requires stable network connectivity (latency considerations)

3. Where to Store Logs

  1. Local Files

    • Pros: Simple, fast to write, minimal setup
    • Cons: Hard to manage across multiple containers/instances, must handle rotation

  2. AWS CloudWatch

    • Pros: Centralized logging for all containers on AWS, integrated with AWS ecosystem.
    • Cons: Additional cost, requires some configuration (IAM roles, etc.)

  3. Other Aggregation Services (ELK, Splunk, Datadog)

    • Pros: Powerful searching, dashboards, alerting capabilities
    • Cons: Might require more infrastructure setup and associated costs

Note: We can also log to both local files for quick reference and a cloud or aggregation service for long-term storage and analytics

4. Good Practices

  1. Consistent Log Levels

    • Don’t spam Trace or Debug in Productio
    • Use Information for normal events, Warning for unusual but non-breaking situations, Error for real failures

  2. Structured Logging

    • Prefer JSON format to enable easy searching and analysis later
    • Example: Log.Information("User {UserId} logged in", userId);

  3. Correlation / Request IDs

    • Generate a unique ID per request to trace it across multiple services
    • Facilitates debugging in distributed systems

  4. Log Rotation & Retention

    • Set up daily or size-based rotation to avoid filling disks
    • Determine how long logs should be retained (e.g., 7 days, 30 days, 90 days), especially if storing locally

  5. Avoid Sensitive Data

    • Do not log passwords, access tokens, or personal information subject to privacy laws

  6. Monitoring & Alerting

    • Utilize thresholds (e.g., error rate) to trigger alerts
    • Logging is only useful if someone reviews it regularly or sets up automated alarms

5. Proposed Recommendation & Next Steps

  1. Short-List Options

    • Serilog (structured logging, easy to integrate with existing ILogger)
    • or Stick with Built-In (if advanced features are not critical right now)

  2. Storage

    • For development: Log to Console and a local rolling file
    • For production: Potentially use AWS CloudWatch or an EFK/ELK stack for centralized logs

  3. Decision Factors

    • Cost (AWS CloudWatch vs. local file + custom aggregator)
    • Team Familiarity (Serilog is quite popular and not too difficult to learn)
    • Future Scalability (if we expect significant growth, a centralized approach is better)

6. References & Further Reading