Logging - alphagov/notifications-manuals GitHub Wiki

We have three logging systems:

• Logit / Kibana - logs for ECS apps and cloudfront. Limited to 14 days.
• AWS Cloudwatch - logs for ECS apps and things in AWS, such as EC2 instances.
• AWS Athena - logs for CloudFront and AWS load balancers

This is our preferred way to search logs for ECS apps, since it's easier to search across all logs or a subset of them. You may need to switch to CloudWatch if you need logs over a longer period (more than 14 days), or if you want to do additional stats processing.

Logs are outputted from the apps to STDOUT/STDERR and captured via ecs and stored inside cloudwatch.

Logs from ECS apps get to Logit via a lambda deployed from notifications-aws. This lambda is deployed from the terraform and can be found here.

Cloudfront logs are shipped to Logit via a Logstash input on the logit side. They pull from the s3 buckets where the logs are stored. Due to the S3 batching up of logs you will always have some delay in the logs appearing in Kibana (5-15 mins).

We provide aws credentials to logit in order to pull from the s3 bucket. See the terraform.

Configuration for this on the logit side was set up by logit support.

We have some field manipulation that can be found inside the logit configuration. The configuration can be viewed by logging into logit and selecting "settings" on the environment you are interested in then select "Logstash Pipelines".

If there are any issues with this process you will need to contact logit support.

While Kibana logs should be reliable, CloudWatch is still useful for long term storage (1 year vs. 14 days) and may be easier to do some advanced processing on data within log messages.

Athena is a query interface on top of structured log files stored in S3 buckets. S3 logs available are:

• *-paas-cloudfront-proxy-logs - request logs for the Admin, API or Document Download apps
• <env>-document-download-logs - S3 logs for Document Download (no table in Athena)
• <env>-notify-alb-access-logs - S3 logs for the Notify ALBs (admin, api, document-download-api, document-download-frontend, tech-docs)