Monitoring and Reporting in AWS - vedratna/aws-learning GitHub Wiki

There are three main monitoring and reporting services in AWS. AWS Cloudwatch AWS CloudTrail AWS Config

Amazon CloudWatch

It is basically used for performance monitoring. Amazon CloudWatch is a service that monitors the health and status of AWS resources in real time. It provides system wide visibility into resource utilization, application performance and operational health by tracking, measuring, reporting, alerting and reacting to events that occur in an environment. It is accessible through following methods:

  1. Amazon CloudWatch console : https://console.aws.amazon.com/cloudwatch/
  2. The AWS CLI
  3. The Amazon CloudWatch API
  4. AWS SDKs Two types of Amazon Monitoring Metrics Basic Monitoring: Minimum period is 5 mins that is 300 secs Detail Monitoring: Minimum period is 1 min that is 60 secs

  • Basic monitoring metrics for Amazon EC2 instances are available at no additional charges

  • All metrics for Amazon EBS volumes, Elastic Load Balancing load balancers and Amazon RDS DB instances are available at no additional charges.

  • Metrics exist only in the AWS Region in which they are created

  • In addition to monitoring AWS resources, Amazon CloudWatch can be used to monitor data produced from applications, scripts and services. A custom metric is any metric provided to Amazon CloudWatch via an agent or API.

  • One minute data points are available for 15 days

  • Five minutes data points are available for 63 days

  • One hour data points are available for 455 days

  • If metrics need to be available for longer than those periods, they can be archived using the GetMetricsStatistics API call.

  • A namespace is a container for a collection of Amazon CloudWatch metrics. Each namespace is isolated from other namespace.

  • The AWS/ namespace is reserved and cannot be used by customers.

  • Some examples of AWS product namespaces are AWS/AutoScaling, AWS/EC2, AWS/EBS, AWS/ELB and AWS/ApplicationELB.

  • A dimension is a name/value pair that uniquely identifies a metric and further clarifies the metric data stored. A metric can have upto 10 dimensions.

Amazon EBS Volume statuses:

  1. OK means everything is fine
  2. Warning means volume is Degraded or Severely Degraded
  3. Impaired means volume has either Stalled or is Not Available.
  4. Insufficient Data means insufficient data to determine the status.
  • You can use metric filters to search for and match terms, phrases, or values in your log events. When a metric filter finds one of the terms, phrases, or values in your log events, you can increment the value of a CloudWatch metric. For example, you can create a metric filter to search for and count the occurrence of the word ERROR in your log events. Metric filters can also extract numerical values from space-delimited log events, such as the latency of web requests. In these examples, you can increment your metric value by the actual numerical value extracted from the log. Custom metric generated through CloudWatch logs can be used for setting CloudWatch Alarm.
  • You can use subscriptions to get access to a real-time feed of log events from CloudWatch Logs and have it delivered to other services such as an Amazon Kinesis stream, an Amazon Kinesis Data Firehose stream, or AWS Lambda for custom processing, analysis, or loading to other systems. When log events are sent to the receiving service, they are Base64 encoded and compressed with the gzip format. To begin subscribing to log events, create the receiving resource, such as a Kinesis stream, where the events will be delivered. A subscription filter defines the filter pattern to use for filtering which log events get delivered to your AWS resource, as well as information about where to send matching log events to. Each log group can have up to two subscription filters associated with it.
  • You can configure a CloudWatch Logs log group to stream data it receives to your Amazon Elasticsearch Service (Amazon ES) cluster in near real-time through a CloudWatch Logs subscription. Internally it uses lambda to send log data to ES endpoint.
  • CloudWatch Logging Subscription cross account (across multiple region) delivery is supported only for Amazon Kinesis stream. It is very useful for centralizing logging data in s3 bucket or Elastic search. For example CloudWatch logs from multiple accounts can deliver logs to subscribed Amazon Kinesis Data stream in central account that can be pushed to Kinesis Firehose (can have lambda in between for transformation purpose) and Firehose can push it to Elastic Search endpoint or store it to s3 bucket in same central account.
  • Amazon CloudWatch Events delivers a near real-time stream of system events that describe changes in Amazon Web Services (AWS) resources. Using simple rules that you can quickly set up, you can match events and route them to one or more target functions or streams. CloudWatch Events becomes aware of operational changes as they occur. CloudWatch Events responds to these operational changes and takes corrective action as necessary, by sending messages to respond to the environment, activating functions, making changes, and capturing state information. You can also use CloudWatch Events to schedule automated actions that self-trigger at certain times using cron or rate expressions
  • Amazon EventBridge is the preferred way to manage your events. CloudWatch Events and EventBridge are the same underlying service and API, but EventBridge provides more features. Changes you make in either CloudWatch or EventBridge will appear in each console.

AWS CloudTrail

AWS cloudTrail is a service that enables governance, compliance, operational auditing and risk auditing of an AWS account. With AWS CloudTrail, it is possible to log, continuously monitor and retain events related to API calls across an AWS infrastructure.

  • CloudTrail logging which sends CloudTrail events to s3 bucket is not enabled by default. Trail is the configuration that enables delivery of events to the s3 bucket you specify. There are two types of trails
  1. Trails that apply to All Regions: This is recommended by AWS and it has advantage that single s3 bucket in one region can be configured to collect logs from all the regions. In case of third party audit, limited access can be given to single s3 bucket where all trails from all regions are getting stored when applied to all regions is chosen.
  2. Trails that apply to One Regions
  • AWS CloudTrail supports five trails per region
  • By default log files are encrypted using Amazon S3 Server-Side Encryption(SSE).
  • Log files are written in json format
  • Log file name is in format bucket/prefix/AWSLogs/AccountID/CloudTrail/region/YYYY/MM/DD/file_name.json.gz
  • Apart from all regions, it is also possible to write logs in to single s3 bucket on specific account from multiple accounts. For that s3 bucket should have proper bucket policy that permits cross account write access to cloud trail from different accounts. Here cross account assume role won't work as cloud trail needs to read the logs on original account and write it to s3 bucket.
  • In case of Auditing by third party account it makes sense to use assume role. (In case of any confusion visit lecture 278. Cloud Trail and IAM use cases.

AWS Config

AWS Config is a fully managed service that provides AWS resource inventory, configuration history and configuration change notifications to enable security and governance. AWS Config can discover existing AWS resource, export a complete inventory of AWS resources with all configuration details and determine how a resource was configured at any point in time.

  • Customer can create upto 50 AWS Config rules in an AWS account by default. This is a soft limit and it can be increased by contacting AWS support.
  • Rules can be set up as change-triggered rule or as a periodic rule