AWS_Integration - kamialie/knowledge_corner GitHub Wiki

Content

Simple Queue Service

Distributed message queuing system. Sends, stores, and receives messages between software components. Provides unlimited throughput and number of messages in a queue. Default retention is 4 days, maximum is 14. Message size is up to 256KB.

Producer sends message to SQS. Consumer polls messages from SQS (up to 10 messages at a time) and deletes them from the queue. It is possible to have multiple producers and consumers (also supports Lambda).

Encryption at rest can be configured using KMS. SQS access can be configured through IAM policies and SQS access policy (resource policy). It is also possible to set up cross-account access.

CloudWatch queue length metric (ApproximateNumberOfMessages) can be used to trigger scaling in ASG (consumers). Create custom metric - queue length / # of instances.

Large messages (over 256 KB up to 2GB) can be handled by storing them in S3 - Amazon SQS Extended Client Library for Java and AWS SDK for Java can manage messages that are stored in S3 directly (the only method to do it).

Queue types

Standard (default) type guarantees at least once delivery, but order is not preserved. Also duplicate messages may appear.

FIFO queue enforces message ordering. As a result has a limited throughput - 300 msg/s without batching, 3000 msg/s with. Also guarantees exactly once delivery and no duplicates. Queue name must end with .fifo. Group ID can be used to group together similar messages. Only one consumer can be configured per group. Guarantees the order within a group.

De-duplication period is 5 minutes (duplicates that arrive within that period is removed).

  • content-based - SHA-256 of message body
  • explicitly provided message duplication id

Message visibility timeout

Time period, when a message that has been polled by one consumer is not visible for others. If message is not deleted within that time period, it becomes visible again, "is placed again in the queue". Default is 30 seconds, maximum 12 hours. ChangeMessageVisibility API call can be used to extend visibility timeout, for example, if consumer knows, it needs more time for processing.

Short/Long polling

Long polling allows optionally waiting for messages when there are none in the queue at the time of request - up to 20 seconds. Can be set up at queue level or particular request with WaitTimeSeconds parameter.

Short polling returns a result immediately, which could lead to many empty responses, if there is nothing in the queue (extra charge since payment is based on responses).

Dead letter queue

A threshold (MaximumReceives) can be set on how many times a message can go back to the queue. Once it's hit, a message is placed in DLQ, basically a different SQS queue. DLQ must follow original queue type - standard or FIFO.

Make sure to analyze failed messages, as they will expire as normal messages do - set a big retention setting.

Redrive to Source feature allows to send message from DLQ back to original queue, after, for example, some fixes are performed on original consumers.

Delay queue

Delay messages, up to 15 minutes, so that consumers don't see them immediately. Default is 0 seconds. Can be set at queue or message (DelaySeconds parameter) level.

For messages that are already in the queue:

  • standard queue has no effect
  • FIFO queue does affect the delay

Examples

Decoupled frontend-backend applications, for example, image processing - queue between frontend that handles client requests and backend that processes requests.

Request-response system - producer places a requests with message ID and response queue name into first queue, while consumer places a response with the same message ID to the queue that producer had specified. This pattern is implemented by SQS Temporary Queue client, which leverages virtual queues instead of creating/deleting them.

Simple Notification Service

Pub/sub service.

Producer sends messages to one topic, which can have multiple receivers. Up to 10 million subscriptions per topic, up to 100000 topics per account. Supports cross-region delivery, e.g. SNS + SQS pattern.

Subscribers:

  • SQS
  • HTTP/S (with delivery retries)
  • Lambda
  • Email
  • SMS
  • Mobile notification
  • Kinesis Firehose

Publish options:

  • topic - create topic, add subscription(s), publish to topic (SDK)
  • direct - create platform app, create a platform endpoint, publish to platform endpoint (mobile apps SDK, works with Google GCM, Apple APNS, Amazon ADM, etc)

Encryption at rest can be configured using KMS. SNS access can be configured through IAM policies and SNS access policy (resource policy). It is also possible to set up cross-account access.

FIFO type enforces ordering by Message Group ID and deduplication by Deduplication ID or Content Based Deduplication. Only FIFO SQS queue can be set as a subscriber. Name has to end with .fifo.

Message filtering

Described in JSON policy that is attached to subscriber. If subscriber doesn't have a filter policy, it receives all messages.

Examples

Fan out pattern - send one message and deliver to multiple targets. For example, S3 Event for a prefix can only have one rule - use SNS to deliver notification to multiple receivers.

Simple Email Service

Scalable and highly available Email service. Mostly used to send marketing, notification, or transactional emails to consumers. Uses pay-as-you-go model. Can both send and receive emails, latter is stored in S3. Incoming emails can trigger Lambda or SNS.

Kinesis

Collects, processes and analyzes streaming data in real-time. App logs, metrics, IoT telemetry data, etc.

  • KPL - Kinesis Producer Library
  • KCL - Kinesis Consumer Library

Data Stream

Capture, process, and store data streams.

One stream is made of multiple shards. Each shard has a fixed read throughput of 5 msg/s (max 2MB/s) and write throughput of 1000 msg/s (max 1MB/s). Shards can be added or removed later on; more shards increase stream's total throughput. Capacity modes:

  • provisioned - choose number of shards, scale manually; pay per hour per shard
  • on-demand - default capacity of 4MB/s or 4000 msg/s, automatically scales based on observed throughput peak in the last 30 days

Producer creates and sends a data record that consists of partition key and data blob (up to 1MB). Sequence number is also added, which is unique per partition key within shard. Kinesis creates a record with the same partition key, adds a unique sequence number (guarantees the order) and sends it to consumers. Only one consumer can be configured per shard.

Partition key is used to determine where record is going to go (which shard). Same partition key guarantees same shard.

Consumption options:

  • shared - 2MB/s per shard for all consumers (pull mode)
  • enhanced - 2MB/s per shard per consumer (push mode)

Billing is based on number of shards. Retention can be set from 1 (default) to 365 days and allows data reprocessing (replay). Data in Kinesis is immutable.

Producers

Producers:

  • SDK
  • Kinesis Producer Library (built on top of SDK)
  • Kinesis agent (built on top of KPL)

PutRecord API request sends the data to Kinesis.

Handling ProvisionedThroughputExceeded error:

  • choose highly distributed partition key
  • implement retries with exponential backoff
  • increase number of shards (shard splitting)

Shard can only be split into 2 in a single operations - old shard is closed and will be deleted once all data has expired. Merging shards is done in a similar fashion.

Consumers

  • SDK
  • Kinesis Client Library - Java library to read from Data Stream; each shard can be read by one KCL instance (1:1 relationship); checkpointing progress is managed via DynamoDB - in case a reader restarts or if shared scale in/down, so that existing readers can split the work or also scale
  • Lambda
  • Kinesis Data Firehose
  • Kinesis Data Analytics

Enhanced fan-out feature provides each stream consumer it's own read throughput (default behavior shares shard throughput evenly across all consumers). Uses SubscribeToShard API, which implements the push model. Classic option uses GetRecords API, which follows pull model.

Data Firehose

Load data streams into AWS data stores. Fully managed, automatically scaled, serverless. Also called Delivery Stream.

Receives data (up to 1MB), applies optional transformation via Lambda formatting and writes data to destination in batches (near real-time - 60 sec minimum latency for non full batches or minimum 32MB of data). Backup bucket can be set up to send all or failed data. Billing is based on data going through.

Producers:

  • SDK, KPL
  • Kinesis agent
  • Kinesis Data Stream
  • CloudWatch
  • AWS IoT

Destinations:

  • AWS
    • S3
    • Redshift (copy through S3)
    • OpenSearch
  • 3rd party (and more)
    • Datadog
    • Splunk
    • MongoDB
  • custom HTTP endpoint

Data Analytics

Real-time data analytics with SQL or Apache Flink. Fully managed, automatically scaled, serverless. Can create streams out of queries. Generally used for time-series analytics, real-time dashboards and metrics.

Data sources:

  • Kinesis Data Stream
  • Kinesis Data Firehose or Managed Apache Kafka for Apache Flink flavor

Destinations:

  • Kinesis Data Stream
  • Kinesis Data Firehose

Video Stream

Optimized service for capturing, processing, and storing video streams.

EventBridge

Receives events from data sources and sends them to targets with optional routing rules and filters or creates events on schedule (cron). Can intercept any API call with CloutTrail integration. Events are in the from of JSON payload.

Event buses can be accessed from different accounts. Types:

  • default - created by AWS (links with legacy CloudWatch Events)
  • partner - receive event from 3rd party services (Zendesk, DataDog, etc)
  • custom - created by client

Events sent to EventBridge event bus can be archived and replayed.

Schema registry allows events to be structured in a certain way to make it predictable for receiving applications. Can be versioned.

Amazon MQ

Managed Apache ActiveMQ. Runs on a dedicated machine. Has both SQS and SNS features.

HA with failover is available. Set up EFS as backend storage and deploy active and standby MQ brokers in different AZs. In failure event attach EFS to standby.

Serverless Application Model

Framework for developing and deploying serverless applications.

All configurations and resources are set in YAML file - Lambda, DynamoDB, etc. Can also run the setup locally!