EC2 and EBS - vedratna/aws-learning GitHub Wiki

  • AMI - It is used to store metadata about os, installed softwares and architecture of the machine as well as reference to the snapshots of associated EBS volumes. AMIs are regional based, however you can copy AMIs from one region to another. While copying it from one region to another it also copies all the snapshots of associated EBS volumes and storage charges would be applicable accordingly.
  • Placement Group (cluster, partition and spread)
  • By default limits for no. of EC2 instances per region is 20.
  • Bastion Host is used to SSH/RDP ec2 instances created in private subnet. Bastion Host itself is created in public subnet.
  • for gp2 every 1GB of EBS volume, 3 IOPS are provisioned. So if you want to increase IOPS performance in gp2, increase your volume. Volume increase upto 3333 Gb for gp2 can increase IOPS performance as it reaches to the limit of 10000 IOPS and after that increase in volume won't increase the IOPS performance. To increase it further migrate to io1. for io1 50 IOPS are provisioned per 1GB upto 32000 IOPS.
  • AWS Fargate is a serverless compute engine for containers that works with both Amazon Elastic Container Service (ECS) and Amazon Elastic Kubernetes Service (EKS). Fargate removes the need to provision and manage servers, lets you specify and pay for resources per application, and improves security through application isolation by design. Fargate allocates the right amount of compute, eliminating the need to choose instances and scale cluster capacity. Fargate runs each task or pod in its own kernel providing the tasks and pods their own isolated compute environment.
  • EC2Rescue can help you diagnose and troubleshoot problems on Amazon EC2 Linux and Windows Server instances. You can run the tool manually or you can run the tool automatically by using Systems Manager Automation and the AWSSupport-ExecuteEC2Rescue document. The AWSSupport-ExecuteEC2Rescue document is designed to perform a combination of Systems Manager actions, AWS CloudFormation actions, and Lambda functions that automate the steps normally required to use EC2Rescue. Systems Manager Automation simplifies common maintenance and deployment tasks of Amazon EC2 instances and other AWS resources. Automation enables you to do the following:
  • Build Automation workflows to configure and manage instances and AWS resources.
  • Create custom workflows or use pre-defined workflows maintained by AWS.
  • Receive notifications about Automation tasks and workflows by using Amazon CloudWatch Events.
  • Monitor Automation progress and execution details by using the Amazon EC2 or the AWS Systems Manager console.
  • Static IP can be provided by Network Load Balancer one per subnet. You can place your Application Load Balancer (layer 7) behind Network Load Balancer (Layer 3) to get advantage of both

EC2 Instance Types

  • General Purpose: T3(burstable general purpose), T3a(AMD), M5(latest general purpose), M5a(AMD), M5n(improved network throughput), M6g(powered by Arm-based AWS Graviton2 processors), A1 (Arm based workloads)
  • Compute Optimised: C6g(powered by Arm-based AWS Graviton2 processors), C5, C5a(AMD), C5n(improved network throughput)
  • Memory Optimised: R6g(powered by Arm-based AWS Graviton2 processors), R5, R5a(AMD), R5n(improved network throughput), X1, X1e, HighMemory(Large InMemory Database), z1d
  • Accelerated Optimised: P3 (General purpose GPU), G4 (Graphic Intensive workload, Machine learning inferences), Inf1 (Build from ground up for machine learning inferences), F1(customizable hardware acceleration with field programmable gate arrays (FPGAs).)
  • Storage Optimised: I3 (very hight random I/O performance), H1(High Disk Throughput), D2(High Disk Throughput)
  • For an additional, low, hourly fee, customers can launch selected Amazon EC2 instances types as EBS-optimized instances. For M6g, M5, M4, C6g, C5, C4, R6g, P3, P2, G3, and D2 instances, this feature is enabled by default at no additional cost. EBS-optimized instances enable EC2 instances to fully use the IOPS provisioned on an EBS volume. EBS optimised instances will have dedicated network interface for EBS IO operation that won't be shared with other network operations.

EBS

  • EBS are durable but stored in single AZ only. Can take snapshots and store it into different AZ or region.
  • EBS Volume type and size can be changed dynamically, that way it is elastic.
  • General Purpose SSD(gp2): Max IOPS: 16000, Max Throughput 250 MiB/S, 1GB-16TB Volume size
  • Provisioned IO SSD(io1 & io2): Max IOPS: 64000, Max Throughput 1000 Mib/S, 4GB-16TB Volume size
  • Throughput HDD(sc1): Max IOPS: 500, Max Throughput 500 MiB/S, Low storage cost, 500GB-16TB Volume size
  • Cold HDD(st1): Max IOPS: 250, Max Throughput 250 Mib/S, Lowest storage cost, 500GB-16TB Volume size
  • The cluster relies on shared storage across all the compute nodes. A Throughput Optimized EBS volume cannot be shared between EC2 instances. The Amazon EBS Multi-Attach feature is only applicable for Provisioned IOPS SSD volumes.
  • RAID - Redundant Array of Independent Disk
  • RAID0 - does stripping no mirroring hence no redundant data, use for high IOPS
  • RAID1 - does mirroring and maintain redundant copy of data, not useful for high IOPS, used for redundancy/durability
  • RAID10 - does both stripping and mirroring, can be used for both high IOPS and redundancy
  • Oracle RAC is not supported by RDS. That is why you need to deploy the database in an EC2 instance and then either create a shell script to automate the backup or use the Data Lifecycle Manager to automate the process. An Oracle Real Application Clusters (RAC) One Node option provides virtualized servers on a single machine. This provides an 'always on' availability for single-instance databases for a fraction of a cost.
  • Amazon Data Lifecycle Manager (DLM) for EBS Snapshots provides a simple, automated way to back up data stored on Amazon EBS volumes. You can define backup and retention schedules for EBS snapshots by creating lifecycle policies based on tags. With this feature, you no longer have to rely on custom scripts to create and manage your backups.

EFS

  • Amazon Elastic File System (Amazon EFS) provides a simple, scalable, fully managed elastic NFS file system for use with AWS Cloud services and on-premises resources. It is built to scale on-demand to petabytes without disrupting applications, growing and shrinking automatically as you add and remove files, eliminating the need to provision and manage capacity to accommodate growth.
  • Amazon EFS can provide very low and consistent operational latency as well as a throughput scale of 10+GB per second.
  • Amazon EFS file systems are distributed across an unconstrained number of storage servers. This distributed data storage design enables file systems to grow elastically to petabyte scale and enables massively parallel access from Amazon EC2 instances to your data. The Amazon EFS-distributed design avoids the bottlenecks and constraints inherent to traditional file servers.
  • This distributed data storage design means that multithreaded applications and applications that concurrently access data from multiple Amazon EC2 instances can drive substantial levels of aggregate throughput and IOPS. Big data and analytics workloads, media processing workflows, content management, and web serving are examples of these applications. In addition, Amazon EFS data is distributed across multiple Availability Zones, providing a high level of durability and availability.
  • Elastic Fabric Adapter (EFA) is a network interface for Amazon EC2 instances that enables customers to run applications requiring high levels of inter-node communications at scale on AWS. Its custom-built operating system (OS) bypass hardware interface enhances the performance of inter-instance communications, which is critical to scaling these applications. EFA’s unique OS bypass networking mechanism provides a low-latency, low-jitter channel for inter-instance communications. This enables your tightly-coupled HPC or distributed machine learning applications to scale to thousands of cores, making your applications run faster.
  • Amazon FSx for Lustre makes it easy and cost-effective to launch and run the popular, high-performance Lustre file system. You use Lustre for workloads where speed matters, such as machine learning, high-performance computing (HPC), video processing, and financial modeling. Amazon FSx file systems provide up to multiple GB/s of throughput and hundreds of thousands of IOPS. The specific amount of throughput and IOPS that your workload can drive on your file system depends on the throughput capacity and storage capacity configuration of your file system, along with the nature of your workload, including the size of the active working set.The open-source Lustre file system is designed for applications that require fast storage—where you want your storage to keep up with your computing capacity. Lustre was built to solve the problem of quickly and cheaply processing the world's ever-growing datasets. It's a widely used file system designed for the fastest computers in the world. It provides submillisecond latencies, up to hundreds of Gbps of throughput, and up to millions of IOPS.