Avalon on AWS: Step By Step - avalonmediasystem/avalon-aws GitHub Wiki

Deploying Avalon on AWS via CloudFormation

First-Time Preparation

These steps will prepare your account to set up and run a Samvera stack and an Avalon instance. This section only needs to be completed once for each AWS account you want to run Avalon under.

  1. Create a CloudFront Key Pair using your AWS root account. Write down your TrustedSigner ID. Upload your CloudFront private key to an S3 bucket in a cloudfront subfolder.

  2. Create an IAM user and access key id/secret using the Programmatic access option.

  3. Create two IAM roles with the specified names and attached policies:

    • aws-elasticbeanstalk-ec2-role
      • AWSElasticBeanstalkWebTier
      • AWSElasticBeanstalkMulticontainerDocker
      • AWSElasticBeanstalkWorkerTier
    • aws-elasticbeanstalk-service-role
      • AWSElasticBeanstalkEnhancedHealth
      • AWSElasticBeanstalkService

  4. Create an EC2 key pair for your stack.

  5. Sign the AWS Marketplace OpenVPN Access Server License Agreement

  6. Create a public hosted zone for your stack in AWS Route 53 and set up the necessary DNS delegation so it will resolve correctly. For example, if you use repo.somedomain.example.edu, you will need to set up the NS records in the somedomain.example.edu zone so that the repo subdomain records resolve.

  7. Clone the avalon-aws repo. Run rake audit_solution_stacks to ensure that the SolutionStackName specified in the templates points to a currently supported version, e.g. "64bit Amazon Linux 2017.03 v2.6.3 running Tomcat 8 Java 8". Follow the rake task's suggestions to make necessary changes to the template files.

  8. Create a public S3 bucket, with versioning enabled. Upload the latest Avalon zip file. Upload the CloudFormation script and the Solr, Zookeeper and Fedora binaries. Alternatively, you can clone or use the iu-east1-avalon bucket directly.

  9. Create an S3 bucket to store Fedora's binary artifacts.

Using CloudFormation

Deploying the stacks

  1. Install the AWS CLI. Run aws configure to setup your region and access key

  2. Create an infrastructure.json file, this will contain the parameters required for standing up your infrastructure stack:

[
  { "ParameterKey": "StackName",           "ParameterValue": "my-infrastructure" },
  { "ParameterKey": "PublicZoneName",      "ParameterValue": "my-avalon-domain.com" },
  { "ParameterKey": "KeyName",             "ParameterValue": "my-EC2-key-name" },
  { "ParameterKey": "S3Bucket",            "ParameterValue": "iu-east1-deploy" },
  { "ParameterKey": "S3BucketEB",          "ParameterValue": "iu-east1-deploy" },
  { "ParameterKey": "S3FedoraFilename",    "ParameterValue": "fcrepo.zip" },
  { "ParameterKey": "VpnUser",             "ParameterValue": "vpnadmin" },
  { "ParameterKey": "VpnPassword",         "ParameterValue": "vpnpassword" },
  { "ParameterKey": "FcrepoBinaryBucket",  "ParameterValue": "my-fedora-bucket" },
  { "ParameterKey": "FcrepoDatabasePassword", "ParameterValue": "my-fedora-password" },
  { "ParameterKey": "DatabasePassword",    "ParameterValue": "my-db-password" }
]

  1. Install the infrastructure stack aws cloudformation create-stack --stack-name my-infrastructure --template-body https://s3.amazonaws.com/iu-east1-deploy/cloudformation/infrastructure/current/templates/infrastructure.yaml --capabilities CAPABILITY_IAM --parameters file://./infrastructure.json

  2. Create an avalon.json file, this will contain the parameters required for standing up your Avalon stack:

[
  { "ParameterKey": "CloudFrontKeyFile",   "ParameterValue": "my-cloudfront-private-key.pem" },
  { "ParameterKey": "CloudFrontKeyId",     "ParameterValue": "my-cloudfront-key-id" },
  { "ParameterKey": "ConfigurationBucket", "ParameterValue": "bucket-with-cloudfront-key" },
  { "ParameterKey": "EmailComments",       "ParameterValue": "[email protected]" },
  { "ParameterKey": "EmailNotifications",  "ParameterValue": "[email protected]" },
  { "ParameterKey": "EmailSupport",        "ParameterValue": "[email protected]" },
  { "ParameterKey": "InfrastructureStack", "ParameterValue": "my-infrastructre" },
  { "ParameterKey": "KeyName",             "ParameterValue": "my-ec2-keyname" },
  { "ParameterKey": "PriceClass",          "ParameterValue": "PriceClass_100" },
  { "ParameterKey": "QueuePrefix",         "ParameterValue": "avalon" },
  { "ParameterKey": "RDSDatabaseName",     "ParameterValue": "avalon" },
  { "ParameterKey": "RDSUsername",         "ParameterValue": "avalon" },
  { "ParameterKey": "RDSPassword",         "ParameterValue": "rdspassword" },
  { "ParameterKey": "S3Bucket",            "ParameterValue": "iu-east1-deploy" },
  { "ParameterKey": "S3BucketEB",          "ParameterValue": "iu-east1-deploy" },
  { "ParameterKey": "S3Key",               "ParameterValue": "avalon/current/avalon.zip" },
  { "ParameterKey": "SecretKeyBase",       "ParameterValue": "[a Base 64 key]" },
  { "ParameterKey": "TrustedSigner",       "ParameterValue": "[id of trusted signer]" },
  { "ParameterKey": "WebappInstanceType",  "ParameterValue": "t2.small" },
  { "ParameterKey": "WorkerInstanceType",  "ParameterValue": "t2.small" }
]
  1. Install the Avalon stack, only after the infrastructure stack has fully deployed aws cloudformation create-stack --stack-name avalon-stack --template-body https://s3.amazonaws.com/iu-east1-deploy/cloudformation/infrastructure/current/templates/avalon/application.yaml --capabilities CAPABILITY_IAM --parameters file://./avalon.json

Cleaning up the stacks

When a stack deployment fails, you'll need to clean out everything before you can deploy again. Generally, CloudFormation does a good job of cleaning up resources it created, but some will need to be deleted manually:

  1. In CloudFront, first Disable, and, later, Delete any endpoints created by the Avalon stack.
  2. In S3, delete newly created S3 buckets, including the masterfiles and derivatives buckets.

Configuring Avalon

Config changes can be made through the environment variables in Elastic Beanstalk's Software Configuration section.

Using CloudFront secured streaming

Configure the Avalon player so that hls.js includes authenticating cookies in its requests https://github.com/nulib/avalon/commit/94cd48565eafda6d50bc8bd78e303afb604ec7c4

Deploying new Avalon releases

Avalon can be updated easily by overwriting the avalon.zip file in the deploy bucket. A triggered S3 event will tell CodePipeline to pick up and deploy this zip file to Elasic Beanstalk's web and worker environments.

VPN

An OpenVPN has been created to allow easy access to the VPCs in the stack. Point your browser to http://vpn.my-infrastructure.mydomain.com, login with your previously chosen username and password, and download the OpenVPN client. After connecting to the VPN, you'll be able to SSH into your EC2 instances using your EC2 key.