Existing S3 Snapshot Repo Configuration - opensearch-project/opensearch-migrations GitHub Wiki
This page lays out additional configuration that is necessary for configuring the Migration tooling to use an existing S3 snapshot repository. These steps are expected to be consolidated in the near future to be a single configuration update.
Before deployment, the CDK context options should specify the desired existing S3 snapshot repository to use for Reindex-from-Snapshot
| Name | Example | Description |
|---|---|---|
<BUCKET_NAME> |
app-us-east-1-bucket | The name of the existing S3 bucket |
<REPOSITORY_PATH> |
app-repo | The path used by the source cluster snapshot repository to store snapshots |
<SOURCE_VERSION> |
ES_7_17 | The source cluster version in the format in the format of OS_x_y or ES_x_y
|
<AWS_REGION> |
us-east-1 | The AWS region of the S3 bucket |
<SNAPSHOT_NAME> |
app-snapshot | The name of the existing snapshot in the source cluster snapshot repository, or the name that will be used if creating a new snapshot (default is rfs-snapshot) |
{
"default": {
...
"reindexFromSnapshotServiceEnabled": true,
"reindexFromSnapshotExtraArgs": "--s3-repo-uri s3://<BUCKET_NAME>/<REPOSITORY_PATH> --source-version <SOURCE_VERSION> --s3-region <AWS_REGION> --snapshot-name <SNAPSHOT_NAME>"
}
}After deployment, the task roles for Reindex-from-Snapshot and the Migration Console will need to be updated to have S3 permissions for the existing S3 bucket.
These roles would look similar to OSMigrations-dev-us-east--reindexfromsnapshotTaskRo-lasdkfj223 and OSMigrations-dev-us-east--migrationconsoleTaskRole1-akdenrt234
A policy similar to below could be added to these task roles:
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "s3:*",
"Resource": [
"<BUCKET_NAME>/*",
"<BUCKET_NAME>"
],
"Effect": "Allow"
}
]
}
After deployment, the migration_services.yaml on the migration-console will need to be updated in order to perform any metadata migrations as well as perform any snapshot actions, such as creating any new snapshots in the existing S3 snapshot repository.
- Access the Migration Console, steps here
- Fill in the snapshot details to use:
snapshot:
snapshot_name: <SNAPSHOT_NAME>
s3:
repo_uri: s3://<BUCKET_NAME>/<REPOSITORY_PATH>
aws_region: <AWS_REGION>
- Use Vim to modify the
migration_services.yamland add the details from the previous step
vim /etc/migration_services.yaml