AWS S3 Storage - webbukkit/dynmap GitHub Wiki

This option offers both an alternative storage solution, and an external web serving solution, allowing the map data to be published to, and then served from, an AWS S3 compatible 'bucket', providing a convenient option for servers on hosting environments where the option for large local storage and/or the need for an additional TCP/IP port for the internal web server is a problem.

To set up this solution, it is required for the server administrator to have an AWS (Amazon Web Services) account. Alternatively, it is possible to use any other S3-Compatible service (See Alternative Services)

Amazon AWS Configuration

Note: while there is a 'free tier' on AWS for new accounts, and this will allow up to about 5GB to be hosted in S3 for free for the first 12 months of an account, it IS very important to realize that it will be quite possible to run up non-trivial monetary charges when using S3. We will offer some advice on how to optimize this, but understand that you can and should monitor any expenses incurred, and not assume this is a free or risk free option.

To set up the AWS S3 storage option, do the following:

  1. If needed, set up an AWS account for yourself (see article here)

  2. Use your account to set up a new S3 bucket, and configure it to act as a static web site (see article here). It is important to select an appropriate region (this controls which part of the world your S3 bucket is hosted - closer to the bulk of your community is probably best) and record the ID of this region (e.g. us-east-1, eu-west-1) (see article about regions). Also, remember the name you've selected for your S3 bucket - this name must be unique among all S3 buckets in the region. When setting up the Static Web Site settings, use 'index.html' for the Index document setting, and 'images/blank.png' for the Error document setting (adjust this setting appropriately if you decide to include a prefix path on your published web - see below). Also, DO NOT ENABLE BUCKET VERSIONING - this can result in unbounded growth in storage use in the bucket, given how often tiles and status data are updated!

  1. Take note of the 'Bucket website endpoint' URL on the Properties tab for the now-configured S3 bucket. This is the default URL for your web site - in order to make this URL more 'friendly', you can configure a CNAME record on your DNS service to direct your desired DNS name to this bucket (DNS configuration is very DNS provider specific process, so you're on your own with sorting that out - every different DNS hoster has documentation on how to do this).
  1. While not necessary, it is VERY strongly suggested that you create a dedicated service IAM user ID under your account, to be used by Dynmap on your server, and to give this user ID the minimum privilege needed to publish data to your S3 bucket. Do NOT use your root account - anyone who gets access to your Minecraft server could use the account to run up arbitrary amounts of charges on your account! To do this process, follow the following procedure (see article here) with these choices suggested:

  • For Select AWS credential type, select Programmatic access

  • Select Attach existing policies directly on Set permissions, and then select Create Policy. Select JSON, and apply the following JSON (with your S3 bucket name substituted for YOURBUCKETNAME):

{ "Version": "2012-10-17", "Statement": [ { "Action": [ "s3:DeleteObject", "s3:GetObject", "s3:ListBucket", "s3:PutObject" ], "Effect": "Allow", "Resource": [ "arn:aws:s3:::YOURBUCKETNAME", "arn:aws:s3:::YOURBUCKETNAME/*" ] } ] }

  • This will give your IAM account just the privileges needed to support access to your web site bucket. When you see 'Review Policy', give the access policy an appropriate name (e.g. 'DynmapBucket'). Then, return to the tab where the user is being created, refresh the Policy list, and type the name in the Search to find the new Policy. Be sure to check the checkbox to grant the user this policy.
  • Once you finish the creation, be careful to record the Access Key ID and Secret Access Key value - the secret key will never be shown again, and you'll need to reset the keys if you lose them. This values will be used for the aws_access_key_id and aws_secret_access_key settings, respectively.
  1. Now, set up the storage option in configuration.txt:

Make sure other storage settings are commented out.

  1. The following additional settings changes should also be done:
  • Uncomment the JsonFileClientUpdateComponent section, and comment or delete the InternalClientUpdateComponent. Also, set allowwebchat to false (it is unsupported), and it is suggested to set writeinterval to 15 (or higher) AND to set the updaterate setting to 1000 times the same value - this will reduce how often chat, player positions, and other data are written to the S3 bucket (which can reduce API call charges (below)), and how often web users' browsers will poll the bucket for those updates.

  • It is strongly suggested that the new defaulttilescale setting be used to reduce the number of tiles posted to S3, while increasing the size of the tiles (a setting of 2 is recommended). This will reduce the number of GET calls needed by the users to load the tiles on their screen, while still generating about the same bandwidth. It is also suggested to use the default image-format setting (jpg-q90) or the webp format, in order to minimize file sizes (png is NOT recommended, as this results in significantly larger files, and associated storage and bandwidth costs).

  • The internal web server must be disabled by setting the disable-webserver setting to true

  • If you wish to publish multiple servers to the same S3 bucket, the prefix setting can be used to put the files for the server under a specific path in the bucket: this will correspond to the map for the server being under that path relative to the Bucket Website Endpoint (e.g. prefix: test123/test would result in a URL akin to http://dynmap.s3-website-us-east-1.amazonaws.com/test123/test/). If set, the Error document setting for the bucket should correspond to one of the prefix values (so that the images/blank.png file from one of the servers will be used for all 40x errors across the servers in the bucket).

  1. That should be it - restart the server and see if it is able to access the bucket. The server will automatically publish the static web site files from Dynmap to the bucket, marker images, and start using the bucket for publishing files.

AWS Cost Considerations

At the present time, the following costs are representative for use of AWS in us-east-1 in the United States - prices in other countries, or in the future, may vary (see pricing here).

  • $0.023 per GB per month for storing data in the S3 bucket
  • $0.005 per 1000 requests to PUT, COPY, POST, LIST (these are done for updating tiles, zoom tiles, and reporting updates, chat messages and the like)
  • $0.0004 per 1000 requests for GET (each tile loaded by each user browser is one of these calls, as are polling for updates)
  • $0.09 per GB transferred from the bucket (tile reads, update reads, etc. Browser caching will help this). In general, data transfer IN to the bucket has no charges (other than the PUT API call, above).

Alternative Services

It is possible to configure dynmap to use most other S3-Compatible storage APIs. Some options are listed below.

  • Cloudflare R2
    • $0.015 per GB per month, $0.0045 per 1000 Update requests, $0.00036 per 1000 GET requests
    • Free egress
    • Free Tier: 10 GB per month, 1 million Update requests per month, 10 million GET requests per month
  • Backblaze B2
    • $0.006 per GB per month, $0.004 per 1000 LIST, COPY requests (2500 free per day), $0.0004 per 1000 GET requests (2500 free per day)
    • Free egress up to 3x amount stored then $0.01/GB, Free PUT/POST
    • Free tier: 10 GB per month
  • Wasabi
    • $0.0068 per GB per month
    • Free egress
    • Free egress and API usage is bound by a fair use agreement, so may not be suitable for dynmap
    • 30 Day free trial (up to 1 TB)
  • Linode Object Storage
  • DigitalOcean Spaces
  • many more...

Most large hosting providers are now offering object storage which is compatible with the S3 API, so there are many options to choose from. It would be well out of scope to detail how to use each one of these services here, so if you are unsure it is recommended you stick with AWS.

Once you have signed up for a service and created a bucket, update the settings in configuration.txt as per the following table.

Setting Set To
type aws_s3
bucketname The name of the bucket you created. Default is dynmap
region If your provider requires that you set a region, set it here. For some providers, this will do nothing or is not required so set to ""
aws_access_key_id Your access key ID
aws_secret_access_key Your access key secret
override_endpoint Your provider's S3 endpoint. If set to "", it assumes an AWS endpoint and generates based on the region setting

Once these settings are configured, refer to Step 6 in Amazon AWS Configuration for additional dynmap config.