S3 Cloud Storage - FLARE-forecast/flare-forecast.github.io GitHub Wiki

Description

The following data is stored on S3 buckets:

  • Input data from sensors
  • Driver data from NOAA
  • Intermediary data generated during FLARE forecasts
  • Forecast outputs and graphs produced at the end of the forecast workflow

To access the S3 storage, we can use different tools such as Rclone or MinIO client.


Setup and Configuration

Rclone

Rclone is a command-line program used to manage files on cloud storage, including S3 object stores.

Install Rclone on Linux

Follow the official Rclone installation guide.

Rclone Configuration

On Ubuntu, the Rclone config file is typically located at:

~/.config/rclone/rclone.conf

Example Configuration File

The content of the file should be similar to this:

[s3flare]
type = s3
provider = Minio
access_key_id = <your_id>
secret_access_key = <your_password>
endpoint = https://s3.flare-forecast.org

Mount FLARE S3 Storage

A crontab job on Jetstream2 VM ensures that S3 storage is mounted on reboot:

@reboot sleep 30 && /usr/bin/rclone mount --read-only s3flare: ~/s3flare-directory

Read-Only Mode

🚀 Read-only mode is strongly recommended to prevent unwanted changes to the S3 objects.

⚠️ **GitHub.com Fallback** ⚠️