Scaling your Workflows: Cloud vs. HPC

In the world of big data, your local computer may not have enough compute resources to process what you need to. There are two ways to scale your compute resources:

1. High Performance Computing via a Supercomputer (Resources and User Guides)
2. Cloud Computing

You can read more about the pros/cons of these and how to use Earth Lab access to these here.

While parallelization and container vs virtual machine (VM) implementations are relatively similar between these two scalable compute infrastructures, the chosen workflow will try to optimize different things. On the university HPC system, you try to game the queue system and maximize your resource allocation. On AWS, you're trying to keep costs down. You can work with Analytics Hub staff to learn more about how to alter your workflows.

FAIR Data Principles

CARE Data Principles

Collaborative Coding Best Practices

Transitioning your code for Open Source Software Development