Cyber Infrastructure Glossary - earthlab/earth-lab-operations GitHub Wiki
Cyber Infrastructure (CI): The collection of interconnected computing systems, networked devices, software, and data that enable the exchange, processing, storage, and analysis of digital information.
Virtual machine (VM): A computer system that is connected to remotely from a local computer
Cluster: A collection of virtual machines, usually consisting of a main VM and several worker VM
Node: Any virtual machine inside of a cluster
Main node: Node used for orchestrating and balancing the load on a cluster. The main node can be responsible for assigning users to each worker node.
Worker node: Node used for running the main application and report to the main node
CPU: Central processing unit. Executes instructions of the computer programs. Most modern computers have several of these, which enables programs to run in parallel
GPU: Graphics processing unit. Like a CPU but specialized at the hardware level for rendering graphics. Also, very good at training and running machine learning models
Disk drive: Hardware which stores the data on a computer.
Disk space: The maximum amount of data that can be added and stored by a disk drive
Volume: A virtual disk drive that is added to a virtual machine to increase its storage. Can usually be detached and reattached, so that virtual machine can be shut down and restarted.
Memory: The amount of the data that a computer can load at one time for use by its programs. This is usually much less than the computer’s disk space. You may be able to store a 50GB data file on your computer, but if your computer’s memory is only 16GB you cannot load the file to be used by a program
Storage: The amount of space for writing data to. Can be used to describe disk space on a personal computer or VM, or space in a data repository like CyVerse Data Store
Host name: Used as a readable name in place of an IP address. “google”, for example, is the hostname for the IP address that google.com is using
IP Address: A unique string of characters, usually numbers separated by a ‘.’ that identifies each computer on a network so that data can be correctly routed to it.
RAM: Random access memory. This is short term memory that the processor is able to use. This is usually much less than the disk space available on a computer. If you wanted to read a 10 GB file into a DataFrame in R or Python, your computer would need at least 10 GB of available RAM to do this.
*Docker: An open source platform that allows developers to build, run, and manage application containers
Container: A standalone package of software that includes everything needed at runtime to execute an application
Image: Template of instructions for building a docker container that can run on the
Docker platform
CyVerse: An NSF-funded project with the mission to provide life scientists with computational infrastructure to handle large datasets and complex analyses
Data Store: A secure data storage and management system that is hosted on the web. Users can upload files and directories and set access rules for each. Data on the store can be transferred in many ways such as on the web, through command line applications, and from Python and R scripts, for example. Earth Lab / ESIIL have practically unlimited space on the Data Store.
CACAO: CyVerse’s Cloud Automation and Continuous Analysis Orchestration, one of multiple web-based user interfaces for using and containerizing a Jetstream2 allocation
Jetstream2: A collection of hardware and software interfaces to NSF funded supercomputing resources. Jetstream2 is where CACAO provisions most runtime environments.
ACCESS: Advanced Cyberinfrastructure Coordination Environment: Services and Support is an NSF funded service which manages each organization’s access to resources on Jetstream2. ACCESS must grant an allocation before using Jetstream2 resources.
AWS: Amazon Web Services is a platform which offers an entire suite of cloud infrastructure tools. Its most widely used resource at Earth Lab is its on demand, pay as you go cloud computing service.
S3: Simple Storage Service allows users to upload files and directories to AWS cloud for long- or short-term storage. S3 data can be accessed from other AWS resources or from a personal computer.
EC2: Elastic Compute Cloud is AWS’s on demand cloud computing platform. It allows users to create EC2 instances, which are virtual computers that can be remotely accessed. Users can customize hardware capabilities and software requirements on each EC2 instance to fit their computing needs.
Elastic IP: AWS owned static IP address that can be assigned to each EC2 instance so that the instance can be accessed remotely.
Security Group: Defines the accessibility of each port on an EC2 instance. For example, if a user wants to use ssh to connect to their instance on port 22, the security group must allow traffic on port 22.
Port: Connection point between a computer and an external or internal device. For example, when a Docker container is used to run a JupyterLab application on port 8888, all of the traffic from the user connected to JupyterLab goes through port 8888 before executing commands on the host computer.
Server: A server is a computer or group of computers that completes requests given to it by another computer and its user or “client”.
Client: Request the activity of one or more servers to complete its tasks. When you access a webpage, your browser is the client that is requesting activity of the servers that the webpage is hosted on.
Core hour: A metric used for keeping track of computer hardware usage. A core hour is defined as the number of cores used multiplied by the number of hours. For example, if you used an 8 core machine for 4 hours, you would have used 32 core hours.
ssh: The “Secure Shell” program allows users to connect computers securely over otherwise insecure networks. It is most used for remote login and command line execution from one machine to another. The command structure for remote login if a key pair is configured is: ssh -i @
User: username of the account that you are trying to log into. ssh will verify the credentials you supplied with the credentials of this user at the specified address.
Address: Network address of the computer to connect to. This is usually a remote IP address.
Key File: Path to a private key file whose public key file has been added to the machine that is being connected to
scp: The “Secure Copy” program allows users to transfer data securely over an otherwise insecure connection. The command structure for file transfer from local to remote if a key pair is configured is: scp -i @:
DNS: Domain Name System maps domain names to IP addresses so that users can connect to resources in a more user-friendly way. Instead of having to remember the IP address for Google, the DNS maps “google.com” to the IP address for you.
API: An Application Programming Interface is a way for a user to interact with an application through code. The most common way of interfacing with an application is through a graphical interface like a website. In contrast, an API allows a user to automate their requests to an application or make the requests as part of a scripting workflow.
Jupyter Notebook: A file structure that allows one to write code into a collection of executable cells. The file can be edited from web browser connected to a Jupyter Lab server.
README: A text file attached to a repository of code that explains things such as: how to download and install the code, how to use the code, licensing, and how to contribute.
Open science: movement to make scientific research (including publications, data, physical samples, and software) and its dissemination accessible to all levels of society, amateur or professional.
HPC: High performance computing. CU Boulder’s HPC has massive clusters of general compute CPU nodes, GPU nodes, and high memory nodes. Code is run on an HPC by submitting jobs that run your scripts. More info on CU’s HPC can be found here: https://curc.readthedocs.io/en/latest/
SLURM: Simple Linux Utility for Resource Management is the program which allows users to submit and manage jobs to the HPC. It is has a command line interface.
PBS: Portable Batch System. A program for submitting and managing jobs on the HPC. CU HPC does not use PBS.
Job: A request for compute time to run code on the HPC. A Job has many configurable parameters like which hardware should be allocated and what the maximum runtime should be.
Queue: Manages the jobs submitted to the HPC that still need to be run. Jobs are organized in the queue by priority. Priority is usually determined by the amount of hardware requested vs what is available.
PetaLibrary: CU HPC’s large data storage system. This is a periodically backed up Linux file system that can store a lot of data. More info can be found here: https://www.colorado.edu/rc/resources/petalibrary
IdentiKey: Your unique ID given by CU Boulder. Is used for many of CU’s federated authentication systems, like log on to AWS.
VPN: A virtual private network is a mechanism for creating a secure connection between two networks using an insecure medium such as the public internet.
Command Line Interface (CLI): An interface to an application by commands executed on a terminal or other shell request actions of the application.
Integrated Development Environment (IDE): Text editing software that has general and coding language specific utilities that increase productivity. Most IDEs are integrated with git and package managers.
Package manager: Framework for installing packages from a remote repository to your local computer.
Conda: Package manager which installs software from the Conda repository. Can be used to make virtual runtime environments.
PIP: Package manager for Python. By default PIP searches for packages hosted on PyPi servers.
CRAN: Package manager for R. CRAN searches for packages that are hosted on CRAN servers.
Mamba: Reimplementation of the Conda manager in C++. Allows for parallel downloading of repository data.
Virtual Environment: A file structure which contains a code interpreter and its software packages needed at runtime. Virtual Environments allow you to have many different versions of an interpreter and installed packages on a single computer.
Unit Testing: Functions which run code and make assertions about the results. Unit tests are usually atomic on the scale of a single use case. A unit testing suite generally will run each line of a project’s code and test that assertions about the results are correct.
Continuous integration and development (CI/CD): Process by which automation is introduced into the development process of applications. For example, when code is pushed to GitHub unit tests can be automatically performed on the code.
Bash script: A computer program designed to be run by Bash.
Bash: The Bourne-Again Shell is an improved Unix shell
Git: A version control system used for tracking changes in a file system
GitHub: A collection of git repositories hosted on the web, often used for version control
GitLab: A collection of git repositories hosted on the web, often used for version control
Git Bash: A cross platform unix shell provided with a git installation. It is mainly used for running git CLI commands, but can be used more generally.