GCP Google Compute Engine - ghdrako/doc_snipets GitHub Wiki

gcloud services enable compute.googleapis.com

It has storage, compute, memory, and network capabilities as well as different operating systems (OSes) already built in, which can be customized to your needs (more CPU, memory, storage, and so on).

If you want to offload even more work to the cloud, you can use an API called Managed Instance Groups (MIGs), which lets you manage multiple VMs as one.

This allows the user's identity to be verified over HTTPS and grants access only if permitted. This service is especially useful for remote workers as it negates the need for a company VPN to authenticate user requests using on-premises networks. Instead, access is via an internet-accessible URL. When remote users need to access an application, a request is forwarded to Cloud IAP, and access will be granted (if permitted). Additionally, without the overhead of a traditional VPN, manageability is simplified for the administrator.

prerequisites: configure firewall rules to block access to the VMs that are hosting your application and only allow access through IAP.

TCP forwarding IAP has a port forwarding feature that allows us to Secure Shell (SSH) or Remote Desktop for Windows (RDP) onto our backends from the public internet, which prevents these services from being exposed openly to the internet. We are required to pass authorization and authentication checks before they get to their intended resources, thus reducing the risk of connecting to these services from the internet. IAP's forwarding feature allows us to connect to random TCP ports on our Compute Engine instances. IAP will then create a listening port on the localhost that forwards all traffic to our instance. IAP will then wrap all traffic from the client in HTTPS and allow access to the interface and port if we successfully pass authentication. This authentication process is based on configured IAM permissions. There are some limitations that we should be aware of. Port forwarding is not intended for bulk transfer of data, so IAP reserves the right to apply rate limits to prevent misuse of the service. Additionally, after 1 hour of inactivity, IAP will automatically disconnect our session, so applications using this should have logic to handle re-establishing a tunnel after being disconnected.

are short-lived instances that last up to 24 hours and can reduce the costs of GCE workloads (especially fault-tolerant workloads) by up to 80%. Preemptible VMs are great options for batch jobs (you can start up as many preemptible VMs as you want) and they shut down gracefully, giving your app time to save its data. Most importantly, preemptible VMs cost significantly less than regular VMs and are easy to create and end.

It's also ideal for bringing existing systems or applications to the cloud (using a migration strategy as simple as lift and shift). On a similar note, GCE is also great for applications that require specific third-party solutions or a specific (or custom) OS or kernel, as it is most likely to be able to accommodate all of your licensing requirements. The powerful VMs are also ideal for large databases and other resource-intensive workloads. Finally, as with GKE, GCE also supports network protocols other than HTTP/S.

If you do not have an external IP, then a bastion host should be used, also referred to as terminal servers. If you need access to resources that have internal IP addresses only, then bastion hosts are configured with external IP addresses and act as a gateway to your internal resources.

Enable this option to encrypt data while in use. This will keep data encrypted in memory during processing without Google having access to the encryption keys

Reservations can be created for VM instances in specific zones using custom or predefined machine types. Use reservations to ensure resources are available for your workloads when you need them; however, as soon as you create a reservation, you will immediately begin paying for it until the reservation is deleted.

Metadata for every virtual machine (VM) instance is stored on a metadata server and can be queried from the Compute Engine API or from the instance itself. Your VM automatically has access to the metadata server API without any additional authorization. Metadata is stored as key:value pairs. We can pull information such as hostname, instance ID, or any custom metadata we apply. There is a default set of metadata keys that are available for VMs running on Compute Engine. You can also use your own custom metadata keys on an individual VM or project. We can also provide configuration information to applications using metadata, which will negate the need to store this data in the application itself. We also use metadata to reference the startup script to be executed during our instance deployment.

When populating the variables for a GCE deployment, we can also set a startup script if we wish. This is a script that will run when your instance boots up or restarts. Common examples would be to install specific software or ensure services are started. These scripts are specified through the same metadata server we previously mentioned and use the startup script metadata keys.

gcloud compute instances create cloudarchitect ––tags http–
server \
––metadata startup–script='#! /bin/bash
# Installs apache and a custom homepage
sudo su –
apt–get update
apt–get install –y apache2
cat <<EOF > /var/www/html/index.html
<html><body><h1>Hello World</h1>
<p>This page was created from a simple start up script!</p>
</body></html>
EOF'

It is also possible to specify a script that resides on Google Cloud Storage.

gcloud compute instances create example–instance ––scopes storage–ro \
––metadata startup–script–url=gs://cloudarchitectbucket/startupscript.sh

We should note that startup scripts will only be executed when a network is available.

gcloud compute instances create www1 \
  --image-family debian-9 \
  --image-project debian-cloud \
  --zone us-central1-a \
  --tags network-lb-tag \
  --metadata startup-script="#! /bin/bash
    sudo apt-get update
    sudo apt-get install apache2 -y
    sudo service apache2 restart
    echo '<!doctype html><html><body><h1>www1</h1></body></html>' | tee /var/www/html/index.html"


gcloud compute firewall-rules create www-firewall-network-lb \
    --target-tags network-lb-tag --allow tcp:80

If we want to specify a shutdown script, then we will browse to the metadata section and add shutdown–script as the key and add the contents of the script as the value.

Preemptibility brings the cost of an instance down but will generally only last a maximum of 24 hours before it is terminated. Certain actions, such as stopping and starting an instance, will reset this counter.

Batch-processing jobs could run on preemptible instances, and if some of the instances are terminated, then the process will slow down but not stop entirely. This allows batch processing to occur without any additional workload on existing instances. We should note, however, that due to the nature of preemptible instances, there is no GCP Service-Level Agreement (SLA) applied, and they can be removed by the provider with only 30 seconds' notification.

GCP offers the ability to harden your VM instance with security controls, which will defend against rootkits, bootkits, and kernel-level malware. GCP uses a virtual Trusted Platform Module (vTPM) to provide a virtual root of trust to verify the identity of the VM and ensure they are part of a specified project or region. The vTPM generates and stores encryption keys at the guest OS level. It should be noted that this does not add any extra cost to your VM deployment.

This option allows us to either delete or keep the boot disk when the instance is deleted

Sole tenancy was introduced to GCE in 2018 and is a physical Compute Engine server designed for your dedicated use. In other words, the underlying host hardware and hypervisor handles only your GCE instances. A use case for this setup could be security requirements. Many companies have compliance or regulations that require a physical separation of their compute resources.

OS Login lets you use Compute Engine Identity and Access Management (IAM) roles to grant or revoke SSH access to your Linux instances. OS Login is an alternative to managing instance access by adding and removing SSH keys in metadata.

OS Login simplifies SSH access management by linking your Linux user account to your Google identity.

Limitations: not suported public GKE clusters,Fedora CoreOS,Windows Server and SQL Server

1. You can enable or disable OS Login by setting metadata values at the instance or project level. Enabling or disabling OS Login in instance metadata overrides the value that is set in project metadata.

# enable on project level
gcloud compute project-info add-metadata \
    --metadata enable-oslogin=TRUE
# enable on instance
gcloud compute instances add-metadata VM_NAME \
    --metadata enable-oslogin=TRUE
# enable during create instance
gcloud compute instances create INSTANCE_NAME  --zone=ZONE \
 --metadata=enable-oslogin=TRUE

2. Configure OS Login roles on user accounts

• roles/compute.osLogin, which doesn't grant administrator permissions
• roles/compute.osAdminLogin, which grants administrator permissions

Troubleshooting

gcloud compute ssh your-server-name --zone us-central1-a --project your-project-id --internal-ip
gcloud compute instances remove-metadata your-server-name --keys=enable-oslogin --zone us-central1-a --project your-project-id
gcloud  --verbosity=debug compute ssh your-server-name --zone us-central1-a --project your-project-id --internal-ip

An image in Compute Engine is a cloud resource that provides a reference to an immutable disk. That disk representation is then encapsulated using a few data formats. For an image to be bootable, it must contain the following:

• A partition table: either a master boot record
• A bootable partition For a disk to be imported as a Compute Engine image, the disk's bytes must be written to a file named disk.raw.

Use operating system images to create boot disks for your instances

• Public Images - (free or payied - premium images) Container-optimized VM images
• Custom Image - Custom image storage -costs Image families - always points to the most recent image in that family so your instance templates and scripts can use that image without having to update references to a specific image version. Custom image families

Creating customized images

• Manual baking - creating a custom image from that instance who is manualy provisioned
• Automated baking - eg. Packer You can also use Packer as part of a Spinnaker pipeline to produce images that are deployed to clusters of instances.
• Importing existing images

Custom images are available per project and can be created from source disks, images, snapshots, or images stored in cloud storage.

• Available globally
• Stored storageLocation (Cloud Storage multi-region,dual region,region)

By default, when creating a machine image from an instance, the machine image is stored in either the Cloud Storage multi-region bucket that contains the source instance, or the geographically closest Cloud Storage multi-region bucket to the source instance.

A machine image is a Compute Engine resource that stores all the configuration, metadata, permissions, volume mapping and data from multiple disks of a virtual machine (VM) instance. When to use:

• Multiple Disk backups -guarantees that the data across disks is captured in a crash-consistent manner at a given time. Compute Engine uses globally consistent timestamps to ensure this guarantee. Note: This consistency guarantee only applies at the I/O operation level or crash level. This guarantee isn't applied at the application level. Consistency guarantees at the application level would require OS and application participation.
• Differential disk backup
• Instance cloning

In contrast to custom images that capture the contents of a single disk, machine images — which can be created whether the source instance is running or stopped — contain multiple disks as well as other data required to create a new instance. This includes instance properties like machine type, labels, volume mapping, and network tags; the data of all attached disks; instance metadata; and permissions, including the service account used to create the instance.

gcloud compute machine-images describe <MACHINE_IMAGE_NAME>                # find the storage size in bytes in field totalStorageBytes
gcloud compute machine-images create <MACHINE_IMAGE_NAME> \
                                     --source-instance=<SOURCE_VM_NAME>   # create machine image
gcloud beta compute instances create VM_NAME --zone=ZONE \
                       --source-machine-image=SOURCE_MACHINE_IMAGE_NAME # create vm from machine image 
gcloud compute machine-images add-iam-policy-binding MACHINE_IMAGE_NAME \
                                --project=MACHINE_IMAGE_PROJECT \
                                --member='ACCOUNT_EMAIL' \
                                --role='roles/compute.admin'             # grant the permissions on the machine image in different project
gcloud projects add-iam-policy-binding MACHINE_IMAGE_PROJECT \
      --member='ACCOUNT_EMAIL' \
      --role='roles/iam.serviceAccountUser'                              # grant permitionon all machines in different project
                                                                         

Snapshots are also global resources, which means they are accessible by any resource in the same project.

Snapshots incrementally back up data from your persistent disks. After you create a snapshot to capture the current state of the disk, you can use it to restore that data to a new disk. Snapshots are incremental and automatically compressed. You can create snapshots from disks even while they are attached to running virtual machine (VM) instances. The lifecycle of a snapshot created from a disk attached to a running VM instances is independent of the lifecycle of the VM instance.

Snapshots are global resources, so any snapshot is accessible by any resource within the same project. Restrictions:

• You cannot change the storage location of an existing snapshot.
• You can snapshot your disks at most once every 10 minutes.
• You cannot edit the data stored in a snapshot.

It is possible to snapshot a persistent disk and use this as part of a new instance creation, even if is still attached to a running instance. Snapshots can be used to create a new custom image or instance. Snapshots can be created for a number of reasons, one of which is data protection. It is recommended to have a snapshot schedule to reduce data loss. Snapshots are also global resources, which means they are accessible by any resource in the same project.

Snapshots are incremental and, therefore, will contain blocks that are different from the previous snapshot. Therefore, if you perform regular snapshots, the cost is less than performing a regular full image of the disk. Each snapshot is stored across multiple locations within GCP for redundancy. Snapshots can be stored across projects with the correct permissions.

Snapshots are incremental and, therefore, will contain blocks that are different from the previous snapshot. Therefore, if you perform regular snapshots, the cost is less than performing a regular full image of the disk. Each snapshot is stored across multiple locations within GCP for redundancy. Snapshots can be stored across projects with the correct permissions.

gcloud compute snapshots create SNAPSHOT_NAME \
    --source-disk SOURCE_DISK \
    --source-disk-zone SOURCE_DISK_ZONE
gcloud compute snapshots describe SNAPSHOT_NAME
gcloud compute snapshots list --project PROJECT_ID

# Share snapshot data across projects
gcloud compute snapshots create SNAPSHOT_NAME \
    --source-disk https://www.googleapis.com/compute/v1/projects/SOURCE_PROJECT_ID/zones/ZONE/disks/SOURCE_DISK_NAME \
    --project DESTINATION_PROJECT_ID

Enable this option to encrypt data while in use. This will keep data encrypted in memory during processing without Google having access to the encryption keys

Enable this option to deploy and launch a Docker container using a Container- Optimized OS (COS). Note you can only deploy one container for each VM instance.

gcloud compute ssh <vmname> -- 'cat /etc/hosts'
gcloud compute scp <source dir> <vmname>:<vmdir> --recurse

Static address cost you if you not using it in vm

# verify comunication with machine
curl http://$(gcloud compute instances list --filter=name:gcelab2 --format='value(EXTERNAL_IP)')