TerraformQuickNotes - henk52/knowledgesharing GitHub Wiki
- Azure providers and network
- Provider Configuration
- Terraform Course - Automate your AWS cloud infrastructure
This is an attempt to set-up a K8s cluster in Azure, using the information in compute resources
See terraform plugin at: Azure VM
- Desired state - what is in the tf files
- Current state - what is in the real world.
- terraform init
- terraform validate
- terraform plan
- terraform apply
x. terraform destroy !!! it seems it also tries to delete the resource group :-(
- tf workspace -h
- terraform workspace new NEW_WORKSPACE
- terraform workspace list
- .terraform/ - gets create when any plugin is initialized
- terraform.tfstate - represents all the states for terraform
- keeps track of how terraform believes the state is.
- .terraform.lock.hcl - keeps track of which module version was used.
- to commit of not to commit
- commit if you want the deployment unaffected by modul updates
- do not commit it, if you want to know immediatly if a module update breakes the deployment.
<BLOCK_TYPE> "<BLOCK_LABEL>" "<BLOCK_LABEL>"
- BLOCK_TYPE
- data - (referencing) - get information from the cloud provider.
- input - (variable) -
- local - (variable) -
- modules - (referencing) -
- output - (variable) -
- provider - (fundamental) -
- resource - (fundamental) -
- terraform -
- variable -
- IDENTIDIER
- Arguments
- Attributes - values exposer by a particular resource.
- meta-arguments - change the behavior of the resource
- count
- depenends_on
- for_each
- lifecycle
- provider
-
Required terraform version
required_version = "~> 1.3.9"
-
list required providers
required_providers {
-
terraform backend
backend "s3" {
-
experimental language features?
-
Passing MetaData to providers?
-
Only constants can be used.
- my not refer to named objects
- e.g. resources, input variables
- may not use any of the terraform language built-in functions.
- my not refer to named objects
-
aka
- terraform settings block
- terraform configuration block
required_providers {
# aka local name.
# it seems this 'aws' could be anything, as long as there is only one provider using that name.
# please use preferred names as used on hashicorp site.
aws = {
source = "hashicorp/aws"
version = "~> 3.0"
}
}
-
source
consists of three parts delimited by '/'- [HOSTNAME/]/
- Hostname: defaults to https://registry.terraform.io
- namespace:
- type:
- [HOSTNAME/]/
- Official - owned and maintained by hashicorp
- Verified - Owned and maintained by third-party. Hashicorp has verified the authenticity of the owner.
- Community - individual maintainers or groups of maintainers
- Archived - no longer maintianed.
- Only use submodules that have the checkmark.
-
Interact with the target, e.g. aws cloud
-
provides all the resource_ types
-
TODO can you configure multiple regions? does it then apply to every region in one call?
-
Create resource
- Create resources that exists in the configuration but are not associated with a real infrastructure object in the state.
-
Destroy resource
- Destroy resources that exists in the state but no longer exist in the configuration.
-
Update in-place resources
- Update in-place resources whose arguments have changed
-
Destroy and re-create
- Destroy and re-create resources whose arguments have changed but which cannot be updated in-place due to remote API limitations.
-
resource "aws_s3_bucket" "name" {
- resource - block type
- aws_s3_bucket - resource type
- name - resource local name
- available in the same module
- but not outside the module.
-
things inside the block is generalled called
- arguments
- argument value
-
meta-arguments - can be used to change the behavior of resources
- e.g.
provider = aws.aws-west-1
- e.g.
-
resource arguments
- e.g.
cidr_block = "10.2.0.0/16"
- argument values can make use of expressions or other tf dynamic language features.
- e.g.
-
required_version = "~> 0.14.8"
- allows: 0.14.9, 0.14.10 etc
- denys: 0.15 and above
-
required_version = "~> 0.14"
- allows: 0.14.x, 0.15.x etc
- denys: 1.x.x and above
See: [www.terraform.io/downloads.html](Install Terraform)
wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
- sudo apt update && sudo apt install terraform
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.
-
Create the main.tf with the required provider, as per Azure Provider documentation
-
terraform init
-
git add .terraform.lock.hcl
-
az login
-
az account show
resource "libvirt_volume" "data-qcow2" {
name = "csgo-data.qcow2"
pool = "default" # List storage pools using virsh pool-list
format = "qcow2"
size = 45097156608
}
The disk can be formated using the cloudinit, see 'prepare an extra disk'
# Configure the Libvirt provider
terraform {
required_providers {
libvirt = {
source = "dmacvicar/libvirt"
version = "0.7.1"
}
}
}
# https://registry.terraform.io/providers/dmacvicar/libvirt/latest
# See: https://registry.terraform.io/providers/multani/libvirt/latest/docs
provider "libvirt" {
uri = "qemu:///system"
}
# Defining VM Volume
resource "libvirt_volume" "terraform-test-qcow2" {
name = "terraform_test.qcow2"
pool = "default" # List storage pools using virsh pool-list
source = "/disk2/jammy-server-cloudimg-amd64.qcow2"
format = "qcow2"
}
# Define KVM domain to create
resource "libvirt_domain" "terraform-test-vm" {
name = "tf_test_vm"
memory = "2048"
vcpu = 2
network_interface {
network_name = "default" # List networks with virsh net-list
bridge = "virbr0"
wait_for_lease = true
}
disk {
volume_id = "${libvirt_volume.terraform-test-qcow2.id}"
}
console {
type = "pty"
target_type = "serial"
target_port = "0"
}
graphics {
type = "spice"
listen_type = "address"
autoport = true
}
}
# Output Server IP
output "ip" {
value = "${libvirt_domain.terraform-test-vm.network_interface.0.addresses.0}"
}
See also:
- resource
resource "TYPE" "INTERNAL_NAME"
- TYPE: e.g. 'aws_instance'
- INTERNAL_NAME: name used by this terraform file to reference this resource
worker_node_ips = [
{
"arch" = "x86_64"
"autostart" = false
"boot_device" = tolist([])
"cloudinit" = "/hdd2/vmstoragepool/k8s_lfs_worker_commoninit_0.iso;45574da4-726b-4d6b-ae9a-94b1fdae66f9"
"cmdline" = tolist(null) /* of map of string */
"console" = tolist([
{
"source_host" = "127.0.0.1"
"source_path" = ""
"source_service" = "0"
"target_port" = "0"
"target_type" = "serial"
"type" = "pty"
},
])
"coreos_ignition" = tostring(null)
"cpu" = tolist([
{
"mode" = "custom"
},
])
"description" = ""
"disk" = tolist([
{
"block_device" = ""
"file" = ""
"scsi" = false
"url" = ""
"volume_id" = "/hdd2/vmstoragepool/k8s_lfs_worker_0.qcow2"
"wwn" = ""
},
])
"emulator" = "/usr/bin/qemu-system-x86_64"
"filesystem" = tolist([])
"firmware" = tostring(null)
"fw_cfg_name" = "opt/com.coreos/config"
"graphics" = tolist([
{
"autoport" = true
"listen_address" = "127.0.0.1"
"listen_type" = "address"
"type" = "spice"
"websocket" = 0
},
])
"id" = "7a41abbd-e9e2-46cc-84d4-b3617677e394"
"initrd" = ""
"kernel" = ""
"machine" = "ubuntu"
"memory" = 4096
"metadata" = tostring(null)
"name" = "k8s_lfs_worker_0_vm"
"network_interface" = tolist([
{
"addresses" = tolist([
"192.168.122.226",
])
"bridge" = ""
"hostname" = ""
"mac" = "52:54:00:E6:E1:A9"
"macvtap" = ""
"network_id" = "e26992cd-dc45-4f9c-bb9b-75ba01de9ca7"
"network_name" = "default"
"passthrough" = ""
"vepa" = ""
"wait_for_lease" = true
},
])
"nvram" = tolist([])
"qemu_agent" = false
"running" = true
"timeouts" = null /* object */
"tpm" = tolist([])
"type" = "kvm"
"vcpu" = 2
"video" = tolist([])
"xml" = tolist([])
},
{
"arch" = "x86_64"
"autostart" = false
"boot_device" = tolist([])
"cloudinit" = "/hdd2/vmstoragepool/k8s_lfs_worker_commoninit_1.iso;36f42771-4372-413c-8eb3-f6210106074d"
"cmdline" = tolist(null) /* of map of string */
"console" = tolist([
{
"source_host" = "127.0.0.1"
"source_path" = ""
"source_service" = "0"
"target_port" = "0"
"target_type" = "serial"
"type" = "pty"
},
])
"coreos_ignition" = tostring(null)
"cpu" = tolist([
{
"mode" = "custom"
},
])
"description" = ""
"disk" = tolist([
{
"block_device" = ""
"file" = ""
"scsi" = false
"url" = ""
"volume_id" = "/hdd2/vmstoragepool/k8s_lfs_worker_1.qcow2"
"wwn" = ""
},
])
"emulator" = "/usr/bin/qemu-system-x86_64"
"filesystem" = tolist([])
"firmware" = tostring(null)
"fw_cfg_name" = "opt/com.coreos/config"
"graphics" = tolist([
{
"autoport" = true
"listen_address" = "127.0.0.1"
"listen_type" = "address"
"type" = "spice"
"websocket" = 0
},
])
"id" = "edea633c-35e4-4115-8c33-edbd8c90537e"
"initrd" = ""
"kernel" = ""
"machine" = "ubuntu"
"memory" = 4096
"metadata" = tostring(null)
"name" = "k8s_lfs_worker_1_vm"
"network_interface" = tolist([
{
"addresses" = tolist([
"192.168.122.91",
])
"bridge" = ""
"hostname" = ""
"mac" = "52:54:00:76:0A:5A"
"macvtap" = ""
"network_id" = "e26992cd-dc45-4f9c-bb9b-75ba01de9ca7"
"network_name" = "default"
"passthrough" = ""
"vepa" = ""
"wait_for_lease" = true
},
])
"nvram" = tolist([])
"qemu_agent" = false
"running" = true
"timeouts" = null /* object */
"tpm" = tolist([])
"type" = "kvm"
"vcpu" = 2
"video" = tolist([])
"xml" = tolist([])
},
]
# https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs
# Create the service principal to use for the terraform operations
# https://learn.microsoft.com/en-us/cli/azure/create-an-azure-service-principal-azure-cli
# First create a csgo resource group, via the web interface
# For the rolle look at the 'Access control(IAM)' entry under the resource group web page.
# Contributor - Grants full access to manage all resources, but does not allow you to assign roles in Azure RBAC, manage assignments in Azure Blueprints, or share image galleries.
# subscriptionID=$(az account show --query id -o tsv)
# mkdir ~/.variouos_credentials && chmod 700 ~/.variouos_credentials
# az ad sp create-for-rbac --name terraform_op --role Contributor --scopes /subscriptions/$subscriptionID/resourceGroups/$resourceGroup > ~/.variouos_credentials/azure.$resourceGroup.terraform_op
# add the following to your private apply script, do not check this in
# export ARM_SUBSCRIPTION_ID="<azure_subscription_id>"
# export ARM_TENANT_ID="<azure_subscription_tenant_id>"
# export ARM_CLIENT_ID="<service_principal_appid>"
# export ARM_CLIENT_SECRET="<service_principal_password>"
variable "csgo_client_access_password" {
description = "Password clients use to connect to the server"
default = "VerySecret"
}
var.csgo_client_access_password
terraform apply -var csgo_client_access_password="WelcomeToMyServer"
- main.tf - the configuration for the resources that you want to create.
- network.tf - common network definitions.
- outputs.tf - what you want output on after the apply.
- providers.tf - definition of the provider. What terraform provider to use.
- private_azure_variables.tf - private information, not to be included in the source control.
- e.g. tokens etc.
- README.md
- server-init-actions.yaml - cloud init file, one per server type.
- server.tf - server definition, one per server type.
- variables.tf - values that are likely to be changed.
Cloud-init commands are run when a VM boots for the first time.
Them commands and files are packed onto an ISO file and that ISO file is mounted by the VM at first boot. Then a std command is called on the ISO and the operations are then executed.
You need the following
#cloud-config
# Add groups to the system
groups:
- hashicorp
# Add users to the system. Users are added after groups are added.
users:
- default
- name: ansible
gecos: ansible
shell: /bin/bash
primary_group: hashicorp
sudo: ALL=(ALL) NOPASSWD:ALL
groups: users, admin
lock_passwd: false
ssh_authorized_keys:
- ${ansible_ssh_public_key}
# install packages
packages:
- ansible
- git
# create format a second disk
disk_setup:
/dev/vdb:
table_type: 'gpt'
layout: True
overwrite: false
fs_setup:
- label: data
device: /dev/vdb1
filesystem: ext4
overwrite: false
mounts:
- [/dev/vdb1, /data, ext4, 'defaults,discard,nofail', '0', '2']
# commands to run at the end
runcmd:
- cd /home/ansible; git clone https://github.com/XXX/ansible_instructions
- cd ansible_instructions && ansible-playbook -v some_playbook.yaml
data "template_file" "user_data" {
template = file("./cloud-init-actions.yaml")
# left hand var names are the var names used in the cloud-init yaml.
vars = {
ansible_ssh_public_key = file(var.ansible_ssh_public_key_filename)
csgo_client_access_password = var.csgo_client_access_password
csgo_server_rcon_password = var.csgo_server_rcon_password
one_for_local_zero_for_global = var.one_for_local_zero_for_global
server_name = var.server_name
steam_server_token = var.steam_server_token
}
}
resource "libvirt_cloudinit_disk" "commoninit" {
name = "csgo_commoninit.iso"
user_data = data.template_file.user_data.rendered
}
resource "libvirt_domain" "csgo-vm" {
...
cloudinit = libvirt_cloudinit_disk.commoninit.id
...
}
- Dynamic Cloud-Init Content with Terraform File Templates
- Terraform : passing variables from resource to cloudinit data block
data "template_file" "user_data" {
template = file("./cloud-init-actions.yaml")
# left hand var names are the var names used in the cloud-init yaml.
vars = {
ansible_ssh_public_key = file(var.ansible_ssh_public_key_filename)
csgo_client_access_password = var.csgo_client_access_password
csgo_server_rcon_password = var.csgo_server_rcon_password
one_for_local_zero_for_global = var.one_for_local_zero_for_global
server_name = var.server_name
steam_server_token = var.steam_server_token
}
}
ansible-playbook --extra-vars "ansible_var_namecsgo_client_access_password=${cloud_init_var_name_csgo_client_access_password}"
copy a local file to the VM being deployed.
- cloud-init part
# provide the CNI deployment file.
# https://cloudinit.readthedocs.io/en/latest/reference/modules.html#write-files
write_files:
- path: /home/ansible/k8s_node_ansible_playbook.yaml
content: !!binary |
${ansible_playbook_file_base64}
permissions: '0640'
-
ansible_playbook_file_base64 is defined in the .tf file.
worker.tf: ansible_playbook_file_base64 = filebase64("../../ansible_playbook/k8s_node_ansible_playbook.yaml")
- in the template_file section
data "template_file" "worker_user_data" {
-
terraform part
data "template_file" "control_plane_user_data" {
template = file("./cloud-init-actions.yaml")
# left hand var names are the var names used in the cloud-init yaml.
vars = {
ansible_ssh_public_key = file(var.ansible_ssh_public_key_filename)
node_type = "control_plane"
cni_installation_file_base64 = filebase64("weave-daemonset-k8s.yaml")
}
}
disk_setup:
/dev/vdb:
table_type: 'gpt'
layout: True
overwrite: false
fs_setup:
- label: data
device: /dev/vdb1
filesystem: ext4
overwrite: false
mounts:
- [/dev/vdb1, /data, ext4, 'defaults,discard,nofail', '0', '2']
See. Connection to libvirt via SSH fails with provider version 0.6.9
- Can you ssh to the server without challenge?
- Probably not, you need to be able to e.g.
ssh [email protected]
- Probably not, you need to be able to e.g.
- Deleteing id_rsa and ~/.ssh/known_hosts fixed the issue
- not sure if deleting known_hosts would have been enough.
│ Error: failed to dial libvirt: ssh: handshake failed: knownhosts: key mismatch
│
│ with provider["registry.terraform.io/dmacvicar/libvirt"],
│ on main.tf line 16, in provider "libvirt":
│ 16: provider "libvirt" {
│
virsh list --all virsh undefine cloudstack_vm
libvirt_volume.os-qcow2: Creation complete after 1m13s [id=/var/lib/libvirt/images/tf_cloudstack.qcow2]
libvirt_cloudinit_disk.commoninit: Creation complete after 1m13s [id=/var/lib/libvirt/images/cloudstack_commoninit.iso;e5958a5a-8adc-4584-92f7-f7a8a617bb68]
libvirt_domain.cloudstack-vm: Creating...
╷
│ Error: error defining libvirt domain: operation failed: domain 'cloudstack_vm' already exists with uuid b63b2a6f-2dc0-493f-af8d-5a5a7561a97a
│
│ with libvirt_domain.cloudstack-vm,
│ on main.tf line 52, in resource "libvirt_domain" "cloudstack-vm":
│ 52: resource "libvirt_domain" "cloudstack-vm" {
│
Solution: How to fix Terraform libvirt permission denied on /var/lib/libvirt/images
- sudo vi /etc/apparmor.d/libvirt/TEMPLATE.qemu
- sudo systemctl restart libvirtd
│ with libvirt_domain.terraform-test-vm,
│ on template.tf line 31, in resource "libvirt_domain" "terraform-test-vm":
│ 31: resource "libvirt_domain" "terraform-test-vm" {
#
# This profile is for the domain whose UUID matches this file.
#
#include <tunables/global>
profile LIBVIRT_TEMPLATE flags=(attach_disconnected) {
#include <abstractions/libvirt-qemu>
/hdd2/vmstoragepool/**.qcow2 rwk,
}
original
#
# This profile is for the domain whose UUID matches this file.
#
#include <tunables/global>
profile LIBVIRT_TEMPLATE flags=(attach_disconnected) {
#include <abstractions/libvirt-qemu>
}
Error: error creating libvirt domain: internal error: process exited while connecting to monitor: 2023-09-07T08:39:13.291493Z qemu-system-x86_64: -blockdev {"driver":"file","filename":"/var/lib/libvirt/images/tf_cloudstack.qcow2","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}: Could not open '/var/lib/libvirt/images/tf_cloudstack.qcow2': Permission denied
libvirt_domain.gitlab-os-vm: Creating...
╷
│ Error: error creating libvirt domain: internal error: process exited while connecting to monitor: 2023-11-07T08:34:22.172477Z qemu-system-x86_64: -blockdev {"driver":"file","filename":"/hdd2/vmstoragepool/gitlab.qcow2","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}: Could not open '/hdd2/vmstoragepool/gitlab.qcow2': Permission denied
│
│ with libvirt_domain.gitlab-os-vm,
│ on gitlab.tf line 14, in resource "libvirt_domain" "gitlab-os-vm":
│ 14: resource "libvirt_domain" "gitlab-os-vm" {
│
╵```
#### Error: error retrieving volume for disk: Storage volume not found: no storage vol with matching name 'csgo.qcow2'
See: [Error retrieving volume for disk](https://github.com/dmacvicar/terraform-provider-libvirt/issues/949)
* rm terraform.tfstate terraform.tfstate.backup
#### Error: error creating libvirt domain: internal error: process exited while connecting to monitor Could not open qcow2 Permission denied
[](https://github.com/dmacvicar/terraform-provider-libvirt/issues/715)
[](https://computingforgeeks.com/use-virt-manager-as-non-root-user/)
sudo usermod -a -G libvirt-qemu cadm
```text
│ Error: error creating libvirt domain: internal error: process exited while connecting to monitor: 2023-09-11T17:22:29.489038Z qemu-system-x86_64: -blockdev {"driver":"file","filename":"/var/lib/libvirt/images/lfs_worker_0.qcow2","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}: Could not open '/var/lib/libvirt/images/lfs_worker_0.qcow2': Permission denied
libvirt_volume.os-qcow2: Creation complete after 56s [id=/hdd2/vmstoragepool/tf_cloudstack.qcow2]
libvirt_domain.cloudstack-vm: Creating...
╷
│ Error: error creating libvirt domain: internal error: process exited while connecting to monitor: 2023-09-12T14:24:30.727925Z qemu-system-x86_64: -blockdev {"driver":"file","filename":"/hdd2/vmstoragepool/tf_cloudstack.qcow2","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}: Could not open '/hdd2/vmstoragepool/tf_cloudstack.qcow2': Permission denied
│
│ with libvirt_domain.cloudstack-vm,
│ on main.tf line 33, in resource "libvirt_domain" "cloudstack-vm":
│ 33: resource "libvirt_domain" "cloudstack-vm" {
│
╵
Error: building account: could not acquire access token to parse claims: clientCredentialsToken: failed to build request
Planning failed. Terraform encountered an error while generating this plan.
╷
│ Error: building account: could not acquire access token to parse claims: clientCredentialsToken: failed to build request: parse "https://login.microsoftonline.com/XXXXXXXXXXX\n/oauth2/v2.0/token": net/url: invalid control character in URL
│
│ with provider["registry.terraform.io/hashicorp/azurerm"],
│ on main.tf line 32, in provider "azurerm":
│ 32: provider "azurerm" {
TODO what is "register Azure Resource Providers." see: Azure Provider Note
??? Is it this thing of storring the state in azure cloud?
Add
provider "azurerm" {
skip_provider_registration = true # This is only required when the User, Service Principal, or Identity running Terraform lacks the permissions to register Azure Resource Providers.
}
│ Error: Error ensuring Resource Providers are registered.
│
│ Terraform automatically attempts to register the Resource Providers it supports to
│ ensure it's able to provision resources.
│
│ If you don't have permission to register Resource Providers you may wish to use the
│ "skip_provider_registration" flag in the Provider block to disable this functionality.
...
│ More information on the "skip_provider_registration" flag can be found here:
│ https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs#skip_provider_registration
Turns out the Basic_A0 was not available in that location
az vm list-skus --location northeurope --size Basic_ --all --output table
azurerm_network_interface_security_group_association.csgo_nsg_association: Creation complete after 1s [id=/subscriptions/XXX/resourceGroups/free_csgo_group/providers/Microsoft.Network/networkInterfaces/csgoNIC|/subscriptions/XXX/resourceGroups/free_csgo_group/providers/Microsoft.Network/networkSecurityGroups/csgo-nsg]
╷
│ Error: creating Linux Virtual Machine: (Name "csgoVM" / Resource Group "free_csgo_group"): compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=409 -- Original Error: Code="SkuNotAvailable" Message="The requested size for resource '/subscriptions/XXX/resourceGroups/free_csgo_group/providers/Microsoft.Compute/virtualMachines/csgoVM' is currently not available in location 'northeurope' zones '' for subscription 'XXX'. Please try another size or deploy to a different location or zones. See https://aka.ms/azureskunotavailable for details."
│
│ with azurerm_linux_virtual_machine.csgo_vm,
│ on main.tf line 107, in resource "azurerm_linux_virtual_machine" "csgo_vm":
│ 107: resource "azurerm_linux_virtual_machine" "csgo_vm" {
this works: destination_port_range = "27015-27017"
│ Error: Incorrect attribute value type
│
│ on main.tf line 119, in resource "azurerm_network_security_group" "csgo_nsg":
│ 119: destination_port_range = ["27015-27017"]
│
│ Inappropriate value for attribute "destination_port_range": string required.