tidb on aks terraform - luyomo/OhMyTiUP GitHub Wiki

ENV

  • Create APP Registration Please refer to APP Registration Creation to create the application registration for terrafrom user. 
    $ az login
$ az account set --subscription="20000000-0000-0000-0000-000000000000"
$ az ad sp create-for-rbac --name sc4tcdemo  --role="Contributor" --scopes="/subscriptions/20000000-0000-0000-0000-000000000000"
Creating 'Contributor' role assignment under scope '/subscriptions/20000000-0000-0000-0000-000000000000'
The output includes credentials that you must protect. Be sure that you do not include these credentials in your code or check the credentials into your source control. For more information, see https://aka.ms/azadsp-cli
{
  "appId": "11111111-1111-1111-1111-11111111",
  "displayName": "sc4tcdemo",
  "password": "333333333333333333333333333.333333333333",
  "tenant": "44444444-4444-4444-4444-44444444"
}
  • Login using service principal 
    az login --service-principal -u 11111111-1111-1111-1111-11111111 -p 333333333333333333333333333.333333333333 --tenant 44444444-4444-4444-4444-44444444
  • Set Terraform login info 
    export ARM_CLIENT_ID=11111111-1111-1111-1111-11111111
export ARM_TENANT_ID=44444444-4444-4444-4444-44444444
export ARM_CLIENT_SECRET=333333333333333333333333333.333333333333
export ARM_SUBSCRIPTION_ID=20000000-0000-0000-0000-000000000000

Install

TiDB Deployment on AKS

  • Download terraform source code 
    $ git clone [email protected]:luyomo/terraform-tidb-on-aks.git
$ cd example/tidb-on-aks
tidb-on-aks:example/tidb-on-aks$ export TF_VAR_register_app_client_id=76f7b8f5-208e-4a71-xxxx-xxxxxxxxxxxx
tidb-on-aks:example/tidb-on-aks$ export TF_VAR_register_app_client_secret=8f7d9027-200e-xxxx-xxxx-xxxxxxxxxxxx
  • Terraform deployment 
    tidb-on-aks:example/tidb-on-aks$ terraform init
Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
tidb-on-aks:example/tidb-on-aks$ terraform plan
tidb-on-aks:example/tidb-on-aks$ terraform apply
  • Fetch kubeconfig to local 
    tidb-on-aks:example/tidb-on-aks$ az aks get-credentials --resource-group jp-presale-test --name tidb-on-aks-demo --admin
Merged "tidb-on-aks-demo-admin" as current context in /home/tidb-on-aks/.kube/config
  • List K8S resources 
    tidb-on-aks:example/tidb-on-aks$ kubectl get ns
NAME              STATUS   AGE
default           Active   33m
kube-node-lease   Active   33m
kube-public       Active   33m
kube-system       Active   33m
tidb-admin        Active   31m
tidb-cluster      Active   31m
tidb-on-aks:example/tidb-on-aks$ more sc.yaml 
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
    kubernetes.io/cluster-service: "true"
  name: tidb-managed-csi
# https://learn.microsoft.com/en-us/rest/api/storagerp/srp_sku_types
parameters:
  skuname: StandardSSD_LRS
provisioner: disk.csi.azure.com
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
mountOptions:
  - nodelalloc
  - noatime

tidb-on-aks:example/tidb-on-aks$ kubectl apply -f conf/sc.yaml 
storageclass.storage.k8s.io/tidb-managed-csi created
tidb-on-aks:example/tidb-on-aks$ kubectl get sc tidb-managed-csi 
NAME               PROVISIONER          RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
tidb-managed-csi   disk.csi.azure.com   Retain          WaitForFirstConsumer   true                   9m20s

tidb-on-aks:example/tidb-on-aks$ diff conf/tidb-cluster.yaml <(curl https://raw.githubusercontent.com/pingcap/tidb-operator/v1.6.0/examples/aks/tidb-cluster.yaml)
4c4
<   name: tidb-on-aks-demo
---
>   name: basic
20d19
<     storageClassName: tidb-managed-csi
30c29
<       dedicated: tidb-on-aks-demo-pd
---
>       dedicated: pd
35c34
<       value: tidb-on-aks-demo-pd
---
>       value: pd
44c43
<               - tidb-on-aks-demo-pd
---
>               - pd
50d48
<     storageClassName: tidb-managed-csi
55c53
<       dedicated: tidb-on-aks-demo-tikv
---
>       dedicated: tikv
60c58
<       value: tidb-on-aks-demo-tikv
---
>       value: tikv
69c67
<               - tidb-on-aks-demo-tikv
---
>               - tikv
76,77c74,75
< #      annotations:
< #        service.beta.kubernetes.io/azure-load-balancer-internal: "true"
---
>       annotations:
>         service.beta.kubernetes.io/azure-load-balancer-internal: "true"
96c94
<       dedicated: tidb-on-aks-demo-tidb
---
>       dedicated: tidb
101c99
<       value: tidb-on-aks-demo-tidb
---
>       value: tidb
110c108
<               - tidb-on-aks-demo-tidb
---
>               - tidb


tidb-on-aks:example/tidb-on-aks$ kubectl apply -f conf/tidb-cluster.yaml -n tidb-cluster
tidb-on-aks:example/tidb-on-aks$ kubectl get pv 
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                       STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
pvc-3cee5f9e-1bc2-43e2-a1c7-f9280c960553   10Gi       RWO            Retain           Bound    tidb-cluster/pd-tidb-on-aks-demo-pd-1       default        <unset>                          31m
pvc-8a8c1312-5490-4ba7-94fd-aa7591c32df5   100Gi      RWO            Retain           Bound    tidb-cluster/tikv-tidb-on-aks-demo-tikv-0   default        <unset>                          31m
pvc-8e26cf8b-69e6-4987-8c39-ed35d1e6b1d8   100Gi      RWO            Retain           Bound    tidb-cluster/tikv-tidb-on-aks-demo-tikv-1   default        <unset>                          31m
pvc-973ee7a8-c828-466b-b32d-a1f2c3d6ae33   10Gi       RWO            Retain           Bound    tidb-cluster/pd-tidb-on-aks-demo-pd-0       default        <unset>                          31m
pvc-a71b3975-1354-4087-ad8b-c8cfd8fc1808   10Gi       RWO            Retain           Bound    tidb-cluster/pd-tidb-on-aks-demo-pd-2       default        <unset>                          31m
pvc-c8f301da-965d-4ae9-8362-85e2283e1929   100Gi      RWO            Retain           Bound    tidb-cluster/tikv-tidb-on-aks-demo-tikv-2   default        <unset>                          31m

tidb-on-aks:example/tidb-on-aks$ kubectl get all -n tidb-cluster 
NAME                                              READY   STATUS    RESTARTS   AGE
pod/tidb-on-aks-demo-discovery-699848b5f8-hpxzm   1/1     Running   0          3m32s
pod/tidb-on-aks-demo-pd-0                         1/1     Running   0          3m32s
pod/tidb-on-aks-demo-pd-1                         1/1     Running   0          3m32s
pod/tidb-on-aks-demo-pd-2                         1/1     Running   0          3m32s
pod/tidb-on-aks-demo-tidb-0                       2/2     Running   0          2m26s
pod/tidb-on-aks-demo-tidb-1                       2/2     Running   0          2m26s
pod/tidb-on-aks-demo-tikv-0                       1/1     Running   0          2m56s
pod/tidb-on-aks-demo-tikv-1                       1/1     Running   0          2m56s
pod/tidb-on-aks-demo-tikv-2                       1/1     Running   0          2m56s

NAME                                 TYPE           CLUSTER-IP    EXTERNAL-IP   PORT(S)                          AGE
service/tidb-on-aks-demo-discovery   ClusterIP      10.0.29.204   <none>        10261/TCP,10262/TCP              3m32s
service/tidb-on-aks-demo-pd          ClusterIP      10.0.25.111   <none>        2379/TCP                         3m32s
service/tidb-on-aks-demo-pd-peer     ClusterIP      None          <none>        2380/TCP,2379/TCP                3m32s
service/tidb-on-aks-demo-tidb        LoadBalancer   10.0.53.63    48.217.7.4    4000:30489/TCP,10080:31506/TCP   2m19s
service/tidb-on-aks-demo-tidb-peer   ClusterIP      None          <none>        10080/TCP                        2m26s
service/tidb-on-aks-demo-tikv-peer   ClusterIP      None          <none>        20160/TCP                        2m56s

NAME                                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/tidb-on-aks-demo-discovery   1/1     1            1           3m32s

NAME                                                    DESIRED   CURRENT   READY   AGE
replicaset.apps/tidb-on-aks-demo-discovery-699848b5f8   1         1         1       3m32s

NAME                                     READY   AGE
statefulset.apps/tidb-on-aks-demo-pd     3/3     3m32s
statefulset.apps/tidb-on-aks-demo-tidb   2/2     2m26s
statefulset.apps/tidb-on-aks-demo-tikv   3/3     2m56s

DashBoard

Grafana

Backup & Restore

Upgrade

AKS Update by TiCDC

AKS Upgrade by PiTR

AKS rolling upgrade

AKS upgrade

Others

Scale out/in

Scale up/Down

TiKV Disk Online extension

Enterprise audit log

HTAP

Troubleshooting

Diag gather

⚠️ **GitHub.com Fallback** ⚠️