kubernetes - deptno/deptno.github.io GitHub Wiki

kubernetes|์ฟ ๋ฒ„๋„คํ‹ฐ์Šค

  • TODO: sudo kubeadm certs check-expiration

pod

  • init container ์„ค์ •์„ ์œ„ํ•ด ์„ ์‹คํ–‰๋˜๊ณ  ์ข…๋ฃŒ๋˜๋Š” ์ปจํ…Œ์ด๋„ˆ

multi container pod ์˜ ๊ฒฝ์šฐ

  1. container ์ค‘ ํ•˜๋‚˜๋งŒ ์ฃฝ์–ด๋„ ํŒŒ๋“œ ๋‚ด์˜ ์ปจํ…Œ์ด๋„ˆ๋“ค์ด ์žฌ์‹œ์ž‘๋˜๋Š” ๋˜๋Š” ๊ฒƒ์ธ์ง€ 503 ์ด ๋œธ
  2. ์‚ฌ์‹ค ํŒŒ๋“œ ์ค‘ ํ•˜๋‚˜๋Š” ์ดˆ๊ธฐ ์„ค์ •์„ ์œ„ํ•œ ๊ฑฐ์˜€๋Š”๋ฐ ์ด๋ฅผ initContainer ๋กœ ๋งŒ๋“ค๊ณ  ๋‚˜๋‹ˆ ์—๋Ÿฌ๊ฐ€ ์•ˆ๋‚จ
  3. ์ •์ƒ ์ข…๋ฃŒ๋˜๋”๋ผ๋„ deployment ํŠน์„ฑ์ƒ ์‹คํ–‰์ค‘์ธ container๊ฐ€ ์œ ์ง€๋˜์–ด์•ผํ•ด์„œ ์˜ค๋ฅ˜๋กœ ๋ณด๊ณ  pod ์žฌ์‹œ์ž‘์ด ๋  ์ˆ˜ ์žˆ์„ ๊ฒƒ ๊ฐ™์Œ
  4. ์„ค์ • ํ›„ ์ข…๋ฃŒ๋˜๋Š” ํŒŒ๋“œ๋“ค์€ initContainer ๋กœ ์˜ฎ๊ธฐ์ž

cronjob

๋ถ„ ์‹œ ์ผ ์›” ์š”์ผ
*/2 * * * * ๋งค 2๋ถ„๋งˆ๋‹ค
1/2 * * * * ๋งค 2๋ถ„๋งˆ๋‹ค(1๋ถ„, 3๋ถ„...)
  • successfulJobsHistoryLimit: [number] ์˜ต์…˜์„ ํฌ๊ฒŒ ์„ค์ •ํ•˜๋ฉด worker node ์˜ cpu, mem ์— ์˜ํ–ฅ์„ ๋ฏธ์นœ๋‹ค
  • suspend: [boolean] ์Šค์ผ€์ค„์„ ์ž ์‹œ ๋ฉˆ์ถ”๋Š” ๊ฒƒ์œผ๋กœ ๋ณด์ด๋Š”๋ฐ ๋””ํ”Œ๋กœ์ด์—ํ• ๋•Œ ์œ ์šฉํ•ด ๋ณด์ž„

storage

StorageSlass ์ถ”๊ฐ€ ์—†์ด nfs mount ๊ฐ€ ๊ฐ€๋Šฅ

pv

access mode

AccessMode ์•ฝ์–ด node:pvc pvc:pod ๋น„๊ณ 
ReadWriteOnce RWO 1:1 1:n
ReadOnlyMany ROX n:1 1:n
ReadWriteMany RWX n:1 1:n eg. nfs
ReadWriteOncePod RWOP 1:1 1:1 1.22+
  • ์•ฝ์–ด๋Š” cli ์—์„œ ์‚ฌ์šฉ

pvc

  • pv:pvc ๋Š” 1:1 ๊ด€๊ณ„
  • pvc ๋Š” namespace ์˜ค๋ธŒ์ ํŠธ์ž„์œผ๋กœ pv ์˜ access mode ์™€ ๋ฌด๊ด€ํ•˜๊ฒŒ namespace ์— ์ข…์†
  • ReadWriteOnce ๋Š” node ๊ธฐ์ค€์œผ๋กœ pod ๋Š” ํ•˜๋‚˜์˜ pvc์— ์—ฌ๋Ÿฟ ๋ถ™์„ ์ˆ˜ ์žˆ์Œ
    • pvc - pod 1:n ์ด ๊ฐ€๋Šฅ
    • pv - pvc ๋Š” 1:1 ReadWriteOnce ์ธ๊ฒฝ์šฐ
    • local - pv ๋Š” n:1 ์ด ๊ฐ€๋Šฅ

error

$ kdel pvc [pvc]
persistentvolumeclaim "[pvc]" deleted
^C # ๋ฉˆ์ถฐ์„œ ๊ฐ•์ œ ์ข…๋ฃŒ

$ k get volumeattachments.storage.k8s.io
No resources found # ์‚ฌ์šฉ ์ฃผ์ฒด๊ฐ€ ์—†๋‹ค

$ pvc get
$ k get pvc
NAME                         STATUS        VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       AGE
[pvc]                        Terminating   pvc-9d3a82a2-49bd-45e7-99b7-e68eb8b76f34   20M        RWO            openebs-hostpath   30h

์‚ญ์ œ๋˜์ง€ ์•Š๊ณ  Terminating ์ƒํƒœ์—์„œ ๋ฉˆ์ถ˜๋‹ค

kubectl patch pvc {PVC_NAME} -p '{"metadata":{"finalizers":null}}'

์œ„ ๋ฐฉ์‹์œผ๋กœ ์‚ญ์ œ๊ฐ€ ๊ฐ€๋Šฅ

  • openebs ์˜ hostpath ์ธ ๊ฒฝ์šฐ ๋ฐ์ดํ„ฐ๋Š” ์‚ด์•„ ๋‚จ์œผ๋‹ˆ ์ฐธ๊ณ 

์ด๋•Œ pv ๋Š” Released ์ƒํƒœ๊ฐ€ ๋˜๋ฉด ๋‹ค๋ฅธ pvc ์™€๋Š” ๋ฐ”์ธ๋”ฉํ•  ์ˆ˜ ์—†๋‹ค. pv ์˜ ์ƒํƒœ๋ฅผ Available ๋กœ ๋ฐ”๊พธ๊ธฐ ์œ„ํ•ด์„œ๋Š” pv ๋ฅผ ์ˆ˜์ •ํ•˜์—ฌ claimRef ๋ฅผ ์ œ๊ฑฐํ•ด์•ผํ•œ๋‹ค

kubectl patch pv [ PV_NAME ] -p '{"spec":{"claimRef": null}}'

  Warning  FailedMount       24s (x7 over 56s)  kubelet            MountVolume.NewMounter initialization failed for volume "pv-static" : path "/var/openebs/local/some-directory" does not exist

ํ•ด๋‹น ๋””๋ ‰ํ† ๋ฆฌ๋Š” ์ˆ˜๋™์œผ๋กœ ๋งŒ๋“ค์–ด์ค€๋‹ค

mount: /var/lib/kubelet/pods/91f95da8-3cea-4f8a-a367-c2b11b3444b5/volumes/kubernetes.io~nfs/test-volume: bad option; for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount.<type> helper program.                                                                                                                                 โ”‚
  • worker node ์— nfs-common ์„ค์น˜
mount.nfs: failed to apply fstab options
  • ๊ถŒํ•œ ์ด์Šˆ

0/1 nodes are available

  Warning  FailedScheduling  29s   default-scheduler  0/1 nodes are available: 1 Too many pods. preemption: 0/1 nodes are available: 1 No
 preemption victims found for incoming pod..

๋ฌดํ•œ๋ฃจํ”„์— ์˜ํ•ด์„œ cronjob ์˜ job ํŒŒ๋“œ๊ฐ€ ์ง€์†์ ์œผ๋กœ ์Œ“์ด๋Š” ๋ฌธ์ œ๊ฐ€ ์žˆ์—ˆ๋Š”๋ฐ ํŒŒ๋“œ๊ฐ€ 160๊ฐœ ์ •๋„๊ฐ€๋˜๋‹ˆ ๋œจ์งˆ ๋ชปํ•œ๋‹ค

  • TODO: ๋…ธ๋“œ๋‹น ํŒŒ๋“œ ๊ฐฏ์ˆ˜์— ๋Œ€ํ•ด์„œ ์กฐ์‚ฌ

disk-pressure

์šฉ๋Ÿ‰ ๋ถ€์กฑ์ด๋‹ค. 2TB ๋””์Šคํฌ๋ฅผ ์‚ฌ์šฉ ์ค‘์ด์—ˆ๋Š”๋ฐ ๋‚˜์ค‘์— ํ™•์ธํ•ด ๋ณด๋‹ˆ ์šฐ๋ถ„ํˆฌ ์„ค์น˜์‹œ์— 100G ํŒŒํ‹ฐ์…˜์„ ์‚ฌ์šฉํ•˜๊ณ  ์žˆ์—ˆ๋‹ค. ๊ทธ๋งˆ์ €๋„ /var/openebs/local, /var/lib/kubelet/pods/... ์—๋„ ๋ฐ์ดํ„ฐ๊ฐ€ ์ค‘์ฒฉ์œผ๋กœ ์Œ“์ด๋ฉด์„œ 50G ๋งŒ์— ๋ฐœ์ƒํ–ˆ๋‹ค.

secret

echo ๋ฅผ ์‚ฌ์šฉํ•˜๋ฉด newline \n ์ด ๋ถ™๊ฒŒ๋œ๋‹ค.

  • echo -n ์„ ์‚ฌ์šฉ
  • - tr -d '\n'
# ํŒŒ์ผ๋กœ ๋ถ€ํ„ฐ ๊ฐ’ ์ƒ์„ฑ
kubectl create secret generic [name] --from-file=[key]=[filename] --from-file=[key]=[filename]
# ์ง์ ‘ ์ž…๋ ฅํ•œ ๊ฐ’์œผ๋กœ ๋ถ€ํ„ฐ ์ƒ์„ฑ
kubectl create secret generic [name] --from-literal=[key]=[base64 encoded value]
# ๊ธฐ์กด ์‹œํฌ๋ฆฟ์— ์ถ”๊ฐ€ ํ˜น์€ ๊ฐ’ ๋ณ€๊ฒฝ
kubectl patch secret [name] -p '{"data": {["key"]: "[based encoded value]"}}' 
# ์‹œํฌ๋ฆฟ ๋„ค์ž„์ŠคํŽ˜์ด์Šค ๋ณต์‚ฌ
kubectl get secret -n [ namespace ] [ secret name ] | kubectl neat | sed "s/namespace: .*/namespace: [ target namespace]/" | kubectl apply -f -

authentication

์œ ์ €์ถ”๊ฐ€

์šฉ์–ด์ •๋ฆฌ

์šฉ์–ด ์„ค๋ช…
CSR Certificate Signing Request

CertificateSigningRequest ์„ ํ†ตํ•œ ์ ‘๊ทผ

  • ์ฃผ์˜
    • ์ถ”ํ›„ RoleBinding ๊ณผ ์ด์–ด์ง€๋Š” ๋ถ€๋ถ„์€ yaml ์•ˆ์—์„œ์˜ ๋‚ด์šฉ๊ณผ๋Š” ์ƒ๊ด€์—†์ด ์ธ์ฆ์„œ ์ƒ์„ฑ์‹œ์— ์ง€์ •ํ•˜๋Š” CN=User / O=Group ์™€ ๊ด€๊ณ„๋œ๋‹ค
  • CertificateSigningRequest ์ƒ์„ฑ
$ openssl genrsa -out user.key 2048
$ openssl req -new -key user.key -out user.csr
$ cat <<EOF | kubectl apply -f -
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: user
spec:
  request: $(cat user.csr | base64)
  signerName: kubernetes.io/kube-apiserver-client
  expirationSeconds: 3600
  usages:
  - client auth
EOF

certificatesigningrequest.certificates.k8s.io/user created

template ํ™” ๊ด€๋ จํ•ด์„œ๋Š” envsubst ์ฐธ์กฐ

  • CertificateSigningRequest approve
$ k get csr
NAME     AGE   SIGNERNAME                            REQUESTOR          REQUESTEDDURATION   CONDITION
user   34s   kubernetes.io/kube-apiserver-client   kubernetes-admin   60m                 Pending

$ kubectl get csr user -o jsonpath='{.status.certificate}'| base64 -d > user.crt
$ kubectl config set-credentials user --client-key=user.key --client-certificate=user.crt --embed-certs
User "user" set.

$ kubectl config set-context user --cluster=local --user=user
Context "user" created.

$ kubectl config use-context user
Switched to context "user".

$ kgp
Error from server (Forbidden): pods is forbidden: User "user.dev" cannot list resource "pods" in API group "" in the namespace "default"

๊ถŒํ•œ์ด ์—†์–ด์„œ ๋˜์ง€ ์•Š๋Š”๋‹ค ๋‹ค์‹œ ์›๋ž˜์˜ context(๊ถŒํ•œ์ด ์žˆ๋Š”) ๋Œ์•„์™€์„œ role, rolebinding ์„ ์ƒ์„ฑํ•œ๋‹ค

  • role, rolebinding ์ƒ์„ฑ
$ kubectl create role test --verb=get --verb=list --resource=pods
role.rbac.authorization.k8s.io/test created

$ kubectl create rolebinding test-user --role=test --user=user
rolebinding.rbac.authorization.k8s.io/test-user created
  • ๋‹ค์‹œ user ๋กœ ๊ถŒํ•œ์„ ํ™•์ธํ•œ๋‹ค
$ k config use-context user
Switched to context "user".

$ kgp
Error from server (Forbidden): pods is forbidden: User "user.dev" cannot list resource "pods" in API group "" in the namespace "default"

$ kgp -n test
Error from server (Forbidden): pods is forbidden: User "user.dev" cannot list resource "pods" in API group "" in the namespace "test"

$ k auth can-i list pods
no

$ k auth can-i list pods -n test
no
$ k api-resources head -1 ; k api-resources | grep pod
NAME                              SHORTNAMES                                      APIVERSION                             NAMESPACED   KIND
pods                              po                                              v1                                     true         Pod

๋ถ€์กฑํ•œ ๊ฒƒ๋“ค์„ role, rolebinding ์— ๋„ฃ๋„๋ก ํ•˜์ž

  • role ์— apiGroups ์— v1 ์ถ”๊ฐ€
  • rolebinding ์— namespace ์ถ”๊ฐ€
$ k use-context user
error: unknown command "use-context" for "kubectl"
$ kgp
NAME                                    READY   STATUS      RESTARTS      AGE
curl                                    1/1     Running     5 (10d ago)   23d

CertificateSigningRequest ํ•œ์‹œ๊ฐ„์ด ์ง€๋‚˜๋ฉด ํ† ํฐ์ด ๋งŒ๋ฃŒ๋˜์–ด ๋กœ๊ทธ์ธ์ด ๋˜์ง€ ์•Š๋Š”๋‹ค. approve ์‹œ์ ์ด ์•„๋‹ˆ๋ผ csr ์„ ์ƒ์„ฑํ•œ ํƒ€์ž„์œผ๋กœ๋ถ€ํ„ฐ ํ•œ์‹œ๊ฐ„์ด๋‹ค, ์ƒ์„ฑ์‹œ ๋„ฃ์–ด์คฌ๋˜ ์•„๋ž˜ prop์— ์˜ํ•ด์„œ ์ •์˜๋œ๋‹ค

  expirationSeconds: 3600
$ kgp
error: You must be logged in to the server (Unauthorized)

ServiceAccount ๋ฅผ ํ†ตํ•œ ์ ‘๊ทผ

error

PEM block type must be CERTIFICATE REQUEST

CertificateSigningRequest ์ƒ์„ฑ์‹œ์— ๋ฐœ์ƒํ•˜๋Š”๋ฐ cat user.csr | base ํ•œ ๊ฐ’์ด ๋“ค์–ด๊ฐ€์•ผํ•œ๋‹ค

setup

local

๋กœ์ปฌ์—์„œ๋Š” minikube ๋ฅผ ์‚ฌ์šฉํ•œ๋‹ค. ๋ฏธ๋‹ˆ์ฟ ๋ฒ ์—์„œ pv ๋Š” local ์€ ์ง€์›๋˜์ง€ ์•Š์œผ๋ฉฐ hostPath ๋งŒ ์ง€์›ํ•œ๋‹ค.

multipass ๋ฅผ ์‚ฌ์šฉํ•˜๋ฉด vm์„ ์ด์šฉํ•˜์—ฌ ์‹ค์ œ์™€ ๊ฐ™์€ ํด๋Ÿฌ์Šคํ„ฐ ๊ตฌํ˜„์ด ๊ฐ€๋Šฅํ•˜๋‹ค.

error

annotate

error: --overwrite is false but found the following declared annotation(s):

kubectl annotate ์‹œ์— --overwrite ์˜ต์…˜์„ ์ถ”๊ฐ€ํ•˜๊ฑฐ๋‚˜ --force๋ฅผ ์ถ”๊ฐ€ํ•ด์„œ ํ•ด๊ฒฐํ•œ๋‹ค

link

โš ๏ธ **GitHub.com Fallback** โš ๏ธ