hellouz's Devlog

[K8S Deploy] RKE2 & Cluster API

안녕유지 — Mon, 23 Feb 2026 06:56:31 +0900

Cloudnet K8s Deploy 7주차 스터디를 진행하며 정리한 글입니다.

RKE2 소개

RKE2(Rancher Kubernetes Engine 2)는 SUSE가 제공하는 엔터프라이즈용 Kubernetes 배포판입니다. 기존 RKE를 보안 중심으로 재설계한 버전으로, CIS Benchmark를 기본적으로 고려해 구성되어 있습니다. Docker 대신 containerd를 사용하며, 컨트롤 플레인 구성 요소가 패키징되어 있어 설치와 운영이 비교적 단순합니다. 폐쇄망 환경도 지원해 금융·공공기관과 같은 보안 요구사항이 높은 환경에 적합합니다.

RKE2 실습

[root@k8s-node1 ~]# curl -sfL https://get.rke2.io --output install.sh
[root@k8s-node1 ~]# chmod +x install.sh
INSTALL_RKE2_CHANNEL=v1.33 ./install.sh
[INFO]  using stable RPM repositories
[INFO]  using 1.33 series from channel stable
Rancher RKE2 Common (v1.33)                                                                                                                                     62  B/s | 659  B     00:10
Rancher RKE2 Common (v1.33)                                                                                                                                    448  B/s | 2.4 kB     00:05
Importing GPG key 0xE257814A:
 Userid     : "Rancher (CI) <ci@rancher.com>"
 Fingerprint: C8CF F216 4551 26E9 B9C9 18BE 925E A29A E257 814A
 From       : https://rpm.rancher.io/public.key
Rancher RKE2 Common (v1.33)                                                                                                                                    172  B/s | 2.6 kB     00:15
Rancher RKE2 1.33 (v1.33)                                                                                                                                       59  B/s | 659  B     00:11
Rancher RKE2 1.33 (v1.33)                                                                                                                                      467  B/s | 2.4 kB     00:05
Importing GPG key 0xE257814A:
 Userid     : "Rancher (CI) <ci@rancher.com>"
 Fingerprint: C8CF F216 4551 26E9 B9C9 18BE 925E A29A E257 814A
 From       : https://rpm.rancher.io/public.key
Rancher RKE2 1.33 (v1.33)                                                                                                                                      382  B/s | 5.9 kB     00:15
Dependencies resolved.
===============================================================================================================================================================================================
 Package                                   Architecture                         Version                                          Repository                                               Size
===============================================================================================================================================================================================
Installing:
 rke2-server                               aarch64                              1.33.8~rke2r1-0.el9                              rancher-rke2-1.33-stable                                8.3 k
Installing dependencies:
 rke2-common                               aarch64                              1.33.8~rke2r1-0.el9                              rancher-rke2-1.33-stable                                 25 M
 rke2-selinux                              noarch                               0.22-1.el9                                       rancher-rke2-common-stable                               22 k

Transaction Summary
===============================================================================================================================================================================================
Install  3 Packages

Total download size: 25 M
Installed size: 113 M
Downloading Packages:
(1/3): rke2-server-1.33.8~rke2r1-0.el9.aarch64.rpm                                                                                                             1.6 kB/s | 8.3 kB     00:05
(2/3): rke2-selinux-0.22-1.el9.noarch.rpm                                                                                                                      4.1 kB/s |  22 kB     00:05
(3/3): rke2-common-1.33.8~rke2r1-0.el9.aarch64.rpm                                                                                                             3.8 MB/s |  25 MB     00:06
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                          3.8 MB/s |  25 MB     00:06
Rancher RKE2 Common (v1.33)                                                                                                                                    471  B/s | 2.4 kB     00:05
Importing GPG key 0xE257814A:
 Userid     : "Rancher (CI) <ci@rancher.com>"
 Fingerprint: C8CF F216 4551 26E9 B9C9 18BE 925E A29A E257 814A
 From       : https://rpm.rancher.io/public.key
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                       1/1
  Running scriptlet: rke2-selinux-0.22-1.el9.noarch                                                                                                                                        1/3
  Installing       : rke2-selinux-0.22-1.el9.noarch                                                                                                                                        1/3
  Running scriptlet: rke2-selinux-0.22-1.el9.noarch                                                                                                                                        1/3
  Installing       : rke2-common-1.33.8~rke2r1-0.el9.aarch64                                                                                                                               2/3
  Installing       : rke2-server-1.33.8~rke2r1-0.el9.aarch64                                                                                                                               3/3
  Running scriptlet: rke2-server-1.33.8~rke2r1-0.el9.aarch64                                                                                                                               3/3
  Running scriptlet: rke2-selinux-0.22-1.el9.noarch                                                                                                                                        3/3
  Running scriptlet: rke2-server-1.33.8~rke2r1-0.el9.aarch64                                                                                                                               3/3
  Verifying        : rke2-selinux-0.22-1.el9.noarch                                                                                                                                        1/3
  Verifying        : rke2-common-1.33.8~rke2r1-0.el9.aarch64                                                                                                                               2/3
  Verifying        : rke2-server-1.33.8~rke2r1-0.el9.aarch64                                                                                                                               3/3

Installed:
  rke2-common-1.33.8~rke2r1-0.el9.aarch64                           rke2-selinux-0.22-1.el9.noarch                           rke2-server-1.33.8~rke2r1-0.el9.aarch64

Complete!
[root@k8s-node1 ~]# rke2 --version
rke2 version v1.33.8+rke2r1 (eb75e3c1774cee5a584259d6fee77eb8cfa9b430)
go version go1.24.12 X:boringcrypto
[root@k8s-node1 ~]# dnf repolist
repo id                                                                                        repo name
appstream                                                                                      Rocky Linux 9 - AppStream
baseos                                                                                         Rocky Linux 9 - BaseOS
extras                                                                                         Rocky Linux 9 - Extras
rancher-rke2-1.33-stable                                                                       Rancher RKE2 1.33 (v1.33)
rancher-rke2-common-stable                                                                     Rancher RKE2 Common (v1.33)
[root@k8s-node1 ~]# tree /etc/yum.repos.d/
/etc/yum.repos.d/
├── rancher-rke2.repo
├── rocky-addons.repo
├── rocky-devel.repo
├── rocky-extras.repo
└── rocky.repo

0 directories, 5 files
[root@k8s-node1 ~]# cat /etc/yum.repos.d/rancher-rke2.repo
[rancher-rke2-common-stable]
name=Rancher RKE2 Common (v1.33)
baseurl=https://rpm.rancher.io/rke2/stable/common/centos/9/noarch
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://rpm.rancher.io/public.key
[rancher-rke2-1.33-stable]
name=Rancher RKE2 1.33 (v1.33)
baseurl=https://rpm.rancher.io/rke2/stable/1.33/centos/9/aarch64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://rpm.rancher.io/public.key

# 디렉터리 생성 확인
[root@k8s-node1 ~]# tree /etc/rancher/
/etc/rancher/
└── rke2

1 directory, 0 files

[root@k8s-node1 ~]# tree /var/lib/rancher/
/var/lib/rancher/
└── rke2
    ├── agent
    │   ├── containerd
    │   │   └── io.containerd.snapshotter.v1.overlayfs
    │   │       └── snapshots
    │   └── logs
    ├── data
    └── server

8 directories, 0 files

# RKE2 설정 : cni 플러그인(canal) 등
[root@k8s-node1 ~]# cat << EOF > /etc/rancher/rke2/config.yaml
write-kubeconfig-mode: "0644"

debug: true

cni: canal

bind-address: 192.168.10.11
advertise-address: 192.168.10.11
node-ip: 192.168.10.11

disable-cloud-controller: true

disable:
  - servicelb
  - rke2-coredns-autoscaler
  - rke2-ingress-nginx
  - rke2-snapshot-controller
  - rke2-snapshot-controller-crd
  - rke2-snapshot-validation-webhook
EOF

# canal cni 플러그인 helm chart values 파일 작성
# https://docs.rke2.io/networking/basic_network_options
[root@k8s-node1 ~]# mkdir -p /var/lib/rancher/rke2/server/manifests/
[root@k8s-node1 ~]# cat << EOF > /var/lib/rancher/rke2/server/manifests/rke2-canal-config.yaml
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-canal
  namespace: kube-system
spec:
  valuesContent: |-
    flannel:
      iface: "enp0s9"
EOF

# coredns 의 autoscaler 미설치를 위한 helm chart values 파일 작성
[root@k8s-node1 ~]# cat << EOF > /var/lib/rancher/rke2/server/manifests/rke2-coredns-config.yaml
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-coredns
  namespace: kube-system
spec:
  valuesContent: |-
    autoscaler:
      enabled: false
EOF


# 모니터링 : 신규 터미널창
Every 2.0s: pstree -a                                                                                                                                       k8s-node1: Mon Feb 23 06:01:54 2026

systemd no_timer_check --switched-root --system --deserialize 31
  |-NetworkManager --no-daemon
  |   `-2*[{NetworkManager}]
  |-VBoxService --pidfile /var/run/vboxadd-service.sh
  |   `-8*[{VBoxService}]
  |-agetty -o -p -- \\u --noclear - linux
  |-anacron -s
  |-atd -f
  |-auditd
  |   |-sedispatch
  |   `-2*[{auditd}]
  |-chronyd -F 2
  |-crond -n
  |-dbus-broker-lau --scope system --audit
  |   `-dbus-broker --log 4 --controller 9 --machine-id 3b2405e60e2a4236814b5ee5095ec74f --max-bytes 536870912 --max-fds 4096 --max-matches 131072 --audit
  |-gpg-agent --homedir /var/cache/dnf/rancher-rke2-common-stable-38dcbd8c1e621b96/pubring --use-standard-socket --daemon
  |   |-scdaemon --multi-server --homedir /var/cache/dnf/rancher-rke2-common-stable-38dcbd8c1e621b96/pubring
  |   |   `-{scdaemon}
  |   `-{gpg-agent}
  |-gpg-agent --homedir /var/cache/dnf/rancher-rke2-1.33-stable-ecd687a1e3012961/pubring --use-standard-socket --daemon
  
  
# RKE2 시작 : 2분 정도 소요 -> coredns 파드까지 정상화 대략 1~2분 추가 소요
[root@k8s-node1 ~]# systemctl enable --now rke2-server.service
Created symlink /etc/systemd/system/multi-user.target.wants/rke2-server.service → /usr/lib/systemd/system/rke2-server.service.

[root@k8s-node1 ~]# pstree -a | grep -v color | grep 'rke2$' -A5
  |-rke2
  |   |-containerd -c /var/lib/rancher/rke2/agent/etc/containerd/config.toml
  |   |   `-12*[{containerd}]
  |   |-kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s...
  |   |   `-16*[{kubelet}]
  |   `-11*[{rke2}]
  
[root@k8s-node1 ~]# pstree -a | grep -v color | grep 'containerd-shim ' -A2
  |-containerd-shim -namespace k8s.io -id88344110551ed4fde4bf09fd9c7
  |   |-etcd --config-file=/var/lib/rancher/rke2/server/db/etcd/config
  |   |   `-9*[{etcd}]
--
  |-containerd-shim -namespace k8s.io -idbb4b8065833472be21fe676308f
  |   |-kube-proxy --cluster-cidr=10.42.0.0/16 --conntrack-max-per-core=0 --conntrack-tcp-timeout-close-wait=0s...
  |   |   `-7*[{kube-proxy}]
--
  |-containerd-shim -namespace k8s.io -id1524b3e2df0bdd9e189009ae36f
  |   |-kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --advertise-address=192.168.10.11...
  |   |   `-11*[{kube-apiserver}]
--
  |-containerd-shim -namespace k8s.io -ide0b4bdbe434721b4a1f44f0349e
  |   |-kube-scheduler --permit-port-sharing=true ...
  |   |   `-9*[{kube-scheduler}]
--
  |-containerd-shim -namespace k8s.io -id6f3b2d3a4c6a211ff764bc472f1
  |   |-kube-controller --permit-port-sharing=true --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins--terminated-pod-gc-thres
  |   |   `-6*[{kube-controller}]
--
  |-containerd-shim -namespace k8s.io -id0e4c3909fbf2e6f18bb5b436134
  |   |-pause
  |   `-10*[{containerd-shim}]
  |-containerd-shim -namespace k8s.io -id68674ccda6bf02469a45f21cf6c
  |   |-pause
  |   `-10*[{containerd-shim}]


# 자격증명 파일 복사
[root@k8s-node1 ~]# ls -l /etc/rancher/rke2/rke2.yaml
-rw-r--r--. 1 root root 2973 Feb 23 06:02 /etc/rancher/rke2/rke2.yaml
[root@k8s-node1 ~]# cp /etc/rancher/rke2/rke2.yaml ~/.kube/config

# /etc/rancher 디렉터리 확인
[root@k8s-node1 ~]# tree /etc/rancher/
/etc/rancher/
├── node
│   └── password
└── rke2
    ├── config.yaml
    ├── rke2-pss.yaml
    └── rke2.yaml

2 directories, 4 files

# 바이너리 파일 확인
[root@k8s-node1 ~]# tree /var/lib/rancher/rke2/bin/
/var/lib/rancher/rke2/bin/
├── containerd
├── containerd-shim-runc-v2
├── crictl
├── ctr
├── kubectl
├── kubelet
└── runc

0 directories, 7 files

# PATH 안 건드리고 표준 위치로 바이너리 노출 설정 : 심볼릭 링크 방식 
[root@k8s-node1 ~]# ln -s /var/lib/rancher/rke2/bin/containerd /usr/local/bin/containerd
[root@k8s-node1 ~]# ln -s /var/lib/rancher/rke2/bin/kubectl /usr/local/bin/kubectl
[root@k8s-node1 ~]# ln -s /var/lib/rancher/rke2/bin/crictl /usr/local/bin/crictl
[root@k8s-node1 ~]# ln -s /var/lib/rancher/rke2/bin/runc /usr/local/bin/runc
[root@k8s-node1 ~]# ln -s /var/lib/rancher/rke2/bin/ctr /usr/local/bin/ctr
[root@k8s-node1 ~]# ln -s /var/lib/rancher/rke2/agent/etc/crictl.yaml /etc/crictl.yaml

[root@k8s-node1 ~]# runc --version
runc version 1.4.0
commit: v1.4.0-0-g8bd78a99
spec: 1.3.0
go: go1.24.11 X:boringcrypto
libseccomp: 2.5.4
[root@k8s-node1 ~]# containerd --version
containerd github.com/k3s-io/containerd v2.1.5-k3s1 e77c15f30e5162d6abab671b0d74ca2243e2916e
[root@k8s-node1 ~]# kubectl version
Client Version: v1.33.8+rke2r1
Kustomize Version: v5.6.0
Server Version: v1.33.8+rke2r1

# 편의성 설정
[root@k8s-node1 ~]# source <(kubectl completion bash)
alias k=kubectl
complete -F __start_kubectl k
echo 'source <(kubectl completion bash)' >> /etc/profile
echo 'alias k=kubectl' >> /etc/profile
echo 'complete -F __start_kubectl k' >> /etc/profile

[root@k8s-node1 ~]# kubectl cluster-info -v=6
I0223 06:09:46.771465   12296 loader.go:402] Config loaded from file:  /root/.kube/config
I0223 06:09:46.772287   12296 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I0223 06:09:46.772307   12296 envvar.go:172] "Feature gate default state" feature="InOrderInformers" enabled=true
I0223 06:09:46.772312   12296 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I0223 06:09:46.772315   12296 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I0223 06:09:46.772319   12296 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
I0223 06:09:46.776487   12296 round_trippers.go:632] "Response" verb="GET" url="https://192.168.10.11:6443/api?timeout=32s" status="200 OK" milliseconds=3
I0223 06:09:46.778659   12296 round_trippers.go:632] "Response" verb="GET" url="https://192.168.10.11:6443/apis?timeout=32s" status="200 OK" milliseconds=0
I0223 06:09:46.789710   12296 round_trippers.go:632] "Response" verb="GET" url="https://192.168.10.11:6443/api/v1/namespaces/kube-system/services?labelSelector=kubernetes.io%2Fcluster-service%3Dtrue" status="200 OK" milliseconds=2
Kubernetes control plane is running at https://192.168.10.11:6443
CoreDNS is running at https://192.168.10.11:6443/api/v1/namespaces/kube-system/services/rke2-coredns-rke2-coredns:udp-53/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

# 노드, 파드 정보 확인
[root@k8s-node1 ~]# kubectl get node -owide
NAME        STATUS   ROLES                       AGE     VERSION          INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                      KERNEL-VERSION                  CONTAINER-RUNTIME
k8s-node1   Ready    control-plane,etcd,master   4m13s   v1.33.8+rke2r1   192.168.10.11   <none>        Rocky Linux 9.6 (Blue Onyx)   5.14.0-570.52.1.el9_6.aarch64   containerd://2.1.5-k3s1
[root@k8s-node1 ~]# helm list -A
NAME               	NAMESPACE  	REVISION	UPDATED                                	STATUS  	CHART                             	APP VERSION
rke2-canal         	kube-system	1       	2026-02-22 21:07:14.396735594 +0000 UTC	deployed	rke2-canal-v3.31.3-build2026020600	v3.31.3
rke2-coredns       	kube-system	1       	2026-02-22 21:07:14.422743523 +0000 UTC	deployed	rke2-coredns-1.45.201             	1.13.1
rke2-metrics-server	kube-system	1       	2026-02-22 21:08:06.964477155 +0000 UTC	deployed	rke2-metrics-server-3.13.007      	0.8.0
rke2-runtimeclasses	kube-system	1       	2026-02-22 21:08:08.967529212 +0000 UTC	deployed	rke2-runtimeclasses-0.1.000       	0.1.0
[root@k8s-node1 ~]# kubectl get pod -A
NAMESPACE     NAME                                         READY   STATUS      RESTARTS   AGE
kube-system   etcd-k8s-node1                               1/1     Running     0          3m34s
kube-system   helm-install-rke2-canal-cn9bh                0/1     Completed   0          4m9s
kube-system   helm-install-rke2-coredns-t87xt              0/1     Completed   0          4m9s
kube-system   helm-install-rke2-metrics-server-wpjzt       0/1     Completed   0          4m9s
kube-system   helm-install-rke2-runtimeclasses-5pn9f       0/1     Completed   0          4m9s
kube-system   kube-apiserver-k8s-node1                     1/1     Running     0          3m34s
kube-system   kube-controller-manager-k8s-node1            1/1     Running     0          3m34s
kube-system   kube-proxy-k8s-node1                         1/1     Running     0          3m34s
kube-system   kube-scheduler-k8s-node1                     1/1     Running     0          3m34s
kube-system   rke2-canal-9wqbn                             2/2     Running     0          2m54s
kube-system   rke2-coredns-rke2-coredns-559595db99-lv2ww   1/1     Running     0          2m55s
kube-system   rke2-metrics-server-fdcdf575d-vxcnw          1/1     Running     0          2m2s

/var/lib/rancher/rke2 디렉터리 : addon helm chart, 인증서

# 디렉터리 확인
[root@k8s-node1 ~]# tree /var/lib/rancher/rke2 -L 1
/var/lib/rancher/rke2
├── agent
├── bin -> /var/lib/rancher/rke2/data/v1.33.8-rke2r1-1b2872361ec5/bin
├── data
└── server

4 directories, 0 files

# server 디렉터리
[root@k8s-node1 ~]# tree /var/lib/rancher/rke2/server/ -L 1
/var/lib/rancher/rke2/server/
├── agent-token -> /var/lib/rancher/rke2/server/token
├── cred
├── db
├── etc
├── manifests
├── node-token -> /var/lib/rancher/rke2/server/token
├── tls
└── token

5 directories, 3 files


[root@k8s-node1 ~]# ls -l /var/lib/rancher/rke2/server/
total 12
lrwxrwxrwx. 1 root root   34 Feb 23 06:02 agent-token -> /var/lib/rancher/rke2/server/token
drwx------. 2 root root 4096 Feb 23 06:02 cred
drwx------. 4 root root   35 Feb 23 06:05 db
drwx------. 2 root root   66 Feb 23 06:02 etc
drwxr-xr-x. 2 root root  180 Feb 23 06:05 manifests
lrwxrwxrwx. 1 root root   34 Feb 23 06:02 node-token -> /var/lib/rancher/rke2/server/token
drwx------. 6 root root 4096 Feb 23 06:02 tls
-rw-------. 1 root root  109 Feb 23 06:02 token

[root@k8s-node1 ~]# cat /var/lib/rancher/rke2/server/node-token
K10dc56af1974bf2a4a0dafabf6ada975428ac30b1cdfaee37aa24bab82bc13ba44::server:0f98c22f8008e212824486083fde55dd
[root@k8s-node1 ~]# cat /var/lib/rancher/rke2/server/token
K10dc56af1974bf2a4a0dafabf6ada975428ac30b1cdfaee37aa24bab82bc13ba44::server:0f98c22f8008e212824486083fde55dd

## helm chart manifests + values 포함
[root@k8s-node1 ~]# cat /var/lib/rancher/rke2/server/manifests/rke2-coredns.yaml
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  annotations:
    helm.cattle.io/chart-url: https://rke2-charts.rancher.io/assets/rke2-coredns/rke2-coredns-1.45.201.tgz
    rke2.cattle.io/inject-cluster-config: "true"
  name: rke2-coredns
  namespace: kube-system
spec:
  bootstrap: true
  chartContent: H4sICO13j2kCA3RtcC5PYUZIbGlVRmRqLnRhcgDcOmd74zaT+5m/AidfP5OS3OLVN62tXJzsOnpsv71CJCThDAI8ALSXKf/9ZgYgKTGyU6+9SrE4mI5pBGQfxUmaGysK7cZPXNXCZQ0v1Ztf8DOBz8XZGfylz+Dv6dnpxcWb6fn0bHp2fjY9P38zmZ5+cnbxhk3e/A98aue5BVV+Lp+hcf9PPkfsWqx5rTwLm8/WxrIYD1lyxB620jH4l7PfzT+8T2G15N6Lgq2lEoAA5LniVgC5lXylgIM3bCVYxZ0DNKnhsTG1ZV6UleJeANtElnwjZgljVlTGSW9sM2OW63wr7HjLbSG0KNqwBLQj9uWTsFYWyH4rGNEzzzfseWucYEW0QYblHFh4xqvq18I6aXSWMESesdHTNJueZdN0VUtVnExOLiYnk4sRLFe1UkujZA6K3KxvjV9a4YT2KBykVx7YcKUa5iqRy3XDuGbcWt4wsw7qLIHDvcit8C4LVPGJlRBj6JKS65p4AJh7cg6pq3kpXMVzEemsWM/Y1vvKzcbjx3olrBboN2nGhcnd2HP36Ma50Wu5qa1IK4O+0p5LDe5DS1LUCBasfAJBqRUb6bxtxsQ/Whu1m7E//AnBezACMJaSajNWNneRwxeiCTi3sJAksH3gMn5lau1nbIoAB3udC2KhZCkDM8byqgaEyaSkp1KUtOXTk8sPksLgPyH2fghuYo1SUm9+VRXcUwiV/OOvNH/iUmH4AV6A3dd2A08n5/+QJF7YUmqOe/jvFvy8FFaaAgwxugAHnE6SBHw419p4QgLY199CbLNc1Q6IU15743KuhM3298PxtUi9ScWTzD3E15orJ0bE7j1fCRU4JU5YQBAPTSUA6SpwvVkiojWlgCCoyfaIF9wgNBpUzBgxJRDfUZEAjHUMgj655STD21qMDqJUxqKmb6fnpwHBCSVybyypCr4zGhPyZR2KQoZs6C2khS6MgTtxxuQHg7gKgKGopLP2iOWtSwj1iCnDi3dccZ2/ALxS3LkIFx+BFPS5WYZo7iEPlq/XMg+JHbWQ1ae8lKoJwMjCB8RrDHK5qtG+GVtCIqIkQ5ZDLYy5CilPeXsf9Ke1mzWDrQED/THjAUs6thFaWEr12kHQEtUa8ozW24KYBNdF9fh+FBLzpSk61yUveHGe55SFCYvlZQZG1a8qHkmwYNduaAYoUkROYAnx+hmWxVJ+yMDErng+i9KjnCj3H1EvdvdufsW6wjK0L4mhc2eUCFzeNW0/+LH6Znuthm2FBdo1cHMS1J9Fc+J62DCIHeliQke3xhYhhYPuhHlnY0NyW1OrAjtBISplGlCGu67GuEhtLNPYZRV7vHTYxLJkKKE1vW9MrLLSWOkbYMdd7MHovYIaOlobt4BVpnAZ+1W71Jc2NHWn0LUcKdOw3LeRkSXDJXSFaxy4MW2tyWFdAiPy0FXbqkgRUIAp8SQUMMtr4mO0Fx+RMVTlfRiGyD6LAQIZAQmCs8hCuyCEe2opuGtMSV1/ZDmv+EoqoCPfSAhuLIRM1+VKWHCWMs/Qc07OaNhxTm400P7zX6/my7/cLh7+8u7m9vov94u7X99cLf76L1kyUB1jgyvgsYSWK5XYiAW6kcK8K569ElK4WVtM20KesqEcWiisqXqU+fv31C558aVWzZ0x/lOQFnzfRUU30n1lNAX/Mzrki65xAX1uylJg8wPs7500eFFKLWlb4/aOIZDSHL6aMsWUwNgV1o2PYpClYTgpeZUailCXBBQQmJJaZH/4OmMZ2QfR+hefV13dCo3q/JSSMk2hqFfGhRDqckUzzgq5XgsrtCcKwo7ry8iBeFB56RsxOUabgpBwQmU519101y5QDSDiFoLjwiQy3BrnWxmosKo3kjpzNzkJa411hIy5K3CQHm8FV37LwP+VkdqjFbPLyeXkuAtaCmolnwQ4yi2tWYkdnoGcXBbc/E6Z/HHGvkljmChAK+r8EeJ5KBojpxlInl5OB5IRSx4STeTE8q7F94bB7GYb1ocPmy9viE/BPe+pdzBI0YpbAHsMirYIZmAJV0zqFBLDZtxWnMnqgr68YjCWNCCinBQRBjkHXrKm3mxf4ceY9wp2dOAmUMvK3O07CiemgaNiOH0II1Nvaj9yfdfUSUb/ELueAtk9c1t8Bz1jY+HzMfY/9ZSh/TtEOc+3YkhC5nQoyphqbwdxitpb58UqDFWhzkIrBD6QbRy/sWfpt1jaKO9CMlPVaNOYDT9HhPlXWv0rW+FWUWqtBFNiDa6sPfjVS9CkOW47NXWsUYajWF8WtkIpkz0bq4qMGJvY7Q6TMebAG2iTV1jPEPIa2Yj98/NW5lsGO8qVY6NCI9mISQ1usOL69v5f9vRZG5OtuM0G0gIZAodFrHeJEx7cIl2/T6iC6V4rd4tbZSwFHLDB+eAZvIB/66LqejmohiHmTW5UEHz4c8RujY8NkcQ/S6Woj4NXH5lcU+HD13dwgAlK0HQE7gBRG1vlqAmVXJwAQhEniaE6T09Oz85J1I/19vnp0M9d/TwYUHExDqvRE9JFlUKcAZdD1SZyiRNw5HSM1tOAVIKOEkPdyxKQRT9E7Ahmrq7QZMek/zuQM8w5iA0GsYFylHSe5HTLjK/hzw63wGBYzVpz5fqgYBgzLI8J5XyjRKQ/JhSIIRnbVQyIkD+M8qeDRT0x1RfErz1GoFElWC1p7FRSFJisThbdi0OxO1iQJjR2xVLhDZM6V3Uhdl4SowA6L3LHLMRAyRuAl7CN1pTBsF8bVZeCSgs9h4FvRorLkqLtgNvHEG3j9vjyX0N53CWP0+MDBQ2asN9VjwkU+3H0txaiiAN0+/q7E6n/FAMumpXsscPOH0m6KUZqcARX10LxpjtxuMAKXe2fQkwBRiEIPu+A5wBbc6kgIh+2VritUUWAujrPQe4OdLpv6H4TP+5gzU8zlNj9eEtPD1h6/oMNnb5gKAVJJXJMR13VnjnQIfeYt9yhuPYdLKewfm3EtYKGx1yMu1fEHbSUV3KMh4bT8RFfr9HEJn2aprmhPMMqFBtliM4WJ8YqTY3zAawbIa5rC8X2HtpIUSv4drPRpgMvPoo8nENEssjtPr78Pwhbup21lJXc59vFx8qCt9ouzfrlR9H0LQySb2yNEvvNw4DnOB0s3Oj9FToc3ucYROLUlvRWM0YHaV72Rv8UiwN3byqjzKb5AjUf7e8djt5Y0EfdzLsSqvUNMqDPAZ8MHPJ4iXtcJS844bse2DUfVUoL7f7n4/EE4rF1j6swNyGPQCiX2u/EJ9dF3z8+E6psjzrA3Vi5Px3Gb8vznnhedTy7rU+Hng7gztd0JrgfKHSGsW9p6ND/9PXXrT6iOyLKaFdZxr799p++h4sE7bjOI6fsDnTiTmR4ILFLvR9H7cOAF3a0SBCOjx/Fcyip4fO8FfpX2oHn3FqGg+YYxmKum2fe/K3452Ca/UzXvBxVeFyb4JRaiOC4cAFVmSIewpRCe0C4+54rkdoJm25qWYgxFsk0HJFKo8cJPu8Whn1q42bhhOh/p6d4o4Qlfq91lR4rRs7r1byvYAuctQkaC9gsluwIE+s1GDwD/7c7luzJinJG5FOUMIgwvHUCcAohqqkU96JHi4/SeQfATsqoFzP6XtbC58UP4Ri6BjD8ISdZttZY7VXcw907tGI1PoqnP1AcU463aoV0tq4Qc1UXG+ETgF13sHcEa0fMjhUL52JhRqapN8ZPp17IZNLtZDL9ZDw5H08ux4EM20kqwKtSOLxtTXtbxgmypPO+eM3BUhIQqkWMmKxYhc0tTMml7uAYJGEBN01Q8n0Tw2AXJ4PHm1t2/+WcOdCR9My51mFNm3wIAhPOJ5cncHXPPjmZTNjpBfxvejJ5exEfXhYCb7FslUmueRqPBzOwNHudgP8Ygjn8mb49yaYXl9nbtxkc8La4/5rtYR/GTfrX2f62F1+Icm5Zd+XquleO+By3p30TdaYUaYeddvWULmnjcvhO9+O+X1zu3ErPFZRSd1AjEs+ewhsU8ybekCS7L1aHdAoUvUKirHxzLeli6aCgku6L6EgsFOpdCbviPiDiD5JJLJfcb2fhoOsZX2FhZ1/wfbxS9ybqIjVuR3e50XkZqCvjnFwpugbJuQMFtIjnduGkELF1aBPhmAtrDlvx/FHoIlhDl91DQwgvzZUU2qe5sN4dtKR9LfXKUTUjpPgG/cEUwOfsZDJ0z5rrvEmDma/65wC3s8lhpwn9JK2hVtr/TAP98APdVlnzBE52CLGoK82VFXfu2djgULQPl+k+aQuCYoV1CQgfeO83n80fFr9e3P1lcftrALediW4pomURFarQhwXchlzdLR72sT+1pmyHqOAsGF5wSIgwtuvTgJDuzjGh+cQF+B4PCeKszJTY8LyJ13ToNxevDePLcQgXMJSN4gvErHsZGIVBpruldVie7oXoyv8GaOsVlp0uRLZCleOVMqtxaM9jujB03frdYn79YZGVxREvYH+xQQlohvQl4qT9DekRA6XmVUWz5s5tZachHfRSuwm6Uj5hX2zwy3Vn9TFerB2zq/Yy57i9Oe6+xJNvqJ908AP6DC4U4yFBloT2tvujiE6H/kb4f06ReS8zanMEFRZcrwH4JPaPxSjEP1vOdwICWT5z7UWBlFfhvvozY+VXmEoK1J334s3qPyAUMsDEIaXiFKXD31ZQskt9F35S48KETQP3Dig003g9QWyGTynz9CuTuxgObcS34RHRuhSJv6/poZ7/F3vv3t02riuK77/9KXi0u0+TObFi59UZ79Xf+nX6mOk5feQm7czdt9NbKxZj61SWdEUpTeZxP/slSJimIcu0XTWP1p41qS2RIAGCIAiCQD7khVWMIcS3RRRHvyuC2C8DyWDBkFtvwb61XmfA9afxnrQUxSv0rlCb0HqvZUiL2xRzCs1ZYI7tszwFkaFEruUnZIvX2vlvbwqiZFCegaK764Dcmn7V/g5PNV5KHLnqUqudAvBKn4Gn56rDsP2xYbGMG0wYOny1mC56zHMkGHhaHRFoSRouC00VtaF1NbCXYACPfucAznY206PXYx1dKricW4pt4QIJLzr6wBcGeVuP9aT6r+ilYdmvy0TonQoQy0YEf+xqSkmiDsoYRVdbH0igrUuwC+18yLr+937nPyYm9JnBwObe2q29AvjIpLZXDAtTDggU2tGAwXoUc1TGADqaTwENcNMEWxxoCklxqkoeQ8Fnuow9+pYkrPiK2Gxk9VwduAL8qtNcrb8krMSoqJ7HwVBJ+gxdVWyurTKtr2CCx8p5GWttwx/6ZixwBQOgqiTuv57J35YjY7s9LftwgQ3oYSnaPBBFuxsALsZR1eWqihjMjK7tdNrx96tOpwvdTtd1PF3f9fTLO5+i++k8B9Q5LqhLOqEytoofFGVjn1FvJiwfGVc4oA0thN5fjN2Kkwjr4AEnodOmxthnWdUYo3a1BZY1xm6JbY2Y0hhb2cBVY5CqMXIxtqyZywGXmLqAoEbfh9ElXC0LzHhiUw9r9LH29g7H0IrtZu11j15G8BD9t0kNcMuuVDk6gCrWRKz0yfiDtdj0O87xhUtQuwYItBX53GeiAB6QslxkUZIEH3kOZMW1l4fGJxtEAR7tGzgsEsl9cBUECQNEq3qnWhqWqJ4fQ2+dmuAaekHlVLl62krPW+1TZHLiiqew5tCVFq6eu4JjUs3JK2NL2sQYm2cVY2yxXYyxhZYxxhbZxlrT/Znypv8YZegHYNQpSkbjee1iRGvvN99Le7GbNnRcQtEdlrD0lQQu2Cj9pNx5lINXCRBZJJ8GSRhbi1YGDq6oqybU49hy0AZOm/W7QV9Qxf/BRRqFbBDloLTmpFnfnr89hvV2GDb1LMoF+fVrVIx+TkXxCpzeX6UJn3pZFqpteObtgI+RVkTQkASE0OMCbRqq9KadVYBMKa3wyhGRBGsPc+CWEDpPN+mUVuAiq9dyOKKyyB+J2S6yNFdFUHUE2FAAzCtCeUeB4ZAjVPR2QdVy2kv9AD6WR6x5qBSZ7331n3kmeJDLtYKUGl8pE7t+6YtSruyXpoCeeqSGZuIkTAthPTeGNW/Pa9kdpS7VQokSMxEXsb7fgqJkTre0uWnWwgQLnvLmnLv1jrIP4JHJhVArzg++PD/w9zpST/bU2wt7B4SDhSyOTmufIjGaOIOZtrRdbhDFUTkGFRf6A3L58krtC4MBByTYWJb38WaE5gXByowF7LGu+ULBOuFhlPNBwZBrtl6cHG9Dk6LgXM8evDaiOjey+wFUA+dMbMbe/oMW4+suKgUgUfrK4W7CC3DKs5SZXQWrnWM/2pnqx+7flS6h38EKqdpRY2Z0o5Z2RNStfIjzDKm5wo4GIENDGry1m5HOB0f+A7qZweXoiwBfTdkcxumZ5jrtFo8yZbJ5UML+b5sPfmbu/5ormruvXr95euoXl8WXv//b2Tvco/d/D/c6+5v7v9fx+eOPNkhU/xd99ZtoFuAn0sIzD20++8TyMknkwkjteoFgAb3L5LcAPI/F8nD81vNi4i4eKPXTvrKl7+kYt/wW9h6XLgHLub6l4RmMrBsf0AnQStURz6vXT55+OH598ubhvS2QJYMiZkNesHbb2EQY9ZqBpwCFtVP23yJNsqAYPfT+8GHz6gNU8a7z3k+wD395DBsXbK4nD94/Q2+ebdK358ekZwBXrNW/SLatuiaKoCiFj0svV8/wx1+e6sBglDLvHnagd89QyTNDOUvwF9adzAVEB3nSY88L5flbBB/Bgn7OP4ERtyz4VOOwwbHnx2i/Mbd7feN+9y+8sPMJnKpUVY0bO7synHXfpp64GLD2p2UG4j5scsxY4F0sHA4CcKnBcDZJBuz+H5Nxsi+8+hIjGCcYsij76z6MFo7XvWkf546SfeOY1Y8RY94SXfXr8PQlQfyZezweANUu3qCXzU503dMkhLatr9b0hw0fD6dHrecpbAMkGXqtVtdnL4IykUMfsOMUCymNLE1jIQvgSAEvsHYE0zofwx8O4r54+AoOmllbby4fRsl5ehanl3CtTQHA7Sab/G619nz2P+Cak7l+oRV82dIuXgSzrzfd+fX/w4jHGbjXFFn8hdf/vYMHD+j6f9TtPNis/9ez/u9+xy6iMThDFMqbDI5YH8J2HVTzHvtuV85KKNWSbtTg91GQy+TKVcFvYbk2WB+ipOKv2jZv9YGc/xiqKQFiJJJ9uZr9Cdu9ZMCO9tXXaHyqNsLMayMwlBnwXfcPD+ED0wYIrisGLpBwGzsExwLVc7/1K9fQVfkC2gAsBJMyIygFV4Yn+8KswlfCiENtD1CWUX35UVlcts6MYICysPiATrBNyUKlqiENUcPwpaGGKUMKrEiuWEwh3VM49R66R4T2IcujpDhn3j9E+x/CI37FAHb5wasZyHQ8ThM8u6inoH6vAVadn8dBEgx52D676s0szRPF9k/JGGmhlr5FntN0tbPrgdeOL0baWacHq6dFRCjRnj7BSDSyOhoimPcfHvM+eLKY11pCCzd+RnPXaPlWvlQU0ct0axYhopibACHz3M49VNQ9e12u81F3uKi7Rlct3Jbt2zXa/rTo3IHfDLx9BH3zPEB7VGWHR8qZAZHUXIFaf6oFT8iTGq4g6BtpMkhlW2IwFWPaPmn/pN5x9vitsfT8hEeRsAiY+5/61g0gPw7AHxIWI1CGoRCqjxofdVNygaCzohmguGPyo1buU15gO7hTSrQTLwujQQFrN5YFga/KgcSHd7NgnhfYfbwwz+Xaj+v7zAVbMQsyD5LhlMq6oDCQDfTHQQZHvaqj02MIMUgz5ab36SNcfE+4sKDPdBr6XKSnRQ7FffWkTcshjeCltahhaZuCqups/1TQm4TPR3gUCAZGY3WaqFd7BfOKF3ojm6SFqhoV1e6jD8GWBCKdFnAI9D/bMzhUB0nwwi4vgahh8yJRDDJPm3PhVxlOf1lE8CyS0MYM/9KBOgF3N36h2ThKEp4rXtG3guEEUMAbc/McT3nZEE4rNVWqA6jAABRAKkpCfmmhNW84zI8XafqRBYXNjDrmgA5NNOKDj/o6sekQFLnS42egPCpUxXEKgzi7j1SqHLr88hAsByj1eqb22yfHkwv+5tmbx8eTEAM76oo8fDHmfVPMkILMFoWCoT3hFuA247TmeczXQQa22ZaSKiZEgecR/rEIaDkN4cGLCiVgbswjX5jYAnZHaX/4/zHdQSbzMcLBtnZKo1Vrhx74GZ/YfAxAajAx6846wMuwDrjN/esMQRHjEMDQ62+TscdBaZAYpL/kmVOW4XwxPngKkx0jHYsUeRuCPJylxQgYBaJuhdlcWQbzbksJNJzIFRS2t2kBMibbSKKGhnN94hIiEkyNEXeN9iZVPQLFPXi/5hHqEABJ3UTRATThUVoWPCfrunvZsHuGnaAdsBuHVkwQEgEW+DRXzvTYCbCUMSlp4SQOdtRn6QWfpxOotncIobBf7XmdfxEJVUbNsXaV9er4yZSdD/EhbPoB1+kjXEf1Irk1JINnL6HbzJusLB7z5FIg/2q1dmuyB5ZdUJtgXNI97bJ+rGHrAfgThnB78QrsQLQYXCeibx7PQbQYfDai5iFhkyc2e7ywx99JGjPV6uWu3YKZHE9IbUId6xVitdoqMiFam/1D/JZI8EX6nyJNTNsOfQx/38hew3iBbHYbuNv4mrcMm33BZl+w2Rds9gWbfcGy+4JJyM812ptU9RDKLdoX0AafcfCqUOVQiyA3Yk2MOhPnkgm8Sc/V5YVaMYdx/CzMqWBRpmPmTUEv0u5MIeVPStZuEyhuaQAfYtyH5HzIL0+zOJID1/OqDbW7S8EEEpoFrqY1Ask9fAT8FmVnusivLDrw92aTaGvDZKOz6uaw+R1g81i4d37Xu70DMbn+9s7Upts7fHU7t3foxUC9LIxyNhuz33VmhCH+X9U4GpBi/kA1bvXUqARbeFV5rlvYdh08hQH1PiCQPfziuYBcC92sbAKUaKDmmD5a5eY9I54TBmtX2RpqOdzyHNQxWGKQ/g9I8Q+Tq7lzWUP70ftznejn+ILseqvUJM4o3jJuIWB9sBwIlRMeju7z4zkjaorBYTo4Py6aAb7JQMKsaVpfSAHSSNhWDzkwz4/1iq11h50KWazuk5o6ZgGsbzwxoEwZVI+skl1GpEx9S7bssTttgR4WdaAJdv6HDpFvBiSpAddF/YN9X93rwfd0E0d+zv2KDHAcCFHPAbiRtW7hoMpaZQ1T4m0mipwHYxeTmAo+XE1ihGsRf3yKknI+B4KsbDvQfBZhtHXVYpSp0AYsSghyj188X4CZusL0/BiiIKyPEZnkO6AGNIaeScQD18xycNa9quIzm6xn4QSeLWqzlrvkNU9lx3xTI6GTDz0pg/i0kPs9M31MV0l5HeSDlnXMq1r/X8QMLqxb7juYGbDh+z/do8PDWf/fvW73YHP/51o+c1Qba7zx3qZ5BamaUEkE9mm3260gi9DFrcfgNVQfpTkGYAJfK3Ufv9v6GCVhjz2e6j2tMS8CSBHSM/ewl1B02rMX5TGeGKCBgmmhy6AvFf5EdgRsvgesKkzsKGUoGIr0X5Lr55aQwCxYdI6BayOpZt0vh1IkoRtUwtZoP23YeRlP0udI4v+Up2UG93A9733Ljq2lnqk7Q/hCiugz9RC2z1Kaqasz8G4ZQDk3UVMxJkYMCX8UXSn8IZfgmVeqFH3zGoD0bIkApKFckGViXpPTW8fYDhTGfgheLNk4RDkZp3h2YSI/CAZUZBKZyZ2ySIhSXaSXnn48hKBj6wTn+nv3aDl6TntSg4CaZYCALa6b/7jlv0voNyD/K/c/H8inNyT/N/J/Ou8c8v8aZD/djqslYDm5T4T9NUlle+orSeB5NBhP29xfFeoXqqX6B0ZWa5vIXPAchYN6DsJbfVHiu9peGIkBeE5fIfkXtB5HDuh0dVRjnIlMx8MBrCvN6xgFuh0j5UkfEE3MoKdqRLQjpeBWLRh7fLGKLQS/UgxUwh/krAWo2MX8y3YdOXH41HXR2RHVaWvcFMauugX9jcj/sygB4z7dBjQo//ceUP1f3gk8+Mrl/0b//1Ez1mYbsOaCI0moIl4zI7gWjIAsVRmAdektShXPGFY7DZQkFF4Xrp1CuuY6PRGTm8+1yX8q9BuU/90u0f9l9Je9o69Y/m/0f5T9X+E2oHmp7KbEOhLZcVYMYL9ZgbyR/8Y85VT7G4j/dVCx/3cOD/auUf5v5P8C3R/1fJRTii9eBtmM7L/o2oK8AeWvXtg0oOsz+XHo+1jEofTPhYqS0DSR5vOIa+aXby0yq6xHFhaOagSV2rIufGhzbnTaph5p3V3VTVzzA7kOY+OaeHTRJGVPHORDzNqwpVb9cZrr0KZie4d5JI+Fx8QoLeMQ0qdFSVBwH4PaIjQxDuJYQ5OV4V9X5ThKeJCbbPV/qL+s2jBw9DwCkXKSNOdxGhRHQJgdBIYdcQODck5gtWkjasHW1rDBSpLUApDvtCveTIXgsr5CcFmtUJtOoxZMXQ0zdah6882t/8ZRBJf/5tf/g+6Dg8r+76BzA+v/Zv03o02Xf7LcU22guvzPBim2l3dMlabdA82Crle0IAzTREfwyWlcH5Xk7gRi1w+imBu5D3ezIG6ZlLC4TlQCwFB/HZUZk3mzwQolpr3DfSOfGQPzRy7MTxWqGF6TD0bJZz/88P2BCrRPPiFPoiDWrw/N26n3Ts4hzKP5GadpVu+sNRvV2XYCAjMN6/jqP+NgSQsAcapAp8G4SbhKl6ecfdhitXSe5nDHkvkaVp4E+ZXbb8z78OH4+YsXj04+fHj84u3pm6fyiyzz4YNX57SmekvpDa0P1E20Kq3t6yq9H/YO99ch9IgHcTEiJLZfLUHk3ved7zsW5QwE0ucoaUMlP8izwM2cwH0brrp5rqKDmB1tBvAuD6C/GbnJyE0H4+3x6RuZlvTlhw8QffjpiRwR5iDo3VOnN/Y/p9bfgP1vn9r/ukcHN+L/tdH/q2c/KBEwLsW8gtM0S2tvFty2QnoY8Vn2wa/DAWDOBqhNb9gCyeQNW5jOrPfQ9AIzc+GYYSVdWv6WX6NzXcm6Q41LCX2sy9PLPPbPmsAt1evAUXgp+wqhCVRf4Ysw3YG32AVcCE2z9mVCTwPA4AzkpY9vATS+Q1g+AmsTUyc4cIG+DvyG3xkt2qq94z57ScqfIW+FsjOX36dbXw/e2nvaeytuarG/+pvVqiGg6ZEW+j9CEBtVGrh+5qFh8+6ejRthgAoL6L92oWo4HxiRZ1HMkWjQMvA0kqxn25DB/1x2gs46u9nPdKvbrP9hwMdpInhxDfa/o71up2r/O7yt6//G/gcXNqZeHU8Uq5zyojEjoIl4XanO2HIhrVc0JUIcHGhaXxo5xYyiujNw0yVKhm/VK/1IJQ9/m5isNDp5vJhJTzYG794XBqnFaOG8Q/A2GdUnngHjohAiA595WYs1xTH7Wh6pYHFE+yFuPgir6iVCm7dFbS1f2ZncDHCS380Ie1d9swJ8TyQ92uN0ckpIqwcriK2/Onfgdl5KvMMOB4FPUsgYWwpzCRlyPvhYhSQXhk/byuur0/q27KBJsOLPt2Jg6BYo4ewzljVdJzkaW7N5Ij8FUTER+fgK0wISRbw2WgDo43MNDyYHoD9N/wfLt6NwEQxtfWKQjscBiJZ398Xo/g673x7AX8l1UcySAWv/fsHapdXXOusRaDGH+/9kYaoTJQHmci6DPQOGd8LS/2Qi5jxje1Ay4fffVzlpUEtNksawOVouT8cqDWn6Z/zQJNBWYue9w7H90OR2PnwZmedBPhQ9ZhVrM6+Nl9Q9+zFkRFhklQKMDW3qLq0DYQjUNiij5Blk0N6dbIFo+RKNXOJiQF5NMpWaIt4aNjpsBTbABQ/yMP2UPFQLEC3AizKLkoLn58GAP4TpWlMmK1SmaSwyxxQ3uTUD05tfFvZIQsJ9SYOhyW5sXqS5PeZtNhMQqMcs+9xk4bZFAzNxFHsQ93BVOG1jcyWw3jxeAAushhQarIx5NBBOaDR9tvlAQL6feNEjFo3lxxsXlR5zMTgdOvxAgrce28UzpJk3Cm04F8LHtSm+j+wSNKX3oXl3kcblmL+ExVqsjXCbjQHAse52Xia7l5pDfdgQVgYIX7bJy5wHISRjxiy0hDa2NNW7zbbuuwXB7gVMeLN8kfpmWmtACyBMSrZsalHpThEyHKAh2YzopBDk1eox2OO+znWwotZyfR9MDGdWe6TK9IXJzY4ywEFeVwNUvzQflVLSlMdGPnI5xCiLNXhCIPPSPwsEv71b883+35h3qQNw8/v/Bw86Xbr/f3B43fc/Nvc/tuqdgDF666TAKAumb9wGAsNLDXsJuw8BvqkLgbjxdxsi1kx7hrXdKcuw4GckLqsJ9dxAAjKXsYXaWlzYLo7ePpc5PCSMt42VGyEWJVf1UYM0bIKP3E74ptiSfvjfL0CdTCkspKPFi3JsDrz1CKDIwCCo7J7/Bknj/ygRAt2PefV3ZLxtNU/FKNg7PJLAoQ9r8JN2SQagQZyNAkJq29bE7r/7w5PKl9fzHqNJ7xFYPwUo3RBMZmKDkgW0Ecr76/19F+MYOUWdvdNwzh0Des/APS7UWkktjOutDyYKG7F9kmSniFa10ByEaaGqMLy3oAh5hUiRGjWSloxL/QWM4Fzl9rialp08IdbUuoqWKXWtDljsOC1uPXR0wyr5uT1Z0cDsAPB5nUHbXCb5Ve7Qc15YxFGvjs0bSiEnJEfXnIbLKWxquFzPcjmF5zJd0pLUdmkoM7G/L6qbmXLzzJ8uqppy1tl2BwE5hnbGyGQdly1vhaq3DP3uNA3h+gAD+fObN8cumxFQ0IXCnIqztq+M51EargQTq1Bo1HK1LDisY8OjfvhvRjkXkJNmOYi01izMc32LaUWYpJZDIzPHHTMGYVeguZnC7bbZFT2s2xSRCkZ5ebjOCmvA6DRgD6dbvN2VYoZTVzJ4V+1rnA5l80XI81yZ18nri4d77umKOiMcLZANnKNsVS7QcaxvE92sUAK726XlTdtHDp+WL2j/oUafxu0/3cODww71/zzqbuw/t8f/8zPsOw2Igy/t+jl/K3uByFoS39JW/jRYYeLwR1k2J9N7z4Ok4faT/1/lfl+YYhz3AU1boRyX0y2y380r6RQJd5u0wvINz7G60VRYaxtTGcMYt7M6ET7U4cJpJxgTttMSnjWd2K5Lyzsz2Y3a5f3ZgtCuAXBa5kNHVSiClYyF0jGGk3KOASSl60YPXX3vllW0WVMoZeUvZQ5dH9EmbZZrGSqbF7/UOY2CzdKwBiZ5TQCuY+Bcw7pZa9KsokHkPMGFlnEgZFsGq02dzvrBQKX5DjLEfFxTHUTGynaegufjKFG4/JQHA348f7O7sBj0b1WAtF9uw6k70hfNwWetXmMoezpT8036kSeY+NPUXlQWPT+WJOyq1lRTcW2jqkOkUYdQbJIsQKaAuzV0/lwMxXuVJtxDdWF+SX2HZ3sWPtgfmGfeskgoz78o5yH7NOLJFBN4JbjKiqXacnQcTwTqOzItbh71nKcDS4+N08qdxWwLW6BVttm9tWy5RZqlcTq8Os3AiUmiBIpWlNiG3NoitFNuoKv2Ms3ZVh3jbptXtnV9u8Y6v8YCj/4+1RMnU4JVnZ/rV3naNHavfjFxnRnQRtyjvb5z+mcdGKx5SlBffa1OzDqN1zuT0/Zpzc88lzCLldeIP7XjKGK5Hf1yZxP1BxLGd/od+jDDmaztxzh1YGbvaxw4P9dTkl65w0uglOGsFqwri/PhQwHz05QiHp9gsK1VfrETv1iYYhnksfpy9gqyUL3myYXdN/mzRxvAYhbIbseGueS5kuswCR2xoWbFOjXj/gxWqhqRRpD7kodQX/AMqrGDpyZOmxo6YmrsXOlzDpMof8A8jBYyCJZYkUOg1tU8Bul+322CQbBXDXCIgbQ+iyCQBngE4TTAJAhpRS4h3EH2ynYl8soWmbXVbfG52OSx1MHY0qdh1IPfvVLWu72vFpGiMX94kjl18eV42gS9LT+3pfoy9tX8prQEoSoAPeizVwtqsYkO9jINZ8+ODjod5lsvsa9upYLyVrWEw1CHX78x/384lHAc/DaX/+FB94Cc/x7udW4m/8/G/5+eSLkOs/B82EzBx0EWnEVxVESy/KPj57iHEv7PgWAeAoCTod2LPY8mkZh9baKeLS52xotgz5qreCz9MyRbgNUjPk7DR6bjm+hDDd82AKq+UW5AmPeCsXm+AvC84i/AWMN+AuMoOSGnp4alrXdYOLisLYzvTGG8e0r3faY8viZjQcYVCp7xUXARTcxLk18ULC29eIzxa4PyP85dkZ+bzP92WMn/2X2wif98zfK/PgaM88Ywlc+efuVHKQj5Sfw39Uxd8T/hYQQZ37WZbI5E9mb4zzPCBm7aTyo/y9Pxj8HgI0+m991z8w62JBOfSryU/RLO/Hk+UUqjTLazVAwJzxjMj8lNenJb3t4Je4f7nvsKPbkm7wBCbrsbdIEMBltFsqeY4/PU9rSgfg/uaDIUZ4ox7eoibCmu7uqI5/Vr4Zv7v1kahpHIS3WZ/KwMh7z4UvG/QNt/UMn/s7/R/29h/jfJFk8MW/yo2IK1ifDXGYCn7qHH1TobHbxZHdy6eVo/UsSdznlNeH2/qTvrLYbA6HDX874Z9T0yMndd/qOjUBu3NUT2Nxz/+bDbpfr/7ZT/G/lv4rqjK5l9pDQ/7DPO9fVE/oT/buQGwCAdZ2kiC5nAU9fpj18l9Vy/fHzXgM++o2WHM72j4lre/Ku0vkaT139fwRWswz0SLpd4Ry23a/z6i/6SC/5aiz0lZIP+7yY4XpsGmTN70x+6GItO33Y8Ng8dK/7dX//dNsAG1v/Og4N9uv/r7O/f6Pq/if+8/rpeCa25ctxnKPJlIz5XAaPh8b/kQ2kAfItdN6bHqpwow6yNciEz4S+p7YuKDVXIgCgGDhBo56Mg6LpRpRyRTOvOf8esbyj/Y4faf7pHnQcb/f925X+5Fybi+bGAIDoii6OCeTteXX7FV6d2LXVlEerFPDFgyK1ih2hZ/5QQGfmarhJf4/6gecV/o28vr2S7NeuNZo3S4Plxb6lQ7RabDwtLcHQJNNGzE8igQAHxYokWxtrQJr4jvEB+1DEBv1Q5CmMD0jyALhCmqK9K+WKltt/kcH9pYF8nmPsKKLw8kOX6AEkMfwxiYM98QsTZZ9Cqo9p6zanre9gifexsFCvTduud7w2Yquv9HmZ50vfp7VRSOP2Ad0ln3siyusUoexaMo/hKU57MgtmXOAOWIVShh/MJ3HeJzsrCXFCZ8wLaXB7A5qD12/q49f9AX+GlAS4bPv/t7h2R/b/cANxU/J+N/b/q5FmTGMipweMN8E3Y5y95CLxi8Mnmwk7ufSve8hv5T83Azdp/peCn8v+gc/P2343/Z/Pif928gF/OEuyWWpv5T2d9w/rfwVHF//vg8Nbl/9z4f9D4QZYoaEALdMQo+vpMt4jhHQ63SBFxtuuo5OpArda7vqr7jeu3Dvk/ThOImvJl5f/e4cF+xf+vc7vl/8b/D1mjzk8AX0fJ0AfuSoX8Z7xLl4aXutTa/uDY+wXdM0vG3BVkyXpEDK3vZOjubxCGkQ72Xr/2uCuSJQlbvyGvdL4cneuWc1PitOqwrorq3+16lcDG3O1yR7voOA101NpEo3WcGS7reltpl+MVNzP+WZqT8suMrkrLexHE0+5MnkD3V6t555QFsv77Ix6Po2Eif7prNrb+w2Uvsv7vP9jYf67n83d2HBRwTiwgCKgeex0k9KyM4hCypUsx+jEYcuG3/s7ejCBiaJmpw1QmJL/EDJx/tAiTpXdYzuOgiC64ijtjPQ+SUAJI+FAtJxB8mJ9HlzzU69C/bfsMAsaxNIGaqksQ1YnFUcL9lv/k9MNpIfsmQTxOx2MJ4JfHpyyMctHyh1Gxq/7q7rf8s9/zXfV38mA03IU/k5/iItmdAjqT+JUZg+A4ovWdLz5l8u9Z8FH+Lcby+/+VRX8J8igtBXv+5KlogTD4bz4oWn4U8mBXlYNHrde/vnp6cgq/OcS6vHP6/8nTR09ePvXHIby5pvnf7RzIaC90/nf2N/f/r+UDsyAHj9dW6x1+e78Fkd9EbxdjR6oVeZtFggVMvmZCee2wYhQUbDAKokSwLC6H8C9sI7I8vYhCLlRZVDhEC2THi38+Oflnq9Xv9wdpItKYt+4xWHEYhOcEmy3DBhnpAMztUXkG/YDyWG0m7ZFlNWZKU4pjAw3/nUCDDsj+/J09T4o8DcsByKNWC0Qb4JMX7CxNCwjJmwHKVbJgb2BDQ0Bvs+nmiaUJ1P4vo9hoALL+jK6zPfFsYqUAaVuMOHv3s0TPtKdw9sVoeyKJ2cSyzaw+f4rieEJ7m/R6UAaBlHMce8dDFiVsXMZFlMUc46CVuZbLRTqR7+wChZ4Y8AS+ChZHsqehhBSnn3qtVnvCPCyQzRhEgOZoQFCNByzM06wdJZM0KIo852nOgDi7px+vJAhEJhJAAeNnA9Z6gGEaioRBAholhwCAl80Jhj8QugKnSJYKWOKuIPp1AWTvUx25D5SAEwWfoJkYFywbUV9yEytMGzXdDZIr6xQDFF/ocSlhqHD30XnEw9lOG1CGojiU/DJLhSydloWAIQeqITUAL8VM+JdH8mWuSrxKQ+XmBObIF5bTlPJwYuk5trIuvSDIu4tgyKM5MIBNjHMeYjjnKSsETJRnEmOh+nZ/HuHvS20lGoyAiILxYhCiMALwphtnOkAGArbmKCCoOgkPOWzTBqCCsD5Aak+iV/f1HNvBphLOQ1CXYBiynLdR3vDQV2LlOOcq3LqIJFowSaYigHX9bgdoDzuiHIWQqjyZ+ir2MAgjjvMah9u0Ab09T2M5AUXvtshS6C1msUPiCkP6NAG8LBoYLo0Se6rPiiHN+u9mnr3f+vvM720muJLdSiwJI64+BVdqdFK5SuVRWNMCC0uwTlnMIEfv/2Pfffcmyr77rsdeSJgMMM/1TlvgZOoDuVSLkmby/ZRuQq8qrx8/Z5EZVDqUQSzS2fE0Yh+HVf4iA7vaCKWDCNao0SCHMTbjBT0wwyaBYX4ryZH73/ud+nGEjvV1oT6bVErPP3d8FalgmWNvk6hgb7hQYuXf2RN+Vg6H8P2nUg4ersthOihhjEDsxUrlGKWfYJA/5VHBd1heJjtMdRuqa8AlAFaauJIFhVkrkeTvgKxtKAWF5i7vMyVmf237LTgABzwqUx7nNKCPXcFKKJv0+J4FYtQCmPjUjORqHQFQuhfspEwSwMyQVPZFPgM2tqmBA6NoAUwHwniL+0N/h/X9CYv0t2kvDRI+mf0pLEJZWUB07DJWDch59J+qF//z5YvvvgP6j4PCAjj+GEY5a2eqR22sR5pptwWYcgr2W4vJH7qNtlqoADZ5Dtu3GWg2ldRT0Gn8y3EM9QgOZiQN+yEPgThWVIS1IbfIZo5K9PY150HMLpSpCLhScSGhn6kybRu6r4pCQZgQ/3r08gWopTyOI8EToRSaX07lZIOZ8CgMNRfDnpgFhepOkWYwHa/SMp+OsQYEJFG8r6MwB9DBKNTST+llJdwyk53i8MhqSvccDn2k2g7/tOMgGZbBkLe18t9j9zTIhxNezYNPuLKANjNQ4bcKF+vujoMo2dWQ9BvFoGVUcP+/RZogfyllYAx6AxcaMx5GIC4UFjzPJYajaDiK5f/qMaAMkED2REkE2AklcWBmzF9vU1ZOXu2GXFKEa8k3mQ3WOjFXOpvarHZlzPlYYQCFiMg09k75Voh0EAUFGkYslgMYumtaKuPCpBF7bEvXVutPdhzkwZiDHF7q86fkfDHIo0yxwo1+oC+4ZqxXXWLfa6/yYX9C+Vvygc6v/1HY92nWi76D3sCmqg6b1gE5Bufi7DxPx9c49kTDXH3s+yapB0Hbjb2sM4v2Vsjz6IKH8AtXA3UYvv2lsO+7+uxG3CQgcQ77cyivsc1UhRuc8s/PX6XFcc4FT4p1sSeuHP3FLZ6qHffVNBUNi6b0EBrEdWHff/fe9HY97O38oP0lWnxVjuXyA6oD1hQ3N/ZdZxk39ph6xY+jcVQIf5CV/QUtmrQCEAQ4Gpdybh+/vSHs+91OZ9xvFPsxHxOp78BeV7gZ7Pe+fxn1G8Ee9mBcmNF3Yo/leQijf+fH3mBPRt+Nva5wF8feunPZX6KKrXOj3VJZPm9m7M0d0v6a2NeHY+vPX/F4oTeQRcr6RV5qoy168SI91Obp2ACe+FM0j72yFX/O2C8ON9avtGg7NxXKRgr/SAAGRyTBdYz9H1bvo1TugfMgM3cYdtjsWwyRDCGnvL/6ddgbl6P+vBaP05BNCrAb/QD2f32mtlPviLg+56MzIpkAJoXlLeN8h/tff4bzzUv0XYTDW3O0UMIxkkgpGeBAEV6HkRiAKR3O/a4swqyJ/R9/fa6ut8iLsD937JHrixRnAJ7jmFranjKJmS5YegbuHYIFuY2+2gP6nzn2ntc45xtHtP5czleShU3KKO7OdG5+W/YZdzqfbT31hz7z9jvCc29zbxz7hVLPSPRJobst9SaCfuBUHHBPfYy5D/SCJ0Q01Af817TO0bFvGHvRXxJ77sT/9u/v58dX6a889lCfnSEAthWdT9w9eLh9q8eeRtbpL2rxKZabZYKbmffv3n/uircwok9/DvZKFWKDOAJjmt4fAiWynAMopQPfPexNxByKdo3MD8sgbotCun5Q6+adknrurY17l3P9Er8hXc+xtfnKdzkLIjYt1HawPMvTUp2Hgi9jwYdX14k9fBrAnl427i86QVD+ejuTbd2/w86GzUK5LuzVFq9J7GGvQnGn2CdpAf55EnGkQCSQIkAI434wuUV3m8ceIk24Rtw99ic/Pno8Tdl+XWMPHWkA+0xkah1HArhPsiIu4OaG8vUMQBKeYn5mjPwnRmkZg/MuEimEveyts+sh9so7GTV9QgY39vKP7VmK7lU2SMUabopcP/ZVv9pVxx69RQxuC/2l05wl4JsVw01AdRHvZjk/y6MUWFYFcYS7jY4dLhSBM8xjrMegor3jydJQ3BldjyRU77taJOUtdyc084Ti+tw27Ly/QRj6r56++fDj81dPPpw+Pfnl+eOnS614PBf9ZVu0XZ5m3b2DZOIFfl1jf8o5uv/5V+N4nbGPowtu8vnXG7PJHm83jAT8q0TeCwTBMoBxjboeWtTJZ33sFRMH8RMeB1enEKkpFP3ZFtUrdsblqHMWz+IdCaYAgEC/FuyPOtC7xrDPeB6loY03bfHn9BNLzwuuDDqyOLjXAgcgAa77BLtR7ItozNOysNCnLf464omFLVQQTFa5kT3eYZ81if15EMVlzt+Mci5GaRxWTzFfRony2JDkAfGvbtrqWujrayiDN1SgYBSqAwwoB3rAOWhAkFga9oblYMB5yEP/5rFXfRFiReyxlht9LCj3PzMkQLL4q3P+52Kf8yCMHELfIfNPJiAQ6bsi8wn2bqFPZD5WvhGhD9jvd/pNjj0R+l+9zCfYE6H/dct8xN4h9L9WmY/YO4T+1yvzg3O1R7vqL9niI13e3LwVk60d7nDhRsrdsecnaWjiOC3lpSzLo8+KE+/bj32Rxjy3znCcFxNMeTf2t/8c7/c04c/gSlp/tf09Zzo4GftfEgBeart7Z7j8ssgD43wrnLad1xl6bAV5HlyBjQtE2iDI2cAAuWPY/5LG5RiHf2XsL3Rly23vTmL/Ek5zRP8zsB8DBBYlJuoDWr2mfHEbsXdcyXFwvqp8R7FPLhxIu7CXEKI8BZGvQ7IEZ3it1437jWNvBUlebeytNf+JuXm7JbZ34GBrZ3LSvUOdVtFz85as9zSo5vLz3lS5GRKgoq7dRN9mYVBwfxxcvk2CiyCK6WmcUdTxMlFirpaV0wrTa2YYeAOhs1KBb36L2m8a+9MyH1p4u7FX5y94ssiCs/QCzuJElPOQFmoc+73Df/Q/C3vZrSeRyEvFkT+W4ZAX/aVY97hakd01tyPA3j1ra7BnKYQxtKrfRewdEtuFva6OBe4Y9mEi9M5jHexhNSaBjJTpSvBiR4GeOC9GQoXfGMtCgyCO1fZeXU55lSa8f4NbVJ6Po0R1/ac8GPDjWrskwT5EhFkE2Kri2i6ThtOAWRPonA0BOHgiXfmNjf1+53OxH2WBywo/z9H25yxg05xZKo4PD0Il3fM0S3MkEWxf8XMrHW8Aezk8J7hQO7eogLYsT1Y9VEod18pv23qP2AeXK2Evy98O7PeawB6vAi3F+S91WTvCjr5IdnYFlIHJrmYDKzMduCb9lNxemV9NedhfRuaD3Mby0/CXbXvOt6eQ7R3b7Zn3BHsIuiKkzMdZ0HeEjoDiJp4Y4g+HU3QafEnOPzzqNzX2SRqugj0Uv2Hsu4B8M9iD6O+75z0s79HvHPC3Q47chK4Hy31z2AeXS2AfXN4i7LeCjwFLUliFtj8Te0xj8jaBYGxhqXbsr4C9+7aTm3IpBW43yxsK+bNAhaJNMCJdEcRmScz1jsCCi/PmFkm9LOcXPClOo2QY82O4BftMn672qy7mLEy5UK72AUTrZEJVYpObw+d4MAtkUGEPb6HzBcFeW6mexcFQ9OuPIVXIU8BPF2fnUB6QzAIBQrBInQtgU9hvJSkL8qHYbgJ7d6g0V3y0KbBrCJX2J8v5EO4BXUF2FhXhNQt2HZR3YE9CpTnio1072mTF87/3Dxua9zXx0hxB0ozLwRTS3QmStii1c/+rDpJGsHfcNCDWjUlpNoDi8zggS0N/7v0FtSuKzO0s/2auFxDsiRfK+q4n8Mgmw53Y4zm8UByuJw68bz32bi8U6npyw9i/e98g9vWRAp3hAQFp90rb/A4XIsQ1jz2NFOgOD3jdBEBXU4gQ1zT2NFKgOzxgc8jf/NjTSIEO7G9m9Bsee30qMw4yGknBHTzBwk9DYQCGPUtzxi8DiJ8OpQREYn/8nOU8CcYYXWl8W2S+43bZV3eljGDvvl12y66UoYN989jTiwa38noB7u+ax55eNPgKrxcQ7N23y27Z9YL9BrF33C772q4XEOypx/E34GZMU5evcI5nr3qQloRNYakDzKgQLOeQpU1xyFSZuvkVj2JPw8W4gicAyha20fl8QmI0GZ+9lhXyT5HQxEoQisrzFoliNqnWPFOJfPLz8SPlDxLkQ174xrrREPa1Gp5b15s3/Lc9VBTB3sTK+jYCZRHsLR+r2irgpkWNuWDg2MK8MzsMY7A8i3JBfv0aFaOfU1G8Ao8ucNba/lzsPWzV6zeCvcN5DfZ46gSn6qvGtmAuYxiOHSZ4kA9GXH5LlZgU2z47gSxnsPCpFNGG1iAbPCCGd/1jj1vSkzTmSuy9xrR//Trsp+VRdmEFLHGHgqS1IOxIH+OOQFqaPrAzGdQkLeRbY8jnwWDEMkwOZSed7bfb4IP4kV89VBDf7Ziv7/tw3qYT8BUppkDEfFf9mY3wzrzcWDRDIeRfo1l+4JnugBX666E65DWZtLQPtcmnhSu25UjJVDWdoBCDf5Uxh9Rfj2IVOROW7vhqhwXTBGk6FLYwUZSAFAppo/ths4ZmAgNnYyZTNRkAEMkspsIwrU4cSpj2ObMGGADMpKn8V1qq7pSCz2RbfGdVer9F0j/uWi+3dfqw4EJSTigyWwmFiY9qyLUDJ9cBtIEAkEEuhgVU5wZOWJoYb4AWdMiUgRR12l0MSKUT1sGVQDHJNcuDBAOTQ8mB8jXIUgmziKD1FqB4lhYj5r15fOwppcx7++TYgyaD6Tm8JHrrNB2DR05ahvbtGUic95Ez76fHTz1QQrxHv5c5t462MCqWxwZKg2hpLjTpa1Xs3EnoXEz8N69DBmeYdmkLNpeoTgBvoHOQrm4nsHjo2Rl5vT7gJcrBSCPSshHZYeNAYiKg+4qsqfZIKCDIn5qjSCx7jDBhoJBc3CqiMSaOe4R6mcpV+qM565nnE8ceMq2jBq13DkPgvGyaVt7rdpQMyjOIqu86ut5uAR/aYcyQXeCx4XbQ38wMQt8UNX25lO1XTPqIqbewlOWt7tFczy2AywXmV1aTIBliEr6FLnItGPdFXmToAm7SogoUujEov9iVLdXMWIJuAXy45NOnDU1CuoWp9qH2wSVFgxJjECEK1E4LLlkvrog5FVUuWs3j8gc7nYk4uWMvlCq3qv3gxygJMdNii2j3Mwl1SVoqsKMNkzTXJ6IFZnlG9vJt/kNfbMp4MjV6mkdSbBTocP+IshzNr74bpgOxWwTio9jNy6QdZKpDwGi7IwOrnaXhlOt2t1vTbqHfpuItCzlkQ+AWVYcl1YRaxi1qzCPlLttSAuhxVr4tojj6XfVih71U9nXrEQgnVNJAQOJEDdNMUcbsslCSmL0oZJ49t+XBp0Av2FZCy/rKRigjo+7AL4VfAC2r2nZhNecwByULBCCuUqbu+3t+R3YZVia9hgCbwRUA6ApuzTgwlD5dVY2afllNFCm6WAFYDOlXjrW8ztMYd65jH4Kv9WqS5uIf8FbYfXB08INagVvW/rDXYpACIfAxyf4uYqSSLfdU39r2o9rimEvC1KHPWwpZ1R6EWZzlUZ3QP2yfXfUUtnqZf50MlHgHoxI3lAunpIPvcUxGxuQ5trQaPZbj4MqMQ5yqJd1EhSV5RiEhKWS/CIYBKCjT4ZE9jc65UGzFE1iAoCaUzEGWodgUu2F0fi58IUVUy2BwxkEMBjnc+9DdRMEEvAoVNGYcylbxKpNxKgtFSIBEMkapkvJq4sGcBJZQA9NSGoCVkDdKLrhEYKgwTaFQhmyN3EfQU5VM+mbD5Tmkxm1DrMxHWaYuOU12G/IpLuWwCYGf9tVXSAnNhFJKAoHDoSaHGY80zyTlcKaSTk2FDmg5djYLzVOK9tF4XBYgsHzN48BggyJGiDaMaQ5w2ZlBEPKHunHNdPbiilLDrm0Nke0Hym2GSFLl7g86RmWN/RdBHCheAidicxTqo2dvnp7oURgFAkcCWEAC+DS71vwMbAjLWmKO5XRGW9UM8M6Vgm7N/ulUUvKrKoTkcP/66OTV81c/9dhpeSbfFmXB2b9evz358PPTFy8/nDx98fTR6dMPrx69fCqfnDzVCoNtGFODObL4CEfo7/+2W4p89yxKdnlywVSi6Bbg3OZlyrIo42BFbrVg/fkYgQkywa0dshoNbv1PFqYtJj98MEqZhytpLb4JuwdgJ5sCT1WdsE07YXaSeSO2223YLqus67q6YadasfhwAbW+UJvVNPmLGlIEqW9lobx+qMR1mCa4TX31+s3THgPpow6Rdhj4FkdpaasOwGu6AyHueCz5OI6GuZLyRWrWaFHAjAGV3tTCvanhBEAaVuFPPI6Rv5blAgSxO4HttVYZEKytC67NB8016WADFwvUN+LmAiD63+7YJ//I99oTs4DKvj7ZkeVpzK2N2Ad4qSwGK7fRkZ+jgwP4Fz70372D7oO/dQ+7B92Dw4Pu4eHfOnvd/YP9v7HOdRBA4hrksiufC4cid0c+N5vfX33tMXhq7y6Je2GrNdFSlfrcZuZnDa9qU1kLwGKNqOhNlM2cg9ZoN6eM2VZ7IB+VGVIpVrgNVAuI4AXAg8+0+OQJmxTtqd0qPgU4pgTutmcK2HritGDBg3GPhfwizQQ+pNFErNJJCkBtiqkuCsHzwhRrgwb1JB2AHRef4kdpDz3WbZmSkfivKAntMul5z6bZtOg4KAajEz7kl3bxLChGPSWYw6AIlE48+xZMaj32v/3vrHG7N4UqzxmCuApQ2XPfdd5PndmsIkrd77F3njKAeO+d0Ez39GLoA9EpPDMISwOzVlofBoZAJANFuFNWmMeh0bnNn5EA4/fqbAnmdDdfrsc8nRVQgbYRicbnFmK5JhKNyta7v/67F/1G1v+Dvdn1v/tgr9PdrP/fzPpviwd1bPdGFltu1celHoWH2WOpX1RoWAKjuhCbdDvWs1VUCEkb2DDmfFDgZiwJ9XJFJNGyK+zS6w20Vl1jcIvVhi3WZIIbkHjkJVZcYHHYeyZZbHMQcbvVHMAsDRsDZmUKXlJD6b73gyz6KU/LDKBXhwfTCptrnEt3tbvs0MSGnM7e7jl6i74yawDrVoGp49REROkK/LjnHuxJLhzV1YivBFuOw9lcuKXgrW95/T/Tx3wf6s0ATaz/h4f7ZP+/9+Bos/5/k+s/Hixb57puZYCyK7UEGPVg+nxF9aC6Rx9ftfXDdgxPe8xDYF6rdqeOFQLzXNYCgrSVYPRaLTRdQmnrPFEVMb/dOgklpa2aGK1BqSeiVLFC11RQEH7rCygq1giuvIknI1NtEAdqvQ19ZQyr8KdD6mwCmPaEn5sFswpMecNJaihXBdUgKgxLw4ajDArXHsfle/llxg+ZEBSP+X0ljimrALyGLk8nJm2GTtw7vP7TRb/Z/X93f4+s/53D/c5m/f+G1//VF/1GDAHyie1C3mM4M1S/W4yctysAUIEUvLGFXNzClRyQ/HbXQcD+Nq6AtH/fynK3+bjXfxMpwb3tb2T93z/odOn5vzwC2Kz/3876rzjuZZCtsO03XLr2bp8x4mHfg4sC6gXxoO+x/T31vDZQpAV0HCVwlK6+Bpfya0d9r42waR/LGqTMSWj1tN9GrK0P/j2MvGzCs0xLeHfLemEYoVbXsbDXC89KSo9p4LZYLZZfUL82Q4i7CcLn1Qbq2X459xRNrSjhQT7POeW+RwSEpyTE/YaAEyHjgZRpCnatoPK0pGqqnXGUSIjdxsAFlwCu0xS8WqHrodS9X6+mbvQ/h9LXmP1nv0P9P44OHxxs9L9vRv/zQNcz7vaG/9iQJxiE0ltCEZzv9emhhoF3eQMDHnUMvY4pV3t1HRxUjN9NolGPuqZRw1KNaxpU+hhlWt1AWeN243SoWFhOD5FVra06bB7YDyUAvAsDY4UQzEeNDJTx7TdwDbfHDvcP9+2HcTmMEtoGGr/O0/xTkIcE+vTCPLTAvvfVf5OWTDZYGw+guQYZnvlWx6tn8xLmnzPtWaV9xp6/YuwRY90f4C7int9traUkLr8IIfvM83E1I/Dbb9C5HhCW/fbbH15D0JH4EqSksfxj/ve+lF5HeNKNxH1o5Z03O6be+/tuasm/v/0m/uP5K/j7CP7IIZUPO/J/+LfrNb2Ab9b/qYhr5vqHa/2X6z71/+h2D4++rfV/c/9jbgiBNew/pNwU1ooXROb2Z9YuoULhr+u8znR1S1paySV6FhirQBEMe+yiK9e0jvV0msCgN5MhAIuMsmDRXQBDq95SVFtlIX1ioHxZa4sLOtyD9yfI+eqXCfqiTjVwIOvvafSQ6I00dGzGizZJxo9wJnCZ4T2aaUBrrlZs9/X5sgK6x/Q1yfbE0DLIIxUTyFSx2rWp0oZwUj3mPcbyj8JQlnidxFezqmiaQe00l0Wfwh1s4TUzSa5xBizPEZS8lPGc5F6lMWtg1PEin8Nz1dH5/IYmI1ptTQ9w/W0hW9ySe0/rX3q6RkZY/54UwRx6u/7tQ4owKXB9+N5ZBX2j/zuU/qbOf/e7h/T+V6e70f+/Hf0f41xY+iJ7AwWXNvrVqfxL6fkV9b4YcaLir2EEJLoL0VxomMfWjPLS9bsdv6uehShan8u+XoI0dYvtelndtH5MkOhhvymh8RTGVhL1lftIoPsV+jdhiOn16U3g4ct6FRUV1FoNiCinqLp8gXFRgFfVsyqExkFjcsINPopyjNPE8iJYj7JNIUsthS6E5xooCXJzowq8C9rnnfYP7/84Ovjr3txtFIR7nlwZ0zblSxXbimR6W5MTAfwpQn+sgVtioEweibcCpGxXrhSttTZdNyUXxCxWvkGGSgdE7lq3R00siBv9b5QFDsWvYfvvIfX/78p/N/7/1/L5f+xdd/OjNhD935+CSa8EjMtdeu+990Q22GYOGwbsX8bfPlqktQQLBGGc8e9M6tGeMFr2PaTd1VXoP1gNojncD4yyGN3Hd5CwPhWBJ2NqMi0ST+2ail0BlGTVnOD+mMBQo8wzVu5a7sATtZj/5KD2ybUs9LMsCalVdi0cZfxJ82euHX7VeiBc5iKAhWTiFOHyr/AfEoXOrXZxuuyncMcLLspFjV61PEeNDmM6tLpS/dSveUfi2Cz13YXdhVWj+OFpMceCJkJUhh2WYg6hwwwDDjFMkYR3kuGcHsIRwS4bAhGBrUxGw1Xl4q9jXwW6XiIS8RrCD5tiD7HzLxZ6qAyllQLRPAbFcltiKBdDMabtMHKL+D53JDId5scgzSp/IUsSXsffNcetTYVRX9+dQBsTTdohSpcMX5zgoSgaOmdjOHTidXeZHLpDCsdffce67z+7BcoXtL0HTrtmkGBsySx2PaHQRjynaytIR/VdjKR0Pn4OWPlZ0gN4gRBJI0CQ91b/J7Hvh1l6SMASFgd/HZgNBJvX/3Cccv7vfHw7+T+D/ud66P2Txb2bW1zz90ClkZpkAI/oYKE6ktDbeRW/In7YsTsWRoAFysA0u8YwvSbtKbGm6gn3qGwr0C+kaoekGjSKNgxWtNfOyhaXG7DzkVth6b89Qethw0oRsHbRE38092e/rUJfYovGyrO5MZ5hDuvFNKE/huN/ssjBy1KJmnO/6fzveD4vz//OpzdU/2vg/xTWS9gEh8z6QhgdVjxolgHEVg0lQHJqV5wg0WCDnN+UkKtwwCFly5QlylOLU9DBqItYkryKrK98lbqHmta28S7kOOFuTXVFsw7BO+oiQhr0Uh/qRPX/ttj/DSIF0lQMRAoiXkKYoPldgUCpYkhYzCne8degisiMbx3bKVkXYPc3yMfxq16q3htRbxOFVu9XV3TyOnW8fypTeCcTJOJM2iDCW0TqxJhZB8WCP1Csh+7UMwOSA19f18MNE7gX039K9116/nc8J/m/3lD//Yb0H5L9tyAHwBm3lH2Gck9epc9WootGHQl7n6iTXnhSQR6pNEii8U68y0/DBdawVsUnX/OdrmM/nNmO7TqlY0rMkJPEzpXvOK++irtwgeLShfbY9uxJ8ZTvU7ZahUtMF3ric1gHUpwSqdWHxe1NbI9DuPRoHprzqgUSIFq8vIQtcVKYfMi2YXQ8wX+dBqsgfZ+7+u/2bPlInLUX98CHjDi5LA7yQX4fJ3EUr49PXHhQDUwQrukkaLP/CiYFdgr9gnyVd2FQraZnyVph4z0q1ex+FqbpBk5cAEUX/zPUcydXQeFaS7mT96AYyocYQoEK6xXOrYBDZ9YODD1Z3c1Jt9cajLpFipl7yXaIRS9KoaRTNQRDp0vxNB/cDrN62vU97KJ2IEVHT+FKfr8daAUvUGSkiUH6X1j/M1Fs1LwIpHn95+m0vP6DM7+d/J9B/6tguFKl23bfAWiq3StBYqaOyqfnkR1pgApNq3UD/jYN1uCijvxhwUnGC0f0FwZJxOqrlr78bdchWdIPdHmrlxTyS3It4Zf0bybY8PUnaa57sfFz5a/+RK68VOP1h0tiAQPs1oahQ/qiDHR7//jfnPTN+d+Zled/Z9Mh/+N2+P8c0jcYA6QLOFDKbh76Q6GhzujG30dRWVCqDqI5iOKAo+hEW024imO4W15xNuVfJ7/rP/GSoWIX4l5uC914vdYmTeWDObmjTT6+xD7wv5wa/x/4fzp3SfyXO8z/3Rz/fyEsznQaEC3VSAmQ2C8J0hD7pbOtihuRx0L+QFLuHCH1A0dpme+H4I9ZpA8AiER9tn2VX7NKWW18mBaJijuNxw/qAsYM4rmK/FrUHH6wYodo30pXYO82RHThMzCXEgh+tVJCdYLx2ADYCgGUttNJN6h7OTdvUv3i7wrx01C1oXJyBi2me1B2P0Ml2AIulk2jlkgElDkeugSKCS7iGvn/vQ1Lzbn+XP533dkk539vNvfmOf+Px1cR/30Df5W+dXn/hyu2BJ7PI0c3XBcEWGsdqiWBCwZiC/xTRZxsmYaJ+LDl9ZNkff1DCn4EJq/FFCHUiNc4mV+8ioKAKwi230cBtLU47PwoeDnk+GCUI0x+Btw72ExwE+pjebY7KjSNNcTCzGIW/EEoGWu/YXtruYEV4LGIvazzE9+FHMH67BQrC5choWSjTQzkh/pHviKw4GDIf2HVgVfg84mrEnknf70XR/Eh/Uul/dsJd7m83tQ/cepLtkSOgT/j/yF4F/4M4knU4IKT1XAAd29rbXvDmF/YXLNwwTiJaTsz3p8bxkNDt8dHI7FLf/FHIs9YNoO/TMi9XOOVq5WJ+hEsgXR38cl5p/pmMrXHjjuCn3HqsWfffANKsnqe/eDml1z5lz04EAAAAAAQ5G89wgLVAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACAAD//+zBsQAAAADAIH/rXbOoBgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJAA7MGxAAAAAMAgf+tds6gGAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAkADswbEAAAAAwCB/612zqAYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACQAOzBsQAAAADAIH/rXbOoBgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJAA7MGxAAAAAMAgf+tds6gGAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAkADswbEAAAAAwCB/612zqAYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACQAOzBsQAAAADAIH/rXbOoBgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJAA7MGxAAAAAMAgf+tds6gGAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAkADswbEAAAAAwCB/612zqAYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACQAOzBsQAAAADAIH/rXbOoBgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJAA7MGxAAAAAMAgf+tds6gGAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAkADswbEAAAAAwCB/612zqAYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACQAOzBsQAAAADAIH/rXbOoBgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJAA7MGxAAAAAMAgf+tds6gGAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAkADswbEAAAAAwCB/612zqAYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACQAOzBsQAAAADAIH/rXbOoBgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJAA7MGxAAAAAMAgf+tds6gGAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAkADswbEAAAAAwCB/612zqAYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACQAOzBsQAAAADAIH/rXbOoBgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJAA7MGxAAAAAMAgf+tds6gGAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAkADswbEAAAAAwCB/612zqAYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACQAOzBsQAAAADAIH/rXbOoBgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJAA7MGxAAAAAMAgf+tds6gGAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAkADswbEAAAAAwCB/612zqAYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACQAOzBsQAAAADAIH/rXbOoBgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJAA7MGxAAAAAMAgf+tds6gGAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAkADswbEAAAAAwCB/612zqAYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACQAOzBsQAAAADAIH/rXbOoBgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJAA7MGxAAAAAMAgf+tds6gGAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAkADtwbEAAAAAwCB/612zqAYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQ3sSQqgAAQAA=
  set:
    global.clusterCIDR: 10.42.0.0/16
    global.clusterCIDRv4: 10.42.0.0/16
    global.clusterDNS: 10.43.0.10
    global.clusterDomain: cluster.local
    global.rke2DataDir: /var/lib/rancher/rke2
    global.serviceCIDR: 10.43.0.0/16
    global.systemDefaultIngressClass: ingress-nginx

워커 노드 추가

[k8s-node1]

# A token that can be used to register other server or agent nodes
[root@k8s-node1 ~]# cat /var/lib/rancher/rke2/server/node-token
K10dc56af1974bf2a4a0dafabf6ada975428ac30b1cdfaee37aa24bab82bc13ba44::server:0f98c22f8008e212824486083fde55dd

# 노드(서버/에이전트)가 RKE2 클러스터에 조인할 때 사용하는 전용 관리/부트스트랩 API 포트 확인
[root@k8s-node1 ~]# ss -tnlp | grep 9345
LISTEN 0      4096   192.168.10.11:9345       0.0.0.0:*    users:(("rke2",pid=6496,fd=6))
LISTEN 0      4096       127.0.0.1:9345       0.0.0.0:*    users:(("rke2",pid=6496,fd=7))
LISTEN 0      4096           [::1]:9345          [::]:*    users:(("rke2",pid=6496,fd=8))

[root@k8s-node1 ~]# watch -d 'kubectl get node; echo; kubectl get pod -n kube-system'
Every 2.0s: kubectl get node; echo; kubectl get pod -n kube-system                                                                                          k8s-node1: Mon Feb 23 06:15:40 2026

NAME        STATUS   ROLES                       AGE     VERSION
k8s-node1   Ready    control-plane,etcd,master   9m49s   v1.33.8+rke2r1

NAME                                         READY   STATUS	 RESTARTS   AGE
etcd-k8s-node1                               1/1     Running     0          9m5s
helm-install-rke2-canal-cn9bh                0/1     Completed   0          9m40s
helm-install-rke2-coredns-t87xt              0/1     Completed   0          9m40s
helm-install-rke2-metrics-server-wpjzt       0/1     Completed   0          9m40s
helm-install-rke2-runtimeclasses-5pn9f       0/1     Completed   0          9m40s
kube-apiserver-k8s-node1                     1/1     Running     0          9m5s
kube-controller-manager-k8s-node1            1/1     Running     0          9m5s
kube-proxy-k8s-node1                         1/1     Running     0          9m5s
kube-scheduler-k8s-node1                     1/1     Running     0          9m5s
rke2-canal-9wqbn                             2/2     Running     0          8m25s
rke2-coredns-rke2-coredns-559595db99-lv2ww   1/1     Running     0          8m26s
rke2-metrics-server-fdcdf575d-vxcnw          1/1     Running     0          7m33s


---
[k8s-node2]
# Run the installer
[root@k8s-node2 ~]# curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE="agent" INSTALL_RKE2_CHANNEL=v1.33 sh -
[INFO]  using stable RPM repositories
[INFO]  using 1.33 series from channel stable

Rancher RKE2 Common (v1.33)                                                                                                                                     62  B/s | 659  B     00:10
Rancher RKE2 Common (v1.33)                                                                                                                                    472  B/s | 2.4 kB     00:05
Importing GPG key 0xE257814A:
 Userid     : "Rancher (CI) <ci@rancher.com>"
 Fingerprint: C8CF F216 4551 26E9 B9C9 18BE 925E A29A E257 814A
 From       : https://rpm.rancher.io/public.key
Rancher RKE2 Common (v1.33)                                                                                                                                    166  B/s | 2.6 kB     00:16
Rancher RKE2 1.33 (v1.33)                                                                                                                                       64  B/s | 659  B     00:10
Rancher RKE2 1.33 (v1.33)                                                                                                                                      472  B/s | 2.4 kB     00:05
Importing GPG key 0xE257814A:
 Userid     : "Rancher (CI) <ci@rancher.com>"
 Fingerprint: C8CF F216 4551 26E9 B9C9 18BE 925E A29A E257 814A
 From       : https://rpm.rancher.io/public.key
Rancher RKE2 1.33 (v1.33)                                                                                                                                      388  B/s | 5.9 kB     00:15
Dependencies resolved.
===============================================================================================================================================================================================
 Package                                   Architecture                         Version                                          Repository                                               Size
===============================================================================================================================================================================================
Installing:
 rke2-agent                                aarch64                              1.33.8~rke2r1-0.el9                              rancher-rke2-1.33-stable                                8.3 k
Installing dependencies:
 rke2-common                               aarch64                              1.33.8~rke2r1-0.el9                              rancher-rke2-1.33-stable                                 25 M
 rke2-selinux                              noarch                               0.22-1.el9                                       rancher-rke2-common-stable                               22 k

Transaction Summary
===============================================================================================================================================================================================
Install  3 Packages

Total download size: 25 M
Installed size: 113 M
Downloading Packages:
(1/3): rke2-selinux-0.22-1.el9.noarch.rpm                                                                                                                      4.2 kB/s |  22 kB     00:05
(2/3): rke2-agent-1.33.8~rke2r1-0.el9.aarch64.rpm                                                                                                              1.5 kB/s | 8.3 kB     00:05
(3/3): rke2-common-1.33.8~rke2r1-0.el9.aarch64.rpm                                                                                                             4.0 MB/s |  25 MB     00:06
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                          4.0 MB/s |  25 MB     00:06
Rancher RKE2 Common (v1.33)                                                                                                                                    472  B/s | 2.4 kB     00:05
Importing GPG key 0xE257814A:
 Userid     : "Rancher (CI) <ci@rancher.com>"
 Fingerprint: C8CF F216 4551 26E9 B9C9 18BE 925E A29A E257 814A
 From       : https://rpm.rancher.io/public.key
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                       1/1
  Running scriptlet: rke2-selinux-0.22-1.el9.noarch                                                                                                                                        1/3
  Installing       : rke2-selinux-0.22-1.el9.noarch                                                                                                                                        1/3
  Running scriptlet: rke2-selinux-0.22-1.el9.noarch                                                                                                                                        1/3
  Installing       : rke2-common-1.33.8~rke2r1-0.el9.aarch64                                                                                                                               2/3
  Installing       : rke2-agent-1.33.8~rke2r1-0.el9.aarch64                                                                                                                                3/3
  Running scriptlet: rke2-agent-1.33.8~rke2r1-0.el9.aarch64                                                                                                                                3/3
  Running scriptlet: rke2-selinux-0.22-1.el9.noarch                                                                                                                                        3/3
  Running scriptlet: rke2-agent-1.33.8~rke2r1-0.el9.aarch64                                                                                                                                3/3
  Verifying        : rke2-selinux-0.22-1.el9.noarch                                                                                                                                        1/3
  Verifying        : rke2-agent-1.33.8~rke2r1-0.el9.aarch64                                                                                                                                2/3
  Verifying        : rke2-common-1.33.8~rke2r1-0.el9.aarch64                                                                                                                               3/3

Installed:
  rke2-agent-1.33.8~rke2r1-0.el9.aarch64                           rke2-common-1.33.8~rke2r1-0.el9.aarch64                           rke2-selinux-0.22-1.el9.noarch

Complete!

[root@k8s-node2 ~]# TOKEN=K10dc56af1974bf2a4a0dafabf6ada975428ac30b1cdfaee37aa24bab82bc13ba44::server:0f98c22f8008e212824486083fde55dd
[root@k8s-node2 ~]# cat << EOF > /etc/rancher/rke2/config.yaml
server: https://192.168.10.11:9345
token: $TOKEN
EOF

[root@k8s-node2 ~]# systemctl enable --now rke2-agent.service
Created symlink /etc/systemd/system/multi-user.target.wants/rke2-agent.service → /usr/lib/systemd/system/rke2-agent.service.

---
[root@k8s-node1 ~]# kubectl get node -owide
NAME        STATUS   ROLES                       AGE     VERSION          INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                      KERNEL-VERSION                  CONTAINER-RUNTIME
k8s-node1   Ready    control-plane,etcd,master   21m     v1.33.8+rke2r1   192.168.10.11   <none>        Rocky Linux 9.6 (Blue Onyx)   5.14.0-570.52.1.el9_6.aarch64   containerd://2.1.5-k3s1
k8s-node2   Ready    <none>                      2m10s   v1.33.8+rke2r1   192.168.10.12   <none>        Rocky Linux 9.6 (Blue Onyx)   5.14.0-570.52.1.el9_6.aarch64   containerd://2.1.5-k3s1

---
[k8s-node2]
# 디렉터리 확인
[root@k8s-node2 ~]# tree /var/lib/rancher/rke2 -L 1
/var/lib/rancher/rke2
├── agent
├── bin -> /var/lib/rancher/rke2/data/v1.33.8-rke2r1-1b2872361ec5/bin
├── data
└── server

4 directories, 0 files

# rke2-agent
[root@k8s-node2 ~]# systemctl status rke2-agent.service --no-pager
● rke2-agent.service - Rancher Kubernetes Engine v2 (agent)
     Loaded: loaded (/usr/lib/systemd/system/rke2-agent.service; enabled; preset: disabled)
     Active: active (running) since Mon 2026-02-23 06:25:20 KST; 3min 8s ago
       Docs: https://github.com/rancher/rke2#readme
    Process: 6372 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 6373 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 6374 (rke2)
      Tasks: 59
     Memory: 2.2G
        CPU: 37.756s
     CGroup: /system.slice/rke2-agent.service
             ├─6374 "/usr/bin/rke2 agent"
             ├─6397 containerd -c /var/lib/rancher/rke2/agent/etc/containerd/config.toml
             ├─6455 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --config-dir=/var/lib/rancher/rke2/agent/etc/kubelet.conf.d --co…
             ├─6518 /var/lib/rancher/rke2/data/v1.33.8-rke2r1-1b2872361ec5/bin/containerd-shim-runc-v2 -namespace k8s.io -id 51b9f02e55775472f04a5e01fd88c2148cfd6d56ab2df6e59023b653d45e39a9 …
             └─6520 /var/lib/rancher/rke2/data/v1.33.8-rke2r1-1b2872361ec5/bin/containerd-shim-runc-v2 -namespace k8s.io -id 677b5182e8e8a2247e49513207b6cb10d57fd11374e83da2bdae951223a2edbc …

Feb 23 06:24:56 k8s-node2 rke2[6374]: I0223 06:24:56.200674    6374 event.go:389] "Event occurred" object="k8s-node2" fieldPath="" kind="Node" apiVersion="" type="Normal" reas…by rke2 are OK"
Feb 23 06:25:20 k8s-node2 rke2[6374]: time="2026-02-23T06:25:20+09:00" level=error msg="Failed to process image event: failed to import /var/lib/rancher/rke2/agent/images/kube-proxy-image.tx…
Feb 23 06:25:20 k8s-node2 rke2[6374]: time="2026-02-23T06:25:20+09:00" level=info msg="Pulling images from /var/lib/rancher/rke2/agent/images/runtime-image.txt"
Feb 23 06:25:20 k8s-node2 rke2[6374]: time="2026-02-23T06:25:20+09:00" level=info msg="Pulling image index.docker.io/rancher/rke2-runtime:v1.33.8-rke2r1"
Feb 23 06:25:20 k8s-node2 rke2[6374]: time="2026-02-23T06:25:20+09:00" level=info msg="Running kubelet --alsologtostderr=false --config-dir=/var/lib/rancher/rke2/agent/etc/kubelet.conf.d --c…
Feb 23 06:25:20 k8s-node2 rke2[6374]: time="2026-02-23T06:25:20+09:00" level=info msg="Annotations and labels have been set successfully on node: k8s-node2"
Feb 23 06:25:20 k8s-node2 rke2[6374]: time="2026-02-23T06:25:20+09:00" level=info msg="rke2 agent is up and running"
Feb 23 06:25:20 k8s-node2 systemd[1]: Started Rancher Kubernetes Engine v2 (agent).
Feb 23 06:25:57 k8s-node2 rke2[6374]: time="2026-02-23T06:25:57+09:00" level=error msg="Failed to process image event: failed to import /var/lib/rancher/rke2/agent/images/runtime-image.txt: …
Feb 23 06:26:37 k8s-node2 rke2[6374]: time="2026-02-23T06:26:37+09:00" level=info msg="Tunnel authorizer set Kubelet Port 0.0.0.0:10250"
Hint: Some lines were ellipsized, use -l to show in full.

[root@k8s-node2 ~]# cat /usr/lib/systemd/system/rke2-agent.service
[Unit]
Description=Rancher Kubernetes Engine v2 (agent)
Documentation=https://github.com/rancher/rke2#readme
Wants=network-online.target
After=network-online.target
Conflicts=rke2-server.service

[Install]
WantedBy=multi-user.target

[Service]
Type=notify
EnvironmentFile=-/etc/default/%N
EnvironmentFile=-/etc/sysconfig/%N
EnvironmentFile=-/usr/lib/systemd/system/%N.env
KillMode=process
Delegate=yes
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Restart=always
RestartSec=5s
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/bin/rke2 agent
ExecStopPost=-/bin/sh -c "systemd-cgls /system.slice/%n | grep -Eo '[0-9]+ (containerd|kubelet)' | awk '{print $1}' | xargs -r kill"

# PATH 안 건드리고 표준 위치로 노출 설정 : 심볼릭 링크 방식 
[root@k8s-node2 ~]# ln -s /var/lib/rancher/rke2/bin/containerd /usr/local/bin/containerd
[root@k8s-node2 ~]# ln -s /var/lib/rancher/rke2/bin/crictl /usr/local/bin/crictl
[root@k8s-node2 ~]# ln -s /var/lib/rancher/rke2/agent/etc/crictl.yaml /etc/crictl.yaml

[root@k8s-node2 ~]# crictl ps
CONTAINER           IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID              POD                    NAMESPACE
4010ec38f31bb       fc62334b90cf6       2 minutes ago       Running             kube-flannel        0                   51b9f02e55775       rke2-canal-522dn       kube-system
d48f2671d4f9f       3b9613c95d89e       2 minutes ago       Running             calico-node         0                   51b9f02e55775       rke2-canal-522dn       kube-system
094341fc3e0ac       603f9fc02b584       3 minutes ago       Running             kube-proxy          0                   677b5182e8e8a       kube-proxy-k8s-node2   kube-system

[root@k8s-node2 ~]# crictl images
IMAGE                                   TAG                            IMAGE ID            SIZE
docker.io/rancher/hardened-calico       v3.31.3-build20260206          3b9613c95d89e       217MB
docker.io/rancher/hardened-flannel      v0.28.1-build20260206          fc62334b90cf6       19.8MB
docker.io/rancher/hardened-kubernetes   v1.33.8-rke2r1-build20260210   603f9fc02b584       187MB
docker.io/rancher/mirrored-pause        3.6                            7d46a07936af9       253kB
docker.io/rancher/rke2-runtime          v1.33.8-rke2r1                 35592a070625a       91.3MB

수동 업그레이드 v1.33 -> v1.34

# 모니터링
[root@k8s-node1 ~]# while true; do curl -s http://192.168.10.12:30000 | grep Hostname; date; sleep 1; done
Mon Feb 23 06:32:25 AM KST 2026
Mon Feb 23 06:32:26 AM KST 2026
Mon Feb 23 06:32:27 AM KST 2026
Mon Feb 23 06:32:28 AM KST 2026
...

# 버전 정보 확인
[root@k8s-node1 ~]# kubectl get node
NAME        STATUS   ROLES                       AGE     VERSION
k8s-node1   Ready    control-plane,etcd,master   27m     v1.33.8+rke2r1
k8s-node2   Ready    <none>                      7m44s   v1.33.8+rke2r1
[root@k8s-node1 ~]# rke2 --version
rke2 version v1.33.8+rke2r1 (eb75e3c1774cee5a584259d6fee77eb8cfa9b430)
go version go1.24.12 X:boringcrypto
[root@k8s-node1 ~]# curl -s https://update.rke2.io/v1-release/channels | jq .data
[
  {
    "id": "stable",
    "type": "channel",
    "links": {
      "self": "https://update.rke2.io/v1-release/channels/stable"
    },
    "name": "stable",
    "latest": "v1.34.4+rke2r1"
  },
  {
    "id": "latest",
    "type": "channel",
    "links": {
      "self": "https://update.rke2.io/v1-release/channels/latest"
    },
    "name": "latest",
    "latest": "v1.35.1+rke2r1",
    "latestRegexp": ".*",
    "excludeRegexp": "(^[^+]+-|v1\\.25\\.5\\+rke2r1|v1\\.26\\.0\\+rke2r1)"
  },
  {
    "id": "v1.18",
    "type": "channel",
    "links": {
      "self": "https://update.rke2.io/v1-release/channels/v1.18"
    },
    "name": "v1.18",
    "latest": "v1.18.20+rke2r1",
    "latestRegexp": "v1\\.18\\..*",
    "excludeRegexp": "^[^+]+-"
  },
  {
    "id": "v1.19",
    "type": "channel",
    "links": {
      "self": "https://update.rke2.io/v1-release/channels/v1.19"
    },
    "name": "v1.19",
    "latest": "v1.19.16+rke2r1",
    "latestRegexp": "v1\\.19\\..*",
    "excludeRegexp": "(^[^+]+-|v1\\.19\\.13\\+rke2r1)"
  },
  {
    "id": "testing",
    "type": "channel",
    "links": {
      "self": "https://update.rke2.io/v1-release/channels/testing"
    },
    "name": "testing",
    "latest": "v1.18.9-beta22+rke2",
    "latestRegexp": "-(alpha|beta|rc)"
  },
  {
    "id": "v1.20",
    "type": "channel",
    "links": {
      "self": "https://update.rke2.io/v1-release/channels/v1.20"
    },
    "name": "v1.20",
    "latest": "v1.20.15+rke2r2",
    "latestRegexp": "v1\\.20\\..*",
    "excludeRegexp": "(^[^+]+-|v1\\.20\\.9\\+rke2r1)"
  },
  {
    "id": "v1.21",
    "type": "channel",
    "links": {
      "self": "https://update.rke2.io/v1-release/channels/v1.21"
    },
    "name": "v1.21",
    "latest": "v1.21.14+rke2r1",
    "latestRegexp": "v1\\.21\\..*",
    "excludeRegexp": "(^[^+]+-|v1\\.21\\.3\\+rke2r2)"
  },
  {
    "id": "v1.22",
    "type": "channel",
    "links": {
      "self": "https://update.rke2.io/v1-release/channels/v1.22"
    },
    "name": "v1.22",
    "latest": "v1.22.17+rke2r1",
    "latestRegexp": "v1\\.22\\..*",
    "excludeRegexp": "^[^+]+-"
  },
  {
    "id": "v1.23",
    "type": "channel",
    "links": {
      "self": "https://update.rke2.io/v1-release/channels/v1.23"
    },
    "name": "v1.23",
    "latest": "v1.23.17+rke2r1",
    "latestRegexp": "v1\\.23\\..*",
    "excludeRegexp": "^[^+]+-"
  },
  {
    "id": "v1.24",
    "type": "channel",
    "links": {
      "self": "https://update.rke2.io/v1-release/channels/v1.24"
    },
    "name": "v1.24",
    "latest": "v1.24.17+rke2r1",
    "latestRegexp": "v1\\.24\\..*",
    "excludeRegexp": "(^[^+]+-|v1\\.24\\.9\\+rke2r1)"
  },
  {
    "id": "v1.25",
    "type": "channel",
    "links": {
      "self": "https://update.rke2.io/v1-release/channels/v1.25"
    },
    "name": "v1.25",
    "latest": "v1.25.16+rke2r2",
    "latestRegexp": "v1\\.25\\..*",
    "excludeRegexp": "(^[^+]+-|v1\\.25\\.5\\+rke2r1)"
  },
  {
    "id": "v1.26",
    "type": "channel",
    "links": {
      "self": "https://update.rke2.io/v1-release/channels/v1.26"
    },
    "name": "v1.26",
    "latest": "v1.26.15+rke2r1",
    "latestRegexp": "v1\\.26\\..*",
    "excludeRegexp": "(^[^+]+-|v1\\.26\\.0\\+rke2r1)"
  },
  {
    "id": "v1.27",
    "type": "channel",
    "links": {
      "self": "https://update.rke2.io/v1-release/channels/v1.27"
    },
    "name": "v1.27",
    "latest": "v1.27.16+rke2r2",
    "latestRegexp": "v1\\.27\\..*",
    "excludeRegexp": "^[^+]+-"
  },
  {
    "id": "v1.28",
    "type": "channel",
    "links": {
      "self": "https://update.rke2.io/v1-release/channels/v1.28"
    },
    "name": "v1.28",
    "latest": "v1.28.15+rke2r1",
    "latestRegexp": "v1\\.28\\..*",
    "excludeRegexp": "^[^+]+-"
  },
  {
    "id": "v1.29",
    "type": "channel",
    "links": {
      "self": "https://update.rke2.io/v1-release/channels/v1.29"
    },
    "name": "v1.29",
    "latest": "v1.29.15+rke2r1",
    "latestRegexp": "v1\\.29\\..*",
    "excludeRegexp": "^[^+]+-"
  },
  {
    "id": "v1.30",
    "type": "channel",
    "links": {
      "self": "https://update.rke2.io/v1-release/channels/v1.30"
    },
    "name": "v1.30",
    "latest": "v1.30.14+rke2r4",
    "latestRegexp": "v1\\.30\\..*",
    "excludeRegexp": "^[^+]+-"
  },
  {
    "id": "v1.31",
    "type": "channel",
    "links": {
      "self": "https://update.rke2.io/v1-release/channels/v1.31"
    },
    "name": "v1.31",
    "latest": "v1.31.14+rke2r1",
    "latestRegexp": "v1\\.31\\..*",
    "excludeRegexp": "^[^+]+-"
  },
  {
    "id": "v1.32",
    "type": "channel",
    "links": {
      "self": "https://update.rke2.io/v1-release/channels/v1.32"
    },
    "name": "v1.32",
    "latest": "v1.32.12+rke2r1",
    "latestRegexp": "v1\\.32\\..*",
    "excludeRegexp": "^[^+]+-"
  },
  {
    "id": "v1.33",
    "type": "channel",
    "links": {
      "self": "https://update.rke2.io/v1-release/channels/v1.33"
    },
    "name": "v1.33",
    "latest": "v1.33.8+rke2r1",
    "latestRegexp": "v1\\.33\\..*",
    "excludeRegexp": "^[^+]+-"
  },
  {
    "id": "v1.34",
    "type": "channel",
    "links": {
      "self": "https://update.rke2.io/v1-release/channels/v1.34"
    },
    "name": "v1.34",
    "latest": "v1.34.4+rke2r1",
    "latestRegexp": "v1\\.34\\..*",
    "excludeRegexp": "^[^+]+-"
  },
  {
    "id": "v1.35",
    "type": "channel",
    "links": {
      "self": "https://update.rke2.io/v1-release/channels/v1.35"
    },
    "name": "v1.35",
    "latest": "v1.35.1+rke2r1",
    "latestRegexp": "v1\\.35\\..*",
    "excludeRegexp": "^[^+]+-"
  }
]


# v1.34 버전 업그레이드! : 아래 Running scriptlet 과정에서 업그레이드 수행됨, app 통신 영향 없었음.
[root@k8s-node1 ~]# curl -sfL https://get.rke2.io | INSTALL_RKE2_CHANNEL=v1.34 sh -
[INFO]  using stable RPM repositories
[INFO]  using 1.34 series from channel stable
Importing GPG key 0xE257814A:
 Userid     : "Rancher (CI) <ci@rancher.com>"
 Fingerprint: C8CF F216 4551 26E9 B9C9 18BE 925E A29A E257 814A
 From       : https://rpm.rancher.io/public.key
Error: Failed to download metadata for repo 'rancher-rke2-1.34-stable': repomd.xml GPG signature verification error: Bad GPG signature
Importing GPG key 0xE257814A:
 Userid     : "Rancher (CI) <ci@rancher.com>"
 Fingerprint: C8CF F216 4551 26E9 B9C9 18BE 925E A29A E257 814A
 From       : https://rpm.rancher.io/public.key
Error: Failed to download metadata for repo 'rancher-rke2-1.34-stable': repomd.xml GPG signature verification error: Bad GPG signature
Rancher RKE2 1.34 (v1.34)                                                                                                                                       59  B/s | 659  B     00:11
Rancher RKE2 1.34 (v1.34)                                                                                                                                      471  B/s | 2.4 kB     00:05
Importing GPG key 0xE257814A:
 Userid     : "Rancher (CI) <ci@rancher.com>"
 Fingerprint: C8CF F216 4551 26E9 B9C9 18BE 925E A29A E257 814A
 From       : https://rpm.rancher.io/public.key
Rancher RKE2 1.34 (v1.34)                                                                                                                                      235  B/s | 3.6 kB     00:15
Package rke2-server-1.33.8~rke2r1-0.el9.aarch64 is already installed.
Dependencies resolved.
===============================================================================================================================================================================================
 Package                                  Architecture                         Version                                             Repository                                             Size
===============================================================================================================================================================================================
Upgrading:
 rke2-common                              aarch64                              1.34.4~rke2r1-0.el9                                 rancher-rke2-1.34-stable                               25 M
 rke2-server                              aarch64                              1.34.4~rke2r1-0.el9                                 rancher-rke2-1.34-stable                              8.3 k

Transaction Summary
===============================================================================================================================================================================================
Upgrade  2 Packages

Total download size: 25 M
Downloading Packages:
(1/2): rke2-server-1.34.4~rke2r1-0.el9.aarch64.rpm                                                                                                             1.5 kB/s | 8.3 kB     00:05
(2/2): rke2-common-1.34.4~rke2r1-0.el9.aarch64.rpm                                                                                                             3.9 MB/s |  25 MB     00:06
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                          3.9 MB/s |  25 MB     00:06
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                       1/1
  Upgrading        : rke2-common-1.34.4~rke2r1-0.el9.aarch64                                                                                                                               1/4
  Upgrading        : rke2-server-1.34.4~rke2r1-0.el9.aarch64                                                                                                                               2/4
  Running scriptlet: rke2-server-1.34.4~rke2r1-0.el9.aarch64                                                                                                                               2/4
  Running scriptlet: rke2-server-1.33.8~rke2r1-0.el9.aarch64                                                                                                                               3/4
  Cleanup          : rke2-server-1.33.8~rke2r1-0.el9.aarch64                                                                                                                               3/4
  Running scriptlet: rke2-server-1.33.8~rke2r1-0.el9.aarch64                                                                                                                               3/4
  Running scriptlet: rke2-common-1.33.8~rke2r1-0.el9.aarch64                                                                                                                               4/4
  Cleanup          : rke2-common-1.33.8~rke2r1-0.el9.aarch64                                                                                                                               4/4
  Running scriptlet: rke2-common-1.33.8~rke2r1-0.el9.aarch64                                                                                                                               4/4
  Verifying        : rke2-common-1.34.4~rke2r1-0.el9.aarch64                                                                                                                               1/4
  Verifying        : rke2-common-1.33.8~rke2r1-0.el9.aarch64                                                                                                                               2/4
  Verifying        : rke2-server-1.34.4~rke2r1-0.el9.aarch64                                                                                                                               3/4
  Verifying        : rke2-server-1.33.8~rke2r1-0.el9.aarch64                                                                                                                               4/4

Upgraded:
  rke2-common-1.34.4~rke2r1-0.el9.aarch64                                                        rke2-server-1.34.4~rke2r1-0.el9.aarch64

Complete!


[root@k8s-node1 ~]# rke2 --version
rke2 version v1.34.4+rke2r1 (c6b97dc03cefec17e8454a6f45b29f4e3d0a81d6)
go version go1.24.12 X:boringcrypto

# 위 스크립트 설치 과정만으로 아래 처럼 파드들이 신규 재생성되었음
## 첫번쨰(etcd, apiserver, kube-proxy) -> 두번째(scheduler, kcm)
[root@k8s-node1 ~]# kubectl get pod -n kube-system --sort-by=.metadata.creationTimestamp | tac
helm-install-rke2-canal-9bdxv                0/1     Completed   0          54s
helm-install-rke2-coredns-nsmb5              0/1     Completed   0          54s
helm-install-rke2-metrics-server-ltsxc       0/1     Completed   0          54s
helm-install-rke2-runtimeclasses-98gw5       0/1     Completed   0          54s
kube-scheduler-k8s-node1                     1/1     Running     0          71s
kube-controller-manager-k8s-node1            1/1     Running     0          73s
kube-proxy-k8s-node1                         1/1     Running     0          102s
kube-apiserver-k8s-node1                     1/1     Running     0          102s
etcd-k8s-node1                               1/1     Running     0          102s
kube-proxy-k8s-node2                         1/1     Running     0          13m
rke2-canal-522dn                             2/2     Running     0          13m
rke2-metrics-server-fdcdf575d-vxcnw          1/1     Running     0          31m
rke2-canal-9wqbn                             2/2     Running     0          32m
rke2-coredns-rke2-coredns-559595db99-lv2ww   1/1     Running     0          32m
NAME                                         READY   STATUS      RESTARTS   AGE

# repo 추가 및 기존 repo 삭제 확인
[root@k8s-node1 ~]# dnf repolist
repo id                                                                                        repo name
appstream                                                                                      Rocky Linux 9 - AppStream
baseos                                                                                         Rocky Linux 9 - BaseOS
extras                                                                                         Rocky Linux 9 - Extras
rancher-rke2-1.34-stable                                                                       Rancher RKE2 1.34 (v1.34)
rancher-rke2-common-stable                                                                     Rancher RKE2 Common (v1.34)

# kube-system 파드 별 컨테이너 이미지 정보 출력
[root@k8s-node1 ~]# kubectl get pods -n kube-system \
  -o custom-columns=\
POD:.metadata.name,\
CONTAINERS:.spec.containers[*].name,\
IMAGES:.spec.containers[*].image
POD                                          CONTAINERS                 IMAGES
etcd-k8s-node1                               etcd                       index.docker.io/rancher/hardened-etcd:v3.6.7-k3s1-build20260126
helm-install-rke2-canal-9bdxv                helm                       rancher/klipper-helm:v0.9.14-build20260210
helm-install-rke2-coredns-nsmb5              helm                       rancher/klipper-helm:v0.9.14-build20260210
helm-install-rke2-metrics-server-ltsxc       helm                       rancher/klipper-helm:v0.9.14-build20260210
helm-install-rke2-runtimeclasses-98gw5       helm                       rancher/klipper-helm:v0.9.14-build20260210
kube-apiserver-k8s-node1                     kube-apiserver             index.docker.io/rancher/hardened-kubernetes:v1.34.4-rke2r1-build20260210
kube-controller-manager-k8s-node1            kube-controller-manager    index.docker.io/rancher/hardened-kubernetes:v1.34.4-rke2r1-build20260210
kube-proxy-k8s-node1                         kube-proxy                 index.docker.io/rancher/hardened-kubernetes:v1.34.4-rke2r1-build20260210
kube-proxy-k8s-node2                         kube-proxy                 index.docker.io/rancher/hardened-kubernetes:v1.33.8-rke2r1-build20260210
kube-scheduler-k8s-node1                     kube-scheduler             index.docker.io/rancher/hardened-kubernetes:v1.34.4-rke2r1-build20260210
rke2-canal-522dn                             calico-node,kube-flannel   rancher/hardened-calico:v3.31.3-build20260206,rancher/hardened-flannel:v0.28.1-build20260206
rke2-canal-9wqbn                             calico-node,kube-flannel   rancher/hardened-calico:v3.31.3-build20260206,rancher/hardened-flannel:v0.28.1-build20260206
rke2-coredns-rke2-coredns-559595db99-lv2ww   coredns                    rancher/hardened-coredns:v1.14.1-build20260206
rke2-metrics-server-fdcdf575d-vxcnw          metrics-server             rancher/hardened-k8s-metrics-server:v0.8.1-build20260206

# 노드 정보 확인
[root@k8s-node1 ~]# kubectl get node -owide
NAME        STATUS   ROLES                       AGE   VERSION          INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                      KERNEL-VERSION                  CONTAINER-RUNTIME
k8s-node1   Ready    control-plane,etcd,master   35m   v1.34.4+rke2r1   192.168.10.11   <none>        Rocky Linux 9.6 (Blue Onyx)   5.14.0-570.52.1.el9_6.aarch64   containerd://2.1.5-k3s1
k8s-node2   Ready    <none>                      16m   v1.33.8+rke2r1   192.168.10.12   <none>        Rocky Linux 9.6 (Blue Onyx)   5.14.0-570.52.1.el9_6.aarch64   containerd://2.1.5-k3s1


---
[k8s-node2]
[root@k8s-node2 ~]# rke2 --version
rke2 version v1.33.8+rke2r1 (eb75e3c1774cee5a584259d6fee77eb8cfa9b430)
go version go1.24.12 X:boringcrypto

[root@k8s-node2 ~]# curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE=agent INSTALL_RKE2_CHANNEL=v1.34 sh -
[INFO]  using stable RPM repositories
[INFO]  using 1.34 series from channel stable
Importing GPG key 0xE257814A:
 Userid     : "Rancher (CI) <ci@rancher.com>"
 Fingerprint: C8CF F216 4551 26E9 B9C9 18BE 925E A29A E257 814A
 From       : https://rpm.rancher.io/public.key
Error: Failed to download metadata for repo 'rancher-rke2-1.34-stable': repomd.xml GPG signature verification error: Bad GPG signature
Importing GPG key 0xE257814A:
 Userid     : "Rancher (CI) <ci@rancher.com>"
 Fingerprint: C8CF F216 4551 26E9 B9C9 18BE 925E A29A E257 814A
 From       : https://rpm.rancher.io/public.key
Error: Failed to download metadata for repo 'rancher-rke2-1.34-stable': repomd.xml GPG signature verification error: Bad GPG signature
Rancher RKE2 1.34 (v1.34)                                                                                                                                     63  B/s | 659  B     00:10
Rancher RKE2 1.34 (v1.34)                                                                                                                                    472  B/s | 2.4 kB     00:05
Importing GPG key 0xE257814A:
 Userid     : "Rancher (CI) <ci@rancher.com>"
 Fingerprint: C8CF F216 4551 26E9 B9C9 18BE 925E A29A E257 814A
 From       : https://rpm.rancher.io/public.key
Rancher RKE2 1.34 (v1.34)                                                                                                                                    239  B/s | 3.6 kB     00:15
Package rke2-agent-1.33.8~rke2r1-0.el9.aarch64 is already installed.
Dependencies resolved.
=============================================================================================================================================================================================
 Package                                  Architecture                         Version                                           Repository                                             Size
=============================================================================================================================================================================================
Upgrading:
 rke2-agent                               aarch64                              1.34.4~rke2r1-0.el9                               rancher-rke2-1.34-stable                              8.3 k
 rke2-common                              aarch64                              1.34.4~rke2r1-0.el9                               rancher-rke2-1.34-stable                               25 M

Transaction Summary
=============================================================================================================================================================================================
Upgrade  2 Packages

Total download size: 25 M
Downloading Packages:
(1/2): rke2-agent-1.34.4~rke2r1-0.el9.aarch64.rpm                                                                                                            1.4 kB/s | 8.3 kB     00:05
(2/2): rke2-common-1.34.4~rke2r1-0.el9.aarch64.rpm                                                                                                           4.0 MB/s |  25 MB     00:06
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                        4.0 MB/s |  25 MB     00:06
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                     1/1
  Upgrading        : rke2-common-1.34.4~rke2r1-0.el9.aarch64                                                                                                                             1/4
  Upgrading        : rke2-agent-1.34.4~rke2r1-0.el9.aarch64                                                                                                                              2/4
  Running scriptlet: rke2-agent-1.34.4~rke2r1-0.el9.aarch64                                                                                                                              2/4
  Running scriptlet: rke2-agent-1.33.8~rke2r1-0.el9.aarch64                                                                                                                              3/4
  Cleanup          : rke2-agent-1.33.8~rke2r1-0.el9.aarch64                                                                                                                              3/4
  Running scriptlet: rke2-agent-1.33.8~rke2r1-0.el9.aarch64                                                                                                                              3/4
  Running scriptlet: rke2-common-1.33.8~rke2r1-0.el9.aarch64                                                                                                                             4/4
  Cleanup          : rke2-common-1.33.8~rke2r1-0.el9.aarch64                                                                                                                             4/4
  Running scriptlet: rke2-common-1.33.8~rke2r1-0.el9.aarch64                                                                                                                             4/4
  Verifying        : rke2-agent-1.34.4~rke2r1-0.el9.aarch64                                                                                                                              1/4
  Verifying        : rke2-agent-1.33.8~rke2r1-0.el9.aarch64                                                                                                                              2/4
  Verifying        : rke2-common-1.34.4~rke2r1-0.el9.aarch64                                                                                                                             3/4
  Verifying        : rke2-common-1.33.8~rke2r1-0.el9.aarch64                                                                                                                             4/4

Upgraded:
  rke2-agent-1.34.4~rke2r1-0.el9.aarch64                                                       rke2-common-1.34.4~rke2r1-0.el9.aarch64

Complete!

[root@k8s-node2 ~]# rke2 --version
rke2 version v1.34.4+rke2r1 (c6b97dc03cefec17e8454a6f45b29f4e3d0a81d6)
go version go1.24.12 X:boringcrypto
[root@k8s-node2 ~]# dnf repolist
repo id                                                                                       repo name
appstream                                                                                     Rocky Linux 9 - AppStream
baseos                                                                                        Rocky Linux 9 - BaseOS
extras                                                                                        Rocky Linux 9 - Extras
rancher-rke2-1.34-stable                                                                      Rancher RKE2 1.34 (v1.34)
rancher-rke2-common-stable                                                                    Rancher RKE2 Common (v1.34)

[root@k8s-node2 ~]# systemctl restart rke2-agent

---
[k8s-node1]

[root@k8s-node1 ~]# kubectl get node -owide
NAME        STATUS   ROLES                       AGE   VERSION          INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                      KERNEL-VERSION                  CONTAINER-RUNTIME
k8s-node1   Ready    control-plane,etcd,master   40m   v1.34.4+rke2r1   192.168.10.11   <none>        Rocky Linux 9.6 (Blue Onyx)   5.14.0-570.52.1.el9_6.aarch64   containerd://2.1.5-k3s1
k8s-node2   Ready    <none>                      21m   v1.34.4+rke2r1   192.168.10.12   <none>        Rocky Linux 9.6 (Blue Onyx)   5.14.0-570.52.1.el9_6.aarch64   containerd://2.1.5-k3s1

# 워커 노드에 kube-proxy 파드만 신규 재기동됨!
[root@k8s-node1 ~]# kubectl get pod -n kube-system --sort-by=.metadata.creationTimestamp | tac
kube-proxy-k8s-node2                         1/1     Running     0          50s
kube-proxy-k8s-node1                         1/1     Running     0          5m49s
helm-install-rke2-canal-9bdxv                0/1     Completed   0          8m22s
helm-install-rke2-coredns-nsmb5              0/1     Completed   0          8m22s
helm-install-rke2-metrics-server-ltsxc       0/1     Completed   0          8m22s
helm-install-rke2-runtimeclasses-98gw5       0/1     Completed   0          8m22s
kube-scheduler-k8s-node1                     1/1     Running     0          8m39s
kube-controller-manager-k8s-node1            1/1     Running     0          8m41s
kube-apiserver-k8s-node1                     1/1     Running     0          9m10s
etcd-k8s-node1                               1/1     Running     0          9m10s
rke2-canal-522dn                             2/2     Running     0          21m
rke2-metrics-server-fdcdf575d-vxcnw          1/1     Running     0          38m
rke2-canal-9wqbn                             2/2     Running     0          39m
rke2-coredns-rke2-coredns-559595db99-lv2ww   1/1     Running     0          39m
NAME                                         READY   STATUS      RESTARTS   AGE

자동 업그레이드 v1.34 -> v1.35

# system-upgrade-controller 설치
[root@k8s-node1 ~]# kubectl apply -f https://github.com/rancher/system-upgrade-controller/releases/latest/download/crd.yaml -f https://github.com/rancher/system-upgrade-controller/releases/latest/download/system-upgrade-controller.yaml
customresourcedefinition.apiextensions.k8s.io/plans.upgrade.cattle.io created
namespace/system-upgrade created
serviceaccount/system-upgrade created
role.rbac.authorization.k8s.io/system-upgrade-controller created
clusterrole.rbac.authorization.k8s.io/system-upgrade-controller created
clusterrole.rbac.authorization.k8s.io/system-upgrade-controller-drainer created
rolebinding.rbac.authorization.k8s.io/system-upgrade created
clusterrolebinding.rbac.authorization.k8s.io/system-upgrade created
clusterrolebinding.rbac.authorization.k8s.io/system-upgrade-drainer created
configmap/default-controller-env created
deployment.apps/system-upgrade-controller created


# 확인
[root@k8s-node1 ~]# kubectl get deploy,pod,cm -n system-upgrade
NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/system-upgrade-controller   1/1     1            1           82s

NAME                                             READY   STATUS    RESTARTS   AGE
pod/system-upgrade-controller-5f667989c7-kdsgf   1/1     Running   0          82s

NAME                               DATA   AGE
configmap/default-controller-env   11     82s
configmap/kube-root-ca.crt         1      82s

# 계획 작성 후 실행 및 확인
# plan 작성 및 실행
[root@k8s-node1 ~]# cat << EOF | kubectl apply -f -
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: server-plan
  namespace: system-upgrade
spec:
  concurrency: 1
  cordon: true
  nodeSelector:
    matchExpressions:
    - key: node-role.kubernetes.io/control-plane
      operator: In
      values:
      - "true"
  serviceAccountName: system-upgrade
  upgrade:
    image: rancher/rke2-upgrade
EOFhannel: https://update.rke2.io/v1-release/channels/latest  # version: v1.35.0+rke2r3 , curl -s https://update.rke2.io/v1-release/channels | jq .data
plan.upgrade.cattle.io/server-plan created
plan.upgrade.cattle.io/agent-plan created

[root@k8s-node1 ~]# kubectl get node -owide
NAME        STATUS   ROLES                       AGE   VERSION          INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                      KERNEL-VERSION                  CONTAINER-RUNTIME
k8s-node1   Ready    control-plane,etcd,master   54m   v1.35.1+rke2r1   192.168.10.11   <none>        Rocky Linux 9.6 (Blue Onyx)   5.14.0-570.52.1.el9_6.aarch64   containerd://2.1.5-k3s1
k8s-node2   Ready    <none>                      34m   v1.35.1+rke2r1   192.168.10.12   <none>        Rocky Linux 9.6 (Blue Onyx)   5.14.0-570.52.1.el9_6.aarch64   containerd://2.1.5-k3s1

Cluster API 소개

Cluster API(CAPI)는 Kubernetes 클러스터 자체를 Kubernetes 리소스로 관리하기 위한 프로젝트입니다.
Kubernetes SIG Cluster Lifecycle에서 개발했으며, 클러스터 생성·업그레이드·삭제 같은 라이프사이클을 선언적으로 관리할 수 있게 합니다.

일반적인 애플리케이션을 Pod로 관리하듯, Cluster API는 Cluster, Machine, MachineDeployment 같은 CRD를 통해 쿠버네티스 클러스터를 코드로 관리합니다. 관리 클러스터(Management Cluster)에서 다른 워크로드 클러스터를 생성하고 제어하는 구조가 기본 개념입니다. AWS, Azure, vSphere 등 다양한 인프라에 대해 Provider 구조를 제공해 멀티 클라우드 환경에서도 일관된 방식으로 클러스터를 운영할 수 있습니다.

Cluster API 실습

kind k8s에 관리용 management 클러스터 설치 init

> kind create cluster --name myk8s --image kindest/node:v1.35.0 --config - <<EOF                                                            07:28:04
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraMounts:
  - hostPath: /var/run/docker.sock
    containerPath: /var/run/docker.sock
  extraPortMappings:
  # sample app
  - containerPort: 30000
    hostPort: 30000
  # kube-ops-view
  - containerPort: 30001
    hostPort: 30001
EOF
Creating cluster "myk8s" ...
 ✓ Ensuring node image (kindest/node:v1.35.0)  
 ✓ Preparing nodes  
 ✓ Writing configuration  
 ✓ Starting control-plane  ️
 ✓ Installing CNI  
 ✓ Installing StorageClass  
Set kubectl context to "kind-myk8s"
You can now use your cluster with:

kubectl cluster-info --context kind-myk8s

Thanks for using kind!  


# (옵션) kube-ops-view
> helm repo add geek-cookbook https://geek-cookbook.github.io/charts/                                                          ○ kind-myk8s 07:29:08
helm install kube-ops-view geek-cookbook/kube-ops-view --version 1.2.2 \
  --set service.main.type=NodePort,service.main.ports.http.nodePort=30001 \
  --set env.TZ="Asia/Seoul" --namespace kube-system
"geek-cookbook" already exists with the same configuration, skipping
NAME: kube-ops-view
LAST DEPLOYED: Mon Feb 23 07:29:11 2026
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
  export NODE_PORT=$(kubectl get --namespace kube-system -o jsonpath="{.spec.ports[0].nodePort}" services kube-ops-view)
  export NODE_IP=$(kubectl get nodes --namespace kube-system -o jsonpath="{.items[0].status.addresses[0].address}")
  echo http://$NODE_IP:$NODE_PORT
  
> brew install clusterctl  

> clusterctl version -o json | jq                                                                                                       14s 07:29:41

{
  "clusterctl": {
    "major": "1",
    "minor": "12",
    "gitVersion": "v1.12.3",
    "gitCommit": "Homebrew",
    "gitTreeState": "clean",
    "buildDate": "2026-02-17T10:43:04Z",
    "goVersion": "go1.26.0",
    "compiler": "gc",
    "platform": "darwin/arm64"
  }
}


# [Docker 프로바이더] Initialize the management cluster : 현재 k8s 를 관리 클러스터로 변환
## Docker 프로바이더는 프로덕션 환경에 사용하도록 설계되지 않았으며 개발 환경 전용
## ClusterTopology관리형 토폴로지 및 ClusterClass 지원을 활성화하는 데 필요한 기능은 다음과 같이 활성화
## https://cluster-api.sigs.k8s.io/tasks/experimental-features/experimental-features
> export CLUSTER_TOPOLOGY=true
> clusterctl init --infrastructure docker
Fetching providers
Installing cert-manager version="v1.19.3"
Waiting for cert-manager to be available...
spec.privateKey.rotationPolicy: In cert-manager >= v1.18.0, the default value changed from `Never` to `Always`.
Installing provider="cluster-api" version="v1.12.3" targetNamespace="capi-system"
spec.privateKey.rotationPolicy: In cert-manager >= v1.18.0, the default value changed from `Never` to `Always`.
Installing provider="bootstrap-kubeadm" version="v1.12.3" targetNamespace="capi-kubeadm-bootstrap-system"
spec.privateKey.rotationPolicy: In cert-manager >= v1.18.0, the default value changed from `Never` to `Always`.
Installing provider="control-plane-kubeadm" version="v1.12.3" targetNamespace="capi-kubeadm-control-plane-system"
spec.privateKey.rotationPolicy: In cert-manager >= v1.18.0, the default value changed from `Never` to `Always`.
Installing provider="infrastructure-docker" version="v1.12.3" targetNamespace="capd-system"
spec.privateKey.rotationPolicy: In cert-manager >= v1.18.0, the default value changed from `Never` to `Always`.

Your management cluster has been initialized successfully!

You can now create your first workload cluster by running the following:

  clusterctl generate cluster [name] --kubernetes-version [version] | kubectl apply -f -
  
> kubectl get crd                                                                                                          29s ○ kind-myk8s 07:30:55

NAME                                                         CREATED AT
certificaterequests.cert-manager.io                          2026-02-22T22:30:33Z
certificates.cert-manager.io                                 2026-02-22T22:30:33Z
challenges.acme.cert-manager.io                              2026-02-22T22:30:33Z
clusterclasses.cluster.x-k8s.io                              2026-02-22T22:30:51Z
clusterissuers.cert-manager.io                               2026-02-22T22:30:33Z
clusterresourcesetbindings.addons.cluster.x-k8s.io           2026-02-22T22:30:51Z
clusterresourcesets.addons.cluster.x-k8s.io                  2026-02-22T22:30:51Z
clusters.cluster.x-k8s.io                                    2026-02-22T22:30:51Z
devclusters.infrastructure.cluster.x-k8s.io                  2026-02-22T22:30:54Z
devclustertemplates.infrastructure.cluster.x-k8s.io          2026-02-22T22:30:54Z
devmachines.infrastructure.cluster.x-k8s.io                  2026-02-22T22:30:55Z
devmachinetemplates.infrastructure.cluster.x-k8s.io          2026-02-22T22:30:55Z
dockerclusters.infrastructure.cluster.x-k8s.io               2026-02-22T22:30:55Z
dockerclustertemplates.infrastructure.cluster.x-k8s.io       2026-02-22T22:30:55Z
dockermachinepools.infrastructure.cluster.x-k8s.io           2026-02-22T22:30:55Z
dockermachinepooltemplates.infrastructure.cluster.x-k8s.io   2026-02-22T22:30:55Z
dockermachines.infrastructure.cluster.x-k8s.io               2026-02-22T22:30:55Z
dockermachinetemplates.infrastructure.cluster.x-k8s.io       2026-02-22T22:30:55Z
extensionconfigs.runtime.cluster.x-k8s.io                    2026-02-22T22:30:52Z
ipaddressclaims.ipam.cluster.x-k8s.io                        2026-02-22T22:30:52Z
ipaddresses.ipam.cluster.x-k8s.io                            2026-02-22T22:30:52Z
issuers.cert-manager.io                                      2026-02-22T22:30:33Z
kubeadmconfigs.bootstrap.cluster.x-k8s.io                    2026-02-22T22:30:53Z
kubeadmconfigtemplates.bootstrap.cluster.x-k8s.io            2026-02-22T22:30:53Z
kubeadmcontrolplanes.controlplane.cluster.x-k8s.io           2026-02-22T22:30:54Z
kubeadmcontrolplanetemplates.controlplane.cluster.x-k8s.io   2026-02-22T22:30:54Z
machinedeployments.cluster.x-k8s.io                          2026-02-22T22:30:52Z
machinedrainrules.cluster.x-k8s.io                           2026-02-22T22:30:52Z
machinehealthchecks.cluster.x-k8s.io                         2026-02-22T22:30:52Z
machinepools.cluster.x-k8s.io                                2026-02-22T22:30:52Z
machines.cluster.x-k8s.io                                    2026-02-22T22:30:52Z

> kubectl get pod -A                                                                                                           ○ kind-myk8s 07:31:54

NAMESPACE                           NAME                                                             READY   STATUS    RESTARTS   AGE
capd-system                         capd-controller-manager-54755cdd6-k6h72                          1/1     Running   0          60s
capi-kubeadm-bootstrap-system       capi-kubeadm-bootstrap-controller-manager-94d8964d9-vh5qz        1/1     Running   0          62s
capi-kubeadm-control-plane-system   capi-kubeadm-control-plane-controller-manager-6796744c76-qs7zm   1/1     Running   0          61s
capi-system                         capi-controller-manager-59c5798655-hnnqx                         1/1     Running   0          63s
cert-manager                        cert-manager-845844dd8-hq627                                     1/1     Running   0          81s
cert-manager                        cert-manager-cainjector-7b5d65fbcb-qnld4                         1/1     Running   0          81s
cert-manager                        cert-manager-webhook-6fcf4cb6c-9n94f                             1/1     Running   0          81s
kube-system                         coredns-7d764666f9-5nlsf                                         1/1     Running   0          3m30s
kube-system                         coredns-7d764666f9-z6hdf                                         1/1     Running   0          3m30s
kube-system                         etcd-myk8s-control-plane                                         1/1     Running   0          3m39s
kube-system                         kindnet-npqqh                                                    1/1     Running   0          3m30s
kube-system                         kube-apiserver-myk8s-control-plane                               1/1     Running   0          3m38s
kube-system                         kube-controller-manager-myk8s-control-plane                      1/1     Running   0          3m38s
kube-system                         kube-ops-view-5c64986f74-twskv                                   1/1     Running   0          2m44s
kube-system                         kube-proxy-tt8bp                                                 1/1     Running   0          3m30s
kube-system                         kube-scheduler-myk8s-control-plane                               1/1     Running   0          3m38s
local-path-storage                  local-path-provisioner-67b8995b4b-vcdgh                          1/1     Running   0          3m30s

관리용 management k8s 클러스터 정보 확인

## capd-system, capi-(kueadm-X/Y, system), cert-manager 네임스페이스가 생성
> kubectl describe -n capi-system deployment.apps/capi-controller-manager | grep feature-gates                                 ○ kind-myk8s 07:31:55

      --feature-gates=MachinePool=true,ClusterTopology=true,RuntimeSDK=false,MachineSetPreflightChecks=true,MachineWaitForVolumeDetachConsiderVolumeAttachments=true,PriorityQueue=false,ReconcilerRateLimiting=false,InPlaceUpdates=false,MachineTaintPropagation=false

# 프로바이더 (타입별)확인 : CAPI 구성요소가 설치된 상태
> kubectl get providers.clusterctl.cluster.x-k8s.io -A                                                                         ○ kind-myk8s 07:33:08

NAMESPACE                           NAME                    AGE     TYPE                     PROVIDER      VERSION
capd-system                         infrastructure-docker   2m14s   InfrastructureProvider   docker        v1.12.3
capi-kubeadm-bootstrap-system       bootstrap-kubeadm       2m16s   BootstrapProvider        kubeadm       v1.12.3
capi-kubeadm-control-plane-system   control-plane-kubeadm   2m15s   ControlPlaneProvider     kubeadm       v1.12.3
capi-system                         cluster-api             2m16s   CoreProvider             cluster-api   v1.12.3


# CAPI의 핵심 컨트롤러 집합 : Cluster / MachineDeployment / MachineSet / Machine CRD 관리, 전체 reconcile orchestration 담당
> kubectl get providers -n capi-system cluster-api -o yaml                                                                     ○ kind-myk8s 07:33:19

apiVersion: clusterctl.cluster.x-k8s.io/v1alpha3
kind: Provider
metadata:
  creationTimestamp: "2026-02-22T22:30:53Z"
  generation: 1
  labels:
    cluster.x-k8s.io/provider: cluster-api
    clusterctl.cluster.x-k8s.io: ""
    clusterctl.cluster.x-k8s.io/core: inventory
  name: cluster-api
  namespace: capi-system
  resourceVersion: "950"
  uid: 60d648d5-ff16-4364-a65a-3c229685137d
providerName: cluster-api
type: CoreProvider
version: v1.12.3


# 노드를 Kubernetes로 부팅시키는 역할 : cloud-init user-data 생성, kubeadm join/init config 생성
> kubectl get providers -n capi-kubeadm-bootstrap-system bootstrap-kubeadm -o yaml                                             ○ kind-myk8s 07:34:00

apiVersion: clusterctl.cluster.x-k8s.io/v1alpha3
kind: Provider
metadata:
  creationTimestamp: "2026-02-22T22:30:53Z"
  generation: 1
  labels:
    cluster.x-k8s.io/provider: bootstrap-kubeadm
    clusterctl.cluster.x-k8s.io: ""
    clusterctl.cluster.x-k8s.io/core: inventory
  name: bootstrap-kubeadm
  namespace: capi-kubeadm-bootstrap-system
  resourceVersion: "1021"
  uid: 34a3a4da-2b1d-48d4-bd42-f5d63642a355
providerName: kubeadm
type: BootstrapProvider
version: v1.12.3

# Control Plane 전용 Machine 관리 : KubeadmControlPlane 리소스 관리, Control Plane 노드 스케일링, etcd 포함 업그레이드 관리
> kubectl get providers -n capi-kubeadm-control-plane-system control-plane-kubeadm -o yaml                                     ○ kind-myk8s 07:34:16

apiVersion: clusterctl.cluster.x-k8s.io/v1alpha3
kind: Provider
metadata:
  creationTimestamp: "2026-02-22T22:30:54Z"
  generation: 1
  labels:
    cluster.x-k8s.io/provider: control-plane-kubeadm
    clusterctl.cluster.x-k8s.io: ""
    clusterctl.cluster.x-k8s.io/core: inventory
  name: control-plane-kubeadm
  namespace: capi-kubeadm-control-plane-system
  resourceVersion: "1080"
  uid: 5488efb2-f513-48f5-94c6-f0c028e636ae
providerName: kubeadm
type: ControlPlaneProvider
version: v1.12.3

# 실제 인프라 리소스 생성 담당 : 실제 Docker 컨테이너를 VM처럼 생성, Dev/Test 용도 (CAPD)
> kubectl get providers -n capd-system infrastructure-docker -o yaml                                                           ○ kind-myk8s 07:34:35

apiVersion: clusterctl.cluster.x-k8s.io/v1alpha3
kind: Provider
metadata:
  creationTimestamp: "2026-02-22T22:30:55Z"
  generation: 1
  labels:
    cluster.x-k8s.io/provider: infrastructure-docker
    clusterctl.cluster.x-k8s.io: ""
    clusterctl.cluster.x-k8s.io/core: inventory
  name: infrastructure-docker
  namespace: capd-system
  resourceVersion: "1176"
  uid: a9f1e357-a3cd-4858-9ed8-f89f453f35ee
providerName: docker
type: InfrastructureProvider
version: v1.12.3

cert manager 확인

> kubectl get crd | grep cert                                                                                                  ○ kind-myk8s 07:35:01

certificaterequests.cert-manager.io                          2026-02-22T22:30:33Z
certificates.cert-manager.io                                 2026-02-22T22:30:33Z
challenges.acme.cert-manager.io                              2026-02-22T22:30:33Z
clusterissuers.cert-manager.io                               2026-02-22T22:30:33Z
issuers.cert-manager.io                                      2026-02-22T22:30:33Z
orders.acme.cert-manager.io                                  2026-02-22T22:30:33Z


> kubectl get deploy,pod,svc,ep,cm,secret,sa -n cert-manager                                                                   ○ kind-myk8s 07:35:12

Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/cert-manager              1/1     1            1           4m39s
deployment.apps/cert-manager-cainjector   1/1     1            1           4m39s
deployment.apps/cert-manager-webhook      1/1     1            1           4m39s

NAME                                           READY   STATUS    RESTARTS   AGE
pod/cert-manager-845844dd8-hq627               1/1     Running   0          4m39s
pod/cert-manager-cainjector-7b5d65fbcb-qnld4   1/1     Running   0          4m39s
pod/cert-manager-webhook-6fcf4cb6c-9n94f       1/1     Running   0          4m39s

NAME                              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)            AGE
service/cert-manager              ClusterIP   10.96.228.252   <none>        9402/TCP           4m39s
service/cert-manager-cainjector   ClusterIP   10.96.50.51     <none>        9402/TCP           4m39s
service/cert-manager-webhook      ClusterIP   10.96.176.129   <none>        443/TCP,9402/TCP   4m39s

NAME                                ENDPOINTS                          AGE
endpoints/cert-manager              10.244.0.7:9402                    4m39s
endpoints/cert-manager-cainjector   10.244.0.6:9402                    4m39s
endpoints/cert-manager-webhook      10.244.0.8:10250,10.244.0.8:9402   4m39s

NAME                         DATA   AGE
configmap/kube-root-ca.crt   1      4m40s

NAME                             TYPE     DATA   AGE
secret/cert-manager-webhook-ca   Opaque   3      4m29s

NAME                                     AGE
serviceaccount/cert-manager              4m40s
serviceaccount/cert-manager-cainjector   4m40s
serviceaccount/cert-manager-webhook      4m40s
serviceaccount/default                   4m40s

> kubectl get issuers.cert-manager.io -A                                                                                       ○ kind-myk8s 07:35:31

NAMESPACE                           NAME                                           READY   AGE
capd-system                         capd-selfsigned-issuer                         True    4m38s
capi-kubeadm-bootstrap-system       capi-kubeadm-bootstrap-selfsigned-issuer       True    4m39s
capi-kubeadm-control-plane-system   capi-kubeadm-control-plane-selfsigned-issuer   True    4m39s
capi-system                         capi-selfsigned-issuer                         True    4m41s

> kubectl get certificaterequests.cert-manager.io -A -owide                                                                    ○ kind-myk8s 07:35:44

NAMESPACE                           NAME                                        APPROVED   DENIED   READY   ISSUER                                         REQUESTER                                         STATUS                                         AGE
capd-system                         capd-serving-cert-1                         True                True    capd-selfsigned-issuer                         system:serviceaccount:cert-manager:cert-manager   Certificate fetched from issuer successfully   4m49s
capi-kubeadm-bootstrap-system       capi-kubeadm-bootstrap-serving-cert-1       True                True    capi-kubeadm-bootstrap-selfsigned-issuer       system:serviceaccount:cert-manager:cert-manager   Certificate fetched from issuer successfully   4m51s
capi-kubeadm-control-plane-system   capi-kubeadm-control-plane-serving-cert-1   True                True    capi-kubeadm-control-plane-selfsigned-issuer   system:serviceaccount:cert-manager:cert-manager   Certificate fetched from issuer successfully   4m50s
capi-system                         capi-serving-cert-1                         True                True    capi-selfsigned-issuer                         system:serviceaccount:cert-manager:cert-manager   Certificate fetched from issuer successfully   4m52s

> kubectl get certificates.cert-manager.io -A -owide                                                                           ○ kind-myk8s 07:36:05

NAMESPACE                           NAME                                      READY   SECRET                                            ISSUER                                         STATUS                                          AGE
capd-system                         capd-serving-cert                         True    capd-webhook-service-cert                         capd-selfsigned-issuer                         Certificate is up to date and has not expired   5m12s
capi-kubeadm-bootstrap-system       capi-kubeadm-bootstrap-serving-cert       True    capi-kubeadm-bootstrap-webhook-service-cert       capi-kubeadm-bootstrap-selfsigned-issuer       Certificate is up to date and has not expired   5m13s
capi-kubeadm-control-plane-system   capi-kubeadm-control-plane-serving-cert   True    capi-kubeadm-control-plane-webhook-service-cert   capi-kubeadm-control-plane-selfsigned-issuer   Certificate is up to date and has not expired   5m13s
capi-system                         capi-serving-cert                         True    capi-webhook-service-cert                         capi-selfsigned-issuer                         Certificate is up to date and has not expired   5m15s

첫 번째 워크로드 클러스터 생성 및 확인

# 첫 번째 워크로드 구성을 위한 환경 변수 설정 : 필요에 맞게 수정.
## The list of service CIDR, default ["10.128.0.0/12"]
> export SERVICE_CIDR=["10.20.0.0/16"]

## The list of pod CIDR, default ["192.168.0.0/16"]
> export POD_CIDR=["10.10.0.0/16"]

## The service domain, default "cluster.local"
> export SERVICE_DOMAIN="myk8s-1.local"

## PSS Disable
> export POD_SECURITY_STANDARD_ENABLED="false"

> clusterctl generate cluster capi-quickstart --flavor development \
  --kubernetes-version v1.34.3 \
  --control-plane-machine-count=3 \
  --worker-machine-count=3 \
  > capi-quickstart.yaml
  
> cat capi-quickstart.yaml | grep -E '^apiVersion:|^kind:'                                                                                  07:37:12
apiVersion: cluster.x-k8s.io/v1beta2
kind: ClusterClass
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: DockerClusterTemplate
apiVersion: controlplane.cluster.x-k8s.io/v1beta2
kind: KubeadmControlPlaneTemplate
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: DockerMachineTemplate
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: DockerMachineTemplate
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: DockerMachinePoolTemplate
apiVersion: bootstrap.cluster.x-k8s.io/v1beta2
kind: KubeadmConfigTemplate
apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster


> kubectl apply -f capi-quickstart.yaml                                                                                        ○ kind-myk8s 07:37:40

clusterclass.cluster.x-k8s.io/quick-start created
dockerclustertemplate.infrastructure.cluster.x-k8s.io/quick-start-cluster created
kubeadmcontrolplanetemplate.controlplane.cluster.x-k8s.io/quick-start-control-plane created
dockermachinetemplate.infrastructure.cluster.x-k8s.io/quick-start-control-plane created
dockermachinetemplate.infrastructure.cluster.x-k8s.io/quick-start-default-worker-machinetemplate created
dockermachinepooltemplate.infrastructure.cluster.x-k8s.io/quick-start-default-worker-machinepooltemplate created
kubeadmconfigtemplate.bootstrap.cluster.x-k8s.io/quick-start-default-worker-bootstraptemplate created
cluster.cluster.x-k8s.io/capi-quickstart created

# 생성 확인 & kubeconfig 자격 증명 & CNI 플러그인 설치 후 확인
> kubectl get cluster -o wide                                                                                                  ○ kind-myk8s 07:38:29

NAME              CLUSTERCLASS   AVAILABLE   CP DESIRED   CP CURRENT   CP READY   CP AVAILABLE   CP UP-TO-DATE   W DESIRED   W CURRENT   W READY   W AVAILABLE   W UP-TO-DATE   PAUSED   PHASE         AGE   VERSION
capi-quickstart   quick-start    False       3            1            0          0              1               3           3           0         0             3              False    Provisioned   71s   v1.34.3


> clusterctl describe cluster capi-quickstart                                                                                               07:39:31

NAME                                                           REPLICAS AVAILABLE READY UP TO DATE STATUS REASON        SINCE MESSAGE
Cluster/capi-quickstart                                        5/6      0         0     5          False  NotAvailable  110s  * WorkersAvailable:
│                                                                                                                               * MachineDeployment capi-quickstart-md-0-z2vw7: 0 available replicas, at least 3 required
│                                                                                                                                 (spec.strategy.rollout.maxUnavailable is 0, spec.replicas is 3)
├─ClusterInfrastructure - DockerCluster/capi-quickstart-gtgmc                                      True   Ready         103s
├─ControlPlane - KubeadmControlPlane/capi-quickstart-m4s8l     2/3      0         0     2          True   Available     28s
│ └─2 Machines...                                                       0         0     2          False  NotReady      31s   See capi-quickstart-m4s8l-c5frx, capi-quickstart-m4s8l-lt9vl
└─Workers
  └─MachineDeployment/capi-quickstart-md-0-z2vw7               3/3      0         0     3          False  NotAvailable  110s  0 available replicas, at least 3 required (spec.strategy.rollout.maxUnavailable is 0, spec.replicas
    │                                                                                                                         is 3)
    └─3 Machines...                                                     0         0     3          False  NotReady      14s   See capi-quickstart-md-0-z2vw7-kv5vb-k7h7t, capi-quickstart-md-0-z2vw7-kv5vb-qkrh4, ...
    
# 워크로드 클러스터 자격증명
> clusterctl get kubeconfig capi-quickstart > capi-quickstart.kubeconfig

# 노드 NotReady 상태 해결 : CNI 플러그인 설치 하자!
> kubectl --kubeconfig=capi-quickstart.kubeconfig apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.27.0/manifests/calico.yaml

poddisruptionbudget.policy/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
serviceaccount/calico-node created
serviceaccount/calico-cni-plugin created
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgpfilters.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrole.rbac.authorization.k8s.io/calico-cni-plugin created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-cni-plugin created
daemonset.apps/calico-node created
deployment.apps/calico-kube-controllers created

> kubectl --kubeconfig=capi-quickstart.kubeconfig get nodes -owide                                                          3s ○ kind-myk8s 07:43:02

NAME                                     STATUS   ROLES           AGE     VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION                        CONTAINER-RUNTIME
capi-quickstart-m4s8l-c5frx              Ready    control-plane   4m37s   v1.34.3   192.168.228.4   <none>        Debian GNU/Linux 12 (bookworm)   6.7.11-orbstack-00143-ge6b82e26cd22   containerd://2.2.0
capi-quickstart-m4s8l-lt9vl              Ready    control-plane   3m22s   v1.34.3   192.168.228.8   <none>        Debian GNU/Linux 12 (bookworm)   6.7.11-orbstack-00143-ge6b82e26cd22   containerd://2.2.0
capi-quickstart-m4s8l-zh5xj              Ready    control-plane   2m3s    v1.34.3   192.168.228.9   <none>        Debian GNU/Linux 12 (bookworm)   6.7.11-orbstack-00143-ge6b82e26cd22   containerd://2.2.0
capi-quickstart-md-0-z2vw7-kv5vb-k7h7t   Ready    <none>          3m52s   v1.34.3   192.168.228.5   <none>        Debian GNU/Linux 12 (bookworm)   6.7.11-orbstack-00143-ge6b82e26cd22   containerd://2.2.0
capi-quickstart-md-0-z2vw7-kv5vb-qkrh4   Ready    <none>          3m53s   v1.34.3   192.168.228.7   <none>        Debian GNU/Linux 12 (bookworm)   6.7.11-orbstack-00143-ge6b82e26cd22   containerd://2.2.0
capi-quickstart-md-0-z2vw7-kv5vb-twppz   Ready    <none>          3m54s   v1.34.3   192.168.228.6   <none>        Debian GNU/Linux 12 (bookworm)   6.7.11-orbstack-00143-ge6b82e26cd22   containerd://2.2.0

[K8s Deploy] Kubespray offline 설치

안녕유지 — Sun, 15 Feb 2026 01:00:15 +0900

Cloudnet K8s Deploy 6주차 스터디를 진행하며 정리한 글입니다.

폐쇄망 환경 소개

폐쇄망에서 서비스 동작을 위해 필요한 주요 구성 요소

NTP Server ← 보통 벤더 어플라이언스 장비(or SW/솔루션)로 기본 이중화(HA)로 구성되며, 인프라 부서/팀에서 관리
DNS Server ← 보통 벤더 어플라이언스 장비(or SW/솔루션)로 기본 이중화(HA)로 구성되며, 인프라 부서/팀에서 관리
Network Gateway (IGW, NATGW) : 내부망 내 다른 네트워크에 서버와 통신 및 필요 시 DMZ/외부망과 통신 시 ← 보통 네트워크 라우터/스위치 장비로 구성되며, 네트워크팀에서 관리
Local (Mirror) YUM/DNF Repository : Linux 패키지 저장소, 예시) reposync + createrepo
Private Container (Image) Registry : 컨테이너 이미지 저장소, 예시) Docker Registry (registry), Harbor
Helm Artifact Repository (Server) : 헬름 차트 저장소, 예시) Chart Museum, OCI (Artifact) Registry
Private PyPI(Python Package Index) Mirror : Python 패키지 저장소, 예시) Devpi
Private Go Module Proxy : 사설 Go 모듈 프록시, 예시) Athens

폐쇄망 서비스 실습

출처 : https://sirzzang.github.io/kubernetes/Kubernetes-Kubespray-08-01-00/

일반적으로 DMZ 서버 → 내부 admin서버(파일 가져오기, kubespary 실행)이지만, PC Spec과 시간 단축을 위해 1대 서버로 구성

Network Gateway (IGW, NATGW)

[k8s-node] 네트워크 기본 설정 : enp0s8 연결 down, enp0s9 디폴트 라우팅 → 외부 통신 확인

# k8s-node1, k8s-node2에 모두동일하게 진행

# enp0s8 연결 내리기 : 실행 직후 부터 외부 통신 X
root@k8s-node1:~# nmcli connection down enp0s8
Connection 'enp0s8' successfully deactivated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/2)

# enp0s8 확인 : 할당된 IP가 제거되고, 외부 통신 라우팅 정보도 삭제됨 
root@k8s-node1:~# cat /etc/NetworkManager/system-connections/enp0s8.nmconnection
[connection]
id=enp0s8
uuid=7f94e839-e070-4bfe-9330-07090381d89f
type=ethernet
autoconnect=false
interface-name=enp0s8
timestamp=1771084427

[ethernet]

[ipv4]
method=auto

[ipv6]
addr-gen-mode=eui64
method=ignore

[proxy]

root@k8s-node1:~# nmcli connection modify enp0s8 connection.autoconnect no

# 외부 통신을 위해 enp0s9 에 디폴트 라우팅 추가 : 우선순위 200 설정
root@k8s-node1:~# nmcli connection modify enp0s9 +ipv4.routes "0.0.0.0/0 192.168.10.10 200"

# 설정 적용하기
root@k8s-node1:~# nmcli connection up enp0s9
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/6)
root@k8s-node1:~# ip route
default via 192.168.10.10 dev enp0s9 proto static metric 200
192.168.10.0/24 dev enp0s9 proto kernel scope link src 192.168.10.11 metric 100

# 외부 통신 확인
root@k8s-node1:~# ping -w 1 -W 1 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.

--- 8.8.8.8 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

root@k8s-node1:~# curl www.google.com
curl: (6) Could not resolve host: www.google.com

# DNS Nameserver 정보 확인
root@k8s-node1:~# cat /etc/resolv.conf
# Generated by NetworkManager
root@k8s-node1:~# cat << EOF > /etc/resolv.conf
nameserver 168.126.63.1
nameserver 8.8.8.8
EOF
root@k8s-node1:~# curl www.google.com

[admin] 라우터/NAT 설정

# 라우팅 설정 : 이미 설정 되어 있음
root@admin:~# sysctl -w net.ipv4.ip_forward=1
net.ipv4.ip_forward = 1

root@admin:~# cat <<EOF | tee /etc/sysctl.d/99-ipforward.conf
net.ipv4.ip_forward = 1
EOF
net.ipv4.ip_forward = 1

# NAT 설정
root@admin:~# iptables -t nat -A POSTROUTING -o enp0s8 -j MASQUERADE
root@admin:~# iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-A POSTROUTING -o enp0s8 -j MASQUERADE

root@admin:~# iptables -t nat -L -n -v
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 1 packets, 104 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MASQUERADE  all  --  *      enp0s8  0.0.0.0/0            0.0.0.0/0
    
    
# 설정 이후 k8s-node1에서 다시 호출 테스트 -> 성공
root@k8s-node1:~# curl www.google.com
<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="ko"><head><meta content="text/html; charset=UTF-8" http-equiv="Content-Type"><meta content="/images/branding/googleg/1x/googleg_standard_color_128dp.png"

# iptables -t nat -D POSTROUTING -o enp0s8 -j MASQUERADE (다시 iptables NAT 설정 제거로, 내부 k8s-node 인터넷 X)
root@admin:~# iptables -t nat -D POSTROUTING -o enp0s8 -j MASQUERADE

NTP Server - Client

[admin] NTP 서버 설정

# 현재 ntp 서버와 타임 동기화 설정 및 상태 확인
root@admin:~# systemctl status chronyd.service --no-pager
● chronyd.service - NTP client/server
     Loaded: loaded (/usr/lib/systemd/system/chronyd.service; enabled; preset: enabled)
     Active: active (running) since Sun 2026-02-15 00:11:31 KST; 52min ago
 Invocation: eac0b93de6794cdda0c6339584f15332
       Docs: man:chronyd(8)
             man:chrony.conf(5)
   Main PID: 651 (chronyd)
      Tasks: 1 (limit: 12336)
     Memory: 4.7M (peak: 5.2M)
        CPU: 53ms
     CGroup: /system.slice/chronyd.service
             └─651 /usr/sbin/chronyd -F 2

Feb 15 00:11:31 localhost systemd[1]: Starting chronyd.service - NTP client/server...
Feb 15 00:11:31 localhost chronyd[651]: chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCDNS +NTS +SECHASH +IPV6 +DEBUG)
Feb 15 00:11:31 localhost chronyd[651]: Frequency -3.765 +/- 4.378 ppm read from /var/lib/chrony/drift
Feb 15 00:11:31 localhost chronyd[651]: Loaded seccomp filter (level 2)
Feb 15 00:11:31 localhost systemd[1]: Started chronyd.service - NTP client/server.

root@admin:~# grep "^[^#]" /etc/chrony.conf
pool 2.rocky.pool.ntp.org iburst
sourcedir /run/chrony-dhcp
driftfile /var/lib/chrony/drift
makestep 1.0 3
rtcsync
ntsdumpdir /var/lib/chrony
logdir /var/log/chrony

# chrony가 어떤 NTP 서버들을 알고 있고, 그중 어떤 서버를 기준으로 시간을 맞추는지를 보여줍니다.
root@admin:~# chronyc sources -v

  .-- Source mode  '^' = server, '=' = peer, '#' = local clock.
 / .- Source state '*' = current best, '+' = combined, '-' = not combined,
| /             'x' = may be in error, '~' = too variable, '?' = unusable.
||                                                 .- xxxx [ yyyy ] +/- zzzz
||      Reachability register (octal) -.           |  xxxx = adjusted offset,
||      Log2(Polling interval) --.      |          |  yyyy = measured offset,
||                                \     |          |  zzzz = estimated error.
||                                 |    |           \
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================

root@admin:~# dig +short 2.rocky.pool.ntp.org
;; communications error to 0.192.249.101#53: timed out
;; communications error to 0.192.249.101#53: timed out
;; communications error to 0.192.249.101#53: timed out
;; no servers could be reached

# chrony 설정
root@admin:~# cp /etc/chrony.conf /etc/chrony.bak
root@admin:~# cat << EOF > /etc/chrony.conf
# 외부 한국 공용 NTP 서버 설정
server pool.ntp.org iburst
server kr.pool.ntp.org iburst

# 내부망(192.168.10.0/24)에서 이 서버에 접속하여 시간 동기화 허용
allow 192.168.10.0/24

# 외부망이 끊겼을 때도 로컬 시계를 기준으로 내부망에 시간 제공 (선택 사항)
local stratum 10

# 로그
logdir /var/log/chrony
EOF

root@admin:~# systemctl restart chronyd.service
root@admin:~# systemctl status chronyd.service --no-pager
● chronyd.service - NTP client/server
     Loaded: loaded (/usr/lib/systemd/system/chronyd.service; enabled; preset: enabled)
     Active: active (running) since Sun 2026-02-15 01:06:13 KST; 4s ago
 Invocation: 2a7e1f6f000d4573bb73f06a4a110466
       Docs: man:chronyd(8)
             man:chrony.conf(5)
    Process: 5248 ExecStart=/usr/sbin/chronyd $OPTIONS (code=exited, status=0/SUCCESS)
   Main PID: 5250 (chronyd)
      Tasks: 1 (limit: 12336)
     Memory: 908K (peak: 2.8M)
        CPU: 31ms
     CGroup: /system.slice/chronyd.service
             └─5250 /usr/sbin/chronyd -F 2

Feb 15 01:06:13 admin systemd[1]: Starting chronyd.service - NTP client/server...
Feb 15 01:06:13 admin chronyd[5250]: chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCDNS +NTS +SECHASH +IPV6 +DEBUG)
Feb 15 01:06:13 admin chronyd[5250]: Initial frequency -3.765 ppm
Feb 15 01:06:13 admin chronyd[5250]: Loaded seccomp filter (level 2)
Feb 15 01:06:13 admin systemd[1]: Started chronyd.service - NTP client/server.


# 상태 확인
root@admin:~# timedatectl status
               Local time: Sun 2026-02-15 01:06:33 KST
           Universal time: Sat 2026-02-14 16:06:33 UTC
                 RTC time: Sat 2026-02-14 16:06:58
                Time zone: Asia/Seoul (KST, +0900)
System clock synchronized: no
              NTP service: active
          RTC in local TZ: no
root@admin:~# chronyc sources -v

  .-- Source mode  '^' = server, '=' = peer, '#' = local clock.
 / .- Source state '*' = current best, '+' = combined, '-' = not combined,
| /             'x' = may be in error, '~' = too variable, '?' = unusable.
||                                                 .- xxxx [ yyyy ] +/- zzzz
||      Reachability register (octal) -.           |  xxxx = adjusted offset,
||      Log2(Polling interval) --.      |          |  yyyy = measured offset,
||                                \     |          |  zzzz = estimated error.
||                                 |    |           \
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================

[k8s-node] NTP 클라이언트 설정

# 상태 확인
root@k8s-node1:~# timedatectl status
               Local time: Sun 2026-02-15 01:07:25 KST
           Universal time: Sat 2026-02-14 16:07:25 UTC
                 RTC time: Sat 2026-02-14 16:07:50
                Time zone: Asia/Seoul (KST, +0900)
System clock synchronized: no
              NTP service: active
          RTC in local TZ: no
root@k8s-node1:~# chronyc sources -v

  .-- Source mode  '^' = server, '=' = peer, '#' = local clock.
 / .- Source state '*' = current best, '+' = combined, '-' = not combined,
| /             'x' = may be in error, '~' = too variable, '?' = unusable.
||                                                 .- xxxx [ yyyy ] +/- zzzz
||      Reachability register (octal) -.           |  xxxx = adjusted offset,
||      Log2(Polling interval) --.      |          |  yyyy = measured offset,
||                                \     |          |  zzzz = estimated error.
||                                 |    |           \
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================


# chrony 설정
root@k8s-node1:~# cp /etc/chrony.conf /etc/chrony.bak
root@k8s-node1:~# cat << EOF > /etc/chrony.conf
server 192.168.10.10 iburst
logdir /var/log/chrony
EOF
root@k8s-node1:~# systemctl restart chronyd.service
root@k8s-node1:~# systemctl status chronyd.service --no-pager
● chronyd.service - NTP client/server
     Loaded: loaded (/usr/lib/systemd/system/chronyd.service; enabled; preset: enabled)
     Active: active (running) since Sun 2026-02-15 01:08:08 KST; 4s ago
 Invocation: fa923efadd824d92aaccc8bf1e7f75bd
       Docs: man:chronyd(8)
             man:chrony.conf(5)
    Process: 5237 ExecStart=/usr/sbin/chronyd $OPTIONS (code=exited, status=0/SUCCESS)
   Main PID: 5240 (chronyd)
      Tasks: 1 (limit: 12336)
     Memory: 828K (peak: 2.8M)
        CPU: 36ms
     CGroup: /system.slice/chronyd.service
             └─5240 /usr/sbin/chronyd -F 2

Feb 15 01:08:08 k8s-node1 systemd[1]: Starting chronyd.service - NTP client/server...
Feb 15 01:08:08 k8s-node1 chronyd[5240]: chronyd version 4.6.1 starting (+CMDMON +NTP +…EBUG)
Feb 15 01:08:08 k8s-node1 chronyd[5240]: Initial frequency -3.765 ppm
Feb 15 01:08:08 k8s-node1 chronyd[5240]: Loaded seccomp filter (level 2)
Feb 15 01:08:08 k8s-node1 systemd[1]: Started chronyd.service - NTP client/server.
Feb 15 01:08:12 k8s-node1 chronyd[5240]: Selected source 192.168.10.10
Hint: Some lines were ellipsized, use -l to show in full.


# 상태 확인
root@k8s-node1:~# timedatectl status
               Local time: Sun 2026-02-15 01:08:33 KST
           Universal time: Sat 2026-02-14 16:08:33 UTC
                 RTC time: Sat 2026-02-14 16:08:59
                Time zone: Asia/Seoul (KST, +0900)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no
root@k8s-node1:~# chronyc sources -v

  .-- Source mode  '^' = server, '=' = peer, '#' = local clock.
 / .- Source state '*' = current best, '+' = combined, '-' = not combined,
| /             'x' = may be in error, '~' = too variable, '?' = unusable.
||                                                 .- xxxx [ yyyy ] +/- zzzz
||      Reachability register (octal) -.           |  xxxx = adjusted offset,
||      Log2(Polling interval) --.      |          |  yyyy = measured offset,
||                                \     |          |  zzzz = estimated error.
||                                 |    |           \
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* admin                        10   6    17    25    -12us[-1163us] +/-  379us


# [admin-lb] 자신의 NTP Server 를 사용하는 클라이언트 확인
root@admin:~# chronyc clients
Hostname                      NTP   Drop Int IntL Last     Cmd   Drop Int  Last
===============================================================================
k8s-node1                       5      0   4   -    11       0      0   -     -
k8s-node2                       4      0   1   -     9       0      0   -     -

DNS Server - Client

[admin] DNS 서버(bind) 설정 : NetworkManager에서 DNS 관리 끄기

# bind 설치
root@admin:~# dnf install -y bind bind-utils
Rocky Linux 10 - BaseOS                                                                                                                                      2.3 MB/s |  15 MB     00:06
Rocky Linux 10 - AppStream                                                                                                                                   413 kB/s | 2.2 MB     00:05
Rocky Linux 10 - Extras                                                                                                                                      1.2 kB/s | 6.2 kB     00:05
Package bind-utils-32:9.18.33-4.el10_0.aarch64 is already installed.
Dependencies resolved.
=============================================================================================================================================================================================
 Package                                            Architecture                       Version                                                    Repository                            Size
=============================================================================================================================================================================================
Installing:
 bind                                               aarch64                            32:9.18.33-10.el10_1.2                                     appstream                            322 k
Upgrading:
 bind-libs                                          aarch64                            32:9.18.33-10.el10_1.2                                     appstream                            1.2 M
 bind-license                                       noarch                             32:9.18.33-10.el10_1.2                                     appstream                             13 k
 bind-utils                                         aarch64                            32:9.18.33-10.el10_1.2                                     appstream                            233 k
 crypto-policies                                    noarch                             20250905-2.gitc7eb7b2.el10_1.1                             baseos                                94 k
 crypto-policies-scripts                            noarch                             20250905-2.gitc7eb7b2.el10_1.1                             baseos                               134 k
 openssl                                            aarch64                            1:3.5.1-7.el10_1                                           baseos                               1.2 M
 openssl-libs                                       aarch64                            1:3.5.1-7.el10_1                                           baseos                               2.1 M
Installing dependencies:
 openssl-fips-provider                              aarch64                            1:3.5.1-7.el10_1                                           baseos                               717 k
Installing weak dependencies:
 bind-dnssec-utils                                  aarch64                            32:9.18.33-10.el10_1.2                                     appstream                            150 k

Transaction Summary
=============================================================================================================================================================================================
Install  3 Packages
Upgrade  7 Packages

Total download size: 6.2 M
Downloading Packages:
(1/10): bind-dnssec-utils-9.18.33-10.el10_1.2.aarch64.rpm                                                                                                     65 kB/s | 150 kB     00:02
(2/10): openssl-fips-provider-3.5.1-7.el10_1.aarch64.rpm                                                                                                     301 kB/s | 717 kB     00:02
(3/10): crypto-policies-20250905-2.gitc7eb7b2.el10_1.1.noarch.rpm                                                                                            1.0 MB/s |  94 kB     00:00
(4/10): crypto-policies-scripts-20250905-2.gitc7eb7b2.el10_1.1.noarch.rpm                                                                                    2.5 MB/s | 134 kB     00:00
(5/10): bind-9.18.33-10.el10_1.2.aarch64.rpm                                                                                                                 132 kB/s | 322 kB     00:02
(6/10): openssl-3.5.1-7.el10_1.aarch64.rpm                                                                                                                   9.8 MB/s | 1.2 MB     00:00
(7/10): openssl-libs-3.5.1-7.el10_1.aarch64.rpm                                                                                                               11 MB/s | 2.1 MB     00:00
(8/10): bind-license-9.18.33-10.el10_1.2.noarch.rpm                                                                                                           81 kB/s |  13 kB     00:00
(9/10): bind-utils-9.18.33-10.el10_1.2.aarch64.rpm                                                                                                           602 kB/s | 233 kB     00:00
(10/10): bind-libs-9.18.33-10.el10_1.2.aarch64.rpm                                                                                                           1.7 MB/s | 1.2 MB     00:00
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                        984 kB/s | 6.2 MB     00:06
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                     1/1
  Running scriptlet: crypto-policies-20250905-2.gitc7eb7b2.el10_1.1.noarch                                                                                                              1/17
  Upgrading        : crypto-policies-20250905-2.gitc7eb7b2.el10_1.1.noarch                                                                                                              1/17
  Running scriptlet: crypto-policies-20250905-2.gitc7eb7b2.el10_1.1.noarch                                                                                                              1/17
  Upgrading        : openssl-libs-1:3.5.1-7.el10_1.aarch64                                                                                                                              2/17
  Installing       : openssl-fips-provider-1:3.5.1-7.el10_1.aarch64                                                                                                                     3/17
  Upgrading        : bind-license-32:9.18.33-10.el10_1.2.noarch                                                                                                                         4/17
  Upgrading        : bind-libs-32:9.18.33-10.el10_1.2.aarch64                                                                                                                           5/17
  Upgrading        : bind-utils-32:9.18.33-10.el10_1.2.aarch64                                                                                                                          6/17
  Installing       : bind-dnssec-utils-32:9.18.33-10.el10_1.2.aarch64                                                                                                                   7/17
  Running scriptlet: bind-32:9.18.33-10.el10_1.2.aarch64                                                                                                                                8/17
  Installing       : bind-32:9.18.33-10.el10_1.2.aarch64                                                                                                                                8/17
warning: /etc/named.conf created as /etc/named.conf.rpmnew

  Running scriptlet: bind-32:9.18.33-10.el10_1.2.aarch64                                                                                                                                8/17
  Upgrading        : openssl-1:3.5.1-7.el10_1.aarch64                                                                                                                                   9/17
  Upgrading        : crypto-policies-scripts-20250905-2.gitc7eb7b2.el10_1.1.noarch                                                                                                     10/17
  Cleanup          : crypto-policies-scripts-20250214-1.gitfd9b9b9.el10_0.1.noarch                                                                                                     11/17
  Cleanup          : openssl-1:3.2.2-16.el10_0.4.aarch64                                                                                                                               12/17
  Cleanup          : bind-utils-32:9.18.33-4.el10_0.aarch64                                                                                                                            13/17
  Cleanup          : bind-libs-32:9.18.33-4.el10_0.aarch64                                                                                                                             14/17
  Cleanup          : openssl-libs-1:3.2.2-16.el10_0.4.aarch64                                                                                                                          15/17
  Cleanup          : crypto-policies-20250214-1.gitfd9b9b9.el10_0.1.noarch                                                                                                             16/17
  Cleanup          : bind-license-32:9.18.33-4.el10_0.noarch                                                                                                                           17/17
  Running scriptlet: crypto-policies-scripts-20250905-2.gitc7eb7b2.el10_1.1.noarch                                                                                                     17/17
  Running scriptlet: bind-license-32:9.18.33-4.el10_0.noarch                                                                                                                           17/17

Upgraded:
  bind-libs-32:9.18.33-10.el10_1.2.aarch64                        bind-license-32:9.18.33-10.el10_1.2.noarch                              bind-utils-32:9.18.33-10.el10_1.2.aarch64
  crypto-policies-20250905-2.gitc7eb7b2.el10_1.1.noarch           crypto-policies-scripts-20250905-2.gitc7eb7b2.el10_1.1.noarch           openssl-1:3.5.1-7.el10_1.aarch64
  openssl-libs-1:3.5.1-7.el10_1.aarch64
Installed:
  bind-32:9.18.33-10.el10_1.2.aarch64                   bind-dnssec-utils-32:9.18.33-10.el10_1.2.aarch64                   openssl-fips-provider-1:3.5.1-7.el10_1.aarch64

Complete!


# named.conf 설정
root@admin:~# cat <<EOF > /etc/named.conf
options {
        listen-on port 53 { any; };
        listen-on-v6 port 53 { ::1; };
        directory       "/var/named";
        dump-file       "/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";
        secroots-file   "/var/named/data/named.secroots";
        recursing-file  "/var/named/data/named.recursing";
        allow-query     { 127.0.0.1; 192.168.10.0/24; };
        allow-recursion { 127.0.0.1; 192.168.10.0/24; };

        forwarders {
                168.126.63.1;
                8.8.8.8;
        };

        recursion yes;

EOF

# 문법 오류 확인 (아무 메시지 없으면 정상)
root@admin:~# named-checkconf /etc/named.conf

# 서비스 활성화 및 시작
root@admin:~# systemctl enable --now named
Created symlink '/etc/systemd/system/multi-user.target.wants/named.service' → '/usr/lib/systemd/system/named.service'.

# DMZ 서버 자체 DNS 설정 (자기 자신 사용)
root@admin:~# echo "nameserver 192.168.10.10" > /etc/resolv.conf

# 확인
root@admin:~# dig +short google.com @192.168.10.10
172.217.221.138
172.217.221.100
172.217.221.101
172.217.221.102
172.217.221.139
172.217.221.113

root@admin:~# dig +short google.com
172.217.221.138
172.217.221.139
172.217.221.102
172.217.221.101
172.217.221.100
172.217.221.113

# NetworkManager에서 DNS 관리 끄기 : 미적용 시, admin 서버 재부팅 시 NetworkManager 가 초기 설정값 덮어쓰움.
root@admin:~# cat /etc/NetworkManager/conf.d/99-dns-none.conf
cat: /etc/NetworkManager/conf.d/99-dns-none.conf: No such file or directory
root@admin:~# cat << EOF > /etc/NetworkManager/conf.d/99-dns-none.conf
[main]
dns=none
EOF
root@admin:~# systemctl restart NetworkManager

[k8s-node] DNS 클라이언트 설정 : NetworkManager에서 DNS 관리 끄기

# NetworkManager에서 DNS 관리 끄기
root@k8s-node1:~# cat /etc/NetworkManager/conf.d/99-dns-none.conf
cat: /etc/NetworkManager/conf.d/99-dns-none.conf: No such file or directory
root@k8s-node1:~# cat << EOF > /etc/NetworkManager/conf.d/99-dns-none.conf
[main]
dns=none
EOF
root@k8s-node1:~# systemctl restart NetworkManager

# DNS 서버 정보 설정
root@k8s-node1:~# echo "nameserver 192.168.10.10" > /etc/resolv.conf

# 확인
root@k8s-node1:~# dig +short google.com @192.168.10.10
172.217.221.113
172.217.221.139
172.217.221.102
172.217.221.101
172.217.221.100
172.217.221.138
root@k8s-node1:~# dig +short google.com
172.217.221.138
172.217.221.101
172.217.221.100
172.217.221.102
172.217.221.139
172.217.221.113

Local (Mirror) YUM/DNF Repository (kubespray offline에 중복 설정되지 않기 위해 삭제)

[admin] Linux 패키지 저장소 : reposync + createrepo + nginx

# 패키지 설치
root@admin:~# dnf install -y dnf-plugins-core createrepo nginx
Last metadata expiration check: 0:09:22 ago on Sun 15 Feb 2026 01:25:46 AM KST.
Package dnf-plugins-core-4.7.0-8.el10.noarch is already installed.
Dependencies resolved.
=============================================================================================================================================================================================
 Package                                                Architecture                          Version                                          Repository                               Size
=============================================================================================================================================================================================
Installing:
 createrepo_c                                           aarch64                               1.1.2-4.el10                                     appstream                                75 k
 nginx                                                  aarch64                               2:1.26.3-1.el10                                  appstream                                33 k
Upgrading:
 dnf-plugins-core                                       noarch                                4.7.0-9.el10                                     baseos                                   43 k
 python3-dnf-plugins-core                               noarch                                4.7.0-9.el10                                     baseos                                  315 k
 yum-utils                                              noarch                                4.7.0-9.el10                                     baseos                                   34 k
Installing dependencies:
 createrepo_c-libs                                      aarch64                               1.1.2-4.el10                                     appstream                               103 k
 nginx-core                                             aarch64                               2:1.26.3-1.el10                                  appstream                               652 k
 nginx-filesystem                                       noarch                                2:1.26.3-1.el10                                  appstream                                11 k
 rocky-logos-httpd                                      noarch                                100.4-7.el10                                     appstream                                24 k

Transaction Summary
=============================================================================================================================================================================================
Install  6 Packages
Upgrade  3 Packages

Total download size: 1.3 M
Downloading Packages:
(1/9): nginx-1.26.3-1.el10.aarch64.rpm                                                                                                                        44 kB/s |  33 kB     00:00
(2/9): createrepo_c-1.1.2-4.el10.aarch64.rpm                                                                                                                  99 kB/s |  75 kB     00:00
(3/9): createrepo_c-libs-1.1.2-4.el10.aarch64.rpm                                                                                                            131 kB/s | 103 kB     00:00
(4/9): nginx-filesystem-1.26.3-1.el10.noarch.rpm                                                                                                             177 kB/s |  11 kB     00:00
(5/9): rocky-logos-httpd-100.4-7.el10.noarch.rpm                                                                                                             252 kB/s |  24 kB     00:00
(6/9): nginx-core-1.26.3-1.el10.aarch64.rpm                                                                                                                  2.9 MB/s | 652 kB     00:00
(7/9): yum-utils-4.7.0-9.el10.noarch.rpm                                                                                                                     120 kB/s |  34 kB     00:00
(8/9): dnf-plugins-core-4.7.0-9.el10.noarch.rpm                                                                                                              100 kB/s |  43 kB     00:00
(9/9): python3-dnf-plugins-core-4.7.0-9.el10.noarch.rpm                                                                                                      708 kB/s | 315 kB     00:00
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                        436 kB/s | 1.3 MB     00:02
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                     1/1
  Running scriptlet: nginx-filesystem-2:1.26.3-1.el10.noarch                                                                                                                            1/12
  Installing       : nginx-filesystem-2:1.26.3-1.el10.noarch                                                                                                                            1/12
  Installing       : nginx-core-2:1.26.3-1.el10.aarch64                                                                                                                                 2/12
  Upgrading        : python3-dnf-plugins-core-4.7.0-9.el10.noarch                                                                                                                       3/12
  Upgrading        : dnf-plugins-core-4.7.0-9.el10.noarch                                                                                                                               4/12
  Installing       : rocky-logos-httpd-100.4-7.el10.noarch                                                                                                                              5/12
  Installing       : createrepo_c-libs-1.1.2-4.el10.aarch64                                                                                                                             6/12
  Installing       : createrepo_c-1.1.2-4.el10.aarch64                                                                                                                                  7/12
  Installing       : nginx-2:1.26.3-1.el10.aarch64                                                                                                                                      8/12
  Running scriptlet: nginx-2:1.26.3-1.el10.aarch64                                                                                                                                      8/12
  Upgrading        : yum-utils-4.7.0-9.el10.noarch                                                                                                                                      9/12
  Cleanup          : yum-utils-4.7.0-8.el10.noarch                                                                                                                                     10/12
  Cleanup          : dnf-plugins-core-4.7.0-8.el10.noarch                                                                                                                              11/12
  Cleanup          : python3-dnf-plugins-core-4.7.0-8.el10.noarch                                                                                                                      12/12
  Running scriptlet: python3-dnf-plugins-core-4.7.0-8.el10.noarch                                                                                                                      12/12

Upgraded:
  dnf-plugins-core-4.7.0-9.el10.noarch                          python3-dnf-plugins-core-4.7.0-9.el10.noarch                          yum-utils-4.7.0-9.el10.noarch
Installed:
  createrepo_c-1.1.2-4.el10.aarch64      createrepo_c-libs-1.1.2-4.el10.aarch64  nginx-2:1.26.3-1.el10.aarch64  nginx-core-2:1.26.3-1.el10.aarch64  nginx-filesystem-2:1.26.3-1.el10.noarch
  rocky-logos-httpd-100.4-7.el10.noarch

Complete!

# 패키지(저장소) 동기화 (reposync) : 외부 저장소(BaseOS, AppStream 등)의 패키지를 로컬 디렉토리로 가져옵니다.
## 미러 저장 디렉터리 생성
root@admin:~# mkdir -p /data/repos/rocky/10
root@admin:~# cd /data/repos/rocky/10
root@admin:/data/repos/rocky/10# dnf repolist
repo id                                                                               repo name
appstream                                                                             Rocky Linux 10 - AppStream
baseos                                                                                Rocky Linux 10 - BaseOS
extras                                                                                Rocky Linux 10 - Extras

## baseos
root@admin:/data/repos/rocky/10# dnf reposync --repoid=baseos    --download-metadata -p /data/repos/rocky/10
...
(1466/1474): xz-5.6.2-4.el10_0.aarch64.rpm                                                                                                                   6.3 MB/s | 476 kB     00:00
(1467/1474): xz-libs-5.6.2-4.el10_0.aarch64.rpm                                                                                                              1.7 MB/s | 109 kB     00:00
(1468/1474): yum-4.20.0-18.el10.rocky.0.1.noarch.rpm                                                                                                         2.1 MB/s |  89 kB     00:00
(1469/1474): yum-utils-4.7.0-9.el10.noarch.rpm                                                                                                               872 kB/s |  34 kB     00:00
(1470/1474): zlib-ng-compat-2.2.3-2.el10.aarch64.rpm                                                                                                         1.6 MB/s |  64 kB     00:00
(1471/1474): zip-3.0-45.el10.aarch64.rpm                                                                                                                     4.8 MB/s | 265 kB     00:00
(1472/1474): zlib-ng-compat-2.2.3-3.el10_1.aarch64.rpm                                                                                                       1.2 MB/s |  63 kB     00:00
(1473/1474): zstd-1.5.5-9.el10.aarch64.rpm                                                                                                                   4.4 MB/s | 453 kB     00:00
(1474/1474): zsh-5.9-15.el10.aarch64.rpm                                                                                                                      25 MB/s | 3.3 MB     00:00

## (참고) 메타데이터 확인 : YUM/DNF 저장소의 핵심 두뇌 역할을 하는 Metadata(메타데이터) 파일들 확인
root@admin:/data/repos/rocky/10# ls -l /data/repos/rocky/10/baseos/repodata/
total 30836
-rw-r--r--. 1 root root    62360 Feb 15 01:37 0fb567b1f4c5f9bb81eed3bafc9f40b252558508fcd362a7fdd6b5e1f4c85e52-comps-BaseOS.aarch64.xml.xz
-rw-r--r--. 1 root root 10561544 Feb 15 01:37 1b34bb7357df23636e2683908e763013b4918a245739769642ebf900f077201d-primary.sqlite.xz
-rw-r--r--. 1 root root   343464 Feb 15 01:37 1de704afeee6d14017ee0a20de05455359ebcb291b70b5fd93020b1f1df76b23-other.sqlite.xz
-rw-r--r--. 1 root root  1631654 Feb 15 01:37 3a1d90c1347a5d6d179d8a779da81b8326d1ca1a7111cfc4bf51d7cef9f831f5-filelists.xml.gz
-rw-r--r--. 1 root root  1082008 Feb 15 01:37 62bf9ca6e90e9fbaf53c10eeb8b08b9d8d2cd0d2ebb04d6e488e9526a19c9982-filelists.sqlite.xz
-rw-r--r--. 1 root root  1624517 Feb 15 01:37 9027e315287283bc4e95bd862e0e09012e4e4aa8e93916708e955731d5462f6d-other.xml.gz
-rw-r--r--. 1 root root   308363 Feb 15 01:37 a55eb7089d714a338318e5acf8e9ff8a682433816e0608a3e0eeefc230153a0e-comps-BaseOS.aarch64.xml
-rw-r--r--. 1 root root   113999 Feb 15 01:37 d5c18881b65c04a309e07af98222226346b5405b7f5b3bb2a4e9a6b9344feb10-updateinfo.xml.gz
-rw-r--r--. 1 root root 15820685 Feb 15 01:37 f49f5be57837217b638f20dbb776b3b1dd5e92daaf3bb0e20c67abf38d2c5e40-primary.xml.gz
-rw-r--r--. 1 root root     4449 Feb 15 01:37 repomd.xml  저장소의 마스터 인덱스입니다. 다른 모든 메타데이터 파일의 위치와 체크섬(Hash) 정보를 담고 있어, 클라이언트가 가장 먼저 다운로드하는 파일입니다.

## appstream

## extras


# 내부 배포용 웹 서버 설정 (nginx)
root@admin:/data/repos/rocky/10# dnf reposync --repoid=appstream --download-metadata -p /data/repos/rocky/10
...
(5274/5279): zlib-ng-compat-devel-2.2.3-2.el10.aarch64.rpm                                                                                                   556 kB/s |  38 kB     00:00
(5275/5279): yggdrasil-0.4.8-2.el10.aarch64.rpm                                                                                                               19 MB/s | 5.2 MB     00:00
(5276/5279): zlib-ng-compat-devel-2.2.3-3.el10_1.aarch64.rpm                                                                                                 343 kB/s |  36 kB     00:00
(5277/5279): zram-generator-1.1.2-14.el10.aarch64.rpm                                                                                                        4.8 MB/s | 399 kB     00:00
(5278/5279): zziplib-0.13.78-2.el10.aarch64.rpm                                                                                                              1.8 MB/s |  89 kB     00:00
(5279/5279): zziplib-utils-0.13.78-2.el10.aarch64.rpm                                                                                                        990 kB/s |  45 kB     00:00

root@admin:/data/repos/rocky/10# du -sh /data/repos/rocky/10/appstream/
15G	/data/repos/rocky/10/appstream/

## extras 
root@admin:/data/repos/rocky/10# dnf reposync --repoid=extras    --download-metadata -p /data/repos/rocky/10
...
(22/26): rpmfusion-free-release-10-1.noarch.rpm                                                                                                               75 kB/s | 9.3 kB     00:00
(23/26): rpmfusion-free-release-tainted-10-1.noarch.rpm                                                                                                       55 kB/s | 7.3 kB     00:00
(24/26): update-motd-0.1-2.el10.noarch.rpm                                                                                                                    90 kB/s |  10 kB     00:00
(25/26): rpaste-0.4.1-1.el10.aarch64.rpm                                                                                                                     1.2 MB/s | 2.6 MB     00:02
(26/26): rocky-backgrounds-extras-100.4-7.el10.noarch.rpm                                                                                                     20 MB/s |  63 MB     00:03

# 내부 배포용 웹 서버 설정 (nginx)
root@admin:/data/repos/rocky/10# cat <<EOF > /etc/nginx/conf.d/repos.conf
server {
    listen 80;
    server_name repo-server;

    location /rocky/10/ {
        autoindex on;                 # 디렉터리 목록 표시
        autoindex_exact_size off;     # 파일 크기 KB/MB/GB 단위로 보기 좋게
        autoindex_localtime on;       # 서버 로컬 타임으로 표시
        root /data/repos;
    }
}
EOF

root@admin:/data/repos/rocky/10# systemctl enable --now nginx
Created symlink '/etc/systemd/system/multi-user.target.wants/nginx.service' → '/usr/lib/systemd/system/nginx.service'.

root@admin:/data/repos/rocky/10# systemctl status nginx.service --no-pager
● nginx.service - The nginx HTTP and reverse proxy server
     Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; preset: disabled)
     Active: active (running) since Sun 2026-02-15 01:52:21 KST; 10s ago
 Invocation: 68a15a1b87cf4bf9a2b4846e6d3713db
    Process: 6027 ExecStartPre=/usr/bin/rm -f /run/nginx.pid (code=exited, status=0/SUCCESS)
    Process: 6029 ExecStartPre=/usr/sbin/nginx -t (code=exited, status=0/SUCCESS)
    Process: 6031 ExecStart=/usr/sbin/nginx (code=exited, status=0/SUCCESS)
   Main PID: 6033 (nginx)
      Tasks: 5 (limit: 12336)
     Memory: 5.9M (peak: 6.8M)
        CPU: 36ms
     CGroup: /system.slice/nginx.service
             ├─6033 "nginx: master process /usr/sbin/nginx"
             ├─6034 "nginx: worker process"
             ├─6035 "nginx: worker process"
             ├─6036 "nginx: worker process"
             └─6037 "nginx: worker process"

Feb 15 01:52:21 admin systemd[1]: Starting nginx.service - The nginx HTTP and reverse proxy server...
Feb 15 01:52:21 admin nginx[6029]: nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
Feb 15 01:52:21 admin nginx[6029]: nginx: configuration file /etc/nginx/nginx.conf test is successful
Feb 15 01:52:21 admin systemd[1]: Started nginx.service - The nginx HTTP and reverse proxy server.

root@admin:/data/repos/rocky/10# ss -tnlp | grep nginx
LISTEN 0      511          0.0.0.0:80        0.0.0.0:*    users:(("nginx",pid=6037,fd=6),("nginx",pid=6036,fd=6),("nginx",pid=6035,fd=6),("nginx",pid=6034,fd=6),("nginx",pid=6033,fd=6))
LISTEN 0      511             [::]:80           [::]:*    users:(("nginx",pid=6037,fd=7),("nginx",pid=6036,fd=7),("nginx",pid=6035,fd=7),("nginx",pid=6034,fd=7),("nginx",pid=6033,fd=7))

# 접속 테스트
root@admin:/data/repos/rocky/10# curl http://192.168.10.10/rocky/10/
<html>
<head><title>Index of /rocky/10/</title></head>
<body>
<h1>Index of /rocky/10/</h1><hr><pre><a href="../">../</a>
<a href="appstream/">appstream/</a>                                         15-Feb-2026 01:43       -
<a href="baseos/">baseos/</a>                                            15-Feb-2026 01:37       -
<a href="extras/">extras/</a>                                            15-Feb-2026 01:51       -
</pre><hr></body>
</html>

[k8s-node] 인터넷이 안 되는 내부 서버에서 admin Linux 패키지 저장소를 바라보게 설정

# 기존 레포 설정 백업
root@k8s-node1:~# tree /etc/yum.repos.d/
/etc/yum.repos.d/
├── rocky-addons.repo
├── rocky-devel.repo
├── rocky-extras.repo
└── rocky.repo

1 directory, 4 files
root@k8s-node1:~# mkdir /etc/yum.repos.d/backup
root@k8s-node1:~# mv /etc/yum.repos.d/*.repo /etc/yum.repos.d/backup/

# 로컬 레포 파일 생성: 서버 IP는 Repo 서버의 IP로 수정
## DNF 클라이언트는 HTTP, HTTPS, FTP, file(로컬 파일시스템 직접 접근) 프로토콜로 저장소에 접근할 수 있다.
root@k8s-node1:~# cat <<EOF > /etc/yum.repos.d/internal-rocky.repo
[internal-baseos]
name=Internal Rocky 10 BaseOS
baseurl=http://192.168.10.10/rocky/10/baseos
enabled=1
gpgcheck=0

[internal-appstream]
name=Internal Rocky 10 AppStream
baseurl=http://192.168.10.10/rocky/10/appstream
enabled=1
gpgcheck=0

[internal-extras]
name=Internal Rocky 10 Extras
baseurl=http://192.168.10.10/rocky/10/extras
enabled=1
gpgcheck=0
EOF

# 내부 서버 repo 정상 동작 확인 : 클라이언트에서 캐시를 비우고 목록을 불러옵니다.
root@k8s-node1:~# dnf clean all
0 files removed
root@k8s-node1:~# dnf repolist
repo id                                    repo name
internal-appstream                         Internal Rocky 10 AppStream
internal-baseos                            Internal Rocky 10 BaseOS
internal-extras                            Internal Rocky 10 Extras
root@k8s-node1:~# dnf makecache
Internal Rocky 10 BaseOS                                        66 MB/s |  15 MB     00:00
Internal Rocky 10 AppStream                                     16 MB/s | 2.2 MB     00:00
Internal Rocky 10 Extras                                        38 kB/s | 6.2 kB     00:00
Metadata cache created.

# 패키지 인스톨 정상 실행 확인
root@k8s-node1:~# dnf install -y nfs-utils
Last metadata expiration check: 0:00:23 ago on Sun 15 Feb 2026 01:54:39 AM KST.
Package nfs-utils-1:2.8.2-3.el10.aarch64 is already installed.
Dependencies resolved.
===============================================================================================
 Package              Architecture     Version                  Repository                Size
===============================================================================================
Upgrading:
 libnfsidmap          aarch64          1:2.8.3-0.el10           internal-baseos           61 k
 nfs-utils            aarch64          1:2.8.3-0.el10           internal-baseos          476 k

Transaction Summary
===============================================================================================
Upgrade  2 Packages

Total download size: 537 k
Downloading Packages:
(1/2): libnfsidmap-2.8.3-0.el10.aarch64.rpm                    5.1 MB/s |  61 kB     00:00
(2/2): nfs-utils-2.8.3-0.el10.aarch64.rpm                       30 MB/s | 476 kB     00:00
-----------------------------------------------------------------------------------------------
Total                                                           31 MB/s | 537 kB     00:00
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                       1/1
  Upgrading        : libnfsidmap-1:2.8.3-0.el10.aarch64                                    1/4
  Running scriptlet: nfs-utils-1:2.8.3-0.el10.aarch64                                      2/4
  Upgrading        : nfs-utils-1:2.8.3-0.el10.aarch64                                      2/4
  Running scriptlet: nfs-utils-1:2.8.3-0.el10.aarch64                                      2/4
  Running scriptlet: nfs-utils-1:2.8.2-3.el10.aarch64                                      3/4
  Cleanup          : nfs-utils-1:2.8.2-3.el10.aarch64                                      3/4
  Running scriptlet: nfs-utils-1:2.8.2-3.el10.aarch64                                      3/4
  Cleanup          : libnfsidmap-1:2.8.2-3.el10.aarch64                                    4/4
  Running scriptlet: libnfsidmap-1:2.8.2-3.el10.aarch64                                    4/4

Upgraded:
  libnfsidmap-1:2.8.3-0.el10.aarch64              nfs-utils-1:2.8.3-0.el10.aarch64

Complete!

## 패키지 정보에 repo 확인
root@k8s-node1:~# dnf info nfs-utils | grep -i repo
Repository   : @System
From repo    : internal-baseos

[admin] 다음 실습을 위해 삭제

root@admin:/data/repos/rocky/10# systemctl disable --now nginx && dnf remove -y nginx
Removed '/etc/systemd/system/multi-user.target.wants/nginx.service'.
Dependencies resolved.
===============================================================================================================================================================================================
 Package                                           Architecture                            Version                                            Repository                                  Size
===============================================================================================================================================================================================
Removing:
 nginx                                             aarch64                                 2:1.26.3-1.el10                                    @appstream                                 120 k
Removing unused dependencies:
 nginx-core                                        aarch64                                 2:1.26.3-1.el10                                    @appstream                                 1.8 M
 nginx-filesystem                                  noarch                                  2:1.26.3-1.el10                                    @appstream                                 141
 rocky-logos-httpd                                 noarch                                  100.4-7.el10                                       @appstream                                  24 k

Transaction Summary
===============================================================================================================================================================================================
Remove  4 Packages

Freed space: 1.9 M
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                       1/1
  Running scriptlet: nginx-2:1.26.3-1.el10.aarch64                                                                                                                                         1/4
  Erasing          : nginx-2:1.26.3-1.el10.aarch64                                                                                                                                         1/4
  Running scriptlet: nginx-2:1.26.3-1.el10.aarch64                                                                                                                                         1/4
  Erasing          : rocky-logos-httpd-100.4-7.el10.noarch                                                                                                                                 2/4
  Erasing          : nginx-core-2:1.26.3-1.el10.aarch64                                                                                                                                    3/4
  Erasing          : nginx-filesystem-2:1.26.3-1.el10.noarch                                                                                                                               4/4
  Running scriptlet: nginx-filesystem-2:1.26.3-1.el10.noarch                                                                                                                               4/4

Removed:
  nginx-2:1.26.3-1.el10.aarch64            nginx-core-2:1.26.3-1.el10.aarch64            nginx-filesystem-2:1.26.3-1.el10.noarch            rocky-logos-httpd-100.4-7.el10.noarch

Complete!

Private Container (Image) Registry (kubespray offline에 중복 설정되지 않기 위해 삭제)

[admin] podman 으로 컨테이너 이미지 저장소 Docker Registry (registry) 기동

# podman 설치 : 기본 설치 되어 있음
root@admin:~# dnf info podman | grep repo
From repo    : AppStream

# podman 확인
root@admin:~# which podman
/usr/bin/podman
root@admin:~# podman --version
podman version 5.4.0

# Registry 이미지 받기
root@admin:~# podman pull docker.io/library/registry:latest
Trying to pull docker.io/library/registry:latest...
Getting image source signatures
Copying blob a447a5de8f4e done   |
Copying blob 92c7580d074a done   |
Copying blob 1cc3d49277b7 done   |
Copying blob 50b5971fe294 done   |
Copying blob 0a52a06d47e0 done   |
Copying config 2f5ec5015b done   |
Writing manifest to image destination
2f5ec5015badd603680de78accbba6eb3e9146f4d642a7ccef64205e55ac518f

root@admin:~# podman images
REPOSITORY                  TAG         IMAGE ID      CREATED      SIZE
docker.io/library/registry  latest      2f5ec5015bad  2 weeks ago  57.3 MB

# Registry 데이터 저장 디렉터리 준비
root@admin:~# mkdir -p /data/registry
root@admin:~# chmod 755 /data/registry

# Docker Registry 컨테이너 실행 (기본, 인증 없음)
root@admin:~# podman run -d --name local-registry -p 5000:5000 -v /data/registry:/var/lib/registry --restart=always docker.io/library/registry:latest
177c986039f32b024f720c1059dcc8a38931cacd0f49ce7ff901bd1c8314a548

# 확인
root@admin:~# podman ps
CONTAINER ID  IMAGE                              COMMAND               CREATED         STATUS         PORTS                   NAMES
177c986039f3  docker.io/library/registry:latest  /etc/distribution...  14 seconds ago  Up 15 seconds  0.0.0.0:5000->5000/tcp  local-registry

root@admin:~# pstree -a
systemd --switched-root --system --deserialize=46 no_timer_check
  ├─NetworkManager --no-daemon
  │   └─3*[{NetworkManager}]
  ├─VBoxService --pidfile /var/run/vboxadd-service.sh
  │   └─8*[{VBoxService}]
  ├─agetty -o -- \\u --noreset --noclear - linux
  ├─anacron -s
  ├─atd -f
  ├─auditd
  │   ├─sedispatch
  │   └─2*[{auditd}]
  ├─chronyd -F 2
  ├─conmon --api-version 1 -c 177c986039f32b024f720c1059dcc8a38931cacd0f49ce7ff901bd1c8314a548 -u 177c986039f32b024f720c1059dcc8a38931cacd0f49ce7ff901bd1c8314a548 -r /usr/bin/crun -b/va
  │   └─registry serve /etc/distribution/config.yml
  │       └─6*[{registry}]
  ├─crond -n
  ├─dbus-broker-lau --scope system --audit
  │   └─dbus-broker --log 4 --controller 9 --machine-id 36e9c95128f6493a896149dc465c48d9 --max-bytes 536870912 --max-fds 4096 --max-matches 131072 --audit
  ├─fwupd
  │   └─5*[{fwupd}]
  ├─gpg-agent --homedir /var/lib/fwupd/gnupg --use-standard-socket --daemon
  │   ├─scdaemon --multi-server --homedir /var/lib/fwupd/gnupg
  │   │   └─{scdaemon}
  │   └─{gpg-agent}
  ├─gssproxy -i
  │   └─5*[{gssproxy}]
  ├─irqbalance
  │   └─{irqbalance}
  ├─lsmd -d
  ├─named -u named -c /etc/named.conf
  │   └─13*[{named}]
  ├─packagekitd
  │   └─3*[{packagekitd}]
  ├─polkitd --no-debug --log-level=err
  │   └─3*[{polkitd}]
  ├─rpcbind -w -f
  ├─rsyslogd -n
  │   └─2*[{rsyslogd}]
  ├─sshd
  ├─sshd-session
  │   └─sshd-session
  │       └─bash
  │           └─pstree -a
  ├─systemd --user
  │   └─(sd-pam)
  ├─systemd-journal
  ├─systemd-logind
  ├─systemd-udevd
  ├─systemd-userdbd
  │   ├─systemd-userwor
  │   ├─systemd-userwor
  │   └─systemd-userwor
  ├─tuned -Es /usr/sbin/tuned -l -P
  │   └─3*[{tuned}]
  └─udisksd
      └─6*[{udisksd}]
      

# Registry 정상 동작 확인
root@admin:~# curl -s http://localhost:5000/v2/_catalog | jq
{
  "repositories": []
}

[admin] 컨테이너 이미지 저장소 Docker Registry 에 이미지 push

# 이미지 가져오기
root@admin:~# podman pull alpine
Resolved "alpine" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/alpine:latest...
Getting image source signatures
Copying blob d8ad8cd72600 done   |
Copying config 1ab49c19c5 done   |
Writing manifest to image destination
1ab49c19c53ebca95c787b482aeda86d1d681f58cdf19278c476bcaf37d96de1

root@admin:~# cat /etc/containers/registries.conf.d/000-shortnames.conf | grep alpine
  "alpine" = "docker.io/library/alpine"
  
root@admin:~# podman images
REPOSITORY                  TAG         IMAGE ID      CREATED      SIZE
docker.io/library/registry  latest      2f5ec5015bad  2 weeks ago  57.3 MB
docker.io/library/alpine    latest      1ab49c19c53e  2 weeks ago  8.99 MB


# tag
root@admin:~# podman tag alpine:latest 192.168.10.10:5000/alpine:1.0
root@admin:~# podman images
REPOSITORY                  TAG         IMAGE ID      CREATED      SIZE
docker.io/library/registry  latest      2f5ec5015bad  2 weeks ago  57.3 MB
192.168.10.10:5000/alpine   1.0         1ab49c19c53e  2 weeks ago  8.99 MB
docker.io/library/alpine    latest      1ab49c19c53e  2 weeks ago  8.99 MB

# 프라이빗 레지스트리에 업로드 : 실패!
root@admin:~# podman push 192.168.10.10:5000/alpine:1.0
Getting image source signatures
Error: trying to reuse blob sha256:45f3ea5848e8a25ca27718b640a21ffd8c8745d342a24e1d4ddfc8c449b0a724 at destination: pinging container registry 192.168.10.10:5000: Get "https://192.168.10.10:5000/v2/": http: server gave HTTP response to HTTPS client

# 기본적으로 컨테이너 엔진들은 HTTPS를 요구합니다. 내부망에서 HTTP로 테스트하려면 Registry 주소를 '안전하지 않은 저장소'로 등록해야 합니다.
# (참고) registries.conf 는 containers-common 설정이라서, 'podman, skopeo, buildah' 등 전부 동일하게 적용됨.
root@admin:~# cp /etc/containers/registries.conf /etc/containers/registries.bak
root@admin:~# cat <<EOF >> /etc/containers/registries.conf
[[registry]]
location = "192.168.10.10:5000"
insecure = true
EOF
root@admin:~# grep "^[^#]" /etc/containers/registries.conf
unqualified-search-registries = ["registry.access.redhat.com", "registry.redhat.io", "docker.io"]
short-name-mode = "enforcing"
[[registry]]
location = "192.168.10.10:5000"
insecure = true

# 프라이빗 레지스트리에 업로드 : 성공!
root@admin:~# podman push 192.168.10.10:5000/alpine:1.0
Getting image source signatures
Copying blob 45f3ea5848e8 done   |
Copying config 1ab49c19c5 done   |
Writing manifest to image destination

# 업로드된 이미지와 태그 조회
root@admin:~# curl -s 192.168.10.10:5000/v2/_catalog | jq
{
  "repositories": [
    "alpine"
  ]
}

root@admin:~# curl -s 192.168.10.10:5000/v2/alpine/tags/list | jq
{
  "name": "alpine",
  "tags": [
    "1.0"
  ]
}

[k8s-node] 컨테이너 이미지 pull

# registries.conf 설정
root@k8s-node1:~# cp /etc/containers/registries.conf /etc/containers/registries.bak
root@k8s-node1:~# cat <<EOF >> /etc/containers/registries.conf
[[registry]]
location = "192.168.10.10:5000"
insecure = true
EOF
root@k8s-node1:~# grep "^[^#]" /etc/containers/registries.conf
unqualified-search-registries = ["registry.access.redhat.com", "registry.redhat.io", "docker.io"]
short-name-mode = "enforcing"
[[registry]]
location = "192.168.10.10:5000"
insecure = true

# 이미지 가져오기
root@k8s-node1:~# podman pull 192.168.10.10:5000/alpine:1.0
Trying to pull 192.168.10.10:5000/alpine:1.0...
Getting image source signatures
Copying blob 9268c2c682e1 done   |
Copying config 1ab49c19c5 done   |
Writing manifest to image destination
1ab49c19c53ebca95c787b482aeda86d1d681f58cdf19278c476bcaf37d96de1
root@k8s-node1:~# podman images
REPOSITORY                 TAG         IMAGE ID      CREATED      SIZE
192.168.10.10:5000/alpine  1.0         1ab49c19c53e  2 weeks ago  8.98 MB

[admin] 다음 실습을 위해 삭제

root@admin:~# podman rm -f local-registry
local-registry

[admin, k8s-node] 다음 실습을 위해 파일 원복

root@admin:~# mv /etc/containers/registries.bak /etc/containers/registries.conf
mv: overwrite '/etc/containers/registries.conf'? yes

root@k8s-node1:~# mv /etc/containers/registries.bak /etc/containers/registries.conf
mv: overwrite '/etc/containers/registries.conf'? yes

root@k8s-node2:~# mv /etc/containers/registries.bak /etc/containers/registries.conf
mv: overwrite '/etc/containers/registries.conf'? yes

Private PyPI (Python Package Index) Mirror (kubespray offline에 중복 설정되지 않기 위해 삭제)

[admin] Python 패키지 저장소 구성 : devpi-server

# devpi-server 설치
## devpi-server :	PyPI 미러/사설 패키지 저장소 서버
## devpi-client	: devpi 서버에 패키지 업로드/관리
## devpi-web : 웹 UI (선택)

root@admin:~# pip install devpi-server devpi-client devpi-web
...
Building wheels for collected packages: pyramid-chameleon
  Building wheel for pyramid-chameleon (pyproject.toml) ... done
  Created wheel for pyramid-chameleon: filename=pyramid_chameleon-0.3-py3-none-any.whl size=14231 sha256=c0393e2f98c5189b0125c91cebab83898944328e7783fff3009f1bbacfbba356
  Stored in directory: /root/.cache/pip/wheels/22/ef/bd/de2f48264f520e5e093528f6a71ce0b8b593413c78182b950d
Successfully built pyramid-chameleon
Installing collected packages: Whoosh, translationstring, repoze.lru, passlib, zope.interface, zope.deprecation, webob, waitress, venusian, typing-extensions, soupsieve, ruamel.yaml, pyproject_hooks, pygments, pycparser, py, pluggy, platformdirs, plaster, pkginfo, PasteDeploy, packaging-legacy, nh3, lazy, itsdangerous, iniconfig, hupper, h11, docutils, defusedxml, Chameleon, certifi, strictyaml, readme-renderer, plaster-pastedeploy, httpcore, devpi_common, cffi, build, beautifulsoup4, anyio, pyramid, httpx, cmarkgfm, check-manifest, argon2-cffi-bindings, pyramid-chameleon, devpi-client, argon2-cffi, devpi-server, devpi-web
Successfully installed Chameleon-4.6.0 PasteDeploy-3.1.0 Whoosh-2.7.4 anyio-4.12.1 argon2-cffi-25.1.0 argon2-cffi-bindings-25.1.0 beautifulsoup4-4.14.3 build-1.4.0 certifi-2026.1.4 cffi-2.0.0 check-manifest-0.51 cmarkgfm-2025.10.22 defusedxml-0.7.1 devpi-client-7.2.0 devpi-server-6.19.1 devpi-web-5.0.1 devpi_common-4.1.1 docutils-0.22.4 h11-0.16.0 httpcore-1.0.9 httpx-0.28.1 hupper-1.12.1 iniconfig-2.3.0 itsdangerous-2.2.0 lazy-1.6 nh3-0.3.3 packaging-legacy-23.0.post0 passlib-1.7.4 pkginfo-1.12.1.2 plaster-1.1.2 plaster-pastedeploy-1.0.1 platformdirs-4.8.0 pluggy-1.6.0 py-1.11.0 pycparser-3.0 pygments-2.19.2 pyproject_hooks-1.2.0 pyramid-2.0.2 pyramid-chameleon-0.3 readme-renderer-44.0 repoze.lru-0.7 ruamel.yaml-0.19.1 soupsieve-2.8.3 strictyaml-1.7.3 translationstring-1.4 typing-extensions-4.15.0 venusian-3.1.1 waitress-3.0.2 webob-1.8.9 zope.deprecation-6.0 zope.interface-8.2

root@admin:~# pip list | grep devpi
devpi-client              7.2.0
devpi-common              4.1.1
devpi-server              6.19.1
devpi-web                 5.0.1

# 서버 데이터 디렉토리 생성 및 초기화 : Initialize new devpi-server instance.
## --serverdir을 지정하지 않으면 기본값은 ~/.devpi/server 입니다.
root@admin:~# devpi-init --serverdir /data/devpi_data
2026-02-15 02:13:10,380 INFO  NOCTX Loading node info from /data/devpi_data/.nodeinfo
2026-02-15 02:13:10,380 INFO  NOCTX generated uuid: 35e9b6a4076e42e194ddb36a6683bc39
2026-02-15 02:13:10,381 INFO  NOCTX wrote nodeinfo to: /data/devpi_data/.nodeinfo
2026-02-15 02:13:10,384 INFO  NOCTX DB: Creating schema
2026-02-15 02:13:10,449 INFO  [Wtx-1] setting password for user 'root'
2026-02-15 02:13:10,449 INFO  [Wtx-1] created user 'root'
2026-02-15 02:13:10,449 INFO  [Wtx-1] created root user
2026-02-15 02:13:10,449 INFO  [Wtx-1] created root/pypi index
2026-02-15 02:13:10,455 INFO  [Wtx-1] fswriter0: committed at 0

root@admin:~# ls -al /data/devpi_data/
total 28
drwxr-xr-x. 2 root root    60 Feb 15 02:13 .
drwxr-xr-x. 5 root root    53 Feb 15 02:13 ..
-rw-------. 1 root root    72 Feb 15 02:13 .nodeinfo
-rw-r--r--. 1 root root     1 Feb 15 02:13 .serverversion
-rw-r--r--. 1 root root 20480 Feb 15 02:13 .sqlite

# devpi 서버 기동 : --host 0.0.0.0은 외부(다른 PC) 접속을 허용하기 위함
## 백그라운드 상시 구동은 systemd 서비스로 등록 설정 할 것
root@admin:~# nohup devpi-server --serverdir /data/devpi_data --host 0.0.0.0 --port 3141 > /var/log/devpi.log 2>&1 &
[1] 6803

# 확인
root@admin:~# ss -tnlp | grep devpi-server
LISTEN 0      1024         0.0.0.0:3141      0.0.0.0:*    users:(("devpi-server",pid=6803,fd=12))

root@admin:~# tail -f /var/log/devpi.log
2026-02-15 02:13:42,857 INFO  [NOTI] [Rtx0] triggering load of initial projectnames for root/pypi
2026-02-15 02:13:42,861 INFO  NOCTX devpi-server version: 6.19.1
2026-02-15 02:13:42,861 INFO  NOCTX serverdir: /data/devpi_data
2026-02-15 02:13:42,861 INFO  NOCTX uuid: 35e9b6a4076e42e194ddb36a6683bc39
2026-02-15 02:13:42,861 INFO  NOCTX serving at url: http://0.0.0.0:3141 (might be http://[0.0.0.0]:3141 for IPv6)
2026-02-15 02:13:42,861 INFO  NOCTX using 50 threads
2026-02-15 02:13:42,861 INFO  NOCTX bug tracker: https://github.com/devpi/devpi/issues
2026-02-15 02:13:42,861 INFO  NOCTX Hit Ctrl-C to quit.
2026-02-15 02:13:42,887 INFO  Serving on http://0.0.0.0:3141
2026-02-15 02:13:46,774 INFO  [IDX] Committing 2500 new documents to search index.
2026-02-15 02:14:04,016 INFO  [Wtx0] Processed a total of 741856 projects and queued 741856
2026-02-15 02:14:04,026 INFO  [Wtx0] fswriter1: committed at 1
2026-02-15 02:14:04,029 INFO  [IDXQ] Thread 'InitialQueueThread' ended
2026-02-15 02:14:04,033 INFO  [NOTI] [Rtx1] indexing 'root/pypi' mirror with 741856 projects
2026-02-15 02:14:04,923 INFO  [IDX] Indexer queue size ~ 299
2026-02-15 02:14:06,589 INFO  [IDX] Committing 2500 new documents to search index.

[admin] devpi-server 필요한 패키지 업로드

# 서버 연결
root@admin:~# devpi use http://192.168.10.10:3141
Warning: insecure http host, trusted-host will be set for pip
using server: http://192.168.10.10:3141/ (not logged in)
no current index: type 'devpi use -l' to discover indices
/root/.config/pip/pip.conf: no config file exists
/root/.config/uv/uv.toml: no config file exists
/root/.pydistutils.cfg: no config file exists
/root/.buildout/default.cfg: no config file exists
always-set-cfg: no

# 로그인 (기본 root 비번은 없음)
root@admin:~# devpi login root --password ""
logged in 'root', credentials valid for 10.00 hours

# 폐쇄망에서 쓸 패키지 미리 받아두기
root@admin:~# pip download jmespath netaddr -d /tmp/pypi-packages
Collecting jmespath
  Downloading jmespath-1.1.0-py3-none-any.whl.metadata (7.6 kB)
Collecting netaddr
  Downloading netaddr-1.3.0-py3-none-any.whl.metadata (5.0 kB)
Downloading jmespath-1.1.0-py3-none-any.whl (20 kB)
Downloading netaddr-1.3.0-py3-none-any.whl (2.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.3/2.3 MB 6.3 MB/s eta 0:00:00
Saved /tmp/pypi-packages/jmespath-1.1.0-py3-none-any.whl
Saved /tmp/pypi-packages/netaddr-1.3.0-py3-none-any.whl
Successfully downloaded jmespath netaddr

root@admin:~# tree /tmp/pypi-packages/
/tmp/pypi-packages/
├── jmespath-1.1.0-py3-none-any.whl
└── netaddr-1.3.0-py3-none-any.whl

1 directory, 2 files

# 팀 또는 프로젝트용 인덱스 생성 (예: prod)
## 인덱스 체이닝 : 상속(Inheritance) - bases=root/pypi 설정을 통해, 외부망이 연결된 환경에서는 자동으로 PyPI에서 캐싱해오고, 폐쇄망에서는 내부 업로드 파일을 우선 조회하도록 구성할 수 있습니다.
root@admin:~# devpi index -c prod bases=root/pypi
http://192.168.10.10:3141/root/prod?no_projects=:
  type=stage
  bases=root/pypi
  volatile=True
  acl_upload=root
  acl_toxresult_upload=:ANONYMOUS:
  mirror_whitelist=
  mirror_whitelist_inheritance=intersection
  
# devpi 인덱스(저장소) 목록 확인
root@admin:~# devpi index -l
root/prod
root/pypi

# 특정 인덱스(저장소)에 패키지 있는지 확인
root@admin:~# devpi use root/prod
Warning: insecure http host, trusted-host will be set for pip
current devpi index: http://192.168.10.10:3141/root/prod (logged in as root)
supported features: push-no-docs, push-only-docs, push-register-project, server-keyvalue-parsing
/root/.config/pip/pip.conf: no config file exists
/root/.config/uv/uv.toml: no config file exists
/root/.pydistutils.cfg: no config file exists
/root/.buildout/default.cfg: no config file exists
always-set-cfg: no

# devpi 서버 root/prod 인덱스(저장소)에 패키지 업로드
root@admin:~# devpi upload /tmp/pypi-packages/*
file_upload of jmespath-1.1.0-py3-none-any.whl to http://192.168.10.10:3141/root/prod/
file_upload of netaddr-1.3.0-py3-none-any.whl to http://192.168.10.10:3141/root/prod/

# 업로드 한 패키지 실제 위치 확인
root@admin:~# tree /data/devpi_data/+files/
/data/devpi_data/+files/
└── root
    └── prod
        └── +f
            ├── a56
            │   └── 63118de4908c9
            │       └── jmespath-1.1.0-py3-none-any.whl
            └── c2c
                └── 6a8ebe5554ce3
                    └── netaddr-1.3.0-py3-none-any.whl

8 directories, 2 files

[k8s-node] pip 설정 및 사용

# 일회성 사용
root@k8s-node1:~# pip list | grep -i jmespath
root@k8s-node1:~# pip install jmespath --index-url http://192.168.10.10:3141/root/prod/+simple --trusted-host 192.168.10.10
Looking in indexes: http://192.168.10.10:3141/root/prod/+simple
Collecting jmespath
  Downloading http://192.168.10.10:3141/root/prod/%2Bf/a56/63118de4908c9/jmespath-1.1.0-py3-none-any.whl (20 kB)
Installing collected packages: jmespath
Successfully installed jmespath-1.1.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
root@k8s-node1:~# pip list | grep -i jmespath
jmespath                  1.1.0

# 현재 devpi-server 없는 패키지 설치 시도 : 성공!
root@k8s-node1:~# pip install cryptography

[admin] 다음 실습을 위해 삭제

root@admin:~# pkill -f "devpi-server --serverdir /data/devpi_data"

[k8s-node] 다음 실습을 위해 삭제

root@k8s-node1:~# rm -rf /etc/pip.conf

kubespray offline 설치 소개

kubespray는 오프라인 환경에서 k8s를 배포를 위한 편의성을 지원

다운로드될 파일 목록과 컨테이너 이미지 목록을 파일별로 생성
오프라인 배포를 위한 컨테이너 이미지 다운로드 및 이미지 레지스트리(저장소)에 등록(업로드)
파일(목록)을 다운로드 하고 Nginx 컨테이너를 실행하여, 파일 다운로드 기능 제공

kubespray offline 설치 실습

온라인 환경에서 download-all.sh로 모든 것을 다운로드

→ outputs를 오프라인 노드로 이동

→ 타겟 스크립트들로 로컬 인프라(nginx, registry) 구성

→ kubespray로 클러스터 배포

# download-all.sh
#!/bin/bash

run() {
    echo "=> Running: $*"
    $* || {
        echo "Failed in : $*"
        exit 1
    }
}

source ./config.sh

#run ./install-docker.sh
#run ./install-nerdctl.sh
run ./precheck.sh
run ./prepare-pkgs.sh || exit 1
run ./prepare-py.sh
run ./get-kubespray.sh
if $ansible_in_container; then
    run ./build-ansible-container.sh
else
    run ./pypi-mirror.sh
fi
run ./download-kubespray-files.sh
run ./download-additional-containers.sh
run ./create-repo.sh
run ./copy-target-scripts.sh

echo "Done."

download-all.sh 스크립트에 하위 스크립트는 다음과 같습니다.

1. prepare-pkgs.sh

시스템 패키지 준비. Python, podman(또는 containerd/nerdctl) 등 오프라인 다운로드에 필요한 기본 시스템 패키지들을 설치합니다.

2. prepare-py.sh

Python 환경 구성. Python venv(가상환경)를 생성하고, kubespray가 요구하는 Python 패키지들을 설치합니다.

3. get-kubespray.sh

Kubespray 소스 다운로드. KUBESPRAY_DIR이 존재하지 않으면 kubespray 소스를 다운로드하고 압축을 해제합니다. 필요한 패치도 적용합니다.

4. 조건 분기: build-ansible-container.sh 또는 pypi-mirror.sh

ansible_in_container가 true이면 → build-ansible-container.sh: Ansible 실행용 컨테이너 이미지를 빌드합니다.
ansible_in_container가 false이면 → pypi-mirror.sh: kubespray가 필요로 하는 Python 패키지들의 PyPI 미러를 로컬에 구성합니다. 오프라인 환경에서 pip install이 가능하도록 하기 위함입니다.

5. download-kubespray-files.sh

핵심 바이너리 및 컨테이너 이미지 다운로드. 내부적으로 kubespray의 generate_list.sh를 실행하여 files.list(바이너리 URL 목록)와 images.list(컨테이너 이미지 목록)를 생성한 뒤, 해당 바이너리 파일들(kubeadm, kubelet, kubectl, CNI 플러그인, etcd, crictl 등)을 다운로드합니다. 컨테이너 이미지는 내부적으로 download-images.sh를 호출하여 pull & save합니다.

6. download-additional-containers.sh

추가 컨테이너 이미지 다운로드. imagelists/*.txt 파일에 기재된 추가 컨테이너 이미지들을 다운로드합니다. 사용자가 커스텀 이미지(예: 모니터링, 로깅 등)를 추가하고 싶을 때 이 텍스트 파일에 repoTag를 넣으면 됩니다.

7. create-repo.sh

OS 패키지 리포지토리 생성. 대상 OS에 따라 RPM(RHEL/Rocky/Alma) 또는 DEB(Ubuntu) 패키지들을 다운로드하여 로컬 리포지토리를 만듭니다. 오프라인 노드에서 yum/apt로 패키지 설치가 가능하게 합니다.

8. copy-target-scripts.sh

타겟 노드용 스크립트 복사. 오프라인 환경의 타겟 노드에서 실행할 스크립트들을 outputs/ 디렉토리로 복사합니다.

모든 아티팩트는 ./outputs/ 디렉토리에 저장되며, 이 디렉토리를 오프라인 타겟 노드로 옮긴 뒤 아래 스크립트들을 순서대로 실행하여 클러스터를 구성합니다.

download-all.sh의 하위 스크립트들은 인터넷이 되는 온라인 머신에서 실행됩니다. 목적은 오프라인 설치에 필요한 모든 파일(바이너리, 컨테이너 이미지, OS 패키지, PyPI 패키지 등)을 미리 다운로드해서 ./outputs/ 디렉토리에 모아두는 것입니다.

타겟 노드 스크립트들은 ./outputs/ 디렉토리를 USB나 SCP 등으로 옮긴 뒤, 인터넷이 안 되는 오프라인 노드에서 실행됩니다. 목적은 다운로드해둔 파일들을 이용해 로컬 인프라(nginx 파일서버, 프라이빗 레지스트리 등)를 구축하는 것입니다.

outputs 디렉터리 이동 후 setup-container.sh 실행 : 추가로 install-containerd.sh 실행될 때 하위 스크립트들은 다음과 같습니다.

# setup-container.sh
#!/bin/bash

# install containerd
./install-containerd.sh

# Load images
echo "==> Load registry, nginx images"
NERDCTL=/usr/local/bin/nerdctl
cd ./images

for f in docker.io_library_registry-*.tar.gz docker.io_library_nginx-*.tar.gz; do
    sudo $NERDCTL load -i $f || exit 1
done

if [ -f kubespray-offline-container.tar.gz ]; then
    sudo $NERDCTL load -i kubespray-offline-container.tar.gz || exit 1
fi

1. setup-container.sh

containerd를 로컬 파일로 설치하고, nginx/registry 컨테이너 이미지를 containerd에 로드합니다.

2. start-nginx.sh

nginx 컨테이너를 시작하여 바이너리 파일 및 OS 패키지를 서빙하는 로컬 파일서버 구동합니다.

3. setup-offline.sh
yum/deb repo 경로와 PyPI 미러 설정을 로컬 nginx 서버를 바라보도록 변경합니다.

4. setup-py.sh

로컬 repo에서 python3와 venv를 설치합니다.

5. start-registry.sh

Docker 프라이빗 레지스트리 컨테이너를 시작합니다.

6. load-push-images.sh

저장된 컨테이너 이미지를 containerd에 로드한 뒤 프라이빗 레지스트리에 tag & push합니다.

# setup-container.sh 실행
root@admin:~/kubespray-offline/outputs# ./setup-container.sh

# start-nginx.sh 실행
root@admin:~/kubespray-offline/outputs# ./start-nginx.sh
===> Stop nginx
===> Start nginx
db995aecc4a109d1eb492b334507615ebf79d4f4106fa16289e472dfa51a5d3d
root@admin:~/kubespray-offline/outputs# nerdctl ps
CONTAINER ID    IMAGE                             COMMAND                   CREATED           STATUS    PORTS    NAMES
db995aecc4a1    docker.io/library/nginx:1.29.4    "/docker-entrypoint.…"    14 seconds ago    Up                 nginx
root@admin:~/kubespray-offline/outputs# ss -tnlp | grep nginx
LISTEN 0      511          0.0.0.0:80         0.0.0.0:*    users:(("nginx",pid=21689,fd=6),("nginx",pid=21688,fd=6),("nginx",pid=21687,fd=6),("nginx",pid=21686,fd=6),("nginx",pid=21652,fd=6))
LISTEN 0      511             [::]:80            [::]:*    users:(("nginx",pid=21689,fd=7),("nginx",pid=21688,fd=7),("nginx",pid=21687,fd=7),("nginx",pid=21686,fd=7),("nginx",pid=21652,fd=7))

# setup-offline.sh 실행
root@admin:~/kubespray-offline/outputs# dnf repolist
repo id                                                                               repo name
appstream                                                                             Rocky Linux 10 - AppStream
baseos                                                                                Rocky Linux 10 - BaseOS
extras                                                                                Rocky Linux 10 - Extras

root@admin:~/kubespray-offline/outputs# cat /etc/redhat-release
Rocky Linux release 10.0 (Red Quartz)

# 스크립트 실행 : Setup yum/deb repo config and PyPI mirror config to use local nginx server.
root@admin:~/kubespray-offline/outputs# ./setup-offline.sh
/bin/rm: cannot remove '/etc/yum.repos.d/offline.repo': No such file or directory
===> Disable all yumrepositories
===> Setup local yum repository
[offline-repo]
name=Offline repo
baseurl=http://localhost/rpms/local/
enabled=1
gpgcheck=0
===> Setup PyPI mirror

# 기존 repo 이름이 .original로 변경되고, offline.repo 추가 확인
root@admin:~/kubespray-offline/outputs# tree /etc/yum.repos.d/
/etc/yum.repos.d/
├── offline.repo
├── rocky-addons.repo.original
├── rocky-devel.repo.original
├── rocky-extras.repo.original
└── rocky.repo.original

1 directory, 5 files

# offline.repo 파일 확인
root@admin:~/kubespray-offline/outputs# cat /etc/yum.repos.d/offline.repo
[offline-repo]
name=Offline repo
baseurl=http://localhost/rpms/local/
enabled=1
gpgcheck=0

# offline repo 확인
root@admin:~/kubespray-offline/outputs# dnf clean all
18 files removed
root@admin:~/kubespray-offline/outputs# dnf repolist
repo id                                                                                         repo name
offline-repo                                                                                    Offline repo

# pip 전역 설정 : pypi mirror 설정 확인
root@admin:~/kubespray-offline/outputs# cat ~/.config/pip/pip.conf
[global]
index = http://localhost/pypi/
index-url = http://localhost/pypi/
trusted-host = localhost

# setup-py.sh 실행
root@admin:~/kubespray-offline/outputs# ./setup-py.sh
===> Install python, venv, etc
Offline repo                                                                                                                                                    11 MB/s |  85 kB     00:00
Package python3-3.12.12-3.el10_1.aarch64 is already installed.
Dependencies resolved.
Nothing to do.
Complete!

# 변수 확인
root@admin:~/kubespray-offline/outputs# source pyver.sh
root@admin:~/kubespray-offline/outputs# echo -e "python_version $python${PY}"
python_version 3.12

# offline-repo 에 패키지 파일 확인
root@admin:~/kubespray-offline/outputs# dnf info python3
Last metadata expiration check: 0:00:39 ago on Sun 15 Feb 2026 03:56:17 AM KST.
Installed Packages
Name         : python3
Version      : 3.12.12
Release      : 3.el10_1
Architecture : aarch64
Size         : 83 k
Source       : python3.12-3.12.12-3.el10_1.src.rpm
Repository   : @System
From repo    : baseos
Summary      : Python 3.12 interpreter
URL          : https://www.python.org/
License      : Python-2.0.1
Description  : Python 3.12 is an accessible, high-level, dynamically typed, interpreted
             : programming language, designed with an emphasis on code readability.
             : It includes an extensive standard library, and has a vast ecosystem of
             : third-party libraries.
             :
             : The python3 package provides the "python3" executable: the reference
             : interpreter for the Python language, version 3.
             : The majority of its standard library is provided in the python3-libs package,
             : which should be installed automatically along with python3.
             : The remaining parts of the Python standard library are broken out into the
             : python3-tkinter and python3-test packages, which may need to be installed
             : separately.
             :
             : Documentation for Python is provided in the python3-docs package.
             :
             : Packages containing additional libraries for Python are generally named with
             : the "python3-" prefix.

root@admin:~/kubespray-offline/outputs# tree rpms/local/ | grep -i python
├── libcap-ng-python3-0.8.4-6.el10.aarch64.rpm
├── python3-3.12.12-3.el10_1.aarch64.rpm
├── python3-dateutil-2.9.0.post0-1.el10_0.noarch.rpm
├── python3-dbus-1.3.2-8.el10.aarch64.rpm
├── python3-devel-3.12.12-3.el10_1.aarch64.rpm
├── python3-dnf-4.20.0-18.el10.rocky.0.1.noarch.rpm
├── python3-dnf-plugins-core-4.7.0-9.el10.noarch.rpm
├── python3-dnf-plugin-versionlock-4.7.0-9.el10.noarch.rpm
├── python3-firewall-2.3.1-1.el10_0.noarch.rpm
├── python3-gobject-base-3.46.0-7.el10.aarch64.rpm
├── python3-gobject-base-noarch-3.46.0-7.el10.noarch.rpm
├── python3-hawkey-0.73.1-12.el10.rocky.0.1.aarch64.rpm
├── python3-libcomps-0.1.21-3.el10.aarch64.rpm
├── python3-libdnf-0.73.1-12.el10.rocky.0.1.aarch64.rpm
├── python3-libs-3.12.12-3.el10_1.aarch64.rpm
├── python3-libselinux-3.9-1.el10.aarch64.rpm
├── python3-nftables-1.1.1-6.el10.aarch64.rpm
├── python3-pip-23.3.2-7.el10.noarch.rpm
├── python3-pip-wheel-23.3.2-7.el10.noarch.rpm
├── python3-rpm-4.19.1.1-20.el10.aarch64.rpm
├── python3-six-1.16.0-16.el10.noarch.rpm
├── python3-systemd-235-11.el10.aarch64.rpm
├── python-unversioned-command-3.12.12-3.el10_1.noarch.rpm

# start-registry.sh 실행
root@admin:~/kubespray-offline/outputs# ./start-registry.sh
===> Start registry
0af8f147ba4fd2efc9b09f8a1adbbfe9a59c2dd29e6b7ea6b4d07aea6613c671

# 관련 변수 확인
root@admin:~/kubespray-offline/outputs# source config.sh
root@admin:~/kubespray-offline/outputs# echo -e "registry_port: $REGISTRY_PORT"
registry_port: 35000

# 확인
root@admin:~/kubespray-offline/outputs# nerdctl ps
CONTAINER ID    IMAGE                               COMMAND                   CREATED           STATUS    PORTS    NAMES
0af8f147ba4f    docker.io/library/registry:3.0.0    "/entrypoint.sh /etc…"    34 seconds ago    Up                 registry
db995aecc4a1    docker.io/library/nginx:1.29.4      "/docker-entrypoint.…"    5 minutes ago     Up                 nginx

root@admin:~/kubespray-offline/outputs# ss -tnlp | grep registry
LISTEN 0      4096               *:35000            *:*    users:(("registry",pid=21814,fd=7))
LISTEN 0      4096               *:5001             *:*    users:(("registry",pid=21814,fd=3))


# 현재는 registry 서버 내부에 저장된 (컨테이너) 이미지 없는 상태 : (참고) REGISTRY_DIR=${REGISTRY_DIR:-/var/lib/registry}
root@admin:~/kubespray-offline/outputs# tree /var/lib/registry/
/var/lib/registry/

0 directories, 0 files


# load-push-images.sh 실행
# 스크립트 실행 전 관련 변수 확인
root@admin:~/kubespray-offline/outputs# echo -e "cpu arch: $IMAGE_ARCH"
cpu arch: arm64

#(옵션) 'registry.k8s.io k8s.gcr.io gcr.io ghcr.io docker.io quay.io' 이외에 추가로 필요한 저장소가 있다면 설정
root@admin:~/kubespray-offline/outputs# echo -e "Additional container registry hosts: $ADDITIONAL_CONTAINER_REGISTRY_LIST"
Additional container registry hosts: myregistry.io

# nerdctl load 할 .tar.gz 파일들
root@admin:~/kubespray-offline/outputs# ls -l images/*.tar.gz
-rw-r--r--. 1 root root  10068768 Feb 15 03:03 images/docker.io_amazon_aws-alb-ingress-controller-v1.1.9.tar.gz
-rw-r--r--. 1 root root 175407403 Feb 15 03:04 images/docker.io_amazon_aws-ebs-csi-driver-v0.5.0.tar.gz
-rw-r--r--. 1 root root  95707259 Feb 15 03:00 images/docker.io_cloudnativelabs_kube-router-v2.1.1.tar.gz
-rw-r--r--. 1 root root   4978225 Feb 15 02:58 images/docker.io_flannel_flannel-cni-plugin-v1.7.1-flannel1.tar.gz
-rw-r--r--. 1 root root  32159259 Feb 15 02:57 images/docker.io_flannel_flannel-v0.27.3.tar.gz
-rw-r--r--. 1 root root 160658472 Feb 15 03:00 images/docker.io_kubeovn_kube-ovn-v1.12.21.tar.gz
-rw-r--r--. 1 root root  73788930 Feb 15 03:05 images/docker.io_kubernetesui_dashboard-v2.7.0.tar.gz
-rw-r--r--. 1 root root  18013963 Feb 15 03:05 images/docker.io_kubernetesui_metrics-scraper-v1.0.8.tar.gz
-rw-r--r--. 1 root root  14991749 Feb 15 03:01 images/docker.io_library_haproxy-3.2.4-alpine.tar.gz
-rw-r--r--. 1 root root  20891493 Feb 15 03:01 images/docker.io_library_nginx-1.28.0-alpine.tar.gz
-rw-r--r--. 1 root root  60177566 Feb 15 03:06 images/docker.io_library_nginx-1.29.4.tar.gz
-rw-r--r--. 1 root root   8986338 Feb 15 03:02 images/docker.io_library_registry-2.8.1.tar.gz
-rw-r--r--. 1 root root  17299815 Feb 15 03:06 images/docker.io_library_registry-3.0.0.tar.gz


# (TS) mac사용자: 아래 내용 추가 후 스크립트 다시 실행
vi load-push-all-images.sh
# 아래 추가 -----------------------
...
load_images() {
    for image in $BASEDIR/images/*.tar.gz; do
        echo "===> Loading $image"
        sudo $NERDCTL load --all-platforms -i $image || exit 1
    done
...

root@admin:~/kubespray-offline/outputs# ./load-push-all-images.sh

root@admin:~/kubespray-offline/outputs# nerdctl images
REPOSITORY                                               TAG                                                             IMAGE ID        CREATED               PLATFORM       SIZE       BLOB SIZE
localhost:35000/kube-proxy                               v1.34.3                                                         fa5ed2c96dd3    30 seconds ago        linux/arm64    78.05MB    75.94MB
localhost:35000/kube-scheduler                           v1.34.3                                                         985575f183de    30 seconds ago        linux/arm64    53.34MB    51.59MB
localhost:35000/kube-controller-manager                  v1.34.3                                                         354700b61969    31 seconds ago        linux/arm64    74.38MB    72.62MB
localhost:35000/kube-apiserver                           v1.34.3                                                         dece5cf2dd3b    31 seconds ago        linux/arm64    86.56MB    84.81MB
localhost:35000/metallb/controller                       v0.13.9                                                         b724b69a4c9b    32 seconds ago        linux/arm64    63.12MB    63.11MB
localhost:35000/metallb/speaker                          v0.13.9                                                         51f18d4f5d4d    32 seconds ago        linux/arm64    111.1MB    111.1MB
localhost:35000/kubernetesui/metrics-scraper             v1.0.8                                                          9115322001e6    33 seconds ago        linux/amd64    42.26MB    42.26MB
localhost:35000/kubernetesui/dashboard                   v2.7.0                                                          c353af0aa3a0    34 seconds ago        linux/arm64    256.1MB    247.6MB
localhost:35000/amazon/aws-ebs-csi-driver                v0.5.0                                                          ff65db28333e    36 seconds ago        linux/amd64    456.9MB    444.1MB
localhost:35000/provider-os/cinder-csi-plugin            v1.30.0                                                         f4caa8cc697d    36 seconds ago        linux/arm64    63.05MB    61.21MB
localhost:35000/sig-storage/csi-node-driver-registrar    v2.4.0                                                          9124c121892e    36 seconds ago        linux/arm64    22.41MB    20.25MB
localhost:35000/sig-storage/livenessprobe                v2.11.0                                                         1b1ba6eb5c8d    37 seconds ago        linux/arm64    22.25MB    20.14MB
localhost:35000/sig-storage/csi-resizer                  v1.9.2                                                          8f191a7ec9cc    37 seconds ago        linux/arm64    64.17MB    62.06MB
localhost:35000/sig-storage/snapshot-controller          v7.0.2                                                          f23cbdb6dd3f    38 seconds ago        linux/arm64    62.28MB    59.56MB
localhost:35000/sig-storage/csi-snapshotter              v6.3.2                                                          2ba4692b39fc    38 seconds ago        linux/arm64    63.98MB    61.87MB
localhost:35000/sig-storage/csi-provisioner              v3.6.2                                                          b34169e8b528    39 seconds ago        linux/arm64    67.74MB    65.63MB
localhost:35000/sig-storage/csi-attacher                 v4.4.2                                                          2aa9f6446ccd    39 seconds ago        linux/arm64    63.48MB    61.37MB
localhost:35000/jetstack/cert-manager-webhook            v1.15.3                                                         2d91656807bb    40 seconds ago        linux/arm64    58.15MB    56.39MB
localhost:35000/jetstack/cert-manager-cainjector         v1.15.3                                                         a13418dc926e    40 seconds ago        linux/arm64    44.65MB    42.89MB
localhost:35000/jetstack/cert-manager-controller         v1.15.3                                                         5114bfbeac23    40 seconds ago        linux/arm64    67.13MB    65.37MB
localhost:35000/amazon/aws-alb-ingress-controller        v1.1.9                                                          e88f69a74c30    41 seconds ago        linux/arm64    37.9MB     37.91MB
localhost:35000/ingress-nginx/controller                 v1.13.3                                                         68a587e5104f    42 seconds ago        linux/arm64    336.3MB    334.2MB
localhost:35000/rancher/local-path-provisioner           v0.0.32                                                         4a3d51575c84    43 seconds ago        linux/arm64    61.37MB    61.35MB
localhost:35000/sig-storage/local-volume-provisioner     v2.5.0                                                          d158fd9f3579    43 seconds ago        linux/arm64    134.7MB    130.4MB
localhost:35000/metrics-server/metrics-server            v0.8.0                                                          87ccea7af925    44 seconds ago        linux/arm64    82.58MB    80.84MB
localhost:35000/library/registry                         2.8.1                                                           b1524398e0af    44 seconds ago        linux/arm64    25.85MB    25.68MB
localhost:35000/cpa/cluster-proportional-autoscaler      v1.8.8                                                          4146047e636f    44 seconds ago        linux/arm64    39.98MB    37.86MB
localhost:35000/dns/k8s-dns-node-cache                   1.25.0                                                          7071feee8b70    45 seconds ago        linux/arm64    90.54MB    88.43MB
localhost:35000/coredns/coredns                          v1.12.1                                                         e674cf21adf3    46 seconds ago        linux/arm64    74.94MB    73.19MB
localhost:35000/library/haproxy                          3.2.4-alpine                                                    71268591d942    46 seconds ago        linux/arm64    33.57MB    33.37MB
localhost:35000/library/nginx                            1.28.0-alpine                                                   bcb5257f77e1    46 seconds ago        linux/arm64    52.73MB    51.23MB
localhost:35000/kube-vip/kube-vip                        v1.0.3                                                          133301efa7cb    47 seconds ago        linux/arm64    58.69MB    58.69MB
localhost:35000/pause                                    3.10.1                                                          3f85f9d8a6bc    47 seconds ago        linux/arm64    516.1kB    516.9kB
localhost:35000/cloudnativelabs/kube-router              v2.1.1                                                          9b0b03a20f4b    48 seconds ago        linux/arm64    216.7MB    216.1MB
localhost:35000/kubeovn/kube-ovn                         v1.12.21                                                        c93b1a2ecea3    50 seconds ago        linux/arm64    513.6MB    501.2MB
localhost:35000/calico/apiserver                         v3.30.6                                                         b15f366465da    51 seconds ago        linux/arm64    113.8MB    113.8MB
localhost:35000/calico/typha                             v3.30.6                                                         3f421c499d47    51 seconds ago        linux/arm64    82.42MB    82.4MB
localhost:35000/calico/kube-controllers                  v3.30.6                                                         7639e6dca06e    52 seconds ago        linux/arm64    117.4MB    117.4MB
localhost:35000/calico/cni                               v3.30.6                                                         0aa5f9cab8de    53 seconds ago        linux/arm64    157.3MB    157.3MB
localhost:35000/calico/node                              v3.30.6                                                         742471336dda    54 seconds ago        linux/arm64    401.5MB    400MB
localhost:35000/flannel/flannel-cni-plugin               v1.7.1-flannel1                                                 332db17b4c4a    54 seconds ago        linux/arm64    11.39MB    11.37MB
localhost:35000/flannel/flannel                          v0.27.3                                                         3b36a8d4db19    55 seconds ago        linux/arm64    102.6MB    101.5MB
localhost:35000/k8snetworkplumbingwg/multus-cni          v4.2.2                                                          db44f5c1fd8e    57 seconds ago        linux/arm64    497.5MB    495.3MB
localhost:35000/cilium/cilium-envoy                      v1.34.10-1762597008-ff7ae7d623be00078865cff1b0672cc5d9bfc6d5    f8eeaa2e6cbe    58 seconds ago        linux/arm64    206.8MB    201.9MB
localhost:35000/cilium/hubble-ui-backend                 v0.13.3                                                         1b0a4d764eaf    59 seconds ago        linux/arm64    68.89MB    67.14MB
localhost:35000/cilium/hubble-ui                         v0.13.3                                                         637e4d5a05ac    59 seconds ago        linux/arm64    35.56MB    31.92MB
localhost:35000/cilium/certgen                           v0.2.4                                                          e3403ce43031    59 seconds ago        linux/arm64    58.68MB    58.68MB
localhost:35000/cilium/hubble-relay                      v1.18.6                                                         5c66f4ecc18e    About a minute ago    linux/arm64    92.61MB    90.86MB
localhost:35000/cilium/operator                          v1.18.6                                                         7bba55333f30    About a minute ago    linux/arm64    258MB      258MB
localhost:35000/cilium/cilium                            v1.18.6                                                         88b892a5f89b    About a minute ago    linux/arm64    732.4MB    725.2MB
localhost:35000/coreos/etcd                              v3.5.26                                                         4b003fe9069c    About a minute ago    linux/arm64    66.06MB    63.34MB
localhost:35000/mirantis/k8s-netchecker-agent            v1.2.2                                                          e07c83f8f083    About a minute ago    linux/amd64    5.681MB    5.856MB
localhost:35000/mirantis/k8s-netchecker-server           v1.2.2                                                          8e0ef348cf54    About a minute ago    linux/amd64    125.8MB    123.7MB
localhost:35000/library/registry                         3.0.0                                                           496d3637ba81    About a minute ago    linux/arm64    57.52MB    57.34MB
localhost:35000/library/nginx                            1.29.4                                                          4c333d291372    About a minute ago    linux/arm64    190.7MB    184MB
registry.k8s.io/sig-storage/snapshot-controller          v7.0.2                                                          f23cbdb6dd3f    About a minute ago    linux/arm64    62.28MB    59.56MB
registry.k8s.io/sig-storage/local-volume-provisioner     v2.5.0                                                          d158fd9f3579    About a minute ago    linux/arm64    134.7MB    130.4MB
registry.k8s.io/sig-storage/livenessprobe                v2.11.0                                                         1b1ba6eb5c8d    About a minute ago    linux/arm64    22.25MB    20.14MB
registry.k8s.io/sig-storage/csi-snapshotter              v6.3.2                                                          2ba4692b39fc    About a minute ago    linux/arm64    63.98MB    61.87MB
registry.k8s.io/sig-storage/csi-resizer                  v1.9.2                                                          8f191a7ec9cc    About a minute ago    linux/arm64    64.17MB    62.06MB
registry.k8s.io/sig-storage/csi-provisioner              v3.6.2                                                          b34169e8b528    About a minute ago    linux/arm64    67.74MB    65.63MB
registry.k8s.io/sig-storage/csi-node-driver-registrar    v2.4.0                                                          9124c121892e    About a minute ago    linux/arm64    22.41MB    20.25MB
registry.k8s.io/sig-storage/csi-attacher                 v4.4.2                                                          2aa9f6446ccd    About a minute ago    linux/arm64    63.48MB    61.37MB
registry.k8s.io/provider-os/cinder-csi-plugin            v1.30.0                                                         f4caa8cc697d    About a minute ago    linux/arm64    63.05MB    61.21MB
registry.k8s.io/pause                                    3.10.1                                                          3f85f9d8a6bc    About a minute ago    linux/arm64    516.1kB    516.9kB
registry.k8s.io/metrics-server/metrics-server            v0.8.0                                                          87ccea7af925    About a minute ago    linux/arm64    82.58MB    80.84MB
registry.k8s.io/kube-scheduler                           v1.34.3                                                         985575f183de    About a minute ago    linux/arm64    53.34MB    51.59MB
registry.k8s.io/kube-proxy                               v1.34.3                                                         fa5ed2c96dd3    About a minute ago    linux/arm64    78.05MB    75.94MB
registry.k8s.io/kube-controller-manager                  v1.34.3                                                         354700b61969    About a minute ago    linux/arm64    74.38MB    72.62MB
registry.k8s.io/kube-apiserver                           v1.34.3                                                         dece5cf2dd3b    About a minute ago    linux/arm64    86.56MB    84.81MB
registry.k8s.io/ingress-nginx/controller                 v1.13.3                                                         68a587e5104f    About a minute ago    linux/arm64    336.3MB    334.2MB
registry.k8s.io/dns/k8s-dns-node-cache                   1.25.0                                                          7071feee8b70    About a minute ago    linux/arm64    90.54MB    88.43MB
registry.k8s.io/cpa/cluster-proportional-autoscaler      v1.8.8                                                          4146047e636f    About a minute ago    linux/arm64    39.98MB    37.86MB
registry.k8s.io/coredns/coredns                          v1.12.1                                                         e674cf21adf3    About a minute ago    linux/arm64    74.94MB    73.19MB
quay.io/metallb/speaker                                  v0.13.9                                                         51f18d4f5d4d    About a minute ago    linux/arm64    111.1MB    111.1MB
quay.io/metallb/controller                               v0.13.9                                                         b724b69a4c9b    About a minute ago    linux/arm64    63.12MB    63.11MB
quay.io/jetstack/cert-manager-webhook                    v1.15.3                                                         2d91656807bb    About a minute ago    linux/arm64    58.15MB    56.39MB
quay.io/jetstack/cert-manager-controller                 v1.15.3                                                         5114bfbeac23    About a minute ago    linux/arm64    67.13MB    65.37MB
quay.io/jetstack/cert-manager-cainjector                 v1.15.3                                                         a13418dc926e    About a minute ago    linux/arm64    44.65MB    42.89MB
quay.io/coreos/etcd                                      v3.5.26                                                         4b003fe9069c    About a minute ago    linux/arm64    66.06MB    63.34MB
quay.io/cilium/operator                                  v1.18.6                                                         7bba55333f30    About a minute ago    linux/arm64    258MB      258MB
quay.io/cilium/hubble-ui                                 v0.13.3                                                         637e4d5a05ac    About a minute ago    linux/arm64    35.56MB    31.92MB
quay.io/cilium/hubble-ui-backend                         v0.13.3                                                         1b0a4d764eaf    About a minute ago    linux/arm64    68.89MB    67.14MB
quay.io/cilium/hubble-relay                              v1.18.6                                                         5c66f4ecc18e    About a minute ago    linux/arm64    92.61MB    90.86MB
quay.io/cilium/cilium                                    v1.18.6                                                         88b892a5f89b    About a minute ago    linux/arm64    732.4MB    725.2MB
quay.io/cilium/cilium-envoy                              v1.34.10-1762597008-ff7ae7d623be00078865cff1b0672cc5d9bfc6d5    f8eeaa2e6cbe    About a minute ago    linux/arm64    206.8MB    201.9MB
quay.io/cilium/certgen                                   v0.2.4                                                          e3403ce43031    About a minute ago    linux/arm64    58.68MB    58.68MB
quay.io/calico/typha                                     v3.30.6                                                         3f421c499d47    About a minute ago    linux/arm64    82.42MB    82.4MB
quay.io/calico/node                                      v3.30.6                                                         742471336dda    About a minute ago    linux/arm64    401.5MB    400MB
quay.io/calico/kube-controllers                          v3.30.6                                                         7639e6dca06e    About a minute ago    linux/arm64    117.4MB    117.4MB
quay.io/calico/cni                                       v3.30.6                                                         0aa5f9cab8de    About a minute ago    linux/arm64    157.3MB    157.3MB
quay.io/calico/apiserver                                 v3.30.6                                                         b15f366465da    About a minute ago    linux/arm64    113.8MB    113.8MB
ghcr.io/kube-vip/kube-vip                                v1.0.3                                                          133301efa7cb    About a minute ago    linux/arm64    58.69MB    58.69MB
ghcr.io/k8snetworkplumbingwg/multus-cni                  v4.2.2                                                          db44f5c1fd8e    2 minutes ago         linux/arm64    497.5MB    495.3MB
rancher/local-path-provisioner                           v0.0.32                                                         4a3d51575c84    2 minutes ago         linux/arm64    61.37MB    61.35MB
mirantis/k8s-netchecker-server                           v1.2.2                                                          8e0ef348cf54    2 minutes ago         linux/amd64    125.8MB    123.7MB
mirantis/k8s-netchecker-agent                            v1.2.2                                                          e07c83f8f083    2 minutes ago         linux/amd64    5.681MB    5.856MB
haproxy                                                  3.2.4-alpine                                                    71268591d942    2 minutes ago         linux/arm64    33.57MB    33.37MB
kubernetesui/metrics-scraper                             v1.0.8                                                          9115322001e6    2 minutes ago         linux/amd64    42.26MB    42.26MB
kubernetesui/dashboard                                   v2.7.0                                                          c353af0aa3a0    2 minutes ago         linux/arm64    256.1MB    247.6MB
kubeovn/kube-ovn                                         v1.12.21                                                        c93b1a2ecea3    2 minutes ago         linux/arm64    513.6MB    501.2MB
flannel/flannel                                          v0.27.3                                                         3b36a8d4db19    2 minutes ago         linux/arm64    102.6MB    101.5MB
flannel/flannel-cni-plugin                               v1.7.1-flannel1                                                 332db17b4c4a    2 minutes ago         linux/arm64    11.39MB    11.37MB
cloudnativelabs/kube-router                              v2.1.1                                                          9b0b03a20f4b    2 minutes ago         linux/arm64    216.7MB    216.1MB
amazon/aws-ebs-csi-driver                                v0.5.0                                                          ff65db28333e    2 minutes ago         linux/amd64    456.9MB    444.1MB
amazon/aws-alb-ingress-controller                        v1.1.9                                                          e88f69a74c30    2 minutes ago         linux/arm64    37.9MB     37.91MB
nginx                                                    1.29.4                                                          4c333d291372    15 minutes ago        linux/arm64    190.7MB    184MB
nginx                                                    1.28.0-alpine                                                   bcb5257f77e1    15 minutes ago        linux/arm64    52.73MB    51.23MB
registry                                                 3.0.0                                                           496d3637ba81    15 minutes ago        linux/arm64    57.52MB    57.34MB
registry                                                 2.8.1                                                           b1524398e0af    15 minutes ago        linux/arm64    25.85MB    25.68MB


# 이미지 저장소 카탈로그 확인
root@admin:~/kubespray-offline/outputs# curl -s http://localhost:35000/v2/_catalog | jq
{
  "repositories": [
    "amazon/aws-alb-ingress-controller",
    "amazon/aws-ebs-csi-driver",
    "calico/apiserver",
    "calico/cni",
    "calico/kube-controllers",
    "calico/node",
    "calico/typha",
    "cilium/certgen",
    "cilium/cilium",
    "cilium/cilium-envoy",
    "cilium/hubble-relay",
    "cilium/hubble-ui",
    "cilium/hubble-ui-backend",
    "cilium/operator",
    "cloudnativelabs/kube-router",
    "coredns/coredns",
    "coreos/etcd",
    "cpa/cluster-proportional-autoscaler",
    "dns/k8s-dns-node-cache",
    "flannel/flannel",
    "flannel/flannel-cni-plugin",
    "ingress-nginx/controller",
    "jetstack/cert-manager-cainjector",
    "jetstack/cert-manager-controller",
    "jetstack/cert-manager-webhook",
    "k8snetworkplumbingwg/multus-cni",
    "kube-apiserver",
    "kube-controller-manager",
    "kube-proxy",
    "kube-scheduler",
    "kube-vip/kube-vip",
    "kubeovn/kube-ovn",
    "kubernetesui/dashboard",
    "kubernetesui/metrics-scraper",
    "library/haproxy",
    "library/nginx",
    "library/registry",
    "metallb/controller",
    "metallb/speaker",
    "metrics-server/metrics-server",
    "mirantis/k8s-netchecker-agent",
    "mirantis/k8s-netchecker-server",
    "pause",
    "provider-os/cinder-csi-plugin",
    "rancher/local-path-provisioner",
    "sig-storage/csi-attacher",
    "sig-storage/csi-node-driver-registrar",
    "sig-storage/csi-provisioner",
    "sig-storage/csi-resizer",
    "sig-storage/csi-snapshotter",
    "sig-storage/livenessprobe",
    "sig-storage/local-volume-provisioner",
    "sig-storage/snapshot-controller"
  ]
}

# kube-apiserver 정보 확인  
root@admin:~/kubespray-offline/outputs# curl -s http://localhost:35000/v2/kube-apiserver/tags/list | jq
{
  "name": "kube-apiserver",
  "tags": [
    "v1.34.3"
  ]
}

## Image Manifest
root@admin:~/kubespray-offline/outputs# curl -s http://localhost:35000/v2/kube-apiserver/tags/list | jq
{
  "name": "kube-apiserver",
  "tags": [
    "v1.34.3"
  ]
}
root@admin:~/kubespray-offline/outputs#
root@admin:~/kubespray-offline/outputs# curl -s http://localhost:35000/v2/kube-apiserver/manifests/v1.34.3 | jq
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "digest": "sha256:cf65ae6c8f700cc27f57b7305c6e2b71276a7eed943c559a0091e1e667169896",
    "size": 2906
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
      "digest": "sha256:378b3db0974f7a5a8767b6329ad310983bc712d0e400ff5faa294f95f869cc8c",
      "size": 327680
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
      "digest": "sha256:8fa10c0194df9b7c054c90dbe482585f768a54428fc90a5b78a0066a123b1bba",
      "size": 40960
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
      "digest": "sha256:4840c7c54023c867f19564429c89ddae4e9589c83dce82492183a7e9f7dab1fa",
      "size": 2406400
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
      "digest": "sha256:114dde0fefebbca13165d0da9c500a66190e497a82a53dcaabc3172d630be1e9",
      "size": 102400
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
      "digest": "sha256:4d049f83d9cf21d1f5cc0e11deaf36df02790d0e60c1a3829538fb4b61685368",
      "size": 1536
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
      "digest": "sha256:af5aa97ebe6ce1604747ec1e21af7136ded391bcabe4acef882e718a87c86bcc",
      "size": 2560
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
      "digest": "sha256:6f1cdceb6a3146f0ccb986521156bef8a422cdbb0863396f7f751f575ba308f4",
      "size": 2560
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
      "digest": "sha256:bbb6cacb8c82e4da4e8143e03351e939eab5e21ce0ef333c42e637af86c5217b",
      "size": 2560
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
      "digest": "sha256:2a92d6ac9e4fcc274d5168b217ca4458a9fec6f094ead68d99c77073f08caac1",
      "size": 1536
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
      "digest": "sha256:1a73b54f556b477f0a8b939d13c504a3b4f4db71f7a09c63afbc10acb3de5849",
      "size": 10240
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
      "digest": "sha256:f4aee9e53c42a22ed82451218c3ea03d1eea8d6ca8fbe8eb4e950304ba8a8bb3",
      "size": 3072
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
      "digest": "sha256:bfe9137a1b044e8097cdfcb6899137a8a984ed70931ed1e8ef0cf7e023a139fc",
      "size": 241664
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
      "digest": "sha256:a40e7263c9df7d0a1f68b0f447449f41f88f6982e9f148d8b9d9b0c22ee7fc26",
      "size": 1837056
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
      "digest": "sha256:f6a99f5dc9ec493ccd88f616c8917a4f3bc01f68f75e3e78e232d6071d36b51f",
      "size": 79827968
    }
  ]
}

# 이미지 저장소의 저장 디렉터리 확인
root@admin:~/kubespray-offline/outputs# tree /var/lib/registry/ -L 5
/var/lib/registry/
└── docker
    └── registry
        └── v2
            ├── blobs
            │   └── sha256
            └── repositories
                ├── amazon
                ├── calico
                ├── cilium
                ├── cloudnativelabs
                ├── coredns
                ├── coreos
                ├── cpa
                ├── dns
                ├── flannel
                ├── ingress-nginx
                ├── jetstack
                ├── k8snetworkplumbingwg
                ├── kube-apiserver
                ├── kube-controller-manager
                ├── kubeovn
                ├── kube-proxy
                ├── kubernetesui
                ├── kube-scheduler
                ├── kube-vip
                ├── library
                ├── metallb
                ├── metrics-server
                ├── mirantis
                ├── pause
                ├── provider-os
                ├── rancher
                └── sig-storage

34 directories, 0 files

# extract-kubespary.sh 실행
root@admin:~/kubespray-offline/outputs# ls -lh files/kubespray-*
-rw-r--r--. 1 root root 2.5M Feb 15 02:44 files/kubespray-2.30.0.tar.gz

# patches 파일 : kubespary-2.18.0 버전에서 patch 되는 내용으로, kubespary-2.30.0과 관계없음
root@admin:~/kubespray-offline/outputs# tree patches/
patches/
└── 2.18.0
    ├── 0001-nerdctl-insecure-registry-config-8339.patch
    ├── 0002-Update-config.toml.j2-8340.patch
    └── 0003-generate-list-8537.patch

2 directories, 3 files

# Extract kubespray tarball and apply all patches.
root@admin:~/kubespray-offline/outputs# ./extract-kubespray.sh

# kubespary 저장소 압축해제된 파일들 확인
root@admin:~/kubespray-offline/outputs# tree kubespray-2.30.0/ -L 1
kubespray-2.30.0/
├── ansible.cfg
├── CHANGELOG.md
├── cluster.yml
├── CNAME
├── code-of-conduct.md
├── _config.yml
├── contrib
├── CONTRIBUTING.md
├── Dockerfile
├── docs
├── extra_playbooks
├── galaxy.yml
├── index.html
├── inventory
├── library
├── LICENSE
├── logo
├── meta
├── OWNERS
├── OWNERS_ALIASES
├── pipeline.Dockerfile
├── playbooks
├── plugins
├── README.md
├── recover-control-plane.yml
├── RELEASE.md
├── remove-node.yml
├── remove_node.yml
├── requirements.txt
├── reset.yml
├── roles
├── scale.yml
├── scripts
├── SECURITY_CONTACTS
├── test-infra
├── tests
├── upgrade-cluster.yml
├── upgrade_cluster.yml
└── Vagrantfile

14 directories, 26 files

Kubespray를 설치하겠습니다.

# check python version
root@admin:~/kubespray-offline# python --version
Python 3.12.12

# venv 실행
root@admin:~/kubespray-offline# python3.12 -m venv ~/.venv/3.12
root@admin:~/kubespray-offline# source ~/.venv/3.12/bin/activate
((3.12) ) root@admin:~/kubespray-offline# which ansible
/root/.venv/3.12/bin/ansible
((3.12) ) root@admin:~/kubespray-offline# tree ~/.venv/3.12/ -L 4
/root/.venv/3.12/
├── bin
│   ├── activate
│   ├── activate.csh
│   ├── activate.fish
│   ├── Activate.ps1
│   ├── ansible
│   ├── ansible-community
│   ├── ansible-config
│   ├── ansible-connection
│   ├── ansible-console
│   ├── ansible-doc
│   ├── ansible-galaxy
│   ├── ansible-inventory
│   ├── ansible-playbook
│   ├── ansible-pull
│   ├── ansible-test
│   ├── ansible-vault
│   ├── jp.py
│   ├── netaddr
│   ├── pip
│   ├── pip3
│   ├── pip3.12
│   ├── __pycache__
│   │   └── jp.cpython-312.pyc
│   ├── pypi-mirror
│   ├── python -> python3.12
│   ├── python3 -> python3.12
│   └── python3.12 -> /usr/bin/python3.12
├── include
│   └── python3.12
├── lib
│   └── python3.12
│       └── site-packages
│           ├── ansible
│           ├── ansible-10.7.0.dist-info
│           ├── ansible_collections
│           ├── ansible_core-2.17.14.dist-info
│           ├── ansible_test
│           ├── cffi
│           ├── cffi-2.0.0.dist-info
│           ├── _cffi_backend.cpython-312-aarch64-linux-gnu.so
│           ├── cryptography
│           ├── cryptography-46.0.3.dist-info
│           ├── _distutils_hack
│           ├── distutils-precedence.pth
│           ├── jinja2
│           ├── jinja2-3.1.6.dist-info
│           ├── jmespath
│           ├── jmespath-1.1.0.dist-info
│           ├── markupsafe
│           ├── markupsafe-3.0.3.dist-info
│           ├── netaddr
│           ├── netaddr-1.3.0.dist-info
│           ├── packaging
│           ├── packaging-26.0.dist-info
│           ├── pip
│           ├── pip-26.0.1.dist-info
│           ├── __pycache__
│           ├── pycparser
│           ├── pycparser-3.0.dist-info
│           ├── pypi_mirror.py
│           ├── python_pypi_mirror-5.2.1.dist-info
│           ├── pyyaml-6.0.3.dist-info
│           ├── resolvelib
│           ├── resolvelib-1.0.1.dist-info
│           ├── setuptools
│           ├── setuptools-82.0.0.dist-info
│           ├── _yaml
│           └── yaml
├── lib64 -> lib
└── pyvenv.cfg

42 directories, 30 files

# kubespary 디렉터리 이동
((3.12) ) root@admin:~/kubespray-offline/outputs# cd /root/kubespray-offline/outputs/kubespray-2.30.0

# Install ansible : 이미 설치 완료된 상태
((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# pip install -U pip
Looking in indexes: http://localhost/pypi/
Requirement already satisfied: pip in /root/.venv/3.12/lib64/python3.12/site-packages (26.0.1)

# offline.yml 파일 복사 후 inventory 복사
((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# cp ../../offline.yml .
((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# cp -r inventory/sample inventory/mycluster
((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# tree inventory/mycluster/
inventory/mycluster/
├── group_vars
│   ├── all
│   │   ├── all.yml
│   │   ├── aws.yml
│   │   ├── azure.yml
│   │   ├── containerd.yml
│   │   ├── coreos.yml
│   │   ├── cri-o.yml
│   │   ├── docker.yml
│   │   ├── etcd.yml
│   │   ├── gcp.yml
│   │   ├── hcloud.yml
│   │   ├── huaweicloud.yml
│   │   ├── oci.yml
│   │   ├── offline.yml
│   │   ├── openstack.yml
│   │   ├── upcloud.yml
│   │   └── vsphere.yml
│   └── k8s_cluster
│       ├── addons.yml
│       ├── k8s-cluster.yml
│       ├── k8s-net-calico.yml
│       ├── k8s-net-cilium.yml
│       ├── k8s-net-custom-cni.yml
│       ├── k8s-net-flannel.yml
│       ├── k8s-net-kube-ovn.yml
│       ├── k8s-net-kube-router.yml
│       ├── k8s-net-macvlan.yml
│       └── kube_control_plane.yml
└── inventory.ini

4 directories, 27 files

# 웹서버와 이미지 저장소 정보 수정 : http_server, registry_host
((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# cat offline.yml
#
# offline.yml sample
#

http_server: "http://YOUR_HOST"
registry_host: "YOUR_HOST:35000"

# Insecure registries for containerd
containerd_registries_mirrors:
  - prefix: "{{ registry_host }}"
    mirrors:
      - host: "http://{{ registry_host }}"
        capabilities: ["pull", "resolve"]
        skip_verify: true

files_repo: "{{ http_server }}/files"
yum_repo: "{{ http_server }}/rpms"
ubuntu_repo: "{{ http_server }}/debs"

# Registry overrides
kube_image_repo: "{{ registry_host }}"
gcr_image_repo: "{{ registry_host }}"
docker_image_repo: "{{ registry_host }}"
quay_image_repo: "{{ registry_host }}"
github_image_repo: "{{ registry_host }}"

local_path_provisioner_helper_image_repo: "{{ registry_host }}/busybox"

# Download URLs: See roles/download/defaults/main.yml of kubespray.
kubeadm_download_url: "{{ files_repo }}/kubernetes/v{{ kube_version }}/kubeadm"
kubectl_download_url: "{{ files_repo }}/kubernetes/v{{ kube_version }}/kubectl"
kubelet_download_url: "{{ files_repo }}/kubernetes/v{{ kube_version }}/kubelet"

# etcd is optional if you **DON'T** use etcd_deployment=host
etcd_download_url: "{{ files_repo }}/kubernetes/etcd/etcd-v{{ etcd_version }}-linux-amd64.tar.gz"

# CNI plugins
cni_download_url: "{{ files_repo }}/kubernetes/cni/cni-plugins-linux-{{ image_arch }}-v{{ cni_version }}.tgz"
# cri-tools
crictl_download_url: "{{ files_repo }}/kubernetes/cri-tools/crictl-v{{ crictl_version }}-{{ ansible_system | lower }}-{{ image_arch }}.tar.gz"

# If using Calico
calicoctl_download_url: "{{ files_repo }}/kubernetes/calico/v{{ calico_ctl_version }}/calicoctl-linux-{{ image_arch }}"
# If using Calico with kdd
calico_crds_download_url: "{{ files_repo }}/kubernetes/calico/v{{ calico_version }}.tar.gz"

# If using cilium
ciliumcli_download_url: "{{ files_repo }}/cilium-cli/v{{ cilium_cli_version }}/cilium-linux-{{ image_arch }}.tar.gz"

# helm
helm_download_url: "{{ files_repo }}/helm-v{{ helm_version }}-linux-{{ image_arch }}.tar.gz"

# crun
crun_download_url: "{{ files_repo }}/crun-{{ crun_version }}-linux-{{ image_arch }}"

# kata
kata_containers_download_url: "{{ files_repo }}/kata-static-{{ kata_containers_version }}-{{ image_arch }}.tar.xz"

# Containerd
runc_download_url: "{{ files_repo }}/runc/v{{ runc_version }}/runc.{{ image_arch }}"
nerdctl_download_url: "{{ files_repo }}/nerdctl-{{ nerdctl_version }}-{{ ansible_system | lower }}-{{ image_arch }}.tar.gz"
containerd_download_url: "{{ files_repo }}/containerd-{{ containerd_version }}-linux-{{ image_arch }}.tar.gz"

# cri-o
crio_download_url: "{{ files_repo }}/cri-o.{{ image_arch }}.v{{ crio_version }}.tar.gz"
skopeo_download_url: "{{ files_repo }}/skopeo/v{{ skopeo_version }}/skopeo-linux-{{ image_arch }}"

# gvisor
gvisor_runsc_download_url: "{{ files_repo }}/gvisor/{{ gvisor_version }}/{{ ansible_architecture }}/runsc"
gvisor_containerd_shim_runsc_download_url: "{{ files_repo }}/gvisor/{{ gvisor_version }}/{{ ansible_architecture }}/containerd-shim-runsc-v1"

# others
youki_download_url: "{{ files_repo }}/youki-{{ youki_version }}-{{ ansible_architecture }}-musl.tar.gz"
yq_download_url: "{{ files_repo }}/yq/v{{ yq_version }}/yq_linux_{{ image_arch }}"

((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# sed -i "s/YOUR_HOST/192.168.10.10/g" offline.yml

((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# cat offline.yml | grep 192.168.10.10
http_server: "http://192.168.10.10"
registry_host: "192.168.10.10:35000"

# 수정 반영된 offline.yml 파일을 inventory 디렉터리 내부로 복사
((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# cp -f offline.yml inventory/mycluster/group_vars/all/offline.yml
cp: overwrite 'inventory/mycluster/group_vars/all/offline.yml'? yes

# inventory 파일 작성
((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# cat <<EOF > inventory/mycluster/inventory.ini
[kube_control_plane]
k8s-node1 ansible_host=192.168.10.11 ip=192.168.10.11 etcd_member_name=etcd1

[etcd:children]
kube_control_plane

[kube_node]
k8s-node2 ansible_host=192.168.10.12 ip=192.168.10.12
EOF

# ansible 연결 확인
# 나는 ping 실패해서 키 등록을 해주었음
((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# ssh-copy-id root@192.168.10.11
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@192.168.10.11's password:

Number of key(s) added: 1

Now try logging into the machine, with: "ssh 'root@192.168.10.11'"
and check to make sure that only the key(s) you wanted were added.

((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# ssh-copy-id root@192.168.10.12
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@192.168.10.12's password:

Number of key(s) added: 1

Now try logging into the machine, with: "ssh 'root@192.168.10.12'"
and check to make sure that only the key(s) you wanted were added.


# ansible 연결 확인
((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# ansible -i inventory/mycluster/inventory.ini all -m ping
[WARNING]: Platform linux on host k8s-node1 is using the discovered Python interpreter at /usr/bin/python3.12, but future installation of another Python interpreter could change the meaning
of that path. See https://docs.ansible.com/ansible-core/2.17/reference_appendices/interpreter_discovery.html for more information.
k8s-node1 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3.12"
    },
    "changed": false,
    "ping": "pong"
}
[WARNING]: Platform linux on host k8s-node2 is using the discovered Python interpreter at /usr/bin/python3.12, but future installation of another Python interpreter could change the meaning
of that path. See https://docs.ansible.com/ansible-core/2.17/reference_appendices/interpreter_discovery.html for more information.
k8s-node2 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3.12"
    },
    "changed": false,
    "ping": "pong"
}

# 각 노드에 offline repo 설정
((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# tree ../playbook/
../playbook/
├── offline-repo.yml
└── roles
    └── offline-repo
        ├── defaults
        │   └── main.yml
        ├── files
        │   └── 99offline
        └── tasks
            ├── Debian.yml
            ├── main.yml
            └── RedHat.yml

6 directories, 6 files

((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# mkdir offline-repo
((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# cp -r ../playbook/ offline-repo/
((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# tree offline-repo/
offline-repo/
└── playbook
    ├── offline-repo.yml
    └── roles
        └── offline-repo
            ├── defaults
            │   └── main.yml
            ├── files
            │   └── 99offline
            └── tasks
                ├── Debian.yml
                ├── main.yml
                └── RedHat.yml

7 directories, 6 files
((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# ansible-playbook -i inventory/mycluster/inventory.ini offline-repo/playbook/offline-repo.yml

PLAY [all] ************************************************************************************************************************************************************************************
Sunday 15 February 2026  04:24:52 +0900 (0:00:00.028)       0:00:00.028 *******

TASK [Gathering Facts] ************************************************************************************************************************************************************************
ok: [k8s-node1]
ok: [k8s-node2]
Sunday 15 February 2026  04:24:53 +0900 (0:00:01.513)       0:00:01.542 *******

TASK [offline-repo : include_tasks] ***********************************************************************************************************************************************************
included: /root/kubespray-offline/outputs/kubespray-2.30.0/offline-repo/playbook/roles/offline-repo/tasks/RedHat.yml for k8s-node2, k8s-node1
Sunday 15 February 2026  04:24:53 +0900 (0:00:00.035)       0:00:01.577 *******

TASK [offline-repo : Install offline yum repo] ************************************************************************************************************************************************
changed: [k8s-node1]
changed: [k8s-node2]

PLAY RECAP ************************************************************************************************************************************************************************************
k8s-node1                  : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
k8s-node2                  : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Sunday 15 February 2026  04:24:54 +0900 (0:00:00.249)       0:00:01.826 *******
===============================================================================
Gathering Facts ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 1.51s
offline-repo : Install offline yum repo ------------------------------------------------------------------------------------------------------------------------------------------------ 0.25s
offline-repo : include_tasks ----------------------------------------------------------------------------------------------------------------------------------------------------------- 0.04s


# k8s-node 확인
((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# ssh k8s-node1 tree /etc/yum.repos.d/
The authenticity of host 'k8s-node1 (192.168.10.11)' can't be established.
ED25519 key fingerprint is SHA256:wOgDB8b6Oi9Dt6gCWDgUS4n+k6Civhv9EtrlCicAnxg.
This host key is known by the following other names/addresses:
    ~/.ssh/known_hosts:1: 192.168.10.11
    ~/.ssh/known_hosts:4: 192.168.10.12
Are you sure you want to continue connecting (yes/no/[fingerprint])? ㅛyes
Warning: Permanently added 'k8s-node1' (ED25519) to the list of known hosts.
/etc/yum.repos.d/
├── backup
│   ├── rocky-addons.repo
│   ├── rocky-devel.repo
│   ├── rocky-extras.repo
│   └── rocky.repo
├── internal-rocky.repo
└── offline.repo

2 directories, 6 files

((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# ssh k8s-node1 dnf repolist
repo id                             repo name
internal-appstream                  Internal Rocky 10 AppStream
internal-baseos                     Internal Rocky 10 BaseOS
internal-extras                     Internal Rocky 10 Extras
offline-repo                        Offline repo for kubespray

((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# ssh k8s-node1 cat /etc/yum.repos.d/offline.repo
[offline-repo]
baseurl = http://192.168.10.10/rpms/local
enabled = 1
gpgcheck = 0
name = Offline repo for kubespray


## 추가로 설치를 위해 기존 repo 제거 : 미실행할 경우, kubespary 실행 시 fail됨
((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# for i in rocky-addons rocky-devel rocky-extras rocky; do
  ssh k8s-node1 "mv /etc/yum.repos.d/$i.repo /etc/yum.repos.d/$i.repo.original"
  ssh k8s-node2 "mv /etc/yum.repos.d/$i.repo /etc/yum.repos.d/$i.repo.original"
done
mv: cannot stat '/etc/yum.repos.d/rocky-addons.repo': No such file or directory
The authenticity of host 'k8s-node2 (192.168.10.12)' can't be established.
ED25519 key fingerprint is SHA256:wOgDB8b6Oi9Dt6gCWDgUS4n+k6Civhv9EtrlCicAnxg.
This host key is known by the following other names/addresses:
    ~/.ssh/known_hosts:1: 192.168.10.11
    ~/.ssh/known_hosts:4: 192.168.10.12
    ~/.ssh/known_hosts:5: k8s-node1
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'k8s-node2' (ED25519) to the list of known hosts.
mv: cannot stat '/etc/yum.repos.d/rocky-addons.repo': No such file or directory
mv: cannot stat '/etc/yum.repos.d/rocky-devel.repo': No such file or directory
mv: cannot stat '/etc/yum.repos.d/rocky-devel.repo': No such file or directory
mv: cannot stat '/etc/yum.repos.d/rocky-extras.repo': No such file or directory
mv: cannot stat '/etc/yum.repos.d/rocky-extras.repo': No such file or directory
mv: cannot stat '/etc/yum.repos.d/rocky.repo': No such file or directory
mv: cannot stat '/etc/yum.repos.d/rocky.repo': No such file or directory


((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# ssh k8s-node1 tree /etc/yum.repos.d/
/etc/yum.repos.d/
├── backup
│   ├── rocky-addons.repo
│   ├── rocky-devel.repo
│   ├── rocky-extras.repo
│   └── rocky.repo
├── internal-rocky.repo
└── offline.repo

2 directories, 6 files
((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# ssh k8s-node1 dnf repolist
repo id                             repo name
internal-appstream                  Internal Rocky 10 AppStream
internal-baseos                     Internal Rocky 10 BaseOS
internal-extras                     Internal Rocky 10 Extras
offline-repo                        Offline repo for kubespray

((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# ssh k8s-node2 tree /etc/yum.repos.d/
/etc/yum.repos.d/
├── backup
│   ├── rocky-addons.repo
│   ├── rocky-devel.repo
│   ├── rocky-extras.repo
│   └── rocky.repo
├── internal-rocky.repo
└── offline.repo

2 directories, 6 files
((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# ssh k8s-node2 dnf repolist
repo id                             repo name
internal-appstream                  Internal Rocky 10 AppStream
internal-baseos                     Internal Rocky 10 BaseOS
internal-extras                     Internal Rocky 10 Extras
offline-repo                        Offline repo for kubespray

# admin-lb 에 kubectl 없는 것 확인
((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# which kubectl
/usr/bin/which: no kubectl in (/root/.venv/3.12/bin:/root/.local/bin:/root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin)

# group vars 실습 환경에 맞게 설정
((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# echo "kubectl_localhost: true" >> inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml # 배포를 수행하는 로컬 머신의 bin 디렉토리에도 kubectl 바이너리를 다운로드
sed -i 's|kube_owner: kube|kube_owner: root|g' inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
sed -i 's|kube_network_plugin: calico|kube_network_plugin: flannel|g' inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
sed -i 's|kube_proxy_mode: ipvs|kube_proxy_mode: iptables|g' inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
sed -i 's|enable_nodelocaldns: true|enable_nodelocaldns: false|g' inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
grep -iE 'kube_owner|kube_network_plugin:|kube_proxy_mode|enable_nodelocaldns:' inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
echo "enable_dns_autoscaler: false" >> inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml

echo "flannel_interface: enp0s9" >> inventory/mycluster/group_vars/k8s_cluster/k8s-net-flannel.yml
grep "^[^#]" inventory/mycluster/group_vars/k8s_cluster/k8s-net-flannel.yml

sed -i 's|helm_enabled: false|helm_enabled: true|g' inventory/mycluster/group_vars/k8s_cluster/addons.yml
sed -i 's|metrics_server_enabled: false|metrics_server_enabled: true|g' inventory/mycluster/group_vars/k8s_cluster/addons.yml
grep -iE 'metrics_server_enabled:' inventory/mycluster/group_vars/k8s_cluster/addons.yml
echo "metrics_server_requests_cpu: 25m"     >> inventory/mycluster/group_vars/k8s_cluster/addons.yml
echo "metrics_server_requests_memory: 16Mi" >> inventory/mycluster/group_vars/k8s_cluster/addons.yml
# Note: cilium needs to set kube_owner to root https://kubespray.io/#/docs/CNI/cilium?id=unprivileged-agent-configuration
kube_owner: root
kube_network_plugin: flannel
kube_proxy_mode: iptables
enable_nodelocaldns: false
flannel_interface: enp0s9
metrics_server_enabled: true


# 지원 버전 정보 확인
((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# cat roles/kubespray_defaults/vars/main/checksums.yml | grep -i kube -A40
kubelet_checksums:
  arm64:
    1.34.3: sha256:765b740e3ad9c590852652a2623424ec60e2dddce2c6280d7f042f56c8c98619
    1.34.2: sha256:3e31b1bee9ab32264a67af8a19679777cd372b1c3a04b5d7621289cf137b357c
    1.34.1: sha256:6a66bc08d6c637fcea50c19063cf49e708fde1630a7f1d4ceca069a45a87e6f1
    1.34.0: sha256:e45a7795391cd62ee226666039153832d3096c0f892266cd968936e18b2b40b0
    1.33.7: sha256:3035c44e0d429946d6b4b66c593d371cf5bbbfc85df39d7e2a03c422e4fe404a
    1.33.6: sha256:7d8b7c63309cfe2da2331a1ae13cce070b9ba01e487099e7881a4281667c131d
    1.33.5: sha256:c6ad0510c089d49244eede2638b4a4ff125258fd29a0649e7eef05c7f79c737f
    1.33.4: sha256:623329b1a5f4858e3a5406d3947807b75144f4e71dde11ef1a71362c3a8619cc
    1.33.3: sha256:3f69bb32debfaf25fce91aa5e7181e1e32f3550f3257b93c17dfb37bed621a9c
    1.33.2: sha256:0fa15aca9b90fe7aef1ed3aad31edd1d9944a8c7aae34162963a6aaaf726e065
    1.33.1: sha256:10540261c311ae005b9af514d83c02694e12614406a8524fd2d0bad75296f70d
    ...
    
# [macOS 사용자] (TS) 이슈 해결
((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# cat inventory/mycluster/group_vars/all/offline.yml | grep amd64
etcd_download_url: "{{ files_repo }}/kubernetes/etcd/etcd-v{{ etcd_version }}-linux-amd64.tar.gz"
((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# sed -i 's/amd64/arm64/g' inventory/mycluster/group_vars/all/offline.yml


# 배포
((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# ansible-playbook -i inventory/mycluster/inventory.ini -v cluster.yml -e kube_version="1.34.3"
PLAY RECAP ************************************************************************************************************************************************************************************
k8s-node1                  : ok=541  changed=126  unreachable=0    failed=0    skipped=850  rescued=0    ignored=1
k8s-node2                  : ok=425  changed=86   unreachable=0    failed=0    skipped=626  rescued=0    ignored=0

Sunday 15 February 2026  05:21:38 +0900 (0:00:00.041)       0:04:55.788 *******
===============================================================================
kubernetes/kubeadm : Join to cluster if needed ---------------------------------------------------------------------------------------------------------------------------------------- 16.16s
kubernetes/control-plane : Kubeadm | Initialize first control plane node (1st try) ----------------------------------------------------------------------------------------------------- 8.72s
etcd : Restart etcd -------------------------------------------------------------------------------------------------------------------------------------------------------------------- 6.88s
etcd : Configure | Ensure etcd is running ---------------------------------------------------------------------------------------------------------------------------------------------- 5.62s
etcd : Configure | Check if etcd cluster is healthy ------------------------------------------------------------------------------------------------------------------------------------ 5.25s
network_plugin/flannel : Flannel | Wait for flannel subnet.env file presence ----------------------------------------------------------------------------------------------------------- 5.25s
download : Download_file | Download item ----------------------------------------------------------------------------------------------------------------------------------------------- 3.91s
container-engine/containerd : Containerd | Unpack containerd archive ------------------------------------------------------------------------------------------------------------------- 3.67s
download : Download_container | Download image if required ----------------------------------------------------------------------------------------------------------------------------- 3.59s
network_plugin/cni : CNI | Copy cni plugins -------------------------------------------------------------------------------------------------------------------------------------------- 3.57s
container-engine/containerd : Download_file | Download item ---------------------------------------------------------------------------------------------------------------------------- 3.50s
kubernetes-apps/metrics_server : Metrics Server | Create manifests --------------------------------------------------------------------------------------------------------------------- 3.43s
container-engine/crictl : Extract_file | Unpacking archive ----------------------------------------------------------------------------------------------------------------------------- 3.30s
etcdctl_etcdutl : Extract_file | Unpacking archive ------------------------------------------------------------------------------------------------------------------------------------- 3.21s
container-engine/runc : Download_file | Download item ---------------------------------------------------------------------------------------------------------------------------------- 3.13s
container-engine/crictl : Download_file | Download item -------------------------------------------------------------------------------------------------------------------------------- 3.12s
container-engine/nerdctl : Download_file | Download item ------------------------------------------------------------------------------------------------------------------------------- 3.11s
kubernetes-apps/helm : Extract_file | Unpacking archive -------------------------------------------------------------------------------------------------------------------------------- 3.09s
container-engine/nerdctl : Extract_file | Unpacking archive ---------------------------------------------------------------------------------------------------------------------------- 2.94s
kubernetes-apps/helm : Download_file | Download item ----------------------------------------------------------------------------------------------------------------------------------- 2.88s


# 설치 후 NetworkManger 에 dns 설정 파일 추가 확인
((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# ssh k8s-node2 cat /etc/NetworkManager/conf.d/dns.conf

[global-dns-domain-*]
servers = 10.233.0.3,192.168.10.10
[global-dns]
searches = default.svc.cluster.local,svc.cluster.local
options = ndots:2,timeout:2,attempts:2

((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# ssh k8s-node2 cat /etc/resolv.conf
nameserver 192.168.10.10

# 설치 후 NetworkManger 에서 특정 NIC은 관리하지 않게 설정 추가 확인
((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# ssh k8s-node2 cat /etc/NetworkManager/conf.d/k8s.conf
[keyfile]
unmanaged-devices+=interface-name:kube-ipvs0;interface-name:nodelocaldns

# kubectl 바이너리 파일을 ansible-playbook 실행한 서버에 다운로드 확인
((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# file inventory/mycluster/artifacts/kubectl
inventory/mycluster/artifacts/kubectl: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, BuildID[sha1]=638a428e8c66d4b9ab3e9f21c1d934c4fe12d447, stripped
((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# ls -l inventory/mycluster/artifacts/kubectl
-rwxr-xr-x. 1 root root 58130616 Feb 15 05:20 inventory/mycluster/artifacts/kubectl
((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# tree inventory/mycluster/
inventory/mycluster/
├── artifacts
│   └── kubectl
├── credentials
│   └── kubeadm_certificate_key.creds
├── group_vars
│   ├── all
│   │   ├── all.yml
│   │   ├── aws.yml
│   │   ├── azure.yml
│   │   ├── containerd.yml
│   │   ├── coreos.yml
│   │   ├── cri-o.yml
│   │   ├── docker.yml
│   │   ├── etcd.yml
│   │   ├── gcp.yml
│   │   ├── hcloud.yml
│   │   ├── huaweicloud.yml
│   │   ├── oci.yml
│   │   ├── offline.yml
│   │   ├── openstack.yml
│   │   ├── upcloud.yml
│   │   └── vsphere.yml
│   └── k8s_cluster
│       ├── addons.yml
│       ├── k8s-cluster.yml
│       ├── k8s-net-calico.yml
│       ├── k8s-net-cilium.yml
│       ├── k8s-net-custom-cni.yml
│       ├── k8s-net-flannel.yml
│       ├── k8s-net-kube-ovn.yml
│       ├── k8s-net-kube-router.yml
│       ├── k8s-net-macvlan.yml
│       └── kube_control_plane.yml
└── inventory.ini

6 directories, 29 files


((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# cp inventory/mycluster/artifacts/kubectl /usr/local/bin/
((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# kubectl version --client=true
Client Version: v1.34.3
Kustomize Version: v5.7.1

# k8s admin 자격증명 확인 
((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# mkdir /root/.kube
((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# scp k8s-node1:/root/.kube/config /root/.kube/
config                                                                                                                                                       100% 5645     4.2MB/s   00:00
((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# sed -i 's/127.0.0.1/192.168.10.11/g' /root/.kube/config
((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# kubectl get deploy,sts,ds -n kube-system -owide
NAME                             READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS       IMAGES                                                     SELECTOR
deployment.apps/coredns          2/2     2            2           7m29s   coredns          192.168.10.10:35000/coredns/coredns:v1.12.1                k8s-app=kube-dns
deployment.apps/metrics-server   1/1     1            1           7m13s   metrics-server   192.168.10.10:35000/metrics-server/metrics-server:v0.8.0   app.kubernetes.io/name=metrics-server,version=0.8.0

NAME                                     DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE     CONTAINERS     IMAGES                                        SELECTOR
daemonset.apps/kube-flannel              0         0         0       0            0           <none>                   7m41s   kube-flannel   192.168.10.10:35000/flannel/flannel:v0.27.3   app=flannel
daemonset.apps/kube-flannel-ds-arm       0         0         0       0            0           <none>                   7m41s   kube-flannel   192.168.10.10:35000/flannel/flannel:v0.27.3   app=flannel
daemonset.apps/kube-flannel-ds-arm64     2         2         2       2            2           <none>                   7m41s   kube-flannel   192.168.10.10:35000/flannel/flannel:v0.27.3   app=flannel
daemonset.apps/kube-flannel-ds-ppc64le   0         0         0       0            0           <none>                   7m41s   kube-flannel   192.168.10.10:35000/flannel/flannel:v0.27.3   app=flannel
daemonset.apps/kube-flannel-ds-s390x     0         0         0       0            0           <none>                   7m41s   kube-flannel   192.168.10.10:35000/flannel/flannel:v0.27.3   app=flannel
daemonset.apps/kube-proxy                2         2         2       2            2           kubernetes.io/os=linux   8m23s   kube-proxy     192.168.10.10:35000/kube-proxy:v1.34.3        k8s-app=kube-proxy

다만, 중간에 ansible이 실패한 경우가 발생하였는데 원인은 다음과 같습니다.

TASK [system_packages : Manage packages] ******************************************************************************************************************************************************
failed: [k8s-node1] (item=remove) => {"ansible_loop_var": "item", "attempts": 4, "changed": false, "item": {"action_label": "remove", "packages": {"systemd-timesyncd": [false, false, false]}, "state": "absent"}, "msg": "Failed to download metadata for repo 'internal-baseos': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried", "rc": 1, "results": []}
FAILED - RETRYING: [k8s-node1]: Manage packages (4 retries left).
failed: [k8s-node2] (item=remove) => {"ansible_loop_var": "item", "attempts": 4, "changed": false, "item": {"action_label": "remove", "packages": {"systemd-timesyncd": [false, false, false]}, "state": "absent"}, "msg": "Failed to download metadata for repo 'internal-baseos': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried", "rc": 1, "results": []}
FAILED - RETRYING: [k8s-node2]: Manage packages (4 retries left).
FAILED - RETRYING: [k8s-node1]: Manage packages (3 retries left).
FAILED - RETRYING: [k8s-node1]: Manage packages (2 retries left).

/etc/yum.repos.d/는 RHEL 계열 리눅스(Rocky, AlmaLinux 등)에서 yum/dnf 패키지 관리자가 참조하는 리포지토리 설정 파일들이 저장되는 디렉토리이고, 이 안에 있는 .repo 파일들이 패키지를 어디서 다운로드할지 정의합니다.

이번 문제의 원인은, 타겟 노드에 기존에 설정되어 있던 internal-rocky.repo 파일이 사내 미러 서버(internal-baseos)를 가리키고 있었는데, 오프라인 환경에서는 해당 서버에 접근할 수 없어서 발생한 것입니다. yum/dnf는 패키지를 제거(remove) 할 때도 활성화된 모든 repo의 메타데이터를 갱신하려고 시도하기 때문에, 접근 불가능한 repo가 하나라도 있으면 작업 전체가 실패합니다.

해결 방법은 해당 repo를 비활성화(enabled=0)하거나 파일을 백업 처리하여, 오프라인 환경에 맞는 로컬 repo만 사용하도록 변경하는 것이었습니다.

[K8s Deploy] Kubespary HA & Upgrade

안녕유지 — Wed, 4 Feb 2026 19:15:17 +0900

Cloudnet K8s Deploy 5주차 스터디를 진행하며 정리한 글입니다.

실습 환경

Worker Client-Side LoadBalancing

Kubespray를 통한 K8s 배포

# inventory.ini 확인
root@admin-lb:~/kubespray# cat /root/kubespray/inventory/mycluster/inventory.ini
[kube_control_plane]
k8s-node1 ansible_host=192.168.10.11 ip=192.168.10.11 etcd_member_name=etcd1
k8s-node2 ansible_host=192.168.10.12 ip=192.168.10.12 etcd_member_name=etcd2
k8s-node3 ansible_host=192.168.10.13 ip=192.168.10.13 etcd_member_name=etcd3

[etcd:children]
kube_control_plane

[kube_node]
k8s-node4 ansible_host=192.168.10.14 ip=192.168.10.14
#k8s-node5 ansible_host=192.168.10.15 ip=192.168.10.15


root@admin-lb:~/kubespray# ansible-inventory -i /root/kubespray/inventory/mycluster/inventory.ini --graph
@all:
  |--@ungrouped:
  |--@etcd:
  |  |--@kube_control_plane:
  |  |  |--k8s-node1
  |  |  |--k8s-node2
  |  |  |--k8s-node3
  |--@kube_node:
  |  |--k8s-node4
  
  
# k8s_cluster.yml # for every node in the cluster (not etcd when it's separate)
root@admin-lb:~/kubespray# 
sed -i 's|kube_owner: kube|kube_owner: root|g' inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
sed -i 's|kube_network_plugin: calico|kube_network_plugin: flannel|g' inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
sed -i 's|kube_proxy_mode: ipvs|kube_proxy_mode: iptables|g' inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
sed -i 's|enable_nodelocaldns: true|enable_nodelocaldns: false|g' inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
grep -iE 'kube_owner|kube_network_plugin:|kube_proxy_mode|enable_nodelocaldns:' inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml

kube_owner: root
kube_network_plugin: flannel
kube_proxy_mode: iptables
enable_nodelocaldns: false

## coredns autoscaler 미설치
root@admin-lb:~/kubespray# echo "enable_dns_autoscaler: false" >> inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml

# flannel 설정 수정
root@admin-lb:~/kubespray# 
echo "flannel_interface: enp0s9" >> inventory/mycluster/group_vars/k8s_cluster/k8s-net-flannel.yml
grep "^[^#]" inventory/mycluster/group_vars/k8s_cluster/k8s-net-flannel.yml
flannel_interface: enp0s9

# addons
root@admin-lb:~/kubespray# 
sed -i 's|metrics_server_enabled: false|metrics_server_enabled: true|g' inventory/mycluster/group_vars/k8s_cluster/addons.yml
grep -iE 'metrics_server_enabled:' inventory/mycluster/group_vars/k8s_cluster/addons.yml
metrics_server_enabled: true

root@admin-lb:~/kubespray# 
echo "metrics_server_requests_cpu: 25m"     >> inventory/mycluster/group_vars/k8s_cluster/addons.yml
echo "metrics_server_requests_memory: 16Mi" >> inventory/mycluster/group_vars/k8s_cluster/addons.yml

# 지원 버전 정보 확인
root@admin-lb:~/kubespray# cat roles/kubespray_defaults/vars/main/checksums.yml | grep -i kube -A40
kubelet_checksums:
  arm64:
    1.33.7: sha256:3035c44e0d429946d6b4b66c593d371cf5bbbfc85df39d7e2a03c422e4fe404a
    1.33.6: sha256:7d8b7c63309cfe2da2331a1ae13cce070b9ba01e487099e7881a4281667c131d
    ...

# 배포
root@admin-lb:~/kubespray# ANSIBLE_FORCE_COLOR=true ansible-playbook -i inventory/mycluster/inventory.ini -v cluster.yml -e kube_version="1.32.9" | tee kubespray_install.log
PLAY RECAP *********************************************************************
k8s-node1                  : ok=529  changed=120  unreachable=0    failed=0    skipped=829  rescued=0    ignored=2
k8s-node2                  : ok=499  changed=111  unreachable=0    failed=0    skipped=822  rescued=0    ignored=2
k8s-node3                  : ok=504  changed=112  unreachable=0    failed=0    skipped=827  rescued=0    ignored=2
k8s-node4                  : ok=437  changed=87   unreachable=0    failed=0    skipped=615  rescued=0    ignored=0

Saturday 07 February 2026  20:10:16 +0900 (0:00:00.140)       0:21:05.756 *****
===============================================================================
download : Download_file | Download item ------------------------------ 128.68s
download : Download_file | Download item ------------------------------ 111.99s
download : Download_container | Download image if required ------------- 84.87s
download : Download_file | Download item ------------------------------- 75.66s
download : Download_file | Download item ------------------------------- 69.44s
container-engine/containerd : Download_file | Download item ------------ 45.78s
download : Download_container | Download image if required ------------- 41.34s
download : Download_container | Download image if required ------------- 36.79s
download : Download_file | Download item ------------------------------- 36.44s
download : Download_container | Download image if required ------------- 36.40s
download : Download_container | Download image if required ------------- 35.93s
download : Download_file | Download item ------------------------------- 34.58s
container-engine/crictl : Download_file | Download item ---------------- 32.44s
download : Download_container | Download image if required ------------- 30.04s
download : Download_container | Download image if required ------------- 21.85s
download : Download_container | Download image if required ------------- 21.77s
container-engine/runc : Download_file | Download item ------------------ 21.39s
container-engine/nerdctl : Download_file | Download item --------------- 19.07s
download : Download_container | Download image if required ------------- 18.06s
kubernetes/kubeadm : Join to cluster if needed ------------------------- 16.17s

# facts 수집 정보 확인
root@admin-lb:~/kubespray# tree /tmp
/tmp
├── k8s-node1
├── k8s-node2
├── k8s-node3
├── k8s-node4
├── k9s_linux_arm64.tar.gz
├── LICENSE
├── README.md
├── systemd-private-f8d17866e8024c7088868da519e0e4f6-chronyd.service-4DUg97
│   └── tmp
├── systemd-private-f8d17866e8024c7088868da519e0e4f6-dbus-broker.service-mkAcPd
│   └── tmp
├── systemd-private-f8d17866e8024c7088868da519e0e4f6-irqbalance.service-ZXH9m9
│   └── tmp
├── systemd-private-f8d17866e8024c7088868da519e0e4f6-polkit.service-eqUWy2
│   └── tmp
├── systemd-private-f8d17866e8024c7088868da519e0e4f6-systemd-logind.service-KGGBUS
│   └── tmp
└── vagrant-shell

11 directories, 8 files

# etcd 백업 확인
root@admin-lb:~/kubespray# for i in {1..3}; do echo ">> k8s-node$i <<"; ssh k8s-node$i tree /var/backups; echo; done
>> k8s-node1 <<
/var/backups
└── etcd-2026-02-07_20:07:46
    ├── member
    │   ├── snap
    │   │   └── db
    │   └── wal
    │       └── 0000000000000000-0000000000000000.wal
    └── snapshot.db

5 directories, 3 files

>> k8s-node2 <<
/var/backups
└── etcd-2026-02-07_20:07:45
    ├── member
    │   ├── snap
    │   │   └── db
    │   └── wal
    │       └── 0000000000000000-0000000000000000.wal
    └── snapshot.db

5 directories, 3 files

>> k8s-node3 <<
/var/backups
└── etcd-2026-02-07_20:07:45
    ├── member
    │   ├── snap
    │   │   └── db
    │   └── wal
    │       └── 0000000000000000-0000000000000000.wal
    └── snapshot.db

5 directories, 3 files


# k8s api 호출 확인 : IP, Domain
root@admin-lb:~/kubespray# for i in {1..3}; do echo ">> k8s-node$i <<"; curl -sk https://192.168.10.1$i:6443/version | grep Version; echo; done
>> k8s-node1 <<
  "gitVersion": "v1.32.9",
  "goVersion": "go1.23.12",

>> k8s-node2 <<
  "gitVersion": "v1.32.9",
  "goVersion": "go1.23.12",

>> k8s-node3 <<
  "gitVersion": "v1.32.9",
  "goVersion": "go1.23.12",
  
# k8s admin 자격증명 확인 : 컨트롤 플레인 노드들은 apiserver 파드가 배치되어 있으니 127.0.0.1:6443 엔드포인트 설정됨
root@admin-lb:~/kubespray# for i in {1..3}; do echo ">> k8s-node$i <<"; ssh k8s-node$i kubectl cluster-info -v=6; echo; done
>> k8s-node1 <<
I0207 20:13:00.225468   26612 loader.go:402] Config loaded from file:  /root/.kube/config
I0207 20:13:00.226142   26612 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I0207 20:13:00.226162   26612 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
I0207 20:13:00.226166   26612 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I0207 20:13:00.226169   26612 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I0207 20:13:00.240440   26612 round_trippers.go:560] GET https://127.0.0.1:6443/api/v1/namespaces/kube-system/services?labelSelector=kubernetes.io%2Fcluster-service%3Dtrue 200 OK in 10 milliseconds
Kubernetes control plane is running at https://127.0.0.1:6443

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

>> k8s-node2 <<
I0207 20:13:00.590682   25937 loader.go:402] Config loaded from file:  /root/.kube/config
I0207 20:13:00.593036   25937 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I0207 20:13:00.593115   25937 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I0207 20:13:00.593126   25937 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I0207 20:13:00.593129   25937 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
I0207 20:13:00.602801   25937 round_trippers.go:560] GET https://127.0.0.1:6443/api?timeout=32s 200 OK in 9 milliseconds
I0207 20:13:00.605851   25937 round_trippers.go:560] GET https://127.0.0.1:6443/apis?timeout=32s 200 OK in 1 milliseconds
I0207 20:13:00.621066   25937 round_trippers.go:560] GET https://127.0.0.1:6443/api/v1/namespaces/kube-system/services?labelSelector=kubernetes.io%2Fcluster-service%3Dtrue 200 OK in 7 milliseconds
Kubernetes control plane is running at https://127.0.0.1:6443

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

>> k8s-node3 <<
I0207 20:13:01.031387   26097 loader.go:402] Config loaded from file:  /root/.kube/config
I0207 20:13:01.031962   26097 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I0207 20:13:01.031979   26097 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
I0207 20:13:01.031982   26097 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I0207 20:13:01.031985   26097 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I0207 20:13:01.051724   26097 round_trippers.go:560] GET https://127.0.0.1:6443/api/v1/namespaces/kube-system/services?labelSelector=kubernetes.io%2Fcluster-service%3Dtrue 200 OK in 12 milliseconds
Kubernetes control plane is running at https://127.0.0.1:6443

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

root@admin-lb:~/kubespray# mkdir /root/.kube
root@admin-lb:~/kubespray# scp k8s-node1:/root/.kube/config /root/.kube/
root@admin-lb:~/kubespray# cat /root/.kube/config | grep server
config                                                                                                     100% 5665     4.3MB/s   00:00
    server: https://127.0.0.1:6443
    
# API Server 주소를 localhost에서 컨트롤 플레인 1번 node IP로 변경 : 1번 node 장애 시, 직접 수동으로 다른 node IP 변경 필요.
root@admin-lb:~/kubespray# kubectl get node -owide -v=6
I0207 20:15:23.672088   13886 loader.go:402] Config loaded from file:  /root/.kube/config
I0207 20:15:23.672737   13886 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
I0207 20:15:23.672754   13886 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I0207 20:15:23.672844   13886 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I0207 20:15:23.672850   13886 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I0207 20:15:23.682914   13886 round_trippers.go:560] GET https://127.0.0.1:6443/api?timeout=32s 200 OK in 9 milliseconds
I0207 20:15:23.686566   13886 round_trippers.go:560] GET https://127.0.0.1:6443/apis?timeout=32s 200 OK in 2 milliseconds
I0207 20:15:23.706069   13886 round_trippers.go:560] GET https://127.0.0.1:6443/api/v1/nodes?limit=500 200 OK in 10 milliseconds
NAME        STATUS   ROLES           AGE     VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                        KERNEL-VERSION                  CONTAINER-RUNTIME
k8s-node1   Ready    control-plane   7m2s    v1.32.9   192.168.10.11   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.aarch64   containerd://2.1.5
k8s-node2   Ready    control-plane   6m51s   v1.32.9   192.168.10.12   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.aarch64   containerd://2.1.5
k8s-node3   Ready    control-plane   6m47s   v1.32.9   192.168.10.13   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.aarch64   containerd://2.1.5
k8s-node4   Ready    <none>          6m13s   v1.32.9   192.168.10.14   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.aarch64   containerd://2.1.5


root@admin-lb:~/kubespray# sed -i 's/127.0.0.1/192.168.10.11/g' /root/.kube/config

root@admin-lb:~/kubespray# kubectl get node -owide -v=6
I0207 20:15:45.768176   13892 loader.go:402] Config loaded from file:  /root/.kube/config
I0207 20:15:45.771057   13892 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I0207 20:15:45.771100   13892 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
I0207 20:15:45.771105   13892 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I0207 20:15:45.771109   13892 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I0207 20:15:45.786541   13892 round_trippers.go:560] GET https://192.168.10.11:6443/api?timeout=32s 200 OK in 15 milliseconds
I0207 20:15:45.793527   13892 round_trippers.go:560] GET https://192.168.10.11:6443/apis?timeout=32s 200 OK in 3 milliseconds
I0207 20:15:45.812924   13892 round_trippers.go:560] GET https://192.168.10.11:6443/api/v1/nodes?limit=500 200 OK in 8 milliseconds
NAME        STATUS   ROLES           AGE     VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                        KERNEL-VERSION                  CONTAINER-RUNTIME
k8s-node1   Ready    control-plane   7m24s   v1.32.9   192.168.10.11   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.aarch64   containerd://2.1.5
k8s-node2   Ready    control-plane   7m13s   v1.32.9   192.168.10.12   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.aarch64   containerd://2.1.5
k8s-node3   Ready    control-plane   7m9s    v1.32.9   192.168.10.13   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.aarch64   containerd://2.1.5
k8s-node4   Ready    <none>          6m35s   v1.32.9   192.168.10.14   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.aarch64   containerd://2.1.5

# 노드별 파드 CIDR 확인
root@admin-lb:~/kubespray# kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.podCIDR}{"\n"}{end}'
k8s-node1	10.233.64.0/24
k8s-node2	10.233.65.0/24
k8s-node3	10.233.66.0/24
k8s-node4	10.233.67.0/24

# etcd 정보 확인 : etcd name 확인
root@admin-lb:~/kubespray# ssh k8s-node1 etcdctl.sh member list -w table
+------------------+---------+-------+----------------------------+----------------------------+------------+
|        ID        | STATUS  | NAME  |         PEER ADDRS         |        CLIENT ADDRS        | IS LEARNER |
+------------------+---------+-------+----------------------------+----------------------------+------------+
|  8b0ca30665374b0 | started | etcd3 | https://192.168.10.13:2380 | https://192.168.10.13:2379 |      false |
| 2106626b12a4099f | started | etcd2 | https://192.168.10.12:2380 | https://192.168.10.12:2379 |      false |
| c6702130d82d740f | started | etcd1 | https://192.168.10.11:2380 | https://192.168.10.11:2379 |      false |
+------------------+---------+-------+----------------------------+----------------------------+------------+

root@admin-lb:~/kubespray# for i in {1..3}; do echo ">> k8s-node$i <<"; ssh k8s-node$i etcdctl.sh endpoint status -w table; echo; done
>> k8s-node1 <<
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|    ENDPOINT    |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 127.0.0.1:2379 | c6702130d82d740f |  3.5.25 |  5.6 MB |      true |      false |         4 |       2687 |               2687 |        |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

>> k8s-node2 <<
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|    ENDPOINT    |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 127.0.0.1:2379 | 2106626b12a4099f |  3.5.25 |  5.6 MB |     false |      false |         4 |       2687 |               2687 |        |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

>> k8s-node3 <<
+----------------+-----------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|    ENDPOINT    |       ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------+-----------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 127.0.0.1:2379 | 8b0ca30665374b0 |  3.5.25 |  5.6 MB |     false |      false |         4 |       2687 |               2687 |        |
+----------------+-----------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

# control 컴포넌트 확인

# 인증서 정보 확인
# 인증서 정보 확인 : ctrl1번 노드만 super-admin.conf 확인
root@admin-lb:~/kubespray# for i in {1..3}; do echo ">> k8s-node$i <<"; ssh k8s-node$i ls -l /etc/kubernetes/super-admin.conf ; echo; done
>> k8s-node1 <<
-rw-------. 1 root root 5693 Feb  7 20:08 /etc/kubernetes/super-admin.conf

>> k8s-node2 <<
ls: cannot access '/etc/kubernetes/super-admin.conf': No such file or directory

>> k8s-node3 <<
ls: cannot access '/etc/kubernetes/super-admin.conf': No such file or directory

# kubespray task에 의해서 호스트에서도 서비스명 도메인 질의를 위해 ns 최상단 추가 등 확인
root@admin-lb:~/kubespray# ssh k8s-node1 cat /etc/resolv.conf
# Generated by NetworkManager
search default.svc.cluster.local svc.cluster.local
nameserver 10.233.0.3
nameserver 168.126.63.1
nameserver 168.126.63.2
options ndots:2 timeout:2 attempts:2

## 현재 k8s join 되지 않는 노드는 기본 dns 설정 상태
root@admin-lb:~/kubespray# ssh k8s-node5 cat /etc/resolv.conf
# Generated by NetworkManager
nameserver 168.126.63.1
nameserver 168.126.63.2

# kubeadm 과 동일하게 kubelet node 최초 join 시 CSR 사용 확인
root@admin-lb:~/kubespray# kubectl get csr
NAME        AGE   SIGNERNAME                                    REQUESTOR                 REQUESTEDDURATION   CONDITION
csr-6k48r   19m   kubernetes.io/kube-apiserver-client-kubelet   system:node:k8s-node1     <none>              Approved,Issued
csr-gb9vm   19m   kubernetes.io/kube-apiserver-client-kubelet   system:bootstrap:kum2ky   <none>              Approved,Issued
csr-knbps   19m   kubernetes.io/kube-apiserver-client-kubelet   system:bootstrap:exiimk   <none>              Approved,Issued
csr-s67nd   18m   kubernetes.io/kube-apiserver-client-kubelet   system:bootstrap:qfuixm   <none>              Approved,Issued

K8s API 엔드포인트

[Case1] HA 컨트플 플레인 노드(3대) + (Worker Client-Side LoadBalancing)

# worker(kubeclt, kube-proxy) -> k8s api

# 워커노드에서 정보 확인
root@admin-lb:~/kubespray# ssh k8s-node4 crictl ps
CONTAINER           IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID              POD                               NAMESPACE
5a8890e37eb3e       bc6c1e09a843d       22 minutes ago      Running             metrics-server      0                   3378bf7da8a54       metrics-server-65fdf69dcb-22c5b   kube-system
62cc3eb40ea31       2f6c962e7b831       23 minutes ago      Running             coredns             0                   91d5017d28880       coredns-664b99d7c7-pztdv          kube-system
0269a1164273d       cadcae92e6360       23 minutes ago      Running             kube-flannel        0                   e31417e29224a       kube-flannel-ds-arm64-7cvdk       kube-system
080f1a580e301       72b57ec14d31e       23 minutes ago      Running             kube-proxy          0                   975654efc1267       kube-proxy-ngmpt                  kube-system
05ea62f151eb2       5a91d90f47ddf       23 minutes ago      Running             nginx-proxy         0                   0529e6ee0ed27       nginx-proxy-k8s-node4             kube-system


root@admin-lb:~/kubespray# ssh k8s-node4 cat /etc/nginx/nginx.conf
error_log stderr notice;

worker_processes 2;
worker_rlimit_nofile 130048;
worker_shutdown_timeout 10s;

events {
  multi_accept on;
  use epoll;
  worker_connections 16384;
}

stream {
  upstream kube_apiserver {
    least_conn;
    server 192.168.10.11:6443;
    server 192.168.10.12:6443;
    server 192.168.10.13:6443;
    }

  server {
    listen        127.0.0.1:6443;
    proxy_pass    kube_apiserver;
    proxy_timeout 10m;
    proxy_connect_timeout 1s;
  }
}

http {
  aio threads;
  aio_write on;
  tcp_nopush on;
  tcp_nodelay on;

  keepalive_timeout 5m;
  keepalive_requests 100;
  reset_timedout_connection on;
  server_tokens off;
  autoindex off;

  server {
    listen 8081;
    location /healthz {
      access_log off;
      return 200;
    }
    location /stub_status {
      stub_status on;
      access_log off;
    }
  }
  }
  
root@admin-lb:~/kubespray# ssh k8s-node4 curl -s localhost:8081/healthz -I
HTTP/1.1 200 OK
Server: nginx
Date: Sat, 07 Feb 2026 11:34:11 GMT
Content-Type: text/plain
Content-Length: 0
Connection: keep-alive

root@admin-lb:~/kubespray# ssh k8s-node4 curl -sk https://127.0.0.1:6443/version | grep Version
  "gitVersion": "v1.32.9",
  "goVersion": "go1.23.12",
  
root@admin-lb:~/kubespray# ssh k8s-node4 ss -tnlp | grep nginx
LISTEN 0      511          0.0.0.0:8081       0.0.0.0:*    users:(("nginx",pid=15051,fd=6),("nginx",pid=15050,fd=6),("nginx",pid=15023,fd=6))
LISTEN 0      511        127.0.0.1:6443       0.0.0.0:*    users:(("nginx",pid=15051,fd=5),("nginx",pid=15050,fd=5),("nginx",pid=15023,fd=5))

# kubelet(client) -> api-server 호출 시 엔드포인트 정보 확인 : https://localhost:6443
root@admin-lb:~/kubespray# ssh k8s-node4 cat /etc/kubernetes/kubelet.conf
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURCVENDQWUyZ0F3SUJBZ0lJYkQzaEF0Si9CYk13RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TmpBeU1EY3hNVEF6TVRWYUZ3MHpOakF5TURVeE1UQTRNVFZhTUJVeApFekFSQmdOVkJBTVRDbXQxWW1WeWJtVjBaWE13Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLCkFvSUJBUUNmOVlOVklZcnVPSVlmUVFlUnB3em1ORjdyb0lxTlhYT29IWC84U0RhcTUzN1Y4SkJnaUl3L3MwWmgKZ04vekk3N28rV1dTZ3Y0UzVIR3NIR29XWnhRNE9xejAvRS9jUFRJK1Q5ZzFhZFBXSDNHSFN6WS8wNWkya1hzagpsNGtHV0xiTVZYRHlMNW5jZFJpY3N2a2ZqTVJzQVpld2tXK0taUjl2aGJMTXh5M0tqUEd5cS9VVVJtTE1JbUxEClg2WHY4MHZVd2h5RjNUaEFCVExGRjE2Um9BaEJlTW53c3RvdkVDelVVQVExOTRlbFFHL2NhWXZsREl1VkVGcGQKckxxY29FdG5YRHp3dzN1SXUzaVdSZjg3Q2VCMVUyY1hrUmllY1k4elh0ZHJST2FpKzl5ZHRxMEkwOGExazN6ZQpVNDVTYUVlaUdQaGVUSWcvVVdEcDJqYzRBSXI5QWdNQkFBR2pXVEJYTUE0R0ExVWREd0VCL3dRRUF3SUNwREFQCkJnTlZIUk1CQWY4RUJUQURBUUgvTUIwR0ExVWREZ1FXQkJUREwwRVh4d3FaK2ttOEJwYnNPL3JpQXJtSjV6QVYKQmdOVkhSRUVEakFNZ2dwcmRXSmxjbTVsZEdWek1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQnJOVUVyZWp4MwpHN0F4WmtwODFqRVBxbk5LV3BoV2Jkc1ZKc3JDU1VySGRwTWkvNm9vZ0VSQ2tHdXArV1RheTVneFU3M3krVStsCjZIUTltcytQRlo5VU4zUjVYOTJZRkU4bmZhQXhvM01YeXFrd2JGTW5ISWlqSS9vMGMzZk9ObFRFWC9URm9wSDEKSVM1eWRxZ3Fxd0k4VmZkeHJ5Ui9LUHVFS3JVQ0lpcGVaVFc0RHRnMGZiY0VBMVJ2UEpoVXhQOHppTTNCYmFRdApUcDZlZHRJYjFqaDREUmxLOXUvcHQrR2R3ZWxNSVpTRHhXc3JGTGNSVEd6R2xFRnBMbFBFNVF4SnB4bVcwTWdxCk1PSkRHQ0NETVc5a2tjeEs0SnVjaG10bEwzYngwVHdVMWsrTXhoSE05YVcrT1dMOEgrRU14RnVyMFNnVlZGU2oKVW9zQU9KdURYMXNLCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    server: https://localhost:6443
  name: default-cluster
contexts:
- context:
    cluster: default-cluster
    namespace: default
    user: default-auth
  name: default-context
current-context: default-context
kind: Config
preferences: {}
users:
- name: default-auth
  user:
    client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
    client-key: /var/lib/kubelet/pki/kubelet-client-current.pem
    
# kube-proxy(client) -> api-server 호출 시 엔드포인트 정보 확인
root@admin-lb:~/kubespray# kc get cm -n kube-system kube-proxy -o yaml
apiVersion: v1
data:
  config.conf: |-
    apiVersion: kubeproxy.config.k8s.io/v1alpha1
    bindAddress: 0.0.0.0
    bindAddressHardFail: false
    clientConnection:
      acceptContentTypes: ""
      burst: 10
      contentType: application/vnd.kubernetes.protobuf
      kubeconfig: /var/lib/kube-proxy/kubeconfig.conf
      qps: 5
    clusterCIDR: 10.233.64.0/18
    configSyncPeriod: 15m0s
    conntrack:
      maxPerCore: 32768
      min: 131072
      tcpBeLiberal: false
      tcpCloseWaitTimeout: 1h0m0s
      tcpEstablishedTimeout: 24h0m0s
      udpStreamTimeout: 0s
      udpTimeout: 0s
    detectLocal:
      bridgeInterface: ""
      interfaceNamePrefix: ""
    detectLocalMode: ""
    enableProfiling: false
    healthzBindAddress: 0.0.0.0:10256
    hostnameOverride: k8s-node1
    iptables:
      localhostNodePorts: null
      masqueradeAll: false
      masqueradeBit: 14
      minSyncPeriod: 0s
      syncPeriod: 30s
    ipvs:
      excludeCIDRs: []
      minSyncPeriod: 0s
      scheduler: rr
      strictARP: false
      syncPeriod: 30s
      tcpFinTimeout: 0s
      tcpTimeout: 0s
      udpTimeout: 0s
    kind: KubeProxyConfiguration
    logging:
      flushFrequency: 0
      options:
        json:
          infoBufferSize: "0"
        text:
          infoBufferSize: "0"
      verbosity: 0
    metricsBindAddress: 127.0.0.1:10249
    mode: iptables
    nftables:
      masqueradeAll: false
      masqueradeBit: null
      minSyncPeriod: 0s
      syncPeriod: 0s
    nodePortAddresses: []
    oomScoreAdj: -999
    portRange: ""
    showHiddenMetricsForVersion: ""
    winkernel:
      enableDSR: false
      forwardHealthCheckVip: false
      networkName: ""
      rootHnsEndpointName: ""
      sourceVip: ""
  kubeconfig.conf: |-
    apiVersion: v1
    kind: Config
    clusters:
    - cluster:
        certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        server: https://127.0.0.1:6443
      name: default
    contexts:
    - context:
        cluster: default
        namespace: default
        user: default
      name: default
    current-context: default
    users:
    - name: default
      user:
        tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
kind: ConfigMap
metadata:
  creationTimestamp: "2026-02-07T11:08:24Z"
  labels:
    app: kube-proxy
  name: kube-proxy
  namespace: kube-system
  resourceVersion: "686"
  uid: 75f71f01-34a8-4a84-92b4-a823f4032c7c
  
root@admin-lb:~/kubespray# kubectl get cm -n kube-system kube-proxy -o yaml | grep 'kubeconfig.conf:' -A18
  kubeconfig.conf: |-
    apiVersion: v1
    kind: Config
    clusters:
    - cluster:
        certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        server: https://127.0.0.1:6443
      name: default
    contexts:
    - context:
        cluster: default
        namespace: default
        user: default
      name: default
    current-context: default
    users:
    - name: default
      user:
        tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
        
# nginx.conf 생성 Task
root@admin-lb:~/kubespray# tree roles/kubernetes/node/tasks/loadbalancer
roles/kubernetes/node/tasks/loadbalancer
├── haproxy.yml
├── kube-vip.yml
└── nginx-proxy.yml

1 directory, 3 files

root@admin-lb:~/kubespray# cat roles/kubernetes/node/tasks/loadbalancer/nginx-proxy.yml
---
- name: Haproxy | Cleanup potentially deployed haproxy
  file:
    path: "{{ kube_manifest_dir }}/haproxy.yml"
    state: absent

- name: Nginx-proxy | Make nginx directory
  file:
    path: "{{ nginx_config_dir }}"
    state: directory
    mode: "0700"
    owner: root

- name: Nginx-proxy | Write nginx-proxy configuration
  template:
    src: "loadbalancer/nginx.conf.j2"
    dest: "{{ nginx_config_dir }}/nginx.conf"
    owner: root
    mode: "0755"
    backup: true

- name: Nginx-proxy | Get checksum from config
  stat:
    path: "{{ nginx_config_dir }}/nginx.conf"
    get_attributes: false
    get_checksum: true
    get_mime: false
  register: nginx_stat

- name: Nginx-proxy | Write static pod
  template:
    src: manifests/nginx-proxy.manifest.j2
    dest: "{{ kube_manifest_dir }}/nginx-proxy.yml"
    mode: "0640"

# nginx.conf jinja2 템플릿 파일
root@admin-lb:~/kubespray# cat roles/kubernetes/node/templates/loadbalancer/nginx.conf.j2
error_log stderr notice;

worker_processes 2;
worker_rlimit_nofile 130048;
worker_shutdown_timeout 10s;

events {
  multi_accept on;
  use epoll;
  worker_connections 16384;
}

stream {
  upstream kube_apiserver {
    least_conn;
    {% for host in groups['kube_control_plane'] -%}
    server {{ hostvars[host]['main_access_ip'] | ansible.utils.ipwrap }}:{{ kube_apiserver_port }};
    {% endfor -%}
  }

  server {
    listen        127.0.0.1:{{ loadbalancer_apiserver_port|default(kube_apiserver_port) }};
    {% if ipv6_stack -%}
    listen        [::1]:{{ loadbalancer_apiserver_port|default(kube_apiserver_port) }};
    {% endif -%}
    proxy_pass    kube_apiserver;
    proxy_timeout 10m;
    proxy_connect_timeout 1s;
  }
}

http {
  aio threads;
  aio_write on;
  tcp_nopush on;
  tcp_nodelay on;

  keepalive_timeout {{ loadbalancer_apiserver_keepalive_timeout }};
  keepalive_requests 100;
  reset_timedout_connection on;
  server_tokens off;
  autoindex off;

  {% if loadbalancer_apiserver_healthcheck_port is defined -%}
  server {
    listen {{ loadbalancer_apiserver_healthcheck_port }};
    {% if ipv6_stack -%}
    listen [::]:{{ loadbalancer_apiserver_healthcheck_port }};
    {% endif -%}
    location /healthz {
      access_log off;
      return 200;
    }
    location /stub_status {
      stub_status on;
      access_log off;
    }
  }
  {% endif %}
}

# nginx static pod 매니페스트 파일 확인
root@admin-lb:~/kubespray# cat roles/kubernetes/node/templates/manifests/nginx-proxy.manifest.j2
apiVersion: v1
kind: Pod
metadata:
  name: {{ loadbalancer_apiserver_pod_name }}
  namespace: kube-system
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    k8s-app: kube-nginx
  annotations:
    nginx-cfg-checksum: "{{ nginx_stat.stat.checksum }}"
spec:
  hostNetwork: true
  dnsPolicy: ClusterFirstWithHostNet
  nodeSelector:
    kubernetes.io/os: linux
  priorityClassName: system-node-critical
  containers:
  - name: nginx-proxy
    image: {{ nginx_image_repo }}:{{ nginx_image_tag }}
    imagePullPolicy: {{ k8s_image_pull_policy }}
    resources:
      requests:
        cpu: {{ loadbalancer_apiserver_cpu_requests }}
        memory: {{ loadbalancer_apiserver_memory_requests }}
    {% if loadbalancer_apiserver_healthcheck_port is defined -%}
    livenessProbe:
      httpGet:
        path: /healthz
        port: {{ loadbalancer_apiserver_healthcheck_port }}
    readinessProbe:
      httpGet:
        path: /healthz
        port: {{ loadbalancer_apiserver_healthcheck_port }}
    {% endif -%}
    volumeMounts:
    - mountPath: /etc/nginx
      name: etc-nginx
      readOnly: true
  volumes:
  - name: etc-nginx
    hostPath:
      path: {{ nginx_config_dir }}


# nginx log 중 alert 해결 : --tags "containerd" 사용
root@admin-lb:~/kubespray# kubectl logs -n kube-system nginx-proxy-k8s-node4
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf is not a file or does not exist
/docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2026/02/07 11:09:10 [notice] 1#1: using the "epoll" event method
2026/02/07 11:09:10 [notice] 1#1: nginx/1.28.0
2026/02/07 11:09:10 [notice] 1#1: built by gcc 14.2.0 (Alpine 14.2.0)
2026/02/07 11:09:10 [notice] 1#1: OS: Linux 6.12.0-55.39.1.el10_0.aarch64
2026/02/07 11:09:10 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 65535:65535
2026/02/07 11:09:10 [notice] 1#1: start worker processes
2026/02/07 11:09:10 [notice] 1#1: start worker process 20
2026/02/07 11:09:10 [notice] 1#1: start worker process 21
2026/02/07 11:09:10 [alert] 20#20: setrlimit(RLIMIT_NOFILE, 130048) failed (1: Operation not permitted)
2026/02/07 11:09:10 [alert] 21#21: setrlimit(RLIMIT_NOFILE, 130048) failed (1: Operation not permitted)
2026/02/07 11:10:15 [error] 20#20: *49 recv() failed (104: Connection reset by peer) while proxying and reading from upstream, client: 127.0.0.1, server: 127.0.0.1:6443, upstream: "192.168.10.13:6443", bytes from/to client:1489/0, bytes from/to upstream:0/1489

root@admin-lb:~/kubespray# ssh k8s-node4 cat /etc/nginx/nginx.conf
error_log stderr notice;

worker_processes 2;
worker_rlimit_nofile 130048;
worker_shutdown_timeout 10s;

events {
  multi_accept on;
  use epoll;
  worker_connections 16384;
}

stream {
  upstream kube_apiserver {
    least_conn;
    server 192.168.10.11:6443;
    server 192.168.10.12:6443;
    server 192.168.10.13:6443;
    }

  server {
    listen        127.0.0.1:6443;
    proxy_pass    kube_apiserver;
    proxy_timeout 10m;
    proxy_connect_timeout 1s;
  }
}

http {
  aio threads;
  aio_write on;
  tcp_nopush on;
  tcp_nodelay on;

  keepalive_timeout 5m;
  keepalive_requests 100;
  reset_timedout_connection on;
  server_tokens off;
  autoindex off;

  server {
    listen 8081;
    location /healthz {
      access_log off;
      return 200;
    }
    location /stub_status {
      stub_status on;
      access_log off;
    }
  }
  }
  
root@admin-lb:~/kubespray# ssh k8s-node4 cat /etc/containerd/config.toml | grep base_runtime_spec
         base_runtime_spec = "/etc/containerd/cri-base.json"

root@admin-lb:~/kubespray# ssh k8s-node4 cat /etc/containerd/cri-base.json | jq | grep rlimits -A 6
    "rlimits": [
      {
        "type": "RLIMIT_NOFILE",
        "hard": 65535,
        "soft": 65535
      }
    ],
    

# 관련 변수명 확인
root@admin-lb:~/kubespray# cat roles/container-engine/containerd/defaults/main.yml
...
containerd_base_runtime_spec_rlimit_nofile: 65535
containerd_default_base_runtime_spec_patch:
  process:
    rlimits:
      - type: RLIMIT_NOFILE
        hard: "{{ containerd_base_runtime_spec_rlimit_nofile }}"
        soft: "{{ containerd_base_runtime_spec_rlimit_nofile }}"
        
# 기본 OCI Spec(Runtime Spec)을 수정(Patch)
root@admin-lb:~/kubespray# cat << EOF >> inventory/mycluster/group_vars/all/containerd.yml
containerd_default_base_runtime_spec_patch:
  process:
    rlimits: []
EOF
root@admin-lb:~/kubespray# grep "^[^#]" inventory/mycluster/group_vars/all/containerd.yml
---
containerd_default_base_runtime_spec_patch:
  process:
    rlimits: []
    

# (신규터미널) 모니터링
root@k8s-node4:~# crictl pods --namespace kube-system --name 'nginx-proxy-*' -q | xargs crictl rmp -f
Stopped sandbox 0529e6ee0ed270e3c19c8d4b8f35387291e172e3ffbd64987e6a3707b0e221e8
Removed sandbox 0529e6ee0ed270e3c19c8d4b8f35387291e172e3ffbd64987e6a3707b0e221e8
root@k8s-node4:~# ssh k8s-node4 crictl inspect --name nginx-proxy | grep rlimits -A6
root@k8s-node4's password:
        "rlimits": [
          {
            "hard": 65535,
            "soft": 65535,
            "type": "RLIMIT_NOFILE"
          }
        ],
        
root@admin-lb:~/kubespray# kubectl logs -n kube-system nginx-proxy-k8s-node4 -f
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf is not a file or does not exist
/docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2026/02/07 13:30:02 [notice] 1#1: using the "epoll" event method
2026/02/07 13:30:02 [notice] 1#1: nginx/1.28.0
2026/02/07 13:30:02 [notice] 1#1: built by gcc 14.2.0 (Alpine 14.2.0)
2026/02/07 13:30:02 [notice] 1#1: OS: Linux 6.12.0-55.39.1.el10_0.aarch64
2026/02/07 13:30:02 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 65535:65535
2026/02/07 13:30:02 [notice] 1#1: start worker processes
2026/02/07 13:30:02 [notice] 1#1: start worker process 20
2026/02/07 13:30:02 [alert] 20#20: setrlimit(RLIMIT_NOFILE, 130048) failed (1: Operation not permitted)
2026/02/07 13:30:02 [notice] 1#1: start worker process 21

[Case2] External LB → HA 컨트플 플레인 노드(3대) + (Worker Client-Side LoadBalancing)

# apiserver static 파드의 bind-address 에 '::' 확인 
root@admin-lb:~/kubespray# kubectl describe pod -n kube-system kube-apiserver-k8s-node1 | grep -E 'address|secure-port'
Annotations:          kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.10.11:6443
      --advertise-address=192.168.10.11
      --bind-address=::
      --kubelet-preferred-address-types=InternalDNS,InternalIP,Hostname,ExternalDNS,ExternalIP
      --secure-port=6443
      
root@admin-lb:~/kubespray# ssh k8s-node1 ss -tnlp | grep 6443
LISTEN 0      4096               *:6443             *:*    users:(("kube-apiserver",pid=25837,fd=3))
root@admin-lb:~/kubespray# ssh k8s-node1 cat /etc/kubernetes/admin.conf | grep server
    server: https://127.0.0.1:6443
    
    
# admin 자격증명(client) -> api-server 호출 시 엔드포인트 정보 확인
root@admin-lb:~/kubespray# ssh k8s-node1 cat /etc/kubernetes/admin.conf | grep server
    server: https://127.0.0.1:6443
    
# super-admin 자격증명(client) -> api-server 호출 시 엔드포인트 정보 확인
root@admin-lb:~/kubespray# ssh k8s-node1 cat /etc/kubernetes/super-admin.conf | grep server
    server: https://192.168.10.11:6443
    
# kubelet(client) -> api-server 호출 시 엔드포인트 정보 확인 : https://127.0.0.1:6443
root@admin-lb:~/kubespray# ssh k8s-node1 cat /etc/kubernetes/kubelet.conf | grep server
    server: https://127.0.0.1:6443
    
# kube-proxy(client) -> api-server 호출 시 엔드포인트 정보 확인
root@admin-lb:~/kubespray# k get cm -n kube-system kube-proxy -o yaml | grep server
        server: https://127.0.0.1:6443

# kube-controller-manager(client) -> api-server 호출 시 엔드포인트 정보 확인
root@admin-lb:~/kubespray# ssh k8s-node1 cat /etc/kubernetes/controller-manager.conf | grep server
    server: https://127.0.0.1:6443
    
# kube-scheduler(client) -> api-server 호출 시 엔드포인트 정보 확인
root@admin-lb:~/kubespray# ssh k8s-node1 cat /etc/kubernetes/scheduler.conf | grep server
    server: https://127.0.0.1:6443

Cilium 과 같은 데몬셋에서 배포된 파드들이 k8s api endpoint 호출을 위해서, 동일한 127.0.0.1::6443 로컬 엔드포인트도 사용 가능합니다.

# kube-ops-view 설치
root@admin-lb:~/kubespray# helm repo add geek-cookbook https://geek-cookbook.github.io/charts/
"geek-cookbook" has been added to your repositories

root@admin-lb:~/kubespray# helm install kube-ops-view geek-cookbook/kube-ops-view --version 1.2.2 \
  --set service.main.type=NodePort,service.main.ports.http.nodePort=30000 \
  --set env.TZ="Asia/Seoul" --namespace kube-system \
  --set image.repository="abihf/kube-ops-view" --set image.tag="latest"
NAME: kube-ops-view
LAST DEPLOYED: Sat Feb  7 22:39:40 2026
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
  export NODE_PORT=$(kubectl get --namespace kube-system -o jsonpath="{.spec.ports[0].nodePort}" services kube-ops-view)
  export NODE_IP=$(kubectl get nodes --namespace kube-system -o jsonpath="{.items[0].status.addresses[0].address}")
  echo http://$NODE_IP:$NODE_PORT

root@admin-lb:~/kubespray# kubectl get deploy,pod,svc,ep -n kube-system -l app.kubernetes.io/instance=kube-ops-view
NAME                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kube-ops-view   0/1     1            0           7s

NAME                                 READY   STATUS              RESTARTS   AGE
pod/kube-ops-view-8484bdc5df-zz47d   0/1     ContainerCreating   0          7s

NAME                    TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
service/kube-ops-view   NodePort   10.233.49.26   <none>        8080:30000/TCP   7s

NAME                      ENDPOINTS   AGE
endpoints/kube-ops-view   <none>      7s


# 샘플 애플리케이션 배포, 반복 호출
root@admin-lb:~/kubespray# cat << EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webpod
spec:
  replicas: 2
  selector:
    matchLabels:
      app: webpod
  template:
    metadata:
      labels:
        app: webpod
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - sample-app
            topologyKey: "kubernetes.io/hostname"
      containers:
      - name: webpod
        image: traefik/whoami
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: webpod
  labels:
    app: webpod
EOFype: NodePort003
deployment.apps/webpod created
service/webpod created

# 배포 확인
root@admin-lb:~/kubespray# kubectl get deploy,svc,ep webpod -owide
NAME                     READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES           SELECTOR
deployment.apps/webpod   2/2     2            2           28s   webpod       traefik/whoami   app=webpod

NAME             TYPE       CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGE   SELECTOR
service/webpod   NodePort   10.233.11.3   <none>        80:30003/TCP   27s   app=webpod

NAME               ENDPOINTS                       AGE
endpoints/webpod   10.233.67.5:80,10.233.67.6:80   28s

[admin-lb] # IP는 node 작업에 따라 변경
root@admin-lb:~/kubespray# while true; do curl -s http://192.168.10.14:30003 | grep Hostname; sleep 1; done
Hostname: webpod-697b545f57-srd2b
Hostname: webpod-697b545f57-srd2b
Hostname: webpod-697b545f57-srd2b
Hostname: webpod-697b545f57-srd2b

# 성공
root@admin-lb:~/kubespray# ssh k8s-node1 curl -s webpod -I
HTTP/1.1 200 OK
Date: Sat, 07 Feb 2026 13:41:49 GMT
Content-Length: 202
Content-Type: text/plain; charset=utf-8

root@admin-lb:~/kubespray# ssh k8s-node1 curl -s webpod.default -I
HTTP/1.1 200 OK
Date: Sat, 07 Feb 2026 13:42:09 GMT
Content-Length: 210
Content-Type: text/plain; charset=utf-8

# 실패
root@admin-lb:~/kubespray# ssh k8s-node1 curl -s webpod.default.svc -I
root@admin-lb:~/kubespray# ssh k8s-node1 curl -s webpod.default.svc.cluster -I

1번 노드로만 들어옴

# 장애 재현 만약 컨트롤 플레인 1번 노드 장애 발생 시 영향도
# [admin-lb] kubeconfig 자격증명 사용 시 정보 확인
root@admin-lb:~/kubespray# cat /root/.kube/config | grep server
    server: https://192.168.10.11:6443

Every 2.0s: kubectl get pod -n kube-system                                                                k8s-node2: Sat Feb  7 22:44:20 2026


## [k8s-node2]
NAME                                READY   STATUS    RESTARTS   AGE
coredns-664b99d7c7-pztdv            1/1     Running   0          154m
coredns-664b99d7c7-r4nzz            1/1     Running   0          154m
kube-apiserver-k8s-node1            1/1     Running   1          155m
kube-apiserver-k8s-node2            1/1     Running   1          155m
kube-apiserver-k8s-node3            1/1     Running   1          155m
kube-controller-manager-k8s-node1   1/1     Running   2          155m
kube-controller-manager-k8s-node2   1/1     Running   2          155m
kube-controller-manager-k8s-node3   1/1     Running   2          155m
kube-flannel-ds-arm64-2mwzk         1/1     Running   0          154m
kube-flannel-ds-arm64-7cvdk         1/1     Running   0          154m
kube-flannel-ds-arm64-fss94         1/1     Running   0          154m
kube-flannel-ds-arm64-p87lw         1/1     Running   0          154m
kube-ops-view-8484bdc5df-zz47d      1/1     Running   0          4m46s
kube-proxy-8pjq2                    1/1     Running   0          155m
kube-proxy-bfbbn                    1/1     Running   0          155m
kube-proxy-ngmpt                    1/1     Running   0          155m
kube-proxy-twxh7                    1/1     Running   0          155m
kube-scheduler-k8s-node1            1/1     Running   1          155m
kube-scheduler-k8s-node2            1/1     Running   1          155m
kube-scheduler-k8s-node3            1/1     Running   1          155m
metrics-server-65fdf69dcb-22c5b     1/1     Running   0          154m
nginx-proxy-k8s-node4               1/1     Running   1          155m

root@k8s-node1:~# poweroff


Every 2.0s: kubectl get pod -n kube-system                                                                k8s-node2: Sat Feb  7 22:49:57 2026

NAME                                READY   STATUS       RESTARTS   AGE
coredns-664b99d7c7-pztdv            1/1     Running      0	    160m
coredns-664b99d7c7-r4nzz            1/1     Running      0	    160m
kube-apiserver-k8s-node1            1/1     Terminated   1	    161m
kube-apiserver-k8s-node2            1/1     Running      1	    161m
kube-apiserver-k8s-node3            1/1     Running      1	    161m
kube-controller-manager-k8s-node1   1/1     Terminated   2	    161m
kube-controller-manager-k8s-node2   1/1     Running      2	    161m
kube-controller-manager-k8s-node3   1/1     Running      2	    161m
kube-flannel-ds-arm64-2mwzk         1/1     Running      0	    160m
kube-flannel-ds-arm64-7cvdk         1/1     Running      0	    160m
kube-flannel-ds-arm64-fss94         1/1     Running      0	    160m
kube-flannel-ds-arm64-p87lw         1/1     Running      0	    160m
kube-ops-view-8484bdc5df-zz47d      1/1     Running      0	    10m
kube-proxy-8pjq2                    1/1     Running      0	    160m

root@k8s-node2:~# kubectl logs -n kube-system nginx-proxy-k8s-node4 -f
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf is not a file or does not exist
/docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2026/02/07 13:30:02 [notice] 1#1: using the "epoll" event method
2026/02/07 13:30:02 [notice] 1#1: nginx/1.28.0
2026/02/07 13:30:02 [notice] 1#1: built by gcc 14.2.0 (Alpine 14.2.0)
2026/02/07 13:30:02 [notice] 1#1: OS: Linux 6.12.0-55.39.1.el10_0.aarch64
2026/02/07 13:30:02 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 65535:65535
2026/02/07 13:30:02 [notice] 1#1: start worker processes
2026/02/07 13:30:02 [notice] 1#1: start worker process 20
2026/02/07 13:30:02 [alert] 20#20: setrlimit(RLIMIT_NOFILE, 130048) failed (1: Operation not permitted)
2026/02/07 13:30:02 [notice] 1#1: start worker process 21
2026/02/07 13:30:02 [alert] 21#21: setrlimit(RLIMIT_NOFILE, 130048) failed (1: Operation not permitted)

# [k8s-node4] 하지만 백엔드 대상 서버가 나머지 2대가 있으니 아래 요청 처리 정상!
root@k8s-node4:~# while true; do curl -sk https://127.0.0.1:6443/version | grep gitVersion ; date; sleep 1; echo ; done
  "gitVersion": "v1.32.9",
Sat Feb  7 10:50:54 PM KST 2026

  "gitVersion": "v1.32.9",
Sat Feb  7 10:50:55 PM KST 2026

  "gitVersion": "v1.32.9",
Sat Feb  7 10:50:56 PM KST 2026


# [admin-lb] 아래 자격증명 서버 정보 수정 필요
root@admin-lb:~# while true; do kubectl get node ; echo ; curl -sk https://192.168.10.12:6443/version | grep gitVersion ; sleep 1; echo ; done
Unable to connect to the server: dial tcp 192.168.10.11:6443: connect: no route to host

  "gitVersion": "v1.32.9",


root@admin-lb:~# sed -i 's/192.168.10.11/192.168.10.12/g' /root/.kube/config

root@admin-lb:~# while true; do kubectl get node ; echo ; curl -sk https://192.168.10.12:6443/version | grep gitVersion ; sleep 1; echo ; done
NAME        STATUS     ROLES           AGE    VERSION
k8s-node1   NotReady   control-plane   164m   v1.32.9
k8s-node2   Ready      control-plane   164m   v1.32.9
k8s-node3   Ready      control-plane   164m   v1.32.9
k8s-node4   Ready      <none>          163m   v1.32.9

  "gitVersion": "v1.32.9",

External LB → HA 컨트플 플레인 노드(3대) : k8s apiserver 호출 설정

root@admin-lb:~/kubespray# curl -sk https://192.168.10.10:6443/version | grep gitVersion
  "gitVersion": "v1.32.9",
  
root@admin-lb:~/kubespray# sed -i 's/192.168.10.12/192.168.10.10/g' /root/.kube/config

# 인증서 SAN list 확인
root@admin-lb:~/kubespray# kubectl get node
E0207 22:57:29.198183   17989 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://192.168.10.10:6443/api?timeout=32s\": tls: failed to verify certificate: x509: certificate is valid for 10.233.0.1, 192.168.10.13, 192.168.10.11, 127.0.0.1, ::1, 192.168.10.12, 10.0.2.15, fd17:625c:f037:2:a00:27ff:fe90:eaeb, not 192.168.10.10"
E0207 22:57:29.207284   17989 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://192.168.10.10:6443/api?timeout=32s\": tls: failed to verify certificate: x509: certificate is valid for 10.233.0.1, 192.168.10.11, 127.0.0.1, ::1, 192.168.10.12, 192.168.10.13, 10.0.2.15, fd17:625c:f037:2:a00:27ff:fe90:eaeb, not 192.168.10.10"
E0207 22:57:29.217142   17989 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://192.168.10.10:6443/api?timeout=32s\": tls: failed to verify certificate: x509: certificate is valid for 10.233.0.1, 192.168.10.12, 192.168.10.11, 127.0.0.1, ::1, 192.168.10.13, 10.0.2.15, fd17:625c:f037:2:a00:27ff:fe90:eaeb, not 192.168.10.10"
E0207 22:57:29.226689   17989 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://192.168.10.10:6443/api?timeout=32s\": tls: failed to verify certificate: x509: certificate is valid for 10.233.0.1, 192.168.10.13, 192.168.10.11, 127.0.0.1, ::1, 192.168.10.12, 10.0.2.15, fd17:625c:f037:2:a00:27ff:fe90:eaeb, not 192.168.10.10"
E0207 22:57:29.247206   17989 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://192.168.10.10:6443/api?timeout=32s\": tls: failed to verify certificate: x509: certificate is valid for 10.233.0.1, 192.168.10.11, 127.0.0.1, ::1, 192.168.10.12, 192.168.10.13, 10.0.2.15, fd17:625c:f037:2:a00:27ff:fe90:eaeb, not 192.168.10.10"
Unable to connect to the server: tls: failed to verify certificate: x509: certificate is valid for 10.233.0.1, 192.168.10.11, 127.0.0.1, ::1, 192.168.10.12, 192.168.10.13, 10.0.2.15, fd17:625c:f037:2:a00:27ff:fe90:eaeb, not 192.168.10.10

# 인증서에 SAN 정보 확인
root@admin-lb:~/kubespray# ssh k8s-node1 cat /etc/kubernetes/ssl/apiserver.crt | openssl x509 -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 6668381285306049874 (0x5c8ad972c987a552)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kubernetes
        Validity
            Not Before: Feb  7 11:03:15 2026 GMT
            Not After : Feb  7 11:08:15 2027 GMT
        Subject: CN=kube-apiserver
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:ae:4b:04:77:da:b0:1e:83:48:3d:af:6f:9f:54:
                    58:51:be:f5:6b:cd:5f:ce:0d:7d:b4:41:c2:99:96:
                    1b:71:07:88:54:0b:f7:0e:b3:96:62:3b:44:4b:0b:
                    43:95:8a:83:3a:23:66:e5:59:15:93:38:c1:99:a2:
                    7a:1a:13:16:9d:43:95:20:aa:6e:7f:8d:3f:67:32:
                    2d:bf:cd:88:9e:96:7b:d4:42:83:87:b8:3d:6f:08:
                    72:d9:9c:00:ab:b0:43:04:90:d8:f6:6d:59:04:14:
                    d9:93:49:31:99:0e:d4:49:70:9f:2b:aa:6d:fb:2f:
                    b3:6f:85:e2:73:66:2f:03:8e:89:8b:d4:c0:63:68:
                    48:5b:a9:db:e7:18:d3:92:ea:39:47:1c:33:42:ff:
                    c7:f3:f4:6e:9b:3b:18:01:35:8c:a7:f2:47:cb:20:
                    80:17:30:4e:e4:79:a6:c6:d2:bb:5c:37:0e:e8:4a:
                    f9:c0:05:70:f6:ed:f6:cd:9e:22:7d:ed:e4:28:05:
                    4f:dc:10:8c:6a:41:ee:25:19:42:de:ed:24:65:68:
                    f1:4a:0e:f9:b7:90:c0:27:e7:fb:9d:99:d7:d6:a1:
                    ce:cb:08:7a:52:df:8d:fd:6a:89:ae:ff:4e:38:d7:
                    e4:96:00:1c:91:79:f8:7c:c3:3b:5e:e5:71:06:89:
                    80:eb
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Authority Key Identifier:
                C3:2F:41:17:C7:0A:99:FA:49:BC:06:96:EC:3B:FA:E2:02:B9:89:E7
            X509v3 Subject Alternative Name:
                DNS:k8s-node1, DNS:k8s-node2, DNS:k8s-node3, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:lb-apiserver.kubernetes.local, DNS:localhost, IP Address:10.233.0.1, IP Address:192.168.10.11, IP Address:127.0.0.1, IP Address:0:0:0:0:0:0:0:1, IP Address:192.168.10.12, IP Address:192.168.10.13, IP Address:10.0.2.15, IP Address:FD17:625C:F037:2:A00:27FF:FE90:EAEB
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        43:8b:5d:07:52:cc:da:f5:70:45:14:26:7d:60:42:f0:39:10:
        62:e5:fa:35:7f:d5:1c:61:b3:48:68:f0:3b:44:a7:0f:85:a3:
        9a:fb:c3:68:35:80:68:f0:19:d3:05:0a:91:b3:dd:c1:7d:2c:
        73:36:04:05:9a:52:ac:93:ff:78:fc:56:88:0f:da:4f:4f:55:
        93:9e:25:59:d2:22:5b:92:e5:53:2c:7a:f8:96:0e:ee:86:0c:
        62:25:e5:1f:b2:47:b3:f6:94:50:f5:a8:5e:c7:ee:e3:1f:98:
        8c:9b:91:a4:6a:25:76:ed:5f:96:b4:97:cf:44:e2:1e:15:7e:
        69:ee:b7:ec:02:90:1d:a6:f3:6c:96:87:29:e0:41:10:63:44:
        01:ee:d6:ed:f0:f5:7b:f2:7b:dc:56:2f:46:eb:8f:11:bb:86:
        20:fb:4e:07:2c:d3:eb:2e:4e:fb:e8:63:7f:d2:7d:79:de:d8:
        7c:de:01:c0:32:42:db:07:42:72:a8:e7:85:09:9e:30:1a:31:
        1d:55:40:a5:ad:ef:e9:bd:f8:d6:fa:76:34:29:9b:dd:34:23:
        cd:a0:98:bc:9d:32:09:1d:b5:b5:88:50:a8:b1:0d:aa:a4:a4:
        a3:97:b2:2c:c2:36:39:65:4e:f0:df:e8:b6:39:77:19:4f:dd:
        8e:1f:f8:24
        
        
root@admin-lb:~/kubespray# ssh k8s-node1 kubectl get cm -n kube-system kubeadm-config -o yaml
apiVersion: v1
data:
  ClusterConfiguration: |
    apiServer:
      certSANs:
      - kubernetes
      - kubernetes.default
      - kubernetes.default.svc
      - kubernetes.default.svc.cluster.local
      - 10.233.0.1
      - localhost
      - 127.0.0.1
      - ::1
      - k8s-node1
      - k8s-node2
      - k8s-node3
      - lb-apiserver.kubernetes.local
      - 192.168.10.11
      - 192.168.10.12
      - 192.168.10.13
      - 10.0.2.15
      - fd17:625c:f037:2:a00:27ff:fe90:eaeb
      extraArgs:
      - name: etcd-compaction-interval
        value: 5m0s
      - name: default-not-ready-toleration-seconds
        value: "300"
      - name: default-unreachable-toleration-seconds
        value: "300"
      - name: anonymous-auth
        value: "True"
      - name: authorization-mode
        value: Node,RBAC
      - name: bind-address
        value: '::'
      - name: apiserver-count
        value: "3"
      - name: endpoint-reconciler-type
        value: lease
      - name: service-node-port-range
        value: 30000-32767
      - name: service-cluster-ip-range
        value: 10.233.0.0/18
      - name: kubelet-preferred-address-types
        value: InternalDNS,InternalIP,Hostname,ExternalDNS,ExternalIP
      - name: profiling
        value: "False"
      - name: request-timeout
        value: 1m0s
      - name: enable-aggregator-routing
        value: "False"
      - name: service-account-lookup
        value: "True"
      - name: storage-backend
        value: etcd3
      - name: allow-privileged
        value: "true"
      - name: event-ttl
        value: 1h0m0s
      extraVolumes:
      - hostPath: /etc/pki/tls
        mountPath: /etc/pki/tls
        name: etc-pki-tls
        readOnly: true
      - hostPath: /etc/pki/ca-trust
        mountPath: /etc/pki/ca-trust
        name: etc-pki-ca-trust
        readOnly: true
    apiVersion: kubeadm.k8s.io/v1beta4
    caCertificateValidityPeriod: 87600h0m0s
    certificateValidityPeriod: 8760h0m0s
    certificatesDir: /etc/kubernetes/ssl
    clusterName: cluster.local
    controlPlaneEndpoint: 192.168.10.11:6443
    controllerManager:
      extraArgs:
      - name: node-monitor-grace-period
        value: 40s
      - name: node-monitor-period
        value: 5s
      - name: cluster-cidr
        value: 10.233.64.0/18
      - name: service-cluster-ip-range
        value: 10.233.0.0/18
      - name: node-cidr-mask-size-ipv4
        value: "24"
      - name: profiling
        value: "False"
      - name: terminated-pod-gc-threshold
        value: "12500"
      - name: bind-address
        value: '::'
      - name: leader-elect-lease-duration
        value: 15s
      - name: leader-elect-renew-deadline
        value: 10s
      - name: configure-cloud-routes
        value: "false"
    dns:
      disabled: true
      imageRepository: registry.k8s.io/coredns
      imageTag: v1.11.3
    encryptionAlgorithm: RSA-2048
    etcd:
      external:
        caFile: /etc/ssl/etcd/ssl/ca.pem
        certFile: /etc/ssl/etcd/ssl/node-k8s-node1.pem
        endpoints:
        - https://192.168.10.11:2379
        - https://192.168.10.12:2379
        - https://192.168.10.13:2379
        keyFile: /etc/ssl/etcd/ssl/node-k8s-node1-key.pem
    imageRepository: registry.k8s.io
    kind: ClusterConfiguration
    kubernetesVersion: v1.32.9
    networking:
      dnsDomain: cluster.local
      podSubnet: 10.233.64.0/18
      serviceSubnet: 10.233.0.0/18
    proxy: {}
    scheduler:
      extraArgs:
      - name: bind-address
        value: '::'
      - name: config
        value: /etc/kubernetes/kubescheduler-config.yaml
      - name: profiling
        value: "False"
      extraVolumes:
      - hostPath: /etc/kubernetes/kubescheduler-config.yaml
        mountPath: /etc/kubernetes/kubescheduler-config.yaml
        name: kubescheduler-config
        readOnly: true
kind: ConfigMap
metadata:
  creationTimestamp: "2026-02-07T11:08:23Z"
  name: kubeadm-config
  namespace: kube-system
  resourceVersion: "204"
  uid: 9ca208bd-47df-4fbf-b4b1-2bedf9915470
  

# 인증서 SAN 에 'IP, Domain' 추가
root@admin-lb:~/kubespray# echo "supplementary_addresses_in_ssl_keys: [192.168.10.10, k8s-api-srv.admin-lb.com]" >> inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
root@admin-lb:~/kubespray# grep "^[^#]" inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml

# (신규터미널) 모니터링
PLAY RECAP **********************************************************************************************************************************
k8s-node1                  : ok=65   changed=11   unreachable=0    failed=0    skipped=62   rescued=0    ignored=0
k8s-node2                  : ok=44   changed=9    unreachable=0    failed=0    skipped=57   rescued=0    ignored=0
k8s-node3                  : ok=44   changed=9    unreachable=0    failed=0    skipped=57   rescued=0    ignored=0

Saturday 07 February 2026  23:01:09 +0900 (0:00:00.077)       0:00:43.080 *****
===============================================================================
kubernetes/control-plane : Kubeadm | Check apiserver.crt SAN hosts ------------------------------------------------------------------- 3.42s
kubernetes/control-plane : Kubeadm | Check apiserver.crt SAN IPs --------------------------------------------------------------------- 3.11s
kubernetes/control-plane : Backup old certs and keys --------------------------------------------------------------------------------- 2.42s
Gather minimal facts ----------------------------------------------------------------------------------------------------------------- 2.33s
kubernetes/preinstall : Create other directories of root owner ----------------------------------------------------------------------- 1.80s
kubernetes/control-plane : Backup old confs ------------------------------------------------------------------------------------------ 1.73s
kubernetes/control-plane : Update server field in component kubeconfigs -------------------------------------------------------------- 1.68s
kubernetes/control-plane : Install | Copy kubectl binary from download dir ----------------------------------------------------------- 1.63s
win_nodes/kubernetes_patch : debug --------------------------------------------------------------------------------------------------- 1.62s
kubernetes/control-plane : Kubeadm | Create kubeadm config --------------------------------------------------------------------------- 1.47s
kubernetes/preinstall : Create kubernetes directories -------------------------------------------------------------------------------- 1.39s
Gather necessary facts (hardware) ---------------------------------------------------------------------------------------------------- 1.32s
kubernetes/control-plane : Install script to renew K8S control plane certificates ---------------------------------------------------- 1.21s
kubernetes/control-plane : Fixup kubelet client cert rotation 1/2 -------------------------------------------------------------------- 0.95s
kubernetes/control-plane : Create kube-scheduler config ------------------------------------------------------------------------------ 0.89s
kubernetes/control-plane : Renew K8S control plane certificates monthly 2/2 ---------------------------------------------------------- 0.77s
kubernetes/control-plane : Kubeadm | regenerate apiserver cert 2/2 ------------------------------------------------------------------- 0.70s
kubernetes/control-plane : Kubeadm | regenerate apiserver cert 1/2 ------------------------------------------------------------------- 0.66s
kubernetes/control-plane : Install kubectl bash completion --------------------------------------------------------------------------- 0.62s
kubernetes/control-plane : Fixup kubelet client cert rotation 2/2 -------------------------------------------------------------------- 0.57s

# 192.168.10.10 엔드포인트 요청 성공!
root@admin-lb:~/kubespray# kubectl get node -v=6
I0207 23:01:27.019907   18704 loader.go:402] Config loaded from file:  /root/.kube/config
I0207 23:01:27.021030   18704 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I0207 23:01:27.021056   18704 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I0207 23:01:27.021075   18704 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
I0207 23:01:27.021080   18704 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I0207 23:01:27.034217   18704 round_trippers.go:560] GET https://192.168.10.10:6443/api?timeout=32s 200 OK in 12 milliseconds
I0207 23:01:27.045862   18704 round_trippers.go:560] GET https://192.168.10.10:6443/apis?timeout=32s 200 OK in 6 milliseconds
I0207 23:01:27.092009   18704 round_trippers.go:560] GET https://192.168.10.10:6443/api/v1/nodes?limit=500 200 OK in 17 milliseconds
NAME        STATUS   ROLES           AGE    VERSION
k8s-node1   Ready    control-plane   172m   v1.32.9
k8s-node2   Ready    control-plane   172m   v1.32.9
k8s-node3   Ready    control-plane   172m   v1.32.9

# ip, domain 둘 다 확인
root@admin-lb:~/kubespray# sed -i 's/192.168.10.10/k8s-api-srv.admin-lb.com/g' /root/.kube/config

# 추가 확인
root@admin-lb:~/kubespray# ssh k8s-node1 cat /etc/kubernetes/ssl/apiserver.crt | openssl x509 -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 8107676958263781548 (0x7084413cbaeea8ac)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kubernetes
        Validity
            Not Before: Feb  7 14:02:16 2026 GMT
            Not After : Feb  7 14:07:16 2027 GMT
        Subject: CN=kube-apiserver
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:e9:f2:37:35:79:e7:b5:57:29:21:ef:21:c5:69:
                    2d:b3:44:e3:91:0c:e5:c7:d3:f5:ec:05:8d:c7:53:
                    07:12:33:fb:54:38:a9:54:4e:28:de:e5:aa:ff:7f:
                    2f:d7:90:05:85:b4:3c:15:4f:fd:52:ea:d4:92:83:
                    65:72:70:88:a4:eb:2b:04:c7:b0:bb:06:24:86:c4:
                    e4:42:02:8f:bf:64:b8:13:2c:60:44:c2:57:bf:df:
                    c0:dc:7e:f3:3f:e5:03:d2:73:83:fd:f3:6e:7f:95:
                    d5:0c:11:da:a8:ec:3d:f2:f0:ee:49:6f:83:36:34:
                    b0:3a:1b:38:84:be:48:2b:2a:6c:76:db:1f:9c:8a:
                    70:92:f2:1e:b4:89:b3:90:f3:a6:86:80:ed:ee:ef:
                    2d:5f:d4:f3:20:19:ec:c9:6c:05:76:1e:f4:81:56:
                    37:b1:51:ac:5e:4b:f2:58:15:48:93:c2:13:94:7a:
                    5d:e7:b4:f7:54:d7:46:3e:72:bc:a6:91:ae:f2:36:
                    ab:ba:f5:e6:12:00:31:1e:0d:8d:d4:a4:1c:9d:53:
                    38:8b:e8:d0:3d:03:17:90:5b:4c:73:b2:b3:7a:c9:
                    bf:2d:dc:3a:6d:f6:e7:97:c4:f8:ed:7e:59:1d:31:
                    47:37:1d:ef:33:ee:76:bc:95:9f:32:c2:cf:27:fd:
                    03:db
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Authority Key Identifier:
                C3:2F:41:17:C7:0A:99:FA:49:BC:06:96:EC:3B:FA:E2:02:B9:89:E7
            X509v3 Subject Alternative Name:
                DNS:k8s-api-srv.admin-lb.com, DNS:k8s-node1, DNS:k8s-node2, DNS:k8s-node3, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:lb-apiserver.kubernetes.local, DNS:localhost, IP Address:10.233.0.1, IP Address:192.168.10.11, IP Address:127.0.0.1, IP Address:0:0:0:0:0:0:0:1, IP Address:192.168.10.10, IP Address:192.168.10.12, IP Address:192.168.10.13, IP Address:10.0.2.15, IP Address:FD17:625C:F037:2:A00:27FF:FE90:EAEB
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        64:f8:3e:f3:ee:cb:d9:55:bc:a2:cc:cb:0b:98:a4:32:21:9f:
        64:0d:63:b9:8e:29:59:5a:47:da:c9:55:84:67:f9:ba:42:1c:
        61:b1:57:bb:5c:6f:f1:9a:57:f9:9a:a6:9e:e4:ce:01:65:6f:
        22:a4:a6:78:9b:92:67:a0:52:51:62:b8:f5:2f:50:87:29:be:
        e1:07:2c:47:40:3e:9f:a2:af:f9:03:f8:2a:ed:cd:38:1e:c3:
        b4:7c:af:3b:a1:e0:d9:3b:b2:d7:87:b4:6f:fe:7a:b1:4a:fb:
        a8:07:72:63:c9:5f:65:42:e0:4b:fc:fd:58:d1:81:e5:90:2d:
        18:28:39:18:f6:4b:22:b4:f3:d4:59:e7:d7:75:b0:83:d1:5e:
        86:0e:28:67:1d:83:73:cd:7a:53:49:fb:de:91:f0:3e:55:9a:
        2f:25:2a:72:58:44:cd:9e:d0:6e:c9:05:04:39:8c:c2:f2:0d:
        b9:a7:57:2f:36:a2:68:54:5a:30:65:2c:06:c0:2e:3e:d1:56:
        7e:61:8c:4f:bf:29:1a:59:ef:9a:c5:16:0f:6d:9a:21:91:89:
        34:52:da:b7:b6:de:9d:7d:4e:d8:c5:91:cb:ac:55:6e:4c:b7:
        32:d6:ed:34:e0:eb:60:6d:b5:75:40:b8:56:fa:ec:de:ea:e1:
        95:a7:6c:df
        
# 해당 cm은 최초 설치 후 자동 업데이트 X, 업그레이드에 활용된다고 하니, 위 처럼 kubeadm config 변경 시 직접 cm도 같이 변경해두자.

        
        
k8s-node4   Ready    <none>          172m   v1.32.9

노드 관리

노드 추가 k8s-node5

# 노드 추가 k8s-node5 
# inventory.ini 수정
root@admin-lb:~/kubespray# cat << EOF > /root/kubespray/inventory/mycluster/inventory.ini
> [kube_control_plane]
k8s-node1 ansible_host=192.168.10.11 ip=192.168.10.11 etcd_member_name=etcd1
k8s-node2 ansible_host=192.168.10.12 ip=192.168.10.12 etcd_member_name=etcd2
k8s-node3 ansible_host=192.168.10.13 ip=192.168.10.13 etcd_member_name=etcd3

[etcd:children]
kube_control_plane

[kube_node]
k8s-node4 ansible_host=192.168.10.14 ip=192.168.10.14
k8s-node5 ansible_host=192.168.10.15 ip=192.168.10.15
EOF

root@admin-lb:~/kubespray# ansible-inventory -i /root/kubespray/inventory/mycluster/inventory.ini --graph
@all:
  |--@ungrouped:
  |--@etcd:
  |  |--@kube_control_plane:
  |  |  |--k8s-node1
  |  |  |--k8s-node2
  |  |  |--k8s-node3
  |--@kube_node:
  |  |--k8s-node4
  |  |--k8s-node5
  
# ansible 연결 확인
root@admin-lb:~/kubespray# ansible -i inventory/mycluster/inventory.ini k8s-node5 -m ping
[WARNING]: Platform linux on host k8s-node5 is using the discovered Python interpreter at /usr/bin/python3.12, but future installation of
another Python interpreter could change the meaning of that path. See https://docs.ansible.com/ansible-
core/2.17/reference_appendices/interpreter_discovery.html for more information.
k8s-node5 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3.12"
    },
    "changed": false,
    "ping": "pong"
}

# 워커노드 추가
root@admin-lb:~/kubespray# ANSIBLE_FORCE_COLOR=true ansible-playbook -i inventory/mycluster/inventory.ini -v scale.yml --limit=k8s-node5 -e kube_version="1.32.9" | tee kubespray_add_worker_node.log

# 모니터링
root@admin-lb:~/kubespray# ANSIBLE_FORCE_COLOR=true ansible-playbook -i inventory/mycluster/inventory.ini -v scale.yml --limit=k8s-node5 -e kube_version="1.32.9" | tee kubespray_add_worker_node.log

# 확인
Every 2.0s: kubectl get node                                                                               admin-lb: Sat Feb  7 23:19:31 2026

NAME        STATUS   ROLES           AGE         VERSION
k8s-node1   Ready    control-plane   3h11m       v1.32.9
k8s-node2   Ready    control-plane   3h10m       v1.32.9
k8s-node3   Ready    control-plane   3h10m       v1.32.9
k8s-node4   Ready    <none>          3h10m       v1.32.9
k8s-node5   Ready    <none>          <invalid>   v1.32.9


root@admin-lb:~/kubespray# kubectl get node -owide
NAME        STATUS   ROLES           AGE     VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                        KERNEL-VERSION                  CONTAINER-RUNTIME
k8s-node1   Ready    control-plane   3h17m   v1.32.9   192.168.10.11   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.aarch64   containerd://2.1.5
k8s-node2   Ready    control-plane   3h17m   v1.32.9   192.168.10.12   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.aarch64   containerd://2.1.5
k8s-node3   Ready    control-plane   3h17m   v1.32.9   192.168.10.13   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.aarch64   containerd://2.1.5
k8s-node4   Ready    <none>          3h16m   v1.32.9   192.168.10.14   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.aarch64   containerd://2.1.5
k8s-node5   Ready    <none>          2m2s    v1.32.9   192.168.10.15   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.aarch64   containerd://2.1.5

# 변경 정보 확인
root@admin-lb:~/kubespray# ssh k8s-node5 tree /etc/kubernetes
/etc/kubernetes
├── kubeadm-client.conf
├── kubelet.conf
├── kubelet.conf.15422.2026-02-07@23:18:51~
├── kubelet-config.yaml
├── kubelet.env
├── manifests
│   └── nginx-proxy.yml
├── pki -> /etc/kubernetes/ssl
└── ssl
    └── ca.crt

4 directories, 7 files
root@admin-lb:~/kubespray# ssh k8s-node5 tree /var/lib/kubelet
/var/lib/kubelet
├── checkpoints
├── config.yaml
├── cpu_manager_state
├── device-plugins
│   └── kubelet.sock
├── kubeadm-flags.env
├── memory_manager_state
├── pki
│   ├── kubelet-client-2026-02-07-23-18-50.pem
│   ├── kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2026-02-07-23-18-50.pem
│   ├── kubelet.crt
│   └── kubelet.key
├── plugins
├── plugins_registry
├── pod-resources
│   └── kubelet.sock
└── pods
    ├── 577f6a66-08fa-4cd2-ae1e-2f104c1915cd
    │   ├── containers
    │   │   ├── install-cni
    │   │   │   └── 0cb09e5c
    │   │   ├── install-cni-plugin
    │   │   │   └── 481866fe
    │   │   └── kube-flannel
    │   │       ├── 7001cd69
    │   │       └── bd84bb93
    │   ├── etc-hosts
    │   ├── plugins
    │   │   └── kubernetes.io~empty-dir
    │   │       ├── wrapped_flannel-cfg
    │   │       │   └── ready
    │   │       └── wrapped_kube-api-access-v7jkz
    │   │           └── ready
    │   └── volumes
    │       ├── kubernetes.io~configmap
    │       │   └── flannel-cfg
    │       │       ├── cni-conf.json -> ..data/cni-conf.json
    │       │       └── net-conf.json -> ..data/net-conf.json
    │       └── kubernetes.io~projected
    │           └── kube-api-access-v7jkz
    │               ├── ca.crt -> ..data/ca.crt
    │               ├── namespace -> ..data/namespace
    │               └── token -> ..data/token
    ├── ec9c3b89-13fd-42fa-b798-a313ac3f4f05
    │   ├── containers
    │   │   └── kube-proxy
    │   │       └── 94f3c4e8
    │   ├── etc-hosts
    │   ├── plugins
    │   │   └── kubernetes.io~empty-dir
    │   │       ├── wrapped_kube-api-access-6lwg5
    │   │       │   └── ready
    │   │       └── wrapped_kube-proxy
    │   │           └── ready
    │   └── volumes
    │       ├── kubernetes.io~configmap
    │       │   └── kube-proxy
    │       │       ├── config.conf -> ..data/config.conf
    │       │       └── kubeconfig.conf -> ..data/kubeconfig.conf
    │       └── kubernetes.io~projected
    │           └── kube-api-access-6lwg5
    │               ├── ca.crt -> ..data/ca.crt
    │               ├── namespace -> ..data/namespace
    │               └── token -> ..data/token
    └── f8f0790f0f374d27632f9ad8c3ae4aaf
        ├── containers
        │   └── nginx-proxy
        │       └── f0c0f90b
        ├── etc-hosts
        ├── plugins
        └── volumes

39 directories, 33 files
root@admin-lb:~/kubespray# ssh k8s-node5 pstree -a
systemd --switched-root --system --deserialize=46 no_timer_check
  |-NetworkManager --no-daemon
  |   `-3*[{NetworkManager}]
  |-VBoxService --pidfile /var/run/vboxadd-service.sh
  |   `-8*[{VBoxService}]
  |-agetty -o -- \\u --noreset --noclear - linux
  |-atd -f
  |-auditd
  |   |-sedispatch
  |   `-2*[{auditd}]
  |-chronyd -F 2
  |-containerd
  |   `-10*[{containerd}]
  |-containerd-shim -namespace k8s.io -id 635b3e5418716e220d6719eb0af617ce7c85d011b535a2c09b8cbde454440e5e-addre
  |   |-nginx
  |   |   |-nginx
  |   |   |   `-32*[{nginx}]
  |   |   `-nginx
  |   |       `-32*[{nginx}]
  |   |-pause
  |   `-11*[{containerd-shim}]
  |-containerd-shim -namespace k8s.io -id 00b1d2366baa0cbb7bb001f589a64d4b293133143e4e44329538c97e27128697-addre
  |   |-flanneld --ip-masq --kube-subnet-mgr --iface=enp0s9
  |   |   `-9*[{flanneld}]
  |   |-pause
  |   `-12*[{containerd-shim}]
  |-containerd-shim -namespace k8s.io -id ad97b97df62815a13dce7ddf8a8f25023c554cf5c95936ce6b2a1f04aabceb59-addre
  |   |-kube-proxy --config=/var/lib/kube-proxy/config.conf --hostname-override=k8s-node5
  |   |   `-6*[{kube-proxy}]
  |   |-pause
  |   `-11*[{containerd-shim}]
  |-crond -n
  |-dbus-broker-lau --scope system --audit
  |   `-dbus-broker --log 4 --controller 9 --machine-id d52e598aa3894a2bbed8c46cd78feb65 --max-bytes 536870912 --max-fds 4096 ...
  |-fwupd
  |   `-5*[{fwupd}]
  |-gpg-agent --homedir /var/lib/fwupd/gnupg --use-standard-socket --daemon
  |   |-scdaemon --multi-server --homedir /var/lib/fwupd/gnupg
  |   |   `-{scdaemon}
  |   `-{gpg-agent}
  |-gssproxy -i
  |   `-5*[{gssproxy}]
  |-irqbalance
  |   `-{irqbalance}
  |-kubelet --v=2 --node-ip=192.168.10.15 --hostname-override=k8s-node5--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-
  |   `-12*[{kubelet}]
  |-lsmd -d
  |-polkitd --no-debug --log-level=err
  |   `-3*[{polkitd}]
  |-rpcbind -w -f
  |-rsyslogd -n
  |   `-2*[{rsyslogd}]
  |-sshd
  |   |-sshd-session
  |   |   `-sshd-session
  |   `-sshd-session
  |       `-sshd-session
  |           `-pstree -a
  |-systemd --user
  |   `-(sd-pam)
  |-systemd-journal
  |-systemd-logind
  |-systemd-udevd
  |-systemd-userdbd
  |   |-systemd-userwor
  |   |-systemd-userwor
  |   `-systemd-userwor
  |-tuned -Es /usr/sbin/tuned -l -P
  |   `-3*[{tuned}]
  `-udisksd
      `-6*[{udisksd}]
      
# 샘플 파드 분배
root@admin-lb:~/kubespray# kubectl get pod -owide
NAME                      READY   STATUS    RESTARTS   AGE   IP            NODE        NOMINATED NODE   READINESS GATES
webpod-697b545f57-q22tz   1/1     Running   0          41m   10.233.67.6   k8s-node4   <none>           <none>
webpod-697b545f57-srd2b   1/1     Running   0          41m   10.233.67.5   k8s-node4   <none>           <none>
root@admin-lb:~/kubespray# kubectl scale deployment webpod --replicas 1
deployment.apps/webpod scaled
root@admin-lb:~/kubespray# kubectl get pod -owide
NAME                      READY   STATUS    RESTARTS   AGE   IP            NODE        NOMINATED NODE   READINESS GATES
webpod-697b545f57-srd2b   1/1     Running   0          46m   10.233.67.5   k8s-node4   <none>           <none>
root@admin-lb:~/kubespray# kubectl scale deployment webpod --replicas 2
deployment.apps/webpod scaled
root@admin-lb:~/kubespray# kubectl get pod -owide
NAME                      READY   STATUS              RESTARTS   AGE     IP            NODE        NOMINATED NODE   READINESS GATES
webpod-697b545f57-gw28s   0/1     ContainerCreating   0          4m50s   <none>        k8s-node5   <none>           <none>
webpod-697b545f57-srd2b   1/1     Running             0          46m     10.233.67.5   k8s-node4   <none>           <none>
root@admin-lb:~/kubespray#

노드 삭제

# webpod deployment 에 pdb 설정 : 해당 정책은 항상 최소 2개의 Pod가 Ready 상태여야 함 , drain / eviction 시 단 하나의 Pod도 축출 불가
root@admin-lb:~/kubespray# kubectl scale deployment webpod --replicas 1
deployment.apps/webpod scaled
root@admin-lb:~/kubespray# kubectl scale deployment webpod --replicas 2
deployment.apps/webpod scaled

root@admin-lb:~/kubespray# cat <<EOF | kubectl apply -f -
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: webpod
  namespace: default
spec:
  maxUnavailable: 0
  selector:
    matchLabels:
      app: webpod
EOF
poddisruptionbudget.policy/webpod created
root@admin-lb:~/kubespray# kubectl get pdb
NAME     MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
webpod   N/A             0                 0                     <invalid>

# 삭제 실패
root@admin-lb:~/kubespray# ansible-playbook -i inventory/mycluster/inventory.ini -v remove-node.yml --list-tags
Using /root/kubespray/ansible.cfg as config file
[WARNING]: Could not match supplied host pattern, ignoring: bastion
[WARNING]: Could not match supplied host pattern, ignoring: k8s_cluster
[WARNING]: Could not match supplied host pattern, ignoring: calico_rr

playbook: remove-node.yml

  play #1 (localhost): Validate nodes for removal	TAGS: []
      TASK TAGS: []

  play #2 (all): Check Ansible version	TAGS: [always]
      TASK TAGS: [always, check]

  play #3 (all): Inventory setup and validation	TAGS: [always]
      TASK TAGS: [always]

  play #4 (bastion[0]): Install bastion ssh config	TAGS: []
      TASK TAGS: [bastion, localhost]

  play #5 (this_is_unreachable): Confirm node removal	TAGS: []
[WARNING]: Could not match supplied host pattern, ignoring: this_is_unreachable
      TASK TAGS: []

  play #6 (k8s_cluster:etcd:calico_rr): Bootstrap hosts for Ansible	TAGS: []
      TASK TAGS: [bootstrap_os, facts, system-packages]

  play #7 (k8s_cluster:etcd:calico_rr): Gather facts	TAGS: [always]
      TASK TAGS: [always]

  play #8 (this_is_unreachable): Reset node	TAGS: []
      TASK TAGS: [containerd, crio, dns, docker, facts, files, ip6tables, iptables, mounts, network, pre-remove, reset, services]

  play #9 (this_is_unreachable): Post node removal	TAGS: []
      TASK TAGS: [post-remove]
      
      
root@admin-lb:~/kubespray# ansible-playbook -i inventory/mycluster/inventory.ini -v remove-node.yml -e node=k8s-node5
      
TASK [Fail if user does not confirm deletion] ***********************************************************************************************
fatal: [k8s-node5]: FAILED! => {"changed": false, "msg": "Delete nodes confirmation failed"}

PLAY RECAP **********************************************************************************************************************************
k8s-node1                  : ok=4    changed=1    unreachable=0    failed=0    skipped=9    rescued=0    ignored=0
k8s-node2                  : ok=4    changed=1    unreachable=0    failed=0    skipped=9    rescued=0    ignored=0
k8s-node3                  : ok=4    changed=1    unreachable=0    failed=0    skipped=9    rescued=0    ignored=0
k8s-node4                  : ok=17   changed=1    unreachable=0    failed=0    skipped=13   rescued=0    ignored=0
k8s-node5                  : ok=5    changed=1    unreachable=0    failed=1    skipped=9    rescued=0    ignored=0
localhost                  : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Saturday 07 February 2026  23:29:25 +0900 (0:00:00.034)       0:00:13.508 *****
===============================================================================
Confirm Execution ------------------------------------------------------------------------------------------------------------------- 11.12s
dynamic_groups : Match needed groups by their old names or definition ---------------------------------------------------------------- 0.27s
validate_inventory : Stop if unsupported version of Kubernetes ----------------------------------------------------------------------- 0.18s
validate_inventory : Stop if auto_renew_certificates is enabled when certificates are managed externally (kube_external_ca_mode is true) --- 0.16s
validate_inventory : Stop if removed tags are used ----------------------------------------------------------------------------------- 0.12s
validate_inventory : Stop if download_localhost is enabled but download_run_once is not ---------------------------------------------- 0.12s
validate_inventory : Guarantee that enough network address space is available for all pods ------------------------------------------- 0.11s
validate_inventory : Check external_cloud_provider value ----------------------------------------------------------------------------- 0.11s
validate_inventory : Check cloud_provider value -------------------------------------------------------------------------------------- 0.08s
validate_inventory : Stop if kata_containers_enabled is enabled when container_manager is docker ------------------------------------- 0.08s
validate_inventory : Check that ipv4 IP range is enough for the nodes ---------------------------------------------------------------- 0.08s
validate_inventory : Warn if `kube_network_plugin` is `none -------------------------------------------------------------------------- 0.08s
validate_inventory : Stop if RBAC is not enabled when dashboard is enabled ----------------------------------------------------------- 0.08s
validate_inventory : Stop if gvisor_enabled is enabled when container_manager is not containerd -------------------------------------- 0.08s
validate_inventory : Warn if `enable_dual_stack_networks` is set --------------------------------------------------------------------- 0.08s
Check that python netaddr is installed ----------------------------------------------------------------------------------------------- 0.07s
validate_inventory : Check that kube_pods_subnet does not collide with kube_service_addresses ---------------------------------------- 0.07s
validate_inventory : Check that kube_pods_subnet is a network range ------------------------------------------------------------------ 0.06s
validate_inventory : Check that kube_service_addresses is a network range ------------------------------------------------------------ 0.06s
Assert that nodes are specified for removal ------------------------------------------------------------------------------------------ 0.05s
          
# pdb 삭제
root@admin-lb:~/kubespray# kubectl delete pdb webpod
poddisruptionbudget.policy "webpod" deleted

# 다시 삭제 시도
PLAY RECAP **********************************************************************************************************************************
k8s-node1                  : ok=25   changed=1    unreachable=0    failed=0    skipped=19   rescued=0    ignored=0
k8s-node2                  : ok=25   changed=1    unreachable=0    failed=0    skipped=19   rescued=0    ignored=0
k8s-node3                  : ok=25   changed=1    unreachable=0    failed=0    skipped=19   rescued=0    ignored=0
k8s-node4                  : ok=38   changed=1    unreachable=0    failed=0    skipped=24   rescued=0    ignored=0
k8s-node5                  : ok=64   changed=21   unreachable=0    failed=0    skipped=32   rescued=0    ignored=0
localhost                  : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Saturday 07 February 2026  23:31:59 +0900 (0:00:00.432)       0:01:21.093 *****
===============================================================================
reset : Reset | delete some files and directories ----------------------------------------------------------------------------------- 14.16s
Confirm Execution ------------------------------------------------------------------------------------------------------------------- 10.33s
system_packages : Manage packages ---------------------------------------------------------------------------------------------------- 9.92s
network_plugin/cni : CNI | Copy cni plugins ------------------------------------------------------------------------------------------ 3.95s
bootstrap_os : Assign inventory name to unconfigured hostnames (non-CoreOS, non-Flatcar, Suse and ClearLinux, non-Fedora) ------------ 2.78s
reset : Reset | stop services -------------------------------------------------------------------------------------------------------- 2.62s
Gather information about installed services ------------------------------------------------------------------------------------------ 1.98s
reset : Reset | remove containerd binary files --------------------------------------------------------------------------------------- 1.85s
Gather necessary facts (hardware) ---------------------------------------------------------------------------------------------------- 1.73s
bootstrap_os : Fetch /etc/os-release ------------------------------------------------------------------------------------------------- 1.53s
reset : Reset | remove services ------------------------------------------------------------------------------------------------------ 1.51s
remove_node/pre_remove : Remove-node | Drain node except daemonsets resource --------------------------------------------------------- 1.48s
reset : Reset | stop containerd and etcd services ------------------------------------------------------------------------------------ 1.40s
reset : Gather active network services ----------------------------------------------------------------------------------------------- 1.30s
Gather minimal facts ----------------------------------------------------------------------------------------------------------------- 1.21s
Gather necessary facts (network) ----------------------------------------------------------------------------------------------------- 1.11s
reset : Flush iptables --------------------------------------------------------------------------------------------------------------- 0.95s
bootstrap_os : Gather host facts to get ansible_distribution_version ansible_distribution_major_version ------------------------------ 0.88s
reset : Set IPv4 iptables default policies to ACCEPT --------------------------------------------------------------------------------- 0.86s
bootstrap_os : Gather facts ---------------------------------------------------------------------------------------------------------- 0.82s

# 확인
root@admin-lb:~/kubespray# kubectl get node -owide
NAME        STATUS   ROLES           AGE     VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                        KERNEL-VERSION                  CONTAINER-RUNTIME
k8s-node1   Ready    control-plane   3h28m   v1.32.9   192.168.10.11   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.aarch64   containerd://2.1.5
k8s-node2   Ready    control-plane   3h27m   v1.32.9   192.168.10.12   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.aarch64   containerd://2.1.5
k8s-node3   Ready    control-plane   3h27m   v1.32.9   192.168.10.13   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.aarch64   containerd://2.1.5
k8s-node4   Ready    <none>          3h27m   v1.32.9   192.168.10.14   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.aarch64   containerd://2.1.5

# 삭제 확인
root@admin-lb:~/kubespray# ssh k8s-node5 tree /etc/kubernetes
/etc/kubernetes  [error opening dir]

0 directories, 0 files
root@admin-lb:~/kubespray# ssh k8s-node5 tree /var/lib/kubelet
/var/lib/kubelet  [error opening dir]

0 directories, 0 files
root@admin-lb:~/kubespray# ssh k8s-node5 pstree -a
systemd --switched-root --system --deserialize=46 no_timer_check
  |-NetworkManager --no-daemon
  |   `-3*[{NetworkManager}]
  |-VBoxService --pidfile /var/run/vboxadd-service.sh
  |   `-8*[{VBoxService}]
  |-agetty -o -- \\u --noreset --noclear - linux
  |-atd -f
  |-auditd
  |   |-sedispatch
  |   `-2*[{auditd}]
  |-chronyd -F 2
  |-crond -n
  |-dbus-broker-lau --scope system --audit
  |   `-dbus-broker --log 4 --controller 9 --machine-id d52e598aa3894a2bbed8c46cd78feb65 --max-bytes 536870912 --max-fds 4096 ...
  |-fwupd
  |   `-5*[{fwupd}]
  |-gpg-agent --homedir /var/lib/fwupd/gnupg --use-standard-socket --daemon
  |   |-scdaemon --multi-server --homedir /var/lib/fwupd/gnupg
  |   |   `-{scdaemon}
  |   `-{gpg-agent}
  |-gssproxy -i
  |   `-5*[{gssproxy}]
  |-irqbalance
  |   `-{irqbalance}
  |-lsmd -d
  |-polkitd --no-debug --log-level=err
  |   `-3*[{polkitd}]
  |-rpcbind -w -f
  |-rsyslogd -n
  |   `-2*[{rsyslogd}]
  |-sshd
  |   |-sshd-session
  |   |   `-sshd-session
  |   `-sshd-session
  |       `-sshd-session
  |           `-pstree -a
  |-systemd --user
  |   `-(sd-pam)
  |-systemd-journal
  |-systemd-logind
  |-systemd-udevd
  |-systemd-userdbd
  |   |-systemd-userwor
  |   |-systemd-userwor
  |   `-systemd-userwor
  |-tuned -Es /usr/sbin/tuned -l -P
  |   `-3*[{tuned}]
  `-udisksd
      `-6*[{udisksd}]
      
# inventory.ini 수정
root@admin-lb:~/kubespray# cat << EOF > /root/kubespray/inventory/mycluster/inventory.ini
[kube_control_plane]
k8s-node1 ansible_host=192.168.10.11 ip=192.168.10.11 etcd_member_name=etcd1
k8s-node2 ansible_host=192.168.10.12 ip=192.168.10.12 etcd_member_name=etcd2
k8s-node3 ansible_host=192.168.10.13 ip=192.168.10.13 etcd_member_name=etcd3

[etcd:children]
kube_control_plane

[kube_node]
k8s-node4 ansible_host=192.168.10.14 ip=192.168.10.14
EOF

다음 실습을 위해 다시 노드 추가

# inventory.ini 수정
root@admin-lb:~/kubespray# cat << EOF > /root/kubespray/inventory/mycluster/inventory.ini
[kube_control_plane]
k8s-node1 ansible_host=192.168.10.11 ip=192.168.10.11 etcd_member_name=etcd1
k8s-node2 ansible_host=192.168.10.12 ip=192.168.10.12 etcd_member_name=etcd2
k8s-node3 ansible_host=192.168.10.13 ip=192.168.10.13 etcd_member_name=etcd3

[etcd:children]
kube_control_plane

[kube_node]
k8s-node4 ansible_host=192.168.10.14 ip=192.168.10.14
k8s-node5 ansible_host=192.168.10.15 ip=192.168.10.15
EOF

# 워커 노드 추가 수행
root@admin-lb:~/kubespray# ANSIBLE_FORCE_COLOR=true ansible-playbook -i inventory/mycluster/inventory.ini -v scale.yml --limit=k8s-node5 -e kube_version="1.32.9" | tee kubespray_add_worker_node.log

# 확인
PLAY RECAP *********************************************************************
k8s-node5                  : ok=407  changed=64   unreachable=0    failed=0    skipped=594  rescued=0    ignored=0

Saturday 07 February 2026  23:39:35 +0900 (0:00:00.029)       0:03:59.403 *****
===============================================================================
network_plugin/flannel : Flannel | Wait for flannel subnet.env file presence -- 27.46s
download : Download_container | Download image if required ------------- 19.62s
download : Download_container | Download image if required ------------- 13.54s
download : Download_container | Download image if required -------------- 9.43s
download : Download_container | Download image if required -------------- 7.57s
system_packages : Manage packages --------------------------------------- 7.06s
container-engine/containerd : Containerd | Unpack containerd archive ---- 3.84s
download : Download_container | Download image if required -------------- 3.50s
container-engine/runc : Download_file | Download item ------------------- 3.31s
container-engine/crictl : Extract_file | Unpacking archive -------------- 3.27s
container-engine/crictl : Download_file | Download item ----------------- 3.17s
container-engine/containerd : Download_file | Download item ------------- 3.16s
container-engine/nerdctl : Download_file | Download item ---------------- 3.14s
network_plugin/cni : CNI | Copy cni plugins ----------------------------- 2.88s
container-engine/nerdctl : Extract_file | Unpacking archive ------------- 2.83s
network_plugin/cni : CNI | Copy cni plugins ----------------------------- 2.81s
container-engine/validate-container-engine : Populate service facts ----- 2.02s
etcd : Gen_certs | update ca-certificates (RedHat) ---------------------- 2.00s
Gather necessary facts (hardware) --------------------------------------- 1.50s
download : Extract_file | Unpacking archive ----------------------------- 1.33s

root@admin-lb:~/kubespray# kubectl get node -owide
NAME        STATUS   ROLES           AGE         VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                        KERNEL-VERSION                  CONTAINER-RUNTIME
k8s-node1   Ready    control-plane   3h31m       v1.32.9   192.168.10.11   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.aarch64   containerd://2.1.5
k8s-node2   Ready    control-plane   3h31m       v1.32.9   192.168.10.12   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.aarch64   containerd://2.1.5
k8s-node3   Ready    control-plane   3h31m       v1.32.9   192.168.10.13   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.aarch64   containerd://2.1.5
k8s-node4   Ready    <none>          3h30m       v1.32.9   192.168.10.14   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.aarch64   containerd://2.1.5
k8s-node5   Ready    <none>          <invalid>   v1.32.9   192.168.10.15   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.aarch64   containerd://2.1.5

[K8s] Kubespray 배포 분석

안녕유지 — Sun, 1 Feb 2026 07:16:10 +0900

Cloudnet K8s Deploy 4주차 스터디를 진행하며 정리한 글입니다.

Kubespray

https://github.com/kubernetes-sigs/kubespray

GitHub - kubernetes-sigs/kubespray: Deploy a Production Ready Kubernetes Cluster

Deploy a Production Ready Kubernetes Cluster. Contribute to kubernetes-sigs/kubespray development by creating an account on GitHub.

github.com

출처 : https://www.redhat.com/ko/blog/kubespray-deploy-kubernetes

Ansible 기반으로 쿠버네티스(Kubernetes) 클러스터를 자동으로 설치, 업그레이드, 관리하기 위한 오픈소스 배포 도구

사전 설정

# user 확인
root@k8s-ctr:~# whoami
pwd
root
/root

# Linux Kernel Requirements : 5.8+ 이상 권장
root@k8s-ctr:~# uname -a
Linux k8s-ctr 6.12.0-55.39.1.el10_0.aarch64 #1 SMP PREEMPT_DYNAMIC Wed Oct 15 11:18:23 EDT 2025 aarch64 GNU/Linux

# Python : 3.10 ~ 3.12 : (참고) bento/rockylinux-9 경우 3.9
root@k8s-ctr:~# which python  && python -V
/usr/bin/python
Python 3.12.9
root@k8s-ctr:~# which python3 && python3 -V
/usr/bin/python3
Python 3.12.9

# pip , git 설치
root@k8s-ctr:~# dnf install -y python3-pip git
root@k8s-ctr:~# which pip  && pip -V
which pip3 && pip3 -V
/usr/bin/pip
pip 23.3.2 from /usr/lib/python3.12/site-packages/pip (python 3.12)
/usr/bin/pip3
pip 23.3.2 from /usr/lib/python3.12/site-packages/pip (python 3.12)

# /etc/hosts 확인
root@k8s-ctr:~# ip -br -c -4 addr
lo               UNKNOWN        127.0.0.1/8
enp0s8           UP             10.0.2.15/24
enp0s9           UP             192.168.10.10/24
root@k8s-ctr:~# cat /etc/hosts
# Loopback entries; do not change.
# For historical reasons, localhost precedes localhost.localdomain:
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
# See hosts(5) for proper format and other examples:
# 192.168.1.10 foo.example.org foo
# 192.168.1.13 bar.example.org bar
192.168.10.10 k8s-ctr
root@k8s-ctr:~# ping -c 1 k8s-ctr
PING k8s-ctr (192.168.10.10) 56(84) bytes of data.
64 bytes from k8s-ctr (192.168.10.10): icmp_seq=1 ttl=64 time=0.081 ms

--- k8s-ctr ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.081/0.081/0.081/0.000 ms

# SSH 접속을 위한 설정
root@k8s-ctr:~# echo "root:qwe123" | chpasswd
root@k8s-ctr:~# cat << EOF >> /etc/ssh/sshd_config
PermitRootLogin yes
PasswordAuthentication yes
EOF
root@k8s-ctr:~# systemctl restart sshd

# Setting SSH Key
root@k8s-ctr:~# ssh-keygen -t rsa -N "" -f /root/.ssh/id_rsa
Generating public/private rsa key pair.
Your identification has been saved in /root/.ssh/id_rsa
Your public key has been saved in /root/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:XcI1pBqBcH9SBfcidaoxqrkg/OY4BpMlCLGgFU+K+t0 root@k8s-ctr
The key's randomart image is:
+---[RSA 3072]----+
|o.o.o.... o+B .  |
|++ + ... + = =   |
|* . .   + O + .  |
|+ .      B B .   |
|.+      S o      |
|+o . . o         |
| o+ o E          |
|  o+.. .         |
| ..+o .          |
+----[SHA256]-----+
root@k8s-ctr:~# ls -l ~/.ssh
total 8
-rw-------. 1 root root 2602 Feb  1 05:14 id_rsa
-rw-r--r--. 1 root root  566 Feb  1 05:14 id_rsa.pub

# ssh-copy-id
root@k8s-ctr:~# ssh-copy-id -o StrictHostKeyChecking=no root@192.168.10.10
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@192.168.10.10's password:

Number of key(s) added: 1

Now try logging into the machine, with: "ssh -o 'StrictHostKeyChecking=no' 'root@192.168.10.10'"
and check to make sure that only the key(s) you wanted were added.

# ssh 접속 확인 : IP, hostname
root@k8s-ctr:~# cat /root/.ssh/authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDL4YHNWCnsX/64y+O0O02ijzTrEHQpHyyJNZzNjcgxSP3alElK+Sv1S156RP7ZYQTntDiIaoZJQMd1rahijK4YFi8vKv32rug8+vO4NEEI2fr4VztCv0LPEE1L8hya9jhFJsLoQB7Z6eWNgh20ChnaekSwE1LfiPm1rT2cAVqSdNteoTz3za8/dsuhrJYqZrhvvqa3VEHjepUx0fxq0l/6s89OwmjdJsRaNF6L4KQyMRJmeIy4nPqDhwNavhqFt0NYrgoKd++JTXqQtJlPjg/Xm2TXFF+tpRNJzaSPUK8HSWUM3iBa7KVKMZ/1/RMXucq5rd0G60nd9bgR+e4xybcAcaZFgVIn8Lutoql9BkKQGK3E9XYn5mzrElrxVhBYkWTT/V5s3FyE0IWIbsLTFVoFHbjAizP8946Cxf9QTUyge0lvb+Q/6fNecyBmapCgshHSTo/Qwm1vECyF6n9f138rC/QmIFjuqa5Mf3PLJ4Azel0swhEpVOIMonnc06QBtpE= root@k8s-ctr
root@k8s-ctr:~# ssh root@192.168.10.10 hostname
k8s-ctr
root@k8s-ctr:~# ssh -o StrictHostKeyChecking=no root@k8s-ctr hostname
Warning: Permanently added 'k8s-ctr' (ED25519) to the list of known hosts.
k8s-ctr
root@k8s-ctr:~# ssh root@k8s-ctr hostname
k8s-ctr

# Clone Kubespray Repository
root@k8s-ctr:~# git clone -b v2.29.1 https://github.com/kubernetes-sigs/kubespray.git /root/kubespray
root@k8s-ctr:~# cd /root/kubespray
root@k8s-ctr:~/kubespray# pip3 install -r /root/kubespray/requirements.txt

# Ansible 버전 확인 
root@k8s-ctr:~/kubespray# which ansible
/usr/local/bin/ansible
root@k8s-ctr:~/kubespray# ansible --version
ansible [core 2.17.14]
  config file = /root/kubespray/ansible.cfg
  configured module search path = ['/root/kubespray/library']
  ansible python module location = /usr/local/lib/python3.12/site-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.12.9 (main, Aug 14 2025, 00:00:00) [GCC 14.2.1 20250110 (Red Hat 14.2.1-7)] (/usr/bin/python3)
  jinja version = 3.1.6
  libyaml = True

Kubespray를 통한 k8s 배포

 root@k8s-ctr:~/kubespray# cp -rfp /root/kubespray/inventory/sample /root/kubespray/inventory/mycluster
root@k8s-ctr:~/kubespray# tree inventory/mycluster/
inventory/mycluster/
├── group_vars
│   ├── all
│   │   ├── all.yml
│   │   ├── aws.yml
│   │   ├── azure.yml
│   │   ├── containerd.yml
│   │   ├── coreos.yml
│   │   ├── cri-o.yml
│   │   ├── docker.yml
│   │   ├── etcd.yml
│   │   ├── gcp.yml
│   │   ├── hcloud.yml
│   │   ├── huaweicloud.yml
│   │   ├── oci.yml
│   │   ├── offline.yml
│   │   ├── openstack.yml
│   │   ├── upcloud.yml
│   │   └── vsphere.yml
│   └── k8s_cluster
│       ├── addons.yml
│       ├── k8s-cluster.yml
│       ├── k8s-net-calico.yml
│       ├── k8s-net-cilium.yml
│       ├── k8s-net-custom-cni.yml
│       ├── k8s-net-flannel.yml
│       ├── k8s-net-kube-ovn.yml
│       ├── k8s-net-kube-router.yml
│       ├── k8s-net-macvlan.yml
│       └── kube_control_plane.yml
└── inventory.ini

4 directories, 27 files

# inventory.ini 작성
root@k8s-ctr:~/kubespray# cat << EOF > /root/kubespray/inventory/mycluster/inventory.ini
k8s-ctr ansible_host=192.168.10.10 ip=192.168.10.10

[kube_control_plane]
k8s-ctr

[etcd:children]
kube_control_plane

[kube_node]
k8s-ctr
EOF

root@k8s-ctr:~/kubespray# grep "^[^#]" inventory/mycluster/group_vars/all/all.yml
---
bin_dir: /usr/local/bin
loadbalancer_apiserver_port: 6443
loadbalancer_apiserver_healthcheck_port: 8081
no_proxy_exclude_workers: false
kube_webhook_token_auth: false
kube_webhook_token_auth_url_skip_tls_verify: false
ntp_enabled: false
ntp_manage_config: false
ntp_servers:
  - "0.pool.ntp.org iburst"
  - "1.pool.ntp.org iburst"
  - "2.pool.ntp.org iburst"
  - "3.pool.ntp.org iburst"
unsafe_show_logs: false
allow_unsupported_distribution_setup: false

# 테스트할 기능 관련 수정
root@k8s-ctr:~/kubespray# sed -i 's|kube_network_plugin: calico|kube_network_plugin: flannel|g' inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
sed -i 's|kube_proxy_mode: ipvs|kube_proxy_mode: iptables|g' inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
sed -i 's|enable_nodelocaldns: true|enable_nodelocaldns: false|g' inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
sed -i 's|auto_renew_certificates: false|auto_renew_certificates: true|g' inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
sed -i 's|# auto_renew_certificates_systemd_calendar|auto_renew_certificates_systemd_calendar|g' inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
grep -iE 'kube_network_plugin:|kube_proxy_mode|enable_nodelocaldns:|^auto_renew_certificates' inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
kube_network_plugin: flannel
kube_proxy_mode: iptables
enable_nodelocaldns: false
auto_renew_certificates: true
auto_renew_certificates_systemd_calendar: "Mon *-*-1,2,3,4,5,6,7 03:00:00"

## flannel 설정 수정  inventory/mycluster/group_vars/k8s_cluster/k8s-net-flannel.yml
root@k8s-ctr:~/kubespray# cat inventory/mycluster/group_vars/k8s_cluster/k8s-net-flannel.yml
# see roles/network_plugin/flannel/defaults/main.yml

## interface that should be used for flannel operations
## This is actually an inventory cluster-level item
# flannel_interface:

## Select interface that should be used for flannel operations by regexp on Name or IP
## This is actually an inventory cluster-level item
## example: select interface with ip from net 10.0.0.0/23
## single quote and escape backslashes
# flannel_interface_regexp: '10\\.0\\.[0-2]\\.\\d{1,3}'

# You can choose what type of flannel backend to use: 'vxlan',  'host-gw' or 'wireguard'
# please refer to flannel's docs : https://github.com/coreos/flannel/blob/master/README.md
# flannel_backend_type: "vxlan"
# flannel_vxlan_vni: 1
# flannel_vxlan_port: 8472
# flannel_vxlan_direct_routing: false

root@k8s-ctr:~/kubespray# echo "flannel_interface: enp0s9" >> inventory/mycluster/group_vars/k8s_cluster/k8s-net-flannel.yml
root@k8s-ctr:~/kubespray# grep "^[^#]" inventory/mycluster/group_vars/k8s_cluster/k8s-net-flannel.yml
flannel_interface: enp0s9


# 테스트할 기능 관련 수정
root@k8s-ctr:~/kubespray# sed -i 's|helm_enabled: false|helm_enabled: true|g' inventory/mycluster/group_vars/k8s_cluster/addons.yml
sed -i 's|metrics_server_enabled: false|metrics_server_enabled: true|g' inventory/mycluster/group_vars/k8s_cluster/addons.yml
sed -i 's|node_feature_discovery_enabled: false|node_feature_discovery_enabled: true|g' inventory/mycluster/group_vars/k8s_cluster/addons.yml
grep -iE 'helm_enabled:|metrics_server_enabled:|node_feature_discovery_enabled:' inventory/mycluster/group_vars/k8s_cluster/addons.yml

helm_enabled: true
metrics_server_enabled: true
node_feature_discovery_enabled: true


# 기본 환경 정보 출력 저장
root@k8s-ctr:~/kubespray# ip addr | tee -a ip_addr-1.txt 
root@k8s-ctr:~/kubespray# ss -tnlp | tee -a ss-1.txt
root@k8s-ctr:~/kubespray# df -hT | tee -a df-1.txt
root@k8s-ctr:~/kubespray# findmnt | tee -a findmnt-1.txt
root@k8s-ctr:~/kubespray# sysctl -a | tee -a sysctl-1.txt

# 지원 버전 정보 확인
root@k8s-ctr:~/kubespray# cat roles/kubespray_defaults/vars/main/checksums.yml | grep -i kube -A40
kubelet_checksums:
  arm64:
    1.33.7: sha256:3035c44e0d429946d6b4b66c593d371cf5bbbfc85df39d7e2a03c422e4fe404a
    1.33.6: sha256:7d8b7c63309cfe2da2331a1ae13cce070b9ba01e487099e7881a4281667c131d
    1.33.5: sha256:c6ad0510c089d49244eede2638b4a4ff125258fd29a0649e7eef05c7f79c737f
    1.33.4: sha256:623329b1a5f4858e3a5406d3947807b75144f4e71dde11ef1a71362c3a8619cc
    1.33.3: sha256:3f69bb32debfaf25fce91aa5e7181e1e32f3550f3257b93c17dfb37bed621a9c
    1.33.2: sha256:0fa15aca9b90fe7aef1ed3aad31edd1d9944a8c7aae34162963a6aaaf726e065
    1.33.1: sha256:10540261c311ae005b9af514d83c02694e12614406a8524fd2d0bad75296f70d
    1.33.0: sha256:ae5a4fc6d733fc28ff198e2d80334e21fcb5c34e76b411c50fff9cb25accf05a
    1.32.10: sha256:21cc3d98550d3a23052d649e77956f2557e7f6119ff1e27dc82b852d006136cd
    1.32.9: sha256:29037381c79152409adacee83448a2bdb67e113f003613663c7589286200ded8
    1.32.8: sha256:d5527714fac08eac4c1ddcbd8a3c6db35f3acd335d43360219d733273b672cce
    1.32.7: sha256:b862a8d550875924c8abed6c15ba22564f7e232c239aa6a2e88caf069a0ab548
    1.32.6: sha256:b045d4f8f96bf934c894f9704ab2931ffa3c6cf78a8d98e457482a6c455dab6d
    1.32.5: sha256:034753a2e308afeb4ce3cf332d38346c6e660252eac93b268fac0e112a56ff46
    1.32.4: sha256:91117b71eb2bb3dd79ec3ed444e058a347349108bf661838f53ee30d2a0ff168
    1.32.3: sha256:5c3c98e6e0fa35d209595037e05022597954b8d764482417a9588e15218f0fe2
    1.32.2: sha256:d74b659bbde5adf919529d079975900e51e10bc807f0fda9dc9f6bb07c4a3a7b
    1.32.1: sha256:8e6d0eeedd9f0b8b38d4f600ee167816f71cf4dacfa3d9a9bb6c3561cc884e95
    1.32.0: sha256:bda9b2324c96693b38c41ecea051bab4c7c434be5683050b5e19025b50dbc0bf
    1.31.14: sha256:e2842f132933b990a8cbc285be3a28ff1cd213fe1a3380e24e37b1d2ce5e0ca6
    1.31.13: sha256:37e8f83b7bc4cb1b6f49d99cb0d23c2c692f9782abc4f03aad37cc7bd504af68
    1.31.12: sha256:3dab6925a2beb59fbfa7df2897e001af95886145f556cafdbde8c4facd7ca516
    1.31.11: sha256:3a0e07fd72709736cd85ce64a2f5505b2bb085fe697417b96ff249febd5357b1
    1.31.10: sha256:bdb7b70e6f17e6a6700c275c0a3e3632252cf34bf482b6a9fb8448efe8a0e287
    1.31.9: sha256:2debf321e74f430c3832e2426766271f4d51e54927e6ad4be0235d31453dace6
    1.31.8: sha256:c071aa506071db5f03a03ea3f406b4250359b08b7ae10eeee3cfb3da05411925
    1.31.7: sha256:c6624e9e0bbf31334893f991f9a85c7018d8073c32147f421f6338bc92ac6f33
    1.31.6: sha256:79b2bae5f578bae643e44ae1a40c834221983ac8e695c82aad79f2dc96c50ada
    1.31.5: sha256:922a96405fdc3ae41e403565d06c5a6c3b733b0c3d0d1d61086b39c6760103d3
    1.31.4: sha256:fb6f02f3324a72307acc11998eb5b1c3778167ae165c98f9d49bd011498e72f8
    1.31.3: sha256:0ec590052f2d1cee158a789d705ca931cbc2556ceed364c4ad754fd36c61be28
    1.31.2: sha256:118e1b0e85357a81557f9264521c083708f295d7c5f954a4113500fd1afca8f8
    1.31.1: sha256:fbd98311e96b9dcdd73d1688760d410cc70aefce26272ff2f20eef51a7c0d1da
    1.31.0: sha256:b310da449a9d2f8b928cab5ca12a6772617ba421023894e061ca2647e6d9f1c3
  amd64:
    1.33.7: sha256:2cea40c8c6929330e799f8fc73233a4b61e63f208739669865e2a23a39c3a007
    1.33.6: sha256:10cd08fe1f9169fd7520123bcdfff87e37b8a4e21c39481faa382f00355b6973
    1.33.5: sha256:8f6106b970259486c5af5cbee404d4f23406d96d99dfb92a6965b299c2a4db0e
    1.33.4: sha256:109bd2607b054a477ede31c55ae814eae8e75543126dc4cea40b04424d843489
--
kubectl_checksums:
  arm:
    1.33.7: sha256:f6b9ac99f4efb406c5184d0a51d9ed896690c80155387007291309cbb8cdd847
    1.33.6: sha256:89bcef827ac8662781740d092cff410744c0653d828b68cc14051294fcd717e6
    1.33.5: sha256:5a3a416a85cfc9f7a348c0c0e6334b7449e00a57288ab5a57286ccf68a4d06af
    1.33.4: sha256:eefd3864ce5440e0ba648b12d53ccffaad97f1c049781b1aa21af6a5278f035f
    1.33.3: sha256:0124dba9e9091b872591cabcbaea7df07069cb132d38d95f3c7bc8d5b8b621a9
    1.33.2: sha256:f3992382aa0ea21f71a976b6fd6a213781c9b58be60c42013950110cf2184f2a
    1.33.1: sha256:6b1cd6e2bf05c6adaa76b952f9c4ea775f5255913974ccdb12145175d4809e93
    1.33.0: sha256:bbb4b4906d483f62b0fc3a0aea3ddac942820984679ad11635b81ee881d69ab3
    1.32.10: sha256:b42bc77586238b43b8c5cdd06086f1ab00190245dd8b66b28822785b177fbde4
    1.32.9: sha256:84629d460b60693ca954e148ce522defd34d18bc5c934836cfaf0268930713dd
    1.32.8: sha256:ed54b52631fdf5ecc4ddb12c47df481f84b5890683beaeaa55dc84e43d2cd023
    1.32.7: sha256:c5416b59afdf897c4fbf08867c8a32b635f83f26e40980d38233fad6b345e37c
    1.32.6: sha256:77fec65c6f08c28f8695de4db877d82d74c881ed3ed110ebfd88cbd4ee3d01dc
    1.32.5: sha256:7270e6ac4b82b5e4bd037dccae1631964634214baa66a9548deb5edd3f79de31
    1.32.4: sha256:bf28793213039690d018bbfa9bcfcfed76a9aa8e18dc299eced8709ca542fcdd
    1.32.3: sha256:f990c878e54e5fac82eac7398ef643acca9807838b19014f1816fa9255b2d3d9
    1.32.2: sha256:e1e6a2fd4571cd66c885aa42b290930660d34a7331ffb576fcab9fd1a0941a83
    1.32.1: sha256:8ccf69be2578d3a324e9fc7d4f3b29bc9743cc02d72f33ba2d0fe30389014bc8
    1.32.0: sha256:6b33ea8c80f785fb07be4d021301199ae9ee4f8d7ea037a8ae544d5a7514684e
    1.31.14: sha256:23860bd774ec2c2cb1f409581c236725673c55506409da846a651ec27c2ca15d
    1.31.13: sha256:875597876f9dcfb2b3197667c0fbb0691cbef3d9522de22875c1a5c02bc04de5
    1.31.12: sha256:8e430e7a192355a60e1398580a861b4724b286ed38ff52a156500d3fae90c583
    1.31.11: sha256:7768bb4e1b79ddac982968e47d9e25f357b7e9c0f08039134815a64062d5ea6f
    1.31.10: sha256:1f3f644609513ed0c6045638e60fc9e9fb5de39c375719601f565e6ad82b9b85
    1.31.9: sha256:54e560eb3ad4b2b0ae95d79d71b2816dfa154b33758e49f2583bec0980f19861
    1.31.8: sha256:65fdd04f5171e44620cc4e0b9e0763b1b3d10b2b15c1f7f99b549d36482015d4
    1.31.7: sha256:870d919f8ef5f5c608bd69c57893937910de6a8ed2c077fc4f0945375f61734d
    1.31.6: sha256:b370a552cd6c9bb5fc42e4e9031b74f35da332f27b585760bacb0d3189d8634d
    1.31.5: sha256:cbb4e470751ef8864ade9d008e848f691ac6cbdee320539797a68a5512b9f7f8
    1.31.4: sha256:055d1672f63fda86c6dfa5a2354d627f908f68bde6bf8394fdc9a99cadc4de19
    1.31.3: sha256:e0d00fbac98e67b774ff1ed9a0e6fc5be5c1f08cc69b0c8b483904ed15ad8c50
    1.31.2: sha256:f2a638bdaa4764e82259ed1548ce2c86056e33a3d09147f7f0c2d4ee5b5e300c
    1.31.1: sha256:51b178c9362a4fbe35644399f113d7f904d306261953a51c5c0a57676e209fa6
    1.31.0: sha256:a4d6292c88c199688a03ea211bea08c8ae29f1794f5deeeef46862088d124baa
  arm64:
    1.33.7: sha256:fa7ee98fdb6fba92ae05b5e0cde0abd5972b2d9a4a084f7052a1fd0dce6bc1de
    1.33.6: sha256:3ab32d945a67a6000ba332bf16382fc3646271da6b7d751608b320819e5b8f38
    1.33.5: sha256:6db7c5d846c3b3ddfd39f3137a93fe96af3938860eefdbf2429805ee1656e381
    1.33.4: sha256:76cd7a2aa59571519b68c3943521404cbce55dafb7d8866f8d0ea2995b396eef
--
kubeadm_checksums:
  arm64:
    1.33.7: sha256:b24eeeff288f9565e11a2527e5aed42c21386596110537adb805a5a2a7b3e9ce
    1.33.6: sha256:ef80c198ca15a0850660323655ebf5c32cc4ab00da7a5a59efe95e4bcf8503ab
    1.33.5: sha256:b1c00657649e35771569d095e531d826bd19baf57bcb53cccf3f91d7d60b7808
    1.33.4: sha256:ef471b454d68ee211e279ddeaebde6ee7a8e14b66ae58e0d0184e967c3595892
    1.33.3: sha256:bf8ed3bc3952e04f29863c6910ae84b359fe7ac1e642ed4d742ceb396e62c6f2
    1.33.2: sha256:21efc1ba54a1cf25ac68208b7dde2e67f6d0331259f432947d83e70b975ad4cc
    1.33.1: sha256:5b3e3a1e18d43522fdee0e15be13a42cee316e07ddcf47ef718104836edebb3e
    1.33.0: sha256:746c0ee45f4d32ec5046fb10d4354f145ba1ff0c997f9712d46036650ad26340
    1.32.10: sha256:a201f246be3d2c35ffa7fc51a1d2596797628f9b1455da52a246b42ce8e1f779
    1.32.9: sha256:377349141e865849355140c78063fa2b87443bf1aecb06319be4de4df8dbd918
    1.32.8: sha256:8dbd3fa2d94335d763b983caaf2798caae2d4183f6a95ebff28289f2e86edf68
    1.32.7: sha256:a2aad7f7b320c3c847dea84c08e977ba8b5c84d4b7102b46ffd09d41af6c4b51
    1.32.6: sha256:f786731c37ce6e89e6b71d5a7518e4d1c633337237e3803615056eb4640bfc8e
    1.32.5: sha256:2956c694ff2891acdc4690b807f87ab48419b4925d3fad2ac52ace2a1160bd17
    1.32.4: sha256:1b9d97b44758dc4da20d31e3b6d46f50af75ac48be887793e16797a43d9c30e7
    1.32.3: sha256:f9d007aaf1468ea862ef2a1a1a3f6f34cc57358742ceaff518e1533f5a794181
    1.32.2: sha256:fd8a8c1c41d719de703bf49c6f56692dd6477188d8f43dcb77019fd8bc30cbd3
    1.32.1: sha256:55a57145708aaa37f716f140ef774ca64b7088b6df5ee8eae182936ad6580328
    1.32.0: sha256:5da9746a449a3b8a8312b6dd8c48dcb861036cf394306cfbc66a298ba1e8fbde
    1.31.14: sha256:ff9d9351423fd9c7b40a39a9be11df077b1f5a40c85b70349ca0ce55cd4fd336
    1.31.13: sha256:30762e5a20eb8a4d52b278fe7d999fd76ab20b63b40cb1e60625bc73c6e11e96
    1.31.12: sha256:88fc31963e833d72d1e26159166591aea537d762debb5cc0f0d059fdc717b43b
    1.31.11: sha256:73dff62190cd26947a088ceb79d4d039a916091e0c80734e9ddd7b2e0b8efb8b
    1.31.10: sha256:01e627449b5f94bc068f7d0680a07abfd118cbf9805c7bce3aea31a46e4a16cc
    1.31.9: sha256:d8f5dbb17ce2dead6aedcc700e4293a9395e246079fcdc1772ab9e5cbfeca906
    1.31.8: sha256:d0d1a6634e397e4f14b1e5f9b4bd55758ea70bfc114728730d25d563952e453e
    1.31.7: sha256:3f95765db3b9ebb0cf2ff213ac3b42a831dd995a48d9a6b1d544137d3f2c3018
    1.31.6: sha256:03b6df27c630f6137be129d2cef49dc4da12077381af8d234a92e451ba2a16d2
    1.31.5: sha256:971904ff1ac2879d968cac1d4865b7c0ce0d9374506bd17bd0b123981803769b
    1.31.4: sha256:4598c2f0c69e60feb47a070376da358f16efe0e1403c6aca97fa8f7ab1d0e7c0
    1.31.3: sha256:8113900524bd1c8b3ce0b3ece0d37f96291cbf359946afae58a596319a5575c8
    1.31.2: sha256:0f9d231569b3195504f8458415e9b3080e23fb6a749fe7752abfc7a2884efadf
    1.31.1: sha256:66195cd53cda3c73c9ae5e49a1352c710c0ea9ce244bbdeb68b917d809f0ea78
    1.31.0: sha256:dbeb84862d844d58f67ad6be64021681a314cda162a04e6047f376f2a9ad0226
  amd64:
    1.33.7: sha256:c10813d54f58ef33bbe6675f3d39c8bd401867743ebc729afdd043265040c31d
    1.33.6: sha256:c1b84cb3482dd79e26629012f432541ccb505c17f5073aa1fdbca26b1e4909fd
    1.33.5: sha256:6761219749c6c67a56a5668dfe65d669e0c1f34d4b280b72de6d74d47c601f1e
    1.33.4: sha256:a109ebcb68e52d3dd605d92f92460c884dcc8b68aebe442404af19b6d9d778ec
    
# 배포: 아래처럼 반드시 ~/kubespray 디렉토리에서 ansible-playbook 를 실행하자!
root@k8s-ctr:~/kubespray# ansible-playbook -i inventory/mycluster/inventory.ini -v cluster.yml -e kube_version="1.33.3" --list-tasks # 배포 전, Task 목록 확인
Using /root/kubespray/ansible.cfg as config file
[WARNING]: Could not match supplied host pattern, ignoring: bastion
[WARNING]: Could not match supplied host pattern, ignoring: k8s_cluster
[WARNING]: Could not match supplied host pattern, ignoring: calico_rr
[WARNING]: Could not match supplied host pattern, ignoring: _kubespray_needs_etcd

playbook: cluster.yml

  play #1 (all): Check Ansible version  TAGS: [always]
    tasks:
      Check {{ minimal_ansible_version }} <= Ansible version < {{ maximal_ansible_version }}    TAGS: [always, check]
      Check that python netaddr is installed    TAGS: [always, check]
      Check that jinja is not too old (install via pip) TAGS: [always, check]

  play #2 (all): Inventory setup and validation TAGS: [always]
    tasks:
      dynamic_groups : Match needed groups by their old names or definition     TAGS: [always]
      validate_inventory : Stop if removed tags are used        TAGS: [always]
      validate_inventory : Stop if kube_control_plane group is empty    TAGS: [always]
      validate_inventory : Stop if etcd group is empty in external etcd mode    TAGS: [always]
      validate_inventory : Warn if `kube_network_plugin` is `none       TAGS: [always]
      validate_inventory : Stop if unsupported version of Kubernetes    TAGS: [always]
      validate_inventory : Stop if known booleans are set as strings (Use JSON format on CLI: -e "{'key': true }")      TAGS: [always]
      validate_inventory : Stop if even number of etcd hosts    TAGS: [always]
      validate_inventory : Guarantee that enough network address space is available for all pods        TAGS: [always]
      validate_inventory : Stop if RBAC is not enabled when dashboard is enabled        TAGS: [always]
      validate_inventory : Check cloud_provider value   TAGS: [always]
      validate_inventory : Check external_cloud_provider value  TAGS: [always]
      validate_inventory : Check that kube_service_addresses is a network range TAGS: [always]
      validate_inventory : Check that kube_pods_subnet is a network range       TAGS: [always]
      validate_inventory : Check that kube_pods_subnet does not collide with kube_service_addresses     TAGS: [always]
      validate_inventory : Check that ipv4 IP range is enough for the nodes     TAGS: [always]
      validate_inventory : Check that kube_service_addresses_ipv6 is a network range    TAGS: [always]
      validate_inventory : Check that kube_pods_subnet_ipv6 is a network range  TAGS: [always]
      validate_inventory : Check that kube_pods_subnet_ipv6 does not collide with kube_service_addresses_ipv6   TAGS: [always]
      validate_inventory : Check that ipv6 IP range is enough for the nodes     TAGS: [always]
      validate_inventory : Stop if unsupported options selected TAGS: [always]
      validate_inventory : Warn if `enable_dual_stack_networks` is set  TAGS: [always]
      validate_inventory : Stop if download_localhost is enabled but download_run_once is not   TAGS: [always]
      validate_inventory : Stop if kata_containers_enabled is enabled when container_manager is docker  TAGS: [always]
      validate_inventory : Stop if gvisor_enabled is enabled when container_manager is not containerd   TAGS: [always]
      validate_inventory : Ensure minimum containerd version    TAGS: [always]
      validate_inventory : Stop if auto_renew_certificates is enabled when certificates are managed externally (kube_external_ca_mode is true)  TAGS: [always]

  play #3 (bastion[0]): Install bastion ssh config      TAGS: []
    tasks:
      bastion-ssh-config : Set bastion host IP and port TAGS: [bastion, localhost]
      bastion-ssh-config : Store the current ansible_user in the real_user fact TAGS: [bastion, localhost]
      bastion-ssh-config : Create ssh bastion conf      TAGS: [bastion, localhost]

  play #4 (k8s_cluster:etcd:calico_rr): Bootstrap hosts for Ansible     TAGS: []
    tasks:
      bootstrap_os : Fetch /etc/os-release      TAGS: [bootstrap_os]
      bootstrap_os : Include vars       TAGS: [bootstrap_os, facts]
      bootstrap_os : Include tasks      TAGS: [bootstrap_os]
      system_packages : Gather OS information   TAGS: [bootstrap_os, system-packages]
      system_packages : Update package management cache (zypper) - SUSE TAGS: [bootstrap_os, system-packages]
      system_packages : Remove legacy docker repo file  TAGS: [bootstrap_os, system-packages]
      system_packages : Install epel-release on RHEL derivatives        TAGS: [bootstrap_os, system-packages]
      system_packages : Manage packages TAGS: [bootstrap_os, system-packages]
      bootstrap_os : Create remote_tmp for it is used by another module TAGS: [bootstrap_os]
      bootstrap_os : Gather facts       TAGS: [bootstrap_os]
      bootstrap_os : Assign inventory name to unconfigured hostnames (non-CoreOS, non-Flatcar, Suse and ClearLinux, non-Fedora) TAGS: [bootstrap_os]
      bootstrap_os : Ensure bash_completion.d folder exists     TAGS: [bootstrap_os]

  play #5 (k8s_cluster:etcd:calico_rr): Gather facts    TAGS: [always]
    tasks:
    ...
    
    
  play #15 (k8s_cluster): Apply resolv.conf changes now that cluster DNS is up  TAGS: []
    tasks:
      adduser : User | Create User Group        TAGS: [kubelet, resolvconf]
      adduser : User | Create User      TAGS: [kubelet, resolvconf]
      
      
root@k8s-ctr:~/kubespray# ANSIBLE_FORCE_COLOR=true ansible-playbook -i inventory/mycluster/inventory.ini -v cluster.yml -e kube_version="1.33.3" | tee kubespray_install.log
Using /root/kubespray/ansible.cfg as config file

PLAY [Check Ansible version] ***************************************************
Sunday 01 February 2026  05:42:20 +0900 (0:00:00.012)       0:00:00.012 ******* 

TASK [Check 2.17.3 <= Ansible version < 2.18.0] ********************************
ok: [k8s-ctr] => {
    "changed": false,
    "msg": "All assertions passed"
}
Sunday 01 February 2026  05:42:20 +0900 (0:00:00.018)       0:00:00.031 ******* 

TASK [Check that python netaddr is installed] **********************************
ok: [k8s-ctr] => {
    "changed": false,
    "msg": "All assertions passed"
}
Sunday 01 February 2026  05:42:20 +0900 (0:00:00.048)       0:00:00.080 ******* 

TASK [Check that jinja is not too old (install via pip)] ***********************
ok: [k8s-ctr] => {
    "changed": false,
    "msg": "All assertions passed"
}

PLAY [Inventory setup and validation] ******************************************
Sunday 01 February 2026  05:42:21 +0900 (0:00:00.022)       0:00:00.102 ******* 

TASK [dynamic_groups : Match needed groups by their old names or definition] ***
changed: [k8s-ctr] => (item={'key': 'k8s_cluster', 'value': ['kube_node', 'kube_control_plane', 'calico_rr']}) => {"add_group": "k8s_cluster", "ansible_loop_var": "item", "changed": true, "item": {"key": "k8s_cluster", "value": ["kube_node", "kube_control_plane", "calico_rr"]}, "parent_groups": ["all"]}
Sunday 01 February 2026  05:42:21 +0900 (0:00:00.041)       0:00:00.144 ******* 

TASK [validate_inventory : Stop if removed tags are used] **********************
ok: [k8s-ctr] => {
    "changed": false,
    "msg": "All assertions passed"
}
Sunday 01 February 2026  05:42:21 +0900 (0:00:00.021)       0:00:00.165 ******* 

TASK [validate_inventory : Stop if kube_control_plane group is empty] **********
ok: [k8s-ctr] => {
    "changed": false,
    "msg": "All assertions passed"
}
Sunday 01 February 2026  05:42:21 +0900 (0:00:00.020)       0:00:00.186 ******* 

TASK [validate_inventory : Stop if etcd group is empty in external etcd mode] ***
ok: [k8s-ctr] => {
    "changed": false,
    "msg": "All assertions passed"
}
Sunday 01 February 2026  05:42:21 +0900 (0:00:00.021)       0:00:00.207 ******* 
Sunday 01 February 2026  05:42:21 +0900 (0:00:00.009)       0:00:00.217 ******* 

TASK [validate_inventory : Stop if unsupported version of Kubernetes] **********
ok: [k8s-ctr] => {
    "changed": false,
    "msg": "All assertions passed"
}
Sunday 01 February 2026  05:42:21 +0900 (0:00:00.025)       0:00:00.242 ******* 

TASK [validate_inventory : Stop if known booleans are set as strings (Use JSON format on CLI: -e "{'key': true }")] ***
ok: [k8s-ctr] => {
    "changed": false,
    "msg": "All assertions passed"
}
Sunday 01 February 2026  05:42:21 +0900 (0:00:00.025)       0:00:00.268 ******* 

TASK [validate_inventory : Stop if even number of etcd hosts] ******************
ok: [k8s-ctr] => {
    "changed": false,
    "msg": "All assertions passed"
}
Sunday 01 February 2026  05:42:21 +0900 (0:00:00.024)       0:00:00.292 ******* 

TASK [validate_inventory : Guarantee that enough network address space is available for all pods] ***
ok: [k8s-ctr] => {
    "changed": false,
    "msg": "All assertions passed"
}
Sunday 01 February 2026  05:42:21 +0900 (0:00:00.026)       0:00:00.318 ******* 
Sunday 01 February 2026  05:42:21 +0900 (0:00:00.009)       0:00:00.328 ******* 
Sunday 01 February 2026  05:42:21 +0900 (0:00:00.010)       0:00:00.339 ******* 
Sunday 01 February 2026  05:42:21 +0900 (0:00:00.009)       0:00:00.348 ******* 

TASK [validate_inventory : Check that kube_service_addresses is a network range] ***
ok: [k8s-ctr] => {
    "changed": false,
    "msg": "All assertions passed"
}
Sunday 01 February 2026  05:42:21 +0900 (0:00:00.049)       0:00:00.398 ******* 

TASK [validate_inventory : Check that kube_pods_subnet is a network range] *****
ok: [k8s-ctr] => {
    "changed": false,
    "msg": "All assertions passed"
}
Sunday 01 February 2026  05:42:21 +0900 (0:00:00.048)       0:00:00.446 ******* 

TASK [validate_inventory : Check that kube_pods_subnet does not collide with kube_service_addresses] ***
ok: [k8s-ctr] => {
    "changed": false,
    "msg": "All assertions passed"
}
Sunday 01 February 2026  05:42:21 +0900 (0:00:00.051)       0:00:00.497 ******* 

TASK [validate_inventory : Check that ipv4 IP range is enough for the nodes] ***
ok: [k8s-ctr] => {
    "changed": false,
    "msg": "All assertions passed"
}
Sunday 01 February 2026  05:42:21 +0900 (0:00:00.048)       0:00:00.546 ******* 
Sunday 01 February 2026  05:42:21 +0900 (0:00:00.016)       0:00:00.562 ******* 
Sunday 01 February 2026  05:42:21 +0900 (0:00:00.015)       0:00:00.578 ******* 
Sunday 01 February 2026  05:42:21 +0900 (0:00:00.014)       0:00:00.592 ******* 
Sunday 01 February 2026  05:42:21 +0900 (0:00:00.013)       0:00:00.606 ******* 

TASK [validate_inventory : Stop if unsupported options selected] ***************
ok: [k8s-ctr] => {
    "changed": false,
    "msg": "All assertions passed"
}
Sunday 01 February 2026  05:42:21 +0900 (0:00:00.025)       0:00:00.631 ******* 
Sunday 01 February 2026  05:42:21 +0900 (0:00:00.013)       0:00:00.645 ******* 
Sunday 01 February 2026  05:42:21 +0900 (0:00:00.008)       0:00:00.654 ******* 
Sunday 01 February 2026  05:42:21 +0900 (0:00:00.009)       0:00:00.663 ******* 
Sunday 01 February 2026  05:42:21 +0900 (0:00:00.009)       0:00:00.673 ******* 

TASK [validate_inventory : Ensure minimum containerd version] ******************
ok: [k8s-ctr] => {
    "changed": false,
    "msg": "All assertions passed"
}
Sunday 01 February 2026  05:42:21 +0900 (0:00:00.023)       0:00:00.696 ******* 
[WARNING]: Could not match supplied host pattern, ignoring: bastion
[WARNING]: Could not match supplied host pattern, ignoring: calico_rr

PLAY [Install bastion ssh config] **********************************************
skipping: no hosts matched

PLAY [Bootstrap hosts for Ansible] *********************************************
Sunday 01 February 2026  05:42:21 +0900 (0:00:00.019)       0:00:00.715 ******* 
[WARNING]: raw module does not support the environment keyword

TASK [bootstrap_os : Fetch /etc/os-release] ************************************
...


root@k8s-ctr:~/kubespray# more kubespray_install.log
Using /root/kubespray/ansible.cfg as config file

PLAY [Check Ansible version] ***************************************************
Sunday 01 February 2026  05:42:20 +0900 (0:00:00.012)       0:00:00.012 ******* 

TASK [Check 2.17.3 <= Ansible version < 2.18.0] ********************************
ok: [k8s-ctr] => {
    "changed": false,
    "msg": "All assertions passed"
}
Sunday 01 February 2026  05:42:20 +0900 (0:00:00.018)       0:00:00.031 ******* 

TASK [Check that python netaddr is installed] **********************************
ok: [k8s-ctr] => {
    "changed": false,
    "msg": "All assertions passed"
}
Sunday 01 February 2026  05:42:20 +0900 (0:00:00.048)       0:00:00.080 ******* 

TASK [Check that jinja is not too old (install via pip)] ***********************
ok: [k8s-ctr] => {
    "changed": false,
    "msg": "All assertions passed"
}

PLAY [Inventory setup and validation] ******************************************


root@k8s-ctr:~/kubespray# kubectl get node -v=6
I0201 05:55:54.381070   35626 loader.go:402] Config loaded from file:  /root/.kube/config
I0201 05:55:54.381828   35626 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I0201 05:55:54.381849   35626 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I0201 05:55:54.381854   35626 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
I0201 05:55:54.381858   35626 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I0201 05:55:54.381861   35626 envvar.go:172] "Feature gate default state" feature="InOrderInformers" enabled=true
I0201 05:55:54.392143   35626 round_trippers.go:632] "Response" verb="GET" url="https://127.0.0.1:6443/api/v1/nodes?limit=500" status="200 OK" milliseconds=7
NAME      STATUS   ROLES           AGE    VERSION
k8s-ctr   Ready    control-plane   4m7s   v1.33.3


root@k8s-ctr:~/kubespray# ip addr | tee -a ip_addr-2.txt 
ss -tnlp | tee -a ss-2.txt
df -hT | tee -a df-2.txt
findmnt | tee -a findmnt-2.txt
sysctl -a | tee -a sysctl-2.txt

Ansible Playbook & Role 분석

# cluster.yml은 Kubespray의 메인 플레이북으로, 클러스터 생성의 전체 과정을 정의
root@k8s-ctr:~/kubespray# cat /root/kubespray/cluster.yml
---
- name: Install Kubernetes
  ansible.builtin.import_playbook: playbooks/cluster.yml

root@k8s-ctr:~/kubespray# cat kubespray_install.log | grep -E 'PLAY'
PLAY [Check Ansible version] ***************************************************
PLAY [Inventory setup and validation] ******************************************
PLAY [Install bastion ssh config] **********************************************
PLAY [Bootstrap hosts for Ansible] *********************************************
PLAY [Gather facts] ************************************************************
PLAY [Prepare for etcd install] ************************************************
PLAY [Add worker nodes to the etcd play if needed] *****************************
PLAY [Install etcd] ************************************************************
PLAY [Install Kubernetes nodes] ************************************************
PLAY [Install the control plane] ***********************************************
PLAY [Invoke kubeadm and install a CNI] ****************************************
PLAY [Install Calico Route Reflector] ******************************************
PLAY [Patch Kubernetes for Windows] ********************************************
PLAY [Install Kubernetes apps] *************************************************
PLAY [Apply resolv.conf changes now that cluster DNS is up] ********************
PLAY RECAP *********************************************************************


root@k8s-ctr:~/kubespray# cat kubespray_install.log | grep -E 'TASK' | wc -l
563


# playbooks/boilerplate.yml → ansible_version.yml → roles (dynamic_groups, validate_inventory) 소개
oot@k8s-ctr:~/kubespray# cat playbooks/boilerplate.yml
---
- name: Check ansible version
  import_playbook: ansible_version.yml

# These are inventory compatibility tasks with two purposes:
# - to ensure we keep compatibility with old style group names
# - to reduce inventory boilerplate (defining parent groups / empty groups)

- name: Inventory setup and validation
  hosts: all
  gather_facts: false
  tags: always
  roles:
    - dynamic_groups
    - validate_inventory

- name: Install bastion ssh config
  hosts: bastion[0]
  gather_facts: false
  environment: "{{ proxy_disable_env }}"
  roles:
    - { role: kubespray_defaults }
    - { role: bastion-ssh-config, tags: ["localhost", "bastion"] }
    

# kubespray_defaults: Kubespray 전체 play / role / task가 참조하는 ‘최상위 기본 변수 집합’ ← bastion 대상이 아니여서 여기서는 Skip 되긴함
# 여기 값은 언제든 override 가능, 하지만 override 안 하면 전부 여기 기준
root@k8s-ctr:~/kubespray# tree roles/kubespray_defaults
roles/kubespray_defaults
├── defaults
│   └── main
│       ├── download.yml
│       └── main.yml
└── vars
    └── main
        ├── checksums.yml
        └── main.yml

5 directories, 4 files

1. 초기화 및 정보 수집 (Boilerplate & Facts)

모든 노드에 공통적으로 필요한 설정을 적용하고, 각 서버의 사양 정보를 수집하여 이후 설치 단계에서 변수로 활용합니다.

2. 인프라 및 엔진 준비 (Prepare for etcd & container-engine)

kubernetes/preinstall: 방화벽, 커널 파라미터, Swap 비활성화 등 K8s 설치를 위한 OS 최적화를 수행합니다.
container-engine: Docker나 Containerd 같은 컨테이너 런타임을 설치합니다.
download: 설치에 필요한 모든 바이너리와 이미지들을 미리 다운로드합니다.

3. 데이터 저장소 및 노드 구성 (Etcd & K8s Nodes)

etcd: 쿠버네티스의 상태 정보를 저장하는 DB 클러스터를 구축합니다.
kubernetes/node: 모든 노드에 Kubelet, Kube-proxy 등 기초 컴포넌트를 설치합니다.

4. 컨트롤 플레인 및 네트워크 (Control Plane & CNI)

control-plane: 마스터 노드에 API 서버, 스케줄러 등을 설정합니다.
kubeadm: kubeadm init 또는 join을 통해 클러스터를 하나로 묶습니다.
network_plugin: Calico, Flannel 등 CNI를 설치하여 파드 간 통신을 가능하게 합니다.

5. 부가 서비스 설치 (Apps & DNS)

Ingress Controller, Storage Provisioner 등 클러스터 운영에 필요한 앱들을 배포하고, 최종적으로 노드의 DNS(resolv.conf)가 클러스터 내부 DNS를 바라보도록 수정합니다.

실습 환경 배포 분석

다운로드 파일 경로 확인 : local_release_dir: "/tmp/releases"

root@k8s-ctr:~/kubespray# tree -ug /tmp/releases/
[root     root    ]  /tmp/releases/
├── [root     root    ]  cni-plugins-linux-arm64-1.8.0.tgz
├── [root     root    ]  containerd-2.1.5-linux-arm64.tar.gz
├── [root     root    ]  containerd-rootless-setuptool.sh
├── [root     root    ]  containerd-rootless.sh
├── [root     118     ]  crictl
├── [root     root    ]  crictl-1.33.0-linux-arm64.tar.gz
├── [root     root    ]  etcd-3.5.25-linux-arm64.tar.gz
├── [vagrant  vagrant ]  etcd-v3.5.25-linux-arm64
│   ├── [vagrant  vagrant ]  Documentation
│   │   ├── [vagrant  vagrant ]  dev-guide
│   │   │   └── [vagrant  vagrant ]  apispec
│   │   │       └── [vagrant  vagrant ]  swagger
│   │   │           ├── [vagrant  vagrant ]  rpc.swagger.json
│   │   │           ├── [vagrant  vagrant ]  v3election.swagger.json
│   │   │           └── [vagrant  vagrant ]  v3lock.swagger.json
│   │   └── [vagrant  vagrant ]  README.md
│   ├── [vagrant  vagrant ]  etcd
│   ├── [vagrant  vagrant ]  etcdctl
│   ├── [vagrant  vagrant ]  etcdutl
│   ├── [vagrant  vagrant ]  README-etcdctl.md
│   ├── [vagrant  vagrant ]  README-etcdutl.md
│   ├── [vagrant  vagrant ]  README.md
│   └── [vagrant  vagrant ]  READMEv2-etcdctl.md
├── [root     root    ]  helm-3.18.4
│   ├── [root     root    ]  helm-3.18.4-linux-arm64.tar.gz
│   └── [root     118     ]  linux-arm64
│       ├── [root     118     ]  helm
│       ├── [root     118     ]  LICENSE
│       └── [root     118     ]  README.md
├── [root     root    ]  images
├── [root     root    ]  kubeadm-1.33.3-arm64
├── [root     root    ]  kubectl-1.33.3-arm64
├── [root     root    ]  kubelet-1.33.3-arm64
├── [root     root    ]  nerdctl
├── [root     root    ]  nerdctl-2.1.6-linux-arm64.tar.gz
└── [root     root    ]  runc-1.3.4.arm64

9 directories, 28 files

설치된 바이너리 확인 : bin_dir: /usr/local/bin

root@k8s-ctr:~/kubespray# cat inventory/mycluster/group_vars/all/all.yml | grep 'bin_dir'
bin_dir: /usr/local/bin

root@k8s-ctr:~/kubespray# cat inventory/mycluster/group_vars/k8s_cluster/addons.yml | grep helm
helm_enabled: true

root@k8s-ctr:~/kubespray# helm version
version.BuildInfo{Version:"v3.18.4", GitCommit:"d80839cf37d860c8aa9a0503fe463278f26cd5e2", GitTreeState:"clean", GoVersion:"go1.24.4"}
root@k8s-ctr:~/kubespray# etcdctl version
etcdctl version: 3.5.25
API version: 3.5
root@k8s-ctr:~/kubespray# containerd --version
containerd github.com/containerd/containerd/v2 v2.1.5 fcd43222d6b07379a4be9786bda52438f0dd16a1
root@k8s-ctr:~/kubespray# kubeadm version -o yaml
clientVersion:
  buildDate: "2025-07-15T18:05:14Z"
  compiler: gc
  gitCommit: 80779bd6ff08b451e1c165a338a7b69351e9b0b8
  gitTreeState: clean
  gitVersion: v1.33.3
  goVersion: go1.24.4
  major: "1"
  minor: "33"
  platform: linux/arm64

설치된 cni 관련 파일 확인 & kube_owner에 uid로 생성되는 파일 목록 확인

root@k8s-ctr:~/kubespray# cat inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
---
# Kubernetes configuration dirs and system namespace.
# Those are where all the additional config stuff goes
# the kubernetes normally puts in /srv/kubernetes.
# This puts them in a sane location and namespace.
# Editing those values will almost surely break something.
kube_config_dir: /etc/kubernetes
kube_script_dir: "{{ bin_dir }}/kubernetes-scripts"
kube_manifest_dir: "{{ kube_config_dir }}/manifests"

# This is where all the cert scripts and certs will be located
kube_cert_dir: "{{ kube_config_dir }}/ssl"

# This is where all of the bearer tokens will be stored
kube_token_dir: "{{ kube_config_dir }}/tokens"

kube_api_anonymous_auth: true

# Where the binaries will be downloaded.
# Note: ensure that you've enough disk space (about 1G)
local_release_dir: "/tmp/releases"
# Random shifts for retrying failed ops like pushing/downloading
retry_stagger: 5

# This is the user that owns tha cluster installation.
kube_owner: kube
...


root@k8s-ctr:~/kubespray# find / -user kube 2>/dev/null
/etc/cni
/etc/cni/net.d
/etc/kubernetes
/etc/kubernetes/manifests
/usr/libexec/kubernetes
/usr/libexec/kubernetes/kubelet-plugins
/usr/libexec/kubernetes/kubelet-plugins/volume
/usr/libexec/kubernetes/kubelet-plugins/volume/exec
/usr/local/bin/kubernetes-scripts
/opt/cni
/opt/cni/bin
/opt/cni/bin/README.md
/opt/cni/bin/static
/opt/cni/bin/host-device
/opt/cni/bin/ipvlan
/opt/cni/bin/dhcp
/opt/cni/bin/LICENSE
/opt/cni/bin/portmap
/opt/cni/bin/tap
/opt/cni/bin/host-local
/opt/cni/bin/vlan
/opt/cni/bin/loopback
/opt/cni/bin/sbr
/opt/cni/bin/firewall
/opt/cni/bin/bandwidth
/opt/cni/bin/bridge
/opt/cni/bin/vrf
/opt/cni/bin/macvlan
/opt/cni/bin/tuning
/opt/cni/bin/dummy
/opt/cni/bin/ptp

root@k8s-ctr:~/kubespray# ls -l /opt
total 0
drwxr-xr-x. 3 kube root  17 Feb  1 05:43 cni
drwx--x--x. 4 root root  28 Feb  1 05:45 containerd
drwxr-xr-x. 7 root root 119 Oct 24 06:12 VBoxGuestAdditions-7.2.4

root@k8s-ctr:~/kubespray# tree -ug /opt/cni
[kube     root    ]  /opt/cni
└── [kube     root    ]  bin
    ├── [kube     root    ]  bandwidth
    ├── [kube     root    ]  bridge
    ├── [kube     root    ]  dhcp
    ├── [kube     root    ]  dummy
    ├── [kube     root    ]  firewall
    ├── [root     root    ]  flannel
    ├── [kube     root    ]  host-device
    ├── [kube     root    ]  host-local
    ├── [kube     root    ]  ipvlan
    ├── [kube     root    ]  LICENSE
    ├── [kube     root    ]  loopback
    ├── [kube     root    ]  macvlan
    ├── [kube     root    ]  portmap
    ├── [kube     root    ]  ptp
    ├── [kube     root    ]  README.md
    ├── [kube     root    ]  sbr
    ├── [kube     root    ]  static
    ├── [kube     root    ]  tap
    ├── [kube     root    ]  tuning
    ├── [kube     root    ]  vlan
    └── [kube     root    ]  vrf

2 directories, 21 files

root@k8s-ctr:~/kubespray# ls -l /etc | grep cni
drwxr-xr-x.  3 kube root     19 Feb  1 05:43 cni

root@k8s-ctr:~/kubespray# tree -ug /etc/cni
[kube     root    ]  /etc/cni
└── [kube     root    ]  net.d
    └── [root     root    ]  10-flannel.conflist

2 directories, 1 file

auto_renew_certificates: true, auto_renew_certificates_systemd_calendar: "Mon *-*-1,2,3,4,5,6,7 03:00:00" 설정확인

root@k8s-ctr:~/kubespray# systemctl list-timers --all --no-pager
NEXT                          LEFT LAST                              PASSED UNIT                         ACTIVATES                     
Sun 2026-02-01 06:30:02 KST   7min Sun 2026-02-01 02:59:38 KST 1h 31min ago dnf-makecache.timer          dnf-makecache.service
Sun 2026-02-01 06:57:58 KST  35min Sun 2026-02-01 05:26:20 KST    56min ago fwupd-refresh.timer          fwupd-refresh.service
Mon 2026-02-02 00:01:34 KST    17h Sun 2026-02-01 05:02:59 KST 1h 19min ago fstrim.timer                 fstrim.service
Mon 2026-02-02 00:36:37 KST    18h Sun 2026-02-01 03:21:21 KST 1h 30min ago logrotate.timer              logrotate.service
Mon 2026-02-02 00:52:56 KST    18h Sun 2026-02-01 01:47:45 KST 1h 36min ago plocate-updatedb.timer       plocate-updatedb.service
Mon 2026-02-02 03:06:43 KST    20h -                                      - k8s-certs-renew.timer        k8s-certs-renew.service
Mon 2026-02-02 04:52:28 KST    22h Sun 2026-02-01 03:21:32 KST 1h 30min ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Sun 2026-02-08 01:00:00 KST 6 days Sun 2026-02-01 01:39:28 KST 1h 45min ago raid-check.timer             raid-check.service

8 timers listed.
root@k8s-ctr:~/kubespray# systemctl status k8s-certs-renew.timer --no-pager
● k8s-certs-renew.timer - Timer to renew K8S control plane certificates
     Loaded: loaded (/etc/systemd/system/k8s-certs-renew.timer; enabled; preset: disabled)
     Active: active (waiting) since Sun 2026-02-01 05:51:55 KST; 30min ago
 Invocation: fee2a54cd8874d9da45c494616978add
    Trigger: Mon 2026-02-02 03:06:43 KST; 20h left
   Triggers: ● k8s-certs-renew.service

Feb 01 05:51:55 k8s-ctr systemd[1]: Started k8s-certs-renew.timer - Timer to renew K8S control plane certificates.

root@k8s-ctr:~/kubespray# cat /etc/systemd/system/k8s-certs-renew.timer
[Unit]
Description=Timer to renew K8S control plane certificates

[Timer]
OnCalendar=Mon *-*-1,2,3,4,5,6,7 03:00:00
RandomizedDelaySec=10min
FixedRandomDelay=yes
Persistent=yes

[Install]
WantedBy=multi-user.target


root@k8s-ctr:~/kubespray# systemctl status k8s-certs-renew.service
○ k8s-certs-renew.service - Renew K8S control plane certificates
     Loaded: loaded (/etc/systemd/system/k8s-certs-renew.service; static)
     Active: inactive (dead)
TriggeredBy: ● k8s-certs-renew.timer

root@k8s-ctr:~/kubespray# cat /usr/local/bin/k8s-certs-renew.sh
#!/bin/bash

echo "## Check Expiration before renewal ##"

/usr/local/bin/kubeadm certs check-expiration

days_buffer=7 # set a time margin, because we should not renew at the last moment
calendar=Mon *-*-1,2,3,4,5,6,7 03:00:00
next_time=$(systemctl show k8s-certs-renew.timer  -p NextElapseUSecRealtime --value)

if [ "${next_time}" == "" ]; then
        echo "## Skip expiry comparison due to fail to parse next elapse from systemd calendar,do renewal directly ##"
else
        current_time=$(date +%s)
        target_time=$(date -d "${next_time} + ${days_buffer} days" +%s) # $next_time - $days_buffer days
        expiry_threshold=$(( ${target_time} - ${current_time} ))
        expired_certs=$(/usr/local/bin/kubeadm certs check-expiration -o jsonpath="{.certificates[?(@.residualTime<${expiry_threshold}.0)]}")
    if [ "${expired_certs}" == "" ];then
                echo "## Skip cert renew and K8S container restart, since all residualTimes are beyond threshold ##"
                exit 0
        fi
fi

echo "## Renewing certificates managed by kubeadm ##"
/usr/local/bin/kubeadm certs renew all

echo "## Restarting control plane pods managed by kubeadm ##"
/usr/local/bin/crictl pods --namespace kube-system --name 'kube-scheduler-*|kube-controller-manager-*|kube-apiserver-*|etcd-*' -q | /usr/bin/xargs /usr/local/bin/crictl rmp -f

echo "## Updating /root/.kube/config ##"
cp /etc/kubernetes/admin.conf /root/.kube/config

echo "## Waiting for apiserver to be up again ##"
until printf "" 2>>/dev/null >>/dev/tcp/127.0.0.1/6443; do sleep 1; done

echo "## Expiration after renewal ##"
/usr/local/bin/kubeadm certs check-expiration

sysctl 관련 작업

root@k8s-ctr:~/kubespray# cat roles/kubernetes/preinstall/tasks/0080-system-configurations.yml
---
# Todo : selinux configuration
- name: Confirm selinux deployed
  stat:
    path: /etc/selinux/config
    get_attributes: false
    get_checksum: false
    get_mime: false
  when:
    - ansible_os_family == "RedHat"
    - "'Amazon' not in ansible_distribution"
  register: slc

- name: Set selinux policy
  ansible.posix.selinux:
    policy: targeted
    state: "{{ preinstall_selinux_state }}"
  when:
    - ansible_os_family == "RedHat"
    - "'Amazon' not in ansible_distribution"
    - slc.stat.exists
  tags:
    - bootstrap_os

- name: Disable IPv6 DNS lookup
  lineinfile:
    dest: /etc/gai.conf
    line: "precedence ::ffff:0:0/96  100"
    state: present
    create: true
    backup: "{{ leave_etc_backup_files }}"
    mode: "0644"
  when:
    - disable_ipv6_dns
    - not ansible_os_family in ["Flatcar", "Flatcar Container Linux by Kinvolk"]
  tags:
    - bootstrap_os

- name: Clean previously used sysctl file locations
  file:
    path: "/etc/sysctl.d/{{ item }}"
    state: absent
  with_items:
    - ipv4-ip_forward.conf
    - bridge-nf-call.conf

- name: Stat sysctl file configuration
  stat:
    path: "{{ sysctl_file_path }}"
    get_attributes: false
    get_checksum: false
    get_mime: false
  register: sysctl_file_stat
  tags:
    - bootstrap_os

- name: Change sysctl file path to link source if linked
  set_fact:
    sysctl_file_path: "{{ sysctl_file_stat.stat.lnk_source }}"
  when:
    - sysctl_file_stat.stat.islnk is defined
    - sysctl_file_stat.stat.islnk
  tags:
    - bootstrap_os

- name: Make sure sysctl file path folder exists
  file:
    name: "{{ sysctl_file_path | dirname }}"
    state: directory
    mode: "0755"

- name: Enable ip forwarding
  ansible.posix.sysctl:
    sysctl_file: "{{ sysctl_file_path }}"
    name: net.ipv4.ip_forward
    value: "1"
    state: present
    reload: true
    ignoreerrors: "{{ sysctl_ignore_unknown_keys }}"
  when: ipv4_stack | bool

- name: Enable ipv6 forwarding
  ansible.posix.sysctl:
    sysctl_file: "{{ sysctl_file_path }}"
    name: net.ipv6.conf.all.forwarding
    value: "1"
    state: present
    reload: true
    ignoreerrors: "{{ sysctl_ignore_unknown_keys }}"
  when: ipv6_stack | bool

- name: Check if we need to set fs.may_detach_mounts
  stat:
    path: /proc/sys/fs/may_detach_mounts
    get_attributes: false
    get_checksum: false
    get_mime: false
  register: fs_may_detach_mounts
  ignore_errors: true  # noqa ignore-errors

- name: Set fs.may_detach_mounts if needed
  ansible.posix.sysctl:
    sysctl_file: "{{ sysctl_file_path }}"
    name: fs.may_detach_mounts
    value: 1
    state: present
    reload: true
    ignoreerrors: "{{ sysctl_ignore_unknown_keys }}"
  when: fs_may_detach_mounts.stat.exists | d(false)

- name: Ensure kubelet expected parameters are set
  ansible.posix.sysctl:
    sysctl_file: "{{ sysctl_file_path }}"
    name: "{{ item.name }}"
    value: "{{ item.value }}"
    state: present
    reload: true
    ignoreerrors: "{{ sysctl_ignore_unknown_keys }}"
  with_items:
    - { name: kernel.keys.root_maxbytes, value: 25000000 }
    - { name: kernel.keys.root_maxkeys, value: 1000000 }
    - { name: kernel.panic, value: 10 }
    - { name: kernel.panic_on_oops, value: 1 }
    - { name: vm.overcommit_memory, value: 1 }
    - { name: vm.panic_on_oom, value: 0 }
  when: kubelet_protect_kernel_defaults | bool

- name: Check dummy module
  community.general.modprobe:
    name: dummy
    state: present
    params: 'numdummies=0'
  when: enable_nodelocaldns

- name: Set additional sysctl variables
  ansible.posix.sysctl:
    sysctl_file: "{{ sysctl_file_path }}"
    name: "{{ item.name }}"
    value: "{{ item.value }}"
    state: present
    reload: true
    ignoreerrors: "{{ sysctl_ignore_unknown_keys }}"
  with_items: "{{ additional_sysctl }}"

- name: Disable fapolicyd service
  failed_when: false
  systemd_service:
    name: fapolicyd
    state: stopped
    enabled: false
  when: disable_fapolicyd
  
  
root@k8s-ctr:~/kubespray# grep "^[^#]" /etc/sysctl.conf
net.ipv4.ip_forward=1
kernel.keys.root_maxbytes=25000000
kernel.keys.root_maxkeys=1000000
kernel.panic=10
kernel.panic_on_oops=1
vm.overcommit_memory=1
vm.panic_on_oom=0
net.ipv4.ip_local_reserved_ports=30000-32767
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-arptables=1
net.bridge.bridge-nf-call-ip6tables=1

root@k8s-ctr:~/kubespray# 
ls -l /etc/sysctl.d/
total 4
lrwxrwxrwx. 1 root root  14 May 18  2025 99-sysctl.conf -> ../sysctl.conf
-rw-r--r--. 1 root root 120 Feb  1 01:39 k8s.conf

kubernetes/preinstall, tags: preinstall : kubelet이 안정적으로 기동되고, kubeadm이 실패하지 않도록 OS 상태를 Kubernetes 친화적으로 만드는 단계

root@k8s-ctr:~/kubespray# tree roles/kubernetes/preinstall/tasks/
roles/kubernetes/preinstall/tasks/
├── 0010-swapoff.yml
├── 0020-set_facts.yml
├── 0040-verify-settings.yml
├── 0050-create_directories.yml
├── 0060-resolvconf.yml
├── 0061-systemd-resolved.yml
├── 0062-networkmanager-unmanaged-devices.yml
├── 0063-networkmanager-dns.yml
├── 0080-system-configurations.yml
├── 0081-ntp-configurations.yml
├── 0100-dhclient-hooks.yml
├── 0110-dhclient-hooks-undo.yml
└── main.yml

1 directory, 13 files
root@k8s-ctr:~/kubespray# cat roles/kubernetes/preinstall/defaults/main.yml
---
# Set to true to allow pre-checks to fail and continue deployment
ignore_assert_errors: false
# Set to false to disable the backup parameter, set to true to accumulate backups of config files.
leave_etc_backup_files: true
nameservers: []
cloud_resolver: []
disable_host_nameservers: false
# Kubespray sets this to true after clusterDNS is running to apply changes to the host resolv.conf
dns_late: false

# Set to true if your network does not support IPv6
# This may be necessary for pulling Docker images from
# GCE docker repository
disable_ipv6_dns: false

# Remove default cluster search domains (``default.svc.{{ dns_domain }}, svc.{{ dns_domain }}``).
remove_default_searchdomains: false

kube_owner: kube
kube_cert_group: kube-cert
kube_config_dir: /etc/kubernetes
kube_cert_dir: "{{ kube_config_dir }}/ssl"
kube_cert_compat_dir: /etc/kubernetes/pki
kubelet_flexvolumes_plugins_dir: /usr/libexec/kubernetes/kubelet-plugins/volume/exec

# Flatcar Container Linux by Kinvolk cloud init config file to define /etc/resolv.conf content
# for hostnet pods and infra needs
resolveconf_cloud_init_conf: /etc/resolveconf_cloud_init.conf

# sysctl_file_path to add sysctl conf to
sysctl_file_path: "/etc/sysctl.d/99-sysctl.conf"

# Minimal memory requirement in MB for safety checks
minimal_node_memory_mb: 1024
minimal_master_memory_mb: 1500

## NTP Settings

# Manage the NTP configuration file.
ntp_manage_config: false
# Specify the NTP servers
# Only takes effect when ntp_manage_config is true.
ntp_servers:
  - "0.pool.ntp.org iburst"
  - "1.pool.ntp.org iburst"
  - "2.pool.ntp.org iburst"
  - "3.pool.ntp.org iburst"
# Restrict NTP access to these hosts.
# Only takes effect when ntp_manage_config is true.
ntp_restrict:
  - "127.0.0.1"
  - "::1"
# Specify whether to filter interfaces
ntp_filter_interface: false
# Specify the interfaces
# Only takes effect when ntp_filter_interface is true
# ntp_interfaces:
#   - ignore wildcard
#   - listen xxx
# The NTP driftfile path
# Only takes effect when ntp_manage_config is true.
# Default value is `/var/lib/ntp/ntp.drift`, for ntpsec use '/var/lib/ntpsec/ntp.drift'
ntp_driftfile: >-
      {% if ntp_package == "ntpsec" -%}
      /var/lib/ntpsec/ntp.drift
      {%- else -%}
      /var/lib/ntp/ntp.drift
      {%- endif -%}
# Only takes effect when ntp_manage_config is true.
ntp_tinker_panic: false

# Force sync time immediately after the ntp installed, which is useful in a newly installed system.
ntp_force_sync_immediately: false

# Set the timezone for your server.  eg: "Etc/UTC","Etc/GMT-8". If not set, the timezone will not change.
ntp_timezone: ""

# Currently known os distributions
supported_os_distributions:
  - 'RedHat'
  - 'CentOS'
  - 'Fedora'
  - 'Ubuntu'
  - 'Debian'
  - 'Flatcar'
  - 'Flatcar Container Linux by Kinvolk'
  - 'Suse'
  - 'openSUSE Leap'
  - 'openSUSE Tumbleweed'
  - 'ClearLinux'
  - 'OracleLinux'
  - 'AlmaLinux'
  - 'Rocky'
  - 'Amazon'
  - 'Kylin Linux Advanced Server'
  - 'UnionTech'
  - 'UniontechOS'
  - 'openEuler'

# Extending some distributions into the redhat os family
redhat_os_family_extensions:
  - "UnionTech"
  - "UniontechOS"

# Sets DNSStubListener=no, useful if you get "0.0.0.0:53: bind: address already in use"
systemd_resolved_disable_stub_listener: "{{ ansible_os_family in ['Flatcar', 'Flatcar Container Linux by Kinvolk'] }}"

# Used to disable File Access Policy Daemon service.
# If service is enabled, the CNI plugin installation will fail
disable_fapolicyd: true

파드 내에서 too many open files 에러 발생 시 ansible-playbook _ --tags "container-engine"

목표는 OS 커널에 프로세스 단위 제한 (ulimit)을 파드에도 적용

# base_runtime_spec
root@k8s-ctr:~/kubespray# cat /etc/containerd/cri-base.json | jq
{
  "ociVersion": "1.2.1",
  "process": {
    "user": {
      "uid": 0,
      "gid": 0
    },
    "cwd": "/",
    "capabilities": {
      "bounding": [
        "CAP_CHOWN",
        "CAP_DAC_OVERRIDE",
        "CAP_FSETID",
        "CAP_FOWNER",
        "CAP_MKNOD",
        "CAP_NET_RAW",
        "CAP_SETGID",
        "CAP_SETUID",
        "CAP_SETFCAP",
        "CAP_SETPCAP",
        "CAP_NET_BIND_SERVICE",
        "CAP_SYS_CHROOT",
        "CAP_KILL",
        "CAP_AUDIT_WRITE"
      ],
      "effective": [
        "CAP_CHOWN",
        "CAP_DAC_OVERRIDE",
        "CAP_FSETID",
        "CAP_FOWNER",
        "CAP_MKNOD",
        "CAP_NET_RAW",
        "CAP_SETGID",
        "CAP_SETUID",
        "CAP_SETFCAP",
        "CAP_SETPCAP",
        "CAP_NET_BIND_SERVICE",
        "CAP_SYS_CHROOT",
        "CAP_KILL",
        "CAP_AUDIT_WRITE"
      ],
      "permitted": [
        "CAP_CHOWN",
        "CAP_DAC_OVERRIDE",
        "CAP_FSETID",
        "CAP_FOWNER",
        "CAP_MKNOD",
        "CAP_NET_RAW",
        "CAP_SETGID",
        "CAP_SETUID",
        "CAP_SETFCAP",
        "CAP_SETPCAP",
        "CAP_NET_BIND_SERVICE",
        "CAP_SYS_CHROOT",
        "CAP_KILL",
        "CAP_AUDIT_WRITE"
      ]
    },
    "rlimits": [
      {
        "type": "RLIMIT_NOFILE",
        "hard": 65535,
        "soft": 65535
      }
    ],
    "noNewPrivileges": true
  },
  "root": {
    "path": "rootfs"
  },
  "mounts": [
    {
      "destination": "/proc",
      "type": "proc",
      "source": "proc",
      "options": [
        "nosuid",
        "noexec",
        "nodev"
      ]
    },
    {
      "destination": "/dev",
      "type": "tmpfs",
      "source": "tmpfs",
      "options": [
        "nosuid",
        "strictatime",
        "mode=755",
        "size=65536k"
      ]
    },
    {
      "destination": "/dev/pts",
      "type": "devpts",
      "source": "devpts",
      "options": [
        "nosuid",
        "noexec",
        "newinstance",
        "ptmxmode=0666",
        "mode=0620",
        "gid=5"
      ]
    },
    {
      "destination": "/dev/shm",
      "type": "tmpfs",
      "source": "shm",
      "options": [
        "nosuid",
        "noexec",
        "nodev",
        "mode=1777",
        "size=65536k"
      ]
    },
    {
      "destination": "/dev/mqueue",
      "type": "mqueue",
      "source": "mqueue",
      "options": [
        "nosuid",
        "noexec",
        "nodev"
      ]
    },
    {
      "destination": "/sys",
      "type": "sysfs",
      "source": "sysfs",
      "options": [
        "nosuid",
        "noexec",
        "nodev",
        "ro"
      ]
    },
    {
      "destination": "/run",
      "type": "tmpfs",
      "source": "tmpfs",
      "options": [
        "nosuid",
        "strictatime",
        "mode=755",
        "size=65536k"
      ]
    }
  ],
  "linux": {
    "resources": {
      "devices": [
        {
          "allow": false,
          "access": "rwm"
        }
      ]
    },
    "cgroupsPath": "/default",
    "namespaces": [
      {
        "type": "pid"
      },
      {
        "type": "ipc"
      },
      {
        "type": "uts"
      },
      {
        "type": "mount"
      },
      {
        "type": "network"
      }
    ],
    "maskedPaths": [
      "/proc/acpi",
      "/proc/asound",
      "/proc/kcore",
      "/proc/keys",
      "/proc/latency_stats",
      "/proc/timer_list",
      "/proc/timer_stats",
      "/proc/sched_debug",
      "/sys/firmware",
      "/sys/devices/virtual/powercap",
      "/proc/scsi"
    ],
    "readonlyPaths": [
      "/proc/bus",
      "/proc/fs",
      "/proc/irq",
      "/proc/sys",
      "/proc/sysrq-trigger"
    ]
  }
}

# 파드 기동하며 확인
root@k8s-ctr:~/kubespray# cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: ubuntu
spec:
  containers:
  - name: ubuntu
    image: ubuntu
    command: ["sh", "-c", "sleep infinity"]
    securityContext:
      privileged: true
EOF
pod/ubuntu created

root@k8s-ctr:~/kubespray# kubectl exec -it ubuntu -- sh -c 'ulimit -a'
time(seconds)        unlimited
file(blocks)         unlimited
data(kbytes)         unlimited
stack(kbytes)        8192
coredump(blocks)     unlimited
memory(kbytes)       unlimited
locked memory(kbytes) unlimited
process              unlimited
nofiles              65535
vmemory(kbytes)      unlimited
locks                unlimited
rtprio               0

### 참고 ###
커널 전역 한계*
 ├─ fs.file-max
 ├─ file-nr
 └─ inode 캐시

프로세스 한계*
 ├─ RLIMIT_NOFILE (ulimit -n)
 ├─ systemd LimitNOFILE
 └─ PAM limits.conf

cgroup 한계
 ├─ pids.max
 └─ systemd slice 제한

파일시스템
 ├─ inode 수
 ├─ dentry 캐시
 └─ mount 옵션

런타임
 ├─ kubelet / containerd*
 ├─ JVM / Nginx
 └─ epoll / socket 사용량

# 커널 전역 한계
root@k8s-ctr:~/kubespray# sysctl fs.file-max
fs.file-max = 9223372036854775807
root@k8s-ctr:~/kubespray# cat /proc/sys/fs/file-max
9223372036854775807

# 현재 사용량 확인
root@k8s-ctr:~/kubespray# cat /proc/sys/fs/file-nr
2720    0       9223372036854775807

# 프로세스 단위 제한 (사용자 및 프로세스 제한 (Shell/User Level))
root@k8s-ctr:~/kubespray# grep "^[^#]" /etc/security/limits.conf
root@k8s-ctr:~/kubespray# cat /etc/security/limits.conf
# /etc/security/limits.conf
#
#This file sets the resource limits for the users logged in via PAM.
#It does not affect resource limits of the system services.
#
#Also note that configuration files in /etc/security/limits.d directory,
#which are read in alphabetical order, override the settings in this
#file in case the domain is the same or more specific.
#That means, for example, that setting a limit for wildcard domain here
#can be overridden with a wildcard setting in a config file in the
#subdirectory, but a user specific setting here can be overridden only
#with a user specific setting in the subdirectory.
#
#Each line describes a limit for a user in the form:
#
#<domain>        <type>  <item>  <value>
#
#Where:
#<domain> can be:
#        - a user name
#        - a group name, with @group syntax
#        - the wildcard *, for default entry
#        - the wildcard %, can be also used with %group syntax,
#                 for maxlogin limit
#
#<type> can have the two values:
#        - "soft" for enforcing the soft limits
#        - "hard" for enforcing hard limits
#
#<item> can be one of the following:
#        - core - limits the core file size (KB)
#        - data - max data size (KB)
#        - fsize - maximum filesize (KB)
#        - memlock - max locked-in-memory address space (KB)
#        - nofile - max number of open file descriptors
#        - rss - max resident set size (KB)
#        - stack - max stack size (KB)
#        - cpu - max CPU time (MIN)
#        - nproc - max number of processes
#        - as - address space limit (KB)
#        - maxlogins - max number of logins for this user
#        - maxsyslogins - max number of logins on the system
#        - priority - the priority to run user process with
#        - locks - max number of file locks the user can hold
#        - sigpending - max number of pending signals
#        - msgqueue - max memory used by POSIX message queues (bytes)
#        - nice - max nice priority allowed to raise to values: [-20, 19]
#        - rtprio - max realtime priority
#
#<domain>      <type>  <item>         <value>
#

#*               soft    core            0
#*               hard    rss             10000
#@student        hard    nproc           20
#@faculty        soft    nproc           20
#@faculty        hard    nproc           50
#ftp             hard    nproc           0
#@student        -       maxlogins       4

# End of file

root@k8s-ctr:~/kubespray# ulimit -a
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) unlimited
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 15495
max locked memory           (kbytes, -l) 8192
max memory size             (kbytes, -m) unlimited
open files                          (-n) 524288
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) 8192
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 15495
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited
root@k8s-ctr:~/kubespray# ulimit -n
524288


# systemd 서비스 레벨 제한
root@k8s-ctr:~/kubespray# systemctl show kubelet | grep LimitNOFILE
LimitNOFILE=524288
LimitNOFILESoft=1024

# kubelet 프로세스 기준 1,000,000 이지만 -> systemd 524,288 이므로 systemd 적용.
root@k8s-ctr:~/kubespray# cat /proc/$(pidof kubelet)/limits | grep open
Max open files            1000000              1000000              files   

# containerd 프로세스 기준
root@k8s-ctr:~/kubespray# systemctl show containerd | grep LimitNOFILE
LimitNOFILE=1048576
LimitNOFILESoft=1048576

root@k8s-ctr:~/kubespray# cat /proc/$(pidof containerd)/limits | grep open
Max open files            1048576              1048576              files 

# 설정 후 적용
# 기본 OCI Spec(Runtime Spec)을 수정(Patch)
root@k8s-ctr:~/kubespray# cat << EOF >> inventory/mycluster/group_vars/all/containerd.yml
containerd_default_base_runtime_spec_patch:
  process:
    rlimits: []
EOF

# 플레이북 실행 
root@k8s-ctr:~/kubespray# ansible-playbook -i inventory/mycluster/inventory.ini -v cluster.yml --tags "container-engine" --limit k8s-ctr -e kube_version="1.33.3"

root@k8s-ctr:~/kubespray# cat /etc/containerd/cri-base.json | jq | grep rlimits
    "rlimits": [],

root@k8s-ctr:~/kubespray# systemctl restart containerd.service

root@k8s-ctr:~/kubespray# systemctl status containerd.service --no-pager
● containerd.service - containerd container runtime
     Loaded: loaded (/etc/systemd/system/containerd.service; enabled; preset: disabled)
     Active: active (running) since Sun 2026-02-01 07:11:21 KST; 2s ago
 Invocation: 0573ba291be54b13b5cb4d9cd0e9ce1d


# 파드 삭제 후 재기동
root@k8s-ctr:~/kubespray# kubectl delete pod ubuntu
pod "ubuntu" deleted

root@k8s-ctr:~/kubespray# cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: ubuntu
spec:
  containers:
  - name: ubuntu
    image: ubuntu
    command: ["sh", "-c", "sleep infinity"]
    securityContext:
      privileged: true
EOF
pod/ubuntu created
 
# 적용 확인
root@k8s-ctr:~/kubespray# kubectl exec -it ubuntu -- sh -c 'ulimit -a'
time(seconds)        unlimited
file(blocks)         unlimited
data(kbytes)         unlimited
stack(kbytes)        8192
coredump(blocks)     unlimited
memory(kbytes)       unlimited
locked memory(kbytes) unlimited
process              unlimited
nofiles              1048576 ## 확인
vmemory(kbytes)      unlimited
locks                unlimited
rtprio               0

다운로드의 경우 공통적으로 download role을 활용함

root@k8s-ctr:~/kubespray# tree roles/download/
roles/download/
├── meta
│   └── main.yml
├── tasks
│   ├── check_pull_required.yml
│   ├── download_container.yml
│   ├── download_file.yml
│   ├── extract_file.yml
│   ├── main.yml
│   ├── prep_download.yml
│   ├── prep_kubeadm_images.yml
│   └── set_container_facts.yml
└── templates
    └── kubeadm-images.yaml.j2

4 directories, 10 files

kubeadm 관련 바이너리, 컨테이너 이미지 다운로드

root@k8s-ctr:~/kubespray# cat roles/download/tasks/prep_kubeadm_images.yml
---
- name: Prep_kubeadm_images | Download kubeadm binary
  include_tasks: "download_file.yml"
  vars:
    download: "{{ download_defaults | combine(downloads.kubeadm) }}"
  when:
    - not skip_downloads | default(false)
    - downloads.kubeadm.enabled

- name: Prep_kubeadm_images | Copy kubeadm binary from download dir to system path
  copy:
    src: "{{ downloads.kubeadm.dest }}"
    dest: "{{ bin_dir }}/kubeadm"
    mode: "0755"
    remote_src: true

- name: Prep_kubeadm_images | Create kubeadm config
  template:
    src: "kubeadm-images.yaml.j2"
    dest: "{{ kube_config_dir }}/kubeadm-images.yaml"
    mode: "0644"
    validate: "{{ kubeadm_config_validate_enabled | ternary(bin_dir + '/kubeadm config validate --config %s', omit) }}"
  when:
    - not skip_kubeadm_images | default(false)

- name: Prep_kubeadm_images | Generate list of required images
  shell: "set -o pipefail && {{ bin_dir }}/kubeadm config images list --config={{ kube_config_dir }}/kubeadm-images.yaml | grep -Ev 'coredns|pause'"
  args:
    executable: /bin/bash
  register: kubeadm_images_raw
  run_once: true
  changed_when: false
  when:
    - not skip_kubeadm_images | default(false)

- name: Prep_kubeadm_images | Parse list of images
  vars:
    kubeadm_images_list: "{{ kubeadm_images_raw.stdout_lines }}"
  set_fact:
    kubeadm_image:
      key: "kubeadm_{{ (item | regex_replace('^(?:.*\\/)*', '')).split(':')[0] }}"
      value:
        enabled: true
        container: true
        repo: "{{ item | regex_replace('^(.*):.*$', '\\1') }}"
        tag: "{{ item | regex_replace('^.*:(.*)$', '\\1') }}"
        groups:
          - k8s_cluster
  loop: "{{ kubeadm_images_list | flatten(levels=1) }}"
  register: kubeadm_images_cooked
  run_once: true
  when:
    - not skip_kubeadm_images | default(false)

- name: Prep_kubeadm_images | Convert list of images to dict for later use
  set_fact:
    kubeadm_images: "{{ kubeadm_images_cooked.results | map(attribute='ansible_facts.kubeadm_image') | list | items2dict }}"
  run_once: true
  when:
    - not skip_kubeadm_images | default(false)
    
root@k8s-ctr:~/kubespray# cat roles/download/templates/kubeadm-images.yaml.j2
apiVersion: kubeadm.k8s.io/{{ kubeadm_config_api_version }}
kind: InitConfiguration
nodeRegistration:
  criSocket: {{ cri_socket }}
---
apiVersion: kubeadm.k8s.io/{{ kubeadm_config_api_version }}
kind: ClusterConfiguration
imageRepository: {{ kubeadm_image_repo }}
kubernetesVersion: v{{ kube_version }}
etcd:
{% if etcd_deployment_type == "kubeadm" %}
  local:
    imageRepository: "{{ etcd_image_repo | regex_replace("/etcd$","") }}"
    imageTag: "{{ etcd_image_tag }}"
{% else %}
  external:
      endpoints:
{% for endpoint in etcd_access_addresses.split(',') %}
      - {{ endpoint }}
{% endfor %}
{% endif %}
dns:
  imageRepository: {{ coredns_image_repo | regex_replace('/coredns(?!/coredns).*$', '') }}
  imageTag: {{ coredns_image_tag }}
  

root@k8s-ctr:~/kubespray# nerdctl -n k8s.io images
REPOSITORY                                             TAG                IMAGE ID        CREATED           PLATFORM       SIZE       BLOB SIZE
ubuntu                                                 <none>             cd1dba651b30    27 minutes ago    linux/arm64    107.8MB    28.87MB
<none>                                                 <none>             cd1dba651b30    27 minutes ago    linux/arm64    107.8MB    28.87MB
ubuntu                                                 latest             cd1dba651b30    27 minutes ago    linux/arm64    107.8MB    28.87MB
registry.k8s.io/nfd/node-feature-discovery             <none>             d3f0fb2d50c2    2 hours ago       linux/arm64    219MB      62.6MB
<none>                                                 <none>             d3f0fb2d50c2    2 hours ago       linux/arm64    219MB      62.6MB
registry.k8s.io/nfd/node-feature-discovery             v0.16.4            d3f0fb2d50c2    2 hours ago       linux/arm64    219MB      62.6MB
<none>                                                 <none>             c69929cfba9e    2 hours ago       linux/arm64    103.1MB    28.2MB
registry.k8s.io/kube-proxy                             v1.33.3            c69929cfba9e    2 hours ago       linux/arm64    103.1MB    28.2MB
<none>                                                 <none>             f3a2ffdd7483    2 hours ago       linux/arm64    73.42MB    19.85MB
registry.k8s.io/kube-scheduler                         v1.33.3            f3a2ffdd7483    2 hours ago       linux/arm64    73.42MB    19.85MB
<none>                                                 <none>             96091626e37c    2 hours ago       linux/arm64    93.34MB    25.09MB
registry.k8s.io/kube-controller-manager                v1.33.3            96091626e37c    2 hours ago       linux/arm64    93.34MB    25.09MB
<none>                                                 <none>             125a8b488def    2 hours ago       linux/arm64    99.89MB    27.35MB
registry.k8s.io/kube-apiserver                         v1.33.3            125a8b488def    2 hours ago       linux/arm64    99.89MB    27.35MB
<none>                                                 <none>             89258156d0e9    2 hours ago       linux/arm64    82.58MB    20.58MB
registry.k8s.io/metrics-server/metrics-server          v0.8.0             89258156d0e9    2 hours ago       linux/arm64    82.58MB    20.58MB
<none>                                                 <none>             69bf675e3567    2 hours ago       linux/arm64    39.98MB    10.43MB
registry.k8s.io/cpa/cluster-proportional-autoscaler    v1.8.8             69bf675e3567    2 hours ago       linux/arm64    39.98MB    10.43MB
<none>                                                 <none>             40384aa1f5ea    2 hours ago       linux/arm64    71.2MB     19.15MB
registry.k8s.io/coredns/coredns                        v1.12.0            40384aa1f5ea    2 hours ago       linux/arm64    71.2MB     19.15MB
<none>                                                 <none>             30f1c0d78e0a    2 hours ago       linux/arm64    52.73MB    21.85MB
nginx                                                  1.28.0-alpine      30f1c0d78e0a    2 hours ago       linux/arm64    52.73MB    21.85MB
<none>                                                 <none>             ee6521f290b2    2 hours ago       linux/arm64    516.1kB    265.5kB
registry.k8s.io/pause                                  3.10               ee6521f290b2    2 hours ago       linux/arm64    516.1kB    265.5kB
<none>                                                 <none>             39d51a8cf650    2 hours ago       linux/arm64    11.39MB    5.136MB
flannel/flannel-cni-plugin                             v1.7.1-flannel1    39d51a8cf650    2 hours ago       linux/arm64    11.39MB    5.136MB
<none>                                                 <none>             478ca1ac04e4    2 hours ago       linux/arm64    102.6MB    33.08MB
flannel/flannel                                        v0.27.3            478ca1ac04e4    2 hours ago       linux/arm64    102.6MB    33.08MB

[K8s Deploy] Kubeadm Deep Dive

안녕유지 — Sat, 24 Jan 2026 19:23:37 +0900

Cloudnet K8s Deploy 3주차 스터디를 진행하며 정리한 글입니다.

Kubeadm 이란

Kubeadm은 Kubernetes에서 제공하는 공식 클러스터 부트스트랩 도구로, kube-apiserver, etcd, kube-controller-manager, kube-scheduler와 같은 컨트롤 플레인 구성요소를 표준화된 방식으로 설치·구성해줍니다.

복잡한 수작업 설정 없이도 kubeadm init, kubeadm join 명령만으로 Kubernetes 클러스터를 빠르고 일관되게 생성할 수 있도록 설계되었습니다.

Kubeadm은 SIG Cluster Lifecycle에서 관리하며, 클러스터의

생성 (init / join)
업그레이드
재구성
초기화(reset)

와 같은 클러스터 라이프사이클 전반의 기반 역할을 담당합니다.

Kubeadm으로 K8s 구성 절차

kubeadm으로 구성할 k8s 정보는 다음과 같습니다.

항목	버전	k8s 버전 호환성
Rocky Linux	10.0-1.6	RHEL 10 소스 기반 배포판으로 RHEL 정보 참고
containerd	v2.1.5	CRI Version(v1), k8s 1.32~1.35 지원 - Link
runc	v1.3.3	정보 조사 필요 https://github.com/opencontainers/runc
kubelet	v1.32.11	k8s 버전 정책 문서 참고 - Docs
kubeadm	v1.32.11	상동
kubectl	v1.32.11	상동
helm	v3.18.6	k8s 1.30.x ~ 1.33.x 지원 - Docs
flannel cni	v0.27.3	k8s 1.28~ 이후 - Release

1. [공통] 사전 설정

1-1. root 권한(로그인 환경) 전환

1-2. Time, NTP 설정 : 인증서 만료 시간, 로그 타임스탬프 등 모든 노드에 동기화된 시간 필요

(⎈|HomeLab:N/A) root@k8s-ctr:~# sudo su -
Last login: Sat Jan 24 17:26:50 KST 2026 on pts/1

# timedatectl 정보 확인
(⎈|HomeLab:N/A) root@k8s-ctr:~# timedatectl status
               Local time: Sat 2026-01-24 17:27:09 KST
           Universal time: Sat 2026-01-24 08:27:09 UTC
                 RTC time: Sat 2026-01-24 10:49:41
                Time zone: Asia/Seoul (KST, +0900)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no
(⎈|HomeLab:N/A) root@k8s-ctr:~# timedatectl set-local-rtc 0
(⎈|HomeLab:N/A) root@k8s-ctr:~# timedatectl status
               Local time: Sat 2026-01-24 17:27:16 KST
           Universal time: Sat 2026-01-24 08:27:16 UTC
                 RTC time: Sat 2026-01-24 10:49:49
                Time zone: Asia/Seoul (KST, +0900)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no
          
# 시스템 타임존(Timezone)을 한국(KST, UTC+9) 으로 설정 : 시스템 시간은 UTC 기준 유지, 표시만 KST로 변환
(⎈|HomeLab:N/A) root@k8s-ctr:~# date
(⎈|HomeLab:N/A) root@k8s-ctr:~# timedatectl set-timezone Asia/Seoul
(⎈|HomeLab:N/A) root@k8s-ctr:~# date
Sat Jan 24 05:27:27 PM KST 2026

# systemd가 시간 동기화 서비스(chronyd) 를 관리하도록 설정되어 있음 : ntpd 대신 chrony 사용 (Rocky 9/10 기본)
(⎈|HomeLab:N/A) root@k8s-ctr:~# date
Sat Jan 24 05:27:27 PM KST 2026
(⎈|HomeLab:N/A) root@k8s-ctr:~# timedatectl status
               Local time: Sat 2026-01-24 17:28:19 KST
           Universal time: Sat 2026-01-24 08:28:19 UTC
                 RTC time: Sat 2026-01-24 10:50:51
                Time zone: Asia/Seoul (KST, +0900)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no
(⎈|HomeLab:N/A) root@k8s-ctr:~# timedatectl set-ntp true


# chronyc 확인
# chrony가 어떤 NTP 서버들을 알고 있고, 그중 어떤 서버를 기준으로 시간을 맞추는지를 보여줍니다.
## Stratum 2: 매우 신뢰도 높은 서버
## Reach 377: 최근 8회 연속 응답 성공 (최대값)
(⎈|HomeLab:N/A) root@k8s-ctr:~# chronyc sources -v

  .-- Source mode  '^' = server, '=' = peer, '#' = local clock.
 / .- Source state '*' = current best, '+' = combined, '-' = not combined,
| /             'x' = may be in error, '~' = too variable, '?' = unusable.
||                                                 .- xxxx [ yyyy ] +/- zzzz
||      Reachability register (octal) -.           |  xxxx = adjusted offset,
||      Log2(Polling interval) --.      |          |  yyyy = measured offset,
||                                \     |          |  zzzz = estimated error.
||                                 |    |           \
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^- 240b:400d:3:3300:aeda:71>     2   8   377   144    +45ms[  +45ms] +/-  102ms
^- mail.innotab.com              3   7   377    94  +6840us[+6840us] +/-   42ms
^* 211.108.117.211               2   8   377   165   -518us[-1073us] +/- 8817us
^- 2401:c080:1c00:24a1:5400>     2   8   377    38  +3131us[+3131us] +/-   78ms

# 현재 시스템 시간이 얼마나 정확한지 종합 성적표
(⎈|HomeLab:N/A) root@k8s-ctr:~# chronyc tracking
Reference ID    : D36C75D3 (211.108.117.211)
Stratum         : 3
Ref time (UTC)  : Sat Jan 24 08:26:04 2026
System time     : 0.000317448 seconds slow of NTP time
Last offset     : -0.000554537 seconds
RMS offset      : 0.520825684 seconds
Frequency       : 7.186 ppm fast
Residual freq   : -0.053 ppm
Skew            : 4.019 ppm
Root delay      : 0.010728337 seconds
Root dispersion : 0.003451450 seconds
Update interval : 256.3 seconds
Leap status     : Normal

1-3. SELinux 설정, firewalld(방화벽) 끄기

# SELinux 설정 : Kubernetes는 Permissive 권장
(⎈|HomeLab:N/A) root@k8s-ctr:~# getenforce
Permissive
(⎈|HomeLab:N/A) root@k8s-ctr:~# sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   permissive
Mode from config file:          permissive
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Memory protection checking:     actual (secure)
Max kernel policy version:      33
(⎈|HomeLab:N/A) root@k8s-ctr:~# setenforce 0
(⎈|HomeLab:N/A) root@k8s-ctr:~# getenforce
Permissive
(⎈|HomeLab:N/A) root@k8s-ctr:~# sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   permissive
Mode from config file:          permissive
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Memory protection checking:     actual (secure)
Max kernel policy version:      33

# 재부팅 시에도 Permissive 적용
(⎈|HomeLab:N/A) root@k8s-ctr:~# cat /etc/selinux/config | grep ^SELINUX
SELINUX=permissive
SELINUXTYPE=targeted
(⎈|HomeLab:N/A) root@k8s-ctr:~# sed -i 's/^SELINUX=enforcing/SELINUX=permissive/' /etc/selinux/config
(⎈|HomeLab:N/A) root@k8s-ctr:~# cat /etc/selinux/config | grep ^SELINUX
SELINUX=permissive
SELINUXTYPE=targeted

# firewalld(방화벽) 끄기
(⎈|HomeLab:N/A) root@k8s-ctr:~# systemctl status firewalld
○ firewalld.service - firewalld - dynamic firewall daemon
     Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; preset: enabled)
     Active: inactive (dead)
       Docs: man:firewalld(1)

Jan 24 14:57:10 localhost systemd[1]: Starting firewalld.service - firewalld - dynamic firewall daemon...
Jan 24 14:57:11 localhost systemd[1]: Started firewalld.service - firewalld - dynamic firewall daemon.
Jan 24 14:57:23 k8s-ctr systemd[1]: Stopping firewalld.service - firewalld - dynamic firewall daemon...
Jan 24 14:57:24 k8s-ctr systemd[1]: firewalld.service: Deactivated successfully.
Jan 24 14:57:24 k8s-ctr systemd[1]: Stopped firewalld.service - firewalld - dynamic firewall daemon.
Jan 24 14:57:24 k8s-ctr systemd[1]: firewalld.service: Consumed 541ms CPU time, 70.7M memory peak.
(⎈|HomeLab:N/A) root@k8s-ctr:~# systemctl disable --now firewalld
(⎈|HomeLab:N/A) root@k8s-ctr:~# systemctl status firewalld
○ firewalld.service - firewalld - dynamic firewall daemon
     Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; preset: enabled)
     Active: inactive (dead)
       Docs: man:firewalld(1)

Jan 24 14:57:10 localhost systemd[1]: Starting firewalld.service - firewalld - dynamic firewall daemon...
Jan 24 14:57:11 localhost systemd[1]: Started firewalld.service - firewalld - dynamic firewall daemon.
Jan 24 14:57:23 k8s-ctr systemd[1]: Stopping firewalld.service - firewalld - dynamic firewall daemon...
Jan 24 14:57:24 k8s-ctr systemd[1]: firewalld.service: Deactivated successfully.
Jan 24 14:57:24 k8s-ctr systemd[1]: Stopped firewalld.service - firewalld - dynamic firewall daemon.
Jan 24 14:57:24 k8s-ctr systemd[1]: firewalld.service: Consumed 541ms CPU time, 70.7M memory peak.

1-4. Swap 비활성화

# Swap 비활성화
(⎈|HomeLab:N/A) root@k8s-ctr:~# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda      8:0    0   64G  0 disk
├─sda1   8:1    0  600M  0 part /boot/efi
└─sda3   8:3    0 59.6G  0 part /
(⎈|HomeLab:N/A) root@k8s-ctr:~# free -h
               total        used        free      shared  buff/cache   available
Mem:           2.8Gi       911Mi       471Mi        19Mi       1.5Gi       1.9Gi
Swap:             0B          0B          0B
(⎈|HomeLab:N/A) root@k8s-ctr:~# free -h | grep Swap
Swap:             0B          0B          0B
(⎈|HomeLab:N/A) root@k8s-ctr:~# swapoff -a
(⎈|HomeLab:N/A) root@k8s-ctr:~# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda      8:0    0   64G  0 disk
├─sda1   8:1    0  600M  0 part /boot/efi
└─sda3   8:3    0 59.6G  0 part /
(⎈|HomeLab:N/A) root@k8s-ctr:~# free -h | grep Swap
Swap:             0B          0B          0B


# 재부팅 시에도 'Swap 비활성화' 적용되도록 /etc/fstab에서 swap 라인 삭제
(⎈|HomeLab:N/A) root@k8s-ctr:~# cat /etc/fstab | grep swap
(⎈|HomeLab:N/A) root@k8s-ctr:~# sed -i '/swap/d' /etc/fstab
(⎈|HomeLab:N/A) root@k8s-ctr:~# cat /etc/fstab | grep swap

1-5. 커널 모듈 및 커널 파라미터 설정

# 커널 모듈 확인
(⎈|HomeLab:N/A) root@k8s-ctr:~# lsmod | grep -iE 'overlay|br_netfilter'
br_netfilter           32768  0
bridge                327680  1 br_netfilter
overlay               200704  16

# 커널 모듈 로드
(⎈|HomeLab:N/A) root@k8s-ctr:~# modprobe overlay
(⎈|HomeLab:N/A) root@k8s-ctr:~# modprobe br_netfilter
(⎈|HomeLab:N/A) root@k8s-ctr:~# lsmod | grep -iE 'overlay|br_netfilter'
br_netfilter           32768  0
bridge                327680  1 br_netfilter
overlay               200704  16


(⎈|HomeLab:N/A) root@k8s-ctr:~# cat <<EOF | tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
overlay
br_netfilter
(⎈|HomeLab:N/A) root@k8s-ctr:~# tree /etc/modules-load.d/
/etc/modules-load.d/
└── k8s.conf

1 directory, 1 file


# 커널 파라미터 설정 : 네트워크 설정 - 브릿지 트래픽이 iptables를 거치도록 함
(⎈|HomeLab:N/A) root@k8s-ctr:~# cat <<EOF | tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
(⎈|HomeLab:N/A) root@k8s-ctr:~# tree /etc/sysctl.d/
/etc/sysctl.d/
├── 99-sysctl.conf -> ../sysctl.conf
└── k8s.conf

1 directory, 2 files

# 설정 적용
(⎈|HomeLab:N/A) root@k8s-ctr:~# sysctl --system
* Applying /usr/lib/sysctl.d/10-default-yama-scope.conf ...
* Applying /usr/lib/sysctl.d/10-map-count.conf ...
* Applying /usr/lib/sysctl.d/50-coredump.conf ...
* Applying /usr/lib/sysctl.d/50-default.conf ...
* Applying /usr/lib/sysctl.d/50-libkcapi-optmem_max.conf ...
* Applying /usr/lib/sysctl.d/50-pid-max.conf ...
* Applying /usr/lib/sysctl.d/50-redhat.conf ...
* Applying /etc/sysctl.d/99-sysctl.conf ...
* Applying /etc/sysctl.d/k8s.conf ...
* Applying /etc/sysctl.conf ...
kernel.yama.ptrace_scope = 0
vm.max_map_count = 1048576
kernel.core_pattern = |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h
kernel.core_pipe_limit = 16
fs.suid_dumpable = 2
kernel.sysrq = 16
kernel.core_uses_pid = 1
net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.cni0.rp_filter = 2
net.ipv4.conf.enp0s8.rp_filter = 2
net.ipv4.conf.enp0s9.rp_filter = 2
net.ipv4.conf.flannel/1.rp_filter = 2
net.ipv4.conf.lo.rp_filter = 2
net.ipv4.conf.veth066523cb.rp_filter = 2
net.ipv4.conf.veth96506d84.rp_filter = 2
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.cni0.accept_source_route = 0
net.ipv4.conf.enp0s8.accept_source_route = 0
net.ipv4.conf.enp0s9.accept_source_route = 0
net.ipv4.conf.flannel/1.accept_source_route = 0
net.ipv4.conf.lo.accept_source_route = 0
net.ipv4.conf.veth066523cb.accept_source_route = 0
net.ipv4.conf.veth96506d84.accept_source_route = 0
net.ipv4.conf.default.promote_secondaries = 1
net.ipv4.conf.cni0.promote_secondaries = 1
net.ipv4.conf.enp0s8.promote_secondaries = 1
net.ipv4.conf.enp0s9.promote_secondaries = 1
net.ipv4.conf.flannel/1.promote_secondaries = 1
net.ipv4.conf.lo.promote_secondaries = 1
net.ipv4.conf.veth066523cb.promote_secondaries = 1
net.ipv4.conf.veth96506d84.promote_secondaries = 1
net.ipv4.ping_group_range = 0 2147483647
net.core.default_qdisc = fq_codel
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
fs.protected_regular = 1
fs.protected_fifos = 1
net.core.optmem_max = 81920
kernel.pid_max = 4194304
kernel.kptr_restrict = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.cni0.rp_filter = 1
net.ipv4.conf.enp0s8.rp_filter = 1
net.ipv4.conf.enp0s9.rp_filter = 1
net.ipv4.conf.flannel/1.rp_filter = 1
net.ipv4.conf.lo.rp_filter = 1
net.ipv4.conf.veth066523cb.rp_filter = 1
net.ipv4.conf.veth96506d84.rp_filter = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1


(⎈|HomeLab:N/A) root@k8s-ctr:~# sysctl net.bridge.bridge-nf-call-iptables
net.bridge.bridge-nf-call-iptables = 1
(⎈|HomeLab:N/A) root@k8s-ctr:~# sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1

1-6. hosts 설정

# hosts 설정
(⎈|HomeLab:N/A) root@k8s-ctr:~# cat /etc/hosts
# Loopback entries; do not change.
# For historical reasons, localhost precedes localhost.localdomain:
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
# See hosts(5) for proper format and other examples:
# 192.168.1.10 foo.example.org foo
# 192.168.1.13 bar.example.org bar
192.168.10.100 k8s-ctr
192.168.10.101 k8s-w1
192.168.10.102 k8s-w2
(⎈|HomeLab:N/A) root@k8s-ctr:~# sed -i '/^127\.0\.\(1\|2\)\.1/d' /etc/hosts
(⎈|HomeLab:N/A) root@k8s-ctr:~# cat << EOF >> /etc/hosts
192.168.10.100 k8s-ctr
192.168.10.101 k8s-w1
192.168.10.102 k8s-w2
EOF
(⎈|HomeLab:N/A) root@k8s-ctr:~# cat /etc/hosts
# Loopback entries; do not change.
# For historical reasons, localhost precedes localhost.localdomain:
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
# See hosts(5) for proper format and other examples:
# 192.168.1.10 foo.example.org foo
# 192.168.1.13 bar.example.org bar
192.168.10.100 k8s-ctr
192.168.10.101 k8s-w1
192.168.10.102 k8s-w2
192.168.10.100 k8s-ctr
192.168.10.101 k8s-w1
192.168.10.102 k8s-w2


# 확인
(⎈|HomeLab:N/A) root@k8s-ctr:~# ping -c 1 k8s-ctr
PING k8s-ctr (192.168.10.100) 56(84) bytes of data.
64 bytes from k8s-ctr (192.168.10.100): icmp_seq=1 ttl=64 time=0.050 ms

--- k8s-ctr ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.050/0.050/0.050/0.000 ms
(⎈|HomeLab:N/A) root@k8s-ctr:~# ping -c 1 k8s-w1
PING k8s-w1 (192.168.10.101) 56(84) bytes of data.
64 bytes from k8s-w1 (192.168.10.101): icmp_seq=1 ttl=64 time=0.644 ms

--- k8s-w1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.644/0.644/0.644/0.000 ms
(⎈|HomeLab:N/A) root@k8s-ctr:~# ping -c 1 k8s-w2
PING k8s-w2 (192.168.10.102) 56(84) bytes of data.
64 bytes from k8s-w2 (192.168.10.102): icmp_seq=1 ttl=64 time=0.511 ms

--- k8s-w2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.511/0.511/0.511/0.000 ms

2. [공통] CRI 설치 : containerd

2-1. containerd(runc) v2.1.5 설치

# dnf == yum, 버전 정보 확인
(⎈|HomeLab:N/A) root@k8s-ctr:~# dnf repolist
repo id                                                                                   repo name
appstream                                                                                 Rocky Linux 10 - AppStream
baseos                                                                                    Rocky Linux 10 - BaseOS
docker-ce-stable                                                                          Docker CE Stable - aarch64
extras                                                                                    Rocky Linux 10 - Extras
kubecolor                                                                                 packages for kubecolor
kubernetes                                                                                Kubernetes
(⎈|HomeLab:N/A) root@k8s-ctr:~# tree /etc/yum.repos.d/
/etc/yum.repos.d/
├── docker-ce.repo
├── kubecolor.repo
├── kubernetes.repo
├── rocky-addons.repo
├── rocky-devel.repo
├── rocky-extras.repo
└── rocky.repo

1 directory, 7 files
(⎈|HomeLab:N/A) root@k8s-ctr:~# dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
Adding repo from: https://download.docker.com/linux/centos/docker-ce.repo
(⎈|HomeLab:N/A) root@k8s-ctr:~# dnf repolist
repo id                                                                                   repo name
appstream                                                                                 Rocky Linux 10 - AppStream
baseos                                                                                    Rocky Linux 10 - BaseOS
docker-ce-stable                                                                          Docker CE Stable - aarch64
extras                                                                                    Rocky Linux 10 - Extras
kubecolor                                                                                 packages for kubecolor
kubernetes                                                                                Kubernetes
(⎈|HomeLab:N/A) root@k8s-ctr:~# tree /etc/yum.repos.d/
/etc/yum.repos.d/
├── docker-ce.repo
├── kubecolor.repo
├── kubernetes.repo
├── rocky-addons.repo
├── rocky-devel.repo
├── rocky-extras.repo
└── rocky.repo

1 directory, 7 files
(⎈|HomeLab:N/A) root@k8s-ctr:~# cat /etc/yum.repos.d/docker-ce.repo
[docker-ce-stable]
name=Docker CE Stable - $basearch
baseurl=https://download.docker.com/linux/centos/$releasever/$basearch/stable
enabled=1
gpgcheck=1
gpgkey=https://download.docker.com/linux/centos/gpg

[docker-ce-stable-source]
name=Docker CE Stable - Sources
baseurl=https://download.docker.com/linux/centos/$releasever/source/stable
enabled=0
gpgcheck=1
gpgkey=https://download.docker.com/linux/centos/gpg

[docker-ce-test]
name=Docker CE Test - $basearch
baseurl=https://download.docker.com/linux/centos/$releasever/$basearch/test
enabled=0
gpgcheck=1
gpgkey=https://download.docker.com/linux/centos/gpg

[docker-ce-test-source]
name=Docker CE Test - Sources
baseurl=https://download.docker.com/linux/centos/$releasever/source/test
enabled=0
gpgcheck=1
gpgkey=https://download.docker.com/linux/centos/gpg


# 설치 가능한 모든 containerd.io 버전 확인
(⎈|HomeLab:N/A) root@k8s-ctr:~# dnf list --showduplicates containerd.io
Docker CE Stable - aarch64                                                                                                                                   398  B/s | 2.0 kB     00:05
Installed Packages
containerd.io.aarch64                                                                    2.1.5-1.el10                                                                       @docker-ce-stable
Available Packages
containerd.io.aarch64                                                                    1.7.23-3.1.el10                                                                    docker-ce-stable
containerd.io.aarch64                                                                    1.7.24-3.1.el10                                                                    docker-ce-stable
containerd.io.aarch64                                                                    1.7.25-3.1.el10                                                                    docker-ce-stable
containerd.io.aarch64                                                                    1.7.26-3.1.el10                                                                    docker-ce-stable
containerd.io.aarch64                                                                    1.7.27-3.1.el10                                                                    docker-ce-stable
containerd.io.aarch64                                                                    1.7.28-1.el10                                                                      docker-ce-stable
containerd.io.aarch64                                                                    1.7.28-2.el10                                                                      docker-ce-stable
containerd.io.aarch64                                                                    1.7.29-1.el10                                                                      docker-ce-stable
containerd.io.aarch64                                                                    2.1.5-1.el10                                                                       docker-ce-stable
containerd.io.aarch64                                                                    2.2.0-2.el10                                                                       docker-ce-stable
containerd.io.aarch64                                                                    2.2.1-1.el10                                                                       docker-ce-stable


# containerd 설치
(⎈|HomeLab:N/A) root@k8s-ctr:~# dnf install -y containerd.io-2.1.5-1.el10
Last metadata expiration check: 0:00:44 ago on Sat 24 Jan 2026 05:39:48 PM KST.
Package containerd.io-2.1.5-1.el10.aarch64 is already installed.
Dependencies resolved.
Nothing to do.
Complete!

# 설치된 파일 확인
(⎈|HomeLab:N/A) root@k8s-ctr:~# which runc && runc --version
/usr/bin/runc
runc version 1.3.3
commit: v1.3.3-0-gd842d771
spec: 1.2.1
go: go1.24.9
libseccomp: 2.5.3
(⎈|HomeLab:N/A) root@k8s-ctr:~# which containerd && containerd --version
/usr/bin/containerd
containerd containerd.io v2.1.5 fcd43222d6b07379a4be9786bda52438f0dd16a1
(⎈|HomeLab:N/A) root@k8s-ctr:~# which containerd-shim-runc-v2 && containerd-shim-runc-v2 -v
/usr/bin/containerd-shim-runc-v2
containerd-shim-runc-v2:
  Version:  v2.1.5
  Revision: fcd43222d6b07379a4be9786bda52438f0dd16a1
  Go version: go1.24.9

(⎈|HomeLab:N/A) root@k8s-ctr:~# which ctr && ctr --version
/usr/bin/ctr
ctr containerd.io v2.1.5


# 기본 설정 생성 및 SystemdCgroup 활성화 (매우 중요)
(⎈|HomeLab:N/A) root@k8s-ctr:~# cat /etc/containerd/config.toml | grep -i systemdcgroup
            SystemdCgroup = false
(⎈|HomeLab:N/A) root@k8s-ctr:~# sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml
(⎈|HomeLab:N/A) root@k8s-ctr:~# cat /etc/containerd/config.toml | grep -i systemdcgroup
            SystemdCgroup = true
            
            
# containerd start 와 enabled            
(⎈|HomeLab:N/A) root@k8s-ctr:~# systemctl enable --now containerd
(⎈|HomeLab:N/A) root@k8s-ctr:~# systemctl status containerd --no-pager
● containerd.service - containerd container runtime
     Loaded: loaded (/usr/lib/systemd/system/containerd.service; enabled; preset: disabled)
     Active: active (running) since Sat 2026-01-24 14:58:47 KST; 2h 45min ago
 Invocation: 15a09f3b93124c9986fb4913578e0154
       Docs: https://containerd.io
   Main PID: 5008 (containerd)
      Tasks: 121
     Memory: 855.5M (peak: 873.2M)
        CPU: 2min 6.573s
     CGroup: /system.slice/containerd.service
             ├─5008 /usr/bin/containerd
             ├─5931 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id c91cc3779b6d53b845614ee32d79e7281e7e6004cabd8ddcd0bfbd27bde927c2 -address /run/containerd/containerd.sock
             ├─5952 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id c3a07a12edde49cc5ce299cbd849e91e2c58b00b215b8991ffb2e1577cf88e92 -address /run/containerd/containerd.sock
             ├─6022 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id 7f865707e4b038307dedfeb9cb8a49a094c9daf1462517efecbff651fc5c6619 -address /run/containerd/containerd.sock
             ├─6039 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id aa47ccd8f7693f24dde5d70646bbf83580e12abffe191f2cbb3b03984ee2e1d9 -address /run/containerd/containerd.sock
             ├─6368 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id 4b69ed2f8eeec3eb4db3fee2291bfa38b54b2de942109ef9409cd48bd4fbc0b4 -address /run/containerd/containerd.sock
             ├─6862 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id fc0ef56b33f8291bbbd5fe23fba807c3347f0991d80a97da022107f3ddfaa55e -address /run/containerd/containerd.sock
             ├─7280 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id cad15631e844399414452ca6597a58165af7c4096d66804ce632c54d54f1c756 -address /run/containerd/containerd.sock
             └─7290 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id 188e8bdda1af31fe0d3591245aeef1bec7258a23ead3d0c423d372905e7d3a5f -address /run/containerd/containerd.sock
             
             
             
(⎈|HomeLab:N/A) root@k8s-ctr:~# systemctl status containerd --no-pager
(⎈|HomeLab:N/A) root@k8s-ctr:~# journalctl -u containerd.service --no-pager
(⎈|HomeLab:N/A) root@k8s-ctr:~# pstree -alnp
(⎈|HomeLab:N/A) root@k8s-ctr:~# systemd-cgls --no-pager


# containerd의 유닉스 도메인 소켓 확인 : kubelet에서 사용 , containerd client 3종(ctr, nerdctr, crictl)도 사용
(⎈|HomeLab:N/A) root@k8s-ctr:~# containerd config dump | grep -n containerd.sock
11:  address = '/run/containerd/containerd.sock'
(⎈|HomeLab:N/A) root@k8s-ctr:~# ls -l /run/containerd/containerd.sock
srw-rw----. 1 root root 0 Jan 24 14:58 /run/containerd/containerd.sock
(⎈|HomeLab:N/A) root@k8s-ctr:~# ss -xl | grep containerd
u_str LISTEN 0      4096                                                /run/containerd/containerd.sock.ttrpc 17656             * 0
u_str LISTEN 0      4096                                                      /run/containerd/containerd.sock 18544             * 0
u_str LISTEN 0      4096   /run/containerd/s/6a1eec9193b24526a055dd3c67beb7a02bb755b4bdef487dfbb58ff75e2dc31c 27398             * 0
u_str LISTEN 0      4096   /run/containerd/s/3c5471325c662956ea7099bf0f607aed6f94b4d5ed7de841d808da09647fd65f 21167             * 0
u_str LISTEN 0      4096   /run/containerd/s/323bfd1bd6ebd2ce9a3420b50eae4533fe80584ab525e264c058efda3c5be61a 23173             * 0
u_str LISTEN 0      4096   /run/containerd/s/05d2b657bfce55010895d0fcbf5ef9c84cbcdb22d51de9420dc5d009153345e8 21609             * 0
u_str LISTEN 0      4096   /run/containerd/s/59d7141628d54c0532476756e3589d8ed513fa390c889248db7d1474c5d10011 32850             * 0
u_str LISTEN 0      4096   /run/containerd/s/96e904766632d214a6e4f0a072e9cb3f650edbe5f67d9ab6d2a7a95a685680ae 31855             * 0
u_str LISTEN 0      4096   /run/containerd/s/6439866fe5db71cda35a979ec11ff91aebc5066eb838654ba8a3e502089642f8 21775             * 0
u_str LISTEN 0      4096   /run/containerd/s/97c57dfb95ff3f013f11b4dc2f2b0e00f04e2b0c6b93c7e67ed34b1cd962e667 21782             * 0
(⎈|HomeLab:N/A) root@k8s-ctr:~# ss -xnp | grep containerd
u_str ESTAB 0      0                                                                                       * 21789             * 21790  users:(("containerd",pid=5008,fd=43))
u_str ESTAB 0      0      /run/containerd/s/323bfd1bd6ebd2ce9a3420b50eae4533fe80584ab525e264c058efda3c5be61a 23262             * 23966  users:(("containerd-shim",pid=6368,fd=12))
u_str ESTAB 0      0                                                   /run/containerd/containerd.sock.ttrpc 20049             * 21731  users:(("containerd",pid=5008,fd=32))
u_str ESTAB 0      0                                                                                       * 21170             * 19349  users:(("containerd",pid=5008,fd=29))
u_str ESTAB 0      0                                                                                       * 19443             * 20065  users:(("containerd",pid=5008,fd=40))
u_str ESTAB 0      0                                                                                       * 23962             * 22331  users:(("containerd-shim",pid=6368,fd=11))
u_str ESTAB 0      0                                                   /run/containerd/containerd.sock.ttrpc 21728             * 21727  users:(("containerd",pid=5008,fd=31))
u_str ESTAB 0      0                                                                                       * 17083             * 18535  users:(("containerd",pid=5008,fd=2),("containerd",pid=5008,fd=1))
u_str ESTAB 0      0                                                   /run/containerd/containerd.sock.ttrpc 21910             * 22665  users:(("containerd",pid=5008,fd=55))
u_str ESTAB 0      0                                                                                       * 21727             * 21728  users:(("containerd-shim",pid=5931,fd=3))
u_str ESTAB 0      0                                                                                       * 22665             * 21910  users:(("containerd-shim",pid=6039,fd=11))
u_str ESTAB 0      0      /run/containerd/s/97c57dfb95ff3f013f11b4dc2f2b0e00f04e2b0c6b93c7e67ed34b1cd962e667 21790             * 21789  users:(("containerd-shim",pid=6039,fd=10))
u_str ESTAB 0      0                                                   /run/containerd/containerd.sock.ttrpc 22331             * 23962  users:(("containerd",pid=5008,fd=15))
u_str ESTAB 0      0      /run/containerd/s/6439866fe5db71cda35a979ec11ff91aebc5066eb838654ba8a3e502089642f8 20065             * 19443  users:(("containerd-shim",pid=6022,fd=10))
u_str ESTAB 0      0                                                                                       * 21731             * 20049  users:(("containerd-shim",pid=5952,fd=11))
u_str ESTAB 0      0      /run/containerd/s/3c5471325c662956ea7099bf0f607aed6f94b4d5ed7de841d808da09647fd65f 19349             * 21170  users:(("containerd-shim",pid=5931,fd=10))
u_str ESTAB 0      0                                                                                       * 23966             * 23262  users:(("containerd",pid=5008,fd=79))
u_str ESTAB 0      0                                                                                       * 28060             * 29048  users:(("containerd",pid=5008,fd=23))
u_str ESTAB 0      0                                                                                       * 21291             * 21292  users:(("containerd",pid=5008,fd=54))
u_str ESTAB 0      0                                                   /run/containerd/containerd.sock.ttrpc 28118             * 25493  users:(("containerd",pid=5008,fd=28))
u_str ESTAB 0      0                                                                                       * 19354             * 21612  users:(("containerd",pid=5008,fd=30))
u_str ESTAB 0      0      /run/containerd/s/96e904766632d214a6e4f0a072e9cb3f650edbe5f67d9ab6d2a7a95a685680ae 32955             * 32954  users:(("containerd-shim",pid=7290,fd=12))
u_str ESTAB 0      0                                                                                       * 21284             * 21285  users:(("containerd",pid=5008,fd=51))
u_str ESTAB 0      0                                                                                       * 25493             * 28118  users:(("containerd-shim",pid=6862,fd=11))
u_str ESTAB 0      0      /run/containerd/s/05d2b657bfce55010895d0fcbf5ef9c84cbcdb22d51de9420dc5d009153345e8 21292             * 21291  users:(("containerd-shim",pid=5952,fd=3))
u_str ESTAB 0      0                                                   /run/containerd/containerd.sock.ttrpc 30422             * 32924  users:(("containerd",pid=5008,fd=99))
u_str ESTAB 0      0                                                                                       * 30332             * 32854  users:(("containerd",pid=5008,fd=96))
u_str ESTAB 0      0      /run/containerd/s/97c57dfb95ff3f013f11b4dc2f2b0e00f04e2b0c6b93c7e67ed34b1cd962e667 23561             * 23560  users:(("containerd-shim",pid=6039,fd=12))
u_str ESTAB 0      0      /run/containerd/s/6a1eec9193b24526a055dd3c67beb7a02bb755b4bdef487dfbb58ff75e2dc31c 29048             * 28060  users:(("containerd-shim",pid=6862,fd=10))
u_str ESTAB 0      0                                                                                       * 32954             * 32955  users:(("containerd",pid=5008,fd=111))
u_str ESTAB 0      0                                                         /run/containerd/containerd.sock 22260             * 20330  users:(("containerd",pid=5008,fd=19))
u_str ESTAB 0      0      /run/containerd/s/323bfd1bd6ebd2ce9a3420b50eae4533fe80584ab525e264c058efda3c5be61a 23950             * 23947  users:(("containerd-shim",pid=6368,fd=10))
u_str ESTAB 0      0      /run/containerd/s/3c5471325c662956ea7099bf0f607aed6f94b4d5ed7de841d808da09647fd65f 21285             * 21284  users:(("containerd-shim",pid=5931,fd=12))
u_str ESTAB 0      0                                                                                       * 30051             * 30810  users:(("containerd",pid=5008,fd=95))
u_str ESTAB 0      0                                                                                       * 32924             * 30422  users:(("containerd-shim",pid=7290,fd=11))
u_str ESTAB 0      0                                                                                       * 23947             * 23950  users:(("containerd",pid=5008,fd=14))
u_str ESTAB 0      0                                                                                       * 23560             * 23561  users:(("containerd",pid=5008,fd=69))
u_str ESTAB 0      0      /run/containerd/s/59d7141628d54c0532476756e3589d8ed513fa390c889248db7d1474c5d10011 32854             * 30332  users:(("containerd-shim",pid=7280,fd=10))
u_str ESTAB 0      0      /run/containerd/s/6a1eec9193b24526a055dd3c67beb7a02bb755b4bdef487dfbb58ff75e2dc31c 30810             * 30051  users:(("containerd-shim",pid=6862,fd=12))
u_str ESTAB 0      0                                                         /run/containerd/containerd.sock 23882             * 20331  users:(("containerd",pid=5008,fd=20))
u_str ESTAB 0      0      /run/containerd/s/05d2b657bfce55010895d0fcbf5ef9c84cbcdb22d51de9420dc5d009153345e8 21612             * 19354  users:(("containerd-shim",pid=5952,fd=10))
u_str ESTAB 0      0      /run/containerd/s/59d7141628d54c0532476756e3589d8ed513fa390c889248db7d1474c5d10011 31941             * 31940  users:(("containerd-shim",pid=7280,fd=12))
u_str ESTAB 0      0                                                                                       * 21956             * 21957  users:(("containerd",pid=5008,fd=72))
u_str ESTAB 0      0      /run/containerd/s/6439866fe5db71cda35a979ec11ff91aebc5066eb838654ba8a3e502089642f8 21957             * 21956  users:(("containerd-shim",pid=6022,fd=12))
u_str ESTAB 0      0      /run/containerd/s/96e904766632d214a6e4f0a072e9cb3f650edbe5f67d9ab6d2a7a95a685680ae 31859             * 31857  users:(("containerd-shim",pid=7290,fd=10))
u_str ESTAB 0      0                                                                                       * 21915             * 21916  users:(("containerd-shim",pid=6022,fd=11))
u_str ESTAB 0      0                                                   /run/containerd/containerd.sock.ttrpc 21916             * 21915  users:(("containerd",pid=5008,fd=56))
u_str ESTAB 0      0                                                                                       * 31940             * 31941  users:(("containerd",pid=5008,fd=115))
u_str ESTAB 0      0                                                                                       * 31857             * 31859  users:(("containerd",pid=5008,fd=97))
u_str ESTAB 0      0                                                   /run/containerd/containerd.sock.ttrpc 31930             * 30421  users:(("containerd",pid=5008,fd=98))
u_str ESTAB 0      0                                                                                       * 30421             * 31930  users:(("containerd-shim",pid=7280,fd=11))


# 플러그인 확인
(⎈|HomeLab:N/A) root@k8s-ctr:~# ctr --address /run/containerd/containerd.sock version
Client:
  Version:  v2.1.5
  Revision: fcd43222d6b07379a4be9786bda52438f0dd16a1
  Go version: go1.24.9

Server:
  Version:  v2.1.5
  Revision: fcd43222d6b07379a4be9786bda52438f0dd16a1
  UUID: d37e58f0-2cca-4578-8f78-74c328cb60e5
(⎈|HomeLab:N/A) root@k8s-ctr:~# ctr plugins ls
TYPE                                      ID                       PLATFORMS         STATUS
io.containerd.content.v1                  content                  -                 ok
io.containerd.image-verifier.v1           bindir                   -                 ok
io.containerd.internal.v1                 opt                      -                 ok
io.containerd.warning.v1                  deprecations             -                 ok
io.containerd.snapshotter.v1              blockfile                linux/arm64/v8    skip
io.containerd.snapshotter.v1              devmapper                linux/arm64/v8    skip
io.containerd.snapshotter.v1              erofs                    linux/arm64/v8    skip
io.containerd.snapshotter.v1              native                   linux/arm64/v8    ok
io.containerd.snapshotter.v1              overlayfs                linux/arm64/v8    ok
io.containerd.snapshotter.v1              zfs                      linux/arm64/v8    skip
io.containerd.event.v1                    exchange                 -                 ok
io.containerd.monitor.task.v1             cgroups                  linux/arm64/v8    ok
io.containerd.metadata.v1                 bolt                     -                 ok
io.containerd.gc.v1                       scheduler                -                 ok
io.containerd.differ.v1                   erofs                    linux/arm64/v8    ok
io.containerd.differ.v1                   walking                  linux/arm64/v8    ok
io.containerd.lease.v1                    manager                  -                 ok
io.containerd.service.v1                  containers-service       -                 ok
io.containerd.service.v1                  content-service          -                 ok
io.containerd.service.v1                  diff-service             -                 ok
io.containerd.service.v1                  images-service           -                 ok
io.containerd.service.v1                  introspection-service    -                 ok
io.containerd.service.v1                  namespaces-service       -                 ok
io.containerd.service.v1                  snapshots-service        -                 ok
io.containerd.shim.v1                     manager                  -                 ok
io.containerd.runtime.v2                  task                     linux/arm64/v8    ok
io.containerd.service.v1                  tasks-service            -                 ok
io.containerd.grpc.v1                     containers               -                 ok
io.containerd.grpc.v1                     content                  -                 ok
io.containerd.grpc.v1                     diff                     -                 ok
io.containerd.grpc.v1                     events                   -                 ok
io.containerd.grpc.v1                     images                   -                 ok
io.containerd.grpc.v1                     introspection            -                 ok
io.containerd.grpc.v1                     leases                   -                 ok
io.containerd.grpc.v1                     namespaces               -                 ok
io.containerd.sandbox.store.v1            local                    -                 ok
io.containerd.transfer.v1                 local                    -                 ok
io.containerd.cri.v1                      images                   -                 ok
io.containerd.cri.v1                      runtime                  linux/arm64/v8    ok
io.containerd.podsandbox.controller.v1    podsandbox               -                 ok
io.containerd.sandbox.controller.v1       shim                     -                 ok
io.containerd.grpc.v1                     sandbox-controllers      -                 ok
io.containerd.grpc.v1                     sandboxes                -                 ok
io.containerd.grpc.v1                     snapshots                -                 ok
io.containerd.streaming.v1                manager                  -                 ok
io.containerd.grpc.v1                     streaming                -                 ok
io.containerd.grpc.v1                     tasks                    -                 ok
io.containerd.grpc.v1                     transfer                 -                 ok
io.containerd.grpc.v1                     version                  -                 ok
io.containerd.monitor.container.v1        restart                  -                 ok
io.containerd.tracing.processor.v1        otlp                     -                 skip
io.containerd.internal.v1                 tracing                  -                 skip
io.containerd.ttrpc.v1                    otelttrpc                -                 ok
io.containerd.grpc.v1                     healthcheck              -                 ok
io.containerd.nri.v1                      nri                      -                 ok
io.containerd.grpc.v1                     cri                      -                 ok

3. [공통] kubeadm, kubelet 및 kubectl 설치

3-1. kubeadm, kubelet 및 kubectl 설치 v1.32.11

# repo 추가
## exclude=... : 실수로 dnf update 시 kubelet 자동 업그레이드 방지
(⎈|HomeLab:N/A) root@k8s-ctr:~# dnf repolist
repo id                                                                                   repo name
appstream                                                                                 Rocky Linux 10 - AppStream
baseos                                                                                    Rocky Linux 10 - BaseOS
docker-ce-stable                                                                          Docker CE Stable - aarch64
extras                                                                                    Rocky Linux 10 - Extras
kubecolor                                                                                 packages for kubecolor
kubernetes                                                                                Kubernetes

(⎈|HomeLab:N/A) root@k8s-ctr:~# tree /etc/yum.repos.d/
/etc/yum.repos.d/
├── docker-ce.repo
├── kubecolor.repo
├── kubernetes.repo
├── rocky-addons.repo
├── rocky-devel.repo
├── rocky-extras.repo
└── rocky.repo

1 directory, 7 files

(⎈|HomeLab:N/A) root@k8s-ctr:~# cat <<EOF | tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.32/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.32/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.32/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.32/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni

(⎈|HomeLab:N/A) root@k8s-ctr:~# dnf makecache
Docker CE Stable - aarch64                                                                                                                                   394  B/s | 2.0 kB     00:05
packages for kubecolor                                                                                                                                       305  B/s | 1.5 kB     00:05
Kubernetes                                                                                                                                                   165  B/s | 1.7 kB     00:10
Rocky Linux 10 - BaseOS                                                                                                                                      412  B/s | 4.3 kB     00:10
Rocky Linux 10 - AppStream                                                                                                                                   371  B/s | 4.3 kB     00:12
Rocky Linux 10 - Extras                                                                                                                                      294  B/s | 3.1 kB     00:10

# 설치
## --disableexcludes=... kubernetes repo에 설정된 exclude 규칙을 이번 설치에서만 무시(1회성 옵션 처럼 사용)
## 설치 가능 버전 확인
(⎈|HomeLab:N/A) root@k8s-ctr:~# dnf list --showduplicates kubelet
Last metadata expiration check: 0:01:25 ago on Sat 24 Jan 2026 05:49:23 PM KST.
Installed Packages
kubelet.aarch64                                                                        1.32.11-150500.1.1                                                                         @kubernetes
(⎈|HomeLab:N/A) root@k8s-ctr:~# dnf list --showduplicates kubelet --disableexcludes=kubernetes
Last metadata expiration check: 0:01:28 ago on Sat 24 Jan 2026 05:49:23 PM KST.
Installed Packages
kubelet.aarch64                                                                        1.32.11-150500.1.1                                                                         @kubernetes
Available Packages
kubelet.aarch64                                                                        1.32.0-150500.1.1                                                                          kubernetes
kubelet.ppc64le                                                                        1.32.0-150500.1.1                                                                          kubernetes
...
(⎈|HomeLab:N/A) root@k8s-ctr:~# dnf list --showduplicates kubeadm --disableexcludes=kubernetes
(⎈|HomeLab:N/A) root@k8s-ctr:~# dnf list --showduplicates kubectl --disableexcludes=kubernetes


## 버전 정보 미지정 시, 제공 가능 최신 버전 설치됨.
(⎈|HomeLab:N/A) root@k8s-ctr:~# dnf install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
Last metadata expiration check: 0:03:51 ago on Sat 24 Jan 2026 05:49:23 PM KST.
Package kubelet-1.32.11-150500.1.1.aarch64 is already installed.
Package kubeadm-1.32.11-150500.1.1.aarch64 is already installed.
Package kubectl-1.32.11-150500.1.1.aarch64 is already installed.
Dependencies resolved.
Nothing to do.
Complete!


# kubelet 활성화 (실제 기동은 kubeadm init 후에 시작됨)
(⎈|HomeLab:N/A) root@k8s-ctr:~# systemctl enable --now kubelet
(⎈|HomeLab:N/A) root@k8s-ctr:~# ps -ef |grep kubelet
root        6124    5952  5 15:03 ?        00:08:47 kube-apiserver --advertise-address=192.168.10.100 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-account-signing-key-file=/etc/kubernetes/pki/sa.key --service-cluster-ip-range=10.96.0.0/16 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
root        6304       1  3 15:03 ?        00:05:54 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime-endpoint=unix:///run/containerd/containerd.sock --node-ip=192.168.10.100 --pod-infra-container-image=registry.k8s.io/pause:3.10
root       53912   42408  0 17:53 pts/2    00:00:00 grep --color=auto kubelet

# 설치 파일들 확인
(⎈|HomeLab:N/A) root@k8s-ctr:~# which kubeadm && kubeadm version -o yaml
/usr/bin/kubeadm
clientVersion:
  buildDate: "2025-12-16T18:06:36Z"
  compiler: gc
  gitCommit: 2195eae9e91f2e72114365d9bb9c670d0c08de12
  gitTreeState: clean
  gitVersion: v1.32.11
  goVersion: go1.24.11
  major: "1"
  minor: "32"
  platform: linux/arm64
(⎈|HomeLab:N/A) root@k8s-ctr:~# which kubectl && kubectl version --client=true
/usr/bin/kubectl
Client Version: v1.32.11
Kustomize Version: v5.5.0
(⎈|HomeLab:N/A) root@k8s-ctr:~# which kubectl && kubectl version --client=true
/usr/bin/kubectl
Client Version: v1.32.11
Kustomize Version: v5.5.0
(⎈|HomeLab:N/A) root@k8s-ctr:~# which kubelet && kubelet --version
/usr/bin/kubelet
Kubernetes v1.32.11
(⎈|HomeLab:N/A) root@k8s-ctr:~# which crictl && crictl version
/usr/bin/crictl
Version:  0.1.0
RuntimeName:  containerd
RuntimeVersion:  v2.1.5
RuntimeApiVersion:  v1

# /etc/crictl.yaml 파일 작성
(⎈|HomeLab:N/A) root@k8s-ctr:~# cat << EOF > /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
EOF


# kubernetes-cni : 파드 네트워크 구성을 위한 CNI 바이너리 파일 확인
(⎈|HomeLab:N/A) root@k8s-ctr:~# ls -al /opt/cni/bin
total 66036
drwxr-xr-x. 2 root root    4096 Jan 24 15:05 .
drwxr-xr-x. 3 root root      17 Jan 24 14:59 ..
-rwxr-xr-x. 1 root root 3239200 Dec 12  2024 bandwidth
-rwxr-xr-x. 1 root root 3731632 Dec 12  2024 bridge
-rwxr-xr-x. 1 root root 9123544 Dec 12  2024 dhcp
-rwxr-xr-x. 1 root root 3379872 Dec 12  2024 dummy
-rwxr-xr-x. 1 root root 3742888 Dec 12  2024 firewall
-rwxr-xr-x. 1 root root 2903098 Jan 24 15:05 flannel
-rwxr-xr-x. 1 root root 3383408 Dec 12  2024 host-device
-rwxr-xr-x. 1 root root 2812400 Dec 12  2024 host-local
-rwxr-xr-x. 1 root root 3380928 Dec 12  2024 ipvlan
-rw-r--r--. 1 root root   11357 Dec 12  2024 LICENSE
-rwxr-xr-x. 1 root root 2953200 Dec 12  2024 loopback
-rwxr-xr-x. 1 root root 3448024 Dec 12  2024 macvlan
-rwxr-xr-x. 1 root root 3312488 Dec 12  2024 portmap
-rwxr-xr-x. 1 root root 3524072 Dec 12  2024 ptp
-rw-r--r--. 1 root root    2343 Dec 12  2024 README.md
-rwxr-xr-x. 1 root root 3091976 Dec 12  2024 sbr
-rwxr-xr-x. 1 root root 2526944 Dec 12  2024 static
-rwxr-xr-x. 1 root root 3516272 Dec 12  2024 tap
-rwxr-xr-x. 1 root root 2956032 Dec 12  2024 tuning
-rwxr-xr-x. 1 root root 3380544 Dec 12  2024 vlan
-rwxr-xr-x. 1 root root 3160560 Dec 12  2024 vrf
(⎈|HomeLab:N/A) root@k8s-ctr:~# tree /opt/cni
/opt/cni
└── bin
    ├── bandwidth
    ├── bridge
    ├── dhcp
    ├── dummy
    ├── firewall
    ├── flannel
    ├── host-device
    ├── host-local
    ├── ipvlan
    ├── LICENSE
    ├── loopback
    ├── macvlan
    ├── portmap
    ├── ptp
    ├── README.md
    ├── sbr
    ├── static
    ├── tap
    ├── tuning
    ├── vlan
    └── vrf

2 directories, 21 files


(⎈|HomeLab:N/A) root@k8s-ctr:~# tree /etc/cni/
/etc/cni/
└── net.d
    └── 10-flannel.conflist

2 directories, 1 file


(⎈|HomeLab:N/A) root@k8s-ctr:~# systemctl is-active kubelet
active
(⎈|HomeLab:N/A) root@k8s-ctr:~# systemctl status kubelet --no-pager
● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/kubelet.service.d
             └─10-kubeadm.conf
     Active: active (running) since Sat 2026-01-24 15:03:08 KST; 2h 53min ago
 Invocation: 5d6b1a8fdda84197b96c991789afd05f
       Docs: https://kubernetes.io/docs/
   Main PID: 6304 (kubelet)
      Tasks: 13 (limit: 18742)
     Memory: 45.7M (peak: 50.1M)
        CPU: 6min 3.091s
     CGroup: /system.slice/kubelet.service
             └─6304 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container…



(⎈|HomeLab:N/A) root@k8s-ctr:~# cat /usr/lib/systemd/system/kubelet.service
(⎈|HomeLab:N/A) root@k8s-ctr:~# cat /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf

(⎈|HomeLab:N/A) root@k8s-ctr:~# tree /etc/kubernetes
/etc/kubernetes
├── admin.conf
├── controller-manager.conf
├── kubelet.conf
├── manifests
│   ├── etcd.yaml
│   ├── kube-apiserver.yaml
│   ├── kube-controller-manager.yaml
│   └── kube-scheduler.yaml
├── pki
│   ├── apiserver.crt
│   ├── apiserver-etcd-client.crt
│   ├── apiserver-etcd-client.key
│   ├── apiserver.key
│   ├── apiserver-kubelet-client.crt
│   ├── apiserver-kubelet-client.key
│   ├── ca.crt
│   ├── ca.key
│   ├── etcd
│   │   ├── ca.crt
│   │   ├── ca.key
│   │   ├── healthcheck-client.crt
│   │   ├── healthcheck-client.key
│   │   ├── peer.crt
│   │   ├── peer.key
│   │   ├── server.crt
│   │   └── server.key
│   ├── front-proxy-ca.crt
│   ├── front-proxy-ca.key
│   ├── front-proxy-client.crt
│   ├── front-proxy-client.key
│   ├── sa.key
│   └── sa.pub
├── scheduler.conf
└── super-admin.conf

4 directories, 31 files

# cgroup , namespace 정보 확인
(⎈|HomeLab:N/A) root@k8s-ctr:~# systemd-cgls --no-pager
(⎈|HomeLab:N/A) root@k8s-ctr:~# lsns

# containerd의 유닉스 도메인 소켓 확인 : kubelet에서 사용 , containerd client 3종(ctr, nerdctr, crictl)도 사용
(⎈|HomeLab:N/A) root@k8s-ctr:~# ls -l /run/containerd/containerd.sock
srw-rw----. 1 root root 0 Jan 24 14:58 /run/containerd/containerd.sock
(⎈|HomeLab:N/A) root@k8s-ctr:~# ss -xl | grep containerd
u_str LISTEN 0      4096                                                /run/containerd/containerd.sock.ttrpc 17656             * 0
u_str LISTEN 0      4096                                                      /run/containerd/containerd.sock 18544             * 0
u_str LISTEN 0      4096   /run/containerd/s/6a1eec9193b24526a055dd3c67beb7a02bb755b4bdef487dfbb58ff75e2dc31c 27398             * 0
u_str LISTEN 0      4096   /run/containerd/s/3c5471325c662956ea7099bf0f607aed6f94b4d5ed7de841d808da09647fd65f 21167             * 0
u_str LISTEN 0      4096   /run/containerd/s/323bfd1bd6ebd2ce9a3420b50eae4533fe80584ab525e264c058efda3c5be61a 23173             * 0
u_str LISTEN 0      4096   /run/containerd/s/05d2b657bfce55010895d0fcbf5ef9c84cbcdb22d51de9420dc5d009153345e8 21609             * 0
u_str LISTEN 0      4096   /run/containerd/s/59d7141628d54c0532476756e3589d8ed513fa390c889248db7d1474c5d10011 32850             * 0
u_str LISTEN 0      4096   /run/containerd/s/96e904766632d214a6e4f0a072e9cb3f650edbe5f67d9ab6d2a7a95a685680ae 31855             * 0
u_str LISTEN 0      4096   /run/containerd/s/6439866fe5db71cda35a979ec11ff91aebc5066eb838654ba8a3e502089642f8 21775             * 0
u_str LISTEN 0      4096   /run/containerd/s/97c57dfb95ff3f013f11b4dc2f2b0e00f04e2b0c6b93c7e67ed34b1cd962e667 21782             * 0
(⎈|HomeLab:N/A) root@k8s-ctr:~# ss -xnp | grep containerd
u_str ESTAB 0      0                                                                                       * 21789             * 21790  users:(("containerd",pid=5008,fd=43))
u_str ESTAB 0      0      /run/containerd/s/323bfd1bd6ebd2ce9a3420b50eae4533fe80584ab525e264c058efda3c5be61a 23262             * 23966  users:(("containerd-shim",pid=6368,fd=12))
u_str ESTAB 0      0                                                   /run/containerd/containerd.sock.ttrpc 20049             * 21731  users:(("containerd",pid=5008,fd=32))
u_str ESTAB 0      0                                                                                       * 21170             * 19349  users:(("containerd",pid=5008,fd=29))
u_str ESTAB 0      0                                                                                       * 19443             * 20065  users:(("containerd",pid=5008,fd=40))
u_str ESTAB 0      0                                                                                       * 23962             * 22331  users:(("containerd-shim",pid=6368,fd=11))
u_str ESTAB 0      0                                                   /run/containerd/containerd.sock.ttrpc 21728             * 21727  users:(("containerd",pid=5008,fd=31))
u_str ESTAB 0      0                                                                                       * 17083             * 18535  users:(("containerd",pid=5008,fd=2),("containerd",pid=5008,fd=1))
u_str ESTAB 0      0                                                   /run/containerd/containerd.sock.ttrpc 21910             * 22665  users:(("containerd",pid=5008,fd=55))
u_str ESTAB 0      0                                                                                       * 21727             * 21728  users:(("containerd-shim",pid=5931,fd=3))
u_str ESTAB 0      0                                                                                       * 22665             * 21910  users:(("containerd-shim",pid=6039,fd=11))
u_str ESTAB 0      0      /run/containerd/s/97c57dfb95ff3f013f11b4dc2f2b0e00f04e2b0c6b93c7e67ed34b1cd962e667 21790             * 21789  users:(("containerd-shim",pid=6039,fd=10))
u_str ESTAB 0      0                                                   /run/containerd/containerd.sock.ttrpc 22331             * 23962  users:(("containerd",pid=5008,fd=15))
u_str ESTAB 0      0      /run/containerd/s/6439866fe5db71cda35a979ec11ff91aebc5066eb838654ba8a3e502089642f8 20065             * 19443  users:(("containerd-shim",pid=6022,fd=10))
u_str ESTAB 0      0                                                                                       * 21731             * 20049  users:(("containerd-shim",pid=5952,fd=11))
u_str ESTAB 0      0      /run/containerd/s/3c5471325c662956ea7099bf0f607aed6f94b4d5ed7de841d808da09647fd65f 19349             * 21170  users:(("containerd-shim",pid=5931,fd=10))
u_str ESTAB 0      0                                                                                       * 23966             * 23262  users:(("containerd",pid=5008,fd=79))
u_str ESTAB 0      0                                                                                       * 28060             * 29048  users:(("containerd",pid=5008,fd=23))
u_str ESTAB 0      0                                                                                       * 21291             * 21292  users:(("containerd",pid=5008,fd=54))
u_str ESTAB 0      0                                                   /run/containerd/containerd.sock.ttrpc 28118             * 25493  users:(("containerd",pid=5008,fd=28))
u_str ESTAB 0      0                                                                                       * 19354             * 21612  users:(("containerd",pid=5008,fd=30))
u_str ESTAB 0      0      /run/containerd/s/96e904766632d214a6e4f0a072e9cb3f650edbe5f67d9ab6d2a7a95a685680ae 32955             * 32954  users:(("containerd-shim",pid=7290,fd=12))
u_str ESTAB 0      0                                                                                       * 21284             * 21285  users:(("containerd",pid=5008,fd=51))
u_str ESTAB 0      0                                                                                       * 25493             * 28118  users:(("containerd-shim",pid=6862,fd=11))
u_str ESTAB 0      0      /run/containerd/s/05d2b657bfce55010895d0fcbf5ef9c84cbcdb22d51de9420dc5d009153345e8 21292             * 21291  users:(("containerd-shim",pid=5952,fd=3))
u_str ESTAB 0      0                                                   /run/containerd/containerd.sock.ttrpc 30422             * 32924  users:(("containerd",pid=5008,fd=99))
u_str ESTAB 0      0                                                                                       * 30332             * 32854  users:(("containerd",pid=5008,fd=96))
u_str ESTAB 0      0      /run/containerd/s/97c57dfb95ff3f013f11b4dc2f2b0e00f04e2b0c6b93c7e67ed34b1cd962e667 23561             * 23560  users:(("containerd-shim",pid=6039,fd=12))
u_str ESTAB 0      0      /run/containerd/s/6a1eec9193b24526a055dd3c67beb7a02bb755b4bdef487dfbb58ff75e2dc31c 29048             * 28060  users:(("containerd-shim",pid=6862,fd=10))
u_str ESTAB 0      0                                                                                       * 32954             * 32955  users:(("containerd",pid=5008,fd=111))
u_str ESTAB 0      0                                                         /run/containerd/containerd.sock 22260             * 20330  users:(("containerd",pid=5008,fd=19))
u_str ESTAB 0      0      /run/containerd/s/323bfd1bd6ebd2ce9a3420b50eae4533fe80584ab525e264c058efda3c5be61a 23950             * 23947  users:(("containerd-shim",pid=6368,fd=10))
u_str ESTAB 0      0      /run/containerd/s/3c5471325c662956ea7099bf0f607aed6f94b4d5ed7de841d808da09647fd65f 21285             * 21284  users:(("containerd-shim",pid=5931,fd=12))
u_str ESTAB 0      0                                                                                       * 30051             * 30810  users:(("containerd",pid=5008,fd=95))
u_str ESTAB 0      0                                                                                       * 32924             * 30422  users:(("containerd-shim",pid=7290,fd=11))
u_str ESTAB 0      0                                                                                       * 23947             * 23950  users:(("containerd",pid=5008,fd=14))
u_str ESTAB 0      0                                                                                       * 23560             * 23561  users:(("containerd",pid=5008,fd=69))
u_str ESTAB 0      0      /run/containerd/s/59d7141628d54c0532476756e3589d8ed513fa390c889248db7d1474c5d10011 32854             * 30332  users:(("containerd-shim",pid=7280,fd=10))
u_str ESTAB 0      0      /run/containerd/s/6a1eec9193b24526a055dd3c67beb7a02bb755b4bdef487dfbb58ff75e2dc31c 30810             * 30051  users:(("containerd-shim",pid=6862,fd=12))
u_str ESTAB 0      0                                                         /run/containerd/containerd.sock 23882             * 20331  users:(("containerd",pid=5008,fd=20))
u_str ESTAB 0      0      /run/containerd/s/05d2b657bfce55010895d0fcbf5ef9c84cbcdb22d51de9420dc5d009153345e8 21612             * 19354  users:(("containerd-shim",pid=5952,fd=10))
u_str ESTAB 0      0      /run/containerd/s/59d7141628d54c0532476756e3589d8ed513fa390c889248db7d1474c5d10011 31941             * 31940  users:(("containerd-shim",pid=7280,fd=12))
u_str ESTAB 0      0                                                                                       * 21956             * 21957  users:(("containerd",pid=5008,fd=72))
u_str ESTAB 0      0      /run/containerd/s/6439866fe5db71cda35a979ec11ff91aebc5066eb838654ba8a3e502089642f8 21957             * 21956  users:(("containerd-shim",pid=6022,fd=12))
u_str ESTAB 0      0      /run/containerd/s/96e904766632d214a6e4f0a072e9cb3f650edbe5f67d9ab6d2a7a95a685680ae 31859             * 31857  users:(("containerd-shim",pid=7290,fd=10))
u_str ESTAB 0      0                                                                                       * 21915             * 21916  users:(("containerd-shim",pid=6022,fd=11))
u_str ESTAB 0      0                                                   /run/containerd/containerd.sock.ttrpc 21916             * 21915  users:(("containerd",pid=5008,fd=56))
u_str ESTAB 0      0                                                                                       * 31940             * 31941  users:(("containerd",pid=5008,fd=115))
u_str ESTAB 0      0                                                                                       * 31857             * 31859  users:(("containerd",pid=5008,fd=97))
u_str ESTAB 0      0                                                   /run/containerd/containerd.sock.ttrpc 31930             * 30421  users:(("containerd",pid=5008,fd=98))
u_str ESTAB 0      0                                                                                       * 30421             * 31930  users:(("containerd-shim",pid=7280,fd=11))

4. [Controlplane node] kubeadm 으로 k8s 클러스터 구성 → Flannel CNI 설치 → 편의성 설치 및 확인

4-1. kubeadm init 수행

https://kubernetes.io/docs/reference/setup-tools/kubeadm/implementation-details/

Implementation details

FEATURE STATE: Kubernetes v1.10 [stable] kubeadm init and kubeadm join together provide a nice user experience for creating a bare Kubernetes cluster from scratch, that aligns with the best-practices. However, it might not be obvious how kubeadm does that.

kubernetes.io

# 기본 환경 정보 출력 저장
(⎈|HomeLab:N/A) root@k8s-ctr:~# crictl images
IMAGE                                     TAG                 IMAGE ID            SIZE
ghcr.io/flannel-io/flannel-cni-plugin     v1.7.1-flannel1     127562bd9047f       5.14MB
ghcr.io/flannel-io/flannel                v0.27.3             d84558c0144bc       33.1MB
registry.k8s.io/coredns/coredns           v1.11.3             2f6c962e7b831       16.9MB
registry.k8s.io/etcd                      3.5.24-0            1211402d28f58       21.9MB
registry.k8s.io/kube-apiserver            v1.32.11            58951ea1a0b5d       26.4MB
registry.k8s.io/kube-controller-manager   v1.32.11            82766e5f2d560       24.2MB
registry.k8s.io/kube-proxy                v1.32.11            dcdb790dc2bfe       27.6MB
registry.k8s.io/kube-scheduler            v1.32.11            cfa17ff3d6634       19.2MB
registry.k8s.io/pause                     3.10                afb61768ce381       268kB
(⎈|HomeLab:N/A) root@k8s-ctr:~# crictl ps
CONTAINER           IMAGE               CREATED             STATE               NAME                      ATTEMPT             POD ID              POD                               NAMESPACE
2e4e767a2f4d6       2f6c962e7b831       3 hours ago         Running             coredns                   0                   cad15631e8443       coredns-668d6bf9bc-smll6          kube-system
70ec7b4c12bde       2f6c962e7b831       3 hours ago         Running             coredns                   0                   188e8bdda1af3       coredns-668d6bf9bc-hk2cc          kube-system
603ffaa0d45b7       d84558c0144bc       3 hours ago         Running             kube-flannel              0                   fc0ef56b33f82       kube-flannel-ds-xlgfs             kube-flannel
6d44b873d3cfd       dcdb790dc2bfe       3 hours ago         Running             kube-proxy                0                   4b69ed2f8eeec       kube-proxy-jn9xt                  kube-system
587e6a63c5b4b       82766e5f2d560       3 hours ago         Running             kube-controller-manager   0                   7f865707e4b03       kube-controller-manager-k8s-ctr   kube-system
ab4f5f0704d1e       cfa17ff3d6634       3 hours ago         Running             kube-scheduler            0                   aa47ccd8f7693       kube-scheduler-k8s-ctr            kube-system
d2b3157d33f9d       58951ea1a0b5d       3 hours ago         Running             kube-apiserver            0                   c3a07a12edde4       kube-apiserver-k8s-ctr            kube-system
1a37a2f0d42f1       1211402d28f58       3 hours ago         Running             etcd                      0                   c91cc3779b6d5       etcd-k8s-ctr                      kube-system

(⎈|HomeLab:N/A) root@k8s-ctr:~# cat /etc/sysconfig/kubelet
(⎈|HomeLab:N/A) root@k8s-ctr:~# tree /etc/kubernetes  | tee -a etc_kubernetes-1.txt
(⎈|HomeLab:N/A) root@k8s-ctr:~# tree /var/lib/kubelet | tee -a var_lib_kubelet-1.txt
(⎈|HomeLab:N/A) root@k8s-ctr:~# tree /run/containerd/ -L 3 | tee -a run_containerd-1.txt
(⎈|HomeLab:N/A) root@k8s-ctr:~# pstree -alnp | tee -a pstree-1.txt
(⎈|HomeLab:N/A) root@k8s-ctr:~# systemd-cgls --no-pager | tee -a systemd-cgls-1.txt
(⎈|HomeLab:N/A) root@k8s-ctr:~# lsns | tee -a lsns-1.txt
(⎈|HomeLab:N/A) root@k8s-ctr:~# ip addr | tee -a ip_addr-1.txt 
(⎈|HomeLab:N/A) root@k8s-ctr:~# ss -tnlp | tee -a ss-1.txt
(⎈|HomeLab:N/A) root@k8s-ctr:~# df -hT | tee -a df-1.txt
(⎈|HomeLab:N/A) root@k8s-ctr:~# findmnt | tee -a findmnt-1.txt
(⎈|HomeLab:N/A) root@k8s-ctr:~# sysctl -a | tee -a sysctl-1.txt


# kubeadm Configuration 파일 작성
(⎈|HomeLab:N/A) root@k8s-ctr:~# cat << EOF > kubeadm-init.yaml
apiVersion: kubeadm.k8s.io/v1beta4
kind: InitConfiguration
bootstrapTokens:
- token: "123456.1234567890123456"
  ttl: "0s"
  usages:
  - signing
  - authentication
nodeRegistration:
  kubeletExtraArgs:
    - name: node-ip
      value: "192.168.10.100"  # 미설정 시 10.0.2.15 맵핑
  criSocket: "unix:///run/containerd/containerd.sock"
localAPIEndpoint:
  advertiseAddress: "192.168.10.100"
---
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: "1.32.11"
networking:
  podSubnet: "10.244.0.0/16"
  serviceSubnet: "10.96.0.0/16"
EOF

# (옵션) 컨테이너 이미지 미리 다운로드 : 특히 업그레이드 작업 시, 작업 시간 단축을 위해서 수행할 것
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubeadm config images pull
W0124 18:17:42.180559   61813 version.go:109] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get "https://cdn.dl.k8s.io/release/stable-1.txt": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
W0124 18:17:42.180725   61813 version.go:110] falling back to the local client version: v1.32.11
[config/images] Pulled registry.k8s.io/kube-apiserver:v1.32.11
[config/images] Pulled registry.k8s.io/kube-controller-manager:v1.32.11
[config/images] Pulled registry.k8s.io/kube-scheduler:v1.32.11
[config/images] Pulled registry.k8s.io/kube-proxy:v1.32.11
[config/images] Pulled registry.k8s.io/coredns/coredns:v1.11.3
[config/images] Pulled registry.k8s.io/pause:3.10
[config/images] Pulled registry.k8s.io/etcd:3.5.24-0

# k8s controlplane 초기화 설정 수행
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubeadm init --config="kubeadm-init.yaml"
[init] Using Kubernetes version: v1.32.11
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action beforehand using 'kubeadm config images pull'
W0124 18:56:07.483873   72139 checks.go:843] detected that the sandbox image "" of the container runtime is inconsistent with that used by kubeadm.It is recommended to use "registry.k8s.io/pause:3.10" as the CRI sandbox image.
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-ctr kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.10.100]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-ctr localhost] and IPs [192.168.10.100 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-ctr localhost] and IPs [192.168.10.100 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "super-admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests"
[kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is healthy after 502.219864ms
[api-check] Waiting for a healthy API server. This can take up to 4m0s
[api-check] The API server is healthy after 4.002549553s
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s-ctr as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node k8s-ctr as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: 123456.1234567890123456
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.10.100:6443 --token 123456.1234567890123456 \
	--discovery-token-ca-cert-hash sha256:ab5c260aff4f6a47ac284ea1576fa616c6b4fc29a048e0c34f93969b16bfb221
    
    
# crictl 확인
(⎈|HomeLab:N/A) root@k8s-ctr:~# crictl images
IMAGE                                     TAG                 IMAGE ID            SIZE
ghcr.io/flannel-io/flannel-cni-plugin     v1.7.1-flannel1     127562bd9047f       5.14MB
ghcr.io/flannel-io/flannel                v0.27.3             d84558c0144bc       33.1MB
registry.k8s.io/coredns/coredns           v1.11.3             2f6c962e7b831       16.9MB
registry.k8s.io/etcd                      3.5.24-0            1211402d28f58       21.9MB
registry.k8s.io/kube-apiserver            v1.32.11            58951ea1a0b5d       26.4MB
registry.k8s.io/kube-controller-manager   v1.32.11            82766e5f2d560       24.2MB
registry.k8s.io/kube-proxy                v1.32.11            dcdb790dc2bfe       27.6MB
registry.k8s.io/kube-scheduler            v1.32.11            cfa17ff3d6634       19.2MB
registry.k8s.io/pause                     3.10                afb61768ce381       268kB

(⎈|HomeLab:N/A) root@k8s-ctr:~# crictl ps
CONTAINER           IMAGE               CREATED             STATE               NAME                      ATTEMPT             POD ID              POD                               NAMESPACE
d97aba979ea45       2f6c962e7b831       43 seconds ago      Running             coredns                   0                   71e8a6f3b251d       coredns-668d6bf9bc-qdwrj          kube-system
9d0e7906023a1       2f6c962e7b831       43 seconds ago      Running             coredns                   0                   12705c71a3d83       coredns-668d6bf9bc-c2g8k          kube-system
d1de205be7f67       dcdb790dc2bfe       43 seconds ago      Running             kube-proxy                0                   f0c4ffcef3bbb       kube-proxy-6gfjf                  kube-system
12407e81bc3b3       cfa17ff3d6634       54 seconds ago      Running             kube-scheduler            0                   9b742330f7f07       kube-scheduler-k8s-ctr            kube-system
374b448fdda3c       1211402d28f58       54 seconds ago      Running             etcd                      0                   b354af512cfe1       etcd-k8s-ctr                      kube-system
852224a5098d3       58951ea1a0b5d       54 seconds ago      Running             kube-apiserver            1                   f9c341e85832b       kube-apiserver-k8s-ctr            kube-system
118aeaecb21fc       82766e5f2d560       54 seconds ago      Running             kube-controller-manager   0                   dd4d7205db8c6       kube-controller-manager-k8s-ctr   kube-system


# kubeconfig 작성
(⎈|HomeLab:N/A) root@k8s-ctr:~# mkdir -p /root/.kube
(⎈|HomeLab:N/A) root@k8s-ctr:~# cp -i /etc/kubernetes/admin.conf /root/.kube/config
cp: overwrite '/root/.kube/config'?
(⎈|HomeLab:N/A) root@k8s-ctr:~# chown $(id -u):$(id -g) /root/.kube/config

# 확인
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kubectl cluster-info
Kubernetes control plane is running at https://192.168.10.100:6443
CoreDNS is running at https://192.168.10.100:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kubectl get node -owide
NAME      STATUS   ROLES           AGE    VERSION    INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                        KERNEL-VERSION                  CONTAINER-RUNTIME
k8s-ctr   Ready    control-plane   4m6s   v1.32.11   192.168.10.100   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.aarch64   containerd://2.1.5

# coredns 의 service name 확인 : kube-dns
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kubectl get svc -n kube-system
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   4m43s

# cluster-info ConfigMap 공개 : cluster-info는 '신원 확인 전, 최소한의 신뢰 부트스트랩 데이터'
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kubectl -n kube-public get configmap cluster-info
NAME           DATA   AGE
cluster-info   2      4m56s
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kubectl -n kube-public get configmap cluster-info -o yaml
apiVersion: v1
data:
  jws-kubeconfig-123456: eyJhbGciOiJIUzI1NiIsImtpZCI6IjEyMzQ1NiJ9..K4CA_0Z5R0GWkao4kj4kgPHTmKl1f-G5GQQUDTX2Lek
  kubeconfig: |
    apiVersion: v1
    clusters:
    - cluster:
        certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURCVENDQWUyZ0F3SUJBZ0lJSXk0ODJreHY4b013RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TmpBeE1qUXdPVFV4TURkYUZ3MHpOakF4TWpJd09UVTJNRGRhTUJVeApFekFSQmdOVkJBTVRDbXQxWW1WeWJtVjBaWE13Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLCkFvSUJBUURZWkxueHAxUWFvMWNmOWVBNlZkalZwb2RFenpwMzk4d2QxWFBvd1VjQm1idzY1RFlWR0c5c1VlcVYKL042elFSREZJMmJQeUVTRDVJM092U0tvVGRWNS9rZ1RSaWFyTFlhYS84WENhTzM5Q05PamlmMjdaQVJYa2xjcwo5czJiODNOU1JycDJiNmJhNitHbkYwbUM1UXlPbE1uZVN6Uys3OFV2L3Jpdk9nQ2tkdzBCSUQvblRoTjdhR3lhCjQvQ1RJaVZCMkVoQXllT2FWSEVER3hLMk1ZNXhWL2lxVjBoVFh3V3B1NnhiZ1UxdlFlbml4dkIweVM3MjgyblEKajEydUpjdTFiUkcwWStDbFMzb1owKzBYRGY5dFNnNU54RTR6aWh2NjFpQVl3MnBTRzNJNU5CYVZXb0NtbkJVeAprbkJBbTRaZXJvNDRTVHZLU0pPMXd5ckQzcnluQWdNQkFBR2pXVEJYTUE0R0ExVWREd0VCL3dRRUF3SUNwREFQCkJnTlZIUk1CQWY4RUJUQURBUUgvTUIwR0ExVWREZ1FXQkJRYnhDdm8xL1k4UW5Bc3VhNG9aelVObjBGb3l6QVYKQmdOVkhSRUVEakFNZ2dwcmRXSmxjbTVsZEdWek1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQ2JkTUkwUGFyYgowbTZHdDRaNFJNRmtxR3d2T0VNdTlYZVRlTVVLanpXSklRVEUxSlpDajFGZnlHbWhBZ3NQaERQcVBabDZyMmFvCmlrOVdhbm12dmxkM3RGWElqbnRPM0ZyVmtOTGREKzYwVlVkMVNuL2ZvSk16VmJLNzlqNWlUUWs1aWJSNnRraXoKSFZnTktvTHlaYjk4SXpFOEdEcVQ2YnV2ZG9UM2xSMW5VWlNNVHJFazZLdXA4MDJjNGN3dUxHT0MrR2I1WUg2bwp0Z3d5d3gxY2hMcGx4VkJkSlBtSld2NHFxNm0vYXZlUmRLNm9jdXIzVWNraGJhYUVweWhwUmRDMDYzWXE2YjdCCnhvV1kvdGRRQzhMNEQ1SlVrSWx6TW9Cdzd5emJrVTJINER6QVYzMHBlakpSdVVSSStxQWFxSERvalBWenNzSG4KY3ByMGFTTlVIWGNnCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
        server: https://192.168.10.100:6443
      name: ""
    contexts: null
    current-context: ""
    kind: Config
    preferences: {}
    users: null
kind: ConfigMap
metadata:
  creationTimestamp: "2026-01-24T09:56:15Z"
  name: cluster-info
  namespace: kube-public
  resourceVersion: "300"
  uid: 5c22db5f-a97c-4a31-a4c3-643b4fd74f74
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kubectl -n kube-public get configmap cluster-info -o jsonpath='{.data.kubeconfig}' | grep certificate-authority-data | cut -d ':' -f2 | tr -d ' ' | base64 -d | openssl x509 -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 2535030548539110019 (0x232e3cda4c6ff283)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kubernetes
        Validity
            Not Before: Jan 24 09:51:07 2026 GMT
            Not After : Jan 22 09:56:07 2036 GMT
        Subject: CN=kubernetes
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:d8:64:b9:f1:a7:54:1a:a3:57:1f:f5:e0:3a:55:
                    d8:d5:a6:87:44:cf:3a:77:f7:cc:1d:d5:73:e8:c1:
                    47:01:99:bc:3a:e4:36:15:18:6f:6c:51:ea:95:fc:
                    de:b3:41:10:c5:23:66:cf:c8:44:83:e4:8d:ce:bd:
                    22:a8:4d:d5:79:fe:48:13:46:26:ab:2d:86:9a:ff:
                    c5:c2:68:ed:fd:08:d3:a3:89:fd:bb:64:04:57:92:
                    57:2c:f6:cd:9b:f3:73:52:46:ba:76:6f:a6:da:eb:
                    e1:a7:17:49:82:e5:0c:8e:94:c9:de:4b:34:be:ef:
                    c5:2f:fe:b8:af:3a:00:a4:77:0d:01:20:3f:e7:4e:
                    13:7b:68:6c:9a:e3:f0:93:22:25:41:d8:48:40:c9:
                    e3:9a:54:71:03:1b:12:b6:31:8e:71:57:f8:aa:57:
                    48:53:5f:05:a9:bb:ac:5b:81:4d:6f:41:e9:e2:c6:
                    f0:74:c9:2e:f6:f3:69:d0:8f:5d:ae:25:cb:b5:6d:
                    11:b4:63:e0:a5:4b:7a:19:d3:ed:17:0d:ff:6d:4a:
                    0e:4d:c4:4e:33:8a:1b:fa:d6:20:18:c3:6a:52:1b:
                    72:39:34:16:95:5a:80:a6:9c:15:31:92:70:40:9b:
                    86:5e:ae:8e:38:49:3b:ca:48:93:b5:c3:2a:c3:de:
                    bc:a7
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment, Certificate Sign
            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Subject Key Identifier:
                1B:C4:2B:E8:D7:F6:3C:42:70:2C:B9:AE:28:67:35:0D:9F:41:68:CB
            X509v3 Subject Alternative Name:
                DNS:kubernetes
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        9b:74:c2:34:3d:aa:db:d2:6e:86:b7:86:78:44:c1:64:a8:6c:
        2f:38:43:2e:f5:77:93:78:c5:0a:8f:35:89:21:04:c4:d4:96:
        42:8f:51:5f:c8:69:a1:02:0b:0f:84:33:ea:3d:99:7a:af:66:
        a8:8a:4f:56:6a:79:af:be:57:77:b4:55:c8:8e:7b:4e:dc:5a:
        d5:90:d2:dd:0f:ee:b4:55:47:75:4a:7f:df:a0:93:33:55:b2:
        bb:f6:3e:62:4d:09:39:89:b4:7a:b6:48:b3:1d:58:0d:2a:82:
        f2:65:bf:7c:23:31:3c:18:3a:93:e9:bb:af:76:84:f7:95:1d:
        67:51:94:8c:4e:b1:24:e8:ab:a9:f3:4d:9c:e1:cc:2e:2c:63:
        82:f8:66:f9:60:7e:a8:b6:0c:32:c3:1d:5c:84:ba:65:c5:50:
        5d:24:f9:89:5a:fe:2a:ab:a9:bf:6a:f7:91:74:ae:a8:72:ea:
        f7:51:c9:21:6d:a6:84:a7:28:69:45:d0:b4:eb:76:2a:e9:be:
        c1:c6:85:98:fe:d7:50:0b:c2:f8:0f:92:54:90:89:73:32:80:
        70:ef:2c:db:91:4d:87:e0:3c:c0:57:7d:29:7a:32:51:b9:44:
        48:fa:a0:1a:a8:70:e8:8c:f5:73:b2:c1:e7:72:9a:f4:69:23:
        54:1d:77:20


(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# curl -s -k https://192.168.10.100:6443/api/v1/namespaces/kube-public/configmaps/cluster-info | jq
{
  "kind": "ConfigMap",
  "apiVersion": "v1",
  "metadata": {
    "name": "cluster-info",
    "namespace": "kube-public",
    "uid": "5c22db5f-a97c-4a31-a4c3-643b4fd74f74",
    "resourceVersion": "300",
    "creationTimestamp": "2026-01-24T09:56:15Z",
    "managedFields": [
      {
        "manager": "kubeadm",
        "operation": "Update",
        "apiVersion": "v1",
        "time": "2026-01-24T09:56:15Z",
        "fieldsType": "FieldsV1",
        "fieldsV1": {
          "f:data": {
            ".": {},
            "f:kubeconfig": {}
          }
        }
      },
      {
        "manager": "kube-controller-manager",
        "operation": "Update",
        "apiVersion": "v1",
        "time": "2026-01-24T09:56:21Z",
        "fieldsType": "FieldsV1",
        "fieldsV1": {
          "f:data": {
            "f:jws-kubeconfig-123456": {}
          }
        }
      }
    ]
  },
  "data": {
    "jws-kubeconfig-123456": "eyJhbGciOiJIUzI1NiIsImtpZCI6IjEyMzQ1NiJ9..K4CA_0Z5R0GWkao4kj4kgPHTmKl1f-G5GQQUDTX2Lek",
    "kubeconfig": "apiVersion: v1\nclusters:\n- cluster:\n    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURCVENDQWUyZ0F3SUJBZ0lJSXk0ODJreHY4b013RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TmpBeE1qUXdPVFV4TURkYUZ3MHpOakF4TWpJd09UVTJNRGRhTUJVeApFekFSQmdOVkJBTVRDbXQxWW1WeWJtVjBaWE13Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLCkFvSUJBUURZWkxueHAxUWFvMWNmOWVBNlZkalZwb2RFenpwMzk4d2QxWFBvd1VjQm1idzY1RFlWR0c5c1VlcVYKL042elFSREZJMmJQeUVTRDVJM092U0tvVGRWNS9rZ1RSaWFyTFlhYS84WENhTzM5Q05PamlmMjdaQVJYa2xjcwo5czJiODNOU1JycDJiNmJhNitHbkYwbUM1UXlPbE1uZVN6Uys3OFV2L3Jpdk9nQ2tkdzBCSUQvblRoTjdhR3lhCjQvQ1RJaVZCMkVoQXllT2FWSEVER3hLMk1ZNXhWL2lxVjBoVFh3V3B1NnhiZ1UxdlFlbml4dkIweVM3MjgyblEKajEydUpjdTFiUkcwWStDbFMzb1owKzBYRGY5dFNnNU54RTR6aWh2NjFpQVl3MnBTRzNJNU5CYVZXb0NtbkJVeAprbkJBbTRaZXJvNDRTVHZLU0pPMXd5ckQzcnluQWdNQkFBR2pXVEJYTUE0R0ExVWREd0VCL3dRRUF3SUNwREFQCkJnTlZIUk1CQWY4RUJUQURBUUgvTUIwR0ExVWREZ1FXQkJRYnhDdm8xL1k4UW5Bc3VhNG9aelVObjBGb3l6QVYKQmdOVkhSRUVEakFNZ2dwcmRXSmxjbTVsZEdWek1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQ2JkTUkwUGFyYgowbTZHdDRaNFJNRmtxR3d2T0VNdTlYZVRlTVVLanpXSklRVEUxSlpDajFGZnlHbWhBZ3NQaERQcVBabDZyMmFvCmlrOVdhbm12dmxkM3RGWElqbnRPM0ZyVmtOTGREKzYwVlVkMVNuL2ZvSk16VmJLNzlqNWlUUWs1aWJSNnRraXoKSFZnTktvTHlaYjk4SXpFOEdEcVQ2YnV2ZG9UM2xSMW5VWlNNVHJFazZLdXA4MDJjNGN3dUxHT0MrR2I1WUg2bwp0Z3d5d3gxY2hMcGx4VkJkSlBtSld2NHFxNm0vYXZlUmRLNm9jdXIzVWNraGJhYUVweWhwUmRDMDYzWXE2YjdCCnhvV1kvdGRRQzhMNEQ1SlVrSWx6TW9Cdzd5emJrVTJINER6QVYzMHBlakpSdVVSSStxQWFxSERvalBWenNzSG4KY3ByMGFTTlVIWGNnCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K\n    server: https://192.168.10.100:6443\n  name: \"\"\ncontexts: null\ncurrent-context: \"\"\nkind: Config\npreferences: {}\nusers: null\n"
  }
}


# kubeadm init 시 생성되는 객체
- Namespace: kube-public
- ConfigMap: cluster-info
- Role + RoleBinding 
>> 대상: system:unauthenticated (인증 안 된 사용자)
>> 권한: get on configmaps/cluster-info
  아직 클러스터 인증서가 없는 노드(worker) 가 (kubeadm join 전) API Server에 처음 접속해서 최소 정보(엔드포인트 + CA)를 얻기 위해 필요

4-2. [k8s-ctr] k8s 관련 작업 편의성 설정 : 자동 완성, kubecolor, kubectx, kubens, kube-ps, helm, k9s 설치

# k8s 관련 작업 편의성 설정
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# echo "sudo su -" >> /home/vagrant/.bashrc
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# source <(kubectl completion bash)
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# source <(kubeadm completion bash)
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# alias k=kubectl
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# complete -o default -F __start_kubectl k
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# echo 'alias k=kubectl' >> /etc/profile
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# echo 'complete -o default -F __start_kubectl k' >> /etc/profile

(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# dnf install -y 'dnf-command(config-manager)'
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# dnf config-manager --add-repo https://kubecolor.github.io/packages/rpm/kubecolor.repo
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# dnf repolist
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# dnf install -y kubecolor

(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# dnf install -y git
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# git clone https://github.com/ahmetb/kubectx /opt/kubectx
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# ln -s /opt/kubectx/kubens /usr/local/bin/kubens
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# ln -s /opt/kubectx/kubectx /usr/local/bin/kubectx

(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# git clone https://github.com/jonmosco/kube-ps1.git /root/kube-ps1
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# cat << "EOT" >> /root/.bash_profile
source /root/kube-ps1/kube-ps1.sh
KUBE_PS1_SYMBOL_ENABLE=true
function get_cluster_short() {
  echo "$1" | cut -d . -f1
}
KUBE_PS1_CLUSTER_FUNCTION=get_cluster_short
KUBE_PS1_SUFFIX=') '
PS1='$(kube_ps1)'$PS1
EOT

4-3. [k8s-ctr] Flannel CNI 설치 v0.27.3

# 현재 k8s 클러스터에 파드 전체 CIDR 확인
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kc describe pod -n kube-system kube-controller-manager-k8s-ctr
Name:                 kube-controller-manager-k8s-ctr
Namespace:            kube-system
Priority:             2000001000
Priority Class Name:  system-node-critical
Node:                 k8s-ctr/192.168.10.100
Start Time:           Sat, 24 Jan 2026 18:56:16 +0900
Labels:               component=kube-controller-manager
                      tier=control-plane
Annotations:          kubernetes.io/config.hash: 7314ab3f0ec6401c196ca943fad44a05
                      kubernetes.io/config.mirror: 7314ab3f0ec6401c196ca943fad44a05
                      kubernetes.io/config.seen: 2026-01-24T18:56:16.097383940+09:00
                      kubernetes.io/config.source: file
Status:               Running
SeccompProfile:       RuntimeDefault
IP:                   192.168.10.100
IPs:
  IP:           192.168.10.100
Controlled By:  Node/k8s-ctr
Containers:
  kube-controller-manager:
    Container ID:  containerd://118aeaecb21fc60deeb9956f919b4b6c8be283fb50b61ad2f6b4402c11101960
    Image:         registry.k8s.io/kube-controller-manager:v1.32.11
    Image ID:      registry.k8s.io/kube-controller-manager@sha256:ce7b2ead5eef1a1554ef28b2b79596c6a8c6d506a87a7ab1381e77fe3d72f55f
    Port:          <none>
    Host Port:     <none>
    Command:
      kube-controller-manager
      --allocate-node-cidrs=true
      --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
      --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf
      --bind-address=127.0.0.1
      --client-ca-file=/etc/kubernetes/pki/ca.crt
      --cluster-cidr=10.244.0.0/16
      --cluster-name=kubernetes
      --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
      --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
      --controllers=*,bootstrapsigner,tokencleaner
      --kubeconfig=/etc/kubernetes/controller-manager.conf
      --leader-elect=true
      --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
      --root-ca-file=/etc/kubernetes/pki/ca.crt
      --service-account-private-key-file=/etc/kubernetes/pki/sa.key
      --service-cluster-ip-range=10.96.0.0/16
      --use-service-account-credentials=true
     
     
# 노드별 파드 CIDR 확인 
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.podCIDR}{"\n"}{end}'
k8s-ctr	10.244.0.0/24

# Deploying Flannel with Helm
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# helm repo add flannel https://flannel-io.github.io/flannel
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "metrics-server" chart repository
...Successfully got an update from the "flannel" chart repository
Update Complete. ⎈Happy Helming!⎈
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kubectl create namespace kube-flannel
cat << EOF > flannel.yaml
podCidr: "10.244.0.0/16"
flannel:
  cniBinDir: "/opt/cni/bin"
  cniConfDir: "/etc/cni/net.d"
  args:
  - "--ip-masq"
  - "--kube-subnet-mgr"
  - "--iface=enp0s9"
  backend: "vxlan"
EOF
namespace/kube-flannel created

(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kc describe ds -n kube-flannel
Name:           kube-flannel-ds
Selector:       app=flannel
Node-Selector:  <none>
Labels:         app=flannel
                app.kubernetes.io/managed-by=Helm
                tier=node
Annotations:    deprecated.daemonset.template.generation: 1
                meta.helm.sh/release-name: flannel
                meta.helm.sh/release-namespace: kube-flannel
Desired Number of Nodes Scheduled: 1
Current Number of Nodes Scheduled: 1
Number of Nodes Scheduled with Up-to-date Pods: 1
Number of Nodes Scheduled with Available Pods: 1
Number of Nodes Misscheduled: 0
Pods Status:  1 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           app=flannel
                    tier=node
  Service Account:  flannel
  Init Containers:
   install-cni-plugin:
    Image:      ghcr.io/flannel-io/flannel-cni-plugin:v1.7.1-flannel1
    Port:       <none>
    Host Port:  <none>
    Command:
      cp
    Args:
      -f
      /flannel
      /opt/cni/bin/flannel
    Environment:  <none>
    Mounts:
      /opt/cni/bin from cni-plugin (rw)
   install-cni:
    Image:      ghcr.io/flannel-io/flannel:v0.27.3
    Port:       <none>
    Host Port:  <none>
    Command:
      cp
    Args:
      -f
      /etc/kube-flannel/cni-conf.json
      /etc/cni/net.d/10-flannel.conflist
    Environment:  <none>
    Mounts:
      /etc/cni/net.d from cni (rw)
      /etc/kube-flannel/ from flannel-cfg (rw)
  Containers:
   kube-flannel:
    Image:      ghcr.io/flannel-io/flannel:v0.27.3
    Port:       <none>
    Host Port:  <none>
    Command:
      /opt/bin/flanneld
      --ip-masq
      --kube-subnet-mgr
      --iface=enp0s9
    Requests:
      cpu:     100m
      memory:  50Mi
    Environment:
      POD_NAME:                    (v1:metadata.name)
      POD_NAMESPACE:               (v1:metadata.namespace)
      EVENT_QUEUE_DEPTH:          5000
      CONT_WHEN_CACHE_NOT_READY:  false
    Mounts:
      /etc/kube-flannel/ from flannel-cfg (rw)
      /run/flannel from run (rw)
      /run/xtables.lock from xtables-lock (rw)
  Volumes:
   run:
    Type:          HostPath (bare host directory volume)
    Path:          /run/flannel
    HostPathType:
   cni-plugin:
    Type:          HostPath (bare host directory volume)
    Path:          /opt/cni/bin
    HostPathType:
   cni:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/cni/net.d
    HostPathType:
   flannel-cfg:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      kube-flannel-cfg
    Optional:  false
   xtables-lock:
    Type:               HostPath (bare host directory volume)
    Path:               /run/xtables.lock
    HostPathType:       FileOrCreate
  Priority Class Name:  system-node-critical
  Node-Selectors:       <none>
  Tolerations:          :NoExecute op=Exists
                        :NoSchedule op=Exists
Events:
  Type    Reason            Age    From                  Message
  ----    ------            ----   ----                  -------
  Normal  SuccessfulCreate  3m57s  daemonset-controller  Created pod: kube-flannel-ds-mqn8t

# flannel cni 바이너리 설치 확인
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# ls -l /opt/cni/bin/
total 66032
-rwxr-xr-x. 1 root root 3239200 Dec 12  2024 bandwidth
-rwxr-xr-x. 1 root root 3731632 Dec 12  2024 bridge
-rwxr-xr-x. 1 root root 9123544 Dec 12  2024 dhcp
-rwxr-xr-x. 1 root root 3379872 Dec 12  2024 dummy
-rwxr-xr-x. 1 root root 3742888 Dec 12  2024 firewall
-rwxr-xr-x. 1 root root 2903098 Jan 24 19:13 flannel
-rwxr-xr-x. 1 root root 3383408 Dec 12  2024 host-device
-rwxr-xr-x. 1 root root 2812400 Dec 12  2024 host-local
-rwxr-xr-x. 1 root root 3380928 Dec 12  2024 ipvlan
-rw-r--r--. 1 root root   11357 Dec 12  2024 LICENSE
-rwxr-xr-x. 1 root root 2953200 Dec 12  2024 loopback
-rwxr-xr-x. 1 root root 3448024 Dec 12  2024 macvlan
-rwxr-xr-x. 1 root root 3312488 Dec 12  2024 portmap
-rwxr-xr-x. 1 root root 3524072 Dec 12  2024 ptp
-rw-r--r--. 1 root root    2343 Dec 12  2024 README.md
-rwxr-xr-x. 1 root root 3091976 Dec 12  2024 sbr
-rwxr-xr-x. 1 root root 2526944 Dec 12  2024 static
-rwxr-xr-x. 1 root root 3516272 Dec 12  2024 tap
-rwxr-xr-x. 1 root root 2956032 Dec 12  2024 tuning
-rwxr-xr-x. 1 root root 3380544 Dec 12  2024 vlan
-rwxr-xr-x. 1 root root 3160560 Dec 12  2024 vrf

# cni 설정 정보 확인
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# tree /etc/cni/net.d/
/etc/cni/net.d/
└── 10-flannel.conflist

1 directory, 1 file
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# cat /etc/cni/net.d/10-flannel.conflist | jq
{
  "name": "cbr0",
  "cniVersion": "0.3.1",
  "plugins": [
    {
      "type": "flannel",
      "delegate": {
        "hairpinMode": true,
        "isDefaultGateway": true
      }
    },
    {
      "type": "portmap",
      "capabilities": {
        "portMappings": true
      }
    }
  ]
}

# coredns 파드 정상 기동 확인
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kubectl get pod -n kube-system -owide
NAME                              READY   STATUS    RESTARTS   AGE   IP               NODE      NOMINATED NODE   READINESS GATES
coredns-668d6bf9bc-c2g8k          1/1     Running   0          21m   10.244.0.4       k8s-ctr   <none>           <none>
coredns-668d6bf9bc-qdwrj          1/1     Running   0          21m   10.244.0.5       k8s-ctr   <none>           <none>
etcd-k8s-ctr                      1/1     Running   0          21m   192.168.10.100   k8s-ctr   <none>           <none>
kube-apiserver-k8s-ctr            1/1     Running   1          21m   192.168.10.100   k8s-ctr   <none>           <none>
kube-controller-manager-k8s-ctr   1/1     Running   0          21m   192.168.10.100   k8s-ctr   <none>           <none>
kube-proxy-6gfjf                  1/1     Running   0          21m   192.168.10.100   k8s-ctr   <none>           <none>
kube-scheduler-k8s-ctr            1/1     Running   0          21m   192.168.10.100   k8s-ctr   <none>           <none>

# network 정보 확인
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# ip -c route | grep 10.244.
10.244.0.0/24 dev cni0 proto kernel scope link src 10.244.0.1
10.244.1.0/24 via 10.244.1.0 dev flannel.1 onlink
10.244.3.0/24 via 10.244.3.0 dev flannel.1 onlink
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 08:00:27:90:ea:eb brd ff:ff:ff:ff:ff:ff
    altname enx08002790eaeb
    inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic noprefixroute enp0s8
       valid_lft 70715sec preferred_lft 70715sec
    inet6 fd17:625c:f037:2:a00:27ff:fe90:eaeb/64 scope global dynamic mngtmpaddr proto kernel_ra
       valid_lft 86384sec preferred_lft 14384sec
    inet6 fe80::a00:27ff:fe90:eaeb/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
3: enp0s9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 08:00:27:e6:47:2d brd ff:ff:ff:ff:ff:ff
    altname enx080027e6472d
    inet 192.168.10.100/24 brd 192.168.10.255 scope global noprefixroute enp0s9
       valid_lft forever preferred_lft forever
    inet6 fe80::6edd:7039:2b9b:4df9/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
    link/ether 46:07:ca:bc:29:2a brd ff:ff:ff:ff:ff:ff
    inet 10.244.0.0/32 scope global flannel.1
       valid_lft forever preferred_lft forever
    inet6 fe80::4407:caff:febc:292a/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
5: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether 3a:80:fe:79:51:dd brd ff:ff:ff:ff:ff:ff
    inet 10.244.0.1/24 brd 10.244.0.255 scope global cni0
       valid_lft forever preferred_lft forever
    inet6 fe80::3880:feff:fe79:51dd/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
8: veth9abddb6f@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default qlen 1000
    link/ether 56:36:ed:f4:e5:83 brd ff:ff:ff:ff:ff:ff link-netns cni-cec5c89a-bc9d-932b-3b31-7c5e46a17a18
    inet6 fe80::5436:edff:fef4:e583/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
9: veth9f16af76@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default qlen 1000
    link/ether 2a:42:20:93:b8:c4 brd ff:ff:ff:ff:ff:ff link-netns cni-87d1e3d6-f0b5-0557-6b53-1b4e7f00588e
    inet6 fe80::2842:20ff:fe93:b8c4/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# bridge link
8: veth9abddb6f@enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 master cni0 state forwarding priority 32 cost 2
9: veth9f16af76@enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 master cni0 state forwarding priority 32 cost 2
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# lsns -t net
        NS TYPE NPROCS   PID USER       NETNSID NSFS                                                COMMAND
4026531840 net     152     1 root    unassigned                                                     /usr/lib/systemd/systemd --switched-root --system --deserialize=46 no_timer_check
4026532129 net       1   645 root    unassigned                                                     ├─/usr/sbin/irqbalance
4026532202 net       1   803 polkitd unassigned                                                     └─/usr/lib/polkit-1/polkitd --no-debug --log-level=err
4026532293 net       2 72912 65535            0 /run/netns/cni-cec5c89a-bc9d-932b-3b31-7c5e46a17a18 /pause
4026532371 net       2 72914 65535            1 /run/netns/cni-87d1e3d6-f0b5-0557-6b53-1b4e7f00588e /pause

4-4. [k8s-ctr] 노드 정보 확인, 기본 환경 정보 출력 비교, sysctl 변경 확인

# # kubelet 활성화 확인 : 실제 기동은 kubeadm init 후에 시작됨
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# systemctl is-active kubelet
active

# 노드 정보 확인 : 일반 워크로드가 Control Plane에 스케줄 X
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kc describe node
Name:               k8s-ctr
Roles:              control-plane
Labels:             beta.kubernetes.io/arch=arm64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=arm64
                    kubernetes.io/hostname=k8s-ctr
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/control-plane=
                    node.kubernetes.io/exclude-from-external-load-balancers=
Annotations:        flannel.alpha.coreos.com/backend-data: {"VNI":1,"VtepMAC":"46:07:ca:bc:29:2a"}
                    flannel.alpha.coreos.com/backend-type: vxlan
                    flannel.alpha.coreos.com/kube-subnet-manager: true
                    flannel.alpha.coreos.com/public-ip: 192.168.10.100
                    kubeadm.alpha.kubernetes.io/cri-socket: unix:///run/containerd/containerd.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Sat, 24 Jan 2026 18:56:13 +0900
Taints:             node-role.kubernetes.io/control-plane:NoSchedule
...


# 기본 환경 정보 출력 저장
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# cat /etc/sysconfig/kubelet
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# tree /etc/kubernetes  | tee -a etc_kubernetes-2.txt
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# tree /var/lib/kubelet | tee -a var_lib_kubelet-2.txt
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# tree /run/containerd/ -L 3 | tee -a run_containerd-2.txt
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# pstree -alnp | tee -a pstree-2.txt
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# systemd-cgls --no-pager | tee -a systemd-cgls-2.txt
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# lsns | tee -a lsns-2.txt
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# ip addr | tee -a ip_addr-2.txt 
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# ss -tnlp | tee -a ss-2.txt
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# df -hT | tee -a df-2.txt
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# findmnt | tee -a findmnt-2.txt
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# sysctl -a | tee -a sysctl-2.txt

# 파일 출력 비교 : 빠져나오기 ':q' -> ':q' => 변경된 부분이 어떤 동작과 역할인지 조사해보기!
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# vi -d etc_kubernetes-1.txt etc_kubernetes-2.txt
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# vi -d var_lib_kubelet-1.txt var_lib_kubelet-2.txt
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# vi -d run_containerd-1.txt run_containerd-2.txt
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# vi -d pstree-1.txt pstree-2.txt
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# vi -d systemd-cgls-1.txt systemd-cgls-2.txt
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# vi -d lsns-1.txt lsns-2.txt
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# vi -d ip_addr-1.txt ip_addr-2.txt
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# vi -d ss-1.txt ss-2.txt
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# vi -d df-1.txt df-2.txt
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# vi -d findmnt-1.txt findmnt-2.txt

# kubelet 에 --protect-kernel-defaults=false 적용되어 관련 코드에 sysctl 커널 파라미터 적용 : 아래 링크 확왼
## 위 설정 시, 커널 튜닝 가능 항목 중 하나라도 kubelet의 기본값과 다르면 오류가 발생합니다
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# vi -d sysctl-1.txt sysctl-2.txt
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kernel.panic = 0 -> 10 변경
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kernel.panic_on_oops = 1 기존값 그대로
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# vm.overcommit_memory = 0 -> 1 변경
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# vm.panic_on_oom = 0 기존값 그대로

4-5. [k8s-ctr] 인증서 확인

(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kc describe cm -n kube-system kubeadm-config
Name:         kubeadm-config
Namespace:    kube-system
Labels:       <none>
Annotations:  <none>

Data
====
ClusterConfiguration:
----
apiServer: {}
apiVersion: kubeadm.k8s.io/v1beta4
caCertificateValidityPeriod: 87600h0m0s
certificateValidityPeriod: 8760h0m0s
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
encryptionAlgorithm: RSA-2048
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.k8s.io
kind: ClusterConfiguration
kubernetesVersion: v1.32.11
networking:
  dnsDomain: cluster.local
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/16
proxy: {}
scheduler: {}



BinaryData
====

Events:  <none>

(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kubeadm certs check-expiration
[check-expiration] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[check-expiration] Use 'kubeadm init phase upload-config --config your-config.yaml' to re-upload it.

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Jan 24, 2027 09:56 UTC   364d            ca                      no
apiserver                  Jan 24, 2027 09:56 UTC   364d            ca                      no
apiserver-etcd-client      Jan 24, 2027 09:56 UTC   364d            etcd-ca                 no
apiserver-kubelet-client   Jan 24, 2027 09:56 UTC   364d            ca                      no
controller-manager.conf    Jan 24, 2027 09:56 UTC   364d            ca                      no
etcd-healthcheck-client    Jan 24, 2027 09:56 UTC   364d            etcd-ca                 no
etcd-peer                  Jan 24, 2027 09:56 UTC   364d            etcd-ca                 no
etcd-server                Jan 24, 2027 09:56 UTC   364d            etcd-ca                 no
front-proxy-client         Jan 24, 2027 09:56 UTC   364d            front-proxy-ca          no
scheduler.conf             Jan 24, 2027 09:56 UTC   364d            ca                      no
super-admin.conf           Jan 24, 2027 09:56 UTC   364d            ca                      no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Jan 22, 2036 09:56 UTC   9y              no
etcd-ca                 Jan 22, 2036 09:56 UTC   9y              no
front-proxy-ca          Jan 22, 2036 09:56 UTC   9y              no

(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# cat /etc/kubernetes/pki/ca.crt | openssl x509 -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 2535030548539110019 (0x232e3cda4c6ff283)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kubernetes
        Validity
            Not Before: Jan 24 09:51:07 2026 GMT
            Not After : Jan 22 09:56:07 2036 GMT
        Subject: CN=kubernetes
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:d8:64:b9:f1:a7:54:1a:a3:57:1f:f5:e0:3a:55:
                    d8:d5:a6:87:44:cf:3a:77:f7:cc:1d:d5:73:e8:c1:
                    47:01:99:bc:3a:e4:36:15:18:6f:6c:51:ea:95:fc:
                    de:b3:41:10:c5:23:66:cf:c8:44:83:e4:8d:ce:bd:
                    22:a8:4d:d5:79:fe:48:13:46:26:ab:2d:86:9a:ff:
                    c5:c2:68:ed:fd:08:d3:a3:89:fd:bb:64:04:57:92:
                    57:2c:f6:cd:9b:f3:73:52:46:ba:76:6f:a6:da:eb:
                    e1:a7:17:49:82:e5:0c:8e:94:c9:de:4b:34:be:ef:
                    c5:2f:fe:b8:af:3a:00:a4:77:0d:01:20:3f:e7:4e:
                    13:7b:68:6c:9a:e3:f0:93:22:25:41:d8:48:40:c9:
                    e3:9a:54:71:03:1b:12:b6:31:8e:71:57:f8:aa:57:
                    48:53:5f:05:a9:bb:ac:5b:81:4d:6f:41:e9:e2:c6:
                    f0:74:c9:2e:f6:f3:69:d0:8f:5d:ae:25:cb:b5:6d:
                    11:b4:63:e0:a5:4b:7a:19:d3:ed:17:0d:ff:6d:4a:
                    0e:4d:c4:4e:33:8a:1b:fa:d6:20:18:c3:6a:52:1b:
                    72:39:34:16:95:5a:80:a6:9c:15:31:92:70:40:9b:
                    86:5e:ae:8e:38:49:3b:ca:48:93:b5:c3:2a:c3:de:
                    bc:a7
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment, Certificate Sign
            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Subject Key Identifier:
                1B:C4:2B:E8:D7:F6:3C:42:70:2C:B9:AE:28:67:35:0D:9F:41:68:CB
            X509v3 Subject Alternative Name:
                DNS:kubernetes
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        9b:74:c2:34:3d:aa:db:d2:6e:86:b7:86:78:44:c1:64:a8:6c:
        2f:38:43:2e:f5:77:93:78:c5:0a:8f:35:89:21:04:c4:d4:96:
        42:8f:51:5f:c8:69:a1:02:0b:0f:84:33:ea:3d:99:7a:af:66:
        a8:8a:4f:56:6a:79:af:be:57:77:b4:55:c8:8e:7b:4e:dc:5a:
        d5:90:d2:dd:0f:ee:b4:55:47:75:4a:7f:df:a0:93:33:55:b2:
        bb:f6:3e:62:4d:09:39:89:b4:7a:b6:48:b3:1d:58:0d:2a:82:
        f2:65:bf:7c:23:31:3c:18:3a:93:e9:bb:af:76:84:f7:95:1d:
        67:51:94:8c:4e:b1:24:e8:ab:a9:f3:4d:9c:e1:cc:2e:2c:63:
        82:f8:66:f9:60:7e:a8:b6:0c:32:c3:1d:5c:84:ba:65:c5:50:
        5d:24:f9:89:5a:fe:2a:ab:a9:bf:6a:f7:91:74:ae:a8:72:ea:
        f7:51:c9:21:6d:a6:84:a7:28:69:45:d0:b4:eb:76:2a:e9:be:
        c1:c6:85:98:fe:d7:50:0b:c2:f8:0f:92:54:90:89:73:32:80:
        70:ef:2c:db:91:4d:87:e0:3c:c0:57:7d:29:7a:32:51:b9:44:
        48:fa:a0:1a:a8:70:e8:8c:f5:73:b2:c1:e7:72:9a:f4:69:23:
        54:1d:77:20
        
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# cat /etc/kubernetes/pki/apiserver.crt | openssl x509 -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 6316045001739227051 (0x57a719771a0c4fab)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kubernetes
        Validity
            Not Before: Jan 24 09:51:07 2026 GMT
            Not After : Jan 24 09:56:07 2027 GMT
        Subject: CN=kube-apiserver
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:e6:16:f7:26:f2:dd:2c:58:be:92:31:a9:dc:31:
                    c2:47:db:b0:3d:b0:07:d5:7b:12:71:b2:6d:bb:e6:
                    f5:89:a9:92:7e:e4:e2:67:70:f6:11:33:1a:e2:10:
                    42:39:c9:a9:84:a7:a2:68:3e:dd:01:b9:6b:05:2a:
                    c3:e9:23:c0:72:49:73:5b:51:8f:51:5e:fd:9c:80:
                    6a:e1:e5:4d:ee:c7:e5:7b:b6:f6:e7:84:30:27:17:
                    7f:e9:87:a9:a2:2b:7d:d0:78:7d:da:7f:bb:6e:59:
                    ed:19:3b:8f:08:a9:bd:c1:aa:4b:05:e4:5b:7f:41:
                    59:c2:b9:74:d6:33:38:8b:27:bb:d6:bb:b9:90:9d:
                    bb:36:13:3e:ce:af:32:33:6a:fb:d4:53:de:46:19:
                    c4:27:99:ce:0f:9d:fa:38:b6:9e:17:65:1f:fb:b2:
                    4c:17:c8:14:39:e8:3f:8e:5e:2f:a2:9d:a0:b2:f7:
                    df:32:18:8a:28:81:89:4b:a4:9e:6f:8b:98:45:75:
                    b6:be:d2:86:19:8f:75:86:e3:7c:28:08:13:2d:8d:
                    5f:1d:df:aa:32:5e:56:a1:e2:16:17:ed:f2:bd:82:
                    f7:b7:4e:2a:2f:e0:04:1c:9b:47:84:ba:ab:05:f2:
                    a1:aa:c2:5c:81:51:55:90:55:63:44:97:42:03:01:
                    2a:1d
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Authority Key Identifier:
                1B:C4:2B:E8:D7:F6:3C:42:70:2C:B9:AE:28:67:35:0D:9F:41:68:CB
            X509v3 Subject Alternative Name:
                DNS:k8s-ctr, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP Address:10.96.0.1, IP Address:192.168.10.100
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        b1:41:49:5a:07:7e:59:23:3d:b0:fa:1e:17:15:3c:a7:ba:a6:
        ee:bd:93:c5:20:a6:56:83:a5:79:fa:25:91:dc:68:6f:c5:fe:
        96:9b:46:dd:10:65:5e:a9:c6:9f:2b:7b:b2:1d:15:e5:52:09:
        45:5e:13:3c:41:87:fd:d7:33:29:c7:72:2c:5a:6a:7b:4d:0a:
        f0:ca:d6:2f:ba:c1:29:0c:a6:40:00:fd:e0:a1:aa:a8:91:cc:
        29:df:93:9b:12:04:bb:7a:ff:45:5b:d4:cd:f5:11:61:f6:34:
        9c:2c:af:39:10:2c:54:2c:41:f2:d5:ab:44:10:0a:ec:ea:04:
        33:ea:d0:ab:8b:6d:e5:17:e7:7c:7a:4f:72:3f:fb:20:e9:aa:
        e1:70:09:08:70:d2:67:6f:04:56:1e:13:80:ff:fe:3d:75:8e:
        10:ad:ea:9b:25:50:17:d0:aa:f2:01:2c:44:0c:41:19:a8:3e:
        7b:da:56:1c:70:e4:38:2e:e2:88:f3:31:18:d3:ac:b9:c4:ac:
        2d:cb:a0:ba:45:cb:06:e4:00:c1:3a:a0:47:79:e8:c9:1b:b5:
        af:0b:82:bf:0b:e1:68:eb:03:6d:14:ca:33:63:64:42:ce:58:
        ab:71:1a:77:a8:28:44:84:69:e7:9a:e2:e5:52:6d:5e:76:34:
        00:80:ba:70
        
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# cat /etc/kubernetes/pki/apiserver-kubelet-client.crt | openssl x509 -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 8855908732919257496 (0x7ae682126cee9998)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kubernetes
        Validity
            Not Before: Jan 24 09:51:07 2026 GMT
            Not After : Jan 24 09:56:07 2027 GMT
        Subject: O=kubeadm:cluster-admins, CN=kube-apiserver-kubelet-client
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:e2:ee:cf:50:27:10:3d:20:49:98:76:26:1a:36:
                    b6:06:b6:47:5e:bd:cc:85:7d:96:98:f5:38:42:c9:
                    ea:58:8d:9c:00:1f:e7:84:07:d8:f8:b5:75:22:77:
                    40:c4:bc:7e:d3:89:6d:f2:a9:b3:88:64:f8:67:5f:
                    34:80:5e:4e:31:8b:6b:22:fb:0f:77:0f:08:d4:4f:
                    b0:59:1d:bb:38:80:24:e6:a2:40:22:a0:95:db:21:
                    84:88:f8:d9:be:12:5d:f9:97:f3:78:b2:1f:8d:ef:
                    3d:13:8d:01:37:32:24:84:43:46:4b:76:98:98:e8:
                    9e:cc:e2:77:33:34:0c:59:8f:ea:0b:b3:9a:58:66:
                    03:37:77:f8:66:56:33:12:aa:68:b3:92:c7:a6:d0:
                    02:40:13:ca:3a:e6:37:34:ce:89:97:27:a7:d2:56:
                    d7:c4:9e:ee:ff:ab:29:1b:82:b4:39:c7:3e:aa:47:
                    4c:34:01:71:e7:a5:bb:a7:4b:cb:58:33:79:8f:91:
                    17:38:fd:d8:07:83:ac:24:90:06:fc:a5:b4:3a:7d:
                    43:27:e7:d6:4b:b2:87:0e:0f:85:f1:85:8c:ee:47:
                    a0:5f:d6:07:17:6d:37:52:27:23:97:9d:7a:03:00:
                    39:65:bf:54:0f:0f:9e:a7:02:98:38:1a:96:d1:68:
                    02:7b
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Authority Key Identifier:
                1B:C4:2B:E8:D7:F6:3C:42:70:2C:B9:AE:28:67:35:0D:9F:41:68:CB
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        53:4b:76:fe:25:29:83:35:c7:31:23:e6:8f:16:23:af:65:bf:
        90:56:ba:bb:f3:31:62:cd:34:e1:82:41:5e:bc:e3:46:1d:e2:
        ca:1e:f2:25:34:d3:10:9d:50:9b:11:6b:31:7d:4f:34:8b:b3:
        a3:10:69:d7:a8:e1:63:4f:ba:a3:22:e1:86:54:58:0b:96:9f:
        e5:a9:2c:93:11:ac:20:2f:7b:6d:e6:e8:ea:ba:56:84:67:a2:
        a5:ae:c9:27:47:4a:72:3d:4d:50:35:0a:67:51:e5:98:0c:48:
        06:23:62:03:61:2c:8a:6d:cb:cd:09:1e:29:42:e5:67:97:e3:
        79:e0:0e:a6:ee:33:cd:3f:cb:88:9e:ed:fb:56:e9:f8:54:cd:
        c8:09:c2:17:c9:78:9f:0c:96:90:1e:dd:b3:6f:ad:75:b1:8b:
        a5:e3:f9:ae:95:7b:b4:4e:44:dc:51:fe:6e:39:fd:24:4f:64:
        9d:6c:d9:6c:6d:6e:b1:98:6d:12:16:36:b9:a0:94:6b:66:9c:
        ed:e7:a5:56:b2:90:b2:38:7f:15:84:c2:0e:34:ca:f2:a3:8e:
        d2:95:b2:76:b7:07:d6:2c:3d:85:1b:19:9d:11:48:20:9b:84:
        69:5a:81:3b:35:6b:fc:2f:2e:f3:94:c3:50:dc:88:69:73:7e:
        3c:ee:6a:19

4-6. [k8s-ctr] kubeconfig 확인

# 관리자 용도
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# cat /etc/kubernetes/admin.conf
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURCVENDQWUyZ0F3SUJBZ0lJSXk0ODJreHY4b013RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TmpBeE1qUXdPVFV4TURkYUZ3MHpOakF4TWpJd09UVTJNRGRhTUJVeApFekFSQmdOVkJBTVRDbXQxWW1WeWJtVjBaWE13Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLCkFvSUJBUURZWkxueHAxUWFvMWNmOWVBNlZkalZwb2RFenpwMzk4d2QxWFBvd1VjQm1idzY1RFlWR0c5c1VlcVYKL042elFSREZJMmJQeUVTRDVJM092U0tvVGRWNS9rZ1RSaWFyTFlhYS84WENhTzM5Q05PamlmMjdaQVJYa2xjcwo5czJiODNOU1JycDJiNmJhNitHbkYwbUM1UXlPbE1uZVN6Uys3OFV2L3Jpdk9nQ2tkdzBCSUQvblRoTjdhR3lhCjQvQ1RJaVZCMkVoQXllT2FWSEVER3hLMk1ZNXhWL2lxVjBoVFh3V3B1NnhiZ1UxdlFlbml4dkIweVM3MjgyblEKajEydUpjdTFiUkcwWStDbFMzb1owKzBYRGY5dFNnNU54RTR6aWh2NjFpQVl3MnBTRzNJNU5CYVZXb0NtbkJVeAprbkJBbTRaZXJvNDRTVHZLU0pPMXd5ckQzcnluQWdNQkFBR2pXVEJYTUE0R0ExVWREd0VCL3dRRUF3SUNwREFQCkJnTlZIUk1CQWY4RUJUQURBUUgvTUIwR0ExVWREZ1FXQkJRYnhDdm8xL1k4UW5Bc3VhNG9aelVObjBGb3l6QVYKQmdOVkhSRUVEakFNZ2dwcmRXSmxjbTVsZEdWek1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQ2JkTUkwUGFyYgowbTZHdDRaNFJNRmtxR3d2T0VNdTlYZVRlTVVLanpXSklRVEUxSlpDajFGZnlHbWhBZ3NQaERQcVBabDZyMmFvCmlrOVdhbm12dmxkM3RGWElqbnRPM0ZyVmtOTGREKzYwVlVkMVNuL2ZvSk16VmJLNzlqNWlUUWs1aWJSNnRraXoKSFZnTktvTHlaYjk4SXpFOEdEcVQ2YnV2ZG9UM2xSMW5VWlNNVHJFazZLdXA4MDJjNGN3dUxHT0MrR2I1WUg2bwp0Z3d5d3gxY2hMcGx4VkJkSlBtSld2NHFxNm0vYXZlUmRLNm9jdXIzVWNraGJhYUVweWhwUmRDMDYzWXE2YjdCCnhvV1kvdGRRQzhMNEQ1SlVrSWx6TW9Cdzd5emJrVTJINER6QVYzMHBlakpSdVVSSStxQWFxSERvalBWenNzSG4KY3ByMGFTTlVIWGNnCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    server: https://192.168.10.100:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: kubernetes-admin
  name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
  user:
    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURLVENDQWhHZ0F3SUJBZ0lJQjk4bTBrUk43ZEF3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TmpBeE1qUXdPVFV4TURkYUZ3MHlOekF4TWpRd09UVTJNRGRhTUR3eApIekFkQmdOVkJBb1RGbXQxWW1WaFpHMDZZMngxYzNSbGNpMWhaRzFwYm5NeEdUQVhCZ05WQkFNVEVHdDFZbVZ5CmJtVjBaWE10WVdSdGFXNHdnZ0VpTUEwR0NTcUdTSWIzRFFFQkFRVUFBNElCRHdBd2dnRUtBb0lCQVFDb1YzdmkKREdZRlgvdnE1eElwVjM4bGw2MTJ4TzVaT3dwOTBsUjRIbk5BK3AyTmwreHlZSG9kTjAxTlUyVk5TMUxmR2dsRwo3ZU12VVpTbjdQdUhFMXVtNXN0a25BY1R0bWxGaUhNWGdGK2phMVM2MkNDTGFBYnV3VWh5WG95eXhaYWtnN1NUCkJuVjdBaWNZSDc2Rk95RS96RFhncEErWWx2QSsrdTJoOWZUZVpBbUcyTVhXUHM2T3dLNmtvdkd4STQ3V3lPSDUKc1pIVlg3OXA1MUN4QWlET1NyN0FMaGl0WFVKOGFDZVhqTGVvNTc5VUpjZENmS3J0d04ybjN2Vm9DeXdFTHNLSgpMQURoRjY0Nk5WTXFuMEYwNXQ4eWtLQjY0Z0h2T2xac01Qa2JXNm5mWS9hZng1TXo1RjVKR3RVYmtiVjR1SGhPCjAwZitycEkySEYxQ3RiOTFBZ01CQUFHalZqQlVNQTRHQTFVZER3RUIvd1FFQXdJRm9EQVRCZ05WSFNVRUREQUsKQmdnckJnRUZCUWNEQWpBTUJnTlZIUk1CQWY4RUFqQUFNQjhHQTFVZEl3UVlNQmFBRkJ2RUsralg5anhDY0N5NQpyaWhuTlEyZlFXakxNQTBHQ1NxR1NJYjNEUUVCQ3dVQUE0SUJBUURRWExaTGY0N0JjM3NyN2xPclIvQUFGcUtWCnZZZGVWa1hHa21SajhPYlV4SjY3U1JHZElkYkFKblQ1WnhYaVhyUkdCZnFDNkVGQjZFVjUxZlNYZVMzcDgyYlkKblRxNmE2eXE5WVY5SU5SalVvQ0RJUEY2WVFYcmNDelJuQWVkaEdLZGtWV2lHbytNWDdEdU5nZW9BUnlBc3hoRwoxMjZVN0ZZRWZLZVhBSVpZMmljSnNUcS94TFd2S0g3VVE1U21JVzlhWHZ3czBXL1A4c25rNm93ODhyMlNzcHZ5CjlOZkpuUEd4K3duUVlrTlM1RkxtV3lncVc1dEZuWC9Wd0psV3N3RFV6QXI0anFQejhMRDNxWUxJSzY2RUhxaEUKaW1EVWFLeG14ZVJSTjB3dDJENitWUnhLMm8yOXJZemRnUHlXYy9PVkpxNWNBVkhpalZ3eURSNGhsN1ltCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBcUZkNzRneG1CVi83NnVjU0tWZC9KWmV0ZHNUdVdUc0tmZEpVZUI1elFQcWRqWmZzCmNtQjZIVGROVFZObFRVdFMzeG9KUnUzakwxR1VwK3o3aHhOYnB1YkxaSndIRTdacFJZaHpGNEJmbzJ0VXV0Z2cKaTJnRzdzRkljbDZNc3NXV3BJTzBrd1oxZXdJbkdCKytoVHNoUDh3MTRLUVBtSmJ3UHZydG9mWDAzbVFKaHRqRgoxajdPanNDdXBLTHhzU09PMXNqaCtiR1IxVisvYWVkUXNRSWd6a3Erd0M0WXJWMUNmR2dubDR5M3FPZS9WQ1hIClFueXE3Y0RkcDk3MWFBc3NCQzdDaVN3QTRSZXVPalZUS3A5QmRPYmZNcENnZXVJQjd6cFdiREQ1RzF1cDMyUDIKbjhlVE0rUmVTUnJWRzVHMWVMaDRUdE5IL3E2U05oeGRRclcvZFFJREFRQUJBb0lCQUNobFdlSjhKQzB0QThneQpLOThFMG91RVVzbFo1M0k5SXo3Zkxvcm1qN1Nyand3dnhUc0xJTEtMRno4emdHOGtZSjRONHVVRTU4dnVrVFFjCnY1MEJ6YkFHMlE3ckRCMjBXNTJtYVN2ZUQ5VW94OXZRU2pyNXV4UW5DSW45VzFqNDVqWFRMdzFLOHYwU0hxeUEKelppdUFFUU5ibTVhSUMzM0ptNk9pMkNlbzJTTURDVUlsZnlFK1hFU2dIL2RPR0ZNU09PeU9nZkZ5eTM5SjA1UgpvSmdsK1RIdVZrNVJWSWgwNlRoZm1ESlczMjdCS2FwT3NLYWg5NFpRRjVQWFVwYzNuY0xLeGVvTEVYNGNpdXFtCk10QmtuSEdTMVFINzYvN09meE1BS08yckpLMUFWR0gyRUo5ZE1PT2NWNCswOUk0dThCRFAweUFNZU1HYlRmTnYKcnBsNENSRUNnWUVBeFdOb3IxOUxWU2M3Z2x5V0RNbEJSZ0NWV3pFd3VyQXZMTmRZTm5PZHhZdkNkSGlKa0xDbwpFL2tteW5uYUdTdEIvNkJ3Vi9MQndUQ1lheHBaZ05XMU1ReTJaZ3lkcUE2SGh5TVNzMkRwQkt3cW1lR2dUeUY0CmVCdjUyNlZwRng2NVhpK20vcmIvUUx3cUhjZlJ5dDc3aStTZEhLV3BWNndabVhwV3kyN3ozOWtDZ1lFQTJsUVUKNTU5TUlGSWp2eGJPR01qSUdROXpOQTZmN3NLckZnY3JXK0NjN1F5czNSRUc1UjVXNjJpN29PcUFZVVk4Rm9YRwpHcnVGMStiSnFUVkpZYTRZVmI1dU81dUNhaGMvcGV0V005SW5RcEZxM0tSUEdTY3JaLzJyVUtCeHY3YjJnay9ECkxWd3dzOHh4cUFKdnpXa2o2TSs2VHVCVVNqR05acnJaa1Ivdjl2MENnWUVBclovdW9teFJTRnJWSnFzb05hRUYKc0h5czQrVVY5dkVvM2VtaUoydDFlU0doYjIvam1ZazZueThHcHcyZUFZdWlaeWVLQ21KM2VlYXorMm5YRnROawpxUHVFcWFrcE9IMW5TMEJYbjc5NzJHZFVwYnpvbFJKYzlGR3ZhenhKZjFQQVBBL3dkWmNrV1ozcDhmNGxGSzBsCldQMUVFY0hLZmxyY3ZicjJBOFhaOEtrQ2dZQStzazZlaFR4VE84TlFLTGhlbmFuNHFGc280OXBCc2wxM0llL3QKbm43eUErWFFSZ2Q0M0ZHUm9LM2c4L2FSK0oxZ3ltR3RZNVIzLzZxQmtPL1Z3U3p6MG8vTlJrY1pPRHZxNWI0SAplNTRTbTdmWVRNYjZMaWxrMzQvR3c0eG14Wi9jcEJNa2Y0anMyUlQ2YmxpMDREQ1R0ck9GMngzWmdJbGVxdUczCnJ6ZzE4UUtCZ1FDZC91U1Y0bjJ5N24vVlV0RzdpTnd0YWp1MjExZnNpUitDMk94Z2wrWm5DelRockJwMnc0elUKTXFCZzJXczFsTXp5eHdNWmlreW5xekdxWnJ2UVlISnpWUU83bEs3MnhSVXBjTmxHaHliU3NkL05UYWhqakRKSAp3Zk4wUUtQYVkvRS84MWJ1QUhSUmhwTUhUWExyWElVQS9HeWxGQzdPWnNZVzBhdlM4U3lFV1E9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=
    
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# cat /etc/kubernetes/super-admin.conf
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURCVENDQWUyZ0F3SUJBZ0lJSXk0ODJreHY4b013RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TmpBeE1qUXdPVFV4TURkYUZ3MHpOakF4TWpJd09UVTJNRGRhTUJVeApFekFSQmdOVkJBTVRDbXQxWW1WeWJtVjBaWE13Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLCkFvSUJBUURZWkxueHAxUWFvMWNmOWVBNlZkalZwb2RFenpwMzk4d2QxWFBvd1VjQm1idzY1RFlWR0c5c1VlcVYKL042elFSREZJMmJQeUVTRDVJM092U0tvVGRWNS9rZ1RSaWFyTFlhYS84WENhTzM5Q05PamlmMjdaQVJYa2xjcwo5czJiODNOU1JycDJiNmJhNitHbkYwbUM1UXlPbE1uZVN6Uys3OFV2L3Jpdk9nQ2tkdzBCSUQvblRoTjdhR3lhCjQvQ1RJaVZCMkVoQXllT2FWSEVER3hLMk1ZNXhWL2lxVjBoVFh3V3B1NnhiZ1UxdlFlbml4dkIweVM3MjgyblEKajEydUpjdTFiUkcwWStDbFMzb1owKzBYRGY5dFNnNU54RTR6aWh2NjFpQVl3MnBTRzNJNU5CYVZXb0NtbkJVeAprbkJBbTRaZXJvNDRTVHZLU0pPMXd5ckQzcnluQWdNQkFBR2pXVEJYTUE0R0ExVWREd0VCL3dRRUF3SUNwREFQCkJnTlZIUk1CQWY4RUJUQURBUUgvTUIwR0ExVWREZ1FXQkJRYnhDdm8xL1k4UW5Bc3VhNG9aelVObjBGb3l6QVYKQmdOVkhSRUVEakFNZ2dwcmRXSmxjbTVsZEdWek1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQ2JkTUkwUGFyYgowbTZHdDRaNFJNRmtxR3d2T0VNdTlYZVRlTVVLanpXSklRVEUxSlpDajFGZnlHbWhBZ3NQaERQcVBabDZyMmFvCmlrOVdhbm12dmxkM3RGWElqbnRPM0ZyVmtOTGREKzYwVlVkMVNuL2ZvSk16VmJLNzlqNWlUUWs1aWJSNnRraXoKSFZnTktvTHlaYjk4SXpFOEdEcVQ2YnV2ZG9UM2xSMW5VWlNNVHJFazZLdXA4MDJjNGN3dUxHT0MrR2I1WUg2bwp0Z3d5d3gxY2hMcGx4VkJkSlBtSld2NHFxNm0vYXZlUmRLNm9jdXIzVWNraGJhYUVweWhwUmRDMDYzWXE2YjdCCnhvV1kvdGRRQzhMNEQ1SlVrSWx6TW9Cdzd5emJrVTJINER6QVYzMHBlakpSdVVSSStxQWFxSERvalBWenNzSG4KY3ByMGFTTlVIWGNnCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    server: https://192.168.10.100:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: kubernetes-super-admin
  name: kubernetes-super-admin@kubernetes
current-context: kubernetes-super-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-super-admin
  user:
    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURKekNDQWcrZ0F3SUJBZ0lJRXdtSVo3Q09ZUnd3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TmpBeE1qUXdPVFV4TURkYUZ3MHlOekF4TWpRd09UVTJNRGRhTURveApGekFWQmdOVkJBb1REbk41YzNSbGJUcHRZWE4wWlhKek1SOHdIUVlEVlFRREV4WnJkV0psY201bGRHVnpMWE4xCmNHVnlMV0ZrYldsdU1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBd2gvdFNJZUUKOU9LTWlQbXU4Vk96UVYzckpTamt4TStDTFNObTdyWUNRYXpqaU1aRm1tUVFnMWRoYk52Ri84M2hpTHkraWkvcAo2cVovaklhNUtwSGZqRjVrN2Z5b3Z3c3JlVjFaL0R0MzBROVZIVWQ2TmV0VlhjcDIzalppeDF2UXI2d0FhNE9SClhBMDV0L1FaRGFhMGVFQXFDSSt5akE2cTM4LzFXVlJnY2lmUzdrRWJiWW4yNE1xN05iRW9IckY3Ym9rTTlIMGYKYVAvdThaU2JoeVJsRjk0UkJwQTJiVENkWUJXd2IzYy92SjJ3aFpSSzRYYW9Xcm01ZXhEZHNFOENUcVIzZlNoeQpySzBhWWN5cDRXSjJuVGdEalNTYXllSHo2YjIzNDJ3NEJrM3BmbndmRXY3bGpqcU9rdTRsQXVtM2JUc2h2STc2ClBnZGtGaTMyd1M0YVpRSURBUUFCbzFZd1ZEQU9CZ05WSFE4QkFmOEVCQU1DQmFBd0V3WURWUjBsQkF3d0NnWUkKS3dZQkJRVUhBd0l3REFZRFZSMFRBUUgvQkFJd0FEQWZCZ05WSFNNRUdEQVdnQlFieEN2bzEvWThRbkFzdWE0bwpaelVObjBGb3l6QU5CZ2txaGtpRzl3MEJBUXNGQUFPQ0FRRUF1Sm5TdHlhTVh4OFhWMDVIVlIxcjlUS29SSDZrClVVZG01NjRVdVN2N21QZlh2WWk3am1Va3pRaU13azZrWXZBWlFXZjFxVmhjcUt2djNtcXVGMGZNLzhHQTNtTEEKeDFhU2JtaUJ2eEZpaE11ZURjVmFObHN2VUNEbkRrQnVKRHFZSFE3dzJDWVhVTG4vZ0hRYndLUXFhbU5GSEdNaApCL3JJMVl3VFZKckJaMk9HbmZYVmNIYlpoL3NMczN6MDExUFh0N2hrMjg3OTZOdUwxL3dMWk1uK3ZNWURTQnBXCnNHeGdVQ3FWYkFobS9IUmZpZWJtYVRXYTJOZUxOaHJvWkpWWmcvYjM4blV4b2xXSTkreGNuRERHemY0YUtsQnIKRmJTdHN2VituQnNYRHRNbjFuc1N4SlpiRk9ueXhXK0ZRZzNVZTdza0RxbHI1dXZrZS9MQVlaM1lxUT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBd2gvdFNJZUU5T0tNaVBtdThWT3pRVjNySlNqa3hNK0NMU05tN3JZQ1FhemppTVpGCm1tUVFnMWRoYk52Ri84M2hpTHkraWkvcDZxWi9qSWE1S3BIZmpGNWs3ZnlvdndzcmVWMVovRHQzMFE5VkhVZDYKTmV0VlhjcDIzalppeDF2UXI2d0FhNE9SWEEwNXQvUVpEYWEwZUVBcUNJK3lqQTZxMzgvMVdWUmdjaWZTN2tFYgpiWW4yNE1xN05iRW9IckY3Ym9rTTlIMGZhUC91OFpTYmh5UmxGOTRSQnBBMmJUQ2RZQld3YjNjL3ZKMndoWlJLCjRYYW9Xcm01ZXhEZHNFOENUcVIzZlNoeXJLMGFZY3lwNFdKMm5UZ0RqU1NheWVIejZiMjM0Mnc0QmszcGZud2YKRXY3bGpqcU9rdTRsQXVtM2JUc2h2STc2UGdka0ZpMzJ3UzRhWlFJREFRQUJBb0lCQUVockFhbURpTm1VSkVvNwpId1MvTFVtTzRGaHgrM25pVVpWR05qR0tLNmhWZDJLQVdObmlSM2kvNGNQcTd0L2hiYWdGaFcxbXQzUkduYUdPClpzaFhOOWFWSmtEVDl4Mmg3SnR2ZEZEUTNIOWNvV3QxVFVXTkg4RUg5VFVyZzhrTVd2c1dCdWdVNG1hOU5sR0cKR2N5S2FwdkxrQUsybkt4OEVrbkJPaTJUZVJGTVhEWUVzalRoa1JIdS9qWHZkRDREYmlkZVJOeGJXOEVaWDFOWQphOHJQQzB6cU52dGQ4YXI5UlBCQmt3WURsMkV2VkJEYUZlUWVFb0xNRWg0TFRaQ3Q5bEcrZ0U3UlNOb3pwNDRuCkRRb3NLOVcxbnNsaGFUK1Rjd3BJUGhtc28ybVpYcmQxVHJKbEw4dkFGSk5OZEFUbjJYWmZFdlM2WTVLWFFUM1AKcnRGVXFPY0NnWUVBejN5MngxS0lYRXRNVnJzZjJ2cStVQjMrWmZPNjVTcXNQYk8rRjZvQzFmaTB3SVdDbFVBSQo5dHdsVTRqbW91bUlJd0R1NEFKYWxqZC9MRUxUVXZvK3BLMFRvRGdDOFF6RWQvaDdkRG84OGthZ2dYOE1rcXFxClgyOVRDM2U5RXNPMUdORXdFdjFHWFJSRzhybENjTktqVHRQVDFIVlkrWHNSSW9MUG8yZUZ3RDhDZ1lFQTc0TmsKOUlrMGtSRGFEeXdncnhqTlBIMDQzY1VwSi91RlMxSGtmZzk2TWVJYTJaR0taQjJSeVFCVzRhVk1TbDRyRFdkWQp4R3dyZzZPbmtPWW5abGg2cXREY0ljdEdFR21WelF2S0N2U0NYazI2aGVjejJnOW91WEE0WUpuZjJ3NFRwRXdsCm4wNEtUejRiSDNZY3ZCM1k4NVFRSEpzWWZobDY5TzY3M3o5c1BGc0NnWUJUWE4wbTZqOEZMZSttN1JuWVptUHUKVm82dXNhVkdpOFdXS05CWU82TldDczI4aUNlMkJYdFVpNUNucGxwYjNBNHBXaWVmY3ZLb1pmVy9kNzNtR2NydgphT1o0dWVoY3B1K004QlhSMWRCRTJ5R0R4ZUxzVG91VE9td1lNR3lUekhQSFc4eS81R2pQM3VTK2dyWnlFLzh2CkhhWi9Od0tmZ2RXWmt3c1BzUGtwQ3dLQmdRRHU4MUI1NHBTdUVYanJZQ1B2YkRmOW5CUnF0RE9lTHdISnBoYm0KYVR5SW5jSVp3MmlrL3hjZHlCZmxvSXJmT3RtSzBzc3RrNWxLQ0xDNUQ5VEk5NGJSK2ZOVVI3OUx5bnJvQ1ZYMQozZ0JlWXYrdWJYNCtrOWJ3QW5STWM5ZHdiTGZOMXlaRnE0Ny9oYjk3Z05Pa0hjYi9JMzE3ZklSUDhjM0lwSkNNClpuTHVOd0tCZ1FETHN4T1Jwck45cE5odWV6akUwa2gvMDdBTU1sdExSUm05Zm9qNDlDU3ZhMTlmVVYyNGdYZ0oKQ0hWcTloZU9lTzI4ZDRrQ0wydUJtUktBSFJKbml3Q0wvdkNVV0IyaXhyS05FUXpnWGhnWUVPSTcyaHYzNndpNAozYjZNTUQ5Mk1IVHlEVEI1dExWbDBZNElqTlp1QXRiMFpHWHdTOVFWL1RmYTBaeTRicWJWSmc9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=
    
# kcm
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# cat /etc/kubernetes/controller-manager.conf
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURCVENDQWUyZ0F3SUJBZ0lJSXk0ODJreHY4b013RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TmpBeE1qUXdPVFV4TURkYUZ3MHpOakF4TWpJd09UVTJNRGRhTUJVeApFekFSQmdOVkJBTVRDbXQxWW1WeWJtVjBaWE13Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLCkFvSUJBUURZWkxueHAxUWFvMWNmOWVBNlZkalZwb2RFenpwMzk4d2QxWFBvd1VjQm1idzY1RFlWR0c5c1VlcVYKL042elFSREZJMmJQeUVTRDVJM092U0tvVGRWNS9rZ1RSaWFyTFlhYS84WENhTzM5Q05PamlmMjdaQVJYa2xjcwo5czJiODNOU1JycDJiNmJhNitHbkYwbUM1UXlPbE1uZVN6Uys3OFV2L3Jpdk9nQ2tkdzBCSUQvblRoTjdhR3lhCjQvQ1RJaVZCMkVoQXllT2FWSEVER3hLMk1ZNXhWL2lxVjBoVFh3V3B1NnhiZ1UxdlFlbml4dkIweVM3MjgyblEKajEydUpjdTFiUkcwWStDbFMzb1owKzBYRGY5dFNnNU54RTR6aWh2NjFpQVl3MnBTRzNJNU5CYVZXb0NtbkJVeAprbkJBbTRaZXJvNDRTVHZLU0pPMXd5ckQzcnluQWdNQkFBR2pXVEJYTUE0R0ExVWREd0VCL3dRRUF3SUNwREFQCkJnTlZIUk1CQWY4RUJUQURBUUgvTUIwR0ExVWREZ1FXQkJRYnhDdm8xL1k4UW5Bc3VhNG9aelVObjBGb3l6QVYKQmdOVkhSRUVEakFNZ2dwcmRXSmxjbTVsZEdWek1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQ2JkTUkwUGFyYgowbTZHdDRaNFJNRmtxR3d2T0VNdTlYZVRlTVVLanpXSklRVEUxSlpDajFGZnlHbWhBZ3NQaERQcVBabDZyMmFvCmlrOVdhbm12dmxkM3RGWElqbnRPM0ZyVmtOTGREKzYwVlVkMVNuL2ZvSk16VmJLNzlqNWlUUWs1aWJSNnRraXoKSFZnTktvTHlaYjk4SXpFOEdEcVQ2YnV2ZG9UM2xSMW5VWlNNVHJFazZLdXA4MDJjNGN3dUxHT0MrR2I1WUg2bwp0Z3d5d3gxY2hMcGx4VkJkSlBtSld2NHFxNm0vYXZlUmRLNm9jdXIzVWNraGJhYUVweWhwUmRDMDYzWXE2YjdCCnhvV1kvdGRRQzhMNEQ1SlVrSWx6TW9Cdzd5emJrVTJINER6QVYzMHBlakpSdVVSSStxQWFxSERvalBWenNzSG4KY3ByMGFTTlVIWGNnCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    server: https://192.168.10.100:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: system:kube-controller-manager
  name: system:kube-controller-manager@kubernetes
current-context: system:kube-controller-manager@kubernetes
kind: Config
preferences: {}
users:
- name: system:kube-controller-manager
  user:
    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURGakNDQWY2Z0F3SUJBZ0lJZEhxUTZHNEtWZ0l3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TmpBeE1qUXdPVFV4TURkYUZ3MHlOekF4TWpRd09UVTJNRGRhTUNreApKekFsQmdOVkJBTVRIbk41YzNSbGJUcHJkV0psTFdOdmJuUnliMnhzWlhJdGJXRnVZV2RsY2pDQ0FTSXdEUVlKCktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQU5Kb2E1aEZ5SEp4cGg4SmxrS0NRaTJVR1FoM2ZPQ3UKdDVCMldINldGTFJlUmhhTlBQekJ2WkttaWxqNzhBNUcwUlBBeWJqNXc5ZS9mNnA0OEhhVU5EZStMcW5EeHcrcgo2R1MzcVJpR0p3RllCUkFUaDQ3QnRnOUdFNG0ySXBudEFvd1RPSE5GSFRZU1Y2aXdIdDlRN1Y2NTk4eCtjNWQwCnhIT1ZmYkd2L1RXdXJhVlFJdkdMMDZ1Y3hWRElYZUt3NWtmbzlPdW9ucHV2NmlPODlJVG5KTGVjZXg3TmF2Q20KYlp2Tmo2SWZ0UXRiM2lQM0lwVDJGZExKT1VORElhTTcxT2x6ZEtSVE5XSUJoWjltYmRyMEVpdnZkT3dqOHduTwphUVRYOHo5VlE2dGgxeE9FeGZheVlRSkEyLzJxbEFtelQrZDdWYmVjdWpyYndSc2F5NUF5N2hFQ0F3RUFBYU5XCk1GUXdEZ1lEVlIwUEFRSC9CQVFEQWdXZ01CTUdBMVVkSlFRTU1Bb0dDQ3NHQVFVRkJ3TUNNQXdHQTFVZEV3RUIKL3dRQ01BQXdId1lEVlIwakJCZ3dGb0FVRzhRcjZOZjJQRUp3TExtdUtHYzFEWjlCYU1zd0RRWUpLb1pJaHZjTgpBUUVMQlFBRGdnRUJBTk55c095cGQ5WjJEN0NRenRJeVBjVU9XKzlVcmM5QVY2Z0thanU2WDJGaXRhYnhrZnMrClZBWEg4QUh6ZmxpTDk1blF6NlVBaG9RQk8yam9Cb3lPM0JWUTMxd1krVjlTQjhESnR2NXBUQURIV1NBNGZnU1oKdFJvT215RHZQaHdWUFFVcDFUcGcwMHBwL3VFZ29jK0NJWTNjYURuN1dsUGNrTHNYNzBvS3poWXpWbjFINk8yegpiNk5PY0pZL3ZlVUY2bXBmRUo1b0NWVVVxc2MzZ2kyQzNoekR5Q3E1UVJZU0pseURlemRodndqRmdDMFpBbGtLCkl5WENXZkJUZjVFMjk4Z1R2L2VOYkNwZkY2ck0wNWRVS29vcXZCSWp6SExNZ09GemxpOGRDV09ndWM5a0J5ME4KTnFERlZ4UWI1c01WZ1RuV0U4KzcyWW8xRnppQ0ltVlpyaFU9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb3dJQkFBS0NBUUVBMG1ocm1FWEljbkdtSHdtV1FvSkNMWlFaQ0hkODRLNjNrSFpZZnBZVXRGNUdGbzA4Ci9NRzlrcWFLV1B2d0RrYlJFOERKdVBuRDE3OS9xbmp3ZHBRME43NHVxY1BIRDZ2b1pMZXBHSVluQVZnRkVCT0gKanNHMkQwWVRpYllpbWUwQ2pCTTRjMFVkTmhKWHFMQWUzMUR0WHJuM3pINXpsM1RFYzVWOXNhLzlOYTZ0cFZBaQo4WXZUcTV6RlVNaGQ0ckRtUitqMDY2aWVtNi9xSTd6MGhPY2t0NXg3SHMxcThLWnRtODJQb2grMUMxdmVJL2NpCmxQWVYwc2s1UTBNaG96dlU2WE4wcEZNMVlnR0ZuMlp0MnZRU0srOTA3Q1B6Q2M1cEJOZnpQMVZEcTJIWEU0VEYKOXJKaEFrRGIvYXFVQ2JOUDUzdFZ0NXk2T3R2Qkd4ckxrREx1RVFJREFRQUJBb0lCQUJsbC8yN3ZKWVlqRCtGNQpQOGtoZmltUVVnRkNvekZnQmNxTGJwMUJNcGlmTktpdVBlbG8zYUJoT0J3THRXdVR3dE0ybDJNYnl6YzA1NDFGCmNnbHVWR3RTS3NIVlh5Y1dJa1JlSXl4UnJVMnRPVmM2ZEVlWVBJalZkYkJPNnhoWGt0SWowOUdlU3l0bXhXd1kKYm5HWWpENitCaHFLUFJ2UVBaS1NQZ3ovNkhuZE53eUg3RmlLNFpVK3NYckpGYTh3Qmw4YVR1U2dGZTg2Zm9qWQppSjNWVkNjNmdqazFjYzBEalRFMUNuTlZaNlVaL2VheXAxcmRocEdMNFcxZkxLNnUwbWJmK3JtTXpsVWF4WU9FCjdZQVl0WFo2SmNIMzZJYS9CeXZTVStsU2FzUU1UaHVqSWlqY0g5NFZDS2IzQm01SzJ0M05nY1hpT0xRSmsxVkgKYVRSZUhwRUNnWUVBNFZiWWZWelVmVVpvT3JKSUJkNGNYWVJoanhHYkxyTWFFVksrMjdPbWsvZFpSRDJmR2c3aQo5Q3V3K3JLbkNIWURabDdZN2N2Q2M0K1FJclJFajVacjIzZHZVYktIMEFKMUZReW5mcTF0Sk5kZzNsQjdDOE04CjJvaThpYmJnTmxGS3RkampxNVJSQXhPRHZsZDhHMHN2R09oS2xZb2pXR0dwVHFpak94Nkg0RlVDZ1lFQTd3bDYKQzZFYVNsSXJLbjZUZkZsd0E1a2x6TnNXdE5NL0NGL2tZMXpyZFlTcWx3ZkR5VmZtQnpjRlpBY0FkUm9aRzF3Zgp5a1c2RTF2RkM4WkhONHN4dXpITk04RXNNRVg3RjBoVi9YcnM2RitLSmJ1TGtib2J0RUQ0SGJtUnNCdVJpdXN0CmZNSkpSSDJ0RXVmWnRUOUVobk9rV3Jka2hmeDEzNUhkVWlBMUlzMENnWUVBcGM3QVo2WkoySkJaRzIrWm5XK3MKMFljYVBpckhWQnFIZ04yeEFIcDFoUVVKVXpSQWdPMFpSRzl0djFwN203Y3lrejRSUXhDZVdXZjJ1QUtMUEZpRApycTU0WTlZSkp4N1h4aEJVb3RxN3A5TXZQUVpkTSsrS05JZE9xOHE3dWx3Z3JDUVdpbWNOSVVWWHVGUXBSdkFRCmpMUklSVGFyQVZxRE9SVFBYeTM4N3kwQ2dZQnF5b0lTOWZ1SDNxUFlUVXBZMEtCQmkwY2UrWFp3Zkx2NVl0WG4KS2xrclhJVFdDcXNHcGRWbnZjWVR4U2tJS0F1MWRIZmpaemxWY3JkYXBrK2syZlB5M0xIL2dEcmNxamNlVkx2TwpEZ0FQWkxlVVdmQmx2NDZtL2l1YkpBK1piUWVkMTZtdnhpRHpqMjRtTnh6Rlk2bWFvOGwybWQ0NEdlMFRYOWhQCjI0SEJ0UUtCZ0ZFaEM1V1U1eFM1ZmRZVmpPdmI4TnRnTjJMdENSMXNFR01vd0trSmJScFRyNzI2ZGt2dDJ6TG8KYlhyampwcFFuUmZqUUJLNm5MemMzWC8yM3J4ZUUyVUVlZlBHNTNIeDJmcEZwMVh3cGtxWnY1SjBkSngyZGU1ZApuR0g3cEUzL1p5c3J3YVdsNmUrc3J4cUZTd1FaOWVkSUVLMVJjTUNlbTlieVVQRHhodlc4Ci0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg==
    
# scheduler
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# cat /etc/kubernetes/scheduler.conf
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURCVENDQWUyZ0F3SUJBZ0lJSXk0ODJreHY4b013RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TmpBeE1qUXdPVFV4TURkYUZ3MHpOakF4TWpJd09UVTJNRGRhTUJVeApFekFSQmdOVkJBTVRDbXQxWW1WeWJtVjBaWE13Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLCkFvSUJBUURZWkxueHAxUWFvMWNmOWVBNlZkalZwb2RFenpwMzk4d2QxWFBvd1VjQm1idzY1RFlWR0c5c1VlcVYKL042elFSREZJMmJQeUVTRDVJM092U0tvVGRWNS9rZ1RSaWFyTFlhYS84WENhTzM5Q05PamlmMjdaQVJYa2xjcwo5czJiODNOU1JycDJiNmJhNitHbkYwbUM1UXlPbE1uZVN6Uys3OFV2L3Jpdk9nQ2tkdzBCSUQvblRoTjdhR3lhCjQvQ1RJaVZCMkVoQXllT2FWSEVER3hLMk1ZNXhWL2lxVjBoVFh3V3B1NnhiZ1UxdlFlbml4dkIweVM3MjgyblEKajEydUpjdTFiUkcwWStDbFMzb1owKzBYRGY5dFNnNU54RTR6aWh2NjFpQVl3MnBTRzNJNU5CYVZXb0NtbkJVeAprbkJBbTRaZXJvNDRTVHZLU0pPMXd5ckQzcnluQWdNQkFBR2pXVEJYTUE0R0ExVWREd0VCL3dRRUF3SUNwREFQCkJnTlZIUk1CQWY4RUJUQURBUUgvTUIwR0ExVWREZ1FXQkJRYnhDdm8xL1k4UW5Bc3VhNG9aelVObjBGb3l6QVYKQmdOVkhSRUVEakFNZ2dwcmRXSmxjbTVsZEdWek1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQ2JkTUkwUGFyYgowbTZHdDRaNFJNRmtxR3d2T0VNdTlYZVRlTVVLanpXSklRVEUxSlpDajFGZnlHbWhBZ3NQaERQcVBabDZyMmFvCmlrOVdhbm12dmxkM3RGWElqbnRPM0ZyVmtOTGREKzYwVlVkMVNuL2ZvSk16VmJLNzlqNWlUUWs1aWJSNnRraXoKSFZnTktvTHlaYjk4SXpFOEdEcVQ2YnV2ZG9UM2xSMW5VWlNNVHJFazZLdXA4MDJjNGN3dUxHT0MrR2I1WUg2bwp0Z3d5d3gxY2hMcGx4VkJkSlBtSld2NHFxNm0vYXZlUmRLNm9jdXIzVWNraGJhYUVweWhwUmRDMDYzWXE2YjdCCnhvV1kvdGRRQzhMNEQ1SlVrSWx6TW9Cdzd5emJrVTJINER6QVYzMHBlakpSdVVSSStxQWFxSERvalBWenNzSG4KY3ByMGFTTlVIWGNnCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    server: https://192.168.10.100:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: system:kube-scheduler
  name: system:kube-scheduler@kubernetes
current-context: system:kube-scheduler@kubernetes
kind: Config
preferences: {}
users:
- name: system:kube-scheduler
  user:
    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUREVENDQWZXZ0F3SUJBZ0lJTzk5SUwxNFUzczR3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TmpBeE1qUXdPVFV4TURkYUZ3MHlOekF4TWpRd09UVTJNRGRhTUNBeApIakFjQmdOVkJBTVRGWE41YzNSbGJUcHJkV0psTFhOamFHVmtkV3hsY2pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCCkJRQURnZ0VQQURDQ0FRb0NnZ0VCQU9qUko1UzdVN0NxS3p2SHJBTXZGb29MUXZ0YXEyZXBySDZNVkwxZk1xNVAKVXZ5OWQ1NVozZmVuaVkySGp0NjBDSnp0S25WaWgyTkRGbnpHNDd1M1pYNURFaTFJU3A5cm03ZFhlU3MxOElUZApSYlVGY094bWJBT2pucDZJV21yQUxtUTgzVHpQazlEcllYZ0dPRVNQbWdGSW04bzY1RWtOdkVZd2V3dEhMWkdGCkxOYkVGcWVrc2I5dVp2OCszT1dFNEpOa3BFMjAwMDBHL1VxajFVMkw2ZElvZWtmc3ZZTGppWEpnU2dXZXZKL2gKdmhkWWRzOHNLcEZ5eVk0MmZEVkUwOXRFR3BJdmdKcUY2U3V1SkR2MFA1SS9wNFg3ck4yUmJ3T3ZYTlVTQUF5LwpkVkcxSUJsNjRwUk5lbWxyM0d4SjlqUkU5ZUo0cHlYVC9INk9wTGthRzVVQ0F3RUFBYU5XTUZRd0RnWURWUjBQCkFRSC9CQVFEQWdXZ01CTUdBMVVkSlFRTU1Bb0dDQ3NHQVFVRkJ3TUNNQXdHQTFVZEV3RUIvd1FDTUFBd0h3WUQKVlIwakJCZ3dGb0FVRzhRcjZOZjJQRUp3TExtdUtHYzFEWjlCYU1zd0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQgpBRlF5ZDN2Y0NMVlBJUW10WGRudlJjUTlYWkRUMVF4ZndRb2FLd2RYbG90OFFVdm5XMklHUWt4RUdVeWdyYlBsCjk3MGl2cU1zOEQ4NnNvMmV1NzVjcnJCL0J1NHJkS1hORStHc2ZhYittdFhBaGFycmxnNDlaQ0o1aklPY2RsM1UKTHh1c0VIOU9VOXRZVFJETEJ6N0J1TWhDM0dmcGM2TTFGTmlzMm1Zd2grNDlaVkNqNENPRStBVCtDc0I4aWNCYworRENTcENqTEltaDEyc1o0KzdVWTN1Nm5kRHo1NEJCWnVtVGhFSUkvdEtyZmxqM2JDSXh5Z3BMa3RKUUl4NnRoCjhzRFpVZktSSVI0NXFqaC9oVWtlNUVmTjMwOGxJTFdhdXUxZUpCcmg5MHNpdWlMekhkanNUc21FVTBQOHo2bEcKNHZ4b0ZaZitVTVBtZ3ZHUzk1Ny9OajQ9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K

# kubelet
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# cat /etc/kubernetes/kubelet.conf
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURCVENDQWUyZ0F3SUJBZ0lJSXk0ODJreHY4b013RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TmpBeE1qUXdPVFV4TURkYUZ3MHpOakF4TWpJd09UVTJNRGRhTUJVeApFekFSQmdOVkJBTVRDbXQxWW1WeWJtVjBaWE13Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLCkFvSUJBUURZWkxueHAxUWFvMWNmOWVBNlZkalZwb2RFenpwMzk4d2QxWFBvd1VjQm1idzY1RFlWR0c5c1VlcVYKL042elFSREZJMmJQeUVTRDVJM092U0tvVGRWNS9rZ1RSaWFyTFlhYS84WENhTzM5Q05PamlmMjdaQVJYa2xjcwo5czJiODNOU1JycDJiNmJhNitHbkYwbUM1UXlPbE1uZVN6Uys3OFV2L3Jpdk9nQ2tkdzBCSUQvblRoTjdhR3lhCjQvQ1RJaVZCMkVoQXllT2FWSEVER3hLMk1ZNXhWL2lxVjBoVFh3V3B1NnhiZ1UxdlFlbml4dkIweVM3MjgyblEKajEydUpjdTFiUkcwWStDbFMzb1owKzBYRGY5dFNnNU54RTR6aWh2NjFpQVl3MnBTRzNJNU5CYVZXb0NtbkJVeAprbkJBbTRaZXJvNDRTVHZLU0pPMXd5ckQzcnluQWdNQkFBR2pXVEJYTUE0R0ExVWREd0VCL3dRRUF3SUNwREFQCkJnTlZIUk1CQWY4RUJUQURBUUgvTUIwR0ExVWREZ1FXQkJRYnhDdm8xL1k4UW5Bc3VhNG9aelVObjBGb3l6QVYKQmdOVkhSRUVEakFNZ2dwcmRXSmxjbTVsZEdWek1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQ2JkTUkwUGFyYgowbTZHdDRaNFJNRmtxR3d2T0VNdTlYZVRlTVVLanpXSklRVEUxSlpDajFGZnlHbWhBZ3NQaERQcVBabDZyMmFvCmlrOVdhbm12dmxkM3RGWElqbnRPM0ZyVmtOTGREKzYwVlVkMVNuL2ZvSk16VmJLNzlqNWlUUWs1aWJSNnRraXoKSFZnTktvTHlaYjk4SXpFOEdEcVQ2YnV2ZG9UM2xSMW5VWlNNVHJFazZLdXA4MDJjNGN3dUxHT0MrR2I1WUg2bwp0Z3d5d3gxY2hMcGx4VkJkSlBtSld2NHFxNm0vYXZlUmRLNm9jdXIzVWNraGJhYUVweWhwUmRDMDYzWXE2YjdCCnhvV1kvdGRRQzhMNEQ1SlVrSWx6TW9Cdzd5emJrVTJINER6QVYzMHBlakpSdVVSSStxQWFxSERvalBWenNzSG4KY3ByMGFTTlVIWGNnCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    server: https://192.168.10.100:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: system:node:k8s-ctr
  name: system:node:k8s-ctr@kubernetes
current-context: system:node:k8s-ctr@kubernetes
kind: Config
preferences: {}
users:
- name: system:node:k8s-ctr
  user:
    client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
    client-key: /var/lib/kubelet/pki/kubelet-client-current.pem
    

(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# ls -l /var/lib/kubelet/pki
total 12
-rw-------. 1 root root 2822 Jan 24 18:56 kubelet-client-2026-01-24-18-56-11.pem
lrwxrwxrwx. 1 root root   59 Jan 24 18:56 kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2026-01-24-18-56-11.pem
-rw-r--r--. 1 root root 2262 Jan 24 18:56 kubelet.crt
-rw-------. 1 root root 1679 Jan 24 18:56 kubelet.

# kubelet 서버 역할 : Subjec, Key Usage, SAN 확인
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# cat /var/lib/kubelet/pki/kubelet.crt | openssl x509 -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 7292065219812151886 (0x65329eaeb1e2524e)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=k8s-ctr-ca@1769248570
        Validity
            Not Before: Jan 24 08:56:10 2026 GMT
            Not After : Jan 24 08:56:10 2027 GMT
        Subject: CN=k8s-ctr@1769248571
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:a8:2c:fa:06:90:a1:40:78:f4:b0:a6:e0:ea:1c:
                    ed:9d:ee:66:95:54:19:db:eb:ed:7b:31:3e:41:d4:
                    bb:19:36:a1:52:c6:9c:08:50:6c:55:d5:b7:e8:40:
                    a5:84:e1:43:92:d1:0a:2a:3d:82:bc:9c:99:47:08:
                    69:3a:73:0a:f0:1e:b9:2b:74:79:34:eb:87:41:91:
                    d7:af:e0:0c:0c:5a:42:44:a9:6a:e2:04:3f:74:99:
                    64:3a:66:f0:c2:16:c3:71:37:e8:f7:9e:b9:72:d3:
                    79:fd:90:2b:c9:fb:d4:db:66:86:4b:2e:77:b4:50:
                    f1:08:5e:d1:78:4a:de:fa:b7:da:37:f1:04:91:41:
                    1b:cd:f9:42:b7:d3:78:1c:7a:dc:d1:ef:50:c4:45:
                    de:42:11:3a:66:dd:46:45:10:3f:3d:bb:68:50:d0:
                    bd:02:8f:58:1c:14:c8:79:e6:c4:5e:c0:21:de:75:
                    3d:bc:f8:dc:7b:4c:41:c3:fc:0f:f3:58:81:f7:b6:
                    ad:42:ae:cb:3b:53:99:e3:22:3d:67:6e:9b:db:0a:
                    53:9a:19:d0:9d:7e:ed:27:75:ed:76:cb:ae:28:ce:
                    1e:91:c8:52:1d:75:a0:e5:dd:d8:7c:18:32:55:0a:
                    b2:20:47:cd:37:0d:be:e2:1c:0a:3d:f4:40:0e:de:
                    b1:a5
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Authority Key Identifier:
                08:C1:8D:45:1A:4C:42:6B:1C:B8:B6:65:E1:8F:81:28:55:94:D6:2B
            X509v3 Subject Alternative Name:
                DNS:k8s-ctr
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        73:3b:6a:c2:e8:0d:68:35:7a:3a:18:b5:b9:34:aa:0b:85:ac:
        73:b3:12:35:f7:4a:2e:98:08:0c:82:29:41:56:06:5b:27:a0:
        08:d0:ab:8d:86:78:f8:fa:b4:e1:af:54:bd:d0:97:fa:b6:9f:
        1f:9e:67:e4:5c:23:22:ec:11:c2:a2:56:4a:45:db:4d:b5:7f:
        49:f0:7e:aa:d6:03:e0:32:a5:f4:ed:e8:d6:fd:f2:85:87:b9:
        25:93:47:d8:33:d9:65:e9:36:61:bf:9a:34:3f:66:e2:42:6d:
        25:0b:99:13:c7:62:c5:97:d9:7c:b4:18:b4:8e:40:ff:6f:cb:
        98:03:8a:78:23:17:83:69:ec:c7:3b:0f:d7:cc:45:6f:f1:05:
        14:3d:f9:57:45:de:21:2f:ac:f7:ee:b2:87:e4:ee:75:f7:16:
        36:42:0a:d0:f5:d5:48:aa:8d:95:99:37:2f:57:48:49:88:f8:
        74:d0:4f:02:cd:a9:4c:23:74:af:dd:c2:93:ea:fc:4e:cc:23:
        3c:c9:f5:37:53:77:62:3e:b3:e9:a5:2b:ee:b3:59:10:7c:9c:
        6c:50:1e:e3:94:37:ae:1d:96:d2:86:7f:d5:a0:c8:24:fc:b8:
        1f:62:6e:38:78:49:9b:13:9f:76:ab:ce:a1:3f:18:18:4d:a3:
        b9:2b:e1:fd
        
# kubelet 클라이언트 역할 : Subjec, Key Usage 확인
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# cat /var/lib/kubelet/pki/kubelet-client-current.pem | openssl x509 -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 8262525664953040483 (0x72aa6350c712fe63)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kubernetes
        Validity
            Not Before: Jan 24 09:51:07 2026 GMT
            Not After : Jan 24 09:56:07 2027 GMT
        Subject: O=system:nodes, CN=system:node:k8s-ctr
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:cd:48:6b:aa:c0:2b:ab:9f:35:8d:db:81:53:12:
                    b9:7e:b4:db:7b:33:8d:5f:0d:8d:45:14:74:0c:41:
                    37:8b:be:98:3b:f7:84:76:77:3d:8a:34:40:4e:0a:
                    e1:1c:39:cc:1e:fc:4b:1b:3c:5b:d3:d9:ec:27:31:
                    b3:05:c8:2a:23:8e:4d:e1:e5:37:94:9f:15:30:01:
                    73:7c:a0:a6:06:eb:27:3f:09:11:04:b8:3d:48:4b:
                    a9:76:25:f6:52:f5:97:62:25:62:ea:78:3f:d7:8b:
                    07:c3:46:cc:e3:b6:ef:34:63:41:cf:9b:3d:b3:9e:
                    93:26:15:8d:0a:5a:35:27:3a:4f:f1:b3:d6:ef:32:
                    01:6c:ca:3c:e7:7b:4b:56:24:0f:8e:2d:14:f9:99:
                    9b:b3:83:90:80:21:9a:85:9d:4f:97:b1:9e:a4:6e:
                    6b:49:45:3c:f5:3d:5a:fc:e8:b7:5f:25:86:60:ce:
                    a4:e3:a9:0c:48:1a:03:e8:f3:be:00:fe:61:36:9a:
                    75:3f:d9:7d:ba:69:90:b0:e8:38:60:86:89:23:71:
                    ee:ff:ad:0a:60:1f:23:19:0d:06:35:0a:04:be:11:
                    0c:b6:be:ff:78:db:e3:8a:63:58:9e:99:45:66:cd:
                    b2:1a:77:2e:fc:93:4f:6e:ab:83:db:ca:9a:fb:48:
                    26:63
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Authority Key Identifier:
                1B:C4:2B:E8:D7:F6:3C:42:70:2C:B9:AE:28:67:35:0D:9F:41:68:CB
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        16:5a:81:34:d1:a0:99:70:f6:0f:53:eb:18:f2:fc:81:76:ad:
        7d:58:62:a9:34:00:f3:7e:94:d8:29:03:ca:67:35:2e:0b:1a:
        2e:15:2c:f0:75:30:49:b2:1f:1d:b0:ec:0d:ec:cd:7b:3b:24:
        4d:61:fe:85:6b:38:e8:69:b5:fb:25:32:3e:d9:0a:2c:1f:83:
        3b:e9:4a:8e:d3:46:04:1d:b2:99:67:87:bd:12:8b:e2:0f:ef:
        9e:83:71:f6:3d:c1:82:75:5d:ca:b1:60:51:9e:ff:c0:3d:d4:
        13:2f:3d:c4:91:60:ff:c8:8d:a8:f4:d4:7c:74:d4:1b:8c:69:
        0d:39:ce:d3:f2:14:63:60:47:85:b9:ae:26:b1:16:b9:16:76:
        22:e7:93:02:4c:60:17:50:b9:94:9b:28:2d:44:49:55:1c:a9:
        d5:1d:1d:ab:cc:08:04:62:be:89:a8:a4:75:17:4e:46:43:c2:
        ff:b8:1b:0e:27:17:e6:09:41:37:43:ee:cd:d2:90:90:27:cb:
        15:65:ff:2b:b7:93:fb:f5:12:10:5b:9a:6e:bc:f8:c9:4d:d7:
        d5:32:b8:7d:f0:78:39:de:df:34:50:ed:78:4e:62:8b:47:d7:
        c1:05:50:67:6f:32:e8:30:60:45:a4:e2:11:37:fa:3e:4e:93:
        7c:a6:22:89

4-7. [k8s-ctr] static pod 확인 : etcd, kube-apiserver, kube-scheduler,kube-controller-manager

# kubeket에 의해 기동되는 static pod 대상 매니페스트 디렉터리 확인
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# tree /etc/kubernetes/manifests/
/etc/kubernetes/manifests/
├── etcd.yaml
├── kube-apiserver.yaml
├── kube-controller-manager.yaml
└── kube-scheduler.yaml

1 directory, 4 files

# kublet 설정 확인
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# cat /var/lib/kubelet/config.yaml
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 0s
    cacheUnauthorizedTTL: 0s
cgroupDriver: systemd
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
containerRuntimeEndpoint: ""
cpuManagerReconcilePeriod: 0s
crashLoopBackOff: {}
evictionPressureTransitionPeriod: 0s
fileCheckFrequency: 0s
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 0s
imageMaximumGCAge: 0s
imageMinimumGCAge: 0s
kind: KubeletConfiguration
logging:
  flushFrequency: 0
  options:
    json:
      infoBufferSize: "0"
    text:
      infoBufferSize: "0"
  verbosity: 0
memorySwap: {}
nodeStatusReportFrequency: 0s
nodeStatusUpdateFrequency: 0s
rotateCertificates: true
runtimeRequestTimeout: 0s
shutdownGracePeriod: 0s
shutdownGracePeriodCriticalPods: 0s
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 0s
syncFrequency: 0s
volumeStatsAggPeriod: 0s

(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# cat /var/lib/kubelet/kubeadm-flags.env
KUBELET_KUBEADM_ARGS="--container-runtime-endpoint=unix:///run/containerd/containerd.sock --node-ip=192.168.10.100 --pod-infra-container-image=registry.k8s.io/pause:3.10"

# static 파드 확인
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kubectl get pod -n kube-system -owide
NAME                              READY   STATUS    RESTARTS   AGE   IP               NODE      NOMINATED NODE   READINESS GATES
coredns-668d6bf9bc-c2g8k          1/1     Running   0          56m   10.244.0.4       k8s-ctr   <none>           <none>
coredns-668d6bf9bc-qdwrj          1/1     Running   0          56m   10.244.0.5       k8s-ctr   <none>           <none>
etcd-k8s-ctr                      1/1     Running   0          56m   192.168.10.100   k8s-ctr   <none>           <none>
kube-apiserver-k8s-ctr            1/1     Running   1          56m   192.168.10.100   k8s-ctr   <none>           <none>
kube-controller-manager-k8s-ctr   1/1     Running   0          56m   192.168.10.100   k8s-ctr   <none>           <none>
kube-proxy-6gfjf                  1/1     Running   0          56m   192.168.10.100   k8s-ctr   <none>           <none>
kube-scheduler-k8s-ctr            1/1     Running   0          56m   192.168.10.100   k8s-ctr   <none>           <none>

# etcd : etcd client 는 https://192.168.10.100:2379 호출, metrics 은 http://127.0.0.1:2381 확인
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# tree /var/lib/etcd/
/var/lib/etcd/
└── member
    ├── snap
    │   └── db
    └── wal
        ├── 0000000000000000-0000000000000000.wal
        └── 0.tmp

4 directories, 3 files

# kube-apiserver
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# cat /etc/kubernetes/manifests/kube-apiserver.yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.10.100:6443
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=192.168.10.100
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=6443
    - --service-account-issuer=https://kubernetes.default.svc.cluster.local
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
    - --service-cluster-ip-range=10.96.0.0/16
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# ss -tnlp | grep apiserver
LISTEN 0      4096                *:6443             *:*    users:(("kube-apiserver",pid=72505,fd=3))


## k8s 내부에서 api 호출 시 : https://10.96.0.1 혹은 https://kubernetes.default.svc.cluster.local
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   61m

NAME                   ENDPOINTS             AGE
endpoints/kubernetes   192.168.10.100:6443   61m


# scheduler
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# cat /etc/kubernetes/manifests/kube-scheduler.yaml
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: kube-scheduler
    tier: control-plane
  name: kube-scheduler
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-scheduler
    - --authentication-kubeconfig=/etc/kubernetes/scheduler.conf
    - --authorization-kubeconfig=/etc/kubernetes/scheduler.conf
    - --bind-address=127.0.0.1
    - --kubeconfig=/etc/kubernetes/scheduler.conf
    - --leader-elect=true
    image: registry.k8s.io/kube-scheduler:v1.32.11
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 127.0.0.1
        path: /livez
        port: 10259
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    name: kube-scheduler
    readinessProbe:
      failureThreshold: 3
      httpGet:
        host: 127.0.0.1
        path: /readyz
        port: 10259
        scheme: HTTPS
      periodSeconds: 1
      timeoutSeconds: 15
    resources:
      requests:
        cpu: 100m
    startupProbe:
      failureThreshold: 24
      httpGet:
        host: 127.0.0.1
        path: /livez
        port: 10259
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    volumeMounts:
    - mountPath: /etc/kubernetes/scheduler.conf
      name: kubeconfig
      readOnly: true
  hostNetwork: true
  priority: 2000001000
  priorityClassName: system-node-critical
  securityContext:
    seccompProfile:
      type: RuntimeDefault
  volumes:
  - hostPath:
      path: /etc/kubernetes/scheduler.conf
      type: FileOrCreate
    name: kubeconfig
status: {}

## tcp 10259 Listen 포트 확인
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# ss -tnlp | grep scheduler
LISTEN 0      4096        127.0.0.1:10259      0.0.0.0:*    users:(("kube-scheduler",pid=72499,fd=3))

## scheduler 파드가 1개 이상일 경우 리더 역할 파드 확인
## Lease는 k8s의 경량 coordination 리소스 : 리더 선출 (Leader Election), 노드/컴포넌트 상태 heartbeat, 저부하(high-scale) 상태 갱신
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kubectl get leases.coordination.k8s.io -n kube-system kube-scheduler -o yaml
apiVersion: coordination.k8s.io/v1
kind: Lease
metadata:
  creationTimestamp: "2026-01-24T09:56:17Z"
  name: kube-scheduler
  namespace: kube-system
  resourceVersion: "5354"
  uid: 78d1cc93-33db-4eeb-abe7-b603fd081c2e
spec:
  acquireTime: "2026-01-24T09:56:17.179818Z"
  holderIdentity: k8s-ctr_3825883f-694e-45f3-a5d2-28094842db8e
  leaseDurationSeconds: 15
  leaseTransitions: 0
  renewTime: "2026-01-24T10:59:51.957723Z"
  
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kubectl get leases.coordination.k8s.io -n kube-system kube-scheduler
NAME             HOLDER                                         AGE
kube-scheduler   k8s-ctr_3825883f-694e-45f3-a5d2-28094842db8e   63m

## Node Heartbeat (Node 상태) : node heartbeat 전용 네임스페이스
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kubectl get lease -n kube-node-lease
NAME      HOLDER    AGE
k8s-ctr   k8s-ctr   64m

# kube-controller-manager
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# cat /etc/kubernetes/manifests/kube-controller-manager.yaml
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: kube-controller-manager
    tier: control-plane
  name: kube-controller-manager
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-controller-manager
    - --allocate-node-cidrs=true
    - --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --bind-address=127.0.0.1
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --cluster-cidr=10.244.0.0/16
    - --cluster-name=kubernetes
    - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
    - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
    - --controllers=*,bootstrapsigner,tokencleaner
    - --kubeconfig=/etc/kubernetes/controller-manager.conf
    - --leader-elect=true
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --root-ca-file=/etc/kubernetes/pki/ca.crt
    - --service-account-private-key-file=/etc/kubernetes/pki/sa.key
    - --service-cluster-ip-range=10.96.0.0/16
    - --use-service-account-credentials=true
    image: registry.k8s.io/kube-controller-manager:v1.32.11
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10257
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    name: kube-controller-manager
    resources:
      requests:
        cpu: 200m
    startupProbe:
      failureThreshold: 24
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10257
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    volumeMounts:
    - mountPath: /etc/ssl/certs
      name: ca-certs
      readOnly: true
    - mountPath: /etc/pki/ca-trust
      name: etc-pki-ca-trust
      readOnly: true
    - mountPath: /etc/pki/tls/certs
      name: etc-pki-tls-certs
      readOnly: true
    - mountPath: /usr/libexec/kubernetes/kubelet-plugins/volume/exec
      name: flexvolume-dir
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true
    - mountPath: /etc/kubernetes/controller-manager.conf
      name: kubeconfig
      readOnly: true
  hostNetwork: true
  priority: 2000001000
  priorityClassName: system-node-critical
  securityContext:
    seccompProfile:
      type: RuntimeDefault
  volumes:
  - hostPath:
      path: /etc/ssl/certs
      type: DirectoryOrCreate
    name: ca-certs
  - hostPath:
      path: /etc/pki/ca-trust
      type: DirectoryOrCreate
    name: etc-pki-ca-trust
  - hostPath:
      path: /etc/pki/tls/certs
      type: DirectoryOrCreate
    name: etc-pki-tls-certs
  - hostPath:
      path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec
      type: DirectoryOrCreate
    name: flexvolume-dir
  - hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: k8s-certs
  - hostPath:
      path: /etc/kubernetes/controller-manager.conf
      type: FileOrCreate
    name: kubeconfig
status: {}

## tcp 10257 Listen 포트 확인
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# ss -tnlp | grep controller
LISTEN 0      4096        127.0.0.1:10257      0.0.0.0:*    users:(("kube-controller",pid=72492,fd=3))

## 노드별 파드 CIDR 확인 
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.podCIDR}{"\n"}{end}'
k8s-ctr	10.244.0.0/24

## kcm 파드가 1개 이상일 경우 리더 역할 파드 확인
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kubectl get lease -n kube-system kube-controller-manager -o yaml
apiVersion: coordination.k8s.io/v1
kind: Lease
metadata:
  creationTimestamp: "2026-01-24T09:56:16Z"
  name: kube-controller-manager
  namespace: kube-system
  resourceVersion: "5571"
  uid: 01a84622-5150-45bf-b53b-fc748026a6e1
spec:
  acquireTime: "2026-01-24T09:56:16.556309Z"
  holderIdentity: k8s-ctr_89f0fb81-6ebf-4b54-992d-0b61acedafd3
  leaseDurationSeconds: 15
  leaseTransitions: 0
  renewTime: "2026-01-24T11:02:38.943222Z"
  
## 컨트롤러별 개별 ServiceAccount + RBAC 사용
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kubectl get sa -n kube-system | grep controller
attachdetach-controller                       0         66m
certificate-controller                        0         66m
clusterrole-aggregation-controller            0         66m
cronjob-controller                            0         66m
daemon-set-controller                         0         66m
deployment-controller                         0         66m
disruption-controller                         0         66m
endpoint-controller                           0         66m
endpointslice-controller                      0         66m
endpointslicemirroring-controller             0         66m
ephemeral-volume-controller                   0         66m
expand-controller                             0         66m
job-controller                                0         66m
namespace-controller                          0         66m
node-controller                               0         66m
pv-protection-controller                      0         66m
pvc-protection-controller                     0         66m
replicaset-controller                         0         66m
replication-controller                        0         66m
resourcequota-controller                      0         66m
service-account-controller                    0         66m
statefulset-controller                        0         66m
ttl-after-finished-controller                 0         66m
ttl-controller                                0         66m
validatingadmissionpolicy-status-controller   0         66m

4-8. 필수 애드온 설치 (coredns, kube-proxy) 확인

# coredns 확인
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kubectl get deploy -n kube-system coredns -owide
NAME      READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES                                    SELECTOR
coredns   2/2     2            2           67m   coredns      registry.k8s.io/coredns/coredns:v1.11.3   k8s-app=kube-dns


# kube-proxy 확인
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kubectl get ds -n kube-system -owide
NAME         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE   CONTAINERS   IMAGES                                SELECTOR
kube-proxy   1         1         1       1            1           kubernetes.io/os=linux   68m   kube-proxy   registry.k8s.io/kube-proxy:v1.32.11   k8s-app=kube-proxy

5. [Worker nodes] kubeadm 으로 k8s 클러스터 join → 확인

5-1. 사전 설정

## k8s-w2도 동일하게 설정

# root 권한(로그인 환경) 전환
root@k8s-w1:~# echo "sudo su -" >> /home/vagrant/.bashrc
root@k8s-w1:~# sudo su -
Last login: Sat Jan 24 20:05:05 KST 2026 on pts/1

# Time, NTP 설정
root@k8s-w1:~# timedatectl set-local-rtc 0

# 시스템 타임존(Timezone)을 한국(KST, UTC+9) 으로 설정 : 시스템 시간은 UTC 기준 유지, 표시만 KST로 변환
root@k8s-w1:~# timedatectl set-timezone Asia/Seoul

# SELinux 설정 : Kubernetes는 Permissive 권장
root@k8s-w1:~# setenforce 0

# 재부팅 시에도 Permissive 적용
root@k8s-w1:~# sed -i 's/^SELINUX=enforcing/SELINUX=permissive/' /etc/selinux/config

# firewalld(방화벽) 끄기
root@k8s-w1:~# systemctl disable --now firewalld

# Swap 비활성화
root@k8s-w1:~# swapoff -a

# 재부팅 시에도 'Swap 비활성화' 적용되도록 /etc/fstab에서 swap 라인 주석 처리
root@k8s-w1:~# sed -i '/swap/d' /etc/fstab

# 커널 모듈 로드
root@k8s-w1:~# modprobe overlay
root@k8s-w1:~# modprobe br_netfilter
root@k8s-w1:~# cat <<EOF | tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
overlay
br_netfilter

# 커널 파라미터 설정 : 네트워크 설정 - 브릿지 트래픽이 iptables를 거치도록 함
root@k8s-w1:~# cat <<EOF | tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1

# 설정 적용
root@k8s-w1:~# sysctl --system >/dev/null 2>&1

# hosts 설정

5-2. CRI 설치 containerd(runc) v2.1.5

# Docker 저장소 추가
root@k8s-w1:~# dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
Adding repo from: https://download.docker.com/linux/centos/docker-ce.repo


# containerd 설치
root@k8s-w1:~# dnf install -y containerd.io-2.1.5-1.el10
Docker CE Stable - aarch64                                   396  B/s | 2.0 kB     00:05
Package containerd.io-2.1.5-1.el10.aarch64 is already installed.
Dependencies resolved.
Nothing to do.
Complete!

# 기본 설정 생성 및 SystemdCgroup 활성화 (매우 중요)
root@k8s-w1:~# containerd config default | tee /etc/containerd/config.toml
version = 3
root = '/var/lib/containerd'
state = '/run/containerd'
temp = ''
disabled_plugins = []
required_plugins = []
oom_score = 0
imports = []
root@k8s-w1:~# sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml

# systemd unit 파일 최신 상태 읽기
root@k8s-w1:~# systemctl daemon-reload

# containerd start 와 enabled
root@k8s-w1:~# systemctl enable --now containerd

5-3. kubeadm, kubelet 및 kubectl 설치 v1.32.11

# repo 추가
root@k8s-w1:~# cat <<EOF | tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.32/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.32/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.32/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.32/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni

# 설치
root@k8s-w1:~# dnf install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
Kubernetes                                                   164  B/s | 1.7 kB     00:10
Package kubelet-1.32.11-150500.1.1.aarch64 is already installed.
Package kubeadm-1.32.11-150500.1.1.aarch64 is already installed.
Package kubectl-1.32.11-150500.1.1.aarch64 is already installed.
Dependencies resolved.
Nothing to do.
Complete!

# kubelet 활성화 (실제 기동은 kubeadm init 후에 시작됨)
root@k8s-w1:~# systemctl enable --now kubelet

# /etc/crictl.yaml 파일 작성
root@k8s-w1:~# cat << EOF > /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
EOF

5-4. kubeadm 으로 k8s join

# 기본 환경 정보 출력 저장
root@k8s-w1:~# crictl images
root@k8s-w1:~# crictl ps
root@k8s-w1:~# cat /etc/sysconfig/kubelet
root@k8s-w1:~# tree /etc/kubernetes  | tee -a etc_kubernetes-1.txt
root@k8s-w1:~# tree /var/lib/kubelet | tee -a var_lib_kubelet-1.txt
root@k8s-w1:~# tree /run/containerd/ -L 3 | tee -a run_containerd-1.txt
root@k8s-w1:~# pstree -alnp | tee -a pstree-1.txt
root@k8s-w1:~# systemd-cgls --no-pager | tee -a systemd-cgls-1.txt
root@k8s-w1:~# lsns | tee -a lsns-1.txt
root@k8s-w1:~# ip addr | tee -a ip_addr-1.txt 
root@k8s-w1:~# ss -tnlp | tee -a ss-1.txt
root@k8s-w1:~# df -hT | tee -a df-1.txt
root@k8s-w1:~# findmnt | tee -a findmnt-1.txt
root@k8s-w1:~# sysctl -a | tee -a sysctl-1.txt

# kubeadm Configuration 파일 작성
root@k8s-w1:~# NODEIP=$(ip -4 addr show enp0s9 | grep -oP '(?<=inet\s)\d+(\.\d+){3}')
root@k8s-w1:~# cat << EOF > kubeadm-join.yaml
apiVersion: kubeadm.k8s.io/v1beta4
kind: JoinConfiguration
discovery:
  bootstrapToken:
    token: "123456.1234567890123456"
    apiServerEndpoint: "192.168.10.100:6443"
    unsafeSkipCAVerification: true
nodeRegistration:
  criSocket: "unix:///run/containerd/containerd.sock"
  kubeletExtraArgs:
    - name: node-ip
      value: "$NODEIP"
EOF


oot@k8s-w1:~# kubeadm join --config="kubeadm-join.yaml"
[preflight] Running pre-flight checks
[preflight] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[preflight] Use 'kubeadm init phase upload-config --config your-config.yaml' to re-upload it.
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is healthy after 1.005192609s
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

# crictl 확인
root@k8s-w1:~# crictl images
IMAGE                                           TAG                 IMAGE ID            SIZE
ghcr.io/flannel-io/flannel-cni-plugin           v1.7.1-flannel1     127562bd9047f       5.14MB
ghcr.io/flannel-io/flannel                      v0.27.3             d84558c0144bc       33.1MB
registry.k8s.io/kube-proxy                      v1.32.11            dcdb790dc2bfe       27.6MB
registry.k8s.io/metrics-server/metrics-server   v0.8.0              bc6c1e09a843d       20.6MB
registry.k8s.io/pause                           3.10                afb61768ce381       268kB
root@k8s-w1:~# crictl ps
CONTAINER           IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID              POD                     NAMESPACE
5dcdfd48b8591       d84558c0144bc       17 seconds ago      Running             kube-flannel        0                   71bc4d12de4dd       kube-flannel-ds-gwnvf   kube-flannel
2da4cf462a2d7       dcdb790dc2bfe       18 seconds ago      Running             kube-proxy          0                   9e69b76d0cf87       kube-proxy-fz6s9        kube-system

# cluster-info cm 호출 가능 확인
root@k8s-w1:~# curl -s -k https://192.168.10.100:6443/api/v1/namespaces/kube-public/configmaps/cluster-info | jq
{
  "kind": "ConfigMap",
  "apiVersion": "v1",
  "metadata": {
    "name": "cluster-info",
    "namespace": "kube-public",
    "uid": "5c22db5f-a97c-4a31-a4c3-643b4fd74f74",
    "resourceVersion": "300",
    "creationTimestamp": "2026-01-24T09:56:15Z",
    "managedFields": [
      {
        "manager": "kubeadm",
        "operation": "Update",
        "apiVersion": "v1",
        "time": "2026-01-24T09:56:15Z",
        "fieldsType": "FieldsV1",
        "fieldsV1": {
          "f:data": {
            ".": {},
            "f:kubeconfig": {}
          }
        }
      },
     ...

5-5. [k8s-ctr] k8s-w1/w2 관련 정보 확인

# join 된 워커 노드 확인
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kubectl get node -owide
NAME      STATUS   ROLES           AGE    VERSION    INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                        KERNEL-VERSION                  CONTAINER-RUNTIME
k8s-ctr   Ready    control-plane   106m   v1.32.11   192.168.10.100   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.aarch64   containerd://2.1.5
k8s-w1    Ready    <none>          110s   v1.32.11   192.168.10.101   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.aarch64   containerd://2.1.5
k8s-w2    Ready    <none>          25s    v1.32.11   192.168.10.102   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.aarch64   containerd://2.1.5

# 노드별 파드 CIDR 확인 
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.podCIDR}{"\n"}{end}'
k8s-ctr	10.244.0.0/24
k8s-w1	10.244.1.0/24
k8s-w2	10.244.2.0/24

# 다른 노드의 파드 CIDR(Per Node Pod CIDR)에 대한 라우팅이 자동으로 커널 라우팅에 추가됨을 확인 : flannel.1 을 통해 VXLAN 통한 라우팅
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# ip -c route | grep flannel
10.244.1.0/24 via 10.244.1.0 dev flannel.1 onlink
10.244.2.0/24 via 10.244.2.0 dev flannel.1 onlink
10.244.3.0/24 via 10.244.3.0 dev flannel.1 onlink

# k8s-ctr 에서 10.244.1.0 IP로 통신 가능(vxlan overlay 사용) 확인
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# ping -c 1 10.244.1.0
PING 10.244.1.0 (10.244.1.0) 56(84) bytes of data.
64 bytes from 10.244.1.0: icmp_seq=1 ttl=64 time=1.01 ms

--- 10.244.1.0 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.006/1.006/1.006/0.000 ms

# 워커 노드에 Taints 정보 확인
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kc describe node k8s-w1
Name:               k8s-w1
Roles:              <none>
Labels:             beta.kubernetes.io/arch=arm64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=arm64
                    kubernetes.io/hostname=k8s-w1
                    kubernetes.io/os=linux
Annotations:        flannel.alpha.coreos.com/backend-data: {"VNI":1,"VtepMAC":"c2:65:5d:ff:bf:f5"}
                    flannel.alpha.coreos.com/backend-type: vxlan
                    flannel.alpha.coreos.com/kube-subnet-manager: true
                    flannel.alpha.coreos.com/public-ip: 192.168.10.101
                    kubeadm.alpha.kubernetes.io/cri-socket: unix:///run/containerd/containerd.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Sat, 24 Jan 2026 20:40:42 +0900
Taints:             <none>
Unschedulable:      false

# k8s-w1 노드에 배치된 파드 확인
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kubectl get pod -A -owide | grep k8s-w1
kube-flannel   kube-flannel-ds-gwnvf             1/1     Running   0          3m20s   192.168.10.101   k8s-w1    <none>           <none>
kube-system    kube-proxy-fz6s9                  1/1     Running   0          3m20s   192.168.10.101   k8s-w1    <none>

5-6. [k8s-w1/w2 노드 정보 확인, 기본 환경 정보 출력 비교, sysctl 변경 확인

# # kubelet 활성화 확인
root@k8s-w1:~# systemctl status kubelet --no-pager
● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/kubelet.service.d
             └─10-kubeadm.conf
     Active: active (running) since Sat 2026-01-24 20:40:43 KST; 3min 52s ago
 Invocation: fea023e4fa34432aa4685ad662845d69
       Docs: https://kubernetes.io/docs/
   Main PID: 89116 (kubelet)
      Tasks: 12 (limit: 12337)
     Memory: 29.6M (peak: 30.4M)
        CPU: 4.132s
     CGroup: /system.slice/kubelet.service
             └─89116 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubel…


# 기본 환경 정보 출력 저장
root@k8s-w1:~# cat /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS=
root@k8s-w1:~# tree /etc/kubernetes  | tee -a etc_kubernetes-2.txt
/etc/kubernetes
├── kubelet.conf
├── manifests
└── pki
    └── ca.crt

3 directories, 2 files
root@k8s-w1:~# cat /etc/kubernetes/kubelet.conf
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURCVENDQWUyZ0F3SUJBZ0lJSXk0ODJreHY4b013RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TmpBeE1qUXdPVFV4TURkYUZ3MHpOakF4TWpJd09UVTJNRGRhTUJVeApFekFSQmdOVkJBTVRDbXQxWW1WeWJtVjBaWE13Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLCkFvSUJBUURZWkxueHAxUWFvMWNmOWVBNlZkalZwb2RFenpwMzk4d2QxWFBvd1VjQm1idzY1RFlWR0c5c1VlcVYKL042elFSREZJMmJQeUVTRDVJM092U0tvVGRWNS9rZ1RSaWFyTFlhYS84WENhTzM5Q05PamlmMjdaQVJYa2xjcwo5czJiODNOU1JycDJiNmJhNitHbkYwbUM1UXlPbE1uZVN6Uys3OFV2L3Jpdk9nQ2tkdzBCSUQvblRoTjdhR3lhCjQvQ1RJaVZCMkVoQXllT2FWSEVER3hLMk1ZNXhWL2lxVjBoVFh3V3B1NnhiZ1UxdlFlbml4dkIweVM3MjgyblEKajEydUpjdTFiUkcwWStDbFMzb1owKzBYRGY5dFNnNU54RTR6aWh2NjFpQVl3MnBTRzNJNU5CYVZXb0NtbkJVeAprbkJBbTRaZXJvNDRTVHZLU0pPMXd5ckQzcnluQWdNQkFBR2pXVEJYTUE0R0ExVWREd0VCL3dRRUF3SUNwREFQCkJnTlZIUk1CQWY4RUJUQURBUUgvTUIwR0ExVWREZ1FXQkJRYnhDdm8xL1k4UW5Bc3VhNG9aelVObjBGb3l6QVYKQmdOVkhSRUVEakFNZ2dwcmRXSmxjbTVsZEdWek1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQ2JkTUkwUGFyYgowbTZHdDRaNFJNRmtxR3d2T0VNdTlYZVRlTVVLanpXSklRVEUxSlpDajFGZnlHbWhBZ3NQaERQcVBabDZyMmFvCmlrOVdhbm12dmxkM3RGWElqbnRPM0ZyVmtOTGREKzYwVlVkMVNuL2ZvSk16VmJLNzlqNWlUUWs1aWJSNnRraXoKSFZnTktvTHlaYjk4SXpFOEdEcVQ2YnV2ZG9UM2xSMW5VWlNNVHJFazZLdXA4MDJjNGN3dUxHT0MrR2I1WUg2bwp0Z3d5d3gxY2hMcGx4VkJkSlBtSld2NHFxNm0vYXZlUmRLNm9jdXIzVWNraGJhYUVweWhwUmRDMDYzWXE2YjdCCnhvV1kvdGRRQzhMNEQ1SlVrSWx6TW9Cdzd5emJrVTJINER6QVYzMHBlakpSdVVSSStxQWFxSERvalBWenNzSG4KY3ByMGFTTlVIWGNnCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    server: https://192.168.10.100:6443
  name: default-cluster
contexts:
- context:
    cluster: default-cluster
    namespace: default
    user: default-auth
  name: default-context
current-context: default-context
kind: Config
preferences: {}
users:
- name: default-auth
  user:
    client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
    client-key: /var/lib/kubelet/pki/kubelet-client-curre

6. 모니터링 툴 설치 : 프로메테우스-스택 설치 → 인증서 익스포터 설치 → 그라파나 대시보드 확인

6-1. metrics-server 설치

# metrics-server
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# helm upgrade --install metrics-server metrics-server/metrics-server --set 'args[0]=--kubelet-insecure-tls' -n kube-system

# 확인
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kubectl top node
NAME      CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)
k8s-ctr   175m         4%       816Mi           29%
k8s-w1    31m          1%       374Mi           21%
k8s-w2    34m          1%       344Mi           19%

(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kubectl top pod -A --sort-by='cpu'
NAMESPACE      NAME                              CPU(cores)   MEMORY(bytes)
kube-system    kube-apiserver-k8s-ctr            57m          202Mi
kube-system    etcd-k8s-ctr                      36m          43Mi
kube-system    kube-controller-manager-k8s-ctr   25m          54Mi
kube-system    kube-scheduler-k8s-ctr            15m          23Mi
kube-flannel   kube-flannel-ds-mqn8t             10m          13Mi
kube-flannel   kube-flannel-ds-gwnvf             10m          13Mi
kube-flannel   kube-flannel-ds-t2tkq             9m           13Mi
kube-system    metrics-server-5dd7b49d79-dqzjl   4m           19Mi
kube-system    coredns-668d6bf9bc-c2g8k          3m           16Mi
kube-system    coredns-668d6bf9bc-qdwrj          3m           16Mi
kube-system    kube-proxy-fz6s9                  2m           16Mi
kube-system    kube-proxy-6gfjf                  2m           16Mi
kube-system    kube-proxy-9cgdt                  1m           16Mi

6-2. kube-prometheus-stack 설치

# repo 추가
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
"prometheus-community" has been added to your repositories

# 파라미터 파일 생성
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# cat <<EOT > monitor-values.yaml
prometheus:
  prometheusSpec:
    scrapeInterval: "20s"
    evaluationInterval: "20s"
    externalLabels:
      cluster: "myk8s-cluster"
  service:
    type: NodePort
    nodePort: 30001

grafana:
  defaultDashboardsTimezone: Asia/Seoul
  adminPassword: prom-operator
  service:
    type: NodePort
    nodePort: 30002

alertmanager:
  enabled: true
defaultRules:
  create: true

kubeProxy:
  enabled: false
prometheus-windows-exporter:
  prometheus:
    monitor:
      enabled: false
EOT


# 배포
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack --version 80.13.3 -f monitor-values.yaml --create-namespace --namespace monitoring

(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# helm list -n monitoring
NAME                 	NAMESPACE 	REVISION	UPDATED                                	STATUS	CHART                        	APP VERSION
kube-prometheus-stack	monitoring	1       	2026-01-24 21:05:06.688856739 +0900 KST	failed	kube-prometheus-stack-80.13.3	v0.87.1

(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kubectl get pod,svc,ingress,pvc -n monitoring
NAME                                                            READY   STATUS              RESTARTS   AGE
pod/alertmanager-kube-prometheus-stack-alertmanager-0           0/2     Init:0/1            0          28s
pod/kube-prometheus-stack-admission-patch-5r4p6                 0/1     ContainerCreating   0          62s
pod/kube-prometheus-stack-grafana-5cb7c586f9-vwsr8              0/3     ContainerCreating   0          63s
pod/kube-prometheus-stack-kube-state-metrics-7846957b5b-rl2ck   1/1     Running             0          63s
pod/kube-prometheus-stack-operator-584f446c98-v2qhc             1/1     Running             0          63s
pod/kube-prometheus-stack-prometheus-node-exporter-854fb        1/1     Running             0          63s
pod/kube-prometheus-stack-prometheus-node-exporter-9wfls        1/1     Running             0          63s
pod/kube-prometheus-stack-prometheus-node-exporter-g7qm6        1/1     Running             0          63s
pod/prometheus-kube-prometheus-stack-prometheus-0               0/2     Init:0/1            0          28s

NAME                                                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
service/alertmanager-operated                            ClusterIP   None            <none>        9093/TCP,9094/TCP,9094/UDP      28s
service/kube-prometheus-stack-alertmanager               ClusterIP   10.96.224.200   <none>        9093/TCP,8080/TCP               64s
service/kube-prometheus-stack-grafana                    NodePort    10.96.113.15    <none>        80:30002/TCP                    64s
service/kube-prometheus-stack-kube-state-metrics         ClusterIP   10.96.132.129   <none>        8080/TCP                        64s
service/kube-prometheus-stack-operator                   ClusterIP   10.96.238.79    <none>        443/TCP                         64s
service/kube-prometheus-stack-prometheus                 NodePort    10.96.55.234    <none>        9090:30001/TCP,8080:31632/TCP   64s
service/kube-prometheus-stack-prometheus-node-exporter   ClusterIP   10.96.13.3      <none>        9100/TCP                        64s
service/prometheus-operated                              ClusterIP   None            <none>        9090/TCP                        28s

# 각각 웹 접속 실행 : NodePort 접속
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# open http://192.168.10.100:30001 # prometheus
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# open http://192.168.10.100:30002 # grafana : 접속 계정 admin / prom-operator

6-3. kube-controller-manager, etcd, kube-scheduler 메트릭 수집 설정

# kube-controller-manager bind-address 127.0.0.1 => 0.0.0.0 변경
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# sed -i 's|--bind-address=127.0.0.1|--bind-address=0.0.0.0|g' /etc/kubernetes/manifests/kube-controller-manager.yaml
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# cat /etc/kubernetes/manifests/kube-controller-manager.yaml | grep bind-address
    - --bind-address=0.0.0.0

# kube-scheduler bind-address 127.0.0.1 => 0.0.0.0 변경
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# sed -i 's|--bind-address=127.0.0.1|--bind-address=0.0.0.0|g' /etc/kubernetes/manifests/kube-scheduler.yaml
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# cat /etc/kubernetes/manifests/kube-scheduler.yaml | grep bind-address
    - --bind-address=0.0.0.0
    
# etcd metrics-url(http) 127.0.0.1 에 192.168.10.100 추가
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# sed -i 's|--listen-metrics-urls=http://127.0.0.1:2381|--listen-metrics-urls=http://127.0.0.1:2381,http://192.168.10.100:2381|g' /etc/kubernetes/manifests/etcd.yaml
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# cat /etc/kubernetes/manifests/etcd.yaml | grep listen-metrics-urls
    - --listen-metrics-urls=http://127.0.0.1:2381,http://192.168.10.100:2381

6-4. k8s 인증서 위치 확인

# Check certificates expiration for a Kubernetes cluster
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kubeadm certs check-expiration
[check-expiration] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[check-expiration] Use 'kubeadm init phase upload-config --config your-config.yaml' to re-upload it.

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Jan 24, 2027 09:56 UTC   364d            ca                      no
apiserver                  Jan 24, 2027 09:56 UTC   364d            ca                      no
apiserver-etcd-client      Jan 24, 2027 09:56 UTC   364d            etcd-ca                 no
apiserver-kubelet-client   Jan 24, 2027 09:56 UTC   364d            ca                      no
controller-manager.conf    Jan 24, 2027 09:56 UTC   364d            ca                      no
etcd-healthcheck-client    Jan 24, 2027 09:56 UTC   364d            etcd-ca                 no
etcd-peer                  Jan 24, 2027 09:56 UTC   364d            etcd-ca                 no
etcd-server                Jan 24, 2027 09:56 UTC   364d            etcd-ca                 no
front-proxy-client         Jan 24, 2027 09:56 UTC   364d            front-proxy-ca          no
scheduler.conf             Jan 24, 2027 09:56 UTC   364d            ca                      no
super-admin.conf           Jan 24, 2027 09:56 UTC   364d            ca                      no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Jan 22, 2036 09:56 UTC   9y              no
etcd-ca                 Jan 22, 2036 09:56 UTC   9y              no
front-proxy-ca          Jan 22, 2036 09:56 UTC   9y              no

# 위 kubelet.conf 에 대한 인증서/키 파일 위치 : worker 노드 동일
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# tree /var/lib/kubelet/pki/
/var/lib/kubelet/pki/
├── kubelet-client-2026-01-24-18-56-11.pem
├── kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2026-01-24-18-56-11.pem
├── kubelet.crt
└── kubelet.key

1 directory, 4 files

6-5. x509 certificate exporter 설치

# w1/w2 에 node label 설정
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kubectl label node k8s-w1 worker="true" --overwrite
node/k8s-w1 labeled
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kubectl label node k8s-w2 worker="true" --overwrite
node/k8s-w2 labeled

(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# cat << EOF > cert-export-values.yaml
# -- hostPaths Exporter
hostPathsExporter:
  hostPathVolumeType: Directory

  daemonSets:
    cp:
      nodeSelector:
        node-role.kubernetes.io/control-plane: ""
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/control-plane
        operator: Exists
      watchFiles:
      - /var/lib/kubelet/pki/kubelet-client-current.pem
      - /var/lib/kubelet/pki/kubelet.crt
      - /etc/kubernetes/pki/apiserver.crt
      - /etc/kubernetes/pki/apiserver-etcd-client.crt
      - /etc/kubernetes/pki/apiserver-kubelet-client.crt
      - /etc/kubernetes/pki/ca.crt
      - /etc/kubernetes/pki/front-proxy-ca.crt
      - /etc/kubernetes/pki/front-proxy-client.crt
      - /etc/kubernetes/pki/etcd/ca.crt
      - /etc/kubernetes/pki/etcd/healthcheck-client.crt
      - /etc/kubernetes/pki/etcd/peer.crt
      - /etc/kubernetes/pki/etcd/server.crt
      watchKubeconfFiles:
      - /etc/kubernetes/admin.conf
      - /etc/kubernetes/controller-manager.conf
      - /etc/kubernetes/scheduler.conf

    nodes:
      nodeSelector:
        worker: "true"
      watchFiles:
      - /var/lib/kubelet/pki/kubelet-client-current.pem
      - /etc/kubernetes/pki/ca.crt

prometheusServiceMonitor:
  create: true
  scrapeInterval: 15s
  scrapeTimeout: 10s
EOF

# helm chart 설치
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# helm repo add enix https://charts.enix.io
"enix" has been added to your repositories
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# helm install x509-certificate-exporter enix/x509-certificate-exporter -n monitoring --values cert-export-values.yaml
NAME: x509-certificate-exporter
LAST DEPLOYED: Sat Jan 24 21:18:08 2026
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
TEST SUITE: None

# 설치 확인
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# helm list -n monitoring
NAME                     	NAMESPACE 	REVISION	UPDATED                                	STATUS  	CHART                           	APP VERSION
kube-prometheus-stack    	monitoring	1       	2026-01-24 21:05:06.688856739 +0900 KST	failed  	kube-prometheus-stack-80.13.3   	v0.87.1
x509-certificate-exporter	monitoring	1       	2026-01-24 21:18:08.623833792 +0900 KST	deployed	x509-certificate-exporter-3.19.1	3.19.1

## x509 대시보드 추가 : grafana sidecar 컨테이너가 configmap 확인 후 추가
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kubectl get cm -n monitoring x509-certificate-exporter-dashboard
NAME                                  DATA   AGE
x509-certificate-exporter-dashboard   1      41s

# 데몬셋 확인 : cp, nodes 각각
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kubectl get ds -n monitoring -l app.kubernetes.io/instance=x509-certificate-exporter
NAME                              DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                            AGE
x509-certificate-exporter-cp      1         1         1       1            1           node-role.kubernetes.io/control-plane=   57s
x509-certificate-exporter-nodes   2         2         2       2            2           worker=true                              57s

# 파드 정보 확인 : IP 확인
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kubectl get pod -n monitoring -l app.kubernetes.io/instance=x509-certificate-exporter -owide
NAME                                    READY   STATUS    RESTARTS   AGE   IP           NODE      NOMINATED NODE   READINESS GATES
x509-certificate-exporter-cp-l7b97      1/1     Running   0          71s   10.244.0.6   k8s-ctr   <none>           <none>
x509-certificate-exporter-nodes-pgdpf   1/1     Running   0          71s   10.244.2.6   k8s-w2    <none>           <none>
x509-certificate-exporter-nodes-rwdrb   1/1     Running   0          71s   10.244.1.7   k8s-w1    <none>           <none>

# 프로메테우스 서비스모니터 수집을 위한 Service(ClusterIP) 정보 확인
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kubectl get svc,ep -n monitoring x509-certificate-exporter
NAME                                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/x509-certificate-exporter   ClusterIP   10.96.189.118   <none>        9793/TCP   86s

NAME                                  ENDPOINTS                                         AGE
endpoints/x509-certificate-exporter   10.244.0.6:9793,10.244.1.7:9793,10.244.2.6:9793   86s

# 컨트롤플레인 노드에 배포된 'x509 익스포터' 파드에 메트릭 호출 확인
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# curl -s 10.244.0.6:9793/metrics | grep '^x509' | head -n 3
x509_cert_expired{filename="apiserver-etcd-client.crt",filepath="/etc/kubernetes/pki/apiserver-etcd-client.crt",issuer_CN="etcd-ca",serial_number="6752242067453093961",subject_CN="kube-apiserver-etcd-client"} 0
x509_cert_expired{filename="apiserver.crt",filepath="/etc/kubernetes/pki/apiserver.crt",issuer_CN="kubernetes",serial_number="6316045001739227051",subject_CN="kube-apiserver"} 0
x509_cert_expired{filename="ca.crt",filepath="/etc/kubernetes/pki/ca.crt",issuer_CN="kubernetes",serial_number="2535030548539110019",subject_CN="kubernetes"} 0

# 워커 노드에 배포된 'x509 익스포터' 파드에 메트릭 호출 확인
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# curl -s 10.244.1.7:9793/metrics | grep '^x509' | head -n 3
x509_cert_expired{filename="ca.crt",filepath="/etc/kubernetes/pki/ca.crt",issuer_CN="kubernetes",serial_number="2535030548539110019",subject_CN="kubernetes"} 0
x509_cert_expired{filename="kubelet-client-current.pem",filepath="/var/lib/kubelet/pki/kubelet-client-current.pem",issuer_CN="kubernetes",serial_number="305345650348149891832660344899669548281",subject_CN="system:node:k8s-w1",subject_O="system:nodes"} 0
x509_cert_not_after{filename="ca.crt",filepath="/etc/kubernetes/pki/ca.crt",issuer_CN="kubernetes",serial_number="2535030548539110019",subject_CN="kubernetes"} 2.084608567e+09

# 프로메테우스 CR 정보 확인 : 서비스모니터와 룰 셀렉터 정보 확인
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kubectl get prometheuses.monitoring.coreos.com -n monitoring -o yaml

# helm 배포 시, label 추가 해둠
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kubectl edit servicemonitors -n monitoring x509-certificate-exporter
...
  labels:
    app.kubernetes.io/instance: x509-certificate-exporter
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: x509-certificate-exporter
    app.kubernetes.io/version: 3.19.1
    helm.sh/chart: x509-certificate-exporter-3.19.1
    release: kube-prometheus-stack
...

(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kubectl get prometheusrules.monitoring.coreos.com -n monitoring x509-certificate-exporter -o yaml | head -n 20

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  ...
  labels:
    app.kubernetes.io/instance: x509-certificate-exporter
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: x509-certificate-exporter
    app.kubernetes.io/version: 3.19.1
    helm.sh/chart: x509-certificate-exporter-3.19.1
    release: kube-prometheus-stack
  name: x509-certificate-exporter
...

6-6. 프로메테우스 확인

서비스모니터에 의해 수집된 타겟 정보 확인

6-7. 그라파나 확인

7. 샘플 애플리케이션 배포

7-1. 샘플 애플리케이션 배포

# 샘플 애플리케이션 배포
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# cat << EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webpod
spec:
  replicas: 2
  selector:
    matchLabels:
      app: webpod
  template:
    metadata:
      labels:
        app: webpod
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - sample-app
            topologyKey: "kubernetes.io/hostname"
      containers:
      - name: webpod
        image: traefik/whoami
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: webpod
  labels:
    app: webpod
spec:
  selector:
    app: webpod
  ports:
  - protocol: TCP
EOFype: ClusterIP0
deployment.apps/webpod created
service/webpod created

7-2. 애플리케이션 확인 & 반복 호출

# 배포 확인
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kubectl get deploy,svc,ep webpod -owide
NAME                     READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES           SELECTOR
deployment.apps/webpod   0/2     2            0           24s   webpod       traefik/whoami   app=webpod

NAME             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE   SELECTOR
service/webpod   ClusterIP   10.96.156.154   <none>        80/TCP    24s   app=webpod

NAME               ENDPOINTS   AGE
endpoints/webpod   <none>      24s

# webpod service clusterip 변수 지정
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# SVCIP=$(kubectl get svc webpod -o jsonpath='{.spec.clusterIP}')
echo $SVCIP
10.96.156.154

# 통신 확인
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# curl -s $SVCIP
Hostname: webpod-697b545f57-wb9hd
IP: 127.0.0.1
IP: ::1
IP: 10.244.1.8
IP: fe80::8c06:f6ff:fe5c:f521
RemoteAddr: 10.244.0.0:20714
GET / HTTP/1.1
Host: 10.96.156.154
User-Agent: curl/8.9.1
Accept: */*

8. kubeadm 인증서 갱신

8-1. 현재 인증서 정보 확인 : 인증서 파일 생성일 → 만료일 확인

# Check certificates expiration for a Kubernetes cluster
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kc describe cm -n kube-system kubeadm-config | grep -i cert
caCertificateValidityPeriod: 87600h0m0s
certificateValidityPeriod: 8760h0m0s
certificatesDir: /etc/kubernetes/pki


 현재 인증서가 UTC 기준 1월 24일 09:56 분 생성되어서, 유효기간 365일(1년) 이후 만료일은 '27년 1월 24일 09:55분.
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kubeadm certs check-expiration -v 6
I0124 21:28:27.266056  117213 loader.go:402] Config loaded from file:  /etc/kubernetes/admin.conf
I0124 21:28:27.266668  117213 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I0124 21:28:27.266688  117213 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
I0124 21:28:27.266693  117213 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I0124 21:28:27.266697  117213 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
[check-expiration] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[check-expiration] Use 'kubeadm init phase upload-config --config your-config.yaml' to re-upload it.
I0124 21:28:27.274926  117213 round_trippers.go:560] GET https://192.168.10.100:6443/api/v1/namespaces/kube-system/configmaps/kubeadm-config?timeout=10s 200 OK in 7 milliseconds
I0124 21:28:27.275676  117213 kubeproxy.go:55] attempting to download the KubeProxyConfiguration from ConfigMap "kube-proxy"
I0124 21:28:27.277450  117213 round_trippers.go:560] GET https://192.168.10.100:6443/api/v1/namespaces/kube-system/configmaps/kube-proxy?timeout=10s 200 OK in 1 milliseconds
I0124 21:28:27.278787  117213 kubelet.go:74] attempting to download the KubeletConfiguration from ConfigMap "kubelet-config"
I0124 21:28:27.280968  117213 round_trippers.go:560] GET https://192.168.10.100:6443/api/v1/namespaces/kube-system/configmaps/kubelet-config?timeout=10s 200 OK in 1 milliseconds
I0124 21:28:27.282732  117213 loader.go:402] Config loaded from file:  /etc/kubernetes/kubelet.conf
I0124 21:28:27.284115  117213 cert_rotation.go:140] Starting client certificate rotation controller
I0124 21:28:27.291268  117213 round_trippers.go:560] POST https://192.168.10.100:6443/apis/authentication.k8s.io/v1/selfsubjectreviews?timeout=10s 201 Created in 7 milliseconds
I0124 21:28:27.293635  117213 round_trippers.go:560] GET https://192.168.10.100:6443/api/v1/nodes/k8s-ctr?timeout=10s 200 OK in 2 milliseconds
I0124 21:28:27.301805  117213 round_trippers.go:560] GET https://192.168.10.100:6443/api/v1/namespaces/kube-system/pods?fieldSelector=spec.nodeName%3Dk8s-ctr&labelSelector=component%3Dkube-apiserver%2Ctier%3Dcontrol-plane 200 OK in 6 milliseconds

I0124 21:28:27.303023  117213 certs.go:360] Overriding the cluster certificate directory with the value from command line flag --cert-dir: /etc/kubernetes/pki
I0124 21:28:27.303375  117213 certs.go:473] validating certificate period for CA certificate
I0124 21:28:27.303874  117213 loader.go:402] Config loaded from file:  /etc/kubernetes/admin.conf
I0124 21:28:27.304200  117213 certs.go:473] validating certificate period for etcd CA certificate
I0124 21:28:27.304793  117213 loader.go:402] Config loaded from file:  /etc/kubernetes/controller-manager.conf
I0124 21:28:27.305372  117213 certs.go:473] validating certificate period for front-proxy CA certificate
I0124 21:28:27.306284  117213 loader.go:402] Config loaded from file:  /etc/kubernetes/scheduler.conf
I0124 21:28:27.307746  117213 loader.go:402] Config loaded from file:  /etc/kubernetes/super-admin.conf
CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Jan 24, 2027 09:56 UTC   364d            ca                      no
apiserver                  Jan 24, 2027 09:56 UTC   364d            ca                      no
apiserver-etcd-client      Jan 24, 2027 09:56 UTC   364d            etcd-ca                 no
apiserver-kubelet-client   Jan 24, 2027 09:56 UTC   364d            ca                      no
controller-manager.conf    Jan 24, 2027 09:56 UTC   364d            ca                      no
etcd-healthcheck-client    Jan 24, 2027 09:56 UTC   364d            etcd-ca                 no
etcd-peer                  Jan 24, 2027 09:56 UTC   364d            etcd-ca                 no
etcd-server                Jan 24, 2027 09:56 UTC   364d            etcd-ca                 no
front-proxy-client         Jan 24, 2027 09:56 UTC   364d            front-proxy-ca          no
scheduler.conf             Jan 24, 2027 09:56 UTC   364d            ca                      no
super-admin.conf           Jan 24, 2027 09:56 UTC   364d            ca                      no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Jan 22, 2036 09:56 UTC   9y              no
etcd-ca                 Jan 22, 2036 09:56 UTC   9y              no
front-proxy-ca          Jan 22, 2036 09:56 UTC   9y              no

8-2. 수동 인증서 갱신 실행

# 샘플 애플리케이션 반복 호출(신규 터미널)
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# SVCIP=$(kubectl get svc webpod -o jsonpath='{.spec.clusterIP}')
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# while true; do curl -s $SVCIP | grep Hostname; sleep 1; done
Hostname: webpod-697b545f57-wb9hd
Hostname: webpod-697b545f57-wb9hd
Hostname: webpod-697b545f57-8ld9r

# 사전 백업 : HA Controlplane 모두
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# cp -r /etc/kubernetes/pki /etc/kubernetes/pki.backup.$(date +%F)
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# ls -l /etc/kubernetes/pki.backup.$(date +%F)
total 56
-rw-r--r--. 1 root root 1281 Jan 24 21:31 apiserver.crt
-rw-r--r--. 1 root root 1123 Jan 24 21:31 apiserver-etcd-client.crt
-rw-------. 1 root root 1675 Jan 24 21:31 apiserver-etcd-client.key
-rw-------. 1 root root 1675 Jan 24 21:31 apiserver.key
-rw-r--r--. 1 root root 1176 Jan 24 21:31 apiserver-kubelet-client.crt
-rw-------. 1 root root 1675 Jan 24 21:31 apiserver-kubelet-client.key
-rw-r--r--. 1 root root 1107 Jan 24 21:31 ca.crt
-rw-------. 1 root root 1675 Jan 24 21:31 ca.key
drwxr-xr-x. 2 root root  162 Jan 24 21:31 etcd
-rw-r--r--. 1 root root 1123 Jan 24 21:31 front-proxy-ca.crt
-rw-------. 1 root root 1675 Jan 24 21:31 front-proxy-ca.key
-rw-r--r--. 1 root root 1119 Jan 24 21:31 front-proxy-client.crt
-rw-------. 1 root root 1675 Jan 24 21:31 front-proxy-client.key
-rw-------. 1 root root 1679 Jan 24 21:31 sa.key
-rw-------. 1 root root  451 Jan 24 21:31 sa.pub
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~#
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# mkdir /etc/kubernetes/backup-conf.$(date +%F)
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# cp /etc/kubernetes/*.conf /etc/kubernetes/backup-conf.$(date +%F)
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# ls -l /etc/kubernetes/backup-conf.$(date +%F)
total 36
-rw-------. 1 root root 5658 Jan 24 21:31 admin.conf
-rw-------. 1 root root 5678 Jan 24 21:31 controller-manager.conf
-rw-------. 1 root root 1974 Jan 24 21:31 kubelet.conf
-rw-------. 1 root root 5626 Jan 24 21:31 scheduler.conf
-rw-------. 1 root root 5682 Jan 24 21:31 super-admin.conf

# 인증서 만료 상태 확인
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kubeadm certs check-expiration
[check-expiration] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[check-expiration] Use 'kubeadm init phase upload-config --config your-config.yaml' to re-upload it.

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Jan 24, 2027 09:56 UTC   364d            ca                      no
apiserver                  Jan 24, 2027 09:56 UTC   364d            ca                      no
apiserver-etcd-client      Jan 24, 2027 09:56 UTC   364d            etcd-ca                 no
apiserver-kubelet-client   Jan 24, 2027 09:56 UTC   364d            ca                      no
controller-manager.conf    Jan 24, 2027 09:56 UTC   364d            ca                      no
etcd-healthcheck-client    Jan 24, 2027 09:56 UTC   364d            etcd-ca                 no
etcd-peer                  Jan 24, 2027 09:56 UTC   364d            etcd-ca                 no
etcd-server                Jan 24, 2027 09:56 UTC   364d            etcd-ca                 no
front-proxy-client         Jan 24, 2027 09:56 UTC   364d            front-proxy-ca          no
scheduler.conf             Jan 24, 2027 09:56 UTC   364d            ca                      no
super-admin.conf           Jan 24, 2027 09:56 UTC   364d            ca                      no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Jan 22, 2036 09:56 UTC   9y              no
etcd-ca                 Jan 22, 2036 09:56 UTC   9y              no
front-proxy-ca          Jan 22, 2036 09:56 UTC   9y              no

# 인증서 전체 갱신 : 기존 cert 삭제 -> CA로 재서명된 새 cert 생성
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kubeadm certs renew all
[renew] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[renew] Use 'kubeadm init phase upload-config --config your-config.yaml' to re-upload it.

certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed
certificate embedded in the kubeconfig file for the super-admin renewed

Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.


# 인증서 만료 상태 확인
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# kubeadm certs check-expiration
[check-expiration] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[check-expiration] Use 'kubeadm init phase upload-config --config your-config.yaml' to re-upload it.

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Jan 24, 2027 12:31 UTC   364d            ca                      no
apiserver                  Jan 24, 2027 12:31 UTC   364d            ca                      no
apiserver-etcd-client      Jan 24, 2027 12:31 UTC   364d            etcd-ca                 no
apiserver-kubelet-client   Jan 24, 2027 12:31 UTC   364d            ca                      no
controller-manager.conf    Jan 24, 2027 12:31 UTC   364d            ca                      no
etcd-healthcheck-client    Jan 24, 2027 12:31 UTC   364d            etcd-ca                 no
etcd-peer                  Jan 24, 2027 12:31 UTC   364d            etcd-ca                 no
etcd-server                Jan 24, 2027 12:31 UTC   364d            etcd-ca                 no
front-proxy-client         Jan 24, 2027 12:31 UTC   364d            front-proxy-ca          no
scheduler.conf             Jan 24, 2027 12:31 UTC   364d            ca                      no
super-admin.conf           Jan 24, 2027 12:31 UTC   364d            ca                      no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Jan 22, 2036 09:56 UTC   9y              no
etcd-ca                 Jan 22, 2036 09:56 UTC   9y              no
front-proxy-ca          Jan 22, 2036 09:56 UTC   9y              no

# ca 인증서는 그대로, 나머지 인증서는 신규 생성
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# ls -lt /etc/kubernetes/pki/
total 56
-rw-r--r--. 1 root root 1119 Jan 24 21:31 front-proxy-client.crt
-rw-------. 1 root root 1679 Jan 24 21:31 front-proxy-client.key
-rw-r--r--. 1 root root 1176 Jan 24 21:31 apiserver-kubelet-client.crt
-rw-------. 1 root root 1675 Jan 24 21:31 apiserver-kubelet-client.key
-rw-r--r--. 1 root root 1123 Jan 24 21:31 apiserver-etcd-client.crt
-rw-------. 1 root root 1675 Jan 24 21:31 apiserver-etcd-client.key
-rw-r--r--. 1 root root 1281 Jan 24 21:31 apiserver.crt
-rw-------. 1 root root 1679 Jan 24 21:31 apiserver.key
-rw-------. 1 root root 1679 Jan 24 18:56 sa.key
-rw-------. 1 root root  451 Jan 24 18:56 sa.pub
drwxr-xr-x. 2 root root  162 Jan 24 18:56 etcd
-rw-r--r--. 1 root root 1123 Jan 24 18:56 front-proxy-ca.crt
-rw-------. 1 root root 1675 Jan 24 18:56 front-proxy-ca.key
-rw-r--r--. 1 root root 1107 Jan 24 18:56 ca.crt
-rw-------. 1 root root 1675 Jan 24 18:56 ca.key

(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# ls -lt /etc/kubernetes/pki/etcd
total 32
-rw-r--r--. 1 root root 1196 Jan 24 21:31 server.crt
-rw-------. 1 root root 1679 Jan 24 21:31 server.key
-rw-r--r--. 1 root root 1196 Jan 24 21:31 peer.crt
-rw-------. 1 root root 1679 Jan 24 21:31 peer.key
-rw-r--r--. 1 root root 1123 Jan 24 21:31 healthcheck-client.crt
-rw-------. 1 root root 1675 Jan 24 21:31 healthcheck-client.key
-rw-r--r--. 1 root root 1094 Jan 24 18:56 ca.crt
-rw-------. 1 root root 1675 Jan 24 18:56 ca.key


# apiserver 인증서
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# cat /etc/kubernetes/pki/apiserver.crt | openssl x509 -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 7287864220267195902 (0x6523b1e5444e2dfe)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kubernetes
        Validity
            Not Before: Jan 24 12:26:46 2026 GMT
            Not After : Jan 24 12:31:46 2027 GMT
            
# control component 의 kubeconfig 신규 생성 확인
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# ls -lt /etc/kubernetes/*.conf
-rw-------. 1 root root 5678 Jan 24 21:31 /etc/kubernetes/super-admin.conf
-rw-------. 1 root root 5626 Jan 24 21:31 /etc/kubernetes/scheduler.conf
-rw-------. 1 root root 5678 Jan 24 21:31 /etc/kubernetes/controller-manager.conf
-rw-------. 1 root root 5654 Jan 24 21:31 /etc/kubernetes/admin.conf
-rw-------. 1 root root 1974 Jan 24 18:56 /etc/kubernetes/kubelet.conf

8-3. control-plane static pod 재기동 & admin.conf kubeconfig 재적용

# 사전 백업 : static pod 매니페스트
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# cp -r /etc/kubernetes/manifests /etc/kubernetes/manifests.backup.$(date +%F)
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# ls -l /etc/kubernetes/manifests.backup.$(date +%F)
total 16
-rw-------. 1 root root 2576 Jan 24 21:34 etcd.yaml
-rw-------. 1 root root 3603 Jan 24 21:34 kube-apiserver.yaml
-rw-------. 1 root root 3102 Jan 24 21:34 kube-controller-manager.yaml
-rw-------. 1 root root 1655 Jan 24 21:34 kube-scheduler.yaml

# static pod 모니터링(신규 터미널)
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# watch -d crictl ps

# static pod manifest 삭제
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# rm -rf /etc/kubernetes/manifests/*.yaml

# static pod manifest 복사 -> 파드 재기동  
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# cp /etc/kubernetes/manifests.backup.$(date +%F)/*.yaml /etc/kubernetes/manifests
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# tree /etc/kubernetes/manifests
/etc/kubernetes/manifests
├── etcd.yaml
├── kube-apiserver.yaml
├── kube-controller-manager.yaml
└── kube-scheduler.yaml

1 directory, 4 files

# 파드 기동 확인 : CA가 바뀌지 않았기 때문에 예전 인증서도 신뢰됨 >> 다만, 예전(?) 인증서의 만료 기간을 놓칠 수 있으니, 같이 갱신 할 것!
Every 2.0s: crictl ps                                                                                                                                      k8s-ctr: Sat Jan 24 21:37:03 2026

CONTAINER           IMAGE               CREATED             STATE               NAME                        ATTEMPT             POD ID              POD
               NAMESPACE
5089984fc24b4       cfa17ff3d6634	22 seconds ago      Running             kube-scheduler              0                   949a0d03486b9       kube-scheduler-k8s-ctr
               kube-system
05f61bb01a7a6       1211402d28f58	22 seconds ago      Running             etcd                        0                   318a29e5cd40b       etcd-k8s-ctr
               kube-system
e510345a46907       82766e5f2d560	22 seconds ago      Running             kube-controller-manager     0                   e1b0c9dbe8b2b       kube-controller-manager-k8s-ctr
               kube-system
7d42851059591       58951ea1a0b5d	22 seconds ago      Running             kube-apiserver              0                   6ce0a9479754e       kube-apiserver-k8s-ctr
               kube-system
81913ea92e88d       11873b3fefc46	18 minutes ago      Running             x509-certificate-exporter   0                   8b832b28f605e       x509-certificate-exporter-cp-l7b97
               monitoring
               

# admin.conf kubeconfig 재적용
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# yes | cp  /etc/kubernetes/admin.conf ~/.kube/config ; echo
(⎈|kubernetes-admin@kubernetes:N/A) root@k8s-ctr:~# chown $(id -u):$(id -g) ~/.kube/config

[K8s Deploy] Ansible 기초

안녕유지 — Sun, 18 Jan 2026 08:39:31 +0900

Cloudnet K8s Deploy 2주차 스터디를 진행하며 정리한 글입니다.

이번 포스팅에서는 Ansible 기초에 대해 알아보겠습니다.

Ansible이란

IT 인프라와 애플리케이션 운영 작업을 코드로 자동화하기 위한 오픈소스 도구입니다.

서버 설정, 애플리케이션 설치, 클라우드 리소스 생성 같은 반복 작업을 YAML 기반 플레이북으로 정의해 여러 환경에 동일하게 적용할 수 있습니다.

Ansible의 특징

1. Agentless

관리 대상 서버에 별도 에이전트 설치가 필요 없음
SSH 기반으로 즉시 자동화 가능
Puppet, Chef 대비 초기 도입과 운영 부담이 적음

2. 멱등성 (Idempotent)

동일한 작업을 여러 번 실행해도 항상 같은 결과
원하는 상태(desired state)를 기준으로 시스템을 유지

3. 쉬운 문법과 풍부한 모듈

YAML 문법으로 사람이 읽고 쓰기 쉬움
파일 관리, 패키지 설치, 서비스 제어부터 클라우드 리소스 관리까지 다양한 모듈과 컬렉션 제공

Ansible 구성 요소 (커뮤니티 Ansible 기준)

전체 아키텍처

Control Node + Managed Node
중앙 제어 노드에서 여러 관리 노드를 동시에 자동화

Control Node

Ansible이 설치되어 실행되는 노드
Linux 기반 환경이면 사용 가능
Ansible은 Python 기반이므로 Python 필수

Managed Node

Ansible이 제어하는 원격 서버
Linux 또는 Windows 가능
에이전트 설치 불필요
Control Node와 SSH 통신 가능, Python 설치 필요

Inventory

관리 대상 노드를 정의한 파일
호스트를 그룹 단위로 관리 가능

Module

Ansible이 작업 수행 시 관리 노드로 전송(push) 하는 스크립트
작업 완료 후 자동 제거
시스템의 상태를 기준으로 동작

Plugin

Ansible 제어 노드에서 실행
로깅, 데이터 변환, 인벤토리 연동 등 핵심 기능 확장
모듈을 보조하는 내부 확장 메커니즘

Playbook

실제 자동화를 정의하는 핵심 파일
YAML 형식으로 작업을 순서대로 기술

Ansible 실습 환경

다음 서버를 생성하여 Ansible 실습을 진행하겠습니다.

Node	OS	Kernel	vCPU	Memory	Disk	NIC2 IP	관리자 계정	(기본) 일반 계정
server	Ubuntu 24.04	6.8.0	2	1.5GB	30GB	10.10.1.10	root / qwe123	vagrant / qwe123
tnode1	상동	상동	2	1.5GB	30GB	10.10.1.11	root / qwe123	vagrant / qwe123
tnode2	상동	상동	2	1.5GB	30GB	10.10.1.12	root / qwe123	vagrant / qwe123
tnode3	Rocky Linux 9	5.14.0	2	1.5GB	60GB	10.10.1.13	root / qwe123	vagrant / qwe123

> vagrant ssh server

root@server:~# whoami
root

# 파이썬 버전 확인
root@server:~# python3 --version
Python 3.12.3

# 설치
root@server:~# apt install software-properties-common -y
root@server:~# add-apt-repository --yes --update ppa:ansible/ansible
root@server:~# apt install ansible -y

# 확인
root@server:~# ansible --version
ansible [core 2.19.5]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.12.3 (main, Aug 14 2025, 17:47:21) [GCC 13.3.0] (/usr/bin/python3)
  jinja version = 3.1.2
  pyyaml version = 6.0.1 (with libyaml v0.2.5)
root@server:~# cat /etc/ansible/ansible.cfg
# Since Ansible 2.12 (core):
# To generate an example config file (a "disabled" one with all default settings, commented out):
#               $ ansible-config init --disabled > ansible.cfg
#
# Also you can now have a more complete file by including existing plugins:
# ansible-config init --disabled -t all > ansible.cfg

# For previous versions of Ansible you can check for examples in the 'stable' branches of each version
# Note that this file was always incomplete  and lagging changes to configuration settings

# for example, for 2.9: https://github.com/ansible/ansible/blob/stable-2.9/examples/ansible.cfg


# ssh-keygen 명령어를 이용하여 SSH 키 생성
root@server:~/my-ansible# tree ~/.ssh
/root/.ssh
└── authorized_keys

1 directory, 1 file

root@server:~/my-ansible# ssh-keygen -t rsa -N "" -f /root/.ssh/id_rsa

# 공개키를 관리 노드에 복사
root@server:~/my-ansible# for i in {1..3}; do sshpass -p 'qwe123' ssh-copy-id -o StrictHostKeyChecking=no root@tnode$i; done

# ssh 접속 테스트
root@server:~/my-ansible# for i in {1..3}; do echo ">> tnode$i <<"; ssh tnode$i hostname; echo; done

# python 정보 확인
root@server:~/my-ansible# for i in {1..3}; do echo ">> tnode$i <<"; ssh tnode$i python3 -V; echo; done
>> tnode1 <<
Python 3.12.3

>> tnode2 <<
Python 3.12.3

>> tnode3 <<
Python 3.9.21

Host 선정

인벤토리 파일

텍스트 파일로, Ansible이 자동화 대상으로 하는 관리 호스트를 지정

ansible config 적용 우선 순위

현재 프로젝트 디렉터리 내에 ansible.cfg 라는 앤서블 환경 설정 파일을 구성 시, -i 옵션을 사용하지 않아도 ansible.cfg 설정 파일에 정의된 인벤토리의 호스트 정보를 확인 가능

ANSIBLE_CONFIG (environment variable if set)
ansible.cfg (in the current directory)
~/.ansible.cfg (in the home directory)
/etc/ansible/ansible.cfg

# IP를 이용한 인벤토리 파일 생성
root@server:~/my-ansible# cat <<EOT > inventory
10.10.1.11
10.10.1.12
10.10.1.13
EOT

root@server:~/my-ansible# ansible-inventory -i ./inventory --list | jq
{
  "_meta": {
    "hostvars": {},
    "profile": "inventory_legacy"
  },
  "all": {
    "children": [
      "ungrouped"
    ]
  },
  "ungrouped": {
    "hosts": [
      "10.10.1.11",
      "10.10.1.12",
      "10.10.1.13"
    ]
  }
}


# 호스트명을 이용한 인벤토리 파일 생성
root@server:~/my-ansible# cat /etc/hosts
127.0.0.1 localhost

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
10.10.1.10 server
10.10.1.11 tnode1
10.10.1.12 tnode2
10.10.1.13 tnode3

root@server:~/my-ansible# cat <<EOT > inventory
tnode1
tnode2
tnode3
EOT

root@server:~/my-ansible# ansible-inventory -i ./inventory --list | jq
{
  "_meta": {
    "hostvars": {},
    "profile": "inventory_legacy"
  },
  "all": {
    "children": [
      "ungrouped"
    ]
  },
  "ungrouped": {
    "hosts": [
      "tnode1",
      "tnode2",
      "tnode3"
    ]
  }
}

# 인벤토리 그룹 구성 / ansible config 적용 
root@server:~/my-ansible# cat <<EOT > inventory
[web]
tnode1
tnode2

[db]
tnode3

[all:children]
web
db
EOT

root@server:~/my-ansible# ansible-inventory -i ./inventory --list | jq
{
  "_meta": {
    "hostvars": {},
    "profile": "inventory_legacy"
  },
  "all": {
    "children": [
      "ungrouped",
      "web",
      "db"
    ]
  },
  "db": {
    "hosts": [
      "tnode3"
    ]
  },
  "web": {
    "hosts": [
      "tnode1",
      "tnode2"
    ]
  }
}

root@server:~/my-ansible# cat <<EOT > ansible.cfg
[defaults]
inventory = ./inventory
EOT

# inventory 목록 확인
root@server:~/my-ansible# ansible-inventory --list | jq


# ansible config 적용 우선 순위 확인
root@server:~/my-ansible# echo $ANSIBLE_CONFIG
root@server:~/my-ansible# cat $PWD/ansible.cfg   # kubespary 실행 시, 디렉터리 위치 고정 이유
root@server:~/my-ansible# ls ~/.ansible.cfg
root@server:~/my-ansible# tree ~/.ansible
root@server:~/my-ansible# cat /etc/ansible/ansible.cfg

#
root@server:~/my-ansible# ansible-config dump
root@server:~/my-ansible# ansible-config list

PlayBook 작성

ansible.cfg 파일을 생성하면 다양한 앤서블 설정을 적용 가능
섹션 제목은 대괄호로 묶여 있으며, 기본적인 실행을 위해 다음 예제와 같이 [defaults]와 [privilege_escalation] 두 개의 섹션으로 구성
- default : ansible 작업을 위한 기본값 설정
- prvilege_escalation : 보안/감사로 인해 원격 호스트에 권한 없는 사용자 연결 후 관리 액세스 권한 에스컬레이션하여 루트 사용자로 가져올 때

root@server:~/my-ansible# cat <<EOT > ansible.cfg
[defaults]
inventory = ./inventory
remote_user = root
ask_pass = false

[privilege_escalation]
become = true
become_method = sudo
become_user = root
become_ask_pass = false
EOT

root@server:~/my-ansible# ansible-inventory -i ./inventory --list | jq
{
  "_meta": {
    "hostvars": {
      "tnode1": {
        "ansible_python_interpreter": "/usr/bin/python3"
      },
      "tnode2": {
        "ansible_python_interpreter": "/usr/bin/python3"
      },
      "tnode3": {
        "ansible_python_interpreter": "/usr/bin/python3"
      }
    },
    "profile": "inventory_legacy"
  },
  "all": {
    "children": [
      "ungrouped",
      "web",
      "db"
    ]
  },
  "db": {
    "hosts": [
      "tnode3"
    ]
  },
  "web": {
    "hosts": [
      "tnode1",
      "tnode2"
    ]
  }
}

# ad hoc ansible ping 모듈
root@server:~/my-ansible# ansible -m ping web
tnode1 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
tnode2 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

root@server:~/my-ansible# ansible -m ping db
tnode3 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

# 옵션 설정으로 암호 입력 후 실행 확인
root@server:~/my-ansible# ansible -m ping --ask-pass web
SSH password: 
tnode1 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
tnode2 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

# 다른 사용자 계정으로 실행 확인
root@server:~/my-ansible# ansible -m ping web -u vagrant
[ERROR]: Task failed: Failed to connect to the host via ssh: vagrant@tnode1: Permission denied (publickey,password).
Origin: <adhoc 'ping' task>

{'action': 'ping', 'args': {}, 'timeout': 0, 'async_val': 0, 'poll': 15}

tnode1 | UNREACHABLE! => {
    "changed": false,
    "msg": "Task failed: Failed to connect to the host via ssh: vagrant@tnode1: Permission denied (publickey,password).",
    "unreachable": true
}
[ERROR]: Task failed: Failed to connect to the host via ssh: vagrant@tnode2: Permission denied (publickey,password).
Origin: <adhoc 'ping' task>

{'action': 'ping', 'args': {}, 'timeout': 0, 'async_val': 0, 'poll': 15}

tnode2 | UNREACHABLE! => {
    "changed": false,
    "msg": "Task failed: Failed to connect to the host via ssh: vagrant@tnode2: Permission denied (publickey,password).",
    "unreachable": true
}


# ad hoc ansible shell 모듈
root@server:~/my-ansible# ansible -m shell -a uptime db
tnode3 | CHANGED | rc=0 >>
 07:34:16 up  1:20,  1 user,  load average: 0.00, 0.00, 0.00
 
root@server:~/my-ansible# ansible -m shell -a "free -h" web
tnode1 | CHANGED | rc=0 >>
               total        used        free      shared  buff/cache   available
Mem:           1.3Gi       264Mi       448Mi       4.8Mi       702Mi       1.0Gi
Swap:          3.7Gi          0B       3.7Gi
tnode2 | CHANGED | rc=0 >>
               total        used        free      shared  buff/cache   available
Mem:           1.3Gi       248Mi       463Mi       4.8Mi       703Mi       1.1Gi
Swap:          3.7Gi          0B       3.7Gi

변수

변수를 사용하여 사용자, 설치하고자 하는 패키지, 재시작할 서비스, 생성 또는 삭제할 파일 명 등 시스템 작업 시 사용되는 다양한 값 저장

변수 종류

추가 변수: 외부에서 플레이북 실행 시 파라미터러 전달되는 변수
플레이 변수: 플레이북 내에서 선언되는 변수 또는 별도 파일로 분리되는 변수
호스트 변수: 특정 호스트에서만 사용하는 변수
그룹 변수: 인벤토리 정의된 호스트 그룹에 적용하는 변수
작업 변수: 플레이북의 수행 결과를 저장 후 후속 작업에 사용할 때 사용된다.

변수 우선 순위

추가변수
플레이 변수
호스트 변수
그룹 변수

# ansible.builtin.user 모듈
### 그룹 변수
# my-ansible/inventory
[web]
tnode1 ansible_python_interpreter=/usr/bin/python3
tnode2 ansible_python_interpreter=/usr/bin/python3

[db]
tnode3 ansible_python_interpreter=/usr/bin/python3

[all:children]
web
db

[all:vars]
user=ansible

# create-user.yml
---

- hosts: all
  tasks:
  - name: Create User {{ user }}
    ansible.builtin.user:
      name: "{{ user }}"
      state: present


root@server:~/my-ansible# ansible-playbook create-user.yml

PLAY [all] ******************************************************************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************************
ok: [tnode1]
ok: [tnode3]
ok: [tnode2]

TASK [Create User ansible] **************************************************************************************************************************************************************
changed: [tnode3]
changed: [tnode2]
changed: [tnode1]

PLAY RECAP ******************************************************************************************************************************************************************************
tnode1                     : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
tnode2                     : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
tnode3                     : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   


# 대상 host에서 ansible 사용자 확인
root@server:~/my-ansible# for i in {1..3}; do echo ">> tnode$i <<"; ssh tnode$i tail -n 3 /etc/passwd; echo; done
>> tnode1 <<
vagrant:x:1000:1000:vagrant:/home/vagrant:/bin/bash
vboxadd:x:999:1::/var/run/vboxadd:/bin/false
ansible:x:1001:1001::/home/ansible:/bin/sh

>> tnode2 <<
vagrant:x:1000:1000:vagrant:/home/vagrant:/bin/bash
vboxadd:x:999:1::/var/run/vboxadd:/bin/false
ansible:x:1001:1001::/home/ansible:/bin/sh

>> tnode3 <<
vagrant:x:1000:1000::/home/vagrant:/bin/bash
vboxadd:x:991:1::/var/run/vboxadd:/bin/false
ansible:x:1001:1001::/home/ansible:/bin/bash


### 호스트 변수
# my-ansible/inventory
[web]
tnode1 ansible_python_interpreter=/usr/bin/python3
tnode2 ansible_python_interpreter=/usr/bin/python3

[db]
tnode3 ansible_python_interpreter=/usr/bin/python3 user=ansible1

[all:children]
web
db

[all:vars]
user=ansible

# my-ansible/create-user1.yml
---

- hosts: db
  tasks:
  - name: Create User {{ user }}
    ansible.builtin.user:
      name: "{{ user }}"
      state: present
      
root@server:~/my-ansible# ansible-playbook create-user1.yml

PLAY [db] *******************************************************************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************************
ok: [tnode3]

TASK [Create User ansible1] *************************************************************************************************************************************************************
changed: [tnode3]

PLAY RECAP ******************************************************************************************************************************************************************************
tnode3                     : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   


# 확인
root@server:~/my-ansible# for i in {1..3}; do echo ">> tnode$i <<"; ssh tnode$i tail -n 3 /etc/passwd; echo; done
>> tnode1 <<
vagrant:x:1000:1000:vagrant:/home/vagrant:/bin/bash
vboxadd:x:999:1::/var/run/vboxadd:/bin/false
ansible:x:1001:1001::/home/ansible:/bin/sh

>> tnode2 <<
vagrant:x:1000:1000:vagrant:/home/vagrant:/bin/bash
vboxadd:x:999:1::/var/run/vboxadd:/bin/false
ansible:x:1001:1001::/home/ansible:/bin/sh

>> tnode3 <<
vboxadd:x:991:1::/var/run/vboxadd:/bin/false
ansible:x:1001:1001::/home/ansible:/bin/bash
ansible1:x:1002:1002::/home/ansible1:/bin/bash


### 플레이 변수
# my-ansible/create-user2.yml
---

- hosts: all
  vars:
    user: ansible2

  tasks:
  - name: Create User {{ user }}
    ansible.builtin.user:
      name: "{{ user }}"
      state: present
      

root@server:~/my-ansible# ansible-playbook create-user2.yml

PLAY [all] ******************************************************************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************************
ok: [tnode1]
ok: [tnode2]
ok: [tnode3]

TASK [Create User ansible2] *************************************************************************************************************************************************************
changed: [tnode3]
changed: [tnode1]
changed: [tnode2]

PLAY RECAP ******************************************************************************************************************************************************************************
tnode1                     : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
tnode2                     : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
tnode3                     : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

# 확인
root@server:~/my-ansible# for i in {1..3}; do echo ">> tnode$i <<"; ssh tnode$i tail -n 3 /etc/passwd; echo; done
>> tnode1 <<
vboxadd:x:999:1::/var/run/vboxadd:/bin/false
ansible:x:1001:1001::/home/ansible:/bin/sh
ansible2:x:1002:1002::/home/ansible2:/bin/sh

>> tnode2 <<
vboxadd:x:999:1::/var/run/vboxadd:/bin/false
ansible:x:1001:1001::/home/ansible:/bin/sh
ansible2:x:1002:1002::/home/ansible2:/bin/sh

>> tnode3 <<
ansible:x:1001:1001::/home/ansible:/bin/bash
ansible1:x:1002:1002::/home/ansible1:/bin/bash
ansible2:x:1003:1003::/home/ansible2:/bin/bash


# ansible.builtin.debug 모듈
### 작업 변수
# my-ansible/create-user4.yml
---

- hosts: db
  tasks:
  - name: Create User {{ user }}
    ansible.builtin.user:
      name: "{{ user }}"
      state: present
    register: result
  
  - ansible.builtin.debug:
      var: result
      

root@server:~/my-ansible# ansible-playbook -e user=ansible5 create-user4.yml

PLAY [db] *******************************************************************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************************
ok: [tnode3]

TASK [Create User ansible5] *************************************************************************************************************************************************************
changed: [tnode3]

TASK [ansible.builtin.debug] ************************************************************************************************************************************************************
ok: [tnode3] => {
    "result": {
        "changed": true,
        "comment": "",
        "create_home": true,
        "failed": false,
        "group": 1004,
        "home": "/home/ansible5",
        "name": "ansible5",
        "shell": "/bin/bash",
        "state": "present",
        "system": false,
        "uid": 1004
    }
}

PLAY RECAP ******************************************************************************************************************************************************************************
tnode3                     : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Facts

ansible이 관리 호스트에서 자동으로 검색한 변수

관리 호스트에서 수집된 일부 팩트에는 다음 내용들이 포함될 수 있습니다.

호스트 이름
커널 버전
네트워크 인터페이스 이름
운영체제 버전
CPU 개수
사용 가능한 메모리
스토리지 장치의 크기 및 여유 공간
등등…

# my-ansible/facts.yml
---

- hosts: db

  tasks:
  - name: Print all facts
    ansible.builtin.debug:
      var: ansible_facts
      
      
# playbook 실행
root@server:~/my-ansible# ansible-playbook facts.yml

PLAY [db] *******************************************************************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************************
ok: [tnode3]

TASK [Print all facts] ******************************************************************************************************************************************************************
ok: [tnode3] => {
    "ansible_facts": {
        "all_ipv4_addresses": [
            "10.10.1.13",
            "10.0.2.15"
        ],
        "all_ipv6_addresses": [
            "fe80::a00:27ff:fe6e:6ffb",
            "fd17:625c:f037:2:a00:27ff:fe1f:ae8b",
            "fe80::a00:27ff:fe1f:ae8b"
        ],
        "ansible_local": {},
        "apparmor": {
            "status": "disabled"
        },
        "architecture": "aarch64",
        "bios_date": "NA",
        "bios_vendor": "NA",
        "bios_version": "NA",
        "board_asset_tag": "NA",
        "board_name": "NA",
   		...
        "env": {
            "BASH_FUNC_which%%": "() {  ( alias;\n eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot $@\n}",
            "DBUS_SESSION_BUS_ADDRESS": "unix:path=/run/user/0/bus",
            "DEBUGINFOD_IMA_CERT_PATH": "/etc/keys/ima:",
            "DEBUGINFOD_URLS": "https://debuginfod.rockylinux.org/ ",
            "HOME": "/root",
            "LANG": "en_US.UTF-8",
            "LESSOPEN": "||/usr/bin/lesspipe.sh %s",
            "LOGNAME": "root",
            "LS_COLORS": "rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.m4a=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.oga=01;36:*.opus=01;36:*.spx=01;36:*.xspf=01;36:",
            "MOTD_SHOWN": "pam",
            "PATH": "/root/.local/bin:/root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin",
            "PWD": "/root",
            "SELINUX_LEVEL_REQUESTED": "",
            "SELINUX_ROLE_REQUESTED": "",
            "SELINUX_USE_CURRENT_RANGE": "",
            "SHELL": "/bin/bash",
            "SHLVL": "1",
            "SSH_CLIENT": "10.10.1.10 40386 22",
            "SSH_CONNECTION": "10.10.1.10 40386 10.10.1.13 22",
            "SSH_TTY": "/dev/pts/0",
            "TERM": "xterm-256color",
            "USER": "root",
            "XDG_RUNTIME_DIR": "/run/user/0",
            "XDG_SESSION_CLASS": "user",
            "XDG_SESSION_ID": "35",
            "XDG_SESSION_TYPE": "tty",
            "_": "/usr/bin/python3",
            "which_declare": "declare -f"
        },
        "fibre_channel_wwn": [],
        "fips": false,
        "form_factor": "NA",
        "fqdn": "tnode3",
        "gather_subset": [
            "all"
        ],
        "hostname": "tnode3",
        "hostnqn": "nqn.2014-08.org.nvmexpress:uuid:b4201204-1135-498b-8536-3a8fe83e7131",
        "interfaces": [
            "enp0s8",
            "enp0s9",
            "lo"
        ],
        "is_chroot": false,
        "iscsi_iqn": "",
        "kernel": "5.14.0-570.52.1.el9_6.aarch64",
        "kernel_version": "#1 SMP PREEMPT_DYNAMIC Wed Oct 15 14:48:33 UTC 2025",
        "processor_cores": 1,
        "processor_count": 2,
        "processor_nproc": 2,
        "processor_threads_per_core": 1,
        "processor_vcpus": 2,
        "product_name": "NA",
        "product_serial": "NA",
        "product_uuid": "NA",
        "product_version": "NA",
        "python": {
            "executable": "/usr/bin/python3",
            "has_sslcontext": true,
            "type": "cpython",
            "version": {
                "major": 3,
                "micro": 21,
                "minor": 9,
                "releaselevel": "final",
                "serial": 0
            },
            "version_info": [
                3,
                9,
                21,
                "final",
                0
            ]
        },
        "python_version": "3.9.21",
        "real_group_id": 0,
        "real_user_id": 0,
        "selinux": {
            "config_mode": "permissive",
            "mode": "permissive",
            "policyvers": 33,
            "status": "enabled",
            "type": "targeted"
        },
        "selinux_python_present": true,
        "service_mgr": "systemd",
        "ssh_host_key_ecdsa_public": "AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA3NDADxITYCQ+Yzl24Rv5+HTMqHFo1A1i6bGwRXfIvZ5ZPFQYMrR2CAi0vU69QVLwdWCJoLoQP8He+NBfrlrV8=",
        "ssh_host_key_ecdsa_public_keytype": "ecdsa-sha2-nistp256",
        "ssh_host_key_ed25519_public": "AAAAC3NzaC1lZDI1NTE5AAAAIKcCJWNOyGZ6SW350FH+dihyL+qJBcvRb7NIvkyes3Lt",
        "ssh_host_key_ed25519_public_keytype": "ssh-ed25519",
        "ssh_host_key_rsa_public": "AAAAB3NzaC1yc2EAAAADAQABAAABgQCQOhfEeFMicbWK5uXUAbgkFqTQRQGptB4FLBmiMIrpVVC0J9PFfBTJL2UIvDg1TCdwQ+DkJhtjffLDjWM/OU75gsYs9Ih0aGgE1zHf+93Wt0tM5+I8z0fTact/+4GaBAuSO4o7rjlebOC5XpgGT7aglCbuUn7UIefvf4m1OIdrWL8szWb6jZLGNH6AOn7itpri2cXWp9pnffr8FHYWIsKyHJWnRGtPXSOlU7/2wc7j0b8M1+H2FpBS8d4Y9+0Jdf0T9wq3tK7UkcW5hOGv09X1h42xMLzaCqivdvP2dxMz5xcvUWrC6g2wQgzg7IkA+hVszF6Nfazu2GYrNOTNyy3Pf+Krb8SthldVg4/Skw7sMkmGQ6TGCO5yTY0ul13sPiEx4SHBbtebJXk075W2+dB1yx6/6lxziUkUeW9jBJf2IVUpSMTf6PGLOefce1pLu/bSdD0pxR6CpukDfUhHaNGJCZpaiOEbk5FK2f9B+9TJ0y+w6ZaXzcfNmPj2hzzhN+8=",
        "ssh_host_key_rsa_public_keytype": "ssh-rsa",
        "swapfree_mb": 3902,
        "swaptotal_mb": 3902,
        "system": "Linux",
        "system_capabilities": [],
        "system_capabilities_enforced": "False",
        "system_vendor": "NA",
        "systemd": {
            "features": "+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified",
            "version": 252
        },
        "uptime_seconds": 6548,
        "user_dir": "/root",
        "user_gecos": "root",
        "user_gid": 0,
        "user_id": "root",
        "user_shell": "/bin/bash",
        "user_uid": 0,
        "userspace_bits": "64",
        "virtualization_role": "NA",
        "virtualization_tech_guest": [],
        "virtualization_tech_host": [],
        "virtualization_type": "NA"
    }
}

PLAY RECAP ******************************************************************************************************************************************************************************
tnode3                     : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0


# facts를 통해 수집된 변수 중 특정 값만 추출
# my-ansible/facts1.yml
---

- hosts: db

  tasks:
  - name: Print all facts
    ansible.builtin.debug:
      msg: >
        The default IPv4 address of {{ ansible_facts.hostname }}
        is {{ ansible_facts.default_ipv4.address }}v



root@server:~/my-ansible# ansible-playbook facts1.yml

PLAY [db] *******************************************************************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************************
ok: [tnode3]

TASK [Print all facts] ******************************************************************************************************************************************************************
ok: [tnode3] => {
    "msg": "The default IPv4 address of tnode3 is 10.0.2.15"
}

PLAY RECAP ******************************************************************************************************************************************************************************
tnode3                     : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

반복문

loop 키워드를 작업에 추가하면 작업을 반복해야 하는 항목의 목록을 값으로 사용

# ~/my-ansible/check-services.yml
# sshd와 rsyslog 서비스가 시작되어 있지 않다면 시작
---
- hosts: all
  tasks:
    - name: Check sshd state on Debian
      ansible.builtin.service:
        name: ssh
        state: started
      when: ansible_facts['os_family'] == 'Debian'

    - name: Check sshd state on RedHat
      ansible.builtin.service:
        name: sshd
        state: started
      when: ansible_facts['os_family'] == 'RedHat'


    - name: Check rsyslog state
      ansible.builtin.service:
        name: rsyslog
        state: started


root@server:~/my-ansible# ansible -m shell -a "pstree |grep sshd" all
tnode1 | CHANGED | rc=0 >>
        |-sshd---sshd---sh---python3---sh-+-grep
tnode3 | CHANGED | rc=0 >>
        |-sshd---sshd---sshd---sh---python3---sh-+-grep
tnode2 | CHANGED | rc=0 >>
        |-sshd---sshd---sh---python3---sh-+-grep
        
        
root@server:~/my-ansible# ansible -m shell -a "pstree |grep rsyslog" all
tnode3 | CHANGED | rc=0 >>
        |-rsyslogd---2*[{rsyslogd}]
tnode2 | CHANGED | rc=0 >>
        |-rsyslogd---3*[{rsyslogd}]
tnode1 | CHANGED | rc=0 >>
        |-rsyslogd---3*[{rsyslogd}]
        
root@server:~/my-ansible# systemctl list-units --type=service
  UNIT                                                                                      LOAD   ACTIVE SUB     DESCRIPTION                                                           >
  apport.service                                                                            loaded active exited  automatic crash report generation
  blk-availability.service                                                                  loaded active exited  Availability of block devices
  console-setup.service                                                                     loaded active exited  Set console font and keymap
  cron.service                                                                              loaded active running Regular background program processing daemon
  dbus.service                                                                              loaded active running D-Bus System Message Bus
  finalrd.service                                                                           loaded active exited  Create final runtime dir for shutdown pivot root
  getty@tty1.service                                                                        loaded active running Getty on tty1
  keyboard-setup.service                                                                    loaded active exited  Set the console keyboard layout
  kmod-static-nodes.service                                                                 loaded active exited  Create List of Static Device Nodes
  lvm2-monitor.service                                                                      loaded active exited  Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress >
  ModemManager.service                                                                      loaded active running Modem Manager
  multipathd.service                                                                        loaded active running Device-Mapper Multipath Device Controller
  plymouth-quit-wait.service                                                                loaded active exited  Hold until boot process finishes up
  plymouth-quit.service                                                                     loaded active exited  Terminate Plymouth Boot Screen
  plymouth-read-write.service                                                               loaded active exited  Tell Plymouth To Write Out Runtime Data
  polkit.service                                                                            loaded active running Authorization Manager
  rsyslog.service                                                                           loaded active running System Logging Service
  setvtrgb.service                                                                          loaded active exited  Set console scheme
  snapd.apparmor.service                                                                    loaded active exited  Load AppArmor profiles managed internally by snapd
  snapd.seeded.service                                                                      loaded active exited  Wait until snapd is fully seeded
  ssh.service                                                                               loaded active running OpenBSD Secure Shell server
  sysstat.service                                                                           loaded active exited  Resets System Activity Logs
  systemd-binfmt.service                                                                    loaded active exited  Set Up Additional Binary Formats
  systemd-fsck@dev-disk-by\x2duuid-58F1\x2dBD52.service                                     loaded active exited  File System Check on /dev/disk/by-uuid/58F1-BD52
  systemd-fsck@dev-disk-by\x2duuid-b8cd9c1e\x2d487d\x2d412b\x2d882b\x2dedeb25fd98bc.service loaded active exited  File System Check on /dev/disk/by-uuid/b8cd9c1e-487d-412b-882b-edeb25f>
  
  
root@server:~/my-ansible# ansible-playbook check-services.yml

PLAY [all] ******************************************************************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************************
ok: [tnode1]
ok: [tnode2]
ok: [tnode3]

TASK [Check sshd state on Debian] *******************************************************************************************************************************************************
skipping: [tnode3]
ok: [tnode2]
ok: [tnode1]

TASK [Check sshd state on RedHat] *******************************************************************************************************************************************************
skipping: [tnode1]
skipping: [tnode2]
ok: [tnode3]

TASK [Check rsyslog state] **************************************************************************************************************************************************************
ok: [tnode1]
ok: [tnode3]
ok: [tnode2]

PLAY RECAP ******************************************************************************************************************************************************************************
tnode1                     : ok=3    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0   
tnode2                     : ok=3    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0   
tnode3                     : ok=3    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0   



# loop 반복문 적용
# my-ansible/check-services1.yml
---
- hosts: all
  tasks:
  - name: Check sshd and rsyslog state
    ansible.builtin.service:
      name: "{{ item }}"
      state: started
    loop:
      - vboxadd-service  # ssh
      - rsyslog
      
root@server:~/my-ansible# ansible-playbook check-services1.yml

PLAY [all] ******************************************************************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************************
ok: [tnode1]
ok: [tnode2]
ok: [tnode3]

TASK [Check sshd and rsyslog state] *****************************************************************************************************************************************************
ok: [tnode3] => (item=vboxadd-service)
ok: [tnode1] => (item=vboxadd-service)
ok: [tnode2] => (item=vboxadd-service)
ok: [tnode3] => (item=rsyslog)
ok: [tnode2] => (item=rsyslog)
ok: [tnode1] => (item=rsyslog)

PLAY RECAP ******************************************************************************************************************************************************************************
tnode1                     : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
tnode2                     : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
tnode3                     : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   


# loop문에 사용하는 아이템을 변수에 저장하면 loop 키워드에 해당 변수명 사용 가능
# my-ansible/check-services2.yml
---
- hosts: all
  vars:
    services:
      - vboxadd-service  # ssh
      - rsyslog

  tasks:
  - name: Check sshd and rsyslog state
    ansible.builtin.service:
      name: "{{ item }}"
      state: started
    loop: "{{ services }}"
    

root@server:~/my-ansible# ansible-playbook check-services2.yml

PLAY [all] ******************************************************************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************************
ok: [tnode1]
ok: [tnode3]
ok: [tnode2]

TASK [Check sshd and rsyslog state] *****************************************************************************************************************************************************
ok: [tnode3] => (item=vboxadd-service)
ok: [tnode1] => (item=vboxadd-service)
ok: [tnode2] => (item=vboxadd-service)
ok: [tnode3] => (item=rsyslog)
ok: [tnode1] => (item=rsyslog)
ok: [tnode2] => (item=rsyslog)

PLAY RECAP ******************************************************************************************************************************************************************************
tnode1                     : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
tnode2                     : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
tnode3                     : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

조건문

조건문을 사용하여 특정 조건이 충족될 때 작업 또는 플레이를 실행

when 문은 조건부로 작업을 실행할 때 테스트할 조건을 값으로 사용

# my-ansible/when_task.yml
---
- hosts: localhost
  vars:
    run_my_task: true

  tasks:
  - name: echo message
    ansible.builtin.shell: "echo test"
    when: run_my_task
    register: result

  - name: Show result
    ansible.builtin.debug:
      var: result
      

root@server:~/my-ansible# ansible-playbook when_task.yml

PLAY [localhost] ************************************************************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************************
ok: [localhost]

TASK [echo message] *********************************************************************************************************************************************************************
changed: [localhost]

TASK [Show result] **********************************************************************************************************************************************************************
ok: [localhost] => {
    "result": {
        "changed": true,
        "cmd": "echo test",
        "delta": "0:00:00.001913",
        "end": "2026-01-18 08:13:22.691144",
        "failed": false,
        "msg": "",
        "rc": 0,
        "start": "2026-01-18 08:13:22.689231",
        "stderr": "",
        "stderr_lines": [],
        "stdout": "test",
        "stdout_lines": [
            "test"
        ]
    }
}

PLAY RECAP ******************************************************************************************************************************************************************************
localhost                  : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   


# run_my_task값을 false로 수정 후 실행 확인
# 테스트 수행되지 않고 건너뜀 (skip)
root@server:~/my-ansible# ansible-playbook when_task.yml

PLAY [localhost] ************************************************************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************************
ok: [localhost]

TASK [echo message] *********************************************************************************************************************************************************************
skipping: [localhost]

TASK [Show result] **********************************************************************************************************************************************************************
ok: [localhost] => {
    "result": {
        "changed": false,
        "false_condition": "run_my_task",
        "skip_reason": "Conditional result was False",
        "skipped": true
    }
}

PLAY RECAP ******************************************************************************************************************************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0

핸들러 및 작업 실패 처리

앤서블에서 핸들러를 사용하려면 notify 문을 사용하여 명시적으로 호출된 경우에만 사용할 수 있습니다.

# my-ansible/handler-sample.yml
---
- hosts: tnode2
  tasks:
    - name: restart rsyslog
      ansible.builtin.service:
        name: "rsyslog"
        state: restarted
      notify:
        - print msg

  handlers:
    - name: print msg
      ansible.builtin.debug:
        msg: "rsyslog is restarted"


root@server:~/my-ansible# ansible-playbook handler-sample.yml

PLAY [tnode2] ***************************************************************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************************
ok: [tnode2]

TASK [restart rsyslog] ******************************************************************************************************************************************************************
changed: [tnode2]

RUNNING HANDLER [print msg] *************************************************************************************************************************************************************
ok: [tnode2] => {
    "msg": "rsyslog is restarted"
}

PLAY RECAP ******************************************************************************************************************************************************************************
tnode2                     : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   


# 한번 더 실행 시 tnode2 노드에 rsyslog 서비스를 재시작하고, print msg라는 핸들러를 호출해 “rsyslog..” 메시지를 출력
root@server:~/my-ansible# ansible-playbook handler-sample.yml

PLAY [tnode2] ***************************************************************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************************
ok: [tnode2]

TASK [restart rsyslog] ******************************************************************************************************************************************************************
changed: [tnode2]

RUNNING HANDLER [print msg] *************************************************************************************************************************************************************
ok: [tnode2] => {
    "msg": "rsyslog is restarted"
}

PLAY RECAP ******************************************************************************************************************************************************************************
tnode2                     : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   


# 작업 실패 무시
my-ansible/ignore-example-1.yml
---
- hosts : tnode1

  tasks:
    - name: Install apache3
      ansible.builtin.apt:
        name: apache3
        state: latest

    - name: Print msg
      ansible.builtin.debug:
        msg: "Before task is ignored"
        
my-ansible/ignore-example-2.yml
---
- hosts : tnode1

  tasks:
    - name: Install apache3
      ansible.builtin.apt:
        name: apache3
        state: latest
      ignore_errors: yes

    - name: Print msg
      ansible.builtin.debug:
        msg: "Before task is ignored"


root@server:~/my-ansible# ansible-playbook ignore-example-1.yml

PLAY [tnode1] ***************************************************************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************************
ok: [tnode1]

TASK [Install apache3] ******************************************************************************************************************************************************************
[ERROR]: Task failed: Module failed: No package matching 'apache3' is available
Origin: /root/my-ansible/ignore-example-1.yml:5:7

3
4   tasks:
5     - name: Install apache3
        ^ column 7

fatal: [tnode1]: FAILED! => {"changed": false, "msg": "No package matching 'apache3' is available"}

PLAY RECAP ******************************************************************************************************************************************************************************
tnode1                     : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   


# 무시됨
root@server:~/my-ansible# ansible-playbook ignore-example-2.yml

PLAY [tnode1] ***************************************************************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************************
ok: [tnode1]

TASK [Install apache3] ******************************************************************************************************************************************************************
[ERROR]: Task failed: Module failed: No package matching 'apache3' is available
Origin: /root/my-ansible/ignore-example-2.yml:5:7

3
4   tasks:
5     - name: Install apache3
        ^ column 7

fatal: [tnode1]: FAILED! => {"changed": false, "msg": "No package matching 'apache3' is available"}
...ignoring

TASK [Print msg] ************************************************************************************************************************************************************************
ok: [tnode1] => {
    "msg": "Before task is ignored"
}

PLAY RECAP ******************************************************************************************************************************************************************************
tnode1                     : ok=3    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=1

롤 구조 소개 및 사용법

롤은 플레이북 내용을 기능 단위로 나누어 공통 부품으로 관리/재사용하기 위한 구조

# 롤 생성
root@server:~/my-ansible# ansible-galaxy role init my-role
- Role my-role was created successfully

root@server:~/my-ansible# tree ./my-role/
./my-role/
├── defaults
│   └── main.yml
├── files
├── handlers
│   └── main.yml
├── meta
│   └── main.yml
├── README.md
├── tasks
│   └── main.yml
├── templates
├── tests
│   ├── inventory
│   └── test.yml
└── vars
    └── main.yml

9 directories, 8 files

# 롤을 이용한 플레이북 개발
```
- 롤 이름 : my-role
- tasks (메인 태스크)
    - install service : httpd 관련 패키지 설치
    - copy html file : index.html 파일 복사
- files (정적 파일)
    - index.html
- handlers (핸들러)
    - restart service : httpd 서비스 재시작
- defaults (가변 변수) : 메인 태스크에서 사용된 변수 선언
    - service_title
- vars (불변 변수) : 메인 태스크와 핸들러에서 사용된 변수 선언
    - service_name : 서비스명
    - src_file_path : 복사할 파일 경로
    - dest_file_path : 파일이 복사될 디렉터리 경로
    - httpd_packages : httpd 관련 패키지 목록
    - supported_distros : 지원 OS 목록
```

# 메인 테스크 작성
# my-ansible/my-role/tasks/main.yml
---
# tasks file for my-role

- name: install service {{ service_title }}
  ansible.builtin.apt:
    name: "{{ item }}"
    state: latest
  loop: "{{ httpd_packages }}"
  when: ansible_facts.distribution in supported_distros

- name: copy conf file
  ansible.builtin.copy:
    src: "{{ src_file_path }}"
    dest: "{{ dest_file_path }}"
  notify: 
    - restart service

# index.html 정적파일 생성
# my-ansible/my-role/files/index.html
root@server:~/my-ansible/my-role# echo "Hello! Ansible" > files/index.html

# 핸들러 작성 
# 특정 태스크가 끝나고 그 다음에 수행해야하는 태스크, service 모듈 이용하여 서비스를 재시작
# my-ansible/my-role/handlers/main.yml
---
# handlers file for my-role

- name: restart service
  ansible.builtin.service:
    name: "{{ service_name }}"
    state: restarted

# default 작성 
# 외부로부터 재정의될 수 있는 기반 변수
root@server:~/my-ansible/my-role# echo 'service_title: "Apache Web Server"' >> defaults/main.yml

# vars 작성
# 한번 정의되면 외부로부터 변수 값을 수정할 수 없음, 롤 내의 플레이북에서만 사용되는 변수로 정의하는 것이 좋음
---
# vars file for my-role

service_name: apache2
src_file_path: ../files/index.html
dest_file_path: /var/www/html
httpd_packages:
  - apache2
  - apache2-doc

supported_distros:
  - Ubuntu


# 플레이북에 롤 추가하기
# 롤 실행을 위해 롤을 호출해주는 플레이북 필요
# my-ansible/role-example.yml
---
- hosts: tnode1

  tasks:
    - name: Print start play
      ansible.builtin.debug:
        msg: "Let's start role play"

    - name: Install Service by role
      ansible.builtin.import_role:
        name: my-role


# 실행 - my-role 내의 각 태스트와 핸들러가 순차적으로 수행
root@server:~/my-ansible# ansible-playbook role-example.yml

PLAY [tnode1] ***************************************************************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************************
ok: [tnode1]

TASK [Print start play] *****************************************************************************************************************************************************************
ok: [tnode1] => {
    "msg": "Let's start role play"
}

TASK [my-role : install service Apache Web Server] **************************************************************************************************************************************
changed: [tnode1] => (item=apache2)
changed: [tnode1] => (item=apache2-doc)

TASK [my-role : copy conf file] *********************************************************************************************************************************************************
changed: [tnode1]

RUNNING HANDLER [my-role : restart service] *********************************************************************************************************************************************
changed: [tnode1]

PLAY RECAP ******************************************************************************************************************************************************************************
tnode1                     : ok=5    changed=3    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

[K8s Deploy] Bootstrap Kubernetes the hard way

안녕유지 — Sat, 10 Jan 2026 17:57:52 +0900

Cloudnet K8s Deploy 1주차 스터디를 진행하며 정리한 글입니다.

이번 포스팅에서는 Kubernetes의 내부 동작을 보다 깊이 이해하기 위해, 자동화된 설치 도구를 벗어나 Kubernetes the Hard Way 방식으로 클러스터를 구성해 봅니다.

CA, TLS 인증서, kubeconfig를 직접 구성하며 Kubernetes 구성 요소 간 mTLS 통신 구조 이해
주요 컴포넌트를 systemd 서비스로 기동하여 실제 운영 환경과 유사한 구조 확인
kind Kubernetes와의 비교를 통해 설정 방식과 동작 차이 분석
macOS / Windows(WSL2)에서 동일하게 실습 가능한 Vagrant 기반 환경 제공
스크립트 자동화 대신 단계별 명령 실행과 설정 파일 설명 중심으로 실습 진행

이번 포스팅에서 구성할 클러스터는 아래와 같습니다.

Name	Description	CPU	RAM	NIC1	NIC2	Hostname
jumpbox	Administration host	2	1536 MB	10.0.2.15	192.168.10.10	jumpbox
server	Kubernetes server	2	2GB	10.0.2.15	192.168.10.100	server.kubernetes.local server
node-0	Kubernetes worker	2	2GB	10.0.2.15	192.168.10.101	node-0.kubernetes.local node-0
node-1	Kubernetes worker	2	2GB	10.0.2.15	192.168.10.102	node-1.kubernetes.local node-1

실습을 진행한 자료는 아래 Github 내용을 바탕으로 진행하였습니다.

https://github.com/kelseyhightower/kubernetes-the-hard-way

GitHub - kelseyhightower/kubernetes-the-hard-way: Bootstrap Kubernetes the hard way. No scripts.

Bootstrap Kubernetes the hard way. No scripts. Contribute to kelseyhightower/kubernetes-the-hard-way development by creating an account on GitHub.

github.com

기본 환경 설정 (Kind, Jumpbox 세팅)

kind와 krew 플러그인 도구를 설치합니다.

> kind create cluster --name myk8s --image kindest/node:v1.32.8 --config - <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 30000
    hostPort: 30000
  - containerPort: 30001
    hostPort: 30001
- role: worker
EOF


> kind get nodes --name myk8s                                                                                                                         13:40:04
myk8s-control-plane
myk8s-worker

> kubens default

VirtualBox와 Vagrant를 설치하고, 실습용 가상머신을 배포합니다.

# Vagrantfile , init_cfg.sh 파일 다운로드
> curl -O https://raw.githubusercontent.com/gasida/vagrant-lab/refs/heads/main/k8s-hardway/Vagrantfile
> curl -O https://raw.githubusercontent.com/gasida/vagrant-lab/refs/heads/main/k8s-hardway/init_cfg.sh

# 실습용 가상 머신 배포
> vagrant up

# 실습용 OS 이미지 자동 다운로드 확인
> vagrant box list                                                                                                                                 5m 30s 13:36:01
bento/debian-12    (virtualbox, 202510.26.0, (arm64))
bento/ubuntu-24.04 (virtualbox, 202502.21.0, (arm64))
bento/ubuntu-24.04 (virtualbox, 202508.03.0, (arm64))

# 배포된 가상머신 확인
> vagrant status                                                                                                                                          13:37:31
Current machine states:

jumpbox                   running (virtualbox)
server                    running (virtualbox)
node-0                    running (virtualbox)
node-1                    running (virtualbox)

This environment represents multiple VMs. The VMs are all listed
above with their current state. For more information about a specific
VM, run `vagrant status NAME`.

> vagrant ssh jumpbox                                                                                                                                 13:41:19
Linux jumpbox 6.1.0-40-arm64 #1 SMP Debian 6.1.153-1 (2025-09-20) aarch64

This system is built by the Bento project by Chef Software
More information can be found at https://github.com/chef/bento

Use of this system is acceptance of the OS vendor EULA and License Agreements.

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
root@jumpbox:~#

Jumpbox에 필요한 파일들을 설치합니다.

설치 항목	버전
k8s 관련 (kube-apiserver, kubelet 등)	1.32.2
etcd	3.6.0
containerd	2.1.0
runc	1.3.0

## vagrant 계정 로그인 시 'sudo su -' 실행으로 root 계정 전환됨
> cat /home/vagrant/.bashrc | tail -n 1
sudo su -

> apt-get update && apt install tree git jq yq unzip vim sshpass -y
> pwd
> git clone --depth 1 https://github.com/kelseyhightower/kubernetes-the-hard-way.git
> cd kubernetes-the-hard-way
> tree
.
├── ca.conf
├── configs
│   ├── 10-bridge.conf
│   ├── 99-loopback.conf
│   ├── containerd-config.toml
│   ├── encryption-config.yaml
│   ├── kube-apiserver-to-kubelet.yaml
│   ├── kubelet-config.yaml
│   ├── kube-proxy-config.yaml
│   └── kube-scheduler.yaml
├── CONTRIBUTING.md
├── COPYRIGHT.md
├── docs
│   ├── 01-prerequisites.md
│   ├── 02-jumpbox.md
│   ├── 03-compute-resources.md
│   ├── 04-certificate-authority.md
│   ├── 05-kubernetes-configuration-files.md
│   ├── 06-data-encryption-keys.md
│   ├── 07-bootstrapping-etcd.md
│   ├── 08-bootstrapping-kubernetes-controllers.md
│   ├── 09-bootstrapping-kubernetes-workers.md
│   ├── 10-configuring-kubectl.md
│   ├── 11-pod-network-routes.md
│   ├── 12-smoke-test.md
│   └── 13-cleanup.md
├── downloads-amd64.txt
├── downloads-arm64.txt
├── LICENSE
├── README.md
└── units
    ├── containerd.service
    ├── etcd.service
    ├── kube-apiserver.service
    ├── kube-controller-manager.service
    ├── kubelet.service
    ├── kube-proxy.service
    └── kube-scheduler.service
    

# CPU 아키텍처 확인
> dpkg --print-architecture
arm64

# CPU 아키텍처 별 다운로드 목록 정보 다름
> cat downloads-$(dpkg --print-architecture).txt


# wget 으로 다운로드 실행
> wget -q --show-progress \
  --https-only \
  --timestamping \
  -P downloads \
  -i downloads-$(dpkg --print-architecture).txt

Computing Resource 프로비저닝

SSH 접속 환경 설정을 구성합니다.

# Machine Database (서버 속성 저장 파일) : IPV4_ADDRESS FQDN HOSTNAME POD_SUBNET
## 참고) server(controlplane)는 kubelet 동작하지 않아서, 파드 네트워크 대역 설정 필요 없음
> cat <<EOF > machines.txt
192.168.10.100 server.kubernetes.local server
192.168.10.101 node-0.kubernetes.local node-0 10.200.0.0/24
192.168.10.102 node-1.kubernetes.local node-1 10.200.1.0/24
EOF

> while read IP FQDN HOST SUBNET; do
  echo "${IP} ${FQDN} ${HOST} ${SUBNET}"
done < machines.txt
192.168.10.100 server.kubernetes.local server
192.168.10.101 node-0.kubernetes.local node-0 10.200.0.0/24
192.168.10.102 node-1.kubernetes.local node-1 10.200.1.0/24



# Configuring SSH Access 설정

# sshd config 설정 파일 확인 : 이미 암호 기반 인증 접속 설정 되어 있음
> grep "^[^#]" /etc/ssh/sshd_config
Include /etc/ssh/sshd_config.d/*.conf
KbdInteractiveAuthentication no
UsePAM yes
X11Forwarding yes
PrintMotd no
AcceptEnv LANG LC_*
Subsystem	sftp	/usr/lib/openssh/sftp-server
UseDNS no
GSSAPIAuthentication no
PasswordAuthentication yes
PermitRootLogin yes

# Generate a new SSH key
> ssh-keygen -t rsa -N "" -f /root/.ssh/id_rsa
Generating public/private rsa key pair.
Your identification has been saved in /root/.ssh/id_rsa
Your public key has been saved in /root/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:fcS3G/41qQLQ94d8DyJFtucYtWin1EOMgBj5pU5lTV8 root@jumpbox
The key's randomart image is:
+---[RSA 3072]----+
|      .+ ..+.o  E|
|      o . +.=.+. |
|       ..= oo*o. |
|       .+o oB.*. |
|       oS o++B+. |
|        ....+=o=.|
|          .. .=+o|
|           .  ..+|
|            ..  .|
+----[SHA256]-----+

> ls -l /root/.ssh
total 8
-rw------- 1 root root 2602 Jan 10 13:51 id_rsa
-rw-r--r-- 1 root root  566 Jan 10 13:51 id_rsa.pub


# Copy the SSH public key to each machine
> while read IP FQDN HOST SUBNET; do
  sshpass -p 'qwe123' ssh-copy-id -o StrictHostKeyChecking=no root@${IP}
done < machines.txt
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh -o 'StrictHostKeyChecking=no' 'root@192.168.10.100'"
and check to make sure that only the key(s) you wanted were added.

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh -o 'StrictHostKeyChecking=no' 'root@192.168.10.101'"
and check to make sure that only the key(s) you wanted were added.

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh -o 'StrictHostKeyChecking=no' 'root@192.168.10.102'"
and check to make sure that only the key(s) you wanted were added.


> while read IP FQDN HOST SUBNET; do
  ssh -n root@${IP} cat /root/.ssh/authorized_keys
done < machines.txt
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDzGyBAxcCEBJtrtNhudPuHchrV7a5t7GRWNq4r7ew/KztHsScspWB3tJIeDd4+TQgb5qexNUmwslBqoxYW6Lp8zoGiFJrVoj9GFO1O4iZimnHCIqzhufmkYp/FQuDnEsQi7tPQAjYPTojZfhvgpW4Z09+gPWpyl40P9KVOvGETtocLAdJACcL+I6uq+2JFKG6GnDHbER7zhXYngJKAizj/2t/Dx6l6SC/RfR8Z8jLUBgEguy8RAPQI8k20qYP4ONl77J8/3aQ7XJQptFODfQRmve1KRf8o509AjoDyktOG7LyAFxxlN8RVEDW5gN857WIzjOdnzirAIkt3+6ExrNIyb/XF4JIHWu36HfuAPUusEzIsP6V6+NxbrwQbZvi4wNgCl5OzQFVjtiDlCoa1DzG6rn4onQzW/OgxWoLENe7M4+5w1zS6hGZ/7OW33sO/DeFZOwl6mrdhoXiVBi10+M4sB/CI55a7aQ5J6VcPogxZq6slcoAFrXdiasyJaAPGXjc= 
root@jumpbox
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDzGyBAxcCEBJtrtNhudPuHchrV7a5t7GRWNq4r7ew/KztHsScspWB3tJIeDd4+TQgb5qexNUmwslBqoxYW6Lp8zoGiFJrVoj9GFO1O4iZimnHCIqzhufmkYp/FQuDnEsQi7tPQAjYPTojZfhvgpW4Z09+gPWpyl40P9KVOvGETtocLAdJACcL+I6uq+2JFKG6GnDHbER7zhXYngJKAizj/2t/Dx6l6SC/RfR8Z8jLUBgEguy8RAPQI8k20qYP4ONl77J8/3aQ7XJQptFODfQRmve1KRf8o509AjoDyktOG7LyAFxxlN8RVEDW5gN857WIzjOdnzirAIkt3+6ExrNIyb/XF4JIHWu36HfuAPUusEzIsP6V6+NxbrwQbZvi4wNgCl5OzQFVjtiDlCoa1DzG6rn4onQzW/OgxWoLENe7M4+5w1zS6hGZ/7OW33sO/DeFZOwl6mrdhoXiVBi10+M4sB/CI55a7aQ5J6VcPogxZq6slcoAFrXdiasyJaAPGXjc= 
root@jumpbox
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDzGyBAxcCEBJtrtNhudPuHchrV7a5t7GRWNq4r7ew/KztHsScspWB3tJIeDd4+TQgb5qexNUmwslBqoxYW6Lp8zoGiFJrVoj9GFO1O4iZimnHCIqzhufmkYp/FQuDnEsQi7tPQAjYPTojZfhvgpW4Z09+gPWpyl40P9KVOvGETtocLAdJACcL+I6uq+2JFKG6GnDHbER7zhXYngJKAizj/2t/Dx6l6SC/RfR8Z8jLUBgEguy8RAPQI8k20qYP4ONl77J8/3aQ7XJQptFODfQRmve1KRf8o509AjoDyktOG7LyAFxxlN8RVEDW5gN857WIzjOdnzirAIkt3+6ExrNIyb/XF4JIHWu36HfuAPUusEzIsP6V6+NxbrwQbZvi4wNgCl5OzQFVjtiDlCoa1DzG6rn4onQzW/OgxWoLENe7M4+5w1zS6hGZ/7OW33sO/DeFZOwl6mrdhoXiVBi10+M4sB/CI55a7aQ5J6VcPogxZq6slcoAFrXdiasyJaAPGXjc= 
root@jumpbox


# 아래는 IP 기반으로 접속 확인
> while read IP FQDN HOST SUBNET; do
  ssh -n root@${IP} hostname
done < machines.txt
server
node-0
node-1


> while read IP FQDN HOST SUBNET; do
  ssh -n root@${IP} cat /etc/hosts
done < machines.txt
127.0.0.1	localhost

# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
192.168.10.10  jumpbox
192.168.10.100 server.kubernetes.local server
192.168.10.101 node-0.kubernetes.local node-0
192.168.10.102 node-1.kubernetes.local node-1
127.0.0.1	localhost

# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
192.168.10.10  jumpbox
192.168.10.100 server.kubernetes.local server
192.168.10.101 node-0.kubernetes.local node-0
192.168.10.102 node-1.kubernetes.local node-1
127.0.0.1	localhost

# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
192.168.10.10  jumpbox
192.168.10.100 server.kubernetes.local server
192.168.10.101 node-0.kubernetes.local node-0
192.168.10.102 node-1.kubernetes.local node-1


# Hostnames 설정

> while read IP FQDN HOST SUBNET; do
  ssh -n root@${IP} cat /etc/hosts
done < machines.txt
127.0.0.1	localhost

# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
192.168.10.10  jumpbox
192.168.10.100 server.kubernetes.local server
192.168.10.101 node-0.kubernetes.local node-0
192.168.10.102 node-1.kubernetes.local node-1
127.0.0.1	localhost

# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
192.168.10.10  jumpbox
192.168.10.100 server.kubernetes.local server
192.168.10.101 node-0.kubernetes.local node-0
192.168.10.102 node-1.kubernetes.local node-1
127.0.0.1	localhost

# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
192.168.10.10  jumpbox
192.168.10.100 server.kubernetes.local server
192.168.10.101 node-0.kubernetes.local node-0
192.168.10.102 node-1.kubernetes.local node-1

> while read IP FQDN HOST SUBNET; do
  ssh -n root@${IP} hostname --fqdn
done < machines.txt
server.kubernetes.local
node-0.kubernetes.local
node-1.kubernetes.local

> cat /etc/hosts
while read IP FQDN HOST SUBNET; do
  sshpass -p 'qwe123' ssh -n -o StrictHostKeyChecking=no root@${HOST} hostname
done < machines.txt
127.0.0.1	localhost

# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
192.168.10.10  jumpbox
192.168.10.100 server.kubernetes.local server
192.168.10.101 node-0.kubernetes.local node-0
192.168.10.102 node-1.kubernetes.local node-1
Warning: Permanently added 'server' (ED25519) to the list of known hosts.
server
Warning: Permanently added 'node-0' (ED25519) to the list of known hosts.
node-0
Warning: Permanently added 'node-1' (ED25519) to the list of known hosts.
node-1


> while read IP FQDN HOST SUBNET; do
  sshpass -p 'qwe123' ssh -n root@${HOST} uname -o -m -n
done < machines.txt
server aarch64 GNU/Linux
node-0 aarch64 GNU/Linux
node-1 aarch64 GNU/Linux

CA 생성 및 TLS 인증서 발급

Kubernetes 클러스터를 구성하는 각 구성 요소가 mTLS 통신을 하기 위해 어떤 인증서(키·CSR·CRT)를 가지고, 어떤 정체성(CN/O)과 용도로 사용되는지를 정리한 표입니다.

항목	개인키	CSR	인증서	참고 정보	X509v3 Extended Key Usage
Root CA	ca.key	X	ca.crt
admin	admin.key	admin.csr	admin.crt	CN = admin, O = system:masters	TLS Web Client Authentication
node-0	node-0.key	node-0.csr	node-0.crt	CN = system:node:node-0, O = system:nodes	TLS Web Server / Client Authentication
node-1	node-1.key	node-1.csr	node-1.crt	CN = system:node:node-1, O = system:nodes	TLS Web Server / Client Authentication
kube-proxy	kube-proxy.key	kube-proxy.csr	kube-proxy.crt	CN = system:kube-proxy, O = system:node-proxier	TLS Web Server / Client Authentication
kube-scheduler	kube-scheduler.key	kube-scheduler	kube-scheduler.crt	CN = system:kube-scheduler, O = system:kube-scheduler	TLS Web Server / Client Authentication
kube-controller-manager	kube-controller-manager.key	kube-controller-manager.csr	kube-controller-manager.crt	CN = system:kube-controller-manager, O = system:kube-controller-manager	TLS Web Server / Client Authentication
kube-api-server	kube-api-server.key	kube-api-server.csr	kube-api-server.crt	CN = kubernetes, SAN: IP(127.0.0.1, 10.32.0.1), DNS(kubernetes,..)	TLS Web Server / Client Authentication
service-accounts	service-accounts.key	service-accounts.csr	service-accounts.crt	CN = service-accounts	TLS Web Client Authentication

구성한 네트워크 대역은 다음과 같습니다.

항목	네트워크 대역 or IP
clusterCIDR	10.200.0.0/16
→ node-0 PodCIDR	10.200.0.0/24
→ node-1 PodCIDR	10.200.1.0/24
ServiceCIDR	10.32.0.0/24
→api clusterIP	10.32.0.1

참고를 위해 kind k8s에서 인증서 관련 파일을 확인해보겠습니다.

> docker exec -it myk8s-control-plane sh -c 'apt update && apt install tree yq jq -y'

> docker exec -i myk8s-control-plane kubeadm certs check-expiration                                                                                    5s 06:59:25

[check-expiration] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[check-expiration] Use 'kubeadm init phase upload-config --config your-config.yaml' to re-upload it.

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Jan 10, 2027 04:03 UTC   364d            ca                      no
apiserver                  Jan 10, 2027 04:03 UTC   364d            ca                      no
apiserver-etcd-client      Jan 10, 2027 04:03 UTC   364d            etcd-ca                 no
apiserver-kubelet-client   Jan 10, 2027 04:03 UTC   364d            ca                      no
controller-manager.conf    Jan 10, 2027 04:03 UTC   364d            ca                      no
etcd-healthcheck-client    Jan 10, 2027 04:03 UTC   364d            etcd-ca                 no
etcd-peer                  Jan 10, 2027 04:03 UTC   364d            etcd-ca                 no
etcd-server                Jan 10, 2027 04:03 UTC   364d            etcd-ca                 no
front-proxy-client         Jan 10, 2027 04:03 UTC   364d            front-proxy-ca          no
scheduler.conf             Jan 10, 2027 04:03 UTC   364d            ca                      no
super-admin.conf           Jan 10, 2027 04:03 UTC   364d            ca                      no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Jan 08, 2036 04:03 UTC   9y              no
etcd-ca                 Jan 08, 2036 04:03 UTC   9y              no
front-proxy-ca          Jan 08, 2036 04:03 UTC   9y              no

> docker exec -it myk8s-control-plane tree /etc/kubernetes                                                                                                06:59:49

/etc/kubernetes
|-- admin.conf
|-- controller-manager.conf
|-- kubelet.conf
|-- manifests
|   |-- etcd.yaml
|   |-- kube-apiserver.yaml
|   |-- kube-controller-manager.yaml
|   `-- kube-scheduler.yaml
|-- pki
|   |-- apiserver-etcd-client.crt
|   |-- apiserver-etcd-client.key
|   |-- apiserver-kubelet-client.crt
|   |-- apiserver-kubelet-client.key
|   |-- apiserver.crt
|   |-- apiserver.key
|   |-- ca.crt
|   |-- ca.key
|   |-- etcd
|   |   |-- ca.crt
|   |   |-- ca.key
|   |   |-- healthcheck-client.crt
|   |   |-- healthcheck-client.key
|   |   |-- peer.crt
|   |   |-- peer.key
|   |   |-- server.crt
|   |   `-- server.key
|   |-- front-proxy-ca.crt
|   |-- front-proxy-ca.key
|   |-- front-proxy-client.crt
|   |-- front-proxy-client.key
|   |-- sa.key
|   `-- sa.pub
|-- scheduler.conf
`-- super-admin.conf

4 directories, 31 files

CA 설정파일, certificate, private key를 생성하겠습니다.

우선 certificate authority를 생성하겠습니다.

CN(Common Name)은 X.509 인증서의 식별자 이름 필드로 인증서 주체(또는 발급자)를 사람이 알아보기 쉽게 부르는 이름입니다.

Issuer는 이 인증서를 발급한 CA의 DN, Subject는 이 인증서의 소유자 DN입니다.

따라서, 현재 이 인증서는 self-signed CA으로, Issuer와 Subject가 CN=CA입니다.

root@jumpbox:~/kubernetes-the-hard-way# openssl genrsa -out ca.key 4096
root@jumpbox:~/kubernetes-the-hard-way# ls -l ca.key
-rw------- 1 root root 3272 Jan 11 07:04 ca.key
root@jumpbox:~/kubernetes-the-hard-way# cat ca.key
-----BEGIN PRIVATE KEY-----
MIIJRAIBADANBgkqhkiG9w0BAQEFAASCCS4wggkqAgEAAoICAQCe03U+G0LIGAZd
GlXG5uInPOFL35RAls8fXKDFjeT1R4F+BZX8mVDR9kxuijGKzZzkAY+4nNiahOcW
Z/eoSBRw5CwVRtIUiRFJA0CgjZ1O8wMgQrnH77GTqrtieJcZdAPu3AAY3uCi2THr
cUfQ6UEygoAY0+rQx2oXQRmsqq7JMJllI6sOxyTqj7amUjl542HlzQSIAQ2Vi1b1
BbLOsLxUr1xcYcJJePOFOw0kAXqQWjlpMx+YXsLEAtOMlCKwnhUljD4a5N0pqDQz
igr8y3Ykt6kgjGLG348wkXmArGNS3lPZrzN+7osFP08rtN3a6C4sYU7y60Fo4Tw4
Jhzp/SCWpG12sYgDO68Do4p9FfSA8JzGjrtBM/Iu0JkQR5nKha+ERJBQudzcfk+x
b3TbCi0Tj8n8bbTvx3pdODhXn+lLxPjj0KbZ7C5bKjoyxBjZhWYYfDSovwuCAIP0
xOq8DWFg+zyPjrqpFZuwyDnWEWVaTrKg7kT8fnBQecFP4u5Wi2L2lDj+FnzuHoyw
KU7IL6UTcUjZj9ICmN9YoX224iHF9yvC6emu74v51X3HrLOf0j+oHxm9orLWcofo
Q3AOc7uJ7sX7nWChpmrQRZ74fHmDYSm7R8/688674rTKYsGJdpNsAoJIyijhlmil
AHQAxSSiyzofVzpEXnCXpmViTLD5kQIDAQABAoICACOJnQowf1jWR1Y9tkOzFZdC
AAdCF6H8HIwf5blObpSTU14nUdAbXgIc4i05rqwoJZq+3rGoJq8sXaidysRUnH5G
D7cTCZ6erMdxetxx02j+5wwo4fjioRExeowl/2kqlFa10y+9bvYBMzJDPpZ4GqZ6
nL7xa/EbLZnWbzA7YLCTtz7yJCxCUvhI/81nSIUFY7pPjVwBIFD9xI8+nyuwT/OC
6V+iYB6geZyuAdq0mgACsCXh0of9bAb/uuYRn1F1jbXBE4RhATt473U4xgyaWe0N
S0KFJvTowNlVM6Ke7/py1MA4e4VDSsf5ynU1Gw8INIypmUixawlUnGzT2lRg1xGD
o6Jjec5usxBLEDoW81NB/IHicn0RxLMnlGwhI4fn2qvOEmIs9j7QkvISNSxUX5+I
/6hE8EQq7SYQcj1BgnkT2vuyEGoBLAnV3PbWfwC89P4+a4ZtsV/7sK66hgTl4c9P
y5B5/oPt6pxYF2T2XeDkP6Szj77hnDCFovfcClqrOPDB9s4HRUTpszdMSykTpjoG
z+9z7JWSTHd/IAaF7wK/3HGfUIfHEh6EV7+jokxdYG3ixswTcqkaplN8tqtJdPKw
Q4wSmBkNfckpTtoWyi47a5GaENg/wDGgGKyyWLPOIQdXxHUTcAIUcY0eUl9cWNCa
hNR7ZqFWyxC6jCN6cSOBAoIBAQDPTqwANmqVlJ93B+OGVKKaJ4ao+nJHSFXR3fN/
nSC8i2gphIlcQS42GsHz4OAYwR+Q1nAX1f5hUmOwdj2TRQsgKBfYSrb4vjjjVK6v
xbPGLC5gn+TJA9J8kt8JelFa4GoOvOBJ/GBhrQwK3gRtaeLtj92io2ab90XbrvXZ
bvVfGc+9zPEzgS3I0WdQDTvWome4avOpP30ULQqVay2XkLl2slO/q7cs0mRhHEV4
H+gp4mswbmUwiW1IFThYCDvOuE4ir3A+WZKyOhTJjF2nsnHGiaMB0RhUa3cC6O1N
DgJDBHlX1EVxINF3z1DnduPTVpv3qL/jq9yn+GGfynCYQvQNAoIBAQDEIZ6TN91+
Rfu9VI+ozwkfBDIW1nbc/Y8KDQd0nWVK14LiPDyxZdhXQm+97jouzcmq45rHr5QM
nCfwo45SeitOA8f4tQewbz5Q1V9td3kOUHva4wf0f0bSAEQLPiYOJPn35pxRMNfH
XuRg1lqMVHGdvVoS/ZR24TEh2fjZjhEMqMAir14QUN8313qTqVziK6Hbm7qJxDcM
RcGf18edtVlV5D/kyucCGSeRUjdJodLB+nCw22i/RvCLhdp2nFXsp67Jmt45Gz7m
YlX3ZM5oU1dWTKELJ8QGutAcphI+P6QhvKlUlO3Xee4+YOC9PWXezsc43hCJ2Qqa
ddhavH9YuyaVAoIBAQCBH0taxooUqQG6Lhu1a7nrsnrZfUT4yUFK+qfT8JaXWW2D
jp3P7HlQEdKEiKiMQp3apPYkopFTJPIa00r/E54eZVqUydS1kiX00qYLrWSQJqys
WmsyGUNdfE2XpaxOokDs9Q/uhVey/wnlKuwvCijiNx2hU41xz86fSpau28HRfk46
tg/2OJu/3U4wOk3vHjDkWRlzbawGiRdygdNf+DDMkb5AQEBTlqgXJHjQnW8Jx5Fk
FR7J4I9EupLhoh8+XuWsDQIok7ofcLXJ47fh3++l7cyuMQKqclUqZtKHkBYwj/TU
k8686+gMbADEavT0iZmT5HPA5ycFKFwGHx2KeQlxAoIBAQChZrlElXik7ZqY1/nR
BW+QIrfFq3/tpIwNzWMRSGWPDYMcAk4//nvmcznZpy4iTh9jjiJVqdu/9NG5ah85
KGhyrxHlTbSiMSmrA6ygPO1WJk2Gd8aYIhV5w9Y7Gk3e3fcHVrPU7MjlvOVb+4pQ
aT2GfcieUCfZHMB+Bw6OmibVsBlNcduKucK4BY9N7NYPGbD9NE5yCRLz+oO4sq4G
zrJRyJ/7/C3WwDSblnkBEzaJgzsxj53Hqu1RJDn/2e+r7OId7mBpxk4g8VOlX0Os
/siksDLnJzZeA2Y37UKeD2lWihcfD+AarrCsjLTlhp1KmgfAeJ5Lklzzd38wTGbP
8lNxAoIBAQCNikVjxJ7XChzch3b6rIcDM3cfL7Et7XDKXMJa/Q9HQy1Lm2tgFVJA
YmHaFpNiq9CVRmyZ6pxpXMEDjClifdpi1utl8LHanDvJRHtKxi0d5oogl4E2wnD7
uVxZu7pX7KjsQSan0YV9Z4/DoW0gKBljoVOxOsUMwrevCN9jJjelo2vyP3bT6F0V
jtxleALdKGcvFL0sSRk3BFQzS6SOgl/7V5CpWvxq7axgArvCabfAG9MAvLMuuXMQ
sRRJ87cgboc7i6nutfldMhWeXM1kFJgvnOwTu8mBEHLOGTkUKewkAgT2EdafIkCK
nCdtHCf+g7W4SvE216mOC6MjpU+IP/+S
-----END PRIVATE KEY-----
root@jumpbox:~/kubernetes-the-hard-way#
root@jumpbox:~/kubernetes-the-hard-way# openssl rsa -in ca.key -text -noout # 개인키 구조 확인
Private-Key: (4096 bit, 2 primes)
modulus:
    00:9e:d3:75:3e:1b:42:c8:18:06:5d:1a:55:c6:e6:
    e2:27:3c:e1:4b:df:94:40:96:cf:1f:5c:a0:c5:8d:
    e4:f5:47:81:7e:05:95:fc:99:50:d1:f6:4c:6e:8a:
    31:8a:cd:9c:e4:01:8f:b8:9c:d8:9a:84:e7:16:67:
    f7:a8:48:14:70:e4:2c:15:46:d2:14:89:11:49:03:
    40:a0:8d:9d:4e:f3:03:20:42:b9:c7:ef:b1:93:aa:
    bb:62:78:97:19:74:03:ee:dc:00:18:de:e0:a2:d9:
    31:eb:71:47:d0:e9:41:32:82:80:18:d3:ea:d0:c7:
    6a:17:41:19:ac:aa:ae:c9:30:99:65:23:ab:0e:c7:
    24:ea:8f:b6:a6:52:39:79:e3:61:e5:cd:04:88:01:
    0d:95:8b:56:f5:05:b2:ce:b0:bc:54:af:5c:5c:61:
    c2:49:78:f3:85:3b:0d:24:01:7a:90:5a:39:69:33:
    1f:98:5e:c2:c4:02:d3:8c:94:22:b0:9e:15:25:8c:
    3e:1a:e4:dd:29:a8:34:33:8a:0a:fc:cb:76:24:b7:
    a9:20:8c:62:c6:df:8f:30:91:79:80:ac:63:52:de:
    53:d9:af:33:7e:ee:8b:05:3f:4f:2b:b4:dd:da:e8:
    2e:2c:61:4e:f2:eb:41:68:e1:3c:38:26:1c:e9:fd:
    20:96:a4:6d:76:b1:88:03:3b:af:03:a3:8a:7d:15:
    f4:80:f0:9c:c6:8e:bb:41:33:f2:2e:d0:99:10:47:
    99:ca:85:af:84:44:90:50:b9:dc:dc:7e:4f:b1:6f:
    74:db:0a:2d:13:8f:c9:fc:6d:b4:ef:c7:7a:5d:38:
    38:57:9f:e9:4b:c4:f8:e3:d0:a6:d9:ec:2e:5b:2a:
    3a:32:c4:18:d9:85:66:18:7c:34:a8:bf:0b:82:00:
    83:f4:c4:ea:bc:0d:61:60:fb:3c:8f:8e:ba:a9:15:
    9b:b0:c8:39:d6:11:65:5a:4e:b2:a0:ee:44:fc:7e:
    70:50:79:c1:4f:e2:ee:56:8b:62:f6:94:38:fe:16:
    7c:ee:1e:8c:b0:29:4e:c8:2f:a5:13:71:48:d9:8f:
    d2:02:98:df:58:a1:7d:b6:e2:21:c5:f7:2b:c2:e9:
    e9:ae:ef:8b:f9:d5:7d:c7:ac:b3:9f:d2:3f:a8:1f:
    19:bd:a2:b2:d6:72:87:e8:43:70:0e:73:bb:89:ee:
    c5:fb:9d:60:a1:a6:6a:d0:45:9e:f8:7c:79:83:61:
    29:bb:47:cf:fa:f3:ce:bb:e2:b4:ca:62:c1:89:76:
    93:6c:02:82:48:ca:28:e1:96:68:a5:00:74:00:c5:
    24:a2:cb:3a:1f:57:3a:44:5e:70:97:a6:65:62:4c:
    b0:f9:91
publicExponent: 65537 (0x10001)
privateExponent:
    23:89:9d:0a:30:7f:58:d6:47:56:3d:b6:43:b3:15:
    97:42:00:07:42:17:a1:fc:1c:8c:1f:e5:b9:4e:6e:
    94:93:53:5e:27:51:d0:1b:5e:02:1c:e2:2d:39:ae:
    ac:28:25:9a:be:de:b1:a8:26:af:2c:5d:a8:9d:ca:
    c4:54:9c:7e:46:0f:b7:13:09:9e:9e:ac:c7:71:7a:
    dc:71:d3:68:fe:e7:0c:28:e1:f8:e2:a1:11:31:7a:
    8c:25:ff:69:2a:94:56:b5:d3:2f:bd:6e:f6:01:33:
    32:43:3e:96:78:1a:a6:7a:9c:be:f1:6b:f1:1b:2d:
    99:d6:6f:30:3b:60:b0:93:b7:3e:f2:24:2c:42:52:
    f8:48:ff:cd:67:48:85:05:63:ba:4f:8d:5c:01:20:
    50:fd:c4:8f:3e:9f:2b:b0:4f:f3:82:e9:5f:a2:60:
    1e:a0:79:9c:ae:01:da:b4:9a:00:02:b0:25:e1:d2:
    87:fd:6c:06:ff:ba:e6:11:9f:51:75:8d:b5:c1:13:
    84:61:01:3b:78:ef:75:38:c6:0c:9a:59:ed:0d:4b:
    42:85:26:f4:e8:c0:d9:55:33:a2:9e:ef:fa:72:d4:
    c0:38:7b:85:43:4a:c7:f9:ca:75:35:1b:0f:08:34:
    8c:a9:99:48:b1:6b:09:54:9c:6c:d3:da:54:60:d7:
    11:83:a3:a2:63:79:ce:6e:b3:10:4b:10:3a:16:f3:
    53:41:fc:81:e2:72:7d:11:c4:b3:27:94:6c:21:23:
    87:e7:da:ab:ce:12:62:2c:f6:3e:d0:92:f2:12:35:
    2c:54:5f:9f:88:ff:a8:44:f0:44:2a:ed:26:10:72:
    3d:41:82:79:13:da:fb:b2:10:6a:01:2c:09:d5:dc:
    f6:d6:7f:00:bc:f4:fe:3e:6b:86:6d:b1:5f:fb:b0:
    ae:ba:86:04:e5:e1:cf:4f:cb:90:79:fe:83:ed:ea:
    9c:58:17:64:f6:5d:e0:e4:3f:a4:b3:8f:be:e1:9c:
    30:85:a2:f7:dc:0a:5a:ab:38:f0:c1:f6:ce:07:45:
    44:e9:b3:37:4c:4b:29:13:a6:3a:06:cf:ef:73:ec:
    95:92:4c:77:7f:20:06:85:ef:02:bf:dc:71:9f:50:
    87:c7:12:1e:84:57:bf:a3:a2:4c:5d:60:6d:e2:c6:
    cc:13:72:a9:1a:a6:53:7c:b6:ab:49:74:f2:b0:43:
    8c:12:98:19:0d:7d:c9:29:4e:da:16:ca:2e:3b:6b:
    91:9a:10:d8:3f:c0:31:a0:18:ac:b2:58:b3:ce:21:
    07:57:c4:75:13:70:02:14:71:8d:1e:52:5f:5c:58:
    d0:9a:84:d4:7b:66:a1:56:cb:10:ba:8c:23:7a:71:
    23:81
prime1:
    00:cf:4e:ac:00:36:6a:95:94:9f:77:07:e3:86:54:
    a2:9a:27:86:a8:fa:72:47:48:55:d1:dd:f3:7f:9d:
    20:bc:8b:68:29:84:89:5c:41:2e:36:1a:c1:f3:e0:
    e0:18:c1:1f:90:d6:70:17:d5:fe:61:52:63:b0:76:
    3d:93:45:0b:20:28:17:d8:4a:b6:f8:be:38:e3:54:
    ae:af:c5:b3:c6:2c:2e:60:9f:e4:c9:03:d2:7c:92:
    df:09:7a:51:5a:e0:6a:0e:bc:e0:49:fc:60:61:ad:
    0c:0a:de:04:6d:69:e2:ed:8f:dd:a2:a3:66:9b:f7:
    45:db:ae:f5:d9:6e:f5:5f:19:cf:bd:cc:f1:33:81:
    2d:c8:d1:67:50:0d:3b:d6:a2:67:b8:6a:f3:a9:3f:
    7d:14:2d:0a:95:6b:2d:97:90:b9:76:b2:53:bf:ab:
    b7:2c:d2:64:61:1c:45:78:1f:e8:29:e2:6b:30:6e:
    65:30:89:6d:48:15:38:58:08:3b:ce:b8:4e:22:af:
    70:3e:59:92:b2:3a:14:c9:8c:5d:a7:b2:71:c6:89:
    a3:01:d1:18:54:6b:77:02:e8:ed:4d:0e:02:43:04:
    79:57:d4:45:71:20:d1:77:cf:50:e7:76:e3:d3:56:
    9b:f7:a8:bf:e3:ab:dc:a7:f8:61:9f:ca:70:98:42:
    f4:0d
prime2:
    00:c4:21:9e:93:37:dd:7e:45:fb:bd:54:8f:a8:cf:
    09:1f:04:32:16:d6:76:dc:fd:8f:0a:0d:07:74:9d:
    65:4a:d7:82:e2:3c:3c:b1:65:d8:57:42:6f:bd:ee:
    3a:2e:cd:c9:aa:e3:9a:c7:af:94:0c:9c:27:f0:a3:
    8e:52:7a:2b:4e:03:c7:f8:b5:07:b0:6f:3e:50:d5:
    5f:6d:77:79:0e:50:7b:da:e3:07:f4:7f:46:d2:00:
    44:0b:3e:26:0e:24:f9:f7:e6:9c:51:30:d7:c7:5e:
    e4:60:d6:5a:8c:54:71:9d:bd:5a:12:fd:94:76:e1:
    31:21:d9:f8:d9:8e:11:0c:a8:c0:22:af:5e:10:50:
    df:37:d7:7a:93:a9:5c:e2:2b:a1:db:9b:ba:89:c4:
    37:0c:45:c1:9f:d7:c7:9d:b5:59:55:e4:3f:e4:ca:
    e7:02:19:27:91:52:37:49:a1:d2:c1:fa:70:b0:db:
    68:bf:46:f0:8b:85:da:76:9c:55:ec:a7:ae:c9:9a:
    de:39:1b:3e:e6:62:55:f7:64:ce:68:53:57:56:4c:
    a1:0b:27:c4:06:ba:d0:1c:a6:12:3e:3f:a4:21:bc:
    a9:54:94:ed:d7:79:ee:3e:60:e0:bd:3d:65:de:ce:
    c7:38:de:10:89:d9:0a:9a:75:d8:5a:bc:7f:58:bb:
    26:95
exponent1:
    00:81:1f:4b:5a:c6:8a:14:a9:01:ba:2e:1b:b5:6b:
    b9:eb:b2:7a:d9:7d:44:f8:c9:41:4a:fa:a7:d3:f0:
    96:97:59:6d:83:8e:9d:cf:ec:79:50:11:d2:84:88:
    a8:8c:42:9d:da:a4:f6:24:a2:91:53:24:f2:1a:d3:
    4a:ff:13:9e:1e:65:5a:94:c9:d4:b5:92:25:f4:d2:
    a6:0b:ad:64:90:26:ac:ac:5a:6b:32:19:43:5d:7c:
    4d:97:a5:ac:4e:a2:40:ec:f5:0f:ee:85:57:b2:ff:
    09:e5:2a:ec:2f:0a:28:e2:37:1d:a1:53:8d:71:cf:
    ce:9f:4a:96:ae:db:c1:d1:7e:4e:3a:b6:0f:f6:38:
    9b:bf:dd:4e:30:3a:4d:ef:1e:30:e4:59:19:73:6d:
    ac:06:89:17:72:81:d3:5f:f8:30:cc:91:be:40:40:
    40:53:96:a8:17:24:78:d0:9d:6f:09:c7:91:64:15:
    1e:c9:e0:8f:44:ba:92:e1:a2:1f:3e:5e:e5:ac:0d:
    02:28:93:ba:1f:70:b5:c9:e3:b7:e1:df:ef:a5:ed:
    cc:ae:31:02:aa:72:55:2a:66:d2:87:90:16:30:8f:
    f4:d4:93:ce:bc:eb:e8:0c:6c:00:c4:6a:f4:f4:89:
    99:93:e4:73:c0:e7:27:05:28:5c:06:1f:1d:8a:79:
    09:71
exponent2:
    00:a1:66:b9:44:95:78:a4:ed:9a:98:d7:f9:d1:05:
    6f:90:22:b7:c5:ab:7f:ed:a4:8c:0d:cd:63:11:48:
    65:8f:0d:83:1c:02:4e:3f:fe:7b:e6:73:39:d9:a7:
    2e:22:4e:1f:63:8e:22:55:a9:db:bf:f4:d1:b9:6a:
    1f:39:28:68:72:af:11:e5:4d:b4:a2:31:29:ab:03:
    ac:a0:3c:ed:56:26:4d:86:77:c6:98:22:15:79:c3:
    d6:3b:1a:4d:de:dd:f7:07:56:b3:d4:ec:c8:e5:bc:
    e5:5b:fb:8a:50:69:3d:86:7d:c8:9e:50:27:d9:1c:
    c0:7e:07:0e:8e:9a:26:d5:b0:19:4d:71:db:8a:b9:
    c2:b8:05:8f:4d:ec:d6:0f:19:b0:fd:34:4e:72:09:
    12:f3:fa:83:b8:b2:ae:06:ce:b2:51:c8:9f:fb:fc:
    2d:d6:c0:34:9b:96:79:01:13:36:89:83:3b:31:8f:
    9d:c7:aa:ed:51:24:39:ff:d9:ef:ab:ec:e2:1d:ee:
    60:69:c6:4e:20:f1:53:a5:5f:43:ac:fe:c8:a4:b0:
    32:e7:27:36:5e:03:66:37:ed:42:9e:0f:69:56:8a:
    17:1f:0f:e0:1a:ae:b0:ac:8c:b4:e5:86:9d:4a:9a:
    07:c0:78:9e:4b:92:5c:f3:77:7f:30:4c:66:cf:f2:
    53:71
coefficient:
    00:8d:8a:45:63:c4:9e:d7:0a:1c:dc:87:76:fa:ac:
    87:03:33:77:1f:2f:b1:2d:ed:70:ca:5c:c2:5a:fd:
    0f:47:43:2d:4b:9b:6b:60:15:52:40:62:61:da:16:
    93:62:ab:d0:95:46:6c:99:ea:9c:69:5c:c1:03:8c:
    29:62:7d:da:62:d6:eb:65:f0:b1:da:9c:3b:c9:44:
    7b:4a:c6:2d:1d:e6:8a:20:97:81:36:c2:70:fb:b9:
    5c:59:bb:ba:57:ec:a8:ec:41:26:a7:d1:85:7d:67:
    8f:c3:a1:6d:20:28:19:63:a1:53:b1:3a:c5:0c:c2:
    b7:af:08:df:63:26:37:a5:a3:6b:f2:3f:76:d3:e8:
    5d:15:8e:dc:65:78:02:dd:28:67:2f:14:bd:2c:49:
    19:37:04:54:33:4b:a4:8e:82:5f:fb:57:90:a9:5a:
    fc:6a:ed:ac:60:02:bb:c2:69:b7:c0:1b:d3:00:bc:
    b3:2e:b9:73:10:b1:14:49:f3:b7:20:6e:87:3b:8b:
    a9:ee:b5:f9:5d:32:15:9e:5c:cd:64:14:98:2f:9c:
    ec:13:bb:c9:81:10:72:ce:19:39:14:29:ec:24:02:
    04:f6:11:d6:9f:22:40:8a:9c:27:6d:1c:27:fe:83:
    b5:b8:4a:f1:36:d7:a9:8e:0b:a3:23:a5:4f:88:3f:
    ff:92
    
    
# Root CA 인증서 생성 : ca.crt
## -x509 : CSR을 만들지 않고 바로 인증서(X.509) 생성, 즉, Self-Signed Certificate
## -noenc : 개인키를 암호화하지 않음, 즉, CA 키(ca.key)에 패스프레이즈 없음
## -config ca.conf : 인증서 세부 정보는 설정 파일에서 읽음 , [req] 섹션 사용됨 - DN 정보 → [req_distinguished_name] , CA 확장 → [ca_x509_extensions]
root@jumpbox:~/kubernetes-the-hard-way# openssl req -x509 -new -sha512 -noenc \
  -key ca.key -days 3653 \
  -config ca.conf \
  -out ca.crt
root@jumpbox:~/kubernetes-the-hard-way# ls -l ca.crt
-rw-r--r-- 1 root root 1899 Jan 11 07:05 ca.crt


# ca.conf 내용 확인
root@jumpbox:~/kubernetes-the-hard-way# cat ca.conf
[req]
distinguished_name = req_distinguished_name
prompt             = no
x509_extensions    = ca_x509_extensions

[ca_x509_extensions]
basicConstraints = CA:TRUE # 이 인증서는 CA 역할 가능
keyUsage         = cRLSign, keyCertSign # cRLSign: 인증서 폐기 목록(CRL) 서명 가능, keyCertSign: 다른 인증서를 서명할 수 있음

[req_distinguished_name]
C   = US
ST  = Washington
L   = Seattle
CN  = CA

[admin]
distinguished_name = admin_distinguished_name
prompt             = no
req_extensions     = default_req_extensions

[admin_distinguished_name]
CN = admin
O  = system:masters

# Service Accounts
#
# The Kubernetes Controller Manager leverages a key pair to generate
# and sign service account tokens as described in the
# [managing service accounts](https://kubernetes.io/docs/admin/service-accounts-admin/)
# documentation.

[service-accounts]
distinguished_name = service-accounts_distinguished_name
prompt             = no
req_extensions     = default_req_extensions

[service-accounts_distinguished_name]
CN = service-accounts

# Worker Nodes
#
# Kubernetes uses a [special-purpose authorization mode](https://kubernetes.io/docs/admin/authorization/node/)
# called Node Authorizer, that specifically authorizes API requests made
# by [Kubelets](https://kubernetes.io/docs/concepts/overview/components/#kubelet).
# In order to be authorized by the Node Authorizer, Kubelets must use a credential
# that identifies them as being in the `system:nodes` group, with a username
# of `system:node:<nodeName>`.

[node-0]
distinguished_name = node-0_distinguished_name
prompt             = no
req_extensions     = node-0_req_extensions

[node-0_req_extensions]
basicConstraints     = CA:FALSE
extendedKeyUsage     = clientAuth, serverAuth
keyUsage             = critical, digitalSignature, keyEncipherment
nsCertType           = client
nsComment            = "Node-0 Certificate"
subjectAltName       = DNS:node-0, IP:127.0.0.1
subjectKeyIdentifier = hash

[node-0_distinguished_name]
CN = system:node:node-0
O  = system:nodes
C  = US
ST = Washington
L  = Seattle

[node-1]
distinguished_name = node-1_distinguished_name
prompt             = no
req_extensions     = node-1_req_extensions

[node-1_req_extensions]
basicConstraints     = CA:FALSE
extendedKeyUsage     = clientAuth, serverAuth
keyUsage             = critical, digitalSignature, keyEncipherment
nsCertType           = client
nsComment            = "Node-1 Certificate"
subjectAltName       = DNS:node-1, IP:127.0.0.1
subjectKeyIdentifier = hash

[node-1_distinguished_name]
CN = system:node:node-1
O  = system:nodes
C  = US
ST = Washington
L  = Seattle


# Kube Proxy Section
[kube-proxy]
distinguished_name = kube-proxy_distinguished_name
prompt             = no
req_extensions     = kube-proxy_req_extensions

[kube-proxy_req_extensions]
basicConstraints     = CA:FALSE
extendedKeyUsage     = clientAuth, serverAuth
keyUsage             = critical, digitalSignature, keyEncipherment
nsCertType           = client
nsComment            = "Kube Proxy Certificate"
subjectAltName       = DNS:kube-proxy, IP:127.0.0.1
subjectKeyIdentifier = hash

[kube-proxy_distinguished_name]
CN = system:kube-proxy
O  = system:node-proxier
C  = US
ST = Washington
L  = Seattle


# Controller Manager
[kube-controller-manager]
distinguished_name = kube-controller-manager_distinguished_name
prompt             = no
req_extensions     = kube-controller-manager_req_extensions

[kube-controller-manager_req_extensions]
basicConstraints     = CA:FALSE
extendedKeyUsage     = clientAuth, serverAuth
keyUsage             = critical, digitalSignature, keyEncipherment
nsCertType           = client
nsComment            = "Kube Controller Manager Certificate"
subjectAltName       = DNS:kube-controller-manager, IP:127.0.0.1
subjectKeyIdentifier = hash

[kube-controller-manager_distinguished_name]
CN = system:kube-controller-manager
O  = system:kube-controller-manager
C  = US
ST = Washington
L  = Seattle


# Scheduler
[kube-scheduler]
distinguished_name = kube-scheduler_distinguished_name
prompt             = no
req_extensions     = kube-scheduler_req_extensions

[kube-scheduler_req_extensions]
basicConstraints     = CA:FALSE
extendedKeyUsage     = clientAuth, serverAuth
keyUsage             = critical, digitalSignature, keyEncipherment
nsCertType           = client
nsComment            = "Kube Scheduler Certificate"
subjectAltName       = DNS:kube-scheduler, IP:127.0.0.1
subjectKeyIdentifier = hash

[kube-scheduler_distinguished_name]
CN = system:kube-scheduler
O  = system:system:kube-scheduler
C  = US
ST = Washington
L  = Seattle


# API Server
#
# The Kubernetes API server is automatically assigned the `kubernetes`
# internal dns name, which will be linked to the first IP address (`10.32.0.1`)
# from the address range (`10.32.0.0/24`) reserved for internal cluster
# services.

[kube-api-server]
distinguished_name = kube-api-server_distinguished_name
prompt             = no
req_extensions     = kube-api-server_req_extensions

[kube-api-server_req_extensions]
basicConstraints     = CA:FALSE
extendedKeyUsage     = clientAuth, serverAuth
keyUsage             = critical, digitalSignature, keyEncipherment
nsCertType           = client, server
nsComment            = "Kube API Server Certificate"
subjectAltName       = @kube-api-server_alt_names
subjectKeyIdentifier = hash

[kube-api-server_alt_names]
IP.0  = 127.0.0.1
IP.1  = 10.32.0.1
DNS.0 = kubernetes
DNS.1 = kubernetes.default
DNS.2 = kubernetes.default.svc
DNS.3 = kubernetes.default.svc.cluster
DNS.4 = kubernetes.svc.cluster.local
DNS.5 = server.kubernetes.local
DNS.6 = api-server.kubernetes.local

[kube-api-server_distinguished_name]
CN = kubernetes
C  = US
ST = Washington
L  = Seattle


[default_req_extensions]
basicConstraints     = CA:FALSE
extendedKeyUsage     = clientAuth
keyUsage             = critical, digitalSignature, keyEncipherment
nsCertType           = client
nsComment            = "Admin Client Certificate"
subjectKeyIdentifier = hash



root@jumpbox:~/kubernetes-the-hard-way# cat ca.crt
-----BEGIN CERTIFICATE-----
MIIFTDCCAzSgAwIBAgIUcSMTtyKoWqyd+YXYFjUVawNiOjkwDQYJKoZIhvcNAQEN
BQAwQTELMAkGA1UEBhMCVVMxEzARBgNVBAgMCldhc2hpbmd0b24xEDAOBgNVBAcM
B1NlYXR0bGUxCzAJBgNVBAMMAkNBMB4XDTI2MDExMDIyMDU1NloXDTM2MDExMTIy
MDU1NlowQTELMAkGA1UEBhMCVVMxEzARBgNVBAgMCldhc2hpbmd0b24xEDAOBgNV
BAcMB1NlYXR0bGUxCzAJBgNVBAMMAkNBMIICIjANBgkqhkiG9w0BAQEFAAOCAg8A
MIICCgKCAgEAntN1PhtCyBgGXRpVxubiJzzhS9+UQJbPH1ygxY3k9UeBfgWV/JlQ
0fZMbooxis2c5AGPuJzYmoTnFmf3qEgUcOQsFUbSFIkRSQNAoI2dTvMDIEK5x++x
k6q7YniXGXQD7twAGN7gotkx63FH0OlBMoKAGNPq0MdqF0EZrKquyTCZZSOrDsck
6o+2plI5eeNh5c0EiAENlYtW9QWyzrC8VK9cXGHCSXjzhTsNJAF6kFo5aTMfmF7C
xALTjJQisJ4VJYw+GuTdKag0M4oK/Mt2JLepIIxixt+PMJF5gKxjUt5T2a8zfu6L
BT9PK7Td2uguLGFO8utBaOE8OCYc6f0glqRtdrGIAzuvA6OKfRX0gPCcxo67QTPy
LtCZEEeZyoWvhESQULnc3H5PsW902wotE4/J/G2078d6XTg4V5/pS8T449Cm2ewu
Wyo6MsQY2YVmGHw0qL8LggCD9MTqvA1hYPs8j466qRWbsMg51hFlWk6yoO5E/H5w
UHnBT+LuVoti9pQ4/hZ87h6MsClOyC+lE3FI2Y/SApjfWKF9tuIhxfcrwunpru+L
+dV9x6yzn9I/qB8ZvaKy1nKH6ENwDnO7ie7F+51goaZq0EWe+Hx5g2Epu0fP+vPO
u+K0ymLBiXaTbAKCSMoo4ZZopQB0AMUkoss6H1c6RF5wl6ZlYkyw+ZECAwEAAaM8
MDowDAYDVR0TBAUwAwEB/zALBgNVHQ8EBAMCAQYwHQYDVR0OBBYEFJ7NNroia0On
2yMpl1Mb0XLfbWHSMA0GCSqGSIb3DQEBDQUAA4ICAQBwpr8+8niGE9UjAKQG2ANk
xzHnMPQNyN85zEqrB0Zx3kiHBtE/FhzQtxh/Idfz3E3tSI8nTMm4lH62rvPvc54F
19K1gWWzhxLkZC+tK20JUUSyuRdKEeO6SHptQYp8FdMeaYYLI/cHuArdS6L9zMia
BL4YXGpm1q3yXyTkwugHRSpYqyBzP8LksDJ57Oa2a+7CLZau5fw3JJODRDsuy9tT
XFKe3OeVA/H6g/AC0S9JkRCNuJ88q8n4KHCE4rK06uA+VIThymwZttylNHCAEF6w
kjmn45JwHKvRcf2m5ad9U5cVwJynBbvvW6/e1qNTh2ftopE/QUkhz3RO/5iLCpHB
MR6VenjcNEa86Nz2Q5V8Tf/8RXzNZ6GSPAAxlPCS43dZJLeVTscvm0TFVsFndBKq
iNYvcPlrXztv+0MRq9zdUjPKX6o/9/dA9y1doFBPQ1aqEhgPdVBrjzrMn7WiEgkg
eaxVVZbz/msqaAujFgEGZgey8AVPB7YcJJ+yuqRWlLnbtzyv1KzXguK+Xb6Ak4Et
yOjlqhH6SJ6oI8072WpqzyghDpoH9kyTjH//1HEK5VNuu1mF10ekwNUBOpBDDuyy
3CKm1RhdPyXF8mILp9OE80koq5K6tFwNS/hDERtQWt/s1CvtqKkSqdYRUIs+0sqt
YTBe2HubYTjmiGN38NfyNA==
-----END CERTIFICATE-----
root@jumpbox:~/kubernetes-the-hard-way# openssl x509 -in ca.crt -text -noout # 인증서 전체 내용 확인
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            71:23:13:b7:22:a8:5a:ac:9d:f9:85:d8:16:35:15:6b:03:62:3a:39
        Signature Algorithm: sha512WithRSAEncryption
        Issuer: C = US, ST = Washington, L = Seattle, CN = CA # kind는 kubernetes임
        Validity
            Not Before: Jan 10 22:05:56 2026 GMT
            Not After : Jan 11 22:05:56 2036 GMT
        Subject: C = US, ST = Washington, L = Seattle, CN = CA # kind는 kubernetes임
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)
                Modulus:
                    00:9e:d3:75:3e:1b:42:c8:18:06:5d:1a:55:c6:e6:
                    e2:27:3c:e1:4b:df:94:40:96:cf:1f:5c:a0:c5:8d:
                    e4:f5:47:81:7e:05:95:fc:99:50:d1:f6:4c:6e:8a:
                    31:8a:cd:9c:e4:01:8f:b8:9c:d8:9a:84:e7:16:67:
                    f7:a8:48:14:70:e4:2c:15:46:d2:14:89:11:49:03:
                    40:a0:8d:9d:4e:f3:03:20:42:b9:c7:ef:b1:93:aa:
                    bb:62:78:97:19:74:03:ee:dc:00:18:de:e0:a2:d9:
                    31:eb:71:47:d0:e9:41:32:82:80:18:d3:ea:d0:c7:
                    6a:17:41:19:ac:aa:ae:c9:30:99:65:23:ab:0e:c7:
                    24:ea:8f:b6:a6:52:39:79:e3:61:e5:cd:04:88:01:
                    0d:95:8b:56:f5:05:b2:ce:b0:bc:54:af:5c:5c:61:
                    c2:49:78:f3:85:3b:0d:24:01:7a:90:5a:39:69:33:
                    1f:98:5e:c2:c4:02:d3:8c:94:22:b0:9e:15:25:8c:
                    3e:1a:e4:dd:29:a8:34:33:8a:0a:fc:cb:76:24:b7:
                    a9:20:8c:62:c6:df:8f:30:91:79:80:ac:63:52:de:
                    53:d9:af:33:7e:ee:8b:05:3f:4f:2b:b4:dd:da:e8:
                    2e:2c:61:4e:f2:eb:41:68:e1:3c:38:26:1c:e9:fd:
                    20:96:a4:6d:76:b1:88:03:3b:af:03:a3:8a:7d:15:
                    f4:80:f0:9c:c6:8e:bb:41:33:f2:2e:d0:99:10:47:
                    99:ca:85:af:84:44:90:50:b9:dc:dc:7e:4f:b1:6f:
                    74:db:0a:2d:13:8f:c9:fc:6d:b4:ef:c7:7a:5d:38:
                    38:57:9f:e9:4b:c4:f8:e3:d0:a6:d9:ec:2e:5b:2a:
                    3a:32:c4:18:d9:85:66:18:7c:34:a8:bf:0b:82:00:
                    83:f4:c4:ea:bc:0d:61:60:fb:3c:8f:8e:ba:a9:15:
                    9b:b0:c8:39:d6:11:65:5a:4e:b2:a0:ee:44:fc:7e:
                    70:50:79:c1:4f:e2:ee:56:8b:62:f6:94:38:fe:16:
                    7c:ee:1e:8c:b0:29:4e:c8:2f:a5:13:71:48:d9:8f:
                    d2:02:98:df:58:a1:7d:b6:e2:21:c5:f7:2b:c2:e9:
                    e9:ae:ef:8b:f9:d5:7d:c7:ac:b3:9f:d2:3f:a8:1f:
                    19:bd:a2:b2:d6:72:87:e8:43:70:0e:73:bb:89:ee:
                    c5:fb:9d:60:a1:a6:6a:d0:45:9e:f8:7c:79:83:61:
                    29:bb:47:cf:fa:f3:ce:bb:e2:b4:ca:62:c1:89:76:
                    93:6c:02:82:48:ca:28:e1:96:68:a5:00:74:00:c5:
                    24:a2:cb:3a:1f:57:3a:44:5e:70:97:a6:65:62:4c:
                    b0:f9:91
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:TRUE
            X509v3 Key Usage:
                Certificate Sign, CRL Sign
            X509v3 Subject Key Identifier:
                9E:CD:36:BA:22:6B:43:A7:DB:23:29:97:53:1B:D1:72:DF:6D:61:D2
    Signature Algorithm: sha512WithRSAEncryption
    Signature Value:
        70:a6:bf:3e:f2:78:86:13:d5:23:00:a4:06:d8:03:64:c7:31:
        e7:30:f4:0d:c8:df:39:cc:4a:ab:07:46:71:de:48:87:06:d1:
        3f:16:1c:d0:b7:18:7f:21:d7:f3:dc:4d:ed:48:8f:27:4c:c9:
        b8:94:7e:b6:ae:f3:ef:73:9e:05:d7:d2:b5:81:65:b3:87:12:
        e4:64:2f:ad:2b:6d:09:51:44:b2:b9:17:4a:11:e3:ba:48:7a:
        6d:41:8a:7c:15:d3:1e:69:86:0b:23:f7:07:b8:0a:dd:4b:a2:
        fd:cc:c8:9a:04:be:18:5c:6a:66:d6:ad:f2:5f:24:e4:c2:e8:
        07:45:2a:58:ab:20:73:3f:c2:e4:b0:32:79:ec:e6:b6:6b:ee:
        c2:2d:96:ae:e5:fc:37:24:93:83:44:3b:2e:cb:db:53:5c:52:
        9e:dc:e7:95:03:f1:fa:83:f0:02:d1:2f:49:91:10:8d:b8:9f:
        3c:ab:c9:f8:28:70:84:e2:b2:b4:ea:e0:3e:54:84:e1:ca:6c:
        19:b6:dc:a5:34:70:80:10:5e:b0:92:39:a7:e3:92:70:1c:ab:
        d1:71:fd:a6:e5:a7:7d:53:97:15:c0:9c:a7:05:bb:ef:5b:af:
        de:d6:a3:53:87:67:ed:a2:91:3f:41:49:21:cf:74:4e:ff:98:
        8b:0a:91:c1:31:1e:95:7a:78:dc:34:46:bc:e8:dc:f6:43:95:
        7c:4d:ff:fc:45:7c:cd:67:a1:92:3c:00:31:94:f0:92:e3:77:
        59:24:b7:95:4e:c7:2f:9b:44:c5:56:c1:67:74:12:aa:88:d6:
        2f:70:f9:6b:5f:3b:6f:fb:43:11:ab:dc:dd:52:33:ca:5f:aa:
        3f:f7:f7:40:f7:2d:5d:a0:50:4f:43:56:aa:12:18:0f:75:50:
        6b:8f:3a:cc:9f:b5:a2:12:09:20:79:ac:55:55:96:f3:fe:6b:
        2a:68:0b:a3:16:01:06:66:07:b2:f0:05:4f:07:b6:1c:24:9f:
        b2:ba:a4:56:94:b9:db:b7:3c:af:d4:ac:d7:82:e2:be:5d:be:
        80:93:81:2d:c8:e8:e5:aa:11:fa:48:9e:a8:23:cd:3b:d9:6a:
        6a:cf:28:21:0e:9a:07:f6:4c:93:8c:7f:ff:d4:71:0a:e5:53:
        6e:bb:59:85:d7:47:a4:c0:d5:01:3a:90:43:0e:ec:b2:dc:22:
        a6:d5:18:5d:3f:25:c5:f2:62:0b:a7:d3:84:f3:49:28:ab:92:
        ba:b4:5c:0d:4b:f8:43:11:1b:50:5a:df:ec:d4:2b:ed:a8:a9:
        12:a9:d6:11:50:8b:3e:d2:ca:ad:61:30:5e:d8:7b:9b:61:38:
        e6:88:63:77:f0:d7:f2:34

Client, Server admin 인증서 생성하겠습니다.

root@jumpbox:~/kubernetes-the-hard-way# openssl genrsa -out admin.key 4096
root@jumpbox:~/kubernetes-the-hard-way# ls -l admin.key
-rw------- 1 root root 3272 Jan 11 07:19 admin.key
root@jumpbox:~/kubernetes-the-hard-way# cat ca.conf
[req]
distinguished_name = req_distinguished_name
prompt             = no
x509_extensions    = ca_x509_extensions

[ca_x509_extensions]
basicConstraints = CA:TRUE
keyUsage         = cRLSign, keyCertSign

[req_distinguished_name]
C   = US
ST  = Washington
L   = Seattle
CN  = CA

[admin] # ca.conf 에 admin 섹션 확인
distinguished_name = admin_distinguished_name
prompt             = no
req_extensions     = default_req_extensions

[admin_distinguished_name]
CN = admin
O  = system:masters

...



# csr 파일 생성 : admin.key 개인키를 사용해 'CN=admin, O=system:masters'인 Kubernetes 관리자용 클라이언트 인증서 요청(admin.csr) 생성
root@jumpbox:~/kubernetes-the-hard-way# openssl req -new -key admin.key -sha256 \
  -config ca.conf -section admin \
  -out admin.csr
root@jumpbox:~/kubernetes-the-hard-way#
root@jumpbox:~/kubernetes-the-hard-way# ls -l admin.csr
-rw-r--r-- 1 root root 1830 Jan 11 07:20 admin.csr
root@jumpbox:~/kubernetes-the-hard-way# openssl req -in admin.csr -text -noout # CSR 전체 내용 확인

Certificate Request:
    Data:
        Version: 1 (0x0)
        Subject: CN = admin, O = system:masters
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)
                Modulus:
                    00:b7:77:93:30:ae:01:ce:08:55:60:66:4e:35:a4:
                    c2:97:cc:10:22:d6:76:7c:02:20:3d:16:c7:11:a5:
                    ce:fa:44:5b:9c:7e:92:39:fa:50:2b:a1:82:ff:e2:
                    14:6f:80:74:7e:e1:58:14:fe:3f:06:c4:2c:1d:81:
                    7c:05:24:12:5f:ed:2d:84:b3:5b:11:7b:b6:ae:00:
                    5d:10:8c:a2:76:8c:37:a1:a3:67:34:75:cb:d3:66:
                    f5:64:86:0c:bc:7b:34:40:ef:e5:07:02:86:30:7e:
                    9b:47:67:97:2d:23:62:95:9d:75:55:b4:d0:33:1a:
                    34:85:da:e9:34:e7:cf:1e:89:2b:a0:c8:25:05:7a:
                    13:5d:94:6d:a1:c7:a3:4d:dd:f8:46:1f:8e:03:4c:
                    97:b3:41:83:9b:bb:08:59:f7:03:25:b6:07:9d:ea:
                    ed:0c:26:f4:22:ba:35:00:64:34:f9:9a:f9:ed:26:
                    e4:35:3c:f9:79:5f:37:69:59:ed:8b:db:7e:f7:42:
                    0c:15:ff:bb:1a:3e:f9:dd:b1:79:56:d1:96:b9:98:
                    ee:a0:76:52:49:8d:a9:73:26:c1:97:6b:fe:7a:eb:
                    39:24:3e:97:f7:e8:c2:85:4c:e3:99:8c:98:f5:76:
                    4f:8d:b1:b2:3e:45:5c:7a:db:c9:47:bd:95:3f:57:
                    09:ad:98:4d:3c:62:f1:82:d3:79:f0:99:f6:95:47:
                    58:c1:e3:e5:e1:f8:2a:a7:d9:f2:27:ae:d9:09:00:
                    6e:bb:96:50:b2:f4:ab:ac:84:db:27:f0:6d:01:8b:
                    26:fc:58:f7:8a:70:bc:e3:0e:c3:49:1e:b2:a2:67:
                    8e:c9:9b:1d:8a:43:90:0a:33:ed:d7:f9:70:f5:0a:
                    61:02:d9:99:49:1d:88:3c:c8:1b:e5:4c:75:28:99:
                    12:9e:6c:83:45:0b:f7:47:ac:ed:4d:31:df:f2:6d:
                    9e:99:26:0a:37:cd:31:3f:d3:b6:80:e4:c9:02:be:
                    d1:21:21:dd:48:0a:03:a8:39:40:14:1d:e7:5b:e2:
                    5b:0e:6b:9c:11:e1:58:da:28:bf:6f:ae:8d:d9:65:
                    0c:d3:e7:ef:00:6e:ee:7c:83:1c:be:e3:10:4b:c3:
                    77:e0:06:2e:54:f7:8a:8d:5c:bc:e9:8f:dc:d4:fb:
                    92:5d:f5:e5:f9:0c:ff:1a:a9:27:bf:28:ae:fd:f9:
                    e8:25:5b:02:11:cf:df:59:f0:ad:f8:d8:1a:cb:ed:
                    9b:00:45:b7:2e:76:63:b3:7b:e2:0f:44:75:a7:e6:
                    a8:eb:53:24:9e:95:a0:56:1b:71:0c:f6:10:a1:b0:
                    e9:c0:bd:c5:d8:3d:9b:d5:04:81:23:ba:7b:e3:88:
                    84:2c:b5
                Exponent: 65537 (0x10001)
        Attributes:
            Requested Extensions:
                X509v3 Basic Constraints:
                    CA:FALSE
                X509v3 Extended Key Usage:
                    TLS Web Client Authentication
                X509v3 Key Usage: critical
                    Digital Signature, Key Encipherment
                Netscape Cert Type:
                    SSL Client
                Netscape Comment:
                    Admin Client Certificate
                X509v3 Subject Key Identifier:
                    4C:56:38:B9:6E:E5:81:0A:70:F1:C9:02:CB:48:5C:8D:CE:B4:65:18
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        65:29:3a:20:5b:28:a2:86:5f:54:aa:0c:e5:57:11:9c:9c:87:
        b0:97:35:1b:f9:a0:f2:3d:c5:6f:e6:b3:73:fc:42:f7:b4:22:
        84:54:58:67:f9:7c:aa:18:e5:d7:2e:06:61:be:e7:53:8c:3c:
        87:49:09:ed:05:f5:b1:03:0a:cd:a8:52:df:a1:f4:92:db:ec:
        c0:f0:b8:ba:2b:10:b5:6b:eb:6a:e5:ec:08:f3:cf:3a:4d:0a:
        07:42:d2:a1:b3:3a:18:7d:d8:ae:07:04:c0:e2:fd:78:b8:31:
        0d:7a:d8:c1:aa:c2:d9:61:cd:ee:2e:6a:e8:bd:83:67:74:28:
        c7:1d:ca:c6:39:cd:cf:8c:3d:d6:57:63:0e:32:ea:0f:58:f5:
        43:db:76:95:b2:34:7f:ef:6a:1e:cb:af:22:14:c0:35:59:dc:
        02:30:69:7f:55:4c:82:ff:b5:e0:ec:eb:2b:2a:9a:b7:9e:c8:
        7f:8e:14:a6:09:82:7c:fa:82:2e:28:57:2e:5d:ec:8e:63:e8:
        a3:69:0a:42:3f:31:e7:5e:7b:fb:09:d5:64:69:30:04:ff:b6:
        67:e0:49:64:a0:2d:c7:c9:18:98:12:9d:3e:ba:c3:d5:41:00:
        a1:89:28:94:0f:90:9e:b6:23:cd:62:c7:b6:dd:cc:f7:da:32:
        84:9d:d2:a5:37:9b:7e:a5:c2:45:65:1e:97:ab:a0:4d:3f:2c:
        ff:00:43:cf:20:c9:37:a6:a3:cf:4f:a8:47:5a:f4:f2:be:10:
        20:19:e6:ee:c7:e1:c4:d2:f2:c1:da:84:f7:23:5d:bb:e5:a3:
        0e:ae:19:d0:4a:1b:7d:76:ff:89:5d:02:b4:26:4f:72:ba:37:
        aa:02:9d:1c:0e:5b:9a:a3:ff:ba:a2:4f:2f:bb:8e:a7:fa:5a:
        8d:78:e1:8e:61:55:f0:48:d3:b8:64:64:db:50:98:e0:3d:66:
        d1:fc:2b:19:f8:8d:da:36:5e:cb:0c:ae:f4:ec:e5:10:62:3e:
        1c:c0:8a:e6:d1:0e:b0:dd:74:80:da:8f:77:76:76:d3:0b:95:
        0f:c4:19:60:7a:e5:49:72:6a:63:0c:9e:8d:46:03:47:d4:36:
        16:36:a7:7a:ee:26:d9:6e:8d:d3:fb:79:40:7b:ff:2c:bb:35:
        be:35:1f:da:47:cb:e3:4d:dd:d7:f4:35:e5:2a:29:f3:32:7e:
        8c:2c:ce:89:6a:ad:0e:a7:a0:8e:77:06:75:eb:0d:10:9a:96:
        5d:10:98:3e:e7:8e:bb:92:dc:6e:2f:ad:14:79:82:ec:bf:14:
        4e:42:08:6a:34:29:f0:a9:5a:e6:58:86:9d:ea:34:92:50:aa:
        c8:4a:33:39:89:02:fd:2e
        

# ca에 csr 요청을 통한 crt 파일 생성
## -req : CSR를 입력으로 받아 인증서를 생성, self-signed 아님, CA가 서명하는 방식
## -days 3653 : 인증서 유효기간 3653일 (약 10년)
## -copy_extensions copyall : CSR에 포함된 모든 X.509 extensions를 인증서로 복사
## -CAcreateserial : CA 시리얼 번호 파일 자동 생성, 다음 인증서 발급 시 재사용, 기본 생성 파일(ca.srl)
openssl x509 -req -days 3653 -in admin.csr \
  -copy_extensions copyall \
  -sha256 -CA ca.crt \
  -CAkey ca.key \
  -CAcreateserial \
  -out admin.crt
Certificate request self-signature ok
subject=CN = admin, O = system:masters


root@jumpbox:~/kubernetes-the-hard-way# ls -l admin.crt
-rw-r--r-- 1 root root 2021 Jan 11 07:25 admin.crt
root@jumpbox:~/kubernetes-the-hard-way# openssl x509 -in admin.crt -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            2b:03:f7:ba:63:e9:26:c8:a5:c1:9f:dd:8b:d8:df:4c:aa:56:f5:6f
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, ST = Washington, L = Seattle, CN = CA
        Validity
            Not Before: Jan 10 22:25:09 2026 GMT
            Not After : Jan 11 22:25:09 2036 GMT
        Subject: CN = admin, O = system:masters # admin

Client, Server 인증서 나머지 전부에 대하여 생성하겠습니다.

# ca.conf 수정
root@jumpbox:~/kubernetes-the-hard-way# cat ca.conf | grep system:kube-scheduler
CN = system:kube-scheduler
O  = system:system:kube-scheduler

root@jumpbox:~/kubernetes-the-hard-way# sed -i 's/system:system:kube-scheduler/system:kube-scheduler/' ca.conf
root@jumpbox:~/kubernetes-the-hard-way# cat ca.conf | grep system:kube-scheduler
CN = system:kube-scheduler
O  = system:kube-scheduler

# 변수 지정
root@jumpbox:~/kubernetes-the-hard-way# certs=(
  "node-0" "node-1"
  "kube-proxy" "kube-scheduler"
  "kube-controller-manager"
  "kube-api-server"
  "service-accounts"
)

# 확인
root@jumpbox:~/kubernetes-the-hard-way# echo ${certs[*]}
node-0 node-1 kube-proxy kube-scheduler kube-controller-manager kube-api-server service-accounts

# 개인키 생성, csr 생성, 인증서 생성
root@jumpbox:~/kubernetes-the-hard-way# for i in ${certs[*]}; do
  openssl genrsa -out "${i}.key" 4096

  openssl req -new -key "${i}.key" -sha256 \
    -config "ca.conf" -section ${i} \
    -out "${i}.csr"

  openssl x509 -req -days 3653 -in "${i}.csr" \
    -copy_extensions copyall \
    -sha256 -CA "ca.crt" \
    -CAkey "ca.key" \
    -CAcreateserial \
    -out "${i}.crt"
done
Certificate request self-signature ok
subject=CN = system:node:node-0, O = system:nodes, C = US, ST = Washington, L = Seattle
Certificate request self-signature ok
subject=CN = system:node:node-1, O = system:nodes, C = US, ST = Washington, L = Seattle
Certificate request self-signature ok
subject=CN = system:kube-proxy, O = system:node-proxier, C = US, ST = Washington, L = Seattle
Certificate request self-signature ok
subject=CN = system:kube-scheduler, O = system:kube-scheduler, C = US, ST = Washington, L = Seattle
Certificate request self-signature ok
subject=CN = system:kube-controller-manager, O = system:kube-controller-manager, C = US, ST = Washington, L = Seattle
Certificate request self-signature ok
subject=CN = kubernetes, C = US, ST = Washington, L = Seattle
Certificate request self-signature ok
subject=CN = service-accounts


root@jumpbox:~/kubernetes-the-hard-way# ls -1 *.crt *.key *.csr
admin.crt
admin.csr
admin.key
ca.crt
ca.key
kube-api-server.crt
kube-api-server.csr
kube-api-server.key
kube-controller-manager.crt
kube-controller-manager.csr
kube-controller-manager.key
kube-proxy.crt
kube-proxy.csr
kube-proxy.key
kube-scheduler.crt
kube-scheduler.csr
kube-scheduler.key
node-0.crt
node-0.csr
node-0.key
node-1.crt
node-1.csr
node-1.key
service-accounts.crt
service-accounts.csr
service-accounts.key


root@jumpbox:~/kubernetes-the-hard-way# openssl x509 -in node-0.crt -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            2b:03:f7:ba:63:e9:26:c8:a5:c1:9f:dd:8b:d8:df:4c:aa:56:f5:70
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, ST = Washington, L = Seattle, CN = CA
        Validity
            Not Before: Jan 10 22:28:48 2026 GMT
            Not After : Jan 11 22:28:48 2036 GMT
        Subject: CN = system:node:node-0, O = system:nodes, C = US, ST = Washington, L = Seattle
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)
                Modulus:
                    00:d3:fe:f2:c0:df:bb:08:76:6d:ef:1b:c5:25:e0:
                    16:66:be:d1:36:b0:2e:6b:9a:c5:d7:79:f9:36:da:
                    2b:cb:95:91:94:1b:75:92:f5:87:26:0b:a8:8c:e5:
                    a0:07:e6:c3:01:71:61:c1:f1:40:43:57:1d:63:bb:
                    51:c6:fd:81:ab:e0:83:40:b0:29:78:6f:22:30:63:
                    17:21:6e:41:31:37:39:b9:af:5f:8c:0c:ad:87:d8:
                    81:95:c6:d4:be:6e:f9:da:1b:1c:c8:a3:0f:72:3c:
                    77:2a:7a:1b:2b:71:40:85:a3:e4:fb:5e:d5:81:14:
                    a0:15:99:ce:6a:c4:1f:50:3a:4f:a9:10:17:bb:dc:
                    e9:e2:ba:03:48:e9:19:ec:8b:94:1d:d6:93:1b:57:
                    46:ee:25:39:d3:9b:3b:4a:1b:8c:71:ff:21:ff:60:
                    78:4f:b2:0a:10:cf:da:73:a9:4b:dc:f1:50:2d:c5:
                    82:0b:44:34:cb:40:87:dc:07:3b:88:1f:ce:c3:85:
                    c5:12:b7:f3:13:4d:77:e0:7d:60:a9:5b:ac:81:3c:
                    3f:79:63:b0:88:27:a9:92:e2:c2:bd:86:0e:e8:f2:
                    ed:bb:bf:2f:ed:14:f0:50:cf:bb:a3:8d:8e:8e:e5:
                    be:45:72:ad:83:78:1e:da:4d:4e:9f:d9:4f:6d:5b:
                    2b:25:8d:80:67:5b:bd:a5:13:b8:d3:1a:35:61:9c:
                    8b:c6:e7:b5:45:6e:d2:5e:82:b5:19:ee:6a:92:14:
                    b7:ef:04:66:8a:b3:94:0a:44:bf:d2:40:c3:62:7d:
                    15:f3:50:4b:9f:76:41:6c:87:15:5c:ac:74:26:cc:
                    c6:51:29:c0:e1:35:52:d1:8c:45:c1:93:17:e2:45:
                    88:8d:9f:23:4e:19:9d:4e:df:40:59:3a:f0:e7:12:
                    57:fe:da:72:f8:a7:1b:4e:be:45:75:5f:32:94:69:
                    11:02:4f:a9:6a:53:b9:23:62:88:42:b6:16:ec:f9:
                    64:31:ea:8d:6d:f1:f1:d1:da:ec:9f:b9:a9:da:a6:
                    27:39:e8:4e:e4:0f:62:11:e2:44:0c:aa:7c:44:19:
                    85:8f:29:35:b8:63:6d:df:85:9d:19:86:c3:49:6f:
                    ad:57:f6:03:3f:8f:c7:56:3b:24:a1:9b:c6:85:6d:
                    70:98:7b:46:2a:06:93:bc:88:0f:33:c4:3d:de:0c:
                    79:e2:c3:0f:68:57:2c:07:bc:b4:e4:04:f5:f8:0b:
                    ce:5e:b7:80:f5:bc:44:df:aa:eb:38:34:c4:a3:e1:
                    34:b3:24:21:42:f4:10:c8:13:a8:32:66:0f:c6:84:
                    bb:6b:ff:58:fa:b9:3d:16:c5:9c:f0:be:2d:9e:7b:
                    4d:b0:f9
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            X509v3 Extended Key Usage:
                TLS Web Client Authentication, TLS Web Server Authentication
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            Netscape Cert Type:
                SSL Client
            Netscape Comment:
                Node-0 Certificate
            X509v3 Subject Alternative Name:
                DNS:node-0, IP Address:127.0.0.1 # DNS:node-0
            X509v3 Subject Key Identifier:
                F0:75:8C:46:52:57:57:BF:C3:36:4E:CB:92:2B:4D:F1:32:D7:69:E0
            X509v3 Authority Key Identifier:
                9E:CD:36:BA:22:6B:43:A7:DB:23:29:97:53:1B:D1:72:DF:6D:61:D2
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        16:15:df:57:75:b0:6d:6e:8d:35:52:d6:43:07:61:cd:f3:00:
        60:d2:6a:1d:81:da:55:98:24:88:ce:c2:59:42:0b:b1:83:a7:
        de:71:41:df:02:b4:53:9e:3b:6c:bd:de:85:97:c1:d7:71:8a:
        33:aa:49:88:67:a6:74:4a:d8:80:68:cb:2e:a1:ae:52:ac:74:
        44:02:41:0c:79:15:6f:d7:48:56:9e:d8:41:0f:b3:f2:31:dc:
        b6:6b:f1:09:e7:af:df:75:6a:fd:be:d5:dd:69:20:8e:66:5f:
        88:f3:13:c3:49:e5:c8:ca:59:ce:83:1c:df:20:38:4f:e3:13:
        93:d4:3d:c7:fb:63:85:d6:5d:14:90:4c:c9:45:42:e3:54:e6:
        e6:b7:c1:ec:b9:a1:68:eb:d5:1b:28:cc:bc:cc:07:94:fa:a2:
        49:c0:14:19:55:b2:42:2b:04:c3:47:39:74:70:67:ac:8b:24:
        1e:0a:e4:86:3a:1d:56:18:f3:96:e2:f4:d6:20:34:be:dd:79:
        2f:b6:d6:4a:ed:c5:09:a3:42:9b:f3:c4:bd:99:ed:5a:05:27:
        ce:2b:b6:48:de:49:39:5d:23:c3:3a:f5:28:1d:37:16:6f:93:
        9b:15:4e:78:27:63:e1:de:1b:1e:36:66:d6:3f:e9:32:f0:41:
        4c:c2:cd:65:06:70:40:a6:0a:55:5f:47:77:3a:eb:03:35:6f:
        5a:06:75:c1:8f:81:f8:fd:2b:58:dd:d9:2d:d6:12:e1:f2:ae:
        14:3e:7b:af:96:25:7d:68:29:d1:4b:bb:56:03:26:d9:a4:4d:
        ae:bd:ac:93:ff:e3:7b:4f:17:d8:b8:d2:3f:a4:a4:da:73:4d:
        1d:a1:b0:1d:b7:68:56:f2:d5:b9:6b:52:b3:cc:09:f5:55:16:
        52:b7:a3:aa:36:54:13:72:7f:01:53:ca:79:e5:5e:84:87:40:
        7e:c4:d7:c3:8b:20:32:e4:67:3f:4d:cd:33:d0:92:03:31:49:
        c7:e2:2e:15:e7:16:a3:fa:dc:f8:cf:61:44:c8:ed:eb:ad:cc:
        50:1a:0e:72:19:30:cb:22:86:6a:bd:24:24:18:b5:27:54:b1:
        c4:96:cd:f3:d4:ba:9b:ec:7d:34:80:76:9e:24:10:b4:56:fa:
        9f:4a:81:60:f2:57:ad:d2:45:ce:65:aa:c8:95:24:6f:e4:6b:
        3f:9f:fc:d6:cb:34:54:09:2a:8c:17:6c:c8:4d:0f:ba:b4:d4:
        c5:ff:e5:14:2c:fb:50:34:c4:f6:87:12:44:ea:e7:b9:e3:84:
        d8:06:1b:b8:a7:a6:26:bd:c3:bf:7e:e0:0c:72:9e:38:42:8f:
        ec:ea:c3:43:ce:b3:26:69


root@jumpbox:~/kubernetes-the-hard-way# openssl x509 -in node-1.crt -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            2b:03:f7:ba:63:e9:26:c8:a5:c1:9f:dd:8b:d8:df:4c:aa:56:f5:71
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, ST = Washington, L = Seattle, CN = CA
        Validity
            Not Before: Jan 10 22:28:50 2026 GMT
            Not After : Jan 11 22:28:50 2036 GMT
        Subject: CN = system:node:node-1, O = system:nodes, C = US, ST = Washington, L = Seattle
        ...
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            X509v3 Extended Key Usage:
                TLS Web Client Authentication, TLS Web Server Authentication
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            Netscape Cert Type:
                SSL Client
            Netscape Comment:
                Node-1 Certificate
            X509v3 Subject Alternative Name:
                DNS:node-1, IP Address:127.0.0.1

root@jumpbox:~/kubernetes-the-hard-way# openssl x509 -in kube-proxy.crt -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            2b:03:f7:ba:63:e9:26:c8:a5:c1:9f:dd:8b:d8:df:4c:aa:56:f5:72
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, ST = Washington, L = Seattle, CN = CA
        Validity
            Not Before: Jan 10 22:28:52 2026 GMT
            Not After : Jan 11 22:28:52 2036 GMT
        Subject: CN = system:kube-proxy, O = system:node-proxier, C = US, ST = Washington, L = Seattle
        ...
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            X509v3 Extended Key Usage:
                TLS Web Client Authentication, TLS Web Server Authentication
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            Netscape Cert Type:
                SSL Client
            Netscape Comment:
                Kube Proxy Certificate
            X509v3 Subject Alternative Name:
                DNS:kube-proxy, IP Address:127.0.0.1
            X509v3 Subject Key Identifier:
                BE:7D:4E:DC:92:C0:AA:B9:6C:41:1E:D5:2F:A2:C6:26:79:F5:63:52
            X509v3 Authority Key Identifier:
                9E:CD:36:BA:22:6B:43:A7:DB:23:29:97:53:1B:D1:72:DF:6D:61:D2


# 아래 컴포넌트에 대해서도 모두 CN = system:컴포넌트, X509v3 Subject Alternative Name:DNS:컴포넌트 동일
root@jumpbox:~/kubernetes-the-hard-way# openssl x509 -in kube-scheduler.crt -text -noout
root@jumpbox:~/kubernetes-the-hard-way# openssl x509 -in kube-controller-manager.crt -text -noout
root@jumpbox:~/kubernetes-the-hard-way# openssl x509 -in kube-api-server.crt -text -noout
root@jumpbox:~/kubernetes-the-hard-way# openssl x509 -in service-accounts.crt -text -noout

각 VM에게 Client, Server 인증서를 나누어줍니다.

root@jumpbox:~/kubernetes-the-hard-way# for host in node-0 node-1; do
  ssh root@${host} mkdir /var/lib/kubelet/

  scp ca.crt root@${host}:/var/lib/kubelet/

  scp ${host}.crt \
    root@${host}:/var/lib/kubelet/kubelet.crt

  scp ${host}.key \
    root@${host}:/var/lib/kubelet/kubelet.key
done
ca.crt                                                                                                                                                       100% 1899     2.5MB/s   00:00
node-0.crt                                                                                                                                                   100% 2147     3.4MB/s   00:00
node-0.key                                                                                                                                                   100% 3272     4.2MB/s   00:00
ca.crt                                                                                                                                                       100% 1899     2.9MB/s   00:00
node-1.crt                                                                                                                                                   100% 2147     3.2MB/s   00:00
node-1.key                                                                                                                                                   100% 3272     5.1MB/s   00:00

root@jumpbox:~/kubernetes-the-hard-way# ssh node-0 ls -l /var/lib/kubelet
total 12
-rw-r--r-- 1 root root 1899 Jan 11 07:35 ca.crt
-rw-r--r-- 1 root root 2147 Jan 11 07:35 kubelet.crt
-rw------- 1 root root 3272 Jan 11 07:35 kubelet.key
root@jumpbox:~/kubernetes-the-hard-way# ssh node-1 ls -l /var/lib/kubelet
total 12
-rw-r--r-- 1 root root 1899 Jan 11 07:35 ca.crt
-rw-r--r-- 1 root root 2147 Jan 11 07:35 kubelet.crt
-rw------- 1 root root 3272 Jan 11 07:35 kubelet.key

root@jumpbox:~/kubernetes-the-hard-way# scp \
  ca.key ca.crt \
  kube-api-server.key kube-api-server.crt \
  service-accounts.key service-accounts.crt \
  root@server:~/
ca.key                                                                                                                                                       100% 3272     2.8MB/s   00:00
ca.crt                                                                                                                                                       100% 1899     2.0MB/s   00:00
kube-api-server.key                                                                                                                                          100% 3268     3.3MB/s   00:00
kube-api-server.crt                                                                                                                                          100% 2354     1.7MB/s   00:00
service-accounts.key                                                                                                                                         100% 3272     3.3MB/s   00:00
service-accounts.crt                                                                                                                                         100% 2004     1.9MB/s   00:00

root@jumpbox:~/kubernetes-the-hard-way# ssh server ls -l /root
total 24
-rw-r--r-- 1 root root 1899 Jan 11 07:36 ca.crt
-rw------- 1 root root 3272 Jan 11 07:36 ca.key
-rw-r--r-- 1 root root 2354 Jan 11 07:36 kube-api-server.crt
-rw------- 1 root root 3268 Jan 11 07:36 kube-api-server.key
-rw-r--r-- 1 root root 2004 Jan 11 07:36 service-accounts.crt
-rw------- 1 root root 3272 Jan 11 07:36 service-accounts.key

인증을 위한 Kubernetes 설정 파일 생성

API Server와 통신을 위한 Client 인증 설정 파일을 작성하겠습니다.

우선, kubelet 전용 kubeconfig 파일을 생성하겠습니다.

각 kubelet이 API Server에 ‘자기 자신’으로 인증·인가받을 수 있도록 kubeconfig(클라이언트 인증 설정 파일)를 만들어주는 단계입니다.

kubelet은 API Server에 클라이언트로 요청을 보낼때, 이 요청들은 익명 요청이 아니라 인증된 요청이어야합니다.

kubelet 인증서의 Subject CN이 반드시 system:node:<node-name> 형태여야 합니다.

따라서, node-0, node-1의 kubeconfig 파일을 생성하겠습니다.

root@jumpbox:~/kubernetes-the-hard-way# kubectl describe pod -n kube-system kube-apiserver-myk8s-control-plane
...
    Command:
      kube-apiserver
      --authorization-mode=Node,RBAC  
...


# config set-cluster
root@jumpbox:~/kubernetes-the-hard-way# kubectl config set-cluster kubernetes-the-hard-way \
  --certificate-authority=ca.crt \
  --embed-certs=true \
  --server=https://server.kubernetes.local:6443 \
  --kubeconfig=node-0.kubeconfig && ls -l node-0.kubeconfig && cat node-0.kubeconfig
Cluster "kubernetes-the-hard-way" set.
-rw------- 1 root root 2758 Jan 11 07:47 node-0.kubeconfig
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUZURENDQXpTZ0F3SUJBZ0lVY1NNVHR5S29XcXlkK1lYWUZqVVZhd05pT2prd0RRWUpLb1pJaHZjTkFRRU4KQlFBd1FURUxNQWtHQTFVRUJoTUNWVk14RXpBUkJnTlZCQWdNQ2xkaGMyaHBibWQwYjI0eEVEQU9CZ05WQkFjTQpCMU5sWVhSMGJHVXhDekFKQmdOVkJBTU1Ba05CTUI0WERUSTJNREV4TURJeU1EVTFObG9YRFRNMk1ERXhNVEl5Ck1EVTFObG93UVRFTE1Ba0dBMVVFQmhNQ1ZWTXhFekFSQmdOVkJBZ01DbGRoYzJocGJtZDBiMjR4RURBT0JnTlYKQkFjTUIxTmxZWFIwYkdVeEN6QUpCZ05WQkFNTUFrTkJNSUlDSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQWc4QQpNSUlDQ2dLQ0FnRUFudE4xUGh0Q3lCZ0dYUnBWeHViaUp6emhTOStVUUpiUEgxeWd4WTNrOVVlQmZnV1YvSmxRCjBmWk1ib294aXMyYzVBR1B1SnpZbW9UbkZtZjNxRWdVY09Rc0ZVYlNGSWtSU1FOQW9JMmRUdk1ESUVLNXgrK3gKazZxN1luaVhHWFFEN3R3QUdON2dvdGt4NjNGSDBPbEJNb0tBR05QcTBNZHFGMEVacktxdXlUQ1paU09yRHNjawo2bysycGxJNWVlTmg1YzBFaUFFTmxZdFc5UVd5enJDOFZLOWNYR0hDU1hqemhUc05KQUY2a0ZvNWFUTWZtRjdDCnhBTFRqSlFpc0o0VkpZdytHdVRkS2FnME00b0svTXQySkxlcElJeGl4dCtQTUpGNWdLeGpVdDVUMmE4emZ1NkwKQlQ5UEs3VGQydWd1TEdGTzh1dEJhT0U4T0NZYzZmMGdscVJ0ZHJHSUF6dXZBNk9LZlJYMGdQQ2N4bzY3UVRQeQpMdENaRUVlWnlvV3ZoRVNRVUxuYzNINVBzVzkwMndvdEU0L0ovRzIwNzhkNlhUZzRWNS9wUzhUNDQ5Q20yZXd1Cld5bzZNc1FZMllWbUdIdzBxTDhMZ2dDRDlNVHF2QTFoWVBzOGo0NjZxUldic01nNTFoRmxXazZ5b081RS9INXcKVUhuQlQrTHVWb3RpOXBRNC9oWjg3aDZNc0NsT3lDK2xFM0ZJMlkvU0FwamZXS0Y5dHVJaHhmY3J3dW5wcnUrTAorZFY5eDZ5em45SS9xQjhadmFLeTFuS0g2RU53RG5PN2llN0YrNTFnb2FacTBFV2UrSHg1ZzJFcHUwZlArdlBPCnUrSzB5bUxCaVhhVGJBS0NTTW9vNFpab3BRQjBBTVVrb3NzNkgxYzZSRjV3bDZabFlreXcrWkVDQXdFQUFhTTgKTURvd0RBWURWUjBUQkFVd0F3RUIvekFMQmdOVkhROEVCQU1DQVFZd0hRWURWUjBPQkJZRUZKN05Ocm9pYTBPbgoyeU1wbDFNYjBYTGZiV0hTTUEwR0NTcUdTSWIzRFFFQkRRVUFBNElDQVFCd3ByOCs4bmlHRTlVakFLUUcyQU5rCnh6SG5NUFFOeU44NXpFcXJCMFp4M2tpSEJ0RS9GaHpRdHhoL0lkZnozRTN0U0k4blRNbTRsSDYycnZQdmM1NEYKMTlLMWdXV3poeExrWkMrdEsyMEpVVVN5dVJkS0VlTzZTSHB0UVlwOEZkTWVhWVlMSS9jSHVBcmRTNkw5ek1pYQpCTDRZWEdwbTFxM3lYeVRrd3VnSFJTcFlxeUJ6UDhMa3NESjU3T2EyYSs3Q0xaYXU1ZnczSkpPRFJEc3V5OXRUClhGS2UzT2VWQS9INmcvQUMwUzlKa1JDTnVKODhxOG40S0hDRTRySzA2dUErVklUaHltd1p0dHlsTkhDQUVGNncKa2ptbjQ1SndIS3ZSY2YybTVhZDlVNWNWd0p5bkJidnZXNi9lMXFOVGgyZnRvcEUvUVVraHozUk8vNWlMQ3BIQgpNUjZWZW5qY05FYTg2TnoyUTVWOFRmLzhSWHpOWjZHU1BBQXhsUENTNDNkWkpMZVZUc2N2bTBURlZzRm5kQktxCmlOWXZjUGxyWHp0diswTVJxOXpkVWpQS1g2by85L2RBOXkxZG9GQlBRMWFxRWhnUGRWQnJqenJNbjdXaUVna2cKZWF4VlZaYnovbXNxYUF1akZnRUdaZ2V5OEFWUEI3WWNKSit5dXFSV2xMbmJ0enl2MUt6WGd1SytYYjZBazRFdAp5T2pscWhINlNKNm9JODA3MldwcXp5Z2hEcG9IOWt5VGpILy8xSEVLNVZOdXUxbUYxMGVrd05VQk9wQkREdXl5CjNDS20xUmhkUHlYRjhtSUxwOU9FODBrb3E1SzZ0RndOUy9oREVSdFFXdC9zMUN2dHFLa1NxZFlSVUlzKzBzcXQKWVRCZTJIdWJZVGptaUdOMzhOZnlOQT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    server: https://server.kubernetes.local:6443
  name: kubernetes-the-hard-way
contexts: null
current-context: ""
kind: Config
preferences: {}
users: null

root@jumpbox:~/kubernetes-the-hard-way# kubectl config set-cluster kubernetes-the-hard-way \
  --certificate-authority=ca.crt \
  --embed-certs=true \
  --server=https://server.kubernetes.local:6443 \
  --kubeconfig=node-1.kubeconfig && ls -l node-1.kubeconfig && cat node-1.kubeconfig
  
# config set-credentials
root@jumpbox:~/kubernetes-the-hard-way# kubectl config set-credentials system:node:node-0 \
  --client-certificate=node-0.crt \
  --client-key=node-0.key \
  --embed-certs=true \
  --kubeconfig=node-0.kubeconfig && cat node-0.kubeconfig
User "system:node:node-0" set.
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUZURENDQXpTZ0F3SUJBZ0lVY1NNVHR5S29XcXlkK1lYWUZqVVZhd05pT2prd0RRWUpLb1pJaHZjTkFRRU4KQlFBd1FURUxNQWtHQTFVRUJoTUNWVk14RXpBUkJnTlZCQWdNQ2xkaGMyaHBibWQwYjI0eEVEQU9CZ05WQkFjTQpCMU5sWVhSMGJHVXhDekFKQmdOVkJBTU1Ba05CTUI0WERUSTJNREV4TURJeU1EVTFObG9YRFRNMk1ERXhNVEl5Ck1EVTFObG93UVRFTE1Ba0dBMVVFQmhNQ1ZWTXhFekFSQmdOVkJBZ01DbGRoYzJocGJtZDBiMjR4RURBT0JnTlYKQkFjTUIxTmxZWFIwYkdVeEN6QUpCZ05WQkFNTUFrTkJNSUlDSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQWc4QQpNSUlDQ2dLQ0FnRUFudE4xUGh0Q3lCZ0dYUnBWeHViaUp6emhTOStVUUpiUEgxeWd4WTNrOVVlQmZnV1YvSmxRCjBmWk1ib294aXMyYzVBR1B1SnpZbW9UbkZtZjNxRWdVY09Rc0ZVYlNGSWtSU1FOQW9JMmRUdk1ESUVLNXgrK3gKazZxN1luaVhHWFFEN3R3QUdON2dvdGt4NjNGSDBPbEJNb0tBR05QcTBNZHFGMEVacktxdXlUQ1paU09yRHNjawo2bysycGxJNWVlTmg1YzBFaUFFTmxZdFc5UVd5enJDOFZLOWNYR0hDU1hqemhUc05KQUY2a0ZvNWFUTWZtRjdDCnhBTFRqSlFpc0o0VkpZdytHdVRkS2FnME00b0svTXQySkxlcElJeGl4dCtQTUpGNWdLeGpVdDVUMmE4emZ1NkwKQlQ5UEs3VGQydWd1TEdGTzh1dEJhT0U4T0NZYzZmMGdscVJ0ZHJHSUF6dXZBNk9LZlJYMGdQQ2N4bzY3UVRQeQpMdENaRUVlWnlvV3ZoRVNRVUxuYzNINVBzVzkwMndvdEU0L0ovRzIwNzhkNlhUZzRWNS9wUzhUNDQ5Q20yZXd1Cld5bzZNc1FZMllWbUdIdzBxTDhMZ2dDRDlNVHF2QTFoWVBzOGo0NjZxUldic01nNTFoRmxXazZ5b081RS9INXcKVUhuQlQrTHVWb3RpOXBRNC9oWjg3aDZNc0NsT3lDK2xFM0ZJMlkvU0FwamZXS0Y5dHVJaHhmY3J3dW5wcnUrTAorZFY5eDZ5em45SS9xQjhadmFLeTFuS0g2RU53RG5PN2llN0YrNTFnb2FacTBFV2UrSHg1ZzJFcHUwZlArdlBPCnUrSzB5bUxCaVhhVGJBS0NTTW9vNFpab3BRQjBBTVVrb3NzNkgxYzZSRjV3bDZabFlreXcrWkVDQXdFQUFhTTgKTURvd0RBWURWUjBUQkFVd0F3RUIvekFMQmdOVkhROEVCQU1DQVFZd0hRWURWUjBPQkJZRUZKN05Ocm9pYTBPbgoyeU1wbDFNYjBYTGZiV0hTTUEwR0NTcUdTSWIzRFFFQkRRVUFBNElDQVFCd3ByOCs4bmlHRTlVakFLUUcyQU5rCnh6SG5NUFFOeU44NXpFcXJCMFp4M2tpSEJ0RS9GaHpRdHhoL0lkZnozRTN0U0k4blRNbTRsSDYycnZQdmM1NEYKMTlLMWdXV3poeExrWkMrdEsyMEpVVVN5dVJkS0VlTzZTSHB0UVlwOEZkTWVhWVlMSS9jSHVBcmRTNkw5ek1pYQpCTDRZWEdwbTFxM3lYeVRrd3VnSFJTcFlxeUJ6UDhMa3NESjU3T2EyYSs3Q0xaYXU1ZnczSkpPRFJEc3V5OXRUClhGS2UzT2VWQS9INmcvQUMwUzlKa1JDTnVKODhxOG40S0hDRTRySzA2dUErVklUaHltd1p0dHlsTkhDQUVGNncKa2ptbjQ1SndIS3ZSY2YybTVhZDlVNWNWd0p5bkJidnZXNi9lMXFOVGgyZnRvcEUvUVVraHozUk8vNWlMQ3BIQgpNUjZWZW5qY05FYTg2TnoyUTVWOFRmLzhSWHpOWjZHU1BBQXhsUENTNDNkWkpMZVZUc2N2bTBURlZzRm5kQktxCmlOWXZjUGxyWHp0diswTVJxOXpkVWpQS1g2by85L2RBOXkxZG9GQlBRMWFxRWhnUGRWQnJqenJNbjdXaUVna2cKZWF4VlZaYnovbXNxYUF1akZnRUdaZ2V5OEFWUEI3WWNKSit5dXFSV2xMbmJ0enl2MUt6WGd1SytYYjZBazRFdAp5T2pscWhINlNKNm9JODA3MldwcXp5Z2hEcG9IOWt5VGpILy8xSEVLNVZOdXUxbUYxMGVrd05VQk9wQkREdXl5CjNDS20xUmhkUHlYRjhtSUxwOU9FODBrb3E1SzZ0RndOUy9oREVSdFFXdC9zMUN2dHFLa1NxZFlSVUlzKzBzcXQKWVRCZTJIdWJZVGptaUdOMzhOZnlOQT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    server: https://server.kubernetes.local:6443
  name: kubernetes-the-hard-way
contexts: null
current-context: ""
kind: Config
preferences: {}
users:
- name: system:node:node-0
  user:
    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUdCRENDQSt5Z0F3SUJBZ0lVS3dQM3VtUHBKc2lsd1ovZGk5amZUS3BXOVhBd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1FURUxNQWtHQTFVRUJoTUNWVk14RXpBUkJnTlZCQWdNQ2xkaGMyaHBibWQwYjI0eEVEQU9CZ05WQkFjTQpCMU5sWVhSMGJHVXhDekFKQmdOVkJBTU1Ba05CTUI0WERUSTJNREV4TURJeU1qZzBPRm9YRFRNMk1ERXhNVEl5Ck1qZzBPRm93YURFYk1Ca0dBMVVFQXd3U2MzbHpkR1Z0T201dlpHVTZibTlrWlMwd01SVXdFd1lEVlFRS0RBeHoKZVhOMFpXMDZibTlrWlhNeEN6QUpCZ05WQkFZVEFsVlRNUk13RVFZRFZRUUlEQXBYWVhOb2FXNW5kRzl1TVJBdwpEZ1lEVlFRSERBZFRaV0YwZEd4bE1JSUNJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBZzhBTUlJQ0NnS0NBZ0VBCjAvN3l3Tis3Q0hadDd4dkZKZUFXWnI3Uk5yQXVhNXJGMTNuNU50b3J5NVdSbEJ0MWt2V0hKZ3Vvak9XZ0IrYkQKQVhGaHdmRkFRMWNkWTd0Unh2MkJxK0NEUUxBcGVHOGlNR01YSVc1Qk1UYzV1YTlmakF5dGg5aUJsY2JVdm03NQoyaHNjeUtNUGNqeDNLbm9iSzNGQWhhUGsrMTdWZ1JTZ0Zabk9hc1FmVURwUHFSQVh1OXpwNHJvRFNPa1o3SXVVCkhkYVRHMWRHN2lVNTA1czdTaHVNY2Y4aC8yQjRUN0lLRU0vYWM2bEwzUEZRTGNXQ0MwUTB5MENIM0FjN2lCL08KdzRYRkVyZnpFMDEzNEgxZ3FWdXNnVHcvZVdPd2lDZXBrdUxDdllZTzZQTHR1Nzh2N1JUd1VNKzdvNDJPanVXKwpSWEt0ZzNnZTJrMU9uOWxQYlZzckpZMkFaMXU5cFJPNDB4bzFZWnlMeHVlMVJXN1NYb0sxR2U1cWtoUzM3d1JtCmlyT1VDa1MvMGtERFluMFY4MUJMbjNaQmJJY1ZYS3gwSnN6R1VTbkE0VFZTMFl4RndaTVg0a1dJalo4alRobWQKVHQ5QVdUcnc1eEpYL3RweStLY2JUcjVGZFY4eWxHa1JBaytwYWxPNUkyS0lRcllXN1Bsa01lcU5iZkh4MGRycwpuN21wMnFZbk9laE81QTlpRWVKRURLcDhSQm1GanlrMXVHTnQzNFdkR1liRFNXK3RWL1lEUDQvSFZqc2tvWnZHCmhXMXdtSHRHS2dhVHZJZ1BNOFE5M2d4NTRzTVBhRmNzQjd5MDVBVDErQXZPWHJlQTlieEUzNnJyT0RURW8rRTAKc3lRaFF2UVF5Qk9vTW1ZUHhvUzdhLzlZK3JrOUZzV2M4TDR0bm50TnNQa0NBd0VBQWFPQnpEQ0J5VEFKQmdOVgpIUk1FQWpBQU1CMEdBMVVkSlFRV01CUUdDQ3NHQVFVRkJ3TUNCZ2dyQmdFRkJRY0RBVEFPQmdOVkhROEJBZjhFCkJBTUNCYUF3RVFZSllJWklBWWI0UWdFQkJBUURBZ2VBTUNFR0NXQ0dTQUdHK0VJQkRRUVVGaEpPYjJSbExUQWcKUTJWeWRHbG1hV05oZEdVd0Z3WURWUjBSQkJBd0RvSUdibTlrWlMwd2h3Ui9BQUFCTUIwR0ExVWREZ1FXQkJUdwpkWXhHVWxkWHY4TTJUc3VTSzAzeE10ZHA0REFmQmdOVkhTTUVHREFXZ0JTZXpUYTZJbXREcDlzaktaZFRHOUZ5CjMyMWgwakFOQmdrcWhraUc5dzBCQVFzRkFBT0NBZ0VBRmhYZlYzV3diVzZOTlZMV1F3ZGh6Zk1BWU5KcUhZSGEKVlpna2lNN0NXVUlMc1lPbjNuRkIzd0swVTU0N2JMM2VoWmZCMTNHS002cEppR2VtZEVyWWdHakxMcUd1VXF4MApSQUpCREhrVmI5ZElWcDdZUVErejhqSGN0bXZ4Q2VldjMzVnEvYjdWM1drZ2ptWmZpUE1UdzBubHlNcFp6b01jCjN5QTRUK01UazlROXgvdGpoZFpkRkpCTXlVVkM0MVRtNXJmQjdMbWhhT3ZWR3lqTXZNd0hsUHFpU2NBVUdWV3kKUWlzRXcwYzVkSEJucklza0hncmtoam9kVmhqemx1TDAxaUEwdnQxNUw3YldTdTNGQ2FOQ20vUEV2Wm50V2dVbgp6aXUyU041Sk9WMGp3enIxS0IwM0ZtK1RteFZPZUNkajRkNGJIalptMWovcE12QkJUTUxOWlFad1FLWUtWVjlICmR6cnJBelZ2V2daMXdZK0IrUDByV04zWkxkWVM0Zkt1RkQ1N3I1WWxmV2dwMFV1N1ZnTW0yYVJOcnIyc2svL2oKZTA4WDJMalNQNlNrMm5OTkhhR3dIYmRvVnZMVnVXdFNzOHdKOVZVV1VyZWpxalpVRTNKL0FWUEtlZVZlaElkQQpmc1RYdzRzZ011Um5QMDNOTTlDU0F6Rkp4K0l1RmVjV28vcmMrTTloUk1qdDY2M01VQm9PY2hrd3l5S0dhcjBrCkpCaTFKMVN4eEpiTjg5UzZtK3g5TklCMm5pUVF0RmI2bjBxQllQSlhyZEpGem1XcXlKVWtiK1JyUDUvODFzczAKVkFrcWpCZHN5RTBQdXJUVXhmL2xGQ3o3VURURTlvY1NST3JudWVPRTJBWWJ1S2VtSnIzRHYzN2dESEtlT0VLUAo3T3JEUTg2ekptaz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    client-key-data: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUpRZ0lCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQ1N3d2dna29BZ0VBQW9JQ0FRRFQvdkxBMzdzSWRtM3YKRzhVbDRCWm12dEUyc0M1cm1zWFhlZmsyMml2TGxaR1VHM1dTOVljbUM2aU01YUFINXNNQmNXSEI4VUJEVngxagp1MUhHL1lHcjRJTkFzQ2w0YnlJd1l4Y2hia0V4TnptNXIxK01ESzJIMklHVnh0Uytidm5hR3h6SW93OXlQSGNxCmVoc3JjVUNGbytUN1h0V0JGS0FWbWM1cXhCOVFPaytwRUJlNzNPbml1Z05JNlJuc2k1UWQxcE1iVjBidUpUblQKbXp0S0c0eHgveUgvWUhoUHNnb1F6OXB6cVV2YzhWQXR4WUlMUkRUTFFJZmNCenVJSDg3RGhjVVN0L01UVFhmZwpmV0NwVzZ5QlBEOTVZN0NJSjZtUzRzSzloZzdvOHUyN3Z5L3RGUEJRejd1ampZNk81YjVGY3EyRGVCN2FUVTZmCjJVOXRXeXNsallCblc3MmxFN2pUR2pWaG5Jdkc1N1ZGYnRKZWdyVVo3bXFTRkxmdkJHYUtzNVFLUkwvU1FNTmkKZlJYelVFdWZka0ZzaHhWY3JIUW16TVpSS2NEaE5WTFJqRVhCa3hmaVJZaU5ueU5PR1oxTzMwQlpPdkRuRWxmKwoybkw0cHh0T3ZrVjFYektVYVJFQ1Q2bHFVN2tqWW9oQ3RoYnMrV1F4Nm8xdDhmSFIydXlmdWFuYXBpYzU2RTdrCkQySVI0a1FNcW54RUdZV1BLVFc0WTIzZmhaMFpoc05KYjYxWDlnTS9qOGRXT3lTaG04YUZiWENZZTBZcUJwTzgKaUE4enhEM2VESG5pd3c5b1Z5d0h2TFRrQlBYNEM4NWV0NEQxdkVUZnF1czROTVNqNFRTekpDRkM5QkRJRTZneQpaZy9HaEx0ci8xajZ1VDBXeFp6d3ZpMmVlMDJ3K1FJREFRQUJBb0lDQURBTlh1cjNDNGI4aVNpa3E4aE5CZWJlCkk3LzRxVno4cFJHVmRUYkl0eHFiV1hjdnpqTzBjYURmRGFKNlBnN3NjSkU0S1FkejdyUVUzL1dHTlNBY3dESzgKWGlQZ2lsWGhJdWZDM2pCV3RISTZhVnc4Y3pUa2ZzVUcvMEI2NUNnU01aSU5pVFcxT0VBVmtwRlBRaTlUcVdOaApaZnJRWVNhMERTRTRUc0NHbWU1dWVUWUNEczhuWU9CaWQzSWtYQnJ3L0c3Uy8xODRKankrdFYvUWxOelVaNzVECkxXcWNiRWpzT3puYVUxTGFNZmJ2djlQUUxBenZ0ZVBJU2NORzVHaVRoU051ZytBZFQveTFGNUlMK1VnS0hvemcKb2lhYXdSM3JsNVZTenY5MGpwYWcwZUEza3Q1elc1R0Z0S3Q2MzdCbVJQMnE0OGVrQjZKRjJhZUQ4MnYvT1VVTQoyR1JadDYrdXhHeWkxblRWSTA2YjZQVDFPQU5zSGZ4OUtvcTBybEdUQnY5TnFYcUtrczBldWRBUmFPUTB3UmFUCityaXA2MFhpalNFRkg2RzFlQXE3UnkvdUtTc0UzbnEwdGs3d2lWbFYwemlleVk2cjFUZnFXZjgvKzBxYndyZDIKalpVemp6UHUwSmJCZGRTeHJVUUpwcTcvSlliMjNKb0dnUVJPaTFQMUhKbjBqUVZXUkdPRUMzK2JKTitvamYzTQorSnhKUndidFNaYUphbnpMckpKbTF2MUFQMENqeENiM2N1b2lnc2JwL2RzOWF4YnFLaTduaVpvZjUwZUJSei9qCmQ3ZkJ1Sm9yQ09sbEpmM2p5ZHpXWTVqY1BSNjYyemczc2IyWUlnc2dacDJTaytCVTM0bXVCc0hsUUx6elFxRXgKc1RlQmNsd3BrQlNaODVoSzRkWjlBb0lCQVFEOUJRbGJjMUNPOFZLaFI0KzdtalJPTTMyMVNxdVlneEkveWFBWgpNUThIWEFaV1Uya084QnZ2bjZDdHI5SHlXQ005U1ZlQWEyNzdzQ1dSTkkxRGVoR0ZPekpJTUFOcDEzaGgva3lICkN3NzY4NTY0MHBtVlh3d1lSejBIN2FuUXJudHpaQ2YwV0lwWE5pd01CNnZvNkh3a1Z5M25LdDRTTWVRY2Y2bVoKYU1saHBrdlM1c3NkU25zUmRPUWZiZGhmdHF4ZXp6dlQ4WmcxNnFTeHM3b3IraytUVjlBbmZqL21HMnV3RHdLWApQb2Q5R3JUdTlJaXBuOFBvZ1Qza01xTzNCUUdJVml4V200cHc3T3dLOUhsYzNpcGR6eUV1YVcyREM0OGRyNm4zCk9lMDZ0ODJGRGZRZHZra0hUN3g2eU9aalBiRXBMNjNSYkpjVktRVFBpb1lFK1JmakFvSUJBUURXZmpVUi9QOWwKVTZjb0VhZEkrbmdvK3pVOU1WQ1VZazZnK3J1U3cvWml2ZTFBY1RUdXlZZmRQMGc1eDlDbkRLeXVhSlpEc0JXSgpiQUNCS2N6RjNYcHBoZ1BJRXA3WkFVYy9KdHZlWUNWYVlqczQ2cnZIbitNNHd2U3gvZ1JKSDhFZnZwVG1Bc1V3CktMMGFVRWVSWnFnTjZMREJRNnVRWFppR3hyd1pRdFZhTVlzVVpXSDAwTzFja0d0Mk5jUWdlOXh0YVZwV3BwY2MKWGxSYWRSK0JXaXFvSTJLdllwK1ZpU3VEQ3FnN0gwWlBVeGY1RHgvcEVYd3l0WDRkUjdaSVdvaEVoV2R2b25NLwp3VEhTSUZHc0ZBQlhVdEUrbkRHNjhJQzh3ejdNR2dEbW1adW5KbXZUN2RnMEQ0ZmtwZnQ2NHlQRHFiZ2VrUnE0CmtNUnFiL28rZmhKekFvSUJBQ1daZVFHQkNtaWpqZjd2am5sM2VleGhGSDQ3WVhLaGo3V3c5Ynlvb0s0Y1M4aksKVG1OczVRbmY4OVc3dWZ4RWdzQVp4UW9OYlFtejdIRSttUHQ5Q1Z0K0o0TzR0dGtrcnZ3SStVQmNNVmF6bTFPbAo0RExsVS92TG9sTXFTMEQ2bk5mUisyb0JPbFdwZERRNWM1K0I1T2pWbDdteGtoeXNFMVQwMTdRM2NTMjBzN0hjCm9EUnpIQVRGRlIxaWluUDJ4bmkwWFJTcTQwNm4xdCtkY0w0V1RYcjg0cTdyd1AwbCtBUEt4enpJaXdQWlV5ZnIKYU4ra0lVM00zdkNwWWREUnlwK0RCc1c4Q1grc2hZcy9OMFhJc2cwTXVjYVczakpLSm5ObjVsOFoyWm9QTkZoZwpjWFYyRUk0b21ENjYyaG5IVWl0bnJYSGdyRGIxeUlRNjlwMklrcmNDZ2dFQUdRT3c5cjhWbmVDS252Nm9EcmUvCktiQmFQaCt4TVF6aDlKNHBIKzJnbjFWZEJPWHU4V1lwWlNFTE1FbmdTSXNPM0d3d0VnYklERFcvUFZEOW81bTMKdS9TNEhpNGx2bE1LMXcrV0ZySUp5U2VwbG5LeG9ZOUE5WTZ1OE04Wit0a3NNYUtOUk9IUDAxVDg4TU9xQlFmKwowMWh1dkt0Q3A5eHlmRTFUQkhNcDVKSGdrV2xkYWlxdmM5c1gzVHBaL0J2VjBpQzJPMkxFUHJlMWlMTWVFSHpPCmVydGR0K3g5aGZmd1RpRUJDNXdUSEE4cHZZOEkvcXBlWEorVDdLQS9YdUxILzQyN1NXYWRvRExNdDUrelJxa3kKZlB5WWtIK3d3UWRwVEJwTkNZcUN6bUhkcURtdjI1QmlaWGZtTmtFeE9keWhnQ0ViNnR5clRCajkveGUrREdteApod0tDQVFFQWlxa2lHTUtBZms2cG5lelNiZUdjVis3bjNyQlB1TXJoTHVyRkNtMllpL1BSOTE1YXlLdWtYdnpVCnNleld5eFY1cUJIblZYdnNpblRYT212THBnVEovdWdzREdRQ3h2bVNaZXNqWkpNcE1VUnUvZmswS2pVK0RqZVgKeFVGZ3p5NVpiVjZlSXlwWmE4OUZqK2RYVnJqWTZsNXVnWjdhZHNJUnN5WkFMQnQwRjNaZ200REFBRWo5RWtMdgpZZjJlbUdwTGpOVE9FOTdlOXNMNC9LWEZ3UmdZbm5kc1hMdVNPQ29DWnBDTG56VENWMTdYeTR5MHlTZ3ZaTUxUCjIxaTI3WVAvNlc0WDUvNmFGVVhIOUFhaHJOMzB2MnpMVmRLeXJHQUk4amtJeHJDWW05UjBLOWR1RGdtYXNRRWYKaC9ka2dDM0x3WHB2djUyTG1rbUtZL09nTUh3OW13PT0KLS0tLS1FTkQgUFJJVkFURSBLRVktLS0tLQo=
    
    
root@jumpbox:~/kubernetes-the-hard-way# kubectl config set-credentials system:node:node-1 \
  --client-certificate=node-1.crt \
  --client-key=node-1.key \
  --embed-certs=true \
  --kubeconfig=node-1.kubeconfig && cat node-1.kubeconfig
  
  
# set-context : default 추가
root@jumpbox:~/kubernetes-the-hard-way# kubectl config set-context default \
  --cluster=kubernetes-the-hard-way \
  --user=system:node:node-0 \
  --kubeconfig=node-0.kubeconfig && cat node-0.kubeconfig
Context "default" created.
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUZURENDQXpTZ0F3SUJBZ0lVY1NNVHR5S29XcXlkK1lYWUZqVVZhd05pT2prd0RRWUpLb1pJaHZjTkFRRU4KQlFBd1FURUxNQWtHQTFVRUJoTUNWVk14RXpBUkJnTlZCQWdNQ2xkaGMyaHBibWQwYjI0eEVEQU9CZ05WQkFjTQpCMU5sWVhSMGJHVXhDekFKQmdOVkJBTU1Ba05CTUI0WERUSTJNREV4TURJeU1EVTFObG9YRFRNMk1ERXhNVEl5Ck1EVTFObG93UVRFTE1Ba0dBMVVFQmhNQ1ZWTXhFekFSQmdOVkJBZ01DbGRoYzJocGJtZDBiMjR4RURBT0JnTlYKQkFjTUIxTmxZWFIwYkdVeEN6QUpCZ05WQkFNTUFrTkJNSUlDSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQWc4QQpNSUlDQ2dLQ0FnRUFudE4xUGh0Q3lCZ0dYUnBWeHViaUp6emhTOStVUUpiUEgxeWd4WTNrOVVlQmZnV1YvSmxRCjBmWk1ib294aXMyYzVBR1B1SnpZbW9UbkZtZjNxRWdVY09Rc0ZVYlNGSWtSU1FOQW9JMmRUdk1ESUVLNXgrK3gKazZxN1luaVhHWFFEN3R3QUdON2dvdGt4NjNGSDBPbEJNb0tBR05QcTBNZHFGMEVacktxdXlUQ1paU09yRHNjawo2bysycGxJNWVlTmg1YzBFaUFFTmxZdFc5UVd5enJDOFZLOWNYR0hDU1hqemhUc05KQUY2a0ZvNWFUTWZtRjdDCnhBTFRqSlFpc0o0VkpZdytHdVRkS2FnME00b0svTXQySkxlcElJeGl4dCtQTUpGNWdLeGpVdDVUMmE4emZ1NkwKQlQ5UEs3VGQydWd1TEdGTzh1dEJhT0U4T0NZYzZmMGdscVJ0ZHJHSUF6dXZBNk9LZlJYMGdQQ2N4bzY3UVRQeQpMdENaRUVlWnlvV3ZoRVNRVUxuYzNINVBzVzkwMndvdEU0L0ovRzIwNzhkNlhUZzRWNS9wUzhUNDQ5Q20yZXd1Cld5bzZNc1FZMllWbUdIdzBxTDhMZ2dDRDlNVHF2QTFoWVBzOGo0NjZxUldic01nNTFoRmxXazZ5b081RS9INXcKVUhuQlQrTHVWb3RpOXBRNC9oWjg3aDZNc0NsT3lDK2xFM0ZJMlkvU0FwamZXS0Y5dHVJaHhmY3J3dW5wcnUrTAorZFY5eDZ5em45SS9xQjhadmFLeTFuS0g2RU53RG5PN2llN0YrNTFnb2FacTBFV2UrSHg1ZzJFcHUwZlArdlBPCnUrSzB5bUxCaVhhVGJBS0NTTW9vNFpab3BRQjBBTVVrb3NzNkgxYzZSRjV3bDZabFlreXcrWkVDQXdFQUFhTTgKTURvd0RBWURWUjBUQkFVd0F3RUIvekFMQmdOVkhROEVCQU1DQVFZd0hRWURWUjBPQkJZRUZKN05Ocm9pYTBPbgoyeU1wbDFNYjBYTGZiV0hTTUEwR0NTcUdTSWIzRFFFQkRRVUFBNElDQVFCd3ByOCs4bmlHRTlVakFLUUcyQU5rCnh6SG5NUFFOeU44NXpFcXJCMFp4M2tpSEJ0RS9GaHpRdHhoL0lkZnozRTN0U0k4blRNbTRsSDYycnZQdmM1NEYKMTlLMWdXV3poeExrWkMrdEsyMEpVVVN5dVJkS0VlTzZTSHB0UVlwOEZkTWVhWVlMSS9jSHVBcmRTNkw5ek1pYQpCTDRZWEdwbTFxM3lYeVRrd3VnSFJTcFlxeUJ6UDhMa3NESjU3T2EyYSs3Q0xaYXU1ZnczSkpPRFJEc3V5OXRUClhGS2UzT2VWQS9INmcvQUMwUzlKa1JDTnVKODhxOG40S0hDRTRySzA2dUErVklUaHltd1p0dHlsTkhDQUVGNncKa2ptbjQ1SndIS3ZSY2YybTVhZDlVNWNWd0p5bkJidnZXNi9lMXFOVGgyZnRvcEUvUVVraHozUk8vNWlMQ3BIQgpNUjZWZW5qY05FYTg2TnoyUTVWOFRmLzhSWHpOWjZHU1BBQXhsUENTNDNkWkpMZVZUc2N2bTBURlZzRm5kQktxCmlOWXZjUGxyWHp0diswTVJxOXpkVWpQS1g2by85L2RBOXkxZG9GQlBRMWFxRWhnUGRWQnJqenJNbjdXaUVna2cKZWF4VlZaYnovbXNxYUF1akZnRUdaZ2V5OEFWUEI3WWNKSit5dXFSV2xMbmJ0enl2MUt6WGd1SytYYjZBazRFdAp5T2pscWhINlNKNm9JODA3MldwcXp5Z2hEcG9IOWt5VGpILy8xSEVLNVZOdXUxbUYxMGVrd05VQk9wQkREdXl5CjNDS20xUmhkUHlYRjhtSUxwOU9FODBrb3E1SzZ0RndOUy9oREVSdFFXdC9zMUN2dHFLa1NxZFlSVUlzKzBzcXQKWVRCZTJIdWJZVGptaUdOMzhOZnlOQT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    server: https://server.kubernetes.local:6443
  name: kubernetes-the-hard-way
contexts:
- context:
    cluster: kubernetes-the-hard-way
    user: system:node:node-0
  name: default
current-context: ""
kind: Config
preferences: {}
users:
- name: system:node:node-0
  user:
    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUdCRENDQSt5Z0F3SUJBZ0lVS3dQM3VtUHBKc2lsd1ovZGk5amZUS3BXOVhBd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1FURUxNQWtHQTFVRUJoTUNWVk14RXpBUkJnTlZCQWdNQ2xkaGMyaHBibWQwYjI0eEVEQU9CZ05WQkFjTQpCMU5sWVhSMGJHVXhDekFKQmdOVkJBTU1Ba05CTUI0WERUSTJNREV4TURJeU1qZzBPRm9YRFRNMk1ERXhNVEl5Ck1qZzBPRm93YURFYk1Ca0dBMVVFQXd3U2MzbHpkR1Z0T201dlpHVTZibTlrWlMwd01SVXdFd1lEVlFRS0RBeHoKZVhOMFpXMDZibTlrWlhNeEN6QUpCZ05WQkFZVEFsVlRNUk13RVFZRFZRUUlEQXBYWVhOb2FXNW5kRzl1TVJBdwpEZ1lEVlFRSERBZFRaV0YwZEd4bE1JSUNJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBZzhBTUlJQ0NnS0NBZ0VBCjAvN3l3Tis3Q0hadDd4dkZKZUFXWnI3Uk5yQXVhNXJGMTNuNU50b3J5NVdSbEJ0MWt2V0hKZ3Vvak9XZ0IrYkQKQVhGaHdmRkFRMWNkWTd0Unh2MkJxK0NEUUxBcGVHOGlNR01YSVc1Qk1UYzV1YTlmakF5dGg5aUJsY2JVdm03NQoyaHNjeUtNUGNqeDNLbm9iSzNGQWhhUGsrMTdWZ1JTZ0Zabk9hc1FmVURwUHFSQVh1OXpwNHJvRFNPa1o3SXVVCkhkYVRHMWRHN2lVNTA1czdTaHVNY2Y4aC8yQjRUN0lLRU0vYWM2bEwzUEZRTGNXQ0MwUTB5MENIM0FjN2lCL08KdzRYRkVyZnpFMDEzNEgxZ3FWdXNnVHcvZVdPd2lDZXBrdUxDdllZTzZQTHR1Nzh2N1JUd1VNKzdvNDJPanVXKwpSWEt0ZzNnZTJrMU9uOWxQYlZzckpZMkFaMXU5cFJPNDB4bzFZWnlMeHVlMVJXN1NYb0sxR2U1cWtoUzM3d1JtCmlyT1VDa1MvMGtERFluMFY4MUJMbjNaQmJJY1ZYS3gwSnN6R1VTbkE0VFZTMFl4RndaTVg0a1dJalo4alRobWQKVHQ5QVdUcnc1eEpYL3RweStLY2JUcjVGZFY4eWxHa1JBaytwYWxPNUkyS0lRcllXN1Bsa01lcU5iZkh4MGRycwpuN21wMnFZbk9laE81QTlpRWVKRURLcDhSQm1GanlrMXVHTnQzNFdkR1liRFNXK3RWL1lEUDQvSFZqc2tvWnZHCmhXMXdtSHRHS2dhVHZJZ1BNOFE5M2d4NTRzTVBhRmNzQjd5MDVBVDErQXZPWHJlQTlieEUzNnJyT0RURW8rRTAKc3lRaFF2UVF5Qk9vTW1ZUHhvUzdhLzlZK3JrOUZzV2M4TDR0bm50TnNQa0NBd0VBQWFPQnpEQ0J5VEFKQmdOVgpIUk1FQWpBQU1CMEdBMVVkSlFRV01CUUdDQ3NHQVFVRkJ3TUNCZ2dyQmdFRkJRY0RBVEFPQmdOVkhROEJBZjhFCkJBTUNCYUF3RVFZSllJWklBWWI0UWdFQkJBUURBZ2VBTUNFR0NXQ0dTQUdHK0VJQkRRUVVGaEpPYjJSbExUQWcKUTJWeWRHbG1hV05oZEdVd0Z3WURWUjBSQkJBd0RvSUdibTlrWlMwd2h3Ui9BQUFCTUIwR0ExVWREZ1FXQkJUdwpkWXhHVWxkWHY4TTJUc3VTSzAzeE10ZHA0REFmQmdOVkhTTUVHREFXZ0JTZXpUYTZJbXREcDlzaktaZFRHOUZ5CjMyMWgwakFOQmdrcWhraUc5dzBCQVFzRkFBT0NBZ0VBRmhYZlYzV3diVzZOTlZMV1F3ZGh6Zk1BWU5KcUhZSGEKVlpna2lNN0NXVUlMc1lPbjNuRkIzd0swVTU0N2JMM2VoWmZCMTNHS002cEppR2VtZEVyWWdHakxMcUd1VXF4MApSQUpCREhrVmI5ZElWcDdZUVErejhqSGN0bXZ4Q2VldjMzVnEvYjdWM1drZ2ptWmZpUE1UdzBubHlNcFp6b01jCjN5QTRUK01UazlROXgvdGpoZFpkRkpCTXlVVkM0MVRtNXJmQjdMbWhhT3ZWR3lqTXZNd0hsUHFpU2NBVUdWV3kKUWlzRXcwYzVkSEJucklza0hncmtoam9kVmhqemx1TDAxaUEwdnQxNUw3YldTdTNGQ2FOQ20vUEV2Wm50V2dVbgp6aXUyU041Sk9WMGp3enIxS0IwM0ZtK1RteFZPZUNkajRkNGJIalptMWovcE12QkJUTUxOWlFad1FLWUtWVjlICmR6cnJBelZ2V2daMXdZK0IrUDByV04zWkxkWVM0Zkt1RkQ1N3I1WWxmV2dwMFV1N1ZnTW0yYVJOcnIyc2svL2oKZTA4WDJMalNQNlNrMm5OTkhhR3dIYmRvVnZMVnVXdFNzOHdKOVZVV1VyZWpxalpVRTNKL0FWUEtlZVZlaElkQQpmc1RYdzRzZ011Um5QMDNOTTlDU0F6Rkp4K0l1RmVjV28vcmMrTTloUk1qdDY2M01VQm9PY2hrd3l5S0dhcjBrCkpCaTFKMVN4eEpiTjg5UzZtK3g5TklCMm5pUVF0RmI2bjBxQllQSlhyZEpGem1XcXlKVWtiK1JyUDUvODFzczAKVkFrcWpCZHN5RTBQdXJUVXhmL2xGQ3o3VURURTlvY1NST3JudWVPRTJBWWJ1S2VtSnIzRHYzN2dESEtlT0VLUAo3T3JEUTg2ekptaz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    client-key-data: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUpRZ0lCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQ1N3d2dna29BZ0VBQW9JQ0FRRFQvdkxBMzdzSWRtM3YKRzhVbDRCWm12dEUyc0M1cm1zWFhlZmsyMml2TGxaR1VHM1dTOVljbUM2aU01YUFINXNNQmNXSEI4VUJEVngxagp1MUhHL1lHcjRJTkFzQ2w0YnlJd1l4Y2hia0V4TnptNXIxK01ESzJIMklHVnh0Uytidm5hR3h6SW93OXlQSGNxCmVoc3JjVUNGbytUN1h0V0JGS0FWbWM1cXhCOVFPaytwRUJlNzNPbml1Z05JNlJuc2k1UWQxcE1iVjBidUpUblQKbXp0S0c0eHgveUgvWUhoUHNnb1F6OXB6cVV2YzhWQXR4WUlMUkRUTFFJZmNCenVJSDg3RGhjVVN0L01UVFhmZwpmV0NwVzZ5QlBEOTVZN0NJSjZtUzRzSzloZzdvOHUyN3Z5L3RGUEJRejd1ampZNk81YjVGY3EyRGVCN2FUVTZmCjJVOXRXeXNsallCblc3MmxFN2pUR2pWaG5Jdkc1N1ZGYnRKZWdyVVo3bXFTRkxmdkJHYUtzNVFLUkwvU1FNTmkKZlJYelVFdWZka0ZzaHhWY3JIUW16TVpSS2NEaE5WTFJqRVhCa3hmaVJZaU5ueU5PR1oxTzMwQlpPdkRuRWxmKwoybkw0cHh0T3ZrVjFYektVYVJFQ1Q2bHFVN2tqWW9oQ3RoYnMrV1F4Nm8xdDhmSFIydXlmdWFuYXBpYzU2RTdrCkQySVI0a1FNcW54RUdZV1BLVFc0WTIzZmhaMFpoc05KYjYxWDlnTS9qOGRXT3lTaG04YUZiWENZZTBZcUJwTzgKaUE4enhEM2VESG5pd3c5b1Z5d0h2TFRrQlBYNEM4NWV0NEQxdkVUZnF1czROTVNqNFRTekpDRkM5QkRJRTZneQpaZy9HaEx0ci8xajZ1VDBXeFp6d3ZpMmVlMDJ3K1FJREFRQUJBb0lDQURBTlh1cjNDNGI4aVNpa3E4aE5CZWJlCkk3LzRxVno4cFJHVmRUYkl0eHFiV1hjdnpqTzBjYURmRGFKNlBnN3NjSkU0S1FkejdyUVUzL1dHTlNBY3dESzgKWGlQZ2lsWGhJdWZDM2pCV3RISTZhVnc4Y3pUa2ZzVUcvMEI2NUNnU01aSU5pVFcxT0VBVmtwRlBRaTlUcVdOaApaZnJRWVNhMERTRTRUc0NHbWU1dWVUWUNEczhuWU9CaWQzSWtYQnJ3L0c3Uy8xODRKankrdFYvUWxOelVaNzVECkxXcWNiRWpzT3puYVUxTGFNZmJ2djlQUUxBenZ0ZVBJU2NORzVHaVRoU051ZytBZFQveTFGNUlMK1VnS0hvemcKb2lhYXdSM3JsNVZTenY5MGpwYWcwZUEza3Q1elc1R0Z0S3Q2MzdCbVJQMnE0OGVrQjZKRjJhZUQ4MnYvT1VVTQoyR1JadDYrdXhHeWkxblRWSTA2YjZQVDFPQU5zSGZ4OUtvcTBybEdUQnY5TnFYcUtrczBldWRBUmFPUTB3UmFUCityaXA2MFhpalNFRkg2RzFlQXE3UnkvdUtTc0UzbnEwdGs3d2lWbFYwemlleVk2cjFUZnFXZjgvKzBxYndyZDIKalpVemp6UHUwSmJCZGRTeHJVUUpwcTcvSlliMjNKb0dnUVJPaTFQMUhKbjBqUVZXUkdPRUMzK2JKTitvamYzTQorSnhKUndidFNaYUphbnpMckpKbTF2MUFQMENqeENiM2N1b2lnc2JwL2RzOWF4YnFLaTduaVpvZjUwZUJSei9qCmQ3ZkJ1Sm9yQ09sbEpmM2p5ZHpXWTVqY1BSNjYyemczc2IyWUlnc2dacDJTaytCVTM0bXVCc0hsUUx6elFxRXgKc1RlQmNsd3BrQlNaODVoSzRkWjlBb0lCQVFEOUJRbGJjMUNPOFZLaFI0KzdtalJPTTMyMVNxdVlneEkveWFBWgpNUThIWEFaV1Uya084QnZ2bjZDdHI5SHlXQ005U1ZlQWEyNzdzQ1dSTkkxRGVoR0ZPekpJTUFOcDEzaGgva3lICkN3NzY4NTY0MHBtVlh3d1lSejBIN2FuUXJudHpaQ2YwV0lwWE5pd01CNnZvNkh3a1Z5M25LdDRTTWVRY2Y2bVoKYU1saHBrdlM1c3NkU25zUmRPUWZiZGhmdHF4ZXp6dlQ4WmcxNnFTeHM3b3IraytUVjlBbmZqL21HMnV3RHdLWApQb2Q5R3JUdTlJaXBuOFBvZ1Qza01xTzNCUUdJVml4V200cHc3T3dLOUhsYzNpcGR6eUV1YVcyREM0OGRyNm4zCk9lMDZ0ODJGRGZRZHZra0hUN3g2eU9aalBiRXBMNjNSYkpjVktRVFBpb1lFK1JmakFvSUJBUURXZmpVUi9QOWwKVTZjb0VhZEkrbmdvK3pVOU1WQ1VZazZnK3J1U3cvWml2ZTFBY1RUdXlZZmRQMGc1eDlDbkRLeXVhSlpEc0JXSgpiQUNCS2N6RjNYcHBoZ1BJRXA3WkFVYy9KdHZlWUNWYVlqczQ2cnZIbitNNHd2U3gvZ1JKSDhFZnZwVG1Bc1V3CktMMGFVRWVSWnFnTjZMREJRNnVRWFppR3hyd1pRdFZhTVlzVVpXSDAwTzFja0d0Mk5jUWdlOXh0YVZwV3BwY2MKWGxSYWRSK0JXaXFvSTJLdllwK1ZpU3VEQ3FnN0gwWlBVeGY1RHgvcEVYd3l0WDRkUjdaSVdvaEVoV2R2b25NLwp3VEhTSUZHc0ZBQlhVdEUrbkRHNjhJQzh3ejdNR2dEbW1adW5KbXZUN2RnMEQ0ZmtwZnQ2NHlQRHFiZ2VrUnE0CmtNUnFiL28rZmhKekFvSUJBQ1daZVFHQkNtaWpqZjd2am5sM2VleGhGSDQ3WVhLaGo3V3c5Ynlvb0s0Y1M4aksKVG1OczVRbmY4OVc3dWZ4RWdzQVp4UW9OYlFtejdIRSttUHQ5Q1Z0K0o0TzR0dGtrcnZ3SStVQmNNVmF6bTFPbAo0RExsVS92TG9sTXFTMEQ2bk5mUisyb0JPbFdwZERRNWM1K0I1T2pWbDdteGtoeXNFMVQwMTdRM2NTMjBzN0hjCm9EUnpIQVRGRlIxaWluUDJ4bmkwWFJTcTQwNm4xdCtkY0w0V1RYcjg0cTdyd1AwbCtBUEt4enpJaXdQWlV5ZnIKYU4ra0lVM00zdkNwWWREUnlwK0RCc1c4Q1grc2hZcy9OMFhJc2cwTXVjYVczakpLSm5ObjVsOFoyWm9QTkZoZwpjWFYyRUk0b21ENjYyaG5IVWl0bnJYSGdyRGIxeUlRNjlwMklrcmNDZ2dFQUdRT3c5cjhWbmVDS252Nm9EcmUvCktiQmFQaCt4TVF6aDlKNHBIKzJnbjFWZEJPWHU4V1lwWlNFTE1FbmdTSXNPM0d3d0VnYklERFcvUFZEOW81bTMKdS9TNEhpNGx2bE1LMXcrV0ZySUp5U2VwbG5LeG9ZOUE5WTZ1OE04Wit0a3NNYUtOUk9IUDAxVDg4TU9xQlFmKwowMWh1dkt0Q3A5eHlmRTFUQkhNcDVKSGdrV2xkYWlxdmM5c1gzVHBaL0J2VjBpQzJPMkxFUHJlMWlMTWVFSHpPCmVydGR0K3g5aGZmd1RpRUJDNXdUSEE4cHZZOEkvcXBlWEorVDdLQS9YdUxILzQyN1NXYWRvRExNdDUrelJxa3kKZlB5WWtIK3d3UWRwVEJwTkNZcUN6bUhkcURtdjI1QmlaWGZtTmtFeE9keWhnQ0ViNnR5clRCajkveGUrREdteApod0tDQVFFQWlxa2lHTUtBZms2cG5lelNiZUdjVis3bjNyQlB1TXJoTHVyRkNtMllpL1BSOTE1YXlLdWtYdnpVCnNleld5eFY1cUJIblZYdnNpblRYT212THBnVEovdWdzREdRQ3h2bVNaZXNqWkpNcE1VUnUvZmswS2pVK0RqZVgKeFVGZ3p5NVpiVjZlSXlwWmE4OUZqK2RYVnJqWTZsNXVnWjdhZHNJUnN5WkFMQnQwRjNaZ200REFBRWo5RWtMdgpZZjJlbUdwTGpOVE9FOTdlOXNMNC9LWEZ3UmdZbm5kc1hMdVNPQ29DWnBDTG56VENWMTdYeTR5MHlTZ3ZaTUxUCjIxaTI3WVAvNlc0WDUvNmFGVVhIOUFhaHJOMzB2MnpMVmRLeXJHQUk4amtJeHJDWW05UjBLOWR1RGdtYXNRRWYKaC9ka2dDM0x3WHB2djUyTG1rbUtZL09nTUh3OW13PT0KLS0tLS1FTkQgUFJJVkFURSBLRVktLS0tLQo=
    
root@jumpbox:~/kubernetes-the-hard-way# kubectl config set-context default \
  --cluster=kubernetes-the-hard-way \
  --user=system:node:node-1 \
  --kubeconfig=node-1.kubeconfig && cat node-1.kubeconfig
  

# use-context : current-context 에 default 추가
root@jumpbox:~/kubernetes-the-hard-way# kubectl config use-context default \
  --kubeconfig=node-0.kubeconfig
Switched to context "default".
root@jumpbox:~/kubernetes-the-hard-way# kubectl config use-context default \
  --kubeconfig=node-1.kubeconfig
Switched to context "default".


root@jumpbox:~/kubernetes-the-hard-way# ls -l *.kubeconfig
-rw------- 1 root root 10161 Jan 11 07:49 node-0.kubeconfig
-rw------- 1 root root 10161 Jan 11 07:49 node-1.kubeconfig

동일한 방식으로 kube-proxy, kube-controller-manager, kube-scheduler, admin kubeconfig 파일 생성 후 나누겠습니다.

root@jumpbox:~/kubernetes-the-hard-way# kubectl config set-cluster kubernetes-the-hard-way \
  --certificate-authority=ca.crt \
  --embed-certs=true \
  --server=https://server.kubernetes.local:6443 \
  --kubeconfig=kube-proxy.kubeconfig
Cluster "kubernetes-the-hard-way" set.

root@jumpbox:~/kubernetes-the-hard-way# kubectl config set-credentials system:kube-proxy \
  --client-certificate=kube-proxy.crt \
  --client-key=kube-proxy.key \
  --embed-certs=true \
  --kubeconfig=kube-proxy.kubeconfig
User "system:kube-proxy" set.

root@jumpbox:~/kubernetes-the-hard-way# kubectl config set-context default \
  --cluster=kubernetes-the-hard-way \
  --user=system:kube-proxy \
  --kubeconfig=kube-proxy.kubeconfig
Context "default" created.

root@jumpbox:~/kubernetes-the-hard-way# kubectl config use-context default \
  --kubeconfig=kube-proxy.kubeconfig
Switched to context "default".

root@jumpbox:~/kubernetes-the-hard-way# cat kube-proxy.kubeconfig
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUZURENDQXpTZ0F3SUJBZ0lVY1NNVHR5S29XcXlkK1lYWUZqVVZhd05pT2prd0RRWUpLb1pJaHZjTkFRRU4KQlFBd1FURUxNQWtHQTFVRUJoTUNWVk14RXpBUkJnTlZCQWdNQ2xkaGMyaHBibWQwYjI0eEVEQU9CZ05WQkFjTQpCMU5sWVhSMGJHVXhDekFKQmdOVkJBTU1Ba05CTUI0WERUSTJNREV4TURJeU1EVTFObG9YRFRNMk1ERXhNVEl5Ck1EVTFObG93UVRFTE1Ba0dBMVVFQmhNQ1ZWTXhFekFSQmdOVkJBZ01DbGRoYzJocGJtZDBiMjR4RURBT0JnTlYKQkFjTUIxTmxZWFIwYkdVeEN6QUpCZ05WQkFNTUFrTkJNSUlDSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQWc4QQpNSUlDQ2dLQ0FnRUFudE4xUGh0Q3lCZ0dYUnBWeHViaUp6emhTOStVUUpiUEgxeWd4WTNrOVVlQmZnV1YvSmxRCjBmWk1ib294aXMyYzVBR1B1SnpZbW9UbkZtZjNxRWdVY09Rc0ZVYlNGSWtSU1FOQW9JMmRUdk1ESUVLNXgrK3gKazZxN1luaVhHWFFEN3R3QUdON2dvdGt4NjNGSDBPbEJNb0tBR05QcTBNZHFGMEVacktxdXlUQ1paU09yRHNjawo2bysycGxJNWVlTmg1YzBFaUFFTmxZdFc5UVd5enJDOFZLOWNYR0hDU1hqemhUc05KQUY2a0ZvNWFUTWZtRjdDCnhBTFRqSlFpc0o0VkpZdytHdVRkS2FnME00b0svTXQySkxlcElJeGl4dCtQTUpGNWdLeGpVdDVUMmE4emZ1NkwKQlQ5UEs3VGQydWd1TEdGTzh1dEJhT0U4T0NZYzZmMGdscVJ0ZHJHSUF6dXZBNk9LZlJYMGdQQ2N4bzY3UVRQeQpMdENaRUVlWnlvV3ZoRVNRVUxuYzNINVBzVzkwMndvdEU0L0ovRzIwNzhkNlhUZzRWNS9wUzhUNDQ5Q20yZXd1Cld5bzZNc1FZMllWbUdIdzBxTDhMZ2dDRDlNVHF2QTFoWVBzOGo0NjZxUldic01nNTFoRmxXazZ5b081RS9INXcKVUhuQlQrTHVWb3RpOXBRNC9oWjg3aDZNc0NsT3lDK2xFM0ZJMlkvU0FwamZXS0Y5dHVJaHhmY3J3dW5wcnUrTAorZFY5eDZ5em45SS9xQjhadmFLeTFuS0g2RU53RG5PN2llN0YrNTFnb2FacTBFV2UrSHg1ZzJFcHUwZlArdlBPCnUrSzB5bUxCaVhhVGJBS0NTTW9vNFpab3BRQjBBTVVrb3NzNkgxYzZSRjV3bDZabFlreXcrWkVDQXdFQUFhTTgKTURvd0RBWURWUjBUQkFVd0F3RUIvekFMQmdOVkhROEVCQU1DQVFZd0hRWURWUjBPQkJZRUZKN05Ocm9pYTBPbgoyeU1wbDFNYjBYTGZiV0hTTUEwR0NTcUdTSWIzRFFFQkRRVUFBNElDQVFCd3ByOCs4bmlHRTlVakFLUUcyQU5rCnh6SG5NUFFOeU44NXpFcXJCMFp4M2tpSEJ0RS9GaHpRdHhoL0lkZnozRTN0U0k4blRNbTRsSDYycnZQdmM1NEYKMTlLMWdXV3poeExrWkMrdEsyMEpVVVN5dVJkS0VlTzZTSHB0UVlwOEZkTWVhWVlMSS9jSHVBcmRTNkw5ek1pYQpCTDRZWEdwbTFxM3lYeVRrd3VnSFJTcFlxeUJ6UDhMa3NESjU3T2EyYSs3Q0xaYXU1ZnczSkpPRFJEc3V5OXRUClhGS2UzT2VWQS9INmcvQUMwUzlKa1JDTnVKODhxOG40S0hDRTRySzA2dUErVklUaHltd1p0dHlsTkhDQUVGNncKa2ptbjQ1SndIS3ZSY2YybTVhZDlVNWNWd0p5bkJidnZXNi9lMXFOVGgyZnRvcEUvUVVraHozUk8vNWlMQ3BIQgpNUjZWZW5qY05FYTg2TnoyUTVWOFRmLzhSWHpOWjZHU1BBQXhsUENTNDNkWkpMZVZUc2N2bTBURlZzRm5kQktxCmlOWXZjUGxyWHp0diswTVJxOXpkVWpQS1g2by85L2RBOXkxZG9GQlBRMWFxRWhnUGRWQnJqenJNbjdXaUVna2cKZWF4VlZaYnovbXNxYUF1akZnRUdaZ2V5OEFWUEI3WWNKSit5dXFSV2xMbmJ0enl2MUt6WGd1SytYYjZBazRFdAp5T2pscWhINlNKNm9JODA3MldwcXp5Z2hEcG9IOWt5VGpILy8xSEVLNVZOdXUxbUYxMGVrd05VQk9wQkREdXl5CjNDS20xUmhkUHlYRjhtSUxwOU9FODBrb3E1SzZ0RndOUy9oREVSdFFXdC9zMUN2dHFLa1NxZFlSVUlzKzBzcXQKWVRCZTJIdWJZVGptaUdOMzhOZnlOQT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    server: https://server.kubernetes.local:6443
  name: kubernetes-the-hard-way
contexts:
- context:
    cluster: kubernetes-the-hard-way
    user: system:kube-proxy
  name: default
current-context: default
kind: Config
preferences: {}
users:
- name: system:kube-proxy
  user:
    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUdFakNDQS9xZ0F3SUJBZ0lVS3dQM3VtUHBKc2lsd1ovZGk5amZUS3BXOVhJd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1FURUxNQWtHQTFVRUJoTUNWVk14RXpBUkJnTlZCQWdNQ2xkaGMyaHBibWQwYjI0eEVEQU9CZ05WQkFjTQpCMU5sWVhSMGJHVXhDekFKQmdOVkJBTU1Ba05CTUI0WERUSTJNREV4TURJeU1qZzFNbG9YRFRNMk1ERXhNVEl5Ck1qZzFNbG93YmpFYU1CZ0dBMVVFQXd3UmMzbHpkR1Z0T210MVltVXRjSEp2ZUhreEhEQWFCZ05WQkFvTUUzTjUKYzNSbGJUcHViMlJsTFhCeWIzaHBaWEl4Q3pBSkJnTlZCQVlUQWxWVE1STXdFUVlEVlFRSURBcFhZWE5vYVc1bgpkRzl1TVJBd0RnWURWUVFIREFkVFpXRjBkR3hsTUlJQ0lqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FnOEFNSUlDCkNnS0NBZ0VBNEt1RGdrWVhrRzhaYVFhb2dmcnhCVFNOOXovZkhKdWxNalBBODRYL0ZHaTYzNFNMSEpSbUZybC8Kb2tzYXRjdGhBaWcwaXVlNW4wWHYwUzUxU2NhUlpHQk1NQ1dBOVVDRUdoOTRGVHZFemp4SkpnZHlLbjEreldHMAp0UFhIM2ROWFNHaWptYmpqWGI5Nmhndm0xMEN3WjZYMWVneksxTnRPUVRRVmR4emhUWUpkZjFYNU5VUERLZHo3CmdsZ1FvS2F2TW5LMmV0T2ZMTGpBamdncWsvaEZYbTloUXNCdDJDWHp6ZlF5Nkc1ZHR6OUJwSGxRT1liamJZNEEKU0grNUR4clk0MGd5dHM0TFdtNmU4bHZCditWUXhubTlJcUhFQ2Vvc3JkRENjWWx5bW9RNmpmY0JtN2dRYXFnUAptY0Q0T0ptVlVEdE1iZ2piTXBxOG4zZ1JrcXFLam9lQTN2d1BFYStvK0UvcHpYQy9udEZ5WEhzQjNsclNkOUNhClN6V2xmVkFyQWlSOVBQdHMwbjFqMCtzTFgxSytMYndQVUh0cjliRWVzV0RSbW9kODE0QXRsY3lPVVkvYlBqbHMKaEtENDF1Qk9ZcU0zVGJucHVQTjBkNDBtOEFJYXpGY0p6RFU4Vk5FLzU1ellIeHlXcEQ3dm5JclZHclVNYVA1NApPcUxKRE56Si9XQmhFcklrRFlEZ1lwSEFIL3RFcEZqc3pKM3BZZllBb2hBaDdIL0JYSHNGYmlPUmpZUnZTckw1CjJaQ3lUQkFpV1QrWEl3SE9aY2EySnhaNXRIejIzQ1EwWFYwaW9WRldldDIxZ0p4MWFKalFEN1RlbXVoT0RDTnEKQmQ0MDY5UjVqOEY2QW5oSjZvaDZIRFBtVGFtRHlOLzNtc2JaU0xUbk9McnUwQzFnS3EwQ0F3RUFBYU9CMURDQgowVEFKQmdOVkhSTUVBakFBTUIwR0ExVWRKUVFXTUJRR0NDc0dBUVVGQndNQ0JnZ3JCZ0VGQlFjREFUQU9CZ05WCkhROEJBZjhFQkFNQ0JhQXdFUVlKWUlaSUFZYjRRZ0VCQkFRREFnZUFNQ1VHQ1dDR1NBR0crRUlCRFFRWUZoWkwKZFdKbElGQnliM2g1SUVObGNuUnBabWxqWVhSbE1Cc0dBMVVkRVFRVU1CS0NDbXQxWW1VdGNISnZlSG1IQkg4QQpBQUV3SFFZRFZSME9CQllFRkw1OVR0eVN3S3E1YkVFZTFTK2l4aVo1OVdOU01COEdBMVVkSXdRWU1CYUFGSjdOCk5yb2lhME9uMnlNcGwxTWIwWExmYldIU01BMEdDU3FHU0liM0RRRUJDd1VBQTRJQ0FRQ05WRURkSlFsSlozNVYKVEM1UmJITExQTEk2SFJOeTVvUm9UUGFRazhPSWFWZTZULzR0Vk15bUNwZklVR1JxQVFqRkZDcVBua3R1QjBEKwpmd1RoL01xMDNYNDRwRTlwWmFEbVIzWk5qVVpkV045NjF2UnJSTTVaVzhzaDIrVk5CN2lpc2VHY3h1ZDN1TnEvCmlEdjV4d3BEQUZib1R5ZDE0VDFsT0NIbW45eFBTcDQ1aGtKWjhYT00yT1dJNTlINWZ0Y2lLNmdCd1Y4OWxMSHQKNGFyMmZ6VEdaR3hPOW04ZHRsbUxXaVg1S1U0VVlyVTN0OW5pUGV3a2g0VzhMM0FvVU1QSlZFa0F1VlNBZE81NgpBTHRibDdJbW5tZ3RRU1gvS0dheDVGdEl5TUhoMGZ0cFlCNFJGQnRBa0FXQ04zN04yNmJGWnB5TEdOVDE3MFArCkU2QTZoVjlWUVZDeXV1TzdSN3VUcFRDR1BmU1VxZGNJbVVDVUxlVUR5d3VnR1R4VTVwSFNiQ1VMTmdDU2FHcWcKeUtkV04vb3dicVY1Tkd6WWdtVnJ3RUV1WWtUYi9RS3h4SWF0UEEyRmFySlBSK2xBRFYyRXVid1V6ZzVJN21DUApjb040WXA1Y2N4ZzQ1bWNDOE5iN0J3OTJHREhYeUZoWGc4UWJuREVUWThwdmQzWWVhTXhCSHF6ZlM5S0lqa25aCkYzd29aQTcwWlVPQlkrMk1NcURRZzUxYnBaM0d6Zkk5L0IxVE92MHpTbXFLa2xpMmRDNHNzVGNyaTNTU3h0emQKY0VNMWlpdVBPaXN5YnVEMWhIeXpDYnVEU0MyZTdXMUU4UmttajRmQjhVVzF0NFYxdHJ3c3hINFd0UGdkMmJKWQpPcEkwR3FEeDU3TEduMFFXSms1OW9lRFFQU0ZEVEE9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
    client-key-data: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUpRd0lCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQ1Mwd2dna3BBZ0VBQW9JQ0FRRGdxNE9DUmhlUWJ4bHAKQnFpQit2RUZOSTMzUDk4Y202VXlNOER6aGY4VWFMcmZoSXNjbEdZV3VYK2lTeHExeTJFQ0tEU0s1N21mUmUvUgpMblZKeHBGa1lFd3dKWUQxUUlRYUgzZ1ZPOFRPUEVrbUIzSXFmWDdOWWJTMDljZmQwMWRJYUtPWnVPTmR2M3FHCkMrYlhRTEJucGZWNkRNclUyMDVCTkJWM0hPRk5nbDEvVmZrMVE4TXAzUHVDV0JDZ3BxOHljclo2MDU4c3VNQ08KQ0NxVCtFVmViMkZDd0czWUpmUE45RExvYmwyM1AwR2tlVkE1aHVOdGpnQklmN2tQR3RqalNESzJ6Z3RhYnA3eQpXOEcvNVZER2ViMGlvY1FKNml5dDBNSnhpWEthaERxTjl3R2J1QkJxcUErWndQZzRtWlZRTzB4dUNOc3ltcnlmCmVCR1Nxb3FPaDREZS9BOFJyNmo0VCtuTmNMK2UwWEpjZXdIZVd0SjMwSnBMTmFWOVVDc0NKSDA4KzJ6U2ZXUFQKNnd0ZlVyNHR2QTlRZTJ2MXNSNnhZTkdhaDN6WGdDMlZ6STVSajlzK09XeUVvUGpXNEU1aW96ZE51ZW00ODNSMwpqU2J3QWhyTVZ3bk1OVHhVMFQvbm5OZ2ZISmFrUHUrY2l0VWF0UXhvL25nNm9za00zTW45WUdFU3NpUU5nT0JpCmtjQWYrMFNrV096TW5lbGg5Z0NpRUNIc2Y4RmNld1Z1STVHTmhHOUtzdm5aa0xKTUVDSlpQNWNqQWM1bHhyWW4KRm5tMGZQYmNKRFJkWFNLaFVWWjYzYldBbkhWb21OQVB0TjZhNkU0TUkyb0YzalRyMUhtUHdYb0NlRW5xaUhvYwpNK1pOcVlQSTMvZWF4dGxJdE9jNHV1N1FMV0FxclFJREFRQUJBb0lDQUFjNS9SVG5wRUpsRUhBc1lxY3JweXovCnI4RlFNS0QrOEtWWXZOQVljMGY0eG1vQ0hXZGg3cGEwaU1Kem4veFErbUhUWVQxR2p0WHNyZllRaEgxSEFjNDAKdXI2aHRqQXRUeEo0Q0tqR0wySmkxZ2QvSGRENEVHZ1N5SGs1bnFBbitCckFJdU1DNjIxTWgyaGdZYUJ5MkRtOQpRTlFZTFQ0RGY5VGhFLytzMGlKb0d4SEpkaWFjYzE2WldRMnNQM3lyWDk5RGlWTVNFNENnd0xEUTJtM0FBcHN6CkFmSGZ0cWpKTDdwa0xHOEZmY1l6SnExZnFaRHdJY2lZVkxJQVNiZU9xWU5HQXp1V0RROXZsM1U4TmxNeDBSTlIKUHlBcUMrQTF4QklqTU4zdUNkMm1leXV0MGtOci9Ec3lCSzV4WW9xUGhobjZqazdMd2lFeWtQYWdLaENKbzhNNgpieGZMbGdnSFhhdSsrbkNvRVF6Q3VKdVZLV2IvOS9OTE1HUmlEYW1PcERtc2xFa2tZSHBhaVBFZ0p2amlpQzJsCnFSMXBEZitJb1YwdWFRMWlWdGJPY3RpeVhYclMrRkFEWVFnaXY2V1BVNnVWRTk2M1hqWk9kWCt1V0RUdGtuZ2MKSmJLejVWeThxQUl0dWlNQWFiZjR4dkdMRG1XSVJQbm9FY2JRZVZUaktOcG8wWkw2Q2kySlpOR0w1cnFqaXBPZwo3ZzhhOVRvREk0aWZYakRXeGd6MmkyNTNJMStML3lzR1hzVklCUnF0Wnl5dDVYejQ1d1BXL3IwMm9hQ21KR0RqCmhDWW5KVVhFUzdBM3ZpQS9OVWdFa1E5eE13cTVibEN3eVlRRlU5VGdiZ3NDWjkybkJNamh2REZZOWtzeUlDRWIKQmV4NjkvWkYwTS9IZFNSS1JwOFJBb0lCQVFENnE0eEQvaXhQbEVuSURWZkhaWFFkb0dDY0dyd09yL2tWR0R6Qgo3a0diK2dTZm5xMTF2NEthNWYzUVluQ1ZEQWVmK3dPL2Viaksvb2UwK2dFc1lXZEk0b0h1YjFHUkVEUk1wdElNCk54bkRuS1BWSUVRL2JadTJla3MydUVuYzNnNTdYbjNFSjNzSWIwU04wSDFYWDZRVkhaLzRkTVcvR3kvZ2dTTlAKbE1TZnRKRXZERFR5dGhCWHVkVm5HQk9qbGwzUlBuQzhHRkdOR1IyNlFoeisrNy9Zb2dvb1hFYlBrelFPQjRMYwpValY1YlI0ck9wWmJLQWR5SnFEbTJQc1F6RjBuSzFxaElFQ3g5ekFodmNwcWljcFhjU0NuRG9WS25pdXdJdEZmCm5OYWNnMXQ4S0JGZmRzaG4vdVFOYjgyQjZXZWJ5WC8yeWc4TkZyNWY5YXc0ejd3SEFvSUJBUURsY25EL1FJeFAKVzdZNVl3L1UzVGxOcEljU1JsTXhpNEs1SGdwRTFXWHR5QWFtb3BvZzdSb1ZsVEhJRVQ4eUpyWW8zQkE4UXMzVApFbGRjWkFMWGI5S3JyVTUvbmlCOHRrWXltSlI5ZDJEdzAyVlZNenlUb2MwQlFGbUlldUd0ejdtbmVPMFlMWXhOClZ0bTlZZVYzNCtNekxueVNZalFYSjluSGlYRm5scnBWcFlGZERYdXJYM2VnWFJZUTRKTWNSUGJzaEJmQVFpRmUKQjV0TXllMEVqTThVKzFBOURLR2ZKL3kyTjF3eDVsR01TQ3B1NzkvcHIvNWp5c001K0VBQ0pNaldpSkNxR3dwegplZkprdXYzdFBjNmJsOFNFaGtIRGMySkNidkNnbmFUNWJjVjJDSlZzRkxsaGNaMkpIaEpSOG1XOXZYZDhkQjFECjIzVDZyZUNCRlY2ckFvSUJBUUN5cTc0aDVMUyswRXE2emsvbjczZVJlblFDTU1na2krZTEwbjNVR1Rnb3JrL0gKV25uM09OY21BNU1qaWxjRzFmbEZYY3p4R1JqUGtCYzR0dzVKRFBCcXNoQzgzQm0rbFZkQjM1Sk9aS1AzbXRubwpHc1NxbFR6dEYwUXZMN0p1UE5VRHVIbGFiRzIwdUdLYjFxVFF0M29rRnVha1dNOXVoVGxlQTQ0Z3RrQmtPWGRXCm1wUFBBRmlydnVqT1FBb1hpeUU1K1JDdjhZR0hXZ1NuYzJKcG1SWEltczgwcjNsRmtYNEVGZlIvaUJWWGhLQ0IKWFpRNFMxTHJXWDlNckJKZ2lkcElCZzNtVlMvbjZmSWp5cnpSclg5TlRuUmd2dWZoSk5hdENGUkdtMnNaU2tQVwpQbFVMdjRYZkluNTQyVjBaSVQzWmZBbTY1WkZPMmd3K0o4Vm5CU0xYQW9JQkFRQ3Y5amdhU1JKUnVlL2h5c3JWCllHTVYzRnVKOUV6UTdrRHVYczlaUXA3TTRGM1paVkgyZGw5T3RnQmJnbjZJbzQrSGZsMGxHY1puWE9XS2dWaDgKaDlGMHY1QTM1cmpiYTU0aUowbVpmdmZsT1BtNWdYV3ZjcUtabHhRU3lzcFFWY0gzWVpIU3ZQdWdSRDd3TWllYwo3alBMaUpjWVhCKytzbDBkM1pZTzcxVXZZYUg0RHVNTTVrWUR3ZCt1U2N3SUhndVFEeTEzMEQ5dXNUNjBnUmM0CktoU1Y3clgyVzcySEpVOTFIcmVodlJSRkZRTVYyUGhMYXlFWXZ1TkJqZFRvei9VR01lMWZ4eVVZc2lqbWYxRkYKbytUVEJtMDdoUXJpTFpicGc5TWVHa2Z1MGZSaE5Pd0MyMUhnSFJYSGxqeE9oQ3JxZkdjcUZwSnErOTZXY01wLwp0RGZKQW9JQkFIWUYxa01kL0I2cUdlVEsxaTlBK3dQQ3AzQXhWTWxjWTVSbVFsWTIvcXRyN3plYVNqTlorUisrCk96OGNBVWFHUkxOVzRiNVkzU3NTaDhVNEZUUGZDN3d3RXJDcnl2SEdMbzcxTlZwdEVRQkFza3BkQkw4VFBHQ2cKMWhSbG0zTWlsSmhGYmxNalp0QVVLWGdsajlaSklZcW9ZRTY2NWlIeHV2ZGdmT2RaU2RoMzhHL3ptODB5czRWZQpRWnNhRjJ5a2RmSjU4M1NWVWluNHViOXRhMll4cU9MTm1iNnhRMkJmWVB4b0luOXB0YTA2bFcxUTNuNDFUOElSCndya1A5QllYZXlVZlRXeTgxazFXdFRYUmV0VnZ1S3JaeDZFdy9oWTNwcjdvTzFnRFlzTVgzVDQ2bFRQS0Z0aVkKOWMzcHphc3JyT01Bd1dDTDFaakxCUHYra1JRT09JTT0KLS0tLS1FTkQgUFJJVkFURSBLRVktLS0tLQo=

...


root@jumpbox:~/kubernetes-the-hard-way# ls -l *.kubeconfig
-rw------- 1 root root  9953 Jan 11 07:53 admin.kubeconfig
-rw------- 1 root root 10305 Jan 11 07:53 kube-controller-manager.kubeconfig
-rw------- 1 root root 10187 Jan 11 07:52 kube-proxy.kubeconfig
-rw------- 1 root root 10211 Jan 11 07:53 kube-scheduler.kubeconfig
-rw------- 1 root root 10161 Jan 11 07:49 node-0.kubeconfig
-rw------- 1 root root 10161 Jan 11 07:49 node-1.kubeconfig

# kubelet, kube-proxy kubeconfig 파일을 node-0, node-1에 복사
root@jumpbox:~/kubernetes-the-hard-way# for host in node-0 node-1; do
  ssh root@${host} "mkdir -p /var/lib/{kube-proxy,kubelet}"

  scp kube-proxy.kubeconfig \
    root@${host}:/var/lib/kube-proxy/kubeconfig \

  scp ${host}.kubeconfig \
    root@${host}:/var/lib/kubelet/kubeconfig
done
kube-proxy.kubeconfig                                                                                                                                        100%   10KB   8.8MB/s   00:00
node-0.kubeconfig                                                                                                                                            100%   10KB  12.1MB/s   00:00
kube-proxy.kubeconfig                                                                                                                                        100%   10KB  12.0MB/s   00:00
node-1.kubeconfig                                                                                                                                            100%   10KB  10.5MB/s   00:00

root@jumpbox:~/kubernetes-the-hard-way# ssh node-0 ls -l /var/lib/*/kubeconfig
-rw------- 1 root root 10161 Jan 11 07:55 /var/lib/kubelet/kubeconfig
-rw------- 1 root root 10187 Jan 11 07:55 /var/lib/kube-proxy/kubeconfig
root@jumpbox:~/kubernetes-the-hard-way# ssh node-1 ls -l /var/lib/*/kubeconfig
-rw------- 1 root root 10161 Jan 11 07:55 /var/lib/kubelet/kubeconfig
-rw------- 1 root root 10187 Jan 11 07:55 /var/lib/kube-proxy/kubeconfig

# kube-controller-manager,kube-scheduler kubeconfig files을 서버로 복사
root@jumpbox:~/kubernetes-the-hard-way# scp admin.kubeconfig \
  kube-controller-manager.kubeconfig \
  kube-scheduler.kubeconfig \
  root@server:~/
admin.kubeconfig                                                                                                                                             100% 9953     9.7MB/s   00:00
kube-controller-manager.kubeconfig                                                                                                                           100%   10KB  12.8MB/s   00:00
kube-scheduler.kubeconfig                                                                                                                                    100%   10KB  14.4MB/s   00:00

root@jumpbox:~/kubernetes-the-hard-way# ssh server ls -l /root/*.kubeconfig
-rw------- 1 root root  9953 Jan 11 07:56 /root/admin.kubeconfig
-rw------- 1 root root 10305 Jan 11 07:56 /root/kube-controller-manager.kubeconfig
-rw------- 1 root root 10211 Jan 11 07:56 /root/kube-scheduler.kubeconfig

데이터 암호화를 위한 설정 및 키 생성

이 단계는 Kubernetes API Server가 Secret을 etcd에 저장할 때 평문이 아닌 암호화된 형태로 저장하도록 설정하는 과정입니다.

# 암호키 생성
root@jumpbox:~/kubernetes-the-hard-way# export ENCRYPTION_KEY=$(head -c 32 /dev/urandom | base64)
echo $ENCRYPTION_KEY
pR499HLO7RNc0VAgKmO9WqlfgE/rqBRsGskyllCp3Xo=


# Create the encryption-config.yaml encryption config file
# (참고) 실제 etcd 값에 기록되는 헤더 : k8s:enc:aescbc:v1:key1:<ciphertext>
root@jumpbox:~/kubernetes-the-hard-way# cat configs/encryption-config.yaml
kind: EncryptionConfiguration  # kube-apiserver가 etcd에 저장할 리소스를 어떻게 암호화할지 정의
apiVersion: apiserver.config.k8s.io/v1  # --encryption-provider-config 플래그로 참조
resources:
  - resources:
      - secrets  # 암호화 대상 Kubernetes 리소스 : 여기서는 Secret 리소스만 암호화
    providers:  # 지원 providers(위 부터 적용됨) : identity, aescbc, aesgcm, kms v2, secretbox
      - aescbc:  # etcd에 저장될 Secret을 AES-CBC 방식으로 암호화
          keys:
            - name: key1  # 키 식별자 (etcd 데이터에 기록됨)
              secret: ${ENCRYPTION_KEY}
      - identity: {}  # 암호화하지 않음 (Plaintext), 주로 하위 호환성 / 점진적 암호화 목적
      
      
root@jumpbox:~/kubernetes-the-hard-way# cat configs/encryption-config.yaml
kind: EncryptionConfiguration
apiVersion: apiserver.config.k8s.io/v1
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: ${ENCRYPTION_KEY}
      - identity: {}
root@jumpbox:~/kubernetes-the-hard-way#
root@jumpbox:~/kubernetes-the-hard-way#
root@jumpbox:~/kubernetes-the-hard-way# envsubst < configs/encryption-config.yaml > encryption-config.yaml
root@jumpbox:~/kubernetes-the-hard-way# cat encryption-config.yaml
kind: EncryptionConfiguration
apiVersion: apiserver.config.k8s.io/v1
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: pR499HLO7RNc0VAgKmO9WqlfgE/rqBRsGskyllCp3Xo=
      - identity: {}
      
# encryption-config.yaml, encryption config file을 controller instance에 복사
root@jumpbox:~/kubernetes-the-hard-way# scp encryption-config.yaml root@server:~/
encryption-config.yaml                                                                                                                                     100%  271   343.0KB/s   00:00
root@jumpbox:~/kubernetes-the-hard-way# ssh server ls -l /root/encryption-config.yaml
-rw-r--r-- 1 root root 271 Jan 11 08:04 /root/encryption-config.yaml

etcd 클러스터 부트스트래핑

server 노드에 etcd 서비스 기동하겠습니다.

# Prerequisites

# hostname 변경 : controller -> server
# http 평문 통신!
# 각 etcd 멤버는 etcd 클러스터에서 모두 고유한 이름을 가져야합니다. 
# etcd 이름을 현재의 compute instance에 맞추어 지정합니다.

root@jumpbox:~/kubernetes-the-hard-way# cat units/etcd.service | grep controller
  --name controller \
  --initial-cluster controller=http://127.0.0.1:2380 \

root@jumpbox:~/kubernetes-the-hard-way# ETCD_NAME=server
root@jumpbox:~/kubernetes-the-hard-way# cat > units/etcd.service <<EOF
[Unit]
Description=etcd
Documentation=https://github.com/etcd-io/etcd

[Service]
Type=notify
ExecStart=/usr/local/bin/etcd \\
  --name ${ETCD_NAME} \\
  --initial-advertise-peer-urls http://127.0.0.1:2380 \\
  --listen-peer-urls http://127.0.0.1:2380 \\
  --listen-client-urls http://127.0.0.1:2379 \\
  --advertise-client-urls http://127.0.0.1:2379 \\
  --initial-cluster-token etcd-cluster-0 \\
  --initial-cluster ${ETCD_NAME}=http://127.0.0.1:2380 \\
  --initial-cluster-state new \\
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF


root@jumpbox:~/kubernetes-the-hard-way# cat units/etcd.service | grep server
  --name server \
  --initial-cluster server=http://127.0.0.1:2380 \


# etcd 바이너리 파일과 systemd 유닛 파일을 서버에 복사합니다. 
root@jumpbox:~/kubernetes-the-hard-way# scp \
  downloads/controller/etcd \
  downloads/client/etcdctl \
  units/etcd.service \
  root@server:~/
etcd                                                                                                                                                       100%   23MB  96.5MB/s   00:00
etcdctl                                                                                                                                                    100%   15MB 135.0MB/s   00:00
etcd.service                                                                                                                                               100%  564     1.2MB/s   00:00


# 아래는 server 가상머신 접속 후 명령 실행

root@jumpbox:~/kubernetes-the-hard-way# ssh root@server

# Bootstrapping an etcd Cluster
# etcd 바이너리 파일 설치
root@server:~# pwd
/root
root@server:~# mv etcd etcdctl /usr/local/bin/

# etcd 서버 설정
root@server:~# mkdir -p /etc/etcd /var/lib/etcd
root@server:~# chmod 700 /var/lib/etcd
root@server:~# cp ca.crt kube-api-server.key kube-api-server.crt /etc/etcd/

# etcd.service, systemd 유닛 파일 생성
root@server:~# mv etcd.service /etc/systemd/system/
root@server:~# tree /etc/systemd/system/
/etc/systemd/system/
├── apt-daily.service -> /dev/null
├── apt-daily-upgrade.service -> /dev/null
├── dbus-org.freedesktop.timesync1.service -> /lib/systemd/system/systemd-timesyncd.service
├── etcd.service
├── getty.target.wants
│   └── getty@tty1.service -> /lib/systemd/system/getty@.service
├── multi-user.target.wants
│   ├── console-setup.service -> /lib/systemd/system/console-setup.service
│   ├── cron.service -> /lib/systemd/system/cron.service
│   ├── e2scrub_reap.service -> /lib/systemd/system/e2scrub_reap.service
│   ├── networking.service -> /lib/systemd/system/networking.service
│   ├── nfs-client.target -> /lib/systemd/system/nfs-client.target
│   ├── remote-fs.target -> /lib/systemd/system/remote-fs.target
│   ├── rpcbind.service -> /lib/systemd/system/rpcbind.service
│   ├── ssh.service -> /lib/systemd/system/ssh.service
│   ├── vboxadd.service -> /lib/systemd/system/vboxadd.service
│   └── vboxadd-service.service -> /lib/systemd/system/vboxadd-service.service
├── network-online.target.wants
│   └── networking.service -> /lib/systemd/system/networking.service
├── paths.target.wants
│   └── acpid.path -> /lib/systemd/system/acpid.path
├── remote-fs.target.wants
│   └── nfs-client.target -> /lib/systemd/system/nfs-client.target
├── sockets.target.wants
│   ├── acpid.socket -> /lib/systemd/system/acpid.socket
│   ├── dm-event.socket -> /lib/systemd/system/dm-event.socket
│   └── rpcbind.socket -> /lib/systemd/system/rpcbind.socket
├── sshd.service -> /lib/systemd/system/ssh.service
├── sysinit.target.wants
│   ├── blk-availability.service -> /lib/systemd/system/blk-availability.service
│   ├── keyboard-setup.service -> /lib/systemd/system/keyboard-setup.service
│   ├── lvm2-lvmpolld.socket -> /lib/systemd/system/lvm2-lvmpolld.socket
│   ├── lvm2-monitor.service -> /lib/systemd/system/lvm2-monitor.service
│   ├── systemd-pstore.service -> /lib/systemd/system/systemd-pstore.service
│   └── systemd-timesyncd.service -> /lib/systemd/system/systemd-timesyncd.service
└── timers.target.wants
    ├── dpkg-db-backup.timer -> /lib/systemd/system/dpkg-db-backup.timer
    ├── e2scrub_all.timer -> /lib/systemd/system/e2scrub_all.timer
    ├── fstrim.timer -> /lib/systemd/system/fstrim.timer
    ├── logrotate.timer -> /lib/systemd/system/logrotate.timer
    └── man-db.timer -> /lib/systemd/system/man-db.timer

9 directories, 33 files

# etcd 서버 시작
root@server:~# systemctl daemon-reload
root@server:~# systemctl enable etcd
Created symlink /etc/systemd/system/multi-user.target.wants/etcd.service → /etc/systemd/system/etcd.service.
root@server:~# systemctl start etcd

root@server:~# systemctl status etcd --no-pager
● etcd.service - etcd
     Loaded: loaded (/etc/systemd/system/etcd.service; enabled; preset: enabled)
     Active: active (running) since Sun 2026-01-11 08:19:06 KST; 2min 11s ago
       Docs: https://github.com/etcd-io/etcd
   Main PID: 2627 (etcd)
      Tasks: 7 (limit: 2096)
     Memory: 9.2M
        CPU: 1.148s
     CGroup: /system.slice/etcd.service
             └─2627 /usr/local/bin/etcd --name server --initial-advertise-peer-urls http://127.0.0.1:2380 --listen-peer-urls http://127.0.0.1:2380 --listen-client-urls http://127.0.0.1:237…

Jan 11 08:19:06 server etcd[2627]: {"level":"info","ts":"2026-01-11T08:19:06.399737+0900","caller":"membership/cluster.go:647","msg":"set initial cluster version","cluster-i…version":"3.6"}
Jan 11 08:19:06 server etcd[2627]: {"level":"info","ts":"2026-01-11T08:19:06.399834+0900","caller":"api/capability.go:76","msg":"enabled capabilities for version","cluster-version":"3.6"}
Jan 11 08:19:06 server etcd[2627]: {"level":"info","ts":"2026-01-11T08:19:06.399881+0900","caller":"etcdserver/server.go:2383","msg":"cluster version is updated","cluster-version":"3.6"}
Jan 11 08:19:06 server etcd[2627]: {"level":"info","ts":"2026-01-11T08:19:06.400006+0900","caller":"etcdmain/main.go:44","msg":"notifying init daemon"}
Jan 11 08:19:06 server etcd[2627]: {"level":"info","ts":"2026-01-11T08:19:06.400040+0900","caller":"version/monitor.go:116","msg":"cluster version differs from storage versi…rsion":"3.5.0"}
Jan 11 08:19:06 server etcd[2627]: {"level":"info","ts":"2026-01-11T08:19:06.400109+0900","caller":"schema/migration.go:65","msg":"updated storage version","new-storage-version":"3.6.0"}
Jan 11 08:19:06 server etcd[2627]: {"level":"info","ts":"2026-01-11T08:19:06.402219+0900","caller":"v3rpc/health.go:63","msg":"grpc service status changed","service":"","status":"SERVING"}
Jan 11 08:19:06 server etcd[2627]: {"level":"info","ts":"2026-01-11T08:19:06.404687+0900","caller":"etcdmain/main.go:50","msg":"successfully notified init daemon"}
Jan 11 08:19:06 server systemd[1]: Started etcd.service - etcd.
Jan 11 08:19:06 server etcd[2627]: {"level":"info","ts":"2026-01-11T08:19:06.407277+0900","caller":"embed/serve.go:220","msg":"serving client traffic insecurely; this is str…27.0.0.1:2379"}
Hint: Some lines were ellipsized, use -l to show in full.

root@server:~# ss -tnlp | grep etcd
LISTEN 0      4096       127.0.0.1:2379      0.0.0.0:*    users:(("etcd",pid=2627,fd=6))
LISTEN 0      4096       127.0.0.1:2380      0.0.0.0:*    users:(("etcd",pid=2627,fd=3))

root@server:~# etcdctl member list
6702b0a34e2cfd39, started, server, http://127.0.0.1:2380, http://127.0.0.1:2379, false

root@server:~# etcdctl member list -w table
+------------------+---------+--------+-----------------------+-----------------------+------------+
|        ID        | STATUS  |  NAME  |      PEER ADDRS       |     CLIENT ADDRS      | IS LEARNER |
+------------------+---------+--------+-----------------------+-----------------------+------------+
| 6702b0a34e2cfd39 | started | server | http://127.0.0.1:2380 | http://127.0.0.1:2379 |      false |
+------------------+---------+--------+-----------------------+-----------------------+------------+

root@server:~# etcdctl endpoint status -w table
+----------------+------------------+------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
|    ENDPOINT    |        ID        |  VERSION   | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |
+----------------+------------------+------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| 127.0.0.1:2379 | 6702b0a34e2cfd39 | 3.6.0-rc.3 |           3.6.0 |   20 kB |  16 kB |                   20% |   0 B |      true |      false |         2 |          4 |                  4 |        |                          |             false |
+----------------+------------------+------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+

컨트롤 플레인 부트스트래핑

server 노드에 api server, scheduler, kube controller manager 서비스 기동하겠습니다.

# api -> kubelet 호출 시 Flow
kube-apiserver (client)
  |
  | (TLS client cert, CN=kubernetes)
  ↓
kubelet API Server 역할 (/stats, /log, /metrics)
  |
  ↓
RBAC 평가:
  User = kubernetes
  → ClusterRoleBinding system:kube-apiserver 매칭
  → ClusterRole system:kube-apiserver-to-kubelet 권한 부여

# kube-apiserver.service 수정 : service-cluster-ip-range 추가
# https://github.com/kelseyhightower/kubernetes-the-hard-way/issues/905
# service-cluster-ip 값은 ca.conf 에 설정한 [kube-api-server_alt_names] 항목의 Service IP 범위
root@jumpbox:~/kubernetes-the-hard-way# cat ca.conf | grep '\[kube-api-server_alt_names' -A2
[kube-api-server_alt_names]
IP.0  = 127.0.0.1
IP.1  = 10.32.0.1


root@jumpbox:~/kubernetes-the-hard-way# cat << EOF > units/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-apiserver \\
  --allow-privileged=true \\
  --apiserver-count=1 \\
  --audit-log-maxage=30 \\
  --audit-log-maxbackup=3 \\
  --audit-log-maxsize=100 \\
  --audit-log-path=/var/log/audit.log \\
  --authorization-mode=Node,RBAC \\
  --bind-address=0.0.0.0 \\
  --client-ca-file=/var/lib/kubernetes/ca.crt \\
  --enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \\
  --etcd-servers=http://127.0.0.1:2379 \\
  --event-ttl=1h \\
  --encryption-provider-config=/var/lib/kubernetes/encryption-config.yaml \\
  --kubelet-certificate-authority=/var/lib/kubernetes/ca.crt \\
  --kubelet-client-certificate=/var/lib/kubernetes/kube-api-server.crt \\
  --kubelet-client-key=/var/lib/kubernetes/kube-api-server.key \\
  --runtime-config='api/all=true' \\
  --service-account-key-file=/var/lib/kubernetes/service-accounts.crt \\
  --service-account-signing-key-file=/var/lib/kubernetes/service-accounts.key \\
  --service-account-issuer=https://server.kubernetes.local:6443 \\
  --service-cluster-ip-range=10.32.0.0/24 \\
  --service-node-port-range=30000-32767 \\
  --tls-cert-file=/var/lib/kubernetes/kube-api-server.crt \\
  --tls-private-key-file=/var/lib/kubernetes/kube-api-server.key \\
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

# kube-apiserver가 kubelet(Node)에 접근할 수 있도록 허용하는 '시스템 내부용 RBAC' 설정
root@jumpbox:~/kubernetes-the-hard-way# cat configs/kube-apiserver-to-kubelet.yaml ; echo
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"  # Kubernetes가 업그레이드 시 자동 관리
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-apiserver-to-kubelet
rules:
  - apiGroups:
      - ""
    resources:
      - nodes/proxy
      - nodes/stats
      - nodes/log
      - nodes/spec
      - nodes/metrics
    verbs:
      - "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:kube-apiserver  # 누가 이 권한을 쓰는가? → kube-apiserver 자신
  namespace: ""
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-apiserver-to-kubelet
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: kubernetes  # 사용자 kubernetes ,이 사용자는 kube-apiserver가 사용하는 클라이언트 인증서의 CN
    

# api-server : Subject CN 확인
root@jumpbox:~/kubernetes-the-hard-way# openssl x509 -in kube-api-server.crt -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            2b:03:f7:ba:63:e9:26:c8:a5:c1:9f:dd:8b:d8:df:4c:aa:56:f5:75
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, ST = Washington, L = Seattle, CN = CA
        Validity
            Not Before: Jan 10 22:28:57 2026 GMT
            Not After : Jan 11 22:28:57 2036 GMT
        Subject: CN = kubernetes, C = US, ST = Washington, L = Seattle
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)
                Modulus:
                    00:96:43:8b:18:ed:f0:ab:92:9b:3d:21:22:28:f7:
                    13:48:d0:5e:26:0e:f9:e8:09:ce:29:8b:e4:f5:2a:
                    7b:bd:33:37:8d:a7:a5:42:28:2c:c0:ee:35:db:11:
                    0a:84:da:d3:ef:c1:7f:fa:bb:66:81:bf:d9:8a:fb:
                    74:c7:ed:20:e6:81:f8:5c:2a:98:18:c7:10:61:6d:
                    e8:0d:44:6b:7a:2c:c4:ad:e0:21:b9:d2:dd:dc:53:
                    56:8c:1c:03:e6:40:bc:e1:7b:32:b5:7b:2b:e6:85:
                    4d:6b:e2:db:20:e6:7b:ac:18:bf:69:ae:ca:0e:1b:
                    45:14:0c:0a:5d:c1:c6:30:7c:5e:d4:87:45:4a:97:
                    54:16:7e:b6:99:16:36:98:30:cd:60:4b:cb:0d:9f:
                    95:30:c5:f6:7c:0e:9d:6f:e2:ea:b1:a1:3a:aa:c6:
                    0e:e8:35:75:55:e3:a1:ef:6b:fd:0f:2e:a1:1b:14:
                    d5:e6:e5:16:67:a0:65:3c:a7:9b:b6:34:44:f6:60:
                    27:64:39:0a:c2:ca:51:db:89:44:6a:6c:cc:47:ec:
                    73:72:05:9d:b2:82:76:4d:60:e3:3a:e8:cf:d4:61:
                    6b:aa:ab:6c:2d:85:13:36:fb:01:04:b7:57:7c:91:
                    5d:08:4f:61:5c:d3:a3:62:8f:1a:2f:51:24:a1:9a:
                    a9:5a:8d:0c:04:e2:e4:16:ad:46:af:99:45:32:a5:
                    aa:a7:b7:6a:37:9b:60:aa:76:24:5f:48:40:76:b1:
                    09:0f:b0:85:04:eb:d5:0d:2e:86:d2:4a:e5:10:e7:
                    36:28:e8:d8:41:ba:e5:5d:98:e2:72:97:cf:6d:89:
                    4b:59:1a:44:f9:72:1d:f8:e9:03:01:3b:ac:61:b6:
                    b0:95:c0:c2:7a:b4:e4:da:7a:54:be:a6:7c:1a:4f:
                    57:66:b3:26:67:0d:7b:f2:60:ce:99:c5:79:3b:86:
                    79:dd:50:67:c3:4a:85:fb:e4:47:61:c8:7f:07:c4:
                    98:1f:d0:f4:e5:53:c4:ad:d9:e9:ab:4c:14:6b:3c:
                    50:c9:5c:6b:d5:7c:cd:bd:c4:a2:2e:e5:57:61:3e:
                    7f:b5:a2:59:42:c8:46:64:31:82:a3:d4:46:55:35:
                    db:cc:4b:ce:54:47:83:d7:7e:56:ec:76:8c:7c:67:
                    00:02:9d:b4:d1:fc:e0:60:a3:c7:9b:6a:28:3e:cf:
                    0b:e4:50:f8:78:32:ac:75:8c:ba:8e:8d:8a:1a:62:
                    ed:1a:74:c6:71:1a:c1:4c:ec:f6:0a:c5:0d:ae:1a:
                    c9:d6:f7:6e:a0:9c:97:48:93:f6:2c:f8:77:6d:8c:
                    50:5a:fd:3d:38:80:c0:fe:81:55:c3:ce:27:e2:07:
                    fa:52:9f
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            X509v3 Extended Key Usage:
                TLS Web Client Authentication, TLS Web Server Authentication
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            Netscape Cert Type:
                SSL Client, SSL Server
            Netscape Comment:
                Kube API Server Certificate
            X509v3 Subject Alternative Name:
                IP Address:127.0.0.1, IP Address:10.32.0.1, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster, DNS:kubernetes.svc.cluster.local, DNS:server.kubernetes.local, DNS:api-server.kubernetes.local
            X509v3 Subject Key Identifier:
                CA:7C:90:05:B6:3F:7B:38:75:8B:AA:2C:09:B9:17:94:60:F2:2F:0F
            X509v3 Authority Key Identifier:
                9E:CD:36:BA:22:6B:43:A7:DB:23:29:97:53:1B:D1:72:DF:6D:61:D2
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        21:c5:6b:cd:66:86:8b:34:99:a3:ab:04:2a:ba:d4:3b:25:86:
        11:ab:f5:97:57:55:4c:c2:b3:57:f2:5f:06:a7:07:b6:10:ef:
        60:8f:eb:af:48:f5:f7:c0:80:fd:50:7d:98:b6:82:a3:6b:44:
        05:4c:06:59:ec:6e:33:be:0f:58:de:bd:94:d1:54:b9:7d:53:
        09:b3:ca:32:e8:c9:64:e3:f4:1a:ab:bd:94:3b:38:fd:10:49:
        8f:fa:17:aa:25:06:b2:78:48:26:7a:1d:e0:1a:73:4f:61:63:
        92:01:e9:37:84:9e:dd:e5:23:ba:7d:8d:9d:c0:83:75:16:95:
        f4:29:95:aa:f2:27:d4:c1:3a:57:d9:65:66:1a:f2:dd:27:5e:
        d0:6c:47:3e:a7:75:06:7c:96:aa:5e:39:3b:e1:d9:be:0b:a7:
        3a:71:a9:0b:49:4a:d0:ca:41:a2:52:0c:d1:a6:62:20:45:d6:
        b3:29:bc:b9:8c:04:d5:b9:41:af:7a:42:5f:2c:00:cf:80:a4:
        54:a4:a0:e2:fa:52:8d:da:57:ac:76:af:1f:ee:3a:d2:0c:2a:
        9a:bb:ea:a2:bf:10:30:ab:f3:20:b4:e6:cc:c9:81:2b:63:c5:
        8d:90:6d:71:84:10:cb:b0:3b:e0:c5:36:61:0a:fe:8b:f5:c1:
        ed:28:ca:e4:c5:8d:8e:9d:9b:86:d4:81:de:26:a0:fd:e4:d1:
        a1:1c:48:e5:83:4e:d7:a8:bd:6f:d6:f7:19:56:a9:34:3e:ff:
        a7:a8:4c:96:c9:b3:a6:eb:2c:ae:a2:b6:58:b3:0a:7d:0b:4d:
        3f:31:a9:25:41:1c:cd:7c:e0:f2:7e:aa:47:a4:2a:6f:9b:30:
        2a:6e:46:cd:50:05:04:69:89:17:66:cd:7b:4b:01:1a:12:50:
        53:2b:e5:f9:34:49:a3:a3:e0:5a:61:11:34:aa:fa:2d:3b:f3:
        22:b6:3c:32:4a:90:ab:3c:f9:01:2f:43:5a:78:80:93:5e:66:
        4d:ad:7b:a2:09:34:17:3b:ce:0b:94:14:f8:5a:01:ab:98:a7:
        ba:4c:0f:70:ff:3b:3c:01:0f:97:9a:6b:5b:76:9e:8b:b5:10:
        49:3d:ae:6a:40:c8:16:80:af:c4:61:b3:28:f7:32:21:bf:7a:
        36:6f:97:f2:ae:b2:06:6c:75:5e:99:47:f2:a6:6b:09:3b:e3:
        b7:cd:29:3c:89:62:29:d7:83:78:13:4b:29:be:41:58:e9:8e:
        a1:a5:07:c2:12:02:ea:85:5c:6d:13:8f:d5:49:a7:25:83:09:
        ca:fc:04:81:0e:d6:5a:03:db:55:e1:0b:c2:de:13:ca:c1:ae:
        36:e9:cc:a3:3e:80:ee:67
        
        
root@jumpbox:~/kubernetes-the-hard-way# cat units/kube-scheduler.service ; echo
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-scheduler \
  --config=/etc/kubernetes/config/kube-scheduler.yaml \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

root@jumpbox:~/kubernetes-the-hard-way# cat configs/kube-scheduler.yaml ; echo
apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
clientConnection:
  kubeconfig: "/var/lib/kubernetes/kube-scheduler.kubeconfig"
leaderElection:
  leaderElect: true
  
root@jumpbox:~/kubernetes-the-hard-way# cat units/kube-controller-manager.service ; echo
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-controller-manager \
  --bind-address=0.0.0.0 \
  --cluster-cidr=10.200.0.0/16 \
  --cluster-name=kubernetes \
  --cluster-signing-cert-file=/var/lib/kubernetes/ca.crt \
  --cluster-signing-key-file=/var/lib/kubernetes/ca.key \
  --kubeconfig=/var/lib/kubernetes/kube-controller-manager.kubeconfig \
  --root-ca-file=/var/lib/kubernetes/ca.crt \
  --service-account-private-key-file=/var/lib/kubernetes/service-accounts.key \
  --service-cluster-ip-range=10.32.0.0/24 \
  --use-service-account-credentials=true \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target


# jumpbox에 연결하고 쿠버네티스 바이너리 파일과 systemd 유닛 파일을 복사합니다.
root@jumpbox:~/kubernetes-the-hard-way# scp \
  downloads/controller/kube-apiserver \
  downloads/controller/kube-controller-manager \
  downloads/controller/kube-scheduler \
  downloads/client/kubectl \
  units/kube-apiserver.service \
  units/kube-controller-manager.service \
  units/kube-scheduler.service \
  configs/kube-scheduler.yaml \
  configs/kube-apiserver-to-kubelet.yaml \
  root@server:~/
kube-apiserver                                                                                                                                               100%   86MB 120.2MB/s   00:00
kube-controller-manager                                                                                                                                      100%   80MB 107.7MB/s   00:00
kube-scheduler                                                                                                                                               100%   61MB  84.2MB/s   00:00
kubectl                                                                                                                                                      100%   53MB  61.5MB/s   00:00
kube-apiserver.service                                                                                                                                       100% 1442     1.4MB/s   00:00
kube-controller-manager.service                                                                                                                              100%  735     1.0MB/s   00:00
kube-scheduler.service                                                                                                                                       100%  281   408.4KB/s   00:00
kube-scheduler.yaml                                                                                                                                          100%  191   288.9KB/s   00:00
kube-apiserver-to-kubelet.yaml                                                                                                                               100%  727     1.0MB/s   00:00

# 확인
root@jumpbox:~/kubernetes-the-hard-way# ssh server ls -l /root
total 286964
-rw------- 1 root root     9953 Jan 11 07:56 admin.kubeconfig
-rw-r--r-- 1 root root     1899 Jan 11 07:36 ca.crt
-rw------- 1 root root     3272 Jan 11 07:36 ca.key
-rw-r--r-- 1 root root      271 Jan 11 08:04 encryption-config.yaml
-rwxr-xr-x 1 root root 90243224 Jan 11 08:28 kube-apiserver
-rw-r--r-- 1 root root     2354 Jan 11 07:36 kube-api-server.crt
-rw------- 1 root root     3268 Jan 11 07:36 kube-api-server.key
-rw-r--r-- 1 root root     1442 Jan 11 08:28 kube-apiserver.service
-rw-r--r-- 1 root root      727 Jan 11 08:28 kube-apiserver-to-kubelet.yaml
-rwxr-xr-x 1 root root 83427480 Jan 11 08:28 kube-controller-manager
-rw------- 1 root root    10305 Jan 11 07:56 kube-controller-manager.kubeconfig
-rw-r--r-- 1 root root      735 Jan 11 08:28 kube-controller-manager.service
-rwxr-xr-x 1 root root 55836824 Jan 11 08:28 kubectl
-rwxr-xr-x 1 root root 64225432 Jan 11 08:28 kube-scheduler
-rw------- 1 root root    10211 Jan 11 07:56 kube-scheduler.kubeconfig
-rw-r--r-- 1 root root      281 Jan 11 08:28 kube-scheduler.service
-rw-r--r-- 1 root root      191 Jan 11 08:28 kube-scheduler.yaml
-rw-r--r-- 1 root root     2004 Jan 11 07:36 service-accounts.crt
-rw------- 1 root root     3272 Jan 11 07:36 service-accounts.key

쿠버네티스 컨트롤 플레인 프로비저닝을 진행하겠습니다.

root@jumpbox:~/kubernetes-the-hard-way# ssh root@server

# 서버 접근
root@server:~# pwd
/root
root@server:~# mkdir -p /etc/kubernetes/config

# 쿠버네티스 바이너리 설치
root@server:~# mv kube-apiserver \
  kube-controller-manager \
  kube-scheduler kubectl \
  /usr/local/bin/
root@server:~# ls -l /usr/local/bin/kube-*
-rwxr-xr-x 1 root root 90243224 Jan 11 08:28 /usr/local/bin/kube-apiserver
-rwxr-xr-x 1 root root 83427480 Jan 11 08:28 /usr/local/bin/kube-controller-manager
-rwxr-xr-x 1 root root 64225432 Jan 11 08:28 /usr/local/bin/kube-scheduler

# 쿠버네티스 api server 설정
root@server:~# mkdir -p /var/lib/kubernetes/
root@server:~# mv ca.crt ca.key \
  kube-api-server.key kube-api-server.crt \
  service-accounts.key service-accounts.crt \
  encryption-config.yaml \
  /var/lib/kubernetes/
root@server:~# ls -l /var/lib/kubernetes/
total 28
-rw-r--r-- 1 root root 1899 Jan 11 07:36 ca.crt
-rw------- 1 root root 3272 Jan 11 07:36 ca.key
-rw-r--r-- 1 root root  271 Jan 11 08:04 encryption-config.yaml
-rw-r--r-- 1 root root 2354 Jan 11 07:36 kube-api-server.crt
-rw------- 1 root root 3268 Jan 11 07:36 kube-api-server.key
-rw-r--r-- 1 root root 2004 Jan 11 07:36 service-accounts.crt
-rw------- 1 root root 3272 Jan 11 07:36 service-accounts.key

# kube-apiserver.service systemd 유닛 파일 생서
root@server:~# mv kube-apiserver.service \
  /etc/systemd/system/kube-apiserver.service
root@server:~# tree /etc/systemd/system
/etc/systemd/system
├── apt-daily.service -> /dev/null
├── apt-daily-upgrade.service -> /dev/null
├── dbus-org.freedesktop.timesync1.service -> /lib/systemd/system/systemd-timesyncd.service
├── etcd.service
├── getty.target.wants
│   └── getty@tty1.service -> /lib/systemd/system/getty@.service
├── kube-apiserver.service
├── multi-user.target.wants
│   ├── console-setup.service -> /lib/systemd/system/console-setup.service
│   ├── cron.service -> /lib/systemd/system/cron.service
│   ├── e2scrub_reap.service -> /lib/systemd/system/e2scrub_reap.service
│   ├── etcd.service -> /etc/systemd/system/etcd.service
│   ├── networking.service -> /lib/systemd/system/networking.service
│   ├── nfs-client.target -> /lib/systemd/system/nfs-client.target
│   ├── remote-fs.target -> /lib/systemd/system/remote-fs.target
│   ├── rpcbind.service -> /lib/systemd/system/rpcbind.service
│   ├── ssh.service -> /lib/systemd/system/ssh.service
│   ├── vboxadd.service -> /lib/systemd/system/vboxadd.service
│   └── vboxadd-service.service -> /lib/systemd/system/vboxadd-service.service
├── network-online.target.wants
│   └── networking.service -> /lib/systemd/system/networking.service
├── paths.target.wants
│   └── acpid.path -> /lib/systemd/system/acpid.path
├── remote-fs.target.wants
│   └── nfs-client.target -> /lib/systemd/system/nfs-client.target
├── sockets.target.wants
│   ├── acpid.socket -> /lib/systemd/system/acpid.socket
│   ├── dm-event.socket -> /lib/systemd/system/dm-event.socket
│   └── rpcbind.socket -> /lib/systemd/system/rpcbind.socket
├── sshd.service -> /lib/systemd/system/ssh.service
├── sysinit.target.wants
│   ├── blk-availability.service -> /lib/systemd/system/blk-availability.service
│   ├── keyboard-setup.service -> /lib/systemd/system/keyboard-setup.service
│   ├── lvm2-lvmpolld.socket -> /lib/systemd/system/lvm2-lvmpolld.socket
│   ├── lvm2-monitor.service -> /lib/systemd/system/lvm2-monitor.service
│   ├── systemd-pstore.service -> /lib/systemd/system/systemd-pstore.service
│   └── systemd-timesyncd.service -> /lib/systemd/system/systemd-timesyncd.service
└── timers.target.wants
    ├── dpkg-db-backup.timer -> /lib/systemd/system/dpkg-db-backup.timer
    ├── e2scrub_all.timer -> /lib/systemd/system/e2scrub_all.timer
    ├── fstrim.timer -> /lib/systemd/system/fstrim.timer
    ├── logrotate.timer -> /lib/systemd/system/logrotate.timer
    └── man-db.timer -> /lib/systemd/system/man-db.timer

9 directories, 35 files


# 쿠버네티스 컨트롤러 매니저 구성
root@server:~# mv kube-controller-manager.kubeconfig /var/lib/kubernetes/
root@server:~# mv kube-controller-manager.service /etc/systemd/system/
root@server:~# mv kube-scheduler.kubeconfig /var/lib/kubernetes/
root@server:~# mv kube-scheduler.yaml /etc/kubernetes/config/
root@server:~# mv kube-scheduler.service /etc/systemd/system/

# 컨트롤러 서비스 시작
root@server:~# systemctl daemon-reload
root@server:~# systemctl enable kube-apiserver kube-controller-manager kube-scheduler
Created symlink /etc/systemd/system/multi-user.target.wants/kube-apiserver.service → /etc/systemd/system/kube-apiserver.service.
Created symlink /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service → /etc/systemd/system/kube-controller-manager.service.
Created symlink /etc/systemd/system/multi-user.target.wants/kube-scheduler.service → /etc/systemd/system/kube-scheduler.service.
root@server:~# systemctl start  kube-apiserver kube-controller-manager kube-scheduler

root@server:~# ss -tlp | grep kube
LISTEN 0      4096               *:6443              *:*    users:(("kube-apiserver",pid=2803,fd=3))
LISTEN 0      4096               *:10257             *:*    users:(("kube-controller",pid=2804,fd=3))
LISTEN 0      4096               *:10259             *:*    users:(("kube-scheduler",pid=2809,fd=3))

root@server:~# systemctl is-active kube-apiserver
active
root@server:~# systemctl status kube-apiserver --no-pager
● kube-apiserver.service - Kubernetes API Server
     Loaded: loaded (/etc/systemd/system/kube-apiserver.service; enabled; preset: enabled)
     Active: active (running) since Sun 2026-01-11 08:38:14 KST; 1min 9s ago
       Docs: https://github.com/kubernetes/kubernetes
   Main PID: 2803 (kube-apiserver)
      Tasks: 9 (limit: 2096)
     Memory: 210.2M
        CPU: 3.860s
     CGroup: /system.slice/kube-apiserver.service
             └─2803 /usr/local/bin/kube-apiserver --allow-privileged=true --apiserver-count=1 --audit-log-maxage=30 --audit-log-maxbackup=3 --audit-log-maxsize=100 --audit-log-path=/var/log/…

Jan 11 08:38:17 server kube-apiserver[2803]: I0111 08:38:17.446400    2803 storage_rbac.go:321] created rolebinding.rbac.authorization.k8s.io/system:controller:cloud-provider in kube-system
Jan 11 08:38:17 server kube-apiserver[2803]: I0111 08:38:17.448425    2803 storage_rbac.go:321] created rolebinding.rbac.authorization.k8s.io/system:controller:token-cleaner in kube-system
Jan 11 08:38:17 server kube-apiserver[2803]: I0111 08:38:17.450674    2803 storage_rbac.go:321] created rolebinding.rbac.authorization.k8s.io/system:controller:bootstrap-signer in kube-public
Jan 11 08:38:17 server kube-apiserver[2803]: I0111 08:38:17.504016    2803 alloc.go:330] "allocated clusterIPs" service="default/kubernetes" clusterIPs={"IPv4":"10.32.0.1"}
Jan 11 08:38:17 server kube-apiserver[2803]: W0111 08:38:17.506560    2803 lease.go:265] Resetting endpoints for master service "kubernetes" to [10.0.2.15]
Jan 11 08:38:17 server kube-apiserver[2803]: I0111 08:38:17.507060    2803 controller.go:615] quota admission added evaluator for: endpoints
Jan 11 08:38:17 server kube-apiserver[2803]: I0111 08:38:17.510095    2803 controller.go:615] quota admission added evaluator for: endpointslices.discovery.k8s.io
Jan 11 08:38:20 server kube-apiserver[2803]: I0111 08:38:20.160901    2803 controller.go:615] quota admission added evaluator for: serviceaccounts
Jan 11 08:38:26 server kube-apiserver[2803]: I0111 08:38:26.306880    2803 apf_controller.go:493] "Update CurrentCL" plName="exempt" seatDemandHighWatermark=2 seatDemandAvg=0.020429668431191…
Jan 11 08:38:36 server kube-apiserver[2803]: I0111 08:38:36.307942    2803 apf_controller.go:493] "Update CurrentCL" plName="exempt" seatDemandHighWatermark=1 seatDemandAvg=0.000540973546383…
Hint: Some lines were ellipsized, use -l to show in full.

root@server:~# systemctl status kube-scheduler --no-pager
● kube-scheduler.service - Kubernetes Scheduler
     Loaded: loaded (/etc/systemd/system/kube-scheduler.service; enabled; preset: enabled)
     Active: active (running) since Sun 2026-01-11 08:38:14 KST; 1min 32s ago
       Docs: https://github.com/kubernetes/kubernetes
   Main PID: 2809 (kube-scheduler)
      Tasks: 8 (limit: 2096)
     Memory: 15.3M
        CPU: 1.321s
     CGroup: /system.slice/kube-scheduler.service
             └─2809 /usr/local/bin/kube-scheduler --config=/etc/kubernetes/config/kube-scheduler.yaml --v=2

Jan 11 08:38:17 server kube-scheduler[2809]: I0111 08:38:17.795270    2809 reflector.go:376] Caches populated for *v1.ReplicationController from k8s.io/client-go/informers/factory.go:160
Jan 11 08:38:17 server kube-scheduler[2809]: I0111 08:38:17.816831    2809 reflector.go:376] Caches populated for *v1.CSIDriver from k8s.io/client-go/informers/factory.go:160
Jan 11 08:38:17 server kube-scheduler[2809]: I0111 08:38:17.889509    2809 reflector.go:376] Caches populated for *v1.StorageClass from k8s.io/client-go/informers/factory.go:160
Jan 11 08:38:17 server kube-scheduler[2809]: I0111 08:38:17.976799    2809 reflector.go:376] Caches populated for *v1.ReplicaSet from k8s.io/client-go/informers/factory.go:160
Jan 11 08:38:18 server kube-scheduler[2809]: I0111 08:38:18.053109    2809 reflector.go:376] Caches populated for *v1.Pod from k8s.io/client-go/informers/factory.go:160
Jan 11 08:38:18 server kube-scheduler[2809]: I0111 08:38:18.104208    2809 reflector.go:376] Caches populated for *v1.Namespace from k8s.io/client-go/informers/factory.go:160
Jan 11 08:38:18 server kube-scheduler[2809]: I0111 08:38:18.104322    2809 reflector.go:376] Caches populated for *v1.PersistentVolumeClaim from k8s.io/client-go/informers/factory.go:160
Jan 11 08:38:20 server kube-scheduler[2809]: I0111 08:38:20.518170    2809 reflector.go:376] Caches populated for *v1.PersistentVolume from k8s.io/client-go/informers/factory.go:160
Jan 11 08:38:20 server kube-scheduler[2809]: I0111 08:38:20.543776    2809 leaderelection.go:257] attempting to acquire leader lease kube-system/kube-scheduler...
Jan 11 08:38:20 server kube-scheduler[2809]: I0111 08:38:20.547972    2809 leaderelection.go:271] successfully acquired lease kube-system/kube-scheduler

root@server:~# systemctl status kube-controller-manager --no-pager
● kube-controller-manager.service - Kubernetes Controller Manager
     Loaded: loaded (/etc/systemd/system/kube-controller-manager.service; enabled; preset: enabled)
     Active: active (running) since Sun 2026-01-11 08:38:14 KST; 1min 38s ago
       Docs: https://github.com/kubernetes/kubernetes
   Main PID: 2804 (kube-controller)
      Tasks: 5 (limit: 2096)
     Memory: 40.2M
        CPU: 2.314s
     CGroup: /system.slice/kube-controller-manager.service
             └─2804 /usr/local/bin/kube-controller-manager --bind-address=0.0.0.0 --cluster-cidr=10.200.0.0/16 --cluster-name=kubernetes --cluster-signing-cert-file=/var/lib/kubernetes/ca.cr…

Jan 11 08:38:24 server kube-controller-manager[2804]: I0111 08:38:24.351097    2804 shared_informer.go:320] Caches are synced for certificate-csrsigning-kubelet-client
Jan 11 08:38:24 server kube-controller-manager[2804]: I0111 08:38:24.351083    2804 shared_informer.go:320] Caches are synced for certificate-csrsigning-legacy-unknown
Jan 11 08:38:24 server kube-controller-manager[2804]: I0111 08:38:24.351111    2804 shared_informer.go:320] Caches are synced for disruption
Jan 11 08:38:24 server kube-controller-manager[2804]: I0111 08:38:24.352466    2804 shared_informer.go:320] Caches are synced for ReplicaSet
Jan 11 08:38:24 server kube-controller-manager[2804]: I0111 08:38:24.352467    2804 shared_informer.go:320] Caches are synced for endpoint_slice
Jan 11 08:38:24 server kube-controller-manager[2804]: I0111 08:38:24.352487    2804 endpointslice_controller.go:288] "Starting service queue worker threads" logger="endpointsli…oller" total=5
Jan 11 08:38:24 server kube-controller-manager[2804]: I0111 08:38:24.352505    2804 endpointslice_controller.go:292] "Starting topology queue worker threads" logger="endpointsl…oller" total=1
Jan 11 08:38:24 server kube-controller-manager[2804]: I0111 08:38:24.352540    2804 shared_informer.go:320] Caches are synced for stateful set
Jan 11 08:38:24 server kube-controller-manager[2804]: I0111 08:38:24.353634    2804 shared_informer.go:320] Caches are synced for legacy-service-account-token-cleaner
Jan 11 08:38:24 server kube-controller-manager[2804]: I0111 08:38:24.353714    2804 shared_informer.go:320] Caches are synced for PVC protection
Hint: Some lines were ellipsized, use -l to show in full.

root@server:~# kubectl cluster-info --kubeconfig admin.kubeconfig
Kubernetes control plane is running at https://127.0.0.1:6443

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.


# kubectl 동작확인
root@server:~# kubectl get node --kubeconfig admin.kubeconfig
No resources found
root@server:~# kubectl get pod -A --kubeconfig admin.kubeconfig
No resources found
root@server:~# kubectl get service,ep --kubeconfig admin.kubeconfig
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.32.0.1    <none>        443/TCP   2m58s

NAME                   ENDPOINTS        AGE
endpoints/kubernetes   10.0.2.15:6443   2m58s
root@server:~# kubectl get clusterroles --kubeconfig admin.kubeconfig
NAME                                                                   CREATED AT
admin                                                                  2026-01-10T23:38:17Z
cluster-admin                                                          2026-01-10T23:38:17Z
edit                                                                   2026-01-10T23:38:17Z
system:aggregate-to-admin                                              2026-01-10T23:38:17Z
system:aggregate-to-edit                                               2026-01-10T23:38:17Z
system:aggregate-to-view                                               2026-01-10T23:38:17Z
system:auth-delegator                                                  2026-01-10T23:38:17Z
system:basic-user                                                      2026-01-10T23:38:17Z
...

API Server가 워커 노드의 Kubelet API를 안전하게 호출할 수 있도록 RBAC 기반 접근 제어를 설정합니다. 이를 통해 메트릭 수집, 로그 조회, 포드 명령 실행과 같은 운영 기능을 API Server를 통해 수행할 수 있습니다.

Kubelet은 Webhook 권한 부여 모드를 사용하며, 요청에 대한 인가 판단은 SubjectAccessReview API를 통해 중앙에서 처리됩니다.

root@server:~# cat kube-apiserver-to-kubelet.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-apiserver-to-kubelet
rules:
  - apiGroups:
      - ""
    resources:
      - nodes/proxy
      - nodes/stats
      - nodes/log
      - nodes/spec
      - nodes/metrics
    verbs:
      - "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:kube-apiserver
  namespace: ""
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-apiserver-to-kubelet
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: kubernetesroot@server:~#

root@server:~# kubectl apply -f kube-apiserver-to-kubelet.yaml --kubeconfig admin.kubeconfig
clusterrole.rbac.authorization.k8s.io/system:kube-apiserver-to-kubelet created
clusterrolebinding.rbac.authorization.k8s.io/system:kube-apiserver created

점프 서버에서 Kubernetes 컨트롤 플레인 정상 동작 확인하겠습니다.

root@jumpbox:~/kubernetes-the-hard-way# curl -s -k --cacert ca.crt https://server.kubernetes.local:6443/version | jq
{
  "major": "1",
  "minor": "32",
  "gitVersion": "v1.32.3",
  "gitCommit": "32cc146f75aad04beaaa245a7157eb35063a9f99",
  "gitTreeState": "clean",
  "buildDate": "2025-03-11T19:52:21Z",
  "goVersion": "go1.23.6",
  "compiler": "gc",
  "platform": "linux/arm64"
}

워커 노드 부트스트래핑

node-0, node-1 노드에 runc, container networking plugins, containerd, kubelet, kube-proxy 설치하겠습니다.

root@jumpbox:~/kubernetes-the-hard-way# cat configs/10-bridge.conf | jq
{
  "cniVersion": "1.0.0",
  "name": "bridge",
  "type": "bridge",
  "bridge": "cni0",
  "isGateway": true,
  "ipMasq": true,
  "ipam": {
    "type": "host-local",
    "ranges": [
      [
        {
          "subnet": "SUBNET"
        }
      ]
    ],
    "routes": [
      {
        "dst": "0.0.0.0/0"
      }
    ]
  }
}

root@jumpbox:~/kubernetes-the-hard-way# cat configs/kubelet-config.yaml | yq # clusterDomain , clusterDNS 없어도 smoke test 까지 잘됨 -> 실습에서 coredns 미사용
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: "0.0.0.0"                            # kubelet HTTPS 서버 바인딩 주소 : 모든 인터페이스에서 10250 포트 수신
authentication:
  anonymous:
    enabled: false                            # 익명 인증 비활성화
  webhook:
    enabled: true                             # 인증 요청을 kube-apiserver에 위임 : ServiceAccount 토큰, bootstrap 토큰 처리 가능
  x509:                                       # kubelet에 접근하는 클라이언트 인증서 검증용 CA
    clientCAFile: "/var/lib/kubelet/ca.crt"   # (상동) 대상 : kube-apiserver, metrics-server, kubectl (직접 접근 시)
authorization:                                
  mode: Webhook                               # 인가 요청을 kube-apiserver에 위임 : Node Authorizer + RBAC 적용됨
cgroupDriver: systemd
containerRuntimeEndpoint: "unix:///var/run/containerd/containerd.sock"  # CRI 엔드포인트
enableServer: true                            # kubelet API 서버 활성화 , false면 apiserver가 kubelet 접근 불가
failSwapOn: false
maxPods: 16                                   # 노드당 최대 파드 수 16개
memorySwap:
  swapBehavior: NoSwap
port: 10250                                   # kubelet HTTPS API 포트 : 로그, exec, stats, metrics 접근에 사용
resolvConf: "/etc/resolv.conf"                # 파드에 전달할 DNS 설정 파일
registerNode: true                            # kubelet이 API 서버에 Node 객체 자동 등록
runtimeRequestTimeout: "15m"                  # CRI 요청 최대 대기 시간 : 이미지 pull, container start 등
tlsCertFile: "/var/lib/kubelet/kubelet.crt"   # TLS 서버 인증서 (kubelet 자신) : kubelet HTTPS 서버의 서버 인증서
tlsPrivateKeyFile: "/var/lib/kubelet/kubelet.key"


root@jumpbox:~/kubernetes-the-hard-way# for HOST in node-0 node-1; do
  SUBNET=$(grep ${HOST} machines.txt | cut -d " " -f 4)
  sed "s|SUBNET|$SUBNET|g" \
    configs/10-bridge.conf > 10-bridge.conf

  sed "s|SUBNET|$SUBNET|g" \
    configs/kubelet-config.yaml > kubelet-config.yaml

  scp 10-bridge.conf kubelet-config.yaml \
  root@${HOST}:~/
done
10-bridge.conf                                                                                                                                               100%  265   413.0KB/s   00:00
kubelet-config.yaml                                                                                                                                          100%  610     1.1MB/s   00:00
10-bridge.conf                                                                                                                                               100%  265   418.8KB/s   00:00
kubelet-config.yaml                                                                                                                                          100%  610     1.1MB/s   00:00

root@jumpbox:~/kubernetes-the-hard-way# ssh node-0 ls -l /root
total 8
-rw-r--r-- 1 root root 265 Jan 11 08:46 10-bridge.conf
-rw-r--r-- 1 root root 610 Jan 11 08:46 kubelet-config.yaml
root@jumpbox:~/kubernetes-the-hard-way# ssh node-1 ls -l /root
total 8
-rw-r--r-- 1 root root 265 Jan 11 08:46 10-bridge.conf
-rw-r--r-- 1 root root 610 Jan 11 08:46 kubelet-config.yaml

# 파일 확인 및 node-0/1에 전달
root@jumpbox:~/kubernetes-the-hard-way# cat configs/99-loopback.conf ; echo
{
  "cniVersion": "1.1.0",
  "name": "lo",
  "type": "loopback"
}
root@jumpbox:~/kubernetes-the-hard-way# cat configs/containerd-config.toml ; echo
version = 2

[plugins."io.containerd.grpc.v1.cri"]
  [plugins."io.containerd.grpc.v1.cri".containerd]
    snapshotter = "overlayfs"
    default_runtime_name = "runc"
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
    runtime_type = "io.containerd.runc.v2"
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true
[plugins."io.containerd.grpc.v1.cri".cni]
  bin_dir = "/opt/cni/bin"
  conf_dir = "/etc/cni/net.d"
root@jumpbox:~/kubernetes-the-hard-way# cat configs/kube-proxy-config.yaml ; echo
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
clientConnection:
  kubeconfig: "/var/lib/kube-proxy/kubeconfig"
mode: "iptables"
clusterCIDR: "10.200.0.0/16"

root@jumpbox:~/kubernetes-the-hard-way# cat units/containerd.service
[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target

[Service]
ExecStartPre=/sbin/modprobe overlay
ExecStart=/bin/containerd
Restart=always
RestartSec=5
Delegate=yes
KillMode=process
OOMScoreAdjust=-999
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity

[Install]
WantedBy=multi-user.target
root@jumpbox:~/kubernetes-the-hard-way# cat units/kubelet.service
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=containerd.service
Requires=containerd.service

[Service]
ExecStart=/usr/local/bin/kubelet \
  --config=/var/lib/kubelet/kubelet-config.yaml \
  --kubeconfig=/var/lib/kubelet/kubeconfig \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
root@jumpbox:~/kubernetes-the-hard-way# cat units/kube-proxy.service
[Unit]
Description=Kubernetes Kube Proxy
Documentation=https://github.com/kubernetes/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-proxy \
  --config=/var/lib/kube-proxy/kube-proxy-config.yaml
Restart=on-failure
RestartSec=5

[Install]


root@jumpbox:~/kubernetes-the-hard-way# for HOST in node-0 node-1; do
  scp \
    downloads/worker/* \
    downloads/client/kubectl \
    configs/99-loopback.conf \
    configs/containerd-config.toml \
    configs/kube-proxy-config.yaml \
    units/containerd.service \
    units/kubelet.service \
    units/kube-proxy.service \
    root@${HOST}:~/
done
containerd                                                                                                                                                   100%   54MB 109.8MB/s   00:00
containerd-shim-runc-v2                                                                                                                                      100% 7808KB 115.1MB/s   00:00
containerd-stress                                                                                                                                            100%   21MB 134.4MB/s   00:00
crictl                                                                                                                                                       100%   37MB 126.0MB/s   00:00
ctr                                                                                                                                                          100%   22MB 128.0MB/s   00:00
kubelet                                                                                                                                                      100%   72MB 114.6MB/s   00:00
kube-proxy                                                                                                                                                   100%   62MB 107.7MB/s   00:00
runc                                                                                                                                                         100%   11MB 116.3MB/s   00:00
kubectl                                                                                                                                                      100%   53MB  99.3MB/s   00:00
99-loopback.conf                                                                                                                                             100%   65   144.9KB/s   00:00
containerd-config.toml                                                                                                                                       100%  470     1.1MB/s   00:00
kube-proxy-config.yaml                                                                                                                                       100%  184   431.0KB/s   00:00
containerd.service                                                                                                                                           100%  352   739.2KB/s   00:00
kubelet.service                                                                                                                                              100%  365   781.0KB/s   00:00
kube-proxy.service                                                                                                                                           100%  268   498.2KB/s   00:00
containerd                                                                                                                                                   100%   54MB 123.9MB/s   00:00
containerd-shim-runc-v2                                                                                                                                      100% 7808KB 123.9MB/s   00:00
containerd-stress                                                                                                                                            100%   21MB 128.7MB/s   00:00
crictl                                                                                                                                                       100%   37MB 129.9MB/s   00:00
ctr                                                                                                                                                          100%   22MB 127.6MB/s   00:00
kubelet                                                                                                                                                      100%   72MB 121.1MB/s   00:00
kube-proxy                                                                                                                                                   100%   62MB 103.1MB/s   00:00
runc                                                                                                                                                         100%   11MB 132.1MB/s   00:00
kubectl                                                                                                                                                      100%   53MB 114.7MB/s   00:00
99-loopback.conf                                                                                                                                             100%   65   116.5KB/s   00:00
containerd-config.toml                                                                                                                                       100%  470   715.5KB/s   00:00
kube-proxy-config.yaml                                                                                                                                       100%  184   204.4KB/s   00:00
containerd.service                                                                                                                                           100%  352   716.2KB/s   00:00
kubelet.service                                                                                                                                              100%  365   684.7KB/s   00:00
kube-proxy.service                                                                                                                                           100%  268   366.2KB/s   00:00

root@jumpbox:~/kubernetes-the-hard-way# for HOST in node-0 node-1; do
  scp \
    downloads/cni-plugins/* \
    root@${HOST}:~/cni-plugins/
done
bandwidth                                                                                                                                                    100% 4492KB  52.1MB/s   00:00
bridge                                                                                                                                                       100% 5065KB 107.0MB/s   00:00
dhcp                                                                                                                                                         100%   12MB 102.8MB/s   00:00
dummy                                                                                                                                                        100% 4623KB 108.5MB/s   00:00
firewall                                                                                                                                                     100% 5088KB 144.1MB/s   00:00
host-device                                                                                                                                                  100% 4611KB 119.7MB/s   00:00
host-local                                                                                                                                                   100% 3886KB 113.2MB/s   00:00
ipvlan                                                                                                                                                       100% 4628KB 114.8MB/s   00:00
LICENSE                                                                                                                                                      100%   11KB  14.6MB/s   00:00
loopback                                                                                                                                                     100% 4030KB 126.9MB/s   00:00
macvlan                                                                                                                                                      100% 4764KB 110.9MB/s   00:00
portmap                                                                                                                                                      100% 4513KB  88.0MB/s   00:00
ptp                                                                                                                                                          100% 4877KB 113.8MB/s   00:00
README.md                                                                                                                                                    100% 2343     3.3MB/s   00:00
sbr                                                                                                                                                          100% 4216KB 113.4MB/s   00:00
static                                                                                                                                                       100% 3498KB 110.7MB/s   00:00
tap                                                                                                                                                          100% 4775KB 122.1MB/s   00:00
tuning                                                                                                                                                       100% 4054KB 108.8MB/s   00:00
vlan                                                                                                                                                         100% 4627KB  80.0MB/s   00:00
vrf                                                                                                                                                          100% 4325KB 116.8MB/s   00:00
bandwidth                                                                                                                                                    100% 4492KB  64.0MB/s   00:00
bridge                                                                                                                                                       100% 5065KB 140.7MB/s   00:00
dhcp                                                                                                                                                         100%   12MB 128.4MB/s   00:00
dummy                                                                                                                                                        100% 4623KB 134.0MB/s   00:00
firewall                                                                                                                                                     100% 5088KB 134.2MB/s   00:00
host-device                                                                                                                                                  100% 4611KB 130.3MB/s   00:00
host-local                                                                                                                                                   100% 3886KB 137.3MB/s   00:00
ipvlan                                                                                                                                                       100% 4628KB 135.9MB/s   00:00
LICENSE                                                                                                                                                      100%   11KB  18.1MB/s   00:00
loopback                                                                                                                                                     100% 4030KB 113.2MB/s   00:00
macvlan                                                                                                                                                      100% 4764KB 124.8MB/s   00:00
portmap                                                                                                                                                      100% 4513KB 124.5MB/s   00:00
ptp                                                                                                                                                          100% 4877KB 129.0MB/s   00:00
README.md                                                                                                                                                    100% 2343     4.5MB/s   00:00
sbr                                                                                                                                                          100% 4216KB 127.5MB/s   00:00
static                                                                                                                                                       100% 3498KB 135.4MB/s   00:00
tap                                                                                                                                                          100% 4775KB 134.7MB/s   00:00
tuning                                                                                                                                                       100% 4054KB 137.7MB/s   00:00
vlan                                                                                                                                                         100% 4627KB 140.3MB/s   00:00
vrf                                                                                                                                                          100% 4325KB 136.6MB/s   00:00


root@jumpbox:~/kubernetes-the-hard-way# ssh node-0 ls -l /root
total 347860
-rw-r--r-- 1 root root      265 Jan 11 08:46 10-bridge.conf
-rw-r--r-- 1 root root       65 Jan 11 08:51 99-loopback.conf
drwxr-xr-x 2 root root     4096 Jan 11 08:51 cni-plugins
-rwxr-xr-x 1 root root 56836190 Jan 11 08:51 containerd
-rw-r--r-- 1 root root      470 Jan 11 08:51 containerd-config.toml
-rw-r--r-- 1 root root      352 Jan 11 08:51 containerd.service
-rwxr-xr-x 1 root root  7995576 Jan 11 08:51 containerd-shim-runc-v2
-rwxr-xr-x 1 root root 22020449 Jan 11 08:51 containerd-stress
-rwxr-xr-x 1 root root 38808389 Jan 11 08:51 crictl
-rwxr-xr-x 1 root root 22806881 Jan 11 08:51 ctr
-rwxr-xr-x 1 root root 55836824 Jan 11 08:51 kubectl
-rwxr-xr-x 1 root root 75235588 Jan 11 08:51 kubelet
-rw-r--r-- 1 root root      610 Jan 11 08:46 kubelet-config.yaml
-rw-r--r-- 1 root root      365 Jan 11 08:51 kubelet.service
-rwxr-xr-x 1 root root 65274008 Jan 11 08:51 kube-proxy
-rw-r--r-- 1 root root      184 Jan 11 08:51 kube-proxy-config.yaml
-rw-r--r-- 1 root root      268 Jan 11 08:51 kube-proxy.service
-rwxr-xr-x 1 root root 11305168 Jan 11 08:51 runc
root@jumpbox:~/kubernetes-the-hard-way# ssh node-1 ls -l /root
total 347860
-rw-r--r-- 1 root root      265 Jan 11 08:46 10-bridge.conf
-rw-r--r-- 1 root root       65 Jan 11 08:51 99-loopback.conf
drwxr-xr-x 2 root root     4096 Jan 11 08:51 cni-plugins
-rwxr-xr-x 1 root root 56836190 Jan 11 08:51 containerd
-rw-r--r-- 1 root root      470 Jan 11 08:51 containerd-config.toml
-rw-r--r-- 1 root root      352 Jan 11 08:51 containerd.service
-rwxr-xr-x 1 root root  7995576 Jan 11 08:51 containerd-shim-runc-v2
-rwxr-xr-x 1 root root 22020449 Jan 11 08:51 containerd-stress
-rwxr-xr-x 1 root root 38808389 Jan 11 08:51 crictl
-rwxr-xr-x 1 root root 22806881 Jan 11 08:51 ctr
-rwxr-xr-x 1 root root 55836824 Jan 11 08:51 kubectl
-rwxr-xr-x 1 root root 75235588 Jan 11 08:51 kubelet
-rw-r--r-- 1 root root      610 Jan 11 08:46 kubelet-config.yaml
-rw-r--r-- 1 root root      365 Jan 11 08:51 kubelet.service
-rwxr-xr-x 1 root root 65274008 Jan 11 08:51 kube-proxy
-rw-r--r-- 1 root root      184 Jan 11 08:51 kube-proxy-config.yaml
-rw-r--r-- 1 root root      268 Jan 11 08:51 kube-proxy.service
-rwxr-xr-x 1 root root 11305168 Jan 11 08:51 runc
root@jumpbox:~/kubernetes-the-hard-way# ssh node-0 ls -l /root/cni-plugins
total 88164
-rwxr-xr-x 1 root root  4600029 Jan 11 08:51 bandwidth
-rwxr-xr-x 1 root root  5186762 Jan 11 08:51 bridge
-rwxr-xr-x 1 root root 12321657 Jan 11 08:51 dhcp
-rwxr-xr-x 1 root root  4734102 Jan 11 08:51 dummy
-rwxr-xr-x 1 root root  5210586 Jan 11 08:51 firewall
-rwxr-xr-x 1 root root  4721690 Jan 11 08:51 host-device
-rwxr-xr-x 1 root root  3979580 Jan 11 08:51 host-local
-rwxr-xr-x 1 root root  4738895 Jan 11 08:51 ipvlan
-rwxr-xr-x 1 root root    11357 Jan 11 08:51 LICENSE
-rwxr-xr-x 1 root root  4127141 Jan 11 08:51 loopback
-rwxr-xr-x 1 root root  4878636 Jan 11 08:51 macvlan
-rwxr-xr-x 1 root root  4621227 Jan 11 08:51 portmap
-rwxr-xr-x 1 root root  4994381 Jan 11 08:51 ptp
-rwxr-xr-x 1 root root     2343 Jan 11 08:51 README.md
-rwxr-xr-x 1 root root  4317592 Jan 11 08:51 sbr
-rwxr-xr-x 1 root root  3582221 Jan 11 08:51 static
-rwxr-xr-x 1 root root  4889353 Jan 11 08:51 tap
-rwxr-xr-x 1 root root  4150810 Jan 11 08:51 tuning
-rwxr-xr-x 1 root root  4738028 Jan 11 08:51 vlan
-rwxr-xr-x 1 root root  4428332 Jan 11 08:51 vrf
root@jumpbox:~/kubernetes-the-hard-way# ssh node-1 ls -l /root/cni-plugins
total 88164
-rwxr-xr-x 1 root root  4600029 Jan 11 08:51 bandwidth
-rwxr-xr-x 1 root root  5186762 Jan 11 08:51 bridge
-rwxr-xr-x 1 root root 12321657 Jan 11 08:51 dhcp
-rwxr-xr-x 1 root root  4734102 Jan 11 08:51 dummy
-rwxr-xr-x 1 root root  5210586 Jan 11 08:51 firewall
-rwxr-xr-x 1 root root  4721690 Jan 11 08:51 host-device
-rwxr-xr-x 1 root root  3979580 Jan 11 08:51 host-local
-rwxr-xr-x 1 root root  4738895 Jan 11 08:51 ipvlan
-rwxr-xr-x 1 root root    11357 Jan 11 08:51 LICENSE
-rwxr-xr-x 1 root root  4127141 Jan 11 08:51 loopback
-rwxr-xr-x 1 root root  4878636 Jan 11 08:51 macvlan
-rwxr-xr-x 1 root root  4621227 Jan 11 08:51 portmap
-rwxr-xr-x 1 root root  4994381 Jan 11 08:51 ptp
-rwxr-xr-x 1 root root     2343 Jan 11 08:51 README.md
-rwxr-xr-x 1 root root  4317592 Jan 11 08:51 sbr
-rwxr-xr-x 1 root root  3582221 Jan 11 08:51 static
-rwxr-xr-x 1 root root  4889353 Jan 11 08:51 tap
-rwxr-xr-x 1 root root  4150810 Jan 11 08:51 tuning
-rwxr-xr-x 1 root root  4738028 Jan 11 08:51 vlan
-rwxr-xr-x 1 root root  4428332 Jan 11 08:51 vrf

워커노드 프로비저닝을 위해 node-0, node-1에 접속하여 실행하겠습니다.

# kubelet ↔ containerd 연결 Flow
kubelet
  ↓ CRI (gRPC)
unix:///var/run/containerd/containerd.sock
  ↓
containerd CRI plugin
  ↓
runc
  ↓
Linux namespaces / cgroups

# Node 0 접속 후 실행 (Node 1도 동일)
root@jumpbox:~/kubernetes-the-hard-way# ssh root@node-0
root@node-0:~# apt-get -y install socat conntrack ipset kmod psmisc bridge-utils
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
kmod is already the newest version (30+20221128-1).
The following additional packages will be installed:
  iptables libip6tc2 libipset13 libnetfilter-conntrack3 libnfnetlink0
  ...
  
# Disable Swap : Verify if swap is disabled:
root@node-0:~# swapon --show

# Create the installation directories
root@node-0:~# mkdir -p \
  /etc/cni/net.d \
  /opt/cni/bin \
  /var/lib/kubelet \
  /var/lib/kube-proxy \
  /var/lib/kubernetes \
  /var/run/kubernetes

# Install the worker binaries:
root@node-0:~# mv crictl kube-proxy kubelet runc /usr/local/bin/
root@node-0:~# mv containerd containerd-shim-runc-v2 containerd-stress /bin/
root@node-0:~# mv cni-plugins/* /opt/cni/bin/

# Configure CNI Networking

# Create the bridge network configuration file:
root@node-0:~# mv 10-bridge.conf 99-loopback.conf /etc/cni/net.d/
root@node-0:~# cat /etc/cni/net.d/10-bridge.conf
{
  "cniVersion": "1.0.0",
  "name": "bridge",
  "type": "bridge",
  "bridge": "cni0",
  "isGateway": true,
  "ipMasq": true,
  "ipam": {
    "type": "host-local",
    "ranges": [
      [{"subnet": "10.200.0.0/24"}]
    ],
    "routes": [{"dst": "0.0.0.0/0"}]
  }
}

# To ensure network traffic crossing the CNI bridge network is processed by iptables, load and configure the br-netfilter kernel module:
root@node-0:~# lsmod | grep netfilter
root@node-0:~# modprobe br-netfilter
root@node-0:~# echo "br-netfilter" >> /etc/modules-load.d/modules.conf
root@node-0:~# lsmod | grep netfilter
br_netfilter           32768  0
bridge                262144  1 br_netfilter

root@node-0:~# echo "net.bridge.bridge-nf-call-iptables = 1"  >> /etc/sysctl.d/kubernetes.conf
root@node-0:~# echo "net.bridge.bridge-nf-call-ip6tables = 1" >> /etc/sysctl.d/kubernetes.conf
root@node-0:~# sysctl -p /etc/sysctl.d/kubernetes.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1


# Configure containerd : Install the containerd configuration files:
root@node-0:~# mkdir -p /etc/containerd/
root@node-0:~# mv containerd-config.toml /etc/containerd/config.toml
root@node-0:~# mv containerd.service /etc/systemd/system/
root@node-0:~# cat /etc/containerd/config.toml ; echo
version = 2

[plugins."io.containerd.grpc.v1.cri"]               # CRI 플러그인 활성화 : kubelet은 이 플러그인을 통해 containerd와 통신
  [plugins."io.containerd.grpc.v1.cri".containerd]  # containerd 기본 런타임 설정
    snapshotter = "overlayfs"                       # 컨테이너 파일시스템 레이어 관리 방식 : Linux표준/성능최적
    default_runtime_name = "runc"                   # 기본 OCI 런타임 : 파드가 별도 지정 없을 경우 runc 사용
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]  # runc 런타임 상세 설정
    runtime_type = "io.containerd.runc.v2"                        # containerd 최신 runc shim
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]  # runc 옵션
    SystemdCgroup = true                                                  # containerd가 cgroup을 systemd로 관리 
[plugins."io.containerd.grpc.v1.cri".cni]           # CNI 설정
  bin_dir = "/opt/cni/bin"                          # CNI 플러그인 바이너리 위치
  conf_dir = "/etc/cni/net.d"                       # CNI 네트워크 설정 파일 위치
  
  
# Configure the Kubelet : Create the kubelet-config.yaml configuration file:
root@node-0:~# mv kubelet-config.yaml /var/lib/kubelet/
root@node-0:~# mv kubelet.service /etc/systemd/system/

# Configure the Kubernetes Proxy
root@node-0:~# mv kube-proxy-config.yaml /var/lib/kube-proxy/
root@node-0:~# mv kube-proxy.service /etc/systemd/system/

# Start the Worker Services
root@node-0:~# systemctl daemon-reload
root@node-0:~# systemctl enable containerd kubelet kube-proxy
Created symlink /etc/systemd/system/multi-user.target.wants/containerd.service → /etc/systemd/system/containerd.service.
Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service → /etc/systemd/system/kubelet.service.
Created symlink /etc/systemd/system/multi-user.target.wants/kube-proxy.service → /etc/systemd/system/kube-proxy.service.
root@node-0:~# systemctl start containerd kubelet kube-proxy

# 확인
root@node-0:~# systemctl status kubelet --no-pager
● kubelet.service - Kubernetes Kubelet
     Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; preset: enabled)
     Active: active (running) since Sun 2026-01-11 08:59:02 KST; 4s ago
       Docs: https://github.com/kubernetes/kubernetes
   Main PID: 2937 (kubelet)
      Tasks: 11 (limit: 2096)
     Memory: 24.5M
        CPU: 277ms
     CGroup: /system.slice/kubelet.service
             └─2937 /usr/local/bin/kubelet --config=/var/lib/kubelet/kubelet-config.yaml --kubeconfig=/var/lib/kubelet/kubeconfig --v=2

Jan 11 08:59:02 node-0 kubelet[2937]: I0111 08:59:02.425947    2937 kubelet_node_status.go:687] "Recording event message for node" node="node-0" event="NodeHasNoDiskPressure"
Jan 11 08:59:02 node-0 kubelet[2937]: I0111 08:59:02.425955    2937 kubelet_node_status.go:687] "Recording event message for node" node="node-0" event="NodeHasSufficientPID"
Jan 11 08:59:02 node-0 kubelet[2937]: I0111 08:59:02.426436    2937 kubelet_node_status.go:75] "Attempting to register node" node="node-0"
Jan 11 08:59:02 node-0 kubelet[2937]: I0111 08:59:02.432949    2937 kubelet_node_status.go:78] "Successfully registered node" node="node-0"
Jan 11 08:59:02 node-0 kubelet[2937]: I0111 08:59:02.655470    2937 kubelet_node_status.go:687] "Recording event message for node" node="node-0" event="NodeReady"
Jan 11 08:59:02 node-0 kubelet[2937]: I0111 08:59:02.655542    2937 kubelet_node_status.go:501] "Fast updating node status as it just became ready"
Jan 11 08:59:03 node-0 kubelet[2937]: I0111 08:59:03.201514    2937 apiserver.go:52] "Watching apiserver"
Jan 11 08:59:03 node-0 kubelet[2937]: I0111 08:59:03.204673    2937 reflector.go:376] Caches populated for *v1.Pod from pkg/kubelet/config/apiserver.go:66
Jan 11 08:59:03 node-0 kubelet[2937]: I0111 08:59:03.204949    2937 kubelet.go:2468] "SyncLoop ADD" source="api" pods=[]
Jan 11 08:59:03 node-0 kubelet[2937]: I0111 08:59:03.213472    2937 desired_state_of_world_populator.go:157] "Finished populating initial desired state of world"
root@node-0:~# systemctl status containerd --no-pager
● containerd.service - containerd container runtime
     Loaded: loaded (/etc/systemd/system/containerd.service; enabled; preset: enabled)
     Active: active (running) since Sun 2026-01-11 08:59:02 KST; 7s ago
       Docs: https://containerd.io
    Process: 2929 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 2936 (containerd)
      Tasks: 8 (limit: 2096)
     Memory: 17.8M
        CPU: 128ms
     CGroup: /system.slice/containerd.service
             └─2936 /bin/containerd

Jan 11 08:59:02 node-0 containerd[2936]: time="2026-01-11T08:59:02.179211032+09:00" level=info msg="Start event monitor"
Jan 11 08:59:02 node-0 containerd[2936]: time="2026-01-11T08:59:02.179221574+09:00" level=info msg="Start cni network conf syncer for default"
Jan 11 08:59:02 node-0 containerd[2936]: time="2026-01-11T08:59:02.179226490+09:00" level=info msg="Start streaming server"
Jan 11 08:59:02 node-0 containerd[2936]: time="2026-01-11T08:59:02.179238532+09:00" level=info msg="Registered namespace \"k8s.io\" with NRI"
Jan 11 08:59:02 node-0 containerd[2936]: time="2026-01-11T08:59:02.179244282+09:00" level=info msg="runtime interface starting up..."
Jan 11 08:59:02 node-0 containerd[2936]: time="2026-01-11T08:59:02.179248324+09:00" level=info msg="starting plugins..."
Jan 11 08:59:02 node-0 containerd[2936]: time="2026-01-11T08:59:02.179258740+09:00" level=info msg="Synchronizing NRI (plugin) with current runtime state"
Jan 11 08:59:02 node-0 containerd[2936]: time="2026-01-11T08:59:02.179524407+09:00" level=info msg=serving... address=/run/containerd/containerd.sock.ttrpc
Jan 11 08:59:02 node-0 containerd[2936]: time="2026-01-11T08:59:02.180191990+09:00" level=info msg=serving... address=/run/containerd/containerd.sock
Jan 11 08:59:02 node-0 containerd[2936]: time="2026-01-11T08:59:02.180276824+09:00" level=info msg="containerd successfully booted in 0.080985s"
root@node-0:~# systemctl status kube-proxy --no-pager
● kube-proxy.service - Kubernetes Kube Proxy
     Loaded: loaded (/etc/systemd/system/kube-proxy.service; enabled; preset: enabled)
     Active: active (running) since Sun 2026-01-11 08:59:02 KST; 10s ago
       Docs: https://github.com/kubernetes/kubernetes
   Main PID: 2930 (kube-proxy)
      Tasks: 5 (limit: 2096)
     Memory: 14.2M
        CPU: 282ms
     CGroup: /system.slice/kube-proxy.service
             └─2930 /usr/local/bin/kube-proxy --config=/var/lib/kube-proxy/kube-proxy-config.yaml

Jan 11 08:59:03 node-0 kube-proxy[2930]: I0111 08:59:03.470716    2930 server.go:499] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jan 11 08:59:03 node-0 kube-proxy[2930]: I0111 08:59:03.474823    2930 config.go:199] "Starting service config controller"
Jan 11 08:59:03 node-0 kube-proxy[2930]: I0111 08:59:03.474988    2930 shared_informer.go:313] Waiting for caches to sync for service config
Jan 11 08:59:03 node-0 kube-proxy[2930]: I0111 08:59:03.475123    2930 config.go:329] "Starting node config controller"
Jan 11 08:59:03 node-0 kube-proxy[2930]: I0111 08:59:03.475178    2930 shared_informer.go:313] Waiting for caches to sync for node config
Jan 11 08:59:03 node-0 kube-proxy[2930]: I0111 08:59:03.475453    2930 config.go:105] "Starting endpoint slice config controller"
Jan 11 08:59:03 node-0 kube-proxy[2930]: I0111 08:59:03.475478    2930 shared_informer.go:313] Waiting for caches to sync for endpoint slice config
Jan 11 08:59:03 node-0 kube-proxy[2930]: I0111 08:59:03.575744    2930 shared_informer.go:320] Caches are synced for node config
Jan 11 08:59:03 node-0 kube-proxy[2930]: I0111 08:59:03.575765    2930 shared_informer.go:320] Caches are synced for endpoint slice config
Jan 11 08:59:03 node-0 kube-proxy[2930]: I0111 08:59:03.575802    2930 shared_informer.go:320] Caches are synced for service config

원격 접근을 위한 kubectl 설정

jumpbox 노드에서 kubectl을 admin 자격 증명으로 사용을 위한 설정을 구성합니다.

# You should be able to ping server.kubernetes.local based on the /etc/hosts DNS entry from a previous lab.
root@jumpbox:~/kubernetes-the-hard-way# curl -s --cacert ca.crt https://server.kubernetes.local:6443/version | jq
{
  "major": "1",
  "minor": "32",
  "gitVersion": "v1.32.3",
  "gitCommit": "32cc146f75aad04beaaa245a7157eb35063a9f99",
  "gitTreeState": "clean",
  "buildDate": "2025-03-11T19:52:21Z",
  "goVersion": "go1.23.6",
  "compiler": "gc",
  "platform": "linux/arm64"
}

# Generate a kubeconfig file suitable for authenticating as the admin user:
root@jumpbox:~/kubernetes-the-hard-way# kubectl config set-cluster kubernetes-the-hard-way \
  --certificate-authority=ca.crt \
  --embed-certs=true \
  --server=https://server.kubernetes.local:6443
Cluster "kubernetes-the-hard-way" set.

root@jumpbox:~/kubernetes-the-hard-way# kubectl config set-credentials admin \
  --client-certificate=admin.crt \
  --client-key=admin.key
User "admin" set.

root@jumpbox:~/kubernetes-the-hard-way# kubectl config set-context kubernetes-the-hard-way \
  --cluster=kubernetes-the-hard-way \
  --user=admin
Context "kubernetes-the-hard-way" created.

root@jumpbox:~/kubernetes-the-hard-way# kubectl config use-context kubernetes-the-hard-way
Switched to context "kubernetes-the-hard-way".

root@jumpbox:~/kubernetes-the-hard-way# kubectl version
Client Version: v1.32.3
Kustomize Version: v5.5.0
Server Version: v1.32.3
root@jumpbox:~/kubernetes-the-hard-way#
root@jumpbox:~/kubernetes-the-hard-way# kubectl get nodes -v=6
I0111 09:20:48.852866    3046 loader.go:402] Config loaded from file:  /root/.kube/config
I0111 09:20:48.853973    3046 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I0111 09:20:48.854086    3046 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
I0111 09:20:48.854173    3046 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I0111 09:20:48.854255    3046 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I0111 09:20:48.854156    3046 cert_rotation.go:140] Starting client certificate rotation controller
I0111 09:20:48.878114    3046 round_trippers.go:560] GET https://server.kubernetes.local:6443/api?timeout=32s 200 OK in 23 milliseconds
I0111 09:20:48.880569    3046 round_trippers.go:560] GET https://server.kubernetes.local:6443/apis?timeout=32s 200 OK in 1 milliseconds
I0111 09:20:48.893465    3046 round_trippers.go:560] GET https://server.kubernetes.local:6443/api/v1/nodes?limit=500 200 OK in 2 milliseconds
NAME     STATUS   ROLES    AGE     VERSION
node-0   Ready    <none>   21m     v1.32.3
node-1   Ready    <none>   2m15s   v1.32.3

root@jumpbox:~/kubernetes-the-hard-way# cat /root/.kube/config
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUZURENDQXpTZ0F3SUJBZ0lVY1NNVHR5S29XcXlkK1lYWUZqVVZhd05pT2prd0RRWUpLb1pJaHZjTkFRRU4KQlFBd1FURUxNQWtHQTFVRUJoTUNWVk14RXpBUkJnTlZCQWdNQ2xkaGMyaHBibWQwYjI0eEVEQU9CZ05WQkFjTQpCMU5sWVhSMGJHVXhDekFKQmdOVkJBTU1Ba05CTUI0WERUSTJNREV4TURJeU1EVTFObG9YRFRNMk1ERXhNVEl5Ck1EVTFObG93UVRFTE1Ba0dBMVVFQmhNQ1ZWTXhFekFSQmdOVkJBZ01DbGRoYzJocGJtZDBiMjR4RURBT0JnTlYKQkFjTUIxTmxZWFIwYkdVeEN6QUpCZ05WQkFNTUFrTkJNSUlDSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQWc4QQpNSUlDQ2dLQ0FnRUFudE4xUGh0Q3lCZ0dYUnBWeHViaUp6emhTOStVUUpiUEgxeWd4WTNrOVVlQmZnV1YvSmxRCjBmWk1ib294aXMyYzVBR1B1SnpZbW9UbkZtZjNxRWdVY09Rc0ZVYlNGSWtSU1FOQW9JMmRUdk1ESUVLNXgrK3gKazZxN1luaVhHWFFEN3R3QUdON2dvdGt4NjNGSDBPbEJNb0tBR05QcTBNZHFGMEVacktxdXlUQ1paU09yRHNjawo2bysycGxJNWVlTmg1YzBFaUFFTmxZdFc5UVd5enJDOFZLOWNYR0hDU1hqemhUc05KQUY2a0ZvNWFUTWZtRjdDCnhBTFRqSlFpc0o0VkpZdytHdVRkS2FnME00b0svTXQySkxlcElJeGl4dCtQTUpGNWdLeGpVdDVUMmE4emZ1NkwKQlQ5UEs3VGQydWd1TEdGTzh1dEJhT0U4T0NZYzZmMGdscVJ0ZHJHSUF6dXZBNk9LZlJYMGdQQ2N4bzY3UVRQeQpMdENaRUVlWnlvV3ZoRVNRVUxuYzNINVBzVzkwMndvdEU0L0ovRzIwNzhkNlhUZzRWNS9wUzhUNDQ5Q20yZXd1Cld5bzZNc1FZMllWbUdIdzBxTDhMZ2dDRDlNVHF2QTFoWVBzOGo0NjZxUldic01nNTFoRmxXazZ5b081RS9INXcKVUhuQlQrTHVWb3RpOXBRNC9oWjg3aDZNc0NsT3lDK2xFM0ZJMlkvU0FwamZXS0Y5dHVJaHhmY3J3dW5wcnUrTAorZFY5eDZ5em45SS9xQjhadmFLeTFuS0g2RU53RG5PN2llN0YrNTFnb2FacTBFV2UrSHg1ZzJFcHUwZlArdlBPCnUrSzB5bUxCaVhhVGJBS0NTTW9vNFpab3BRQjBBTVVrb3NzNkgxYzZSRjV3bDZabFlreXcrWkVDQXdFQUFhTTgKTURvd0RBWURWUjBUQkFVd0F3RUIvekFMQmdOVkhROEVCQU1DQVFZd0hRWURWUjBPQkJZRUZKN05Ocm9pYTBPbgoyeU1wbDFNYjBYTGZiV0hTTUEwR0NTcUdTSWIzRFFFQkRRVUFBNElDQVFCd3ByOCs4bmlHRTlVakFLUUcyQU5rCnh6SG5NUFFOeU44NXpFcXJCMFp4M2tpSEJ0RS9GaHpRdHhoL0lkZnozRTN0U0k4blRNbTRsSDYycnZQdmM1NEYKMTlLMWdXV3poeExrWkMrdEsyMEpVVVN5dVJkS0VlTzZTSHB0UVlwOEZkTWVhWVlMSS9jSHVBcmRTNkw5ek1pYQpCTDRZWEdwbTFxM3lYeVRrd3VnSFJTcFlxeUJ6UDhMa3NESjU3T2EyYSs3Q0xaYXU1ZnczSkpPRFJEc3V5OXRUClhGS2UzT2VWQS9INmcvQUMwUzlKa1JDTnVKODhxOG40S0hDRTRySzA2dUErVklUaHltd1p0dHlsTkhDQUVGNncKa2ptbjQ1SndIS3ZSY2YybTVhZDlVNWNWd0p5bkJidnZXNi9lMXFOVGgyZnRvcEUvUVVraHozUk8vNWlMQ3BIQgpNUjZWZW5qY05FYTg2TnoyUTVWOFRmLzhSWHpOWjZHU1BBQXhsUENTNDNkWkpMZVZUc2N2bTBURlZzRm5kQktxCmlOWXZjUGxyWHp0diswTVJxOXpkVWpQS1g2by85L2RBOXkxZG9GQlBRMWFxRWhnUGRWQnJqenJNbjdXaUVna2cKZWF4VlZaYnovbXNxYUF1akZnRUdaZ2V5OEFWUEI3WWNKSit5dXFSV2xMbmJ0enl2MUt6WGd1SytYYjZBazRFdAp5T2pscWhINlNKNm9JODA3MldwcXp5Z2hEcG9IOWt5VGpILy8xSEVLNVZOdXUxbUYxMGVrd05VQk9wQkREdXl5CjNDS20xUmhkUHlYRjhtSUxwOU9FODBrb3E1SzZ0RndOUy9oREVSdFFXdC9zMUN2dHFLa1NxZFlSVUlzKzBzcXQKWVRCZTJIdWJZVGptaUdOMzhOZnlOQT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    server: https://server.kubernetes.local:6443
  name: kubernetes-the-hard-way
contexts:
- context:
    cluster: kubernetes-the-hard-way
    user: admin
  name: kubernetes-the-hard-way
current-context: kubernetes-the-hard-way
kind: Config
preferences: {}
users:
- name: admin
  user:
    client-certificate: /root/kubernetes-the-hard-way/admin.crt
    client-key: /root/kubernetes-the-hard-way/admin.key

Pod 네트워크 라우팅 구성

node-0, node-1에 PodCIDR과 통신을 위한 OS커널에 라우팅 설정을 구성합니다.

구성하려는 네트워크 대역은 다음과 같습니다.

항목	네트워크 대역 or IP
clusterCIDR	10.200.0.0/16
→ node-0 PodCIDR	10.200.0.0/24
→ node-1 PodCIDR	10.200.1.0/24
ServiceCIDR	10.32.0.0/24
→ api clusterIP	10.32.0.1

# Print the internal IP address and Pod CIDR range for each worker instance:
root@jumpbox:~/kubernetes-the-hard-way# SERVER_IP=$(grep server machines.txt | cut -d " " -f 1)
NODE_0_IP=$(grep node-0 machines.txt | cut -d " " -f 1)
NODE_0_SUBNET=$(grep node-0 machines.txt | cut -d " " -f 4)
NODE_1_IP=$(grep node-1 machines.txt | cut -d " " -f 1)
NODE_1_SUBNET=$(grep node-1 machines.txt | cut -d " " -f 4)
echo $SERVER_IP $NODE_0_IP $NODE_0_SUBNET $NODE_1_IP $NODE_1_SUBNET
192.168.10.100 192.168.10.101 10.200.0.0/24 192.168.10.102 10.200.1.0/24

root@jumpbox:~/kubernetes-the-hard-way# ssh server ip -c route
default via 10.0.2.2 dev eth0
10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.15
192.168.10.0/24 dev eth1 proto kernel scope link src 192.168.10.100

root@jumpbox:~/kubernetes-the-hard-way# ssh root@server <<EOF
  ip route add ${NODE_0_SUBNET} via ${NODE_0_IP}
  ip route add ${NODE_1_SUBNET} via ${NODE_1_IP}
EOF
Pseudo-terminal will not be allocated because stdin is not a terminal.
Linux server 6.1.0-40-arm64 #1 SMP Debian 6.1.153-1 (2025-09-20) aarch64

This system is built by the Bento project by Chef Software
More information can be found at https://github.com/chef/bento

Use of this system is acceptance of the OS vendor EULA and License Agreements.

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.

root@jumpbox:~/kubernetes-the-hard-way# ssh server ip -c route
default via 10.0.2.2 dev eth0
10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.15
10.200.0.0/24 via 192.168.10.101 dev eth1
10.200.1.0/24 via 192.168.10.102 dev eth1
192.168.10.0/24 dev eth1 proto kernel scope link src 192.168.10.100

root@jumpbox:~/kubernetes-the-hard-way# ssh node-0 ip -c route
default via 10.0.2.2 dev eth0
10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.15
192.168.10.0/24 dev eth1 proto kernel scope link src 192.168.10.101

root@jumpbox:~/kubernetes-the-hard-way# ssh root@node-0 <<EOF
  ip route add ${NODE_1_SUBNET} via ${NODE_1_IP}
EOF
Pseudo-terminal will not be allocated because stdin is not a terminal.
Linux node-0 6.1.0-40-arm64 #1 SMP Debian 6.1.153-1 (2025-09-20) aarch64

This system is built by the Bento project by Chef Software
More information can be found at https://github.com/chef/bento

Use of this system is acceptance of the OS vendor EULA and License Agreements.

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.

root@jumpbox:~/kubernetes-the-hard-way# ssh node-0 ip -c route
default via 10.0.2.2 dev eth0
10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.15
10.200.1.0/24 via 192.168.10.102 dev eth1
192.168.10.0/24 dev eth1 proto kernel scope link src 192.168.10.101


root@jumpbox:~/kubernetes-the-hard-way# ssh node-1 ip -c route
default via 10.0.2.2 dev eth0
10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.15
192.168.10.0/24 dev eth1 proto kernel scope link src 192.168.10.102

root@jumpbox:~/kubernetes-the-hard-way# ssh root@node-1 <<EOF
  ip route add ${NODE_0_SUBNET} via ${NODE_0_IP}
EOF
Pseudo-terminal will not be allocated because stdin is not a terminal.
Linux node-1 6.1.0-40-arm64 #1 SMP Debian 6.1.153-1 (2025-09-20) aarch64

This system is built by the Bento project by Chef Software
More information can be found at https://github.com/chef/bento

Use of this system is acceptance of the OS vendor EULA and License Agreements.

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.

root@jumpbox:~/kubernetes-the-hard-way# ssh node-1 ip -c route
default via 10.0.2.2 dev eth0
10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.15
10.200.0.0/24 via 192.168.10.101 dev eth1
192.168.10.0/24 dev eth1 proto kernel scope link src 192.168.10.102

Smoke Test

최종 구성도는 포스팅 상단에 작성한 다음 그림과 같습니다.

kubernetes 동작 테스트를 진행하겠습니다.

root@jumpbox:~/kubernetes-the-hard-way# kubectl create secret generic kubernetes-the-hard-way --from-literal="mykey=mydata"
secret/kubernetes-the-hard-way created

root@jumpbox:~/kubernetes-the-hard-way# kubectl get secret kubernetes-the-hard-way
NAME                      TYPE     DATA   AGE
kubernetes-the-hard-way   Opaque   1      5s

root@jumpbox:~/kubernetes-the-hard-way# kubectl get secret kubernetes-the-hard-way -o yaml
apiVersion: v1
data:
  mykey: bXlkYXRh
kind: Secret
metadata:
  creationTimestamp: "2026-01-11T00:33:42Z"
  name: kubernetes-the-hard-way
  namespace: default
  resourceVersion: "4565"
  uid: ce1640d0-b57d-4e66-9edd-168e2580710b
type: Opaque

root@jumpbox:~/kubernetes-the-hard-way# kubectl get secret kubernetes-the-hard-way -o jsonpath='{.data.mykey}' ; echo
bXlkYXRh

root@jumpbox:~/kubernetes-the-hard-way# kubectl get secret kubernetes-the-hard-way -o jsonpath='{.data.mykey}' | base64 -d ; echo
mydata

# Print a hexdump of the kubernetes-the-hard-way secret stored in etcd
## etcdctl get … : etcd 내부 key 직접 조회, kubernetes API 우회(매우 강력한 접근)
## Secret 리소스의 etcd 실제 저장 경로: /registry/<resource>/<namespace>/<name> -> /registry/secrets/default/kubernetes-the-hard-way
root@jumpbox:~/kubernetes-the-hard-way# ssh root@server \
    'etcdctl get /registry/secrets/default/kubernetes-the-hard-way | hexdump -C'
00000000  2f 72 65 67 69 73 74 72  79 2f 73 65 63 72 65 74  |/registry/secret|
00000010  73 2f 64 65 66 61 75 6c  74 2f 6b 75 62 65 72 6e  |s/default/kubern|
00000020  65 74 65 73 2d 74 68 65  2d 68 61 72 64 2d 77 61  |etes-the-hard-wa|
00000030  79 0a 6b 38 73 3a 65 6e  63 3a 61 65 73 63 62 63  |y.k8s:enc:aescbc|
00000040  3a 76 31 3a 6b 65 79 31  3a 8a 92 59 2e 76 12 be  |:v1:key1:..Y.v..|
00000050  c6 bf 8a 5e 14 50 fe d4  a8 0a 8c 02 95 bd 38 06  |...^.P........8.|
00000060  53 5d 71 8d 33 db 1b 35  3c 57 b3 ca 81 ca 52 ab  |S]q.3..5<W....R.|
00000070  d9 7b 5b c1 cf 1d 6a e9  26 be 67 b9 33 a6 ba 38  |.{[...j.&.g.3..8|
00000080  0a ad af d3 0e 1e d7 ad  59 54 4b 1e 93 28 12 80  |........YTK..(..|
00000090  67 55 44 dd 4a 00 9c c7  58 5a 10 6b 47 76 42 76  |gUD.J...XZ.kGvBv|
000000a0  64 5d da cc 11 7d f7 38  60 c7 d8 70 ac ea 75 49  |d]...}.8`..p..uI|
000000b0  fd d8 83 6a 5e bb 9f 8f  60 73 f5 9f 95 e6 05 00  |...j^...`s......|
000000c0  2e af a9 63 38 be a7 5a  7e e4 a2 23 52 fd 47 63  |...c8..Z~..#R.Gc|
000000d0  d1 12 41 3b 36 4d 7a ce  d7 0c 25 02 d8 40 d5 a7  |..A;6Mz...%..@..|
000000e0  04 ee 55 b1 20 5f e6 02  b6 ec a8 ab 7c fa 51 81  |..U. _......|.Q.|
000000f0  c3 78 18 64 f8 1a e0 3d  44 d9 57 3b 80 4d d5 35  |.x.d...=D.W;.M.5|
00000100  df 2f 79 c4 5c 1e 40 a5  93 ba 5f 52 bd f6 c2 fd  |./y.\.@..._R....|
00000110  03 c7 ff e0 59 68 c1 ec  6c 02 af 94 56 45 10 6d  |....Yh..l...VE.m|
00000120  d1 ce 21 d3 42 f8 12 e8  2e 77 46 8a fd 4f 99 29  |..!.B....wF..O.)|
00000130  a8 ba 9c 26 fe 55 cd 12  f7 29 2e 7d 0b ee 12 c6  |...&.U...).}....|
00000140  ad a2 60 83 56 b6 af 14  a4 13 20 1c ef 4e 95 19  |..`.V..... ..N..|
00000150  c1 b6 dd fc 78 40 66 82  07 0a                    |....x@f...|
0000015a

... # Kubernetes Secret이 etcd에 AES-CBC 방식으로 정상 암호화되어 저장되고 있음을 증명하는 출력
# k8s:enc	: Kubernetes 암호화 포맷
# aescbc	: 암호화 알고리즘 (AES-CBC)
# v1	    : encryption provider 버전
# key1	  : 사용된 encryption key 이름
# 이후 데이터는 암호화된 데이터

[Cilium] Tetragon

안녕유지 — Sun, 7 Sep 2025 19:48:40 +0900

Cloudnet Cilium 8주차 스터디를 진행하며 정리한 글입니다.

이번 포스팅에서는 쿠버네티스 환경에서 보안을 강화하기 위한 오픈소스 도구 중 Tetragon에 대해 알아보겠습니다.

Tetragon

Tetragon은 보안에 중요한 이벤트를 커널 단에서 감지하고, 필요 시 대응까지 할 수 있습니다.

Process: 실행 및 권한 변경 이벤트
System: 시스템 콜 활동
I/O: 네트워크 및 파일 접근

특히 Kubernetes 환경에서 실행되면, Tetragon은 네임스페이스, 포드, 워크로드 단위로 이벤트를 인식합니다.
단순히 OS 레벨이 아니라 이 Pod 안에서 실행된 프로세스가 문제다라는 식으로 컨텍스트 기반 추적이 가능하다는 점이 큰 장점입니다.

https://isovalent.com/blog/post/top-tetragon-use-cases/

1) eBPF 실시간

Tetragon의 가장 큰 특징은 eBPF를 통한 실시간 이벤트 처리입니다.

보통 보안/관찰 툴은 이벤트를 커널 → 유저 스페이스로 올린 후 필터링/분석합니다. 이 과정에서 성능저하가 일반적으로 발생하지만 Tetragon은 다릅니다.

정책과 필터를 커널 레벨 eBPF에 직접 적용
이벤트를 사용자 공간으로 전송하기 전에 커널에서 직접 필터링/차단/대응
컨텍스트 전환 오버헤드 감소 → 성능 효율↑

예를 들어, read()와 같은 호출은 너무 자주 발생합니다. 이런 이벤트를 모두 올려 처리하면 리소스 낭비가 심한데, Tetragon은 커널에서 미리 걸러 필요한 이벤트만 전달합니다.

2) eBPF 유연성

Tetragon은 Linux 커널의 거의 모든 함수에 후킹할 수 있습니다.

이를 통해

인수, 반환 값 추적
실행 파일 이름, 파일 속성, 프로세스 메타데이터 확인
특정 함수 기반의 필터 적용

즉, 보안 담당자가 원하는 대로 추적 정책(Tracing Policy) 을 정의할 수 있어 사용자 공간에서 변조될 위험 없이 커널 깊숙한 곳에서 안전하게 추적이 가능합니다.
이는 단순한 system call 추적 방식보다 안정적이고 신뢰성이 높습니다.

3) eBPF 커널 인식

Tetragon은 커널 레벨 상태와 Kubernetes 정보를 결합하여 더 정교한 정책을 적용할 수 있습니다.

예를 들어

애플리케이션이 권한 상승을 시도할 때 → 커널에서 이벤트 발생 전에 탐지
특정 네임스페이스 내의 파일 접근 차단 정책 적용
소켓, FD, 프로세스 네임스페이스를 매핑하여 Pod 단위 보안 정책 실행

이런 방식으로 실행 후 탐지가 아닌, 실시간 차단,경고가 가능합니다.

Tetragon 설치

# Helm 설치
root@k8s-ctr:~# helm repo add cilium https://helm.cilium.io
root@k8s-ctr:~# helm repo update
root@k8s-ctr:~# helm install tetragon cilium/tetragon -n kube-system
NAME: tetragon
LAST DEPLOYED: Sun Sep  7 19:35:19 2025
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None


# 데모 어플리케이션 배포
root@k8s-ctr:~#
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.18.1/examples/minikube/http-sw-app.yaml
root@k8s-ctr:~# kubectl get pods
NAME                         READY   STATUS    RESTARTS   AGE
deathstar-86f85ffb4d-9k6d4   1/1     Running   0          76s
deathstar-86f85ffb4d-kkfwk   1/1     Running   0          76s
tiefighter                   1/1     Running   0          75s
xwing                        1/1     Running   0          75s

Tetragon 실행 모니터링

Tetragon은 실행 이벤트를 JSON 로그와 GRPC 스트림을 통해 공개합니다. 사용자는 이를 통해 시스템의 모든 실행 과정을 관찰할 수 있습니다.

# 여러 노드가 있는 클러스터에서는 사용하는 Tetragon Pod가 "xwing" Pod와 동일한 노드에 있어야 실행 이벤트를 캡처할 수 있습니다.
# 이 명령을 사용하면 "xwing" Pod와 동일한 Kubernetes 노드에 있는 Tetragon Pod의 이름을 가져올 수 있습니다.
root@k8s-ctr:~# POD=$(kubectl -n kube-system get pods -l 'app.kubernetes.io/name=tetragon' -o name --field-selector spec.nodeName=$(kubectl get pod xwing -o jsonpath='{.spec.nodeName}'))
root@k8s-ctr:~# echo $POD
pod/tetragon-whvgw

# 터미널1
# 일치하는 Pod를 식별한 후 해당 Pod를 대상으로 지정하여 명령을 실행합니다 : Tetragon에서 캡처한 실행 이벤트를 반환
root@k8s-ctr:~# kubectl exec -ti -n kube-system $POD -c tetragon -- tetra getevents --pods xwing
{"process_exec":{"process":{"exec_id":"azhzLXcxOjEzMzU1NjY2ODQ2NDY3Ojc3OTk=","pid":7799,"uid":0,"cwd":"/","binary":"/usr/bin/bash","arguments":"-c \"curl https://ebpf.io/applications/#tetragon\"","flags":"execve rootcwd clone","start_time":"2025-09-07T10:40:12.435259269Z","auid":4294967295,"pod":{"namespace":"default","name":"xwing","container":{"id":"containerd://bb255f4a2fdad513391f8f7687c6f0001ced83d8e968a4f018bfab129968fffe","name":"spaceship","image":{"id":"quay.io/cilium/json-mock@sha256:5aad04835eda9025fe4561ad31be77fd55309af8158ca8663a72f6abb78c2603","name":"sha256:56b43d7e51feffe637c2837a8c70da02be98a51099533f288c78fa369f5c6991"},"start_time":"2025-09-07T10:37:37Z","pid":18,"security_context":{}},"pod_labels":{"app.kubernetes.io/name":"xwing","class":"xwing","org":"alliance"},"workload":"xwing","workload_kind":"Pod"},"docker":"bb255f4a2fdad513391f8f7687c6f00","parent_exec_id":"azhzLXcxOjEzMTM2ODEwMDAwMDAwOjc1MDI=","tid":7799,"in_init_tree":false},"parent":{"exec_id":"azhzLXcxOjEzMTM2ODEwMDAwMDAwOjc1MDI=","pid":7502,"uid":0,"cwd":"/run/containerd/io.containerd.runtime.v2.task/k8s.io/c929beffd65aa3b52155428fbb6d2c0bcc482cbef7ab8fcd65ca3019389165ee","binary":"/usr/bin/containerd-shim-runc-v2","arguments":"-namespace k8s.io -id c929beffd65aa3b52155428fbb6d2c0bcc482cbef7ab8fcd65ca3019389165ee -address /run/containerd/containerd.sock","flags":"procFS auid","start_time":"2025-09-07T10:36:33.578411760Z","auid":4294967295,"parent_exec_id":"azhzLXcxOjEwMDAwMDAwOjE=","tid":7502,"in_init_tree":false}},"node_name":"k8s-w1","time":"2025-09-07T10:40:12.435258227Z","node_labels":{"beta.kubernetes.io/arch":"arm64","beta.kubernetes.io/os":"linux","kubernetes.io/arch":"arm64","kubernetes.io/hostname":"k8s-w1","kubernetes.io/os":"linux"}}
{"process_exec":{"process":{"exec_id":"azhzLXcxOjEzMzU1Njc5NDQwNTkyOjc3OTk=","pid":7799,"uid":0,"cwd":"/","binary":"/usr/bin/curl","arguments":"https://ebpf.io/applications/#tetragon","flags":"execve rootcwd","start_time":"2025-09-07T10:40:12.447853727Z","auid":4294967295,"pod":{"namespace":"default","name":"xwing","container":{"id":"containerd://bb255f4a2fdad513391f8f7687c6f0001ced83d8e968a4f018bfab129968fffe","name":"spaceship","image":{"id":"quay.io/cilium/json-mock@sha256:5aad04835eda9025fe4561ad31be77fd55309af8158ca8663a72f6abb78c2603","name":"sha256:56b43d7e51feffe637c2837a8c70da02be98a51099533f288c78fa369f5c6991"},"start_time":"2025-09-07T10:37:37Z","pid":18,"security_context":{}},"pod_labels":{"app.kubernetes.io/name":"xwing","class":"xwing","org":"alliance"},"workload":"xwing","workload_kind":"Pod"},"docker":"bb255f4a2fdad513391f8f7687c6f00","parent_exec_id":"azhzLXcxOjEzMzU1NjY2ODQ2NDY3Ojc3OTk=","tid":7799,"in_init_tree":false},"parent":{"exec_id":"azhzLXcxOjEzMzU1NjY2ODQ2NDY3Ojc3OTk=","pid":7799,"uid":0,"cwd":"/","binary":"/usr/bin/bash","arguments":"-c \"curl https://ebpf.io/applications/#tetragon\"","flags":"execve rootcwd clone","start_time":"2025-09-07T10:40:12.435259269Z","auid":4294967295,"pod":{"namespace":"default","name":"xwing","container":{"id":"containerd://bb255f4a2fdad513391f8f7687c6f0001ced83d8e968a4f018bfab129968fffe","name":"spaceship","image":{"id":"quay.io/cilium/json-mock@sha256:5aad04835eda9025fe4561ad31be77fd55309af8158ca8663a72f6abb78c2603","name":"sha256:56b43d7e51feffe637c2837a8c70da02be98a51099533f288c78fa369f5c6991"},"start_time":"2025-09-07T10:37:37Z","pid":18,"security_context":{}},"pod_labels":{"app.kubernetes.io/name":"xwing","class":"xwing","org":"alliance"},"workload":"xwing","workload_kind":"Pod"},"docker":"bb255f4a2fdad513391f8f7687c6f00","parent_exec_id":"azhzLXcxOjEzMTM2ODEwMDAwMDAwOjc1MDI=","tid":7799,"in_init_tree":false}},"node_name":"k8s-w1","time":"2025-09-07T10:40:12.447853269Z","node_labels":{"beta.kubernetes.io/arch":"arm64","beta.kubernetes.io/os":"linux","kubernetes.io/arch":"arm64","kubernetes.io/hostname":"k8s-w1","kubernetes.io/os":"linux"}}
{"process_exit":{"process":{"exec_id":"azhzLXcxOjEzMzU1Njc5NDQwNTkyOjc3OTk=","pid":7799,"uid":0,"cwd":"/","binary":"/usr/bin/curl","arguments":"https://ebpf.io/applications/#tetragon","flags":"execve rootcwd","start_time":"2025-09-07T10:40:12.447853727Z","auid":4294967295,"pod":{"namespace":"default","name":"xwing","container":{"id":"containerd://bb255f4a2fdad513391f8f7687c6f0001ced83d8e968a4f018bfab129968fffe","name":"spaceship","image":{"id":"quay.io/cilium/json-mock@sha256:5aad04835eda9025fe4561ad31be77fd55309af8158ca8663a72f6abb78c2603","name":"sha256:56b43d7e51feffe637c2837a8c70da02be98a51099533f288c78fa369f5c6991"},"start_time":"2025-09-07T10:37:37Z","pid":18,"security_context":{}},"pod_labels":{"app.kubernetes.io/name":"xwing","class":"xwing","org":"alliance"},"workload":"xwing","workload_kind":"Pod"},"docker":"bb255f4a2fdad513391f8f7687c6f00","parent_exec_id":"azhzLXcxOjEzMzU1NjY2ODQ2NDY3Ojc3OTk=","tid":7799,"in_init_tree":false},"parent":{"exec_id":"azhzLXcxOjEzMzU1NjY2ODQ2NDY3Ojc3OTk=","pid":7799,"uid":0,"cwd":"/","binary":"/usr/bin/bash","arguments":"-c \"curl https://ebpf.io/applications/#tetragon\"","flags":"execve rootcwd clone","start_time":"2025-09-07T10:40:12.435259269Z","auid":4294967295,"pod":{"namespace":"default","name":"xwing","container":{"id":"containerd://bb255f4a2fdad513391f8f7687c6f0001ced83d8e968a4f018bfab129968fffe","name":"spaceship","image":{"id":"quay.io/cilium/json-mock@sha256:5aad04835eda9025fe4561ad31be77fd55309af8158ca8663a72f6abb78c2603","name":"sha256:56b43d7e51feffe637c2837a8c70da02be98a51099533f288c78fa369f5c6991"},"start_time":"2025-09-07T10:37:37Z","pid":18,"security_context":{}},"pod_labels":{"app.kubernetes.io/name":"xwing","class":"xwing","org":"alliance"},"workload":"xwing","workload_kind":"Pod"},"docker":"bb255f4a2fdad513391f8f7687c6f00","parent_exec_id":"azhzLXcxOjEzMTM2ODEwMDAwMDAwOjc1MDI=","tid":7799,"in_init_tree":false},"time":"2025-09-07T10:40:13.858683478Z"},"node_name":"k8s-w1","time":"2025-09-07T10:40:13.858681020Z","node_labels":{"beta.kubernetes.io/arch":"arm64","beta.kubernetes.io/os":"linux","kubernetes.io/arch":"arm64","kubernetes.io/hostname":"k8s-w1","kubernetes.io/os":"linux"}}

# 터미널2
root@k8s-ctr:~# kubectl exec -ti xwing -- bash -c 'curl https://ebpf.io/applications/#tetragon'
...
 stroke-linejoin="round" stroke-width="1.4" d="m3.938 12.688 1.75-11.375M8.313 12.688l1.75-11.375M3.063 4.813h8.75M2.188 9.188h8.75"></path></svg></span>DeepFlow</a></h3><h4 class="mt-0.5 font-sans text-lg font-medium leading-snug">Highly Automated Observability Platform powered by eBPF</h4><p class="mt-2.5">DeepFlow is a highly automated observability platform built for cloud native developers. Based on eBPF, DeepFlow innovatively implements an automated distributed tracing mechanism: AutoTracing. Microservice processes, service mesh sidecars, and network interfaces along the way are included as tracing spans, for every distributed transaction, without any code instrumentation. DeepFlow can automatically generate golden RED metrics for any process in cloud native environment.</p><div class="mt-3.5 flex flex-wrap gap-x-5 gap-y-3.5"><a class="inline-flex leading-none items-center transition-colors duration-200 text-sm font-semibold text-black hover:text-gray-40 relative uppercase leading-none tracking-[0.03em] before:absolute before:-right-3 before:top-1/2 before:block before:h-1 before:w-1 before:-translate-y-1/2 before:rounded-full before:bg-gray-90 last:before:hidden" href="https://github.com/deepflowys/deepflow" target="_blank" rel="noreferrer">GitHub</a><a class="inline-flex leading-none items-center transition-colors duration-200 text-sm font-semibold text-black hover:text-gray-40 relative uppercase leading-none tracking-[0.03em] before:absolute before:-right-3 before:top-1/2 before:block before:h-1 before:w-1 before:-translate-y-1/2 before:rounded-full before:bg-gray-90 last:before:hidden" href="https://deepflow.yunshan.net/community.html" target="_blank" rel="noreferrer">Website</a></div></div></li><li class="flex items-start space-x-8 pb-10 pt-7 sm:flex-col sm:space-x-0 sm:pb-7 sm:pt-6 cursor-pointer transition-colors duration-200 rounded-lg" id="retina"><div class="mb-4 w-full max-w-[116px] shrink-0 sm:max-w-[90px] xs:max-w-[80px]"><img src="/static/retina-b4a41f24d7ba93a6010fbd4649fc0a41.svg" alt="Retina" loading="lazy" width="116" height="116"/></div><div class="flex-1 min-w-0"><h3 id="retina" class="group relative flex w-fit heading-6xl font-semibold"><a class="anchor inline-flex text-black before:!hidden hover:!text-black" href="#retina" tabindex="-1" aria-hidden="true"><span class="absolute top-1/2 flex h-full -translate-x-full -translate-y-[calc(50%-0.15rem)] items-center justify-center px-2.5 opacity-0 transition-opacity duration-200 hover:opacity-100 group-hover:opacity-100 sm:hidden -right-16"><svg xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 14 14" class="w-3.5 text-gray-60"><path stroke="currentColor" stroke-linecap="square" stroke-linejoin="round" stroke-width="1.4" d="m3.938 12.688 1.75-11.375M8.313 12.688l1.75-11.375M3.063 4.813h8.75M2.188 9.188h8.75"></path></svg></span>Retina</a></h3><h4 class="mt-0.5 font-sans text-lg font-medium leading-snug">
...

Tetragon 파일 접속 모니터링

추적 정책으로 민감 파일 모니터링도 가능합니다.

YAML 구성 파일을 통해 Tetragon에 추적 정책을 추가하면 Tetragon의 기본 실행 추적 기능을 확장할 수 있습니다.

이러한 정책은 커널에서 필터링을 수행하여 커널에서 실행 중인 BPF 프로그램에서 관심 있는 이벤트만 사용자 공간에 게시되도록 합니다.이를 통해 사용량이 많은 시스템에서도 오버헤드를 낮게 유지할 수 있습니다.

https://isovalent.com/blog/post/file-monitoring-with-ebpf-and-tetragon-part-1/

root@k8s-ctr:~# cat file_monitoring.yaml
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "file-monitoring-filtered"
spec:
  kprobes:
  - call: "security_file_permission"
    syscall: false
    return: true
    args:
    - index: 0
      type: "file" # (struct file *) used for getting the path
    - index: 1
      type: "int" # 0x04 is MAY_READ, 0x02 is MAY_WRITE
    returnArg:
      index: 0
      type: "int"
    returnArgAction: "Post"
    selectors:
    - matchArgs:      
      - index: 0
        operator: "Prefix"
        values:
        - "/boot"           # Reads to sensitive directories
        - "/root/.ssh"      # Reads to sensitive files we want to know about
        - "/etc/shadow"
        - "/etc/profile"
        - "/etc/sudoers"
        - "/etc/pam.conf"   # Reads global shell configs bash/csh supported
        - "/etc/bashrc"
        - "/etc/csh.cshrc"
        - "/etc/csh.login"  # Add additional sensitive files here
      - index: 1
        operator: "Equal"
        values:
        - "4" # MAY_READ
   
   
   
root@k8s-ctr:~# kubectl apply -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/quickstart/file_monitoring.yaml
root@k8s-ctr:~# kubectl get tracingpolicy
NAME                       AGE
file-monitoring-filtered   12s


# 터미널2 : 이벤트를 생성하기위애 정책에 참조된 중요한(민감한) 파일을 읽어보세요
root@k8s-ctr:~# kubectl exec -ti xwing -- bash -c 'cat /etc/shadow'
root:*:19709:0:99999:7:::
daemon:*:19709:0:99999:7:::
bin:*:19709:0:99999:7:::
sys:*:19709:0:99999:7:::
sync:*:19709:0:99999:7:::
games:*:19709:0:99999:7:::
man:*:19709:0:99999:7:::
lp:*:19709:0:99999:7:::
mail:*:19709:0:99999:7:::
news:*:19709:0:99999:7:::
uucp:*:19709:0:99999:7:::
proxy:*:19709:0:99999:7:::
www-data:*:19709:0:99999:7:::
backup:*:19709:0:99999:7:::
list:*:19709:0:99999:7:::
irc:*:19709:0:99999:7:::
_apt:*:19709:0:99999:7:::
nobody:*:19709:0:99999:7:::
node:!:19710:0:99999:7:::

# 터미널 1
root@k8s-ctr:~# POD=$(kubectl -n kube-system get pods -l 'app.kubernetes.io/name=tetragon' -o name --field-selector spec.nodeName=$(kubectl get pod xwing -o jsonpath='{.spec.nodeName}'))
  process default/xwing /usr/bin/bash -c "cat /etc/shadow"
  process default/xwing /usr/bin/cat /etc/shadow
  read    default/xwing /usr/bin/cat /etc/shadow
  read    default/xwing /usr/bin/cat /etc/shadow
  read    default/xwing /usr/bin/cat /etc/shadow
  read    default/xwing /usr/bin/cat /etc/shadow
  exit    default/xwing /usr/bin/cat /etc/shadow



# 터미널2 : 이벤트를 생성하기위애 정책에 참조된 중요한(민감한) 파일을 읽어보세요
root@k8s-ctr:~# kubectl exec -ti xwing -- bash -c 'echo foo >> /etc/bar'

# 터미널 1
root@k8s-ctr:~# kubectl exec -ti -n kube-system $POD -c tetragon -- tetra getevents -o compact --pods xwing
  process default/xwing /usr/bin/bash -c "echo foo >> /etc/bar"
  write   default/xwing /usr/bin/bash /etc/bar
  write   default/xwing /usr/bin/bash /etc/bar
  exit    default/xwing /usr/bin/bash -c "echo foo >> /etc/bar"

[Cilium] Cilium Security

안녕유지 — Sun, 7 Sep 2025 18:41:24 +0900

Cloudnet Cilium 8주차 스터디를 진행하며 정리한 글입니다.

Clilium Security

쿠버네티스 기본 네트워크 정책(NetworkPolicy)은 트래픽을 제한할 수 있습니다.
하지만 여기에는 몇 가지 한계가 있습니다.

L3/L4까지만 지원 → IP와 포트 기반 제어에 머무름
Identity 개념 없음 → 파드가 재시작되어 IP가 바뀌면 정책 관리가 번거로움
애플리케이션 레벨(L7) 제어 불가 → HTTP 메서드나 경로 단위의 보안은 불가능

이러한 빈틈을 메워주는 것이 바로 Cilium Security입니다.
Cilium은 Identity 기반 보안을 중심으로, L3 → L4 → L7까지 계층적으로 보안을 확장할 수 있습니다.

Cilium vs Kubernetes NetworkPolicy

구분	Kubernetes NetworkPolicy	Cilium NetworkPolicy
지원 레벨	L3, L4	L3, L4, L7
기준	IP, Namespace, Label	Identity(Label 기반)
정책 범위	네임스페이스 단위	네임스페이스 + 클러스터 전역 (CCNP)
특징	기본적인 Pod 간 통신 제한	Envoy 프록시를 통한 L7 제어, Identity 기반 정책

Cilium을 사용하면 IP 중심 -> Identity 중심으로 보안을 전환할 수 있고, 추가로 L7까지 제어할 수 있다는 점에서 확장성이 훨씬 뛰어납니다.

Identity 기반 보안 (Layer 3)

Identity

Cilium에서 말하는 Identity는 파드나 컨테이너를 구분하는 고유한 이름표 같은 개념입니다.
이 이름표는 레이블(Label)에서 파생되며, 같은 레이블을 가진 파드들은 같은 Identity를 공유합니다.

예를 들어,

role=frontend 라벨을 가진 파드들은 모두 같은 Identity
role=backend 라벨을 가진 파드들은 또 다른 Identity

이렇게 그룹화됩니다. Identity가 만들어지는 과정은 다음과 같습니다.

파드가 생성되면 Cilium이 이를 감지하고 해당 파드를 엔드포인트로 등록합니다.
등록된 엔드포인트에 붙은 레이블을 확인해서, 그 레이블 조합에 맞는 Identity 번호를 부여합니다.
파드의 레이블이 바뀌면 → Cilium이 이를 감지 → 엔드포인트 상태를 waiting-for-identity로 전환 → 새로운 Identity를 다시 할당합니다.

즉, 레이블이 Identity의 기준점이고, 따라서 정책도 레이블 중심으로 자동 재적용됩니다.

따라서, 현재 배포 되어있는 라벨을 변경하면 identity 값이 즉시 변경된 것을 알 수 있습니다. 다만, 대규모 클러스터에서 자주 labels를 변경하면 identity 할당이 빈번해져 성능 저하가 발생할 수 있으므로, identity-relevant labels를 제한하는 것이 권장된다고 합니다.

> kubectl get ciliumendpoints.cilium.io -n kube-system                                       

NAME                              SECURITY IDENTITY   ENDPOINT STATE   IPV4           IPV6
coredns-674b8bbfcf-8qhhb          18821               ready            10.244.0.181
coredns-674b8bbfcf-xfbpx          18821               ready            10.244.0.61
hubble-relay-fdd49b976-vkxqt      16039               ready            10.244.2.66
hubble-ui-655f947f96-2dz8s        5126                ready            10.244.0.148
metrics-server-5dd7b49d79-cfgcw   8383                ready            10.244.1.107

> kubectl get ciliumidentities.cilium.io                                                    

NAME    NAMESPACE            AGE
13988   default              7d13h
16039   kube-system          7d13h
18821   kube-system          7d13h
1980    default              7d13h
25791   local-path-storage   7d13h
44434   cilium-monitoring    7d13h
50195   cilium-monitoring    7d13h
5126    kube-system          7d13h
8383    kube-system          7d13h

# coredns deployment 에 spec.template.metadata.labels에 아래 추가 시 반영 확인
# coredns가 롤링되면서 반영이 됨
> kubectl edit deploy -n kube-system coredns
...
  template:
    metadata:
      labels:
        app: testing # 추가
        k8s-app: kube-dns
        
> kubectl exec -it -n kube-system ds/cilium -- cilium identity list
ID      LABELS
1       reserved:host
        reserved:kube-apiserver
2       reserved:world
3       reserved:unmanaged
4       reserved:health
5       reserved:init
6       reserved:remote-node
7       reserved:kube-apiserver
        reserved:remote-node
8       reserved:ingress
9       reserved:world-ipv4
10      reserved:world-ipv6
1980    k8s:app=iperf3-client
        k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default
        k8s:io.cilium.k8s.policy.cluster=kind-myk8s
        k8s:io.cilium.k8s.policy.serviceaccount=default
        k8s:io.kubernetes.pod.namespace=default
5126    k8s:app.kubernetes.io/name=hubble-ui
        k8s:app.kubernetes.io/part-of=cilium
        k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system
        k8s:io.cilium.k8s.policy.cluster=kind-myk8s
        k8s:io.cilium.k8s.policy.serviceaccount=hubble-ui
        k8s:io.kubernetes.pod.namespace=kube-system
        k8s:k8s-app=hubble-ui
8383    k8s:app.kubernetes.io/instance=metrics-server
        k8s:app.kubernetes.io/name=metrics-server
        k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system
        k8s:io.cilium.k8s.policy.cluster=kind-myk8s
        k8s:io.cilium.k8s.policy.serviceaccount=metrics-server
        k8s:io.kubernetes.pod.namespace=kube-system
13988   k8s:app=iperf3-server
        k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default
        k8s:io.cilium.k8s.policy.cluster=kind-myk8s
        k8s:io.cilium.k8s.policy.serviceaccount=default
        k8s:io.kubernetes.pod.namespace=default
16039   k8s:app.kubernetes.io/name=hubble-relay
        k8s:app.kubernetes.io/part-of=cilium
        k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system
        k8s:io.cilium.k8s.policy.cluster=kind-myk8s
        k8s:io.cilium.k8s.policy.serviceaccount=hubble-relay
        k8s:io.kubernetes.pod.namespace=kube-system
        k8s:k8s-app=hubble-relay
18821   k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system
        k8s:io.cilium.k8s.policy.cluster=kind-myk8s
        k8s:io.cilium.k8s.policy.serviceaccount=coredns
        k8s:io.kubernetes.pod.namespace=kube-system
        k8s:k8s-app=kube-dns
25791   k8s:app=local-path-provisioner
        k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=local-path-storage
        k8s:io.cilium.k8s.policy.cluster=kind-myk8s
        k8s:io.cilium.k8s.policy.serviceaccount=local-path-provisioner-service-account
        k8s:io.kubernetes.pod.namespace=local-path-storage
44434   k8s:app=grafana
        k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-monitoring
        k8s:io.cilium.k8s.policy.cluster=kind-myk8s
        k8s:io.cilium.k8s.policy.serviceaccount=default
        k8s:io.kubernetes.pod.namespace=cilium-monitoring
47953   k8s:app=testing
        k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system
        k8s:io.cilium.k8s.policy.cluster=kind-myk8s
        k8s:io.cilium.k8s.policy.serviceaccount=coredns
        k8s:io.kubernetes.pod.namespace=kube-system
        k8s:k8s-app=kube-dns
50195   k8s:app=prometheus
        k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-monitoring
        k8s:io.cilium.k8s.policy.cluster=kind-myk8s
        k8s:io.cilium.k8s.policy.serviceaccount=prometheus-k8s
        k8s:io.kubernetes.pod.namespace=cilium-monitoring
57638   k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system
        k8s:io.cilium.k8s.policy.cluster=kind-myk8s
        k8s:io.cilium.k8s.policy.serviceaccount=coredns
        k8s:io.kubernetes.pod.namespace=kube-system
        k8s:k8s-app=kube-dns
        k8s:study=8w

Identity 관리

ID는 전체 클러스터에서 유효합니다. 즉, 여러 클러스터 노드에서 여러 개의 포드 또는 컨테이너가 시작되더라도 ID 관련 레이블을 공유하는 경우 모든 포드 또는 컨테이너가 단일 ID를 확인하고 공유합니다.

엔드포인트 ID를 확인하는 작업은 분산 키-값 저장소를 통해 수행됩니다.

포트 제한 (Layer 4)

L4 정책은 인바운드·아웃바운드 포트를 제한합니다.
예를 들어 frontend는 443만 나가고, backend는 443에서만 받을 수 있게 할 수 있습니다.

정책이 없을 때 → 모든 포트 허용
첫 정책이 로드되면 → 화이트리스트 모드 전환
L4 규칙이 한 번이라도 붙으면 해당 포드는 지정되지 않은 모든 포트 접근이 자동 차단 -> 운영자 입장에서는 실수로 허용 안 한 포트가 열리는 상황을 예방할 수 있는 구조

애플리케이션 프로토콜 수준 제어 (Layer 7)

Cilium은 Envoy 프록시를 통해 L7까지 제어할 수 있습니다.

특징적인 점은, 단순히 패킷을 드롭하는 게 아니라 HTTP 403 같은 거부 응답을 반환한다는 것입니다.
덕분에 운영자는 정책 때문에 막힌 건지, 네트워크 장애인지 쉽게 구분할 수 있습니다.

Cilium의 DNS 기반 정책은 DNS-IP 매핑 추적과 같은 복잡한 측면을 관리하는 동시에 액세스 제어를 쉽게 지정할 수 있는 메커니즘을 제공합니다.

DNS 기반 정책을 사용하여 클러스터 외부 서비스에 대한 이탈 액세스 제어
패턴(또는 와일드카드)을 사용하여 DNS 도메인 하위 집합을 허용 목록에 추가
외부 서비스 접근 제한을 위한 DNS, 포트 및 L7 규칙 결합

실습 1) DNS Egress 적용 - mediabot 파드가 api.github.com에만 액세스하도록 허용

실습을 위한 예제 어플리케이션을 배포하겠습니다.

root@k8s-ctr:~# cat << EOF > dns-sw-app.yaml
apiVersion: v1
kind: Pod
metadata:
  name: mediabot
  labels:
    org: empire
    class: mediabot
    app: mediabot
spec:
  containers:
  - name: mediabot
    image: quay.io/cilium/json-mock:v1.3.8@sha256:5aad04835eda9025fe4561ad31be77fd55309af8158ca8663a72f6abb78c2603
EOF

root@k8s-ctr:~# kubectl apply -f dns-sw-app.yaml
root@k8s-ctr:~# kubectl wait pod/mediabot --for=condition=Ready

root@k8s-ctr:~# kubectl exec -it -n kube-system ds/cilium -- cilium identity list                

ID      LABELS
1       reserved:host
        reserved:kube-apiserver
2       reserved:world
3       reserved:unmanaged
4       reserved:health
5       reserved:init
6       reserved:remote-node
7       reserved:kube-apiserver
        reserved:remote-node
8       reserved:ingress
9       reserved:world-ipv4
10      reserved:world-ipv6
1980    k8s:app=iperf3-client
        k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default
        k8s:io.cilium.k8s.policy.cluster=kind-myk8s
        k8s:io.cilium.k8s.policy.serviceaccount=default
        k8s:io.kubernetes.pod.namespace=default
5126    k8s:app.kubernetes.io/name=hubble-ui
        k8s:app.kubernetes.io/part-of=cilium
        k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system
        k8s:io.cilium.k8s.policy.cluster=kind-myk8s
        k8s:io.cilium.k8s.policy.serviceaccount=hubble-ui
        k8s:io.kubernetes.pod.namespace=kube-system
        k8s:k8s-app=hubble-ui
8383    k8s:app.kubernetes.io/instance=metrics-server
        k8s:app.kubernetes.io/name=metrics-server
        k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system
        k8s:io.cilium.k8s.policy.cluster=kind-myk8s
        k8s:io.cilium.k8s.policy.serviceaccount=metrics-server
        k8s:io.kubernetes.pod.namespace=kube-system
13988   k8s:app=iperf3-server
        k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default
        k8s:io.cilium.k8s.policy.cluster=kind-myk8s
        k8s:io.cilium.k8s.policy.serviceaccount=default
        k8s:io.kubernetes.pod.namespace=default
16039   k8s:app.kubernetes.io/name=hubble-relay
        k8s:app.kubernetes.io/part-of=cilium
        k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system
        k8s:io.cilium.k8s.policy.cluster=kind-myk8s
        k8s:io.cilium.k8s.policy.serviceaccount=hubble-relay
        k8s:io.kubernetes.pod.namespace=kube-system
        k8s:k8s-app=hubble-relay
18821   k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system
        k8s:io.cilium.k8s.policy.cluster=kind-myk8s
        k8s:io.cilium.k8s.policy.serviceaccount=coredns
        k8s:io.kubernetes.pod.namespace=kube-system
        k8s:k8s-app=kube-dns
25791   k8s:app=local-path-provisioner
        k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=local-path-storage
        k8s:io.cilium.k8s.policy.cluster=kind-myk8s
        k8s:io.cilium.k8s.policy.serviceaccount=local-path-provisioner-service-account
        k8s:io.kubernetes.pod.namespace=local-path-storage
44434   k8s:app=grafana
        k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-monitoring
        k8s:io.cilium.k8s.policy.cluster=kind-myk8s
        k8s:io.cilium.k8s.policy.serviceaccount=default
        k8s:io.kubernetes.pod.namespace=cilium-monitoring
47953   k8s:app=testing
        k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system
        k8s:io.cilium.k8s.policy.cluster=kind-myk8s
        k8s:io.cilium.k8s.policy.serviceaccount=coredns
        k8s:io.kubernetes.pod.namespace=kube-system
        k8s:k8s-app=kube-dns
50195   k8s:app=prometheus
        k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-monitoring
        k8s:io.cilium.k8s.policy.cluster=kind-myk8s
        k8s:io.cilium.k8s.policy.serviceaccount=prometheus-k8s
        k8s:io.kubernetes.pod.namespace=cilium-monitoring
57609   k8s:app=mediabot
        k8s:class=mediabot
        k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default
        k8s:io.cilium.k8s.policy.cluster=kind-myk8s
        k8s:io.cilium.k8s.policy.serviceaccount=default
        k8s:io.kubernetes.pod.namespace=default
        k8s:org=empire
57638   k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system
        k8s:io.cilium.k8s.policy.cluster=kind-myk8s
        k8s:io.cilium.k8s.policy.serviceaccount=coredns
        k8s:io.kubernetes.pod.namespace=kube-system
        k8s:k8s-app=kube-dns
        k8s:study=8w
        
root@k8s-ctr:~# kubectl get pods
NAME       READY   STATUS    RESTARTS   AGE
mediabot   1/1     Running   0          72s                       1/1     Running   0          75s

root@k8s-ctr:~# kubectl exec mediabot -- curl -I -s https://api.github.com | head -1
HTTP/2 200

root@k8s-ctr:~# kubectl exec mediabot -- curl -I -s --max-time 5 https://support.github.com | head -1
HTTP/2 302

mediabot 파드가 api.github.com에만 액세스하도록 허용하는 CiliumNetworkPolicy를 배포하여 확인해보겠습니다.

root@k8s-ctr:~# cat << EOF | kubectl apply -f -
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "fqdn"
spec:
  endpointSelector:
    matchLabels:
      org: empire
      class: mediabot
  egress:
  - toFQDNs:
    - matchName: "api.github.com"
  - toEndpoints:
    - matchLabels:
        "k8s:io.kubernetes.pod.namespace": kube-system
        "k8s:k8s-app": kube-dns
    toPorts:
EOF     - matchPattern: "*"
ciliumnetworkpolicy.cilium.io/fqdn created

root@k8s-ctr:~# kubectl get cnp
NAME   AGE   VALID
fqdn   18s   True

root@k8s-ctr:~# kubectl exec -it -n kube-system ds/cilium -- cilium policy selectors
SELECTOR                                                                                                                                                                      LABELS         USERS   IDENTITIES
&LabelSelector{MatchLabels:map[string]string{any.class: mediabot,any.org: empire,k8s.io.kubernetes.pod.namespace: default,},MatchExpressions:[]LabelSelectorRequirement{},}   default/fqdn   1       37240

root@k8s-ctr:~# cilium config view | grep -i dns
dnsproxy-enable-transparent-mode                  true
dnsproxy-socket-linger-timeout                    10
hubble-metrics                                    dns drop tcp flow port-distribution icmp httpV2:exemplars=true;labelsContext=source_ip,source_namespace,source_workload,destination_ip,destination_namespace,destination_workload,traffic_direction
tofqdns-dns-reject-response-code                  refused
tofqdns-enable-dns-compression                    true
tofqdns-endpoint-max-ip-per-hostname              1000
tofqdns-idle-connection-grace-period              0s
tofqdns-max-deferred-connection-deletes           10000
tofqdns-preallocate-identities                    true
tofqdns-proxy-response-max-delay                  100ms

root@k8s-ctr:~# kubectl exec mediabot -- curl -I -s https://api.github.com | head -1
HTTP/2 200

root@k8s-ctr:~# kubectl exec mediabot -- curl -I -s --max-time 5 https://support.github.com | head -1
command terminated with exit code 28

root@k8s-ctr:~# cilium hubble port-forward&
hubble observe --pod mediabot
Sep  7 10:07:36.152: kube-system/coredns-674b8bbfcf-p856r:53 (ID:48269) <> default/mediabot (ID:37240) pre-xlate-rev TRACED (UDP)
Sep  7 10:07:36.152: kube-system/kube-dns:53 (world) <> default/mediabot (ID:37240) post-xlate-rev TRANSLATED (UDP)
Sep  7 10:07:36.153: default/mediabot:44734 (ID:37240) -> kube-system/coredns-674b8bbfcf-p856r:53 (ID:48269) to-endpoint FORWARDED (UDP)
Sep  7 10:07:36.154: default/mediabot:44734 (ID:37240) <> kube-system/coredns-674b8bbfcf-p856r (ID:48269) pre-xlate-rev TRACED (UDP)
Sep  7 10:07:36.154: default/mediabot:44734 (ID:37240) <> kube-system/coredns-674b8bbfcf-p856r (ID:48269) pre-xlate-rev TRACED (UDP)
Sep  7 10:07:36.154: default/mediabot:42884 (ID:37240) <> support.github.com:443 (world) policy-verdict:none EGRESS DENIED (TCP Flags: SYN)
Sep  7 10:07:36.154: default/mediabot:44734 (ID:37240) <- kube-system/coredns-674b8bbfcf-p856r:53 (ID:48269) to-network FORWARDED (UDP)
Sep  7 10:07:36.154: default/mediabot:42884 (ID:37240) <> support.github.com:443 (world) Policy denied DROPPED (TCP Flags: SYN)
Sep  7 10:07:36.157: default/mediabot:33584 (ID:37240) -> kube-system/coredns-674b8bbfcf-8297z:53 (ID:48269) to-endpoint FORWARDED (UDP)
Sep  7 10:07:36.157: default/mediabot:33584 (ID:37240) <> kube-system/coredns-674b8bbfcf-8297z (ID:48269) pre-xlate-rev TRACED (UDP)
Sep  7 10:07:36.158: default/mediabot:33584 (ID:37240) <> kube-system/coredns-674b8bbfcf-8297z (ID:48269) pre-xlate-rev TRACED (UDP)
Sep  7 10:07:36.158: default/mediabot:33584 (ID:37240) <- kube-system/coredns-674b8bbfcf-8297z:53 (ID:48269) to-network FORWARDED (UDP)
Sep  7 10:07:36.167: default/mediabot:53861 (ID:37240) -> kube-system/coredns-674b8bbfcf-p856r:53 (ID:48269) to-endpoint FORWARDED (UDP)
Sep  7 10:07:36.167: default/mediabot:53861 (ID:37240) <> kube-system/coredns-674b8bbfcf-p856r (ID:48269) pre-xlate-rev TRACED (UDP)
Sep  7 10:07:36.168: default/mediabot:53861 (ID:37240) <> kube-system/coredns-674b8bbfcf-p856r (ID:48269) pre-xlate-rev TRACED (UDP)
Sep  7 10:07:36.186: default/mediabot:53861 (ID:37240) <- kube-system/coredns-674b8bbfcf-p856r:53 (ID:48269) to-network FORWARDED (UDP)
Sep  7 10:07:37.197: default/mediabot:42884 (ID:37240) <> support.github.com:443 (world) policy-verdict:none EGRESS DENIED (TCP Flags: SYN)
Sep  7 10:07:37.197: default/mediabot:42884 (ID:37240) <> support.github.com:443 (world) Policy denied DROPPED (TCP Flags: SYN)
Sep  7 10:07:38.221: default/mediabot:42884 (ID:37240) <> support.github.com:443 (world) policy-verdict:none EGRESS DENIED (TCP Flags: SYN)
Sep  7 10:07:38.221: default/mediabot:42884 (ID:37240) <> support.github.com:443 (world) Policy denied DROPPED (TCP Flags: SYN)
Sep  7 10:07:39.646: default/mediabot:56174 (ID:37240) <> support.github.com:443 (world) policy-verdict:none EGRESS DENIED (TCP Flags: SYN)
Sep  7 10:07:39.646: default/mediabot:56174 (ID:37240) <> support.github.com:443 (world) Policy denied DROPPED (TCP Flags: SYN)
Sep  7 10:07:40.488: default/mediabot:57742 (ID:37240) <> support.github.com:443 (world) policy-verdict:none EGRESS DENIED (TCP Flags: SYN)
Sep  7 10:07:40.488: default/mediabot:57742 (ID:37240) <> support.github.com:443 (world) Policy denied DROPPED (TCP Flags: SYN)


# 로깅 활성화
root@k8s-ctr:~# kubectl logs -n kube-system -l k8s-app=kube-dns -f
maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
.:53
[INFO] plugin/reload: Running configuration SHA512 = 1b226df79860026c6a52e67daa10d7f0d57ec5b023288ec00c5e05f93523c894564e15b91770d3a07ae1cfbe861d15b37d4a0027e69c546ab112970993a3b03b
CoreDNS-1.12.0
linux/arm64, go1.23.3, 51e11f1
[WARNING] plugin/health: Local health request to "http://:8080/health" failed: Get "http://:8080/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[WARNING] plugin/health: Local health request to "http://:8080/health" took more than 1s: 1.351010835s
.:53
[INFO] plugin/reload: Running configuration SHA512 = 1b226df79860026c6a52e67daa10d7f0d57ec5b023288ec00c5e05f93523c894564e15b91770d3a07ae1cfbe861d15b37d4a0027e69c546ab112970993a3b03b
CoreDNS-1.12.0
linux/arm64, go1.23.3, 51e11f1
[WARNING] plugin/health: Local health request to "http://:8080/health" failed: Get "http://:8080/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[WARNING] plugin/health: Local health request to "http://:8080/health" failed: Get "http://:8080/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[WARNING] plugin/health: Local health request to "http://:8080/health" failed: Get "http://:8080/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.31.2/tools/cache/reflector.go:243: watch of *v1.Service ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
[INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.31.2/tools/cache/reflector.go:243: watch of *v1.Namespace ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
[INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.31.2/tools/cache/reflector.go:243: watch of *v1.EndpointSlice ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding


# cilium 파드 및 단축키 지정
root@k8s-ctr:~# export CILIUMPOD0=$(kubectl get -l k8s-app=cilium pods -n kube-system --field-selector spec.nodeName=k8s-ctr -o jsonpath='{.items[0].metadata.name}')
root@k8s-ctr:~# export CILIUMPOD1=$(kubectl get -l k8s-app=cilium pods -n kube-system --field-selector spec.nodeName=k8s-w1  -o jsonpath='{.items[0].metadata.name}')
root@k8s-ctr:~# export CILIUMPOD2=$(kubectl get -l k8s-app=cilium pods -n kube-system --field-selector spec.nodeName=k8s-w2  -o jsonpath='{.items[0].metadata.name}')
root@k8s-ctr:~# echo $CILIUMPOD0 $CILIUMPOD1 $CILIUMPOD2

## 단축키(alias) 지정
root@k8s-ctr:~# alias c0="kubectl exec -it $CILIUMPOD0 -n kube-system -c cilium-agent -- cilium"
root@k8s-ctr:~# alias c1="kubectl exec -it $CILIUMPOD1 -n kube-system -c cilium-agent -- cilium"
root@k8s-ctr:~# alias c2="kubectl exec -it $CILIUMPOD2 -n kube-system -c cilium-agent -- cilium"

##
root@k8s-ctr:~# c0 fqdn cache list
root@k8s-ctr:~# c1 fqdn cache list
root@k8s-ctr:~# c2 fqdn cache list
Endpoint   Source       FQDN                  TTL   ExpirationTime             IPs
553        connection   api.github.com.       0     2025-09-07T10:15:25.047Z   20.200.245.245
553        connection   support.github.com.   0     2025-09-07T10:15:25.047Z   185.199.108.133
553        connection   support.github.com.   0     2025-09-07T10:15:25.047Z   185.199.110.133
553        connection   support.github.com.   0     2025-09-07T10:15:25.047Z   185.199.109.133
553        connection   support.github.com.   0     2025-09-07T10:15:25.047Z   185.199.111.133

root@k8s-ctr:~# c0 fqdn names
root@k8s-ctr:~# c1 fqdn names
root@k8s-ctr:~# c2 fqdn names
{
  "DNSPollNames": null,
  "FQDNPolicySelectors": [
    {
      "regexString": "^api[.]github[.]com[.]$",
      "selectorString": "MatchName: api.github.com, MatchPattern: "
    }
  ]
}

실습 2) DNS Egress 적용 - 모든 GitHub 하위 도메인에 액세스

root@k8s-ctr:~# kubectl delete cnp fqdn
root@k8s-ctr:~# c1 fqdn cache clean -f
FQDN proxy cache cleared
root@k8s-ctr:~# c2 fqdn cache clean -f
FQDN proxy cache cleared

root@k8s-ctr:~# cat dns-pattern.yaml
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "fqdn"
spec:
  endpointSelector:
    matchLabels:
      org: empire
      class: mediabot
  egress:
  - toFQDNs:
    - matchName: "*.github.com"
  - toEndpoints:
    - matchLabels:
        "k8s:io.kubernetes.pod.namespace": kube-system
        "k8s:k8s-app": kube-dns
    toPorts:
    - ports:
      - port: "53"
        protocol: ANY
      rules:
        dns:
        - matchPattern: "*"
        


root@k8s-ctr:~# kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.18.1/examples/kubernetes-dns/dns-pattern.yaml
root@k8s-ctr:~# c1 fqdn names
{
  "DNSPollNames": null,
  "FQDNPolicySelectors": []
}
root@k8s-ctr:~# c2 fqdn names
{
  "DNSPollNames": null,
  "FQDNPolicySelectors": [
    {
      "regexString": "^[-a-zA-Z0-9_]*[.]github[.]com[.]$",
      "selectorString": "MatchName: , MatchPattern: *.github.com"
    }
  ]
}
root@k8s-ctr:~# c1 fqdn cache list
Endpoint   Source   FQDN   TTL   ExpirationTime   IPs
root@k8s-ctr:~# c2 fqdn cache list
Endpoint   Source   FQDN   TTL   ExpirationTime   IPs

root@k8s-ctr:~# kubectl get cnp
NAME   AGE   VALID
fqdn   66s   True

root@k8s-ctr:~# kubectl exec mediabot -- curl -I -s https://support.github.com | head -1
HTTP/2 302
root@k8s-ctr:~# kubectl exec mediabot -- curl -I -s https://gist.github.com | head -1
HTTP/2 302
root@k8s-ctr:~# kubectl exec mediabot -- curl -I -s --max-time 5 https://github.com | head -1
HTTP/2 200

*.github.com 하위 도메인에 대해 통신 되는 것을 확인할 수 있습니다.

실습 3) DNS Egress 적용 - DNS, Port 조합 적용

root@k8s-ctr:~# kubectl delete cnp fqdn
ciliumnetworkpolicy.cilium.io "fqdn" deleted

root@k8s-ctr:~# dns-port.yaml
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "fqdn"
spec:
  endpointSelector:
    matchLabels:
      org: empire
      class: mediabot
  egress:
  - toFQDNs:
    - matchPattern: "*.github.com"
    toPorts:
    - ports:
      - port: "443"
        protocol: TCP
  - toEndpoints:
    - matchLabels:
        "k8s:io.kubernetes.pod.namespace": kube-system
        "k8s:k8s-app": kube-dns
    toPorts:
    - ports:
      - port: "53"
        protocol: ANY
      rules:
        dns:
        - matchPattern: "*"
        
root@k8s-ctr:~# kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.18.1/examples/kubernetes-dns/dns-port.yaml
ciliumnetworkpolicy.cilium.io/fqdn created
root@k8s-ctr:~# c1 fqdn names
{
  "DNSPollNames": null,
  "FQDNPolicySelectors": []
}
root@k8s-ctr:~# c2 fqdn names
{
  "DNSPollNames": null,
  "FQDNPolicySelectors": [
    {
      "regexString": "^[-a-zA-Z0-9_]*[.]github[.]com[.]$",
      "selectorString": "MatchName: , MatchPattern: *.github.com"
    }
  ]
}
root@k8s-ctr:~# c1 fqdn cache list
Endpoint   Source   FQDN   TTL   ExpirationTime   IPs
root@k8s-ctr:~# c2 fqdn cache list
Endpoint   Source       FQDN                  TTL   ExpirationTime             IPs
553        connection   support.github.com.   0     2025-09-07T10:32:18.053Z   185.199.109.133
553        connection   gist.github.com.      0     2025-09-07T10:32:18.053Z   20.200.245.247
553        connection   github.com.           0     2025-09-07T10:32:18.053Z   20.200.245.247
553        connection   cilium.io.            0     2025-09-07T10:32:18.053Z   104.198.14.52
553        connection   support.github.com.   0     2025-09-07T10:32:18.053Z   185.199.111.133
553        connection   support.github.com.   0     2025-09-07T10:32:18.053Z   185.199.108.133
553        connection   support.github.com.   0     2025-09-07T10:32:18.053Z   185.199.110.133
root@k8s-ctr:~# c1 fqdn cache clean -f
FQDN proxy cache cleared
root@k8s-ctr:~# c2 fqdn cache clean -f
FQDN proxy cache cleared

root@k8s-ctr:~# kubectl exec mediabot -- curl -I -s https://support.github.com | head -1
HTTP/2 302
root@k8s-ctr:~# kubectl exec mediabot -- curl -I -s --max-time 5 http://support.github.com | head -1
command terminated with exit code 28

80포트에 대해서는 dropped 되고, 443 포트에 대해서는 통신 되는 것을 확인할 수 있습니다.

[Cilium] Cilium Performance

안녕유지 — Sun, 31 Aug 2025 04:11:37 +0900

Cloudnet Cilium 7주차 스터디를 진행하며 정리한 글입니다.

이번 포스팅에서는 Cilium Metrics 대시보드를 살펴보고, 간단한 쿼리문을 확인해보겠습니다.

실습 환경 준비

kind 클러스터에 cilium cni를 비롯한 필요한 컴포넌트를 설치하겠습니다.

# Prometheus Target connection refused bind-address 설정 : kube-controller-manager , kube-scheduler , etcd , kube-proxy
> kind create cluster --name myk8s --image kindest/node:v1.33.2 --config - <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 30000
    hostPort: 30000
  - containerPort: 30001
    hostPort: 30001
  - containerPort: 30002
    hostPort: 30002
  - containerPort: 30003
    hostPort: 30003
  kubeadmConfigPatches: # Prometheus Target connection refused bind-address 설정
  - |
    kind: ClusterConfiguration
    controllerManager:
      extraArgs:
        bind-address: 0.0.0.0
    etcd:
      local:
        extraArgs:
          listen-metrics-urls: http://0.0.0.0:2381
    scheduler:
      extraArgs:
        bind-address: 0.0.0.0
  - |
    kind: KubeProxyConfiguration
    metricsBindAddress: 0.0.0.0
networking:
  disableDefaultCNI: true
  kubeProxyMode: none
  podSubnet: "10.244.0.0/16"   # cluster-cidr
kubeadmConfigPatches:
- |
  kind: ClusterConfiguration
  controllerManager:
    extraArgs:
      allocate-node-cidrs: "true"
      cluster-cidr: "10.244.0.0/16"
      node-cidr-mask-size: "22"
EOF

# node 별 PodCIDR 확인
> kubectl get nodes -o jsonpath='{.items[*].spec.podCIDR}'
10.244.0.0/22

# cilium cni 설치
> cilium install --version 1.18.1 --set ipam.mode=kubernetes --set ipv4NativeRoutingCIDR=172.20.0.0/16 \
--set routingMode=native --set autoDirectNodeRoutes=true --set endpointRoutes.enabled=true --set directRoutingSkipUnreachable=true \
--set kubeProxyReplacement=true --set bpf.masquerade=true \
--set endpointHealthChecking.enabled=false --set healthChecking=false \
--set hubble.enabled=true --set hubble.relay.enabled=true --set hubble.ui.enabled=true \
--set hubble.ui.service.type=NodePort --set hubble.ui.service.nodePort=30003 \
--set prometheus.enabled=true --set operator.prometheus.enabled=true --set envoy.prometheus.enabled=true --set hubble.metrics.enableOpenMetrics=true \
--set hubble.metrics.enabled="{dns,drop,tcp,flow,port-distribution,icmp,httpV2:exemplars=true;labelsContext=source_ip\,source_namespace\,source_workload\,destination_ip\,destination_namespace\,destination_workload\,traffic_direction}" \
--set debug.enabled=true  # --dry-run-helm-values

# hubble ui
> open http://127.0.0.1:30003


# metrics-server
> helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/
> helm upgrade --install metrics-server metrics-server/metrics-server --set 'args[0]=--kubelet-insecure-tls' -n kube-system


# Install Prometheus & Grafana 
> kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.18.1/examples/kubernetes/addons/prometheus/monitoring-example.yaml

# NodePort 설정
> kubectl patch svc -n cilium-monitoring prometheus -p '{"spec": {"type": "NodePort", "ports": [{"port": 9090, "targetPort": 9090, "nodePort": 30001}]}}'
> kubectl patch svc -n cilium-monitoring grafana -p '{"spec": {"type": "NodePort", "ports": [{"port": 3000, "targetPort": 3000, "nodePort": 30002}]}}'

쿼리문 이해하기

topk(5, avg(rate(cilium_bpf_map_ops_total{k8s_app="cilium", pod=~"$pod"}[5m])) by (pod, map_name, operation))

다음 쿼리문은 최근 5분 동안 Cilium Pod에서 발생한 BPF map 연산 중, Pod·맵·연산별로 초당 평균 연산 횟수를 계산한 뒤, 그 중 상위 5개를 보여달라는 의미를 나타냅니다.

1) cilium_bpf_map_ops_total

메트릭 이름
Cilium이 eBPF 맵(BPF map)에 대해 수행한 연산 횟수(total counter) 를 기록하는 카운터형 메트릭
라벨 정보:
- pod : 어느 Cilium pod에서 발생했는지
- map_name : 어떤 BPF map에 대한 연산인지
- operation : 연산 종류 (lookup, update, delete …)

2) {k8s_app="cilium", pod=~"$pod"}

라벨 필터링
k8s_app="cilium" : Cilium Pod에서 발생한 메트릭만 선택
pod=~"$pod" : Grafana 변수 $pod에 매칭되는 Pod만 선택 (정규식 매칭)

3) rate(...[5m])

카운터 메트릭 → 속도(초당 증가율) 변환
cilium_bpf_map_ops_total은 누적값(counter)이므로, 단순 값은 의미가 없어요.
rate(...[5m]) : 최근 5분 동안의 초당 증가율을 계산 → “초당 몇 번의 map 연산이 발생했는가” 로 해석 가능

4) avg(...) by (pod, map_name, operation)

동일한 (pod, map_name, operation) 조합별로 평균값을 집계
즉, 특정 Pod에서 어떤 map에 대해 어떤 연산이 얼마나 평균적으로 발생하는가를 구함

5) topk(5, …)

계산된 값 중 상위 5개만 출력
예: 가장 많은 연산이 일어난 BPF map 연산 5개

이 메트릭을 프로메테우스 쿼리를 보면서 확인해보겠습니다.

cilium_bpf_map_ops_total

cilium_bpf_map_ops_total{k8s_app="cilium", pod="cilium-sbnd5"}

# 최근 5분 간의 데이터로 증가율 계산 rate(cilium_bpf_map_ops_total{k8s_app="cilium"}[5m])

# 여러 시계열(metric series)의 값의 평균 avg(rate(cilium_bpf_map_ops_total{k8s_app="cilium"}[5m]))

쿠버네티스 환경에서 속도 측정 테스트

iperf3는 네트워크 성능 벤치마크 도구로, TCP/UDP 기반의 대역폭, 지터, 패킷 손실률 등을 측정할 수 있습니다.
쿠버네티스 환경에서도 Pod 간 통신 성능을 확인할 때 자주 사용됩니다.

# 배포
> cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: iperf3-server
spec:
  selector:
    matchLabels:
      app: iperf3-server
  replicas: 1
  template:
    metadata:
      labels:
        app: iperf3-server
    spec:
      containers:
      - name: iperf3-server
        image: networkstatic/iperf3
        args: ["-s"]
        ports:
        - containerPort: 5201
---
apiVersion: v1
kind: Service
metadata:
  name: iperf3-server
spec:
  selector:
    app: iperf3-server
  ports:
    - name: tcp-service
      protocol: TCP
      port: 5201
      targetPort: 5201
    - name: udp-service
      protocol: UDP
      port: 5201
      targetPort: 5201
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: iperf3-client
spec:
  selector:
    matchLabels:
      app: iperf3-client
  replicas: 1
  template:
    metadata:
      labels:
        app: iperf3-client
    spec:
      containers:
      - name: iperf3-client
        image: networkstatic/iperf3
        command: ["sleep"]
        args: ["infinity"]
EOF
deployment.apps/iperf3-server created
service/iperf3-server created
deployment.apps/iperf3-client created


# 확인 : 서버와 클라이언트가 어떤 노드에 배포되었는지 확인
kubectl get deploy,svc,pod -owide

NAME                            READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS      IMAGES                 SELECTOR
deployment.apps/iperf3-client   1/1     1            1           18s   iperf3-client   networkstatic/iperf3   app=iperf3-client
deployment.apps/iperf3-server   1/1     1            1           18s   iperf3-server   networkstatic/iperf3   app=iperf3-server

NAME                    TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)             AGE   SELECTOR
service/iperf3-server   ClusterIP   10.96.112.49   <none>        5201/TCP,5201/UDP   18s   app=iperf3-server
service/kubernetes      ClusterIP   10.96.0.1      <none>        443/TCP             15m   <none>

NAME                                 READY   STATUS    RESTARTS   AGE   IP             NODE                  NOMINATED NODE   READINESS GATES
pod/iperf3-client-688ff6565d-vj7pw   1/1     Running   0          18s   10.244.1.34    myk8s-control-plane   <none>           <none>
pod/iperf3-server-5d54b669cc-h7sl8   1/1     Running   0          18s   10.244.0.204   myk8s-control-plane   <none>           <none>

1) TCP 단방향 테스트 (기본 모드)

동작: client → server 로 TCP 스트림 1개를 5초간 송신
결과:
- 평균 ~54 Gbps 속도 확인
- Retr(재전송) 횟수가 수천 건 → 패킷 손실이나 혼잡제어로 인해 재전송 발생
- sender와 receiver 측정 값이 동일 → TCP는 신뢰성 보장 전송이므로 데이터 손실 없음

TCP는 혼잡 제어(Cwnd, Retransmit)에 따라 실제 성능이 영향을 받는다는 것을 알 수 있습니다.

# 클라이언트 파드에서 아래 명령 실행
> kubectl exec -it deploy/iperf3-client -- iperf3 -c iperf3-server -t 5

Connecting to host iperf3-server, port 5201
[  5] local 10.244.1.34 port 60780 connected to 10.96.112.49 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  6.03 GBytes  51.8 Gbits/sec  678    655 KBytes
[  5]   1.00-2.00   sec  6.30 GBytes  54.1 Gbits/sec  704   1.17 MBytes
[  5]   2.00-3.00   sec  6.42 GBytes  55.1 Gbits/sec  1205   1.16 MBytes
[  5]   3.00-4.00   sec  6.45 GBytes  55.4 Gbits/sec  628   1.04 MBytes
[  5]   4.00-5.00   sec  6.33 GBytes  54.4 Gbits/sec  516   1.05 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-5.00   sec  31.5 GBytes  54.1 Gbits/sec  3731             sender
[  5]   0.00-5.00   sec  31.5 GBytes  54.1 Gbits/sec                  receiver

iperf Done.


# 서버 파드 로그 확인 : 기본 5201 포트 Listen
> kubectl logs -l app=iperf3-server -f
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
Accepted connection from 10.244.1.34, port 60768
[  5] local 10.244.0.204 port 5201 connected to 10.244.1.34 port 60780
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  6.02 GBytes  51.8 Gbits/sec
[  5]   1.00-2.00   sec  6.30 GBytes  54.1 Gbits/sec
[  5]   2.00-3.00   sec  6.41 GBytes  55.1 Gbits/sec
[  5]   3.00-4.00   sec  6.45 GBytes  55.4 Gbits/sec
[  5]   4.00-5.00   sec  6.33 GBytes  54.4 Gbits/sec
[  5]   5.00-5.00   sec  3.62 MBytes  18.7 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-5.00   sec  31.5 GBytes  54.1 Gbits/sec                  receiver

2) UDP 사용, 역방향 모드(-R)

동작: client → server 로 UDP 트래픽을 목표 대역폭(20Gbps)으로 송신
결과:
- client 송신 기준: 3.2 Gbps 전송 성공
- server 수신 기준: 1.38 Gbps만 수신 (57% 손실)
- Jitter는 0ms 수준 → 지터는 안정적이나, 손실률이 매우 큼

UDP는 신뢰성 보장이 없어 손실이 바로 드러나는 것을 알 수 있습니다.

> kubectl exec -it deploy/iperf3-client -- iperf3 -c iperf3-server -u -b 20G

Connecting to host iperf3-server, port 5201
[  5] local 10.244.1.34 port 54994 connected to 10.96.112.49 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   397 MBytes  3.33 Gbits/sec  287487
[  5]   1.00-2.00   sec   371 MBytes  3.11 Gbits/sec  268600
[  5]   2.00-3.00   sec   323 MBytes  2.71 Gbits/sec  233792
[  5]   3.00-4.00   sec   378 MBytes  3.17 Gbits/sec  273661
[  5]   4.00-5.00   sec   372 MBytes  3.12 Gbits/sec  269267
[  5]   5.00-6.00   sec   402 MBytes  3.37 Gbits/sec  290937
[  5]   6.00-7.00   sec   409 MBytes  3.43 Gbits/sec  296220
[  5]   7.00-8.00   sec   403 MBytes  3.38 Gbits/sec  292149
[  5]   8.00-9.00   sec   405 MBytes  3.39 Gbits/sec  292948
[  5]   9.00-10.00  sec   391 MBytes  3.28 Gbits/sec  283167
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  3.76 GBytes  3.23 Gbits/sec  0.000 ms  0/2788228 (0%)  sender
[  5]   0.00-10.00  sec  1.61 GBytes  1.38 Gbits/sec  0.000 ms  1593901/2787362 (57%)  receiver

iperf Done.


> kubectl logs -l app=iperf3-server -f
-----------------------------------------------------------
Server listening on 5201 (test #2)
-----------------------------------------------------------
Accepted connection from 10.244.1.34, port 50736
[  5] local 10.244.0.204 port 5201 connected to 10.244.1.34 port 54994
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-1.00   sec   166 MBytes  1.40 Gbits/sec  0.002 ms  167003/287486 (58%)
[  5]   1.00-2.00   sec   154 MBytes  1.29 Gbits/sec  0.096 ms  157170/268436 (59%)
[  5]   2.00-3.00   sec   133 MBytes  1.12 Gbits/sec  0.143 ms  137238/233572 (59%)
[  5]   3.00-4.00   sec   162 MBytes  1.36 Gbits/sec  0.006 ms  157074/274040 (57%)
[  5]   4.00-5.00   sec   166 MBytes  1.40 Gbits/sec  0.000 ms  148829/269271 (55%)
[  5]   5.00-6.00   sec   173 MBytes  1.45 Gbits/sec  0.034 ms  165514/290920 (57%)
[  5]   6.00-7.00   sec   175 MBytes  1.47 Gbits/sec  0.208 ms  168177/294898 (57%)
[  5]   7.00-8.00   sec   177 MBytes  1.48 Gbits/sec  0.191 ms  164541/292408 (56%)
[  5]   8.00-9.00   sec   180 MBytes  1.50 Gbits/sec  0.189 ms  163227/293238 (56%)
[  5]   9.00-10.00  sec   163 MBytes  1.37 Gbits/sec  0.000 ms  165128/283093 (58%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  1.61 GBytes  1.38 Gbits/sec  0.000 ms  1593901/2787362 (57%)  receiver

3) TCP, 쌍방향 모드(-R)

동작: client→server, server→client 양쪽으로 동시에 TCP 스트림 송수신
결과:
- client 송신(TX-C): 평균 ~14.8 Gbps
- client 수신(RX-C): 평균 ~14.4 Gbps
- server 송신(TX-S): 평균 ~14.4 Gbps
- server 수신(RX-S): 평균 ~14.8 Gbps
- 총합 약 29 Gbps (14~15 Gbps × 2 방향)

양방향 모드에서는 링크 대역폭이 절반 수준으로 분배되며, 양쪽 송수신에서 Retr(재전송)이 발생할 수 있다는 것을 알 수 있습니다.

> kubectl exec -it deploy/iperf3-client -- iperf3 -c iperf3-server -t 5 --bidir

Connecting to host iperf3-server, port 5201
[  5] local 10.244.1.34 port 57120 connected to 10.96.112.49 port 5201
[  7] local 10.244.1.34 port 57130 connected to 10.96.112.49 port 5201
[ ID][Role] Interval           Transfer     Bitrate         Retr  Cwnd
[  5][TX-C]   0.00-1.00   sec  3.05 GBytes  26.2 Gbits/sec  2540    806 KBytes
[  7][RX-C]   0.00-1.00   sec  2.77 GBytes  23.8 Gbits/sec
[  5][TX-C]   1.00-2.00   sec  2.24 GBytes  19.2 Gbits/sec  1286    707 KBytes
[  7][RX-C]   1.00-2.00   sec  2.10 GBytes  18.0 Gbits/sec
[  5][TX-C]   2.00-3.00   sec  1.02 GBytes  8.76 Gbits/sec  431   1.06 MBytes
[  7][RX-C]   2.00-3.00   sec  1.22 GBytes  10.5 Gbits/sec
[  5][TX-C]   3.00-4.00   sec  1.48 GBytes  12.7 Gbits/sec  535    711 KBytes
[  7][RX-C]   3.00-4.00   sec  1.30 GBytes  11.2 Gbits/sec
[  5][TX-C]   4.00-5.00   sec   852 MBytes  7.15 Gbits/sec  552    344 KBytes
[  7][RX-C]   4.00-5.00   sec   999 MBytes  8.38 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-5.00   sec  8.62 GBytes  14.8 Gbits/sec  5344             sender
[  5][TX-C]   0.00-5.00   sec  8.62 GBytes  14.8 Gbits/sec                  receiver
[  7][RX-C]   0.00-5.00   sec  8.37 GBytes  14.4 Gbits/sec  1942             sender
[  7][RX-C]   0.00-5.00   sec  8.37 GBytes  14.4 Gbits/sec                  receiver

iperf Done.


> kubectl logs -l app=iperf3-server -f
-----------------------------------------------------------
Server listening on 5201 (test #3)
-----------------------------------------------------------
Accepted connection from 10.244.1.34, port 57108
[  5] local 10.244.0.204 port 5201 connected to 10.244.1.34 port 57120
[  8] local 10.244.0.204 port 5201 connected to 10.244.1.34 port 57130
[ ID][Role] Interval           Transfer     Bitrate         Retr  Cwnd
[  5][RX-S]   0.00-1.00   sec  3.05 GBytes  26.2 Gbits/sec
[  8][TX-S]   0.00-1.00   sec  2.77 GBytes  23.8 Gbits/sec  396    960 KBytes
[  5][RX-S]   1.00-2.01   sec  2.23 GBytes  19.1 Gbits/sec
[  8][TX-S]   1.00-2.01   sec  2.10 GBytes  17.9 Gbits/sec  516    745 KBytes
[  5][RX-S]   2.01-3.00   sec  1.02 GBytes  8.83 Gbits/sec
[  8][TX-S]   2.01-3.00   sec  1.22 GBytes  10.5 Gbits/sec   67    911 KBytes
[  5][RX-S]   3.00-4.00   sec  1.48 GBytes  12.7 Gbits/sec
[  8][TX-S]   3.00-4.00   sec  1.30 GBytes  11.2 Gbits/sec  175    807 KBytes
[  5][RX-S]   4.00-5.00   sec   856 MBytes  7.18 Gbits/sec
[  8][TX-S]   4.00-5.00   sec   996 MBytes  8.35 Gbits/sec  788    732 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][RX-S]   0.00-5.00   sec  8.62 GBytes  14.8 Gbits/sec                  receiver
[  8][TX-S]   0.00-5.00   sec  8.37 GBytes  14.4 Gbits/sec  1942             sender

4) TCP 다중 스트림(30개), -P(number of parallel client streams to run)

동작: TCP 스트림 2개를 병렬 실행 (멀티 커넥션 상황)
결과:
- 각 스트림은 ~32 Gbps → 합산 64 Gbps 근처
- Retr(재전송) 횟수는 개별 스트림에서 많음 (3천 건/초)
- 서버 로그에서도 합산 64 Gbps 확인

단일 스트림으로는 대역폭을 다 못 쓰는 경우가 많기 때문에 멀티 스트림(-P 옵션)을 주면 네트워크 스택/큐 병렬화로 성능이 더 높게 나오는 것을 알 수 있습니다.

> kubectl exec -it deploy/iperf3-client -- iperf3 -c iperf3-server -t 10 -P 2

Connecting to host iperf3-server, port 5201
[  5] local 10.244.1.34 port 54498 connected to 10.96.112.49 port 5201
[  7] local 10.244.1.34 port 54512 connected to 10.96.112.49 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  3.91 GBytes  33.6 Gbits/sec  676    745 KBytes
[  7]   0.00-1.00   sec  3.86 GBytes  33.1 Gbits/sec  1002    700 KBytes
[SUM]   0.00-1.00   sec  7.77 GBytes  66.7 Gbits/sec  1678
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec  3.80 GBytes  32.6 Gbits/sec  727    467 KBytes
[  7]   1.00-2.00   sec  3.78 GBytes  32.5 Gbits/sec  1491    680 KBytes
[SUM]   1.00-2.00   sec  7.58 GBytes  65.1 Gbits/sec  2218
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec  3.89 GBytes  33.4 Gbits/sec  2109    471 KBytes
[  7]   2.00-3.00   sec  3.86 GBytes  33.2 Gbits/sec  962    462 KBytes
[SUM]   2.00-3.00   sec  7.76 GBytes  66.6 Gbits/sec  3071
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec  3.38 GBytes  29.1 Gbits/sec  2651    369 KBytes
[  7]   3.00-4.00   sec  3.34 GBytes  28.7 Gbits/sec  1341    539 KBytes
[SUM]   3.00-4.00   sec  6.72 GBytes  57.7 Gbits/sec  3992
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec  3.65 GBytes  31.4 Gbits/sec  1422    537 KBytes
[  7]   4.00-5.00   sec  3.66 GBytes  31.4 Gbits/sec  734    465 KBytes
[SUM]   4.00-5.00   sec  7.31 GBytes  62.8 Gbits/sec  2156
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec  3.81 GBytes  32.7 Gbits/sec  1923    503 KBytes
[  7]   5.00-6.00   sec  3.79 GBytes  32.5 Gbits/sec  902    392 KBytes
[SUM]   5.00-6.00   sec  7.60 GBytes  65.3 Gbits/sec  2825
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec  3.71 GBytes  31.9 Gbits/sec  1239    515 KBytes
[  7]   6.00-7.00   sec  3.70 GBytes  31.8 Gbits/sec  830    322 KBytes
[SUM]   6.00-7.00   sec  7.41 GBytes  63.7 Gbits/sec  2069
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec  3.79 GBytes  32.5 Gbits/sec  1020    481 KBytes
[  7]   7.00-8.00   sec  3.70 GBytes  31.8 Gbits/sec  1359    494 KBytes
[SUM]   7.00-8.00   sec  7.49 GBytes  64.3 Gbits/sec  2379
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec  4.14 GBytes  35.6 Gbits/sec  1995    378 KBytes
[  7]   8.00-9.00   sec  4.09 GBytes  35.2 Gbits/sec  824    601 KBytes
[SUM]   8.00-9.00   sec  8.23 GBytes  70.8 Gbits/sec  2819
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec  3.41 GBytes  29.3 Gbits/sec  1534    617 KBytes
[  7]   9.00-10.00  sec  3.34 GBytes  28.7 Gbits/sec  1203    359 KBytes
[SUM]   9.00-10.00  sec  6.75 GBytes  58.0 Gbits/sec  2737
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  37.5 GBytes  32.2 Gbits/sec  15296             sender
[  5]   0.00-10.00  sec  37.5 GBytes  32.2 Gbits/sec                  receiver
[  7]   0.00-10.00  sec  37.1 GBytes  31.9 Gbits/sec  10648             sender
[  7]   0.00-10.00  sec  37.1 GBytes  31.9 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  74.6 GBytes  64.1 Gbits/sec  25944             sender
[SUM]   0.00-10.00  sec  74.6 GBytes  64.1 Gbits/sec                  receiver

iperf Done.


> kubectl logs -l app=iperf3-server -f 
-----------------------------------------------------------
Server listening on 5201 (test #4)
-----------------------------------------------------------
Accepted connection from 10.244.1.34, port 54488
[  5] local 10.244.0.204 port 5201 connected to 10.244.1.34 port 54498
[  8] local 10.244.0.204 port 5201 connected to 10.244.1.34 port 54512
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  3.90 GBytes  33.5 Gbits/sec
[  8]   0.00-1.00   sec  3.85 GBytes  33.1 Gbits/sec
[SUM]   0.00-1.00   sec  7.75 GBytes  66.6 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec  3.80 GBytes  32.6 Gbits/sec
[  8]   1.00-2.00   sec  3.78 GBytes  32.5 Gbits/sec
[SUM]   1.00-2.00   sec  7.57 GBytes  65.1 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec  3.89 GBytes  33.5 Gbits/sec
[  8]   2.00-3.00   sec  3.86 GBytes  33.2 Gbits/sec
[SUM]   2.00-3.00   sec  7.75 GBytes  66.6 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec  3.38 GBytes  29.1 Gbits/sec
[  8]   3.00-4.00   sec  3.34 GBytes  28.7 Gbits/sec
[SUM]   3.00-4.00   sec  6.72 GBytes  57.7 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec  3.65 GBytes  31.4 Gbits/sec
[  8]   4.00-5.00   sec  3.66 GBytes  31.4 Gbits/sec
[SUM]   4.00-5.00   sec  7.31 GBytes  62.8 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec  3.81 GBytes  32.7 Gbits/sec
[  8]   5.00-6.00   sec  3.79 GBytes  32.5 Gbits/sec
[SUM]   5.00-6.00   sec  7.60 GBytes  65.3 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec  3.71 GBytes  31.9 Gbits/sec
[  8]   6.00-7.00   sec  3.70 GBytes  31.8 Gbits/sec
[SUM]   6.00-7.00   sec  7.41 GBytes  63.7 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec  3.79 GBytes  32.5 Gbits/sec
[  8]   7.00-8.00   sec  3.70 GBytes  31.8 Gbits/sec
[SUM]   7.00-8.00   sec  7.49 GBytes  64.3 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec  4.14 GBytes  35.6 Gbits/sec
[  8]   8.00-9.00   sec  4.09 GBytes  35.2 Gbits/sec
[SUM]   8.00-9.00   sec  8.24 GBytes  70.7 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec  3.41 GBytes  29.3 Gbits/sec
[  8]   9.00-10.00  sec  3.34 GBytes  28.7 Gbits/sec
[SUM]   9.00-10.00  sec  6.75 GBytes  58.0 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  10.00-10.00  sec  2.62 MBytes  27.8 Gbits/sec
[  8]  10.00-10.00  sec  2.62 MBytes  29.2 Gbits/sec
[SUM]  10.00-10.00  sec  5.25 MBytes  55.5 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  37.5 GBytes  32.2 Gbits/sec                  receiver
[  8]   0.00-10.00  sec  37.1 GBytes  31.9 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  74.6 GBytes  64.1 Gbits/sec                  receiver