스터디/Cilium

[Cilium] L2 Announcements

안녕유지 2025. 8. 10. 04:22
Cloudnet Cilium 4주차 스터디를 진행하며 정리한 글입니다.

 

이번 포스팅에서는 Cilium Layer 2 Announcement를 활용한 LoadBalancer External IP 외부 노출에 대해 알아보겠습니다.

Cilium Layer 2 Announcement using ARP

Kubernetes 환경에서 LoadBalancer 타입 서비스의 External IP를 외부 네트워크에서 접근하려면, 일반적으로 BGP 광고를 사용하거나 클라우드 로드밸런서가 필요합니다. 하지만 온프레미스나 사무실·캠퍼스 네트워크처럼 BGP 라우팅이 없는 환경에서는 이 방식이 불가능합니다.

Cilium은 이를 해결하기 위해 Layer 2 Announcement 기능을 제공합니다.
이 기능을 사용하면 External IP 또는 LoadBalancer IP에 대해 ARP(IPv4) / NDP(IPv6) 응답을 보내, 같은 L2 네트워크 내에서 해당 IP가 존재하는 것처럼 동작시킬 수 있습니다.

 

https://isovalent.com/blog/post/migrating-from-metallb-to-cilium/#l2-announcement-with-arp

 

 

root@k8s-ctr:~# arping -i eth1 $LBIP -c 100000
ARPING 192.168.10.211
Timeout
Timeout
Timeout
Timeout
^C
--- 192.168.10.211 statistics ---
5 packets transmitted, 0 packets received, 100% unanswered (0 extra)

# 설정 업그레이드
root@k8s-ctr:~# helm upgrade cilium cilium/cilium --namespace kube-system --version 1.18.0 --reuse-values \
   --set l2announcements.enabled=true
Release "cilium" has been upgraded. Happy Helming!
NAME: cilium
LAST DEPLOYED: Sun Aug 10 04:14:07 2025
NAMESPACE: kube-system
STATUS: deployed
REVISION: 4
TEST SUITE: None
NOTES:
You have successfully installed Cilium with Hubble Relay and Hubble UI.

Your release version is 1.18.0.

For any further help, visit https://docs.cilium.io/en/v1.18/gettinghelp

# 확인
root@k8s-ctr:~# kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg config --all | grep EnableL2Announcements
EnableL2Announcements             : true

root@k8s-ctr:~# cilium config view | grep enable-l2
enable-l2-announcements                           true
enable-l2-neigh-discovery                         false

 

root@k8s-ctr:~# cat << EOF | kubectl apply -f -
apiVersion: "cilium.io/v2alpha1"  # not v2
kind: CiliumL2AnnouncementPolicy
metadata:
  name: policy1
spec:
  serviceSelector:
    matchLabels:
      app: webpod
  nodeSelector:
    matchExpressions:
      - key: kubernetes.io/hostname
        operator: NotIn
        values:
          - k8s-w0
EOFoadBalancerIPs: true
ciliuml2announcementpolicy.cilium.io/policy1 created

root@k8s-ctr:~# kubectl -n kube-system get lease | grep "cilium-l2announce"
cilium-l2announce-default-webpod       k8s-w1                                                                      26s


root@k8s-ctr:~# kubectl -n kube-system get lease/cilium-l2announce-default-webpod -o yaml | yq
{
  "apiVersion": "coordination.k8s.io/v1",
  "kind": "Lease",
  "metadata": {
    "creationTimestamp": "2025-08-09T19:16:22Z",
    "name": "cilium-l2announce-default-webpod",
    "namespace": "kube-system",
    "resourceVersion": "23436",
    "uid": "9ec94298-dc6f-41e3-b08d-66b41c31a5d6"
  },
  "spec": {
    "acquireTime": "2025-08-09T19:16:22.558405Z",
    "holderIdentity": "k8s-w1",
    "leaseDurationSeconds": 15,
    "leaseTransitions": 0,
    "renewTime": "2025-08-09T19:17:08.431817Z"
  }
}

 

# cilium 파드 이름 지정
root@k8s-ctr:~# export CILIUMPOD0=$(kubectl get -l k8s-app=cilium pods -n kube-system --field-selector spec.nodeName=k8s-ctr -o jsonpath='{.items[0].metadata.name}')
root@k8s-ctr:~# export CILIUMPOD1=$(kubectl get -l k8s-app=cilium pods -n kube-system --field-selector spec.nodeName=k8s-w1  -o jsonpath='{.items[0].metadata.name}')
root@k8s-ctr:~# export CILIUMPOD2=$(kubectl get -l k8s-app=cilium pods -n kube-system --field-selector spec.nodeName=k8s-w0  -o jsonpath='{.items[0].metadata.name}')
root@k8s-ctr:~# echo $CILIUMPOD0 $CILIUMPOD1 $CILIUMPOD2
cilium-k94qs cilium-p5gfr cilium-kcgnf

# 현재 해당 IP에 대한 리더가 위치한 노드의 cilium-agent 파드 내에서 정보 확인
root@k8s-ctr:~# kubectl exec -n kube-system $CILIUMPOD0 -- cilium-dbg shell -- db/show l2-announce
IP   NetworkInterface
root@k8s-ctr:~# kubectl exec -n kube-system $CILIUMPOD1 -- cilium-dbg shell -- db/show l2-announce
IP               NetworkInterface
192.168.10.211   eth1
root@k8s-ctr:~# kubectl exec -n kube-system $CILIUMPOD2 -- cilium-dbg shell -- db/show l2-announce
IP   NetworkInterface


root@k8s-ctr:~# kubectl -n kube-system logs ds/cilium | grep "l2"

Found 3 pods, using pod/cilium-kcgnf
time=2025-08-09T19:13:59.476909317Z level=info source=/go/src/github.com/cilium/cilium/pkg/option/config.go:1206 msg="  --enable-l2-announcements='true'"
time=2025-08-09T19:13:59.476917651Z level=info source=/go/src/github.com/cilium/cilium/pkg/option/config.go:1206 msg="  --enable-l2-neigh-discovery='false'"
time=2025-08-09T19:13:59.476926734Z level=info source=/go/src/github.com/cilium/cilium/pkg/option/config.go:1206 msg="  --enable-l2-pod-announcements='false'"
time=2025-08-09T19:13:59.485008526Z level=info source=/go/src/github.com/cilium/cilium/pkg/option/config.go:1206 msg="  --l2-announcements-lease-duration='15s'"
time=2025-08-09T19:13:59.485079067Z level=info source=/go/src/github.com/cilium/cilium/pkg/option/config.go:1206 msg="  --l2-announcements-renew-deadline='5s'"
time=2025-08-09T19:13:59.485098859Z level=info source=/go/src/github.com/cilium/cilium/pkg/option/config.go:1206 msg="  --l2-announcements-retry-period='2s'"
time=2025-08-09T19:13:59.485111442Z level=info source=/go/src/github.com/cilium/cilium/pkg/option/config.go:1208 msg="  --l2-pod-announcements-interface=''"
time=2025-08-09T19:13:59.485124651Z level=info source=/go/src/github.com/cilium/cilium/pkg/option/config.go:1208 msg="  --l2-pod-announcements-interface-pattern=''"
time=2025-08-09T19:14:00.488790443Z level=debug source=/go/src/github.com/cilium/cilium/vendor/github.com/cilium/hive/cell/invoke.go:38 msg=Invoking function="l2announcer.init.func1 (pkg/l2announcer/l2announcer.go:51)"
time=2025-08-09T19:14:00.491883901Z level=debug source=/go/src/github.com/cilium/cilium/vendor/github.com/cilium/hive/cell/invoke.go:59 msg=Invoked duration=2.94525ms function="l2announcer.init.func1 (pkg/l2announcer/l2announcer.go:51)"
time=2025-08-09T19:14:00.538191651Z level=debug source=/go/src/github.com/cilium/cilium/vendor/github.com/cilium/hive/cell/invoke.go:38 msg=Invoking function="l2responder.NewL2ResponderReconciler (.../datapath/l2responder/l2responder.go:74)"
time=2025-08-09T19:14:00.538347026Z level=debug source=/go/src/github.com/cilium/cilium/vendor/github.com/cilium/hive/cell/invoke.go:59 msg=Invoked duration=145.5µs function="l2responder.NewL2ResponderReconciler (.../datapath/l2responder/l2responder.go:74)"
time=2025-08-09T19:14:00.629093693Z level=debug source=/go/src/github.com/cilium/cilium/pkg/k8s/synced/resources.go:96 msg="waiting for cache to synchronize" module=agent.controlplane.k8s-synced resource=crd:ciliuml2announcementpolicies.cilium.io
time=2025-08-09T19:14:00.679605735Z level=debug source=/go/src/github.com/cilium/cilium/pkg/k8s/synced/resources.go:101 msg="canceled cache synchronization" module=agent.controlplane.k8s-synced resource=crd:ciliuml2announcementpolicies.cilium.io
time=2025-08-09T19:14:01.957104985Z level=debug source=/go/src/github.com/cilium/cilium/vendor/github.com/cilium/hive/cell/lifecycle.go:126 msg="Executing start hook" function="*resource.resource[*github.com/cilium/cilium/pkg/k8s/apis/cilium.io/v2alpha1.CiliumL2AnnouncementPolicy].Start (agent.controlplane.l2-announcer)"
time=2025-08-09T19:14:01.957127152Z level=debug source=/go/src/github.com/cilium/cilium/vendor/github.com/cilium/hive/cell/lifecycle.go:136 msg="Start hook executed" function="*resource.resource[*github.com/cilium/cilium/pkg/k8s/apis/cilium.io/v2alpha1.CiliumL2AnnouncementPolicy].Start (agent.controlplane.l2-announcer)" duration=8.125µs
time=2025-08-09T19:14:01.957145777Z level=debug source=/go/src/github.com/cilium/cilium/vendor/github.com/cilium/hive/cell/lifecycle.go:126 msg="Executing start hook" function="*job.groupHooks.Start (agent.controlplane.l2-announcer)"
time=2025-08-09T19:14:01.957153235Z level=debug source=/go/src/github.com/cilium/cilium/vendor/github.com/cilium/hive/cell/lifecycle.go:136 msg="Start hook executed" function="*job.groupHooks.Start (agent.controlplane.l2-announcer)" duration=708ns
time=2025-08-09T19:14:01.957250485Z level=debug source=/go/src/github.com/cilium/cilium/vendor/github.com/cilium/hive/cell/lifecycle.go:126 msg="Executing start hook" function="*job.queuedJob.Start (agent.controlplane.l2-announcer)"
time=2025-08-09T19:14:01.957303277Z level=debug source=/go/src/github.com/cilium/cilium/vendor/github.com/cilium/hive/cell/lifecycle.go:136 msg="Start hook executed" function="*job.queuedJob.Start (agent.controlplane.l2-announcer)" duration=2.083µs
time=2025-08-09T19:14:01.957314527Z level=debug source=/go/src/github.com/cilium/cilium/vendor/github.com/cilium/hive/cell/lifecycle.go:126 msg="Executing start hook" function="*job.queuedJob.Start (agent.controlplane.l2-announcer)"
time=2025-08-09T19:14:01.957320693Z level=debug source=/go/src/github.com/cilium/cilium/vendor/github.com/cilium/hive/cell/lifecycle.go:136 msg="Start hook executed" function="*job.queuedJob.Start (agent.controlplane.l2-announcer)" duration=1.083µs
time=2025-08-09T19:14:01.992649777Z level=debug source=/go/src/github.com/cilium/cilium/vendor/github.com/cilium/hive/job/oneshot.go:134 msg="Starting one-shot job" module=agent.controlplane.l2-announcer name=l2-announcer-run func=l2announcer.(*L2Announcer).run
time=2025-08-09T19:14:01.994295693Z level=debug source=/go/src/github.com/cilium/cilium/vendor/github.com/cilium/hive/job/timer.go:172 msg="Starting timer job" module=agent.controlplane.l2-announcer name=l2-announcer-lease-gc func="l2announcer.NewL2Announcer.func1 (pkg/l2announcer/l2announcer.go:131)"
time=2025-08-09T19:14:02.006441777Z level=debug source=/go/src/github.com/cilium/cilium/pkg/hive/health/provider.go:89 msg="upserting health status" module=health lastLevel=none reporter-id=agent.controlplane.l2-announcer.job-l2-announcer-run status="agent.controlplane.l2-announcer.job-l2-announcer-run: [OK] Running"
time=2025-08-09T19:14:02.006651777Z level=debug source=/go/src/github.com/cilium/cilium/pkg/hive/health/provider.go:89 msg="upserting health status" module=health lastLevel=none reporter-id=agent.controlplane.l2-announcer.timer-job-l2-announcer-lease-gc status="agent.controlplane.l2-announcer.timer-job-l2-announcer-lease-gc: [OK] Primed"
time=2025-08-09T19:14:02.015368277Z level=debug source=/go/src/github.com/cilium/cilium/vendor/github.com/cilium/hive/cell/lifecycle.go:126 msg="Executing start hook" function="l2respondermap.newMap.func1 (.../maps/l2respondermap/l2_responder_map4.go:46) (agent.datapath.maps)"
time=2025-08-09T19:14:02.015685485Z level=debug source=/go/src/github.com/cilium/cilium/pkg/ebpf/map_register.go:22 msg="Registered BPF map" module=agent.datapath.maps file-path=/sys/fs/bpf/tc/globals/cilium_l2_responder_v4
time=2025-08-09T19:14:02.015774027Z level=debug source=/go/src/github.com/cilium/cilium/vendor/github.com/cilium/hive/cell/lifecycle.go:136 msg="Start hook executed" function="l2respondermap.newMap.func1 (.../maps/l2respondermap/l2_responder_map4.go:46) (agent.datapath.maps)" duration=376.875µs
time=2025-08-09T19:14:02.015794402Z level=debug source=/go/src/github.com/cilium/cilium/vendor/github.com/cilium/hive/cell/lifecycle.go:126 msg="Executing start hook" function="*job.groupHooks.Start (agent.datapath.l2-responder)"
time=2025-08-09T19:14:02.015816235Z level=debug source=/go/src/github.com/cilium/cilium/vendor/github.com/cilium/hive/cell/lifecycle.go:136 msg="Start hook executed" function="*job.groupHooks.Start (agent.datapath.l2-responder)" duration=1.084µs
time=2025-08-09T19:14:02.01583436Z level=debug source=/go/src/github.com/cilium/cilium/vendor/github.com/cilium/hive/cell/lifecycle.go:126 msg="Executing start hook" function="*job.queuedJob.Start (agent.datapath.l2-responder)"
time=2025-08-09T19:14:02.015845027Z level=debug source=/go/src/github.com/cilium/cilium/vendor/github.com/cilium/hive/cell/lifecycle.go:136 msg="Start hook executed" function="*job.queuedJob.Start (agent.datapath.l2-responder)" duration=3.333µs
time=2025-08-09T19:14:02.015857735Z level=debug source=/go/src/github.com/cilium/cilium/vendor/github.com/cilium/hive/cell/lifecycle.go:126 msg="Executing start hook" function="*job.groupHooks.Start (agent.datapath.l2-pod-announcements-garp)"
time=2025-08-09T19:14:02.015864818Z level=debug source=/go/src/github.com/cilium/cilium/vendor/github.com/cilium/hive/cell/lifecycle.go:136 msg="Start hook executed" function="*job.groupHooks.Start (agent.datapath.l2-pod-announcements-garp)" duration=416ns
time=2025-08-09T19:14:02.042840819Z level=debug source=/go/src/github.com/cilium/cilium/vendor/github.com/cilium/hive/job/oneshot.go:134 msg="Starting one-shot job" module=agent.datapath.l2-responder name=l2-responder-reconciler func=l2responder.(*l2ResponderReconciler).run
time=2025-08-09T19:14:02.042989152Z level=debug source=/go/src/github.com/cilium/cilium/pkg/hive/health/provider.go:89 msg="upserting health status" module=health lastLevel=none reporter-id=agent.datapath.l2-responder.job-l2-responder-reconciler status="agent.datapath.l2-responder.job-l2-responder-reconciler: [OK] Running"
time=2025-08-09T19:14:02.049619819Z level=debug source=/go/src/github.com/cilium/cilium/pkg/datapath/l2responder/l2responder.go:195 msg="l2 announcer table full reconciliation" module=agent.datapath.l2-responder
time=2025-08-09T19:15:01.99483443Z level=debug source=/go/src/github.com/cilium/cilium/vendor/github.com/cilium/hive/job/timer.go:184 msg="Timer job triggered" module=agent.controlplane.l2-announcer name=l2-announcer-lease-gc func="l2announcer.NewL2Announcer.func1 (pkg/l2announcer/l2announcer.go:131)"
time=2025-08-09T19:15:02.02308618Z level=debug source=/go/src/github.com/cilium/cilium/vendor/github.com/cilium/hive/job/timer.go:200 msg="Timer job finished" module=agent.controlplane.l2-announcer name=l2-announcer-lease-gc func="l2announcer.NewL2Announcer.func1 (pkg/l2announcer/l2announcer.go:131)"
time=2025-08-09T19:16:01.994835126Z level=debug source=/go/src/github.com/cilium/cilium/vendor/github.com/cilium/hive/job/timer.go:184 msg="Timer job triggered" module=agent.controlplane.l2-announcer name=l2-announcer-lease-gc func="l2announcer.NewL2Announcer.func1 (pkg/l2announcer/l2announcer.go:131)"
time=2025-08-09T19:16:02.003833126Z level=debug source=/go/src/github.com/cilium/cilium/vendor/github.com/cilium/hive/job/timer.go:200 msg="Timer job finished" module=agent.controlplane.l2-announcer name=l2-announcer-lease-gc func="l2announcer.NewL2Announcer.func1 (pkg/l2announcer/l2announcer.go:131)"
time=2025-08-09T19:17:01.995790279Z level=debug source=/go/src/github.com/cilium/cilium/vendor/github.com/cilium/hive/job/timer.go:184 msg="Timer job triggered" module=agent.controlplane.l2-announcer name=l2-announcer-lease-gc func="l2announcer.NewL2Announcer.func1 (pkg/l2announcer/l2announcer.go:131)"
time=2025-08-09T19:17:02.044427529Z level=debug source=/go/src/github.com/cilium/cilium/vendor/github.com/cilium/hive/job/timer.go:200 msg="Timer job finished" module=agent.controlplane.l2-announcer name=l2-announcer-lease-gc func="l2announcer.NewL2Announcer.func1 (pkg/l2announcer/l2announcer.go:131)"
time=2025-08-09T19:18:01.995064891Z level=debug source=/go/src/github.com/cilium/cilium/vendor/github.com/cilium/hive/job/timer.go:184 msg="Timer job triggered" module=agent.controlplane.l2-announcer name=l2-announcer-lease-gc func="l2announcer.NewL2Announcer.func1 (pkg/l2announcer/l2announcer.go:131)"
time=2025-08-09T19:18:02.006271641Z level=debug source=/go/src/github.com/cilium/cilium/vendor/github.com/cilium/hive/job/timer.go:200 msg="Timer job finished" module=agent.controlplane.l2-announcer name=l2-announcer-lease-gc func="l2announcer.NewL2Announcer.func1 (pkg/l2announcer/l2announcer.go:131)"

 

# router VM : LBIP로 curl 요청 확인
root@router:~# curl --connect-timeout 1 $LBIP
Hostname: webpod-697b545f57-9ns5w
IP: 127.0.0.1
IP: ::1
IP: 172.20.2.152
IP: fe80::a43c:f7ff:fe42:d57a
RemoteAddr: 172.20.1.179:39190
GET / HTTP/1.1
Host: 192.168.10.211
User-Agent: curl/8.5.0
Accept: */*

root@router:~# arp -a
? (192.168.20.100) at 08:00:27:51:27:06 [ether] on eth2
? (192.168.10.101) at 08:00:27:81:5f:e3 [ether] on eth1
? (192.168.10.211) at 08:00:27:81:5f:e3 [ether] on eth1
_gateway (10.0.2.2) at 52:55:0a:00:02:02 [ether] on eth0
? (10.0.2.3) at 52:55:0a:00:02:03 [ether] on eth0
? (192.168.10.100) at 08:00:27:9b:e0:6f [ether] on eth1

root@router:~# curl -s $LBIP
Hostname: webpod-697b545f57-9nnfd
IP: 127.0.0.1
IP: ::1
IP: 172.20.1.164
IP: fe80::e81d:79ff:fe56:7550
RemoteAddr: 192.168.10.200:40722
GET / HTTP/1.1
Host: 192.168.10.211
User-Agent: curl/8.5.0
Accept: */*

root@router:~# curl -s $LBIP | grep Hostname
Hostname: webpod-697b545f57-9ns5w
root@router:~# curl -s $LBIP | grep RemoteAddr
RemoteAddr: 172.20.1.179:54962
# 리더 노드가 아닌 다른 노드에 webpod 통신 시, SNAT 됨 : arp 동작(리더 노드)으로 인한 제약 사항
root@router:~# while true; do curl -s --connect-timeout 1 $LBIP | grep Hostname; sleep 0.1; done
Hostname: webpod-697b545f57-9nnfd
Hostname: webpod-697b545f57-mn7r5
Hostname: webpod-697b545f57-mn7r5
Hostname: webpod-697b545f57-9nnfd
Hostname: webpod-697b545f57-mn7r5
Hostname: webpod-697b545f57-mn7r5
Hostname: webpod-697b545f57-9ns5w
Hostname: webpod-697b545f57-9ns5w
Hostname: webpod-697b545f57-9ns5w
^C
root@router:~# while true; do curl -s --connect-timeout 1 $LBIP | grep RemoteAddr; sleep 0.1; done
RemoteAddr: 172.20.1.179:57356
RemoteAddr: 172.20.1.179:57358
RemoteAddr: 192.168.10.200:57364
RemoteAddr: 192.168.10.200:57370
RemoteAddr: 172.20.1.179:57372
RemoteAddr: 172.20.1.179:57384
RemoteAddr: 192.168.10.200:57400
RemoteAddr: 172.20.1.179:57404
RemoteAddr: 172.20.1.179:57414
RemoteAddr: 192.168.10.200:57422
RemoteAddr: 172.20.1.179:57426
^C

'스터디 > Cilium' 카테고리의 다른 글

[Cilium] Cluster Mesh  (2) 2025.08.17
[Cilium] BGP Control Plane  (5) 2025.08.17
[Cilium] Service LB-IPAM  (4) 2025.08.10
[Cilium] Overlay Network (Encapsulation) mode  (0) 2025.08.10
[Cilium] Native Routing Mode  (1) 2025.08.10

'스터디/Cilium'의 다른글

  • 현재글[Cilium] L2 Announcements

관련글

티스토리툴바