实现多集群网络通信

参考文档：https://docs.cilium.io/en/stable/network/clustermesh/clustermesh/

k8s 集群版本 1.32.8
Cilium 版本 v1.17.4

指定群集名称和ID

必须为每个集群分配一个唯一的人类可读名称以及一个数字集群ID（1-255）。集群名称必须遵守以下约束：

它必须最多包含32个字符；
它必须以小写字母数字字符开头和结尾；
它可能包含小写字母数字字符和之间的破折号。

#集群b 
export CLUSTER2=cluster-b
 
#集群c
export CLUSTER3=cluster-c

##### 之前安装过需要卸载重新安装
# root@k8s-master02:~# cilium uninstall --kubeconfig /root/config/cluster-b.yaml
# root@k8s-master02:~# cilium uninstall --kubeconfig /root/config/cluster-c.yaml
cilium install --set cluster.name=$CLUSTER2 --set cluster.id=2 --kubeconfig /root/config/cluster-b.yaml
cilium install --set cluster.name=$CLUSTER3 --set cluster.id=3 --kubeconfig /root/config/cluster-c.yaml

验证

cilium config view | grep cluster-id
 
cilium config view | grep cluster-name

共享证书授权中心(可选)

如果您计划跨集群运行Hubble Relay，最好在集群之间共享证书颁发机构（CA），因为它将使跨集群的mTLS能够正常工作。

您可以将包含CA的库伯内特斯秘密从一个集群复制到另一个集群：

kubectl --kubeconfig /root/config/cluster-b.yaml get secret \
  -n kube-system cilium-ca -o yaml | \
  kubectl --kubeconfig /root/config/cluster-c.yaml create -f -

启用 Clustermesh

- LoadBalancer（推荐）: 使用 LoadBalancer 类型的 Service 发布服务，这样可以使用稳定的 LoadBalancer IP，通常是最佳选择。
- NodePort: 使用 NodePort 类型的 Service 发布服务，如果一个节点消失，Cluster Mesh 将不得不重新连接到另一个节点，可能会造成网络的中断。
- ClusterIP: 使用 ClusterIP 类型的 Service 发布服务，这要求 ClusterIP 在集群间是可路由的。

# 集群B
root@k8s-master02:~# cilium clustermesh enable  --service-type LoadBalancer

# 集群C
root@k8s-master03:~# cilium clustermesh enable  --service-type LoadBalancer

等待群集网格组件通过调用cilium clustermesh status --wait。如果您正在使用LoadBalancer类型的服务，那么这也将等待LoadBalancer被分配一个IP。

cilium clustermesh status --kubeconfig /root/config/cluster-b.yaml --wait
cilium clustermesh status --kubeconfig /root/config/cluster-c.yaml --wait

连接集群

1. 修改 cluster-b.yaml

root@k8s-master02:~/config# vim cluster-b.yaml

2. 修改 cluster-c.yaml

root@k8s-master02:~/config# vim cluster-c.yaml

3. 合并

KUBECONFIG=/root/config/cluster-b.yaml:/root/config/cluster-c.yaml \
kubectl config view --flatten > /root/config/merged.yaml

4.检查：

kubectl config get-contexts --kubeconfig /root/config/merged.yaml

你应该能看到：

root@k8s-master02:~/config# kubectl config get-contexts --kubeconfig /root/config/merged.yaml
CURRENT   NAME        CLUSTER     AUTHINFO          NAMESPACE
*         cluster-b   cluster-b   cluster-b-admin   
          cluster-c   cluster-c   cluster-c-admin

这样虽然两个 CLUSTER 字段都是 kubernetes，但 context 已经能区分了，cilium clustermesh connect 就能用。

5.连接集群

最后，连接集群。这一步只需要在一个方向上完成。连接将自动在两个方向上建立：

root@k8s-master02:~# KUBECONFIG=/root/config/merged.yaml cilium clustermesh connect --context $CLUSTER2 --destination-context $CLUSTER3

连接集群可能需要一点时间。您可以运行cilium clustermesh status --wait以等待连接成功：

root@k8s-master02:~# KUBECONFIG=/root/config/merged.yaml cilium clustermesh status --context $CLUSTER2 --wait

✅ Service "clustermesh-apiserver" of type "LoadBalancer" found
✅ Cluster access information is available:
  - 192.168.139.81:2379
✅ Deployment clustermesh-apiserver is ready
ℹ️  KVStoreMesh is enabled

✅ All 2 nodes are connected to all clusters [min:1 / avg:1.0 / max:1]
✅ All 1 KVStoreMesh replicas are connected to all clusters [min:1 / avg:1.0 / max:1]

🔌 Cluster Connections:
  - cluster-c: 2/2 configured, 2/2 connected - KVStoreMesh: 1/1 configured, 1/1 connected

🔀 Global services: [ min:0 / avg:0.0 / max:0 ]

实现案例多集群访问

1. 实现多集群全局负载均衡

root@k8s-master02:~/cilium# cat cluster-b-deploy.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: serve
spec:
  replicas: 1
  selector:
    matchLabels:
      app: serve
  template:
    metadata:
      labels:
        app: serve
    spec:
      containers:
        - name: serve
          image: jeremyot/serve:0a40de8
          args:
            - '--message=''hello from cluster cluster-b (Node: {{env "NODE_NAME"}} Pod: {{env "POD_NAME"}} Address: {{addr}})'''
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
---
apiVersion: v1
kind: Service
metadata:
  name: serve
  annotations:
    service.cilium.io/global: "true" # 开启集群网络互通
spec:
  ports:
    - port: 80
      targetPort: 8080
  selector:
    app: serve

root@k8s-master02:~/cilium# cat cluster-c-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: serve
spec:
  replicas: 1
  selector:
    matchLabels:
      app: serve
  template:
    metadata:
      labels:
        app: serve
    spec:
      containers:
        - name: serve
          image: jeremyot/serve:0a40de8
          args:
            - '--message=''hello from cluster cluster-c (Node: {{env "NODE_NAME"}} Pod: {{env "POD_NAME"}} Address: {{addr}})'''
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
---
apiVersion: v1
kind: Service
metadata:
  name: serve
  annotations:
    service.cilium.io/global: "true" # 开启集群网络互通
spec:
  ports:
    - port: 80
      targetPort: 8080
  selector:
    app: serve

root@k8s-master02:~/cilium# KUBECONFIG=/root/config/merged.yaml kubectl apply -f cluster-b-deploy.yaml  --context cluster-b
root@k8s-master02:~/cilium# KUBECONFIG=/root/config/merged.yaml kubectl apply -f cluster-c-deploy.yaml  --context cluster-c

验证访问

root@k8s-master02:~/cilium# kubectl --kubeconfig /root/config/cluster-c.yaml run -i --rm --restart=Never --image=jeremyot/request:0a40de8 request -- --duration=5s --address=serve
If you don't see a command prompt, try pressing enter.
2025/09/01 15:33:22 'hello from cluster cluster-b (Node: k8s-node02 Pod: serve-7bbd7bc4dd-9kvv7 Address: 10.245.1.150)'
2025/09/01 15:33:23 'hello from cluster cluster-c (Node: k8s-node03 Pod: serve-86f865878f-xf7cg Address: 10.246.1.14)'
2025/09/01 15:33:24 'hello from cluster cluster-c (Node: k8s-node03 Pod: serve-86f865878f-xf7cg Address: 10.246.1.14)'
pod "request" deleted

集群B/C 交替访问

2. 禁止全局服务共享

---
apiVersion: v1
kind: Service
metadata:
  name: serve
  annotations:
    service.cilium.io/global: "true" # 开启集群网络互通
    service.cilium.io/shared: "false"  # 添加这个注解即可
spec:
  ports:
    - port: 80
      targetPort: 8080
  selector:
    app: serve

关闭 cluster-b 共享

root@k8s-master02:~/cilium# KUBECONFIG=/root/config/merged.yaml kubectl annotate service serve service.cilium.io/shared="false" --overwrite --context cluster-b

验证访问

oot@k8s-master02:~/cilium# kubectl --kubeconfig /root/config/cluster-c.yaml run -i --rm --restart=Never --image=jeremyot/request:0a40de8 request -- --duration=10s --address=serve
If you don't see a command prompt, try pressing enter.
2025/09/01 15:41:09 'hello from cluster cluster-c (Node: k8s-node03 Pod: serve-86f865878f-xf7cg Address: 10.246.1.14)'
2025/09/01 15:41:10 'hello from cluster cluster-c (Node: k8s-node03 Pod: serve-86f865878f-xf7cg Address: 10.246.1.14)'
2025/09/01 15:41:11 'hello from cluster cluster-c (Node: k8s-node03 Pod: serve-86f865878f-xf7cg Address: 10.246.1.14)'
2025/09/01 15:41:12 'hello from cluster cluster-c (Node: k8s-node03 Pod: serve-86f865878f-xf7cg Address: 10.246.1.14)'
2025/09/01 15:41:13 'hello from cluster cluster-c (Node: k8s-node03 Pod: serve-86f865878f-xf7cg Address: 10.246.1.14)'
2025/09/01 15:41:14 'hello from cluster cluster-c (Node: k8s-node03 Pod: serve-86f865878f-xf7cg Address: 10.246.1.14)'
2025/09/01 15:41:15 'hello from cluster cluster-c (Node: k8s-node03 Pod: serve-86f865878f-xf7cg Address: 10.246.1.14)'
2025/09/01 15:41:16 'hello from cluster cluster-c (Node: k8s-node03 Pod: serve-86f865878f-xf7cg Address: 10.246.1.14)'
2025/09/01 15:41:17 'hello from cluster cluster-c (Node: k8s-node03 Pod: serve-86f865878f-xf7cg Address: 10.246.1.14)'
pod "request" deleted

只能访问到自己集群C的数据

root@k8s-master02:~/cilium# kubectl --kubeconfig /root/config/cluster-b.yaml run -i --rm --restart=Never --image=jeremyot/request:0a40de8 request -- --duration=10s --address=serve
If you don't see a command prompt, try pressing enter.
2025/09/01 15:42:10 'hello from cluster cluster-c (Node: k8s-node03 Pod: serve-86f865878f-xf7cg Address: 10.246.1.14)'
2025/09/01 15:42:11 'hello from cluster cluster-b (Node: k8s-node02 Pod: serve-7bbd7bc4dd-9kvv7 Address: 10.245.1.150)'
2025/09/01 15:42:12 'hello from cluster cluster-c (Node: k8s-node03 Pod: serve-86f865878f-xf7cg Address: 10.246.1.14)'
2025/09/01 15:42:13 'hello from cluster cluster-c (Node: k8s-node03 Pod: serve-86f865878f-xf7cg Address: 10.246.1.14)'
2025/09/01 15:42:14 'hello from cluster cluster-b (Node: k8s-node02 Pod: serve-7bbd7bc4dd-9kvv7 Address: 10.245.1.150)'
2025/09/01 15:42:15 'hello from cluster cluster-b (Node: k8s-node02 Pod: serve-7bbd7bc4dd-9kvv7 Address: 10.245.1.150)'
2025/09/01 15:42:16 'hello from cluster cluster-b (Node: k8s-node02 Pod: serve-7bbd7bc4dd-9kvv7 Address: 10.245.1.150)'
2025/09/01 15:42:17 'hello from cluster cluster-b (Node: k8s-node02 Pod: serve-7bbd7bc4dd-9kvv7 Address: 10.245.1.150)'
pod "request" deleted

集群B 依旧可以访问到集群C

3. 实现集群 C 访问集群 B的服务

root@k8s-master02:~/cilium# cat cluster-b-deploy.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: serve
spec:
  replicas: 1
  selector:
    matchLabels:
      app: serve
  template:
    metadata:
      labels:
        app: serve
    spec:
      containers:
        - name: serve
          image: jeremyot/serve:0a40de8
          args:
            - '--message=''hello from cluster cluster-b (Node: {{env "NODE_NAME"}} Pod: {{env "POD_NAME"}} Address: {{addr}})'''
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
---
apiVersion: v1
kind: Service
metadata:
  name: serve
  annotations:
    service.cilium.io/global: "true" # 开启集群网络互通
spec:
  ports:
    - port: 80
      targetPort: 8080
  selector:
    app: serve

# 只部署 svc
root@k8s-master02:~/cilium# cat cluster-c-svc.yaml 
apiVersion: v1
kind: Service
metadata:
  name: serve
  annotations:
    service.cilium.io/global: "true" # 开启集群网络互通
spec:
  ports:
    - port: 80
      targetPort: 8080
  selector:
    app: serve

    
root@k8s-master02:~/cilium# KUBECONFIG=/root/config/merged.yaml kubectl apply -f cluster-b-deploy.yaml  --context cluster-b
root@k8s-master02:~/cilium# KUBECONFIG=/root/config/merged.yaml kubectl apply -f cluster-c-svc.yaml  --context cluster-c

验证访问

root@k8s-master02:~/cilium# kubectl --kubeconfig /root/config/cluster-c.yaml run -i --rm --restart=Never --image=jeremyot/request:0a40de8 request -- --duration=10s --address=serve
If you don't see a command prompt, try pressing enter.
2025/09/01 15:51:15 'hello from cluster cluster-b (Node: k8s-node02 Pod: serve-7bbd7bc4dd-dm4wg Address: 10.245.1.234)'
2025/09/01 15:51:16 'hello from cluster cluster-b (Node: k8s-node02 Pod: serve-7bbd7bc4dd-dm4wg Address: 10.245.1.234)'
2025/09/01 15:51:17 'hello from cluster cluster-b (Node: k8s-node02 Pod: serve-7bbd7bc4dd-dm4wg Address: 10.245.1.234)'
2025/09/01 15:51:18 'hello from cluster cluster-b (Node: k8s-node02 Pod: serve-7bbd7bc4dd-dm4wg Address: 10.245.1.234)'
2025/09/01 15:51:19 'hello from cluster cluster-b (Node: k8s-node02 Pod: serve-7bbd7bc4dd-dm4wg Address: 10.245.1.234)'
2025/09/01 15:51:20 'hello from cluster cluster-b (Node: k8s-node02 Pod: serve-7bbd7bc4dd-dm4wg Address: 10.245.1.234)'
2025/09/01 15:51:21 'hello from cluster cluster-b (Node: k8s-node02 Pod: serve-7bbd7bc4dd-dm4wg Address: 10.245.1.234)'
2025/09/01 15:51:22 'hello from cluster cluster-b (Node: k8s-node02 Pod: serve-7bbd7bc4dd-dm4wg Address: 10.245.1.234)'
pod "request" deleted

集群C 成功访问集群B