Kubernetes-网络原理(5) - ClusterIP

我们看完的了简单的网络通讯,在Kubernetes中还有一个很重要的ClusterIP的概念。如下图所示,这个是K8s实现服务负载均衡的核心基础。

部署环境

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-world-server
labels:
app: hello-world-server
spec:
replicas: 3
selector:
matchLabels:
app: hello-world-server
template:
metadata:
labels:
app: hello-world-server
spec:
containers: - name: hello-world-server
image: python:2.7
imagePullPolicy: Always
command: [“/bin/bash”]
args: [“-c”, “echo \”<p>Hi from $(hostname)</p>\” > index.html; python -m SimpleHTTPServer 8080”]
ports: - name: http
containerPort: 8080 - name: curl-client
image: yannxia/ubuntu-with-tcpdump
imagePullPolicy: Always
command: [“/bin/sh”]
args: [“-c”, “while true; do sleep 10; done;”]
1
2
3
4
5
6
7
8
9
10
11
12
13
kind: Service
apiVersion: v1
metadata:
name: hello-world-service
spec:
selector:
app: hello-world-server
ports:

- protocol: TCP
port: 80
targetPort: 8080

敲个小命令

1
2
$ kubectl apply -f hello-world-server-deployment.yaml
$ kubectl apply -f hello-world-server-service.yaml

检查下环境

1
2
3
4
5
6
7
8
9
10
11
12
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
c2c-network-demo 2/2 Running 0 18h 192.168.1.2 k8s-worker-1 <none> <none>
c2c-network-demo-w2 2/2 Running 0 22h 192.168.2.3 k8s-worker-2 <none> <none>
hello-world-server-6469546c67-bbtjp 2/2 Running 0 71s 192.168.2.4 k8s-worker-2 <none> <none>
hello-world-server-6469546c67-d6j9n 2/2 Running 0 71s 192.168.1.3 k8s-worker-1 <none> <none>
hello-world-server-6469546c67-fbsvv 2/2 Running 0 72s 192.168.1.4 k8s-worker-1 <none> <none>

$ kubectl get services -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
hello-world-service ClusterIP 10.100.3.59 <none> 80/TCP 16s app=hello-world-server
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 24h <none>

此时我们有了一个 CLUSTER-IP:10.100.3.59

探索

在我们的服务器上执行 curl

1
2
3
4
5
6
7
8
9
10
➜  ~ curl 10.100.3.59
<p>Hi from hello-world-server-6469546c67-bbtjp</p>
➜ ~ curl 10.100.3.59
<p>Hi from hello-world-server-6469546c67-d6j9n</p>
➜ ~ curl 10.100.3.59
<p>Hi from hello-world-server-6469546c67-bbtjp</p>
➜ ~ curl 10.100.3.59
<p>Hi from hello-world-server-6469546c67-d6j9n</p>
➜ ~ curl 10.100.3.59
<p>Hi from hello-world-server-6469546c67-fbsvv</p>

我们发现我们的请求会均衡的转发到不同的Pod上去了,显然也达成了一个load balance的功能。
在我们的 worker-2 的节点上,我们检查我们的路由表:

1
2
3
4
5
6
7
8
9
10
11
12
➜  ~ route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.12.0.1 0.0.0.0 UG 0 0 0 ens192
10.12.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ens192
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
192.168.0.0 10.12.22.150 255.255.255.0 UG 0 0 0 ens192
192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 *
192.168.1.2 0.0.0.0 255.255.255.255 UH 0 0 0 calid9d486e54a8
192.168.1.3 0.0.0.0 255.255.255.255 UH 0 0 0 cali11d6478e38f
192.168.1.4 0.0.0.0 255.255.255.255 UH 0 0 0 calief5eb3245cc
192.168.2.0 10.12.22.152 255.255.255.0 UG 0 0 0 ens192

没有任意匹配的路由表(除了 0.0.0.0)。

ClusterIP 模式

按照官网的说明,ClusterIP 有三种工作模式

  • userspace(since v1.0)
  • iptables (since v1.1)
  • ipvs (since v1.9)
1
➜  ~ iptables -L | grep 10.100.3.59

我们可以从 kube-proxy 的启动日志中发现:

1
2
3
4
W0321 06:39:55.760705       1 server_others.go:295] Flag proxy-mode="" unknown, assuming iptables proxy
I0321 06:39:55.778302 1 server_others.go:148] Using iptables Proxier.
I0321 06:39:55.778589 1 server_others.go:178] Tearing down inactive rules.
I0321 06:39:56.188482 1 server.go:483] Version: v1.13.4

如果没有特意的指定,我们所使用的还是iptables 的方式

Debug Iptable

我们从本机访问出去的连接会经历 POSTROUTINGOUTPUT 这两个Chain,我们给OUTPUT 增加一个 DEBUGRule

1
iptables  -A OUTPUT -t raw -p tcp --destination 10.100.3.59 -j TRACE

/var/log/kern.log 中我们看到

  1. 我们再去查询下 iptables 查询 ➌ 处的

    1
    2
    3
    4
    5
    ➜  ~ iptables -L KUBE-SEP-M7XDXP4L3HZCSYD6 -t nat --line-number
    Chain KUBE-SEP-M7XDXP4L3HZCSYD6 (1 references)
    num target prot opt source destination
    1 KUBE-MARK-MASQ all -- 192.168.2.4 anywhere
    2 DNAT tcp -- anywhere anywhere tcp to:192.168.2.4:8080

    在这里,我们使用了使用第二条的 DNAT 方式将数据包转换了 IP目标地址

  2. 我们沿着记录往上追溯,查询 ➋ 处

    1
    2
    3
    4
    5
    6
    7
    8
    9
     ➜  ~ iptables -t nat -L KUBE-SVC-5MRENC7Q6ZQR6GKR
    Chain KUBE-SVC-5MRENC7Q6ZQR6GKR (1 references)
    target prot opt source destination
    # 192.168.1.3
    KUBE-SEP-PG5HHQCRLP6OHBWJ all -- anywhere anywhere statistic mode random probability 0.33332999982
    # 192.168.1.4
    KUBE-SEP-Z7HABNZOJAQBPZC7 all -- anywhere anywhere statistic mode random probability 0.50000000000
    # 192.168.2.4
    KUBE-SEP-M7XDXP4L3HZCSYD6 all -- anywhere anywhere

    我们发现了我们是这里直接将数据进行随机的选择,(为什么不是均等的:黑脸问号),我们现在得到 Kubernats 针对 ClusterIP 的处理方式如下

all-in-one

附录