【K8S排錯(cuò)】在集群的POD內(nèi)不能訪問clusterIP和service

排錯(cuò)背景：在一次生產(chǎn)環(huán)境的部署過程中，配置文件中配置的訪問地址為集群的Service，配置好后發(fā)現(xiàn)服務(wù)不能正常訪問，遂啟動(dòng)了一個(gè)busybox進(jìn)行測(cè)試，測(cè)試發(fā)現(xiàn)在busybox中，能通過coredns正常的解析到IP，然后去ping了一下service，發(fā)現(xiàn)不能ping通，ping clusterIP也不能ping通。

排錯(cuò)經(jīng)歷：首先排查了kube-proxy是否正常，發(fā)現(xiàn)啟動(dòng)都是正常的，然后也重啟了，還是一樣ping不通，然后又排查了網(wǎng)絡(luò)插件，也重啟過flannel，依然沒有任何效果。后來想到自己的另一套k8s環(huán)境，是能正常ping通service的，就對(duì)比這兩套環(huán)境檢查配置，發(fā)現(xiàn)所有配置中只有kube-proxy的配置有一點(diǎn)差別，能ping通的環(huán)境kube-proxy使用了--proxy-mode=ipvs ,不能ping通的環(huán)境使用了默認(rèn)模式（iptables）。

iptables沒有具體設(shè)備響應(yīng)。

然后就是開始經(jīng)過多次測(cè)試，添加--proxy-mode=ipvs 后，清空node上防火墻規(guī)則，重啟kube-proxy后就能正常的ping通了。

在學(xué)習(xí)K8S的時(shí)候，自己一直比較忽略底層流量轉(zhuǎn)發(fā)，也即IPVS和iptables的相關(guān)知識(shí)，認(rèn)為不管哪種模式，只要能轉(zhuǎn)發(fā)訪問到pod就可以，不用太在意這些細(xì)節(jié)，以后還是得更加仔細(xì)才行。

補(bǔ)充：kubeadm 部署方式修改kube-proxy為 ipvs模式。

默認(rèn)情況下，我們部署的kube-proxy通過查看日志，能看到如下信息：Flag proxy-mode="" unknown，assuming iptables proxy

[root@k8s-master?~]#?kubectl?logs?-n?kube-system?kube-proxy-ppdb6?
W1013?06:55:35.773739???????1?proxier.go:513]?Failed?to?load?kernel?module?ip_vs?with?modprobe.?You?can?ignore?this?message?when?kube-proxy?is?running?inside?container?without?mounting?/lib/modules
W1013?06:55:35.868822???????1?proxier.go:513]?Failed?to?load?kernel?module?ip_vs_rr?with?modprobe.?You?can?ignore?this?message?when?kube-proxy?is?running?inside?container?without?mounting?/lib/modules
W1013?06:55:35.869786???????1?proxier.go:513]?Failed?to?load?kernel?module?ip_vs_wrr?with?modprobe.?You?can?ignore?this?message?when?kube-proxy?is?running?inside?container?without?mounting?/lib/modules
W1013?06:55:35.870800???????1?proxier.go:513]?Failed?to?load?kernel?module?ip_vs_sh?with?modprobe.?You?can?ignore?this?message?when?kube-proxy?is?running?inside?container?without?mounting?/lib/modules
W1013?06:55:35.876832???????1?server_others.go:249]?Flag?proxy-mode=""?unknown,?assuming?iptables?proxy
I1013?06:55:35.890892???????1?server_others.go:143]?Using?iptables?Proxier.
I1013?06:55:35.892136???????1?server.go:534]?Version:?v1.15.0
I1013?06:55:35.909025???????1?conntrack.go:100]?Set?sysctl?'net/netfilter/nf_conntrack_max'?to?131072
I1013?06:55:35.909053???????1?conntrack.go:52]?Setting?nf_conntrack_max?to?131072
I1013?06:55:35.919298???????1?conntrack.go:83]?Setting?conntrack?hashsize?to?32768
I1013?06:55:35.945969???????1?conntrack.go:100]?Set?sysctl?'net/netfilter/nf_conntrack_tcp_timeout_established'?to?86400
I1013?06:55:35.946044???????1?conntrack.go:100]?Set?sysctl?'net/netfilter/nf_conntrack_tcp_timeout_close_wait'?to?3600
I1013?06:55:35.946623???????1?config.go:96]?Starting?endpoints?config?controller
I1013?06:55:35.946660???????1?controller_utils.go:1029]?Waiting?for?caches?to?sync?for?endpoints?config?controller
I1013?06:55:35.946695???????1?config.go:187]?Starting?service?config?controller
I1013?06:55:35.946713???????1?controller_utils.go:1029]?Waiting?for?caches?to?sync?for?service?config?controller
I1013?06:55:36.047121???????1?controller_utils.go:1036]?Caches?are?synced?for?endpoints?config?controller
I1013?06:55:36.047195???????1?controller_utils.go:1036]?Caches?are?synced?for?service?config?controller

這里我們需要修改kube-proxy的配置文件,添加mode 為ipvs。

[root@k8s-master?~]#?kubectl?edit?cm?kube-proxy?-n?kube-system
...
ipvs:
??????excludeCIDRs:?null
??????minSyncPeriod:?0s
??????scheduler:?""
??????strictARP:?false
??????syncPeriod:?30s
????kind:?KubeProxyConfiguration
????metricsBindAddress:?127.0.0.1:10249
????mode:?"ipvs"
???...

ipvs模式需要注意的是要添加ip_vs相關(guān)模塊：

cat?>?/etc/sysconfig/modules/ipvs.modules?<<EOF
?#!/bin/bash?
?modprobe?--?ip_vs?
?modprobe?--?ip_vs_rr?
?modprobe?--?ip_vs_wrr?
?modprobe?--?ip_vs_sh?
?modprobe?--?nf_conntrack_ipv4?
EOF

chmod?755?/etc/sysconfig/modules/ipvs.modules?&&?bash?/etc/sysconfig/modules/ipvs.modules?&&?lsmod?|?grep?-e?ip_vs?-e?nf_conntrack_ipv4

重啟kube-proxy 的pod

[root@k8s-master?~]#?kubectl?get?pod?-n?kube-system?|?grep?kube-proxy?|awk?'{system("kubectl?delete?pod?"$1"?-n?kube-system")}'
pod?"kube-proxy-62gvr"?deleted
pod?"kube-proxy-n2rml"?deleted
pod?"kube-proxy-ppdb6"?deleted
pod?"kube-proxy-rr9cg"?deleted

在pod重啟后再查看日志，發(fā)現(xiàn)模式已經(jīng)變?yōu)閕pvs了。

[root@k8s-master?~]#?kubectl?get?pod?-n?kube-system?|grep?kube-proxy
kube-proxy-cbm8p?????????????????????1/1?????Running???0??????????85s
kube-proxy-d97pn?????????????????????1/1?????Running???0??????????83s
kube-proxy-gmq6s?????????????????????1/1?????Running???0??????????76s
kube-proxy-x6tcg?????????????????????1/1?????Running???0??????????81s
[root@k8s-master?~]#?kubectl?logs?-n?kube-system?kube-proxy-cbm8p?
I1013?07:34:38.685794???????1?server_others.go:170]?Using?ipvs?Proxier.
W1013?07:34:38.686066???????1?proxier.go:401]?IPVS?scheduler?not?specified,?use?rr?by?default
I1013?07:34:38.687224???????1?server.go:534]?Version:?v1.15.0
I1013?07:34:38.692777???????1?conntrack.go:52]?Setting?nf_conntrack_max?to?131072
I1013?07:34:38.693378???????1?config.go:187]?Starting?service?config?controller
I1013?07:34:38.693391???????1?controller_utils.go:1029]?Waiting?for?caches?to?sync?for?service?config?controller
I1013?07:34:38.693406???????1?config.go:96]?Starting?endpoints?config?controller
I1013?07:34:38.693411???????1?controller_utils.go:1029]?Waiting?for?caches?to?sync?for?endpoints?config?controller
I1013?07:34:38.793684???????1?controller_utils.go:1036]?Caches?are?synced?for?endpoints?config?controller
I1013?07:34:38.793688???????1?controller_utils.go:1036]?Caches?are?synced?for?service?config?controller

再次測(cè)試ping service

[root@k8s-master?~]#?kubectl?exec?-it?dns-test?sh
/?#?ping?nginx-service
PING?nginx-service?(10.1.58.65):?56?data?bytes
64?bytes?from?10.1.58.65:?seq=0?ttl=64?time=0.033?ms
64?bytes?from?10.1.58.65:?seq=1?ttl=64?time=0.069?ms
64?bytes?from?10.1.58.65:?seq=2?ttl=64?time=0.094?ms
64?bytes?from?10.1.58.65:?seq=3?ttl=64?time=0.057?ms
^C
---?nginx-service?ping?statistics?---
4?packets?transmitted,?4?packets?received,?0%?packet?loss
round-trip?min/avg/max?=?0.033/0.063/0.094?ms