Kubernetes群集部署之ETCD

1、Master組件

●kube-apiserver
Kubernetes API,集群的統(tǒng)一入口，各組件協(xié)調(diào)者，以RESTful API提供接口服務(wù)，所有對(duì)象資源的增刪改查和監(jiān)聽操作都交給APIServer處理后再提交給Etcd存儲(chǔ)。

●kube-controller-manager
處理集群中常規(guī)后臺(tái)任務(wù)，一個(gè) 資源對(duì)應(yīng)一個(gè)控制 器，而ControllerManager就是負(fù)責(zé)管理這些控制器的。

●kube-scheduler
根據(jù)調(diào)度算法為新創(chuàng)建的Pod選擇一個(gè)Node節(jié)點(diǎn)，可以任意部署，可以部署在同一個(gè)節(jié)點(diǎn)上，也可以部署在不同的節(jié)點(diǎn)上。

●etcd
分布式鍵值存儲(chǔ)系統(tǒng)。用于保存集群狀態(tài)數(shù)據(jù)，比如Pod、Service 等對(duì)象信息。

2、Node組件

●kubelet
kubelet是Master在Node節(jié)點(diǎn)上的Agent，管理本機(jī)運(yùn)行容器的生命周期，比如創(chuàng)建容器、Pod掛載數(shù)據(jù)卷、下 載secret、獲取容器和節(jié)點(diǎn)狀態(tài)等工作。kubelet將 每個(gè)Pod轉(zhuǎn)換成一組容器。

●kube-proxy
在Node節(jié)點(diǎn)上實(shí)現(xiàn)Pod網(wǎng)絡(luò)代理，維護(hù)網(wǎng)絡(luò)規(guī)則和四層負(fù)載均衡工作。

●docker或rocket
容器引擎，運(yùn)行容器。

工作原理：

1、準(zhǔn)備包含應(yīng)用程序的Deployment的yml文件，然后通過kubectl客戶端工具發(fā)送給ApiServer。

2、ApiServer接收到客戶端的請(qǐng)求并將資源內(nèi)容存儲(chǔ)到數(shù)據(jù)庫(etcd)中。

3、Controller組件(包括scheduler、replication、endpoint)監(jiān)控資源變化并作出反應(yīng)。

4、ReplicaSet檢查數(shù)據(jù)庫變化，創(chuàng)建期望數(shù)量的pod實(shí)例。

5、Scheduler再次檢查數(shù)據(jù)庫變化，發(fā)現(xiàn)尚未被分配到具體執(zhí)行節(jié)點(diǎn)(node)的Pod，然后根據(jù)一組相關(guān)規(guī)則將pod分配到可以運(yùn)行它們的節(jié)點(diǎn)上，并更新數(shù)據(jù)庫，記錄pod分配情況。

6、Kubelete監(jiān)控?cái)?shù)據(jù)庫變化，管理后續(xù)pod的生命周期，發(fā)現(xiàn)被分配到它所在的節(jié)點(diǎn)上運(yùn)行的那些pod。如果找到新pod，則會(huì)在該節(jié)點(diǎn)上運(yùn)行這個(gè)新pod。

附：kuberproxy運(yùn)行在集群各個(gè)主機(jī)上，管理網(wǎng)絡(luò)通信，如服務(wù)發(fā)現(xiàn)、負(fù)載均衡。當(dāng)有數(shù)據(jù)發(fā)送到主機(jī)時(shí)，將其路由到正確的pod或容器。對(duì)于從主機(jī)上發(fā)出的數(shù)據(jù)，它可以基于請(qǐng)求地址發(fā)現(xiàn)遠(yuǎn)程服務(wù)器，并將數(shù)據(jù)正確路由，在某些情況下會(huì)使用輪循調(diào)度算法(Round-robin)將請(qǐng)求發(fā)送到集群中的多個(gè)實(shí)例。

Kubernetes核心概念

1、Pod

●最小部署單元

●一組容器的集合

●一個(gè)Pod中的容器共享網(wǎng)絡(luò)命名空間

●Pod是短暫的

2、Controllers

●ReplicaSet :確保 預(yù)期的Pod副本數(shù)量

●Deployment:無狀態(tài)應(yīng)用 部署

●StatefulSet :有狀態(tài)應(yīng)用部署

●DaemonSet:確保所有Node運(yùn)行同一個(gè)Pod

●Job:一次性任務(wù)

●Cronjob:定時(shí)任務(wù)

更高級(jí)層次對(duì)象，部署和管理Pod

3、Service

●防止Pod失聯(lián)

●定義一組Pod的訪問策略

●Label: 標(biāo)簽，附加到某個(gè)資源上，用于關(guān)聯(lián)對(duì)象、查詢和篩選

●Namespaces :命 名空間，將對(duì)象邏輯上隔離

●Annotations :注釋

Kubernetes集群部署

1.官方提供的三種部署方式

●minikube

Minikube是一個(gè)工具，可以在本地快速運(yùn)行一個(gè)單點(diǎn)的Kubernetes，僅用于嘗試Kubernetes或日常開發(fā)的用戶使用。
部署地址: https://kubernetes.io/docs/setup/minikube/

●kubeadm

Kubeadm也是一個(gè)工具，提供kubeadm init和kubeadm join,用于快速部署Kubernetes集群。

部署地址:https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm/

●二進(jìn)制包

推薦，從官方下載發(fā)行版的二進(jìn)制包，手動(dòng)部署每個(gè)組件，組成Kubernetes集群。

下載地址:https://github.com/kubernetes/kubernetes/releases

2. Kubernetes 平臺(tái)環(huán)境規(guī)劃

3.自簽SSL證書

4. Etcd數(shù)據(jù)庫集群部署

? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Kubernetes二進(jìn)制部署
k8s軟件包：

鏈接：https://pan.baidu.com/s/1oN2wkGZ_7parS8sMaaogGw?
提取碼：lbjx

k8s部署規(guī)劃：
master：192.168.35.128 ? ? ?kube-apiserver kube-controller-manager kube-scheduler etcd
node1：192.168.35.195 ? ? ? kubelet kube-proxy docker flannel etcd
node2：192.168.35.138 ? ? ? kubelet kube-proxy docker flannel etcd

master操作：
[root@localhost ~]# mkdir k8s
[root@localhost ~]# cd k8s/
[root@localhost k8s]# ls ? ?//從宿主機(jī)拖進(jìn)來
etcd-cert.sh ?etcd.sh
[root@localhost k8s]# mkdir etcd-cert
[root@localhost k8s]# mv etcd-cert.sh etcd-cert
上面?etcd-cert.sh etcd.sh腳本

vim etcd.sh

#!/bin/bash
#example: ./etcd.sh etcd01 192.168.1.10 etcd02=https://192.168.1.11:2380,etcd03=https://192.168.1.12:2380

ETCD_NAME=$1
ETCD_IP=$2
ETCD_CLUSTER=$3

WORK_DIR=/opt/etcd

cat <<EOF >$WORK_DIR/cfg/etcd
#[Member]
ETCD_NAME="${ETCD_NAME}"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://${ETCD_IP}:2380"
ETCD_LISTEN_CLIENT_URLS="https://${ETCD_IP}:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://${ETCD_IP}:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://${ETCD_IP}:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://${ETCD_IP}:2380,${ETCD_CLUSTER}"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

cat <<EOF >/usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=${WORK_DIR}/cfg/etcd
ExecStart=${WORK_DIR}/bin/etcd \
--name=\${ETCD_NAME} \
--data-dir=\${ETCD_DATA_DIR} \
--listen-peer-urls=\${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=\${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
--advertise-client-urls=\${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=\${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=\${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=\${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=new \
--cert-file=${WORK_DIR}/ssl/server.pem \
--key-file=${WORK_DIR}/ssl/server-key.pem \
--peer-cert-file=${WORK_DIR}/ssl/server.pem \
--peer-key-file=${WORK_DIR}/ssl/server-key.pem \
--trusted-ca-file=${WORK_DIR}/ssl/ca.pem \
--peer-trusted-ca-file=${WORK_DIR}/ssl/ca.pem
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable etcd
systemctl restart etcd

vim etcd-cert.sh

cat > ca-config.json <<EOF
{
? "signing": {
? ? "default": {
? ? ? "expiry": "87600h"
? ? },
? ? "profiles": {
? ? ? "www": {
? ? ? ? ?"expiry": "87600h",
? ? ? ? ?"usages": [
? ? ? ? ? ? "signing",
? ? ? ? ? ? "key encipherment",
? ? ? ? ? ? "server auth",
? ? ? ? ? ? "client auth"
? ? ? ? ]
? ? ? }
? ? }
? }
}
EOF

cat > ca-csr.json <<EOF
{
? ? "CN": "etcd CA",
? ? "key": {
? ? ? ? "algo": "rsa",
? ? ? ? "size": 2048
? ? },
? ? "names": [
? ? ? ? {
? ? ? ? ? ? "C": "CN",
? ? ? ? ? ? "L": "Beijing",
? ? ? ? ? ? "ST": "Beijing"
? ? ? ? }
? ? ]
}
EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

#-----------------------

cat > server-csr.json <<EOF
{
? ? "CN": "etcd",
? ? "hosts": [
? ? "10.206.240.188",
? ? "10.206.240.189",
? ? "10.206.240.111"
? ? ],
? ? "key": {
? ? ? ? "algo": "rsa",
? ? ? ? "size": 2048
? ? },
? ? "names": [
? ? ? ? {
? ? ? ? ? ? "C": "CN",
? ? ? ? ? ? "L": "BeiJing",
? ? ? ? ? ? "ST": "BeiJing"
? ? ? ? }
? ? ]
}
EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server

下載官方包?

[root@localhost k8s]# vim cfssl.sh
curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o /usr/local/bin/cfssl
curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /usr/local/bin/cfssljson
curl -L https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -o /usr/local/bin/cfssl-certinfo
chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson /usr/local/bin/cfssl-certinfo

[root@localhost k8s]# bash cfssl.sh ?//下載cfssl官方包
? % Total ? ?% Received % Xferd ?Average Speed ? Time ? ?Time ? ? Time ?Current
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Dload ?Upload ? Total ? Spent ? ?Left ?Speed
100 ?9.8M ?100 ?9.8M ? ?0 ? ? 0 ?77052 ? ? ?0 ?0:02:14 ?0:02:14 --:--:-- 94447
? % Total ? ?% Received % Xferd ?Average Speed ? Time ? ?Time ? ? Time ?Current
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Dload ?Upload ? Total ? Spent ? ?Left ?Speed
100 2224k ?100 2224k ? ?0 ? ? 0 ?66701 ? ? ?0 ?0:00:34 ?0:00:34 --:--:-- 71949
? % Total ? ?% Received % Xferd ?Average Speed ? Time ? ?Time ? ? Time ?Current
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Dload ?Upload ? Total ? Spent ? ?Left ?Speed
100 6440k ?100 6440k ? ?0 ? ? 0 ?74368 ? ? ?0 ?0:01:28 ?0:01:28 --:--:-- 93942

[root@localhost k8s]# ls /usr/local/bin/
cfssl ?cfssl-certinfo ?cfssljson
//cfssl 生成證書工具、cfssljson通過傳入json文件生成證書、cfssl-certinfo查看證書信息
定義證書

[root@localhost k8s]# cd etcd-cert/

//定義ca證書
cat > ca-config.json <<EOF
{
? "signing": {
? ? "default": {
? ? ? "expiry": "87600h"
? ? },
? ? "profiles": {
? ? ? "www": {
? ? ? ? ?"expiry": "87600h",
? ? ? ? ?"usages": [
? ? ? ? ? ? "signing",
? ? ? ? ? ? "key encipherment",
? ? ? ? ? ? "server auth",
? ? ? ? ? ? "client auth" ? ??
? ? ? ? ] ?
? ? ? }?
? ? } ? ? ? ??
? }
}
EOF?

實(shí)現(xiàn)證書簽名

//實(shí)現(xiàn)證書簽名
cat > ca-csr.json <<EOF?
{ ??
? ? "CN": "etcd CA",
? ? "key": {
? ? ? ? "algo": "rsa",
? ? ? ? "size": 2048
? ? },
? ? "names": [
? ? ? ? {
? ? ? ? ? ? "C": "CN",
? ? ? ? ? ? "L": "Beijing",
? ? ? ? ? ? "ST": "Beijing"
? ? ? ? }
? ? ]
}
EOF
生產(chǎn)證書，生成ca-key.pem ?ca.pem

//生產(chǎn)證書，生成ca-key.pem ?ca.pem
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

2020/01/15 18:15:15 [INFO] generating a new CA key and certificate from CSR
2020/01/15 18:15:15 [INFO] generate received request
2020/01/15 18:15:15 [INFO] received CSR
2020/01/15 18:15:15 [INFO] generating key: rsa-2048
2020/01/15 18:15:15 [INFO] encoded CSR
2020/01/15 18:15:15 [INFO] signed certificate with serial number 661808851940283859099066838380794010566731982441
指定etcd三個(gè)節(jié)點(diǎn)之間的通信驗(yàn)證

//指定etcd三個(gè)節(jié)點(diǎn)之間的通信驗(yàn)證
cat > server-csr.json <<EOF
{
? ? "CN": "etcd",
? ? "hosts": [
? ? "192.168.35.128",
? ? "192.168.35.195",
? ? "192.168.35.138"
? ? ],
? ? "key": {
? ? ? ? "algo": "rsa",
? ? ? ? "size": 2048
? ? },
? ? "names": [
? ? ? ? {
? ? ? ? ? ? "C": "CN",
? ? ? ? ? ? "L": "BeiJing",
? ? ? ? ? ? "ST": "BeiJing"
? ? ? ? }
? ? ]
}
EOF

生成ETCD證書 server-key.pem ? server.pem

//生成ETCD證書 server-key.pem ? server.pem
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server

www server-csr.json | cfssljson -bare server
2020/01/15 18:24:09 [INFO] generate received request
2020/01/15 18:24:09 [INFO] received CSR
2020/01/15 18:24:09 [INFO] generating key: rsa-2048
2020/01/15 18:24:09 [INFO] encoded CSR
2020/01/15 18:24:09 [INFO] signed certificate with serial number 613252568370198035643630635602034323043189506463
2020/01/15 18:24:09 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
復(fù)制軟件包到centos7中

[root@localhost etcd-cert]# cd /root/k8s/
[root@localhost k8s]# ls ? ? ? ? ? ? ?//直接拉取到目錄下
cfssl.sh ? etcd.sh ? ? ? ? ? ? ? ? ? ? ? ? ?flannel-v0.10.0-linux-amd64.tar.gz
etcd-cert ?etcd-v3.3.10-linux-amd64.tar.gz ?kubernetes-server-linux-amd64.tar.gz

解壓

[root@localhost k8s]# tar zxvf etcd-v3.3.10-linux-amd64.tar.gz

[root@localhost k8s]# ls etcd-v3.3.10-linux-amd64
Documentation ?etcd ?etcdctl ?README-etcdctl.md ?README.md ?READMEv2-etcdctl.md

[root@localhost k8s]# mkdir /opt/etcd/{cfg,bin,ssl} -p ? ?//配置文件，命令文件，證書

[root@localhost k8s]# mv etcd-v3.3.10-linux-amd64/etcd etcd-v3.3.10-linux-amd64/etcdctl /opt/etcd/bin/
證書拷貝

//證書拷貝
[root@localhost k8s]# cp etcd-cert/*.pem /opt/etcd/ssl/

//進(jìn)入卡住狀態(tài)等待其他節(jié)點(diǎn)加入
[root@localhost k8s]# bash etcd.sh etcd01 192.168.35.128 etcd02=https://192.168.35.195:2380,etcd03=https://192.168.35.138:2380

使用另外一個(gè)會(huì)話打開，會(huì)發(fā)現(xiàn)etcd進(jìn)程已經(jīng)開啟

[root@localhost ~]# ps aux | grep etcd
root ? ? ? 4653 ?0.3 ?0.6 10523616 12140 ? ? ? ?Ssl ?19:49 ? 0:00 /opt/etcd/bin/etcd --name=etcd01 --data-dir=/var/lib/etcd/default.etcd --listen-peer-urls=https://192.168.35.128:2380 --listen-client-urls=https://192.168.35.128:2379,http://127.0.0.1:2379 --advertise-client-urls=https://192.168.35.128:2379 --initial-advertise-peer-urls=https://192.168.35.128:2380 --initial-cluster=etcd01=https://192.168.35.128:2380,etcd02=https://192.168.35.195:2380,etcd03=https://192.168.35.138:2380 --initial-cluster-token=etcd-cluster --initial-cluster-state=new --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --peer-cert-file=/opt/etcd/ssl/server.pem --peer-key-file=/opt/etcd/ssl/server-key.pem --trusted-ca-file=/opt/etcd/ssl/ca.pem --peer-trusted-ca-file=/opt/etcd/ssl/ca.pem
root ? ? ? 4719 ?0.0 ?0.0 112676 ? 984 pts/2 ? ?R+ ? 19:50 ? 0:00 grep --color=auto etcd
關(guān)閉防火墻

systemctl stop firewalld.service?
setenforce 0

拷貝證書去其他節(jié)點(diǎn)

[root@localhost k8s]# scp -r /opt/etcd/ root@192.168.35.195:/opt/
[root@localhost k8s]# scp -r /opt/etcd/ root@192.168.35.138:/opt
啟動(dòng)腳本拷貝其他節(jié)點(diǎn)

[root@localhost k8s]# scp /usr/lib/systemd/system/etcd.service root@192.168.35.195:/usr/lib/systemd/system/
[root@localhost k8s]# scp /usr/lib/systemd/system/etcd.service root@192.168.35.138:/usr/lib/systemd/system/
在node01節(jié)點(diǎn)修改
[root@localhost ~]# vim /opt/etcd/cfg/etcd

#[Member]
ETCD_NAME="etcd02"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.35.195:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.35.195:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.35.195:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.35.195:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://192.168.35.128:2380,etcd02=https://192.168.35.195:2380,etcd03=https://192.168.35.138:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

在node02節(jié)點(diǎn)修改
[root@localhost ~]# vim /opt/etcd/cfg/etcd

#[Member]
ETCD_NAME="etcd03"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.35.138:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.35.138:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.35.138:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.35.138:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://192.168.35.128:2380,etcd02=https://192.168.35.195:2380,etcd03=https://192.168.35.138:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

先在master上開啟服務(wù)
[root@localhost system]# cd /root/k8s/
[root@localhost k8s]# bash etcd.sh etcd01 192.168.35.128 etcd02=https://192.168.35.195:2380,etcd03=https://192.168.35.138:2380
在node1和node2上開啟服務(wù)
[root@localhost ~]# systemctl start etcd
[root@localhost ~]# systemctl status etcd
再去master查看，會(huì)發(fā)現(xiàn)同步成功
[root@localhost k8s]# bash etcd.sh etcd01 192.168.35.128 etcd02=https://192.168.35.195:2380,etcd03=https://192.168.35.138:2380

檢查群集狀態(tài)
[root@localhost k8s]# cd etcd-cert/
[root@localhost etcd-cert]# /opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://192.168.35.128:2379,https://192.168.35.195:2379,https://192.168.35.138:2379" cluster-health
member 12a96220ac829a49 is healthy: got healthy result from https://192.168.35.195:2379
member 76797989afd0ecba is healthy: got healthy result from https://192.168.35.128:2379
member ff469df2baaba1da is healthy: got healthy result from https://192.168.35.138:2379
cluster is healthy