kubeadm使用外部etcd集群tls部署kubernetes

環(huán)境：ubuntu 16.04.2

? ? ? ? ? ? ?cpu 4? 內(nèi)存 8G

? ? ? ? ? ? ?內(nèi)核4.4.0-119

ip地址：192.168.0.62

? ? ? ? ? ? ? ? ?192.168.0.63

? ? ? ? ? ? ? ? ?192.168.0.64

?etcd版本： 3.2.12

?kubernetes版本：1.11.5

?

?一、部署etcd集群(需要sudo或者root權(quán)限)

?1生成證書及etcd的二進制文件包，工具下載地址

?wget -O /bin/cfssl? https://pkg.cfssl.org/R1.2/cfssl_linux-amd64?

wget -O /bin/cfssljson https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64?

chmod +x /bin/cfssl*

wget?https://github.com/etcd-io/etcd/releases/download/v3.2.18/etcd-v3.2.18-linux-amd64.tar.gz

tar xf?etcd-v3.2.18-linux-amd64.tar.gz

cp?etcd-v3.2.18-linux-amd64/etcd* /usr/bin/

2.ca-config.json配置文件，修改過期時間為10年（紅色部分）

內(nèi)容如下：

{

? ? "signing": {

? ? ? ? "default": {

? ? ? ? ? ? "expiry": "87600h"

? ? ? ? },

? ? ? ? "profiles": {

? ? ? ? ? ? "etcd": {

? ? ? ? ? ? ? ? "expiry": "87600h",

? ? ? ? ? ? ? ? "usages": [

? ? ? ? ? ? ? ? ? ? "signing",

? ? ? ? ? ? ? ? ? ? "key encipherment",

? ? ? ? ? ? ? ? ? ? "client auth",

? ? ? ? ? ? ? ? ? ? "server auth"

? ? ? ? ? ? ? ? ]

? ? ? ? ? ? }

? ? ? ? }

? ? }

}

3.ca-csr.json配置文件如下：

{

? "CN": "etcd",

? "key": {

? ? "algo": "rsa",

? ? "size": 2048

? },

? "names": [

? ? {

? ? ? "C": "CN",

? ? ? "ST": "shanghai",

? ? ? "L": "shanghai",

? ? ? "O": "etcd",

? ? ? "OU": "System"

? ? }

? ]

}

4.etcd集群的etcd-csr.json

{

? "CN": "etcd",

? "hosts": [

? ? "127.0.0.1",

? ??"192.168.0.62",

? ? "192.168.0.63",

? ? "192.168.0.64"

? ],

? "key": {

? ? "algo": "rsa",

? ? "size": 2048

? },

? "names": [

? ? {

? ? ? "C": "CN",

? ? ? "ST": "shanghai",

? ? ? "L": "shanghai",

? ? ? "O": "etcd",

? ? ? "OU": "System"

? ? }

? ]

}

4.生成證書并自簽名

cfssl gencert -initca ca-csr.json | cfssljson -bare ca

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=etcd etcd-csr.json | cfssljson -bare etcd

復(fù)制pem文件到你指定的目錄，3臺主機都要復(fù)制的

不建議使用目錄/etc/kubernetes/pki/etcd

mkdir -p?/etc/etcdCA

cp *.pem /etc/etcdCA

5.etcd的配置文件如下，紅色部分自行更改。復(fù)制配置文件為/etc/default/etcd

ETCD_NAME=test-node62

ETCD_DATA_DIR="/var/lib/etcd/"

ETCD_LISTEN_PEER_URLS="https://192.168.0.62:2380"

ETCD_LISTEN_CLIENT_URLS="https://192.168.0.62:2379,https://127.0.0.1:4001"

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.62:2380"

ETCD_INITIAL_CLUSTER="test-node62=https://192.168.0.62:2380,test-node63=https://192.168.0.63:2380,test-node64=https://192.168.0.64:2380"

ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-sdn"

ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.62:2379"

CLIENT_CERT_AUTH="true"

ETCD_CA_FILE="/etc/etcdCA/ca.pem"

ETCD_CERT_FILE="/etc/etcdCA/etcd.pem"

ETCD_KEY_FILE="/etc/etcdCA/etcd-key.pem"

PEER_CLIENT_CERT_AUTH="true"

ETCD_PEER_CA_FILE="/etc/etcdCA/ca.pem"

ETCD_PEER_CERT_FILE="/etc/etcdCA/etcd.pem"

ETCD_PEER_KEY_FILE="/etc/etcdCA/etcd-key.pem"

6.創(chuàng)建用戶和服務(wù)并授權(quán)

useradd etcd

chmod 755?/etc/etcdCA/*

echo '[Unit]

Description=etcd - highly-available key value store

Documentation=https://github.com/coreos/etcd

Documentation=man:etcd

After=network.target

Wants=network-online.target

[Service]

Environment=DAEMON_ARGS=

Environment=ETCD_NAME=%H

Environment=ETCD_DATA_DIR=/var/lib/etcd/default

EnvironmentFile=-/etc/default/%p

Type=notify

User=etcd

PermissionsStartOnly=true

#ExecStart=/bin/sh -c "GOMAXPROCS=$(nproc) /usr/bin/etcd $DAEMON_ARGS"

ExecStart=/usr/bin/etcd $DAEMON_ARGS

Restart=on-abnormal

#RestartSec=10s

#LimitNOFILE=65536

[Install]

WantedBy=multi-user.target

Alias=etcd3.service'? ?>/lib/systemd/system/etcd.service

7.啟動服務(wù)

systemctl start etcd

8.檢查集群狀態(tài)

export?ETCDCTL_API=3
etcdctl??\
??--cacert=/etc/etcdCA/ca.pem?\
??--cert=/etc/etcdCA/etcd.pem?\
??--key=/etc/etcdCA/etcd-key.pem?\
??--endpoints=192.168.0.62:2379,192.168.0.63:2379,192.168.0.64:2379?\
??endpoint?health

看到下圖就ok了

二、部署kubernetes

1. 安裝docker-ce （18.06.3）

sudo?apt-get?update
sudo?apt-get?install?\
????apt-transport-https?\
????ca-certificates?\
????curl?\
????software-properties-common
curl?-fsSL?https://download.docker.com/linux/ubuntu/gpg?|?sudo?apt-key?add?-
sudo?apt-key?fingerprint?0EBFCD88
sudo?add-apt-repository?\
???"deb?[arch=amd64]?https://download.docker.com/linux/ubuntu?\
???$(lsb_release?-cs)?\
???stable"
sudo?apt-get?update
sudo?apt-get?install?docker-ce=18.06.3~ce~3-0~ubuntu

2.安裝kubernetes包

apt-get?update?&&?apt-get?install?-y?apt-transport-https
curl?https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg?|?apt-key?add?-?
cat?<<EOF?>/etc/apt/sources.list.d/kubernetes.list
deb?https://mirrors.aliyun.com/kubernetes/apt/?kubernetes-xenial?main
EOF
apt-get?install?-y?kubelet=1.11.5-00?kubeadm=1.11.5-00?kubectl=1.11.5-00

3.使用配置文件進行安裝，配置文件（kubeadm-config.yaml）如下

apiVersion:?kubeadm.k8s.io/v1alpha1
kind:?MasterConfiguration
networking:
??podSubnet:?172.16.0.0/16
??serviceSubnet:?10.96.0.0/12
etcd:
??endpoints:
??-?https://192.168.0.62:2379
??-?https://192.168.0.63:2379
??-?https://192.168.0.64:2379
??caFile:?/etc/etcdCA/ca.pem
??certFile:?/etc/etcdCA/etcd.pem
??keyFile:?/etc/etcdCA/etcd-key.pem
kubernetesVersion:?v1.11.5
kubeProxy:
??config:
????mode:?"ipvs"

4.加載需要的kubernetes鏡像

A="kube-proxy-amd64:v1.11.5
kube-apiserver-amd64:v1.11.5
kube-controller-manager-amd64:v1.11.5
kube-scheduler-amd64::v1.11.5
pause:3.1"
for?i?in?$A;do
	docker?pull?mirrorgooglecontainers/$i
	docker?tag?mirrorgooglecontainers/$i?k8s.gcr.io/$i
done
docker?pull?coredns/coredns:1.1.3
docker?tag?coredns/coredns:1.1.3?k8s.gcr.io/coredns:1.1.3

5.安裝master,出現(xiàn)下圖就master的安裝好了

kubeadm init --config /path/kubeadm-config.yaml

6.授權(quán)客戶端訪問

? mkdir -p $HOME/.kube

? sudo cp -f /etc/kubernetes/admin.conf $HOME/.kube/config

? sudo chown $(id -u):$(id -g) $HOME/.kube/config

7.安裝客戶端（請先執(zhí)行1.安裝docker-ce 2.kubernetes包和4.加載需要的kubernetes鏡像）

執(zhí)行master生成后的kubeadm jion ，需要root或者sudo權(quán)限

如上圖是：

kubeadm join 192.168.0.62:6443 --token 4msj6v.plj3rcsq89c4y4mn --discovery-token-ca-cert-hash sha256:7fb655510bc0af2dda7e401a45932709c473b0f33acef0794924b54715512bbc

三、安裝calico插件

wget?https://github.com/projectcalico/calico/releases/download/v2.6.12/release-v2.6.12.tgz
tar?xf?release-v2.6.12.tgz
cd?release-v2.6.12/k8s-manifests/hosted
sed?-i?'s?http://127.0.0.1:2379?https://192.168.0.62:2379,https://192.168.0.63:2379,https://192.168.0.64:2379?g'?calico.yaml
cat?/etc/etcdCA/etcd-key.pem|base64?-w?0?>?ETCD-KEY
cat?/etc/etcdCA/ca.pem|base64?-w?0?>?ETCD-CA
cat?/etc/etcdCA/etcd.pem|base64?-w?0?>?ETCD-CERT
sed?-i?"s?#?etcd-key:?null?etcd-key:?$(cat?ETCD-KEY)?g"?calico.yaml
sed?-i?"s?#?etcd-ca:?null?etcd-ca:?$(cat?ETCD-CA)?g"?calico.yaml
sed?-i?"s?#?etcd-cert:?null?etcd-cert:?$(cat?ETCD-CERT)?g"?calico.yaml
sed?-i?'s?etcd_ca:?""?etcd_ca:?"/calico-secrets/etcd-ca"?g'?calico.yaml
sed?-i?'s?etcd_cert:?""?etcd_cert:?"/calico-secrets/etcd-cert"?g'?calico.yaml
sed?-i?'s?etcd_key:?""?etcd_key:?"/calico-secrets/etcd-key"?g'?calico.yaml
kubectl?apply?-f?calico.yaml
kubectl?apply?-f?rbac-kdd.yaml

四、查看狀態(tài)

至此k8s的基礎(chǔ)部分完成

補充calico 3.10部分

wget?https://github.com/projectcalico/calico/releases/download/v3.10.2/release-v3.10.2.tgz
tar?xf?release-v3.10.2.tgz
cd?release-v3.10.2/k8s-manifests
sed?-i?'s?http://<ETCD_IP>:<ETCD_PORT>?https://192.168.0.62:2379,https://192.168.0.63:2379,https://192.168.0.64:2379?g'?calico-etcd.yaml
cat?/etc/etcdCA/etcd-key.pem|base64?-w?0?>?ETCD-KEY
cat?/etc/etcdCA/ca.pem|base64?-w?0?>?ETCD-CA
cat?/etc/etcdCA/etcd.pem|base64?-w?0?>?ETCD-CERT
sed?-i?"s?#?etcd-key:?null?etcd-key:?$(cat?ETCD-KEY)?g"?calico-etcd.yaml
sed?-i?"s?#?etcd-ca:?null?etcd-ca:?$(cat?ETCD-CA)?g"?calico-etcd.yaml
sed?-i?"s?#?etcd-cert:?null?etcd-cert:?$(cat?ETCD-CERT)?g"?calico-etcd.yaml
sed?-i?'s?etcd_ca:?""?etcd_ca:?"/calico-secrets/etcd-ca"?g'?calico-etcd.yaml
sed?-i?'s?etcd_cert:?""?etcd_cert:?"/calico-secrets/etcd-cert"?g'?calico-etcd.yaml
sed?-i?'s?etcd_key:?""?etcd_key:?"/calico-secrets/etcd-key"?g'?calico-etcd.yaml
sed?-i?'s?192.168.0.0/16?172.16.0.0/16?g'?calico-etcd.yaml
kubectl?apply?-f?calico-etcd.yaml

注意：加密的etcd集群和明文的etcd集群不能通用