溫馨提示×
您好，登錄后才能下訂單哦！
點(diǎn)擊 登錄注冊(cè) 即表示同意《億速云用戶(hù)服務(wù)條款》
您好，登錄后才能下訂單哦！
搭建K8S集群 kubernetes 1.11.3
1.1 實(shí)驗(yàn)架構(gòu)：
kubernetes架構(gòu)
node1: master 10.192.44.129
node2: node2 10.192.44.127
node3: node3 10.192.44.126
etcd架構(gòu)
node1: master 10.192.44.129
node2: node 10.192.44.127
node3: node 10.192.44.126
harbor服務(wù)器
redhat128.example.com
10.192.44.128
2.安裝
2.1配置系統(tǒng)相關(guān)參數(shù)（每臺(tái)）：
2.1.1 臨時(shí)禁用selinux
sed -i 's/SELINUX=permissive/SELINUX=disabled/' /etc/sysconfig/selinux
setenforce 0
2.1.2 臨時(shí)關(guān)閉swap ,永久關(guān)閉直接注釋fstab中swap行
swapoff -a
2.1.3 開(kāi)啟forward
iptables -P FORWARD ACCEPT
2.1.3 配置轉(zhuǎn)發(fā)相關(guān)參數(shù)，否則可能會(huì)出錯(cuò)
cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
vm.swappiness=0
EOF
sysctl --system
2.1.4 配置hosts
10.192.44.126 node3
10.192.44.127 node2
10.192.44.128 redhat128
10.192.44.129 node1
2.1.5 安裝docker 參考我此前的blog。
2.1.6 時(shí)間同步
yum install ntpdate -y &&ntpdate 0.asia.pool.ntp.org
3.創(chuàng)建TLS證書(shū)和秘鑰（master節(jié)點(diǎn)）
3.1 生成的證書(shū)文件如下：
ca-key.pem #根私鑰
ca.pem #根證書(shū)
kubernetes-key.pem #集群私鑰
kubernetes.pem #集群證書(shū)
kube-proxy.pem #proxy私鑰-node節(jié)點(diǎn)進(jìn)行認(rèn)證
kube-proxy-key.pem #proxy證書(shū)-node節(jié)點(diǎn)進(jìn)行認(rèn)證
admin.pem #管理員私鑰-主要用于kubectl認(rèn)證
admin-key.pem #管理員證書(shū)-主要用于kubectl認(rèn)證
知識(shí)點(diǎn)補(bǔ)充：
TLS: TLS 的作用就是對(duì)通訊加密，防止中間人竊聽(tīng)；同時(shí)如果證書(shū)不信任的話(huà)根本就無(wú)法與 apiserver 建立連接，更不用提有沒(méi)有權(quán)限向 apiserver 請(qǐng)求指定內(nèi)容。
RBAC作用：RBAC 中規(guī)定了一個(gè)用戶(hù)或者用戶(hù)組(subject)具有請(qǐng)求哪些 api 的權(quán)限；在配合 TLS 加密的時(shí)候，實(shí)際上 apiserver 讀取客戶(hù)端證書(shū)的 CN 字段作為用戶(hù)名，讀取 O 字段作為用戶(hù)組。
總結(jié)：想要與 apiserver 通訊就必須采用由 apiserver CA 簽發(fā)的證書(shū)，這樣才能形成信任關(guān)系，建立 TLS 連接；第二，可以通過(guò)證書(shū)的 CN、O 字段來(lái)提供 RBAC 所需的用戶(hù)與用戶(hù)組。
3.2 下載安裝CFSSL(用于簽名，驗(yàn)證和捆綁TLS證書(shū)的HTTP API工具)（master節(jié)點(diǎn)）
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 /usr/local/bin/cfssljson
chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssl-certinfo /usr/local/bin/cfssljson
3.3創(chuàng)建CA(Certificate Authority)（master節(jié)點(diǎn)）
mkdir /root/ssl
cd /root/ssl
cfssl print-defaults config > config.json
cfssl print-defaults csr > csr.json
# 根據(jù)config.json文件的格式創(chuàng)建如下的ca-config.json文件
# 過(guò)期時(shí)間設(shè)置成了 87600h
cat > ca-config.json <<EOF
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"kubernetes": {
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
],
"expiry": "87600h"
}
}
}
}
EOF
知識(shí)點(diǎn)：
ca-config.json：可以定義多個(gè) profiles，分別指定不同的過(guò)期時(shí)間、使用場(chǎng)景等參數(shù)；后續(xù)在簽名證書(shū)時(shí)使用某個(gè) profile；
signing：表示該證書(shū)可用于簽名其它證書(shū)；生成的 ca.pem 證書(shū)中 CA=TRUE；
server auth：表示client可以用該 CA 對(duì)server提供的證書(shū)進(jìn)行驗(yàn)證；
client auth：表示server可以用該CA對(duì)client提供的證書(shū)進(jìn)行驗(yàn)證；
3.4 創(chuàng)建證書(shū)請(qǐng)求
cat > ca-csr.json <<EOF
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "GuangDong",
"L": "ShenZhen",
"O": "k8s",
"OU": "System"
}
],
"ca": {
"expiry": "87600h"
}
}
EOF
知識(shí)點(diǎn)：
"CN"：Common Name，kube-apiserver 從證書(shū)中提取該字段作為請(qǐng)求的用戶(hù)名 (User Name)
"O"：Organization，kube-apiserver 從證書(shū)中提取該字段作為請(qǐng)求用戶(hù)所屬的組 (Group)
3.5 生成CA證書(shū)和私鑰
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
3.6 創(chuàng)建kubernetes證書(shū)請(qǐng)求文件
cat > kubernetes-csr.json <<EOF
{
"CN": "kubernetes",
"hosts": [
"127.0.0.1",
"10.192.44.129",
"10.192.44.128",
"10.192.44.126",
"10.192.44.127",
"10.254.0.1",
"*.kubernetes.master",
"localhost",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "GuangDong",
"L": "ShenZhen",
"O": "k8s",
"OU": "System"
}
]
}
EOF
知識(shí)點(diǎn)：
這個(gè)證書(shū)目前專(zhuān)屬于 apiserver加了一個(gè) *.kubernetes.master 域名以便內(nèi)部私有 DNS 解析使用(可刪除)；至于很多人問(wèn)過(guò) kubernetes 這幾個(gè)能不能刪掉，答案是不可以的；因?yàn)楫?dāng)集群創(chuàng)建好后，default namespace 下會(huì)創(chuàng)建一個(gè)叫 kubenretes 的 svc，有一些組件會(huì)直接連接這個(gè) svc 來(lái)跟 api 通訊的，證書(shū)如果不包含可能會(huì)出現(xiàn)無(wú)法連接的情況；其他幾個(gè) kubernetes 開(kāi)頭的域名作用相同
hosts包含的是授權(quán)范圍，不在此范圍的的節(jié)點(diǎn)或者服務(wù)使用此證書(shū)就會(huì)報(bào)證書(shū)不匹配錯(cuò)誤。
10.254.0.1是指kube-apiserver 指定的 service-cluster-ip-range 網(wǎng)段的第一個(gè)IP。
3.7 生成kubernetes證書(shū)和私鑰
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes
3.8 創(chuàng)建admin證書(shū)
cat > admin-csr.json <<EOF
{
"CN": "admin",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "GuangDong",
"L": "ShenZhen",
"O": "system:masters",
"OU": "System"
}
]
}
EOF
3.9 生成admin證書(shū)和私鑰
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin
知識(shí)點(diǎn)：
這個(gè)admin 證書(shū)，是將來(lái)生成管理員用的kube config 配置文件用的，現(xiàn)在我們一般建議使用RBAC 來(lái)對(duì)kubernetes 進(jìn)行角色權(quán)限控制， kubernetes 將證書(shū)中的CN 字段 作為User， O 字段作為 Group
3.10 創(chuàng)建Kube-proxy 證書(shū)
cat > kube-proxy-csr.json <<EOF
{
"CN": "system:kube-proxy",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "GuangDong",
"L": "ShenZhen",
"O": "k8s",
"OU": "System"
}
]
}
EOF
3.11 生成kube-proxy客戶(hù)端證書(shū)和私鑰
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
3.12 校驗(yàn)證書(shū)
openssl x509 -noout -text -in kubernetes.pem
3.13分發(fā)證書(shū) 將生成的證書(shū)和秘鑰文件（后綴名為.pem）拷貝到所有機(jī)器的 /etc/kubernetes/ssl 目錄下備用
mkdir -p /etc/kubernetes/ssl
scp *.pem {node2,node3}:/etc/kubernetes/ssl
4.創(chuàng)建kubeconfig文件 （master節(jié)點(diǎn)）
4.1 生成token文件
export BOOTSTRAP_TOKEN=$(head -c 16 /dev/urandom | od -An -t x | tr -d ' ')
export KUBE_APISERVER="https://10.192.44.129:6443"
echo $BOOTSTRAP_TOKEN
cat > token.csv <<EOF
${BOOTSTRAP_TOKEN},kubelet-bootstrap,10001,"system:bootstrappers"
EOF
cp token.csv /etc/kubernetes/
知識(shí)點(diǎn)：不要質(zhì)疑 system:bootstrappers 用戶(hù)組是否寫(xiě)錯(cuò)了，有疑問(wèn)請(qǐng)參考官方文， https://kubernetes.io/docs/admin/kubelet-tls-bootstrapping/
4.2 創(chuàng)建kubelete-kubeconfig文件
kubeconfig 設(shè)置其實(shí)是權(quán)限配置文件，是對(duì)k8s集群層面的訪(fǎng)問(wèn)控制。如果不使用--kubeconfig=xx.kubeconfig,他就會(huì)默認(rèn)保存在~/.kube/conf中文件，然后作為默認(rèn)配置文件。其實(shí)通過(guò)kubeadm配置也會(huì)發(fā)現(xiàn)，他要求你將kubeconfig拷貝到~/.kube/conf。
cd /etc/kubernetes/ssl
4.2.1設(shè)置集群參數(shù)
kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=bootstrap.kubeconfig
4.2.2設(shè)置客戶(hù)端認(rèn)證參數(shù)
kubectl config set-credentials kubelet-bootstrap \
--token=${BOOTSTRAP_TOKEN} \
--kubeconfig=bootstrap.kubeconfig
4.2.3設(shè)置上下文參數(shù)
kubectl config set-context default \
--cluster=kubernetes \
--user=kubelet-bootstrap \
--kubeconfig=bootstrap.kubeconfig
4.2.4設(shè)置默認(rèn)上下文
kubectl config use-context default --kubeconfig=bootstrap.kubeconfig
4.3 創(chuàng)建kube-proxy文件
4.3.1 設(shè)置集群參數(shù)
kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=kube-proxy.kubeconfig
4.3.2 設(shè)置客戶(hù)端認(rèn)證參數(shù)
kubectl config set-credentials kube-proxy \
--client-certificate=/etc/kubernetes/ssl/kube-proxy.pem \
--client-key=/etc/kubernetes/ssl/kube-proxy-key.pem \
--embed-certs=true \
--kubeconfig=kube-proxy.kubeconfig
4.3.3 設(shè)置上下文參數(shù)
kubectl config set-context default \
--cluster=kubernetes \
--user=kube-proxy \
--kubeconfig=kube-proxy.kubeconfig
4.3.4 設(shè)置默認(rèn)上下文
kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
4.4 分發(fā)kubeconfig 證書(shū)
scp bootstrap.kubeconfig kube-proxy.kubeconfig {node2,node3}:/etc/kubernetes/
4.5 創(chuàng)建 admin kubeconfig文件
4.5.1 設(shè)置集群參數(shù)
kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=admin.conf
4.5.2設(shè)置客戶(hù)端認(rèn)證參數(shù)
kubectl config set-credentials admin \
--client-certificate=/etc/kubernetes/ssl/admin.pem \
--embed-certs=true \
--client-key=/etc/kubernetes/ssl/admin-key.pem \
--kubeconfig=admin.conf
4.5.3設(shè)置上下文參數(shù)
kubectl config set-context default \
--cluster=kubernetes \
--user=admin \
--kubeconfig=admin.conf
4.5.4設(shè)置默認(rèn)上下文
kubectl config use-context default --kubeconfig=admin.conf
4.6 創(chuàng)建高級(jí)審計(jì)文件
cat >> audit-policy.yaml <<EOF
# Log all requests at the Metadata level.
apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
- level: Metadata
EOF
4.7 文件拷貝：
#cp ~/.kube/config /etc/kubernetes/kubelet.kubeconfig (#關(guān)于這一步當(dāng)時(shí)我是添加node節(jié)點(diǎn)出問(wèn)題，如果沒(méi)有問(wèn)題請(qǐng)忽略這操作，下面的kubelet.kubeconfig一樣）
scp /etc/kubernetes/{kubelet.kubeconfig,bootstrap.kubeconfig,kube-proxy.kubeconfig} node2:/etc/kubernetes/
scp /etc/kubernetes/{kubelet.kubeconfig,bootstrap.kubeconfig,kube-proxy.kubeconfig} node3:/etc/kubernetes/
5 創(chuàng)建etcd集群
5.1創(chuàng)建etcd啟動(dòng)服務(wù)（每臺(tái)）
cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos
[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
ExecStart=/usr/local/bin/etcd \
--name ${ETCD_NAME} \
--cert-file=/etc/kubernetes/ssl/kubernetes.pem \
--key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
--peer-cert-file=/etc/kubernetes/ssl/kubernetes.pem \
--peer-key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
--trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
--peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
--initial-advertise-peer-urls ${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--listen-peer-urls ${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls ${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
--advertise-client-urls ${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-cluster-token ${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster infra1=https://172.20.0.113:2380,infra2=https://172.20.0.114:2380,infra3=https://172.20.0.115:2380 \
--initial-cluster-state new \
--data-dir=${ETCD_DATA_DIR}
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
知識(shí)點(diǎn)： systemd是配置管理驅(qū)動(dòng)服務(wù)的。 環(huán)境變量 = -/ "-"表示抑制錯(cuò)誤，即發(fā)生錯(cuò)誤的時(shí)候，也不影響其他命令的執(zhí)行。
5.2 編輯配置文件(以ectd1為例，etcd2,etcd3注意替換IP地址)
mkdir /etc/etcd && vim /etc/etcd/etcd.conf
cat > /etc/etcd/etcd.conf << EOF
ETCD_NAME=infra1
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_LISTEN_PEER_URLS="https://10.192.44.129:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.192.44.129:2379"
#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.192.44.129:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="https://10.192.44.129:2379"
5.3啟動(dòng)etcd服務(wù)器，記得創(chuàng)建/var/lib/etcd。
mkdir /var/lib/etcd
systemctl enable etcd && systemctl start etcd
6 部署master節(jié)點(diǎn)：（好像需要自己到服務(wù)器文件解壓）
6.1 下載kubernetes 文件
下載kubernetes (v1.11.3)
wget https://github.com/kubernetes/kubernetes/releases/download/v1.11.3/kubernetes.tar.gz
tar -xzvf kubernetes.tar.gz
cd kubernetes
./cluster/get-kube-binaries.sh
#如果不行，請(qǐng)手動(dòng)操作
cd server/
tar -zxvf kubernetes-server-linux-amd64.tar.gz
cp kubernetes/server/bin/kube-apiserver /usr/local/bin/kube-apiserver
cp kubernetes/server/bin/kube-controller-manager /usr/local/bin/kube-controller-manager
cp kubernetes/server/bin/kube-scheduler /usr/local/bin/kube-scheduler
chmod +x /usr/local/bin/{kube-apiserver,kube-controller-manager,kube-scheduler}
6.2配置系統(tǒng)服務(wù)啟動(dòng)kube-apiserver，kube-controller-manager,kube-scheduler
6.2.1創(chuàng)建kube-apiserver.service
cat > /usr/lib/systemd/system/kube-apiserver.service << EOF
[Unit]
Description=Kubernetes API Service
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
After=etcd.service
[Service]
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/apiserver
ExecStart=/usr/local/bin/kube-apiserver \
$KUBE_LOGTOSTDERR \
$KUBE_LOG_LEVEL \
$KUBE_ETCD_SERVERS \
$KUBE_API_ADDRESS \
$KUBE_API_PORT \
$KUBELET_PORT \
$KUBE_ALLOW_PRIV \
$KUBE_SERVICE_ADDRESSES \
$KUBE_ADMISSION_CONTROL \
$KUBE_API_ARGS
Restart=on-failure
Type=notify
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
6.2.2 創(chuàng)建kube-controller-manager.service
cat > /usr/lib/systemd/system/kube-controller-manager.service << EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/controller-manager
ExecStart=/usr/local/bin/kube-controller-manager \
$KUBE_LOGTOSTDERR \
$KUBE_LOG_LEVEL \
$KUBE_MASTER \
$KUBE_CONTROLLER_MANAGER_ARGS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
6.2.3 創(chuàng)建kube-scheduler.service
cat > /usr/lib/systemd/system/kube-scheduler.service << EOF
[Unit]
Description=Kubernetes Scheduler Plugin
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/scheduler
ExecStart=/usr/local/bin/kube-scheduler \
$KUBE_LOGTOSTDERR \
$KUBE_LOG_LEVEL \
$KUBE_MASTER \
$KUBE_SCHEDULER_ARGS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
6.2.4 編輯/etc/kubernetes/config文件
cat > /etc/kubernetes/config << EOF
###
# kubernetes system config
#
# The following values are used to configure various aspects of all
# kubernetes services, including
#
# kube-apiserver.service
# kube-controller-manager.service
# kube-scheduler.service
# kubelet.service
# kube-proxy.service
# logging to stderr means we get it in the systemd journal
KUBE_LOGTOSTDERR="--logtostderr=true"
# journal message level, 0 is debug
KUBE_LOG_LEVEL="--v=0"
# Should this cluster be allowed to run privileged docker containers
KUBE_ALLOW_PRIV="--allow-privileged=true"
# How the controller-manager, scheduler, and proxy find the apiserver
#KUBE_MASTER="--master=http://test-001.jimmysong.io:8080"
KUBE_MASTER="--master=http://10.192.44.129:8080"
EOF
6.2.5 編輯apiserver配置文件
cat > /etc/kubernetes/apiserver << EOF
###
## kubernetes system config
##
## The following values are used to configure the kube-apiserver
##
#
## The address on the local server to listen to.
#KUBE_API_ADDRESS="--insecure-bind-address=test-001.jimmysong.io"
KUBE_API_ADDRESS="--advertise-address=10.192.44.129 --bind-address=10.192.44.129 --insecure-bind-address=10.192.44.129"
#
## The port on the local server to listen on.
KUBE_API_PORT="--secure-port=6443"
#
## Port minions listen on
#KUBELET_PORT="--kubelet-port=10250"
#
## Comma separated list of nodes in the etcd cluster
KUBE_ETCD_SERVERS="--etcd-servers=https://10.192.44.129:2379,https://10.192.44.127:2379,https://10.192.44.126:2379"
#
## Address range to use for services
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"
#
## default admission control policies
KUBE_ADMISSION_CONTROL="--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction"
#
## Add your own!
KUBE_API_ARGS="--anonymous-auth=false \
--authorization-mode=Node,RBAC \
--kubelet-https=true \
--kubelet-timeout=3s \
--enable-bootstrap-token-auth \
--enable-garbage-collector \
--enable-logs-handler \
--token-auth-file=/etc/kubernetes/token.csv \
--service-node-port-range=30000-32767 \
--tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem \
--tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
--client-ca-file=/etc/kubernetes/ssl/ca.pem \
--service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \
--etcd-cafile=/etc/kubernetes/ssl/ca.pem \
--etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem \
--etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem \
--etcd-compaction-interval=5m0s \
--etcd-count-metric-poll-period=1m0s \
--enable-swagger-ui=true \
--apiserver-count=3 \
--log-flush-frequency=5s \
--audit-log-maxage=30 \
--audit-log-maxbackup=3 \
--audit-log-maxsize=100 \
--audit-log-path=/var/lib/audit.log \
--audit-policy-file=/etc/kubernetes/audit-policy.yaml \
--storage-backend=etcd3 \
--event-ttl=1h"
EOF
6.2.6 編輯controller-manager配置文件
cat > /etc/kubernetes/controller-manager << EOF
###
# The following values are used to configure the kubernetes controller-manager
# defaults from config and apiserver should be adequate
# Add your own!
KUBE_CONTROLLER_MANAGER_ARGS="--address=127.0.0.1
--service-cluster-ip-range=10.254.0.0/16
--cluster-name=kubernetes
--cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem
--cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem
--service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem
--root-ca-file=/etc/kubernetes/ssl/ca.pem
--leader-elect=true"
EOF
6.2.7 編輯scheduler配置文件
cat > /etc/kubernetes/scheduler << EOF
###
# kubernetes scheduler config
# default config should be adequate
# Add your own!
KUBE_SCHEDULER_ARGS="--leader-elect=true --address=127.0.0.1 --algorithm-provider=DefaultProvider"
6.2.8 啟動(dòng)服務(wù)
systemctl daemon-reload
systemctl enable kueb-apiserver kube-controller-manager kube-scheduler
systemctl start kueb-apiserver kube-controller-manager kube-scheduler
6.2.9 驗(yàn)證master節(jié)點(diǎn)功能
kubectl get componentstatuses
如下：
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {"health": "true"}
etcd-1 Healthy {"health": "true"}
etcd-2 Healthy {"health": "true"}
6.2.10 kubectl命令補(bǔ)全
echo "source <(kubectl completion bash)" >> ~/.bashrc
source ~/.bashrc
7. 安裝flannel網(wǎng)絡(luò)插件
7.1 通過(guò)yum安裝配置flannel（每節(jié)點(diǎn)）
yum install -y flannel
7.2 配置服務(wù)文件（每節(jié)點(diǎn)）
cat > /usr/lib/systemd/system/flanneld.service << EOF
[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
After=network-online.target
Wants=network-online.target
After=etcd.service
Before=docker.service
[Service]
Type=notify
EnvironmentFile=/etc/sysconfig/flanneld
EnvironmentFile=-/etc/sysconfig/docker-network
ExecStart=/usr/bin/flanneld-start \
-etcd-endpoints=${FLANNEL_ETCD_ENDPOINTS} \
-etcd-prefix=${FLANNEL_ETCD_PREFIX} \
$FLANNEL_OPTIONS
ExecStartPost=/usr/libexec/flannel/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker
Restart=on-failure
[Install]
WantedBy=multi-user.target
RequiredBy=docker.service
知識(shí)點(diǎn)：mk-docker-opts.sh生成環(huán)境變量/run/flannel/subnet.env，/run/docker_opts.env。后續(xù)要docker要調(diào)用其配置文件。
7.3 創(chuàng)建flanneld配置文件（每節(jié)點(diǎn)）
cat > /etc/sysconfig/flanneld << EOF
# Flanneld configuration options
# etcd url location. Point this to the server where etcd runs
FLANNEL_ETCD_ENDPOINTS="https://10.192.44.129:2379,https://10.192.44.127:2379,https://10.192.44.126:2379"
# etcd config key. This is the configuration key that flannel queries
# For address range assignment
FLANNEL_ETCD_PREFIX="/kube-centos/network"
# Any additional options that you want to pass
FLANNEL_OPTIONS="-etcd-cafile=/etc/kubernetes/ssl/ca.pem -etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem -etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem"
7.4 在etcd創(chuàng)建網(wǎng)絡(luò)配置（每節(jié)點(diǎn)，gw模式）
etcdctl --endpoints=https://10.192.44.129:2379,https://10.192.44.127:2379,https://10.192.44.126:2379 \
--ca-file=/etc/kubernetes/ssl/ca.pem \
--cert-file=/etc/kubernetes/ssl/kubernetes.pem \
--key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
mkdir /kube-centos/network
etcdctl --endpoints=https://10.192.44.129:2379,https://10.192.44.127:2379,https://10.192.44.126:2379 \
--ca-file=/etc/kubernetes/ssl/ca.pem \
--cert-file=/etc/kubernetes/ssl/kubernetes.pem \
--key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
mk /kube-centos/network/config '{"Network":"172.30.0.0/16","SubnetLen":24,"Backend":{"Type":"host-gw"}}'
7.5 啟動(dòng)flannel（每節(jié)點(diǎn)）
systemctl daemon-reload
systemctl restart flanneld
systemctl start flanneld
7.6 查看etcd內(nèi)容(隨便一個(gè)節(jié)點(diǎn)執(zhí)行就行了，因?yàn)閿?shù)據(jù)是同步的)
etcdctl --endpoints=https://10.192.44.129:2379 \
--ca-file=/etc/kubernetes/ssl/ca.pem \
--cert-file=/etc/kubernetes/ssl/kubernetes.pem \
--key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
ls /kube-centos/network/subnets
etcdctl --endpoints=https://10.192.44.129:2379 \
--ca-file=/etc/kubernetes/ssl/ca.pem \
--cert-file=/etc/kubernetes/ssl/kubernetes.pem \
--key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
get /kube-centos/network/config
7.7 將flannel啟動(dòng)后生成的環(huán)境變量添加到docker的systemd目錄。（每節(jié)點(diǎn)）
vim /usr/lib/systemd/system/docker.service
EnvironmentFile=-/run/flannel/docker
systemctl daemon-reload && systemctl restart docker
7.8 更改dockerd啟動(dòng)配置(每節(jié)點(diǎn))
vim /usr/lib/systemd/system/docker.service
EnvironmentFile=-/run/flannel/docker
ExecStart=/usr/bin/dockerd \
$DOCKER_OPT_BIP \
$DOCKER_OPT_IPMASQ \
$DOCKER_OPT_MTU \
--log-driver=json-file
8.部署node節(jié)點(diǎn)
8.1 TLS bootstrapping配置(master節(jié)點(diǎn))
cd /etc/kubernetes
kubectl create clusterrolebinding kubelet-bootstrap \
--clusterrole=system:node-bootstrapper \
--user=kubelet-bootstrap
kubectl create clusterrolebinding kubelet-nodes \
--clusterrole=system:node \
--group=system:nodes
知識(shí)點(diǎn)：
kubelet 啟動(dòng)時(shí)向 kube-apiserver 發(fā)送 TLS bootstrapping 請(qǐng)求，需要先將 bootstrap token 文件中的 kubelet-bootstrap 用戶(hù)賦予 system:node-bootstrapper cluster 角色(role)， 然后 kubelet 才能有權(quán)限創(chuàng)建認(rèn)證請(qǐng)求(certificate signing requests)：
kubelet 通過(guò)認(rèn)證后向 kube-apiserver 發(fā)送 register node 請(qǐng)求，需要先將 kubelet-nodes 用戶(hù)賦予 system:node cluster角色(role) 和 system:nodes 組(group)， 然后 kubelet 才能有權(quán)限創(chuàng)建節(jié)點(diǎn)請(qǐng)求：
8.2 下載kubelet和kube-proxy 二進(jìn)制文件（每節(jié)點(diǎn)）
wget https://github.com/kubernetes/kubernetes/releases/download/v1.11.3/kubernetes.tar.gz
tar -xzvf kubernetes.tar.gz
cd kubernetes
./cluster/get-kube-binaries.sh
#如果不行，請(qǐng)手動(dòng)操作
cd server/
tar -zxvf kubernetes-server-linux-amd64.tar.gz
cp kubernetes/server/bin/kubelet /usr/local/bin/kubelet
cp kubernetes/server/bin/kube-proxy /usr/local/bin/kube-proxy
chmod +x /usr/local/bin/{kubelet,kube-proxy}
8.3 配置系統(tǒng)服務(wù)啟動(dòng)kubelet，kube-proxy
8.3.1 創(chuàng)建kubelete
cat > /usr/lib/systemd/system/kubelet.service << EOF
[Unit]
Description=Kubernetes Kubelet Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service
[Service]
WorkingDirectory=/var/lib/kubelet
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/kubelet
ExecStart=/usr/local/bin/kubelet \
$KUBE_LOGTOSTDERR \
$KUBE_LOG_LEVEL \
$KUBELET_ADDRESS \
$KUBELET_PORT \
$KUBELET_HOSTNAME \
$KUBE_ALLOW_PRIV \
$KUBELET_POD_INFRA_CONTAINER \
$KUBELET_ARGS
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
8.3.2 創(chuàng)建Kube-proxy
cat > /usr/lib/systemd/system/kube-proxy.service << EOF
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
[Service]
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/proxy
ExecStart=/usr/local/bin/kube-proxy \
$KUBE_LOGTOSTDERR \
$KUBE_LOG_LEVEL \
$KUBE_MASTER \
$KUBE_PROXY_ARGS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
8.3.3 創(chuàng)建conf文件
cd /etc/kubernetes
cat >/etc/kubernetes/config<<EOF
KUBE_LOGTOSTDERR="--logtostderr=true"
KUBE_LOG_LEVEL="--v=2"
EOF
8.3.4 創(chuàng)建kubelete-conf文件(master)
cat > /etc/kubernetes/kubelet << EOF
###
## kubernetes kubelet (minion) config
#
## The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)
KUBELET_ADDRESS="--address=10.192.44.129"
#
## The port for the info server to serve on
#KUBELET_PORT="--port=10250"
#
## You may leave this blank to use the actual hostname
KUBELET_HOSTNAME="--hostname-override=master"
#
## location of the api-server
## COMMENT THIS ON KUBERNETES 1.8+
#
## pod infrastructure container
KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=gcr.io/google_containers/pause-amd64:3.0"
#
## Add your own!
KUBELET_ARGS="--cluster-dns=10.254.0.2 --bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --cert-dir=/etc/kubernetes/ssl --cluster-domain=cluster.local. --allow-privileged=true --serialize-image-pulls=false --fail-swap-on=false --log-dir=/var/log/kubernetes/kubelet"
EOF
8.3.5 創(chuàng)建kubelete-conf文件(node2)
cat > /etc/kubernetes/kubelet << EOF
###
## kubernetes kubelet (minion) config
#
## The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)
KUBELET_ADDRESS="--address=10.192.44.127"
#
## The port for the info server to serve on
#KUBELET_PORT="--port=10250"
#
## You may leave this blank to use the actual hostname
KUBELET_HOSTNAME="--hostname-override=node2"
#
## location of the api-server
## COMMENT THIS ON KUBERNETES 1.8+
#
## pod infrastructure container
KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=gcr.io/google_containers/pause-amd64:3.0"
#
## Add your own!
KUBELET_ARGS="--cluster-dns=10.254.0.2 --bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --cert-dir=/etc/kubernetes/ssl --cluster-domain=cluster.local. --allow-privileged=true --serialize-image-pulls=false --fail-swap-on=false --log-dir=/var/log/kubernetes/kubelet"
EOF
8.3.6 創(chuàng)建kubelete-conf文件(node3)
cat > /etc/kubernetes/kubelet << EOF
###
## kubernetes kubelet (minion) config
#
## The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)
KUBELET_ADDRESS="--address=10.192.44.126"
#
## The port for the info server to serve on
#KUBELET_PORT="--port=10250"
#
## You may leave this blank to use the actual hostname
KUBELET_HOSTNAME="--hostname-override=node3"
#
## location of the api-server
## COMMENT THIS ON KUBERNETES 1.8+
#
## pod infrastructure container
KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=gcr.io/google_containers/pause-amd64:3.0"
#
## Add your own!
KUBELET_ARGS="--cluster-dns=10.254.0.2 --bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --cert-dir=/etc/kubernetes/ssl --cluster-domain=cluster.local. --allow-privileged=true --serialize-image-pulls=false --fail-swap-on=false --log-dir=/var/log/kubernetes/kubelet"
EOF
8.3.7 創(chuàng)建kube-proxy文件(master)
cat > /etc/kubernetes/proxy << EOF
###
# kubernetes proxy config
# default config should be adequate
# Add your own!
KUBE_PROXY_ARGS="--bind-address=10.192.44.129 --hostname-override=master --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig --cluster-cidr=10.254.0.0/16 --log-dir=/var/log/kubernetes/proxy"
EOF
8.3.7 創(chuàng)建kube-proxy文件(node2)
cat > /etc/kubernetes/proxy << EOF
###
# kubernetes proxy config
# default config should be adequate
# Add your own!
KUBE_PROXY_ARGS="--bind-address=10.192.44.127 --hostname-override=node2 --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig --cluster-cidr=10.254.0.0/16 --log-dir=/var/log/kubernetes/proxy"
EOF
8.3.8 創(chuàng)建kube-proxy文件(node3)
cat > /etc/kubernetes/proxy << EOF
###
# kubernetes proxy config
# default config should be adequate
# Add your own!
KUBE_PROXY_ARGS="--bind-address=10.192.44.126 --hostname-override=node3 --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig --cluster-cidr=10.254.0.0/16 --log-dir=/var/log/kubernetes/proxy"
EOF
8.3.9 啟動(dòng)kubelet
systemctl daemon-reload && systemctl enable kubelet && systemctl start kubelet
8.3.10 啟動(dòng)kube-proxy
systemctl daemon-reload && systemctl enable kube-proxy && systemctl start kube-proxy
8.3.11 查看證書(shū)申請(qǐng)請(qǐng)求（node節(jié)點(diǎn)自動(dòng)去kubeapi節(jié)點(diǎn)申請(qǐng)）
kubectl get csr
8.3.12 master節(jié)點(diǎn)允許請(qǐng)求 ，查看證書(shū)請(qǐng)求狀態(tài)
kubectl certificate approve node-csr-Yiiv675wUCvQl3HH11jDr0cC9p3kbrXWrxvG3EjWGoE
kubectl describe csr node-csr-Yiiv675wUCvQl3HH11jDr0cC9p3kbrXWrxvG3EjWGoE
狀態(tài)標(biāo)注下如下：
kubectl describe csr node-csr-hsBS9OyhOa8rK_Q48ee81giH17t6Nk4FL9IRWRt4ygw
Name: node-csr-hsBS9OyhOa8rK_Q48ee81giH17t6Nk4FL9IRWRt4ygw
Labels: <none>
Annotations: <none>
CreationTimestamp: Thu, 22 Nov 2018 20:19:09 +0800
Requesting User: kubelet-bootstrap
Status: Approved,Issued
Subject:
Common Name: system:node:node3
Serial Number:
Organization: system:nodes
8.3.13 查看節(jié)點(diǎn)狀態(tài)
kubectl get nodes
8.3.14 創(chuàng)建測(cè)試
vim deploy.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nginx-deployment
spec:
replicas: 2
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: ngxin:1.7.9
ports:
- containerPort: 80
kubectl create -f deploy.yaml
kubectl scale deployment nginx --replicas=4
9.部署集群DNS(CoreDNS)
9.1 下載coredns配置文件，如下:
coredns.yaml.sed
apiVersion: v1
kind: ServiceAccount
metadata:
name: coredns
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:coredns
rules:
- apiGroups:
- ""
resources:
- endpoints
- services
- pods
- namespaces
verbs:
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:coredns
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:coredns
subjects:
- kind: ServiceAccount
name: coredns
namespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:
name: coredns
namespace: kube-system
data:
Corefile: |
.:53 {
errors
health
kubernetes CLUSTER_DOMAIN REVERSE_CIDRS {
pods insecure
upstream
fallthrough in-addr.arpa ip6.arpa
}
prometheus :9153
proxy . /etc/resolv.conf
cache 30
}
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: coredns
namespace: kube-system
labels:
k8s-app: kube-dns
kubernetes.io/name: "CoreDNS"
spec:
replicas: 2
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
selector:
matchLabels:
k8s-app: kube-dns
template:
metadata:
labels:
k8s-app: kube-dns
spec:
serviceAccountName: coredns
tolerations:
- key: "CriticalAddonsOnly"
operator: "Exists"
containers:
- name: coredns
image: coredns/coredns:1.1.1
imagePullPolicy: IfNotPresent
args: [ "-conf", "/etc/coredns/Corefile" ]
volumeMounts:
- name: config-volume
mountPath: /etc/coredns
ports:
- containerPort: 53
name: dns
protocol: UDP
- containerPort: 53
name: dns-tcp
protocol: TCP
- containerPort: 9153
name: metrics
protocol: TCP
livenessProbe:
httpGet:
path: /health
port: 8080
scheme: HTTP
initialDelaySeconds: 60
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 5
dnsPolicy: Default
volumes:
- name: config-volume
configMap:
name: coredns
items:
- key: Corefile
path: Corefile
---
apiVersion: v1
kind: Service
metadata:
name: kube-dns
namespace: kube-system
annotations:
prometheus.io/scrape: "true"
labels:
k8s-app: kube-dns
kubernetes.io/cluster-service: "true"
kubernetes.io/name: "CoreDNS"
spec:
selector:
k8s-app: kube-dns
clusterIP: CLUSTER_DNS_IP
ports:
- name: dns
port: 53
protocol: UDP
- name: dns-tcp
port: 53
protocol: TCP
9.2 編寫(xiě)部署腳本
cat > deploy.sh << EOF
#!/bin/bash
# Deploys CoreDNS to a cluster currently running Kube-DNS.
SERVICE_CIDR=${1:-10.254.0.0/16}
POD_CIDR=${2:-172.30.0.0/16}
CLUSTER_DNS_IP=${3:-10.254.0.2}
CLUSTER_DOMAIN=${4:-cluster.local}
YAML_TEMPLATE=${5:-`pwd`/coredns.yaml.sed}
sed -e s/CLUSTER_DNS_IP/$CLUSTER_DNS_IP/g -e s/CLUSTER_DOMAIN/$CLUSTER_DOMAIN/g -e s?SERVICE_CIDR?$SERVICE_CIDR?g -e s?POD_CIDR?$POD_CIDR?g $YAML_TEMPLATE > coredns.yaml
EOF
知識(shí)點(diǎn)：根據(jù)自己的node網(wǎng)絡(luò)，cluster修改自己的地址段。
9.3 部署coredns
chmod + deploy.sh
./deploy.sh
kubectl create -f coredns.yaml
9.4 驗(yàn)證dns服務(wù)
9.4.1 創(chuàng)建deployment
cat > busyboxdeploy.yaml << EOF
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: busybox-deployment
spec:
replicas: 2
template:
metadata:
labels:
app: busybox
spec:
containers:
- name: busybox
image: busybox
ports:
- containerPort: 80
args: ["/bin/sh","-c","sleep 1000"]
EOF
9.4.2 進(jìn)入pod，ping自己的SVC
kubectl exec busybox-deployment-6679c4bb96-86kfg -it -- /bin/sh
# ping kubernetes
# ...
# 雖然因?yàn)榫W(wǎng)絡(luò)的問(wèn)題ping不同，但是可以解析出名稱(chēng)。
10. 部署heapster
10.1.下載yaml文件
mkdir heapter
cd hapster
wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/grafana.yaml
wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/heapster.yaml
wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/influxdb.yaml
wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/rbac/heapster-rbac.yaml
10.2. 修改yaml的container鏡像源文件（默認(rèn)使用goolge鏡像源，我們下載不到只能改成其他人上傳至dockerhub上的）
10.2.1 修改grafana.yaml
k8s.gcr.io/heapster-grafana-amd64:v5.0.4 mirrorgooglecontainers/heapster-grafana-amd64:v5.0.4
10.2.2 修改heapster.yaml
k8s.gcr.io/heapster-amd64:v1.5.4 cnych/heapster-amd64:v1.5.4
10.2.3 修改influxdb.yaml
k8s.gcr.io/heapster-influxdb-amd64:v1.5.2 fishchen/heapster-influxdb-amd64:v1.5.2
10.3 查看heapster狀態(tài)
kubectl get svc -n kube-system
10.4 在master設(shè)置代理可以允許外部訪(fǎng)問(wèn)
kubectl proxy --port=8096 --address="10.192.44.129" --accept-hosts='^*$'
11.部署dashboard
11.1 下載dashboard的yaml文件
wget https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml -O kubernetes-dashboard.yaml
11.2 修改如下：（使用的是官方鏡像，但是更換了images，添加了nodePort）
# ------------------- Dashboard Secret ------------------- #
apiVersion: v1
kind: Secret
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-certs
namespace: kube-system
type: Opaque
---
# ------------------- Dashboard Service Account ------------------- #
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kube-system
---
# ------------------- Dashboard Role & Role Binding ------------------- #
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: kubernetes-dashboard-minimal
namespace: kube-system
rules:
# Allow Dashboard to create 'kubernetes-dashboard-key-holder' secret.
- apiGroups: [""]
resources: ["secrets"]
verbs: ["create"]
# Allow Dashboard to create 'kubernetes-dashboard-settings' config map.
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["create"]
# Allow Dashboard to get, update and delete Dashboard exclusive secrets.
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs"]
verbs: ["get", "update", "delete"]
# Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map.
- apiGroups: [""]
resources: ["configmaps"]
resourceNames: ["kubernetes-dashboard-settings"]
verbs: ["get", "update"]
# Allow Dashboard to get metrics from heapster.
- apiGroups: [""]
resources: ["services"]
resourceNames: ["heapster"]
verbs: ["proxy"]
- apiGroups: [""]
resources: ["services/proxy"]
resourceNames: ["heapster", "http:heapster:", "https:heapster:"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: kubernetes-dashboard-minimal
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: kubernetes-dashboard-minimal
subjects:
- kind: ServiceAccount
name: kubernetes-dashboard
namespace: kube-system
---
# ------------------- Dashboard Deployment ------------------- #
kind: Deployment
apiVersion: apps/v1beta2
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kube-system
spec:
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
k8s-app: kubernetes-dashboard
template:
metadata:
labels:
k8s-app: kubernetes-dashboard
spec:
containers:
- name: kubernetes-dashboard
image: siriuszg/kubernetes-dashboard-amd64:v1.10.0
ports:
- containerPort: 8443
protocol: TCP
args:
- --auto-generate-certificates
# Uncomment the following line to manually specify Kubernetes API server Host
# If not specified, Dashboard will attempt to auto discover the API server and connect
# to it. Uncomment only if the default does not work.
# - --apiserver-host=http://my-address:port
volumeMounts:
- name: kubernetes-dashboard-certs
mountPath: /certs
# Create on-disk volume to store exec logs
- mountPath: /tmp
name: tmp-volume
livenessProbe:
httpGet:
scheme: HTTPS
path: /
port: 8443
initialDelaySeconds: 30
timeoutSeconds: 30
volumes:
- name: kubernetes-dashboard-certs
secret:
secretName: kubernetes-dashboard-certs
- name: tmp-volume
emptyDir: {}
serviceAccountName: kubernetes-dashboard
# Comment the following tolerations if Dashboard must not be deployed on master
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
---
# ------------------- Dashboard Service ------------------- #
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
kubernetes.io/cluster-service: 'true'
name: kubernetes-dashboard
namespace: kube-system
spec:
type: NodePort
ports:
- port: 443
nodePort: 30000
targetPort: 8443
selector:
k8s-app: kubernetes-dashboard
11.3 部署dashboard
kubectl create -f kubernetes-dashboard.yaml
11.4 查看部署狀態(tài)
kubectl get services kubernetes-dashboard -n kube-system
kubectl get pods -n kube-system
11.5 創(chuàng)建token，以及利用token創(chuàng)建kubeconfig
k8s嚴(yán)格執(zhí)行了權(quán)限訪(fǎng)問(wèn)控制。此時(shí)賬戶(hù)必須是sa（service account）賬戶(hù),不能使用用戶(hù)名和密碼的認(rèn)證方式。以我的理解,上文kubeconfig的用戶(hù)針對(duì)的集群的訪(fǎng)問(wèn)和控制。dashboard需要是對(duì)pod層面的訪(fǎng)問(wèn)控制。
11.5.1 創(chuàng)建toekn，后續(xù)可以直接使用token訪(fǎng)問(wèn)。
kubectl create sa dashboard-admin -n kube-system
kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin
ADMIN_SECRET=$(kubectl get secrets -n kube-system | grep dashboard-admin | awk '{print $1}')
DASHBOARD_LOGIN_TOKEN=$(kubectl describe secret -n kube-system ${ADMIN_SECRET} | grep -E '^token' | awk '{print $2}')
echo ${DASHBOARD_LOGIN_TOKEN}
11.5.2 利用token創(chuàng)建kubeconfig文件訪(fǎng)問(wèn)形式
# 設(shè)置集群參數(shù)
kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=https://10.192.44.129:6443 \
--kubeconfig=dashboard.kubeconfig
# 設(shè)置客戶(hù)端認(rèn)證參數(shù)，使用上面創(chuàng)建的 Token
kubectl config set-credentials dashboard_user \
--token=${DASHBOARD_LOGIN_TOKEN} \
--kubeconfig=dashboard.kubeconfig
# 設(shè)置上下文參數(shù)
kubectl config set-context default \
--cluster=kubernetes \
--user=dashboard_user \
--kubeconfig=dashboard.kubeconfig
# 設(shè)置默認(rèn)上下文
kubectl config use-context default --kubeconfig=dashboard.kubeconfig
11.6 訪(fǎng)問(wèn)dashboard
11.6.1 kubernetes-dashboard 服務(wù)暴露了 NodePort，可以使用 https://NodeIP:NodePort 地址訪(fǎng)問(wèn) dashboard；
https://10.192.44.129:30000
問(wèn)題記錄：如果使用chrome訪(fǎng)問(wèn)的話(huà)，會(huì)提示NET:ERR_CERT_INVALID錯(cuò)誤，這是證書(shū)的額問(wèn)題。但是可以使用firefox訪(fǎng)問(wèn)。
或者導(dǎo)入ca和admin證書(shū)。如上文創(chuàng)建。但是導(dǎo)入windos要變換格式。
下面是關(guān)于生成證書(shū)的命令
生成p12格式證書(shū)
openssl pkcs12 -export -in admin.pem -out admin.p12 -inkey admin-key.pem
生成cer格式證書(shū)
openssl x509 -in admin.pem -outform der -out admin.cer
11.6.2 通過(guò) kube-apiserver 訪(fǎng)問(wèn) dashboard；
https://10.192.44.129:6443/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy
11.6.3 通過(guò) kubectl proxy 訪(fǎng)問(wèn) dashboard：
kubectl proxy --address='10.192.44.129' --port=8086 --accept-hosts='^*$'
訪(fǎng)問(wèn)：http://10.192.44.129:8096/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/ 問(wèn)題記錄：通過(guò)kubectl proxy方式訪(fǎng)問(wèn)，雖然可以成功訪(fǎng)問(wèn)到登錄界面，但是卻無(wú)法登錄，這是因?yàn)镈ashboard只允許localhost和127.0.0.1使用HTTP連接進(jìn)行訪(fǎng)問(wèn)，而其它地址只允許使用HTTPS。因此，如果需要在非本機(jī)訪(fǎng)問(wèn)Dashboard的話(huà)，只能選擇其他訪(fǎng)問(wèn)方式。
12.部署EFK
EFK是三個(gè)開(kāi)源軟件的縮寫(xiě)，分別表示：Elasticsearch ,Fluentd, Kibana ,
12.1. 安裝ELK(我是直接粘貼復(fù)制文件的)
git clone https://github.com/kubernetes/kubernetes.git
cd /opt/k8s/kubernetes/cluster/addons/fluentd-elasticsearch
12.2. 替換容器鏡像（默認(rèn)是谷歌鏡像，你懂得）
12.3.1 替換es-statefulset.yaml中鏡像
xxlaila/elasticsearch:v6.3.0
12.3.2 替換fluentd-es-ds.yaml中鏡像
vavikast/fluentd-elasticsearch:v2.2.0
12.3.3 替換kibana-deployment.yaml中鏡像
mintel/kibana-oss:6.3.2
12.3.4 意外情況（最新版本EFK中的fluentd-es-configmap.yaml文件配置有點(diǎn)問(wèn)題，我還沒(méi)有詳細(xì)研究為什么，下面是我根據(jù)https://github.com/kubernetes/minikube/blob/master/deploy/addons/efk/更改如下）
kind: ConfigMap
apiVersion: v1
metadata:
name: fluentd-es-config-v0.1.6
namespace: kube-system
labels:
addonmanager.kubernetes.io/mode: Reconcile
data:
system.conf: |-
<system>
root_dir /tmp/fluentd-buffers/
</system>
containers.input.conf: |-
<source>
@id fluentd-containers.log
@type tail
path /var/log/containers/*.log
pos_file /var/log/es-containers.log.pos
time_format %Y-%m-%dT%H:%M:%S.%NZ
tag raw.kubernetes.*
read_from_head true
<parse>
@type multi_format
<pattern>
format json
time_key time
time_format %Y-%m-%dT%H:%M:%S.%NZ
</pattern>
<pattern>
format /^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/
time_format %Y-%m-%dT%H:%M:%S.%N%:z
</pattern>
</parse>
</source>
# Detect exceptions in the log output and forward them as one log entry.
<match raw.kubernetes.**>
@id raw.kubernetes
@type detect_exceptions
remove_tag_prefix raw
message log
stream stream
multiline_flush_interval 5
max_bytes 500000
max_lines 1000
</match>
output.conf: |-
# Enriches records with Kubernetes metadata
<filter kubernetes.**>
@type kubernetes_metadata
</filter>
<match **>
@id elasticsearch
@type elasticsearch
@log_level info
include_tag_key true
host elasticsearch-logging
port 9200
logstash_format true
<buffer>
@type file
path /var/log/fluentd-buffers/kubernetes.system.buffer
flush_mode interval
retry_type exponential_backoff
flush_thread_count 2
flush_interval 5s
retry_forever
retry_max_interval 30
chunk_limit_size 2M
queue_limit_length 8
overflow_action block
</buffer>
</match>
12.3. 給node設(shè)置標(biāo)簽（因?yàn)閒luentd-es-ds.yaml文件中設(shè)置了nodeselector，如果你不設(shè)置，則無(wú)法部署DS）
kubectl get nodes
kubectl label nodes master beta.kubernetes.io/fluentd-ds-ready=true
kubectl label nodes node2 beta.kubernetes.io/fluentd-ds-ready=true
kubectl label nodes node3 beta.kubernetes.io/fluentd-ds-ready=true
12.4 執(zhí)行定義文件
kubectl creat -f ./
12.5檢查執(zhí)行結(jié)果
kubectl get pods -n kube-system -o wide|grep -E 'elasticsearch|fluentd|kibana'
參考blog：https://jimmysong.io/kubernetes-handbook/practice/create-tls-and-secret-key.html
https://mritd.me/2018/01/07/kubernetes-tls-bootstrapping-note/
https://juejin.im/post/5b45cea9f265da0f652370ce
http://www.ruanyifeng.com/blog/2016/03/systemd-tutorial-part-two.html
https://www.cnblogs.com/RainingNight/p/deploying-k8s-dashboard-ui.html
https://github.com/opsnull/follow-me-install-kubernetes-cluster
這是一篇費(fèi)時(shí)費(fèi)力的文章，遇到很多坑，一次次跌倒又爬起來(lái)，最終才算完成，真是不動(dòng)手不知道辛苦。
我希望自己的寫(xiě)的東西可以有所記錄，同時(shí)也希望與你們有所分享。這篇文章參考了很牛人的博客，我已經(jīng)貼到最后。目前來(lái)說(shuō)還有一些遇到的問(wèn)題，以及我自己的拙見(jiàn)，我沒(méi)有貼出來(lái)，等我想好怎么更好的排版，我再來(lái)完善，加油。
免責(zé)聲明：本站發(fā)布的內(nèi)容（圖片、視頻和文字）以原創(chuàng)、轉(zhuǎn)載和分享為主，文章觀點(diǎn)不代表本網(wǎng)站立場(chǎng)，如果涉及侵權(quán)請(qǐng)聯(lián)系站長(zhǎng)郵箱：is@yisu.com進(jìn)行舉報(bào)，并提供相關(guān)證據(jù)，一經(jīng)查實(shí)，將立刻刪除涉嫌侵權(quán)內(nèi)容。
億速云公眾號(hào)
手機(jī)網(wǎng)站二維碼
