在k8s中安裝jenkins并實(shí)現(xiàn)動(dòng)態(tài)生成 jenkins slave

安裝jenkins
1、創(chuàng)建一個(gè)命名空間
$ kubectl create namespace kube-ops
2、為jenkins創(chuàng)建pvc（也可以使用存儲(chǔ)類創(chuàng)建）

apiVersion: v1
kind: PersistentVolume
metadata:
  name: opspv
spec:
  capacity:
    storage: 2Gi
  accessModes:
  - ReadWriteMany
  persistentVolumeReclaimPolicy: Delete
  nfs:
    server: 192.168.1.244
    path: /data/k8s
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: opspvc
  namespace: kube-ops
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 2Gi

3、創(chuàng)建jenkins需要的rbac權(quán)限

apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins2
  namespace: kube-ops
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: jenkins2
rules:
  - apiGroups: ["extensions", "apps"]
    resources: ["deployments"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get","list","watch"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: jenkins2
  namespace: kube-ops
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: jenkins2
subjects:
  - kind: ServiceAccount
    name: jenkins2
    namespace: kube-ops

也可以為ServiceAccoun綁定一個(gè)系統(tǒng)現(xiàn)有的 cluster-admin 集群角色權(quán)限
4、創(chuàng)建jenkins pod
$ docker pull docker.io/jenkins/jenkins:lts
$ docker pull cnych/jenkins:jnlp6

---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: jenkins2
  namespace: kube-ops
spec:
  template:
    metadata:
      labels:
        app: jenkins2
    spec:
      terminationGracePeriodSeconds: 10
      serviceAccount: jenkins2
      containers:
      - name: jenkins
        image: docker.io/jenkins/jenkins:lts
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
          name: web
          protocol: TCP
        - containerPort: 50000
          name: agent
          protocol: TCP
        resources:
          limits:
            cpu: 1000m
            memory: 1Gi
          requests:
            cpu: 500m
            memory: 512Mi
        livenessProbe:
          httpGet:
            path: /login
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 5
          failureThreshold: 12
        readinessProbe:
          httpGet:
            path: /login
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 5
          failureThreshold: 12
        volumeMounts:
        - name: jenkinshome
          subPath: jenkins2
          mountPath: /var/jenkins_home
        env:
        - name: LIMITS_MEMORY
          valueFrom:
            resourceFieldRef:
              resource: limits.memory
              divisor: 1Mi
        - name: JAVA_OPTS
          value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvision
er.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 -Duser.timezone=Asia/Shanghai      
      securityContext:
        fsGroup: 1000
      volumes:
      - name: jenkinshome
        persistentVolumeClaim:
          claimName: opspvc
---
apiVersion: v1
kind: Service
metadata:
  name: jenkins2
  namespace: kube-ops
  labels:
    app: jenkins2
spec:
  selector:
    app: jenkins2
  type: NodePort
  ports:
  - name: web
    port: 8080
    targetPort: web
    nodePort: 30003
  - name: agent
    port: 50000
    targetPort: agent

$ kubectl apply -f jenkins2.yaml
$ kubectl get pod -n kube-ops
jenkins2-76644dbc9b-llcsp 0/1 Running 0 #不能正常啟動(dòng)
$ kubectl describe pod jenkins2-76644dbc9b-llcsp -n kube-ops
$ kubectl logs -f jenkins2-76644dbc9b-llcsp -n kube-ops
5、在nfs服務(wù)器上修改jenkins持久目錄的權(quán)限并重新創(chuàng)建jenkins pod
$ chown -R 1000 /data/k8s/jenkins2 #在192.168.1.244上
$ kubectl delete -f jenkins2.yaml
$ kubectl apply -f jenkins2.yaml
$ kubectl get pod -n kube-ops
jenkins2-76644dbc9b-llcsp 1/1 Running 0
為什么是1000？
上述鏡像的Dockerfile文件中定義的是：user=jenkins group=jenkins uid=1000 gid=1000
Dockerfile文件的地址：
https://github.com/jenkinsci/docker/blob/master/Dockerfile
ARG user=jenkins
ARG group=jenkins
ARG uid=1000
ARG gid=1000
ARG http_port=8080
ARG agent_port=50000
ARG JENKINS_HOME=/var/jenkins_home
$ kubectl get svc -n kube-ops
jenkins2 NodePort 10.105.121.176 <none> 8080:30003/TCP,50000:30936/TCP
http://192.168.1.243:30003
初始密碼在nfs服務(wù)器上
$ cat /data/k8s/jenkins2/secrets/initialAdminPassword

在jenkins上創(chuàng)建kubernetes云
enkins Master 和 Jenkins Slave 以 Pod 形式運(yùn)行在 Kubernetes 集群的 Node 上，Master 運(yùn)行在其中一個(gè)節(jié)點(diǎn)，并且將其配置數(shù)據(jù)存儲(chǔ)到一個(gè) Volume 上去，Slave 運(yùn)行在各個(gè)節(jié)點(diǎn)上，并且它不是一直處于運(yùn)行狀態(tài)，它會(huì)按照需求動(dòng)態(tài)的創(chuàng)建并自動(dòng)刪除
這種方式的工作流程大致為：當(dāng) Jenkins Master 接受到 Build 請(qǐng)求時(shí)，會(huì)根據(jù)配置的 Label 動(dòng)態(tài)創(chuàng)建一個(gè)運(yùn)行在 Pod 中的 Jenkins Slave 并注冊(cè)到 Master 上，當(dāng)運(yùn)行完 Job 后，這個(gè) Slave 會(huì)被注銷并且這個(gè) Pod 也會(huì)自動(dòng)刪除，恢復(fù)到最初狀態(tài)。
1、安裝插件
安裝kubernetes plugin， 點(diǎn)擊 Manage Jenkins -> Manage Plugins -> Available -> Kubernetes plugin
2、增加kubernetes云
點(diǎn)擊 Manage Jenkins —> Configure System —> (拖到最下方)Add a new cloud —> 選擇 Kubernetes，然后填寫 Kubernetes 和 Jenkins 配置信息----連接測(cè)試
name：kubernetes
Kubernetes 地址：https://kubernetes.default.svc.cluster.local
Kubernetes 命名空間：kube-ops
Jenkins 地址：http://jenkins2.kube-ops.svc.cluster.local:8080（jenkins2是svc）
3、添加pod模板
添加pod模板----Kubernetes Pod Template
名稱：jnlp
命名空間：kube-ops
標(biāo)簽列表：dongyali-jnlp
4、添加容器模板
添加容器----Container Template
名稱：jnlp
Docker 鏡像：cnych/jenkins:jnlp6（Jenkins 版本在 2.176.x以下的鏡像名字去掉6）
工作目錄：/home/jenkins/agent
運(yùn)行的命令：清空
命令參數(shù)：清空
5、添加兩個(gè)卷
添加卷----Host Path Volume
主機(jī)路徑：/var/run/docker.sock
掛載路徑：/var/run/docker.sock
主機(jī)路徑：/root/.kube
掛載路徑：/root/.kube
6、可能需要配置ServiceAccount
$ kubectl get sa -n kube-ops
jenkins2 1 14h
點(diǎn)擊添加卷下面的高級(jí)----Service Account----jenkins2
7、用shell測(cè)試 Kubernetes 動(dòng)態(tài)生成 jenkins slave
新建任務(wù)----名字----自由風(fēng)格
通用----勾選限制項(xiàng)目的運(yùn)行節(jié)點(diǎn)----標(biāo)簽表達(dá)式：dongyali-jnlp
構(gòu)建----執(zhí)行shell----輸入如下內(nèi)容----保存立即構(gòu)建

echo "測(cè)試 Kubernetes 動(dòng)態(tài)生成 jenkins slave"
echo "==============docker in docker==========="
docker info
echo "=============kubectl============="
kubectl get pods

觀察 Kubernetes 集群中 Pod 的變化：
$ kubectl get pod -n kube-ops
jenkins2-76644dbc9b-llcsp 1/1 Running 0 3h59m
jnlp-tl1km 1/1 Running 0 44s
當(dāng)任務(wù)運(yùn)行完畢，jnlp這個(gè)slave pod就會(huì)自動(dòng)消失。
8、用pipeline測(cè)試 Kubernetes 動(dòng)態(tài)生成 jenkins slave
新建任務(wù)----名字----流水線
在流水線腳本中輸入如下內(nèi)容----保存立即構(gòu)建

node('dongyali-jnlp') {
    stage('Clone') {
      echo "1.Clone Stage"
    }
    stage('Test') {
      echo "2.Test Stage"
    }
    stage('Build') {
      echo "3.Build Stage"
    }
    stage('Deploy') {
      echo "4. Deploy Stage"
    }
}