本文共 13038 字，大约阅读时间需要 43 分钟。

【k8s实战一】Jenkins 部署应用到 Kubernetes

01 本文主旨

目标是演示整个Jenkins从源码构建镜像到部署镜像到Kubernetes集群过程。

为了简化流程与容易重现文中效果，做出如下操作：

• 使用 VBox 创建两台虚拟机
• 使用 Minikube 初始化简单的k8s集群
• 使用 GitHub 作为代码仓库，本文内容所有配置文件均会上传至此仓库

02 CI/CD流程

根据我画的这张图，Jenkins任务整体的流程分为五个步骤：

• 获取源码
• 构建制品
• 构建镜像并推送
• 发起部署请求
• k8s调度pod，拉取镜像，部署完成

03 准备虚拟机

准备两个虚拟机，均安装好docker，参考

设置Jenkins机器免密登录k8s master主机

ssh-keygen -t rsassh-copy-id hellxz@192.168.87.129

验证免密效果

ssh hellxz@192.168.87.129exit

04 部署registry

部署registry之前，可以先将私有仓库配置到 /etc/docker/daemon.json，两台虚拟机均需要配置

sudo mkdir -p /etc/dockersudo vim /etc/docker/daemon.json

内容如下：

{    "insecure-registries": ["192.168.87.129:5000"]}

接着在 192.168.87.129 虚拟机上部署registry，生产环境推荐使用Harbor

docker run -d --name registry -p 5000:5000 registry

05 配置maven

在jenkins虚拟机安装maven

wget https://mirrors.aliyun.com/apache/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gztar zxvf apache-maven-3.6.3-bin.tar.gzmv apache-maven-3.6.3 maven-3.6.3

修改 /etc/profile，添加 M2_HOME 环境变量映射到 maven 目录，重写 PATH 添加 maven 的 bin 目录，如图31、32行

06 Tomcat部署Jenkins

在jenkins虚拟机 192.168.87.128上执行命令下载 tomcat 与 jenkins.war

#下载tomcatwget https://mirrors.tuna.tsinghua.edu.cn/apache/tomcat/tomcat-9/v9.0.41/bin/apache-tomcat-9.0.41.tar.gz#解压并重命名tar -zxf apache-tomcat-9.0.41.tar.gzmv apache-tomcat-9.0.41 /home/hellxz/jenkins#进入jenkins安装目录cd ~/jenkins/webapps#删除webapps下示例项目rm -rf *#下载最新版jenkins.warwget https://mirrors.ustc.edu.cn/jenkins/war/latest/jenkins.war#安装openjdk、gitsudo apt-updatesudo apt-get install -y default-jdk default-jdk-headless git

设置Jenkins家目录，编辑 /etc/profile，添加 JENKINS_HOME 环境变量

刷新配置

source /etc/profile

修改tomcat下 conf/context.xml，加大静态资源缓存，提前解决启动 Jenkins 时的报错

启动Jenkins

cd ~/jenkins/bin./startup.sh

访问 ，安装推荐插件，安装加速可参考

07 部署Kubernetes集群

部署一个可以通过 kubectl 命令部署应用的k8s环境即可，部署k8s环境不是本文重点，以下为参考

#添加阿里apt kubernetes源sudo apt-get update && sudo apt-get install -y apt-transport-httpscurl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | sudo apt-key add - sudo tee /etc/apt/sources.list.d/kubernetes.list <<-'EOF'deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial mainEOFsudo apt-get update#安装kubectlsudo apt-get install -y kubectl#下载minikube二进制文件curl -Lo minikube https://kubernetes.oss-cn-hangzhou.aliyuncs.com/minikube/releases/v1.16.0/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/#部署k8s环境，默认使用docker驱动。读者也可使用vboxminikube start -–insecure-registry=192.168.87.129:5000

本文使用Minikube做测试，坑比较多，想趟坑可以先参考下

关于Minikube的吐槽

写文时，用的Win10创建Linux无桌面虚拟机，在虚拟机基础上装的minikube，minikube默认使用Docker在虚拟机中又创建了个Docker容器，里边才是k8s环境，所以想在虚拟机外访问套了两层的内网应用服务，可能还需要在虚拟机中加Nginx，通过它转发请求到虚拟机中的内网地址……虽然疯狂套娃挺坑的，但为了演示下效果还是可以的。 PS：在Linux/MacOS环境直接装Minikube，是可以通过浏览器直接看的。由于未直接在Win环境使用Docker Desktop，所以未确定Win下是否可用。

08 准备Demo

Demo分五部分：测试代码、Jenkins构建脚本、docker镜像构建脚本、k8s部署配置yaml、部署脚本，均处在同一仓库中

仓库地址：

目录结构：

demo

使用start.spring.io生成的只添加spring-boot-starter-web依赖，在启动类上加了个 /test 的接口

Dockerfile

FROM openjdk:8-jdk#设置工作目录WORKDIR /root#复制制品jar到/root/app.jar位置ADD target/*.jar app.jar#JVM参数，给后期调优使用ENV JVM_OPTS=""#启动服务CMD java ${JVM_OPTS} \    -Djava.security.egd=file:/dev/./urandom \    -jar app.jar

shell

artifact2image.sh 制品转镜像Shell脚本，本文为了简单未安装Jenkins关于docker的插件

#!/bin/bash# -- 构建镜像并推送私有仓库 --set -eu #如有报错或取不到变量情况停止执行#声名常量IMG_REGISTRY="192.168.87.129:5000" #镜像仓库IMG_NAME="cicd-demo"IMG_TAG=`date "+%Y%M%d_%H%M"`      #镜像标签，如 20201223_1351IMG_FULL_NAME="${IMG_REGISTRY}/${IMG_NAME}:${IMG_TAG}" #镜像上传与拉取的名称#构建镜像docker build -t ${IMG_FULL_NAME} .#推送镜像docker push ${IMG_FULL_NAME}#删除本地镜像docker rmi ${IMG_FULL_NAME}#修改deploy.yaml的镜像标签sed -i "s#{{IMAGE_NAME}}#${IMG_FULL_NAME}#g" deploy.yaml

deploy2k8s.sh 部署镜像到k8s环境脚本

#!/bin/bash# -- 部署image到k8s --# !注意：需要提前做ssh免密登录set -eu#定义常量PROJECT_NAME="cicd-demo"UPLOAD_DIR="/home/hellxz/apps/${PROJECT_NAME}"FILE_NAME="${UPLOAD_DIR}/deploy.yaml"SSH_USER="hellxz"SSH_IP="192.168.87.129"#首先删除待上传目录的同名文件ssh ${SSH_USER}@${SSH_IP} "rm -rf ${FILE_NAME}"#确保部署文件目录存在ssh ${SSH_USER}@${SSH_IP} "mkdir -p ${UPLOAD_DIR}"#远程复制部署文件scp -r deploy.yaml ${SSH_USER}@${SSH_IP}:${FILE_NAME}#远程执行部署命令ssh ${SSH_USER}@${SSH_IP} "kubectl apply -f ${FILE_NAME}"

Jenkinsfile

pipeline{    agent any    stages {        //由于源码和Jenkinsfile处于同一仓库，在Jenkins项目执行时，会先将Jenkinsfile所在的仓库克隆下来，为了简单，这里就不重复添加拉取源码的操作了        //其它情况，如使用多个仓库一些构建的，这种就需要额外添加拉取代码的stage了。        stage('Build Artifact') {            steps{                sh label:'maven building', script: 'mvn clean package -DskipTests'            }        }        stage('Build Image'){            steps{                sh label:'image building', script: '/bin/bash artifact2image.sh'            }        }        stage('Deploy k8s'){            steps{                sh label:'deploy image to k8s', script: '/bin/bash deploy2k8s.sh'            }        }    }    post {        success{            //成功清理工作空间，失败保留现场            cleanWs()        }    }}

deploy.yaml

apiVersion: apps/v1   #api版本kind: Deployment      #对象类型metadata:             #元信息标识  name: cicd-demo  namespace: default  labels:             #自定义Deployment绑定的标签    app: cicd-demospec:                 #期待运行状态  replicas: 1         #部署实例数  selector:    matchLabels:      app: cicd-demo  template:           #pod    metadata:      labels:         #与spec.selector.matchLabels相同        app: cicd-demo    spec:             #pod容器期望的状态      containers:        - name: cicd-demo          image: {{IMAGE_NAME}}          imagePullPolicy: IfNotPresent          env:            - name: JVM_OPTS              value: "-Xms128m -Xmx256m"          ports:            - containerPort: 8080---apiVersion: v1kind: Servicemetadata:  name: cicd-demo-svc  namespace: default  labels:    service: cicd-demo-svcspec:  selector:            #匹配pod标签    app: cicd-demo  type: NodePort     #svc类型，ExternalName, ClusterIP, NodePort, and LoadBalancer  ports:    - name: cicd-demo-port      protocol: TCP      port: 8080      nodePort: 30000

09 创建Jenkins流水线项目

访问 并登录，以下创建新流水线demo构建项目

部署与验证

执行构建demo程序

日志输出：

Started by user adminObtained Jenkinsfile from git https://github.com/hellxz/cicd-demo.gitRunning in Durability level: MAX_SURVIVABILITY[Pipeline] Start of Pipeline[Pipeline] nodeRunning on Jenkins in /home/hellxz/jenkins/home/workspace/demo[Pipeline] {[Pipeline] stage[Pipeline] { (Declarative: Checkout SCM)[Pipeline] checkoutSelected Git installation does not exist. Using DefaultThe recommended git tool is: NONENo credentials specifiedCloning the remote Git repositoryCloning repository https://github.com/hellxz/cicd-demo.git > git init /home/hellxz/jenkins/home/workspace/demo # timeout=10Fetching upstream changes from https://github.com/hellxz/cicd-demo.git > git --version # timeout=10 > git --version # 'git version 2.17.1' > git fetch --tags --progress -- https://github.com/hellxz/cicd-demo.git +refs/heads/*:refs/remotes/origin/* # timeout=10 > git config remote.origin.url https://github.com/hellxz/cicd-demo.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10Avoid second fetch > git rev-parse refs/remotes/origin/master^{commit} # timeout=10Checking out Revision 1fc086590e32c001c8b65a4c575f4aa1b5b20413 (refs/remotes/origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 1fc086590e32c001c8b65a4c575f4aa1b5b20413 # timeout=10Commit message: "update" > git rev-list --no-walk 1fc086590e32c001c8b65a4c575f4aa1b5b20413 # timeout=10[Pipeline] }[Pipeline] // stage[Pipeline] withEnv[Pipeline] {[Pipeline] stage[Pipeline] { (Build Artifact)[Pipeline] sh (maven building)+ mvn clean package -DskipTestsNOTE: Picked up JDK_JAVA_OPTIONS:  --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED[INFO] Scanning for projects...[INFO] [INFO] ----------------------< online.hellxz:cicd-demo >-----------------------[INFO] Building cicd-demo 0.0.1[INFO] --------------------------------[ jar ]---------------------------------[INFO] [INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ cicd-demo ---[INFO] [INFO] --- maven-resources-plugin:3.2.0:resources (default-resources) @ cicd-demo ---[INFO] Using 'UTF-8' encoding to copy filtered resources.[INFO] Using 'UTF-8' encoding to copy filtered properties files.[INFO] Copying 1 resource[INFO] Copying 0 resource[INFO] The encoding used to copy filtered properties files have not been set. This means that the same encoding will be used to copy filtered properties files as when copying other filtered resources. This might not be what you want! Run your build with --debug to see which files might be affected. Read more at https://maven.apache.org/plugins/maven-resources-plugin/examples/filtering-properties-files.html[INFO] [INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ cicd-demo ---[INFO] Changes detected - recompiling the module![INFO] Compiling 1 source file to /home/hellxz/jenkins/home/workspace/demo/target/classes[INFO] [INFO] --- maven-resources-plugin:3.2.0:testResources (default-testResources) @ cicd-demo ---[INFO] Using 'UTF-8' encoding to copy filtered resources.[INFO] Using 'UTF-8' encoding to copy filtered properties files.[INFO] skip non existing resourceDirectory /home/hellxz/jenkins/home/workspace/demo/src/test/resources[INFO] [INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ cicd-demo ---[INFO] No sources to compile[INFO] [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ cicd-demo ---[INFO] Tests are skipped.[INFO] [INFO] --- maven-jar-plugin:3.2.0:jar (default-jar) @ cicd-demo ---[INFO] Building jar: /home/hellxz/jenkins/home/workspace/demo/target/cicd-demo-0.0.1.jar[INFO] [INFO] --- spring-boot-maven-plugin:2.4.1:repackage (repackage) @ cicd-demo ---[INFO] Replacing main artifact with repackaged archive[INFO] ------------------------------------------------------------------------[INFO] BUILD SUCCESS[INFO] ------------------------------------------------------------------------[INFO] Total time:  3.248 s[INFO] Finished at: 2020-12-23T11:51:20Z[INFO] ------------------------------------------------------------------------[Pipeline] }[Pipeline] // stage[Pipeline] stage[Pipeline] { (Build Image)[Pipeline] sh (image building)+ /bin/bash artifact2image.shSending build context to Docker daemon  17.26MBStep 1/5 : FROM openjdk:8-jdk ---> 89f100fa8f9fStep 2/5 : WORKDIR /root ---> Using cache ---> 17670bfa01f1Step 3/5 : ADD target/*.jar app.jar ---> b7e4292e4f80Step 4/5 : ENV JVM_OPTS="" ---> Running in 0be8c16c9c8aRemoving intermediate container 0be8c16c9c8a ---> 3ef66b100082Step 5/5 : CMD java -Djava.security.egd=file:/dev/./urandom     ${JVM_OPTS}     -jar app.jar ---> Running in c8209a642a2dRemoving intermediate container c8209a642a2d ---> bc7859f9b4feSuccessfully built bc7859f9b4feSuccessfully tagged 192.168.87.129:5000/cicd-demo:20205123_1151The push refers to repository [192.168.87.129:5000/cicd-demo]44cbb90cc2c3: Preparingf85e383859a1: Preparingffb4778f8a52: Preparinge528f2c31deb: Preparingc5f4367d4a59: Preparingceecb62b2fcc: Preparing193bc1d68b80: Preparingf0e10b20de19: Preparingceecb62b2fcc: Waiting193bc1d68b80: Waitingf0e10b20de19: Waitingffb4778f8a52: Layer already existse528f2c31deb: Layer already existsf85e383859a1: Layer already exists193bc1d68b80: Layer already existsc5f4367d4a59: Layer already existsceecb62b2fcc: Layer already existsf0e10b20de19: Layer already exists44cbb90cc2c3: Pushed20205123_1151: digest: sha256:24976370741b8e20a7e8c7d18e13f4b72bb492f9c77908927d3b17b8e1ce75d4 size: 2006Untagged: 192.168.87.129:5000/cicd-demo:20205123_1151Untagged: 192.168.87.129:5000/cicd-demo@sha256:24976370741b8e20a7e8c7d18e13f4b72bb492f9c77908927d3b17b8e1ce75d4Deleted: sha256:bc7859f9b4fefc5fdcb39bab4c77dff94980d70fcd21b0b3755f317f517e4291Deleted: sha256:3ef66b100082a9d6949802c53fe57442d0e156216f0ba7a544b4e44d5a80e7a4Deleted: sha256:b7e4292e4f807a13f800b5acc8adb0a84926e5ce5c4cc850a2769503558efcb9Deleted: sha256:2afe950a3c18304e26f7d6b3f3bc48d9de631eb8827203bfa273d361647d0951[Pipeline] }[Pipeline] // stage[Pipeline] stage[Pipeline] { (Deploy k8s)[Pipeline] sh (deploy image to k8s)+ /bin/bash deploy2k8s.shdeployment.apps/cicd-demo createdservice/cicd-demo-svc created[Pipeline] }[Pipeline] // stage[Pipeline] stage[Pipeline] { (Declarative: Post Actions)[Pipeline] cleanWs[WS-CLEANUP] Deleting project workspace...[WS-CLEANUP] Deferred wipeout is used...[WS-CLEANUP] done[Pipeline] }[Pipeline] // stage[Pipeline] }[Pipeline] // withEnv[Pipeline] }[Pipeline] // node[Pipeline] End of PipelineFinished: SUCCESS

可以看到以下输出，说明程序已部署到k8s集群中了

deployment.apps/cicd-demo createdservice/cicd-demo-svc created

ssh登录k8s的机器验证

总结

简单总结下，Jenkins与Kubernetes的集成，更多地是Jenkins如何控制Kubernetes部署的流程，文中例子是通过 构建并上传镜像 和 ssh发送配置和命令 实现的。

本文是参考慕课网上的，快速扫了下视频实践所得，是了解其思路后自行实现的。

实际操作上才发现，脑子说会了，然后双手却没有跟上，还是要动手啊！

操作了一天半左右，最开始还想着用vagrant初始化VM，由于工作上有个虚拟机是VBox5的，升级新版不能用。搞了半天放弃了Vagrant（VBox6.1升级了命令参数不兼容旧版，导致Vagrant为兼容它把几乎所有版本Vagrant都改了……）。

文中难免因为安装部分的描述而过于冗长，感谢大家能看到这里，最后再次感谢慕课网的

参考

转载地址：http://yqauz.baihongyu.com/