kubernetes系列教程（八）Pod健康检查机制-深蓝源码网

时间: 2020-09-03 00:08:26 人气: 2272 评论: 0

写在前面

上一篇文章中kubernetes系列教程（七）深入玩转pod调度介绍了kubernetes中Pod的调度机制，通过实战演练介绍Pod调度到node的几种方法：1. 通过nodeName固定选择调度，2. 通过nodeSelector定向选择调度，3. 通过node Affinity亲和力调度，接下来介绍kubernetes系列教程pod的健康检查机制。

1. 健康检查

1.1 健康检查概述

应用在运行过程中难免会出现错误，如程序异常，软件异常，硬件故障，网络故障等，kubernetes提供Health Check健康检查机制，当发现应用异常时会自动重启容器，将应用从service服务中剔除，保障应用的高可用性。k8s定义了三种探针Probe：

readiness probes 准备就绪检查，通过readiness是否准备接受流量，准备完毕加入到endpoint，否则剔除
liveness probes 在线检查机制，检查应用是否可用，如死锁，无法响应，异常时会自动重启容器
startup probes 启动检查机制，应用一些启动缓慢的业务，避免业务长时间启动而被前面的探针kill掉

每种探测机制支持三种健康检查方法，分别是命令行exec，httpGet和tcpSocket，其中exec通用性最强，适用与大部分场景，tcpSocket适用于TCP业务，httpGet适用于web业务。

exec 提供命令或shell的检测，在容器中执行命令检查，返回码为0健康，非0异常
httpGet http协议探测，在容器中发送http请求，根据http返回码判断业务健康情况
tcpSocket tcp协议探测，向容器发送tcp建立连接，能建立则说明正常

每种探测方法能支持几个相同的检查参数，用于设置控制检查时间：

initialDelaySeconds 初始第一次探测间隔，用于应用启动的时间，防止应用还没启动而健康检查失败
periodSeconds 检查间隔，多久执行probe检查，默认为10s
timeoutSeconds 检查超时时长，探测应用timeout后为失败
successThreshold 成功探测阈值，表示探测多少次为健康正常，默认探测1次

1.2 exec命令行健康检查

许多应用程序运行过程中无法检测到内部故障，如死锁，出现故障时通过重启业务可以恢复，kubernetes提供liveness在线健康检查机制，我们以exec为例，创建一个容器启动过程中创建一个文件/tmp/liveness-probe.log，10s后将其删除，定义liveness健康检查机制在容器中执行命令ls -l /tmp/liveness-probe.log，通过文件的返回码判断健康状态，如果返回码非0，暂停20s后kubelet会自动将该容器重启。

1. 定义一个容器，启动时创建一个文件，健康检查时ls -l /tmp/liveness-probe.log返回码为0，健康检查正常，10s后将其删除，返回码为非0，健康检查异常

[root@node-1 demo]# cat centos-exec-liveness-probe.yaml
apiVersion: v1
kind: Pod
metadata:
  name: exec-liveness-probe
  annotations:
    kubernetes.io/description: "exec-liveness-probe"
spec:
  containers:
  - name: exec-liveness-probe
    image: centos:latest
    imagePullPolicy: IfNotPresent
    args:    #容器启动命令，生命周期为30s
    - /bin/sh
    - -c
    - touch /tmp/liveness-probe.log && sleep 10 && rm -f /tmp/liveness-probe.log && sleep 20
    livenessProbe:
      exec:  #健康检查机制，通过ls -l /tmp/liveness-probe.log返回码判断容器的健康状态
        command:
        - ls 
        - l 
        - /tmp/liveness-probe.log
      initialDelaySeconds: 1
      periodSeconds: 5
      timeoutSeconds: 1

2. 应用配置生成容器

[root@node-1 demo]# kubectl apply -f centos-exec-liveness-probe.yaml 
pod/exec-liveness-probe created

3. 查看容器的event日志，容器启动后，10s以内容器状态正常，11s开始执行liveness健康检查，检查异常，触发容器重启

[root@node-1 demo]# kubectl describe pods exec-liveness-probe | tail
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  28s                default-scheduler  Successfully assigned default/exec-liveness-probe to node-3
  Normal   Pulled     27s                kubelet, node-3    Container image "centos:latest" already present on machine
  Normal   Created    27s                kubelet, node-3    Created container exec-liveness-probe
  Normal   Started    27s                kubelet, node-3    Started container exec-liveness-probe
  #容器已启动
  Warning  Unhealthy  20s (x2 over 25s)  kubelet, node-3    Liveness probe failed: /tmp/liveness-probe.log
ls: cannot access l: No such file or directory #执行健康检查，检查异常
  Warning  Unhealthy  15s  kubelet, node-3  Liveness probe failed: ls: cannot access l: No such file or directory
ls: cannot access /tmp/liveness-probe.log: No such file or directory
  Normal  Killing  15s  kubelet, node-3  Container exec-liveness-probe failed liveness probe, will be restarted
  #重启容器

4. 查看容器重启次数，容器不停的执行，重启次数会响应增加，可以看到RESTARTS的次数在持续增加

[root@node-1 demo]# kubectl get pods exec-liveness-probe 
NAME                  READY   STATUS    RESTARTS   AGE
exec-liveness-probe   1/1     Running   6          5m19s

1.3 httpGet健康检查

1. httpGet probe主要主要用于web场景，通过向容器发送http请求，根据返回码判断容器的健康状态，返回码小于4xx即表示健康，如下定义一个nginx应用，通过探测http://<container>:port/index.html的方式判断健康状态

[root@node-1 demo]# cat nginx-httpGet-liveness-readiness.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: nginx-httpget-livess-readiness-probe 
  annotations:
    kubernetes.io/description: "nginx-httpGet-livess-readiness-probe"
spec:
  containers:
  - name: nginx-httpget-livess-readiness-probe
    image: nginx:latest
    ports:
    - name: http-80-port
      protocol: TCP
      containerPort: 80
    livenessProbe:   #健康检查机制，通过httpGet实现实现检查
      httpGet:
        port: 80
        scheme: HTTP
        path: /index.html
      initialDelaySeconds: 3
      periodSeconds: 10
      timeoutSeconds: 3

2. 生成pod并查看健康状态

[root@node-1 demo]# kubectl apply -f nginx-httpGet-liveness-readiness.yaml 
pod/nginx-httpget-livess-readiness-probe created
[root@node-1 demo]# kubectl get pods nginx-httpget-livess-readiness-probe 
NAME                                   READY   STATUS    RESTARTS   AGE
nginx-httpget-livess-readiness-probe   1/1     Running   0          6s

3. 模拟故障，将pod中的path文件所属文件删除，此时发送http请求时会健康检查异常，会触发容器自动重启

查询pod所属的节点
[root@node-1 demo]# kubectl get pods nginx-httpget-livess-readiness-probe -o wide 
NAME                                   READY   STATUS    RESTARTS   AGE    IP            NODE     NOMINATED NODE   READINESS GATES
nginx-httpget-livess-readiness-probe   1/1     Running   1          3m9s   10.244.2.19   node-3   <none>           <none>

登录到pod中将文件删除
[root@node-1 demo]# kubectl exec -it nginx-httpget-livess-readiness-probe /bin/bash
root@nginx-httpget-livess-readiness-probe:/# ls -l /usr/share/nginx/html/index.html 
-rw-r--r-- 1 root root 612 Sep 24 14:49 /usr/share/nginx/html/index.html
root@nginx-httpget-livess-readiness-probe:/# rm -f /usr/share/nginx/html/index.html

4. 再次查看pod的列表，此时会RESTART的次数会增加1，表示重启重启过一次，AGE则多久前重启的时间

[root@node-1 demo]# kubectl get pods nginx-httpget-livess-readiness-probe 
NAME                                   READY   STATUS    RESTARTS   AGE
nginx-httpget-livess-readiness-probe   1/1     Running   1          4m22s

5. 查看pod的详情，观察容器重启的情况，通过Liveness 检查容器出现404错误，触发重启。

[root@node-1 demo]# kubectl describe pods nginx-httpget-livess-readiness-probe | tail
Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  5m45s                  default-scheduler  Successfully assigned default/nginx-httpget-livess-readiness-probe to node-3
  Normal   Pulling    3m29s (x2 over 5m45s)  kubelet, node-3    Pulling image "nginx:latest"
  Warning  Unhealthy  3m29s (x3 over 3m49s)  kubelet, node-3    Liveness probe failed: HTTP probe failed with statuscode: 404
  Normal   Killing    3m29s                  kubelet, node-3    Container nginx-httpget-livess-readiness-probe failed liveness probe, will be restarted
  Normal   Pulled     3m25s (x2 over 5m41s)  kubelet, node-3    Successfully pulled image "nginx:latest"
  Normal   Created    3m25s (x2 over 5m40s)  kubelet, node-3    Created container nginx-httpget-livess-readiness-probe
  Normal   Started    3m25s (x2 over 5m40s)  kubelet, node-3    Started container nginx-httpget-livess-readiness-probe

1.4 tcpSocket健康检查

1. tcpsocket健康检查适用于TCP业务，通过向指定容器建立一个tcp连接，可以建立连接则健康检查正常，否则健康检查异常，依旧以nignx为例使用tcp健康检查机制，探测80端口的连通性

[root@node-1 demo]# cat nginx-tcp-liveness.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: nginx-tcp-liveness-probe
  annotations:
    kubernetes.io/description: "nginx-tcp-liveness-probe"
spec:
  containers:
  - name: nginx-tcp-liveness-probe 
    image: nginx:latest
    ports:
    - name: http-80-port
      protocol: TCP
      containerPort: 80
    livenessProbe:  #健康检查为tcpSocket，探测TCP 80端口
      tcpSocket:
       port: 80
      initialDelaySeconds: 3
      periodSeconds: 10
      timeoutSeconds: 3

2. 应用配置创建容器

[root@node-1 demo]# kubectl apply -f nginx-tcp-liveness.yaml 
pod/nginx-tcp-liveness-probe created

[root@node-1 demo]# kubectl get pods nginx-tcp-liveness-probe 
NAME                       READY   STATUS    RESTARTS   AGE
nginx-tcp-liveness-probe   1/1     Running   0          6s

3. 模拟故障，获取pod所属节点，登录到pod中，安装查看进程工具htop

获取pod所在node [root@node-1 demo]# kubectl get pods nginx-tcp-liveness-probe -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-tcp-liveness-probe 1/1 Running 0 99s 10.244.2.20 node-3 <none> <none> 登录到pod中 [root@node-1 demo]# kubectl exec -it nginx-httpget-livess-readiness-probe /bin/bash #执行apt-get update更新和apt-get install htop安装工具 root@nginx-httpget-livess-readiness-probe:/# apt-get update Get:1 http://cdn-fastly.deb.debian.org/debian buster InRelease [122 kB] Get:2 http://security-cdn.debian.org/debian-security buster/updates InRelease [39.1 kB] Get:3 http://cdn-fastly.deb.debian.org/debian buster-updates InRelease [49.3 kB] Get:4 http://security-cdn.debian.org/debian-security buster/updates/main amd64 Packages [95.7 kB] Get:5 http://cdn-fastly.deb.debian.org/debian buster/main amd64 Packages [7899 kB] Get:

技术沙龙教程文章热点综合

分享链接