Health Check — Kubernetes

Health probes can be used by container orchestrators and load balancers to check an app’s status.

Identify any early signs of health issues

Kubernetes probes allow us to validate the state of the pods running within our cluster.

Pod and Container Status

Pods have phases and conditions. Containers have states. These status properties can and will be changed based on probe results.

Pod Phases

• Pending: Accepted by the cluster, containers are not set up yet.
• Running: At least one container is in a running, starting, or restarting state.
• Succeeded: All of the containers exited with a status code of zero; the pod will not be restarted.
• Failed: All containers have terminated and at least one container exited with a status code of non-zero.
• Unknown: The state of the pod can not be determined.

Pod Conditions

• PodScheduled: A Node has been successfully selected to schedule the pod, and scheduling is completed.
• ContainersReady: All the containers are ready.
• Initialized: Init containers are started.
• Ready: The pod is able to serve requests; hence it needs to be included in the service and load balancers.

Container States

The container has three simple states.

• Waiting: Required processes are running for a successful startup.
• Running: The container is executing.
• Terminated: Container started execution and finished by either success or failure.

Common Probe Parameters

• initialDelaySeconds: Specifies the number of seconds to wait before a probe is initiated. (default value: 0)
• periodSeconds: Defines the frequency (seconds) of performing probes. (default value: 10)
• timeoutSeconds: Defines the maximum time (seconds) a probe should take to complete. (default value: 1)
• successThreshold: Minimum consecutive successes for the probe to be considered successful after having failed. (default value: 1)
• failureThreshold: Defines the number of probe failures before Kubernetes gives up. In the case of liveness probes, this means restarting the pod. (default value: 3)

LivenessProbe → Whether the application is up and running

Create a yaml file called node-app and paste in the following configuration settings:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: node-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: node-app
  template:
    metadata:
      labels:
        app: node-app
    spec:
      containers:
      - name: nodejs-app
        image: mercybassey/memory-leak  
        ports:
        - containerPort: 3005
        livenessProbe:
          httpGet:
            path: /health-check  
            port: 3005
          initialDelaySeconds: 15
          periodSeconds: 5

The liveness probe associated with each pod sends HTTP GET requests to the “health-check” endpoint on port 3005 of the container, starting 15 seconds after the pod launches and repeating every 5 seconds thereafter.

To create this deployment

kubectl apply -f node-app.yml

Run the kubectl commands to view the deployment and the pods created by the deployment

kubectl get deployment
kubectl get pods

Create a “yaml” file named “service.yaml” with the following configuration:

apiVersion: v1
kind: Service
metadata:
  name: nodejs-service
spec:
  selector:
    app: node-app
  type: NodePort
  ports:
    - port: 3005
      targetPort: 3005
      nodePort: 30005

To create this service object

kubectl apply -f service.yml

Run the following kubectl command to view the service

kubectl get services

Create a “yaml” file named “pod” with the following code:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: siege
spec:
  replicas: 1
  selector:
    matchLabels:
      app: siege
  template:
    metadata:
      labels:
        app: siege
    spec:
      containers:
      - name: siege
        image: dockersec/siege
        command: ["sieg", "-c", "40", "-t", "1M", "http://10.99.132.238:3005/"]

Now use the following kubectl command to create and view this deployment

kubectl apply -f pod.yml
kubectl get deployments

To observe the liveness probe in action, you can periodically monitor the status of the pods by executing the following command

kubectl get pods

CrashLoopBackOff is a Kubernetes state that indicates a restart loop is happening in a pod. It’s a common error message that occurs when a Kubernetes container fails to start up properly for some reason, then repeatedly crashes.

Readiness Probe → Whether it is ready to accept requests

We will use a “yaml” file called “cache” with the following configuration settings.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: node-cache-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: node-cache-app
  template:
    metadata:
      labels:
        app: node-cache-app
    spec:
      containers:
      - name: nodejs-app
        image: mercybassey/cache  
        ports:
        - containerPort: 3005
        readinessProbe:
          httpGet:
            path: /ready  
            port: 3005
          initialDelaySeconds: 10
          periodSeconds: 5

The readiness probe is configured to make an HTTP GET request to the /ready endpoint of the application every 5 seconds (periodSeconds: 5), but it will wait for 10 seconds (initialDelaySeconds: 10) before performing the first probe.

To create the deployment

kubectl apply -f cache.yml
kubectl get deployments

kubectl get pods

Initially, the readiness status of the pods shows 0/1, indicating that the containers in the pods are not ready yet to serve traffic. However, after the cache check operation is complete (after about 10 seconds), the readiness status changes to 1/1, signifying that the application is now ready to serve traffic.

Startup Probe → This probe determines whether a container has started or not

To deploy this application create a “yaml” file called “startup” with the following configuration settings

apiVersion: apps/v1
kind: Deployment
metadata:
  name: node-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: node-app
  template:
    metadata:
      labels:
        app: node-app
    spec:
      containers:
      - name: nodejs-app
        image: mercybassey/startup
        ports:
        - containerPort: 3005
        startupProbe:
          httpGet:
            path: /
            port: 3005
          failureThreshold: 30
          periodSeconds: 1
        livenessProbe:
          httpGet:
            path: /health
            port: 3005
          initialDelaySeconds: 45
          periodSeconds: 5

Now execute the following command to create and view this deployment

kubectl apply -f startup.yml
kubectl get deployments

You can also confirm that the pods also have a ready state of 3/3