Chapter 14

Deployment Rollouts

In this session we will see how to do deployment rollouts and rollbacks

Subsections of Deployment Rollouts

Deployment & Replicaset

The following are typical use cases for Deployments:

  • Create a Deployment to rollout a ReplicaSet. The ReplicaSet creates Pods in the background. Check the status of the rollout to see if it succeeds or not.
  • Declare the new state of the Pods by updating the PodTemplateSpec of the Deployment. A new ReplicaSet is created and the Deployment manages moving the Pods from the old ReplicaSet to the new one at a controlled rate. - Each new ReplicaSet updates the revision of the Deployment.
  • Rollback to an earlier Deployment revision if the current state of the Deployment is not stable. Each rollback updates the revision of the Deployment.
  • Scale up the Deployment to facilitate more load.
  • Pause the Deployment to apply multiple fixes to its PodTemplateSpec and then resume it to start a new rollout.
  • Use the status of the Deployment as an indicator that a rollout has stuck.
  • Clean up older ReplicaSets that you don’t need anymore.

Deployment Deployment

Rollout Demo

Creating a Deployment

$ vi nginx-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80
$ kubectl create -f nginx-deployment.yaml

Output

deployment.apps/nginx-deployment created
$ kubectl get all

Output

NAME                                    READY   STATUS    RESTARTS   AGE
pod/nginx-deployment-76bf4969df-c7jrz   1/1     Running   0          38s --+
pod/nginx-deployment-76bf4969df-gl5cv   1/1     Running   0          38s   |-----> These Pods are spawned by ReplicaSet nginx-deployment-76bf4969df
pod/nginx-deployment-76bf4969df-kgmx9   1/1     Running   0          38s --+

NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
service/kubernetes   ClusterIP   172.168.0.1      <none>        443/TCP        3d23h

NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-deployment   3/3     3            3           38s ---------> Deployment

NAME                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-deployment-76bf4969df   3         3         3       38s ---> ReplicaSet

As we already know , Deployment controller uses labels to select the Pods In Yaml spec , you can see below selector fields.

selector:
  matchLabels:
    app: nginx

Lets examine the Pods. We can see that there are two labels. The pod-template-hash label is added by the Deployment controller to every ReplicaSet that a Deployment creates or adopts.

$ kubectl get pods --show-labels

Output

NAME                                READY   STATUS    RESTARTS   AGE     LABELS
nginx-deployment-76bf4969df-hm5hf   1/1     Running   0          3m44s   app=nginx,pod-template-hash=76bf4969df
nginx-deployment-76bf4969df-tqn2k   1/1     Running   0          3m44s   app=nginx,pod-template-hash=76bf4969df
nginx-deployment-76bf4969df-wjmqp   1/1     Running   0          3m44s   app=nginx,pod-template-hash=76bf4969df

You may see the parameters of a replicaset using below command.

$ kubectl get replicasets nginx-deployment-76bf4969df -o yaml --export

We can update the nginx image to a new version using set image.

Here we are executing both set image and rollout status together so that we can monitor the status.

$ kubectl set image deployment nginx-deployment nginx=nginx:1.9.1 ; kubectl rollout status deployment nginx-deployment

Output

deployment.extensions/nginx-deployment image updated
Waiting for deployment "nginx-deployment" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "nginx-deployment" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "nginx-deployment" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "nginx-deployment" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "nginx-deployment" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "nginx-deployment" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "nginx-deployment" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "nginx-deployment" rollout to finish: 1 old replicas are pending termination...
deployment "nginx-deployment" successfully rolled out

To view the history of rollout

$ kubectl rollout history deployment nginx-deployment

Output

deployment.extensions/nginx-deployment
REVISION  CHANGE-CAUSE
1         <none>
2         <none>

To see the changes we made on each version, we can use below commands

$ kubectl rollout history deployment nginx-deployment --revision=1

First revision have image nginx:1.7.9

deployment.extensions/nginx-deployment with revision #1
Pod Template:
  Labels:       app=nginx
        pod-template-hash=76bf4969df
  Containers:
   nginx:
    Image:      nginx:1.7.9
    Port:       80/TCP
    Host Port:  0/TCP
    Environment:        <none>
    Mounts:     <none>
  Volumes:      <none>

First revision have image nginx:1.9.1

$ kubectl rollout history deployment nginx-deployment --revision=2
deployment.extensions/nginx-deployment with revision #2
Pod Template:
  Labels:       app=nginx
        pod-template-hash=779fcd779f
  Containers:
   nginx:
    Image:      nginx:1.9.1
    Port:       80/TCP
    Host Port:  0/TCP
    Environment:        <none>
    Mounts:     <none>
  Volumes:      <none>

Now lets rollback the update.

$ kubectl rollout undo deployment nginx-deployment ;kubectl rollout status deployment nginx-deployment

Output

deployment.extensions/nginx-deployment rolled back
Waiting for deployment "nginx-deployment" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "nginx-deployment" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "nginx-deployment" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "nginx-deployment" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "nginx-deployment" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "nginx-deployment" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "nginx-deployment" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "nginx-deployment" rollout to finish: 1 old replicas are pending termination...
deployment "nginx-deployment" successfully rolled out

Now we have revision 2 and 3.

When we rollback version 1 become 3 . In this way the latest active one and the previous revision will be the highest and second highest revisions respectively. This logic will allow quick rollbacks.

$ kubectl rollout history deployment nginx-deployment
deployment.extensions/nginx-deployment
REVISION  CHANGE-CAUSE
2         <none>
3         <none>

Check each revision changes

$ kubectl rollout history deployment nginx-deployment --revision=2
deployment.extensions/nginx-deployment with revision #2
Pod Template:
  Labels:       app=nginx
        pod-template-hash=779fcd779f
  Containers:
   nginx:
    Image:      nginx:1.9.1
    Port:       80/TCP
    Host Port:  0/TCP
    Environment:        <none>
    Mounts:     <none>
  Volumes:      <none>
$ kubectl rollout history deployment nginx-deployment --revision=3
deployment.extensions/nginx-deployment with revision #3
Pod Template:
  Labels:       app=nginx
        pod-template-hash=76bf4969df
  Containers:
   nginx:
    Image:      nginx:1.7.9
    Port:       80/TCP
    Host Port:  0/TCP
    Environment:        <none>
    Mounts:     <none>
  Volumes:      <none>

Lets check the status of replicaset

$ kubectl get rs
NAME                          DESIRED   CURRENT   READY   AGE
nginx-deployment-76bf4969df   3         3         3       145m
nginx-deployment-779fcd779f   0         0         0       69m

Here we can see two replicaset

$ kubectl describe rs nginx-deployment-76bf4969df
Name:           nginx-deployment-76bf4969df
Namespace:      default
Selector:       app=nginx,pod-template-hash=76bf4969df
Labels:         app=nginx
                pod-template-hash=76bf4969df
Annotations:    deployment.kubernetes.io/desired-replicas: 3
                deployment.kubernetes.io/max-replicas: 4
                deployment.kubernetes.io/revision: 3
                deployment.kubernetes.io/revision-history: 1
Controlled By:  Deployment/nginx-deployment
Replicas:       3 current / 3 desired  <<<<<<<<<<<<<<-------------------------
Pods Status:    3 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:  app=nginx
           pod-template-hash=76bf4969df
  Containers:
   nginx:
    Image:        nginx:1.7.9 <<<<<<<<<<<<<<-------------------------
    Port:         80/TCP
    Host Port:    0/TCP
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Events:           <none>
$ kubectl describe rs nginx-deployment-779fcd779f
Name:           nginx-deployment-779fcd779f
Namespace:      default
Selector:       app=nginx,pod-template-hash=779fcd779f
Labels:         app=nginx
                pod-template-hash=779fcd779f
Annotations:    deployment.kubernetes.io/desired-replicas: 3
                deployment.kubernetes.io/max-replicas: 4
                deployment.kubernetes.io/revision: 2
Controlled By:  Deployment/nginx-deployment
Replicas:       0 current / 0 desired <<<<<<<<<<<<<<-------------------------
Pods Status:    0 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:  app=nginx
           pod-template-hash=779fcd779f
  Containers:
   nginx:
    Image:        nginx:1.9.1 <<<<<<<<<<<<<<-------------------------
    Port:         80/TCP
    Host Port:    0/TCP
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Events:           <none>

We can pause and resume a rollout using below commands

$ kubectl rollout pause deployment nginx-deployment
$ kubectl rollout resume deployment nginx-deployment

We can use revisionHistoryLimit field in a Deployment to specify how many old ReplicaSets for this Deployment you want to retain. The rest will be garbage-collected in the background. By default, it is 10

We can read more about strategy here