Upgrading Consul on Kubernetes components

This topic describes considerations and strategies for upgrading Consul deployments running on Kubernetes clusters. In addition to upgrading the version of Consul, you may need to update your Helm chart or the release version of the Helm chart.

Version-specific upgrade requirements

As of Consul v1.14.0, Kubernetes deployments use Consul Dataplane instead of client agents. If you upgrade Consul from a version that uses client agents to a version that uses dataplanes, you must follow specific steps to update your Helm chart and remove client agents from the existing deployment. Refer to Upgrading to Consul Dataplane for more information.

The v1.0.0 release of the Consul on Kubernetes Helm chart also introduced a change to the externalServers[].hosts parameter. Previously, you were able to enter a provider lookup as a string in this field. Now, you must include exec= at the start of a string containing a provider lookup. Otherwise, the string is treated as a DNS name. Refer to the go-netaddrs library and command line tool for more information.

Upgrade types

We recommend updating Consul on Kubernetes when:

You change your Helm configuration
A new Helm chart is released
You want to upgrade your Consul version

The upgrade procedure you use depends on the type of upgrade you are performing.

Helm configuration changes

If you make a change to your Helm values file, you need to perform a helm upgrade for those changes to take effect.

Determine your current installed chart version.

$ helm list --filter consul --namespace consul
NAME    NAMESPACE   REVISION    UPDATED                                 STATUS      CHART           APP VERSION
consul  consul      2           2022-02-02 21:49:45.647678 -0800 PST    deployed    consul-0.40.0   1.11.2

In this example, version 0.40.0 (from consul-k8s:0.40.0) is being used, and Consul on Kubernetes is installed in the consul namespace.

Perform a helm upgrade and make sure that you specify the current chart version:

$ helm upgrade consul hashicorp/consul --namespace consul --version 0.40.0 --values /path/to/my/values.yaml

Note: If you don't pass the --version flag when upgrading a Helm chart, Helm uses the most up-to-date version of the chart in its local cache, which may result in an unintended version upgrade.

Upgrade Helm chart version

You may wish to upgrade your Helm chart version to take advantage of new features and bug fixes, or because you want to upgrade your Consul version and it requires a certain Helm chart version.

Update your local Helm repository cache:
```
$ helm repo update
```

List all available versions. The console lists version 0.40.0 in the following example.

$ helm search repo hashicorp/consul --versions
NAME                CHART VERSION   APP VERSION DESCRIPTION
hashicorp/consul    0.40.0          1.11.2      Official HashiCorp Consul Chart
hashicorp/consul    0.39.0          1.11.1      Official HashiCorp Consul Chart
hashicorp/consul    0.38.0          1.10.4      Official HashiCorp Consul Chart
hashicorp/consul    0.37.0          1.10.4      Official HashiCorp Consul Chart
...

To determine which version you have installed, issue the following command:

$ helm list --filter consul --namespace consul
NAME    NAMESPACE   REVISION    UPDATED                                 STATUS      CHART           APP VERSION
consul  consul      2           2022-02-02 21:49:45.647678 -0800 PST    deployed    consul-0.39.0   1.11.1

In this example, version 0.39.0 (from consul-k8s:0.39.0) is being used.

Check the changelog for any breaking changes from that version and any versions in between: CHANGELOG.md.
Check the Consul on Kubernetes Version Compatibility matrix. Each 1.x version of the chart corresponds to a specific 1.x version of Consul. You may need to upgrade your Consul version to match the chart version you want to upgrade to. For example, chart version 1.3.1 must be used with Consul version 1.17.x. To set the Consul version, set global.image in your values.yaml file, for example:
```
global:
  image: "hashicorp/consul:1.17.5"
```
You can leave the global.image value unset to use the latest supported version of Consul. version automatically.

Upgrade by performing a helm upgrade with the --version flag set to the version you want to upgrade to:

$ helm upgrade consul hashicorp/consul --namespace consul --version 0.40.0 --values /path/to/my/values.yaml

Upgrade Consul version

If a new version of Consul is released, you need to perform a Helm upgrade to update to the new version. Before you upgrade to a new version:

Read the Upgrading Consul documentation.
Read any specific instructions for the version you want to upgrade to, as well as the Consul changelog for that version.
Read our Compatibility Matrix to ensure your current Helm chart version supports this Consul version. If it does not, you may need to also upgrade your Helm chart version at the same time.
Set global.image in your values.yaml to the desired version:
values.yaml
```
global:
  image: "hashicorp/consul:1.11.1"
```

Determine the version of your existing Helm installation. The following example shows that version 0.39.0 is installed. The version is derived from the CHART column.

$ helm list --filter consul --namespace consul
NAME    NAMESPACE   REVISION    UPDATED                                 STATUS      CHART           APP VERSION
consul  consul      2           2022-02-02 21:49:45.647678 -0800 PST    deployed    consul-0.39.0   1.11.1

Check the Consul on Kubernetes Version Compatibility matrix. Each 1.x version of the chart corresponds to a specific 1.x version of Consul. You may need to upgrade your chart version to match the Consul version you want to upgrade to.

Perform a helm upgrade:

$ helm upgrade consul hashicorp/consul --namespace consul --version 0.39.0 --values /path/to/my/values.yaml

Note: If you don't pass the --version flag when upgrading a Helm chart, Helm uses the most up-to-date version of the chart in its local cache, which may result in an unintended version upgrade.

Determine scope of changes

Before upgrading, it is important to understand the changes that affect your cluster. For example, you need to take more care if your upgrade results in the Consul server StatefulSet being redeployed.

There is no built-in functionality in Helm that shows what a helm upgrade changes. There is, however, a Helm plugin helm-diff that can be used.

Install helm-diff with:

$ helm plugin install https://github.com/databus23/helm-diff

If you are updating your values.yaml file, do so now.
Take the same helm upgrade command you were planning to issue but perform helm diff upgrade instead of helm upgrade:
```
$ helm diff upgrade consul hashicorp/consul --namespace consul --version 0.40.0 --values /path/to/your/values.yaml
```
This command prints out the manifests that will be updated and their diffs.

To see only updated objects, add | grep "has changed":

$ helm diff upgrade consul hashicorp/consul --namespace consul --version 0.40.0 --values /path/to/your/values.yaml |
  grep "has changed"

Take specific note if consul-server, StatefulSet is listed, as it means your Consul server statefulset will be redeployed.
If your Consul server statefulset needs to be redeployed, follow the same pattern for upgrades as on other platforms by redeploying servers one by one. Refer to Upgrading Consul for more information.
If neither the server statefulset is not being redeployed, then you can continue with the Helm upgrade without any specific sequence to follow.

Upgrade Consul servers

Initiate the server upgrade:

Change the global.image value to the desired Consul version.
Set the server.updatePartition value to the number of server replicas. By default, there are 3 servers, so you would set this value to 3.
Set the updateStrategy for clients to OnDelete.
values.yaml
```
global:
  image: 'consul:123.456'
server:
  updatePartition: 3
```
The updatePartition value controls how many instances of the server cluster are updated. Only instances with an index greater than the updatePartition value are updated (zero-indexed). Therefore, by setting it equal to replicas, updates should not occur immediately.
Next, perform the upgrade:
```
$ helm upgrade consul hashicorp/consul --namespace consul --version <your-version> --values /path/to/your/values.yaml
```
This command does not cause the servers to redeploy, although the resource is updated. If everything is stable, decrease the updatePartition value by one and performing helm upgrade again. This will cause the first Consul server to be stopped and restarted with the new image.
Wait until the Consul server cluster is healthy again (30s to a few minutes). This can be confirmed by issuing consul members on one of the previous servers, and ensuring that all servers are listed and are alive.
Decrease updatePartition by one and upgrade again. Continue until updatePartition is 0. At this point, you may remove the updatePartition configuration. Your server upgrade is complete.

Convert multi-port workloads during an upgrade

By default, the Consul Helm chart disables multi-port service registration with the connectInject.multiportServiceRegistration.enabled value set to false. When registration is disabled, the service mesh injector rejects any Pod that would register more than one named port in a single Consul service, unless the consul.hashicorp.com/connect-service-port annotation selects exactly one port. Refer to Configure multi-port services on Kubernetes for more information.

Existing workloads that expose several container ports and do not set that annotation keep running after the upgrade. When Kubernetes recreates their Pods, the injector rejects the new Pods. For StatefulSets, this leaves ordinals stranded, and for DaemonSets, new nodes receive no Pod.

To avoid this outcome, Consul can run a conversion Job during helm upgrade that updates existing workloads before Kubernetes recreates their Pods. The conversion Job and the conversionStrategy Helm value are available in Consul on Kubernetes Helm chart version 2.0.4 and later.

Conversion strategies

Set connectInject.multiportServiceRegistration.conversionStrategy to one of the following values:

Strategy	Behavior
`NONE`	Default. Consul does not create the conversion Job and leaves existing workloads untouched.
`TRANSLATE`	For each eligible workload that selects more than one port, Consul sets `consul.hashicorp.com/connect-service-port` in the Pod template to a single port. Consul keeps the workload in the service mesh.
`DECOMMISSION`	For each eligible workload, Consul sets `consul.hashicorp.com/connect-inject: "false"` in the Pod template. This setting removes the workload from the service mesh. Consul applies this strategy to every eligible mesh-injected workload, not only workloads that select multiple ports.

Both strategies modify the Pod template, so Kubernetes starts a rolling update for each workload that changes. The Job does not update workloads that already comply.

The TRANSLATE strategy selects the port using the following rules:

If the Pod template sets consul.hashicorp.com/connect-service-default-port, Consul selects that port. The value must match the name or number of one of the ports the workload would register. If it does not, the Job fails for that workload.
Otherwise, Consul selects the first port that the workload would register.

Consul does not convert workloads that use the legacy consul.hashicorp.com/connect-service annotation with several comma-separated service names. These workloads register one Consul service for each name and are not affected by the multi-port registration setting.

Job behavior

Helm creates the Job as a post-upgrade hook in the release namespace. The Job runs the consul-k8s-control-plane convert-multiport-services command under the service account of the service mesh injector.

The chart creates the Job when all of the following conditions are true:

You run helm upgrade. Helm does not create the Job during helm install.
connectInject.enabled is true.
connectInject.multiportServiceRegistration.enabled is false.
connectInject.multiportServiceRegistration.conversionStrategy is TRANSLATE or DECOMMISSION.

The Job converts Deployments, StatefulSets, and DaemonSets. It evaluates a workload only when the workload is eligible for injection, which means one of the following is true:

The Pod template sets consul.hashicorp.com/connect-inject: "true".
The Pod template does not set the annotation, and connectInject.default is true.

The Job honors the same namespace scoping as the injector. A namespace is in scope when it passes the connectInject.k8sAllowNamespaces, connectInject.k8sDenyNamespaces, and connectInject.namespaceSelector filters. The Job always skips the kube-system and kube-public namespaces and the namespace that contains the Consul release. Skipping the release namespace prevents DECOMMISSION from restarting Consul components.

The Job also scans in-scope Pods for workloads that it cannot rewrite, such as bare Pods and Pods owned by Argo Rollouts, OpenShift DeploymentConfigs, or operator-managed custom resources. If any of these Pods select more than one port, the Job fails and lists them. Set consul.hashicorp.com/connect-service-port to a single port on each listed workload, or remove the workload from the service mesh, and then run the upgrade again.

The Job continues after a failure to convert one workload so that the log reports the state of every workload. The Job then exits with an error. Because the Job is a Helm hook, a failed Job fails the helm upgrade command. The Job retries up to three times.

Helm deletes the Job after it succeeds and before it creates the Job for the next upgrade. A failed Job remains in the cluster so that you can inspect it.

The chart validates the strategy when it renders. The helm upgrade command fails before Helm applies any change when conversionStrategy is not NONE, TRANSLATE, or DECOMMISSION. It also fails when you set conversionStrategy to TRANSLATE or DECOMMISSION while multiportServiceRegistration.enabled is true.

Run the conversion

Set the conversion strategy in your Helm values file.

values.yaml

connectInject:
  multiportServiceRegistration:
    enabled: false
    conversionStrategy: TRANSLATE

Upgrade the Helm release.

$ helm upgrade consul hashicorp/consul --namespace consul --version <your-version> --values /path/to/your/values.yaml

Helm waits for the conversion Job to complete before it marks the upgrade as successful.

Verify the result. For a TRANSLATE conversion, check the Pod template annotations of a converted workload. The following example shows a Deployment named web.

$ kubectl get deployment web --output jsonpath='{.spec.template.metadata.annotations}'
{"consul.hashicorp.com/connect-inject":"true","consul.hashicorp.com/connect-service-port":"api-port"}

If the upgrade fails, review the Job logs. The log line multi-port conversion complete reports the strategy and the number of workloads that Consul updated. Error lines list each workload that Consul could not convert.
```
$ kubectl logs job/consul-connect-inject-multiport-conversion --namespace consul
```

After the conversion completes and Kubernetes finishes the rolling updates, you can set conversionStrategy back to NONE. The Job is idempotent, so running it again does not change workloads that already comply.

If your workloads need multi-port registration and you run Consul Enterprise, set connectInject.multiportServiceRegistration.enabled to true instead of converting the workloads. In this configuration, you must leave conversionStrategy set to NONE.

Upgrading to Consul Dataplane

In earlier versions, Consul on Kubernetes used client agents in its deployments. As of v1.14.0, Consul uses Consul Dataplane in Kubernetes deployments instead of client agents.

If you upgrade Consul from a version that uses client agents to a version the uses dataplanes, complete the following steps to upgrade your deployment safely and without downtime.

If ACLs are enabled, you must first upgrade to consul-k8s 0.49.8 or above. These versions expose the setting connectInject.prepareDataplanesUpgrade which is required for no-downtime upgrades when ACLs are enabled.
Set connectInject.prepareDataplanesUpgrade to true and then perform the upgrade to 0.49.8 or above (whichever is the latest in the 0.49.x series)
```
connectInject:
  prepareDataplanesUpgrade: true
```
Consul dataplanes disables Consul clients by default, but during an upgrade you need to ensure Consul clients continue to run. Edit your Helm chart configuration and set the client.enabled field to true and specify an action for Consul to take during the upgrade process in the client.updateStrategy field:
```
client:
  enabled: true
  updateStrategy: |
    type: OnDelete
```
Follow our recommended procedures to upgrade servers on Kubernetes deployments to upgrade Helm values for the new version of Consul. The latest version of consul-k8s components may be in a CrashLoopBackoff state during the performance of the server upgrade from versions <1.14.x until all Consul servers are on versions >=1.14.x. Components in CrashLoopBackoff will not negatively affect the cluster because older versioned components will still be operating. Once all servers have been fully upgraded, the latest consul-k8s components will automatically restore from CrashLoopBackoff and older component versions will be spun down.
Run kubectl rollout restart to restart your service mesh applications. Restarting service mesh application causes Kubernetes to re-inject them with the webhook for dataplanes.
Restart all gateways in your service mesh.
Now that all services and gateways are using Consul dataplanes, disable client agents in your Helm chart by deleting the client stanza or setting client.enabled to false and running a consul-k8s or Helm upgrade.
If ACLs are enabled, outdated ACL tokens will persist a result of the upgrade. You can manually delete the tokens to declutter your Consul environment.
Outdated connect-injector tokens have the following description: token created via login: {"component":"connect-injector"}. Do not delete the tokens that have a description where pod is a key, for example token created via login: {"component":"connect-injector","pod":"default/consul-connect-injector-576b65747c-9547x"}). The dataplane-enabled connect inject pods use these tokens.
You can also review the creation date for the tokens and only delete the injector tokens created before your upgrade, but do not delete all old tokens without considering if they are still in use. Some tokens, such as the server tokens, are still necessary.

Configuring TLS on an existing cluster

If you already have a Consul cluster deployed on Kubernetes and would like to turn on TLS for internal Consul communication, refer to Configuring TLS on an Existing Cluster.