Upgrade procedures

This page covers recurring Consul Enterprise upgrade operations after installation.

Consul is a long-running agent on any nodes participating in a Consul cluster. These nodes consistently communicate with each other. As such, protocol level compatibility and ease of upgrades are important to remember when using Consul. After successfully installing and running Consul Enterprise, it's important to understand the backup and restore procedures to ensure data durability and system availability during unforeseen events.

Upgrades

Upgrading Consul Enterprise is a critical operational task that requires careful planning and execution to ensure service availability. All software upgrades and major server configuration changes should be tested in a non-production environment before being promoted to your production environment. This section elaborates on the key considerations when upgrading Consul Enterprise server instances.

Autopilot automated upgrades

The architecture deployed when following the Installation Guide leverages Autopilot automated upgrades to streamline the upgrade process. This feature automatically transitions to a new set of Consul server agents once ready, allowing the existing server agents to serve traffic during the transition.

When operating within an AWS environment, leverage lifecycle hooks to halt the promotion of new Auto Scaling Groups (ASGs) or the teardown of old ones until the lifecycle hook post-validation is completed. Leverage the UpgradeVersionTag to perform safe promotion without changing the Consul version. This allows you to adopt an immutable deployment style and use automated upgrade promotion when changing any server configurations to ensure any changes that could affect server availability are validated as part of the upgrade promotion process. The Terraform module recommended as part of the Installation Guide incorporates these automated upgrade features when deploying to EC2 instances.

Rolling upgrades with configuration management tools

In scenarios without AutoPilot, conduct rolling upgrades using traditional configuration management tools. It is imperative to incorporate health checks within this automation to prevent breaking the server quorum and triggering a server outage. The health checks should be robust enough to identify potential issues that could jeopardize the upgrade or the overall system stability.

Validate each server node's health before upgrading the next node in your server pool. This ensures that service remains operational during the upgrade as long as you do not exceed your deployment's failure tolerance. Even when not leveraging the automated upgrades feature, you can still query the Autopilot state to determine your current failure tolerance and health states of individual server nodes as you progress through an upgrade.

$ curl https://127.0.0.1:8501/v1/operator/autopilot/state
{
    "Healthy": true,
    "FailureTolerance": 1,
    "OptimisticFailureTolerance": 4,
    "Servers": {
        "5e26a3af-f4fc-4104-a8bb-4da9f19cb278": {},
        "10b71f14-4b08-4ae5-840c-f86d39e7d330": {},
        "1fd52e5e-2f72-47d3-8cfc-2af760a0c8c2": {},
        "63783741-abd7-48a9-895a-33d01bf7cb30": {},
        "6cf04fd0-7582-474f-b408-a830b5471285": {}
    },
    "Leader": "5e26a3af-f4fc-4104-a8bb-4da9f19cb278",
    "Voters": [
        "5e26a3af-f4fc-4104-a8bb-4da9f19cb278",
        "10b71f14-4b08-4ae5-840c-f86d39e7d330",
        "1fd52e5e-2f72-47d3-8cfc-2af760a0c8c2"
    ],
    "RedundancyZones": {
        "az1": {},
        "az2": {},
        "az3": {}
    },
    "ReadReplicas": [
        "63783741-abd7-48a9-895a-33d01bf7cb30",
        "6cf04fd0-7582-474f-b408-a830b5471285"
    ],
    "Upgrade": {}
}

Regular upgrade practice

Familiarize yourself with the upgrade procedures and make it a practice to carry out upgrades regularly. This proactive approach helps capitalize on new features and improvements while avoiding the risks of falling significantly behind on versions. While it's important to avoid bleeding-edge versions until they've proven stable in non-production environments, ensure you don't lag to the extent that the version gap introduces substantial risk during upgrades. In production, promote new releases only after they have achieved stability in non-production deployments.

Downgrade considerations

Downgrades are not supported unless you revert the server state to the pre-upgrade version. This implies that a thorough backup and validation process is important before any upgrade activity to ensure you can roll back to the previous version with the active data before the upgrade.

Upgrade clients

When deploying client agents to the nodes running your services, it is important to keep the client version in sync with the server version. Once your server agents have been upgraded to a new version, begin rolling out updates to the client agents by leveraging configuration management or by layering the new Consul software version into your system image and following an immutable deployment process to refresh the base image of all of your nodes.

Upgrade references

For more information about upgrading Consul, refer to General Upgrade Process(opens in new tab) and Upgrading Specific Versions(opens in new tab).

For more information about autopilot, refer to Autopilot(opens in new tab) and Automate upgrades with Consul Enterprise(opens in new tab).

For information about Kubernetes-specific upgrade instructions, refer to Upgrading Consul on Kubernetes components(opens in new tab).