Failover between regions with HCP Consul Dedicated

This topic describes the disaster recovery setup for HCP Consul Dedicated clusters, including how to configure multiple HCP Consul Dedicated clusters to handle a total region failure by automatically routing service traffic to instances deployed in other regions.

For more information about service failover strategies Consul supports, refer to Failover overview in the Consul documentation.

Introduction

HCP Consul Dedicated clusters are designed to recover from almost all disasters automatically. However, we recommend that you set up a few resources of your own to minimize network disruption during a total region failure. Because HCP Consul Dedicated clusters are deployed to a user-specified region, you must use multiple clusters to architect against a region failure.

To implement a minimal failover strategy, deploy two HCP Consul Dedicated clusters in separate regions, with separate instances of the same service deployed in each region. You should deploy Consul clusters in the same region as your services to satisfy both latency requirements and limit the blast radius of large-scale disasters. Then when one region fails, you can failover to services deployed in the other region.

Create a new HCP Consul Dedicated cluster in an alternative region

1. Sign in to the HCP Portal.
2. Select the organization or project where you want to create the new cluster.
3. Click Consul.
4. From the Consul Overview, click Create a Consul cluster.
5. Use the workflow to create a new HCP Consul Dedicated cluster. Give your cluster a name, select a size, and configure accessibility.

It usually takes between 5 and 10 minutes to create the new cluster.

Configure and deploy services to alternative region

After the new HCP Consul Dedicated cluster is created, configure Consul and deploy all necessary services to the second region. You can deploy all of your services or just a critical subset, depending on your recovery time or recovery point objectives.

You must also register these services to the Consul datacenter in order to route traffic from another region to them.

Setup a global failover policy

During a total region outage, you are not able to communicate with the services and the Consul cluster in that region. Therefore, you must set up a global failover policy that can reroute network traffic to your alternative region and the services running there. This failover policy should be triggered by your own disaster recovery procedures.

Setup cluster peering between clusters

While not required to recover from a regional outage, to provide additional resiliency against service outages, we recommend that you peer the two Consul clusters and setup sameness groups.