HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream

Disaster recovery — initial setup

This page covers disaster-recovery design and initial recovery preparation at install time. Recurring failover tests and DR procedures belong in the Administration Guide.

Disaster recovery

Consul Enterprise allows Disaster Recovery of multiple servers as well as cluster recovery. Redundancy Zones, Automated Backups, and Cloud Service Provider backup methodologies all aim at increasing the resiliency of the cluster in terms of an outage. As Consul leverages the Raft consensus protocol, if a single failed server is recoverable, bringing it back online involves two options; bringing the failed server back online with the same IP address, or forcing the server to leave and bringing on a new server with a new IP address. Autopilot helps with this process by cleaning up dead servers (by default, a check is run every 200ms) such that new servers can be created via an immutable process with new IPs and Autopilot will remove the previously failed server for you.

In the event of a loss of quorum in the Consul cluster or a complete loss of the Consul cluster, the quickest path to remediation would be to add new servers and then proceed with manual recovery with peers.json(opens in new tab) and same is true for the Kubernetes Consul recovery with exception(opens in new tab).

Depending on the quorum size and number of failed servers, the recovery process will vary. In the event of complete failure it is beneficial to have a backup process(opens in new tab).

As such, utilizing backup mechanisms of AWS is recommended. At a minimum, the S3 bucket that snapshots are being stored into should be replicated to a different bucket. We recommend this replicated bucket to be in a different AWS region; while S3 is Highly Available within the region, this will also protect against a regional failure of the cluster.

Disaster recovery

To recover a Consul on Kubernetes primary datacenter from a disaster or during a long term outage you will need:

  • A recent backup of Consul's internal state store
  • A current backup of the Consul secrets

Consul Enterprise enables you to run the snapshot agent within your environment as a service (Systemd) or scheduled through other means. Once running, the snapshot agent service operates as a highly available process that integrates with the snapshot API to automatically manage taking snapshots, rotating backups, and sending backup files offsite to Amazon S3 (or another S3-compatible endpoint).

Configure Consul snapshot agent

Terraform deployment configures the Consul snapshot agent definition values to meet the organizations Recovery Point Objective (RPO) requirements. Update the snapshot_intervalvalues in the TF auto.tfvars file to apply the changes. This will help you configure the S3 bucket of choice in a given region in terms of disaster recovery scenarios and configure a snapshot interval to meet the RPO requirement for the organization’s Service Level Agreement (SLA).

{
  "snapshot_agent": {
    "token": "",
    "datacenter": "",
    "ca_file": "",
    "ca_path": "",
    "cert_file": "",
    "key_file": "",
    "tls_server_name": "",
    "log": {
    "level": "INFO",
    "enable_syslog": false,
    "syslog_facility": "LOCAL0"
  },
  "snapshot": {
    "interval": "1h",
    "retain": 30,
    "stale": false,
    "service": "consul-snapshot",
    "deregister_after": "72h",
    "lock_key": "consul-snapshot/lock",
    "max_failures": 3,
    "local_scratch_path": ""
  },
  "local_storage": {
    "path": "."
  },
  "aws_storage": {
    "access_key_id": "",
    "secret_access_key": "",
    "s3_region": "",
    "s3_bucket": "",
    "s3_key_prefix": "consul-snapshot",
    "s3_server_side_encryption": false,
    "s3_static_snapshot_name": ""
  }
}

Helm configuration - Consul snapshot agent

# [Enterprise Only] Values for setting up and running snapshot agents
# (https://consul.io/commands/snapshot/agent)
# within the Consul clusters. They are required to be co-located with Consul clients,
# so will inherit the clients' nodeSelector, tolerations and affinity.
server:
  snapshotAgent:
    caCert: null
    configSecret:
    secretKey: null
    secretName: null
    enabled: true
    interval: 1h
    resources:
      limits:
      cpu: 50m
      memory: 50Mi
    requests:
      cpu: 50m
      memory: 50 Mi

Shared services

Certificates

Consul EKS deployment can enable TLS for Consul servers, clients, and allowable consul-k8s components. Enabling TLS will generate a certificate authority, and server and client certificates.

HashiCorp recommends to provide certificates that have been signed by a single certificate authority (CA) and update them as part of Kubernetes or Vault secret to enable TLS for the cluster.

The Helm configuration below enables TLS on the Consul cluster. More information regarding these configuration parameters can be found on the consul-helm repository(opens in new tab).

Enable TLS - helm configuration

tls:
  enabled: true
  enableAutoEncrypt: true
  serverAdditionalDNSSANs: []
  serverAdditionalIPSANs: []
    verify: true
    httpsOnly: true
  caCert:
    secretName: null
    secretKey: null
  caKey:
    secretName: null
    secretKey: null
Secrets

Consul Kubernetes (EKS) deployment requires all the above specified required secrets. If not provided, Helm deployment will generate the secrets and update the Kubernetes secrets. If you specify Vault-as-backend, or provide your own secrets, you need to update the helm values for the Consul deployment.

Edit this page on GitHub