Vault Enterprise replication
Enterprise
Appropriate Vault Enterprise license or HCP Vault Dedicated cluster required.
Vault Enterprise replication copies data from one Vault cluster, the primary, to one or more other Vault clusters, the secondaries. Use disaster recovery (DR) replication to keep a standby cluster that can take over when the primary fails. Use performance replication to add clusters that serve client requests close to the clients.
Replication works between clusters, not between the nodes of one cluster. Nodes inside a cluster share data through the cluster's storage backend, such as integrated storage, and through high availability mode. Replication links whole clusters, typically in different data centers or regions.
If you are new to Vault replication, checkout the disaster recovery, and performance replication tutorials.
Introduction
A replication set has one primary cluster and one or more secondary clusters. The primary is the system of record. It asynchronously ships every change to its secondaries as write-ahead log (WAL) entries, and the secondaries apply them in near real time. When a secondary falls too far behind for WAL streaming, it compares Merkle trees with the primary and syncs only the keys that differ. Refer to Replication internals for the design.
Each type of replication is a separate relationship. A cluster can be a DR primary and a performance primary at the same time, and a performance secondary can have its own DR secondary.
Disaster recovery and performance replication
The two types copy the same configuration but serve different goals:
Disaster recovery replication keeps a warm standby cluster. A DR secondary mirrors everything on the primary, including tokens and leases, and does not serve client requests. When the primary fails, you promote the DR secondary, and clients continue with the tokens and leases they already have.
Performance replication adds clusters that serve client requests. A performance secondary handles reads locally and forwards writes to replicated data to the primary. Each performance secondary issues and manages its own tokens and leases.
The following table compares the two types:
| Capability | DR replication | Performance replication |
|---|---|---|
| Mirrors the configuration of the primary cluster | Yes | Yes |
| Mirrors auth methods, secrets engines, audit devices, and policies | Yes | Yes |
| Mirrors local mounts from the primary cluster | Yes | No. Local mounts on the primary are not visible to performance secondaries. |
| Mirrors tokens and leases from the primary cluster | Yes | No. Each secondary manages its own tokens and leases. After a promotion, clients must reauthenticate. |
| Serves client requests | No, until you promote it | Yes |
| Serves requests while disconnected from the primary | No | Partially. Refer to Disconnected performance secondaries. |
| Supports path filters to limit replicated data | No | Yes |
Replicated data
Vault classifies every storage entry into one of three categories:
replicated: all secondaries receive the entry.local: only DR secondaries receive the entry.ignored: no secondary receives the entry.
Local means local to one cluster. A DR secondary is a standby copy of its primary, so it receives local data too. A performance secondary is a separate cluster that serves its own clients, so it receives only replicated data.
Every secrets engine and auth method mount is either shared or local. A shared
mount, the default, replicates its data to all secondaries. A local mount, which
you create with the -local flag, keeps its data on the cluster where you
create it and on that cluster's DR secondaries. Refer to Local
mounts.
Some secrets engines keep part of a shared mount's data local. On a shared PKI mount, issuers, keys, and roles replicate, so every performance cluster issues certificates from the same issuers. Each cluster stores the certificates it issues in its own storage, so it lists only its own certificates and publishes its own certificate revocation list (CRL). The certificate works anywhere, because the client holds it. Only Vault's stored record stays on the issuing cluster. To read or revoke that record, send the request to the issuing cluster, or set up cross-cluster revocation. Refer to PKI replicated data sets and Cross-cluster CRLs.
Tokens and leases are local data. A performance secondary issues its own, and a DR secondary holds a copy of the primary's. Ignored data replicates nowhere. For example, Vault ignores the integrated storage autopilot configuration, so a DR secondary can use a different autopilot configuration than its primary.
Choose a replication type
Choose the type based on the outcome you need:
| You need to | Use |
|---|---|
| Recover from the loss of a cluster, data center, or region | DR replication |
| Fail over without forcing clients to reauthenticate | DR replication |
| Upgrade with a fallback cluster, or test a recovery plan | DR replication |
| Serve clients in other regions with lower latency | Performance replication |
Scale read-heavy or encryption-heavy workloads, such as transit | Performance replication |
| Keep some secrets out of a region, for example for data residency | Performance replication with path filters |
| Protect a performance cluster against failure | DR replication on that cluster |
Most production deployments use both. Each performance cluster has a DR secondary, so the loss of any one cluster does not lose its data or its tokens. Performance replication alone is not a disaster recovery solution, because performance secondaries do not receive tokens, leases, or local mounts.
Shared design considerations
The following considerations apply to both types. The page for each type covers the considerations specific to it.
Network
Replication uses two ports on every cluster, and traffic must flow in both directions between the primary and secondary clusters:
- The API port, 8200 by default. The secondary calls the primary's API to unwrap the activation token, unless you activate with a public key.
- The cluster port, 8201 by default. All replication traffic uses mutual TLS (mTLS) on this port.
Refer to the ports reference for every port Vault uses.
Load balancers
The API port and the cluster port have different load balancer requirements.
When a load balancer fronts Vault, set api_addr and cluster_addr explicitly
on every node. Refer to the high availability
parameters.
API port
The API port, 8200 by default, accepts a network-level (layer 4) load balancer
or an application-level (layer 7) load balancer that terminates TLS. The TLS
certificate you set in the listener configuration with tls_cert_file applies
only to this port.
A secondary calls the primary's API port to unwrap its activation token. If a load balancer fronts that port:
- Set
primary_api_addron the secondary when the API address in the token differs from the load balancer address. - Make sure the secondary trusts the certificate that the load balancer
presents, or pass its CA with the
ca_fileorca_pathparameter.
Cluster port
Vault generates its own certificates for cluster traffic, so a load balancer
cannot terminate TLS on the cluster port, 8201 by default. Use a network-level
(layer 4) load balancer that passes TCP through without terminating TLS, and
route it to the active node of the primary cluster. Set primary_cluster_addr
when you enable the primary so that the primary gives secondaries the load
balancer address.
For bootstrap errors that load balancers cause, refer to Troubleshoot bootstrap and token errors.
Seals and key holders
Each cluster keeps its own seal type, so a Shamir primary can replicate to an auto-unseal secondary, and the reverse. Enabling replication does not change how the secondary unseals. It changes which key shares the secondary accepts.
Vault replaces the secondary's unseal or recovery keys, and their share count and threshold, with the primary's. After replication starts, the primary's key holders use their existing shares on the secondary, for example to generate a DR operation token or to unseal a Shamir secondary. Refer to seal wrap for how seal wrapping works across clusters.
The following table shows which keys the secondary uses after you enable replication:
| Primary seal | Secondary seal before | Secondary seal after | Secondary recovery keys after | Effect on the secondary |
|---|---|---|---|---|
| Shamir | Shamir | Primary's Shamir configuration and unseal keys | Not applicable | Seal configuration and unseal keys replaced with the primary's |
| Shamir | Auto | Unchanged | Primary's unseal keys | Recovery configuration and keys replaced with the primary's seal configuration and keys |
| Auto | Auto | Unchanged | Primary's recovery keys | Recovery configuration and keys replaced with the primary's |
| Auto | Shamir | Primary's recovery configuration and keys | Not applicable | Seal configuration and keys replaced with the primary's recovery configuration and keys |
Clusters that use auto-unseal have recovery keys instead of unseal keys. After you enable replication, the holders of the primary's key shares are the people who can unseal the secondary and generate a DR operation token.
Storage and configuration
Every cluster in a replication set needs a storage backend that supports transactional updates, such as integrated storage or Consul.
The clusters do not need the same backend, but matching backends and seal types reduces work during an outage. Replication does not copy the server configuration file, so keep the configuration of each primary and secondary securely stored in-case you need to restore the file.
Vault versions
A secondary can run a later Vault version than its primary, which lets you upgrade secondaries first. Replicating from a later version to an older version is not supported. Do not run replicated clusters on different versions longer than necessary. Refer to Upgrade replicated deployments.
Architecture guidance
For reference architectures, sizing, and operating procedures, refer to the HashiCorp Validated Design for Vault Enterprise and the Well-Architected Framework guidance on resilience.
Security model
When you enable a cluster as a primary, it generates a self-signed certificate authority (CA). For each secondary, the primary uses the CA to issue a private key and certificate and packages them, with the primary's addresses, into a secondary activation token. The primary and secondary use the certificate for mutual TLS on the cluster port.
The primary never shares its CA certificate, and no secondary has access to another secondary's certificate. A secondary's certificate works only with the primary that issued it, so after you revoke a secondary, its certificate gives it no access to the primary or to any other cluster. An attacker cannot make a secondary connect to a cluster the attacker controls without an operator giving the secondary a new activation token, which is an ACL-protected operation.
Vault uses Application-Layer Protocol Negotiation (ALPN) on the cluster port, so the same port handles request forwarding and replication with separate roots of trust.
Secondary activation tokens
A secondary activation token grants access to all Vault data. Treat it like a root token:
- Generate the secondary token only when you are ready to use it.
- The secondary token has a time to live (TTL) of 30 minutes by default.
- Keep more than one person watching secondary token from generation until use.
- The secondary token is single use token. After it activates a secondary, Vault rejects it.
After a secondary activates, Vault stores its cluster information behind the encrypted barrier.
Activation methods
Vault offers two ways to protect the connection details in a secondary activation token.
Activate with a wrapping token
By default, the primary returns the token as a response-wrapped JSON Web Token (JWT). When you enable the secondary or update its primary, the secondary calls the primary's API to unwrap the token. This call needs the primary's API port to be reachable from the secondary, and the secondary must trust the primary's TLS certificate.
Activate with a public key
When the secondary cannot reach the primary's API port, the primary can encrypt the connection details with a public key from the secondary instead. The secondary then makes no API call to the primary during activation, and only the cluster port must be open.
- On the secondary, generate a public key with the DR or performance generate public key endpoint.
- On the primary, generate the secondary activation token with the public key
in the
secondary_public_keyparameter. - On the secondary, enable replication or update the primary with the token.
The setup pages show both methods for each replication type.
Workflows
Most replication work follows one of these workflows.
Set up replication
The setup workflow is the same for both types:
- Enable replication on the primary cluster.
- Generate a secondary activation token on the primary.
- Enable replication on the secondary cluster with the token. This step deletes all existing data on the secondary.
- Check the replication status on both clusters.
Refer to Enable DR replication and Enable performance replication.
Operate replication
Ongoing operation covers these tasks:
- Monitor the replication status and replication metrics on every cluster.
- Test failover on a schedule.
- Fail over when a primary fails, then fail back or replace the failed primary.
- Troubleshoot secondaries that disconnect or fall behind.
Guidance
Use the following pages to set up, operate, and troubleshoot replication.
Tutorials
- To set up DR replication, prepare for failover, and practice planned, unplanned, and emergency failovers, complete the Set up disaster recovery replication tutorials.
- To set up performance replication and path filters, complete the Enable performance replication tutorial.