Enable ACLs in WAN-federated datacenters

This page describes how to set up Consul's access control list (ACL) system in WAN-federated Consul datacenters. You can configure tokens, policies, and roles to work across multiple datacenters. ACL replication has several benefits:

It enables authentication of nodes and services between multiple datacenters.
The secondary datacenter can provide failover for all ACL components created in the primary datacenter.
Sharing policies reduces redundancy for the operator.

Workflow overview

The overall process to setup ACL replication for multiple datacenters that are connected through WAN federation consists of the following steps:

Set the primary_datacenter parameter on all Consul agents in the primary datacenter.
Create the replication token.
Configure the primary_datacenter and enable_token_replication parameters on all Consul servers in the secondary datacenter.
Apply the replication token to all the servers in the secondary datacenter.
Configure the primary_datacenter and enable_token_replication parameters on all Consul clients in the secondary datacenter.

Requirements

Consul v1.4.0 or later

Configure ACLs in the primary datacenter

In a federated Consul deployment, one of the datacenters is designated the primary datacenter. You should add the acl configuration block to the primary datacenter server's configuration file as shown in the following example.

ACL Configuration in Primary

datacenter = "dc1",
primary_datacenter= "dc1",
acl {
  enabled = true,
  default_policy = "deny",
  down_policy = "extend-cache",
  enable_token_persistence = true
}

{
  "datacenter": "dc1",
  "primary_datacenter": "dc1",
  "acl": {
    "enabled": true,
    "default_policy": "deny",
    "down_policy": "extend-cache",
    "enable_token_persistence": true
  }
}

Refer to the ACL stanza in the Agent reference docs for more detailed descriptions of each attribute.

Enterprise deployments

Most enterprise deployments have security requirements that prevent specifying tokens in configuration files. The configuration example also sets the enable_token_persistence attribute so that the token is stored to disk in the agent's data directory. Any changes to the token through the Consul HTTP API persists to the same location, and the value in the configuration file is ignored.

The primary_datacenter parameter sets the primary datacenter to have authority for all ACL information. You should also set this parameter on the Consul clients, so that the they can forward API requests to the servers.

Start the agent. If your Consul servers are already up, you need to restart the Consul agents for ACL related configuration changes to take effect.

$ consul agent -config-file=server.hcl

Complete this process on all server agents. If you are configuring ACLs for the first time, you must bootstrap the ACL system for the cluster first.

Create the replication token

You must configure servers in secondary datacenters to point to the primary datacenter. You need to provide the ACL replication token to the secondary datacenters. Replication tokens are required to replicate ACL tokens and to create configuration entries and auth methods in connected secondary datacenters.

Replication tokens require the following permissions:

Permission	Description
`acl = "write"`	The permission allows you to replicate tokens.
`operator = "write"`	This permission enables the `proxy-default` configuration entries to be replicated and enables CA certificate signing in the secondary datacenter.
`policy = "read"` and `intentions = "read"` in the `service_prefix` field	These permissions enable `service-default` configuration entries, CA, and intention data to be replicated for all services.

Replication Token Policy

acl = "write"

operator = "write"

service_prefix "" {
  policy = "read"
  intentions = "read"
}

operator = "write"

service_prefix "" {
  policy = "read"
  intentions = "read"
}

namespace_prefix "" {
  acl = "write"

  service_prefix "" {
    policy = "read"
    intentions = "read"
  }
}

Now that you have defined the ACL rules, create a replication policy.

$ consul acl policy create -name replication -rules @replication-policy.hcl
ID:           240f1d01-6517-78d3-ec32-1d237f92ab58
Name:         replication
Description: Datacenters:
Rules: acl = "write"

operator = "write"

service_prefix "" { policy = "read" intentions = "read" }

Create the replication token with your newly created replication policy.

$ consul acl token create -description "replication token" -policy-name replication
AccessorID:   67d55dc1-b667-1835-42ab-64658d64a2ff
SecretID:     fc48e84d-3f4d-3646-4b6a-2bff7c4aaffb
Description:  replication token
Local:        false
Create Time:  2023-05-09 18:34:23.288392523 +0000 UTC
Policies:
  240f1d01-6517-78d3-ec32-1d237f92ab58 - replication

Configure Consul servers in the secondary datacenter

Set the primary_datacenter parameter to the name of your primary datacenter and enable_token_replication to true on all the servers.

Configure ACLs for Consul servers in dc2

datacenter = "dc2",
primary_datacenter= "dc1",
acl {
  enabled = true,
  default_policy = "deny",
  down_policy = "extend-cache",
  enable_token_persistence = true,
  enable_token_replication = true
}

{
  "datacenter": "dc2",
  "primary_datacenter": "dc1",
  "acl": {
    "enabled": true,
    "default_policy": "deny",
    "down_policy": "extend-cache",
    "enable_token_persistence": true,
    "enable_token_replication": true
  }
}

Start the servers on the secondary datacenter.

consul agent -config-file=server.hcl

Repeat this process on all the servers in the secondary datacenter.

Warning

When enabling ACL token replication in secondary datacenters, global tokens present in the secondary datacenter are lost. For production environments, we recommend configuring ACL replication when you initially bootstrap your datacenter.

If you are using Consul Enterprise and the admin partitions, only ACL tokens in the default partition are replicated to other datacenters.

Apply the replication token to Consul servers

Apply the replication token to all Consul servers in the secondary datacenter with the CLI. Replace <token> with the replication token's secret ID value you created earlier.

$ consul acl set-agent-token replication <token>
ACL token "replication" set successfully

Repeat this process on all servers the secondary datacenter. If you are configuring ACLs for the first time, you must also set the agent token.

Configure clients in secondary datacenters

Set the primary_datacenter parameter to the name of your primary datacenter and enable_token_replication to true on all the servers.

Configure ACLs for Consul clients in dc2

datacenter = "dc2",
primary_datacenter= "dc1",
acl {
  enabled = true,
  default_policy = "deny",
  down_policy = "extend-cache",
  enable_token_persistence = true,
  enable_token_replication = true
}

{
  "datacenter": "dc2",
  "primary_datacenter": "dc1",
  "acl": {
    "enabled": true,
    "default_policy": "deny",
    "down_policy": "extend-cache",
    "enable_token_persistence": true,
    "enable_token_replication": true
  }
}

Start the agent. If your Consul servers are already up, you need to restart the Consul agents for ACL related configuration changes to take effect.

$ consul agent -config-file=client.hcl

Repeat this process on all clients. If you are configuring ACLs for the first time, you must also set the agent token on all clients.

Check replication

Now that you have set up ACL replication, use the HTTP API to check the configuration in the secondary datacenter.

$ curl http://localhost:8500/v1/acl/replication?pretty
{
  "Enabled": true,
  "Running": true,
  "SourceDatacenter": "primary_dc",
  "ReplicationType": "tokens",
  "ReplicatedIndex": 19,
  "ReplicatedTokenIndex": 22,
  "LastSuccess": "2019-05-09T18:54:09Z",
  "LastError": "0001-01-01T00:00:00Z"
}

Notice ReplicationType is set to tokens. This means tokens, policies, and roles are replicated from the primary datacenter to the secondary one. You can call the /v1/acl/replication endpoint in the primary datacenter, but it does not return replication configuration.