HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream
Nomad
Consul

Configure Consul ACL with Nomad Workload Identities

  • 19min
  • |
  • Nomad
  • Consul
  • Interactive

Nomad and Consul provide several integration points when deployed together. Workloads running in Nomad can register services in Consul's catalog for service discovery or service mesh, and access configuration values from Consul KV. Nomad agents can also use Consul to discover other Nomad agents and automatically form a cluster.

Production deployments of Nomad and Consul must always run with the Access Control List (ACL) system enabled since it protects against unauthorized access to the cluster. When ACLs are enabled, both Nomad and Consul must be properly configured in order for their integrations to work.

Nomad can generate workload identities for tasks and services, which are represented as JSON Web Tokens (JWT) signed by Nomad. These identities can be used as proof to third parties that a workload was actually created and is managed by Nomad. If the third party is configured to trust Nomad, it can automatically grant specific access and permissions to Nomad workloads.

In this tutorial, you will:

  • Start a Nomad and Consul agent with ACL enabled.
  • Generate ACL tokens to access Consul and Nomad.
  • Configure Consul to accept workload identities from Nomad.
  • Configure Nomad to automatically generate and sign workload identities for services and tasks that need access to Consul.
  • Deploy sample Nomad jobs that interact with Consul.

Launch Terminal

This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.

Prerequisites

This tutorial requires you to have basic familiarity with Nomad and Consul. If you are new to these tools, complete the Nomad Get Started and Consul Get Started tutorials before following this one.

You will need the following tools installed:

Start the Consul agent

Create a directory for the tutorial on your local machine, change into that directory, and create a file named consul.hcl to store the Consul agent configuration. Add the following contents to it and save the file.

consul.hcl

datacenter = "dc1"
node_name  = "host01"

acl {
  enabled        = true
  default_policy = "deny"
}

Start a Consul dev agent using consul.hcl as the configuration file.

$ consul agent -dev -config-file 'consul.hcl'
==> Starting Consul agent...
               Version: '1.17.0'
            Build Date: '2023-11-03 14:56:56 +0000 UTC'
               Node ID: 'f9625116-3884-cd0b-01fa-2bc2e0d9a69d'
...

Dev Agents

This tutorial uses development agents for Consul and Nomad as a quick way to get started. Dev agents have ephemeral state and should not be used in production environments. They also run in the foreground of your terminal so do not close the terminal window or you will need to rerun the agent configuration steps again.

Open another terminal window in the same directory and bootstrap the Consul ACL system. This terminal will act as the main terminal session where you will run commands.

$ consul acl bootstrap
AccessorID:       0164c5f0-4186-f24c-5fc9-28e2d4e6902e
SecretID:         da0315e9-8000-4c5e-c470-1c60d79314b9
Description:      Bootstrap Token (Global Management)
Local:            false
Create Time:      2023-11-15 17:57:48.724704 -0500 EST
Policies:
   00000000-0000-0000-0000-000000000001 - global-management

Copy the value for SecretID and set it as the environment variable CONSUL_HTTP_TOKEN.

$ export CONSUL_HTTP_TOKEN=...

Create a node ACL token for the Consul agent.

$ consul acl token create -node-identity 'host01:dc1'
AccessorID:       18ddc57f-72e6-2b4f-9b20-b8c8d58d41d7
SecretID:         e7fc8ebd-8234-2316-15fe-f59d4f7a5463
Description:
Local:            false
Create Time:      2023-11-15 18:06:59.674682 -0500 EST
Node Identities:
   host01 (Datacenter: dc1)

Copy the value for SecretID.

Apply the ACL token to the local Consul agent. Replace the <SecretID> placeholder with the value from the previous token creation command.

$ consul acl set-agent-token agent <SecretID>
ACL token "agent" set successfully

Create a file named consul-policy-nomad-agents.hcl to store the Consul ACL rules that grant the necessary permissions to Nomad agents. Add the following contents to it and save the file.

consul-policy-nomad-agents.hcl

agent_prefix "" {
  policy = "read"
}

node_prefix "" {
  policy = "write"
}

service_prefix "" {
  policy = "write"
}

Create a Consul ACL policy named nomad-agents with the rules defined in the consul-policy-nomad-agents.hcl file.

$ consul acl policy create -name 'nomad-agents' -description 'Policy for Nomad agents' -rules '@consul-policy-nomad-agents.hcl'
ID:           7a0fe00b-f7e6-809c-2227-bb0638b873bd
Name:         nomad-agents
Description:  Policy for Nomad agents
Datacenters:
Rules:
agent_prefix "" {
  policy = "read"
}

node_prefix "" {
  policy = "write"
}

service_prefix "" {
  policy = "write"
}

Create a Consul ACL token for the Nomad agent using the nomad-agents ACL policy.

$ consul acl token create -policy-name 'nomad-agents'
AccessorID:       3f436657-823a-95e3-4755-79f3e1e43c8e
SecretID:         df179fd2-3211-3641-5901-a57331c14611
Description:
Local:            false
Create Time:      2023-11-15 18:23:39.572365 -0500 EST
Policies:
   a5ee20ed-7158-89be-9a19-be213d106d24 - nomad-agents

Save the value of SecretID for the Consul ACL token. You will use it in the next section to configure Nomad.

Start the Nomad agent

Create a file named nomad.hcl. Add the following contents to it, replace the placeholder <Consul token SecretID> text with the value of SecretID for the Consul ACL token, and save the file.

nomad.hcl

acl {
  enabled = true
}

consul {
  address = "127.0.0.1:8500"
  token   = "<Consul token SecretID>"

  service_identity {
    aud = ["consul.io"]
    ttl = "1h"
  }

  task_identity {
    aud = ["consul.io"]
    ttl = "1h"
  }
}

Tokens in Configuration Files

We do not recommend placing tokens in configuration files for production environments and is done so in this tutorial as an illustration. When possible, place the token value in the CONSUL_HTTP_TOKEN environment variable.

The consul block in this configuration file provides the information necessary for Nomad to connect to Consul.

It also defines two default workload identities, service_identity and task_identity, that are automatically added to jobs that need access to Consul. Without them, you would need to define an identity block in your jobs for each service and task that needs access to Consul.

Open another terminal window in the same directory and start the Nomad dev agent.

$ sudo nomad agent -dev -config 'nomad.hcl'
==> Loaded configuration from nomad.hcl
==> Starting Nomad agent...
==> Nomad agent configuration:

       Advertise Addrs: HTTP: 127.0.0.1:4646; RPC: 127.0.0.1:4647; Serf: 127.0.0.1:4648
            Bind Addrs: HTTP: [127.0.0.1:4646]; RPC: 127.0.0.1:4647; Serf: 127.0.0.1:4648
                Client: true
             Log Level: DEBUG
               Node Id: af9bef00-2e83-b704-2d6c-c62bc005431f
                Region: global (DC: dc1)
                Server: true
               Version: 1.7.0

==> Nomad agent started! Log data will stream in below:
...

Return to your main terminal window and bootstrap the Nomad ACL system.

$ nomad acl bootstrap
Accessor ID  = d1de8625-8556-0932-a25c-3aa71bfc0134
Secret ID    = 7f10099a-936c-3f3a-8783-f0980493e54b
Name         = Bootstrap Token
Type         = management
Global       = true
Create Time  = 2023-11-16 01:09:26.565422 +0000 UTC
Expiry Time  = <none>
Create Index = 23
Modify Index = 23
Policies     = n/a
Roles        = n/a

Copy the value of Secret ID and set it as the environment variable NOMAD_TOKEN.

$ export NOMAD_TOKEN=...

Using Bootstrap Tokens

The initial ACL bootstrap tokens from Consul and Nomad have full access to the cluster and should not be used for regular day-to-day operations in a production environment. They are used in this tutorial as an illustration.

We recommend creating ACL policies and tokens with only a level of access necessary to perform the required operations.

Verify that the Nomad agent was able to register itself in the Consul catalog.

$ consul catalog services
consul
nomad
nomad-client

Configure Consul for services workload identities

Create a Consul ACL JWT auth method

Create a file named consul-auth-method-nomad-workloads.json. Add the following contents to it and save the file.

consul-auth-method-nomad-workloads.json

{
  "JWKSURL": "http://127.0.0.1:4646/.well-known/jwks.json",
  "JWTSupportedAlgs": ["RS256"],
  "BoundAudiences": ["consul.io"],
  "ClaimMappings": {
    "nomad_namespace": "nomad_namespace",
    "nomad_job_id": "nomad_job_id",
    "nomad_task": "nomad_task",
    "nomad_service": "nomad_service"
  }
}

This file contains important information for creating a JWT auth method in Consul.

  • JWKSURL is the URL that Consul uses to contact Nomad and retrieve the data necessary to validate Nomad workload identities. In a production environment, this should resolve to multiple Nomad agents via a reverse proxy, load balancer, or DNS entry to prevent a single point of failure.
  • BoundAudiences is a list of strings that configures Consul to only accept JWTs that have one of these audience values. The default identities in the nomad.hcl configuration file match the values in consul-auth-method-nomad-workloads.json.
  • ClaimMappings are values extracted from the Nomad workload identity. You will reference them in upcoming steps.

Create a Consul JWT auth method named nomad-workloads using the consul-auth-method-nomad-workloads.json file.

$ consul acl auth-method create -name 'nomad-workloads' -type 'jwt' -description 'JWT auth method for Nomad services and workloads' -config '@consul-auth-method-nomad-workloads.json'
Name:          nomad-workloads
Type:          jwt
Description:   JWT auth method for Nomad services and workloads
Config:
{
  "BoundAudiences": [
    "consul.io"
  ],
  "ClaimMappings": {
    "nomad_job_id": "nomad_job_id",
    "nomad_namespace": "nomad_namespace",
    "nomad_service": "nomad_service",
    "nomad_task": "nomad_task"
  },
  "JWKSURL": "http://127.0.0.1:4646/.well-known/jwks.json",
  "JWTSupportedAlgs": [
    "RS256"
  ]
}

This auth method creates an endpoint for generating Consul ACL tokens from Nomad workload identities.

Create a Consul ACL binding rule for Nomad services

Create a binding rule to associate Consul ACL tokens generated from the nomad-workloads auth method to a service identity.

$ consul acl binding-rule create -method 'nomad-workloads' -description 'Binding rule for services registered from Nomad' -bind-type 'service' -bind-name '${value.nomad_service}' -selector '"nomad_service" in value'
ID:           3a5c1a4b-58b6-e779-dfc0-4e3a434b6763
AuthMethod:   nomad-workloads
Description:  Binding rule for services registered from Nomad
BindType:     service
BindName:     ${value.nomad_service}
Selector:     "nomad_service" in value

The service identity automatically grants the token the permissions to manage the service itself and to query other services in the Consul cluster.

The -bind-name '${value.nomad_service}' flag restricts the token to only be able to modify services with the same name as defined in the Nomad job.

The value of ${value.nomad_service} is extracted from the workload identity used to register the service, as defined in the ClaimMappings configuration of the nomad-workloads auth method, and is dynamically set to the name of the service being registered by Nomad.

The -selector '"nomad_service" in value' flag ensures this rule only applies to workload identities used for service registration because they are the only ones that have the nomad_service claim.

Run a Nomad job to register a service in Consul

Create a file named httpd.nomad.hcl. Add the following contents to it and save the file.

httpd.nomad.hcl

job "httpd" {
  group "httpd" {
    network {
      port "http" {}
    }

    service {
      provider = "consul"
      name     = "httpd"
      port     = "http"
    }

    task "httpd" {
      driver = "docker"

      config {
        image   = "busybox:1.36"
        command = "httpd"
        args    = ["-f", "-p", "${NOMAD_PORT_http}"]
        ports   = ["http"]
      }
    }
  }
}

This Nomad job runs an HTTP server and registers it as service in the Consul service catalog.

The job does not specify any identity for Consul, so the service_identity from the nomad.hcl agent configuration file is used.

Run the httpd.nomad.hcl job file and wait for the deployment to complete.

$ nomad job run 'httpd.nomad.hcl'
==> 2023-11-15T20:15:11-05:00: Monitoring evaluation "3e32e483"
    2023-11-15T20:15:11-05:00: Evaluation triggered by job "httpd"
    2023-11-15T20:15:12-05:00: Evaluation within deployment: "3ab75837"
    2023-11-15T20:15:12-05:00: Allocation "020dc113" created: node "6b4dc583", group "db"
    2023-11-15T20:15:12-05:00: Evaluation status changed: "pending" -> "complete"
==> 2023-11-15T20:15:12-05:00: Evaluation "3e32e483" finished with status "complete"
==> 2023-11-15T20:15:12-05:00: Monitoring deployment "3ab75837"
  ✓ Deployment "3ab75837" successful

    2023-11-15T20:15:23-05:00
    ID          = 3ab75837
    Job ID      = httpd
    Job Version = 0
    Status      = successful
    Description = Deployment completed successfully

    Deployed
    Task Group  Desired  Placed  Healthy  Unhealthy  Progress Deadline
    db          1        1       1        0          2023-11-15T20:25:21-05:00

Verify that the httpd service is in the Consul catalog.

$ consul catalog services
consul
httpd
nomad
nomad-client

Retrieve the job service definition from Nomad.

$ nomad job inspect -t '{{sprig_toPrettyJson (index .TaskGroups 0).Services}}' 'httpd'
[
  {
    "Name": "httpd",
    "Tags": null,
    "CanaryTags": null,
...

Note that it has an Identity for Consul.

[
  {
    "Name": "httpd",
    "Tags": null,
    "CanaryTags": null,
    "EnableTagOverride": false,
    "PortLabel": "db",
    "AddressMode": "auto",
    "Address": "",
    "Checks": null,
    "CheckRestart": null,
    "Connect": null,
    "Meta": null,
    "CanaryMeta": null,
    "TaggedAddresses": null,
    "TaskName": "",
    "OnUpdate": "require_healthy",
    "Identity": {
      "Name": "consul-service_httpd-http",
      "Audience": [
        "consul.io"
      ],
      "ChangeMode": "",
      "ChangeSignal": "",
      "Env": false,
      "File": false,
      "ServiceName": "httpd",
      "TTL": 3600000000000
    },
    "Provider": "consul",
    "Cluster": "default"
  }
]

This Identity block is the workload identity that was automatically injected by Nomad based on the configuration provided in the service_identity block in the nomad.hcl file.

Nomad uses this identity to retrieve a Consul ACL token that is allowed to register the service in Consul.

Configure Consul for tasks workload identities

Create a Consul ACL binding rule for Nomad tasks

Create a binding rule to map ACL tokens from the nomad-workloads auth method to ACL roles with names that match the pattern nomad-tasks-${value.nomad_namespace}.

$ consul acl binding-rule create -method 'nomad-workloads' -description 'Binding rule for Nomad tasks' -bind-type 'role' -bind-name 'nomad-tasks-${value.nomad_namespace}' -selector '"nomad_service" not in value'
ID:           025fba50-e3f2-84c9-163e-f42e415ea478
AuthMethod:   nomad-workloads
Description:  Binding rule for Nomad tasks
BindType:     role
BindName:     nomad-tasks-${value.nomad_namespace}
Selector:     "nomad_service" not in value

The value of ${value.nomad_namespace} is dynamically set to the Nomad namespace of the workload requesting the Consul token. You can also use other values from the auth method ClaimMappings, such as nomad_job_id and nomad_task for more fine-grained mapping of roles.

The -selector '"nomad_service" not in value' flag ensures this rule only applies to workload identities that are not used for service registration because they do not have the nomad_service claim.

Create a Consul ACL policy for Nomad tasks

Create a file named consul-policy-nomad-tasks.hcl. Add the following contents to it and save the file. This policy allows tokens to access any service and KV path in Consul. In production environments, you should adjust the policy should to meet specific access requirements.

consul-policy-nomad-tasks.hcl

key_prefix "" {
  policy = "read"
}

node_prefix "" {
  policy = "read"
}

service_prefix "" {
  policy = "read"
}

Create a Consul ACL policy named nomad-tasks using the file consul-policy-nomad-tasks.hcl.

$ consul acl policy create -name 'nomad-tasks' -description 'ACL policy used by Nomad tasks' -rules '@consul-policy-nomad-tasks.hcl'
ID:           8fc8b321-10d6-0bee-cbe4-9a40812e4bd7
Name:         nomad-tasks
Description:  ACL policy used by Nomad tasks
Datacenters:
Rules:
key_prefix "" {
  policy = "read"
}

node_prefix "" {
  policy = "read"
}

service_prefix "" {
  policy = "read"
}

Create a Consul ACL role for Nomad tasks

Create a Consul ACL role named nomad-tasks-default that matches the name pattern for jobs in the default namespace using the rules set in the file consul-policy-nomad-tasks.hcl.

$ consul acl role create -name 'nomad-tasks-default' -description 'ACL role for Nomad tasks in the default Nomad namespace' -policy-name 'nomad-tasks'
ID:           13c4c9dc-a808-7503-ab07-2e7ef710d398
Name:         nomad-tasks-default
Description:  ACL role for Nomad tasks in the default Nomad namespace
Policies:
   8fc8b321-10d6-0bee-cbe4-9a40812e4bd7 - nomad-tasks

Run a Nomad job to read data from Consul

Write test data to Consul's KV store.

$ consul kv put 'httpd/config/cache' '50'; consul kv put 'httpd/config/maxconn' '100'; consul kv put 'httpd/config/minconn' '3'
Success! Data written to: httpd/config/cache
Success! Data written to: httpd/config/maxconn
Success! Data written to: httpd/config/minconn

Create a file named consul-info.nomad.hcl. Add the following contents to it and save the file.

consul-info.nomad.hcl

job "consul-info" {
  type = "batch"

  group "consul-info" {
    task "consul-info" {
      driver = "docker"

      consul {}

      config {
        image   = "busybox:1.36"
        command = "/bin/sh"
        args    = ["-c", "exit 0"]
      }

      template {
        data        = <<EOF
Consul Services:
{{- range services}}
  * {{.Name}}{{end}}

Consul KV for "httpd/config":
{{- range ls "httpd/config"}}
  * {{.Key}}: {{.Value}}{{end}}
EOF
        destination = "local/consul-info.txt"
      }
    }
  }
}

Note the template block that reads the names of all services registered in Consul and all the keys and values stored in the KV path httpd/config. This information is then written to a file.

Also note how the job does not have any identities for Consul. Nomad uses the task_identity defined in the Nomad agent configuration to retrieve a Consul ACL token for the task.

Run the Nomad job in consul-info.nomad.hcl.

$ nomad job run 'consul-info.nomad.hcl'
==> 2023-11-15T21:22:03-05:00: Monitoring evaluation "94df1181"
    2023-11-15T21:22:03-05:00: Evaluation triggered by job "consul-info"
    2023-11-15T21:22:04-05:00: Allocation "d3504ad8" created: node "6b4dc583", group "consul-info"
    2023-11-15T21:22:04-05:00: Evaluation status changed: "pending" -> "complete"
==> 2023-11-15T21:22:04-05:00: Evaluation "94df1181" finished with status "complete"

Retrieve the allocation information and wait until the status is complete. You may need to run the command a few times before the status changes to complete.

$ nomad job allocs 'consul-info'
ID        Node ID   Task Group   Version  Desired  Status    Created    Modified
d3504ad8  6b4dc583  consul-info  0        run      complete  2m27s ago  2m27s ago

Print the file created by the job.

$ nomad alloc fs "$(nomad job allocs -t '{{with index . 0}}{{.ID}}{{end}}' 'consul-info')" 'consul-info/local/consul-info.txt'
Consul Services:
  * consul
  * httpd
  * nomad
  * nomad-client

Consul KV for "httpd/config":
  * cache: 50
  * maxconn: 100
  * minconn: 3

The job can access the Consul API and retrieve the list of services in the catalog as well as the values in the KV store under the httpd/config path.

Retrieve the task definition from Nomad.

$ nomad job inspect -t '{{sprig_toPrettyJson (index (index .TaskGroups 0).Tasks 0)}}' 'consul-info'
{
  "Name": "consul-info",
  "Driver": "docker",
  "User": "",
  "Lifecycle": null,
  "Config": {
...

Note that it has an identity for Consul in its Identities list.

{
  "Name": "consul-info",
  "Driver": "docker",
  "User": "",
  "Lifecycle": null,
  "Config": {
  ...
  "Identities": [
    {
      "Name": "consul_default",
      "Audience": [
        "consul.io"
      ],
      "ChangeMode": "",
      "ChangeSignal": "",
      "Env": false,
      "File": false,
      "ServiceName": "",
      "TTL": 3600000000000
    }
  ],
  "Actions": null
}

This is the workload identity that Nomad automatically injects based on the task_identity block in the nomad.hcl file.

Nomad uses this identity to retrieve a Consul ACL token with permissions to read services and KV values as defined by the nomad-tasks ACL policy.

Next steps

In this tutorial, you configured Nomad and Consul to communicate with ACL enabled. You also configured Nomad to automatically add workload identities for services and tasks that need access to Consul.

You then deployed a Nomad job to register a service in Consul and another job to read data from it. Both jobs used their workload identities to receive a Consul ACL token properly scoped to the work they needed to do.

This process required several steps, with certain values having to match each other in order for everything to work properly.

There are two resources available to help you automate these steps.

You may continue exploring additional integrations between Nomad and Consul or learn how to similarly integrate Nomad and Vault with ACL enabled.

Collection Overview

Consul

 Next

Secure jobs with service mesh