Identity and access management

The access control lists (ACL) system in Nomad provides authentication and authorization. This gives Nomad administrators fine-grained control over what users and applications can do within the Nomad environment.

Properly structuring ACL policies and roles ensures that only authorized users and applications can interact with the Nomad cluster, and only to the level they need to. This helps prevent accidental or malicious actions that could disrupt the cluster or compromise sensitive data.

Structuring ACLs is organization-dependent. These are the best practices.

• Create role-based policies:
  • Define roles like developer, operator, and admin instead of one-off tokens.
• Use the principle of least privilege:
  • Grant minimal permissions necessary for each role.
• Use namespaces:
  • Use namespaces to isolate different tenants, teams, applications or environments.
• Use wildcards judiciously:
  • Be cautious with wildcards to avoid overly broad permissions.
• Avoid static/long-lived tokens:
  • Use either short lived Nomad ACL tokens, the Vault Nomad secrets engine(opens in new tab) for time-bound dynamic tokens, or Nomad single sign-on with OIDC(opens in new tab) which allows trusting external OIDC-capable identity providers, such as Okta, Auth0, EntraID for human access, or GitHub Actions, GitLab CI or Vault for machine access.
• Implement a robust offboarding process for any static tokens:
  • Ensure timely removal of access when users leave/applications no longer need access.
• Regular access audits:
  • Review ACL policies and any static tokens periodically.
• Enable audit logs:
  • Use audit logs to monitor and detect suspicious activity.
• Use Sentinel(opens in new tab):
  • Sentinel is a policy as code framework that allows you to enforce fine-grained policies at job deploy time. Use these to enforce extra security or business logic (such as to only authorize Docker containers from a specific Docker registry, or disallow deploys on Friday afternoon, and so on.)
• Treat namespaces and ACL policies as code.
  • Store them in a version control system, ideally manage them with Terraform(opens in new tab), and use CI/CD pipelines to deploy them to Nomad. Review and audit changes and apply them consistently across all environments.

ACL management

ACLs comprise of capabilities, policies, roles, and tokens.

Attach policies to tokens through the correct definition of roles. Use tokens to authenticate to Nomad (use Vault or an SSO identity provider to broker this authentication). Policies control which actions users can access in Nomad Enterprise.

Use Terraform to manage namespaces, ACL policies and roles, as it allows for easier version control and code review, as well as for an easier understanding of all the policies in use in the Nomad Enterprise cluster. Used Terraform to also enable a self-service model using Terraform child modules, and to render common combinations of namespace/associated policies.

An example of a policy that allows submitting jobs and a few related activities in a specific namespace, using raw HCL for Nomad or Terraform is below.

# Allow reading resources within the "dev" namespace as well as submitting
# jobs to this namespace, without allowing access to view log output or inspect
# the filesystem.
namespace "dev" {
  capabilities = ["submit-job", "list-jobs", "read-job", "parse-job"]
}

Create such a policy in Nomad using the command-line tool or API as follows.

nomad acl policy apply -description "Allow reading resources within the 'dev' namespace as well as submitting jobs to this namespace, without allowing access to view log output or inspect the filesystem." dev-policy dev.hcl

Create the same policy using the Terraform nomad_acl_policy(opens in new tab) resource from the Nomad Terraform provider(opens in new tab).

resource "nomad_acl_policy" "dev" {
  name        = "dev-policy"
  description = "Allow reading resources within the 'dev' namespace as well as submitting jobs to this namespace, without allowing access to view log output or inspect the filesystem."

  rules_hcl = <<EOT
namespace "dev" {
  capabilities = [
    "submit-job",
    "list-jobs",
    "read-job",
    "parse-job"
  ]
}
EOT
}

Attach the created policy to an ACL role, or use it directly by creating a token with it. Preferably, use SSO to authenticate users and applications to roles.

SSO using OIDC

Using an external identity provider (IdP) for authentication and authorization is a recommended pattern for security and simplicity. Nomad supports OIDC (OpenID Connect) for Single Sign-On (SSO) with external identity providers such as Okta, Auth0, EntraID, and so on. for human auth, and GitHub Actions, GitLab CI for machine auth. This allows users, pipelines, and applications to authenticate to Nomad using their existing identity, and Nomad to trust the IdP to provide the necessary information about the user, such as their username, groups, metadata, and so on. This is more secure than static tokens, as it allows for centralized management of users and their access, which is short-lived and time bound. It also allows for revocation of access when users leave the organization or no longer need access.

To enable OIDC, you need to configure the necessary ACL policies and ACL roles in Nomad, and create an auth method (using the command-line tool(opens in new tab) or preferably, the nomad_acl_auth_method(opens in new tab) Terraform resource).

Workload identity

The ACL system is what enables Nomad to provide Workload Identity to each workload running in Nomad, and serve as an OIDC identity provider. This allows to uniquely and securely authenticate and authorize workloads running in Nomad without any static credentials needed. This enables workloads running in Nomad to authenticate to Nomad itself, Vault, assume a role in AWS IAM/GCP IAM, or any other OIDC/JWT compatible system.

Internal

By default, each allocation in Nomad gets an identity which gives it implicit access to the Nomad API. This identity is unique to the allocation, and gives it read-only access to Nomad variables in the respective path for that allocation's job/group/task (nomad/jobs/job/task_group/task). Attach ACL policies explicitly to jobs/groups/tasks to enhance that identity, and provide granular access to alternative Nomad variable paths, or the Nomad Task API which allows for secure and local access to the Nomad API from within tasks (perfect for load balancers, control tasks, automation, and so on.)

Third party OIDC-compatible systems

Configure the OIDC issuer in Nomad to use workload identity for auth to OIDC-compatible third parties (such as Vault, AWS IAM, and GCP IAM). Configure trust between the external system and Nomad as an identity provider (IdP).

The exact flow and configuration depends on the third party system trusting Nomad.

• Vault: Nomad workloads can authenticate to the OIDC/JWT Vault Authentication method.
• AWS IAM: Nomad workloads can assume AWS IAM roles.
• GCP Workload Identity Federation: Federate GCP IAM identities to Nomad workloads.