Configure Certificate Authority (CA) for Consul on Kubernetes

"--- layout: docs page_title: Configure Certificate Authority (CA) for Consul on Kubernetes description: >- Consul includes a built-in CA, but when bootstrapping a cluster on k8s, you can configure your service mesh to use a custom certificate provider instead. Learn how to configure Vault as an external CA in primary and secondary datacenters and manually rotate Vault tokens. ---"

NOTE: The instructions below should only be used for initially bootstrapping a cluster with Consul K8s 0.38.0+. To update the Connect CA provider on an existing cluster or to update any properties, such as tokens, of the CA provider, please use the Update CA Configuration Endpoint.

Consul has support for different certificate authority (CA) providers to be used with the Consul Service Mesh. Please see Connect Certificate Management for the information on the providers we currently support.

If Connect is enabled, the built-in Consul CA is automatically enabled for the Connect CA. To configure an external CA provider via the Consul Helm chart, you need to follow three steps:

Create a configuration file containing your provider information.
Create a Kubernetes secret containing the configuration file.
Reference the Kubernetes secret in the server.extraVolumes value in the Helm chart.

To configure the Vault Connect Provider please see Vault as the Service Mesh Certificate Provider on Kubernetes.

NOTE: The following instructions are only valid for Consul-k8s 0.37.0 and prior.

Below we will go over the process for configuring Vault as the Connect CA. However, other providers can similarly be configured during initial bootstrap of the cluster by providing the appropriate ca_config and ca_provider values for the provider you're using.

Configuring Vault as a Connect CA (Consul K8s 0.37.0 and earlier)

NOTE: If using Vault as your Connect CA, it's highly recommended to run a Consul version >= 1.8.5 that supports token auto-renewal. With this feature, if the Vault token is renewable then Consul will automatically renew the token periodically. Without this feature, you will need to manually rotate the Vault token before it expires.

Primary Datacenter

To configure Vault as a CA provider for Consul Connect, first, create a provider configuration JSON file. Please refer to Vault as a Connect CA for the configuration options. You will need to provide a Vault token to the token property. Please refer to these docs for the permissions that the token needs to have. This token should be renewable.

To provide a CA, you first need to create a Kubernetes secret containing the CA. For example, you may create a secret with the Vault CA like so:

kubectl create secret generic vault-ca --from-file vault.ca=/path/to/your/vault/ca

And then reference it like this in the provider configuration:

{
  "connect": [
    {
      "ca_config": [
        {
          "address": "https://vault:8200",
          "intermediate_pki_path": "dc1/connect-intermediate",
          "root_pki_path": "connect-root",
          "token": "s.VgQvaXl8xGFO1RUxAPbPbsfN",
          "ca_file": "/consul/userconfig/vault-ca/vault.ca"
        }
      ],
      "ca_provider": "vault"
    }
  ]
}

This example configuration file is pointing to a Vault instance running in the same Kubernetes cluster, which has been deployed with TLS enabled. Note that the ca_file is pointing to the file location based on the Kubernetes secret for the Vault CA that we have created before. We will provide that secret later in the Helm values for our Consul cluster.

NOTE: If you have used Kubernetes CA to sign Vault's certificate, such as shown in Standalone Server with TLS, you don't need to create a Kubernetes secret with Vault's CA and can reference the CA directly by setting ca_file to /var/run/secrets/kubernetes.io/serviceaccount/ca.crt.

Next, create a Kubernetes secret with this configuration file.

$ kubectl create secret generic vault-config --from-file=config=vault-config.json

We will provide this secret and the Vault CA secret, to the Consul server via the server.extraVolumes Helm value.

global:
  name: consul
server:
  extraVolumes:
    - type: secret
      name: vault-config
      load: true
      items:
        - key: config
          path: vault-config.json
    - type: secret
      name: vault-ca
      load: false
connectInject:
  enabled: true

Finally, install the Helm chart using the above config file:

$ helm install consul --values config.yaml hashicorp/consul

Verify that the CA provider is set correctly:

$ kubectl exec consul-server-0 -- curl --silent http://localhost:8500/v1/connect/ca/configuration\?pretty
{
  "Provider": "vault",
  "Config": {
    "Address": "https://vault:8200",
    "CAFile": "/consul/userconfig/vault-server-tls/vault.ca",
    "IntermediateCertTTL": "8760h",
    "IntermediatePKIPath": "connect-intermediate",
    "LeafCertTTL": "72h",
    "RootPKIPath": "connect-root",
    "Token": "s.VgQvaXl8xGFO1RUxAPbPbsfN"
  },
  "State": null,
  "ForceWithoutCrossSigning": false,
  "CreateIndex": 5,
  "ModifyIndex": 5
}

Secondary Datacenters

To configure Vault as the Connect CA in secondary datacenters, you need to make sure that the Root CA is the same, but the intermediate is different for each datacenter. In the connect configuration for a secondary datacenter, you can specify a intermediate_pki_path that is, for example, prefixed with the datacenter for which this configuration is intended. You will similarly need to create a Vault token and a Kubernetes secret with Vault's CA in each secondary Kubernetes cluster.

{
  "connect": [
    {
      "ca_config": [
        {
          "address": "https://vault:8200",
          "intermediate_pki_path": "dc2/connect-intermediate",
          "root_pki_path": "connect-root",
          "token": "s.VgQvaXl8xGFO1RUxAPbPbsfN",
          "ca_file": "/consul/userconfig/vault-ca/vault.ca"
        }
      ],
      "ca_provider": "vault"
    }
  ]
}

Note that all secondary datacenters need to have access to the same Vault instance as the primary.

Manually Rotating Vault Tokens

If running Consul < 1.8.5 or using a Vault token that is not renewable then you will need to manually renew or rotate the Vault token before it expires.

Rotating Vault Token

The ca_config and ca_provider options defined in the Consul agent configuration are only used when initially bootstrapping the cluster. Once the cluster is running, subsequent changes to the ca_provider config are ignored–even if consul reload is run or the servers are restarted.

To update any settings under these keys, you must use Consul's Update CA Configuration API or the consul connect ca set-config command.

Renewing Vault Token

To renew the Vault token, use the vault token renew CLI command or API.