Export secrets as environment variables with Vault Agent

Applications often require secrets to be available as environment variables. Traditionally, shell scripts or cloud-init scripts set the environment variables before the application starts. This approach has limitations, such as the need to manually update the environment variables when the secrets change.

Consul Template and Envconsul tools have been widely used by the Vault practitioners to help integrate Vault in their existing solutions.

The Vault process supervisor mode retrieves secrets from Vault, and creates environment variables using Consul Template markup.

Prerequisites

This tutorial was tested on macOS using an x86_64 based and Apple silicon-based processors.

To perform the tasks described in this tutorial, you need to have:

• Vault binary 1.14.0 or later installed.
• Git installed.

Set up the lab

1. Launch the HCP Portal and login.

2. Click Vault in the left navigation pane.

3. In the Vault clusters pane, click vault-cluster.

4. Under Cluster URLs, click Public Cluster URL.

5. In a terminal, set the VAULT_ADDR environment variable to the copied address.

   $ export VAULT_ADDR=<Public_Cluster_URL>
   

6. Return to the Overview page and click Generate token.

7. Copy the Admin Token.

8. Return to the terminal and set the VAULT_TOKEN environment variable.

   $ export VAULT_TOKEN=<token>
   

9. Set the VAULT_NAMESPACE environment variable to admin.

   $ export VAULT_NAMESPACE=admin
   

   The admin namespace is the top-level namespace automatically created by HCP Vault. All CLI operations default to use the namespace defined in this environment variable.

   Vault server is ready.

Set up test secrets

1. Clone the learn-vault-agent-envconsul repository which contains the example configuration used in this tutorial.

   $ git clone https://github.com/hashicorp-education/learn-vault-agent-envconsul
   

2. Change into the learn-vault-agent-envconsul directory.

   $ cd learn-vault-agent-envconsul
   

   The folder includes the following files:

   .
   ├── README.md
   ├── kv-demo.sh
   ├── pki-agent-config.hcl
   ├── pki-demo.sh
   └── setup-secrets.sh
   
   0 directories, 5 files
   

   Working directory

   The remainder of the commands in this tutorial must run from this directory.

3. Run the setup-secrets.sh script. This script enables kv-v2 secrets engine at web-team/ and dev-app/ paths and creates initial test data. Also, it configures PKI secrets engine to generate certificate which you will use in a later section.

   $ ./setup-secrets.sh
   

4. Read the secrets created at web-team/data/api-keys.

   $ vault kv get -mount=web-team api-keys
   
   ===== Secret Path =====
   web-team/data/api-keys
   
   ======= Metadata =======
   Key                Value
   ---                -----
   created_time       2023-06-13T21:05:24.965199Z
   custom_metadata    <nil>
   deletion_time      n/a
   destroyed          false
   version            1
   
   ==== Data ====
   Key     Value
   ---     -----
   key1    4984h33sd
   key2    o48r3eqf
   

5. Read the secrets created at dev-app/data/creds.

   $ vault kv get -mount=dev-app creds
   
   === Secret Path ===
   dev-app/data/creds
   
   ======= Metadata =======
   Key                Value
   ---                -----
   created_time       2023-06-13T21:05:25.042033Z
   custom_metadata    <nil>
   deletion_time      n/a
   destroyed          false
   version            1
   
   ====== Data ======
   Key         Value
   ---         -----
   password    my-long-password
   user        tester1
   

6. Read the secrets created at dev-app/data/creds/database/db-admin.

   $ vault kv get -mount=dev-app creds/database/db-admin
   
   ============ Secret Path ============
   dev-app/data/creds/database/db-admin
   
   ======= Metadata =======
   Key                Value
   ---                -----
   created_time       2023-06-13T21:05:25.10932Z
   custom_metadata    <nil>
   deletion_time      n/a
   destroyed          false
   version            1
   
   ====== Data ======
   Key         Value
   ---         -----
   password    o3ir23ir
   username    admin
   

   Vault is running and ready to demonstrate Vault Agent's ability to export secrets as environment variables.

   If you are not familiar with Vault policies, secrets engines, or auth methods, review the Vault get started tutorials.

Generate a Vault Agent config file

Vault Agent introduced the generate-config subcommand to auto generate the agent configuration file based on the user input.

The generate-config subcommand supports the kv-v1 and kv-v2 secrets engines.

Example usage:

$ vault agent generate-config [options] <path_to_config>

Run the generate-config subcommand

Generate a Vault Agent configuration file named agent-config.hcl in the current working directory. If your application needs to pull secrets from multiple paths, you can provide multiple -path options as well as wildcards (*).

1. Generate a Vault Agent configuration file named agent-config.hcl. Pass the kv-demo.sh script as the command to execute.

   $ vault agent generate-config -type="env-template" \
       -exec="./kv-demo.sh" \
       -path="web-team/api-keys" \
       -path="dev-app/*" \
       agent-config.hcl
   

   Output:

   Successfully generated "agent-config.hcl" configuration file!
   Warning: the generated file uses 'token_file' authentication method, which is
   not suitable for production environments.
   

   In the absence of the output file name, the command will generate a configuration file named agent.hcl.

2. Open the generated configuration file with your preferred text editor to examine.

   $ cat agent-config.hcl
   

   Example output:

   agent-config.hcl

   auto_auth {
      method {
         type = "token_file"
   
         config {
            token_file_path = "/Users/username/.vault-token"
         }
      }
   }
   
   template_config {
      static_secret_render_interval = "5m"
      exit_on_retry_failure         = true
   }
   
   vault {
      address = "https://vault-cluster-public-vault-XXXXXXXXX.XXXXXXXXX.XX.hashicorp.cloud:8200"
   }
   
   env_template "API_KEYS_KEY1" {
      contents             = "{{ with secret \"web-team/data/api-keys\" }}{{ .Data.data.key1 }}{{ end }}"
      error_on_missing_key = true
   }
   env_template "API_KEYS_KEY2" {
      contents             = "{{ with secret \"web-team/data/api-keys\" }}{{ .Data.data.key2 }}{{ end }}"
      error_on_missing_key = true
   }
   env_template "CREDS_PASSWORD" {
      contents             = "{{ with secret \"dev-app/data/creds\" }}{{ .Data.data.password }}{{ end }}"
      error_on_missing_key = true
   }
   env_template "CREDS_USER" {
      contents             = "{{ with secret \"dev-app/data/creds\" }}{{ .Data.data.user }}{{ end }}"
      error_on_missing_key = true
   }
   env_template "DB_ADMIN_PASSWORD" {
      contents             = "{{ with secret \"dev-app/data/creds/database/db-admin\" }}{{ .Data.data.password }}{{ end }}"
      error_on_missing_key = true
   }
   env_template "DB_ADMIN_USERNAME" {
      contents             = "{{ with secret \"dev-app/data/creds/database/db-admin\" }}{{ .Data.data.username }}{{ end }}"
      error_on_missing_key = true
   }
   
   exec {
      command                   = ["./kv-demo.sh"]
      restart_on_secret_changes = "always"
      restart_stop_signal       = "SIGTERM"
   }
   

   The auto_auth stanza is using token_file as a placeholder. You can change the stanza to use your desired auth method.

   See the following tutorials for other auto auth examples:

   • Vault Agent with AWS demonstrates AWS auth method
   • Vault Agent with Kubernetes demonstrates Kubernetes auth method

Process supervisor mode

Each generated env_template block maps to a secret key of a KV path that you passed in the command.

The exec block sets the kv-demo.sh script as the commands that Vault Agent to execute.

Open and examine the kv-demo.sh file. The script reads secrets as environment variables.

$ cat kv-demo.sh

#!/bin/sh

echo
echo "==== KV DEMO APP ===="
echo "API_KEYS_KEY1     = ${API_KEYS_KEY1}"
echo "API_KEYS_KEY2     = ${API_KEYS_KEY2}"
echo "CREDS_USER        = ${CREDS_USER}"
echo "CREDS_PASSWORD    = ${CREDS_PASSWORD}"
echo "DB_ADMIN_USERNAME = ${DB_ADMIN_USERNAME}"
echo "DB_ADMIN_PASSWORD = ${DB_ADMIN_PASSWORD}"

This application does not need to know about Vault.

Start a Vault Agent

Start a Vault Agent instance that connects to the Vault server running at VAULT_ADDR.

1. From your current terminal, print the working directory and Vault environment variables.

   $ pwd && printenv | grep VAULT_
   
   /Users/yourradusername/learn-vault-agent-envconsul
   VAULT_ADDR=https://vault-cluster-public-vault-XXXXXXXXX.XXXXXXXXX.XX.hashicorp.cloud:8200
   VAULT_NAMESPACE=admin
   VAULT_TOKEN=hvs.S3ASAM3...STR33T
   

2. Open a new terminal to run Vault Agent.

3. Change to the working directory printed in the previous step.

   Example:

   $ cd /Users/yourradusername/learn-vault-agent-envconsul
   

4. Set the VAULT_ADDR environment variable.

   $ export VAULT_ADDR=<Public_Cluster_URL>
   

5. Set the VAULT_NAMESPACE environment variable.

   $ export VAULT_NAMESPACE=admin
   

6. Set the VAULT_TOKEN environment variable from the HCP Portal.

   $ export VAULT_TOKEN=<admin_token>
   

7. Log into Vault. This step stores the authentication token in the expected directory for the Vault Agent.

   $ vault login $VAULT_TOKEN
   

8. Start a Vault Agent.

   $ vault agent -config=agent-config.hcl -log-level=error
   

   Example output:

   ==> Vault Agent started! Log data will stream in below:
   
   ==> Vault Agent configuration:
   
            Api Address 1: http://bufconn
                        Cgo: disabled
                  Log Level: error
                  Version: Vault v1.14.0-rc1, built 2023-06-06T18:12:53Z
               Version Sha: 1327dc6728853611f62708e28dbac3ce6cabad1a
   
   ==== KV DEMO APP ====
   API_KEYS_KEY1     = 4984h33sd
   API_KEYS_KEY2     = o48r3eqf
   CREDS_USER        = tester1
   CREDS_PASSWORD    = my-long-password
   DB_ADMIN_USERNAME = admin
   DB_ADMIN_PASSWORD = o3ir23ir
   

   Vault Agent stops and displays the secret values read from Vault.

Restarts on secrets change

The secret values can change while your application is running. Examine the behavior when the secret values change.

1. Add sleep 100 in the kv-demo.sh to mimic a long running application.

   kv-demo.sh

   $ printf '\nsleep 100' >> kv-demo.sh && cat kv-demo.sh
   
   #!/bin/sh
   
   echo
   echo "==== DEMO APP ===="
   echo "API_KEYS_KEY1     = ${API_KEYS_KEY1}"
   echo "API_KEYS_KEY2     = ${API_KEYS_KEY2}"
   echo "CREDS_USER        = ${CREDS_USER}"
   echo "CREDS_PASSWORD    = ${CREDS_PASSWORD}"
   echo "DB_ADMIN_USERNAME = ${DB_ADMIN_USERNAME}"
   echo "DB_ADMIN_PASSWORD = ${DB_ADMIN_PASSWORD}"
   
   sleep 100
   

2. Edit the agent-config.sh file and set the static_secret_render_interval to 10 seconds. The static_secret_render_interval setting makes Vault Agent pull the secrets every 10 seconds.

   $ sed -i '' 's/static_secret_render_interval = "5m"/static_secret_render_interval = "10s"/' agent-config.hcl \
   && cat agent-config.hcl
   

   Example output:

   agent-config.hcl

   auto_auth {
      method {
         type = "token_file"
   
         config {
            token_file_path = "/Users/username/.vault-token"
         }
      }
   }
   
   template_config {
      static_secret_render_interval = "10s"
      exit_on_retry_failure         = true
   }
   
   vault {
      address = "$VAULT_ADDR"
   }
   
   ...snip...
   

3. Start the Vault Agent again.

   $ vault agent -config=agent-config.hcl -log-level=error
   
   ==> Vault Agent started! Log data will stream in below:
   
   ==> Vault Agent configuration:
   
            Api Address 1: http://bufconn
                        Cgo: disabled
                  Log Level: error
                  Version: Vault v1.14.0-rc1, built 2023-06-06T18:12:53Z
               Version Sha: 1327dc6728853611f62708e28dbac3ce6cabad1a
   
   ==== KV DEMO APP ====
   API_KEYS_KEY1     = 4984h33sd
   API_KEYS_KEY2     = o48r3eqf
   CREDS_USER        = tester1
   CREDS_PASSWORD    = my-long-password
   DB_ADMIN_USERNAME     = admin
   DB_ADMIN_PASSWORD = o3ir23ir
   

4. Switch back to the terminal where you connect to Vault.

5. Update the secret values at web-team/data/api-keys.

   $ vault kv put web-team/api-keys \
      key1="new-key-value-1" \
      key2="new-key-value-2"
   

   Return to the terminal where Vault Agent is running and watch the output.

   ==> Vault Agent started! Log data will stream in below:
   
   ==> Vault Agent configuration:
   
            Api Address 1: http://bufconn
                        Cgo: disabled
                  Log Level: error
                  Version: Vault v1.14.0-rc1, built 2023-06-06T18:12:53Z
               Version Sha: 1327dc6728853611f62708e28dbac3ce6cabad1a
   
   ==== KV DEMO APP ====
   API_KEYS_KEY1     = 4984h33sd
   API_KEYS_KEY2     = o48r3eqf
   CREDS_USER        = tester1
   CREDS_PASSWORD    = my-long-password
   DB_ADMIN_USERNAME = admin
   DB_ADMIN_PASSWORD = o3ir23ir
   
   ==== KV DEMO APP ====
   API_KEYS_KEY1     = new-key-value-1
   API_KEYS_KEY2     = new-key-value-2
   CREDS_USER        = tester1
   CREDS_PASSWORD    = my-long-password
   DB_ADMIN_USERNAME = admin
   DB_ADMIN_PASSWORD = o3ir23ir
   

6. Press Ctrl + C to stop the running Vault Agent.

Work with dynamic secrets

The generate-config subcommand supports kv-v1 and kv-v2 secrets engines; however, the Vault Agent's process supervisor mode supports dynamic secrets that Envconsul supports today. The difference is that you have to manually define the env_template blocks.

1. The setup-secrets.sh script configured the PKI secrets engine.

   $ vault secrets list
   

   Example output:

   Path          Type         Accessor              Description
   ----          ----         --------              -----------
   cubbyhole/    cubbyhole    cubbyhole_595f8b0f    per-token private secret storage
   dev-app/      kv           kv_f16c3aac           n/a
   identity/     identity     identity_cd1c4e99     identity store
   pki/          pki          pki_ca9e7398          n/a
   pki_int/      pki          pki_331d3c63          n/a
   secret/       kv           kv_1bb31236           key/value secret storage
   sys/          system       system_a6bb36e2       system endpoints used for control, policy and debugging
   web-team/     kv           kv_c5dd184d           n/a
   

2. Request a certificate.

   $ vault write -field=certificate pki_int/issue/example-dot-com \
      common_name="test.example.com" ttl="24h"
   

   Example output:

   -----BEGIN CERTIFICATE-----
   MIIDZjCCAk6gAwIBAgIUZl818nTOMzvFdlhcnE5YYZ/WC5UwDQYJKoZIhvcNAQEL
   BQAwLTErMCkGA1UEAxMiZXhhbXBsZS5jb20gSW50ZXJtZWRpYXRlIEF1dGhvcml0
   eTAeFw0yMzA2MTMyMzE3MjdaFw0yMzA2MTQyMzE3NTdaMBsxGTAXBgNVBAMTEHRl
   c3QuZXhhbXBsZS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDV
   IrsKREu2+lQbNbdYgYzMYAidevxVV3K4kWe9MYQWW4xLus/DtuBZXwqPkPsl3UGs
   NRjv4gE9hvb3lGAcs5Ze4f27sK/FvXgcOXy/p1wJ8JQV+rMHq/cg3dcEvYvY/f2c
   IMeR0vMJb58POjM4iWSgvYfvQtO+4r0hPydGGafubhorRvlvEMfiNfghDG3wSoeJ
   Z0OmwQenBY6ASsJZ32qJbuiVszSCHIgOP5s6GfF6KKkKj3ojr/aH++ff+ckDzFbW
   /uz0IGV2hhdGEnnSjlWn0uaPR6xFGqprp1csAAZtdY50q6PYZ7Ev8Oe3zFPNM4sj
   uXETsXcjRUUusC1vGxYrAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgOoMB0GA1Ud
   JQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAdBgNVHQ4EFgQUcu+At+HNbaGOD8CV
   voKRxf9Q4JMwHwYDVR0jBBgwFoAUOiYGxwIjDoTz389uuIGb9VWmEKUwGwYDVR0R
   BBQwEoIQdGVzdC5leGFtcGxlLmNvbTANBgkqhkiG9w0BAQsFAAOCAQEAQbHfB/gc
   pG9n2Pb9wfw9wek+Ko1abmf2rgkqD44z6lvS+kpRi1SIwY2oUEhXvILdl098M4D7
   zRhWfCYyBuN+jVuGpZjqVlc3nS976JzBil3MzlfvUSbscoao0oh6unaBfYlGlgR+
   STNjjeQDx+bVWL3os8u3WqY+XlyVVHnBAQME+9Hgiamzv0gKE2vEb93KZ33rozjf
   wSTNzEfvQOHTAjkMZ2rwAu2prhLq4e8jUM8BA4QxFiw9R+vx1Dnu3Kz3W0eLhQ2W
   XG9YH57A234vOc/dcW6teNq8QzYLuqzA4oNKpBWiWT7uLt2hP0L3OtUliqDZAzFZ
   pxt9m7E2V9qQcQ==
   -----END CERTIFICATE-----
   

3. Open the pki-agent-config.hcl to examine the env_template block it defines.

   $ cat pki-agent-config.hcl
   
   env_template "CERT" {
      contents = "{{ with pkiCert \"pki_int/issue/example-dot-com\" \"common_name=test.example.com\" \"ttl=24h\"}}{{ .Data.Cert }}{{ end }}"
      error_on_missing_key = true
   }
   
   exec {
      command                   = ["./pki-demo.sh"]
      restart_on_secret_changes = "always"
      restart_stop_signal       = "SIGTERM"
   }
   

4. Open the pki-demo.sh file to review.

   $ cat pki-demo.sh
   
   #!/bin/sh
   
   echo
   echo "==== CERT DEMO APP ===="
   echo "CERT = ${CERT}"
   

   This mock application uses a certificate stored in the CERT environment variable. Similar to the kv-demo.sh, this application does not need to know anything about Vault.

5. Switch back to the terminal where you started the Vault Agent earlier.

6. Run a Vault Agent with the env_template definition.

   $ vault agent -config=agent-config.hcl \
      -config=pki-agent-config.hcl \
      -log-level=error
   

   Example output:

   ==> Vault Agent started! Log data will stream in below:
   
   ==> Vault Agent configuration:
   
            Api Address 1: http://bufconn
                        Cgo: disabled
                  Log Level: error
                  Version: Vault v1.14.0-rc1, built 2023-06-06T18:12:53Z
               Version Sha: 1327dc6728853611f62708e28dbac3ce6cabad1a
   
   ==== CERT DEMO APP ====
   CERT = -----BEGIN CERTIFICATE-----
   MIIDZjCCAk6gAwIBAgIUdTJEGBZtcnxlo8f0mMLVzKvXbtgwDQYJKoZIhvcNAQEL
   BQAwLTErMCkGA1UEAxMiZXhhbXBsZS5jb20gSW50ZXJtZWRpYXRlIEF1dGhvcml0
   eTAeFw0yMzA2MTMyMzQ3MDNaFw0yMzA2MTQyMzQ3MzNaMBsxGTAXBgNVBAMTEHRl
   c3QuZXhhbXBsZS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC3
   GXTT5UlMnH5OJKZbx7n9DJs4K4lgrKdirpD4mVL7O4ba3KUA/mam9M6LAqn5YzB6
   UX6y1+zIyNaakqltxLVxiyLWodqix3fjN9g6jzis5JAQM3UIgVYPBpg5ptKzpVzH
   ASvuDonnPflBYQG609rD56RK8ycyhq0bcUVCa6pEQI3EtJaqj2woOTVWyauo+r7g
   jrwr49BL0ye0T1IoTSs3l0E5Pk6ZcTDEbK3rNZ+YWSzwcg6bAL1U8kOW3B3jINwI
   GYXPKVVLM2v6TTYpeYB/fZ6FeQeDasVU3iy3N16HdIHbI2vAQqUkTZcv/LM1ALgf
   DbEYOUvAcdNCZKifhDTdAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgOoMB0GA1Ud
   JQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAdBgNVHQ4EFgQUBIsJvc4reYPkq1Vy
   tuf3NMAHvD0wHwYDVR0jBBgwFoAUOiYGxwIjDoTz389uuIGb9VWmEKUwGwYDVR0R
   BBQwEoIQdGVzdC5leGFtcGxlLmNvbTANBgkqhkiG9w0BAQsFAAOCAQEAZyN5dSKo
   9E8G6kaArX6Jfq8CGdX1jhaqFH8SljYZ5/WngzJwuQNkvCU42TpAbvHsBfen93T/
   VfGgPPh6GohwhvLbVQjTLzGBlFTq+oQJs0ntQxjL/CMEfqg0aLH1OnZuVULc9J6z
   LaOHhsV4EHx7c5iSuwwun7NOJgysadboS8EE4BOqkKehaEz/ZXVVM5sJhRpi8KfR
   I9EA+FNG6URKllW72OM2kbg9HVQsxsIPdG+o/JDsCz48yMwxJDyLAxMH/YtOVrbF
   j55N7ooFPw/h3BhMP+0zzDrQrl9GPYRCe//10zmTA+4/q2vQ3nPMVeInEa+QdTBi
   hxz5k4LuDjhuoA==
   -----END CERTIFICATE-----
   

   Similar to what you observed in the restarts on secrets change section, if the certificate expires and the application is still running, Vault Agent can fetch a new certificate.

Clean up

Summary

You learned the use of the generate-config subcommand and the process supervisor mode to take advantage of secrets presented as environment variables.

Vault Agent can handle the authentication and secrets retrieval so that your application can remain Vault unaware. This reduces the barrier to adopting Vault and keep your applications secure.

There are several tutorials demonstrates the use of Vault Agent. See the available Vault Agent tutorials.

Help and reference

• Vault Agent overview
• Vault Agent generate-config
• Vault Agent's Process Supervisor Mode
• Vault Agent Auto-Auth