Vault
Export secrets as environment variables with Vault Agent
Applications often require secrets to be available as environment variables. Traditionally, shell scripts or cloud-init scripts set the environment variables before the application starts. This approach has limitations, such as the need to manually update the environment variables when the secrets change.
Consul Template and Envconsul tools have been widely used by the Vault practitioners to help integrate Vault in their existing solutions.
The Vault process supervisor mode retrieves secrets from Vault, and creates environment variables using Consul Template markup.
Prerequisites
This tutorial was tested on macOS using an x86_64 based and Apple silicon-based processors.
To perform the tasks described in this tutorial, you need to have:
- Vault binary 1.14.0 or later installed.
- Git installed.
Set up the lab
Note
If you do not have access to an HCP Vault Dedicated cluster, visit the Create a Vault Cluster on HCP tutorial.
Launch the HCP Portal and login.
Click Vault in the left navigation pane.
In the Vault clusters pane, click vault-cluster.
Under Cluster URLs, click Public Cluster URL.
In a terminal, set the
VAULT_ADDR
environment variable to the copied address.$ export VAULT_ADDR=<Public_Cluster_URL>
Return to the Overview page and click Generate token.
Copy the Admin Token.
Return to the terminal and set the
VAULT_TOKEN
environment variable.$ export VAULT_TOKEN=<token>
Set the
VAULT_NAMESPACE
environment variable toadmin
.$ export VAULT_NAMESPACE=admin
The
admin
namespace is the top-level namespace automatically created by HCP Vault. All CLI operations default to use the namespace defined in this environment variable.Vault server is ready.
Set up test secrets
Clone the learn-vault-agent-envconsul repository which contains the example configuration used in this tutorial.
$ git clone https://github.com/hashicorp-education/learn-vault-agent-envconsul
Change into the
learn-vault-agent-envconsul
directory.$ cd learn-vault-agent-envconsul
The folder includes the following files:
. ├── README.md ├── kv-demo.sh ├── pki-agent-config.hcl ├── pki-demo.sh └── setup-secrets.sh 0 directories, 5 files
Working directory
The remainder of the commands in this tutorial must run from this directory.
Run the
setup-secrets.sh
script. This script enables kv-v2 secrets engine atweb-team/
anddev-app/
paths and creates initial test data. Also, it configures PKI secrets engine to generate certificate which you will use in a later section.$ ./setup-secrets.sh
Read the secrets created at
web-team/data/api-keys
.$ vault kv get -mount=web-team api-keys ===== Secret Path ===== web-team/data/api-keys ======= Metadata ======= Key Value --- ----- created_time 2023-06-13T21:05:24.965199Z custom_metadata <nil> deletion_time n/a destroyed false version 1 ==== Data ==== Key Value --- ----- key1 4984h33sd key2 o48r3eqf
Read the secrets created at
dev-app/data/creds
.$ vault kv get -mount=dev-app creds === Secret Path === dev-app/data/creds ======= Metadata ======= Key Value --- ----- created_time 2023-06-13T21:05:25.042033Z custom_metadata <nil> deletion_time n/a destroyed false version 1 ====== Data ====== Key Value --- ----- password my-long-password user tester1
Read the secrets created at
dev-app/data/creds/database/db-admin
.$ vault kv get -mount=dev-app creds/database/db-admin ============ Secret Path ============ dev-app/data/creds/database/db-admin ======= Metadata ======= Key Value --- ----- created_time 2023-06-13T21:05:25.10932Z custom_metadata <nil> deletion_time n/a destroyed false version 1 ====== Data ====== Key Value --- ----- password o3ir23ir username admin
Vault is running and ready to demonstrate Vault Agent's ability to export secrets as environment variables.
If you are not familiar with Vault policies, secrets engines, or auth methods, review the Vault get started tutorials.
Generate a Vault Agent config file
Vault Agent introduced the generate-config subcommand to auto generate the agent configuration file based on the user input.
The generate-config
subcommand supports the
kv-v1 and kv-v2
secrets engines.
Example usage:
$ vault agent generate-config [options] <path_to_config>
Option | Description |
---|---|
-type | env-template is the only supported type of agent configuration |
-exec | Pass the command to execute in agent process. The default is env |
-path | Path to a kv-v1 or kv-v2 secret. Multiple paths, and tail '*' wildcards supported. |
Run the generate-config subcommand
Generate a Vault Agent configuration file named agent-config.hcl
in the
current working directory. If your application needs to pull secrets from
multiple paths, you can provide multiple -path
options as well as wildcards (*
).
Generate a Vault Agent configuration file named
agent-config.hcl
. Pass thekv-demo.sh
script as the command to execute.$ vault agent generate-config -type="env-template" \ -exec="./kv-demo.sh" \ -path="web-team/api-keys" \ -path="dev-app/*" \ agent-config.hcl
Output:
Successfully generated "agent-config.hcl" configuration file! Warning: the generated file uses 'token_file' authentication method, which is not suitable for production environments.
In the absence of the output file name, the command will generate a configuration file named
agent.hcl
.Open the generated configuration file with your preferred text editor to examine.
$ cat agent-config.hcl
Example output:
agent-config.hcl
auto_auth { method { type = "token_file" config { token_file_path = "/Users/username/.vault-token" } } } template_config { static_secret_render_interval = "5m" exit_on_retry_failure = true } vault { address = "https://vault-cluster-public-vault-XXXXXXXXX.XXXXXXXXX.XX.hashicorp.cloud:8200" } env_template "API_KEYS_KEY1" { contents = "{{ with secret \"web-team/data/api-keys\" }}{{ .Data.data.key1 }}{{ end }}" error_on_missing_key = true } env_template "API_KEYS_KEY2" { contents = "{{ with secret \"web-team/data/api-keys\" }}{{ .Data.data.key2 }}{{ end }}" error_on_missing_key = true } env_template "CREDS_PASSWORD" { contents = "{{ with secret \"dev-app/data/creds\" }}{{ .Data.data.password }}{{ end }}" error_on_missing_key = true } env_template "CREDS_USER" { contents = "{{ with secret \"dev-app/data/creds\" }}{{ .Data.data.user }}{{ end }}" error_on_missing_key = true } env_template "DB_ADMIN_PASSWORD" { contents = "{{ with secret \"dev-app/data/creds/database/db-admin\" }}{{ .Data.data.password }}{{ end }}" error_on_missing_key = true } env_template "DB_ADMIN_USERNAME" { contents = "{{ with secret \"dev-app/data/creds/database/db-admin\" }}{{ .Data.data.username }}{{ end }}" error_on_missing_key = true } exec { command = ["./kv-demo.sh"] restart_on_secret_changes = "always" restart_stop_signal = "SIGTERM" }
The
auto_auth
stanza is usingtoken_file
as a placeholder. You can change the stanza to use your desired auth method.See the following tutorials for other auto auth examples:
- Vault Agent with AWS demonstrates AWS auth method
- Vault Agent with Kubernetes demonstrates Kubernetes auth method
Process supervisor mode
Each generated env_template
block maps to a secret key of a KV path that you
passed in the command.
The exec
block sets the kv-demo.sh
script as the commands that Vault Agent
to execute.
Open and examine the kv-demo.sh
file. The script reads secrets as environment
variables.
$ cat kv-demo.sh
#!/bin/sh
echo
echo "==== KV DEMO APP ===="
echo "API_KEYS_KEY1 = ${API_KEYS_KEY1}"
echo "API_KEYS_KEY2 = ${API_KEYS_KEY2}"
echo "CREDS_USER = ${CREDS_USER}"
echo "CREDS_PASSWORD = ${CREDS_PASSWORD}"
echo "DB_ADMIN_USERNAME = ${DB_ADMIN_USERNAME}"
echo "DB_ADMIN_PASSWORD = ${DB_ADMIN_PASSWORD}"
This application does not need to know about Vault.
Start a Vault Agent
Start a Vault Agent instance that connects to the Vault server running at
VAULT_ADDR
.
From your current terminal, print the working directory and Vault environment variables.
$ pwd && printenv | grep VAULT_ /Users/yourradusername/learn-vault-agent-envconsul VAULT_ADDR=https://vault-cluster-public-vault-XXXXXXXXX.XXXXXXXXX.XX.hashicorp.cloud:8200 VAULT_NAMESPACE=admin VAULT_TOKEN=hvs.S3ASAM3...STR33T
Open a new terminal to run Vault Agent.
Change to the working directory printed in the previous step.
Example:
$ cd /Users/yourradusername/learn-vault-agent-envconsul
Set the
VAULT_ADDR
environment variable.$ export VAULT_ADDR=<Public_Cluster_URL>
Set the
VAULT_NAMESPACE
environment variable.$ export VAULT_NAMESPACE=admin
Set the
VAULT_TOKEN
environment variable from the HCP Portal.$ export VAULT_TOKEN=<admin_token>
Log into Vault. This step stores the authentication token in the expected directory for the Vault Agent.
$ vault login $VAULT_TOKEN
Start a Vault Agent.
$ vault agent -config=agent-config.hcl -log-level=error
Example output:
==> Vault Agent started! Log data will stream in below: ==> Vault Agent configuration: Api Address 1: http://bufconn Cgo: disabled Log Level: error Version: Vault v1.14.0-rc1, built 2023-06-06T18:12:53Z Version Sha: 1327dc6728853611f62708e28dbac3ce6cabad1a ==== KV DEMO APP ==== API_KEYS_KEY1 = 4984h33sd API_KEYS_KEY2 = o48r3eqf CREDS_USER = tester1 CREDS_PASSWORD = my-long-password DB_ADMIN_USERNAME = admin DB_ADMIN_PASSWORD = o3ir23ir
Vault Agent stops and displays the secret values read from Vault.
Restarts on secrets change
The secret values can change while your application is running. Examine the behavior when the secret values change.
Add
sleep 100
in thekv-demo.sh
to mimic a long running application.kv-demo.sh
$ printf '\nsleep 100' >> kv-demo.sh && cat kv-demo.sh #!/bin/sh echo echo "==== DEMO APP ====" echo "API_KEYS_KEY1 = ${API_KEYS_KEY1}" echo "API_KEYS_KEY2 = ${API_KEYS_KEY2}" echo "CREDS_USER = ${CREDS_USER}" echo "CREDS_PASSWORD = ${CREDS_PASSWORD}" echo "DB_ADMIN_USERNAME = ${DB_ADMIN_USERNAME}" echo "DB_ADMIN_PASSWORD = ${DB_ADMIN_PASSWORD}" sleep 100
Edit the
agent-config.sh
file and set thestatic_secret_render_interval
to 10 seconds. Thestatic_secret_render_interval
setting makes Vault Agent pull the secrets every 10 seconds.$ sed -i '' 's/static_secret_render_interval = "5m"/static_secret_render_interval = "10s"/' agent-config.hcl \ && cat agent-config.hcl
Example output:
agent-config.hcl
auto_auth { method { type = "token_file" config { token_file_path = "/Users/username/.vault-token" } } } template_config { static_secret_render_interval = "10s" exit_on_retry_failure = true } vault { address = "$VAULT_ADDR" } ...snip...
Start the Vault Agent again.
$ vault agent -config=agent-config.hcl -log-level=error ==> Vault Agent started! Log data will stream in below: ==> Vault Agent configuration: Api Address 1: http://bufconn Cgo: disabled Log Level: error Version: Vault v1.14.0-rc1, built 2023-06-06T18:12:53Z Version Sha: 1327dc6728853611f62708e28dbac3ce6cabad1a ==== KV DEMO APP ==== API_KEYS_KEY1 = 4984h33sd API_KEYS_KEY2 = o48r3eqf CREDS_USER = tester1 CREDS_PASSWORD = my-long-password DB_ADMIN_USERNAME = admin DB_ADMIN_PASSWORD = o3ir23ir
Switch back to the terminal where you connect to Vault.
Update the secret values at
web-team/data/api-keys
.$ vault kv put web-team/api-keys \ key1="new-key-value-1" \ key2="new-key-value-2"
Return to the terminal where Vault Agent is running and watch the output.
==> Vault Agent started! Log data will stream in below: ==> Vault Agent configuration: Api Address 1: http://bufconn Cgo: disabled Log Level: error Version: Vault v1.14.0-rc1, built 2023-06-06T18:12:53Z Version Sha: 1327dc6728853611f62708e28dbac3ce6cabad1a ==== KV DEMO APP ==== API_KEYS_KEY1 = 4984h33sd API_KEYS_KEY2 = o48r3eqf CREDS_USER = tester1 CREDS_PASSWORD = my-long-password DB_ADMIN_USERNAME = admin DB_ADMIN_PASSWORD = o3ir23ir ==== KV DEMO APP ==== API_KEYS_KEY1 = new-key-value-1 API_KEYS_KEY2 = new-key-value-2 CREDS_USER = tester1 CREDS_PASSWORD = my-long-password DB_ADMIN_USERNAME = admin DB_ADMIN_PASSWORD = o3ir23ir
Press Ctrl + C to stop the running Vault Agent.
Work with dynamic secrets
The generate-config subcommand supports kv-v1 and kv-v2 secrets
engines; however, the Vault Agent's process supervisor
mode supports dynamic
secrets that Envconsul supports today. The difference is that you have to
manually define the env_template
blocks.
The
setup-secrets.sh
script configured the PKI secrets engine.$ vault secrets list
Example output:
Path Type Accessor Description ---- ---- -------- ----------- cubbyhole/ cubbyhole cubbyhole_595f8b0f per-token private secret storage dev-app/ kv kv_f16c3aac n/a identity/ identity identity_cd1c4e99 identity store pki/ pki pki_ca9e7398 n/a pki_int/ pki pki_331d3c63 n/a secret/ kv kv_1bb31236 key/value secret storage sys/ system system_a6bb36e2 system endpoints used for control, policy and debugging web-team/ kv kv_c5dd184d n/a
Request a certificate.
$ vault write -field=certificate pki_int/issue/example-dot-com \ common_name="test.example.com" ttl="24h"
Example output:
-----BEGIN CERTIFICATE----- MIIDZjCCAk6gAwIBAgIUZl818nTOMzvFdlhcnE5YYZ/WC5UwDQYJKoZIhvcNAQEL BQAwLTErMCkGA1UEAxMiZXhhbXBsZS5jb20gSW50ZXJtZWRpYXRlIEF1dGhvcml0 eTAeFw0yMzA2MTMyMzE3MjdaFw0yMzA2MTQyMzE3NTdaMBsxGTAXBgNVBAMTEHRl c3QuZXhhbXBsZS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDV IrsKREu2+lQbNbdYgYzMYAidevxVV3K4kWe9MYQWW4xLus/DtuBZXwqPkPsl3UGs NRjv4gE9hvb3lGAcs5Ze4f27sK/FvXgcOXy/p1wJ8JQV+rMHq/cg3dcEvYvY/f2c IMeR0vMJb58POjM4iWSgvYfvQtO+4r0hPydGGafubhorRvlvEMfiNfghDG3wSoeJ Z0OmwQenBY6ASsJZ32qJbuiVszSCHIgOP5s6GfF6KKkKj3ojr/aH++ff+ckDzFbW /uz0IGV2hhdGEnnSjlWn0uaPR6xFGqprp1csAAZtdY50q6PYZ7Ev8Oe3zFPNM4sj uXETsXcjRUUusC1vGxYrAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgOoMB0GA1Ud JQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAdBgNVHQ4EFgQUcu+At+HNbaGOD8CV voKRxf9Q4JMwHwYDVR0jBBgwFoAUOiYGxwIjDoTz389uuIGb9VWmEKUwGwYDVR0R BBQwEoIQdGVzdC5leGFtcGxlLmNvbTANBgkqhkiG9w0BAQsFAAOCAQEAQbHfB/gc pG9n2Pb9wfw9wek+Ko1abmf2rgkqD44z6lvS+kpRi1SIwY2oUEhXvILdl098M4D7 zRhWfCYyBuN+jVuGpZjqVlc3nS976JzBil3MzlfvUSbscoao0oh6unaBfYlGlgR+ STNjjeQDx+bVWL3os8u3WqY+XlyVVHnBAQME+9Hgiamzv0gKE2vEb93KZ33rozjf wSTNzEfvQOHTAjkMZ2rwAu2prhLq4e8jUM8BA4QxFiw9R+vx1Dnu3Kz3W0eLhQ2W XG9YH57A234vOc/dcW6teNq8QzYLuqzA4oNKpBWiWT7uLt2hP0L3OtUliqDZAzFZ pxt9m7E2V9qQcQ== -----END CERTIFICATE-----
Open the
pki-agent-config.hcl
to examine theenv_template
block it defines.$ cat pki-agent-config.hcl env_template "CERT" { contents = "{{ with pkiCert \"pki_int/issue/example-dot-com\" \"common_name=test.example.com\" \"ttl=24h\"}}{{ .Data.Cert }}{{ end }}" error_on_missing_key = true } exec { command = ["./pki-demo.sh"] restart_on_secret_changes = "always" restart_stop_signal = "SIGTERM" }
Open the
pki-demo.sh
file to review.$ cat pki-demo.sh #!/bin/sh echo echo "==== CERT DEMO APP ====" echo "CERT = ${CERT}"
This mock application uses a certificate stored in the CERT environment variable. Similar to the
kv-demo.sh
, this application does not need to know anything about Vault.Switch back to the terminal where you started the Vault Agent earlier.
Run a Vault Agent with the
env_template
definition.$ vault agent -config=agent-config.hcl \ -config=pki-agent-config.hcl \ -log-level=error
Example output:
==> Vault Agent started! Log data will stream in below: ==> Vault Agent configuration: Api Address 1: http://bufconn Cgo: disabled Log Level: error Version: Vault v1.14.0-rc1, built 2023-06-06T18:12:53Z Version Sha: 1327dc6728853611f62708e28dbac3ce6cabad1a ==== CERT DEMO APP ==== CERT = -----BEGIN CERTIFICATE----- MIIDZjCCAk6gAwIBAgIUdTJEGBZtcnxlo8f0mMLVzKvXbtgwDQYJKoZIhvcNAQEL BQAwLTErMCkGA1UEAxMiZXhhbXBsZS5jb20gSW50ZXJtZWRpYXRlIEF1dGhvcml0 eTAeFw0yMzA2MTMyMzQ3MDNaFw0yMzA2MTQyMzQ3MzNaMBsxGTAXBgNVBAMTEHRl c3QuZXhhbXBsZS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC3 GXTT5UlMnH5OJKZbx7n9DJs4K4lgrKdirpD4mVL7O4ba3KUA/mam9M6LAqn5YzB6 UX6y1+zIyNaakqltxLVxiyLWodqix3fjN9g6jzis5JAQM3UIgVYPBpg5ptKzpVzH ASvuDonnPflBYQG609rD56RK8ycyhq0bcUVCa6pEQI3EtJaqj2woOTVWyauo+r7g jrwr49BL0ye0T1IoTSs3l0E5Pk6ZcTDEbK3rNZ+YWSzwcg6bAL1U8kOW3B3jINwI GYXPKVVLM2v6TTYpeYB/fZ6FeQeDasVU3iy3N16HdIHbI2vAQqUkTZcv/LM1ALgf DbEYOUvAcdNCZKifhDTdAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgOoMB0GA1Ud JQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAdBgNVHQ4EFgQUBIsJvc4reYPkq1Vy tuf3NMAHvD0wHwYDVR0jBBgwFoAUOiYGxwIjDoTz389uuIGb9VWmEKUwGwYDVR0R BBQwEoIQdGVzdC5leGFtcGxlLmNvbTANBgkqhkiG9w0BAQsFAAOCAQEAZyN5dSKo 9E8G6kaArX6Jfq8CGdX1jhaqFH8SljYZ5/WngzJwuQNkvCU42TpAbvHsBfen93T/ VfGgPPh6GohwhvLbVQjTLzGBlFTq+oQJs0ntQxjL/CMEfqg0aLH1OnZuVULc9J6z LaOHhsV4EHx7c5iSuwwun7NOJgysadboS8EE4BOqkKehaEz/ZXVVM5sJhRpi8KfR I9EA+FNG6URKllW72OM2kbg9HVQsxsIPdG+o/JDsCz48yMwxJDyLAxMH/YtOVrbF j55N7ooFPw/h3BhMP+0zzDrQrl9GPYRCe//10zmTA+4/q2vQ3nPMVeInEa+QdTBi hxz5k4LuDjhuoA== -----END CERTIFICATE-----
Similar to what you observed in the restarts on secrets change section, if the certificate expires and the application is still running, Vault Agent can fetch a new certificate.
Clean up
You can stop the Vault dev server by pressing Ctrl+C where the server is running. Or, execute the following command.
$ pgrep -f vault | xargs kill
Unset the
VAULT_ADDR
environment variable.$ unset VAULT_ADDR
Summary
You learned the use of the generate-config subcommand and the process supervisor mode to take advantage of secrets presented as environment variables.
Vault Agent can handle the authentication and secrets retrieval so that your application can remain Vault unaware. This reduces the barrier to adopting Vault and keep your applications secure.
There are several tutorials demonstrates the use of Vault Agent. See the available Vault Agent tutorials.