Manage monitoring agents secrets

No matter how you deploy and manage your monitoring agents, you will want to securely manage their secrets. Your deployment method depends on the platform where you run your applications and whether you network your services traditionally or through a service mesh.

Most monitoring agents need authentication credentials or tokens to securely connect back to the central monitoring platform. For example, Datadog agents need an API key to associate the agent with your Datadog account. Exposing these secrets directly in Packer templates, Terraform configurations, or in your configuration management tools is insecure.

We recommend that you retrieve the required credentials from a secure secret management solution like HashiCorp Vault during the deployment process.

• If you use Packer, you can directly pull dynamic secrets from Vault when you build the golden image.
• If you use configuration management tools like Ansible, you can integrate it with Vault to securely inject monitoring credentials when you configure the monitoring agent.
• When you deploy your image with Terraform, you can leverage user scripts (startup/cloud-init scripts) to fetch the credentials from Vault.

This secret injection approach keeps credentials out of your codebase even as you adopt “as-code” best practices to configure your monitoring agents. Vault lets you securely store and distribute secrets through strict access controls and auditing capabilities.

HashiCorp resources:

• The Packer vault function lets you read secrets from Vault and use the secrets within your template as user variables.
• The Terraform Vault provider lets you connect to your Vault cluster and manage its resources. You can use the Vault provider to read dynamic secrets and provide them to your user scripts.
• The Vault Agent containers tutorial shows you how to inject secrets into Kubernetes pods. Nomad supports a Vault integration that lets you read secrets from Vault and use it in your Nomad templates. You can use these methods to provide Vault secrets to configure your monitoring agents on the respective container orchestrator.

External resources:

The following resources let you read vault secrets from configuration management tools:

• Ansible supports Vault integration with the hashi_vault lookup plugin.
• The Using HashiCorp's Vault with Chef blog post provides strategies for retrieving secrets from Vault using Chef at runtime, at configuration time, within a Chef resource/provider, and within the application logic itself.
• Puppet supports Vault integration with the vault_lookup module.

Next steps

In this section of Setup monitoring agents, you learned how to manage secrets for your monitoring agents. Manage monitoring agents secrets is part of the Define and automate processes pillar.