Remediate leaked secrets

Once you identify a leaked secret and store it in your secrets manager, you need to plan to remediate the leaked secret.

If you have not already done so, form an incident response team. This team should include technical representatives from the security, operations, and development teams, as well as business and legal representatives.

The remediation process will vary based on:

The type of system the leaked secret gives access to.
The type of data available in that system.
Whether you have to report the incident to a government or industry regulator.

In any scenario, part of your remediation process should include communication with your stakeholders, both internal (such as development teams) and external (such as customers), to ensure they stay informed of any actions they may need to take.

The National Institute of Standards and Technology (NIST) is a helpful resource for defining requirements for incident response. NIST SP 800-61: Computer Security Incident Handling Guide identifies four stages of incident response:

Preparation: Ensure you have the right tools and processes in place to identify, manage, and remediate incidents.
Detection and analysis: Identify the incident, and analyze the impact of the incident.
Containment, eradication, and recovery: Contain the incident, remove the cause of the incident, and recover from the incident.
Post-incident activity: Review the incident, and update your processes to prevent similar incidents in the future.

HashiCorp resources:

External resources:

Remediate leaked secrets

Now that you have set up HCP Vault Radar to identify leaked secrets and HashiCorp Vault to manage your secrets, you can begin to remediate the leaked secret.

Like the incident response plan, the remediation process will vary. One of the biggest factors revolves around the data source of the leaked secret.

Leaked secret in documentation

If you identify a leaked secret in documentation, the first step is to identify what teams, workflows, or applications depend on this secret. A common practice is to document requirements in a configuration management database (CMDB). Similarly, if you are using Terraform to manage your Vault policies, you can cross-reference the policies to determine what teams or applications are using the secret.

If the processes using the secret already retrieve the secret from Vault, you can:

Rotate or change the secret.
Remove the plaintext secret from the documentation.
Update the secret value in Vault.

Any process retrieving the secret will receive the updated secret value from Vault.

Leaked secrets in source code

When you identify a leaked secret in source code, the process is more involved and may require involving other teams.

If the application or workflow built from the source code requires access to this secret, you must update the source code to reference it in a secure manner.

There are several options to retrieve a secret from Vault and you may use one or more methods during the remediation process. Each method has unique advantages and disadvantages. You must weigh the impact of each option against any business or regulatory requirements.

For native integration, you need to refactor your application or service to integrate your application directly with Vault using the secrets engine for the type of secrets you need. This may be your best approach if you maintain consistent development practices across your application teams, such as the use of standardized frameworks like Spring Boot, or retrieve all application configurations from a configuration server.

Use the Vault Secrets Operator for Kubernetes workloads or Vault secret sync to copy the secrets from Vault to your platform (for example, AWS, Azure, or GCP) built-in secrets management solution. Once you can retrieve the secret from the built-in secrets management solution, update the source code to reference the secret as required by your platform.
Use the Vault Agent to write the secret to a secure location such as an environment variable. Then, update the source code to reference the environment variable. This option introduces the least amount of friction, and is a good first step to quickly remove the secret from your source code before refactoring the code to use the Vault Secrets Operator, Vault SDK or API.
Update your application using the Vault SDK, or your programming language's native API/secrets management library. This is the most secure option, but may require significant development effort. Refactoring a legacy application may not be possible. In that scenario, consider the other methods described.

Once you update the data source to retrieve the secret from Vault, you can rotate the secret and securely store it in Vault.

HashiCorp resources:

Next steps

In this section of how to manage leaked secrets, you learned about how to remediate leaked secrets in your organization. Managing leaked secrets is part of the Secure systems pillar.

In the next section, you will learn how to prevent leaked secrets.