Seal/Unseal

When you start a Vault server, it starts in a sealed state. In this state, Vault can access the physical storage, but it cannot decrypt any of the data on it.

Unsealing is the process of obtaining the plaintext root key that is necessary to read the decryption key. Prior to unsealing, the only possible Vault operations are to unseal the Vault and check the status of the server.

Why?

Vault encrypts the data using an encryption key (in the keyring) and stores them in its storage backend. To protect this encryption key, Vault encrypts it using another encryption key known as the root key and stores it with the data.

To decrypt the data, Vault needs the root key so that it can decrypt the encryption key. Unsealing is the process of getting access to this root key. Vault encrypts the root key using the unseal key, and stores it alongside all other Vault data.

To summarize, Vault encrypts most data using the encryption key in the keyring. To get the keyring, Vault uses the root key to decrypt it. The root key itself requires the unseal key to decrypt it.

Shamir seals

Instead of distributing the unseal key to an operator as a single key, the default Vault configuration uses an algorithm known as Shamir's Secret Sharing to split the key into shares.

Vault requires a certain threshold of shares to reconstruct the unseal key. Vault operators add shares one at a time in any order until Vault has enough shares to reconstruct the key. Then, Vault uses the unseal key to decrypt the root key. This is the Vault unseal process.

Unsealing

You can run the unseal process using the vault operator unseal command or the API. The unseal process is stateful. You can enter each key using multiple mechanisms from multiple client machines. The process allows each share of the root key to reside on a distinct client machine for better security.

When you use the Shamir seal with multiple nodes, you must unseal each node with the required threshold of shares. Vault does not distribute partially unsealed nodes across the cluster.

Once you unseal a Vault node, it remains unsealed until one of the following happens:

1. You reseal it using the API.
2. You restart the server.
3. Vault's storage layer encounters an unrecoverable error.

Unsealing nodes can make automating a Vault installation difficult. Automated tools can easily install, configure, and start Vault, but unsealing it using Shamir seals is a manual process. For most users, auto unseal provides a better experience.

Sealing

There is also an API to seal the Vault. The API deletes the root key from memory and requires another unseal process to restore it. Sealing the Vault requires only a single operator with root privileges.

If there is a detected intrusion, you can seal the Vault data quickly to try to minimize damages. Users cannot access the data from a sealed Vault without access to the root key shares.

Auto unseal

Auto unseal aids in reducing the operational complexity of keeping the unseal key secure. This feature delegates the responsibility of securing the unseal key from users to a trusted device or service. At startup, Vault connects to the trusted device or service and prompts it to decrypt the root key from storage.

There are certain Vault operations besides unsealing that require a quorum of users to perform, for example, generating a root token. When you use a Shamir seal, you must provide the unseal keys to authorize these operations. When you use auto unseal, these operations require recovery keys instead.

Just as the Shamir seal initialization process yields unseal keys, initializing operations with auto unseal yields recovery keys.

It is still possible to seal a Vault node using the API. When you use the API, Vault remains sealed until you restart it or use the unseal API. Auto unseal requires the recovery key fragments instead of the unseal key fragments that the Shamir seal provides. The process remains the same.

For a list of examples and supported providers, refer to the seal documentation.

When you enable DR replication in Vault Enterprise, performance standby nodes on the DR cluster seal themselves, so you must restart them to unseal them.

Recovery key

When you initialize Vault using an HSM or KMS, Vault returns recovery keys to the operator rather than unseal keys. Vault generates recovery keys from an internal recovery key that is split via Shamir's Secret Sharing, similar to Vault's treatment of unseal keys when you run it without an HSM or KMS.

When you perform an operation such as generate-root that uses recovery keys, Vault automatically selects the recovery keys for this purpose, rather than the barrier unseal keys.

Initialization

When you initialize Vault, it performs the split according to the following CLI flags and their API equivalents in the /sys/init endpoint:

• recovery-shares: The number of shares into which to split the recovery key. This value is equivalent to the recovery_shares value in the API endpoint.
• recovery-threshold: The threshold of shares required to reconstruct the recovery key. This value is equivalent to the recovery_threshold value in the API endpoint.
• recovery-pgp-keys: The PGP keys to use to encrypt the returned recovery key shares. This value is equivalent to the recovery_pgp_keys value in the API endpoint, although as with pgp_keys the object in the API endpoint is an array, not a string.

If you don't set these options, Vault does not initialize; therefore, it will not generate the keys. Refer to Configuration for more details.

Rekeying

Unseal key

You can rekey Vault's unseal keys using a vault operator rekey operation from the CLI or the matching API calls. Vault authorizes the rekey operation when you meet the threshold of recovery keys. After rekeying, Vault encrypts the new barrier key with the configured HSM or KMS, and stores it in its storage backend.

Recovery key

You can rekey the recovery key to change the number of shares/threshold or to target different key holders using different PGP keys. When you use the Vault CLI, it performs the rekey operation by using the -target=recovery flag to vault operator rekey.

Using the API, you perform the rekey operation with the same parameters as the normal /sys/rekey endpoint; however, the API prefix for the rekey operation is at /sys/rekey-recovery-key rather than /sys/rekey.

Seal migration

You cannot perform the seal migration process without downtime. Due to the technical underpinnings of the seal implementations, the process requires that you briefly take the whole cluster down. While experiencing some downtime may be unavoidable, we believe that switching seals is a rare event and that the inconvenience of the downtime is an acceptable trade-off.

You should make sure to create a backup before you start the seal migration process, in case something goes wrong.

The seal migration operation requires both the old and new seals to be available during the migration. For example, if you migrate from auto unseal to Shamir seals, the service backing the auto unseal must be available during the migration.

Migration post Vault 1.16.0 via Seal HA for Auto Seals (Enterprise)

You can perform an online migration between most auto-unseal types using seal high availability (seal HA) without any downtime. Seal HA does not support migrating Shamir seals.

1. Edit the Vault configuration, and add the new target seal configuration.
2. Send the Vault process the SIGHUP signal to trigger a configuration reload.
3. Monitor the sys/sealwrap/rewrap endpoints, to confirm that rewrap is running, and/or sys/seal-backend-status, endpoints, waiting for fully_wrapped to be true, indicating all seal wrapped values are now wrapped by the new seal. The logs also contain information about the rewrap progress.
4. Edit the Vault configuration, and remove the old seal configuration.
5. Send the Vault process the SIGHUP signal, again allowing rewrapping to complete.

Migration post Vault 1.5.1

The following steps are common for seal migrations between any supported kinds and for any storage backend.

1. Take a standby node down and update the seal configuration.

   • If the migration is from Shamir seal to auto seal, add the new auto seal block to the configuration.
   • If the migration is from auto seal to Shamir seal, add disabled = "true" to the old seal block.
   • If the migration is from auto seal to another auto seal, add disabled = "true" to the old seal block and add the new auto seal block.

   Now, bring the standby node back up and run the unseal command on each key, by supplying the -migrate flag.

   • Supply Shamir unseal keys if the old seal was Shamir, which is migrated as the recovery keys for the auto seal.
   • Supply recovery keys if the old seal is one of auto seals, which is migrated as the recovery keys of the new auto seal, or as Shamir unseal keys if the new seal is Shamir.

2. Perform step 1 for all the standby nodes, one at a time. You must bring back each downed standby node before moving on to the other standby nodes, specifically when you use integrated storage, because it helps to retain the quorum.

3. Step down the active node. One of the standby nodes will become the new active node. When you use integrated storage, ensure that Vault reaches a quorum and elects a leader.

4. The new active node performs the migration. Monitor the server log in the active node to witness the completion of the seal migration process. Wait for a little while for the migration information to replicate to all the nodes, if you use integrated storage. In Vault Enterprise, switching an auto seal implies that the seal wrapped storage entries get rewrapped. Monitor the log and wait until this process is complete. Look for seal re-wrap completed in the log.

1. Seal migration is now completed. Take down the old active node, update its configuration to use the new seal blocks (completely unaware of the old seal type) , and bring it back up. It will be auto-unsealed if the new seal is one of the auto seals, or will require unseal keys if the new seal is Shamir.

2. You can now update the configuration files of all the nodes to have only the new seal information. You can restart standby nodes right away, and you can restart the active node upon a leadership change.

Migration pre 1.5.1

Migration from Shamir to auto unseal

To migrate from Shamir keys to auto unseal, take your server cluster offline and update the seal with the appropriate seal configuration. Bring your server back up and leave the rest of the nodes offline if you use multi-server mode, then run the unseal process with the -migrate flag and bring the rest of the cluster online.

All unseal commands must specify the -migrate flag. Once you enter the required threshold of unseal keys, Vault migrates the unseal keys to recovery keys.

$ vault operator unseal -migrate

Migration from auto unseal to Shamir

To migrate from auto unseal to Shamir keys, take your server cluster offline and update the seal configuration and add disabled = "true" to the seal block. This flag allows the migration to use this information to decrypt the key, but will not unseal Vault.

When you bring your server back up, run the unseal process with the -migrate flag and use the recovery keys to perform the migration. All unseal commands must specify the -migrate flag. Once you enter the required threshold of recovery keys, Vault migrates the recovery keys that it will use as unseal keys.

Migration from auto unseal to auto unseal

To migrate from auto unseal to a different auto unseal configuration, take your server cluster offline and update the existing seal configuration and add disabled = "true" to the seal block. Then add another seal block to describe the new seal.

When you bring your server back up, run the unseal process with the -migrate flag and use the recovery keys to perform the migration. All unseal commands must specify the -migrate flag. Once you enter the required threshold of recovery keys, Vault keeps the recovery keys and uses them as recovery keys in the new seal.

Migration with integrated storage

Integrated storage uses the Raft protocol, which requires a quorum of servers to be online before the cluster is functional. Therefore, bringing the cluster back up one node at a time with the updated seal configuration does not work in this case.

Follow the same steps for each kind of migration described above with the exception that after you take the cluster offline, you should update the seal configurations of all the nodes appropriately and bring them all back up. When the quorum of nodes are back up, Raft elects a leader and the leader node that performs the migration. The migrated information is replicated to all other cluster peers. When the peers eventually become the leader, migration does not happen again on the peer nodes.

Seal high availability EnterpriseEnterprise

Seal high availability (seal HA) lets you configure more than one auto seal mechanism such that Vault can tolerate the temporary loss of a seal service or device. When you configure seal HA with at least two and no more than three auto seals, Vault can also start up and unseal if one of the configured seals is still available, though Vault remains in a degraded mode in this case. While seals are unavailable, seal wrapping and entropy augmentation can still occur using the remaining seals, and Vault rewraps any values produced while a seal is down with all the seals when they become healthy again.

You should choose two seals that are unlikely to become unavailable at the same time. For example, you may choose KMS keys in two cloud regions, from two different providers, or a mix of HSM, KMS, or transit seals.

When you configure an additional seal or remove one seal at a time and restart Vault, Vault automatically detects that it needs to rewrap CSPs and seal wrapped values, and starts the process. You can monitor seal rewrapping using the logs or the sys/seal-status endpoint. Changes to the seal configuration are not allowed while a rewrap is in progress or if one could not complete successfully.

In addition to its high availability benefits, you can also use seal HA to migrate between two auto seals in a simplified manner. For more information, refer to Migration post Vault 1.16.0 via seal HA for auto seals.

Note that Shamir seals are not auto seals, and you cannot include Shamir seals in a seal HA setup. Auto seals support seal wrap, so the loss of the auto seal does not necessarily leave Vault in a fully available state. However, Shamir seals do not support seal wrap.

Use and configuration

Refer to the configuration section for details on how to configure seal HA.

Seal rewrapping

Whenever you change a seal configuration, Vault must rewrap all CSPs and seal wrapped values to ensure each value has an entry that is encrypted by all configured seals. Vault detects these configuration changes automatically, and triggers a rewrap. Rewraps can take some time, depending on the number of seal wrapped values. While rewrapping is in progress, you cannot make configuration changes to the seals.

You can monitor the progress of the rewrap using the sys/sealwrap/rewrap endpoint.

Limitations and known issues

To limit complexity and increase safety, there are some limitations to the use and configuration of seal HA:

• You must configure Vault for a single seal at the time of initialization, then you can add extra seals.
• You must add or remove seals one at a time.
• You can only use auto seals in HA configurations. You cannot mix Shamir and auto seals.
• You can configure a maximum of three seals.
• As you must wrap seal wrapped values by all configured seals, it is possible that large values may fail to persist because Vault multiplies the size of the entry by the number of seals, causing it to exceed the storage entry size limit. An example would be storing a large document in KVv2 with seal wrapping enabled.
• It is not possible to rotate the data encryption key nor the recovery keys unless all seals are healthy.