Protect against resource exhaustion

1. Enable a file audit device which outputs to /var/log/vault-audit.log (or your desired file location).

   $ vault audit enable file file_path="/var/log/vault-audit.log"
   

2. To enable the audit logging for rate limit quotas, execute the following command.

   $ vault write sys/quotas/config enable_rate_limit_audit_logging=true
   

3. Read the quota configuration to verify.

   $ vault read sys/quotas/config
   
   Key                                   Value
   ---                                   -----
   absolute_rate_limit_exempt_paths      []
   enable_rate_limit_audit_logging       true
   enable_rate_limit_response_headers    false
   rate_limit_exempt_paths               []
   

1. Enable file audit device.

   First, create the HTTP request payload specifying the audit log path to be /var/log/vault-audit.log (or your desired file location).

   $ tee audit-payload.json <<EOF
   {
     "type": "file",
     "options": {
         "file_path": "/var/log/vault-audit.log"
     }
   }
   EOF
   

   Use the sys/audit endpoint to enable file audit log.

   $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
       --request POST \
       --data @audit-payload.json \
       $VAULT_ADDR/v1/sys/audit/file
   

   Set the target file_path to your desired location.

2. Create the HTTP request payload to enable the rate limit quota audit logging.

   $ tee payload.json <<EOF
   {
      "enable_rate_limit_audit_logging": true
   }
   EOF
   

3. Invoke the sys/quotas/config endpoint.

   $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
       --request POST \
       --data @payload.json \
       $VAULT_ADDR/v1/sys/quotas/config
   

4. Read the quota configuration to verify.

   $ curl -s --header "X-Vault-Token: $VAULT_TOKEN" \
       $VAULT_ADDR/v1/sys/quotas/config | jq -r ".data"
   

   Example output:

   {
      "absolute_rate_limit_exempt_paths": [],
      "enable_rate_limit_audit_logging": true,
      "enable_rate_limit_response_headers": false,
      "rate_limit_exempt_paths": []
   }
   

1. Create a rate limit quota named, "global-rate" which limits inbound workload to 1500 requests per second.

   $ vault write sys/quotas/rate-limit/global-rate rate=1500
   Success! Data written to: sys/quotas/rate-limit/global-rate
   

2. Read the global-rate rule to verify its configuration.

   $ vault read sys/quotas/rate-limit/global-rate
   
   Key               Value
   ---               -----
   block_interval    0
   inheritable       true
   interval          1
   name              global-rate
   path              n/a
   rate              1500
   role              n/a
   type              rate-limit
   

   Note

   In absence of path, this quota rule applies to the global level instead of a specific mount or namespace.

3. Create a rate limit quota named, "transit-limit" which limits the access to the Transit secrets engine to be 1000 requests per minute (60 seconds).

   First, enable Transit secrets engine at transit.

   $ vault secrets enable transit
   Success! Enabled the transit secrets engine at: transit/
   

   Now, create a rate limit quota.

   $ vault write sys/quotas/rate-limit/transit-limit \
      path="transit" \
      rate=1000 \
      interval=60
   

   Output:

   Success! Data written to: sys/quotas/rate-limit/transit-limit
   

4. Read the transit-limit rule to verify its configuration.

   $ vault read sys/quotas/rate-limit/transit-limit
   

   Output:

   Key               Value
   ---               -----
   block_interval    0
   inheritable       true
   interval          60
   name              transit-limit
   path              transit/
   rate              1000
   role              n/a
   type              rate-limit
   

   You configured the rate limit to 1000 requests per minute (60 seconds).

Path granularity

You can set the path to be deeper than the mount point (in this example, transit/).

1. Create a rate limit quota named, "transit-order" to limit the data encryption requests using orders key to be 500 per second.

   First, create an encryption key named, "orders".

   $ vault write -f transit/keys/orders
   
   Key                       Value
   ---                       -----
   allow_plaintext_backup    false
   auto_rotate_period        0s
   deletion_allowed          false
   derived                   false
   exportable                false
   imported_key              false
   keys                      map[1:1695147293]
   latest_version            1
   min_available_version     0
   min_decryption_version    1
   min_encryption_version    0
   name                      orders
   supports_decryption       true
   supports_derivation       true
   supports_encryption       true
   supports_signing          false
   type                      aes256-gcm96
   

   Now, create the "transit-order" rate limit quota.

   $ vault write sys/quotas/rate-limit/transit-order \
      path="transit/encrypt/orders" \
      rate=500
   

   Output:

   Success! Data written to: sys/quotas/rate-limit/transit-order
   

2. Verify the rate limit quota configuration.

   $ vault read sys/quotas/rate-limit/transit-order
   

   Output:

   Key               Value
   ---               -----
   block_interval    0
   inheritable       true
   interval          1
   name              transit-order
   path              transit/encrypt/orders
   rate              500
   role              n/a
   type              rate-limit
   

Create a rate limit quota named, "global-rate" which limits inbound workload to 1500 requests per second.

1. Invoke the sys/quotas/rate-limit endpoint.

   $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
       --request POST \
       --data '{ "rate": 1500 }' \
       $VAULT_ADDR/v1/sys/quotas/rate-limit/global-rate
   

2. Read the newly created global-rate quota rule.

   $ curl -s --header "X-Vault-Token: $VAULT_TOKEN" \
       $VAULT_ADDR/v1/sys/quotas/rate-limit/global-rate | jq -r ".data"
   

   Output:

   {
      "block_interval": 0,
      "inheritable": true,
      "interval": 1,
      "name": "global-rate",
      "path": "",
      "rate": 1500,
      "role": "",
      "type": "rate-limit"
   }
   

   Note

   In absence of path, this quota rule applies to the root namespace instead of a specific mount or namespace.

3. Create a rate limit quota named, "transit-limit" which limits the access to the Transit secrets engine to be 1000 requests per minute (60 seconds).

   First, enable Transit secrets engine at transit.

   $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
       --request POST \
       --data '{"type":"transit"}' \
       $VAULT_ADDR/v1/sys/mounts/transit
   

   Now, create a "transit-limit" rate limit quota.

   $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
      --request POST \
      --data '{"path": "transit", "rate": 1000, "interval": 60 }' \
      $VAULT_ADDR/v1/sys/quotas/rate-limit/transit-limit
   

4. Read the transit-limit rule to verify its configuration.

   $ curl -s --header "X-Vault-Token: $VAULT_TOKEN" \
      $VAULT_ADDR/v1/sys/quotas/rate-limit/transit-limit | jq -r ".data"
   

   Output:

   {
      "block_interval": 0,
      "inheritable": true,
      "interval": 60,
      "name": "transit-limit",
      "path": "transit/",
      "rate": 1000,
      "role": "",
      "type": "rate-limit"
   }
   

   You configured the rate limit to 1000 requests per minute (60 seconds).

Path granularity

You can set the path to be deeper than the mount point (in this example, transit/).

1. Create a rate limit quota named, "transit-order" to limit the data encryption requests using orders key to be 500 per second.

   First, create an encryption key named, "orders".

   $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
       --request POST \
       $VAULT_ADDR/v1/transit/keys/orders
   

   Now, create the "transit-order" rate limit quota.

   $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
      --request POST \
      --data '{ "path": "transit/encrypt/orders", "rate": 500 }' \
      $VAULT_ADDR/v1/sys/quotas/rate-limit/transit-order
   

2. Verify the rate limit quota configuration.

   $ curl -s --header "X-Vault-Token: $VAULT_TOKEN" \
      $VAULT_ADDR/v1/sys/quotas/rate-limit/transit-order | jq -r ".data"
   

   Output:

   {
      "block_interval": 0,
      "inheritable": true,
      "interval": 1,
      "name": "transit-order",
      "path": "transit/encrypt/orders",
      "rate": 500,
      "role": "",
      "type": "rate-limit"
   }
   

You can use HashiCorp Terraform to set the rate limit quotas.

1. Create a file named, main.tf with the following content.

   # Use Vault provider
   provider vault {}
   
   # Create "global-rate" which limits inbound workload to 1500 requests per second
   resource "vault_quota_rate_limit" "global" {
      name = "global-rate"
      path = ""
      rate = 1500
   }
   
   # Create "transit-limit" which limits the access to the Transit secrets engine to be 1000 requests per minute (60 seconds)
   resource "vault_quota_rate_limit" "transit-limit" {
      name = "transit-limit"
      path = "transit/"
      rate = 1000
      interval = 60
   
      depends_on = [ vault_mount.transit ]
   }
   
   # Path granularity: Create a rate limit quota, "transit-order" to limit the data encryption requests using orders key to be 500 per second
   resource "vault_quota_rate_limit" "transit-order" {
      name = "transit-order"
      path = "transit/encrypt/orders"
      rate = 500
   
   depends_on = [ vault_mount.transit, vault_transit_secret_backend_key.key ]
   }
   
   # Enable transit secrets engine & create a test key
   resource "vault_mount" "transit" {
      path                      = "transit"
      type                      = "transit"
      description               = "Test resource quota"
   }
   
   resource "vault_transit_secret_backend_key" "key" {
      backend = vault_mount.transit.path
      name    = "orders"
   }
   

   The main.tf performs the following operations:

   • Create a rate limit quota named, "global-rate" which limits inbound workload to 1500 requests per second (line 5 - 9)

   • Create a rate limit quota named, "transit-limit" which limits the access to the Transit secrets engine to be 1000 requests per minute (line 12 - 19)

   • Create a rate limit quota named, "transit-order" to limit the data encryption requests using orders key to be 500 per second (line 22 - 28)

   • Enable transit secrets engine for testing (line 31 - 35)

   • Create an encryption key named, orders for testing (line 37 - 40)

2. Initialize Terraform.

   $ terraform init
   
   Initializing the backend...
   
   Initializing provider plugins...
   ...snip...
   
   Terraform has been successfully initialized!
   

3. After you run terraform init, you can verify that it will create the resources with terraform plan.

   $ terraform plan
   
   An execution plan has been generated and is shown below.
   Resource actions are indicated with the following symbols:
   ...snip...
   
   Plan: 5 to add, 0 to change, 0 to destroy.
   

   You should note resources listed in the output.

4. Deploy the resources with terraform apply.

   $ terraform apply -auto-approve
   
   Terraform used the selected providers to generate the following execution plan.
   ...snip...
   Apply complete! Resources: 5 added, 0 changed, 0 destroyed.
   

To learn more about Terraform, visit the Terraform tutorials site.

1. Create a namespace, "us-west".

   $ vault namespace create us-west
   
   Key                Value
   ---                -----
   custom_metadata    map[]
   id                 AdrqM
   path               us-west/
   

2. Enable database secrets engine at postgres in the us-west namespace.

   $ vault secrets enable -ns=us-west -path=postgres database
   Success! Enabled the database secrets engine at: postgres/
   

3. Create a lease count quota named, "db-creds" which limits the incoming requests for a new set of DB credentials to 100 concurrent, valid leases maximum.

   $ vault write sys/quotas/lease-count/db-creds \
      max_leases=100 \
      path="us-west/postgres"
   

   Output:

   Success! Enabled the database secrets engine at: postgres/
   

   Note

   Similar to rate limit quotas, the quota rules apply globally in the absence of path. In the example, you created a quota rule against the us-west/postgres path. If a global quota rule exists on the root namespace, the quota rule defined on a specific path takes precedence.

4. Verify the configuration.

   $ vault read sys/quotas/lease-count/db-creds
   
   Key            Value
   ---            -----
   counter        0
   inheritable    false
   max_leases     100
   name           db-creds
   path           us-west/postgres/
   role           n/a
   type           lease-count
   

Lease count on a role

You can set lease count on the role if the target auth method has the concept of roles.

1. Create a role named "webapp" for approle auth method.

   First, enable the approle auth method.

   $ vault auth enable approle
   Success! Enabled approle auth method at: approle/
   

   Create "webapp" role.

   $ vault write auth/approle/role/webapp \
      token_policies="webapp-policy" \
      token_ttl=4h \
      token_max_ttl=12h
   

   Output:

   Success! Data written to: auth/approle/role/webapp
   

2. Create a lease count quota named "webapp-tokens" which limits the creation of token for the webapp role to maximum of 100.

   $ vault write sys/quotas/lease-count/webapp-tokens \
      max_leases=100 \
      path="auth/approle" \
      role="webapp"
   

   Output:

   Success! Data written to: sys/quotas/lease-count/webapp-tokens
   

   If you want to set it for the approle auth method enable in the us-west namespace, the path should be us-west/auth/approle.

3. Verify the configuration.

   $ vault read sys/quotas/lease-count/webapp-tokens
   

   Output:

   Key            Value
   ---            -----
   counter        0
   inheritable    true
   max_leases     100
   name           webapp-tokens
   path           auth/approle/
   role           webapp
   type           lease-count
   

1. Create a namespace, "us-west".

   $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
        --request POST \
        $VAULT_ADDR/v1/sys/namespaces/us-west
   

2. To learn about this feature, enable database secrets engine at "postgres" in the us-west namespace.

   $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
      --header "X-Vault-Namespace: us-west" \
      --request POST \
      --data '{"type":"database"}' \
      $VAULT_ADDR/v1/sys/mounts/postgres
   

3. Create a lease count quota named, "db-creds" which limits the incoming requests for a new set of DB credentials to 100 concurrent, valid leases maximum.

   First, create the HTTP request payload defining the parameters for db-creds quota rule.

   $ tee payload.json <<EOF
   {
      "max_leases": 100,
      "path": "us-west/postgres"
   }
   EOF
   

   Now, invoke the sys/quotas/lease-count endpoint to create the quota rule.

   $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
      --request POST \
      --data @payload.json \
      $VAULT_ADDR/v1/sys/quotas/lease-count/db-creds
   

   Note

   Similar to rate limit quotas, the quota rules apply globally in the absence of path. In the example, you created a quota rule against the us-west/postgres path. If a global quota rule exists on the root namespace, the quota rule defined on a specific path takes precedence.

4. Verify the configuration.

   $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
      $VAULT_ADDR/v1/sys/quotas/lease-count/db-creds | jq -r ".data"
   

   Output:

   {
      "counter": 0,
      "max_leases": 100,
      "name": "db-creds",
      "path": "us-west/postgres/",
      "role": "",
      "type": "lease-count"
   }
   

Lease count on a role

If you are running Vault 1.12 or later, you can set lease count on the role if the target auth method has the concept of roles.

1. To demonstrate this feature, you are going to use the approle auth method. Enable the approle auth method.

   $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
      --request POST \
      --data '{"type": "approle"}' \
      $VAULT_ADDR/v1/sys/auth/approle
   

2. Create a role named "webapp" for approle auth method.

   First, create the API request payload.

   $ tee payload.json <<EOF
   {
      "token_policies": "webapp-policy",
      "token_ttl": "4h",
      "token_max_ttl": "12h"
   }
   EOF
   

   Now, create a "webapp" role.

   $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
      --request POST \
      --data @payload.json \
      $VAULT_ADDR/v1/auth/approle/role/webapp
   

3. Create a lease count quota named "webapp-tokens" which limits the creation of token for the webapp role to maximum of 100.

   First, create the request payload.

   $ tee payload_lcq.json <<EOF
   {
      "max_leases": 100,
      "path": "auth/approle",
      "role": "webapp"
   }
   EOF
   

   If you want to set it for the approle auth method enabled in the us-west namespace, the path should be us-west/auth/approle.

   Now, invoke the sys/quotas/lease-count endpoint to create the quota rule.

   $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
      --request POST \
      --data @payload_lcq.json \
      $VAULT_ADDR/v1/sys/quotas/lease-count/webapp-tokens
   

4. Verify the configuration.

   $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
      $VAULT_ADDR/v1/sys/quotas/lease-count/webapp-tokens | jq -r ".data"
   

   Output:

   {
      "counter": 0,
      "max_leases": 100,
      "name": "webapp-tokens",
      "path": "auth/approle/",
      "role": "webapp",
      "type": "lease-count"
   }
   

You can use Terraform to set lease count quotas.

1. Create a main.tf file with the following content.

   provider vault {}
   
   # Create a "us-west" namespace
   resource "vault_namespace" "us-west" {
      path = "us-west"
   }
   
   # Create a "db-creds" lease count quota
   resource "vault_quota_lease_count" "db-creds" {
      name = "db-creds"
      path = "us-west/postgres"
      max_leases = 100
   
      depends_on = [ vault_mount.database ]
   }
   
   # Create a "webapp-tokens" lease count quotas
   resource "vault_quota_lease_count" "webapp-tokens" {
      name = "webapp-tokens"
      path = "auth/approle"
      role = "webapp"
      max_leases = 100
   
      depends_on = [ vault_approle_auth_backend_role.webapp ]
   }
   
   # Create a "postgres" mount
   resource "vault_mount" "database" {
      path = "postgres"
      type = "database"
      namespace = "us-west"
      description = "This is a test mount"
   }
   
   # Enable approle auth method
   resource "vault_auth_backend" "approle" {
      type = "approle"
   }
   
   # Create a role named, "webapp"
   resource "vault_approle_auth_backend_role" "webapp" {
      backend = vault_auth_backend.approle.path
      role_name = "webapp"
      token_ttl = 14400
      token_max_ttl = 43200
      token_policies = ["default", "webapp-policy"]
   
      depends_on = [ vault_auth_backend.approle ]
   }
   

   The main.tf performs the following operations:

   • Create a lease count quota named, "db-creds" which limits the incoming requests for a new set of DB credentials to 100 concurrent, valid leases maximum (line 9 - 15)
   • Create a lease count quota named "webapp-tokens" which limits the creation of token for the webapp role to maximum of 100 (line 18 - 25)
   • Create a namespace named, "us-west" (line 4 - 6)
   • Create a "postgres" mount at postgres/ under us-west namespace (line 28 - 33)
   • Enable approle auth method (line 36 - 38)
   • Create an approle role named, "webapp" (line 41 - 49)

2. Initialize Terraform.

   $ terraform init
   
   Initializing the backend...
   
   Initializing provider plugins...
   ...snip...
   
   Terraform has been successfully initialized!
   

3. After you run terraform init, you can verify that it will create the resources with terraform plan.

   $ terraform plan
   
   An execution plan has been generated and is shown below.
   Resource actions are indicated with the following symbols:
   ...snip...
   
   Plan: 6 to add, 0 to change, 0 to destroy.
   

   You should note resources listed in the output.

4. Deploy the resources with terraform apply.

   $ terraform apply -auto-approve
   
   Terraform used the selected providers to generate the following execution plan.
   ...snip...
   Apply complete! Resources: 6 added, 0 changed, 0 destroyed.
   

1. Read the current settings.

   $ vault read sys/quotas/lease-count/default
   
   Key            Value
   ---            -----
   counter        0
   inheritable    true
   max_leases     300000
   name           default
   path           n/a
   role           n/a
   type           lease-count
   

2. Use the sys/quotas/lease-count/default endpoint to adjust the default lease count quota.

   $ vault write sys/quotas/lease-count/default \
      max_leases=10000
   

   Output:

   Success! Data written to: sys/quotas/lease-count/default
   

   You can read the default settings again to verify the changes.

1. Read the current settings.

   $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
      $VAULT_ADDR/v1/sys/quotas/lease-count/default | jq -r ".data"
   

   Output:

   {
      "counter": 0,
      "inheritable": true,
      "max_leases": 300000,
      "name": "default",
      "path": "",
      "role": "",
      "type": "lease-count"
   }
   

2. Use the sys/quotas/lease-count/default endpoint to adjust the default lease

   $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
      --request POST \
      --data '{ "max_leases": 10000 }' \
      $VAULT_ADDR/v1/sys/quotas/lease-count/default
   

   You can read the default settings again to verify the changes.

You can use Terraform to set lease count quotas.

The following Terraform file adjusts the global lease count quota. The new maximum number of leases is 10,000 instead of 300,000.

resource "vault_quota_lease_count" "default" {
  name = "default"
  max_leases = 10000
}

1. Remember that you created a global-rate quota rule. This quota rule applies to the global level instead of a specific mount or namespace.

   $ vault read sys/quotas/rate-limit/global-rate
   
   Key               Value
   ---               -----
   block_interval    0
   inheritable       true
   interval          1
   name              global-rate
   path              n/a
   rate              1500
   role              n/a
   type              rate-limit
   

2. Create a quota rule on the us-west namespace which its child namespaces will inherit. The rate limit is 500 requests per minute.

   $ vault write sys/quotas/rate-limit/us-west \
      path="us-west" \
      rate=500 \
      interval=1m \
      inheritable=true
   

   Output:

   Success! Data written to: sys/quotas/rate-limit/us-west
   

3. Verify the quota rule.

   $ vault read sys/quotas/rate-limit/us-west
   
   Key               Value
   ---               -----
   block_interval    0
   inheritable       true
   interval          60
   name              us-west
   path              us-west/
   rate              500
   role              n/a
   type              rate-limit
   

1. Remember that you created a global-rate quota rule. This quota rule applies to the global level instead of a specific mount or namespace.

   $ curl -s --header "X-Vault-Token: $VAULT_TOKEN" \
       $VAULT_ADDR/v1/sys/quotas/rate-limit/global-rate | jq -r ".data"
   

   Output:

   {
      "block_interval": 0,
      "inheritable": true,
      "interval": 1,
      "name": "global-rate",
      "path": "",
      "rate": 1500,
      "role": "",
      "type": "rate-limit"
   }
   

2. Create a quota rule on the us-west namespace which its child namespace inherits. The rate limit is 500 requests per minute.

   $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
      --request POST \
      --data '{"path": "us-west", "rate": 500, "interval": 60, "inheritable": true }' \
      $VAULT_ADDR/v1/sys/quotas/rate-limit/us-west
   

3. Verify the quota rule.

   $ curl -s --header "X-Vault-Token: $VAULT_TOKEN" \
      $VAULT_ADDR/v1/sys/quotas/rate-limit/us-west | jq -r ".data"
   

   Output:

   {
      "block_interval": 0,
      "inheritable": true,
      "interval": 60,
      "name": "us-west",
      "path": "us-west/",
      "rate": 500,
      "role": "",
      "type": "rate-limit"
   }
   

You can use Terraform to create a quota rule on the us-west namespace which its child namespaces will inherit. The following Terraform configuration sets the rate limit to 500 requests per minute.

provider vault {}

# Create a "us-west" namespace
resource "vault_namespace" "us-west" {
   path = "us-west"
}

# Create a "us-west" rate limit quota 
resource "vault_quota_rate_limit" "us-west" {
  name = "us-west"
  path = "us-west"
  rate = 500
  interval = 60
  inheritable = true
}