Import existing resources in bulk

You can configure queries that instruct Terraform to search your existing infrastructure for unmanaged resources. Terraform can also generate configuration for importing the resources it finds so that you can import them to your Terraform workspace in bulk. For information about importing single resources or small batches of resources, refer to Import single resources for instructions.

Note

This feature is currently in beta. Do not use beta functionality in production environments.

Introduction

For organizations with large sets of infrastructure resources, manually identifying and importing them is tedious and labor intensive, even when using third-party tools or custom scripts. To alleviate this burden, you can write HCL-based queries and run them with the Terraform CLI to retrieve unmanaged resources so that you can import them in bulk.

Complete the following steps to find and import resources in bulk:

Search for resources: Create list blocks to search for existing resources.
Generate configuration: Use the Terraform command to generate resource and import blocks for importing the resources. The configuration also includes the resource identity.
Import resources: Copy the generated configuration into your main.tf file and run a terraform apply command to import the resources.
After importing resources, you can remove the generated import block or keep it as an historical record.

Requirements

Terraform v1.12 or newer is required to search for resources according to their resource identities. HCP Terraform uses resource identities to determine when resources are managed by another workspace. Refer to Import existing resources to state for more information.

Supported providers

Verify that the resource type you want to search for is supported. For the initial beta release, you can use list blocks to query for the following resource types. This list may change. Refer to your provider documentation for the most up-to-date information.

AWS provider

AWS CC provider

All resources in this provider support list blocks. Because this provider is fully generated and doesn't accept provider-specific configurations, omit the config block when defining queries for AWS CC resources.

Define a query

To search for resources, you must create a standard .tf configuration and a .tfquery.hcl configuration.

Standard configurations

If your .tf configuration does not already have a required_providers block, add it so that Terraform can install all provider plugins required to create and manage resources specified in the configuration. Refer to the required_providers reference for details on how to configure this block.

If you have an HCP Terraform account, you can compare the resources Terraform discovers to resources you are already managing with Terraform in the HCP Terraform UI. To use this functionality, you must configure the cloud block in your .tf configuration to connect to HCP Terraform. Refer to the cloud block reference for details on how to configure this block.

Query configurations

Create a file with a .tfquery.hcl extension and add list blocks to define queries that read existing infrastructure and return lists of unmanaged resources.

Each list block requires an inline argument that specifies the type of resource you are querying. The type is specific to your provider. The block also requires an inline argument that declares a local name for the list of results returned by the query. Refer to the list block reference for configuration details.

The following example defines a list block named prod that queries aws_instances:

list "aws_instance" "prod" {
  # . . .
}

Add the following arguments to your list block:

provider: This argument is required. It specifies which provider configuration Terraform should use to perform the query. The list block retrieves the provider configuration from the terraform block in your main.tf configuration, but you can declare a provider block in your query file with an alternate configuration and reference it in the provider argument in your list block.
config: Add the config block to your list block and define provider-specific arguments to build your query. Refer to your provider documentation for supported arguments and values.
include_resource: By default, Terraform retrieves only resource identities, but you can set the include_resource argument to true so that Terraform can also retrieve all available resource attributes. To reference resource attributes retrieved by the list block, you must enable the include_resource argument. Setting this argument to true may affect performance.
limit: By default, Terraform retrieves up to 100 resources per list block, but you can use the limit argument to specify a higher or lower number of results.

The list block also supports several Terraform meta-arguments, which are arguments built into Terraform configuration language that configure how Terraform creates and manages infrastructure objects. For example, you can use the count meta-argument to create multiple instances of the resource list returned by the query. Refer to Meta-arguments for more information.

In the following example, Terraform applies provder arguments to query for aws_instance resources in the us-east-2 region. The query also filters results prefixed with prod- and stage- and that are in a running state. The limit argument, which is not a provider-specific argument, instructs Terraform to return the first 50 results:

list "aws_instance" "prod" {
  provider = aws
  limit = 50
  config {
    region = "us-east-2"
    filter {
      name   = "tag:Name"
      values = ["prod-*", "staging-*"]
    }
    filter {
        name = "instance-state-name"
        values = ["running"]
    }
  }
}

Parameterize your query configuration

You can parameterize your .tfquery.hcl file so that you can reuse it with a different set of inputs. Refer to Set configuration parameters for instructions on how to parameterize configurations.

Add the following blocks to your query file to parameterize the configuration:

variable: Add the variable block to your query file to define variables that people who run the query configuration can provide at runtime. Refer to the variable block reference for configuration details.
locals: Add the locals block to your query file to define temporary variables scoped to the query configuration. Refer to the locals block reference for configuration details.

Specify a custom provider configuration

The query file checks the main.tf configuration for provider configurations, but you can also declare one or more provider blocks in the query configuration file to define alternate provider configurations. To use one of the alternate provider configurations, add the provider argument to your list block and reference the name of the provider configuration. Refer to the provider block reference for information about how to configure provider blocks.

Query the infrastructure

After configuring the query configuration, you can search for resources using the Terraform CLI or using HCP Terraform if applicable.

HCP Terraform

To run queries from the HCP Terraform UI, you must configure a connnection to HCP Terraform and copy configuration files from your working directory to your workspace in HCP. Refer to Connect to HCP Terraform for instructions.

If HCP Terraform is connected to your version control system (VCS), you can also check your configurations into your VCS before running the query in HCP Terraform. Refer to Connect to VCS Providers for more information.

After copying the configuration to your HCP Terraform workspace, you can either run the query locally using the Terraform CLI or run the query in the HCP Terraform UI. Refer to Import existing resources to state in the HCP Terraform documentation for instructions on how to run queries and import resources from the UI.

Refer to Import existing resources to state in bulk in the HCP Terraform documentation for details.

Terraform CLI

Run the terraform query command to retrieve the resource types specified in your list blocks. The command prints the results to your console. By default, each result includes the following information:

Reference to the list block that queried for resources formatted as list.<type>.<label>.
Identity of a discovered resource.

The provider may also include other information, such as a description of the resource. Refer to your provider documentation for details.

The number of results depends on the provider, as well as any query constraints you configured in the query configuration file.

To generate machine-readable results, you can include the -json flag:

$ terraform query -json

Generate configuration for importing results

To generate configuration, run the terraform query command and add the -generate-config-out flag. The flag expects a path to a generated file named generated.tf. The generated.tf file contains the resource and import blocks, including resource identities, necessary for importing results.

The following example generates configuration to the working directory:

$ terraform query -generate-config-out=generated.tf

Even when connected to HCP Terraform or Terraform Enterprise, Terraform creates the generated.tf file and stores it on the local workstation when running the terraform query command with the -generate-config-out flag.

After generating the results file, you must remove the file before rerunning the command to generate a new results file. Rerunning the command when a generated.tf already exists at the specified path results in an error.

Import resources

Copy the import and resource blocks from the generated.tf file to your main.tf configuration and run the terraform apply command to import the resources.

Next steps

You can discard the generated.tf file after importing your resources. You can either remove import blocks after you've imported them or leave them in your configuration as a record of the resource's origin for future module maintainers. For more information on maintaining configurations over time, refer to Refactoring.