Backup and restore initial setup

This page covers initial preparation for backup and restore. Recurring backup execution and restoration runbooks belong in the Administration Guide.

Definitions

Business continuity (BC) is a corporate capability. This capability exists whenever organizations can continue to deliver their products and services at acceptable, predefined levels whenever disruptive incidents occur. When we refer to environments, these map to engineering/pre-development, development, various test (system integration testing, user acceptance testing and so on) and production. Reserve dedicated engineering instances for the platform team only, for replication, backup/restore and upgrade testing.

Two factors heavily determine your organization's ability to achieve BC:

1. Recovery Time Objective (RTO)(opens in new tab) is the target time set for the resumption of product, service, or activity delivery after an incident. For example, if an organization has an RTO of one hour, they aim to have their services running within one hour of the service disruption.
2. Recovery Point Objective (RPO)(opens in new tab) is the maximum tolerable period of data loss during an incident. For example, if an organization has an RPO of one hour, they can tolerate the loss of a maximum of one hour's data before the service disruption.

Based on these definitions, we recommend establishing the valid RTO/RPO for your Terraform Enterprise instance and approach BC accordingly. These factors determine your backup frequency and other considerations discussed below.

In this guide:

• Reference to a public cloud availability zone (AZ) is equivalent to a single VMware-based data center.
• Reference to a public cloud multi-availability zone is equivalent to a multi-data center VMware deployment.
• The main AZ is the primary. Any other AZs in the same region are the secondary. The secondary region is a business continuity/failover region only and is not an active secondary location. Consider all availability zones equal for the purposes of illustration.

Best practices

Maintain the backup and restore process

When you deploy Terraform Enterprise:

• Test the backup and restoration process and measure the recovery time to ensure it satisfies your organization's RTO/RPO.
• Document the backup and restoration process.
• Arrange for staff who did not write this document to run a test restore using it. This measure increases confidence in the backup and restore process.
• Use a regular schedule to test the backup and restoration process to ensure the documentation is reliable. This increases familiarity with the backup and restoration process and covers situations where staff leave.

Manage sensitive values

For automated deployments, you must manage several sensitive values. On Kubernetes, refer to this support page for base information(opens in new tab). The HVD deployment modules also provide further information specific to the deployment method and target platform - see below for further details. The methods below do not back up these data. We recommend HashiCorp Vault(opens in new tab) for storage of all pipeline secrets. Do not store any of these sensitive values in version control or allow them to leak into shell histories. Do not store any of these sensitive values in version control or allow them to leak into shell histories.

We recommend HashiCorp Vault(opens in new tab) for storage of all pipeline secrets.

Process audit logs

Audit log processing helps you identify the root cause during a data recovery incident.

Follow the guidance on this Terraform Enterprise logs resource page(opens in new tab) to aggregate and index logs from the Terraform Enterprise node(s) using a central logging platform such as Splunk, ELK, or a cloud-native solution. Use these as a diagnostic tool in the event of outage, scanning them case insensitively for ERROR and FATAL messages as part of root cause analysis.

Terraform Enterprise backup API

Terraform Enterprise has a Backup API(opens in new tab) but this primarily facilitates migrations from one operational mode to another. Only use the backup API for such migrations. Use cloud-native tooling instead for day-to-day backup and recovery on public cloud, and standard approaches for on-premise deployments as detailed below.

Initial considerations

The following recommendations improve your security posture, reduce the effort required to maintain an optimal Terraform Enterprise instance, and speed up deployment time during a restoration. Apply all relevant points in the list below for your target platform (VM-based or Kubernetes-based deployments).

• Harden server images using CIS benchmarking(opens in new tab).
• Secure Terraform Enterprise deployments on both single-tenant and shared Kubernetes clusters. If using Kubernetes clusters, lock down all APIs properly using automated configuration capabilities such as Ansible.
• If using VM-based deployments of Terraform Enterprise, use single-tenant, immutable instances using automation by repaving instances with patched images rather than patching them in place. This process requires you to maintain the setup configuration in the code used to deploy the system and is out of the scope of this document.
• Remove all unnecessary packages from the operating system.
• Ensure to store deployment configuration in a version control system and use git best practices on the repositories used to version the code.