Configure dashboards and alerts

As the number of services you manage and maintain grows, manually managing monitoring components like dashboards and alerts become unsustainable. This can lead to inconsistencies across environments, observability gaps where issues go undetected, and potential security risks. We recommend adopting monitoring-as-code (MaC) to manage these configurations to solve these challenges as your organization scales.

With monitoring-as-code (MaC), you can adopt many of the best practices as infrastructure-as-code (IaC), such as:

Consistent configuration: MaC lets you consistently deploy standardized monitoring setups across teams and environments. Terraform lets you create modules that include standard monitoring configurations with built-in reasonable defaults. A range of monitoring tools also offer official Terraform providers and modules your organization can use.
For example, the team responsible for ensuring data integrity can make changes to the Terraform modules and propagate those changes throughout the organization.
Automated provisioning: As your organization scales, you can configure infrastructure and service deployments to automatically trigger monitoring dashboards.
Auditable changes: All changes to monitoring components are traceable through version control.
Policy-compliant resources: You can use Sentinel and Open Policy Agent (OPA) to ensure your monitoring resources are secure and compliant with your organization's policies.

While the codified approach offers significant benefits, designing complex monitoring dashboards and alert rules directly in code can be challenging initially.

To balance this, we recommend an iterative workflow. First, leverage your monitoring tool's UI to visually design and build dashboards, layouts, and alert rules. This allows you to fully utilize the robust querying capabilities and intuitive interfaces provided by monitoring solutions. Once you have functional prototypes, import those configurations into Terraform code and standardize them as a Terraform module. From there, your organization can consume and modify the Terraform modules to create consistent monitoring dashboards and alerts across your infrastructure and services.

This approach combines the flexibility of visually designing your dashboards first with the consistency and maintainability of managing it as code.

External resources:

New Relic's article provides additional insights into why organizations should adopt monitoring-as-code.

Next steps

In this section of Configure dashboards and alerts, you learned the benefits of monitoring-as-code and how to use Terraform to define and manage dashboards and alerts on your infrastructure and services. Configuring dashboards and alerts is part of the Define and automate processes pillar.

Refer to the following documents to learn how to deploy monitoring tools with Terraform.