Configure and deploy your monitoring agent on VMs

You can configure and deploy monitoring agents in different ways:

• Use golden images with the monitoring agent already configured. (Recommended)
• Use a post-provisioning script or configuration management tools to configure the monitoring agent.

We recommend using golden images that include the pre-installed monitoring agents, which developers can use to build their applications. These produce faster, more consistent deployments and reduce the risk of flawed deployments.

Post-provisioning workflows require more time and effort to install and configure software on each individual system after provisioning, which can lead to inconsistencies and increased maintenance overhead.

We recommend you use Terraform to deploy and manage your virtual machines. Terraform can deploy virtual machines from golden images created by Packer or execute configuration management scripts to install and configure agents. Adopting infrastructure-as-code practices with Terraform provides a consistent, versionable workflow for your infrastructure.

How you deploy monitoring agents depends on how you configure and install them on virtual machines. The following sections provide resources for deploying pre-built images created with Packer, as well as deploying virtual machines and installing agents through post-provisioning scripts.

Create golden image with monitoring agent

We recommend creating a base machine image that already has your chosen monitoring agent installed. This ensures consistency, since each service will automatically send monitoring data to your central monitoring platform. Your organization can use Packer with configuration management tools, like Ansible, to create consistent and centrally-managed images. These preconfigured “golden images” have all the necessary software, dependencies, and security patches to run your services.

Platform teams can use Packer to codify and build golden images across multiple platforms. Application teams then create new images using the golden images as a starting point, and add their own services to them. They can then deploy the images to cloud providers like AWS, Azure, and Google Cloud using tools like Terraform. When you build the service image from the golden image, it includes a properly configured monitoring agent that sends both node and service metrics to your central monitoring platform.

If platform teams need to update the monitoring agent or install security patches, they can rebuild the golden image and notify any downstream teams that rely on it so they can update their own images. HCP Packer registry automates this process by tracking artifact metadata, and providing developers the correct information through Packer or Terraform. HCP Packer also lets you revoke artifacts to remove them from use if they become outdated or have security vulnerabilities. By using HCP Packer, you can automate your image pipelines and ensure all your machine images are secure and follow your organization's rules and policies.

HashiCorp resources:

• The Build a golden image pipeline with HCP Packer tutorial guides you through using HCP Packer to build parent and child application images and deploy them with Terraform.
• The Deploy a Packer generated AMI with Terraform tutorial shows you how to build an AMI using Packer and deploy to AWS.
• The Automate Packer with GitHub Actions tutorials guides you through a complete GitHub Actions workflow to automatically build and manage different versions of an image artifact. Automating image pipelines with HCP Packer is a companion video that covers the same topic.
• The Standardize artifacts across multiple cloud providers tutorial guides you through using Packer and HCP Packer to standardize artifacts across multi-cloud and hybrid environments. You will also deploy the image artifacts to both AWS and Azure.
• The Ansible Packer provisioner runs Ansible playbooks. This provisioner expects that Ansible is already installed on the guest or remote machine.
• To learn more about immutable infrastructure, view Armon's What is mutable vs. immutable infrastructure? video.

External resources:

• DataDog provides resources to configure their agent using a configuration management tool like Ansible, Chef, Puppet, and Saltstack.
• New Relic provides guides to automatically configure its agent with Ansible, Chef, and Puppet.
• AWS provides instructions on how to install its AWS Unified CloudWatch Agent on Linux virtual machines.

Automate during post-provision

If your organization or environment prohibits you from using golden image pipelines you can automatically install monitoring agents after you create the virtual machine (VM).

For example, you can write a script that installs the monitoring agent and configures it to send data to your central monitoring platform. When you use a tool like Terraform to deploy the VM, you can tell it to run this script after the VM launches. Each cloud provider has a different method to run these scripts. In AWS for example, you can provide this script to the aws_instance resource with the user_data argument.

Another option is to use configuration management tools like Chef, Puppet, or Ansible. These tools provide a structured way to consistently configure software across your services.

These post-provisioning methods may take longer to set up the monitoring agents and services after creating the VM than using a machine image. The monitoring agent configuration can become inconsistent if the script or configuration has errors. You also incur the maintenance burden of managing the script lifecycle to support more services as your infrastructure changes.

External resources:

• AWS - You can define your post-provisioning scripts in the user_data argument of the aws_instance resource. Refer to the AWS guide for AWS specific guidance on user data scripts.
• Azure - You can define your post-provisioning scripts in the custom_data argument of the azurerm_virtual_machine resource. Refer to the Azure guide for Azure specific guidance on user data scripts.
• GCP - You can define your post-provisioning scripts in the metadata_startup_script (or metadata.startup_script) argument of the google_compute_instance resource. Refer to the Google Cloud guide for Google Cloud specific guidance on startup scripts.

Next steps

In this section of Setup monitoring agents, you learned how to configure and deploy monitoring agents on virtual machines. Configure and deploy your monitoring agent on VMs is part of the Define and automate processes pillar.