Performance replication operations
Monitoring performance replication
The health of Vault performance replication should be monitored. Monitoring Vault performance replication should include ensuring that:
- Both the primary and secondary clusters are online and in a healthy state
- Network communication between the primary and secondary clusters has not become disrupted
- Replication subsystem in Vault is operating in a healthy state on primary and secondary clusters and that data sync is keeping pace between the two
For item number one, refer to the Vault observability setup to ensure that telemetry has been configured on primary and secondary clusters, and the outlined key metrics are being monitored for general cluster health.
For item number two, a synthetic check can be run periodically against the sys/replication/performance/status API endpoint on both clusters to check the value of the “state” field. If the value of the state field is “idle” then this indicates that communication may have become disrupted and replication is no longer taking place. The progress and health of synchronization between primary and secondary clusters can be checked by querying the sys/replication/performance/status API endpoint on both clusters and comparing the values of last_performance_wal on the primary and last_remote_wal on the secondary. After writing a piece of replicated data, the last_remote_wal on the secondary should match the last_performance_wal on the primary for a short period of time.
For item number three, there are specific Vault telemetry values that should be monitored to indicate the health of the replication subsystem:
| Metric Name | Description |
|---|---|
| vault.wal_persistwals | Time taken to persist a WAL to storage |
| vault.wal_flushready | Time taken to flush a ready WAL to storage |
These metrics indicate the time being taken in the replication subsystem to process write ahead logs (WALs). Both of these metrics should be monitored for a situation where the values continue to grow without returning to the expected baseline. In a high-load scenario, WALs can accumulate, putting pressure on Vault’s storage backend.