Network coordinates

This page describes the role of network coordinates in Consul's catalog operations.

Introduction

Consul uses a network tomography system to compute network coordinates for nodes in the cluster. These coordinates allow the network round trip time (RTT) to be estimated between any two nodes. Consul can use RTT when returning services. For example, it can find the service node that is nearest to the one making the request, or it can fail over to services in the next closest datacenter.

Consul uses the the Serf library, which also provides gossip communication between agents in a cluster, to determine network coordinates and their RTT.

Network coordinates in Consul

Network coordinates manifest in several ways inside Consul:

• The consul rtt command queries the network round trip time between any two nodes.
• The Catalog API endpoints and Health API endpoints can sort the results of queries based on the network round trip time from a given node by including the ?near= parameter.
• Prepared queries can automatically fail over services to other Consul datacenters based on network round trip times.
• The Coordinate endpoint exposes raw network coordinates for use in other applications.

Although Consul uses Serf to manage both the LAN gossip pool and the WAN gossip pool, network coordinates are not compatible between these two pools. LAN network coordinates only make work with other LAN coordinates, and WAN network coordinates only make work with other WAN coordinates.

Working with Coordinates

You can use the network coordinates for two nodes to calculate their RTT.

The following example shows a response from the Coordinate API endpoint.

[
  {
    "Node": "agent-one",
    "Segment": "",
    "Coord": {
        "Adjustment": 0.1,
        "Error": 1.5,
        "Height": 0.02,
        "Vec": [0.34,0.68,0.003,0.01,0.05,0.1,0.34,0.06]
    }
  }
]

All values are floating point numbers in units of seconds. The exception is the Error term, which is not used for distance calculations.

The following Go code calculates the distance between two coordinates:

import (
    "math"
    "time"

    "github.com/hashicorp/serf/coordinate"
)

func dist(a *coordinate.Coordinate, b *coordinate.Coordinate) time.Duration {
    // Coordinates will always have the same dimensionality, so this is
    // just a sanity check.
    if len(a.Vec) != len(b.Vec) {
        panic("dimensions aren't compatible")
    }

    // Calculate the Euclidean distance plus the heights.
    sumsq := 0.0
    for i := 0; i < len(a.Vec); i++ {
        diff := a.Vec[i] - b.Vec[i]
        sumsq += diff * diff
    }
    rtt := math.Sqrt(sumsq) + a.Height + b.Height

    // Apply the adjustment components, guarding against negatives.
    adjusted := rtt + a.Adjustment + b.Adjustment
    if adjusted > 0.0 {
        rtt = adjusted
    }

    // Go's times are natively nanoseconds, so we convert from seconds.
    const secondsToNanoseconds = 1.0e9
    return time.Duration(rtt * secondsToNanoseconds)
}