AI observability
Consul Agentic Runtime Fabric observability connects model access, tool calls, policy decisions, and operational outcomes. Metrics, traces, and audit evidence serve different purposes, while higher-order key performance indicators (KPIs) also require application-provided outcomes.
The AI Inference Gateway and MCP sidecars provide automatic instrumentation for supported model and tool traffic without application tracing SDKs. The compliance audit trail and enforced audit-delivery mode described by the architecture are separate design capabilities, not inference-gateway configuration documented here. Higher-order KPIs require additional application and analysis integrations.
Enterprise
Appropriate Consul Enterprise license required.
Introduction
The architecture separates three independently configurable signal types:
| Signal | Purpose | Boundary |
|---|---|---|
| Metrics | Aggregate traffic, latency, errors, usage, and policy outcomes. | Aggregation and delivery loss make metrics unsuitable as a billing ledger. |
| Traces | Connect request processing and application workflow activity. | Sampling and incomplete instrumentation limit what a trace can establish. |
| Audit evidence | Describe attributable governance decisions for investigation. | Durable, complete, tamper-evident retention requires delivery and storage controls beyond ordinary telemetry. |
Each component emits the events it can observe. An AI Inference Gateway cannot supply tool-business outcomes or agent workflow completion merely because those operations belong to the same application.
Correlation and model attribution
Request and trace identifiers connect observations across processing stages. Inference processing before and after routing must remain correlatable even though it uses separate external-processing streams.
A request identifier is a join key, not a workload identity or authorization credential. Validate untrusted correlation values and keep record identity separate from a caller-controlled identifier.
Requested-model and served-model attribution answer different questions. Preserve both where available, and correlate approved fallback and retry evidence with the relevant attempt. The architecture requires actual-model attribution to be available to authorized users. A client-facing interface alone does not establish which model served a request.
Metrics and traces
Inference metrics describe request outcomes, stage duration, token usage, streaming behavior, identity resolution, and configured PII processing. Personally identifiable information (PII) signals identify detector types and actions, not detected values.
Metrics use bounded dimensions. Do not promote request identifiers, trace identifiers, user identifiers, full workload identities, arbitrary paths, or request content into metric labels. Unbounded labels increase memory use and can expose sensitive information.
Tracing describes the gateway and tool-call paths. Applications propagate compatible trace context so proxy-generated spans form a correlated workflow trace; they do not need to emit spans themselves. Proxy spans do not instrument internal agent reasoning, local tool business logic, or application-defined task outcomes.
The inference implementation exports metrics through Prometheus and optionally through OpenTelemetry Protocol (OTLP). Tracing requires explicit enablement and an OTLP destination. These exporters are best-effort: a telemetry-sink failure must not change the inference request outcome.
Agentic Analytics consumes correlated trace evidence for cycle detection. It checks repeated workflow steps and their results, including repeated calls to the same tool with the same arguments. Consul uses policy decisions based on these findings to open agent-scoped circuits. Trace collection alone does not enable circuit enforcement.
Proposed audit delivery
The audit design separates admission evidence from completion evidence. A denied request needs a decision record even though it never reaches a provider. An admitted request can also need a completion record containing outcome and usage attribution.
The design calls for explicit loss accounting and correlation across these records. Sampling, buffer exhaustion, process termination, and sink failures can create gaps. A local buffer or standard-output stream does not by itself provide durable or tamper-evident storage.
The proposed enforced-delivery mode is an explicit exception to best-effort emission: admission can depend on the configured audit-delivery guarantee. Do not infer this guarantee from ordinary metrics, tracing, or logs, and do not assume that the mode exists without component support.
Audit retention, access control, integrity, and downstream delivery remain operational responsibilities. Audit evidence can support compliance work. Its presence does not establish certification or regulatory compliance.
Operational metrics
Auto-instrumentation supplies observations, not finished metrics. Configure your metrics or analytics backend to aggregate them. The following measures are calculation guidance, not built-in metric names.
| Measure | Calculation inputs | Calculation rule |
|---|---|---|
| Request rate | Logical model requests or tool calls over a defined time window. | Count each exchange once, not each network, payload, and processor span. |
| Protocol error rate | Failed observed exchanges divided by all observed exchanges in the same population and window. | Include tool and JSON-RPC errors. HTTP success alone does not show tool success. |
| Request latency | Request durations as distributions or percentiles for one observation point. | Do not mix Envoy network timing with processor timing, or add sibling spans as sequential work. |
| Token usage | Usage reported in model responses, attributed by model and request. | Treat missing usage as unknown, not zero. Count a response observed on both sides of a gateway once. |
Define the observation point, denominator, time window, and retry treatment before you compare results. Sampled traces describe the observed population, not all traffic. Use complete request counters for traffic totals where available, and do not present sampled span counts as exact totals.
Higher-order KPIs
Higher-order KPIs combine network observations with application semantics. Define the population, outcome criteria, time window, and treatment of incomplete workflows before interpreting a result.
| KPI | Required inputs | Interpretation limit |
|---|---|---|
| Task completion rate | Application-defined task starts, terminal outcomes, and correlation across calls. | Successful HTTP requests or model responses do not prove task completion. |
| Tool success rate | Tool protocol outcomes and application-defined business results. | A protocol success can still represent an unsuccessful business operation. |
| Cost per successful task | Attributed usage, maintained provider pricing, and confirmed task outcomes. | Missing usage, retries, pricing changes, and incomplete tasks affect the result. |
| Autonomous versus escalated outcomes | Application approval events and final task outcomes. | Avoid treating every approval request or timeout as a completed task. |
| Model quality or routing effectiveness | Application evaluations and confirmed served-model attribution. | Latency, token counts, and availability are not quality measurements. |
These are analysis requirements, not a claim that the gateway emits each KPI directly. Keep operational cost estimates separate from an authoritative invoice or financial ledger.
Privacy and content boundaries
Keep prompts, responses, tool arguments, and tool results out of routine metrics, decision logs, and audit records. Content-bearing payload traces require explicit data-handling approval and filtering or redaction before export, as described in auto-instrumentation privacy controls. Do not enable that capture when the deployed path cannot enforce those controls. Keep matched PII, credentials, and tokens out of telemetry and audit records. Provider errors can contain request data, so prefer bounded error categories over raw error messages.
Correlated metadata can still be sensitive. Apply access controls and retention limits to identity, model usage, policy decisions, and application outcomes even when content capture is disabled.
Dependencies and boundaries
The observability component owns signal definitions, privacy rules, delivery modes, and KPI prerequisites. Specialized proxies automatically capture supported network interactions. Applications preserve trace context and supply business outcomes that proxy observations cannot establish. Agentic Analytics evaluates the available evidence; missing or redacted content can limit semantic analysis.