AI Inference Gateway

The AI Inference Gateway governs agent access to large language models (LLMs) in Consul Agentic Runtime Fabric. It provides a supported inference interface between applications and model providers, with policy controls for routing, usage, and sensitive data.

Agents reach the gateway over their Consul service mesh path. The gateway forwards each request to a model registered in the Consul catalog. To learn how model, tool, and identity paths fit together, refer to the Consul Agentic Runtime Fabric architecture.

Introduction

Applications send inference requests to a gateway instead of integrating network policy separately for each provider. The gateway accepts an OpenAI-compatible request format and translates supported operations to the selected provider's format. Interface compatibility does not imply support for every provider feature or equivalent behavior between models.

Consul supplies gateway configuration and eligible backend information. Envoy enforces mesh connectivity policy and performs routing and retries. The external processor validates and transforms requests and responses. It does not independently select routes or run a retry loop.

Deployment prerequisites

You need a valid Consul Enterprise license to use the AI Inference Gateway. Configure the license in your Consul deployment, then enable the consul-ai feature gate, which is disabled by default:

$ consul operator feature set consul-ai enabled

The gateway requires Envoy 1.38.2 or later. Pull the csl-inf-rt image alongside the other required Agentic Runtime Fabric components. On Kubernetes, set ai.enabled to true to enable the feature gate, and set the gateway image with the ai.inferenceGateway.image Helm value.

Replace <registry> with the registry path supplied for your distribution and <version> with a compatible release tag. For Docker, run:

$ docker pull <registry>/csl-inf-rt:<version>

For Podman, run:

$ podman pull <registry>/csl-inf-rt:<version>

Authenticate to the registry when required and pin an approved release tag or digest in your deployment manifest. Deploy the image using the configuration supplied with that release. Pulling the image does not configure model backends, provider credentials, mesh routes, or gateway policy.

Request processing

The inference path separates processing before and after Envoy selects an upstream:

The upstream processor uses information about the backend that Envoy selected. A retry against another backend must use that backend's adapter and model information rather than reuse an incompatible translated request.

Model selection and fallback

Model selection has two distinct inputs: the caller's request and the operator's policy. The architecture requires the gateway to preserve an explicit model choice when policy permits it. Policy can reject a disallowed request. It must not silently substitute an unrelated model.

Caller-approved fallback constrains which alternatives are acceptable. A routing policy's ability to reach another backend is not, by itself, permission to change the requested model. Treat retrying a request and changing models as separate decisions.

For requests that delegate model selection to the gateway, eligible providers and routing policy determine the selection. Provider translation changes the wire format. It does not establish that two models have the same capabilities, quality, or data-handling terms.

The requested model and the model that serves the request are separate attribution values. The architecture requires actual-model attribution to be available to authorized users, including when an approved fallback occurs.

Usage controls

Token-aware usage controls govern admission and account for completed work. Admission can depend on estimated consumption because final provider usage is unavailable before a request completes. Usage accounting must distinguish estimates from reported consumption, including incomplete streams and retries.

Keep rate-limit decisions, observed token usage, and financial accounting distinct. A token count is not a price, and gateway telemetry is not a billing ledger. Cost analysis requires a maintained pricing source and a defined policy for missing or estimated usage.

Data protection and streaming

Personally identifiable information (PII) processing applies the configured detection and handling rules to request or response content. Request-side controls act before data reaches a provider. Response-side controls act before processed content returns to the caller.

Streaming introduces an additional boundary: content already delivered to the caller cannot be recalled. Validate detector coverage, buffering behavior, and supported stream formats for your deployment. Do not interpret configured PII controls as a guarantee that all sensitive content is detected.

Operational metrics and decision logs record bounded metadata, not prompts, responses, detected PII values, credentials, or provider error text that can contain request content. Content-bearing payload traces require separate approval and filtering or redaction under the auto-instrumentation privacy controls.

Automatic instrumentation

The AI Inference Gateway is a specialized proxy that automatically instruments supported model exchanges. Gateway tracing is off by default. Enable it with the Observability.Tracing block in the inference-gateway configuration entry, as described in Configure auto-instrumentation. Its Envoy and protocol-aware external-processing components generate network and AI payload spans without application tracing SDKs or manual spans. Applications preserve trace context across calls; model usage and responses do not establish business-level task completion.

Agentic Analytics can use the correlated evidence for cycle detection. Detection and agent-scoped enforcement require the analytics, policy, and protection-state path in addition to gateway tracing.

Dependencies and boundaries

• Mesh identity and policy establish the workload's permitted network path to the gateway and its backends.
• Provider adapters define supported request formats, response formats, streaming behavior, and translation limits.
• Routing rules define eligible backends. Application consent and operator policy constrain model substitution.
• Usage consumers require consistent attribution across admission, provider usage, and completion evidence.
• Managed-model authentication is separate from delegated user identity and external credential injection.

The inference component owns adapter behavior, streaming, routing-rule integration, accounting, and PII processing. It does not own tool business logic or determine whether a model response satisfies the user's task.

Related documentation

• AI observability
• External credential injection
• Identity plane
• Auto-instrumentation
• Cycle detection