Edge Gateway (Data Plane) Component

​

Availability

Note: The Edge Gateway is sometimes referred to as the Microgateway in configuration files. However, Edge Gateway is the preferred terminology.

• Unified Access Point: Provides a single, consistent endpoint for applications to interact with various LLMs.
• Security Enforcement: Handles authentication, authorization, and applies security policies.
• Policy Management: Enforces rules related to budget limits, model access, and applies custom Filters.
• Observability: Logs detailed analytics data for each request, feeding the Analytics & Monitoring system.
• Vendor Abstraction: Hides the complexities of different LLM provider APIs, especially through the OpenAI-compatible endpoint.

​

Edge Gateway Variants

​

Core Features

1. Request Routing: Incoming requests include an llmSlug in their path (e.g., /llm/call/{llmSlug}/...). The Proxy uses this slug (auto-generated from the LLM configuration name) to identify the target LLM Configuration and route the request accordingly.
2. Authentication & Authorization:
   • Validates the API key provided by the client application.
   • Identifies the associated Application and User.
   • Checks if the Application/team has permission to access the requested LLM Configuration based on RBAC rules.
   • Applies the same access check to every authentication method: App API key, Authorization: Bearer App secret, or an auth plugin. The check also applies to every entry point: /llm/, /ai/, /anthropic/, /v1, and /datasource/. An App can call only the LLMs and data sources that it has a grant for. It can also reach the fallbacks of a granted LLM’s failover waterfall, but only through that waterfall. A request for any other LLM, or for a slug that does not exist, returns 403.
   • If an endpoint has auth plugins attached, only those plugins authenticate its requests on the Edge Gateway. The Edge Gateway refuses App keys on that endpoint.
3. Policy Enforcement: Before forwarding the request to the backend LLM, the Proxy enforces policies defined in the LLM Configuration or globally:
   • Budget Checks: Verifies if the estimated cost exceeds the configured Budgets for the App or LLM.
   • Model Access: Ensures the requested model is allowed for the specific LLM configuration.
   • Filters: Applies configured request Filters to modify the incoming request payload.
4. Analytics Logging: After receiving the response from the backend LLM (and potentially applying response Filters), the Proxy logs detailed information about the interaction (user, app, model, tokens used, cost, latency, etc.) to the Analytics database.

​

Endpoints

​

How the Gateway Builds the Vendor URL

​

Main Ingress

• Model strings have a prefix. Send {llmSlug}/{model}, for example openai-prod/gpt-4o. The prefix selects the LLM. The gateway removes the prefix before it sends the request to the vendor. A model string without a prefix returns 400. The gateway splits the string at the first slash, so model names that contain slashes (for example, fine-tune IDs) stay complete.
• OpenAI format only. Requests and responses use the OpenAI Chat Completions schema for all vendors. To use a vendor’s SDK and native parameters, use the vendor-native endpoint.
• Streaming. The gateway streams the response when the request contains stream: true.
• Model discovery. GET /v1/models lists all the models that the authenticated App can call, with the prefix included. The gateway builds this list from the allowed models and the default model of each LLM. An LLM that has no allowed models and no default model is not in the list.
• Legacy completions. The gateway also serves /v1/completions for clients that use the older completions API.

curl -X POST "https://your-gateway-host/v1/chat/completions" \
  -H "Authorization: Bearer YOUR_APP_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "my-openai-config/gpt-4-turbo",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

# One client for every LLM in the App
import openai

client = openai.OpenAI(
    base_url="https://your-gateway-host/v1",
    api_key="YOUR_APP_API_KEY"
)

response = client.chat.completions.create(
    model="my-openai-config/gpt-4-turbo",  # or "my-bedrock-config/claude-3-5-sonnet"
    messages=[{"role": "user", "content": "Hello!"}]
)
print(response.choices[0].message.content)

​

Vendor-Native Endpoint

• You get access to all vendor features. When a vendor adds a new parameter, it works immediately, without a gateway change.
• One URL serves streaming and non-streaming requests. The gateway examines the request and selects the correct path.

# Anthropic LLM, Anthropic request format
curl -X POST "https://your-gateway-host/llm/call/my-anthropic-config/v1/messages" \
  -H "Authorization: Bearer YOUR_APP_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-3-5-sonnet-20241022",
    "max_tokens": 1024,
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

​

OpenAI-Compatible Endpoint

• Use this endpoint when a client must use only one LLM. Also use it when a tool cannot send a model string with a slug prefix.
• Vendor features that the OpenAI schema cannot express are not available.
• Vertex and Hugging Face LLMs are not supported on this endpoint or on the Main Ingress. A request returns 400 with the code unsupported_vendor, and the gateway does not send the request to the vendor. Use /llm/rest/{llmSlug} or /llm/stream/{llmSlug} for these vendors. In a failover waterfall, the gateway skips such a fallback and tries the next one.

# Pinned to one LLM
import openai

client = openai.OpenAI(
    base_url="https://your-gateway-host/ai/my-openai-config/v1",
    api_key="YOUR_APP_API_KEY"
)

response = client.chat.completions.create(
    model="gpt-4-turbo",  # plain model name, no slug prefix
    messages=[{"role": "user", "content": "Hello!"}]
)
print(response.choices[0].message.content)

​

Anthropic Messages Endpoint

• This endpoint supports Bedrock LLMs only. Other vendors already accept their own API on the vendor-native endpoint. A request to a non-Bedrock LLM returns 400.
• Set the endpoint URL as ANTHROPIC_BASE_URL, and set the App key as ANTHROPIC_AUTH_TOKEN (or ANTHROPIC_API_KEY). The client adds /v1/messages to the URL.

export ANTHROPIC_BASE_URL="https://your-gateway-host/anthropic/my-bedrock-config"
export ANTHROPIC_AUTH_TOKEN="YOUR_APP_API_KEY"

​

Model Discovery in Claude Code

export CLAUDE_CODE_ENABLE_GATEWAY_MODEL_DISCOVERY=1

{"data":[{"type":"model","id":"eu.anthropic.claude-3-5-sonnet-20241022-v2:0","display_name":"Bedrock Claude — eu.anthropic.claude-3-5-sonnet-20241022-v2:0"}],"has_more":false}

• The models request uses the same authentication and App access check as /v1/messages. It does not call a model, so it does not use budget, run filters, or record usage.
• The list is empty ("data":[]) when the LLM has no default model. It is also empty when the allowed models of the LLM do not include the default model. In both cases, Claude Code then shows its built-in list.
• Use Claude Code v2.1.223 or later. Claude Code keeps a model ID only if the ID contains “claude” or “anthropic”. Versions v2.1.129 to v2.1.222 required the ID to start with one of these words. For this reason, they hid region-prefixed IDs such as us.anthropic... or eu.anthropic....
• Claude Code does not show application inference profiles. An application inference profile ARN (arn:aws:bedrock:...:application-inference-profile/...) does not contain “claude” or “anthropic”, so Claude Code drops it. Requests still work. Only the picker entry is missing. System inference profile ARNs contain “anthropic”, so Claude Code shows them.
• Do not redirect this path. Claude Code treats any redirect as a failed discovery, including an http:// to https:// redirect at your ingress. Set ANTHROPIC_BASE_URL to the final URL.

​

Legacy Endpoints

• /llm/rest/{llmSlug}/... accepts only non-streaming (synchronous) requests.
• /llm/stream/{llmSlug}/... accepts only Server-Sent Events streaming requests.

​

Configure the Main Ingress

• It adds a leading slash if there is none.
• It removes a trailing slash and any empty or dot segments.
• It refuses / and uses the default, so that the Main Ingress cannot take the full gateway path.

​

LLM Slug

​

Model Router

​

Configuration Reference

​

Troubleshooting