Skip to content

Use An OpenAI-Compatible Proxy

Direct LGOS is the protocol reference, while the maintained demo UIs enter through either LiteLLM or Bifrost. A proxy must carry the native Responses contract without an LGOS-specific response adapter or plugin. LiteLLM and Bifrost remain deployment choices and are not part of the package.

Native Responses Requirements

Configure a standard /v1 OpenAI base URL and verify the proxy preserves:

  • POST /v1/responses, including store: false, safety_identifier, user, string-valued metadata, client function tools, registered custom tools, and web_search;
  • typed Responses SSE events, item IDs, output indices, sequence numbers, and assistant phase values;
  • complete function_call items and matching function_call_output items for stateless tool continuation;
  • complete custom_tool_call and custom_tool_call_output items for LGOS-executed custom tools;
  • previous_response_id plus matching function_call_output items for interrupt continuation;
  • polling-only background creation, GET /v1/responses/{response_id}, and POST /v1/responses/{response_id}/cancel, with the opaque ID sufficient for lifecycle routing after both client and gateway restarts;
  • standard OpenAI error type, param, and code values;
  • Files upload, list, retrieve, content, and delete operations through one file namespace independent of graph routing; and
  • downstream disconnect propagation to the upstream streaming request.

LGOS does not require the proxy to retain Responses itself. Foreground work rejects store: true; an opted-in background graph accepts it, and the background engine keeps the result either way. conversation remains unsupported, and previous_response_id is reserved for interrupt resumes. The client owns the ordinary conversation input ledger. A proxy must not silently turn store: false into a stored response.

GET /v1/models is sufficient for ordinary graph selection. A client that uses LGOS descriptions, feature discovery, or runtime-settings forms also needs GET /v1/models/{model} and the namespaced lgos property, or a gateway catalog containing the equivalent metadata. The demo's LiteLLM integration reads that extension from native /model/info after model sync. Bifrost exposes it as a JSON string in additional_attributes.lgos on native /v1/models after catalog sync. Those extensions improve presentation but are not prerequisites for a standard Responses request.

Verify The Deployed Version

Test the actual image digest and configuration with the ordinary OpenAI SDK. At minimum cover non-streaming and streaming text, more than one commentary item, function continuation, file upload and input, standard errors, and disconnect cleanup. A proxy that returns valid final text can still be incompatible if it synthesizes a new stream or drops phase and call IDs.

Path Current demo result Intended use
Direct LGOS Full maintained contract Protocol reference and diagnostics
LiteLLM managed routing Native streaming, commentary, Files, file input, continuation, and successful Responses spend logging pass; error metadata is rewritten LiteLLM-selected UI inference and Files
Bifrost native /v1/models Complete LGOS metadata in additional_attributes.lgos after declarative sync Bifrost-selected UI discovery and settings
Bifrost normalized route Native Responses fields, Files, file input, commentary phase, continuation, store: false, and upstream error type and param pass; model-detail extensions are unavailable Bifrost-selected UI inference and Files

These results describe the bundled configuration. Exact image tags and digests are provided by DEMO_LITELLM_IMAGE in demo/.env.example for LiteLLM and demo/docker/apps/bifrost.yml for Bifrost.

The demo Docker guide documents the pinned LiteLLM image, native-stream configuration, and test command. The Bifrost guide records its exact remaining strict expected failures. Do not hide an upstream failure with a Chat fallback, custom proxy plugin, or LGOS-specific response field.

Background Lifecycle Matrix

The background path is tested separately because create-only routing is not enough. Retrieval and cancellation carry only the saved Response ID; a gateway must route that ID to any LGOS API replica using the same background engine.

Pinned path Background create, poll, cancel Constraint
Direct LGOS Pass Replicas must use the same background engine.
LiteLLM managed /v1 Pass Preserve LiteLLM's opaque client-visible ID; LGOS emits whole-second created_at values for LiteLLM parser compatibility.
Bifrost /openai/v1 Pass Send the create model's provider prefix, such as lgos, as the provider query parameter on retrieve and cancel; without it, Bifrost uses its built-in openai provider.

These are results for the pinned demo images and configurations, not promises about other gateway releases. Run just demo/test-background-gateway --editable against the exact route you plan to expose. Test with a newly constructed SDK client and restart the gateway between create and retrieve as part of deployment acceptance.

Routing

A gateway may expose provider-qualified model IDs such as lgos/simple-graph. Its native Responses route may require a documented provider selector or may translate the prefix itself. Keep that behavior in gateway configuration and send the unqualified graph name upstream. Clients connected directly to LGOS use the registered graph name unchanged.

LiteLLM's documented Responses endpoint uses concrete database-backed deployments with model_info.supports_native_streaming: true. The LGOS-owned sync command registers graph routing and metadata through LiteLLM's native management API. Run sync after graph changes.

The demo defaults to the pinned public homeserver-litellm image, configured to honor the deployment capability and preserve text deltas and commentary through managed /v1/responses. DEMO_LITELLM_IMAGE can select an alternative compatible image; see Docker Compose for configuration and validation. The integration suite checks native text deltas and commentary through both demo graph providers.

The maintained UIs use OPENAI_GATEWAY_TYPE=litellm|bifrost. With LiteLLM, their catalog readers use native /model/info, take descriptions and capabilities from model_info.lgos, and send model_name unchanged to managed /v1/responses. Files use managed /v1 with the configured litellm_proxy provider. No per-provider catalog routes or implicit model prefixes are needed. This keeps LiteLLM's routing, accounting, policy, retry, and fallback features available for inference. The error-normalization limitation still applies when LiteLLM is selected.

The bundled Bifrost custom providers enable only normalized OpenAI routes. The native Responses route preserves phase, multiple commentary items, file input, function continuation, and upstream error type and param. It still omits LGOS extensions from normalized model detail. The UIs read native /v1/models, whose synchronized additional_attributes.lgos contains the full extension. Responses send the provider-qualified catalog ID to native /openai/v1/responses, and Files use normalized /v1 with the dedicated lgos-files provider. No plugin or response adapter is required.

LiteLLM recovers managed deployment routing from its opaque Response ID. Bifrost routes model-less retrieve and cancel by the provider query parameter, which OpenAI SDKs send with extra_query.

Direct Chat Compatibility

Chat Completions remains available for direct compatibility clients running simple graphs. If such a client is placed behind a proxy, verify modern tool calls, metadata, usage, and stream cancellation separately. Complex features such as streaming status commentary, checkpointed persistence, selecting server tools, and interrupts use the Responses API.

Request Correlation

Forward a trusted gateway-generated X-Request-ID unchanged. At the first trusted edge, discard or replace a public client's value unless the deployment explicitly permits caller-controlled correlation data. LGOS validates only the header's shape; never use it for identity or authorization. See Production Logging and Request Correlation.