Demo Architecture¶

The demo is a complete deployment of langgraph-openai-serve (LGOS): two chat UIs, an AI gateway, two LGOS APIs, and the services they share. It is shaped like a production system on purpose. Every component is an independent application, and the request path between them is OpenAI-compatible HTTP and MCP, so you can replace a UI, the gateway, or an LGOS API without changing the others.

This page explains how the parts work together, from the outside in. For what happens inside one LGOS API process, see Package Architecture.

Three ideas shape the whole stack:

• One gateway in, one gateway out. Every UI request and every model call passes the gateway. It authenticates callers, owns the model catalog, and holds the only upstream provider key.
• Graphs are models. Each LGOS API publishes its LangGraph graphs as OpenAI models, so a UI selects a graph the same way it selects a model.
• Clients own conversations. The UIs store transcripts. LGOS stores only paused runs and the data a graph saves on purpose.

Components¶

flowchart TB
  uis["Chat UIs<br/>Chainlit, Open WebUI"]
  gateway["AI gateway<br/>LiteLLM or Bifrost"]

  subgraph lgos["LGOS APIs"]
    direction LR
    api["Demo API<br/>example graphs"]
    coding["Coding-agent API<br/>Codex"]
  end

  files["Files API<br/>S3-backed"]
  dbhub["DBHub<br/>read-only MCP"]
  upstream["Upstream model API<br/>OpenAI by default"]

  uis -->|"models, Responses, Files,<br/>speech, MCP"| gateway
  gateway -->|"graph requests"| api & coding
  api & coding -.->|"model calls"| gateway
  gateway -->|"uploads and downloads"| files
  gateway -->|"MCP tools"| dbhub
  gateway -->|"model calls<br/>and speech"| upstream
  api -->|"file reads and writes"| files

Solid arrows are client requests. Dotted arrows are the model calls a graph makes while it runs; they return to the same gateway as separate requests. The optional background worker and Hatchet are shown in Background Mock.

The optional OpenTelemetry overlay adds a collector for traces, metrics, and logs without changing these paths.

External Services¶

The stack relies on these services outside Compose:

How Requests Flow¶

sequenceDiagram
  participant UI as Chainlit or Open WebUI
  participant GW as AI gateway
  participant API as LGOS API
  participant LLM as Upstream model API

  UI->>GW: Responses request, model lgos/simple-graph
  GW->>API: same request, model simple-graph
  API->>API: run the graph on the conversation
  API->>GW: Chat Completions call, model openai/gpt-4.1-mini
  GW->>LLM: model call with the provider key
  LLM-->>API: tokens, through the gateway
  API-->>UI: Responses stream, through the gateway

The lgos/ prefix identifies graph models. Gateway routing maps each graph to its API without exposing the deployment name in the public model ID. The API runs the graph and streams standard Responses events: text, status updates, citations, and tool calls. When the graph needs an LLM, it calls the same gateway with an openai/* model ID, so the API never holds a provider key.

Feature flows build on this path and live with their graphs:

• File attachments: File Input
• Gateway MCP tools: PostgreSQL Through Native MCP
• Background Responses: Background Mock
• Human review: Interruptible Human Review

Model Catalog¶

The UIs learn about graphs from the gateway catalog, not from the APIs. At startup, the catalog sync copies each LGOS API's /v1/models metadata into the gateway's native catalog, and the UIs use it to show settings forms, enable attachments, and connect MCP tools. See LiteLLM Model Sync and Bifrost metadata; each gateway's routes are listed in Docker Compose.

State Ownership¶

Recovery behavior lives in Persistent Plot Agent and Interruptible Human Review.

From Demo To Production¶

Every component runs the way it would in production. The demo takes a few shortcuts that a real deployment replaces: