2 variants
A3.1
Self-hosted
You run the gateway in your cloud. Most control, most to operate.Data flow
- 1Apps and dev containers call one OpenAI-compatible endpoint with a virtual key.
- 2The gateway maps each key to a user or team through SSO groups for quotas and chargeback.
- 3Guardrails check app and chat prompts for PII, secrets and injection before anything leaves. Coding-agent traffic is too large to scan economically.
- 4The gateway pulls real provider keys from the secrets manager. Teams never see them.
- 5Requests route to the right provider with fallback, rate limits and budgets.
- 6Every request logs user, model, tokens, cost and guardrail verdict.
Self-hosted: build or buy
Buy + glueWhat you can buy for each box in the diagram, the open source option, and the part you still build yourself.
| Component | Buy | Open source | You build | Maturity |
|---|---|---|---|---|
| Virtual keys for apps and dev containers | LiteLLM Enterprise or Kong Gateway Enterprise | LiteLLM OSS virtual keys | Key issuance flow and putting keys into dev containers | GA |
| AI Gateway | LiteLLM Enterprise (annual, sales quote) or Kong Gateway Enterprise self-hosted | LiteLLM OSS, Envoy AI Gateway 1.0, Kong OSS ai-proxy | HA deploy with Postgres and Redis, version pinning, upgrade process | GA |
| SSO and group claims | Okta, Entra or Google. LiteLLM SSO beyond 5 users needs Enterprise | OIDC plugins for Kong or Envoy | Sync IdP groups to gateway teams, budgets and model access | GA |
| Guardrails and DLP | Amazon Bedrock Guardrails standalone API, Lakera (LiteLLM Enterprise), Kong AI Sanitizer and Semantic Prompt Guard (Enterprise or Plus add-on) | Microsoft Presidio, LLM Guard | Choose which traffic to inspect, tune policies, handle false positives | GA |
| Secrets manager for provider keys | AWS Secrets Manager, HashiCorp Vault. LiteLLM native integration needs Enterprise | External Secrets Operator, Secrets Store CSI driver | Rotation runbook and revoking old direct keys | GA |
| Logs to SIEM | Your existing SIEM. LiteLLM per-team logging and GCS/Azure export need Enterprise | LiteLLM callbacks to OpenTelemetry, S3, Datadog | Metadata-only SIEM stream, prompt bodies in a bucket with retention | GA |
| Model providers | OpenAI, Anthropic and Google Gemini APIs | · | Fallback order and per-team model allowlists | GA |
Rollout steps
| # | Step | Who | Effort |
|---|---|---|---|
| 1 | Deploy LiteLLM from a pinned, cosign-verified GHCR image with Postgres, Redis and two or more replicas | Platform engineer | 3-5 days |
| 2 | Inject provider keys from your secrets manager and revoke direct keys | Platform engineer | 2-3 days |
| 3 | Connect SSO, map IdP groups to teams, set budgets and rate limits | Security engineer | 3-5 days |
| 4 | Issue per-developer keys and set base URLs in dev containers | Platform engineer | 2-4 days |
| 5 | Send metadata to the SIEM and keep prompt bodies in a bucket with 30-90 day retention, or off | Security engineer | 3-5 days |
| 6 | Add guardrails to app and chat traffic and tune them on real prompts | Security engineer | 1-3 weeks |
| 7 | Set up pinned upgrades, an egress allowlist and bypass detection | Platform engineer | Ongoing, a few hours a week |
What you still build
- Sync from IdP groups to gateway teams and budgets
- Routing rules by data class, such as sending regulated data only to BAA-covered endpoints
- Fail-safe behavior: fail closed on regulated routes, fail open with alerts for dev traffic
- Removing prompt text from logs before they reach the SIEM
- Detecting teams that bypass the gateway with direct provider keys
Gotchas
- LiteLLM's PyPI package was compromised on March 24, 2026 (versions 1.82.7 and 1.82.8). Deploy pinned, signed GHCR images, never unpinned pip installs.
- LiteLLM OSS has no native secrets manager integration and caps SSO at 5 users. Inject keys as env vars or buy Enterprise.
- Guardrail APIs bill by text volume, so scanning coding-agent traffic with very large prompts gets expensive and noisy fast.
- Gateway logs hold your most sensitive prompts. Treat the log store as a new sensitive data store.
- A gateway outage stops all AI traffic, so run two or more replicas and plan a break-glass path.
A3.2
Managed or aggregator
A gateway vendor runs it. Live in a day, one more vendor in the path. An aggregator such as OpenRouter is the same pattern, but it also sells you the model access.Data flow
- 1Apps and dev containers call the vendor's endpoint with a per-team gateway token.
- 2SSO, usually on enterprise tiers, controls who can manage keys, routes and budgets in the gateway console.
- 3Vendor guardrails inspect prompts if enabled.
- 4Provider keys live with the vendor, or stay yours with bring-your-own-key. An aggregator such as OpenRouter replaces your provider contracts with its own: one key and one bill for hundreds of models, and BYOK changes the billing, not the data path.
- 5The gateway handles routing, caching, fallback and rate limits. An aggregator also picks among upstream providers with different retention terms, so require ZDR and a provider allowlist on the account. OpenRouter publishes no BAA, so keep PHI off it.
- 6Logs sit in the vendor's store. Export them to your bucket or SIEM.
Managed or aggregator: build or buy
Mostly buyWhat you can buy for each box in the diagram, the open source option, and the part you still build yourself.
Two ways to run it
1Managed gateway
Teams that already have contracts with OpenAI, Anthropic or Google and want one control point in front of them.
- Your provider keys through BYOK, so your BAA and data terms still apply
- SSO, per-team tokens and budgets
- Guardrails and DLP on app traffic
- Log export to your bucket or SIEM
- Covers
- Keys, budgets, routing, logging and guardrails across the providers you already use.
- Misses
- Model breadth beyond your own contracts. The gateway vendor is one more subprocessor.
- Effort
- Days
- Needs
- Cloudflare AI Gateway, Prisma AIRS AI Gateway, Vercel AI Gateway or Kong Konnect
2Aggregator such as OpenRouter
Product teams that want hundreds of models, provider fallback and one bill, for data that isn't regulated.
- ZDR and a provider allowlist set on the account
- Key in your secrets manager
- Usage export to your bucket or SIEM
- Covers
- Model breadth and failover with almost no setup.
- Misses
- Your own provider contracts: a third party sees every prompt, even with BYOK, and there's no BAA. Upstream retention terms vary.
- Effort
- A day
- Needs
- OpenRouter Standard, Business or Enterprise
Components
| Component | Buy | Open source | You build | Maturity |
|---|---|---|---|---|
| Managed gateway | Cloudflare AI Gateway, Prisma AIRS AI Gateway (formerly Portkey), Vercel AI Gateway, Kong Konnect Plus or Enterprise | Portkey gateway (open source) | Per-team token issuance | GA |
| SSO to the gateway console | Okta, Entra or Google. Prisma AIRS AI Gateway SSO needs Enterprise | · | · | GA |
| Vendor guardrails | Cloudflare DLP (full profiles need Zero Trust DLP), Cloudflare Guardrails, Prisma AIRS AI Gateway guardrails (Production and up), Kong AI plugins | · | Policy tuning and choosing which traffic to inspect | GA |
| Vendor log store and export | Cloudflare Workers Logs and Logpush, Vercel Trace Drains (Pro and Enterprise), Prisma AIRS AI Gateway logs | · | Export to your bucket, prompt removal, SIEM field mapping | GA |
| Provider keys | Cloudflare Secrets Store BYOK, Vercel BYOK, Prisma AIRS AI Gateway virtual keys | · | Rotation runbook | GA |
| ZDR and provider allowlist | Vercel team-wide ZDR and provider allowlist (Pro and Enterprise) | · | · | GA |
| Model providers | OpenAI, Anthropic and Google Gemini APIs | · | · | GA |
| Aggregator, such as OpenRouter | OpenRouter Standard, Business or Enterprise: one key, routing, billing and an activity log. ZDR routing on all plans, data-policy routing on Standard and up, Managed Policy Enforcement on Enterprise. | · | Provider allowlist review and a usage export to your bucket or SIEM | GA |
Rollout steps
| # | Step | Who | Effort |
|---|---|---|---|
| 1 | Confirm the vendor's BAA, DPA, subprocessors and log retention before any regulated data flows | Security engineer | 1-2 days |
| 2 | Create the gateway, add your provider keys and issue per-team tokens | Platform engineer | 1-2 days |
| 3 | Point apps and dev containers at the gateway and revoke direct keys | Platform engineer | 2-3 days |
| 4 | Set up SSO, budgets, alerts and log export to your bucket | Security engineer | 2-4 days |
| 5 | Add guardrails to app traffic once the basics are stable | Security engineer | 1-2 weeks |
| 6 | If you use an aggregator, turn on ZDR and a provider allowlist on the account and get the DPA signed before production traffic | Security engineer | 1-2 days |
What you still build
- Routing by data class, since managed gateways route by model, not by what's in the prompt
- Break-glass plan for a vendor outage that doesn't quietly bring back direct keys
- Removing prompt text from exported logs and mapping fields for the SIEM
- Any gating by data sensitivity in front of an aggregator, since it can't tell which prompts are sensitive
Gotchas
- Portkey is now Prisma AIRS AI Gateway under Palo Alto Networks. Expect tier and roadmap changes.
- SSO, HIPAA BAAs and VPC hosting are on enterprise tiers, which are quote only.
- The gateway vendor becomes a subprocessor, and a business associate if prompts contain PHI. Your model provider's BAA doesn't cover them.
- Vercel retries failed BYOK requests on its own credentials, so some traffic can run outside your provider contract.
- Vendor log stores keep prompts unless you turn body logging off.
- Cloudflare gateways created on or after September 24, 2026 log to Workers Logs, so check retention and pricing.
- OpenRouter BYOK still sends every prompt through OpenRouter. It changes billing, not the data path.
- We found no public BAA from OpenRouter, so don't send PHI through it.
- Long-tail providers behind an aggregator have different retention terms. Enforce ZDR on the account, not per request.
Common use cases
- Internal chat and assistants
- Shared API access across teams
- Cost attribution and chargeback
- Model evaluations and provider failover
Key risks
- Gateway outage stops all AI traffic
- Gateway logs hold sensitive prompts
- Bypass if teams keep direct keys
- Aggregators add a subprocessor with variable retention and no BAA
- Routing through a gateway can fall outside a vendor's BAA
Fast path
- SIMPLIFYManaged gateway instead of self-hosting
- DEFERPrompt DLP. Use ZDR terms and a data policy first
- SIMPLIFYBucket with retention before SIEM
- KEEPProvider keys only in a secrets manager or BYOK
- KEEPPinned, signed gateway images, never unpinned pip installs
- KEEPZDR and provider allowlist on any aggregator