2 variants
A4.1
Managed cloud AI
The cloud runs the model inside its compliance boundary. You get IAM identity and private networking.Data flow
- 1Users sign in to an internal app.
- 2The workload gets short-lived credentials from its IAM role. No API keys exist.
- 3Retrieval pulls sensitive context from data stores inside the VPC.
- 4Model calls go through a PrivateLink endpoint, never the public internet. In-region inference profiles keep data in region.
- 5Provider guardrails filter each invocation inside the cloud's compliance boundary (BAA, FedRAMP, region).
- 6Invocation logging is off by default and misses Bedrock's newer mantle endpoint, so turn it on, and keep apps on bedrock-runtime or log in the app. With CloudTrail, both land in your account and stream to the SIEM.
Managed cloud AI: build or buy
Buy + glueWhat you can buy for each box in the diagram, the open source option, and the part you still build yourself.
| Component | Buy | Open source | You build | Maturity |
|---|---|---|---|---|
| Workload identity | AWS IAM roles, Entra managed identities, Google Cloud service accounts | · | Least-privilege policies per app, plus an IAM condition requiring a guardrail | GA |
| Workload and vector DB | EKS or ECS, OpenSearch Serverless, Aurora pgvector, Azure AI Search, AlloyDB | pgvector, Qdrant | Per-user access control on retrieved documents | GA |
| Private networking | AWS PrivateLink for bedrock-runtime and bedrock-mantle, Azure Private Endpoints, Google Private Service Connect | · | Private DNS, no NAT on AI subnets, SCPs | GA |
| Invocation logs and CloudTrail | Bedrock model invocation logging to S3 or CloudWatch, Azure API Management LLM logging, Agent Platform request-response logging | · | Turn logging on, encrypt, set retention, send metadata to SIEM | GA |
| Provider guardrails | Amazon Bedrock Guardrails, Azure AI Content Safety, Google Model Armor | · | Policy tuning per app | GA |
| Managed models | Amazon Bedrock, Microsoft Foundry, Gemini Enterprise Agent Platform (formerly Vertex AI) | · | Pin regions and inference profiles | GA |
| Separate account for PHI workloads | AWS Organizations, Azure subscriptions, Google Cloud projects | · | SCPs for approved regions and models, and a deny on disabling logging | GA |
Rollout steps
| # | Step | Who | Effort |
|---|---|---|---|
| 1 | Create a dedicated AI account with SCPs for regions, models and logging, and confirm BAA coverage | Cloud security engineer | 2-4 days |
| 2 | Create VPC endpoints for bedrock-runtime, bedrock-mantle, S3, KMS and logs, with private DNS and no NAT | Platform engineer | 2-3 days |
| 3 | Turn on invocation logging to an encrypted S3 bucket and keep apps on bedrock-runtime, or log in the app | Security engineer | 2-3 days |
| 4 | Give each workload its own IAM role and build retrieval with per-user filtering | Application engineer | 1-3 weeks |
| 5 | Add provider guardrails and require them in IAM policy | Security engineer | 1-2 weeks |
| 6 | Stream log metadata to the SIEM and keep prompt bodies in a bucket with retention | Security engineer | 3-5 days |
What you still build
- Per-user access control on documents pulled into prompts
- App-level logging for calls native logs miss, like bedrock-mantle on AWS or Azure without API Management
- Removing prompt text before logs reach the SIEM
- SCPs and policy checks that stop logging, region or model drift
- A gateway in front if apps span more than one cloud
Gotchas
- Bedrock invocation logging is off by default. Turn it on before launch.
- Bedrock invocation logging doesn't capture calls on the bedrock-mantle endpoint, which AWS recommends for new apps. Use bedrock-runtime or log in your app.
- CloudTrail records who called a model, not what they sent. Prompts only appear in invocation logs.
- Azure diagnostic logs don't include prompt or response text. You need API Management LLM logging.
- Google's request-response logging is opt-in, sampled and writes to BigQuery.
- Global cross-region inference can process data outside your region. Use in-region or geo profiles for residency.
- Catalogs differ by cloud. Bedrock now has OpenAI and Anthropic models, but Gemini is only on Google, and features can trail the first-party APIs.
- Azure Foundry needs three private DNS zones, and a missing zone silently falls back to the public endpoint.
A4.2
Self-hosted open weights
You run the models on GPUs you own. Nothing leaves, and you own everything.Data flow
- 1Users and batch jobs call one internal, OpenAI-compatible endpoint.
- 2The gateway handles SSO, quotas and routing to the right model size.
- 3New open-weight models are imported deliberately, never pulled at runtime.
- 4The registry scans, signs and pins versions before GPU nodes load them. Scanners catch malicious files, not backdoored weights, so evaluate each new model.
- 5Outbound internet is denied, so prompts and outputs never leave the cluster. Offline settings let the stack start without it.
- 6Gateway and inference logs stay in-cluster and forward to the SIEM.
Self-hosted open weights: build or buy
Mostly customWhat you can buy for each box in the diagram, the open source option, and the part you still build yourself.
| Component | Buy | Open source | You build | Maturity |
|---|---|---|---|---|
| Inference servers | NVIDIA NIM with NVIDIA AI Enterprise. GPUs from AWS, Azure, Google Cloud or neoclouds like CoreWeave, Lambda, Nebius | vLLM, SGLang, KServe, llm-d | Capacity planning, autoscaling, model evals | GA |
| Internal gateway | LiteLLM Enterprise, Kong Gateway Enterprise | LiteLLM OSS, Envoy AI Gateway 1.0 | SSO, quotas, routing by model size and data class | GA |
| Model registry with scanning and signing | Prisma AIRS Model Security (formerly Protect AI Guardian), HiddenLayer | ModelScan, picklescan, OpenSSF model-signing, MLflow, Harbor | Quarantine import, scan, sign, pin by digest, verify at deploy | GA |
| Egress deny | Cloud security groups, no NAT gateway | Cilium, Calico network policies | Allowlist for internal registry and patch mirrors | GA |
| Cluster logs | Your existing SIEM | OpenTelemetry, Fluent Bit | Join gateway and inference logs by request ID, remove prompt text | GA |
| Model source | Hugging Face, model vendors | · | Deliberate import process, never pulled at runtime | GA |
Rollout steps
| # | Step | Who | Effort |
|---|---|---|---|
| 1 | Pick models with evals on your real tasks and load-test GPU sizing | ML engineer | 1-2 weeks |
| 2 | Build the GPU node pool, autoscaling and subnets with no egress | Platform engineer | 2-3 weeks |
| 3 | Build the import pipeline: quarantine, scan, require safetensors, review config code, sign, pin | Security engineer | 2-3 weeks |
| 4 | Stand up the internal gateway with SSO, quotas and routing | Platform engineer | 1 week |
| 5 | Forward gateway and inference logs to the SIEM and set up a CVE watch for vLLM, CUDA and drivers | Security engineer | 1 week |
| 6 | Run monthly upgrades, model refreshes and capacity reviews | Platform engineer | Ongoing, 0.5-3 FTE depending on size |
What you still build
- Model import pipeline with scanning, signing, digest pinning and signature checks at deploy
- Behavioral evals that look for backdoors and quality regressions
- Routing by model size and data class
- GPU capacity planning and autoscaling
- CVE response process for the serving stack
Gotchas
- GPU cost and the people to run it dominate. Compare against your managed token bill at the same volume.
- vLLM had several 2026 code execution CVEs tied to remote model code. Run 0.28.0 or later with trust_remote_code off.
- Scanners catch malicious pickle files, not backdoored weights. You still need evals.
- With all egress denied, set offline flags like HF_HUB_OFFLINE and LiteLLM's local cost map, or startup can break.
- Prisma AIRS Model Security scans only public Hugging Face repos directly. Download private models and scan them locally.
- Plan two to four months to stand this up, not two weeks.
Common use cases
- PHI and PII document processing
- Regulated customer support bots
- Source code and IP analysis
- Residency-bound or air-gapped workloads
Key risks
- Model catalogs differ by cloud
- Invocation logging off by default, with gaps on newer endpoints
- Malicious or backdoored open weights
- You own GPU capacity and every serving-stack CVE
Fast path
- KEEPIAM roles, no static keys
- KEEPSeparate cloud account for PHI AI workloads
- KEEPPrivate networking or egress deny
- KEEPInvocation logging on, plus app-level logs
- KEEPRegistry with scanning and pinning (self-hosted)
- DEFERProvider guardrails at launch
- SIMPLIFYLogs to bucket before SIEM