How AI Features in Your Product works
Data flow
- 1A signed-in customer user calls an AI feature. Every request carries the user's tenant ID.
- 2Customer identity is checked on every call, never inferred from the prompt.
- 3Per-tenant settings check opt-outs and retention before retrieval, so an opted-out customer's data is never fetched or sent.
- 4Retrieval filters by tenant before ranking, and your own caches are keyed by tenant, so one customer's documents never reach another's answer.
- 5Prompts and retrieved context go through the gateway, where guardrails flag likely injection, PII and unsafe output. Least-privilege actions are the real control.
- 6Model calls use a Track A model with no training on customer data, and ZDR or a BAA where needed.
- 7Any action the feature takes goes through the toolshed with the customer's own permissions.
- 8Logs are kept per tenant for audit and abuse detection.
- 9Every release is red-teamed for prompt injection, cross-tenant leaks and unsafe output.
Build or buy
Buy + glueWhat you can buy for each box in the diagram, the open source option, and the part you still build yourself.
| Component | Buy | Open source | You build | Maturity |
|---|---|---|---|---|
| Customer identity | Your existing app auth: Auth0, WorkOS, Clerk or Amazon Cognito | Keycloak, Ory | Take the tenant ID from the session or token and pass it through every service, including async jobs. Never take it from the prompt. | GA |
| Tenant data and vectors | Pinecone (namespace per tenant, Standard or Enterprise), Weaviate Cloud (multi-tenancy), Amazon OpenSearch Serverless | pgvector with Postgres row-level security, Weaviate, Qdrant, OpenSearch | One retrieval layer that can't run without a tenant ID. Ban direct vector DB calls in code review or lint. | GA |
| Tenant AI settings | No dedicated product. Feature flag tools like LaunchDarkly or Statsig can hold per-tenant toggles. | OpenFeature | Opt-out checked before retrieval and the model call. Retention jobs that delete across logs, vectors, caches and the provider. | No product |
| Gateway and guardrails | Amazon Bedrock Guardrails, Azure AI Content Safety Prompt Shields, Check Point AI Guardrails (formerly Lakera), Portkey, Cloudflare AI Gateway | LiteLLM, NeMo Guardrails, Microsoft Presidio, LLM Guard, Llama Guard | Tenant ID in gateway metadata. Thresholds tuned per feature. A block, redact or flag policy. | GA |
| Model | Anthropic, OpenAI, Amazon Bedrock, Azure OpenAI, Google Vertex AI, with no-training terms plus zero data retention or a BAA | · | Confirm in the contract and config that each feature you use is covered by zero data retention. | GA |
| Actions via the toolshed | MCP gateways and identity providers that support on-behalf-of token exchange, such as Auth0 | MCP SDKs, agentgateway | Tools called with the end user's own token, scoped to their tenant. Writes need user confirmation. | Preview |
| Per-tenant AI logs | Datadog LLM Observability, Langfuse Cloud | Langfuse, OpenTelemetry GenAI conventions | Tenant tag on every span. Access control by tenant, per-tenant export and delete, abuse rules. | GA |
| Red team and evals | Promptfoo Enterprise (now owned by OpenAI), Check Point AI red teaming | Promptfoo, Garak, PyRIT | Two-tenant canary tests in CI. Tenant A plants markers that tenant B must never see. | GA |
Rollout steps
| # | Step | Who | Effort |
|---|---|---|---|
| 1 | Get no-training terms in writing, request zero data retention and sign a BAA if you need one | Founder or legal, with security lead | 1-2 weeks elapsed |
| 2 | Build one tenant-scoped retrieval layer using namespaces or row-level security | Backend engineer | 2-4 weeks |
| 3 | Key every cache you run by tenant, or turn shared caches off | Backend engineer | 0.5-1 week |
| 4 | Check opt-out and retention settings before retrieval and before any model call | Backend engineer | 2-4 weeks |
| 5 | Put a gateway in front of all model calls with prompt-attack and PII filters | Platform engineer | 2-3 weeks |
| 6 | Tag every log and trace with tenant ID and restrict who can read them | Platform engineer | 1-3 weeks |
| 7 | Add cross-tenant canary tests and an injection suite to CI | Security engineer | 2-3 weeks |
| 8 | Run an outside AI pentest before launch | Outside consultant | 1-3 weeks |
| 9 | Tune output classifiers once real traffic exists | Security engineer | 1-2 weeks |
What you still build
- A retrieval wrapper that won't run without a tenant ID
- Tenant-keyed semantic, response and embedding caches
- Opt-out enforcement that runs before retrieval, not after
- Retention deletes that reach the vector DB, gateway logs, observability tool and caches
- Tenant-tagged logs with per-tenant access, export and delete
- Two-tenant canary tests that run in CI before each release
Gotchas
- Provider prompt caches can't be keyed per tenant. Anthropic isolates by workspace on its own API, and Bedrock and Google Cloud isolate only by organization. Key your own caches by tenant.
- Injection classifiers catch direct jailbreaks but miss much of the injection hidden in uploaded files. Limit what actions can do and confirm writes.
- Zero data retention needs provider approval and may not cover every feature. Check each one.
- Logs hold prompts, so a shared log view is a cross-tenant leak.
- Generic jailbreak suites won't find your tenancy bugs. You need your own canary tests.
- Guardrail cost grows with traffic, input and output length, and the number of policies you turn on.
Common use cases
- In-product chat, search and summaries
- Customer support bots
- AI agents inside your app
- Document and data analysis for customers
Key risks
- One customer's data in another customer's answer
- Prompt injection from customer-uploaded content
- Customer data used for training without consent
- Shared caches or logs visible across tenants
Fast path
- KEEPTenant ID enforced on every retrieval and on your own caches
- KEEPNo training on customer data, in contract and config
- SIMPLIFYPer-tenant opt-out before per-feature controls
- DEFEROutput classifiers. Keep regex PII and citation checks from day one
- KEEPCross-tenant tests in CI before each release