An API gateway, an AI (or LLM) gateway and a secrets manager answer different questions. The first usually protects APIs you host. The second routes and guards calls to models. The third stores credentials. None of them, on its own, governs the outbound call an application or agent makes to a third-party API you do not host.
This article is a layer map. No layer is obsolete. The useful question is which layer does what, and how they sit together.
The architecture review that named five products and one gap
An enterprise architect opened a review with five boxes already drawn: Okta, Kong, a WAF, Vault, and an LLM gateway the platform team had bought to cap token spend. The proposed agent would read the CRM and draft a renewal email. Every box had an owner. None of the boxes sat on the HTTPS call from the agent to Salesforce. Kong would see the organisation’s own APIs. The LLM gateway would see the model hop. Vault would hand out the CRM token. Okta would issue the user token the agent borrowed. The CRM call itself had no policy and no dedicated audit.
The meeting did not need a sixth logo for its own sake. It needed a name for the missing hop.
What does each layer control?
Identity says who. The inbound gateway and WAF protect what you publish. The AI gateway governs model traffic. The secrets manager stores keys. Outbound API governance decides what each call to a vendor API may do and return.
| Layer | Traffic direction | What it controls | What it does not | Typical owner |
|---|---|---|---|---|
| Identity provider | Identity plane | Who the human or workload is; groups, roles, SSO, MFA | The shape of a call to a third-party API, or the fields that return | IAM / identity |
| API gateway (inbound) | Inbound, to APIs you host | Authn/authz, rate limits, schemas on your APIs | Calls your apps and agents make to vendors you do not host | Platform / API |
| AI / LLM gateway | Agent or app to model | Model routing, model credentials, token spend, sometimes prompt/output guardrails | Vendor API calls and vendor response bodies (unless it is also the outbound proxy) | AI platform |
| WAF | Inbound, to your edge | Known web attacks against published sites and APIs | Outbound request shape, outbound credentials, vendor responses | Security / network |
| Secrets manager | Control plane | Storage, rotation, checkout of secrets | Whether the caller that checked out a key uses it within scope | Security / platform |
| RequestRocket | Outbound, to APIs you do not host | Per-call scope, credential isolation, response filters, meters, audit | Identity proof, inbound protection of your APIs, model routing | Security / platform (complements the rows above) |
Identity tells you who; RequestRocket enforces what each outbound call may do. That sentence is the whole complementary claim.
For the category definition see outbound API governance. For the agent credential pattern see API access without the vendor key. For how identity programmes stop at the call, see non-human identity for AI agents.
Three architectures that use the layers together
These are patterns, not customer stories.
1. Human app, third-party API
A back-office app calls Xero. Users sign in with the identity provider. The app’s own API sits behind the inbound gateway and WAF. The Xero OAuth grant lives in the secrets manager or on the outbound proxy’s target credential. The outbound proxy issues the app a scoped caller credential, allows invoice reads, and logs each call. The inbound gateway never sees Xero. It does not need to.
2. Agent plus model plus CRM
An assistant calls a model through the AI gateway and calls Salesforce through the outbound proxy. The AI gateway caps tokens and can redact prompt text. The outbound proxy holds the Salesforce token, allows GET on one object, and strips fields the model does not need. Vault can still store the Salesforce secret if policy requires a vault. The proxy is what stops the agent holding that secret.
3. MCP client in a regulated entity
A desktop MCP client talks to an approved MCP server. The server does not hold vendor keys. It calls the outbound proxy. The identity provider still authenticates the human who launched the client. The WAF still protects the entity’s own portals. CPS 230 and CPS 234 still apply to the vendor relationship. The proxy is where third- and fourth-party use of the API becomes a per-call record.
How to choose without a takedown
Start from the hop you are trying to govern.
- If the question is “who is this person or workload?”, that is the identity provider.
- If the question is “who may call our API?”, that is the inbound gateway, often with a WAF in front.
- If the question is “which model, at what cost, with what prompt policy?”, that is the AI gateway.
- If the question is “where do we store the vendor key?”, that is the secrets manager.
- If the question is “what may this caller do to an API we do not host, and what may come back?”, that is outbound API governance.
Buying a stronger inbound gateway does not close the last question. Neither does a new model router. The layers compose.
A common procurement mistake is to ask one vendor to absorb the row below them. The inbound-gateway RFP grows an “AI” chapter. The LLM-gateway RFP grows an “API connector” chapter. The identity RFP grows “agentic authorization.” Each of those expansions can be real. They still fail the test in the table if the product never sits on the HTTPS call to an API you do not host, never keeps the vendor key off the agent, or never filters the response. Write the test into the RFP. Do not write a category slogan.
Another mistake is ownership. The AI platform team buys the model gateway. The API team owns Kong. Security owns the WAF and the vault. The seam sits in the white space on the diagram. Name an owner for outbound calls to third-party APIs before the first agent reaches production. The complementary architecture is easier to draw than to staff.
OWASP API10:2023, Unsafe Consumption of APIs, is the inbound-gateway community naming the outbound problem: teams trust third-party payloads more than user input. An AI agent inherits that trust and then writes the result into a context window. The layer that consumes the API is the layer that needs the grant.
If the review still has only five boxes, add the hop as a named row, assign an owner, and keep the other five. That is the whole of this comparison.
Where RequestRocket fits
RequestRocket is the outbound row in the table. It holds vendor credentials, applies rules and filters, meters callers and records the audit. It does not replace Okta, Kong, a WAF, an LLM gateway or Vault. It sits on the hop those products leave to the application.
FAQ
Is an AI gateway an API gateway for third-party APIs?
Usually not. Most AI gateways sit on the model hop. A third-party CRM call only goes through them if you have forced that traffic through the same proxy, which most teams have not.
Can our inbound gateway do egress if we add a new cluster?
Some gateways can be pointed at outbound traffic. The test is whether policy is bound to each caller and request shape for APIs you do not host, whether the vendor key stays off the agent, and whether you filter the response. The product name is secondary.
Do we still need a secrets manager if the proxy holds keys?
Yes, if your policy requires a vault, HSM or existing rotation workflow. The proxy consumes the secret. It does not make the vault obsolete.
Where does a service-mesh egress gateway fit?
It is a network control: which hostnames a workload may reach. It is complementary. It does not replace per-call method, path and field rules.
Is RequestRocket an AI gateway?
No. It provides no models and does not route prompts. It governs outbound API and MCP calls.
Who typically owns the outbound layer?
Often the same platform or security team that owns the inbound gateway, working with IAM. The owner of the AI gateway is sometimes a different team. Write the RACI down so the seam does not sit between them.
Book a walkthrough of RequestRocket
Sources
- NIST CSF 2.0 (layer mapping only; Functions described in the pillar)
- OWASP API10:2023 Unsafe Consumption of APIs