The AI API billing problem
AI API costs have a unique failure mode: they’re invisible until they’re not. A well-behaved agent calling GPT-4 for 1,000 requests a day is manageable. The same agent, with a retry loop bug introduced in a Friday deploy, hitting the API 100,000 times over a weekend is a different story — and you’ll find out on Monday when the vendor invoice arrives.
This isn’t hypothetical. Token-based billing means costs scale with both call volume and payload size. Both are controllable at the gateway layer before they become billing events.
Lever 1: Default request rate limits
The client configuration sets rate limit defaults — the values that new proxies in your account are created with, unless you specify different limits at creation time. For an AI cost management account, set conservative defaults that reflect your overall vendor budget:
PUT /clients/{clientId}/configuration
{
"configuration": {
"limits": {
"requestsPerMinute": 60,
"requestsPerDay": 2000
}
}
}New proxies are created with these values unless you supply different limits at creation time. Once created, each proxy’s rate limits are its own — changing the account defaults does not retroactively affect existing proxies.
Lever 2: Per-proxy rate limits and retry caps
Each proxy can define its own rate limits independently. For cost-sensitive AI proxies, set the limits at the proxy level to give each integration its own budget:
PUT /clients/{clientId}/proxies/{proxyId}
{
"proxyMaxRetries": 3,
"proxyMaxBackoff": 15,
"proxyMaxRequestsPerMinute": 50,
"proxyMaxRequestsPerDay": 5000
}proxyMaxRequestsPerMinute caps burst traffic — a short-lived retry loop can’t hammer the API more than 60 times per minute. proxyMaxRequestsPerDay caps cumulative spend — once 2,000 requests have been processed in a day, the gateway returns a rate-limit error until the window resets.
proxyMaxRetries: 2 means a failed request gets at most two retries — three total attempts. proxyMaxBackoff: 10 caps the exponential backoff at 10 seconds, preventing the total retry window from extending indefinitely. A call that fails all three attempts returns the error to the caller, which is the correct behaviour for cost management.
Lever 3: Strip large fields from responses before they reach the agent
AI APIs often return metadata alongside the content you actually need. The full OpenAI response includes the model name, usage statistics (prompt_tokens, completion_tokens, total_tokens), finish reason, system fingerprint, and other fields. Your agent might only need the message.content.
A filter that strips everything except the content field from an OpenAI-style chat completion response:
POST /clients/{clientId}/proxies/{proxyId}/filters
{
"methods": ["POST"],
"requestPath": {
"path": { "pattern": "^/v1/chat/completions$" },
"presence": "must_exist"
},
"operations": [
{
"effect": "destroy",
"jsonPath": { "pattern": "^(model|system_fingerprint|service_tier)$" },
"notes": "Strip metadata fields not needed by agent"
},
{
"effect": "destroy",
"jsonPath": { "pattern": "^choices\\.\\d+\\.(logprobs|finish_reason|index)$" },
"notes": "Strip per-choice metadata"
}
],
"notes": "Response trimming: return only message content to agent"
}This doesn’t reduce billing (you’ve already made the call) but it reduces the size of the context window the response feeds into downstream model calls — which does reduce the token count for any chained LLM call that processes the result.
Lever 4: Allow only the endpoints you need
Every AI provider exposes endpoints beyond chat completions: embeddings, image generation, audio transcription, fine-tuning, assistants, files, and more. An agent that should only be doing text generation has no business calling /v1/images/generations or /v1/audio/transcriptions.
A proxyDefaultRuleEffect: "deny" proxy with explicit allow rules limits the callable surface:
POST /clients/{clientId}/proxies/{proxyId}/rules
{
"effect": "allow",
"methods": ["POST"],
"path": {
"path": { "pattern": "^/v1/chat/completions$" },
"presence": "must_exist"
},
"priority": 10,
"notes": "Permit chat completions only"
}POST /clients/{clientId}/proxies/{proxyId}/rules
{
"effect": "allow",
"methods": ["POST"],
"path": {
"path": { "pattern": "^/v1/embeddings$" },
"presence": "must_exist"
},
"priority": 9,
"notes": "Permit embeddings"
}Everything else — image generation, audio, fine-tuning — is denied by default. If an agent starts calling unexpected endpoints (potentially due to a prompt injection or a model hallucinating an API call), the gateway stops it.
Lever 5: Monitor spend signals via telemetry
The telemetry API gives you a real-time view of call volume trends without waiting for vendor invoices:
GET /clients/{clientId}/proxies/{proxyId}/telemetry
?interval=hour
&limit=48If call volume spikes significantly above the normal hourly average, you have an early warning signal — before it becomes a billing line item. Set up an alert in your observability stack on the hourly request count for your AI proxy to catch anomalies within the same hour they occur.
What the gateway can’t do
A few limitations worth naming:
- Token count — the gateway doesn’t inspect or count the tokens in request or response bodies. Token-level cost caps require application logic or a purpose-built AI cost management tool.
- Model selection — which model is requested is part of the request body; the gateway doesn’t change it (though an authorization rule can deny requests that specify a disallowed model).
- Vendor-side caching — this is outside the gateway’s control; vendor-native caching (where available) is complementary.
Next steps
Rate limits, retry caps, response filtering, and endpoint restrictions can all be configured in minutes on any existing proxy. Read the RequestRocket configuration and filter documentation or start for free.
FAQ
Can a gateway cut AI spend without changing model code?
Yes. Caps on request rate, narrower allow-lists, and filters that drop large unused fields all reduce calls and tokens before the vendor bills them.
What will the gateway not do?
It will not pick a cheaper model for you, and it will not see spend that never passed through the proxy.
How do response filters reduce cost?
If the agent does not need a large field, destroy it before the body is returned. Later prompts do not resend that text.
Where should the first cap go?
On the proxy that serves the agent, tighter than the account default, so one workflow cannot exhaust the vendor quota.