Archive every API request to AWS S3 with custom policies

Why you might need request archiving

The request log built into RequestRocket covers most operational needs — you can retrieve requests by proxy and time window, inspect individual records, and pull aggregate telemetry. But regulated industries often need more: a tamper-evident, long-lived archive of every API call with full request and response payloads, stored in infrastructure you control.

Use cases that require this level of archiving:

• SOC 2 / ISO 27001 compliance — demonstrating that access to sensitive APIs is fully logged.
• GDPR audit trails — proving which personal data was accessed, when, and by whom.
• Billing reconciliation — cross-referencing internal API call counts against vendor invoices.
• Forensic debugging — replaying historical requests to reproduce a reported issue.

The architecture

RequestRocket’s filter system evaluates response body and response status on the way back from the upstream API. A custom policy can:

1. Capture the incoming request path, method, and headers.
2. Capture the upstream response body and status.
3. Emit these to an S3-bound pipeline (via a webhook, Lambda, or streaming export).

The filter variables system extracts values from request headers, response headers, response body, and response status into named variables that can be used across filter operations.

Step 1: Set up a filter that captures response data

This filter applies to all requests on a proxy and uses variables to read key fields from the response, making them available for downstream archiving:

POST /clients/{clientId}/proxies/{proxyId}/filters
{
  "methods": ["GET", "POST", "PUT", "PATCH", "DELETE"],
  "variables": [
    {
      "name": "responseStatus",
      "source": "responseStatus",
      "key": "status"
    },
    {
      "name": "correlationId",
      "source": "headers",
      "key": "X-Correlation-ID"
    },
    {
      "name": "callerCredential",
      "source": "headers",
      "key": "X-RR-Credential-ID"
    }
  ],
  "operations": [],
  "notes": "Archive policy: capture correlation ID and response status for audit log"
}

Step 2: Redact before archiving

If archiving full payloads, strip secrets and PII first. A destroy operation on sensitive fields ensures the archived version never contains credentials or regulated personal data:

POST /clients/{clientId}/proxies/{proxyId}/filters
{
  "methods": ["GET", "POST", "PUT", "PATCH", "DELETE"],
  "operations": [
    {
      "effect": "destroy",
      "jsonPath": {
        "pattern": "\\.(password|token|apiKey|secret|ssn|creditCard|card_number|cvv|taxId)$",
        "flags": "i"
      },
      "notes": "Redact secrets and PII before payload reaches archive"
    }
  ],
  "notes": "Pre-archive redaction policy"
}

This filter runs before the response is returned to the caller. The redacted payload is what the caller receives and what gets archived.

Step 3: Use request logs as the archiving source

For most compliance use cases, the RequestRocket request log is sufficient as the primary audit record. Each log entry is queryable by time window:

GET /clients/{clientId}/proxies/{proxyId}/requests
  ?processedAfter=2026-04-01T00:00:00Z
  &processedBefore=2026-04-30T23:59:59Z

To build a scheduled archive job that exports to S3:

1. Call the requests endpoint with a time window covering the last period (hourly, daily).
2. For each request, retrieve the full record: GET /clients/{clientId}/proxies/{proxyId}/requests/{requestId}.
3. Write each record as a JSON object to your S3 bucket under a path like s3://your-bucket/api-audit/{clientId}/{proxyId}/{date}/{requestId}.json.

The resulting S3 objects are durable, queryable via Athena, and can be retained under your own lifecycle policies.

Step 4: Scope archiving to sensitive proxies

You probably don’t need to archive every proxy — internal health checks and low-sensitivity APIs don’t require the same treatment as a payment or HR data API. Apply archive-specific filters only to the proxies that require them:

POST /clients/{clientId}/proxies
{
  "proxyName": "payroll-api-archived",
  "proxyRegion": "us-east-1",
  "proxyProxyCredentialId": "<internal-service-key-id>",
  "proxyTargetId": "<payroll-api-target-id>",
  "proxyTargetCredentialId": "<payroll-api-credential-id>",
  "proxyDefaultRuleEffect": "deny",
  "proxyAlias": "payroll-api"
}

Then attach the redaction and variable-capture filters specifically to payroll-api-archived’s proxyId.

Compliance considerations

A few practical points for regulated use cases:

• Retention period — configure S3 lifecycle policies to retain records for your required period (typically 7 years for financial records, 6 years for GDPR in some jurisdictions).
• Tamper evidence — enable S3 Object Lock with COMPLIANCE mode to prevent deletion before the retention period expires.
• Encryption at rest — use S3 SSE-KMS with a customer-managed key so you control access to the encryption material.
• Access logging — enable S3 server access logging so you have a log of who accessed the archive itself.

What the request log already handles

For teams without a requirement to archive to their own S3 bucket, RequestRocket’s built-in request log provides a queryable audit trail without additional infrastructure. The processedAfter / processedBefore query parameters on the requests endpoint let compliance teams pull records for any time window.

Next steps

Whether you need full S3 archiving or just a reliable audit trail, the gateway provides a single, consistent record of every API call. Read the RequestRocket documentation on filters and request logs, or start for free.