| Limit | Default | Counted |
|---|---|---|
| Requests | 40 a second | Every call that is admitted under the limits (below), whatever the model |
| Input tokens | 100,000 a second | The usage.input_tokens of each answered request and of every session frame, charged once it is answered |
Ceilings per plan#
Your plan sets a ceiling on these limits: the gateway applies the lower of the workspace's own limits and the plan's ceiling, in requests and in input tokens a second. The ceilings rise with the plan; an upgrade raises them at once. Over the ceiling the answer is the same 429 as below. A workspace with members and no plan serves no decisions (402 plan_required), so it has no ceiling to reach.
| Plan | Requests a second | Input tokens a second | API keys | Members |
|---|---|---|---|---|
| Starter | 5 | 50,000 | 3 | 2 |
| Pro | 25 | 250,000 | 10 | 5 |
| Business | 100 | 1,000,000 | 100 | 25 |
How they count#
- Per workspace. The limits belong to the workspace, and every key in it shares them. One model or another makes no difference: there is one count.
- Lower limits and a spend cap per key. A key can carry its own lower limits, in requests and input tokens a second, and a Decisions-only or Publishable key a daily spend cap (
429key_spend_cap_reachedpast it), so one device or one page cannot spend what the workspace can: see API keys. - Per second. Each limit is a budget for one second, a fixed window. The token limit refuses a request once the tokens already charged in the current second reach it; a request's tokens are charged after it is answered.
- What counts:
/v1/decide,/v1/uploads, opening a session and connecting its stream, and the Collections API calls, one request each. Session frames count their input tokens only.GET /v1/modelsis not limited. The usage routes,GET /v1/usageandGET /v1/keys/{key_id}/usage, do not count either: they have their own limit, 10 reads a second per workspace for the two together. - Your workspace's values are on the Rate limits tab of API keys.
Batches and sessions#
| Call | Counted as | Own limits |
|---|---|---|
| Batch calls: create, add, finalize, poll, results, cancel, list | Requests, like every call | None beyond the request rate |
| The requests inside a batch | Not counted: they run when there is room | 10 active batches and 100,000 requests in active batches (draft, queued and running) per key; over them, 429 batch_limit_exceeded |
| Opening a session | One request, and its fixed part's tokens | Open sessions per key, 4 unless we set yours otherwise; over it, 429 session_limit_exceeded, "This key already holds <n> open sessions, the limit." |
| Frames on an open session | Input tokens only: each frame's usage.input_tokens counts against your workspace's token rate, never its request rate | Over the token rate, a frame error rate_limit_error with retry_after_ms; the socket stays open. Per frame: text or JSON of at most the session's max_frame_tokens, or one image |
When you hit a limit#
The API answers 429 with Retry-After, the wait in whole seconds, and retry-after-ms, the same wait in milliseconds: the rest of the current second, so Retry-After is 1 and retry-after-ms 1 to 1000. Wait that long and retry the same request; nothing was charged for the refused one. Retrying sooner only earns another 429.
HTTP/1.1 429 Too Many RequestsRetry-After: 1retry-after-ms: 840x-request-id: req_3f6c1a9e0b2d4c7f8a1e5d9bcontent-type: application/json{"detail": {"error_type": "rate_limit_error", "message": "Rate limit reached. Retry after 1 s."}}When we are overloaded: 529#
A 529 is not your limit: it means the work already waiting leaves no room to answer your request in time, and you are told at once rather than after a timeout. It carries the same two headers: usually 1 second; while a model is starting, the rest of its boot window, up to 600 seconds; 30 seconds while its servers are unhealthy; 1 to 2 seconds when our key store is busy. Wait at least that long, add jitter, and back off further if it repeats. A 529 is not billed.
Staying under them#
- Bound concurrency when you fan out live calls (see Fan-out) instead of firing everything at once.
- Move work nothing waits on into a batch job: its requests do not count against your limits, and they cost less than live calls.
- Ask all questions about an input in one request: one request instead of many, and the state counted once.
- Trim states. Input tokens a second is usually the limit a backlog hits first.
- Give each service its own key with its own lower limits: the workspace's limits are shared, so a key's own cap keeps one service's backlog from using up the rest.
- Need a higher ceiling or reserved GPUs? Dedicated capacity has limits set to your traffic.