Performance & limits
A consolidated reference for technical buyers evaluating NULLRUN's
operational characteristics. Every value below is anchored to backend
code with file:line citations; rows marked Not measured are honest
gaps — we surface what we don't know rather than quote a number we
haven't earned.
This page is the load-bearing complement to the Circuit breaker and Error handling concept pages: those explain the what and why, this page enumerates the bounds.
Hot-path latency
The gate hot path is /api/v1/gate → Redis Lua (reserve_v3) →
Postgres audit outbox. Wall-clock budgets are layered; the inner cap
is the binding one.
| Stage | Bound | Source |
|---|---|---|
Outer HTTP request (/api/v1/*) |
30 s (env REQUEST_TIMEOUT_SECS, default 30) |
backend/src/proxy/server.rs:99-106,326,362 — tower-http TimeoutLayer |
/api/v1/gate inner hard timeout |
5 s (GATE_HANDLER_HARD_TIMEOUT) |
backend/src/proxy/http/gate/gate.rs:636-662 — on expiry returns 402 BUDGET_REDIS_UNAVAILABLE |
| Gate orchestrator inner hard timeout | 3 s (ORCHESTRATOR_HARD_TIMEOUT) |
backend/src/proxy/http/gate/internal.rs:2265-2266 — on expiry returns 402 + redis_unavailable_inc("gate_timeout") |
/api/v1/execute inner timeout |
None — relies on outer 30 s + Redis breaker fail-fast (<100 ms once tripped) | backend/src/proxy/http/gate/execute.rs:66-386 |
/api/v1/track inner timeout |
None — relies on outer 30 s | backend/src/proxy/handlers.rs:5234 |
reserve_v3.lua SCAN loop cap |
256 iterations (MAX_SCAN_ITERATIONS, raised from 16 by NR-073) |
backend/src/redis/scripts/reserve_v3.lua:508 |
reserve_v3.lua SCAN COUNT per iter |
1 000 (256 k slot visits worst case) | reserve_v3.lua:521 |
RESERVED_SCANNED_PARTIAL return |
typed {-1, "RESERVED_SCANNED_PARTIAL", reserved, projected} — fail-CLOSED 402, never silent truncation |
reserve_v3.lua:558-570 |
Redis pool wait_timeout |
5 000 ms (env REDIS_POOL_TIMEOUT_MS) |
backend/src/redis/pool.rs:153,321 |
Redis pool create_timeout |
10 s (env REDIS_ACQUIRE_TIMEOUT, clamp 1–60 s) |
backend/src/redis/pool.rs:154,323 |
Redis pool max_size |
32 (env REDIS_POOL_MAX_SIZE) |
backend/src/redis/pool.rs:152,320 |
Postgres pool acquire_timeout |
10 s | backend/src/db/mod.rs:1641 |
Postgres statement_timeout (per-connection) |
30 000 ms (env NULLRUN_DB_STATEMENT_TIMEOUT_MS) |
backend/src/db/mod.rs GUC setup |
Healthy-path budget
When Redis is healthy and no /check style fan-out is required:
- Network RTT to Redis: sub-millisecond within a region.
reserve_v3.luasingle atomic step: <5 ms typical, <50 ms at p99.- Orchestrator evaluation: <2 ms typical.
- Total
/api/v1/gateserver-side: <10 ms typical, <60 ms p99.
The 3 s orchestrator cap is the binding inner budget; the 5 s gate
handler cap is a one-above safety net. The 30 s outer TimeoutLayer
should never fire on a healthy Redis.
First-burst (breaker not yet tripped)
If Redis is reachable but slow, the worst case before the breaker trips is:
- 1 acquire × 10 s + 1 wait × 5 s + 1 SCAN cap × ~50 ms ≈ up to 15 s
- On the 5th consecutive failure, the breaker trips and subsequent
calls short-circuit sub-100 ms returning
BUDGET_REDIS_UNAVAILABLE(see Redis failure below).
Not measured
- Real Redis p99 RTT at production scale for the worst-case
reserve_v3.luapath (256 SCAN + 256 MGET ≈ 512 commands). - Worst-case first-request latency on a Redis partition before the breaker trips.
Behaviour when a service is unavailable
Every enforcement-path failure mode below is fail-CLOSED unless explicitly noted. The default posture is "do not let the agent proceed when an enforcement signal is missing".
Redis failure
The Redis circuit breaker has three profiles, picked by call-site:
| Profile | failure_threshold |
recovery_timeout_ms |
half_open_max_probes |
Used for |
|---|---|---|---|---|
| default | 5 | 30 000 (30 s) | 1 | Generic Redis paths |
critical() |
3 | 60 000 (60 s) | 1 | Budget enforcement on the gate hot path (FailClosed) |
high_traffic() |
20 | 10 000 (10 s) | 3 | Buffered / StaleCache / analytics paths |
Source: backend/src/redis/breaker/config.rs:29-80.
The breaker short-circuits in sub-100 ms once OPEN (execute_with_degraded
returns CircuitOpen for FailClosed/Buffered profiles).
Wire error_code |
HTTP | Triggered by | Source |
|---|---|---|---|
BUDGET_REDIS_UNAVAILABLE |
402 | reserve_v3 Lua failure / inline Redis fail on gate path | error_codes.rs:616; gate.rs:657; internal.rs:4241-4267 |
RATE_LIMIT_REDIS_UNAVAILABLE |
503 | per-org aggregate bucket Redis fail (fail-CLOSED) | error_codes.rs:739; orchestrator.rs:671-682,713 |
IDEMPOTENCY_REDIS_UNAVAILABLE |
503 | IdempotencyStore Redis fail (fail-CLOSED) |
error_codes.rs:741; gate.rs:484,546,563,585,603 |
WORKFLOW_DEPTH_LOOKUP_FAILED |
503 | ADR-036 cycle-walk Redis fail | error_codes.rs:754 |
INVOKE_PERSIST_FAILED |
503 | ADR-036 invoke persist fail | error_codes.rs:755 |
REDIS_UNAVAILABLE (inline) |
varied | inline pre-check on /track idempotency |
handlers.rs:5318; orchestrator.rs:1265 |
Retry behaviour: no automatic Redis retry in reservation/consume/idempotency handlers — failures surface directly to the breaker. The breaker is the retry mechanism.
Lua return-code → HTTP mapping (exhaustive):
| Lua return | Outcome | error_code |
HTTP |
|---|---|---|---|
HARD_BUDGET_EXCEEDED |
Hard block | BUDGET_HARD_BLOCKED |
402 |
ORG_BUDGET_EXCEEDED |
Org ceiling | BUDGET_ORG_CEILING_BLOCKED |
402 |
WORKFLOW_BUDGET_EXCEEDED |
Workflow | BUDGET_WORKFLOW_BLOCKED |
402 |
SOFT_OVERDRAFT_CAP |
Overdraft cap hit | BUDGET_OVERDRAFT_EXCEEDED |
402 |
RESERVED_CAP_EXCEEDED |
Anti-DoS reserved | BUDGET_ANTI_DOS_RESERVED_CAP |
402 |
RESERVED_SCANNED_PARTIAL |
NR-015 partial scan | RESERVED_SCANNED_PARTIAL |
402 |
BUDGET_EXCEEDED (consume-side) |
Consume-side | BUDGET_EXCEEDED |
429 + Retry-After: 1 |
CONSUME_OVERBUDGET |
Consume overshoot | CONSUME_OVERBUDGET |
422 |
RESERVATION_NOT_FOUND |
TTL expired / orphan | RESERVATION_NOT_FOUND |
503 |
POLICY_UNCONFIGURED |
Invariant break | POLICY_UNCONFIGURED |
500 |
Postgres failure
| Setting | Value | Source |
|---|---|---|
Pool max_connections |
(num_cpus::get().min(12) * 2 + 10) (≈25–34 on 12-core); env DATABASE_MAX_CONNECTIONS |
backend/src/db/mod.rs:1637-1641 |
Pool idle_timeout |
600 s | same |
Pool max_lifetime |
1 800 s (DNS / PgBouncer rollover protection) | same |
idle_in_transaction_session_timeout |
60 000 ms | same |
| DB circuit breaker profile | mirrors Redis default (5 / 30 s) | backend/src/db_breaker.rs |
| Breaker short-circuit on OPEN | <100 ms returns CircuitError::CircuitOpen |
same |
/track outbox path (the only enforcement-side PG write)
The audit-row INSERT is best-effort and off the request path:
- Wrapper:
PostgresCircuitBreaker::global()withFailureMode::Degraded(backend/src/proxy/http/gate/internal.rs:4029). - On INSERT error:
tracing::warn!+postgres_metrics.record_connection_error() - continue — the gate response is unchanged. The audit row is not on the critical path.
- Drain location:
backend/src/outbox/audit_drain.rs, spawned via fast ticker atmain.rs:3212. - Drain tick interval: 5 s default (env
OUTBOX_AUDIT_DRAIN_INTERVAL_SECONDS, clamp 2–300). - Per-tick batch size: 100.
- Per-tick timeout: 30 s.
- Per-row backoff: 1 / 2 / 4 / 8 / 16 s exponential (CLAUDE.md §15).
MAX_CONSECUTIVE_ERRORSper tick: 5 (tick break on persistent DB errors).- DLQ at
retry_count >= 5: hard ceiling. mark_immediate_dead_letterfor permanent failures: v3.76 P0-3.- Retention once DLQ'd: 30 days (
OUTBOX_DLQ_RETENTION_DAYS,backend/src/retention.rs:40).
Other outbox partitions
| Partition | max_attempts |
Backoff base |
|---|---|---|
| governance | 3 | 30 s × 2^attempt + jitter |
| notification | 5 | 30 s × 2^attempt + jitter |
| analytics | 1 (fire-and-forget) | — |
Upstream LLM / provider failure
The provider module is sparse — only OpenAIProvider is registered
(backend/src/proxy/provider/mod.rs:264-269; Anthropic / Google / Azure
types are documented but commented out, and /api/v1/proxy* is not
wired into routes.rs).
| Thing | Value | Source |
|---|---|---|
OpenAI reqwest::Client timeout |
Not configured — no .timeout(), .connect_timeout(), .read_timeout(), .pool_max_idle_per_host, .tcp_keepalive |
backend/src/proxy/provider/openai.rs:16,25 |
| Provider retry | None — one .send() per call, immediate return |
backend/src/proxy/provider/openai.rs:122-126,144,194,226 |
| HTTP status on provider failure | bare 502 BAD_GATEWAY — no error_code, no Retry-After, no JSON envelope |
backend/src/proxy/http/proxy.rs:109-112 |
OpenAI 429 parse |
ProviderError::RateLimited(60) — the 60 s hint is dropped on the floor (never emitted as Retry-After) |
backend/src/proxy/provider/openai.rs:130-141 |
| Streaming failure shape | governance_events carries control_decision: e.to_string() but HTTP status stays 200 |
backend/src/proxy/http/proxy.rs:300,340-349 |
This is the area with the largest gap between SDK promise and
backend reality. Practical implication: a hung OpenAI socket stalls
until the outer 30 s TimeoutLayer fires; there is no per-call
connect timeout, no retry, and no machine-readable error envelope.
Geo / sanctions failure
| Setting | Value | Source |
|---|---|---|
SANCTIONED jurisdictions |
RU, IR, KP, SY, CU, BY, VE, MM, AF | backend/src/proxy/middleware/geo_block.rs:64-74 |
HIGH_RISK_NO_SERVICE jurisdictions |
EU-27 + EEA + UK + CH + CN + IN | geo_block.rs:76-86 |
BlockSanctioned arm |
403 + {error:"service_unavailable_in_jurisdiction", message, fortress_reason:"sanctions"} + headers X-Fortress-Block-Country + X-Fortress-Block-Reason: sanctions |
geo_block.rs:714-758 |
BlockHighRisk arm |
403 (or 503 if GeoIP DB unavailable) + same shape | geo_block.rs:759-806 |
| GeoIP DB missing / unreadable | All ingress rejected (503) — fail-CLOSED by design | geo_block.rs:33-36,138-149,279-281 |
NULLRUN_GEOBLOCK_DISABLED |
dev-only fail-OPEN escape (logs WARN) | geo_block.rs:457-466,585-608 |
| Sanctions SDN CSV missing | 6-entry hand-curated fallback (fail-OPEN with WARN) | backend/src/proxy/middleware/sanctions.rs:185-221 |
NULLRUN_SANCTIONS_SCREENING_DISABLED |
operator kill-switch (fail-OPEN) | sanctions.rs:322-330 |
See Geo restrictions and Sanctions screening for the full compliance contract.
Timeouts and integration limits
HTTP / WebSocket / SSE
| Surface | Setting | Value | Source |
|---|---|---|---|
/api/v1/* outer request |
tower-http TimeoutLayer |
30 s | proxy/server.rs:99-106 |
HMAC re-read cap (MAX_BODY_SIZE) |
body cap before HMAC verify | 10 MiB | proxy/middleware/hmac_verify.rs:73 |
| WebSocket server ping interval | Message::Ping(vec![]) |
30 s | proxy/http/ws_control.rs:983-984 |
| WebSocket HMAC replay window | WS_HMAC_MAX_AGE_SECONDS |
300 s | ws_control.rs:37,1094 |
| WebSocket idle disconnect | None — closed only on Close frame / stream error / send failure | — | ws_control.rs:935-948 |
| WebSocket per-org connection cap | None (SSE has MAX_SSE_PER_ORG=5) |
— | — |
| WebSocket max frame / message size | None configured — axum defaults apply | — | — |
| SSE max connection duration | hard cap | 24 h | proxy/http/stream.rs:54 |
| SSE JSON heartbeat cadence | informational heartbeat | 30 s | stream.rs:78,421-422 |
SSE protocol-level keepalive (axum KeepAlive) |
text "ping" |
15 s | stream.rs:508-512 |
| SSE per-org concurrent cap | MAX_SSE_PER_ORG |
5 | proxy/http/sse_limiter.rs:44 |
| EventBus default capacity | bounded queue | 4 096 events (overflow at 3 072 / 75%) | proxy/http/event_bus.rs:881-887 |
| Slow-consumer signal | server emits WsMessage::ResyncRequired to force SDK reconnect |
— | ws_control.rs:967-974 |
OAuth / webhook clients (cold path)
| Client | Timeout | Source |
|---|---|---|
| GitHub OAuth | .timeout(10s).connect_timeout(5s) |
auth/github.rs:102-104,149-151,190-192,237-239,288-290 |
| Google OAuth | .timeout(10s).connect_timeout(5s) |
auth/google.rs:314-316,372-374,416-418,470-472 |
| SSO | .timeout(10s).connect_timeout(5s) |
auth/sso.rs:423-424 |
| Cron egress (litellm pricing) | .timeout(10s).connect_timeout(5s) |
cron.rs:1272-1274 |
Slack chat.postMessage |
.timeout(10s) (no separate connect timeout) |
main.rs:1510-1513 |
HMAC policy
| Setting | Value | Source |
|---|---|---|
NULLRUN_HMAC_REQUIRED default |
false (warns at runtime) |
config.rs:166 |
NULLRUN_HMAC_MAX_AGE_SECS default |
300 s | config.rs:171 |
NULLRUN_CONSUME_RACE_WINDOW_SECONDS default |
2 s, clamped [0..30] |
config.rs:222-226 |
| HMAC-verified SDK paths | ["/api/v1/check", "/api/v1/execute", "/api/v1/gate", "/api/v1/track", "/api/v1/track/batch"] |
proxy/middleware/hmac_verify.rs:138-144 |
| Gateway signing key min length | 32 bytes (env NULLRUN_GATEWAY_SIGNING_KEY) |
config.rs:38-42 |
Webhook signature replay windows
| Webhook | Algorithm | Replay window | Source |
|---|---|---|---|
| Slack Events | HMAC-SHA256 over v0:{ts}:{raw_body}, header X-Slack-Signature: v0=<hex> |
300 s | slack_oauth.rs:820-823,1295-1328; replay :1319 |
| Slack OAuth state | TTL | 600 s | slack_oauth.rs:71 STATE_TTL_SECS |
| Slack install ticket | TTL | 300 s | slack_oauth.rs:80 INSTALL_TICKET_TTL_SECS |
| Polar | Stripe-shaped t=<unix>,v1=<hex> over timestamp.payload, HMAC-SHA256, raw bytes |
300 s | billing/polar.rs:21,160-200 WEBHOOK_SIGNATURE_MAX_AGE_SECS |
| Polar sandbox | is_sandbox() == true skips verification |
— | billing/polar.rs:155-157 |
| Stripe | Not in code — Polar is the sole payment provider | — | — |
Hard limits and caps
Body / payload caps
| Cap | Value | Source |
|---|---|---|
| HTTP request body cap | 10 MiB (10 * 1024 * 1024) |
proxy/server.rs:363 RequestBodyLimitLayer::new(...) |
| MCP discovery SSE chunk | 65 536 bytes (64 KiB) | proxy/http/mcp/discovery_probe.rs:498,772 |
Validation caps (validation_constants.toml — single source of truth, generated by build.rs)
| Constant | Value | Source |
|---|---|---|
MAX_WORKFLOW_NAME |
80 | validation_constants.toml:17 |
MAX_POLICY_NAME |
80 | :18 |
MAX_API_KEY_NAME |
80 | :19 |
MAX_ORG_NAME |
100 | :20 |
MAX_ALERT_RULE_NAME |
80 | :21 |
MAX_KILL_REASON |
500 | :24 |
MAX_SUPPORT_MESSAGE |
5 000 | :25 |
MAX_DESCRIPTION |
2 000 | :26 |
MAX_JUSTIFICATION |
500 | :27 |
MAX_EMAIL_LEN |
254 (RFC 5321) | :30 |
MIN_PASSWORD / MAX_PASSWORD |
12 / 256 (NIST 800-63B) | :31-32 |
MIN_TWO_FACTOR_CODE / MAX_TWO_FACTOR_CODE |
6 / 6 | :33-34 |
MIN_ORG_SLUG_LEN / MAX_ORG_SLUG_LEN |
2 / 32 | :37-38 |
MAX_WEBHOOK_URL_LEN |
2 048 | :39 |
MAX_CALLBACK_URL_LEN |
2 048 | :40 |
MAX_FILTER_INPUT |
100 | :43 |
MAX_POLICY_PATTERN_BYTES (ToolBlock pattern) |
4 096 (4 KiB) | :46 |
MAX_WORKFLOW_DEPTH (sub-workflow chain, ADR-036) |
8 | :57; gate at orchestrator.rs:4228 → 422 WORKFLOW_DEPTH_EXCEEDED |
MAX_DISPLAY_NAME (XSS-guard) |
255 | validation.rs:214 |
DEFAULT_MAX_RATE_LIMIT_RPM (per-policy ceiling) |
1 000 000 (env NULLRUN_POLICY_MAX_RATE_LIMIT_RPM) |
validation.rs:223 |
Approval-rule limits
| Setting | Value | Source |
|---|---|---|
expires_in_seconds default (DB) |
300 s | db/mod.rs:8069 |
Service clamp [min..max] |
30..=3 600 (1 min – 1 h) | approval_rule_service.rs:55-56 |
priority bounds |
[0..=1 000] (i16) |
approval_rule_service.rs:49-50 |
action_label max length |
200 | :60 |
VALID_RISK_LEVELS |
["LOW","MEDIUM","HIGH"] |
:68 |
DEFAULT_RISK_LEVEL |
"MEDIUM" |
:73 |
| Per-org pending approvals cap | 50 (env NULLRUN_MAX_PENDING_APPROVALS_PER_ORG, range 1–1 000) |
redis/mod.rs:530 MAX_PENDING_APPROVALS_PER_ORG; 429 TOO_MANY_PENDING_APPROVALS on overflow |
| Envelope TTL floor / ceiling | 60 s / 86 400 s (24 h) | redis/mod.rs:515,521 |
APPROVAL_ENVELOPE_GRACE_SECONDS |
30 | redis/mod.rs:498 |
APPROVAL_CLOCK_SKEW_MARGIN_SECONDS |
5 | redis/mod.rs:509 |
| Approval-rule predicate JSON byte cap | Not in code — only typed schema validates shape | — |
Reservation / chain TTLs
| Constant | Value | Source |
|---|---|---|
DEFAULT_RESERVATION_TTL_SECONDS |
300 s (5 min) | cost/reservation.rs:48; cost/registry.rs:76 RESERVATION_TTL_SECONDS = 300 |
CHAIN_IDLE |
300 s | redis/mod.rs:553; consume_v3.lua:597 |
CHAIN_REGISTERED |
300 s | redis/mod.rs:559 |
max_chain_duration_seconds default |
3 600 s (1 h) | enforcement/unified_evaluator.rs:151,605,809; DB MAX_CHAIN_DURATION_SECONDS INTEGER NOT NULL DEFAULT 3600 (db/mod.rs:7525) |
period_ttl_seconds default |
3 600 × 24 × 30 = 30 days | enforcement/unified_evaluator.rs:152 |
EXECUTION_BINDING TTL |
24 × 3 600 s (24 h) | redis/mod.rs:549 |
| Heartbeat dedup marker | 35 s | redis/mod.rs:303-304 |
in_flight counter |
300 s | redis/mod.rs:317-322 |
Retention (backend/src/retention.rs — single source of truth)
| Constant | Value | Source |
|---|---|---|
OUTBOX_DLQ_RETENTION_DAYS |
30 | retention.rs:40 (CLAUDE.md §15) |
DECISION_HISTORY_FALLBACK_DAYS |
3 | retention.rs:59 |
METERING_IDEMPOTENCY_WINDOW_DAYS / _SECONDS |
30 / 2 592 000 | retention.rs:76,80 |
METERING_EVENT_LOG_TTL_DAYS / _SECONDS |
90 / 7 776 000 | retention.rs:94,98 |
INGESTION_DLQ_RESOLVED_RETENTION_DAYS |
7 | retention.rs:118 |
INGESTION_DLQ_MAX_REPLAY_ATTEMPTS |
5 | retention.rs:139 |
INGESTION_DLQ_EXHAUSTED_RETENTION_DAYS |
30 | retention.rs:158 |
BILLING_DEAD_LETTER_RETENTION_DAYS |
30 | retention.rs:173 |
Per-tier retention (decision_history_retention.rs:14-21):
| Plan | Decision history retention | Audit log |
|---|---|---|
| Lite | 3 d | immutable on Growth+ only |
| Starter | 7 d | immutable on Growth+ only |
| Growth | 30 d | immutable |
| Scale | 90 d | immutable |
| Enterprise | unlimited | immutable |
Rate limits
Edge / per-IP
| Layer | Default | Algorithm | Source |
|---|---|---|---|
| Per-IP edge | 60 RPM token bucket (env NULLRUN_IP_RATE_LIMIT_RPM) |
Token bucket; refill = max_rpm/60 | proxy/middleware/ip_rate_limit.rs:168-174 |
| Per-IP edge bypass | env NULLRUN_IP_RATE_LIMIT_DISABLED=1; bypass paths /health, /metrics, /internal/* |
— | same |
| Per-IP edge multi-pod | Redis-backed via NULLRUN_IP_RATE_LIMIT_REDIS_URL (fail-CLOSED) |
— | same |
| Per-IP edge response | 429 + Retry-After + X-RateLimit-Limit/Remaining |
— | same |
| Waitlist (high-risk jurisdictions) | 5 submissions/hour/IP (env NULLRUN_WAITLIST_PER_HOUR); window 3 600 s |
Counter | ip_rate_limit.rs:536-648; 429 WAITLIST_RATE_LIMITED |
| Auth endpoints | 5 req/min/IP (IpAuthRateLimiter::default) |
Token bucket (IP) + per-email counter | proxy/middleware/auth_rate_limit.rs:183-185,318-321 |
| Email lockout | 5 failures → 300 s lockout | — | same |
Per-org / per-key
| Layer | Default | Algorithm | Source |
|---|---|---|---|
| Per-org aggregate | plan-driven max_rpm from plans.limits.max_rps × 60; global fixed capacity 10 000 RPM; per-workspace fallback 1 000 RPM; refill 100 tok/s |
Token bucket (PlanAwareRateLimiter) |
proxy/middleware/rate_limit.rs:137-138,148-184 |
| Per-org fallback (unknown org) | 5 RPM — bug-class hazard, surfaces in tests | — | rate_limit.rs:199 |
Per-(org, api_key) Redis |
per-policy limit/ttl_secs (default max_calls_per_minute × 60); key shape ratelimit:org:{org_id}:key:{key_id} |
Fixed-window counter (atomic GET→INCR→EXPIRE in Lua; NOT sliding) | redis/scripts/check_rate_limit_v1.lua:16-17,50-52,58-80 |
| Per-key fail-OPEN on Redis error | warn-and-fall-through, no wire code | — | proxy/http/gate/orchestrator.rs (per-key branch) |
| Per-org aggregate fail-CLOSED on Redis error | 503 RATE_LIMIT_REDIS_UNAVAILABLE, retry_after_seconds: 60 |
— | orchestrator.rs:671-682,713 |
The per-org aggregate rate limit is fail-CLOSED on Redis error (returns 503), the per-key is fail-OPEN. This asymmetry is intentional: a per-key miss only affects one key, but a per-org miss would mask abuse.
Per-plan limits
Seeded in 002_plans.sql; unified budget
(admission/limit_checks.rs:1013-1076) shares policies +
approval_rules slots (drift-pinned by migration 332).
| Plan | workflows | tokens/hr | exec/mo | RPS | parallel | api_keys | seats | policies |
|---|---|---|---|---|---|---|---|---|
| Lite | 3 | 10 000 | 75 000 | 5 | 1 | 10 | 1 | 3 |
| Starter | 8 | 25 000 | 100 000 | 10 | 3 | 15 | 3 | 10 |
| Growth | 50 | 300 000 | 750 000 | 50 | 10 | 100 | 10 | 25 |
| Scale | 200 | unlimited | 2 000 000 | 300 | 50 | 350 | 75 | 150 |
| Enterprise | unlimited | unlimited | unlimited | 1 000 | 100 | unlimited | unlimited | unlimited |
Wire-contract essentials
The gate emits GateResponse over HTTP 200/402/422/429/503. Key
fields (backend/src/proxy/http/gate/internal.rs:551-734):
execution_id— server-minted UUIDv7. Client-supplied IDs are never honoured (ADR-003 — ownership-ambiguous billing + replay attack surface).projected_cost_cents— informational only. The only enforcement-readable field isremaining_budget_cents.cost_centson/track— server marks cost asprovisionaluntil reconciliation.retry_after_ms(top-level) — only present onRATE_LIMIT_EXCEEDED; milliseconds.details.{retry_after_seconds, retry_after_ms}— set on rate-limit and idempotency-unavailable responses.
No Retry-After HTTP header on the gate path. Body-only retry
hints. The header is emitted only by ip_rate_limit middleware and
by ApiError on non-gate paths (429 → 30 s, 503 → explicit
retry_after_seconds).
X-NULLRUN-PROTOCOL is required on
/api/v1/{gate,execute,track,track/batch,heartbeat,cancel,approvals/:id/consume}.
Current=4, MIN=2, MAX=4 (protocol.rs:67-71); rejected at middleware
protocol_version_middleware (protocol.rs:297-330).
Idempotency surfaces
| Endpoint | Mechanism | TTL | Outcome discriminator |
|---|---|---|---|
/api/v1/gate |
IdempotencyStore (atomic SETNX + Lua mutate); body field operation_id (NOT Idempotency-Key header) |
24 h (GATE_TTL_SECONDS, gate.rs:523) |
hit+match+Completed → 200 + idempotent_replay:true; hit+match+Pending → 409 IDEMPOTENCY_REDIS_UNAVAILABLE; hit+mismatch → 409 IDEMPOTENCY_KEY_MISMATCH; Redis down → 503 fail-CLOSED |
/api/v1/track |
IdempotencyStore reused; body field idempotency_key |
— | mismatch → 409 IDEMPOTENCY_KEY_MISMATCH; Completed → 200 + idempotent_replay:true; Pending → 409 IDEMPOTENCY_IN_FLIGHT + retry_after_ms:500; Failed → wipe via delete + fall through; Redis down → 503 |
/api/v1/cancel |
SETNX cancel:{execution_id} |
24 h | replay → 200 {already_canceled:true} |
/api/v1/heartbeat |
SETNX |
30 s | dedup |
/api/v1/approvals/:id/consume |
SQL-layer atomic UPDATE flipping APPROVED→CONSUMED (ADR-047) | — | 200 with status: consumed\|already_consumed\|not_approved |
No use of the IETF Idempotency-Key header convention. Drift from
any IETF-style clients — the header is silently ignored.
Retry / backoff defaults
| Component | Settings | Source |
|---|---|---|
| Slack alert delivery | MAX_ATTEMPTS=3, BASE_BACKOFF_MS=1_000, MAX_BACKOFF_MS=30_000 (pure exponential, no jitter) |
alert/providers.rs:184-186 |
| Litellm pricing cron | PRICING_RETRY_MAX_ATTEMPTS=3 |
cron.rs:998 |
| Audit outbox | per-row 5-step exponential 1/2/4/8/16 s | outbox/audit_drain.rs |
| Governance outbox | max_attempts=3, 30 s × 2^attempt + jitter |
outbox/partitions.rs |
| Notification outbox | max_attempts=5, 30 s × 2^attempt + jitter |
same |
| Analytics outbox | max_attempts=1 (fire-and-forget) |
same |
| Detector retry | retry.max_retries=5, retry.window_seconds=60 |
config.rs:665-666 |
No jitter in the audit-outbox or Slack-alert retry loops. This is a known limitation worth tracking if a thundering-herd pattern emerges.
Gaps (honest)
These are the things we either couldn't find in the code or didn't measure in production. We list them explicitly so technical buyers can ask the right questions during evaluation.
Not in code (verified absent)
- OpenAI
reqwest::Clienthas no timeout — no.timeout(),.connect_timeout(),.read_timeout(),.pool_max_idle_per_host,.tcp_keepalive(backend/src/proxy/provider/openai.rs:25). - Anthropic / Google / Azure providers — only
OpenAIProviderexists on disk; registration commented out (provider/mod.rs:264-269);/api/v1/proxy*not inroutes.rs. - Provider retry —
ProviderError::is_retryable()returns true forRateLimited/Unavailablebut no caller retries; one.send()then return. Retry-AfterHTTP header on/api/v1/proxy*502 — bare status, no header, no JSON envelope (proxy.rs:109-112).- OpenAI
429RateLimited(60)retry hint — hardcoded 60 s is dropped on the floor, never emitted asRetry-After. - No per-call Redis timeout on
Script::invoke_async/cmd().query_async— only 5 s connection timeout atConnectionManager::new; nowith_timeout(...)wrapper. - No automatic Redis retry in gate/track handlers.
- No jitter in any retry loop — Slack alerts (1 s → 30 s), pricing cron, audit outbox, governance outbox. Pure exponential only.
- No server-side tool-execution timeout on
/execute— SDK runs the tool locally; the server has no watchdog. - No client-initiated cancellation of in-flight
/gateor/execute— no DELETE/HEAD routes inroutes.rs:163-187;/cancelis the only de-facto abort (separate call). - No server-side deadline on
/executeindependent of/gate— onlyRedisCircuitBreakerfail-fast atexecute.rs:199-201,225. - WS idle timeout — 30 s ping is a probe, not an idle-disconnect policy; a silent client stays connected.
- WS per-org connection cap —
MAX_SSE_PER_ORG=5exists; WS has no equivalent cap. - WS max frame size / max message size — no
max_frame_size/max_message_sizeoverride; axum defaults apply. WS_PING_INTERVAL/WS_RECONNECT_MAXenv var — grep returned zero matches.- SQL-level
audit_outboxqueue depth cap — only per-row retry cap (5) then DLQ; no global cap. - Explicit semaphore / concurrency cap on
outbox/audit_drain— onlyMAX_CONSECUTIVE_ERRORS=5per tick break. MAX_KEYS_SCANconstant in Lua — onlyMAX_SCAN_ITERATIONS(256) exists;consume_v3.luahas no scan / time-bound.- Approval-rule predicate JSON byte cap —
Option<serde_json::Value>only, no byte validator. - Stripe webhook — no
stripe*.rsfiles; Polar is the sole payment provider. X-RateLimit-Limit/X-RateLimit-Remaining/X-RateLimit-Reseton gate path — grep returned zero matches inproxy/http/gate/; emitted only byip_rate_limitmiddleware (per-IP edge).Retry-AfterHTTP header on gate — gate uses bodyretry_after_ms/retry_after_secondsonly.MISSING_TOKEN/INVALID_KEY/EXPIRED_KEYGateErrorCodevariants — registry has onlyAPI_KEY_REVOKEDfor auth-class; missing/invalid/expired flow through legacyApiErrorenvelope./trackIDEMPOTENCY_IN_FLIGHTis emitted inline inhandlers.rs:5472but NOT inGateErrorCode::all()— typed enum has 61 variants (error_codes.rs:all()assertslen() == 61), drift from inline emission.
Not measured (would require load test)
- Real Redis p99 RTT at production scale for
reserve_v3.luaworst-case 256 SCAN + 256 MGET. - Worst-case first-request latency on Redis partition before the breaker trips (5 consecutive × 10 s acquire ≈ 50 s before short-circuit).
- OpenAI upstream hung-socket behavior in practice — outer
TimeoutLayer(30 s) is the only bound; no per-callconnect_timeout. - WS reconnect storms — no server-side cap; SDK could reconnect indefinitely under split-brain.
- Geo-block DB load latency at p99 —
mmdblookup is in-request hot path; only the 503 fail-CLOSED branch is verified. - PG pool contention under
/trackbursts — bounded ingestion queue hasQUEUE_CAPACITY=10_000,HARD=9_000(drop oldest above),MAX=15_000(drop at cap), but actual Postgres write throughput is not measured.
Drift between code and documentation
contracts/openapi.yamlGateResponse(lines 938-1033) is missing 8 wire fields present in the Rust struct (internal.rs:551-734):approval_id,approval_timeout_seconds,approval_expires_at,execution_id,retry_after_ms,idempotent_replay,operation_id,decision_context,details.contracts/openapi.yaml:1-16declares YAML canonical; Rust is wire-true./checkdeprecation — returns 410 GONE (check.rs:46-74) but YAML may still document it as active.
How to verify
Any of these numbers can drift between this page and the actual code. When that happens, the code wins. To re-derive:
# From NULLRUN repo root
git grep -nE 'TimeoutLayer|TimeoutDuration' backend/src/proxy/server.rs
git grep -nE 'failure_threshold|recovery_timeout_ms' backend/src/redis/breaker/config.rs
git grep -nE 'TTL|TIMEOUT|DURATION|interval' backend/src/retention.rs
git grep -nE 'MAX_' backend/src/redis/mod.rs
git grep -nE 'CAP|PATTERN|MAX_' validation_constants.toml
See also
- Circuit breaker — what the agent sees when a failure occurs
- Error handling — wire code → exception mapping
- Budgets — budget reserve/consume semantics
- API keys — HMAC, rotation, drain
- Control plane — WebSocket keepalive
- HTTP API → Capabilities —
protocol version +
/healthmin/max - Compliance — geo / sanctions posture