# NullRun Docs -- Full Text Corpus > Concatenated Markdown of every page in nav order, for > agents that prefer a single-fetch corpus. See llms.txt > for the structured index. Regenerate this file when > content changes via infra/scripts/build-llms-full.ps1. Source: https://github.com/nullrunio/nullrun-docs Generated against main branch. --- # Source: https://docs.nullrun.io/ # File: docs/index.md --- title: NullRun — runtime decision layer for tool-using AI agents description: Runtime decision layer for tool-using AI agents. Gates every tool and model call through allow/block/require_approval before execution. home: true ---

Runtime decision layer for tool-using AI agents

Before your agent executes a supported tool or model call, the SDK asks the gate.
allow, block, or require_approval — backed by tool patterns, budgets, rate limits, and human approvals.

Get started → Get an API key How the gate works
NullRun dashboard home showing the workflow control panel. NullRun dashboard home showing the workflow control panel.

How it fits together

```mermaid flowchart LR Agent["Your agent
(Python SDK)"] -->|"@protect"| Gate Gate -->|"budget pre-flight
policy fetch"| Gateway["NullRun gateway"] Gateway -->|"plan limit
rate limit
ToolBlock check"| Decision{"allow?"} Decision -->|"yes"| Body["wrapped function runs"] Decision -->|"no"| Block["raise NullRunBlockedException"] Gateway -.->|"control plane
(WebSocket)"| Kill["kill / pause
from dashboard"] ```

What you get out of the box

Budget gate
Set a per-workflow cap in cents. The SDK asks the gateway "any budget left?" before every @protect call — no round-trip cost when the answer is "yes". Hard blocks on overrun; soft mode allows a bounded overrun when an active chain is present.
Action-bound approvals
Operator approves sensitive calls via typed predicates (money_amount / tool_parameters). Every approval is bound to the exact action payload via a SHA-256 action_digest — the grant is refused if the SDK then executes a different amount or different arguments.
Real-time kill / pause
A WebSocket control plane pushes killed / paused to every connected SDK. WorkflowKilledInterrupt reaches the top of the agent loop, not a swallowed except Exception.
ToolBlock policy
Server-side glob-pattern rules (mcp://payments/refund*, bash, db.drop) decide which canonical tool names are allowed. Always Hard: fails closed on transport error, regardless of the budget's enforcement_mode.
Auto-instrumentation
nullrun.init() patches OpenAI, Anthropic, LangGraph, OpenAI Agents, Mistral, Gemini, Cohere, Bedrock, LlamaIndex, CrewAI, and AutoGen — cost tracking without @protect.
Audit chain
Every gate decision (allow / block / require_approval) is recorded in an append-only audit log with a tamper-evident hash chain. The chain is recompute-verifiable on demand via the audit-log verify endpoint.

Wire it up in 30 lines

```mermaid sequenceDiagram participant U as Your code participant SDK as nullrun SDK participant G as NullRun gateway participant DB as Dashboard U->>SDK: from nullrun import init, protect U->>SDK: init(api_key="nr_live_...") Note over SDK: fetches HMAC secret via /api/v1/auth/verify U->>SDK: with workflow("user-123"):
@protect
def step(): ... loop every @protect call U->>SDK: step() SDK->>G: POST /api/v1/gate (with projected cost) G-->>SDK: {decision: "allow"} SDK->>SDK: run wrapped function SDK->>G: POST /api/v1/track (actual cost) end DB->>G: operator clicks Kill G-->>SDK: WS push: StateChange(killed) SDK->>U: raise WorkflowKilledInterrupt ```
End-to-end: SDK wiring, gate evaluation, kill signal path.

Managed runtime, not a self-hosted deployment

NullRun runs as a managed control plane at nullrun.io. There is no self-hosted deployment option today. The Python SDK runs inside your process and talks to the hosted gateway over HTTPS; the dashboard at nullrun.io hosts the control plane. See the docs for the SDK surface and /about for the runtime contract.

--- # Source: https://docs.nullrun.io/getting-started/onboarding/ # File: docs/getting-started/onboarding.md --- title: Onboarding description: Wire NullRun into an existing agent in fifteen minutes: install, key, decorate, set a budget, ship. --- # First agent This is the recommended path from "I have an LLM app" to "NullRun is gating my spend and tools". Each step links out to deeper docs only when you need them. ## 1. Sign up and create an API key 1. Go to [nullrun.io](https://nullrun.io) and sign in. 2. In the sidebar, under **Access**, open **API keys**, then click **New API key** in the top right. 3. Pick a name (e.g. `"my-first-agent"`) and the workflow you want the key bound to. Each key is **workflow-scoped** — it represents one agent run, not one workspace. 4. Copy the key (`nr_live_…`) shown once and store it somewhere safe (env var, secret manager). You'll need it in step 3.
NullRun New API key dialog, with the workflow dropdown and key name field. NullRun New API key dialog, with the workflow dropdown and key name field.
API keys · New key
## 2. Install the SDK ```bash title="shell" pip install "nullrun[openai]" # raw openai SDK + tracking pip install "nullrun[langgraph]" # if you're using LangGraph pip install "nullrun[agents]" # if you're using OpenAI Agents SDK pip install "nullrun[all]" # every vendor extra — heaviest install ``` See [Install](install.md#optional-extras) for the full list of extras. For this walk-through `nullrun[openai]` is enough. ## 3. Wire NullRun into your code Pick the pattern that matches what you have today: ### A. You already call `client.chat.completions.create(...)` ```python title="my_agent.py" import nullrun from openai import OpenAI from nullrun import init_or_die, protect, shutdown # 1. One line — reads NULLRUN_API_KEY from env if not passed. init_or_die(api_key="nr_live_...") client = OpenAI() # 2. @protect gates every call through NullRun before it runs. @protect def answer(prompt: str) -> str: response = client.chat.completions.create( model="gpt-4o-mini", messages=[{"role": "user", "content": prompt}], ) return response.choices[0].message.content # 3. shutdown() flushes pending events and closes the WS cleanly # — register via atexit in production scripts. if __name__ == "__main__": try: print(answer("What does NullRun do?")) finally: shutdown() ``` Every call inside `answer()` is cost-attributed. `@protect` is the **gate** (budget pre-flight + kill check + sensitive-tool decision), not the tracking mechanism — tracking is handled automatically by auto-instrumentation. ### B. You use a framework (LangGraph / CrewAI / OpenAI Agents / AutoGen / LlamaIndex) Auto-instrumentation does the same thing — see [Use with LangGraph](../how-to/langgraph.md) or any of the other [framework how-tos](../how-to/llm-frameworks.md). Most of the time the only line you add is the `init()` call. ## 4. Set a budget In the dashboard, open the workflow your key is bound to and set a `budget_cents`. A reasonable starter budget: | Use case | Suggestion | |---|---| | Personal / dev experiment | `500` ($5) per period | | Single-tenant internal tool | `2000` ($20) per period | | Customer-facing AI feature | `10000` ($100) per period with alerts | Periods are either calendar-month UTC (Lite) or your billing cycle (paid plans via Polar). See [Budgets → Period rollover](../concepts/budgets.md) for the detail. ## 5. Run and observe ```bash title="shell" python my_agent.py ``` Then open the dashboard → **Workflows** → your workflow → **Executions**. You'll see every `/gate` call (one per `@protect`-wrapped invocation), the policy verdict (`allow` / `block` / `rate_limit`), and the cost. For real-time spend, hit [`GET /api/v1/orgs/{org_id}/status`](../reference/http-api.md#common-request-patterns) — it returns `current_spend_cents`, `budget_cents`, `time_to_exhaustion`, and your plan caps in a single call (see the **Single-call status** example under "Common request patterns"). ## 6. Tighten or loosen Common next steps, in rough order of how often they're needed: 1. **Block a tool** the agent shouldn't touch — see [Tool policies](../concepts/tool-policies.md) and the recommended ToolBlock starter list in the [Tool catalog](../reference/llm-tool-catalog.md#recommended-toolblock-starter-list). 2. **Allow over-budget for long agents** — see [Chain context → soft mode](../concepts/workflow.md#chain-context). 3. **Forward every error to Sentry** — see [Error handling → on_error hook](../concepts/error-handling.md). 4. **Pre-flight keys before risky calls** — see [Human approval](../concepts/human-approval.md). ## What this walk-through didn't cover - **Multi-process / multi-key** patterns — see [Run multiple agents](../how-to/multi-agent.md). - **Self-hosted gateway** — see your on-prem deployment runbook. - **Streaming responses** — see [Stream with chain heartbeat](../how-to/streaming.md). ## Where to read next - [Concepts → Circuit breaker](../concepts/circuit-breaker.md) — the mental model behind `@protect`. - [Concepts → Error handling](../concepts/error-handling.md) — the three-layer error model. - [Concepts → Workflow context](../concepts/workflow.md) — what the `with nullrun.workflow(...)` block does. --- # Source: https://docs.nullrun.io/getting-started/tour/ # File: docs/getting-started/tour.md title: 5-minute tour maturity: stable description: Five-minute walkthrough of the NullRun dashboard, policies, and SDK — enough to evaluate the platform end-to-end. # 5-minute tour This is the shortest path from "I have never used NullRun" to "I shipped an agent to production and it tripped a budget cap." It mirrors the dashboard tour at `nullrun.io/onboarding` — same five screens, same five CLI commands. > **If you want to install first**, jump to [Install](install.md) and > come back. This tour assumes `nullrun` is already installed and an > API key is exported as `NULLRUN_API_KEY`. ## What you will build A LangGraph agent that: 1. Calls `gpt-4o-mini` through the NullRun gate 2. Has a hard $0.50 budget per workflow 3. Trips the circuit breaker when it tries to call `send_email` 4. Recovers cleanly after you raise the budget You will see each step land in the dashboard as a real audit row. ## Step 1 — Create an organization If you do not already have one, open `nullrun.io/onboarding`, pick a name ("Acme AI"), and click **Create**. The dashboard provisions an `organization_id` and a default `policy_id` that allows everything except destructive tools. !!! note "What you see" On the dashboard home, a single card shows: organization name, default policy name (`Permissive`), and an "API keys" tile that is empty until step 2. ## Step 2 — Create an API key In the dashboard: 1. In the left sidebar, under **Access**, click **API keys**. 2. Click **New API key** in the top right. 3. In the dialog, pick the **Workflow** the key belongs to (the tour-agent workflow you just created) and name the key `tour-agent`. Pick an expiration — **Never**, **24 hours**, **7 days**, **30 days**, or **90 days**. 4. Click **Create**. 5. Copy the `nr_live_…` public identifier and the HMAC secret. The secret is shown **once** — store it in your secrets manager immediately.
API keys page showing the New key button highlighted in the top right. API keys page showing the New key button highlighted in the top right.
API keys · New key
Export both in your shell: ```bash title="shell" export NULLRUN_API_KEY="nr_live_xxxxxxxxxxxxxxxx" export NULLRUN_SECRET_KEY="hmac_xxxxxxxxxxxxxxxxxxxx" ``` !!! warning "Where the HMAC secret lives" The SDK pulls the HMAC secret via `POST /api/v1/auth/verify` on first use, then caches it in memory. Re-exporting the env var does NOT invalidate an existing cached secret — restart your process to pick up a new one. ## Step 3 — Wire up the agent Create a file `tour_agent.py`: ```python title="tour_agent.py" import os import nullrun from nullrun import init, protect, NullRunBudgetError from langchain_openai import ChatOpenAI init(api_key=os.environ["NULLRUN_API_KEY"]) llm = ChatOpenAI(model="gpt-4o-mini") @protect def ask(question: str) -> str: return llm.invoke(question).content if __name__ == "__main__": for i in range(20): try: print(f"[{i}]", ask("Tell me a one-sentence joke.")) except NullRunBudgetError as exc: print(f"[{i}] BLOCKED:", nullrun.format_user_message(exc)) break ``` Run it: ```bash title="shell" pip install "nullrun[langgraph]" langgraph langchain-openai python tour_agent.py ``` You will see ~7–10 successful LLM calls, then `BLOCKED: You've used all your support credits. Upgrade to keep chatting.` (or whatever your catalog wording is). ## Step 4 — Watch the decisions Open `nullrun.io/control-center/audit` (the dashboard **Audit log** page, under **Governance** in the sidebar). You will see: - One row per `@protect` call across four columns: **Time**, **Decision**, **Rule**, **Actor**. - `decision = allow` for the first ~7–10 rows. The **FilterBar** above the table has period preset chips (1h / 24h / Today / 7d / 30d / 90d / All), decision chips, and an event-type selector. - `decision = block` on the last row with `error_code = NR-B004`, `wire = BUDGET_HARD_BLOCKED` — click the row to open the **DetailPanel** and see the budget snapshot at the time of the block.
Audit log page — Allow / Deny / Require approval filter chips and the events table. Audit log page — Allow / Deny / Require approval filter chips and the events table.
Governance · Audit log
Each row is recorded by the gateway's audit pipeline and surfaces in your own customer's audit trail identically. ## Step 5 — Trip a ToolBlock Edit `tour_agent.py` and add a second protected function: ```python title="tour_agent.py" @protect def send_email(to: str, body: str) -> None: # Pretend SMTP call. print(f"SMTP → {to}: {body}") ``` Then call it from `__main__`: ```python title="tour_agent.py" # After the loop: try: send_email("test@example.com", "hi from the tour") except nullrun.NullRunToolBlockedError as exc: print(f"BLOCKED:", nullrun.format_user_message(exc)) ``` Run it again. The dashboard shows a `decision = block` row with `error_code = NR-T001`, `wire = TOOL_BLOCKED`. New organizations ship with a permissive default policy; if your admin has added a stricter default, you may see additional blocks. To allow `send_email`, open **Policies** in the sidebar, find the tool-block rule that matches `send_email`, and either narrow the pattern or scope it to a different workflow. ## Step 6 — Raise the budget and try again Back in the dashboard: 1. Open the tour-agent workflow and stay on the **Overview** tab. 2. In the budget card, raise the cap to `$5.00` (500 cents). 3. Save. Re-run `tour_agent.py`. The loop now completes all 20 calls. The **Overview** tab's spend bar shows ~$0.40 used (depending on token counts), and the progress bar sits at ~8%. ## What next? | You want to… | Open | | --- | --- | | Understand the gate in depth | [Concepts → Circuit breaker](../concepts/circuit-breaker.md) | | Wire up multiple agents | [How-to → Run multiple agents](../how-to/multi-agent.md) | | Add an approval flow for sensitive tools | [Concepts → Human approval](../concepts/human-approval.md) | | Stream responses | [How-to → Stream responses](../how-to/streaming.md) | | Deploy to production behind your gateway | [Configuration → Behaviour](configuration.md#behaviour) | !!! tip "Where to send feedback" Email `support@nullrun.io` with the dashboard's **Help → Send feedback** form filled in. Include the workflow ID (top-right of any dashboard page) and the failing row's `decision_id`. --- # Source: https://docs.nullrun.io/getting-started/install/ # File: docs/getting-started/install.md --- title: Install description: Install the NullRun Python SDK with pip, create an API key in the dashboard, and verify the gate is reachable from your environment. --- # Install ## Python SDK ```bash title="shell" pip install nullrun ``` Verify: ```bash title="shell" python -c "from nullrun import protect; print('ok')" ``` > **No local mode.** If `init()` is called without an API key, the > SDK raises `NullRunAuthenticationError` at first use. There is no > offline / local-only fallback. ## API key Sign in at [nullrun.io](https://nullrun.io), open **API keys**, and create a key. Each key is minted with a public identifier (`nr_live_...`) plus a server-side HMAC secret. The SDK transparently obtains the HMAC secret via: ```http POST /api/v1/auth/verify ``` on first use, so you only need to pass the API key: ```python title="app.py" import nullrun nullrun.init(api_key="nr_live_...") ``` The public `init()` surface takes `api_key` (and optionally `api_url`, `debug`). The HMAC secret is **not** a constructor argument — it is read from `NULLRUN_SECRET_KEY` or returned by `/api/v1/auth/verify`. For env-var setup (`NULLRUN_API_KEY`, `NULLRUN_SECRET_KEY`, and other runtime flags), see [Configuration](configuration.md). ## Auto-instrumentation `nullrun.init()` patches the underlying HTTP transport (`httpx`) and the agent framework modules it can detect in `sys.modules`: | Detected | Coverage | | --- | --- | | `openai` ≥ 1.0 | HTTP transport hook | | `openai-agents` | Agent framework hook | | `anthropic` | HTTP transport hook | | `langgraph` | Graph runtime hook (`invoke` / `stream` / `ainvoke` / `astream`) | | `langchain` | Callback manager hook | | `llama-index` | LlamaIndex tool/agent hook | | `crewai` | CrewAI EventBus bridge (1.15+) | | `autogen` | AutoGen agent runtime hook | | `mistralai`, `google-genai`, `cohere`, `boto3` (bedrock) | per-vendor extractors | The Gemini vendor extra is `google-genai` (the actively maintained package, ≥ 1.0); the older `google.generativeai` package is **not** supported. Install with `pip install "nullrun[gemini]"`. In every case the call is cost-tracked automatically — `@protect` is not required for tracking. `@protect` is the **gate** layer (budget pre-flight + kill/pause + sensitive-tool decision). ## Optional extras | Extra | Installs | | --- | --- | | `nullrun[opentelemetry]` | `opentelemetry-api`, `opentelemetry-sdk` | | `nullrun[langgraph]` | `langgraph` | | `nullrun[openai]` | `openai` | | `nullrun[anthropic]` | `anthropic` | | `nullrun[mistral]` | `mistralai` | | `nullrun[gemini]` | `google-genai` | | `nullrun[cohere]` | `cohere` | | `nullrun[bedrock]` | `boto3` | | `nullrun[agents]` | `openai-agents` | | `nullrun[langchain]` | `langchain-core` | | `nullrun[llama-index]` | `llama-index-core` | | `nullrun[crewai]` | `crewai` | | `nullrun[autogen]` | `autogen-agentchat`, `autogen-ext[openai]` | | `nullrun[fastapi]` | `fastapi`, `starlette`, `httpx` (server-framework integration) | | `nullrun[opentelemetry]` | `opentelemetry-api`, `opentelemetry-sdk` | | `nullrun[all]` | every vendor extra | ```bash title="shell" pip install "nullrun[langgraph]" pip install "nullrun[all]" ``` > Note: `nullrun[openai]` is for the raw `openai` SDK — it is **not** > the OpenAI Agents SDK. For agents use `nullrun[agents]`. --- # Source: https://docs.nullrun.io/getting-started/quickstart/ # File: docs/getting-started/quickstart.md --- title: Quickstart description: Decorate your first tool with @protect and ship it through the NullRun gate in under thirty lines of code. --- # Quickstart Wrap any function with **`@nullrun.protect`** to track its cost, tools, and behaviour, and let NullRun halt it when it goes off the rails. ```python title="app.py" from openai import OpenAI from nullrun import init_or_die, guarded, protect, workflow, shutdown init_or_die(api_key="nr_live_...") # exits cleanly if api_key missing client = OpenAI() with workflow("my-first-agent"): # scopes the gate to a workflow @guarded # catches NullRunError, prints @protect # the catalog user-message, def answer(prompt: str) -> str: # sys.exit(1) — zero boilerplate response = client.chat.completions.create( model="gpt-4o-mini", messages=[{"role": "user", "content": prompt}], ) return response.choices[0].message.content if __name__ == "__main__": try: print(answer("What does NullRun do?")) finally: shutdown() ``` > The `with workflow("..."):` block binds every `@protect` call inside > to a named workflow — required, otherwise the SDK falls back to an > ad-hoc workflow_id with no budget policy attached. For production, > the workflow name should match the dashboard workflow your API key > is bound to. Every call inside `answer()` is cost-attributed and governed by your workspace policy. On any policy outcome (budget cap, tool block, rate limit, transport outage), `@guarded` prints the catalog wording on stderr and exits `1`. ## What gets tracked - LLM tokens in and out - Cost in cents (per-call and aggregate) - Latency - Tool calls (if you use a framework integration) ## What can go wrong See [Troubleshooting](../troubleshooting.md) for the full table of expected behaviours (budget cap, loop, sensitive-tool, gateway down, kill/pause, etc.) and recovery steps. For the three-layer error model, see [Concepts → Error handling](../concepts/error-handling.md). ## Next - [Concepts → Circuit breaker](../concepts/circuit-breaker.md) - [Concepts → Control plane](../concepts/control-plane.md) - [Concepts → Error handling](../concepts/error-handling.md) - [How-to → Set a hard cost cap](../how-to/cost-cap.md) - [How-to → Use with LangGraph](../how-to/langgraph.md) --- # Source: https://docs.nullrun.io/getting-started/configuration/ # File: docs/getting-started/configuration.md --- title: Configuration description: Every NullRun SDK environment variable, transport option, and fail-CLOSED guard documented with safe defaults. --- # Configuration NullRun reads configuration from environment variables. `nullrun.init()` only needs the API key — everything else has sensible defaults. Variables are read by the Python SDK process. The gateway is operated by the NullRun team and exposes no user-facing runtime flags. ## SDK env vars Read by `nullrun.init` and the SDK transport. None of these affect the gateway. | Variable | Default | Description | | --- | --- | --- | | `NULLRUN_API_KEY` | unset (required) | API key from the NullRun dashboard (`nr_live_...`). Missing at `init()` raises `NullRunAuthenticationError` (NR-C001). | | `NULLRUN_API_URL` | `https://api.nullrun.io` | Gateway REST base URL. The WebSocket control plane URL is derived from this as `wss:///ws/control/{org_id}` — `{org_id}` is the `organization_id` returned by `POST /api/v1/auth/verify`, and is **not** a separate env var. | | `NULLRUN_SECRET_KEY` | unset | HMAC-SHA256 signing secret. The SDK signs every request automatically when this is set. | | `NULLRUN_ENV` | unset | Environment tag (`production` / `staging` / ...). | | `NULLRUN_APPROVAL_TIMEOUT_SECONDS` | `300` | SDK-side wait for the `approval_resolved` WS push before fail-CLOSED kill. | | `NULLRUN_REQUEST_TIMEOUT` | `30` | HTTP request timeout in seconds. | | `NULLRUN_TRANSPORT` | `ws` | Control-plane transport mode (`ws` or `http`). | | `NULLRUN_GATE_CACHE_DISABLE` | unset | `=1` disables the SDK's local gate cache (forces a fresh gate evaluation on every `@protect` call). | | `NULLRUN_TLS_CLIENT_CERT` / `NULLRUN_TLS_CLIENT_KEY` / `NULLRUN_TLS_CA_CERT` | unset | Optional mTLS material for the SDK-to-gateway connection. | | `NULLRUN_MAX_RESPONSE_BYTES` | library default | Cap on captured LLM response body size for span metadata. | ## Developer and CI overrides !!! danger "Production-safe default: do NOT set these in production traffic" The variables below override the gate's safety defaults. They exist for local SDK development and CI only. Exporting them in a production environment silently disables protection — your agent will run un-gated. | Variable | Effect | When to use | | --- | --- | --- | | `NULLRUN_SKIP_BUDGET_CHECK=1` | Fully bypasses the gate on every `@protect` call in the process. **For local SDK development and CI only** — do not export in production environments. Production with this flag set silently skips every policy check. | Local SDK experiments, integration tests where you want to verify business logic without gate noise. | | `NULLRUN_ALLOW_SKIP_BUDGET_CHECK=1` | Acknowledges the previous flag in CI logs so an audit reviewer can see the bypass was deliberate. Has no effect by itself; safe to set alongside the previous flag in CI. | CI pipelines that intentionally skip the gate. | | `NULLRUN_SENSITIVE_FAIL_OPEN=1` | Returns a permissive result instead of failing-CLOSED when a sensitive-tool transport error blocks the gate call. | Legacy environments without a working transport for sensitive-tool lookups — modern installs should leave this unset. | If a CI test "passes only with `NULLRUN_SKIP_BUDGET_CHECK=1`" that's a signal the gate is blocking what it should not — fix the gate, not the bypass. ## Server-side configuration NullRun runs as a managed service; the gateway is operated by the NullRun team and exposes no user-facing runtime flags. ## Behaviour The HTTP request timeout is configurable via `NULLRUN_REQUEST_TIMEOUT` (default `30`s). The control-plane transport (WS push vs. HTTP polling fallback) is configured by `NULLRUN_TRANSPORT` (default `ws`). The default is WS push with HTTP polling fallback when the WS connection drops more than 10 times in a row. HMAC signature window (`NULLRUN_HMAC_MAX_AGE_SECS`, default `300`s) is a server-side setting. The SDK signs every request automatically when `NULLRUN_SECRET_KEY` is set. ## See also - [HTTP API](../reference/http-api.md) - [Control plane](../concepts/control-plane.md) - [Circuit breaker](../concepts/circuit-breaker.md) --- # Source: https://docs.nullrun.io/concepts/circuit-breaker/ # File: docs/concepts/circuit-breaker.md title: Circuit breaker maturity: stable description: How NullRun's circuit breaker trips on a budget overrun, recovers after a cooldown, and propagates a kill signal across in-flight calls. # Circuit breaker The circuit breaker stops your agent when something goes wrong. When the agent is hitting the budget cap, calling a tool your policies forbid, or being asked by an operator to stop — the gate returns `block` and the SDK raises an exception, even if the agent's code doesn't know to stop. The underlying mechanism is a single `/api/v1/gate` evaluation per `@protect`-wrapped call that returns `allow` / `block` / `require_approval`. In the dashboard, a tripped breaker shows up as the workflow's status flipping from **Active** to **Killed** or as a flood of **block** decisions in the **Audit log**. ## When does it trip? The gate reacts to three categories of situation. Each is a separate decision path inside `/gate`, but to you it all looks the same: the next call rejects. | Situation | What you see | Where in the dashboard | |---|---|---| | **Budget exceeded** (Hard mode) | Every call returns `block`; SDK raises `NullRunBudgetError` with `error_code = "NR-B004"` | Audit log, then the spend bar hits 100% | | **Tool blocked** by policy | `block`; SDK raises `NullRunToolBlockedError` with `error_code = "NR-T001"` | Audit log | | **Operator kill** | `WorkflowKilledInterrupt` (alias `NullRunWorkflowKilledError`) raised mid-call | Workflow status flips to **Killed** | Rate limiting (429) and budget soft-mode blocks are returned by the same gate but with different codes. SDK surfaces them as `error_code = "NR-R001"` and `error_code = "NR-B004"`. See [Budgets](budgets.md#soft-mode) and [Policies](policies.md). The first two are automatic — the gate enforces them on every call. The third needs you to click **Kill** in the dashboard or call `POST /api/v1/workflows/{id}/kill`. ## What the agent sees When the breaker trips, the SDK raises an exception. The exact exception depends on what tripped it: | Trip cause | Exception | Class | |---|---|---| | Budget exceeded | `NullRunBudgetError` (`error_code = "NR-B004"`) | `NullRunError` (Exception) | | Tool blocked | `NullRunBlockedException` (`error_code = "NR-T001"`) | `NullRunError` (Exception) | | Operator kill | `WorkflowKilledInterrupt` (alias `NullRunWorkflowKilledError`) | `NullRunError` (Exception) | The kill signal inherits from `NullRunError`, so `try/except Exception:` catches it like every other SDK error. To handle kill specifically — checkpoint state, notify a supervisor, exit cleanly — catch `NullRunWorkflowKilledError` (preferred) or `WorkflowKilledInterrupt` explicitly. See [Error handling → Kill signal](../concepts/error-handling.md#kill-signal) for the recommended handler shape. If you use the zero-boilerplate helpers from the SDK, you don't have to write any of this — `@guarded` catches the standard exceptions (including the kill signal), prints the catalog wording, and exits 1. To handle kill distinctly, use the un-`@guarded` `protect()` form. ## When the gateway is unreachable Sometimes the gateway itself is down — DNS, network, an outage. The mental model: critical paths (budget reservation, ToolBlock, aggregate rate limit) refuse to run when the gateway can't be reached; secondary signals (per-key rate limit) may let calls through. When the gateway rejects because of an infrastructure failure, you'll see a clear HTTP error from the SDK. If you're seeing persistent infrastructure failures, contact support. ## When the breaker recovers After the gateway comes back, the gate transitions automatically to normal mode. No operator action needed — the next `/gate` call succeeds if the policy allows it. If the gate is blocking too often (every call rejects), look at: 1. The **Audit log** for the workflow. The reason column tells you why each call was blocked. 2. The workflow's **Overview** tab — the spend vs. cap bar shows whether you're consistently hitting the budget. Raise the cap or switch to a cheaper model if so. 3. **Effective policy** (on the **Policies** tab). A policy you added recently may be too strict — try narrowing patterns or scoping to one workflow before rolling out org-wide. ## Common scenarios ### "My agent suddenly stopped responding" Open the workflow in the dashboard. Check the state: | Status | What happened | |---|---| | **Active** | The agent is fine — check the application logs for the actual error | | **Paused** | You paused it (or an operator did). Click **Resume** to restart. See [Control plane](control-plane.md). | | **Killed** | You killed it (or an operator did). Create a new workflow or re-activate. | If the status is Active but every call rejects, open the **Audit log** and filter by `decision = block`. The reason column shows the pattern that matched. ### "My agent was working yesterday and is blocked today" Look at the workflow's **Overview** tab — the spend bar. The budget probably rolled over (new month or billing cycle renewal) and the new period started with an empty counter. Raise the cap or wait for the next reset. ### "I want to test my agent without the breaker tripping" Use a **separate workflow** with its own (low or zero) budget. Don't disable the gate — bypassing it is a dev/test opt-out that the SDK flags with a `RuntimeWarning`. ## See also - [Budgets](budgets.md) — the most common trip cause - [Tool policies](tool-policies.md) — your own blocking rules - [Human approval](human-approval.md) — the alternative to blocking for sensitive operations you actually want to allow - [Troubleshooting](../troubleshooting.md) — common "why is my agent blocked?" questions --- # Source: https://docs.nullrun.io/concepts/budgets/ # File: docs/concepts/budgets.md title: Budgets maturity: stable description: Hard and soft budget enforcement, billing-period rollover, and the reserve / consume invariant that protects against implicit re-reservation. # Budgets A **budget** is the most important number on the dashboard. It's the maximum amount of money a workflow is allowed to spend in a billing period. Set it too low and your agent stops working. Set it too high and a runaway agent burns through real money before you notice. This page covers what the budget controls, how the dashboard shows it, and what happens at each boundary. ## Where you see it On the **Workflows** detail page, the budget appears as a progress bar near the top: ``` Spend this period $47.30 of $50.00 (95%) ████████████████████████░░ Time to exhaustion ~16 hours at current rate ``` Three numbers: - **Spend this period** — total cents spent since the last period rollover. Resets automatically. - **Budget** — the cap. Set this in workflow settings. - **Time to exhaustion** — at the current rate of spend, when the budget will run out. Useful for "should I raise the cap?". ## What the budget covers The budget covers **spend**, not calls. Calls are rate-limited separately — see [Policies](policies.md). "Spend" is calculated from token counts reported by your LLM provider. The dashboard knows the per-model pricing for every model the SDK tracks: - **Input tokens** × input rate - **Output tokens** × output rate - **Cache read** / **cache write** tokens (if your provider exposes them) at their respective rates - **Reasoning tokens** for o1/o3-style models at the reasoning rate The total spend is the sum across all `@protect` calls inside the workflow, across the current period. ## Periods A "period" is the window after which the spend counter resets. NullRun has two period sources: | Plan | Period source | When it resets | |---|---|---| | **Lite** (free) | Calendar month UTC | 1st of each month at 00:00 UTC | | **Paid** (Starter / Growth / Scale) | Your billing cycle (Polar subscription) | Set when you subscribed; on renewal | The dashboard shows the period start and end dates next to the spend bar. When the period rolls over, the spend counter resets to zero and the budget applies fresh. ## What happens at the boundary Three scenarios, depending on the workflow's [enforcement mode](policies.md#budgetlimit-extra-fields): ### Hard mode (default) ``` Spending → $49.95 of $50.00 Next @protect call: #2.00 projected gate decision: block SDK raises: NullRunBudgetError (NR-B004) @guarded: prints friendly message, sys.exit(1) ```
Hard mode — the projected cost of the next call exceeds the remaining budget. The gate returns `block` before the model runs.
The agent stops cleanly at the boundary. No partial charge — the projected cost is reserved when the gate approves, and the actual cost is reported after the LLM returns. If the call is denied, no charge happens. ### Soft mode Soft mode lets the agent run past its budget when an active chain is present, up to the configured overdraft cap (`max_overdraft_cents` or `max_overdraft_percent`, whichever is lower). The chain returns to standard Hard mode once the cap is exhausted. See [Policies → BudgetLimit extra fields](policies.md#budgetlimit-extra-fields) for the full configuration contract. ## How to set the budget The first time you create a workflow, no budget cap is configured. `max_budget_cents == 0` means **"no per-key budget configured"** — the gate passes through to the org-level plan cap, not "block everything" — so the agent runs against the org's default policy until you raise the per-key cap. To set the budget: 1. Open the workflow. 2. Click **Settings**. 3. Find **Budget** and enter cents (`$50` = `5000`). 4. Save.
Workflow detail — Overview tab. The Budget card sits at the top showing spent / cap. Workflow detail — Overview tab. The Budget card sits at the top showing spent / cap.
Workflows · Budget card
Reasonable starting budgets: | Use case | Suggested budget | |---|---| | Personal / dev experiment | $5 (500 cents) per period | | Single-tenant internal tool | $20 (2000 cents) per period | | Customer-facing AI feature | $100 (10000 cents) per period, plus an alert at 80% | The dashboard warns you when spend crosses 80% of the cap and again at 100%. Configure alert destinations under **Notifications** in the sidebar (Channels + Alert rules + Event subscriptions matrix). ## What happens when you change the budget mid-period - **Raise**: the new cap takes effect immediately. The next gate call uses the new cap. - **Lower below current spend**: the agent doesn't get retroactive refunds, but every call from this point onward rejects until the spend drops (which only happens at period rollover, since the counter is monotonic within a period). ## Why cents, not dollars The dashboard stores everything in cents to avoid floating-point rounding in pricing math. The `budget_cents` field in the API is always an integer. If you set `budget_cents: 5000`, your cap is exactly $50.00, no rounding errors. ## Reservation and consumption The gate reserves your projected cost before the model runs and reconciles the actual cost after. If the LLM call returns a cost that meaningfully exceeds the reservation, the `/track` commit rejects with `CONSUME_OVERBUDGET` (HTTP **422**, `error_code = "NR-O001"`) — no implicit re-reserve, ever. The tolerance is a fixed cents value (`policies.consume_epsilon_cents`, default **1¢**); no percentage-based epsilon is supported. ## Approximate budget endpoint If you want to show "you've used X of Y" in a custom dashboard or notification without enrolling in the full NullRun dashboard, the gateway exposes an approximate-spend endpoint: ```bash title="shell" curl "https://api.nullrun.io/api/v1/budget/approximate" \ -H "Authorization: Bearer ***" ``` The response carries `current_spend_cents_estimate`, an `is_approximate: true` flag, a `source` field, a `confidence` level (`High` / `Medium` / `Low`), and `last_updated_at`. **Use this for display only** — never for enforcement, rate-limit logic, or agent-side gating. When the source is unavailable the endpoint returns `503 BUDGET_DATA_UNAVAILABLE`; render that as "data unavailable", never as `≈ $0 spent`. ## See also - [Workflows](workflow.md) — where the budget lives - [Policies](policies.md) — rate limits (separate from budget) and soft-mode fields - [Troubleshooting](../troubleshooting.md#why-is-my-call-being-rejected-with-nullrunblockedexception) --- # Source: https://docs.nullrun.io/concepts/sensitive-tools/ # File: docs/concepts/sensitive-tools.md title: Sensitive tools maturity: stable description: Mark a tool @sensitive to make it fail-CLOSED on transport errors, with no opt-out — the safest class for irreversible actions. # Sensitive tools A **sensitive tool** is one that should never run without a human paying attention. Sending an email, moving money, deleting a record — all of these have consequences the agent can't easily undo. The way to express "this tool needs review" in NullRun is a **[ToolBlock policy](tool-policies.md)** — a glob pattern that fails the gate at `/gate` evaluation time, regardless of the SDK's local context. This page covers the *recommended* patterns and how to wire them. !!! info "`@sensitive` vs `ToolBlock`" Two distinct mechanisms, often confused: - **`@sensitive` (SDK-side)** — a parameterless decorator that marks a function so its kwargs are extracted into the `BusinessImpact` predicate bag. Used with **approval rules** that evaluate a typed predicate. Affects the SDK only; the gate still has the final say. - **`ToolBlock` (server-side)** — a policy rule evaluated by the gate on every `/gate` call. The gate fails-CLOSED if it cannot reach Redis or the policy cache to evaluate. This is the canonical "this tool is forbidden" mechanism. Use `@sensitive` when you want a typed `BusinessImpact` approval flow (e.g. "refunds over $500 need approval"). Use `ToolBlock` when you want a hard rule ("never call `bash`"). ## What a sensitive tool is, in policy terms There is no built-in tool catalogue shipped by the SDK — every enforcement decision is evaluated by the gate on every `/gate` call, so you can't accidentally miss a tool you didn't register locally. You express "sensitive" with one of two complementary mechanisms: - **`@sensitive` (SDK-side)** — marks a function so its kwargs flow into the `BusinessImpact` predicate bag used by approval rules (typed `money_amount` / `tool_parameters`). Use this when you want a typed predicate — e.g. "refunds over $500 need approval". - **`ToolBlock` (server-side)** — a policy rule evaluated by the gate. The gate fails-CLOSED if it cannot reach Redis or the policy cache to evaluate. Use this for hard rules — "never call `bash`". For the typed predicate wiring, see [Decorators & extractors → `money_outflow(...)`](../reference/decorators.md#money_outflow-typed-money-impact) and [`tool_params(...)`](../reference/decorators.md#tool_params-free-form-argument-bag). Recommended starter patterns (see [Tool catalog → Recommended ToolBlock starter list](../reference/llm-tool-catalog.md#recommended-toolblock-starter-list) for the maintained list): | Category | Pattern examples | |---|---| | Money | `mcp://payments/refund*`, `mcp://stripe/charge`, `mcp://stripe/refund` | | Email & messaging | `mcp://gmail/send`, `mcp://slack/post`, `send_email` | | Database destructive | `mcp://postgres/drop_table`, `mcp://postgres/delete_row`, `execute_sql` | | External API writes | `mcp://*/post`, `mcp://*/put`, `mcp://*/delete` | | Files & storage | `mcp://s3/delete`, `file_delete`, `bash` | | Admin | `mcp://admin/delete_user`, `mcp://admin/disable_user` | These are the canonical tool names a policy matches against. The **exact name** comes from your MCP server / framework integration — see the tool catalog for the curated list with risk ratings. ## Why the SDK does not ship a built-in list A built-in "sensitive tools" SDK list would force every framework to register its tools against NullRun's expectations — and would be silently wrong for any tool not on the list. The current model inverts this: - You write a ToolBlock policy that names the tools you care about. - The policy is evaluated server-side on every `/gate` call. - The decision is returned to the SDK as `TOOL_BLOCKED` (403); the SDK raises `NullRunToolBlockedError` with `error_code = "NR-T001"`. The gate **does not inspect tool arguments** — it cannot distinguish two calls to the same tool by payload. If you want a narrower rule (e.g. "block refunds over $500"), use a typed `BusinessImpact` predicate: the SDK extracts the argument bag and the gate evaluates a DNF of up to 5 named parameters against Equals / OneOf / NumericRange / Regex / Exists matchers. See [Human approval → typed predicates](human-approval.md#typed-predicates). ## Why ToolBlock is enforced at the gate ToolBlock is enforced at the gate: sensitive operations never run when the policy engine is unreachable. If the gate returns `403 TOOL_BLOCKED` (SDK `error_code = "NR-T001"`), the SDK raises before your function body executes. ToolBlock is **always Hard**, regardless of the budget's `enforcement_mode`. ## What's NOT in a ToolBlock policy A ToolBlock policy matches **tool name only** — not: - prompt content or semantic intent - the recipient of a payment (use a typed predicate instead) - tool arguments beyond the `BusinessImpact` extraction - the tool's runtime sandbox (that's your infrastructure concern) Read operations are never sensitive regardless of the tool. The canonical name alone decides. ## Where the sensitive list lives You write the policy in the dashboard under **Policies** (sidebar under **Governance**). Click **New policy**, pick **Tool block** as the policy type, and the modal shows the **Tool pattern** field where you enter the glob(s). The dashboard shows you the canonical tool name for every framework integration. Your policy applies to: - All workflows under the org (default) - A specific workflow (scope to `workflow_id`) - A specific API key (scope to `api_key_id`) Per the [aggregation rules](../concepts/policies.md#aggregation): ToolBlock patterns **union** across applicable policies — every pattern that matches fires. ## Audit trail When a sensitive tool is blocked, the **audit log** records the block with reason `TOOL_BLOCKED` (SDK `error_code = "NR-T001"`), the pattern that matched, and the workflow + api_key + tool_name. The audit log is hash-chained — see [Audit records](../concepts/error-handling.md#audit-trail). This gives you a complete audit trail of every blocked attempt, regardless of whether the block came from your policy or from the default `TOOL_BLOCKED` rejection of an unknown tool name. For sensitive tools you want to allow after explicit human review, pair them with an **approval rule** instead of removing them from the blocking surface. The approval row in the dashboard gives you the audit trail, and the SHA-256 `action_digest` ensures the grant is bound to the exact action payload the SDK sent on `/gate`. See [Human approval](human-approval.md). ## See also - [Decorators & extractors → `@sensitive`](../reference/decorators.md#sensitive-the-per-tool-policy-marker) — the two `@sensitive` forms (bare + factory), the `money_outflow(...)` / `tool_params(...)` impact extractors, and the `_nullrun_extractor` contract that ties them to `/execute` - [Tool policies](tool-policies.md) — the actual rule structure - [Tool catalog](../reference/llm-tool-catalog.md) — recommended patterns with risk ratings - [Human approval](human-approval.md) — the safer alternative to disabling a ToolBlock rule - [Circuit breaker → fail-CLOSED matrix](../concepts/circuit-breaker.md#when-the-gateway-is-unreachable) --- # Source: https://docs.nullrun.io/concepts/workflow/ # File: docs/concepts/workflow.md title: Workflows maturity: stable description: Group agent calls into a named workflow, propagate parent_trace_id, and bind cost to a logical unit instead of a single session. # Workflows A **workflow** is one agent you run. In the dashboard it shows up under **Workflows** in the left sidebar. Each workflow has its own budget and its own list of API keys. ## What you see in the dashboard The **Workflows** page lists every workflow you've created. Each row shows: - The workflow's name (you picked this when you created it) - Whether it's **Active**, **Paused**, or **Killed** - Total spend for the current billing period - How many API keys are bound to it - When it last saw traffic Click a workflow to open its detail page. The detail page has six tabs: | Tab | What it shows | |---|---| | **Overview** | Name, status (Active / Paused / Killed), current spend vs. the installed budget cap, applied policies, and the **Pause** / **Resume** / **Kill** / **Delete** controls. This is where you change the budget cap. | | **Policies** | The policies scoped to this workflow. Rate limit, budget limit, and tool block entries — same primitives as the org-level Policies page, filtered to this workflow. | | **Executions** | Every gate call your agent made — allowed, blocked, rate-limited. The raw list the gate uses to decide what your agent can do. | | **Traces** | Hierarchical view of one agent run — each LLM call, each tool call, with timing and cost. | | **API keys** | The API keys bound to this workflow. Use **Generate API key** in the top-right to mint one; the raw key value is shown only once at creation. | | **Coverage** | MCP servers and tools observed on this workflow in the last 30 days, with the "discovered but not registered" panel for un-enrolled servers. | ## How to create one 1. In the dashboard sidebar, click **Workflows**. 2. Click **New workflow** in the top right. 3. Give it a name (e.g. `"production-support-bot"`). The name shows up everywhere — keep it short. Names are 1–255 characters: letters, digits, space, and `_ . , - & ( )` are allowed. 4. Optionally set an **External ID** — alphanumeric with `-` and `_`, up to 64 characters — for integrations that need to look up the workflow from your own systems (e.g. a GitHub repo name or a customer account id). 5. Click **Create**. The budget cap is configured on the **Overview** tab after creation via a budget-limit policy or the installed budget control — there is no starting budget on the dialog itself.
Workflows list with the New workflow button highlighted in the top right. Workflows list with the New workflow button highlighted in the top right.
Workflows · New workflow
Create workflow dialog open — Workflow name field and External ID optional field. Create workflow dialog open — Workflow name field and External ID optional field.
Workflows · Create dialog
Workflow detail page — Overview tab with budget card, applied policies, Pause and Kill controls. Workflow detail page — Overview tab with budget card, applied policies, Pause and Kill controls.
Workflows · Workflow detail
You'll land on the new workflow's detail page. From there: - **Mint an API key** under the **API keys** tab. The key value (`nr_live_...`) is shown **once** — copy it into your secret manager immediately. - **Point your SDK at it**: `nullrun.init(api_key=...)` picks up the key; the workflow binding happens server-side. ## How to control one Each workflow has three states that you control from the dashboard or via the API: **Active**, **Paused**, and **Killed**. Both Pause and Kill reach your running SDK over a WebSocket push; the agent doesn't have to wait for the next call to learn. See [Control plane](control-plane.md) for the full contract, the exceptions each state raises, and how the signal travels over the WebSocket. ## The workflow's settings Five things you control per workflow: - **Budget** — the per-period cap in cents. Set this first. The dashboard shows a horizontal bar of how much you've spent vs. the cap. - **Enforcement mode** — `Hard` (block on budget exceeded) or `Soft` (allow over-budget up to an overdraft cap, when there's an active chain). Full configuration in [Policies → BudgetLimit extra fields](policies.md#budgetlimit-extra-fields). - **Human approvals** — turn on to require operator approval for dangerous tools (payments, deletes, external API mutations). Available on Growth+ plans. - **Tool block list** — the patterns the agent must not call. See [Tool policies](tool-policies.md). - **Trace retention** — how long to keep detailed per-call traces (default 30 days, plan-gated up to 90). ## Chain context A **chain** is a logical grouping across multiple `@protect` calls inside one user request, declared via `with chain(...)`. Chains are auto-registered on the first `/gate` call: the chain transitions from `null → ACTIVE` atomically. ### When chains end A chain dies on the **first** of: - `op="end"` is reached in the context manager - 5 minutes of `/gate` inactivity (idle TTL) - `max_chain_duration_seconds` exceeded (default 3600) For long streams, send a `POST /heartbeat` every 30 seconds — see [Heartbeat → how-to](../how-to/streaming.md#chain-heartbeat). ### Why chains exist Chains exist primarily to enable **soft-mode budget gating**: with an active chain, the gate allows the agent to run past its budget up to an overdraft cap (`max_overdraft_cents` or `max_overdraft_percent`, whichever is lower). Full soft-mode contract in [Policies → BudgetLimit extra fields](policies.md#budgetlimit-extra-fields). ## How the workflow ends A workflow doesn't have an explicit "end" state in the sense of a final commit. Instead: - The workflow stays **Active** across many agent runs. Each run is a sequence of `@protect` calls. - A run is **logically ended** when the agent's loop returns or throws. - A workflow is **paused** or **killed** when you decide, or when plan limits (max workflows per plan) cause auto-pause. There is no "clean up the workflow when done" step. Active workflows keep their policy, budget, and key bindings. Re-run the agent next week and the same workflow handles it. ## See also - [Budgets](budgets.md) — the budget cap and how rollover works - [Policies](policies.md) — what rules attach to a workflow - [Control plane](control-plane.md) — how Kill / Pause reach your agent - [API keys](api-keys.md) — how to mint a key bound to this workflow --- # Source: https://docs.nullrun.io/concepts/tracing/ # File: docs/concepts/tracing.md title: Tracing maturity: stable description: OpenTelemetry-style spans for every gate decision, with parent_trace_id propagation so the dashboard renders a true waterfall. # Tracing A **trace** is everything that happened during one run of your agent. In the dashboard they live under **Executions** and **Traces** in the sidebar. Each execution is one agent run; the trace view shows the nested structure of every LLM call, every tool call, and how long each took. If a user reports "the agent did something weird at 14:30", the tracing tab is where you go to see exactly what happened. ## What you see in the dashboard The **Executions** page lists every agent run. Each row shows: - **Workflow** — which workflow ran this - **Started at** — timestamp - **Duration** — total run time - **Status** — completed / failed / killed - **Cost** — total cost for this run - **LLM calls** — how many LLM invocations
Executions page listing every agent run with workflow, duration, status and cost columns. Executions page listing every agent run with workflow, duration, status and cost columns.
Executions
Traces page with the waterfall of LLM and tool calls for a single execution. Traces page with the waterfall of LLM and tool calls for a single execution.
Traces · Waterfall
Click an execution to open the **Trace** view. The trace is a hierarchical tree: ``` Run "user-123-research" 2m 14s $0.42 ├─ Step 1: plan 0.3s $0.01 │ └─ llm.call (claude-sonnet-4-5) 0.3s $0.01 ├─ Step 2: research 45s $0.18 │ ├─ llm.call (claude-sonnet-4-5) 12s $0.06 │ ├─ tool.call (tavily_search) 8s — │ └─ llm.call (claude-sonnet-4-5) 22s $0.12 ├─ Step 3: write 30s $0.12 │ └─ llm.call (claude-sonnet-4-5) 30s $0.12 └─ Step 4: review 17s $0.11 └─ llm.call (claude-sonnet-4-5) 17s $0.11 ``` Three things you can read off this tree at a glance: - **Where the time went** — the longest step is where to optimise. - **Where the money went** — same, but for cost. - **What the agent did** — each tool call and LLM call is clickable, showing the full request/response. ## How a trace is built When you use the SDK's `@protect` decorator or `with workflow(...)` context manager, the SDK automatically creates spans: | Action | What gets a span | |---|---| | `@protect` decorator | One span per gate call | | `with workflow("name"):` | One span for the whole workflow run | | `with chain("id"):` | One span for the chain | | `with span("phase"):` | One span for the named phase | You don't have to add tracing manually — it comes from the decorators and context managers you already use. The SDK sends trace metadata alongside every `/gate` and `/track` call. For nested agent orchestrations (a supervisor calling sub-agents), each sub-agent's spans are nested under the supervisor's. The trace view shows the tree; the **Cost** column rolls up automatically. ## What each span contains Click any span in the trace tree to see: - **Span ID** — unique identifier (UUID) - **Parent span ID** — for nesting - **Started at** / **Duration** — timing - **Status** — completed / failed / killed - **Inputs** — the prompt metadata sent to the LLM (truncated if huge). **Prompt content is NOT stored** — NullRun never persists raw prompt text or LLM response bodies. See [Audit records → What is NOT stored](../concepts/error-handling.md#what-is-not-stored). - **Outputs** — the LLM's response metadata (token counts, model, finish reason). **Raw completions are NOT stored.** - **Cost** — input + output tokens × model rate - **Tool calls** — every tool the span invoked (with arguments) - **Decision** — the gate verdict (`allow` / `block` / `require_approval`) and which policy triggered it For blocked calls, the **Decision** row is the most useful — it links to the policy that matched and shows the rule. ## How long traces are kept Trace retention follows your plan's `history_days` window: | Plan | Trace retention | |---|---| | Lite | 3 days | | Starter | 7 days | | Growth | 30 days | | Scale | 90 days | | Enterprise | unlimited | After the retention window expires, the trace is removed from the dashboard; the aggregated cost information stays (it's summarised per workflow per period). If you need longer retention for compliance, you can export traces from the dashboard as JSON via the **Export** button on the Executions page. The exported shape matches the wire format. ## Span identifiers and correlation Each span has three identifiers: | Field | Purpose | |---|---| | `trace_id` | The whole agent run — same across every span in one execution | | `span_id` | One call — unique per `@protect` invocation | | `parent_trace_id` | For sub-agents — the orchestration trace they belong to | You can search the dashboard by any of these. If a customer reports a problem with `trace_id = abc-123`, you can pull the full trace and every decision tied to it from the audit log. ## How to use tracing during development When you're building a new agent, traces tell you: - **Is the agent slow?** — sort by duration, see which LLM call takes the most time. - **Is the agent hitting the budget?** — look for spans with `decision = block / NR-B004`. - **Is the agent calling tools you didn't expect?** — the trace shows every tool call with arguments. When you're debugging a production issue, traces answer: - **What did the agent do at 14:30 yesterday?** — filter by time range, click each execution, walk the trace. - **Why did the call to `send_email` fail?** — the trace shows the call's status and decision. If it was blocked, the linked policy explains why. - **How much did this single run cost?** — the top of the trace shows the total; the leaves show the per-call breakdown. ## Common questions ### "My trace shows nothing" If `init()` was never called or the API key is missing, the SDK runs in error mode and no spans are recorded. Check the SDK logs for `NullRunAuthenticationError`. ### "My trace is incomplete — only some spans show up" The SDK buffers events and flushes on a timer. If your process crashes before the flush, the in-flight spans are lost. Use `nullrun.shutdown(flush=True)` in your `finally` block to ensure everything reaches the gateway. ### "Why are some spans duplicated?" The SDK's auto-instrumentation emits one span per LLM call. If you also call `track_llm` manually for the same call, you'll see two spans. Pick one or the other — the auto-instrumentation is enough for the standard OpenAI / Anthropic / Gemini / Cohere clients. ## See also - [Workflow context](workflow.md) — how `workflow()` scopes spans - [Error handling](error-handling.md) — errors that span blocks - [Reference → SDK API → track_*](../reference/sdk-api.md) — manual span creation --- # Source: https://docs.nullrun.io/concepts/control-plane/ # File: docs/concepts/control-plane.md title: Control plane (real-time control) maturity: stable description: Real-time WebSocket channel for kill, pause, and approval_resolved — the operator's runtime control surface for live agents. # Control plane (real-time control) The **control plane** is the live channel between the dashboard and your running agent. When you click **Pause**, **Resume**, or **Kill** in the dashboard, the signal reaches your SDK through the WebSocket push channel. Without the control plane, the dashboard would only tell the agent something happened on its next `/gate` call. With it, the agent learns in real time. ## What the dashboard can do From the workflow detail page (or the top-level **Workflows** list): | Action | Effect on the agent | |---|---| | **Pause** | Every call starts raising `WorkflowPausedException` (a `NullRunError` subclass). Resume to undo. | | **Resume** | Unpause — calls resume normally. | | **Kill** | Every call raises `WorkflowKilledInterrupt` (alias `NullRunWorkflowKilledError`). The agent loop dies. | For the agent, the difference between Pause and Kill: - **Pause** — recoverable. The agent can catch `WorkflowPausedException`, do clean-up, and either retry or wait. - **Kill** — terminal. The exception inherits from `NullRunError` and is caught by `except Exception:` like every other SDK error — handle it explicitly if you need to checkpoint state before the process exits. The agent doesn't have to wait for the next `@protect` call to learn. If it's mid-LLM-call when you click Kill, the SDK raises the exception at the next yield boundary inside the agent's loop. ## How the signal reaches your SDK The dashboard pushes signals over a WebSocket connection that the SDK opens automatically when `init()` runs. The connection is authenticated with the same API key the SDK uses for `/gate` and `/track`, plus HMAC signature verification. The SDK keeps the connection alive with background heartbeats. If the WebSocket disconnects (network blip, firewall, gateway restart), the SDK falls back to polling `GET /api/v1/status/:workflow_id` once per second until the WebSocket comes back. From the agent's perspective, the control plane still applies — kill/pause still arrive on the next gate or yield boundary. The control-plane transport is auto-negotiated by the SDK — WS push in production traffic with HTTP-polling fallback when the WS connection drops repeatedly. You don't need to opt in or pass any flag to `init()`; the SDK handles both transports internally. For most agents this is invisible: `init()` opens the WS, and the gateway's Pause / Kill / `approval_resolved` signals arrive in real time without any further setup. ## What your agent sees {#how-the-sdk-reacts} The two exceptions your agent code will encounter: ```python from nullrun import WorkflowKilledInterrupt @nullrun.protect def my_agent_step(prompt): # ... agent logic ... return result try: my_agent_step("do something") except NullRunWorkflowKilledError: # Operator killed the workflow. Re-raise, or checkpoint then re-raise. raise except WorkflowPausedException: # Operator paused the workflow. Wait or exit cleanly. raise ``` Both signals inherit from `NullRunError`, so `@guarded` catches both (prints catalog wording, exits 1). To handle kill distinctly — checkpoint state, notify a supervisor, then exit — wrap the un-`@guarded` call in your own try/except. See [Error handling → Kill signal](../concepts/error-handling.md#kill-signal) for the recommended handler shape. For Pause, you have more flexibility. Most production agents catch `WorkflowPausedException`, save their state to durable storage, wait a few seconds, and resume. Some simply exit and let a supervisor process restart them when the workflow is unpaused. ## Approval events The same WebSocket push channel carries the second event type the SDK needs: **`approval_resolved`**. When the gate returns `decision = require_approval` on a `/gate` call, the parked SDK agent's thread waits on a `threading.Event` until the operator clicks Approve or Deny on the dashboard. The `approval_resolved` WS push wakes the event; the SDK resumes the agent with the operator's outcome. The complete approval flow is documented in [Human approval → Approval resume flow](human-approval.md#approval-resume-flow). If the approval timeout expires, the SDK raises `WorkflowKilledInterrupt`. There is no silent approval — operators must decide explicitly. ## What if the SDK is disconnected? If the WebSocket is down and polling is also blocked, the SDK can't learn about a kill until the next `/gate` call. In practice this window is at most one LLM-call duration — typically seconds, never minutes. The dashboard records the kill timestamp. When the SDK reconnects, it queries the workflow's state and acts on the most recent kill — even if the kill happened during the disconnection. The agent picks up the kill on the next call, with the original timestamp preserved in the audit log. ## Common operations ### Pause a runaway agent 1. Open **Workflows** in the sidebar. 2. Find the row whose status is **Active** but whose spend is suspiciously climbing. 3. Click the row, then click **Pause**. 4. The dashboard shows "Pause sent" with the timestamp. 5. Within ~1 second, the agent stops calling LLM. ### Resume after a pause 1. Same workflow page. 2. Click **Resume**. 3. The agent's next call succeeds. ### Kill an agent that won't stop 1. **Workflows** → workflow row → **Kill**. 2. The agent receives `WorkflowKilledInterrupt` (or its typed alias `NullRunWorkflowKilledError`) on the next yield point inside its loop. See [Error handling](../concepts/error-handling.md#kill-signal). 3. The signal inherits from `NullRunError`, so a bare `except Exception:` arm catches it. If you want a clean shutdown on kill, catch the typed exception **explicitly** and re-raise it — the kill contract is "operator's word is final". ### Verify the signal arrived After clicking Pause / Kill, the workflow's status flips immediately in the dashboard. If the agent doesn't respond, check the SDK logs — the WebSocket connection state is logged at startup and on every reconnect. ## See also - [Workflows → how to control one](workflow.md#how-to-control-one) - [Human approval](human-approval.md) — similar flow for tool approvals - [Troubleshooting](../troubleshooting.md) — "why did my workflow pause without me doing anything?" --- # Source: https://docs.nullrun.io/concepts/api-keys/ # File: docs/concepts/api-keys.md title: API keys maturity: stable description: Scopes, two-phase rotation, revocation, and the binding between an API key, its workflow, and its policy cache. # API keys An **API key** is how your code authenticates with the NullRun gateway. The key identifies a single workflow, gives the agent the permissions it needs, and (optionally) expires on a date you choose. ## Where you see it in the dashboard API keys live under **Access → API keys** in the left sidebar. The counter at the top of the page (`N / `) tells you how many keys your org has versus your plan's cap. The page shows every key with its name, workflow, last-used timestamp, and expiration date. ## The mental model Each workflow needs at least one API key to run. The key is what the SDK uses to identify itself when it talks to the gateway. The gateway uses the key to look up: - Which workflow is calling (so it can apply the right policies) - Which permissions the key has (`gate` / `execute` / `track` / `verify`) - Whether the key is still valid (not revoked, not expired) You mint keys through the dashboard, paste them into your application's environment, and the SDK takes care of the rest. ## How to create a key 1. **Access → API keys → New API key**. 2. Pick a workflow to bind the key to. The dropdown lists every workflow in your org. (Each key is **workflow-scoped** — one key represents one agent run, not one workspace.) 3. Pick an **Expires** window: **Never** (default), **24 hours**, **7 days**, **30 days**, or **90 days**. Keys without an expiration are valid until revoked. 4. Click **Create**. Scopes (`gate` / `execute` / `track` / `verify`) are auto-assigned to every new key and are not user-customizable in the dialog — the gateway needs all four to do its job. The dashboard shows the new key value **once** — a string starting with `nr_live_...`. Copy it into your secret manager **immediately**. The dashboard will never show it again.
API keys list with the New key button highlighted in the top right. API keys list with the New key button highlighted in the top right.
API keys · New key
New API key dialog open — Key name field, Workflow dropdown, Create button. New API key dialog open — Key name field, Workflow dropdown, Create button.
API keys · New key dialog
## What's in the response When you create a key, the dashboard shows: - **Key** — the public value (`nr_live_xxx...`). Use this in your SDK. - **HMAC secret key** — a second 32-byte hex string for request signing. Treat it like a password; never commit it to source control. The SDK stores it under `NULLRUN_SECRET_KEY`. - **Key prefix** — the first 12 characters, used in list views. - **Workflow** — the bound workflow (you picked this on creation). - **Scopes** — the permissions you granted. The dashboard shows the full key and secret **exactly once**. After you close the modal, the values are gone forever. If you lose them, you must rotate the key (see below). ## How the SDK uses the key The SDK needs two values from you: ```bash title="env" export NULLRUN_API_KEY=nr_live_xxx... export NULLRUN_SECRET_KEY=... ``` The `api_key` is the public value the SDK sends on every request. The `hmac_secret` is used for HMAC-SHA256 request signing — the gateway verifies every request came from a holder of the secret. In production deployments, HMAC is required. Without it, every SDK request returns 401. You can pass the API key directly to `init()`: ```python import nullrun from nullrun import init init(api_key="nr_live_xxx...") ``` Or set it via environment variable before the SDK starts: ```bash title="env" export NULLRUN_API_KEY=nr_live_xxx... python my_agent.py ``` The HMAC secret is read from `NULLRUN_SECRET_KEY` in the environment. It cannot be passed to `init()` — you set it once per process. ## Scopes Each key has a list of permissions — what it can do. Only the four values below are accepted; any other value is rejected at the API key creation step. | Scope | What it allows | |---|---| | `gate` | Call `/api/v1/gate` (the policy decision endpoint). Required for any `@protect`-wrapped call. | | `execute` | Call `/api/v1/execute` (the post-approval re-check after a `require_approval` decision). | | `track` | Call `/api/v1/track` (the spend tracking endpoint). Required for any LLM call. | | `verify` | Call `/api/v1/auth/verify` (the auth handshake on first use). Almost always needed. | | `*` | Wildcard — all of the above. The default if you don't specify. | For most agents, the defaults work. A telemetry-only ingestor needs just `track`. A read-only CI checker needs just `verify`. ## How to rotate a key Rotating creates a new key and invalidates the old one. In-flight calls finish normally. ## How to revoke a key Revoking means deleting the key. Useful when: - The key was leaked publicly - The agent is decommissioned - The workflow is being deleted Use `POST /api/v1/orgs/{org_id}/api-keys/{key_id}/rotate` first to generate a replacement, then `DELETE` the old one. The key stops working immediately — no grace period. ## Listing and searching The **API keys** page lists every key in your org. You can search by name (substring match), filter by workflow, or filter by status (active / revoked). Each row shows: - **Name** — what you set when creating - **Workflow** — the bound workflow - **Prefix** — first 12 characters of the key (`nr_live_abc...`) - **Last used** — when the SDK last made a request with this key - **Expires** — when the key stops working (or "Never") - **Status** — active / revoked Click a row to see full details. The full key value is never shown again — only the prefix. ## Common questions ### "How many keys do I need?" One per workflow, minimum. For production: - **One key per environment** — separate keys for production, staging, dev. Makes it easy to revoke staging without affecting production. - **One key per service** — if your agent runs in three containers, give each its own key. Makes it easy to rotate one without restarting the others. Do not disable the gate in production — you'll lose enforcement. ### "Can I share a key between two workflows?" No. Each key is bound to exactly one workflow at creation time. If you need the same agent logic against two workflows (for example, A/B testing), create two keys and switch between them based on your A/B routing. ### "What happens when my key expires?" The key stops working at the expiration timestamp. Calls return `401 api_key_expired`. Rotate the key (which generates a new secret but keeps the same key value) or create a new key entirely. ### "Can I see who used a key?" The **Last used** column shows the most recent activity. The audit log shows every individual call. The audit log records the key prefix, not the full key — so you can correlate usage without exposing the secret. ## See also - [Workflows](workflow.md) — what the key is bound to - [Troubleshooting](../troubleshooting.md) — "why am I getting 401?" - [Configuration](../getting-started/configuration.md) — env vars for keys --- # Source: https://docs.nullrun.io/concepts/policies/ # File: docs/concepts/policies.md title: Policies maturity: stable description: How BudgetLimit, RateLimit, ToolBlock, and LoopDetection policies are aggregated — most-restrictive-wins semantics across scopes. # Policies A **policy** is a rule attached to your organization or a single workflow. In the dashboard they live under **Governance → Policies**. Each policy answers one question: - "Is this call allowed, blocked, or does it need a human to approve?" ## What you see in the dashboard The **Policies** page lists every policy in your org. Each row shows: - **Name** — you set this when you created the policy - **Type** — what the policy caps (see the table below) - **Scope** — applies to the whole org, or only one workflow - **Active** toggle — on/off without deleting - **Effective from** — when the policy was last edited Click a policy to edit it. Changes apply to the next gate call — there's no need to redeploy your agent. ## The three policy types | Type | What it controls | Example value | |---|---|---| | **BudgetLimit** | Maximum spend per workflow per period | `5000` ($50.00) | | **RateLimit** | Maximum calls per minute | `60` (one call per second sustained) | | **ToolBlock** | Tools the agent must not call | `["send_*", "db.drop", "stripe.charge"]` | Each type has a JSON config payload — see the [Tool policies](tool-policies.md) page for the glob-match syntax inside `ToolBlock`. ## BudgetLimit — extra fields A `BudgetLimit` policy can carry these optional fields: | Field | Default | What it does | |---|---|---| | `enforcement_mode` | `"Hard"` | `Hard` blocks on budget exceeded. `Soft` allows a bounded overdraft when an active chain is present. | | `max_overdraft_cents` | `0` | Maximum overdraft in cents (per-org aggregate). Both `cents` and `percent` apply — the lower cap wins. | | `max_overdraft_percent` | `0` | Maximum overdraft as percent of the budget. | | `max_chain_duration_seconds` | `3600` | Maximum duration of a chain started under this policy before the gate refuses. | The gate reads these fields from every applicable `BudgetLimit` and uses **most-restrictive-wins**: `enforcement_mode` (Hard > Soft), `max_overdraft_cents` (min), `max_overdraft_percent` (min). ### Soft mode requirements Soft mode requires **all three**: 1. The policy uses `enforcement_mode = Soft` (not Hard) 2. An **active `chain_id`** exists (declared via `with chain(...)`) 3. The projected cost stays within `max_overdraft_cents` and `max_overdraft_percent` If any of the three is missing, soft mode is unavailable and the gate behaves as Hard. Multiple parallel chains on the same org share one overdraft counter — N concurrent chains do **not** multiply the overdraft cap. A chain dies on the first of: `op="end"`, 5 minutes of `/gate` inactivity (idle TTL), or exceeding `max_chain_duration_seconds`. Chain time is read server-side, eliminating clock skew between backend nodes. ## Aggregation When two policies in the merged set compete, the engine picks the **most restrictive** one for numeric caps and the **union** for tool patterns. | Field | If two policies disagree | |---|---| | `budget_cents` | The smaller number wins | | `max_calls_per_minute` | The smaller number wins | | `enforcement_mode` | `Hard` beats `Soft` | | `max_overdraft_cents` | The smaller number wins | | `max_overdraft_percent` | The smaller number wins | | `tool_pattern` / `blocked_tools` / `tools` | Both lists are merged (a tool blocked anywhere is blocked everywhere) | You can't accidentally un-block a tool the org blocks. There is no "allow" rule that overrides a "block" — the system is conservative on purpose. ## Org-level vs workflow-level A policy has one of two scopes: - **Org** — applies to every workflow in your organization. Useful for "all our agents must block `db.drop`" or "everyone gets 60 calls/min". - **Workflow** — applies to one workflow only. Useful for "this specific agent gets $500/month" or "this one agent can call `send_email`". Both scopes apply at the same time. There's no "overrides" — both sets of rules run together. The dashboard's **Effective policy** tab on a workflow page shows the merged set. ## Templates The dashboard ships with **templates** — pre-built policies for common patterns. To enable one: 1. On the **Policies** page, click **Templates**. 2. Pick a template (e.g. "Cap dev workflow at 100c/min" or "Block all write tools"). 3. Click **Enable**. The template materialises as a real policy in your org using its config and name. Disable reverses it. Templates save you from hand-authoring JSON. ## Plan gating Some policy features are plan-restricted: | Feature | Available on | |---|---| | `BudgetLimit` policies | All plans | | `RateLimit` policies | All plans | | `ToolBlock` policies | Growth+ | | Approval rules with typed predicates | Growth+ | If you try to create a feature your plan doesn't include, the dashboard shows the feature greyed out with an "Upgrade" link. ## Approval rules — separate from ToolBlock Approval rules are **not** a `ToolBlock` policy with an `action = require_approval` field. They are a separate rule object with `tool_patterns`, a projected-cost threshold, a typed `BusinessImpact` predicate, and operational metadata. When a rule fires, the gate returns `decision = "require_approval"` and the SDK parks until the operator clicks Approve / Deny. See [Human approval](human-approval.md) for the full flow, the typed `action_digest` binding, and the WebSocket push resume path. ## How to create one 1. **Governance → Policies → New policy**. 2. Pick the type (BudgetLimit / RateLimit / ToolBlock). 3. Pick the scope (Org or specific workflow). 4. Fill in the config. The dashboard validates the JSON in real time and shows errors before you save. 5. Save. The policy is active immediately.
Policies list with the New policy button highlighted in the top right. Policies list with the New policy button highlighted in the top right.
Governance · Policies · New policy
To test a new policy before rolling it out broadly, scope it to one workflow. The dashboard's **Effective policy** tab on that workflow's detail page shows the merged result so you can see exactly what your agent will see. ## What gets logged Every policy decision is recorded in **Governance → Audit log**. You can filter by: - Workflow - Decision type (`allow` / `block` / `require_approval`) - Time window - Tool name (for `ToolBlock` matches) The audit log is the source of truth for "why did my agent stop working at 14:32 yesterday?". Pair it with [Traces](tracing.md) to see the exact request that triggered the decision. ## See also - [Tool policies](tool-policies.md) — the `ToolBlock` matching rules - [Budgets](budgets.md) — how `BudgetLimit` interacts with the period rollover - [Human approval](human-approval.md) — typed `BusinessImpact` rules that produce `require_approval` - [Workflows](workflow.md) — where the merged policy is applied --- # Source: https://docs.nullrun.io/concepts/tool-policies/ # File: docs/concepts/tool-policies.md title: Tool policies maturity: stable description: Glob patterns for tool names with a 4 KB cap per pattern, union semantics across applicable scopes, and validation traps to avoid. # Tool policies A `ToolBlock` policy decides which tools the agent is allowed to call and which it can't. In the dashboard these rules live under a policy of type **ToolBlock** — see [Policies](policies.md) for the general overview. This page covers how to write the patterns inside the policy. ## Where you see it in the dashboard When you create or edit a policy and pick **ToolBlock** as the type, the dashboard shows a JSON editor for the `tool_pattern`, `blocked_tools`, or `tools` array. The "Test pattern" preview at the bottom lets you paste a tool name and see whether any pattern matches — useful for debugging.
Policies list with the New policy button highlighted in the top right. Policies list with the New policy button highlighted in the top right.
Governance · Policies · New policy
## What a tool name looks like The agent calls tools by name. The canonical tool name format: | Type | Format | Example | |---|---|---| | Built-in tool | lowercase string | `bash`, `file_write`, `execute_code` | | MCP tool | `mcp://{server}/{tool}` | `mcp://filesystem/read` | | Custom tool | `custom:{name}` | `custom:my_tool` | The policy matcher is name-based. The SDK sends the tool name to the gate, the gate checks it against every active ToolBlock policy, and the verdict comes back as `allow`, `block`, or `require_approval`. ## How to write the patterns Each entry in a ToolBlock policy is one of: - **Exact name** — `"stripe.charge"` blocks only that one tool. - **Glob** — `"send_*"` blocks anything starting with `send_`. - **`*` alone** — blocks everything. Each entry is capped at **4096 bytes**. The cap exists because the matcher scans every pattern on every gate call — a 10 MB pattern would burn CPU on each call. The matcher runs case-insensitively against the canonical tool name. ## What a ToolBlock policy does A ToolBlock policy **blocks** tool calls whose name matches one of its patterns. There is no `action = require_approval` field on a ToolBlock — for "I want a human to approve before this tool runs", create an **approval rule** instead (see [Human approval](human-approval.md)). The two are separate rule objects; ToolBlock and approval rules don't share a configuration schema. ToolBlock is **always Hard**: it never lets through, regardless of the budget's `enforcement_mode`. See [Reliability matrix](../concepts/circuit-breaker.md#when-the-gateway-is-unreachable). If the gate cannot evaluate the ToolBlock check (Redis or policy cache unavailable), it fails closed — `403 TOOL_BLOCKED` (SDK `error_code = "NR-T001"`). The agent never runs an unverified sensitive operation. ToolBlock and approval rules are distinct rule objects — they don't share a configuration schema. A ToolBlock policy always *blocks*; to require human review first, use an approval rule instead. ## A worked example Suppose your agent has these tools: `tavily_search`, `send_email`, `db.write`, `db.drop`, `stripe.charge`, `read_file`. You want to: - Allow read-only operations (`tavily_search`, `read_file`) - Block destructive operations (`db.drop`, `stripe.charge`) - Require approval for any outbound communication (`send_email`) - Allow normal DB writes (`db.write`) but block `db.drop` Two ToolBlock policies and one approval rule: !!! info "The example below shows the request payload shape — what you POST to the dashboard / API. It's not the storage schema, just the public input format." ```json title="tool_block_policy.json" { "policies": [ { "name": "Block destructive", "type": "ToolBlock", "scope": "Org", "config": { "tool_pattern": ["db.drop", "stripe.*"] } } ], "approval_rules": [ { "name": "Outbound needs approval", "tool_patterns": ["send_email"], "action_label": "send email to customer", "expires_in_seconds": 300 } ] } ``` `send_email` now triggers the approval flow (a human clicks **Approve** in the dashboard before the call goes through). `db.drop` and `stripe.charge` are blocked outright. The two policies do not interact — ToolBlock checks and approval-rule checks are independent paths in the gate. ## Validation at policy creation The dashboard rejects invalid patterns at save time: | Error | Cause | Fix | |---|---|---| | `400 bare_string_pattern` | `"pattern": "send_*"` instead of `"pattern": ["send_*"]` | Always use an array, even for one entry | | `400 pattern_too_long` | An entry longer than the per-pattern byte cap | Split into multiple patterns | | `400 invalid_glob` | Contains control characters | Remove `\n`, `\r`, `\t` | ## Plan gating `ToolBlock` policies require the Custom Policies feature, which is on **Growth+** plans. Lite and Starter can have `BudgetLimit` and `RateLimit` policies, but not ToolBlock. On Lite / Starter, the dashboard shows ToolBlock policy creation greyed out with an "Upgrade" link. ## How to debug a block you didn't expect If your agent reports `error_code = "NR-T001"` (wire `TOOL_BLOCKED`) on a call you think should be allowed: 1. Open the workflow in the dashboard. 2. Click **Effective policy**. The merged set shows every ToolBlock pattern that could match. 3. Open the **Audit log** and filter by `decision = block` and the tool name. The log shows which pattern matched. 4. If a pattern is too broad (`*` matches everything), narrow it in the policy editor. 5. If the pattern is wrong entirely, deactivate the policy and re-create it with the correct list. ## See also - [Policies](policies.md) — the dashboard view, aggregation rules, and most-restrictive-wins semantics - [Sensitive tools](sensitive-tools.md) — the policy-driven way to express "this tool needs review" (not a built-in SDK list) - [Tool catalog](../reference/llm-tool-catalog.md) — common tool names with risk ratings - [Human approval](human-approval.md) — the approval-rule path, distinct from ToolBlock --- # Source: https://docs.nullrun.io/concepts/human-approval/ # File: docs/concepts/human-approval.md title: Human approval maturity: beta description: Bind approvals to a typed BusinessImpact predicate and a SHA-256 action_digest so the grant refuses if the action payload drifts. # Human approval Some operations need a human to click **Approve** before they run. Sending an email to a customer, moving money, deleting a record — operations where you want a paper trail and a conscious decision. In the dashboard, pending approvals live under **Approvals** in the sidebar. When the agent hits an approval rule, the call pauses. The agent stays paused until a human clicks **Approve** or **Deny**, or the approval times out. ## Approval rules — separate from ToolBlock Approval rules are **not** `ToolBlock` policies with an `action = require_approval` field. They are a separate concept with these fields: | Field | Purpose | |---|---| | `name` | Display name for the rule | | `tool_patterns` | Glob patterns matching the tool name | | `per_call_threshold_cents` | Projected-cost threshold (estimated tokens × model rate) | | `action_predicate` | Typed `BusinessImpact` predicate | | `priority` | Ordering for tied rules | | `expires_in_seconds` | How long the operator has to decide | | `action_label` | Display label shown in the dashboard | When an SDK calls a tool that matches an approval rule, the gate returns `decision = "require_approval"` and parks the SDK on a `threading.Event` until the operator clicks Approve / Deny. ## Predicate kinds Two predicate fields can fire the same approval rule: - **`per_call_threshold_cents`** — projected execution cost in cents, evaluated against the SDK-reported `estimated_tokens`. - **`action_predicate`** — a typed condition over a structured `BusinessImpact` extracted from the live function call. When both are set, the rule fires only when **both** pass. Either may be `None`, in which case it does not contribute. A rule with both `None` matches every call. ### Typed predicates Two predicate kinds are supported on `action_predicate`: 1. **`money_amount`** — per-call monetary threshold. ```json { "kind": "money_amount", "direction": "outflow", "operator": "gt", "threshold_minor": 5000, "currency": "USD" } ``` 2. **`tool_parameters`** — DNF over up to 5 named parameters with Equals / OneOf / NumericRange / Regex / Exists matchers. ```json { "kind": "tool_parameters", "trigger_logic": "any", "conditions": [ {"param_name": "refund_amount", "matcher": {"kind": "numeric_range", "min": 500, "max": null}}, {"param_name": "recipient", "matcher": {"kind": "regex", "pattern": "^(?!internal@).*"}} ] } ``` The `tool_parameters` predicate rides on the SDK's `ToolParamsExtractor`. With the default `include_all=True` mode, every kwarg of `@sensitive`-decorated functions flows into the predicate bag (positional args are dropped; `f64` / set / custom objects are filtered; PII-masked sentinels like `"***"` for `password` / `token` / `api_key` keys are stripped before wire). ## `action_digest` — tamper-evident binding When the gate fires an approval rule with a typed `BusinessImpact`, it computes a SHA-256 digest of the canonical-JSON `{"kind":"money_amount", direction, operator, threshold_minor, currency, extractor_id, extractor_version}` (Money variant) or the `ToolCallParams` envelope (ToolCall variant). The digest is stored on the approval row. After the operator clicks Approve, the SDK's post-approval `/execute` re-check sends the live `business_impact` and `action_digest` back to the gate. The grant consume is atomic — concurrent re-checks are serialized, and the gate surfaces `Allow` / `DigestMismatch` / `NotFound` / `Expired` / `ReplayRejected` outcomes. ## Approval resume flow The complete flow, end-to-end: 1. SDK sends `/api/v1/gate` with the live `BusinessImpact`. Gate evaluates rules. Match fires. 2. Gate creates a pending approval record with the `business_impact`, `action_digest`, and an `expires_at` set server-side (clamped `[1, 3600]` s from `expires_in_seconds`). The gateway then emits an `approval_required` alert to configured channels. 3. Gate returns `decision = "require_approval"` plus `approval_id`, `approval_timeout_seconds`, `approval_expires_at`. 4. SDK parks on `threading.Event.wait(timeout=approval_timeout_seconds)`. 5. Operator clicks Approve / Deny in the dashboard (or auto-deny timer fires). 6. Backend publishes `ApprovalResolved` event on the WS push channel. 7. SDK wakes the parked thread; agent resumes with the operator's outcome. If the WS push is silent for `approval_timeout_seconds`, the SDK **fails CLOSED**: `WorkflowKilledInterrupt` (alias `NullRunWorkflowKilledError`) is raised and the agent dies. A silent network must not silently approve a privileged action. There is no `/status` HTTP-poll fallback for approvals — deliberate, the operator's word is final. ## What you see in the dashboard The **Approvals** page lists every pending, approved, denied, and expired request. Each row shows: - The tool the agent wanted to call (e.g. `send_email`) - The workflow that requested it - The action digest (first 16 hex chars — full digest in tooltip) - For typed predicates: the rendered impact summary - `Money`: `Spend $499.00 USD · 4.99× above the $100.00 limit` - `ToolCall`: `tool:stripe.charge` + raw `params` key/value block - How long ago it was created - How long until `expires_at`
Approvals page listing every pending, approved, denied and expired request. Approvals page listing every pending, approved, denied and expired request.
Approvals
Click an approval to see the full context — what the agent was trying to do, the tool's arguments, and any notes you attached. ## How to approve or deny In the **Approvals** page, click an open request. You see: 1. The agent's goal (what it was trying to accomplish) 2. The tool it wants to call (e.g. `send_email`) 3. The typed impact summary or the projected cost 4. The action digest (the SHA-256 binding) 5. How long the approval has been pending Two buttons: - **Approve** — the gate releases the reservation, the agent's call resumes. On `/execute`, the gate re-checks the `action_digest` against the live payload and refuses on mismatch (returns `DigestMismatch`). - **Deny** — the gate rejects, the agent sees `WorkflowKilledInterrupt` (alias `NullRunWorkflowKilledError`). The agent can catch it and clean up; most agents don't. If you don't click either within the approval's `expires_at` window, the request expires. The SDK raises `WorkflowKilledInterrupt` (alias `NullRunWorkflowKilledError`) after `approval_timeout_seconds` (server-clamped `[1, 3600]` s). The agent can retry or give up. ## Notification channels When an approval is created, the gateway notifies every active channel configured on your org: - **Slack** — uses your org's installed Slack OAuth. - **Webhook** — generic HTTPS POST with HMAC-SHA256 signature (`X-NullRun-Signature`, 5-minute clock-skew tolerance, 10-minute nonce replay defence). Disable a channel per-user or per-channel under **Notifications** in the sidebar (the page has Channels, Alert rules, and an Event subscriptions matrix).
Approval rules page with the New rule button highlighted in the top right. Approval rules page with the New rule button highlighted in the top right.
Governance · Approval rules · New rule
## Programmatic approval (for automations) The dashboard is for humans. If you want a CI bot or on-call rotation to approve requests programmatically, the same endpoints are exposed via REST: ```bash title="approve_via_api.sh" curl -X POST "https://api.nullrun.io/api/v1/orgs/$ORG_ID/approvals/$APPROVAL_ID/approve" \ -H "Authorization: Bearer ***" # Or deny explicitly curl -X POST "https://api.nullrun.io/api/v1/orgs/$ORG_ID/approvals/$APPROVAL_ID/deny" \ -H "Authorization: Bearer ***" ``` Both endpoints are idempotent — calling approve on an already-approved request returns `409 approval_already_decided`; calling deny twice on the same request is a no-op. Use these in your incident-response automation: an approval surfaces in Slack, your bot detects the `risk_level = high`, and approves or denies based on your runbook. ## When to use approval instead of blocking Approval makes sense when: - The operation is sensitive but **you want the agent to be able to do it** under human review (sending customer emails, creating invoices, deploying builds). - The blast radius is bounded (a single email vs. an entire database drop). - You have someone on-call who can review within minutes. Blocking (not approval) makes more sense when: - The operation is never legitimate (`db.drop` in a read-only workflow). - The blast radius is unbounded (admin operations, mass deletes). - No one is on-call to review approvals in time. Approval is a feature, not a default. Most teams should default to blocking and switch specific patterns to approval as the need arises. ## What's logged Every approval decision is in **Governance → Audit log**. You can filter by: - Approver (which user clicked Approve/Deny) - Workflow - Tool name - Time window - Outcome (approved / denied / expired) The audit log is the source of truth for "who approved this?" — both for compliance and for incident review. The action digest is the immutable anchor that proves the operator approved the exact payload the SDK sent on `/gate` (not "any refund" — the exact amount and arguments). ## SDK-side extraction The `action_digest` is **produced** by the SDK at extraction time. For the Python SDK, this happens inside `@sensitive` when you attach an `impact=` extractor — see [Decorators & extractors → `money_outflow(...)`](../reference/decorators.md#money_outflow-typed-money-impact) and [Decorators & extractors → `tool_params(...)`](../reference/decorators.md#tool_params-free-form-argument-bag). The extracted `BusinessImpact` is canonicalised (keys sorted recursively, compact JSON, `nullrun/v1/business_impact:` prefix) and SHA-256-hashed; the digest flows onto the wire on both `/gate` and `/execute`. A drift between SDK and backend is a P0 security regression covered by the SDK's source-pin tests. ## See also - [Tool policies](tool-policies.md) — `ToolBlock` rules (no `require_approval` action; that's a separate entity) - [Sensitive tools](sensitive-tools.md) — when blocking is enough - [Workflows → operator controls](workflow.md#how-to-control-one) — Pause / Kill work the same way as approval - [API keys](api-keys.md) — how to mint a key bound to a workflow --- # Source: https://docs.nullrun.io/concepts/approvals/ # File: docs/concepts/approvals.md --- title: Approvals (UI surface) maturity: stable description: The terminal-feed approvals dashboard — pending rows, friction-level buttons, the click-to-Dialog detail, and the history tab. --- # Approvals (UI surface) The **Approvals** page is where a human reviews and decides every `require_approval` decision the gate returns. It lives at `/control-center/approvals` (sidebar badge counts pending requests) and is gated by the `approvals` plan feature — Growth and above. This page covers the **UI surface** — terminal-feed rows, the friction-level approve flow, the click-to-Dialog detail panel, and the history tab. The wire contract (action_digest, typed predicates, plan-tier gating) lives in [Human approval](human-approval.md). Programmatic decision-making (REST endpoints, idempotency, retry semantics) is at the bottom of this page; the API reference is in [HTTP API → approvals](../reference/http-api.md#approvals). ## Page layout — terminal feed The pending queue renders as a **terminal feed**: hairline-divided rows in the spirit of the audit log + terminal-window vocabulary, not bordered cards. Each row reads as a continuous log line; the operator's eye locks onto the icon-prefix marker before parsing the rest of the row. ### Status prefix markers The first character of every row is a marker that encodes status and tone: | Marker | Tone | Status | |---|---|---| | `●` | state-block | `pending` | | `✓` | state-allow | `approved` (history tab) | | `✗` | state-flag | `denied` (history tab) | | `⌧` | fg-muted | `expired` / `consumed` (history tab) | ### Row anatomy From left to right: 1. **Prefix marker + workflow name + actor label** ("requested by X"). 2. **Hero amount** — for money-kind approvals, the spend line is back on the row (reverted to inline from the dialog-only placement) with the ▲ N× above $X limit relationship encoder so the operator sees both the value and why it's over the limit in one glance. Tabular-nums at 28px semibold. 3. **Why this needs approval** — the rule label, deep-linkable to the rule's config page. 4. **Inline live countdown** — a colour-shifting bar + pipe + tabular `mm:ss` label that shrinks as the review window runs out. Colour flips green → amber → coral at 40% / 15% of the remaining window. 5. **Action button(s)** — see below. For **tool-call approvals** (money kind = `tool_call`), the hero amount is replaced by the operator-approved tool name + the raw parameter bag, so the operator sees exactly what the SDK is about to run. The `action_digest` is the tamper-evident binding, not a display artefact — the dashboard shows the bag verbatim, never reconstructed from the digest. When the SDK forwarded `tool_class="mcp"` annotations, the row also renders a class badge (`MCP tool` / `builtin` / `custom` / `unknown`) plus a chip row for `destructive`, `read-only`, `open-world` (each chip shows `yes` / `no` / `unknown`). ## Friction-level approve flow The action button label encodes the friction level — operators never fire an action without seeing the value they are approving: - **Low risk** → single-click `[ approve ]`. - **Medium risk** → `[ approve ]` → `[ type 499.00 to confirm ]`. - **High risk** → `[ approve ]` → `[ type 1,000.00 ]` → `[ type refund_customer to confirm ]`. The amount being approved is surfaced inside the button label itself, not only in the confirmation step. The deny path is a single click on every risk level — see the human-approval page for why deny is unconditional. ## Click-to-Dialog Clicking anywhere on a row (outside the action button) opens a Dialog with the full detail panel: - Hero summary (amount / tool name + parameter bag). - **Why this needs approval** — the matched rule's human-readable predicate (`amount ≥ $50 USD`, `ANY(amount ≥ 5000, region IN [EU,US])`). - **Technical details** accordion — open by default after 2026-08-31, because the closed chevron alone failed to signal that the rule_id / digest / execution_id rows lived behind the disclosure. Rows: Action fingerprint, Execution ID, Rule + rule label, Tool patterns, Per-call threshold, Rule priority (lower = higher), Review window, Trust level chip (`typed impact` / `LLM-cost only`), Rule created, and the rendered Action predicate. The Dialog intentionally has **no Approve / Deny controls** — the friction-level flow lives on the row, and the Dialog is for review, not decision. ## History tab The history view is the same page at `?tab=history` — a tab strip in the page header switches between **Pending** (default) and **History**. Old `/approvals/history` URLs redirect to `?tab=history` so existing links keep working. History rows are filtered to the last 30 days by default and support the same search / status filters as the pending feed. Resolved rows are grouped by outcome (`approved`, `denied`, `expired`, `consumed`) with the same prefix-marker vocabulary (✓ / ✗ / ⌧) so an operator can scan a week of decisions in one glance. ### Bulk toolbar A hairline-divided toolbar above the feed exposes **Approve all** and **Deny all** when more than one row is selected. Both bulk actions require the same friction-level confirmations as the single-row flow. ## Page chrome - **Plan gate** — the page itself renders a `TierGate` upgrade prompt for plans without the `approvals` feature. The sidebar link is also hidden for those plans. - **SSE live update** — every new approval request lands in the feed within a few seconds without refresh; the badge count in the sidebar updates in lockstep. - **Audit trail** — every approve / deny decision is recorded in the audit log (`Audit log` under **Governance**) with the decided_by UUID, decided_at timestamp, and the operator label (or `System` for server-side expiry). ## Programmatic approval For CI bots and on-call rotations, the same endpoints are exposed via REST and the page chrome has no opinion: ```bash title="approve_via_api.sh" curl -X POST "https://api.nullrun.io/api/v1/orgs/$ORG_ID/approvals/$APPROVAL_ID/approve" \ -H "Authorization: Bearer ***" # Or deny explicitly curl -X POST "https://api.nullrun.io/api/v1/orgs/$ORG_ID/approvals/$APPROVAL_ID/deny" \ -H "Authorization: Bearer ***" ``` The full endpoint catalog — idempotency rules (`409 approval_already_decided`), the post-approval `/execute` binding, and the digest-mismatch drift cases (`NR-A013` / `NR-A014`) — is in [HTTP API → approvals](../reference/http-api.md#approvals). ## Where to read next - [Human approval](human-approval.md) — wire contract, action digest, typed predicates, plan-tier gating. - [HTTP API → approvals](../reference/http-api.md#approvals) — REST endpoints for programmatic decision-making. - [Audit log](error-handling.md#audit-trail) — every decision lands in the hash-chained audit log; the operator + `decided_by` UUID + `decided_at` are searchable. --- # Source: https://docs.nullrun.io/concepts/error-handling/ # File: docs/concepts/error-handling.md title: Error handling maturity: stable description: The full NullRun exception hierarchy, kill-signal semantics, and the multi-layer fail-CLOSED contract that protects production traffic. # Error handling Errors in NullRun come in three layers, designed for three audiences: your code, your monitoring, and your end users. The SDK does most of the work — you pick how much of each layer to use. !!! tip "Quick reference" | Audience | Hook / Class | Catches | |---|---|---| | Your code | `except NullRunDecision` | Expected policy outcomes (budget, tool block, pause) | | Your code | `except NullRunInfrastructureError` | Transport / 5xx / auth / config failures | | Your code | `except NullRunWorkflowKilledError` (or `WorkflowKilledInterrupt`) | Operator kill — terminal; caught by `except Exception:`, handle explicitly if you need to checkpoint before exit | | Your monitoring | `@nullrun.on_error` hook | Every `NullRunError`, fired before propagation | | Your end user | `@guarded` / `format_user_message` | Friendly text from the catalog | ## Where errors appear in the dashboard Every error the SDK raises lands in **Governance → Audit log** — every decision ever recorded, hash-chained and filterable by workflow, time range, decision type, and tool name. The reason column shows `BUDGET_HARD_BLOCKED`, `TOOL_BLOCKED`, `RATE_LIMIT_EXCEEDED`, etc. Useful both for "what just happened?" and for compliance review / incident forensics.
Audit log page listing every gate decision ever made by the org. Audit log page listing every gate decision ever made by the org.
Governance · Audit log
The audit log is the source of truth for "did the agent call the right thing?". Pair it with [Traces](tracing.md) for full context. ## The three layers | Layer | Who consumes it | What they see | Purpose | |---|---|---|---| | **1. Structured exception** | Your Python code | Exception type, error code, what to do next | Your code decides: retry, fail, surface to UI | | **2. `on_error` hook** | Sentry / Datadog / logs | Same exception + context (workflow, tool, stage) | Observability: you see every error in your existing dashboards | | **3. `@guarded` / `format_user_message`** | End user | One friendly sentence from a catalog | The user gets a clean message, not a stack trace | The SDK ships all three. You decide how much to use. ## Layer 1 — the structured exception Every NullRun exception carries four fields your code can branch on: | Field | What it is | Example | |---|---|---| | `error_code` | Stable machine-readable identifier | `NR-B004`, `NR-R001`, `NR-T001` | | `user_action` | What to do next | `Wait 30s, then retry` | | `retryable` | True if retry-after-backoff makes sense | True for rate limit, False for budget | | `docs_url` | URL to the per-code docs page | `https://docs.nullrun.io/reference/errors#sdk-exception-hierarchy-python` | The full catalog lives in that reference page; the standard set is: - `NR-B004` — workflow budget exhausted - `NR-B002` — gateway 5xx - `NR-B006` — post-approval budget re-check failed on the same envelope as the original `/gate`. The SDK raises `NullRunBudgetRecheckFailedError`. Operator must re-approve or the workflow can no longer run. - `NR-R001` — per-workflow rate limit - `NR-R002` — rate-limit Redis unavailable - `NR-T001` — tool block list hit - `NR-CH001` — chain context invalid - `NR-W004` — workflow soft-deleted or killed - `NR-A003` — API key rejected - `NR-A010` — approval row exists, status `PENDING` — operator has not decided yet - `NR-A011` — operator explicitly denied the approval — terminal, request a fresh grant - `NR-A012` — approval expired (`expires_at` is in the past) - `NR-A013` — business-impact digest drifted since operator approval — re-approval required - `NR-A014` — capability digest drifted (silent capability-gain attack surface) — re-approval required - `NR-A015` — grant already consumed by a prior `/execute` (replay rejected) - `NR-P001` — wire-protocol version mismatch - `NR-O001` — actual cost > reservation + ε (HTTP 422) - `NR-X001` — generic catch-all raised when a policy block matches a code the SDK does not have a dedicated class for. Match on `NullRunBlockedException` and read `.error_code` if you want specific handling. For the exception classes used to surface these codes, see [Reference → Errors → SDK exception hierarchy](../reference/errors.md#sdk-exception-hierarchy-python). The wire code is still available via the response body or `.status_code` when you need it for metrics / dashboards. You catch a specific exception type and inspect the fields: ```python from nullrun.breaker.exceptions import RateLimitError @nullrun.protect def my_agent(prompt): try: return call_llm(prompt) except RateLimitError as exc: # exc.error_code = "NR-R001" # exc.retryable = True # exc.retry_after = 30 (seconds) # exc.upgrade_url = "..." (link to upgrade plan) time.sleep(exc.retry_after) return call_llm(prompt) ``` For most cases you don't need to import specific types — catching the parent `NullRunError` and reading `error_code` is enough. ## Layer 2 — the `on_error` hook For Sentry / Datadog / your log aggregator, register a hook that fires for every `NullRunError` **before** it propagates: ```python import nullrun import sentry_sdk @nullrun.on_error def _to_sentry(err, ctx): sentry_sdk.capture_exception(err, extra={ "code": err.error_code, "retryable": err.retryable, "stage": ctx.stage, "workflow_id": ctx.workflow_id, "tool_name": ctx.tool_name, }) ``` The hook fires **once per error**, in registration order. Hook exceptions are caught and logged at DEBUG — a misbehaving Sentry can't break your agent. The context object (`ctx`) carries: `stage` (init / transport / track / gate), `workflow_id`, `tool_name`, `api_key_prefix` (first 12 chars of the API key, never the full value), `correlation_id` (per-request UUID), `timestamp`, `extra` (vendor-specific dict). Multiple hooks are supported: ```python @nullrun.on_error def _to_sentry(err, ctx): ... @nullrun.on_error def _to_log(err, ctx): log.warning("NullRun error", extra={"code": err.error_code}) ``` The hook fires for every `NullRunError` subclass — **including the kill signal** (`WorkflowKilledInterrupt` and its typed alias `NullRunWorkflowKilledError`). If you want to skip kill inside the hook, filter on `error_code` (`"NR-W002"`). ## Layer 3 — `@guarded` and `format_user_message` For scripts that just want "run the agent and print a friendly message on failure", use the zero-boilerplate helpers: ```python from nullrun import init_or_die, guarded, protect, shutdown init_or_die() @guarded @protect def my_agent(prompt): return call_llm(prompt) if __name__ == "__main__": try: print(my_agent("What does NullRun do?")) finally: shutdown() ``` What your terminal looks like on a rate-limit hit: ``` $ python my_agent.py Too many requests. Please wait a moment and try again. $ echo $? 1 ``` `@guarded` catches every `NullRunError` — which now includes the kill signal (`WorkflowKilledInterrupt` / `NullRunWorkflowKilledError` both inherit from `NullRunError`) — prints the catalog wording to stderr, and exits with code 1. To handle kill distinctly (for example, checkpoint state before exit), use the un-`@guarded` `protect()` form and add your own `except NullRunWorkflowKilledError:` arm. `@guarded` is for scripts and one-shots. For long-running services you want explicit handling — see [Server frameworks](#server-frameworks) below. ### Branded wording If you want your own error messages (e.g. "You've used all your support credits" instead of the default wording), call `set_user_message` once at startup: ```python import nullrun nullrun.set_user_message( "NR-B004", "You've used all your support credits. Upgrade to keep chatting.", ) ``` Overrides live in a per-process dict. They don't persist across processes and aren't synced to the gateway — they're presentation sugar on top of the catalog. ## Server frameworks For FastAPI / aiohttp / Flask / Django, you don't want `@guarded` (it's a CLI helper). Instead, catch the exception in your request handler and return an appropriate HTTP status: ```python from nullrun import NullRunError @app.post("/chat") async def chat(req: ChatRequest): try: return await run_agent(req.message) except NullRunError as exc: # Return the catalog wording as the user-facing message, # log the structured fields server-side. raise HTTPException( status_code=exc.status_code or 503, detail={"message": nullrun.format_user_message(exc), "code": exc.error_code} ) ``` The mapping from exception to HTTP status is documented in [Reference → Errors → Decision subclasses to HTTP](../reference/errors.md#mapping-decision-subclasses-to-http). ## Audit trail Every decision is recorded in the audit log; you can fetch the full log via the API. The audit log is the source of truth for "did the agent call the right thing?". Pair it with [Traces](tracing.md) for full context. ## What is NOT stored NullRun never persists: - **Prompt content** or **LLM response payloads**. The gate receives only `model`, `tool`, `tools`, `estimated_tokens`, and optional `business_impact` typed payload. - **Tool arguments** beyond the typed `BusinessImpact` extraction. Operators do not write JSONPath rules over tool payloads. - **MCP interaction payloads** — only the canonical tool name is logged. - **Card numbers, CVC, expiry month/year** — Polar is the merchant of record. Subscriptions carry only `payment_method_brand` and `payment_method_last4`. - **OAuth refresh tokens** — the IdP owns session lifetime. Email addresses and prompts are hashed or redacted at the log and trace-span boundary so plaintext does not reach the structured log store. Uppercase `KEY=VALUE` pairs are rewritten to `KEY=[REDACTED]` before bytes reach stdout. ## Kill signal The operator kill signal arrives as `WorkflowKilledInterrupt` or its typed alias `NullRunWorkflowKilledError` (recommended). Both inherit from `NullRunError`, so a bare `except Exception:` arm catches the kill alongside every other SDK error: ```python try: my_agent(prompt) except Exception: log.error("agent failed", exc_info=True) # WorkflowKilledInterrupt IS caught here. ``` If you want kill-specific handling — checkpointing state, notifying a supervisor, exiting with a clean reason — catch the typed alias **explicitly** and re-raise it after handling (the kill contract is "operator's word is final"): ```python from nullrun import NullRunWorkflowKilledError try: my_agent(prompt) except NullRunWorkflowKilledError: persist_state() raise except NullRunError: log.error("agent failed", exc_info=True) ``` `@guarded` catches kill via the standard `NullRunError` arm — it prints the catalog wording and exits 1. To keep the process alive on kill (checkpoint, notify a supervisor, then exit), use the un-`@guarded` `protect()` form with your own `except NullRunWorkflowKilledError:` arm above. ## See also - [Reference → Errors](../reference/errors.md) — full catalog - [Troubleshooting](../troubleshooting.md) — common questions and their fixes - [Use with FastAPI](../how-to/fastapi.md) — exception handling inside ASGI handlers - [Tracing](tracing.md) — how errors map to spans --- # Source: https://docs.nullrun.io/concepts/mcp-servers/ # File: docs/concepts/mcp-servers.md --- title: MCP servers (Action sources) maturity: stable description: What an Action Source is, how the dashboard tells verified servers apart from observed ones, and how to act on drift or stale catalogs. --- # MCP servers (Action sources) The **MCP servers** page in the dashboard is the operator's view of the Model Context Protocol servers your agents actually call. Each row in the page is one **action source** — the gateway's canonical name for "one MCP server (or built-in provider) that the SDK has talked to". The page lives under **Governance → MCP servers** in the sidebar. This page covers: - What an Action Source is and where it comes from. - How the dashboard splits **verification** (operator-registered, probe-driven) from **observation** (SDK-driven, last 30 days). - What **drift** means, the four states that qualify, and the one state that looks like drift but isn't. - How to **enroll** a discovered source and how to **write an approval rule** straight from a catalog action. For the canonical tool-name format (`mcp://server/tool`) used in policies and approvals, see [Tool policies](tool-policies.md). For how tool patterns and approval rules differ, see [Human approval](human-approval.md). ## What an Action Source is An Action Source is one entry in the unified table the dashboard renders for "every MCP-style server we know about". A row appears either because: - **You registered it** with a probe URL (`Add action source`), and the scheduler polled it and got a tool catalog back. - **The SDK called it** in the last 30 days. The observation helper fires on every `/check` call and adds the source automatically. A row from the first path has a `verification` block; a row from the second path has only an `observation` block and sits in the "Discovered but not registered" panel below the main list with an explicit **Enroll** CTA. ## Page layout The page header reads **Action sources** and shows the count of distinct sources in the observation window: - **N action sources in the last 30 days**, or - **No MCP action sources yet** when the org is brand new. The right-side action button is **Add action source** — clicking it opens a dialog where you paste the MCP probe URL and (optionally) a label. The scheduler polls new sources every 60 seconds until the first successful probe lands. ### Metric strip Four cards above the list give the operator a glance at the state of the org's tool surface: | Card | What it counts | |---|---| | **Action sources** | Total registered or observed in the window. | | **Verified** | Sources whose last probe returned a catalog matching observation. Sub-label shows `N unverified — verify now` (a deep link to `?filter=unverified`) or `all sources verified`. | | **Tool calls** | Total SDK-driven calls across every source in the last 30 days. | | **Drift** | Real drift only — see below. Sub-label distinguishes `N verification pending — not drift` from `Tools match the upstream catalog`. | While the page is loading, every card shows `—` rather than `0` — the dashboard never confuses "I don't know yet" with "the answer is zero". ### Filter bar Two controls above the list: - **Search source or action** — substring match against the source URL or any catalog action name. - **Status chips** — `All` / `Unverified` / `Stale` / `Drift`. Deep links via `?filter=` so e.g. the dashboard's "verify now" affordance drops the operator on the right view. ## What each row contains Each row is a hairline-divided tile with three blocks side by side or stacked: ### Verification block The left block answers "did the probe succeed?": - **Verified** — the last probe returned a catalog and it matches observation. Sub-line shows `Last verified · re-polls every `. - **Stale** — the last probe succeeded but is older than the re-poll interval. The next scheduled probe will refresh. - **Failed** — the last probe errored. The first 200 chars of the error body are shown inline (errors are usually JSON Schema validation payloads, multi-line HTML, or stack traces); a `Show full body` toggle reveals the rest. - **Never polled** — source was just added; first poll is pending. - **No probe URL registered** — the source exists only because the SDK called it. An inline **Enroll** CTA opens the same dialog used by `Add action source`. ### Observation block The middle block answers "did the SDK actually use this?": - **N distinct actions called** plus **M total calls in the last 30 days** when the SDK is active. - "The SDK hasn't called any action from this source yet" when the source is registered but unused. ### Catalog drilldown A `
` toggle below the row, labelled `Actions known (N)`, expands the catalog. Every row in the catalog shows: - The action name (`mcp://server/tool`). - An **Origin** badge: `Probe` (came from a successful probe), `Observed` (came from an SDK call), or `Probe + observed` (both). - A **Create approval rule** deep link to `/control-center/policies/approval-rules?prefill_source=…&prefill_action=…` so the operator can write a typed-predicate rule for one specific action without typing the path. ## Drift — and what isn't drift The Drift card counts **real drift only**. There are three states that qualify as drift: 1. **`unannounced` mismatch** — the SDK called actions the last probe never listed. Either the upstream catalog moved or someone added tools without re-probing. Write approval rules for any destructive verb before they are used. 2. **`disappeared` mismatch with prior SDK activity** — the probe once succeeded AND the SDK used to call this source, but the calls have stopped in the last 30 days. Usually means an upstream server upgrade. Review to confirm it isn't the agent silently failing over to a different server. 3. **`schema_drift === true`** — an action's input schema (keys and types of its argument bag) changed within the window. Pin the new schema before allowing the action. A **`disappeared` source with no prior SDK activity** is NOT drift — it is verification-pending. The probe never landed and the SDK never called anything, so we have no baseline to compare against. The Drift card surfaces these as `N verification pending — not drift`; the `Unverified` filter is the right view to work through them. A row that meets any of the three drift criteria renders a red-bordered **Drift callout** above the catalog drilldown, with a per-cause title and one-sentence body explaining what changed and what to do. ## Discovered but not registered Below the main list, sources the SDK called but you have not enrolled show up in a separate panel with the heading **Discovered but not registered**. Each row has an **Enroll** button that opens the add-source dialog pre-filled with the URL the SDK last used. Enrolling moves the source into the main list and starts the probe scheduler on it. ## Where to read next - [Tool policies](tool-policies.md) — the `mcp://server/tool` canonical-name format and the `ToolBlock` matching rules. - [Human approval](human-approval.md) — how to write a typed approval rule for one action (the deep link in the catalog drilldown lands here). - [Sensitive tools](sensitive-tools.md) — recommended starter patterns for destructive actions. --- # Source: https://docs.nullrun.io/concepts/alerts/ # File: docs/concepts/alerts.md --- title: Alerts maturity: stable description: Severity-tiered alerts surfaced in the dashboard — what the four KPI tiles mean, the filter chips, the snooze and dismiss flows. --- # Alerts The **Alerts** page surfaces every operational signal the gateway fires that the operator should look at — blocked incidents, threshold breaches, system events. It lives at `/control-center/alerts` and is gated by the `alerts` plan feature (Starter and above). On Lite plans the sidebar link is hidden and direct URLs render an upgrade prompt. The page reads from the same `alerts` feed that drives the sidebar bell badge (count next to **Alerts**), so dismissing or snoozing on the page brings the badge in line immediately. ## Page header The header reads **Alerts** with a running subtitle that breaks down the live state: ``` 12 active · 5 resolved · 3 critical · 6 warning · 2 info ``` Zero-count tiers collapse out of the subtitle so a clean org shows just `0 active · 0 resolved` with no visual noise. The counts come from the unified `AlertListMeta` payload (per ADR-037); pre-fix, the **info** tier was silently dropped from the subtitle, which hid the third of three severities from operators. The header action is **Dismiss all (N)** when at least one active alert exists; clicking it opens a confirmation Dialog ("Dismiss N active alerts? This action cannot be undone.") with Cancel and the destructive confirm button. ## Metric strip — not a row of four cards The four tiles above the filter chips are: | Tile | Sub-line example | |---|---| | **Action sources** | `Add one to begin` / `registered or observed` | | **Verified** | `3 unverified — verify now` (deep link to `?filter=unverified`) / `all sources verified` | | **Tool calls** | `Last 30 days across all sources` | | **Drift** | `Tools match the upstream catalog` / `4 verification pending — not drift` | If the list is still loading, every tile shows `—` rather than `0`, so the operator never confuses "I don't know yet" with "the answer is zero". ## Filter chips — two orthogonal dimensions Two filter dimensions run side by side above the list: - **Severity** — `All` / `Critical` / `Warning` / `Info`. - **Category** — `All` / `Prevented` / `System`. Severity is applied client-side (small enum, response shape unchanged); Category is also pushed to the server via `useAlerts({ category })` so the wire doesn't even ship the filtered-out rows. The combination of the two narrows the list independently — `Critical + Prevented` is the typical "what incidents did the breaker actually stop today" view. ## Alert row anatomy Each row is an `AlertCard` rendered as a hairline-divided block. The components from top to bottom: - **Severity left-border** — critical/warning/info accent. - **Icon-avatar** — incident type (Wallet for budget_block, ShieldAlert for tool_block, Gauge for spend thresholds). - **Title + body** — structured for `budget_block` rows: "Projected vs budget" stats + a horizontal threshold bar (current spend over threshold_cents, live from the wire). - **Timestamp + workflow name** — when the alert is workflow- scoped; system alerts omit the workflow chip. - **Snooze dropdown** — `1h` / `4h` / `24h` / `3d` / `7d`. The snoozed row is hidden from the active list until the snooze expires; a "Snoozed until …" footer line plus a live countdown appears on the row while the snooze is active. - **Dismiss** — single click, the row collapses into the Resolved section. ## Resolved section Beneath the active list, a **Resolved today** section shows dismissed alerts from the current calendar day. Each row renders the same `AlertCard` with a `resolved` flag — icon, title, body, but no Snooze / Dismiss actions. The resolved section is collapsed automatically when there are no resolved alerts. ## How to wire up alerts The **Set up alerts** button in the top-right of the header takes the operator to **Notifications** (`/control-center/notifications`) where they can: - Add Slack, Email, or Webhook channels. - Configure threshold rules (e.g. "spend reaches 80% of cap"). - Subscribe the org's events to the enabled channels. The Alerts page is the **read** surface; Notifications is the **configure** surface. They share the same wire, so a channel that fires lands both in the page and in the channel that the operator subscribed to. ## API hooks For automations, the same actions are exposed via REST and are mirrored in the audit log: - `POST /api/orgs/alerts/{id}/snooze` — `{ hours: number }` body. - `POST /api/orgs/alerts/dismiss-all` — dismiss every active alert for the org in one call. Use sparingly; the gateway still writes one audit row per dismissal. The plan-tier gate (`alerts` feature) is enforced server-side on every handler — a Lite user cannot dismiss alerts by hitting the API directly even if the page itself doesn't render. ## Where to read next - [Notifications](notifications.md) — how to add channels and subscribe events. - [Audit log](error-handling.md#audit-trail) — every dismiss / snooze is recorded as an audit row. --- # Source: https://docs.nullrun.io/concepts/notifications/ # File: docs/concepts/notifications.md --- title: Notifications maturity: stable description: Alert channels (Slack / Email / Webhook), threshold rules, and the per-event subscription matrix that controls where every signal lands. --- # Notifications The **Notifications** page is the configure surface for every outbound signal NullRun sends. It lives at `/control-center/notifications` in the sidebar and is gated by the Starter plan and above. The page has three sections in this order: 1. **Channels** — where signals can land (Slack / Email / Webhook). 2. **Alert rules** — threshold rules that fire when a value crosses. 3. **Event subscriptions** — which events reach which channels. It is the **configure** surface for [Alerts](alerts.md) (the read surface); channels created here appear in the Alert rules editor, and alerts dismissed on the Alerts page keep their wire-side notification enabled. ## Channels The top section is a 2-up grid of channel cards. Each card carries: - **Icon tile** + **Channel name**. - **Masked URL** in mono for webhook channels (Slack channels show the channel name; legacy email channels have been retired — see the migration note below). - **Status dot** — neutral for idle, faint for `last_sent` (so the operator can see at a glance whether the channel has fired recently). - **Edit link** + **Send test** icon-btn + **On / off** switch row. ### Adding a channel Click **+ Add channel** in the section header. The dialog supports: - **Slack** — OAuth-branded setup with Slack-specific help text. Pre-pivot rows that store `installation_id` / `channel_id` in config are preserved on edit so the existing connection doesn't break. - **Webhook** — generic HTTPS POST. The signing secret is optional; if set, the receiver verifies `X-NullRun-Signature` (HMAC-SHA256, 5-minute clock-skew tolerance, 10-minute nonce replay defence). Both Slack and generic webhook store on the backend as `channel_type: "webhook"` with `config: { type: "webhook", url: ... }` — Slack incoming webhooks accept POST JSON out of the box, so no Block Kit transform is needed for MVP. !!! note "Email variant removed" The Email channel type was removed on 2026-08-17 (P1-43) and is no longer available in the dialog. Legacy rows continue to render in the list but cannot be re-created. ### Send test The **Send test** button on each card posts a synthetic payload to the channel; the toast reports success or surfaces the backend's error message. Use this after creating or editing a channel to confirm your URL / OAuth installation actually delivers before relying on it for production signals. ## Alert rules The middle section is a list of threshold rules. Each rule renders as a card with: - **Left-border accent by severity** — info / warning / critical. - **Inline gauge bar** showing `last_observed_value / threshold_cents` live from the wire. - **Last-fired timestamp** + **enabled toggle** + **delete link** on the right. Rule editing is a form inside an `AlertRulesSection` dialog. The form shape mirrors the wire contract — name, severity, threshold in cents, and which channels the rule routes to. ## Event subscriptions matrix The bottom section is the matrix that decides which event reaches which channel. Events are grouped by area: - **Workflow activity** — `workflow.killed`, `workflow.paused`, `workflow.resumed`, `workflow.created`. - **Governance & access** — `approval.created`, `approval.decided`, `policy.changed`, `key.created`, `key.revoked`. - **Team** — `member.invited`, `member.joined`, `member.removed`. - **Digest** — weekly spend digest, monthly quota report. Each event is a row; each channel is a column. A cell shows a chip-dot when the event is enabled for that channel; no chip means the event is disabled for that channel. The right edge of each row has a master on/off toggle that flips every channel at once. A footer legend explains the chip-dot semantics (`●` = enabled, no chip = disabled). ## Plan gating The Notifications page itself renders a `TierGate` upgrade prompt for plans without Starter; the sidebar link is also hidden. The upgrade card links to **Billing & Plan** (`/control-center/billing`) and to the public pricing page (which hosts the comparison table) so operators can inspect feature deltas before committing. The plan-tier gate is enforced server-side on every `/api/alert_channels` and `/api/alert_rules` handler — a Lite user cannot POST to the API directly even if the page itself doesn't render. ## API hooks For automations, the same actions are exposed via REST: - `GET /api/alert_channels` / `POST` / `PATCH /{id}` / `DELETE /{id}`. - `POST /api/alert_channels/{id}/test` — fire a synthetic payload. - `GET /api/alert_rules` / `POST` / `PATCH /{id}` / `DELETE /{id}`. The full endpoint catalog is in [HTTP API → alert channels](../reference/http-api.md) (and the alert-rules section, when split out). ## Where to read next - [Alerts](alerts.md) — the read surface for what fired. - [Audit log](error-handling.md#audit-trail) — every channel and rule mutation is recorded as an audit row. --- # Source: https://docs.nullrun.io/concepts/billing/ # File: docs/concepts/billing.md --- title: Billing & Plan maturity: stable description: The merged Billing & Plan page — two tabs (subscription / payment / invoices, and quota / plan comparison / feature availability) sharing one fetch and one loading state. --- # Billing & Plan The **Billing & Plan** page combines subscription / payment / invoice history with quota / plan-comparison / feature-gate information under one URL. It lives at `/control-center/billing` in the sidebar. Before the merge, these were two separate pages (`/control-center/billing` and `/control-center/plan`) that shared most of their data. The merged view avoids two round-trips for the "should I upgrade and how do I pay" question the operator actually has. The active tab is encoded in the URL via `?tab=billing|plan` so the view is shareable + reload-safe. `?tab=plan` is the common inbound from upgrade prompts (the at-risk banner, the `TierGate`, the legacy `/control-center/plan` URL which now redirects). Anything else — and the default — is the **Billing** tab. ## The Billing tab Default landing. The header reads **Billing** with the subtitle "Subscription, payment method and invoice history." The hero card surfaces: - **Plan name + price** (e.g. `Starter · $49/mo · Renews Sep 30`). - **Status pill** — `Active` / `Trialing` / `Past due`. - **Manage subscription** button (opens the customer portal — see the migration note below). - **Update payment method** button. Below the hero: - **Current period end** — RFC-3339 timestamp. - **Payment method** — `Brand · last4` (Visa, Mastercard, Amex). Full PAN and CVC are never persisted; Polar is the merchant of record. - **Invoice history** — table of `Date / Invoice number / Amount / Status / Download`. Each PDF download wraps the blob in `URL.createObjectURL` and opens it; `window.open` can't attach the bearer token. !!! note "Customer portal retired" The Polar customer portal is no longer a product surface (2026-07-07). Both **Manage subscription** and **Update payment method** controls now render a `mailto:support@nullrun.io` deep-link with a pre-filled subject + body. Auto-checkout on first mount (when `pending_checkout_plan` is set in sessionStorage) is preserved. ### Lite orgs Lite is the free tier — there is no `billing_subscriptions` row. The hero reads `$0/mo · Free tier` and the payment-method / invoices sections collapse. ## The Plan tab Default landing when `?tab=plan` is set. The header reads **Plan** with the subtitle "Quota usage, plan comparison and feature availability." The tab surfaces: - **Quota usage cards** — every plan cap (workflows, policies, api_keys, seats, executions) with `used / limit` and a percentage. The executions card surfaces `executions_period_kind` so the operator knows whether the reset is **calendar_month UTC** (Lite) or the **Polar billing-cycle anchor** (paid plans) or the **lite_rolling_period**. - **At-risk banner** — when `quota.at_risk` is true, the page renders a callout with the projected hit date (`projected_hit_in_days`) and a CTA to upgrade. - **Plan comparison table** — every public plan catalog row, with the per-tier feature column (Approval rules, Audit log, MCP servers, Notifications, …). The current plan row is highlighted and disabled. - **Per-tier feature-gate panel** — a tighter view of which features are on/off at the current plan, with upgrade CTAs. The catalog comes from `GET /api/v1/plans`, which is unauthenticated and lives outside the `createApiClient` factory, so the page shell fetches it once and threads it through both tabs. ### Billing period toggle Above the comparison table, a `BillingPeriodToggle` switches between **Monthly** and **Yearly** price columns. The yearly column is computed via `computeYearlyPriceCents` so the discount matches the public pricing page. ## Auto-checkout (post-signup) When a user lands on `/control-center/billing` directly with a `pending_checkout_plan` in sessionStorage (post-signup flow), the Billing tab is the right destination — it shows the **Manage subscription** portal button after a successful checkout returns. The Plan tab doesn't get this side effect because the Plan tab is a comparison, not a payment surface. ## Upgrading from anywhere The same Billing & Plan page is where every upgrade prompt in the dashboard lands. The redirect contract is: - TierGate on a gated page → `?tab=plan`. - At-risk banner (any page) → `?tab=plan`. - `Upgrade plan` button in a feature empty-state → `?tab=plan`. All three deep links land on the Plan tab so the user sees the comparison table before being asked to pay. ## Where to read next - [Pricing page](https://nullrun.io/pricing) — public plan catalog (the Billing page reads from the same endpoint). - [Workspace & Org](organization.md) — for changing the org name / contact email / DPA acceptance. --- # Source: https://docs.nullrun.io/concepts/team/ # File: docs/concepts/team.md --- title: Team maturity: stable description: Members, invites, and the four-role matrix (owner / admin / operator / viewer) that governs what each teammate can do. --- # Team The **Team** page is the org-membership surface. It lists every member and every pending invite, surfaces the per-role capability matrix, and lets owners + admins invite or remove people. It lives at `/control-center/team` in the sidebar under **Access** and is gated by the `team` plan feature (Starter and above). ## Roles and what each can do There are four roles, ranked from most to least permissive: | Role | Capabilities | |---|---| | **Owner** | Everything an admin can do, plus transfer org ownership, delete the org, manage billing. There is always at least one owner; the last owner cannot be demoted. | | **Admin** | Invite / remove members, change roles for non-owner members, edit all policies, manage API keys, configure notifications. Cannot delete the org or change billing. | | **Operator** | Use the dashboard read/write — view workflows, executions, traces, audit log; approve / deny pending requests; create / edit policies and API keys. Cannot change team membership or billing. | | **Viewer** | Read-only — view workflows, executions, audit log, MCP servers, but cannot mutate anything (including approve / deny). | The full capability matrix is also rendered as a section inside the page (so an admin can confirm what they're granting before sending an invite). ## Members table The members table is sortable by role (asc / desc) and shows: - **Avatar** (initials in colour tile, or OAuth avatar for GitHub / Google users). - **Name + email** (masked via `maskEmail` for non-self rows to prevent screen-shoulder disclosure). - **Role** (select dropdown for owner / admin; non-owners show a select for the other three roles). - **Joined at** (RFC-3339 timestamp; older rows predating the migration render `—`). - **Remove** button (with confirmation dialog; the last owner cannot be removed). Owners and the current user are pinned near the top of the list regardless of sort order, so an admin never accidentally scrolls past themselves. ## Invites Above the members table is the **Invite** panel with an email field + role selector. The dialog rejects: - **Self-invites** — `You cannot invite yourself`. - **Existing members** — `This person is already a member`. - **Duplicate pending invites** — `Invite already sent to this email`. Only owners and admins see the invite panel; operators and viewers see a read-only members list. After sending, the invite appears in a separate **Pending invites** section below the members table. Each pending row shows: - **Email + role** + **Token** (copyable deep link). - **Last send status** — `pending` / `sent` / `failed` (with SMTP error text on `failed`). - **Last successful delivery** timestamp. - **Resend** and **Revoke** buttons. The invite link is `/invite?token=`; the deep link is stable until the invite is revoked or accepted. ## Seat quota The page header shows `N / seats used`. The seat count includes both active members and pending invites, so an admin sees the quota cost of every outstanding invite in real time. When the org hits the seat cap, the invite panel disables the send button and surfaces an upgrade prompt — `Team seat limit reached — N of N seats used` — that links to **Billing & Plan** (`?tab=plan`). ## Plan gating The Team page itself renders a `TierGate` upgrade prompt for plans without the `team` feature; the sidebar link is also hidden. Lite users cannot view the page, and the backend rejects every member / invite mutation with `403 seat_feature_disabled`. ## Audit trail Every invite send, resend, revoke, role change, and removal is recorded in the audit log with the actor's `decided_by` UUID. Admins can search the audit log by `action = team.*` to reconstruct who did what to whom. ## Where to read next - [Organization](organization.md) — for changing the org name, contact email, and DPA acceptance. - [Billing & Plan](billing.md) — the Plan tab is where seat upgrades are purchased. - [Audit log](error-handling.md#audit-trail) — every team mutation leaves a row. --- # Source: https://docs.nullrun.io/concepts/organization/ # File: docs/concepts/organization.md --- title: Organization maturity: stable description: Org name / slug / contact email, DPA acceptance, and the irreversible delete-org flow at the bottom of the page. --- # Organization The **Organization** page is the workspace-level config surface — the org's display name, slug, billing contact email, DPA acceptance, and the irreversible delete flow. It lives at `/control-center/organization` in the sidebar (no plan gate; every plan can edit). This page is **not** the billing surface — that lives at [Billing & Plan](billing.md). This page covers org identity and legal/compliance metadata only. ## Identity The top section is the org identity form: - **Name** — the display name shown in the dashboard header, email headers, and audit rows. Editable by owners; takes effect immediately on save. - **Slug** — the URL-safe identifier; immutable post-creation. Used in deep links, invite URLs, and webhook URLs. - **Contact email** — the address Polar and our support team contact for billing and security notifications. Owners can edit; the change triggers a confirmation re-auth (password or 2FA). The Save button is disabled when the form is unchanged or when the current user lacks the `owner` role. The success toast shows the new name; the page does not navigate. ## DPA / Compliance A dedicated section surfaces the org's Data Processing Agreement acceptance status. The section reads: - `accepted v2026-06-25 on 2026-08-15` (the **latest** acceptance), or - `not yet accepted` (when the org has never accepted). Below the latest line, a history of every prior acceptance — `version · accepted_at · accepted_method`. Acceptance methods on the wire are `in_product` (clicked through the dashboard) or `signed_pdf` (an offline acceptance that ops imported). When the latest version is unaccepted, the section renders an inline **Accept DPA** button. Acceptance is idempotent on `(org_id, dpa_version)` so a re-click is harmless — the backend returns the existing row with `created: false`. The `accepted_method` for the dashboard button is `in_product`. DPA fetch + acceptance failures do not block the page render — the section shows an inline retryable error and the rest of the page stays interactive. ## Members (linked from here) The page does not host the members table itself — the link in the team section takes the operator to the **Team** page (`/control-center/team`) under **Access**. Plan-gated to Starter+. ## Delete organization (irreversible) At the bottom of the page, the **Danger zone** section exposes the irreversible delete flow. The button is **Delete organization** and is only rendered when the current user has the `owner` role AND is the only remaining owner; otherwise the button is hidden and a one-line explainer tells the operator which precondition is missing. The confirmation dialog requires the operator to type the org name verbatim into a confirm field — `Type "acme-ai" to confirm`. Submitting fires a single `DELETE /api/v1/orgs/{org_id}` that cascades: - All workflows, executions, traces. - All policies, approval rules, alerts, audit rows. - All API keys (server-minted; the org loses access immediately). - All team memberships + invites. The dashboard then signs the operator out and redirects to `nullrun.io`. There is no undo. The 30-day audit-log retention still applies — audit rows are tombstoned rather than destroyed, so a post-delete forensic query through support is still possible within the retention window. After 30 days, the audit rows are purged. ## Audit trail Every identity change (name, contact email) and every DPA acceptance is recorded in the audit log with the actor's `decided_by` UUID. The irreversible delete-org flow writes a single audit row before cascade, marked `action = org.delete`, which is retained for the full retention window regardless of subsequent tombstones. ## Where to read next - [Team](team.md) — invites, role matrix, seat quota. - [Billing & Plan](billing.md) — for changing the plan or seat count. - [Audit log](error-handling.md#audit-trail) — every identity change leaves a row. --- # Source: https://docs.nullrun.io/concepts/profile/ # File: docs/concepts/profile.md --- title: Profile settings maturity: stable description: Personal info, two-factor auth, sessions, and the delete-account flow — the per-user surface, distinct from the per-org Organization page. --- # Profile settings The **Profile settings** page is the per-user surface: personal info, two-factor auth, password, sessions, and the delete-account flow. It lives at `/control-center/profile` in the sidebar (no plan gate). This page is **not** the org-level **Organization** page — that one edits org identity, slug, billing email, and DPA acceptance. Profile is the *person* who is currently signed in. ## Profile hero The hero card shows: - **Avatar** — initials on a colour tile (chosen server-side via `avatar_color`), OR the OAuth avatar (populated for GitHub / Google users, null for email-registered users). The OAuth image takes precedence over the initials fallback when both exist. - **Display name + email** + **Member since** timestamp. ## Personal info A two-column form with: - **Display name** — editable; saves on submit. - **Email** — editable, but the change triggers a confirmation re-auth (password or 2FA) and a verification email to the new address. Until the operator clicks the verification link, the email chip reads `Verification pending`. - **Resend verification email** — visible when the email is unverified. Disabled for 60 seconds after each click. A **Saved** indicator next to the Save button acknowledges the last write without taking up screen space. ## Security A two-column security section covers: - **Password** — visible when the user has a password (email- registered users). OAuth-only users (no `has_password` flag) see a one-line explainer + a **Set password** button that links to the password-set flow. The password change form requires the **current** password and validates the **new** password server-side. The dialog does not unmount on success — the user can change another field without re-entering the password. - **Two-factor auth (TOTP)** — three states: - **Not configured** — `Enable 2FA` opens a modal that generates a TOTP secret, renders a QR code, and asks for the first 6-digit code to confirm. The current password is required (`requireCurrentPassword`) before the secret is shown. - **Enabled** — `Disable 2FA` and `Regenerate backup codes` (and `Regenerate secret`, depending on the build). Both require the current password and a fresh TOTP code. - **Pending recovery** — for users who lost their device; the `Regenerate` flow can also be used to rotate secrets. The 2FA status object (`{ enabled: boolean, backup_codes_remaining: number, last_used_at: string | null }`) drives the chip in the section header. ## Sessions The **Session section** is a read-only card showing the information security needs to know: - **Last login** timestamp and IP. - **Active session count** (the current browser + every other logged-in device the user has). - **Sign out of all other devices** — single click, immediately invalidates every other session but keeps the current browser signed in. The action is logged in the audit log under `action = session.terminate_all`. - **Log out** — single click, signs out the current browser only. ## Display preferences (retired) The currency toggle was retired on 2026-08-11 (Audit P3-6 closure). Operators read "cost shown in EUR" as "I will be billed in EUR", which the wire contract forbids — backend billing is always USD-cents, and the section name + the per-card `USD` suffix already communicate that. The component is retained for any future surface that genuinely needs display conversion. ## Danger zone The **Delete account** button at the bottom of the page opens a modal that: - Requires typing the user's display name verbatim (`Type "Anatolii" to confirm`). - Requires entering the current TOTP code **if 2FA is enabled** (otherwise just the password). - Calls `DELETE /api/v1/me`, which tombstones the user, removes every session, and invalidates every API key the user personally minted. Org-level data (workflows, policies, audit rows) is untouched — that is the **Organization → Delete organization** flow, not this one. - Signs the operator out and redirects to `nullrun.io`. There is no undo. Audit rows authored by the deleted user remain in the org's audit log (`decided_by` UUID preserved) so a post-delete forensic query still works within the retention window. ## Where to read next - [Organization](organization.md) — the org-level identity page (different from this per-user page). - [Audit log](error-handling.md#audit-trail) — every profile change (password / 2FA / email / sessions) leaves a row. --- # Source: https://docs.nullrun.io/how-to/langgraph/ # File: docs/how-to/langgraph.md --- title: Langgraph description: Auto-instrument a LangGraph agent with @protect, or wrap nodes manually when you need fine-grained control over the gate decision. --- # Protect a LangGraph agent Install with the LangGraph extra: ```bash title="shell" pip install "nullrun[langgraph]" langgraph langchain-openai ``` `nullrun.init()` auto-instruments LangGraph — it attaches the NullRun callback to any compiled graph once `init()` runs. **No manual callback wiring needed** (the legacy direct-import path still works but is discouraged). ```python title="langgraph_agent.py" from langchain_openai import ChatOpenAI from langgraph.graph import END, MessagesState, StateGraph from nullrun import init init(api_key="nr_live_...") llm = ChatOpenAI(model="gpt-4o-mini") def chat(state: MessagesState): return {"messages": [llm.invoke(state["messages"])]} # `StateGraph(MessagesState)` replaces the deprecated # `langgraph.graph.MessageGraph` (removed in langgraph 1.0). graph = StateGraph(MessagesState) graph.add_node("chat", chat) graph.add_edge("chat", END) graph.set_entry_point("chat") app = graph.compile() result = app.invoke({"messages": [{"role": "user", "content": "Hi"}]}) ``` Every LLM call inside the graph is now cost-attributed and gated by your workspace policy. The same auto-instrumentation path works for any LangChain `Runnable` and most LangGraph node types. ## Manual wrapper (advanced) If you need to attach the callback manually — e.g. inside a library that re-compiles graphs after `init()` ran — the canonical wrapper is: ```python title="langgraph_manual_wrapper.py" from nullrun.toolbox.langgraph import wrapper app = wrapper(graph.compile()) ``` `wrapper` wraps the compiled app's `.invoke` and `.stream` methods to inject the NullRun callback into the LangChain `config["callbacks"]` list per call. The control-plane kill/pause subscription is **independent** — it's started automatically by `init()` and works for every `@protect` call in the process regardless of whether you used `wrapper()` or relied on the auto-instrumentation path above. ## See also - [Quickstart](../getting-started/quickstart.md) - [Examples → LangGraph](https://github.com/nullrunio/nullrun-examples/blob/master/examples/langgraph_basic.py) --- # Source: https://docs.nullrun.io/how-to/openai-agents/ # File: docs/how-to/openai-agents.md --- title: Openai Agents description: Install the nullrun[agents] extra and gate every tool call from an OpenAI Agents SDK workflow. --- # Use with OpenAI Agents Install: ```bash title="shell" pip install "nullrun[agents]" openai-agents ``` Wrap the `Runner.run_sync` call (or any sync / async runner) with `@protect`: ```python title="openai_agents_protect.py" from agents import Agent, Runner from nullrun import init, protect init(api_key="nr_live_...") @protect def ask(prompt: str) -> str: agent = Agent( name="assistant", instructions="Answer in one sentence.", ) result = Runner.run_sync(agent, prompt) return result.final_output print(ask("What is the capital of France?")) ``` `@protect` tracks every tool call the agent makes and halts the run if the workflow exceeds budget, hits a sensitive tool, or is rate-limited by the policy. ## See also - [Examples → OpenAI Agents](https://github.com/nullrunio/nullrun-examples/blob/master/examples/openai_agents_basic.py) --- # Source: https://docs.nullrun.io/how-to/crewai/ # File: docs/how-to/crewai.md --- title: Crewai description: Wrap CrewAI tools and tasks with @protect so the NullRun gate evaluates every crew action before it executes. --- # CrewAI Install (CrewAI **1.15+** required): ```bash title="shell" pip install "nullrun[crewai]" ``` The current patch subscribes to the crewai `EventBus` and translates each lifecycle event into a `runtime.track_event` call. ```python title="crewai_crew.py" import nullrun from crewai import Agent, Crew, Task nullrun.init(api_key="nr_live_...") researcher = Agent( role="Researcher", goal="Answer the question", backstory="Concise and accurate.", ) task = Task( description="What does NullRun do?", agent=researcher, expected_output="Two sentences.", ) crew = Crew(agents=[researcher], tasks=[task]) result = crew.kickoff() ``` The CrewAI integration automatically tracks crew / agent / task / tool lifecycle events. Token totals still come from the crew's usage metrics after kickoff — the SDK reports the canonical `(model, prompt_tokens, completion_tokens)` tuple on every billable row. When crewai's events module is not importable (pre-1.15 crewai or a stripped-down third-party build), only the per-event span bridge is skipped; the post-run cost attribution still works. ## See also - [LLM frameworks](llm-frameworks.md) - [Quickstart](../getting-started/quickstart.md) --- # Source: https://docs.nullrun.io/how-to/fastapi/ # File: docs/how-to/fastapi.md --- title: Fastapi description: Bind a FastAPI request to a NullRun workflow, propagate trace context, and map gate errors to the right HTTP status code. --- # Use with FastAPI Install with the FastAPI extra (pulls in `fastapi`, `starlette`, and the `httpx`-based transport the SDK needs at runtime): ```bash title="shell" pip install "nullrun[fastapi]" fastapi uvicorn ``` `nullrun.integrations.fastapi.install(app)` is a one-line setup that turns every NullRun exception in your agent API into a clean JSON response. Kill signals, budget caps, transport outages, and tool blocks all render as proper HTTP responses with end-user-safe text in the body. ```python title="fastapi_app.py" from fastapi import FastAPI import nullrun from nullrun.integrations.fastapi import install nullrun.init(api_key="nr_live_...") app = FastAPI() install(app) @app.post("/chat") @nullrun.protect def chat(message: str) -> dict: return {"reply": agent.run(message)} ``` ## What `install()` registers `install(app)` wires the SDK exceptions to FastAPI's handler chain: every `NullRunError` subclass — including `WorkflowKilledInterrupt` / `NullRunWorkflowKilledError` — is routed to the appropriate `app.add_exception_handler` based on its category. | Exception | Mechanism | HTTP | Body | | --- | --- | --- | --- | | `NullRunError` (budget, tool block, rate limit, soft block, etc.) | `app.add_exception_handler` | per `error_code` | `user_message`, `category: "decision"`, `retryable` | | Infrastructure errors (transport, 5xx, auth, config) | `app.add_exception_handler` | `503` | `user_message`, `category: "infrastructure"`, `retryable` | | `WorkflowKilledInterrupt` (alias `NullRunWorkflowKilledError`) | `app.add_exception_handler` | `503` | `user_message`, `category: "killed"` | `Retry-After` is set on the response whenever the exception carries a `retry_after` (gateway 429) or `resume_after` (workflow pause) attribute. ## HTTP status mapping | `error_code` | Category | HTTP | Notes | | --- | --- | --- | --- | | `NR-B004` | decision | `402` | `retryable: false` — user must upgrade or wait for next cycle. Covers `BUDGET_HARD_BLOCKED`, `BUDGET_SOFT_BLOCKED`, `BUDGET_OVERDRAFT_EXCEEDED`, `BUDGET_ANTI_DOS_RESERVED_CAP`, `BUDGET_PERIOD_NOT_STARTED` | | `NR-R001` | decision | `429` | `Retry-After` from `.retry_after` | | `RATE_LIMIT_REDIS_UNAVAILABLE` | decision | `503` | Aggregate rate limit fails closed | | `NR-T001` | decision | `403` | The action itself is forbidden | | `WORKFLOW_INACTIVE` | decision | `403` | Workflow was soft-deleted or killed | | `CHAIN_MAX_DURATION_EXCEEDED` | decision | `402` | Chain exceeded `max_chain_duration_seconds` | | `BUDGET_REDIS_UNAVAILABLE` | infrastructure | `402` | `retryable: true` — money math fail-CLOSED | | `BUDGET_DATA_UNAVAILABLE` | infrastructure | `503` | Approximate-budget lookup: all sources down | `WorkflowKilledInterrupt` always maps to `503`. See [Reference → Errors](../reference/errors.md) for the full catalog. ## Locale resolution By default the integration reads `Accept-Language` from the request. Pass a custom resolver when the locale comes from somewhere else (session cookie, JWT claim, upstream header): ```python app = FastAPI() # Locale from a session cookie, falling back to "en". install( app, locale_resolver=lambda req: req.cookies.get("locale", "en"), ) ``` A buggy resolver degrades silently to `"en"`. ## Custom exception mapping If you want to override `install()`'s defaults for a single endpoint (rare), wrap the agent call and map the exception yourself: ```python title="custom_mapping.py" from fastapi import HTTPException from nullrun import ( NullRunDecision, NullRunInfrastructureError, WorkflowKilledInterrupt, format_user_message, ) @app.post("/chat") @nullrun.protect def chat(message: str) -> dict: try: return {"reply": agent.run(message)} except NullRunDecision as exc: # Expected policy outcome — pass it to the client as-is raise HTTPException( status_code=exc.status_code or 403, detail={ "message": format_user_message(exc), "code": exc.error_code, "retryable": exc.retryable, }, ) except NullRunInfrastructureError as exc: # System failure — log to Sentry, return generic 503 sentry_sdk.capture_exception(exc) raise HTTPException( status_code=exc.status_code or 503, detail={"message": format_user_message(exc), "code": exc.error_code}, ) ``` `WorkflowKilledInterrupt` is caught by the `NullRunError` handler chain and surfaced as a `503` with `category: "killed"`. Catch it explicitly in your endpoint if you need to checkpoint before the response is returned. ## Response body shape ```json { "error_code": "NR-B004", "user_message": "You've reached the usage limit for this conversation. Please try again later.", "category": "decision", "retryable": false } ``` | Field | Type | Notes | | --- | --- | --- | | `error_code` | `string` | Stable machine-readable identifier | | `user_message` | `string` | End-user-safe text. Safe to render verbatim in a UI | | `category` | `"decision"` \| `"infrastructure"` \| `"killed"` | Coarse classification for client-side branching | | `retryable` | `bool` | Mirrors the SDK exception's `.retryable` | ## Per-deployment wording overrides To brand the wording for a single deployment, call `nullrun.set_user_message(...)` once at startup: ```python nullrun.set_user_message( "NR-B004", "You've used all your support credits. Upgrade to keep chatting.", ) ``` ## Limitations - `app.add_exception_handler` is last-wins — if you already register a `NullRunError` handler, `install()` overwrites it. Re-order your `install()` call to last if you need custom precedence. - Kill middleware is process-global state. The locale resolver is stored at module level; if you serve multiple FastAPI apps from one process with different locale policies, the last `install()` call wins. Per-app middleware (`app.add_middleware(NullRunMiddleware, locale_resolver=...)`) is the supported escape hatch. ## See also - [Quickstart](../getting-started/quickstart.md) - [Errors](../reference/errors.md) --- # Source: https://docs.nullrun.io/how-to/llm-frameworks/ # File: docs/how-to/llm-frameworks.md title: LLM frameworks maturity: stable description: Coverage matrix for OpenAI, Anthropic, Mistral, Gemini, Cohere, Bedrock, LangChain, LlamaIndex, CrewAI, AutoGen, and the raw openai SDK. # LLM frameworks `nullrun.init()` patches the underlying HTTP transport (`httpx`) and the agent framework modules it can detect in `sys.modules`. Every patch wraps the vendor import in `try/except ImportError`, so you can install one extra group without crashing on `init()`. In every case, the LLM call gets `track_llm` events automatically — **no `@protect` required for cost tracking**. `@protect` is the **gate** layer (budget pre-flight + kill / pause / sensitive-tool decision). > The Gemini vendor extra is `google-genai` (the actively maintained > package, ≥ 1.0); the older `google.generativeai` package is **not** > supported. ## Coverage matrix | Provider | Install extra | Auto-instrumented | Tested end-to-end | Patcher | | --- | --- | --- | --- | --- | | OpenAI (`openai`) | `nullrun[openai]` | ✅ | ✅ | `httpx` transport hook | | Anthropic (`anthropic`) | `nullrun[anthropic]` | ✅ | ✅ | `httpx` transport hook | | OpenAI Agents (`openai-agents`) | `nullrun[agents]` | ✅ | ✅ | `patch_openai_agents` | | Mistral (`mistralai`) | `nullrun[mistral]` | ✅ | ⚠️ extractor only | per-vendor extractor | | Gemini (`google-genai`) | `nullrun[gemini]` | ✅ | ⚠️ extractor only | per-vendor extractor | | Cohere (`cohere`) | `nullrun[cohere]` | ✅ | ⚠️ extractor only | per-vendor extractor | | AWS Bedrock (`boto3`) | `nullrun[bedrock]` | ⚠️ partial | ⚠️ extractor only | `boto3` event-stream hook | | LangChain (`langchain`) | `nullrun[langchain]` | ✅ | ✅ | `patch_langchain_callback` | | LangGraph (`langgraph`) | `nullrun[langgraph]` | ✅ | ✅ | `patch_langgraph_compiled` | | LlamaIndex (`llama-index`) | `nullrun[llama]` | ✅ | ⚠️ extractor only | `instrumentation.llama_index` | | CrewAI (`crewai`) | `nullrun[crewai]` | ✅ | ⚠️ extractor only | `instrumentation.crewai` | | AutoGen (`autogen-agentchat`) | `nullrun[autogen]` | ✅ | ⚠️ extractor only | `instrumentation.autogen` | | Raw `openai` SDK | `nullrun[openai]` | ✅ | ✅ | `httpx` transport hook | > "Tested end-to-end" means: a multi-roundtrip test exists that > verifies tokens flow from the vendor response into `/api/v1/track`. > "Extractor only" means the unit test covers the JSON parsing, but > no full integration test confirms the bytes-on-the-wire → track > chain. Verify against your real workload before relying on it. ## Install everything ```bash title="shell" pip install "nullrun[all]" ``` Installs every vendor extra. The `[all]` meta-extra lives at `pyproject.toml` and pulls every individual extra in one go. ## How the httpx transport hook works The httpx transport hook wraps the response handler for any HTTP client built on `httpx` (the `openai` SDK and the `anthropic` SDK both use `httpx` under the hood). On every response, the hook: 1. Reads the JSON body. 2. Extracts token counts from the vendor's `usage` block (`usage.prompt_tokens` / `usage.completion_tokens` for OpenAI, `usage.input_tokens` / `usage.output_tokens` for Anthropic). 3. Emits a `track_llm` event with the extracted tokens. The backend recomputes cost from the org's pricing policy — the SDK only reports token counts, never dollar amounts. ## Detection logic If your framework is installed, the SDK patches it automatically on `init()`. The detection logic walks `sys.modules` looking for known packages — `openai`, `openai-agents`, `anthropic`, `langgraph`, `langchain`, `mistralai`, `google-genai`, `cohere`, `boto3` (bedrock), `llama_index`, `crewai`, `autogen_agentchat` — and applies the appropriate patch. Order matters: if your code imports `openai` before `init()`, the hook is in place before the first request. If you import after `init()`, the SDK patches at import time on next `init()` call — or you can call `nullrun.patch()` explicitly. ## Provider-specific notes ### Anthropic Reasoning tokens (for o1-style extended-thinking models) are tracked at the reasoning rate configured in your pricing policy. The hook reads `usage.reasoning_tokens` when present. ### Mistral The hook watches `mistralai` ≥ 1.0 (`MistralClient` and `MistralAsyncClient`). Earlier `mistralai<1` clients have a different response shape; the extractor handles both with a duck-type check on `usage.prompt_tokens` / `usage.completion_tokens`. ### Bedrock Bedrock uses AWS event streams (`InvokeModelWithResponseStream`), not plain JSON responses. The hook attaches to the `boto3` event-stream parser. Token counts come from `invocationMetrics.inputTokenCount` / `outputTokenCount` in the final `messageStop` event. **Streaming-only** — non-streaming Bedrock calls must be reported via `track_llm` manually. ### LangGraph The `nullrun[langgraph]` extra wraps `Pregel.invoke` / `.ainvoke` / `.stream` / `.astream` so every node that calls an LLM goes through the gate. See [Protect a LangGraph agent](langgraph.md) for the canonical wiring pattern and the manual `wrapper()` escape hatch. ### CrewAI / AutoGen Multi-agent frameworks spawn sub-agents that each make their own LLM calls. The hook fires per call, so cost attribution lands in the right `agent_id` automatically (the framework passes `agent_name` through to the SDK contextvar). ## When auto-instrumentation can't see the call Some patterns bypass the auto-instrumentation: - Custom HTTP transport (not `httpx`) — use [`track_llm`](../reference/sdk-api.md#track_llm-manual-usage) - Streaming chunks where the SDK is constructed before `init()` — call `nullrun.patch()` after the late imports - A framework not listed above — file an issue at `github.com/nullrunio/nullrun-sdk-python` The catch-all `track_llm(input_tokens=…, output_tokens=…, model=…)` is the escape hatch for any of these. ## See also - [Protect a LangGraph agent](langgraph.md) — full LangGraph example - [Use with OpenAI Agents](openai-agents.md) — `openai-agents` extra - [Use with FastAPI](fastapi.md) — request-scoped SDK context - [Manual cost / event tracking](custom-tracking.md) — `track_llm` / `track_tool` / `track_event` --- # Source: https://docs.nullrun.io/how-to/cost-cap/ # File: docs/how-to/cost-cap.md --- title: Cost Cap description: Set a per-workflow hard cost cap with alert thresholds, and use Soft mode with an active chain to allow a controlled overdraft. --- # Set a hard cost cap A cost cap is the simplest way to make sure an agent can't blow past your budget. It works at two levels: 1. **Per-workflow** — set on a workflow, halts the run when cumulative cost exceeds the cap. 2. **Per-call** — projected cost from the SDK, rejects any single call that would exceed the cap. ## Per-workflow In the dashboard, open the workflow and set the budget. Or via the HTTP API: ```bash title="set_budget.sh" curl -X PATCH https://api.nullrun.io/api/v1/orgs/$ORG_ID/workflows/$WORKFLOW_ID \ -H "X-API-Key: *** \ -H "X-Signature: $(compute_hmac)" \ -H "X-Signature-Timestamp: $(date +%s)" \ -H "Content-Type: application/json" \ -d '{"budget_cents": 500}' ``` > Auth uses `X-API-Key` plus an HMAC-SHA256 signature over > `timestamp:api_key:body_hash` (see the > [HTTP API reference](../reference/http-api.md#authentication)) > and the SDK's `NULLRUN_SECRET_KEY`. Bearer session tokens are for > dashboard / admin endpoints only. Then in the SDK: ```python title="budgeted_run.py" import nullrun from nullrun import init, protect init(api_key="nr_live_...") with nullrun.workflow("my-workflow"): @protect def run(): ... ``` Cumulative cost > 500¢ → `NullRunBudgetError` raised on the next gate call with `error_code = "NR-B004"` (wire `BUDGET_HARD_BLOCKED`). For non-budget policy blocks (tool block, sensitive tool) the generic `NullRunBlockedException` is raised with `error_code = "NR-T001"` (wire `TOOL_BLOCKED`). See [Errors](../reference/errors.md) for the full catalog and the recommended `except` pattern. `max_budget_cents == 0` means **"no per-key budget configured"**, not "block everything" — the gate passes through to the org-level plan cap. See [Budgets → How to set the budget](../concepts/budgets.md#how-to-set-the-budget). ## Per-call The SDK does not project per-call cost on its own — the per-call cap is enforced by the workspace policy on the gateway. When the policy carries a per-call threshold, the `/gate` call rejects any single call whose projected cost would exceed the cap *before* the model is invoked. The SDK raises `NullRunBlockedException` with `error_code = "NR-B004"` (wire `BUDGET_HARD_BLOCKED`). If you need to skip pre-flight in tests, use a workflow with a low budget instead. ## See also - [Budgets](../concepts/budgets.md) — reservation lifecycle and the pre-flight `/gate` end-to-end - [Errors](../reference/errors.md) - [Examples → cost cap demo](https://github.com/nullrunio/nullrun-examples/blob/master/examples/cost_cap_demo.py) --- # Source: https://docs.nullrun.io/how-to/multi-agent/ # File: docs/how-to/multi-agent.md --- title: Multi Agent description: Run multiple agents in one workflow, propagate parent_trace_id, and aggregate cost across the team. --- # Run multiple agents (multi-key / multi-process) NullRun's `init()` is intended to be called **once per process**. The SDK's runtime is a process-scoped singleton — transport pool, WebSocket subscription, and event batch buffer. If you call `init()` twice in the same process, behaviour depends on the runtime implementation; the supported pattern is one `init()` per process and one process per workflow key. For everything beyond a single one-shot script, run **one process per key**. This page shows the three patterns that cover real workloads. ## Pattern 1 — multiple agents on one host (one process per key) The simplest production deployment. Each workflow gets its own process, its own env var, its own log stream. Common supervisors include systemd, Docker Compose, and Kubernetes — pick whichever fits the platform. The rule is the same regardless of supervisor: each process gets its own `NULLRUN_API_KEY` so the dashboard's **Workflows** view shows separate per-workflow spend, kill/pause works independently, and you can restart one without affecting the other. ## Pattern 2 — fan-out inside one container (multiprocessing.Pool) When you have one entrypoint but N workflows to run, use `multiprocessing.Pool` so each child initializes its own SDK runtime and its own key: ```python title="fanout.py" import multiprocessing as mp import nullrun from nullrun import init_or_die, protect def _agent_main(key: str, prompt: str) -> str: # Each child process initializes its own runtime with its own key. init_or_die(api_key=key) @protect def step(p: str) -> str: return your_llm_call(p) return step(prompt) def fan_out(jobs: list[tuple[str, str]]) -> list[str]: # jobs is [(key, prompt), ...] — one key per workflow. with mp.Pool(processes=len(jobs)) as pool: async_results = [ pool.apply_async(_agent_main, args=(k, p)) for k, p in jobs ] return [r.get(timeout=120) for r in async_results] ``` Each child gets its own copy of the SDK state, so each `init()` runs cleanly with no shutdown collisions. **Do not** call `init()` once at the parent and share the runtime across children — that's the multi-key-in-one-process anti-pattern and you'll get shutdown warnings the moment the first child finishes. Pick the pool start method that matches your platform (see the `multiprocessing` docs). The rule is the same regardless: each child is a fresh interpreter and runs its own `init()` independently. ## Pattern 3 — one entrypoint, multiple keys, hard process boundary If you have one CLI / API server that needs to handle requests for many workflows, route at the **process level** rather than the **function level**: ```python title="router.py" import os import subprocess def run_workflow(key: str, prompt: str) -> str: """Spawn a fresh subprocess for each request. Each one is its own SDK singleton, so multi-key isolation is automatic.""" result = subprocess.run( ["python", "agent.py", prompt], env={**os.environ, "NULLRUN_API_KEY": key}, capture_output=True, text=True, timeout=120, check=True, ) return result.stdout ``` The subprocess startup cost (~150 ms for `init()` + WebSocket connect) is the price for clean isolation. For high-throughput paths, see Pattern 2 — multiprocessing keeps workers warm in a pool. ## What doesn't work Calling `init()` more than once in the same process is not a supported pattern. The runtime singleton is process-scoped, and mixing multiple keys in one process leads to interleaved events on the wrong workflow. The supported alternative is one process per key (Pattern 1) or one subprocess per request (Pattern 3). ## What if I want a single dashboard view across all my processes? You don't need anything special — the dashboard already aggregates per workflow across all processes holding that workflow's key. As long as every subprocess binds to the **same** workflow (i.e. uses the same key), all their `/gate` and `/track` calls land on the same workflow record in the backend. The case where this **doesn't** hold is the "many workflows, one process" anti-pattern above: each `init()` call swaps the active key but the prior workflow's events have already gone to the prior key's workflow. ## See also - [Configuration → env vars](../getting-started/configuration.md) - [Concepts → API keys](../concepts/api-keys.md) — workflow-scoping and the `1:1` binding between key and workflow - [Concepts → Workflow context](../concepts/workflow.md) --- # Source: https://docs.nullrun.io/how-to/multi-agent-orchestration/ # File: docs/how-to/multi-agent-orchestration.md --- title: Multi Agent Orchestration description: Orchestrate sub-agents with shared kill semantics: a top-level trip propagates to every child through the control plane. --- # Multi-agent orchestration When one agent delegates to sub-agents — a LangGraph supervisor, a CrewAI crew, an OpenAI Agents `Runner` with handoffs, or a custom orchestrator — NullRun tracks each sub-agent independently under the same workflow. The rule is: **each `with workflow(...)` block creates its own budget pool. Nesting does NOT inherit budget — the inner block has its own pool.** ## LangGraph supervisor → sub-agents ```python title="langgraph_orchestrator.py" from typing import TypedDict from langgraph.graph import END, StateGraph from langchain_openai import ChatOpenAI import nullrun from nullrun import init_or_die, protect, workflow, shutdown init_or_die() llm = ChatOpenAI(model="gpt-4o-mini") class State(TypedDict): topic: str research: str draft: str @protect def research_node(state: State) -> State: """Sub-agent — its @protect is a gate for the LLM call only.""" out = llm.invoke(f"Research {state['topic']}") return {"research": out.content} @protect def writer_node(state: State) -> State: out = llm.invoke(f"Write a draft using: {state['research']}") return {"draft": out.content} def supervisor(state: State) -> str: return END # or "research" / "writer" with workflow("research-supervisor"): graph = StateGraph(State) graph.add_node("research", research_node) graph.add_node("writer", writer_node) graph.add_conditional_edges("supervisor", supervisor) graph.set_entry_point("supervisor") app = graph.compile() app.invoke({"topic": "LLM cost trends"}) ``` Each `research_node` / `writer_node` is `@protect`-wrapped, so the gate runs per node invocation, not per `app.invoke()` call. `@protect` on each sub-agent ensures the gate runs before the LLM call. The `with workflow("research-supervisor")` block scopes budget attribution to the `research-supervisor` workflow — every LLM call inside counts against that workflow's budget. Nesting `with workflow("research-subagent")` inside does **not** inherit the outer workflow's budget: each `workflow()` block creates its own `workflow_id` with its own budget pool. To share a budget across the whole orchestration tree, use ONE `workflow()` block around the orchestrator (the pattern above). To give each sub-agent an independent budget, use distinct workflow names — but they become separate budget pools: ```python title="separate_workflows.py" with workflow("research-supervisor"): with workflow("research-subagent") as research_wf: research_node(state) # counts against research-subagent's budget with workflow("writer-subagent") as writer_wf: writer_node(state) # counts against writer-subagent's budget ``` This is the **independent-pools** pattern — useful when sub-agents have distinct budget allocations (e.g. one sub-agent handles paid API calls, another is read-only), but you lose the "one cap protects everything" guarantee. Full LangGraph, CrewAI, and OpenAI Agents orchestration examples live in [`nullrun-examples/examples/`](https://github.com/nullrunio/nullrun-examples/tree/master/examples) — `langgraph_basic.py`, `crewai_basic.py`, and `openai_agents_basic.py` show the independent-pools wiring. ## Operator kill across the tree When an operator hits **Kill** in the dashboard, the WS push delivers a `state_change(killed)` to **every** connected SDK client holding the workflow's key. If multiple `@protect` calls are in-flight across the orchestration tree, they all receive the kill signal at their next yield boundary. See [Control plane → kill contract](../concepts/control-plane.md#how-the-sdk-reacts) for the wire-level details. ## Common pitfalls | Pitfall | Symptom | Fix | |---|---|---| | Missing `with workflow(...)` around the orchestrator | Each sub-agent gets its own ad-hoc workflow_id, no shared budget pool across the tree | Wrap the whole tree in one workflow block | | Each sub-agent has its own key | Sub-agents share nothing — kill signal only reaches the one bound to the killed workflow | Use one key for the orchestrator and let sub-agents inherit | | Catching only `Exception` around the orchestration loop with no kill handler | Kill still propagates but with no cleanup hook | Catch `NullRunWorkflowKilledError` explicitly first if you need to checkpoint sub-agent state | ## See also - [Workflow context](../concepts/workflow.md) — how `workflow()` scopes events - [Chain context](../concepts/workflow.md#chain-context) — soft mode for multi-step orchestrations - [Use with LangGraph](langgraph.md) — single-agent LangGraph example - [Use with OpenAI Agents](openai-agents.md) — single-agent example --- # Source: https://docs.nullrun.io/how-to/streaming/ # File: docs/how-to/streaming.md --- title: Streaming description: Use @protect on a stream iterator so the gate's Cancel decision can stop a live response the moment an overrun is detected. --- # Stream LLM responses The SDK tracks streaming responses correctly — every chunk is forwarded to your caller in real time, and the cost is reported from the final chunk (which carries the `usage` block). ## The pattern ```python title="streaming_agent.py" import nullrun from openai import AsyncOpenAI from nullrun import init_or_die, protect init_or_die() client = AsyncOpenAI() @protect async def stream_answer(prompt: str): stream = await client.chat.completions.create( model="gpt-4o-mini", messages=[{"role": "user", "content": prompt}], stream=True, ) async for chunk in stream: yield chunk.choices[0].delta.content or "" ``` The transport hook reads the final `usage` block before emitting `/track`, while forwarding chunks to your caller in real time. ## Long streams and soft mode A long stream that exceeds the chain idle TTL (300s) will be killed mid-chunk. The SDK sends a wall-clock heartbeat every **30 seconds** per policy (configurable in `[10s, 120s]`) — not per chunk. For multi-minute responses, use a `chain` context to keep the gate alive: ```python @protect def long_stream(prompt: str): with chain("my-long-stream", op="start"): stream = client.chat.completions.create( model="gpt-4o-mini", messages=[{"role": "user", "content": prompt}], stream=True, ) for chunk in stream: yield chunk.choices[0].delta.content or "" ``` For budget headroom, set `enforcement_mode = "Soft"` on the policy. See [Chain context](../concepts/workflow.md#chain-context). ### Chain heartbeat The SDK keeps the chain alive with a wall-clock heartbeat every **30 seconds** by default (configurable per policy in `[10s, 120s]`). The interval is time-based, not chunk-based: a slow stream with one chunk per minute still gets a heartbeat; a fast stream does not spam them. If the chain dies (idle TTL expired, max duration exceeded, or `op="end"`), the SDK raises `WorkflowKilledInterrupt` at the next `yield` boundary. ## Kill signal mid-stream An operator hit on **Kill** raises `WorkflowKilledInterrupt` (alias `NullRunWorkflowKilledError`) at the next `yield` boundary. It is a `NullRunError` subclass — caught by `except Exception:` like every other SDK error. If you want kill-specific handling (close the stream, flush state), catch the typed alias explicitly first and re-raise after. ### Cancellation latency The kill signal typically arrives at the SDK within ~100 ms of the operator clicking **Kill** (WebSocket push path). If the WebSocket is unavailable and polling fallback is active, latency rises to the poll interval (default **1 s**) plus the next `/gate` boundary. For long-running streams, keep the WS connection healthy — set `NULLRUN_TRANSPORT=ws` (the default) and avoid restrictive outbound firewalls on the SDK host. ```python from nullrun import WorkflowKilledInterrupt @protect async def stream_kill_safe(prompt: str): stream = await client.chat.completions.create( model="gpt-4o-mini", messages=[{"role": "user", "content": prompt}], stream=True, ) try: async for chunk in stream: yield chunk.choices[0].delta.content or "" except WorkflowKilledInterrupt: await stream.close() raise ``` ## Tracking without auto-instrumentation If the SDK's httpx transport hook can't see your custom streaming client (a vendor SDK that bypasses httpx), call `track_llm` manually after the stream ends. Use `stream_options={"include_usage": True}` so the final chunk carries the usage block; otherwise you have to estimate. See [OpenAI streaming reference](https://platform.openai.com/docs/api-reference/chat-streaming). ```python from nullrun import init_or_die, protect, track_llm @protect def custom_stream(prompt: str): stream = client.chat.completions.create( model="gpt-4o-mini", messages=[{"role": "user", "content": prompt}], stream=True, stream_options={"include_usage": True}, ) final = None for chunk in stream: final = chunk yield chunk.choices[0].delta.content or "" if final and getattr(final, "usage", None): track_llm( input_tokens=final.usage.prompt_tokens, output_tokens=final.usage.completion_tokens, model="gpt-4o-mini", ) ``` Without `track_llm()` the budget counter is never credited and the next `/gate` may reject the next call based on stale spend. ## Common pitfalls | Pitfall | Symptom | Fix | |---|---|---| | Heartbeat every N chunks | Chain dies silently during slow streams | Heartbeat on a wall-clock timer (30s default) | | `await stream.close()` after kill | Half-written chunks can leak to the caller | Wrap the stream in `try/finally`, always close | | Catching only `Exception` around the loop with no kill handler | Kill still propagates but with no cleanup hook | Catch `NullRunWorkflowKilledError` explicitly first to close the stream | | Forgetting `track_llm()` after a manual stream | Dashboard shows zero cost, budget never decremented | Always report final usage, even via estimation | ## See also - [Chain context → soft mode](../concepts/workflow.md#chain-context) - [Errors → kill contract](../reference/errors.md#sdk-exception-hierarchy-python) - [Use with FastAPI](../how-to/fastapi.md) — streaming inside ASGI handlers --- # Source: https://docs.nullrun.io/how-to/custom-tracking/ # File: docs/how-to/custom-tracking.md --- title: Custom Tracking description: Manually report cost and events with track_llm, track_tool, and track_event when auto-instrumentation doesn't fit your runtime. --- # Manual cost / event tracking Most of the time auto-instrumentation handles cost tracking — the httpx transport hook reads `usage` from OpenAI / Anthropic / Gemini / Cohere responses and emits `track_llm` automatically. Use `track_llm`, `track_tool`, and `track_event` manually when: - your LLM client bypasses httpx (Bedrock via boto3, Cohere on a raw socket, an offline batch reading cached completions); - you proxy the LLM call and the auto-instrumentation hook sees your proxy's response (zero usage) instead of the upstream's; - you call a tool that isn't an HTTP call (database query, state transition, side-effect-bearing custom function); - you have a custom business event (milestone, retry attempt, A/B variant) that you want in the decision log. If your SDK wraps the standard OpenAI / Anthropic / Gemini / Cohere clients, do **not** call `track_llm` manually — auto-instrumentation will fire and you'll double-count. ## The three trackers | API | Purpose | Required fields | | --- | --- | --- | | `track_llm(input_tokens, output_tokens, model, ...)` | Manual LLM cost | `input_tokens`, `output_tokens`; `model` recommended | | `track_tool(tool_name, duration_ms, ...)` | Manual tool cost | `tool_name` (must match `ToolBlock` patterns) | | `track_event(event_type, ...)` | Arbitrary observability | `event_type` (becomes a filterable category) | Without `track_llm` the budget counter is never credited for the call — the next `/gate` may reject based on stale spend. ## Example ```python title="track_custom.py" import nullrun from nullrun import track_llm, track_tool, track_event # After your custom LLM call returns: track_llm( input_tokens=response.usage.prompt_tokens, output_tokens=response.usage.completion_tokens, model="custom-llm-v1", latency_ms=response.elapsed_ms, metadata={"vendor": "internal", "trace_id": "abc-123"}, ) # After a tool call (regardless of success/failure): track_tool( tool_name="send_email", duration_ms=240, is_retry=False, metadata={"to": "user@example.com"}, ) # Arbitrary business events: track_event("agent.milestone", step="research_complete", elapsed_secs=42) track_event("agent.error", code="validation_failed", field="email") ``` `track_tool`'s `tool_name` flows through the policy engine — a `ToolBlock` policy with pattern `send_*` catches a manual call to `track_tool("send_email", ...)`. Use the same tool names you would pass to auto-instrumentation so policy enforcement stays consistent. ## When the SDK can't see the call If your tool isn't called from inside `@protect`, wrap the manual tracking in `@protect` so the gate still runs: ```python from nullrun import protect, track_llm @protect def call_custom_llm(prompt): response = my_custom_client.complete(prompt) track_llm( input_tokens=response.usage.input, output_tokens=response.usage.output, model="custom-llm-v1", ) return response.text ``` ## Caveats - **Buffering**: `track_*` events don't go straight to the gateway — they buffer in the runtime's event batch and flush on the next `@protect` call or `flush_interval_ms`. If your process exits before the flush, the events are lost; call `shutdown(flush=True)` in your `finally` block. - **Idempotency**: each `track_*` call gets a fresh UUID. Calling it twice with the same payload produces two events. For retries, gate the call yourself. ## See also - [SDK API → track_llm / track_tool / track_event](../reference/sdk-api.md#track_llm-manual-usage) - [LLM frameworks](../how-to/llm-frameworks.md) — non-httpx vendors (Bedrock, Cohere) that use manual tracking --- # Source: https://docs.nullrun.io/how-to/ci-cd/ # File: docs/how-to/ci-cd.md --- title: Ci Cd description: Fail-CLOSED gate integration in CI, with smoke-test scripts that verify the gate is reachable before a deploy. --- # CI / CD integration Wire NullRun into your build pipeline so policy mistakes are caught before they hit production. The pattern uses [`synthetic_sdk_load.py`](https://github.com/nullrunio/nullrun-examples) — a CLI tool from the official examples repo that drives the real SDK against the real gateway with no LLM cost. ## Pre-prod validation pattern ```mermaid flowchart LR A[CI job] --> B[set NULLRUN_API_KEY to pre-prod key] B --> C[synthetic_sdk_load.py
--interval 0.1 --workers 20
--requests 1000 --yes] C --> D{policy OK?} D -- yes --> E[deploy to prod] D -- no --> F[fail build] ``` `/gate` runs through the **same policy** your production key uses, just with synthetic tokens. The exit code tells your CI whether the policy misbehaved. ## GitHub Actions example ```yaml title=".github/workflows/pre-prod.yml" name: pre-prod-validation on: pull_request: branches: [master] jobs: nullrun-smoke: runs-on: ubuntu-latest timeout-minutes: 5 steps: - uses: actions/checkout@v4 - name: Install load tool run: | git clone https://github.com/nullrunio/nullrun-examples.git pip install -e nullrun-examples - name: Run synthetic load env: NULLRUN_API_KEY: ${{ secrets.NULLRUN_PRE_PROD_KEY }} NULLRUN_API_URL: ${{ secrets.NULLRUN_PRE_PROD_URL }} run: | set -euo pipefail python -m synthetic_sdk_load \ --interval 0.05 \ --workers 10 \ --requests 200 \ --model gpt-4o-mini \ --yes echo "load passed" ``` `synthetic_sdk_load.py` exits `1` on any exception (network error, budget block, etc.) and `0` on a clean run. `set -euo pipefail` ensures the step fails fast on any error. The exit code is your CI gate. ## Common failure modes | Failure | What it means | Fix | |---|---|---| | `NullRunBudgetError` on every request | Your pre-prod budget is too tight for synthetic traffic | Raise `budget_cents` on the pre-prod workflow, or use a higher-cap test key | | `429 NR-R001` | `max_calls_per_minute` is too low for the synthetic load | Raise the rate limit on the pre-prod workflow | | Connection refused / DNS error | `NULLRUN_API_URL` is wrong or the gateway is down in this environment | Verify the URL; add a `/health/live` check before the load step | | HMAC 401 | `NULLRUN_SECRET_KEY` is not set in CI | Add the secret to the repo / org / environment secrets store | ## What this catches - **Budget too tight** — if a developer sets `budget_cents: 100` on a workflow that needs `$50/day` to run, the pre-prod load surfaces it before prod. - **Tool block too broad** — a `ToolBlock` pattern that accidentally matches every tool name surfaces as `NR-T001` on every call. - **Workflow not bound to the right key** — if the pre-prod key was rotated but the workflow binding is stale, the load fails with a clean 401 instead of mysteriously going through. - **Soft-mode misconfiguration** — if the policy says `Soft` but no `max_overdraft_cents` is set, every over-budget call hits a hard block. The smoke load exposes this. ## What this does **not** catch - **Real LLM cost** — synthetic tokens are random, not actual spend. Run `synthetic_sdk_load.py` for policy shape; use a real staging workflow with a small `budget_cents` for cost projections. - **Per-model pricing drift** — verify your models are priced correctly by checking the dashboard **Cost** tab after one real call. - **WS push timing** — synthetic load doesn't exercise kill/pause paths. Trigger them manually via the dashboard during the smoke-test pass. ## Pipeline integration checklist Add these checks to your CI before merging anything that touches the SDK, the gateway, or policy configuration: ```yaml title=".github/workflows/pre-prod.yml" jobs: nullrun-smoke: steps: - name: 1. Health check run: | curl -fs "${NULLRUN_API_URL}/health/live" \ || (echo "gateway down" && exit 1) - name: 2. Synthetic load env: { NULLRUN_API_KEY: ${{ secrets.NULLRUN_PRE_PROD_KEY }} } run: python -m synthetic_sdk_load --interval 0.05 --workers 10 --requests 200 --yes - name: 3. Capabilities probe run: | # Verify the gateway capabilities are present before promoting to prod curl -fs "${NULLRUN_API_URL}/api/v1/capabilities" \ | python -c "import json,sys; c=json.load(sys.stdin); \ assert c['capabilities_ok'], 'gateway capabilities not ready'" - name: 4. Approval pause/resume (manual) # Trigger an approval-required call, click approve in the # dashboard, confirm the SDK resumes without throwing. run: echo "manual step — see docs/how-to/human-approval.md" ``` Step 4 is manual by design — the approval pause/resume flow requires a human to click Approve in the dashboard, which a CI job can't do. Run it on every release candidate as part of the release checklist. ## See also - [`synthetic_sdk_load.py`](https://github.com/nullrunio/nullrun-examples) - [Troubleshooting](../troubleshooting.md) — common failure modes and how to read SDK logs - [Configuration → env vars](../getting-started/configuration.md) - [Reference → HTTP API → Capabilities](../reference/http-api.md#capabilities) --- # Source: https://docs.nullrun.io/reference/sdk-api/ # File: docs/reference/sdk-api.md title: SDK API maturity: stable description: Reference for every NullRun SDK symbol: init, @protect, @sensitive, workflow, chain, exceptions, manual tracking, and transport hooks. # SDK API The Python SDK lives in [`nullrunio/nullrun-sdk-python`](https://github.com/nullrunio/nullrun-sdk-python). Package name on PyPI: **`nullrun`**. ```bash title="shell" pip install nullrun # core only pip install "nullrun[langgraph]" pip install "nullrun[agents]" # openai-agents pip install "nullrun[all]" # every optional extra ``` Auto-instrumentation for httpx-based libraries (`openai`, `anthropic`, `openai-agents`, …) is on by default once `init()` runs — see [Auto-instrumentation](../getting-started/install.md#auto-instrumentation). ## Top-level ```python title="public_surface.py" from nullrun import init, init_or_die, protect, workflow, span, agent, chain, track_llm, track_tool, track_event ``` ### `init()` vs `init_or_die()` — which one to use | Helper | Behaviour | Use when | |---|---|---| | `init(api_key=None, api_url=None, debug=False)` | Raises `NullRunAuthenticationError` if `api_key` is missing or env var unset. Returns the runtime. | Production / apps where you want to handle "no api_key" yourself (e.g. surface a friendly error to your UI) | | `init_or_die(*, api_key=None, api_url=None, debug=False, exit_code=1)` | Catches the `NullRunAuthenticationError` exception, prints the catalog user-message to stderr, calls `sys.exit(exit_code)`. Returns the runtime otherwise. | One-shot scripts, CLI tools, examples, anything where a missing key is a hard error | `init_or_die` is `init` plus an `try/except NullRunAuthenticationError → sys.exit(1)`. The chain `@guarded` decorator does the same for callsite-level errors. || Symbol | Purpose | In `__all__` | |---|---|---|---| | `init(api_key=None, api_url=None, debug=False)` | Initialise the SDK singleton. `api_key` is required (read from `NULLRUN_API_KEY` if not passed). The HMAC secret, batch size, flush interval, and transport mode are **not** parameters here — set them via env vars. Negotiates protocol version with the gateway on first call. | ✅ | | `init_or_die(*, api_key=None, api_url=None, debug=False, exit_code=1)` | Like `init` but exits cleanly with `exit_code` (default 1) if no API key is configured. See the table above. | ✅ | | `@protect` | Wrap a function for **gate** enforcement (budget pre-flight + kill/pause check + sensitive-tool decision). Takes no kwargs. Always pair with `@guarded` for the zero-boilerplate exit-on-block pattern. | ✅ | | `@sensitive` | Parameterless decorator. Marks a function as a sensitive tool — `@protect` will pre-check before the body runs. If the gate is unreachable, the call is rejected. Place `@sensitive` outside `@protect` so registration runs first. | ✅ (lazy import) | | `@guarded` | Decorator that wraps a function so any `NullRunError` raised inside is converted to `format_user_message(exc)` on stderr and `sys.exit(1)`. The kill signal (`WorkflowKilledInterrupt` / `NullRunWorkflowKilledError`) is also a `NullRunError` subclass, so `@guarded` catches it too. Use the un-`@guarded` `protect()` form if you need to handle kill distinctly. | ✅ | | `with nullrun.handle(*, exit_code=1):` | Context manager form of `@guarded` — apply to a region of code rather than a single function. | ✅ | | `workflow(name=None)` | Context manager. Sets the `workflow_id` contextvar that `@protect` and `track_*` attach to events. | (lazy) | | `chain(chain_id: str, op: str = "start")` | Context manager for soft-mode budget gate. `op="start"` registers the chain; `op="continue"` extends TTL; `op="end"` closes it. | (lazy) | | `span(name=None)` | Context manager for nested trace spans. | (lazy) | | `agent(name=None)` | Context manager for agent identity. | (lazy) | | `set_call_context(model=None, tools=None)` | Per-call context the SDK forwards to `/gate` so the backend's budget + tool-block enforcement sees real values. | (lazy) | | `on_error(hook)` | Register a global error hook. Fires for every `NullRunError` subclass — including the kill signal (`WorkflowKilledInterrupt` / `NullRunWorkflowKilledError`) — BEFORE the exception propagates. Multiple hooks supported; fires in registration order; hook exceptions are caught and DEBUG-logged. Filter inside the hook by `error_code` (`"NR-W002"`) if you need to skip kill. Returns an idempotent unregister callable. | ✅ | | `track_llm(input_tokens, output_tokens=0, **kwargs)` | Manual escape hatch for non-HTTP LLM calls. Returns the backend's decision dict. Buffers into the event batch and flushes on the next `@protect` call or `flush_interval_ms`. `**kwargs` are forwarded to the transport layer (e.g. `model`, `latency_ms`, `metadata`). | ✅ | | `track_tool(tool_name, duration_ms=None, **kwargs)` | Manual tool-call tracking. `**kwargs` are forwarded to the transport layer (e.g. `is_retry`, `metadata`). | ✅ | | `track_event(event_type, **kwargs)` | Catch-all for custom events. | ✅ | | `format_user_message(exc, locale="en")` | Render a `NullRunError` as an end-user-facing string from the SDK's default catalog. Use this in place of `str(exc)` when showing exceptions to end users — see [User-facing messages](#user-facing-messages) below. | ✅ | | `set_user_message(code, text)` | Override the user-facing message for a specific `error_code` for the lifetime of this process. Pass `text=""` to clear. | ✅ | | `get_user_message(code)` | Look up the raw user-facing message for an `error_code`. Returns the per-process override if set, otherwise the catalog default, otherwise the generic fallback. | (lazy) | | `shutdown(timeout=2.0, flush=True)` | Gracefully shut down the runtime: send a clean WebSocket close frame, drain in-flight events, stop background threads. Safe to register via `atexit`. | ✅ | | `status()` | Synchronous snapshot of the runtime state as a frozen `NullRunStatus` dataclass (`ok` / `degraded` / `offline` / `misconfigured`). Thread-safe, side-effect-free. Raises `NullRunConfigError` if the runtime hasn't been initialised yet. | ✅ | Rows marked **lazy** are exposed under `nullrun.*` via `__getattr__` on first access; they do not appear in `dir(nullrun)` until used. ### `track_llm` manual usage Use `track_llm` when auto-instrumentation can't see the LLM call — a custom HTTP client that bypasses `httpx`, an offline batch job, a test fixture. The signature mirrors the data the auto-instrumentation extractor reads from OpenAI / Anthropic / Gemini / Cohere response bodies: ```python title="track_llm_manual.py" import nullrun from nullrun import track_llm # After your custom LLM call returns: track_llm( input_tokens=response.usage.prompt_tokens, output_tokens=response.usage.completion_tokens, model="custom-model-v1", latency_ms=response.elapsed_ms, metadata={"vendor": "custom", "trace_id": "..."}, ) ``` Without `track_llm`, the SDK has nothing to report to the gateway — the budget counter is never credited, and the next `/gate` call may reject based on stale spend. Call `track_llm` once per real LLM call. ### `track_tool` manual usage ```python title="track_tool_manual.py" from nullrun import track_tool track_tool( tool_name="send_email", duration_ms=240, is_retry=False, metadata={"to": "user@example.com"}, ) ``` Use it when a non-LLM tool call happens outside the auto-instrumentation hooks (e.g. a custom agent framework, or a tool wrapped in your own function). The `tool_name` flows through to the policy engine — a `ToolBlock` policy with `pattern = "send_*"` will catch a manual call to `track_tool("send_email", ...)`. ### `track_event` catch-all ```python title="track_event_manual.py" from nullrun import track_event track_event("agent.milestone", step="research_complete", elapsed_secs=42) ``` Accepts arbitrary keyword arguments as the event payload. Use for custom observability signals (milestones, errors, business events) that you want in the decision log alongside `track_llm` / `track_tool`. ### Custom user messages See [User-facing messages → Per-deployment branding](#per-deployment-branding) below for `set_user_message` / `get_user_message` usage. The curated public surface in `dir(nullrun)` is the `__all__` list in `nullrun/__init__.py`: `__version__`, `init`, `protect`, `track_llm`, `track_tool`, `track_event`, `shutdown`, `on_error`, `status`, `format_user_message`, `set_user_message`, `handle`, `guarded`, `init_or_die`, plus the structured exception names `NullRunError`, `NullRunAuthError`, `NullRunConfigError`, `NullRunBackendError`, `NullRunBudgetError`, `NullRunToolBlockedError`, and `WorkflowKilledInterrupt` (the kill signal). The legacy names (`WorkflowPausedException`, `WorkflowKilledException`, `NullRunAuthenticationError`, `NullRunBlockedException`) remain available via `from nullrun import X` for backward compatibility. ## Exceptions All raised from `nullrun.breaker.exceptions`. Every public SDK exception inherits from `NullRunError` and carries four structured fields: `error_code` (machine-readable, e.g. `"NR-B004"`), `user_action` (imperative hint), `retryable` (bool), `docs_url`. See [Errors](errors.md#sdk-exception-hierarchy-python) for the full hierarchy diagram. | Class | When | Notes | | --- | --- | --- | | `NullRunError` | Structured base for every user-facing SDK exception | Inherits `BreakerError`. Carries `.error_code`, `.user_action`, `.retryable`, `.docs_url`. | | `NullRunConfigError` | SDK misconfigured (e.g. missing `api_key`) | Code family for config errors. Never retryable. | | `NullRunAuthenticationError` | Missing / invalid `X-API-Key`, bad HMAC | 401 / 403. Carries `.message` for backward compat. | | `NullRunAuthError` | 401 specifically (key rejected) | Subclass of `NullRunAuthenticationError`. Carries `.status_code` (the wire HTTP status). | | `NullRunTransportError` | Gateway unreachable | Carries `.source` (e.g. `NETWORK_ERROR` / `GATEWAY_ERROR` / `BREAKER_OPEN` / `AUTH_ERROR`) and `.endpoint`. Retryable. | | `NullRunBackendError` | 5xx from the gateway | Subclass of `NullRunTransportError`. Code `NR-B002` family. Retryable. | | `RateLimitError` | HTTP 429 | Subclass of `NullRunTransportError`. Carries `.retry_after`, `.upgrade_url`, `.body`. Code `NR-R001`. Retryable. | | `NullRunRateLimitRedisError` | 503 — Redis reservation failed | Subclass of `NullRunInfrastructureError`. Code `NR-R002`. | | `NullRunProtocolError` | Backend returned 400 `PROTOCOL_TOO_OLD` | Carries `.min_required_version`. Upgrade SDK past the min required protocol version. | | `NullRunBlockedException` | Generic policy block | Inspect `.workflow_id`, `.reason`, `.action`, `.tool_name`, `.details`. Carries `.status_code` (the wire HTTP status, e.g. 402 budget, 403 cross-org, 422 `CONSUME_OVERBUDGET`, 429 cap-reached). **No** `.message` — use `str(exc)`. | | `NullRunBudgetError` | Budget exhausted | Subclass of `NullRunBlockedException`. Code `NR-B004`. | | `NullRunToolBlockedError` | Tool in block list | Subclass of `NullRunBlockedException`. Code `NR-T001`. Carries `.tool_name`. | | `NullRunChainError` | Chain-mode gate check failed | Subclass of `NullRunDecision`. Code `NR-CH001`. | | `NullRunConsumeOverbudgetError` | 422 — actual cost > reservation + ε | Subclass of `NullRunDecision`. Surfaces over-budget commit events. | | `NullRunWorkflowInactiveError` | 403 — workflow paused / killed cross-org | Subclass of `NullRunDecision`. Code `NR-W004`. | | `BreakerTransportError` | Transport misconfiguration (events cannot be delivered after retries) | Subclass of `BreakerError` (NOT `NullRunError`). Carries `.events_lost`, `.buffer_size`. | | `InsecureTransportError` | HTTP used where HTTPS required | Subclass of `BreakerTransportError`. | | `WorkflowPausedException` | Paused via control plane | Subclass of `NullRunError`. Carries `.workflow_id`, `.reason`, `.resume_after`. | | `WorkflowKilledException` | Killed via control plane (legacy parent) | `BaseException` subclass. **Deprecated** — emits `DeprecationWarning` on construction. Use `NullRunWorkflowKilledError` directly. | | `WorkflowKilledInterrupt` | Kill arrived mid-call | Subclass of `NullRunError` (was `BaseException` before SDK 0.16.x — now caught by `except Exception:` like every other SDK error). | | `NullRunWorkflowKilledError` | Kill arrived mid-call (typed alias) | Preferred subclass of `WorkflowKilledInterrupt`. Same wire semantics; use this for typed `except` arms. | ## Catch-all pattern ```python title="catch_all_pattern.py" import nullrun from nullrun import WorkflowKilledInterrupt, init, protect from nullrun.breaker.exceptions import ( NullRunBlockedException, RateLimitError, WorkflowPausedException, ) init(api_key="nr_live_...") try: step() except WorkflowKilledInterrupt: raise # always re-raise — kill must reach the top except NullRunBlockedException: ... # budget / tool block / workflow inactive / chain except RateLimitError as exc: time.sleep(exc.retry_after) except WorkflowPausedException: ... # paused — resume via WS / API, then retry ``` The full annotated tutorial (handler ordering rationale, observability hooks, exception hierarchy walkthrough) lives in [Use with FastAPI → HTTP status mapping](../how-to/fastapi.md#http-status-mapping). For global observability (Sentry, OpenTelemetry, structured logs), register a hook with `nullrun.on_error(...)` instead of wrapping every call site. The hook fires for every `NullRunError` subclass BEFORE the exception propagates. Hook exceptions are caught and DEBUG-logged — a misbehaving hook cannot break the SDK. ## User-facing messages `nullrun.format_user_message(exc, locale="en")` renders a `NullRunError` (or any object with an `error_code` attribute) as an end-user-facing string. **Use this instead of `str(exc)` whenever the message might be shown to a person who is not the developer** — `str(exc)` contains internal identifiers like `workflow_id` and `budget_cents` that leak the SDK's internals into product UI. ```python title="format_user_message.py" import nullrun from nullrun import NullRunBudgetError @nullrun.protect def chatbot(message: str) -> str: return agent.run(message) try: reply = chatbot(message) except NullRunBudgetError as exc: # Show the user a clean message instead of the raw exception text # ("Workflow wf-31a blocked: budget_cents=500 exceeded..."). return nullrun.format_user_message(exc) ``` ### Why the SDK owns the wording The catalog of default messages is part of the NullRun product so every deployment sees consistent wording for a given `error_code`. ### Per-deployment branding If a deployment wants its own wording for a single code (e.g. a branded "out of credits" message), call `set_user_message` once at startup: ```python title="set_user_message.py" import nullrun # Override the default message for budget-exceeded. Pass "" to clear. nullrun.set_user_message( "NR-B004", "You've used all your support credits. Upgrade to keep chatting.", ) ``` Overrides live in a per-process dict and are checked before the catalog default. They do not persist across processes and are not synced to the backend — they are pure presentation sugar. ### Locale `format_user_message(exc, locale)` accepts a locale code; in this SDK version **only English (`"en"`) is shipped** and any other value falls back to the English message. The parameter is reserved for future locale packs and matches the structure that user-message overrides will take when they land. ### What if `error_code` is unknown or missing? Objects without `error_code` (plain `Exception`, raw values) get a generic fallback (`"Something went wrong. Please try again."`). The function never raises and never returns an empty string. ## See also - [Decorators & extractors](decorators.md) — deep-dive on `@protect`, `@sensitive`, `@guarded`, `money_outflow`, `tool_params`, `set_call_context`, and the workflow / span / chain context managers - [Errors](errors.md) - [Errors → Decision vs. infrastructure](errors.md#decision-vs-infrastructure) - [Use with FastAPI](../how-to/fastapi.md) - [Auto-instrumentation](../getting-started/install.md#auto-instrumentation) - [Control plane](../concepts/control-plane.md) --- # Source: https://docs.nullrun.io/reference/http-api/ # File: docs/reference/http-api.md --- title: Http Api description: Every NullRun HTTP endpoint: /api/v1/gate, /api/v1/track, /api/v1/capabilities, /api/v1/heartbeat, and the control-plane WebSocket protocol. --- # HTTP API This page lists the endpoints a user — or their SDK — actually calls. Internal admin endpoints (pricing backfill, gateway operator APIs) are not exposed here; see the gateway repo for the full internal surface. Base URL: - Production: `https://api.nullrun.io/api/v1` - WebSocket control plane: `wss://api.nullrun.io/ws/control/{org_id}` The `{org_id}` for the WebSocket comes from the credential bundle returned by `POST /api/v1/auth/verify` (`organization_id` field). The SDK negotiates this automatically — only set the URL by hand when you are building a custom WebSocket client. The OpenAPI spec is generated from the router and is the source of truth. ## Authentication Two schemes — they are **not interchangeable**. ### `X-API-Key` + HMAC (SDK → gateway) All SDK-traffic endpoints (`/track`, `/track/batch`, `/gate`, `/execute`, `/check`) require: | Header | Value | | --- | --- | | `X-API-Key` | API key (`nr_live_...`) | | `X-Signature` | hex(HMAC-SHA256(`secret_key`, `timestamp:api_key:body_sha256`)) | | `X-Signature-Timestamp` | Unix epoch seconds (rejected if older than `NULLRUN_HMAC_MAX_AGE_SECS`, default 300) | | `X-Workflow-Id` *(optional)* | binds the call to a workflow context | The SDK computes and signs every request automatically once `NULLRUN_API_KEY` and `NULLRUN_SECRET_KEY` are set. The gateway default for `NULLRUN_HMAC_REQUIRED` is `false` for backward compatibility — operators must set it explicitly to `true` in production. When `NULLRUN_HMAC_REQUIRED=true`, unsigned SDK requests are rejected with 401 and the SDK-auth middleware emits a per-request WARN so the gap is visible in logs. > SDK requests to `/api/v1/orgs/{org_id}/*` are also checked for > org-mismatch: the org claimed in the URL must match the org the > API key was minted for. Mismatch → 403. ### `Authorization: Bearer ` (dashboard / admin) Dashboard endpoints (`/orgs/...`, `/admin/...`) use session tokens obtained from `POST /api/v1/auth/login` or the OAuth flow (`/auth/oauth/register`). Pass as `Authorization: Bearer `. ## SDK endpoints These are the endpoints the SDK calls. You will not normally hit them by hand, but the contracts are stable and the OpenAPI spec documents every field. | Method | Path | Called by | | --- | --- | --- | | `POST` | `/api/v1/auth/verify` | `init()` — verifies the API key and returns the full credential bundle (HMAC `secret_key`, org context, plan, `workflow_id`, `key_version`, etc.) | | `GET` | `/api/v1/capabilities` | `init()` — protocol negotiation | | `POST` | `/api/v1/gate` | Pre-flight + policy evaluation + budget reservation (called from `@protect` entry) | | `POST` | `/api/v1/track` | Single-event commit (`reservation_id` + `idempotency_key`) | | `POST` | `/api/v1/track/batch` | Batched events (≤ 100 per batch) — opt-in fallback when single-event is disabled | | `GET` | `/api/v1/orgs/{org_id}/policies` | Policy fetch (called from SDK on first `@protect` and on `policy_invalidated` WS push) | | `GET` | `/api/v1/orgs/{org_id}/workflows/{workflow_id}` | Workflow lookup (called from SDK on first gate per workflow) | | `GET` | `/api/v1/orgs/{org_id}/status` | Control-plane poll fallback (only used when WS is down) | | `POST` | `/api/v1/heartbeat` | Time-based cadence heartbeat | | `POST` | `/api/v1/cancel` | Cancel an in-flight execution. Idempotent; the reservation TTL cleans up even if the call doesn't reach the gateway. | | Method | Path | Status | | --- | --- | --- | | `POST` | `/api/v1/check` | Deprecated — returns `410 Gone` with `replacement: /api/v1/gate`. Use `/gate` for new integrations. | | `POST` | `/api/v1/execute` | Live — per-tool invocation with gate pre-flight and budget reservation (thin adapter in front of the unified gate engine). | ## Auth | Method | Path | Purpose | | --- | --- | --- | | `POST` | `/api/v1/auth/register` | Create account, returns API key + secret | | `POST` | `/api/v1/auth/login` | Dashboard session token | | `POST` | `/api/v1/auth/verify` | Verify API key + return full credential bundle | | `POST` | `/api/v1/auth/oauth/register` | OAuth signup (returns API key + secret) | #### `POST /api/v1/auth/verify` response fields | Field | Type | Notes | | --- | --- | --- | | `organization_id` | string (UUID) | Canonical org identifier | | `organization_name` | string \| null | Canonical org display name | | `plan` | string | Plan tier (`lite`, `starter`, `growth`, `scale`, `enterprise_unlimited`, …) | | `features` | object | Plan-feature flags resolved for this org | | `limits` | object | Plan-limits block (workflows, seats, …) | | `role` | string \| null | Member role; `null` on API-key path — SDK treats `null` as "role unknown, escalate to session auth" | | `workflow_id` | string \| null | Workflow this API key is bound to; `null` on unbound / legacy keys | | `secret_key` | string \| null | HMAC secret the SDK uses to sign requests — distinct from the API key itself | | `key_version` | number \| null | Current active HMAC key version; SDK compares against its cached value to detect rotation | ## Workflows | Method | Path | Purpose | | --- | --- | --- | | `POST` | `/api/v1/orgs/{org_id}/workflows` | Create workflow | | `GET` | `/api/v1/orgs/{org_id}/workflows` | List workflows | | `GET` | `/api/v1/orgs/{org_id}/workflows/{workflow_id}` | Get workflow | | `PATCH` | `/api/v1/orgs/{org_id}/workflows/{workflow_id}` | Update (budget, name, …) | | `POST` | `/api/v1/orgs/{org_id}/workflows/{workflow_id}/pause` | Pause (broadcasts `state_change` over WS) | | `POST` | `/api/v1/orgs/{org_id}/workflows/{workflow_id}/resume` | Resume | | `POST` | `/api/v1/orgs/{org_id}/workflows/{workflow_id}/kill` | Kill (broadcasts `state_change` over WS) | ## Policies | Method | Path | Purpose | | --- | --- | --- | | `POST` | `/api/v1/orgs/{org_id}/policies` | Create | | `GET` | `/api/v1/orgs/{org_id}/policies` | List (a single policy is not addressable by id — read it from the list response) | | `PATCH` | `/api/v1/orgs/{org_id}/policies/{policy_id}` | Update | | `DELETE` | `/api/v1/orgs/{org_id}/policies/{policy_id}` | Delete | | `POST` | `/api/v1/orgs/{org_id}/policies/{policy_id}/toggle` | Toggle a policy active/inactive (dashboard PATCH 404 fix) | | `GET` | `/api/v1/orgs/{org_id}/policies/templates` | List policy templates | | `POST` | `/api/v1/orgs/{org_id}/policies/templates/{template_id}/enable` | Enable a template | | `DELETE` | `/api/v1/orgs/{org_id}/policies/templates/{template_id}` | Disable a template | Most-restrictive-wins composition across applicable policies — see [Concepts → Policies](../concepts/policies.md). ## Approvals Programmatic approval / denial — useful for on-call bots and CI runbooks. Dashboard uses the same endpoints internally. See [Concepts → Human approval](../concepts/human-approval.md) for the end-to-end flow. | Method | Path | Purpose | | --- | --- | --- | | `POST` | `/api/v1/orgs/{org_id}/approvals/{approval_id}/approve` | Approve a pending approval. Idempotent — already-approved returns `409 approval_already_decided`. | | `POST` | `/api/v1/orgs/{org_id}/approvals/{approval_id}/deny` | Deny a pending approval. The SDK raises `WorkflowKilledInterrupt` for the parked agent. | ## Executions, audit, observability | Method | Path | Purpose | | --- | --- | --- | | `GET` | `/api/v1/orgs/{org_id}/executions` | List executions | | `GET` | `/api/v1/orgs/{org_id}/executions/{execution_id}` | One execution | | `GET` | `/api/v1/orgs/{org_id}/audit-log` | Audit trail | | `GET` | `/api/v1/orgs/{org_id}/audit-log/export` | Start an async audit-log export job (returns a job id) | | `POST` | `/api/v1/orgs/{org_id}/audit-log/export` | Same — POST variant, useful from web forms | | `GET` | `/api/v1/orgs/{org_id}/audit-log/export/{job_id}/status` | Poll export job status (`pending` / `ready` / `failed`) | | `GET` | `/api/v1/orgs/{org_id}/audit-log/export/{job_id}/download` | Download the exported file once status is `ready` | | `GET` | `/api/v1/orgs/{org_id}/traces` | List trace spans | | `GET` | `/api/v1/orgs/{org_id}/traces/{trace_id}` | One trace (full span tree) | | `GET` | `/api/v1/orgs/{org_id}/incidents` | Active and recent incidents (rate-limit outages, budget overruns, etc.) | | `GET` | `/api/v1/orgs/{org_id}/dashboard` | Dashboard payload | | `GET` | `/api/v1/orgs/{org_id}/control-center` | Single-call control-center view (workflows + recent decisions + alerts) | | `GET` | `/api/v1/orgs/{org_id}/usage` | Per-key usage breakdown (canonical) | | `GET` | `/api/v1/orgs/{org_id}/quota` | Per-key usage breakdown (legacy alias of `/usage` — same payload) | | `GET` | `/api/v1/budget/approximate` | Approximate budget view for UI display — see [Budgets → Approximate budget endpoint](../concepts/budgets.md#approximate-budget-endpoint) | | `GET` | `/api/v1/orgs/{org_id}/status` | Single-call dashboard status (budget + rate + plan limits + time-to-exhaustion) | ### Cancellations `POST /api/v1/cancel` cancels an in-flight execution. Cancellation is idempotent — calling it twice on the same execution is a no-op and releases the reservation by TTL even if the call never reaches the gateway. See [Control plane](../concepts/control-plane.md) for the related kill / pause endpoints. ## Org management | Method | Path | Purpose | | --- | --- | --- | | `GET/PATCH/DELETE` | `/api/v1/orgs/{org_id}` | Org settings, update, delete | | `GET` | `/api/v1/orgs/{org_id}/api-keys` | List keys | | `POST` | `/api/v1/orgs/{org_id}/api-keys` | Mint key | | `DELETE` | `/api/v1/orgs/{org_id}/api-keys/{key_id}` | Revoke key | | `POST` | `/api/v1/orgs/{org_id}/api-keys/{key_id}/rotate` | Rotate the HMAC secret in place | | `GET` | `/api/v1/workflows/{workflow_id}/api-keys` | Per-workflow key listing | | `GET` | `/api/v1/orgs/{org_id}/members` | Members | | `PATCH` | `/api/v1/orgs/{org_id}/members/{member_id}` | Update member role | | `DELETE` | `/api/v1/orgs/{org_id}/members/{member_id}` | Remove member | | `POST` | `/api/v1/orgs/{org_id}/invites` | Invite | | `DELETE` | `/api/v1/orgs/{org_id}/invites/{invite_id}` | Revoke invite | | `POST` | `/api/v1/orgs/{org_id}/invites/{invite_id}/resend` | Resend invite email | | `GET` | `/api/v1/invites/{token}` | Public invite info | | `POST` | `/api/v1/invites/{token}/accept` | Public invite accept | | `POST` | `/api/v1/invites/{token}/decline` | Public invite decline | ## Alerts | Method | Path | Purpose | | --- | --- | --- | | `GET` | `/api/v1/orgs/{org_id}/alerts` | Active alerts | | `POST` | `/api/v1/orgs/{org_id}/alerts/{alert_id}/dismiss` | Dismiss | | `GET/POST` | `/api/v1/orgs/{org_id}/alert-channels` | Channels | | `GET/PATCH` | `/api/v1/orgs/{org_id}/notification-settings` | Per-user settings | ## Health Health endpoints are registered on the gateway's top-level router — they are **not** under `/api/v1`. They return `200 OK` when healthy, `503` otherwise, with a JSON body listing each dependency's status. | Method | Path | Purpose | | --- | --- | --- | | `GET` | `/health` | Alias for `/health/live` | | `GET` | `/healthz` | Alias for `/health/live` (Kubernetes convention) | | `GET` | `/health/live` | Liveness — process is up and accepting connections | | `GET` | `/health/ready` | Readiness — Postgres + Redis reachable | | `GET` | `/health/startup` | Startup — `200` after migrations complete, `503` while booting | ## Capabilities The capabilities endpoint reports the wire-contract version the gateway supports. The SDK calls this on `init()` to negotiate the protocol version and to surface a startup warning if the SDK is older than what the gateway requires. | Method | Path | Purpose | | --- | --- | --- | | `GET` | `/api/v1/capabilities` | Report `min_protocol_version` / `max_protocol_version`, `sdk_min_version`, server version + build timestamp, and the `capabilities.*` feature flags | When `init()` detects that the SDK is older than the gateway's required minimum version, it emits a warning so the operator sees the gap before the first `/gate` call fails with `400 PROTOCOL_TOO_OLD`. The current wire-protocol version is **4** (min supported: **2**). `init()` negotiates the version automatically via `/capabilities`; you do not need to set anything by hand. ## Heartbeat Long-running workflows post a time-based cadence heartbeat so the gateway can detect an orphaned workflow whose agent process has crashed without sending a kill / pause signal. The recommended cadence is advertised in `capabilities.heartbeat_interval_seconds` (default 30s). | Method | Path | Purpose | | --- | --- | --- | | `POST` | `/api/v1/heartbeat` | Time-based cadence heartbeat; body is `{ chain_id }` only. `workflow_id` is derived from the API key, not passed in the body. | The SDK posts heartbeats automatically inside the `NullRunRuntime` background thread once `init()` has run; operators do not need to call it manually. ## WebSocket control plane | Path | Purpose | | --- | --- | | `WS /ws/control/{org_id}` | Real-time kill/pause/policy-invalidated/key-rotated events (HMAC-signed on connect) | Server → client message types: `initial_state`, `state_change`, `policy_invalidated`, `key_rotated`, `resync_required`, `error`, `pong`, `approval_resolved`, `subscribed`. Client → server message types: `ack`. See [Control plane](../concepts/control-plane.md) for the full protocol and the SDK reaction matrix. ## Common request patterns The examples below use the dashboard's `Authorization: Bearer ` header (the token comes from `POST /api/v1/auth/login`). For SDK-traffic endpoints substitute `X-API-Key` + HMAC headers — see the [Authentication](#authentication) section above. ```bash title="shell" TOKEN=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9... ORG_ID=8a3b1c7d-... ``` !!! note "Compute the HMAC signature" For SDK endpoints the `X-Signature` and `X-Signature-Timestamp` headers are required. The signature is `HMAC-SHA256(secret_key, "::")`. See [Authentication → X-API-Key + HMAC](#x-api-key-hmac-sdk-gateway). ### Create a workflow ```bash title="shell" curl -X POST "https://api.nullrun.io/api/v1/orgs/$ORG_ID/workflows" \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{ "name": "production-bot", "description": "Customer-facing assistant", "budget_cents": 5000, "human_approvals_enabled": false }' # → 201 { "id": "wf_abc...", "name": "production-bot", ... } ``` ### Mint an API key bound to a workflow ```bash title="shell" curl -X POST "https://api.nullrun.io/api/v1/orgs/$ORG_ID/workflows/wf_abc.../api-keys" \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{ "name": "prod-bot-key", "scopes": ["gate", "track", "verify"] }' # → 201 { # "id": "key_...", # "key": "nr_live_xxxxxxxxxxxx", ← shown ONCE, store it # "secret_key": "sk_...", ← shown ONCE, store it # "workflow_id": "wf_abc...", # "key_prefix": "nr_live_xxxxxx" # } ``` The raw `key` and `secret_key` are **never returned again** — losing them means rotating the key. See [API keys → How to create a key](../concepts/api-keys.md#how-to-create-a-key). ### Kill / pause / resume a running workflow ```bash title="shell" # Kill — broadcasts state_change(killed) over WS to every connected SDK curl -X POST "https://api.nullrun.io/api/v1/orgs/$ORG_ID/workflows/wf_abc.../kill" \ -H "Authorization: Bearer $TOKEN" # Pause curl -X POST "https://api.nullrun.io/api/v1/orgs/$ORG_ID/workflows/wf_abc.../pause" \ -H "Authorization: Bearer $TOKEN" # Resume curl -X POST "https://api.nullrun.io/api/v1/orgs/$ORG_ID/workflows/wf_abc.../resume" \ -H "Authorization: Bearer $TOKEN" ``` ### Single-call status (current spend / budget / time-to-exhaustion) ```bash title="shell" curl "https://api.nullrun.io/api/v1/orgs/$ORG_ID/status" \ -H "Authorization: Bearer $TOKEN" # → 200 { # "current_spend_cents": 2340, # "budget_cents": 5000, # "time_to_exhaustion_secs": 86400, # "rate_used": 12, # "rate_limit_per_min": 60, # "plan_caps": { ... } # } ``` Useful for dashboards and alerts — one call returns everything you need to show "you've used X of Y". ## See also - [Errors](errors.md) - [Control plane](../concepts/control-plane.md) --- # Source: https://docs.nullrun.io/reference/errors/ # File: docs/reference/errors.md --- title: Errors description: Full NullRun error-code reference: NR-B004 budget blocks, NR-T001 transport errors, NR-R001 refusals, and decision vs infrastructure classes. --- # Error codes The canonical `ApiErrorCode` enum is the source of truth for every non-2xx response the gateway returns. This page maps each code to: - the SDK exception the Python SDK raises when it sees that error - the HTTP status code the gateway returns - when it happens There are **two parallel taxonomies** you may see: - **Gateway error slugs** — short SCREAMING_SNAKE_CASE strings in the `error` field of every non-2xx response. Listed below in *Gateway error codes*. - **NR-* codes** — the SDK's user-facing `error_code` field on every exception. The full catalog is below in *NR-* error code catalog*. For the **three-layer error model** (structured exceptions → `on_error` hook → `format_user_message` / `@guarded`) and the boundary between developer-facing and end-user-facing wording, see [Concepts → Error handling](../concepts/error-handling.md). ## Gateway error codes (`error` field on every non-2xx response) The canonical catalog lives in the gateway. The `error` slug is the stable, machine-readable identifier; `message` is human-safe; `code` is a legacy SCREAMING_SNAKE_CASE alias kept for backward compatibility. | `error` slug | HTTP | When | SDK exception | | --- | --- | --- | --- | | `bad_request` | 400 | Generic 400 — invalid input that isn't a validation failure | `NullRunConfigError` (or `NullRunError`) | | `unauthorized` | 401 | Missing or invalid `X-API-Key` / expired session / HMAC mismatch | `NullRunAuthenticationError` (`NullRunAuthError` for 401 specifically) | | `forbidden` | 403 | Authenticated but not allowed (incl. CSRF mismatch, org-mismatch on `/orgs/*`) | `NullRunAuthenticationError` | | `not_found` | 404 | Resource doesn't exist or isn't visible | (no exception — caller handles) | | `conflict` | 409 | Idempotency conflict, duplicate, "already a member", "invite already pending", "cannot demote last owner", etc. | `NullRunError` | | `validation_error` | **422** | Request body / params failed schema validation | `NullRunConfigError` | | `plan_limit_exceeded` | **422** | Generic plan cap hit (workflows, seats, api_keys). Body `details.resource` carries which dimension. | `NullRunBlockedException` | | `workflow_limit_reached` | **422** | Workflow-specific active-workflow cap hit | `NullRunBlockedException` | | `rate_limit_exceeded` | 429 | Per-minute / per-day rate cap. Body carries `retry_after` (seconds). | `RateLimitError` (carries `.retry_after`, `.upgrade_url`) | | `internal_error` | 500 | Server-side bug | `NullRunBackendError` (retryable) | | `not_implemented` | 501 | Feature not yet implemented | `NullRunError` | | (also `internal_error`) | 503 | `ApiError::ServiceUnavailable` — transient downstream failure on an enforcement path. Carries `retry_after`. | `NullRunBackendError` (retryable) | > **Plan limit slugs (api_keys / seats / policies / executions)** all > surface as `plan_limit_exceeded` with `details.resource` set to the > dimension name (`"api_keys"`, `"seats"`, `"workflows"`, …). There > is no separate slug per dimension — read `details.resource`. ## SDK exception hierarchy (Python) Every public SDK exception inherits from `NullRunError` and carries four structured fields: `error_code` (machine-readable, e.g. `"NR-B004"`), `user_action` (imperative hint), `retryable` (bool), `docs_url`. ``` NullRunError (Exception) ├── NullRunDecision (marker — expected policy outcomes) │ ├── NullRunBlockedException (policy / budget / loop / sensitive block) │ │ ├── NullRunBudgetError (budget exhausted — NR-B004) │ │ └── NullRunToolBlockedError (tool in block list — NR-T001) │ ├── WorkflowPausedException (paused via control plane) │ └── NullRunWorkflowKilledError (kill via control plane — NR-W002; │ preferred typed alias of │ `WorkflowKilledInterrupt`) └── NullRunInfrastructureError (marker — system failures) ├── NullRunConfigError (misconfiguration, e.g. missing api_key) ├── NullRunAuthenticationError (401 / 403) │ └── NullRunAuthError (401 specifically) └── NullRunTransportError (transport failures) ├── NullRunBackendError (5xx — retryable) └── RateLimitError (429 — carries .retry_after, .upgrade_url) ``` `NullRunDecision` and `NullRunInfrastructureError` are **marker classes**, not exception classes themselves. They exist so host code can `except NullRunDecision` to catch every expected policy outcome (budget, tool block, pause) and `except NullRunInfrastructureError` to catch every system failure (transport, backend 5xx, auth rejection, config error) — see [Decision vs. infrastructure](#decision-vs-infrastructure) below for the recommended handling pattern. `NullRunBlockedException` carries `.workflow_id`, `.reason`, `.action` (`"block"` / `"kill"` / `"pause"`), `.tool_name` (when the block is tool-scoped), and `.details` (free-form). There is **no** `.message` attribute — use `str(exc)`. `NullRunWorkflowKilledError` (and its parent `WorkflowKilledInterrupt`) inherit from `NullRunError`, so a bare `except Exception:` catches the kill signal. For kill-specific handling — checkpointing state, notifying a supervisor, etc. — catch the typed exception explicitly. ## The default path: zero lines of error handling For the common "run an agent and print a friendly message on failure" case, the three public helpers do the work — no `try/except NullRunError` required. | Helper | Catches | For | |---|---|---| | `init_or_die(api_key=...)` | `NullRunError` raised by `init()` (typically a config / auth family code) | Startup; one-shot script entry point | | `@guarded` | Any `NullRunError` raised inside the wrapped function | Standard agent loop | | `with nullrun.handle():` | Any `NullRunError` raised inside the block | Region of code (e.g. a graph `invoke`) | All three propagate non-`NullRunError` exceptions (anything that isn't an SDK error) as honest tracebacks. `NullRunError` subclasses — including the kill signal `NullRunWorkflowKilledError` — are caught and converted to catalog wording. For the full design rationale and the boundary between "what NullRun tells the developer" and "what the developer tells their end users", see [Concepts → Error handling](../concepts/error-handling.md). ## Decision vs. infrastructure The public exception hierarchy splits `NullRunError` into two marker subclasses by **what kind of event** the exception represents. The split is additive — every existing `except NullRunError:` and `except NullRunBlockedException:` clause keeps matching. New code can use the marker classes to write a two-branch handler that captures the right behaviour for each category. | Marker | What it covers | Why it matters | | --- | --- | --- | | `NullRunDecision` | Expected policy outcomes — budget cap, tool block, loop detection, workflow pause, per-workflow rate limit | The enforcement layer is doing its job. UX explains the decision and (where applicable) offers an upgrade or alternative action. | | `NullRunInfrastructureError` | System failures — network unreachable, gateway 5xx, auth rejection, config error | The SDK could not reach or query the policy engine. UX is a generic "service unavailable"; operators triage via `error_code`, `retryable`, and for transport errors, `source` / `endpoint`. | ### Recommended handler shape ```python title="decision_vs_infra_handler.py" import nullrun from nullrun import ( NullRunDecision, NullRunInfrastructureError, ) try: result = agent.run(message) except NullRunDecision as d: # Expected — surface to the user, log to product analytics, # tag the conversation with d.error_code for cohort analysis. return d.user_message() if hasattr(d, "user_message") else str(d) except NullRunInfrastructureError as e: # System failure — alert ops, retry with backoff, do NOT # surface internal text to the end user. The catalog has a # generic message for every infrastructure error code. sentry.capture_exception(e) return nullrun.format_user_message(e) ``` ### Mapping decision subclasses to HTTP When you build a server-framework integration (FastAPI, aiohttp, Telegram bot, Slack handler), map each category to the right HTTP status. The headline cases are below; every `NullRunDecision` subclass carries `.status_code` so framework integrations can map the field directly instead of hard-coding. | Category | HTTP status | Notes | | --- | --- | --- | | `NullRunDecision` — budget exhausted (`NR-B004`) | `402` | Honour `.retry_after` from the `RateLimitError` if set; budget-exhausted `NullRunBudgetError` exposes the same field via `.details.retry_after` | | `NullRunDecision` — tool blocked (`NR-T001`) | `403` | User did nothing wrong, but the action is forbidden | | `NullRunDecision` — workflow paused | `503` | Set `Retry-After` from `.resume_after` | | `NullRunInfrastructureError` — rate-limit Redis (`NR-R002`) | `503` | `NullRunRateLimitRedisError` — the rate limiter is degraded | | `WorkflowKilledInterrupt` | `503` | Special ASGI middleware required — see [Use with FastAPI](../how-to/fastapi.md) | Other decision categories (`CONSUME_OVERBUDGET` → 422, `CHAIN_ORG_MISMATCH` → 403, `CHAIN_MAX_DURATION_EXCEEDED` → 402, `WORKFLOW_INACTIVE` → 403, `PROTOCOL_TOO_OLD` → 400, generic `NullRunInfrastructureError` → 503) follow the same pattern: read `exc.status_code` from the wire and map it directly. Every `NullRunDecision` subclass carries `.status_code` (the wire HTTP status the backend returned). The FastAPI integration maps this field to the response status automatically; in custom integrations read `exc.status_code` rather than hard-coding the default above. The NullRun SDK ships a reference FastAPI integration that applies this mapping for you — see [Use with FastAPI](../how-to/fastapi.md) for a one-line setup. ## HTTP status summary | Status | Meaning | SDK action | | --- | --- | --- | | 200 | OK | — | | 400 | Bad request | Inspect `message`, fix request | | 401 | Bad API key / HMAC | Refresh key / check `NULLRUN_SECRET_KEY` | | 403 | Forbidden | Check role / scope | | 404 | Not found | Caller handles (workflow/policy may have been deleted) | | 409 | Conflict | Inspect `message` (already-member, invite-already-pending, etc.) | | 422 | Validation / plan limit | Inspect `details` (for plan limits, `details.resource` + `details.current` + `details.limit`) | | 429 | Rate limit | Honour `Retry-After`; check `upgrade_url` | | 5xx | Gateway error | Retry with backoff; sensitive tools fail-closed | When the gateway is unreachable, the SDK raises `NullRunTransportError` with `source` set to one of `NETWORK_ERROR`, `GATEWAY_ERROR`, `AUTH_ERROR`. ## NR-* error code catalog Stable, machine-readable identifiers on every SDK exception. The catalog splits into two families: - **decision / enforcement codes** — what the gate decided (block, deny, require approval, …). Most are surfaced as `NullRunDecision` subclasses in Python. - **infrastructure codes** — transport / backend / config failures. Surfaced as `NullRunInfrastructureError` subclasses. The three-layer error model and the boundary between developer-facing and end-user-facing wording lives in [Concepts → Error handling](../concepts/error-handling.md). ### Decision / enforcement codes | `error_code` | When | HTTP | SDK class | | --- | --- | --- | --- | | `NR-B004` | Workflow budget exhausted | 402 | `NullRunBudgetError` | | `NR-B006` | Post-approval budget re-check failed on the same envelope as the original `/gate` | 503 | `NullRunBudgetRecheckFailedError` | | `NR-O001` | Actual cost > reservation + ε | 422 | `NullRunConsumeOverbudgetError` | | `NR-R001` | Per-workflow rate limit | 429 | `RateLimitError` | | `NR-T001` | Tool in block list | 403 | `NullRunToolBlockedError` | | `NR-CH001` | Chain context invalid (CHAIN_MAX_DURATION_EXCEEDED) | 402 | `NullRunChainError` | | `NR-W001` | Workflow does not exist or is not visible to this API key | 404 | `NullRunError` | | `NR-W002` | Operator kill signal (via dashboard **Kill** button or WS push) | n/a (raised) | `NullRunWorkflowKilledError` (alias `WorkflowKilledInterrupt`) | | `NR-W004` | Workflow soft-deleted, killed, or paused | 403/503 | `WorkflowPausedException` / kill signal | | `NR-A003` | API key rejected | 401 | `NullRunAuthError` | | `NR-A004` | Approval response missing — gate returned `require_approval` but no row found on `/execute` | 403 | `NullRunApprovalResponseMissingError` | | `NR-A010` | Approval exists, status `PENDING` — operator has not decided yet | 403 | `NullRunApprovalNotYetApprovedError` | | `NR-A011` | Operator explicitly denied the approval | 403 | `NullRunApprovalDeniedError` | | `NR-A012` | Approval expired (`expires_at` in the past) | 403 | `NullRunApprovalExpiredError` | | `NR-A013` | Business-impact digest drifted since approval — re-approval required | 403 | `NullRunApprovalDigestMismatchError` | | `NR-A014` | Capability digest drifted since approval — re-approval required | 403 | `NullRunApprovalToolDigestMismatchError` | | `NR-A015` | Grant already consumed (replay rejected) | 403 | `NullRunApprovalReplayRejectedError` | | `NR-A016` | Approval database unavailable — transient 5xx on the approval row lookup | 503 | `NullRunApprovalDbUnavailableError` (fail-CLOSED — retry with backoff) | | `NR-X001` | Generic policy block — no dedicated subclass | varies | `NullRunBlockedException` (default) | Approval grant-consume codes (NR-A010..NR-A015) are most often seen inside a running agent flow: the SDK has parked for human review and the operator has acted (or the grant aged out). Match on the specific subclass first; fall back to `NullRunBlockedException` if you don't care about the exact cause. ### Infrastructure codes | `error_code` | When | HTTP | SDK class | | --- | --- | --- | --- | | `NR-B002` | Gateway 5xx | 500/503 | `NullRunBackendError` | | `NR-B003` | Budget Redis unavailable (fail-CLOSED on the budget path) | 402 | `NullRunBudgetError` | | `NR-B005` | Budget data unavailable (approximate-budget lookup, all sources down) | 503 | `NullRunBackendError` | | `NR-R002` | Rate-limit Redis unavailable | 503 | `NullRunRateLimitRedisError` | | `NR-C001` | Missing or invalid `NULLRUN_API_KEY` at `init()` | n/a (raised) | `NullRunAuthenticationError` | | `NR-P001` | Wire-protocol version mismatch | 400 | `NullRunProtocolError` | ## See also - [Concepts → Error handling](../concepts/error-handling.md) — the three-layer model, minimal-boilerplate helpers, dev/end-user boundary - [SDK API](sdk-api.md) - [SDK API → User-facing messages](sdk-api.md#user-facing-messages) - [Use with FastAPI](../how-to/fastapi.md) - [HTTP API](http-api.md) - [Circuit breaker](../concepts/circuit-breaker.md) - [Sensitive tools](../concepts/sensitive-tools.md) --- # Source: https://docs.nullrun.io/reference/llm-tool-catalog/ # File: docs/reference/llm-tool-catalog.md --- title: Llm Tool Catalog description: Per-model input and output pricing for every LLM NullRun understands, with capability flags for streaming, tools, and structured output. --- # Tool catalog A reference list of the tool names LLM agents commonly expose, tagged with a default risk rating you can use as a starting point when you register `@sensitive` patterns. The catalog covers three sources that NullRun sees in production: - **LangChain built-in toolkits** (`langchain-community` — search, SQL, Gmail, Slack, GitHub, file system, vector stores, code interpreters) - **Anthropic / OpenAI hosted tools** (OpenAI code interpreter, e2b sandbox, Riza JS exec) - **MCP servers** (official + community — filesystem, git, github, postgres, sqlite, redis, puppeteer, memory, time) Names are normalised to `snake_case` — that's the convention LangChain's docs recommend. Risk rating: | Rating | Meaning | | --- | --- | | `low` | Read-only or reversible. Safe to call without a policy decision. | | `medium` | Mutates external state but the change is reversible (issue created, draft email, S3 put). | | `high` | **Side effects you can't easily undo** — files written or deleted, money moved, messages sent, code executed, infra changed. Mark `@sensitive` and route through a human approval gate. | ## Search & retrieval | Tool | Risk | | --- | --- | | `tavily_search`, `tavily_search_results_json` | low | | `duckduckgo_search`, `duckduck_results_json` | low | | `google_search`, `google_search_results_json` | low | | `bing_search` | low | | `brave_search` | low | | `web_search`, `search_web` | low | | `fetch`, `fetch_url` | low | | `requests_get` | low | | `wikipedia`, `arxiv`, `pubmed_search`, `semantic_scholar` | low | | `news_search`, `search_news` | low | | `requests_post`, `requests_put`, `requests_patch` | medium | | `requests_delete` | high | ## File system | Tool | Risk | | --- | --- | | `read_file`, `file_read` | low | | `list_directory`, `get_file_info`, `search_files`, `file_search` | low | | `copy_file`, `create_directory` | medium | | `write_file`, `file_write`, `create_file`, `edit_file` | high | | `delete_file`, `file_delete`, `move_file` | high | ## Code execution Any tool that evaluates arbitrary code is `high` by definition — the blast radius is the entire host or sandbox the agent reaches. `python_repl`, `python_repl_ast`, `execute_python`, `code_interpreter` (OpenAI built-in), `e2b_code_interpreter`, `execute_javascript` (Riza), `execute_code`, `run_command`, `run_bash_command`, `terminal`, `bash`, `shell`, `repl` — all **`high`**. Register a blanket pattern (see the starter list below) rather than enumerating them. ## Databases | Tool | Risk | | --- | --- | | `sql_db_schema`, `sql_db_query_checker`, `read_query` (sqlite) | low | | `list_tables`, `describe_table` | low | | `sql_db_query`, `query_sql_database`, `db_query` | medium | | `redis_set` | medium | | `execute_sql`, `run_sql`, `db_write`, `write_query`, `create_table` | high | | `db_delete`, `redis_delete` | high | ## Git & GitHub | Tool | Risk | | --- | --- | | `git_status`, `git_diff`, `git_log` | low | | `github_get_issue`, `github_get_pull_request`, `github_list_repos`, `github_get_file`, `github_search_code` | low | | `github_create_issue`, `github_update_issue`, `github_close_issue`, `github_create_pull_request`, `github_create_repo`, `github_create_branch`, `git_add`, `git_checkout` | medium | | `git_commit`, `github_merge_pull_request`, `github_push_files`, `github_delete_repo` | high | ## Email & messaging | Tool | Risk | | --- | --- | | `gmail_get_message`, `gmail_search`, `office365_search_emails`, `slack_get_channel`, `slack_get_messages` | low | | `gmail_create_draft`, `office365_create_draft` | medium | | `send_email`, `send_gmail`, `gmail_send_message`, `gmail_delete_message`, `office365_send_email`, `slack_send_message`, `slack_schedule_message`, `send_sms` | high | ## Calendar & tasks | Tool | Risk | | --- | --- | | `office365_search_events`, `get_calendar_events` | low | | `office365_create_event`, `create_calendar_event`, `create_task`, `complete_task` | medium | | `delete_calendar_event`, `delete_task` | high | ## Cloud & infrastructure | Tool | Risk | | --- | --- | | `s3_get_object`, `s3_list_objects`, `ec2_describe_instances`, `kubernetes_get` | low | | `s3_put_object`, `docker_run` | medium | | `s3_delete_object`, `s3_delete`, `ec2_start_instance`, `ec2_stop_instance`, `ec2_terminate_instance`, `lambda_invoke`, `kubernetes_apply`, `kubernetes_delete`, `docker_stop` | high | ## Finance & payments `stripe_charge`, `stripe_create_customer`, `stripe_refund`, `stripe_create_payment`, `create_invoice`, `send_payment` — all **`high`**. Reads only (`get_balance_sheet`, `get_income_statement`, `get_cash_flow`) drop to `low`. ## Memory & vector stores | Tool | Risk | | --- | --- | | `vector_store_search`, `memory_retrieve`, `search_nodes` | low | | `vector_store_add`, `memory_store`, `create_entities`, `add_observations` | medium | | `vector_store_delete`, `memory_delete`, `delete_entities` | high | ## Browser & scraping | Tool | Risk | | --- | --- | | `puppeteer_screenshot`, `browser_screenshot`, `scrape_page`, `extract_content` | low | | `puppeteer_navigate`, `puppeteer_click`, `puppeteer_fill`, `browser_navigate`, `browser_click` | medium | | `puppeteer_evaluate` | high | !!! note `puppeteer_evaluate` runs JS in the browser page context — treat it as `high` (cross-origin requests, DOM injection, credential theft). `browser_navigate` and `browser_click` are `medium` because they take actions on whatever URL the agent picks. ## Recommended ToolBlock starter list The SDK does **not** ship a built-in sensitive tool list (see [Sensitive tools](../concepts/sensitive-tools.md) for the rationale). You express "this tool needs review" with a `ToolBlock` policy on the server, evaluated by the gate on every `/gate` call. The recommended starter patterns below map to that policy mechanism. Create a ToolBlock policy in the dashboard under **Policies → New policy** (pick **Tool block** as the type). The dashboard renders the canonical tool name for every framework integration so you can match against the right string: ```json title="tool_block_policy.json" { "policies": [ { "name": "Sensitive starter (matches the catalog)", "type": "ToolBlock", "scope": "Org", "config": { "tool_pattern": [ "stripe.*", "charge", "send_payment", "create_invoice", "refund", "send_email", "send_gmail", "send_message", "send_sms", "slack_send.*", "office365_send.*", "delete_file", "file_delete", "write_file", "file_write", "execute_sql", "run_sql", "db_write", "db_delete", "write_query", "create_table", "s3_delete.*", "ec2_terminate.*", "ec2_stop.*", "lambda_invoke", "kubernetes_delete", "kubernetes_apply", "python_repl.*", "bash", "shell", "terminal", "execute_.*", "run_command", "run_bash_command", "git_commit", "github_merge.*", "github_push.*", "github_delete.*", "memory_delete", "vector_store_delete", "delete_entities" ] } } ] } ``` Patterns are glob-matched against the tool name the agent requested, case-insensitively — `"Stripe.Charge"` will match `"stripe.*"`. For finer-grained rules (e.g. "block refunds over $500", "require approval for sends to non-`@internal` addresses"), use the typed `BusinessImpact` predicate (`money_amount` or `tool_parameters`) — see [Human approval → typed predicates](../concepts/human-approval.md#typed-predicates). ## See also - [Sensitive tools](../concepts/sensitive-tools.md) — the policy- driven way to express "this tool needs review" (no built-in SDK list, all server-side via ToolBlock) - [Tool policies](../concepts/tool-policies.md) — glob patterns, per-tool block / allow rules - [Human approval](../concepts/human-approval.md) — typed `BusinessImpact` predicates for narrower rules --- # Source: https://docs.nullrun.io/compliance/index/ # File: docs/compliance/index.md --- title: Index description: NullRun's compliance posture: geo-block at the network edge, sanctions screening at signup, and what to expect when rules degrade. --- # Compliance NullRun enforces geo and sanctions restrictions at the edge gateway. Two cooperating layers control jurisdiction-based access: | Layer | Purpose | Reference | | --- | --- | --- | | Geo restrictions | Classify every inbound request by source country and apply allow / hard-block / waitlist actions. | [Geographic restrictions](geo-restrictions.md) | | Sanctions screening | Match signup name and email against the OFAC SDN list (with EU / UK / UN lists supported as additional CSVs). | [Sanctions screening](sanctions-screening.md) | Sanctions violations are strict-liability; see legal review for full rationale. A regression on either layer is a compliance incident. --- # Source: https://docs.nullrun.io/compliance/geo-restrictions/ # File: docs/compliance/geo-restrictions.md title: Geographic restrictions maturity: stable description: IP-level blocklists for sanctioned jurisdictions, with the runtime status codes a client sees when a request is geo-blocked. # Geographic restrictions NullRun's edge gateway classifies every inbound request by source country and applies one of three actions: - **Allow** — request proceeds normally. - **Hard block** — request is rejected with **403** (`service_unavailable_in_jurisdiction`) or **503** (`geoip_unavailable`). - **Waitlist redirect** — a compliance-blocked visitor on the marketing site is 302-redirected to `/waitlist` so the lead is captured without exposing the API surface. The classification happens before authentication and before per-account quota checks, so blocked traffic never touches the database. ## Why this is needed Sanctions violations are strict-liability; see legal review for full rationale. A Terms-of-Service clause alone is not enough — a regulator will infer targeting from the fact that the API endpoint is reachable from a sanctioned IP space. Hard-blocking at the edge is the only reliable signal. The same logic applies to the other comprehensive-sanctions regimes (OFAC, EU, UK, UN) for the sanctioned-country blocklist. A single accepted signup or payment from one of those jurisdictions is a criminal-law violation, not a civil one. ## Blocklist The blocklist has two tiers. ### Tier 1 — Sanctioned (strict-liability block) | Code | Country | Rationale | | --- | --- | --- | | `RU` | Russia | OFAC + EU + UK comprehensive | | `IR` | Iran | OFAC comprehensive | | `KP` | DPRK | OFAC + UN comprehensive | | `SY` | Syria | OFAC + EU comprehensive | | `CU` | Cuba | OFAC comprehensive | | `BY` | Belarus | Post-2022 UK + EU sectoral | | `VE` | Venezuela | Partial — signups blocked; existing read-only API access preserved (write operations blocked) | | `MM` | Myanmar | OFAC + EU restrictive measures | | `AF` | Afghanistan | Post-2021 sanctions regime | | `ZW` | Zimbabwe | OFAC selective sanctions | Sanctioned requests are blocked with **403** even on the marketing site — no waitlist, no email capture. Strict liability does not allow the "we will email you when we do" bridge. ### Tier 2 — High-risk / no-service (compliance block) | Code | Region | Rationale | | --- | --- | --- | | `AT BE BG HR CY CZ DK EE FI FR DE GR HU IE IT LV LT LU MT NL PL PT RO SK SI ES SE` | EU-27 | GDPR + active enforcement | | `IS NO LI` | EEA / EFTA | Treated like EU for our purposes | | `CH` | Switzerland | FADP — high compliance burden | | `GB` | United Kingdom | UK GDPR + ICO + class actions | | `US CA` | United States / Canada | CCPA + state patchwork | | `CN` | China | PIPL + data localisation | | `IN` | India | DPDPA 2023 + criminal penalties for officers | For high-risk countries: - **`/api/*` and `/ws/*`** → 403 `service_unavailable_in_jurisdiction` - **Marketing site** (anything NOT under `/api/` or `/ws/`) → 302 to `/waitlist?cc=`. ## Decision matrix ```mermaid flowchart TD R["Request arrives
at the edge"] --> E{"Extract
client IP"} E -->|None| L["Log WARN, allow
(should not happen in prod)"] E -->|Loopback /
private / CGNAT| L2["Allow
(bypass IP)"] E -->|Public IP| B{"GeoIP DB
available?"} B -->|No| H["503 geoip_unavailable
(fail-CLOSED)"] B -->|Yes| L3["Look up country"] L3 --> S{"Sanctioned
country?"} S -->|Yes| H2["403 service_unavailable_in_jurisdiction
(strict-liability block)"] S -->|No| H3{"High-risk
country?"} H3 -->|No| A["Allow"] H3 -->|Yes| P{"On marketing
site?"} P -->|Yes| W["302 → /waitlist?cc=…"] P -->|No| H4["403 service_unavailable_in_jurisdiction"] ``` ## Fail-CLOSED posture The geo-block is **fail-CLOSED**: if the GeoIP database is missing, unreadable, or returns an error, **all** ingress is rejected with **503**. The rationale: > If the GeoIP database is missing or unreadable, ALL ingress is > rejected (503) so the operator notices the misconfiguration. ## Operator overrides Geo-block posture is operator-controlled at the platform level; users cannot override it. ## What is bypassed The geo-block **never** blocks: - **Localhost and private IPs** — `127.0.0.0/8`, `10/8`, `172.16/12`, `192.168/16`, `169.254/16`, `100.64.0.0/10` (CGNAT), IPv6 `fc00::/7` (ULA), `fe80::/10` (link-local). These are pod-to-pod traffic, monitoring agents, or the operator's local-dev loopback; none of them can themselves trigger GDPR. - **Health and metrics** — `/health`, `/healthz`, `/ready`, `/readyz`. These are infrastructure-internal probes and must never be geo-blocked. - **The waitlist endpoint** — `POST /api/v1/waitlist`. The marketing site redirects compliance-blocked visitors here; if the geo-block then 403'd the form POST, the lead-capture flow would be broken. The waitlist has its own rate limit of 5 submissions per hour per IP. ## Audit headers Every blocked response carries two headers for observability and debugging: | Header | Meaning | | --- | --- | | `x-nullrun-fortress-block: sanctions` | Blocked by the Tier-1 sanctions list. | | `x-nullrun-fortress-block: waitlist` | Marketing-site redirect to `/waitlist`. | | `x-nullrun-fortress-country: ` | The resolved ISO 3166-1 alpha-2 country code. Absent when the GeoIP database is unavailable. | These headers are **not** logged at INFO level (the country code is PII under GDPR) — they appear at WARN. ## Runbook — keeping the GeoIP database live The NullRun team maintains the GeoIP database; contact support if geo-block seems misclassified. --- # Source: https://docs.nullrun.io/compliance/sanctions-screening/ # File: docs/compliance/sanctions-screening.md title: Sanctions screening maturity: stable description: OFAC SDN screening on signup, the degraded-fallback semantics when the screening service is unavailable, and the audit trail. # Sanctions screening The geo-block stops ingress from sanctioned countries at the edge. **Sanctions screening** is the second layer: a name/email/handle check on every signup that catches the case where a designated individual travels, uses a VPN, or signs up through a non-sanctioned-country proxy. It runs on both the standard signup form and the OAuth registration flow. The screening runs against the OFAC SDN list (with EU / UK / UN list support). ## Why both layers OFAC's comprehensive-sanctions regimes are **strict-liability**. A single accepted signup or payment from a designated person is a criminal-law violation. The geo-block is the always-on defence; sanctions screening is the secondary layer: - A designated individual travelling abroad and signing up from a hotel Wi-Fi in a non-sanctioned country. - A designated individual using a commercial VPN that exits in Armenia or Singapore. - A designated individual signing up via OAuth (Google / GitHub) from a non-sanctioned IP, where the only signal we have is the email handle or display name. The full list of sanctioned jurisdictions is in [Geographic restrictions → Blocklist](geo-restrictions.md#blocklist). ## List source The screening matches against the OFAC SDN list. Source: The EU consolidated list and the UK HMT consolidated list are also supported. !!! tip "Refresh cadence" OFAC SDN: refresh **daily**. MaxMind GeoLite2: weekly. EU / UK consolidated: as published (typically monthly). Restart the gateway after each CSV update to pick up the new file. ## Screening logic For each signup the screening runs: 1. **Normalise** the name and email (catches full-width homoglyphs like `ＡＢＣ` → `ABC`) and lower-case them. 2. **Tokenise** on whitespace and non-alphanumeric characters. 3. **Drop short noise** — tokens shorter than 3 characters are skipped (so `Mr.`, `de`, `la`, `Jr.` do not contribute). 4. **Match** — if **any** token of the name or the email appears in the SDN token set, the signup is rejected. The matching is intentionally aggressive. False positives are cheap (rejected signup, the user retries with a different email); false negatives carry regulatory exposure. ## Screen outcomes The screening returns one of three results: | Result | Meaning | What happens | | --- | --- | --- | | Clean | No SDN token matched. | Allow signup. | | Match | The name or email contained a known SDN token. | Reject with 403. The matched display name and the field that hit are logged at WARN for audit. | | Degraded | Screening ran but the table is the hand-curated fallback (CSV missing or unparseable). | Allow signup. A separate WARN log + an ops-counter flag the misconfiguration. The geo-block is still on — the IP-level defence is intact. | On a `Match` the response body is a generic 403 — the matched display name is **not** echoed to the client to avoid confirming the screening target. ## Known limitations - **Cyrillic / Latin homoglyphs are NOT collapsed.** A Cyrillic `а` stays Cyrillic after normalisation; only the full-width Latin / ASCII cases collapse. A designated individual could circumvent name-based screening by transliterating their name to a homoglyph script. A non-Latin name from a sanctioned-country IP is still blocked by the geo-block. - **No email-domain match.** Emails are tokenised on `@` and `.`, but the resulting tokens (e.g. `gmail`, `mail`) are common enough that matching them would produce false positives. The name tokens are the primary signal; the email is a secondary, weaker signal. --- # Source: https://docs.nullrun.io/troubleshooting/ # File: docs/troubleshooting.md --- title: Troubleshooting description: Common NullRun questions answered: why is my agent blocked, how to debug a gate decision, what to do when a budget doesn't reset. --- # Troubleshooting What to expect when NullRun is doing its job — and how to recover when it isn't. > **Format:** symptom → diagnosis → fix. If your question isn't here, > see the [Errors reference](reference/errors.md) or open a ticket > from the dashboard's **Help → Send feedback** form (include the > `workflow_id` and the failing row's `decision_id`). ## What can go wrong (and how NullRun reacts) | Situation | Default behaviour | Exception raised | | --- | --- | --- | | Workflow exceeds budget (Hard mode) | Halt at next `/gate` call | `NullRunBudgetError` (`error_code = "NR-B004"`) | | Soft mode over-budget | Allow bounded overrun if chain active, otherwise block | `NullRunBudgetError` (`error_code = "NR-B004"`) | | Agent calls a sensitive tool | Block the call before the function body runs (per ToolBlock policy) | `NullRunToolBlockedError` (`error_code = "NR-T001"`) | | Gateway unreachable, budget gate | **Fail-CLOSED** — 402 | `NullRunBackendError` | | Gateway unreachable, per-key rate limit | **Fail-OPEN** (secondary signal; budget gate is the backstop) | `NullRunBackendError` (warn-logged) | | Gateway unreachable, aggregate rate limit | **Fail-CLOSED** — 503 | `NullRunRateLimitRedisError` | | Workflow killed via dashboard | Raise at next `/gate` call (or at WS push receipt) | `WorkflowKilledInterrupt` (alias `NullRunWorkflowKilledError`) | | Workflow paused via dashboard | Raise at next `/gate` call | `WorkflowPausedException` | | Missing `api_key` on `init()` | Raise at first SDK call | `NullRunAuthenticationError` | | HMAC signature missing / stale | Reject the request (401) | `NullRunAuthenticationError` | | Plan monthly / per-dimension cap reached | Reject the request (422 `plan_limit_exceeded`; `details.resource` names the dimension) | `NullRunBlockedException` (HTTP 422 via `exc.status_code`) | | Consume over-budget on commit | Reject the `/track` commit (422; actual cost > reserved + ε) | `NullRunConsumeOverbudgetError` | | Per-minute rate cap reached | Reject the request (429 with `Retry-After`) | `RateLimitError` | | Chain expired (`max_chain_duration_seconds` exceeded) | 402 | `NullRunChainError` | | Protocol version too old | 400 | `NullRunProtocolError` | > Critical paths refuse to run when the gateway is unreachable; > secondary signals may let calls through. ## What happens when the NullRun service is unavailable NullRun's service is the gateway that evaluates every `/check` and `/track` call. When it's down, behaviour is intentionally asymmetric: **enforcement paths fail-CLOSED** (the safest choice — never let a tool run that should have been blocked), while **secondary signals** (per-key rate limits, cost-event outbox writes, dashboard reads) may fail-OPEN or be queued, because blocking on them would lose data without protecting the budget. This table describes every surface that can be affected. If a row isn't here, the surface behaves as documented in its own concept page. | Surface | What you observe during an outage | How to handle it | | --- | --- | --- | | **Active `/check` — budget gate** | Fail-CLOSED. SDK raises `NullRunBackendError`; the next `@protect`-wrapped call is refused. No implicit re-reserve. The reservation TTL eventually releases the cents. | Catch the exception; retry with exponential backoff. Long outages will exceed your tool timeout. The budget counter is never decremented by a call the gate never approved. | | **Active `/check` — sensitive-tool gate** | Fail-CLOSED. The sensitive tool body never runs. SDK raises `NullRunBackendError`. | Treat as **indeterminate** — don't retry the side-effect blindly. Surface the error to the user and let a human decide. This is the canonical reason `@sensitive` is the default for irreversible actions. | | **Active `/check` — per-key rate limit** | Fail-OPEN (secondary signal). SDK warns and the call proceeds. The budget gate remains the backstop. | No action required. The budget gate still applies on the next call. | | **Active `/check` — aggregate (per-org) rate limit** | Fail-CLOSED — 503 `NullRunRateLimitRedisError`. | Back off and retry with jitter. This is a true outage of the aggregator, not a transient blip. | | **Active `/track` — cost commit** | Returns 200 with the cost event queued in the SDK's local outbox. The inference already happened; blocking would lose the cost record. | None required. The SDK persists the event locally and the outbox drains when the gateway returns. **No cost record is lost during the outage window.** | | **Control plane (WebSocket)** | Connection drops. SDK reconnects with exponential backoff. The local snapshot of workflow status (active / paused / killed) survives. | No operator action — reconnect is automatic. Long outages mean no live kill/pause signals reach the SDK; the next `/check` call picks them up server-side. | | **In-flight approval request** | Held server-side; not surfaced to operators until the gateway returns. The SDK continues to wait for an approval decision (subject to your approval timeout). | If your approval timeout is short, expect `NullRunApprovalTimeoutError`. The pending request is preserved server-side and reappears in the approvals inbox once the gateway recovers — operators can still answer it. | | **Dashboard UI** | Pages return 503; read paths may serve cached fragments where possible. The top banner shows "NullRun is currently unavailable." | Refresh once `GET /health/ready` returns 200. Read-only views (audit log, dashboards) resume first; writes (kill, approve, edit) resume once the gateway is fully ready. | | **HTTP API (programmatic)** | 502 / 503 / 504 on read and write paths. Writes are rejected — the server has no record of success, so there is no implicit retry. | Idempotent reads (`GET`) can be retried freely. Writes (`POST /kill`, `POST /approve`) should not be retried blindly — gate them behind your own idempotency keys if your client retries. | | **Cost-event outbox (reconciliation)** | Events queue in Redis; the drain loop resumes when the gateway returns. | None — reconciliation is automatic. The outbox catches up on the next gateway tick. Provisional reservations eventually reconcile to final `cost_events` rows. | | **Alerts & notifications** | Rule evaluation pauses (the gateway can't see new events to score). Outbound delivery depends on channel: Slack messages buffer at Slack's edge; email and webhook channels drop. | Check the channel after recovery. Slack messages sent during the outage arrive late but are not lost; webhook deliveries need a replay tool. See [Notifications](concepts/notifications.md). | | **Configured workflows, policies, MCP servers, API keys** | Read-only. Nothing can be created, edited, killed, or revoked until the gateway returns. **Already-active rules continue to enforce** on the next gate call — the gate caches the merged Effective Policy. | Plan configuration changes outside the outage window. Operators can still read existing state from cached dashboard fragments. | | **`GET /health/live`** | Always 200 if the binary is running — even when downstream deps are down. | Use this for liveness probes. **Do not use it as a "is NullRun usable" signal** — it will lie during a Redis or Postgres outage. | | **`GET /health/ready`** | 200 when DB + Redis + policy cache are reachable; 503 otherwise. | Use this for readiness probes and to page on. This is the signal that flips first as the service recovers. | | **`GET /api/v1/capabilities`** | 200 with the cached protocol version when the gateway can read from cache; 503 if Redis is unreachable. | Treat 5xx as "stay on the version you already have" — don't auto-upgrade during an outage. The SDK already pins the version it probed at startup. | ### Operator playbook during an outage 1. **Confirm the scope.** Check `GET /health/ready` (or the status page). If `/health/ready` is 503 but `/health/live` is 200, the gateway process is up but Redis or Postgres is unreachable — every enforcement path will fail-CLOSED. 2. **Watch the recovery cascade.** `/health/ready` flips first, then `/api/v1/capabilities`, then the WebSocket reconnects, then the cost-event outbox finishes draining, then dashboard writes unlock. Each layer takes a few seconds; the whole cascade usually finishes inside a minute. 3. **Audit the outage window afterwards.** Open the workflow's detail page and filter the audit log to the outage window. Every decision (including the fail-CLOSED ones) is retained — nothing is lost, and the row counts reconcile against the cost-events outbox. 4. **Don't disable the gate to "fix" the outage.** Setting `NULLRUN_SKIP_BUDGET_CHECK=1` or `NULLRUN_SENSITIVE_FAIL_OPEN=1` bypasses the gate entirely and is unsafe in production. Let the gate fail-CLOSED; catch and retry in your code. ## Common runtime questions ### "Why is my call being rejected with `NullRunBlockedException`?" The most common causes, in order of frequency: 1. **Budget exhausted** — your `policy.budget_cents` ran out, or the per-org plan cap (`max_executions_per_month`, `history_days`, etc.) was hit. Either raise the cap in the dashboard or wait for the next billing cycle. 2. **Tool blocked by ToolBlock policy** — the function name matches a glob pattern in an active ToolBlock policy. Inspect the merged Effective Policy on the workflow's detail page to see which patterns are in scope. 3. **Workflow inactive** — the workflow was soft-deleted, paused, or killed. The gate returns 403 `WORKFLOW_INACTIVE` (SDK surfaces as `error_code = "NR-W004"`). Restore the workflow from the dashboard or create a new one. 4. **Consume over-budget on `/track`** — actual cost exceeded the reservation + ε. The gate returns 422 `CONSUME_OVERBUDGET` and refuses to commit. Report the actual cost accurately from the LLM response. ### "Why is my workflow paused / killed without me doing anything?" Two usual suspects: - **Operator action** — open the workflow's detail page; the audit log shows the actor and timestamp. - **Plan or workflow limit** — `max_workflows_per_plan` was hit (Lite 5, Starter 25, Growth 150, Scale 500), causing auto-pause. Check the plan picker for your tier's cap. ### "Why is the SDK raising `NullRunAuthenticationError`?" - `NULLRUN_API_KEY` is unset or the key was revoked. - `NULLRUN_SECRET_KEY` is unset. Set both `NULLRUN_API_KEY` and `NULLRUN_SECRET_KEY`. - The host clock skew between your SDK process and the gateway is too large for the HMAC signature window. Sync the host clock. - The protocol header `X-NULLRUN-PROTOCOL` is missing or below the gateway's min_required_version. The SDK auto-probes capabilities on first call; upgrade past the min version. ### "Why are some calls tracked and others aren't?" `@protect` fires on the functions it's wrapped around. Plain LLM calls (no `@protect`, no auto-instrumented framework) are **invisible** to NullRun. If you use a framework that the SDK auto-instruments (see [How-to → LLM frameworks](how-to/llm-frameworks.md)), you do not need `@protect` to get cost tracking. ## Health endpoints | Endpoint | Purpose | | --- | --- | | `GET /health/live` | Process liveness (always 200 if the binary is running) | | `GET /health/ready` | Dependency readiness (DB + Redis; 503 if down) | | `GET /api/v1/capabilities` | Gateway protocol version + feature flags | ## See also - [Errors → exception hierarchy](reference/errors.md) - [Concepts → Circuit breaker](concepts/circuit-breaker.md) - [Concepts → Control plane (WebSocket)](concepts/control-plane.md) - [Concepts → Budgets](concepts/budgets.md) - [Reference → HTTP API](reference/http-api.md)