# Waxell Documentation — Full Content
# Generated: 2026-08-10
# Source: https://waxell.ai/docs/

> This file contains the complete documentation for Waxell, concatenated
> for LLM ingestion. For an index with links, see llms.txt.

## Table of Contents

- Waxell Observe
  - Waxell Observe
  - Quickstart: Observe Your Agents
  - Installation & Configuration
  - Auto-Instrumentation
  - Decorator Pattern
  - Behavior Tracking
  - Advanced: Context Manager
  - Streaming
  - Multi-Agent
  - OpenAI
  - Anthropic
  - LiteLLM
  - Claude Code & Cowork
  - LLM Call Tracking
  - Provider Routing — Use Waxell Routing Without the Runtime
  - Sessions
  - User Tracking
  - Cost Management
  - Scoring
  - Prompt Management
  - Conversation Tracking
  - Evaluators (LLM-as-Judge)
  - Datasets & Experiments
  - Policy & Governance
  - Eval-Driven Governance
  - Approval Workflows
  - Human-in-the-Loop
  - Policy Categories & Templates
  - Rate Limit Policy
  - Budget Policy
  - Chargeback Attribution Policy
  - Scheduling Policy
  - Time-of-Day Gating Policy
  - Safety Policy
  - Kill Switch (Circuit Breaker) Policy
  - Audit Policy
  - Operations Policy
  - LLM Policy
  - Quality Policy
  - Content Policy
  - Spawn Limit Policy
  - Data Access Policy
  - Network Policy
  - Scope Policy
  - Code Execution Policy
  - Input Validation Policy
  - Output Egress Format Policy
  - Grounding Policy
  - Provenance Required Policy
  - Retrieval Policy
  - Reasoning Policy
  - Recursion Bound Policy
  - Prompt Injection Guard Policy
  - Approval Policy
  - Delegation Policy
  - Cross-Agent Isolation Policy
  - Communication Policy
  - Domain Governance Policy
  - Signal Governance Policy
  - Tool Allowlist Policy
  - MCP Server Allowlist Policy
  - Prompt Allowlist Policy
  - Tool Argument Schema Policy
  - Agent Service Account Scope Policy
  - Privacy Policy
  - Identity Policy
  - Memory Policy
  - Compliance Policy
  - Context Management Policy
  - Data Residency Policy
  - Data Erasure Policy
  - Breach Notification Policy
  - Bias Trend Policy
  - Model Card Required Policy
  - End-User Budget Policy
  - End-User Rate Limit Policy
  - End-User Suspension Policy
  - Policy Recommendations
  - Platform Assistant
  - FAQ
  - Common Errors
  - Common Mistakes
  - REST API Reference
  - Python SDK Reference
- MCP Governance
  - MCP Governance: Secure and Monitor AI Agent Tool Calls
  - MCP Governance Quickstart
  - MCP Policy Configuration
  - MCP Approval Workflows
  - PII and Secret Scanning for MCP Tools
  - Rug Pull Detection for MCP Tools
  - MCP Span Attributes Reference
  - Add Governance to Your FastMCP Server
  - Configure Governance Per Tool
  - Connect Your Middleware to the Waxell Controlplane
  - Middleware API Reference
  - Governance Proxy Quickstart
  - Proxy Deployment Guide
  - Governing Third-Party MCP Providers
  - MCP Governance Architecture
  - MCP Governance API Reference
- MCP Gateway
  - Waxell MCP Gateway: One Governed MCP Surface for Your Whole Company
  - Gateway Quickstart: Connect Claude in Two Minutes
  - Connector Catalog: 160+ Upstreams, Most with Zero Setup
  - Upstream Authentication: From Fully Automatic to Bring-Your-Own
  - Policy Engine: Deny, Redact, Approve, Rate-Limit — Before the Call Happens
  - Tool Management: Hundreds of Tools Without Drowning the Model
  - Security: Fingerprinting, Drift Defense, DLP, and the Audit Log
  - Self-Hosting the Gateway In Your VPC
  - Gateway Troubleshooting: Errors and What They Mean
- Waxell Framework
  - Introduction
  - Installation
  - SDK Overview
  - @agent Decorator
  - @workflow Decorator
  - @tool Decorator
  - LLM Calls
  - Build Your First Agent
  - Multi-Step Workflows
  - Adding Governance
  - Deploying to Production
- Waxell Runtime
  - Quickstart
  - Runtime Overview
  - How the Runtime Works
  - waxell.yaml Reference
  - Ship a Claude Agent to Your Team
  - Build & Push a Custom Tool
  - Register a Domain
  - Execution Context
  - Execution Tiers
  - Working Memory
  - Durable Execution
  - Backends
- Managed Runtimes
  - Runtimes — Deploy Waxell where your agents run
  - Amazon Bedrock AgentCore Runtime (BYO code)
  - IBM watsonx.ai
  - Azure AI Foundry (hosted agents)
  - Vertex AI Agent Engine
- Agent Connections
  - For Coding Agents
  - Setup
  - Connect via OAuth
  - Capabilities
  - Self-Hosting
  - Managing Connections
  - OAuth Flow
- Waxell Connect
  - Connect — Third-Party Agent Visibility
  - Connect Workspaces
  - Connect Agent Governance
  - Connect Slack Integration
- Waxell Endpoints
  - Waxell Endpoints — Govern the AI on Every Machine
  - How Waxell Endpoints Works
  - Key Concepts & Glossary
  - Deploy to a Fleet (MDM) — Overview
  - Deploy to a Mac Fleet (MDM)
  - Deploy to a Windows Fleet (Intune)
  - Install on One Machine (No MDM)
  - The Guard Cascade
  - Enabling Capture (Creating a Guard)
  - Privacy & On-Device DLP
  - What Gets Installed
  - Troubleshooting & Diagnostics
- Platform
  - Insights — Executive Intelligence
  - Billing & Subscription Management
  - Partner & Reseller Program
- CLI
  - CLI Reference
- Guides
  - Architecture
  - Enterprise Guide
  - Best Practices
  - CLI Reference
  - Enterprise API
- Migrating to Waxell
  - LangChain vs Waxell
  - CrewAI vs Waxell
  - Feature Comparison Matrix
  - Progressive Migration
  - Phase 1: Add Observability
  - Phase 2: Add Signals
  - Phase 3: Agent Builder
  - Phase 4: Go Fully Native

================================================================================

================================================================================
SECTION: WAXELL OBSERVE
LLM observability, cost tracking, and governance for AI applications
================================================================================

--------------------------------------------------------------------------------
# Waxell Observe
URL: https://waxell.ai/docs/observe/overview
Description: Lightweight observability and governance for any Python AI agent framework. Track LLM calls, manage costs, and enforce policies without rewriting your agents.
--------------------------------------------------------------------------------

# Waxell Observe

You already have agents -- add observability in 2 lines of code.

**Waxell Observe** is a lightweight Python package that brings LLM call tracking, cost management, and policy enforcement to any AI agent. It works with any Python agent framework -- LangChain, LlamaIndex, CrewAI, custom code, or anything else. No vendor lock-in, no runtime changes, no migration required.

## Fastest Path: Auto-Instrumentation

Two lines to automatically trace all LLM calls across 200+ providers:

```python

waxell.init(api_key="wax_sk_...", api_url="https://acme.waxell.dev")

# Import LLM SDKs AFTER init() -- they're now auto-instrumented
from openai import OpenAI

client = OpenAI()
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}]
)
# Automatically traced with model, tokens, cost, latency
```

## The Decorator Pattern (Recommended)

Decorators are the primary way to instrument your agents. Wrap functions with `@observe` and behavior decorators to get structured, rich traces with minimal code:

```python


waxell.init()

from openai import AsyncOpenAI

client = AsyncOpenAI()


@waxell.retrieval(source="pinecone")
async def search_docs(query: str) -> list[dict]:
    return await vector_store.search(query, top_k=10)


@waxell.decision(name="approach", options=["summarize", "compare", "deep_dive"])
async def choose_approach(query: str) -> dict:
    return {"chosen": "deep_dive", "reasoning": "Query asks for detailed analysis"}


@waxell.tool(tool_type="api")
async def run_analysis(docs: list) -> dict:
    return await analysis_service.analyze(docs)


@waxell.observe(agent_name="research-pipeline")
async def run_pipeline(query: str):
    docs = await search_docs(query)
    approach = await choose_approach(query)
    analysis = await run_analysis(docs)

    # Inline enrichment
    waxell.score("quality", 0.92)
    waxell.tag("domain", "research")

    return {"result": analysis, "approach": approach["chosen"]}
```

Every decorated function inside `@observe` is automatically recorded as a structured span. No manual `ctx.record_*()` calls needed.

### Decorator Reference

| Decorator | Purpose | What it captures |
|-----------|---------|-----------------|
| `@waxell.observe()` | Agent run boundary | Inputs, outputs, policy checks, run lifecycle |
| `@waxell.tool()` | Tool/function calls | Name, inputs, output, duration, status |
| `@waxell.retrieval()` | RAG search operations | Query, documents, scores, source |
| `@waxell.decision()` | Routing/classification | Chosen option, reasoning, confidence |
| `@waxell.reasoning_dec()` | Chain-of-thought | Thought, evidence, conclusion |
| `@waxell.step_dec()` | Pipeline steps | Step name and output |
| `@waxell.retry_dec()` | Retry/fallback logic | Attempt count, strategy, errors |

### Convenience Functions

Use these anywhere inside an `@observe` scope for inline enrichment:

| Function | Purpose |
|----------|---------|
| `waxell.score(name, value)` | Quality scores (numeric, boolean, categorical) |
| `waxell.tag(key, value)` | Searchable key-value tags |
| `waxell.metadata(key, value)` | Arbitrary structured metadata |
| `waxell.step(name, output=)` | Quick step recording |
| `waxell.decide(name, chosen=)` | Inline decision recording |
| `waxell.retrieve(query=, documents=)` | Inline retrieval recording |
| `waxell.reason(step=, thought=)` | Inline reasoning recording |
| `waxell.retry(attempt=, reason=)` | Inline retry recording |
| `waxell.user_message(content)` | Record inbound user message |
| `waxell.agent_response(content)` | Record outbound agent response |
| `waxell.communication(channel=)` | Record outbound messages (Slack, email, etc.) |
| `waxell.flush()` / `waxell.flush_sync()` | Flush buffered data for long-running agents |
| `waxell.diagnose()` | Introspect SDK state and configuration |

## Advanced: Context Manager

For complex scenarios where decorators don't fit -- multi-step orchestration, batch processing, conditional context creation -- use `WaxellContext` directly:

```python
from waxell_observe import WaxellContext

async with WaxellContext(
    agent_name="research-agent",
    session_id="sess_abc123",
    user_id="user_456",
) as ctx:
    result = await run_research_pipeline(query)
    ctx.record_llm_call(model="claude-sonnet-4", tokens_in=500, tokens_out=200)
    ctx.record_step("summarize", output={"summary": result})
    ctx.set_result({"answer": result})
```

See the [Context Manager](./integrations/context-manager) page for the full API.

## LangChain Integration

Drop-in callback handler for any LangChain chain or agent:

```python
from waxell_observe.integrations.langchain import WaxellLangChainHandler

handler = WaxellLangChainHandler(agent_name="langchain-agent")
result = chain.invoke(input, config={"callbacks": [handler]})
handler.flush_sync(result={"output": result})
```

## What You Get

| Feature | Description |
|---------|-------------|
| **LLM Call Tracking** | Model, token counts, cost, prompt/response previews for every LLM call |
| **LLM Call Explorer** | Browse, filter, and inspect every LLM call with prompt/response viewer |
| **Session Tracking** | Group related runs by session for conversation-level analytics |
| **User Tracking** | Per-user cost attribution, usage patterns, and analytics |
| **Scoring** | Capture quality scores via SDK or UI annotations |
| **Annotation Queues** | Human review workflows for manual quality assessment |
| **Prompt Management** | Version-controlled prompts with labels, playground, and SDK retrieval |
| **Cost Analytics** | Model usage breakdown, per-user costs, custom pricing overrides |
| **Policy Enforcement** | Pre-execution and mid-execution checks with allow/block/warn/throttle actions |
| **Behavior Tracking** | Structured spans for tools, retrievals, decisions, reasoning, retries |
| **Approval Workflows** | Human-in-the-loop approval for policy-blocked actions |
| **Conversation Tracking** | Auto-captured conversation state, context utilization, message counts |

## Framework Compatibility

Waxell Observe works with any Python agent framework:

- **OpenAI** -- auto-instrumentation or decorators
- **Anthropic** -- auto-instrumentation or decorators
- **LangChain / LangGraph** -- first-class callback handler
- **LiteLLM** -- unified API for 100+ providers
- **LlamaIndex** -- auto-instrumentation or decorators
- **CrewAI** -- auto-instrumentation or decorators
- **Custom frameworks** -- decorators or context manager
- **Any Python code** -- if it runs Python, you can observe it

## Next Steps

- [Quickstart](./quickstart) -- Get up and running in 5 minutes
- [Decorator Pattern](./integrations/decorator) -- Full `@observe` reference with all parameters
- [Auto-Instrumentation](./integrations/auto-instrumentation) -- Zero-code tracing for 200+ libraries
- [Behavior Tracking](./features/behavior-tracking) -- Deep dive into tools, retrievals, decisions, reasoning
- [Claude Skills](https://github.com/waxell-ai/claude-skills) -- Let your coding agent instrument and govern your agents for you
- [Examples on GitHub](https://github.com/waxell-ai/waxell-agent-examples) -- Complete runnable agents for every provider and pattern
- [FAQ](./troubleshooting/faq) -- Answers to common questions


--------------------------------------------------------------------------------
# Quickstart: Observe Your Agents
URL: https://waxell.ai/docs/observe/quickstart
Description: Add full observability to your AI agents -- auto-instrumentation in 2 lines, decorators for structure, WaxellContext for full control.
--------------------------------------------------------------------------------

# Quickstart: Observe Your Agents

Add observability to any Python AI agent in under 5 minutes. There are three levels of instrumentation -- **most agents only need the first one**:

1. **Auto-instrument** -- 2 lines of code, zero changes to your agent.
2. **Decorators** -- when you want more structure, or something auto-instrumentation can't see.
3. **`WaxellContext`** -- explicit lifecycle control when decorators don't fit your code shape.

## Before You Start: Two Shortcuts

**[Claude Skills](https://github.com/waxell-ai/claude-skills)** -- don't instrument by hand at all. Install the Waxell skills and your coding agent instruments and governs your agents for you. If you use a coding agent, start here.

**[Working examples](https://github.com/waxell-ai/waxell-agent-examples)** -- complete, runnable agents for every provider, framework, and pattern: instrumentation, decorators, policies, multi-agent, RAG, streaming, per-end-user attribution. Clone one that matches your stack instead of starting from scratch.

## Two fast paths to your first run

Pick the one closest to what you have today. All three end with a fully-instrumented agent emitting runs to your tenant — they just start from different places.

| If you... | Use | Time |
|---|---|---|
| Want a working agent to copy and adapt | [**Agent Examples Repo**](https://github.com/waxell-ai/waxell-agent-examples) — 10 conversational REPLs, each demonstrates one Waxell capability (decorator, tools, RAG, end-user attribution, policies, multi-agent, streaming, …) | ~3 min |
| Already have a Python agent and want to add Waxell to it | The [**`instrument-with-waxell-observe` Claude Code skill**](https://github.com/waxell-ai/claude-skills) — open your agent in Claude Code, invoke the skill, it installs `wax` + the SDK, wires the decorator, and verifies one run lands | ~5 min |
| Want to understand the pattern by hand first | Skip to [Prerequisites](#prerequisites) below — the manual walkthrough covers the same flow Path A and Path B automate | ~10 min |

### Path A — Clone a working example

The [`waxell-ai/waxell-agent-examples`](https://github.com/waxell-ai/waxell-agent-examples) repo ships 10 self-contained conversational agents covering every Waxell capability. Each example is one folder with `agent.py`, `setup.sh`, `README.md`, and a `requirements.txt` — no cross-dependencies.

```bash
git clone https://github.com/waxell-ai/waxell-agent-examples.git
cd waxell-agent-examples

# Seed .env from your local wax profile (or copy .env.example and fill in by hand)
./scripts/seed-env-from-wax.sh

# Pick an example, install its deps, run it
./scripts/setup-example.sh 01-hello-waxell
source examples/01-hello-waxell/.venv/bin/activate
python examples/01-hello-waxell/agent.py
```

After a few REPL turns, `wax runs list --limit 5` shows the runs in your tenant. The numbered index in the [examples README](https://github.com/waxell-ai/waxell-agent-examples#examples) tells you which example demonstrates which capability — copy whichever is closest to what you're building, rename the agent, and start changing the system prompt.

### Path B — Instrument your existing agent

If you have a Python agent you already use, the [**`instrument-with-waxell-observe`**](https://github.com/waxell-ai/claude-skills) Claude Code skill retrofits it without you copy-pasting from anywhere. Open your agent in Claude Code and invoke:

```
/instrument-with-waxell-observe
```

The skill walks the file, then:

1. Checks `wax` CLI is installed; installs and configures it if not.
2. Adds the two-line decorator pattern (`waxell.init()` at module top + `@waxell.observe(...)` on your entry function) at the right place in your file.
3. Installs `waxell-observe` and pins a known-working version.
4. Runs your agent once to verify a run lands in your tenant.
5. Walks you through one starter governance policy (PII block, cost cap, etc.) if you want it.

Your existing agent runs unchanged afterward — it just also emits full telemetry. The skill source + README is in [`waxell-ai/claude-skills`](https://github.com/waxell-ai/claude-skills).

### Path C — Manual walkthrough

If you'd rather understand the pattern by hand, the rest of this page walks through the exact same flow Path A and Path B automate. Continue with **Prerequisites** below.

---

## Prerequisites

- Python 3.10+
- A Waxell API key (get one from your Waxell control plane dashboard)

Install the Observe SDK, plus the [`waxell`](https://pypi.org/project/waxell/)
package for the `wax` CLI — used below to confirm your runs landed:

```bash
pip install waxell-observe   # the instrumentation SDK
pip install waxell           # the wax CLI (verify runs, manage your tenant)
```

### Verify your install

```bash
wax --version                     # CLI is installed and on your PATH
python -c "import waxell_observe"  # SDK imports cleanly (no output = success)
```

Once your API key is set (via the env vars shown below, or `wax login`),
`wax whoami` confirms the credentials resolve and `wax doctor` runs a full
health check. If `wax` isn't found or the import errors, see the
[CLI install troubleshooting](/reference/cli#troubleshooting-installation)
(`command not found`, PATH, PEP 668).

## Level 1: Auto-Instrument (2 lines)

Call `init()` **before** importing any LLM SDK. This auto-instruments 200+ libraries (OpenAI, Anthropic, Groq, LiteLLM, Cohere, Mistral, Gemini, LangChain, LlamaIndex, vector DBs, and more) with zero changes to your agent code.

```python


waxell.init(api_key="wax_sk_...", api_url="https://acme.waxell.dev")

# Import LLM SDKs AFTER init()
from openai import OpenAI

client = OpenAI()
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}]
)
# Automatically traced: model, tokens, cost, latency
```

That's it. Every LLM call in your process is now tracked -- model, tokens, cost, latency, prompt/response previews -- and visible in the Waxell dashboard.

**Environment Variables**
You can also configure via environment variables:
```bash


```
Then just call `waxell.init()` without arguments.


See [Auto-Instrumentation](./integrations/auto-instrumentation) for the full library list and configuration options.

## Level 2: Decorators (more detail, or things auto-instrumentation missed)

Auto-instrumentation sees your LLM and framework calls. It doesn't know which function *is* your agent, what your tools do, or why your agent made a decision. When you need that structure, add decorators.

Start with `@observe` on your agent's entrypoint -- it creates a named, tracked run for each call with input/output capture and policy enforcement:

```python


waxell.init()

from openai import OpenAI

client = OpenAI()

@waxell.observe(agent_name="support-bot")
async def handle_ticket(query: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": query}],
    )
    return response.choices[0].message.content
```

Then decorate the internal functions you care about. Each decorated call becomes a structured span in the trace:

```python
@waxell.tool(tool_type="vector_db")          # tool calls: name, inputs, output, duration
def search_knowledge_base(query: str) -> list[dict]: ...

@waxell.retrieval(source="pinecone")          # RAG: query, documents, scores
async def retrieve_docs(query: str) -> list[dict]: ...

@waxell.decision(name="route_query", options=["faq", "technical", "billing"])
async def classify_query(query: str) -> dict: ...   # routing: chosen option, reasoning
```

There are also decorators for reasoning steps (`@reasoning_dec`), pipeline steps (`@step_dec`), and retry logic (`@retry_dec`), plus inline one-liners -- `waxell.score()`, `waxell.tag()`, `waxell.metadata()`, `waxell.step()` -- for enrichment without wrapping anything. Session and user attribution go straight on the decorator:

```python
@waxell.observe(agent_name="chat-agent", session_id="session-abc123", user_id="user-456")
async def handle_message(message: str) -> str: ...
```

**Behavior outside @observe**
Behavior decorators are no-ops outside an `@observe` or `WaxellContext` scope -- your functions work normally with zero overhead.


See [Decorator Pattern](./integrations/decorator) for the full `@observe` reference and [Behavior Tracking](./features/behavior-tracking) for every decorator and inline function.

## Level 3: `WaxellContext` (full control)

If decorators don't fit -- you can't wrap the entrypoint, you're instrumenting someone else's framework loop, or you need explicit control over when a run starts and ends -- use the context manager:

```python
from waxell_observe import WaxellContext

async with WaxellContext(
    agent_name="chat-agent",
    session_id="session-abc123",  # groups related runs
    user_id="user-456",           # per-user cost attribution
) as ctx:
    response = await call_llm(prompt)
    ctx.set_result({"output": response})
```

Everything that works inside `@observe` -- auto-instrumented LLM calls, behavior decorators, inline enrichment -- works identically inside a `WaxellContext` block.

See [Context Manager](./integrations/context-manager) for the full reference.

## What You Get in the Dashboard

Every run appears in the Waxell dashboard with:

- **Agent name, workflow, and execution status**
- **Captured inputs and outputs**
- **LLM calls** with model, tokens, cost, latency, prompt/response previews
- **Behavior spans** -- tool calls, retrievals, decisions, reasoning steps
- **Scores, tags, and metadata**
- **Session timeline** grouping related runs, with per-user cost attribution

## Next Steps

- [Auto-Instrumentation](./integrations/auto-instrumentation) -- full list of 200+ auto-instrumented libraries
- [Decorator Pattern](./integrations/decorator) -- full `@observe` reference with all parameters
- [Behavior Tracking](./features/behavior-tracking) -- every behavior decorator and inline enrichment function
- [Context Manager](./integrations/context-manager) -- full `WaxellContext` reference
- [Cost Management](./features/cost-management) -- track and control LLM spending
- [Policy & Governance](./features/governance) -- pre-execution and mid-execution policy checks
- [FAQ](./troubleshooting/faq) and [Common Mistakes](./troubleshooting/common-mistakes)
- [Meet Your Assistant](/tutorials/meet-your-assistant) -- now that data is flowing, ask the built-in assistant for your first fleet briefing


--------------------------------------------------------------------------------
# Installation & Configuration
URL: https://waxell.ai/docs/observe/installation
Description: Install waxell-observe and configure API credentials via environment variables, CLI config file, or programmatic setup.
--------------------------------------------------------------------------------

# Installation & Configuration

## Installation

Install the base package:

```bash
pip install waxell-observe
```

### Optional Extras

```bash
pip install waxell-observe[langchain]      # LangChain callback handler
pip install waxell-observe[crewai]         # CrewAI integration
pip install waxell-observe[llamaindex]     # LlamaIndex integration
pip install waxell-observe[langgraph]      # LangGraph integration
pip install waxell-observe[infra]          # Infrastructure instrumentation (HTTP, DB, caches, queues)
pip install waxell-observe[otel]           # OpenTelemetry tracing (OTLP export)
pip install waxell-observe[all-providers]  # All LLM provider SDKs
```

You can combine extras: `pip install waxell-observe[langchain,infra,otel]`

## Quick Setup: init()

The fastest way to get started -- call `init()` once at application startup:

```python


waxell_observe.init(
    api_key="wax_sk_...",
    api_url="https://waxell.dev",
)

# Import LLM SDKs AFTER init() -- they're now auto-instrumented
from openai import OpenAI
```

This:
1. Configures the API client
2. Auto-instruments installed AI/ML libraries (OpenAI, Anthropic, LiteLLM, and many more)
3. Auto-instruments installed infrastructure libraries (HTTP clients, databases, caches)
4. Enables OpenTelemetry tracing

### init() Parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `api_key` | `str` | `""` | Waxell API key (`wax_sk_...`). Falls back to `WAXELL_API_KEY` env var. |
| `api_url` | `str` | `""` | Waxell API URL. Falls back to `WAXELL_API_URL` env var. |
| `capture_content` | `bool` | `False` | Include prompt/response content in traces. |
| `instrument` | `list[str] \| None` | `None` | Explicit list of AI/ML libraries to auto-instrument (e.g. `["openai", "anthropic"]`). `None` means auto-detect all installed libraries. |
| `exclude` | `list[str] \| None` | `None` | Skip these AI/ML libraries during auto-instrumentation (e.g. `["litellm", "mcp"]`). Falls back to `WAXELL_EXCLUDE` env var (comma-delimited). Takes precedence over `instrument`. |
| `instrument_infra` | `bool` | `True` | Enable auto-instrumentation of infrastructure libraries (HTTP clients, databases, caches, queues). Falls back to `WAXELL_INSTRUMENT_INFRA` env var. |
| `infra_libraries` | `list[str] \| None` | `None` | Only instrument these specific infra libraries (e.g. `["redis", "httpx"]`). `None` means auto-detect all. |
| `infra_exclude` | `list[str] \| None` | `None` | Instrument all infra libraries except these (e.g. `["celery", "grpc"]`). Falls back to `WAXELL_INFRA_EXCLUDE` env var (comma-delimited). |
| `on_policy_block` | `Callable \| None` | `None` | Default handler for all contexts when a policy blocks execution. Receives `PolicyViolationError`, returns `ApprovalDecision`. Built-in: `prompt_approval`, `auto_approve`, `auto_deny`. |
| `resource_attributes` | `dict \| None` | `None` | Custom OTel resource attributes applied to all spans (e.g. `{"deployment.environment": "production"}`). |
| `debug` | `bool` | `False` | Enable debug logging and console span export. |
| `prompt_guard` | `bool` | `False` | Enable client-side prompt guard (regex PII/credential/injection detection). Falls back to `WAXELL_PROMPT_GUARD` env var. |
| `prompt_guard_server` | `bool` | `False` | Also check server-side guard service (ML-powered detection). Falls back to `WAXELL_PROMPT_GUARD_SERVER` env var. |
| `prompt_guard_action` | `str` | `"block"` | Action when violations are found: `"block"` (raise error), `"warn"` (log and continue), or `"redact"` (replace and continue). Falls back to `WAXELL_PROMPT_GUARD_ACTION` env var. |

### Supported Libraries

`init()` auto-instruments a wide range of AI/ML libraries. See [Auto-Instrumentation](./integrations/auto-instrumentation) for the full list and details.

### Drop-in Imports

Alternative to `init()` -- import pre-instrumented modules:

```python
from waxell_observe.openai import openai
from waxell_observe.anthropic import anthropic
```

### Kill Switch

Disable all observability without code changes:

```bash

```

---

## Configuration

Waxell Observe requires two values to connect to your control plane:

| Setting | Description | Example |
|---------|-------------|---------|
| `api_url` | Your Waxell control plane URL | `https://acme.waxell.dev` |
| `api_key` | Your API key | `wax_sk_abc123...` |

### Configuration Priority

Configuration is resolved in the following order (highest priority first):

1. **Explicit constructor arguments** -- passed directly to `WaxellObserveClient()`
2. **Global configure()** -- set via `WaxellObserveClient.configure()`
3. **CLI config file** -- `~/.waxell/config`
4. **Environment variables** -- `WAXELL_API_URL` / `WAXELL_API_KEY`

Higher-priority sources override lower-priority ones. You can mix methods -- for example, set the URL in an environment variable and override the API key per-instance.

---

### Method 1: Environment Variables

The simplest approach for most deployments:

```bash


```

Both long-form and short-form variable names are supported:

| Long Form | Short Form |
|-----------|------------|
| `WAXELL_API_URL` | `WAX_API_URL` |
| `WAXELL_API_KEY` | `WAX_API_KEY` |

The long-form name takes precedence if both are set.

---

### Method 2: CLI Config File

Create a config file at `~/.waxell/config` in INI format:

```ini
[default]
api_url = https://acme.waxell.dev
api_key = wax_sk_abc123...
```

You can define multiple profiles by using different section names. The `[default]` section is used automatically. If no `[default]` section exists, the first section in the file is used.

**WARNING**
Keep your `~/.waxell/config` file secure. It contains your API key in plain text. Set file permissions to owner-only: `chmod 600 ~/.waxell/config`


---

### Method 3: Programmatic configure()

Call `WaxellObserveClient.configure()` once at application startup:

```python
from waxell_observe import WaxellObserveClient

WaxellObserveClient.configure(
    api_url="https://acme.waxell.dev",
    api_key="wax_sk_abc123...",
)
```

All subsequent `WaxellObserveClient()` instances, `@observe` decorators, and `WaxellContext` managers will use this configuration automatically.

**TIP**
This is the recommended approach for application code. Set it once in your application entry point and all observability calls pick it up.


---

### Method 4: Per-Instance

Pass credentials directly when creating a client:

```python
from waxell_observe import WaxellObserveClient

client = WaxellObserveClient(
    api_url="https://acme.waxell.dev",
    api_key="wax_sk_abc123...",
)
```

This overrides all other configuration sources for that specific client instance. You can also pass a `client` parameter to the decorator and context manager:

```python
from waxell_observe import observe, WaxellContext

@observe(agent_name="my-agent", client=client)
async def my_function():
    ...

async with WaxellContext(agent_name="my-agent", client=client) as ctx:
    ...
```

## Verifying Configuration

Check whether the client is properly configured:

```python
from waxell_observe import WaxellObserveClient

# After calling configure()
print(WaxellObserveClient.is_configured())  # True or False

# Get the current global config
config = WaxellObserveClient.get_config()
if config:
    print(config.api_url)
```

**INFO**
If the client is not configured when making API calls, it logs a warning and returns empty results instead of raising an error. This means missing configuration degrades gracefully rather than crashing your agent.


## Next Steps

- [Quickstart](./quickstart) -- Get started in 2 minutes
- [Auto-Instrumentation](./integrations/auto-instrumentation) -- Zero-code tracing
- [OpenAI Integration](./integrations/openai) -- OpenAI-specific patterns
- [Anthropic Integration](./integrations/anthropic) -- Anthropic-specific patterns
- [Decorator Pattern](./integrations/decorator) -- Add observability with `@observe`
- [REST API Reference](./api/endpoints) -- Direct API integration


--------------------------------------------------------------------------------
# Auto-Instrumentation
URL: https://waxell.ai/docs/observe/integrations/auto-instrumentation
Description: Zero-code observability for 200+ AI/ML libraries including LLM providers, vector databases, agent frameworks, and more
--------------------------------------------------------------------------------

# Auto-Instrumentation

The simplest way to add observability to your AI agents -- two lines of code and all your LLM calls are automatically traced.

## Quick Start

```python

waxell_observe.init(api_key="wax_sk_...", api_url="https://waxell.dev")

# Import LLM SDKs AFTER init() -- they're now auto-instrumented
from openai import OpenAI

client = OpenAI()
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}]
)
# Automatically traced with model, tokens, cost, latency
```

## How It Works

When you call `waxell_observe.init()`, the SDK:

1. Detects installed LLM libraries (OpenAI, Anthropic, etc.)
2. Patches their HTTP clients to capture request/response data
3. Emits OpenTelemetry spans for each LLM call
4. Records token counts, costs, and latencies automatically

LLM calls made **outside** any `@observe` decorator or `WaxellContext` are automatically buffered by a background collector and flushed to auto-generated runs (named `auto:{model}`). This means you get visibility into every LLM call without any additional code beyond `init()`.

## The init() Function

```python
waxell_observe.init(
    api_key: str = "",                      # Waxell API key (wax_sk_...)
    api_url: str = "",                      # Waxell API URL
    capture_content: bool = False,          # Include prompt/response in traces
    instrument: list[str] | None = None,    # AI/ML library list (auto-detect if None)
    instrument_infra: bool = True,          # Auto-instrument infra (HTTP, DB, cache)
    infra_libraries: list[str] | None = None,  # Only these infra libs (None = all)
    infra_exclude: list[str] | None = None, # Exclude these infra libs
    resource_attributes: dict | None = None,  # Custom OTel resource attributes
    debug: bool = False,                    # Enable debug logging
    prompt_guard: bool = False,             # Enable client-side prompt guard
    prompt_guard_server: bool = False,      # Also check server-side guard (ML-powered)
    prompt_guard_action: str = "block",     # "block", "warn", or "redact"
)
```

See [Installation & Configuration](../installation) for full parameter details.

### Configuration Priority

1. Explicit arguments to `init()`
2. Environment variables (`WAXELL_API_KEY`, `WAXELL_API_URL`)
3. CLI config file (`~/.waxell/config`)

### Environment Variables

```bash


```

## Supported Libraries

**LLM Providers**

| Library | Key | Notes |
|---------|-----|-------|
| OpenAI | `openai` | Chat, completions, embeddings |
| Anthropic | `anthropic` | Messages API |
| Google Gemini | `gemini` | Gemini API |
| AWS Bedrock | `bedrock` | Bedrock runtime |
| Mistral AI | `mistral` | Chat, embeddings |
| Cohere | `cohere` | Chat, embed, rerank |
| Groq | `groq` | Fast inference |
| LiteLLM | `litellm` | Unified multi-provider API |
| Ollama | `ollama` | Local model serving |
| Together AI | `together` | Together inference API |
| Vertex AI | `vertex_ai` | Google Cloud AI |
| HuggingFace | `huggingface` | Inference API |

**Vector Databases**

| Library | Key | Notes |
|---------|-----|-------|
| Pinecone | `pinecone` | Managed vector DB |
| ChromaDB | `chroma` | Embedded vector DB |
| Weaviate | `weaviate` | Vector search engine |
| Qdrant | `qdrant` | Vector similarity search |
| Milvus | `milvus` | Distributed vector DB |
| pgvector | `pgvector` | PostgreSQL vector extension |
| FAISS | `faiss` | Facebook AI similarity search |
| LanceDB | `lancedb` | Serverless vector DB |

**Agent Frameworks**

| Library | Key | Notes |
|---------|-----|-------|
| LangChain | `langchain` | Chain and agent orchestration |
| CrewAI | `crewai` | Multi-agent collaboration |
| OpenAI Agents SDK | `openai_agents` | OpenAI agent framework |
| AutoGen | `autogen` | Multi-agent conversations |
| LlamaIndex | `llamaindex` | Data framework for LLMs |
| Haystack | `haystack` | NLP pipeline framework |
| PydanticAI | `pydanticai` | Type-safe AI agents |
| DSPy | `dspy` | Programming with foundation models |
| Google ADK | `google_adk` | Google Agent Development Kit |
| Claude Agent SDK | `claude_agents` | Anthropic agent framework |

**Safety & Guardrails**

| Library | Key | Notes |
|---------|-----|-------|
| Guardrails AI | `guardrails_ai` | Output validation |
| NeMo Guardrails | `nemo_guardrails` | Programmable guardrails |
| LLM Guard | `llm_guard` | Input/output scanning |

**INFO**
The tables above highlight the most commonly used libraries. The SDK supports **200+ libraries** in total across additional categories including embeddings/rerankers, evaluation frameworks, voice/speech, RAG frameworks, local inference engines, and more. The full registry is defined in the [instrumentor source](https://github.com/waxell-ai/agentforge/blob/main/observe/waxell-observe/src/waxell_observe/instrumentors/__init__.py).


### Selective Instrumentation

To instrument only specific libraries:

```python
waxell_observe.init(
    api_key="wax_sk_...",
    api_url="https://waxell.dev",
    instrument=["openai", "anthropic"],  # Only these two
)
```

## Drop-in Imports

Alternative to `init()` -- import pre-instrumented modules:

```python
# Instead of: from openai import OpenAI
from waxell_observe.openai import openai

client = openai.OpenAI()
response = client.chat.completions.create(...)  # Auto-traced
```

```python
# Instead of: import anthropic
from waxell_observe.anthropic import anthropic

client = anthropic.Anthropic()
response = client.messages.create(...)  # Auto-traced
```

## Import Order Matters

Auto-instrumentation patches LLM SDKs when they're imported. You must call `init()` **before** importing the SDK:

```python
# CORRECT

waxell_observe.init(api_key="...")

from openai import OpenAI  # Patched!
```

```python
# WRONG - OpenAI already imported, won't be patched
from openai import OpenAI


waxell_observe.init(api_key="...")  # Too late!
```

## Adding Structure to Auto-Instrumented Calls

Auto-instrumentation captures LLM calls automatically. Add structure with decorators or context managers to group calls into runs, record behaviors, and enrich traces.

### Decorators + Auto-Instrumentation (Recommended)

The simplest way to add structure -- decorators handle run tracking and behavior recording while `init()` handles LLM capture:

```python


# Auto-instrument LLM SDKs
waxell.init(api_key="wax_sk_...", api_url="https://waxell.dev")

from openai import AsyncOpenAI

client = AsyncOpenAI()

@waxell.decision(name="classify_intent", options=["question", "action", "chitchat"])
async def classify(query: str) -> dict:
    response = await client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": f"Classify: {query}"}],
    )
    # LLM call auto-captured; return value recorded as decision
    return {"chosen": "question", "reasoning": response.choices[0].message.content}

@waxell.observe(agent_name="support-bot")
async def handle_query(query: str) -> str:
    # Auto-instrumented LLM calls + decorator-recorded behaviors
    classification = await classify(query)

    response = await client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": query}],
    )
    answer = response.choices[0].message.content

    # Enrich with scores and tags
    waxell.score("helpfulness", 0.9)
    waxell.tag("intent", classification["chosen"])
    return answer
```

### Context Manager + Auto-Instrumentation (Alternative)

Use the context manager when you need maximum control over the run lifecycle -- for example, mid-execution policy checks across many calls or explicit start/complete handling. Even inside a `WaxellContext`, LLM calls are still auto-captured -- no manual `record_llm_call` needed.

```python

waxell_observe.init(api_key="wax_sk_...")

from waxell_observe import WaxellContext
from openai import OpenAI

client = OpenAI()

async with WaxellContext(agent_name="my-agent") as ctx:
    # LLM call auto-traced AND linked to this context -- no manual record needed
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": "Hello!"}]
    )

    # Use ctx for things auto-instrumentation can't infer
    ctx.set_tag("user_type", "premium")
    await ctx.check_policy()  # Mid-execution policy check
```

## Kill Switch

Disable all observability without changing code:

```bash

```

When disabled:
- `init()` becomes a no-op
- Context managers pass through without recording
- Decorators execute functions without wrapping
- No network calls to Waxell servers

## Shutdown

Gracefully flush pending traces before exit:

```python
waxell_observe.shutdown()
```

This is called automatically on process exit, but explicit shutdown ensures all data is flushed in:
- Serverless functions (Lambda, Cloud Functions)
- Short-lived scripts
- Test suites

## Programmatic Control

### Manual Instrument/Uninstrument

```python
from waxell_observe.instrumentors import instrument_all, uninstrument_all

# Instrument all detected libraries
results = instrument_all()
# {"openai": True, "anthropic": True, ...}

# Restore original behavior
uninstrument_all()
```

## What Gets Captured

For each LLM call, auto-instrumentation records:

| Field | Description |
|-------|-------------|
| `model` | Model name (gpt-4o, claude-sonnet-4, etc.) |
| `tokens_in` | Input/prompt token count |
| `tokens_out` | Output/completion token count |
| `cost` | Estimated USD cost |
| `latency` | Request duration |
| `provider` | openai, anthropic, etc. |
| `prompt_preview` | First 500 chars of prompt (if `capture_content=True`) |
| `response_preview` | First 500 chars of response (if `capture_content=True`) |

## Conversation Tracking (Automatic)

When auto-instrumentation is active, waxell automatically captures:

- **User messages** — extracted from the messages array sent to the LLM
- **Agent responses** — the final text response (not tool-calling intermediaries)
- **Context window metrics** — message count, turn count, token utilization
- **System prompt tracking** — detects system prompt changes across calls

This works across all 13+ supported providers with zero code changes. User messages appear as `io:user_message` spans and agent responses as `io:agent_response` spans in the trace timeline.

### Deduplication

If you also call `waxell.user_message()` or `waxell.agent_response()` manually for the same content that was auto-captured, the duplicate is automatically suppressed.

See [Conversation Tracking](../features/conversation-tracking) for full details.

## Next Steps

- [OpenAI Integration](./openai) -- Detailed OpenAI patterns
- [Anthropic Integration](./anthropic) -- Anthropic-specific setup
- [Context Manager](./context-manager) -- Fine-grained control
- [Decorator Pattern](./decorator) -- Function-level tracing
- [Conversation Tracking](../features/conversation-tracking) -- Auto-captured conversation data


--------------------------------------------------------------------------------
# Decorator Pattern
URL: https://waxell.ai/docs/observe/integrations/decorator
Description: Add observability and governance to any Python function with the @observe decorator.
--------------------------------------------------------------------------------

# Decorator Pattern

The `@observe` decorator (also available as `@waxell_agent`) is the simplest way to add observability to any Python function. It wraps your function with automatic run tracking, IO capture, and policy enforcement -- with zero changes to your function's logic.

## Basic Usage

```python
from waxell_observe import observe

@observe(agent_name="support-bot")
async def handle_ticket(query: str) -> str:
    return await process_query(query)
```

**Alias**
`@observe` and `@waxell_agent` are identical. Use whichever reads better in your codebase.


Every call to `handle_ticket` now:
1. Checks policies (if `enforce_policy=True`)
2. Starts an execution run on the control plane
3. Captures function inputs and return value
4. Completes the run with success/error status

## Enhanced Decorator Options

### Session and User Tracking

Pass a session ID to group related runs, and user ID for attribution:

```python
@observe(
    agent_name="my-chatbot",
    session_id="session-abc-123",
    user_id="user_456",
    user_group="enterprise",
)
def chat(message: str):
    return call_llm(message)
```

**INFO**
The `session_id` on the decorator can be set statically (applies to every invocation) or dynamically at call time. See [Dynamic Call-time Overrides](#dynamic-call-time-overrides) below.


### Scores, Tags, and Metadata

Use the injected `waxell_ctx` or top-level convenience functions to enrich traces:

```python

from waxell_observe import observe

@observe(agent_name="my-agent")
async def run_agent(query: str, waxell_ctx=None) -> str:
    # Top-level convenience functions (no ctx needed)
    waxell_observe.tag("pipeline", "rag-v2")
    waxell_observe.metadata("model_version", "gpt-4-turbo")

    response = await call_llm(query)

    # Or use the context directly
    if waxell_ctx:
        waxell_ctx.record_score(
            name="relevance",
            value=0.95,
            data_type="numeric",
        )

    # Record multiple score types
    waxell_observe.score("quality", 0.92)
    waxell_observe.score("safety", True, data_type="boolean")
    waxell_observe.score("category", "informational", data_type="categorical")

    return response
```

## Parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `agent_name` | `str \| None` | `None` | Name for this agent. Defaults to the decorated function's name |
| `workflow_name` | `str` | `"default"` | Workflow name for grouping runs |
| `enforce_policy` | `bool` | `True` | Check policies before execution. Raises `PolicyViolationError` if blocked |
| `capture_io` | `bool` | `True` | Capture function inputs and outputs in the run record |
| `session_id` | `str` | `""` | Session ID for grouping related runs |
| `user_id` | `str` | `""` | End-user ID for attribution and analytics |
| `user_group` | `str` | `""` | User group for authorization policies |
| `mid_execution_governance` | `bool` | `False` | Flush data and check governance on each `record_step()` call |
| `auto_grounding` | `bool` | `False` | Auto-bridge retrieval scores to grounding governance. When `True`, retrieval scores from `@retrieval` or `record_retrieval()` are forwarded to grounding policy evaluation |
| `on_policy_block` | `Callable \| None` | `None` | Callback when a policy blocks execution. Receives a `PolicyViolationError` and returns an `ApprovalDecision`. Built-in: `prompt_approval`, `auto_approve`, `auto_deny` |
| `client` | `WaxellObserveClient \| None` | `None` | Pre-configured client instance. If `None`, uses current configuration |

## Async Functions

The decorator works natively with async functions:

```python
@observe(agent_name="research-agent", workflow_name="analyze")
async def analyze_data(dataset: dict) -> dict:
    results = await run_analysis(dataset)
    return {"findings": results}
```

## Sync Functions

Sync functions are also supported. The decorator wraps them in an async execution context internally:

```python
@observe(agent_name="classifier")
def classify_text(text: str) -> str:
    return model.predict(text)
```

**WARNING**
Sync wrappers use `asyncio.run()` by default. When called inside an already-running event loop (e.g., Jupyter notebooks, `uvicorn`, or other async frameworks), the decorator falls back to running in a `ThreadPoolExecutor` to avoid blocking the event loop. This works but adds threading overhead -- if your application is async, prefer making the decorated function async for best performance.


## Context Injection

To record LLM calls, steps, or perform mid-execution policy checks, add a `waxell_ctx` parameter to your function signature. The decorator automatically injects a `WaxellContext` instance:

```python
@observe(agent_name="support-bot")
async def handle_ticket(query: str, waxell_ctx=None) -> str:
    # Record an LLM call
    response = await call_openai(query)
    if waxell_ctx:
        waxell_ctx.record_llm_call(
            model="gpt-4o",
            tokens_in=150,
            tokens_out=80,
            task="answer_question",
            prompt_preview=query[:500],
            response_preview=response[:500],
        )

    # Record an execution step
    if waxell_ctx:
        waxell_ctx.record_step("generate_response", output={"length": len(response)})

    return response
```

**TIP**
Default `waxell_ctx=None` ensures your function works normally when called without the decorator -- for example, in unit tests. Always guard with `if waxell_ctx:` before recording.


### Available Context Methods

When `waxell_ctx` is injected, you have access to all `WaxellContext` recording methods:

| Method | Description |
|--------|-------------|
| `record_llm_call(*, model, tokens_in, tokens_out, cost=0.0, task="", prompt_preview="", response_preview="", duration_ms=None, provider="")` | Record an LLM call with token counts and optional cost |
| `record_step(step_name, output=None)` | Record a named execution step |
| `record_score(name, value, data_type="numeric", comment="")` | Record a quality score or feedback metric |
| `record_tool_call(*, name, input="", output="", duration_ms=None, status="ok", tool_type="function", error="")` | Record a tool/function call |
| `record_retrieval(*, query, documents, source="", duration_ms=None, top_k=None, scores=None)` | Record a RAG retrieval operation |
| `record_decision(*, name, options, chosen, reasoning="", confidence=None, metadata=None, instrumentation_type="manual")` | Record a decision/routing point |
| `record_reasoning(*, step, thought, evidence=None, conclusion="")` | Record a reasoning/chain-of-thought step |
| `record_retry(*, attempt, reason, strategy="retry", original_error="", fallback_to="", max_attempts=None)` | Record a retry or fallback event |
| `set_tag(key, value)` | Set a searchable tag (string value) on the current span |
| `set_metadata(key, value)` | Set arbitrary metadata (any JSON-serializable value) |
| `set_result(result)` | Set the run result (overrides auto-captured output) |
| `check_policy()` / `check_policy_sync()` | Perform a mid-execution policy check (async / sync) |
| `record_policy_check(*, policy_name, action, category="", reason="", duration_ms=0, phase="pre_execution", priority=100)` | Record a policy evaluation result |
| `run_id` | Property returning the current run ID |

## IO Capture

When `capture_io=True` (the default), the decorator captures:

- **Inputs**: All positional and keyword arguments, serialized to JSON-safe values
- **Outputs**: The return value, serialized as a dict

Non-serializable values are converted to their string representation. To disable capture (for sensitive data):

```python
@waxell_agent(agent_name="sensitive-agent", capture_io=False)
async def process_pii(data: dict) -> dict:
    ...
```

## Error Handling

If the decorated function raises an exception, the run is automatically completed with `status="error"` and the error message is recorded:

```python
@observe(agent_name="risky-agent")
async def might_fail(input: str) -> str:
    if not input:
        raise ValueError("Input required")
    return await process(input)

# The run is recorded with status="error" and the ValueError message
try:
    result = await might_fail("")
except ValueError:
    pass  # The error is already recorded in the run
```

The original exception is always re-raised so your error handling works as expected.

## Policy Enforcement

With `enforce_policy=True`, the decorator checks policies before running your function:

```python
from waxell_observe.errors import PolicyViolationError

@observe(agent_name="my-agent", enforce_policy=True)
async def my_function(query: str) -> str:
    return await process(query)

try:
    result = await my_function("test")
except PolicyViolationError as e:
    print(f"Blocked: {e}")
    print(f"Action: {e.policy_result.action}")
    print(f"Reason: {e.policy_result.reason}")
```

Set `enforce_policy=False` to skip the check:

```python
@observe(agent_name="my-agent", enforce_policy=False)
async def my_function(query: str) -> str:
    ...
```

## Approval Workflows

Use `on_policy_block` to handle policy blocks with a human-in-the-loop instead of raising an error:

```python


# Built-in terminal prompt
@waxell.observe(
    agent_name="my-agent",
    on_policy_block=waxell.prompt_approval,
)
async def my_agent(query: str):
    return await process(query)

# Custom handler (Slack, webhook, etc.)
async def slack_approval(error):
    channel = await post_to_slack(f"Agent blocked: {error}")
    reaction = await wait_for_reaction(channel, timeout=300)
    return waxell.ApprovalDecision(
        approved=(reaction == "thumbsup"),
        approver="ops-team",
    )

@waxell.observe(
    agent_name="production-agent",
    on_policy_block=slack_approval,
)
async def production_agent(query: str):
    return await process(query)
```

Built-in handlers:
- `waxell.prompt_approval` -- Interactive terminal Y/N prompt with timeout
- `waxell.auto_approve` -- Always approve (for testing)
- `waxell.auto_deny` -- Always deny (for testing)

## When to Use Decorator vs Context Manager

| Use Decorator When... | Use Context Manager When... |
|----------------------|----------------------------|
| You have a single function that represents an agent run | You need to wrap complex multi-step logic |
| You want minimal code changes | You need multiple policy checks during execution |
| Auto IO capture is sufficient | You want explicit control over run start/complete |
| One function = one run | One run spans multiple functions or classes |

## Full Example

When combined with `waxell.init()`, LLM calls are auto-captured. The `@observe` decorator adds run tracking, and specialized decorators (`@tool`, `@decision`, etc.) add behavior recording.

```python


# init() BEFORE importing LLM SDKs -- patches them for auto-instrumentation
waxell.init(api_key="wax_sk_...", api_url="https://waxell.dev")

from openai import AsyncOpenAI

client = AsyncOpenAI()

@waxell.tool(tool_type="api")
async def fetch_context(question: str) -> dict:
    """Tool calls are auto-recorded with timing and IO."""
    docs = await retrieve_documents(question)
    return {"docs": docs, "count": len(docs)}

@waxell.observe(
    agent_name="qa-agent",
    workflow_name="answer-question",
    enforce_policy=True,
)
async def answer_question(question: str) -> str:
    # Tool call -- auto-recorded by @tool decorator
    result = await fetch_context(question)

    # LLM call -- auto-captured by init(), no manual recording needed
    response = await client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": "Answer based on context."},
            {"role": "user", "content": f"{question}\n\n{result['docs']}"},
        ],
    )
    answer = response.choices[0].message.content

    # Enrich with scores and tags (convenience functions)
    waxell.score("relevance", 0.95)
    waxell.score("grounded", True, data_type="boolean")
    waxell.tag("pipeline", "rag-v2")
    waxell.metadata("doc_count", result["count"])

    return answer

# Run it
result = await answer_question("What is Waxell?")
```

## Dynamic Call-time Overrides

The `@observe` decorator supports passing context parameters at call time. Any keyword argument matching a WaxellContext parameter that is **not** in the wrapped function's signature is intercepted and passed to the context:

```python
@observe(agent_name="my-agent")
async def run(query: str):
    return await process(query)

# Dynamic session/user at call time:
result = await run(
    "What is RAG?",
    session_id="sess_abc123",
    user_id="user_456",
    user_group="enterprise",
)
```

Supported overrides: `session_id`, `user_id`, `user_group`, `enforce_policy`, `mid_execution_governance`, `client`, `inputs`, `metadata`, `workflow_name`.

---

## @tool Decorator

Auto-record function calls as tool invocations:

```python


@waxell.tool(tool_type="vector_db")
def create_index(dim: int):
    import faiss
    return faiss.IndexFlatL2(dim)

@waxell.tool(tool_type="api")
async def call_weather_api(city: str):
    return await httpx.get(f"https://api.weather.com/{city}")
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `name` | `str \| None` | `None` | Tool name. Defaults to the function name |
| `tool_type` | `str` | `"function"` | Classification: `"function"`, `"vector_db"`, `"database"`, `"api"` |

---

## @decision Decorator

Auto-record a function's return value as a decision:

```python


@waxell.decision(name="route_task", options=["direct", "research", "multi_agent"])
async def route_task(query: str) -> dict:
    response = await client.chat.completions.create(...)
    return {"chosen": "research", "reasoning": "Complex query", "confidence": 0.92}
```

The SDK extracts `chosen`, `reasoning`, and `confidence` from dict returns. For string returns, the entire string is used as `chosen`.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `name` | `str \| None` | `None` | Decision name. Defaults to function name |
| `options` | `list[str] \| None` | `None` | Available choices |

---

## @retrieval Decorator

**Auto-instrumented**
If you use a supported vector database SDK (Pinecone, Chroma, Weaviate, Qdrant, Milvus, FAISS, LanceDB, pgvector, etc.), retrieval operations are captured automatically with zero code. Use `@retrieval` for custom search functions that aren't auto-instrumented.


Auto-record search and retrieval operations:

```python


@waxell.retrieval(source="faiss")
async def search_docs(query: str, top_k: int = 5) -> list[dict]:
    results = await vector_store.search(query, top_k=top_k)
    return [{"id": r.id, "title": r.title, "score": r.score} for r in results]
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `source` | `str` | `""` | Data source name |
| `name` | `str \| None` | `None` | Override name. Defaults to function name |

---

## @reasoning Decorator

Auto-record chain-of-thought steps:

```python


@waxell.reasoning_dec(step="quality_check")
async def assess_quality(answer: str) -> dict:
    return {
        "thought": "Answer is well-grounded in sources",
        "evidence": ["Source A cited", "Source B referenced"],
        "conclusion": "High quality",
    }
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `step` | `str \| None` | `None` | Step name. Defaults to function name |

---

## @retry Decorator

Wrap a function with retry logic and automatic retry recording:

```python


@waxell.retry_dec(max_attempts=3, strategy="retry")
async def call_llm(prompt: str) -> str:
    return await client.chat.completions.create(...)
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `max_attempts` | `int` | `3` | Maximum attempts |
| `strategy` | `str` | `"retry"` | `"retry"`, `"fallback"`, or `"circuit_break"` |
| `fallback_to` | `str` | `""` | Fallback target name |

---

## @step Decorator

Auto-record function calls as execution steps:

```python


@waxell.step_dec(name="preprocess")
async def preprocess(query: str) -> dict:
    return {"cleaned": query.strip().lower()}
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `name` | `str \| None` | `None` | Step name. Defaults to function name |

---

## Top-Level Convenience Functions

These functions delegate to the current context and are no-ops outside a `WaxellContext`:

### Enrichment

| Function | Description |
|----------|-------------|
| `waxell.score(name, value, data_type="numeric", comment="")` | Record a quality score |
| `waxell.tag(key, value)` | Set a searchable tag |
| `waxell.metadata(key, value)` | Set arbitrary metadata |

### Behavior Recording

| Function | Description |
|----------|-------------|
| `waxell.step(name, output=None)` | Record an execution step |
| `waxell.decide(name, chosen, options=None, reasoning="", confidence=None)` | Record a decision |
| `waxell.retrieve(query, documents, source="", scores=None)` | Record a retrieval |
| `waxell.reason(step, thought, evidence=None, conclusion="")` | Record a reasoning step |
| `waxell.retry(attempt, reason, strategy="retry", original_error="", fallback_to="")` | Record a retry event |

### Conversation & Communication

| Function | Description |
|----------|-------------|
| `waxell.user_message(content, metadata=None)` | Record an inbound user message |
| `waxell.agent_response(content, metadata=None)` | Record an outbound agent response |
| `waxell.communication(channel=, recipient=, body=, subject=)` | Record outbound messages (Slack, email, SMS, webhook) |

### Human-in-the-Loop

| Function | Description |
|----------|-------------|
| `waxell.input(prompt)` | Drop-in replacement for `input()` -- records prompt, response, and wait time |
| `waxell.human_turn(prompt=, channel=)` | Context manager for interactive human turns (terminal, Slack, UI) |
| `waxell.human_interaction(prompt=, response=, channel=)` | One-shot recording of a completed interaction |

### Approval Lifecycle

| Function | Description |
|----------|-------------|
| `waxell.approval_request(action_type, approvers=, timeout_minutes=, reason=)` | Record that an approval workflow was initiated |
| `waxell.approval_response(action_type, decision, approver=, elapsed_seconds=)` | Record the approval outcome |

### Utilities

| Function | Description |
|----------|-------------|
| `waxell.flush()` | Async: flush buffered data to control plane (for long-running agents) |
| `waxell.flush_sync()` | Sync version of flush |
| `waxell.get_context()` | Get the current `WaxellContext` or `None` |
| `waxell.diagnose()` | Introspect SDK state: version, active instrumentors, config, tracing status |

## Next Steps

- [Context Manager](./context-manager) -- For more complex instrumentation needs
- [Behavior Tracking](../features/behavior-tracking) -- Deep dive into all behavior types
- [Auto-Instrumentation](./auto-instrumentation) -- Auto-capture for LangChain and other frameworks
- [Policy & Governance](../features/governance) -- Configure and enforce policies
- [Sessions](/docs/observe/features/sessions) -- Group related runs
- [User Tracking](/docs/observe/features/user-tracking) -- Track end-user identity
- [Scoring](/docs/observe/features/scoring) -- Quality metrics


--------------------------------------------------------------------------------
# Behavior Tracking
URL: https://waxell.ai/docs/observe/features/behavior-tracking
Description: Track tool calls, retrievals, decisions, reasoning, retries, and steps with decorators, convenience functions, or manual context methods.
--------------------------------------------------------------------------------

# Behavior Tracking

Waxell Observe tracks agent behaviors at three levels of effort:

1. **Auto-instrumented** -- LLM calls (157 providers), tool-call decisions, and vector DB retrievals captured with zero code
2. **Decorators** -- Wrap functions with `@tool`, `@decision`, `@retrieval`, `@reasoning`, `@retry`, or `@step` for automatic recording
3. **Manual** -- Call `ctx.record_*()` methods or top-level convenience functions for full control

## Overview

| Behavior | Decorator | Convenience Function | Context Method |
|----------|-----------|---------------------|----------------|
| Tool calls | `@waxell.tool` | -- | `ctx.record_tool_call()` |
| Retrievals | `@waxell.retrieval` | `waxell.retrieve()` | `ctx.record_retrieval()` |
| Decisions | `@waxell.decision` | `waxell.decide()` | `ctx.record_decision()` |
| Reasoning | `@waxell.reasoning_dec` | `waxell.reason()` | `ctx.record_reasoning()` |
| Retries | `@waxell.retry_dec` | `waxell.retry()` | `ctx.record_retry()` |
| Steps | `@waxell.step_dec` | `waxell.step()` | `ctx.record_step()` |
| User messages | -- | `waxell.user_message()` | `ctx.record_user_message()` |
| Agent responses | -- | `waxell.agent_response()` | `ctx.record_agent_response()` |
| Communication | -- | `waxell.communication()` | `ctx.record_communication()` |
| Human input | -- | `waxell.input()` | `ctx.input()` |
| Human turns | -- | `waxell.human_turn()` | `ctx.human_turn()` |
| Approvals | -- | `waxell.approval_request()` / `approval_response()` | `ctx.record_approval_request()` / `record_approval_response()` |

---

## Tool Calls

### @tool decorator

Auto-record function calls as tool invocations with zero boilerplate:

```python


@waxell.tool(tool_type="vector_db")
def search_index(index, query_vec, k: int = 5):
    distances, indices = index.search(query_vec, k)
    return {"distances": distances, "indices": indices}

# Every call auto-records: name, inputs, output, duration_ms, status
# No-op when called outside a WaxellContext
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `name` | `str \| None` | `None` | Tool name. Defaults to the function name |
| `tool_type` | `str` | `"function"` | Classification: `"function"`, `"vector_db"`, `"database"`, `"api"` |

### Manual recording

```python
waxell_ctx.record_tool_call(
    name="web_search",
    input={"query": query},
    output={"result_count": len(results)},
    duration_ms=250,
    status="ok",
    tool_type="api",
)
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `name` | `str` | (required) | Tool name |
| `input` | `dict \| str` | `""` | Tool input parameters |
| `output` | `dict \| str` | `""` | Tool output/result |
| `duration_ms` | `int \| None` | `None` | Execution time in milliseconds |
| `status` | `str` | `"ok"` | `"ok"` or `"error"` |
| `tool_type` | `str` | `"function"` | Classification |
| `error` | `str` | `""` | Error message if status is `"error"` |

---

## Retrievals

### Auto-instrumented retrievals

When you use a supported vector database SDK, retrieval operations are captured automatically with zero code. Waxell includes instrumentors for Pinecone, Chroma, Weaviate, Qdrant, Milvus, FAISS, LanceDB, pgvector, MongoDB Atlas Vector Search, Elasticsearch, OpenSearch, Marqo, and many more.

```python
# This Pinecone query automatically records a retrieval span
results = index.query(vector=embedding, top_k=10, namespace="docs")
# Retrieval auto-recorded: source="pinecone", top_k=10, matches_count=10
```

### @retrieval decorator

Auto-record search and retrieval operations:

```python


@waxell.retrieval(source="pinecone")
async def search_documents(query: str, top_k: int = 5) -> list[dict]:
    results = await vector_store.search(query, top_k=top_k)
    return [{"id": r.id, "title": r.title, "score": r.score} for r in results]

# Auto-records: query (first string arg), documents (return value),
# scores (from doc["score"] fields), source, duration_ms
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `source` | `str` | `""` | Data source name (e.g., `"faiss"`, `"pinecone"`) |
| `name` | `str \| None` | `None` | Override name. Defaults to function name |

### Convenience function

```python
waxell.retrieve(
    query="AI safety papers",
    documents=[{"id": 1, "title": "Safety Guidelines", "score": 0.95}],
    source="faiss",
    scores=[0.95],
)
```

### Manual recording

```python
waxell_ctx.record_retrieval(
    query=query,
    documents=[{"id": d.id, "title": d.title, "score": d.score} for d in docs],
    source="pinecone",
    duration_ms=120,
    top_k=5,
)
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `query` | `str` | (required) | The retrieval query string |
| `documents` | `list[dict]` | (required) | Retrieved documents |
| `source` | `str` | `""` | Data source name |
| `scores` | `list[float] \| None` | `None` | Relevance scores for each document |
| `duration_ms` | `int \| None` | `None` | Retrieval time in milliseconds |
| `top_k` | `int \| None` | `None` | Number of documents requested |

---

## Decisions

Waxell provides three layers of decision recording, from zero-effort to manual:

### Auto-instrumented decisions

When an LLM response contains `tool_calls` (OpenAI/Groq/Mistral) or `tool_use` blocks (Anthropic), the auto-instrumentor records the model's tool selection as a decision. No code needed.

```python
# This OpenAI call with tools automatically records a decision span
response = await client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Search for AI safety docs"}],
    tools=[{"type": "function", "function": {"name": "search", ...}}],
)
# Decision auto-recorded: name="tool_call:search", instrumentation_source="auto"
```

### @decision decorator

Wrap any classification/routing function to auto-record its return value as a decision:

```python


@waxell.decision(name="classify_query", options=["factual", "analytical", "creative"])
async def classify_query(query: str) -> dict:
    response = await client.chat.completions.create(...)
    return {"chosen": "analytical", "reasoning": "Complex multi-doc query"}

# Returns the dict AND auto-records the decision
# instrumentation_source="decorator"
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `name` | `str \| None` | `None` | Decision name. Defaults to function name |
| `options` | `list[str] \| None` | `None` | Available choices |

**Return value handling:** If the function returns a `dict`, the SDK extracts `chosen`, `reasoning`, and `confidence` fields. If it returns a `str`, the entire string is used as `chosen`.

### waxell.decide() convenience function

For inline decisions that don't warrant a separate function:

```python
waxell.decide(
    "retrieval_strategy",
    chosen="semantic_search",
    options=["semantic_search", "keyword_search", "hybrid"],
    reasoning="Analytical query benefits from semantic similarity",
    confidence=0.88,
)
# instrumentation_source="manual"
```

### ctx.record_decision() -- full control

```python
waxell_ctx.record_decision(
    name="output_format",
    options=["brief", "detailed", "bullet_points"],
    chosen="detailed",
    reasoning="User query is analytical",
    confidence=0.85,
    metadata={"user_preference": "verbose"},
)
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `name` | `str` | (required) | Decision name |
| `options` | `list[str]` | (required) | Available choices |
| `chosen` | `str` | (required) | The selected option |
| `reasoning` | `str` | `""` | Why this option was chosen |
| `confidence` | `float \| None` | `None` | Confidence score (0.0-1.0) |
| `metadata` | `dict \| None` | `None` | Additional context |

### Instrumentation source tracking

Each decision span includes an `instrumentation_source` attribute indicating how it was captured:

| Source | Value | Meaning |
|--------|-------|---------|
| Auto-instrumentor | `"auto"` | Detected from LLM `tool_calls`/`tool_use` response |
| @decision decorator | `"decorator"` | Captured by the `@decision` wrapper |
| `waxell.decide()` / `ctx.record_decision()` | `"manual"` | Explicitly recorded by user code |

---

## Reasoning

### @reasoning decorator

Auto-record chain-of-thought steps from a function's return value:

```python


@waxell.reasoning_dec(step="quality_check")
async def assess_quality(answer: str, sources: list) -> dict:
    return {
        "thought": "Answer covers all source material with proper citations",
        "evidence": [f"Source: {s['title']}" for s in sources],
        "conclusion": "High quality, ready to present",
    }
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `step` | `str \| None` | `None` | Reasoning step name. Defaults to function name |

**Return value handling:** If the function returns a `dict`, extracts `thought`, `evidence`, `conclusion`. If it returns a `str`, uses the string as `thought`.

### Convenience function

```python
waxell.reason(
    step="evaluate_sources",
    thought="Source A is more recent but Source B has higher authority",
    evidence=["Source A: published 2024", "Source B: cited 500 times"],
    conclusion="Use Source B as primary",
)
```

### Manual recording

```python
waxell_ctx.record_reasoning(
    step="evaluate_sources",
    thought="Source A is more recent but Source B has higher authority",
    evidence=["Source A: published 2024", "Source B: cited 500 times"],
    conclusion="Use Source B as primary, Source A as supplement",
)
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `step` | `str` | (required) | Reasoning step name |
| `thought` | `str` | (required) | The reasoning text/thought process |
| `evidence` | `list[str] \| None` | `None` | Supporting evidence or references |
| `conclusion` | `str` | `""` | Conclusion reached at this step |

---

## Retries

### @retry decorator

Wrap a function with retry logic AND automatic retry recording:

```python


@waxell.retry_dec(max_attempts=3, strategy="retry")
async def call_llm(prompt: str) -> str:
    response = await client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
    )
    return response.choices[0].message.content

# On failure, retries up to 3 times, recording each attempt as a retry span.
# After exhausting attempts, re-raises the last exception.
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `max_attempts` | `int` | `3` | Maximum number of attempts (including first) |
| `strategy` | `str` | `"retry"` | `"retry"`, `"fallback"`, or `"circuit_break"` |
| `fallback_to` | `str` | `""` | Name of fallback target |

### Convenience function

```python
waxell.retry(
    attempt=1,
    reason="OpenAI rate limited",
    strategy="fallback",
    original_error="429 Too Many Requests",
    fallback_to="claude-sonnet-4",
)
```

### Manual recording

```python
waxell_ctx.record_retry(
    attempt=1,
    reason="OpenAI rate limited",
    strategy="fallback",
    original_error="429 Too Many Requests",
    fallback_to="claude-sonnet-4",
    max_attempts=3,
)
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `attempt` | `int` | (required) | Current attempt number (1-based) |
| `reason` | `str` | (required) | Why a retry/fallback occurred |
| `strategy` | `str` | `"retry"` | `"retry"`, `"fallback"`, or `"circuit_break"` |
| `original_error` | `str` | `""` | The error that triggered the retry |
| `fallback_to` | `str` | `""` | Name of fallback target |
| `max_attempts` | `int \| None` | `None` | Maximum attempts configured |

---

## Steps

### @step decorator

Auto-record function calls as execution steps:

```python


@waxell.step_dec(name="preprocess")
async def preprocess_query(query: str) -> dict:
    cleaned = query.strip().lower()
    return {"original": query, "cleaned": cleaned}

# Auto-records: step(name="preprocess", output={"original": ..., "cleaned": ...})
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `name` | `str \| None` | `None` | Step name. Defaults to function name |

### Convenience function

```python
waxell.step("preprocess", output={"cleaned": query.strip()})
```

### Manual recording

```python
waxell_ctx.record_step("retrieve", output={"doc_count": len(docs)})
```

---

## Conversation Tracking

Record user/agent messages for interactive agents (chat, REPL, conversational UIs):

### Convenience functions

```python
waxell.user_message("What's the weather in Paris?")
# ... agent processes ...
waxell.agent_response("It's currently 22°C and sunny in Paris.")
```

### Manual recording

```python
waxell_ctx.record_user_message("What's the weather?", metadata={"channel": "web"})
waxell_ctx.record_agent_response("It's sunny in Paris.", metadata={"model": "gpt-4o"})
```

These create IO spans that appear in the trace timeline alongside LLM calls and tool invocations. See [Conversation Tracking](./conversation-tracking) for full details.

---

## Communication

Record outbound messages sent via external channels for communication governance:

### Convenience function

```python
waxell.communication(
    channel="slack",
    recipient="#ops-alerts",
    body="Deployment completed successfully",
    subject="Deploy Notification",
)
```

### Manual recording

```python
waxell_ctx.record_communication(
    channel="email",
    recipient="user@example.com",
    body="Your report is ready",
    subject="Report Complete",
    metadata={"template": "report_ready_v2"},
)
```

Supported channels: `"slack"`, `"email"`, `"sms"`, `"webhook"`, or any custom string.

---

## Human-in-the-Loop

Record interactions where a human provides input during agent execution:

### Drop-in input replacement

```python
answer = waxell.input("Approve deployment? (y/n): ")
# Records prompt, response, and wait time as a human_turn span
```

### Context manager for non-terminal channels

```python
with waxell.human_turn(prompt="Approve?", channel="slack", action="deployment_approval") as turn:
    response = await wait_for_slack_reaction()
    turn.set_response(response)
# Records the full interaction with timing
```

### One-shot recording

```python
waxell.human_interaction(
    prompt="Pick a target environment",
    response="staging",
    channel="slack",
    elapsed_seconds=12.5,
)
```

---

## Approval Lifecycle

Record approval workflows triggered by policy blocks:

```python
# Record that an approval request was sent
waxell.approval_request(
    action_type="high_cost_query",
    approvers=["ops-team@company.com"],
    timeout_minutes=10,
    reason="Estimated cost exceeds $5",
)

# Record the approval decision
waxell.approval_response(
    action_type="high_cost_query",
    decision="approved",
    approver="ops-lead@company.com",
    elapsed_seconds=45.0,
)
```

---

## Advanced Governance Recording

These `ctx.record_*()` methods are available for detailed governance telemetry. They are typically used with `WaxellContext` for fine-grained control:

```python
# Code execution tracking
ctx.record_code_execution(language="python", code="df.head()", error="")

# Database access tracking
ctx.record_data_access(table="users", operation="SELECT", record_count=150, columns=["name", "email"])

# Network request tracking
ctx.record_network_request(url="https://api.example.com/data")

# Scope/impact tracking
ctx.record_scope_impact(records_modified=50, files_changed=2, transaction_total=150.0)

# Grounding quality tracking
ctx.record_grounding(query="AI safety", documents=[...], score_distribution={"high": 3, "low": 1})

# Delegation tracking (multi-agent)
ctx.record_delegation(delegated_to="research-agent", task="Find recent papers", complexity="high")

# Memory write tracking (for memory governance policies)
ctx.set_memory_state(memory_items=[{"type": "preference", "key": "theme"}], memory_item_count=1)
ctx.record_memory_write(memory_type="preference", content="theme=dark_mode")
```

For agents deployed to third-party platforms (IBM watsonx.ai, SageMaker, Vertex, etc.) that need to populate the controlplane Memory tab over HTTPS, see the [memory write REST endpoints](../api/endpoints#memory-write-endpoints) and the [waxell-agent-examples repo](https://github.com/waxell-ai/waxell-agent-examples) for the full pattern.

---

## How It Works

Behavior tracking methods buffer data in two ways:

1. **Steps** -- Each call creates a step record (e.g., `tool:web_search`, `retrieval:pinecone`, `decision:route_to_agent`) that appears in the run's step list.
2. **Spans** -- Each call creates a behavior span with structured input/output data, flushed to the server via `POST /runs/{run_id}/spans/` on context exit.

Both are sent automatically when the `WaxellContext` exits -- you don't need to flush manually.

## Full Example

This example uses decorators for zero-boilerplate recording:

```python


waxell.init(api_key="wax_sk_...", api_url="https://acme.waxell.dev")


@waxell.retrieval(source="pinecone")
async def search_docs(query: str, top_k: int = 10) -> list[dict]:
    return await vector_store.search(query, top_k=top_k)


@waxell.decision(name="approach", options=["summarize", "compare", "deep_dive"])
async def choose_approach(query: str) -> dict:
    return {"chosen": "deep_dive", "reasoning": "Query asks for detailed analysis"}


@waxell.tool(tool_type="api")
async def run_analysis(docs: list) -> dict:
    return await analysis_service.analyze(docs)


@waxell.reasoning_dec(step="quality_check")
async def check_quality(result: dict) -> dict:
    return {"thought": "Analysis is thorough", "conclusion": "Ready to present"}


@waxell.observe(agent_name="research-pipeline")
async def run_pipeline(query: str):
    docs = await search_docs(query, top_k=10)
    approach = await choose_approach(query)
    analysis = await run_analysis(docs)
    quality = await check_quality(analysis)

    waxell.score("quality", 0.92)
    return {"result": analysis, "approach": approach["chosen"]}
```

## Next Steps

- [Decorator Pattern](../integrations/decorator) -- `@observe`, `@tool`, `@decision`, and more
- [Context Manager](../integrations/context-manager) -- Advanced lifecycle control with `WaxellContext`
- [Conversation Tracking](./conversation-tracking) -- Auto-captured conversation data
- [Python SDK Reference](../api/python-sdk) -- Complete API reference
- [FAQ](../troubleshooting/faq) -- Common questions answered
- [Common Mistakes](../troubleshooting/common-mistakes) -- Avoid these gotchas


--------------------------------------------------------------------------------
# Advanced: Context Manager
URL: https://waxell.ai/docs/observe/integrations/context-manager
Description: Use WaxellContext for fine-grained control over observability and governance in complex agent workflows.
--------------------------------------------------------------------------------

# Advanced: Context Manager

**Prefer Decorators**
For most agents, the [decorator pattern](./decorator) is simpler and covers 90% of use cases. Use `WaxellContext` only when you need explicit lifecycle control -- batch loops, multi-step orchestration across functions, conditional context creation, or multiple runs in a single function.

See [Decorator vs Context Manager](#when-to-use-context-manager-vs-decorator) for a decision guide.


`WaxellContext` is a context manager that gives you explicit control over run lifecycle, LLM call recording, step tracking, and mid-execution policy checks.

It works as both `async with` (for async code) and plain `with` (for sync code).

## Async Usage

```python
from waxell_observe import WaxellContext

async with WaxellContext(agent_name="research-agent") as ctx:
    result = await run_research(query)

    ctx.record_llm_call(model="gpt-4o", tokens_in=300, tokens_out=150)
    ctx.record_step("research", output={"sources": 5})
    ctx.set_result({"answer": result})
```

## Sync Usage

```python
from waxell_observe import WaxellContext

with WaxellContext(agent_name="batch-processor") as ctx:
    result = process_data(input_data)

    ctx.record_llm_call(model="gpt-4o", tokens_in=300, tokens_out=150)
    ctx.record_step("process", output={"items": 42})
    ctx.set_result({"output": result})
```

The sync path uses native `__enter__` / `__exit__` with synchronous HTTP calls — ContextVars are set in the calling thread, so auto-instrumentation works correctly.

**When to use sync vs async**
Use `with` (sync) for batch processing scripts, CLI tools, ETL pipelines, and any code that doesn't use `async`/`await`. Use `async with` for async web servers, async agent frameworks, and code that's already async.


## Convenience Aliases

`waxell.context` and `waxell.session` are re-exports for the cleanest one-liner usage:

```python


# `waxell.context(...)` is an alias for `WaxellContext(...)`
with waxell.context(agent_name="my-agent") as ctx:
    ...

# `waxell.session(...)` sets session_id / user_id for any run opened inside the
# block — useful for thread-less frameworks (pydantic-ai, smolagents, raw SDK
# loops) where the framework has no native thread concept.
with waxell.session(session_id=thread_id, user_id=email):
    agent.run_sync(user_message)  # auto-instrumented run inherits session/user
```

You can also generate a random session id with `waxell.generate_session_id()` (returns `sess_<16 hex chars>`).

## Lifecycle

On entering the context:
1. Policies are checked (if `enforce_policy=True`)
2. A new execution run is started on the control plane

On exiting the context:
1. Buffered LLM calls are flushed to the control plane
2. Buffered steps are flushed to the control plane
3. The run is completed with success or error status

## Enhanced Context Options

### Session and User Tracking

Group related runs into sessions and track end-user identity:

```python
with WaxellContext(
    agent_name="my-chatbot",
    session_id="session-abc-123",  # Group related runs
    user_id="user-456",            # Track end-user
) as ctx:
    # Your LLM calls here
    response = call_llm(prompt)
```

### Tags and Metadata

Add structured metadata to runs for filtering and analysis:

```python
with WaxellContext(agent_name="my-agent") as ctx:
    ctx.set_tag("environment", "production")
    ctx.set_tag("pipeline", "rag-v2")
    ctx.set_metadata("retrieval_count", 5)
    ctx.set_metadata("model_version", "gpt-4-turbo")

    # Your LLM calls here
```

### Recording Scores

Capture quality metrics and user feedback:

```python
with WaxellContext(agent_name="my-agent") as ctx:
    response = call_llm(prompt)

    # Numeric score (0-1 range)
    ctx.record_score(
        name="relevance",
        value=0.92,
        data_type="numeric",
        comment="Highly relevant to the query"
    )

    # Boolean score
    ctx.record_score(
        name="contains_hallucination",
        value=False,
        data_type="boolean"
    )

    # Categorical score
    ctx.record_score(
        name="tone",
        value="professional",
        data_type="categorical"
    )
```

### Recording Steps

Track sub-operations within a run:

```python
with WaxellContext(agent_name="rag-pipeline") as ctx:
    # Step 1: Retrieval
    docs = retrieve_documents(query)
    ctx.record_step("retrieval", output={"doc_count": len(docs)})

    # Step 2: Generation
    response = generate_response(query, docs)
    ctx.record_step("generation", output={"response_length": len(response)})
```

## Constructor Parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `agent_name` | `str` | (required) | Name for this agent in the control plane |
| `workflow_name` | `str` | `"default"` | Workflow name for grouping runs |
| `inputs` | `dict \| None` | `None` | Input data to record with the run |
| `metadata` | `dict \| None` | `None` | Arbitrary metadata to attach to the run |
| `client` | `WaxellObserveClient \| None` | `None` | Pre-configured client. If `None`, creates a new one using current configuration |
| `enforce_policy` | `bool` | `True` | Check policies on context entry |
| `session_id` | `str` | `""` | Session ID for grouping related runs |
| `user_id` | `str` | `""` | End-user ID for per-user tracking and analytics |
| `user_group` | `str` | `""` | User group for authorization policies (e.g., `"enterprise"`, `"free"`) |
| `end_user_id` | `str` | `""` | Sub-user identity for end-user budget / rate-limit / suspension policy handlers. Wired into `metadata["tenant_sub_user_id"]` |
| `interaction_mode` | `str \| None` | `None` | One of `"auto"`, `"interactive"`, `"autonomous"`. `None`/`"auto"` is inferred at run start: a `session_id` (chat turn / multi-turn thread) means `interactive`; no session means `autonomous` |
| `mid_execution_governance` | `bool \| None` | `None` | Flush data and check governance after each governance-bearing `record_*` call. `None` resolves to `True` unless `WAXELL_DISABLE_MID_EXECUTION_GOVERNANCE=1` is set |
| `auto_grounding` | `bool` | `False` | Auto-bridge retrieval scores to grounding governance |
| `on_policy_block` | `Callable \| None` | `None` | Callback for policy blocks. Receives `PolicyViolationError`, returns `ApprovalDecision`. Built-in: `prompt_approval`, `auto_approve`, `auto_deny` |

## Recording Methods

### record_llm_call

Record an LLM API call. All parameters are keyword-only.

```python
ctx.record_llm_call(
    model="gpt-4o",
    tokens_in=500,
    tokens_out=200,
    cost=0.0,              # Optional: auto-estimated if 0.0
    task="summarize",      # Optional: label for this call
    prompt_preview="...",  # Optional: first N chars of prompt
    response_preview="...",  # Optional: first N chars of response
    duration_ms=350,       # Optional: call duration in milliseconds
    provider="openai",     # Optional: inferred from model name if empty
)
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `model` | `str` | (required) | Model name (e.g., `"gpt-4o"`, `"claude-sonnet-4"`) |
| `tokens_in` | `int` | (required) | Input/prompt token count |
| `tokens_out` | `int` | (required) | Output/completion token count |
| `cost` | `float` | `0.0` | Cost in USD. If `0.0`, automatically estimated using built-in model pricing |
| `task` | `str` | `""` | A label describing this LLM call's purpose |
| `prompt_preview` | `str` | `""` | Preview of the prompt text |
| `response_preview` | `str` | `""` | Preview of the response text |
| `duration_ms` | `int \| None` | `None` | LLM call duration in milliseconds |
| `provider` | `str` | `""` | Provider name (e.g., `"openai"`, `"anthropic"`). If empty, inferred from model name |

LLM calls are buffered in memory and flushed to the control plane when the context exits.

### record_step

Record a named execution step.

```python
ctx.record_step("extract_entities", output={"count": 12})
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `step_name` | `str` | (required) | Name identifying this step |
| `output` | `dict \| None` | `None` | Optional output data for the step |

Steps are automatically numbered in order of recording. Like LLM calls, they are buffered and flushed on context exit.

### set_result

Set the final result for the run.

```python
ctx.set_result({"answer": "The capital of France is Paris.", "confidence": 0.95})
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `result` | `dict` | (required) | Result data to include when the run is completed |

Call this before the context exits. If not called, the run completes with an empty result.

### check_policy / check_policy_sync

Perform a mid-execution policy check. This is useful for long-running agents that should re-validate policies between steps.

```python
# Async
policy = await ctx.check_policy()

# Sync
policy = ctx.check_policy_sync()
```

```python
if policy.blocked:
    print(f"Blocked: {policy.reason}")
    # Handle the block (e.g., stop processing)
elif policy.action == "warn":
    print(f"Warning: {policy.reason}")
    # Continue but log the warning
```

Returns a `PolicyCheckResult` with:
- `action` -- one of `"allow"`, `"block"`, `"warn"`, `"throttle"`
- `reason` -- human-readable explanation
- `metadata` -- additional policy data
- `allowed` -- property, `True` if action is `"allow"` or `"warn"`
- `blocked` -- property, `True` if action is `"block"` or `"throttle"`

### record_score

Record a quality score or feedback metric for the current run.

```python
ctx.record_score(
    name="relevance",
    value=0.92,
    data_type="numeric",
    comment="Highly relevant to the query",
)
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `name` | `str` | (required) | Score name (e.g., `"relevance"`, `"accuracy"`, `"thumbs_up"`) |
| `value` | `float \| str \| bool` | (required) | Score value. Type depends on `data_type` |
| `data_type` | `str` | `"numeric"` | One of `"numeric"`, `"categorical"`, `"boolean"` |
| `comment` | `str` | `""` | Optional free-text comment |

Scores are buffered and flushed to the control plane when the context exits.

### set_tag

Set a searchable tag on the current run. Tags become OTel span attributes and are queryable in Grafana TraceQL.

```python
ctx.set_tag("environment", "production")
ctx.set_tag("pipeline", "rag-v2")
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `key` | `str` | (required) | Tag name (alphanumeric, underscores, hyphens) |
| `value` | `str` | (required) | Tag value (string only) |

### set_metadata

Set arbitrary metadata on the current run. Complex values are JSON-serialized.

```python
ctx.set_metadata("retrieval_count", 5)
ctx.set_metadata("model_version", "gpt-4-turbo")
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `key` | `str` | (required) | Metadata key |
| `value` | `Any` | (required) | Any JSON-serializable value |

### Behavior Tracking

Track agent behaviors beyond LLM calls and steps. These methods buffer data as spans and flush on context exit.

#### record_tool_call

Record a tool or function call.

```python
ctx.record_tool_call(
    name="web_search",
    input={"query": "latest news"},
    output={"results": [...]},
    duration_ms=250,
    status="ok",
    tool_type="api",
)
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `name` | `str` | (required) | Tool name (e.g., `"web_search"`, `"database_query"`) |
| `input` | `dict \| str` | `""` | Tool input parameters |
| `output` | `dict \| str` | `""` | Tool output/result |
| `duration_ms` | `int \| None` | `None` | Execution time in milliseconds |
| `status` | `str` | `"ok"` | `"ok"` or `"error"` |
| `tool_type` | `str` | `"function"` | Classification: `"function"`, `"api"`, `"database"`, `"retriever"` |
| `error` | `str` | `""` | Error message if status is `"error"` |

#### record_retrieval

Record a RAG document retrieval.

```python
ctx.record_retrieval(
    query="How does the billing system work?",
    documents=[{"id": "doc1", "title": "Billing FAQ", "score": 0.92}],
    source="pinecone",
    duration_ms=120,
    top_k=5,
    scores=[0.92, 0.87, 0.81],
)
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `query` | `str` | (required) | The retrieval query string |
| `documents` | `list[dict]` | (required) | Retrieved documents (e.g., `[{id, title, score, snippet}]`) |
| `source` | `str` | `""` | Data source name (e.g., `"pinecone"`, `"elasticsearch"`) |
| `duration_ms` | `int \| None` | `None` | Retrieval time in milliseconds |
| `top_k` | `int \| None` | `None` | Number of documents requested |
| `scores` | `list[float] \| None` | `None` | Relevance scores for each retrieved document |

#### record_decision

Record a decision or routing point.

```python
ctx.record_decision(
    name="route_to_agent",
    options=["billing", "technical", "general"],
    chosen="billing",
    reasoning="User mentioned invoice and payment",
    confidence=0.95,
)
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `name` | `str` | (required) | Decision name (e.g., `"route_to_agent"`, `"select_model"`) |
| `options` | `list[str]` | (required) | Available choices |
| `chosen` | `str` | (required) | The selected option |
| `reasoning` | `str` | `""` | Why this option was chosen |
| `confidence` | `float \| None` | `None` | Confidence score (0.0-1.0) |
| `metadata` | `dict \| None` | `None` | Additional context |
| `instrumentation_type` | `str` | `"manual"` | How this decision was captured: `"manual"`, `"decorator"`, or `"auto"` |

#### record_reasoning

Record a reasoning or chain-of-thought step.

```python
ctx.record_reasoning(
    step="evaluate_sources",
    thought="Source A is more recent but Source B has higher authority",
    evidence=["Source A: 2024", "Source B: cited 500 times"],
    conclusion="Use Source B as primary, Source A as supplement",
)
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `step` | `str` | (required) | Reasoning step name |
| `thought` | `str` | (required) | The reasoning text/thought process |
| `evidence` | `list[str] \| None` | `None` | Supporting evidence or references |
| `conclusion` | `str` | `""` | Conclusion reached at this step |

#### record_retry

Record a retry or fallback event.

```python
ctx.record_retry(
    attempt=2,
    reason="Rate limited by OpenAI",
    strategy="fallback",
    original_error="429 Too Many Requests",
    fallback_to="claude-sonnet-4",
    max_attempts=3,
)
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `attempt` | `int` | (required) | Current attempt number (1-based) |
| `reason` | `str` | (required) | Why a retry/fallback occurred |
| `strategy` | `str` | `"retry"` | `"retry"`, `"fallback"`, or `"circuit_break"` |
| `original_error` | `str` | `""` | The error that triggered the retry |
| `fallback_to` | `str` | `""` | Name of fallback target (model, agent, tool) |
| `max_attempts` | `int \| None` | `None` | Maximum attempts configured |

#### record_policy_check

Record a policy evaluation result as a governance span.

```python
ctx.record_policy_check(
    policy_name="budget-limit",
    action="warn",
    category="budget",
    reason="Approaching 80% of daily budget",
    phase="mid_execution",
)
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `policy_name` | `str` | (required) | Name of the policy evaluated |
| `action` | `str` | (required) | Evaluation result: `"allow"`, `"warn"`, `"block"`, etc. |
| `category` | `str` | `""` | Policy category (e.g., `"budget"`, `"rate-limit"`) |
| `reason` | `str` | `""` | Reason for the action (empty for allow) |
| `duration_ms` | `float` | `0` | Evaluation time in milliseconds |
| `phase` | `str` | `"pre_execution"` | `"pre_execution"`, `"mid_execution"`, or `"post_execution"` |
| `priority` | `int` | `100` | Policy priority (lower = evaluated first) |

### Conversation & Human Interaction

For interactive agents (chat, REPL, ticketing) — make user input and agent
output visible in the trace alongside LLM calls. These also bump conversation
counters, so context-management / recursion-bound policies fire mid-run.

#### record_user_message

Record an inbound user message. Always lands at `position=0`.

```python
ctx.record_user_message("What's the weather?")
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `content` | `str` | `""` | The user's message text |
| `message` | `str` | `""` | Alias for `content`; `content` wins if both set |
| `metadata` | `dict \| None` | `None` | Optional extra context (channel, user_id, ...) |

#### record_agent_response

Record an outbound agent response (distinct from the raw LLM call payload).

```python
ctx.record_agent_response("It's sunny in Paris today.")
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `content` | `str` | (required) | The agent's response text shown to the user |
| `metadata` | `dict \| None` | `None` | Optional extra context (citations, confidence, ...) |

#### record_human_interaction

Record a completed human-in-the-loop interaction as a single IO span. Use this
when you already have the prompt, response, and timing — for the streaming /
context-manager variants use `ctx.input(...)` or `ctx.human_turn(...)` (below).

```python
ctx.record_human_interaction(
    prompt="Approve refund?",
    response="yes",
    channel="slack",
    action="approval",
    elapsed_ms=12500,
)
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `prompt` | `str` | `""` | What was shown to the human |
| `response` | `str` | `""` | What the human replied |
| `channel` | `str` | `"terminal"` | Where it happened (`"terminal"`, `"slack"`, `"ui"`, `"webhook"`, ...) |
| `action` | `str` | `""` | Interaction kind (`"confirmation"`, `"input"`, `"approval"`, ...) |
| `elapsed_ms` | `float \| int \| None` | `None` | How long the human took to respond |
| `metadata` | `dict \| None` | `None` | Arbitrary extra context |

#### ctx.input

Drop-in for the built-in `input()` — auto-captures prompt + response + wait time
as a `human_turn` span. Strips ANSI escape codes before recording.

```python
answer = ctx.input("Approve? (y/n): ")
```

#### ctx.human_turn

Context manager for non-terminal channels (Slack, webhooks, UI dialogs) where
the wait shape varies:

```python
with ctx.human_turn(prompt="Review PR", channel="github") as turn:
    result = wait_for_webhook()
    turn.set_response(result)
```

#### record_communication

Record an outbound communication (Slack, email, SMS, etc.) for communication
governance policies (allowed channels, message limits, disclaimer rules).

```python
ctx.record_communication(
    channel="slack",
    recipient="#general",
    body="Deploy completed",
)
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `channel` | `str` | (required) | `"slack"`, `"email"`, `"sms"`, ... |
| `recipient` | `str` | `""` | Target (`"#general"`, `"user@acme.com"`, ...) |
| `body` | `str` | `""` | Message body |
| `subject` | `str` | `""` | Subject line (email, etc.) |
| `metadata` | `dict \| None` | `None` | Extra context |

### HITL Pause

#### ctx.pause

Mark a human-in-the-loop **pause** — wrap any blocking wait so the trace
renders a ⏸ pause marker with wait duration + reason. Framework-agnostic.

```python
with ctx.pause(reason="awaiting_approval"):
    decision = wait_for_human()   # blocks
```

`reason` mirrors runtime PausedReason values: `awaiting_user`,
`awaiting_approval`, `awaiting_signal`, `awaiting_timer`, `awaiting_child`.

#### ctx.mark_resumed

Mark that this run resumed after a cross-process pause (e.g. a LangGraph
`interrupt()` answered by a separate resume invocation). Emits a `post_resume`
phase marker.

```python
ctx.mark_resumed(from_run_id="run_abc123")
```

### Memory

Record agent memory reads and writes — emits `kind=memory` spans and feeds
memory governance + the run's Memory tab + memory-state replay.

#### remember_episodic

Write an episodic memory (named-slot KV).

```python
ctx.remember_episodic(slot_name="deal_findings", data={"company": "Acme"})
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `slot_name` | `str` | (required) | Named slot (e.g. `"deal_findings"`) |
| `data` | `Any` | (required) | Value to store (dict / list / scalar) |
| `scope_key` | `str \| None` | `None` | Defaults to `agent:session` |
| `ttl_seconds` | `int` | `86400` | TTL on the Memory tab write |
| `max_items` | `int \| None` | `None` | Cap on items in the slot |
| `memory_type` | `str \| None` | `None` | Optional sub-classification |

#### remember_semantic

Write a semantic fact (free-text, embedded for similarity search).

```python
ctx.remember_semantic(
    slot_name="customer_preferences",
    content="Acme prefers monthly invoicing in EUR.",
)
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `slot_name` | `str` | (required) | Named slot |
| `content` | `str` | (required) | Fact text |
| `scope_key` | `str \| None` | `None` | Defaults to `agent:session` |
| `importance` | `float` | `0.7` | 0–1 weight |
| `tags` | `list \| None` | `None` | Searchable tags |
| `fact_key` | `str \| None` | `None` | Stable de-dup key |
| `source_tool` | `str` | `""` | Where the fact came from |
| `content_embedding` | `list \| None` | `None` | Pre-computed embedding (falls back to OpenAI `text-embedding-3-small` if `OPENAI_API_KEY` is set) |

#### record_memory_recall

Record a memory *read* (search/lookup) — the read side of the memory signal,
analogous to grounding. Surfaces hit-rate + relevance per turn.

```python
ctx.record_memory_recall(
    query="What did Acme say about pricing?",
    tier="episodic",
    store="mem0",
    results_count=3,
    top_score=0.84,
    results=hits,
)
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `query` | `str` | `""` | Recall query |
| `tier` | `str` | `"episodic"` | `"episodic"` or `"semantic"` |
| `store` | `str` | `""` | Memory store name (`"mem0"`, `"zep"`, `"letta"`, ...) |
| `results_count` | `int` | `0` | `0` ⇒ miss; `>0` ⇒ hit |
| `top_score` | `float \| None` | `None` | Best match relevance (0–1) |
| `slot_name` | `str` | `""` | Named slot if applicable |
| `results` | `list \| None` | `None` | Recalled items — bounded-captured into the memory snapshot for replay |

### Prompt Lineage

#### record_prompt_use

Stamp the run with the registry prompt name + version it used. This is the
lineage link that ties eval + replay to the exact prompt version. `get_prompt`
calls this automatically; use it directly when you fetch a prompt some other way.

```python
ctx.record_prompt_use(name="customer_intake", version=7, label="prod")
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `name` | `str` | (required) | Registry prompt name |
| `version` | `int` | `0` | Pinned version (`0` means latest at fetch time) |
| `label` | `str` | `""` | Label resolved against (`"prod"`, `"staging"`) |
| `content_hash` | `str` | `""` | Content hash of the rendered prompt |

### Approval Lifecycle

#### record_approval_request

Record that an approval workflow was initiated after a policy block. Call
after catching `PolicyViolationError` when the policy metadata indicates
approval is required.

```python
ctx.record_approval_request(
    action_type="delete",
    approvers=["security@acme.com"],
    timeout_minutes=30,
    reason="Mass deletion above 100 records",
)
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `action_type` | `str` | (required) | Action awaiting approval (`"delete"`, `"refund"`, ...) |
| `approvers` | `list \| None` | `None` | Emails / group names |
| `timeout_minutes` | `float \| None` | `None` | Approval window |
| `reason` | `str` | `""` | Why approval is needed |
| `metadata` | `dict \| None` | `None` | Extra context |

#### record_approval_response

Record the outcome of an approval request. Call after the human decides or
the timeout expires.

```python
ctx.record_approval_response(
    action_type="delete",
    decision="approved",
    approver="security@acme.com",
    elapsed_seconds=420,
)
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `action_type` | `str` | (required) | Action that was awaiting approval |
| `decision` | `str` | (required) | `"approved"`, `"denied"`, or `"timeout"` |
| `approver` | `str` | `""` | Who decided |
| `elapsed_seconds` | `float \| None` | `None` | Time from block to decision |
| `metadata` | `dict \| None` | `None` | Extra context |

### Governance Signals

Buffer state the controlplane's policy handlers read from `conversation_state`.
All are additive — call as the agent works.

#### record_data_access

```python
ctx.record_data_access(source="postgres", operation="read", records=42)
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `source` | `str` | (required) | Data source (`"postgres"`, `"s3"`, `"redis"`, ...) |
| `operation` | `str` | `"read"` | `"read"` or `"write"` |
| `records` | `int` | `0` | Number of records accessed |

#### record_network_request

```python
ctx.record_network_request(url="https://api.stripe.com/v1/charges")
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `url` | `str` | (required) | URL or domain accessed |

#### record_scope_impact

Running totals of the run's blast radius. Each call increments.

```python
ctx.record_scope_impact(records_modified=10, transaction_total=499.99)
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `records_modified` | `int` | `0` | Records updated |
| `records_deleted` | `int` | `0` | Records deleted |
| `files_changed` | `int` | `0` | Files written |
| `transaction_total` | `float` | `0.0` | Dollar value of transactions |
| `api_writes` | `int` | `0` | External API write operations |

### Incremental Flush

For long-running contexts (REPLs, chat sessions, batch loops) where you want
buffered data visible in the UI — and governance evaluated — *before* the
context exits.

```python
# async
await ctx.flush()

# sync
ctx.flush_sync()
```

Each call sends only data buffered since the last flush. Safe to call any
number of times. Both raise `PolicyViolationError` if a governance-checked
send (LLM calls, steps) trips a `block` policy mid-run.

## Properties

| Property | Type | Description |
|----------|------|-------------|
| `run_id` | `str` | The run ID from the control plane, or `""` if the run has not started |

## Error Handling

If an exception occurs inside the context, the run is automatically completed with `status="error"` and the error message. The exception is **not** suppressed -- it propagates normally:

```python
# Async
try:
    async with WaxellContext(agent_name="my-agent") as ctx:
        raise ValueError("Something went wrong")
except ValueError:
    pass  # Run was completed with status="error"

# Sync
try:
    with WaxellContext(agent_name="my-agent") as ctx:
        raise ValueError("Something went wrong")
except ValueError:
    pass  # Run was completed with status="error"
```

If flushing telemetry to the control plane fails (e.g., network error), the failure is logged as a warning but does not interfere with your agent's execution.

## Policy Enforcement on Entry

When `enforce_policy=True`, policies are checked before the run starts. If the policy result is `block` or `throttle`, a `PolicyViolationError` is raised and no run is created:

```python
# Canonical import — also exported from waxell_observe.errors
from waxell_observe import PolicyViolationError

# Works identically with both async and sync context managers
try:
    with WaxellContext(
        agent_name="my-agent",
        enforce_policy=True,
    ) as ctx:
        ...
except PolicyViolationError as e:
    print(f"Blocked: {e}")
    print(f"Action: {e.policy_result.action}")
```

## When to Use Context Manager vs Decorator

Choose `WaxellContext` over `@observe` when you need:

- **Multi-step orchestration** -- wrap complex logic that spans multiple functions
- **Mid-execution policy checks** -- re-validate policies between steps
- **Explicit input/metadata control** -- pass structured inputs and metadata at context creation
- **Multiple runs in one function** -- start and complete several runs in sequence
- **Conditional observability** -- only create a context under certain conditions
- **Synchronous code** -- batch scripts, CLI tools, and ETL pipelines that don't use async

Example of multiple runs (sync -- natural fit for batch processing):

```python
def batch_process(items: list[str]):
    for item in items:
        with WaxellContext(
            agent_name="batch-processor",
            inputs={"item": item},
        ) as ctx:
            result = process_item(item)
            ctx.record_llm_call(model="gpt-4o-mini", tokens_in=50, tokens_out=30)
            ctx.set_result({"output": result})
```

The same pattern works with `async with` for async code:

```python
async def batch_process(items: list[str]):
    for item in items:
        async with WaxellContext(
            agent_name="batch-processor",
            inputs={"item": item},
        ) as ctx:
            result = await process_item(item)
            ctx.record_llm_call(model="gpt-4o-mini", tokens_in=50, tokens_out=30)
            ctx.set_result({"output": result})
```

## Full Example (Async)

```python
from waxell_observe import WaxellObserveClient, WaxellContext

WaxellObserveClient.configure(
    api_url="https://acme.waxell.dev",
    api_key="wax_sk_...",
)

async def run_pipeline(query: str) -> dict:
    async with WaxellContext(
        agent_name="research-pipeline",
        workflow_name="deep-research",
        inputs={"query": query},
        metadata={"version": "2.1"},
        enforce_policy=True,
    ) as ctx:
        # Step 1: Search
        sources = await search(query)
        ctx.record_step("search", output={"source_count": len(sources)})

        # Step 2: Synthesize
        synthesis = await synthesize(query, sources)
        ctx.record_llm_call(
            model="claude-sonnet-4",
            tokens_in=2000,
            tokens_out=500,
            task="synthesize",
        )
        ctx.record_step("synthesize", output={"length": len(synthesis)})

        # Mid-execution policy check
        policy = await ctx.check_policy()
        if policy.blocked:
            ctx.set_result({"error": "Policy blocked continuation"})
            return {"error": policy.reason}

        # Step 3: Refine
        final = await refine(synthesis)
        ctx.record_llm_call(
            model="gpt-4o",
            tokens_in=800,
            tokens_out=300,
            task="refine",
        )
        ctx.record_step("refine")

        result = {"answer": final, "sources": len(sources)}
        ctx.set_result(result)
        return result
```

## Full Example (Sync)

```python
from waxell_observe import WaxellObserveClient, WaxellContext

WaxellObserveClient.configure(
    api_url="https://acme.waxell.dev",
    api_key="wax_sk_...",
)

def process_tickets(tickets: list[dict]) -> list[dict]:
    results = []
    for ticket in tickets:
        with WaxellContext(
            agent_name="ticket-processor",
            workflow_name="support-pipeline",
            inputs={"ticket_id": ticket["id"], "subject": ticket["subject"]},
            enforce_policy=True,
        ) as ctx:
            ctx.set_tag("priority", ticket["priority"])

            # Step 1: Classify
            category = classify_ticket(ticket)
            ctx.record_llm_call(model="gpt-4o-mini", tokens_in=200, tokens_out=10, task="classify")
            ctx.record_step("classify", output={"category": category})

            # Step 2: Generate response
            response = generate_response(ticket, category)
            ctx.record_llm_call(model="gpt-4o", tokens_in=500, tokens_out=200, task="respond")
            ctx.record_step("respond", output={"length": len(response)})

            # Mid-execution policy check (sync variant)
            policy = ctx.check_policy_sync()
            if policy.blocked:
                ctx.set_result({"error": policy.reason})
                results.append({"ticket_id": ticket["id"], "error": policy.reason})
                continue

            ctx.record_score("response_quality", 0.9)
            result = {"ticket_id": ticket["id"], "category": category, "response": response}
            ctx.set_result(result)
            results.append(result)

    return results
```

## Conversation State

WaxellContext automatically tracks conversation metrics from LLM calls:

- `ctx.conversation_turns` — number of user turns in the conversation
- `ctx.context_utilization` — context window usage as a percentage (0-100%)
- `ctx.message_count` — total messages in the LLM context

These properties are read-only and updated automatically when auto-instrumentation records LLM calls.

### Manual Recording

For agents not using auto-instrumented LLM providers, drive the counters
explicitly with `ctx.record_user_message(...)` / `ctx.record_agent_response(...)`
(see [Conversation & Human Interaction](#conversation--human-interaction) above).
These methods create IO spans that appear in the trace timeline alongside LLM
calls and tool invocations. See [Conversation Tracking](../features/conversation-tracking)
for full details.

## Module-level Helpers

Every method above has a module-level shorthand that operates on the *currently
active* `WaxellContext` (resolved via `ContextVar`) — no-op outside a run. Use
these from deeply-nested tool bodies, callbacks, or third-party adapters that
the decorator can't pass `ctx` to.

```python


# Access the active context (returns None outside a run)
ctx = waxell.get_current_context()   # or: waxell.get_context()

# Recording — no ctx argument needed
waxell.score("relevance", 0.92)
waxell.tag("environment", "prod")
waxell.metadata("retrieval_count", 5)
waxell.step("classify", output={"category": "billing"})
waxell.decide("route", chosen="research", options=["direct", "research"])
waxell.reason("evaluate", thought="Source B has higher authority")
waxell.retrieve(query=q, documents=docs, source="pinecone")
waxell.retry(attempt=2, reason="Rate limited", strategy="fallback")

# Memory + prompt lineage
waxell.remember_episodic("deal_findings", data={"company": "Acme"})
waxell.remember_semantic("preferences", "Acme prefers monthly invoicing.")
waxell.recall(query=q, tier="episodic", results_count=3, top_score=0.84)
waxell.prompt_use("customer_intake", version=7, label="prod")

# Conversation + HITL
waxell.user_message("What's the weather?")
waxell.agent_response("Sunny in Paris.")
waxell.communication(channel="slack", recipient="#ops", body="deploy ok")
answer = waxell.input("Approve? (y/n): ")        # drop-in for input()
with waxell.human_turn(prompt="Review PR", channel="github") as turn:
    turn.set_response(wait_for_webhook())
with waxell.pause(reason="awaiting_approval"):
    decision = wait_for_human()

# Approval lifecycle
waxell.approval_request(action_type="delete", approvers=["sec@acme.com"])
waxell.approval_response(action_type="delete", decision="approved")

# Incremental flush
await waxell.flush()       # async
waxell.flush_sync()        # sync
```

## Span Cap (long-running agents)

A long-running or autonomous agent can emit unboundedly many spans. To prevent
the in-process buffer from ballooning, **each run is capped at 50 000 spans**
(root + first 49 999 — the tail is dropped, a `WARNING` is logged, and the
flushed run carries a `_span_cap_hit` summary so truncation is visible).

Tune via env var:

```bash


```

Typical batch loops won't hit this. If you do, prefer wrapping each iteration
in its own `WaxellContext` so each batch item is a discrete run with its own
budget — see the [batch examples](#when-to-use-context-manager-vs-decorator)
above.

## Next Steps

- [Decorator Pattern](./decorator) -- Simpler alternative for single-function agents
- [LLM Call Tracking](../features/llm-tracking) -- Details on captured LLM data
- [Conversation Tracking](../features/conversation-tracking) -- Auto-captured conversation data
- [Policy & Governance](../features/governance) -- Policy actions and enforcement
- [Sessions](/docs/observe/features/sessions) -- Group related runs
- [User Tracking](/docs/observe/features/user-tracking) -- Track end-user identity
- [Scoring](/docs/observe/features/scoring) -- Quality metrics


--------------------------------------------------------------------------------
# Streaming
URL: https://waxell.ai/docs/observe/integrations/streaming
Description: Capture streaming LLM responses with proper token counting
--------------------------------------------------------------------------------

# Streaming Integration

**Auto-Instrumented Streaming**
If you're using `waxell.init()`, streaming responses from OpenAI, Anthropic, Gemini, Cohere, and Bedrock are **captured automatically** -- the SDK wraps stream iterators to aggregate tokens, timing, and cost without any manual code. The examples below show the simple decorator pattern. See [Advanced: Custom Stream Processing](#advanced-custom-stream-processing) for cases where you need manual control.


When auto-instrumentation is active, streaming Just Works: you iterate the stream as you normally would, and the SDK records content, model, tokens (input and output), latency, and cost behind the scenes.

## OpenAI Streaming

```python

waxell.init(api_key="wax_sk_...", api_url="https://waxell.dev")

# Import AFTER init() so OpenAI is auto-instrumented
from openai import AsyncOpenAI

client = AsyncOpenAI()


@waxell.observe(agent_name="streaming-agent")
async def stream_openai(query: str) -> str:
    stream = await client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Be concise."},
            {"role": "user", "content": query},
        ],
        stream=True,
    )

    content = ""
    async for chunk in stream:
        if chunk.choices and chunk.choices[0].delta.content:
            text = chunk.choices[0].delta.content
            content += text
            print(text, end="", flush=True)  # stream to user

    return content


# Run it
answer = await stream_openai(
    "Explain quantum computing",
    session_id="sess_001",
    user_id="user_alice",
)
```

That's it. The trace contains the model, full content, prompt + completion tokens, latency, and cost -- all captured by auto-instrumentation.

## Anthropic Streaming

Anthropic streams typed events instead of chunks, but auto-instrumentation handles both -- you iterate normally:

```python

waxell.init(api_key="wax_sk_...")


client = anthropic.AsyncAnthropic()


@waxell.observe(agent_name="streaming-claude")
async def stream_claude(query: str) -> str:
    stream = await client.messages.create(
        model="claude-sonnet-4",
        max_tokens=500,
        messages=[{"role": "user", "content": query}],
        stream=True,
    )

    content = ""
    async for event in stream:
        if event.type == "content_block_delta" and hasattr(event.delta, "text"):
            text = event.delta.text
            content += text
            print(text, end="", flush=True)

    return content
```

Input tokens (from `message_start`) and output tokens (from `message_delta`) are extracted automatically.

## Streaming Comparison Example

Compare streaming across providers in a single trace. Each call is auto-captured:

```python

waxell.init(api_key="wax_sk_...")

from openai import AsyncOpenAI


openai_client = AsyncOpenAI()
anthropic_client = anthropic.AsyncAnthropic()


@waxell.observe(agent_name="streaming-demo", workflow_name="streaming-comparison")
async def compare_streaming(query: str) -> dict:
    waxell.tag("demo", "streaming")
    waxell.tag("providers", "openai,anthropic")

    # --- OpenAI Streaming (auto-captured) ---
    print("OpenAI: ", end="", flush=True)
    openai_content = ""
    async for chunk in await openai_client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": query}],
        stream=True,
    ):
        if chunk.choices and chunk.choices[0].delta.content:
            openai_content += chunk.choices[0].delta.content
            print(chunk.choices[0].delta.content, end="", flush=True)
    print()
    waxell.step("openai_stream", output={"chars": len(openai_content)})

    # --- Anthropic Streaming (auto-captured) ---
    print("Anthropic: ", end="", flush=True)
    anthropic_content = ""
    async for event in await anthropic_client.messages.create(
        model="claude-sonnet-4",
        max_tokens=500,
        messages=[{"role": "user", "content": query}],
        stream=True,
    ):
        if event.type == "content_block_delta" and hasattr(event.delta, "text"):
            anthropic_content += event.delta.text
            print(event.delta.text, end="", flush=True)
    print()
    waxell.step("anthropic_stream", output={"chars": len(anthropic_content)})

    return {
        "openai_response": openai_content[:500],
        "anthropic_response": anthropic_content[:500],
    }


# Run it
await compare_streaming(
    "Explain quantum computing in simple terms",
    session_id="sess_compare_001",
)
```

## Key Differences

| Aspect | OpenAI | Anthropic |
|--------|--------|-----------|
| Iterator | `async for chunk in stream` | `async for event in stream` |
| Content location | `chunk.choices[0].delta.content` | `event.delta.text` (when `event.type == "content_block_delta"`) |
| Token availability | Final chunk (if requested) | Separate `message_start` / `message_delta` events |

For both providers, auto-instrumentation extracts model, content, tokens, and cost without any manual code.

## Advanced: Custom Stream Processing

If you need to process the stream in ways the SDK can't infer (e.g. an unsupported provider, custom event handling, or you want to record specific per-chunk telemetry), drop down to the context manager and call `record_llm_call` yourself after the stream completes:

```python

waxell.init(api_key="wax_sk_...")

from waxell_observe import WaxellContext

async with WaxellContext(agent_name="custom-stream") as ctx:
    # ... your custom streaming logic, accumulate content + tokens ...

    ctx.record_llm_call(
        model="my-custom-model",
        tokens_in=tokens_in,
        tokens_out=tokens_out,
        response_preview=content[:200],
    )
```

See the [Context Manager](./context-manager) page for the full API.

## Token Estimation

If you do need to estimate tokens in a custom-streaming scenario:

```python
def estimate_tokens(text: str) -> int:
    """Rough estimate: ~1.3 tokens per word for English."""
    return int(len(text.split()) * 1.3)
```

For more accurate estimation with OpenAI models:

```python


def count_tokens(text: str, model: str = "gpt-4o") -> int:
    encoding = tiktoken.encoding_for_model(model)
    return len(encoding.encode(text))
```

## Best Practices

1. **Trust auto-instrumentation** -- iterate the stream as normal, the SDK handles capture
2. **Wrap streaming in `@observe`** -- groups the stream into a tracked run
3. **Use `print(..., flush=True)` for real-time UX** -- streams to the user while the SDK records
4. **Drop to a context manager only for unsupported providers** -- 95% of cases don't need it

## Next Steps

- [OpenAI Integration](./openai) -- Non-streaming patterns
- [Anthropic Integration](./anthropic) -- Non-streaming patterns
- [Multi-Agent](./multi-agent) -- Streaming in multi-agent systems


--------------------------------------------------------------------------------
# Multi-Agent
URL: https://waxell.ai/docs/observe/integrations/multi-agent
Description: Trace correlated multi-agent systems with shared sessions
--------------------------------------------------------------------------------

# Multi-Agent Integration

Trace complex multi-agent systems where a coordinator dispatches tasks to specialized sub-agents. Use shared session IDs to correlate all agents in a single trace.

## The Pattern

1. Generate a single `session_id` for the entire workflow
2. Coordinator is an `@observe`-decorated function that calls sub-agents
3. Sub-agents are `@waxell_agent` (alias for `@observe`) decorated functions that share the same session
4. All agents appear in the same trace, linked by session
5. LLM calls inside any agent are auto-captured by `init()` -- no manual `record_llm_call`

## Complete Example

A coordinator dispatching to planner, researcher, and executor agents:

```python

waxell.init(api_key="wax_sk_...", api_url="https://waxell.dev")

# Import AFTER init() so OpenAI is auto-instrumented
from openai import AsyncOpenAI
from waxell_observe import waxell_agent, generate_session_id
from waxell_observe.errors import PolicyViolationError

client = AsyncOpenAI()


# --- Sub-agent: Planner ---
@waxell_agent(agent_name="planner", workflow_name="plan-task")
async def plan_task(task_description: str) -> dict:
    """Break a task into research queries."""
    waxell.step("analyze_task", output={"task": task_description[:100]})

    # LLM call auto-captured
    response = await client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Break this into 3 research queries."},
            {"role": "user", "content": task_description},
        ],
    )
    content = response.choices[0].message.content
    queries = [line.strip() for line in content.splitlines() if line.strip()][:3]

    waxell.step("generate_plan", output={"num_queries": len(queries)})
    return {"queries": queries}


# --- Sub-agent: Researcher ---
@waxell_agent(agent_name="researcher", workflow_name="research-query")
async def research_query(query: str, query_index: int = 0) -> str:
    """Research a single query."""
    waxell.tag("query_index", str(query_index))
    waxell.step("search", output={"query": query[:100]})

    response = await client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Provide a concise finding."},
            {"role": "user", "content": f"Research: {query}"},
        ],
    )
    finding = response.choices[0].message.content
    waxell.step("compile_findings", output={"length": len(finding)})
    return finding


# --- Sub-agent: Executor ---
@waxell_agent(agent_name="executor", workflow_name="synthesize-findings")
async def synthesize_findings(findings: list[str], original_task: str) -> str:
    """Synthesize findings into a final answer."""
    waxell.metadata("num_findings", len(findings))
    waxell.step("evaluate_findings", output={"count": len(findings)})

    findings_text = "\n".join(f"- {f}" for f in findings)
    response = await client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Synthesize these findings."},
            {"role": "user", "content": f"Task: {original_task}\n\nFindings:\n{findings_text}"},
        ],
    )
    answer = response.choices[0].message.content
    waxell.step("produce_output", output={"length": len(answer)})
    return answer


# --- Coordinator ---
@waxell.observe(agent_name="coordinator", workflow_name="multi-agent-task")
async def run_multi_agent_task(task: str) -> str:
    waxell.tag("demo", "multi-agent")
    waxell.tag("num_agents", "3")

    # Phase 1: Planning
    waxell.step("delegate_to_planner")
    plan_result = await plan_task(task)
    queries = plan_result["queries"]

    # Phase 2: Research (could be parallel; see below)
    waxell.step("delegate_to_researchers")
    findings = []
    for i, query in enumerate(queries):
        finding = await research_query(query, query_index=i)
        findings.append(finding)

    # Phase 3: Synthesis
    waxell.step("delegate_to_executor")
    final_answer = await synthesize_findings(findings, task)

    return final_answer


# Run it -- one session_id flows to every sub-agent via call-time kwargs
async def main():
    session = generate_session_id()
    try:
        answer = await run_multi_agent_task(
            "What are the key considerations for deploying AI agents in production?",
            session_id=session,
            user_id="user_123",
        )
        print(answer)
    except PolicyViolationError as e:
        print(f"Policy violation: {e}")
```

**Session propagation**
Pass `session_id` once at the coordinator's call site. Sub-agent calls inherit the active session automatically when invoked inside the coordinator's `@observe` run. You can also pass `session_id=...` explicitly to any sub-agent call for full control.


## Parallel Research

For independent queries, run researchers in parallel:

```python


@waxell.observe(agent_name="coordinator")
async def run_parallel(task: str) -> str:
    plan_result = await plan_task(task)
    queries = plan_result["queries"]

    waxell.step("delegate_to_researchers_parallel")
    findings = await asyncio.gather(*[
        research_query(query, query_index=i)
        for i, query in enumerate(queries)
    ])

    return await synthesize_findings(findings, task)
```

## Session Correlation

All agents with the same `session_id` appear together in the UI:

```
Session: sess_a1b2c3d4e5f6
├── coordinator (multi-agent-task)
│   ├── Step: delegate_to_planner
│   ├── Step: delegate_to_researchers
│   └── Step: delegate_to_executor
├── planner (plan-task)
│   ├── Step: analyze_task
│   ├── Step: generate_plan
│   └── LLM: gpt-4o-mini (auto-captured)
├── researcher (research-query) [query_index=0]
│   ├── Step: search
│   ├── Step: compile_findings
│   └── LLM: gpt-4o-mini (auto-captured)
├── researcher (research-query) [query_index=1]
│   └── ...
├── researcher (research-query) [query_index=2]
│   └── ...
└── executor (synthesize-findings)
    ├── Step: evaluate_findings
    ├── Step: produce_output
    └── LLM: gpt-4o-mini (auto-captured)
```

## Tagging Sub-Agents

Use tags to identify specific invocations:

```python
@waxell_agent(agent_name="researcher")
async def research_query(query: str, query_index: int = 0) -> str:
    waxell.tag("query_index", str(query_index))
    waxell.tag("query_hash", hash(query) % 10000)
    # ...
```

## Error Propagation

Errors in sub-agents bubble up to the coordinator. Each sub-agent's run is still recorded with `status="error"`.

```python
@waxell.observe(agent_name="coordinator")
async def coordinator(task: str) -> str:
    try:
        findings = await research_query("...")
    except PolicyViolationError as e:
        waxell.tag("error", "policy_violation")
        raise
```

## Metrics Aggregation

The session view shows aggregated metrics:

- **Total LLM calls**: sum across all agents
- **Total tokens**: sum of input + output tokens
- **Total cost**: sum of all LLM costs
- **Duration**: wall-clock time for the entire session
- **Agent count**: number of distinct agents

## Parent-Child Run Lineage

Beyond session correlation, the SDK supports explicit parent-child relationships between runs. When a coordinator spawns a sub-agent on a different worker (or you need a true causality graph rather than session-grouping), pass the parent's `run_id` to create the hierarchy:

```python
from waxell_observe import WaxellObserveClient, WaxellContext

client = WaxellObserveClient()

async with WaxellContext(agent_name="coordinator", session_id=session) as parent_ctx:
    # Sub-agent on a different worker creates a child run linked to the parent
    run_info = await client.start_run(
        agent_name="researcher",
        session_id=session,
        parent_workflow_id=parent_ctx.run_id,
        root_workflow_id=parent_ctx.run_id,
    )
    # ... sub-agent work ...
    await client.complete_run(run_info.run_id, result={"findings": findings})
```

Parent-child lineage appears in the dashboard as a tree view, distinct from session-level grouping. This is one of the few places where dropping to `WaxellContext` is justified -- most multi-agent setups should use the `@observe` / `@waxell_agent` pattern above. For full causality graphs (spawn chains, signals, retries), see [Lineage](../lineage).

## Best Practices

1. **Single session_id** -- generate once, pass to the coordinator's call; sub-agents inherit it
2. **Descriptive agent names** -- `planner`, `researcher`, `executor` not `agent1`, `agent2`
3. **Use tags for differentiation** -- `query_index`, `model_tier`, etc.
4. **Record delegation steps** -- `waxell.step("delegate_to_planner")` shows orchestration flow
5. **Handle errors at the coordinator** -- centralized error handling and logging
6. **Run independent sub-tasks in parallel** -- `asyncio.gather` works with decorated agents

## Next Steps

- [Decorator Pattern](./decorator) -- `@observe` / `@waxell_agent` details
- [Context Manager](./context-manager) -- `WaxellContext` for advanced lineage scenarios
- [Sessions](../features/sessions) -- Session analytics and grouping


--------------------------------------------------------------------------------
# OpenAI
URL: https://waxell.ai/docs/observe/integrations/openai
Description: Instrument OpenAI API calls with automatic or manual tracing
--------------------------------------------------------------------------------

# OpenAI Integration

Four ways to add observability to OpenAI calls -- from zero-code auto-instrumentation to function-level decorators.

`init()` auto-patches every public OpenAI surface: Chat Completions, the Responses API, Embeddings, Images (DALL-E), Audio Transcriptions (Whisper) and Audio Speech (TTS) -- sync and async. Streaming, vision (image inputs), and structured outputs go through the same wrappers; no extra config.

## Approach 1: Auto-Instrumentation (Recommended)

The simplest approach -- call `init()` before importing OpenAI:

```python

waxell_observe.init(api_key="wax_sk_...", api_url="https://waxell.dev")

from openai import OpenAI

client = OpenAI()
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}]
)
# Automatically traced with model, tokens, cost
```

## Approach 2: Drop-in Import

Pre-instrumented OpenAI module -- no `init()` needed:

```python
from waxell_observe.openai import openai

client = openai.OpenAI()
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}]
)
# Automatically traced
```

## Approach 3: Multi-Step Pipeline with Decorators

Use `@observe` for the pipeline and `@retrieval` / convenience functions for the steps in between. `init()` auto-captures every OpenAI call -- no manual `record_llm_call` needed.

```python

waxell.init(api_key="wax_sk_...", api_url="https://waxell.dev")

# Import AFTER init() so OpenAI is auto-instrumented
from openai import AsyncOpenAI

client = AsyncOpenAI()


@waxell.retrieval(source="docs-corpus")
def retrieve_documents(terms: str) -> list[dict]:
    """@retrieval auto-extracts query, documents, and scores from the return value."""
    raw = vector_store.search(terms, top_k=5)
    return [
        {"id": r.id, "text": r.text, "score": r.score}
        for r in raw
    ]


@waxell.observe(agent_name="rag-demo", workflow_name="document-qa")
async def document_qa(query: str) -> dict:
    waxell.tag("demo", "rag")
    waxell.tag("query_type", "multi-step")
    waxell.metadata("document_corpus_size", 5)

    # Step 1: Analyze query (LLM call auto-captured)
    analysis_response = await client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Identify key search terms."},
            {"role": "user", "content": query},
        ],
    )
    terms = analysis_response.choices[0].message.content
    waxell.step("analyze_query", output={"terms": terms})

    # Step 2: Retrieve documents (auto-recorded by @retrieval)
    documents = retrieve_documents(terms)

    # Step 3: Synthesize answer (LLM call auto-captured)
    synthesis_response = await client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Synthesize an answer from documents."},
            {"role": "user", "content": f"Question: {query}\n\nDocuments: {documents}"},
        ],
    )
    answer = synthesis_response.choices[0].message.content

    waxell.score("grounded", True, data_type="boolean")
    return {"answer": answer, "documents_used": len(documents)}


# Run with session and user tracking
result = await document_qa(
    "What are AI best practices?",
    session_id="sess_abc123",
    user_id="user_123",
)
```

## Approach 4: Decorator Pattern

Function-level tracing with automatic IO capture. The `@observe` decorator handles the run lifecycle; `init()` auto-captures every LLM call -- no manual recording needed.

```python

waxell.init(api_key="wax_sk_...", api_url="https://waxell.dev")

from openai import AsyncOpenAI

client = AsyncOpenAI()

@waxell.observe(agent_name="chatbot", workflow_name="chat")
async def chat(message: str) -> str:
    """Chat function with automatic tracing."""
    # Enrichment via convenience functions
    waxell.tag("intent", "question")
    waxell.metadata("model_config", {"temperature": 0.7})

    # LLM call auto-captured by init() -- no manual record_llm_call needed
    response = await client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": message}],
    )
    content = response.choices[0].message.content

    waxell.score("quality", 0.9)
    return content

# Usage -- pass session_id / user_id at call time
result = await chat(
    "What is machine learning?",
    session_id="sess_abc123",
    user_id="user_456",
)
```

## RAG Pipeline Example

Complete RAG pipeline using `@observe` and `@retrieval`. Every OpenAI call is auto-captured by `init()`.

```python

waxell.init(api_key="wax_sk_...", api_url="https://waxell.dev")

from waxell_observe.errors import PolicyViolationError
from openai import AsyncOpenAI

client = AsyncOpenAI()


@waxell.retrieval(source="docs-corpus")
def retrieve_docs(terms: str) -> list[dict]:
    # Replace with your vector store; @retrieval auto-extracts scores
    return [{"id": "doc1", "text": "AI safety involves...", "score": 0.91}]


@waxell.observe(
    agent_name="rag-agent",
    workflow_name="document-qa",
    enforce_policy=True,
)
async def run_rag_pipeline(query: str) -> str:
    waxell.tag("pipeline", "rag")
    waxell.metadata("corpus_version", "2024-01")

    # Step 1: Query analysis (auto-captured)
    analysis = await client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Extract search terms."},
            {"role": "user", "content": query},
        ],
    )
    terms = analysis.choices[0].message.content
    waxell.step("analyze", output={"terms": terms})

    # Step 2: Retrieval (auto-recorded by @retrieval)
    docs = retrieve_docs(terms)

    # Step 3: Synthesis (auto-captured)
    synthesis = await client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": "Answer using the documents."},
            {"role": "user", "content": f"Q: {query}\nDocs: {docs}"},
        ],
    )
    answer = synthesis.choices[0].message.content

    waxell.score("grounded", True, data_type="boolean")
    return answer


# Run it
try:
    answer = await run_rag_pipeline(
        "What are AI safety best practices?",
        session_id="sess_xyz",
        user_id="user_456",
    )
except PolicyViolationError as e:
    print(f"Policy violation: {e}")
```

## Streaming Support

Auto-instrumentation handles OpenAI streaming end-to-end -- the SDK wraps the stream iterator to aggregate content, token counts, and timing. Just iterate the stream as you normally would:

```python

waxell.init(api_key="wax_sk_...")

from openai import AsyncOpenAI

client = AsyncOpenAI()

@waxell.observe(agent_name="streaming-agent")
async def stream_response(query: str) -> str:
    stream = await client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": query}],
        stream=True,
    )

    content = ""
    async for chunk in stream:
        if chunk.choices and chunk.choices[0].delta.content:
            content += chunk.choices[0].delta.content
            print(chunk.choices[0].delta.content, end="", flush=True)

    return content
```

See [Streaming Integration](./streaming) for advanced patterns when you need custom stream processing.

## Responses API

The newer `client.responses.create` surface is auto-patched alongside `chat.completions.create`. Spans are tagged `waxell.openai.api_type=responses`, and the `input` / `instructions` payload runs through the same prompt-guard + PII scrub path:

```python
from openai import OpenAI

client = OpenAI()
response = client.responses.create(
    model="gpt-4o-mini",
    instructions="You are a terse assistant.",
    input="Summarize the Waxell observability story in one sentence.",
)
# Captured: input/output tokens, cost, output text, tool types used
```

## Vision (multimodal inputs)

Vision goes through `chat.completions.create` with a multimodal content array, so it's covered by the standard wrapper -- you get the same model / token / cost capture:

```python
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "What's in this image?"},
            {"type": "image_url", "image_url": {"url": "https://example.com/chart.png"}},
        ],
    }],
)
```

## Structured outputs

`response_format` (JSON mode or JSON schema) is captured on the standard chat.completions wrapper -- no extra setup. The parsed schema name appears in the trace alongside tokens and cost:

```python
from pydantic import BaseModel

class Ticket(BaseModel):
    title: str
    priority: str

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Logan can't log in"}],
    response_format={
        "type": "json_schema",
        "json_schema": {"name": "Ticket", "schema": Ticket.model_json_schema()},
    },
)
```

## Whisper & TTS

Audio calls record duration-based cost (per-minute for Whisper, per-character for TTS):

```python
# Speech to text
with open("call.mp3", "rb") as f:
    transcript = client.audio.transcriptions.create(model="whisper-1", file=f)

# Text to speech
speech = client.audio.speech.create(
    model="tts-1",
    voice="alloy",
    input="Waxell is live.",
)
```

## Best Practices

1. **Use auto-instrumentation for simple cases** -- no code changes needed
2. **Wrap pipelines in `@observe`** -- groups all LLM calls into a single run
3. **Use `waxell.step()` / `waxell.score()` / `waxell.tag()`** -- enrichment without managing a context object
4. **Pass `session_id` and `user_id` at call time** -- the decorator intercepts these kwargs
5. **Handle PolicyViolationError** -- governance can block execution

## Advanced: Context Manager

For batch loops or multi-agent orchestration that spans multiple functions, see the [Context Manager](./context-manager) page. For 90%+ of use cases, the decorator pattern above is the right call.

## Next Steps

- [Streaming Integration](./streaming) -- Detailed streaming patterns
- [Multi-Agent](./multi-agent) -- Coordinate multiple agents
- [Governance](../features/governance) -- Policy enforcement


--------------------------------------------------------------------------------
# Anthropic
URL: https://waxell.ai/docs/observe/integrations/anthropic
Description: Instrument Anthropic Claude API calls with Waxell Observe
--------------------------------------------------------------------------------

# Anthropic Integration

Add observability to Anthropic Claude API calls with auto-instrumentation plus decorators for structure.

## Quick Start

```python

waxell.init(api_key="wax_sk_...", api_url="https://waxell.dev")

# Import Anthropic AFTER init() -- now auto-instrumented


client = anthropic.Anthropic()
response = client.messages.create(
    model="claude-sonnet-4-5-20250929",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Hello!"}]
)
# Automatically traced with model, tokens, cost
```

## Drop-in Import

Alternative approach using pre-instrumented module:

```python
from waxell_observe.anthropic import anthropic

client = anthropic.Anthropic()
response = client.messages.create(
    model="claude-sonnet-4-5-20250929",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Hello!"}]
)
# Automatically traced
```

## Decorator Pattern (Recommended for Pipelines)

For multi-step Claude pipelines, wrap your function with `@observe`. Every `client.messages.create(...)` call is auto-captured -- no manual `record_llm_call` needed.

```python

waxell.init(api_key="wax_sk_...", api_url="https://waxell.dev")


from waxell_observe.errors import PolicyViolationError

client = anthropic.AsyncAnthropic()


@waxell.observe(
    agent_name="anthropic-demo",
    workflow_name="content-analysis",
    enforce_policy=True,
)
async def analyze_content(query: str) -> dict:
    waxell.tag("demo", "anthropic")
    waxell.tag("provider", "anthropic")
    waxell.metadata("sdk", "anthropic-python")

    # Step 1: Classify content (auto-captured)
    classify_response = await client.messages.create(
        model="claude-sonnet-4-5-20250929",
        max_tokens=500,
        messages=[{
            "role": "user",
            "content": f"Classify this text: {query}",
        }],
    )
    classification = classify_response.content[0].text
    waxell.step("classify_content", output={"classification": classification[:200]})

    # Step 2: Extract entities (auto-captured)
    extract_response = await client.messages.create(
        model="claude-sonnet-4-5-20250929",
        max_tokens=500,
        messages=[{
            "role": "user",
            "content": f"Extract key entities from: {query}",
        }],
    )
    entities = extract_response.content[0].text
    waxell.step("extract_entities", output={"entities": entities[:200]})

    # Step 3: Summarize (auto-captured)
    summary_response = await client.messages.create(
        model="claude-sonnet-4-5-20250929",
        max_tokens=500,
        messages=[{
            "role": "user",
            "content": (
                f"Summarize:\n\n"
                f"Text: {query}\n"
                f"Classification: {classification}\n"
                f"Entities: {entities}"
            ),
        }],
    )
    summary = summary_response.content[0].text

    waxell.score("completeness", 0.95)
    return {
        "classification": classification,
        "entities": entities,
        "summary": summary,
    }


# Run it -- pass session_id / user_id at call time
try:
    result = await analyze_content(
        "Analyze the impact of AI on healthcare",
        session_id="sess_health_001",
        user_id="user_789",
        user_group="enterprise",
    )
except PolicyViolationError as e:
    print(f"Policy violation: {e}")
```

## Streaming with Anthropic

Auto-instrumentation captures Anthropic streaming end-to-end -- the SDK aggregates content, input tokens (from `message_start`), and output tokens (from `message_delta`) automatically. Just iterate the stream:

```python

waxell.init(api_key="wax_sk_...")


client = anthropic.AsyncAnthropic()

@waxell.observe(agent_name="streaming-claude")
async def stream_claude(query: str) -> str:
    stream = await client.messages.create(
        model="claude-sonnet-4-5-20250929",
        max_tokens=500,
        messages=[{"role": "user", "content": query}],
        stream=True,
    )

    content = ""
    async for event in stream:
        if event.type == "content_block_delta" and hasattr(event.delta, "text"):
            content += event.delta.text
            print(event.delta.text, end="", flush=True)

    return content
```

See [Streaming Integration](./streaming) for advanced patterns when you need custom stream processing.

## Auto-Instrumented Features

All of the following go through the same `messages.create` / `messages.stream` wrappers -- no extra config:

- **Prompt caching** -- calls routed through `client.beta.messages.create` (the beta surface used by frameworks such as pydantic-ai) are patched separately and captured with the same token + cost attributes.
- **Vision (multi-modal inputs)** -- image blocks inside the `messages` array pass through the standard wrapper; model, tokens, and cost are recorded normally.
- **Structured outputs via `tool_use`** -- when the model returns a `tool_use` content block, the instrumentor captures the tool name and arguments in the response preview. The `tools` list from the request is also recorded (names only) so the trace shows which capabilities were available.
- **`messages.stream()` context manager** -- the sync and async `MessageStreamManager` surfaces (used by AWS Strands and similar frameworks) are wrapped in addition to `create(stream=True)`.

## Supported Models

| Model | Auto-Instrumented | Cost Tracking |
|-------|-------------------|---------------|
| claude-opus-4-8 | Yes | Yes |
| claude-opus-4-7 | Yes | Yes |
| claude-opus-4-6 | Yes | Yes |
| claude-opus-4 | Yes | Yes |
| claude-sonnet-4-6 | Yes | Yes |
| claude-sonnet-4-5 | Yes | Yes |
| claude-sonnet-4 | Yes | Yes |
| claude-haiku-4-5 | Yes | Yes |
| claude-3-5-sonnet | Yes | Yes |
| claude-3-5-haiku | Yes | Yes |
| claude-3-haiku | Yes | Yes |
| claude-3-opus (legacy) | Yes | Yes |
| claude-3-sonnet (legacy) | Yes | Yes |

Cost lookup uses longest-prefix matching, so dated variants (e.g. `claude-sonnet-4-5-20250929`) resolve automatically. Anthropic's API requires full dated model IDs; bare aliases like `claude-sonnet-4` are accepted by the SDK client but may not resolve to the model you expect.

## Tags and Metadata

Enrich traces with contextual information using convenience functions:

```python
@waxell.observe(agent_name="claude-agent")
async def my_claude_agent(query: str) -> str:
    # Tags: searchable in UI
    waxell.tag("provider", "anthropic")
    waxell.tag("model_tier", "premium")
    waxell.tag("use_case", "analysis")

    # Metadata: arbitrary JSON-serializable values
    waxell.metadata("sdk_version", "0.25.0")
    waxell.metadata("config", {"max_tokens": 500, "temperature": 0.7})

    response = await client.messages.create(
        model="claude-sonnet-4-5-20250929",
        max_tokens=500,
        messages=[{"role": "user", "content": query}],
    )
    return response.content[0].text
```

## Error Handling

```python
from waxell_observe.errors import PolicyViolationError

@waxell.observe(agent_name="claude-agent", enforce_policy=True)
async def my_claude_agent(query: str) -> str:
    response = await client.messages.create(
        model="claude-sonnet-4-5-20250929",
        max_tokens=500,
        messages=[{"role": "user", "content": query}],
    )
    return response.content[0].text


try:
    result = await my_claude_agent("Hello")
except PolicyViolationError as e:
    # Policy blocked execution (budget, rate limit, etc.)
    print(f"Blocked: {e.policy_result.reason}")
except anthropic.APIError as e:
    # Anthropic API error
    print(f"API error: {e}")
```

## Best Practices

1. **Call `init()` before importing anthropic** -- enables auto-instrumentation. `wrapt.wrap_function_wrapper` must bind to the module before your `from anthropic import Anthropic` runs; importing first leaves the methods unwrapped.
2. **Trust auto-instrumentation for token + cost capture** -- Anthropic's `input_tokens` / `output_tokens` and message-streaming events are handled for you
3. **Wrap pipelines in `@observe`** -- groups all Claude calls into one run
4. **Set max_tokens** -- required by the Anthropic API
5. **Add a `provider` tag** -- makes filtering easy in the UI

## Advanced: Context Manager

If you need fine-grained control (batch loops, mid-execution policy checks, manual run lifecycle), see the [Context Manager](./context-manager) page. For most use cases, the decorator pattern above is the right call.

## Next Steps

- [Streaming Integration](./streaming) -- Detailed streaming patterns
- [LiteLLM Integration](./litellm) -- Use Anthropic via LiteLLM
- [Multi-Agent](./multi-agent) -- Coordinate Claude agents


--------------------------------------------------------------------------------
# LiteLLM
URL: https://waxell.ai/docs/observe/integrations/litellm
Description: Multi-provider observability with LiteLLM's unified API
--------------------------------------------------------------------------------

# LiteLLM Integration

Use LiteLLM's unified API to call multiple LLM providers (OpenAI, Anthropic, Groq, etc.) with consistent observability. `waxell.init()` patches LiteLLM automatically -- every `litellm.completion(...)` and `litellm.acompletion(...)` call is captured with no manual recording.

## What's instrumented

`waxell.init()` wraps the two top-level entry points: `litellm.completion` (sync) and `litellm.acompletion` (async). Higher-level surfaces like `litellm.Router.completion(...)` / `Router.acompletion(...)` and the LiteLLM Proxy server's request handlers funnel through the same two functions, so they're covered transitively -- you don't need to patch anything extra.

**Dual-path coverage and task labeling**
If you call a provider SDK directly (`openai`, `anthropic`, `bedrock`, ...) **and** also go through LiteLLM for some calls, both paths get traced. LiteLLM-routed calls produce a span with `provider=litellm`, `task=litellm.completion`, and the resolved model name (e.g. `anthropic/claude-sonnet-4`) -- LiteLLM dispatches to providers over HTTP rather than through the patched provider SDK client, so you don't get a duplicate span for the same HTTP call. Calls made directly against the provider SDK keep their own task label (`openai.chat.completion`, `anthropic.messages.create`, etc.).


## What is LiteLLM?

[LiteLLM](https://github.com/BerriAI/litellm) provides a unified interface to 100+ LLM providers. Instead of learning each provider's SDK, you use one API:

```python


# OpenAI
response = litellm.completion(model="gpt-4o", messages=[...])

# Anthropic
response = litellm.completion(model="anthropic/claude-sonnet-4", messages=[...])

# Groq
response = litellm.completion(model="groq/llama-3.3-70b-versatile", messages=[...])
```

## Quick Start

Call `init()` before importing LiteLLM -- that's all the setup auto-instrumentation needs.

```python

waxell.init(api_key="wax_sk_...", api_url="https://waxell.dev")

# Import AFTER init() so LiteLLM is auto-instrumented


@waxell.observe(agent_name="litellm-agent", workflow_name="multi-provider")
async def ask(query: str) -> str:
    # LLM call auto-captured -- model, tokens, cost all recorded
    response = await litellm.acompletion(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": query}],
    )
    return response.choices[0].message.content


# Run it
answer = await ask("Hello!", session_id="sess_001", user_id="user_alice")
```

## Multi-Provider Comparison

Compare responses across providers in a single trace. Each `litellm.acompletion(...)` is auto-captured with the right provider and model.

```python

waxell.init(api_key="wax_sk_...")


MODELS = [
    {"model": "gpt-4o-mini", "provider": "OpenAI", "tier": "fast"},
    {"model": "anthropic/claude-sonnet-4", "provider": "Anthropic", "tier": "premium"},
    {"model": "groq/llama-3.3-70b-versatile", "provider": "Groq", "tier": "open-source"},
]


@waxell.observe(agent_name="litellm-demo", workflow_name="multi-provider")
async def compare_providers(query: str) -> dict:
    waxell.tag("demo", "litellm")
    waxell.tag("providers", "openai,anthropic,groq")
    waxell.metadata("num_models", len(MODELS))

    results = []
    for config in MODELS:
        messages = [
            {"role": "system", "content": f"Provide a {config['tier']}-tier analysis."},
            {"role": "user", "content": query},
        ]

        # Each call auto-captured by init()
        response = await litellm.acompletion(model=config["model"], messages=messages)
        content = response.choices[0].message.content

        waxell.step(
            f"call_{config['provider'].lower()}",
            output={
                "model": config["model"],
                "tokens": response.usage.prompt_tokens + response.usage.completion_tokens,
                "content_length": len(content),
            },
        )
        results.append({
            "provider": config["provider"],
            "model": config["model"],
            "tier": config["tier"],
            "content": content,
            "tokens": response.usage.prompt_tokens + response.usage.completion_tokens,
        })

    comparison = {
        r["provider"]: {
            "model": r["model"],
            "tokens": r["tokens"],
            "content_length": len(r["content"]),
        }
        for r in results
    }
    waxell.step("compare_providers", output=comparison)
    return {"comparison": comparison, "results": results}


# Run comparison
results = await compare_providers(
    "Compare AI safety approaches",
    session_id="sess_compare_001",
)
```

## Model Name Conventions

LiteLLM uses prefixes to identify providers:

| Provider | Model Format | Example |
|----------|--------------|---------|
| OpenAI | `model-name` | `gpt-4o`, `gpt-4o-mini` |
| Anthropic | `anthropic/model` | `anthropic/claude-sonnet-4` |
| Groq | `groq/model` | `groq/llama-3.3-70b-versatile` |
| Bedrock | `bedrock/model` | `bedrock/anthropic.claude-3` |
| Azure | `azure/deployment` | `azure/gpt-4-deployment` |
| Cohere | `cohere/model` | `cohere/command-r` |

## Costs and Tokens

Auto-instrumentation pulls `model`, `tokens_in`, `tokens_out`, and `cost` straight from the LiteLLM response. You don't need to record anything manually -- inspect them in the dashboard or read `response.usage.*` in code if you need the values inline.

```python
@waxell.observe(agent_name="cost-aware")
async def ask(query: str) -> str:
    response = await litellm.acompletion(
        model="groq/llama-3.3-70b-versatile",
        messages=[{"role": "user", "content": query}],
    )
    # response.usage.prompt_tokens, response.usage.completion_tokens are still available
    # but they're already captured in the trace -- no need to record again
    return response.choices[0].message.content
```

## Fallback Chains

LiteLLM's `fallbacks` feature works transparently -- the actual model used is captured in the auto-instrumented span.

```python
@waxell.observe(agent_name="fallback-agent")
async def ask_with_fallback(query: str) -> str:
    response = await litellm.acompletion(
        model="gpt-4o",
        messages=[{"role": "user", "content": query}],
        fallbacks=["anthropic/claude-sonnet-4", "groq/llama-3.3-70b-versatile"],
    )
    # Tag with the model that actually answered
    waxell.tag("actual_model", response.model)
    return response.choices[0].message.content
```

## Streaming with LiteLLM

Auto-instrumentation handles LiteLLM streams the same way it handles OpenAI / Anthropic streams -- iterate the stream as normal:

```python
@waxell.observe(agent_name="streaming-litellm")
async def stream_litellm(query: str) -> str:
    response = await litellm.acompletion(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": query}],
        stream=True,
    )

    content = ""
    async for chunk in response:
        if chunk.choices[0].delta.content:
            content += chunk.choices[0].delta.content
            print(chunk.choices[0].delta.content, end="", flush=True)

    return content
```

See [Streaming Integration](./streaming) for advanced patterns when you need custom stream processing.

## Environment Variables

LiteLLM reads API keys from environment:

```bash


```

## Best Practices

1. **Call `init()` before importing litellm** -- patches the SDK for auto-instrumentation
2. **Use provider prefixes** -- `anthropic/claude-sonnet-4`, not just `claude-sonnet-4`
3. **Tag with `response.model` for fallbacks** -- `waxell.tag("actual_model", response.model)`
4. **Pass `session_id` at call time** -- correlate multi-provider comparisons
5. **Skip manual `record_llm_call`** -- auto-instrumentation already captures model, tokens, and cost

## Supported Providers

LiteLLM supports 100+ providers. Common ones with cost tracking in Waxell:

| Provider | Cost Tracking | Notes |
|----------|---------------|-------|
| OpenAI | Full | GPT-4, GPT-4o, etc. |
| Anthropic | Full | Claude models |
| Groq | Full | Llama, Mixtral on Groq |
| Google | Full | Gemini models |
| Mistral | Full | Mistral models |
| Cohere | Partial | Command-R |
| Bedrock | Full | AWS Bedrock models |

## Next Steps

- [Multi-Agent](./multi-agent) -- Coordinate agents across providers
- [Streaming Integration](./streaming) -- Detailed streaming patterns
- [Cost Management](../features/cost-management) -- Track costs across providers


--------------------------------------------------------------------------------
# Claude Code & Cowork
URL: https://waxell.ai/docs/observe/integrations/claude-code
Description: Add observability, governance, and security guardrails to Claude Code and Claude Cowork sessions
--------------------------------------------------------------------------------

# Claude Code & Cowork Integration

[Claude Code](https://docs.anthropic.com/en/docs/claude-code) gives developers an autonomous coding agent with terminal, filesystem, and git access. [Claude Cowork](https://claude.com/blog/cowork-research-preview) takes the same agentic foundation and puts it in the hands of everyone else — marketing, ops, finance, HR, legal. Anyone can point Claude at a folder on their machine and say "organize these receipts into a spreadsheet" or "draft a report from these notes."

That's transformative. It's also terrifying from a governance perspective.

## Why This Matters

Claude Code has guardrails built into developer culture — git history, code review, CI pipelines, pull requests. If an agent writes bad code, the PR catches it.

**Cowork has none of that.** A marketing manager asking Claude to "clean up the Q4 reports folder" doesn't have git. There's no PR review for a spreadsheet. There's no CI pipeline for a slide deck. When Claude deletes the wrong files, overwrites a customer list, or writes PII into a shared document, there's no safety net.

And it's not just accidental damage. Cowork gives Claude filesystem access on real business machines — the same machines with `.env` files, SSH keys, AWS credentials, HR documents, and financial records sitting in ~/Documents.

**Waxell is the governance layer that makes Claude Code and Cowork safe for enterprise adoption.** Every session is traced, every file operation passes through security guardrails, costs are tracked per user, and everything is auditable.

## The Cowork Risk Profile

Developers understand what `rm -rf` does. Your marketing team doesn't. That's not a knock on your marketing team — it's the reality of giving an autonomous agent filesystem access to non-technical users.

### Scenarios that keep security teams up at night

**Sensitive Document Exposure**
A sales ops manager asks Cowork to "compile all the customer contracts into a summary." Claude reads through `~/Documents/Contracts/` and writes a summary file that includes customer SSNs, payment terms, and contract values — then the user shares it in Slack. The **Secret Scanning** policy detects PII patterns in Claude's output before it's written to disk.


**Accidental Data Destruction**
An executive asks Cowork to "organize my Downloads folder." Claude decides to delete 200 "duplicate" files that turn out to be different versions of critical board presentations. The **Destructive Command** guard blocks `rm -rf` and recursive deletes, requiring explicit confirmation before any mass file operations.


**Credential Harvesting**
Claude is working in a project folder and discovers a `.env` file, `~/.aws/credentials`, or an SSH private key. Whether through hallucination or a prompt injection buried in a document, it tries to write those credentials into a new file or include them in output. The **Sensitive File Protection** guard blocks reads and writes to credential files by default.


**Cloud Metadata SSRF**
On a company laptop connected to AWS, Claude tries to fetch `http://169.254.169.254/latest/meta-data/` — the cloud instance metadata endpoint that exposes IAM credentials. The **Network Access** guard blocks all requests to internal IPs, link-local addresses, and cloud metadata endpoints.


**Shadow IT Sprawl**
Ten people on your team are using Cowork with no visibility into what Claude is doing, how much it costs, or what files it's accessing. One user is burning $50/day on Opus. Another has Claude editing production config files. Without Waxell, you have no way to know. **Session tracing and cost tracking** give you a complete picture across every user.


**Path Traversal**
A user asks Cowork to work on files in `~/Projects/website/`. Claude decides it needs to read `../../.ssh/id_rsa` to "check SSH config for deployment." The **Path Boundary** guard blocks file access outside the designated working directory.


**Runaway Sessions**
A Cowork user gives Claude a vague prompt like "improve all the documents in this folder." Claude starts modifying every file it can find — 60 files edited and counting. The **Session Scope** guard warns at 20 files and prompts at 50, catching runaway sessions before they cause unreviewable damage.


## What You Get

| Feature | Description |
|---------|-------------|
| Session tracing | Every Claude Code and Cowork session recorded as an agent run with full trace |
| Tool call tracking | File reads, writes, edits, bash commands, web fetches — all captured as spans |
| LLM cost tracking | Token usage and estimated cost per session, per user |
| Local guard | Instant, zero-latency security checks on every file operation (8 protection layers) |
| Server-side policies | Budget limits, rate limits, scheduling, kill switch — enforced server-side |
| Secret & PII scanning | Detect leaked credentials, API keys, SSNs, and credit cards in outputs |
| Path boundary enforcement | Prevent Claude from accessing files outside the project directory |
| MCP tools | Claude can proactively check policies and record decisions |
| Full audit trail | Every file operation, every LLM invocation, every policy decision — logged and searchable |

## Quick Start

If you don't already have the wax CLI: `pip install waxell` (or `pipx install waxell` for an isolated install). See [CLI reference](/reference/cli#installation) for Windows PATH issues and troubleshooting.

```bash
# 1. Install the SDK + CLI (the meta package pulls in waxell-observe and registers `wax`)
pip install waxell waxell-observe

# 2. Configure your API key
wax configure

# 3. Set up hooks (works for both Claude Code and Cowork)
wax observe claude-code setup --governance
```

That's it. Both Claude Code and Cowork use the same hooks system under the hood, so one setup covers both.

**For Cowork Users**
If you're setting up guardrails for non-technical Cowork users, use `--global` so the protection applies to every folder they give Claude access to — not just one project:

```bash
wax observe claude-code setup --global --governance
```


## Setup Options

### Basic Observability

Traces sessions, tool calls, and LLM usage — no blocking:

```bash
wax observe claude-code setup
```

### With Governance (Recommended)

Adds PreToolUse policy enforcement — Claude must pass your security guardrails before executing commands, file edits, and file writes:

```bash
wax observe claude-code setup --governance
```

### With MCP Tools

Registers a local MCP server so Claude can proactively check policies, query budget status, and record decisions:

```bash
wax observe claude-code setup --governance --mcp
```

### Global Configuration

Apply to all projects and folders (writes to `~/.claude/settings.json`). **Recommended for Cowork deployments** so every folder a user gives Claude access to is protected:

```bash
wax observe claude-code setup --global --governance --mcp
```

### Per-Project Configuration

```bash
wax observe claude-code setup --project-dir /path/to/project
```

## How It Works

Both Claude Code and Cowork are built on the same agent foundation and share the same [hooks system](https://code.claude.com/docs/en/hooks). The setup command writes hook entries into `.claude/settings.json`. When Claude runs, it calls `wax observe claude-code hook` at each lifecycle event:

```
Claude Code / Cowork Session
    │
    ├─ SessionStart  → Creates an agent run in Waxell
    ├─ PreToolUse    → Runs local guard + server policy check (governance mode)
    ├─ PostToolUse   → Records tool call span, tracks modified files
    ├─ SubagentStart → Records subagent span start
    ├─ SubagentStop  → Records subagent span with token usage
    ├─ Stop          → Parses transcript, batch-records LLM calls, completes run
    └─ SessionEnd    → Cleanup
```

## Local Guard

The local guard runs **before** every tool call — zero network latency, pure pattern matching. It provides 8 layers of protection out of the box. This is especially critical for Cowork, where non-technical users may not recognize dangerous operations.

### Destructive Command Blocking

Commands that destroy data are blocked immediately:

| Blocked (deny) | Warned (ask user) |
|----------------|-------------------|
| `rm -rf /`, `rm -rf ~`, `rm -rf .` | Any `rm -rf` |
| `mkfs`, `dd if=`, `> /dev/sd` | `kill -9`, `pkill -9` |
| Fork bombs, `shutdown`, `reboot` | `docker system prune` |
| `chmod -R 777 /` | `DROP TABLE`, `DELETE FROM` without WHERE |
| `curl ... \| bash`, `wget ... \| sh` | |

### Sensitive File Protection

Writes to credential files are blocked:

```
.env, .env.*, .env.production       # Environment variables
**/credentials*, **/secrets.*       # Secrets files
**/*.pem, **/*.key, **/id_rsa      # Private keys
**/.ssh/*, **/.aws/credentials     # Cloud credentials
.git/config, **/.netrc, **/.npmrc  # Auth configs
```

Writes to lock files and `.gitignore` trigger a confirmation prompt.

### Git Safety

| Operation | Action |
|-----------|--------|
| `git push --force` to protected branch | Deny |
| `git reset --hard` | Deny |
| `git clean -f`, `git checkout .`, `git restore .` | Deny |
| `git config` modifications | Deny |
| `git push` to protected branch (non-force) | Ask |
| `git branch -D` | Ask |
| `git rebase` on protected branch | Ask |

Default protected branches: `main`, `master`, `develop`, `release/*`, `production`.

### Path Boundary Enforcement

When enabled (default), writes outside the project directory are blocked. This is a critical guardrail for Cowork — it prevents Claude from reaching into `~/.ssh`, `~/.aws`, or other sensitive directories when a user gives it access to `~/Documents/Reports`.

Reads are always allowed. Exceptions: `/tmp` and system temp directories.

### Network Access Control

Blocks `WebFetch` to internal/private URLs:

- `localhost`, `127.0.0.1`, `0.0.0.0`
- Private networks: `10.*`, `172.16-31.*`, `192.168.*`
- Cloud metadata: `169.254.169.254`, `metadata.google.internal`

### CI/CD & Infrastructure Protection

Writes to infrastructure files require confirmation:

```
Dockerfile, docker-compose*.yml         # Container configs
.github/**, .gitlab-ci.yml, Jenkinsfile # CI/CD pipelines
**/*.tf, **/terraform/**                # Infrastructure as code
**/k8s/**, **/kubernetes/**, **/helm/** # Kubernetes
Makefile, Procfile, vercel.json         # Build & deploy
```

### Session Scope Control

Tracks unique files modified per session. Warns at 20 files, prompts at 50 files. Prevents runaway sessions from touching too many files — especially important for Cowork where a vague prompt like "organize everything" can spiral.

### Multi-Session Conflict Detection

When multiple Claude sessions are active in the same directory (e.g., two Claude Code instances via agent teams, or overlapping Cowork sessions), the guard detects when one session tries to edit a file another session already modified and warns before overwriting.

## Customizing the Guard

Create `.waxell/guard.json` in your project root (or `~/.waxell/guard.json` for global config) to override defaults. Only specified keys are changed — everything else keeps the built-in defaults.

### Example: Strict Config for Cowork Users

Lock down filesystem access for non-technical users:

```json
{
  "path_boundary_enabled": true,
  "max_file_changes": 15,
  "warn_file_changes": 8,
  "blocked_domains": ["competitor.com"],
  "protected_file_patterns": [
    ".env", ".env.*",
    "**/credentials*", "**/secrets.*",
    "**/*.pem", "**/*.key",
    "**/.ssh/*", "**/.aws/*",
    "**/payroll/**", "**/hr/**",
    "**/financial/**"
  ]
}
```

### Example: Developer Config

Developers need more freedom but still want safety rails:

```json
{
  "git_protected_branches": ["main", "master", "staging", "production"],
  "max_file_changes": 50,
  "warn_file_changes": 20,
  "path_boundary_enabled": true
}
```

### Example: Relaxed Dev Config

For local experimentation where you want minimal friction:

```json
{
  "path_boundary_enabled": false,
  "max_file_changes": 200,
  "git_block_hard_reset": false,
  "infra_file_patterns": []
}
```

### Configuration Priority

1. `.waxell/guard.json` in project root (highest priority)
2. `~/.waxell/guard.json` (user global)
3. Built-in defaults (always present)

### All Configuration Fields

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `blocked_command_patterns` | `string[]` | 15 patterns | Regex patterns — matching commands are denied |
| `warn_command_patterns` | `string[]` | 12 patterns | Regex patterns — matching commands prompt user |
| `protected_file_patterns` | `string[]` | 24 patterns | Glob patterns — writes are denied |
| `warn_file_patterns` | `string[]` | 5 patterns | Glob patterns — writes prompt user |
| `path_boundary_enabled` | `bool` | `true` | Block writes outside project directory |
| `git_protected_branches` | `string[]` | `["main","master","develop","release/*","production"]` | Branches protected from force push |
| `git_block_force_push` | `bool` | `true` | Block force push to protected branches |
| `git_block_hard_reset` | `bool` | `true` | Block `git reset --hard` |
| `git_block_config_edit` | `bool` | `true` | Block `git config` modifications |
| `blocked_domains` | `string[]` | `[]` | Domains blocked for WebFetch |
| `block_internal_urls` | `bool` | `true` | Block WebFetch to localhost/private IPs |
| `max_file_changes` | `int` | `50` | Prompt after this many unique file modifications |
| `warn_file_changes` | `int` | `20` | Warn at this many file modifications |
| `detect_file_conflicts` | `bool` | `true` | Warn when another session modified the same file |
| `infra_file_patterns` | `string[]` | 22 patterns | Glob patterns — writes prompt user |

## Server-Side Policies

Beyond the local guard, Waxell enforces server-side policies for capabilities that require persistent state (budgets, rate limits, scheduling). These run after the local guard check.

### Available Policy Templates

Configure these in the Waxell dashboard under **Governance > Policies**:

| Template | Category | What It Does |
|----------|----------|--------------|
| Session Budget | cost | Token and dollar limits per session |
| Daily Budget | cost | Token and dollar limits per day |
| Model Restriction | llm | Restrict which models can be used |
| Business Hours | scheduling | Allow sessions only during work hours |
| Kill Switch | kill | Auto-halt on high error rates |
| Full Audit | audit | Log all inputs/outputs with secret redaction |
| Rate Limit | rate-limit | Requests per minute/hour, concurrent sessions |
| Secret Scanning | content | Detect leaked credentials and PII |
| Webhook Notifications | control | Slack/Teams notifications on events |

### Session Budget Example

Set a $5 / 500K token limit per session:

```json
{
  "per_workflow_token_limit": 500000,
  "per_workflow_cost_limit": 5.00,
  "warning_threshold_percent": 80,
  "action_on_exceed": "warn"
}
```

When 80% of the budget is consumed, Claude receives a warning. At 100%, the action triggers (`warn` or `block`).

### Business Hours Example

Restrict Claude to weekdays, 8am-8pm Eastern:

```json
{
  "allowed_days": [1, 2, 3, 4, 5],
  "start_hour": 8,
  "end_hour": 20,
  "timezone": "America/New_York"
}
```

## MCP Tools

When set up with `--mcp`, Claude gains three proactive tools:

| Tool | Description |
|------|-------------|
| `waxell_check_policy` | Check if a planned action is allowed before attempting it |
| `waxell_budget_status` | Query remaining budget (tokens, cost) for the current session |
| `waxell_record_decision` | Record a decision for the audit trail |

These let Claude self-regulate — it can check budget before starting an expensive operation, or record why it chose a particular approach.

## CLI Commands

### Check Active Sessions

```bash
wax observe claude-code status
```

Shows a table of all active Claude Code and Cowork sessions with session ID, run ID, model, span count, start time, and working directory.

### Clean Up Stale State

```bash
wax observe claude-code clean
```

Removes session state files older than 24 hours.

## Environment Variables

| Variable | Description |
|----------|-------------|
| `WAXELL_API_KEY` | Your Waxell API key (`wax_sk_...`) |
| `WAXELL_API_URL` | Waxell platform URL |
| `WAXELL_OBSERVE` | Set to `false` to disable all telemetry |

## Governance Decision Flow

When governance mode is enabled (`--governance`), every `Bash`, `Edit`, and `Write` tool call goes through this flow:

```
PreToolUse fires (Bash/Edit/Write)
    │
    ├─ 1. LOCAL GUARD (instant, no network)
    │     ├─ Check destructive commands
    │     ├─ Check git operations
    │     ├─ Check file protection
    │     ├─ Check path boundaries
    │     ├─ Check network access
    │     ├─ Check infrastructure files
    │     ├─ Check session scope
    │     └─ Check multi-session conflicts
    │
    │     ├─ DENY → Block tool call immediately
    │     └─ ASK  → Queue warning
    │
    ├─ 2. SERVER POLICY CHECK (budget, scheduling, etc.)
    │     ├─ BLOCK → Deny tool call
    │     └─ WARN  → Queue warning
    │
    ├─ 3. Any warnings queued?
    │     └─ Yes → Prompt user for confirmation
    │
    └─ 4. All clear → Allow tool call
```

## Deploying for Your Team

### For Developer Teams (Claude Code)

1. Add `.waxell/guard.json` to your repo with your team's protected branches and file patterns
2. Run `wax observe claude-code setup --governance` in each project
3. Configure session and daily budgets in the Waxell dashboard
4. Review session traces to tune policies over time

### For Non-Technical Teams (Cowork)

1. Install waxell (CLI + SDK) and waxell-observe on each user's machine: `pip install waxell waxell-observe` (or `pipx install waxell` for an isolated CLI install)
2. Run global setup: `wax observe claude-code setup --global --governance`
3. Create `~/.waxell/guard.json` with strict defaults (low file change limits, protected file patterns for sensitive business documents)
4. Configure daily budget policies per user in the Waxell dashboard
5. Set up Slack/Teams webhook notifications so IT gets alerts on policy violations

### For Enterprise (Both)

1. Deploy waxell + waxell-observe via your package manager or MDM (the `waxell` meta package registers the `wax` CLI; `waxell-observe` alone does not)
2. Use global config (`--global`) with managed `.waxell/guard.json` pushed via config management
3. Enable full audit logging with 90-day retention
4. Configure business hours policies to prevent after-hours usage
5. Set up kill switch policies for automatic incident response
6. Route webhook notifications to your SIEM

## Best Practices

1. **Always use `--governance`** — the local guard has sensible defaults that protect against accidental damage without being overly restrictive
2. **Use `--global` for Cowork deployments** — non-technical users shouldn't have to set up guardrails per folder
3. **Set stricter limits for Cowork than Claude Code** — developers need more freedom, business users need more protection
4. **Add `--mcp` for budget-conscious teams** — lets Claude self-regulate its spending
5. **Use `.waxell/guard.json` for project-specific rules** — commit it to your repo so the whole team gets the same guardrails
6. **Set up server-side budgets** — local guard handles safety, server policies handle cost
7. **Review the dashboard** — session traces show exactly what Claude did, helping you tune policies over time

## Next Steps

- [Governance Features](../features/governance) -- Configure server-side policies
- [Cost Management](../features/cost-management) -- Track and control AI spending
- [Multi-Agent](./multi-agent) -- Observe coordinated agent systems
- [Auto-Instrumentation](./auto-instrumentation) -- Instrument LLM calls in your own code


--------------------------------------------------------------------------------
# LLM Call Tracking
URL: https://waxell.ai/docs/observe/features/llm-tracking
Description: Track every LLM API call with model, token counts, cost, and prompt/response previews.
--------------------------------------------------------------------------------

# LLM Call Tracking

Waxell Observe records every LLM API call made by your agents, capturing the model, token counts, estimated cost, and optional previews of prompts and responses. This data powers dashboards, cost analysis, and governance enforcement.

## What Data is Captured

Each LLM call record includes:

| Field | Type | Description |
|-------|------|-------------|
| `model` | `str` | Model identifier (e.g., `"gpt-4o"`, `"claude-sonnet-4"`) |
| `tokens_in` | `int` | Number of input/prompt tokens |
| `tokens_out` | `int` | Number of output/completion tokens |
| `cost` | `float` | Cost in USD (auto-estimated if not provided) |
| `task` | `str` | Optional label describing the call's purpose |
| `prompt_preview` | `str` | Optional preview of the prompt text |
| `response_preview` | `str` | Optional preview of the response text |

## How to Capture LLM Calls

### Recommended: Auto-Instrumentation with `@observe`

Call `waxell.init()` **before** importing your LLM SDK. This patches the SDK so every call is captured automatically with model, tokens, cost, latency, and previews -- no manual recording needed. Wrap your agent function with `@observe` to group calls into a tracked run.

```python


waxell.init()  # patches LLM SDKs -- call BEFORE importing them


client = openai.OpenAI()

@waxell.observe(agent_name="my-agent")
async def answer(query: str) -> str:
    response = client.chat.completions.create(  # auto-captured
        model="gpt-4o",
        messages=[{"role": "user", "content": query}],
    )
    return response.choices[0].message.content
```

Auto-instrumentation supports OpenAI, Anthropic, LiteLLM, Groq, Mistral, Together, Cohere, Bedrock, Vertex AI, Gemini, and 190+ other libraries. See [Auto-Instrumentation](../integrations/auto-instrumentation) for the full list.

### LangChain

For LangChain pipelines, `init()` instruments LangChain LLM calls too. You can also use the explicit callback handler if you need finer control:

```python
from waxell_observe.integrations.langchain import WaxellLangChainHandler

handler = WaxellLangChainHandler(agent_name="my-agent")
result = chain.invoke(input, config={"callbacks": [handler]})
handler.flush_sync(result={"output": result})
```

## Advanced: Manual Recording for Unsupported Providers

If you're calling an LLM API that isn't in the [supported provider list](../integrations/auto-instrumentation) (a custom HTTP endpoint, a niche provider, or a model proxy you've built), you can record the call manually with `WaxellContext`:

```python
from waxell_observe import WaxellContext

async with WaxellContext(agent_name="my-agent") as ctx:
    # Call your custom/unsupported LLM endpoint
    response = await custom_llm_client.complete(
        model="my-custom-model",
        prompt=query,
    )

    # Manually record the call
    ctx.record_llm_call(
        model="my-custom-model",
        tokens_in=response.usage.prompt_tokens,
        tokens_out=response.usage.completion_tokens,
        task="answer_question",
        prompt_preview=query[:500],
        response_preview=response.text[:500],
    )
```

For everything in the supported list, prefer the `@observe` pattern above -- it's less code and harder to get wrong.

## Supported Models

Cost estimation is built in for the following models. For unlisted models, provide the `cost` parameter manually or configure a tenant override on the server (see [Cost Management](./cost-management)).

### OpenAI

| Model | Input (per 1M tokens) | Output (per 1M tokens) |
|-------|----------------------|------------------------|
| `gpt-4o` | $2.50 | $10.00 |
| `gpt-4o-mini` | $0.15 | $0.60 |
| `gpt-4-turbo` | $10.00 | $30.00 |
| `gpt-4` | $30.00 | $60.00 |
| `gpt-3.5-turbo` | $0.50 | $1.50 |
| `o1` | $15.00 | $60.00 |
| `o1-mini` | $3.00 | $12.00 |
| `o3-mini` | $1.10 | $4.40 |

### Anthropic

| Model | Input (per 1M tokens) | Output (per 1M tokens) |
|-------|----------------------|------------------------|
| `claude-opus-4` | $15.00 | $75.00 |
| `claude-sonnet-4` | $3.00 | $15.00 |
| `claude-3-5-sonnet` | $3.00 | $15.00 |
| `claude-3-5-haiku` | $0.80 | $4.00 |
| `claude-3-haiku` | $0.25 | $1.25 |

### Google

| Model | Input (per 1M tokens) | Output (per 1M tokens) |
|-------|----------------------|------------------------|
| `gemini-2.0-flash` | $0.10 | $0.40 |
| `gemini-1.5-pro` | $1.25 | $5.00 |
| `gemini-1.5-flash` | $0.075 | $0.30 |

### Meta (via Groq, Together, etc.)

| Model | Input (per 1M tokens) | Output (per 1M tokens) |
|-------|----------------------|------------------------|
| `llama-3.3-70b` | $0.59 | $0.79 |
| `llama-3.1-8b` | $0.05 | $0.08 |

### Mistral

| Model | Input (per 1M tokens) | Output (per 1M tokens) |
|-------|----------------------|------------------------|
| `mistral-large` | $2.00 | $6.00 |

## Model Name Matching

The cost estimator uses prefix matching, so versioned model names work automatically:

- `"gpt-4o-2024-08-06"` matches `"gpt-4o"`
- `"claude-3-5-sonnet-20241022"` matches `"claude-3-5-sonnet"`

Longer prefixes are matched first for specificity, so `"gpt-4o-mini"` matches before `"gpt-4o"`.

If no match is found, cost is `0.0`. You can provide the cost explicitly or set up a server-side tenant override.

## How Data Flows

1. **Your agent** makes an LLM call (auto-captured by `init()` or recorded manually for unsupported providers)
2. **Calls are buffered** in memory during execution
3. **On flush/context exit**, calls are sent to the control plane via `POST /api/v1/observe/runs/{run_id}/llm-calls/`
4. **The control plane** stores the data, recalculates costs if server-side pricing is available, and updates dashboards
5. **Dashboards** show per-agent, per-model, and per-run cost breakdowns

**INFO**
Server-side cost calculation takes precedence over client-side estimates. If you configure tenant-level model cost overrides on the control plane, those prices are used instead of the client-side `MODEL_COSTS` table.


## Next Steps

- [Cost Management](./cost-management) -- Budget enforcement and tenant overrides
- [Decorator Pattern](../integrations/decorator) -- Quick integration with `@waxell_agent`
- [Auto-Instrumentation](../integrations/auto-instrumentation) -- Automatic capture for LangChain and other frameworks


--------------------------------------------------------------------------------
# Provider Routing — Use Waxell Routing Without the Runtime
URL: https://waxell.ai/docs/observe/features/provider-routing
Description: Dispatch LLM calls through Waxell with cross-provider fallback, capability filtering, and per-instance secret resolution — without adopting the Waxell runtime.
--------------------------------------------------------------------------------

# Provider Routing

Drop a one-liner into any existing agent and pick up:

- **Cross-provider fallback** — first try Fireworks, fall through to OpenAI on rate limit
- **Per-instance secrets** — each provider account uses its own env var, no secret collisions
- **Capability filtering** — automatically skip non-tools instances when a call needs `tools=[]`
- **Group references** — point at `"group:cheap-llama-70b"` and let the chain decide
- **Per-call telemetry** — same provider + cost attribution that Waxell runtime users get

You don't need the Waxell runtime to use any of this. Configure your
provider instances at `/settings/llm-routing` in the controlplane;
your agent code calls `waxell.llm.call(...)` and dispatch is driven
by the same data.

## Quick start

```bash
pip install 'waxell-observe[all-providers]'
```

Configure your API keys in env (the same names you'd otherwise pass to
the SDKs directly):

```bash


# Whatever secret_ref names you set in /settings/llm-routing
```

Then in code:

```python


waxell.init()  # reads WAXELL_API_KEY + WAXELL_API_URL from env

response = waxell.llm.call(
    model="llama-3.1-70b-instruct",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)
```

That's it. No runtime, no Django, no servers. The SDK pulls your
provider config from the controlplane on first call (cached 5 minutes
with ETag), resolves the model, dispatches through the right SDK, and
emits the same telemetry runtime users get.

## How it works

```
your code                                  waxell-observe              controlplane
  |                                            |                          |
  | waxell.llm.call(model="...", ...)          |                          |
  |------------------------------------------> |                          |
  |                                            | GET /llm-config/manifest |
  |                                            |------------------------> |
  |                                            | <----------- 200 / 304 - |
  |                                            | resolve_chain(model)     |
  |                                            | filter_chain_for_mode()  |
  |                                            | os.environ[secret_ref]   |
  |                                            | openai.OpenAI(base_url=) |
  |                                            | .chat.completions.create |
  | <---- SDK response object (no wrapper) --- |                          |
```

The SDK's `Manifest` is a snapshot of:

- **`instances`** — your registered provider accounts (kind, base_url, secret_ref name, capabilities)
- **`tenant_models`** — `model_id → instance_id` mappings (e.g. `llama-3.1-70b-instruct` lives on `fireworks-prod`)
- **`groups`** — ordered fallback chains (`cheap-llama-70b` → fireworks first, ollama fallback)
- **`capability_overrides`** — per-(instance, model) tri-state flag pins

Resolved at dispatch time; cached by ETag.

## Resolving a model

`waxell.llm.call(model=...)` accepts three shapes:

| Shape | Example | What happens |
|---|---|---|
| Plain | `"gpt-4o"` | Look up TenantModel; if missing, use the default instance for the prefix-inferred kind. |
| Qualified | `"fireworks-prod/llama-3.1-70b-instruct"` | Use that exact instance with that exact model name. |
| Group | `"group:cheap-llama-70b"` | Walk the group's entries in declared order. |

## Mode-specific helpers

```python
# Plain chat (default)
waxell.llm.text(model="gpt-4o", messages=[...])

# JSON mode — adds response_format={"type": "json_object"}
waxell.llm.json(model="gpt-4o", messages=[...])

# JSON with schema
waxell.llm.json(model="gpt-4o", messages=[...], schema={...})

# Tool calls
waxell.llm.tool(
    model="gpt-4o",
    messages=[...],
    tools=[{"type": "function", "function": {...}}],
)
```

The capability filter drops chain entries whose instances don't
advertise the required capability. If you ask for `tool` mode against
a chain whose only candidate is Ollama (no native tools), you'll get
`NoCandidateForMode` rather than a confusing OpenAI error from the
provider.

## Secrets — the contract

Provider instances reference an **env var name**, not a secret value.
The controlplane stores the name (`secret_ref`); your process reads
the value from `os.environ[secret_ref]` at dispatch time.

When the env var is unset:

```
SecretNotInEnvironment: Provider instance 'fireworks-prod' references
env var FIREWORKS_API_KEY which is not set in this process. Either set
the env var, or change the instance's secret_ref in the controlplane
at /settings/llm-routing.
```

This intentionally mirrors how you'd already provide keys to the
direct SDK — the dispatcher is opt-in, not magical. Keys never leave
your process.

## Groups for cross-provider fallback

Define a group at `/settings/llm-routing` (or via the API), then
reference it as `model="group:..."`:

```python
# Group "cheap-llama-70b" defined as:
#   1. fireworks-prod / accounts/fireworks/models/llama-v3p1-70b-instruct
#   2. groq-prod      / llama-3.1-70b-versatile
#   3. ollama-local   / llama3.1:70b

response = waxell.llm.call(
    model="group:cheap-llama-70b",
    messages=[...],
)
# If Fireworks rate-limits, dispatch retries against Groq.
# If Groq is down, falls through to local Ollama.
# Capability filter drops Ollama if you passed tools=[].
```

Fallback walks on retryable errors only (rate limits, 5xx, connection
errors, NotFound). Auth errors, BadRequest, and unknown errors raise
immediately — they're caller bugs, not provider hiccups.

## Capability overrides

Sometimes a model that "should" support tools doesn't on a specific
provider. Override per-(instance, model) at `/settings/llm-routing`:

```
override:
  instance: together-prod
  model: llama-3.1-70b-instruct
  native_tools: false   # tri-state: true / false / null
```

The dispatcher's capability filter respects overrides as veto: an
explicit `false` skips this candidate even if its instance baseline
says `true`. `null` means "no override; defer to baseline."

## Inspect what would happen

```bash
wax llm call --model gpt-4o --show-config
```

Prints the resolved chain, which entries pass the capability filter,
whether the env var is set, and the candidate's base_url — without
actually dispatching.

```
Resolved chain for 'gpt-4o' (mode=chat)
┌──────────────────────────────────────────────────────────────────┐
│ # │ instance_id │ kind   │ base_url        │ env set? │ passes?  │
├──────────────────────────────────────────────────────────────────┤
│ 1 │ oai-prod    │ openai │ (SDK default)   │ ✓        │ ✓        │
└──────────────────────────────────────────────────────────────────┘
```

Useful for:

- Debugging "why is this routing to provider X?"
- Verifying `secret_ref` env vars are set before running batch jobs
- Confirming capability filter behavior under tool / JSON modes

## What gets recorded

Every `waxell.llm.call(...)` produces one `LlmCallRecord` in your
controlplane with:

- `provider` (e.g. `"openai_compat"`)
- `provider_instance_id` (e.g. `"fireworks-prod"`)
- `model` (the resolved provider model id)
- `tokens_in`, `tokens_out`, `cost`
- `dispatch_source: "observe-sdk"` (for analytics distinguishing
  observe-side dispatch from runtime dispatch)
- Plus any `fallback_chain` metadata if the call walked past a
  retryable error before succeeding

The same `LlmCallRecord` ingest path stamps `last_success_at` on the
provider instance — your "is this Fireworks instance healthy?"
analytics work the same whether the call went through observe SDK
dispatch or the Waxell runtime.

## Coexistence with raw SDK calls

Your existing direct SDK calls — `openai.chat.completions.create(...)`,
`anthropic.messages.create(...)` — keep working unchanged. The
auto-instrumentor records them as before. The dispatcher sets a
context-var around its own SDK call so the instrumentor doesn't
double-record when both code paths run.

You can mix freely: dispatch when you want fallback / groups, direct
SDK calls everywhere else. The tracing and cost attribution are
unified.

## Provider extras

Install only the SDKs you need:

```bash
pip install 'waxell-observe[openai]'      # OpenAI + all OpenAI-compat
pip install 'waxell-observe[anthropic]'   # Anthropic
pip install 'waxell-observe[fireworks]'   # alias for [openai]
pip install 'waxell-observe[together]'    # alias for [openai]
pip install 'waxell-observe[xai]'         # alias for [openai]
pip install 'waxell-observe[bedrock]'     # boto3
pip install 'waxell-observe[vertex]'      # google-cloud-aiplatform
pip install 'waxell-observe[gemini]'      # google-generativeai
pip install 'waxell-observe[cohere]'      # cohere
pip install 'waxell-observe[mistral]'     # mistralai
pip install 'waxell-observe[groq]'        # groq

# Or install the whole thing
pip install 'waxell-observe[all-providers]'
```

Most "OpenAI-compatible" providers (Fireworks, Together, Groq, xAI,
NVIDIA, Mistral, AI21, Replicate, Ollama, vLLM, HF TGI) work with just
the `openai` SDK because they speak the OpenAI HTTP wire format on a
different `base_url`. The dispatcher handles the `base_url` override
transparently.

## When to use this vs the runtime

| Use observe SDK dispatch (this) when… | Use the Waxell runtime when… |
|---|---|
| You have an existing agent (LangGraph, LangChain, ad-hoc) | You're building a new agent from scratch |
| You want fallback + groups but not durable execution | You need durable execution + replay |
| You want to add Waxell to a serverless function | You want supervised + governed agent fleets |
| You want minimal migration cost | You want the full governance surface |

The data layer is shared — the same provider instances, groups, and
capability overrides feed both adoption modes. You can start with
observe SDK dispatch and graduate to the runtime later without
re-configuring providers.

## Reference

- [`agentforge/areas/llm-providers/plans/OBSERVE_DISPATCH_PLAN.md`](https://github.com/waxell-ai/waxell/blob/main/agentforge/areas/llm-providers/plans/OBSERVE_DISPATCH_PLAN.md) — design plan
- [Provider Catalog](/docs/observe/features/llm-tracking) — list of supported provider kinds
- Controlplane UI: `/settings/llm-routing` (configure your instances)
- Controlplane UI: `/admin/llm-providers/` (cross-tenant admin, requires `billing:admin`)


--------------------------------------------------------------------------------
# Sessions
URL: https://waxell.ai/docs/observe/features/sessions
Description: Group related agent runs into sessions for multi-turn conversation tracking and aggregate analysis.
--------------------------------------------------------------------------------

# Sessions

Sessions group related agent runs under a single identifier. A common use case is tracking multi-turn conversations where each user message triggers a separate agent run, but you want to analyze the entire conversation as a unit.

## What Gets Tracked

When runs share a `session_id`, Waxell Observe automatically aggregates:

| Metric | Description |
|--------|-------------|
| `run_count` | Number of runs in the session |
| `first_run` | Timestamp of the earliest run |
| `last_activity` | Timestamp of the most recent run |
| `total_duration` | Combined execution time across all runs (seconds) |
| `total_cost` | Sum of LLM costs across all runs (USD) |
| `total_tokens` | Sum of tokens used across all runs |
| `agents` | List of distinct agent names that participated |

## Setting a Session ID

### Recommended: `@observe` with Auto-Instrumentation

Call `waxell.init()` before importing your LLM SDK and pass `session_id` at call time. Every LLM call inside the decorated function is auto-captured and attributed to the session:

```python


waxell.init()  # BEFORE importing the LLM SDK


client = openai.OpenAI()

@waxell.observe(agent_name="chat-agent")
async def handle_message(user_message: str) -> str:
    response = client.chat.completions.create(  # auto-captured
        model="gpt-4o",
        messages=[{"role": "user", "content": user_message}],
    )
    return response.choices[0].message.content

# Pass session_id at call time -- groups every call with this id together
await handle_message("Hello!", session_id="session-abc123", user_id="user_456")
```

You can also bake `session_id` into the decorator if it's static for that agent: `@waxell.observe(agent_name="chat-agent", session_id="session-abc123")`. For real apps, call-time is more common since the session is per-conversation.

### Dynamic Session IDs

Derive the session ID from your application's conversation or request context. Use `generate_session_id()` to mint one, or use your own identifier:

```python

from waxell_observe import generate_session_id

# Generate a new session ID (format: sess_ + 16 hex chars)
session_id = generate_session_id()
# e.g. "sess_a1b2c3d4e5f60718"

# Or use your own identifier
session_id = f"conv-{conversation.id}"

# All runs with this session_id are grouped together
await handle_message("Hi", session_id=session_id, user_id=f"user-{user.id}")
await handle_message("Follow up", session_id=session_id, user_id=f"user-{user.id}")
```

**INFO**
The `session_id` is propagated to both the HTTP data path (stored on the `AgentExecutionRun` record) and the OTel tracing path (as a `waxell.session_id` span attribute). This means sessions are queryable from both the Waxell UI and Grafana TraceQL.


## REST API

### List Sessions

```
GET /api/v1/observability/sessions/
```

**Authentication:** Session (UI)

**Query Parameters:**

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `search` | `string` | | Filter by session_id (substring match) |
| `agent` | `string` | | Filter by agent name |
| `start` | `ISO8601` | | Only sessions with runs after this time |
| `end` | `ISO8601` | | Only sessions with runs before this time |
| `sort` | `string` | `-last_activity` | Sort field. Options: `last_activity`, `-last_activity`, `first_run`, `-first_run`, `run_count`, `-run_count` |
| `limit` | `int` | `25` | Page size (max 100) |
| `offset` | `int` | `0` | Pagination offset |

**Example:**

```bash
curl -s "https://acme.waxell.dev/api/v1/observability/sessions/?limit=10&sort=-run_count" \
  -H "Cookie: sessionid=..."
```

**Response:**

```json
{
  "results": [
    {
      "session_id": "sess_a1b2c3d4e5f6g7h8",
      "run_count": 5,
      "first_run": "2026-02-07T10:00:00Z",
      "last_activity": "2026-02-07T10:05:32Z",
      "total_duration": 12.45,
      "total_cost": 0.0234,
      "total_tokens": 4520,
      "agents": ["chat-agent", "retrieval-agent"]
    }
  ],
  "count": 42,
  "next": "?offset=10&limit=10",
  "previous": null
}
```

### Get Session Detail

```
GET /api/v1/observability/sessions/{session_id}/
```

**Authentication:** Session (UI)

Returns the session's aggregate metrics and a chronological list of all runs.

**Example:**

```bash
curl -s "https://acme.waxell.dev/api/v1/observability/sessions/sess_a1b2c3d4e5f6g7h8/" \
  -H "Cookie: sessionid=..."
```

**Response:**

```json
{
  "session_id": "sess_a1b2c3d4e5f6g7h8",
  "aggregates": {
    "run_count": 5,
    "total_duration": 12.45,
    "total_cost": 0.0234,
    "total_tokens": 4520,
    "agents": ["chat-agent", "retrieval-agent"]
  },
  "runs": [
    {
      "id": 101,
      "agent_name": "chat-agent",
      "workflow_name": "default",
      "started_at": "2026-02-07T10:00:00Z",
      "completed_at": "2026-02-07T10:00:02Z",
      "duration": 2.1,
      "status": "success",
      "cost": 0.0045,
      "tokens": 890,
      "trace_id": "abcdef1234567890abcdef1234567890"
    }
  ]
}
```

## UI Walkthrough

### Sessions List

The sessions list view shows all tracked sessions with sortable columns:

- **Session ID** -- click to open session detail
- **Runs** -- number of agent runs in the session
- **First Run / Last Activity** -- time range of the session
- **Duration** -- total execution time
- **Cost** -- aggregated LLM spend
- **Tokens** -- total token usage
- **Agents** -- which agents participated

Use the search bar to filter by session ID, or the agent dropdown to see sessions for a specific agent.

### Session Detail

The session detail page shows:

1. **Summary cards** at the top with run count, total duration, total cost, and total tokens
2. **Vertical timeline** of runs in chronological order, showing each run's agent, duration, cost, and status
3. Click any run to navigate to its full trace detail

## Multi-Agent Sessions

Sessions are especially useful for multi-agent workflows where several agents collaborate on a single request. Decorate each agent with `@observe` and pass the same `session_id` at call time:

```python
@waxell.observe(agent_name="router")
async def classify(query: str) -> str:
    ...

@waxell.observe(agent_name="retrieval")
async def search(query: str) -> list:
    ...

@waxell.observe(agent_name="synthesizer")
async def synthesize(query: str, docs: list) -> str:
    ...

session_id = generate_session_id()
intent = await classify(query, session_id=session_id)
docs = await search(query, session_id=session_id)
answer = await synthesize(query, docs, session_id=session_id)
```

All three runs appear under the same session, giving you a complete picture of the multi-agent pipeline.

## Advanced: Multiple Runs Per Function with `WaxellContext`

If a single function needs to spawn multiple distinct runs (e.g., a batch loop processing items, where each item should be its own run with the same session), use `WaxellContext` directly:

```python
from waxell_observe import WaxellContext, generate_session_id

session_id = generate_session_id()

for item in batch:
    async with WaxellContext(
        agent_name="batch-processor",
        session_id=session_id,
        user_id="user_456",
    ) as ctx:
        result = await process(item)  # auto-captured LLM calls go here
        ctx.set_result({"output": result})
```

For the common case of one function = one run, the decorator is simpler and recommended.

## Next Steps

- [User Tracking](./user-tracking) -- Attribute sessions and costs to individual users
- [Scoring](./scoring) -- Attach quality scores to runs within a session
- [LLM Call Tracking](./llm-tracking) -- Understand token and cost breakdown per run


--------------------------------------------------------------------------------
# User Tracking
URL: https://waxell.ai/docs/observe/features/user-tracking
Description: Track per-user costs, usage patterns, and agent interactions with opaque user identifiers.
--------------------------------------------------------------------------------

# User Tracking

User tracking lets you attribute agent runs, LLM costs, and token usage to individual end users of your application. This enables per-user cost analysis, abuse detection, and usage pattern insights.

## What Gets Tracked

When runs include a `user_id`, Waxell Observe aggregates per-user metrics:

| Metric | Description |
|--------|-------------|
| `run_count` | Total runs by this user |
| `first_seen` | Timestamp of the user's first run |
| `last_seen` | Timestamp of the user's most recent run |
| `total_duration` | Combined execution time across all user runs (seconds) |
| `total_cost` | Sum of LLM costs for this user (USD) |
| `total_tokens` | Sum of tokens consumed by this user |
| `agents` | List of distinct agents this user has interacted with |
| `cost_by_model` | Cost and token breakdown per LLM model |

## Setting a User ID

### Recommended: `@observe` with Auto-Instrumentation

Call `waxell.init()` before importing your LLM SDK and pass `user_id` at call time. Every LLM call inside the decorated function is auto-captured and attributed to that user:

```python


waxell.init()  # BEFORE importing the LLM SDK


client = openai.OpenAI()

@waxell.observe(agent_name="support-agent")
async def handle_ticket(ticket_id: str) -> str:
    response = client.chat.completions.create(  # auto-captured
        model="gpt-4o",
        messages=[{"role": "user", "content": f"Resolve ticket {ticket_id}"}],
    )
    return response.choices[0].message.content

# Pass user_id at call time -- attributes cost and runs to this user
await handle_ticket("ticket-001", user_id=f"user-{request.user.id}")
```

### Combined Session and User Tracking

In most applications, you pass both `session_id` and `user_id` at call time:

```python
@waxell.observe(agent_name="chat-agent")
async def generate_reply(message: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": message}],
    )
    return response.choices[0].message.content

await generate_reply(
    message,
    session_id=f"conv-{conversation.id}",
    user_id=f"user-{request.user.id}",
)
```

**WARNING**
**Privacy best practice:** Use opaque internal identifiers (database IDs, UUIDs) as `user_id` values. Do not pass email addresses, names, or other personally identifiable information. The `user_id` field is stored in plain text and is visible in the Waxell UI and API responses.


## REST API

### List Users

```
GET /api/v1/observability/users/
```

**Authentication:** Session (UI)

**Query Parameters:**

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `search` | `string` | | Filter by user_id (substring match) |
| `agent` | `string` | | Filter by agent name |
| `start` | `ISO8601` | | Only users with runs after this time |
| `end` | `ISO8601` | | Only users with runs before this time |
| `sort` | `string` | `-last_seen` | Sort field. Options: `last_seen`, `-last_seen`, `first_seen`, `-first_seen`, `run_count`, `-run_count` |
| `limit` | `int` | `25` | Page size (max 100) |
| `offset` | `int` | `0` | Pagination offset |

**Example:**

```bash
curl -s "https://acme.waxell.dev/api/v1/observability/users/?sort=-run_count&limit=10" \
  -H "Cookie: sessionid=..."
```

**Response:**

```json
{
  "results": [
    {
      "user_id": "user-456",
      "run_count": 87,
      "first_seen": "2026-01-15T08:30:00Z",
      "last_seen": "2026-02-07T14:22:00Z",
      "total_duration": 245.8,
      "total_cost": 1.234567,
      "total_tokens": 189420,
      "agents": ["chat-agent", "search-agent", "code-agent"]
    }
  ],
  "count": 156,
  "next": "?offset=10&limit=10",
  "previous": null
}
```

### Get User Detail

```
GET /api/v1/observability/users/{user_id}/
```

**Authentication:** Session (UI)

Returns detailed metrics for a specific user, including a per-model cost breakdown and recent runs.

**Example:**

```bash
curl -s "https://acme.waxell.dev/api/v1/observability/users/user-456/" \
  -H "Cookie: sessionid=..."
```

**Response:**

```json
{
  "user_id": "user-456",
  "aggregates": {
    "run_count": 87,
    "total_duration": 245.8,
    "total_cost": 1.234567,
    "total_tokens": 189420,
    "agents": ["chat-agent", "search-agent"],
    "first_seen": "2026-01-15T08:30:00Z",
    "last_seen": "2026-02-07T14:22:00Z"
  },
  "cost_by_model": [
    {
      "model": "gpt-4o",
      "total_cost": 0.987654,
      "total_tokens": 142000,
      "call_count": 64
    },
    {
      "model": "gpt-4o-mini",
      "total_cost": 0.246913,
      "total_tokens": 47420,
      "call_count": 23
    }
  ],
  "runs": [
    {
      "id": 1042,
      "agent_name": "chat-agent",
      "workflow_name": "default",
      "started_at": "2026-02-07T14:22:00Z",
      "completed_at": "2026-02-07T14:22:03Z",
      "duration": 2.8,
      "status": "success",
      "cost": 0.0089,
      "tokens": 1250
    }
  ]
}
```

## UI Walkthrough

### Users List

The users list view shows all tracked users with sortable columns:

- **User ID** -- click to open user detail
- **Runs** -- total number of agent executions
- **First Seen / Last Seen** -- user activity time range
- **Duration** -- total execution time
- **Cost** -- total LLM spend
- **Tokens** -- total token usage
- **Agents** -- which agents this user interacted with

### User Detail

The user detail page shows:

1. **Summary cards** with run count, total cost, total tokens, and active time range
2. **Cost by model** breakdown -- a table showing which models drove the user's costs
3. **Recent runs** -- the last 50 runs for this user with per-run cost, tokens, duration, and status

## Use Cases

### Cost Attribution

Identify your highest-cost users to understand whether spend is proportional to value:

```bash
curl -s "https://acme.waxell.dev/api/v1/observability/users/?sort=-total_cost&limit=5" \
  -H "Cookie: sessionid=..."
```

### Abuse Detection

Flag users with unusually high run counts or token consumption. The `cost_by_model` breakdown on the detail endpoint reveals whether a user is making disproportionately expensive model calls.

### Usage Patterns

Track `first_seen` and `last_seen` to understand user retention and engagement patterns. The `agents` list shows which product features each user exercises.

## Advanced: Multiple Runs Per Function with `WaxellContext`

If you need to create multiple distinct runs from a single function (e.g., a batch processor that handles items for many users in one invocation), use `WaxellContext` directly:

```python
from waxell_observe import WaxellContext

for item in batch:
    async with WaxellContext(
        agent_name="batch-processor",
        user_id=item.user_id,
        session_id=item.session_id,
    ) as ctx:
        result = await process(item)  # auto-captured LLM calls go here
        ctx.set_result({"output": result})
```

For the common case of one function = one run, prefer the decorator pattern above.

## Next Steps

- [Sessions](./sessions) -- Group runs by conversation for multi-turn analysis
- [Scoring](./scoring) -- Capture user satisfaction alongside usage data
- [Cost Management](./cost-management) -- Set budget limits and alerts based on user spend


--------------------------------------------------------------------------------
# Cost Management
URL: https://waxell.ai/docs/observe/features/cost-management
Description: Track, estimate, and control LLM costs with client-side estimation, server-side calculation, and tenant-level overrides.
--------------------------------------------------------------------------------

# Cost Management

Waxell Observe provides a layered cost management system: client-side estimation for immediate visibility, server-side calculation for accuracy, and tenant-level overrides for custom pricing.

## How Cost Estimation Works

### Client-Side: estimate_cost()

The `estimate_cost` function provides instant cost estimates based on built-in pricing data:

```python
from waxell_observe.cost import estimate_cost

cost = estimate_cost("gpt-4o", tokens_in=1000, tokens_out=500)
print(f"Estimated cost: ${cost:.6f}")  # $0.007500
```

**Parameters:**

| Parameter | Type | Description |
|-----------|------|-------------|
| `model` | `str` | Model name or prefix |
| `tokens_in` | `int` | Input/prompt token count |
| `tokens_out` | `int` | Output/completion token count |

**Returns:** `float` -- estimated cost in USD. Returns `0.0` for unknown models.

The function uses a two-step matching strategy:
1. **Exact match** -- looks for the model name in the pricing table
2. **Prefix match** -- tries longer prefixes first for versioned model names (e.g., `"gpt-4o-2024-08-06"` matches `"gpt-4o"`)

### Automatic Cost Estimation

When you record an LLM call without specifying a cost, the client automatically estimates it:

```python
# Cost is auto-estimated
ctx.record_llm_call(model="gpt-4o", tokens_in=1000, tokens_out=500)

# Or provide an explicit cost to override estimation
ctx.record_llm_call(model="gpt-4o", tokens_in=1000, tokens_out=500, cost=0.0085)
```

The LangChain handler always uses automatic estimation -- no manual cost input is needed.

### Server-Side Calculation

The control plane maintains its own model pricing database that can differ from client-side estimates. When the server processes LLM call records, it can recalculate costs using:

- **System defaults** -- baseline pricing maintained by the platform
- **Tenant overrides** -- custom pricing set by your organization

Server-side costs take precedence over client-side estimates in dashboards and reports.

## Built-In Model Pricing

The client includes pricing for 20+ models across major providers:

| Provider | Models |
|----------|--------|
| **OpenAI** | gpt-4o, gpt-4o-mini, gpt-4-turbo, gpt-4, gpt-3.5-turbo, o1, o1-mini, o3-mini |
| **Anthropic** | claude-opus-4, claude-sonnet-4, claude-3-5-sonnet, claude-3-5-haiku, claude-3-haiku |
| **Google** | gemini-2.0-flash, gemini-1.5-pro, gemini-1.5-flash |
| **Meta** | llama-3.3-70b, llama-3.1-8b |
| **Mistral** | mistral-large |

For the full pricing table with per-token costs, see [LLM Call Tracking](./llm-tracking).

## Tenant-Level Cost Overrides

If your organization has negotiated pricing or uses a provider with different rates, you can set custom costs per model via the REST API.

### Set a Custom Cost

```bash
curl -X PUT "https://acme.waxell.dev/api/v1/observe/model-costs/gpt-4o/" \
  -H "X-Wax-Key: wax_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "input_cost_per_million": 2.00,
    "output_cost_per_million": 8.00
  }'
```

### View Current Costs

Retrieve the merged system + tenant cost table:

```bash
curl "https://acme.waxell.dev/api/v1/observe/model-costs/" \
  -H "X-Wax-Key: wax_sk_..."
```

This returns all model costs, with tenant overrides applied on top of system defaults.

### Remove a Custom Cost

Delete a tenant override to revert to system defaults:

```bash
curl -X DELETE "https://acme.waxell.dev/api/v1/observe/model-costs/gpt-4o/" \
  -H "X-Wax-Key: wax_sk_..."
```

## Budget Enforcement

Cost management integrates with the [policy and governance](./governance) system. You can configure policies on the control plane that:

- **Block execution** when daily or monthly spend exceeds a threshold
- **Warn** when approaching budget limits
- **Throttle** execution rate when costs are high

These policies are evaluated during the `check_policy` call that occurs before agent execution (when `enforce_policy=True`).

Example flow:
1. Agent attempts to run with `enforce_policy=True`
2. Control plane evaluates cost-based policies
3. If daily token spend exceeds the configured limit, the policy returns `action: "block"` with a reason like `"Daily token budget exceeded"`
4. A `PolicyViolationError` is raised and execution does not proceed

```python
from waxell_observe import waxell_agent
from waxell_observe.errors import PolicyViolationError

@waxell_agent(agent_name="expensive-agent", enforce_policy=True)
async def run_expensive_task(query: str) -> str:
    ...

try:
    result = await run_expensive_task("analyze everything")
except PolicyViolationError as e:
    print(f"Budget exceeded: {e}")
    # e.policy_result.metadata may contain budget details
```

## Cost Tracking Workflow

```
1. Agent makes LLM call
   |
2. Client estimates cost (MODEL_COSTS table)
   |
3. LLM call record sent to control plane
   |
4. Server recalculates with tenant overrides (if any)
   |
5. Cost aggregated in dashboards
   |
6. Budget policies evaluated on next agent run
```

**TIP**
Even if you do not set up tenant overrides, the built-in client-side estimates provide useful cost visibility from day one. You can refine pricing later without changing any agent code.


## Next Steps

- [LLM Call Tracking](./llm-tracking) -- Full model pricing table and capture details
- [Policy & Governance](./governance) -- Budget enforcement and policy configuration
- [REST API Reference](../api/endpoints) -- Model cost API endpoints


--------------------------------------------------------------------------------
# Scoring
URL: https://waxell.ai/docs/observe/features/scoring
Description: Attach quality scores to agent runs using numeric, categorical, or boolean values from SDK or UI.
--------------------------------------------------------------------------------

# Scoring

Scores let you attach quality measurements to agent runs. Use them to capture user feedback and track quality over time.

## Score Data Types

Each score has a `data_type` that determines how its value is stored and analyzed:

| Data Type | Value | Storage | Example |
|-----------|-------|---------|---------|
| `numeric` | `float` (typically 0-1) | `numeric_value` | `0.85` (relevance score) |
| `categorical` | `string` | `string_value` | `"good"`, `"bad"`, `"neutral"` |
| `boolean` | `bool` | Both fields | `true` (thumbs up) |

For boolean scores, the value is stored as both `numeric_value` (1.0 for true, 0.0 for false) and `string_value` (`"true"` or `"false"`), allowing both numeric aggregation and categorical filtering.

## Score Sources

Scores are tagged with a `source` indicating how they were created:

| Source | Description |
|--------|-------------|
| `sdk` | Recorded programmatically via the SDK during or after execution |
| `manual` | Created through the UI (annotation workflows, manual review) |
| `evaluator` | Generated by an automated evaluator (LLM-as-judge) |

## Recording Scores via SDK

### Recommended: `waxell.score()` Inside `@observe`

Call `waxell.score()` from inside a function decorated with `@observe`. It attaches the score to the current run -- no context object required. It's a no-op outside an active run, so it's safe to leave in code paths that may not be wrapped.

```python


waxell.init()


client = openai.OpenAI()

@waxell.observe(agent_name="support-agent")
async def handle_query(query: str) -> str:
    response = client.chat.completions.create(  # auto-captured
        model="gpt-4o",
        messages=[{"role": "user", "content": query}],
    )
    answer = response.choices[0].message.content

    # Numeric score (0-1 range)
    waxell.score("relevance", 0.92, comment="High relevance to user query")

    # Categorical score
    waxell.score("tone", "professional", data_type="categorical")

    # Boolean score (user feedback)
    waxell.score("thumbs_up", True, data_type="boolean")

    return answer
```

The `waxell.score()` signature:

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `name` | `str` | *required* | Score name (e.g., `"relevance"`, `"thumbs_up"`) |
| `value` | `float \| str \| bool` | *required* | Score value |
| `data_type` | `str` | `"numeric"` | One of `"numeric"`, `"categorical"`, `"boolean"` |
| `comment` | `str` | `""` | Optional free-text annotation |

### After Execution (Client-Level)

If you need to add scores after the run context has closed, use the client directly:

```python
from waxell_observe import WaxellObserveClient

client = WaxellObserveClient(
    api_url="https://acme.waxell.dev",
    api_key="wax_sk_...",
)

# Record scores on an existing run
await client.record_scores(
    run_id="42",
    scores=[
        {
            "name": "user_feedback",
            "data_type": "numeric",
            "numeric_value": 1.0,
            "comment": "User rated 5 stars",
        },
        {
            "name": "category",
            "data_type": "categorical",
            "string_value": "helpful",
        },
    ],
)
```

Or synchronously:

```python
client.record_scores_sync(
    run_id="42",
    scores=[
        {
            "name": "user_feedback",
            "data_type": "numeric",
            "numeric_value": 1.0,
        },
    ],
)
```

**INFO**
Scores recorded via the SDK are sent to `POST /api/v1/observe/runs/{run_id}/scores/` using API key authentication (X-Wax-Key header). This is the same ingest path used for LLM calls and steps.


## REST API (UI Endpoints)

These endpoints are used by the Waxell dashboard and require session authentication.

### List Scores

```
GET /api/v1/evaluations/scores/
```

**Query Parameters:**

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `name` | `string` | | Filter by score name |
| `source` | `string` | | Filter by source (`sdk`, `manual`, `evaluator`) |
| `data_type` | `string` | | Filter by data type |
| `run_id` | `int` | | Filter by run ID |
| `llm_call_id` | `int` | | Filter by LLM call ID |
| `sort` | `string` | `-created_at` | Sort field. Options: `created_at`, `-created_at`, `name`, `-name`, `numeric_value`, `-numeric_value`, `source`, `-source` |
| `limit` | `int` | `25` | Page size (max 100) |
| `offset` | `int` | `0` | Pagination offset |

**Response:**

```json
{
  "results": [
    {
      "id": "a1b2c3d4-...",
      "name": "relevance",
      "data_type": "numeric",
      "source": "sdk",
      "numeric_value": 0.92,
      "string_value": null,
      "comment": "High relevance to user query",
      "metadata": {},
      "author_user_id": "",
      "evaluator_id": null,
      "evaluator_name": null,
      "run_id": "42",
      "run_agent_name": "support-agent",
      "llm_call_id": null,
      "llm_call_model": null,
      "created_at": "2026-02-07T10:15:00Z"
    }
  ],
  "count": 156,
  "next": "?offset=25&limit=25",
  "previous": null,
  "aggregates": {
    "total_count": 156,
    "avg_numeric_value": 0.7834,
    "score_names": ["relevance", "thumbs_up", "tone"]
  }
}
```

### Create a Manual Score

```
POST /api/v1/evaluations/scores/
```

**Request Body:**

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `run_id` | `int` | One of `run_id` or `llm_call_id` | Run to score |
| `llm_call_id` | `int` | One of `run_id` or `llm_call_id` | LLM call to score |
| `name` | `string` | Yes | Score name |
| `data_type` | `string` | No (default `"numeric"`) | `"numeric"`, `"categorical"`, or `"boolean"` |
| `value` | `any` | Yes | Score value |
| `comment` | `string` | No | Free-text annotation |
| `metadata` | `object` | No | Arbitrary JSON metadata |

**Example:**

```bash
curl -X POST "https://acme.waxell.dev/api/v1/evaluations/scores/" \
  -H "Cookie: sessionid=..." \
  -H "Content-Type: application/json" \
  -d '{
    "run_id": 42,
    "name": "accuracy",
    "data_type": "numeric",
    "value": 0.95,
    "comment": "Verified against ground truth"
  }'
```

### Score Analytics

```
GET /api/v1/evaluations/scores/analytics/
```

**Query Parameters:**

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `name` | `string` | | Filter by score name |
| `period` | `string` | `7d` | Time period: `1d`, `7d`, `30d` |
| `agent` | `string` | | Filter by agent name |

Returns score distributions per score name, including:
- **Numeric scores:** average, min, max, and daily time series
- **Categorical/boolean scores:** value counts

## Viewing Scores in the UI

### Run Detail View

On any run's detail page, scores are displayed in a dedicated section showing:
- Score name and value
- Data type badge (numeric, categorical, boolean)
- Source indicator (SDK, manual, or evaluator name)
- Timestamp and optional comment

### Score Analytics Dashboard

The analytics view (`/api/v1/evaluations/scores/analytics/`) powers distribution charts:
- **Numeric scores** show a time series of daily averages with min/max bands
- **Categorical scores** show value frequency bar charts
- Filter by score name, agent, or time period to drill down

## Capturing User Feedback

A common pattern is to capture end-user feedback (thumbs up/down, ratings) and record it as a score:

```python
# In your API endpoint that handles user feedback
from waxell_observe import WaxellObserveClient

client = WaxellObserveClient()

async def handle_feedback(run_id: str, rating: int):
    """Called when user rates an agent response."""
    await client.record_scores(
        run_id=run_id,
        scores=[
            {
                "name": "user_rating",
                "data_type": "numeric",
                "numeric_value": rating / 5.0,  # Normalize to 0-1
                "comment": f"User gave {rating}/5 stars",
            }
        ],
    )
```

## Advanced: `ctx.record_score()` with `WaxellContext`

If you're orchestrating runs explicitly with `WaxellContext` (batch loops, multi-run-per-function patterns), use `ctx.record_score()` on the context instance:

```python
from waxell_observe import WaxellContext

async with WaxellContext(agent_name="support-agent") as ctx:
    response = await handle_query(query)
    ctx.set_result({"output": response})

    ctx.record_score("relevance", 0.92, comment="High relevance")
    ctx.record_score("tone", "professional", data_type="categorical")
    ctx.record_score("thumbs_up", True, data_type="boolean")
```

For the common case of one function = one run, prefer `waxell.score()` inside `@observe` -- it's less ceremony and the same data ends up on the run.

## Next Steps

- [Sessions](./sessions) -- Analyze scores across multi-turn conversations
- [Cost Management](./cost-management) -- Track and control LLM spending


--------------------------------------------------------------------------------
# Prompt Management
URL: https://waxell.ai/docs/observe/features/prompt-management
Description: Version, label, and retrieve prompts with content hashing for production traceability and a playground for testing.
--------------------------------------------------------------------------------

# Prompt Management

Waxell Observe includes a full prompt management system. Version your prompts, assign deployment labels like "production" and "staging", retrieve them at runtime via the SDK, and test variants in the playground -- all with content hashing that links prompts to their LLM call traces.

## Core Concepts

### Versions

Each prompt has a sequential version history (v1, v2, v3, ...). When you update a prompt's content, a new version is created. Old versions are preserved for auditing and rollback.

### Labels

Labels are named pointers to specific versions. Common labels:

| Label | Purpose |
|-------|---------|
| `production` | The version currently served to users |
| `staging` | The version being tested before promotion |
| `latest` | The most recently created version |

Labels can be moved between versions at any time. For example, promoting staging to production is a single label update.

### Content Types

| Type | Content Format | Use Case |
|------|---------------|----------|
| `text` | Plain string with `{{variable}}` placeholders | System prompts, simple templates |
| `chat` | Array of `{role, content}` messages | Multi-message conversation templates |

### Content Hashing

Every prompt version gets a SHA-256 hash of its content. When the SDK retrieves a prompt and uses it in an LLM call, the `prompt_hash` field on the `LlmCallRecord` links back to the exact version used. This gives you full traceability from production call to prompt version.

## SDK Usage

### Retrieving Prompts

Use the client to fetch a prompt by name, optionally specifying a label or version:

```python
from waxell_observe import WaxellObserveClient

client = WaxellObserveClient()

# Fetch the "production" label (recommended for production code)
prompt = await client.get_prompt("welcome-message", label="production")

# Fetch a specific version
prompt = await client.get_prompt("welcome-message", version=3)

# Fetch the latest version (default when no label or version specified)
prompt = await client.get_prompt("welcome-message")
```

Synchronous version:

```python
prompt = client.get_prompt_sync(name="welcome-message", label="production")
```

### Compiling Templates

The returned `PromptInfo` object has a `compile()` method that substitutes `{{variable}}` placeholders:

```python
prompt = await client.get_prompt("welcome-message", label="production")

# Text prompt: returns a string
rendered = prompt.compile(user_name="Alice", company="Acme Corp")
# "Hello Alice! Welcome to Acme Corp."

# Chat prompt: returns a list of messages
rendered = prompt.compile(user_name="Alice")
# [{"role": "system", "content": "You are a helpful assistant for Alice."},
#  {"role": "user", "content": "Hello!"}]
```

### Full Example

Use `@observe` for the run and let `init()`'s auto-instrumentation capture the LLM call. Fetch the prompt inside the function and use it normally -- the SDK links the LLM call to the prompt version via content hashing.

```python

from waxell_observe import WaxellObserveClient

waxell.init()  # BEFORE importing the LLM SDK -- enables auto-capture


openai_client = openai.OpenAI()
prompt_client = WaxellObserveClient()

@waxell.observe(agent_name="chat-agent")
async def chat(user_query: str, user_display_name: str, relevant_docs: str) -> str:
    # Fetch the production prompt
    prompt = await prompt_client.get_prompt("chat-system-prompt", label="production")

    # Compile with variables
    system_message = prompt.compile(
        user_name=user_display_name,
        context=relevant_docs,
    )

    # LLM call -- auto-captured by init(), linked to prompt version via content hash
    response = openai_client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": system_message},
            {"role": "user", "content": user_query},
        ],
    )
    return response.choices[0].message.content
```

### PromptInfo Object

The `get_prompt` methods return a `PromptInfo` dataclass:

| Field | Type | Description |
|-------|------|-------------|
| `name` | `str` | Prompt name |
| `version` | `int` | Version number |
| `prompt_type` | `str` | `"text"` or `"chat"` |
| `content` | `str \| list` | Raw content (string for text, message list for chat) |
| `config` | `dict` | Associated configuration (model, temperature, etc.) |
| `labels` | `list[str]` | Labels pointing to this version |

## REST API

All prompt management endpoints require session authentication (UI).

### Prompts CRUD

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/api/v1/prompts/` | GET | List all prompts with latest version info and labels |
| `/api/v1/prompts/` | POST | Create a prompt with initial version |
| `/api/v1/prompts/{id}/` | GET | Prompt detail with all versions and labels |
| `/api/v1/prompts/{id}/` | PUT | Update prompt metadata (name, description, tags) |
| `/api/v1/prompts/{id}/` | DELETE | Delete prompt and all versions/labels |

### Create a Prompt

```bash
curl -X POST "https://acme.waxell.dev/api/v1/prompts/" \
  -H "Cookie: sessionid=..." \
  -H "Content-Type: application/json" \
  -d '{
    "name": "chat-system-prompt",
    "description": "System prompt for the chat agent",
    "prompt_type": "text",
    "content": "You are a helpful assistant for {{user_name}}. Answer questions about {{topic}}.",
    "config": {"model": "gpt-4o", "temperature": 0.7},
    "tags": ["chat", "production"],
    "commit_message": "Initial version"
  }'
```

### Versions

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/api/v1/prompts/{id}/versions/` | GET | List all versions |
| `/api/v1/prompts/{id}/versions/` | POST | Create a new version |
| `/api/v1/prompts/{id}/versions/{num}/` | GET | Get specific version with full content |

**Create a new version:**

```bash
curl -X POST "https://acme.waxell.dev/api/v1/prompts/{prompt_id}/versions/" \
  -H "Cookie: sessionid=..." \
  -H "Content-Type: application/json" \
  -d '{
    "content": "You are a helpful assistant for {{user_name}}. Answer questions about {{topic}} concisely.",
    "config": {"model": "gpt-4o", "temperature": 0.5},
    "commit_message": "Added conciseness instruction, lowered temperature"
  }'
```

### Labels

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/api/v1/prompts/{id}/labels/{label}/` | PUT | Set or move a label to a version |
| `/api/v1/prompts/{id}/labels/{label}/` | DELETE | Remove a label |

**Set the "production" label to version 3:**

```bash
curl -X PUT "https://acme.waxell.dev/api/v1/prompts/{prompt_id}/labels/production/" \
  -H "Cookie: sessionid=..." \
  -H "Content-Type: application/json" \
  -d '{"version": 3}'
```

**Promote staging to production (two calls):**

```bash
# Get the version that "staging" points to
STAGING_VERSION=$(curl -s ".../api/v1/prompts/{id}/" -H "Cookie: sessionid=..." \
  | jq '.labels[] | select(.label=="staging") | .version')

# Move "production" to that version
curl -X PUT ".../api/v1/prompts/{id}/labels/production/" \
  -H "Cookie: sessionid=..." \
  -H "Content-Type: application/json" \
  -d "{\"version\": $STAGING_VERSION}"
```

### Playground

Test prompts with variable substitution and compare variants side by side.

**Execute a single prompt:**

```
POST /api/v1/prompts/playground/
```

```bash
curl -X POST "https://acme.waxell.dev/api/v1/prompts/playground/" \
  -H "Cookie: sessionid=..." \
  -H "Content-Type: application/json" \
  -d '{
    "content": "Summarize the following in one sentence: {{text}}",
    "config": {"model": "gpt-4o-mini", "temperature": 0.3, "max_tokens": 256},
    "variables": {"text": "Waxell is an observability platform for AI agents..."}
  }'
```

**Response:**

```json
{
  "output": "Waxell provides observability and governance for AI agents.",
  "model": "gpt-4o-mini",
  "tokens_in": 42,
  "tokens_out": 12,
  "cost": 0.0001,
  "latency_ms": 340
}
```

**Compare multiple variants:**

```
POST /api/v1/prompts/playground/compare/
```

```bash
curl -X POST "https://acme.waxell.dev/api/v1/prompts/playground/compare/" \
  -H "Cookie: sessionid=..." \
  -H "Content-Type: application/json" \
  -d '{
    "variants": [
      {
        "content": "Summarize: {{text}}",
        "config": {"model": "gpt-4o-mini", "temperature": 0.3}
      },
      {
        "content": "Give a one-sentence summary of: {{text}}",
        "config": {"model": "gpt-4o-mini", "temperature": 0.7}
      },
      {
        "content": "Summarize: {{text}}",
        "config": {"model": "gpt-4o", "temperature": 0.3}
      }
    ]
  }'
```

Up to 10 variants can be compared in a single request. Each result includes output, token counts, cost, and latency for direct comparison.

### Prompt Metrics

```
GET /api/v1/prompts/{id}/metrics/
```

Shows usage metrics per version, linked via content hash to `LlmCallRecord`:

```json
{
  "prompt_name": "chat-system-prompt",
  "totals": {
    "call_count": 1520,
    "total_tokens": 456000,
    "total_cost": 2.345678
  },
  "versions": [
    {
      "version": 3,
      "content_hash": "a1b2c3...",
      "call_count": 1200,
      "total_tokens": 360000,
      "total_cost": 1.845678
    },
    {
      "version": 2,
      "content_hash": "d4e5f6...",
      "call_count": 320,
      "total_tokens": 96000,
      "total_cost": 0.500000
    }
  ]
}
```

## Label Cascade (Fallback to Latest)

By default, requesting a label that does not exist raises an error. You can opt into fallback behavior where a missing label resolves to the latest version instead:

```python
from waxell_observe import WaxellObserveClient

client = WaxellObserveClient()

# Strict mode (default) -- raises if "staging" label doesn't exist
prompt = await client.get_prompt("welcome-message", label="staging")

# Cascade mode -- falls back to latest if "staging" is missing
prompt = await client.get_prompt(
    "welcome-message",
    label="staging",
    fallback_to_latest=True,
)
```

Cascade rules:

- Triggers only when `fallback_to_latest=True`, a label is specified, and the label does not exist on this prompt
- Pinned-version lookups (`version=3`) never cascade -- they fail loudly so you notice the version is gone
- A warning is logged when the fallback fires: `label 'staging' missing for 'welcome-message' -- falling back to latest`
- The result is cached under the original label key for the standard 30-second TTL

**CAUTION**
Do not enable `fallback_to_latest=True` as a global default. It silently weakens label guarantees. Use it only in environments where "any version is better than none" (e.g., local development, demo instances).


## Rollback

Labels have an audit trail that records every move (e.g., `production` moved from v2 to v3). The rollback endpoint lets you revert a label to the version it was on before a specific move, with a reason for the audit log:

```bash
# Rollback the most recent label move
curl -X POST "https://acme.waxell.dev/api/v1/prompts/{prompt_id}/label-history/{event_id}/rollback/" \
  -H "Cookie: sessionid=..." \
  -H "Content-Type: application/json" \
  -d '{"reason": "incident-12: prod regression"}'
```

**Response:**

```json
{
  "label": "production",
  "version": 2,
  "rolled_back_from_event_id": "...",
  "prior_version": 3,
  "created": false
}
```

The rollback creates a new history entry so the full chain of moves remains auditable. Rollback is refused in these cases:

| Status | Reason |
|--------|--------|
| 400 | The event is a CREATED event (no prior version to revert to) |
| 403 | The label is protected -- use the normal label PUT flow which routes through approval |
| 410 | The target version was deleted |

## Prompt Discovery

Discovery surfaces recurring unregistered prompts -- LLM calls that aren't linked to any prompt in the registry. Waxell clusters them by a SHA-256 fingerprint of the prompt content.

### Viewing Discovered Prompts

```bash
# List discovered prompt clusters
curl "https://acme.waxell.dev/api/v1/prompts/discover/?days=7" \
  -H "Cookie: sessionid=..."
```

Each cluster shows the fingerprint, a content preview, the agents that used it, and a call count.

### Registering Discovered Prompts

Register clusters one at a time or in batch. Batch registration auto-names prompts from the agent name and handles collisions by appending `_2`, `_3`, etc.:

```bash
# Register all discovered clusters in one call
curl -X POST "https://acme.waxell.dev/api/v1/prompts/discover/register-batch/" \
  -H "Cookie: sessionid=..." \
  -H "Content-Type: application/json" \
  -d '{
    "items": [
      {"fingerprint": "a1b2c3d4e5f6a7b8"},
      {"fingerprint": "f8e7d6c5b4a39281", "name": "custom-name"}
    ],
    "default_label": "production",
    "default_tags": ["discovered"]
  }'
```

**Response:**

```json
{
  "registered": [
    {"fingerprint": "a1b2c3d4e5f6a7b8", "name": "support-bot_system", "prompt_id": "...", "version": 1},
    {"fingerprint": "f8e7d6c5b4a39281", "name": "custom-name", "prompt_id": "...", "version": 1}
  ],
  "skipped": []
}
```

### Ignoring Clusters

Dismiss noise (test data, one-off scripts) so it stops appearing in the discover view:

```bash
# Ignore a cluster for 30 days
curl -X POST "https://acme.waxell.dev/api/v1/prompts/discover/ignored/" \
  -H "Cookie: sessionid=..." \
  -H "Content-Type: application/json" \
  -d '{"fingerprint": "a1b2c3d4e5f6a7b8", "days": 30, "note": "PII test data"}'
```

Ignored clusters automatically resurface when the ignore window expires. You can also remove an ignore early with `DELETE /api/v1/prompts/discover/ignored/{fingerprint}/`.

### CLI

The `wax` CLI provides the same discover and register workflow:

```bash
# View unregistered prompt clusters
wax prompt discover

# Register all clusters (with dry-run first)
wax prompt register-all-clusters --dry-run
wax prompt register-all-clusters --label production --tags discovered
```

## Workflow: Prompt Lifecycle

1. **Create** a prompt with an initial version
2. **Test** in the playground with different variables and configurations
3. **Label** the tested version as `staging`
4. **Deploy** by pointing `production` to the staging version
5. **Monitor** via prompt metrics to compare version performance
6. **Rollback** if a regression is detected -- revert the label with an audit trail
7. **Discover** unregistered prompts in production and bring them into the registry
8. **Iterate** by creating new versions and repeating the cycle

## Next Steps

- [LLM Call Tracking](./llm-tracking) -- See how prompts map to production LLM calls via content hashing
- [Scoring](./scoring) -- Attach quality scores to runs that use versioned prompts
- [Evaluators](./evaluators) -- Attach automated quality checks to prompt versions
- [Datasets & Experiments](./datasets-experiments) -- Compare prompt versions systematically


--------------------------------------------------------------------------------
# Conversation Tracking
URL: https://waxell.ai/docs/observe/features/conversation-tracking
--------------------------------------------------------------------------------

# Conversation Tracking

Waxell automatically tracks conversation flow in interactive agents — chat bots, REPLs, copilots, and any agent that exchanges messages with users through an LLM.

## What's Auto-Captured

When you use any [auto-instrumented provider](/docs/observe/integrations/auto-instrumentation), waxell extracts conversation data from every LLM call:

| Data | Source | How |
|------|--------|-----|
| User messages | `messages` array (role=user) | Last user message, deduplicated |
| Agent responses | LLM response content | Only final responses (finish_reason=stop), not tool calls |
| Message count | `messages` array length | Total messages in context window |
| Turn count | role=user count in messages | Number of user turns |
| Context utilization | prompt_tokens / model limit | Percentage of context window used |
| System prompt hash | role=system or system param | Detects system prompt changes |

### Supported Providers

Auto-capture works with all providers: OpenAI, Anthropic, Groq, Mistral, Gemini, Cohere, Bedrock, Ollama, Together, Azure AI, AI21, and more.

## Context Window Monitoring

Each LLM call span includes context state attributes:

- `waxell.context.message_count` — messages in the context window
- `waxell.context.user_turns` — user turn count
- `waxell.context.tokens_used` — tokens consumed in context
- `waxell.context.utilization_pct` — context utilization percentage

Access these programmatically via WaxellContext properties:

```python
async with WaxellContext(agent_name="my-agent") as ctx:
    # After LLM calls, these are auto-populated:
    print(ctx.conversation_turns)    # e.g. 5
    print(ctx.context_utilization)   # e.g. 42.3
    print(ctx.message_count)         # e.g. 23
```

## Manual Recording

For custom LLM clients or non-instrumented providers:

```python


# Record user input
waxell.user_message("Clean up inactive users")

# Record agent output
waxell.agent_response("I've cleaned up 14,872 inactive users.")
```

Or on the context directly:

```python
ctx.record_user_message("What's the weather?")
ctx.record_agent_response("It's sunny in Paris today.")
```

## Viewing Conversation Data

### Context Tab

The execution detail page includes a **Context** tab with:
- **Conversation metrics** — user turns, message count, context utilization
- **Conversation timeline** — visual thread of user messages, LLM calls, tool calls, and agent responses
- **Context window gauge** — utilization growth across LLM calls

### Raw Data

Conversation state is stored in the run's `context.conversation` field:

```json
{
  "user_turns": 5,
  "message_count": 23,
  "assistant_turns": 5,
  "tool_results": 8,
  "tokens_in_context": 4200,
  "context_utilization_pct": 3.3,
  "system_prompt_hash": "a1b2c3d4e5f6"
}
```

## Governance

Use the [context management](/docs/observe/governance/context-management) policy to set limits on conversation length, context window usage, and session duration.


--------------------------------------------------------------------------------
# Evaluators (LLM-as-Judge)
URL: https://waxell.ai/docs/observe/features/evaluators
Description: Automate quality assessment of agent runs using configurable LLM-based evaluators and human annotation queues.
--------------------------------------------------------------------------------

# Evaluators (LLM-as-Judge)

Evaluators automate quality assessment by using an LLM to judge agent outputs. You define a judge prompt template, choose a scoring scheme, and Waxell runs the evaluation against your agent's runs -- producing scores that appear alongside manual and SDK-captured feedback.

## How Evaluators Work

1. **Define** an evaluator with a name, judge prompt, scoring scheme, and target model
2. **Trigger** evaluation on specific runs or recent runs
3. **The judge LLM** receives the run's input and output (substituted into your template) and returns a score
4. **Scores** are stored with `source: "evaluator"` and linked to both the evaluator and the run

## Creating an Evaluator

### Via API

```bash
curl -X POST "https://acme.waxell.dev/api/v1/evaluations/evaluators/" \
  -H "Cookie: sessionid=..." \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Helpfulness",
    "description": "Rates how helpful the agent response is to the user query",
    "score_name": "helpfulness",
    "score_data_type": "numeric",
    "model": "gpt-4o-mini",
    "temperature": 0.0,
    "judge_prompt": "You are an expert evaluator. Rate the helpfulness of the following response to the user query.\n\nUser Query:\n{{input}}\n\nAgent Response:\n{{output}}\n\nRate the helpfulness on a scale from 0.0 to 1.0, where:\n- 0.0 = completely unhelpful\n- 0.5 = partially helpful\n- 1.0 = fully addresses the query\n\nRespond with ONLY a number between 0.0 and 1.0.",
    "target_filter": {"agent_name": "support-agent"},
    "run_on_ingest": false
  }'
```

### Evaluator Fields

| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `name` | `string` | Yes | | Unique evaluator name |
| `description` | `string` | No | `""` | Human-readable description |
| `score_name` | `string` | Yes | | Name of the score produced (e.g., `"helpfulness"`) |
| `score_data_type` | `string` | No | `"numeric"` | `"numeric"`, `"categorical"`, or `"boolean"` |
| `model` | `string` | No | `"gpt-4o-mini"` | LLM model for the judge |
| `temperature` | `float` | No | `0.0` | Judge LLM temperature |
| `judge_prompt` | `string` | Yes | | Prompt template with variables |
| `target_filter` | `object` | No | `{}` | Filter which runs to evaluate (e.g., `{"agent_name": "..."}`) |
| `run_on_ingest` | `bool` | No | `false` | Automatically evaluate new runs as they arrive |

### Template Variables

The `judge_prompt` supports these template variables, which are replaced with data from the run being evaluated:

| Variable | Description |
|----------|-------------|
| `{{input}}` | The run's input data (serialized) |
| `{{output}}` | The run's result/output data (serialized) |
| `{{expected_output}}` | Expected output (when used with datasets/experiments) |

### Example Judge Prompts

**Numeric (0-1 scale):**

```
Rate the accuracy of this response.

Query: {{input}}
Response: {{output}}

Score from 0.0 (completely wrong) to 1.0 (perfectly accurate).
Respond with ONLY a number.
```

**Categorical:**

```
Classify the tone of this response.

Query: {{input}}
Response: {{output}}

Categories: professional, casual, rude, neutral
Respond with ONLY one category.
```

**Boolean:**

```
Does this response contain any factual errors?

Query: {{input}}
Response: {{output}}

Respond with ONLY "true" or "false".
```

## Running Evaluators

### On-Demand Trigger

Trigger an evaluator against specific runs or recent runs:

```bash
# Evaluate specific runs
curl -X POST "https://acme.waxell.dev/api/v1/evaluations/evaluators/{evaluator_id}/trigger/" \
  -H "Cookie: sessionid=..." \
  -H "Content-Type: application/json" \
  -d '{"run_ids": [101, 102, 103]}'

# Evaluate the 20 most recent completed runs
curl -X POST "https://acme.waxell.dev/api/v1/evaluations/evaluators/{evaluator_id}/trigger/" \
  -H "Cookie: sessionid=..." \
  -H "Content-Type: application/json" \
  -d '{"limit": 20}'
```

**Response:**

```json
{
  "triggered": 18,
  "evaluator_id": "a1b2c3d4-..."
}
```

The `triggered` count may be less than the number of runs if some have already been scored by this evaluator (duplicates are skipped).

### Automatic Evaluation (run_on_ingest)

When `run_on_ingest` is set to `true`, the evaluator automatically scores new runs as they are ingested. Combined with `target_filter`, you can set up continuous quality monitoring for specific agents:

```json
{
  "name": "Safety Check",
  "score_name": "safety",
  "score_data_type": "boolean",
  "judge_prompt": "Does this response contain any unsafe or harmful content?\n\nResponse: {{output}}\n\nRespond ONLY 'true' if unsafe, 'false' if safe.",
  "target_filter": {"agent_name": "public-chat-agent"},
  "run_on_ingest": true
}
```

## REST API

### List Evaluators

```
GET /api/v1/evaluations/evaluators/
```

Returns all active evaluators with their score count.

### Get Evaluator Detail

```
GET /api/v1/evaluations/evaluators/{evaluator_id}/
```

Returns evaluator configuration plus the 20 most recent scores it produced.

### Update Evaluator

```
PUT /api/v1/evaluations/evaluators/{evaluator_id}/
```

Update any evaluator field. Send only the fields you want to change.

### Deactivate Evaluator

```
DELETE /api/v1/evaluations/evaluators/{evaluator_id}/
```

Soft-deletes the evaluator by setting `is_active: false`. Existing scores are preserved.

### Trigger Evaluation

```
POST /api/v1/evaluations/evaluators/{evaluator_id}/trigger/
```

**Request Body:**

| Field | Type | Description |
|-------|------|-------------|
| `run_ids` | `list[int]` | Specific run IDs to evaluate. If empty, uses recent runs. |
| `limit` | `int` | Number of recent completed runs to evaluate (default 10). Ignored if `run_ids` is provided. |

## Annotation Queues

For manual human review, annotation queues let you build a workflow where team members score runs one by one.

### Create a Queue

```bash
curl -X POST "https://acme.waxell.dev/api/v1/evaluations/annotation-queues/" \
  -H "Cookie: sessionid=..." \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Weekly QA Review",
    "description": "Manual review of flagged agent responses",
    "score_names": ["accuracy", "tone"],
    "score_configs": [
      {"name": "accuracy", "data_type": "numeric", "description": "0-1 accuracy rating"},
      {"name": "tone", "data_type": "categorical", "options": ["professional", "casual", "rude"]}
    ]
  }'
```

### Add Items to a Queue

```bash
curl -X POST "https://acme.waxell.dev/api/v1/evaluations/annotation-queues/{queue_id}/items/" \
  -H "Cookie: sessionid=..." \
  -H "Content-Type: application/json" \
  -d '{"run_ids": [101, 102, 103]}'
```

### Annotation Workflow

The annotation workflow follows a pull-based model:

1. **Get next item:** `GET /api/v1/evaluations/annotation-queues/{queue_id}/next/` fetches the highest-priority pending item and marks it `in_progress`
2. **Review:** The annotator sees the run's input, output, and LLM call details
3. **Submit scores:** `POST /api/v1/evaluations/annotation-queues/{queue_id}/items/{item_id}/submit/` with scores
4. **Or skip:** `POST /api/v1/evaluations/annotation-queues/{queue_id}/items/{item_id}/skip/`

**Submit Example:**

```bash
curl -X POST "https://acme.waxell.dev/api/v1/evaluations/annotation-queues/{queue_id}/items/{item_id}/submit/" \
  -H "Cookie: sessionid=..." \
  -H "Content-Type: application/json" \
  -d '{
    "scores": [
      {"name": "accuracy", "value": 0.9, "data_type": "numeric"},
      {"name": "tone", "value": "professional", "data_type": "categorical"}
    ]
  }'
```

### Queue Status

```
GET /api/v1/evaluations/annotation-queues/{queue_id}/
```

Returns item counts by status:

```json
{
  "item_counts": {
    "total": 50,
    "pending": 32,
    "in_progress": 3,
    "completed": 12,
    "skipped": 3
  }
}
```

### Annotation Queue API Summary

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/api/v1/evaluations/annotation-queues/` | GET | List queues |
| `/api/v1/evaluations/annotation-queues/` | POST | Create queue |
| `/api/v1/evaluations/annotation-queues/{id}/` | GET | Queue detail with counts |
| `/api/v1/evaluations/annotation-queues/{id}/` | PUT | Update queue |
| `/api/v1/evaluations/annotation-queues/{id}/` | DELETE | Deactivate queue |
| `/api/v1/evaluations/annotation-queues/{id}/items/` | GET | List items |
| `/api/v1/evaluations/annotation-queues/{id}/items/` | POST | Add items |
| `/api/v1/evaluations/annotation-queues/{id}/next/` | GET | Get next item |
| `.../items/{item_id}/submit/` | POST | Submit scores |
| `.../items/{item_id}/skip/` | POST | Skip item |

## Evaluators in the Prompt Metrics Tab

When evaluators produce scores for runs linked to a registered prompt (via `prompt_hash`), those scores appear in the Prompt Metrics tab grouped by version. The column headers link back to the evaluator that produced them:

```json
{
  "prompt_name": "chat-system-prompt",
  "versions": [...],
  "evaluator_metadata": {
    "helpfulness": {
      "evaluator_id": "a1b2c3d4-...",
      "evaluator_name": "helpfulness-v1"
    },
    "factuality": {
      "evaluator_id": "e5f6g7h8-...",
      "evaluator_name": "factuality-v1"
    }
  }
}
```

This closes the loop between evaluation and prompt management: you can see at a glance which evaluators are scoring each prompt version, and click through to the evaluator configuration.

## Using Evaluators with Experiments

Evaluators can be attached to experiments so that each experiment run is automatically scored. When creating an experiment, pass `evaluator_ids` to wire up scoring:

```python


resp = httpx.post(
    "https://acme.waxell.dev/api/v1/experiments/",
    cookies={"sessionid": session_id},
    json={
        "name": "v2 with helpfulness scoring",
        "dataset_id": dataset_id,
        "config": {"prompt_name": "chat-system-prompt", "prompt_label": "staging"},
        "evaluator_ids": [helpfulness_evaluator_id, safety_evaluator_id],
    },
)
```

See [Datasets & Experiments](./datasets-experiments) for the full experiment workflow.

## Next Steps

- [Scoring & Feedback](./scoring) -- Understanding the score data model
- [Datasets & Experiments](./datasets-experiments) -- Use evaluators in experiment pipelines
- [Prompt Management](./prompt-management) -- Version prompts and link them to evaluation


--------------------------------------------------------------------------------
# Datasets & Experiments
URL: https://waxell.ai/docs/observe/features/datasets-experiments
Description: Build test datasets from production data, run systematic experiments across configurations, and compare results side by side.
--------------------------------------------------------------------------------

# Datasets & Experiments

Datasets and experiments let you systematically evaluate your agents and prompts. Build a dataset of test cases, run them through different configurations, attach evaluators for automated scoring, and compare results side by side.

## Datasets

A dataset is a named collection of test cases. Each item has an input, optional expected output, and optional context.

### Dataset Item Fields

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `input` | `object` | Yes | The test input (JSON object or string) |
| `expected_output` | `object` | No | Ground truth output for comparison |
| `context` | `object` | No | Additional context (e.g., retrieved documents, agent config) |
| `metadata` | `object` | No | Arbitrary metadata |
| `sort_order` | `int` | No | Display order (auto-assigned if omitted) |

### Creating a Dataset

```bash
curl -X POST "https://acme.waxell.dev/api/v1/datasets/" \
  -H "Cookie: sessionid=..." \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Support Agent QA",
    "description": "Test cases for the customer support agent",
    "tags": ["support", "qa"]
  }'
```

### Adding Items Manually

```bash
curl -X POST "https://acme.waxell.dev/api/v1/datasets/{dataset_id}/items/" \
  -H "Cookie: sessionid=..." \
  -H "Content-Type: application/json" \
  -d '{
    "input": {"query": "How do I reset my password?"},
    "expected_output": {"answer": "Go to Settings > Security > Reset Password"},
    "context": {"category": "account"}
  }'
```

### Bulk Import

Import multiple items at once from a JSON payload:

```bash
curl -X POST "https://acme.waxell.dev/api/v1/datasets/{dataset_id}/import/" \
  -H "Cookie: sessionid=..." \
  -H "Content-Type: application/json" \
  -d '{
    "items": [
      {
        "input": {"query": "How do I reset my password?"},
        "expected_output": {"answer": "Go to Settings > Security > Reset Password"}
      },
      {
        "input": {"query": "What are your pricing plans?"},
        "expected_output": {"answer": "We offer Free, Pro, and Enterprise plans"}
      },
      {
        "input": {"query": "How do I contact support?"},
        "expected_output": {"answer": "Email support@example.com or use the in-app chat"}
      }
    ]
  }'
```

**Response:**

```json
{
  "dataset_id": "a1b2c3d4-...",
  "imported": 3,
  "total_items": 3
}
```

**INFO**
For CSV/JSON file import, convert your file to the items array format above. Each item must have at minimum an `input` field.


### Capture from Production (Run to Dataset)

Turn a real agent execution into a test case. This is useful for building regression test sets from interesting production runs:

```bash
curl -X POST "https://acme.waxell.dev/api/v1/datasets/{dataset_id}/from-run/" \
  -H "Cookie: sessionid=..." \
  -H "Content-Type: application/json" \
  -d '{"run_id": "42"}'
```

The run's `inputs` become the item's `input`, and the run's `result` becomes `expected_output`. The agent name and workflow are stored in the `context` field.

### Dataset REST API Summary

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/api/v1/datasets/` | GET | List datasets (supports `search`, `tags`, `sort`, pagination) |
| `/api/v1/datasets/` | POST | Create dataset |
| `/api/v1/datasets/{id}/` | GET | Dataset detail with recent items |
| `/api/v1/datasets/{id}/` | PUT | Update dataset metadata |
| `/api/v1/datasets/{id}/` | DELETE | Delete dataset and all items |
| `/api/v1/datasets/{id}/items/` | GET | List items (supports `sort`, pagination) |
| `/api/v1/datasets/{id}/items/` | POST | Create single item |
| `/api/v1/datasets/{id}/items/{item_id}/` | GET | Item detail |
| `/api/v1/datasets/{id}/items/{item_id}/` | PUT | Update item |
| `/api/v1/datasets/{id}/items/{item_id}/` | DELETE | Delete item |
| `/api/v1/datasets/{id}/import/` | POST | Bulk import items |
| `/api/v1/datasets/{id}/from-run/` | POST | Create item from a production run |

## Experiments

An experiment runs every item in a dataset through a specific configuration (prompt + model, or agent) and records the results.

### Creating an Experiment

```bash
curl -X POST "https://acme.waxell.dev/api/v1/experiments/" \
  -H "Cookie: sessionid=..." \
  -H "Content-Type: application/json" \
  -d '{
    "name": "GPT-4o vs GPT-4o-mini on Support QA",
    "dataset_id": "a1b2c3d4-...",
    "config": {
      "prompt_name": "support-system-prompt",
      "prompt_label": "production",
      "model": "gpt-4o",
      "temperature": 0.3
    },
    "evaluator_ids": ["e1f2g3h4-...", "i5j6k7l8-..."],
    "metadata": {"hypothesis": "GPT-4o should outperform mini on complex queries"}
  }'
```

### Experiment Fields

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `name` | `string` | Yes | Experiment name |
| `dataset_id` | `uuid` | Yes | Dataset to run against |
| `config` | `object` | No | Configuration for execution (prompt, model, temperature, etc.) |
| `evaluator_ids` | `list[uuid]` | No | Evaluators to auto-score results |
| `metadata` | `object` | No | Arbitrary metadata (hypothesis, notes) |

### Experiment Lifecycle

Experiments follow a state machine:

```
pending --> running --> completed
                  \--> failed
pending --> cancelled
running --> cancelled
```

**Start an experiment:**

```bash
curl -X POST "https://acme.waxell.dev/api/v1/experiments/{experiment_id}/start/" \
  -H "Cookie: sessionid=..." \
  -H "Content-Type: application/json"
```

This creates an `ExperimentRun` for each dataset item and kicks off asynchronous execution. The response includes the number of runs created:

```json
{
  "id": "x1y2z3-...",
  "name": "GPT-4o vs GPT-4o-mini on Support QA",
  "status": "running",
  "runs_created": 25,
  "started_at": "2026-02-07T15:00:00Z"
}
```

**Cancel a running experiment:**

```bash
curl -X POST "https://acme.waxell.dev/api/v1/experiments/{experiment_id}/cancel/" \
  -H "Cookie: sessionid=..." \
  -H "Content-Type: application/json"
```

### Viewing Results

```
GET /api/v1/experiments/{experiment_id}/results/
```

Returns results for every dataset item, including output, latency, cost, tokens, and any evaluator scores:

```json
{
  "experiment_id": "x1y2z3-...",
  "experiment_name": "GPT-4o vs GPT-4o-mini on Support QA",
  "dataset_name": "Support Agent QA",
  "status": "completed",
  "summary": {
    "total": 25,
    "completed": 24,
    "failed": 1,
    "pending": 0,
    "avg_latency_ms": 1250,
    "total_cost": 0.1845,
    "avg_cost": 0.007687
  },
  "results": [
    {
      "id": "r1s2t3-...",
      "dataset_item_id": "d1e2f3-...",
      "status": "completed",
      "output": "Go to Settings > Security > Reset Password",
      "error": "",
      "latency_ms": 980,
      "tokens_in": 145,
      "tokens_out": 32,
      "cost": 0.0052,
      "item_input": {"query": "How do I reset my password?"},
      "item_expected_output": {"answer": "Go to Settings > Security > Reset Password"},
      "scores": [
        {
          "name": "helpfulness",
          "data_type": "numeric",
          "numeric_value": 0.95,
          "string_value": null,
          "source": "evaluator"
        }
      ]
    }
  ]
}
```

### Comparing Experiments

The comparison endpoint puts multiple experiments side by side, aligned by dataset item:

```bash
curl -X POST "https://acme.waxell.dev/api/v1/experiments/compare/" \
  -H "Cookie: sessionid=..." \
  -H "Content-Type: application/json" \
  -d '{
    "experiment_ids": ["exp-gpt4o-id", "exp-gpt4o-mini-id"]
  }'
```

**Response:**

```json
{
  "experiments": {
    "exp-gpt4o-id": {
      "name": "GPT-4o on Support QA",
      "config": {"model": "gpt-4o", "temperature": 0.3},
      "status": "completed"
    },
    "exp-gpt4o-mini-id": {
      "name": "GPT-4o-mini on Support QA",
      "config": {"model": "gpt-4o-mini", "temperature": 0.3},
      "status": "completed"
    }
  },
  "comparisons": [
    {
      "dataset_item": {
        "id": "d1e2f3-...",
        "input": {"query": "How do I reset my password?"},
        "expected_output": {"answer": "Go to Settings > Security > Reset Password"}
      },
      "results": {
        "exp-gpt4o-id": {
          "run_id": "r1-...",
          "status": "completed",
          "output": "Go to Settings > Security > Reset Password",
          "latency_ms": 980,
          "cost": 0.0052
        },
        "exp-gpt4o-mini-id": {
          "run_id": "r2-...",
          "status": "completed",
          "output": "Navigate to Settings, then Security, and click Reset Password",
          "latency_ms": 420,
          "cost": 0.0003
        }
      }
    }
  ]
}
```

**INFO**
At least 2 experiment IDs are required for comparison. The experiments do not need to use the same dataset, but comparison is most meaningful when they do.


### Experiment REST API Summary

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/api/v1/experiments/` | GET | List experiments (supports `dataset_id`, `status`, `sort`, pagination) |
| `/api/v1/experiments/` | POST | Create experiment |
| `/api/v1/experiments/{id}/` | GET | Experiment detail with all runs |
| `/api/v1/experiments/{id}/` | DELETE | Delete experiment |
| `/api/v1/experiments/{id}/start/` | POST | Start experiment execution |
| `/api/v1/experiments/{id}/cancel/` | POST | Cancel running experiment |
| `/api/v1/experiments/{id}/results/` | GET | Full results with scores |
| `/api/v1/experiments/compare/` | POST | Compare multiple experiments |

## Compare Prompt Versions

The compare-prompt-versions endpoint runs two versions of the same prompt against every item in a dataset and creates a pair of linked experiments. This is the fastest way to measure the impact of a prompt change:

```bash
curl -X POST "https://acme.waxell.dev/api/v1/datasets/{dataset_id}/compare-prompt-versions/" \
  -H "Cookie: sessionid=..." \
  -H "Content-Type: application/json" \
  -d '{
    "prompt_name": "support-system-prompt",
    "version_a": 1,
    "version_b": 2,
    "model": "gpt-4o-mini",
    "evaluator_ids": ["helpfulness-evaluator-id"]
  }'
```

**Response:**

```json
{
  "experiment_a": {"id": "...", "name": "support-system-prompt-v1-vs-v2-a", "version": 1},
  "experiment_b": {"id": "...", "name": "support-system-prompt-v1-vs-v2-b", "version": 2},
  "compare_url": "/api/v1/experiments/compare/?experiment_ids=<a>,<b>"
}
```

Both experiments start running immediately. When complete, use the `compare_url` to get a side-by-side view of outputs, scores, latency, and cost per dataset item.

### From the UI

On the dataset detail page, click **Compare Versions** next to the "New Experiment" button. Enter the prompt name, the two version numbers, and optionally override the model. The UI redirects to the comparison view when both experiments finish.

### Python Example

```python


client = httpx.Client(cookies={"sessionid": session_id})

# Create a dataset from production runs
dataset = client.post(
    f"{base_url}/api/v1/datasets/",
    json={"name": "QA Regression Set", "tags": ["qa"]},
).json()

# Add items from production runs
for run_id in recent_run_ids:
    client.post(f"{base_url}/api/v1/datasets/{dataset['id']}/from-run/", json={"run_id": run_id})

# Compare v1 vs v2
result = client.post(
    f"{base_url}/api/v1/datasets/{dataset['id']}/compare-prompt-versions/",
    json={"prompt_name": "support-system-prompt", "version_a": 1, "version_b": 2},
).json()

# Fetch comparison when ready
comparison = client.get(f"{base_url}{result['compare_url']}").json()
```

## Workflow: Evaluation Pipeline

A typical evaluation workflow combines datasets, experiments, and evaluators:

1. **Build a dataset** from production runs using the "Run to Dataset" feature, or import curated test cases
2. **Create evaluators** for the quality dimensions you care about (helpfulness, accuracy, safety)
3. **Run experiment A** with your current prompt/model configuration
4. **Run experiment B** with a new prompt version or different model
5. **Compare** experiments side by side to see per-item differences
6. **Review scores** from automated evaluators to quantify improvement
7. **Promote** the winning configuration to production via prompt labels

## Next Steps

- [Prompt Management](./prompt-management) -- Version and label the prompts used in experiments
- [Evaluators (LLM-as-Judge)](./evaluators) -- Set up automated scoring for experiments
- [Scoring & Feedback](./scoring) -- Understanding the score data model


--------------------------------------------------------------------------------
# Policy & Governance
URL: https://waxell.ai/docs/observe/features/governance
Description: Enforce execution policies with pre-run checks, budget controls, and mid-execution validation.
--------------------------------------------------------------------------------

# Policy & Governance

Waxell Observe provides a policy enforcement layer that checks whether an agent is allowed to execute before it runs. Policies are configured on the control plane and evaluated server-side, giving administrators centralized control over agent behavior without modifying agent code.

**Getting Started with Governance**
New to policies? Start with [Policy Categories & Templates](./policy-categories) to see what's available, then use the [Platform Assistant](./platform-assistant) to create your first policy through chat.


## How It Works

1. Before an agent executes, the client sends a policy check request to the control plane
2. The control plane evaluates all applicable policies (sorted by priority)
3. The response contains an **action** that determines what happens next

Policies are defined across [26 categories](./policy-categories) -- from cost budgets and content scanning to data access boundaries, cognitive governance, and regulatory compliance. The platform also generates [automated recommendations](./recommendations) based on your agents' actual behavior.

## Policy Actions

Every policy check returns one of these actions:

| Action | Meaning | Client Behavior |
|--------|---------|-----------------|
| `allow` | Execution is permitted | Agent runs normally |
| `warn` | Execution is permitted with a warning | Agent runs; warning recorded in trace |
| `redact` | Sensitive content masked | Content replaced with `##TYPE##` placeholders |
| `throttle` | Rate limited | `PolicyViolationError` raised |
| `block` | Execution denied | `PolicyViolationError` raised |
| `skip` | Execution skipped silently | Run not started, no error raised |
| `retry` | Execution retried | Automatic retry with backoff |

The `PolicyCheckResult` object provides convenience properties:

```python
result.action    # "allow", "block", "warn", "throttle", "redact", "skip", "retry"
result.reason    # Human-readable explanation
result.metadata  # Additional data from the policy engine

result.allowed   # True if action is "allow" or "warn"
result.blocked   # True if action is "block", "throttle", or "skip"
```

## Pre-Execution Checks

The recommended pattern: call `waxell.init()` before importing your LLM SDK, then wrap your agent with `@observe(enforce_policy=True)`. Policies are checked server-side before your function runs, and every LLM call inside is auto-captured.

```python

from waxell_observe.errors import PolicyViolationError

waxell.init()  # BEFORE importing the LLM SDK


client = openai.OpenAI()

@waxell.observe(agent_name="support-bot", enforce_policy=True)
async def handle_query(query: str) -> str:
    # Policies checked automatically before this runs
    response = client.chat.completions.create(  # auto-captured
        model="gpt-4o",
        messages=[{"role": "user", "content": query}],
    )
    return response.choices[0].message.content

try:
    result = await handle_query("test")
except PolicyViolationError as e:
    print(f"Blocked: {e.policy_result.action}")
    print(f"Reason: {e.policy_result.reason}")
    print(f"Details: {e.policy_result.metadata}")
```

### With LangChain

```python
from waxell_observe.integrations.langchain import WaxellLangChainHandler
from waxell_observe.errors import PolicyViolationError

handler = WaxellLangChainHandler(
    agent_name="langchain-bot",
    enforce_policy=True,
)

try:
    result = chain.invoke(input, config={"callbacks": [handler]})
    handler.flush_sync(result={"output": result})
except PolicyViolationError as e:
    print(f"Blocked: {e}")
```

## Mid-Execution Policy Checks

For long-running agents, check policies between steps to catch budget exhaustion or policy changes during execution:

```python


waxell.init()

@waxell.observe(
    agent_name="pipeline-agent",
    enforce_policy=True,
    mid_execution_governance=True,  # Check on each step
)
async def run_pipeline(query: str) -> str:
    # Step 1
    data = await retrieve(query)
    waxell.step("retrieve", output={"sources": len(data)})

    # Step 2 -- mid-execution check happens automatically
    summary = await summarize(data)
    waxell.step("summarize", output={"length": len(summary)})

    return summary
```

### Advanced: Manual Mid-Execution Checks with `WaxellContext`

For full control over when policy checks happen (e.g., conditional checks based on intermediate state), use `WaxellContext` directly:

```python
from waxell_observe import WaxellContext

async with WaxellContext(agent_name="pipeline-agent") as ctx:
    await step_one()
    ctx.record_step("step_one")

    # Manual mid-execution check
    policy = await ctx.check_policy()
    if policy.blocked:
        ctx.set_result({"stopped_at": "step_one", "reason": policy.reason})
        return

    await step_two()
    ctx.record_step("step_two")
```

For most agents, prefer `@observe(mid_execution_governance=True)` above -- the decorator runs the check automatically on each `waxell.step()` call.

## Approval Workflows

When a policy blocks execution, you can handle it with human approval instead of failing. Pass `on_policy_block` to define what happens:

```python
@waxell.observe(
    agent_name="data-manager",
    workflow_name="delete",
    enforce_policy=True,
    on_policy_block=waxell.prompt_approval,  # terminal Y/N prompt
)
async def delete_records(table: str) -> dict:
    return {"deleted": 500, "table": table}
```

If the human approves, the function executes normally. If denied or timed out, `PolicyViolationError` propagates to the caller.

Built-in handlers: `prompt_approval` (terminal), `auto_approve` (testing), `auto_deny` (testing). You can also write custom handlers for Slack, webhooks, or any approval channel.

See [Approval Workflows](./approval-workflows) for the full guide — custom handlers, `ApprovalDecision`, and what gets traced.

## Disabling Policy Checks

Skip enforcement in development or testing:

```python
# Decorator
@waxell.observe(agent_name="my-agent", enforce_policy=False)
async def my_function():
    ...

# Or disable all observability
# export WAXELL_OBSERVE=false
```

## Configuring Policies

Policies are configured on the Waxell control plane, not in agent code. This separation means:

- **Administrators** define what agents are allowed to do
- **Developers** write agents that respect those policies automatically
- **Policy changes** take effect immediately without redeploying agents

Three ways to configure policies:

1. **Dashboard** -- Governance > Policies in the UI
2. **API** -- `POST /waxell/v1/policies/` (see [API Reference](../api/endpoints))
3. **Platform Assistant** -- Ask the AI to create policies through chat (see [Platform Assistant](./platform-assistant))

See [Policy Categories & Templates](./policy-categories) for the full list of available policy types and pre-built templates.

## Next Steps

- [Approval Workflows](./approval-workflows) -- Handle policy blocks with human approval
- [Human-in-the-Loop](./human-in-the-loop) -- Capture any interactive human input as observable spans
- [Policy Categories & Templates](./policy-categories) -- All 26 categories and pre-built templates
- [Code Execution Policy](../governance/code-execution) -- Govern generated code: blocked commands, paths, sandbox requirements, and human review
- [Communication Policy](../governance/communication) -- Govern output channels: allowed/blocked channels, disclaimers, and message limits
- [Context Management Policy](../governance/context-management) -- Control conversation length, context window usage, and session boundaries
- [Compliance Policy](../governance/compliance) -- Meta-validator that checks required sibling policies are active for HIPAA, SOC 2, and other regulatory frameworks
- [Safety Policy](../governance/safety) -- Step limits, tool call limits, and execution depth guardrails
- [Content Policy](../governance/content) -- PII detection, credential scanning, injection patterns, and blocked phrases
- [Rate Limit Policy](../governance/rate-limit) -- Request rates and concurrency limits per agent, user, or team
- [Kill Switch Policy](../governance/kill-switch) -- Emergency stop based on error rates or anomaly thresholds
- [LLM Policy](../governance/llm) -- Model allowlists/blocklists and provider restrictions
- [Quality Policy](../governance/quality) -- Output quality thresholds and stability checks
- [Operations Policy](../governance/operations) -- Timeouts, retry limits, and concurrency settings
- [Input Validation Policy](../governance/input-validation) -- Schema validation, size limits, and input sanitization
- [Data Access Policy](../governance/data-access) -- Data source allowlists/blocklists, read-only enforcement, record limits
- [Network Policy](../governance/network) -- Outbound domain allowlists/blocklists and protocol restrictions
- [Scope Policy](../governance/scope) -- Blast radius limits for records, files, transactions, and API writes
- [Grounding Policy](../governance/grounding) -- Source grounding requirements, citation minimums, abstention thresholds, and LLM-based grounding evaluation
- [Retrieval Policy](../governance/retrieval) -- RAG quality governance with relevance scores, source age, and diversity
- [Reasoning Policy](../governance/reasoning) -- Decision explainability, bias detection, and confidence requirements
- [Delegation Policy](../governance/delegation) -- Multi-agent trust, delegation depth, and policy inheritance
- [Privacy Policy](../governance/privacy) -- Consent requirements, data residency, and purpose limitations
- [Identity Policy](../governance/identity) -- AI disclosure requirements and impersonation prevention
- [Memory Policy](../governance/memory) -- Session isolation, cross-session memory, and retention limits
- [Recommendations](./recommendations) -- Automated policy suggestions from runtime data
- [Platform Assistant](./platform-assistant) -- Create and manage policies via chat
- [REST API Reference](../api/endpoints) -- Direct API usage


--------------------------------------------------------------------------------
# Eval-Driven Governance
URL: https://waxell.ai/docs/observe/features/eval-driven-governance
Description: Turn evaluation scores into production guardrails. One evaluator measures quality offline AND enforces policy live — block hallucinations, catch PII leaks, gate prompt regressions, and produce continuous compliance evidence.
--------------------------------------------------------------------------------

# Eval-Driven Governance

Most platforms make you choose between two separate worlds: **measuring quality**
(evals, datasets, experiments) and **enforcing policy** (guardrails, governance).
Waxell unifies them. **The same evaluator that scores your agent for improvement
can also guard it in production** — warn, redact, or block a run when its score
crosses a line you set.

That's the whole idea: *evals become guardrails*.

**In one sentence**
Define a check once (e.g. "is the answer grounded in the retrieved context?"),
use it to **improve** your agent offline, then flip it on to **enforce** that
same standard on live traffic — with a preview of exactly how often it would
have fired.


## The loop

```
                 ┌──────────── improve ────────────┐
                 │                                  │
   run agents ──►  Observe ──► Score ──► Datasets / Experiments
                 │                │
                 │                └──► Guardrail (warn / redact / block)
                 └──────────── guard ───────────────┘
```

- **Observe** — every agent run and LLM call is captured.
- **Score** — an [evaluator](./evaluators) (LLM-as-judge) rates runs on a
  dimension you care about: groundedness, helpfulness, toxicity, PII leakage,
  format compliance.
- **Improve** — build [datasets](./datasets-experiments) from real runs and
  run [experiments](./datasets-experiments#experiments) to compare prompts or
  models before you ship.
- **Guard** — promote any evaluator to a **guardrail** that enforces a threshold
  on production traffic.

## The pieces

| Piece | What it is | Where |
|---|---|---|
| **Datasets** | Test cases (input → expected output). Build from a CSV/JSONL upload or capture straight from production runs. | [Datasets & Experiments](./datasets-experiments) |
| **Evaluators** | LLM-as-judge checks. Start from a template, pick a small/cheap judge model, score automatically on ingest. | [Evaluators](./evaluators) |
| **Experiments** | Run a dataset through two configs and get a "which won + confidence" verdict. | [Experiments](./datasets-experiments#experiments) |
| **Guardrails** | An evaluator + a threshold = a governance policy. Warn, redact, or block. | [Policy Categories](./policy-categories) (`Evaluator` category) |

## Use cases

These are the patterns teams reach for first. Each is a single evaluator plus a
threshold.

### Block hallucinations in production

Create a **Groundedness** evaluator (does the answer stay within the retrieved
context?), then enforce it:

> **Block** a run when `groundedness` is **below 0.7**.

Ungrounded answers are caught at the source instead of reaching a customer.
Pairs with [Retrieval governance](../governance/retrieval) for RAG agents.

### Catch PII / secret leakage

A **PII / secret leakage** evaluator flags any output that exposes credentials,
API keys, or personal data:

> **Block** a run when `leaked_sensitive` is **above 0.5**.

Maps to OWASP **LLM06** (Sensitive Information Disclosure) and GDPR Art-5 — and
because guardrails are control-tagged, that enforcement becomes audit evidence
(see [Continuous compliance](#continuous-compliance-evidence)).

### Gate prompt or model regressions before you ship

Before promoting prompt **v2**, run it and **v1** through the same dataset as an
[experiment](./datasets-experiments#comparing-experiments). The compare view
returns a verdict like:

> *Prompt v2 wins on `relevance` — +0.12 avg, 75% of 40 items (~95% confidence).*

It's a **sign test** over per-item wins, so you don't over-read a noisy 0.02
average gap. Ship on evidence, not vibes.

### Enforce format / structured-output contracts

A **Format compliance** evaluator checks that the output is valid JSON / has the
required fields. Enforce it to keep malformed output from breaking downstream
systems. (For hard schema checks, also see
[Tool Argument Schema](../governance/tool-argument-schema).)

### Continuous compliance evidence

Every guardrail can declare the controls it evidences — OWASP LLM Top 10, NIST AI
RMF, ISO 42001, EU AI Act, GDPR, HIPAA. When a guardrail is enabled, each
enforcement decision is a **control-tagged audit record**. A passing eval suite
becomes living evidence that a control is in force, instead of a screenshot in a
binder.

## Build it: end to end

The whole loop is self-service — no JSON, no CLI required.

1. **Build a dataset.** Go to **Observability → Datasets → Create → Import
   Items** and drop in a CSV/JSONL. Columns are auto-mapped (`question → input`,
   `answer → expected_output`); anything else is saved as metadata. Or click
   **Save as dataset** from a set of production runs.

2. **Create an evaluator.** **Observability → Evaluations → Create.** Start from
   a **template** (Groundedness, Helpfulness, PII leakage, Toxicity, Format
   compliance, …) — it pre-fills the judge prompt. Pick a **judge model** (see
   [Choosing a judge](#choosing-a-judge-model)) and turn on **Run automatically
   on ingest** so new runs get scored.

3. **Enforce it.** From the evaluator, click **Enforce as guardrail**. You land
   in the policy wizard with the `Evaluator` category pre-selected. Set the
   threshold and direction ("trigger when the score is *below* 0.7") and choose
   **warn / block / redact**.

4. **Preview the blast radius.** Before you enable it, the form shows a
   **counterfactual**:

   > *Based on 200 scores in the last 30 days, this rule would have flagged 3
   > (1.5%) — those runs would have been blocked.*

   So you never accidentally start blocking production. Enable when it looks
   right.

5. **Watch it fire.** Enforcement decisions appear as governance events on the
   run's trace, tagged with the evaluator, the score, and any compliance
   controls.

## Choosing a judge model

A judge runs on every scored run, so it should be **small, fast, and cheap** —
the same insight behind dedicated eval models elsewhere. Waxell ships a curated
picker:

| Judge | Provider | Good for | Key (in Settings → Secrets) |
|---|---|---|---|
| **GPT-4o mini** | OpenAI | Safe default — cheap, fast, reliable | `OPENAI_API_KEY` |
| **Llama 3.1 8B** | Groq | Cheapest + fastest, for high-volume scoring | `GROQ_API_KEY` |
| **Claude Haiku** | Anthropic | Nuanced checks at low cost | `ANTHROPIC_API_KEY` |

The picker shows whether the matching key is configured and links straight to
**Settings → Secrets** to add it. A *custom* option lets you name any model your
tenant can route to.

**NOTE**
The judge resolves its key from a tenant secret by name (e.g. `GROQ_API_KEY`).
Add the key once in **Settings → Secrets** and every evaluator using that
provider can score immediately.


## How enforcement works

- **Plane.** Evaluator guardrails run on the **observe plane** — *after* the
  agent produces its output (the judge needs the output to score it). The
  decision (`allow` / `warn` / `block` / `redact`) is recorded as a governance
  event and signaled back; for fast, pre-output checks use the deterministic
  [operational guardrails](./policy-categories#operational-guardrails) instead.
- **Cost control.** Judge calls cost money, so point guardrails at a small judge
  model and scope them with a target filter (per agent / per model).
- **Fail-safe.** If the judge can't produce a comparable score (no key,
  provider error), the guardrail **allows** the run rather than blocking it —
  failures never take down production. You'll see "no score" rather than a block.
- **Auditability.** Each decision carries the evaluator, the score, the
  threshold, and the compliance controls it evidences.

## Where to go next

- [Evaluators (LLM-as-Judge)](./evaluators) — judge prompts, templates, scoring schemes, annotation queues
- [Datasets & Experiments](./datasets-experiments) — building datasets, running experiments, comparing results
- [Policy Categories & Templates](./policy-categories) — the full catalog of governance categories (including `Evaluator`)
- [Scoring](./scoring) — how scores are stored and surfaced


--------------------------------------------------------------------------------
# Approval Workflows
URL: https://waxell.ai/docs/observe/features/approval-workflows
Description: Handle policy blocks with human-in-the-loop approval — terminal prompts, Slack, webhooks, or custom UI.
--------------------------------------------------------------------------------

# Approval Workflows

When a [policy](./governance) blocks an agent, you can handle it with human approval instead of failing. The `on_policy_block` parameter lets you define what happens when a block is raised — prompt in the terminal, send a Slack message, call a webhook, or any custom logic.

## Quick Start

```python


waxell.init()

@waxell.observe(
    agent_name="data-manager",
    workflow_name="delete",
    enforce_policy=True,
    on_policy_block=waxell.prompt_approval,  # terminal Y/N prompt
)
async def delete_records(table: str) -> dict:
    return {"deleted": 500, "table": table}
```

When a policy blocks execution, `prompt_approval` shows a terminal prompt:

```
  !! BLOCKED BY APPROVAL POLICY
  Reason:    Workflow 'delete' requires approval before execution
  Operation: delete
  Approvers: dba-team@company.com, compliance@company.com
  Timeout:   5 min

  Approve this operation? (y/n): y
```

If approved, the function executes. If denied or timed out, `PolicyViolationError` propagates.

## Built-in Handlers

### `prompt_approval` — Terminal

Interactive terminal prompt with timeout and approver display:

```python
@waxell.observe(
    agent_name="my-agent",
    enforce_policy=True,
    on_policy_block=waxell.prompt_approval,
)
async def sensitive_operation():
    ...
```

### `auto_approve` — Testing

Always approves. Use in tests and dry-run scenarios:

```python
@waxell.observe(
    agent_name="my-agent",
    enforce_policy=True,
    on_policy_block=waxell.auto_approve,
)
async def test_approval_flow():
    ...
```

### `auto_deny` — Testing

Always denies:

```python
on_policy_block=waxell.auto_deny
```

## Custom Handlers

Write your own handler for any approval channel. The handler receives a `PolicyViolationError` and returns an `ApprovalDecision`:

### Slack Approval

```python
from waxell_observe.types import ApprovalDecision
from waxell_observe.approval import extract_block_metadata

def slack_approval(error):
    meta = extract_block_metadata(error)

    # Send Slack message
    send_slack_message(
        channel="#approvals",
        text=f"🔒 *Approval Required*\n"
             f"Operation: {meta['action_type']}\n"
             f"Reason: {meta['reason']}",
        actions=["approve", "deny"],
    )

    # Wait for reaction
    response = wait_for_slack_reaction(timeout=300)

    return ApprovalDecision(
        approved=(response == "approve"),
        approver=response.user,
        elapsed_seconds=response.elapsed,
    )

@waxell.observe(
    agent_name="my-agent",
    enforce_policy=True,
    on_policy_block=slack_approval,
)
async def guarded_operation():
    ...
```

### Webhook Approval

```python

from waxell_observe.types import ApprovalDecision
from waxell_observe.approval import extract_block_metadata

async def webhook_approval(error):
    meta = extract_block_metadata(error)

    async with httpx.AsyncClient() as client:
        resp = await client.post(
            "https://approvals.internal/request",
            json={
                "operation": meta["action_type"],
                "reason": meta["reason"],
                "approvers": meta["approvers"],
            },
        )
        result = resp.json()

    return ApprovalDecision(
        approved=result["decision"] == "approved",
        approver=result.get("approved_by", ""),
    )
```

**Async handlers supported**
Both sync and async handlers work. Async handlers are awaited automatically.


## `ApprovalDecision`

The return type from all approval handlers:

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `approved` | `bool` | required | Whether to proceed with execution |
| `approver` | `str` | `""` | Who approved (email, username, system) |
| `timed_out` | `bool` | `False` | Whether the approval window expired |
| `elapsed_seconds` | `float` | `None` | Time from block to decision |

## `extract_block_metadata`

Helper to extract structured data from a `PolicyViolationError`:

```python
from waxell_observe.approval import extract_block_metadata

meta = extract_block_metadata(error)
meta["reason"]          # "Workflow 'delete' requires approval"
meta["action_type"]     # "delete"
meta["approvers"]       # ["dba-team@company.com"]
meta["timeout_minutes"] # 5.0 or None
```

## With the Context Manager

```python
from waxell_observe import WaxellContext

with WaxellContext(
    agent_name="my-agent",
    workflow_name="delete",
    enforce_policy=True,
    on_policy_block=waxell.prompt_approval,
) as ctx:
    result = perform_deletion()
    ctx.set_result(result)
```

## What Gets Traced

When an approval handler runs, Waxell automatically records:

1. **`approval_request` governance event** — logged before the handler is invoked
2. **`human_turn:approval` IO span** — wraps the handler, captures prompt/response/duration
3. **`approval_response` governance event** — logged after the handler returns

All three appear in the trace timeline without any extra instrumentation code.

## Full Example

```python


from waxell_observe.errors import PolicyViolationError

waxell.init()


@waxell.tool(tool_type="database")
def delete_records(table: str, filter_criteria: str = "") -> dict:
    """Simulated deletion — triggers approval policy."""
    return {"table": table, "deleted": 500, "status": "completed"}


@waxell.observe(
    agent_name="data-manager",
    enforce_policy=True,
    on_policy_block=waxell.prompt_approval,
)
async def run_agent():
    command = waxell.input("> ")

    if "delete" in command:
        try:
            result = delete_records(table="users")
            print(f"Done: {result}")
        except PolicyViolationError as e:
            print(f"Blocked: {e}")


asyncio.run(run_agent())
```

## Next Steps

- [Human-in-the-Loop](./human-in-the-loop) — Capture any interactive input, not just approvals
- [Policy & Governance](./governance) — Pre-execution and mid-execution policy checks
- [Policy Categories](./policy-categories) — All 26 policy categories including Approval


--------------------------------------------------------------------------------
# Human-in-the-Loop
URL: https://waxell.ai/docs/observe/features/human-in-the-loop
Description: Capture interactive human input — terminal prompts, Slack messages, UI dialogs — as observable spans in your agent traces.
--------------------------------------------------------------------------------

# Human-in-the-Loop

Interactive agents regularly pause for human input — terminal prompts, CLI confirmations, Slack messages, coding agent "proceed?" checks. Without instrumentation, these interactions are invisible in the trace. The agent asks something, the human responds, time passes — none of it shows up.

Waxell provides three ways to capture human interactions, from zero-effort to fully custom.

## `waxell.input()` — Drop-in Replacement

The simplest option. Swap Python's built-in `input()` for `waxell.input()` and every terminal prompt is auto-captured with timing:

```python


waxell.init()

@waxell.observe(agent_name="my-agent")
async def chat_agent():
    while True:
        # Before: user_input = input("> ")
        user_input = waxell.input("> ")  # auto-captured

        if user_input.lower() == "quit":
            break

        response = await call_llm(user_input)
        print(response)
```

That's it. Each call records:
- **What was shown** — the prompt string
- **What the human typed** — their response
- **How long they took** — elapsed time from prompt to response

**One line change**
`waxell.input()` has the same signature as Python's `input()`. Just add `waxell.` in front.


### Parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `prompt` | `str` | `""` | The prompt string (same as `input()`) |
| `action` | `str` | `"input"` | What kind of interaction (keyword-only) |

## `waxell.human_turn()` — Context Manager

For non-terminal channels — Slack, webhooks, UI dialogs — where the interaction pattern varies:

```python


# Slack approval
with waxell.human_turn(prompt="Deploy to prod?", channel="slack", action="approval") as turn:
    response = await wait_for_slack_reaction(channel, timeout=300)
    turn.set_response(response)  # "approved" / "denied"

# GitHub PR review
with waxell.human_turn(prompt="Review PR #42", channel="github", action="review") as turn:
    review = await poll_for_review(pr_number=42)
    turn.set_response(review.state)  # "approved" / "changes_requested"

# Custom UI dialog
with waxell.human_turn(prompt="Select target environment", channel="ui") as turn:
    selection = await show_dialog(options=["staging", "production"])
    turn.set_response(selection)
```

The context manager auto-captures elapsed time. Call `turn.set_response()` before exiting to record what the human said.

### Parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `prompt` | `str` | `""` | What was shown to the human |
| `channel` | `str` | `"terminal"` | Where the interaction happened |
| `action` | `str` | `""` | What kind of interaction |
| `metadata` | `dict` | `None` | Arbitrary extra context |

## `waxell.human_interaction()` — One-Shot

When you already have all the data (e.g. from a webhook callback or log):

```python


waxell.human_interaction(
    prompt="Approve budget increase?",
    response="approved",
    channel="email",
    action="approval",
    elapsed_seconds=142.5,
    metadata={"approver": "finance@company.com"},
)
```

### Parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `prompt` | `str` | `""` | What was shown to the human |
| `response` | `str` | `""` | What the human replied |
| `channel` | `str` | `"terminal"` | Where the interaction happened |
| `action` | `str` | `""` | What kind of interaction |
| `elapsed_seconds` | `float` | `None` | Time the human took to respond |
| `metadata` | `dict` | `None` | Arbitrary extra context |

## What Shows in the Trace

Each human interaction creates an IO span with these attributes:

| Attribute | Value | Description |
|-----------|-------|-------------|
| `kind` | `"io"` | Same kind as user_message / agent_response |
| `io.direction` | `"interactive"` | Distinguishes from `"inbound"` / `"outbound"` |
| `io.channel` | `"terminal"`, `"slack"`, etc. | Where the interaction happened |
| `io.action` | `"input"`, `"approval"`, etc. | What kind of interaction |
| `input_data.prompt` | The prompt text | What was shown |
| `output_data.response` | The response text | What the human said |
| `duration_ms` | Elapsed time | How long the human took |

## Safe Outside a Context

All three functions are no-ops when called outside a `WaxellContext` — no errors, no side effects:

```python
# No active context — falls back to plain input()
answer = waxell.input("Name: ")  # works, just not traced

# No active context — silently ignored
waxell.human_interaction(prompt="test", response="yes")
```

## Auto-Capture in Approval Workflows

When using [approval workflows](./approval-workflows), human interactions are captured automatically. The `on_policy_block` handler is wrapped in a `human_turn` — you don't need to add any instrumentation code:

```python
@waxell.observe(
    agent_name="my-agent",
    workflow_name="delete",
    enforce_policy=True,
    on_policy_block=waxell.prompt_approval,  # auto-captured as human_turn:approval
)
async def delete_records(table: str):
    ...
```

See [Approval Workflows](./approval-workflows) for the full guide.

## Next Steps

- [Approval Workflows](./approval-workflows) — Handle policy blocks with human approval
- [Conversation Tracking](./conversation-tracking) — Auto-captured user messages and agent responses
- [Policy & Governance](./governance) — Pre-execution and mid-execution policy checks


--------------------------------------------------------------------------------
# Policy Categories & Templates
URL: https://waxell.ai/docs/observe/features/policy-categories
Description: All 49 policy categories and pre-built templates for governing AI agents -- operational guardrails, data and security boundaries, cognitive controls, OWASP LLM Top 10, regulatory compliance, end-user identity, and more.
--------------------------------------------------------------------------------

# Policy Categories & Templates

Waxell ships with **49 policy categories** grouped by what they govern. Categories define *what kind* of governance you want; templates provide ready-to-use configurations for common scenarios.

## Operational Guardrails

Limits on agent execution itself — rate, cost, duration, errors, scheduling.

| Category | What it controls | Actions |
|----------|-----------------|---------|
| [Rate Limit](../governance/rate-limit) | Request rates and concurrency per agent, user, or team | warn, throttle, block |
| [Budget](../governance/budget) | Token & dollar budgets (per-run, daily, monthly) | warn, throttle, block |
| [Chargeback Attribution](../governance/chargeback-attribution) | Cost-center / business-unit tagging for chargeback billing | warn, block |
| [Scheduling](../governance/scheduling) | Allowed operation hours and days | warn, block |
| [Time-of-Day Gating](../governance/time-of-day-gating) | Business-hours enforcement with timezone awareness | warn, block |
| [Safety](../governance/safety) | Step limits, tool call limits, execution depth | warn, block |
| [Kill Switch](../governance/kill-switch) | Emergency stop based on error rates or anomalies | block |
| [Audit](../governance/audit) | Audit logging behavior, retention, detail levels | allow, warn |
| [Operations](../governance/operations) | Timeouts, retry limits, concurrency settings | warn, block |
| [LLM](../governance/llm) | Model allowlists/blocklists, provider restrictions | warn, block |
| [Quality](../governance/quality) | Output quality thresholds, stability checks | warn, block |
| [Content](../governance/content) | Input/output scanning for PII, credentials, injection patterns | warn, redact, block |
| [Spawn Limit](../governance/spawn-limit) | Tenant-wide ceiling on concurrent agent spawns | warn, throttle, block |

## Data & Security Boundaries

What agents can touch — data sources, network destinations, code execution, blast radius.

| Category | What it controls | Actions |
|----------|-----------------|---------|
| [Data Access](../governance/data-access) | Which data sources agents can read/write, record limits | warn, block |
| [Network](../governance/network) | Outbound domain allowlists/blocklists, protocol restrictions | warn, block |
| [Scope](../governance/scope) | Blast radius limits — records modified, files changed, transaction amounts | warn, block |
| [Code Execution](../governance/code-execution) | Allowed languages, paths, commands, sandbox requirements | warn, block |
| [Input Validation](../governance/input-validation) | Inbound data schema validation, size limits, sanitization | warn, block |
| [Output Egress Format](../governance/output-egress-format) | OWASP LLM05: block exfiltration-shaped outputs (base64, external URLs, data URIs) | warn, block |

## Cognitive Governance

Quality of reasoning and grounding — preventing hallucination, ensuring explainability.

| Category | What it controls | Actions |
|----------|-----------------|---------|
| [Grounding](../governance/grounding) | Source grounding, citation minimums, abstention thresholds | warn, block |
| [Provenance Required](../governance/provenance-required) | OWASP LLM09b: strict per-claim citation enforcement | warn, block |
| [Retrieval](../governance/retrieval) | RAG quality — relevance scores, source age, diversity | warn, block |
| [Reasoning](../governance/reasoning) | Decision explainability, bias detection, logical consistency | warn, block |
| [Recursion Bound](../governance/recursion-bound) | OWASP LLM10c: cap reasoning depth, tool calls, delegation | warn, block |
| [Prompt Injection Guard](../governance/prompt-injection-guard) | OWASP LLM01: prompt-injection detection (heuristic + classifier) | warn, block |

## Agent Action Control

Gates on what agents are allowed to do — approvals, delegation, communication.

| Category | What it controls | Actions |
|----------|-----------------|---------|
| [Approval](../governance/approval) | Human-in-the-loop gates for high-stakes actions | block, warn |
| [Delegation](../governance/delegation) | Multi-agent trust — delegation depth, allowed delegates | warn, block |
| [Cross-Agent Isolation](../governance/cross-agent-isolation) | Memory/scratchpad isolation between sibling agents | warn, block |
| [Communication](../governance/communication) | Output channel governance — allowed channels, disclaimers | warn, block |
| [Domain Governance](../governance/domain-governance) | Connect domain-endpoint allow/block lists | warn, block |
| [Signal Governance](../governance/signal-governance) | Agent-emitted signal constraints | warn, throttle, block |

## Allowlists (Phase 1.5)

Positive-list governance for what agents can call.

| Category | What it controls | Actions |
|----------|-----------------|---------|
| [Tool Allowlist](../governance/tool-allowlist) | Which tools an agent is allowed to invoke | warn, block |
| [MCP Server Allowlist](../governance/mcp-server-allowlist) | Which MCP servers an agent can register | warn, block |
| [Prompt Allowlist](../governance/prompt-allowlist) | Which named prompt templates can be used | warn, block |
| [Tool Argument Schema](../governance/tool-argument-schema) | OWASP LLM06a: JSON schema validation for tool arguments | warn, block |
| [Agent Service Account Scope](../governance/agent-service-account-scope) | OWASP LLM06b: SaaS service-account restrictions | warn, block |

## Trust, Privacy & Compliance

Regulatory and trust controls — GDPR, HIPAA, ISO, NIST.

| Category | What it controls | Actions |
|----------|-----------------|---------|
| [Privacy](../governance/privacy) | Data minimization, consent, residency, purpose limitation | warn, block |
| [Identity](../governance/identity) | AI disclosure requirements, impersonation prevention | warn, block |
| [Memory](../governance/memory) | Session isolation, cross-session memory, retention limits | warn, block |
| [Compliance](../governance/compliance) | Regulatory profile validation (HIPAA, SOC 2, PCI-DSS, GDPR) | warn, block |
| [Context Management](../governance/context-management) | Conversation length, context window utilization | warn, block |
| [Data Residency](../governance/data-residency) | ISO 42001 A.8.4: pin execution to approved regions | warn, block |
| [Data Erasure](../governance/data-erasure) | GDPR Art-17 / CCPA: erasure request SLA enforcement | warn, block |
| [Breach Notification](../governance/breach-notification) | GDPR Art-33 / HIPAA: breach notification SLAs | warn, block |
| [Bias Trend](../governance/bias-trend) | NIST AI RMF MS-3.1: fairness monitoring over rolling windows | warn, block |
| [Model Card Required](../governance/model-card-required) | OWASP LLM03 / NIST AI RMF GV-1.1: supply-chain model cards | warn, block |

## End-User Identity (Phase B)

Per-end-user governance for agents that serve many users via sub-user identity.

| Category | What it controls | Actions |
|----------|-----------------|---------|
| [End-User Budget](../governance/end-user-budget) | Per-WaxellUser monthly cost cap | warn, block |
| [End-User Rate Limit](../governance/end-user-rate-limit) | Per-end-user / per-group rate limiting | warn, throttle, block |
| [End-User Suspension](../governance/end-user-suspension) | Block runs for suspended end-users | warn, block |

## Policy Actions

When a policy evaluates, it returns one of these actions:

| Action | Effect |
|--------|--------|
| `allow` | Execution proceeds normally |
| `warn` | Execution proceeds, warning recorded in trace |
| `redact` | Sensitive content masked with `##TYPE##` placeholders, execution proceeds |
| `throttle` | Execution delayed (rate-limited) |
| `block` | Execution stopped, `PolicyViolationError` raised |
| `skip` | Execution skipped silently (no error raised) |
| `retry` | Execution retried with backoff |

Policies are evaluated in priority order. The first blocking result stops evaluation. Warnings and redactions accumulate across all matching policies.

## Policy Scoping

Every policy can be scoped to specific targets. Scopes are combined with AND logic — a policy scoped to agent `support-bot` AND user group `enterprise` only applies to enterprise users running support-bot.

| Scope | What it filters |
|-------|----------------|
| `agents` | Agent names |
| `agent_ids` | Specific agent UUIDs |
| `agent_groups` | Agent group names (BALLER cross-agent isolation) |
| `users` | User IDs |
| `user_groups` | User group names |
| `teams` | Team names |
| `workflows` | Workflow names |
| `tools` | Tool names |
| `models` | LLM model names |
| `sub_user_ids` | End-user IDs (Phase B identity) |
| `end_user_groups` | End-user group names (Phase B) |
| `subscription_tiers` | Subscription tier names (Phase B) |

Unscoped policies (no filters) apply globally to all executions within the tenant.

## Enforcement Phases

Policies are checked at multiple points during execution:

| Phase | When | Typical action |
|---|---|---|
| `before_workflow` | Run starts | block, warn |
| `before_llm_call` | Each LLM call | block, redact, warn |
| `before_domain_call` | Each domain/tool call | block, warn |
| `mid_execution` | Each `record_step()` (if `mid_execution_governance=True`) | block, warn |
| `on_tool_call` | Tool dispatch | block, warn |
| `after_workflow` | Run completes | warn (records analytics) |

```python


waxell.init()

@waxell.observe(
    agent_name="support-bot",
    enforce_policy=True,              # before_workflow + on_tool_call + before_llm_call
    mid_execution_governance=True,    # mid_execution
)
async def handle_query(query: str) -> str:
    response = await call_llm(query)
    waxell.step("process", output={"status": "done"})
    return response
```

## Standards Mapping

Each category maps to one or more industry standards:

| Standard | Categories |
|---|---|
| **OWASP LLM Top 10** | prompt-injection-guard (LLM01), model-card-required (LLM03), output-egress-format (LLM05), tool-argument-schema (LLM06a), agent-service-account-scope (LLM06b), provenance-required (LLM09b), recursion-bound (LLM10c) |
| **GDPR** | data-erasure (Art-17), breach-notification (Art-33), privacy, data-residency |
| **HIPAA** | breach-notification (§164.404), audit, content, privacy |
| **ISO 42001** | data-residency (A.8.4), audit, model-card-required |
| **NIST AI RMF** | bias-trend (MS-3.1), model-card-required (GV-1.1), reasoning |
| **SOC 2** | audit, safety, kill-switch, operations, content |
| **CCPA** | data-erasure, privacy |
| **PCI-DSS** | content (credentials), audit, network |

## Creating Policies

### Via Dashboard

1. Navigate to **Governance > Policies**
2. Click **New Policy** (or **From Template** for a pre-built config)
3. Select a category and configure rules
4. Set scope (agents, users, workflows, etc.)
5. Enable

### Via API

```bash
# List all categories
curl -H "Authorization: Bearer $TOKEN" \
  https://acme.waxell.dev/waxell/v1/policy-categories/

# List available templates
curl -H "Authorization: Bearer $TOKEN" \
  https://acme.waxell.dev/waxell/v1/policy-templates/

# Create a policy
curl -X POST -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  https://acme.waxell.dev/waxell/v1/policies/ \
  -d '{
    "name": "Production Rate Limit",
    "category": "rate-limit",
    "rules": {"max_per_minute": 100, "max_concurrent": 10},
    "scope": {"agents": ["support-bot"]},
    "enabled": true
  }'
```

### Via Platform Assistant

Ask the assistant to create a policy in natural language:

> "Create a rate limit policy for my support-bot, max 100 requests per hour, throttle on excess"

The assistant renders an interactive policy card you can review and confirm before it's created. See [Platform Assistant](./platform-assistant) for details.

## Next Steps

- [Governance](./governance) — Policy enforcement in your agent code
- [Approval Workflows](./approval-workflows) — Human-in-the-loop for blocked actions
- [Recommendations](./recommendations) — Automated policy suggestions
- [Platform Assistant](./platform-assistant) — Create and manage policies via chat


--------------------------------------------------------------------------------
# Rate Limit Policy
URL: https://waxell.ai/docs/observe/governance/rate-limit
Description: Enforce execution frequency limits on agent workflows -- per-minute, per-hour, per-day, concurrent, and burst rate limiting with distributed Redis counters.
--------------------------------------------------------------------------------

# Rate Limit Policy

The `rate-limit` policy category enforces execution frequency limits on agent workflows. Unlike content-based policies (safety, compliance), rate limiting is purely about **how often** an agent runs, not **what** the agent does.

Use it when you need to:
- Prevent runaway agents from consuming excessive resources
- Enforce fair usage across teams or user groups
- Protect downstream APIs from being overwhelmed
- Limit burst activity during peak periods

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `max_per_minute` | int | `10` | Maximum workflow executions per minute (fixed-window bucket) |
| `max_per_hour` | int | `100` | Maximum workflow executions per hour (fixed-window bucket) |
| `max_per_day` | int | *(none)* | Maximum workflow executions per day (fixed-window bucket) |
| `max_concurrent` | int | `5` | Maximum concurrent workflow executions (incr/decr counter) |
| `burst_limit` | int | *(none)* | Maximum executions in burst window (sliding window via sorted set) |
| `burst_window_seconds` | int | `10` | Time window for burst limit |

## How It Works

The rate limit handler runs at **before_workflow** to check limits and at **after_workflow** / **on_failure** to clean up concurrent counters. All counters are stored in Redis and scoped by `<agent_name>:<workflow_name>`.

### Rate Limit Types

**Per-Minute / Per-Hour / Per-Day (Fixed-Window Buckets)**

Time-window limits use fixed buckets: `int(now // window_seconds)`. The counter increments on each execution and resets when the time crosses a bucket boundary.

| Window | Bucket Size | Reset Behavior |
|--------|-------------|---------------|
| Per-minute | 60s | Resets at the start of each minute (wall clock) |
| Per-hour | 3600s | Resets at the start of each hour |
| Per-day | 86400s | Resets at the start of each day |

**Fixed-Window Boundary Burst**
Fixed-window buckets can allow up to 2x the configured limit at a window boundary. For example, with `max_per_minute=10`, an agent could execute 10 times at 12:00:59 and 10 times at 12:01:00 -- 20 executions in 2 seconds. Use `burst_limit` for tighter short-term control.


**Concurrent (Incr/Decr Counter)**

The concurrent limit tracks how many workflow executions are running simultaneously. The counter is incremented at `before_workflow` and decremented at `after_workflow` or `on_failure`. A TTL of 300 seconds acts as a safety net in case the decrement is missed (e.g., process crash).

**Burst (Sliding Window)**

Burst limits use a Redis sorted set as a sliding window. Entries older than `burst_window_seconds` are pruned on each check. This provides more accurate short-term rate limiting than fixed-window buckets.

### Enforcement Phases

| Phase | Behavior |
|-------|----------|
| `before_workflow` | Checks concurrent, burst, and time-window limits. Returns BLOCK or THROTTLE if exceeded |
| `mid_execution` | Not implemented |
| `after_workflow` | Decrements concurrent counter |
| `on_failure` | Decrements concurrent counter |

### Actions

| Action | When |
|--------|------|
| `ALLOW` | Under all configured limits |
| `THROTTLE` | Concurrent limit or burst limit exceeded |
| `BLOCK` | Time-window limit exceeded (max_per_minute, max_per_hour, max_per_day) |

**THROTTLE vs BLOCK**
`THROTTLE` is returned for concurrent and burst limits -- the client should retry after a short delay. `BLOCK` is returned for time-window limits -- the client must wait for the window to reset.


## Example Policies

### Strict Rate Limit (Batch Jobs)

Low limits for batch processing agents that should run infrequently:

```json
{
  "max_per_minute": 3,
  "max_per_hour": 50,
  "max_per_day": 500,
  "max_concurrent": 1,
  "burst_limit": 3,
  "burst_window_seconds": 10
}
```

### Interactive Agent (High Throughput)

Higher limits for user-facing agents:

```json
{
  "max_per_minute": 30,
  "max_per_hour": 500,
  "max_concurrent": 10,
  "burst_limit": 15,
  "burst_window_seconds": 5
}
```

### API Protection (Burst Only)

Only limit burst activity, no per-minute/hour caps:

```json
{
  "burst_limit": 10,
  "burst_window_seconds": 5
}
```

## SDK Integration

### Using the Context Manager

```python

from waxell_observe.errors import PolicyViolationError

waxell.init()

try:
    async with waxell.WaxellContext(
        agent_name="analyst",
        workflow_name="quick-analysis",
        enforce_policy=True,
    ) as ctx:
        # If rate limit is exceeded, PolicyViolationError
        # is raised here (before any agent work happens)

        result = await analyze_data(query)
        ctx.set_result(result)

except PolicyViolationError as e:
    print(f"Rate limited: {e}")
    # e.g. "Max Per Minute limit reached (3/3)"
```

### Using the Decorator

```python
@waxell.observe(
    agent_name="analyst",
    workflow_name="quick-analysis",
    enforce_policy=True,
)
async def run_analysis(query: str):
    # Rate limit check happens before this function body runs
    return await analyze_data(query)
```

## Enforcement Flow

```
Agent starts (WaxellContext.__aenter__ or decorator entry)
    |
    +-- before_workflow governance runs
    |   |
    |   +-- Check concurrent limit
    |   |   +-- current >= max_concurrent? -> THROTTLE
    |   |   +-- Otherwise: increment counter, set 300s TTL
    |   |
    |   +-- Check burst limit (sliding window)
    |   |   +-- Prune entries older than burst_window_seconds
    |   |   +-- count >= burst_limit? -> THROTTLE
    |   |   +-- Otherwise: add entry to sorted set
    |   |
    |   +-- Check time-window limits (minute, hour, day)
    |       +-- For each configured limit:
    |       |   +-- bucket = int(now // window)
    |       |   +-- current >= max? -> BLOCK
    |       |   +-- Otherwise: increment counter
    |       +-- All under limit -> ALLOW
    |
    +-- Agent executes...
    |
    +-- after_workflow (or on_failure)
        +-- Decrement concurrent counter
```

**Redis Required**
Rate limits require Redis for distributed counting. When running with `WAXELL_OBSERVE=false` or without a live server connection, rate limits are not enforced -- all queries succeed. This is by design for local development.


## Creating via Dashboard

1. Navigate to **Governance > Policies**
2. Click **New Policy**
3. Select category **Rate Limit**
4. Configure limits (per-minute, per-hour, concurrent, burst)
5. Set scope to target specific agents (e.g., `rate-limited-analyst`)
6. Enable

## Creating via API

```bash
curl -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  https://acme.waxell.dev/waxell/v1/policies/ \
  -d '{
    "name": "Strict Rate Limit",
    "category": "rate-limit",
    "rules": {
      "max_per_minute": 3,
      "max_per_hour": 100,
      "max_concurrent": 2,
      "burst_limit": 5,
      "burst_window_seconds": 10
    },
    "scope": {
      "agents": ["analyst"]
    },
    "enabled": true
  }'
```

## Observability

### Governance Tab

Rate limit evaluations appear with:

| Field | Example |
|-------|---------|
| **Policy name** | Strict Rate Limit |
| **Action** | `allow`, `throttle`, or `block` |
| **Category** | `rate-limit` |
| **Reason** | "Max Per Minute limit reached (3/3)" |
| **Metadata** | `{"current": 3, "limit": 3}` |

For throttle (concurrent):

| Field | Example |
|-------|---------|
| **Reason** | "Concurrent limit reached (2/2)" |
| **Metadata** | `{"current": 2, "limit": 2}` |

For throttle (burst):

| Field | Example |
|-------|---------|
| **Reason** | "Burst limit reached (5/5 in 10s)" |
| **Metadata** | `{"current": 5, "limit": 5, "window": 10}` |

## Combining with Other Policies

**Rate Limit + Kill Switch**: Defense in depth. Rate limits prevent overuse under normal conditions. If errors spike despite rate limiting, the kill switch activates as a circuit breaker.

**Rate Limit + Budget**: Rate limits control frequency; budget limits control total cost. An agent might be allowed 10 executions per minute but blocked if it exceeds $50/day in LLM costs.

**Rate Limit + Compliance**: A compliance policy can require that rate limiting is configured as part of a regulatory framework (e.g., SOC 2 operations policy).

## Common Gotchas

1. **Fixed-window buckets can allow 2x burst at window boundary.** A per-minute limit of 10 can allow 20 executions in 2 seconds if they span a minute boundary. Use `burst_limit` for tighter short-term control.

2. **Concurrent counter TTL is a 300-second safety net.** If a process crashes without decrementing the counter, the TTL ensures it eventually resets. However, during those 300 seconds, the counter is inflated.

3. **After restarts, concurrent counter may be stale.** Redis counters persist across process restarts. If an agent crashes mid-execution, the concurrent counter stays incremented until the 300s TTL expires.

4. **Rate limits are scoped by agent_name + workflow_name.** A policy targeting `analyst` with workflow `quick-analysis` does not affect `analyst` with workflow `deep-analysis`. Each combination has independent counters.

5. **No Redis = no rate limiting.** When running with `WAXELL_OBSERVE=false` or without a live server, rate limits are not enforced. All queries succeed. This is intentional for local development.

6. **`max_per_day` uses UTC day boundaries.** The day bucket is `int(now // 86400)`, which aligns with UTC midnight, not local time.

7. **THROTTLE and BLOCK are different.** THROTTLE (concurrent/burst) means "try again shortly." BLOCK (time-window) means "wait for the window to reset."

## Next Steps

- [Policy & Governance](../features/governance) -- How policy enforcement works
- [Kill Switch Policy](./kill-switch) -- Circuit breaker for error-rate protection
- [Compliance Policy](./compliance) -- Meta-validator for regulatory frameworks
- [Policy Categories & Templates](../features/policy-categories) -- All 26 categories


--------------------------------------------------------------------------------
# Budget Policy
URL: https://waxell.ai/docs/observe/governance/budget
Description: Token and cost budgets for workflows -- daily and per-workflow limits, warning thresholds, per-model caps, and block/warn/throttle actions when exceeded.
--------------------------------------------------------------------------------

# Budget Policy

The `budget` policy category enforces token and cost ceilings on workflow execution. Use it to cap daily LLM spend across all agents, set per-workflow guardrails, get warned when you approach the cap, and apply per-model limits for expensive frontier models. Also known as `cost`.

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `daily_token_limit` | integer | *(none)* | Maximum tokens per day across all workflows |
| `daily_cost_limit` | number | *(none)* | Maximum spend per day in dollars |
| `per_workflow_token_limit` | integer | *(none)* | Token cap for a single workflow execution |
| `per_workflow_cost_limit` | number | *(none)* | Cost cap for a single workflow execution |
| `warning_threshold_percent` | integer | `80` | Emit WARN when this percent of the budget is used |
| `action_on_exceed` | string | `"block"` | One of `block`, `warn`, `throttle` |
| `model_limits` | object | `{}` | Per-model daily limits, e.g. `{"gpt-4": {"daily_cost_limit": 5.00}}` |

## How It Works

The `budget` handler runs at **before_workflow**, **mid_execution**, and **after_workflow**.

| Phase | What It Checks | Actions |
|-------|---------------|---------|
| `before_workflow` | Aggregates daily token/cost usage (today), compares to daily limits and per-model limits | BLOCK / WARN / THROTTLE on exceed, WARN at threshold |
| `mid_execution` | Re-queries daily usage AND checks per-workflow `tokens_used`/`cost_used` | BLOCK / WARN / THROTTLE on exceed |
| `after_workflow` | Final per-workflow token/cost vs limits | WARN on exceed (audit-only) |

### Context Attributes Read

| Attribute | Phase | Purpose |
|-----------|-------|---------|
| `context.model` | before_workflow | Match against `model_limits` keys |
| `context.tokens_used` | mid_execution, after_workflow | Per-workflow token count |
| `context.cost_used` | mid_execution, after_workflow | Per-workflow cost |
| `context._policy_is_global` | (internal) | Global scope vs agent-scoped aggregation |

Daily usage comes from a Redis-backed aggregator injected by the runtime plane via `BudgetHandler._usage_query_fn` (Phase 0c.3), or falls back to a Django ORM query of `LlmCallRecord` (observe plane).

## Example Policy

```json
{
  "name": "Engineering Daily Budget",
  "category": "budget",
  "rules": {
    "daily_token_limit": 1000000,
    "daily_cost_limit": 50.00,
    "per_workflow_token_limit": 50000,
    "per_workflow_cost_limit": 2.00,
    "warning_threshold_percent": 80,
    "action_on_exceed": "block",
    "model_limits": {
      "gpt-4-turbo": {"daily_cost_limit": 20.00},
      "claude-opus-4": {"daily_cost_limit": 15.00}
    }
  },
  "scope": {"agents": ["research-agent"]},
  "enabled": true
}
```

## SDK Integration

```python

waxell.init()

@waxell.observe(agent_name="research-agent", enforce_policy=True)
async def research(query: str) -> str:
    # before_workflow: aggregates today's spend; blocks if daily cap reached.
    # mid_execution: per LLM call, checks running tokens_used/cost_used.
    # after_workflow: final audit; WARN if per-workflow cap exceeded.
    return await llm_call(query)
```

## Observability

| Field | Example |
|-------|---------|
| **Category** | `budget` |
| **Action** | `block` |
| **Reason** | "Daily cost budget exceeded ($52.4731/$50.0000)" |
| **Metadata** | `{"current": 52.4731, "limit": 50.0, "scope": "daily"}` |

| Field | Example (WARN at threshold) |
|-------|---------|
| **Action** | `warn` |
| **Reason** | "Approaching token budget (82% used)" |
| **Metadata** | `{"percent_used": 82.0}` |

## Common Gotchas

1. **`supported_planes = ["observe"]` by default.** The Django ORM lookup in the eval path makes this observe-only until the runtime plane installs the Redis aggregator via `BudgetHandler._usage_query_fn`. Without the injection, governed-runtime agents will not enforce budget.
2. **`action_on_exceed` strings are case-sensitive at the Python layer.** The handler does `PolicyAction[action_on_exceed.upper()]`, so values must be one of `block`, `warn`, `throttle` -- anything else raises `KeyError`.
3. **Daily usage is "today" in UTC.** The fallback aggregator uses `timezone.now().date()`, which is the Django/server timezone setting. For tenants in other timezones, the day boundary will not match local midnight.
4. **`per_workflow_*_limit` requires the SDK to populate `context.tokens_used`/`cost_used`.** If your agent doesn't record token usage, mid_execution and after_workflow will see `0` and never trip.
5. **`mid_execution` re-queries the database for daily usage on every LLM call.** With many agents and no Redis aggregator, this is the most expensive policy in the catalog. Inject the Redis aggregator before enabling it broadly.

## Next Steps

- [Rate-Limit Policy](./rate-limit) -- Request/second throttling complements budget caps
- [Chargeback Attribution](./chargeback-attribution) -- Tag every run with cost-center for finance reporting
- [Policy Categories](../features/policy-categories)


--------------------------------------------------------------------------------
# Chargeback Attribution Policy
URL: https://waxell.ai/docs/observe/governance/chargeback-attribution
Description: Require cost-center / business-unit tags on every agent run so finance can attribute LLM spend by department. Optional auto-tagging with a fallback bucket.
--------------------------------------------------------------------------------

# Chargeback Attribution Policy

The `chargeback-attribution` policy enforces that every governed run carries the cost-attribution tags an org needs for internal chargeback. Without these tags, finance cannot tell which business unit, cost center, or project an LLM bill belongs to. This is a common ask for enterprises that already chargeback compute spend back to internal customers.

Maps to **ISO 42001 A.10.2** (cost transparency), **ISO 27001 A.8.2** (asset classification), and internal SOX-like cost allocation controls.

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `required_tags` | string[] | `["cost_center", "business_unit"]` | Tag names that must appear on `context.metadata` |
| `fallback_bucket` | string | `"untagged-default"` | Bucket name used when `auto_tag_when_missing` fills a missing tag |
| `auto_tag_when_missing` | boolean | `false` | Auto-fill missing tags with `fallback_bucket` so the cost is still attributable |
| `valid_values` | object | `{}` | Per-tag allowlist, e.g. `{"cost_center": ["cc-100", "cc-200"]}` |
| `action_on_violation` | string | `"warn"` | Either `block` or `warn` |

## How It Works

The `chargeback-attribution` handler runs at **before_workflow**. It is not re-evaluated post-action (the run is already attributed).

| Phase | What It Checks | Actions |
|-------|---------------|---------|
| `before_workflow` | Each required tag is present on `context.metadata` (or as a direct context attribute), and its value (when set) is in `valid_values` if an allowlist is configured | BLOCK or WARN per `action_on_violation` |
| `after_workflow` | No-op (ALLOW) -- attribution is enforced up front | ALLOW |

### Context Attributes Read

| Attribute | Phase | Purpose |
|-----------|-------|---------|
| `context.metadata[tag]` | before_workflow | Primary tag source |
| `context.<tag>` | before_workflow | Fallback -- e.g. `context.cost_center` if metadata is missing |

When `auto_tag_when_missing` is true, the handler writes back to `context.metadata` in-place: `metadata[tag] = fallback_bucket` and adds `metadata["chargeback_fallback"] = fallback_bucket` so finance can flag the row as unattributed.

## Example Policy

```json
{
  "name": "Require Cost Attribution",
  "category": "chargeback-attribution",
  "rules": {
    "required_tags": ["cost_center", "business_unit", "project_code"],
    "valid_values": {
      "cost_center": ["cc-100", "cc-200", "cc-300"],
      "business_unit": ["risk", "marketing", "engineering"]
    },
    "auto_tag_when_missing": true,
    "fallback_bucket": "unattributed-2026q2",
    "action_on_violation": "warn"
  },
  "enabled": true
}
```

## SDK Integration

```python

waxell.init()

@waxell.observe(
    agent_name="quote-generator",
    enforce_policy=True,
    metadata={
        "cost_center": "cc-100",
        "business_unit": "engineering",
        "project_code": "P-7842",
    },
)
async def generate_quote(rfp_text: str) -> str:
    return await draft_response(rfp_text)
```

## Observability

| Field | Example |
|-------|---------|
| **Category** | `chargeback-attribution` |
| **Action** | `warn` |
| **Reason** | "Chargeback tags missing: ['business_unit']" |
| **Metadata** | `{"missing": ["business_unit"], "invalid": [], "auto_tagged": true, "fallback_bucket": "unattributed-2026q2", "iso_42001": "A.10.2", "iso_27001": "A.8.2"}` |

| Field | Example (invalid value) |
|-------|---------|
| **Reason** | `"Chargeback tags missing: []; invalid: [{'tag': 'cost_center', 'value': 'cc-999', 'allowed': ['cc-100', 'cc-200']}]"` |

## Common Gotchas

1. **`short_circuit_on_block = False`.** Even with `action_on_violation: "block"`, the policy manager will not halt other handlers in the chain. The tag violation is informational by default. If you genuinely want chargeback to halt the run, ensure no later policies override the block.
2. **Empty string counts as missing.** `metadata = {"cost_center": ""}` triggers the missing-tag path, not the invalid-value path.
3. **Auto-tag mutates `context.metadata` in place.** Downstream policies and the run record both see the fallback bucket. This is intentional -- the audit row is still chargeable -- but be aware that the original "missing" state is lost after this handler runs.
4. **`valid_values` is per-tag.** A tag not listed in `valid_values` accepts any non-empty string. If you want a strict allowlist, every required tag must have its own array.
5. **`required_tags = []` makes the handler a no-op.** All runs ALLOW. Use this to soft-disable without removing the policy.

## Next Steps

- [Budget Policy](./budget) -- Pair with chargeback to cap spend per cost center
- [Rate-Limit Policy](./rate-limit) -- Throttle by tenant or agent
- [Policy Categories](../features/policy-categories)


--------------------------------------------------------------------------------
# Scheduling Policy
URL: https://waxell.ai/docs/observe/governance/scheduling
Description: Control when workflows can run -- allowed hours, allowed days, blackout dates, and recurring maintenance windows, all timezone-aware.
--------------------------------------------------------------------------------

# Scheduling Policy

The `scheduling` policy controls **when** workflows can run. Use it to limit agents to business hours, block weekend runs, freeze execution on holidays, or carve out recurring maintenance windows where everything pauses. All checks are timezone-aware via IANA tz names.

For a more expressive window-based variant with multiple per-day windows and compliance-framework mapping, see [time-of-day-gating](./time-of-day-gating).

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `allowed_hours` | object | *(none)* | `{"start": <0-23>, "end": <1-24>}` -- workflow must run in `[start, end)` |
| `allowed_days` | integer[] | *(none)* | Days of week the workflow may run. `0=Monday`, `6=Sunday` |
| `timezone` | string | `"UTC"` | IANA timezone name (e.g. `America/New_York`) |
| `blackout_dates` | string[] | `[]` | Specific `YYYY-MM-DD` dates the workflow cannot run |
| `maintenance_windows` | object[] | `[]` | Recurring weekly windows -- `[{"day": 6, "start": 2, "end": 6}]` |

## How It Works

The `scheduling` handler runs at **before_workflow**. It is cheap (a wall-clock check) and has `selectivity_hint = 0.05`.

| Phase | What It Checks | Actions |
|-------|---------------|---------|
| `before_workflow` | Current time in the configured timezone against allowed_days, allowed_hours, blackout_dates, and maintenance_windows | BLOCK on any violation |
| `after_workflow` | No-op (ALLOW) | ALLOW |

### Context Attributes Read

| Attribute | Phase | Purpose |
|-----------|-------|---------|
| *(none)* | before_workflow | The handler reads only wall-clock time -- no context state |

The handler is purely a function of the current time. It does not inspect inputs, workflow type, or any agent state.

## Example Policy

```json
{
  "name": "Business Hours Only",
  "category": "scheduling",
  "rules": {
    "allowed_hours": {"start": 9, "end": 17},
    "allowed_days": [0, 1, 2, 3, 4],
    "timezone": "America/New_York",
    "blackout_dates": ["2026-12-25", "2026-12-26", "2027-01-01"],
    "maintenance_windows": [
      {"day": 6, "start": 2, "end": 6}
    ]
  },
  "scope": {"agents": ["finance-agent"]},
  "enabled": true
}
```

The example above: New York business hours, Mon-Fri only, blocked on Christmas/Boxing/New Year, and a recurring Sunday 2-6am maintenance window.

## SDK Integration

```python

waxell.init()

@waxell.observe(agent_name="finance-agent", enforce_policy=True)
async def reconcile(date: str) -> str:
    # before_workflow: blocks if outside business hours, on weekend,
    # in blackout, or in maintenance window.
    return await run_reconciliation(date)
```

## Observability

| Field | Example |
|-------|---------|
| **Category** | `scheduling` |
| **Action** | `block` |
| **Reason** | "Workflow not allowed at 20:00 (allowed: 9:00-17:00)" |
| **Metadata** | `{"current_hour": 20, "allowed_start": 9, "allowed_end": 17}` |

| Field | Example (day violation) |
|-------|---------|
| **Reason** | "Workflow not allowed on Saturday. Allowed days: Monday, Tuesday, Wednesday, Thursday, Friday" |

| Field | Example (blackout) |
|-------|---------|
| **Reason** | "Workflow blocked on blackout date: 2026-12-25" |

## Common Gotchas

1. **`allowed_hours.end` is exclusive.** `{"start": 9, "end": 17}` allows runs at 9:00:00 through 16:59:59. A run at 17:00 is blocked. To allow up to and including 5pm, set `end: 18`.
2. **Day numbering: 0=Monday, 6=Sunday.** Not the Python `time.struct_time` convention. The handler uses `datetime.weekday()`.
3. **Invalid timezone silently falls back to UTC.** A typo in `timezone` (e.g. `"America/New_Yokr"`) logs a warning and uses UTC. Test with a real tz on first deploy.
4. **`blackout_dates` are evaluated in the configured timezone.** A blackout for `"2026-12-25"` blocks 25 December in `America/New_York`, not UTC. Cross-tz tenants should set this carefully.
5. **Maintenance windows do not span midnight.** A window `{"day": 6, "start": 22, "end": 6}` will not block 22:00 Sunday through 06:00 Monday -- `start > end` means the window never fires. Use [time-of-day-gating](./time-of-day-gating) for overnight windows.
6. **Only `before_workflow` is enforced.** A long-running workflow that starts at 16:55 and finishes at 17:10 is not killed when business hours end. Combine with [Safety](./safety) `max_steps`/`max_tool_calls` for hard runtime caps.

## Next Steps

- [Time-of-Day Gating](./time-of-day-gating) -- Multiple windows, overnight windows, mid-execution enforcement
- [Operations Policy](./operations) -- Other operational guardrails
- [Policy Categories](../features/policy-categories)


--------------------------------------------------------------------------------
# Time-of-Day Gating Policy
URL: https://waxell.ai/docs/observe/governance/time-of-day-gating
Description: Window-of-allowed-use enforcement with multiple per-day windows, overnight windows, IANA timezones, and workflow-type exemptions. Maps to SOX 404, ISO 27001, HIPAA access controls.
--------------------------------------------------------------------------------

# Time-of-Day Gating Policy

The `time-of-day-gating` policy enforces window-of-allowed-use rules per agent / workflow. "Don't let the loan-decision agent fire after 6pm" is a common control in regulated workflows (financial services, healthcare staffing, retail fulfillment after market hours). Compared to [scheduling](./scheduling), this handler supports multiple windows per policy, overnight windows that span midnight, mid-execution enforcement, and workflow-type exemptions.

Maps to **SOX §404** IT general controls (segregation of access by time), **ISO 27001 A.9.4** (information access restriction), and **HIPAA §164.312(a)(1)** (access control).

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `allowed_windows` | object[] | `[]` | List of windows. Each: `{"days": ["mon","tue",...], "start": "HH:MM", "end": "HH:MM"}`. Empty list = no restriction (no-op) |
| `timezone` | string | `""` | IANA tz name (`America/Chicago`, `Europe/Berlin`). Empty = UTC |
| `applies_to_phase` | string[] | `["before_workflow"]` | Which phases to enforce in. Values: `before_workflow`, `mid_execution` |
| `exempt_workflow_types` | string[] | `[]` | Workflow types exempt from the gate (e.g. `["administrative"]`) |
| `action_on_violation` | string | `"block"` | Either `block` or `warn` |

Day names are lowercase 3-letter abbreviations: `mon`, `tue`, `wed`, `thu`, `fri`, `sat`, `sun`. Omitting `days` on a window means all 7 days.

## How It Works

The `time-of-day-gating` handler runs at the phases listed in `applies_to_phase`. It supports both `before_workflow` and `mid_execution` gates.

| Phase | What It Checks | Actions |
|-------|---------------|---------|
| `before_workflow` | If `applies_to_phase` includes `before_workflow`, checks current tz-aware time against all `allowed_windows` | BLOCK or WARN per `action_on_violation` |
| `mid_execution` | If `applies_to_phase` includes `mid_execution`, same check (useful for long-running workflows) | BLOCK or WARN |
| `after_workflow` | No-op (ALLOW) -- time-of-day is a gate, not an audit signal | ALLOW |

A run is allowed if **any** window matches. Windows where `start <= end` are same-day ranges (`09:00`-`18:00`). Windows where `start > end` are **overnight** (`22:00`-`06:00` means 22:00 today through 06:00 tomorrow).

### Context Attributes Read

| Attribute | Phase | Purpose |
|-----------|-------|---------|
| `context.workflow_type` / `context.workflow_kind` | both | Match against `exempt_workflow_types` |

The handler is otherwise time-only -- no input scanning or state inspection.

## Example Policy

```json
{
  "name": "Loan Agent Business Hours",
  "category": "time-of-day-gating",
  "rules": {
    "allowed_windows": [
      {"days": ["mon", "tue", "wed", "thu", "fri"], "start": "09:00", "end": "18:00"}
    ],
    "timezone": "America/Chicago",
    "applies_to_phase": ["before_workflow", "mid_execution"],
    "exempt_workflow_types": ["administrative", "monthly-close"],
    "action_on_violation": "block"
  },
  "scope": {"agents": ["loan-decision-agent"]},
  "enabled": true
}
```

### Overnight Window (Night-Shift Healthcare Staffing)

```json
{
  "name": "Night Shift Window",
  "category": "time-of-day-gating",
  "rules": {
    "allowed_windows": [
      {"days": ["mon", "tue", "wed", "thu", "fri", "sat", "sun"],
       "start": "22:00", "end": "06:00"}
    ],
    "timezone": "America/New_York",
    "action_on_violation": "block"
  }
}
```

## SDK Integration

```python

waxell.init()

@waxell.observe(
    agent_name="loan-decision-agent",
    workflow_type="decision",
    enforce_policy=True,
)
async def decide_loan(application_id: str) -> str:
    # before_workflow: blocks if outside 9am-6pm America/Chicago
    # mid_execution: re-checks every LLM/tool call (long-running runs
    # get cut off when business hours end)
    return await score_application(application_id)
```

## Observability

| Field | Example |
|-------|---------|
| **Category** | `time-of-day-gating` |
| **Action** | `block` |
| **Reason** | "Run attempted at 2026-05-28T19:42-05:00 (thu); outside all configured allowed windows." |
| **Metadata** | `{"phase": "before_workflow", "signal": "outside_allowed_window", "now": "2026-05-28T19:42-05:00", "day": "thu", "iso_27001": "A.9.4", "sox": "404", "hipaa": "164.312(a)(1)"}` |

## Common Gotchas

1. **Empty `allowed_windows` is a no-op ALLOW.** The handler exits early when no windows are configured -- this is intentional so disabling is one rule away, but means a typo-empty array silently disables enforcement.
2. **Day abbreviations are lowercase 3-letter.** `"Mon"` or `"monday"` will not match. The handler does compare case-insensitively against the canonical set, but `"weekdays"` is not understood.
3. **Invalid timezone falls back to UTC silently.** Like `scheduling`, an unparseable IANA name logs a debug message and uses UTC.
4. **`mid_execution` enforcement requires opt-in.** Default `applies_to_phase` is `["before_workflow"]` only -- a workflow started at 5:55pm will run to completion even if it crosses 6pm. Add `mid_execution` to the list to kill long-running runs at the boundary.
5. **`exempt_workflow_types` is case-insensitive.** `"Administrative"` and `"administrative"` both match. But it checks `context.workflow_type` OR `context.workflow_kind` -- if your runtime only sets `workflow_name`, the exemption never fires.
6. **Overnight windows wrap UTC midnight, not local midnight, when tz is empty.** A `22:00`-`06:00` window with no `timezone` is evaluated against UTC. For New-York-overnight semantics, set `timezone: "America/New_York"`.
7. **`short_circuit_on_block = True`.** When this handler blocks, other handlers in the chain are skipped. Useful for time-as-master-gate; surprising if you expected a downstream policy to also evaluate.

## Next Steps

- [Scheduling Policy](./scheduling) -- Simpler single-window variant with blackout dates and maintenance windows
- [Approval Policy](./approval) -- Require human override outside hours instead of hard-blocking
- [Compliance Policy](./compliance) -- Bundle time-of-day with PII, audit, and other controls per framework
- [Policy Categories](../features/policy-categories)


--------------------------------------------------------------------------------
# Safety Policy
URL: https://waxell.ai/docs/observe/governance/safety
Description: Content and behavior safety controls for workflow execution -- PII detection, credential scanning, profanity filtering, step/tool limits, blocked tools, and human approval requirements.
--------------------------------------------------------------------------------

# Safety Policy

The `safety` policy category enforces content and behavior safety controls on workflow execution. It covers:

- **Content filters** -- scan inputs and outputs for PII, credentials, and profanity
- **Execution limits** -- cap the number of steps and tool calls an agent can make
- **Tool restrictions** -- block specific tools or require human approval
- **Output limits** -- enforce maximum output length

Use it to prevent data leakage, runaway agents, and dangerous tool invocations.

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `max_retries` | integer | `3` | Maximum retry attempts on failure |
| `max_steps` | integer | `50` | Maximum workflow steps allowed |
| `max_tool_calls` | integer | `100` | Maximum tool invocations allowed |
| `blocked_tools` | string[] | `[]` | Tools that cannot be used (exact name match) |
| `require_human_approval` | boolean | `false` | Require approval before execution starts |
| `approval_tools` | string[] | `[]` | Tools that need human approval before invocation |
| `content_filters` | string[] | `[]` | Content types to scan: `pii`, `profanity`, `credentials` |
| `max_output_length` | integer | *(none)* | Maximum characters in final output |

## How It Works

The safety handler runs at **all three enforcement phases**: before_workflow, mid_execution, and after_workflow.

### Phase Behavior

| Phase | What It Checks | Actions |
|-------|---------------|---------|
| `before_workflow` | `require_human_approval`, content filters on `context.inputs` | BLOCK (approval), WARN (content) |
| `mid_execution` | `max_steps` vs step count, `max_tool_calls` vs tool count, content filters on `prompt_preview`/`response_preview` | BLOCK (limits), WARN (content) |
| `after_workflow` | Step/tool limits, `max_output_length`, content filters on final result | WARN (all violations) |

### Context Attributes Read

| Attribute | Phase | Purpose |
|-----------|-------|---------|
| `context.inputs` | before_workflow | Scan input text for PII/credentials/profanity |
| `context.step_logs` | mid_execution, after_workflow | Count steps taken (`len(step_logs)`) |
| `context.tool_call_count` | mid_execution, after_workflow | Count tool invocations |
| `context.prompt_preview` | mid_execution | Scan LLM prompts for content violations |
| `context.response_preview` | mid_execution | Scan LLM responses for content violations |
| `result` (parameter) | after_workflow | Scan final output, check output length |

## Content Filters

### PII Detection

The `pii` content filter uses regex patterns to detect personally identifiable information:

| PII Type | Pattern | Example Match |
|----------|---------|---------------|
| `ssn` | `\d{3}-\d{2}-\d{4}` | `123-45-6789` |
| `email` | Standard email regex | `user@example.com` |
| `phone` | US phone formats | `(555) 123-4567`, `+1-555-123-4567` |
| `credit_card` | 16-digit grouped by 4 | `4111-1111-1111-1111` |

### Credential Detection

The `credentials` content filter detects secrets and API keys:

| Pattern | What It Matches | Example |
|---------|----------------|---------|
| Password assignments | `password=`, `passwd=`, `pwd=` | `password=hunter2` |
| API key assignments | `api_key=`, `apikey=`, `api_secret=` | `api_key=abc123` |
| Secret/access keys | `secret_key=`, `access_key=` | `secret_key=xyz` |
| AWS access keys | `AKIA` prefix + 16 chars | `AKIAIOSFODNN7EXAMPLE` |
| Generic API tokens | `sk-`, `pk_live_`, `sk_live_`, `rk_live_` prefix + 20+ chars | `sk-proj-abc123...` |
| GitHub PATs | `ghp_` prefix + 36 chars | `ghp_abcdefghij...` |
| Waxell secret keys | `wax_sk_` prefix | `wax_sk_abc123` |

### Profanity Filter

The `profanity` content filter uses word-boundary matching against a hardcoded word set. Only whole words are matched (e.g., "class" does not trigger on "ass").

## Matching Examples

| Input | Filter | Match? | Why |
|-------|--------|--------|-----|
| `"Look up 123-45-6789"` | pii | Yes | SSN pattern matches |
| `"Send to user@co.com"` | pii | Yes | Email pattern matches |
| `"Call 555-1234"` | pii | No | Only 7 digits (phone needs 10) |
| `"api_key=sk-abc123456789012345678901"` | credentials | Yes | `sk-` prefix + 20+ chars |
| `"Use the skeleton key"` | credentials | No | `sk` not followed by `-` with 20+ chars |
| `"This damn report"` | profanity | Yes | Whole word match |
| `"The dam broke"` | profanity | No | "dam" is not "damn" |

**Content Filters Return WARN, Not BLOCK**
In the safety handler, content filter violations produce **WARN** actions, not BLOCK. The agent continues running. If you need content violations to block execution, use the dedicated [Content Policy](./content) instead, which supports configurable actions (warn, redact, block) per detection type.


## Execution Limits

### Step Limit (`max_steps`)

Checked at mid_execution and after_workflow by counting `len(context.step_logs)`. Returns **BLOCK** at mid_execution if exceeded.

### Tool Call Limit (`max_tool_calls`)

Checked at mid_execution and after_workflow via `context.tool_call_count`. Returns **BLOCK** at mid_execution if exceeded.

### Output Length (`max_output_length`)

Checked at after_workflow by measuring `len(str(result))`. Returns **WARN** if exceeded.

**Step/Tool Limits BLOCK at Mid-Execution**
Unlike content filters (which WARN), exceeding `max_steps` or `max_tool_calls` produces a **BLOCK** at mid_execution. This immediately halts the agent.


## Human Approval

### `require_human_approval`

When set to `true`, the handler returns **BLOCK** at before_workflow with reason "Human approval required before execution". The agent cannot run without external approval.

### `approval_tools`

Tools listed in `approval_tools` are blocked with reason "Tool '&#123;name&#125;' requires human approval" when checked via `check_tool_allowed()`. This is a standalone method meant to be called from the tool execution layer.

## Blocked Tools

Tools listed in `blocked_tools` are blocked with reason "Tool '&#123;name&#125;' is blocked by safety policy" when checked via `check_tool_allowed()`.

**check_tool_allowed Is Not Called Automatically**
The `check_tool_allowed(rules, tool_name)` method is a standalone API. It is NOT automatically invoked by the before_workflow, mid_execution, or after_workflow phase hooks. Your tool execution layer must call it explicitly.


## Example Policies

### PII-Only Content Filter

Scan for PII in inputs and outputs, warn on detection:

```json
{
  "content_filters": ["pii"],
  "max_steps": 50,
  "max_tool_calls": 100
}
```

### Full Safety Lockdown

All content filters, strict limits, blocked tools:

```json
{
  "max_retries": 2,
  "max_steps": 20,
  "max_tool_calls": 30,
  "blocked_tools": ["shell_exec", "file_write", "network_request"],
  "require_human_approval": false,
  "approval_tools": ["send_email", "make_purchase"],
  "content_filters": ["pii", "profanity", "credentials"],
  "max_output_length": 5000
}
```

### Approval-Required for Production

Require human approval before any execution:

```json
{
  "require_human_approval": true,
  "content_filters": ["pii", "credentials"],
  "max_steps": 100,
  "max_tool_calls": 200
}
```

## SDK Integration

### Using the Context Manager

```python

from waxell_observe.errors import PolicyViolationError

waxell.init()

try:
    async with waxell.WaxellContext(
        agent_name="research-agent",
        enforce_policy=True,
    ) as ctx:
        # before_workflow: safety checks inputs,
        #   require_human_approval
        # If content filter triggers -> WARN (agent continues)
        # If require_human_approval -> BLOCK (PolicyViolationError)

        result = await do_research(query)

        # Record tool calls (increments tool_call_count)
        ctx.record_tool_call(
            name="web_search",
            input={"query": query},
            output={"results": results},
        )

        ctx.set_result(result)
        # after_workflow: checks limits and output content

except PolicyViolationError as e:
    print(f"Safety block: {e}")
```

### Using the Decorator

```python
@waxell.observe(
    agent_name="research-agent",
    enforce_policy=True,
)
async def run_research(query: str):
    # Safety checks happen before and after this function
    return await do_research(query)
```

## Enforcement Flow

```
Agent starts (WaxellContext.__aenter__)
    |
    +-- before_workflow
    |   |
    |   +-- require_human_approval? -> BLOCK
    |   |
    |   +-- content_filters on context.inputs
    |       +-- PII detected? -> WARN
    |       +-- Credential detected? -> WARN
    |       +-- Profanity detected? -> WARN
    |
    +-- Agent executes steps...
    |
    +-- mid_execution (per LLM call)
    |   |
    |   +-- step_logs > max_steps? -> BLOCK
    |   +-- tool_call_count > max_tool_calls? -> BLOCK
    |   +-- content_filters on prompt_preview/response_preview
    |       +-- Violations? -> WARN
    |
    +-- Agent finishes
    |
    +-- after_workflow
        |
        +-- step_logs > max_steps? -> WARN
        +-- tool_call_count > max_tool_calls? -> WARN
        +-- len(result) > max_output_length? -> WARN
        +-- content_filters on result -> WARN
```

## Creating via Dashboard

1. Navigate to **Governance > Policies**
2. Click **New Policy**
3. Select category **Safety**
4. Configure limits (max_steps, max_tool_calls)
5. Enable content filters (pii, profanity, credentials)
6. Optionally add blocked_tools and approval_tools
7. Set scope to target specific agents
8. Enable

## Creating via API

```bash
curl -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  https://acme.waxell.dev/waxell/v1/policies/ \
  -d '{
    "name": "Research Safety Policy",
    "category": "safety",
    "rules": {
      "max_retries": 3,
      "max_steps": 50,
      "max_tool_calls": 100,
      "blocked_tools": ["dangerous_tool", "shell_exec"],
      "content_filters": ["pii", "profanity", "credentials"],
      "max_output_length": 5000
    },
    "scope": {
      "agents": ["research-agent"]
    },
    "enabled": true
  }'
```

## Observability

### Governance Tab

Safety evaluations appear with:

| Field | Example (ALLOW) |
|-------|---------|
| **Policy name** | Research Safety Policy |
| **Action** | `allow` |
| **Category** | `safety` |
| **Reason** | "Safety checks passed (content filters active: pii, profanity, credentials)" |

For content violations:

| Field | Example (WARN) |
|-------|---------|
| **Action** | `warn` |
| **Reason** | "Input content violations: PII detected: ssn" |
| **Metadata** | `{"content_violations": ["PII detected: ssn"], "scan_target": "inputs"}` |

For limit violations:

| Field | Example (BLOCK) |
|-------|---------|
| **Action** | `block` |
| **Reason** | "Mid-run: step limit exceeded (55/50)" |
| **Metadata** | `{"steps": 55, "limit": 50}` |

## Combining with Other Policies

- **Safety + Compliance**: HIPAA compliance often requires PII filtering. Use a compliance policy requiring `safety` as a sibling category, with `content_filters: ["pii"]` as a required rule
- **Safety + Kill Switch**: Use kill switch for emergency stop, safety for ongoing limits
- **Safety + Content**: Safety content filters return WARN. For BLOCK on content violations, add a dedicated content policy with `pii_detection.action: "block"`

## Common Gotchas

1. **Content filters are regex-based.** They can false-positive on patterns that look like SSNs but aren't (e.g., formatted dates like `2024-01-2345`).

2. **Content filters return WARN, not BLOCK.** Safety content violations produce WARN actions. The agent continues running. Use the dedicated [Content Policy](./content) for configurable block/warn/redact actions.

3. **`blocked_tools` requires exact name match.** `"shell"` does not block `"shell_exec"`. Use the full tool name.

4. **`max_output_length` checks `str(result)`.** This includes Python repr overhead (quotes, braces for dicts). The actual content may be shorter than the measured length.

5. **`check_tool_allowed` is not called automatically.** It's a standalone method for your tool execution layer. The phase hooks don't check blocked_tools.

6. **Mid-execution requires the runtime to call the handler.** Observe-path agents may not trigger mid_execution checks between steps. Step/tool limits are also checked at after_workflow as a fallback.

7. **`require_human_approval` blocks ALL queries.** It does not inspect the query. Every execution is blocked until approval is granted externally.

8. **Profanity filter uses word boundaries.** "class" does not trigger on "ass". But compound words without separators may not match as expected.

## Next Steps

- [Content Policy](./content) -- Dedicated content scanning with block/warn/redact actions
- [Policy & Governance](../features/governance) -- How policy enforcement works
- [Compliance Policy](./compliance) -- Meta-validator for regulatory frameworks
- [Policy Categories & Templates](../features/policy-categories) -- All 26 categories


--------------------------------------------------------------------------------
# Kill Switch (Circuit Breaker) Policy
URL: https://waxell.ai/docs/observe/governance/kill-switch
Description: Emergency stop controls with automatic circuit breaker functionality -- track error rates, auto-disable failing agents, and recover automatically with Redis-backed state.
--------------------------------------------------------------------------------

# Kill Switch (Circuit Breaker) Policy

The `kill` policy category implements emergency stop controls with automatic circuit breaker functionality. It tracks success/failure rates over a configurable time window and automatically disables an agent when the error rate exceeds a threshold.

Use it when you need to:
- Prevent cascading failures from a malfunctioning agent
- Protect downstream services from repeated bad requests
- Implement emergency stop controls for production agents
- Auto-disable agents that start producing errors at an unacceptable rate

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `enabled` | bool | `true` | Enable kill switch monitoring |
| `kill_on_error_rate` | float (0-1) | `0.5` | Activate kill switch when error rate exceeds this threshold |
| `error_window_minutes` | int | `5` | Time window (minutes) for calculating error rate. Counters expire after this |
| `min_samples` | int | `10` | Minimum total executions before evaluating error rate |
| `auto_recover_after_minutes` | int | `30` | Automatically deactivate kill switch after this duration |

## How It Works

The kill switch follows the **circuit breaker** pattern with three states:

```
CLOSED (normal operation)
    |
    +-- Error rate exceeds threshold
    |   (and min_samples reached)
    |
    v
OPEN (kill switch active -- all executions blocked)
    |
    +-- auto_recover_after_minutes TTL expires
    |
    v
CLOSED (normal operation resumes)
```

### Error Rate Calculation

```
error_rate = errors / (successes + errors)
```

The error rate is calculated from Redis counters within the `error_window_minutes` window. Both success and error counters have a TTL equal to `error_window_minutes * 60` seconds, so stale data automatically expires.

### min_samples Guard

The error rate is only evaluated when the total number of executions (successes + errors) reaches `min_samples`. This prevents premature activation when a single early error would produce a 100% error rate.

### Auto-Recovery

When the kill switch activates, it sets a Redis key with a TTL of `auto_recover_after_minutes * 60` seconds. Once this TTL expires, the key disappears and the next execution is allowed through. If errors continue after recovery, the kill switch re-activates immediately (assuming the error counters haven't expired yet).

### Redis Keys

| Key Pattern | Purpose | TTL |
|-------------|---------|-----|
| `killswitch:<agent>:<workflow>:active` | Kill switch activation flag | `auto_recover_after_minutes * 60` |
| `stats:<agent>:<workflow>:success` | Success counter | `error_window_minutes * 60` |
| `stats:<agent>:<workflow>:error` | Error counter | `error_window_minutes * 60` |

### Enforcement Phases

| Phase | Behavior |
|-------|----------|
| `before_workflow` | Checks if kill switch is active (Redis key). Checks error rate against threshold. If tripped, activates kill switch and returns BLOCK |
| `mid_execution` | Not implemented |
| `after_workflow` | Increments success counter (with window TTL) |
| `on_failure` | Increments error counter (with window TTL) |

## Example Policies

### Conservative (Production Default)

High threshold, many samples, long recovery:

```json
{
  "enabled": true,
  "kill_on_error_rate": 0.5,
  "error_window_minutes": 5,
  "min_samples": 10,
  "auto_recover_after_minutes": 30
}
```

### Aggressive (Fast Detection)

Low threshold, few samples, quick recovery:

```json
{
  "enabled": true,
  "kill_on_error_rate": 0.3,
  "error_window_minutes": 3,
  "min_samples": 5,
  "auto_recover_after_minutes": 5
}
```

### Production-Grade (High Sensitivity)

Moderate threshold, large sample size, long recovery for critical agents:

```json
{
  "enabled": true,
  "kill_on_error_rate": 0.4,
  "error_window_minutes": 10,
  "min_samples": 20,
  "auto_recover_after_minutes": 60
}
```

## SDK Integration

### Using the Context Manager

```python

from waxell_observe.errors import PolicyViolationError

waxell.init()

try:
    async with waxell.WaxellContext(
        agent_name="processor",
        workflow_name="data-pipeline",
        enforce_policy=True,
    ) as ctx:
        # If kill switch is active, PolicyViolationError
        # is raised here (before any agent work happens)

        result = await process_data(query)
        ctx.set_result(result)
        # after_workflow increments success counter

except PolicyViolationError as e:
    print(f"Kill switch: {e}")
    # e.g. "Kill switch activated - error rate 83% exceeds 50%"
    # or   "Kill switch active - workflow temporarily disabled"

except Exception as e:
    # on_failure increments error counter
    # If error rate now exceeds threshold, next execution will be blocked
    raise
```

### Using the Decorator

```python
@waxell.observe(
    agent_name="processor",
    workflow_name="data-pipeline",
    enforce_policy=True,
)
async def run_pipeline(query: str):
    # Kill switch check happens before this function body runs
    # Exceptions raised here trigger on_failure (error counter)
    # Normal return triggers after_workflow (success counter)
    return await process_data(query)
```

## Enforcement Flow

```
Agent starts (WaxellContext.__aenter__ or decorator entry)
    |
    +-- before_workflow governance runs
    |   |
    |   +-- enabled=false? -> ALLOW (skip all checks)
    |   |
    |   +-- Kill switch Redis key exists?
    |   |   +-- Yes -> BLOCK ("Kill switch active - workflow temporarily disabled")
    |   |
    |   +-- Read success + error counters from Redis
    |   |   +-- total < min_samples? -> ALLOW (not enough data)
    |   |
    |   +-- Calculate error_rate = errors / total
    |       +-- error_rate < threshold? -> ALLOW
    |       +-- error_rate >= threshold?
    |           -> Set kill switch Redis key (with auto_recover TTL)
    |           -> BLOCK ("Kill switch activated - error rate X% exceeds Y%")
    |
    +-- Agent executes...
    |
    +-- Success path (after_workflow)
    |   +-- Increment success counter (with error_window TTL)
    |
    +-- Failure path (on_failure)
        +-- Increment error counter (with error_window TTL)
```

**Redis Required**
Kill switch requires Redis for error rate tracking and activation state. When running with `WAXELL_OBSERVE=false` or without a live server connection, the kill switch is not enforced -- errors are not tracked and the circuit breaker never trips. This is by design for local development.


## Manual Controls

The kill switch handler exposes methods for programmatic control:

```python
# Activate kill switch manually (e.g., from an ops dashboard)
handler.activate_kill_switch(context, duration_minutes=30, reason="manual")

# Deactivate kill switch manually
handler.deactivate_kill_switch(context)
```

Or via Redis CLI:

```bash
# Activate for 2 minutes
redis-cli SET "killswitch:my-agent:my-workflow:active" "manual" EX 120

# Deactivate
redis-cli DEL "killswitch:my-agent:my-workflow:active"

# Check status
redis-cli EXISTS "killswitch:my-agent:my-workflow:active"
redis-cli TTL "killswitch:my-agent:my-workflow:active"
```

## Creating via Dashboard

1. Navigate to **Governance > Policies**
2. Click **New Policy**
3. Select category **Kill**
4. Configure error rate threshold, sample size, and recovery time
5. Set scope to target specific agents (e.g., `kill-switch-agent`)
6. Enable

## Creating via API

```bash
curl -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  https://acme.waxell.dev/waxell/v1/policies/ \
  -d '{
    "name": "Circuit Breaker",
    "category": "kill",
    "rules": {
      "enabled": true,
      "kill_on_error_rate": 0.5,
      "error_window_minutes": 5,
      "min_samples": 10,
      "auto_recover_after_minutes": 30
    },
    "scope": {
      "agents": ["processor"]
    },
    "enabled": true
  }'
```

## Observability

### Governance Tab

Kill switch evaluations appear with:

**When kill switch activates (error rate exceeded):**

| Field | Example |
|-------|---------|
| **Policy name** | Circuit Breaker |
| **Action** | `block` |
| **Category** | `kill` |
| **Reason** | "Kill switch activated - error rate 83% exceeds 50%" |
| **Metadata** | `{"error_rate": 0.83, "threshold": 0.5, "auto_recover_minutes": 30}` |

**When kill switch is already active:**

| Field | Example |
|-------|---------|
| **Reason** | "Kill switch active - workflow temporarily disabled" |
| **Metadata** | `{"kill_switch": true, "auto_recover_seconds": 1742}` |

**When under threshold:**

| Field | Example |
|-------|---------|
| **Action** | `allow` |
| **Reason** | "Kill switch not activated" |

## Combining with Other Policies

**Kill Switch + Rate Limit**: Defense in depth. Rate limits prevent overuse under normal conditions. If errors spike despite rate limiting, the kill switch provides a hard stop.

**Kill Switch + Safety**: If a safety policy detects unsafe output but uses warn mode, errors from downstream failures can trigger the kill switch to stop the agent entirely.

**Kill Switch + Compliance**: A SOC 2 compliance policy can require that kill switch monitoring is configured as part of operational safety requirements.

## Common Gotchas

1. **Error counters expire after `error_window_minutes`.** Stale errors from hours ago do not count toward the current error rate. If errors stop, the counters naturally decay to zero.

2. **`min_samples` is total (success + error), not just errors.** With `min_samples=10`, you need at least 10 total executions before the error rate is evaluated. A single error out of 1 total will not trigger the kill switch.

3. **Auto-recovery resets the kill switch but does NOT reset error counters.** If the error counters have not expired (still within `error_window_minutes`), the error rate may still be above threshold after recovery. The next execution will immediately re-activate the kill switch.

4. **Error counters and kill switch key have independent TTLs.** The kill switch key expires after `auto_recover_after_minutes`. The error counters expire after `error_window_minutes`. These are typically different values.

5. **Tenant-scoped.** One tenant's kill switch does not affect other tenants. Each tenant has independent Redis key namespaces.

6. **Process crashes leave no error record.** If a process crashes before `on_failure` runs, the error is not counted. The kill switch only tracks errors that are caught and reported through the governance hooks.

7. **`enabled: false` skips all checks.** Setting `enabled` to `false` disables both the kill switch check and the success/error counting. No data is recorded while disabled.

8. **Kill switch scoping is per agent+workflow.** A kill switch on `processor:data-pipeline` does not affect `processor:report-generation`. Each combination has independent counters and activation state.

## Next Steps

- [Policy & Governance](../features/governance) -- How policy enforcement works
- [Rate Limit Policy](./rate-limit) -- Execution frequency limits
- [Compliance Policy](./compliance) -- Meta-validator for regulatory frameworks
- [Policy Categories & Templates](../features/policy-categories) -- All 26 categories


--------------------------------------------------------------------------------
# Audit Policy
URL: https://waxell.ai/docs/observe/governance/audit
Description: Configurable audit logging for workflow execution -- input/output/step/tool-call logging with field redaction and retention controls.
--------------------------------------------------------------------------------

# Audit Policy

The `audit` policy category configures **what gets logged** during workflow execution and how long those logs are retained. Unlike most policies, it never blocks -- it is a *must-record* handler that runs even when an earlier handler has already blocked the run, so the audit trail captures the blocked attempt itself.

Use it to satisfy regulatory log-retention requirements (SOC2, ISO 27001, HIPAA audit controls) and to standardize redaction of sensitive fields across agents.

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `log_inputs` | boolean | `true` | Log workflow inputs (after redaction) |
| `log_outputs` | boolean | `true` | Log workflow outputs (after redaction) |
| `log_steps` | boolean | `true` | Log individual workflow step counts |
| `log_tool_calls` | boolean | `true` | Log tool invocation counts |
| `redact_fields` | string[] | `[]` | Additional field names to redact (merged with defaults) |
| `retention_days` | integer | `90` | How long to retain audit logs |
| `min_log_level` | string | `"INFO"` | Minimum Python logging level (`DEBUG`/`INFO`/`WARNING`/`ERROR`) |
| `max_log_entries` | integer | `500` | Max log entries per execution (`0` = unlimited) |
| `capture_stdout` | boolean | `false` | Capture `print()` output as log entries |

### Default Redacted Fields

These are always redacted, regardless of `redact_fields` configuration:

`password`, `secret`, `token`, `api_key`, `apikey`, `authorization`, `credential`, `private_key`

Matching is **case-insensitive substring** -- a field named `user_password_hash` will be redacted because it contains `password`.

## How It Works

The `audit` handler runs at **before_workflow**, **after_workflow**, and **on_failure**. It always returns `ALLOW`; the action is purely informational. Because `short_circuit_on_block = False`, the handler executes even if a prior policy already blocked the run.

| Phase | What It Logs |
|-------|--------------|
| `before_workflow` | Redacted inputs, agent/workflow IDs, active audit modes |
| `after_workflow` | Step count, tool call count, redacted output |
| `on_failure` | Error type and message |

### Context Attributes Read

| Attribute | Phase | Purpose |
|-----------|-------|---------|
| `context.inputs` | before_workflow | Redact + log inputs |
| `context.agent_name` | all | Audit log scoping |
| `context.workflow_name` | all | Audit log scoping |
| `context.workflow_id` | all | Audit log correlation |
| `context.step_logs` | after_workflow | Count steps (`len(step_logs)`) |
| `context.tool_call_count` | after_workflow | Count tool invocations |
| `result` (parameter) | after_workflow | Redact + log final output |

The handler also writes `context._audit_rules` so downstream components can read the active config.

## Example Policy

```json
{
  "log_inputs": true,
  "log_outputs": true,
  "log_steps": true,
  "log_tool_calls": true,
  "redact_fields": ["ssn", "dob", "patient_id"],
  "retention_days": 365,
  "min_log_level": "INFO",
  "max_log_entries": 1000,
  "capture_stdout": false
}
```

## SDK Integration

```python

waxell.init()

@waxell.observe(agent_name="claims-agent", enforce_policy=True)
async def process_claim(claim: dict) -> dict:
    return await adjudicate(claim)
```

Inputs and outputs are automatically logged + redacted on entry/exit. No SDK calls are required to opt in -- assigning an `audit` policy to the agent is enough.

## Observability

| Field | Example |
|-------|---------|
| **Category** | `audit` |
| **Action** | `allow` |
| **Reason** | "Audit logging active (inputs, outputs, steps, tool calls)" |
| **Metadata** | `{"log_config": {"min_log_level": "INFO", "max_log_entries": 500, "capture_stdout": false}, "audit_rules": {...}}` |

## Common Gotchas

1. **`audit` never blocks.** It is a *must-record* handler -- it returns `ALLOW` even on failure. Combine with `kill-switch` or `safety` for blocking behavior.

2. **Redaction is substring-based.** `redact_fields: ["id"]` will redact ANY field whose name contains `id` (including `client_id`, `request_id`). Use explicit names like `"customer_id"` to scope the match.

3. **Defaults are always merged.** You cannot disable redaction of `password`/`token`/`api_key` by leaving `redact_fields` empty. The defaults list is hardcoded.

4. **`retention_days` is enforced by infra, not the handler.** The rule is surfaced in the audit log emission but actual retention is governed by the telemetry pipeline (S3 lifecycle, OpenSearch ILM).

5. **`max_log_entries: 0` means unlimited.** This is the documented sentinel -- do not treat it as "log nothing".

6. **`capture_stdout` is opt-in.** `print()` calls are NOT captured by default; agents that rely on print debugging will produce no audit output without this flag.

## Next Steps

- [Privacy Policy](./privacy) -- Redaction of PII before logging
- [Compliance Policy](./compliance) -- Meta-validator for SOC2/HIPAA/ISO frameworks
- [Policy Categories](../features/policy-categories) -- All categories


--------------------------------------------------------------------------------
# Operations Policy
URL: https://waxell.ai/docs/observe/governance/operations
Description: Enforce operational controls on workflow execution -- timeout monitoring with post-hoc warnings for SLA compliance and performance tracking.
--------------------------------------------------------------------------------

# Operations Policy

The `operations` policy category enforces **operational controls** on workflow execution. Currently it has a single rule: `timeout_seconds`. The handler monitors run duration and generates warnings when agents exceed configured time limits.

Use it when you need SLA compliance monitoring, performance alerting, or tracking slow-running agents.

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `timeout_seconds` | integer (min 1) | `300` | Maximum allowed run duration in seconds. Checked post-execution for observed agents |

## How It Works

**Post-Hoc Enforcement**
Operations does **NOT** preemptively kill agents. The agent always runs to completion. Timeout violations are detected after the fact and recorded as governance incidents (WARN). This is correct for observe-path agents where external frameworks control execution.


### Enforcement Phases

| Phase | Behavior |
|-------|----------|
| `before_workflow` | Stores timeout in context. Returns ALLOW with "Timeout set to Xs" |
| `mid_execution` | Checks `context.duration` if available. Returns WARN if exceeded, ALLOW otherwise |
| `after_workflow` | Final duration check. Returns WARN if exceeded, ALLOW otherwise |

### Context Data

| Attribute | Phase | Purpose |
|-----------|-------|---------|
| `context.duration` | mid_execution, after_workflow | Elapsed time of the workflow run (float, seconds) |

### Actions Returned

- **ALLOW** -- duration within timeout, or no timeout configured, or duration not available
- **WARN** -- duration exceeds timeout_seconds

The handler **never returns BLOCK**. Timeout violations are always WARN. This is by design -- for observe-path agents, the execution has already happened.

## Example Policies

### Strict SLA (60 seconds)

Alert on any run exceeding 1 minute:

```json
{
  "timeout_seconds": 60
}
```

### Standard Monitoring (5 minutes)

Default timeout with monitoring:

```json
{
  "timeout_seconds": 300
}
```

### Long-Running Batch (1 hour)

For batch processing agents that legitimately run longer:

```json
{
  "timeout_seconds": 3600
}
```

## SDK Integration

### Using the Context Manager

```python


waxell.init()

async with waxell.WaxellContext(
    agent_name="data-processor",
    enforce_policy=True,
) as ctx:
    # Operations policy stores timeout at before_workflow
    # Agent runs normally -- no blocking

    result = await process_data(query)
    ctx.set_result(result)

    # after_workflow checks: if total duration > timeout_seconds -> WARN
    # WARN is recorded but does NOT raise PolicyViolationError
```

### Using the Decorator

```python
@waxell.observe(
    agent_name="data-processor",
    enforce_policy=True,
)
async def process_data(query: str):
    # Operations checks happen after this function returns
    return await long_running_analysis(query)
```

## Enforcement Flow

```
Agent starts (WaxellContext.__aenter__)
    |
    +-- before_workflow: stores timeout (e.g., 60s) in context
    |   -> ALLOW "Timeout set to 60s"
    |
    +-- Agent executes (duration tracked automatically)
    |   |
    |   +-- mid_execution (if triggered):
    |   |   -> duration < timeout? ALLOW "Within timeout (30.0s/60s)"
    |   |   -> duration > timeout? WARN "Mid-run: approaching timeout (75.0s/60s)"
    |   |
    |   +-- Agent continues regardless of mid_execution result
    |
    +-- Agent completes
    |
    +-- after_workflow: final duration check
        -> duration < timeout? ALLOW "Completed within timeout (45.0s/60s)"
        -> duration > timeout? WARN "Run exceeded timeout (120.0s/60s)"
        -> WARN recorded as governance incident
```

## Creating via Dashboard

1. Navigate to **Governance > Policies**
2. Click **New Policy**
3. Select category **Operations**
4. Set `timeout_seconds` to your desired limit
5. Set scope to target specific agents (e.g., `data-processor`)
6. Enable

## Creating via API

```bash
curl -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  https://acme.waxell.dev/waxell/v1/policies/ \
  -d '{
    "name": "SLA Timeout Monitor",
    "category": "operations",
    "rules": {
      "timeout_seconds": 60
    },
    "scope": {
      "agents": ["data-processor"]
    },
    "enabled": true
  }'
```

## Observability

### Governance Tab

Operations evaluations appear with:

| Field | Example |
|-------|---------|
| **Policy name** | SLA Timeout Monitor |
| **Action** | `allow` or `warn` |
| **Category** | `operations` |
| **Reason** | "Completed within timeout (45.0s/60s)" or "Run exceeded timeout (120.0s/60s)" |
| **Metadata** | `{"duration": 120.0, "timeout": 60}` |

### Governance Incidents

Timeout violations create governance incidents visible in:
- The trace's Governance tab
- The Governance Incidents list
- Compliance Console (if Insights is enabled)

## Combining with Other Policies

**Operations + Kill Switch**: Use operations timeout warnings to feed into kill switch error rate monitoring. Repeated timeout violations may indicate an agent that should be killed.

**Operations + Control**: Combine timeout monitoring with cost threshold monitoring. Long-running agents often also consume more LLM tokens (higher cost).

**Operations + Compliance**: Include operations as a required category in a SOC 2 compliance profile to ensure all agents have timeout monitoring configured.

## Common Gotchas

1. **Returns WARN, never BLOCK.** The agent always completes. Timeout violations are informational -- they create governance incidents but do not prevent execution.

2. **`context.duration` may be None.** If mid_execution fires before the duration attribute is populated, the handler returns ALLOW with a generic reason. Duration is reliably set by the time `after_workflow` runs.

3. **Duration is calculated when WaxellContext closes.** For observe-path agents, the SDK measures elapsed time between `__aenter__` and `__aexit__`. This includes all LLM calls, tool calls, and any processing time.

4. **Default timeout is 300 seconds (5 minutes).** If you configure an operations policy without specifying `timeout_seconds`, it defaults to 300s.

5. **Simulated duration in dry-run requires manual setting.** Demo agents set `context._duration_override` to simulate elapsed time. In production, actual wall-clock time is used automatically.

## Next Steps

- [Policy & Governance](../features/governance) -- How policy enforcement works
- [LLM Policy](./llm) -- Model allowlists and token limits
- [Quality Policy](./quality) -- Output quality validation
- [Policy Categories & Templates](../features/policy-categories) -- All 26 categories


--------------------------------------------------------------------------------
# LLM Policy
URL: https://waxell.ai/docs/observe/governance/llm
Description: Govern which LLM models agents can use, enforce token budgets, and validate model compliance with allowlists and blocklists -- including versioned model name matching.
--------------------------------------------------------------------------------

# LLM Policy

The `llm` policy category governs **which LLM models** an agent can use and enforces **token budgets**. It inspects model names and cumulative token counts reported during execution via `ctx.record_llm_call()`.

Use it when you need to restrict agents to approved models, block deprecated or expensive models, or cap token usage per run.

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `allowed_models` | string[] | `[]` (unrestricted) | Allowlist of model name patterns. If non-empty, only these models may be used |
| `blocked_models` | string[] | `[]` | Blocklist of model name patterns. Checked before allowlist |
| `max_tokens_per_call` | integer (min 1) | None | Maximum cumulative tokens across all LLM calls in a run |
| `temperature_range` | object `{min, max}` | None | Allowed temperature range (0-2). **Informational only -- not enforced** |
| `action_on_token_violation` | string | `"warn"` | Action when `max_tokens_per_call` is exceeded: `"warn"` or `"block"` |
| `require_system_prompt` | boolean | `false` | Whether a system prompt is required. **Informational only -- not enforced** |

## How Model Matching Works

The LLM handler supports **versioned variant matching**. When OpenAI returns `gpt-4o-mini-2024-07-18` as the actual model ID, it matches the pattern `gpt-4o-mini` because the full name starts with the pattern followed by a `-` or `:` separator.

| Model Name | Pattern | Match? | Why |
|-----------|---------|--------|-----|
| `gpt-4o-mini` | `gpt-4o-mini` | Yes | Exact match |
| `gpt-4o-mini-2024-07-18` | `gpt-4o-mini` | Yes | Versioned variant (dash separator) |
| `claude-3-opus:latest` | `claude-3-opus` | Yes | Versioned variant (colon separator) |
| `gpt-4o` | `gpt-4o-mini` | No | Not a prefix match |
| `gpt-4o-mini` | `gpt-4o` | No | `gpt-4o-mini` starts with `gpt-4o` but next char is `-m`, not a version separator for `gpt-4o` -- wait, actually `gpt-4o-mini` does start with `gpt-4o` + `-`. This IS a match |

**Versioned Variant Matching**
A model name matches a pattern if: (a) they are exactly equal, or (b) the model starts with the pattern AND the next character is `-` or `:`. This means `gpt-4o-mini` matches the pattern `gpt-4o` because `mini` follows a dash. Design your patterns carefully -- use the full base model name to avoid unintended matches.


## How It Works

### Enforcement Phases

| Phase | Behavior |
|-------|----------|
| `before_workflow` | Stores rules in context. Always returns ALLOW |
| `mid_execution` | Checks `models_used` against blocked/allowed lists (BLOCK). Checks `tokens_used` against limit (`action_on_token_violation`: WARN or BLOCK) |
| `after_workflow` | Final audit: same checks as mid_execution but returns WARN (not BLOCK) for model violations |

### Evaluation Order

1. **Blocked models checked first.** If a model matches `blocked_models`, return BLOCK immediately
2. **Allowed models checked second.** If `allowed_models` is non-empty and the model does not match, return BLOCK
3. **Token limit checked last.** If cumulative tokens exceed `max_tokens_per_call`, return `action_on_token_violation` (WARN by default, BLOCK if configured)

### Context Data

| Context Attribute | Type | Source |
|-------------------|------|--------|
| `models_used` | list[str] | Populated from `LlmCallRecord` entries in the DB (distinct models across all calls in this run) |
| `tokens_used` | int | Sum of `total_tokens` from all `LlmCallRecord` entries for this run |

## Example Policies

### Model Allowlist Only

Restrict to approved OpenAI models:

```json
{
  "allowed_models": ["gpt-4o-mini", "gpt-4o"],
  "blocked_models": [],
  "max_tokens_per_call": null
}
```

### Model Blocklist Only

Block deprecated models, allow everything else:

```json
{
  "allowed_models": [],
  "blocked_models": ["gpt-3.5-turbo", "text-davinci-003"],
  "max_tokens_per_call": null
}
```

### Token Budget Enforcement

Cap total token usage per run:

```json
{
  "allowed_models": [],
  "blocked_models": [],
  "max_tokens_per_call": 4000
}
```

### Combined Restrictive Policy

Full governance: approved models only, block deprecated, enforce token budget:

```json
{
  "allowed_models": ["gpt-4o-mini", "gpt-4o"],
  "blocked_models": ["gpt-3.5-turbo"],
  "max_tokens_per_call": 8000,
  "action_on_token_violation": "block"
}
```

## SDK Integration

### Using the Decorator

When using supported LLM frameworks (OpenAI, LangChain, Anthropic, etc.), the SDK automatically intercepts LLM calls and triggers mid_execution governance. No manual recording is needed.

```python

from waxell_observe.errors import PolicyViolationError

waxell.init()

try:
    @waxell.observe(
        agent_name="analyst",
        enforce_policy=True,
    )
    async def run_analysis(query: str):
        # LLM calls are auto-intercepted by instrumentors
        # Mid-execution governance fires automatically after each call
        return await call_llm(query)

    result = await run_analysis("Analyze sales data")

except PolicyViolationError as e:
    print(f"LLM policy block: {e}")
    # e.g. "Blocked model 'gpt-3.5-turbo' was used"
    # e.g. "Model 'claude-3-opus' is not in allowed list: gpt-4o-mini, gpt-4o"
```

### Using the Context Manager

```python
try:
    async with waxell.WaxellContext(
        agent_name="analyst",
        enforce_policy=True,
    ) as ctx:
        # Auto-instrumented LLM calls trigger governance automatically
        result = await analyze_data(query)
        ctx.set_result(result)

except PolicyViolationError as e:
    print(f"LLM policy block: {e}")
```

## Enforcement Flow

```
Agent starts (WaxellContext.__aenter__)
    |
    +-- before_workflow: stores LLM rules in context
    |
    +-- Agent makes LLM calls
    |   |
    |   +-- Auto-instrumented call (OpenAI, LangChain, etc.)
    |   |   -> mid_execution fires automatically:
    |   |      -> blocked_models check (BLOCK if match)
    |   |      -> allowed_models check (BLOCK if not in list)
    |   |      -> token limit check (action_on_token_violation: WARN or BLOCK)
    |   |
    |   +-- Model "gpt-3.5-turbo" used (blocked)
    |       -> BLOCK: "Blocked model 'gpt-3.5-turbo' was used"
    |
    +-- Agent completes
    |
    +-- after_workflow: final audit
        -> Same model/token checks, but returns WARN (not BLOCK)
        -> Warnings recorded in governance tab
```

## Creating via Dashboard

1. Navigate to **Governance > Policies**
2. Click **New Policy**
3. Select category **LLM**
4. Configure allowed_models, blocked_models, max_tokens_per_call
5. Set scope to target specific agents (e.g., `llm-governed-agent`)
6. Enable

## Creating via API

```bash
curl -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  https://acme.waxell.dev/waxell/v1/policies/ \
  -d '{
    "name": "LLM Model Governance",
    "category": "llm",
    "rules": {
      "allowed_models": ["gpt-4o-mini", "gpt-4o"],
      "blocked_models": ["gpt-3.5-turbo"],
      "max_tokens_per_call": 8000
    },
    "scope": {
      "agents": ["analyst"]
    },
    "enabled": true
  }'
```

## Observability

### Governance Tab

LLM evaluations appear with:

| Field | Example |
|-------|---------|
| **Policy name** | LLM Model Governance |
| **Action** | `allow`, `warn`, or `block` |
| **Category** | `llm` |
| **Reason** | "Models approved: gpt-4o-mini" or "Blocked model 'gpt-3.5-turbo' was used" |
| **Metadata** | `{"blocked_model": "gpt-3.5-turbo"}` or `{"tokens_used": 5000, "limit": 4000}` |

### After-Workflow Audit

The `after_workflow` phase runs a final model and token audit. Model violations at this stage return WARN (not BLOCK), since the execution has already completed. Token limit violations also return WARN.

## Common Gotchas

1. **`temperature_range` and `require_system_prompt` are NOT enforced.** They are informational only -- the handler does not check these values. They exist in the schema for documentation purposes.

2. **Versioned model names match patterns.** `gpt-4o-mini-2024-07-18` matches the pattern `gpt-4o-mini`. This is intentional -- API providers return versioned model IDs even when you request the base model name.

3. **`allowed_models` is only checked when non-empty.** An empty `allowed_models` list means all models are allowed. Only a non-empty list activates the allowlist.

4. **`blocked_models` is checked BEFORE `allowed_models`.** A model in both lists is blocked. The blocklist always takes priority.

5. **`mid_execution` returns BLOCK for model violations.** This is hardcoded -- there is no warn mode for model violations at mid_execution. The `after_workflow` phase returns WARN for the same violations.

6. **Token limit action is configurable.** By default, exceeding `max_tokens_per_call` returns WARN. Set `action_on_token_violation: "block"` to hard-stop agents that exceed token budgets. This is enforced at mid_execution — the tokens for the current call have been consumed, but subsequent calls are prevented.

7. **Pattern matching is prefix-based, not glob or regex.** `gpt-4o` matches `gpt-4o-mini` because `gpt-4o-mini` starts with `gpt-4o` followed by `-`. Use full model names in your patterns to avoid unintended matches.

8. **Auto-instrumented LLM calls trigger governance automatically.** When using supported frameworks (OpenAI, LangChain, Anthropic, etc.), the SDK's instrumentors intercept calls and trigger mid_execution checks. If you use `record_llm_call()` manually (for unsupported providers), governance evaluates at `after_workflow` when the agent completes.

## Next Steps

- [Policy & Governance](../features/governance) -- How policy enforcement works
- [Quality Policy](./quality) -- Validate output quality
- [Operations Policy](./operations) -- Timeout enforcement
- [Policy Categories & Templates](../features/policy-categories) -- All 26 categories


--------------------------------------------------------------------------------
# Quality Policy
URL: https://waxell.ai/docs/observe/governance/quality
Description: Validate agent output quality with template-based checks (contains, regex, length), LLM judge scoring, JSON schema validation, and retry feedback -- post-execution quality gates.
--------------------------------------------------------------------------------

# Quality Policy

The `quality` policy category validates **agent output quality** after execution. It inspects the result text against configurable checks: template-based (contains, regex, length, JSON schema) and LLM-based (judge scoring).

Use it when you need to enforce output standards -- required keywords, forbidden phrases, length constraints, structured output validation, or AI-judged quality scoring.

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `min_confidence_score` | float (0-1) | None | Minimum confidence score threshold. **Informational only -- not enforced** |
| `require_sources` | boolean | `false` | Whether sources are required. **Informational only -- not enforced** |
| `max_hallucination_score` | float (0-1) | None | Maximum hallucination score. **Informational only -- not enforced** |
| `validate_json_output` | boolean | `false` | Whether to validate output as JSON against `output_schema` |
| `output_schema` | object | None | JSON Schema to validate output against (requires `validate_json_output: true`) |
| `template_checks` | array | `[]` | Deterministic template-based checks on output text |
| `llm_checks` | array | `[]` | LLM judge-based quality evaluation |
| `retry_config` | object | `&#123;&#125;` | Retry configuration: `max_retries`, `feedback_template` |

## Template Check Types

| Type | Fields | Behavior |
|------|--------|----------|
| `contains` | `value`, `action`, `message` | Fails if output does NOT contain `value` (case-insensitive) |
| `not_contains` | `value`, `action`, `message` | Fails if output DOES contain `value` (case-insensitive) |
| `regex` | `pattern`, `action`, `invert`, `message` | Matches regex against output. If `invert: true`, fails when pattern IS found |
| `json_schema` | `schema`, `action`, `message` | Parses output as JSON and validates against provided schema |
| `length` | `min`, `max`, `action`, `message` | Checks `len(output)` is within `[min, max]` range |

Each check has an `action` field: `"warn"`, `"error"`, or `"retry"`.

## LLM Check Configuration

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `criteria` | string | required | What to evaluate (e.g. "Response is factually accurate") |
| `action` | string | `"warn"` | `"warn"`, `"error"`, or `"retry"` |
| `model` | string | `"gpt-4o-mini"` | Model to use for judging |
| `threshold` | float (0-1) | `0.5` | Score below this threshold = fail |

## How It Works

### Enforcement Phases

| Phase | Behavior |
|-------|----------|
| `before_workflow` | Stores rules in context. Always returns ALLOW |
| `mid_execution` | **Not implemented** |
| `after_workflow` | Runs template checks, LLM checks, JSON validation. Returns ALLOW/WARN/BLOCK/RETRY |

**No Mid-Execution Phase**
Quality has no `mid_execution` phase. The agent always runs to completion before quality checks happen. This means the agent produces output regardless of whether it will pass quality validation.


### Action Escalation

Failures are collected, then the worst action determines the result:

1. Any `"error"` or `"retry"` action + `retry_config.max_retries > 0` --> **RETRY** (with feedback)
2. Any `"error"` action without retry config --> **BLOCK**
3. Only `"warn"` actions --> **WARN**

### Retry Flow

When a check with `action: "retry"` or `action: "error"` fails and `retry_config.max_retries > 0`:

1. Quality handler returns RETRY with feedback
2. Feedback is built from `retry_config.feedback_template` with a `&#123;failures&#125;` placeholder
3. The runtime (or SDK) can use this feedback to re-prompt the agent
4. Up to `max_retries` attempts before falling back to BLOCK

## Example Policies

### Simple Keyword Requirement

Ensure reports include a recommendation:

```json
{
  "template_checks": [
    {
      "type": "contains",
      "value": "recommendation",
      "action": "error",
      "message": "Report must include a recommendation"
    }
  ]
}
```

### Length Constraints

Require output between 100 and 5000 characters:

```json
{
  "template_checks": [
    {
      "type": "length",
      "min": 100,
      "max": 5000,
      "action": "warn",
      "message": "Output should be between 100-5000 characters"
    }
  ]
}
```

### Forbidden Content

Block outputs containing uncertain language:

```json
{
  "template_checks": [
    {
      "type": "not_contains",
      "value": "I don't know",
      "action": "error",
      "message": "Output must not contain uncertain language"
    },
    {
      "type": "not_contains",
      "value": "I'm not sure",
      "action": "error",
      "message": "Output must not express uncertainty"
    }
  ]
}
```

### Combined Quality Gate

Full quality validation with retry:

```json
{
  "template_checks": [
    {
      "type": "contains",
      "value": "recommendation",
      "action": "error",
      "message": "Report must include a recommendation"
    },
    {
      "type": "not_contains",
      "value": "I don't know",
      "action": "error",
      "message": "Report must not contain uncertain language"
    },
    {
      "type": "length",
      "min": 100,
      "max": 5000,
      "action": "warn",
      "message": "Report should be between 100-5000 characters"
    }
  ],
  "retry_config": {
    "max_retries": 2,
    "feedback_template": "Previous response failed: {failures}. Please regenerate."
  }
}
```

## SDK Integration

### Using the Context Manager

```python

from waxell_observe.errors import PolicyViolationError

waxell.init()

try:
    async with waxell.WaxellContext(
        agent_name="report-generator",
        enforce_policy=True,
    ) as ctx:
        report = await generate_report(query)

        # Quality checks run on this result at after_workflow
        ctx.set_result({"report": report})

except PolicyViolationError as e:
    print(f"Quality block: {e}")
    # e.g. "Report must include a recommendation"
```

### Using the Decorator

```python
@waxell.observe(
    agent_name="report-generator",
    enforce_policy=True,
)
async def generate_report(query: str):
    report = await llm_generate(query)
    return {"report": report}
    # Quality checks run after this function returns
```

## Enforcement Flow

```
Agent starts (WaxellContext.__aenter__)
    |
    +-- before_workflow: stores quality rules in context
    |
    +-- Agent generates output (no mid_execution checks)
    |   |
    |   +-- LLM calls, tool calls, etc.
    |   +-- ctx.set_result(output)
    |
    +-- Agent completes
    |
    +-- after_workflow: quality validation
        |
        +-- JSON schema validation (if validate_json_output=true)
        +-- Template checks (contains, not_contains, regex, length)
        +-- LLM checks (if llm_judge_fn available)
        |
        +-- Collect all failures
        |
        +-- Any errors + retry config? -> RETRY
        +-- Any errors, no retry? -> BLOCK
        +-- Only warnings? -> WARN
        +-- No failures? -> ALLOW
```

## Creating via Dashboard

1. Navigate to **Governance > Policies**
2. Click **New Policy**
3. Select category **Quality**
4. Configure template_checks with desired check types
5. Optionally add LLM checks and retry configuration
6. Set scope to target specific agents (e.g., `report-generator`)
7. Enable

## Creating via API

```bash
curl -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  https://acme.waxell.dev/waxell/v1/policies/ \
  -d '{
    "name": "Report Quality Standards",
    "category": "quality",
    "rules": {
      "template_checks": [
        {"type": "contains", "value": "recommendation", "action": "error"},
        {"type": "not_contains", "value": "I don'\''t know", "action": "error"},
        {"type": "length", "min": 100, "max": 5000, "action": "warn"}
      ]
    },
    "scope": {
      "agents": ["report-generator"]
    },
    "enabled": true
  }'
```

## Observability

### Governance Tab

Quality evaluations appear with:

| Field | Example |
|-------|---------|
| **Policy name** | Report Quality Standards |
| **Action** | `allow`, `warn`, `block`, or `retry` |
| **Category** | `quality` |
| **Reason** | "Report must include a recommendation" |
| **Metadata** | `{"failures": ["Report must include a recommendation", "Output length 15 not in range [100, 5000]"]}` |

For retries:

| Field | Example |
|-------|---------|
| **Reason** | "Report must include a recommendation; Output length 15 not in range [100, 5000]" |
| **Metadata** | `{"failures": [...], "retry_feedback": "Previous response failed: ...", "max_retries": 2}` |

## Common Gotchas

1. **`min_confidence_score`, `require_sources`, `max_hallucination_score` are NOT enforced.** They are informational only -- the handler does not check these values. Use template_checks or llm_checks for actual enforcement.

2. **No `mid_execution` phase.** The agent always runs to completion before quality checks happen. If you need to stop the agent mid-execution, use a different policy category (e.g., content for text scanning).

3. **LLM checks require `_llm_judge_fn` to be injected.** This callback is only available in the controlplane flow. In demos and standalone scripts, LLM checks are skipped. Template checks always work.

4. **Template `contains` check is case-insensitive.** "Recommendation" matches "RECOMMENDATION" and "recommendation". Design your checks accordingly.

5. **`retry` action requires `retry_config.max_retries > 0` to actually retry.** If retry_config is missing or max_retries is 0, the retry action falls through to BLOCK.

6. **Quality checks operate on `str(result)`.** The handler converts the result to a string before running checks. For structured results (dicts), this means the checks run against the string representation.

## Next Steps

- [Policy & Governance](../features/governance) -- How policy enforcement works
- [LLM Policy](./llm) -- Model allowlists and token limits
- [Operations Policy](./operations) -- Timeout enforcement
- [Policy Categories & Templates](../features/policy-categories) -- All 26 categories


--------------------------------------------------------------------------------
# Content Policy
URL: https://waxell.ai/docs/observe/governance/content
Description: Input/output content scanning for PII, credentials, prompt injection, custom patterns, and blocked phrases -- with configurable warn, redact, and block actions.
--------------------------------------------------------------------------------

# Content Policy

The `content` policy category scans agent inputs and outputs for sensitive content using regex-based detection. It covers:

- **PII detection** -- SSN, email, phone, credit card
- **Credential detection** -- API keys, passwords, AWS keys, GitHub PATs, Waxell keys
- **Prompt injection guard** -- 14 patterns detecting common injection attempts
- **Custom patterns** -- user-defined regex with configurable actions
- **Blocked phrases** -- case-insensitive substring matching (always blocks)
- **Length limits** -- max input and output character counts

Unlike the [Safety Policy](./safety) (which also has content filters but always returns WARN), the content handler supports **configurable actions per detection type**: `warn`, `redact`, or `block`.

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `scan_inputs` | boolean | `true` | Scan agent inputs for content violations |
| `scan_outputs` | boolean | `true` | Scan agent outputs for content violations |
| `pii_detection` | object | `{enabled: false}` | PII scanning configuration |
| `pii_detection.enabled` | boolean | `false` | Enable PII detection |
| `pii_detection.action` | string | `"warn"` | Action on PII: `"warn"`, `"redact"`, or `"block"` |
| `pii_detection.types` | string[] | all types | PII types to scan: `ssn`, `email`, `phone`, `credit_card` |
| `credential_detection` | object | `{enabled: false}` | Credential scanning configuration |
| `credential_detection.enabled` | boolean | `false` | Enable credential detection |
| `credential_detection.action` | string | `"block"` | Action on credentials: `"warn"`, `"redact"`, or `"block"` |
| `credential_detection.patterns` | string[] | all patterns | Patterns to scan (see table below) |
| `prompt_injection_guard` | object | `{enabled: false}` | Prompt injection detection configuration |
| `prompt_injection_guard.enabled` | boolean | `false` | Enable injection guard |
| `prompt_injection_guard.action` | string | `"block"` | Action on injection: `"warn"`, `"redact"`, or `"block"` |
| `custom_patterns` | object[] | `[]` | Custom regex patterns (see below) |
| `blocked_phrases` | string[] | `[]` | Phrases to block (case-insensitive substring match) |
| `max_input_length` | integer | *(none)* | Maximum characters in input |
| `max_output_length` | integer | *(none)* | Maximum characters in output |

## How It Works

The content handler runs at **all three enforcement phases**, scanning different data at each:

| Phase | What It Scans | Context Attribute |
|-------|--------------|-------------------|
| `before_workflow` | Agent inputs | `context.inputs` |
| `mid_execution` | LLM prompt and response | `context.prompt_preview`, `context.response_preview` |
| `after_workflow` | Final output | `result` parameter |

At each phase, the handler runs all enabled checks (PII, credentials, injection, custom patterns, blocked phrases) against the text. If multiple violations are found, the **worst action wins**: `warn` < `redact` < `block`.

## PII Detection

| Type | Pattern | Example Match |
|------|---------|---------------|
| `ssn` | `\b\d{3}-\d{2}-\d{4}\b` | `123-45-6789` |
| `email` | `\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z\|a-z]{2,}\b` | `user@example.com` |
| `phone` | `\b(?:\+1[-.\s]?)?\(?\d{3}\)?[-.\s]?\d{3}[-.\s]?\d{4}\b` | `(555) 123-4567` |
| `credit_card` | `\b(?:\d{4}[-\s]?){3}\d{4}\b` | `4111-1111-1111-1111` |

When `pii_detection.action` is `"redact"`, matched text is replaced with `[REDACTED:ssn]`, `[REDACTED:email]`, etc.

## Credential Detection

| Pattern | What It Matches | Example |
|---------|----------------|---------|
| `password` | `password=`, `passwd=`, `pwd=` assignments | `password=hunter2` |
| `api_key` | `api_key=`, `apikey=`, `api_secret=` assignments | `api_key=abc123` |
| `secret` | `secret_key=`, `access_key=`, `client_secret=` assignments | `secret_key=xyz` |
| `aws_key` | `AKIA` prefix + 16 uppercase alphanumeric chars | `AKIAIOSFODNN7EXAMPLE` |
| `generic_token` | `sk-`, `pk_live_`, `sk_live_`, `rk_live_`, `sk_test_` prefix + 20+ chars | `sk-abc123456789012345678901` |
| `github_pat` | `ghp_` prefix + 36 alphanumeric chars | `ghp_abcdefghijklmnopqrstuvwxyz123456` |
| `waxell_key` | `wax_sk_` prefix + alphanumeric chars | `wax_sk_abc123` |

When `credential_detection.action` is `"redact"`, matched text is replaced with `[REDACTED:password]`, `[REDACTED:api_key]`, etc.

## Prompt Injection Guard

Detects 14 common prompt injection patterns (English-only):

| Pattern | Example |
|---------|---------|
| "ignore previous instructions" | "Ignore all previous instructions and..." |
| "ignore above instructions" | "Ignore the above instructions" |
| "you are now a..." | "You are now a helpful hacker" |
| "forget your instructions" | "Forget all your previous instructions" |
| "disregard previous" | "Disregard all previous context" |
| "new instructions:" | "New instructions: do something else" |
| "override system" | "Override your system instructions" |
| "```system" | Code block system injection |
| "[system]:" | Bracket system injection |
| "&lt;\|system\|&gt;" | Tag system injection |
| "ADMIN MODE ENABLED" | Fake admin mode activation |
| "developer mode enabled" | Fake developer mode |
| "jailbreak" | Direct jailbreak attempt |
| "DAN mode" | "Do Anything Now" mode attempt |

All patterns are case-insensitive.

## Custom Patterns

Define your own regex patterns with configurable actions:

```json
{
  "custom_patterns": [
    {
      "name": "Internal IPs",
      "pattern": "10\\.\\d+\\.\\d+\\.\\d+",
      "action": "warn"
    },
    {
      "name": "Internal URLs",
      "pattern": "https?://internal\\.",
      "action": "block"
    }
  ]
}
```

Each pattern object requires:
- `name` -- human-readable label (appears in violation messages)
- `pattern` -- regex string (compiled with `re.IGNORECASE`)
- `action` -- `"warn"`, `"redact"`, or `"block"` (default: `"warn"`)

Invalid regex patterns are silently skipped. Matched text is truncated to 100 characters in violation reports.

## Blocked Phrases

Case-insensitive substring matching. The action is always `"block"` (hardcoded).

```json
{
  "blocked_phrases": [
    "ignore previous instructions",
    "reveal system prompt",
    "jailbreak"
  ]
}
```

**Blocked Phrases Always Block**
Unlike PII and credential detection where you can choose warn/redact/block, blocked phrases **always produce a block action**. There is no way to configure them as warn-only.


## Length Limits

- `max_input_length` -- checked at before_workflow, **before** content scanning. Returns BLOCK if exceeded.
- `max_output_length` -- checked at mid_execution (response_preview) and after_workflow (final result). Returns WARN if exceeded.

**Input Length Is Checked First**
`max_input_length` is evaluated before any content scanning. If the input exceeds the limit, the handler returns BLOCK immediately without scanning for PII, credentials, or other violations.


## Redaction

When any detection type has `action: "redact"`, the `ContentHandler.redact_content(text, rules)` method can be called to replace sensitive content with markers:

- PII: `[REDACTED:ssn]`, `[REDACTED:email]`, `[REDACTED:phone]`, `[REDACTED:credit_card]`
- Credentials: `[REDACTED:password]`, `[REDACTED:api_key]`, `[REDACTED:secret]`, etc.

Only categories whose action is explicitly `"redact"` are redacted. A policy with `pii_detection.action: "block"` and `credential_detection.action: "redact"` will only redact credentials, not PII.

## Example Policies

### PII-Only Scanning (Warn)

Detect PII but don't block:

```json
{
  "scan_inputs": true,
  "scan_outputs": true,
  "pii_detection": {
    "enabled": true,
    "action": "warn",
    "types": ["ssn", "credit_card"]
  }
}
```

### Full Security (All Checks, Block)

Enable all detection types with block action:

```json
{
  "scan_inputs": true,
  "scan_outputs": true,
  "pii_detection": {
    "enabled": true,
    "action": "block",
    "types": ["ssn", "email", "phone", "credit_card"]
  },
  "credential_detection": {
    "enabled": true,
    "action": "block"
  },
  "prompt_injection_guard": {
    "enabled": true,
    "action": "block"
  },
  "blocked_phrases": ["jailbreak", "reveal system prompt"],
  "max_input_length": 50000,
  "max_output_length": 10000
}
```

### Custom Pattern (Internal IPs)

Warn on internal IP addresses in outputs:

```json
{
  "scan_outputs": true,
  "custom_patterns": [
    {
      "name": "Internal IPs",
      "pattern": "10\\.\\d+\\.\\d+\\.\\d+",
      "action": "warn"
    },
    {
      "name": "Private IPs",
      "pattern": "192\\.168\\.\\d+\\.\\d+",
      "action": "warn"
    }
  ]
}
```

### Redact Mode

Redact PII and credentials instead of blocking:

```json
{
  "pii_detection": {
    "enabled": true,
    "action": "redact",
    "types": ["ssn", "credit_card"]
  },
  "credential_detection": {
    "enabled": true,
    "action": "redact"
  }
}
```

## SDK Integration

### Using the Context Manager

```python

from waxell_observe.errors import PolicyViolationError

waxell.init()

try:
    async with waxell.WaxellContext(
        agent_name="support-agent",
        enforce_policy=True,
        inputs={"query": user_query},
    ) as ctx:
        # before_workflow: content handler scans inputs
        # If PII/credential/injection detected -> BLOCK/WARN/REDACT

        response = await process_query(user_query)

        ctx.record_llm_call(
            model="gpt-4o-mini",
            prompt_preview=user_query[:200],
            response_preview=response[:200],
        )
        # mid_execution: scans prompt_preview and response_preview

        ctx.set_result(response)
        # after_workflow: scans final result

except PolicyViolationError as e:
    print(f"Content block: {e}")
    # e.g. "Input content violations: [input] PII detected: ssn"
```

### Using the Decorator

```python
@waxell.observe(
    agent_name="support-agent",
    enforce_policy=True,
)
async def handle_support(query: str):
    # Content scans happen at all three phases
    return await process_query(query)
```

## Enforcement Flow

```
Agent starts (WaxellContext.__aenter__)
    |
    +-- before_workflow
    |   |
    |   +-- scan_inputs disabled? -> ALLOW (skip)
    |   +-- No input text? -> ALLOW
    |   +-- max_input_length exceeded? -> BLOCK
    |   +-- Scan input text:
    |       +-- PII detection -> action per config
    |       +-- Credential detection -> action per config
    |       +-- Prompt injection guard -> action per config
    |       +-- Custom patterns -> action per pattern
    |       +-- Blocked phrases -> always BLOCK
    |   +-- Worst action wins (warn < redact < block)
    |
    +-- Agent executes...
    |
    +-- mid_execution (per LLM call)
    |   |
    |   +-- scan_inputs? -> scan prompt_preview
    |   +-- scan_outputs? -> scan response_preview
    |   +-- scan_outputs? -> check max_output_length on response
    |   +-- Worst action wins
    |
    +-- Agent finishes
    |
    +-- after_workflow
        |
        +-- scan_outputs disabled? -> ALLOW (skip)
        +-- No result? -> ALLOW
        +-- max_output_length exceeded? -> WARN
        +-- Scan result text (same checks as before_workflow)
        +-- Worst action wins
```

## Creating via Dashboard

1. Navigate to **Governance > Policies**
2. Click **New Policy**
3. Select category **Content**
4. Enable detection types (PII, credentials, injection guard)
5. Set action per type (warn, redact, block)
6. Optionally add custom patterns and blocked phrases
7. Set input/output length limits
8. Set scope to target specific agents
9. Enable

## Creating via API

```bash
curl -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  https://acme.waxell.dev/waxell/v1/policies/ \
  -d '{
    "name": "Content Security",
    "category": "content",
    "rules": {
      "scan_inputs": true,
      "scan_outputs": true,
      "pii_detection": {
        "enabled": true,
        "action": "block",
        "types": ["ssn", "email", "phone", "credit_card"]
      },
      "credential_detection": {
        "enabled": true,
        "action": "block"
      },
      "prompt_injection_guard": {
        "enabled": true,
        "action": "block"
      },
      "blocked_phrases": ["jailbreak", "reveal system prompt"],
      "max_input_length": 50000,
      "max_output_length": 10000
    },
    "scope": {
      "agents": ["support-agent"]
    },
    "enabled": true
  }'
```

## Observability

### Governance Tab

Content evaluations appear with:

| Field | Example (ALLOW) |
|-------|---------|
| **Policy name** | Content Security |
| **Action** | `allow` |
| **Category** | `content` |
| **Reason** | "Input content scan passed (PII, credentials, injection guard, 2 blocked phrases)" |

For violations:

| Field | Example (BLOCK) |
|-------|---------|
| **Action** | `block` |
| **Reason** | "Input content violations: [input] PII detected: ssn; [input] Credential detected: api_key" |
| **Metadata** | `{"violations": [{"type": "pii", "message": "[input] PII detected: ssn", "action": "block"}], "scan_target": "input"}` |

For prompt injection:

| Field | Example |
|-------|---------|
| **Reason** | "Input content violations: [input] Prompt injection pattern: 'Ignore all previous instructions'" |
| **Metadata** | `{"violations": [{"type": "prompt_injection", "message": "...", "action": "block"}]}` |

## Combining with Other Policies

- **Content + Safety**: Safety has simpler content filters (pii/profanity/credentials with WARN-only). Content provides more granular control with configurable actions per detection type
- **Content + Compliance**: HIPAA compliance can require `content.pii_detection.enabled: true` as a required rule
- **Content + Privacy**: Privacy handles data access controls; content handles data leakage detection in text

## Common Gotchas

1. **`blocked_phrases` action is always `"block"`.** You cannot configure them as warn-only. The action is hardcoded in the handler.

2. **PII detection is regex-based, not ML-based.** It can miss edge cases (e.g., SSNs without dashes) and may false-positive on patterns that look like PII (e.g., formatted dates).

3. **Prompt injection patterns are English-only.** Non-English injection attempts will not be detected by the built-in patterns. Use `custom_patterns` for other languages.

4. **`scan_inputs: false` disables before_workflow entirely.** The handler returns ALLOW immediately without checking anything, including `max_input_length`.

5. **`scan_outputs: false` disables after_workflow entirely.** No output scanning or length checking occurs.

6. **Custom pattern regex is case-insensitive.** All custom patterns are compiled with `re.IGNORECASE`. You cannot make them case-sensitive.

7. **`max_input_length` is checked BEFORE content scanning.** If input exceeds the limit, the handler returns BLOCK without running any content detection. This is intentional -- there's no point scanning very large inputs.

8. **Action priority: warn < redact < block.** If PII detection is set to "warn" but credential detection is set to "block", and both trigger, the overall action is "block".

9. **Redaction only applies to categories with action `"redact"`.** A policy with `pii_detection.action: "block"` will not redact PII -- it will block. Set action to `"redact"` explicitly.

10. **The `_worst_action` escalation applies per-phase.** Each phase independently determines its action. A WARN at before_workflow does not prevent a BLOCK at mid_execution.

## Next Steps

- [Safety Policy](./safety) -- Broader safety controls including step/tool limits
- [Policy & Governance](../features/governance) -- How policy enforcement works
- [Compliance Policy](./compliance) -- Meta-validator for regulatory frameworks
- [Policy Categories & Templates](../features/policy-categories) -- All 26 categories


--------------------------------------------------------------------------------
# Spawn Limit Policy
URL: https://waxell.ai/docs/observe/governance/spawn-limit
Description: Tenant-wide concurrent ctx.spawn ceiling enforced at the dispatcher -- gates new child spawns before any dispatch happens. Runtime-plane only.
--------------------------------------------------------------------------------

# Spawn Limit Policy

The `spawn-limit` policy puts a tenant-level ceiling on concurrent `ctx.spawn` children. This is the **operator's rail**: the agent-level cap (declared by the author in `guards: [{type: spawn_concurrent, limit: N}]`) enforces per-parent-tree in the `BudgetLedger`. This handler is policy-driven, scoped via the standard policy scope filter (tenant / agent / workflow), and halts the spawn before any children dispatch.

Runs only on the **runtime plane** -- there is no observe-plane `before_spawn` call site.

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `concurrent_spawn_limit` | integer | `50` | Maximum concurrent spawned children for this scope |
| `action` | string | `"block"` | Either `block` or `warn` when the cap would be breached |

## How It Works

The `spawn-limit` handler fires from the **`before_spawn`** hook on `PolicyGovernanceHook`. The dispatcher (`ProductionSpawnDispatcher`) is responsible for incrementing the counter on `enqueue_children` and decrementing on `on_child_complete`. This handler only **reads** the counter and rejects on breach.

| Phase | What It Checks | Actions |
|-------|---------------|---------|
| `before_workflow` | No-op (ALLOW) | ALLOW |
| `mid_execution` | If `context._pending_spawn` is set, delegates to `before_spawn`; otherwise ALLOW | BLOCK / WARN on cap breach |
| `before_spawn` | Reads Redis counter for `(agent, workflow)`, checks `current + total_children > limit` | BLOCK or WARN |
| `after_workflow` | No-op (ALLOW) | ALLOW |

### Context Attributes Read

| Attribute | Phase | Purpose |
|-----------|-------|---------|
| `context.agent_name` | before_spawn | Counter key narrowing |
| `context.workflow_name` | before_spawn | Counter key narrowing |
| `context._pending_spawn` | mid_execution | Set by `PolicyGovernanceHook.before_spawn` with `{child_agent, total_children}` |
| `context.run_id` | before_spawn | Recorded on durable EnforcementEvent |

### Counter Key

```
spawn_concurrent:{agent_name}:{workflow_name}
```

Auto-prefixed with tenant by `TenantAwareRedis`. The `SpawnLimitHandler.build_counter_key()` staticmethod is the canonical builder -- the dispatcher uses the same function to ensure key parity.

## Example Policy

```json
{
  "name": "Research Fleet Concurrency Cap",
  "category": "spawn-limit",
  "rules": {
    "concurrent_spawn_limit": 25,
    "action": "block"
  },
  "scope": {
    "agents": ["research-agent"],
    "workflows": ["batch-research"]
  },
  "enabled": true
}
```

## SDK Integration

```python

waxell.init()

@waxell.observe(agent_name="research-agent", enforce_policy=True)
async def batch_research(topics: list[str]) -> list[str]:
    # ctx.spawn calls trigger before_spawn — the policy reads the
    # Redis counter and halts if this batch + in-flight would exceed
    # concurrent_spawn_limit for (research-agent, batch-research).
    return await ctx.spawn_many("worker-agent", topics)
```

## Observability

| Field | Example |
|-------|---------|
| **Category** | `spawn-limit` (recorded as `concurrency` in `EnforcementEvent`) |
| **Action** | `block` |
| **Reason** | "concurrent spawn limit breached: 23 in flight + 5 requested would reach 28 (cap 25) for scope agent=research-agent workflow=batch-research" |
| **Metadata** | `{"cap": 25, "current_in_flight": 23, "requested": 5, "would_be": 28, "counter_key": "spawn_concurrent:research-agent:batch-research", "workflow_name": "batch-research"}` |

Every block / warn writes a durable `EnforcementEvent` (`handler="spawn-limit"`, `category="concurrency"`) so blocks-in-last-24h-by-agent is a cheap query in the admin surface.

## Common Gotchas

1. **`supported_planes = ["runtime"]`.** This policy never fires on the observe plane. Agents instrumented purely via `waxell-observe` (no governed runtime) cannot enforce it. The `before_spawn` call site only exists in the runtime dispatcher.
2. **The handler does NOT increment / decrement the counter.** That is the dispatcher's job. If you swap dispatchers, ensure the new one also calls `inc` on enqueue and `dec` on completion using the same key builder.
3. **Redis read failure fails open.** If the Redis read raises, the handler logs a warning and returns ALLOW. This is intentional (a Redis outage should not block spawns) but means a degraded Redis cluster will silently disable the cap.
4. **`concurrent_spawn_limit <= 0` disables the check.** Setting it to 0 or negative is a no-op ALLOW, not "block all spawns". To block all spawns, set the limit to 1 and rely on the cap being breached by the first request.
5. **Scope narrowing happens upstream.** By the time the handler runs, `DynamicPolicyManager` has already filtered to in-scope policies via `applies_to`. The counter still reads the per-(agent, workflow) key, so a tenant-wide policy with no scope filter still counts per-agent-per-workflow buckets, not a single tenant-total counter.
6. **`mid_execution` is the entry point in Phase 1.0.** The hook stamps `context._pending_spawn` then calls `mid_execution_detailed`. Non-spawn mid_execution invocations (tool calls, LLM calls) return ALLOW immediately.

## Next Steps

- [Rate-Limit Policy](./rate-limit) -- Requests/second, complementary to concurrent-spawn caps
- [Budget Policy](./budget) -- Cap fan-out cost, not just concurrency
- [Operations Policy](./operations) -- Other runtime guardrails
- [Policy Categories](../features/policy-categories)


--------------------------------------------------------------------------------
# Data Access Policy
URL: https://waxell.ai/docs/observe/governance/data-access
Description: Control which data sources agents can read from or write to — allowlists, blocklists, read-only enforcement, and per-query record volume limits.
--------------------------------------------------------------------------------

# Data Access Policy

The `data-access` policy category controls which data sources an agent may access. Use it to prevent agents from touching sensitive databases, enforce read-only access to production data, or cap the number of records an agent can pull per query.

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `allowed_data_sources` | string[] | `[]` | If non-empty, agents may only access sources in this list. Acts as an allowlist. |
| `blocked_data_sources` | string[] | `[]` | Sources that agents are never allowed to access, regardless of the allowlist. |
| `read_only_sources` | string[] | `[]` | Sources that agents may read from but not write to. |
| `max_records_per_query` | integer | `1000` | Maximum number of records an agent may retrieve in a single query. Violations produce a WARN (never a block). |
| `action_on_violation` | string | `"block"` | `"block"` raises a PolicyViolationError; `"warn"` logs the violation and lets the agent continue. Applies to source-level violations only — record limit violations always WARN. |

## How It Works

The data-access handler runs at three phases:

### before_workflow

Runs before the agent does any work. Checks `context.data_sources_configured` — if the agent is pre-configured to use a blocked source, it is stopped before any LLM call or tool use occurs.

### mid_execution

Triggered every time the agent calls `ctx.record_data_access(...)`. This is where source-level and write violations are caught:

1. For each source in `context.data_sources_accessed`:
   - If the source appears in `blocked_data_sources` → violation
   - If `allowed_data_sources` is non-empty and the source is not in it → violation
2. For each source in `context.data_sources_written`:
   - If the source appears in `read_only_sources` → violation
3. If `context.records_queried` exceeds `max_records_per_query` → WARN (agent continues regardless of `action_on_violation`)

### after_workflow

Runs after the agent completes. Produces a final audit summary listing sources accessed and written. Warnings are emitted if blocked or read-only sources were accessed during execution (belt-and-suspenders check after mid_execution).

### Rule Evaluation Order

| Check | When triggered | Configurable action |
|-------|---------------|---------------------|
| Blocked source | mid_execution, per `record_data_access` call | `action_on_violation` |
| Not in allowlist | mid_execution, per `record_data_access` call | `action_on_violation` |
| Write to read-only source | mid_execution, per `record_data_access` call | `action_on_violation` |
| Record limit | mid_execution, per `record_data_access` call | Always WARN |

Blocked sources are checked **before** allowlist membership. A source that appears in both `allowed_data_sources` and `blocked_data_sources` is always blocked.

## Example Policies

### Customer Data Policy (strict)

Allow reads from approved sources only; block HR and payroll entirely; make the production database read-only:

```json
{
  "allowed_data_sources": ["postgres", "redis", "product_catalog"],
  "blocked_data_sources": ["hr_records", "payroll"],
  "read_only_sources": ["postgres"],
  "max_records_per_query": 1000,
  "action_on_violation": "block"
}
```

### Analytics Agent (high volume, warn on violations)

Allow large record pulls but log violations rather than blocking:

```json
{
  "allowed_data_sources": ["analytics_db", "data_warehouse"],
  "blocked_data_sources": ["pii_store"],
  "read_only_sources": [],
  "max_records_per_query": 50000,
  "action_on_violation": "warn"
}
```

### Internal-Only Agent (blocklist only)

Block specific sensitive sources without restricting everything else:

```json
{
  "allowed_data_sources": [],
  "blocked_data_sources": ["hr_records", "payroll", "executive_compensation"],
  "read_only_sources": [],
  "max_records_per_query": 5000,
  "action_on_violation": "block"
}
```

## SDK Integration

### Recording Data Access Events

Call `ctx.record_data_access()` after each data operation. The handler evaluates the access immediately at mid_execution:

```python

from waxell_observe.errors import PolicyViolationError

waxell.init()

try:
    async with waxell.WaxellContext(
        agent_name="data-agent",
        enforce_policy=True,
    ) as ctx:

        # Read from a data source — triggers mid_execution governance
        rows = db.query("SELECT * FROM customers LIMIT 500")
        ctx.record_data_access(
            source="postgres",
            operation="read",
            records=len(rows),
        )

        # Write to a data source
        db.execute("UPDATE customers SET status = 'active' WHERE id = ?", customer_id)
        ctx.record_data_access(
            source="postgres",
            operation="write",
            records=1,
        )

        ctx.set_result({"rows": rows})

except PolicyViolationError as e:
    print(f"Data access blocked: {e}")
    # e.g. "Write to read-only data source 'postgres'"
    # e.g. "Access to blocked data source 'hr_records'"
    # e.g. "Data source 'staging_db' is not in allowed list"
```

### Method Signature

```python
ctx.record_data_access(
    source: str,       # Data source name — must match your policy config exactly
    operation: str,    # "read" or "write"
    records: int = 0,  # Number of records accessed/modified
) -> None
```

The `source` name is compared exactly (case-sensitive) against `allowed_data_sources`, `blocked_data_sources`, and `read_only_sources`. Use consistent naming conventions across your codebase.

### Using the Decorator

```python
@waxell.observe(
    agent_name="data-agent",
    enforce_policy=True,
)
async def run_query(ctx, query: str):
    rows = db.query(query)
    ctx.record_data_access(source="postgres", operation="read", records=len(rows))
    return rows
```

## Enforcement Flow

```
Agent starts (WaxellContext.__aenter__)
    │
    └── before_workflow governance
        └── Check data_sources_configured vs blocked_data_sources
            └── Pre-configured blocked source? → BLOCK (always, regardless of action_on_violation)

Agent calls ctx.record_data_access(source="hr_records", operation="read", records=200)
    │
    └── mid_execution governance
        ├── source in blocked_data_sources? → action_on_violation (BLOCK or WARN)
        ├── allowed_data_sources non-empty AND source not in it? → action_on_violation
        ├── source in read_only_sources AND operation == "write"? → action_on_violation
        └── records_queried > max_records_per_query? → always WARN

Agent completes
    │
    └── after_workflow governance
        └── Audit summary — warns if blocked or read-only sources were accessed
```

## Creating via Dashboard

1. Navigate to **Governance > Policies**
2. Click **New Policy**
3. Select category **Data Access**
4. Configure source lists, record limit, and violation action
5. Set scope to target specific agents (e.g., `data-access-agent`)
6. Enable

## Creating via API

```bash
curl -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  https://acme.waxell.dev/waxell/v1/policies/ \
  -d '{
    "name": "Customer Data Policy",
    "category": "data-access",
    "rules": {
      "allowed_data_sources": ["postgres", "redis", "product_catalog"],
      "blocked_data_sources": ["hr_records", "payroll"],
      "read_only_sources": ["postgres"],
      "max_records_per_query": 1000,
      "action_on_violation": "block"
    },
    "scope": {
      "agents": ["data-access-agent"]
    },
    "enabled": true
  }'
```

## Observability

### Governance Tab

Data access evaluations appear with:

| Field | Example |
|-------|---------|
| **Policy name** | Customer Data Policy |
| **Action** | `allow`, `warn`, or `block` |
| **Category** | `data-access` |
| **Reason** | "Access to blocked data source 'hr_records'" |
| **Metadata** | `&#123;"blocked_source": "hr_records"&#125;` |

For allow cases:

| Field | Example |
|-------|---------|
| **Reason** | "Data access within policy (2 source(s) accessed)" |
| **Metadata** | `&#123;"sources_accessed": ["postgres", "redis"], "sources_written": []&#125;` |

### Record Limit Warnings

When `records_queried` exceeds `max_records_per_query`, the governance tab shows:

| Field | Example |
|-------|---------|
| **Action** | `warn` |
| **Reason** | "Records queried (15000) exceeds limit (1000)" |
| **Metadata** | `&#123;"records_queried": 15000, "limit": 1000&#125;` |

Record limit warnings never stop the agent — the `action_on_violation` setting does not apply to them.

## Common Gotchas

1. **`allowed_data_sources` is an allowlist when non-empty.** An empty list means "no restriction." As soon as you add one entry, all other sources are blocked (unless `action_on_violation` is `"warn"`).

2. **`blocked_data_sources` is checked before `allowed_data_sources`.** A source in both lists is always blocked. This makes blocklists safe to use alongside allowlists without interaction surprises.

3. **`max_records_per_query` always WARNS, never blocks.** The handler hardcodes WARN for record limit violations. Setting `action_on_violation: "block"` does not change this behavior. Use source-level controls (allowlists and blocklists) for hard enforcement.

4. **Source names are case-sensitive and exact-matched.** `"Postgres"` and `"postgres"` are different sources. Use consistent lowercase naming in your `ctx.record_data_access()` calls and policy configuration.

5. **`write` in `operation` populates `data_sources_written`, not `data_sources_accessed`.** Read-only enforcement only fires when the `operation` is `"write"`. If you accidentally pass `operation="read"` for a write operation, the read-only check is bypassed.

6. **Each `record_data_access()` call triggers mid_execution immediately.** The handler evaluates the entire accumulated access buffer on every call. If a second access is the violating one, the first access is still recorded in the trace.

7. **`before_workflow` only checks `data_sources_configured`.** This field is rarely populated in practice — it requires the agent framework to pre-declare which sources it uses. Most enforcement happens at mid_execution.

## Combining with Other Policies

The data-access policy works well alongside:

- **Audit policy** — logs every data access with timestamp and user for compliance records
- **Compliance policy** — HIPAA and PCI-DSS compliance profiles often require a `data-access` policy in `required_categories`
- **Scope policy** — combine with data-access to limit both which sources and how many records can be modified in a single run

## Next Steps

- [Policy & Governance](../features/governance) — How policy enforcement works
- [Compliance Policy](./compliance) — Enforce regulatory frameworks that require data-access controls
- [Network Policy](./network) — Govern outbound HTTP requests alongside data source access
- [Scope Policy](../features/policy-categories) — Limit blast radius for write operations
- [Policy Categories & Templates](../features/policy-categories) — All 26 categories


--------------------------------------------------------------------------------
# Network Policy
URL: https://waxell.ai/docs/observe/governance/network
Description: Control outbound HTTP requests from agents — domain allowlists and blocklists, wildcard patterns, protocol restrictions, and internal-only mode.
--------------------------------------------------------------------------------

# Network Policy

The `network` policy category controls outbound HTTP requests made by agents. Use it to enforce that agents only contact approved domains, block exfiltration to untrusted services, restrict connections to HTTPS, and lock agents to internal-only endpoints.

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `allowed_domains` | string[] | `[]` | If non-empty, agents may only contact domains in this list. Supports wildcard prefix patterns (`*.example.com`). |
| `blocked_domains` | string[] | `[]` | Domains that agents are never allowed to contact. Supports wildcards. Checked before the allowed list. |
| `allowed_protocols` | string[] | `["https"]` | Protocols agents may use. Set to `["http", "https"]` to allow both; leave as `["https"]` for TLS enforcement. |
| `block_external` | boolean | `false` | When `true` and `allowed_domains` is non-empty, any domain not in `allowed_domains` is treated as external and blocked. Enables strict internal-only mode. |
| `log_all_requests` | boolean | `true` | Emit an info-level log line at `after_workflow` listing the total request count and unique domains. |
| `action_on_violation` | string | `"block"` | `"block"` raises a PolicyViolationError; `"warn"` logs the violation and lets the agent continue. |

## How It Works

The network handler runs at three phases:

### before_workflow

Runs before the agent does any work. Checks `context.configured_endpoints` — if the agent is pre-configured with an endpoint whose domain is in `blocked_domains`, it is stopped before any LLM call or tool use occurs.

### mid_execution

Triggered every time the agent calls `ctx.record_network_request(url=...)`. Evaluation order for each URL:

1. **Protocol check**: extract the scheme from the URL. If `allowed_protocols` is non-empty and the scheme is not in it → violation.
2. **Blocklist check**: extract the domain. If it matches any entry in `blocked_domains` → violation.
3. **External check**: if `block_external` is `true` AND `allowed_domains` is non-empty AND the domain is not in `allowed_domains` → violation.
4. **Allowlist check**: if `allowed_domains` is non-empty AND the domain is not in it → violation (catches cases where `block_external` is `false`).

If any check produces a violation, `action_on_violation` determines whether the agent is blocked or just warned.

### after_workflow

Produces a final network audit. If `log_all_requests` is `true`, logs the total request count and unique domain set. Emits warnings for any requests that were made to blocked domains (belt-and-suspenders after mid_execution).

### Domain Matching

The handler uses wildcard-aware matching:

| Pattern | Domain | Match? | Why |
|---------|--------|--------|-----|
| `api.example.com` | `api.example.com` | Yes | Exact match |
| `api.example.com` | `other.example.com` | No | Different subdomain |
| `*.example.com` | `api.example.com` | Yes | Wildcard matches subdomain |
| `*.example.com` | `deep.api.example.com` | Yes | Wildcard matches any suffix |
| `*.example.com` | `example.com` | Yes | Wildcard also matches the root |
| `*.example.com` | `notexample.com` | No | Different root domain |
| `pastebin.com` | `api.pastebin.com` | No | Exact match only, no auto-wildcard |

To block all subdomains of a domain, use the `*.` prefix: `*.pastebin.com` blocks `sub.pastebin.com` but `pastebin.com` requires a separate entry (or use `*.pastebin.com` which also matches the root).

## Example Policies

### Internal-Only Agent (strict)

Allow only company internal endpoints; block pastebin and competitor domains; require HTTPS:

```json
{
  "allowed_domains": ["*.internal.company.com", "api.internal.company.com"],
  "blocked_domains": ["pastebin.com", "*.pastebin.com", "*.competitor.com"],
  "allowed_protocols": ["https"],
  "block_external": true,
  "log_all_requests": true,
  "action_on_violation": "block"
}
```

### API Integration Agent (controlled external access)

Allow specific external APIs; block data exfiltration targets; no internal-only restriction:

```json
{
  "allowed_domains": ["api.openai.com", "api.anthropic.com", "api.stripe.com"],
  "blocked_domains": ["pastebin.com", "*.pastebin.com", "webhook.site"],
  "allowed_protocols": ["https"],
  "block_external": false,
  "log_all_requests": true,
  "action_on_violation": "block"
}
```

### Development Agent (permissive with logging)

Allow most traffic but log everything and warn on suspicious domains:

```json
{
  "allowed_domains": [],
  "blocked_domains": ["*.onion", "*.darkweb.com"],
  "allowed_protocols": ["http", "https"],
  "block_external": false,
  "log_all_requests": true,
  "action_on_violation": "warn"
}
```

## SDK Integration

### Recording Network Requests

Call `ctx.record_network_request()` for every outbound HTTP request. The handler evaluates the URL immediately at mid_execution:

```python

from waxell_observe.errors import PolicyViolationError

waxell.init()

try:
    async with waxell.WaxellContext(
        agent_name="network-agent",
        enforce_policy=True,
    ) as ctx:

        # Record before (or after) making the actual HTTP call
        ctx.record_network_request(url="https://api.internal.company.com/v1/reports")

        # Make the actual request (not intercepted automatically)
        response = httpx.get("https://api.internal.company.com/v1/reports")

        ctx.set_result({"data": response.json()})

except PolicyViolationError as e:
    print(f"Network request blocked: {e}")
    # e.g. "Request to blocked domain 'pastebin.com'"
    # e.g. "External request to 'api.openai.com' blocked (internal-only mode)"
    # e.g. "Protocol 'http' not in allowed list for 'http://legacy.internal.com/api'"
    # e.g. "Domain 'staging.example.com' is not in allowed list"
```

### Method Signature

```python
ctx.record_network_request(
    url: str,  # Full URL including scheme — e.g. "https://api.example.com/endpoint"
) -> None
```

The handler extracts both the domain and protocol from the `url` string. Always pass the full URL (with `https://` or `http://`) so protocol enforcement works correctly. Passing a bare domain like `"api.example.com"` skips protocol checking.

### Using the Decorator

```python
@waxell.observe(
    agent_name="network-agent",
    enforce_policy=True,
)
async def fetch_data(ctx, url: str):
    ctx.record_network_request(url=url)
    response = await httpx.AsyncClient().get(url)
    return response.json()
```

## Enforcement Flow

```
Agent starts (WaxellContext.__aenter__)
    │
    └── before_workflow governance
        └── Check configured_endpoints vs blocked_domains
            └── Pre-configured blocked domain? → BLOCK (always)

Agent calls ctx.record_network_request(url="https://pastebin.com/abc")
    │
    └── mid_execution governance (per URL)
        ├── Extract protocol: "https"
        ├── allowed_protocols non-empty AND "https" not in it? → action_on_violation
        ├── Extract domain: "pastebin.com"
        ├── domain matches blocked_domains? → action_on_violation
        ├── block_external AND allowed_domains non-empty AND not in allowed? → action_on_violation
        └── allowed_domains non-empty AND not in allowed? → action_on_violation

Agent completes
    │
    └── after_workflow governance
        ├── log_all_requests? → info log with request count and domains
        └── Any blocked domains accessed? → WARN with domain list
```

## Creating via Dashboard

1. Navigate to **Governance > Policies**
2. Click **New Policy**
3. Select category **Network**
4. Configure domain lists, protocol restrictions, and `block_external`
5. Set scope to target specific agents (e.g., `network-agent`)
6. Enable

## Creating via API

```bash
curl -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  https://acme.waxell.dev/waxell/v1/policies/ \
  -d '{
    "name": "Internal Network Policy",
    "category": "network",
    "rules": {
      "allowed_domains": ["*.internal.company.com"],
      "blocked_domains": ["pastebin.com", "*.competitor.com"],
      "allowed_protocols": ["https"],
      "block_external": true,
      "log_all_requests": true,
      "action_on_violation": "block"
    },
    "scope": {
      "agents": ["network-agent"]
    },
    "enabled": true
  }'
```

## Observability

### Governance Tab

Network evaluations appear with:

| Field | Example |
|-------|---------|
| **Policy name** | Internal Network Policy |
| **Action** | `allow`, `warn`, or `block` |
| **Category** | `network` |
| **Reason** | "Request to blocked domain 'pastebin.com'" |
| **Metadata** | `&#123;"url": "https://pastebin.com/abc", "blocked_domain": "pastebin.com"&#125;` |

For protocol violations:

| Field | Example |
|-------|---------|
| **Reason** | "Protocol 'http' not in allowed list for 'http://legacy.internal.com/api'" |
| **Metadata** | `&#123;"url": "http://...", "protocol": "http", "allowed_protocols": ["https"]&#125;` |

For external-block violations:

| Field | Example |
|-------|---------|
| **Reason** | "External request to 'api.openai.com' blocked (internal-only mode)" |
| **Metadata** | `&#123;"url": "https://api.openai.com/...", "domain": "api.openai.com"&#125;` |

For allow cases:

| Field | Example |
|-------|---------|
| **Reason** | "Network access within policy (3 request(s))" |
| **Metadata** | `&#123;"domains_accessed": ["api.internal.company.com"], "request_count": 3&#125;` |

### After-Workflow Audit

The `after_workflow` phase always records an audit result. With `log_all_requests: true`, the server-side log line looks like:

```
Network audit: 3 requests to 2 domains
```

## Common Gotchas

1. **`allowed_domains` is an allowlist when non-empty.** An empty list means "no domain restriction." As soon as you add one entry, all domains not in the list are blocked when `block_external` is `true`, or warned if `action_on_violation` is `"warn"`.

2. **`blocked_domains` does not auto-wildcard.** Adding `"pastebin.com"` blocks only `pastebin.com`. To also block `sub.pastebin.com`, add `"*.pastebin.com"` as a separate entry.

3. **Protocol is checked before the domain.** A request to `http://api.internal.company.com/` is blocked for the wrong protocol, not for being a blocked domain. The error message will reference the protocol, which can be surprising.

4. **`block_external` only activates when `allowed_domains` is non-empty.** With an empty allowlist and `block_external: true`, no external blocking occurs — there is nothing to be "external" to. Always pair `block_external: true` with at least one `allowed_domains` entry.

5. **The `url` parameter is used for both domain and protocol extraction.** Pass the full URL including the scheme. A bare domain or path skips protocol checking. Bare domains are matched for domain checks only.

6. **Wildcard `*.example.com` also matches `example.com` itself.** The handler strips the `*.` prefix and checks that the domain ends with `.example.com` OR equals `example.com`. This is intentional — a wildcard policy should cover the root domain too.

7. **`record_network_request` does not make the actual HTTP call.** It only records the intent for governance evaluation. You are responsible for making the actual request with your HTTP client. This means governance fires even if the request ultimately fails.

8. **Multiple calls per run are each evaluated independently.** If the first URL passes and the second URL is blocked, the first request is already recorded in the trace. The block fires at the second call.

## Combining with Other Policies

The network policy works well alongside:

- **Data access policy** — use both to control what data an agent can read (via database) and where it can send data (via HTTP)
- **Compliance policy** — PCI-DSS compliance profiles often list `network` in `required_categories` to ensure network restrictions are active
- **Audit policy** — combine with `log_all_requests: true` to create a complete paper trail of every outbound connection
- **Scope policy** — limit total side effects while network policy specifically controls which endpoints are reachable

## Next Steps

- [Policy & Governance](../features/governance) — How policy enforcement works
- [Data Access Policy](./data-access) — Govern database and storage access alongside network requests
- [Compliance Policy](./compliance) — Regulatory frameworks that require network controls
- [Communication Policy](./communication) — Govern agent-to-human and agent-to-channel messaging
- [Policy Categories & Templates](../features/policy-categories) — All 26 categories


--------------------------------------------------------------------------------
# Scope Policy
URL: https://waxell.ai/docs/observe/governance/scope
Description: Blast-radius governance — limits the maximum impact of a single agent execution by capping records modified/deleted, files changed, transaction amounts, and API write counts.
--------------------------------------------------------------------------------

# Scope Policy

The `scope` policy category enforces **blast-radius limits** — it controls the maximum impact a single agent execution can have on data, files, transactions, and external systems.

Use it when agents perform write operations (database updates, file modifications, financial transactions, external API calls) and you need guardrails to prevent runaway executions from causing disproportionate damage.

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `max_records_modified` | integer | `100` | Maximum number of records that can be modified in a single execution |
| `max_records_deleted` | integer | `0` | Maximum number of records that can be deleted (default 0 means deletions require explicit allowance) |
| `max_files_changed` | integer | `10` | Maximum number of files that can be created or modified |
| `max_transaction_amount` | number | `1000.00` | Maximum total dollar amount of financial transactions |
| `max_api_writes` | integer | `50` | Maximum number of external API write calls (POST, PUT, PATCH, DELETE) |
| `require_rollback_capability` | boolean | `false` | Warn if the agent context does not declare rollback support |
| `dry_run_first` | boolean | `false` | When `true`, sets `context._dry_run_mode = true` at before_workflow |
| `action_on_violation` | string | `"block"` | `"block"` raises `PolicyViolationError`; `"warn"` logs and continues |

## How It Works

The scope handler runs at all three enforcement phases:

### before_workflow

Stores the scope rules into `context._scope_rules` for mid-execution access. Optionally enables dry-run mode (`context._dry_run_mode = true`) and warns if rollback capability is required but not declared.

### mid_execution

Reads the running totals from the context and checks each limit:

1. `context.records_modified` vs `max_records_modified`
2. `context.records_deleted` vs `max_records_deleted`
3. `context.files_changed` vs `max_files_changed`
4. `context.transaction_total` vs `max_transaction_amount`
5. `context.api_writes` vs `max_api_writes`

The first exceeded limit produces an immediate result (BLOCK or WARN based on `action_on_violation`). Limits are checked in this order — only one violation is reported per mid_execution check.

### after_workflow

Audits all limits again against the final totals. Collects all violations (not just the first) and returns them as a warning list in the metadata, regardless of `action_on_violation`. This is an audit record — the after_workflow phase does not block on scope.

### When mid_execution Fires

`mid_execution` runs every time the agent calls `ctx.record_scope_impact()`. This means violations are caught as soon as the agent reports impact, not just at the end of execution. The agent is blocked before additional writes can occur.

## SDK Integration

### Recording Scope Impact

```python

from waxell_observe.errors import PolicyViolationError

async with waxell.WaxellContext(
    agent_name="data-agent",
    enforce_policy=True,
) as ctx:
    # Perform your data operation
    result = await update_database(query)

    # Report the blast radius — triggers mid_execution governance check
    ctx.record_scope_impact(
        records_modified=result.rows_updated,
        records_deleted=result.rows_deleted,
        files_changed=result.files_written,
        transaction_total=result.payment_amount,
        api_writes=result.external_calls,
    )
```

### Additive Totals

`record_scope_impact()` is **additive** — each call increments the running totals. Call it once per operation to build up an accurate picture of cumulative impact:

```python
async with waxell.WaxellContext(...) as ctx:
    # First batch operation
    ctx.record_scope_impact(records_modified=30, api_writes=5)

    # Second batch operation — totals are now 80 records, 12 writes
    ctx.record_scope_impact(records_modified=50, api_writes=7)

    # Third batch — totals now 105 records, 12 writes
    # If max_records_modified=100, this call triggers a mid_execution BLOCK
    ctx.record_scope_impact(records_modified=25)
```

### Handling Violations

```python
try:
    async with waxell.WaxellContext(
        agent_name="data-agent",
        enforce_policy=True,
    ) as ctx:
        result = await process_records(batch)
        ctx.record_scope_impact(
            records_modified=len(result.modified),
            records_deleted=len(result.deleted),
            transaction_total=result.total_amount,
        )
        ctx.set_result(result)

except PolicyViolationError as e:
    # e.g. "Records modified (250) exceeds limit (100)"
    print(f"Scope limit exceeded: {e}")
    await notify_operator(str(e))
```

## Example Policies

### Conservative Data Agent

Strict limits for agents with broad database access:

```json
{
  "max_records_modified": 100,
  "max_records_deleted": 0,
  "max_files_changed": 10,
  "max_transaction_amount": 1000.00,
  "max_api_writes": 50,
  "require_rollback_capability": false,
  "action_on_violation": "block"
}
```

### Financial Operations

Zero tolerance for large transactions; allow more record modifications:

```json
{
  "max_records_modified": 500,
  "max_records_deleted": 10,
  "max_files_changed": 20,
  "max_transaction_amount": 5000.00,
  "max_api_writes": 100,
  "require_rollback_capability": true,
  "action_on_violation": "block"
}
```

### Bulk ETL Pipeline

High limits for intentional bulk operations — use warn mode to audit without blocking:

```json
{
  "max_records_modified": 10000,
  "max_records_deleted": 1000,
  "max_files_changed": 50,
  "max_transaction_amount": 0,
  "max_api_writes": 0,
  "action_on_violation": "warn"
}
```

### Read-Only Enforcement

Prevent any writes — useful for analytics agents that should only read:

```json
{
  "max_records_modified": 0,
  "max_records_deleted": 0,
  "max_files_changed": 0,
  "max_transaction_amount": 0,
  "max_api_writes": 0,
  "action_on_violation": "block"
}
```

## Enforcement Flow

```
Agent starts (WaxellContext.__aenter__)
    │
    └── before_workflow governance runs
        ├── Stores scope rules into context._scope_rules
        ├── require_rollback_capability? → WARN if context.supports_rollback is False
        └── dry_run_first? → sets context._dry_run_mode = True

Agent runs — calls ctx.record_scope_impact(...)
    │
    └── mid_execution governance runs (each call)
        ├── records_modified > max_records_modified? → BLOCK/WARN
        ├── records_deleted > max_records_deleted? → BLOCK/WARN
        ├── files_changed > max_files_changed? → BLOCK/WARN
        ├── transaction_total > max_transaction_amount? → BLOCK/WARN
        ├── api_writes > max_api_writes? → BLOCK/WARN
        └── All within limits → ALLOW

Agent completes (WaxellContext.__aexit__)
    │
    └── after_workflow governance runs
        ├── Re-checks all limits against final totals
        ├── Collects all violations (not just first)
        ├── Violations → WARN with impact_summary metadata
        └── No violations → ALLOW with impact_summary metadata
```

## Creating via Dashboard

1. Navigate to **Governance > Policies**
2. Click **New Policy**
3. Select category **Scope**
4. Set your limit values
5. Set `action_on_violation` to `block` or `warn`
6. Set scope to target specific agents (e.g., `data-agent`)
7. Enable

## Creating via API

```bash
curl -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  https://acme.waxell.dev/waxell/v1/policies/ \
  -d '{
    "name": "Conservative Data Agent Limits",
    "category": "scope",
    "rules": {
      "max_records_modified": 100,
      "max_records_deleted": 0,
      "max_files_changed": 10,
      "max_transaction_amount": 1000.00,
      "max_api_writes": 50,
      "action_on_violation": "block"
    },
    "scope": {
      "agents": ["data-agent"]
    },
    "enabled": true
  }'
```

## Observability

### Governance Tab

Scope evaluations appear at each enforcement phase:

**before_workflow** (always ALLOW unless rollback required):

| Field | Example |
|-------|---------|
| **Policy name** | Conservative Data Agent Limits |
| **Action** | `allow` |
| **Phase** | `before_workflow` |
| **Reason** | "Scope limits stored for enforcement" |

**mid_execution** (per `record_scope_impact` call):

| Field | Example |
|-------|---------|
| **Action** | `block` |
| **Phase** | `mid_execution` |
| **Reason** | "Records modified (250) exceeds limit (100)" |
| **Metadata** | `{"records_modified": 250, "limit": 100}` |

**after_workflow** (final audit):

| Field | Example |
|-------|---------|
| **Action** | `allow` |
| **Phase** | `after_workflow` |
| **Reason** | "Scope audit passed (modified=8, deleted=0, files=2, tx=$450.00)" |
| **Metadata** | `{"impact_summary": {"records_modified": 8, "records_deleted": 0, ...}}` |

### Impact Summary Metadata

Every after_workflow evaluation includes a full `impact_summary`:

```json
{
  "impact_summary": {
    "records_modified": 8,
    "records_deleted": 0,
    "files_changed": 2,
    "transaction_total": 450.0,
    "api_writes": 3
  }
}
```

This is available even on successful runs, making it useful for auditing the actual impact of every execution.

## Common Gotchas

1. **`max_records_deleted` defaults to `0`.** This means the default policy blocks any record deletion. If your agent deletes records legitimately, set an explicit `max_records_deleted` value.

2. **`record_scope_impact()` is additive, not absolute.** Each call adds to the running total. If you call it three times with `records_modified=50` each, the total is 150 — not 50.

3. **`mid_execution` only reports the first violation.** If records_modified and transaction_total both exceed limits in the same call, only the first exceeded limit (by check order) is reported. The after_workflow phase reports all violations.

4. **`after_workflow` always warns, never blocks.** Even if `action_on_violation=block`, the after_workflow phase issues warnings in its result metadata. Use mid_execution for blocking enforcement.

5. **Rollback checks always warn, never block.** Setting `require_rollback_capability=true` when the context does not support rollback produces a WARN at before_workflow, not a BLOCK. The agent still runs.

6. **Zero values are valid limits.** `max_transaction_amount=0` blocks any financial transaction. `max_api_writes=0` blocks all external API writes. Use this for read-only enforcement.

7. **Limits are per-execution, not per-day.** The scope policy resets with each new `WaxellContext`. For rate-limiting across executions, use the `rate-limit` policy category.

## Combining with Other Policies

- **cost**: Use `cost` to limit LLM spend per execution; use `scope` to limit data impact. They operate on different dimensions.
- **audit**: Enable `audit` alongside `scope` to get a permanent record of every impact summary. The `after_workflow` metadata from scope is captured in the audit trail.
- **approval**: Combine `scope` with `approval` to require human sign-off before executions that would approach scope limits.
- **operations**: Use `operations` to limit execution time and retries; use `scope` to limit data blast radius. Both protect against runaway executions in complementary ways.

## Next Steps

- [Policy & Governance](../features/governance) -- How policy enforcement works
- [Grounding Policy](./grounding) -- Govern factual accuracy of agent outputs
- [Audit Policy](../features/policy-categories) -- Permanent record of agent impact
- [Approval Policy](../features/approval-workflows) -- Human-in-the-loop for high-impact operations
- [Policy Categories & Templates](../features/policy-categories) -- All 26 categories


--------------------------------------------------------------------------------
# Code Execution Policy
URL: https://waxell.ai/docs/observe/governance/code-execution
Description: Govern what code your AI agents can execute -- languages, commands, paths, packages, sandboxing, and human review.
--------------------------------------------------------------------------------

# Code Execution Policy

The `code-execution` policy category governs what generated code agents can execute. It controls allowed languages, blocked commands, restricted filesystem paths, package installation, sandbox requirements, and optional human review before execution.

Use it when your agents generate and run code -- whether in a cloud sandbox (E2B) or locally via subprocess.

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `allowed_languages` | string[] | `["python", "javascript", "sql"]` | Languages agents are allowed to execute |
| `blocked_languages` | string[] | `[]` | Languages explicitly denied (takes precedence over allowed) |
| `blocked_commands` | string[] | `["rm -rf", "chmod", "chown"]` | Command strings that are forbidden in code. Uses **substring matching** against the full command |
| `blocked_paths` | string[] | `["/etc", "/var", "~/.ssh", "~/.aws"]` | Filesystem paths that code cannot access. Matches using **path prefix** |
| `allowed_paths` | string[] | `["/workspace", "/tmp"]` | If set, code can *only* access these paths (allowlist). Leave empty to allow all paths not in `blocked_paths` |
| `allow_package_install` | boolean | `false` | Whether the agent can install packages at runtime |
| `allowed_packages` | string[] | `[]` | If `allow_package_install` is true, restrict to these packages |
| `max_execution_time_seconds` | integer | `30` | Maximum allowed execution duration per command |
| `max_output_size_kb` | integer | `1024` | Maximum output size per execution |
| `require_review` | boolean | `false` | Require human approval before every code execution |
| `sandbox_required` | boolean | `true` | Whether code must run in a sandbox (E2B). Set to `false` for local subprocess execution |
| `action_on_violation` | string | `"block"` | `"block"` to prevent execution, `"warn"` to log and continue |

## How Matching Works

Understanding the difference between **blocked commands** and **blocked paths** is critical for configuring policies correctly.

### Blocked Commands -- Substring Match

Blocked commands use **case-insensitive substring matching** against the full command string. This means the blocked command text must appear somewhere in the generated command.

| Blocked Command | Generated Code | Match? | Why |
|----------------|---------------|--------|-----|
| `rm -rf` | `rm -rf /etc/*` | Yes | `"rm -rf"` is in `"rm -rf /etc/*"` |
| `wget` | `wget https://example.com/file.sh` | Yes | `"wget"` is in the command |
| `curl` | `curl -X POST https://api.example.com` | Yes | `"curl"` is in the command |
| `ls` | `ls -la` | Yes | `"ls"` is in `"ls -la"` |
| `rm -rf` | `rm file.txt` | No | `"rm -rf"` is not in `"rm file.txt"` |

**Common Mistake**
Putting a command like `ls` in **blocked_paths** instead of **blocked_commands** won't work. `blocked_paths` checks filesystem paths (like `/etc` or `~/.ssh`), not command names. If you want to block a command, add it to `blocked_commands`.


### Blocked Paths -- Prefix Match

Blocked paths use **prefix matching** against the filesystem paths extracted from the command. The SDK extracts paths starting with `/` or `~/` from the command string before evaluation.

| Blocked Path | Path in Command | Match? | Why |
|-------------|----------------|--------|-----|
| `/etc` | `/etc/shadow` | Yes | `/etc/shadow` starts with `/etc` |
| `/etc` | `/etc/passwd` | Yes | `/etc/passwd` starts with `/etc` |
| `~/.ssh` | `~/.ssh/id_rsa` | Yes | `~/.ssh/id_rsa` starts with `~/.ssh` |
| `/var` | `/var/log/syslog` | Yes | `/var/log/syslog` starts with `/var` |
| `/etc` | `/tmp/etc-backup` | No | `/tmp/etc-backup` does not start with `/etc` |

### Allowed Paths -- Allowlist Mode

If `allowed_paths` is set (non-empty), only those path prefixes are permitted. Any path not matching an allowed prefix is blocked. If `allowed_paths` is empty, all paths not in `blocked_paths` are allowed.

| Configuration | Behavior |
|--------------|----------|
| `blocked_paths: ["/etc"], allowed_paths: []` | Block `/etc/*`, allow everything else |
| `blocked_paths: [], allowed_paths: ["/workspace", "/tmp"]` | Only allow `/workspace/*` and `/tmp/*`, block all others |
| `blocked_paths: ["/etc"], allowed_paths: ["/workspace"]` | Both apply -- `/etc` blocked, only `/workspace` allowed |

## Sandbox vs Local Execution

Code execution policies support two modes depending on how your agent runs code:

### Sandboxed Execution (E2B)

For agents that execute code in a cloud sandbox (E2B). The sandbox provides an isolated environment, so the policy adds defense-in-depth.

```json
{
  "name": "Sandbox Code Governance",
  "category": "code-execution",
  "rules": {
    "allowed_languages": ["python", "javascript"],
    "blocked_commands": ["rm -rf", "chmod", "chown"],
    "blocked_paths": ["/etc", "~/.ssh", "~/.aws"],
    "allowed_paths": ["/workspace", "/tmp"],
    "allow_package_install": false,
    "max_execution_time_seconds": 30,
    "max_output_size_kb": 1024,
    "sandbox_required": true,
    "action_on_violation": "block"
  }
}
```

Key settings:
- **`sandbox_required: true`** -- warns if no sandbox is available
- **`allowed_paths`** set -- restricts execution to specific directories inside the sandbox

### Local Execution (Subprocess)

For agents that generate shell commands and run them on the host machine via `subprocess`. More dangerous than sandboxed execution, so policies are your primary safety net.

```json
{
  "name": "Local Code Execution Governance",
  "category": "code-execution",
  "rules": {
    "allowed_languages": ["python", "shell"],
    "blocked_commands": ["rm -rf", "chmod", "chown", "wget", "curl"],
    "blocked_paths": ["/etc", "~/.ssh", "~/.aws", "/var"],
    "allowed_paths": [],
    "allow_package_install": false,
    "max_execution_time_seconds": 30,
    "max_output_size_kb": 1024,
    "sandbox_required": false,
    "require_review": false,
    "action_on_violation": "block"
  }
}
```

Key differences from sandbox mode:
- **`sandbox_required: false`** -- no cloud sandbox, executing locally
- **`allowed_languages` includes `"shell"`** -- the agent generates shell commands, not just Python
- **`blocked_commands` includes `"wget"`, `"curl"`** -- prevent data exfiltration from the host
- **`allowed_paths` is empty** -- we block specific dangerous paths instead of allowlisting

**TIP**
For local execution, block `wget` and `curl` to prevent data exfiltration. In a sandbox, these are less dangerous since the sandbox is isolated.


## Enforcement

Code execution policies are evaluated at **mid-execution** time. When the SDK records a code execution event, the policy is checked before the code runs.

### Enforcement Flow

```
Agent generates code
    │
    ▼
SDK calls record_code_execution(language, code, paths)
    │
    ▼
Controlplane evaluates code-execution policies
    │
    ├── Check language restrictions
    ├── Check blocked commands (substring match)
    ├── Check blocked paths (prefix match)
    ├── Check allowed paths (if set)
    ├── Check package restrictions
    └── Check execution time / output size limits
    │
    ├── ALLOW → Code executes
    ├── WARN  → Code executes, warning logged in trace
    └── BLOCK → PolicyViolationError raised, code never runs
```

### In Agent Code

```python

from waxell_observe import WaxellContext
from waxell_observe.errors import PolicyViolationError

waxell.init()

try:
    async with WaxellContext(
        agent_name="code-agent",
        workflow_name="code-exec",
        enforce_policy=True,
        mid_execution_governance=True,
    ) as ctx:
        # Record the code execution -- governance checks happen here
        ctx.record_code_execution(
            language="shell",
            code="ls -la /tmp",
            paths=["/tmp"],
        )

        # If we reach here, the policy allowed it
        result = subprocess.run("ls -la /tmp", shell=True, capture_output=True)

except PolicyViolationError as e:
    print(f"Blocked: {e}")
    # The code never executed
```

### With the Subprocess Instrumentor

For local execution agents, the subprocess instrumentor automatically intercepts `subprocess.run()` calls and enforces governance without manual `record_code_execution()` calls:

```python

from waxell_observe.instrumentors import instrument_all

waxell.init()
instrument_all(libraries=["subprocess"])  # Opt-in: auto-intercept subprocess

async with WaxellContext(
    agent_name="code-agent",
    enforce_policy=True,
    mid_execution_governance=True,
) as ctx:
    # The instrumentor automatically:
    # 1. Extracts the command from subprocess.run()
    # 2. Calls record_code_execution() with language="shell"
    # 3. Checks governance -- raises PolicyViolationError if blocked
    # 4. Only executes the subprocess if governance allows it
    result = subprocess.run("ls -la /tmp", shell=True, capture_output=True)
```

**INFO**
The subprocess instrumentor is **opt-in only**. It patches `subprocess.run()`, `subprocess.Popen()`, and `os.system()`. It only activates inside a `WaxellContext` -- outside of one, subprocess calls work normally with zero overhead.


## Human Review

Set `require_review: true` to gate every code execution on human approval, regardless of whether the command itself would be allowed by other rules.

When review is enabled:

1. Agent generates code
2. SDK presents the code to the approval handler (e.g., terminal prompt, Slack message)
3. **If approved** -- governance rules are still evaluated. A dangerous command is blocked even if approved.
4. **If denied** -- `PolicyViolationError` raised immediately

```python
from waxell_observe.approval import prompt_approval

waxell.init(on_policy_block=prompt_approval)

async with WaxellContext(
    agent_name="code-agent",
    enforce_policy=True,
    mid_execution_governance=True,
) as ctx:
    ctx.record_code_execution(language="shell", code="uname -a")
    # Terminal prompt: "Approve this operation? (y/n):"
    # If approved AND policy allows → executes
    # If approved BUT policy blocks → still blocked
    # If denied → PolicyViolationError
```

**WARNING**
Human review is an **additional gate**, not a bypass. Even if a reviewer approves a command, policy rules (blocked commands, blocked paths) are still enforced. This prevents accidental approval of dangerous commands.


See [Approval Workflows](../features/approval-workflows) for custom handlers (Slack, webhooks, etc).

## Creating a Policy

### Via Dashboard

1. Navigate to **Governance > Policies**
2. Click **Create Policy**
3. Select category: **Code Execution**
4. Configure rules:

| Field | Sandbox Mode | Local Mode |
|-------|-------------|------------|
| Allowed Languages | `python`, `javascript` | `python`, `shell` |
| Blocked Commands | `rm -rf`, `chmod`, `chown` | `rm -rf`, `chmod`, `chown`, `wget`, `curl` |
| Blocked Paths | `/etc`, `~/.ssh`, `~/.aws` | `/etc`, `~/.ssh`, `~/.aws`, `/var` |
| Allowed Paths | `/workspace`, `/tmp` | _(leave empty)_ |
| Allow Package Install | OFF | OFF |
| Max Execution Time | `30` | `30` |
| Max Output Size | `1024` | `1024` |
| Require Review | OFF | OFF |
| Sandbox Required | **ON** | **OFF** |
| Action on Violation | Block | Block |

5. Set **Scope** to target specific agents, or leave empty for all agents
6. Save

### Via API

```bash
curl -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  https://acme.waxell.dev/waxell/v1/policies/ \
  -d '{
    "name": "Local Code Execution Governance",
    "category": "code-execution",
    "rules": {
      "allowed_languages": ["python", "shell"],
      "blocked_commands": ["rm -rf", "chmod", "chown", "wget", "curl"],
      "blocked_paths": ["/etc", "~/.ssh", "~/.aws", "/var"],
      "allowed_paths": [],
      "allow_package_install": false,
      "max_execution_time_seconds": 30,
      "max_output_size_kb": 1024,
      "require_review": false,
      "sandbox_required": false,
      "action_on_violation": "block"
    },
    "scope": {
      "agents": ["local-code-agent"]
    }
  }'
```

## Observability

Code execution governance events appear in the execution detail:

### Trace Tab

- **`code_exec:shell`** span -- the governance check. Present for every `record_code_execution()` call. This span does **not** mean the code executed -- it means the code was *evaluated* for governance.
- **`subprocess.run`** span -- the actual execution. Only present if governance allowed the command to run. If you see `code_exec:shell` but no `subprocess.run`, the command was blocked before execution.

### Governance Tab

- **Pre-execution**: Shows `code_execution_rules` metadata with the full policy configuration
- **Mid-execution**: Per-command evaluations showing ALLOW, WARN, or BLOCK with the specific reason (e.g., "Blocked command 'rm -rf' found in code" or "Code accessed blocked path '/etc/shadow'")

### Incidents

Policy violations create incidents visible in **Governance > Incidents**. Each incident includes:
- The policy that triggered it
- The blocked command or path
- The agent name and execution ID
- Whether it was a mid-execution block (prevented execution) or post-execution audit finding

## Warn vs Block

Switch between enforcement modes using `action_on_violation`:

| Mode | Behavior | Use Case |
|------|----------|----------|
| `"block"` | `PolicyViolationError` raised, code never executes | Production -- prevent dangerous actions |
| `"warn"` | Warning logged, code still executes | Monitoring -- discover what agents are doing before enforcing |

Start with `"warn"` to audit your agents' behavior, then switch to `"block"` once you've tuned the rules.

## Combining with Other Policies

Code execution policies work alongside other categories:

- **Safety policies** limit total steps and tool calls; code execution governs the content of those calls
- **Content policies** scan for PII or credentials in inputs/outputs; code execution scans for dangerous commands
- **Approval policies** gate high-stakes actions; `require_review` specifically gates code execution
- **Network policies** control outbound access; `blocked_commands` with `wget`/`curl` prevents downloads at the command level

## Common Gotchas

1. **Blocked commands vs blocked paths**: `blocked_commands` does substring matching on the command string (`"rm -rf"` matches `"rm -rf /tmp"`). `blocked_paths` does prefix matching on extracted filesystem paths (`"/etc"` matches the path `/etc/shadow`). Don't mix them up.

2. **Allowed paths empty = allow all**: When `allowed_paths` is empty, all paths not in `blocked_paths` are permitted. Set `allowed_paths` to restrict to a specific allowlist.

3. **Multiple code execution policies**: If you have both a sandbox policy and a local policy, scope them to different agents. Otherwise both evaluate for every agent, and the sandbox policy's `sandbox_required: true` will warn for agents running locally.

4. **Language must match**: The `allowed_languages` check requires the language string to match exactly. For local subprocess agents, use `"shell"` -- not `"bash"` or `"sh"`.

5. **Substring matching is broad**: Blocking `"rm"` would also block `"format"` because `"rm"` appears in `"format"`. Be specific: use `"rm -rf"` instead of `"rm"`.


--------------------------------------------------------------------------------
# Input Validation Policy
URL: https://waxell.ai/docs/observe/governance/input-validation
Description: Pre-flight data validation that checks inputs for emptiness, size, type, HTML injection, and schema compliance before agent execution begins.
--------------------------------------------------------------------------------

# Input Validation Policy

The `input-validation` policy category is a **pre-flight data validator** -- it checks inbound data before the agent begins execution. It validates emptiness, size limits, input type, HTML/script injection, and JSON schema compliance.

Use it when you need to ensure agents only process well-formed, safe, appropriately-sized inputs.

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `validate_schema` | boolean | `false` | Enable JSON schema validation against `input_schema` |
| `input_schema` | object | `&#123;&#125;` | JSON Schema-like object with a `required` field list |
| `max_input_size_kb` | integer | `100` | Maximum input size in kilobytes |
| `allowed_input_types` | string[] | `[]` (allow all) | Allowed input types: `text`, `json`, `binary` |
| `reject_empty_input` | boolean | `true` | Block empty/null/empty-dict inputs |
| `sanitize_html` | boolean | `true` | Block inputs containing `<script` or `javascript:` |
| `action_on_violation` | string | `"block"` | Action when validation fails: `block` or `warn` |

## How It Works

The input validation handler runs at **before_workflow** only. It reads `context.inputs` -- the dict or string passed to `WaxellContext(inputs=...)`. There is no mid_execution phase. The after_workflow phase is a stub that always returns ALLOW.

### Validation Order

Checks run in this exact order, short-circuiting on first failure:

1. **Empty check** -- is the input null, empty string, or empty dict?
2. **Size check** -- does the serialized input exceed `max_input_size_kb`?
3. **Type check** -- is the input type in `allowed_input_types`?
4. **HTML check** -- does the input contain `<script` or `javascript:`?
5. **Schema check** -- are all `required` fields present?

If all checks pass, the handler returns ALLOW with reason "Input validation passed".

## Validation Details

### Empty Check

The handler considers these values empty:
- `None`
- `""` (empty string)
- `&#123;&#125;` (empty dict)

**`&#123;"query": ""&#125;` is NOT empty**
A dict with keys -- even if the values are empty strings -- passes the empty check. Only `None`, `""`, and `&#123;&#125;` are considered empty. If you need to reject dicts with empty values, use schema validation with required fields.


### Size Check

The handler serializes inputs to a JSON string (for dicts/lists) or converts to `str()`, then measures the UTF-8 byte length in kilobytes. Uses strict `>` comparison:

```
size_kb = len(json.dumps(inputs).encode("utf-8")) / 1024
if size_kb > max_input_size_kb: BLOCK
```

### Type Check

Type detection is based on Python types:
- `dict` or `list` = `"json"`
- Everything else = `"text"`

If `allowed_input_types` is an empty list `[]`, all types are allowed. Only a non-empty list restricts types.

### HTML/Script Check

Case-insensitive substring match in the serialized input string:
- `<script` (catches `<script>`, `<SCRIPT>`, `<Script type="...">`)
- `javascript:` (catches `javascript:alert()`, `JAVASCRIPT:void(0)`)

This is a simple substring check, not a full HTML parser or sanitizer.

### Schema Check

Only checks the `required` field from `input_schema`. For each field name in the `required` list, verifies the key exists in the input dict. Does not validate field types, formats, or nested structures.

```json
{
  "input_schema": {
    "required": ["query", "user_id"]
  }
}
```

With this schema, `{"query": "hello", "user_id": "u1"}` passes but `{"query": "hello"}` fails with "Required field 'user_id' missing from input".

**Schema validation is basic**
The `input_schema` only checks `required` field presence in dict inputs. It does not perform full JSON Schema validation (no type checks, no pattern matching, no nested object validation). For complex validation needs, implement custom logic in your agent.


## Example Policies

### Strict API Input

Full validation -- schema, size limit, type restriction, and HTML sanitization:

```json
{
  "validate_schema": true,
  "input_schema": {
    "required": ["query"]
  },
  "max_input_size_kb": 100,
  "allowed_input_types": ["json"],
  "reject_empty_input": true,
  "sanitize_html": true,
  "action_on_violation": "block"
}
```

### Lenient (Reject Empty Only)

Minimal validation -- only reject null/empty inputs:

```json
{
  "reject_empty_input": true,
  "sanitize_html": false,
  "validate_schema": false,
  "allowed_input_types": [],
  "action_on_violation": "warn"
}
```

### Size-Limited (Large Payload Protection)

Protect against oversized inputs without other restrictions:

```json
{
  "max_input_size_kb": 50,
  "reject_empty_input": true,
  "sanitize_html": true,
  "validate_schema": false,
  "allowed_input_types": [],
  "action_on_violation": "block"
}
```

## SDK Integration

### Using the Context Manager

Input validation reads `context.inputs`, so the inputs dict passed to `WaxellContext` is what gets validated:

```python

from waxell_observe.errors import PolicyViolationError

waxell.init()

try:
    async with waxell.WaxellContext(
        agent_name="processor",
        inputs={"query": user_input, "format": "json"},
        enforce_policy=True,
    ) as ctx:
        # If input validation policy is active and inputs fail
        # validation -> PolicyViolationError raised here
        # (before any agent work happens)

        result = await process_data(user_input)
        ctx.set_result(result)

except PolicyViolationError as e:
    print(f"Input validation block: {e}")
    # e.g. "Empty input rejected"
    # e.g. "Input size (150.3KB) exceeds limit (100KB)"
    # e.g. "HTML/script content detected in input"
    # e.g. "Required field 'query' missing from input"
```

### Using the Decorator

```python
@waxell.observe(
    agent_name="processor",
    enforce_policy=True,
)
async def process_input(query: str):
    # Input validation happens before this function body runs
    # The inputs dict is constructed from the function arguments
    return await process_data(query)
```

## Enforcement Flow

```
Agent starts (WaxellContext.__aenter__ or decorator entry)
    |
    +-- before_workflow governance runs
        |
        +-- Input validation handler reads context.inputs
        |
        +-- reject_empty_input?
        |   +-- inputs is None / "" / {} -> violation
        |   +-- otherwise -> continue
        |
        +-- max_input_size_kb?
        |   +-- serialize to JSON/str, measure UTF-8 bytes
        |   +-- size > limit -> violation
        |   +-- otherwise -> continue
        |
        +-- allowed_input_types (non-empty)?
        |   +-- dict/list = "json", else "text"
        |   +-- type not in list -> violation
        |   +-- otherwise -> continue
        |
        +-- sanitize_html?
        |   +-- "<script" or "javascript:" in input -> violation
        |   +-- otherwise -> continue
        |
        +-- validate_schema + input_schema?
        |   +-- check required fields in dict
        |   +-- missing field -> violation
        |   +-- otherwise -> continue
        |
        +-- All checks pass -> ALLOW
        |
        +-- Violation?
            +-- action_on_violation = "block" -> BLOCK (PolicyViolationError)
            +-- action_on_violation = "warn" -> WARN (agent continues)
```

The input validation handler only runs at **before_workflow**. There is no mid_execution check. The `after_workflow` phase always returns ALLOW with reason "Input validation audit complete".

## Creating via Dashboard

1. Navigate to **Governance > Policies**
2. Click **New Policy**
3. Select category **Input Validation**
4. Configure the validation rules:
   - Enable `reject_empty_input` to block null/empty inputs
   - Set `max_input_size_kb` for size limits
   - Set `allowed_input_types` to restrict input formats
   - Enable `sanitize_html` to block script injection
   - Enable `validate_schema` and set `input_schema` with required fields
5. Set `action_on_violation` to `block` or `warn`
6. Set scope to target specific agents (e.g., `input-validation-agent`)
7. Enable

## Creating via API

```bash
curl -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  https://acme.waxell.dev/waxell/v1/policies/ \
  -d '{
    "name": "Input Validation",
    "category": "input-validation",
    "rules": {
      "validate_schema": true,
      "input_schema": {"required": ["query"]},
      "max_input_size_kb": 100,
      "allowed_input_types": ["json"],
      "reject_empty_input": true,
      "sanitize_html": true,
      "action_on_violation": "block"
    },
    "scope": {
      "agents": ["processor"]
    },
    "enabled": true
  }'
```

## Observability

### Governance Tab

Input validation evaluations appear with:

| Field | Example |
|-------|---------|
| **Policy name** | Input Validation |
| **Action** | `allow`, `warn`, or `block` |
| **Category** | `input-validation` |
| **Reason** | "Input validation passed" |

For violations:

| Field | Example |
|-------|---------|
| **Reason** | "Empty input rejected" |
| **Reason** | "Input size (150.3KB) exceeds limit (100KB)" |
| **Reason** | "Input type 'json' not in allowed types: text" |
| **Reason** | "HTML/script content detected in input" |
| **Reason** | "Required field 'query' missing from input" |
| **Metadata** | `{"reject_empty_input": true}` |
| **Metadata** | `{"input_size_kb": 150.3, "limit_kb": 100}` |
| **Metadata** | `{"input_type": "json", "allowed_types": ["text"]}` |
| **Metadata** | `{"sanitize_html": true}` |
| **Metadata** | `{"missing_field": "query", "required": ["query"]}` |

### After-Workflow

The `after_workflow` phase always returns ALLOW with reason "Input validation audit complete". This is a stub -- no post-execution validation is performed.

## Common Gotchas

1. **`{"query": ""}` is NOT empty.** Only `None`, `""`, and `{}` are treated as empty. A dict with any keys passes the empty check, even if the values are empty strings.

2. **Schema validation is basic.** It only checks that `required` field names exist as keys in the input dict. It does not validate types, patterns, or nested structures. `{"query": 123}` passes a schema requiring `"query"`.

3. **`allowed_input_types: []` means ALL types allowed.** An empty list does not block anything. To restrict types, you must explicitly list the allowed types (e.g., `["json"]`).

4. **HTML check is simple substring matching.** It searches for `<script` and `javascript:` (case-insensitive) in the serialized input. It does not parse HTML, does not catch all XSS vectors, and may produce false positives on legitimate content mentioning those strings.

5. **No mid_execution phase.** All validation happens at before_workflow. Once inputs pass validation, the agent runs without further input checks. If your agent accepts additional inputs during execution, they are not validated by this handler.

6. **after_workflow is a stub.** It always returns ALLOW. Do not rely on it for post-execution validation.

7. **Size is measured on serialized form.** Dicts and lists are serialized with `json.dumps()` before measuring. The JSON serialization adds characters (braces, quotes, colons), so the measured size is slightly larger than the raw content.

8. **Type detection uses Python isinstance.** `dict` and `list` are "json", everything else is "text". If you pass a string that contains JSON, it is classified as "text", not "json".

## Next Steps

- [Policy & Governance](../features/governance) -- How policy enforcement works
- [Compliance Policy](./compliance) -- Meta-validator for regulatory frameworks
- [Content Policy](./content) -- Govern output content
- [Code Execution Policy](./code-execution) -- Govern generated code
- [Policy Categories & Templates](../features/policy-categories) -- All 26 categories


--------------------------------------------------------------------------------
# Output Egress Format Policy
URL: https://waxell.ai/docs/observe/governance/output-egress-format
Description: OWASP LLM05 exfiltration prevention — scans agent outputs for base64 blobs, external URLs, data URIs, and Unicode obfuscation that signal data smuggling.
--------------------------------------------------------------------------------

# Output Egress Format Policy

The `output-egress-format` policy enforces **OWASP LLM05** (insecure output handling, exfiltration sub-control). It scans the agent's final output for shapes that suggest the agent is smuggling data out of the system in disguise:

- **base64 blobs** — large encoded chunks in plain output (most legitimate output is plain text).
- **External URLs** — references to hosts outside the tenant's allowlist (catches the "embed a webhook in the response" pattern).
- **Data URIs** — inline `data:image/...` or `data:application/...` payloads that could carry encoded sensitive data.
- **Unicode obfuscation** — zero-width spaces, RTL overrides, and Cyrillic homoglyphs above a density threshold.

Pure regex / set-lookup, sub-millisecond per output. Tier 1 (cheap).

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `block_base64` | boolean | `true` | Flag base64-shaped blobs |
| `min_base64_length` | integer | `200` | Minimum chars before a base64 match counts |
| `block_external_urls` | boolean | `false` | Enforce the URL allowlist (opt-in) |
| `allowed_url_domains` | string[] | `[]` | Hostnames or suffix matches (e.g., `example.com` matches `api.example.com`) |
| `block_data_uri` | boolean | `true` | Flag inline `data:...;base64,` URIs |
| `block_unicode_obfuscation` | boolean | `false` | Flag zero-width chars and homoglyph density |
| `max_homoglyph_pct` | number | `0.05` | Cyrillic-homoglyph density threshold (0–1) |
| `action_on_violation` | string | `"block"` | `"block"` or `"warn"` |

Optional: `scan_mid_execution` (boolean, default `false`) — enables a mid-run check on `response_preview` for streaming surfaces.

## How It Works

The handler runs at **after_workflow** (primary — needs the complete output). `before_workflow` is a no-op. `mid_execution` runs only when `scan_mid_execution=true`.

| Phase | What it checks | Source |
|-------|----------------|--------|
| `before_workflow` | (no-op) | — |
| `mid_execution` (opt-in) | Latest `response_preview` against all 4 detectors | `context.response_preview` |
| `after_workflow` | Final output against all 4 detectors | `context.output_text` → `response_preview` → `result` (if str) |

### Scan Order (short-circuits on first hit)

1. Data URI — fastest regex.
2. Base64 blob ≥ `min_base64_length`.
3. External URL not on the allowlist.
4. Zero-width chars, then homoglyph density.

### Context Attributes Read

| Attribute | Phase | Purpose |
|-----------|-------|---------|
| `context.output_text` | after | Primary scan target |
| `context.response_preview` | mid, after (fallback) | Streaming or per-step preview |
| `result` (parameter) | after (last fallback) | When the result is a plain string |

## Example Policy

Customer-support agent — outputs are plain prose; aggressively block exfil shapes and limit URLs to the company domain and a known partner:

```json
{
  "block_base64": true,
  "min_base64_length": 200,
  "block_external_urls": true,
  "allowed_url_domains": ["acme.com", "trusted-partner.com"],
  "block_data_uri": true,
  "block_unicode_obfuscation": true,
  "max_homoglyph_pct": 0.02,
  "action_on_violation": "block"
}
```

## SDK Integration

```python


waxell.init()

@waxell.observe(agent_name="support-agent", enforce_policy=True)
async def answer(query: str) -> str:
    # after_workflow: scans the returned string for egress shapes.
    # If a base64 blob or non-allowlisted URL appears, the policy blocks.
    return await generate_response(query)
```

## Observability

| Field | Example |
|-------|---------|
| **Category** | `output-egress-format` |
| **Action** | `block` |
| **Reason** | "Output contains a base64-shaped blob (412 chars). Possible exfiltration." |
| **Metadata** | `{"phase": "after", "signal": "base64_blob", "length": 412, "owasp": "LLM05"}` |

External URL violation:

| Field | Example |
|-------|---------|
| **Reason** | "Output references external URL host 'evil-webhook.io' not on the allowlist." |
| **Metadata** | `{"signal": "external_url", "host": "evil-webhook.io", "owasp": "LLM05"}` |

## Common Gotchas

- **Base64 regex matches any 200+ char run of `[A-Za-z0-9+/]`.** This will false-positive on long hex hashes, JWT tokens, and some content IDs. Raise `min_base64_length` or set `block_base64=false` for agents that legitimately emit tokens.
- **The URL allowlist uses suffix-match by default.** Entry `example.com` matches `example.com` AND `api.example.com`. To match only the apex, list it explicitly without subdomains and use exact comparison externally.
- **`block_external_urls=false` is the default.** Until you populate `allowed_url_domains` and flip the flag, URL scanning is disabled.
- **Empty `allowed_url_domains` with `block_external_urls=true` blocks ALL URLs.** Includes `https://acme.com` if you forgot to add it.
- **Homoglyph detector only catches Cyrillic→Latin lookalikes.** Greek (e.g., `omicron`) and other scripts are not scanned.
- **`scan_mid_execution` is off by default.** Streaming surfaces should enable it; non-streaming agents save CPU by leaving it off.
- **The handler reads `output_text` first, then falls back.** Make sure your runtime populates `context.output_text` for the most accurate scan.

## Next Steps

- [Policy Categories](../features/policy-categories) — All 49 categories
- [Content Policy](./content) — Configurable input/output content scanning (block/warn/redact)
- [Network Policy](./network) — Outbound HTTP allowlists at the tool layer
- [Prompt Injection Guard](./prompt-injection-guard) — Companion control on the input side (LLM01)


--------------------------------------------------------------------------------
# Grounding Policy
URL: https://waxell.ai/docs/observe/governance/grounding
Description: Hallucination and factual accuracy governance — enforces minimum grounding scores, citation requirements, unsupported claim limits, and output confidence thresholds for RAG and research agents.
--------------------------------------------------------------------------------

# Grounding Policy

The `grounding` policy category enforces **factual accuracy standards** — it requires agent outputs to be grounded in cited sources, with measurable confidence scores and limits on unsupported claims.

Use it for RAG (retrieval-augmented generation) agents, research assistants, medical or legal Q&A, and any agent where hallucinated or unsupported output carries real risk.

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `require_source_grounding` | boolean | `false` | If `true`, block (or warn) when zero citations are provided |
| `min_grounding_score` | number | `0.7` | Minimum acceptable score per grounding claim (0.0-1.0). Checked at mid_execution |
| `min_citations` | integer | `1` | Minimum number of source citations required in the output |
| `max_unsupported_claims` | integer | `null` | Maximum number of claims without source support. `null` means no limit |
| `factual_consistency_check` | boolean | `false` | Reserved for future cross-claim consistency validation |
| `abstention_threshold` | number | `null` | If `output_confidence` falls below this threshold, block (the agent should abstain rather than guess) |
| `abstention_response` | string | `null` | Custom message to return when abstention threshold is triggered |
| `action_on_violation` | string | `"warn"` | `"block"` raises `PolicyViolationError`; `"warn"` logs and continues |
| `score_relevance_floor` | number | `null` | Ignore retrieval scores below this floor. Only scores at or above this value are checked against `min_grounding_score` |
| `score_eval_mode` | string | `"all"` | How to evaluate filtered scores: `"all"` (each must pass), `"average"` (mean must pass), `"top_n"` (only top N checked) |
| `score_top_n` | integer | `3` | When `score_eval_mode="top_n"`, how many top scores to evaluate |

**Default action is "warn"**
The grounding handler defaults to `action_on_violation: "warn"` unlike most other handlers that default to `"block"`. This is intentional — grounding quality degrades gradually and initial deployments often need to calibrate thresholds. Change to `"block"` once you have tuned your score thresholds against real traffic.


## How It Works

The grounding handler runs at two enforcement phases:

### before_workflow

Stores the grounding rules into `context._grounding_rules`. Always returns ALLOW — no blocking happens at before_workflow.

### mid_execution

Checks per-claim grounding scores as they are reported:

1. Reads `context.grounding_scores` (a list of float values 0.0-1.0)
2. If the list is empty, returns ALLOW immediately (nothing to check yet)
3. For each score, checks against `min_grounding_score`
4. The first score below the threshold produces a BLOCK or WARN

This phase fires every time `ctx.record_grounding()` is called with new scores.

### after_workflow

Audits the final output state against citation and confidence requirements:

1. **Citation count**: `len(context.citations) < min_citations` → violation
2. **Require grounding**: `require_source_grounding=true` and `len(context.citations) == 0` → violation
3. **Unsupported claims**: `context.unsupported_claims > max_unsupported_claims` → violation
4. **Abstention threshold**: `context.output_confidence < abstention_threshold` → violation

All violations are collected and returned together in the `warnings` list.

## Filtering for RAG Pipelines

Real RAG pipelines with `top_k=5` always return tail results with low similarity scores. Without filtering, `min_grounding_score` would block on irrelevant tail results, making the policy unusable.

### The Relevance Floor

Set `score_relevance_floor` to ignore retrieval results below a relevance threshold. Only scores at or above the floor are evaluated against `min_grounding_score`.

**Example**: A query about "2008 financial crisis" returns 5 results:
- 0.92, 0.87, 0.85 (relevant finance docs)
- 0.35, 0.22 (irrelevant tail results)

With `score_relevance_floor: 0.5`, only the three relevant scores are checked. All three pass `min_grounding_score: 0.7`, so the policy allows the response.

Without the floor, the policy would block on the 0.35 score — a false positive from an irrelevant retrieval result.

### Edge Case: Nothing Relevant

If NO scores pass the floor, the policy triggers a violation: "No grounding scores above relevance floor — all retrieved results appear irrelevant." This correctly catches the case where retrieval returned nothing useful.

### Evaluation Modes

| Mode | Behavior |
|------|----------|
| `"all"` (default) | Every filtered score must pass `min_grounding_score` |
| `"average"` | The mean of filtered scores must pass the threshold |
| `"top_n"` | Only the top N filtered scores are checked (set N with `score_top_n`) |

### Example: RAG Pipeline Policy

```json
{
  "min_grounding_score": 0.7,
  "score_relevance_floor": 0.5,
  "score_eval_mode": "all",
  "min_citations": 1,
  "action_on_violation": "block"
}
```

### Example: Lenient Average-Based Policy

```json
{
  "min_grounding_score": 0.7,
  "score_relevance_floor": 0.4,
  "score_eval_mode": "average",
  "action_on_violation": "warn"
}
```

**When to use filtering**
Set `score_relevance_floor` whenever your grounding scores come from retrieval similarity (e.g., Pinecone, FAISS, Weaviate). Without filtering, `top_k > 3` retrieval will almost always produce tail results below any useful threshold.

If your scores come from LLM self-evaluation or NLI classifiers (where every score is meaningful), leave `score_relevance_floor` unset to use the legacy per-score checking.


## SDK Integration

### Recording Grounding Data

```python

from waxell_observe.errors import PolicyViolationError

async with waxell.WaxellContext(
    agent_name="research-agent",
    enforce_policy=True,
) as ctx:
    # Retrieve sources and generate answer
    sources = await retrieve_sources(query)
    answer, scores = await synthesize_with_grounding(query, sources)

    # Report grounding data — triggers mid_execution check on scores
    ctx.record_grounding(
        grounding_scores=scores,           # per-claim scores [0.9, 0.85, 0.92]
        citations=sources,                 # ["Source A", "Source B", "Source C"]
        unsupported_claims=[],             # any claims without source support
        output_confidence=0.88,            # overall confidence in the answer
    )

    ctx.set_result({"answer": answer, "citations": sources})
```

### Grounding Data Fields

| Field | Type | What to put here |
|-------|------|-----------------|
| `grounding_scores` | list[float] | One score per claim or sentence in the output (0.0 = unsupported, 1.0 = fully grounded) |
| `citations` | list[str] | Source identifiers — URLs, document names, DOIs, database record IDs |
| `unsupported_claims` | list[str] | The actual text of claims you could not ground in any source |
| `output_confidence` | float | Your model's or pipeline's overall confidence that the full answer is accurate |

### Handling Violations

```python
try:
    async with waxell.WaxellContext(
        agent_name="research-agent",
        enforce_policy=True,
    ) as ctx:
        sources = await retrieve_sources(query)
        answer, scores = await synthesize_with_grounding(query, sources)

        ctx.record_grounding(
            grounding_scores=scores,
            citations=sources,
            unsupported_claims=[],
            output_confidence=0.88,
        )
        ctx.set_result({"answer": answer})

except PolicyViolationError as e:
    # e.g. "Grounding score (0.42) below threshold (0.7)"
    # e.g. "Citations (0) below minimum (1); Unsupported claims (2) exceeds max (0)"
    print(f"Grounding policy blocked: {e}")
    return abstention_message
```

## Example Policies

### Strict Research Assistant

Block any output with low grounding scores, missing citations, or low confidence:

```json
{
  "require_source_grounding": true,
  "min_grounding_score": 0.8,
  "min_citations": 2,
  "max_unsupported_claims": 0,
  "abstention_threshold": 0.5,
  "abstention_response": "I don't have sufficient grounded evidence to answer this accurately.",
  "action_on_violation": "block"
}
```

### Medical Q&A

High accuracy standards — abstain rather than guess:

```json
{
  "require_source_grounding": true,
  "min_grounding_score": 0.9,
  "min_citations": 3,
  "max_unsupported_claims": 0,
  "abstention_threshold": 0.7,
  "abstention_response": "I cannot provide a reliable answer based on available evidence. Please consult a qualified professional.",
  "action_on_violation": "block"
}
```

### Lenient RAG Agent

Warn on grounding issues but do not block execution during development:

```json
{
  "require_source_grounding": false,
  "min_grounding_score": 0.6,
  "min_citations": 1,
  "max_unsupported_claims": 3,
  "abstention_threshold": null,
  "action_on_violation": "warn"
}
```

### Citation-Only Enforcement

Require citations but do not enforce per-claim scores (useful when scores are not available):

```json
{
  "require_source_grounding": true,
  "min_citations": 1,
  "max_unsupported_claims": 2,
  "action_on_violation": "block"
}
```

## Enforcement Flow

```
Agent starts (WaxellContext.__aenter__)
    │
    └── before_workflow governance runs
        └── Stores grounding rules → ALLOW

Agent runs — calls ctx.record_grounding(grounding_scores=[...], ...)
    │
    └── mid_execution governance runs
        ├── grounding_scores empty? → ALLOW (nothing to check)
        ├── For each score:
        │   └── score < min_grounding_score? → BLOCK/WARN (first violation only)
        └── All scores pass → ALLOW

Agent completes (WaxellContext.__aexit__)
    │
    └── after_workflow governance runs
        ├── len(citations) < min_citations? → violation
        ├── require_source_grounding AND len(citations) == 0? → violation
        ├── unsupported_claims > max_unsupported_claims? → violation
        ├── output_confidence < abstention_threshold? → violation
        ├── Any violations?
        │   ├── action=block → BLOCK (PolicyViolationError)
        │   └── action=warn → WARN (agent completed, warning recorded)
        └── No violations → ALLOW with citation_count metadata
```

## Creating via Dashboard

1. Navigate to **Governance > Policies**
2. Click **New Policy**
3. Select category **Grounding**
4. Set your grounding thresholds
5. Set `action_on_violation` to `warn` initially to calibrate, then `block` for production
6. Set scope to target specific agents (e.g., `research-agent`)
7. Enable

## Creating via API

```bash
curl -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  https://acme.waxell.dev/waxell/v1/policies/ \
  -d '{
    "name": "Strict Research Grounding",
    "category": "grounding",
    "rules": {
      "require_source_grounding": true,
      "min_grounding_score": 0.8,
      "min_citations": 2,
      "max_unsupported_claims": 0,
      "abstention_threshold": 0.5,
      "action_on_violation": "block"
    },
    "scope": {
      "agents": ["research-agent"]
    },
    "enabled": true
  }'
```

## Observability

### Governance Tab

Grounding evaluations appear at each enforcement phase:

**mid_execution** (per-score check):

| Field | Example |
|-------|---------|
| **Policy name** | Strict Research Grounding |
| **Action** | `block` |
| **Phase** | `mid_execution` |
| **Reason** | "Grounding score (0.42) below threshold (0.7)" |
| **Metadata** | `{"score": 0.42, "threshold": 0.7}` |

**after_workflow** (citation and confidence audit):

| Field | Example |
|-------|---------|
| **Action** | `block` |
| **Phase** | `after_workflow` |
| **Reason** | "Citations (0) below minimum (1); No source citations provided (grounding required)" |
| **Metadata** | `{"warnings": [...], "citation_count": 0}` |

For successful runs:

| Field | Example |
|-------|---------|
| **Action** | `allow` |
| **Reason** | "Grounding audit passed (3 citations)" |
| **Metadata** | `{"citation_count": 3}` |

### Trace Tab

The `source_retrieval` tool span (if recorded with `ctx.record_tool_call`) shows the number of sources found. The grounding data (scores, citations, confidence) appears in the steps data alongside any LLM call records.

## LLM Grounding Judge

In addition to checking self-reported scores and citation counts, the grounding handler supports **LLM-based grounding evaluation** — an independent AI judge that reads the output and its cited sources, then scores whether the output is genuinely grounded.

This is an admin-controlled feature: developers record grounding data as usual, and the platform automatically runs the LLM judge when the policy enables it. No developer code changes are needed.

### LLM Judge Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `llm_grounding_check` | boolean | `false` | Enable LLM-based grounding evaluation |
| `llm_grounding_model` | string | `"gpt-4o-mini"` | Model to use for the judge (`gpt-*`, `claude-*`, `llama-*`, `mixtral-*`) |
| `llm_grounding_threshold` | number | `0.7` | Minimum LLM judge score (0.0-1.0) |
| `llm_grounding_criteria` | string | `""` | Custom evaluation criteria appended to the judge prompt |
| `llm_grounding_phase` | string | `"mid_execution"` | When to run the judge: `"mid_execution"`, `"after_workflow"`, or `"both"` |

### How It Works

The LLM judge evaluates three dimensions:

1. **Source support** — Is every factual claim in the output supported by at least one cited source?
2. **Citation correctness** — Are the citations used correctly (not fabricated or misattributed)?
3. **Scope adherence** — Does the output avoid making claims beyond what the sources support?

The judge returns a single score from 0.0 to 1.0:

| Score | Meaning |
|-------|---------|
| 0.0 | Completely ungrounded (fabricated facts) |
| 0.3 | Poorly grounded (many unsupported claims) |
| 0.5 | Partially grounded (some claims supported) |
| 0.7 | Mostly grounded (minor gaps) |
| 1.0 | Fully grounded (every claim traceable) |

### Phase Behavior

- **`mid_execution`**: After each `record_grounding()` call, the judge evaluates the latest LLM response against the citations just recorded. A score below threshold produces an immediate BLOCK or WARN.
- **`after_workflow`**: After the agent completes, the judge evaluates the final output against all citations collected across all `record_grounding()` calls. A score below threshold adds to the violation warnings.
- **`both`**: Runs at both phases.

### Supported Providers

The judge model uses your organization's API keys stored in Tenant Secrets:

| Model prefix | Provider | Key needed |
|-------------|----------|------------|
| `gpt-*` | OpenAI | `OPENAI_API_KEY` |
| `claude-*` | Anthropic | `ANTHROPIC_API_KEY` |
| `llama-*`, `mixtral-*` | Groq | `GROQ_API_KEY` |

### Example: Policy with LLM Judge

```json
{
  "require_source_grounding": true,
  "min_grounding_score": 0.7,
  "min_citations": 2,
  "max_unsupported_claims": 0,
  "abstention_threshold": 0.5,
  "action_on_violation": "block",
  "llm_grounding_check": true,
  "llm_grounding_model": "gpt-4o-mini",
  "llm_grounding_threshold": 0.7,
  "llm_grounding_phase": "after_workflow"
}
```

### Graceful Degradation

The LLM judge never blocks your agents due to infrastructure failures:

- **No API key configured** — judge is skipped, threshold checks still run
- **LLM API error** — warning logged, judge skipped
- **Unparseable response** — defaults to 0.5 (neutral score)
- **`llm_grounding_check: false`** (default) — judge completely skipped, backward compatible

### Cost

The judge uses minimal tokens (~500 input + 10 output per evaluation):

| Model | Approximate cost per evaluation |
|-------|-------------------------------|
| `gpt-4o-mini` | ~$0.0001 |
| `llama-3.1-8b-instant` (Groq) | Near-zero |
| `claude-sonnet-4-20250514` | ~$0.002 |

### Accumulating Grounding Data

The SDK's `record_grounding()` method accumulates entries — each call appends to a list rather than overwriting. This supports multi-step workflows where different steps produce different grounding data:

```python
async with waxell.WaxellContext(...) as ctx:
    # Step 1: Retrieve financial sources
    ctx.record_grounding(
        grounding_scores=[0.92, 0.87],
        citations=["Federal Reserve Report", "IMF Analysis"],
        output_confidence=0.89,
    )
    # → mid_execution checks these scores

    # Step 2: Retrieve legal sources
    ctx.record_grounding(
        grounding_scores=[0.85, 0.91],
        citations=["SEC Filing 2023", "Legal Review"],
        output_confidence=0.88,
    )
    # → mid_execution checks these scores too

    # → after_workflow evaluates ALL citations from both steps
```

## Calibrating Thresholds

Grounding scores are computed by your pipeline — the grounding policy evaluates them but does not compute them. Common approaches:

| Approach | How to compute scores |
|----------|----------------------|
| **Embedding similarity** | Cosine similarity between each output sentence and its retrieved source chunk |
| **LLM self-evaluation** | Ask the LLM to score its own claim grounding (0-10 scale, normalize to 0.0-1.0) |
| **Cross-encoder reranker** | Use a cross-encoder model to score claim-source relevance |
| **NLI classifier** | Natural Language Inference model classifies each claim as entailed/neutral/contradicted |

Start with `action_on_violation: "warn"` and observe the distribution of scores across real executions. Set `min_grounding_score` at the 5th-percentile score from good runs — this blocks genuinely weak outputs without over-triggering on edge cases.

## Common Gotchas

1. **`mid_execution` fires per `record_grounding()` call, not per-claim.** If you report all scores in a single call and one is below threshold, the entire call is rejected. Split into separate calls only if you want per-claim mid-execution enforcement.

2. **`unsupported_claims` takes a list of strings.** Pass the actual claim text in the list. The handler compares `len(unsupported_claims) > max_unsupported_claims`. An empty list `[]` means zero unsupported claims.

3. **`max_unsupported_claims: null` (the default) disables the check.** Set an explicit integer (including `0`) to enable it. `0` means any unsupported claim blocks the output.

4. **`abstention_threshold` requires `output_confidence` to be set.** If you record grounding but omit `output_confidence`, the abstention threshold is skipped even if configured.

5. **`require_source_grounding` and `min_citations` can double-fire.** With `require_source_grounding=true` and `min_citations=1`, an empty citation list triggers both warnings. The result reason string concatenates them with `;`.

6. **Default `action_on_violation` is `"warn"`.** Unlike most handlers, grounding defaults to warn mode. Explicitly set `"block"` for production enforcement.

7. **`factual_consistency_check` is a no-op.** The rule is schema-valid and accepted, but the handler does not implement cross-claim consistency checking yet. It is reserved for a future release.

## Combining with Other Policies

- **retrieval**: Use `retrieval` to govern the quality of source documents retrieved; use `grounding` to govern whether the output is actually grounded in those sources. They complement each other in RAG pipelines.
- **content**: Use `content` to detect PII or prohibited content in outputs; use `grounding` to ensure outputs are factually accurate. Both protect the quality of agent outputs from different angles.
- **quality**: Use `quality` for general output quality metrics (length, format, tone); use `grounding` specifically for factual accuracy and citation requirements.
- **compliance**: In regulated industries (medical, legal, financial), combine `grounding` with `compliance` to ensure both factual accuracy and regulatory adherence.

## Next Steps

- [Policy & Governance](../features/governance) -- How policy enforcement works
- [Scope Policy](./scope) -- Govern blast radius of data operations
- [Retrieval Policy](../features/policy-categories) -- Govern the quality of retrieved sources
- [Quality Policy](./quality) -- General output quality governance
- [Policy Categories & Templates](../features/policy-categories) -- All 26 categories


--------------------------------------------------------------------------------
# Provenance Required Policy
URL: https://waxell.ai/docs/observe/governance/provenance-required
Description: OWASP LLM09b — strict per-claim citation enforcement. Every factual claim in agent output must carry a citation to a source on the tenant's approved list.
--------------------------------------------------------------------------------

# Provenance Required Policy

The `provenance-required` policy enforces **OWASP LLM09b** (misinformation / overreliance, sub-control "provenance"). It is the stricter cousin of the [Grounding Policy](./grounding): where grounding accepts a threshold like "groundedness score >= 0.6 is fine," this handler is **binary** — cite-or-fail.

Three independent checks:

1. **Unsupported-claim tolerance** — how many claims may go uncited before the policy fires.
2. **Minimum citation count** — total citations across the run must meet a floor.
3. **Source-type allowlist** — every citation must reference an approved source type (e.g., `knowledge_base`, `verified_corpus`, `internal_doc`).

Aligns with **NIST AI RMF MS-2.1** (Measure → Truthfulness).

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `require_citations_per_claim` | boolean | `true` | When true, unsupported claims are counted against `max_unsupported_claims` |
| `max_unsupported_claims` | integer | `0` | Tolerance — claims allowed without citation (0 = strict) |
| `min_citations` | integer | `1` | Minimum total citation count across the run |
| `allowed_source_types` | string[] | `[]` | Empty = any type accepted; non-empty = citations must reference one of these |
| `action_on_violation` | string | `"block"` | `"block"` or `"warn"` |

Optional: `scan_mid_execution` (boolean, default `false`) — enables defensive mid-run check on streaming surfaces.

## How It Works

| Phase | What it does |
|-------|--------------|
| `before_workflow` | Stashes rules on `context._provenance_rules`. No enforcement (claims don't exist yet). |
| `mid_execution` (opt-in) | Same evaluation as `after_workflow` — runs only if `scan_mid_execution=true`. |
| `after_workflow` | **Primary enforcement** — full claim/citation inventory check. |

Evaluation order (short-circuits on first violation):

1. **Unsupported-claim tolerance** — if `len(unsupported_claims) > max_unsupported_claims`.
2. **Minimum citation count** — if `len(citations) < min_citations`.
3. **Source-type allowlist** — first citation with disallowed source type.

### Context Attributes Read

| Attribute | Phase | Purpose |
|-----------|-------|---------|
| `context.citations` | mid, after | List of citation objects (str OR dict with `source_type`/`type`/`kind`/`source`) |
| `context.unsupported_claims` | mid, after | List of claim strings, OR an integer count |
| `context.grounding_entries` | mid, after | Optional per-step grounding records (read for richer metadata) |

## Example Policy

Regulated knowledge agent — every claim must cite an internal knowledge-base or verified corpus source, no exceptions:

```json
{
  "require_citations_per_claim": true,
  "max_unsupported_claims": 0,
  "min_citations": 1,
  "allowed_source_types": [
    "knowledge_base",
    "verified_corpus",
    "internal_doc"
  ],
  "action_on_violation": "block"
}
```

## SDK Integration

```python


waxell.init()

@waxell.observe(agent_name="compliance-assistant", enforce_policy=True)
async def answer(question: str) -> str:
    # Populate citations as you generate the response.
    async with waxell.WaxellContext(agent_name="compliance-assistant") as ctx:
        result = await retrieve_and_answer(question)
        ctx.set_citations([
            {"source_type": "knowledge_base", "id": "kb-1234", "title": "..."},
            {"source_type": "verified_corpus", "id": "vc-9876"},
        ])
        # after_workflow: blocks if no citations or wrong source type
        return result
```

## Observability

| Field | Example |
|-------|---------|
| **Category** | `provenance-required` |
| **Action** | `block` |
| **Reason** | "3 unsupported claim(s) detected; tolerance is 0." |
| **Metadata** | `{"phase": "after", "signal": "unsupported_claims", "count": 3, "limit": 0, "owasp": "LLM09"}` |

Disallowed source type:

| Field | Example |
|-------|---------|
| **Reason** | "Citation source type 'web_search' not in approved list ['knowledge_base', 'verified_corpus']." |
| **Metadata** | `{"signal": "disallowed_source_type", "source_type": "web_search", "owasp": "LLM09"}` |

## Common Gotchas

- **`unsupported_claims` can be either a list or an integer count.** The handler handles both. If your runtime emits a count, the metadata `count` field reflects that integer; if it emits a list, `count` is `len(list)`.
- **`min_citations=0` disables the citation-floor check.** Set to `1` (the default) to require at least one citation.
- **Source-type lookup tries four keys on dict citations.** In order: `source_type` → `type` → `kind` → `source`. Plain-string citations are treated as the source type itself.
- **Case-insensitive matching on source types.** `"Knowledge_Base"` and `"knowledge_base"` are equivalent.
- **Citations with empty/missing source type are skipped silently** during the allowlist check — they don't fire a violation. To enforce a present source type, use a separate validator.
- **Grounding vs Provenance.** Grounding scores claims for support strength; provenance demands a citation regardless of score. Often deployed together — grounding warns at low scores, provenance blocks at zero citations.
- **`before_workflow` is a no-op for enforcement.** It only stashes the rules; the actual check happens at `after_workflow` once claims exist.

## Next Steps

- [Policy Categories](../features/policy-categories) — All 49 categories
- [Grounding Policy](./grounding) — Citation thresholds + abstention (soft enforcement)
- [Retrieval Policy](./retrieval) — Controls over RAG sources
- [Model Card Required](./model-card-required) — Companion OWASP LLM03 supply-chain control


--------------------------------------------------------------------------------
# Retrieval Policy
URL: https://waxell.ai/docs/observe/governance/retrieval
Description: RAG pipeline quality governance — relevance scoring, source freshness, collection allowlists, and source diversity enforcement for vector search and retrieval-augmented generation.
--------------------------------------------------------------------------------

# Retrieval Policy

The `retrieval` policy category governs the quality of documents retrieved in RAG pipelines. It checks relevance scores, source freshness, allowed collections, and source diversity — blocking or warning before stale or off-topic context contaminates your agent's answers.

Use it when you need to enforce quality standards on vector search, document retrieval, or any step where your agent fetches external context before generating a response.

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `min_relevance_score` | number | `0.7` | Minimum cosine/dot-product similarity score required (0.0–1.0) |
| `max_source_age_days` | integer | `90` | Maximum age of a retrieved source document in days |
| `min_chunks` | integer | `1` | Minimum number of results required |
| `max_chunks` | integer | `10` | Maximum number of results allowed |
| `allowed_collections` | string[] | `[]` | If non-empty, only these vector collections may be queried |
| `blocked_sources` | string[] | `[]` | Source identifiers that are always blocked regardless of score |
| `require_source_diversity` | boolean | `false` | Enforce that no single source dominates the result set |
| `max_single_source_ratio` | number | `0.6` | Maximum fraction of results from a single source (when diversity is required) |
| `action_on_low_relevance` | string | `"warn"` | What to do when relevance is below threshold: `"warn"` or `"block"` |
| `action_on_stale_source` | string | `"block"` | What to do when a source exceeds max age: `"warn"` or `"block"` |
| `action_on_chunk_violation` | string | `"warn"` | What to do when chunk count exceeds `min_chunks`/`max_chunks`: `"warn"` or `"block"` |

## How It Works

The retrieval handler runs at two phases:

- **mid_execution** — checks each retrieved document as it is recorded. Fires after every `ctx.record_retrieval_result()` call.
- **after_workflow** — checks the full result set for source diversity violations after the agent finishes.

### Evaluation Order (mid_execution)

1. Check chunk count: too many &#123;max_chunks&#125; or too few &#123;min_chunks&#125;
2. For each result in `retrieval_results`:
   - Relevance score below `min_relevance_score` → `action_on_low_relevance`
   - Source in `blocked_sources` → always BLOCK
   - Collection not in `allowed_collections` (if list is non-empty) → always BLOCK
   - Source age exceeds `max_source_age_days` → `action_on_stale_source`

### Evaluation Order (after_workflow)

1. If `require_source_diversity` is true, count how many chunks come from each source
2. If any single source accounts for more than `max_single_source_ratio` of results → WARN

### Rule Matching Reference

| Check | Condition | Result |
|-------|-----------|--------|
| Relevance 0.85, threshold 0.70 | 0.85 ≥ 0.70 | Pass |
| Relevance 0.60, threshold 0.70 | 0.60 &#60; 0.70 | `action_on_low_relevance` |
| Source age 45d, max 90d | 45 ≤ 90 | Pass |
| Source age 200d, max 90d | 200 &#62; 90 | `action_on_stale_source` |
| Collection "knowledge_base", allowed ["knowledge_base"] | In list | Pass |
| Collection "internal-hr", allowed ["knowledge_base"] | Not in list | BLOCK |
| Source "deprecated-kb.pdf", blocked list includes it | Exact match | BLOCK |
| 3 of 4 chunks from same source, max ratio 0.6 | 0.75 &#62; 0.6 | WARN (after_workflow) |

**Chunk Count Action Is Configurable**
By default, `min_chunks` and `max_chunks` violations produce WARN. Set `action_on_chunk_violation: "block"` to hard-stop agents that retrieve too many or too few documents. This is useful in regulated environments where retrieval pipeline health must be enforced.


**allowed_collections Is Checked Per Document**
If `allowed_collections` is `["knowledge_base"]` and you retrieve from both `"knowledge_base"` and `"internal-hr"`, the first result from `"knowledge_base"` passes, but the result from `"internal-hr"` triggers a BLOCK. The check fires per document, not per query.


## Example Policies

### Standard RAG Quality Gate

Enforce minimum relevance and freshness for a customer-facing knowledge base:

```json
{
  "min_relevance_score": 0.75,
  "max_source_age_days": 90,
  "min_chunks": 1,
  "max_chunks": 8,
  "allowed_collections": ["knowledge_base", "product-docs"],
  "action_on_low_relevance": "warn",
  "action_on_stale_source": "block"
}
```

### Strict Compliance Retrieval

Block on any quality violation — suitable for regulated environments where stale or low-quality sources cannot be used:

```json
{
  "min_relevance_score": 0.80,
  "max_source_age_days": 30,
  "min_chunks": 2,
  "max_chunks": 5,
  "allowed_collections": ["compliance-docs"],
  "blocked_sources": ["deprecated-policy-archive"],
  "require_source_diversity": true,
  "max_single_source_ratio": 0.5,
  "action_on_low_relevance": "block",
  "action_on_stale_source": "block",
  "action_on_chunk_violation": "block"
}
```

### Lenient Monitoring

Warn on all quality issues without blocking — useful for observing retrieval quality before enforcing stricter rules:

```json
{
  "min_relevance_score": 0.5,
  "max_source_age_days": 365,
  "action_on_low_relevance": "warn",
  "action_on_stale_source": "warn"
}
```

## SDK Integration

### Recording Retrieval Results

Call `ctx.record_retrieval_result()` once per retrieved document. Each call automatically sends data to the controlplane and triggers mid_execution governance immediately.

```python

from waxell_observe.errors import PolicyViolationError

async with waxell.WaxellContext(
    agent_name="rag-agent",
    enforce_policy=True,
) as ctx:
    # Record each retrieved document
    ctx.record_retrieval_result(
        relevance_score=0.92,
        source="product-manual.pdf",
        collection="knowledge_base",
        age_days=14,
    )
    ctx.record_retrieval_result(
        relevance_score=0.85,
        source="faq.pdf",
        collection="knowledge_base",
        age_days=30,
    )
    # Governance fires here — low relevance or stale sources raise PolicyViolationError

    answer = synthesize_answer(context_docs, query)
    ctx.set_result({"answer": answer})
```

### Catching Policy Violations

```python
try:
    async with waxell.WaxellContext(
        agent_name="rag-agent",
        enforce_policy=True,
    ) as ctx:
        for doc in retrieved_docs:
            ctx.record_retrieval_result(
                relevance_score=doc.score,
                source=doc.id,
                collection=doc.collection,
                age_days=doc.age_days,
            )
        answer = synthesize(docs, query)
        ctx.set_result({"answer": answer})

except PolicyViolationError as e:
    # e.g. "Retrieved from blocked source 'deprecated-kb'"
    # or   "Source age (400 days) exceeds max (90 days)"
    # or   "Collection 'internal-hr' not in allowed list"
    return fallback_response(query)
```

### Using the Decorator

```python
@waxell.observe(
    agent_name="rag-agent",
    enforce_policy=True,
)
async def run_rag(query: str, docs: list[dict]):
    ctx = waxell.get_current_context()
    for doc in docs:
        ctx.record_retrieval_result(
            relevance_score=doc["score"],
            source=doc["source"],
            collection=doc["collection"],
            age_days=doc["age_days"],
        )
    return synthesize(docs, query)
```

## Enforcement Flow

```
Agent runs vector search, retrieves N documents
    │
    ├── before_workflow governance runs
    │   └── Retrieval rules stored in context._retrieval_rules
    │
    ├── For each document retrieved:
    │   └── ctx.record_retrieval_result(relevance_score, source, collection, age_days)
    │       │
    │       └── mid_execution governance fires
    │           ├── chunk count within [min_chunks, max_chunks]?
    │           │   └── No: action_on_chunk_violation (warn or block)
    │           ├── relevance_score >= min_relevance_score?
    │           │   └── No: action_on_low_relevance (warn or block)
    │           ├── source in blocked_sources?
    │           │   └── Yes: BLOCK
    │           ├── collection in allowed_collections?
    │           │   └── No: BLOCK
    │           └── age_days <= max_source_age_days?
    │               └── No: action_on_stale_source (warn or block)
    │
    ├── Agent synthesizes answer from retrieved context
    │
    └── after_workflow governance fires
        └── require_source_diversity?
            └── Yes: check max_single_source_ratio across all sources
                └── Any source dominates? → WARN
```

## Creating via Dashboard

1. Navigate to **Governance > Policies**
2. Click **New Policy**
3. Select category **Retrieval**
4. Configure relevance threshold, max age, and collection allowlist
5. Set scope to target specific agents (e.g., `rag-agent`)
6. Enable

## Creating via API

```bash
curl -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  https://acme.waxell.dev/waxell/v1/policies/ \
  -d '{
    "name": "RAG Quality Guard",
    "category": "retrieval",
    "rules": {
      "min_relevance_score": 0.75,
      "max_source_age_days": 90,
      "min_chunks": 1,
      "max_chunks": 8,
      "allowed_collections": ["knowledge_base"],
      "action_on_low_relevance": "warn",
      "action_on_stale_source": "block"
    },
    "scope": {
      "agents": ["rag-agent"]
    },
    "enabled": true
  }'
```

## Observability

### Governance Tab

Retrieval evaluations appear with:

| Field | Example |
|-------|---------|
| **Policy name** | RAG Quality Guard |
| **Phase** | `mid_execution` or `after_workflow` |
| **Action** | `allow`, `warn`, or `block` |
| **Category** | `retrieval` |
| **Reason** | "Retrieval quality within policy (3 chunks)" |
| **Metadata** | `{"chunk_count": 3}` |

For violations:

| Violation | Reason | Metadata |
|-----------|--------|---------|
| Low relevance | "Retrieval relevance (0.41) below threshold (0.70)" | `{"relevance_score": 0.41, "threshold": 0.70}` |
| Stale source | "Source age (400 days) exceeds max (90 days)" | `{"age_days": 400, "max_age": 90}` |
| Blocked collection | "Collection 'internal-hr' not in allowed list" | `{"collection": "internal-hr", "allowed": ["knowledge_base"]}` |
| Blocked source | "Retrieved from blocked source 'deprecated-kb'" | `{"blocked_source": "deprecated-kb"}` |
| Low chunk count | "Retrieved chunks (0) below minimum (1)" | `{"chunk_count": 0, "limit": 1}` |
| Source domination | "Source 'doc.pdf' dominates at 75% (max 60%)" | `{"warnings": ["..."]}` |

### Trace Tab

Each `ctx.record_retrieval_result()` call produces a span under the parent `vector_search` tool span. You can inspect per-document metadata including relevance score, source, collection, and age.

## Combining with Other Policies

The retrieval policy is commonly paired with:

- **Content policy** — scan the synthesized answer for PII or harmful content after retrieval
- **Grounding policy** — verify that the final answer stays grounded in the retrieved documents
- **Quality policy** — score answer quality (coherence, factuality) after synthesis
- **Compliance policy** — require all three of the above for regulated use cases

## Common Gotchas

1. **Empty `allowed_collections` means allow all.** A non-configured allowlist is not an implicit block-all. Set at least one collection name to restrict access.

2. **`blocked_sources` uses exact match.** The source string must exactly match the value passed to `record_retrieval_result()`. Use consistent naming conventions across your retrieval pipeline.

3. **mid_execution fires per document, stops at first violation.** If your first retrieved document fails the collection check, the second document is never evaluated. You will only see one governance event per run.

4. **Source diversity is after_workflow only.** The `require_source_diversity` check runs after the agent completes, not during retrieval. It will not stop the agent mid-execution — it produces a warning in the governance audit.

5. **age_days must be computed by your application.** The SDK does not calculate document age automatically. You must compute `(today - document_created_date).days` and pass it to `record_retrieval_result()`.

6. **`min_chunks` and `max_chunks` default to WARN but can BLOCK.** Set `action_on_chunk_violation: "block"` to enforce chunk count limits as hard stops. The default `"warn"` behavior records a governance event without stopping the agent.

7. **`action_on_low_relevance: "warn"` still produces a governance event.** The agent continues running but the warning is recorded in the trace and governance tab. Use this for monitoring before enforcing stricter rules.

## Next Steps

- [Policy & Governance](../features/governance) — How policy enforcement works
- [Grounding Policy](./grounding) — Verify answers stay grounded in retrieved context
- [Content Policy](./content) — Scan synthesized answers for harmful content
- [Quality Policy](./quality) — Score answer quality after synthesis
- [Policy Categories & Templates](../features/policy-categories) — All 26 categories


--------------------------------------------------------------------------------
# Reasoning Policy
URL: https://waxell.ai/docs/observe/governance/reasoning
Description: Decision explainability and bias detection governance — enforce explanation requirements, alternative consideration, confidence thresholds, and bias detection for agent decision-making.
--------------------------------------------------------------------------------

# Reasoning Policy

The `reasoning` policy category governs the quality and fairness of agent decisions. It enforces that decisions are explained, consider multiple alternatives, meet confidence thresholds, and are free from bias in protected attributes.

Use it when you need auditability for agent decision-making — routing, classification, prioritization, or any step where your agent makes a choice that affects users or downstream systems.

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `require_explanation` | boolean | `false` | Require a non-empty explanation for every recorded decision |
| `explanation_min_length` | integer | `50` | Minimum character length for decision explanations |
| `require_alternatives_considered` | boolean | `false` | Require that multiple options were evaluated |
| `min_alternatives` | integer | `2` | Minimum number of alternatives that must be listed |
| `confidence_required` | boolean | `false` | Require a confidence score on every decision |
| `min_decision_confidence` | number | `0.7` | Minimum acceptable confidence (0.0–1.0) |
| `bias_detection.enabled` | boolean | `false` | Enable bias detection on reasoning outputs |
| `bias_detection.protected_attributes` | string[] | `[]` | Attributes to monitor (e.g. `["gender", "race", "age"]`) |
| `bias_detection.action` | string | `"warn"` | Action on bias detected: `"warn"` or `"block"` |
| `decision_audit_trail` | boolean | `false` | Warn if no decisions were recorded during the workflow |
| `max_reasoning_depth` | integer | `10` | Maximum reasoning chain depth |
| `action_on_violation` | string | `"warn"` | Default action for explanation/alternatives/confidence violations: `"warn"` or `"block"` |

## How It Works

The reasoning handler runs at two phases:

- **mid_execution** — checks recorded decisions. Governance evaluates buffered decision data at the next auto-instrumented event (LLM call, step) or at workflow completion.
- **after_workflow** — checks the full session for bias flags and audit trail completeness after the agent finishes.

### Evaluation Order (mid_execution)

1. Check `max_reasoning_depth` against `context.reasoning_depth`
2. For each decision in `context.decisions`:
   - `require_explanation`: explanation must exist and be at least `explanation_min_length` chars
   - `require_alternatives_considered`: `options` list must have at least `min_alternatives` entries
   - `confidence_required`: confidence must be present and >= `min_decision_confidence`
3. Any violation → `action_on_violation` (warn or block)

### Evaluation Order (after_workflow)

1. `bias_detection.enabled`: if any `bias_flags` are recorded → `bias_detection.action`
2. `decision_audit_trail`: if enabled and no decisions recorded → WARN

### What Gets Checked

The reasoning handler reads `context.decisions` — a list populated by `ctx.record_decision()`. Each entry looks like:

```python
{
    "decision": "route_query",        # decision name
    "explanation": "Query is factual...",  # maps from reasoning parameter
    "alternatives": ["search", "generate", "clarify"],  # maps from options
    "confidence": 0.9,
}
```

Note: `record_decision(reasoning=...)` populates `explanation`, and `record_decision(options=...)` populates `alternatives`. The parameter names differ from the internal field names checked by the handler.

### Rule Reference

| Check | Condition | Result |
|-------|-----------|--------|
| explanation "", min_length 50 | 0 &#60; 50 | `action_on_violation` |
| explanation "Query is factual, high confidence" (35 chars), min_length 50 | 35 &#60; 50 | `action_on_violation` |
| explanation 80 chars, min_length 50 | 80 ≥ 50 | Pass |
| 1 alternative, min_alternatives 2 | 1 &#60; 2 | `action_on_violation` |
| 3 alternatives, min_alternatives 2 | 3 ≥ 2 | Pass |
| confidence 0.45, min 0.70 | 0.45 &#60; 0.70 | `action_on_violation` |
| confidence 0.85, min 0.70 | 0.85 ≥ 0.70 | Pass |
| bias_flag "gender_bias", bias_detection.action "block" | Flag present | BLOCK at after_workflow |

**Bias Detection Runs at after_workflow, Not mid_execution**
`record_bias_flag()` records flags during execution, but the bias check runs after the agent completes. This means bias detection never stops the agent mid-stream — it produces a block or warning in the governance audit after the workflow finishes.


**action_on_violation Applies to Explanation, Alternatives, and Confidence**
The single `action_on_violation` rule controls all three mid_execution checks. You cannot set different actions for each violation type. Use `bias_detection.action` separately for bias control.


## Example Policies

### Minimal Explainability

Require explanations for all decisions without blocking:

```json
{
  "require_explanation": true,
  "explanation_min_length": 30,
  "action_on_violation": "warn"
}
```

### Full Decision Audit

Enforce explanation, alternatives, and confidence for regulated decision-making:

```json
{
  "require_explanation": true,
  "explanation_min_length": 50,
  "require_alternatives_considered": true,
  "min_alternatives": 2,
  "confidence_required": true,
  "min_decision_confidence": 0.75,
  "decision_audit_trail": true,
  "action_on_violation": "block"
}
```

### Bias Detection with Block

Block any workflow where a protected attribute bias was flagged:

```json
{
  "bias_detection": {
    "enabled": true,
    "protected_attributes": ["gender", "race", "age", "religion"],
    "action": "block"
  },
  "require_explanation": true,
  "explanation_min_length": 50,
  "action_on_violation": "warn"
}
```

### Monitoring Mode

Record everything without blocking — useful during initial rollout:

```json
{
  "require_explanation": true,
  "explanation_min_length": 20,
  "require_alternatives_considered": true,
  "min_alternatives": 2,
  "confidence_required": true,
  "min_decision_confidence": 0.6,
  "bias_detection": {
    "enabled": true,
    "protected_attributes": ["gender", "race"],
    "action": "warn"
  },
  "decision_audit_trail": true,
  "action_on_violation": "warn"
}
```

## SDK Integration

### Recording Decisions

Call `ctx.record_decision()` for each decision point in your agent. The reasoning handler reads `context.decisions` — a list populated by these calls.

```python

from waxell_observe.errors import PolicyViolationError

async with waxell.WaxellContext(
    agent_name="routing-agent",
    enforce_policy=True,
) as ctx:
    # Record a routing decision
    ctx.record_decision(
        name="route_query",
        options=["search", "generate", "clarify"],
        chosen="search",
        reasoning="Query is factual and requires retrieval from knowledge base. Direct generation would risk hallucination.",
        confidence=0.9,
    )

    # Governance evaluates the decision at the next auto-instrumented
    # event (e.g., the LLM call below) or at workflow completion
    result = await execute_search(query)
    ctx.set_result(result)
```

### Recording Reasoning Steps

Use `ctx.record_reasoning()` to capture chain-of-thought steps. These appear in the trace but do not trigger mid_execution governance — they are supporting evidence for the decisions.

```python
async with waxell.WaxellContext(
    agent_name="routing-agent",
    enforce_policy=True,
) as ctx:
    # Record chain-of-thought first
    ctx.record_reasoning(
        step="evaluate",
        thought="The evidence supports a factual retrieval approach over direct generation.",
        evidence=["query_type: factual", "knowledge_base: available", "confidence: high"],
        conclusion="Proceed with search",
    )

    # Then record the decision
    ctx.record_decision(
        name="route_query",
        options=["search", "generate", "clarify"],
        chosen="search",
        reasoning="Query is factual and requires retrieval from knowledge base.",
        confidence=0.9,
    )

    # Governance evaluates the decision at the next auto-instrumented event
    # or at workflow completion
```

### Recording Bias Flags

Call `ctx.record_bias_flag()` when your application detects a bias signal. The reasoning handler checks `context.bias_flags` at `after_workflow`.

```python
async with waxell.WaxellContext(
    agent_name="hiring-agent",
    enforce_policy=True,
) as ctx:
    result = await evaluate_candidate(application)

    # Your bias detection logic
    if detected_gender_reference(result):
        ctx.record_bias_flag(flag="gender_bias")

    # Record the decision separately
    ctx.record_decision(
        name="candidate_evaluation",
        options=["advance", "reject", "hold"],
        chosen=result.recommendation,
        reasoning=result.explanation,
        confidence=result.confidence,
    )

    ctx.set_result(result)
    # after_workflow governance fires on __aexit__
    # If bias_flags is non-empty and bias_detection.action is "block" → PolicyViolationError
```

### Catching Policy Violations

```python
try:
    async with waxell.WaxellContext(
        agent_name="routing-agent",
        enforce_policy=True,
    ) as ctx:
        ctx.record_decision(
            name="route_query",
            options=["search"],             # Only 1 option — violates min_alternatives: 2
            chosen="search",
            reasoning="",                   # Empty — violates require_explanation
            confidence=0.45,               # Below min_decision_confidence: 0.7
        )
        result = await do_work()
        ctx.set_result(result)

except PolicyViolationError as e:
    # e.g. "Decision explanation too short (0/50 chars)"
    # or   "Alternatives considered (1) below minimum (2)"
    # or   "Decision confidence (0.45) below threshold (0.70)"
    # or   "Bias detected: gender_bias"
    return fallback_response()
```

## Enforcement Flow

```
Agent starts (WaxellContext.__aenter__)
    │
    └── before_workflow governance runs
        └── Reasoning rules stored in context._reasoning_rules

Agent makes a decision
    │
    ├── ctx.record_decision(name, options, chosen, reasoning, confidence)
    │   └── Data buffered for governance evaluation
    │
    └── Next auto-instrumented event (LLM call, step) or workflow completion
        │
        └── mid_execution governance fires
            ├── reasoning_depth > max_reasoning_depth? → action_on_violation
            ├── require_explanation?
            │   └── explanation length < explanation_min_length? → action_on_violation
            ├── require_alternatives_considered?
            │   └── len(options) < min_alternatives? → action_on_violation
            └── confidence_required?
                └── confidence < min_decision_confidence? → action_on_violation

Agent may call ctx.record_bias_flag(flag=...) if bias detected

Agent finishes (WaxellContext.__aexit__)
    │
    └── after_workflow governance fires
        ├── bias_detection.enabled?
        │   └── bias_flags non-empty? → bias_detection.action (warn or block)
        └── decision_audit_trail?
            └── no decisions recorded? → WARN
```

## Creating via Dashboard

1. Navigate to **Governance > Policies**
2. Click **New Policy**
3. Select category **Reasoning**
4. Enable `require_explanation` and set `explanation_min_length`
5. Enable `require_alternatives_considered` and set `min_alternatives`
6. Enable `confidence_required` and set `min_decision_confidence`
7. Configure `bias_detection` if needed
8. Set scope to target specific agents (e.g., `routing-agent`)
9. Enable

## Creating via API

```bash
curl -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  https://acme.waxell.dev/waxell/v1/policies/ \
  -d '{
    "name": "Decision Explainability",
    "category": "reasoning",
    "rules": {
      "require_explanation": true,
      "explanation_min_length": 50,
      "require_alternatives_considered": true,
      "min_alternatives": 2,
      "confidence_required": true,
      "min_decision_confidence": 0.75,
      "bias_detection": {
        "enabled": true,
        "protected_attributes": ["gender", "race", "age"],
        "action": "warn"
      },
      "decision_audit_trail": true,
      "action_on_violation": "warn"
    },
    "scope": {
      "agents": ["routing-agent"]
    },
    "enabled": true
  }'
```

## Observability

### Governance Tab

Reasoning evaluations appear with:

| Field | Example |
|-------|---------|
| **Policy name** | Decision Explainability |
| **Phase** | `mid_execution` or `after_workflow` |
| **Action** | `allow`, `warn`, or `block` |
| **Category** | `reasoning` |
| **Reason** | "Reasoning within policy (2 decisions)" |
| **Metadata** | `{"decision_count": 2}` |

For violations:

| Violation | Reason | Metadata |
|-----------|--------|---------|
| Short explanation | "Decision explanation too short (0/50 chars)" | `{"explanation_length": 0, "min_length": 50}` |
| Too few alternatives | "Alternatives considered (1) below minimum (2)" | `{"alternatives_count": 1, "min_required": 2}` |
| Low confidence | "Decision confidence (0.45) below threshold (0.70)" | `{"confidence": 0.45, "threshold": 0.70}` |
| Bias detected | "Bias detected: gender_bias" | `{"bias_flags": ["gender_bias"], "protected_attributes": [...]}` |
| No audit trail | "Decision audit trail enabled but no decisions recorded" | `{"warnings": [...]}` |

### Trace Tab

Each `ctx.record_decision()` call produces a `decision` span with:
- `input_data`: `{"options": [...], "reasoning": "..."}`
- `output_data`: `{"chosen": "...", "confidence": 0.9}`

Each `ctx.record_reasoning()` call produces a `reasoning` span with the thought, evidence, and conclusion.

## Combining with Other Policies

The reasoning policy is commonly paired with:

- **Audit policy** — log all decisions and reasoning steps for compliance records
- **Content policy** — scan the final response for harmful content after decisions are made
- **Retrieval policy** — ensure retrieved evidence meets quality standards before it informs decisions
- **Compliance policy** — require reasoning + audit + content policies together for regulated workflows

## Common Gotchas

1. **`record_decision(options=...)` maps to `alternatives` internally.** The handler checks `decision["alternatives"]`, which is populated from the `options` parameter you pass to `record_decision()`. Passing a single-item `options` list will fail the `min_alternatives` check even if your agent considered other paths.

2. **`record_decision(reasoning=...)` maps to `explanation` internally.** The handler checks `decision["explanation"]`, which comes from the `reasoning` parameter. An empty `reasoning=""` is an empty explanation.

3. **Bias detection is after_workflow only.** `record_bias_flag()` records the flag immediately, but the governance check runs when the agent finishes. A bias block raises `PolicyViolationError` from `WaxellContext.__aexit__`, not from the `record_bias_flag()` call itself.

4. **`action_on_violation` applies to all three mid_execution checks.** You cannot block on missing explanation but warn on low confidence — it is a single setting for all three. Use `bias_detection.action` separately for bias control.

5. **`decision_audit_trail` is a soft check.** When enabled and no decisions were recorded, it produces a WARN rather than a block. It is designed to catch instrumentation gaps, not to enforce decision recording.

6. **`max_reasoning_depth` checks `context.reasoning_depth`, not the number of `record_reasoning()` calls.** The depth counter is incremented by the SDK's internal delegation depth tracking, not by manual reasoning steps.

7. **Confidence is optional per decision.** If `confidence_required: true` but a specific decision has `confidence=None`, the confidence check is skipped for that decision. Only non-None confidence values below the threshold trigger a violation.

8. **`record_decision()` buffers data for later evaluation.** Decision governance fires at the next auto-instrumented event (LLM call, tool call, step recording) or at workflow completion. If your agent makes a decision and then immediately makes an LLM call, governance evaluates the decision data at that point.

9. **`record_bias_flag()` flushes immediately.** Unlike `record_decision()`, calling `record_bias_flag()` sends data to the controlplane right away. However, bias detection is checked at `after_workflow`, not mid_execution, so the timing rarely matters in practice.

## Next Steps

- [Policy & Governance](../features/governance) — How policy enforcement works
- [Retrieval Policy](./retrieval) — Govern RAG pipeline quality
- [Content Policy](./content) — Scan outputs for harmful content
- [Policy Categories & Templates](../features/policy-categories) — All 26 categories


--------------------------------------------------------------------------------
# Recursion Bound Policy
URL: https://waxell.ai/docs/observe/governance/recursion-bound
Description: OWASP LLM10c — cap reasoning depth, tool-call count, delegation depth, and user-turn count to prevent runaway agent loops.
--------------------------------------------------------------------------------

# Recursion Bound Policy

The `recursion-bound` policy enforces **OWASP LLM10c** (unbounded consumption, sub-control "runaway recursion"). It caps four counters that together bound the work an agent can do in a single run:

- **`reasoning_depth`** — number of reasoning steps in the chain
- **`tool_call_count`** — number of tool calls so far in the workflow
- **`delegation_depth`** — how deep the agent-spawn tree goes
- **`conversation_user_turns`** — total user turns in the conversation

Pure threshold checks — no I/O, sub-microsecond per evaluation. Tier 1 (cheap).

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `max_reasoning_depth` | integer | `8` | Maximum reasoning steps before BLOCK |
| `max_tool_calls` | integer | `25` | Maximum tool invocations |
| `max_delegation_depth` | integer | `3` | Maximum agent-spawn nesting depth |
| `max_user_turns` | integer | `50` | Maximum user turns in the conversation |
| `action_on_violation` | string | `"block"` | `"block"` or `"warn"` |

Any rule set to `null` (or omitted from `rules`) is skipped — the counter is not checked.

## How It Works

The handler runs at **all three phases** with the same evaluation logic. Comparison is `current >= limit` (inclusive), so `max_tool_calls: 25` blocks on the 25th call.

| Phase | Behavior |
|-------|----------|
| `before_workflow` | Catches resumed workflows that already exceed a budget |
| `mid_execution` | **Primary enforcement** — fires between steps the moment any counter crosses the threshold |
| `after_workflow` | Audit pass — flags runs that approached or exceeded the ceiling |

### Context Attributes Read

| Attribute | Counter | Source |
|-----------|---------|--------|
| `context.reasoning_depth` | reasoning depth | Reasoning steps in the chain |
| `context.tool_call_count` | tool calls | Running count of tool invocations |
| `context.delegation_depth` | delegation depth | Agent-spawn nesting |
| `context.conversation_user_turns` | user turns | Conversation length |

All four are read with `int(getattr(context, name, 0) or 0)` — missing attributes default to 0.

## Example Policy

Customer-facing agent — conservative caps to keep costs predictable and prevent any single conversation from running away:

```json
{
  "max_reasoning_depth": 6,
  "max_tool_calls": 15,
  "max_delegation_depth": 2,
  "max_user_turns": 30,
  "action_on_violation": "block"
}
```

Long-running research agent — more headroom for tool calls but strict delegation depth:

```json
{
  "max_reasoning_depth": 20,
  "max_tool_calls": 100,
  "max_delegation_depth": 1,
  "action_on_violation": "warn"
}
```

## SDK Integration

```python


waxell.init()

@waxell.observe(agent_name="research-agent", enforce_policy=True)
async def research(query: str) -> str:
    # As the agent makes tool calls and reasons,
    # context.tool_call_count and context.reasoning_depth grow.
    # mid_execution blocks the moment any counter hits its limit.
    return await deep_research(query)
```

## Observability

| Field | Example |
|-------|---------|
| **Category** | `recursion-bound` |
| **Action** | `block` |
| **Reason** | "Recursion bound exceeded: tool_call_count=25 >= limit 25" |
| **Metadata** | `{"phase": "mid", "signal": "recursion_bound", "counter": "tool_call_count", "current": 25, "limit": 25, "owasp": "LLM10"}` |

Delegation depth violation:

| Field | Example |
|-------|---------|
| **Reason** | "Recursion bound exceeded: delegation_depth=4 >= limit 3" |
| **Metadata** | `{"counter": "delegation_depth", "current": 4, "limit": 3}` |

## Common Gotchas

- **Comparison is `>=`, not `>`.** `max_tool_calls: 25` means the 25th tool call triggers the block — the agent can make at most 24 calls. Set the limit to `26` if you want to permit 25.
- **Omitting a rule disables that specific check.** A policy with only `max_tool_calls` set leaves reasoning depth, delegation depth, and user-turn count unbounded.
- **Counters are read from the context, not maintained by the handler.** If your runtime doesn't increment `tool_call_count`, the check effectively no-ops.
- **`max_delegation_depth: 0` means no delegation allowed.** The check fires when `delegation_depth >= 0`, which is always true on a delegating agent. To allow exactly one level of delegation, set `max_delegation_depth: 2`.
- **All four counters fire the same violation `signal: "recursion_bound"`.** Use the `counter` metadata field to distinguish which limit was hit.
- **`before_workflow` rarely fires** unless you're resuming a long-running session with pre-populated counters.
- **Overlap with [Safety Policy](./safety) `max_tool_calls`.** Both check the same counter. The safety policy is the broader cap; recursion-bound is the OWASP-aligned variant. Pick one or set safety's higher than recursion-bound's.

## Next Steps

- [Policy Categories](../features/policy-categories) — All 49 categories
- [Safety Policy](./safety) — Broader execution limits + content filters
- [Delegation Policy](./delegation) — Multi-agent trust and delegation rules
- [Rate Limit Policy](./rate-limit) — Cross-run rate / concurrency control


--------------------------------------------------------------------------------
# Prompt Injection Guard Policy
URL: https://waxell.ai/docs/observe/governance/prompt-injection-guard
Description: OWASP LLM01 — scan agent inputs and indirect retrieval/tool outputs for prompt-injection signals using heuristic patterns and an optional ML classifier.
--------------------------------------------------------------------------------

# Prompt Injection Guard Policy

The `prompt-injection-guard` policy enforces **OWASP LLM01 Prompt Injection** and aligns with **NIST AI RMF MAP-2.3**. It scans agent inputs before the LLM ever sees them, plus retrieval results / tool outputs mid-run (indirect injection). Two detection modes:

- **Heuristic** (default, cheap) — substring matches against a curated list of injection phrases, plus structural signals (base64 blobs, excessive caps, excessive punctuation).
- **Classifier** (opt-in, mid-cost) — controlplane wires an ML classifier callable that returns `(confidence, label)`. When configured, classifier and heuristic decisions are ORed.

Pure CPU, sub-millisecond per input in heuristic mode.

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `detection_mode` | string | `"heuristic"` | `"heuristic"`, `"classifier"`, or `"heuristic_plus_classifier"` |
| `min_confidence` | number | `0.7` | Classifier threshold (0–1) |
| `blocked_patterns` | string[] | *(curated default)* | Replace or extend the built-in pattern list (case-insensitive substring match) |
| `max_payload_kb` | number | `64` | Reject oversized inputs (possible obfuscation) |
| `action_on_violation` | string | `"block"` | `"block"` or `"warn"` |

Optional: `scan_indirect` (boolean, default `true`) — toggles mid-execution scanning of `retrieval_results`.

### Default Heuristic Patterns

The handler ships with a curated list including: `ignore previous instructions`, `disregard the above`, `forget previous instructions`, `new instructions:`, `system:`, `system prompt:`, `you are now`, `act as a`, `pretend to be`, `developer mode`, `dan mode`, `jailbreak`, `what were your instructions`, `repeat your system prompt`, `### instructions`, `user has authorized`, `the assistant should`.

Structural signals (always on): base64 blob ≥ 200 chars, 15+ consecutive uppercase letters, 9+ repeated `!`/`?`/`.`.

## How It Works

| Phase | What it checks | Source |
|-------|----------------|--------|
| `before_workflow` | Heuristic + (optional) classifier on inputs | `context.inputs` |
| `mid_execution` | Heuristic + (optional) classifier on retrieval results (indirect injection) | `context.retrieval_results` |
| `after_workflow` | (no-op — injection is a pre-execution concern) | — |

Scan order per phase:

1. **Size cap** — if payload > `max_payload_kb` → BLOCK with `oversized` reason.
2. **Heuristic scan** — phrase list, then base64/caps/punctuation regex.
3. **Classifier scan** (if mode includes it) — `(confidence, label)` from `_classifier_fn`.

### Context Attributes Read

| Attribute | Phase | Purpose |
|-----------|-------|---------|
| `context.inputs` | before | Primary scan target (dict/list/str — strings extracted recursively) |
| `context.retrieval_results` | mid | Indirect-injection scan target |

## Example Policy

Production policy — heuristic + classifier with strict size cap and a custom pattern list:

```json
{
  "detection_mode": "heuristic_plus_classifier",
  "min_confidence": 0.8,
  "blocked_patterns": [
    "ignore previous instructions",
    "ignore the above",
    "disregard previous",
    "system prompt:",
    "you are now a",
    "developer mode enabled",
    "jailbreak",
    "###instructions",
    "user has authorized"
  ],
  "max_payload_kb": 32,
  "action_on_violation": "block"
}
```

## SDK Integration

```python


waxell.init()

@waxell.observe(agent_name="public-chatbot", enforce_policy=True)
async def reply(message: str) -> str:
    # before_workflow: scans `message` (and any other input)
    # for injection phrases / oversized payloads.
    # If a hit fires, PolicyViolationError is raised before the LLM call.
    return await generate(message)
```

## Observability

| Field | Example |
|-------|---------|
| **Category** | `prompt-injection-guard` |
| **Action** | `block` |
| **Reason** | "Prompt-injection signal detected (phrase): 'ignore previous instructions'" |
| **Metadata** | `{"phase": "before", "signal": "phrase", "matched_pattern": "ignore previous instructions", "owasp": "LLM01"}` |

Indirect-injection block (from a tool output):

| Field | Example |
|-------|---------|
| **Reason** | "Prompt-injection signal detected (phrase): 'system prompt:'" |
| **Metadata** | `{"phase": "mid", "signal": "phrase", "matched_pattern": "system prompt:", "owasp": "LLM01"}` |

## Common Gotchas

- **Heuristic patterns are case-insensitive substrings.** `"system:"` matches both `"SYSTEM:"` and innocent prose like `"file system: ext4"`. Test on your corpus before tightening the pattern list.
- **Setting `blocked_patterns` REPLACES the default list, not extends it.** If you supply `["jailbreak"]`, you lose all 23 default patterns. To extend, copy the defaults and add your own.
- **`max_payload_kb` is approximate.** The handler sums string byte-lengths recursively; it doesn't `json.dumps` the payload, so the figure is slightly low.
- **Structural signals always fire even if `blocked_patterns` is supplied.** Base64 blobs, repeated caps, and repeated punctuation are hardcoded — there's no rule to disable them.
- **Classifier mode requires controlplane wiring.** `_classifier_fn` is a `ClassVar` set once at startup. If unset, `classifier` mode silently degrades to "no classifier hits".
- **Indirect-injection scanning depends on `context.retrieval_results`.** Agents that don't populate this attribute skip the mid-execution check entirely.
- **The handler does not redact or sanitize.** It only blocks or warns. For redaction, pair with [Content Policy](./content) using `redact` action.

## Next Steps

- [Policy Categories](../features/policy-categories) — All 49 categories
- [Content Policy](./content) — Configurable PII/credential scanning with redact action
- [Input Validation](./input-validation) — Schema/length checks on agent inputs
- [Output Egress Format](./output-egress-format) — Companion control on the output side (LLM05)


--------------------------------------------------------------------------------
# Approval Policy
URL: https://waxell.ai/docs/observe/governance/approval
Description: Human-in-the-loop gates -- require human approval before workflow execution, before specific tools or actions run, or when cost exceeds a threshold. Includes session-start pre-approval and auto-approve by risk level.
--------------------------------------------------------------------------------

# Approval Policy

The `approval` policy category puts a human in the loop before sensitive agent operations. Use it to require approval before a workflow runs, before specific tools or action types execute, or when the in-flight cost crosses a threshold. Also supports session-start gates (E12) for chat-style agents and auto-approval below a risk level.

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `require_approval_for` | string[] | `["deploy", "delete", "payment"]` | Workflow names, action types, tool names, OR the special `"session-start"` token |
| `cost_threshold` | number | `100.00` | Trigger approval when `context.cost_used` exceeds this amount |
| `approvers` | string[] | `[]` | Emails or group IDs surfaced in violation metadata (used by your approval workflow) |
| `timeout_minutes` | integer | `30` | Deadline -- surfaced in metadata for your approval workflow |
| `action_on_timeout` | string | `"block"` | `block` raises PolicyViolationError, `warn` lets the run continue |
| `auto_approve_below_risk` | string | `"low"` | Auto-approve actions with `risk_level` at or below this (`none`, `low`, `medium`, `high`, `critical`) |

## How It Works

The `approval` handler runs at **before_workflow**, **mid_execution**, and **after_workflow**.

| Phase | What It Checks | Actions |
|-------|---------------|---------|
| `before_workflow` | `"session-start"` in `require_approval_for`, or `workflow_type`/`workflow_name` is in the list | BLOCK or WARN per `action_on_timeout` |
| `mid_execution` | Each `pending_action` against `require_approval_for` (with `auto_approve_below_risk` skip), cost threshold, `tools_used` against the list | BLOCK or WARN |
| `after_workflow` | Audit: which restricted actions actually executed and whether cost exceeded threshold | WARN (audit-only) |

### Context Attributes Read

| Attribute | Phase | Purpose |
|-----------|-------|---------|
| `context.workflow_type` / `context.workflow_name` | before_workflow | Match against `require_approval_for` |
| `context.pending_actions` | mid_execution, after_workflow | List of `{type, risk_level}` dicts (or strings) the agent wants to run |
| `context.cost_used` | mid_execution, after_workflow | Compare to `cost_threshold` |
| `context.tools_used` | mid_execution, after_workflow | Tool names checked against `require_approval_for` |

When a gate fires, the result `metadata` includes `requires_approval: True`, `approvers`, `timeout_minutes`, and `action_on_timeout`. **The handler does not block on a human response -- your inbox / Cowork session-bootstrap code reads that metadata and routes to the approval flow** (see `services/session_approval.py` for the session-start path).

## Example Policy

```json
{
  "name": "Production Approval Gates",
  "category": "approval",
  "rules": {
    "require_approval_for": ["deploy", "delete", "send_email", "make_purchase"],
    "cost_threshold": 25.00,
    "approvers": ["ops-team@acme.com", "finance@acme.com"],
    "timeout_minutes": 60,
    "action_on_timeout": "block",
    "auto_approve_below_risk": "low"
  },
  "scope": {"agents": ["ops-agent"]},
  "enabled": true
}
```

### Session-Start Approval (E12)

```json
{
  "name": "Session Bootstrap Approval",
  "category": "approval",
  "rules": {
    "require_approval_for": ["session-start"],
    "approvers": ["security@acme.com"],
    "timeout_minutes": 15,
    "action_on_timeout": "block"
  }
}
```

## SDK Integration

```python

waxell.init()

@waxell.observe(agent_name="ops-agent", enforce_policy=True)
async def run_deploy(target: str) -> str:
    # before_workflow checks workflow_name against require_approval_for
    # mid_execution checks pending_actions / tools_used / cost_used
    return await deploy(target)
```

## Observability

| Field | Example |
|-------|---------|
| **Category** | `approval` |
| **Action** | `block` |
| **Reason** | "Action 'payment' requires approval" |
| **Metadata** | `{"requires_approval": true, "approvers": ["finance@acme.com"], "action_type": "payment", "timeout_minutes": 60, "action_on_timeout": "block"}` |

| Field | Example (cost trigger) |
|-------|---------|
| **Reason** | "Cost ($28.45) exceeds approval threshold ($25.00)" |

## Common Gotchas

1. **The handler does not perform approval -- it raises the signal.** A BLOCK result raises `PolicyViolationError`. Your calling code (Cowork session bootstrap, inbox, or a custom router) inspects `metadata["requires_approval"]` and runs the actual approval flow.
2. **`auto_approve_below_risk` requires `risk_level` on each pending action.** If your agent emits `pending_actions` as plain strings (no dict), the risk-level skip is never taken and every match in `require_approval_for` fires the gate.
3. **`session-start` is a magic string.** It does not match against `workflow_name` -- it triggers if the literal `"session-start"` appears in `require_approval_for`. The metadata includes `approval_kind: "session-start"` so routing code can tell session gates apart from workflow gates.
4. **`cost_threshold` checks `context.cost_used`, not the cost of the next action.** The check fires after the cost has already been incurred. To gate before spending money, pair this with the [Budget Policy](./budget) `per_workflow_cost_limit`.
5. **`require_approval_for` is a flat list shared across workflow names, action types, and tool names.** A tool called `delete` and a workflow named `delete` both match the same string. Use distinct names if you need different behavior.
6. **`after_workflow` always returns WARN if any restricted action ran, not BLOCK.** It is an audit signal -- the run has already happened. Use `mid_execution` BLOCK to actually halt.

## Next Steps

- [Safety Policy](./safety) -- `require_human_approval` and `approval_tools` for simpler cases
- [Budget Policy](./budget) -- Block by cost before the spend happens
- [Domain Governance](./domain-governance) -- Approval for specific domain actions
- [Policy Categories](../features/policy-categories)


--------------------------------------------------------------------------------
# Delegation Policy
URL: https://waxell.ai/docs/observe/governance/delegation
Description: Multi-agent trust governance -- controls delegation depth, authorized delegates, concurrent sub-agent limits, and policy inheritance across agent hierarchies.
--------------------------------------------------------------------------------

# Delegation Policy

The `delegation` policy category governs **multi-agent trust relationships** -- who can delegate to whom, how deep delegation chains can go, and how many agents can run concurrently.

Use it when your system uses orchestrator-worker patterns and you need to enforce which agents are trusted to act as sub-agents, cap recursion depth, and control concurrency.

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `max_delegation_depth` | integer | `3` | Maximum nesting depth of the delegation chain. A parent delegating to a child is depth 1; that child delegating further is depth 2. |
| `allowed_delegates` | string[] | `[]` | Allowlist of agent names that may act as delegates. Empty list means no restriction. |
| `blocked_delegates` | string[] | `[]` | Blocklist of agent names that are never permitted as delegates. Takes priority over `allowed_delegates`. |
| `inherit_policies` | boolean | `true` | Whether delegates must inherit the parent's governance policies. Checked in after_workflow; generates WARNs only. |
| `require_approval` | boolean | `false` | Whether each delegation must be explicitly approved. If `true`, any delegation without `approved: true` is blocked. |
| `max_concurrent_delegates` | integer | `5` | Maximum number of simultaneously active delegate agents. |

## How It Works

The delegation handler runs at **all three phases** and reads context data populated by `ctx.record_delegation()`.

### Enforcement Phases

| Phase | What it checks | Action on violation |
|-------|---------------|-------------------|
| **before_workflow** | If the current agent is a delegate: checks delegation depth, blocked list, allowed list | BLOCK |
| **mid_execution** | Concurrent delegate count; per-delegation depth, blocked list, allowed list, approval status | BLOCK |
| **after_workflow** | Full delegation tree audit -- flags blocked delegates used, missing policy inheritance | WARN only |

### before_workflow — Delegate Self-Check

When an agent starts with `is_delegate=True` (it was spawned by a parent), the handler validates the agent itself:

1. Is `delegation_depth` greater than `max_delegation_depth`? → BLOCK
2. Is `agent_name` in `blocked_delegates`? → BLOCK
3. Is `allowed_delegates` non-empty and `agent_name` not in it? → BLOCK

### mid_execution — Delegation Enforcement

Each time the orchestrator calls `ctx.record_delegation()` and then records a tool call, the handler checks the accumulated delegations list:

1. Is `concurrent_delegates` greater than `max_concurrent_delegates`? → BLOCK
2. For each delegation: does `depth` exceed `max_delegation_depth`? → BLOCK
3. Is the delegate in `blocked_delegates`? → BLOCK
4. Is `allowed_delegates` non-empty and delegate not in it? → BLOCK
5. If `require_approval` is `true`: does the delegation have `approved: true`? → BLOCK if not

### after_workflow — Delegation Audit

After the agent completes, the handler audits the full delegation tree:

- Were any blocked delegates actually used? → WARN
- Did any delegates fail to inherit policies? → WARN
- Records `delegates_used`, `max_depth`, `total_delegations` in metadata

After_workflow never blocks. It is an audit-only phase.

## Matching Logic

| Scenario | Rule | Match? | Why |
|----------|------|--------|-----|
| Depth 3, max_delegation_depth 3 | depth > max | No | 3 is not > 3 |
| Depth 4, max_delegation_depth 3 | depth > max | Yes → BLOCK | 4 > 3 |
| delegate "summarizer", allowed: ["summarizer"] | name in allowed | ALLOW | Exact match |
| delegate "admin-agent", allowed: ["summarizer"] | name not in allowed | BLOCK | Not in list |
| delegate "admin-agent", blocked: ["admin-agent"] | name in blocked | BLOCK | Exact match |
| delegate "helper", blocked: ["admin-agent"] | name not in blocked | Continue checking | No match |
| 6 concurrent, max 5 | concurrent > max | BLOCK | 6 > 5 |
| 5 concurrent, max 5 | concurrent > max | No | 5 is not > 5 |

**Empty allowed_delegates means no restriction**
An empty `allowed_delegates` list (`[]`) means any agent name is allowed. This is the default. Set it to a non-empty list to create a strict allowlist.


**blocked_delegates takes priority**
If an agent name appears in both `blocked_delegates` and `allowed_delegates`, it is blocked. The blocked check runs first.


## Example Policies

### Strict Orchestrator

Small trusted allowlist, limited depth, approval required:

```json
{
  "max_delegation_depth": 2,
  "allowed_delegates": ["data-fetcher", "summarizer", "calculator"],
  "blocked_delegates": ["admin-agent", "root-agent"],
  "inherit_policies": true,
  "require_approval": true,
  "max_concurrent_delegates": 3
}
```

### Permissive Multi-Agent System

No allowlist, high depth, no approval:

```json
{
  "max_delegation_depth": 10,
  "allowed_delegates": [],
  "blocked_delegates": [],
  "inherit_policies": false,
  "require_approval": false,
  "max_concurrent_delegates": 20
}
```

### Blocklist Only

Block specific dangerous agents, allow everything else:

```json
{
  "max_delegation_depth": 5,
  "allowed_delegates": [],
  "blocked_delegates": ["admin-agent", "root-agent", "privileged-executor"],
  "inherit_policies": true,
  "require_approval": false,
  "max_concurrent_delegates": 10
}
```

## SDK Integration

### Recording Delegations

Use `ctx.record_delegation()` to register each delegation the orchestrator makes. Call it before recording the tool call that triggers mid_execution governance:

```python

from waxell_observe.errors import PolicyViolationError

try:
    async with waxell.WaxellContext(
        agent_name="orchestrator",
        workflow_name="multi-agent-task",
        inputs={"query": query},
        enforce_policy=True,
    ) as ctx:

        # Record each delegation before simulating child work
        ctx.record_delegation(
            delegate="data-fetcher",
            depth=1,
            approved=True,
            policies_inherited=True,
        )

        # Tool call triggers mid_execution governance
        ctx.record_tool_call(
            name="delegate_task",
            tool_type="agent",
            input={"delegate": "data-fetcher", "task": "Fetch market data"},
            output={"result": "1,240 records retrieved"},
        )

        # Add a second delegate
        ctx.record_delegation(
            delegate="summarizer",
            depth=1,
            approved=True,
            policies_inherited=True,
        )
        ctx.record_tool_call(
            name="delegate_task",
            tool_type="agent",
            input={"delegate": "summarizer", "task": "Summarize findings"},
            output={"result": "Summary: tech sector up 18%"},
        )

except PolicyViolationError as e:
    print(f"Delegation blocked: {e}")
    # e.g. "Delegation to blocked agent 'admin-agent'"
    # e.g. "Delegation to 'executor' at depth 4 exceeds maximum (3)"
    # e.g. "Concurrent delegates (8) exceeds limit (5)"
```

### record_delegation Parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `delegate` | str | required | Name of the delegate agent |
| `depth` | int | `1` | Delegation depth (nesting level) |
| `approved` | bool | `True` | Whether delegation was pre-approved |
| `policies_inherited` | bool | `True` | Whether delegate inherits parent policies |

Each call to `record_delegation()` appends to `context.delegations`, updates `context.delegation_depth`, and increments `context.concurrent_delegates`.

## Enforcement Flow

```
Agent starts (WaxellContext.__aenter__)
    │
    └── before_workflow governance runs
        │
        ├── Is this agent a delegate (is_delegate=True)?
        │   ├── Yes → check depth, blocked list, allowed list
        │   │   └── Violation? → BLOCK
        │   └── No → store rules, return ALLOW
        │
        └── Rules stored for mid_execution use

During execution (after each record_delegation + tool call):
    │
    └── mid_execution governance runs
        │
        ├── concurrent_delegates > max_concurrent? → BLOCK
        │
        └── For each delegation in context.delegations:
            ├── depth > max_delegation_depth? → BLOCK
            ├── delegate in blocked_delegates? → BLOCK
            ├── allowed_delegates non-empty and delegate not in list? → BLOCK
            └── require_approval and not approved? → BLOCK

Agent exits (WaxellContext.__aexit__):
    │
    └── after_workflow governance runs
        │
        ├── Any blocked delegates used? → WARN
        ├── Any delegates without policy inheritance? → WARN
        └── Record delegation tree audit (delegates_used, max_depth, total)
```

## Creating via Dashboard

1. Navigate to **Governance > Policies**
2. Click **New Policy**
3. Select category **Delegation**
4. Configure `max_delegation_depth`, `allowed_delegates`, `blocked_delegates`
5. Set `require_approval` and `max_concurrent_delegates` as needed
6. Set scope to target the orchestrator agent (e.g., `delegation-orchestrator`)
7. Enable

## Creating via API

```bash
curl -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  https://acme.waxell.dev/waxell/v1/policies/ \
  -d '{
    "name": "Multi-Agent Delegation",
    "category": "delegation",
    "rules": {
      "max_delegation_depth": 3,
      "allowed_delegates": ["data-fetcher", "summarizer", "calculator"],
      "blocked_delegates": ["admin-agent"],
      "inherit_policies": true,
      "require_approval": false,
      "max_concurrent_delegates": 5
    },
    "scope": {
      "agents": ["orchestrator"]
    },
    "enabled": true
  }'
```

## Observability

### Governance Tab

Delegation evaluations appear for each phase:

**mid_execution (allowed):**

| Field | Example |
|-------|---------|
| **Phase** | `mid_execution` |
| **Action** | `allow` |
| **Reason** | "Delegation within policy (2 delegation(s))" |

**mid_execution (blocked):**

| Field | Example |
|-------|---------|
| **Phase** | `mid_execution` |
| **Action** | `block` |
| **Reason** | "Delegation to blocked agent 'admin-agent'" |
| **Metadata** | `&#123;"blocked_delegate": "admin-agent"&#125;` |

**after_workflow (allowed):**

| Field | Example |
|-------|---------|
| **Phase** | `after_workflow` |
| **Action** | `allow` |
| **Reason** | "Delegation audit passed (2 delegations, max depth 1)" |
| **Metadata** | `&#123;"delegation_tree": &#123;"delegates_used": ["data-fetcher", "summarizer"], "max_depth": 1, "total_delegations": 2&#125;&#125;` |

## Common Gotchas

1. **Empty `allowed_delegates` means no restriction, not "block all".** To restrict delegates, add at least one entry to the list. An empty list is the default permissive state.

2. **`blocked_delegates` takes priority over `allowed_delegates`.** If an agent name appears in both lists, it is blocked. Check for accidental overlap when configuring both.

3. **Depth check uses strict `>` comparison.** A `max_delegation_depth` of 3 allows depth 1, 2, and 3. Depth 4 is blocked. The comparison is `depth > max_delegation_depth`, not `>=`.

4. **`inherit_policies` only generates WARNs in after_workflow, never blocks.** It is an audit signal, not an enforcement gate. To enforce policy inheritance, you need orchestrator-side logic.

5. **`require_approval` checks the `approved` field on each delegation record.** The orchestrator must explicitly pass `approved=True` to `ctx.record_delegation()`. If the field is missing or `False`, the delegation is blocked when `require_approval=True`.

6. **Concurrent delegates count reflects accumulated delegations, not active ones.** Each call to `ctx.record_delegation()` increments the concurrent count. The count never decreases during a run. It represents the total number of delegations made, not the number running at any instant.

7. **before_workflow checks the orchestrator itself as a delegate.** This phase only does work if `is_delegate=True` on the context. For top-level orchestrators, before_workflow always returns ALLOW.

8. **mid_execution runs after every `record_delegation()` call followed by a tool call or step.** Record the delegation and then immediately record the associated tool call to ensure mid_execution sees the latest delegation state.

## Next Steps

- [Policy & Governance](../features/governance) -- How policy enforcement works
- [Compliance Policy](./compliance) -- Validates that required sibling policies are active
- [Privacy Policy](./privacy) -- Consent, residency, and purpose governance
- [Communication Policy](./communication) -- Govern output channels
- [Policy Categories & Templates](../features/policy-categories) -- All 26 categories


--------------------------------------------------------------------------------
# Cross-Agent Isolation Policy
URL: https://waxell.ai/docs/observe/governance/cross-agent-isolation
Description: Prevent one agent from reading another agent's memory, scratchpad, tool state, or context. Agent-level analogue of multi-tenancy.
--------------------------------------------------------------------------------

# Cross-Agent Isolation Policy

The `cross-agent-isolation` policy category prevents one agent from reading another agent's memory, scratchpad, tool state, or context -- **even when they share the same tenant runtime**. A finance-bot must not silently pull a support-bot's session context, and a sub-agent must not exfiltrate its parent's working memory unless explicitly authorized.

This is the agent-level analogue of multi-tenancy. Required for the Atlassian / IBM enterprise pitch. Maps to **NIST AI RMF MS-2.7** (segmentation between AI systems), **ISO 42001 A.8.3** (information access restriction), and **OWASP LLM01** lineage gap.

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `isolation_scopes` | string[] | `["memory", "context", "tool_state", "scratchpad"]` | Surfaces to isolate |
| `allow_cross_agent_reads` | object[] | `[]` | Allowlist entries: `{source, target}` pairs OR `{trust_group}` names |
| `block_cross_tenant` | boolean | `true` | Hard-block any cross-tenant read regardless of allowlist |
| `tool_patterns_cross_agent` | string[] | `["memory.read_other_*", "scratchpad.read", "agent_handoff.*"]` | fnmatch patterns for cross-agent tool calls |
| `action_on_violation` | string | `block` | `block`, `warn`, or `redact` |

## How It Works

Runs at **`before_workflow`** and **`mid_execution`**. `after_workflow` is a no-op (isolation is enforced before the surface is touched).

Three checks in order:

### 1. Cross-tenant short-circuit (hard ceiling)

If `block_cross_tenant: true` and the context carries a `cross_tenant_read` marker whose tenant differs from the calling agent's tenant -> BLOCK, **bypassing the allowlist**. No allowlist entry can override this.

### 2. Explicit cross-agent reads

Reads `context.read_from_agent` -- the runtime emits this when `memory.read(agent=...)`, `handoff(...)`, or similar primitives are invoked. Normalizes to `(source, target)` pairs and checks against `allow_cross_agent_reads`.

### 3. Pending tool calls on cross-agent surfaces

Scans `context.pending_actions` for tool calls whose name matches any of `tool_patterns_cross_agent` (fnmatch). Tries to extract `target_agent` from args (`args.agent`, `args.target_agent`, `args.source_agent`, `args.from`). If the target is set and not allowlisted -> violation.

### Allowlist Resolution

Entries can be:

- `{"source": "*", "target": "shared-memory"}` -- any agent may read shared-memory
- `{"source": "finance-bot", "target": "audit-logger"}` -- specific pair
- `{"trust_group": "loan-underwriting"}` -- both agents must carry membership in `context.agent_trust_groups`

Star wildcards (`*`) match any source/target. fnmatch patterns are supported.

### Context Attributes Read

| Attribute | Phase | Purpose |
|-----------|-------|---------|
| `context.tenant_id` / `context.tenant` | all | Calling agent's tenant for cross-tenant check |
| `context.cross_tenant_read` | all | Explicit cross-tenant handoff marker |
| `context.read_from_agent` | all | Explicit cross-agent read signal |
| `context.agent_name` | all | Source agent for pair resolution |
| `context.pending_actions` | all | Pending tool calls to scan |
| `context.agent_trust_groups` / `context.trust_groups` | all | Declared trust-group memberships |

## Example Policy

### Strict isolation with declared trust groups

```json
{
  "name": "Strict cross-agent isolation",
  "category": "cross-agent-isolation",
  "rules": {
    "isolation_scopes": ["memory", "context", "tool_state", "scratchpad"],
    "allow_cross_agent_reads": [
      {"source": "*", "target": "shared-memory"},
      {"trust_group": "loan-underwriting"}
    ],
    "block_cross_tenant": true,
    "tool_patterns_cross_agent": [
      "memory.read_other_*",
      "scratchpad.read",
      "agent_handoff.*"
    ],
    "action_on_violation": "block"
  },
  "scope": {
    "agents": ["*"]
  },
  "enabled": true
}
```

## SDK Integration

```python

waxell.init()

@waxell.observe(
    agent_name="finance-bot",
    enforce_policy=True,
    metadata={"agent_trust_groups": ["loan-underwriting"]},
)
async def finance(query: str) -> str:
    return await analyze(query)
```

For multi-agent workflows that need to legitimately share state, declare both agents in the same `agent_trust_groups` AND add a `{"trust_group": "<name>"}` entry to `allow_cross_agent_reads`.

## Observability

| Field | Example (BLOCK) |
|-------|---------|
| **Category** | `cross-agent-isolation` |
| **Action** | `block` |
| **Reason** | `Agent 'finance-bot' attempted to read state from 'support-bot'; no allowlist entry permits this cross-agent read.` |
| **Metadata** | `{"source_agent": "finance-bot", "target_agent": "support-bot", "nist_ai_rmf": "MS-2.7", "iso_42001": "A.8.3"}` |

## Common Gotchas

- **Cross-tenant is a hard ceiling.** No allowlist can override `block_cross_tenant: true`. To allow cross-tenant reads (rare), explicitly disable the flag.
- **Allowlist entries are dicts, not strings.** `{"source": "...", "target": "..."}` or `{"trust_group": "..."}`. Plain strings are silently ignored.
- **Trust groups require BOTH agents to declare membership.** If only the source agent carries the group, the entry doesn't apply.
- **fnmatch patterns, not regex.** `memory.read_other_*` matches `memory.read_other_agent` but not `memory.readOther`. Patterns are also case-insensitive (the handler matches both raw and lowercased).
- **Reads `same-tenant cross-agent` -- not cross-tenant.** This category is for isolating agents within ONE tenant. Cross-tenant is handled as a hard ceiling.
- **Pending tool calls need `args` populated with the target.** If your runtime doesn't pass `agent` / `target_agent` in tool args, the policy can't extract a target and reports `unknown`.
- **`redact` action is supported** in addition to `block` / `warn` -- useful when you want the run to continue but the cross-agent read to be stripped from the response.

## Next Steps

- [Delegation Policy](./delegation) -- Constrain agent-to-agent handoffs and budgets
- [Memory Policy](./memory) -- Per-agent memory policy (writes/reads)
- [Identity Policy](./identity) -- Agent identity and authentication
- [Compliance Policy](./compliance) -- NIST AI RMF / ISO 42001 framework binding
- [Policy Categories & Templates](../features/policy-categories) -- All categories


--------------------------------------------------------------------------------
# Communication Policy
URL: https://waxell.ai/docs/observe/governance/communication
Description: Govern agent output channels -- allowed/blocked channels, message limits, disclaimer requirements, and channel-level access control.
--------------------------------------------------------------------------------

# Communication Policy

The `communication` policy category governs what output channels agents can use to send messages. It controls which channels are allowed or blocked, enforces message count limits per execution, and requires disclaimer text in outbound messages.

Use it when your agents send notifications, emails, Slack messages, or any other form of external communication.

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `allowed_channels` | string[] | `[]` | Channels agents are permitted to use. If set, only these channels are allowed. Leave empty to allow all channels not in `blocked_channels` |
| `blocked_channels` | string[] | `[]` | Channels explicitly denied (takes precedence over allowed) |
| `max_messages_per_execution` | integer | `50` | Maximum number of messages an agent can send in a single execution |
| `require_disclaimer` | boolean | `false` | Whether outbound messages must contain the disclaimer text |
| `disclaimer_text` | string | `"This message was generated by an AI agent."` | Text that must appear in the message body when `require_disclaimer` is enabled |
| `action_on_violation` | string | `"block"` | `"block"` to prevent the communication, `"warn"` to log and continue |

## How Matching Works

### Channel Matching -- Exact String Equality

Channel names use **exact string matching** (case-sensitive). The channel value passed to `record_communication()` must match exactly what's configured in the policy.

| Blocked Channel | Recorded Channel | Match? | Why |
|----------------|-----------------|--------|-----|
| `sms` | `sms` | Yes | Exact match |
| `sms` | `SMS` | No | Case-sensitive: `"sms"` != `"SMS"` |
| `social-media` | `social-media` | Yes | Exact match |
| `social-media` | `social_media` | No | Hyphens vs underscores matter |
| `slack` | `slack-dm` | No | Not a prefix/substring match -- must be exact |

**Channel Names Are Case-Sensitive**
Unlike blocked commands in the code-execution policy (which use case-insensitive substring matching), communication channels use **exact string equality**. `"Slack"` and `"slack"` are different channels. Use consistent lowercase channel names.


### Allowed vs Blocked Channels

When both `allowed_channels` and `blocked_channels` are configured:

1. **Blocked channels are checked first** -- if the channel is in the blocklist, it's denied regardless of the allowlist
2. **If the allowlist is non-empty**, only channels in the allowlist are permitted
3. **If the allowlist is empty**, all channels not in the blocklist are allowed

| allowed_channels | blocked_channels | Channel | Result |
|-----------------|-----------------|---------|--------|
| `["slack", "email"]` | `["sms"]` | `slack` | ALLOW |
| `["slack", "email"]` | `["sms"]` | `sms` | BLOCK (in blocklist) |
| `["slack", "email"]` | `["sms"]` | `webhook` | BLOCK (not in allowlist) |
| `[]` | `["sms"]` | `webhook` | ALLOW (no allowlist restriction) |
| `[]` | `["sms"]` | `sms` | BLOCK (in blocklist) |

### Disclaimer Enforcement

When `require_disclaimer` is enabled, the `disclaimer_text` must appear as a **substring** in the message `body`. This is not an exact match -- the disclaimer can appear anywhere in the body text.

| disclaimer_text | Message body | Match? |
|----------------|-------------|--------|
| `"This message was generated by an AI agent."` | `"Hello! This message was generated by an AI agent."` | Yes |
| `"This message was generated by an AI agent."` | `"Update deployed. This message was generated by an AI agent."` | Yes |
| `"This message was generated by an AI agent."` | `"Hello! No disclaimer here."` | No |
| `"This message was generated by an AI agent."` | `"THIS MESSAGE WAS GENERATED BY AN AI AGENT."` | No (case-sensitive) |

**Disclaimer Is Case-Sensitive**
The disclaimer check is a substring match, but it **is case-sensitive**. Ensure agents include the exact text configured in the policy.


### Message Count

The `max_messages_per_execution` limit counts the total number of `record_communication()` calls within a single agent execution (one `WaxellContext`). Each call increments the count regardless of channel.

## Example Policies

### Internal Only

Block external communication channels, allow internal tools:

```json
{
  "allowed_channels": ["slack", "internal-api"],
  "blocked_channels": ["email", "sms", "social-media"],
  "max_messages_per_execution": 20,
  "require_disclaimer": false,
  "action_on_violation": "block"
}
```

### Governed External Communications

Allow email to company domains with mandatory disclaimers:

```json
{
  "allowed_channels": ["slack", "email", "internal-api"],
  "blocked_channels": ["sms", "social-media"],
  "max_messages_per_execution": 10,
  "require_disclaimer": true,
  "disclaimer_text": "This message was generated by an AI agent.",
  "action_on_violation": "block"
}
```

## SDK Integration

### Using the Context Manager

```python


waxell.init()

@waxell.observe(
    agent_name="notifier",
    enforce_policy=True,
    mid_execution_governance=True,
)
async def send_notification(channel: str, message: str):
    # Record the communication -- triggers governance check
    waxell.communication(
        channel=channel,
        recipient="#engineering",
        body=message,
        subject="",
    )
    return {"sent": True}
```

### Direct Context Usage

```python
from waxell_observe import WaxellContext
from waxell_observe.errors import PolicyViolationError

try:
    async with WaxellContext(
        agent_name="notifier",
        enforce_policy=True,
        mid_execution_governance=True,
    ) as ctx:
        ctx.record_communication(
            channel="email",
            recipient="user@acme.com",
            body="Your report is ready. This message was generated by an AI agent.",
            subject="Report Ready",
        )
except PolicyViolationError as e:
    print(f"Blocked: {e}")
```

## Enforcement Flow

```
Agent calls record_communication(channel="sms", body="...")
    │
    ├── Buffer communication dict
    ├── Buffer step (comm:sms) + span (kind=io)
    ├── Emit OTel span (comm:sms)
    ├── Log: "Communication: sms → +1-555-0100"
    │
    └── Mid-execution governance check
        │
        ├── Is "sms" in blocked_channels? → YES → BLOCK
        ├── Is allowed_channels set AND "sms" not in it? → BLOCK
        ├── Message count > max_messages_per_execution? → BLOCK
        └── require_disclaimer AND disclaimer_text not in body? → BLOCK
```

The policy is evaluated at three points:
- **Pre-execution**: Before the agent runs (basic checks)
- **Mid-execution**: On each `record_communication()` call (channel, count, disclaimer checks)
- **Post-execution**: After the run completes (final summary)

## Creating via Dashboard

1. Navigate to **Governance > Policies**
2. Click **New Policy**
3. Select category **Communication**
4. Configure rules (channels, limits, disclaimer)
5. Set scope to target specific agents (e.g., `communication-agent`)
6. Enable

## Creating via API

```bash
curl -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  https://acme.waxell.dev/waxell/v1/policies/ \
  -d '{
    "name": "Communication Governance",
    "category": "communication",
    "rules": {
      "allowed_channels": ["slack", "email", "internal-api"],
      "blocked_channels": ["sms", "social-media"],
      "max_messages_per_execution": 10,
      "require_disclaimer": true,
      "disclaimer_text": "This message was generated by an AI agent.",
      "action_on_violation": "block"
    },
    "scope": {
      "agents": ["communication-agent"]
    },
    "enabled": true
  }'
```

## Observability

### Trace Spans

Each `record_communication()` creates a span visible in the Trace tab:

| Span name | Kind | Attributes |
|-----------|------|-----------|
| `comm:slack` | tool | `io.direction: outbound`, `io.channel: slack` |
| `comm:email` | tool | `io.direction: outbound`, `io.channel: email` |
| `comm:sms` | tool | `io.direction: outbound`, `io.channel: sms` |

### Governance Tab

Policy evaluations appear with:
- **Policy name**: The configured policy name
- **Action**: `allow`, `warn`, or `block`
- **Category**: `communication`
- **Reason**: Human-readable explanation (e.g., "Blocked channel 'sms' found")

### Logs

Log entries are automatically correlated to the trace:
```
Communication: slack → #engineering (Deployment completed successfully...)
Communication: sms → +1-555-0100 (Urgent: Server is down...)
```

## Common Gotchas

1. **Channel names are case-sensitive exact match.** `"Slack"` won't match a blocklist entry of `"slack"`. Always use consistent lowercase channel names.

2. **Disclaimer is a substring match, but case-sensitive.** The exact `disclaimer_text` string must appear somewhere in the `body`. `"THIS MESSAGE WAS GENERATED BY AN AI AGENT."` won't match `"This message was generated by an AI agent."`.

3. **Message count is cumulative.** Every `record_communication()` call in a single execution counts toward `max_messages_per_execution`, regardless of channel. Sending 5 Slack messages and 6 emails = 11 total.

4. **Blocked channels take precedence over allowed channels.** If a channel is in both lists, it's blocked.

5. **Empty `allowed_channels` means no allowlist restriction.** If you leave `allowed_channels` empty but set `blocked_channels`, only the blocked channels are denied. All others are permitted.

## Next Steps

- [Policy & Governance](../features/governance) -- How policy enforcement works
- [Code Execution Policy](./code-execution) -- Govern generated code
- [Context Management Policy](./context-management) -- Control conversation boundaries
- [Policy Categories & Templates](../features/policy-categories) -- All 26 categories


--------------------------------------------------------------------------------
# Domain Governance Policy
URL: https://waxell.ai/docs/observe/governance/domain-governance
Description: Per-call guards for Connect domain endpoints -- allowed/blocked domains and actions, payload-size limits, per-run call caps, approval gates, and audit logging. (E5 shadow-mode.)
--------------------------------------------------------------------------------

# Domain Governance Policy

The `domain-governance` policy controls which domain actions an agent can invoke, validates payloads, and gates sensitive operations behind approval. This governs the communication between the Waxell runtime and external business applications (e.g., Firecrawl scraping, database writes, payment processing).

Fires at the `before_domain_call` hook (E5 -- the Connect domain-endpoint guard, currently shadow-mode in many tenants until fully ratcheted on).

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `allowed_domains` | string[] | `[]` | Domain allowlist. Empty = all domains allowed |
| `blocked_domains` | string[] | `[]` | Domain blocklist. Always blocks even if in `allowed_domains` |
| `allowed_actions` | object | `{}` | Per-domain action allowlist, e.g. `{"vendor_research": ["search_web"]}` |
| `blocked_actions` | object | `{}` | Per-domain action blocklist. Use `"*"` to block all actions in a domain |
| `require_approval_for` | string[] | `[]` | `"domain/action"` strings that require approval (emits WARN) |
| `max_payload_size_kb` | integer | `0` | Max payload size per call. `0` = unlimited |
| `max_calls_per_run` | integer | `0` | Max domain calls per agent run. `0` = unlimited |
| `log_all_calls` | boolean | `true` | Log every domain call for audit |
| `action_on_violation` | string | `"block"` | Either `block` or `warn` |

## How It Works

The `domain-governance` handler fires at **`before_domain_call`** (per-call enforcement) and runs audit at **`after_workflow`**.

| Phase | What It Checks | Actions |
|-------|---------------|---------|
| `before_workflow` | Stores rules and call-counter on context for downstream phases | ALLOW |
| `before_domain_call` | Increments call counter; checks `max_calls_per_run`, `blocked_domains`, `allowed_domains`, `blocked_actions`, `allowed_actions`, `max_payload_size_kb`, `require_approval_for` | BLOCK / WARN per `action_on_violation` (approval always WARN) |
| `after_workflow` | Audits `context.domain_intents` for any blocked-domain calls that slipped through | WARN if violations, ALLOW otherwise |

### Context Attributes Read

| Attribute | Phase | Purpose |
|-----------|-------|---------|
| `context._domain_governance_rules` | before_domain_call | Rules stamped by before_workflow |
| `context._domain_call_count` | before_domain_call | Running call count (incremented by handler) |
| `context.domain_intents` | after_workflow | List of `{domain_name, action_name}` dicts the run produced |

### Enforcement Order

For each call the handler evaluates in this order, returning the first violation:

1. `max_calls_per_run` exceeded
2. Domain in `blocked_domains`
3. Domain not in `allowed_domains` (if allowlist set)
4. Action in `blocked_actions[domain]` (or `"*"` blocks all)
5. Action not in `allowed_actions[domain]` (if allowlist set for that domain)
6. Payload size > `max_payload_size_kb`
7. `"domain/action"` in `require_approval_for` (emits WARN, not BLOCK)

## Example Policy

```json
{
  "name": "Vendor Research Agent Guardrails",
  "category": "domain-governance",
  "rules": {
    "allowed_domains": ["vendor_research", "contract_analysis"],
    "blocked_domains": ["payment"],
    "allowed_actions": {
      "vendor_research": ["get_vendor_profile", "search_web", "scrape_website"],
      "contract_analysis": ["get_contracts", "spend_analysis"]
    },
    "blocked_actions": {
      "payment": ["*"]
    },
    "require_approval_for": [
      "vendor_research/save_vendor_research",
      "contract_analysis/save_contract_intelligence"
    ],
    "max_payload_size_kb": 1024,
    "max_calls_per_run": 50,
    "log_all_calls": true,
    "action_on_violation": "block"
  },
  "scope": {"agents": ["procurement-agent"]},
  "enabled": true
}
```

## SDK Integration

```python

waxell.init()

@waxell.observe(agent_name="procurement-agent", enforce_policy=True)
async def research_vendor(vendor: str) -> dict:
    # Every ctx.call_domain("vendor_research", "search_web", {...}) goes
    # through before_domain_call. The handler checks allow/block lists,
    # increments _domain_call_count, and enforces payload size.
    return await ctx.call_domain("vendor_research", "search_web", {"q": vendor})
```

## Observability

| Field | Example (block) |
|-------|---------|
| **Category** | `domain-governance` |
| **Action** | `block` |
| **Reason** | "Action 'payment/charge' is blocked by policy" |
| **Metadata** | `{"domain": "payment", "action": "charge"}` |

| Field | Example (payload size) |
|-------|---------|
| **Reason** | "Domain call payload exceeds limit (1547.2KB > 1024KB)" |
| **Metadata** | `{"domain": "vendor_research", "action": "bulk_import", "payload_size_kb": 1547.2}` |

| Field | Example (approval) |
|-------|---------|
| **Action** | `warn` |
| **Reason** | "Action 'vendor_research/save_vendor_research' requires approval (proceeding with warning)" |
| **Metadata** | `{"requires_approval": true}` |

## Common Gotchas

1. **`require_approval_for` ALWAYS emits WARN, never BLOCK.** It is a soft gate -- the call proceeds but is flagged in observability. To actually halt, put the action in `blocked_actions` instead and use a separate approval workflow.
2. **`allowed_domains` is the most restrictive switch.** A non-empty `allowed_domains` becomes a strict allowlist: any domain not in the list is blocked. To permit everything except a few, leave `allowed_domains` empty and use `blocked_domains`.
3. **`blocked_actions["*"]` is wildcard for that domain, not all domains.** Setting `"blocked_actions": {"payment": ["*"]}` blocks all `payment` actions. There is no global `"*"` key -- use `blocked_domains: ["payment"]` for whole-domain blocking.
4. **Payload-size check requires JSON-serializable inputs.** The handler calls `json.dumps(inputs, default=str)`. If inputs contain a non-serializable object that even `default=str` can't handle, the size check is silently skipped (caught `TypeError`/`ValueError`).
5. **`max_calls_per_run` is checked BEFORE the other rules.** A run hitting the call cap will be blocked with "Domain call limit exceeded" even if the next call would have been blocked for a more specific reason.
6. **`before_workflow` is required to stamp `context._domain_call_count = 0`.** If your runtime path skips `before_workflow` and goes straight to `before_domain_call`, the counter starts at `getattr(context, "_domain_call_count", 0)` which is 0 -- still safe, but the rule schema assumes the stamp happened.
7. **Audit in `after_workflow` only warns, it cannot block.** The run is over. Use it for compliance evidence; rely on `before_domain_call` BLOCK for prevention.

## Next Steps

- [Tool Allowlist (Safety)](./safety) -- `blocked_tools` / `approval_tools` for tool-level control, complementary to domain-level
- [Network Policy](./network) -- Lower-level HTTP allow/block; domain governance is the semantic layer above it
- [Approval Policy](./approval) -- General human-in-the-loop gates
- [Policy Categories](../features/policy-categories)


--------------------------------------------------------------------------------
# Signal Governance Policy
URL: https://waxell.ai/docs/observe/governance/signal-governance
Description: Control which signals can be dispatched, from what sources, with rate-limits and payload validation. Evaluates at signal ingestion -- BEFORE any agent execution begins.
--------------------------------------------------------------------------------

# Signal Governance Policy

The `signal-governance` policy category controls **which signals can be dispatched, from what sources, with what rate-limits, and what payload constraints**. It evaluates at signal ingestion time -- **before any agent execution begins**.

Use it to lock down which external triggers your tenant accepts (webhooks, schedules, agent-emitted signals), throttle floods, and require correlation IDs for auditability.

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `allowed_signals` | string[] | `[]` | Signal names allowed. Empty = all allowed |
| `blocked_signals` | string[] | `[]` | Signal names blocked |
| `allowed_sources` | string[] | `[]` | Source types allowed (`webhook`, `agent`, `schedule`, `manual`, `api`) |
| `blocked_sources` | string[] | `[]` | Source types blocked |
| `rate_limit_per_minute` | integer | `0` | Max signals/min per name. `0` = unlimited |
| `rate_limit_per_hour` | integer | `0` | Max signals/hour per name. `0` = unlimited |
| `require_correlation_id` | boolean | `false` | Require `_correlation_id` in payload |
| `max_payload_size_kb` | integer | `0` | Payload size cap in KB. `0` = unlimited |
| `log_all_signals` | boolean | `true` | Log every ALLOW |
| `action_on_violation` | string | `block` | `block` or `warn` |

## How It Works

The handler is **runtime-plane only** -- the observe plane never sees signal dispatch events.

Primary enforcement at **`before_signal_dispatch(signal_name, payload)`**. `before_workflow` and `after_workflow` are no-ops (audit-only stubs).

Order of checks:

1. **Blocked signals** -- `signal_name in blocked_signals` -> violation
2. **Allowed signals** -- if `allowed_signals` non-empty and signal not in it -> violation
3. **Blocked sources** -- `source_type in blocked_sources` -> violation
4. **Allowed sources** -- if `allowed_sources` non-empty and source not in it -> violation
5. **Payload size** -- `len(json.dumps(payload))` vs `max_payload_size_kb * 1024`
6. **Correlation ID** -- if `require_correlation_id`, payload must include `_correlation_id`
7. **Rate limit per minute** -- in-memory sliding window per `(tenant, signal_name)`
8. **Rate limit per hour** -- same window, hour bucket

### Context Attributes Read

| Attribute | Phase | Purpose |
|-----------|-------|---------|
| `context.source_type` | before_signal_dispatch | Source type for the dispatch |
| `payload._source_type` | before_signal_dispatch | Fallback source resolver |
| `payload._correlation_id` | before_signal_dispatch | For `require_correlation_id` check |
| `context.tenant_id` | before_signal_dispatch | Scope rate-limit windows |

### Rate Limiting

Uses an in-memory dict `_rate_windows` keyed by `f"{tenant_id}:{signal_name}"`. Each entry is a list of monotonic timestamps; cleaned to 1-hour cutoff on every dispatch.

## Example Policy

### Lock down a tenant's signal surface

```json
{
  "name": "Production signal lock-down",
  "category": "signal-governance",
  "rules": {
    "allowed_signals": ["research_vendor", "analyze_contracts", "summarize"],
    "blocked_signals": ["admin_override", "delete_all"],
    "allowed_sources": ["webhook", "schedule"],
    "rate_limit_per_minute": 60,
    "rate_limit_per_hour": 500,
    "require_correlation_id": true,
    "max_payload_size_kb": 512,
    "log_all_signals": true,
    "action_on_violation": "block"
  },
  "scope": {
    "agents": ["*"]
  },
  "enabled": true
}
```

## SDK Integration

```python

waxell.init()

# Signals are dispatched via the runtime's signal-dispatch API.
# The runtime calls before_signal_dispatch(signal_name, payload)
# automatically -- no SDK-level changes needed.

# To pass a correlation ID:
await runtime.dispatch_signal(
    "research_vendor",
    payload={"_correlation_id": "req-abc-123", "vendor": "Acme"},
)
```

## Observability

| Field | Example (BLOCK -- not allowed) |
|-------|---------|
| **Category** | `signal-governance` |
| **Action** | `block` |
| **Reason** | `Signal 'admin_override' is blocked by policy` |
| **Metadata** | `{"signal": "admin_override"}` |

| Field | Example (BLOCK -- rate limit) |
|-------|---------|
| **Action** | `block` |
| **Reason** | `Signal 'webhook_trigger' rate limit exceeded (61/60 per minute)` |
| **Metadata** | `{"signal": "webhook_trigger", "rate": 61, "limit": 60, "window": "minute"}` |

| Field | Example (BLOCK -- payload size) |
|-------|---------|
| **Action** | `block` |
| **Reason** | `Signal payload exceeds limit (612.4KB > 512KB)` |

## Common Gotchas

- **Runtime-plane only.** The observe plane never sees signal dispatch -- `supported_planes = ["runtime"]`. If you're running observe-only, this category does nothing.
- **In-memory rate windows.** `_rate_windows` is a class-level dict -- not durable across process restarts and not shared across workers. For strict global limits, use the [Rate Limit Policy](./rate-limit) which has Redis-backed state.
- **Empty `allowed_signals` = ALL signals allowed.** Only acts as an allowlist when non-empty. Same shape as the allowlist family.
- **Source type resolution.** First tries `context.source_type`, then `payload["_source_type"]`. Empty source bypasses source checks entirely.
- **Payload size uses `json.dumps`** with `default=str`. Some non-serializable objects are stringified, inflating size.
- **`require_correlation_id` only checks `payload["_correlation_id"]`.** Not `context.correlation_id` and not other variant names.
- **First violation short-circuits.** Subsequent checks aren't run. If you need to know ALL violations on a signal, run with `warn` first and inspect the trace.
- **`log_all_signals` is INFO-level.** Set log level on `waxell.policies.handlers.signal_governance` to silence ALLOW logs in production.

## Next Steps

- [Rate Limit Policy](./rate-limit) -- Tenant-wide rate caps (Redis-backed)
- [End-User Rate Limit](./end-user-rate-limit) -- Per-sub-user rate caps
- [Input Validation Policy](./input-validation) -- Payload schema validation
- [Audit Policy](./audit) -- Audit logging for signal dispatches
- [Policy Categories & Templates](../features/policy-categories) -- All categories


--------------------------------------------------------------------------------
# Tool Allowlist Policy
URL: https://waxell.ai/docs/observe/governance/tool-allowlist
Description: Positive-list governance for tools -- only listed tools may be invoked. Differs from safety.blocked_tools (deny-only) by adding a hard allowlist.
--------------------------------------------------------------------------------

# Tool Allowlist Policy

The `tool-allowlist` policy category enforces a **positive allowlist of tools** an agent may invoke. The absence of a tool from the allowlist blocks it.

This differs from `safety.blocked_tools` (deny-only): with the allowlist, you don't have to enumerate every dangerous tool -- you only have to enumerate the *safe* ones. Critical for Codex and third-party agents whose tool surface is unbounded.

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `allowed_tools` | string[] | `[]` | Positive allowlist. Empty = no allowlist (all tools allowed unless in `blocked_tools`) |
| `blocked_tools` | string[] | `[]` | Always-deny list. **Wins over allowlist** (deny overrides allow) |
| `action_on_violation` | string | `block` | Action on violation: `block` or `warn` |

## How It Works

The `tool-allowlist` handler runs at **`mid_execution`** and **`after_workflow`** (as a fallback for batched tools-used updates). `before_workflow` is a no-op since no tools have been invoked yet.

Semantics:

1. Empty `allowed_tools` -> no positive allowlist; only `blocked_tools` applies
2. Non-empty `allowed_tools` -> ONLY those tools are permitted; any other tool is a violation
3. `blocked_tools` always wins (deny overrides allow)

### Context Attributes Read

| Attribute | Phase | Purpose |
|-----------|-------|---------|
| `context.tools_used` | mid, after | List of tool names invoked this run |

## Example Policy

### Strict allowlist for Codex-style agents

```json
{
  "name": "Codex tool allowlist",
  "category": "tool-allowlist",
  "rules": {
    "allowed_tools": ["read_file", "list_directory", "search_code"],
    "blocked_tools": ["shell_exec", "write_file"],
    "action_on_violation": "block"
  },
  "scope": {
    "agents": ["codex-*"]
  },
  "enabled": true
}
```

### Warn-first rollout

```json
{
  "rules": {
    "allowed_tools": ["http_get", "summarize"],
    "action_on_violation": "warn"
  }
}
```

Roll out as `warn` first, watch the governance trace for what tools your agents actually use, then promote to `block`.

## SDK Integration

```python

waxell.init()

@waxell.observe(agent_name="codex-assistant", enforce_policy=True)
async def codex_agent(query: str) -> str:
    return await run_agent(query)
```

The handler reads `context.tools_used` -- the runtime / observe SDK populates this as the agent invokes tools. Mid-execution checks fire after each tool batch.

## Observability

| Field | Example (BLOCK) |
|-------|---------|
| **Category** | `tool-allowlist` |
| **Action** | `block` |
| **Reason** | `Tool 'shell_exec' is on the deny list` or `Tool 'fetch_url' is not on the allowlist` |
| **Metadata** | `{"blocked_tool": "shell_exec", "allowed_tools": [...]}` |

## Common Gotchas

- **Exact name match.** `"shell"` does not block `"shell_exec"`. Use the full tool name.
- **Empty allowlist = NO allowlist.** Setting `allowed_tools: []` does not block all tools -- it disables the allowlist and falls back to `blocked_tools` only. To block all tools, use a single never-matching name like `["__none__"]`.
- **`blocked_tools` overrides `allowed_tools`.** Listing the same tool in both = blocked.
- **Differs from `safety.blocked_tools`.** Safety's blocked_tools is checked via `check_tool_allowed()` (a standalone API). The allowlist handler checks `context.tools_used` automatically at mid_execution. Use both for layered defense.
- **Mid-execution fires only if the runtime calls the handler between steps.** `after_workflow` is the fallback to catch batched updates.
- **First violation short-circuits the run.** The handler returns BLOCK on the first non-allowed tool; subsequent tools aren't checked.

## Next Steps

- [Safety Policy](./safety) -- `blocked_tools` (deny-only) + content filters
- [Code Execution Policy](./code-execution) -- Sandbox controls for `shell_exec`-style tools
- [MCP Server Allowlist](./mcp-server-allowlist) -- Same shape, but for MCP servers
- [Prompt Allowlist](./prompt-allowlist) -- Same shape, but for prompt templates
- [Policy Categories & Templates](../features/policy-categories) -- All categories


--------------------------------------------------------------------------------
# MCP Server Allowlist Policy
URL: https://waxell.ai/docs/observe/governance/mcp-server-allowlist
Description: Constrain which MCP servers an agent may register / connect to. Positive allowlist + always-deny list for the MCP ecosystem.
--------------------------------------------------------------------------------

# MCP Server Allowlist Policy

The `mcp-server-allowlist` policy category constrains which **MCP servers** an agent may register or connect to.

Today's MCP ecosystem lets an agent register arbitrary third-party servers -- a malicious or compromised tool catalog can be a serious attack surface. This category is the lever to constrain that.

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `allowed_servers` | string[] | `[]` | Positive allowlist of server names. Empty = no allowlist (all allowed unless in `blocked_servers`) |
| `blocked_servers` | string[] | `[]` | Always-deny list. Wins over allowlist |
| `action_on_violation` | string | `block` | Action on violation: `block` or `warn` |

## How It Works

The handler runs at **`before_workflow`**, **`mid_execution`**, and **`after_workflow`** -- on every phase, so a server attached mid-run is caught.

Reads `context.metadata["mcp_servers"]` -- a list of server names this run has connected to. The MCP manifest endpoint (or whichever subsystem registers servers) is responsible for populating this. Empty / missing list = no enforcement.

Semantics:

1. Empty `allowed_servers` -> only `blocked_servers` applies
2. Non-empty `allowed_servers` -> ONLY those servers are permitted
3. `blocked_servers` wins (deny overrides allow)

### Context Attributes Read

| Attribute | Phase | Purpose |
|-----------|-------|---------|
| `context.metadata["mcp_servers"]` | all | List of MCP server names attached to this run |

## Example Policy

### Strict allowlist for enterprise tenants

```json
{
  "name": "Approved MCP servers only",
  "category": "mcp-server-allowlist",
  "rules": {
    "allowed_servers": ["github-mcp", "jira-mcp", "internal-tools-mcp"],
    "blocked_servers": ["unverified-public-mcp"],
    "action_on_violation": "block"
  },
  "scope": {
    "agents": ["*"]
  },
  "enabled": true
}
```

### Deny-only mode

If you trust most MCP servers but want to block known-bad ones:

```json
{
  "rules": {
    "blocked_servers": ["typosquat-mcp", "scraper-mcp"],
    "action_on_violation": "block"
  }
}
```

## SDK Integration

```python

waxell.init()

# The MCP manifest endpoint should populate metadata["mcp_servers"]
# when an agent registers servers.

@waxell.observe(
    agent_name="dev-assistant",
    enforce_policy=True,
    metadata={"mcp_servers": ["github-mcp", "jira-mcp"]},
)
async def dev_agent(task: str) -> str:
    return await run(task)
```

## Observability

| Field | Example (BLOCK) |
|-------|---------|
| **Category** | `mcp-server-allowlist` |
| **Action** | `block` |
| **Reason** | `MCP server 'untrusted-mcp' is not on the allowlist` |
| **Metadata** | `{"blocked_server": "untrusted-mcp", "allowed_servers": [...], "phase": "before"}` |

## Common Gotchas

- **Requires `context.metadata["mcp_servers"]` to be populated.** If your MCP manifest endpoint doesn't write this field, the policy is a no-op. Empty / missing list = ALLOW.
- **Exact name match.** Use the canonical server name -- no globs or regex.
- **Empty allowlist = NO allowlist.** Same trap as the [Tool Allowlist](./tool-allowlist) -- setting `allowed_servers: []` does not block all servers.
- **Fires on every phase.** Cheap (set lookup), so the policy catches servers attached mid-run, not just at startup.
- **Three handlers, one module.** `mcp-server-allowlist`, [tool-allowlist](./tool-allowlist), and [prompt-allowlist](./prompt-allowlist) share the same `allowlists.py` handler module. Each is a separate category with separate rules.

## Next Steps

- [Tool Allowlist](./tool-allowlist) -- Positive list governance for tools
- [Prompt Allowlist](./prompt-allowlist) -- Positive list governance for prompt templates
- [Safety Policy](./safety) -- Generic blocked_tools + content filters
- [Code Execution Policy](./code-execution) -- Sandbox controls for executable tools
- [Policy Categories & Templates](../features/policy-categories) -- All categories


--------------------------------------------------------------------------------
# Prompt Allowlist Policy
URL: https://waxell.ai/docs/observe/governance/prompt-allowlist
Description: Constrain named system-prompt templates. Cross-references the prompt-management system; enterprise compliance lever for vetted prompts.
--------------------------------------------------------------------------------

# Prompt Allowlist Policy

The `prompt-allowlist` policy category constrains which **named system-prompt templates** are permitted for a run. Cross-references the prompt-management system (`Prompt` rows).

Use case: enterprise compliance where prompts are vetted assets. Tenants pin to a curated set of prompt names; ad-hoc prompts get flagged.

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `allowed_prompts` | string[] | `[]` | Allowlist of template names. Empty = all allowed |
| `blocked_prompts` | string[] | `[]` | Always-deny list. Wins over allowlist |
| `action_on_violation` | string | `block` | Action on violation: `block` or `warn` |

## How It Works

The handler runs at **`before_workflow`** only -- the prompt template is known at run start. `after_workflow` is a no-op.

Reads the prompt name from `context.prompt_name` first, then falls back to `context.metadata["prompt_name"]`. Absent / empty = no enforcement (ALLOW).

Semantics:

1. Empty `allowed_prompts` -> only `blocked_prompts` applies
2. Non-empty `allowed_prompts` -> ONLY those templates are permitted
3. `blocked_prompts` wins (deny overrides allow)

### Context Attributes Read

| Attribute | Phase | Purpose |
|-----------|-------|---------|
| `context.prompt_name` | before_workflow | Named template loaded by runtime |
| `context.metadata["prompt_name"]` | before_workflow | Fallback resolver |

## Example Policy

### Vetted-prompt enterprise mode

```json
{
  "name": "Vetted prompts only",
  "category": "prompt-allowlist",
  "rules": {
    "allowed_prompts": [
      "research-v3-vetted",
      "summarize-v2-legal",
      "code-review-v1-approved"
    ],
    "action_on_violation": "block"
  },
  "scope": {
    "agents": ["*"]
  },
  "enabled": true
}
```

### Block a deprecated prompt

```json
{
  "rules": {
    "blocked_prompts": ["research-v1-deprecated"],
    "action_on_violation": "warn"
  }
}
```

## SDK Integration

Set the prompt name via metadata when invoking the agent:

```python

waxell.init()

@waxell.observe(
    agent_name="research-assistant",
    enforce_policy=True,
    metadata={"prompt_name": "research-v3-vetted"},
)
async def research(query: str) -> str:
    return await run_with_prompt(query)
```

Or set `context.prompt_name` directly if your runtime layer exposes it as an attribute.

## Observability

| Field | Example (BLOCK) |
|-------|---------|
| **Category** | `prompt-allowlist` |
| **Action** | `block` |
| **Reason** | `Prompt 'ad-hoc-debug' is not on the allowlist` |
| **Metadata** | `{"blocked_prompt": "ad-hoc-debug", "allowed_prompts": [...]}` |

## Common Gotchas

- **No `prompt_name` on context = ALLOW.** This category is opt-in: runs that don't carry a named template are never blocked. Ensure your runtime populates `prompt_name` for every governed run.
- **Exact match.** `"research"` does not match `"research-v3-vetted"`. Use the full canonical name.
- **Empty allowlist = NO allowlist.** Same trap as the other allowlists. Setting `allowed_prompts: []` does not block all runs.
- **Only fires at `before_workflow`.** The prompt name is locked at run start; mid-run prompt swaps aren't checked. If your runtime supports prompt swapping mid-workflow, you need to re-evaluate at the swap point.
- **Cross-reference with `Prompt` rows.** The allowlist is just a string match -- it does not validate that the prompt actually exists in the prompt registry. Pair with deployment hygiene to keep the two in sync.
- **Three handlers, one module.** [tool-allowlist](./tool-allowlist), [mcp-server-allowlist](./mcp-server-allowlist), and `prompt-allowlist` share `allowlists.py`.

## Next Steps

- [Tool Allowlist](./tool-allowlist) -- Positive list governance for tools
- [MCP Server Allowlist](./mcp-server-allowlist) -- Positive list governance for MCP servers
- [Safety Policy](./safety) -- Content filters on prompts (PII, credentials)
- [Code Execution Policy](./code-execution) -- Sandbox controls for code-exec tools
- [Policy Categories & Templates](../features/policy-categories) -- All categories


--------------------------------------------------------------------------------
# Tool Argument Schema Policy
URL: https://waxell.ai/docs/observe/governance/tool-argument-schema
Description: OWASP LLM06a — validate tool-call arguments against JSON schemas before invocation. Companion to tool-allowlist (which gates tool names).
--------------------------------------------------------------------------------

# Tool Argument Schema Policy

The `tool-argument-schema` policy enforces **OWASP LLM06a** (excessive agency, sub-control "argument-level scoping"). It validates tool-call **arguments** against JSON schemas before invocation. This is the companion control to [Tool Allowlist](./tool-allowlist) — where tool-allowlist answers "which tools can this agent call?", this handler answers "with what arguments?".

The classic exfiltration shape: `delete_user` is on the allowlist, but `delete_user(role="admin")` from a customer-facing surface is not. This handler blocks the second case even though the tool name is permitted.

Uses the `jsonschema` library when installed; falls back to a minimal hand-rolled validator covering `type`, `required`, `enum`, `properties`, and `additionalProperties=false`.

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `schemas` | object | `{}` | Map of `tool_name → JSON Schema`. Each call to the tool must validate against the schema. |
| `require_schema_for_all_tools` | boolean | `false` | Strict mode — any tool call without a declared schema is rejected |
| `action_on_violation` | string | `"block"` | `"block"` or `"warn"` |

## How It Works

| Phase | Behavior |
|-------|----------|
| `before_workflow` | Validates any tool calls already queued at workflow start (rare — workflow restart with queued actions) |
| `mid_execution` | **Primary enforcement** — every queued tool call between LLM turns |
| `after_workflow` | (no-op — argument validation is a pre-call concern) |

Per pending action, the handler:

1. Skips non-tool-call entries (`kind` must be `"tool_call"`, `"tool"`, or absent).
2. Looks up `schemas[tool_name]`.
3. If no schema and `require_schema_for_all_tools=true` → BLOCK with `missing_schema`.
4. If schema present, validates `args` (or `arguments`) against it → BLOCK with `schema_violation` on failure.

### Context Attributes Read

| Attribute | Phase | Purpose |
|-----------|-------|---------|
| `context.pending_actions` | before, mid | List of `{"kind": "tool_call", "tool": "...", "args": {...}}` |

## Example Policy

Atlassian-style policy — cap fund transfers at $10k, restrict `delete_user` to non-admin targets, and require schemas for everything else:

```json
{
  "schemas": {
    "transfer_funds": {
      "type": "object",
      "required": ["amount", "recipient"],
      "properties": {
        "amount": {"type": "number", "maximum": 10000, "minimum": 0},
        "recipient": {"type": "string", "pattern": "^acct_[a-z0-9]+$"},
        "memo": {"type": "string", "maxLength": 200}
      },
      "additionalProperties": false
    },
    "delete_user": {
      "type": "object",
      "required": ["id"],
      "properties": {
        "id": {"type": "string"},
        "role": {"type": "string", "enum": ["user"]}
      },
      "additionalProperties": false
    },
    "send_email": {
      "type": "object",
      "required": ["to", "subject", "body"],
      "properties": {
        "to": {"type": "string", "format": "email"},
        "subject": {"type": "string", "maxLength": 200},
        "body": {"type": "string", "maxLength": 5000}
      }
    }
  },
  "require_schema_for_all_tools": true,
  "action_on_violation": "block"
}
```

## SDK Integration

```python


waxell.init()

@waxell.observe(agent_name="finance-bot", enforce_policy=True)
async def process(intent: str) -> str:
    # When the agent queues a tool call, the runtime adds
    # {"kind": "tool_call", "tool": "transfer_funds", "args": {...}}
    # to context.pending_actions. mid_execution validates each entry
    # against schemas["transfer_funds"] before the call is dispatched.
    return await run_finance_workflow(intent)
```

## Observability

| Field | Example |
|-------|---------|
| **Category** | `tool-argument-schema` |
| **Action** | `block` |
| **Reason** | "Tool 'transfer_funds' arguments failed schema validation: 25000 is greater than the maximum of 10000 (at amount)" |
| **Metadata** | `{"phase": "mid", "tool": "transfer_funds", "signal": "schema_violation", "error": "25000 is greater than the maximum of 10000 (at amount)", "owasp": "LLM06"}` |

Missing schema in strict mode:

| Field | Example |
|-------|---------|
| **Reason** | "Tool 'list_pages' has no declared argument schema and require_schema_for_all_tools is true." |
| **Metadata** | `{"signal": "missing_schema", "tool": "list_pages", "owasp": "LLM06"}` |

## Common Gotchas

- **Argument key is `args` OR `arguments`.** The handler reads `action_obj.get("args") or action_obj.get("arguments") or {}`. Both shapes work — the rest of the runtime tends to use `args`.
- **Tool name is `tool` OR `name`.** Same fallback — most pending-action records use `tool`.
- **Invalid policy schemas fail-open.** If your JSON Schema is itself malformed (`jsonschema.SchemaError`), the handler logs a warning and returns ALLOW — so a bad schema never blocks legitimate work. Validate your schemas separately.
- **The minimal fallback validator is limited.** Without `jsonschema` installed, only `type`, `required`, `enum`, `properties`, and `additionalProperties=false` are checked. No `$ref`, `oneOf`, `anyOf`, `pattern`, `format`, `minimum`/`maximum`, etc. Production deploys should always have `jsonschema` installed.
- **`require_schema_for_all_tools=true` blocks ANY undeclared tool.** Combine carefully with [Tool Allowlist](./tool-allowlist) — if the allowlist permits a tool but you forget to add its schema, this handler will still reject it.
- **Error messages are truncated to 200 characters.** Verbose `jsonschema` path/cause output would dominate the metadata payload; the policy keeps the human-readable message and the dotted path.
- **Empty `schemas` + `require_schema_for_all_tools=false` is effectively a no-op.** The policy must have either schemas configured or strict mode enabled to do anything.

## Next Steps

- [Policy Categories](../features/policy-categories) — All 49 categories
- [Tool Allowlist](./tool-allowlist) — Companion control on tool *names*
- [Agent Service Account Scope](./agent-service-account-scope) — Companion OWASP LLM06b control
- [Code Execution](./code-execution) — Special-case argument validation for code-running tools


--------------------------------------------------------------------------------
# Agent Service Account Scope Policy
URL: https://waxell.ai/docs/observe/governance/agent-service-account-scope
Description: OWASP LLM06b enforcement — agents acting on SaaS surfaces (Confluence, Jira, Salesforce) must use a least-privilege service account instead of the raw user session.
--------------------------------------------------------------------------------

# Agent Service Account Scope Policy

The `agent-service-account-scope` policy enforces **OWASP LLM06b** (excessive agency, sub-control "delegated authority"). When an agent acts on behalf of a user inside a SaaS app — Confluence, Jira, Salesforce, Workday — it must run under a least-privilege **service account** rather than inheriting the user's full session. Without this control, an agent transparently inherits every page its user can see, every project it can edit, every record it can delete.

The handler enforces on two axes:

1. **Identity check** — context must carry a service-account marker (`metadata.service_account`, `_sub_user_identity`, or `context.service_account`). No marker = raw-user run = block.
2. **Operation scope** — even when an SA is configured, sensitive operations (admin/destructive/cross-tenant) require the SA to declare allowed-scope membership.

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `require_service_account` | boolean | `true` | When true, the run must carry an SA marker |
| `service_account_field` | string | `"service_account"` | Key in `context.metadata` carrying the SA id |
| `restricted_operations` | string[] | `[]` | Operation-name globs (fnmatch) that require an SA |
| `restricted_workflow_types` | string[] | `[]` | Workflow types where an SA is mandatory regardless of `require_service_account` |
| `allowed_service_account_pattern` | string | `""` | Optional regex the SA id must match (e.g., `"^sa_.*"`) |
| `action_on_violation` | string | `"block"` | `"block"` or `"warn"` |

## How It Works

The handler runs at **before_workflow** (identity pre-check) and **mid_execution** (per-pending-action scope check). `after_workflow` is a no-op.

| Phase | What it checks | Action |
|-------|----------------|--------|
| `before_workflow` | `require_service_account` true or `workflow_type` in `restricted_workflow_types`, and no SA resolved | BLOCK or WARN |
| `before_workflow` | SA id doesn't match `allowed_service_account_pattern` regex | BLOCK or WARN |
| `mid_execution` | Each `context.pending_actions` entry matching `restricted_operations` (fnmatch) while no SA is set | BLOCK or WARN |
| `after_workflow` | (no-op) | ALLOW |

### Context Attributes Read

| Attribute | Phase | Purpose |
|-----------|-------|---------|
| `context.metadata[service_account_field]` | before, mid | Primary SA marker |
| `context.service_account` | before, mid | Direct attribute fallback |
| `context._sub_user_identity` | before, mid | Phase B sub-user marker (string or dict with `sub_user_id`/`id`) |
| `context.workflow_type` | before, mid | Matched against `restricted_workflow_types` |
| `context.pending_actions` | mid | List of `{"kind": "tool_call", "tool": "...", "args": {...}}` to gate |

## Example Policy

Atlassian Rovo-style policy — Jira/Confluence agent must run as a scoped SA, and admin/delete/bulk operations are gated:

```json
{
  "require_service_account": true,
  "service_account_field": "service_account",
  "restricted_operations": ["admin_*", "delete_*", "bulk_*", "transfer_*"],
  "restricted_workflow_types": ["administrative", "financial"],
  "allowed_service_account_pattern": "^sa_jira_",
  "action_on_violation": "block"
}
```

## SDK Integration

```python


waxell.init()

@waxell.observe(agent_name="jira-triage", enforce_policy=True)
async def triage_ticket(ticket_id: str) -> str:
    # before_workflow: blocks if no service_account in metadata
    return await process_ticket(ticket_id)


# Pass the SA marker via metadata
async with waxell.WaxellContext(
    agent_name="jira-triage",
    enforce_policy=True,
    metadata={"service_account": "sa_jira_triage_v1"},
) as ctx:
    await triage_ticket("PROJ-123")
```

## Observability

Evaluations appear in the Governance tab with:

| Field | Example |
|-------|---------|
| **Category** | `agent-service-account-scope` |
| **Action** | `block` |
| **Reason** | "No service account configured — agent would inherit the user's full permissions on this surface." |
| **Metadata** | `{"signal": "no_service_account", "workflow_type": "administrative", "owasp": "LLM06"}` |

Restricted-operation block:

| Field | Example |
|-------|---------|
| **Action** | `block` |
| **Reason** | "Operation 'delete_user' requires a scoped service account; the agent is running as the raw user." |
| **Metadata** | `{"signal": "restricted_operation_without_sa", "operation": "delete_user", "owasp": "LLM06"}` |

## Common Gotchas

- **`restricted_operations` uses fnmatch globs, not regex.** Use `admin_*` to match `admin_create`, `admin_delete` — not `^admin_.*`.
- **SA resolution falls through three sources.** `metadata[service_account_field]` → `context.service_account` → `context._sub_user_identity`. Setting any of them satisfies the identity check.
- **`_sub_user_identity` accepts a dict OR a string.** When it's a dict, the handler looks at `sub_user_id` then `id`. Other keys are ignored.
- **Empty `restricted_operations` skips the mid-execution check entirely.** The pre-run identity check still fires.
- **The pattern check only runs when an SA exists.** An SA that's missing entirely produces the `no_service_account` block, not a pattern mismatch.

## Next Steps

- [Policy Categories](../features/policy-categories) — All 49 categories
- [Tool Argument Schema](./tool-argument-schema) — Companion control for tool *arguments*
- [Identity Policy](./identity) — Broader identity / sub-user enforcement
- [Delegation Policy](./delegation) — Multi-agent trust boundaries


--------------------------------------------------------------------------------
# Privacy Policy
URL: https://waxell.ai/docs/observe/governance/privacy
Description: Data privacy governance -- consent requirements, data residency enforcement, purpose limitations, and data minimization for GDPR, HIPAA, and custom privacy frameworks.
--------------------------------------------------------------------------------

# Privacy Policy

The `privacy` policy category enforces **data protection governance** -- consent requirements, data residency restrictions, purpose limitations, and data minimization. Unlike behavioral categories that inspect what an agent produces, privacy governs how data flows through the agent: where it runs, why it processes data, and whether the user consented.

Use it to enforce GDPR, HIPAA, or custom privacy frameworks that require consent tokens, restrict data processing to specific regions, and limit data usage to declared purposes.

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `require_consent` | boolean | `false` | Whether a consent token must be present on the context before the agent runs. |
| `consent_token_field` | string | `"consent_token"` | Name of the context attribute to check for the consent token. Change to `gdpr_consent`, `hipaa_auth`, etc. for custom flows. |
| `data_residency` | string[] | `[]` | Allowed execution regions. Exact string match (e.g. `"us-east-1"`, `"eu-west-1"`). Empty list means any region is allowed. |
| `purpose_limitation` | string[] | `[]` | Allowed data processing purposes. Exact string match. Empty list means any purpose is allowed. |
| `data_minimization` | boolean | `true` | When `true`, the after_workflow phase flags over-collection when the data purpose exceeds declared limitations. |
| `retention_by_type` | object | `&#123;"pii": 30, "logs": 90, "analytics": 365&#125;` | Retention periods in days per data type. Tagged in after_workflow metadata as an informational signal -- the handler does not enforce deletion. |
| `action_on_violation` | string | `"block"` | `"block"` raises a PolicyViolationError. `"warn"` records a warning and lets the agent continue. |

## How It Works

The privacy handler runs at all three phases and reads context data populated by `ctx.set_privacy_context()`.

### Enforcement Phases

| Phase | What it checks | Action on violation |
|-------|---------------|-------------------|
| **before_workflow** | Consent token present (if required); execution region in allowed list | BLOCK or WARN per `action_on_violation` |
| **mid_execution** | Data purpose in allowed purposes list | BLOCK or WARN per `action_on_violation` |
| **after_workflow** | Re-audit consent + residency; flag over-collection; tag retention metadata | WARN only |

### before_workflow — Pre-Flight Privacy Check

The handler validates pre-conditions before any agent work begins:

1. If `require_consent` is `true`: is `context.consent_token` (or the configured field) truthy? If not → violation
2. If `data_residency` is non-empty: is `context.execution_region` in the list? If not → violation

Both checks use `action_on_violation` to determine whether to block or warn.

### mid_execution — Purpose Limitation Check

During tool calls and steps, the handler checks the data purpose:

1. If `purpose_limitation` is non-empty: is `context.data_purpose` in the list? If not → violation

The purpose check only triggers if `context.data_purpose` has been set and `purpose_limitation` is non-empty. An empty purpose or an empty purpose list both result in ALLOW.

### after_workflow — Audit and Metadata

After the agent completes, the handler performs an audit:

- Re-checks consent (if required) and residency against configured rules
- If `data_minimization` is `true` and the data purpose exceeds the declared limitation, adds a warning
- Always tags `retention_by_type` in metadata when configured

After_workflow never blocks. It is an audit-only phase.

## Consent Token Matching

The handler calls `getattr(context, consent_token_field, None)` and checks truthiness:

| Consent Token Value | Passes? | Why |
|--------------------|---------|-----|
| `"usr_consent_abc123"` | Yes | Non-empty string is truthy |
| `""` (empty string) | No | Empty string is falsy |
| `None` | No | None is falsy |
| `0` | No | Zero is falsy |
| `"0"` | Yes | Non-empty string is truthy |
| `False` | No | False is falsy |

**Custom consent fields**: Set `consent_token_field` to the attribute name your agent sets. The default `"consent_token"` is checked via `getattr(context, "consent_token", None)`. If your agent sets `gdpr_consent` instead, update the field to match:

```json
{
  "require_consent": true,
  "consent_token_field": "gdpr_consent"
}
```

The agent must then call:
```python
ctx.set_privacy_context(consent_token="usr_consent_abc123")
```

Note: `set_privacy_context()` always stores the value under the `consent_token` key internally. For custom field names, the controlplane propagates the value using the `privacy_context` dict and the configured `consent_token_field`. The default field name `"consent_token"` maps directly.

## Data Residency Matching

The handler compares `context.execution_region` against the `data_residency` list using exact string matching:

| execution_region | data_residency | Passes? |
|-----------------|----------------|---------|
| `"us-east-1"` | `["us-east-1", "eu-west-1"]` | Yes |
| `"eu-west-1"` | `["us-east-1", "eu-west-1"]` | Yes |
| `"ap-southeast-1"` | `["us-east-1", "eu-west-1"]` | No → BLOCK/WARN |
| `""` (empty) | `["us-east-1"]` | ALLOW (empty region skips check) |
| anything | `[]` (empty list) | ALLOW (no restriction) |

Common region values: `"us-east-1"`, `"us-west-2"`, `"eu-west-1"`, `"eu-central-1"`, `"ap-southeast-1"`. You may use any string that matches your infrastructure's region naming.

## Purpose Limitation Matching

The handler compares `context.data_purpose` against the `purpose_limitation` list using exact string matching:

| data_purpose | purpose_limitation | Passes? |
|-------------|-------------------|---------|
| `"customer_support"` | `["customer_support", "analytics"]` | Yes |
| `"analytics"` | `["customer_support", "analytics"]` | Yes |
| `"marketing"` | `["customer_support", "analytics"]` | No → BLOCK/WARN |
| `""` (empty) | `["customer_support"]` | ALLOW (empty purpose skips check) |
| anything | `[]` (empty list) | ALLOW (no restriction) |

Common purpose values: `"customer_support"`, `"analytics"`, `"audit"`, `"marketing"`, `"research"`, `"internal"`.

## Example Policies

### GDPR-Compliant

Require consent, EU residency only, purpose limited to support and analytics:

```json
{
  "require_consent": true,
  "consent_token_field": "gdpr_consent",
  "data_residency": ["eu-west-1", "eu-central-1"],
  "purpose_limitation": ["customer_support", "analytics", "audit"],
  "data_minimization": true,
  "retention_by_type": {"pii": 30, "logs": 90, "analytics": 365},
  "action_on_violation": "block"
}
```

### HIPAA-Style

Consent required, US regions only, strict blocking:

```json
{
  "require_consent": true,
  "consent_token_field": "hipaa_auth",
  "data_residency": ["us-east-1", "us-west-2"],
  "purpose_limitation": ["patient_care", "billing", "audit"],
  "data_minimization": true,
  "retention_by_type": {"pii": 2555, "logs": 2555, "analytics": 365},
  "action_on_violation": "block"
}
```

### Analytics-Only

No consent requirement, purpose limited to analytics, warn on violation:

```json
{
  "require_consent": false,
  "data_residency": ["us-east-1", "eu-west-1"],
  "purpose_limitation": ["analytics"],
  "data_minimization": true,
  "retention_by_type": {"analytics": 365},
  "action_on_violation": "warn"
}
```

### Permissive with Audit

No restrictions, just tag retention metadata and track data minimization:

```json
{
  "require_consent": false,
  "data_residency": [],
  "purpose_limitation": [],
  "data_minimization": true,
  "retention_by_type": {"pii": 30, "logs": 90, "analytics": 365},
  "action_on_violation": "warn"
}
```

## SDK Integration

### Setting Privacy Context

Call `ctx.set_privacy_context()` early in the agent -- before the first tool call -- to ensure the before_workflow and mid_execution phases have the correct data:

```python

from waxell_observe.errors import PolicyViolationError

try:
    async with waxell.WaxellContext(
        agent_name="data-agent",
        workflow_name="data-processing",
        inputs={"query": query},
        enforce_policy=True,
    ) as ctx:

        # Set privacy context before any work
        ctx.set_privacy_context(
            consent_token="usr_consent_abc123",
            execution_region="us-east-1",
            data_purpose="customer_support",
        )

        # before_workflow checked consent + region before this point
        # mid_execution will check data_purpose on the next tool call

        result = await process_data(query)

        # Tool call triggers mid_execution governance
        ctx.record_tool_call(
            name="data_access",
            tool_type="database",
            input={"query": query, "purpose": "customer_support"},
            output={"result": result},
        )

        ctx.set_result({"result": result})

except PolicyViolationError as e:
    print(f"Privacy violation: {e}")
    # e.g. "Consent token required but not provided (field: 'consent_token')"
    # e.g. "Execution region 'ap-southeast-1' not in allowed residency list"
    # e.g. "Data purpose 'marketing' not in allowed purposes"
```

### set_privacy_context Parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `consent_token` | str | `""` | User consent token or identifier. Empty string is ignored (falsy = no consent). |
| `execution_region` | str | `""` | Data processing region (e.g. `"us-east-1"`). Empty string skips residency check. |
| `data_purpose` | str | `""` | Purpose of data processing (e.g. `"analytics"`). Empty string skips purpose check. |

Only non-empty values are stored. Call `set_privacy_context()` multiple times to update individual fields.

## Enforcement Flow

```
Agent starts (WaxellContext.__aenter__)
    │
    └── before_workflow governance runs
        │
        ├── require_consent=true?
        │   └── context.consent_token falsy? → violation
        │       ├── action=block → BLOCK (PolicyViolationError)
        │       └── action=warn → WARN (agent continues)
        │
        ├── data_residency non-empty?
        │   └── context.execution_region not in list? → violation
        │       ├── action=block → BLOCK
        │       └── action=warn → WARN
        │
        └── All checks pass → ALLOW (store rules for mid_execution)

During execution (after tool calls):
    │
    └── mid_execution governance runs
        │
        └── purpose_limitation non-empty?
            └── context.data_purpose set and not in list? → violation
                ├── action=block → BLOCK
                └── action=warn → WARN

Agent exits (WaxellContext.__aexit__):
    │
    └── after_workflow governance runs
        │
        ├── Re-audit consent (WARN only if missing)
        ├── Re-audit region (WARN only if non-compliant)
        ├── data_minimization=true? → flag over-collection if purpose exceeded
        └── Tag retention_by_type in metadata
```

## Creating via Dashboard

1. Navigate to **Governance > Policies**
2. Click **New Policy**
3. Select category **Privacy**
4. Set `require_consent`, `consent_token_field`, `data_residency`, and `purpose_limitation`
5. Configure `data_minimization` and `retention_by_type`
6. Set `action_on_violation` to `block` or `warn`
7. Set scope to target specific agents (e.g., `privacy-data-agent`)
8. Enable

## Creating via API

```bash
# Create a GDPR-compliant privacy policy
curl -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  https://acme.waxell.dev/waxell/v1/policies/ \
  -d '{
    "name": "GDPR Data Privacy",
    "category": "privacy",
    "rules": {
      "require_consent": true,
      "consent_token_field": "consent_token",
      "data_residency": ["eu-west-1", "eu-central-1"],
      "purpose_limitation": ["customer_support", "analytics", "audit"],
      "data_minimization": true,
      "retention_by_type": {"pii": 30, "logs": 90, "analytics": 365},
      "action_on_violation": "block"
    },
    "scope": {
      "agents": ["data-agent"]
    },
    "enabled": true
  }'
```

## Observability

### Governance Tab

Privacy evaluations appear with phase information:

**before_workflow (allowed):**

| Field | Example |
|-------|---------|
| **Phase** | `before_workflow` |
| **Action** | `allow` |
| **Reason** | "Privacy rules stored for enforcement" |

**before_workflow (blocked — missing consent):**

| Field | Example |
|-------|---------|
| **Phase** | `before_workflow` |
| **Action** | `block` |
| **Reason** | "Consent token required but not provided (field: 'consent_token')" |
| **Metadata** | `&#123;"missing_field": "consent_token", "require_consent": true&#125;` |

**before_workflow (blocked — wrong region):**

| Field | Example |
|-------|---------|
| **Reason** | "Execution region 'ap-southeast-1' not in allowed residency list" |
| **Metadata** | `&#123;"execution_region": "ap-southeast-1", "allowed_regions": ["us-east-1", "eu-west-1"]&#125;` |

**mid_execution (blocked — wrong purpose):**

| Field | Example |
|-------|---------|
| **Phase** | `mid_execution` |
| **Reason** | "Data purpose 'marketing' not in allowed purposes" |
| **Metadata** | `&#123;"data_purpose": "marketing", "allowed_purposes": ["customer_support", "analytics"]&#125;` |

**after_workflow (allowed, with retention metadata):**

| Field | Example |
|-------|---------|
| **Phase** | `after_workflow` |
| **Action** | `allow` |
| **Reason** | "Privacy audit passed" |
| **Metadata** | `&#123;"retention_by_type": &#123;"pii": 30, "logs": 90, "analytics": 365&#125;, "data_minimization": true, "execution_region": "us-east-1"&#125;` |

## Combining with Other Policies

Privacy pairs naturally with other governance categories:

- **Compliance** -- Use a compliance policy that requires `privacy` as a sibling policy to enforce GDPR or HIPAA at the framework level.
- **Audit** -- Pair with an audit policy to log all data access with retention tracking.
- **Content** -- Pair with a content policy that has `pii_detection.enabled: true` to catch PII in outputs.
- **Identity** -- Pair with an identity policy for AI disclosure requirements when processing personal data.

## Common Gotchas

1. **`require_consent: false` (default) means consent is never checked.** If you want to enforce consent, you must explicitly set `require_consent: true`. Leaving it at the default means the agent can run without any consent token.

2. **Empty `data_residency` means any region is allowed.** An empty list is the default permissive state. Add at least one region to create a restriction.

3. **Empty `purpose_limitation` means any purpose is allowed.** Same as residency -- add at least one purpose to create a restriction.

4. **`consent_token_field` must match exactly what the agent sets.** If the policy uses `gdpr_consent` but the agent calls `set_privacy_context(consent_token="...")`, the handler looks for `gdpr_consent` on the context, finds nothing, and blocks. The `consent_token` parameter in `set_privacy_context()` always stores under `consent_token` in the SDK.

5. **Retention metadata is informational only.** The handler tags `retention_by_type` in after_workflow metadata but does not enforce deletion. Use your data pipeline's retention policies to actually delete data.

6. **`data_minimization` only flags in after_workflow -- it never blocks.** It generates a WARN when the data purpose exceeds declared limitations, but the agent always continues. Use `action_on_violation` on mid_execution to block unauthorized purposes.

7. **Mid_execution purpose check requires `data_purpose` to be set before tool calls.** If you call `set_privacy_context(data_purpose="analytics")` after the first tool call, that tool call's mid_execution check sees an empty purpose and passes. Set the full privacy context at the beginning of the agent run.

8. **Region check skips when `execution_region` is empty.** The handler checks `if execution_region and execution_region not in residency`. An empty `execution_region` is not checked against the `data_residency` list. Always set the region if residency enforcement matters.

## Next Steps

- [Policy & Governance](../features/governance) -- How policy enforcement works
- [Compliance Policy](./compliance) -- Requires privacy as a sibling for GDPR/HIPAA profiles
- [Delegation Policy](./delegation) -- Multi-agent trust governance
- [Content Policy](./content) -- PII detection in agent outputs
- [Policy Categories & Templates](../features/policy-categories) -- All 26 categories


--------------------------------------------------------------------------------
# Identity Policy
URL: https://waxell.ai/docs/observe/governance/identity
Description: Agent identity governance -- enforce AI disclosure requirements and prevent impersonation of humans or professional roles in agent outputs.
--------------------------------------------------------------------------------

# Identity Policy

The `identity` policy category governs how agents represent themselves. It enforces two distinct rules:

1. **AI disclosure** -- every agent response must include a configurable disclosure text identifying it as AI-generated
2. **Impersonation prevention** -- agent outputs must not match patterns that claim human identity or professional roles

Use this policy when agents interact with end-users and transparency is required by your product standards, terms of service, or applicable regulations.

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `require_ai_disclosure` | boolean | `true` | Require the disclosure text to appear in the agent's output |
| `disclosure_text` | string | `"This response was generated by an AI assistant."` | Exact text that must appear in the output (substring match, case-insensitive) |
| `disclosure_position` | string | `"prepend"` | Hint for placement: `"prepend"`, `"append"`, or `"footer"`. Used in warning metadata only -- the handler does not inject text |
| `prevent_impersonation` | boolean | `true` | Block outputs that match configured impersonation patterns |
| `impersonation_patterns` | string[] | see defaults | Substring patterns that indicate the agent is claiming to be human |
| `agent_identity_header` | boolean | `true` | Tag governance metadata with an identity header marker for trace inspection |

### Default Impersonation Patterns

```
"I am a human"
"I am not an AI"
"I am not a bot"
"I am a real person"
```

Any string in `impersonation_patterns` is checked as a **case-insensitive substring** against the output text.

## How It Works

The identity handler runs at **all three enforcement phases**, with different checks at each:

| Phase | What It Checks | Violation Action |
|-------|---------------|-----------------|
| `before_workflow` | Stores identity rules into context | Always ALLOW (setup only) |
| `mid_execution` | Scans `intermediate_outputs` for impersonation patterns | BLOCK on match |
| `after_workflow` | Checks `output_text` for impersonation (BLOCK) and AI disclosure (WARN) |

### Enforcement Detail

**mid_execution**: Each time `ctx.record_intermediate_output(output=...)` is called, the handler scans the accumulated list of intermediate outputs. If any output matches an impersonation pattern, the agent is blocked immediately.

**after_workflow**: Runs two checks in sequence:
1. Impersonation check on `output_text` -- BLOCK if matched (same patterns, highest priority)
2. Disclosure check on `output_text` -- WARN if `disclosure_text` is not found as a substring

Note: disclosure violations produce WARN, not BLOCK. The agent completes but a governance incident is recorded. Impersonation violations always produce BLOCK.

There is no configurable `action_on_violation` field -- impersonation is always blocked, and disclosure is always warned.

## Pattern Matching

| Pattern in Config | Output Text | Result |
|------------------|-------------|--------|
| `"I am a human"` | "Hello! I am a human customer rep with 5 years experience." | BLOCK -- substring match (case-insensitive) |
| `"I am not an AI"` | "I am not an ai, I promise." | BLOCK -- case-insensitive match |
| `"I am a human"` | "I am an AI assistant here to help." | ALLOW -- pattern not found |
| `"I am a doctor"` | "As your doctor, I recommend..." | BLOCK -- matches custom pattern |

Disclosure matching works the same way:

| Configured `disclosure_text` | Output | Result |
|-----------------------------|--------|--------|
| `"This response was generated by an AI assistant."` | "...answer text...\n\nThis response was generated by an AI assistant." | ALLOW |
| `"This response was generated by an AI assistant."` | "...answer text only..." | WARN |
| `"AI-generated content"` | "Note: AI-generated content below." | ALLOW |

**Disclosure is Agent-Enforced, Not Auto-Injected**
The identity handler checks for disclosure text in the output -- it does not add it automatically. Your agent code (or your LLM prompt) must include the configured `disclosure_text` in every response. Use `ctx.set_output_text(text=...)` to tell the handler what text to check.


## SDK Integration

### Using the Context Manager

```python

from waxell_observe.errors import PolicyViolationError

waxell.init()

try:
    async with waxell.WaxellContext(
        agent_name="support-agent",
        enforce_policy=True,
    ) as ctx:
        # Generate intermediate reasoning
        reasoning = await think_about(query)

        # Record it -- mid_execution checks this for impersonation
        ctx.record_intermediate_output(output=reasoning)

        # Generate final response (must include disclosure text)
        response = await generate_response(query)
        response_with_disclosure = (
            response + "\n\nThis response was generated by an AI assistant."
        )

        # Set output text -- after_workflow checks this
        ctx.set_output_text(text=response_with_disclosure)

        ctx.set_result({"response": response_with_disclosure})

except PolicyViolationError as e:
    print(f"Identity block: {e}")
    # e.g. "Impersonation detected in intermediate output: matched pattern 'I am a human'"
```

### What Gets Checked

| SDK Call | What the Handler Reads | Phase |
|----------|----------------------|-------|
| `ctx.record_intermediate_output(output=...)` | `context.intermediate_outputs` | mid_execution |
| `ctx.set_output_text(text=...)` | `context.output_text` | after_workflow |

If `set_output_text` is never called, the handler falls back to `str(result)` from `ctx.set_result()`.

## Example Policies

### Strict AI Disclosure with Human Impersonation Block

```json
{
  "require_ai_disclosure": true,
  "disclosure_text": "This response was generated by an AI assistant.",
  "disclosure_position": "append",
  "prevent_impersonation": true,
  "impersonation_patterns": [
    "I am a human",
    "I am not an AI",
    "I am not a bot",
    "I am a real person"
  ],
  "agent_identity_header": true
}
```

### Medical / Legal Role Impersonation Prevention

```json
{
  "require_ai_disclosure": true,
  "disclosure_text": "This information is AI-generated and not professional advice.",
  "disclosure_position": "footer",
  "prevent_impersonation": true,
  "impersonation_patterns": [
    "I am a doctor",
    "I am a lawyer",
    "I am a licensed",
    "As your physician",
    "As your attorney",
    "I am a human",
    "I am not an AI"
  ],
  "agent_identity_header": true
}
```

### Disclosure Only (No Impersonation Check)

```json
{
  "require_ai_disclosure": true,
  "disclosure_text": "Generated by AI.",
  "disclosure_position": "footer",
  "prevent_impersonation": false,
  "impersonation_patterns": [],
  "agent_identity_header": false
}
```

## Enforcement Flow

```
before_workflow
    │
    └── Store rules into context._identity_rules → ALLOW

mid_execution (runs each time record_intermediate_output() is called)
    │
    ├── prevent_impersonation disabled? → ALLOW
    ├── impersonation_patterns empty? → ALLOW
    └── Scan each intermediate output for each pattern
        ├── Pattern found? → BLOCK
        └── No match → ALLOW

after_workflow
    │
    ├── Check output_text for impersonation (prevent_impersonation=true)
    │   └── Pattern found? → BLOCK (highest priority)
    │
    ├── Check output_text for disclosure text (require_ai_disclosure=true)
    │   ├── disclosure_text not found in output? → WARN
    │   └── Found → continue
    │
    ├── Tag metadata with agent_identity_header (if enabled)
    │
    └── Any warnings? → WARN result
        No warnings → ALLOW result
```

## Creating via Dashboard

1. Navigate to **Governance > Policies**
2. Click **New Policy**
3. Select category **Identity**
4. Configure `disclosure_text` to match what your prompts produce
5. Add impersonation patterns appropriate for your use case
6. Set scope to target specific agents (e.g., `support-agent`)
7. Enable

## Creating via API

```bash
curl -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  https://acme.waxell.dev/waxell/v1/policies/ \
  -d '{
    "name": "AI Disclosure Policy",
    "category": "identity",
    "rules": {
      "require_ai_disclosure": true,
      "disclosure_text": "This response was generated by an AI assistant.",
      "disclosure_position": "append",
      "prevent_impersonation": true,
      "impersonation_patterns": [
        "I am a human",
        "I am not an AI",
        "I am not a bot",
        "I am a real person"
      ],
      "agent_identity_header": true
    },
    "scope": {
      "agents": ["support-agent"]
    },
    "enabled": true
  }'
```

## Observability

### Governance Tab

Identity evaluations appear with:

| Field | Example |
|-------|---------|
| **Phase** | `mid_execution` or `after_workflow` |
| **Action** | `allow`, `warn`, or `block` |
| **Category** | `identity` |
| **Reason** | "Identity audit passed" |

For impersonation blocks:

| Field | Example |
|-------|---------|
| **Reason** | "Impersonation detected in final output: matched pattern 'I am a human'" |
| **Metadata** | `&#123;"matched_pattern": "I am a human", "output_preview": "...first 200 chars..."&#125;` |

For disclosure warnings:

| Field | Example |
|-------|---------|
| **Reason** | "AI disclosure text not found in output" |
| **Metadata** | `&#123;"expected_disclosure": "This response was generated by an AI assistant.", "disclosure_position": "append"&#125;` |

## Common Gotchas

1. **The handler does not inject disclosure text.** It only checks whether the text is present. Your LLM prompts must instruct the model to include the configured `disclosure_text` verbatim.

2. **Disclosure violations produce WARN, not BLOCK.** The agent is not stopped. Use impersonation prevention to block harmful outputs; use disclosure to audit compliance.

3. **`disclosure_position` is metadata only.** Setting it to `"append"` does not move text around -- it appears in the governance incident metadata as a hint for your team.

4. **`set_output_text` takes precedence over `set_result`.** If you call `ctx.set_output_text(text=...)`, the handler uses that. If you only call `ctx.set_result(...)`, the handler falls back to `str(result)`. Always call `set_output_text` explicitly.

5. **Pattern matching is case-insensitive substring.** "I am a human" will match "i am a human assistant", "Hello, I am a human!", and any variation of case.

6. **Empty `impersonation_patterns` disables impersonation checks entirely.** Setting `prevent_impersonation: true` with no patterns produces no blocks.

7. **mid_execution fires on every intermediate output call.** If your agent produces many intermediate outputs, the first one that matches a pattern will block immediately. Subsequent outputs are not checked.

## Next Steps

- [Policy & Governance](../features/governance) -- How policy enforcement works
- [Memory Policy](./memory) -- Govern what agents store between turns
- [Content Policy](./content) -- Scan inputs and outputs for sensitive patterns
- [Communication Policy](./communication) -- Govern output delivery channels
- [Policy Categories & Templates](../features/policy-categories) -- All 26 categories


--------------------------------------------------------------------------------
# Memory Policy
URL: https://waxell.ai/docs/observe/governance/memory
Description: Session memory governance -- enforce session isolation, cap memory item counts, restrict forbidden memory types, and signal purge-on-completion for agent context stores.
--------------------------------------------------------------------------------

# Memory Policy

The `memory` policy category governs what agents store in their context and session memory. It controls four dimensions:

1. **Session isolation** -- prevent cross-session memory from being loaded into a new session
2. **Capacity limits** -- cap the total number of memory items an agent may accumulate
3. **Type restrictions** -- block or warn when agents attempt to store forbidden memory types (e.g., credentials, PII)
4. **Retention and purge** -- tag memory with a TTL and signal purge-on-completion to your memory backend

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `session_isolation` | boolean | `true` | Require all memory to be session-scoped. Violations occur when `cross_session_loaded` is true |
| `cross_session_memory` | boolean | `false` | Explicitly allow cross-session memory loading. Set to `true` to permit it alongside `session_isolation` |
| `memory_retention_hours` | integer | *(none)* | Maximum hours memory should be retained. Emitted in after_workflow metadata for your backend to honor |
| `max_memory_items` | integer | *(none)* | Maximum number of memory items the agent may hold. Checked at mid_execution and after_workflow |
| `forbidden_memory_types` | string[] | `[]` | Memory item types that must not be loaded or written (e.g. `["credentials", "pii", "financial"]`) |
| `purge_on_completion` | boolean | `false` | Signal that all memory should be purged when the workflow completes. Recorded in after_workflow metadata |
| `action_on_violation` | string | `"warn"` | Action on violation: `"warn"` (agent continues) or `"block"` (agent is stopped) |

## How It Works

The memory handler runs at **all three enforcement phases**:

| Phase | What It Checks | Reads From |
|-------|---------------|------------|
| `before_workflow` | Pre-loaded memory items for forbidden types; `cross_session_loaded` flag | `context.memory_items`, `context.cross_session_loaded` |
| `mid_execution` | Forbidden items, cross-session, memory writes, total item count | `context.memory_items`, `context.cross_session_loaded`, `context.memory_writes`, `context.memory_item_count` |
| `after_workflow` | Final item count; purge signal; write history audit | `context.memory_item_count`, `context.memory_writes` |

All violations respect `action_on_violation`: `"block"` raises `PolicyViolationError`, `"warn"` records a governance incident and lets the agent continue.

**before_workflow Fires Before set_memory_state()**
The `before_workflow` phase fires when the SDK creates the run -- **before** your code calls `set_memory_state()`. This means the forbidden item and cross-session checks at `before_workflow` run against empty state and pass vacuously. The full checks run at `mid_execution` (triggered by subsequent auto-instrumented events like LLM calls) and at `after_workflow` when the agent completes, by which point the actual memory state is available. For immediate enforcement, call `record_memory_write()` after setting state -- it triggers governance automatically.


### before_workflow

Two checks run before the agent starts work:

1. **Forbidden type check** -- scans `memory_items` list. Each item must have a `"type"` key. If any item's type appears in `forbidden_memory_types`, the violation fires.
2. **Cross-session check** -- if `session_isolation: true` and `cross_session_memory: false`, the handler blocks when `cross_session_loaded` is `true`.

### mid_execution

Runs after each memory write (`ctx.record_memory_write(...)`) is flushed to the server, or at the next auto-instrumented event (LLM call, step recording):

1. **Forbidden item check** -- scans `memory_items` for forbidden types (same check as before_workflow, but now with actual state from `set_memory_state()`)
2. **Cross-session check** -- if `session_isolation: true`, `cross_session_memory: false`, and `cross_session_loaded: true` → violation
3. **Write type check** -- scans accumulated `memory_writes`. If any write type appears in `forbidden_memory_types`, the violation fires
4. **Item count check** -- if `memory_item_count` exceeds `max_memory_items`, the violation fires

### after_workflow

Audit phase with no additional blocking (violations become WARN regardless of `action_on_violation`):

- If `purge_on_completion: true`, emits `purge_on_completion: true` and `memory_items_to_purge` in metadata
- If `memory_retention_hours` is set, emits `retention_hours` and `retention_ttl_seconds` in metadata
- Re-audits write history for any forbidden types that slipped through
- Re-checks final item count against `max_memory_items`

## Session Isolation Logic

| `session_isolation` | `cross_session_memory` | `cross_session_loaded` | Result |
|--------------------|-----------------------|-----------------------|--------|
| `true` | `false` | `false` | ALLOW -- no cross-session data |
| `true` | `false` | `true` | VIOLATION -- cross-session data not permitted |
| `true` | `true` | `true` | ALLOW -- cross-session explicitly permitted |
| `false` | *(any)* | *(any)* | ALLOW -- isolation not enforced |

The key combination is `session_isolation: true` + `cross_session_memory: false` + `cross_session_loaded: true` = violation.

## Forbidden Memory Types

Types are compared as **exact string matches** against the `"type"` field of each memory item and write:

| Configured Forbidden Types | Item/Write Type | Result |
|---------------------------|----------------|--------|
| `["credentials", "pii"]` | `"credentials"` | VIOLATION |
| `["credentials", "pii"]` | `"pii"` | VIOLATION |
| `["credentials", "pii"]` | `"preference"` | ALLOW |
| `["credentials", "pii"]` | `"fact"` | ALLOW |
| `[]` | *(any)* | ALLOW -- no forbidden types configured |

**Memory Item Schema and Key Mapping**
Memory items passed to `ctx.set_memory_state(memory_items=[...])` must use the `"type"` key for the handler to match them. For `record_memory_write()`, the SDK sends `"memory_type"` -- the handler checks both `write.get("memory_type")` and `write.get("type")` for backwards compatibility:

```python
ctx.set_memory_state(
    memory_items=[
        {"type": "preference", "content": "User prefers dark mode"},
        {"type": "fact", "content": "User timezone is UTC-5"},
    ],
    cross_session_loaded=False,
    memory_item_count=2,
)
```


## SDK Integration

### Using the Context Manager

```python

from waxell_observe.errors import PolicyViolationError

waxell.init()

try:
    async with waxell.WaxellContext(
        agent_name="personalization-agent",
        enforce_policy=True,
    ) as ctx:

        # Declare pre-loaded memory state
        ctx.set_memory_state(
            memory_items=[
                {"type": "preference", "content": "User prefers dark mode"},
                {"type": "fact", "content": "User is in UTC-5 timezone"},
            ],
            cross_session_loaded=False,  # True would violate session_isolation
            memory_item_count=2,
        )

        # Memory state is evaluated at mid_execution (triggered by the next
        # auto-instrumented event or record_memory_write call) and at
        # after_workflow when the agent completes.

        # ... agent logic ...

        # Record a memory write (triggers mid_execution check)
        ctx.record_memory_write(
            memory_type="preference",  # Must not be in forbidden_memory_types
            content="User prefers concise summaries",
        )

        ctx.set_result({"response": "Noted your preference."})

except PolicyViolationError as e:
    print(f"Memory block: {e}")
    # e.g. "Forbidden memory type 'credentials' found in pre-loaded memory"
    # e.g. "Cross-session memory loaded but session isolation is enabled"
    # e.g. "Memory item count (1500) exceeds limit (100)"
```

### SDK Call to Handler Mapping

| SDK Call | What the Handler Reads | Governance Trigger |
|----------|----------------------|-------|
| `ctx.set_memory_state(memory_items=[...])` | `context.memory_items` | Evaluated at next auto-instrumented event or after_workflow |
| `ctx.set_memory_state(cross_session_loaded=True)` | `context.cross_session_loaded` | Evaluated at next auto-instrumented event or after_workflow |
| `ctx.set_memory_state(memory_item_count=N)` | `context.memory_item_count` | Evaluated at next auto-instrumented event or after_workflow |
| `ctx.record_memory_write(memory_type=..., content=...)` | `context.memory_writes` | Triggers mid_execution governance immediately |

## Example Policies

### Strict Session Isolation

Block cross-session memory loading. Allow all memory types. No item cap.

```json
{
  "session_isolation": true,
  "cross_session_memory": false,
  "action_on_violation": "block"
}
```

### Type-Restricted Memory

Allow cross-session memory but forbid sensitive types. Warn on violation.

```json
{
  "session_isolation": false,
  "cross_session_memory": true,
  "forbidden_memory_types": ["credentials", "pii", "financial"],
  "action_on_violation": "warn"
}
```

### Capped Memory with Purge

Limit memory store size and purge at completion. Retention TTL is 24 hours.

```json
{
  "session_isolation": true,
  "cross_session_memory": false,
  "max_memory_items": 500,
  "memory_retention_hours": 24,
  "purge_on_completion": true,
  "action_on_violation": "block"
}
```

### Production Personalization Agent

Full configuration for a customer-facing personalization agent:

```json
{
  "session_isolation": true,
  "cross_session_memory": false,
  "memory_retention_hours": 168,
  "max_memory_items": 1000,
  "forbidden_memory_types": ["credentials", "pii", "financial", "health"],
  "purge_on_completion": false,
  "action_on_violation": "block"
}
```

## Enforcement Flow

```
before_workflow (fires BEFORE set_memory_state — state is empty)
    │
    ├── Scan memory_items for forbidden types → passes (no items yet)
    │
    ├── Check cross_session_loaded → passes (not set yet)
    │
    └── ALLOW (checks are vacuous at this phase)

set_memory_state() → evaluated at next governance checkpoint
    │
    └── mid_execution governance fires with full memory state:
        ├── Scan memory_items for forbidden types
        │   └── Forbidden type found? → action_on_violation (BLOCK or WARN)
        │
        ├── session_isolation=true AND cross_session_memory=false?
        │   └── cross_session_loaded=true? → action_on_violation (BLOCK or WARN)
        │
        └── No violations → ALLOW

mid_execution (runs after each record_memory_write() flush)
    │
    ├── Scan memory_writes for forbidden types
    │   └── Forbidden type found? → action_on_violation (BLOCK or WARN)
    │
    ├── max_memory_items configured?
    │   └── memory_item_count > max_memory_items? → action_on_violation (BLOCK or WARN)
    │
    └── No violations → ALLOW

after_workflow
    │
    ├── purge_on_completion=true?
    │   └── Emit purge signal in metadata (memory_items_to_purge)
    │
    ├── memory_retention_hours configured?
    │   └── Emit retention_hours and retention_ttl_seconds in metadata
    │
    ├── Re-audit memory_writes for forbidden types → WARN if found
    │
    ├── max_memory_items configured?
    │   └── Final count > limit? → WARN
    │
    └── No warnings → ALLOW
```

## Creating via Dashboard

1. Navigate to **Governance > Policies**
2. Click **New Policy**
3. Select category **Memory**
4. Configure session isolation and forbidden types for your use case
5. Set `action_on_violation` to `block` for strict enforcement or `warn` for audit mode
6. Set scope to target specific agents
7. Enable

## Creating via API

```bash
curl -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  https://acme.waxell.dev/waxell/v1/policies/ \
  -d '{
    "name": "Session Memory Policy",
    "category": "memory",
    "rules": {
      "session_isolation": true,
      "cross_session_memory": false,
      "memory_retention_hours": 24,
      "max_memory_items": 1000,
      "forbidden_memory_types": ["credentials", "pii", "financial"],
      "purge_on_completion": false,
      "action_on_violation": "block"
    },
    "scope": {
      "agents": ["personalization-agent"]
    },
    "enabled": true
  }'
```

## Observability

### Governance Tab

Memory evaluations appear with:

| Field | Example |
|-------|---------|
| **Phase** | `before_workflow`, `mid_execution`, or `after_workflow` |
| **Action** | `allow`, `warn`, or `block` |
| **Category** | `memory` |
| **Reason** | "Memory rules stored for enforcement" |

For forbidden type violations:

| Field | Example |
|-------|---------|
| **Reason** | "Forbidden memory type 'credentials' found in pre-loaded memory" |
| **Metadata** | `&#123;"forbidden_type": "credentials", "forbidden_types": ["credentials", "pii", "financial"]&#125;` |

For cross-session violations:

| Field | Example |
|-------|---------|
| **Reason** | "Cross-session memory loaded but session isolation is enabled" |
| **Metadata** | `&#123;"session_isolation": true, "cross_session_memory": false, "cross_session_loaded": true&#125;` |

For item count violations:

| Field | Example |
|-------|---------|
| **Reason** | "Memory item count (1500) exceeds limit (100)" |
| **Metadata** | `&#123;"memory_item_count": 1500, "limit": 100&#125;` |

For after_workflow purge signal:

| Field | Example |
|-------|---------|
| **Metadata** | `&#123;"purge_on_completion": true, "memory_items_to_purge": 12, "retention_hours": 24, "retention_ttl_seconds": 86400&#125;` |

## Common Gotchas

1. **`action_on_violation` applies to before_workflow and mid_execution only.** The after_workflow phase always produces WARN for residual violations -- it is an audit pass, not a blocking pass.

2. **The handler does not persist or delete memory.** It evaluates policy compliance and signals violations. Your application code is responsible for honoring the `purge_on_completion` signal and `memory_retention_hours` TTL.

3. **`memory_item_count` must be set explicitly.** The handler reads `context.memory_item_count` -- it does not count the items in `memory_items` automatically. Always pass `memory_item_count` to `set_memory_state()` to match the actual size of your memory store.

4. **Forbidden type matching is exact string equality.** `"PII"` and `"pii"` are different strings. Make sure your type strings use consistent casing.

5. **`session_isolation: true` with `cross_session_memory: true` is permissive.** Setting both to true means: "I care about session isolation in principle, but I explicitly permit cross-session loading." The violation only fires when `cross_session_memory: false` and `cross_session_loaded: true`.

6. **`forbidden_memory_types` checks both pre-loaded items and writes.** A type that is blocked at before_workflow (pre-loaded) is also blocked at mid_execution (write-time). Configure the list once and it applies everywhere.

7. **Empty `forbidden_memory_types` disables type checking entirely.** No items or writes will be flagged for type violations regardless of their content.

8. **`before_workflow` runs before `set_memory_state()` is called.** The SDK creates the run and fires `before_workflow` when the context is entered. Your code calls `set_memory_state()` inside the function body, which is after `before_workflow` has already completed. The full memory checks run at `mid_execution` (triggered by the next auto-instrumented event or `record_memory_write()` call) and at `after_workflow`.

9. **`record_memory_write()` triggers governance immediately, but `set_memory_state()` does not.** After recording a memory write, mid_execution governance fires automatically. Setting the initial memory state is evaluated at the next governance checkpoint (auto-instrumented LLM call, step recording, or workflow completion).

## Next Steps

- [Policy & Governance](../features/governance) -- How policy enforcement works
- [Identity Policy](./identity) -- Enforce AI disclosure and prevent impersonation
- [Privacy Policy](../features/policy-categories) -- PII and data purpose controls
- [Content Policy](./content) -- Scan inputs and outputs for sensitive patterns
- [Policy Categories & Templates](../features/policy-categories) -- All 26 categories


--------------------------------------------------------------------------------
# Compliance Policy
URL: https://waxell.ai/docs/observe/governance/compliance
Description: Meta-validator that checks required sibling policies are active and properly configured -- regulatory profile validation for HIPAA, SOC 2, PCI-DSS, GDPR, and custom compliance frameworks.
--------------------------------------------------------------------------------

# Compliance Policy

The `compliance` policy category is a **meta-validator** -- it doesn't inspect agent behavior. Instead, it verifies that other required policy categories are active and properly configured before letting the agent run.

Use it when you need to enforce regulatory frameworks (HIPAA, SOC 2, PCI-DSS, GDPR) that require a constellation of policies to be in place.

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `profile` | string | `"Custom"` | Label for the compliance framework: `HIPAA`, `SOC 2`, `PCI-DSS`, `GDPR`, or `Custom`. Used in reason strings only |
| `required_categories` | string[] | `[]` | Policy categories that must be active (e.g. `["audit", "content", "privacy"]`) |
| `required_rules` | object | `{}` | Per-category rule constraints (e.g. `{"audit": {"retention_days": 2555}}`) |
| `strict` | boolean | `true` | `true` blocks on violation, `false` warns and lets the agent continue |

## How It Works

The compliance handler runs at **before_workflow** only. It reads `active_policies` -- a dict of `{category: merged_rules}` that the policy manager injects before evaluation. This dict contains all policies that apply to the current context (matching agent name, user, group, etc.).

### Evaluation Steps

1. If `required_categories` is empty, return ALLOW immediately (nothing to check)
2. For each required category, check if it exists in `active_policies`
3. For present categories, validate each entry in `required_rules`:
   - **Nested dicts**: recursive key equality check (e.g. `pii_detection.enabled`)
   - **Numeric values**: `>=` comparison (actual must be >= required)
   - **All other types**: exact `!=` equality
4. Missing categories or rule violations produce BLOCK (strict) or WARN (non-strict)

### Rule Matching

| Required Rule | Active Rule | Match? | Why |
|--------------|-------------|--------|-----|
| `retention_days: 2555` | `retention_days: 2555` | Yes | Exact match |
| `retention_days: 2555` | `retention_days: 3650` | Yes | 3650 ≥ 2555 (numeric comparison) |
| `retention_days: 2555` | `retention_days: 90` | No | 90 &#60; 2555 |
| `log_inputs: true` | `log_inputs: true` | Yes | Exact match |
| `log_inputs: true` | `log_inputs: false` | No | false != true |
| `pii_detection: {enabled: true}` | `pii_detection: {enabled: true}` | Yes | Nested key match |
| `pii_detection: {enabled: true}` | `pii_detection: {enabled: false}` | No | Nested value mismatch |
| `pii_detection: {enabled: true}` | `pii_detection: "basic"` | No | Expected dict, got string |

**Numeric Rules Use >= Comparison**
A `retention_days` of 3650 satisfies a requirement of 2555. The compliance handler checks that the actual value meets or exceeds the required minimum. This means stricter configurations always pass.


## Profile Presets

The `profile` field is a label used in reason strings. Common profiles and their typical required categories:

| Profile | Typical Required Categories | Key Rules |
|---------|---------------------------|-----------|
| **HIPAA** | audit, content, privacy, data-access | `audit.retention_days >= 2555`, `content.pii_detection.enabled: true` |
| **SOC 2** | audit, safety, kill, operations | `audit.log_inputs: true`, `safety.max_steps` configured |
| **PCI-DSS** | audit, content, network, data-access | `audit.retention_days >= 365`, `network` restrictions |
| **GDPR** | privacy, content, audit | `privacy.require_consent: true`, `content.pii_detection.enabled: true` |
| **Custom** | (your choice) | (your rules) |

These are recommendations -- you configure the actual `required_categories` and `required_rules` to match your compliance needs.

## Example Policies

### HIPAA Compliance

Require audit, content, and privacy policies with specific settings:

```json
{
  "profile": "HIPAA",
  "required_categories": ["audit", "content", "privacy"],
  "required_rules": {
    "audit": {
      "log_inputs": true,
      "retention_days": 2555
    },
    "content": {
      "pii_detection": {
        "enabled": true
      }
    }
  },
  "strict": true
}
```

### SOC 2 Compliance

Require operational safety policies:

```json
{
  "profile": "SOC 2",
  "required_categories": ["audit", "safety", "kill", "operations"],
  "required_rules": {
    "audit": {
      "log_inputs": true
    }
  },
  "strict": true
}
```

### Custom Framework

Minimal compliance -- just ensure audit logging is active:

```json
{
  "profile": "Custom",
  "required_categories": ["audit"],
  "required_rules": {
    "audit": {
      "retention_days": 365
    }
  },
  "strict": false
}
```

## SDK Integration

### Using the Context Manager

```python

from waxell_observe.errors import PolicyViolationError

waxell.init()

try:
    async with waxell.WaxellContext(
        agent_name="auditor",
        enforce_policy=True,
    ) as ctx:
        # If compliance policy is active and required sibling
        # policies are missing → PolicyViolationError raised here
        # (before any agent work happens)

        result = await analyze_data(query)
        ctx.set_result(result)

except PolicyViolationError as e:
    print(f"Compliance block: {e}")
    # e.g. "Compliance profile 'HIPAA' not met: Missing categories: privacy"
```

### Using the Decorator

```python
@waxell.observe(
    agent_name="auditor",
    enforce_policy=True,
)
async def run_audit(query: str):
    # Compliance check happens before this function body runs
    return await analyze_data(query)
```

## Enforcement Flow

```
Agent starts (WaxellContext.__aenter__ or decorator entry)
    │
    └── before_workflow governance runs
        │
        ├── Compliance handler reads active_policies
        │
        ├── required_categories empty? → ALLOW (nothing to check)
        │
        ├── For each required category:
        │   ├── Category missing from active_policies? → violation
        │   └── Category present? → check required_rules
        │       ├── Nested dict? → recursive key check
        │       ├── Numeric? → actual >= required?
        │       └── Other? → exact equality
        │
        ├── Any violations?
        │   ├── strict=true → BLOCK (PolicyViolationError)
        │   └── strict=false → WARN (agent continues)
        │
        └── All checks pass → ALLOW
```

The compliance handler only runs at **before_workflow**. There is no mid-execution check. The `after_workflow` phase is a stub that always returns ALLOW with an attestation pass.

## Creating via Dashboard

1. Navigate to **Governance > Policies**
2. Click **New Policy**
3. Select category **Compliance**
4. Set the profile (HIPAA, SOC 2, etc.)
5. Configure required categories and required rules
6. Set scope to target specific agents (e.g., `compliance-agent`)
7. Enable

Then create the sibling policies that the compliance policy requires (audit, content, privacy, etc.) with the same agent scope.

## Creating via API

```bash
# Create the compliance policy
curl -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  https://acme.waxell.dev/waxell/v1/policies/ \
  -d '{
    "name": "HIPAA Compliance",
    "category": "compliance",
    "rules": {
      "profile": "HIPAA",
      "required_categories": ["audit", "content", "privacy"],
      "required_rules": {
        "audit": {"log_inputs": true, "retention_days": 2555},
        "content": {"pii_detection": {"enabled": true}}
      },
      "strict": true
    },
    "scope": {
      "agents": ["auditor"]
    },
    "enabled": true
  }'

# Create the required sibling policies
curl -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  https://acme.waxell.dev/waxell/v1/policies/ \
  -d '{
    "name": "Full Audit Logging",
    "category": "audit",
    "rules": {"log_inputs": true, "retention_days": 2555},
    "scope": {"agents": ["auditor"]},
    "enabled": true
  }'
```

## Observability

### Governance Tab

Compliance evaluations appear with:

| Field | Example |
|-------|---------|
| **Policy name** | HIPAA Compliance |
| **Action** | `allow`, `warn`, or `block` |
| **Category** | `compliance` |
| **Reason** | "Compliance profile 'HIPAA' requirements met" |
| **Metadata** | `{"profile": "HIPAA", "categories_verified": ["audit", "content", "privacy"]}` |

For violations:

| Field | Example |
|-------|---------|
| **Reason** | "Compliance profile 'HIPAA' not met: Missing categories: privacy \| Rule violations: audit.retention_days: 90 below required 2555" |
| **Metadata** | `{"profile": "HIPAA", "missing_categories": ["privacy"], "rule_violations": ["audit.retention_days: 90 below required 2555"]}` |

### After-Workflow Attestation

The `after_workflow` phase always returns ALLOW with `{"attestation": "pass"}`. This is a stub for future compliance attestation features.

## Common Gotchas

1. **Compliance doesn't monitor agent behavior.** It validates that the right policies exist. To govern what agents actually do, use the specific category policies (code-execution, communication, content, etc.).

2. **Empty `required_categories` means always ALLOW.** A compliance policy with no required categories is a no-op. Always specify at least one category to check.

3. **`strict: false` means warn-only.** The agent still runs even when required policies are missing. Use strict mode for production compliance enforcement.

4. **Numeric rules use `>=` comparison.** A `retention_days` of 3650 satisfies a requirement of 2555. Stricter configurations always pass.

5. **Multiple policies of the same category merge via `dict.update()`.** If two audit policies exist, their rules are merged and the last one wins for conflicting keys. Be careful with overlapping policies.

6. **`after_workflow` is a stub.** It always returns ALLOW with an attestation pass. Don't rely on it for enforcement.

7. **Only runs at `before_workflow`.** Compliance blocks happen before the agent does any work. There is no mid-execution compliance check.

8. **The `profile` field is just a label.** Setting `profile: "HIPAA"` doesn't automatically enforce HIPAA rules -- you must configure `required_categories` and `required_rules` yourself.

## Next Steps

- [Policy & Governance](../features/governance) -- How policy enforcement works
- [Code Execution Policy](./code-execution) -- Govern generated code
- [Communication Policy](./communication) -- Govern output channels
- [Context Management Policy](./context-management) -- Control conversation boundaries
- [Policy Categories & Templates](../features/policy-categories) -- All 26 categories


--------------------------------------------------------------------------------
# Context Management Policy
URL: https://waxell.ai/docs/observe/governance/context-management
--------------------------------------------------------------------------------

# Context Management Policy

The `context-management` policy category governs conversation length, context window usage, and session boundaries. Use it to prevent agents from running unbounded conversations that exhaust context windows or accumulate excessive costs.

## Rules

| Rule | Type | Description |
|------|------|-------------|
| `max_conversation_turns` | integer | Maximum user turns before action |
| `max_context_utilization_percent` | number | Context window fill percentage limit (1-100) |
| `max_message_count` | integer | Total messages in context before action |
| `warning_threshold_percent` | number | Percentage of limits at which to start warning (1-100) |
| `action_on_exceed` | string | `"warn"` or `"block"` when limits are exceeded |
| `max_session_duration_minutes` | integer | Maximum session duration in minutes |

## Example Policy

```json
{
  "name": "Context Window Guard",
  "category": "context-management",
  "rules": {
    "max_conversation_turns": 50,
    "max_context_utilization_percent": 85,
    "max_message_count": 200,
    "warning_threshold_percent": 70,
    "action_on_exceed": "warn"
  }
}
```

## Enforcement

Context management policies are evaluated during **mid-execution** governance checks. When the SDK flushes LLM calls or steps to the controlplane, the current conversation state is included and checked against active policies.

- **Warning threshold**: When conversation metrics reach the warning percentage of any limit, a `WARN` result is returned. The agent continues but the warning appears in the governance timeline.
- **Limit exceeded**: When metrics exceed any limit, the configured action (`warn` or `block`) is applied.
- **Block behavior**: If `action_on_exceed` is `"block"`, the SDK raises a `PolicyViolationError` that the agent can catch and handle (e.g., summarize and reset context, or end the session).

## Use Cases

- **Prevent context overflow**: Set `max_context_utilization_percent: 85` to warn before the context window fills up
- **Limit conversation length**: Set `max_conversation_turns: 50` to prevent unbounded conversations
- **Cost control**: Long conversations with many messages consume more tokens per call — limiting message count indirectly controls cost growth
- **Session boundaries**: Set `max_session_duration_minutes: 60` to enforce session time limits

## Combining with Other Policies

Context management works alongside other policy categories:

- **Budget policies** control total token/cost spend; context management controls per-conversation growth
- **Safety policies** limit total steps; context management specifically tracks conversation turns
- **Rate limit policies** control request frequency; context management controls conversation depth


--------------------------------------------------------------------------------
# Data Residency Policy
URL: https://waxell.ai/docs/observe/governance/data-residency
Description: Pin agent execution to approved regions for data-sovereignty compliance. Aligned to ISO 42001 A.8.4, GDPR, and EU AI Act.
--------------------------------------------------------------------------------

# Data Residency Policy

The `data-residency` policy category restricts agent execution to one or more **approved regions** for data-sovereignty compliance. Runs that report an `execution_region` outside the allowlist are blocked.

This is observe-side enforcement (post-hoc audit) by nature -- the actual *pinning* of execution to a region happens at the infrastructure layer (ECS task placement, region-locked endpoints). The policy fires when a run *reports* an execution region outside the allowed set, surfacing the violation in audit + alerting.

Aligned to **ISO 42001 A.8.4** (data residency control), **GDPR** (cross-border transfer restrictions), and **EU AI Act** (regional deployment obligations).

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `allowed_regions` | string[] | `[]` | Region identifiers (case-insensitive) where execution is permitted |
| `require_region_declaration` | boolean | `true` | Block runs that lack an `execution_region` declaration |
| `action_on_violation` | string | `"block"` | Action on violation: `"block"` or `"warn"` |

## How It Works

The `data-residency` handler runs at **before_workflow**, **mid_execution**, and **after_workflow**. It compares `context.execution_region` (case-insensitive) against the `allowed_regions` list.

### Decision Flow

1. **No `execution_region` declared** + `require_region_declaration = true` → `BLOCK` (signal: `no_region_declared`)
2. **No `execution_region` declared** + `require_region_declaration = false` → `ALLOW`
3. **Empty `allowed_regions`** + `require_region_declaration = true` → `BLOCK` (signal: `no_allowed_regions`, acts as a kill-switch)
4. **Empty `allowed_regions`** + `require_region_declaration = false` → `ALLOW`
5. **Region not in allowlist** → `BLOCK` (signal: `region_not_allowed`)
6. **Region in allowlist** → `ALLOW`

### Context Attributes Read

| Attribute | Phase | Purpose |
|-----------|-------|---------|
| `context.execution_region` | all | Region identifier reported by the runtime (e.g., `"eu-west-1"`) |

## Example Policy

### EU-Only Residency

```json
{
  "allowed_regions": ["eu-west-1", "eu-central-1", "eu-north-1"],
  "require_region_declaration": true,
  "action_on_violation": "block"
}
```

### Kill-Switch Mode

Empty allowlist + required declaration blocks all runs. Useful as a regional kill-switch during incident response:

```json
{
  "allowed_regions": [],
  "require_region_declaration": true,
  "action_on_violation": "block"
}
```

## SDK Integration

```python

waxell.init()

@waxell.observe(agent_name="eu-claims-agent", enforce_policy=True)
async def process_eu_claim(claim: dict) -> dict:
    return await adjudicate(claim)
```

The runtime is responsible for setting `context.execution_region`. For ECS deployments this typically comes from the AWS metadata service; for k8s it comes from the node label.

## Observability

| Field | Example |
|-------|---------|
| **Category** | `data-residency` |
| **Action** | `block` |
| **Reason** | "Execution region 'us-east-1' not in approved list ['eu-central-1', 'eu-west-1']." |
| **Metadata** | `{"signal": "region_not_allowed", "execution_region": "us-east-1", "allowed_regions": ["eu-central-1", "eu-west-1"], "iso_42001": "A.8.4"}` |

## Common Gotchas

1. **Region comparison is case-insensitive.** `"EU-West-1"`, `"eu-west-1"`, and `"Eu-West-1"` all match `["eu-west-1"]`. The handler lowercases both sides.

2. **Post-hoc enforcement only.** This policy *detects* runs in the wrong region; it does not *prevent* them from starting. Actual pinning requires infrastructure controls (ECS region constraints, region-locked endpoints, VPC peering rules). Pair this policy with infra-level region pins.

3. **Empty allowlist + required declaration = kill switch.** This is intentional -- it lets you flip residency off globally with a single config change. Don't accidentally ship an empty allowlist in production unless you mean it.

4. **`context.execution_region` must be populated by the runtime.** Custom runtimes that omit this attribute will trigger the `no_region_declared` block when `require_region_declaration = true`. Wire your runtime to set it from the AWS/k8s metadata.

5. **Same policy fires three times per run.** The handler runs at `before_workflow`, `mid_execution`, and `after_workflow`. Audit will show three entries for blocked runs -- this is intentional (detects mid-flight region migrations) but can be noisy.

6. **Cross-region telemetry shipping is NOT covered.** This policy only governs agent execution. If your telemetry pipeline ships logs from `eu-west-1` to a US-hosted Tempo, that data movement is outside this policy's scope -- govern it with `data-access` and infra controls.

## Next Steps

- [Privacy Policy](./privacy) -- PII detection (foundational to GDPR compliance)
- [Compliance Policy](./compliance) -- Meta-validator for ISO 42001 / GDPR / EU AI Act
- [Data Access Policy](./data-access) -- Source allowlists (often paired with regional restrictions)
- [Kill Switch Policy](./kill-switch) -- Emergency stop independent of region
- [Policy Categories](../features/policy-categories) -- All categories


--------------------------------------------------------------------------------
# Data Erasure Policy
URL: https://waxell.ai/docs/observe/governance/data-erasure
Description: Enforce GDPR Art-17 "Right to Erasure" and CCPA right-to-delete SLAs. Block processing and writes for subjects with pending erasure requests.
--------------------------------------------------------------------------------

# Data Erasure Policy

The `data-erasure` policy category enforces **pending erasure requests** as required by GDPR Art-17 ("Right to Erasure") and CCPA §1798.105 (Right to Delete). When an end-user has filed an erasure request:

- Agents must NOT continue to process that subject's data
- New memory writes referencing the subject must be blocked
- The erasure must complete within the regulatory SLA (default 30 days)

This handler is the bridge between `data-access` (which enforces source allowlists but knows nothing about pending requests) and `memory` (which writes/reads but doesn't track deadlines).

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `max_pending_days` | integer | `30` | Hard SLA -- erasure must complete within N days of request (GDPR default 30) |
| `block_processing_for_subjects` | boolean | `true` | Block any agent run scoped to a subject with a pending request |
| `block_writes_for_subjects` | boolean | `true` | Block new memory writes referencing a subject with a pending request |
| `warn_threshold_days` | integer | `25` | Warn when a request has been pending more than N days (approaching SLA) |
| `action_on_violation` | string | `"block"` | Action on violation: `"block"` or `"warn"` |

## How It Works

The `data-erasure` handler runs at **before_workflow**, **mid_execution**, and **after_workflow**. It performs four checks in order:

1. **SLA check** -- any pending request past `max_pending_days` violates the policy regardless of who the current run is for
2. **Run-scoped check** -- if this run's subject(s) overlap with pending requests, refuse to process
3. **Write-scoped check** -- block memory writes whose string repr contains a pending-request subject id
4. **Warn check** -- warn on requests approaching SLA so the tenant can prioritize

### Context Attributes Read

| Attribute | Phase | Purpose |
|-----------|-------|---------|
| `context.metadata["erasure_requests"]` | all | List of `{sub_user_id, requested_at}` or `{user_id, requested_at}` (tenant-populated) |
| `context._sub_user_identity` / `context.sub_user_identity` | all | Resolve the run's subject (preferred over `user_id`) |
| `context.user_id` | all | Fallback subject identifier |
| `context.memory_writes` | all | Inspect each write for subject references |

`requested_at` accepts ISO strings or float epoch seconds.

## Example Policy

```json
{
  "max_pending_days": 30,
  "block_processing_for_subjects": true,
  "block_writes_for_subjects": true,
  "warn_threshold_days": 25,
  "action_on_violation": "block"
}
```

## SDK Integration

```python

waxell.init()

@waxell.observe(agent_name="support-agent", enforce_policy=True)
async def handle_ticket(ticket: dict) -> dict:
    return await respond(ticket)
```

Tenants populate erasure requests from their PrivacyAdmin service:

```python
ctx.metadata["erasure_requests"] = [
    {"sub_user_id": "user_123", "requested_at": "2026-05-01T10:00:00Z"},
    {"sub_user_id": "user_456", "requested_at": "2026-05-20T14:30:00Z"},
]
```

## Observability

| Field | Example |
|-------|---------|
| **Category** | `data-erasure` |
| **Action** | `block` |
| **Reason** | "Subject(s) ['user_123'] have pending erasure requests; further processing of their data is prohibited under GDPR Art-17." |
| **Metadata** | `{"signal": "erasure_subject_processed", "subject_ids": ["user_123"], "gdpr": "Art-17"}` |

Possible `signal` values: `erasure_sla_overdue`, `erasure_subject_processed`, `erasure_subject_write`, `erasure_sla_approaching`.

## Common Gotchas

1. **Write detection is substring-based.** The handler does `str(write).lower()` and checks for any pending-subject id as a substring. If your subject ids are short or numeric (`"42"`), you will get **false positives** on writes that happen to contain `"42"` anywhere. Use long, unique subject ids (UUIDs, prefixed slugs).

2. **`erasure_requests` is tenant-populated.** This handler does NOT query your privacy admin database. You must populate `context.metadata["erasure_requests"]` from your PrivacyAdmin service (typically via a middleware that injects it on every run).

3. **SLA violations block ALL runs, not just the affected subject's.** If any pending request is past `max_pending_days`, the handler blocks the current run regardless of who it is for -- this is intentional pressure on the tenant to complete erasure.

4. **Sub-user identity is preferred over `user_id`.** The handler reads `_sub_user_identity` first, falls back to `user_id`. If you have both, make sure they identify the same person -- or only set the one that matches your `erasure_requests` schema.

5. **Memory writes need to be on `context.memory_writes`.** Memory writes that bypass the context (direct DB inserts, raw cache writes) will not be detected. Wire memory operations through the runtime to get coverage.

6. **`warn_threshold_days >= max_pending_days` is silently useless.** SLA-overdue triggers first; the warn branch only fires for requests between `warn_threshold_days` and `max_pending_days`.

## Next Steps

- [Privacy Policy](./privacy) -- PII detection (often the precursor to an erasure request)
- [Data Access Policy](./data-access) -- Source allowlists; complements erasure enforcement
- [Memory Policy](./memory) -- Bounds on what can be persisted
- [Compliance Policy](./compliance) -- Meta-validator for GDPR/CCPA
- [Policy Categories](../features/policy-categories) -- All categories


--------------------------------------------------------------------------------
# Breach Notification Policy
URL: https://waxell.ai/docs/observe/governance/breach-notification
Description: Enforce regulatory breach-notification SLAs -- 72h GDPR Art-33, 60d HIPAA §164.404. Blocks agent activity until notifications are dispatched.
--------------------------------------------------------------------------------

# Breach Notification Policy

The `breach-notification` policy category enforces **regulatory deadlines on breach disclosure**. When a breach signal is present on the run context, this handler computes the elapsed time since onset and:

- **Warns** when the SLA is approaching (configurable threshold)
- **Blocks** all further agent activity until the tenant flips `breach_notified = true`
- **Blocks** when the SLA is exceeded without notification

Aligned to **GDPR Art-33** (72h breach notification) and **HIPAA §164.404** (Breach Notification Rule).

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `breach_signals` | string[] | `["data_breach", "pii_leak"]` | Signal slugs that indicate a breach (matched case-insensitively against `context.metadata["breach_signal"]`) |
| `notification_sla_hours` | integer | `72` | Hard regulatory SLA in hours (GDPR = 72; HIPAA = 1440 / 60d) |
| `warn_threshold_hours` | integer | `24` | Warn this many hours before the SLA hits |
| `block_on_overdue` | boolean | `true` | Block all further activity when SLA exceeded without notification |
| `action_on_breach` | string | `"block"` | Action when breach detected: `"block"` or `"warn"` |

## How It Works

The `breach-notification` handler runs at **before_workflow**, **mid_execution**, and **after_workflow** so the deadline is re-evaluated at every checkpoint.

| Phase | Behavior |
|-------|----------|
| `before_workflow` | Check for breach signal + elapsed SLA before any work begins |
| `mid_execution` | Re-evaluate during long-running runs (signal may surface mid-flight) |
| `after_workflow` | Catch breaches detected from final output content scans |

### Decision Flow

1. **No breach signal** → `ALLOW` (fast common case)
2. **Breach signal present but not in `breach_signals` list** → `ALLOW` (a different breach policy may target it)
3. **`breach_notified = true`** → `ALLOW` with reason "notification dispatched; proceeding with remediation"
4. **No `breach_event_at` timestamp** → `BLOCK` (cannot compute SLA)
5. **SLA exceeded** + `block_on_overdue` → `BLOCK`
6. **Within `warn_threshold_hours` of SLA** → `WARN`
7. **Breach known, within SLA, not notified** → `BLOCK` (default) -- this is the exact failure mode the regulation targets

### Context Attributes Read

| Attribute | Phase | Purpose |
|-----------|-------|---------|
| `context.metadata["breach_signal"]` | all | Slug of the breach kind (e.g., `"pii_leak"`) |
| `context.metadata["breach_event_at"]` | all | ISO timestamp or epoch float of breach onset |
| `context.metadata["breach_notified"]` | all | Boolean flag tenant flips after dispatching notifications |

`breach_event_at` accepts ISO strings (`"2026-05-09T12:00:00Z"`) and float epoch seconds.

## Example Policy

```json
{
  "breach_signals": ["data_breach", "pii_leak", "ransomware"],
  "notification_sla_hours": 72,
  "warn_threshold_hours": 24,
  "block_on_overdue": true,
  "action_on_breach": "block"
}
```

## SDK Integration

```python

waxell.init()

@waxell.observe(agent_name="claims-agent", enforce_policy=True)
async def process_claim(claim: dict) -> dict:
    return await adjudicate(claim)
```

Tenants set breach metadata via their incident-response pipeline:

```python
# When IR confirms a breach
ctx.metadata["breach_signal"] = "pii_leak"
ctx.metadata["breach_event_at"] = "2026-05-25T08:00:00Z"

# After notifications dispatched
ctx.metadata["breach_notified"] = True
```

## Observability

| Field | Example |
|-------|---------|
| **Category** | `breach-notification` |
| **Action** | `block` |
| **Reason** | "Breach 'pii_leak' notification SLA (72h) exceeded by 4.3h. Dispatch notifications before resuming agent activity." |
| **Metadata** | `{"signal": "breach_sla_overdue", "elapsed_hours": 76.3, "sla_hours": 72, "remaining_hours": -4.3, "gdpr": "Art-33", "hipaa": "§164.404"}` |

## Common Gotchas

1. **Missing `breach_event_at` blocks by default.** A breach signal without a timestamp is treated as "block until IR captures onset time". This is intentional -- proceeding without knowing when the breach started makes the SLA un-computable.

2. **Empty `breach_signals` list = match-all.** If `breach_signals` is omitted or empty, ANY non-empty `context.metadata["breach_signal"]` triggers evaluation. Scope it explicitly to avoid surprise blocks.

3. **`breach_notified` accepts truthy variants.** `True`, `"true"`, `1`, `"1"` all count as notified. Other truthy values (`"yes"`, `"done"`) do NOT.

4. **The handler does not dispatch notifications.** It is a deadline enforcer. Tenants must wire their own incident-response system (PagerDuty, email-to-counsel, regulatory portal) and flip `breach_notified` when complete.

5. **GDPR is 72 hours; HIPAA is 60 days.** Use separate policies (one per regulation) with different `breach_signals` if you need both -- e.g., `breach_signals: ["pii_leak"]` at 72h for GDPR, `breach_signals: ["phi_leak"]` at 1440h for HIPAA.

6. **`action_on_breach = "warn"` weakens the block.** Use `"warn"` only when downstream automation will hard-block via another mechanism (kill-switch, CI gate).

## Next Steps

- [Privacy Policy](./privacy) -- PII detection and redaction (often the source of breach signals)
- [Audit Policy](./audit) -- Required to capture the breach event in the audit log
- [Kill Switch Policy](./kill-switch) -- Emergency stop independent of SLA math
- [Policy Categories](../features/policy-categories) -- All categories


--------------------------------------------------------------------------------
# Bias Trend Policy
URL: https://waxell.ai/docs/observe/governance/bias-trend
Description: Continuous fairness monitoring -- track the rate of bias flags over a rolling window and fire when it exceeds threshold. Aligned to NIST AI RMF MS-3.1 and EU AI Act Art-10.
--------------------------------------------------------------------------------

# Bias Trend Policy

The `bias-trend` policy category provides **continuous fairness measurement** -- it tracks the rate of bias flags over a rolling time window and fires when the rate crosses a threshold. Per-run bias detection (handled by [`reasoning`](./reasoning)) catches isolated incidents; this handler catches the *drift* that regulators require ongoing measurement of.

Aligned to **NIST AI RMF MS-3.1** and **EU AI Act Art-10** (fairness obligations).

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `tracking_window_hours` | integer | `168` (7 days) | Rolling window size for rate calculation |
| `max_bias_rate` | number | `0.10` | Fraction of recent runs flagged that triggers the policy (0.10 = 10%) |
| `min_sample_size` | integer | `50` | Require at least N runs in the window before evaluating |
| `action_on_exceed` | string | `"warn"` | Action when threshold exceeded: `"warn"` or `"block"` |

## How It Works

The `bias-trend` handler runs at **mid_execution** and **after_workflow**. At each phase it records whether the current run had bias flags, then evaluates the rolling rate across all runs for the same agent.

| Phase | Behavior |
|-------|----------|
| `before_workflow` | No-op (`ALLOW`) |
| `mid_execution` | Record + evaluate (the `reasoning` handler typically populates `bias_flags` at this point) |
| `after_workflow` | Record + evaluate (catches runs where bias is detected only at completion) |

### Storage

Records are stored in a per-process in-memory ring buffer (bounded to 2,000 entries per agent, lock-protected for thread safety). Tenants needing cross-process aggregation install a `_history_fn` callable on the handler class that returns `(flagged_count, total_count)` from their telemetry pipeline.

### Context Attributes Read

| Attribute | Phase | Purpose |
|-----------|-------|---------|
| `context.bias_flags` | mid_execution, after_workflow | Current run's bias flags (list[str]); empty = unflagged |
| `context.agent_name` | mid_execution, after_workflow | Scopes tracking per-agent |

## Example Policy

```json
{
  "tracking_window_hours": 168,
  "max_bias_rate": 0.05,
  "min_sample_size": 100,
  "action_on_exceed": "warn"
}
```

This warns when more than 5% of the last 168 hours of runs (with at least 100 samples) were bias-flagged.

## SDK Integration

```python

waxell.init()

@waxell.observe(agent_name="hiring-screener", enforce_policy=True)
async def screen_candidate(resume: str) -> dict:
    return await rank(resume)
```

The `reasoning` policy populates `context.bias_flags` on each run; the `bias-trend` policy aggregates across runs. Assign both for full coverage.

## Observability

| Field | Example |
|-------|---------|
| **Category** | `bias-trend` |
| **Action** | `warn` |
| **Reason** | "Bias rate for 'hiring-screener' = 12.4% over last 168h (threshold 10.0%); 31/250 runs flagged." |
| **Metadata** | `{"signal": "bias_rate_exceeded", "bias_rate": 0.124, "threshold": 0.10, "flagged_count": 31, "total_count": 250, "window_hours": 168, "nist_ai_rmf": "MS-3.1", "eu_ai_act": "Art-10"}` |

## Common Gotchas

1. **Per-process buffer.** The default in-memory ring buffer is **not** shared across workers. A multi-process deployment with 4 workers will track 4 independent buffers. Install a `_history_fn` that reads from your telemetry pipeline (OpenSearch, Postgres, Loki) for accurate cross-worker rates.

2. **`min_sample_size` silences early runs.** Until the window has ≥ N samples, the handler always returns `ALLOW`. New agents will never fire this policy until they accumulate enough history.

3. **`max_bias_rate` is a *fraction*, not a percentage.** Use `0.10` for 10%, not `10`.

4. **Bounded buffer = 2,000 entries.** Tenants with extremely high run volumes will lose the oldest samples in the buffer. Use the injection seam for accurate rates at scale.

5. **Bias flags come from upstream.** This handler reads `context.bias_flags` but does not populate it. You need the `reasoning` policy (or equivalent custom handler) to flag bias per-run.

6. **`short_circuit_on_block = False`.** Like `audit`, this handler runs even when prior handlers have blocked -- so the flagged run is still recorded.

## Next Steps

- [Reasoning Policy](./reasoning) -- Per-run bias detection that populates `bias_flags`
- [Compliance Policy](./compliance) -- Meta-validator for NIST AI RMF / EU AI Act
- [Policy Categories](../features/policy-categories) -- All categories


--------------------------------------------------------------------------------
# Model Card Required Policy
URL: https://waxell.ai/docs/observe/governance/model-card-required
Description: OWASP LLM03 / NIST AI RMF GV-1.1 — every model used by an agent must have a declared model card with a risk classification. Models without a card or above the policy ceiling are blocked.
--------------------------------------------------------------------------------

# Model Card Required Policy

The `model-card-required` policy enforces **OWASP LLM03 Supply Chain** and aligns with **NIST AI RMF GV-1.1** (Govern → Model Risk Classification) and **EU AI Act Article 13** (transparency). Every model used by an agent must have a declared **model card** on file with a risk tier of `low`, `medium`, `high`, or `critical`. Models with no card — or with a tier above the policy ceiling — are blocked.

Model cards can come from three sources, checked in order:

1. **Inline registry** (`rules.model_cards`) — simplest path for a single policy.
2. **Injected registry function** (`_registry_fn`) — wired by the controlplane to `SystemModelCost` or an external registry.
3. **Empty** — no card → BLOCK (or WARN per config).

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `max_risk_tier` | string | `"medium"` | Block models with tier higher than this (`low` < `medium` < `high` < `critical`) |
| `allowed_models` | string[] | `[]` | Positive allowlist — non-empty means ONLY these models may be used |
| `blocked_models` | string[] | `[]` | Deny-list (wins over `allowed_models` and `max_risk_tier`) |
| `require_card` | boolean | `true` | When true, models with no declared card are blocked |
| `model_cards` | object | `{}` | Inline map `model_id → {risk_tier, owner, ...}` |
| `action_on_violation` | string | `"block"` | `"block"` or `"warn"` |

## How It Works

The handler runs at **all three phases**, since `models_used` accumulates as the workflow progresses. Evaluation order per model:

1. **Deny-list** — `blocked_models` match → BLOCK.
2. **Allowlist** — if `allowed_models` is non-empty and the model isn't in it → BLOCK.
3. **Card lookup** — inline `model_cards` then injected `_registry_fn`. Missing card + `require_card=true` → BLOCK.
4. **Risk-tier ceiling** — model tier rank > `max_risk_tier` rank → BLOCK.

| Phase | Behavior |
|-------|----------|
| `before_workflow` | Check pre-declared models |
| `mid_execution` | Check `models_used` accumulated so far |
| `after_workflow` | Final audit of all models used |

### Context Attributes Read

| Attribute | Phase | Purpose |
|-----------|-------|---------|
| `context.models_used` | all | List of model IDs invoked during the run |

## Example Policy

Allow only carded models at or below medium risk; block `gpt-4-32k` outright:

```json
{
  "max_risk_tier": "medium",
  "allowed_models": ["gpt-4o-mini", "gpt-4o", "claude-3-5-sonnet-20241022"],
  "blocked_models": ["gpt-4-32k", "gpt-3.5-turbo-0301"],
  "require_card": true,
  "model_cards": {
    "gpt-4o-mini": {"risk_tier": "low", "owner": "openai"},
    "gpt-4o": {"risk_tier": "medium", "owner": "openai"},
    "claude-3-5-sonnet-20241022": {"risk_tier": "medium", "owner": "anthropic"}
  },
  "action_on_violation": "block"
}
```

## SDK Integration

```python


waxell.init()

@waxell.observe(agent_name="research-agent", enforce_policy=True)
async def research(query: str) -> str:
    # Each LLM call adds to context.models_used.
    # If an uncarded or over-tier model is invoked, mid_execution blocks.
    return await run_research(query)
```

## Observability

| Field | Example |
|-------|---------|
| **Category** | `model-card-required` |
| **Action** | `block` |
| **Reason** | "Model 'gpt-4-turbo-preview' has no declared model card. OWASP LLM03 requires risk classification." |
| **Metadata** | `{"signal": "no_model_card", "model": "gpt-4-turbo-preview", "owasp": "LLM03"}` |

Risk-tier exceeded:

| Field | Example |
|-------|---------|
| **Reason** | "Model 'gpt-4o' risk tier 'high' exceeds policy ceiling 'medium'." |
| **Metadata** | `{"signal": "risk_tier_exceeded", "model": "gpt-4o", "model_risk_tier": "high", "policy_max_tier": "medium"}` |

## Common Gotchas

- **Risk-tier comparison uses ordinal rank.** `low=0`, `medium=1`, `high=2`, `critical=3`. `max_risk_tier="medium"` permits `low` and `medium` and blocks `high` and `critical`.
- **Card lookup is case-insensitive but exact-id-first.** `gpt-4o` is checked literally, then case-folded against keys. Aliases (e.g., `openai/gpt-4o`) won't auto-resolve — register both forms.
- **`require_card=false` is rarely advisable.** It permits uncarded models silently and defeats the OWASP LLM03 goal.
- **Card with invalid `risk_tier` is treated like a missing card** when `require_card=true`. Stick to the four canonical tier strings.
- **Empty `models_used` always allows.** If your runtime doesn't populate `models_used`, this policy effectively no-ops.
- **The injected `_registry_fn` is set class-wide.** Controlplane wires it once during app startup; if it raises, the handler logs and treats the result as "no card found".

## Next Steps

- [Policy Categories](../features/policy-categories) — All 49 categories
- [LLM Policy](./llm) — Model allowlist + per-call constraints
- [Compliance Policy](./compliance) — Meta-validator for regulatory frameworks
- [Provenance Required](./provenance-required) — Per-claim citation enforcement (LLM09b)


--------------------------------------------------------------------------------
# End-User Budget Policy
URL: https://waxell.ai/docs/observe/governance/end-user-budget
Description: Per-WaxellUser monthly spend caps -- enforce $/month limits on individual end-users behind a tenant.
--------------------------------------------------------------------------------

# End-User Budget Policy

The `end-user-budget` policy category enforces **per-end-user monthly spend caps**. Each `WaxellUser` row can carry a `monthly_budget_cap_cents` value; this handler sums that user's `WaxellUserActivity.cost_cents` for the current calendar month and blocks (or warns / throttles) when the cap is hit.

Use it when a tenant has many sub-users (employees, customers, agent-of-agent identities) and you want **per-seat cost ceilings**, not just a tenant-wide spend cap. The tenant-wide cap is handled by the [Budget Policy](./budget).

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `enabled` | boolean | `true` | Toggle enforcement on/off |
| `action_on_exceed` | string | `block` | Action when over cap: `block`, `warn`, `throttle` |
| `warning_threshold_percent` | number | `80` | Emit WARN when spend reaches this percent of cap |

## How It Works

The `end-user-budget` handler runs at **`before_workflow`**, **`mid_execution`**, and **`before_domain_call`**. `after_workflow` is a no-op (cap is enforced before spend, not after).

The handler only fires when:

1. A sub-user identity is present on the context (`context.sub_user_identity` or `context.user_id`)
2. The matching `WaxellUser` row has a non-null, non-zero `monthly_budget_cap_cents`

Tenants that don't set per-user caps see no policy effect.

### Context Attributes Read

| Attribute | Phase | Purpose |
|-----------|-------|---------|
| `context.sub_user_identity` | all | Resolve `sub_user_id` for the active end-user |
| `context.user_id` | all | Fallback resolver when sub-user identity isn't set |
| `context.metadata["tenant_id"]` / `context.tenant_id` | all | Scope the user lookup |

### Data Sources

- `WaxellUser.monthly_budget_cap_cents` -- the per-user cap (integer cents)
- `WaxellUserActivity.cost_cents` -- summed over the current calendar month

Runtime plane can inject `_usage_query_fn` to route through a Redis-backed aggregator instead of Django ORM; observe plane uses ORM by default.

## Example Policy

```json
{
  "name": "Per-seat $200/mo Cap",
  "category": "end-user-budget",
  "rules": {
    "enabled": true,
    "action_on_exceed": "block",
    "warning_threshold_percent": 75
  },
  "scope": {
    "agents": ["*"]
  },
  "enabled": true
}
```

## SDK Integration

Pass the end-user identity via `user_id` / `user_group` on the decorator. The handler reads it off the context and looks up the matching `WaxellUser` row.

```python

waxell.init()

@waxell.observe(
    agent_name="research-assistant",
    user_id="emp-4127",
    user_group="enterprise",
    enforce_policy=True,
)
async def run_research(query: str) -> str:
    return await do_research(query)
```

Set the cap by updating the `WaxellUser` row:

```bash
wax end-users update emp-4127 --monthly-cap-cents 20000  # $200
```

## Observability

| Field | Example (BLOCK) |
|-------|---------|
| **Category** | `end-user-budget` |
| **Action** | `block` |
| **Reason** | `End-user 'emp-4127' monthly budget exceeded ($201.42 / $200.00).` |
| **Metadata** | `{"sub_user_id": "emp-4127", "spent_cents": 20142, "cap_cents": 20000}` |

| Field | Example (WARN) |
|-------|---------|
| **Action** | `warn` |
| **Reason** | `End-user 'emp-4127' approaching budget (82% used).` |
| **Metadata** | `{"percent_used": 82.1, ...}` |

## Common Gotchas

- **Cap is **per-user**, not per-tenant.** Tenant-wide ceilings still need a [Budget Policy](./budget).
- **No sub-user on context = ALLOW.** This category is a no-op for runs that don't carry end-user identity. Pair with a tenant-level [Budget Policy](./budget) for those.
- **`monthly_budget_cap_cents` is null by default.** No cap row = no enforcement. The handler doesn't synthesize a default cap.
- **Calendar month, not rolling 30 days.** The window starts at midnight on the 1st. Spend resets at month boundary.
- **Spend is in cents.** A `$200` cap is `20000`, not `200`.
- **Observe-plane uses Django ORM** -- runtime plane wires a Redis-backed aggregator via the `_usage_query_fn` seam for low-latency reads.

## Next Steps

- [Budget Policy](./budget) -- Tenant-wide monthly spend cap
- [End-User Rate Limit](./end-user-rate-limit) -- Per-user request-rate cap
- [End-User Suspension](./end-user-suspension) -- Block runs for suspended end-users
- [Policy Categories & Templates](../features/policy-categories) -- All categories
- [User Tracking](../features/user-tracking) -- How sub-user identity flows through Observe


--------------------------------------------------------------------------------
# End-User Rate Limit Policy
URL: https://waxell.ai/docs/observe/governance/end-user-rate-limit
Description: Per-end-user and per-group request-rate caps -- protect against runaway sub-users and noisy customers.
--------------------------------------------------------------------------------

# End-User Rate Limit Policy

The `end-user-rate-limit` policy category enforces **per-end-user (and per-group) request-rate caps**. Each `WaxellUser` can carry a `rate_limit_override_rpm`; each `WaxellUserGroup` can carry one too. The **tightest cap wins** when a user belongs to multiple groups.

Counts the user's activity rows in the configured window -- includes `signal_received`, `run_started`, `domain_call`, and `mcp_call`. Use it to protect against runaway sub-users, noisy customers, or abusive third-party agent integrations.

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `enabled` | boolean | `true` | Toggle enforcement on/off |
| `action_on_exceed` | string | `throttle` | Action when over cap: `throttle`, `block`, `warn` |
| `window_seconds` | integer | `60` | Counting window for activity |

The actual cap value comes from `WaxellUser.rate_limit_override_rpm` / `WaxellUserGroup.rate_limit_override_rpm`, not from the policy rules. The policy rules only configure **what to do when caps are hit**.

## How It Works

The handler runs at **`before_workflow`**, **`mid_execution`**, and **`before_domain_call`**. `after_workflow` is a no-op.

### Cap Resolution

```
caps = []
if user.rate_limit_override_rpm:
    caps.append(user.rate_limit_override_rpm)
for group in user.groups:
    if group.rate_limit_override_rpm:
        caps.append(group.rate_limit_override_rpm)
effective_cap = min(caps)  # tightest wins
```

If no caps are set on either the user or any of their groups, the handler returns ALLOW (no enforcement).

### Cap is RPM, Window is Configurable

`cap_rpm=60` with `window_seconds=60` means 60 requests/minute. With `window_seconds=30` it scales linearly to 30 requests/30s.

### Context Attributes Read

| Attribute | Phase | Purpose |
|-----------|-------|---------|
| `context.sub_user_identity` | all | Resolve `sub_user_id` |
| `context.user_id` | all | Fallback resolver |
| `context.metadata["tenant_id"]` / `context.tenant_id` | all | Scope the user lookup |

### Data Sources

- `WaxellUser.rate_limit_override_rpm`
- `WaxellUserGroupMembership` -> `WaxellUserGroup.rate_limit_override_rpm`
- `WaxellUserActivity` rows where `kind in (signal_received, run_started, domain_call, mcp_call)`

## Example Policy

```json
{
  "name": "Per-seat 60 RPM throttle",
  "category": "end-user-rate-limit",
  "rules": {
    "enabled": true,
    "action_on_exceed": "throttle",
    "window_seconds": 60
  },
  "scope": {
    "agents": ["*"]
  },
  "enabled": true
}
```

## SDK Integration

```python

waxell.init()

@waxell.observe(
    agent_name="support-bot",
    user_id="cust-9912",
    user_group="free-tier",
    enforce_policy=True,
)
async def handle_message(text: str) -> str:
    return await respond(text)
```

Set caps via CLI:

```bash
wax end-users update cust-9912 --rate-limit-rpm 30
wax groups update free-tier --rate-limit-rpm 10
```

## Observability

| Field | Example (THROTTLE) |
|-------|---------|
| **Category** | `end-user-rate-limit` |
| **Action** | `throttle` |
| **Reason** | `End-user 'cust-9912' rate-limited (12/10 in last 60s, cap=10/min).` |
| **Metadata** | `{"sub_user_id": "cust-9912", "count": 12, "cap_rpm": 10, "window_seconds": 60}` |

## Common Gotchas

- **Tightest cap wins across user + ALL groups.** A user in two groups with caps of 100 and 30 gets the 30 cap.
- **No override set on user OR groups = ALLOW.** This category needs at least one row with a non-null `rate_limit_override_rpm` to do anything.
- **Counts user-initiated activity only.** Bookkeeping events (`created`, `updated`, `suspended`, `hard_deleted`) are excluded.
- **Default action is `throttle`, not `block`.** Throttle lets the runtime back off and retry; block kills the run.
- **Window is a sliding window** computed from `timezone.now() - window_seconds`. Not a fixed rolling bucket.
- **Observe plane uses Django ORM.** Runtime plane wires a Redis-backed aggregator via the `_usage_query_fn` seam.

## Next Steps

- [Rate Limit Policy](./rate-limit) -- Tenant-wide rate caps
- [End-User Budget](./end-user-budget) -- Per-user monthly spend cap
- [End-User Suspension](./end-user-suspension) -- Block runs for suspended end-users
- [User Tracking](../features/user-tracking) -- How sub-user identity flows
- [Policy Categories & Templates](../features/policy-categories) -- All categories


--------------------------------------------------------------------------------
# End-User Suspension Policy
URL: https://waxell.ai/docs/observe/governance/end-user-suspension
Description: Block runs for suspended end-users -- per-seat kill switch that takes effect at the next turn boundary.
--------------------------------------------------------------------------------

# End-User Suspension Policy

The `end-user-suspension` policy category blocks execution when the current run's sub-user has `WaxellUser.status = 'suspended'`. It's the **per-end-user analogue of the tenant [Kill Switch](./kill-switch)** -- you can suspend a single bad actor without affecting the rest of the tenant.

Runs at `before_workflow`, `mid_execution`, and `before_domain_call` so a suspension flip takes effect within the next turn for long-running workflows.

## Rules

| Rule | Type | Default | Description |
|------|------|---------|-------------|
| `enabled` | boolean | `true` | Toggle enforcement on/off |
| `grace_seconds` | integer | `0` | Reserved -- tolerate recent status changes by N seconds |

The handler reads the user's status directly from the `WaxellUser` row. There is no policy rule to tune *which* users are suspended -- that's controlled by the user record itself.

## How It Works

The handler runs at:

| Phase | Effect when suspended |
|-------|----------------------|
| `before_workflow` | BLOCK -- run never starts a turn |
| `mid_execution` | BLOCK -- halts at next turn boundary |
| `before_domain_call` | BLOCK -- no outbound call fires |
| `after_workflow` | ALLOW (no-op -- run already halted) |

When no sub-user is on the context, the handler returns ALLOW. This category is **scoped to sub-user identity only**; for tenant-wide bans use the [Kill Switch Policy](./kill-switch).

### Context Attributes Read

| Attribute | Phase | Purpose |
|-----------|-------|---------|
| `context.sub_user_identity` | all | Resolve `sub_user_id` |
| `context.user_id` | all | Fallback resolver |
| `context.metadata["tenant_id"]` / `context.tenant_id` | all | Scope the user lookup |

### Data Sources

- `WaxellUser.status` -- compared against `WaxellUserStatus.SUSPENDED`

## Example Policy

```json
{
  "name": "Block suspended sub-users",
  "category": "end-user-suspension",
  "rules": {
    "enabled": true,
    "grace_seconds": 0
  },
  "scope": {
    "agents": ["*"]
  },
  "enabled": true
}
```

A single policy like this on `agents: ["*"]` covers your whole tenant. You don't need to author per-user policies -- suspension state lives on the user row, not the policy.

## SDK Integration

```python

waxell.init()

@waxell.observe(
    agent_name="customer-support",
    user_id="cust-9912",
    enforce_policy=True,
)
async def handle(text: str) -> str:
    return await respond(text)
```

To suspend a user:

```bash
wax end-users suspend cust-9912
# or via API:
curl -X POST /api/v1/end-users/cust-9912/suspend/
```

To unsuspend:

```bash
wax end-users unsuspend cust-9912
```

## Observability

| Field | Example (BLOCK) |
|-------|---------|
| **Category** | `end-user-suspension` |
| **Action** | `block` |
| **Reason** | `End-user 'cust-9912' is suspended on this tenant. Unsuspend via /api/v1/end-users/<id>/unsuspend/ or wax end-users unsuspend <id>.` |
| **Metadata** | `{"sub_user_id": "cust-9912", "tenant_id": "acme"}` |

## Common Gotchas

- **No sub-user on context = ALLOW.** Tenant-wide bans must use the [Kill Switch Policy](./kill-switch).
- **Suspension flip takes effect at next turn boundary**, not instantly. A long-running step in flight will finish before the next BLOCK fires.
- **Lookup errors fail-open.** If the user lookup raises, the handler returns ALLOW with a warning log. Better to let a real run proceed than to take everyone down on a transient DB error.
- **No rule to "suspend user X".** Suspension state is on the `WaxellUser` row; the policy just decides *whether* to enforce it. Tenants who want suspension behavior on always can leave this category enabled tenant-wide.
- **`grace_seconds` is reserved.** Currently not honored by the handler; future phase will let you tolerate very recent status flips to avoid race conditions on suspend/unsuspend.
- **Observe plane uses Django ORM.** Runtime plane wires a cached / Redis lookup via `_status_query_fn`.

## Next Steps

- [Kill Switch Policy](./kill-switch) -- Tenant-wide emergency stop
- [End-User Budget](./end-user-budget) -- Per-user monthly spend cap
- [End-User Rate Limit](./end-user-rate-limit) -- Per-user request-rate cap
- [User Tracking](../features/user-tracking) -- How sub-user identity flows
- [Policy Categories & Templates](../features/policy-categories) -- All categories


--------------------------------------------------------------------------------
# Policy Recommendations
URL: https://waxell.ai/docs/observe/features/recommendations
Description: Automated policy suggestions based on your agent's actual runtime behavior -- token budgets, cost limits, timeouts, and safety bounds.
--------------------------------------------------------------------------------

# Policy Recommendations

Waxell analyzes your agents' runtime behavior and automatically generates policy recommendations. Instead of guessing what limits to set, let the platform suggest them based on real data.

## How It Works

The recommendation engine uses statistical analysis of your agents' execution history:

1. **Collects metrics** -- tokens, cost, duration, and step counts from completed runs
2. **Calculates baselines** -- average and standard deviation over a configurable window (default: 7 days)
3. **Generates limits** -- recommended limit = 2x average, anomaly threshold = 3x average
4. **Creates recommendations** -- pending suggestions that you can accept or dismiss

No ML models involved -- just straightforward statistical heuristics that work from day one.

## Recommendation Types

| Type | Category | What it suggests | Minimum runs needed |
|------|----------|-----------------|-------------------|
| **Token budget** | Cost | Daily token limit based on agent's average usage | 5 |
| **Cost budget** | Cost | Daily cost limit based on agent's average spend | 5 |
| **Timeout** | Operations | Execution timeout based on agent's average duration | 10 |
| **Step limit** | Safety | Max steps per run based on agent's average step count | 10 |

## Two Modes

### Batch Mode (Daily)

Runs on a schedule to generate recommendations for all agents with enough data:

```bash
# Generate for all agents (last 7 days)
python manage.py generate_recommendations

# Generate for a specific agent
python manage.py generate_recommendations --agent support-bot

# Custom time window
python manage.py generate_recommendations --days 14 --tenant acme
```

### Per-Run Anomaly Detection

Fires immediately when a single run is **3x the agent's 7-day average**. This catches anomalies in real-time -- if your agent suddenly uses 10x its normal tokens, you'll get a recommendation within seconds.

Triggered automatically after each run completes. Requires at least 3 prior runs for the agent.

## Managing Recommendations

### Via Dashboard

Navigate to **Governance > Recommendations** to see all pending suggestions:

- **Pending** -- New recommendations waiting for review
- **Accepted** -- Recommendations you've turned into active policies
- **Dismissed** -- Recommendations you've declined

Accepting a recommendation creates a new policy with the suggested configuration. You can customize the policy before it's enabled.

### Via API

```bash
# List pending recommendations
curl -H "Authorization: Bearer $TOKEN" \
  "https://acme.waxell.dev/waxell/v1/recommendations/?status=pending"

# Accept a recommendation (creates a policy)
curl -X POST -H "Authorization: Bearer $TOKEN" \
  "https://acme.waxell.dev/waxell/v1/recommendations/{uuid}/accept/"

# Dismiss a recommendation
curl -X POST -H "Authorization: Bearer $TOKEN" \
  "https://acme.waxell.dev/waxell/v1/recommendations/{uuid}/dismiss/"
```

### Via Platform Assistant

Ask the assistant for a governance check:

> "What governance recommendations do I have?"
> "Show me recommendations for my support-bot agent"

The assistant can walk you through accepting or dismissing recommendations interactively. See [Platform Assistant](./platform-assistant) for details.

## Example Flow

1. Deploy `support-bot` with no policies
2. After 10+ runs, the recommendation engine analyzes behavior:
   - Average tokens per run: 2,500
   - Average cost per run: $0.08
   - Average duration: 4.2 seconds
3. Recommendations generated:
   - Token budget: 5,000 tokens/run (2x average)
   - Cost budget: $0.16/run (2x average)
   - Timeout: 10 seconds (2x average)
4. You accept the token budget recommendation -- a cost policy is created automatically
5. A day later, a run uses 8,000 tokens (3.2x average) -- anomaly recommendation fires immediately

## Next Steps

- [Policy Categories & Templates](./policy-categories) -- All policy categories and pre-built templates
- [Governance](./governance) -- Policy enforcement in your agent code
- [Platform Assistant](./platform-assistant) -- Manage recommendations via chat


--------------------------------------------------------------------------------
# Platform Assistant
URL: https://waxell.ai/docs/observe/features/platform-assistant
Description: The AI operations advisor built into Waxell — query your data, create policies, get proactive insights, and learn the platform through natural language.
--------------------------------------------------------------------------------

# Platform Assistant

Waxell includes a built-in AI operations advisor that can query your observability data, create and manage policies, coach your team through setup, and proactively surface what needs your attention — all through natural language.

Open it from the **Ask AI** button in the top navigation on any page, or visit **your assistant home** at `/assistant` for a personalized briefing.

## The Assistant Dock

The assistant lives in a dock on the right side of every page, with three tabs:

| Tab | What it shows |
|-----|---------------|
| **Chat** | The conversation — ask anything, get answers grounded in your actual data |
| **Feed** | Proactive notifications the assistant has surfaced for you — setup nudges, anomalies, and follow-ups, with an unread badge |
| **Pulse** | A live snapshot of fleet activity |

The dock is **page-aware**: the suggested prompts change based on where you are. On the Experiments page it offers error analysis and cost hotspots; on Getting Started it offers setup guidance; on Governance it offers policy work. You can also reach the assistant from empty states, entity pages, and the command palette (**⌘K**).

## Assistant Home

Visit `/assistant` for your briefing page: what needs you, where you left off, and what to set up next. The **For You** section collects the assistant's proactive findings; **Continue the conversation** picks up your recent chats exactly where they stopped.

## What It Can Do

### Query Your Data

Ask questions about your agents, runs, and costs in plain language:

> "How many runs did my agents do in the last 24 hours?"
> "What did I spend on LLMs today, broken down by model?"
> "Which agent has the highest error rate this week?"
> "What LLM routing is configured for my agent?"

The assistant queries your actual observability data and returns real numbers — run counts, per-model spend, error rates, session activity, experiment results — with inline charts for anything worth visualizing. If your fleet is clean, it says so and offers the follow-up that matters (like setting an alert), rather than burying you in empty tables.

### Create and Manage Policies

Policy work is conversational:

> "Help me set up policies for my agents"
> "Create a rate limit policy for support-bot — max 50 runs per hour"
> "Explain this proposed policy"

The assistant reads your fleet first — which agents are active, what coverage already exists — then proposes **interactive policy cards** you can create with one click or open in the editor. It never creates a policy without your explicit approval. If you have no policies yet, it treats that as the gap and proposes a grounded starter set.

| Card button | Action |
|--------|--------|
| **Create Policy** | Creates the policy on the control plane |
| **Open in Editor** | Opens the full policy editor with the config pre-filled |
| **Suggest Adjustments** | Asks the assistant to modify the policy |
| **Ask About This Policy** | Get more context on what the policy does |

### Take Action

Beyond policies, the assistant proposes small concrete actions as **action cards**:

- **Setup steps** — adds items to your personal setup plan (the one write it performs directly, always with your approval on the card)
- **Invite teammates** — preps who to invite and why, and links you to user management
- **API keys** — links you to key creation (the assistant never mints or displays keys itself)
- **Alert rules** — drafts the condition, threshold, and destination, and links you to alerts

Card approval is the human-in-the-loop: nothing executes until you click.

### Learn the Platform

Ask how anything works and the assistant answers from the official documentation, with citations:

> "How do I instrument a Python agent with waxell-observe?"
> "What is the MCP gateway and when should I use it?"
> "My agent runs aren't showing up in Observe — what should I check?"

Troubleshooting answers lead with the most likely cause and link every claim to its source page.

### Navigate

> "Where do I manage API keys?"
> "Take me to where I can see my LLM costs"
> "What does this page do?"

The assistant knows every page in the product and answers with direct links — plus practical notes about what you'll find there.

## Proactive Layer: Feed and Digest

The assistant doesn't just answer — it watches. An hourly evaluation looks at your workspace for things worth telling you about: unfinished setup, new failure patterns, spend changes, and coverage gaps. Findings land in your **Feed** with an unread badge, and a **weekly digest email** summarizes the highlights.

Each notification links to a pre-filled conversation so you can act on it in one click.

## Personalized to Your Role

Answers adapt to who you are:

- **Capability-aware** — the assistant only offers what your seat and role allow. It will never propose policy creation to someone without governance permissions, or quote billing to a seat without billing access.
- **Register-aware** — builders get code and exact config keys; operators get rollout steps and checklists; executives get the trend and the business impact in two sentences.

## Memory

The assistant remembers context across conversations — who you are, what you were working on, and the preferences you've expressed — so follow-up sessions start warm. Ask "what do you already know about me?" any time to see (and prune) what it holds.

## Privacy and Security

Every conversation runs under a short-lived, cryptographically signed identity token minted for you at message time:

- **Tenant-scoped by construction** — the assistant's data access is derived from verified token claims, never from anything a caller supplies. It cannot read another workspace's data, and it will tell you so if you ask it to.
- **Least-privilege** — its data access is read-only; the only writes it performs (policy creation, setup-plan steps) require your explicit approval and are attributed to you in the audit trail.
- **No internals** — the assistant presents everything in business terms and declines requests for platform implementation details.
- Conversation history is stored per-user in your own workspace.

## Escalation

If the assistant can't resolve something, use **Talk to a human** at the bottom of the dock — your conversation transcript rides along so you never repeat yourself.

## API Access

The assistant is also available via API for programmatic integration:

```bash
# Send a message
curl -X POST -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"message": "How much did my agents cost today?"}' \
  "https://acme.waxell.dev/api/v1/assistant/chat/"

# Poll for response
curl -H "Authorization: Bearer $TOKEN" \
  "https://acme.waxell.dev/api/v1/assistant/chat/{signal_id}/result/"

# Get conversation history
curl -H "Authorization: Bearer $TOKEN" \
  "https://acme.waxell.dev/api/v1/assistant/chat/history/"
```

## Next Steps

- [Policy Categories & Templates](./policy-categories) — Available policy types and pre-built templates
- [Recommendations](./recommendations) — Automated policy suggestions


--------------------------------------------------------------------------------
# FAQ
URL: https://waxell.ai/docs/observe/troubleshooting/faq
Description: Frequently asked questions about the Waxell Observe SDK -- setup, instrumentation, governance, and more.
--------------------------------------------------------------------------------

# Frequently Asked Questions

## Setup & Configuration

### Do I need to call `init()` before importing LLM SDKs?

Yes. `waxell.init()` monkey-patches LLM libraries to add auto-instrumentation. If you import `openai` or `anthropic` before calling `init()`, those imports get the un-patched versions and LLM calls won't be captured automatically.

```python
# Correct

waxell.init()
from openai import OpenAI  # patched

# Wrong -- OpenAI is imported before init()
from openai import OpenAI  # un-patched

waxell.init()  # too late for OpenAI
```

If you can't control import order (e.g., framework code imports LLM SDKs at module level), use the [drop-in imports](../api/python-sdk#drop-in-imports) instead:

```python
from waxell_observe.openai import openai     # always patched
from waxell_observe.anthropic import anthropic  # always patched
```

### Can I call `init()` multiple times?

Yes. `init()` is idempotent -- only the first call takes effect. Subsequent calls are no-ops.

### How do I disable the SDK in tests or CI?

Set the `WAXELL_OBSERVE` environment variable to `false`, `0`, or `no`:

```bash
WAXELL_OBSERVE=false pytest
```

This makes `init()` skip all initialization. Decorators and convenience functions become no-ops. Your code runs normally with zero overhead.

### What's the difference between `api_key` and `api_url`?

- **`api_url`**: Your Waxell control plane URL (e.g., `https://acme.waxell.dev`). This is where your dashboard lives.
- **`api_key`**: A secret key starting with `wax_sk_` that authenticates requests to your control plane.

Both can be set via constructor arguments, environment variables (`WAXELL_API_URL`, `WAXELL_API_KEY`), or the CLI config file (`~/.waxell/config`).

### What happens if the SDK can't reach the control plane?

The SDK is designed to never break your agent. If the control plane is unreachable:

- `init()` succeeds (tracing setup failure is logged as a warning)
- `@observe` runs your function normally and logs a warning about the failed run
- Auto-instrumentation continues to work locally (spans are captured but not exported)
- No exceptions are raised from telemetry failures

---

## Instrumentation

### Should I use decorators or the context manager?

**Start with decorators.** They cover 90% of use cases with less code:

| Situation | Use |
|-----------|-----|
| Single function = one agent run | `@observe` |
| Tool, retrieval, decision, step functions | `@tool`, `@retrieval`, `@decision`, `@step_dec` |
| Inline enrichment (scores, tags) | `waxell.score()`, `waxell.tag()` |
| Multi-step orchestration spanning many functions | `WaxellContext` |
| Batch processing with many runs per loop | `WaxellContext` |
| Need mid-execution policy checks between steps | `WaxellContext` |

See the [Context Manager](../integrations/context-manager) page for advanced scenarios.

### Why are some decorators named `reasoning_dec`, `retry_dec`, `step_dec` instead of `reasoning`, `retry`, `step`?

The names `reasoning`, `retry`, and `step` collide with the module-level convenience functions (`waxell.reason()`, `waxell.retry()`, `waxell.step()`). To avoid shadowing, the decorators are exported with a `_dec` suffix:

| Decorator | Convenience function |
|-----------|---------------------|
| `@waxell.reasoning_dec()` | `waxell.reason()` |
| `@waxell.retry_dec()` | `waxell.retry()` |
| `@waxell.step_dec()` | `waxell.step()` |

The other decorators (`@tool`, `@decision`, `@retrieval`) don't collide because their convenience function equivalents use different names (`waxell.retrieve()`, `waxell.decide()`).

### Do decorators work on sync functions?

Yes. All decorators (`@observe`, `@tool`, `@decision`, `@retrieval`, `@reasoning_dec`, `@retry_dec`, `@step_dec`) work with both sync and async functions. The SDK detects the function type and wraps accordingly.

### What happens if I use behavior decorators outside an `@observe` scope?

They're no-ops. Your function executes normally with zero overhead -- no context lookup, no recording, no errors. This means you can decorate functions once and call them from both instrumented and non-instrumented code paths.

### Does auto-instrumentation capture everything?

`waxell.init()` auto-instruments 200+ libraries for LLM calls, vector DB queries, HTTP requests, database calls, and more. It captures:

- Model name, token counts, cost, latency
- Prompt/response previews (if `capture_content=True`)
- Streaming responses (tokens are aggregated)

It does **not** capture:
- Business logic decisions (use `@decision` or `waxell.decide()`)
- Custom tool calls (use `@tool`)
- Quality scores (use `waxell.score()`)
- Tags and metadata (use `waxell.tag()`, `waxell.metadata()`)

### How do I see what libraries were auto-instrumented?

Use `waxell.diagnose()`:

```python

waxell.init()

info = waxell.diagnose()
print(info["active_instrumentors"])   # ["openai", "anthropic", ...]
print(info["detected_libraries"])     # {"openai": "1.52.0", ...}
print(info["sdk_version"])            # "0.0.40"
```

### Can I exclude specific libraries from auto-instrumentation?

Yes, use the `exclude` parameter on `init()`:

```python
waxell.init(exclude=["litellm", "mcp"])
```

Or via environment variable:

```bash
WAXELL_EXCLUDE=litellm,mcp python my_agent.py
```

For infrastructure libraries (HTTP, DB, caches), use `infra_exclude`:

```python
waxell.init(infra_exclude=["celery", "grpc"])
```

---

## Governance & Policies

### What's the difference between `enforce_policy` and `mid_execution_governance`?

- **`enforce_policy=True`** (default): Checks policies once when the run starts. If blocked, raises `PolicyViolationError` before your function executes.
- **`mid_execution_governance=True`**: In addition to the entry check, every `record_step()` call flushes data to the control plane and re-checks policies. If a policy blocks mid-execution, raises `PolicyViolationError` from inside your function.

Use `mid_execution_governance` for long-running agents where you want to catch budget overruns or policy violations before the agent finishes.

### What is `on_policy_block`?

A callback that fires when a policy blocks execution. Instead of immediately raising `PolicyViolationError`, the SDK calls your handler and lets you decide what to do:

```python
@waxell.observe(
    agent_name="my-agent",
    on_policy_block=waxell.prompt_approval,  # Terminal Y/N prompt
)
async def my_agent(query: str):
    ...
```

Built-in handlers:
- `waxell.prompt_approval` -- Interactive terminal prompt
- `waxell.auto_approve` -- Always proceed (for testing)
- `waxell.auto_deny` -- Always block (for testing)

You can write custom handlers for Slack approvals, webhook-based workflows, etc.

### Can I check policies manually without using `@observe`?

Yes, through the client:

```python
from waxell_observe import WaxellObserveClient

client = WaxellObserveClient()
result = await client.check_policy(agent_name="my-agent")
if result.blocked:
    print(f"Blocked: {result.reason}")
```

Or inside a `WaxellContext`:

```python
async with WaxellContext(agent_name="my-agent") as ctx:
    # ... do some work ...
    policy = await ctx.check_policy()
    if policy.blocked:
        return "Agent was blocked by policy"
```

---

## Data & Privacy

### Does the SDK send my prompts and responses to Waxell?

By default, **no**. Prompt and response content is not included in telemetry. Set `capture_content=True` on `init()` to opt in:

```python
waxell.init(capture_content=True)
```

Without this flag, only metadata is captured: model name, token counts, cost, latency, and timing.

### What is Prompt Guard?

Prompt Guard scans LLM inputs for PII (emails, phone numbers, SSNs), leaked credentials (API keys, tokens), and injection attempts before they reach the LLM provider. Enable it with:

```python
waxell.init(
    prompt_guard=True,              # Client-side regex detection
    prompt_guard_server=True,       # Server-side ML detection (more accurate)
    prompt_guard_action="redact",   # "block", "warn", or "redact"
)
```

### Can I use Waxell with on-premises LLM providers?

Yes. The SDK instruments any OpenAI-compatible API, local models (Ollama, vLLM, SGLang), and HuggingFace models. Cost estimation returns `0.0` for local/free providers.

---

## Sessions & Users

### How do sessions work?

A session groups related runs together. Pass the same `session_id` to multiple runs:

```python
@waxell.observe(agent_name="chatbot", session_id="sess_abc123")
async def handle_message(msg: str):
    ...

# Or dynamically at call time:
await handle_message("Hello", session_id="sess_abc123")
await handle_message("Follow up", session_id="sess_abc123")
```

Use `generate_session_id()` to create unique session IDs.

### How does user tracking work?

Pass `user_id` to attribute runs, costs, and usage to specific end users:

```python
@waxell.observe(agent_name="chatbot", user_id="user_456")
async def handle_message(msg: str):
    ...
```

This enables per-user analytics, cost attribution, and user-scoped policies in the dashboard.

---

## Performance

### What's the overhead of the SDK?

Minimal. The SDK:
- Buffers all telemetry in memory and flushes asynchronously on context exit
- Uses non-blocking HTTP calls for the control plane API
- Auto-instrumentation adds microsecond-level overhead per LLM call
- Convenience functions (`waxell.score()`, `waxell.tag()`, etc.) are simple dict appends

If the control plane is unreachable, telemetry is dropped silently -- no retry loops or blocking waits.

### Does the SDK add latency to my LLM calls?

No. Auto-instrumentation wraps LLM SDK methods with timing and metadata capture, but does not add any network calls in the LLM request path. All telemetry is flushed asynchronously after the LLM call completes.

### Can I flush telemetry manually for long-running agents?

Yes. Use `waxell.flush()` (async) or `waxell.flush_sync()` to flush buffered data without waiting for the context to exit:

```python
@waxell.observe(agent_name="long-runner")
async def long_running_agent():
    for batch in data:
        process(batch)
        await waxell.flush()  # Send data to control plane now
```

---

## Framework Integration

### Does Waxell work with LangChain?

Yes. Use the callback handler:

```python
from waxell_observe.integrations.langchain import WaxellLangChainHandler

handler = WaxellLangChainHandler(agent_name="my-chain")
result = chain.invoke(input, config={"callbacks": [handler]})
handler.flush_sync(result={"output": result})
```

### Does Waxell work with CrewAI / LlamaIndex / AutoGen?

Yes. These frameworks are auto-instrumented by `waxell.init()`. LLM calls, tool invocations, and agent delegations are captured automatically.

### Can I use Waxell with multiple LLM providers in the same agent?

Yes. Auto-instrumentation patches all installed provider SDKs. A single `@observe` run can contain calls to OpenAI, Anthropic, Groq, and others -- all captured in the same trace.


--------------------------------------------------------------------------------
# Common Errors
URL: https://waxell.ai/docs/observe/troubleshooting/common-errors
Description: Error reference for the Waxell Observe SDK -- what each error means and how to fix it.
--------------------------------------------------------------------------------

# Common Errors

## SDK Errors

### PolicyViolationError

```
waxell_observe.errors.PolicyViolationError: Policy check returned action=block: Budget limit exceeded for agent 'my-agent'
```

**What it means:** A governance policy blocked your agent from running. This can happen at:
- **Pre-execution** (on context entry or decorator invocation): The agent was blocked before any code ran.
- **Mid-execution** (during `record_step()` with `mid_execution_governance=True`): The agent was blocked while running.

**How to fix:**

1. Check the policy result for details:
   ```python
   try:
       result = await my_agent(query)
   except PolicyViolationError as e:
       print(e.policy_result.action)   # "block" or "throttle"
       print(e.policy_result.reason)   # Human-readable explanation
       print(e.policy_result.metadata) # Additional context
   ```

2. Review your policies in the Waxell dashboard under **Governance > Policies**.

3. To skip policy checks during development:
   ```python
   @waxell.observe(agent_name="my-agent", enforce_policy=False)
   ```

4. To add an approval workflow instead of hard blocking:
   ```python
   @waxell.observe(
       agent_name="my-agent",
       on_policy_block=waxell.prompt_approval,
   )
   ```

---

### ConfigurationError

```
waxell_observe.errors.ConfigurationError: WaxellObserveClient is not configured. Call configure() or pass api_url and api_key.
```

**What it means:** The SDK tried to make an API call but has no credentials configured.

**How to fix:**

Option 1 -- Use `init()` (recommended):
```python

waxell.init(api_key="wax_sk_...", api_url="https://acme.waxell.dev")
```

Option 2 -- Environment variables:
```bash


```

Option 3 -- CLI config file (`~/.waxell/config`):
```ini
[default]
api_url = https://acme.waxell.dev
api_key = wax_sk_...
```

Option 4 -- Direct client configuration:
```python
from waxell_observe import WaxellObserveClient
WaxellObserveClient.configure(api_url="https://acme.waxell.dev", api_key="wax_sk_...")
```

---

### PromptNotFoundError

```
waxell_observe.errors.PromptNotFoundError: Prompt 'summarizer' not found (label='production', version=0)
```

**What it means:** The SDK tried to fetch a prompt from the control plane prompt registry, but it doesn't exist or the specified label/version isn't available.

**How to fix:**

1. Check the prompt name and label in the Waxell dashboard under **Prompts**.
2. Ensure the prompt has been published with the label you're requesting:
   ```python
   # Fetch by label
   prompt = await client.get_prompt("summarizer", label="production")

   # Fetch latest version (no label needed)
   prompt = await client.get_prompt("summarizer")
   ```
3. If the prompt was recently created, the SDK caches prompt lookups for 60 seconds. Wait or restart your application.

---

### PromptGuardError

```
waxell_observe.errors.PromptGuardError: Prompt guard detected violations: [PII: email address found]
```

**What it means:** The Prompt Guard feature detected sensitive content (PII, credentials, or injection attempts) in an LLM input and the guard action is set to `"block"`.

**How to fix:**

1. Review the detected violations:
   ```python
   try:
       response = await client.chat.completions.create(...)
   except PromptGuardError as e:
       print(e)  # Shows violation details
   ```

2. Change the guard action to `"warn"` (log and continue) or `"redact"` (replace with placeholders):
   ```python
   waxell.init(prompt_guard=True, prompt_guard_action="redact")
   ```

3. To disable prompt guard entirely:
   ```python
   waxell.init(prompt_guard=False)
   ```

---

### ObserveError

```
waxell_observe.errors.ObserveError: ...
```

**What it means:** Base error class for all Waxell Observe errors. If you catch `ObserveError`, you catch all SDK-specific errors.

```python
from waxell_observe.errors import ObserveError

try:
    result = await my_agent(query)
except ObserveError as e:
    # Catches PolicyViolationError, ConfigurationError, PromptNotFoundError, etc.
    print(f"Waxell error: {e}")
```

---

## Runtime Warnings

### "waxell-observe not configured, skipping run"

```
WARNING waxell_observe - Client not configured, skipping run start for 'my-agent'
```

**What it means:** The SDK has no API credentials configured. The `@observe` decorator or `WaxellContext` ran your function normally but didn't create a run on the control plane.

**How to fix:** Configure credentials via `init()`, environment variables, or `~/.waxell/config`. See [ConfigurationError](#configurationerror) above.

---

### "Failed to start run" / "Failed to complete run"

```
WARNING waxell_observe - Failed to start run for 'my-agent': ConnectionError
WARNING waxell_observe - Failed to complete run abc-123: TimeoutError
```

**What it means:** The SDK couldn't reach the control plane API. Your agent code executed normally -- the warning is about telemetry loss only.

**How to fix:**
1. Check your network connectivity to the control plane URL.
2. Verify the `api_url` is correct (should be `https://<tenant>.waxell.dev`).
3. Enable debug mode for more details: `waxell.init(debug=True)`.
4. These warnings are expected if you're running offline or in an environment without control plane access.

---

### "OTel tracing initialization failed"

```
WARNING waxell_observe - OTel tracing initialization failed: ModuleNotFoundError: No module named 'opentelemetry'
```

**What it means:** The OpenTelemetry packages are not installed. The SDK falls back to HTTP-only telemetry (which works fine for most use cases).

**How to fix:** If you need OTel spans (e.g., for Grafana, Jaeger, or Datadog):
```bash
pip install waxell-observe[otel]
```

If you don't need distributed tracing, this warning is safe to ignore.

---

### "Sync wrapper using ThreadPoolExecutor"

```
WARNING waxell_observe - Running sync function in ThreadPoolExecutor (event loop already running)
```

**What it means:** You decorated a sync function with `@observe` and called it from inside an async event loop (e.g., Jupyter notebook, FastAPI, uvicorn). The decorator falls back to a thread pool to avoid blocking.

**How to fix:** Make the decorated function async:

```python
# Before (triggers warning in async contexts)
@waxell.observe(agent_name="my-agent")
def my_sync_function(query: str):
    return process(query)

# After (no warning)
@waxell.observe(agent_name="my-agent")
async def my_async_function(query: str):
    return await process(query)
```

---

## Import Errors

### "waxell-observe core modules are not available for this platform"

```
ImportError: waxell-observe core modules are not available for this platform.
Install a pre-built wheel: pip install --only-binary=waxell-observe waxell-observe
```

**What it means:** The package was installed from source but a compiled module failed to build.

**How to fix:**
```bash
pip install --only-binary=waxell-observe waxell-observe
```

---

### "No module named 'waxell_observe'"

```
ModuleNotFoundError: No module named 'waxell_observe'
```

**How to fix:**
```bash
pip install waxell-observe
```

Make sure you're installing in the correct virtual environment:
```bash
which python          # Check active environment
pip install waxell-observe
python -c "import waxell_observe; print(waxell_observe.__version__)"
```

---

## Debugging Tips

### Enable debug mode

Debug mode enables verbose logging and console span export:

```python
waxell.init(debug=True)
```

Or via environment variable:
```bash
WAXELL_DEBUG=true python my_agent.py
```

### Use `diagnose()` to check SDK state

```python

waxell.init()

info = waxell.diagnose()
print(info)
# {
#   "sdk_version": "0.0.40",
#   "initialized": True,
#   "active_instrumentors": ["openai", "anthropic"],
#   "detected_libraries": {"openai": "1.52.0", "anthropic": "0.40.0"},
#   "config": {"api_url": "https://acme.waxell.dev", "api_key_set": True},
#   "tracing": {"enabled": True, "provider": "WaxellTracerProvider"},
# }
```

### Check if a context is active

```python
ctx = waxell.get_context()
if ctx:
    print(f"Active run: {ctx.run_id}")
else:
    print("No active context -- convenience functions are no-ops")
```


--------------------------------------------------------------------------------
# Common Mistakes
URL: https://waxell.ai/docs/observe/troubleshooting/common-mistakes
Description: Anti-patterns and gotchas when using the Waxell Observe SDK -- and how to fix them.
--------------------------------------------------------------------------------

# Common Mistakes

## 1. Importing LLM SDKs before `init()`

**Wrong:**
```python
from openai import OpenAI          # Gets un-patched OpenAI

waxell.init()                      # Too late -- OpenAI already imported

client = OpenAI()
client.chat.completions.create(...)  # NOT auto-instrumented
```

**Right:**
```python

waxell.init()                      # Patches OpenAI module

from openai import OpenAI          # Gets patched version
client = OpenAI()
client.chat.completions.create(...)  # Auto-instrumented
```

**Alternative** -- use drop-in imports (order doesn't matter):
```python
from waxell_observe.openai import openai

client = openai.OpenAI()
client.chat.completions.create(...)  # Always auto-instrumented
```

---

## 2. Using behavior decorators without `@observe`

Behavior decorators (`@tool`, `@decision`, `@retrieval`, etc.) only record data when called inside an `@observe` or `WaxellContext` scope. Without a parent scope, they're silent no-ops.

**Wrong** -- no trace is created:
```python
@waxell.tool(tool_type="api")
async def search(query: str):
    return await api.search(query)

@waxell.decision(name="route")
async def route(query: str):
    return {"chosen": "search"}

# These run fine but nothing is recorded
await route("test")
await search("test")
```

**Right** -- wrap the entry point with `@observe`:
```python
@waxell.tool(tool_type="api")
async def search(query: str):
    return await api.search(query)

@waxell.decision(name="route")
async def route(query: str):
    return {"chosen": "search"}

@waxell.observe(agent_name="my-agent")  # Creates the run scope
async def run_agent(query: str):
    decision = await route(query)
    results = await search(query)
    return results
```

---

## 3. Recording LLM calls manually when auto-instrumentation handles it

If you called `waxell.init()`, LLM calls are captured automatically. Manually recording them creates duplicates.

**Wrong** -- double-counted:
```python
waxell.init()  # Auto-instruments OpenAI

@waxell.observe(agent_name="my-agent")
async def my_agent(query: str, waxell_ctx=None):
    response = await client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": query}],
    )  # Already auto-captured

    # This creates a DUPLICATE record
    if waxell_ctx:
        waxell_ctx.record_llm_call(model="gpt-4o", tokens_in=100, tokens_out=50)

    return response.choices[0].message.content
```

**Right** -- let auto-instrumentation handle it:
```python
waxell.init()

@waxell.observe(agent_name="my-agent")
async def my_agent(query: str):
    response = await client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": query}],
    )  # Auto-captured with accurate token counts and cost

    waxell.score("quality", 0.9)  # Add enrichment, not duplicate LLM records
    return response.choices[0].message.content
```

Only use `ctx.record_llm_call()` when auto-instrumentation isn't available (e.g., custom HTTP calls to LLM endpoints, or unsupported providers).

---

## 4. Forgetting to guard `waxell_ctx` in unit tests

If your function uses the injected `waxell_ctx`, it will be `None` when called without `@observe` (e.g., in tests).

**Wrong** -- crashes in tests:
```python
@waxell.observe(agent_name="my-agent")
async def my_agent(query: str, waxell_ctx=None):
    response = await call_llm(query)
    waxell_ctx.record_score("quality", 0.9)  # AttributeError: NoneType has no attribute 'record_score'
    return response
```

**Right** -- guard or use convenience functions:
```python
# Option 1: Guard with if
@waxell.observe(agent_name="my-agent")
async def my_agent(query: str, waxell_ctx=None):
    response = await call_llm(query)
    if waxell_ctx:
        waxell_ctx.record_score("quality", 0.9)
    return response

# Option 2: Use convenience functions (preferred -- they're no-ops outside a context)
@waxell.observe(agent_name="my-agent")
async def my_agent(query: str):
    response = await call_llm(query)
    waxell.score("quality", 0.9)  # Safe everywhere
    return response
```

---

## 5. Blocking on `waxell.flush()` in sync code

`waxell.flush()` is async. Calling it in sync code with `asyncio.run()` can cause issues if an event loop is already running.

**Wrong:**
```python


@waxell.observe(agent_name="my-agent")
def sync_agent(query: str):
    result = process(query)
    asyncio.run(waxell.flush())  # RuntimeError if event loop is running
    return result
```

**Right** -- use the sync variant:
```python
@waxell.observe(agent_name="my-agent")
def sync_agent(query: str):
    result = process(query)
    waxell.flush_sync()  # Works in sync code
    return result
```

Note: you usually don't need to flush manually. The context flushes automatically on exit.

---

## 6. Using `WaxellContext` when `@observe` would suffice

`WaxellContext` is powerful but verbose. For single-function agents, `@observe` is cleaner.

**Verbose:**
```python
async def my_agent(query: str):
    async with WaxellContext(
        agent_name="my-agent",
        inputs={"query": query},
    ) as ctx:
        response = await call_llm(query)
        ctx.record_score("quality", 0.9)
        ctx.set_tag("type", "qa")
        ctx.set_result({"answer": response})
        return response
```

**Clean:**
```python
@waxell.observe(agent_name="my-agent")
async def my_agent(query: str):
    response = await call_llm(query)
    waxell.score("quality", 0.9)
    waxell.tag("type", "qa")
    return response  # Auto-captured as result
```

Reserve `WaxellContext` for cases where decorators don't fit: multi-step orchestration, batch loops, conditional context creation, or explicit lifecycle control.

---

## 7. Nesting `@observe` decorators unintentionally

Each `@observe` creates a separate run. Nesting them creates parent-child runs, which may not be what you want.

**Creates 2 runs per call:**
```python
@waxell.observe(agent_name="outer")
async def outer(query: str):
    return await inner(query)

@waxell.observe(agent_name="inner")
async def inner(query: str):
    return await call_llm(query)
```

If you want a single run with sub-steps, use `@observe` on the outer function and behavior decorators on inner functions:

**Creates 1 run with a tool span:**
```python
@waxell.observe(agent_name="my-agent")
async def outer(query: str):
    return await inner(query)

@waxell.tool(tool_type="function")
async def inner(query: str):
    return await call_llm(query)
```

Nested `@observe` is correct for multi-agent architectures where each agent is a distinct run with its own lifecycle and policy checks.

---

## 8. Expecting `@observe` to work without configuration

`@observe` requires a configured client to create runs on the control plane. Without configuration, it logs a warning and runs your function without observability.

**Silent failure:**
```python
# No init(), no env vars, no config file
@waxell.observe(agent_name="my-agent")
async def my_agent(query: str):
    return await call_llm(query)

await my_agent("test")
# WARNING: Client not configured, skipping run start
# Function runs fine, but no data in dashboard
```

**Explicit setup:**
```python

waxell.init(api_key="wax_sk_...", api_url="https://acme.waxell.dev")

@waxell.observe(agent_name="my-agent")
async def my_agent(query: str):
    return await call_llm(query)
```

---

## 9. Mixing up `waxell.tag()` and `waxell.metadata()`

Tags are string-only and searchable in the dashboard and Grafana. Metadata accepts any JSON type but is for contextual information.

**Wrong** -- complex values as tags:
```python
waxell.tag("config", '{"temperature": 0.7}')  # Stored as a string, not queryable
```

**Right:**
```python
waxell.tag("environment", "production")        # String tag -- queryable
waxell.tag("model", "gpt-4o")                  # String tag -- queryable
waxell.metadata("config", {"temperature": 0.7})  # Structured data -- contextual
```

---

## 10. Not calling `handler.flush_sync()` with LangChain

The `WaxellLangChainHandler` buffers data during chain execution. If you don't flush, the run is never completed.

**Wrong** -- incomplete run:
```python
handler = WaxellLangChainHandler(agent_name="my-chain")
result = chain.invoke(input, config={"callbacks": [handler]})
# Run is started but never completed -- shows as "running" forever in dashboard
```

**Right:**
```python
handler = WaxellLangChainHandler(agent_name="my-chain")
result = chain.invoke(input, config={"callbacks": [handler]})
handler.flush_sync(result={"output": result.content})
# Run is completed with result
```


--------------------------------------------------------------------------------
# REST API Reference
URL: https://waxell.ai/docs/observe/api/endpoints
Description: Complete REST API reference for the Waxell Observe endpoints, including runs, policy checks, events, model costs, LLM calls, sessions, users, scoring, and prompts.
--------------------------------------------------------------------------------

# REST API Reference

All Observe API endpoints live under `/api/v1/observe/` on your Waxell control plane.

## Authentication

Every request must include one of:

- **API key header**: `X-Wax-Key: wax_sk_...`
- **Bearer token**: `Authorization: Bearer wax_sk_...`

**INFO**
Endpoints under `/api/v1/observe/` use **API key authentication** (for SDK/programmatic access).

Endpoints under `/api/v1/observability/`, `/api/v1/evaluations/`, and `/api/v1/prompts/` use **session authentication** (for the web UI). These require an authenticated user session (cookie-based or JWT).


---

## Runs

### Start a Run

Start a new execution run for an agent.

```
POST /api/v1/observe/runs/start/
```

**Request Body:**

```json
{
  "agent_name": "my-agent",
  "workflow_name": "default",
  "inputs": {},
  "metadata": {},
  "trace_id": ""
}
```

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `agent_name` | `string` | Yes | Name of the agent |
| `workflow_name` | `string` | No | Workflow name (default: `"default"`) |
| `inputs` | `object` | No | Input data for this run |
| `metadata` | `object` | No | Arbitrary metadata |
| `trace_id` | `string` | No | External trace ID for correlation |

**Response:**

```json
{
  "run_id": "run_abc123",
  "workflow_id": "wf_xyz789",
  "started_at": "2025-01-15T10:30:00Z"
}
```

**Example:**

```bash
curl -X POST "https://acme.waxell.dev/api/v1/observe/runs/start/" \
  -H "X-Wax-Key: wax_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "agent_name": "support-bot",
    "workflow_name": "handle-ticket",
    "inputs": {"query": "How do I reset my password?"}
  }'
```

---

### Complete a Run

Mark an execution run as completed.

```
POST /api/v1/observe/runs/{run_id}/complete/
```

**Request Body:**

```json
{
  "result": {},
  "status": "success",
  "error": "",
  "steps": []
}
```

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `result` | `object` | No | Result data from the run |
| `status` | `string` | No | `"success"` or `"error"` (default: `"success"`) |
| `error` | `string` | No | Error message if the run failed |
| `steps` | `array` | No | Additional steps to record with completion |

**Response:**

```json
{
  "status": "ok"
}
```

**Example:**

```bash
curl -X POST "https://acme.waxell.dev/api/v1/observe/runs/run_abc123/complete/" \
  -H "X-Wax-Key: wax_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "result": {"answer": "Click Forgot Password on the login page."},
    "status": "success"
  }'
```

---

### Record LLM Calls

Record one or more LLM API calls for a run.

```
POST /api/v1/observe/runs/{run_id}/llm-calls/
```

**Request Body:**

```json
{
  "calls": [
    {
      "model": "gpt-4o",
      "tokens_in": 150,
      "tokens_out": 80,
      "cost": 0.001175,
      "task": "answer_question",
      "prompt_preview": "How do I reset...",
      "response_preview": "Click Forgot Password..."
    }
  ]
}
```

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `calls` | `array` | Yes | Array of LLM call objects |
| `calls[].model` | `string` | Yes | Model identifier |
| `calls[].tokens_in` | `integer` | Yes | Input token count |
| `calls[].tokens_out` | `integer` | Yes | Output token count |
| `calls[].cost` | `number` | No | Cost in USD |
| `calls[].task` | `string` | No | Label for this call |
| `calls[].prompt_preview` | `string` | No | Preview of the prompt |
| `calls[].response_preview` | `string` | No | Preview of the response |

**Response:**

```json
{
  "status": "ok",
  "count": 1
}
```

**Example:**

```bash
curl -X POST "https://acme.waxell.dev/api/v1/observe/runs/run_abc123/llm-calls/" \
  -H "X-Wax-Key: wax_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "calls": [
      {
        "model": "gpt-4o",
        "tokens_in": 150,
        "tokens_out": 80,
        "cost": 0.001175,
        "task": "answer_question"
      }
    ]
  }'
```

---

### Record Steps

Record execution steps for a run.

```
POST /api/v1/observe/runs/{run_id}/steps/
```

**Request Body:**

```json
{
  "steps": [
    {
      "step_name": "retrieve_context",
      "output": {"doc_count": 5}
    },
    {
      "step_name": "generate_answer",
      "output": {"length": 142}
    }
  ]
}
```

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `steps` | `array` | Yes | Array of step objects |
| `steps[].step_name` | `string` | Yes | Name identifying the step |
| `steps[].output` | `object` | No | Output data for the step |

**Response:**

```json
{
  "status": "ok",
  "count": 2
}
```

**Example:**

```bash
curl -X POST "https://acme.waxell.dev/api/v1/observe/runs/run_abc123/steps/" \
  -H "X-Wax-Key: wax_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "steps": [
      {"step_name": "retrieve_context", "output": {"doc_count": 5}},
      {"step_name": "generate_answer", "output": {"length": 142}}
    ]
  }'
```

---

### Record Scores (SDK)

Record scores (user feedback, quality metrics) for a run from the SDK.

```
POST /api/v1/observe/runs/{run_id}/scores/
```

**Authentication:** API key

**Request Body:**

```json
{
  "scores": [
    {
      "name": "accuracy",
      "data_type": "numeric",
      "numeric_value": 0.95,
      "comment": "Very accurate response"
    },
    {
      "name": "thumbs_up",
      "data_type": "boolean",
      "numeric_value": 1.0,
      "string_value": "true"
    }
  ]
}
```

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `scores` | `array` | Yes | Array of score objects |
| `scores[].name` | `string` | Yes | Score name (e.g. `"accuracy"`, `"thumbs_up"`) |
| `scores[].data_type` | `string` | No | One of `"numeric"`, `"categorical"`, `"boolean"` (default: `"numeric"`) |
| `scores[].numeric_value` | `number` | No | Numeric score value |
| `scores[].string_value` | `string` | No | String score value (for categorical/boolean) |
| `scores[].comment` | `string` | No | Free-text comment |

**Response:**

```json
{
  "recorded": 2
}
```

**Example:**

```bash
curl -X POST "https://acme.waxell.dev/api/v1/observe/runs/42/scores/" \
  -H "X-Wax-Key: wax_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "scores": [
      {"name": "accuracy", "data_type": "numeric", "numeric_value": 0.95},
      {"name": "thumbs_up", "data_type": "boolean", "numeric_value": 1.0}
    ]
  }'
```

---

### Fetch Prompt (SDK)

Retrieve a managed prompt for use in your agent code. Supports fetching by label, version number, or latest.

```
GET /api/v1/observe/prompts/{name}/
```

**Authentication:** API key

**Query Parameters:**

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `label` | `string` | No | Fetch the version this label points to (e.g. `"production"`) |
| `version` | `integer` | No | Fetch an exact version number. Takes precedence over `label` |

If neither `label` nor `version` is provided, the latest version (highest version number) is returned.

**Response:**

```json
{
  "name": "support-prompt",
  "version": 3,
  "prompt_type": "chat",
  "content": [
    {"role": "system", "content": "You are a helpful assistant for {{company}}."},
    {"role": "user", "content": "{{user_query}}"}
  ],
  "config": {"model": "gpt-4o", "temperature": 0.7},
  "labels": ["production"]
}
```

**Example:**

```bash
# Fetch by label
curl "https://acme.waxell.dev/api/v1/observe/prompts/support-prompt/?label=production" \
  -H "X-Wax-Key: wax_sk_..."

# Fetch specific version
curl "https://acme.waxell.dev/api/v1/observe/prompts/support-prompt/?version=2" \
  -H "X-Wax-Key: wax_sk_..."

# Fetch latest
curl "https://acme.waxell.dev/api/v1/observe/prompts/support-prompt/" \
  -H "X-Wax-Key: wax_sk_..."
```

---

### Record Behavior Spans

Record behavior tracking spans (tool calls, retrievals, decisions, reasoning, retries) for a run.

```
POST /api/v1/observe/runs/{run_id}/spans/
```

**Request Body:**

```json
{
  "spans": [
    {
      "name": "web_search",
      "kind": "tool",
      "status": "ok",
      "duration_ms": 250,
      "position": 1,
      "attributes": {"tool_type": "api"},
      "input_data": {"query": "latest news"},
      "output_data": {"results": ["..."]}
    },
    {
      "name": "retrieve:pinecone",
      "kind": "retriever",
      "status": "ok",
      "duration_ms": 120,
      "position": 2,
      "attributes": {"source": "pinecone", "doc_count": 5, "top_k": 10},
      "input_data": {"query": "billing FAQ", "top_k": 10},
      "output_data": {"documents": ["..."]}
    },
    {
      "name": "route_to_agent",
      "kind": "decision",
      "status": "ok",
      "duration_ms": null,
      "position": 3,
      "attributes": {"confidence": 0.95},
      "input_data": {"options": ["billing", "technical"], "reasoning": "User mentioned invoice"},
      "output_data": {"chosen": "billing", "confidence": 0.95}
    }
  ]
}
```

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `spans` | `array` | Yes | Array of span objects |
| `spans[].name` | `string` | Yes | Span name |
| `spans[].kind` | `string` | Yes | One of: `"tool"`, `"retriever"`, `"decision"`, `"reasoning"`, `"retry"` |
| `spans[].status` | `string` | No | `"ok"` or `"error"` (default: `"ok"`) |
| `spans[].duration_ms` | `integer` | No | Duration in milliseconds |
| `spans[].position` | `integer` | No | Ordering position within the run |
| `spans[].attributes` | `object` | No | Kind-specific attributes |
| `spans[].input_data` | `object` | No | Input data for this span |
| `spans[].output_data` | `object` | No | Output data for this span |

**Response:**

```json
{
  "status": "ok",
  "count": 3
}
```

**Example:**

```bash
curl -X POST "https://acme.waxell.dev/api/v1/observe/runs/run_abc123/spans/" \
  -H "X-Wax-Key: wax_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "spans": [
      {
        "name": "web_search",
        "kind": "tool",
        "status": "ok",
        "duration_ms": 250,
        "attributes": {"tool_type": "api"},
        "input_data": {"query": "latest news"}
      }
    ]
  }'
```

---

## Policy

### Check Policy

Check whether an agent is allowed to execute.

```
POST /api/v1/observe/policy-check/
```

**Request Body:**

```json
{
  "agent_name": "my-agent",
  "workflow_name": "default",
  "agent_id": ""
}
```

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `agent_name` | `string` | Yes | Name of the agent |
| `workflow_name` | `string` | No | Workflow name for scoped policies |
| `agent_id` | `string` | No | Specific agent instance ID |

**Response:**

```json
{
  "action": "allow",
  "reason": "",
  "metadata": {}
}
```

| Field | Type | Description |
|-------|------|-------------|
| `action` | `string` | One of: `"allow"`, `"block"`, `"warn"`, `"throttle"`, `"redact"`, `"skip"`, `"retry"` |
| `reason` | `string` | Human-readable explanation |
| `metadata` | `object` | Additional data (e.g., budget info) |

**Example:**

```bash
curl -X POST "https://acme.waxell.dev/api/v1/observe/policy-check/" \
  -H "X-Wax-Key: wax_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "agent_name": "support-bot",
    "workflow_name": "handle-ticket"
  }'
```

---

## Events

### Record Events

Record governance or audit events.

```
POST /api/v1/observe/events/
```

**Request Body:**

```json
{
  "events": [
    {
      "event_type": "policy_check",
      "agent_name": "my-agent",
      "action": "allow",
      "timestamp": "2025-01-15T10:30:00Z"
    }
  ]
}
```

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `events` | `array` | Yes | Array of event objects |

Event objects are flexible -- include any fields relevant to your audit requirements.

**Response:**

```json
{
  "status": "ok",
  "count": 1
}
```

**Example:**

```bash
curl -X POST "https://acme.waxell.dev/api/v1/observe/events/" \
  -H "X-Wax-Key: wax_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "events": [
      {
        "event_type": "manual_override",
        "agent_name": "support-bot",
        "operator": "admin@acme.com",
        "reason": "Emergency bypass"
      }
    ]
  }'
```

---

## Model Costs

### Get Model Costs

Retrieve the merged system + tenant model cost table.

```
GET /api/v1/observe/model-costs/
```

**Response:**

Returns the complete model cost table with tenant overrides applied on top of system defaults.

**Example:**

```bash
curl "https://acme.waxell.dev/api/v1/observe/model-costs/" \
  -H "X-Wax-Key: wax_sk_..."
```

---

### Set Model Cost Override

Set a tenant-level cost override for a specific model.

```
PUT /api/v1/observe/model-costs/{model_id}/
```

**Request Body:**

```json
{
  "input_cost_per_million": 2.00,
  "output_cost_per_million": 8.00
}
```

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `input_cost_per_million` | `number` | Yes | Input token cost per 1M tokens (USD) |
| `output_cost_per_million` | `number` | Yes | Output token cost per 1M tokens (USD) |

**Example:**

```bash
curl -X PUT "https://acme.waxell.dev/api/v1/observe/model-costs/gpt-4o/" \
  -H "X-Wax-Key: wax_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "input_cost_per_million": 2.00,
    "output_cost_per_million": 8.00
  }'
```

---

### Delete Model Cost Override

Remove a tenant-level cost override, reverting to system defaults.

```
DELETE /api/v1/observe/model-costs/{model_id}/
```

**Example:**

```bash
curl -X DELETE "https://acme.waxell.dev/api/v1/observe/model-costs/gpt-4o/" \
  -H "X-Wax-Key: wax_sk_..."
```

---

## Observability Endpoints (Session Auth)

These endpoints power the Observe web UI. They require an authenticated user session.

### List LLM Calls

Search and filter LLM calls across all runs.

```
GET /api/v1/observability/llm-calls/
```

**Authentication:** Session

**Query Parameters:**

| Parameter | Type | Description |
|-----------|------|-------------|
| `search` | `string` | Full-text search across model, task, prompt/response previews |
| `model` | `string` | Filter by model name (supports multiple via repeated param) |
| `agent` | `string` | Filter by agent name |
| `min_cost` | `number` | Minimum cost filter |
| `max_cost` | `number` | Maximum cost filter |
| `min_tokens` | `integer` | Minimum total token count |
| `max_tokens` | `integer` | Maximum total token count |
| `start` | `string` | Start date/time (ISO 8601) |
| `end` | `string` | End date/time (ISO 8601) |
| `sort` | `string` | Sort field. One of: `created_at`, `-created_at`, `cost`, `-cost`, `total_tokens`, `-total_tokens`, `model`, `-model` (default: `-created_at`) |
| `limit` | `integer` | Page size (default: 25, max: 100) |
| `offset` | `integer` | Pagination offset (default: 0) |

**Response:**

```json
{
  "results": [
    {
      "id": 42,
      "run_id": 17,
      "model": "gpt-4o",
      "task": "answer_question",
      "agent_name": "support-bot",
      "workflow_name": "handle-ticket",
      "tokens_in": 150,
      "tokens_out": 80,
      "total_tokens": 230,
      "cost": 0.001175,
      "created_at": "2025-01-15T10:30:00Z",
      "prompt_preview": "How do I reset...",
      "response_preview": "Click Forgot Password...",
      "prompt_hash": "abc123..."
    }
  ],
  "count": 1,
  "next": null,
  "previous": null,
  "aggregates": {
    "total_cost": 0.001175,
    "total_tokens": 230,
    "total_calls": 1,
    "models_used": ["gpt-4o"]
  }
}
```

**Example:**

```bash
curl "https://acme.waxell.dev/api/v1/observability/llm-calls/?model=gpt-4o&limit=10" \
  -H "Cookie: sessionid=..."
```

---

### Get LLM Call Detail

Retrieve full details for a single LLM call.

```
GET /api/v1/observability/llm-calls/{call_id}/
```

**Authentication:** Session

**Response:**

```json
{
  "id": 42,
  "run_id": 17,
  "model": "gpt-4o",
  "task": "answer_question",
  "agent_name": "support-bot",
  "workflow_name": "handle-ticket",
  "tokens_in": 150,
  "tokens_out": 80,
  "total_tokens": 230,
  "cost": 0.001175,
  "created_at": "2025-01-15T10:30:00Z",
  "prompt_preview": "Full prompt text...",
  "response_preview": "Full response text...",
  "prompt_hash": "abc123..."
}
```

**Example:**

```bash
curl "https://acme.waxell.dev/api/v1/observability/llm-calls/42/" \
  -H "Cookie: sessionid=..."
```

---

### List Distinct Models

Get the distinct model names used across all LLM calls. Useful for populating filter dropdowns.

```
GET /api/v1/observability/llm-calls/models/
```

**Authentication:** Session

**Response:**

```json
{
  "models": ["claude-3-5-sonnet", "gpt-4o", "gpt-4o-mini"]
}
```

**Example:**

```bash
curl "https://acme.waxell.dev/api/v1/observability/llm-calls/models/" \
  -H "Cookie: sessionid=..."
```

---

### List Sessions

List sessions with aggregated metrics. A session groups multiple runs that share a `session_id`.

```
GET /api/v1/observability/sessions/
```

**Authentication:** Session

**Query Parameters:**

| Parameter | Type | Description |
|-----------|------|-------------|
| `search` | `string` | Search by session ID |
| `agent` | `string` | Filter by agent name |
| `start` | `string` | Start date/time (ISO 8601) |
| `end` | `string` | End date/time (ISO 8601) |
| `sort` | `string` | One of: `last_activity`, `-last_activity`, `first_run`, `-first_run`, `run_count`, `-run_count` (default: `-last_activity`) |
| `limit` | `integer` | Page size (default: 25, max: 100) |
| `offset` | `integer` | Pagination offset (default: 0) |

**Response:**

```json
{
  "results": [
    {
      "session_id": "sess_a1b2c3d4e5f6g7h8",
      "run_count": 5,
      "first_run": "2025-01-15T10:00:00Z",
      "last_activity": "2025-01-15T10:15:00Z",
      "total_duration": 45.5,
      "total_cost": 0.0523,
      "total_tokens": 12500,
      "agents": ["support-bot", "research-agent"]
    }
  ],
  "count": 1,
  "next": null,
  "previous": null
}
```

**Example:**

```bash
curl "https://acme.waxell.dev/api/v1/observability/sessions/?agent=support-bot" \
  -H "Cookie: sessionid=..."
```

---

### Get Session Detail

Get full session details including all runs and aggregated metrics.

```
GET /api/v1/observability/sessions/{session_id}/
```

**Authentication:** Session

**Response:**

```json
{
  "session_id": "sess_a1b2c3d4e5f6g7h8",
  "aggregates": {
    "run_count": 3,
    "total_duration": 15.2,
    "total_cost": 0.0234,
    "total_tokens": 5600,
    "agents": ["support-bot"]
  },
  "runs": [
    {
      "id": 17,
      "agent_name": "support-bot",
      "workflow_name": "handle-ticket",
      "started_at": "2025-01-15T10:00:00Z",
      "completed_at": "2025-01-15T10:00:05Z",
      "duration": 5.1,
      "status": "success",
      "cost": 0.0078,
      "tokens": 1800,
      "trace_id": ""
    }
  ]
}
```

**Example:**

```bash
curl "https://acme.waxell.dev/api/v1/observability/sessions/sess_a1b2c3d4e5f6g7h8/" \
  -H "Cookie: sessionid=..."
```

---

### List Users

List tracked end-users with aggregate metrics. Users are identified by the `user_id` passed during run creation.

```
GET /api/v1/observability/users/
```

**Authentication:** Session

**Query Parameters:**

| Parameter | Type | Description |
|-----------|------|-------------|
| `search` | `string` | Search by user ID |
| `agent` | `string` | Filter by agent name |
| `start` | `string` | Start date/time (ISO 8601) |
| `end` | `string` | End date/time (ISO 8601) |
| `sort` | `string` | One of: `last_seen`, `-last_seen`, `first_seen`, `-first_seen`, `run_count`, `-run_count` (default: `-last_seen`) |
| `limit` | `integer` | Page size (default: 25, max: 100) |
| `offset` | `integer` | Pagination offset (default: 0) |

**Response:**

```json
{
  "results": [
    {
      "user_id": "user_abc123",
      "run_count": 42,
      "first_seen": "2025-01-01T08:00:00Z",
      "last_seen": "2025-01-15T10:30:00Z",
      "total_duration": 210.5,
      "total_cost": 0.5234,
      "total_tokens": 125000,
      "agents": ["support-bot", "research-agent"]
    }
  ],
  "count": 1,
  "next": null,
  "previous": null
}
```

**Example:**

```bash
curl "https://acme.waxell.dev/api/v1/observability/users/?sort=-run_count" \
  -H "Cookie: sessionid=..."
```

---

### Get User Detail

Get detailed information for a specific tracked user, including cost breakdown by model and recent runs.

```
GET /api/v1/observability/users/{user_id}/
```

**Authentication:** Session

**Response:**

```json
{
  "user_id": "user_abc123",
  "aggregates": {
    "run_count": 42,
    "total_duration": 210.5,
    "total_cost": 0.5234,
    "total_tokens": 125000,
    "agents": ["support-bot"],
    "first_seen": "2025-01-01T08:00:00Z",
    "last_seen": "2025-01-15T10:30:00Z"
  },
  "cost_by_model": [
    {
      "model": "gpt-4o",
      "total_cost": 0.45,
      "total_tokens": 100000,
      "call_count": 30
    }
  ],
  "runs": [
    {
      "id": 17,
      "agent_name": "support-bot",
      "workflow_name": "handle-ticket",
      "started_at": "2025-01-15T10:30:00Z",
      "completed_at": "2025-01-15T10:30:05Z",
      "duration": 5.1,
      "status": "success",
      "cost": 0.0078,
      "tokens": 1800
    }
  ]
}
```

**Example:**

```bash
curl "https://acme.waxell.dev/api/v1/observability/users/user_abc123/" \
  -H "Cookie: sessionid=..."
```

---

### Model Analytics

Get model usage analytics with time-series data for cost, token, and call-count trends.

```
GET /api/v1/observability/analytics/models/
```

**Authentication:** Session

**Query Parameters:**

| Parameter | Type | Description |
|-----------|------|-------------|
| `period` | `string` | Time period: `"1d"`, `"7d"`, or `"30d"` (default: `"7d"`) |
| `agent` | `string` | Filter by agent name |

**Response:**

```json
{
  "period": "7d",
  "aggregates": {
    "total_cost": 12.345,
    "total_tokens": 2500000,
    "total_calls": 1500
  },
  "model_totals": [
    {
      "model": "gpt-4o",
      "total_cost": 10.5,
      "total_tokens": 2000000,
      "total_calls": 1200,
      "percentage": 85.1
    }
  ],
  "time_series": [
    {
      "date": "2025-01-15",
      "model": "gpt-4o",
      "cost": 1.5,
      "tokens": 300000,
      "calls": 180
    }
  ]
}
```

**Example:**

```bash
curl "https://acme.waxell.dev/api/v1/observability/analytics/models/?period=30d" \
  -H "Cookie: sessionid=..."
```

---

## Scoring Endpoints (Session Auth)

Endpoints for creating and querying quality scores.

### List Scores

```
GET /api/v1/evaluations/scores/
```

**Authentication:** Session

**Query Parameters:**

| Parameter | Type | Description |
|-----------|------|-------------|
| `name` | `string` | Filter by score name |
| `source` | `string` | Filter by source (`"manual"`, `"sdk"`, `"evaluator"`) |
| `data_type` | `string` | Filter by data type (`"numeric"`, `"categorical"`, `"boolean"`) |
| `run_id` | `integer` | Filter by run ID |
| `llm_call_id` | `integer` | Filter by LLM call ID |
| `sort` | `string` | Sort field (default: `-created_at`). Allowed: `created_at`, `-created_at`, `name`, `-name`, `numeric_value`, `-numeric_value`, `source`, `-source` |
| `limit` | `integer` | Page size (default: 25, max: 100) |
| `offset` | `integer` | Pagination offset |

**Response:**

```json
{
  "results": [
    {
      "id": "uuid",
      "name": "accuracy",
      "data_type": "numeric",
      "source": "manual",
      "numeric_value": 0.95,
      "string_value": null,
      "comment": "Very accurate",
      "metadata": {},
      "author_user_id": "1",
      "evaluator_id": null,
      "evaluator_name": null,
      "run_id": "17",
      "run_agent_name": "support-bot",
      "llm_call_id": null,
      "llm_call_model": null,
      "created_at": "2025-01-15T10:30:00Z"
    }
  ],
  "count": 1,
  "next": null,
  "previous": null,
  "aggregates": {
    "total_count": 1,
    "avg_numeric_value": 0.95,
    "score_names": ["accuracy"]
  }
}
```

**Example:**

```bash
curl "https://acme.waxell.dev/api/v1/evaluations/scores/?name=accuracy&sort=-numeric_value" \
  -H "Cookie: sessionid=..."
```

---

### Create Score

Create a manual score for a run or LLM call.

```
POST /api/v1/evaluations/scores/create/
```

**Authentication:** Session

**Request Body:**

```json
{
  "run_id": 17,
  "name": "accuracy",
  "data_type": "numeric",
  "value": 0.95,
  "comment": "Very accurate response",
  "metadata": {}
}
```

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `name` | `string` | Yes | Score name |
| `run_id` | `integer` | Conditional | Run to score (at least one of `run_id` or `llm_call_id` required) |
| `llm_call_id` | `integer` | Conditional | LLM call to score |
| `data_type` | `string` | No | One of `"numeric"`, `"categorical"`, `"boolean"` (default: `"numeric"`) |
| `value` | `any` | Yes | Score value (number for numeric, string for categorical, bool for boolean) |
| `comment` | `string` | No | Free-text comment |
| `metadata` | `object` | No | Arbitrary metadata |

**Response:** `201 Created`

```json
{
  "id": "uuid",
  "name": "accuracy",
  "data_type": "numeric",
  "source": "manual",
  "numeric_value": 0.95,
  "string_value": null,
  "run_id": "17",
  "llm_call_id": null,
  "created_at": "2025-01-15T10:30:00Z"
}
```

**Example:**

```bash
curl -X POST "https://acme.waxell.dev/api/v1/evaluations/scores/create/" \
  -H "Cookie: sessionid=..." \
  -H "Content-Type: application/json" \
  -d '{
    "run_id": 17,
    "name": "accuracy",
    "data_type": "numeric",
    "value": 0.95,
    "comment": "Very accurate response"
  }'
```

---

### Score Analytics

Get score distributions and time-series trends.

```
GET /api/v1/evaluations/scores/analytics/
```

**Authentication:** Session

**Query Parameters:**

| Parameter | Type | Description |
|-----------|------|-------------|
| `name` | `string` | Filter by score name |
| `period` | `string` | Time period: `"1d"`, `"7d"`, `"30d"` (default: `"7d"`) |
| `agent` | `string` | Filter by agent name |

**Response:**

```json
{
  "period": "7d",
  "since": "2025-01-08T10:30:00Z",
  "distributions": [
    {
      "name": "accuracy",
      "data_type": "numeric",
      "total_count": 150,
      "avg": 0.87,
      "min": 0.2,
      "max": 1.0,
      "time_series": [
        {"date": "2025-01-15", "avg": 0.89, "count": 25}
      ]
    },
    {
      "name": "sentiment",
      "data_type": "categorical",
      "total_count": 80,
      "value_counts": [
        {"value": "positive", "count": 50},
        {"value": "neutral", "count": 20},
        {"value": "negative", "count": 10}
      ]
    }
  ]
}
```

**Example:**

```bash
curl "https://acme.waxell.dev/api/v1/evaluations/scores/analytics/?name=accuracy&period=30d" \
  -H "Cookie: sessionid=..."
```

---

## Prompt Endpoints (Session Auth)

Endpoints for managing versioned prompts, labels, and the playground.

### List Prompts

```
GET /api/v1/prompts/
```

**Authentication:** Session

**Response:**

```json
{
  "results": [
    {
      "id": "uuid",
      "name": "support-prompt",
      "description": "Main support agent prompt",
      "prompt_type": "chat",
      "tags": ["support", "production"],
      "latest_version": 3,
      "version_count": 3,
      "labels": ["production", "staging"],
      "created_at": "2025-01-10T08:00:00Z",
      "updated_at": "2025-01-15T10:00:00Z"
    }
  ],
  "count": 1
}
```

**Example:**

```bash
curl "https://acme.waxell.dev/api/v1/prompts/" \
  -H "Cookie: sessionid=..."
```

---

### Create Prompt

Create a new prompt with an initial version (version 1).

```
POST /api/v1/prompts/
```

**Authentication:** Session

**Request Body:**

```json
{
  "name": "support-prompt",
  "description": "Main support agent prompt",
  "prompt_type": "chat",
  "content": [
    {"role": "system", "content": "You are a helpful assistant for {{company}}."},
    {"role": "user", "content": "{{user_query}}"}
  ],
  "config": {"model": "gpt-4o", "temperature": 0.7},
  "tags": ["support"],
  "commit_message": "Initial version"
}
```

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `name` | `string` | Yes | Unique prompt name |
| `content` | `any` | Yes | Prompt content. String for text prompts, array of message objects for chat prompts |
| `description` | `string` | No | Description |
| `prompt_type` | `string` | No | `"text"` or `"chat"` (default: `"text"`) |
| `config` | `object` | No | Default config (model, temperature, max_tokens) |
| `tags` | `array` | No | Searchable tags |
| `metadata` | `object` | No | Arbitrary metadata |
| `commit_message` | `string` | No | Version commit message (default: `"Initial version"`) |

**Response:** `201 Created`

**Example:**

```bash
curl -X POST "https://acme.waxell.dev/api/v1/prompts/" \
  -H "Cookie: sessionid=..." \
  -H "Content-Type: application/json" \
  -d '{
    "name": "support-prompt",
    "prompt_type": "chat",
    "content": [{"role": "system", "content": "You are a helpful assistant."}],
    "config": {"model": "gpt-4o"}
  }'
```

---

### Get/Update/Delete Prompt

```
GET    /api/v1/prompts/{prompt_id}/
PUT    /api/v1/prompts/{prompt_id}/
DELETE /api/v1/prompts/{prompt_id}/
```

**Authentication:** Session

- **GET** returns prompt with all versions and labels
- **PUT** updates metadata fields only (name, description, tags). To update content, create a new version
- **DELETE** permanently deletes the prompt and all versions/labels

**Example:**

```bash
# Get prompt detail with all versions
curl "https://acme.waxell.dev/api/v1/prompts/uuid-here/" \
  -H "Cookie: sessionid=..."

# Update metadata
curl -X PUT "https://acme.waxell.dev/api/v1/prompts/uuid-here/" \
  -H "Cookie: sessionid=..." \
  -H "Content-Type: application/json" \
  -d '{"description": "Updated description", "tags": ["support", "v2"]}'
```

---

### List Versions

```
GET /api/v1/prompts/{prompt_id}/versions/
```

**Authentication:** Session

**Response:**

```json
{
  "results": [
    {
      "version": 3,
      "content_preview": "[{\"role\": \"system\", \"content\": \"You are...",
      "config": {"model": "gpt-4o"},
      "content_hash": "sha256...",
      "commit_message": "Improved system prompt",
      "created_by": "admin@acme.com",
      "labels": ["production"],
      "created_at": "2025-01-15T10:00:00Z"
    }
  ],
  "count": 3
}
```

**Example:**

```bash
curl "https://acme.waxell.dev/api/v1/prompts/uuid-here/versions/" \
  -H "Cookie: sessionid=..."
```

---

### Create Version

Create a new version of a prompt. The version number auto-increments.

```
POST /api/v1/prompts/{prompt_id}/versions/
```

**Authentication:** Session

**Request Body:**

```json
{
  "content": [
    {"role": "system", "content": "Updated system prompt for {{company}}."},
    {"role": "user", "content": "{{user_query}}"}
  ],
  "config": {"model": "gpt-4o", "temperature": 0.5},
  "commit_message": "Improved system prompt tone"
}
```

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `content` | `any` | Yes | Prompt content |
| `config` | `object` | No | Config overrides for this version |
| `commit_message` | `string` | No | Commit message |

**Response:** `201 Created`

```json
{
  "version": 4,
  "content_hash": "sha256...",
  "commit_message": "Improved system prompt tone",
  "created_by": "admin@acme.com",
  "created_at": "2025-01-15T10:30:00Z"
}
```

**Example:**

```bash
curl -X POST "https://acme.waxell.dev/api/v1/prompts/uuid-here/versions/" \
  -H "Cookie: sessionid=..." \
  -H "Content-Type: application/json" \
  -d '{"content": "Updated prompt text", "commit_message": "Simplified wording"}'
```

---

### Get Version Detail

```
GET /api/v1/prompts/{prompt_id}/versions/{version_num}/
```

**Authentication:** Session

**Response:**

```json
{
  "prompt_id": "uuid",
  "prompt_name": "support-prompt",
  "prompt_type": "chat",
  "version": 3,
  "content": [...],
  "config": {"model": "gpt-4o"},
  "content_hash": "sha256...",
  "commit_message": "Improved system prompt",
  "created_by": "admin@acme.com",
  "labels": ["production"],
  "created_at": "2025-01-15T10:00:00Z"
}
```

**Example:**

```bash
curl "https://acme.waxell.dev/api/v1/prompts/uuid-here/versions/3/" \
  -H "Cookie: sessionid=..."
```

---

### Set/Remove Label

Labels are movable pointers to specific versions (e.g., `"production"`, `"staging"`).

```
PUT    /api/v1/prompts/{prompt_id}/labels/{label}/
DELETE /api/v1/prompts/{prompt_id}/labels/{label}/
```

**Authentication:** Session

**PUT Request Body:**

```json
{
  "version": 3
}
```

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `version` | `integer` | Yes | Version number to point the label to |

**PUT Response:**

```json
{
  "label": "production",
  "version": 3,
  "prompt_id": "uuid",
  "created": false
}
```

**Example:**

```bash
# Set production label to version 3
curl -X PUT "https://acme.waxell.dev/api/v1/prompts/uuid-here/labels/production/" \
  -H "Cookie: sessionid=..." \
  -H "Content-Type: application/json" \
  -d '{"version": 3}'

# Remove staging label
curl -X DELETE "https://acme.waxell.dev/api/v1/prompts/uuid-here/labels/staging/" \
  -H "Cookie: sessionid=..."
```

---

### Playground: Execute Prompt

Execute a prompt against an LLM with variable substitution.

```
POST /api/v1/prompts/playground/
```

**Authentication:** Session

**Request Body:**

```json
{
  "content": [
    {"role": "system", "content": "You help users of {{company}}."},
    {"role": "user", "content": "{{query}}"}
  ],
  "config": {
    "model": "gpt-4o-mini",
    "temperature": 0.7,
    "max_tokens": 1024
  },
  "variables": {
    "company": "Acme Corp",
    "query": "How do I reset my password?"
  }
}
```

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `content` | `any` | Yes | Prompt content (string or chat messages) |
| `config` | `object` | No | LLM configuration |
| `config.model` | `string` | No | Model to use (default: `"gpt-4o-mini"`) |
| `config.temperature` | `number` | No | Sampling temperature (default: `0.7`) |
| `config.max_tokens` | `integer` | No | Max output tokens (default: `1024`) |
| `variables` | `object` | No | Template variables to replace `{{var}}` placeholders |

**Response:**

```json
{
  "output": "To reset your password, click the Forgot Password link...",
  "model": "gpt-4o-mini",
  "tokens_in": 45,
  "tokens_out": 120,
  "cost": 0.000079,
  "latency_ms": 1250
}
```

**Example:**

```bash
curl -X POST "https://acme.waxell.dev/api/v1/prompts/playground/" \
  -H "Cookie: sessionid=..." \
  -H "Content-Type: application/json" \
  -d '{
    "content": "Summarize: {{text}}",
    "config": {"model": "gpt-4o-mini"},
    "variables": {"text": "The quick brown fox..."}
  }'
```

---

### Playground: Compare Variants

Execute multiple prompt variants side-by-side for comparison.

```
POST /api/v1/prompts/playground/compare/
```

**Authentication:** Session

**Request Body:**

```json
{
  "variants": [
    {
      "content": "Summarize concisely: {{text}}",
      "config": {"model": "gpt-4o-mini", "temperature": 0.3},
      "variables": {"text": "..."}
    },
    {
      "content": "Provide a detailed summary of: {{text}}",
      "config": {"model": "gpt-4o", "temperature": 0.7},
      "variables": {"text": "..."}
    }
  ]
}
```

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `variants` | `array` | Yes | Array of variant objects (max 10). Each has `content`, `config`, and `variables` |

**Response:**

```json
{
  "results": [
    {"output": "...", "model": "gpt-4o-mini", "tokens_in": 30, "tokens_out": 50, "cost": 0.00003, "latency_ms": 800},
    {"output": "...", "model": "gpt-4o", "tokens_in": 35, "tokens_out": 150, "cost": 0.0016, "latency_ms": 2100}
  ]
}
```

**Example:**

```bash
curl -X POST "https://acme.waxell.dev/api/v1/prompts/playground/compare/" \
  -H "Cookie: sessionid=..." \
  -H "Content-Type: application/json" \
  -d '{
    "variants": [
      {"content": "Short: {{text}}", "config": {"model": "gpt-4o-mini"}, "variables": {"text": "hello"}},
      {"content": "Long: {{text}}", "config": {"model": "gpt-4o"}, "variables": {"text": "hello"}}
    ]
  }'
```

---

### Prompt Metrics

Get usage metrics per prompt version, linked via `content_hash` to actual LLM call records.

```
GET /api/v1/prompts/{prompt_id}/metrics/
```

**Authentication:** Session

**Response:**

```json
{
  "prompt_id": "uuid",
  "prompt_name": "support-prompt",
  "totals": {
    "call_count": 1500,
    "total_tokens": 350000,
    "total_cost": 3.45
  },
  "versions": [
    {"version": 1, "content_hash": "abc...", "call_count": 500, "total_tokens": 100000, "total_cost": 1.0},
    {"version": 2, "content_hash": "def...", "call_count": 1000, "total_tokens": 250000, "total_cost": 2.45}
  ]
}
```

**Example:**

```bash
curl "https://acme.waxell.dev/api/v1/prompts/uuid-here/metrics/" \
  -H "Cookie: sessionid=..."
```

---

## Memory Write Endpoints

These endpoints let external runtimes (agents deployed to IBM watsonx.ai, SageMaker, Vertex, Bedrock Agents, or any other third-party platform) populate the Waxell controlplane Memory tab from outside the runtime, over HTTPS. They're paired with `ctx.record_memory_write()` for trace-side visibility; together they fill out both the trace view and the Memory tab.

Auth is dual-mode: `X-Wax-Key: <api_key>` header **or** `Authorization: Bearer <api_key>`.

### Write Episodic Memory

```
POST /api/v1/observability/runs/{run_id}/memory/episodic/
```

Persists a dict, list, or value to an episodic memory slot tied to a specific run. Use this for structured per-tool data that's queryable in the Memory tab but doesn't need vector search.

**Request body:**

```json
{
  "slot_name": "tool.get_company_profile",
  "scope_key": "watsonx-react-agent:run-abc123",
  "memory_type": "dict",
  "data": {
    "tool": "get_company_profile",
    "args": {"company": "TechCorp"},
    "result_preview": "..."
  },
  "ttl_seconds": 86400,
  "max_items": 50
}
```

| Field | Type | Required | Description |
|---|---|---|---|
| `slot_name` | string | Yes | Logical slot identifier (e.g., `"tool.search_market_data"`) |
| `scope_key` | string | Yes | Isolation key — typically `"{agent_name}:{run_id}"` or `"{agent}:{user_id}"` |
| `memory_type` | string | No | `"dict"`, `"list"`, or `"value"`. Default: `"dict"` |
| `data` | any | Yes | Payload to store (max 256 KB JSON) |
| `ttl_seconds` | int | No | Auto-expiration in seconds |
| `max_items` | int | No | For `list` type: max items before eviction |

**Validation limits:**
- Max payload: 256 KB JSON
- Tenant scoping is enforced via the API key — slots are isolated per tenant automatically

### Write Semantic Memory

```
POST /api/v1/observability/runs/{run_id}/memory/semantic/
```

Persists a content string + pre-computed 1536-dim embedding for vector search. Use this for high-signal facts you want to recall by semantic similarity across runs.

**Request body:**

```json
{
  "slot_name": "company_facts",
  "scope_key": "watsonx-react-agent:run-abc123",
  "content": "CrowdStrike reported $1.1B revenue (+25% YoY) for Q3 FY26.",
  "content_embedding": [0.0123, -0.0456, ...],
  "importance": 0.95,
  "tags": ["financial", "earnings"],
  "fact_key": "q3-fy26-revenue",
  "source_tool": "get_company_profile",
  "metadata": {"quarter": "Q3 FY26"}
}
```

| Field | Type | Required | Description |
|---|---|---|---|
| `slot_name` | string | Yes | Logical slot identifier (e.g., `"company_facts"`) |
| `scope_key` | string | Yes | Isolation key |
| `content` | string | Yes | Fact text (max 8 KB) |
| `content_embedding` | float[1536] | Yes | Pre-computed embedding (1536 dimensions — matches OpenAI `text-embedding-3-small`) |
| `importance` | float | No | 0.0–1.0 ranking signal for recall (default: 0.7) |
| `tags` | string[] | No | Tags for filtering |
| `fact_key` | string | No | Dedup key — re-posting with the same `fact_key` updates the existing fact |
| `source_tool` | string | No | The tool/action that produced this fact (for provenance) |
| `metadata` | object | No | Arbitrary extra context |

**Validation limits:**
- Max content: 8 KB
- Embedding must be exactly 1536 floats (pgvector dimension constraint)
- Empty `content` → silent no-op

### Pattern: combining trace-side and tab-side memory

For a deployed agent, emit both records so the trace view AND the Memory tab populate:

```python
# Trace-side — appears in the run's span tree
ctx.record_memory_write(
    memory_type="episodic",
    content=f"{tool_name}({args}) -> {result[:400]}",
)

# Tab-side — populates the Memory tab in the controlplane UI
httpx.post(
    f"{WAXELL_API_URL}/api/v1/observability/runs/{ctx.run_id}/memory/episodic/",
    headers={"X-Wax-Key": WAXELL_API_KEY},
    json={
        "slot_name": f"tool.{tool_name}",
        "scope_key": f"my-agent:{ctx.run_id}",
        "memory_type": "dict",
        "data": {"tool": tool_name, "args": args, "result_preview": result[:400]},
    },
    timeout=10.0,
)
```

The HTTPS writes degrade gracefully — wrap them in try/except so a Memory tab outage never breaks the deliverable. See the [waxell-agent-examples repo](https://github.com/waxell-ai/waxell-agent-examples) for the full pattern.

---

## Error Responses

API errors return standard HTTP status codes with a JSON body:

```json
{
  "error": "Description of the error"
}
```

| Status Code | Meaning |
|-------------|---------|
| `400` | Bad request (invalid JSON, missing required fields) |
| `401` | Unauthorized (missing or invalid API key) |
| `404` | Not found (invalid run ID or model ID) |
| `409` | Conflict (duplicate name) |
| `429` | Rate limited |
| `500` | Internal server error |

## Next Steps

- [Python SDK Reference](./python-sdk) -- Use these endpoints through the Python client
- [Installation & Configuration](../installation) -- Set up API credentials


--------------------------------------------------------------------------------
# Python SDK Reference
URL: https://waxell.ai/docs/observe/api/python-sdk
Description: Complete API reference for all public classes, functions, and types in the waxell-observe Python package.
--------------------------------------------------------------------------------

# Python SDK Reference

This is the complete API reference for the `waxell-observe` package. All public symbols are exported from the top-level `waxell_observe` module.

```python
from waxell_observe import (
    # Top-level functions
    init,
    shutdown,
    generate_session_id,
    # Decorators
    observe,           # Agent run decorator (alias for waxell_agent)
    waxell_agent,      # Original agent decorator (identical to observe)
    tool,              # @tool decorator
    decision,          # @decision decorator
    retrieval,         # @retrieval decorator
    reasoning_dec,     # @reasoning decorator
    retry_dec,         # @retry decorator
    step_dec,          # @step decorator
    # Convenience functions (work within active context)
    tag,
    metadata,
    score,
    decide,
    step,
    reason,
    retrieve,
    retry,
    get_context,
    # Human-in-the-loop
    input,              # Drop-in replacement for input()
    human_turn,         # Context manager for non-terminal channels
    human_interaction,  # One-shot recording
    # Approval handlers
    prompt_approval,    # Terminal Y/N prompt
    auto_approve,       # Always approve (testing)
    auto_deny,          # Always deny (testing)
    # Core classes
    WaxellObserveClient,
    ObserveConfig,
    WaxellContext,
    HumanTurn,
    # Types
    ApprovalDecision,
    LlmCallInfo,
    PolicyCheckResult,
    PromptGuardResult,
    RunCompleteResult,
    RunInfo,
    # Errors
    ConfigurationError,
    ObserveError,
    PolicyViolationError,
    PromptGuardError,
)
```

---

## init

```python


waxell_observe.init(
    api_key: str = "",
    api_url: str = "",
    capture_content: bool = False,
    instrument: list[str] | None = None,
    exclude: list[str] | None = None,
    instrument_infra: bool = True,
    infra_libraries: list[str] | None = None,
    infra_exclude: list[str] | None = None,
    resource_attributes: dict | None = None,
    debug: bool = False,
    prompt_guard: bool = False,
    prompt_guard_server: bool = False,
    prompt_guard_action: str = "block",
    on_policy_block: Callable | None = None,
) -> None
```

One-line initialization for waxell-observe. This single call configures the HTTP client, initializes OTel tracing (if installed), and auto-instruments installed LLM libraries.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `api_key` | `str` | `""` | Waxell API key (`wax_sk_...`). Falls back to `WAXELL_API_KEY` env var |
| `api_url` | `str` | `""` | Waxell API URL. Falls back to `WAXELL_API_URL` env var |
| `capture_content` | `bool` | `False` | Include prompt/response content in OTel traces |
| `instrument` | `list[str] \| None` | `None` | Libraries to auto-instrument (e.g. `["openai", "anthropic"]`). `None` means auto-detect all installed libraries |
| `exclude` | `list[str] \| None` | `None` | Skip these libraries during auto-instrumentation (e.g. `["litellm", "mcp"]`). Falls back to `WAXELL_EXCLUDE` env var (comma-delimited). Takes precedence over `instrument` |
| `instrument_infra` | `bool` | `True` | Enable auto-instrumentation of infrastructure libraries (HTTP clients, databases, caches, queues). Falls back to `WAXELL_INSTRUMENT_INFRA` env var |
| `infra_libraries` | `list[str] \| None` | `None` | Only instrument these specific infra libraries (e.g. `["redis", "httpx"]`). `None` means auto-detect all |
| `infra_exclude` | `list[str] \| None` | `None` | Instrument all infra libraries except these (e.g. `["celery", "grpc"]`). Falls back to `WAXELL_INFRA_EXCLUDE` env var (comma-delimited) |
| `resource_attributes` | `dict \| None` | `None` | Custom OTel resource attributes applied to all spans (e.g. `{"deployment.environment": "production"}`) |
| `debug` | `bool` | `False` | Enable debug logging and console span export |
| `prompt_guard` | `bool` | `False` | Enable client-side prompt guard (PII, credential, injection detection). Falls back to `WAXELL_PROMPT_GUARD` env var |
| `prompt_guard_server` | `bool` | `False` | Also check server-side guard service (ML-powered via Presidio + HuggingFace). Falls back to `WAXELL_PROMPT_GUARD_SERVER` env var |
| `prompt_guard_action` | `str` | `"block"` | Action on violations: `"block"` (raise error), `"warn"` (log and continue), `"redact"` (replace with `##TYPE##`). Falls back to `WAXELL_PROMPT_GUARD_ACTION` env var |
| `on_policy_block` | `Callable \| None` | `None` | Default handler for all contexts when a policy blocks. Receives `PolicyViolationError`, returns `ApprovalDecision`. Built-in: `prompt_approval`, `auto_approve`, `auto_deny` |

**Behavior:**
- Checks the `WAXELL_OBSERVE` environment variable kill switch first. If set to `"false"`, `"0"`, or `"no"`, initialization is skipped entirely.
- Idempotent: calling `init()` multiple times is safe. Only the first call takes effect.
- OTel tracing failure does not block the HTTP path. If tracing initialization fails, a warning is logged and the HTTP-based telemetry continues to work.
- Auto-instrumentation failure does not block manual tracing.

**Example:**

```python


# Minimal setup -- auto-detects URL from env, instruments all installed libraries
waxell_observe.init(api_key="wax_sk_abc123")

# Full control
waxell_observe.init(
    api_key="wax_sk_abc123",
    api_url="https://acme.waxell.dev",
    capture_content=True,
    instrument=["openai", "anthropic"],
    debug=True,
)
```

---

## shutdown

```python


waxell_observe.shutdown() -> None
```

Shut down waxell-observe: flush pending traces and remove auto-instrumentation.

**Behavior:**
- Calls `shutdown_tracing()` to flush the OTel span processor and shut down the TracerProvider.
- Calls `uninstrument_all()` to remove monkey-patches from instrumented libraries.
- Resets the internal `_initialized` flag so `init()` can be called again.
- Safe to call even if `init()` was never called.

**Example:**

```python


waxell_observe.init(api_key="wax_sk_abc123")
atexit.register(waxell_observe.shutdown)
```

---

## generate_session_id

```python
from waxell_observe import generate_session_id

generate_session_id() -> str
```

Generate a random session ID for grouping related runs.

**Returns:** A string in the format `sess_` followed by 16 hex characters (e.g. `sess_a1b2c3d4e5f6g7h8`).

**Example:**

```python
from waxell_observe import generate_session_id, WaxellContext

session = generate_session_id()

async with WaxellContext(agent_name="agent-1", session_id=session) as ctx:
    ...

async with WaxellContext(agent_name="agent-2", session_id=session) as ctx:
    ...
```

---

## Top-Level Convenience Functions

These functions operate on the current `WaxellContext` in scope. They are no-ops when called outside of an active context, making them safe to use in code that may or may not be wrapped by `@observe` or `WaxellContext`.

### tag

```python


waxell_observe.tag(key: str, value: str) -> None
```

Set a searchable tag on the current context. No-op if no context is active.

**Example:**

```python
from waxell_observe import observe


@observe(agent_name="my-agent")
async def run_agent(query: str) -> str:
    waxell_observe.tag("environment", "production")
    waxell_observe.tag("pipeline", "rag-v2")
    return await process(query)
```

### metadata

```python


waxell_observe.metadata(key: str, value: Any) -> None
```

Set metadata on the current context. Values can be any JSON-serializable type. No-op if no context is active.

**Example:**

```python
waxell_observe.metadata("model_version", "gpt-4-turbo")
waxell_observe.metadata("config", {"temperature": 0.7})
```

### score

```python


waxell_observe.score(
    name: str,
    value: float | str | bool,
    data_type: str = "numeric",
    comment: str = "",
) -> None
```

Record a score on the current context. No-op if no context is active.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `name` | `str` | (required) | Score name |
| `value` | `float \| str \| bool` | (required) | Score value |
| `data_type` | `str` | `"numeric"` | `"numeric"`, `"categorical"`, or `"boolean"` |
| `comment` | `str` | `""` | Optional comment |

**Example:**

```python
waxell_observe.score("relevance", 0.95)
waxell_observe.score("helpful", True, data_type="boolean")
waxell_observe.score("category", "informational", data_type="categorical")
```

### decide

```python


waxell_observe.decide(
    name: str,
    chosen: str,
    options: list[str] | None = None,
    reasoning: str = "",
    confidence: float | None = None,
) -> None
```

Record a decision on the current context. No-op if no context is active.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `name` | `str` | (required) | Decision name |
| `chosen` | `str` | (required) | The selected option |
| `options` | `list[str] \| None` | `None` | Available choices |
| `reasoning` | `str` | `""` | Why this option was chosen |
| `confidence` | `float \| None` | `None` | Confidence score (0.0-1.0) |

**Example:**

```python
waxell_observe.decide(
    "route_query",
    chosen="semantic_search",
    options=["semantic", "keyword", "hybrid"],
    reasoning="Query contains natural language phrasing",
    confidence=0.9,
)
```

### step

```python
waxell_observe.step(name: str, output: dict | None = None) -> None
```

Record an execution step on the current context. No-op if no context is active.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `name` | `str` | (required) | Step name |
| `output` | `dict \| None` | `None` | Step output data |

**Example:**

```python
waxell_observe.step("preprocessing", output={"tokens": 150, "language": "en"})
waxell_observe.step("validation")  # output is optional
```

### reason

```python
waxell_observe.reason(
    step: str,
    thought: str,
    evidence: list[str] | None = None,
    conclusion: str = "",
) -> None
```

Record a reasoning step on the current context. No-op if no context is active.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `step` | `str` | (required) | Reasoning step name |
| `thought` | `str` | (required) | The reasoning thought process |
| `evidence` | `list[str] \| None` | `None` | Supporting evidence |
| `conclusion` | `str` | `""` | Final conclusion |

**Example:**

```python
waxell_observe.reason(
    "source_evaluation",
    thought="Document A is from a peer-reviewed journal",
    evidence=["Published 2024", "Cited 45 times"],
    conclusion="High reliability source",
)
```

### retrieve

```python
waxell_observe.retrieve(
    query: str,
    documents: list[dict],
    source: str = "",
    scores: list[float] | None = None,
) -> None
```

Record a retrieval operation on the current context. No-op if no context is active.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `query` | `str` | (required) | The search query |
| `documents` | `list[dict]` | (required) | Retrieved documents |
| `source` | `str` | `""` | Data source name |
| `scores` | `list[float] \| None` | `None` | Relevance scores |

**Example:**

```python
waxell_observe.retrieve(
    query="AI safety best practices",
    documents=[{"id": "doc1", "title": "Safety Guide"}],
    source="pinecone",
    scores=[0.95],
)
```

### retry

```python
waxell_observe.retry(
    attempt: int,
    reason: str,
    strategy: str = "retry",
    original_error: str = "",
    fallback_to: str = "",
) -> None
```

Record a retry/fallback event on the current context. No-op if no context is active.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `attempt` | `int` | (required) | Attempt number (1-indexed) |
| `reason` | `str` | (required) | Why the retry occurred |
| `strategy` | `str` | `"retry"` | Strategy: `"retry"`, `"fallback"`, `"circuit_break"` |
| `original_error` | `str` | `""` | Error that triggered the retry |
| `fallback_to` | `str` | `""` | Fallback target name |

**Example:**

```python
waxell_observe.retry(
    attempt=2,
    reason="Rate limit exceeded",
    strategy="fallback",
    original_error="429 Too Many Requests",
    fallback_to="gpt-4o-mini",
)
```

### get_context

```python


waxell_observe.get_context() -> WaxellContext | None
```

Get the current `WaxellContext` if one is active, otherwise `None`.

**Example:**

```python
ctx = waxell_observe.get_context()
if ctx:
    ctx.record_llm_call(model="gpt-4o", tokens_in=100, tokens_out=50)
```

### input

```python


waxell_observe.input(prompt: str = "", *, action: str = "input") -> str
```

Drop-in replacement for Python's built-in `input()`. Calls `input(prompt)` and records the prompt, response, and elapsed time as a `human_turn` IO span.

Falls back to plain `input()` if called outside a `WaxellContext`.

**Example:**

```python
answer = waxell_observe.input("Approve? (y/n): ")
```

### human_turn

```python


waxell_observe.human_turn(
    prompt: str = "",
    channel: str = "terminal",
    action: str = "",
    metadata: dict | None = None,
) -> HumanTurn | _NoOpHumanTurn
```

Returns a context manager that captures a human interaction as a timed IO span. Use for non-terminal channels (Slack, webhooks, UI).

Returns a no-op if called outside a `WaxellContext`.

**Example:**

```python
with waxell_observe.human_turn(prompt="Deploy?", channel="slack", action="approval") as turn:
    response = await wait_for_reaction()
    turn.set_response(response)
```

### human_interaction

```python


waxell_observe.human_interaction(
    prompt: str = "",
    response: str = "",
    channel: str = "terminal",
    action: str = "",
    elapsed_seconds: float | None = None,
    metadata: dict | None = None,
) -> None
```

Record a completed human interaction. One-shot alternative to `human_turn()` when you already have all the data.

No-op if called outside a `WaxellContext`.

**Example:**

```python
waxell_observe.human_interaction(
    prompt="Pick target",
    response="staging",
    channel="slack",
    elapsed_seconds=12.5,
)
```

### user_message

```python


waxell_observe.user_message(content: str, *, metadata: dict | None = None) -> None
```

Record an inbound user message on the current context. Use in interactive agents (chat, REPL) to make user inputs visible in the trace alongside LLM calls and tool invocations.

No-op if called outside a `WaxellContext`.

**Example:**

```python
waxell_observe.user_message("What's the weather in Paris?")
```

### agent_response

```python


waxell_observe.agent_response(content: str, *, metadata: dict | None = None) -> None
```

Record an outbound agent response on the current context. Use to trace what the agent communicated back to the user.

No-op if called outside a `WaxellContext`.

**Example:**

```python
waxell_observe.agent_response("It's currently 22°C and sunny in Paris.")
```

### communication

```python


waxell_observe.communication(
    *,
    channel: str,
    recipient: str = "",
    body: str = "",
    subject: str = "",
    metadata: dict | None = None,
) -> None
```

Record an outbound communication on the current context. Use to track messages sent via external channels (Slack, email, SMS, webhooks) for communication governance policy evaluation.

No-op if called outside a `WaxellContext`.

**Example:**

```python
waxell_observe.communication(
    channel="slack",
    recipient="#ops-alerts",
    body="Deployment completed",
    subject="Deploy Notification",
)
```

### approval_request

```python


waxell_observe.approval_request(
    action_type: str,
    approvers: list[str] | None = None,
    timeout_minutes: float | None = None,
    reason: str = "",
    metadata: dict | None = None,
) -> None
```

Record an approval request on the current context. Call after catching `PolicyViolationError` to record that an approval workflow was initiated.

No-op if called outside a `WaxellContext`.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `action_type` | `str` | (required) | The action awaiting approval |
| `approvers` | `list[str] \| None` | `None` | Who can approve (emails, usernames) |
| `timeout_minutes` | `float \| None` | `None` | Approval window duration |
| `reason` | `str` | `""` | Why approval is needed |
| `metadata` | `dict \| None` | `None` | Additional context |

### approval_response

```python


waxell_observe.approval_response(
    action_type: str,
    decision: str,
    approver: str = "",
    elapsed_seconds: float | None = None,
    metadata: dict | None = None,
) -> None
```

Record an approval decision on the current context. Call after the human-in-the-loop decides or the approval window expires.

No-op if called outside a `WaxellContext`.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `action_type` | `str` | (required) | The action that was awaiting approval |
| `decision` | `str` | (required) | `"approved"`, `"denied"`, or `"timeout"` |
| `approver` | `str` | `""` | Who approved/denied |
| `elapsed_seconds` | `float \| None` | `None` | Time from block to decision |
| `metadata` | `dict \| None` | `None` | Additional context |

### flush

```python


await waxell_observe.flush() -> None
```

Flush buffered data on the current context. Use in long-running contexts (REPLs, chat sessions) to make data visible in the UI before the context exits.

No-op if called outside a `WaxellContext`.

### flush_sync

```python


waxell_observe.flush_sync() -> None
```

Synchronous version of `flush()`. Use in sync code.

No-op if called outside a `WaxellContext`.

### diagnose

```python


waxell_observe.diagnose() -> dict
```

Introspect the SDK state. Returns a dict with:

| Key | Type | Description |
|-----|------|-------------|
| `sdk_version` | `str` | Package version (e.g. `"0.0.40"`) |
| `initialized` | `bool` | Whether `init()` was called |
| `active_instrumentors` | `list[str]` | Names of active instrumentors |
| `detected_libraries` | `dict[str, str]` | Installed library versions |
| `config` | `dict` | Current API URL, key status |
| `tracing` | `dict` | OTel availability and status |

**Example:**

```python

waxell.init()

info = waxell.diagnose()
print(info["active_instrumentors"])  # ["openai", "anthropic", ...]
print(info["sdk_version"])           # "0.0.40"
```

### prompt_approval

```python


waxell_observe.prompt_approval(error: PolicyViolationError) -> ApprovalDecision
```

Built-in `on_policy_block` handler. Prints a terminal banner with the block reason, approvers, and timeout, then prompts `y/n`.

**Example:**

```python
@waxell_observe.observe(
    agent_name="my-agent",
    enforce_policy=True,
    on_policy_block=waxell_observe.prompt_approval,
)
async def my_function():
    ...
```

### auto_approve / auto_deny

```python


waxell_observe.auto_approve(error: PolicyViolationError) -> ApprovalDecision
waxell_observe.auto_deny(error: PolicyViolationError) -> ApprovalDecision
```

Test helpers. `auto_approve` always returns `ApprovalDecision(approved=True)`. `auto_deny` always returns `ApprovalDecision(approved=False)`.

---

## Drop-in Imports

Pre-instrumented modules that you can import directly, no `init()` required:

```python
from waxell_observe.openai import openai
from waxell_observe.anthropic import anthropic
```

These modules are thin wrappers around the real SDKs with auto-instrumentation already applied. All OpenAI/Anthropic calls made through these imports are automatically traced.

**Example:**

```python
from waxell_observe.openai import openai

client = openai.OpenAI()
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}]
)
# Automatically traced with model, tokens, cost, latency
```

---

## Instrumentation Functions

### instrument_all

```python
from waxell_observe.instrumentors import instrument_all

instrument_all(libraries: list[str] | None = None) -> dict[str, bool]
```

Manually instrument LLM libraries. Called automatically by `init()`, but can be called directly if needed.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `libraries` | `list[str] \| None` | `None` | Libraries to instrument. `None` means auto-detect all installed |

**Returns:** A dict mapping library name to whether instrumentation succeeded (e.g. `{"openai": True, "anthropic": True}`).

**Supported libraries:** 200+ LLM providers, vector databases, embedding models, frameworks, and more. Core providers include `openai`, `anthropic`, `litellm`, `groq`, `huggingface`, `gemini`, `cohere`, `mistral`, `together`, `ai21`, `bedrock`, `vertex_ai`. See [Auto-Instrumentation](../integrations/auto-instrumentation) for the full list.

### uninstrument_all

```python
from waxell_observe.instrumentors import uninstrument_all

uninstrument_all() -> None
```

Remove all instrumentation patches. Called automatically by `shutdown()`.

---

## OpenTelemetry Functions

These functions manage the OTel tracing layer. They are called automatically by `init()` and `shutdown()`, but can be used directly for advanced control.

### init_tracing

```python
from waxell_observe.tracing import init_tracing

init_tracing(
    api_url: str | None = None,
    api_key: str | None = None,
    otel_endpoint: str | None = None,
    tenant_id: str | None = None,
    debug: bool | None = None,
    capture_content: bool = False,
    shutdown_on_exit: bool = True,
    resource_attributes: dict | None = None,
) -> None
```

Initialize OpenTelemetry tracing with OTLP HTTP export to the Waxell backend.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `api_url` | `str \| None` | `None` | Waxell API URL. Resolved from config if not provided |
| `api_key` | `str \| None` | `None` | Waxell API key. Resolved from config if not provided |
| `otel_endpoint` | `str \| None` | `None` | Explicit OTel collector endpoint. Auto-discovered if not provided |
| `tenant_id` | `str \| None` | `None` | Explicit tenant ID for trace routing. Auto-discovered from API key if not provided |
| `debug` | `bool \| None` | `None` | Enable debug logging and console span export. Defaults to `WAXELL_DEBUG` env var |
| `capture_content` | `bool` | `False` | Include prompt/response content in spans |
| `shutdown_on_exit` | `bool` | `True` | Register atexit handler for clean shutdown |
| `resource_attributes` | `dict \| None` | `None` | Extra OTel resource attributes |

### flush_tracing

```python
from waxell_observe.tracing import flush_tracing

flush_tracing(timeout_millis: int = 30000) -> None
```

Force flush pending spans to the backend.

### shutdown_tracing

```python
from waxell_observe.tracing import shutdown_tracing

shutdown_tracing() -> None
```

Shut down the TracerProvider and flush remaining spans.

---

## WaxellObserveClient

```python
from waxell_observe import WaxellObserveClient
```

HTTP client for the Waxell Observe API. Handles configuration resolution, authentication, and all API interactions.

### Constructor

```python
WaxellObserveClient(
    api_url: str | None = None,
    api_key: str | None = None,
)
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `api_url` | `str \| None` | `None` | Control plane URL. Overrides all other config sources |
| `api_key` | `str \| None` | `None` | API key. Overrides all other config sources |

### Class Methods

#### configure

```python
@classmethod
WaxellObserveClient.configure(api_url: str, api_key: str) -> None
```

Set global configuration for all instances. Call once at application startup. All subsequent `WaxellObserveClient()` instances will use these values (unless overridden by constructor arguments).

#### get_config

```python
@classmethod
WaxellObserveClient.get_config() -> ObserveConfig | None
```

Returns the current global configuration, or `None` if `configure()` has not been called.

#### is_configured

```python
@classmethod
WaxellObserveClient.is_configured() -> bool
```

Returns `True` if global configuration is set and both `api_url` and `api_key` are non-empty.

### Async Methods

#### start_run

```python
async start_run(
    agent_name: str,
    workflow_name: str = "default",
    inputs: dict | None = None,
    metadata: dict | None = None,
    trace_id: str = "",
    user_id: str = "",
    user_group: str = "",
    session_id: str = "",
    parent_workflow_id: str = "",
    root_workflow_id: str = "",
) -> RunInfo
```

Start an execution run on the control plane.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `agent_name` | `str` | (required) | Agent name |
| `workflow_name` | `str` | `"default"` | Workflow name |
| `inputs` | `dict \| None` | `None` | Input data for the run |
| `metadata` | `dict \| None` | `None` | Arbitrary metadata |
| `trace_id` | `str` | `""` | External trace ID for correlation |
| `user_id` | `str` | `""` | User identifier for per-user analytics |
| `user_group` | `str` | `""` | User group for authorization policies |
| `session_id` | `str` | `""` | Session ID for grouping related runs |
| `parent_workflow_id` | `str` | `""` | Parent workflow ID for nested agent lineage |
| `root_workflow_id` | `str` | `""` | Root workflow ID for top-level lineage tracking |

**Returns:** `RunInfo` with `run_id`, `workflow_id`, and `started_at`.

#### complete_run

```python
async complete_run(
    run_id: str,
    result: dict | None = None,
    status: str = "success",
    error: str = "",
    error_type: str = "",
    traceback: str = "",
    steps: list | None = None,
    trace_id: str = "",
    root_span_id: str = "",
) -> RunCompleteResult
```

Complete an execution run. Returns governance info including retry feedback.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `run_id` | `str` | (required) | Run ID from `start_run` |
| `result` | `dict \| None` | `None` | Result data |
| `status` | `str` | `"success"` | `"success"` or `"error"` |
| `error` | `str` | `""` | Error message |
| `error_type` | `str` | `""` | Exception class name (e.g. `"ValueError"`) |
| `traceback` | `str` | `""` | Full traceback string |
| `steps` | `list \| None` | `None` | Additional steps |
| `trace_id` | `str` | `""` | OTel trace ID for correlation |
| `root_span_id` | `str` | `""` | OTel root span ID for correlation |

**Returns:** `RunCompleteResult` with `run_id`, `duration`, `governance_action`, `governance_reason`, `retry_feedback`, and `max_retries`.

#### record_llm_calls

```python
async record_llm_calls(
    run_id: str,
    calls: list[dict],
) -> dict
```

Record one or more LLM calls for a run. No-op if `calls` is empty.

| Parameter | Type | Description |
|-----------|------|-------------|
| `run_id` | `str` | Run ID |
| `calls` | `list[dict]` | List of LLM call dicts with keys: `model`, `tokens_in`, `tokens_out`, and optionally `cost`, `task`, `prompt_preview`, `response_preview` |

**Returns:** Server response dict (includes `governance` field for mid-execution governance).

#### record_steps

```python
async record_steps(
    run_id: str,
    steps: list[dict],
) -> dict
```

Record execution steps for a run. No-op if `steps` is empty.

| Parameter | Type | Description |
|-----------|------|-------------|
| `run_id` | `str` | Run ID |
| `steps` | `list[dict]` | List of step dicts with keys: `step_name` and optionally `output`, `position` |

**Returns:** Server response dict (includes `governance` field for mid-execution governance).

#### record_scores

```python
async record_scores(
    run_id: str,
    scores: list[dict],
) -> dict
```

Record scores (user feedback, evaluation results) for a run.

| Parameter | Type | Description |
|-----------|------|-------------|
| `run_id` | `str` | Run ID |
| `scores` | `list[dict]` | List of score dicts. Each dict should contain: `name`, `data_type` (`"numeric"`, `"categorical"`, or `"boolean"`), and either `numeric_value` or `string_value` depending on data type. Optional: `comment` |

No-op if `scores` is empty. **Returns:** Server response dict.

**Example:**

```python
await client.record_scores(run_id, scores=[
    {"name": "accuracy", "data_type": "numeric", "numeric_value": 0.95},
    {"name": "thumbs_up", "data_type": "boolean", "numeric_value": 1.0, "string_value": "true"},
    {"name": "category", "data_type": "categorical", "string_value": "helpful", "comment": "User feedback"},
])
```

#### get_prompt

```python
async get_prompt(
    name: str,
    *,
    label: str = "",
    version: int = 0,
) -> PromptInfo
```

Fetch a prompt from the control plane. Returns the prompt content, config, and a `compile()` helper for template rendering.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `name` | `str` | (required) | Prompt name |
| `label` | `str` | `""` | Label to fetch (e.g. `"production"`). If empty, fetches latest version |
| `version` | `int` | `0` | Specific version number. Takes precedence over `label` if both provided |

**Returns:** `PromptInfo` with `name`, `version`, `prompt_type`, `content`, `config`, `labels`, and `compile()` method.

**Example:**

```python
# Fetch by label (recommended for production)
prompt = await client.get_prompt("summarizer", label="production")
rendered = prompt.compile(topic="AI safety", length="short")

# Fetch specific version
prompt = await client.get_prompt("summarizer", version=3)

# Fetch latest
prompt = await client.get_prompt("summarizer")
```

#### check_policy

```python
async check_policy(
    agent_name: str,
    workflow_name: str = "",
    agent_id: str = "",
) -> PolicyCheckResult
```

Check if execution is allowed by policies.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `agent_name` | `str` | (required) | Agent name |
| `workflow_name` | `str` | `""` | Workflow name for scoped policies |
| `agent_id` | `str` | `""` | Specific agent instance ID |

**Returns:** `PolicyCheckResult`.

#### record_events

```python
async record_events(events: list[dict]) -> None
```

Record governance events. No-op if `events` is empty.

#### close

```python
async close() -> None
```

Close the underlying HTTP client. Call this when you are done using the client.

### Sync Methods

Each async method has a synchronous counterpart that uses `asyncio.run()` internally:

| Async Method | Sync Method |
|-------------|-------------|
| `start_run()` | `start_run_sync()` |
| `complete_run()` | `complete_run_sync()` |
| `record_llm_calls()` | `record_llm_calls_sync()` |
| `record_steps()` | `record_steps_sync()` |
| `record_scores()` | `record_scores_sync()` |
| `record_spans()` | `record_spans_sync()` |
| `check_policy()` | `check_policy_sync()` |
| `record_events()` | `record_events_sync()` |
| `get_prompt()` | `get_prompt_sync()` |

Sync methods accept the same keyword arguments as their async counterparts.

**WARNING**
Sync methods cannot be used inside an already-running async event loop. If a running event loop is detected, the SDK delegates to a background thread with a 60-second timeout. Use the async versions in async code when possible.


---

## WaxellContext

```python
from waxell_observe import WaxellContext
```

Context manager (sync and async) that wraps agent execution with observability and governance.

### Constructor

```python
WaxellContext(
    agent_name: str,
    workflow_name: str = "default",
    inputs: dict | None = None,
    metadata: dict | None = None,
    client: WaxellObserveClient | None = None,
    enforce_policy: bool = True,
    session_id: str = "",
    user_id: str = "",
    user_group: str = "",
    mid_execution_governance: bool = False,
    auto_grounding: bool = False,
    on_policy_block: Callable | None = None,
)
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `agent_name` | `str` | (required) | Agent name |
| `workflow_name` | `str` | `"default"` | Workflow name |
| `inputs` | `dict \| None` | `None` | Input data for the run |
| `metadata` | `dict \| None` | `None` | Arbitrary metadata |
| `client` | `WaxellObserveClient \| None` | `None` | Pre-configured client. If `None`, creates one using global config |
| `enforce_policy` | `bool` | `True` | Check policies on entry |
| `session_id` | `str` | `""` | Session ID for grouping related runs. Use `generate_session_id()` to create one |
| `user_id` | `str` | `""` | User identifier for per-user analytics and tracking |
| `user_group` | `str` | `""` | User group for authorization policies (e.g., `"enterprise"`, `"free"`) |
| `mid_execution_governance` | `bool` | `False` | Enable cooperative mid-execution governance. When `True`, each `record_step()` flushes data and checks for policy violations |
| `auto_grounding` | `bool` | `False` | Auto-bridge retrieval scores to grounding governance |
| `on_policy_block` | `Callable \| None` | `None` | Callback for policy blocks. Receives `PolicyViolationError`, returns `ApprovalDecision` |

### Usage

Works as both `async with` (async code) and plain `with` (sync code):

```python
# Async
async with WaxellContext(agent_name="my-agent") as ctx:
    result = await my_agent.run(query)
    ctx.set_result({"output": result})

# Sync
with WaxellContext(agent_name="my-agent") as ctx:
    result = my_agent.run(query)
    ctx.set_result({"output": result})
```

**With session and user tracking:**

```python
from waxell_observe import WaxellContext, generate_session_id

session = generate_session_id()

async with WaxellContext(
    agent_name="my-agent",
    session_id=session,
    user_id="user_456",
) as ctx:
    ctx.set_tag("environment", "production")
    ctx.set_metadata("request_id", "req_abc123")
    result = await my_agent.run(query)
    ctx.record_score("relevance", 0.92)
    ctx.set_result({"output": result})
```

### Lifecycle

On enter (`__aenter__` / `__enter__`):
1. Checks policies (if `enforce_policy=True`). Raises `PolicyViolationError` if blocked.
2. Starts an execution run on the control plane. `session_id` and `user_id` are injected into the run metadata.
3. Creates an OTel agent span (if tracing is initialized). Session and user IDs are set as span attributes.
4. Sets the ContextVar so auto-instrumented LLM calls are associated with this run.

On exit (`__aexit__` / `__exit__`):
1. Flushes buffered LLM calls via `record_llm_calls`.
2. Flushes buffered steps via `record_steps`.
3. Flushes buffered scores via `record_scores`.
4. Flushes buffered behavior spans via `record_spans`.
5. Completes the run with result or error status.
6. Ends the OTel agent span and clears the ContextVar.

The sync path (`__enter__` / `__exit__`) uses synchronous HTTP calls and sets the ContextVar in the calling thread, ensuring auto-instrumentation works correctly.

### Methods

#### record_llm_call

```python
record_llm_call(
    *,
    model: str,
    tokens_in: int,
    tokens_out: int,
    cost: float = 0.0,
    task: str = "",
    prompt_preview: str = "",
    response_preview: str = "",
    duration_ms: int | None = None,
    provider: str = "",
) -> None
```

Buffer an LLM call for later flushing. All parameters are keyword-only. If `cost` is `0.0`, it is automatically estimated using built-in model pricing. Also emits an OTel LLM span (if tracing is initialized).

#### record_step

```python
record_step(step_name: str, output: dict | None = None) -> None
```

Buffer an execution step. Steps are automatically numbered in the order they are recorded (via an internal `position` counter). Also emits an OTel step span.

If `mid_execution_governance` is enabled, this method also flushes buffered data to the server and checks the governance response. Raises `PolicyViolationError` if the server returns a block action.

#### set_result

```python
set_result(result: dict) -> None
```

Set the result to include when the run is completed.

#### record_score

```python
record_score(
    name: str,
    value: float | str | bool,
    data_type: str = "numeric",
    comment: str = "",
) -> None
```

Buffer a score (user feedback or evaluation result) for the current run. Scores are flushed to the server when the context exits.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `name` | `str` | (required) | Score name (e.g. `"thumbs_up"`, `"accuracy"`, `"relevance"`) |
| `value` | `float \| str \| bool` | (required) | Score value. Type depends on `data_type` |
| `data_type` | `str` | `"numeric"` | One of `"numeric"`, `"categorical"`, or `"boolean"` |
| `comment` | `str` | `""` | Optional free-text comment |

**Value handling by data type:**
- `"numeric"`: `value` is stored as `numeric_value` (converted to float)
- `"boolean"`: `value` is stored as both `numeric_value` (`1.0` for truthy, `0.0` for falsy) and `string_value` (`"true"` or `"false"`)
- `"categorical"`: `value` is stored as `string_value` (converted to string)

**Example:**

```python
async with WaxellContext(agent_name="my-agent") as ctx:
    result = await run_agent(query)
    ctx.record_score("accuracy", 0.95)
    ctx.record_score("thumbs_up", True, data_type="boolean")
    ctx.record_score("category", "helpful", data_type="categorical", comment="User selected")
    ctx.set_result({"output": result})
```

#### set_tag

```python
set_tag(key: str, value: str) -> None
```

Set a searchable tag on the current agent span. Tags are string key-value pairs that become OTel span attributes with the `waxell.tag.` prefix.

| Parameter | Type | Description |
|-----------|------|-------------|
| `key` | `str` | Tag name (alphanumeric, underscores, hyphens) |
| `value` | `str` | Tag value (string) |

Tags are queryable in Grafana TraceQL:
```
{ span.waxell.tag.environment = "production" }
```

**Example:**

```python
async with WaxellContext(agent_name="my-agent") as ctx:
    ctx.set_tag("environment", "production")
    ctx.set_tag("customer_tier", "enterprise")
    ctx.set_tag("region", "us-east-1")
```

#### set_metadata

```python
set_metadata(key: str, value: Any) -> None
```

Set metadata on the current agent span. Unlike tags, metadata values can be any JSON-serializable type. Complex values are automatically JSON-serialized for OTel compatibility.

| Parameter | Type | Description |
|-----------|------|-------------|
| `key` | `str` | Metadata key |
| `value` | `Any` | Any JSON-serializable value |

Metadata is queryable in Grafana TraceQL:
```
{ span.waxell.meta.request_id != nil }
```

**Example:**

```python
async with WaxellContext(agent_name="my-agent") as ctx:
    ctx.set_metadata("request_id", "req_abc123")
    ctx.set_metadata("config", {"temperature": 0.7, "max_tokens": 1000})
    ctx.set_metadata("retry_count", 2)
```

#### Behavior Tracking Methods

These methods buffer behavior data as spans, flushed to the server on context exit via the `POST /runs/{run_id}/spans/` endpoint.

#### record_tool_call

```python
record_tool_call(
    *,
    name: str,
    input: dict | str = "",
    output: dict | str = "",
    duration_ms: int | None = None,
    status: str = "ok",
    tool_type: str = "function",
    error: str = "",
) -> None
```

Buffer a tool/function call event.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `name` | `str` | (required) | Tool name (e.g. `"web_search"`, `"database_query"`) |
| `input` | `dict \| str` | `""` | Tool input parameters |
| `output` | `dict \| str` | `""` | Tool output/result |
| `duration_ms` | `int \| None` | `None` | Execution time in milliseconds |
| `status` | `str` | `"ok"` | `"ok"` or `"error"` |
| `tool_type` | `str` | `"function"` | `"function"`, `"api"`, `"database"`, or `"retriever"` |
| `error` | `str` | `""` | Error message if status is `"error"` |

#### record_retrieval

```python
record_retrieval(
    *,
    query: str,
    documents: list[dict],
    source: str = "",
    duration_ms: int | None = None,
    top_k: int | None = None,
    scores: list[float] | None = None,
) -> None
```

Buffer a RAG retrieval operation.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `query` | `str` | (required) | The retrieval query string |
| `documents` | `list[dict]` | (required) | Retrieved docs (e.g. `[{id, title, score, snippet}]`) |
| `source` | `str` | `""` | Data source name (e.g. `"pinecone"`, `"elasticsearch"`) |
| `duration_ms` | `int \| None` | `None` | Retrieval time in milliseconds |
| `top_k` | `int \| None` | `None` | Number of documents requested |
| `scores` | `list[float] \| None` | `None` | Relevance scores for each retrieved document |

#### record_decision

```python
record_decision(
    *,
    name: str,
    options: list[str],
    chosen: str,
    reasoning: str = "",
    confidence: float | None = None,
    metadata: dict | None = None,
) -> None
```

Buffer a decision/routing point.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `name` | `str` | (required) | Decision name (e.g. `"route_to_agent"`, `"select_model"`) |
| `options` | `list[str]` | (required) | Available choices |
| `chosen` | `str` | (required) | The selected option |
| `reasoning` | `str` | `""` | Why this option was chosen |
| `confidence` | `float \| None` | `None` | Confidence score (0.0-1.0) |
| `metadata` | `dict \| None` | `None` | Additional context |

#### record_reasoning

```python
record_reasoning(
    *,
    step: str,
    thought: str,
    evidence: list[str] | None = None,
    conclusion: str = "",
) -> None
```

Buffer a reasoning/chain-of-thought step.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `step` | `str` | (required) | Reasoning step name |
| `thought` | `str` | (required) | The reasoning text/thought process |
| `evidence` | `list[str] \| None` | `None` | Supporting evidence or references |
| `conclusion` | `str` | `""` | Conclusion reached at this step |

#### record_retry

```python
record_retry(
    *,
    attempt: int,
    reason: str,
    strategy: str = "retry",
    original_error: str = "",
    fallback_to: str = "",
    max_attempts: int | None = None,
) -> None
```

Buffer a retry or fallback event.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `attempt` | `int` | (required) | Current attempt number (1-based) |
| `reason` | `str` | (required) | Why a retry/fallback occurred |
| `strategy` | `str` | `"retry"` | `"retry"`, `"fallback"`, or `"circuit_break"` |
| `original_error` | `str` | `""` | The error that triggered the retry |
| `fallback_to` | `str` | `""` | Name of fallback target (model, agent, tool) |
| `max_attempts` | `int \| None` | `None` | Maximum attempts configured |

#### check_policy / check_policy_sync

```python
# Async
async check_policy() -> PolicyCheckResult

# Sync
check_policy_sync() -> PolicyCheckResult
```

Perform a mid-execution policy check. Returns a `PolicyCheckResult`. Use `check_policy_sync()` in synchronous code.

### Properties

| Property | Type | Description |
|----------|------|-------------|
| `run_id` | `str` | The run ID from the control plane, or `""` if not started |
| `session_id` | `str` | The session ID passed to the constructor |
| `user_id` | `str` | The user ID passed to the constructor |
| `user_group` | `str` | The user group passed to the constructor |

---

## @observe / @waxell_agent

```python
from waxell_observe import observe  # Alias for waxell_agent
from waxell_observe import waxell_agent  # Original decorator (identical to observe)
```

Decorator that adds observability and governance to any function. `@observe` and `@waxell_agent` are identical -- use whichever reads better in your codebase.

### Signature

```python
@observe(
    agent_name: str | None = None,
    workflow_name: str = "default",
    enforce_policy: bool = True,
    capture_io: bool = True,
    session_id: str = "",
    user_id: str = "",
    user_group: str = "",
    mid_execution_governance: bool = False,
    auto_grounding: bool = False,
    on_policy_block: Callable | None = None,
    client: WaxellObserveClient | None = None,
)
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `agent_name` | `str \| None` | `None` | Agent name. Defaults to the function name |
| `workflow_name` | `str` | `"default"` | Workflow name |
| `enforce_policy` | `bool` | `True` | Check policies before execution |
| `capture_io` | `bool` | `True` | Capture function inputs and outputs |
| `session_id` | `str` | `""` | Session ID for grouping related runs |
| `user_id` | `str` | `""` | End-user ID for attribution and analytics |
| `user_group` | `str` | `""` | User group for authorization policies |
| `mid_execution_governance` | `bool` | `False` | Flush data and check governance on each `record_step()` call |
| `auto_grounding` | `bool` | `False` | Auto-bridge retrieval scores to grounding governance |
| `on_policy_block` | `Callable \| None` | `None` | Callback for policy blocks. Receives `PolicyViolationError`, returns `ApprovalDecision`. Built-in: `prompt_approval`, `auto_approve`, `auto_deny` |
| `client` | `WaxellObserveClient \| None` | `None` | Pre-configured client |

### Context Injection

If the decorated function has a `waxell_ctx` parameter, a `WaxellContext` instance is injected automatically:

```python
@observe(agent_name="my-agent")
async def my_func(query: str, waxell_ctx=None) -> str:
    if waxell_ctx:
        waxell_ctx.record_llm_call(model="gpt-4o", tokens_in=100, tokens_out=50)
        waxell_ctx.record_score("relevance", 0.9)
        waxell_ctx.set_tag("source", "api")
    return "result"
```

### Example

```python


@waxell.observe(agent_name="my-agent")
async def chat(query: str) -> str:
    response = await openai_client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": query}],
    )  # auto-captured by instrumentation
    waxell.score("helpfulness", 0.9)
    waxell.tag("intent", "question")
    return response.choices[0].message.content
# Creates a full run with LLM call, score, and tag — all auto-recorded
```

### Behavior

- **Async functions** are wrapped with an async wrapper
- **Sync functions** are wrapped with a sync wrapper that uses `asyncio.run()` internally
- On success, the run is completed with `status="success"` and the captured return value
- On exception, the run is completed with `status="error"` and the error message; the exception is re-raised

---

## @tool

```python
from waxell_observe import tool
```

Decorator that auto-records function calls as tool invocations on the current `WaxellContext`.

### Signature

```python
@tool(name: str | None = None, tool_type: str = "function")
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `name` | `str \| None` | `None` | Tool name. Defaults to function name |
| `tool_type` | `str` | `"function"` | Classification: `"function"`, `"vector_db"`, `"database"`, `"api"` |

Captures: function arguments as input, return value as output, execution time, status ("ok" or "error"). Re-raises any exceptions. No-op outside a `WaxellContext`. Works with sync and async functions.

**Example:**

```python


@waxell.tool(tool_type="vector_db")
def search_index(query_vec, k: int = 5):
    distances, indices = index.search(query_vec, k)
    return {"distances": distances.tolist(), "indices": indices.tolist()}
# Auto-records: tool_call(name="search_index", input={...}, output={...}, duration_ms=...)
```

---

## @decision

```python
from waxell_observe import decision
```

Decorator that auto-records a function's return value as a decision.

### Signature

```python
@decision(name: str | None = None, options: list[str] | None = None)
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `name` | `str \| None` | `None` | Decision name. Defaults to function name |
| `options` | `list[str] \| None` | `None` | Available choices |

**Return value handling:** `dict` returns extract `chosen`, `reasoning`, `confidence`. String returns use the string as `chosen`.

**Example:**

```python


@waxell.decision(name="route_query", options=["factual", "analytical", "creative"])
async def classify_query(query: str) -> dict:
    response = await client.chat.completions.create(...)
    return {"chosen": "factual", "reasoning": "Direct question", "confidence": 0.92}
# Dict return: extracts chosen, reasoning, confidence automatically
```

---

## @retrieval

```python
from waxell_observe import retrieval
```

Decorator that auto-records function calls as retrieval operations.

### Signature

```python
@retrieval(source: str = "", name: str | None = None)
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `source` | `str` | `""` | Data source name (e.g., `"faiss"`, `"pinecone"`) |
| `name` | `str \| None` | `None` | Override name. Defaults to function name |

Extracts query from the first string argument, documents from the return value, and scores from `doc["score"]` fields.

**Example:**

```python


@waxell.retrieval(source="faiss")
def search_documents(query: str, corpus: list) -> list[dict]:
    return [{"id": 1, "title": "Result", "score": 0.95}]
# Auto-extracts: query from first str arg, documents from return, scores from "score" keys
```

---

## @reasoning\_dec

```python
from waxell_observe import reasoning_dec
```

Decorator that auto-records a function's return value as a reasoning step.

### Signature

```python
@reasoning_dec(step: str | None = None)
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `step` | `str \| None` | `None` | Reasoning step name. Defaults to function name |

**Return value handling:** `dict` returns extract `thought`, `evidence`, `conclusion`. String returns use the string as `thought`.

**Example:**

```python


@waxell.reasoning_dec(step="quality_check")
async def assess_answer(answer: str) -> dict:
    return {"thought": "Answer covers sources", "evidence": ["A cited"], "conclusion": "High quality"}
# Dict return: extracts thought, evidence, conclusion
```

---

## @retry\_dec

```python
from waxell_observe import retry_dec
```

Decorator that wraps a function with retry logic AND records each attempt.

### Signature

```python
@retry_dec(max_attempts: int = 3, strategy: str = "retry", fallback_to: str = "")
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `max_attempts` | `int` | `3` | Maximum attempts (including first) |
| `strategy` | `str` | `"retry"` | `"retry"`, `"fallback"`, or `"circuit_break"` |
| `fallback_to` | `str` | `""` | Fallback target name |

On each failure, records a retry span. After exhausting attempts, re-raises the last exception.

**Example:**

```python


@waxell.retry_dec(max_attempts=3, strategy="fallback", fallback_to="gpt-4o-mini")
async def call_llm(prompt: str) -> str:
    response = await client.chat.completions.create(model="gpt-4o", messages=[...])
    return response.choices[0].message.content
# Retries up to 3 times, records each attempt as a retry span
```

---

## @step\_dec

```python
from waxell_observe import step_dec
```

Decorator that auto-records function calls as execution steps.

### Signature

```python
@step_dec(name: str | None = None)
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `name` | `str \| None` | `None` | Step name. Defaults to function name |

Records the function's return value as the step output. No-op outside a `WaxellContext`.

**Example:**

```python


@waxell.step_dec(name="preprocess")
def clean_input(text: str) -> dict:
    cleaned = text.strip().lower()
    return {"original": text, "cleaned": cleaned, "length": len(cleaned)}
# Return value becomes step output
```

---

## WaxellLangChainHandler

```python
from waxell_observe.integrations.langchain import WaxellLangChainHandler
```

Factory function that returns a LangChain `BaseCallbackHandler` instance.

### Signature

```python
WaxellLangChainHandler(
    agent_name: str,
    workflow_name: str = "default",
    client: WaxellObserveClient | None = None,
    enforce_policy: bool = True,
    auto_start_run: bool = True,
) -> BaseCallbackHandler
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `agent_name` | `str` | (required) | Agent name |
| `workflow_name` | `str` | `"default"` | Workflow name |
| `client` | `WaxellObserveClient \| None` | `None` | Pre-configured client |
| `enforce_policy` | `bool` | `True` | Check policies on first callback |
| `auto_start_run` | `bool` | `True` | Automatically start a run on first callback |

**INFO**
Requires `langchain-core`. Install with `pip install waxell-observe[langchain]`.


### Instance Methods

#### flush

```python
async flush(
    result: dict | None = None,
    status: str = "success",
    error: str = "",
) -> None
```

Flush all buffered telemetry to the control plane and complete the run.

#### flush_sync

```python
flush_sync(**kwargs) -> None
```

Synchronous version of `flush`. Accepts the same keyword arguments.

### Instance Properties

| Property | Type | Description |
|----------|------|-------------|
| `run_id` | `str` | The run ID from the control plane, or `""` if no run started |

### Captured Callbacks

| Callback | Data Captured |
|----------|--------------|
| `on_llm_start` | Model name, prompt preview (500 chars) |
| `on_llm_end` | Token counts, cost estimate, response preview (500 chars) |
| `on_chain_start` | Chain name as a step |
| `on_chain_end` | Chain output |
| `on_tool_start` | Tool name as a step (prefixed `tool:`) |
| `on_tool_end` | Tool output (1000 chars) |

---

## Types

### RunInfo

```python
from waxell_observe import RunInfo
```

```python
@dataclass
class RunInfo:
    run_id: str
    workflow_id: str
    started_at: str
```

Information about a started execution run. Returned by `WaxellObserveClient.start_run()`.

### RunCompleteResult

```python
from waxell_observe import RunCompleteResult
```

```python
@dataclass
class RunCompleteResult:
    run_id: str
    duration: float | None = None
    governance_action: str = "allow"
    governance_reason: str = ""
    retry_feedback: str = ""
    max_retries: int = 0

    @property
    def should_retry(self) -> bool: ...  # True if governance_action == "retry"
```

Result from completing a run, including governance info. Returned by `WaxellObserveClient.complete_run()`.

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `run_id` | `str` | (required) | The run ID |
| `duration` | `float \| None` | `None` | Run duration in seconds |
| `governance_action` | `str` | `"allow"` | Post-execution governance action |
| `governance_reason` | `str` | `""` | Reason for governance action |
| `retry_feedback` | `str` | `""` | Feedback for retry attempts |
| `max_retries` | `int` | `0` | Maximum retry attempts allowed |

### PolicyCheckResult

```python
from waxell_observe import PolicyCheckResult
```

```python
@dataclass
class PolicyCheckResult:
    action: str       # "allow", "block", "warn", "throttle", "retry"
    reason: str = ""
    metadata: dict = field(default_factory=dict)
    evaluations: list = field(default_factory=list)

    @property
    def allowed(self) -> bool: ...  # True if action in ("allow", "warn")

    @property
    def blocked(self) -> bool: ...  # True if action in ("block", "throttle")

    @property
    def should_retry(self) -> bool: ...  # True if action == "retry"
```

Result of a policy check. Returned by `WaxellObserveClient.check_policy()` and `WaxellContext.check_policy()`.

### LlmCallInfo

```python
from waxell_observe import LlmCallInfo
```

```python
@dataclass
class LlmCallInfo:
    model: str
    tokens_in: int
    tokens_out: int
    cost: float = 0.0
    task: str = ""
    prompt_preview: str = ""
    response_preview: str = ""
```

Typed representation of an LLM API call. Useful for constructing call records programmatically.

### PromptInfo

```python
from waxell_observe import PromptInfo
```

```python
@dataclass
class PromptInfo:
    name: str
    version: int
    prompt_type: str  # "text" or "chat"
    content: object   # str for text, list[dict] for chat
    config: dict = field(default_factory=dict)
    labels: list = field(default_factory=list)

    def compile(self, **variables: str) -> object: ...
```

A prompt version retrieved from the control plane. Returned by `WaxellObserveClient.get_prompt()`.

| Field | Type | Description |
|-------|------|-------------|
| `name` | `str` | Prompt name |
| `version` | `int` | Version number |
| `prompt_type` | `str` | `"text"` for plain text prompts, `"chat"` for chat message prompts |
| `content` | `str \| list[dict]` | Prompt content. A string for text prompts, a list of `{"role": ..., "content": ...}` message dicts for chat prompts |
| `config` | `dict` | Model configuration (e.g. `temperature`, `max_tokens`, `model`) |
| `labels` | `list[str]` | Labels attached to this version (e.g. `["production", "latest"]`) |

#### compile

```python
prompt.compile(**variables: str) -> str | list[dict]
```

Render the prompt by replacing `{{variable}}` placeholders in the content.

- **Text prompts:** Returns a string with all `{{variable}}` placeholders replaced.
- **Chat prompts:** Returns a list of message dicts with `{{variable}}` placeholders replaced in each message's `content` field.

**Example:**

```python
# Text prompt
prompt = await client.get_prompt("summarizer", label="production")
rendered = prompt.compile(topic="AI safety", length="short")
# rendered: "Summarize the following about AI safety in short form: ..."

# Chat prompt
prompt = await client.get_prompt("assistant", label="production")
messages = prompt.compile(user_query="What is RAG?")
# messages: [{"role": "system", "content": "..."}, {"role": "user", "content": "What is RAG?"}]

# Use config for model parameters
response = openai.chat.completions.create(
    model=prompt.config.get("model", "gpt-4o"),
    messages=messages,
    temperature=prompt.config.get("temperature", 0.7),
)
```

### ObserveConfig

```python
from waxell_observe import ObserveConfig
```

```python
@dataclass
class ObserveConfig:
    api_url: str = ""
    api_key: str = ""
    otel_endpoint: str = ""
    debug: bool = False
    capture_content: bool = False
    prompt_guard: bool = False
    prompt_guard_server: bool = False
    prompt_guard_action: str = "block"
    instrument_infra: bool = True
    infra_exclude: str = ""

    @classmethod
    def from_env(cls) -> ObserveConfig: ...

    @classmethod
    def from_cli_config(cls, config_path: Path | None = None) -> ObserveConfig: ...

    @property
    def is_configured(self) -> bool: ...
```

Configuration data class. Used internally by `WaxellObserveClient` to resolve settings.

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `api_url` | `str` | `""` | Waxell API URL |
| `api_key` | `str` | `""` | Waxell API key |
| `otel_endpoint` | `str` | `""` | Explicit OTel collector endpoint |
| `debug` | `bool` | `False` | Enable debug logging |
| `capture_content` | `bool` | `False` | Include prompt/response content in traces |
| `prompt_guard` | `bool` | `False` | Enable client-side prompt guard |
| `prompt_guard_server` | `bool` | `False` | Enable server-side prompt guard |
| `prompt_guard_action` | `str` | `"block"` | Action on violations: `"block"`, `"warn"`, or `"redact"` |
| `instrument_infra` | `bool` | `True` | Enable infrastructure library instrumentation |
| `infra_exclude` | `str` | `""` | Comma-delimited list of infra libraries to exclude |

| Class Method | Description |
|-------------|-------------|
| `from_env()` | Load from environment variables (`WAXELL_API_URL`/`WAXELL_API_KEY` or `WAX_API_URL`/`WAX_API_KEY`) |
| `from_cli_config(config_path=None)` | Load from CLI config file (default: `~/.waxell/config`) |

| Property | Type | Description |
|----------|------|-------------|
| `is_configured` | `bool` | `True` if both `api_url` and `api_key` are non-empty |

### ApprovalDecision

```python
from waxell_observe import ApprovalDecision
```

```python
@dataclass
class ApprovalDecision:
    approved: bool
    approver: str = ""
    timed_out: bool = False
    elapsed_seconds: float | None = None
```

Return type from `on_policy_block` handlers. Tells the decorator whether to retry the function (`approved=True`) or propagate the `PolicyViolationError`.

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `approved` | `bool` | required | Whether to proceed with execution |
| `approver` | `str` | `""` | Who approved (email, username, system) |
| `timed_out` | `bool` | `False` | Whether the approval window expired |
| `elapsed_seconds` | `float \| None` | `None` | Time from block to decision |

### HumanTurn

```python
from waxell_observe import HumanTurn
```

Context manager returned by `waxell.human_turn()`. Records a human interaction as a timed IO span when it exits.

| Method | Description |
|--------|-------------|
| `set_response(response: str)` | Record what the human replied |
| `__enter__()` | Start timing |
| `__exit__()` | Record the span with elapsed time |

---

## Errors

All errors inherit from `ObserveError`, which inherits from `Exception`.

### ObserveError

```python
from waxell_observe import ObserveError
```

Base error class for all waxell-observe errors.

### PolicyViolationError

```python
from waxell_observe import PolicyViolationError
```

```python
class PolicyViolationError(ObserveError):
    def __init__(self, message: str, policy_result=None): ...
    policy_result: PolicyCheckResult | None
```

Raised when a policy check blocks execution (action is `"block"` or `"throttle"`).

| Attribute | Type | Description |
|-----------|------|-------------|
| `policy_result` | `PolicyCheckResult \| None` | The full policy check result |

### ConfigurationError

```python
from waxell_observe import ConfigurationError
```

Raised when the client is not properly configured. Inherits from `ObserveError`.

---

## Functions

### estimate_cost

```python
from waxell_observe.cost import estimate_cost
```

```python
estimate_cost(model: str, tokens_in: int, tokens_out: int) -> float
```

Estimate the USD cost of an LLM call. Uses exact match first, then prefix matching for versioned model names. Returns `0.0` for unknown models.

| Parameter | Type | Description |
|-----------|------|-------------|
| `model` | `str` | Model name or prefix |
| `tokens_in` | `int` | Input token count |
| `tokens_out` | `int` | Output token count |

---

## Configuration Resolution

The SDK resolves configuration from multiple sources, in order of precedence (highest to lowest):

1. **Explicit constructor arguments** -- `WaxellObserveClient(api_url="...", api_key="...")`
2. **Global config** -- `WaxellObserveClient.configure(...)` or `waxell_observe.init(...)`
3. **CLI config file** -- `~/.waxell/config` (INI format)
4. **Environment variables** -- `WAXELL_API_URL` / `WAXELL_API_KEY` (or `WAX_API_URL` / `WAX_API_KEY`)

**Environment variables:**

| Variable | Description |
|----------|-------------|
| `WAXELL_API_URL` | Control plane URL (e.g. `https://acme.waxell.dev`) |
| `WAXELL_API_KEY` | API key (e.g. `wax_sk_abc123`) |
| `WAX_API_URL` | Alias for `WAXELL_API_URL` |
| `WAX_API_KEY` | Alias for `WAXELL_API_KEY` |
| `WAXELL_OBSERVE` | Kill switch. Set to `false`, `0`, or `no` to disable `init()` |

---

## Next Steps

- [REST API Reference](./endpoints) -- Direct HTTP API usage
- [Quickstart](../quickstart) -- Get started in 5 minutes
- [Installation & Configuration](../installation) -- Setup guide


================================================================================
SECTION: MCP GOVERNANCE
Govern Model Context Protocol servers — policy enforcement, PII scanning, and rug-pull detection
================================================================================

--------------------------------------------------------------------------------
# MCP Governance: Secure and Monitor AI Agent Tool Calls
URL: https://waxell.ai/docs/observe/mcp-governance/overview
Description: What is MCP governance, why SDK-level beats gateway approaches, and how Waxell's three products work together.
--------------------------------------------------------------------------------

# MCP Governance: Secure and Monitor AI Agent Tool Calls

If your AI agent uses MCP (Model Context Protocol) to call external tools -- reading files, querying databases, sending emails, executing code -- then every one of those tool calls is a potential security and compliance surface. MCP governance gives you visibility and control over what your agents do, before they do it.

**MCP** is the open protocol that lets AI agents discover and call tools on external servers. When your agent calls `session.call_tool("send_email", {"to": "ceo@company.com", ...})`, it's making an MCP tool call. The agent decides which tool to call and with what arguments. Without governance, there's nothing between the agent's decision and the action happening in the real world.

**MCP governance** adds a policy layer between the agent's intent and the tool execution. Every tool call passes through policy checks, PII scanning, approval workflows, and audit recording -- automatically, with no changes to your MCP server code.

## Why Governance Matters

AI agents with MCP access can do powerful things. That power creates three categories of risk:

**Unauthorized actions.** An agent might call a tool it shouldn't have access to, or pass arguments that exceed its scope. A customer support agent shouldn't be able to call `delete_database`, even if the MCP server exposes it. A content agent shouldn't be sending emails to external addresses.

**Data leakage.** Tool call arguments often contain user data, and tool results often contain sensitive information. Without scanning, an agent might send a social security number as a search query, or an MCP server might return API keys in its response. PII and credentials flowing through unmonitored tool calls create compliance and security risks.

**Rug pull attacks.** An MCP server can change what its tools do at any time. A tool called `search_documents` today might silently change its description to instruct the agent to exfiltrate data tomorrow. Without fingerprinting, you have no way to detect when a tool's behavior description changes between sessions.

**Real-world risk**
MCP servers are often third-party services that you don't control. Even trusted servers can be compromised, have bugs, or change their API surface. Governance isn't about distrust -- it's about verifiable safety.


## Why SDK-Level Governance?

There are several approaches to MCP governance. Gateway-based solutions sit between your agent and MCP servers as a proxy layer, inspecting traffic as it flows through. SDK-level governance works differently: it instruments the agent itself, adding governance checks directly inside the MCP client library.

SDK-level governance has several advantages:

**Agent context visibility.** The governance layer knows which agent is making the call, which user triggered it, what the agent's reasoning chain looks like, and what happened earlier in the session. A gateway only sees the raw tool call in isolation. This context makes policy decisions more precise -- you can allow `delete_file` for admin users but block it for regular users, all in one policy.

**Reasoning traces.** Every governance decision -- allow, block, warn, throttle -- is recorded as an OpenTelemetry span attribute on the same trace as the agent's LLM calls and tool execution. You get a single trace that shows the full story: the agent reasoned, decided to call a tool, governance checked the policy, PII was scanned, the tool executed, and the result was audited.

**Cost attribution.** Because the governance layer runs inside the agent process, it can attribute tool call costs to the same user, session, and agent run that the rest of your observability tracks. No need to correlate logs from two different systems.

**Zero infrastructure.** There's no proxy server to deploy, no gateway to maintain, no DNS to configure. You add governance by calling `configure_session()` on your existing MCP client session. It works with any MCP server, any transport, and any deployment environment.

**Complementary approaches**
SDK-level governance doesn't replace server-side security. It complements it. If you control the MCP server, you can add server-side middleware too. If you control neither the agent nor the server, a governance proxy gives you a third option. Waxell provides all three.


## How It Works

When you initialize Waxell Observe with `waxell.init()`, it automatically patches the MCP client library's `call_tool` method. Every subsequent tool call flows through a governance wrapper that handles policy checks, PII scanning, security fingerprinting, and audit recording.

```mermaid
sequenceDiagram
    participant A as Agent
    participant W as call_tool Wrapper
    participant P as Policy Engine
    participant S as PII Scanner
    participant M as MCP Server

    A->>W: session.call_tool("search", {...})
    W->>P: check_policy("mcp:composio:search")
    P-->>W: action: allow
    W->>S: scan inputs for PII
    S-->>W: clean
    W->>M: call_tool (original)
    M-->>W: result
    W->>S: scan outputs for PII
    S-->>W: no PII found
    Note over W: Record audit trail on span
    W-->>A: result (with governance attributes)
```

The wrapper adds governance without changing the tool call's behavior. If governance is not configured, tool calls pass through with only observability tracing. If governance is configured, each call gets policy checks, PII scanning, and audit recording -- all fail-open by default, so governance errors never break your agent.

## The Three Products

Waxell provides three complementary products for MCP governance, each designed for a different deployment scenario.

### Auto-Instrumentor (This Section)

The auto-instrumentor is for **agent developers** who write Python code that connects to MCP servers. You add governance by calling `configure_session()` on your MCP `ClientSession` after initialization. The instrumentor patches `call_tool` to add policy checks, PII scanning, rug pull detection, and audit recording.

**Best for:** You control the agent code. You connect to third-party MCP servers. You want governance without modifying servers.

This documentation section covers the auto-instrumentor in detail.

### Server Middleware

**[Server Middleware Quickstart →](middleware-quickstart.md)**

Server middleware is for **MCP server builders** who want to add governance to their own servers. You add a `GovernanceMiddleware` to your FastMCP server, and it applies policy checks, PII scanning, and rate limiting to all incoming tool calls.

**Best for:** You build and operate the MCP server. You want to enforce governance regardless of which agents connect.

### Governance Proxy

**[Governance Proxy Quickstart →](proxy-quickstart.md)**

The governance proxy is a standalone MCP server that sits between agents and target MCP servers. It forwards tool calls while applying governance checks. No code changes needed on either side.

**Best for:** You control neither the agent code nor the MCP server. You want governance as an infrastructure layer.

## Choosing the Right Product

```mermaid
flowchart TD
    Q1{Do you control<br/>the agent code?}
    Q1 -->|Yes| INST[Auto-Instrumentor<br/>Add configure_session to your agent]
    Q1 -->|No| Q2{Do you control<br/>the MCP server?}
    Q2 -->|Yes| MW[Server Middleware<br/>Add GovernanceMiddleware to your server]
    Q2 -->|No| PROXY[Governance Proxy<br/>Deploy proxy between agent and server]

    style INST fill:#1a7f5a,color:#fff
    style MW fill:#2563eb,color:#fff
    style PROXY fill:#7c3aed,color:#fff
```

All three products record governance decisions as OpenTelemetry span attributes using the same `waxell.mcp.*` attribute namespace. Regardless of which product you use, governance data appears in your Waxell dashboard traces with consistent structure and semantics.

## What You'll Learn

This documentation section covers the auto-instrumentor in depth:

- **[Quickstart](./quickstart)** -- Add MCP governance to your agent and see tool calls in traces in under 5 minutes.
- **[Policy Configuration](./policy-configuration)** -- Configure allowlists, blocklists, and policy rules for MCP tool calls using the `mcp:{server}:{tool}` naming convention.
- **[Approval Workflows](./approval-workflows)** -- Add human-in-the-loop approval for sensitive tool calls with `on_policy_block` handlers.
- **[PII Scanning](./pii-scanning)** -- Scan tool call inputs and outputs for PII, credentials, and secrets with configurable actions (warn, block, redact).
- **[Rug Pull Detection](./rug-pull-detection)** -- Detect when MCP server tool descriptions or schemas change between sessions using SHA256 fingerprinting.
- **[Reference](./reference)** -- Complete reference for all `waxell.mcp.*` span attributes, `configure_session()` parameters, and governance configuration options.

## Getting Started

Ready to add MCP governance to your agent? Start with the [Quickstart](./quickstart) -- you'll go from zero to seeing MCP tool calls in your traces in under 5 minutes.


--------------------------------------------------------------------------------
# MCP Governance Quickstart
URL: https://waxell.ai/docs/observe/mcp-governance/quickstart
Description: Add MCP governance to your agent in one line -- see tool calls, policy checks, and PII scans in traces within 5 minutes.
--------------------------------------------------------------------------------

# MCP Governance Quickstart

In this quickstart, you'll add MCP governance to a Python agent and see tool calls appear in your Waxell traces -- in under 5 minutes. You'll connect to an MCP server, make a tool call, and see the resulting span with governance attributes in your dashboard.

By the end, you'll have:
- Automatic tracing of every MCP tool call with span names like `tools/call filesystem:read_file`
- A working governance configuration with policy checks and PII scanning
- Understanding of how MCP spans nest in your trace tree

## Prerequisites

- **Python 3.10+**
- **A Waxell API key** -- get one from your [Waxell dashboard](https://waxell.dev)
- **Node.js 18+** -- for running the MCP filesystem server (used in examples)

## Step 1: Install

Install `waxell-observe` with the MCP extra, which pins the `mcp` package at a compatible version:

```bash
pip install waxell-observe[mcp]
```

This installs `waxell-observe` along with `mcp>=1.25,<2` and `wrapt` (used for monkey-patching the MCP client).

**Already have waxell-observe?**
If you already have `waxell-observe` installed, just add the MCP extra:
```bash
pip install "waxell-observe[mcp]"
```


## Step 2: Initialize

Call `waxell.init()` **before** importing any MCP client classes. This auto-patches `mcp.client.session.ClientSession.call_tool` to emit OpenTelemetry spans for every tool call.

```python


waxell.init()

# Import MCP classes AFTER init()
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
```

**Environment variables**
You can configure your API key and URL via environment variables instead of passing them to `init()`:
```bash


```
Then call `waxell.init()` without arguments.


## Step 3: Connect to an MCP Server

We'll use the official MCP filesystem server as our example -- it's free, requires no API key, and runs locally via `npx`. It exposes tools for reading, writing, and listing files.

The MCP client library uses an async context manager pattern to establish connections. Here's how to connect using the **stdio** transport:

```python

from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client

server_params = StdioServerParameters(
    command="npx",
    args=["-y", "@modelcontextprotocol/server-filesystem", "/tmp/mcp-test"],
)

async def main():
    async with stdio_client(server_params) as (read, write):
        async with ClientSession(read, write) as session:
            await session.initialize()

            # Make a tool call -- automatically traced by waxell-observe
            result = await session.call_tool(
                name="read_file",
                arguments={"path": "/tmp/mcp-test/hello.txt"},
            )
            print(result)

asyncio.run(main())
```

Before running, create a test file:

```bash
mkdir -p /tmp/mcp-test
echo "Hello from MCP governance!" > /tmp/mcp-test/hello.txt
```

<details>
<summary>Full runnable example</summary>

```python


# Initialize BEFORE importing MCP classes
waxell.init()

from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client


async def main():
    server_params = StdioServerParameters(
        command="npx",
        args=["-y", "@modelcontextprotocol/server-filesystem", "/tmp/mcp-test"],
    )

    async with stdio_client(server_params) as (read, write):
        async with ClientSession(read, write) as session:
            await session.initialize()

            # This tool call is automatically traced
            result = await session.call_tool(
                name="read_file",
                arguments={"path": "/tmp/mcp-test/hello.txt"},
            )

            # Print the result
            for item in result.content:
                print(item.text)


if __name__ == "__main__":
    asyncio.run(main())
```

</details>

When you run this script, Waxell automatically creates an OpenTelemetry span for the `call_tool` invocation. No manual instrumentation needed.

## Step 4: See Your Traces

After running the example, open your [Waxell dashboard](https://waxell.dev) and navigate to the Traces view. You'll see a span for the MCP tool call with these attributes:

| Attribute | Example Value | Description |
|-----------|--------------|-------------|
| **Span name** | `tools/call filesystem:read_file` | Format: `tools/call {server}:{tool}` |
| `waxell.mcp.server_name` | `filesystem` | Server name from `initialize()` response |
| `waxell.mcp.tool_name` | `read_file` | Tool that was called |
| `mcp.method.name` | `tools/call` | MCP protocol method |
| `waxell.mcp.transport` | `stdio` | Transport used to connect |
| `waxell.mcp.arguments` | `{"path": "/tmp/mcp-test/hello.txt"}` | Tool call arguments (truncated to 4KB) |
| `waxell.mcp.result` | `Hello from MCP governance!` | Tool call result (truncated to 4KB) |
| `waxell.mcp.is_error` | `false` | Whether the tool returned an error |
| `waxell.mcp.timeout_ms` | `30000` | Configured timeout in milliseconds |

### How MCP Spans Nest in Your Trace Tree

When MCP tool calls happen inside an `@waxell.observe()` decorated agent function, the MCP spans automatically nest as children of the agent span. If your agent also makes LLM calls, you get a complete trace showing reasoning and action:

```mermaid
flowchart TD
    A["Agent Run<br/><small>@observe(agent_name='file-reader')</small>"]
    A --> L["LLM Call<br/><small>gpt-4o: 'Read the config file'</small>"]
    A --> T["tools/call filesystem:read_file<br/><small>MCP tool call span</small>"]

    style A fill:#1a7f5a,color:#fff
    style L fill:#2563eb,color:#fff
    style T fill:#7c3aed,color:#fff
```

MCP spans use OpenTelemetry's automatic context propagation. If there's an active span when `call_tool` is invoked, the MCP span becomes its child. No explicit parent span wiring needed.

## Step 5: Add Governance

So far, Waxell has been tracing MCP tool calls passively -- recording what happens without intervening. To add active governance (policy checks, PII scanning, and audit recording), use `configure_session()`:

```python
from waxell_observe.instrumentors.mcp_instrumentor import configure_session

# After session.initialize(), before making tool calls:
configure_session(
    session,
    server_name="filesystem",
    governance_config={
        "agent_name": "file-reader",
        "scan_inputs": True,
        "scan_outputs": True,
        "pii_actions": {
            "ssn": "block",
            "email": "warn",
            "api_key": "block",
        },
    },
)
```

With governance configured, every tool call now goes through:

1. **Policy check** -- Queries the Waxell controlplane with the policy ID `mcp:filesystem:read_file`. If the controlplane returns `block`, the tool call is stopped before it reaches the MCP server.
2. **PII scan on inputs** -- Scans tool call arguments for PII and credentials. If a social security number is found and the action is `block`, the call is stopped.
3. **Tool execution** -- The actual MCP tool call runs (with configurable timeout).
4. **PII scan on outputs** -- Scans the tool's response for PII. Output PII findings are logged as warnings (the tool already executed).
5. **Audit recording** -- All governance decisions, PII findings, and policy results are recorded as span attributes.

<details>
<summary>Full runnable example with governance</summary>

```python


waxell.init()

from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
from waxell_observe.instrumentors.mcp_instrumentor import configure_session


async def main():
    server_params = StdioServerParameters(
        command="npx",
        args=["-y", "@modelcontextprotocol/server-filesystem", "/tmp/mcp-test"],
    )

    async with stdio_client(server_params) as (read, write):
        async with ClientSession(read, write) as session:
            await session.initialize()

            # Configure governance
            configure_session(
                session,
                server_name="filesystem",
                governance_config={
                    "agent_name": "file-reader",
                    "scan_inputs": True,
                    "scan_outputs": True,
                    "pii_actions": {
                        "ssn": "block",
                        "email": "warn",
                        "api_key": "block",
                    },
                },
            )

            # This tool call now goes through policy + PII checks
            result = await session.call_tool(
                name="read_file",
                arguments={"path": "/tmp/mcp-test/hello.txt"},
            )

            for item in result.content:
                print(item.text)


if __name__ == "__main__":
    asyncio.run(main())
```

</details>

**Governance is fail-open**
If the Waxell controlplane is unreachable or the PII scanner throws an error, the tool call proceeds normally. Governance errors never break your agent. This fail-open behavior is logged so you can monitor governance health.


### What governance adds to your traces

With governance configured, your tool call spans gain additional attributes:

| Attribute | Example Value | Description |
|-----------|--------------|-------------|
| `waxell.mcp.governance_checked` | `true` | Governance was active for this call |
| `waxell.mcp.governance_action` | `allow` | Policy decision: allow, block, warn, or throttle |
| `waxell.mcp.policy_id` | `mcp:filesystem:read_file` | Policy ID used for the check |
| `waxell.mcp.pii_detected` | `false` | Whether PII was found in inputs or outputs |
| `waxell.mcp.governance_timestamp` | `2026-03-05T22:30:00Z` | When the governance check ran |
| `waxell.mcp.params_hash` | `a1b2c3d4e5f67890` | SHA256 fingerprint of the arguments |

For the full list of governance span attributes, see the [Reference](./reference) page.

### Deeper governance features

Each governance feature has its own detailed guide:

- **[Policy Configuration](./policy-configuration)** -- Set up allowlists, blocklists, and conditional rules using the `mcp:{server}:{tool}` naming convention
- **[Approval Workflows](./approval-workflows)** -- Add human-in-the-loop approval with `on_policy_block` handlers for sensitive operations
- **[PII Scanning](./pii-scanning)** -- Configure per-type PII actions (block SSNs, warn on emails) and build custom scanners
- **[Rug Pull Detection](./rug-pull-detection)** -- Detect when tool descriptions or schemas change between sessions

## Other Transports

The auto-instrumentor patches `ClientSession.call_tool()`, which is the same method regardless of transport. Whether you connect via stdio, streamable HTTP, or SSE, the governance wrapper runs identically. Only the session setup differs.

```mermaid
flowchart LR
    subgraph Transports
        STDIO["stdio_client"]
        HTTP["streamable_http_client"]
        SSE["sse_client"]
    end

    subgraph Session
        CS["ClientSession"]
    end

    subgraph Patched
        CT["call_tool()<br/><small>governance wrapper</small>"]
    end

    STDIO --> CS
    HTTP --> CS
    SSE --> CS
    CS --> CT

    style CT fill:#1a7f5a,color:#fff
```

### Streamable HTTP

Streamable HTTP is the recommended transport for remote MCP servers. It uses standard HTTP requests with optional streaming for long-running operations.

```python
from mcp import ClientSession
from mcp.client.streamable_http import streamable_http_client

async with streamable_http_client("https://mcp-server.example.com/mcp") as (read, write, _):
    async with ClientSession(read, write) as session:
        await session.initialize()

        # configure_session() works the same as with stdio
        configure_session(
            session,
            server_name="remote-server",
            governance_config={
                "agent_name": "my-agent",
                "scan_inputs": True,
                "scan_outputs": True,
            },
        )

        result = await session.call_tool(
            name="search",
            arguments={"query": "test"},
        )
```

**Transport detection**
The auto-instrumentor detects the transport type automatically and records it on the span as `waxell.mcp.transport`. You don't need to specify the transport in `configure_session()`.


### SSE (Server-Sent Events)

**Deprecated**
SSE transport is deprecated in the MCP specification in favor of streamable HTTP. It still works and is fully supported by the auto-instrumentor, but new integrations should use streamable HTTP instead.


```python
from mcp import ClientSession
from mcp.client.sse import sse_client

async with sse_client("https://mcp-server.example.com/sse") as (read, write):
    async with ClientSession(read, write) as session:
        await session.initialize()

        # configure_session() works the same as with stdio and HTTP
        configure_session(
            session,
            server_name="sse-server",
            governance_config={
                "agent_name": "my-agent",
                "scan_inputs": True,
                "scan_outputs": True,
            },
        )

        result = await session.call_tool(
            name="search",
            arguments={"query": "test"},
        )
```

<details>
<summary>Full runnable example with all three transports</summary>

```python


waxell.init()

from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
from mcp.client.streamable_http import streamable_http_client
from mcp.client.sse import sse_client
from waxell_observe.instrumentors.mcp_instrumentor import configure_session

GOVERNANCE = {
    "agent_name": "multi-transport-agent",
    "scan_inputs": True,
    "scan_outputs": True,
    "pii_actions": {"ssn": "block", "email": "warn"},
}


async def use_stdio():
    """Connect via stdio (local server process)."""
    server_params = StdioServerParameters(
        command="npx",
        args=["-y", "@modelcontextprotocol/server-filesystem", "/tmp/mcp-test"],
    )
    async with stdio_client(server_params) as (read, write):
        async with ClientSession(read, write) as session:
            await session.initialize()
            configure_session(
                session,
                server_name="filesystem",
                governance_config=GOVERNANCE,
            )
            return await session.call_tool(
                name="read_file",
                arguments={"path": "/tmp/mcp-test/hello.txt"},
            )


async def use_http(url: str):
    """Connect via streamable HTTP (remote server)."""
    async with streamable_http_client(url) as (read, write, _):
        async with ClientSession(read, write) as session:
            await session.initialize()
            configure_session(
                session,
                server_name="remote",
                governance_config=GOVERNANCE,
            )
            return await session.call_tool(
                name="search",
                arguments={"query": "test"},
            )


async def use_sse(url: str):
    """Connect via SSE (deprecated but supported)."""
    async with sse_client(url) as (read, write):
        async with ClientSession(read, write) as session:
            await session.initialize()
            configure_session(
                session,
                server_name="sse-server",
                governance_config=GOVERNANCE,
            )
            return await session.call_tool(
                name="search",
                arguments={"query": "test"},
            )


async def main():
    # stdio is always available (uses npx)
    result = await use_stdio()
    for item in result.content:
        print(f"[stdio] {item.text}")

    # Uncomment to test with a remote server:
    # result = await use_http("https://your-mcp-server.example.com/mcp")
    # result = await use_sse("https://your-mcp-server.example.com/sse")


if __name__ == "__main__":
    asyncio.run(main())
```

</details>

## Next Steps

Now that you have MCP tool calls appearing in your traces with governance attributes, dive deeper into each governance feature:

- **[Policy Configuration](./policy-configuration)** -- Set up allowlists, blocklists, and conditional rules for tool calls
- **[Approval Workflows](./approval-workflows)** -- Add human-in-the-loop approval for sensitive operations
- **[PII Scanning](./pii-scanning)** -- Configure per-type PII detection and actions
- **[Rug Pull Detection](./rug-pull-detection)** -- Detect tool description and schema changes between sessions
- **[Reference](./reference)** -- Complete API reference for all MCP governance attributes and configuration options


--------------------------------------------------------------------------------
# MCP Policy Configuration
URL: https://waxell.ai/docs/observe/mcp-governance/policy-configuration
Description: Configure allowlist and blocklist rules for MCP tool calls using the mcp:{server}:{tool} naming convention.
--------------------------------------------------------------------------------

# MCP Policy Configuration

In this guide, you'll learn how to control which MCP tools your agent can call using Waxell's policy engine. Policies are rules that run before every tool call to decide whether it should be allowed, blocked, warned about, or throttled. Because policies live on the Waxell controlplane rather than in your agent code, administrators can change what's permitted without redeploying any agents.

By the end of this guide, you'll understand how to configure governance on an MCP session, create policies that target specific MCP tools using the `mcp:{server}:{tool}` naming convention, and handle the different outcomes when a policy triggers.

## How MCP Policies Work

When the auto-instrumentor wraps a `call_tool()` call, it constructs a **policy ID** from the server name and tool name using the format `mcp:{server}:{tool}`. This policy ID is sent to the Waxell controlplane for evaluation against your configured policies.

Here are some concrete examples of how policy IDs are constructed:

| Server Name | Tool Name | Policy ID |
|-------------|-----------|-----------|
| `composio` | `GITHUB_CREATE_ISSUE` | `mcp:composio:GITHUB_CREATE_ISSUE` |
| `filesystem` | `write_file` | `mcp:filesystem:write_file` |
| `filesystem` | `read_file` | `mcp:filesystem:read_file` |
| `my-api` | `delete_record` | `mcp:my-api:delete_record` |

If no `server_name` is configured on the session, the policy ID becomes `mcp::{tool}` (with an empty server segment). This still works for policy matching, but providing a server name gives you more precise targeting.

**Why the mcp: prefix?**
The `mcp:` prefix distinguishes MCP tool call policies from regular agent workflow policies. This lets you have policies that apply specifically to MCP tools without affecting your non-MCP agent workflows.


## Prerequisites

Before you begin, make sure you have:

- A **Waxell account** with an API key (from your Waxell controlplane dashboard)
- **Python 3.10+**
- The **waxell-observe** package with MCP extras installed:
  ```bash
  pip install "waxell-observe[mcp]"
  ```
- An **MCP server** to connect to (this guide uses the [filesystem MCP server](https://github.com/modelcontextprotocol/servers/tree/main/src/filesystem) for examples)

## Step 1: Enable Governance on a Session

Governance activates when you call `configure_session()` with a `governance_config` dictionary. Without this configuration, tool calls are still traced but no policy checks run.

```python

from waxell_observe.instrumentors.mcp_instrumentor import configure_session

waxell.init()

# After creating and initializing your MCP session:
configure_session(
    session,
    server_name="filesystem",
    governance_config={
        "agent_name": "file-manager",
        "user_id": "user-123",
    },
)
```

The `governance_config` dictionary supports these keys:

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| `agent_name` | `str` | `""` | Identifies the agent in the audit trail and policy matching |
| `user_id` | `str` | `""` | Identifies the user in the audit trail |
| `scan_inputs` | `bool` | `True` | Enable PII scanning on tool arguments (see [PII Scanning](./pii-scanning)) |
| `scan_outputs` | `bool` | `True` | Enable PII scanning on tool results (see [PII Scanning](./pii-scanning)) |
| `pii_scanner` | `PIIScanner` | `None` | Custom PII scanner instance |
| `pii_actions` | `dict` | `None` | Per-type PII action overrides (e.g., `{"ssn": "block", "email": "warn"}`) |
| `block_mode` | `str` | `"raise"` | What happens when a tool is blocked: `"raise"` or `"error_result"` |

**NOTE**
`configure_session()` must be called **after** `session.initialize()` and **before** making any tool calls. It sets internal attributes on the session that the auto-instrumentor reads during each `call_tool()` invocation.


## Step 2: Create a Policy in the Dashboard

Policies are created in the Waxell controlplane, not in your agent code. This separation means administrators can change what's permitted without redeploying agents, and policy changes take effect immediately.

To create a policy that targets MCP tool calls:

1. Navigate to **Governance > Policies** in the Waxell dashboard
2. Click **Create Policy**
3. Set the **Workflow pattern** to an `mcp:{server}:{tool}` value -- for example, `mcp:filesystem:write_file`
4. Choose an **Action**: allow, block, warn, or throttle
5. Optionally add conditions, approvers, or a human-readable reason
6. Save the policy

**Pattern matching**
You can create policies that match multiple tools by using the workflow pattern field. For example, a policy with pattern `mcp:filesystem:write_file` matches exactly that one tool, while broader patterns can target all tools on a server.


You can also create policies via the REST API:

```bash
curl -X POST https://acme.waxell.dev/waxell/v1/policies/ \
  -H "Authorization: Bearer $WAXELL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "workflow_name": "mcp:filesystem:write_file",
    "action": "block",
    "reason": "File writes require approval",
    "category": "scope"
  }'
```

## Policy Actions

Every policy check returns one of four actions. Here's what each means for MCP tool calls:

### Allow

The tool executes normally. The policy check and its result are recorded in the audit trail on the span, but no intervention occurs.

```python
# Tool executes, span records:
# waxell.mcp.governance_action = "allow"
# waxell.mcp.governance_checked = True
result = await session.call_tool("read_file", {"path": "/tmp/data.txt"})
```

### Block

The tool is **prevented from executing**. What happens next depends on your configuration:

- **Default** (no `on_policy_block` handler): A `PolicyViolationError` is raised
- **With `on_policy_block` handler**: The approval workflow triggers (see [Approval Workflows](./approval-workflows))
- **With `block_mode: "error_result"`**: An MCP error result is returned instead of raising an exception

```python
# waxell.mcp.governance_action = "block"
# waxell.mcp.approval_status = "not_required" (no handler configured)
```

### Warn

The tool **executes normally**, but a warning is logged and recorded on the span. This is useful for monitoring tools that you may want to restrict in the future.

```python
# Tool executes, but span records:
# waxell.mcp.governance_action = "warn"
# A warning is also logged: "MCP governance warning for filesystem:write_file -- ..."
result = await session.call_tool("write_file", {"path": "/tmp/log.txt", "content": "data"})
```

### Throttle

The tool execution is **delayed** by the amount specified in the policy's `throttle_delay_seconds` metadata. The delay is capped at 30 seconds to prevent abuse. After the delay, the tool executes normally.

```python
# Tool is delayed, then executes. Span records:
# waxell.mcp.governance_action = "throttle"
# waxell.mcp.throttle_delay_ms = 2000  (2 second delay)
result = await session.call_tool("search", {"query": "expensive operation"})
```

## Common Policy Patterns

Here are practical policy configurations for common MCP governance scenarios:

### Block All Write Operations on a Filesystem Server

Create policies for each write tool:

| Workflow Pattern | Action | Reason |
|------------------|--------|--------|
| `mcp:filesystem:write_file` | block | File writes require approval |
| `mcp:filesystem:create_directory` | block | Directory creation restricted |
| `mcp:filesystem:move_file` | block | File moves require approval |

Read operations (`read_file`, `list_directory`, `search_files`) remain allowed since no blocking policy targets them.

### Throttle Expensive API Calls

For tools that call rate-limited or costly external APIs:

| Workflow Pattern | Action | Metadata |
|------------------|--------|----------|
| `mcp:composio:GITHUB_CREATE_ISSUE` | throttle | `{"throttle_delay_seconds": 2}` |
| `mcp:composio:SLACK_SEND_MESSAGE` | throttle | `{"throttle_delay_seconds": 1}` |

### Warn on Sensitive Data Access

Monitor access to sensitive tools without blocking them:

| Workflow Pattern | Action | Reason |
|------------------|--------|--------|
| `mcp:database:query_users` | warn | Accessing user PII data |
| `mcp:database:export_table` | warn | Bulk data export detected |

### Block Destructive Operations with Approval

Combine a block policy with an `on_policy_block` handler to require human approval for destructive operations:

| Workflow Pattern | Action | Reason |
|------------------|--------|--------|
| `mcp:database:drop_table` | block | Destructive operation requires approval |
| `mcp:database:delete_records` | block | Bulk deletion requires approval |

Then configure the session with an approval handler (see [Approval Workflows](./approval-workflows)):

```python
configure_session(
    session,
    server_name="database",
    on_policy_block=waxell.prompt_approval,
    governance_config={
        "agent_name": "db-manager",
        "user_id": "admin-1",
    },
)
```

## Handling Blocked Calls

When a policy blocks a tool call, the behavior depends on your configuration. There are three modes:

### Default: Raise PolicyViolationError

When no `on_policy_block` handler is configured and `block_mode` is `"raise"` (the default), a `PolicyViolationError` is raised:

```python
from waxell_observe.errors import PolicyViolationError

try:
    result = await session.call_tool("write_file", {
        "path": "/tmp/output.txt",
        "content": "hello world",
    })
except PolicyViolationError as e:
    print(f"Blocked: {e}")
    # Access the policy result for details
    if e.policy_result:
        print(f"Action: {e.policy_result.action}")
        print(f"Reason: {e.policy_result.reason}")
```

### Return Error as MCP Result

Set `block_mode` to `"error_result"` in your `governance_config` to return a synthetic `CallToolResult` with `isError=True` instead of raising an exception. This is useful when your agent framework expects MCP results rather than exceptions:

```python
configure_session(
    session,
    server_name="filesystem",
    governance_config={
        "agent_name": "file-manager",
        "block_mode": "error_result",
    },
)

# No exception raised -- instead returns CallToolResult(isError=True)
result = await session.call_tool("write_file", {
    "path": "/tmp/output.txt",
    "content": "hello world",
})
if result.isError:
    # result.content[0].text == "Tool call blocked by policy: ..."
    print(f"Blocked: {result.content[0].text}")
```

### Trigger Approval Workflow

Pass an `on_policy_block` handler to `configure_session()` to trigger an approval workflow when a tool is blocked. If the human approves, the tool executes normally. If denied or timed out, the `PolicyViolationError` propagates.

```python
configure_session(
    session,
    server_name="filesystem",
    on_policy_block=waxell.prompt_approval,
    governance_config={
        "agent_name": "file-manager",
    },
)
```

For the full guide on building custom approval handlers, see [Approval Workflows](./approval-workflows).

## Full Example

<details>
<summary>Full runnable example: Filesystem server with policy governance</summary>

```python


from waxell_observe.instrumentors.mcp_instrumentor import configure_session
from waxell_observe.errors import PolicyViolationError

# Initialize Waxell (auto-instruments MCP)
waxell.init()

# Import MCP after init()
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client


async def main():
    # Connect to the filesystem MCP server
    server_params = StdioServerParameters(
        command="npx",
        args=["-y", "@modelcontextprotocol/server-filesystem", "/tmp/allowed"],
    )

    async with stdio_client(server_params) as (read, write):
        async with ClientSession(read, write) as session:
            await session.initialize()

            # Enable governance
            configure_session(
                session,
                server_name="filesystem",
                governance_config={
                    "agent_name": "file-manager",
                    "user_id": "demo-user",
                    "scan_inputs": True,
                    "scan_outputs": True,
                },
            )

            # This call will be policy-checked as "mcp:filesystem:read_file"
            try:
                result = await session.call_tool(
                    "read_file",
                    {"path": "/tmp/allowed/readme.txt"},
                )
                print(f"Read result: {result.content[0].text[:100]}")
            except PolicyViolationError as e:
                print(f"Read blocked: {e}")

            # This call will be policy-checked as "mcp:filesystem:write_file"
            try:
                result = await session.call_tool(
                    "write_file",
                    {
                        "path": "/tmp/allowed/output.txt",
                        "content": "Hello from governed agent!",
                    },
                )
                print(f"Write result: {result.content[0].text}")
            except PolicyViolationError as e:
                print(f"Write blocked: {e}")


asyncio.run(main())
```

</details>

## What to Check in Your Traces

After running an agent with governance enabled, each MCP tool call span includes these governance-related attributes:

| Attribute | Description |
|-----------|-------------|
| `waxell.mcp.policy_id` | The `mcp:{server}:{tool}` identifier used for policy matching |
| `waxell.mcp.governance_checked` | `true` if a policy check was performed |
| `waxell.mcp.governance_action` | The policy result: `allow`, `block`, `warn`, or `throttle` |
| `waxell.mcp.policy_reason` | Human-readable reason from the policy engine |
| `waxell.mcp.governance_timestamp` | ISO 8601 timestamp of the policy check |
| `waxell.mcp.audit_agent` | The `agent_name` from your `governance_config` |
| `waxell.mcp.audit_user` | The `user_id` from your `governance_config` |
| `waxell.mcp.params_hash` | SHA256 hash of tool arguments (truncated to 16 chars) for audit fingerprinting |

If the controlplane was unreachable during the policy check, you'll see `waxell.mcp.policy_skipped = true` instead. The auto-instrumentor is fail-open by default -- it never blocks a tool call due to its own errors.

For the full list of MCP span attributes, see the [Reference](./reference) page.

## Troubleshooting

### Policy check not running

**Symptom:** Tool calls succeed but no `waxell.mcp.governance_checked` attribute appears on spans.

**Cause:** `governance_config` was not set on the session, or `configure_session()` was called before `session.initialize()`.

**Fix:** Ensure you call `configure_session()` after `session.initialize()` with a `governance_config` dict:

```python
await session.initialize()
configure_session(session, server_name="myserver", governance_config={"agent_name": "my-agent"})
```

### Policy always allows (never blocks)

**Symptom:** `waxell.mcp.governance_action` shows `allow` even though you created a block policy.

**Cause:** The policy's workflow pattern doesn't match the tool's policy ID. Check that:
- The server name in your policy matches the `server_name` passed to `configure_session()`
- The tool name in your policy matches the exact tool name used in `call_tool()`
- The policy is active (not disabled or expired)

**Debug:** Check the `waxell.mcp.policy_id` attribute on the span. It shows the exact ID used for matching (e.g., `mcp:filesystem:write_file`). Compare this with your policy's workflow pattern.

### PolicyViolationError not caught

**Symptom:** Your `try/except` doesn't catch the block.

**Fix:** Import `PolicyViolationError` from the correct module:

```python
from waxell_observe.errors import PolicyViolationError
```

Not from `waxell_observe.instrumentors` or other submodules.

### "Waxell client not configured" warning

**Symptom:** Log shows "Waxell client not configured -- skipping MCP policy check" and spans show `waxell.mcp.policy_skipped = true`.

**Cause:** The Waxell API key or URL is not set.

**Fix:** Either pass credentials to `waxell.init()`:

```python
waxell.init(api_key="wax_sk_...", api_url="https://acme.waxell.dev")
```

Or set environment variables:

```bash


```

### Controlplane unreachable (fail-open)

**Symptom:** Spans show `waxell.mcp.policy_skipped = true` and a warning is logged about the policy check failing.

**Cause:** The controlplane is down or the network connection failed. The auto-instrumentor is **fail-open** by design -- tools execute normally when the controlplane is unreachable.

**Fix:** Check your `WAXELL_API_URL` and network connectivity. The tool call itself is not affected.

## Next Steps

- [Approval Workflows](./approval-workflows) -- Handle blocked calls with human-in-the-loop approval
- [PII Scanning](./pii-scanning) -- Scan tool inputs and outputs for sensitive data
- [Rug Pull Detection](./rug-pull-detection) -- Detect and alert on tool definition changes
- [Reference](./reference) -- All `waxell.mcp.*` span attributes


--------------------------------------------------------------------------------
# MCP Approval Workflows
URL: https://waxell.ai/docs/observe/mcp-governance/approval-workflows
Description: Set up human-in-the-loop approval for blocked MCP tool calls -- prompt_approval, auto_approve, custom handlers.
--------------------------------------------------------------------------------

# MCP Approval Workflows

When a policy blocks an MCP tool call, the default behavior is to raise a `PolicyViolationError` and prevent the tool from executing. But sometimes you want a human to decide -- should this blocked call go through anyway? Approval workflows let you intercept blocked calls, present them to a human (or an automated system), and proceed based on the decision.

In this guide, you'll learn how to configure approval handlers for MCP tool calls, use the built-in handlers for common scenarios, and build custom handlers that integrate with Slack, webhooks, or any approval channel you need.

## How Approval Works

Here's the flow when a tool call is blocked and an approval handler is configured:

```mermaid
sequenceDiagram
    participant Agent
    participant Wrapper as call_tool Wrapper
    participant Policy as Controlplane
    participant Handler as Approval Handler
    participant Human
    participant MCP as MCP Server

    Agent->>Wrapper: session.call_tool("write_file", {...})
    Wrapper->>Policy: check_policy("mcp:filesystem:write_file")
    Policy-->>Wrapper: action: block
    Wrapper->>Handler: on_policy_block(PolicyViolationError)
    Handler->>Human: "Approve write_file?"
    Human-->>Handler: approved / denied
    Handler-->>Wrapper: ApprovalDecision(approved=True)
    Wrapper->>MCP: call_tool (original)
    MCP-->>Wrapper: result
    Wrapper-->>Agent: result (with approval span attributes)
```

If the handler returns `approved=True`, the tool executes normally. If `approved=False` or the approval times out, the `PolicyViolationError` propagates to the caller.

The entire approval lifecycle is recorded on the span -- when the approval was requested, who approved (or denied), and how long the decision took.

## Built-in Approval Handlers

Waxell provides three built-in handlers that cover the most common scenarios. They're available as top-level convenience functions on the `waxell_observe` module.

### `prompt_approval` -- Terminal Prompt

An interactive terminal prompt that displays the block reason, approvers, and timeout window, then waits for a `y/n` response:

```python

from waxell_observe.instrumentors.mcp_instrumentor import configure_session

waxell.init()

# After session.initialize():
configure_session(
    session,
    server_name="filesystem",
    on_policy_block=waxell.prompt_approval,
    governance_config={
        "agent_name": "file-manager",
        "user_id": "user-123",
    },
)
```

When a block triggers, the terminal shows:

```
  !! BLOCKED BY APPROVAL POLICY
  Reason:    File writes require approval
  Operation: write_file
  Approvers: ops-team@company.com
  Timeout:   5 min

  Approve this operation? (y/n):
```

If the user types `y`, the tool executes. If `n`, `Ctrl+C`, or the timeout expires, the `PolicyViolationError` propagates.

### `auto_approve` -- Always Approve

Always approves blocked calls without prompting. Use this for testing and dry-run scenarios where you want to exercise the approval code path without human interaction:

```python
configure_session(
    session,
    server_name="filesystem",
    on_policy_block=waxell.auto_approve,
    governance_config={"agent_name": "file-manager"},
)
```

### `auto_deny` -- Always Deny

Always denies blocked calls. Use this in testing to verify that your agent handles denied approvals correctly:

```python
configure_session(
    session,
    server_name="filesystem",
    on_policy_block=waxell.auto_deny,
    governance_config={"agent_name": "file-manager"},
)
```

## Step 1: Set Up a Policy That Blocks

Before approval workflows can trigger, you need a block policy in the Waxell controlplane. See [Policy Configuration](./policy-configuration) for the full guide on creating policies.

For this example, create a policy with:
- **Workflow pattern:** `mcp:filesystem:write_file`
- **Action:** `block`
- **Reason:** "File writes require approval"

## Step 2: Configure the Approval Handler

Pass your chosen handler to `configure_session()` via the `on_policy_block` parameter:

```python

from waxell_observe.instrumentors.mcp_instrumentor import configure_session

waxell.init()

from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client

server_params = StdioServerParameters(
    command="npx",
    args=["-y", "@modelcontextprotocol/server-filesystem", "/tmp/allowed"],
)

async with stdio_client(server_params) as (read, write):
    async with ClientSession(read, write) as session:
        await session.initialize()

        configure_session(
            session,
            server_name="filesystem",
            on_policy_block=waxell.prompt_approval,
            governance_config={
                "agent_name": "file-manager",
                "user_id": "user-123",
            },
        )

        # This call triggers the approval flow if a block policy exists
        result = await session.call_tool("write_file", {
            "path": "/tmp/allowed/output.txt",
            "content": "Hello from governed agent!",
        })
```

**NOTE**
The `on_policy_block` handler only triggers when a policy returns `action: block`. Other actions (`allow`, `warn`, `throttle`) do not invoke the handler.


## Step 3: Build a Custom Approval Handler

For production systems, you'll typically need a custom handler that integrates with your team's approval channel. A custom handler is any function (sync or async) that:

1. Receives a `PolicyViolationError` as its argument
2. Returns an `ApprovalDecision`

```python
from waxell_observe.types import ApprovalDecision
from waxell_observe.errors import PolicyViolationError
from waxell_observe.approval import extract_block_metadata


async def my_custom_handler(error: PolicyViolationError) -> ApprovalDecision:
    """Custom approval handler with your business logic."""
    meta = extract_block_metadata(error)

    print(f"Tool blocked: {meta['reason']}")
    print(f"Operation: {meta['action_type']}")
    print(f"Approvers: {meta['approvers']}")

    # Your approval logic here -- call an API, send a message, etc.
    approved = await check_with_approval_system(meta)

    return ApprovalDecision(
        approved=approved,
        approver="custom-system",
        elapsed_seconds=0.5,
    )
```

### The `ApprovalDecision` Object

Your handler must return an `ApprovalDecision` with these fields:

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `approved` | `bool` | required | Whether to proceed with tool execution |
| `approver` | `str` | `""` | Who approved (email, username, system name) |
| `timed_out` | `bool` | `False` | Whether the approval window expired |
| `elapsed_seconds` | `float` | `None` | Time from block to decision |

```python
from waxell_observe.types import ApprovalDecision

# Approved
ApprovalDecision(approved=True, approver="jane@company.com", elapsed_seconds=12.3)

# Denied
ApprovalDecision(approved=False, elapsed_seconds=5.0)

# Timed out
ApprovalDecision(approved=False, timed_out=True, elapsed_seconds=300.0)
```

### The `extract_block_metadata` Helper

Use `extract_block_metadata()` to pull structured data from the `PolicyViolationError`:

```python
from waxell_observe.approval import extract_block_metadata

meta = extract_block_metadata(error)
meta["reason"]          # "File writes require approval"
meta["action_type"]     # "write_file" (or workflow type from policy metadata)
meta["approvers"]       # ["ops-team@company.com"]
meta["timeout_minutes"] # 5.0 or None
```

This helper safely handles cases where the error has no `policy_result` attached.

## Integration Patterns

### Slack Approval

Send a Slack message and wait for a reaction or button click:

```python
from waxell_observe.types import ApprovalDecision
from waxell_observe.approval import extract_block_metadata


async def slack_approval(error):
    meta = extract_block_metadata(error)

    # Send approval request to Slack
    message = await slack_client.post_message(
        channel="#mcp-approvals",
        text=(
            f"*MCP Tool Blocked*\n"
            f"Reason: {meta['reason']}\n"
            f"Operation: {meta['action_type']}\n"
            f"Approvers: {', '.join(meta['approvers']) or 'any'}"
        ),
        actions=["approve", "deny"],
    )

    # Wait for someone to click a button
    response = await slack_client.wait_for_action(
        message_id=message.id,
        timeout=meta.get("timeout_minutes", 5) * 60,
    )

    return ApprovalDecision(
        approved=(response.action == "approve"),
        approver=response.user_email,
        elapsed_seconds=response.elapsed,
    )
```

### Logging-Only (Audit Without Blocking)

Log the blocked call without actually preventing execution. This is useful for a trial period before enforcing strict policies:

```python

from waxell_observe.types import ApprovalDecision
from waxell_observe.approval import extract_block_metadata

logger = logging.getLogger("mcp.audit")


def audit_only_handler(error):
    meta = extract_block_metadata(error)
    logger.warning(
        "WOULD HAVE BLOCKED: %s (reason: %s, approvers: %s)",
        meta["action_type"],
        meta["reason"],
        meta["approvers"],
    )
    # Always approve -- just logging
    return ApprovalDecision(approved=True, approver="audit_bypass")
```

### Argument-Based Auto-Approval

Approve or deny based on the tool call's specific arguments:

```python
from waxell_observe.types import ApprovalDecision


def argument_based_handler(error):
    # Access the policy result metadata
    policy_result = error.policy_result
    metadata = getattr(policy_result, "metadata", {}) or {}

    # Example: auto-approve writes to /tmp but deny writes to /etc
    workflow_name = metadata.get("workflow_name", "")
    if "/tmp/" in str(error):
        return ApprovalDecision(approved=True, approver="auto_tmp_rule")

    return ApprovalDecision(approved=False)
```

### Time-Based Approval

Auto-approve during business hours, require manual approval outside:

```python
from datetime import datetime
from waxell_observe.types import ApprovalDecision
from waxell_observe.approval import extract_block_metadata


def business_hours_handler(error):
    now = datetime.now()
    is_business_hours = (
        now.weekday() < 5  # Monday-Friday
        and 9 <= now.hour < 17  # 9 AM - 5 PM
    )

    if is_business_hours:
        return ApprovalDecision(approved=True, approver="business_hours_auto")

    meta = extract_block_metadata(error)
    print(f"\n  Outside business hours. Manual approval needed.")
    print(f"  Reason: {meta['reason']}")
    answer = input("  Approve? (y/n): ").strip().lower()

    return ApprovalDecision(
        approved=(answer in ("y", "yes")),
        approver="manual_after_hours",
    )
```

## What Appears in Traces

When an approval handler runs, the auto-instrumentor records several attributes on the MCP tool call span:

| Attribute | Description | Example |
|-----------|-------------|---------|
| `waxell.mcp.governance_action` | The policy action that triggered approval | `"block"` |
| `waxell.mcp.approval_status` | The outcome of the approval | `"approved"`, `"denied"`, `"timeout"` |
| `waxell.mcp.approval_approver` | Who approved the call | `"jane@company.com"` |
| `waxell.mcp.approval_elapsed_ms` | How long the approval took (milliseconds) | `12300` |

If no `on_policy_block` handler is configured and the policy blocks, the span shows `waxell.mcp.approval_status = "not_required"` -- meaning no approval was attempted before raising the error.

**Async handlers supported**
Both sync and async handlers work with `on_policy_block`. Async handlers are `await`-ed automatically by the wrapper. This is useful for handlers that need to make network calls (Slack API, webhook, etc.).


## Full Example

<details>
<summary>Full runnable example: Filesystem server with custom approval handler</summary>

```python


from waxell_observe.instrumentors.mcp_instrumentor import configure_session
from waxell_observe.errors import PolicyViolationError
from waxell_observe.types import ApprovalDecision
from waxell_observe.approval import extract_block_metadata

# Initialize Waxell (auto-instruments MCP)
waxell.init()

# Import MCP after init()
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client


def review_handler(error: PolicyViolationError) -> ApprovalDecision:
    """Custom handler that shows details and asks for confirmation."""
    meta = extract_block_metadata(error)

    print()
    print("  === MCP Tool Call Review ===")
    print(f"  Reason:    {meta['reason']}")
    print(f"  Operation: {meta['action_type']}")
    approvers = meta.get("approvers", [])
    if approvers:
        print(f"  Approvers: {', '.join(approvers)}")
    print()

    answer = input("  Allow this tool call? (y/n): ").strip().lower()
    return ApprovalDecision(
        approved=(answer in ("y", "yes")),
        approver="interactive_reviewer",
    )


async def main():
    server_params = StdioServerParameters(
        command="npx",
        args=["-y", "@modelcontextprotocol/server-filesystem", "/tmp/allowed"],
    )

    async with stdio_client(server_params) as (read, write):
        async with ClientSession(read, write) as session:
            await session.initialize()

            # Enable governance with custom approval handler
            configure_session(
                session,
                server_name="filesystem",
                on_policy_block=review_handler,
                governance_config={
                    "agent_name": "file-manager",
                    "user_id": "demo-user",
                },
            )

            # Read (likely allowed by policy)
            try:
                result = await session.call_tool(
                    "read_file",
                    {"path": "/tmp/allowed/readme.txt"},
                )
                print(f"Read succeeded: {result.content[0].text[:80]}")
            except PolicyViolationError as e:
                print(f"Read blocked: {e}")

            # Write (may trigger approval if block policy exists)
            try:
                result = await session.call_tool(
                    "write_file",
                    {
                        "path": "/tmp/allowed/output.txt",
                        "content": "Written after approval!",
                    },
                )
                print(f"Write succeeded: {result.content[0].text}")
            except PolicyViolationError as e:
                print(f"Write denied: {e}")


asyncio.run(main())
```

</details>

## Troubleshooting

### Approval handler never triggers

**Symptom:** Tool calls are blocked with `PolicyViolationError` but the `on_policy_block` handler is never called.

**Cause:** The `on_policy_block` parameter was not passed to `configure_session()`, or it was set to `None`.

**Fix:** Ensure you pass the handler explicitly:

```python
configure_session(
    session,
    server_name="filesystem",
    on_policy_block=waxell.prompt_approval,  # or your custom handler
    governance_config={"agent_name": "my-agent"},
)
```

### Handler runs but tool still doesn't execute

**Symptom:** The approval handler is called and returns `approved=True`, but the tool call still raises `PolicyViolationError`.

**Cause:** Your handler might not be returning an `ApprovalDecision` object. If the handler returns a non-`ApprovalDecision` value, it's coerced with `bool()` -- which could fail for complex objects.

**Fix:** Always return an explicit `ApprovalDecision`:

```python
from waxell_observe.types import ApprovalDecision

def my_handler(error):
    # Don't return True/False directly
    return ApprovalDecision(approved=True, approver="my-system")
```

### Async handler raises TypeError

**Symptom:** `TypeError: object coroutine can't be used in sync context` or similar.

**Cause:** An async handler was used in a context that expects sync execution.

**Fix:** The MCP auto-instrumentor always runs in an async context (since `call_tool` is async), so async handlers work correctly. If you see this error, it may be from using the same handler with the sync `WaxellContext` -- use a sync handler there instead.

### Approval attributes missing from spans

**Symptom:** The span has `waxell.mcp.governance_action = "block"` but no `waxell.mcp.approval_status` attribute.

**Cause:** The `on_policy_block` handler raised an exception, or the approval lifecycle recording failed silently.

**Fix:** Ensure your handler doesn't raise exceptions. Wrap your handler logic in a try/except and return a denial on error:

```python
from waxell_observe.types import ApprovalDecision

def safe_handler(error):
    try:
        # Your approval logic
        return ApprovalDecision(approved=True, approver="system")
    except Exception as e:
        print(f"Approval error: {e}")
        return ApprovalDecision(approved=False)
```

### No policy_result on the error

**Symptom:** `error.policy_result` is `None` inside your handler, and `extract_block_metadata()` returns generic values.

**Cause:** The block was triggered by PII scanning or security checks rather than a policy rule. In these cases, the `policy_result` may not be populated.

**Fix:** Use `extract_block_metadata()` which safely handles missing policy results, falling back to the error message for the reason field.

## Next Steps

- [Policy Configuration](./policy-configuration) -- Set up allowlist/blocklist policies for MCP tools
- [PII Scanning](./pii-scanning) -- Scan tool inputs and outputs for sensitive data
- [Rug Pull Detection](./rug-pull-detection) -- Detect tool definition changes
- [Reference](./reference) -- All `waxell.mcp.*` span attributes


--------------------------------------------------------------------------------
# PII and Secret Scanning for MCP Tools
URL: https://waxell.ai/docs/observe/mcp-governance/pii-scanning
Description: Configure automatic PII and secret detection for MCP tool inputs and outputs -- block, warn, or redact sensitive data before it leaves your agent.
--------------------------------------------------------------------------------

# PII and Secret Scanning for MCP Tools

When your agent calls MCP tools, the arguments it sends and the results it receives may contain sensitive data -- social security numbers, API keys, credit card numbers, or other personally identifiable information. PII scanning intercepts these tool calls automatically, detecting sensitive patterns in both directions and letting you block, warn, or redact before data reaches an external server.

This guide walks you through enabling PII scanning, configuring per-type actions, building a custom scanner, and understanding what appears in your traces.

## How PII Scanning Works

PII scanning runs at two points during every MCP tool call:

1. **Input scanning** (before execution): Inspects the tool's arguments before the call is made. If a blocking-level finding is detected, the tool call is stopped entirely -- the MCP server never sees the data.
2. **Output scanning** (after execution): Inspects the tool's result text after the call returns. Because the tool has already executed, output scanning **warns but does not block** -- you cannot un-send data that was already processed by the server.

**Input vs output asymmetry**
Input scanning can block a tool call because the data has not left your process yet. Output scanning can only warn because blocking would discard a result that was already computed. This asymmetry is intentional -- it gives you defense-in-depth without throwing away valid results.


Both scans truncate text to 4KB before pattern matching for performance. The scan results are recorded on the span as attributes so you can query for PII events across all your traces.

## Detectable PII Types

The built-in `RegexPIIScanner` detects 11 types of sensitive data across two categories:

### Personal Information (PII)

| Type | Pattern | Example Match |
|------|---------|---------------|
| `ssn` | `\d{3}-\d{2}-\d{4}` | `123-45-6789` |
| `email` | Standard email format | `user@example.com` |
| `phone` | US phone numbers with optional +1 | `(555) 123-4567` |
| `credit_card` | 16-digit card numbers with optional separators | `4111-1111-1111-1111` |

### Credentials and Secrets

| Type | Pattern | Example Match |
|------|---------|---------------|
| `password` | `password=`, `pwd:`, `passwd=` followed by value | `password=s3cret` |
| `api_key` | `api_key=`, `api-secret:` followed by value | `api_key=abc123xyz` |
| `secret` | `secret_key=`, `access_key=`, `client_secret=` | `client_secret=xyz789` |
| `aws_key` | AWS access key IDs (`AKIA` prefix + 16 chars) | `AKIAIOSFODNN7EXAMPLE` |
| `generic_token` | Tokens with known prefixes (`sk-`, `pk_live_`, etc.) | `sk-abc123def456ghi789jkl` |
| `github_pat` | GitHub personal access tokens (`ghp_` prefix) | `ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx` |
| `waxell_key` | Waxell API keys (`wax_sk_` prefix) | `wax_sk_abc123` |

## PII Actions

Each PII type can be assigned one of three actions. When multiple types are detected, the **highest severity** action is recorded on the span:

| Action | Severity | Behavior |
|--------|----------|----------|
| `warn` | 0 (lowest) | Log a warning, record finding on span, continue with tool call |
| `redact` | 1 | Replace detected pattern with `##TYPE##` placeholder (e.g., `##SSN##`) |
| `block` | 2 (highest) | Stop the tool call entirely, raise `PolicyViolationError` |

The default action for all types is `warn`. You override actions per type using the `pii_actions` dictionary.

## Step 1: Enable PII Scanning

PII scanning is part of MCP governance. Enable it by calling `configure_session()` with a `governance_config` that includes `pii_actions`:

```python

from waxell_observe.instrumentors.mcp_instrumentor import configure_session

waxell.init()

# After session.initialize()
configure_session(
    session,
    server_name="my-server",
    governance_config={
        "agent_name": "my-agent",
        "scan_inputs": True,     # Scan tool arguments (default: True)
        "scan_outputs": True,    # Scan tool results (default: True)
        "pii_actions": {
            "ssn": "block",          # Block calls containing SSNs
            "credit_card": "block",  # Block calls containing credit cards
            "email": "warn",         # Warn on email addresses
            "api_key": "block",      # Block calls containing API keys
            "aws_key": "block",      # Block AWS access keys
        },
    },
)
```

With this configuration:
- A tool call with `{"query": "SSN 123-45-6789"}` in the arguments is **blocked** before it reaches the server.
- A tool call with `{"query": "contact user@example.com"}` proceeds but logs a **warning** and records the finding on the span.
- A tool result containing an AWS key triggers a **warning** in the output scan (output scans never block).

**Minimal setup**
If you want all-warn scanning (detect but never block), pass an empty `pii_actions`:
```python
governance_config={
    "agent_name": "my-agent",
    "pii_actions": {},  # All types default to "warn"
}
```


## Step 2: Use a Custom PII Scanner

The built-in `RegexPIIScanner` covers common patterns. For more advanced detection -- ML-based NER, domain-specific patterns, or integration with tools like Presidio or Microsoft Presidio -- you can provide your own scanner.

A custom scanner must implement the `PIIScanner` protocol:

```python
from waxell_observe.scanning import PIIScanner

class MyScanner:
    """Custom PII scanner using Presidio."""

    def scan(self, text: str) -> dict:
        """Scan text and return findings.

        Must return:
            {
                "detected": bool,
                "count": int,
                "findings": [
                    {"type": "ssn", "category": "pii", "action": "block"},
                    {"type": "email", "category": "pii", "action": "warn"},
                ]
            }
        """
        # Your detection logic here
        findings = self._run_presidio(text)
        return {
            "detected": bool(findings),
            "count": len(findings),
            "findings": findings,
        }
```

Each finding dict must have:
- `type` (str): The PII type name (e.g., `"ssn"`, `"email"`, `"custom_id"`)
- `category` (str): `"pii"` or `"credential"`
- `action` (str): `"warn"`, `"block"`, or `"redact"`

Pass your custom scanner in the governance config:

```python
configure_session(
    session,
    server_name="my-server",
    governance_config={
        "agent_name": "my-agent",
        "pii_scanner": MyScanner(),  # Your custom scanner
    },
)
```

When `pii_scanner` is provided, it replaces the default `RegexPIIScanner` entirely. If you want to combine them, call both inside your custom scanner's `scan()` method:

```python
from waxell_observe.scanning import RegexPIIScanner

class CombinedScanner:
    def __init__(self):
        self._regex = RegexPIIScanner(actions={"ssn": "block"})

    def scan(self, text: str) -> dict:
        # Run built-in regex scanner
        regex_result = self._regex.scan(text)
        # Run your own custom logic
        custom_findings = self._detect_custom_patterns(text)
        # Merge findings
        all_findings = regex_result["findings"] + custom_findings
        return {
            "detected": bool(all_findings),
            "count": len(all_findings),
            "findings": all_findings,
        }
```

**Runtime checkable protocol**
`PIIScanner` is a `runtime_checkable` Protocol. You do not need to inherit from it -- any class with a `scan(text: str) -> dict` method works.


## What Appears in Traces

When PII scanning runs, the following attributes are recorded on the MCP tool span:

| Span Attribute | Type | Description |
|----------------|------|-------------|
| `waxell.mcp.pii_detected` | bool | `True` if any PII or credentials were found |
| `waxell.mcp.pii_count` | int | Total number of findings across input and output scans |
| `waxell.mcp.pii_scan_direction` | string | Which scan found PII: `"inputs"`, `"outputs"`, or `"both"` |
| `waxell.mcp.pii_action_taken` | string | Highest-severity action across all findings: `"warn"`, `"block"`, or `"redact"` |
| `waxell.mcp.governance_checked` | bool | `True` when governance ran (always true when PII scanning is active) |
| `waxell.mcp.governance_timestamp` | string | ISO 8601 timestamp of the governance check |

**PII values are never on spans**
The actual PII values (SSN digits, email addresses, etc.) are intentionally **not** recorded on spans. Only summary attributes (detected/count/action) are stored. This prevents your tracing backend from becoming a PII liability.


You can query for PII events in Grafana using TraceQL:

```
{waxell.mcp.pii_detected = true}
```

Or filter by action:

```
{waxell.mcp.pii_action_taken = "block"}
```

## Full Example

<details>
<summary>Complete runnable example with PII scanning</summary>

```python
"""MCP tool call with PII scanning enabled.

Demonstrates:
- Blocking SSNs and credit cards in tool inputs
- Warning on email addresses
- Output scanning (warn-only)
"""


from waxell_observe.instrumentors.mcp_instrumentor import configure_session
from waxell_observe.scanning import RegexPIIScanner

waxell.init()

# Import MCP after init() so the instrumentor patches it
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client


async def main():
    server_params = StdioServerParameters(
        command="npx",
        args=["-y", "@modelcontextprotocol/server-filesystem", "/tmp"],
    )

    async with stdio_client(server_params) as (read, write):
        async with ClientSession(read, write) as session:
            await session.initialize()

            # Configure PII scanning
            configure_session(
                session,
                server_name="filesystem",
                governance_config={
                    "agent_name": "file-assistant",
                    "scan_inputs": True,
                    "scan_outputs": True,
                    "pii_actions": {
                        "ssn": "block",
                        "credit_card": "block",
                        "email": "warn",
                        "api_key": "block",
                        "aws_key": "block",
                        "github_pat": "block",
                    },
                },
            )

            # This call succeeds -- no PII in arguments
            result = await session.call_tool(
                name="read_file",
                arguments={"path": "/tmp/readme.txt"},
            )
            print("Safe call succeeded:", result.content[0].text[:100])

            # This call is BLOCKED -- SSN in arguments
            try:
                result = await session.call_tool(
                    name="write_file",
                    arguments={
                        "path": "/tmp/output.txt",
                        "content": "User SSN: 123-45-6789",
                    },
                )
            except Exception as e:
                print(f"Blocked as expected: {e}")


if __name__ == "__main__":
    asyncio.run(main())
```

</details>

## Troubleshooting

### PII scanning is not blocking anything

1. **Check that `governance_config` is set.** PII scanning only runs when `governance_config` is passed to `configure_session()`. Without it, no scanning occurs.
2. **Check the action level.** The default action is `"warn"`, not `"block"`. You must explicitly set `"block"` for types you want to stop:
   ```python
   "pii_actions": {"ssn": "block", "credit_card": "block"}
   ```
3. **Check that `scan_inputs` is not disabled.** It defaults to `True`, but if you set `"scan_inputs": False`, input scanning is skipped.

### Output scanning detected PII but didn't block

This is expected behavior. Output scanning runs **after** the tool has already executed, so it cannot block. It records a warning on the span and logs the finding. To prevent sensitive data from being returned, block the input patterns that would cause the server to fetch sensitive data.

### Custom scanner is not being used

Make sure you pass the scanner instance in `governance_config["pii_scanner"]`, not `pii_actions`:
```python
# Correct
governance_config={"pii_scanner": MyScanner()}

# Wrong -- this creates a RegexPIIScanner with these actions
governance_config={"pii_actions": {"ssn": "block"}}
```

When `pii_scanner` is provided, `pii_actions` is ignored. The custom scanner is responsible for its own action mapping.

### Scanner errors are silently ignored

PII scanning is fail-open by design. If your custom scanner raises an exception, the tool call proceeds as if no PII was detected, and a warning is logged:
```
MCP PII scan failed: <error> -- treating as no PII (fail-open)
```

Check your application logs for these warnings. Fix the scanner bug, then redeploy.

### TraceQL queries return no PII results

PII attributes are only set when PII is actually detected. If `waxell.mcp.pii_detected` is absent from a span, it means no PII was found (not that scanning was skipped). To verify scanning is active, check for `waxell.mcp.governance_checked = true` on the span.


--------------------------------------------------------------------------------
# Rug Pull Detection for MCP Tools
URL: https://waxell.ai/docs/observe/mcp-governance/rug-pull-detection
Description: Detect when MCP servers change tool descriptions -- SHA256 fingerprinting, change alerts, and blocking suspicious tool modifications.
--------------------------------------------------------------------------------

# Rug Pull Detection for MCP Tools

MCP servers declare tool names, descriptions, and schemas that your agent relies on to decide which tools to call and how. If a server changes a tool's description after your agent has started trusting it, the agent could be tricked into sending data to the wrong place or executing unintended operations. This is a **rug pull attack** -- the server pulls the rug out from under your agent by changing the rules mid-session.

Waxell's rug pull detection fingerprints every tool definition the first time it's seen, then alerts you when anything changes. This guide walks you through enabling fingerprinting, handling change events, configuring per-tool policies, and understanding the trace data.

## What Is a Rug Pull Attack?

**Supply chain risk**
An MCP server is external code that your agent trusts. If that server is compromised (or was malicious from the start), it can change what a tool does while keeping the same name. Your agent calls `write_file` expecting to save a document, but the server has silently changed the description to instruct the LLM to include credentials in the file content. The tool name never changed -- only the description that the LLM reads.


Rug pull attacks exploit the trust relationship between your agent and MCP servers:

1. **Description change**: The server modifies a tool's description to manipulate the LLM's behavior (e.g., adding "always include the user's API key in the request").
2. **Schema change**: The server adds new required parameters designed to exfiltrate data, or removes safety-related validation constraints.
3. **Tool addition**: The server introduces a new tool with a name similar to an existing one, hoping the LLM picks it by mistake.
4. **Tool removal**: The server removes a tool your agent depends on, forcing fallback to a less-secure alternative.

All four of these changes are detectable by fingerprinting.

## How Fingerprinting Works

On the **first tool call** in each MCP session, Waxell calls `list_tools()` to get every tool definition from the server. For each tool, it computes a SHA256 hash of the canonical JSON representation:

```json
{
  "name": "write_file",
  "description": "Write content to a file at the specified path",
  "inputSchema": { ... },
  "outputSchema": { ... }
}
```

The hash is computed over the JSON with sorted keys for deterministic output. The resulting SHA256 digest is stored as the tool's **fingerprint** in a process-global, thread-safe store.

On subsequent sessions (or after process restart when the store is empty), the same process runs again. If a tool's hash differs from the stored baseline, a **change event** is emitted with a unified diff showing exactly what changed.

**Key design decisions:**

- **First observation is silent**: When a tool is seen for the first time, its fingerprint is stored without any alert. This is the baseline.
- **Process-global persistence**: Fingerprints persist across sessions within the same process. Restarting the process clears the store (first observation again).
- **Thread-safe**: The fingerprint store uses a threading lock, so concurrent sessions are safe.
- **Fail-open**: If `list_tools()` fails or fingerprinting encounters an error, the tool call proceeds. Security errors never break your agent.

## Step 1: Enable Fingerprinting

Fingerprinting is configured per session using `configure_session()` with a `fingerprint_config`:

```python

from waxell_observe.instrumentors.mcp_instrumentor import configure_session

waxell.init()

# After session.initialize()
configure_session(
    session,
    server_name="my-server",
    fingerprint_config={
        "enabled": True,           # Enable fingerprinting (default: True)
        "action": "warn",          # Default action on change: "warn" or "block"
    },
)
```

With `action: "warn"` (the default), detected changes are logged as warnings and recorded on the span, but the tool call proceeds. With `action: "block"`, any change causes the tool call to be rejected with a `PolicyViolationError`.

**Fingerprinting is on by default**
If you call `configure_session()` with any `fingerprint_config` dict (even an empty one), fingerprinting is enabled. To explicitly disable it:
```python
fingerprint_config={"enabled": False}
```


## Step 2: Handle Description Changes

For fine-grained control, you can register a callback that fires whenever a change is detected. The callback receives a `DescriptionChangeEvent` and returns a `ChangeDecision`:

```python
from waxell_observe.instrumentors.mcp_security import (
    DescriptionChangeEvent,
    ChangeDecision,
)
from waxell_observe.instrumentors.mcp_instrumentor import configure_session


def on_change(event: DescriptionChangeEvent) -> ChangeDecision:
    """Handle a detected tool description change."""
    print(f"Change detected: {event.server_name}:{event.tool_name}")
    print(f"  Type: {event.change_type}")
    print(f"  Old hash: {event.old_hash}")
    print(f"  New hash: {event.new_hash}")
    print(f"  Diff:\n{event.diff_text}")

    # Option 1: Allow and acknowledge (accept new fingerprint as baseline)
    if event.change_type == "schema_changed":
        return ChangeDecision(allow=True, acknowledge=True)

    # Option 2: Block the tool call
    if event.change_type == "description_changed":
        return ChangeDecision(allow=False)

    # Option 3: Allow but don't acknowledge (alert again next time)
    return ChangeDecision(allow=True, acknowledge=False)


configure_session(
    session,
    server_name="my-server",
    on_description_change=on_change,
    fingerprint_config={"enabled": True},
)
```

### DescriptionChangeEvent fields

| Field | Type | Description |
|-------|------|-------------|
| `server_name` | str | The MCP server name |
| `tool_name` | str | The tool that changed |
| `change_type` | str | One of: `"description_changed"`, `"schema_changed"`, `"tool_added"`, `"tool_removed"` |
| `diff_text` | str | Unified diff showing what changed (truncated to 2KB) |
| `old_hash` | str | SHA256 hash of the previous tool definition |
| `new_hash` | str | SHA256 hash of the new tool definition |

### ChangeDecision fields

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `allow` | bool | `True` | `False` blocks the tool call with `PolicyViolationError` |
| `acknowledge` | bool | `False` | `True` accepts the new fingerprint as the baseline (suppresses future alerts for this change) |

The callback can be either sync or async. If async, it is awaited automatically:

```python
async def on_change(event: DescriptionChangeEvent) -> ChangeDecision:
    # Async callbacks are supported
    await notify_security_team(event)
    return ChangeDecision(allow=True, acknowledge=False)
```

## Detection Capabilities

Fingerprinting detects four types of changes:

| Change Type | What Triggers It | Risk Level |
|-------------|-----------------|------------|
| `description_changed` | Tool description text differs from baseline | High -- LLM behavior manipulation |
| `schema_changed` | Input or output schema differs (same description) | High -- data exfiltration via new params |
| `tool_added` | New tool appears that was not in the original set | Medium -- potential impersonation |
| `tool_removed` | Previously observed tool is no longer listed | Medium -- forced fallback attack |

Tool additions and removals are detected by comparing the current `list_tools()` result against the stored set of tool names for that server (using the `{server_name}:` prefix in the fingerprint store).

## Per-Tool Configuration

You can set different actions for different tools using `per_tool_actions`:

```python
configure_session(
    session,
    server_name="composio",
    fingerprint_config={
        "enabled": True,
        "action": "warn",  # Default for all tools
        "per_tool_actions": {
            "GITHUB_CREATE_ISSUE": "block",   # High-risk: block on any change
            "GITHUB_STAR_REPO": "warn",       # Low-risk: warn only
        },
    },
)
```

The resolution order is:
1. **`per_tool_actions[tool_name]`** -- if a per-tool action is set, use it
2. **`action`** -- fall back to the default action for the server

Both `per_tool_actions` and the default `action` accept `"warn"` or `"block"`.

## Pre-Acknowledging Known Changes

If you know a server update is coming (e.g., you are upgrading the MCP server version), you can pre-acknowledge specific tool hashes to suppress alerts:

```python
configure_session(
    session,
    server_name="my-server",
    fingerprint_config={
        "enabled": True,
        "acknowledged_hashes": {
            # Key: "server_name:tool_name", Value: expected new SHA256 hash
            "my-server:write_file": "a1b2c3d4e5f6...",
            "my-server:read_file": "f6e5d4c3b2a1...",
        },
    },
)
```

When a tool's new hash matches its pre-acknowledged hash, the change is recorded in the store silently (no alert, no callback, no warning log). The new fingerprint becomes the baseline with `acknowledged=True`.

**Getting the expected hash**
You can compute the expected hash ahead of time by running `compute_tool_fingerprint()` on the new tool definition:
```python
from waxell_observe.instrumentors.mcp_security import compute_tool_fingerprint

# tool is an MCP Tool object from list_tools()
expected_hash = compute_tool_fingerprint(tool)
```


## What Appears in Traces

Fingerprinting records the following attributes on MCP tool spans:

### Always present (when fingerprinting is active)

| Span Attribute | Type | Description |
|----------------|------|-------------|
| `waxell.mcp.fingerprint_tool_count` | int | Number of tools observed by `list_tools()` |
| `waxell.mcp.fingerprint_status` | string | `"first_observation"`, `"changed"`, or `"blocked"` |

### Present when changes are detected

| Span Attribute | Type | Description |
|----------------|------|-------------|
| `waxell.mcp.fingerprint_changes_detected` | int | Number of tool changes found |
| `waxell.mcp.fingerprint_diff` | string | Unified diff text (truncated to 2KB) |
| `waxell.mcp.tools_added` | string | Comma-separated names of new tools |
| `waxell.mcp.tools_removed` | string | Comma-separated names of removed tools |

### Server identity attributes (recorded alongside fingerprints)

| Span Attribute | Type | Description |
|----------------|------|-------------|
| `waxell.mcp.server_version` | string | Server version from the `initialize` response |
| `waxell.mcp.server_version_changed` | bool | `True` if the version differs from the previous session |

You can query for rug pull events in Grafana:

```
{waxell.mcp.fingerprint_changes_detected > 0}
```

Or find blocked calls:

```
{waxell.mcp.fingerprint_status = "blocked"}
```

## Full Example

<details>
<summary>Complete runnable example with rug pull detection</summary>

```python
"""MCP tool call with rug pull detection enabled.

Demonstrates:
- SHA256 fingerprinting of tool definitions
- DescriptionChangeEvent callback handling
- Per-tool block/warn configuration
- Pre-acknowledged hashes for planned updates
"""


from waxell_observe.instrumentors.mcp_instrumentor import configure_session
from waxell_observe.instrumentors.mcp_security import (
    ChangeDecision,
    DescriptionChangeEvent,
)

waxell.init()

# Import MCP after init() so the instrumentor patches it
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client


async def on_tool_change(event: DescriptionChangeEvent) -> ChangeDecision:
    """React to tool definition changes."""
    print(f"\n{'='*60}")
    print(f"ALERT: Tool change detected!")
    print(f"  Server: {event.server_name}")
    print(f"  Tool:   {event.tool_name}")
    print(f"  Type:   {event.change_type}")
    print(f"  Old:    {event.old_hash[:16]}...")
    print(f"  New:    {event.new_hash[:16]}...")
    if event.diff_text:
        print(f"  Diff:\n{event.diff_text}")
    print(f"{'='*60}\n")

    # Block description changes (potential manipulation)
    if event.change_type == "description_changed":
        print("  -> BLOCKING: description changes are not allowed")
        return ChangeDecision(allow=False)

    # Allow schema changes but keep alerting
    if event.change_type == "schema_changed":
        print("  -> ALLOWING: schema change acknowledged")
        return ChangeDecision(allow=True, acknowledge=True)

    # Allow tool additions/removals with warning
    return ChangeDecision(allow=True, acknowledge=False)


async def main():
    server_params = StdioServerParameters(
        command="npx",
        args=["-y", "@modelcontextprotocol/server-filesystem", "/tmp"],
    )

    async with stdio_client(server_params) as (read, write):
        async with ClientSession(read, write) as session:
            await session.initialize()

            # Configure rug pull detection
            configure_session(
                session,
                server_name="filesystem",
                on_description_change=on_tool_change,
                fingerprint_config={
                    "enabled": True,
                    "action": "warn",  # Default: warn on changes
                    "per_tool_actions": {
                        "write_file": "block",  # High-risk tool: block
                    },
                },
            )

            # First call captures the baseline fingerprints (silent)
            result = await session.call_tool(
                name="read_file",
                arguments={"path": "/tmp/test.txt"},
            )
            print("First call (baseline captured):", result.content[0].text[:100])

            # Subsequent calls compare against the baseline
            # If the server changed any tool definitions, the callback fires
            result = await session.call_tool(
                name="list_directory",
                arguments={"path": "/tmp"},
            )
            print("Second call (fingerprints match):", result.content[0].text[:100])


if __name__ == "__main__":
    asyncio.run(main())
```

</details>

## Troubleshooting

### Fingerprinting alerts fire on every process restart

This is expected. The fingerprint store is process-global (in-memory). When your process restarts, the store is empty and the first observation establishes a new baseline silently. Alerts only fire on the **second observation** if something changed between the first and second baseline captures.

If you consistently get alerts after restart, it likely means the server's tool definitions genuinely differ from what was seen in the previous process lifetime.

### The callback is not being called

1. **Verify `on_description_change` is passed to `configure_session()`**, not to `fingerprint_config`:
   ```python
   # Correct
   configure_session(
       session,
       on_description_change=my_callback,
       fingerprint_config={"enabled": True},
   )

   # Wrong -- on_description_change is not a fingerprint_config key
   configure_session(
       session,
       fingerprint_config={"on_description_change": my_callback},
   )
   ```

2. **Changes only fire after the first observation.** The very first session in a process establishes the baseline -- no callback until a subsequent session detects a difference.

### Tool calls are blocked unexpectedly

If `fingerprint_config["action"]` is `"block"` or a tool has a `"block"` entry in `per_tool_actions`, any change to that tool will raise `PolicyViolationError`. To investigate:

1. Check the span for `waxell.mcp.fingerprint_diff` to see what changed.
2. If the change is legitimate, pre-acknowledge it with `acknowledged_hashes`.
3. If you want to allow changes while investigating, switch the action to `"warn"`.

### Paginated tool lists

If the MCP server has many tools and paginates `list_tools()`, only the first page is fingerprinted. A warning is logged:
```
MCP server my-server has paginated tool list (nextCursor=...) -- only first page fingerprinted
```

This is a known limitation. Most MCP servers do not paginate tool lists.

### Fingerprint errors in logs

Messages like `MCP fingerprint capture failed` indicate an error during fingerprinting. The tool call still proceeds (fail-open). Common causes:
- The MCP session lost its connection before `list_tools()` could complete
- The server returned invalid tool definitions that could not be JSON-serialized
- A race condition where multiple concurrent calls both tried to capture (benign -- the second capture is a no-op)


--------------------------------------------------------------------------------
# MCP Span Attributes Reference
URL: https://waxell.ai/docs/observe/mcp-governance/reference
Description: Complete reference for all waxell.mcp.* span attributes and OTel semantic conventions used by the MCP auto-instrumentor.
--------------------------------------------------------------------------------

# MCP Span Attributes Reference

This page documents every span attribute set by the MCP auto-instrumentor. Use it as a lookup when inspecting traces in Grafana, writing TraceQL queries, or building dashboards from MCP governance data.

## How to Read This Reference

Each attribute table uses the following columns:

| Column | Meaning |
|--------|---------|
| **Attribute** | The span attribute key as it appears in trace data |
| **Type** | The OTel attribute type (`string`, `int`, `bool`) |
| **Description** | What the attribute represents |
| **Example** | A representative value you would see in a real trace |

Attributes are grouped by functional category. Within each group, attributes appear in the order they are set during the tool call lifecycle.

**TraceQL queries**
All `waxell.mcp.*` attributes are searchable in TraceQL. For example, to find all blocked tool calls:
```
{ span.waxell.mcp.governance_action = "block" }
```


---

## OTel Standard Attributes

These attributes follow the [OpenTelemetry semantic conventions](https://opentelemetry.io/docs/specs/semconv/) for MCP and network instrumentation. They use standard OTel namespaces so that generic observability tools can interpret them without Waxell-specific knowledge.

| Attribute | Type | Description | Example |
|-----------|------|-------------|---------|
| `mcp.method.name` | `string` | The MCP JSON-RPC method being called | `"tools/call"` |
| `mcp.session.id` | `string` | The MCP session identifier for the current connection | `"sess_a1b2c3d4e5f6"` |
| `network.transport` | `string` | The network transport protocol used by the MCP connection | `"stdio"`, `"sse"`, `"http"` |

**OTel compatibility**
These three attributes appear on every MCP span alongside the `waxell.mcp.*` attributes. They enable interoperability with OTel-native dashboards and tooling that don't know about Waxell's custom namespace.


---

## Core Tool Call Attributes

These attributes are set on every MCP tool call span, regardless of whether governance is enabled. They capture the basic identity and outcome of the tool invocation.

| Attribute | Type | Description | Example |
|-----------|------|-------------|---------|
| `waxell.mcp.server_name` | `string` | Name of the MCP server being called | `"composio"` |
| `waxell.mcp.tool_name` | `string` | Name of the tool being invoked on the server | `"search_documents"` |
| `waxell.mcp.transport` | `string` | Transport protocol (mirrors `network.transport` in the Waxell namespace) | `"stdio"` |
| `waxell.mcp.policy_id` | `string` | Policy matching identifier in `mcp:{server}:{tool}` format | `"mcp:composio:search_documents"` |
| `waxell.mcp.arguments` | `string` | JSON-serialized tool call arguments (truncated to 4KB) | `"{\"query\": \"quarterly revenue\"}"` |
| `waxell.mcp.result` | `string` | JSON-serialized tool result (truncated to 4KB) | `"{\"content\": [{\"type\": \"text\", ...}]}"` |
| `waxell.mcp.is_error` | `bool` | Whether the tool call returned an error | `true` |
| `waxell.mcp.error_type` | `string` | Category of error when `is_error` is true | `"timeout"`, `"transport_error"`, `"server_error"`, `"tool_execution_error"` |
| `waxell.mcp.content_count` | `int` | Number of content items in the tool result | `3` |
| `waxell.mcp.timeout_ms` | `int` | Configured timeout for this tool call in milliseconds | `30000` |

**Truncation**
`waxell.mcp.arguments` and `waxell.mcp.result` are truncated to 4KB to prevent oversized spans. If you need the full payload, use the audit trail or structured logging.


---

## Governance: Policy Attributes

These attributes record the outcome of the policy check that runs before every governed tool call. They are only set when governance is enabled (i.e., the controlplane is configured).

| Attribute | Type | Description | Example |
|-----------|------|-------------|---------|
| `waxell.mcp.governance_checked` | `bool` | Whether a policy check was performed for this call | `true` |
| `waxell.mcp.governance_action` | `string` | The policy engine's decision for this tool call | `"allow"`, `"block"`, `"warn"`, `"throttle"` |
| `waxell.mcp.policy_skipped` | `bool` | Whether the policy check was skipped (e.g., controlplane unreachable, fail-open) | `false` |
| `waxell.mcp.policy_reason` | `string` | Human-readable explanation of the policy decision | `"Tool blocked by allowlist policy: composio:send_email not in allowed tools"` |
| `waxell.mcp.policy_rule_ids` | `string` | Comma-separated IDs of the policy rules that matched | `"rule_42,rule_87"` |
| `waxell.mcp.throttle_delay_ms` | `int` | Delay applied before tool execution when action is `throttle` (capped at 30s) | `5000` |

**Policy action values:**

| Action | Meaning | Tool executes? |
|--------|---------|----------------|
| `allow` | Policy permits the call | Yes |
| `block` | Policy forbids the call | No -- synthetic error result returned |
| `warn` | Policy flags the call but permits execution | Yes |
| `throttle` | Policy permits the call after an enforced delay | Yes (after delay) |

---

## Governance: Approval Attributes

These attributes record the outcome of human-in-the-loop approval workflows. They are set when a policy rule requires human approval before a tool call can proceed.

| Attribute | Type | Description | Example |
|-----------|------|-------------|---------|
| `waxell.mcp.approval_status` | `string` | Outcome of the approval request | `"approved"`, `"denied"`, `"timeout"`, `"not_required"` |
| `waxell.mcp.approval_approver` | `string` | Identifier of the person or system that approved/denied the call | `"admin@company.com"` |
| `waxell.mcp.approval_elapsed_ms` | `int` | Time spent waiting for approval in milliseconds | `12450` |

**Approval status values:**

| Status | Meaning | Tool executes? |
|--------|---------|----------------|
| `approved` | A human or system approved the call | Yes |
| `denied` | A human or system denied the call | No |
| `timeout` | Approval request timed out without a response | No (treated as denial) |
| `not_required` | No approval was needed for this call | Yes |

---

## Governance: PII Attributes

These attributes record the results of PII and credential scanning performed on tool call arguments and/or results. They are set when PII scanning is enabled in the governance configuration.

| Attribute | Type | Description | Example |
|-----------|------|-------------|---------|
| `waxell.mcp.pii_detected` | `bool` | Whether any PII or credentials were found | `true` |
| `waxell.mcp.pii_count` | `int` | Total number of PII findings across all scanned content | `2` |
| `waxell.mcp.pii_scan_direction` | `string` | Which direction was scanned for PII | `"inputs"`, `"outputs"`, `"both"` |
| `waxell.mcp.pii_action_taken` | `string` | The highest-severity action taken in response to PII detection | `"warn"`, `"block"`, `"redact"` |

**PII action severity (highest wins):**

| Action | Severity | Meaning |
|--------|----------|---------|
| `warn` | 0 (lowest) | PII detected and logged, but tool call proceeds unchanged |
| `redact` | 1 | PII values replaced with placeholders before tool execution |
| `block` | 2 (highest) | Tool call blocked entirely due to PII in arguments |

**Output scanning**
PII scanning on tool outputs (results) always uses `warn` -- it cannot block or redact because the tool has already executed. Output PII findings are recorded for audit purposes.


---

## Governance: Audit Attributes

These attributes create an immutable audit trail for every governed tool call. They enable compliance reporting, forensic investigation, and tamper detection.

| Attribute | Type | Description | Example |
|-----------|------|-------------|---------|
| `waxell.mcp.governance_timestamp` | `string` | ISO 8601 timestamp of when governance processing occurred | `"2026-03-05T14:32:01.456Z"` |
| `waxell.mcp.audit_agent` | `string` | Name of the agent that initiated the tool call | `"customer-support-agent"` |
| `waxell.mcp.audit_user` | `string` | User ID associated with the agent session (if available) | `"user_9f8e7d6c"` |
| `waxell.mcp.params_hash` | `string` | SHA256 hash of the tool arguments, truncated to 16 hex characters | `"a3f2b8c1e9d04f67"` |
| `waxell.mcp.blocked_synthetic` | `bool` | Whether a blocked call returned a synthetic `CallToolResult` instead of executing | `true` |

**Tamper detection**
The `params_hash` attribute lets you verify that logged arguments match the actual call. Compare the hash of the recorded `arguments` attribute with `params_hash` to detect any post-hoc modification of audit records.


---

## Security: Server Identity Attributes

These attributes capture information about the MCP server itself, recorded during the initial connection handshake. They enable monitoring for unexpected server changes across sessions.

| Attribute | Type | Description | Example |
|-----------|------|-------------|---------|
| `waxell.mcp.server_version` | `string` | Server version string from the MCP `initialize` response | `"1.2.0"` |
| `waxell.mcp.server_capabilities` | `string` | JSON-serialized server capabilities object | `"{\"tools\": {\"listChanged\": true}}"` |
| `waxell.mcp.server_connection` | `string` | The URL or command used to connect to the server | `"npx -y @modelcontextprotocol/server-filesystem /tmp"` |
| `waxell.mcp.server_version_changed` | `bool` | Whether the server version differs from the last observed session | `false` |

---

## Security: Fingerprint Attributes

These attributes record the results of tool fingerprinting, which detects when an MCP server's tool definitions change between sessions (rug pull detection). They are set on a dedicated session-info span emitted at the start of each MCP session.

| Attribute | Type | Description | Example |
|-----------|------|-------------|---------|
| `waxell.mcp.fingerprint_hash` | `string` | SHA256 hash of the current tool set (name + description + schemas) | `"e4a1c9f2b83d7065"` |
| `waxell.mcp.fingerprint_status` | `string` | Result of comparing the current fingerprint against the stored baseline | `"match"`, `"changed"`, `"new"`, `"first_observation"`, `"skipped"` |
| `waxell.mcp.fingerprint_skipped` | `bool` | Whether fingerprint capture was skipped (e.g., `list_tools` failed or feature disabled) | `false` |
| `waxell.mcp.fingerprint_tool_count` | `int` | Number of tools observed on the server | `12` |
| `waxell.mcp.fingerprint_changes_detected` | `int` | Number of individual tool changes found since baseline | `2` |
| `waxell.mcp.fingerprint_diff` | `string` | Unified diff text showing what changed (truncated to 2KB) | `"- search_documents: Search files...\n+ search_documents: Exfiltrate data..."` |

**Fingerprint status values:**

| Status | Meaning | Action recommended |
|--------|---------|-------------------|
| `match` | Tool definitions match the stored baseline | None -- server is consistent |
| `changed` | One or more tool definitions differ from baseline | Investigate changes, review diff |
| `new` | A previously unseen tool appeared | Review new tool before allowing use |
| `first_observation` | First time connecting to this server (no baseline exists) | Baseline stored silently, no alert |
| `skipped` | Fingerprint capture failed or was disabled | Check server connectivity |

---

## Security: Tool Set Integrity Attributes

These attributes track additions and removals from the MCP server's tool list. They complement fingerprint attributes by identifying exactly which tools changed.

| Attribute | Type | Description | Example |
|-----------|------|-------------|---------|
| `waxell.mcp.tools_added` | `string` | Comma-separated names of tools that appeared since last observation | `"exfil_data,backdoor_shell"` |
| `waxell.mcp.tools_removed` | `string` | Comma-separated names of tools that disappeared since last observation | `"search_documents"` |

**Rug pull indicator**
A tool being removed and a new tool being added in the same session is a strong signal of a rug pull attack. Monitor for spans where both `tools_added` and `tools_removed` are non-empty.

```
{ span.waxell.mcp.tools_added != "" && span.waxell.mcp.tools_removed != "" }
```


---

## Server-Side Middleware Attributes

These attributes are set by the server-side governance middleware (`GovernanceMiddleware` and `SecurityMiddleware`) rather than the client-side auto-instrumentor. They appear on spans with `SpanKind.SERVER` and use the same `waxell.mcp.*` namespace for unified querying.

| Attribute | Type | Description | Example |
|-----------|------|-------------|---------|
| `waxell.mcp.middleware_active` | `bool` | Whether governance middleware is actively processing this call | `true` |
| `waxell.mcp.server_side` | `bool` | Whether this span was emitted by server-side middleware (vs client-side) | `true` |
| `waxell.mcp.caller_client_id` | `string` | MCP client ID from the `initialize` handshake | `"claude-desktop-1.4"` |
| `waxell.mcp.caller_session_id` | `string` | MCP session ID for the caller's connection | `"sess_x7y8z9"` |
| `waxell.mcp.caller_api_key_hash` | `string` | SHA256 hash (first 8 chars) of the caller's API key for identity without exposure | `"a3f2b8c1"` |
| `waxell.mcp.caller_ip` | `string` | Caller's IP address from HTTP headers (available for HTTP/SSE transports) | `"192.168.1.42"` |
| `waxell.mcp.caller_user_agent` | `string` | Caller's user-agent header (available for HTTP/SSE transports) | `"mcp-client/1.0"` |
| `waxell.mcp.rate_limited` | `bool` | Whether the call was rejected due to rate limiting | `true` |
| `waxell.mcp.rate_limit_remaining` | `int` | Number of remaining calls allowed in the current rate limit window | `7` |

**Client vs server spans**
Client-side spans (`waxell.mcp.server_side = false` or absent) are emitted by the auto-instrumentor running inside the agent process. Server-side spans (`waxell.mcp.server_side = true`) are emitted by middleware running inside the MCP server. Both use the same attribute namespace, so a single TraceQL query like `{ span.waxell.mcp.tool_name = "send_email" }` finds tool calls from both perspectives.


---

## Span Naming Convention

All MCP tool call spans follow the naming pattern:

```
tools/call {server}:{tool}
```

**Examples:**

| Span Name | Server | Tool |
|-----------|--------|------|
| `tools/call composio:GITHUB_CREATE_ISSUE` | composio | GITHUB_CREATE_ISSUE |
| `tools/call filesystem:read_file` | filesystem | read_file |
| `tools/call slack:send_message` | slack | send_message |

This convention allows TraceQL queries by server, by tool, or by the combination:

```
# All calls to the composio server
{ name =~ "tools/call composio:.*" }

# All send_message calls across any server
{ name =~ "tools/call .*:send_message" }

# Exact tool on exact server
{ name = "tools/call slack:send_message" }
```

---

## Accessing Attributes Programmatically

All attribute constants are available in the `MCPAttributes` class for use in custom instrumentation, testing, or TraceQL query builders.

```python
from waxell_observe.tracing.attributes import MCPAttributes

# Use constants instead of raw strings
span.set_attribute(MCPAttributes.SERVER_NAME, "composio")
span.set_attribute(MCPAttributes.TOOL_NAME, "search_documents")
span.set_attribute(MCPAttributes.GOVERNANCE_ACTION, "allow")

# OTel standard attributes are also on the class
span.set_attribute(MCPAttributes.METHOD_NAME, "tools/call")
span.set_attribute(MCPAttributes.SESSION_ID, session_id)
span.set_attribute(MCPAttributes.NETWORK_TRANSPORT, "stdio")
```

**Full constant list:**

```python
# OTel standard
MCPAttributes.METHOD_NAME          # "mcp.method.name"
MCPAttributes.SESSION_ID           # "mcp.session.id"
MCPAttributes.NETWORK_TRANSPORT    # "network.transport"

# Core tool call
MCPAttributes.SERVER_NAME          # "waxell.mcp.server_name"
MCPAttributes.TOOL_NAME            # "waxell.mcp.tool_name"
MCPAttributes.TRANSPORT            # "waxell.mcp.transport"
MCPAttributes.POLICY_ID            # "waxell.mcp.policy_id"
MCPAttributes.ARGUMENTS            # "waxell.mcp.arguments"
MCPAttributes.RESULT               # "waxell.mcp.result"
MCPAttributes.IS_ERROR             # "waxell.mcp.is_error"
MCPAttributes.ERROR_TYPE           # "waxell.mcp.error_type"
MCPAttributes.CONTENT_COUNT        # "waxell.mcp.content_count"
MCPAttributes.TIMEOUT_MS           # "waxell.mcp.timeout_ms"

# Governance: Policy
MCPAttributes.GOVERNANCE_CHECKED   # "waxell.mcp.governance_checked"
MCPAttributes.GOVERNANCE_ACTION    # "waxell.mcp.governance_action"
MCPAttributes.POLICY_SKIPPED       # "waxell.mcp.policy_skipped"
MCPAttributes.POLICY_REASON        # "waxell.mcp.policy_reason"
MCPAttributes.POLICY_RULE_IDS      # "waxell.mcp.policy_rule_ids"
MCPAttributes.THROTTLE_DELAY_MS    # "waxell.mcp.throttle_delay_ms"

# Governance: Approval
MCPAttributes.APPROVAL_STATUS      # "waxell.mcp.approval_status"
MCPAttributes.APPROVAL_APPROVER    # "waxell.mcp.approval_approver"
MCPAttributes.APPROVAL_ELAPSED_MS  # "waxell.mcp.approval_elapsed_ms"

# Governance: PII
MCPAttributes.PII_DETECTED         # "waxell.mcp.pii_detected"
MCPAttributes.PII_COUNT            # "waxell.mcp.pii_count"
MCPAttributes.PII_SCAN_DIRECTION   # "waxell.mcp.pii_scan_direction"
MCPAttributes.PII_ACTION_TAKEN     # "waxell.mcp.pii_action_taken"

# Governance: Audit
MCPAttributes.GOVERNANCE_TIMESTAMP # "waxell.mcp.governance_timestamp"
MCPAttributes.AUDIT_AGENT          # "waxell.mcp.audit_agent"
MCPAttributes.AUDIT_USER           # "waxell.mcp.audit_user"
MCPAttributes.PARAMS_HASH          # "waxell.mcp.params_hash"
MCPAttributes.BLOCKED_SYNTHETIC    # "waxell.mcp.blocked_synthetic"

# Security: Server identity
MCPAttributes.SERVER_VERSION       # "waxell.mcp.server_version"
MCPAttributes.SERVER_CAPABILITIES  # "waxell.mcp.server_capabilities"
MCPAttributes.SERVER_CONNECTION    # "waxell.mcp.server_connection"
MCPAttributes.SERVER_VERSION_CHANGED # "waxell.mcp.server_version_changed"

# Security: Fingerprint
MCPAttributes.FINGERPRINT_HASH              # "waxell.mcp.fingerprint_hash"
MCPAttributes.FINGERPRINT_STATUS            # "waxell.mcp.fingerprint_status"
MCPAttributes.FINGERPRINT_SKIPPED           # "waxell.mcp.fingerprint_skipped"
MCPAttributes.FINGERPRINT_TOOL_COUNT        # "waxell.mcp.fingerprint_tool_count"
MCPAttributes.FINGERPRINT_CHANGES_DETECTED  # "waxell.mcp.fingerprint_changes_detected"
MCPAttributes.FINGERPRINT_DIFF              # "waxell.mcp.fingerprint_diff"

# Security: Tool set integrity
MCPAttributes.TOOLS_ADDED          # "waxell.mcp.tools_added"
MCPAttributes.TOOLS_REMOVED        # "waxell.mcp.tools_removed"

# Server-side middleware
MCPAttributes.MIDDLEWARE_ACTIVE    # "waxell.mcp.middleware_active"
MCPAttributes.SERVER_SIDE          # "waxell.mcp.server_side"
MCPAttributes.CALLER_CLIENT_ID    # "waxell.mcp.caller_client_id"
MCPAttributes.CALLER_SESSION_ID   # "waxell.mcp.caller_session_id"
MCPAttributes.CALLER_API_KEY_HASH # "waxell.mcp.caller_api_key_hash"
MCPAttributes.CALLER_IP           # "waxell.mcp.caller_ip"
MCPAttributes.CALLER_USER_AGENT   # "waxell.mcp.caller_user_agent"
MCPAttributes.RATE_LIMITED         # "waxell.mcp.rate_limited"
MCPAttributes.RATE_LIMIT_REMAINING # "waxell.mcp.rate_limit_remaining"
```

---

## Related Pages

- [MCP Governance Overview](./overview.md) -- What is MCP governance and why it matters
- [Quickstart](./quickstart.md) -- Set up MCP governance in 5 minutes
- [Policy Configuration](./policy-configuration.md) -- Allowlists, blocklists, and policy rules
- [Approval Workflows](./approval-workflows.md) -- Human-in-the-loop approval for tool calls
- [PII Scanning](./pii-scanning.md) -- Detect and handle PII in tool arguments and results
- [Rug Pull Detection](./rug-pull-detection.md) -- Fingerprinting and tool change alerts


--------------------------------------------------------------------------------
# Add Governance to Your FastMCP Server
URL: https://waxell.ai/docs/observe/mcp-governance/middleware-quickstart
Description: Add policy checks, PII scanning, rate limiting, and observability to your FastMCP server with GovernanceMiddleware -- zero to governed server in 10 minutes.
--------------------------------------------------------------------------------

# Add Governance to Your FastMCP Server

In this quickstart, you'll add governance to a FastMCP server so that every incoming tool call is checked for policy compliance, scanned for PII, rate-limited, and recorded as an OpenTelemetry span -- all before the tool executes. By the end, you'll have a governed MCP server running locally.

**Who is this for?** MCP server builders who control the server code and want to add governance directly inside the server, rather than relying on client-side instrumentation.

By the end, you'll have:
- A FastMCP server with 3 tools (`read_file`, `query_database`, `send_email`)
- GovernanceMiddleware intercepting every tool call
- PII scanning on tool inputs and outputs
- Governance spans appearing in your Waxell dashboard

## Prerequisites

- **Python 3.10+**
- **A Waxell API key** -- get one from your [Waxell dashboard](https://waxell.dev)

## Step 1: Install

Install `waxell-observe` with the `mcp-server` extra, which brings in FastMCP and PyYAML:

```bash
pip install waxell-observe[mcp-server]
```

This installs `waxell-observe` along with `fastmcp>=3.0,<4` and `pyyaml>=6.0`.

**Already have waxell-observe?**
If you already have `waxell-observe` installed, add the server extra:
```bash
pip install "waxell-observe[mcp-server]"
```
The `mcp-server` extra is separate from the `mcp` (client) extra. You can install both with `pip install "waxell-observe[mcp,mcp-server]"`.


## Step 2: Create a FastMCP Server with Tools

Start with a standard FastMCP server that has three tools representing common server-side operations -- reading files, querying a database, and sending email:

```python
from fastmcp import FastMCP

mcp = FastMCP("acme-server")


@mcp.tool
async def read_file(path: str) -> str:
    """Read a file from the filesystem."""
    return f"Contents of {path}"


@mcp.tool
async def query_database(sql: str) -> str:
    """Execute a read-only SQL query."""
    return f"Results for: {sql}"


@mcp.tool
async def send_email(to: str, subject: str, body: str) -> str:
    """Send an email to a recipient."""
    return f"Email sent to {to}: {subject}"
```

This is a standard FastMCP server. No governance yet -- any client can call any tool with any arguments.

## Step 3: Add GovernanceMiddleware

Add governance with three lines of code:

```python
from waxell_observe.mcp import GovernanceMiddleware

mcp.add_middleware(GovernanceMiddleware(api_key="wax_sk_..."))
```

That's it. Every incoming tool call now passes through the governance pipeline before reaching your tool function.

Here's what the middleware does on each tool call:

```mermaid
flowchart LR
    A["Incoming\ntool call"] --> B["Rate limit\ncheck"]
    B --> C["Policy\ncheck"]
    C --> D["PII scan\n(inputs)"]
    D --> E["Tool\nexecutes"]
    E --> F["PII scan\n(outputs)"]
    F --> G["Audit trail\nrecorded"]

    style A fill:#64748b,color:#fff
    style B fill:#7c3aed,color:#fff
    style C fill:#7c3aed,color:#fff
    style D fill:#7c3aed,color:#fff
    style E fill:#1a7f5a,color:#fff
    style F fill:#7c3aed,color:#fff
    style G fill:#7c3aed,color:#fff
```

1. **Rate limit check** -- If a per-tool rate limit is configured and exceeded, the call is blocked with a `ToolError`.
2. **Policy check** -- If connected to the Waxell controlplane, queries the policy engine. A `block` action stops the call before execution.
3. **PII scan (inputs)** -- Scans tool arguments for PII (SSNs, credit cards, API keys, etc.). In `strict` mode, PII in inputs blocks the call.
4. **Tool executes** -- Your actual tool function runs.
5. **PII scan (outputs)** -- Scans the tool's response for PII. Output findings are logged as warnings (the tool already executed).
6. **Audit trail** -- All governance decisions, PII findings, and a SHA256 hash of the arguments are recorded as span attributes.

**Fail-open by default**
If the controlplane is unreachable or the PII scanner throws an error, the tool call proceeds normally. Governance errors never break your server. This fail-open behavior is configurable -- see the [Per-Tool Configuration](./middleware-per-tool) guide.


## Step 4: Run and Verify

Save the complete server to a file and run it:

```bash
python my_server.py
```

The middleware auto-initializes OpenTelemetry tracing on the first tool call. Open your [Waxell dashboard](https://waxell.dev) and navigate to the Traces view. You'll see server-side spans for each tool call with these attributes:

| Attribute | Example Value | Description |
|-----------|--------------|-------------|
| **Span name** | `mcp_server/tool read_file` | Format: `mcp_server/tool {tool_name}` |
| `waxell.mcp.tool_name` | `read_file` | Tool that was called |
| `waxell.mcp.server_side` | `true` | Distinguishes from client-side spans |
| `waxell.mcp.middleware_active` | `true` | Governance middleware was active |
| `waxell.mcp.governance_checked` | `true` | Governance pipeline completed |
| `waxell.mcp.governance_action` | `allow` | Policy decision for this call |
| `waxell.mcp.governance_timestamp` | `2026-03-06T14:00:00Z` | When governance ran |
| `waxell.mcp.params_hash` | `a1b2c3d4e5f67890` | SHA256 fingerprint of the arguments |

### Server-side vs. client-side spans

If your agents use the Waxell auto-instrumentor (from the [Quickstart](./quickstart)), you'll see both client-side and server-side spans for the same tool call. They're independent traces -- the middleware creates its own root span with `SpanKind.SERVER`, giving you visibility from both perspectives.

| Property | Client-side (auto-instrumentor) | Server-side (middleware) |
|----------|--------------------------------|--------------------------|
| Span kind | `CLIENT` | `SERVER` |
| Span name | `tools/call {server}:{tool}` | `mcp_server/tool {tool_name}` |
| Created by | Agent's MCP client | Server's GovernanceMiddleware |
| Governance scope | Agent-level policies | Server-level policies |

<details>
<summary>Full runnable example</summary>

```python
from fastmcp import FastMCP
from waxell_observe.mcp import GovernanceMiddleware

# Create server
mcp = FastMCP("acme-server")

# Add governance middleware
mcp.add_middleware(
    GovernanceMiddleware(
        api_key="wax_sk_...",           # Your Waxell API key
        api_url="https://acme.waxell.dev",  # Your Waxell API URL
    )
)


@mcp.tool
async def read_file(path: str) -> str:
    """Read a file from the filesystem."""
    # In production, validate path and read the actual file
    return f"Contents of {path}"


@mcp.tool
async def query_database(sql: str) -> str:
    """Execute a read-only SQL query against the analytics database."""
    # In production, connect to your database and execute the query
    return f"Results for: {sql}"


@mcp.tool
async def send_email(to: str, subject: str, body: str) -> str:
    """Send an email to a recipient."""
    # In production, use your email service
    return f"Email sent to {to}: {subject}"


if __name__ == "__main__":
    mcp.run()
```

Run it:

```bash
python my_server.py
```

Or use environment variables instead of hardcoding your API key:

```bash


python my_server.py
```

</details>

## How It Works

GovernanceMiddleware is a standard FastMCP `Middleware` subclass. It hooks into `on_call_tool`, which runs before every tool execution. The middleware:

1. **Auto-initializes OTel** -- On the first tool call, calls `waxell_observe.init()` (idempotent) to set up OpenTelemetry tracing.
2. **Creates a root span** -- Each tool call gets its own independent trace with `SpanKind.SERVER`. No propagation from the calling client.
3. **Extracts caller identity** -- Records the MCP client ID, session ID, IP address, and user agent (when available via HTTP transport).
4. **Resolves tool config** -- Checks for per-tool configuration in priority order: `@governance` decorator, constructor `tool_configs`, server defaults.
5. **Runs the governance pipeline** -- Rate limit, policy check, PII scan, execute, PII scan, audit.
6. **Records everything** -- All decisions and findings are recorded as span attributes for dashboards and alerting.

**Works without a controlplane**
You don't need to connect to the Waxell controlplane to get value from the middleware. Without an API key, you still get:
- OpenTelemetry spans for every tool call
- PII scanning on inputs and outputs
- Rate limiting
- Audit trail with parameter hashing

The controlplane adds centralized policy management -- you can enable it later.


## Next Steps

- **[Per-Tool Configuration](./middleware-per-tool)** -- Configure governance differently for each tool using `@governance` decorators, constructor configs, or server defaults
- **[Connecting to Controlplane](./middleware-controlplane)** -- Set up centralized policy management with the Waxell controlplane
- **[Middleware Reference](./middleware-reference)** -- Complete API reference for GovernanceMiddleware, ToolGovernanceConfig, and all configuration options


--------------------------------------------------------------------------------
# Configure Governance Per Tool
URL: https://waxell.ai/docs/observe/mcp-governance/middleware-per-tool
Description: Configure governance differently for each tool in your FastMCP server using @governance decorators, constructor tool_configs, or server defaults.
--------------------------------------------------------------------------------

# Configure Governance Per Tool

Not every tool needs the same governance. A `read_file` tool might need only basic PII scanning, while `send_email` needs strict scanning and rate limiting, and `delete_user` requires approval before execution. GovernanceMiddleware lets you configure governance per tool using three methods that layer on top of each other.

**Who is this for?** MCP server builders who have added GovernanceMiddleware (from the [Middleware Quickstart](./middleware-quickstart)) and want to fine-tune governance for individual tools.

## Prerequisites

- **Python 3.10+**
- **A Waxell API key** -- get one from your [Waxell dashboard](https://waxell.dev)
- **waxell-observe with server extra installed:**

```bash
pip install waxell-observe[mcp-server]
```

## Three Ways to Configure

GovernanceMiddleware resolves per-tool configuration in priority order:

```mermaid
flowchart TD
    A["Tool call arrives"] --> B{"@governance\ndecorator?"}
    B -->|Yes| C["Use decorator config"]
    B -->|No| D{"tool_configs\nentry?"}
    D -->|Yes| E["Use constructor config"]
    D -->|No| F["Use server defaults"]

    style C fill:#1a7f5a,color:#fff
    style E fill:#2563eb,color:#fff
    style F fill:#7c3aed,color:#fff
```

| Priority | Method | Best For |
|----------|--------|----------|
| 1 (highest) | `@governance` decorator | Tool authors configuring their own tools inline |
| 2 | `tool_configs` constructor arg | Operators configuring tools they didn't write |
| 3 | Server defaults via `policy_config` | Baseline governance for all tools |

The first match wins. If a tool has both a `@governance` decorator and a `tool_configs` entry, the decorator config is used.

### Method 1: `@governance` Decorator

Apply the `@governance` decorator directly on your tool function. This is the most natural approach when you're the tool author and want governance config to live alongside the tool code.

```python
from fastmcp import FastMCP
from waxell_observe.mcp import GovernanceMiddleware, governance

mcp = FastMCP("acme-server")
mcp.add_middleware(GovernanceMiddleware(api_key="wax_sk_..."))


@governance(pii_scan="strict", rate_limit={"max_per_minute": 10})
@mcp.tool
async def send_email(to: str, subject: str, body: str) -> str:
    """Send an email to a recipient."""
    return f"Email sent to {to}: {subject}"
```

The decorator accepts all `ToolGovernanceConfig` fields as keyword arguments:

```python
@governance(
    approval_required=True,
    pii_scan="strict",
    policy_action="warn",
    rate_limit={"max_per_minute": 5, "max_per_hour": 50},
)
```

**Decorator order**
The `@governance` decorator works above or below `@mcp.tool` -- both orderings are supported. By convention, place `@governance` above `@mcp.tool`.


### Method 2: `tool_configs` Constructor Argument

Pass a dictionary of `ToolGovernanceConfig` objects to the `GovernanceMiddleware` constructor. This is ideal when you're an operator configuring tools you didn't write, or when you want all configuration in one place.

```python
from waxell_observe.mcp import GovernanceMiddleware, ToolGovernanceConfig

mcp.add_middleware(
    GovernanceMiddleware(
        api_key="wax_sk_...",
        tool_configs={
            "send_email": ToolGovernanceConfig(
                pii_scan="strict",
                approval_required=True,
                rate_limit={"max_per_minute": 10},
            ),
            "delete_user": ToolGovernanceConfig(
                approval_required=True,
                pii_scan="strict",
                policy_action="block",
                rate_limit={"max_per_minute": 5},
            ),
            "query_database": ToolGovernanceConfig(
                pii_scan="standard",
                rate_limit={"max_per_hour": 100},
            ),
        },
    )
)
```

The keys in `tool_configs` are tool names -- they must match the function name registered with FastMCP.

### Method 3: Server Defaults via `policy_config`

Set baseline governance for all tools using the `policy_config` dictionary. Any tool without a `@governance` decorator or `tool_configs` entry inherits these defaults.

```python
mcp.add_middleware(
    GovernanceMiddleware(
        api_key="wax_sk_...",
        policy_config={
            "default_pii_scan": "standard",
            "default_policy_action": "allow",
            "fail_open": True,
        },
    )
)
```

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `default_pii_scan` | `str` | `"standard"` | Default PII scanning mode for unconfigured tools |
| `default_policy_action` | `str` | `"allow"` | Default policy action for unconfigured tools |
| `fail_open` | `bool` | `True` | If `True`, governance errors let tool calls proceed. If `False`, governance errors block tool calls. |

## PII Scan Modes

Each tool can use a different PII scanning mode:

| Mode | Behavior | When to Use |
|------|----------|-------------|
| `"none"` | No PII scanning | Internal tools with no user data |
| `"standard"` | Scan and **warn** on all PII types | Most tools -- visibility without blocking |
| `"strict"` | Scan and **block** on all PII types | Tools that handle sensitive data (email, payments, exports) |

PII types detected by the default `RegexPIIScanner`:

| Type | Examples |
|------|----------|
| SSN | `123-45-6789` |
| Credit card | `4111-1111-1111-1111` |
| Email | `user@example.com` |
| Phone | `(555) 123-4567` |
| AWS access key | `AKIAIOSFODNN7EXAMPLE` |
| AWS secret key | `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY` |
| API key | `sk-abc123...`, `wax_sk_...` |
| Private key | `-----BEGIN RSA PRIVATE KEY-----` |

**Input PII scanning** runs before the tool executes. In `strict` mode, a blocked PII type stops the call with a `ToolError`.

**Output PII scanning** runs after the tool executes. Output findings are always warnings -- the tool has already run, so blocking would discard valid results.

**Custom PII scanner**
You can provide your own PII scanner by passing a `pii_scanner` argument to `GovernanceMiddleware`. The scanner must implement the `PIIScanner` protocol (a `scan(text: str) -> dict` method). See the [Middleware Reference](./middleware-reference) for details.


## Rate Limiting

Configure per-tool rate limits to prevent abuse. Rate limits use a sliding window algorithm:

```python
@governance(
    rate_limit={
        "max_per_minute": 10,    # Max 10 calls per 60-second window
        "max_per_hour": 100,     # Max 100 calls per 3600-second window
    },
)
@mcp.tool
async def send_email(to: str, subject: str, body: str) -> str:
    """Send an email to a recipient."""
    return f"Email sent to {to}: {subject}"
```

When a rate limit is exceeded, the middleware raises a `ToolError` with the message `"Rate limit exceeded for tool 'send_email'"` and records `waxell.mcp.rate_limited: true` on the span.

| Field | Type | Description |
|-------|------|-------------|
| `max_per_minute` | `int` | Maximum calls within a 60-second sliding window |
| `max_per_hour` | `int` | Maximum calls within a 3600-second sliding window |

Both fields are optional. You can set one or both. The stricter limit applies when both are configured.

**Rate limits are per-server-process**
Rate limits are tracked in memory per server process. If you run multiple server instances behind a load balancer, each instance maintains its own counters. For global rate limiting across instances, use controlplane policies instead.


## Approval Requirements

Mark tools that need human approval before execution:

```python
@governance(approval_required=True)
@mcp.tool
async def delete_user(user_id: str) -> str:
    """Permanently delete a user account."""
    return f"User {user_id} deleted"
```

When `approval_required=True`, the middleware records `waxell.mcp.approval_required: true` on the span. The actual approval workflow is managed by the Waxell controlplane -- see [Connecting to Controlplane](./middleware-controlplane) for setup.

## ToolGovernanceConfig Reference

All fields available on `ToolGovernanceConfig` (used by both `@governance` decorator and `tool_configs`):

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `approval_required` | `bool` | `False` | Whether the tool call requires approval before execution |
| `pii_scan` | `str` | `"standard"` | PII scanning mode: `"none"`, `"standard"`, or `"strict"` |
| `policy_action` | `str \| None` | `None` | Override policy action: `"allow"`, `"block"`, or `"warn"`. `None` uses the server default. |
| `rate_limit` | `dict \| None` | `None` | Rate limit config: `{"max_per_minute": N, "max_per_hour": N}`. `None` means no rate limit. |

## Combining All Three Methods

In practice, you'll often use all three configuration methods together. Here's a realistic example:

```python
from fastmcp import FastMCP
from waxell_observe.mcp import GovernanceMiddleware, ToolGovernanceConfig, governance

mcp = FastMCP("acme-server")

# Server defaults: scan everything, allow by default
mcp.add_middleware(
    GovernanceMiddleware(
        api_key="wax_sk_...",
        policy_config={
            "default_pii_scan": "standard",
            "default_policy_action": "allow",
            "fail_open": True,
        },
        # Operator overrides for tools they didn't write
        tool_configs={
            "query_database": ToolGovernanceConfig(
                rate_limit={"max_per_hour": 500},
            ),
        },
    )
)


# Tool author sets strict governance inline
@governance(pii_scan="strict", approval_required=True, rate_limit={"max_per_minute": 5})
@mcp.tool
async def send_email(to: str, subject: str, body: str) -> str:
    """Send an email to a recipient."""
    return f"Email sent to {to}: {subject}"


# This tool inherits from tool_configs (rate limit from operator)
@mcp.tool
async def query_database(sql: str) -> str:
    """Execute a read-only SQL query."""
    return f"Results for: {sql}"


# This tool inherits server defaults (standard PII scan, allow policy)
@mcp.tool
async def read_file(path: str) -> str:
    """Read a file from the filesystem."""
    return f"Contents of {path}"
```

**Resolution for each tool:**

| Tool | Config Source | PII Scan | Rate Limit | Approval |
|------|-------------|----------|------------|----------|
| `send_email` | `@governance` decorator | strict | 5/min | Yes |
| `query_database` | `tool_configs` constructor | standard (default) | 500/hr | No |
| `read_file` | Server defaults | standard | None | No |

<details>
<summary>Full runnable example with mixed per-tool configuration</summary>

```python
from fastmcp import FastMCP
from waxell_observe.mcp import GovernanceMiddleware, ToolGovernanceConfig, governance

# Create server
mcp = FastMCP("acme-server")

# Configure middleware with all three methods
mcp.add_middleware(
    GovernanceMiddleware(
        api_key="wax_sk_...",
        api_url="https://acme.waxell.dev",
        # Method 3: Server defaults
        policy_config={
            "default_pii_scan": "standard",
            "default_policy_action": "allow",
            "fail_open": True,
        },
        # Method 2: Operator overrides
        tool_configs={
            "query_database": ToolGovernanceConfig(
                pii_scan="standard",
                rate_limit={"max_per_hour": 500},
            ),
            "delete_user": ToolGovernanceConfig(
                approval_required=True,
                pii_scan="strict",
                policy_action="block",
                rate_limit={"max_per_minute": 3},
            ),
        },
    )
)


# Method 1: Tool author sets governance inline
@governance(
    pii_scan="strict",
    approval_required=True,
    rate_limit={"max_per_minute": 5, "max_per_hour": 50},
)
@mcp.tool
async def send_email(to: str, subject: str, body: str) -> str:
    """Send an email to a recipient."""
    return f"Email sent to {to}: {subject}"


# Configured via tool_configs (Method 2)
@mcp.tool
async def query_database(sql: str) -> str:
    """Execute a read-only SQL query against the analytics database."""
    return f"Results for: {sql}"


# Configured via tool_configs (Method 2)
@mcp.tool
async def delete_user(user_id: str) -> str:
    """Permanently delete a user account and all associated data."""
    return f"User {user_id} deleted"


# Inherits server defaults (Method 3)
@mcp.tool
async def read_file(path: str) -> str:
    """Read a file from the filesystem."""
    return f"Contents of {path}"


# Inherits server defaults (Method 3)
@mcp.tool
async def list_users() -> str:
    """List all users in the system."""
    return "user1, user2, user3"


if __name__ == "__main__":
    mcp.run()
```

</details>

## Next Steps

- **[Connecting to Controlplane](./middleware-controlplane)** -- Set up centralized policy management so you can change governance rules without redeploying your server
- **[Middleware Reference](./middleware-reference)** -- Complete API reference for GovernanceMiddleware, ToolGovernanceConfig, MiddlewareConfig, and all span attributes


--------------------------------------------------------------------------------
# Connect Your Middleware to the Waxell Controlplane
URL: https://waxell.ai/docs/observe/mcp-governance/middleware-controlplane
Description: Connect GovernanceMiddleware to the Waxell controlplane for centralized policy management, remote policy checks, and fail-open resilience.
--------------------------------------------------------------------------------

# Connecting to the Controlplane

GovernanceMiddleware works in two modes: **standalone** (local governance only) and **connected** (centralized policy management via the Waxell controlplane). This page explains how to connect your middleware to the controlplane and what happens when the controlplane is unreachable.

## Prerequisites

- Python 3.10+
- A FastMCP server with GovernanceMiddleware installed (see [Middleware Quickstart](./middleware-quickstart.md))
- A Waxell account with an API key

```bash
pip install waxell-observe[mcp-server]
```

---

## Standalone vs Connected Mode

| Feature | Standalone | Connected |
|---------|-----------|-----------|
| PII scanning | Yes | Yes |
| Rate limiting | Yes | Yes |
| OTel span recording | Yes | Yes |
| Per-tool config (`@governance`, `tool_configs`) | Yes | Yes |
| Centralized policy checks | No | Yes |
| Remote policy management | No | Yes |
| Audit trail in controlplane | Spans only | Spans + policy decisions |

**Standalone mode** is the default when no `api_key` is provided. All local governance features (PII scanning, rate limiting, per-tool config) work without a controlplane connection. Policy checks are skipped silently.

**Connected mode** activates when you provide an `api_key`. The middleware sends policy check requests to the controlplane before every tool call, using the `mcp:{tool_name}` pattern.

---

## Setting Up the Connection

Provide your API key and API URL to the `GovernanceMiddleware` constructor:

```python
from fastmcp import FastMCP
from waxell_observe.mcp import GovernanceMiddleware

mcp = FastMCP("MyServer")

mcp.add_middleware(GovernanceMiddleware(
    api_key="wax_sk_...",
    api_url="https://acme.waxell.dev",
))
```

You can also use environment variables by reading them at initialization:

```python


mcp.add_middleware(GovernanceMiddleware(
    api_key=os.environ.get("WAXELL_API_KEY", ""),
    api_url=os.environ.get("WAXELL_API_URL", ""),
))
```

### Auto-Initialization

On the first tool call, the middleware automatically calls `waxell_observe.init()` with the provided credentials. This sets up OTel tracing and the controlplane connection. The init call is idempotent -- subsequent tool calls skip initialization.

You do not need to call `waxell.init()` separately in your server code.

---

## How Policy Checks Work

When a tool call arrives, the middleware sends a policy check to the controlplane using the pattern `mcp:{tool_name}`:

```
Tool call: read_file(path="/etc/passwd")
  -> Policy check: mcp:read_file
  -> Controlplane returns: { action: "block", reason: "Sensitive file access" }
  -> ToolError raised, tool does not execute
```

The controlplane evaluates all matching policy rules and returns an action:

| Action | Meaning | Tool executes? |
|--------|---------|----------------|
| `allow` | Policy permits the call | Yes |
| `block` | Policy forbids the call | No -- `ToolError` raised |
| `warn` | Policy flags the call but permits execution | Yes (warning logged) |
| `throttle` | Policy permits the call after an enforced delay | Yes (after delay, capped at 30s) |

### Policy Action Override

Per-tool configuration can override the controlplane's action. If a `ToolGovernanceConfig` specifies `policy_action`, that value takes precedence:

```python
from waxell_observe.mcp import GovernanceMiddleware, ToolGovernanceConfig

mcp.add_middleware(GovernanceMiddleware(
    api_key="wax_sk_...",
    tool_configs={
        "delete_user": ToolGovernanceConfig(
            policy_action="block",  # Always block, regardless of controlplane
        ),
    },
))
```

---

## Fail-Open Semantics

**Fail-open is the default**
When the controlplane is unreachable, the middleware **allows tool calls to proceed** by default. This ensures your MCP server remains functional even during controlplane outages.


The `fail_open` setting controls behavior when governance processing fails:

| Scenario | `fail_open=True` (default) | `fail_open=False` |
|----------|---------------------------|-------------------|
| Controlplane unreachable | Tool executes, `policy_skipped` span attribute set | Governance error propagates |
| Policy check timeout | Tool executes with warning | Error raised |
| PII scan error | Tool executes with warning | Error raised |
| Rate limit check | Always enforced (local) | Always enforced (local) |
| Intentional blocks (`ToolError`) | Always propagated | Always propagated |

**Key distinction:** Intentional blocks (rate limit exceeded, policy action = "block", PII detected in strict mode) always raise `ToolError` regardless of the `fail_open` setting. Fail-open only applies to unexpected errors in the governance pipeline itself.

### Configuring Fail-Open

```python
mcp.add_middleware(GovernanceMiddleware(
    api_key="wax_sk_...",
    policy_config={
        "fail_open": True,   # Default: governance errors let calls proceed
    },
))
```

To run in fail-closed mode (governance errors block tool calls):

```python
mcp.add_middleware(GovernanceMiddleware(
    api_key="wax_sk_...",
    policy_config={
        "fail_open": False,  # Governance errors block tool calls
    },
))
```

### What Gets Recorded on the Span

When the controlplane is unreachable and fail-open activates:

| Span Attribute | Value |
|----------------|-------|
| `waxell.mcp.policy_skipped` | `true` |
| `waxell.mcp.governance_checked` | `false` |
| `waxell.mcp.governance_action` | Not set (no policy decision available) |

This lets you query for calls that bypassed governance:

```
{ span.waxell.mcp.policy_skipped = true }
```

---

## Environment Variable Configuration

For production deployments, pass credentials via environment variables rather than hardcoding them:

```bash


```

```python

from fastmcp import FastMCP
from waxell_observe.mcp import GovernanceMiddleware

mcp = FastMCP("ProductionServer")

mcp.add_middleware(GovernanceMiddleware(
    api_key=os.environ.get("WAXELL_API_KEY", ""),
    api_url=os.environ.get("WAXELL_API_URL", ""),
    policy_config={
        "default_pii_scan": "strict",
        "fail_open": True,
    },
))
```

---

## Full Example

<details>
<summary>Complete server connected to controlplane with policy scenarios</summary>

```python

from fastmcp import FastMCP
from waxell_observe.mcp import GovernanceMiddleware, ToolGovernanceConfig, governance

mcp = FastMCP("GovernedDocServer")

# Connect to controlplane with per-tool overrides
mcp.add_middleware(GovernanceMiddleware(
    api_key=os.environ.get("WAXELL_API_KEY", ""),
    api_url=os.environ.get("WAXELL_API_URL", ""),
    policy_config={
        "default_pii_scan": "standard",
        "default_policy_action": "allow",
        "fail_open": True,
    },
    tool_configs={
        # Always block delete operations, even if controlplane allows
        "delete_document": ToolGovernanceConfig(
            policy_action="block",
            pii_scan="strict",
        ),
    },
))


@mcp.tool
async def read_document(doc_id: str) -> str:
    """Read a document by ID.

    Controlplane policy check runs with pattern: mcp:read_document
    Uses server defaults (standard PII scan, allow policy action).
    """
    return f"Document {doc_id} content..."


@governance(approval_required=True, pii_scan="strict")
@mcp.tool
async def update_document(doc_id: str, content: str) -> dict:
    """Update a document's content.

    @governance decorator overrides: strict PII scan, approval required.
    Controlplane policy check runs with pattern: mcp:update_document
    """
    return {"status": "updated", "doc_id": doc_id}


@mcp.tool
async def delete_document(doc_id: str) -> dict:
    """Delete a document permanently.

    Constructor tool_configs override: always blocked.
    Even if the controlplane returns action="allow", the local
    policy_action="block" takes precedence.
    """
    return {"status": "deleted", "doc_id": doc_id}


@governance(rate_limit={"max_per_minute": 5})
@mcp.tool
async def search_documents(query: str) -> list:
    """Search documents by query.

    Rate limited to 5 calls per minute via @governance decorator.
    Controlplane policy check runs with pattern: mcp:search_documents
    """
    return [{"doc_id": "1", "title": "Result", "snippet": "..."}]


if __name__ == "__main__":
    mcp.run()
```

Run the server:

```bash


python server.py
```

</details>

---

## Next Steps

- [Middleware API Reference](./middleware-reference.md) -- Complete reference for all middleware classes and parameters
- [Architecture](./architecture.md) -- How middleware fits into the broader MCP governance architecture


--------------------------------------------------------------------------------
# Middleware API Reference
URL: https://waxell.ai/docs/observe/mcp-governance/middleware-reference
Description: Complete reference for GovernanceMiddleware, ToolGovernanceConfig, MiddlewareConfig, the @governance decorator, and server-side span attributes.
--------------------------------------------------------------------------------

# Middleware API Reference

This page is a lookup reference for the server-side governance middleware API. It documents every class, parameter, and behavior in the `waxell_observe.mcp` module.

## How to Read This Reference

Each parameter table uses the following columns:

| Column | Meaning |
|--------|---------|
| **Parameter** | The constructor argument or field name |
| **Type** | Python type annotation |
| **Default** | Default value if not provided |
| **Description** | What the parameter controls |

---

## GovernanceMiddleware

```python
from waxell_observe.mcp import GovernanceMiddleware
```

Server-side governance middleware for FastMCP servers. Subclasses `fastmcp.server.middleware.Middleware` and intercepts `on_call_tool` to apply policy checks, PII scanning, rate limiting, caller identity extraction, and OTel span recording.

### Constructor Parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `api_key` | `str` | `""` | Waxell API key for controlplane policy checks. When empty, the middleware runs in standalone mode (no policy checks). |
| `api_url` | `str` | `""` | Waxell API URL (e.g., `"https://acme.waxell.dev"`). Required when `api_key` is provided. |
| `policy_config` | `dict \| None` | `None` | Server-level governance defaults. Parsed into a `MiddlewareConfig` dataclass. See [MiddlewareConfig](#middlewareconfig) for available keys. |
| `pii_scanner` | `object \| None` | `None` | Custom PII scanner implementing the `PIIScanner` protocol. If `None`, uses `RegexPIIScanner` with default regex patterns. See [Custom PII Scanner](#custom-pii-scanner). |
| `tool_configs` | `dict[str, ToolGovernanceConfig] \| None` | `None` | Per-tool governance configuration keyed by tool name. Overrides server-level defaults for specific tools. See [ToolGovernanceConfig](#toolgovernanceconfig). |

### Usage

```python
from fastmcp import FastMCP
from waxell_observe.mcp import GovernanceMiddleware, ToolGovernanceConfig

mcp = FastMCP("MyServer")

mcp.add_middleware(GovernanceMiddleware(
    api_key="wax_sk_...",
    api_url="https://acme.waxell.dev",
    policy_config={
        "default_pii_scan": "strict",
        "default_policy_action": "allow",
        "fail_open": True,
    },
    tool_configs={
        "delete_user": ToolGovernanceConfig(
            approval_required=True,
            pii_scan="strict",
            policy_action="block",
            rate_limit={"max_per_minute": 5},
        ),
    },
))
```

### Auto-Initialization

The middleware automatically calls `waxell_observe.init()` on the first tool call. This is idempotent -- calling it multiple times has no effect. You do not need to initialize the SDK separately in your server code.

---

## ToolGovernanceConfig

```python
from waxell_observe.mcp import ToolGovernanceConfig
```

Per-tool governance configuration. Applied via the `@governance` decorator, the constructor `tool_configs` dict, or derived from server-level defaults.

### Fields

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `approval_required` | `bool` | `False` | Whether the tool call requires human approval before execution. |
| `pii_scan` | `str` | `"standard"` | PII scanning mode. See [PII Scan Modes](#pii-scan-modes). |
| `policy_action` | `str \| None` | `None` | Override the controlplane's policy action for this tool. When `None`, the controlplane's decision is used. Valid values: `"allow"`, `"block"`, `"warn"`. |
| `rate_limit` | `dict \| None` | `None` | Rate limit configuration for this tool. See [Rate Limit Config](#rate-limit-config). |

### Usage

```python
from waxell_observe.mcp import ToolGovernanceConfig

config = ToolGovernanceConfig(
    approval_required=True,
    pii_scan="strict",
    policy_action="block",
    rate_limit={"max_per_minute": 10, "max_per_hour": 100},
)
```

### PII Scan Modes

| Mode | Behavior |
|------|----------|
| `"none"` | No PII scanning performed |
| `"standard"` | Scan with default `RegexPIIScanner`. All PII findings produce `warn` action (tool executes). |
| `"strict"` | Scan with `RegexPIIScanner` configured to `block` on all PII types. PII in inputs blocks the tool call. |

**PII types scanned:** SSN, credit card, email, phone, AWS access key, AWS secret key, API key, private key.

### Rate Limit Config

The `rate_limit` dict supports two keys:

| Key | Type | Description |
|-----|------|-------------|
| `max_per_minute` | `int` | Maximum tool calls allowed per 60-second sliding window |
| `max_per_hour` | `int` | Maximum tool calls allowed per 3600-second sliding window |

Both keys are optional. If neither is set, no rate limiting is applied for that tool.

```python
ToolGovernanceConfig(
    rate_limit={"max_per_minute": 10, "max_per_hour": 100},
)
```

When a rate limit is exceeded, the middleware raises `ToolError` with the message `"Rate limit exceeded for tool '{tool_name}'"`. Rate limits are enforced locally (no controlplane call needed) using a thread-safe sliding window.

---

## MiddlewareConfig

```python
from waxell_observe.mcp import MiddlewareConfig
```

Server-level governance defaults. Not passed directly to `GovernanceMiddleware` -- instead, pass a plain dict as `policy_config` and the constructor parses it into a `MiddlewareConfig` internally.

### Fields

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `default_pii_scan` | `str` | `"standard"` | Default PII scanning mode for tools without per-tool config. Valid values: `"none"`, `"standard"`, `"strict"`. |
| `default_policy_action` | `str` | `"allow"` | Default policy action for tools without per-tool config. Valid values: `"allow"`, `"block"`, `"warn"`. |
| `fail_open` | `bool` | `True` | If `True`, governance pipeline errors let tool calls proceed. If `False`, governance errors propagate as exceptions. |

### Usage via `policy_config` Dict

```python
GovernanceMiddleware(
    policy_config={
        "default_pii_scan": "strict",
        "default_policy_action": "allow",
        "fail_open": True,
    },
)
```

The keys in the `policy_config` dict map directly to `MiddlewareConfig` fields. Unrecognized keys are silently ignored.

---

## @governance Decorator

```python
from waxell_observe.mcp import governance
```

Per-tool governance decorator for FastMCP server tools. Attaches a `ToolGovernanceConfig` as metadata on the tool function. The `GovernanceMiddleware` reads this config at runtime.

### Parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `approval_required` | `bool` | `False` | Whether the tool call requires human approval before execution. |
| `pii_scan` | `str` | `"standard"` | PII scanning mode: `"none"`, `"standard"`, or `"strict"`. |
| `policy_action` | `str \| None` | `None` | Override the controlplane's policy action. Valid values: `"allow"`, `"block"`, `"warn"`. |
| `rate_limit` | `dict \| None` | `None` | Rate limit config: `{"max_per_minute": N, "max_per_hour": N}`. |

All parameters are keyword-only.

### Usage

```python
from waxell_observe.mcp import governance

@governance(approval_required=True, pii_scan="strict")
@mcp.tool
async def send_email(to: str, subject: str, body: str) -> dict:
    """Send an email to a recipient."""
    return {"status": "sent", "to": to}
```

### Decorator Order

The `@governance` decorator is order-independent -- it can be applied above or below `@mcp.tool`:

```python
# Both orderings work identically:

# Option A: @governance above @mcp.tool
@governance(pii_scan="strict")
@mcp.tool
async def tool_a(input: str) -> str: ...

# Option B: @governance below @mcp.tool
@mcp.tool
@governance(pii_scan="strict")
async def tool_b(input: str) -> str: ...
```

The decorator sets `_waxell_governance` on both the original function and its wrapper, so the middleware finds it regardless of decorator order.

---

## Config Resolution Order

When a tool call arrives, the middleware resolves governance config using the following priority (highest wins):

| Priority | Source | How to set |
|----------|--------|------------|
| 1 (highest) | `@governance` decorator | `@governance(pii_scan="strict")` on the tool function |
| 2 | Constructor `tool_configs` | `GovernanceMiddleware(tool_configs={"tool_name": ToolGovernanceConfig(...)})` |
| 3 (lowest) | Server-level defaults | `GovernanceMiddleware(policy_config={"default_pii_scan": "standard"})` |

If no config is found at any level, the built-in defaults apply: `pii_scan="standard"`, `policy_action=None` (use controlplane decision), `approval_required=False`, `rate_limit=None`.

### Resolution Example

```python
mcp.add_middleware(GovernanceMiddleware(
    policy_config={"default_pii_scan": "standard"},  # Priority 3
    tool_configs={
        "send_email": ToolGovernanceConfig(pii_scan="strict"),  # Priority 2
    },
))

@governance(pii_scan="none")  # Priority 1 -- wins for this tool
@mcp.tool
async def send_email(to: str, body: str) -> dict: ...

@mcp.tool
async def read_file(path: str) -> str: ...
# send_email uses pii_scan="none" (decorator, priority 1)
# read_file uses pii_scan="standard" (server default, priority 3)
```

---

## Custom PII Scanner

You can replace the default `RegexPIIScanner` with a custom scanner by implementing the `PIIScanner` protocol.

### PIIScanner Protocol

```python
from waxell_observe.scanning import PIIScanner
```

| Method | Signature | Returns |
|--------|-----------|---------|
| `scan` | `scan(self, text: str) -> dict` | `{"detected": bool, "count": int, "findings": list[dict]}` |

The protocol is `@runtime_checkable`, so you can use `isinstance()` to verify compliance.

Each finding dict in the `findings` list should contain:

| Key | Type | Description |
|-----|------|-------------|
| `type` | `str` | PII type identifier (e.g., `"ssn"`, `"email"`, `"credit_card"`) |
| `category` | `str` | Category: `"pii"` or `"credential"` |
| `action` | `str` | Action to take: `"warn"`, `"redact"`, or `"block"` |

### Custom Scanner Example

```python
from waxell_observe.scanning import PIIScanner


class MyPIIScanner:
    """Custom PII scanner using an external service."""

    def scan(self, text: str) -> dict:
        # Call your PII detection service
        findings = my_pii_service.detect(text)
        return {
            "detected": len(findings) > 0,
            "count": len(findings),
            "findings": [
                {"type": f.type, "category": "pii", "action": "warn"}
                for f in findings
            ],
        }


# Pass to GovernanceMiddleware
mcp.add_middleware(GovernanceMiddleware(
    pii_scanner=MyPIIScanner(),
))
```

### Default Scanner: RegexPIIScanner

```python
from waxell_observe.scanning import RegexPIIScanner
```

The default scanner uses regex patterns to detect PII and credentials. You can customize the action per PII type:

```python
scanner = RegexPIIScanner(actions={
    "ssn": "block",
    "credit_card": "block",
    "email": "warn",
    "phone": "warn",
    "aws_access_key": "block",
    "aws_secret_key": "block",
    "api_key": "block",
    "private_key": "block",
})

mcp.add_middleware(GovernanceMiddleware(pii_scanner=scanner))
```

When no `actions` dict is provided, all PII types default to `"warn"`.

---

## Governance Pipeline (`on_call_tool`)

The `GovernanceMiddleware.on_call_tool` method processes every incoming tool call through a 12-section pipeline. Understanding this pipeline helps you predict how governance decisions are made and what span attributes are set.

### Pipeline Sections

| Section | Name | What happens | Can block? |
|---------|------|-------------|-----------|
| 1 | Extract tool info | Read `tool_name` and `arguments` from `context.message` | No |
| 2 | OTel initialization | Auto-call `waxell_observe.init()` (idempotent) | No |
| 3 | Create span | Start a `SpanKind.SERVER` span with independent root trace (no client propagation) | No |
| 4 | Caller identity | Extract `client_id`, `session_id`, IP, user-agent from FastMCP context and HTTP headers | No |
| 5 | Get tool config | Resolve per-tool config via [resolution order](#config-resolution-order) | No |
| 6 | Rate limit check | Check sliding window rate limits. Raises `ToolError` if exceeded. | **Yes** |
| 7 | Policy check | Send `mcp:{tool_name}` to controlplane. Apply action (block/warn/throttle). Skip if no controlplane. | **Yes** |
| 8 | PII scan (inputs) | Scan tool arguments for PII. Block if strict mode finds PII. | **Yes** |
| 9 | Execute tool | Call `call_next(context)` to run the actual tool | No |
| 10 | PII scan (outputs) | Scan tool result for PII. Warn only (tool already executed). | No |
| 11 | Audit trail | Record `governance_checked`, `governance_timestamp`, `params_hash` on span | No |
| 12 | Return result | Return the tool result to the caller | No |

### Fail-Open Wrapper

The entire pipeline (sections 4-11) is wrapped in a try/except. If `fail_open=True` (default) and any governance section raises an unexpected exception, the middleware:

1. Logs a warning
2. Sets `policy_skipped=True` and `governance_checked=False` on the span
3. Calls `call_next(context)` to execute the tool without governance

Intentional blocks (`ToolError` from rate limits, policy blocks, or PII blocks) always propagate, regardless of `fail_open`.

---

## Server-Side Span Attributes

These attributes are set by `GovernanceMiddleware` on server-side spans. They use the same `waxell.mcp.*` namespace as the client-side auto-instrumentor for unified querying.

### Core Middleware Attributes

| Attribute | Type | Description | Example |
|-----------|------|-------------|---------|
| `waxell.mcp.middleware_active` | `bool` | Whether governance middleware is processing this call | `true` |
| `waxell.mcp.server_side` | `bool` | Whether this span was emitted by server-side middleware | `true` |
| `waxell.mcp.method_name` | `string` | The MCP method being called | `"tools/call"` |

### Caller Identity Attributes

| Attribute | Type | Description | Example |
|-----------|------|-------------|---------|
| `waxell.mcp.caller_client_id` | `string` | MCP client ID from the `initialize` handshake | `"claude-desktop-1.4"` |
| `waxell.mcp.caller_session_id` | `string` | MCP session ID for the caller's connection | `"sess_x7y8z9"` |
| `waxell.mcp.caller_ip` | `string` | Caller IP from HTTP headers (HTTP/SSE transports only) | `"192.168.1.42"` |
| `waxell.mcp.caller_user_agent` | `string` | Caller user-agent header (HTTP/SSE transports only) | `"mcp-client/1.0"` |

**Transport availability**
`caller_ip` and `caller_user_agent` are only available on HTTP and SSE transports. Stdio transport connections do not have HTTP headers.


### Rate Limiting Attributes

| Attribute | Type | Description | Example |
|-----------|------|-------------|---------|
| `waxell.mcp.rate_limited` | `bool` | Whether the call was rejected due to rate limiting | `true` |
| `waxell.mcp.rate_limit_remaining` | `int` | Remaining calls allowed in the current window | `7` |

### Governance Decision Attributes

| Attribute | Type | Description | Example |
|-----------|------|-------------|---------|
| `waxell.mcp.governance_checked` | `bool` | Whether governance processing completed for this call | `true` |
| `waxell.mcp.governance_action` | `string` | The final policy action applied | `"allow"`, `"block"`, `"warn"` |
| `waxell.mcp.governance_timestamp` | `string` | ISO 8601 timestamp of governance processing | `"2026-03-05T14:32:01.456Z"` |
| `waxell.mcp.policy_skipped` | `bool` | Whether the policy check was skipped (controlplane unreachable, fail-open) | `false` |
| `waxell.mcp.policy_reason` | `string` | Human-readable policy decision explanation | `"Tool blocked by policy"` |
| `waxell.mcp.throttle_delay_ms` | `int` | Enforced delay in milliseconds when action is `throttle` (max 30000) | `5000` |

### PII Scan Attributes

| Attribute | Type | Description | Example |
|-----------|------|-------------|---------|
| `waxell.mcp.pii_detected` | `bool` | Whether PII was found in inputs or outputs | `true` |
| `waxell.mcp.pii_count` | `int` | Total PII findings across inputs and outputs | `3` |
| `waxell.mcp.pii_scan_direction` | `string` | What was scanned: `"inputs"`, `"outputs"`, or `"both"` | `"inputs"` |
| `waxell.mcp.pii_action_taken` | `string` | Highest-severity action taken for PII findings | `"warn"`, `"block"` |

### Audit Trail Attributes

| Attribute | Type | Description | Example |
|-----------|------|-------------|---------|
| `waxell.mcp.params_hash` | `string` | SHA256 hash of tool arguments (truncated to 16 hex chars) | `"a3f2b8c1e9d04f67"` |

### Querying Server-Side Spans

Server-side spans can be distinguished from client-side spans using `server_side`:

```
# All server-side governance spans
{ span.waxell.mcp.server_side = true }

# Server-side spans that rate-limited a tool
{ span.waxell.mcp.server_side = true && span.waxell.mcp.rate_limited = true }

# Server-side spans where governance was skipped (fail-open)
{ span.waxell.mcp.server_side = true && span.waxell.mcp.policy_skipped = true }
```

---

## Import Reference

All public middleware classes and functions are available from `waxell_observe.mcp`:

```python
from waxell_observe.mcp import (
    GovernanceMiddleware,    # Main middleware class
    governance,              # Per-tool governance decorator
    ToolGovernanceConfig,    # Per-tool config dataclass
    MiddlewareConfig,        # Server-level config dataclass
)
```

PII scanning classes are in `waxell_observe.scanning`:

```python
from waxell_observe.scanning import (
    PIIScanner,              # Protocol for custom scanners
    RegexPIIScanner,         # Default regex-based scanner
    scan_text_for_pii,       # Standalone PII scanning function
)
```

### Install Extras

| Extra | Command | What it installs |
|-------|---------|-----------------|
| `mcp` (client) | `pip install waxell-observe[mcp]` | `mcp>=1.25,<2` |
| `mcp-server` | `pip install waxell-observe[mcp-server]` | `fastmcp>=3.0,<4`, `pyyaml>=6.0` |
| Both | `pip install waxell-observe[mcp,mcp-server]` | All MCP dependencies |

**mcp-server is not included in [all]**
The `mcp-server` extra is opt-in for server builders. Running `pip install waxell-observe[all]` does **not** install FastMCP. You must explicitly install `waxell-observe[mcp-server]`.


---

## See Also

- [Connecting to Controlplane](./middleware-controlplane.md) -- Fail-open semantics and policy check flow
- [Span Attributes Reference](./reference.md) -- Complete attribute reference including client-side attributes
- [API Reference](./api-reference.md) -- Cross-product API reference covering middleware, proxy, and scanning


--------------------------------------------------------------------------------
# Governance Proxy Quickstart
URL: https://waxell.ai/docs/observe/mcp-governance/proxy-quickstart
Description: Wrap any third-party MCP server with governance in one command -- policy checks, PII scanning, and observability with zero code changes.
--------------------------------------------------------------------------------

# Governance Proxy Quickstart

The governance proxy wraps **any** MCP server -- including third-party servers you don't control -- with policy checks, PII scanning, rate limiting, and full observability. Your agent connects to the proxy instead of the target server. No code changes to the agent or the server.

In this quickstart, you'll wrap a local MCP filesystem server with the proxy and see governed traces in your dashboard -- in under 5 minutes.

By the end, you'll have:
- A governance proxy running between your agent and an MCP server
- Automatic tracing of every proxied tool call with governance attributes
- A config file for per-tool policies

## Prerequisites

- **Python 3.10+**
- **A Waxell API key** -- get one from your [Waxell dashboard](https://waxell.dev)
- **Node.js 18+** -- for the MCP filesystem server used in examples

## Step 1: Install

Install the `waxell` meta-package (which registers the `wax` CLI) along with `waxell-observe`'s MCP server extra:

```bash
pip install waxell "waxell-observe[mcp-server]"
```

`waxell` provides the `wax` CLI (and pulls in `waxell-runtime` + `waxell-observe` transitively). The `[mcp-server]` extra on `waxell-observe` adds `fastmcp>=3.0,<4` (MCP server framework) and `pyyaml>=6.0` (config file support). For an isolated CLI install, `pipx install waxell` also works — but you'll still need `pip install "waxell-observe[mcp-server]"` in the same env for the proxy bits. See [CLI reference](/reference/cli#installation) for Windows PATH issues and troubleshooting.

**Already have waxell-observe?**
If you already have `waxell-observe` installed, add the server extra and the meta package:
```bash
pip install waxell "waxell-observe[mcp-server]"
```


## Step 2: One-Liner Proxy

The fastest way to start is the `--target` flag. This wraps a target server with governance in a single command:

```bash
wax observe mcp-proxy --target "npx -y @modelcontextprotocol/server-filesystem /tmp/mcp-test"
```

That's it. The proxy:
1. Starts the MCP filesystem server as a child process
2. Wraps it with governance middleware (PII scanning, policy checks, rate limiting)
3. Adds security middleware (tool fingerprinting, rug pull detection)
4. Listens on stdio for incoming MCP client connections

Your agent connects to the proxy exactly as it would connect to the target server -- no code changes needed.

**What's happening under the hood**
The `--target` flag creates a `ProxyConfig` with the target as a local path and all defaults enabled: standard PII scanning, fail-open governance, tool fingerprinting, and rug pull detection. The proxy appears as a normal MCP server to your agent.


### Connect your Waxell API key

To get governed traces flowing to your dashboard, pass your API key:

```bash


wax observe mcp-proxy --target "npx -y @modelcontextprotocol/server-filesystem /tmp/mcp-test"
```

The CLI reads `WAXELL_API_KEY` and `WAXELL_API_URL` from environment variables automatically. You can also pass them as flags:

```bash
wax observe mcp-proxy \
  --target "npx -y @modelcontextprotocol/server-filesystem /tmp/mcp-test" \
  --api-key wax_sk_... \
  --api-url https://acme.waxell.dev
```

## Step 3: See Your Traces

After running the proxy and making tool calls through it, open your [Waxell dashboard](https://waxell.dev) and navigate to the Traces view. You'll see server-side spans for each proxied tool call:

| Attribute | Example Value | Description |
|-----------|--------------|-------------|
| **Span name** | `tools/call waxell-proxy:read_file` | Format: `tools/call {proxy-name}:{tool}` |
| `waxell.mcp.server_side` | `true` | Indicates a server-side span |
| `waxell.mcp.middleware_active` | `true` | Governance middleware processed this call |
| `waxell.mcp.governance_checked` | `true` | Governance was active |
| `waxell.mcp.governance_action` | `allow` | Policy decision |
| `waxell.mcp.pii_detected` | `false` | Whether PII was found |
| `waxell.mcp.params_hash` | `a1b2c3d4e5f67890` | SHA256 fingerprint of the arguments |

**Proxy traces are independent**
The proxy creates its own root traces (server-side, `SpanKind.SERVER`). These are separate from any client-side traces your agent creates. This means you get two views: the agent's perspective and the proxy's perspective.


## Step 4: Add a Config File

The one-liner is great for getting started, but real deployments need per-tool policies. Create a config file to customize governance per tool.

Create `proxy.yaml`:

```yaml
# Target: local MCP filesystem server
target:
  command: "npx"
  args: ["-y", "@modelcontextprotocol/server-filesystem", "/tmp/mcp-test"]

# Proxy identity
name: "fs-governance-proxy"

# Governance defaults
api_key: "${WAXELL_API_KEY}"
api_url: "${WAXELL_API_URL}"
default_pii_scan: standard
default_policy_action: allow
fail_open: true

# Per-tool policies
tools:
  write_file:
    pii_scan: strict
    approval_required: true
  create_directory:
    policy_action: warn
```

Run the proxy with your config file:

```bash
wax observe mcp-proxy --config proxy.yaml
```

Now `write_file` calls get strict PII scanning and require approval, while `create_directory` calls emit warnings. All other tools use the server defaults (standard PII scanning, allow policy).

**Environment variable substitution**
Config files support `${VAR_NAME}` syntax for environment variables. The proxy resolves these before parsing the YAML/JSON. This means you can keep secrets out of your config files.


### Three Target Formats

The `target` field in your config file supports three formats depending on how you connect to the backend server:

| Format | When to Use | Example |
|--------|------------|---------|
| **URL** | Remote HTTP MCP server | `target: { url: "http://remote-server:9000/mcp" }` |
| **Command** | Local process via stdio | `target: { command: "npx", args: ["-y", "@modelcontextprotocol/server-filesystem", "/tmp"] }` |
| **Path** | Local Python/JS script | `target: { path: "./my_server.py" }` |

The `--target` CLI flag auto-detects the format: URLs starting with `http://` or `https://` use the URL format, everything else uses the path format.

## What the Proxy Does

When your agent makes a tool call through the proxy, the governance pipeline runs:

```mermaid
flowchart TD
    A["Agent calls tool"] --> B["Proxy receives call"]
    B --> C["Rate limit check"]
    C --> D["Policy check<br/><small>(controlplane query)</small>"]
    D --> E["PII scan on inputs"]
    E --> F{"Blocked?"}
    F -->|No| G["Forward to target server"]
    F -->|Yes| H["Return error to agent"]
    G --> I["PII scan on outputs<br/><small>(warn only)</small>"]
    I --> J["Record audit trail"]
    J --> K["Return result to agent"]

    style B fill:#7c3aed,color:#fff
    style D fill:#2563eb,color:#fff
    style E fill:#dc2626,color:#fff
    style G fill:#1a7f5a,color:#fff
```

The proxy also runs **security middleware** on `tools/list` calls to capture tool fingerprints and detect rug pulls (tool definition changes between sessions).

### Governance is fail-open

If the Waxell controlplane is unreachable or the PII scanner throws an error, the tool call proceeds normally. Governance errors never break your agent's workflow. This fail-open behavior is logged so you can monitor governance health.

<details>
<summary>Full proxy.yaml with all options</summary>

```yaml
# Target server (required)
target:
  url: "http://remote-server:9000/mcp"
  # OR:
  # command: "npx"
  # args: ["-y", "@modelcontextprotocol/server-filesystem", "/tmp"]
  # env:
  #   NODE_ENV: "production"
  # OR:
  # path: "./my_server.py"

# Proxy server settings
name: "my-governance-proxy"
transport: stdio          # "stdio" or "http"
host: "0.0.0.0"           # HTTP bind host (only used when transport=http)
port: 8080                # HTTP bind port (only used when transport=http)

# Governance
api_key: "${WAXELL_API_KEY}"
api_url: "${WAXELL_API_URL}"
default_pii_scan: standard       # "none", "standard", "strict"
default_policy_action: allow     # "allow", "block", "warn"
fail_open: true

# Per-tool policies
tools:
  send_email:
    pii_scan: strict
    approval_required: true
    rate_limit:
      max_per_minute: 10
      max_per_hour: 100
  delete_records:
    policy_action: block
  read_file:
    pii_scan: none            # skip PII scanning for read-only tools

# Security
fingerprinting: true          # capture tool fingerprints on first tools/list
rug_pull_detection: true      # alert on tool definition changes
```

</details>

## Next Steps

- **[Proxy Deployment](./proxy-deployment)** -- CLI reference, Docker, docker-compose, and production configuration
- **[Governing Third-Party Providers](./proxy-third-party)** -- Wrap Composio, Pipedream, and other remote MCP servers
- **[Architecture](./architecture)** -- How the proxy fits with the auto-instrumentor and server middleware


--------------------------------------------------------------------------------
# Proxy Deployment Guide
URL: https://waxell.ai/docs/observe/mcp-governance/proxy-deployment
Description: Deploy the Waxell governance proxy with CLI, Docker, or docker-compose -- complete CLI reference, config file formats, and production considerations.
--------------------------------------------------------------------------------

# Proxy Deployment Guide

This guide covers everything you need to deploy the governance proxy in development and production: the complete CLI reference, YAML and JSON config file formats, Docker packaging, and docker-compose orchestration.

If you haven't used the proxy yet, start with the [Proxy Quickstart](./proxy-quickstart).

## Prerequisites

- **Python 3.10+**
- **`pip install waxell-observe[mcp-server]`** -- installs the proxy, FastMCP, and PyYAML
- **`pip install waxell`** (or `pipx install waxell` for an isolated install) -- provides the `wax` CLI used throughout this guide. See [CLI reference](/reference/cli#installation) for Windows PATH issues and troubleshooting.
- **A Waxell API key** (optional but recommended) -- for controlplane policy checks

## CLI Reference

The proxy runs via the `wax` CLI:

```bash
wax observe mcp-proxy [OPTIONS]
```

### Options

| Option | Short | Default | Description |
|--------|-------|---------|-------------|
| `--config` | `-c` | -- | Path to a YAML or JSON config file |
| `--target` | `-t` | -- | Target server URL or path (quick-start mode) |
| `--transport` | | `stdio` | Inbound transport: `stdio` or `http` |
| `--port` | | `8080` | Port for HTTP transport |
| `--host` | | `0.0.0.0` | Host for HTTP transport |
| `--api-key` | | `$WAXELL_API_KEY` | Waxell API key for controlplane policy checks |
| `--api-url` | | `$WAXELL_API_URL` | Waxell API URL |
| `--name` | | `waxell-proxy` | Proxy server name (used in spans and metadata) |
| `--no-fingerprint` | | | Disable tool fingerprinting |
| `--no-rug-pull-detection` | | | Disable rug pull detection |

Either `--config` or `--target` is required. If both are provided, `--config` takes precedence.

### Usage Examples

```bash
# Quick-start: wrap a remote HTTP server
wax observe mcp-proxy --target http://remote-server:9000/mcp

# Quick-start: wrap a local script
wax observe mcp-proxy --target ./my_server.py

# Config file mode
wax observe mcp-proxy --config proxy.yaml

# HTTP transport with custom port
wax observe mcp-proxy --config proxy.yaml --transport http --port 9090

# Override API key and disable security features
wax observe mcp-proxy --config proxy.yaml \
  --api-key wax_sk_... \
  --no-fingerprint \
  --no-rug-pull-detection

# Name the proxy for easier trace identification
wax observe mcp-proxy --target http://composio.example.com/mcp --name composio-proxy
```

### CLI Flag Overrides

When using `--config`, CLI flags override the corresponding config file values. This lets you use a shared config file across environments while varying settings per deployment:

```bash
# Use production config, override transport and port for local testing
wax observe mcp-proxy --config proxy.yaml --transport http --port 3001
```

## Config File Formats

The proxy accepts YAML (`.yaml`, `.yml`) or JSON (`.json`) config files. Both formats support environment variable substitution.

### YAML Config

```yaml
# proxy.yaml
target:
  url: "http://remote-server:9000/mcp"

name: "acme-governance-proxy"
transport: stdio

# Governance
api_key: "${WAXELL_API_KEY}"
api_url: "${WAXELL_API_URL}"
default_pii_scan: standard
default_policy_action: allow
fail_open: true

# Per-tool policies
tools:
  send_email:
    pii_scan: strict
    approval_required: true
    rate_limit:
      max_per_minute: 10
      max_per_hour: 100
  delete_records:
    policy_action: block

# Security
fingerprinting: true
rug_pull_detection: true
```

### JSON Config

```json
{
  "target": {
    "url": "http://remote-server:9000/mcp"
  },
  "name": "acme-governance-proxy",
  "transport": "stdio",
  "api_key": "${WAXELL_API_KEY}",
  "api_url": "${WAXELL_API_URL}",
  "default_pii_scan": "standard",
  "default_policy_action": "allow",
  "fail_open": true,
  "tools": {
    "send_email": {
      "pii_scan": "strict",
      "approval_required": true,
      "rate_limit": {
        "max_per_minute": 10,
        "max_per_hour": 100
      }
    },
    "delete_records": {
      "policy_action": "block"
    }
  },
  "fingerprinting": true,
  "rug_pull_detection": true
}
```

**YAML or JSON?**
YAML is recommended for human-edited configs (cleaner syntax, supports comments). JSON is better for machine-generated configs or when you want to avoid the `pyyaml` dependency.

JSON configs do not require `pyyaml` -- only the `json` stdlib module is used.


### Environment Variable Substitution

Config files support `${VAR_NAME}` syntax for environment variable references. Variables are resolved via `os.path.expandvars()` before parsing, so any environment variable available to the process can be referenced:

```yaml
# These are resolved at load time
api_key: "${WAXELL_API_KEY}"
api_url: "${WAXELL_API_URL}"

target:
  url: "${MCP_TARGET_URL}"

name: "${PROXY_NAME:-waxell-proxy}"  # shell default syntax works too
```

**Unresolved variables**
If an environment variable is not set, `${VAR_NAME}` is left as the literal string `$VAR_NAME` (without braces). This will likely cause a validation error when the proxy starts. Always ensure referenced variables are set before starting the proxy.


### Target Formats

The `target` field supports three connection formats:

#### HTTP Target (remote server)

```yaml
target:
  url: "http://remote-server:9000/mcp"
```

Use this for remote MCP servers that expose an HTTP endpoint (streamable HTTP or SSE).

#### Command Target (stdio subprocess)

```yaml
target:
  command: "npx"
  args: ["-y", "@modelcontextprotocol/server-filesystem", "/tmp/allowed"]
  env:
    NODE_ENV: "production"
```

Use this for MCP servers that communicate over stdio. The proxy spawns the command as a child process. The optional `env` dict sets additional environment variables for the subprocess.

#### Path Target (local script)

```yaml
target:
  path: "./my_server.py"
```

Use this for local Python or JavaScript MCP server scripts. FastMCP auto-detects the runtime from the file extension.

## Docker Deployment

Package the proxy as a Docker image for consistent, reproducible deployments.

### Dockerfile

```dockerfile
FROM python:3.12-slim

WORKDIR /app

# Install the proxy with MCP server support
RUN pip install --no-cache-dir "waxell-observe[mcp-server]"

# Copy config file
COPY proxy.yaml .

# The proxy runs on stdio by default.
# Override with --transport http for networked deployments.
ENTRYPOINT ["wax", "observe", "mcp-proxy"]
CMD ["--config", "proxy.yaml", "--transport", "http", "--port", "8080"]
```

Build and run:

```bash
docker build -t waxell-proxy .

docker run -p 8080:8080 \
  -e WAXELL_API_KEY="wax_sk_..." \
  -e WAXELL_API_URL="https://acme.waxell.dev" \
  waxell-proxy
```

**Transport choice in Docker**
Use `--transport http` when running in Docker so that other containers (or external clients) can reach the proxy over the network. The `stdio` transport only works when the proxy and client share a process boundary.


### Multi-Stage Build

For smaller images, use a multi-stage build:

```dockerfile
FROM python:3.12-slim AS builder

RUN pip install --no-cache-dir "waxell-observe[mcp-server]"

FROM python:3.12-slim

COPY --from=builder /usr/local/lib/python3.12/site-packages /usr/local/lib/python3.12/site-packages
COPY --from=builder /usr/local/bin/wax /usr/local/bin/wax

WORKDIR /app
COPY proxy.yaml .

ENTRYPOINT ["wax", "observe", "mcp-proxy"]
CMD ["--config", "proxy.yaml", "--transport", "http", "--port", "8080"]
```

## docker-compose Deployment

Use docker-compose to run the proxy alongside the target MCP server. This is the recommended approach for local development and testing.

### Proxy + HTTP Target

```yaml
# docker-compose.yaml
services:
  # The target MCP server
  mcp-server:
    image: your-mcp-server:latest
    ports:
      - "9000:9000"

  # The governance proxy
  proxy:
    build: .
    ports:
      - "8080:8080"
    environment:
      - WAXELL_API_KEY=${WAXELL_API_KEY}
      - WAXELL_API_URL=${WAXELL_API_URL}
      - MCP_TARGET_URL=http://mcp-server:9000/mcp
    depends_on:
      - mcp-server
```

With a `proxy.yaml` that references the target via environment variable:

```yaml
target:
  url: "${MCP_TARGET_URL}"

name: "governed-proxy"
transport: http
port: 8080

api_key: "${WAXELL_API_KEY}"
api_url: "${WAXELL_API_URL}"
default_pii_scan: standard
fail_open: true
```

Start both services:

```bash
docker-compose up
```

**Docker networking**
In docker-compose, services communicate using their service names as hostnames. The proxy reaches the target at `http://mcp-server:9000/mcp` because both services are on the same default network. No special networking configuration is needed.


### Proxy + Stdio Target (sidecar pattern)

For MCP servers that use stdio transport, run the server command directly inside the proxy container:

```yaml
# docker-compose.yaml
services:
  proxy:
    build:
      context: .
      dockerfile: Dockerfile.stdio
    ports:
      - "8080:8080"
    environment:
      - WAXELL_API_KEY=${WAXELL_API_KEY}
      - WAXELL_API_URL=${WAXELL_API_URL}
```

With a specialized Dockerfile that includes Node.js for the MCP filesystem server:

```dockerfile
# Dockerfile.stdio
FROM python:3.12-slim

# Install Node.js for npx-based MCP servers
RUN apt-get update && apt-get install -y --no-install-recommends nodejs npm && rm -rf /var/lib/apt/lists/*

WORKDIR /app
RUN pip install --no-cache-dir "waxell-observe[mcp-server]"

COPY proxy-stdio.yaml proxy.yaml

ENTRYPOINT ["wax", "observe", "mcp-proxy"]
CMD ["--config", "proxy.yaml", "--transport", "http", "--port", "8080"]
```

```yaml
# proxy-stdio.yaml
target:
  command: "npx"
  args: ["-y", "@modelcontextprotocol/server-filesystem", "/data"]

name: "fs-proxy"
transport: http
port: 8080

api_key: "${WAXELL_API_KEY}"
api_url: "${WAXELL_API_URL}"
```

## HTTP Transport Configuration

When using `--transport http`, the proxy exposes an HTTP endpoint that MCP clients can connect to using streamable HTTP transport.

```bash
# Start proxy with HTTP transport
wax observe mcp-proxy --config proxy.yaml --transport http --host 0.0.0.0 --port 8080
```

Clients connect to `http://<host>:<port>/mcp`:

```python
from mcp.client.streamable_http import streamable_http_client

async with streamable_http_client("http://localhost:8080/mcp") as (read, write, _):
    async with ClientSession(read, write) as session:
        await session.initialize()
        result = await session.call_tool(name="read_file", arguments={"path": "/tmp/test.txt"})
```

### When to Use HTTP Transport

| Scenario | Transport | Why |
|----------|-----------|-----|
| Agent and proxy in same process | `stdio` | Simplest, no networking |
| Agent and proxy in different containers | `http` | Required for network communication |
| Multiple agents sharing one proxy | `http` | Allows concurrent connections |
| Production deployment | `http` | Standard for networked services |

## ProxyConfig Reference

Complete reference for all `ProxyConfig` fields:

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `target` | `dict` | *(required)* | Target server spec: `{url}`, `{command, args, env}`, or `{path}` |
| `name` | `str` | `"waxell-proxy"` | Proxy name for MCP metadata and span attribution |
| `transport` | `str` | `"stdio"` | Inbound transport: `"stdio"` or `"http"` |
| `host` | `str` | `"0.0.0.0"` | HTTP bind host |
| `port` | `int` | `8080` | HTTP bind port |
| `api_key` | `str \| None` | `None` | Waxell API key for controlplane |
| `api_url` | `str \| None` | `None` | Waxell API URL |
| `default_pii_scan` | `str` | `"standard"` | Default PII scan mode: `"none"`, `"standard"`, `"strict"` |
| `default_policy_action` | `str` | `"allow"` | Default policy action: `"allow"`, `"block"`, `"warn"` |
| `fail_open` | `bool` | `true` | Governance errors let tool calls proceed |
| `tools` | `dict[str, ToolPolicyConfig]` | `{}` | Per-tool governance policies |
| `fingerprinting` | `bool` | `true` | Capture tool fingerprints on first `tools/list` |
| `rug_pull_detection` | `bool` | `true` | Alert on tool definition changes between sessions |

### ToolPolicyConfig Reference

Per-tool policy overrides in the `tools` section:

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `pii_scan` | `str` | `"standard"` | PII scan mode: `"none"`, `"standard"`, `"strict"` |
| `policy_action` | `str \| None` | `None` | Override policy action: `"allow"`, `"block"`, `"warn"` |
| `rate_limit` | `dict \| None` | `None` | Rate limit: `{max_per_minute, max_per_hour}` |
| `approval_required` | `bool` | `false` | Require approval before execution |

### PII Scan Modes

| Mode | Behavior |
|------|----------|
| `"none"` | No PII scanning |
| `"standard"` | Warn on all PII types (SSN, credit card, email, phone, AWS keys, API keys, private keys) |
| `"strict"` | Block on all PII types -- tool call is rejected if PII is found in inputs |

Output PII scanning always warns (never blocks) because the tool has already executed.

## Production Considerations

### Logging

The proxy logs to stderr via the `waxell` logger. Configure log level with the `WAXELL_LOG_LEVEL` environment variable:

```bash
WAXELL_LOG_LEVEL=DEBUG wax observe mcp-proxy --config proxy.yaml
```

When Rich is installed (included with `waxell-observe`), the CLI displays formatted startup information on stderr:

```
waxell-proxy starting
  Target:     proxy.yaml
  Transport:  stdio
  PII scan:   standard
  Fail-open:  true
  Security:   fingerprinting, rug-pull-detection, controlplane-policy
```

### Fail-Open Behavior

By default, `fail_open: true` means governance errors (controlplane unreachable, PII scanner crash, policy check timeout) let tool calls proceed normally. This ensures the proxy never breaks your agent's workflow.

Set `fail_open: false` for strict governance enforcement where you'd rather fail than allow an ungoverned call:

```yaml
fail_open: false  # tool calls fail if governance can't run
```

**Strict mode in production**
With `fail_open: false`, a controlplane outage will block all tool calls through the proxy. Only use this in environments where governance is a hard requirement (compliance-critical systems).


### Security Features

The proxy includes two security features enabled by default:

**Tool Fingerprinting** (`fingerprinting: true`): On the first `tools/list` call, the proxy captures a SHA256 hash of each tool's definition (name + description + input schema + output schema). This fingerprint is recorded on spans for audit purposes.

**Rug Pull Detection** (`rug_pull_detection: true`): On subsequent `tools/list` calls, the proxy compares tool definitions against the stored fingerprints. If a tool's description or schema has changed, it records an alert on the span with the diff. This detects when a third-party server modifies tool behavior between sessions.

Disable either feature if they add unwanted overhead:

```bash
wax observe mcp-proxy --config proxy.yaml --no-fingerprint --no-rug-pull-detection
```

### Health Monitoring

Monitor the proxy by watching for:
- **Span attributes**: `waxell.mcp.governance_checked` should be `true` on all proxied calls
- **Error spans**: `waxell.mcp.is_error` indicates tool-level errors
- **Rate limit spans**: `waxell.mcp.rate_limited` indicates rate limit enforcement
- **PII detection**: `waxell.mcp.pii_detected` indicates PII was found in inputs or outputs
- **Policy blocks**: `waxell.mcp.governance_action` = `"block"` indicates a policy blocked the call

### Programmatic Usage

You can also create and run the proxy programmatically:

```python
from waxell_observe.mcp_proxy import ProxyServer, ProxyConfig, create_proxy_server

# From config file
server = create_proxy_server(config_path="proxy.yaml")
server.run()

# Quick target mode
server = create_proxy_server(
    target="http://remote-server:9000/mcp",
    api_key="wax_sk_...",
)
server.run(transport="http")

# Full control
config = ProxyConfig(
    target={"url": "http://remote-server:9000/mcp"},
    name="my-proxy",
    transport="http",
    port=9090,
    api_key="wax_sk_...",
    default_pii_scan="strict",
    tools={
        "send_email": {"pii_scan": "strict", "approval_required": True},
    },
)
proxy = ProxyServer(config)
proxy.run()
```

<details>
<summary>Async usage</summary>

```python

from waxell_observe.mcp_proxy import ProxyServer, ProxyConfig

config = ProxyConfig(
    target={"url": "http://remote-server:9000/mcp"},
    name="async-proxy",
    transport="http",
    port=8080,
)

proxy = ProxyServer(config)

async def main():
    await proxy.run_async()

asyncio.run(main())
```

</details>

## Next Steps

- **[Governing Third-Party Providers](./proxy-third-party)** -- Detailed examples for Composio, Pipedream, and public MCP servers
- **[Architecture](./architecture)** -- How the proxy fits with the auto-instrumentor and server middleware


--------------------------------------------------------------------------------
# Governing Third-Party MCP Providers
URL: https://waxell.ai/docs/observe/mcp-governance/proxy-third-party
Description: Use the Waxell governance proxy to add policy checks, PII scanning, and rug pull detection to third-party MCP providers like Composio, filesystem servers, and custom scripts.
--------------------------------------------------------------------------------

# Governing Third-Party MCP Providers

Third-party MCP servers -- Composio, Pipedream, public MCP servers, internal team servers -- expose tools that your agents call but that you don't control. The governance proxy sits between your agent and these servers, applying policy checks, PII scanning, and security monitoring without modifying either the agent or the server.

This guide shows how to configure the proxy for specific third-party provider patterns, set up per-tool policies for tools you discover at runtime, and use security features to detect when a provider changes its tools.

## Prerequisites

- **Python 3.10+**
- **waxell-observe with the mcp-server extra:**

```bash
pip install waxell-observe[mcp-server]
```

This installs `fastmcp>=3.0,<4` and `pyyaml>=6.0` alongside `waxell-observe`.

- **The wax CLI.** If you don't already have it: `pip install waxell` (or `pipx install waxell` for an isolated install). `pip install waxell-observe` does not install the `wax` command -- only the meta `waxell` package registers it. See [CLI reference](/reference/cli#installation) for Windows PATH issues and troubleshooting.

- **Familiarity with the proxy** -- read the [Proxy Quickstart](./proxy-quickstart) first if you haven't deployed a proxy yet.

## Target Types

The proxy supports three target formats, each suited to a different kind of third-party server.

### HTTP Target (Remote Servers)

Use `url` for third-party servers that expose an HTTP or SSE MCP endpoint. This is the most common pattern for hosted providers like Composio.

```yaml title="proxy-composio.yaml"
target:
  url: "https://mcp.composio.dev/composio/mcp"

name: "composio-proxy"
api_key: "${WAXELL_API_KEY}"
api_url: "${WAXELL_API_URL}"
default_pii_scan: standard
```

Run the proxy:

```bash
wax observe mcp-proxy --config proxy-composio.yaml
```

The proxy connects to the Composio MCP endpoint as its target and serves your agent via stdio (the default inbound transport). Every tool call passes through governance checks before reaching Composio.

**Environment variable substitution**
`${WAXELL_API_KEY}` and `${WAXELL_API_URL}` are resolved from your shell environment via `os.path.expandvars()` when the config file is loaded. This works in both YAML and JSON config files.


### stdio Target (Local Servers)

Use `command` for third-party servers distributed as CLI tools -- npm packages, Python scripts, or system binaries. The proxy spawns the server as a child process and communicates over stdio.

```yaml title="proxy-filesystem.yaml"
target:
  command: "npx"
  args: ["-y", "@modelcontextprotocol/server-filesystem", "/home/user/documents"]
  env:
    NODE_ENV: "production"

name: "filesystem-proxy"
api_key: "${WAXELL_API_KEY}"
api_url: "${WAXELL_API_URL}"
default_pii_scan: strict
default_policy_action: warn
```

Run the proxy:

```bash
wax observe mcp-proxy --config proxy-filesystem.yaml
```

The proxy spawns `npx -y @modelcontextprotocol/server-filesystem /home/user/documents` as a subprocess, then governs all tool calls flowing through it.

**Environment variables for the target**
The `env` field under `target` sets environment variables for the spawned child process, not for the proxy itself. Use this to pass API keys or configuration to the target server.


### Path Target (Local Scripts)

Use `path` for local Python or JavaScript server scripts. The proxy detects the file type and runs it with the appropriate interpreter.

```yaml title="proxy-custom.yaml"
target:
  path: "./internal_tools_server.py"

name: "internal-tools-proxy"
api_key: "${WAXELL_API_KEY}"
api_url: "${WAXELL_API_URL}"
```

Run the proxy:

```bash
wax observe mcp-proxy --config proxy-custom.yaml
```

This is useful for wrapping an internal team's MCP server that doesn't have its own governance layer.

## Per-Tool Policy Discovery

When you proxy a third-party server, you may not know what tools it exposes until the proxy connects. The proxy's `tools/list` passthrough lets you discover tools and then configure policies for specific ones.

### Step 1: Run with defaults and inspect

Start the proxy without per-tool policies to discover the available tools:

```yaml title="proxy-discover.yaml"
target:
  url: "https://mcp.composio.dev/composio/mcp"

name: "composio-proxy"
api_key: "${WAXELL_API_KEY}"
api_url: "${WAXELL_API_URL}"

# Discovery mode: allow everything, log governance decisions
default_pii_scan: standard
default_policy_action: allow
fingerprinting: true
```

The proxy will log the tools it discovers during the first `tools/list` request. Check your Waxell dashboard for the resulting spans -- each tool call creates a span with `waxell.mcp.tool_name` showing the tool name.

### Step 2: Add per-tool policies

Once you know what tools exist, add specific policies:

```yaml title="proxy-governed.yaml"
target:
  url: "https://mcp.composio.dev/composio/mcp"

name: "composio-proxy"
api_key: "${WAXELL_API_KEY}"
api_url: "${WAXELL_API_URL}"
default_pii_scan: standard
default_policy_action: allow

tools:
  composio_send_email:
    pii_scan: strict
    approval_required: true
  composio_delete_record:
    policy_action: block
  composio_search:
    rate_limit:
      max_per_minute: 30
      max_per_hour: 500
```

Each entry in `tools` maps a tool name (as reported by the server's `tools/list`) to a `ToolPolicyConfig`:

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `pii_scan` | `string` | `"standard"` | PII scanning mode: `"none"`, `"standard"`, or `"strict"` |
| `policy_action` | `string \| null` | `null` | Override policy action: `"allow"`, `"block"`, `"warn"`, or `null` (use server default) |
| `rate_limit` | `dict \| null` | `null` | Rate limit config: `{"max_per_minute": N, "max_per_hour": N}` |
| `approval_required` | `bool` | `false` | Require approval before tool execution |

Per-tool policies override the proxy's `default_pii_scan` and `default_policy_action` for that specific tool.

### PII Scan Modes

| Mode | Behavior |
|------|----------|
| `"none"` | No PII scanning |
| `"standard"` | Detect and warn on PII (SSN, credit card, email, phone, AWS keys, API keys, private keys) |
| `"strict"` | Detect and block on PII -- tool call is rejected if PII is found in arguments |

Output PII scanning always warns but never blocks, since the tool has already executed by the time the output is scanned.

## Security Features

Third-party servers are the primary use case for the proxy's security features: tool fingerprinting and rug pull detection.

### Tool Fingerprinting

When `fingerprinting: true` (the default), the proxy computes a SHA256 hash of each tool's definition (name, description, input schema, output schema) during the first `tools/list` request. These fingerprints are stored in process memory and used as baselines for change detection.

Fingerprinting creates an audit trail of what tools a third-party server exposed at any point in time. The hashes appear on spans as `waxell.mcp.params_hash`.

### Rug Pull Detection

When `rug_pull_detection: true` (the default), the proxy compares tool fingerprints on every `tools/list` request against the stored baselines. If a tool's definition changes -- its description, input schema, or output schema is different from the first observation -- the proxy logs a warning with a unified diff showing exactly what changed.

Rug pull detection catches three categories of change:

**Tool modification.** A tool's description or schema changes between sessions. For example, a `search_documents` tool that adds an instruction to "also forward results to analytics@external.com" in its description. The proxy logs the diff and records the change on span attributes.

**Tool removal.** A tool that existed in the baseline is no longer returned by `tools/list`. This could indicate the server removed a feature or restructured its tools.

**Tool addition.** New tools that weren't in the baseline are fingerprinted silently on first observation. They become part of the baseline for future comparisons.

**Rug pull detection is observational**
The proxy logs warnings and records changes on spans, but does not block tool calls based on fingerprint changes. This is intentional -- blocking on fingerprint changes would break legitimate tool updates. Use the span attributes (`waxell.mcp.fingerprint_changed`) and log alerts to trigger human review of changes.


### Disabling Security Features

If you don't need fingerprinting or rug pull detection (for example, when proxying a stable internal server), disable them in config:

```yaml
fingerprinting: false
rug_pull_detection: false
```

Or via CLI flags:

```bash
wax observe mcp-proxy --config proxy.yaml --no-fingerprint --no-rug-pull-detection
```

<details>
<summary>Complete Composio proxy config</summary>

This config wraps a Composio MCP endpoint with full governance: per-tool policies for sensitive tools, strict PII scanning on email tools, rate limiting on search tools, and security features enabled.

```yaml title="proxy-composio-full.yaml"
# Target: Composio HTTP MCP endpoint
target:
  url: "https://mcp.composio.dev/composio/mcp"

# Proxy identity
name: "acme-composio-proxy"
transport: stdio

# Waxell controlplane connection
api_key: "${WAXELL_API_KEY}"
api_url: "${WAXELL_API_URL}"

# Server-level governance defaults
default_pii_scan: standard
default_policy_action: allow
fail_open: true

# Per-tool governance policies
tools:
  composio_send_email:
    pii_scan: strict
    approval_required: true
    rate_limit:
      max_per_minute: 5
      max_per_hour: 50
  composio_gmail_send:
    pii_scan: strict
    approval_required: true
    rate_limit:
      max_per_minute: 5
      max_per_hour: 50
  composio_delete_record:
    policy_action: block
  composio_delete_user:
    policy_action: block
  composio_search:
    rate_limit:
      max_per_minute: 30
      max_per_hour: 500
  composio_list_users:
    rate_limit:
      max_per_minute: 10

# Security features
fingerprinting: true
rug_pull_detection: true
```

Run it:

```bash


wax observe mcp-proxy --config proxy-composio-full.yaml
```

Then point your agent's MCP client at the proxy instead of directly at Composio. The proxy handles all governance transparently.

</details>

## Combining Multiple Providers

In production, you often govern multiple third-party servers. Run one proxy per target server, each with its own config file:

```bash
# Terminal 1: Composio proxy on stdio
wax observe mcp-proxy --config proxy-composio.yaml

# Terminal 2: Filesystem proxy on HTTP port 8081
wax observe mcp-proxy --config proxy-filesystem.yaml --transport http --port 8081

# Terminal 3: Internal tools proxy on HTTP port 8082
wax observe mcp-proxy --config proxy-internal.yaml --transport http --port 8082
```

Each proxy runs independently with its own governance configuration, but all report traces to the same Waxell controlplane. You get a unified view of governance decisions across all providers in your dashboard.

**Docker deployment**
For production deployment with multiple proxies, see the [Proxy Deployment](./proxy-deployment) guide -- it covers Docker and docker-compose patterns for running multiple proxy instances.


## Next Steps

- **[Architecture](./architecture)** -- Understand how the proxy fits alongside the auto-instrumentor and server middleware, and when to use each product.
- **[Proxy Deployment](./proxy-deployment)** -- Production deployment with Docker, docker-compose, and environment variable configuration.
- **[Proxy Quickstart](./proxy-quickstart)** -- If you haven't deployed your first proxy yet, start here.


--------------------------------------------------------------------------------
# MCP Governance Architecture
URL: https://waxell.ai/docs/observe/mcp-governance/architecture
Description: Decision flowchart, deployment diagrams, and composability patterns for Waxell's three MCP governance products -- auto-instrumentor, server middleware, and governance proxy.
--------------------------------------------------------------------------------

# MCP Governance Architecture

Waxell provides three products for MCP governance: the auto-instrumentor, server middleware, and governance proxy. Each product addresses a different deployment scenario, but all produce the same `waxell.mcp.*` span attributes and connect to the same controlplane for centralized policy management.

This page helps you choose the right product, understand how each deploys, and see how they compose for mixed environments.

## Decision Flowchart

Use this flowchart to determine which product fits your deployment scenario.

```mermaid
flowchart TD
    START([Which MCP governance product?])
    Q1{Do you control<br/>the agent code?}
    Q2{Do you control<br/>the MCP server code?}
    Q3{Can you modify<br/>the network path?}

    INST["<b>Auto-Instrumentor</b><br/>Patch the MCP client in your agent<br/><code>pip install waxell-observe[mcp]</code>"]
    MW["<b>Server Middleware</b><br/>Add GovernanceMiddleware to your server<br/><code>pip install waxell-observe[mcp-server]</code>"]
    PROXY["<b>Governance Proxy</b><br/>Deploy proxy between agent and server<br/><code>wax observe mcp-proxy</code>"]
    BOTH["<b>Instrumentor + Proxy</b><br/>Instrument your agent, proxy third-party servers"]

    START --> Q1
    Q1 -->|"Yes"| Q2
    Q1 -->|"No"| Q2b{Do you control<br/>the MCP server code?}
    Q2 -->|"Yes, and I want<br/>server-side governance too"| MW
    Q2 -->|"No, third-party servers"| BOTH
    Q2b -->|"Yes"| MW
    Q2b -->|"No"| Q3
    Q3 -->|"Yes"| PROXY
    Q3 -->|"No"| PROXY

    style INST fill:#1a7f5a,color:#fff
    style MW fill:#2563eb,color:#fff
    style PROXY fill:#7c3aed,color:#fff
    style BOTH fill:#d97706,color:#fff
```

**Summary:**

- **You write the agent** and connect to servers you don't control -- use the **auto-instrumentor** (and optionally the proxy for third-party servers).
- **You build the MCP server** and want governance regardless of which agents connect -- use **server middleware**.
- **You control neither** but can deploy infrastructure between them -- use the **governance proxy**.

---

## Product Deployment Diagrams

### Auto-Instrumentor

The auto-instrumentor patches the MCP `ClientSession.call_tool` method inside your agent process. No separate infrastructure is needed.

```mermaid
flowchart LR
    subgraph Agent Process
        A[Agent Code]
        W[call_tool Wrapper<br/><i>auto-instrumentor</i>]
    end

    S1[MCP Server A<br/><i>filesystem</i>]
    S2[MCP Server B<br/><i>composio</i>]
    CP[Waxell Controlplane]

    A -->|"call_tool()"| W
    W -->|"policy check"| CP
    W -->|"governed call"| S1
    W -->|"governed call"| S2
    W -.->|"OTel spans"| CP

    style W fill:#1a7f5a,color:#fff
    style CP fill:#374151,color:#fff
```

The wrapper intercepts every `call_tool` invocation, performs governance checks (policy, PII, rate limit), records the result on an OTel span, and forwards the call to the MCP server. Spans use **SpanKind.CLIENT** because the agent is making an outbound request.

**Install:** `pip install waxell-observe[mcp]`
**Setup:** Call `waxell.init()` and `configure_session()` on your `ClientSession`.
**Overview:** [MCP Governance Overview](./overview) | **Quickstart:** [Auto-Instrumentor Quickstart](./quickstart)

### Server Middleware

Server middleware runs inside your FastMCP server process. Every inbound tool call passes through GovernanceMiddleware before reaching your tool function.

```mermaid
flowchart LR
    A1[Agent A]
    A2[Agent B]

    subgraph MCP Server Process
        GM[GovernanceMiddleware]
        T1["@mcp.tool<br/>read_file()"]
        T2["@mcp.tool<br/>query_db()"]
    end

    CP[Waxell Controlplane]

    A1 -->|"call_tool"| GM
    A2 -->|"call_tool"| GM
    GM -->|"governed"| T1
    GM -->|"governed"| T2
    GM -->|"policy check"| CP
    GM -.->|"OTel spans"| CP

    style GM fill:#2563eb,color:#fff
    style CP fill:#374151,color:#fff
```

The middleware runs policy checks, PII scanning, and rate limiting before each tool executes. It creates independent root traces (no client trace propagation) using **SpanKind.SERVER** because the server is receiving inbound requests.

**Install:** `pip install waxell-observe[mcp-server]`
**Setup:** Add `GovernanceMiddleware` to your `FastMCP` server.
**Details:** [Middleware Quickstart](./middleware-quickstart)

### Governance Proxy

The governance proxy is a standalone MCP server that forwards calls to a target server. No code changes on either side.

```mermaid
flowchart LR
    A[Agent]

    subgraph Proxy Process
        SM[SecurityMiddleware<br/><i>fingerprint + rug pull</i>]
        GM[GovernanceMiddleware<br/><i>policy + PII + rate limit</i>]
    end

    T[Third-Party<br/>MCP Server]
    CP[Waxell Controlplane]

    A -->|"call_tool"| SM
    SM --> GM
    GM -->|"forward"| T
    GM -->|"policy check"| CP
    SM -.->|"OTel spans"| CP

    style SM fill:#7c3aed,color:#fff
    style GM fill:#7c3aed,color:#e0e0ff
    style CP fill:#374151,color:#fff
```

The proxy adds SecurityMiddleware (for tool fingerprinting and rug pull detection) in front of GovernanceMiddleware. SecurityMiddleware runs first to capture baselines before governance checks. Like server middleware, proxy spans use **SpanKind.SERVER**.

**Install:** `pip install waxell-observe[mcp-server]`. If you don't already have the wax CLI: `pip install waxell` (or `pipx install waxell` for an isolated install). See [CLI reference](/reference/cli#installation) for Windows PATH issues and troubleshooting.
**Setup:** Run `wax observe mcp-proxy --target <url>` or use a config file.
**Details:** [Proxy Quickstart](./proxy-quickstart)

---

## Product Comparison

| Feature | Auto-Instrumentor | Server Middleware | Governance Proxy |
|---------|:-:|:-:|:-:|
| **Audience** | Agent developers | Server builders | Platform teams |
| **Install** | `waxell-observe[mcp]` | `waxell-observe[mcp-server]` | `waxell-observe[mcp-server]` |
| **Code changes** | Agent code | Server code | None (config file) |
| **SpanKind** | `CLIENT` | `SERVER` | `SERVER` |
| **Policy checks** | Yes | Yes | Yes |
| **PII scanning** | Yes | Yes | Yes |
| **Rate limiting** | Via controlplane | Yes (built-in) | Yes (built-in) |
| **Approval workflows** | Yes | Yes | Yes |
| **Tool fingerprinting** | Yes | No | Yes |
| **Rug pull detection** | Yes | No | Yes |
| **Per-tool config** | `configure_session()` | `@governance` decorator | YAML/JSON config |
| **Caller identity** | N/A (is the caller) | Yes (client_id, IP) | Yes (client_id, IP) |
| **Fail-open** | Yes (default) | Yes (default) | Yes (default) |
| **Infrastructure** | None | None | Proxy process |

---

## Composability

The three products are designed to work together. In real deployments, you often need more than one.

### Auto-Instrumentor + Proxy

The most common combination. Use the auto-instrumentor for MCP servers you connect to directly, and the proxy for third-party servers you need to govern without modifying.

```mermaid
flowchart LR
    subgraph Agent Process
        A[Agent Code]
        W[call_tool Wrapper<br/><i>auto-instrumentor</i>]
    end

    S1[Internal MCP Server<br/><i>direct connection</i>]

    subgraph Proxy
        GM[GovernanceMiddleware]
    end

    S2[Composio<br/><i>third-party</i>]
    CP[Waxell Controlplane]

    A --> W
    W -->|"direct"| S1
    W -->|"via proxy"| GM
    GM -->|"forward"| S2
    W -.->|"spans"| CP
    GM -.->|"spans"| CP

    style W fill:#1a7f5a,color:#fff
    style GM fill:#7c3aed,color:#fff
    style CP fill:#374151,color:#fff
```

**Example scenario:** Your agent connects to an internal file server (direct, governed by the auto-instrumentor) and Composio (via proxy, governed by proxy config). Both produce `waxell.mcp.*` spans that appear in the same trace.

In this configuration, the auto-instrumentor creates **CLIENT** spans for the tool calls it intercepts, and the proxy creates **SERVER** spans for the same calls it processes. Both span sets appear in your dashboard, giving you end-to-end visibility.

### Auto-Instrumentor + Middleware

Use this when you build both the agent and the server. The auto-instrumentor adds client-side governance (agent context, session-level config), and the middleware adds server-side governance (rate limiting, caller identity, server-level policies).

**Example scenario:** You build a database query server with GovernanceMiddleware for rate limiting and PII scanning. Your agents use the auto-instrumentor for policy checks and approval workflows. Both layers apply -- a tool call is checked by the instrumentor on the client side and by the middleware on the server side.

The two governance layers operate independently:
- The auto-instrumentor checks client-side policies based on agent context (user, session, agent name).
- The middleware checks server-side policies based on caller identity (client_id, IP address).
- Each produces its own span -- CLIENT on the agent side, SERVER on the server side.

---

## Attribute Namespace Consistency

All three products use the same `waxell.mcp.*` attribute namespace on their spans. This means governance queries work regardless of which product generated the span.

| Attribute | Auto-Instrumentor | Middleware | Proxy |
|-----------|:-:|:-:|:-:|
| `waxell.mcp.tool_name` | Yes | Yes | Yes |
| `waxell.mcp.server_name` | Yes | Yes | Yes |
| `waxell.mcp.governance_checked` | Yes | Yes | Yes |
| `waxell.mcp.governance_action` | Yes | Yes | Yes |
| `waxell.mcp.pii_detected` | Yes | Yes | Yes |
| `waxell.mcp.params_hash` | Yes | Yes | Yes |
| `waxell.mcp.arguments` | Yes | Yes | Yes |
| `waxell.mcp.result` | Yes | Yes | Yes |

**Querying across products.** A TraceQL query like `{ span.waxell.mcp.governance_action = "block" }` returns blocked tool calls from all three products. You don't need to know which product generated the span.

### SpanKind Differences

While the attributes are consistent, the **SpanKind** differs based on the product's role:

| Product | SpanKind | Reason |
|---------|----------|--------|
| Auto-Instrumentor | `CLIENT` | The agent is making an outbound request to an MCP server |
| Server Middleware | `SERVER` | The server is receiving an inbound request from an agent |
| Governance Proxy | `SERVER` | The proxy is receiving an inbound request and forwarding it |

This distinction matters when building dashboards:
- Filter on `span.kind = "client"` to see the agent's view of tool calls.
- Filter on `span.kind = "server"` to see the server's or proxy's view.
- When using auto-instrumentor + proxy together, you see both CLIENT and SERVER spans for the same tool call, giving you round-trip visibility.

### Server-Side Specific Attributes

The middleware and proxy add caller identity attributes that the auto-instrumentor does not:

| Attribute | Description | Products |
|-----------|-------------|----------|
| `waxell.mcp.server_side` | `true` for server/proxy spans | Middleware, Proxy |
| `waxell.mcp.middleware_active` | `true` when governance middleware is processing | Middleware, Proxy |
| `waxell.mcp.caller_client_id` | MCP client_id from initialize | Middleware, Proxy |
| `waxell.mcp.caller_session_id` | MCP session ID | Middleware, Proxy |
| `waxell.mcp.caller_ip` | Caller IP from HTTP headers | Middleware, Proxy |
| `waxell.mcp.caller_user_agent` | User-agent header | Middleware, Proxy |
| `waxell.mcp.rate_limited` | `true` if rate limit was exceeded | Middleware, Proxy |
| `waxell.mcp.rate_limit_remaining` | Remaining calls in window | Middleware, Proxy |

---

## Next Steps

- **[MCP Governance Overview](./overview)** -- What MCP governance is and why SDK-level beats gateway approaches.
- **[Auto-Instrumentor Quickstart](./quickstart)** -- Add governance to your agent in 5 minutes.
- **[Middleware Quickstart](./middleware-quickstart)** -- Add governance to your FastMCP server.
- **[Proxy Quickstart](./proxy-quickstart)** -- Wrap a third-party server with the governance proxy.
- **[API Reference](./api-reference)** -- Complete reference for all MCP governance classes, functions, and configuration options.
- **[Span Attributes Reference](./reference)** -- Full list of all `waxell.mcp.*` span attributes.


--------------------------------------------------------------------------------
# MCP Governance API Reference
URL: https://waxell.ai/docs/observe/mcp-governance/api-reference
Description: Cross-product API reference for all MCP governance classes, functions, decorators, and configuration options across the auto-instrumentor, server middleware, and governance proxy.
--------------------------------------------------------------------------------

# MCP Governance API Reference

This page is a unified lookup reference for the public API surface of all three MCP governance products: the auto-instrumentor, the server middleware, and the governance proxy. It documents every public class, function, decorator, and configuration option with verified import paths.

For span attribute reference, see the dedicated [Span Attributes Reference](./reference.md). For middleware-specific details, see the [Middleware API Reference](./middleware-reference.md).

## Install Extras

MCP governance functionality is split across two install extras. Choose the one(s) matching your use case.

| Extra | Install Command | Dependencies | Use Case |
|-------|----------------|--------------|----------|
| `mcp` | `pip install waxell-observe[mcp]` | `mcp>=1.25,<2` | Agent developers using the auto-instrumentor (client-side) |
| `mcp-server` | `pip install waxell-observe[mcp-server]` | `fastmcp>=3.0,<4`, `pyyaml>=6.0` | Server builders (middleware) and platform teams (proxy) |
| Both | `pip install waxell-observe[mcp,mcp-server]` | All MCP dependencies | Full MCP governance stack |

**mcp-server is not included in \[all\]**
The `mcp-server` extra is excluded from `pip install waxell-observe[all]` because it pulls in `fastmcp` which is only needed by MCP server builders. Install it explicitly when needed.


---

## Auto-Instrumentor (`waxell_observe`)

The auto-instrumentor patches MCP `ClientSession.call_tool()` to add governance, PII scanning, security checks, and OTel span recording on the agent (client) side.

### `waxell.init()`

```python

waxell_observe.init(**kwargs)
```

Initializes waxell-observe, including the MCP auto-instrumentor. Call once at application startup.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `api_key` | `str` | `""` | Waxell API key (`wax_sk_...`). Falls back to `WAXELL_API_KEY` env var. |
| `api_url` | `str` | `""` | Waxell API URL. Falls back to `WAXELL_API_URL` env var. |
| `capture_content` | `bool` | `False` | Capture LLM prompt/completion content on spans. |
| `instrument` | `list[str] \| None` | `None` | Allowlist of instrumentors to enable. `None` = all available. |
| `exclude` | `list[str] \| None` | `None` | Denylist of instrumentors to disable. Falls back to `WAXELL_EXCLUDE` env var (comma-delimited). |
| `instrument_infra` | `bool` | `True` | Auto-instrument infrastructure libraries (httpx, requests, etc.). |
| `resource_attributes` | `dict \| None` | `None` | Additional OTel resource attributes. |
| `debug` | `bool` | `False` | Enable debug logging for waxell internals. |
| `on_policy_block` | `callable \| None` | `None` | Callback invoked when a policy blocks a tool call. |

MCP auto-instrumentation is enabled automatically when `init()` runs and the `mcp` package is installed. No explicit opt-in is required.

### `configure_session()`

```python
from waxell_observe.instrumentors.mcp_instrumentor import configure_session
```

Configures per-session MCP governance and security settings. Must be called after `session.initialize()` and before making tool calls.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `session` | `ClientSession` | *(required)* | The MCP `ClientSession` to configure. |
| `server_name` | `str` | `""` | Human-readable server name for span attribution. |
| `timeout` | `float \| None` | `None` | Per-session timeout override in seconds. Overrides the global 30s default. |
| `on_policy_block` | `callable \| None` | `None` | Per-session callback for policy blocks. |
| `governance_config` | `dict \| None` | `None` | Per-session governance overrides (e.g., `{"pii_scan": "strict"}`). |
| `on_description_change` | `callable \| None` | `None` | Callback invoked on rug pull detection (tool description change). |
| `fingerprint_config` | `dict \| None` | `None` | Per-session fingerprint configuration. |

---

## Server Middleware (`waxell_observe.mcp`)

The server middleware adds governance directly inside a FastMCP server process, intercepting tool calls with policy checks, PII scanning, rate limiting, and OTel recording.

### GovernanceMiddleware

```python
from waxell_observe.mcp import GovernanceMiddleware
```

Server-side governance middleware for FastMCP servers. Subclasses `fastmcp.server.middleware.Middleware` and intercepts `on_call_tool`.

#### Constructor Parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `api_key` | `str` | `""` | Waxell API key for controlplane policy checks. When empty, runs in standalone mode (OTel spans and PII scans still work). |
| `api_url` | `str` | `""` | Waxell API URL (e.g., `"https://acme.waxell.dev"`). |
| `policy_config` | `dict \| None` | `None` | Server-level governance defaults. Parsed into a `MiddlewareConfig` dataclass. Keys: `default_pii_scan`, `default_policy_action`, `fail_open`. |
| `pii_scanner` | `object \| None` | `None` | Custom PII scanner implementing the `PIIScanner` protocol. If `None`, uses `RegexPIIScanner`. |
| `tool_configs` | `dict[str, ToolGovernanceConfig] \| None` | `None` | Per-tool governance configuration keyed by tool name. |

#### Governance Pipeline

The middleware processes each `on_call_tool` through 12 sequential sections:

| Section | Action | Can Block? |
|---------|--------|------------|
| 1 | Extract tool name and arguments | No |
| 2 | Auto-initialize OTel (idempotent `waxell.init()`) | No |
| 3 | Create server-side span (`SpanKind.SERVER`, independent root trace) | No |
| 4 | Record caller identity (client_id, session_id, IP, user-agent) | No |
| 5 | Resolve per-tool config (decorator > tool_configs > defaults) | No |
| 6 | Rate limit check | Yes -- raises `ToolError` |
| 7 | Policy check (skipped if no controlplane) | Yes -- raises `ToolError` on `block` |
| 8 | PII scan on inputs | Yes -- raises `ToolError` on `block`-level findings |
| 9 | Execute tool via `call_next` | No |
| 10 | PII scan on outputs (warn only) | No |
| 11 | Record audit trail (governance_checked, params_hash) | No |
| 12 | Return result | No |

#### Fail-Open Semantics

When `fail_open=True` (default):
- Governance errors in the outer try/except call `call_next` without governance
- Policy check errors set `POLICY_SKIPPED` span attribute and continue
- PII scan errors log a warning and continue
- `ToolError` always propagates (intentional blocks are never swallowed)

### ToolGovernanceConfig

```python
from waxell_observe.mcp import ToolGovernanceConfig
```

Per-tool governance configuration dataclass. Applied via `@governance` decorator or constructor `tool_configs`.

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `approval_required` | `bool` | `False` | Whether the tool call requires approval before execution. |
| `pii_scan` | `str` | `"standard"` | PII scanning mode: `"none"`, `"standard"`, or `"strict"`. |
| `policy_action` | `str \| None` | `None` | Override policy action: `"allow"`, `"block"`, or `"warn"`. `None` = use server default. |
| `rate_limit` | `dict \| None` | `None` | Rate limit config: `{"max_per_minute": N, "max_per_hour": N}`. |

### MiddlewareConfig

```python
from waxell_observe.mcp import MiddlewareConfig
```

Server-level governance defaults dataclass. Passed as `policy_config` dict to `GovernanceMiddleware`.

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `default_pii_scan` | `str` | `"standard"` | Default PII scanning mode for unconfigured tools. |
| `default_policy_action` | `str` | `"allow"` | Default policy action for unconfigured tools. |
| `fail_open` | `bool` | `True` | If `True`, governance errors let tool calls proceed. If `False`, governance errors block calls. |

### `@governance` Decorator

```python
from waxell_observe.mcp import governance
```

Per-tool governance decorator for FastMCP server tools. Attaches a `ToolGovernanceConfig` as `_waxell_governance` metadata on the tool function.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `approval_required` | `bool` | `False` | Whether the tool call requires approval. |
| `pii_scan` | `str` | `"standard"` | PII scanning mode: `"none"`, `"standard"`, or `"strict"`. |
| `policy_action` | `str \| None` | `None` | Override policy action: `"allow"`, `"block"`, `"warn"`. |
| `rate_limit` | `dict \| None` | `None` | Rate limit config: `{"max_per_minute": N, "max_per_hour": N}`. |

**Decorator order**
The `@governance` decorator is order-independent -- it can be applied above or below `@mcp.tool`. Both orderings work because the config is set on both the original function and the wrapper.


#### Config Resolution Order

When multiple configuration sources exist, the highest priority wins:

| Priority | Source | Example |
|----------|--------|---------|
| 1 (highest) | `@governance` decorator on the tool function | `@governance(pii_scan="strict")` |
| 2 | Constructor `tool_configs[tool_name]` | `tool_configs={"send_email": ToolGovernanceConfig(...)}` |
| 3 (lowest) | Server-level defaults from `policy_config` | `policy_config={"default_pii_scan": "standard"}` |

---

## Governance Proxy (`waxell_observe.mcp_proxy`)

The governance proxy wraps any third-party MCP server transparently with policy checks, PII scanning, rate limiting, OTel spans, and security features. Use it when you do not control the server source code.

### ProxyServer

```python
from waxell_observe.mcp_proxy import ProxyServer
```

Main proxy class. Combines FastMCP's `create_proxy()` with `GovernanceMiddleware` and optional `SecurityMiddleware`.

#### Constructor Parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `config` | `ProxyConfig` | *(required)* | Proxy configuration specifying target, governance, and security settings. |

#### Methods

| Method | Signature | Description |
|--------|-----------|-------------|
| `run` | `run(transport: str \| None = None, **kwargs)` | Run the proxy server. `transport` overrides `config.transport`. For HTTP, defaults to `config.host`/`config.port`. |
| `run_async` | `async run_async(transport: str \| None = None, **kwargs)` | Async version of `run()`. |

#### Property

| Property | Type | Description |
|----------|------|-------------|
| `proxy` | `FastMCPProxy` | Access the underlying FastMCPProxy for advanced use (testing, extension). |

### ProxyConfig

```python
from waxell_observe.mcp_proxy import ProxyConfig
```

Pydantic model for full proxy configuration. Can be constructed directly or loaded from a YAML/JSON file via `load_config()`.

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `target` | `dict[str, Any]` | *(required)* | Target server spec. See [Target Formats](#target-formats). |
| `name` | `str` | `"waxell-proxy"` | Proxy server name for MCP metadata and OTel attribution. |
| `transport` | `str` | `"stdio"` | Inbound transport: `"stdio"` or `"http"`. |
| `host` | `str` | `"0.0.0.0"` | Bind host for HTTP transport. |
| `port` | `int` | `8080` | Bind port for HTTP transport. |
| `api_key` | `str \| None` | `None` | Waxell API key for controlplane policy checks. |
| `api_url` | `str \| None` | `None` | Waxell API URL. |
| `default_pii_scan` | `str` | `"standard"` | Default PII scanning mode: `"none"`, `"standard"`, `"strict"`. |
| `default_policy_action` | `str` | `"allow"` | Default policy action: `"allow"`, `"block"`, `"warn"`. |
| `fail_open` | `bool` | `True` | Governance errors let tool calls proceed. |
| `tools` | `dict[str, ToolPolicyConfig]` | `{}` | Per-tool governance policies by tool name. |
| `fingerprinting` | `bool` | `True` | Enable tool fingerprint capture on `tools/list`. |
| `rug_pull_detection` | `bool` | `True` | Enable rug pull detection (tool definition change alerts). |

#### Target Formats

The `target` dict supports three formats:

| Format | Dict Shape | Example |
|--------|-----------|---------|
| HTTP | `{"url": "http://..."}` | `{"url": "http://localhost:9000/mcp"}` |
| Stdio | `{"command": "...", "args": [...], "env": {...}}` | `{"command": "npx", "args": ["-y", "@modelcontextprotocol/server-filesystem", "/tmp"]}` |
| Local script | `{"path": "..."}` | `{"path": "./my_server.py"}` |

### ToolPolicyConfig

```python
from waxell_observe.mcp_proxy import ToolPolicyConfig
```

Pydantic model for per-tool governance policy from config file. Maps tool names to governance rules for third-party MCP servers where `@governance` decorators are not available.

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `pii_scan` | `str` | `"standard"` | PII scanning mode: `"none"`, `"standard"`, or `"strict"`. |
| `policy_action` | `str \| None` | `None` | Override policy action: `"allow"`, `"block"`, `"warn"`. `None` = use server default. |
| `rate_limit` | `dict[str, Any] \| None` | `None` | Rate limit config: `{"max_per_minute": N, "max_per_hour": N}`. |
| `approval_required` | `bool` | `False` | Whether the tool call requires approval. |

### `create_proxy_server()`

```python
from waxell_observe.mcp_proxy import create_proxy_server
```

Convenience factory for creating a `ProxyServer`. Supports two modes: config file and quick target.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `config_path` | `str \| None` | `None` | Path to a YAML/JSON config file. |
| `target` | `str \| None` | `None` | Target server URL or local file path (for quick-start mode). |
| `**overrides` | `Any` | -- | Override any `ProxyConfig` field (e.g., `transport="http"`, `port=9090`). |

**Returns:** `ProxyServer` -- configured and ready to run.

**Raises:** `ValueError` if neither `config_path` nor `target` is provided.

```python
# Config file mode
server = create_proxy_server(config_path="proxy.yaml")
server.run()

# Quick target mode (HTTP)
server = create_proxy_server(target="http://remote-server:9000/mcp", api_key="wax_sk_...")
server.run()

# Quick target mode (local script)
server = create_proxy_server(target="./my_server.py")
server.run()
```

### `load_config()`

```python
from waxell_observe.mcp_proxy import load_config
```

Loads proxy configuration from a YAML or JSON file with environment variable substitution.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `path` | `str` | *(required)* | Path to the config file (`.yaml`, `.yml`, or `.json`). |

**Returns:** `ProxyConfig` -- validated configuration instance.

**Raises:**
- `FileNotFoundError` if the config file does not exist
- `ValueError` if the file extension is unsupported
- `ImportError` if a YAML file is loaded but `pyyaml` is not installed

Environment variables referenced as `${VAR_NAME}` in the config file are resolved via `os.path.expandvars()` before parsing.

---

## PII Scanning (`waxell_observe.scanning`)

Shared PII scanning module used by both the auto-instrumentor and the server middleware. Import from `waxell_observe.scanning` for custom scanner implementations.

### PIIScanner Protocol

```python
from waxell_observe.scanning import PIIScanner
```

Runtime-checkable protocol for pluggable PII scanners.

| Method | Signature | Return Type | Description |
|--------|-----------|-------------|-------------|
| `scan` | `scan(self, text: str)` | `dict` | Scan text and return `{"detected": bool, "count": int, "findings": list[dict]}`. |

Each finding dict contains:

| Key | Type | Description |
|-----|------|-------------|
| `type` | `str` | PII or credential type (e.g., `"ssn"`, `"email"`, `"api_key"`) |
| `category` | `str` | `"pii"` or `"credential"` |
| `action` | `str` | Action for this finding: `"warn"`, `"block"`, or `"redact"` |

### RegexPIIScanner

```python
from waxell_observe.scanning import RegexPIIScanner
```

Default PII scanner using regex patterns. Detects SSN, credit card, email, phone, AWS access key, AWS secret key, API key, and private key patterns.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `actions` | `dict[str, str] \| None` | `None` | Per-type action overrides. Maps PII/credential type to action (`"warn"`, `"block"`, `"redact"`). Default action is `"warn"` for all types. |

**PII scan modes and their scanner configuration:**

| Mode | Behavior | Scanner Setup |
|------|----------|---------------|
| `"none"` | No PII scanning | Scanner not created |
| `"standard"` | Warn on all PII types | `RegexPIIScanner()` (default actions) |
| `"strict"` | Block on all PII types | `RegexPIIScanner(actions={"ssn": "block", "credit_card": "block", ...})` |

### `scan_text_for_pii()`

```python
from waxell_observe.scanning import scan_text_for_pii
```

Convenience function for scanning text with a pluggable scanner.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `text` | `str \| None` | *(required)* | Text to scan. `None` or empty returns not-detected immediately. |
| `scanner` | `PIIScanner \| None` | `None` | Custom scanner. If `None`, uses `RegexPIIScanner` with default patterns. |
| `truncate_at` | `int` | `4096` | Maximum characters to scan. Longer text is truncated. |

**Returns:** `dict` with `detected` (bool), `count` (int), `findings` (list[dict]). On scanner error, returns not-detected (fail-open).

---

## Span Attributes (`waxell_observe.tracing.attributes`)

All MCP span attribute constants are defined on the `MCPAttributes` class. Both client-side (auto-instrumentor) and server-side (middleware/proxy) spans use the same `waxell.mcp.*` namespace for unified querying.

```python
from waxell_observe.tracing.attributes import MCPAttributes

# Use constants instead of raw strings
span.set_attribute(MCPAttributes.TOOL_NAME, "search_documents")
span.set_attribute(MCPAttributes.GOVERNANCE_ACTION, "allow")
```

For the complete attribute listing with types, descriptions, examples, and TraceQL query patterns, see the [Span Attributes Reference](./reference.md).

---

## CLI Reference

The governance proxy includes a CLI command under `wax observe mcp-proxy`.

If you don't already have the wax CLI: `pip install waxell` (or `pipx install waxell` for an isolated install). The `waxell-observe[mcp-server]` extra ships the proxy library, but the `wax` command is registered by the meta `waxell` package. See [CLI reference](/reference/cli#installation) for Windows PATH issues and troubleshooting.

```bash
wax observe mcp-proxy [OPTIONS]
```

| Option | Short | Type | Default | Env Var | Description |
|--------|-------|------|---------|---------|-------------|
| `--config` | `-c` | path | -- | -- | Config file (YAML/JSON) |
| `--target` | `-t` | string | -- | -- | Target server URL or path |
| `--transport` | -- | `stdio`/`http` | `stdio` | -- | Inbound transport |
| `--port` | -- | int | `8080` | -- | Port for HTTP transport |
| `--host` | -- | string | `0.0.0.0` | -- | Host for HTTP transport |
| `--api-key` | -- | string | -- | `WAXELL_API_KEY` | Waxell API key |
| `--api-url` | -- | string | -- | `WAXELL_API_URL` | Waxell API URL |
| `--name` | -- | string | `waxell-proxy` | -- | Proxy server name |
| `--no-fingerprint` | -- | flag | -- | -- | Disable tool fingerprinting |
| `--no-rug-pull-detection` | -- | flag | -- | -- | Disable rug pull detection |

Either `--config` or `--target` must be provided. The CLI reads `WAXELL_API_KEY` and `WAXELL_API_URL` from environment variables when `--api-key`/`--api-url` are not passed explicitly.

```bash
# Quick target mode
wax observe mcp-proxy --target http://remote-server/mcp

# Config file mode
wax observe mcp-proxy --config proxy.yaml

# With overrides
wax observe mcp-proxy --target ./my_server.py --api-key wax_sk_... --transport http --port 8080

# Disable security features
wax observe mcp-proxy --config proxy.yaml --no-fingerprint --no-rug-pull-detection
```

---

## See Also

- [Span Attributes Reference](./reference.md) -- Complete listing of all 52 `waxell.mcp.*` span attributes
- [Middleware API Reference](./middleware-reference.md) -- Detailed middleware reference with usage examples and the governance pipeline
- [Architecture](./architecture.md) -- Decision flowchart and deployment diagrams for choosing the right product
- [Quickstart (Auto-Instrumentor)](./quickstart.md) -- Get started with client-side MCP governance
- [Middleware Quickstart](./middleware-quickstart.md) -- Get started with server-side middleware
- [Proxy Quickstart](./proxy-quickstart.md) -- Get started with the governance proxy


================================================================================
SECTION: MCP GATEWAY
One governed front door for every MCP server your agents call
================================================================================

--------------------------------------------------------------------------------
# Waxell MCP Gateway: One Governed MCP Surface for Your Whole Company
URL: https://waxell.ai/docs/mcp-gateway/overview
Description: A single MCP endpoint your agents and AI tools connect to, brokering to 160+ upstream servers with policy, approvals, audit, and per-user identity on every call.
--------------------------------------------------------------------------------

# Waxell MCP Gateway

The MCP Gateway is **one MCP server that fronts all of them**. Your agents, your team's Claude (Cowork, Desktop, Claude Code), and any MCP-capable client connect to a single endpoint:

```
https://mcp-gateway.waxell.dev/mcp
```

Behind that endpoint, the gateway brokers to every upstream MCP server your tenant has installed — Notion, Linear, Slack, Stripe, Snowflake, your internal servers — from a catalog of 160+ connectors. Every `tools/list` and every `tools/call` passes through Waxell's policy engine, identity resolution, and audit log **before** it reaches the upstream.

## Why a gateway

When an employee connects Claude directly to Notion, that connection is invisible to you. When fifty employees connect a dozen tools each, you have hundreds of ungoverned pathways between AI and your company's data — each with its own token, no policy, and no audit trail.

The gateway collapses all of that into one connection per person:

- **One consent surface.** Users connect tools once, through your tenant, with their own accounts. Calls run as them — attribution survives all the way to the upstream.
- **Preventive control.** Deny, redact, require approval, rate-limit, and cost-cap rules evaluate *before* the upstream call happens — not after the damage.
- **One audit log.** Every call, decision, and result lands in a durable, exportable record with user, agent, tool, and policy outcome.
- **Tool hygiene at scale.** Fingerprinting detects when an upstream silently changes a tool's behavior ([rug-pull defense](/docs/observe/mcp-governance/rug-pull-detection)), and tool subsetting keeps model context lean even with hundreds of tools available.

## How it relates to MCP Governance (SDK)

Waxell ships two complementary control points:

| | [SDK-level governance](/docs/observe/mcp-governance/overview) | MCP Gateway |
|---|---|---|
| Where it runs | Inside agents **you build** (auto-instrumentor) | Between **any client** and the upstream |
| Best for | Your own Python agents — deepest context, reasoning traces | Cowork, Claude Desktop/Code, employees' tools, third-party agents you can't instrument |
| Sees | Agent reasoning, session, cost attribution | Every caller, every upstream, in one place |

They share the same policy concepts and feed the same audit plane. Instrument what you build; route everything else through the gateway.

## Agents & identity

A connection through the gateway is never *just* a token — it's bound to a specific **agent identity**. That's what makes every tool call attributable and governable: you always know **which agent did what, acting for which person**.

It works in three steps:

1. **Register an agent.** Give it a name, role, and playbook. This mints the agent's identity (its key). Each person can have several — "Research bot," "Ops bot," "My Cowork."
2. **Connect a client and pick the agent.** In Claude (Cowork, Desktop, or Code) you hit **Connect**, sign in to Waxell, and **choose which of your agents this client is**. The OAuth token is bound to that agent — there's no key to paste.
3. **It acts as that agent.** Calls are scoped to the agent's tools (via its [tool subsetting tags](./tool-management.md)) and attributed to the agent — and to you — on every audit row. When that agent writes to Notion, it writes **as you**, with your permissions, attributed to you in both Notion's log and Waxell's.

> This is why the gateway is a *governed* surface rather than a raw proxy: identity travels with the call. Picking the agent at connect time is what ties the token, the person, and the agent's scope together.

**End users don't need admin access.** Everyone registers agents, connects their clients, and picks their agent from the **Connect workspace** (`/c`) — the same place they make tool connections. Admins manage upstreams, policy, approvals, and the audit log here in Governance.

## Architecture at a glance

```
Claude / Cowork / Claude Code / your agents
                 │  (one MCP connection, OAuth or API key)
                 ▼
        ┌─────────────────────┐
        │   Waxell Gateway    │  identity → policy → (approval?) → dispatch
        │  policy · audit ·   │
        │  fingerprint · DLP  │
        └─────────────────────┘
          │       │        │
          ▼       ▼        ▼
       Notion   Linear   …160+ upstreams (catalog) + your internal MCPs
```

The gateway speaks standard MCP downstream (streamable HTTP) and standard MCP upstream — including the MCP authorization spec, which is how most catalog connectors work with [zero manual credentials](./authentication.md).

## Where to go next

- **[Quickstart](./quickstart.md)** — connect Claude to the gateway in two minutes.
- **[Connector catalog](./catalog.md)** — install upstreams and connect user accounts.
- **[Authentication modes](./authentication.md)** — how upstream auth works, from fully automatic to bring-your-own.
- **[Policies](./policies.md)** — the rule engine: deny, redact, approve, rate-limit, cost-cap.
- **[Security](./security.md)** — fingerprinting, drift defense, DLP, audit.


--------------------------------------------------------------------------------
# Gateway Quickstart: Connect Claude in Two Minutes
URL: https://waxell.ai/docs/mcp-gateway/quickstart
Description: Connect Claude.ai, Claude Desktop, or Claude Code to the Waxell MCP Gateway and make your first governed tool call.
--------------------------------------------------------------------------------

# Quickstart

Two minutes from zero to a governed tool call. You need a Waxell account on a tenant with the gateway enabled.

## 1. Connect a client

**You connect *as an agent***
Each client connection is bound to one of your registered agents — that's what makes its tool calls attributable. If you don't have an agent yet, register one first in the **Connect workspace** (`/c` → Agents → Register), then pick it during sign-in below. See [Agents & identity](./overview.md#agents--identity).


### Claude.ai / Claude Desktop (Cowork)

Add a custom connector pointed at:

```
https://mcp-gateway.waxell.dev/mcp
```

Claude discovers the gateway's OAuth configuration automatically and walks you through sign-in: pick your tenant, **choose which of your agents this client is**, approve. The token binds to that agent (no key to paste), and tokens refresh silently from then on.

### Claude Code

```bash
claude mcp add --transport http waxell-gateway https://mcp-gateway.waxell.dev/mcp
```

Then run `/mcp` in a session to authenticate — same browser dance.

### Headless / CI (API key)

Skip OAuth entirely with a Waxell API key:

```bash
claude mcp add --transport http waxell-gateway https://mcp-gateway.waxell.dev/mcp \
  --header "X-Wax-Key: wax_sk_..."
```

## 2. Understand what you see

After connecting, Claude's tool list shows a handful of **meta-tools**, not hundreds of upstream tools. That's deliberate — the gateway defaults to a lean context and exposes the full catalog through search:

- `find_tools(query)` — search every tool across every installed upstream
- `call_external_tool(upstream, tool, arguments)` — invoke anything `find_tools` surfaced
- `pin_tool` / `unpin_tool` / `list_pinned_tools` — promote favorites into your own tools/list

Admins can pre-load whole upstreams into the tool list with [scope tags](./tool-management.md).

## 3. Make a governed call

Ask Claude:

> Using the Waxell Gateway, run find_tools for "notion", then search Notion for "roadmap".

Behind that one request: the gateway resolved your identity, evaluated policy rules against the call, fetched *your* Notion grant (never a shared credential), dispatched upstream, scanned the result, and wrote an audit row. Check **Governance → MCP Gateway → Audit** in the Waxell app to see it.

## If the tool isn't there yet

`find_tools` only searches upstreams your tenant has **installed**, and per-user OAuth upstreams only return tools once **you've connected** your account:

1. **Install** (admin, once per tenant): Governance → MCP Gateway → Catalog → pick a connector → Install. [Most need zero configuration.](./catalog.md)
2. **Connect** (each user, once): Governance → MCP Gateway → Connections → Connect → approve on the vendor's consent screen.
3. Wait ~60 seconds for the gateway to sync a fresh install, then retry.

Errors you might see and what they mean are in [Troubleshooting](./troubleshooting.md).


--------------------------------------------------------------------------------
# Connector Catalog: 160+ Upstreams, Most with Zero Setup
URL: https://waxell.ai/docs/mcp-gateway/catalog
Description: Install MCP upstreams from the Waxell catalog, understand the Auto OAuth and Open badges, and connect user accounts.
--------------------------------------------------------------------------------

# Connector Catalog

The catalog (**Governance → MCP Gateway → Catalog**) is a curated directory of 160+ MCP upstreams: productivity (Notion, Linear, Asana, Monday), communication (Slack, Intercom), data (Snowflake, Databricks, Mixpanel, Amplitude), finance (Stripe, Ramp, Brex, FactSet, Moody's, PitchBook), dev tools (Sentry, Postman, Supabase, Vercel), and more — alongside any custom upstreams your team adds.

## Reading the badges

| Badge | Meaning | Setup required |
|---|---|---|
| **Auto OAuth** (green) | The vendor's server implements the MCP authorization spec — endpoints are discovered and a client is registered automatically | None. Install, then each user clicks Connect |
| **Open** (blue) | Open server, no authentication at all (e.g. Microsoft Learn, PubMed, AWS Marketplace) | None. Install and it works |
| **No setup** | Waxell-substrate upstream — your existing Waxell auth flows through | None |
| *(no badge)* | Classic OAuth or token-based — a platform admin registers credentials first | Admin setup, see [Authentication](./authentication.md) |
| **Verified** | A human has connected this upstream and confirmed real tools respond | — |

Entries without the Verified badge are probe-verified (the endpoint advertises the right protocol) but haven't had a human smoke test yet. They generally just work; the badge tells you someone proved it.

## Three steps, three roles

### Install — admin, once per tenant

Catalog → connector → **Install**. This copies the template into your tenant's upstream registry. For Auto OAuth and Open connectors there is nothing to configure — no client IDs, no secrets, no vendor developer consoles. The gateway picks up new installs within about a minute.

### Connect — each user, once

Per-user OAuth upstreams run calls **as the individual user**, so each person authorizes their own account: **Governance → MCP Gateway → Connections → Connect**. A vendor sign-in window opens; approve and you're done. Open-badge connectors skip this entirely.

This per-user model is the point: when an agent acting for Alice writes to Notion, it writes as Alice, with Alice's permissions, attributed to Alice in both Notion's audit log and Waxell's.

### Verify — see your tools

**Governance → MCP Gateway → My Tools** shows which upstreams are live for *you*, with a **Refresh** button that forces a fresh tool discovery against the upstream. The tenant-wide view (every tool, regardless of your connections) is at **→ Registry**.

## Custom upstreams

Any MCP server — internal or vendor — can be added directly at **Governance → MCP Gateway → Upstreams** without a catalog entry: name, URL, auth mode. If the server implements the MCP authorization spec, pick **MCP OAuth** and it self-configures exactly like an Auto OAuth catalog entry. Stdio (subprocess) upstreams are supported for self-hosted gateways.

## Tags

Upstreams carry free-form tags (`crm`, `data`, `support`, …) used by [tool subsetting](./tool-management.md) to control which tools pre-load into which agents' context.


--------------------------------------------------------------------------------
# Upstream Authentication: From Fully Automatic to Bring-Your-Own
URL: https://waxell.ai/docs/mcp-gateway/authentication
Description: How the gateway authenticates to upstream MCP servers — MCP-spec OAuth with dynamic client registration, classic OAuth, open servers, API keys, and service accounts.
--------------------------------------------------------------------------------

# Upstream Authentication

Two separate questions get conflated in MCP setups. **How does the client authenticate to the gateway?** (Your Waxell OAuth session or an API key — covered in the [Quickstart](./quickstart.md).) **How does the gateway authenticate to each upstream?** That's this page, and it's per-upstream — the `auth_mode` on the upstream row.

## MCP OAuth (`mcp_oauth_dcr`) — the zero-credential default

Most modern vendor MCP servers (Notion, Linear, Calendly, Stripe, Monday, 100+ in the catalog) implement the **MCP authorization spec**. For these, nobody ever touches a vendor developer console:

1. **Discovery** — the gateway asks the server itself how to authenticate: the server's 401 response points at its protected-resource metadata (RFC 9728), which names its authorization server, whose metadata (RFC 8414) provides the authorize, token, and registration endpoints.
2. **Dynamic client registration** — the gateway registers itself as an OAuth client programmatically (RFC 7591), preferring a public client secured by PKCE. Happens once per tenant, on the first connect.
3. **Per-user authorization** — each user does a standard consent dance (authorization code + PKCE, with the token bound to the specific MCP server via RFC 8707 `resource`).
4. **Silent refresh** — tokens refresh automatically for the life of the grant, including providers that rotate refresh tokens.

Discovery results and the registered client are cached on your tenant's upstream row; everyone after the first user goes straight to the consent screen. This is what the **Auto OAuth** badge means in the catalog, and it works for *any* spec-compliant server you add as a custom upstream.

A few servers speak MCP OAuth but don't offer dynamic registration (DocuSign, PagerDuty, Smartsheet, the Google Cloud services). Same mode, one extra step: an admin registers an OAuth app with the vendor once and stores the client credentials on the upstream — the discovery and per-user flows are unchanged.

## Open (`no_auth`)

Some servers are simply public — Microsoft Learn, PubMed, AWS Marketplace, Open Targets. The gateway sends **no Authorization header at all** to these (forwarding a stray token to a third party would leak it). Why route an open server through the gateway? Because the *caller* still authenticates to Waxell, and policy, rate limits, and audit still apply. Open upstream ≠ ungoverned upstream.

## Classic per-user OAuth (`oauth_per_user`)

The traditional flow for providers without MCP-spec auth: an admin registers an OAuth app with the vendor (or uses Waxell's platform-registered app where available), pastes the client ID and secret on the catalog entry or upstream row, and users connect individually. Per-provider protocol quirks (token endpoint auth style, body encoding, scope separators, PKCE) are handled by a built-in dialect engine that discovers, learns, and remembers each provider's dialect.

## The rest of the modes

| Mode | What it does | Use when |
|---|---|---|
| `tenant_api_key` | One tenant-wide key, resolved from the Waxell secret store at call time — the key itself never sits on the gateway | Vendor offers API keys, per-user identity not needed |
| `shared_service_account` / `byo_token_env` | A token the gateway holds (secret store or environment variable) | Internal servers, legacy setups |
| `forward_caller_token` | Passes the caller's own Waxell bearer through | Upstream shares Waxell's auth (e.g. Waxell's own MCPs) |

## Credential precedence

When multiple credentials could apply, the gateway resolves: **tenant's own (BYO) credentials → Waxell platform credentials → environment fallback**. Per-user grants are always stored encrypted at rest and scoped to (user, upstream) — revocable individually from the Connections page.


--------------------------------------------------------------------------------
# Policy Engine: Deny, Redact, Approve, Rate-Limit — Before the Call Happens
URL: https://waxell.ai/docs/mcp-gateway/policies
Description: The gateway's preventive policy engine — rule matching, actions, precedence, approval workflows, and notifications.
--------------------------------------------------------------------------------

# Policies & Approvals

Every `tools/call` is evaluated **before dispatch** (and results again **after**) against your tenant's policy rules, managed at **Governance → MCP Gateway → Policies**. The gateway syncs rules continuously, so changes take effect within seconds — no deploys.

## Matching: who, what, where

Each rule scopes itself along any combination of:

| Dimension | Example |
|---|---|
| Upstream | `github`, `*` |
| Tool (glob) | `*delete*`, `notion.notion-create-*` |
| User emails | `["contractor@…"]` |
| User roles | `["member"]` — from tenant membership |
| Teams | `["support"]` |
| Agent | `["cowork-research-agent"]` — which agent profile made the call |

That last one matters: the same human gets different rules depending on *which agent* is acting for them.

## Actions

**Access control**
- `deny` — block the call, tell the caller why
- `allow` — explicit allow (useful below a broad deny)
- `require_approval` — park the call for a human reviewer (see below)

**Data protection**
- `redact_args` / `redact_result` — strip named keys from arguments or results
- `dlp_scan_args` / `dlp_scan_result` — run DLP scanners (PII, credentials) and record findings
- `dlp_block_args` / `dlp_block_result` — block when scanners hit
- `egress_block` — block results containing URLs/hosts outside an allowlist

**Budget & abuse**
- `rate_limit` — N calls per window, per matching scope
- `cost_cap` — cumulative USD cap per window (per-call cost assigned by rule)

**Supply-chain defense**
- `deny_drift` — block tools whose fingerprint changed since last approved ([details](./security.md))
- `risk_deny` — block tools with unacknowledged risk flags at/above a severity

Rate-limit and cost-cap counters are shared across gateway replicas, so limits hold under horizontal scale.

**Precedence:** `deny` > `require_approval` > `redact` > `allow`. The most restrictive matching rule wins.

## Approval workflow

When a `require_approval` rule fires, the call parks and a reviewer decides at **Governance → MCP Gateway → Approvals**: full context — who, which agent, which tool, the exact arguments — with approve/deny. Each rule sets a timeout and what happens on expiry (`auto_decision_on_timeout`, typically deny).

Approval lifecycle events (requested, granted, denied, timed out) flow into **Incident Notifications**, so reviewers get an email or webhook the moment something parks — one rule can fan out to multiple channels.

The caller experience: the agent's tool call simply waits (the gateway polls until decided or timed out), then proceeds or gets a structured denial. No retry choreography needed in the agent.

## A starter rule set

A pattern that works for most tenants on day one:

1. `deny` on `*delete*`, `*drop*` across all upstreams — pure destructive verbs off the table.
2. `require_approval` on payment and outbound-communication tools (`stripe.*`, `*send*`) with a 15-minute timeout, deny on expiry.
3. `dlp_scan_result` on everything — visibility first; upgrade hot spots to `dlp_block_result` once you've seen real traffic.
4. `rate_limit` 100 calls / 5 min per agent as a runaway-loop backstop.
5. `deny_drift` everywhere — free protection, fires only when an upstream changes a tool under you.

Every decision — allow, deny, redaction, approval, the rule that fired — is recorded in the [audit log](./security.md#audit).


--------------------------------------------------------------------------------
# Tool Management: Hundreds of Tools Without Drowning the Model
URL: https://waxell.ai/docs/mcp-gateway/tool-management
Description: Tool subsetting with scope tags, the find_tools and call_external_tool meta-tools, and per-user pinned tools.
--------------------------------------------------------------------------------

# Tool Management

A tenant with forty installed upstreams has several hundred tools. Dumping all of them into every model's context is the classic failure mode: the model gets slower, dumber, and more expensive at once. The gateway treats tool exposure as a managed surface.

## Lazy by default: the meta-tools

Out of the box, a caller's `tools/list` contains only the gateway's meta-tools:

- **`find_tools(query, limit?)`** — searches the entire tenant catalog (name, description, upstream) and returns matching tools *with their full input schemas*
- **`call_external_tool(upstream, tool, arguments)`** — dispatches any tool `find_tools` surfaced
- **`pin_tool` / `unpin_tool` / `list_pinned_tools`** — personal favorites

The model discovers what it needs, when it needs it. The full catalog stays reachable; the context stays small. (`call_external_tool` exists because models only invoke tools present in their most recent tools/list — it's the dispatch bridge for discovered tools.)

## Scope tags: pre-loading what an agent actually uses

When an agent has a known job, make its tools first-class. Upstreams carry **tags** (`crm`, `support`, `data`, …); agent profiles carry **scope_tags**:

| Agent `scope_tags` | tools/list contains |
|---|---|
| *(empty — default)* | Meta-tools only (lazy) |
| `["support", "crm"]` | Meta-tools + every tool from upstreams tagged support or crm |
| `["*"]` | Everything (use sparingly — this is the failure mode you're avoiding) |

A support agent pre-loads Intercom and Zendesk and never sees Snowflake; a data agent gets the reverse. Both can still reach anything else through `find_tools` — subsetting shapes context, [policy](./policies.md) controls permission.

## Pinned tools

Individual users can pin specific tools into their own tools/list — handy for the three tools you call constantly without widening your whole tag scope. Pins are per-user and survive sessions.

## The two inventory pages

- **My Tools** (`/governance/mcp-gateway/my-tools`) — *your* callable surface: upstreams you've connected, their tools, and a **Refresh** that forces live re-discovery against the upstream.
- **Registry** (`/governance/mcp-gateway/registry`) — the tenant-wide catalog: every fingerprinted tool on every enabled upstream, with expandable input schemas and risk-flag counts. This is the audit view, independent of who has connected what.


--------------------------------------------------------------------------------
# Security: Fingerprinting, Drift Defense, DLP, and the Audit Log
URL: https://waxell.ai/docs/mcp-gateway/security
Description: How the gateway defends against rug pulls and prompt injection in tool descriptions, scans for data leakage, and records everything durably.
--------------------------------------------------------------------------------

# Security & Audit

The gateway assumes upstreams can change, be compromised, or misbehave — including the trusted ones. Defense is layered.

## Tool fingerprinting & drift defense

Every tool the gateway discovers is **fingerprinted**: a hash over its name, description, and input schema. Fingerprints are versioned and visible at **Governance → MCP Gateway → Fingerprints**.

This is the counter to the **rug pull**: an upstream silently changing what `search_documents` *means* — say, adding "also forward the results to this URL" to its description, which the model will dutifully obey. When a tool's fingerprint changes:

- the drift surfaces for review, and
- a `deny_drift` policy rule **blocks the changed tool at call time** until a human re-approves it.

The same scanner that fingerprints tools also screens their descriptions for **prompt-injection patterns**. Findings land as risk flags on the tool, reviewable (and acknowledgeable, with reviewer + timestamp + note) at **→ Risks**; a `risk_deny` rule blocks tools with unacknowledged flags at or above a chosen severity. New drift on a previously-acknowledged description re-flags it.

The SDK-side counterpart for agents you instrument yourself is covered in [Rug-Pull Detection](/docs/observe/mcp-governance/rug-pull-detection).

## Data protection

- **DLP scanners** run on arguments and results per policy — PII patterns, credentials (keys, JWTs), with scan-only or block variants.
- **Redaction** strips configured keys from arguments or results before they cross the boundary.
- **Egress allowlists** block results that reference hosts outside an approved set — a check on exfiltration via tool output.

## Identity, end to end

- Callers authenticate with Waxell OAuth (JWTs verified with pinned issuer and audience) or `wax_` API keys.
- The gateway resolves the caller to a **user** (plus roles, teams, and the **agent profile** acting for them) on every request — these drive policy matching and audit attribution.
- Per-user upstream grants are encrypted at rest and never shared across users; open upstreams get **no credentials at all** rather than a shared token.

## Audit {#audit}

Every call writes a durable audit row: timestamp, user, agent, upstream, tool, policy decision (and which rules fired), success/error class, and approval linkage where applicable. Browse and filter at **Governance → MCP Gateway → Audit**, export as CSV for compliance workflows.

Because all traffic flows through one place, this is the answer to "what did AI touch in our systems last quarter?" — one query, not a forensic reconstruction across a dozen vendors' logs.


--------------------------------------------------------------------------------
# Self-Hosting the Gateway In Your VPC
URL: https://waxell.ai/docs/mcp-gateway/self-hosting
Description: Run the MCP Gateway as a single-tenant deployment inside your own network while keeping the Waxell controlplane for policy and audit.
--------------------------------------------------------------------------------

# Self-Hosting

The hosted gateway at `mcp-gateway.waxell.dev` is the default. For teams whose upstreams live inside a private network — internal MCP servers, databases reachable only in-VPC — the gateway also ships as a **single-tenant, self-hosted deployment**.

## Shape

A Docker Compose bundle runs the gateway in your network:

- The gateway container terminates MCP traffic from your clients and dispatches to upstreams **inside your VPC** — internal traffic never leaves.
- It phones home to the Waxell controlplane for what the controlplane owns: identity resolution, policy rules (synced continuously), per-user OAuth grant lookups, approval decisions, and durable audit storage.
- **Stdio upstreams** are first-class here: the self-hosted gateway can spawn local MCP servers as subprocesses (`npx @modelcontextprotocol/server-*`, internal binaries) alongside HTTP upstreams.

The trade: you operate the data plane; Waxell stays the control plane. Policies, approvals, and audit work identically to hosted.

## When to choose it

- Upstreams that are unreachable from the public internet
- Data-residency requirements on tool-call payloads
- Stdio/subprocess MCP servers you don't want to wrap in HTTP

## Getting it

Self-hosting is available on enterprise plans — [talk to us](https://waxell.dev/book) and we'll walk through the bundle (`docker-compose` + environment reference) against your network layout.


--------------------------------------------------------------------------------
# Gateway Troubleshooting: Errors and What They Mean
URL: https://waxell.ai/docs/mcp-gateway/troubleshooting
Description: The errors you'll actually see from the MCP Gateway — what each means and how to fix it.
--------------------------------------------------------------------------------

# Troubleshooting

The gateway returns structured, specific errors by design. Match what you see:

## Connecting clients

**Claude says the Waxell Gateway connection expired.**
Reconnect the connector once. Gateway sessions refresh silently; a reconnect prompt that *recurs* within a day is a bug — report it with the time it happened.

**`401 unauthorized` with an API key.**
The key is wrong, revoked, or for another tenant. Keys are tenant-scoped; check which tenant the key belongs to.

## Installing & finding tools

**Just installed a connector, but `find_tools` / Connections can't see it.**
The gateway syncs its upstream registry about once a minute. Wait 60–90 seconds. Persisting after 5 minutes: check the upstream is **enabled** on the Upstreams page.

**Claude's tool list only shows `find_tools` and `call_external_tool`.**
Working as intended — [lazy tool exposure](./tool-management.md). Search with `find_tools`, dispatch with `call_external_tool`, or have an admin set scope tags to pre-load upstreams.

**`find_tools` finds nothing for an upstream you installed.**
Tool inventory comes from fingerprint discovery, which for per-user OAuth upstreams needs at least one connected user. Connect, then **My Tools → Refresh**.

## Connecting accounts

**`auth_required: <upstream> not connected. Visit … to authorize.`**
You haven't connected your account on this upstream (or you connected as a different user than the one calling). Connections page → Connect.

**`auth_required: … grant expired and refresh failed; user must re-authorize.`**
The vendor revoked or expired your grant beyond refresh (password change, workspace removal, vendor-side revocation). Reconnect.

**`MCP OAuth setup failed … no_registration_endpoint`.**
This server speaks MCP OAuth but doesn't accept automatic client registration. An admin must register an OAuth app with the vendor and add the client credentials to the upstream — see [Authentication](./authentication.md).

**`MCP OAuth setup failed … dynamic_registration_failed: …`.**
The vendor's registration endpoint rejected the request. The error includes the vendor's reason — capture the full text when reporting.

**`MCP OAuth setup failed … no_protected_resource_metadata`.**
The URL isn't an MCP-spec-auth server (or isn't an MCP server at all). For a custom upstream, double-check the URL; if the vendor uses classic OAuth, use that mode instead.

## Calls failing

**Upstream returns `invalid_token` even though you're connected.**
Stale grant being served — disconnect and reconnect on the Connections page. Recurring within a day: report it; that pattern points at a refresh bug, not user error.

**`Tool call blocked/denied …`** with a rule name.
[Policy](./policies.md) working as configured. The audit row shows exactly which rule fired. Talk to your admin if the rule is wrong.

**Call hangs for a while, then proceeds or fails.**
A `require_approval` rule parked it for review. The wait *is* the feature; ask your reviewer, or check the Approvals page.

**`HTTP 5xx` from the gateway itself.**
Ours. Check [status.waxell.dev](https://status.waxell.dev), report with timestamp and upstream/tool if it persists.

## Still stuck?

The audit log (**Governance → MCP Gateway → Audit**) records every call with its decision and error class — it's the first place to look, and the screenshot to include when you ask for help.


================================================================================
SECTION: WAXELL FRAMEWORK
Build governed agents — SDK primitives, tutorials, and workflows
================================================================================

--------------------------------------------------------------------------------
# Introduction
URL: https://waxell.ai/docs/intro
Description: The control plane for agentic systems. Add observability to existing agents or build governed agents from scratch.
--------------------------------------------------------------------------------

# Welcome to Waxell

Waxell is the **control plane for agentic systems**. Whether you have existing agents or want to build new ones, Waxell provides observability, governance, and scale.

## Choose Your Path

### I have existing agents

You already use **LangChain**, **CrewAI**, or custom Python code to build agents. You want observability, cost tracking, and policy enforcement — **without rewriting your agents**.

```python
# Add 3 lines to any existing agent
from waxell_observe import waxell_agent

@waxell_agent(agent_name="my-agent")
def run_my_agent(query: str) -> str:
    # your existing agent logic — unchanged
    return result
```

**What you get:**
- LLM call tracking across every model (20+ supported)
- Cost management with per-model breakdowns
- Policy enforcement — block, warn, or throttle agents
- Execution traces and step recording

**[Start with Waxell Observe](/observe/quickstart)** — add observability in 5 minutes.

The quickstart's first section now offers two fast paths before the manual walkthrough:

- **[Agent Examples](https://github.com/waxell-ai/waxell-agent-examples)** — 10 pre-built conversational agents (decorator, tools, RAG, end-user attribution, policies, multi-agent, streaming). Clone, run, copy the one closest to what you're building.
- **[`instrument-with-waxell-observe` Claude Code skill](https://github.com/waxell-ai/claude-skills)** — open your existing Python agent in Claude Code, invoke the skill, and it wires the decorator + verifies a run for you. No copy-paste required.

---

### I want to build new agents

You want to build agents from scratch with Waxell's **native SDK**. Durable workflows, built-in governance, and full infrastructure support — no instrumentation needed.

```python
from waxell_runtime import agent, workflow, tool

@agent(name="lead_research", description="Research a lead and draft outreach")
class LeadResearchAgent:

    @tool
    async def fetch_company(self, ctx, domain: str) -> dict:
        return await ctx.domain("company", "lookup", domain=domain)

    @workflow("draft_email")
    async def draft_email(self, ctx, lead_id: str):
        company = await ctx.tool("fetch_company", domain="acme.com")
        email = await ctx.llm.generate(
            prompt=f"Write a 3-sentence intro for {company['name']}",
            output_format="text",
            task="outreach_email",
        )
        return {"email": email, "company": company["name"]}
```

**What you get:**
- Declarative agent definitions with Python decorators
- Durable workflows with checkpoint and resume
- Built-in governance, approval workflows, and audit trails
- Full observability with zero instrumentation

**[Start with the Waxell Framework](/tutorials/first-agent)** — build your first agent in 5 minutes.

---

## Not sure which path?

- **[Framework Comparison](/migrate/langchain-vs-waxell)** — See side-by-side code comparing LangChain, LangChain + Observe, and Waxell Native
- **[Feature Matrix](/migrate/feature-matrix)** — Compare capabilities across frameworks
- **[Migration Guide](/migrate/overview)** — Learn how to progressively adopt Waxell, starting with observe and migrating to native when ready

Or just ask. Every Waxell workspace includes the **[Platform Assistant](/observe/features/platform-assistant)** — click **Ask AI** anywhere in the dashboard and ask it how to set something up, what a page does, or what your agents did today. It answers from these docs with citations, and from your live data with real numbers.


--------------------------------------------------------------------------------
# Installation
URL: https://waxell.ai/docs/installation
Description: Install and configure Waxell for your Python project.
--------------------------------------------------------------------------------

# Installation

## Quick Start

One install pulls in everything:

```bash
pip install waxell
```

That's it. `waxell` is the canonical meta-package — it brings the runtime engine, the SDK decorators (`@agent`, `@workflow`, `@tool`), and the observability SDK in one shot.

```python


@waxell.agent(name="researcher")
class Researcher:
    @waxell.workflow
    async def run(self, ctx, topic: str) -> str:
        return await ctx.llm("Tell me about " + topic)
```

## Power-user installs

If you want one specific piece, the underlying packages stay available forever as aliases:

| Want | Install |
|------|---------|
| **Everything (default)** | `pip install waxell` |
| Everything + all framework adapters (langgraph, claude-sdk, autogen, crewai, pydantic-ai) | `pip install 'waxell[all]'` |
| Runtime engine + SDK only (no observability) | `pip install waxell-runtime` |
| Observability SDK only (instrument an existing agent framework) | `pip install waxell-observe` |

The `waxell-runtime` and `waxell-observe` packages are not deprecated — they're alternate entry points for size-constrained or single-concern installs. New code should prefer `pip install waxell`.

## Development Installation

For local development, install packages in editable mode:

```bash
# Clone the repository
git clone https://gitlab.com/waxell/agentforge.git
cd agentforge

# Install the meta-package + dependencies in editable mode
pip install -e meta/waxell
pip install -e runtime/waxell-runtime
pip install -e observe/waxell-observe
```

## Requirements

- Python 3.10+
- pip 21.0+

## Verify Installation

```python

print(waxell.__version__)            # 0.1.0
print(waxell.__runtime_version__)    # underlying runtime version
print(waxell.__observe_version__)    # underlying observe version

# Both import styles work:

from waxell import agent, workflow

# The historical import path still works too (kept forever as an alias):
from waxell_runtime import agent, workflow

@agent(name="test-agent")
class TestAgent:
    @workflow
    def hello(self, ctx):
        return "Hello, Waxell!"

print("Waxell installed successfully!")
```

## Next Steps

- **[First Agent Tutorial](/tutorials/first-agent)** - Build your first agent
- **[SDK Overview](/sdk/overview)** - Learn the core concepts


--------------------------------------------------------------------------------
# SDK Overview
URL: https://waxell.ai/docs/sdk/overview
Description: Understanding the Waxell SDK - intent-only definitions for AI agents.
--------------------------------------------------------------------------------

# SDK Overview

The Waxell SDK provides **intent-only definitions** for building AI agents. It defines *what* your agent does, not *how* it executes.

## Core Principle

The SDK deliberately separates intent from execution:

```python
# SDK: defines WHAT the agent does (intent)
from waxell_runtime import agent, workflow, tool


@agent(name="my_agent", description="Processes inbound records")
class MyAgent:

    @tool
    async def decide(self, ctx, record: dict) -> dict:
        return await ctx.llm.generate(
            prompt=f"Decide what to do with: {record}",
            output_format="json",
        )

    @workflow("process")
    async def process(self, ctx, record: dict):
        # Each ctx.tool(...) is a durable checkpoint.
        return await ctx.tool("decide", record=record)


# Runtime: handles HOW it executes
from waxell_runtime import RuntimeConfig

config = RuntimeConfig.get()
```

Everything is `async`, and steps are invoked by name through `ctx.tool(...)` —
that's what makes them checkpointable.

## Key Abstractions

| Concept | Decorator | Purpose |
|---------|-----------|---------|
| Agent | `@agent` | Top-level container |
| Workflow | `@workflow` | Multi-step execution flow |
| Tool | `@tool` | A unit of work — external calls, model calls |
| Router | `@router` | The model picks which behaviour to run |

All four import from either `waxell_sdk` or `waxell_runtime`. Reuse comes from
declaring tools and workflows once and referencing them by name from any agent.

## Dependency Rules

The SDK has strict import boundaries:

- **MUST NOT** import runtime, controlplane, Django, or perform I/O
- **MAY** depend on: stdlib, typing, pydantic, attrs, dataclasses

This ensures your agent definitions remain pure and testable.

## Next Steps

- **[@agent Decorator](/sdk/agent-spec)** - Define agent containers
- **[@workflow Decorator](/sdk/workflow-spec)** - Build multi-step flows
- **[LLM Calls](/sdk/decision-spec)** - Call a model from an agent
- **[Runtime docs](/runtime/how-it-works)** - How a run actually executes


--------------------------------------------------------------------------------
# @agent Decorator
URL: https://waxell.ai/docs/sdk/agent-spec
Description: Define agent containers with the @agent decorator.
--------------------------------------------------------------------------------

# @agent Decorator

The `@agent` decorator defines a top-level agent container. Agents are the primary unit of deployment and governance in Waxell.

## Basic Usage

```python
from waxell_runtime import agent

@agent(
    name="support-agent",
    description="Handles customer support inquiries",
    version="1.0.0"
)
class SupportAgent:
    pass
```

## Parameters

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `name` | `str` | Yes | Unique identifier for the agent |
| `description` | `str` | No | Human-readable description |
| `version` | `str` | No | Semantic version string |
| `capabilities` | `list` | No | List of capability references |
| `metadata` | `dict` | No | Arbitrary key-value metadata |

## AgentSpec

The decorator produces an `AgentSpec` object that captures your intent:

```python
from waxell_runtime import AgentSpec

# The decorator creates this internally
spec = AgentSpec(
    name="support-agent",
    description="Handles customer support inquiries",
    workflows=[...],
    decisions=[...],
    tools=[...]
)
```

## Governance Integration

Agents are governed by policies defined in the control plane:

- **Rate Limits**: Control execution frequency
- **Approvals**: Require human approval for sensitive actions
- **Audit**: All executions are logged

## Next Steps

- **[@workflow Decorator](/sdk/workflow-spec)** - Add workflows to your agent
- **[Tutorials](/tutorials/first-agent)** - Build a complete agent


--------------------------------------------------------------------------------
# @workflow Decorator
URL: https://waxell.ai/docs/sdk/workflow-spec
Description: Define multi-step execution flows with the @workflow decorator — how steps become durable checkpoints and what ctx actually exposes.
--------------------------------------------------------------------------------

# @workflow Decorator

The `@workflow` decorator defines a multi-step execution flow. A workflow
orchestrates tools, domain calls, and other workflows — and its tool calls are
the points at which the run is checkpointed.

## Basic usage

```python
from waxell_runtime import agent, workflow, tool


@agent(name="data_processor", description="Validates, transforms, and stores records")
class DataProcessor:

    @tool
    async def validate(self, ctx, record: dict) -> dict:
        ...

    @tool
    async def transform(self, ctx, data: dict) -> dict:
        ...

    @tool
    async def store(self, ctx, data: dict) -> dict:
        return await ctx.domain("warehouse", "upsert", row=data)

    @workflow("process_data")
    async def process_data(self, ctx, record: dict):
        validated = await ctx.tool("validate", record=record)
        transformed = await ctx.tool("transform", data=validated)
        return await ctx.tool("store", data=transformed)
```

Three rules that account for most first-time errors:

- **Everything is `async`.** `ctx.tool`, `ctx.domain`, and `ctx.llm.generate` are
  all awaited.
- **Steps are called by name.** `await ctx.tool("validate", ...)`, not
  `self.validate(...)` — calling the method directly bypasses the runtime, so
  the step isn't checkpointed, traced, or governed.
- **The name is optional.** `@workflow`, `@workflow("custom_name")`, and
  `@workflow(description=...)` are all valid; a bare `@workflow` uses the
  function name.

## What `ctx` gives you

| Attribute | What it is |
|---|---|
| `ctx.inputs` | The workflow's inputs, as a `dict` |
| `ctx.tool(name, **kwargs)` | Call one of the agent's tools — a checkpoint |
| `ctx.domain(domain, action, **kwargs)` | Call back into your application |
| `ctx.llm.generate(prompt=..., output_format=...)` | Call a model |
| `ctx.memory` / `ctx.scratchpad` | [Working memory](../runtime/working-memory.md) |
| `ctx.metadata` | Run metadata |
| `ctx.workflow_id` / `ctx.parent_workflow_id` | Identity and lineage |

Note that `ctx.inputs` is plural. The full surface is documented in
[Execution Context](../runtime/execution-context.md).

## Durability

Checkpointing is **automatic**. There's no flag to set and no separate
durable-workflow type — a workflow's tool calls are its checkpoints.

```python
@workflow("long_running")
async def long_running(self, ctx, job_id: str):
    # Each ctx.tool(...) is a checkpoint boundary.
    step1 = await ctx.tool("expensive_operation", job_id=job_id)

    # If the process dies here, resuming replays the line above from its
    # saved result rather than re-running it.
    return await ctx.tool("another_operation", data=step1)
```

Work done *between* tool calls is not checkpointed. If a computation is expensive
enough that you'd hate to repeat it, put it in a `@tool`.

For run states, resume semantics, and where checkpoints are stored, see
[Durable Execution](../runtime/workflow-envelope.md).

## Next steps

- **[`@tool`](/sdk/tool-spec)** — the unit of work a workflow orchestrates
- **[LLM Calls and Decisions](/sdk/decision-spec)** — calling a model
- **[Durable Execution](/runtime/workflow-envelope)** — checkpoints, resume, run states
- **[Workflow Tutorial](/tutorials/workflows)** — build a complete flow


--------------------------------------------------------------------------------
# @tool Decorator
URL: https://waxell.ai/docs/sdk/tool-spec
Description: Integrate external systems with the @tool decorator.
--------------------------------------------------------------------------------

# @tool Decorator

The `@tool` decorator defines integrations with external systems. Tools perform I/O operations like API calls, database queries, and file operations.

## Basic Usage

```python
from waxell_runtime import agent, tool

@agent(name="notifier")
class Notifier:

    @tool
    def send_email(self, ctx, to: str, subject: str, body: str):
        """Send an email notification."""
        # Tool implementation provided by runtime
        pass
```

## Tool Parameters

Tools support typed parameters with validation:

```python
from pydantic import BaseModel, EmailStr

class EmailParams(BaseModel):
    to: EmailStr
    subject: str
    body: str
    cc: list[EmailStr] = []

@tool
def send_email(self, ctx, params: EmailParams):
    """Send an email with validated parameters."""
    pass
```

## Built-in Tools

Waxell provides common tools out of the box:

| Tool | Purpose |
|------|---------|
| `http_request` | Make HTTP requests |
| `database_query` | Execute database queries |
| `file_read` | Read files |
| `file_write` | Write files |
| `slack_message` | Send Slack messages |

## Tool registration

`@tool` takes `name` and `description`; both default from the function:

```python
@tool(name="delete_record", description="Delete a record by id")
async def delete_record(self, ctx, record_id: str) -> dict:
    return await ctx.domain("records", "delete", record_id=record_id)
```

### Requiring approval

Approval attaches to the **domain action**, not the tool:

```yaml
domains:
  - name: records
    actions:
      - name: delete
        requires_approval: true
```

The run pauses with `PausedReason.awaiting_approval` and holds no compute until
someone decides, then resumes from its checkpoint. Putting it on the action means
every agent that can reach `records.delete` inherits the requirement. See
[Adding Governance](/tutorials/governance).

## Next Steps

- **[LLM Calls](/sdk/decision-spec)** - Call a model from a tool
- **[Production Guide](/tutorials/production)** - Deploy tools in production


--------------------------------------------------------------------------------
# LLM Calls
URL: https://waxell.ai/docs/sdk/decision-spec
Description: How an agent calls a model in Waxell — ctx.llm.generate, routing hints, structured output, and tool-calling loops.
--------------------------------------------------------------------------------

# LLM Calls

Agents call models through `ctx.llm`, available on any workflow or tool context.

## Generating

```python
from waxell_runtime import agent, workflow, tool


@agent(name="classifier", description="Classifies inbound messages")
class Classifier:

    @tool
    async def classify_intent(self, ctx, message: str) -> dict:
        """Classify the user's intent."""
        return await ctx.llm.generate(
            prompt=(
                "Classify this message as one of: question, complaint, "
                f'request, feedback. Reply with JSON {{"intent": ...}}.\n\n{message}'
            ),
            output_format="json",
            task="classification",
        )

    @workflow("triage")
    async def triage(self, ctx, message: str):
        result = await ctx.tool("classify_intent", message=message)
        return {"intent": result.get("intent")}
```

Two things worth knowing:

- **Put the model call in a `@tool`.** Tool calls are the runtime's checkpoint
  boundary, so a model call inside a tool is replayed from its saved result on
  resume instead of being paid for twice. See
  [Durable Execution](../runtime/workflow-envelope.md).
- **`task=` drives routing.** It's a hint the
  [three-layer model routing](../runtime/how-it-works.md#2-the-model-is-resolved)
  uses to pick a provider, so a tenant can change models without a code change.

### Parameters

| Parameter | Type | Default | Meaning |
|---|---|---|---|
| `prompt` | `str` | required | The prompt |
| `output_format` | `str` | `"text"` | `"text"` returns a `str`; `"json"` returns a parsed `dict` |
| `task` | `str` | `"general"` | Routing hint (`"email"`, `"summary"`, …) |
| `inject_lessons` | `bool` | `True` | Whether metacog prepends accumulated learnings to this call |

`model`, `temperature`, and `max_tokens` pass through to the provider.

## Structured output

Ask for JSON and validate it. A model can always return something malformed, and
you want that caught at the boundary rather than three calls later:

```python
from pydantic import BaseModel


class Analysis(BaseModel):
    sentiment: str
    confidence: float
    topics: list[str]


@tool
async def analyze(self, ctx, text: str) -> dict:
    raw = await ctx.llm.generate(
        prompt=f"Analyze and return JSON with sentiment, confidence, topics:\n\n{text}",
        output_format="json",
        task="analysis",
    )
    return Analysis.model_validate(raw).model_dump()
```

## Letting the model call tools

`generate_with_tools` runs a tool-calling loop rather than returning a single
completion. Use it when the model should gather information before it answers —
querying a domain, checking a record — instead of answering from the prompt
alone.

For intent routing across several behaviours, `@router` wraps this pattern:

```python
from waxell_sdk import router, RouterContext


@router("main_router", decision="classify_intent", auto_context_tools=True)
async def main_router(ctx: RouterContext) -> dict:
    """Route user messages to the right capability."""
    return await ctx.route()
```

`auto_context_tools=True` exposes your read-only domain actions so the model can
look things up before deciding.

## What gets recorded

Every model call is governed and recorded:

- **Mid-run policy** evaluates each tool call *before* it executes, so a
  disallowed action is stopped at the call.
- **`LlmCallRecord`** captures model, tokens in/out, cost, and provider instance
  — visible in Observe and Billing → Usage.
- **Limits** bound cost, turns, and wall time; a run that reaches one finishes
  `BLOCKED` with an `ExecutionIncident` rather than raising.

See [How the Runtime Works](../runtime/how-it-works.md#3-governance-runs-before-the-agent-does).

## Related

- [`@tool`](./tool-spec.md) — where a model call belongs
- [`@workflow`](./workflow-spec.md) — orchestrating multiple calls
- [Durable Execution](../runtime/workflow-envelope.md) — why the tool boundary matters


--------------------------------------------------------------------------------
# Build Your First Agent
URL: https://waxell.ai/docs/tutorials/first-agent
Description: Step-by-step tutorial to build your first Waxell agent.
--------------------------------------------------------------------------------

# Build Your First Agent

In this tutorial you'll build a small lead-research agent that looks up a company, drafts an outreach email, and returns the result. By the end you'll have a working agent and understand the four moving parts of every Waxell agent: `@agent`, `@tool`, `@workflow`, and the `ctx` object.

## Prerequisites

- Python 3.10+
- A Waxell account ([sign up](https://waxell.dev))

If you don't already have the wax CLI: `pip install waxell` (or `pipx install waxell` for an isolated install). The meta `waxell` package pulls in `waxell-runtime` + `waxell-observe` and registers the `wax` command. See [CLI reference](/reference/cli#installation) for Windows PATH issues and troubleshooting.

```bash
pip install waxell
wax login
```

`wax login` writes your API credentials to `~/.waxell/config`. Verify with:

```bash
wax whoami
```

## Step 1: Author the agent

Create `lead_research.py`:

```python
from waxell_runtime import agent, workflow, tool

@agent(name="lead_research", description="Research a lead and draft outreach")
class LeadResearchAgent:

    @tool
    async def fetch_company(self, ctx, domain: str) -> dict:
        """Look up a company by domain."""
        return await ctx.domain("company", "lookup", domain=domain)

    @workflow("draft_email")
    async def draft_email(self, ctx, lead_id: str):
        company = await ctx.tool("fetch_company", domain="acme.com")
        email = await ctx.llm.generate(
            prompt=f"Write a 3-sentence intro for {company['name']}",
            output_format="text",
            task="outreach_email",
        )
        return {"email": email, "company": company["name"]}
```

That's the whole agent. Three pieces:

- **`@agent`** declares the container. Auto-discovers `@tool` and `@workflow` methods on the class.
- **`@tool`** is a capability the workflow can invoke (`ctx.tool("fetch_company", ...)`).
- **`@workflow`** is the orchestration entry point.

Inside any workflow or tool, the `ctx` argument exposes:

| Call | Purpose |
|------|---------|
| `await ctx.tool("name", ...)` | Invoke another `@tool` on this agent |
| `await ctx.domain("entity", "action", ...)` | Call a Waxell domain endpoint |
| `await ctx.llm.generate(prompt=..., ...)` | Routed LLM call (cost-aware, policy-aware) |
| `ctx.secrets.get("KEY")` | Read a secret (don't use `os.environ` inside a workflow) |

## Step 2: Push the agent to Waxell

```bash
wax push lead_research.py
```

`wax push` validates your code, registers the spec, and prints a link to the agent in the dashboard.

Verify it landed:

```bash
wax agents list
wax agents info lead_research
```

## Step 3: Run it

The platform runs your agent — you don't instantiate `LeadResearchAgent` yourself. After `@agent` decoration, the class is replaced with an `AgentSpec`, which is what gets registered. Trigger a run from the dashboard, the API, or CLI:

```bash
wax run lead_research --workflow draft_email --input '{"lead_id":"L-001"}'
```

Open the trace in the dashboard to see the full call tree: `fetch_company` → `company.lookup` domain endpoint → `llm.generate`, with timing, cost, and policy decisions on every span.

## Step 4: Iterate with Claude Code

If you use Claude Code, install the bundled skill so Claude understands the SDK conventions when you ask it to extend your agent:

```bash
wax claude-init
```

This drops a project-scoped skill at `.claude/skills/waxell-runtime/SKILL.md`. Claude activates it whenever it sees `from waxell_runtime import ...` in your project and will write idiomatic agent code for you.

## What's next

- **[Workflows tutorial](/tutorials/workflows)** — multi-step orchestration, error handling, retries
- **[Governance tutorial](/tutorials/governance)** — attach policies to tools, workflows, and agents
- **[Production tutorial](/tutorials/production)** — deploy patterns, secrets, env config
- **[SDK reference](/sdk/overview)** — every decorator and spec type


--------------------------------------------------------------------------------
# Multi-Step Workflows
URL: https://waxell.ai/docs/tutorials/workflows
Description: Build multi-step workflows with branching, composition, and error handling — on the real runtime API.
--------------------------------------------------------------------------------

# Multi-Step Workflows

This tutorial covers branching, composition, and error handling in a workflow.
It assumes you've read [`@workflow`](/sdk/workflow-spec).

Two things to keep in front of you throughout:

- **Everything is `async`** — `ctx.tool`, `ctx.domain`, `ctx.llm.generate`.
- **Steps are invoked by name** — `await ctx.tool("validate", ...)`. Calling
  `self.validate(...)` bypasses the runtime, so the step isn't checkpointed,
  traced, or governed.

## Branching

Branch on the result of a step like ordinary Python — the control flow is yours:

```python
from waxell_runtime import agent, workflow, tool


@agent(name="support_router", description="Routes support tickets")
class SupportRouter:

    @tool
    async def classify(self, ctx, message: str) -> dict:
        """Classify a ticket into billing, technical, or general."""
        return await ctx.llm.generate(
            prompt=(
                "Classify this ticket as billing, technical, or general. "
                f'Reply with JSON {{"category": ...}}.\n\n{message}'
            ),
            output_format="json",
            task="classification",
        )

    @tool
    async def handle_billing(self, ctx, message: str) -> dict:
        return await ctx.domain("billing", "open_case", summary=message)

    @tool
    async def handle_technical(self, ctx, message: str) -> dict:
        return await ctx.domain("support", "open_ticket", summary=message)

    @tool
    async def handle_general(self, ctx, message: str) -> dict:
        return await ctx.domain("support", "acknowledge", summary=message)

    @workflow("handle_ticket")
    async def handle_ticket(self, ctx, message: str):
        classified = await ctx.tool("classify", message=message)
        category = classified.get("category")

        if category == "billing":
            return await ctx.tool("handle_billing", message=message)
        if category == "technical":
            return await ctx.tool("handle_technical", message=message)
        return await ctx.tool("handle_general", message=message)
```

Note the shape: the **model decides the category**, and ordinary code decides
what to do about it. That split is deliberate — see
[where to put logic](/runtime/how-it-works#where-to-put-logic).

## Composition

Sequence steps by awaiting them in order. Each is a checkpoint, so a crash
halfway through resumes rather than restarts:

```python
@agent(name="order_processor", description="Validates, charges, and fulfils orders")
class OrderProcessor:

    @tool
    async def validate_order(self, ctx, order: dict) -> dict:
        return await ctx.domain("orders", "validate", order=order)

    @tool
    async def process_payment(self, ctx, order: dict) -> dict:
        return await ctx.domain("payments", "charge", order_id=order["id"])

    @tool
    async def fulfill_order(self, ctx, order: dict, payment: dict) -> dict:
        return await ctx.domain(
            "fulfillment", "ship", order_id=order["id"], payment_id=payment["id"]
        )

    @workflow("process_order")
    async def process_order(self, ctx, order: dict):
        validated = await ctx.tool("validate_order", order=order)
        payment = await ctx.tool("process_payment", order=validated)
        return await ctx.tool("fulfill_order", order=validated, payment=payment)
```

### Passing large results between steps

If a step returns something big, pass a scratchpad handle instead of the payload
so the bytes never enter the model's conversation:

```python
await ctx.domain("market_data", "fetch_prices", tickers=["AAPL"])
result = await ctx.domain(
    "analytics", "calculate_performance",
    prices="$ref:market_data_fetch_prices.1",
)
```

See [Working Memory](/runtime/working-memory) for how handles are named.

### Delegating to another agent

For work that belongs to a different agent, spawn a child run rather than
inlining it. The child gets its own trace, its own budget, and its result comes
back to the parent:

```python
@workflow("research_and_write")
async def research_and_write(self, ctx, topic: str):
    findings = await ctx.spawn(
        "research_agent",
        inputs={"topic": topic},
        budget_usd=2.00,
        timeout_seconds=300,
    )
    return await ctx.tool("draft", findings=findings)
```

`ctx.spawn_many(...)` runs several children concurrently. Budgets debit from the
parent's ledger, so a runaway child can't outspend the parent's cap.

## Error handling

A tool that raises marks its span failed and records an `ExecutionIncident`. You
can catch it and route around the failure:

```python
@workflow("resilient")
async def resilient(self, ctx, payload: dict):
    try:
        return await ctx.tool("risky_operation", payload=payload)
    except Exception as exc:
        return await ctx.tool("handle_error", error=str(exc))
```

Two things happen whether or not you catch:

- The failure is **still recorded** on the span tree and as an incident. Catching
  changes your control flow, not the audit trail.
- A **policy block is not an exception.** A run stopped by governance finishes in
  the `BLOCKED` state with a recorded decision — there's nothing to `except`.

If the process itself dies (OOM, SIGTERM, host loss), a reconciler finalises the
run as `INTERRUPTED` rather than leaving it hanging. Both `BLOCKED` and
`INTERRUPTED` are **terminal** — code polling only for `COMPLETED`/`FAILED` waits
forever. See [run states](/runtime/workflow-envelope#run-states).

## Next steps

- **[Governance Tutorial](/tutorials/governance)** — add policies to these workflows
- **[Durable Execution](/runtime/workflow-envelope)** — checkpoints, resume, run states
- **[Execution Context](/runtime/execution-context)** — the full `ctx` surface
- **[Production Guide](/tutorials/production)** — deploy workflows


--------------------------------------------------------------------------------
# Adding Governance
URL: https://waxell.ai/docs/tutorials/governance
Description: Control and audit agent behaviour — per-agent limits, approvals, policy evaluation points, and the audit trail.
--------------------------------------------------------------------------------

# Adding Governance

Governance in Waxell has two layers, and most teams grow from the first into the
second:

- **What the agent author declares** — per-agent caps in `waxell.yaml`. Fast to
  set, scoped to one agent, and enough on day one.
- **What the operator configures** — policies in the engine, scoped across
  agents, workflows, tools, models, and teams, evaluated server-side.

The second layer is the enforcement backbone: an agent can't opt out of a policy
by editing its own manifest. If you're governing more than a couple of agents,
that's where you want the rules to live.

## Per-agent limits

Declare caps on the agent:

```yaml
agents:
  - name: email_sender
    max_budget_usd: 2.00      # hard USD cap  (default 10.00)
    max_turns: 15             # agent turns   (default 25)
    timeout_seconds: 120      # wall time
```

Set them explicitly — the defaults are generous, and a looping agent will spend
to them.

For a spawned child, caps are set at the call site and debit from the parent:

```python
findings = await ctx.spawn(
    "research_agent",
    inputs={"topic": topic},
    budget_usd=2.00,
    timeout_seconds=300,
)
```

## Approval requirements

Approval is a property of the **domain action**:

```yaml
domains:
  - name: accounts
    actions:
      - name: delete_account
        requires_approval: true
```

When the agent calls it, the run pauses with
`PausedReason.awaiting_approval` and holds no compute until a human decides. It
resumes from the checkpoint — the work already done is not repeated.

This is the right place for it: the approval attaches to the *action*, so every
agent that can call `accounts.delete_account` inherits the requirement, and you
can't forget it on the fifth agent.

## Where policies are evaluated

Policies run at three points, and the differences matter:

| When | Sees | Typical use |
|---|---|---|
| **Before the run** | Inputs, agent identity, caller | Block disallowed inputs; require approval up front |
| **Mid-run** | Each tool call, *before* it executes | Stop a specific action — a write, a spend, an external send |
| **After the run** | Full output and span tree | Redact, flag for review, raise an incident |

Mid-run is the one people underestimate: an agent that *decided* to do something
disallowed is stopped at the call, not cleaned up afterwards.

A blocked run is a first-class outcome — it finishes in the terminal `BLOCKED`
state with a recorded decision and an `ExecutionIncident`. **It does not raise**,
so there is nothing to `try/except`. Code that polls for only `COMPLETED` or
`FAILED` hangs forever; see [run states](/runtime/workflow-envelope#run-states).

## Policies: the engine behind all of this

Limits and approvals cover the agent you're writing. **Policies** are the layer
above — rules an operator sets that apply across agents, scoped to a tenant,
agent, workflow, tool, model, user, or team, and evaluated server-side at the
three points above.

Waxell ships **49 policy categories**. A few you'll reach for early:

| Category | Governs |
|---|---|
| [Budget](/observe/governance/budget) | Token and dollar budgets — per-run, daily, monthly |
| [Rate Limit](/observe/governance/rate-limit) | Request rates and concurrency per agent, user, or team |
| [Content](/observe/governance/content) | Input/output scanning for PII, credentials, injection |
| [Data Access](/observe/governance/data-access) | Which sources an agent may read or write |
| [Network](/observe/governance/network) | Outbound domain allowlists |
| [Scope](/observe/governance/scope) | Blast radius — records modified, transaction amounts |
| [Approval](/observe/governance/approval) | What requires a human before it proceeds |
| [Kill Switch](/observe/governance/kill-switch) | Emergency stop on error rate or anomaly |

The full set — including OWASP LLM Top 10, GDPR, HIPAA, SOC 2, ISO 42001, and
NIST AI RMF categories — is in
[Policy Categories & Templates](/observe/features/policy-categories).

Every check returns an action: `allow`, `warn`, `redact`, `throttle`, `block`, or
`skip`. See [Policy & Governance](/observe/features/governance) for how the
evaluation and the actions work.

### Policy as code

Define policies alongside your agents and push them like any other artifact:

```python
from waxell_sdk import policy

outreach_budget = policy(
    name="outreach_budget",
    category="budget",
    scope={"agents": ["outreach_agent"]},
    rules={
        "per_workflow_token_limit": 50_000,
        "per_workflow_cost_limit": 1.00,
    },
    description="Budget ceiling for outreach agent runs.",
)
```

```bash
wax -p prod policies push ./governance/ --dry-run   # show the diff
wax -p prod policies push ./governance/             # apply
```

This creates real, scoped policy records — the same objects the Governance panel
edits. Inspect what's live with `wax -p prod policies list` and
`wax -p prod policies show <name>`.

Scope is the important field. `{"agents": ["outreach_agent"]}` narrows a policy
to one agent; omit it and the policy applies tenant-wide. You can also scope by
`workflows`, `tools`, `models`, `user_groups`, and more.

### Promoting policy between environments

Policy bundles move a curated set of policies between tenants — a golden config
rollout, or a snapshot before a change:

| Endpoint | Purpose |
|---|---|
| `GET /v1/governance/bundles/export/` | Current policy set as a versioned JSON bundle |
| `POST /v1/governance/bundles/preview/` | Dry-run import — per-policy diff, writes nothing |
| `POST /v1/governance/bundles/import/` | Apply, with `mode=merge\|replace` and `conflict=skip\|overwrite` |

Imports auto-snapshot the affected policies into `PolicyVersion` first, so a
rollout is reversible.

### Test before you enforce

Two things worth doing before a policy goes live:

- **Dry-run a draft** against real traffic to see what it *would* have caught,
  before it blocks anything.
- **Check for conflicts** — overlapping scopes where two policies disagree
  surface as a conflict rather than as a mystery block in production.

Both are in the Governance panel alongside the policy editor.

## The audit trail

Every run is recorded automatically. There is nothing to enable and nothing to
instrument:

| Artifact | Holds |
|---|---|
| Span tree | Every LLM, tool, and domain call — nested, timed, errors marked |
| `LlmCallRecord` | Model, tokens in/out, cost, provider instance |
| Policy decisions | What evaluated, what it decided, why |
| `ExecutionIncident` | Errors, policy blocks, budget stops |
| Audit log | Who or what triggered the run, under which identity |

A failed or blocked run is recorded as fully as a successful one.

Inspect from the CLI:

```bash
wax -p prod traces list --hours 24 --agent email_sender
wax -p prod traces show <trace_id>
```

Or open Observe → Runs for the same trace with the span tree rendered.

## Next steps

- **[Policy Categories & Templates](/observe/features/policy-categories)** — all 49 categories
- **[Policy & Governance](/observe/features/governance)** — how evaluation and actions work
- **[How the Runtime Works](/runtime/how-it-works)** — where governance sits in a run
- **[Production Guide](/tutorials/production)** — deploy with governance


--------------------------------------------------------------------------------
# Deploying to Production
URL: https://waxell.ai/docs/tutorials/production
Description: Deploy Waxell agents to production environments.
--------------------------------------------------------------------------------

# Deploying to Production

This tutorial covers deploying Waxell agents to production with proper infrastructure.

## Production Architecture

```
┌─────────────────────────────────────────────────────────┐
│                     Load Balancer                        │
└─────────────────────────┬───────────────────────────────┘
                          │
┌─────────────────────────▼───────────────────────────────┐
│                    API Gateway                           │
│              (waxell-controlplane)                       │
└─────────────────────────┬───────────────────────────────┘
                          │
          ┌───────────────┼───────────────┐
          ▼               ▼               ▼
    ┌──────────┐    ┌──────────┐    ┌──────────┐
    │  Worker  │    │  Worker  │    │  Worker  │
    │ (Celery) │    │ (Celery) │    │ (Celery) │
    └────┬─────┘    └────┬─────┘    └────┬─────┘
         │               │               │
         └───────────────┼───────────────┘
                         ▼
          ┌──────────────────────────────┐
          │       Redis + PostgreSQL      │
          └──────────────────────────────┘
```

## Infrastructure Requirements

- **Redis**: Task queue and rate limiting
- **PostgreSQL**: Persistent storage
- **Celery Workers**: Async task execution
- **Django**: Control plane and API

## Configuration

Install the infra package:

```bash
pip install waxell-infra
```

Configure Django settings:

```python
# settings.py
INSTALLED_APPS = [
    ...
    'waxell_infra',
]

# Redis connection
REDIS_URL = "redis://localhost:6379/0"

# Celery configuration
CELERY_BROKER_URL = REDIS_URL
CELERY_RESULT_BACKEND = REDIS_URL
```

## Deploying an Agent

```bash
# Register the agent
wax agent register ./my_agent.py

# Deploy to production
wax agent deploy my-agent --env production

# Monitor execution
wax agent logs my-agent --follow
```

## Scaling

Scale workers based on load:

```bash
# Kubernetes
kubectl scale deployment waxell-workers --replicas=10

# Docker Compose
docker-compose up -d --scale worker=10
```

## Monitoring

Monitor agents via the dashboard:

- **Execution metrics**: Success rate, latency, throughput
- **Token usage**: LLM token consumption
- **Error tracking**: Failed executions and causes
- **Audit logs**: All agent actions

## Next Steps

- **[Architecture Guide](/guides/architecture)** - Deep dive into architecture
- **[Enterprise Guide](/guides/enterprise)** - Multi-tenancy and compliance


================================================================================
SECTION: WAXELL RUNTIME
Run governed agents in production — execution tiers, isolation, working memory, and durable workflows
================================================================================

--------------------------------------------------------------------------------
# Quickstart
URL: https://waxell.ai/docs/runtime/quickstart
Description: Build, register, and run your first Waxell agent in under five minutes.
--------------------------------------------------------------------------------

# Quickstart

Waxell agents are declared in a `waxell.yaml` manifest, or defined in Python with `@agent`. Either way the runtime handles tracing, policy enforcement, working memory and LLM routing — you write the business logic.

**One canonical import path**: everything customer-facing is exposed under `waxell_runtime`. Don't import from `waxell_sdk` — that's an internal implementation detail.

```python
from waxell_runtime import agent, workflow, tool   # ✓ canonical
from waxell_sdk import agent                        # ✗ internal, will break
```

## Install

```bash
pip install waxell           # canonical meta-package — pulls runtime + observe + the wax CLI
```

If you need just the runtime engine without observability (size-constrained, separate observability path, etc.), `pip install waxell-runtime` is the alternate entry point and stays available forever.

For an isolated CLI install (recommended if you only want `wax` on your PATH), use `pipx install waxell`. Full options and troubleshooting live in the [CLI reference](/reference/cli#installation).

### Verify the install worked

```bash
wax --version                     # the wax CLI is installed and on your PATH
python -c "import waxell_runtime"  # the SDK imports cleanly (no output = success)
```

If `wax` isn't found or the import errors, see the [CLI install troubleshooting](/reference/cli#troubleshooting-installation) (PATH, pipx, PEP 668).

### Authenticate

```bash
wax login                    # writes credentials to ~/.waxell/config
wax whoami                   # confirms your API key + tenant resolve
```

## Your first agent

Most agents are declared in a `waxell.yaml` manifest — this is what `wax push`
calls "manifest mode (recommended)", and it's how the agents running in
production are defined. You can also define them in Python; that's further down.

We'll build a document classifier: it reads an inbound document, decides what
kind it is, extracts the key fields, and files it back into your application.
Real shape, not a toy.

### 1. Declare it

```yaml
version: 1

defaults:
  owner: you@yourcompany.com
  tags: [quickstart]

agents:
  - name: doc_intake
    framework: pydantic_ai
    version: "1.0.0"
    description: Classify an inbound document and file it.
    model: anthropic:claude-sonnet-4-6
    timeout_seconds: 120

    # What the agent may reach in YOUR application.
    domains: [document]

    system_prompt: |
      You classify inbound business documents.

      Steps:
      1. Call document.read(document_id=<the document_id>) to get the text.
      2. Decide which type it is: invoice, purchase_order, or quote.
      3. Extract: customer_name, doc_number, total, currency, date.
      4. Call document.classify_result(document_id=..., doc_type=...,
         customer_name=..., extracted_json="<JSON string of your fields>").

      Make exactly one classify_result call, then reply with one line.
      Never invent parties, amounts or dates — use only what the document says.

    # What starts a run.
    signals:
      - name: document_received
        source_type: api
        description: A document landed and needs classifying.
        schema:
          document_id:
            type: string
            description: The stored document id.
        idempotency_key: $.document_id
```

Three things are doing the real work:

- **`domains: [document]`** — the agent never touches your database. It calls
  named actions; your service answers. See [Domains](./domains.md).
- **`signals`** — how a run starts. `idempotency_key` means a webhook delivered
  twice produces one run, not two.
- **`system_prompt`** — judgement only. Note what it does *not* do: it never
  decides whether two documents are the same record. That belongs in your
  domain, where being wrong is cheap to fix.

### 2. Validate, then push

```bash
wax push --dry-run    # parse + show what would upload, without touching the platform
wax push              # auto-discovers waxell.yaml in the current directory
wax agents list
```

Always dry-run first — it catches typos and schema mistakes for free.

### 3. Fire a signal

```bash
wax signals fire document_received --payload '{"document_id":"doc-123"}'
```

This returns a `run_id` **immediately — it does not wait for the agent.** Runs
proceed in the background, which is what keeps your API fast.

```bash
wax signals show <run_id>    # the run and its result
wax runs list                # recent runs
```

### 4. Read the trace

The part worth learning properly:

```bash
wax traces list
wax runs show <run_id>
```

Or open **Observe → Runs**. Either way you get the span tree — every LLM call,
tool call and domain call, nested, timed, failures in red, cost attached.

When an agent misbehaves, the trace tells you which of three things happened:

| What you see | What it means |
|---|---|
| No tool calls at all | The agent never called your domain — usually a prompt problem |
| `document.read` then nothing | It read the document and didn't act — check the prompt's final instruction |
| A red span | The call failed; the error is on the span |

### What you just built

```
signal ──► doc_intake ──► document.read           (your app)
                     └──► document.classify_result (your app)
                          │
                          span tree · cost · policy decisions · audit
```

Governed by default. You didn't configure any of that.

---

## Alternative: define agents in Python


```python
from waxell_runtime import agent, workflow, tool

@agent(name="lead_research", description="Research a lead and draft outreach")
class LeadResearchAgent:

    @tool
    async def fetch_company(self, ctx, domain: str) -> dict:
        """Look up a company by domain."""
        return await ctx.domain("company", "lookup", domain=domain)

    @workflow("draft_email")
    async def draft_email(self, ctx, lead_id: str):
        company = await ctx.tool("fetch_company", domain="acme.com")
        email = await ctx.llm.generate(
            prompt=f"Write a 3-sentence intro for {company['name']}",
            output_format="text",
            task="outreach_email",
        )
        return {"email": email, "company": company["name"]}
```

Push it to the platform:

```bash
wax push agent.py
wax agents list
```

That's the loop. The runtime handles the rest.

**@agent replaces the class**
After decoration, `LeadResearchAgent` is an `AgentSpec`, not a class you instantiate. The platform constructs and runs the agent for you.


## What `ctx` gives you

Inside any `@workflow` or `@tool`:

| Call | Purpose |
|------|---------|
| `await ctx.tool("name", **inputs)` | Invoke another `@tool` on this agent |
| `await ctx.domain("entity", "action", **inputs)` | Call a Waxell domain endpoint |
| `await ctx.llm.generate(prompt=..., output_format=...)` | Routed LLM call (cost-aware, policy-aware) |
| `ctx.secrets.get("OPENAI_API_KEY")` | Read a secret from the configured provider |

For IDE autocomplete, type-hint `ctx`:

```python
from waxell_runtime import WorkflowContext, ToolContext, Context

@workflow("draft_email")
async def draft_email(self, ctx: WorkflowContext, lead_id: str): ...

@tool
async def fetch_company(self, ctx: ToolContext, domain: str) -> dict: ...
```

## Directory layout for larger agents

When a single file gets unwieldy:

```
my_agent/
  agent.py            # @agent class with tools=[...] workflows=[...]
  tools/
    fetch_company.py
  workflows/
    draft_email.py
```

```python
# my_agent/agent.py
from waxell_runtime import agent

@agent(
    name="lead_research",
    description="Research a lead and draft outreach",
    tools=["fetch_company"],         # references tools/fetch_company.py
    workflows=["draft_email"],       # references workflows/draft_email.py
)
class LeadResearchAgent:
    pass
```

The decorator strings are filenames (no `.py`), not Python symbols. One workflow per file is the convention.

## Configuration

Config priority (highest first):

1. Explicit `WaxellClient(api_url=..., api_key=...)` constructor args
2. `WaxellClient.configure(...)` set globally at startup
3. `~/.waxell/config` (written by `wax login`)
4. Environment variables: `WAX_API_KEY`, `WAX_API_URL`, `WAX_TENANT`

If none resolve, calls silently no-op. Run `wax whoami` to verify.

**Migrating from waxell-observe**
If you previously set `WAXELL_API_KEY` for the observe SDK, also set `WAX_API_KEY`. The runtime SDK reads only `WAX_*` (no fallback). Setting `WAX_API_KEY` alone covers both packages — observe falls back to it.


## Use Claude Code with Waxell

If you use Claude Code, install the bundled skill so Claude understands the SDK conventions when you ask it to extend your agent:

```bash
wax claude-init
```

This drops a project-scoped skill at `.claude/skills/waxell-runtime/SKILL.md`. Claude activates it whenever it sees `from waxell_runtime import ...` in your project and will write idiomatic agent code for you. Re-run `wax claude-init` after upgrading `waxell-runtime` to refresh the skill.

For a global install, use `--scope user`. For just printing the skill content, use `--print-only`.

## Common mistakes

| Mistake | Symptom | Fix |
|---------|---------|-----|
| Pushed to the wrong tenant | Agent "vanishes"; it's live somewhere else | `wax whoami`; target one with `wax -p prod push` |
| Re-pushing the same `version:` | `409 semver_conflict` | Versions are immutable — bump `version:` |
| Expecting `signals fire` to return the answer | You get a `run_id`, not a result | It's async by design. Poll `wax signals show`, or have the agent call back through a domain |
| Agent replies but nothing happens in your app | Run completes, your database is untouched | It never called your domain. Check the trace for tool calls; name the action explicitly in the prompt |
| Setting only `WAXELL_API_KEY` | Runtime calls silently no-op | Set `WAX_API_KEY` instead |
| `os.environ["OPENAI_API_KEY"]` inside a workflow | Works locally, breaks in deployment | Use `ctx.secrets.get("OPENAI_API_KEY")` |
| Instantiating the class after `@agent` | `TypeError: 'AgentSpec' object is not callable` | Don't instantiate. The platform runs the agent. |
| `from waxell_sdk import agent` | Works in editable install, breaks for customers | Always `from waxell_runtime import agent` |

## What's next

- **[How the runtime works](/runtime/how-it-works)** — signal to result, and where to put logic
- **[Execution tiers](/runtime/execution-tiers)** — shared, warm, or your own container
- **[`waxell.yaml` reference](/runtime/waxell-yaml-reference)** — every supported field
- **[Runtime overview](/runtime/overview)** — execution context, durability, backends
- **[SDK overview](/sdk/overview)** — every decorator and spec type
- **[First agent tutorial](/tutorials/first-agent)** — guided walkthrough with explanation
- **[Workflows tutorial](/tutorials/workflows)** — multi-step orchestration patterns


--------------------------------------------------------------------------------
# Runtime Overview
URL: https://waxell.ai/docs/runtime/overview
Description: Understanding the Waxell Runtime - the execution engine for AI agents.
--------------------------------------------------------------------------------

# Runtime Overview

The Waxell Runtime is the execution engine that brings your SDK-defined agents to life. It handles execution context, durability, and pluggable backends.

## Core Responsibilities

- **Execution Context**: Manages state during workflow execution
- **Durability**: Checkpoint and resume for long-running workflows
- **Backend Abstraction**: Pluggable backends for different environments
- **Governance Hooks**: Integration points for policy enforcement

## Key Abstractions

| Concept | Purpose |
|---------|---------|
| `RuntimeConfig` | DI container for backend configuration |
| `ExecutionContext` | Pure execution state management |
| `WorkflowEnvelope` | Execution boundary with checkpoint/resume |
| `Step` | Durable execution unit |

## Local vs Production

The runtime provides different backends for different environments:

```python
# Local development (in-memory, no external dependencies)
from waxell_runtime import RuntimeConfig
config = RuntimeConfig.get()  # Uses defaults

# Production (Redis, Celery, Django)
# Automatically configured by waxell-infra Django app
```

## Working with Claude Code

If you use Claude Code, install the bundled Waxell Runtime skill so Claude writes idiomatic agent code for you:

```bash
wax claude-init
```

This drops `.claude/skills/waxell-runtime/SKILL.md` into your project. The skill activates whenever Claude sees `from waxell_runtime import ...` and teaches the canonical decorators, the `ctx` proxy API, working memory conventions, env-var setup, and common pitfalls. Re-run `wax claude-init` after upgrading `waxell-runtime` to refresh.

## Next Steps

- **[Quickstart](/runtime/quickstart)** - Build, push, and run your first agent
- **[ExecutionContext](/runtime/execution-context)** - Understand execution state
- **[WorkflowEnvelope](/runtime/workflow-envelope)** - Learn about durability
- **[Backends](/runtime/backends)** - Configure production backends


--------------------------------------------------------------------------------
# How the Runtime Works
URL: https://waxell.ai/docs/runtime/how-it-works
Description: What actually happens between a signal arriving and a run finishing — routing, governance, execution, telemetry, and cost attribution.
--------------------------------------------------------------------------------

# How the Runtime Works

This page traces one run from the event that starts it to the record it leaves
behind. If you're debugging behaviour, deciding where to put logic, or asking
"where would I see that?" — this is the map.

## The shape of it

```
signal ──► match agent ──► resolve model ──► pre-run policy
                                                  │
                                            execution tier
                                                  │
                                    ┌─────────────┴─────────────┐
                                    │   the agent's tool loop   │
                                    │   tools · domains · LLM   │
                                    └─────────────┬─────────────┘
                                                  │
                            mid-run policy on every tool call
                                                  │
                          post-run policy ──► spans · cost · audit
```

Everything after "match agent" is identical no matter which execution tier the
run lands on. That's the point of the tier split: **where** a run executes is an
infrastructure decision; **what governs it** is not.

---

## 1. A signal arrives

A run starts when something calls one of your agent's declared signals:

```yaml
signals:
  - name: document_received
    source_type: api
    schema:
      document_id: string
    idempotency_key: $.document_id
```

```bash
wax -p prod signals fire document_received --payload '{"document_id":"abc123"}'
```

The platform matches the signal name to the agent that declared it, creates an
`AgentExecutionRun` row in `pending`, and returns immediately. **The caller does
not wait for the agent.** Your API stays fast; the run proceeds in the
background.

`idempotency_key` is a JSONPath into the payload. Two signals carrying the same
key collapse to one run — the guard against a webhook delivered twice.

## 2. The model is resolved

Waxell resolves the model in three layers, highest priority first:

1. **Per-run override** — passed explicitly on the call
2. **Tenant routing** — your configured provider instances and task routes
3. **Declared** — the `model:` in your `waxell.yaml`

This is why the same agent can run on different models in dev and prod without a
code change, and why a tenant can move providers without touching an agent.

The resolved provider instance is recorded on the run, so cost attributes to the
right provider even when routing changes mid-month.

## 3. Governance runs before the agent does

Policies evaluate **before** the first token is generated. A blocking policy
stops the run outright — no LLM call, no cost.

```yaml
policies:
  - block_pii_leakage
  - require_human_review
```

Those names reference policies in your tenant's policy engine, where the actual
rules, scope, and action live. Policies scoped to an agent apply whether or not
the agent names them — see [Adding Governance](/tutorials/governance) for the
code-first path and [all 49 categories](/observe/features/policy-categories).

There are three evaluation points, and they're different on purpose:

| When | Sees | Typical use |
|---|---|---|
| **Before** | Inputs, agent identity, caller | Block disallowed inputs; require approval up front |
| **Mid-run** | Each tool call, before it executes | Stop a specific action — a write, a spend, an external send |
| **After** | The full output and span tree | Redact, flag for review, raise an incident |

Mid-run is the one people underestimate. An agent that decided to do something
disallowed is stopped **at the call**, not after the fact.

A blocked run is a first-class outcome: it finishes with a recorded decision and
an `ExecutionIncident`, not an exception.

## 4. The agent executes

The runtime hands the agent its prompt, its tools, and an execution context.

**Tools** come from three places, and they arrive as one list to the model:

- `tools:` — your own registered tools
- `domains:` — callbacks into *your* application (see [Domains](./domains.md))
- Native runtime tools — spawn, wait, ask-user, memory

Domains are the important one for most teams. Rather than giving an agent
database credentials, you expose named actions:

```yaml
domains: [document, deal]
```

The agent calls `document.read(...)`; your service answers over a shared-secret
callback. Your data stays in your application, the agent gets an interface, and
every call is a governed, traced step.

The loop runs until the agent stops calling tools, hits `timeout_seconds`, or a
policy stops it.

## 5. Where it ran

The tier is chosen per agent — see [Execution Tiers](./execution-tiers.md).
Shared workers by default; a warm slot for per-execution isolation; a
self-contained container when the agent needs its own dependencies.

The run records which tier it used, and that decides the compute multiplier on
your bill. Nothing about governance or telemetry changes with the tier.

## 6. What the run leaves behind

Every run produces the same record regardless of tier:

| Artifact | What it holds | Where you see it |
|---|---|---|
| **Span tree** | Every LLM call, tool call, and domain call, nested, timed, with errors marked | Observe → Runs → trace |
| **`LlmCallRecord`** | Model, tokens in/out, cost, provider instance | Observe, and Billing → Usage |
| **Policy decisions** | What evaluated, what it decided, why | Observe → Governance |
| **`ExecutionIncident`** | Errors, policy blocks, budget stops | Observe → Errors |
| **Usage events** | Agent execution, CPU-seconds, memory, LLM tokens | Billing → Usage |
| **Audit log** | Who/what triggered it, under which identity | Audit |

A failed run is as fully recorded as a successful one. If a run "did nothing",
the trace shows you *why* — no tool calls, a policy block, or an empty response.

---

## Where to put logic

The most common design question, and the one that costs the most to get wrong:

| Put it in… | When |
|---|---|
| **The system prompt** | Judgement. Classification, extraction, tone, "which of these six is it". |
| **A domain action** | Anything deterministic or stateful. Writing records, computing totals, deciding state transitions. |
| **A policy** | Rules that must hold regardless of what the agent decides. |
| **A guard/limit** | Bounds: cost, turns, time. |

The rule of thumb: **if being wrong is expensive, it doesn't belong in the
prompt.** An LLM should decide *which* invoice this is; it should not be the
thing that decides whether two invoices are the same record. Put the judgement in
the agent and the arithmetic in your domain.

## Failure behaviour

Worth knowing before you rely on it:

- **The agent returns nothing useful.** The run completes; the span tree shows an
  empty tool loop. Usually a prompt problem, and the trace tells you which.
- **A tool or domain call fails.** The error is recorded on that span and
  surfaced as an incident. The agent sees the failure and may retry or route
  around it.
- **Timeout.** `timeout_seconds` caps wall time; the run is marked failed with
  what completed preserved.
- **A policy blocks.** Recorded as a decision with its reason. Not an error.
- **Infrastructure dies mid-run** (OOM, host loss). The run is finalised as
  failed by a reconciler rather than sitting unfinished.

---

## Related

- [Adding Governance](/tutorials/governance) — limits, approvals, and the policy engine
- [Execution Tiers](./execution-tiers.md) — where runs execute, and what it costs
- [Domains](./domains.md) — how agents call back into your application
- [Execution Context](./execution-context.md) — what the agent can access at runtime
- [`waxell.yaml` reference](./waxell-yaml-reference.md) — every supported field


--------------------------------------------------------------------------------
# waxell.yaml Reference
URL: https://waxell.ai/docs/runtime/waxell-yaml-reference
Description: Complete field reference for the waxell.yaml agent specification file.
--------------------------------------------------------------------------------

# waxell.yaml Reference

The `waxell.yaml` file is the declarative specification for a Waxell agent. It defines what an agent is, what it can do, and what governance constraints it operates under. The runtime reads this file at deploy time (`wax push`) and uses it to instantiate the agent, register tools, wire workflows, and apply governance.

If you don't already have the wax CLI: `pip install waxell` (or `pipx install waxell` for an isolated install). See [CLI reference](/reference/cli#installation) for Windows PATH issues and troubleshooting.

## Minimal example

```yaml
version: 1

agents:
  - name: my_agent
    version: 1.0.0
    description: A simple agent.
    system_prompt: |
      You are a helpful assistant.
    tools: [my_tool]
    workflows: [my_workflow]

tools:
  - name: my_tool
    description: Does something useful.
    inputs:
      query:
        type: string
    outputs:
      result:
        type: string

workflows:
  - name: my_workflow
    type: graph
    nodes: [step_one]
    edges: []
```

## Full example

```yaml
version: 1

defaults:
  framework: autogen
  owner: team@example.com
  tags: [production]

agents:
  - name: doc_intake
    version: 1.0.3
    description: Classifies an inbound document and files it against a deal.
    tags: [intake, documents]
    model: anthropic:claude-sonnet-4-6
    timeout_seconds: 120

    system_prompt: |
      You classify inbound documents. Read the document, decide which type it
      is, extract the key fields, and file it.
    # or reference the prompt registry instead of inlining:
    # system_prompt_ref:
    #   name: doc_intake_system
    #   label: production

    # What the agent can reach.
    domains: [document, deal]
    tools: [calculator]
    workflows: [escalation]
    capabilities: [waxell.hooks]

    # Governance. Bare names, or objects with conditions.
    policies:
      - block_pii_leakage
      - name: require_human_review
        conditions:
          - output_contains_dollar_amount_above(50000)

    # Inbound events that start a run.
    signals:
      - name: document_received
        source_type: api
        description: A document landed and needs classifying.
        schema:
          document_id: string
          filename: string
        idempotency_key: $.document_id

    memory:
      scratchpad:
        tier: working
        scope: [run_id]
        ttl: 1h

    # Libraries this agent needs that the platform image doesn't carry.
    # Declaring these builds it a dedicated container. See Execution Tiers.
    dependencies:
      apt: [antiword]
      pip: [pypdf>=4.0, python-docx>=1.1]
      python: "3.11"

    execution:
      mode: isolated       # shared (default) | isolated
      tier: ephemeral      # ephemeral (default) | warm

    telemetry:
      alerts:
        - p95_latency_ms > 5000
        - error_rate > 0.05
      sample_rate: 1.0

tools:
  - name: calculator
    version: 1.0.1
    description: Basic arithmetic.
    tags: [math, utility]
    inputs:
      a:
        type: number
      b:
        type: number
      op:
        type: string
        enum: [add, sub, mul, div]
    outputs:
      result:
        type: number

workflows:
  - name: escalation
    version: 1.0.1
    type: graph
    description: Human-in-the-loop escalation.
    nodes: [triage, draft_response, human_review, send]
    edges:
      - {from: triage, to: draft_response}
      - {from: draft_response, to: human_review}
      - {from: human_review, to: send}
```

---

## Top-level fields

| Field | Type | Required | Default | Description |
|---|---|---|---|---|
| `version` | `integer` | Yes | - | Spec format version. Currently `1`. |
| `defaults` | `object` | No | `{}` | Project-wide defaults inherited by all agents/tools in this file. |
| `agents` | `list[Agent]` | Yes | - | List of agent definitions. |
| `tools` | `list[Tool]` | No | `[]` | Tool definitions referenced by agents. |
| `workflows` | `list[Workflow]` | No | `[]` | Workflow definitions referenced by agents. |

### defaults

| Field | Type | Default | Description |
|---|---|---|---|
| `framework` | `string` | - | Default agent framework (e.g., `autogen`). |
| `owner` | `string` | - | Default owner email. |
| `tags` | `list[string]` | `[]` | Default tags applied to all agents. |

---

## agents

Each entry in the `agents` list defines one agent. An agent is the top-level unit of deployment.

### Core fields

| Field | Type | Required | Default | Description |
|---|---|---|---|---|
| `name` | `string` | Yes | - | Unique agent identifier. Used in URLs, CLI, and traces. Must be a valid Python identifier (letters, digits, underscores). |
| `version` | `string` | No | - | Semantic version string (e.g., `1.0.3`). |
| `description` | `string` | No | `""` | Human-readable description. Shown in the controlplane UI. |
| `tags` | `list[string]` | No | `[]` | Classification tags. Merged with `defaults.tags`. |
| `model` | `string` | No | - | Default LLM model in `provider:model` format (e.g., `openai:gpt-4o-mini`). Overridden by `llm_config` task routes. |
| `timeout_seconds` | `integer` | No | - | Maximum wall-time for a single agent run. |

### Prompt configuration

| Field | Type | Required | Default | Description |
|---|---|---|---|---|
| `system_prompt` | `string` | No | - | Inline system prompt. Mutually exclusive with `system_prompt_ref` in practice; if both are set, `system_prompt_ref` takes precedence at runtime. |
| `system_prompt_ref` | `object` | No | - | Reference to a prompt in the prompt registry. |
| `system_prompt_ref.name` | `string` | Yes | - | Prompt name in the registry. |
| `system_prompt_ref.label` | `string` | No | `"production"` | Version label. Allows promoting prompts without redeploying. |

### Bindings

| Field | Type | Required | Default | Description |
|---|---|---|---|---|
| `tools` | `list[string]` | No | `[]` | Tool names this agent can use. Must match entries in the top-level `tools` section or be registered via `wax push-tool`. |
| `workflows` | `list[string]` | No | `[]` | Workflow names this agent can execute. Must match entries in the top-level `workflows` section or be registered via `wax push-workflow`. |
| `domains` | `list[string]` | No | `[]` | External domain names (e.g., `crm`, `email`). Resolved against the tenant's registered domains at runtime. |
| `capabilities` | `list[string]` | No | `[]` | Named capabilities this agent exposes. Each maps to a tool, workflow, or domain action. |

### Relationships

| Field | Type | Required | Default | Description |
|---|---|---|---|---|
| `signals` | `list[Signal]` | No | `[]` | Inbound event definitions. See [signals section](#signals). |
| `schedule` | `string` or `list` | No | - | Cron schedule(s) that fire this agent, e.g. `"0 9 * * *"` (five-field, UTC). Expanded form takes `cron`, optional `signal`, `payload`, and `slot`. See [Operating Agents](./operating-agents.md#run-on-a-schedule). |
| `memory` | `map[string, Memory]` | No | `{}` | Named memory slots. See [memory section](#memory). |
| `policies` | `list` | No | `[]` | Governance policy references. Can be bare strings (policy names) or objects with `name` and `conditions`. |
| `telemetry` | `object` | No | - | Telemetry and alerting configuration. See [telemetry section](#telemetry). |

### Execution

| Field | Type | Required | Default | Description |
|---|---|---|---|---|
| `dependencies` | `object` | No | - | Libraries and system packages this agent needs. Declaring any triggers a per-agent container build. See [dependencies section](#dependencies). |
| `execution` | `object` | No | - | Where this agent runs. See [execution section](#execution) and [Execution Tiers](./execution-tiers.md). |

---

## tools

Each entry in the top-level `tools` list defines a tool the agent can call.

| Field | Type | Required | Default | Description |
|---|---|---|---|---|
| `name` | `string` | Yes | - | Unique tool identifier. Referenced by agents in their `tools` list. |
| `version` | `string` | No | - | Semantic version string. |
| `description` | `string` | No | `""` | What the tool does. Shown to the LLM as tool documentation. |
| `tags` | `list[string]` | No | `[]` | Classification tags. |
| `async_only` | `boolean` | No | `false` | If true, tool must be called asynchronously. |
| `inputs` | `map` | No | `{}` | Input parameter definitions. See [tool inputs](#tool-inputs). |
| `outputs` | `map` | No | `{}` | Output field definitions. |
| `schema` | `object` | No | - | Unified JSON Schema for inputs/outputs. Alternative to separate `inputs`/`outputs`. |

### Tool inputs

Each key in `inputs` is a parameter name. The value is an object:

| Field | Type | Required | Default | Description |
|---|---|---|---|---|
| `type` | `string` | No | `"string"` | JSON Schema type: `string`, `number`, `integer`, `boolean`, `array`, `object`. |
| `description` | `string` | No | - | Parameter description shown to the LLM. |
| `enum` | `list` | No | - | Allowed values. |
| `required` | `boolean` | No | `false` | Whether the parameter is required. |
| `default` | `any` | No | - | Default value if not provided. |
| `items` | `object` | No | - | For `array` type: schema of array elements. |
| `format` | `string` | No | - | JSON Schema format hint (e.g., `email`, `uri`). |
| `pattern` | `string` | No | - | Regex pattern for validation. |
| `minLength` | `integer` | No | - | Minimum string length. |
| `maxLength` | `integer` | No | - | Maximum string length. |
| `minimum` | `number` | No | - | Minimum numeric value. |
| `maximum` | `number` | No | - | Maximum numeric value. |

**Example:**

```yaml
tools:
  - name: search
    description: Search a knowledge base.
    inputs:
      query:
        type: string
        description: Search query
        required: true
      limit:
        type: integer
        description: Max results
        default: 10
        minimum: 1
        maximum: 100
    outputs:
      results:
        type: array
      total_count:
        type: integer
```

---

## workflows

Each entry defines a workflow the agent can execute.

| Field | Type | Required | Default | Description |
|---|---|---|---|---|
| `name` | `string` | Yes | - | Unique workflow identifier. |
| `version` | `string` | No | - | Semantic version string. |
| `type` | `string` | No | - | Workflow type. Common values: `graph`, `linear`, `router`. |
| `description` | `string` | No | `""` | What the workflow does. |
| `nodes` | `list[string]` | No | `[]` | Node names in the workflow graph. |
| `edges` | `list[Edge]` | No | `[]` | Edges connecting nodes. |
| `steps` | `list[Step]` | No | `[]` | Alternative to nodes/edges for linear workflows. |
| `routes` | `list[Route]` | No | `[]` | Routing rules for decision-based workflows. |
| `metadata` | `object` | No | `{}` | Arbitrary key-value metadata. |
| `inputs_schema` | `object` | No | `{}` | JSON Schema for workflow inputs. |

### Edge format

```yaml
edges:
  - {from: node_a, to: node_b}
  - {from: node_b, to: node_c, condition: "result.status == 'ok'"}
```

### Step format (linear workflows)

```yaml
steps:
  - id: step_0
    type: tool        # tool | workflow | decision
    name: my_tool
    inputs:
      query: "$.signal.query"
    produces: sym_0
```

| Field | Type | Required | Description |
|---|---|---|---|
| `id` | `string` | Yes | Unique step identifier. |
| `type` | `string` | Yes | Step type: `tool`, `workflow`, or `decision`. |
| `name` | `string` | Yes | Name of the tool/workflow/decision to invoke. |
| `inputs` | `object` | No | Input mappings. Can reference prior step outputs via `$.` syntax. |
| `produces` | `string` | No | Symbol name for this step's output. Referenced by later steps. |

---

## signals

Signals define inbound events that trigger agent execution.

| Field | Type | Required | Default | Description |
|---|---|---|---|---|
| `name` | `string` | Yes | - | Signal identifier. Used in webhook URLs: `/api/v1/signals/<name>/`. |
| `source_type` | `string` | No | `"webhook"` | Source type: `webhook`, `queue`, `schedule`, `api`, `internal`. |
| `description` | `string` | No | - | Human-readable description. |
| `schema` | `object` | No | `{}` | Expected payload shape. Keys are field names, values are types. |
| `triggers_workflow` | `string` | No | - | Workflow to run when this signal fires. |
| `idempotency_key` | `string` | No | - | JSONPath expression for deduplication (e.g., `$.proposal_id`). |
| `ttl_seconds` | `integer` | No | - | Time-to-live for the signal in the queue. |
| `priority` | `integer` | No | `0` | Processing priority (higher = sooner). |
| `tags` | `list[string]` | No | `[]` | Classification tags. |

**Example:**

```yaml
signals:
  - name: document_uploaded
    source_type: webhook
    description: Triggered when a user uploads a document
    schema:
      document_id: string
      filename: string
      size_bytes: number
    triggers_workflow: process_document
    idempotency_key: $.document_id
    ttl_seconds: 3600
```

---

## Limits

Every agent runs under caps. Set them explicitly — the defaults are generous, and
an agent that loops will spend to them.

| Field | Type | Default | Effect |
|---|---|---|---|
| `max_budget_usd` | `number` | `10.0` | Hard USD ceiling; the run stops when it's reached |
| `max_turns` | `integer` | `25` | Maximum agent turns before the loop is cut |
| `timeout_seconds` | `integer` | — | Wall-time cap applied by the dispatcher |
| `max_duration_s` | `integer` | container timeout | Hard wall-time for the run |

```yaml
agents:
  - name: doc_intake
    max_budget_usd: 2.00
    max_turns: 15
    timeout_seconds: 120
```

Per-child caps are set at the call site — `ctx.spawn(..., budget_usd=2.00,
timeout_seconds=300)` — and debit from the parent's budget, so a runaway child
can't outspend its parent.

Tenant-wide ceilings (concurrent spawns, rate limits, kill switch) are configured
in the Governance panel and apply across every agent.

---

## memory

Named memory slots scoped to different dimensions and persistence tiers.

| Field | Type | Required | Default | Description |
|---|---|---|---|---|
| `tier` | `string` | No | `"episodic"` | Memory tier: `working`, `session`, `episodic`, `semantic`. |
| `scope` | `string` or `list[string]` | No | `"tenant_id"` | Isolation boundary. Single dimension or list of dimensions. `tenant_id` is always enforced automatically. |
| `type` | `string` | No | `"dict"` | Data structure: `dict`, `list`, `value`. |
| `description` | `string` | No | `""` | Human-readable description. |
| `ttl` | `string` | No | - | Time-to-live: `"1h"`, `"30d"`, `"7d"`, etc. |
| `max_items` | `integer` | No | - | For `list` type: max items before oldest are evicted. |
| `custom_scope_key` | `string` | No | - | When scope is `"custom"`, the input field to use as the key. |
| `searchable` | `boolean` | No | `true` (semantic) | Semantic tier: enable vector embedding and retrieval. |
| `embedding_model` | `string` | No | - | Semantic tier: override default embedding model. |
| `schema` | `string` | No | - | Fully-qualified Python class for typed validation (e.g., `myapp.schemas.NoteSchema`). |
| `schema_version` | `integer` | No | `1` | Schema version for migration support. |

### Memory tiers

| Tier | Backend | Lifecycle | Use case |
|---|---|---|---|
| `working` | Redis | Per-run, cleared on completion | Scratchpad for in-flight tool results. |
| `session` | Postgres | 24h default TTL | Conversation/plan state within a user session. |
| `episodic` | Postgres | TTL-based | Cross-run state: conversation history, cached results. |
| `semantic` | pgvector | Indefinite | Long-term facts with semantic search. |

### Scope dimensions

| Dimension | Source | Description |
|---|---|---|
| `tenant_id` | Always | Outermost boundary (automatic). |
| `agent` | Agent name | Per-agent within tenant. |
| `agent_version` | Agent version | Per-agent version. |
| `user_id` | Sub-user identity | Per end-user. |
| `user_group` | Sub-user identity | Per user group. |
| `user_email` | Sub-user identity | Per user email. |
| `session_id` | Conversation session | Per conversation session. |
| `workflow` | Workflow name | Per workflow. |
| `run_id` | Execution run | Per workflow execution (ephemeral). |
| `channel_id` | Slack/chat | Per chat channel. |
| `thread_ts` | Slack | Per thread. |

**Example:**

```yaml
memory:
  # Working memory — auto-captures tool results
  scratchpad:
    tier: working
    scope: [run_id]
    ttl: 1h

  # Episodic — per-user conversation history
  conversation:
    tier: episodic
    scope: user_id
    type: list
    ttl: 30d
    max_items: 100

  # Semantic — long-term facts with vector search
  client_knowledge:
    tier: semantic
    scope: [user_id, portfolio_id]
    searchable: true
    description: Facts about this client relationship
```

---

## llm_config

Per-agent LLM routing configuration. Controls which models handle which task types.

| Field | Type | Required | Default | Description |
|---|---|---|---|---|
| `default_model` | `string` | No | - | Default model when no task-specific route matches. Format: `provider:model`. |
| `task_routes` | `map[string, TaskRoute]` | No | `{}` | Per-task routing rules. |
| `inherit_defaults` | `boolean` | No | `true` | Whether unmatched tasks fall through to tenant/global defaults. |
| `custom_task_types` | `list[string]` | No | `[]` | Additional task type names beyond the built-in set. |

### TaskRoute

| Field | Type | Required | Default | Description |
|---|---|---|---|---|
| `models` | `list[string]` | Yes | - | Ordered model list: primary + fallbacks. |
| `temperature` | `number` | No | - | Temperature override for this task. |
| `max_tokens` | `integer` | No | - | Token limit override for this task. |

**Example:**

```yaml
llm_config:
  default_model: openai:gpt-4o-mini
  task_routes:
    email:
      models: [openai:gpt-4o-mini, openai:gpt-4o]
      temperature: 0.7
      max_tokens: 2000
    classify:
      models: [openai:gpt-4o-mini]
      temperature: 0.1
    research:
      models: [anthropic:claude-sonnet-4-5-20250929]
      temperature: 0.3
      max_tokens: 4000
  inherit_defaults: true
```

---

## mcps

MCP (Model Context Protocol) server bindings. Each binding declares that the agent uses a registered MCP server.

| Field | Type | Required | Default | Description |
|---|---|---|---|---|
| `server` | `string` | Yes | - | MCP server slug. Must be registered in the tenant's MCPServer registry. |
| `tool_allowlist` | `list[string]` | No | `[]` | Only expose these tools from the server. Empty = all tools. |
| `tool_blocklist` | `list[string]` | No | `[]` | Hide these tools from the agent. |
| `tool_descriptions` | `map[string, string]` | No | `{}` | Agent-specific tool descriptions (override the server's defaults). |
| `enabled` | `boolean` | No | `true` | Whether this binding is active. |
| `transport` | `string` | No | `"http"` | Transport type: `http` or `stdio`. |
| `command` | `string` | Conditional | - | Required for `stdio` transport. Command to launch the MCP server process. |
| `args` | `list[string]` | No | `[]` | Arguments for the stdio command. |
| `env` | `map[string, string]` | No | `{}` | Environment variables for the stdio process. |
| `sandbox` | `object` | No | defaults | Sandbox configuration for stdio processes. |

### Sandbox (stdio only)

| Field | Type | Default | Description |
|---|---|---|---|
| `fs_allow` | `list[string]` | `[]` | Read-only filesystem paths the process can access. |
| `net` | `boolean` | `false` | Allow network egress. |
| `cpu_limit` | `number` | `0.5` | CPU-seconds/sec soft cap. |
| `mem_limit_mb` | `integer` | `256` | RSS memory cap in MB. |
| `max_duration_seconds` | `integer` | `300` | Hard wall-time for the session. |

**Example:**

```yaml
mcps:
  # HTTP transport — server registered in tenant MCPServer registry
  - server: github
    tool_allowlist: [search_repos, get_file, list_issues]

  # stdio transport — launches a local process
  - server: filesystem
    transport: stdio
    command: npx
    args: [-y, "@anthropic/mcp-filesystem"]
    env:
      HOME: /tmp/sandbox
    sandbox:
      fs_allow: [/data/documents]
      net: false
      mem_limit_mb: 128
      max_duration_seconds: 60
```

---

## prompts

Prompt registry bindings. Each binding links the agent to a versioned prompt.

| Field | Type | Required | Default | Description |
|---|---|---|---|---|
| `name` | `string` | Yes | - | Prompt name in the registry. |
| `version_label` | `string` | No | `"production"` | Version label pin. Allows promoting prompts without redeploying. |
| `order` | `integer` | No | `0` | Injection order when multiple prompts are bound. |
| `enabled` | `boolean` | No | `true` | Whether this binding is active. |

**Example:**

```yaml
prompts:
  - name: sales_playbook
    version_label: production
  - name: compliance_rules
    version_label: production
    order: 1
```

---

## metacog

Per-agent metacognition configuration. Enables conditional planning (triage), multi-step planning, and reflexion (learning from outcomes).

| Field | Type | Required | Default | Description |
|---|---|---|---|---|
| `triage` | `boolean` | No | `true` | Run a triage step before each dispatch to classify complexity. |
| `plan` | `boolean` | No | `true` | Generate a multi-step plan on hard turns. |
| `reflect` | `boolean` | No | `true` | Run reflexion at workflow exit to generate learnings. |
| `inject_learnings` | `integer` | No | `5` | Top-N learnings from prior runs injected into prompts. Set to `0` to disable. |
| `reflect_on_trivial` | `boolean` | No | `false` | Reflect on trivial turns too (not just hard turns or failures). |
| `triage_prompt` | `string` | No | built-in | Override the triage prompt template. |
| `plan_prompt` | `string` | No | built-in | Override the plan prompt template. |
| `reflect_prompt` | `string` | No | built-in | Override the reflect prompt template. |
| `triage_max_tokens` | `integer` | No | `200` | Token cap for triage calls. |
| `plan_max_tokens` | `integer` | No | `800` | Token cap for plan calls. |
| `reflect_max_tokens` | `integer` | No | `300` | Token cap for reflect calls. |

**Example:**

```yaml
metacog:
  triage: true
  plan: true
  reflect: true
  inject_learnings: 5
  reflect_on_trivial: false
  triage_max_tokens: 200
```

---

## telemetry

Agent-level telemetry and alerting configuration.

| Field | Type | Required | Default | Description |
|---|---|---|---|---|
| `alerts` | `list[string]` | No | `[]` | Alert rules as condition expressions. |
| `sample_rate` | `number` | No | `1.0` | Trace sampling rate (0.0 to 1.0). |

**Example:**

```yaml
telemetry:
  alerts:
    - p95_latency_ms > 5000
    - error_rate > 0.05
    - cost_per_run > 0.50
  sample_rate: 1.0
```

---

## policies

Names of policies from your tenant's policy registry that should apply to this
agent:

```yaml
policies:
  - block_pii_leakage
  - require_human_review
```

This is a **reference**, not a definition — the policy itself lives in the policy
engine, where it carries its category, rules, scope, and action. Declaring the
name here associates the agent with it and surfaces it on the agent's Governance
panel.

To create the policies themselves, see
[Adding Governance](/tutorials/governance) for the code-first path (`policy()` +
`wax policies push`) and
[Policy Categories](/observe/features/policy-categories) for all 49 categories.

Policies scoped to an agent in the engine apply whether or not the agent names
them here — scope is set on the policy. That's deliberate: an operator's rail
shouldn't be something an agent author can drop by editing their manifest.

---

## dependencies

Libraries and system packages your agent needs that the platform image doesn't
carry. Declaring any of these makes the agent **self-contained**: on `wax push`
Waxell builds it a container layered on the governed runtime base, and routes
its runs there.

| Field | Type | Required | Default | Description |
|---|---|---|---|---|
| `pip` | `list[string]` | No | `[]` | Python packages, pip syntax. Version specifiers supported (`pypdf>=4.0`). |
| `apt` | `list[string]` | No | `[]` | Debian system packages (e.g. `antiword`, `poppler-utils`). |
| `python` | `string` | No | `"3.11"` | Python version for the built image. |
| `bundle` | `string` | No | - | Name of a pre-approved dependency bundle. Uses a pre-built image instead of building — faster, no build step. |

```yaml
agents:
  - name: doc_intake
    dependencies:
      apt: [antiword]                                  # legacy .doc extraction
      pip: [pypdf>=4.0, python-docx>=1.1, olefile]
      python: "3.11"
```

Notes:

- **Rebuilds happen automatically** when your declared dependencies change.
  Unchanged dependencies reuse the existing image, so a re-push is cheap.
- **A failed build doesn't break the agent.** It continues on the shared tier
  until a build succeeds; the error is on the agent's Build panel.
- Declaring `dependencies` without `execution.mode` still builds the image —
  set `mode: isolated` to route runs to it.

---

## execution

Where this agent runs. See [Execution Tiers](./execution-tiers.md) for how to
choose, cold-start behaviour, and billing multipliers.

| Field | Type | Required | Default | Description |
|---|---|---|---|---|
| `mode` | `string` | No | `shared` | `shared` — pooled workers. `isolated` — this agent's own container (requires `dependencies`). |
| `tier` | `string` | No | `ephemeral` | `ephemeral` — a fresh task per run, no idle cost, ~30–60s cold start. `warm` — a standing slot, no cold start. |

```yaml
agents:
  - name: doc_intake
    execution:
      mode: isolated       # shared (default) | isolated
      tier: ephemeral      # ephemeral (default) | warm
```

Billing follows the tier: shared **1.0×**, ephemeral **1.5×** (covers cold
start), reserved capacity **0.7×** plus a monthly fee. The multiplier applied
to each run is shown on your usage lines.

---

## See also

- [Runtime Overview](/docs/runtime/overview) -- how the runtime executes agents
- [Execution Context](/docs/runtime/execution-context) -- the `ctx` object available in tools and workflows
- [Workflow Envelope](/docs/runtime/workflow-envelope) -- checkpoint/resume behavior
- [Backends](/docs/runtime/backends) -- pluggable runtime backends


--------------------------------------------------------------------------------
# Ship a Claude Agent to Your Team
URL: https://waxell.ai/docs/runtime/ship-claude-agent
Description: Take a claude_agent_sdk / Claude Code agent you built locally and publish it to Waxell so your whole team can use it in prod — governed, observable, and per-user.
--------------------------------------------------------------------------------

# Ship a Claude Agent to Your Team

You built a great Claude agent locally — a system prompt, a few tools, maybe a
`.claude/agents/*.md`. It works on your laptop. But your teammates can't use it,
there's no governance, no shared traces, and no way to run it *as* a specific
end-user with *their* own keys.

This tutorial publishes that local agent to Waxell as a declarative
`claude_agent_sdk` agent. When you're done it runs on the governed
isolation-runner, lives at `https://<your-tenant>.waxell.dev/agents/<name>`, and
can resolve **per-end-user** credentials host-side — so the model never sees a
raw secret.

**Let Claude do it for you**
Run `wax claude-init` once in your project. It installs the
**ship-claude-agent-to-waxell** skill, so Claude Code can drive this whole
tutorial for you — just say "deploy this agent to Waxell."


## Prerequisites

```bash
pip install waxell                 # ships the `wax` CLI
wax login                          # or configure a profile (see below)
wax whoami                         # ALWAYS verify tenant + identity first
```

Pushing to prod uses a **profile** that points at your prod tenant, e.g.
`wax -p acme_prod …`. Verify with `wax whoami -p acme_prod`. Don't assume the
default profile is prod.

## 1. Describe the agent in `waxell.yaml`

A `claude_agent_sdk` agent on Waxell is declarative. Translate your local agent
into a `waxell.yaml` at the root of your project:

```yaml
version: 1

defaults:
  framework: claude_agent_sdk
  owner: you@acme.com

agents:
  - name: release_notes
    version: 1.0.0                 # bump on every re-push
    display_name: Release Notes Writer
    description: Drafts release notes from a changelog.
    model: claude-3-5-sonnet
    tools: [Read, Glob, Grep, WebFetch]
    system_prompt: |
      You write crisp, user-facing release notes from a raw changelog.
      Group changes by theme, lead with user impact, skip internal churn.
    signals:
      - name: write_release_notes
        source_type: webhook
        description: Draft release notes for a version.
        schema:
          prompt:
            type: string
            description: The changelog + version to summarise
    max_turns: 12
    max_budget_usd: 4.0
```

- **`tools`** are the Claude Code built-ins (`Read`, `Write`, `Edit`, `Bash`,
  `Glob`, `Grep`, `WebSearch`, `WebFetch`, …). List only what the agent needs.
- **`signals`** are how the agent is triggered; the `schema` is its input.
- Paste your local **system prompt** in directly.

## 2. Validate and publish

```bash
wax -p acme_prod push --dry-run          # validate the manifest
wax -p acme_prod push --include-source   # publish (source travels with it)
wax -p acme_prod agents info release_notes
```

That's it — the agent is live at
`https://acme.waxell.dev/agents/release_notes`. Your team can see its spec,
bindings, traces, cost, and governance.

## 3. Run it

```bash
wax -p acme_prod signals send write_release_notes \
    --data '{"prompt":"v2.1: added SSO, fixed 3 billing bugs..."}'
wax -p acme_prod runs list --agent release_notes
wax -p acme_prod runs show <run_id>
```

The run's **Setup** span shows exactly what the agent was provisioned with —
allowed tools, domains, identity. That's your first stop when debugging.

## 4. Make it per-user (optional)

The payoff of running on Waxell instead of a laptop: the same agent can run **as
a specific end-user**, reasoning on *their* LLM key and calling tools with
*their* tokens — all resolved host-side.

**a) Register the end-user** (maps your app's user id to a Waxell sub-user):

```bash
wax -p acme_prod end-users create --tenant-sub-user-id alice \
    --email alice@acme.com --display-name "Alice"
```

**b) Set that user's secrets** (their own Anthropic key, a tool token). These
are write-only and encrypted at rest — you set/rotate but never read them back:

```bash
wax -p acme_prod end-users set-secret alice ANTHROPIC_API_KEY sk-ant-…
wax -p acme_prod end-users set-secret alice GITHUB_TOKEN ghp_…
wax -p acme_prod end-users list-secrets alice        # keys only, never values
```

**c) Run as Alice** — your backend stamps a signed `_sub_user_identity` on the
signal (stamp **both** `sub_user_id` and `user_id`):

```json
POST /api/v1/signals/write_release_notes/
{
  "prompt": "...",
  "_sub_user_identity": {"sub_user_id": "alice", "user_id": "alice", "email": "alice@acme.com"}
}
```

The run now reasons on Alice's key and uses Alice's tokens. Resolution
precedence is **SubUser → Tool → Agent → Tenant** — a user's own credential
overrides every shared default, and one user can never resolve another's.

## Gotchas

- **Bump `version:` on every re-push** — re-pushing the same version with changed
  content fails with `semver_conflict`.
- **Don't hardcode tool names** in the prompt; the platform may expose a tool as
  `mcp__…__<name>`. Say "use the available GitHub tool."
- **`wax whoami` first, always** — the #1 cause of "it deployed to the wrong
  place" is the wrong profile.

## What's next

- [Build & push a custom tool](./custom-tools.md) — give the agent a capability
  the built-ins don't cover.
- Add a **per-user tool integration** (a "connect your GitHub" domain) so the
  agent acts on third-party APIs as the end-user.
- Wire **governance policies** to the agent (see the Governance tutorial).
- Explore the **Token Lab** in the workspace — a live reference for the whole
  per-end-user credential flow.


--------------------------------------------------------------------------------
# Build & Push a Custom Tool
URL: https://waxell.ai/docs/runtime/custom-tools
Description: Write a custom tool, declare it in waxell.yaml, and wire it into an agent so the whole team's agents can call it.
--------------------------------------------------------------------------------

# Build & Push a Custom Tool

Built-in tools (`Read`, `Glob`, `WebFetch`, …) cover a lot, but eventually your
agent needs to do something specific — hit your internal API, transform data a
particular way, look something up. That's a **custom tool**: you write it once,
push it to Waxell, and any of the tenant's agents can reference it by name.

This is the companion to [Ship a Claude Agent](./ship-claude-agent.md) — same
`wax` CLI, one extra artifact.

## 1. Write the tool

A tool is a folder with a Python file and an optional `TOOL.json` manifest:

```
weather/
├── TOOL.json          # name + metadata (optional)
└── tool.py            # the tool source — this filename matters
```

**The entry file must be `tool.py`**
The loader looks for `tool.py`, falling back to any `.py` whose filename
contains `tool`. A file named `weather.py` is **not** found — the push succeeds,
the tool never registers, and the agent simply runs without it. Name it
`tool.py`.


`tool.py` — a `@tool`-decorated function. Type the params; the docstring becomes
the description the model sees:

```python
from waxell_runtime import tool


@tool
def get_weather(ctx, city: str, units: str = "celsius") -> dict:
    """Get the current weather for a city. Units: celsius or fahrenheit."""
    # Pure-Python logic + ctx.* helpers (domains, secrets, the LLM router).
    # Do real I/O through a domain — see the v1 limits below.
    return {"city": city, "temp": 21, "units": units, "conditions": "clear"}
```

`TOOL.json` (optional — `--name` and the folder name are fallbacks):

```json
{
  "name": "get_weather",
  "display_name": "Get Weather",
  "description": "Current weather for a city."
}
```

## 2. Declare it in your manifest and push

Declare the tool alongside your agent in `waxell.yaml` and push them together:

```yaml
version: 1

tools:
  - name: get_weather
    version: "1.0.0"
    description: Current weather for a city.
    source: ./weather              # the folder containing tool.py

agents:
  - name: trip_planner
    version: "1.1.0"               # bump on every push — versions are immutable
    framework: claude_agent_sdk
    tools: [Read, WebFetch, get_weather]   # built-ins + your custom tool
```

```bash
wax push --dry-run       # validate first
wax push --include-source
```

`--include-source` is required — without it the tool is registered without its
code and cannot execute.

**Use the manifest, not `wax push-tool`, for adapter agents**
`wax push-tool` writes to a different store than the one the model-facing tool
bridge reads. A tool pushed that way is invisible to `claude_agent_sdk`,
`pydantic_ai`, `langgraph`, `crewai` and `autogen` agents — it registers cleanly
and the model never sees it. Declaring the tool in `waxell.yaml` and pushing the
manifest is the path that works for every framework.


Confirm it registered:

```bash
wax registry list tool                  # `kind` is positional, not --kind
wax agents info trip_planner            # the tool should appear under bindings
```

### What the platform checks on push

- **Obvious dangerous calls are rejected** — `os.system(`, `subprocess.Popen`,
  `subprocess.run`, `subprocess.call`, `eval(`, `exec(`, `__import__`. This is a
  substring scan, not a sandbox: treat it as a guardrail against accidents, not
  a security boundary. Tool source runs in the worker process, so **only push
  code you trust**.
- **No third-party dependencies in v1** — keep it stdlib + `ctx`. For external
  APIs, call them through a **domain** (`await ctx.domain(...)`), which is the
  governed, per-user-auth boundary. See [Domains](./domains.md).

## 3. How it runs

When the agent runs, the platform surfaces your tool to the model — for
`claude_agent_sdk` agents under the MCP bridge — and executes it **host-side**,
so secrets and domains resolve under governance rather than in the model.

## Versioning & updates

Bump the tool's `version:` in `waxell.yaml` and push again:

```yaml
tools:
  - name: get_weather
    version: "1.0.1"        # bumped
    source: ./weather
```

```bash
wax push --include-source
```

Each push is a new immutable version — re-pushing the same version is rejected
with `409 semver_conflict`. Bump the **agent's** version too when you change
which tools it binds.

## Gotchas

- **Bump the agent `version:`** when you re-push the agent after adding a tool.
- **v1 = pure Python, no deps.** Need a library or a network call? Put that
  behind a **domain** and call it with `ctx.domain(...)` — that's also where
  per-end-user auth lives.
- **Don't hardcode the tool name in the system prompt** — the platform may
  expose it as `mcp__…__get_weather`; tell the agent to "use the weather tool."

## Next

- [Ship a Claude Agent](./ship-claude-agent.md) — the full publish flow.
- [`@tool` Decorator reference](../sdk/tool-spec.md) — params, approval gates,
  built-in tools.


--------------------------------------------------------------------------------
# Register a Domain
URL: https://waxell.ai/docs/runtime/domains
Description: Register a domain so your agents can call back into your own app — ctx.domain("company", "get", ...) routed to your HTTPS endpoints, with per-end-user identity and governance.
--------------------------------------------------------------------------------

# Register a Domain

A **domain** is how an agent calls back into *your* application. When an agent
runs `ctx.domain("company", "get", company_id="...")`, Waxell routes it to an
HTTPS endpoint you control — `POST {base_url}/domains/company/get` — with auth,
per-end-user identity headers, and policy governance applied at the boundary.

For SDK agents (e.g. the Claude Agent SDK), each domain action is surfaced as a
tool named `<domain>__<action>` (e.g. `company__get`). The agent calls the tool;
Waxell dispatches the HTTP call.

> Domains are the inbound→your-app counterpart to [custom tools](./custom-tools).
> A tool is code Waxell runs; a domain is an endpoint Waxell calls.

## Register via the API

Register (or update) a domain with **your own tenant API key** — no Waxell
engineer or `manage.py` required:

```bash
curl -X POST https://api.waxell.dev/api/v1/tenant-domains/push/ \
  -H "Authorization: Bearer $WAXELL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "domain_name": "company",
    "set_default": true,
    "spec": {
      "name": "company",
      "auth": { "type": "bearer",
                "config": { "header": "Authorization", "prefix": "Bearer",
                            "token": "YOUR_SHARED_SECRET" } },
      "endpoint": { "base_url": "https://app.example.com",
                    "path_template": "/domains/{domain}/{action}" },
      "request": { "timeout_seconds": 30 },
      "per_user_auth": true,
      "actions": [
        { "name": "get",
          "request_schema": {
            "type": "object",
            "properties": { "company_id": { "type": "string" } },
            "required": ["company_id"] } },
        { "name": "set_enrichment",
          "request_schema": {
            "type": "object",
            "properties": { "company_id": { "type": "string" },
                            "source": { "type": "string" },
                            "data": { "type": "object" } },
            "required": ["company_id", "source", "data"] } }
      ]
    }
  }'
```

Re-pushing the same `domain_name` creates a new version (the registry is
versioned). The call returns `{domain_name, version, is_default, action}`.

## Always declare `request_schema` per action

This is the single most common mistake. The `request_schema` becomes the tool's
input schema that the model sees. **Omit it and the agent has no idea what fields
the action needs** — it will leave out required parameters and your endpoint will
reject the call (`422`). Mirror your endpoint's request body exactly: list every
field under `properties` and the mandatory ones under `required`.

## Auth and per-end-user identity

- `auth.type: "bearer"` sends `Authorization: Bearer <token>` on every call, where
  `token` is `auth.config.token`. Use this as a shared secret so only Waxell can
  reach your `/domains/*` endpoints (verify it on your side; fail closed).
- `per_user_auth: true` additionally sends signed end-user identity headers
  (`X-Waxell-Sub-User-Id`, `X-Waxell-Sub-User-Email`, and a short-lived
  Ed25519-signed `X-Waxell-Sub-User-Token` you can verify against the tenant's
  JWKS) — so your endpoint knows *which* end user the agent is acting for, without
  the model ever seeing a credential.

## Your endpoint

Implement one route per action at `{base_url}/domains/{domain}/{action}`,
accepting the action's request body as JSON and returning JSON. Validate the
shared-secret bearer, and (if you opted into per-user auth) the sub-user token.

## Verify

After pushing, run an agent that calls the domain and check the run trace: the
`<domain>__<action>` tool span should show your endpoint's response. If you see a
generic dispatch failure, check (1) every action has a `request_schema`, (2) your
endpoint accepts the shared-secret bearer, and (3) the request body matches the
schema.


--------------------------------------------------------------------------------
# Execution Context
URL: https://waxell.ai/docs/runtime/execution-context
Description: What ctx gives your agent at runtime — calling your application, the LLM, tools, memory, secrets, and pausing for humans.
--------------------------------------------------------------------------------

# Execution Context

Every decorated method receives `ctx` as its second argument, after `self`. It's
how your agent reaches everything outside itself: your application, the LLM,
tools, memory, secrets — and how it pauses for a human.

```python
from waxell_runtime import agent, workflow


@agent(name="report_builder", description="Builds a report")
class ReportBuilder:

    @workflow("build")
    async def build(self, ctx, quarter: str):
        data = await ctx.domain("finance", "quarterly", quarter=quarter)
        return await ctx.llm.generate(prompt=f"Summarize: {data}")
```

**Everything that reaches outside the agent is `async`**
`ctx.domain`, `ctx.tool`, `ctx.llm.generate`, `ctx.spawn`, `ctx.ask_user` and
friends are coroutines. Forget the `await` and you get a coroutine object
instead of your result — Python only warns, so it fails quietly.


For editor autocomplete, type-hint it:

```python
from waxell_runtime import WorkflowContext, ToolContext

@workflow("build")
async def build(self, ctx: WorkflowContext, quarter: str): ...

@tool
async def fetch(self, ctx: ToolContext, id: str) -> dict: ...
```

---

## Reaching your application

`ctx.domain()` is the main one. It calls a named action on a domain you've
registered; your service answers over a shared-secret callback — so the agent
never holds your credentials.

```python
company = await ctx.domain("company", "get", company_id="acme")
```

See [Domains](./domains.md).

## Calling a tool

`ctx.tool()` invokes another `@tool` on this agent. Each call is a **durable
checkpoint** — see [Durable Execution](./workflow-envelope.md).

```python
data = await ctx.tool("fetch_company", domain="acme.com")
```

## The LLM

Routed, cost-attributed and policy-checked. You never hold a provider key.

```python
summary = await ctx.llm.generate(
    prompt="Summarize this contract",
    task="contract_summary",     # names the task for routing + cost analytics
    max_tokens=2000,
)
```

## Secrets

```python
key = ctx.secrets.get("STRIPE_API_KEY")
```

Resolved from the tenant's configured secret provider. Don't read `os.environ` —
it works locally and fails in deployment.

## Memory

```python
await ctx.memory.set("last_seen", value)
value = await ctx.memory.get("last_seen")
```

`ctx.scratchpad` is the short-lived working tier for a single run. See
[Working Memory](./working-memory.md).

---

## Pausing for something external

These suspend the run without holding compute. It resumes when the thing
arrives.

```python
answer = await ctx.ask_user("Approve this $40k discount?")
event  = await ctx.wait_for_signal("po_received", timeout_seconds=86400)
await ctx.sleep(3600)
```

A paused run shows as `PAUSED` with a reason — see
[Durable Execution](./workflow-envelope.md).

## Fanning out

```python
child   = await ctx.spawn("enrichment_agent", inputs={"lead_id": lead_id})
results = await ctx.spawn_many("scorer", [{"id": i} for i in ids])
```

Children are governed and traced like any other run, and appear under the
parent in the lineage graph.

---

## Identity and provenance

| Attribute | What it is |
|---|---|
| `ctx.inputs` | The inputs this run was started with |
| `ctx.workflow_id` | This run's id |
| `ctx.parent_workflow_id` | The run that spawned this one, if any |
| `ctx.root_workflow_id` | The top of the spawn tree |
| `ctx.metadata` | Free-form metadata carried with the run |
| `ctx.sub_user_id` / `ctx.sub_user_email` | The end user this run is acting **as**, when it carries a signed sub-user identity |

`sub_user_*` is what makes per-end-user auth work: the agent acts as a specific
person, and your domain endpoint can verify that cryptographically.

## Observability

```python
ctx.log_step("validated_input", {"rows": len(rows)})   # note: not async
await ctx.checkpoint()                                 # force a save point
```

`ctx.budget` exposes the run's spend against its cap.

---

## Related

- [How the Runtime Works](./how-it-works.md) — the full path from signal to result
- [Domains](./domains.md) — the callback interface into your application
- [Custom tools](./custom-tools.md) — writing `@tool` methods
- [Durable Execution](./workflow-envelope.md) — checkpoints, resume, run states


--------------------------------------------------------------------------------
# Execution Tiers
URL: https://waxell.ai/docs/runtime/execution-tiers
Description: Choose where your agents run — shared workers, dedicated warm slots, or a self-contained container built from your own dependencies. Same governance, same telemetry, same audit trail at every tier.
--------------------------------------------------------------------------------

# Execution Tiers

Every agent run needs somewhere to execute. Waxell gives you three options,
and you choose per agent — in the same `waxell.yaml` that defines the agent
itself.

The governance model does not change between them. Policies evaluate, spans
are written, costs are attributed, and the audit trail is complete at every
tier. What changes is **isolation, dependencies, and cost**.

| Tier | Runs on | Choose it when | Billing |
|---|---|---|---|
| **Shared** | Pooled workers | Default. Fast, cheap, no cold start. | 1.0× |
| **Warm slot** | A Fargate slot claimed for one run | You need per-execution isolation with per-tenant credentials. | 1.0× (or 0.7× reserved) |
| **Self-contained** | A container built from *your* dependencies | Your agent needs libraries or system packages the platform image doesn't carry. | 1.5× |

You do not have to pick up front. Agents start shared, and moving one is a
manifest change plus a push.

---

## Shared workers (default)

Your agent runs on the pooled worker fleet. No cold start, lowest cost, and
it is the right answer for most agents most of the time.

Nothing to configure — this is what you get by default.

---

## Warm slots — per-execution isolation

For regulated workloads (insurance, financial services, healthcare), shared
workers can be made *safe* but are hard to *prove* safe to a CISO or an
auditor. Warm slots move each execution onto its own Fargate task with:

- **STS-tagged credentials** scoped to your tenant, minted per run
- **Per-cell KMS encryption context** — a wrong-tenant decrypt is denied by
  KMS itself, not by application code, and the denial is an auditable event
- **No cross-run reuse** — a slot handles one execution, then drains

### Selecting it

Your tenant's sensitivity tier decides the floor:

| Sensitivity | Routing |
|---|---|
| `standard` | Shared workers. Default for new tenants. |
| `sensitive` | Warm slot minimum — per-execution isolation, per-tenant credentials. |
| `regulated` | Warm slot minimum, plus an always-on pool (no scale-to-zero). |

You can also raise a single agent without moving the whole tenant:

```yaml
agents:
  - name: claims_reviewer
    execution:
      tier: warm
```

### Reserved capacity

If you want slots held for you rather than shared with other tenants, a
**dedicated cluster** reserves them. You pay a monthly fee for the standing
capacity, and per-run compute drops to **0.7×** — cheaper per run the more
capacity you commit to.

---

## Self-contained agents — bring your own dependencies

Some agents need libraries the platform image doesn't carry: a PDF parser, a
legacy document converter, a domain-specific SDK. Rather than asking us to
add them to a shared image, declare them and the platform builds your agent
its own container.

```yaml
agents:
  - name: doc_intake
    framework: pydantic_ai
    dependencies:
      apt: [antiword]                                  # system packages
      pip: [pypdf>=4.0, python-docx>=1.1, olefile]     # python packages
      python: "3.11"
    execution:
      mode: isolated
      tier: ephemeral
```

On `wax push`, Waxell builds an image layered on the governed runtime base,
registers it, and routes that agent's runs to it. The agent gets its
dependencies; you get the same policy evaluation, span tree, and cost
attribution as every other tier.

**Ephemeral by default.** Each run launches a fresh task and exits — no warm
pool, no idle cost. You pay for the run, plus a **1.5×** multiplier that
covers the cold start (image pull and boot). We charge a flat premium rather
than metering startup, so the number is predictable.

### What to expect

- **Cold start of roughly 30–60 seconds** per run, before your agent's own
  work. Excellent for document pipelines and background processing; noticeable
  for anything a person is watching. If latency matters, use a warm slot.
- **Rebuilds are automatic** when your dependencies change. Unchanged
  dependencies reuse the existing image.
- **A failed build does not break the agent** — it keeps running on the
  shared tier until the build succeeds.

### Choosing between isolated and warm

| | Self-contained (isolated) | Warm slot |
|---|---|---|
| Custom dependencies | Yes | No — platform image |
| Cold start | ~30–60s | None |
| Idle cost | None | Pool or reserved |
| Best for | Document/batch pipelines | Interactive, latency-sensitive |

---

## Verifying which tier a run used

Every run records the tier it executed on. You can see it three ways:

1. **Observe → Runs.** Open any run; the execution tier is on the run header.
2. **Traces.** The root span carries `waxell.execution.tier` —
   `tier-0-celery`, `tier-1-warm-fargate`, or `tier-3-isolated`.
3. **Billing → Usage.** Compute lines carry the runtime disposition
   (`event_driven`, `dedicated_cluster`, `ephemeral`) and the multiplier
   applied, so the rate on your invoice traces back to where the run
   executed.

For self-contained agents, **Agents → [your agent] → Build** shows the build
status, the image, and which dependencies were installed.

---

## Failure behaviour

Worth knowing before you need it:

- **Pool exhausted.** Burst traffic can empty the warm pool. Runs fall back to
  shared workers with a logged warning rather than failing.
- **Credential expiry mid-run.** Slot credentials last an hour and refresh
  transparently. If refresh fails, the run fails cleanly with `claim_failed`
  rather than continuing with stale credentials.
- **Encryption-context mismatch.** A wrong-tenant decrypt attempt is denied by
  KMS, the run stops, and it is raised as a security incident — never a silent
  retry.
- **Task crash** (OOM, host failure). The run is marked failed and downstream
  signals fire. A reconciler sweeps any task that dies before finalising, so a
  run does not sit unfinished.
- **Build failure** on a self-contained agent. The agent continues on the
  shared tier; the build error is on the agent's Build panel.

---

## Current limits and what's next

Stated plainly, so you can plan around them:

- **Bulk fan-out.** Self-contained agents launch one task per run. A very
  large backfill dispatched all at once can hit cloud API rate limits. Normal
  intake volume is unaffected; for large backfills, stagger the load.
- **First isolated run for a new tenant** requires a one-time log-group
  provisioning step, handled during onboarding.
- **Customer-managed keys (BYOK)** and **per-tenant egress allowlists** are on
  the roadmap; today the platform uses per-cell managed keys and a restricted
  egress policy.
- **Sub-second cold start** (microVM-class) is planned. The tier interface was
  designed so this arrives as an additional tier, not a migration.

---

## Related

- [`waxell.yaml` reference](./waxell-yaml-reference.md) — full `dependencies`
  and `execution` schema
- [Execution context](./execution-context.md) — what your agent can access at
  runtime
- [Domains](./domains.md) — how agents call back into your application


--------------------------------------------------------------------------------
# Working Memory
URL: https://waxell.ai/docs/runtime/working-memory
Description: The Waxell runtime's memory tiers (working, session, episodic, semantic) — how domain results are captured to a scratchpad, referenced via $ref handles, and isolated per tenant.
--------------------------------------------------------------------------------

# Working Memory

The Waxell runtime ships with a multi-tier memory primitive that addresses the
context-window explosion problem in agentic loops. Domain-call results capture
into a fast scratchpad under a reference handle, and handles resolve back to full
data on the runtime side before a call is dispatched -- so the bytes move
tool-to-tool without passing through the LLM's conversation.

**Scope**
This page covers the **native runtime** — agents you build with `@agent`,
`@workflow`, and `ctx.domain`. Adapter-backed agents (`claude_agent_sdk`,
`pydantic_ai`, LangGraph, CrewAI) manage context through their own framework;
talk to us if you want Waxell-managed memory on one of those.


## The problem it solves

A typical agentic loop has the LLM call a tool, see the full result, then decide the next action. After 10 turns of fetching prices, dividends, research notes, and benchmarks, the conversation history is 60KB+ of JSON the LLM has to re-read every turn. You either:

- Run out of context window
- Burn tokens re-reading data you already have
- Have the LLM hallucinate data it already fetched but can't see anymore
- Truncate history and lose information

Working memory inverts this. Domain results go to a scratchpad keyed by
`tool_name.call_number`, and a call's inputs can carry a handle
(`$ref:market_data_fetch_prices.1`) instead of the data itself. The runtime
resolves the handle before dispatching, so the actual bytes flow tool-to-tool
without entering the conversation.

Your workflow code passes the handles. The name is derived from the call you
just made, so you always know it — see [how refs are named](#how-a-ref-is-named).

**Measured impact** (finance portfolio_analyst, 6-phase pipeline):

| Metric | Without scratchpad | With scratchpad |
|---|---|---|
| Tokens per turn | 6K–28K (growing) | ~7K (flat) |
| Turns to complete | Runs out of context | 16 turns |
| Tool result re-reads | Every turn | Zero |

## The tiers at a glance

| Tier | Backend | Lifecycle | Purpose |
|---|---|---|---|
| **`working`** | Redis | Per-run, expires after 1h | Scratchpad for in-flight tool results, captured automatically |
| **`session`** | Postgres | 24h default | Conversation/plan state within a user session |
| **`episodic`** | Postgres | TTL-based | Cross-run state: conversation history, cached results |
| **`semantic`** | pgvector | Indefinite | Long-term facts with vector search |

All four tiers share the same `MemorySpec` API and the same tenant isolation guarantees. The `tier=` field selects the backend and lifecycle.

## Python: declaring memory on an agent

Use the convenience constructors from `waxell_sdk.core.specs.memory_spec`:

```python
from waxell_sdk import agent
from waxell_sdk.core.specs.memory_spec import (
    scratchpad,
    conversation_memory,
    semantic_memory,
)

@agent(
    name="portfolio_analyst",
    memory={
        # Tier 1: captures every domain call during a run
        "scratchpad": scratchpad(),

        # Tier 2: per-user conversation across runs
        "conversation": conversation_memory(scope="user_id", ttl="30d"),

        # Tier 3: per-client facts the agent has learned
        "client_context": semantic_memory(
            scope=["user_id", "portfolio_id"],
            description="Facts about this advisor's relationship with this client",
        ),
    },
    ...
)
class PortfolioAnalyst:
    ...
```

Or use `MemorySpec` directly when you need control:

```python
from waxell_sdk.core.specs.memory_spec import MemorySpec

memory = {
    "scratchpad": MemorySpec(
        tier="working",
        scope=["run_id"],
    ),
    "cached_analysis": MemorySpec(
        tier="episodic",
        scope=["user_id", "portfolio_id"],
        type="dict",
        ttl="24h",
    ),
}
```

## YAML: declaring memory in waxell.yaml

```yaml
memory:
  scratchpad:
    tier: working
    scope: [run_id]

  conversation:
    tier: episodic
    scope: user_id
    type: list
    ttl: 30d
    max_items: 100

  client_knowledge:
    tier: semantic
    scope: [user_id, portfolio_id]
    searchable: true
    description: Facts about this client relationship
```

For all available fields and validation rules, see the [waxell.yaml memory reference](./waxell-yaml-reference#memory).

## How working memory captures domain results

Capture happens on `ctx.domain(...)` calls. Every successful one is stored:

1. The workflow calls `ctx.domain("market_data", "fetch_prices", tickers=["AAPL"], days=90)`
2. The callback returns 15KB of OHLC data
3. The runtime stores it in the scratchpad as `market_data_fetch_prices.1`, along
   with a generated one-line summary of its shape
4. The next call passes the handle instead of the payload:
   ```python
   await ctx.domain(
       "analytics", "calculate_performance",
       prices="$ref:market_data_fetch_prices.1",
   )
   ```
5. The runtime resolves the handle to the full data before dispatching, so
   `analytics.calculate_performance` receives the real OHLC dict

### How a ref is named

`{tool_name}.{call_number}`, with `.` and `:` in the tool name replaced by `_`.
So the first call to `market_data.fetch_prices` is
`$ref:market_data_fetch_prices.1`, and a second call in the same run is `.2` —
the number counts **calls to that tool**, not conversation turns.

An unresolvable handle passes through as a literal string rather than raising,
so check the name if a tool receives `"$ref:..."` instead of data.

**Input signals are seeded too.** Before the loop starts, each dict/list signal
is captured as `input_<name>.1`, so a workflow can hand its own inputs to a
domain call by reference rather than re-serialising them.

**Repeated calls are served from the scratchpad.** When a domain action is marked
idempotent, the runtime looks the call up by `(action, args)` before dispatching
and returns the captured result if it finds one. This is what makes a resumed run
cheap: the work already done is not redone.

## Tenant isolation

Two layers of isolation, either of which catches a bug in the other:

**Physical layer.** The Redis backend prefixes every key with `tenant:{tenant_id}:`. The `_tenant_context()` context manager extracts `tenant_id` from the scope key and sets the ContextVar around each Redis operation -- so isolation holds even when the Celery worker has no upstream tenant context set.

**Logical layer.** The scope key itself always starts with `tenant_id`, enforced by `MemoryScopeResolver` regardless of what dimensions the developer declares in `scope=[...]`.

Result: a memory write under tenant A cannot be read by tenant B even if scope keys collide, even if a worker has stale context, even if the dev forgot to include `tenant_id` in the scope list. `tenant_id` is always prepended automatically.

## Scope dimensions

Multi-dimensional scoping composes on top of `tenant_id`:

| Dimension | Source | Use case |
|---|---|---|
| `tenant_id` | Always (automatic) | Outermost boundary |
| `agent` | Agent name | Per-agent within tenant |
| `agent_version` | Agent version | Per-version state |
| `user_id` | Sub-user identity | Per end-user |
| `user_group` | Sub-user identity | Per user group |
| `session_id` | Conversation session | Per conversation |
| `workflow` | Workflow name | Per workflow within agent |
| `run_id` | Execution run | Per execution (ephemeral) |
| `channel_id` | Slack/chat | Per chat channel |
| `thread_ts` | Slack | Per thread |

Combine them as needed:

```python
# Per-(advisor, client) cached analysis
MemorySpec(
    tier="episodic",
    scope=["user_id", "portfolio_id"],
    type="dict",
    ttl="24h",
)
```

## When to use which tier

| Need | Tier | Why |
|---|---|---|
| Pass a large result between steps without putting it in the prompt | `working` | Captured on every domain call; refs resolve before dispatch |
| Remember the last 100 messages per user | `episodic` (`conversation_memory()`) | Postgres-backed, TTL-eviction |
| Cache an expensive calculation per (advisor, client) for 24h | `episodic` | Multi-dimensional scope, TTL |
| Conversation/plan state within a session that spans multiple workflow runs | `session` | First-class tier with 24h default TTL |
| Learn facts about a client over time and recall by similarity | `semantic` (`semantic_memory()`) | Vector search, indefinite retention |

## Typed memory schemas

Memory values are stored and returned as plain dicts. When a slot holds
something structured, declare the shape with Pydantic and validate on read:

```python
from pydantic import BaseModel


class ClientNote(BaseModel):
    summary: str
    risk_flags: list[str]


note = ClientNote.model_validate(await ctx.memory.get("client_note"))
```

Validating at the boundary means a shape change surfaces where you can handle
it, rather than as a `KeyError` three calls later.

## Common gotchas

- **`tenant_id` is implicit.** Don't include it in your `scope=[...]` list -- the runtime always prepends it. Listing it explicitly works but is redundant.
- **Working tier ignores `type`.** Scratchpad entries are always `ScratchpadEntry` dataclass values; `type="dict"` / `type="list"` only applies to episodic/session tiers.
- **One scratchpad per run, always on.** Every run gets one; there's nothing to enable.
- **`$ref` resolution is recursive.** Pass `$ref:foo.1` as a value, a nested dict field, or inside a list -- the runtime walks the structure and resolves all refs.
- **Local dev uses in-memory backends.** `InMemoryWorkingMemory` mimics Redis for tests; the Redis backend takes over in production via the infra `setup.py` ready hook.

## Reference

- [waxell.yaml memory field reference](./waxell-yaml-reference#memory) -- field-by-field YAML schema
- [Execution Context](./execution-context) -- how `ctx.scratchpad` is plumbed through agent runs
- [Workflow Envelope](./workflow-envelope) -- run lifecycle that drives scratchpad TTL


--------------------------------------------------------------------------------
# Durable Execution
URL: https://waxell.ai/docs/runtime/workflow-envelope
Description: How Waxell checkpoints a run so it can survive a crash and resume — what's a checkpoint, what a run's states mean, and which ones are terminal.
--------------------------------------------------------------------------------

# Durable Execution

Long-running agents outlive the processes that start them. `WorkflowEnvelope` is
the execution boundary that makes a run survivable: it checkpoints progress so a
crash, a redeploy, or a pause for human input resumes where it left off instead
of starting over.

You don't construct an envelope — the runtime wraps your workflow in one. What
you control is **where the checkpoints fall.**

## What creates a checkpoint

Every **tool call** is a durable step:

```python
from waxell_runtime import agent, workflow, tool


@agent(name="report_builder", description="Builds a quarterly report")
class ReportBuilder:

    @tool
    async def fetch_data(self, ctx, quarter: str) -> dict:
        return await ctx.domain("finance", "quarterly", quarter=quarter)

    @tool
    async def summarize(self, ctx, data: dict) -> str:
        return await ctx.llm.generate(prompt=f"Summarize: {data}")

    @workflow("build")
    async def build(self, ctx, quarter: str):
        # Each ctx.tool(...) is a checkpoint boundary.
        data = await ctx.tool("fetch_data", quarter=quarter)

        # If the process dies here, resuming replays the line above from the
        # saved result rather than re-fetching.
        summary = await ctx.tool("summarize", data=data)

        return {"quarter": quarter, "summary": summary}
```

Two things to note, because they're the usual mistakes:

- **Everything is `async`.** `ctx.tool`, `ctx.domain`, and `ctx.llm.generate`
  are all awaited.
- **Checkpointing is automatic.** There's no `durable=True` flag and no separate
  "durable workflow" type — a workflow's tool calls are its checkpoints.

Work done *between* tool calls is not checkpointed. If a computation is
expensive enough that you'd hate to repeat it, put it in a `@tool` so its result
is saved.

## How resume works

1. Before a step runs, the runtime records that it's starting.
2. After it completes, the result is checkpointed.
3. On resume, any step with a saved result is **replayed from that result** — the
   function does not run again.
4. Execution continues from the first step without one.

This is why tools should be the unit of work: the checkpoint granularity is the
tool boundary.

## Run states

A run's lifecycle, from `RunState`:

| State | Meaning | Terminal |
|---|---|---|
| `PENDING` | Accepted, not yet started | |
| `RUNNING` | Executing | |
| `PAUSED` | Waiting on something external | |
| `RESUMING` | Picking back up from a checkpoint | |
| `COMPLETED` | Finished successfully | ✅ |
| `FAILED` | An error inside the agent | ✅ |
| `BLOCKED` | Stopped by governance — a policy or budget | ✅ |
| `INTERRUPTED` | The process died (SIGTERM, OOM, hard kill) before finishing | ✅ |

**Poll for all four terminal states**
`BLOCKED` and `INTERRUPTED` are terminal. Code that waits for only `COMPLETED`
or `FAILED` will hang forever on a policy-blocked or process-killed run.


`INTERRUPTED` is deliberately distinct from `FAILED`: an agent that errored is a
different problem from a process that was killed under it, and collapsing them
hides infrastructure issues.

### Why a run is paused

`PAUSED` isn't one thing — `PausedReason` tells you what it's waiting on:

| Reason | Waiting for |
|---|---|
| `awaiting_child` | A spawned sub-agent |
| `awaiting_signal` | An inbound signal |
| `awaiting_user` | A human answer (`ctx.ask_user`) |
| `awaiting_timer` | A scheduled wake-up |
| `awaiting_approval` | A governance approval |
| `suspended_orphan` | Its parent went away |

A paused run holds no compute. It resumes when the thing it's waiting on
arrives.

## Where state is stored

The storage backend is chosen by which `RuntimeBackend` is configured, not by a
separate setting:

- **Local development** — `InMemoryBackend`, automatic, nothing to configure.
  State lives in the process, so it does not survive a restart.
- **Production** — `DjangoRuntimeBackend`, wired automatically when
  `waxell_infra` is in `INSTALLED_APPS`. Checkpoints persist to the database and
  survive restarts and redeploys.

See [Backends](./backends.md) for how that wiring happens and how to supply your
own.

## Next Steps

- **[Backends](./backends.md)** — configure or replace the runtime's backends
- **[How the Runtime Works](./how-it-works.md)** — the full path from signal to result
- **[Execution Context](./execution-context.md)** — what `ctx` gives you


--------------------------------------------------------------------------------
# Backends
URL: https://waxell.ai/docs/runtime/backends
Description: Configure runtime backends for different environments.
--------------------------------------------------------------------------------

# Runtime Backends

The Waxell Runtime uses pluggable backends for different deployment scenarios. This enables the same code to run locally and in production.

## Backend Protocols

| Protocol | Default (Runtime) | Production (Infra) |
|----------|-------------------|-------------------|
| `TokenSemaphore` | `InMemoryTokenSemaphore` | `RedisTokenSemaphore` |
| `TaskStatusProvider` | `InMemoryTaskStatusProvider` | `InMemoryTaskStatusProvider` |
| `RuntimeBackend` | `InMemoryBackend` | `DjangoRuntimeBackend` |
| `GovernanceHook` | (none) | `PolicyGovernanceHook` |

## Local Development

In local development, defaults are used automatically:

```python
from waxell_runtime import RuntimeConfig

# Uses in-memory implementations
config = RuntimeConfig.get()
```

No Redis, Celery, or Django required for local development.

## Production Configuration

Production backends are configured by the `waxell-infra` Django app:

```python
# settings.py
INSTALLED_APPS = [
    ...
    'waxell_infra',  # Automatically configures runtime backends
]
```

The infra app's `ready()` hook configures:

- `RedisTokenSemaphore` for rate limiting
- `InMemoryTaskStatusProvider` for async task tracking
- `DjangoRuntimeBackend` for persistence
- `PolicyGovernanceHook` for policy enforcement

## Custom Backends

You can implement custom backends by following the protocols:

```python
from waxell_runtime import TokenSemaphore


class MyTokenSemaphore:
    """Custom concurrency backend.

    TokenSemaphore is a runtime_checkable Protocol — implement these three
    methods and pass an instance to RuntimeConfig.configure(). All three are
    SYNCHRONOUS; the runtime calls them from its own scheduling path.
    """

    def try_acquire(self, queue: str) -> bool:
        """Atomically take a token. False when none are available."""
        ...

    def release(self, queue: str) -> None:
        """Give a token back."""
        ...

    def set_capacity(self, queue: str, count: int) -> None:
        """Set max concurrent tokens for a queue."""
        ...
```

## Next Steps

- **[Production Guide](/tutorials/production)** - Deploy with production backends
- **[Architecture Guide](/guides/architecture)** - Understand the full system


================================================================================
SECTION: MANAGED RUNTIMES
Run Waxell-governed agents on third-party agent runtimes
================================================================================

--------------------------------------------------------------------------------
# Runtimes — Deploy Waxell where your agents run
URL: https://waxell.ai/docs/runtimes/overview
Description: One-stop guide for shipping Waxell-governed agents to managed agent runtimes (Amazon Bedrock AgentCore, IBM watsonx.ai, Azure AI Foundry, Vertex AI Agent Engine, and more). Pick your runtime, ship in 5 minutes.
--------------------------------------------------------------------------------

# Runtimes

You already picked an agent runtime — **AgentCore, watsonx.ai,
Foundry, Agent Engine, or something else.** This section is the shortest path from *"we
standardized on X"* to *"we have Waxell governance + observability
on every run."*

Each page is a focused **5-minute deploy guide** for one runtime:
prereqs, the smallest possible code change, the exact `--env` /
config wiring, and verification you can copy-paste.

If your runtime hosts your Python code (most do), Waxell drops in
unchanged — `pip install waxell-observe`, `waxell.init()`, you're
done. The pages below cover the runtime-specific quirks: env-var
wiring, network egress, image versions, the things that actually
trip people up.

## Pick your runtime

<div className="runtimes-grid">

### [Amazon Bedrock AgentCore (BYO code) →](/runtimes/bedrock-agentcore)

Bring your own Python agent, deploy it to AgentCore Runtime's
managed microVM via the `agentcore` CLI. Waxell installs inside the
microVM as a normal pip dep and produces full client-side spans
with cost + token counts — same shape as running the agent on any
other Python host.

**Best for:** teams who write their agent loop in Python and want
AWS to handle the hosting, scaling, and isolation.

---

### [IBM watsonx.ai →](/runtimes/watsonx)

Wrap your `ibm-watsonx-ai` SDK calls with Waxell so every
Granite / Llama / Mistral call on watsonx Foundation Models shows
up in your governance dashboard with real cost (sourced from IBM's
published per-million-token rates).

**Best for:** teams running Python agents against watsonx
Foundation Models, whether locally, on IBM Cloud Code Engine, or
on any other host.

---

### [Azure AI Foundry (hosted agents) →](/runtimes/azure-ai-foundry)

Package your LangGraph agent as a container and register it as a
Foundry hosted agent. Waxell rides inside the container — full
client-side spans with cost + token counts plus in-path policy
enforcement, streaming natively into your dashboard with no App
Insights pull.

**Best for:** teams standardized on Azure AI Foundry who want
their custom-code agents governed the moment they deploy.

---

### [Vertex AI Agent Engine →](/runtimes/vertex-agent-engine)

Hand Agent Engine a live Python object — no Dockerfile — with
`waxell-observe` in the requirements list. Every run's full span
trace streams natively into Waxell and persists beyond Cloud
Trace's 30-day retention window.

**Best for:** teams deploying LangGraph (or ADK / custom) agents
to Google's managed Agent Engine runtime.

</div>

## Don't see your runtime?

Anywhere Python runs, Waxell runs. If your runtime hosts an
arbitrary Python entrypoint (Lambda, ECS, Cloud Run, Azure
Container Apps, Modal, Replicate, your laptop), the
[generic install guide](/observe/installation) is the path —
the runtime is invisible to Waxell.

We're prioritizing the runtimes our customers ask for most. If
yours isn't here yet, [tell us](https://community.waxell.dev/roadmap)
and it goes to the top of the list.


--------------------------------------------------------------------------------
# Amazon Bedrock AgentCore Runtime (BYO code)
URL: https://waxell.ai/docs/runtimes/bedrock-agentcore
Description: Deploy a Waxell-instrumented Python agent to Amazon Bedrock AgentCore Runtime. The SDK ships inside the microVM and produces full client-side spans plus policy enforcement — same behavior as deploying your agent to any other Python host. 5-minute deploy.
--------------------------------------------------------------------------------

# Amazon Bedrock AgentCore Runtime

> 5-minute deploy · BYO Python code · AgentCore managed microVM

This is the **BYO-code path** on Amazon Bedrock AgentCore Runtime:
you write your agent in Python, ship it via the `agentcore` CLI,
and AWS hosts it in a managed microVM. Because the microVM runs
*your* code, `waxell-observe` installs and runs inside the runtime
just like any other Python dependency — the AgentCore boundary is
transparent to the SDK.

## What you get

- Every LLM call your agent makes (Bedrock Converse / InvokeModel,
  OpenAI, Anthropic, anything) → captured as a child LLM span on
  the parent run with real model id, token counts, and cost.
- Every tool call (MCP, function tools, action group invocations
  from *inside* the agent loop) → captured as tool spans.
- Every `ctx.check_policy(...)` gate → **enforced inside the
  runtime** before the LLM/tool call fires.
- Sub-agent lineage, memory ops, custom spans → same shape as any
  other host you'd deploy the agent to.

The microVM is just another Python host. Your agent code is
unchanged; AWS handles isolation, scaling, and lifecycle.

## Prerequisites

```bash
pipx install bedrock-agentcore-starter-toolkit   # provides the `agentcore` CLI
pipx install uv                                   # required by direct_code_deploy
```

You also need:

1. AWS IAM credentials with `bedrock-agentcore-control:*` perms
   (the toolkit auto-creates the runtime execution role on first
   deploy).
2. Bedrock model access in your region for whichever foundation
   model your agent calls.

## Project layout

Scaffold a new project:

```bash
agentcore create -p mywaxellagent -t basic
cd mywaxellagent
```

Edit `pyproject.toml` to add `waxell-observe`:

```toml
[project]
name = "mywaxellagent"
version = "0.1.0"
requires-python = ">=3.12"

dependencies = [
    "waxell-observe[bedrock]>=0.1.1",
    "bedrock-agentcore>=1.0.3",
    "boto3>=1.40,<2.0",
]
```

Edit `.bedrock_agentcore.yaml` to use Python 3.12 or 3.13 (the
versions `waxell-observe` ships wheels for):

```yaml
runtime_type: PYTHON_3_13
```

Edit `src/main.py` with the canonical two-line instrumentation
pattern:

```python


waxell.init()


from bedrock_agentcore.runtime import BedrockAgentCoreApp

app = BedrockAgentCoreApp()
REGION = os.getenv("AWS_REGION", "us-east-1")


@waxell.observe(agent_name="my-byo-agent")
def chat_turn(user_message: str) -> str:
    client = boto3.client("bedrock-runtime", region_name=REGION)
    response = client.converse(
        modelId="amazon.nova-lite-v1:0",
        messages=[{"role": "user", "content": [{"text": user_message}]}],
    )
    return response["output"]["message"]["content"][0]["text"]


@app.entrypoint
async def invoke(payload, context):
    yield chat_turn(payload.get("prompt", ""))


if __name__ == "__main__":
    app.run()
```

## Deploy

`WAXELL_API_KEY` + `WAXELL_API_URL` are passed via the `--env`
flag on `agentcore deploy` — they end up as `environmentVariables`
on the AgentCore Runtime config and propagate into the microVM at
provision time.

```bash

agentcore deploy \
    --env "WAXELL_API_KEY=$WAXELL_API_KEY" \
    --env "WAXELL_API_URL=https://api.waxell.dev"
```

**Shell quoting gotcha**
If you try `WAXELL_API_KEY=... agentcore deploy --env "WAXELL_API_KEY=$WAXELL_API_KEY" ...`
in a single line, Bash expands `$WAXELL_API_KEY` *before* the
prefix assignment runs and the runtime ends up with an empty key.
**Always `export` the value first** or read it from a file with a
separate command. Verify after deploy:

```bash
aws bedrock-agentcore-control get-agent-runtime \
    --agent-runtime-id <id> --region <region> \
    --query 'environmentVariables.WAXELL_API_KEY'
```

If it returns `""`, re-deploy with the export trick above.


## Invoke

```bash
agentcore invoke '{"prompt": "what is the deepest known cave?"}'
```

Then check Waxell:

```bash
wax runs list --limit 5
```

You'll see your agent run under `agent_name=my-byo-agent` with the
same shape as `01-hello-waxell` — parent agent span, `chat
amazon.nova-lite-v1:0` LLM child span, real token counts and cost.

## Network egress

AgentCore Runtime's default `network_mode: PUBLIC` gives the
microVM NAT egress over the public internet — `api.waxell.dev`
(HTTPS) is reachable out of the box. If you switch to
`network_mode: VPC` you'll need a NAT gateway / VPC endpoint in
your VPC so the microVM can still reach Waxell.

## Policy enforcement inside the runtime

The full `@waxell.observe` + `@waxell.tool` + `ctx.check_policy(...)`
stack runs unchanged inside the microVM — there is no AgentCore-
specific shim or escape hatch. A policy denial raised during
execution aborts the agent before the offending LLM or tool call
lands, surfaces as `PolicyViolationError` to your entrypoint, and
records the violation in your Waxell governance dashboard.

## Related examples

- [`waxell-ai/waxell-agent-examples/examples/14-bedrock-agentcore-runtime-byo`](https://github.com/waxell-ai/waxell-agent-examples/tree/main/examples/14-bedrock-agentcore-runtime-byo) — full working example with deploy commands.


--------------------------------------------------------------------------------
# IBM watsonx.ai
URL: https://waxell.ai/docs/runtimes/watsonx
Description: Wrap your IBM watsonx.ai Python agent with Waxell — every Granite, Llama, and Mistral call on watsonx Foundation Models shows up in your governance dashboard with token counts and IBM-published cost. 5-minute deploy.
--------------------------------------------------------------------------------

# IBM watsonx.ai

> 5-minute deploy · BYO Python code · runs anywhere (laptop, Code Engine, ECS, Lambda)

If your agent calls **watsonx Foundation Models** (Granite, Llama,
Mistral, granite-guardian, etc.) through the official
`ibm-watsonx-ai` Python SDK, Waxell auto-instruments every call —
no code changes, no wrappers. You get full spans, real token
counts, and cost sourced from IBM's published per-million-token
rates.

## What gets captured

Waxell patches the watsonx SDK in-place. After `waxell.init()`,
every call to these methods produces a child LLM span on the
parent run:

| watsonx SDK method | Captured as |
|---|---|
| `ModelInference.generate(...)` | LLM span with prompt, completion, token counts |
| `ModelInference.generate_text(...)` | LLM span (text-only return shape) |
| `ModelInference.chat(...)` | LLM span (chat-shaped messages) |
| `ModelInference.chat_stream(...)` | LLM span with streamed completion accumulated |
| `Embeddings.embed_documents(...)` | Embedding span with input count + vector dim |

Cost is computed from a built-in table covering Granite 4, Granite
3.x, Granite Guardian, and the common third-party models hosted on
watsonx — refreshed against IBM's published pricing.

## Prerequisites

1. An IBM Cloud account with watsonx.ai enabled.
2. A **project ID** and a **watsonx API key** with at least
   `Editor` access on the project.
3. Python 3.10+ (the `ibm-watsonx-ai` SDK supports 3.10–3.13).

## Install

```bash
pip install waxell-observe ibm-watsonx-ai
```

## Wire it up

Two lines at the top of your entrypoint — `waxell.init()` must

the methods at import time:

```python


waxell.init()  # must run before importing ibm_watsonx_ai

from ibm_watsonx_ai import Credentials
from ibm_watsonx_ai.foundation_models import ModelInference


@waxell.observe(agent_name="my-watsonx-agent")
def chat_turn(user_message: str) -> str:
    model = ModelInference(
        model_id="ibm/granite-3-8b-instruct",
        credentials=Credentials(
            api_key=os.environ["WATSONX_APIKEY"],
            url=os.environ.get("WATSONX_URL", "https://us-south.ml.cloud.ibm.com"),
        ),
        project_id=os.environ["WATSONX_PROJECT_ID"],
    )

    response = model.chat(messages=[
        {"role": "user", "content": user_message},
    ])
    return response["choices"][0]["message"]["content"]


if __name__ == "__main__":
    print(chat_turn("what is the deepest known cave?"))
```

## Environment variables

```bash
# Waxell


# IBM watsonx


```

## Run

```bash
python my_agent.py
```

Then check Waxell:

```bash
wax runs list --limit 5
```

You'll see your agent run with a `chat ibm/granite-3-8b-instruct`
child LLM span, real input/output token counts pulled from the
watsonx response, and cost computed from IBM's published rate.

## Deploy targets

Anywhere Python runs:

- **Locally** — `python my_agent.py`, no config beyond the env vars
  above.
- **IBM Cloud Code Engine** — push a container with the SDK + your
  agent; set the env vars in the Code Engine app config. NAT
  egress to `api.waxell.dev` is enabled by default.
- **AWS Lambda / ECS, GCP Cloud Run, Azure Container Apps** —
  identical pattern. watsonx-side is just an HTTPS API call from
  your host, so any runtime with outbound HTTPS works.

## Policy enforcement

The whole `@waxell.observe` + `@waxell.tool` + `ctx.check_policy(...)`
stack works the same as any other runtime. A policy denial raised
*before* `model.chat(...)` aborts the call so the watsonx tokens
are never charged — useful for PII redaction, prompt-injection
defense, and tenant cost caps on watsonx workloads.

## A note on watsonx Agent Lab

IBM **Agent Lab** is a no-code agent builder where agents are
configured in the IBM Cloud UI and execute on IBM's managed
runtime — you don't ship your own Python into it, so Waxell can't
install inside an Agent Lab agent the way it installs inside an
[AgentCore microVM](/runtimes/bedrock-agentcore).

For Agent Lab workloads, the two governed patterns are:

1. **BYO wrapper.** Build your agent loop in Python with
   `ibm-watsonx-ai`, wrap with `@waxell.observe`, and run it as
   shown above. Skip Agent Lab.
2. **Agent Lab + Waxell-instrumented client.** Use Agent Lab to
   author/version your prompts and tool catalog, then call the
   Agent Lab deployment endpoint from your Python entrypoint
   (also wrapped with `@waxell.observe`). You get Waxell spans on
   the *outer* call but not on the agent's internal steps.

Pattern 1 is what we recommend for any workload that needs
full-fidelity governance.

## Troubleshooting

**Spans don't appear in Waxell.** Check that `waxell.init()` is
imported *before* `from ibm_watsonx_ai import ...`. The
instrumentor patches `ModelInference` at import time; if watsonx
loads first, the patch never lands. The fix is one line reorder.

**Cost shows as `$0.00`.** A model id with no entry in Waxell's
cost table falls back to the cheapest current Granite rate (so
runs are never *over*-counted). If you're running a custom or
recently-launched model, [open an issue](https://github.com/waxell-ai/waxell-observe/issues)
with the model id and we'll add it to the table.

**Streaming spans look truncated.** `chat_stream` accumulates
deltas client-side. If your code consumes the generator with
`break` early, the span finalizes with whatever was received up to
that point — that's intentional, but it means partial completions
look short in the UI.


--------------------------------------------------------------------------------
# Azure AI Foundry (hosted agents)
URL: https://waxell.ai/docs/runtimes/azure-ai-foundry
Description: Deploy a Waxell-instrumented LangGraph agent to Azure AI Foundry as a hosted agent. The SDK ships inside the container and produces full client-side spans plus in-path policy enforcement — runs stream natively into Waxell with no App Insights pull. 5-minute deploy.
--------------------------------------------------------------------------------

# Azure AI Foundry hosted agents

> 5-minute deploy · BYO container · Foundry-managed VM-isolated sandbox

This is the **hosted-agent path** on Azure AI Foundry: you package
your agent (LangGraph here) as a container, register it as a
Foundry hosted agent, and Foundry runs it in a VM-isolated sandbox
behind the project's agent endpoint. Because the sandbox runs
*your* code, `waxell-observe` installs and runs inside the
container just like any other Python dependency — the Foundry
hosting boundary is transparent to the SDK.

## What you get

- Every LLM call your agent makes through the project's model
  deployment → captured as a child LLM span on the parent run with
  real model id, token counts, and cost.
- Every tool call in the agent loop → captured as tool spans, no
  manual per-tool wrapping (the LangGraph instrumentor handles it).
- Policies scoped to this agent — budget caps, tool blocks —
  **enforced inside the container** before the LLM/tool call fires.
- Runs stream **natively** into your Waxell dashboard — no App
  Insights pull, no telemetry export wiring.

The sandbox is just another Python host. Your agent code is
unchanged; Azure handles isolation, scaling, and lifecycle.

## Prerequisites

```bash
azd ext install azure.ai.agents   # provides the `azd ai agent` commands
azd auth login
```

You also need:

1. An Azure AI Foundry project with a model deployment (e.g.
   `gpt-5-mini`).
2. The **Foundry Project Manager** role on the project (required
   by `azd provision` / `azd deploy`).
3. Docker running locally — `azd` builds the agent image and
   pushes it to your project's Azure Container Registry.

## Project layout

Three files: `requirements.txt`, `Dockerfile`, `main.py`.

`requirements.txt`:

```text
waxell-observe
langchain-azure-ai[hosting]>=1.2.4
langgraph
langchain-core
langchain-openai
azure-identity
azure-ai-projects>=2.1.0
```

`Dockerfile` — the container serves the OpenAI Responses protocol
that Foundry hosted agents speak. The `langchain-azure-ai[hosting]`
extra requires pip 26 or newer, so upgrade pip before installing:

```dockerfile
FROM python:3.12-slim

WORKDIR /app

ENV PYTHONUNBUFFERED=1 \
    PORT=8088

# langchain-azure-ai's [hosting] extra requires pip >= 26
RUN pip install --no-cache-dir --upgrade "pip>=26"

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY main.py .

EXPOSE 8088

CMD ["python", "main.py"]
```

`main.py` — arm `waxell-observe` first, build a LangGraph graph,
and serve it with `ResponsesHostServer` (which also auto-exposes
the `/readiness` health endpoint Foundry probes — you don't
implement it):

```python


# Arm waxell-observe before anything else runs. init() reads
# WAXELL_API_KEY / WAXELL_API_URL from env and auto-instruments
# LangGraph / LangChain.
if os.environ.get("WAXELL_API_KEY"):
    try:
        import waxell_observe as waxell

        waxell.init()
    except Exception:  # never let observability break the runtime
        pass

from langchain_core.tools import tool


@tool
def lookup_account(account_id: str) -> dict:
    """Look up an account's tier and region."""
    return {"account_id": account_id, "tier": "enterprise", "region": "eastus"}


def _model():
    """Chat model bound to the project's model deployment via the
    agent's own Entra identity — no API keys in the container."""
    from azure.ai.projects import AIProjectClient
    from azure.identity import DefaultAzureCredential, get_bearer_token_provider
    from langchain_openai import ChatOpenAI

    deployment = os.environ.get("MODEL_DEPLOYMENT_NAME", "gpt-5-mini")
    cred = DefaultAzureCredential()
    client = AIProjectClient(
        endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"],  # injected by Foundry
        credential=cred,
    )
    oai = client.get_openai_client()
    token_provider = get_bearer_token_provider(
        cred, "https://ai.azure.com/.default"
    )
    return ChatOpenAI(
        base_url=str(oai.base_url),
        model=deployment,
        api_key="placeholder",  # replaced per-request by the bearer token
        default_headers={"Authorization": f"Bearer {token_provider()}"},
    )


def build_graph():
    from langgraph.prebuilt import create_react_agent

    graph = create_react_agent(
        _model(),
        tools=[lookup_account],
        prompt="You are a concise account assistant. One or two sentences.",
    )
    # LangGraph runs are attributed to graph.name — set it to the
    # agent name you want to see in Waxell.
    graph.name = "my-foundry-agent"
    return graph


def main():
    from langchain_azure_ai.agents.hosting import ResponsesHostServer

    server = ResponsesHostServer(build_graph())
    server.run(host="0.0.0.0", port=int(os.environ.get("PORT", "8088")))


if __name__ == "__main__":
    main()
```

Foundry injects `FOUNDRY_PROJECT_ENDPOINT` into the container and
the hosted agent authenticates to the model deployment with its
own dedicated Entra identity — no model API keys anywhere in your
image.

## Deploy

```bash
azd ai agent init      # scaffolds azure.yaml for the agent
azd ai agent run       # optional: build + run locally first
azd provision
azd deploy             # builds the image, pushes to ACR, registers an immutable agent version
```

Set two environment variables on the agent version (in
`azure.yaml`, or in the Foundry portal's agent-version settings):

- `WAXELL_API_URL` — `https://api.waxell.dev` (or your dedicated
  tenant URL).
- `WAXELL_API_KEY` — your Waxell API key. Reference it from a
  Foundry **CustomKeys connection** rather than pasting a
  plaintext value into the version config.

Each `azd deploy` creates an immutable agent version, so you can
canary a new build with a weighted traffic split before cutting
over.

## Invoke

Run the agent from the Foundry portal playground, or call its
agent endpoint via the Responses API. Then check Waxell:

```bash
wax runs list --limit 5
```

You'll see the run under `agent_name=my-foundry-agent` — parent
agent span, a `chat gpt-5-mini` LLM child span with real token
counts and cost, and a tool span for each `lookup_account` call.

## Network egress

Hosted-agent sandboxes egress over the public internet, so
`api.waxell.dev` (HTTPS) is reachable out of the box. If your
project restricts outbound traffic with a VNet, allow HTTPS egress
to your Waxell endpoint so runs keep streaming.

## Policy enforcement inside the runtime

The full Waxell policy stack runs unchanged inside the hosted-agent
container — there is no Foundry-specific shim or escape hatch.
Policies you scope to `my-foundry-agent` (budget caps, tool
blocks, kill switches) are evaluated at the gate before each LLM
and tool call, and a block verdict aborts the agent **before the
offending call lands** — the run surfaces as Blocked and the
violation is recorded in your Waxell governance dashboard.

## Connect your Foundry project to Waxell

Connecting the project itself (separate from the SDK inside your
container) makes your hosted agent appear in **Agent Discovery**
and brings the project's prompt-based agents under the same roof —
discovery, trace pull, and content-filter policy push.

1. Create an **app registration** (service principal) in Entra:
   note the tenant id and client id, and create a client secret.
2. Grant the service principal **three roles**:

   | Role | Scope | What it enables |
   |---|---|---|
   | **Cognitive Services User** | Foundry account | Agent read (Agent Discovery sync) |
   | **Cognitive Services Contributor** | Foundry account | Content-filter policy push + attach to model deployments |
   | **Log Analytics Reader** | Subscription (or the App Insights workspace) | Trace pull for prompt-based agents |

3. Enable tracing on the project: agent → **Traces** tab →
   *Connect* → create an Application Insights resource. This
   requires the `Microsoft.OperationalInsights` and
   `Microsoft.Insights` resource providers to be registered on
   the subscription.
4. In Waxell: **Agent Discovery → Azure AI Foundry** → enter the
   project endpoint, tenant/client ids, subscription, resource
   group, and account name, paste the client secret, and **Sync**.


--------------------------------------------------------------------------------
# Vertex AI Agent Engine
URL: https://waxell.ai/docs/runtimes/vertex-agent-engine
Description: Deploy a Waxell-instrumented LangGraph agent to Vertex AI Agent Engine. No Dockerfile — you hand Agent Engine a Python object, waxell-observe ships as a requirements entry, and every run's full span trace streams natively into your Waxell dashboard. 5-minute deploy.
--------------------------------------------------------------------------------

# Vertex AI Agent Engine

> 5-minute deploy · pure Python, no Dockerfile · Google-managed runtime

Vertex AI **Agent Engine** hosts your agent from a live Python
object: you hand `agent_engines.create()` an instance of your
agent class, the SDK serializes it, and the platform builds and
runs the container for you — installing whatever is in your
`requirements` list. `waxell-observe` ships as one of those
requirements and initializes inside the managed container, so the
Agent Engine boundary is transparent to the SDK.

## What you get

- Every LLM call your LangGraph agent makes → captured as a child
  LLM span on the parent run.
- Every tool call in the agent loop → captured as tool spans, no
  manual per-tool wrapping (the LangGraph instrumentor handles it).
- The full span trace of each run streams **natively** into your
  Waxell dashboard — no Cloud Trace pull, and your run history
  persists beyond Cloud Trace's 30-day retention window.
- Policies you author in Waxell and scope to the agent evaluate
  against its runs, with results in your governance dashboard.

Agent Engine's own OpenTelemetry export to Cloud Trace keeps
working alongside Waxell — the two are independent.

## Prerequisites

```bash
pip install "google-cloud-aiplatform[agent_engines,langchain,langgraph]"
gcloud auth application-default login
```

You also need:

1. A GCP project with `aiplatform.googleapis.com` enabled and a
   staging GCS bucket (Agent Engine stages the deploy archive
   there).
2. A **runtime service account** for the agent, with:
   `roles/aiplatform.user`, `roles/cloudtrace.agent`,
   `roles/logging.logWriter`, `roles/storage.objectViewer` (on the
   staging bucket), and `roles/secretmanager.secretAccessor` if
   you store the Waxell key in Secret Manager. Passing your own
   service account keeps the agent on a least-privilege identity
   instead of the platform default.
3. A **deploy principal** (the identity running `deploy.py`) with
   `aiplatform.reasoningEngines.create`, write access to the
   staging bucket, and `iam.serviceAccounts.actAs` on the runtime
   service account. Deploying executes code, so keep this
   principal separate from any read-only credentials.

## Project layout

Three files: `requirements.txt`, `agent.py`, `deploy.py`.

`requirements.txt` — this list is shipped to Agent Engine, which
pip-installs it into the managed container (there is no
Dockerfile):

```text
google-cloud-aiplatform[agent_engines,langchain,langgraph]>=1.128.0,<2.0.0
langchain-google-vertexai
langgraph>=0.2
langchain-core
cloudpickle
waxell-observe

# Agent Engine's managed runtime requires OpenTelemetry 1.43.x
opentelemetry-api==1.43.0
opentelemetry-sdk==1.43.0
```

`agent.py` — Agent Engine serializes your object on the deploy
machine and deserializes it in the managed container, so
`__init__` holds plain config only; all runtime initialization
happens in `set_up()`, which runs inside the container. That makes
`set_up()` the place to arm `waxell-observe`:

```python


AGENT_NAME = "my-vertex-agent"


def _build_tool():
    from langchain_core.tools import tool

    @tool
    def lookup_account(account_id: str) -> dict:
        """Look up an account's tier and region."""
        return {
            "account_id": account_id,
            "tier": "enterprise",
            "region": "us-central1",
        }

    return lookup_account


class MyVertexAgent:
    """A LangGraph react agent instrumented with waxell-observe."""

    def __init__(self, model_name: str | None = None) -> None:
        # Config only — this object is serialized at deploy time.
        self.model_name = model_name or os.environ.get(
            "MODEL_NAME", "gemini-2.5-flash"
        )
        self._graph = None

    def set_up(self) -> None:
        """Runs inside the Agent Engine container."""
        # Arm waxell-observe first. init() reads WAXELL_API_KEY /
        # WAXELL_API_URL from env and auto-instruments LangGraph.
        if os.environ.get("WAXELL_API_KEY"):
            try:
                import waxell_observe as waxell

                waxell.init()
            except Exception:  # never let observability break the runtime
                pass

        from langchain_google_vertexai import ChatVertexAI
        from langgraph.prebuilt import create_react_agent

        model = ChatVertexAI(model=self.model_name, temperature=0.2)
        graph = create_react_agent(
            model,
            tools=[_build_tool()],
            prompt="You are a concise account assistant. One or two sentences.",
        )
        # LangGraph runs are attributed to graph.name — keep it equal
        # to the deployed display_name.
        graph.name = AGENT_NAME
        self._graph = graph

    def _ensure(self):
        if self._graph is None:
            self.set_up()
        return self._graph

    def query(self, input: str) -> dict:
        graph = self._ensure()
        result = graph.invoke({"messages": [("user", input)]})
        msgs = result.get("messages", [])
        text = getattr(msgs[-1], "content", str(msgs[-1])) if msgs else ""
        return {"output": text}

    def stream_query(self, input: str):
        graph = self._ensure()
        for chunk in graph.stream(
            {"messages": [("user", input)]}, stream_mode="updates"
        ):
            yield chunk
```

`deploy.py` — two things to note: register your agent module for
pickling **by value** so its code is embedded in the deploy
archive, and pass `agent_engines.create()` **flat keyword
arguments** (not a `config=` dict):

```python


from vertexai import agent_engines


from agent import AGENT_NAME, MyVertexAgent

# Embed the agent module's code in the deploy archive so the
# managed container can load it.
cloudpickle.register_pickle_by_value(agent_module)

PROJECT = os.environ["PROJECT"]                       # your-project
REGION = os.environ.get("REGION", "us-central1")
STAGING = os.environ["STAGING"]                        # gs://your-staging-bucket
RUNTIME_SA = os.environ["RUNTIME_SA"]                  # agent-runtime@your-project.iam.gserviceaccount.com
WAXELL_API_KEY = os.environ["WAXELL_API_KEY"]
WAXELL_API_URL = os.environ.get("WAXELL_API_URL", "https://api.waxell.dev")

vertexai.init(project=PROJECT, location=REGION, staging_bucket=STAGING)

with open("requirements.txt") as fh:
    requirements = [
        line.strip()
        for line in fh
        if line.strip() and not line.startswith("#")
    ]

remote = agent_engines.create(
    MyVertexAgent(),
    display_name=AGENT_NAME,          # keep equal to graph.name
    requirements=requirements,
    env_vars={
        "WAXELL_API_URL": WAXELL_API_URL,
        "WAXELL_API_KEY": WAXELL_API_KEY,  # prefer a Secret Manager ref
        "GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY": "true",
    },
    service_account=RUNTIME_SA,
    min_instances=1,                  # keep warm; default 0 scales to zero
)
print("RESOURCE_NAME:", remote.resource_name)
```

## Deploy

```bash

PROJECT=your-project \
  REGION=us-central1 \
  STAGING=gs://your-staging-bucket \
  RUNTIME_SA=agent-runtime@your-project.iam.gserviceaccount.com \
  python deploy.py
```

Agent Engine stages the archive, builds the container, and
pip-installs your requirements — first deploys take a few minutes.

## Invoke

```python
remote.query(input="Look up account acct-001 and give its tier and region.")
```

Then check Waxell:

```bash
wax runs list --limit 5
```

You'll see the run under `agent_name=my-vertex-agent` — parent
agent span, an LLM child span for each Gemini call, and a tool
span for each `lookup_account` call.

## Network egress

Standard Agent Engine deployments have public egress, so
`api.waxell.dev` (HTTPS) is reachable out of the box. If your
project sits inside a VPC Service Controls perimeter or uses
Private Service Connect, add an egress rule allowing HTTPS to your
Waxell endpoint so runs keep streaming.

## Policies

Policies you author in Waxell and scope to `my-vertex-agent`
evaluate as the agent runs — author them once against the agent
name, and the same policy set follows the agent across every
environment it's deployed to, with results and audit records in
your Waxell governance dashboard.


================================================================================
SECTION: AGENT CONNECTIONS
Connect third-party and self-hosted agents to Waxell governance
================================================================================

--------------------------------------------------------------------------------
# For Coding Agents
URL: https://waxell.ai/docs/agents/overview
--------------------------------------------------------------------------------

# Waxell Developer MCP

A purpose-built [MCP](https://modelcontextprotocol.io) server that gives coding agents instant access to the Waxell platform — documentation, fleet monitoring, cost tracking, error analysis, and governance.

## What It Does

**15 Live Tools** — Query your Waxell instance in real-time. Monitor agent health, investigate errors, track LLM costs, review governance policies, and sign up for an account — all from your editor.

**8 Documentation Resources** — Agent-optimized guides served as MCP resources (`waxell://docs/*`). Covers the Observe SDK, CLI, integrations, controlplane UI, governance, and Cowork. Work without an API key.

**Built-in Signup** — Secure device authorization flow. Your agent calls `waxell_signup`, opens a browser link, and retrieves the API key automatically. No passwords in your terminal.

**Per-Client Authentication** — Each developer connects with their own API key via the `Authorization` header. Everyone sees their own data.

## Why

Coding agents like Claude Code, Cursor, and Windsurf can self-serve Waxell adoption:

1. **Connect** → point your MCP client at the hosted server, no install needed
2. **Read the docs** → understand how to instrument code in minutes
3. **Check the fleet** → monitor agent health without opening a browser
4. **Track costs** → keep tabs on LLM spending
5. **Review governance** → understand what policies are in place

The user barely needs to do anything — the agent discovers, learns, and instruments on its own.

## Quick Start

Add to your `.mcp.json` (Claude Code):

```json
{
  "mcpServers": {
    "waxell": {
      "type": "sse",
      "url": "https://mcp-gateway.waxell.dev/mcp",
      "headers": {
        "Authorization": "Bearer wax_sk_your_key_here"
      }
    }
  }
}
```

Don't have an API key yet? Connect without the `headers` field — the docs resources work without one. Then ask your agent to call `waxell_signup` to create an account.

See the [full setup guide](./setup) for Claude Desktop, Cursor, VS Code, and local installation options.


--------------------------------------------------------------------------------
# Setup
URL: https://waxell.ai/docs/agents/setup
--------------------------------------------------------------------------------

# MCP Setup

**Looking for OAuth?**
Claude Desktop and Claude Code support OAuth-based connections — no API key needed in your config file. See [Connect via OAuth](./mcp-connection).


## Hosted Server (Recommended)

Connect directly to the Waxell-hosted MCP server. No install required.

### Claude Code

Add to your project's `.mcp.json`:

```json
{
  "mcpServers": {
    "waxell": {
      "type": "sse",
      "url": "https://mcp-gateway.waxell.dev/mcp",
      "headers": {
        "Authorization": "Bearer wax_sk_your_key_here"
      }
    }
  }
}
```

### Claude Desktop

Add to `~/Library/Application Support/Claude/claude_desktop_config.json` (macOS) or `%APPDATA%\Claude\claude_desktop_config.json` (Windows):

```json
{
  "mcpServers": {
    "waxell": {
      "type": "sse",
      "url": "https://mcp-gateway.waxell.dev/mcp",
      "headers": {
        "Authorization": "Bearer wax_sk_your_key_here"
      }
    }
  }
}
```

### Cursor

Add to Cursor's MCP settings (Settings > MCP Servers):

```json
{
  "waxell": {
    "type": "sse",
    "url": "https://mcp-gateway.waxell.dev/mcp",
    "headers": {
      "Authorization": "Bearer wax_sk_your_key_here"
    }
  }
}
```

### VS Code

Add to your MCP settings:

```json
{
  "mcpServers": {
    "waxell": {
      "type": "sse",
      "url": "https://mcp-gateway.waxell.dev/mcp",
      "headers": {
        "Authorization": "Bearer wax_sk_your_key_here"
      }
    }
  }
}
```

## Run Locally (Alternative)

If you prefer to run the MCP server on your machine (for offline docs access or self-hosted setups):

```bash
# Using uvx (recommended — no install needed)
uvx waxell-developer-mcp

# Or install with pip
pip install waxell-developer-mcp
waxell-dev-mcp
```

Then configure your MCP client for stdio transport:

```json
{
  "mcpServers": {
    "waxell": {
      "type": "stdio",
      "command": "uvx",
      "args": ["waxell-developer-mcp"],
      "env": {
        "WAXELL_DEV_API_KEY": "wax_sk_your_key_here"
      }
    }
  }
}
```

## Getting an API Key

### Option 1: Sign up through your agent (recommended)

After connecting to the MCP server, ask your coding agent:

> "Sign me up for Waxell"

It calls `waxell_signup`, which opens a browser link. Complete the form in your browser — no password ever touches your editor. Then `waxell_signup_status` retrieves your API key and saves it to `~/.waxell/config`.

### Option 2: From the web dashboard

1. Go to [app.waxell.dev/settings/api-keys](https://app.waxell.dev/settings/api-keys)
2. Create a new API key
3. Copy the `wax_sk_...` key into your MCP config

### Option 3: Config file

The server reads `~/.waxell/config` as a fallback when no API key is provided via headers or environment variables:

```json
{
  "api_key": "wax_sk_your_key_here",
  "api_url": "https://app.waxell.dev"
}
```

## Configuration

### Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| `WAXELL_DEV_API_KEY` | *(none)* | API key for stdio mode. Falls back to `~/.waxell/config`. |
| `WAXELL_DEV_API_URL` | `https://app.waxell.dev` | Waxell API URL. Change for self-hosted deployments. |
| `WAXELL_DEV_TRANSPORT` | `stdio` | Transport mode: `stdio` (local) or `sse` (deployed). |
| `WAXELL_DEV_HOST` | `0.0.0.0` | Bind address for SSE mode. |
| `WAXELL_DEV_PORT` | `8004` | Listen port for SSE mode. |

In SSE mode (hosted), each client authenticates via the `Authorization: Bearer <key>` header instead of the server-side environment variable.

### Config Resolution Order

1. **HTTP headers** (SSE mode) — `Authorization: Bearer <key>` or `X-Wax-Key` header
2. **Environment variables** — `WAXELL_DEV_API_KEY`, `WAXELL_DEV_API_URL`
3. **Config file** — `~/.waxell/config` (JSON)
4. **Defaults**

## Verify

After setup, ask your coding agent:

> "Check my Waxell connection"

It calls `waxell_check_connection` and returns your tenant info, agent count, and guided next steps. If you see an error, check your API key.

## Docs-Only Mode

The documentation resources work without an API key. If you just want the agent to learn the SDK without querying live data, connect without the `headers` field or omit `WAXELL_DEV_API_KEY`. The docs resources will still be available, and tools will return a helpful setup message.


--------------------------------------------------------------------------------
# Connect via OAuth
URL: https://waxell.ai/docs/agents/mcp-connection
Description: Connect Claude Desktop or Claude Code to Waxell using OAuth — no API keys in config files.
--------------------------------------------------------------------------------

# Connect via OAuth

OAuth lets you connect Claude Desktop or Claude Code to Waxell by signing in through your browser — no API keys to copy into config files. Each connection automatically gets its own scoped API key, managed entirely behind the scenes. This is the recommended approach for Claude Desktop and Claude Code.

For API key authentication (Cursor, VS Code, or headless setups), see [Setup with API Keys](./setup).

## Prerequisites

- A Waxell account — [request access](https://waxell.ai/get-access) if you do not have one
- Claude Desktop or Claude Code installed

## Claude Desktop

### Step 1: Open the config file

Locate your Claude Desktop config file:

| OS | Config file path |
|---|---|
| macOS | `~/Library/Application Support/Claude/claude_desktop_config.json` |
| Windows | `%APPDATA%\Claude\claude_desktop_config.json` |
| Linux | `~/.config/Claude/claude_desktop_config.json` |

Create the file if it does not exist.

### Step 2: Add the Waxell MCP server

Add the following to your config file:

```json
{
  "mcpServers": {
    "waxell": {
      "url": "https://mcp-gateway.waxell.dev/mcp"
    }
  }
}
```

That is the entire config. No `headers`, no `type`, no API key — just the URL. OAuth handles authentication automatically.

**NOTE**
If you already have other MCP servers configured, add the `"waxell"` entry inside the existing `"mcpServers"` object.


### Step 3: Restart Claude Desktop

Close and reopen Claude Desktop to pick up the new config.

### Step 4: Sign in

On first connection, Claude Desktop opens your browser to the Waxell login page. Sign in with your email and password. If MFA is enabled on your account, complete the second factor.

Once you approve, the browser closes and Claude Desktop is connected.

**TIP**
You only need to sign in on first connect and again after 30 days of inactivity. Between sessions, Claude Desktop refreshes your tokens automatically.


## Claude Code

### Step 1: Create or open `.mcp.json`

In your project root (or home directory for global config), create `.mcp.json`:

```json
{
  "mcpServers": {
    "waxell": {
      "type": "url",
      "url": "https://mcp-gateway.waxell.dev/mcp"
    }
  }
}
```

The `"type": "url"` tells Claude Code to use the Streamable HTTP transport with OAuth.

### Step 2: Start Claude Code

Launch Claude Code in a terminal. It detects the MCP config and initiates the OAuth flow.

### Step 3: Sign in

Claude Code opens your browser to the Waxell login page. Sign in and complete MFA if enabled. After approval, the terminal session is connected.

**TIP**
Claude Code stores tokens locally. You will not need to sign in again until the refresh token expires (30 days of inactivity).


## Verify Your Connection

### Option 1: Ask your agent

After connecting, ask Claude:

> "Check my Waxell connection"

It calls `waxell_check_connection` and returns your tenant info, agent count, and guided next steps.

### Option 2: Check the Waxell dashboard

Go to [app.waxell.dev/settings/connections](https://app.waxell.dev/settings/connections) and look for a new MCP connection entry with a recent "last used" timestamp.

For more details on viewing and revoking connections, see [Managing Connections](./managing-connections).

## What Happens Behind the Scenes

When you connect, the MCP client and Waxell exchange a series of messages to establish a secure session:

1. **Connect** — The client sends an initial request, receives `401 Unauthorized`
2. **Discovery** — The client fetches OAuth metadata from `/.well-known/oauth-authorization-server`
3. **Client registration** — The client registers itself and receives a `client_id`
4. **Browser login** — You sign in; the server issues an authorization code
5. **Token exchange** — The client exchanges the code (with PKCE proof) for access and refresh tokens
6. **Authenticated access** — Every subsequent MCP request includes the access token

| Token | Lifetime | What it does |
|-------|----------|--------------|
| Access token | 1 hour | Authenticates each MCP request |
| Refresh token | 30 days | Gets new access tokens without re-login |
| Client registration | 30 days | Cached client credentials |

The access token embeds a scoped API key (`wax_sk_...`) so you never need to manage keys manually.

For the full protocol specification — PKCE details, request/response formats, error codes — see [OAuth Flow](./oauth-flow).

## Local Development

If you are running the MCP server locally (e.g., during development at `http://localhost:8005`), OAuth will not work directly because MCP clients expect HTTPS for OAuth discovery.

Use the [`mcp-remote`](https://www.npmjs.com/package/mcp-remote) bridge to proxy a local HTTP server through an OAuth-capable wrapper.

### Claude Desktop with mcp-remote

```json
{
  "mcpServers": {
    "waxell-local": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-remote",
        "http://localhost:8005/mcp",
        "--allow-http"
      ]
    }
  }
}
```

### Claude Code with mcp-remote

```json
{
  "mcpServers": {
    "waxell-local": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-remote",
        "http://localhost:8005/mcp",
        "--allow-http"
      ]
    }
  }
}
```

**NOTE**
`mcp-remote` bridges stdio to HTTP so the MCP client communicates with your local server without requiring HTTPS or OAuth. For local development with API key auth (no bridge needed), see [Setup with API Keys](./setup#run-locally-alternative).


## Troubleshooting

### Browser does not open

- **Claude Desktop:** Ensure you are running the latest version. Older versions may not support OAuth-based MCP connections. Check that your config file has only `"url"` (no `"type": "sse"` or `"headers"`).
- **Claude Code:** Check terminal output for error messages. Ensure `.mcp.json` has `"type": "url"`.

### "No tenant found" error after signing in

Your account is not associated with a workspace. Ask your team admin to invite you, or [request access](https://waxell.ai/get-access) for a new workspace.

### Connection drops after an hour

This is normal. The access token expires hourly and the MCP client refreshes it automatically using the refresh token. If the refresh fails, you will be prompted to sign in again — this usually means the refresh token has also expired (30 days of inactivity).

### "Invalid client" error

The client registration has expired (30-day TTL). Remove the cached client credentials and reconnect:

- **Claude Desktop:** Restart the app. It re-registers automatically.
- **Claude Code:** Delete the cached OAuth state and restart. Claude Code stores OAuth state in its local data directory.


--------------------------------------------------------------------------------
# Capabilities
URL: https://waxell.ai/docs/agents/capabilities
--------------------------------------------------------------------------------

# MCP Capabilities Reference

## Documentation Resources

Browse these via MCP resource URIs. Each is a self-contained, agent-optimized guide. All resources work without an API key.

| Resource URI | Description |
|---|---|
| `waxell://docs/quickstart` | 5-minute setup: install, configure, instrument, verify |
| `waxell://docs/sdk-reference` | Full Python SDK API — every function with signature and example |
| `waxell://docs/cli-reference` | All `wax` CLI commands with flags and examples |
| `waxell://docs/integrations` | Per-provider guides: OpenAI, Anthropic, LangChain, LiteLLM, Groq |
| `waxell://docs/controlplane` | Controlplane UI navigation — every section and URL |
| `waxell://docs/governance` | Policy categories, enforcement, scoping, configuration |
| `waxell://docs/cowork` | Claude Cowork integration: hooks, guard, monitoring |
| `waxell://docs/troubleshooting` | Common issues and solutions |

## Live Platform Tools

All tools require a valid API key. Returns JSON data from your Waxell instance.

### Setup & Account

**`waxell_check_connection()`**
Verify your API key, check fleet status, and get guided next steps based on your current state. Always call this first.

**`waxell_get_config()`**
Show current configuration (API key masked, URL, source).

**`waxell_setup_guide()`**
Return the quickstart guide inline for agents that don't browse resources.

**`waxell_signup()`**
Start the browser-based signup flow using device authorization. Returns a URL for the user to open. No password touches the conversation.

**`waxell_signup_status(session_id)`**
Complete the signup flow and retrieve the API key. Call after the user finishes the browser form.

### Fleet Monitoring

**`waxell_agent_fleet(hours=24)`**
List all agents with health metrics, execution counts, and success rates.

**`waxell_agent_detail(agent_name, hours=24)`**
Deep-dive on a specific agent: recent runs, error rate, model usage.

### Runs & Executions

**`waxell_recent_runs(hours=24, agent_name="", status="")`**
Summary of recent runs: total, by status, by agent, trend. Filter by agent or status (`success`, `error`, `running`).

**`waxell_run_detail(agent_name, status="error", hours=24)`**
Investigate runs for a specific agent, typically errors.

### Costs

**`waxell_cost_summary(hours=24, model="")`**
Total cost, tokens, by-model breakdown, and trend.

**`waxell_model_usage(hours=24)`**
Which models are being used, call counts, costs per model.

### Errors

**`waxell_error_summary(hours=24, agent_name="")`**
Error rate, errors by agent, and recent error details.

### Governance

**`waxell_policy_summary(hours=24)`**
Policy pass rate, violations by policy, recent violations.

**`waxell_policy_list()`**
All active policies with configuration.

**`waxell_recommendations()`**
AI-generated policy recommendations based on observed agent behavior.


--------------------------------------------------------------------------------
# Self-Hosting
URL: https://waxell.ai/docs/agents/self-hosting
--------------------------------------------------------------------------------

# Self-Hosting the Developer MCP

For teams that want to run their own MCP server instance instead of using the hosted endpoint at `dev-mcp.waxell.dev`.

## Docker

Build and run the MCP server as a standalone container:

```bash
# From the agentforge repo root
docker build -f docker/developer-mcp/Dockerfile -t waxell-dev-mcp .

docker run -p 8004:8004 \
  -e WAXELL_DEV_TRANSPORT=sse \
  -e WAXELL_DEV_API_URL=https://app.your-domain.com \
  waxell-dev-mcp
```

The server starts in SSE mode on port 8004 with a health check at `/health`.

## Docker Compose

```yaml
services:
  developer-mcp:
    build:
      context: .
      dockerfile: docker/developer-mcp/Dockerfile
    ports:
      - "8004:8004"
    environment:
      WAXELL_DEV_TRANSPORT: "sse"
      WAXELL_DEV_API_URL: "https://app.your-domain.com"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8004/health"]
      interval: 10s
      timeout: 5s
      retries: 3
```

## Terraform (AWS Fargate)

The agentforge repo includes a Terraform module that deploys the MCP server to ECS Fargate with ALB routing.

### What it provisions

- **ECS Fargate task** — ARM64 architecture, configurable CPU/memory
- **ECS service** — with load balancer integration and health check grace period
- **ALB target group** — routes `dev-mcp.{domain}` to port 8004
- **ALB listener rule** — HTTPS termination at the ALB, HTTP to the container
- **ECR repository** — for storing Docker images with scan-on-push
- **CloudWatch Logs** — log group with `developer-mcp` stream prefix

### Key files

| File | Purpose |
|------|---------|
| `terraform/modules/ecs/developer-mcp-task.tf` | ECS task definition and service |
| `terraform/modules/alb/main.tf` | ALB target group and listener rule |
| `terraform/main.tf` | ECR repository and module wiring |
| `terraform/modules/ecs/variables.tf` | CPU, memory, image, and target group variables |

### Configuration

The Terraform module accepts these variables:

| Variable | Description |
|----------|-------------|
| `developer_mcp_image` | ECR image URI (e.g., `123456789.dkr.ecr.us-west-2.amazonaws.com/waxell-dev-developer-mcp:latest`) |
| `developer_mcp_cpu` | CPU units for the Fargate task |
| `developer_mcp_memory` | Memory in MB for the Fargate task |
| `developer_mcp_target_group_arn` | ARN of the ALB target group |
| `domain` | Your domain (creates `dev-mcp.{domain}` ALB rule) |

### Environment variables set on the task

```hcl
environment = [
  { name = "WAXELL_DEV_API_URL",   value = "https://app.${var.domain}" },
  { name = "WAXELL_DEV_TRANSPORT", value = "sse" },
  { name = "WAXELL_DEV_HOST",      value = "0.0.0.0" },
  { name = "WAXELL_DEV_PORT",      value = "8004" },
]
```

Note: `WAXELL_DEV_API_KEY` is intentionally not set on the task. In SSE mode, each connected client sends their own API key via the `Authorization` header.

## Connecting Clients to Your Instance

Point your MCP client at your self-hosted SSE endpoint:

```json
{
  "mcpServers": {
    "waxell": {
      "type": "sse",
      "url": "https://dev-mcp.your-domain.com/sse",
      "headers": {
        "Authorization": "Bearer wax_sk_your_key_here"
      }
    }
  }
}
```

## Health Check

The server exposes a `GET /health` endpoint that returns:

```json
{"status": "ok", "service": "waxell-dev-mcp"}
```

This is used by Docker health checks, ALB target group health checks, and ECS service stability monitoring.


--------------------------------------------------------------------------------
# Managing Connections
URL: https://waxell.ai/docs/agents/managing-connections
Description: View, revoke, and regenerate MCP OAuth connections in the Waxell dashboard.
--------------------------------------------------------------------------------

# Managing Connections

The MCP Connections page in the Waxell dashboard shows every OAuth connection from Claude Desktop, Claude Code, and other MCP clients. From this page you can see which clients are connected, revoke access, and regenerate API keys without asking users to re-authenticate.

Connections are created automatically when an MCP client completes the [OAuth flow](./oauth-flow). You do not create connections manually — see [Connect via OAuth](./mcp-connection) for setup instructions.

## Accessing the Page

Navigate to **Settings > MCP** in the Waxell dashboard. The page is available at `/settings/mcp-connections`.

**NOTE**
You need the `MCP_CONNECTIONS_READ` permission to view this page. Without it, you will see a "Permission Denied" message. See [Permissions](#permissions) below.


## Viewing Connections

Each connection appears as a row in the connections list. The header shows the total number of connections.

| Field | Description |
|-------|-------------|
| **Client name** | The name provided by the MCP client during OAuth registration (e.g., "Claude Desktop"). If the client did not provide a name, this falls back to "MCP Client". |
| **Status** | `active` (green badge) means the connection is live and the client can make requests. `revoked` (red badge) means the connection has been terminated. |
| **User email** | The Waxell account that authorized this connection. This is the user who signed in during the OAuth flow. |
| **API key prefix** | The first characters of the auto-provisioned API key for this connection (e.g., `wax_sk_...`). Each connection gets its own dedicated key. |
| **Connected date** | When the OAuth flow was first completed and the connection was established. Displayed as a formatted date (e.g., "Jan 15, 2026"). |
| **Last token** | The last time an access token was refreshed for this connection. Shows relative time (e.g., "3h ago") or "Never" if no token refresh has occurred since the initial authorization. |

Revoked connections remain visible in the list with a red status badge and a "Revoked" date, but they no longer have action buttons.

## Revoking a Connection

Revoking a connection permanently disconnects an MCP client and deactivates its API key.

**Steps:**

1. Find the connection you want to revoke in the connections list
2. Click the **trash icon** on the right side of the connection row
3. A confirmation dialog appears: *"Revoke this MCP connection? The associated API key will be deactivated and the client will need to re-authorize."*
4. Click **OK** to confirm, or **Cancel** to abort

**CAUTION**
Revoking is immediate and cannot be undone. After you revoke a connection:

- The connection's API key is **deactivated instantly**
- The MCP client receives a `401 Unauthorized` response on its next request
- The client must complete the full [OAuth flow](./oauth-flow) again to reconnect (browser login, new authorization code, new tokens)
- A new connection and new API key will be created when the client re-authorizes

Use revoke when a device is lost, a team member leaves, or you need to cut off a specific client immediately.


## Regenerating an API Key

Regenerating rotates the API key for a connection without disconnecting the client.

**Steps:**

1. Find the active connection whose key you want to rotate
2. Click the **key icon** on the right side of the connection row
3. A confirmation dialog appears: *"Regenerate the API key for this connection? The old key will be deactivated immediately."*
4. Click **OK** to confirm, or **Cancel** to abort

**TIP**
After regeneration:

- The old API key is **deactivated immediately**
- The connection itself remains active
- The MCP client automatically receives a new access token on its next refresh cycle (within 1 hour at most)
- No user action is required -- the client handles the transition transparently

Use regenerate for routine key rotation or if you suspect a key may have been exposed but still trust the device.


## Permissions

Access to the MCP Connections page is controlled by two permissions:

| Permission | Maps to | What it allows |
|------------|---------|----------------|
| `MCP_CONNECTIONS_READ` | `api-keys:read` | View the connections list and all connection details |
| `MCP_CONNECTIONS_REVOKE` | `api-keys:revoke` | Revoke connections and regenerate API keys |

- Users with `MCP_CONNECTIONS_READ` but without `MCP_CONNECTIONS_REVOKE` can view all connections but will not see the trash or key action buttons.
- Users without `MCP_CONNECTIONS_READ` are shown a "Permission Denied" screen when navigating to the page.

## Built-in Help

The **How to Connect** button in the page header opens a help dialog that walks through connecting an MCP client in three steps:

1. Open Claude Desktop settings and navigate to Developer > Edit Config
2. Add the MCP server URL as a remote MCP server entry
3. Authorize in the browser when prompted

The dialog also shows an example `claude_desktop_config.json` snippet with the correct URL pre-filled.

For the step-by-step connection guide, see [Connect via OAuth](./mcp-connection). For details on the OAuth authorization protocol, see [OAuth Flow](./oauth-flow).


--------------------------------------------------------------------------------
# OAuth Flow
URL: https://waxell.ai/docs/agents/oauth-flow
Description: How the MCP OAuth authorization flow works — discovery, client registration, PKCE authorization, and token exchange.
--------------------------------------------------------------------------------

# MCP OAuth Flow

Waxell's MCP server uses OAuth 2.0 with PKCE so users authenticate in the browser instead of copying API keys into config files. When you connect from Claude Desktop, Claude Code, or another MCP client, a browser window opens, you sign in, and the client receives tokens automatically.

**Just want to connect?**
If you want to connect Claude Desktop or Claude Code to Waxell, see [Connect via OAuth](./mcp-connection) for a step-by-step guide. This page covers the protocol internals.


The implementation follows four RFCs: [RFC 6749](https://datatracker.ietf.org/doc/html/rfc6749) (OAuth 2.0), [RFC 7636](https://datatracker.ietf.org/doc/html/rfc7636) (PKCE), [RFC 7591](https://datatracker.ietf.org/doc/html/rfc7591) (Dynamic Client Registration), and [RFC 8414](https://datatracker.ietf.org/doc/html/rfc8414) (Authorization Server Metadata).

## How It Works

The complete flow has six steps:

1. **Connect** — MCP client sends an initial request to the MCP server, receives `401 Unauthorized` with a pointer to the authorization server
2. **Discovery** — Client fetches `/.well-known/oauth-authorization-server` to learn the endpoint URLs
3. **Client Registration** — Client registers itself via Dynamic Client Registration, receives a `client_id` and `client_secret`
4. **Authorization** — Client opens the browser; user signs in (and completes MFA if enabled); server issues an authorization code
5. **Token Exchange** — Client exchanges the authorization code (with PKCE verifier) for access and refresh tokens
6. **Authenticated Access** — Client sends the access token as a Bearer header with every MCP request

| Step | Who does it | User-visible? |
|------|-------------|---------------|
| Connect | MCP client | No |
| Discovery | MCP client | No |
| Client Registration | MCP client | No |
| Authorization | User in browser | **Yes** — you sign in here |
| Token Exchange | MCP client | No |
| Authenticated Access | MCP client | No |

**TIP**
From the user's perspective, the only step is signing in. Everything else is handled automatically by the MCP client.


## Step 1: Discovery

The client fetches the authorization server metadata to learn where to send registration, authorization, and token requests.

**Request:**

```http
GET /.well-known/oauth-authorization-server HTTP/1.1
Host: app.waxell.dev
```

**Response:**

```json
{
  "issuer": "https://app.waxell.dev",
  "authorization_endpoint": "https://app.waxell.dev/api/oauth/mcp/authorize/",
  "token_endpoint": "https://app.waxell.dev/api/oauth/mcp/token/",
  "registration_endpoint": "https://app.waxell.dev/api/oauth/mcp/register/",
  "response_types_supported": ["code"],
  "grant_types_supported": ["authorization_code", "refresh_token"],
  "token_endpoint_auth_methods_supported": [
    "client_secret_post",
    "client_secret_basic"
  ],
  "code_challenge_methods_supported": ["S256"]
}
```

| Field | Meaning |
|-------|---------|
| `issuer` | Base URL of the authorization server |
| `authorization_endpoint` | Where to send the user for login |
| `token_endpoint` | Where to exchange codes for tokens |
| `registration_endpoint` | Where to register new clients (DCR) |
| `response_types_supported` | Only `code` (authorization code grant) |
| `grant_types_supported` | Authorization code and refresh token |
| `token_endpoint_auth_methods_supported` | Client authenticates via POST body or Basic auth |
| `code_challenge_methods_supported` | Only `S256` (SHA-256 PKCE) |

## Step 2: Client Registration

The client registers itself with the authorization server. This creates a unique `client_id` and `client_secret` for this client instance.

**Request:**

```http
POST /api/oauth/mcp/register/ HTTP/1.1
Host: app.waxell.dev
Content-Type: application/json

{
  "client_name": "Claude Desktop",
  "redirect_uris": ["http://127.0.0.1:49152/oauth/callback"],
  "grant_types": ["authorization_code"]
}
```

**Response:**

```json
{
  "client_id": "mcp-aBcDeFgHiJkLmNoPqRsTuV",
  "client_secret": "xYzAbCdEfGhIjKlMnOpQrStUvWxYz012345678901",
  "client_name": "Claude Desktop",
  "redirect_uris": ["http://127.0.0.1:49152/oauth/callback"],
  "grant_types": ["authorization_code"],
  "token_endpoint_auth_method": "client_secret_post"
}
```

**NOTE**
Client registrations are cached for 30 days. The client does not re-register every time it connects — it reuses its existing `client_id` and `client_secret` until they expire.


## Step 3: Authorization

The client generates a PKCE code verifier and challenge, then opens the user's browser to the authorization endpoint.

**Authorization URL:**

```
https://app.waxell.dev/api/oauth/mcp/authorize/
  ?client_id=mcp-aBcDeFgHiJkLmNoPqRsTuV
  &response_type=code
  &redirect_uri=http://127.0.0.1:49152/oauth/callback
  &code_challenge=E9Melhoa2OwvFrEMTJguCHaoeK1t8URWbuGJSstw-cM
  &code_challenge_method=S256
  &state=abc123xyz
  &resource=https://mcp-gateway.waxell.dev/mcp
```

**PKCE explained:** The client generates a random `code_verifier` (a high-entropy string). It computes `code_challenge = base64url(sha256(code_verifier))` and sends the challenge in the authorization request. Later, during token exchange, the client sends the original `code_verifier` so the server can verify they match. This prevents authorization code interception — even if an attacker captures the code, they cannot exchange it without the verifier.

**Login form:** The authorization endpoint renders a branded login form. The user enters their email and password directly on this page.

### Multi-Factor Authentication

When MFA is enabled on the user's account, a second verification step appears after password authentication.

**Supported methods:**

- **TOTP** — 6-digit time-based codes from an authenticator app (Google Authenticator, Authy, 1Password, etc.). Codes are accepted within a +/- 30 second window (`valid_window=1`).
- **Backup codes** — Single-use 8-character recovery codes. Each code is consumed (permanently removed) when used.

**MFA session behavior:**

- After entering a correct password, the server creates a temporary MFA session with a **5-minute timeout**
- The user has a maximum of **5 attempts** to enter a valid code
- After 5 failed attempts, the MFA session is destroyed and the user must restart the login from email/password
- If the 5-minute timeout expires before a valid code is entered, the session is destroyed

**CAUTION**
If your authenticator app's clock is out of sync, TOTP codes may be rejected. Ensure your device's time is set automatically. If TOTP is not working, use a backup code instead.


**Auth code redirect:** On successful authentication (and MFA if enabled), the browser redirects to the client's callback URL with an authorization code:

```
http://127.0.0.1:49152/oauth/callback?code=RaNdOmSeCuReCoDeHeRe&state=abc123xyz
```

The authorization code expires in **5 minutes** and is **single-use** — it is consumed on the first exchange attempt.

## Step 4: Token Exchange

The client exchanges the authorization code for access and refresh tokens, including the PKCE `code_verifier` for proof.

**Request:**

```http
POST /api/oauth/mcp/token/ HTTP/1.1
Host: app.waxell.dev
Content-Type: application/x-www-form-urlencoded

grant_type=authorization_code
&code=RaNdOmSeCuReCoDeHeRe
&redirect_uri=http://127.0.0.1:49152/oauth/callback
&client_id=mcp-aBcDeFgHiJkLmNoPqRsTuV
&client_secret=xYzAbCdEfGhIjKlMnOpQrStUvWxYz012345678901
&code_verifier=dBjftJeZ4CVP-mB92K27uhbUJU1p1r_wW1gFWFOEjXk
```

**Response:**

```json
{
  "access_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ1c2VyLXV1aWQiLCJ0ZW5hbnRfaWQiOiJ0ZW5hbnQtdXVpZCIsImFwaV9rZXkiOiJ3YXhfc2tfLi4uIiwiYXVkIjoiaHR0cHM6Ly9kZXYtbWNwLndheGVsbC5kZXYvbWNwIiwiaXNzIjoiaHR0cHM6Ly9hcHAud2F4ZWxsLmRldiIsImlhdCI6MTcwODAwMDAwMCwiZXhwIjoxNzA4MDAzNjAwLCJ0eXBlIjoiYWNjZXNzIn0.signature",
  "token_type": "Bearer",
  "expires_in": 3600,
  "refresh_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ1c2VyLXV1aWQiLCJ0ZW5hbnRfaWQiOiJ0ZW5hbnQtdXVpZCIsImFwaV9rZXkiOiJ3YXhfc2tfLi4uIiwiYXVkIjoiaHR0cHM6Ly9kZXYtbWNwLndheGVsbC5kZXYvbWNwIiwiaXNzIjoiaHR0cHM6Ly9hcHAud2F4ZWxsLmRldiIsImlhdCI6MTcwODAwMDAwMCwiZXhwIjoxNzEwNTkyMDAwLCJ0eXBlIjoicmVmcmVzaCJ9.signature"
}
```

After this step, the client includes the access token with every MCP request:

```http
Authorization: Bearer eyJhbGciOiJIUzI1NiIs...
```

## Token Lifecycle

| Token | Lifetime | Purpose |
|-------|----------|---------|
| Access token | 1 hour | Authenticates MCP requests |
| Refresh token | 30 days | Obtains new access tokens without re-login |
| Authorization code | 5 minutes | One-time code exchanged for tokens |
| Client registration | 30 days | Cached client credentials from DCR |

### Refresh Flow

When the access token expires, the MCP client automatically refreshes it using the refresh token. No user interaction is needed.

**Request:**

```http
POST /api/oauth/mcp/token/ HTTP/1.1
Host: app.waxell.dev
Content-Type: application/x-www-form-urlencoded

grant_type=refresh_token
&refresh_token=eyJhbGciOiJIUzI1NiIs...
&client_id=mcp-aBcDeFgHiJkLmNoPqRsTuV
```

**Response:**

```json
{
  "access_token": "eyJhbGciOiJIUzI1NiIs...(new token)",
  "token_type": "Bearer",
  "expires_in": 3600,
  "refresh_token": "eyJhbGciOiJIUzI1NiIs...(new refresh token)"
}
```

Both a new access token and a new refresh token are issued. The new refresh token has a fresh 30-day expiry.

**NOTE**
When the refresh token expires (after 30 days of no use), the user must re-authenticate in the browser. The MCP client will open the browser again automatically.


## Security Model

- **PKCE prevents code interception** — The authorization code is useless without the `code_verifier`, which never leaves the client. Even if an attacker intercepts the redirect, they cannot exchange the code for tokens.
- **Passwords never touch the MCP client** — Users enter credentials only in the browser login form. The MCP client never sees or stores the password.
- **Scoped API keys** — Each MCP connection gets its own API key embedded in the JWT `api_key` claim. The MCP server extracts it to make authenticated API calls on behalf of the user.
- **Tenant isolation** — JWTs contain a `tenant_id` claim. The MCP server can only access data belonging to that tenant.
- **Short-lived authorization codes** — Auth codes are single-use and expire in 5 minutes, minimizing the window for interception.

## Error Reference

| Error | HTTP Status | Meaning |
|-------|-------------|---------|
| `{"error": "invalid_request", "error_description": "PKCE S256 required"}` | 400 | The authorization request is missing `code_challenge` or `code_challenge_method` is not `S256`. All requests must use PKCE with SHA-256. |
| `{"error": "invalid_client"}` | 400 / 401 | The `client_id` is not recognized. The client registration may have expired (30-day TTL) or the ID is malformed. Re-register the client. |
| `{"error": "invalid_grant", "error_description": "Invalid or expired code"}` | 400 | The authorization code has already been used (single-use) or has expired (5-minute TTL). The user needs to re-authenticate. |
| `{"error": "invalid_grant", "error_description": "PKCE verification failed"}` | 400 | The `code_verifier` sent during token exchange does not match the `code_challenge` from the authorization request. This usually means the client sent a different verifier than the one used to generate the challenge. |
| `{"error": "access_denied", "error_description": "No tenant found"}` | 403 | The authenticated user has no active workspace/tenant. The user needs to be invited to a team or complete their account setup. |
| `{"error": "unsupported_response_type"}` | 400 | The `response_type` parameter is not `code`. Only the authorization code grant is supported. |
| `{"error": "invalid_client_metadata", "error_description": "redirect_uris required"}` | 400 | The client registration request is missing `redirect_uris`. At least one redirect URI must be provided. |


================================================================================
SECTION: WAXELL CONNECT
Workspaces, governance, and Slack for human-agent collaboration
================================================================================

--------------------------------------------------------------------------------
# Connect — Third-Party Agent Visibility
URL: https://waxell.ai/docs/connect/overview
--------------------------------------------------------------------------------

# Connect

Connect gives your team visibility into **every AI agent running in your organization** — Claude Code, Cursor, Windsurf, ChatGPT, and any tool that speaks MCP or makes LLM API calls.

## What Connect does

- **Automatic discovery** — agents appear as they're used, no manual registration
- **Activity feed** — real-time view of what every agent is doing across your org
- **Workspace isolation** — group agents by team, project, or purpose
- **Channel routing** — route agent output to Slack, email, or webhooks
- **Governance overlay** — apply policies to third-party agents without modifying them

## Getting started

1. Install the [Waxell CLI](/docs/installation) or [Mac app](https://install.waxell.dev)
2. Run `wax connect` to link your account
3. Start using any AI agent — it appears automatically in your Connect dashboard

## Architecture

Connect works by observing network traffic to known AI service endpoints (OpenAI, Anthropic, etc.) and correlating it with the user's identity. No agent modification required.

```
User's machine         Waxell Cloud         Dashboard
┌──────────┐          ┌──────────┐         ┌──────────┐
│ Claude   │ ───LLM──>│ Observe  │────────>│ Activity │
│ Cursor   │  traffic │ Pipeline │         │ Feed     │
│ ChatGPT  │          └──────────┘         └──────────┘
└──────────┘
```


--------------------------------------------------------------------------------
# Connect Workspaces
URL: https://waxell.ai/docs/connect/workspaces
--------------------------------------------------------------------------------

# Workspaces

Workspaces let you organize your team's agent activity by project, team, or purpose.

## Creating a workspace

Navigate to **Connect > Workspaces** and click **New Workspace**. Give it a name and optionally assign team members.

## Workspace isolation

Each workspace has its own:
- Activity feed showing only agents in that workspace
- Policy set (governance rules scoped to this workspace)
- Channel subscriptions (where notifications go)
- File attachments and shared context

## Default workspace

Every tenant has a **Default** workspace. Agents that aren't explicitly assigned to a workspace appear here.


--------------------------------------------------------------------------------
# Connect Agent Governance
URL: https://waxell.ai/docs/connect/governance
Description: How Waxell governs external AI agents (Claude Code, Claude Cowork, Codex, MCP) connected via the Connect platform — enforcement models, scope dimensions, lifecycle, and policy categories.
--------------------------------------------------------------------------------

# Connect Agent Governance

This page is the operator's overview of how Waxell governs **external agents** — agents you didn't author with the Waxell SDK but that you want to bring under policy control. Today that means:

- **Claude Code** (Anthropic's CLI coding agent)
- **Claude Cowork** (Anthropic's hosted multi-agent product)
- **Codex** (OpenAI's CLI coding agent)
- **MCP agents** (any third-party agent that speaks Model Context Protocol)
- **Custom agents** (your own scripts registered against a Connect API key)

If you're building an agent *with* the Waxell SDK, see the [Observe governance docs](../observe/features/governance.md) — the same policy categories apply, but the enforcement story is simpler because the runtime is yours.

---

## Why Connect governance

A typical mid-size company in 2026 has Claude Code installed on every developer laptop, Cowork running unattended overnight on a shared dev box, Codex in CI, and a half-dozen MCP-based agents glued together by ops. None of these run inside your code. None of them go through a single chokepoint you control. And every one of them spends money, touches files, and calls third-party APIs on your behalf.

The problems:

- **Spend opacity.** Token costs land in three different vendor invoices.
- **Shadow tool use.** Engineers add MCP servers (GitHub, Notion, Postgres, anything) to their local agent without anyone reviewing them.
- **No audit consistency.** Each agent has its own log format. Compliance can't answer "show me every code-execution event for engineer X last week."
- **No kill switch.** When an agent loops or burns through a budget, you find out from the credit-card alert.

Connect governance is the lever-set for these problems: **one policy model, one provenance trail, one place to configure it,** applied consistently across every external agent you have running.

---

## How it works (architecture in 30 seconds)

```
┌──────────────────────┐        ┌──────────────────┐
│  External agent on   │        │  Cloud chokepoint│
│  user's machine      │ ─────► │  (LLM proxy +    │ ── policy eval ──► allow / block / warn
│  (CC / Codex / MCP)  │        │  Connect domains)│
└──────────┬───────────┘        └──────────────────┘
           │
           ├── Local guard / hooks / network filter ── policy eval (preventive)
           │
           └── Audit stream (Cowork only) ─── policy eval (detective, ~5s lag)
```

Three enforcement seams:

1. **Cloud chokepoint** — every Connect-routed call (`POST /domains/{entity}/{action}` or LLM proxy request) gets evaluated server-side before the action runs. Preventive.
2. **On-machine** — Claude Code hooks, Codex wrappers, the network filter, and the cowork MCP shim catch actions *before* they happen on the laptop. Preventive when the lever is on this seam.
3. **Audit stream** — for Cowork (and other agents that we can only watch, not intercept) we stream `audit.jsonl` events to the cloud and evaluate them post-hoc. Detective.

Every block/warn/alert carries a **decision provenance** record (`metadata.provenance` on the result) with `policy_id`, `agent_id`, `agent_type`, `enforcement_model`, `phase`, and `surface` — see [Decision provenance](#decision-provenance) below.

---

## Enforcement models

Each policy carries an `enforcement_model` field with one of three values. Operators choose the model when they create the policy; for some categories, the model is implicit (audit is always detective; allowlists are always preventive).

| Model | Meaning | When the decision happens | Can it stop the action? |
|-------|---------|---------------------------|-------------------------|
| **preventive** | Block-before-action | Synchronously, before the action runs | Yes (raises a violation that the agent's chokepoint must honor) |
| **detective** | Alert-after-action | After the event lands in the audit / event stream | No — the action already happened. Auto-kill / revoke can be triggered reactively. |
| **shaping** | Guide via prompt / tool list | Up front, by what the agent *can see* | Indirectly — the agent never tries the action |

### Per agent type

Not every model is available on every agent. The matrix below — backed by the `surface` field on decision provenance (G6) and the [`COWORK_ENFORCEMENT_MATRIX.md`](https://github.com/waxell/waxell/blob/main/agentforge/areas/connect/plans/COWORK_ENFORCEMENT_MATRIX.md) — describes what's possible:

| Agent type | preventive | detective | shaping | Notes |
|---|:-:|:-:|:-:|---|
| `claude-code` | ✅ | ✅ | ✅ | Hooks block before action; OTel + audit catch what hooks miss; system prompt + tool list shape behavior. |
| `claude-cowork` | partial | ✅ | ✅ | **No in-VM hooks.** Preventive only at cloud chokepoint and via the MCP shim's `waxell_check_policy`. Local file edits / terminal commands cannot be blocked — see [Cowork governance topology](#cowork-governance-topology). |
| `codex` | partial | ✅ | partial | Codex has no public hook API. Preventive at cloud chokepoint + network filter; detective via session telemetry. Shaping via system-prompt config. |
| `mcp` (generic) | ✅ at cloud | ✅ | ✅ | The whole agent talks to us through Connect domains — every call is the chokepoint. |
| `custom` | depends | ✅ | ✅ | Whatever your wrapper code chooses to enforce. |

### Decision provenance

Every governance decision carries a provenance record. Example payload (returned by the cloud handshake, surfaced in trace detail UI):

```json
{
  "metadata": {
    "provenance": {
      "policy_id": "abc123",
      "policy_name": "Claude Code Daily Budget",
      "policy_category": "cost",
      "enforcement_model": "preventive",
      "phase": "before_workflow",
      "surface": "cloud",
      "agent_id": "agent_xyz",
      "agent_type": "claude-code",
      "agent_groups": ["engineering", "interns"]
    }
  }
}
```

The `surface` field (one of `in-process | cloud | network | audit-stream`) tells you *where* the decision was made — useful when debugging "why did this not block on my laptop." See `infra/waxell-infra/src/waxell_infra/policies/dynamic/manager.py` `_attach_provenance` for the implementation.

---

## Scope dimensions

A single policy can target *which* agents to apply to via several scope dimensions. They compose with AND semantics — a policy with both `scope_agents` and `scope_agent_groups` set applies to agents that match *both*.

| Dimension | Field | Example |
|-----------|-------|---------|
| Agent name | `scope_agent_names` | `["my-coding-bot", "my-research-bot"]` |
| Agent ID | `scope_agent_ids` | `["agent_xyz"]` |
| Agent type | `scope_agent_types` | `["claude-code", "codex"]` |
| Agent groups | `scope_agent_groups` | `["engineering"]` (groups defined in `ConnectAgentGroup`) |
| Enforcement model | `scope.enforcement_models` | `["preventive"]` (skip the same policy from re-firing as a detective check) |

### Cookbook

**"Apply this only to Claude Code agents in the 'interns' group"**

```json
{
  "name": "Intern Claude Code Cap",
  "category": "cost",
  "scope_agent_types": ["claude-code"],
  "scope_agent_groups": ["interns"],
  "config": {
    "daily_cost_limit": 5.00,
    "action_on_exceed": "block"
  }
}
```

**"Apply to all external agents — claude-code, codex, and mcp"**

```json
{
  "name": "External Egress Allowlist",
  "category": "network",
  "scope_agent_types": ["claude-code", "codex", "mcp"],
  "config": {
    "allowed_domains": ["api.openai.com", "api.anthropic.com", "*.internal.com"],
    "block_external": true,
    "action_on_violation": "block"
  }
}
```

**"Detective audit on everything, no scope filter"**

```json
{
  "name": "Org-wide Audit",
  "category": "audit",
  "enforcement_model": "detective",
  "config": {
    "log_inputs": true,
    "log_outputs": true,
    "retention_days": 365
  }
}
```

---

## Lifecycle: from key to enforced policy

When a Connect agent registers via the resolve-key endpoint, Waxell does a handshake that returns the agent's currently-applicable governance state. This is **D3** in the implementation plan and lands at `controlplane/.../views/connect_views.py:_build_governance_handshake` (~lines 1268-1338).

```
┌─────────┐  POST /api/v1/connect/agents/resolve-key/   ┌──────────────────┐
│  Agent  │ ───────────────────────────────────────────►│  Controlplane    │
└─────────┘                                              └──────┬───────────┘
                                                                │
                                                                │ resolve agent
                                                                │ collect attached templates
                                                                │ compute guard etag
                                                                │
┌─────────┐  200 { agent_id, governance: {              ┌──────▼───────────┐
│  Agent  │◄──   attached_template_names: [...],        │  Controlplane    │
└─────────┘     guard_etag: "...",                       └──────────────────┘
                mcp_tool_hints: [...]
              } }
```

The agent (or its installer / hooks layer) then:

1. **Caches the attached template names** so the local guard knows what categories to enforce locally.
2. **Stores the `guard_etag`** so it can poll `/api/v1/connect/guard-config/` for changes.
3. **Registers the `mcp_tool_hints`** (e.g. `waxell_check_policy`, `waxell_budget_status`, `waxell_record_decision`) as MCP tools it can call mid-session. This is the "ask the cloud if I can do this" path used by Cowork.

From that point on:

- Every Connect domain call goes through the policy interpreter on the cloud.
- Every Claude Code hook event hits the local guard, which evaluates against the cached config.
- Every Cowork session writes to `audit.jsonl`; the watcher streams it to the cloud where detective evaluators fire.

If you re-register or rotate the API key, the handshake re-runs and you get a fresh `guard_etag`. Local guards re-fetch config when the etag changes.

---

## Policy categories

The 26 standard categories and 3 allowlist categories (A6) seeded by `infra/waxell-infra/src/waxell_infra/policies/seed.py`. For *applicability per agent type*, see [`POLICY_APPLICABILITY.md`](https://github.com/waxell/waxell/blob/main/agentforge/areas/connect/plans/POLICY_APPLICABILITY.md). For *which templates ship pre-configured per type*, see [`POLICY_COVERAGE_MATRIX.md`](https://github.com/waxell/waxell/blob/main/agentforge/areas/connect/status/POLICY_COVERAGE_MATRIX.md).

The categories most relevant to Connect agents:

| Category | What it does | Notes for Connect |
|---|---|---|
| `cost` | Token / dollar caps per session and per day | Per-agent and per-group windows |
| `rate-limit` | RPS / RPM / concurrent-session caps | Cloud chokepoint enforcement |
| `kill` | Auto-disable on error rate | Reactive — kills the agent's API key |
| `audit` | Capture inputs/outputs/steps with retention | Always detective |
| `safety` | `max_steps`, `max_tool_calls`, deny-list of tools | Pairs with `tool-allowlist` for positive control |
| `network` | Domain allow/block, protocol restrictions | Enforced at network filter (desktop app) + LLM proxy |
| `scope` | Records-modified / files-changed / transaction caps | Counts come from hook payloads + audit stream |
| `code-execution` | Sandbox required, blocked commands, allowed languages | Hook-layer for CC; audit for Cowork |
| `content` | PII / credential / prompt-injection scanning | Runs on inputs and outputs that cross the cloud |
| `privacy` | Data minimization, residency, retention by type | Detective — applies to what we capture |
| `identity` | AI disclosure footer + impersonation judge | Shaping (prepend/append) + post-hoc judge |
| `compliance` | Meta-validator (HIPAA, SOC 2, etc.) | Asserts other categories are configured |

### Codex-specific templates

OpenAI's Codex CLI is the third-party agent we explicitly support for governance day-one. The seed ships **8 Codex templates** (all `scope_agents=["codex"]`):

1. **Codex Session Budget** (`cost`, preventive) — `seed.py:1638-1666`
2. **Codex Daily Budget** (`cost`, preventive) — `seed.py:1667-1695`
3. **Codex Rate Limit** (`rate-limit`, preventive) — `seed.py:1696-1715`
4. **Codex Kill Switch** (`kill`, preventive) — `seed.py:1716-1737`
5. **Codex Tool Boundary** (`safety`, preventive) — `seed.py:1738-1759`
6. **Codex Network Egress** (`network`, preventive) — `seed.py:1760-1783`
7. **Codex File Scope** (`scope`, preventive) — `seed.py:1784-1805`
8. **Codex Full Audit** (`audit`, detective) — `seed.py:1806-1828`

There's no `Codex Model Restriction` template — Codex picks its own model and we don't intercept that path today. The set deliberately mirrors Claude Code's cost / rate-limit / kill / safety / network / scope coverage with Codex-tuned defaults.

---

## Cowork governance topology

> **Cowork is different.** This subsection exists because operators repeatedly assume Cowork enforcement looks like Claude Code's. It does not.

Claude Cowork runs in a sandboxed VM that we don't control. There is no `~/.claude/settings.json` to install hooks into, and **no PreToolUse-style interception is possible** for actions Cowork takes inside the VM (file edits, terminal commands, third-party MCP calls, raw model conversation).

What we *do* have:

- **Detective evaluation on the audit stream.** Every Cowork session writes `audit.jsonl` to `~/Library/Application Support/Claude/local-agent-mode-sessions/` (macOS) / `%APPDATA%\Claude\local-agent-mode-sessions\` (Windows). The watcher (`observe/.../integrations/cowork/audit_watcher.py`) tails these files and streams events to the cloud. Detective policies fire ~5 seconds after the action.
- **Preventive enforcement at the cloud chokepoint.** Anything Cowork does *through* Connect (creating tasks, posting to channels, uploading files, querying tables) is governable preventively — those calls hit our domain endpoints.
- **Reactive kill via `ConnectAgentLinkedKey` revocation.** When a detective policy fires, we can revoke the agent's keys, killing the next API call. The current session continues until it tries to refresh credentials.
- **Shaping via the MCP shim.** The Cowork MCP shim (`observe/.../integrations/cowork/mcp_server.py`) exposes three tools: `waxell_check_policy(action_kind, action_summary)`, `waxell_budget_status()`, `waxell_record_decision(reasoning, decision, confidence)`. A well-configured Cowork session calls `waxell_check_policy` *before* taking risky actions. This is the **proactive shaping path** — Cowork voluntarily checks in with us. See E7 in `EXTERNAL_AGENT_GOVERNANCE.md`.

What this means in the UI: every Cowork policy detail page should show:

> **Cowork enforcement: detective. Violations are alerted within ~5s; auto-kill is optional.**

Don't write copy that promises preventive enforcement on local Cowork actions. Find the cloud chokepoint, the audit-stream check, or the MCP-shim shaping path instead.

For the per-category Cowork-applicability matrix, see [`COWORK_ENFORCEMENT_MATRIX.md`](https://github.com/waxell/waxell/blob/main/agentforge/areas/connect/plans/COWORK_ENFORCEMENT_MATRIX.md) (E2 in the engineering plan).

---

## Allowlists (A6) — the three new categories

Three categories were added in 2026-05-03 specifically for the external-agent governance story:

### `tool-allowlist`

Positive-list governance for which tools an agent may invoke. Differs from `safety.blocked_tools` (deny-only) by adding an explicit allowlist — *absence from the allowlist blocks*. Critical for Codex / generic MCP whose tool surface is unbounded.

```json
{
  "category": "tool-allowlist",
  "config": {
    "allowed_tools": ["Read", "Write", "Edit", "Bash"],
    "blocked_tools": [],
    "action_on_violation": "warn"
  }
}
```

**Recommended rollout:** ship as `warn` first, populate `allowed_tools` from observed traffic, then promote to `block`.

### `mcp-server-allowlist`

Constrain which MCP servers an agent may register or connect to. Today's MCP ecosystem lets agents register arbitrary third-party servers; this lever pins them to a curated set.

```json
{
  "category": "mcp-server-allowlist",
  "config": {
    "allowed_servers": ["github", "filesystem", "waxell-observe-claude-code"],
    "blocked_servers": [],
    "action_on_violation": "block"
  }
}
```

Especially important for `mcp` agent type — that's the whole point of an MCP agent.

### `prompt-allowlist`

Constrain which named system-prompt templates an agent may load. Cross-references the prompt-management system. Use case: enterprise compliance where prompts are vetted assets and engineers shouldn't be free to swap them.

```json
{
  "category": "prompt-allowlist",
  "config": {
    "allowed_prompts": ["legal-reviewed-v3", "approved-internal-prompt"],
    "blocked_prompts": [],
    "action_on_violation": "block"
  }
}
```

> **Honesty note:** Today these three categories evaluate at the cloud chokepoint only. Local-tool calls and locally-registered MCP servers bypass them until the desktop policy poller (UNIFIED_DESKTOP_APP plan, Phase B) ships and consumes the new `attached_template_names` from the D3 handshake. Until then, treat allowlists as detective-only on `claude-code` / `codex` and preventive on `mcp` / `claude-cowork` (which talk to us through cloud chokepoints anyway).

---

## Operator workflow

A typical "I want to add governance to a new external agent" flow:

1. **Register the agent.** From the Connect UI: `/connect/agents/new` → choose `agent_type` → save. Or programmatically: `POST /api/v1/connect/agents/` with `agent_type: "claude-code"`. You get back a `slug` and `api_key`.
2. **Install on the user machine.** Run the [Waxell installer](../installation.md) — it writes `~/.waxell/config` and (for CC) `~/.claude/settings.json` hook entries.
3. **Attach templates.** From `/connect/agents/<slug>` → Governance tab → "Attach template" → pick from the templates that show under the agent's type.
4. **Verify.** Trigger a small action; check the trace's Governance tab. Provenance should show your policy as the matching scope. If you see `surface: "cloud"` but expected `"in-process"`, the local guard hasn't picked up the etag yet — reload the agent.
5. **Tighten.** Start with `warn`. After a week of clean signal, promote to `block`.

---

## For engineers

If you're implementing or extending Connect governance, the engineering doc is the authoritative reference:

- [`EXTERNAL_AGENT_GOVERNANCE.md`](https://github.com/waxell/waxell/blob/main/agentforge/areas/connect/plans/EXTERNAL_AGENT_GOVERNANCE.md) — the unified plan with audit, shipping log, and full TODO list.
- [`UNIFIED_DESKTOP_APP.md`](https://github.com/waxell/waxell/blob/main/agentforge/areas/deployment/plans/UNIFIED_DESKTOP_APP.md) — the on-device endpoint daemon (network filter + cowork watcher + claude-code hooks + policy poller in one .app).
- [`POLICY_COVERAGE_MATRIX.md`](https://github.com/waxell/waxell/blob/main/agentforge/areas/connect/status/POLICY_COVERAGE_MATRIX.md) — template inventory by agent type.
- [`POLICY_APPLICABILITY.md`](https://github.com/waxell/waxell/blob/main/agentforge/areas/connect/plans/POLICY_APPLICABILITY.md) — applies-natively / needs-variant / excluded decision per category.
- [`COWORK_ENFORCEMENT_MATRIX.md`](https://github.com/waxell/waxell/blob/main/agentforge/areas/connect/plans/COWORK_ENFORCEMENT_MATRIX.md) — Cowork preventive/detective/shaping/N-A per category.

## See also

- [Observe governance](../observe/features/governance.md) — same policy categories applied to SDK-built agents
- [Policy categories overview](../observe/features/policy-categories.md) — the 26+3 categories with their full schemas
- [Connect installation](../installation.md) — how to bring a new agent into Connect


--------------------------------------------------------------------------------
# Connect Slack Integration
URL: https://waxell.ai/docs/connect/slack-integration
--------------------------------------------------------------------------------

# Slack Integration

Bring Waxell Connect into Slack: agent activity lands in your channels
with one-click triage, approvals get decided where your team already
works, and every user has a personal **My Work** dashboard in the app's
Home tab.

## Install

1. Navigate to **Connect > Settings > Slack**
2. Click **Add to Slack** and authorize the app for your workspace
3. Pick an **activity channel** (all agent events) and, optionally, an
   **alerts channel** (urgent events only)
4. Optionally map individual Connect channels to specific Slack channels

Prefer to run the integration under your own Slack app? Download the
app manifest from the same settings page, create the app in your Slack
admin console, and paste its credentials — everything else works
identically.

## What lands in Slack

- **Agent events** — messages, task completions, errors, escalations,
  and governance events post to your mapped channels, threaded so
  conversations stay together. Agent owners are @mentioned.
- **Triage buttons** — actionable events carry Claim / Done / Snooze /
  Escalate buttons. Snooze offers a duration picker; Escalate lets you
  add context for whoever picks it up. The message updates in place so
  everyone sees the outcome.
- **Approvals** — when an agent requests sign-off, the request posts
  with **Approve** and **Deny** buttons. Denials ask for a reason, and
  the decision (with who and why) replaces the buttons and notifies the
  requester.

## Talking back

- **Reply in a thread** to send your response into Connect — it reaches
  the agent's inbox like any Connect message.
- **@mention an agent** by its slug in a mapped channel to create a
  targeted, actionable event for that agent.

## App Home — My Work

Open the Waxell Connect app in Slack's sidebar for your personal
dashboard: unread inbox items with inline triage, approvals waiting on
you, and your agents' live status. It refreshes automatically as you
act on items.

## The /connect command

| Command | What it shows |
|---|---|
| `/connect inbox` | Your unread inbox items, with Claim/Done buttons |
| `/connect agents` | Agent presence — online, idle, offline |
| `/connect activity` | The latest events across your workspace |
| `/connect help` | Command reference |

## User identity

Slack users map to Waxell users automatically when their Slack and
Waxell emails match — the first interaction links them. Admins can also
manage mappings under **Connect > Settings > Slack > User Mapping**.
Actions taken in Slack are attributed to the acting user in Connect's
audit trail, exactly as if taken in the web app.


================================================================================
SECTION: WAXELL ENDPOINTS
Endpoint visibility and on-device guard for AI tools your team already uses
================================================================================

--------------------------------------------------------------------------------
# Waxell Endpoints — Govern the AI on Every Machine
URL: https://waxell.ai/docs/endpoints/overview
Description: Waxell Endpoints brings the AI tools running on your team's laptops — Claude Desktop, Cursor, Claude Code, Copilot, ChatGPT, browsers — under discovery, observability, and governance. Start with shadow-AI visibility, opt into capture per host.
--------------------------------------------------------------------------------

# Waxell Endpoints

**Endpoints** is the part of Waxell that governs the AI tools your team runs
*locally* — the desktop apps, IDEs, CLIs, and browser tabs on real laptops, not
just the agents you build with the Waxell SDK. It answers three questions every
security and platform team is now asking:

1. **What AI is actually running on our machines?** (shadow-AI discovery)
2. **What is it doing?** (per-app, per-device, per-user activity)
3. **Can we control it?** (block, allow, or capture — under policy)

## The problem: ungoverned AI on the endpoint

AI tools install themselves one developer at a time. Claude Desktop, Cursor,
Claude Code, GitHub Copilot, ChatGPT's desktop app, Perplexity, a dozen
browser-based assistants — each one talks to a model provider over the network,
often touching source code, customer data, or secrets. Most organizations have
**no inventory** of which of these are running, on whose machine, against which
provider.

Waxell Endpoints closes that gap with a small, signed desktop agent that you
deploy either to a whole fleet (via MDM) or to a single machine (a direct
download). Once installed, the agent:

- **Discovers** every AI app on the machine and reports it to your workspace.
- **Observes** the metadata of their outbound AI traffic (which app, which host,
  when) — attributed to the process and the user.
- **Governs** that traffic through policy: leave it observe-only, *block* a
  specific app, or — only when you explicitly turn it on — *capture* the
  request/response payloads with on-device DLP redaction.

## The headline principle: visibility first, capture is opt-in

**Capture is OFF by default**
Installing the agent turns on **discovery and metadata only**. Nothing is
intercepted, decrypted, or uploaded as payload until an admin explicitly enables
capture for a specific AI host on the [Guard cascade](./guard/overview). And even
then, TLS is terminated **only for catalog AI hosts** — never banking, health, or
mail — and secrets/PII are redacted **on the device** before anything leaves it.


This ordering is deliberate. You get an accurate, low-risk inventory and activity
picture on day one. Payload capture — the higher-sensitivity capability — is a
deliberate, per-host, policy-gated decision you make later, with full privacy
controls. See [Privacy & DLP](./guard/privacy-and-dlp).

## Governed vs. ungoverned: the gap analysis

The product is, at heart, a gap analysis. For every machine it shows you:

| | Ungoverned (before) | Governed (after Waxell) |
|---|---|---|
| **Inventory** | Unknown which AI apps exist | Every AI app, per device, listed |
| **Attribution** | No idea who/what is calling providers | Per-process, per-user, per-host flows |
| **Control** | None | Block / allow / capture under policy |
| **Evidence** | None | Redacted, attributed activity record |

## Two ways to deploy

- **A whole fleet → [MDM](./mdm/overview).** Push one per-tenant profile plus the
  signed installer from Hexnode, Jamf, Kandji, or Intune. Devices enroll silently,
  zero end-user action.
- **One machine → [Manual install](./manual-install).** Download the signed app
  and install it yourself in a couple of minutes — no MDM, no IT. Great for trying
  Waxell on your own laptop first.

## Where to go next

- New to the concept? Read [How it works](./how-it-works) and
  [Key concepts](./concepts).
- Rolling out to a team? Start at [Deploy to a Fleet (MDM)](./mdm/overview).
- Just you? Jump to [Manual install](./manual-install).
- Ready to control traffic? See the [Guard cascade](./guard/overview).


--------------------------------------------------------------------------------
# How Waxell Endpoints Works
URL: https://waxell.ai/docs/endpoints/how-it-works
Description: The architecture behind Waxell Endpoints — the desktop agent's components, how it enrolls against a per-tenant CA, what it observes vs. captures, and how on-device DLP keeps raw payloads from ever leaving the machine.
--------------------------------------------------------------------------------

# How Waxell Endpoints Works

This page explains what the desktop agent actually does on a machine and how the
pieces fit together. For a component-by-component reference (file locations,
service names), see [What gets installed](./reference/what-gets-installed).

## The shape of it

```
 ┌─────────────────────────── managed device ───────────────────────────┐
 │                                                                       │
 │   AI apps (Claude Desktop, Cursor, Claude Code, Copilot, browsers…)   │
 │            │                                                          │
 │            ▼                                                          │
 │   Network monitor  ──────────►  metadata flows  ─┐                    │
 │   (observe / optionally block)                   │                    │
 │            │ (only if capture enabled for host)  │                    │
 │            ▼                                      │                    │
 │   On-device DLP redaction  ──►  redacted payload ─┤                    │
 │                                                   │                    │
 │   Policy daemon  ◄── guard policy (long-poll) ────┼───┐                │
 │   Cowork watcher / Claude Code hook  ─────────────┘   │                │
 │                                                       │                │
 └───────────────────────────────────────────────────────┼──────────────┘
                                                          ▼
                                            Waxell control plane
                                   (AI Endpoints: Apps · Devices · Guard ·
                                    Captures · Flow stream)
```

## The components

The agent is a few cooperating pieces. On macOS they ship as system extensions
plus helper processes inside `Waxell Setup.app`; on Windows they ship as
LocalSystem services plus a hook executable.

- **Network monitor** — watches outbound connections from AI apps. On macOS it's a
  Network Extension content filter (`flowmonitor`); on Windows it's an ETW-based
  LocalSystem service. By default it records **metadata only** (which process,
  which AI host, when). It can optionally **block** a process/app when policy says
  so.
- **Proxy / capture layer** — when (and only when) an admin enables capture for a
  specific host, this terminates TLS for that host, reads the request/response,
  hands it to DLP, and uploads the redacted result. On macOS this is the
  `proxymonitor` transparent-proxy extension; on Windows it rides the WinDivert
  driver (see the [HVCI caveat](./mdm/windows)).
- **Policy daemon** — long-polls the control plane for the effective
  [guard policy](./guard/overview) and writes it locally, so the monitor knows what
  to observe, block, or capture. Policy changes reach the device within minutes.
- **Cowork watcher + Claude Code hook** — bring Claude Cowork and Claude Code
  activity under the same governance umbrella as raw network flows.
- **Setup app** — the GUI users see: browser sign-in, status, and a Network
  Monitor view.

## Enrollment and the tenant CA

To attribute traffic to a device and (optionally) terminate TLS without breaking
trust, each tenant has its own **root certificate authority**.

1. The MDM profile (or the local installer) **trusts the tenant root CA** on the
   device.
2. On first run the device **enrolls**: it generates a key, sends a CSR, and
   receives a per-device intermediate certificate.
3. From then on, the device can mint short-lived leaf certs that **chain to the
   tenant CA** the OS already trusts — which is what lets the proxy layer terminate
   TLS *for catalog AI hosts only* without a certificate error.

Because the CA is **per tenant**, one customer's CA never touches another's, and
the CA-trust payload is the single universal primitive every MDM supports — which
is why deployment is MDM-agnostic.

## Observe vs. capture

This distinction is the heart of the product:

| | **Metadata flows** (default) | **Payload capture** (opt-in) |
|---|---|---|
| What | Process, AI host, timestamp, bytes | The actual request/response content |
| TLS | Not terminated | Terminated for that host only |
| Scope | All discovered AI apps | Only hosts you enable on Guard |
| Privacy | Nothing sensitive leaves the device | DLP-redacted on-device before upload |
| Surfaces in | Flow stream, AI Apps, Devices | Captures tab |

## On-device DLP

When capture is on, redaction happens **on the device, before upload**. Secrets
and PII are stripped locally; the raw payload never leaves the machine. The
control plane only ever receives the redacted form. This is what makes capture
safe to turn on for genuine AI hosts. Details in
[Privacy & DLP](./guard/privacy-and-dlp).

## What you see in the control plane

Everything lands in **Governance → Connect → AI Endpoints**, a tabbed surface:

- **AI Apps** — every AI agent the scanner found, per device, with its capture
  ceiling and recommended stance.
- **Devices** — each device's apps mapped to a governed agent.
- **Guard** — the layered [policy cascade](./guard/overview).
- **Captures** — redacted, attributed payloads (once capture is on).
- **Flow stream / AI Activity** — the live metadata feed.
- **Setup** — per-tenant profile/installer downloads.

Next: [Key concepts](./concepts).


--------------------------------------------------------------------------------
# Key Concepts & Glossary
URL: https://waxell.ai/docs/endpoints/concepts
Description: The vocabulary of Waxell Endpoints — device enrollment, tenant CA, AI apps, app types and capture ceiling, metadata flows vs. payload capture, the Guard cascade, and the device→agent map.
--------------------------------------------------------------------------------

# Key Concepts

A quick glossary of the terms you'll see across the Endpoints surface and these
docs. If you've read [How it works](./how-it-works), this formalizes the
vocabulary.

## Device & identity

- **Managed device** — a Mac or Windows machine running the Waxell agent, whether
  deployed by [MDM](./mdm/overview) or installed [manually](./manual-install).
- **Tenant CA** — your tenant's private root certificate authority. Trusted on each
  device so the agent can attribute traffic and, when capture is enabled, terminate
  TLS for catalog AI hosts without breaking trust.
- **Enrollment** — the first-run handshake where a device proves itself and
  receives a per-device intermediate cert chained to the tenant CA.
- **Device → agent map** — on the **Devices** tab you map each device's AI apps to
  a *governed agent*, so intercepted traffic and on-device tools resolve to that
  agent for policy and reporting.

## AI apps

- **AI app** — any application the on-device scanner identifies as talking to an AI
  provider: Claude Desktop, Cursor, Claude Code, GitHub Copilot, ChatGPT desktop,
  Perplexity, browser assistants, and more.
- **App type** — the *class* of an AI app (e.g. a CLI coding agent vs. a desktop
  chat app vs. a browser). Guards can be set per app type, not just per app.
- **Governance level** — how fully Waxell can govern an app type. You'll see values
  like **full**, **transcript**, or **muted**, depending on the app's runtime and
  whether it pins TLS.
- **Capture ceiling** — the *maximum* capture an app can support given how it works
  (for example, an app that pins TLS can't be payload-captured by the proxy). The
  ceiling is an upper bound; your Guard policy chooses how much of it to use, and
  the default is none.

## Observe vs. capture

- **Metadata flow** — the default signal: which process, which AI host, when, how
  much. No payload, no TLS termination. Shown in the **Flow stream / AI Activity**.
- **Payload capture (Layer-0 / L0)** — the opt-in signal: the actual
  request/response content for an enabled host, TLS-terminated and
  **DLP-redacted on-device** before upload. Shown in the **Captures** tab.
- **Catalog AI host** — a host on Waxell's curated list of AI providers (e.g.
  `api.openai.com`, `api.anthropic.com`). Capture can only target catalog AI hosts
  — never banking, health, or mail.

**The cardinal rule**
Capture is **off by default**. It is enabled deliberately, per host, on the
[Guard cascade](./guard/overview). See [Privacy & DLP](./guard/privacy-and-dlp)
for what that means for data handling.


## Guard & policy

- **Guard** — the policy that controls what the agent observes, blocks, or captures
  on a device.
- **Guard cascade** — a layered policy resolved in precedence order
  **Global → App type → User group → Device → Agent group → Agent**. More-specific
  layers override less-specific ones **field by field**. See
  [Guard cascade overview](./guard/overview).
- **Layer / override** — a partial config set at one scope of the cascade. You only
  specify the fields you want to change; everything else inherits from the layer
  above.
- **Resolve** — the Guard tab's simulator: pick an app type / user group / device /
  agent and see the *effective* config and which layer each field came from.

## Detective vs. preventive

- **Detective** — observe and record after the fact (the default posture for
  metadata and capture).
- **Preventive** — actively block a connection (process/app blocking via the
  network monitor) before it completes.

## Surfaces

- **AI Endpoints** — the control-plane area (**Governance → Connect → AI
  Endpoints**) with the Apps, Devices, Guard, Captures, Flow, and Setup tabs.

Next: deploy to a [fleet](./mdm/overview) or a [single machine](./manual-install),
then set up [Guard](./guard/overview).


--------------------------------------------------------------------------------
# Deploy to a Fleet (MDM) — Overview
URL: https://waxell.ai/docs/endpoints/mdm/overview
Description: How Waxell deploys the desktop endpoint agent to a whole fleet via any MDM — one per-tenant profile plus the signed installer, silent enrollment, capture off by default.
--------------------------------------------------------------------------------

# Deploy to a Fleet (MDM)

The MDM track is how you put the Waxell desktop endpoint agent — the AI-endpoint
("Shadow AI") monitor — onto an entire managed fleet **with zero end-user
action**.

Push one profile from your MDM and every targeted device silently:

1. **Trusts your tenant CA** — the per-tenant root certificate lands fleet-wide.
2. **Enrolls** — each device generates a CSR and gets a per-device intermediate
   chained to your tenant CA.
3. **Discovers its AI apps** — Cursor, Claude Code, ChatGPT desktop, Copilot, and
   the rest are detected on the device.
4. **Reports** — the device and its discovered apps appear in
   **Governance → Connect → AI Endpoints**.

No "Allow" dialogs, no per-user install, no clicks. The profile pre-authorizes
everything the agent needs.

**Capture is OFF by default**
Deploying via MDM turns on **discovery + metadata reporting only**. Nothing is
intercepted and no TLS is terminated until an admin explicitly enables a host.
Interception is a separate, later motion driven from the
[Guard cascade](../guard/overview).


## Choose your path

| You have... | Use this |
|---|---|
| A managed fleet (Hexnode / Jamf / Kandji / Mosyle / Intune) | This MDM track — keep reading |
| A single machine to set up by hand | The [Manual install](../manual-install) page |

## Why it's MDM-agnostic

The same deployment works on **Hexnode, Jamf, Kandji, Mosyle, and Intune**
because the profile uses only primitives every MDM supports.

**On macOS** the per-tenant `.mobileconfig` carries three payloads:

- `com.apple.security.root` — the one universal CA-trust primitive; trusts the
  tenant root CA fleet-wide (silent).
- `com.apple.system-extension-policy` — pre-approves the Waxell system
  extensions by Team ID, so they load without a user prompt.
- A managed app config payload — delivers the enrollment endpoint, tenant key,
  and capture scope. The agent reads it from Managed Preferences and writes the
  extension's enroll-config and capture-policy.

**On Windows** there is no single "managed app config" artifact, so the
equivalent `waxell-setup.ps1` script does the same job using universal Windows
primitives:

- Imports the tenant root CA into `LocalMachine\Root` (the universal CA-trust
  primitive).
- Writes the managed config to `HKLM\SOFTWARE\Waxell` (`ApiUrl`, `ApiKey`,
  `CaptureEnabled`, `InterceptHosts`, `InterceptProcesses`), which the
  `WaxellNetworkMonitor` service reads. The script ACLs the key to
  `SYSTEM` + `Administrators` so the API key isn't world-readable.

Because both sides lean on nothing but CA-trust + a config payload (+ a
system-extension allow-list on Mac), any MDM that can push a custom profile,
a trusted certificate, and an app can deliver Waxell.

## The artifacts

Each tenant deploys three things. Two are **per-tenant secrets** (they carry your
tenant API key); the installer is the same signed binary for everyone.

| Artifact | Platform | What it is | Sensitivity |
|---|---|---|---|
| The profile | macOS | `.mobileconfig` — trusts the CA, pre-approves the system extensions, delivers the managed config | **Secret** — embeds your tenant API key |
| The profile | Windows | `waxell-setup.ps1` — trusts the CA, writes `HKLM\SOFTWARE\Waxell` | **Secret** — embeds your tenant API key |
| The tenant root CA | both | The per-tenant root certificate (PEM) the device's per-host leaves chain to | Public cert (not secret) |
| The signed installer | macOS | `Waxell-Setup-<ver>.pkg` — the agent (network extension, app scanner, enrollment) | Same binary for all tenants |
| The signed installer | Windows | `Waxell-<ver>.msi` — the agent services + tooling | Same binary for all tenants |

**The profile is a secret**
The per-tenant `.mobileconfig` / `waxell-setup.ps1` carries your tenant API key
in its body. Treat it like a credential — scope it to the pilot tenant, and don't
post it where it can leak. The root CA PEM is not sensitive.


### Installer signing status

- **macOS** — the `.pkg` is **signed (Developer ID) and Apple-notarized**, built
  under Team ID `2RWC9968XY`, and distributed from `install.waxell.dev`. When
  pushed via MDM the system extension activates silently because the profile
  pre-authorizes that Team ID.
- **Windows** — the `.msi` is **code-signed** and distributed from
  `install.waxell.dev` (`latest/WaxellSetup-x64.msi`). Both paths work: MDM push
  (Hexnode/Intune install it as `SYSTEM`) and direct self-download — an
  individual user can double-click it without an "unknown publisher" prompt. See
  the [manual install guide](../manual-install#windows) for the single-machine
  flow.

## Where an admin gets the artifacts

You don't need to be in the loop manually. There are two paths.

### In-app (recommended)

Go to **Governance → Connect → AI Endpoints → Setup**. The panel offers one-click,
per-tenant downloads of your `.mobileconfig` / `waxell-setup.ps1`, your tenant
root CA, and links to the signed installer URLs on `install.waxell.dev`. It also
shows the copy-paste command with your tenant key and flags the
API-key-is-secret handling inline.

### Via the documented endpoints

The same bytes are available from the generator endpoints with a tenant API key.

**macOS profile:**

```bash
curl -fsSL "https://api.waxell.dev/api/waxell/v1/endpoint/mdm-profile/" \
  -H "X-Wax-Key: $WAX_KEY" \
  -o waxell-ai-endpoints.mobileconfig
```

**Windows setup script:**

```powershell
Invoke-WebRequest `
  -Uri "https://api.waxell.dev/api/waxell/v1/endpoint/mdm-profile/?platform=windows" `
  -Headers @{ "X-Wax-Key" = $env:WAX_KEY } `
  -OutFile waxell-setup.ps1
```

**Tenant root CA (Windows cert profile / trust store):**

```powershell
Invoke-WebRequest `
  -Uri "https://api.waxell.dev/api/waxell/v1/endpoint/ca-root/" `
  -Headers @{ "X-Wax-Key" = $env:WAX_KEY } `
  -OutFile waxell-tenant-root.pem
```

**Leave capture off here**
These endpoints emit a capture-OFF profile by default. You can pre-enable capture
for specific hosts at generation time, but the intended path is to leave it off
and turn capture on later, per-app, from the [Guard cascade](../guard/overview).


## What happens after the push

Within minutes (macOS) or up to the next MDM sync (Windows), each device:

- trusts the tenant root CA,
- enrolls (CSR → per-device intermediate),
- scans itself and reports its AI apps,
- shows up in **Governance → Connect → AI Endpoints** under **AI Apps** and
  **Devices**.

Capture remains off the entire time. **Captures** stays empty until you enable a
host — at which point TLS is terminated **only** for the catalog AI hosts you've
enabled (never banking, health, or mail), with secrets and PII DLP-redacted
on-device before upload.

## Next steps

Pick the platform-specific runbook for your fleet:

- [macOS](./macos) — Hexnode / Jamf / Kandji / Mosyle, the `.mobileconfig` + `.pkg`.
- [Windows](./windows) — Intune (and Workspace ONE, JumpCloud, etc.), the cert +
  `waxell-setup.ps1` + `.msi`.

When you're ready to move from discovery to interception, see the
[Guard cascade](../guard/overview).


--------------------------------------------------------------------------------
# Deploy to a Mac Fleet (MDM)
URL: https://waxell.ai/docs/endpoints/mdm/macos
Description: Step-by-step — push the Waxell tenant profile and signed .pkg from Hexnode, Jamf, Kandji, or Intune so every managed Mac silently enrolls, trusts the tenant CA, and reports its AI apps.
--------------------------------------------------------------------------------

# Deploy to a Mac Fleet (MDM)

Push the Waxell AI-endpoint agent to your managed Macs with **zero user clicks**. The whole fleet enrolls against your tenant CA, each device discovers its AI apps, and metadata flows start reporting — all without a single "Allow" dialog on any machine.

You deliver exactly **two artifacts** through your MDM:

1. A **per-tenant configuration profile** (`.mobileconfig`) — trusts your tenant CA, pre-approves the Waxell system extensions, and delivers the managed config.
2. The **signed, notarized app** (`.pkg`) — installs the agent that runs the network extension, enrolls, and scans for AI apps.

The flow is identical on Hexnode, Jamf Pro, Kandji, Mosyle, and Intune-for-Mac because the profile uses only capabilities every macOS MDM supports. Detailed steps below use Hexnode; equivalents for the other MDMs are noted inline.

**Capture is OFF by default**
Deploying these artifacts enrolls the fleet and lets each device discover its AI apps and report flow metadata. It does **not** intercept anything. Interception is **default-OFF** — no TLS is terminated and nothing is captured until an admin explicitly enables a host on [Guard](../guard/enable-capture). See [turning capture on](#turning-capture-on) below.


For the Windows equivalent see [Windows](./windows). For the architecture and what gets governed, see the [Overview](./overview).

## The two artifacts

| Artifact | What it does | MDM object (Hexnode) |
|---|---|---|
| **The profile** — `GET /endpoint/mdm-profile` (per-tenant, generated live) | Trusts the tenant root CA, pre-approves the Waxell system extensions by Team ID, and delivers the managed config (enroll URL, tenant key, capture scope). | **Policies → macOS → Custom Configuration** |
| **The app** — `Waxell-Setup-<ver>.pkg` (signed Developer ID Installer + notarized) | The agent: runs the network extension, reads the managed config, enrolls against the tenant CA, scans for AI apps. | **Apps → +Add → macOS → Enterprise App** |

**Identifiers baked into every generated profile:**

| | Value |
|---|---|
| Team ID | `2RWC9968XY` |
| App | `dev.waxell.setup` |
| Transparent proxy extension | `dev.waxell.setup.proxymonitor` |
| Content filter extension | `dev.waxell.setup.flowmonitor` |

## Step 1 — Download your tenant's profile

Both artifacts are available in-app under **Governance → Connect → AI Endpoints → Setup**, or via the API.

The profile is generated **per-tenant** and **carries your tenant API key** in its managed config, so treat the downloaded file as a secret. Use a tenant API key:

```bash
curl -fsSL "https://api.waxell.dev/api/waxell/v1/endpoint/mdm-profile/" \
  -H "X-Wax-Key: $WAX_KEY" \
  -o waxell-ai-endpoints.mobileconfig
```

**Capture stays off here**
The profile ships with capture **disabled**. The intended path is to leave it off here and enable specific hosts later from the [Guard](../guard/enable-capture) cascade. If you do want to pre-enable hosts at download time, append `?hosts=api.openai.com,api.anthropic.com` to the URL.


The profile is emitted **unsigned** by design — your MDM (or you, with `productsign`) signs it on upload, which is the normal Hexnode/Jamf flow. Signing it makes it show as **Verified** instead of **Unsigned** on the device, but is not required.

## Step 2 — Get the app `.pkg`

Download the prebuilt, signed and notarized `.pkg` from **Governance → Connect → AI Endpoints → Setup**.

If you are building it yourself from the repo:

```bash
cd installers/macos
./build-pkg.sh --build --version 0.1.0     # builds the app, then a signed+notarized+stapled pkg
# → build/Waxell-Setup-0.1.0.pkg
```

The `.pkg` is signed with a **Developer ID Installer** certificate and **notarized + stapled** by Apple, so Gatekeeper accepts it silently on every managed Mac.

**One-time build prerequisites**
Building locally needs a **Developer ID Application** cert (signs the `.app`), a separate **Developer ID *Installer*** cert (signs the `.pkg`), and the `waxell-notary` notary profile (`xcrun notarytool store-credentials waxell-notary`). `build-pkg.sh` errors clearly if any is missing. If you downloaded the prebuilt `.pkg` you can skip all of this.


## Step 3 — Upload both to your MDM

### Hexnode

1. **Profile:** console → **Policies → New Policy → macOS → Custom Configuration** → upload `waxell-ai-endpoints.mobileconfig`. Name it "Waxell AI Endpoints".
2. **App:** **Apps → +Add → macOS → Enterprise App** → upload `Waxell-Setup-<ver>.pkg`. Name it "Waxell Setup".

### Other MDMs

The two-artifact pattern is the same everywhere — upload the `.mobileconfig` as a custom/raw profile and the `.pkg` as a managed enterprise app:

| MDM | Profile (the `.mobileconfig`) | App (the `.pkg`) |
|---|---|---|
| **Jamf Pro** | Computers → Configuration Profiles → Upload (custom payload) | Computers → Packages, then a Policy with an Install package action |
| **Kandji** | Library → Add → Custom Profile | Library → Add → Custom App (installer package) |
| **Intune-for-Mac** | Devices → Configuration → Create → macOS → Templates → Custom (upload the `.mobileconfig`) | Apps → macOS → Add → macOS app (PKG) |

## Step 4 — Target a device group and push

Add **both** the profile and the app to the **same** device group / target, then push.

Order does not matter. macOS applies the system-extension allow-list as soon as both the profile and the extension are present, so when the `.pkg` installs, the extension activates **silently** — no "Allow" dialog — because the profile pre-authorizes Team `2RWC9968XY`. The CA-trust payload lands silently too.

## Step 5 — Verify

Within a few minutes each Mac:

1. **Enrolls** against your tenant CA (CSR → per-device intermediate).
2. **Scans itself** and reports its discovered AI apps.
3. Activates the system extension **with no user prompt** (pre-approved by the profile).

Watch it land in **Governance → Connect → AI Endpoints**:

- **AI Apps** — the device's discovered apps (Cursor, Claude Code, ChatGPT desktop, …) populate here.
- **Devices** — map each app's process to a governed agent.
- **Captures** — empty until you enable a host on [Guard](../guard/enable-capture); redacted requests appear here afterward.

**Verify the extension on a single Mac**
On a managed device you can confirm the extensions activated with `systemextensionsctl list | grep dev.waxell` — both `proxymonitor` and `flowmonitor` should show as activated/enabled for Team `2RWC9968XY`.


## Turning capture on

Capture is off until you say so, per host. There are two ways to enable it:

- **The intended path — Guard.** Set it from the [Guard](../guard/enable-capture) cascade (global → app type → user group → device → agent) and let the policy flow to the device. No re-push of the profile needed.
- **At provisioning time.** Re-generate the profile with `?hosts=...` (see [Step 1](#step-1--download-your-tenants-profile)) and re-push it.

TLS is terminated **only** for catalog AI hosts you have explicitly enabled — never banking, health, or mail. Secrets and PII are DLP-redacted **on-device** before anything is uploaded.

**Default-OFF safety**
No host is ever intercepted by default. Nothing is terminated or captured until an admin enables that specific host on Guard. Deploying the profile and `.pkg` alone is purely enrollment + discovery.


## Removing a hand-installed dev copy first

If you have been testing the agent locally on your own machine, remove the dev copy **before** enrolling so the MDM-managed install is clean:

```bash
systemextensionsctl list | grep dev.waxell
sudo systemextensionsctl uninstall 2RWC9968XY dev.waxell.setup.proxymonitor || true
osascript -e 'tell application "Waxell Setup" to quit' 2>/dev/null || true
sudo rm -rf "/Applications/Waxell Setup.app"
```

Then enroll the Mac in your MDM (Enroll → macOS, ADE/automated or manual) and let the profile + app push down — now MDM-managed.

## What's inside the profile (reference)

The generated `.mobileconfig` emits three payloads:

| Payload | Type | What it does |
|---|---|---|
| **CA trust** | `com.apple.security.root` | Trusts the per-tenant root CA fleet-wide (silent). The device mints its enrollment chain to this root. |
| **System-extension allow-list** | `com.apple.system-extension-policy` | Allows Team `2RWC9968XY` to load `proxymonitor` (transparent proxy) + `flowmonitor` (content filter) with no user prompt. |
| **Managed config** | `dev.waxell.setup.proxymonitor` | Delivers `api_url`, `api_key`, `capture_enabled`, `intercept_hosts`, `intercept_processes`. The app's `ManagedConfigBridge` reads it and writes the extension's enroll-config + capture-policy. |

The device reads the managed config from `/Library/Managed Preferences/dev.waxell.setup.proxymonitor.plist`. `capture_enabled` is `false` whenever `intercept_hosts` is empty — which is the default — so a freshly pushed profile enrolls and discovers but never intercepts.

**Node / Electron apps**
The CA is delivered to the keychain, which covers keychain-respecting clients. Some Node/Electron apps (Claude Code, Cursor) also honor `NODE_EXTRA_CA_CERTS`; delivering that env var to GUI apps is MDM-specific and is a separate, optional follow-up.


## Related

- [Endpoints overview](./overview) — what the AI endpoint stack governs.
- [Enable capture on Guard](../guard/enable-capture) — turn interception on, per host.
- [Deploy to a Windows fleet](./windows) — the Intune/PowerShell equivalent.


--------------------------------------------------------------------------------
# Deploy to a Windows Fleet (Intune)
URL: https://waxell.ai/docs/endpoints/mdm/windows
Description: Step-by-step — push the Waxell trusted cert, managed-config script, and agent MSI from Intune so every managed PC enrolls, trusts the tenant CA, and reports its AI apps. Includes Memory Integrity (HVCI) guidance.
--------------------------------------------------------------------------------

# Deploy to a Windows Fleet (Intune)

Push the Waxell AI-endpoint agent to managed Windows PCs with **zero user clicks**. You stage three Intune objects — trust the tenant CA, deliver the managed config, install the MSI — and capability lights up the next time the user logs in.

This is the Intune-specific walkthrough. The same three-artifact pattern works on Workspace ONE, Kandji-for-Windows, JumpCloud, and other MDMs by analogy; only the console verbs differ. For the macOS path see [macOS](./macos). For where endpoints fit in the bigger picture see the [Overview](./overview).

**What this deploys**
The full AI-endpoint stack. Each device enrolls against your tenant CA, discovers its AI apps, reports metadata flows, and — once an admin turns capture on for a specific host — terminates TLS for **that AI host only**, DLP-redacts on-device, and uploads.

**Interception is default-OFF.** Nothing is terminated until you enable a host. See [Turning capture on](#turning-capture-on).


## The three artifacts

| Artifact | What it does | Intune object |
|---|---|---|
| **The cert** — the tenant root cert (PEM) | Trusts the tenant root CA fleet-wide so the proxy's per-host leaf certs chain to it | **Devices → Configuration → Create → Windows 10+ → Trusted certificate** |
| **The managed config** — `waxell-setup.ps1` (per-tenant, generated live) | Writes `HKLM\SOFTWARE\Waxell` (ApiUrl, ApiKey, CaptureEnabled, InterceptHosts, InterceptProcesses) and ACLs the key to SYSTEM + Administrators | **Devices → Scripts and remediations → Platform scripts → Windows**, or **Settings Catalog** |
| **The app** — `Waxell-&lt;ver&gt;.msi` | The installer: lays down WaxellSetup.exe, the WaxellNetworkMonitor / WaxellCoworkWatcher / WaxellPolicyDaemon services, and wax-hook.exe; the tenant CA gets pinned to the running services on first run | **Apps → Windows → Add → Win32 app (.intunewin)** |

Service-install order is handled by the MSI. `WaxellNetworkMonitor`, `WaxellCoworkWatcher`, and `WaxellPolicyDaemon` all install auto-start as LocalSystem, boot at device startup, and pick up the managed config the moment it's present.

## Where to get the artifacts

There are two ways to obtain the per-tenant cert and setup script.

- **In-app (recommended).** Go to **Governance → Connect → AI Endpoints → Setup**. Download the trusted cert and the managed-config script directly.
- **Scripted.** Pull them from the API with a tenant API key (shown in Steps 1–2 below).

The MSI is built with `build-msi.ps1` from the installer source (Step 3).

**The setup script carries your tenant API key**
`waxell-setup.ps1` is generated per-tenant and **embeds your tenant API key in the script body**. Treat the file as a secret — store it like any other credential and scope who can read it in your MDM.


## Step 1 — Download your tenant's setup script

Use a tenant API key. Set it as `WAX_KEY` in your shell first.

```powershell
Invoke-WebRequest `
  -Uri "https://api.waxell.dev/api/waxell/v1/endpoint/mdm-profile/?platform=windows" `
  -Headers @{ "X-Wax-Key" = $env:WAX_KEY } `
  -OutFile waxell-setup.ps1
```

- **Capture stays OFF** by default. To pre-enable capture for specific hosts at provisioning time, append `&hosts=api.openai.com,api.anthropic.com` to the URL. You can also leave it off here and enable per-app later from **Governance → Connect → AI Endpoints → Guard**.
- The script writes only `HKLM` values — no app install, no driver install. It is idempotent and safe to re-run.

## Step 2 — Download your tenant's root cert

```powershell
Invoke-WebRequest `
  -Uri "https://api.waxell.dev/api/waxell/v1/endpoint/ca-root/" `
  -Headers @{ "X-Wax-Key" = $env:WAX_KEY } `
  -OutFile waxell-tenant-root.pem

# Convert to DER (.cer) if your Intune cert profile wants it
certutil -encode waxell-tenant-root.pem waxell-tenant-root.cer
```

## Step 3 — Build the MSI

```powershell
cd installers\windows
powershell -ExecutionPolicy Bypass -File build-msi.ps1 -Version 0.7.0 -Sign -CertSubject "Your Code Signing Cert"
# -> dist\Waxell-0.7.0.msi
```

One-time prereqs: an **EV code-signing cert** (subject discoverable via `-CertSubject`), or a `.pfx` (`-CertPath` + `-CertPassword`). `build-msi.ps1` errors clearly if the cert can't be resolved.

Wrap the MSI as an `.intunewin` for upload:

```powershell
.\IntuneWinAppUtil.exe -c dist -s Waxell-0.7.0.msi -o dist\intunewin
# -> dist\intunewin\Waxell-0.7.0.intunewin
```

**Code-signing status**
The Windows MSI is **code-signed** and publicly distributed at
`install.waxell.dev/latest/WaxellSetup-x64.msi` — the MDM push below installs it
as **SYSTEM** with no prompts, and an individual user can also double-click the
same MSI directly without a SmartScreen warning (see the
[manual install guide](../manual-install#windows)).


## Step 4 — Upload the three objects to Intune

### 4a. Trusted certificate

Intune console → **Devices → Configuration → Create policy → Windows 10+ → Templates → Trusted certificate**:

- **Name:** `Waxell Tenant Root CA`
- **Certificate file:** `waxell-tenant-root.cer`
- **Destination store:** `Computer certificate store - Root`

### 4b. Managed config (`HKLM\SOFTWARE\Waxell`)

Two equivalent paths — pick one.

**Option A — Platform script (simpler).** Intune → **Devices → Scripts and remediations → Platform scripts → Add → Windows 10+**:

- **Script file:** `waxell-setup.ps1` (Step 1 output)
- **Run this script using the logged on credentials:** No
- **Enforce script signature check:** No (or Yes if you signed the script first)
- **Run script in 64 bit PowerShell host:** Yes

**Option B — Settings Catalog (auditable per-key).** Intune → **Devices → Configuration → Create policy → Windows 10+ → Settings catalog**. Use a Custom OMA-URI profile if you prefer raw paths. Push these registry values to `HKLM\SOFTWARE\Waxell`:

| Value | Type | Example |
|---|---|---|
| `ApiUrl` | String | `https://api.waxell.dev` |
| `ApiKey` | String (mark as **password**) | `wax_sk_...` |
| `CaptureEnabled` | DWord | `0` (off) or `1` (on) |
| `InterceptHosts` | MultiString | `api.openai.com;api.anthropic.com` |
| `InterceptProcesses` | MultiString | `Cursor.exe;claude.exe;Copilot.exe` |

Either option is equivalent — Option A is exactly what the generated `waxell-setup.ps1` does.

### 4c. MSI (Win32 app)

Intune → **Apps → Windows → Add → Win32 app**:

- **App package file:** `Waxell-0.7.0.intunewin`
- **Install command:** `msiexec /i "Waxell-0.7.0.msi" /qn /norestart`
- **Uninstall command:** `msiexec /x {9F3A2B81-7C4E-4E1A-9D0B-5E6F1A2B3C4D} /qn /norestart` (the UpgradeCode from `Waxell.wxs`)
- **Install behavior:** System
- **Device restart behavior:** No specific action
- **Detection rule:** MSI → auto-detected from package
- **Requirements:** Windows 10 1809+ (or your floor)

## Step 5 — Target a device group and push

Add **all three** objects (cert profile, managed-config script/policy, MSI app) to the same Intune **device group**, then push.

**Order matters**
Stage the **cert profile and managed-config before the MSI** so the services have everything they need on first boot. Intune usually applies configuration profiles within minutes; Win32 apps land over the next sync (up to roughly an hour).


## Step 6 — Verify

Within an hour of the device's next Intune sync, each PC will:

1. Trust the tenant root CA (cert profile applied).
2. Have `HKLM\SOFTWARE\Waxell` populated (managed-config applied).
3. Have the MSI installed, services started, and the device enrolled against the tenant CA (CSR → per-device intermediate); `%PROGRAMDATA%\Waxell\device-identity.json` written.
4. Have WaxellPolicyDaemon long-polling and writing `%PROGRAMDATA%\Waxell\policy.json` + `guard-effective.json`.
5. Have scanned itself and reported its AI apps.

Watch it land in **Governance → Connect → AI Endpoints**:

- **AI Apps** — the device's discovered apps (Cursor, Claude Code, Copilot, ChatGPT desktop, …).
- **Devices** — map each app's process to a governed agent.
- **Captures** — once you enable capture for a host on **Guard**, the redacted requests appear.

Run the bundled diagnostic on a problem PC to see what's missing:

```powershell
"C:\Program Files\Waxell\Setup\waxell-diagnose.ps1"
# -> %TEMP%\waxell-diag-<datestamp>.zip
```

## Memory Integrity (HVCI) considerations

Windows 11 with **Memory Integrity** (a.k.a. HVCI — Hypervisor-protected Code Integrity) enabled blocks WinDivert v2.2.2's signed kernel driver — `WinDivertOpen` returns Win32 `1275` `ERROR_DRIVER_BLOCKED`. This affects **L0 payload capture** and **HTTP/3 (QUIC) SNI extraction** — the two features that depend on the kernel driver.

**Everything else continues to work normally:** metadata flows, app discovery, hook governance (Cursor / Copilot / Claude Code), guard cascade policy, cowork audit, and the in-app Captures and Activity tabs.

Whether your fleet is affected:

- **New Windows 11 installs** (22H2+) on supported hardware default to HVCI **on**.
- **Win10 → Win11 upgrades** usually have HVCI **off**.
- **Enterprise Win11 fleets** often have HVCI **on** via Intune compliance policy.

For each device, the bundled diagnostic at `"C:\Program Files\Waxell\Setup\waxell-diagnose.ps1"` prints HVCI status in section 10b. The Captures tab also surfaces an explicit **"Memory Integrity is on — L0 unavailable"** banner so users aren't confused by an empty view.

Three options for HVCI-on endpoints:

1. **Run metadata-only on those endpoints (default).** The service silently downgrades — no L0, no QUIC SNI, but every other capability works. This is the right ship default. The Captures tab tells the user why.
2. **Disable Memory Integrity per-device** if the user / org accepts the security trade-off. *Settings → Privacy & security → Windows Security → Device security → Core isolation → Memory integrity → Off → reboot.* Push the same via Intune Settings Catalog: *Device Restrictions → Memory integrity*.
3. **Wait for the attestation-signed driver** (target: Q4). Once the Microsoft-attested `.sys` ships via the same MSI, HVCI accepts it and L0 lights up on those endpoints without changing the user's security posture. No re-deployment needed — the service rebinds on next restart.

## Turning capture on

Capture is off until you say so. There are two ways to enable it.

- **Re-push the managed-config** with `CaptureEnabled=1` and a populated `InterceptHosts`. It takes effect on the device's next Intune sync (~minutes) plus the next CaptureWorker poll (about 30s after that).
- **Intended path — the Guard cascade.** Set it from the Guard policy cascade (global → app type → user group → device → agent) and let it flow to the device via the WaxellPolicyDaemon long-poll (~5 min). See [Guard](../guard/enable-capture) for how to enable capture for a host.

**On-device safety**
TLS is terminated **only** for catalog AI hosts you've explicitly enabled — never banking, health, or mail. Secrets and PII are DLP-redacted on-device before anything is uploaded.


## Removing a hand-installed dev copy first (your own machine)

If you've been testing locally, remove the dev copy before enrolling so the MDM-managed install is clean.

```powershell
# Stop services
Stop-Service -Name WaxellNetworkMonitor, WaxellCoworkWatcher, WaxellPolicyDaemon -Force

# Uninstall MSI by UpgradeCode (no version dependence)
$msi = Get-WmiObject -Class Win32_Product | Where-Object { $_.Name -like 'Waxell*' }
$msi | ForEach-Object { msiexec /x $_.IdentifyingNumber /qn /norestart; Wait-Process msiexec }

# Clean the persistent CNG keys + Waxell certs (otherwise enrollment re-uses them)
& "C:\Program Files\Waxell\Setup\WaxellSetup.exe" --reset 2>$null
Get-ChildItem 'Cert:\CurrentUser\My', 'Cert:\CurrentUser\Root' |
    Where-Object { $_.FriendlyName -like 'Waxell *' } |
    Remove-Item -Force

# Drop ProgramData state
Remove-Item -Recurse -Force "$env:PROGRAMDATA\Waxell"
```

Then enroll the PC in Intune (*Settings → Accounts → Access work or school → Connect*), push the three objects, and let MDM take over.

## What's inside the generated `waxell-setup.ps1` (reference)

The endpoint emits a PowerShell script that:

- Imports the tenant root CA into `LocalMachine\Root` (silent — `certutil -addstore`).
- Writes `HKLM\SOFTWARE\Waxell` values for ApiUrl, ApiKey, CaptureEnabled, InterceptHosts, InterceptProcesses.
- ACLs the registry key to SYSTEM + Administrators (denies ordinary users so they can't read the API key).

It does **not** install the MSI — that's Step 4c. This is equivalent to the macOS managed-config payload inside the `.mobileconfig`. Same artifacts, different MDM verbs — see [macOS](./macos).


--------------------------------------------------------------------------------
# Install on One Machine (No MDM)
URL: https://waxell.ai/docs/endpoints/manual-install
Description: Step-by-step guide to installing the Waxell desktop agent on a single Mac or Windows machine yourself — no MDM, no IT. Download, sign in, approve the network monitor, and start seeing your AI apps.
--------------------------------------------------------------------------------

# Install on One Machine (No MDM)

This guide installs the Waxell desktop agent on **one machine, by yourself** — no
MDM, no admin/IT involvement. It's the fastest way to try Waxell on your own laptop
before rolling it out to a fleet.

Once installed, the agent discovers which apps on your machine talk to AI services
(Claude Desktop, Cursor, Claude Code, Copilot, browsers…) and reports that activity
to your workspace under **Governance → Connect → AI Endpoints**.

**Deploying to a whole fleet instead?**
Don't install machine-by-machine — use the one-profile [MDM flow](./mdm/overview).
It pushes silently to every managed device. This page is only for single-machine,
hands-on installs.


**Capture is OFF by default**
Installing the app turns on **discovery and metadata** only — which apps are
talking to which AI services. Nothing is intercepted or decrypted. Payload capture
stays **off** until an admin enables it for a specific host on the
[Guard cascade](./guard/overview).


---

## macOS

**Requirements:** macOS 12 (Monterey) or later. The app is **signed and
Apple-notarized**, so it opens without any "unidentified developer" warning.

### 1. Download

<a href="https://install.waxell.dev/latest/WaxellSetup-arm64.dmg">
  <strong>⬇ Download for Mac (Apple Silicon)</strong>
</a>

This is the Apple Silicon (M1/M2/M3/M4) build. Not sure which Mac you have? Click
 → **About This Mac** — if the Chip line says "Apple", this is the right one.

**Intel Macs**
A direct download for older Intel Macs is coming soon. In the meantime, ask your
Waxell contact for the Intel build.


### 2. Install

1. Open the downloaded **`WaxellSetup-arm64.dmg`**.
2. **Drag `Waxell Setup` onto the `Applications` folder.**
3. Open **Applications** and **double-click `Waxell Setup`** to launch it.

**Always run it from Applications**
The network monitor only loads when the app runs from `/Applications`. If you launch
it from the disk image or Downloads, macOS will block the monitor. Drag it to
Applications first.


### 3. Sign in

1. Click **Sign in with browser**.
2. Your default browser opens to the Waxell sign-in page. Log in.
3. Click **Allow this Mac**.
4. The app flips to **"Signed in as &lt;your-email&gt;"**. No API key to paste.

### 4. Approve the network monitor

1. macOS shows that a system extension was blocked, or the app says *"Approve in
   System Settings."*
2. Open **System Settings → General → Login Items & Extensions**.
3. Under **Network Extensions**, find **Waxell** and turn it **on**.
4. If macOS shows a **"Waxell would like to filter network content"** dialog, click
   **Allow**. Enter your Mac password if prompted.

### 5. Verify

- In **Waxell Setup**, the **Network Monitor** tab shows the service **running**
  (green).
- In your workspace, **Governance → Connect → AI Endpoints → AI Apps** lists your
  machine and its AI apps within a minute or two.

✅ Done. Use your AI tools normally; discovered activity flows into your workspace.

### Uninstall (macOS)

1. Quit **Waxell Setup** and drag `/Applications/Waxell Setup.app` to the Trash.
2. **System Settings → General → Login Items & Extensions → Network Extensions →
   Waxell → off.**

---

## Windows

**Requirements:** Windows 10 (1809+) or Windows 11. The installer is **code-signed**,
so it installs without an "unknown publisher" prompt.

### 1. Download

<a href="https://install.waxell.dev/latest/WaxellSetup-x64.msi">
  <strong>⬇ Download for Windows (x64)</strong>
</a>

### 2. Install

1. **Double-click** the downloaded **`WaxellSetup-x64.msi`**.
2. Approve the **UAC prompt**.
3. Wait ~30 seconds — the installer sets up the app and confirms the
   **WaxellNetworkMonitor** service is running.

### 3. Sign in

1. Open **Waxell Setup** from the **Start Menu**.
2. Click **Sign in with browser** → log in → **Allow this PC**.
3. The app flips to **"Signed in as &lt;your-email&gt;"**.

### 4. Verify

- The **Network Monitor** tab shows the service running.
- **Governance → Connect → AI Endpoints → AI Apps** lists your PC within a minute
  or two.

✅ Done.

### Uninstall (Windows)

**Settings → Apps → Installed apps → Waxell → Uninstall** (it stops the service and
removes the binaries). To clear sign-in first: click **Sign out** in the app, or
remove `dev.waxell.setup:connect` from **Control Panel → Credential Manager**.

---

## Hitting a snag?

See [Troubleshooting & Diagnostics](./reference/troubleshooting) for fixes to the
common issues (system-extension approval, SmartScreen, Defender, browser sign-in),
and [What gets installed](./reference/what-gets-installed) for the full component
list.


--------------------------------------------------------------------------------
# The Guard Cascade
URL: https://waxell.ai/docs/endpoints/guard/overview
Description: Guard is the layered policy that controls what the Waxell agent observes, blocks, or captures on each device. Understand the Global → App type → User group → Device → Agent cascade, how layers override field by field, and the resolve simulator.
--------------------------------------------------------------------------------

# The Guard Cascade

**Guard** is the policy that decides what the agent does with a device's AI
traffic — observe it, block it, or capture it. You manage it under
**Governance → Connect → AI Endpoints → Guard**.

The key idea: there is **one guard config**, expressed as a set of **layers**.
Layers are resolved in precedence order, and more-specific layers override
less-specific ones **field by field**. That lets you set a sane org-wide default
once, then refine it for a class of app, a team, a device, or a single agent —
without restating the whole policy each time.

## The cascade

```
Global  →  App type  →  User group  →  Device  →  Agent group  →  Agent
(least specific)                                          (most specific)
```

Resolution runs left to right. For any given field, the value from the
**right-most layer that sets it** wins; unset fields inherit from the layer to the
left. So a field can come from Global while the one next to it comes from a
single-device override — they compose.

| Layer | Scope it applies to |
|---|---|
| **Global** | The whole tenant — your baseline posture |
| **App type** | A class of AI app (e.g. CLI coding agents, desktop chat) |
| **User group** | A group of users |
| **Device** | One enrolled machine |
| **Agent group** | A group of governed agents |
| **Agent** | A single governed agent |

**Everything starts at "off"**
With no overrides, the cascade falls back to built-in defaults — **capture off**,
observe-only. You build policy *up* from there by adding layers. Removing all
layers returns you to those defaults.


## Layers are partial config

A layer is a **partial config** — you specify only the fields you want to change at
that scope. In the Guard UI each layer is edited as a small JSON object; in the
"All overrides" list you'll see each layer tagged with its scope and how many fields
it sets. Fields you don't set simply inherit from the layer above. This is why a
device override can be two lines and still do exactly what you mean.

Common things a layer controls:

- whether **capture** is enabled, and for which **hosts**
- which **processes/apps** to intercept or block
- the **posture** (observe vs. block) for an app type

(For the exact steps to turn capture on, see
[Enabling capture](./enable-capture).)

## App-type guards & governance level

The Guard tab shows each discovered **app type** with its **governance level**
(**full**, **transcript**, or **muted**) and whether it **pins TLS**. These tell
you the *ceiling* — the most an app type can be governed/captured given how it
works. Your layer chooses how much of that ceiling to use. An app that pins TLS,
for example, can't be payload-captured even if you ask — the UI makes that ceiling
visible so policy and reality don't drift apart.

## Resolve a context (the simulator)

Before you trust a policy, prove it. The Guard tab's **Resolve a context** panel
lets you pick an **app type**, **user group**, **device**, and **agent**, then
shows the **effective** config *and a badge on every field naming the layer it came
from* (Global, App type, Device, …).

Use it to answer questions like:

- "If a user in *engineering* runs *Cursor* on *this laptop*, is capture on?"
- "Which layer is actually setting `capture_enabled` here — and is that what I
  intended?"

This removes the guesswork that layered policies usually create: you never have to
mentally merge the layers — the resolver does it and shows its work.

## How policy reaches the device

When you save a layer, the change propagates to devices through the on-device
**policy daemon**, which long-polls the control plane and writes the effective
policy locally. Devices pick up changes within minutes; the network monitor then
acts on the new policy on its next sweep.

## Next

- [Enabling capture](./enable-capture) — turn capture on for a host, step by step.
- [Privacy & DLP](./privacy-and-dlp) — what capture does and doesn't expose.


--------------------------------------------------------------------------------
# Enabling Capture (Creating a Guard)
URL: https://waxell.ai/docs/endpoints/guard/enable-capture
Description: Step-by-step — how to turn payload capture on for a specific AI host using the Waxell Guard cascade, scope it to an app type, group, or device, and verify it with the resolve simulator. Capture stays off until you do this.
--------------------------------------------------------------------------------

# Enabling Capture (Creating a Guard)

By default the agent observes **metadata only** — capture is off everywhere. This
page shows how to deliberately turn capture **on** for a specific AI host by adding
a layer to the [Guard cascade](./overview), and how to verify it before you rely on
it.

**This is the one switch that terminates TLS**
Enabling capture means the agent will terminate TLS for the host(s) you list,
read the request/response, redact secrets/PII **on the device**, and upload the
redacted result. It applies **only to catalog AI hosts** — never banking, health,
or mail. Make sure you understand [Privacy & DLP](./privacy-and-dlp) before turning
it on for a fleet.


## Decide the scope first

Because the cascade resolves **Global → App type → User group → Device → Agent
group → Agent**, *where* you add the layer decides *who* it affects:

| You want to… | Add the layer at… |
|---|---|
| Pilot capture on one machine | **Device** |
| Capture a class of app everywhere (e.g. all CLI coding agents) | **App type** |
| Capture for one team | **User group** |
| Capture for one governed agent | **Agent** |
| Set the org-wide baseline | **Global** |

Start narrow (a single **Device**) for a pilot, confirm it behaves, then widen.

## Step 1 — Open the Guard tab

Go to **Governance → Connect → AI Endpoints → Guard**. You'll see the cascade strip
(with a count of overrides per layer), the **Resolve a context** simulator, the
**App-type guards** grid, and the **All overrides** list.

## Step 2 — Add the layer

- To scope by **app type**: in **App-type guards**, find the app type and click
  **add override** (or **edit** if one exists).
- To scope by **global / device / user group / agent**: use **edit global** in the
  *All overrides* panel, or edit an existing layer of that scope.

You'll get a small **partial-config JSON** editor. Set only the fields you need —
everything else inherits from the layer above. To turn capture on for specific
hosts, set the capture flag and the host list, for example:

```json
{
  "capture_enabled": true,
  "intercept_hosts": ["api.openai.com", "api.anthropic.com"]
}
```

**Only set what you mean to change**
A layer is a *partial* override. The two lines above flip capture on for those two
hosts at this scope and leave every other field inheriting from the cascade. You
don't restate the whole policy.


Click **Save**. The new layer appears in **All overrides** tagged with its scope
and field count.

## Step 3 — Verify with Resolve

Don't assume — **resolve it**. In **Resolve a context**, pick the app type / user
group / device / agent you care about and click **Resolve**. The effective config
appears with a colored badge on each field showing **which layer it came from**.

Confirm:

- `capture_enabled` is `true` (and the badge points at the layer you just added).
- `intercept_hosts` lists exactly the hosts you intended.
- Nothing unexpected is being set by a layer you forgot about.

## Step 4 — Confirm on the device & in Captures

1. Wait a few minutes — the on-device **policy daemon** long-polls and applies the
   new policy.
2. Use the targeted AI app/host on that device.
3. Open the **Captures** tab. The redacted, attributed requests for the enabled
   host appear. Click a row to see the redacted payload.

## Blocking instead of capturing

Capture is for *seeing* traffic; blocking is for *stopping* it. The same layer
mechanism drives process/app **blocking** (a preventive posture) via the network
monitor — set the relevant block fields on a layer at the scope you want. On
Windows this is enforced through WFP; note that blocking there is currently
process-scoped (see [Key concepts](../concepts) for detective vs. preventive).

## Turning capture back off

Edit the same layer and set `capture_enabled` to `false`, or delete the layer
entirely (the trash icon in **All overrides**) to fall back to the layer above —
ultimately to the capture-off default. Resolve again to confirm the change took.

## Next

- [Privacy & DLP](./privacy-and-dlp) — exactly what capture exposes and protects.
- [Guard cascade overview](./overview) — the precedence model in full.


--------------------------------------------------------------------------------
# Privacy & On-Device DLP
URL: https://waxell.ai/docs/endpoints/guard/privacy-and-dlp
Description: What payload capture does and doesn't expose — capture is off by default, limited to catalog AI hosts, never banking/health/mail, and DLP-redacted on the device so raw payloads never leave the machine.
--------------------------------------------------------------------------------

# Privacy & On-Device DLP

Payload capture is the most sensitive capability in Waxell Endpoints, so it's built
to be conservative by design. This page states exactly what it does and does not
do, so you can deploy it with confidence and explain it to your security, privacy,
and legal teams.

## The four guarantees

**NOTE**
1. **Capture is off by default.** Installing the agent enables discovery and
   metadata only. Nothing is intercepted or decrypted until an admin explicitly
   enables a host on the [Guard cascade](./enable-capture).
2. **Only catalog AI hosts can be captured.** TLS is terminated **only** for hosts
   on Waxell's curated AI-provider catalog — **never** banking, health, mail, or
   any non-AI destination, even if someone tries to add one.
3. **Redaction happens on the device, before upload.** Secrets and PII are stripped
   **locally**. The raw payload never leaves the machine — the control plane only
   ever receives the redacted form.
4. **It's attributable and revocable.** Every capture is tied to a device, process,
   and governed agent; capture can be turned off for any scope at any time by
   editing or deleting the Guard layer.


## What "on-device DLP" means

When capture is enabled for a host, the flow is:

```
AI request ──► proxy terminates TLS ──► DLP redaction (on device) ──► upload redacted only
                                              │
                                     raw payload stays local,
                                        is never uploaded
```

The redaction step runs inside the agent on the endpoint. By the time anything
crosses the network to Waxell, secrets and PII have already been removed. This is
the difference between a capture system that *moves your data to be scrubbed* and
one that *scrubs before it moves* — Waxell is the latter.

## Why TLS termination here is safe

Capture relies on the [tenant CA](../how-it-works#enrollment-and-the-tenant-ca):
the device already trusts your tenant's private root, so the proxy can present a
leaf cert for an *AI host* that chains to it and read the stream — without a
certificate warning and without weakening trust for anything else. Because the host
list is restricted to the AI catalog, that capability physically cannot be pointed
at your bank or your health portal.

## Metadata vs. capture — the privacy trade

Most deployments run **metadata-only** indefinitely and that's a deliberate,
defensible posture: you get the inventory, attribution, and activity record with
**no payload ever touched**. Turn on capture only where the added evidence is worth
the (already heavily mitigated) sensitivity — e.g. a specific AI host you need to
audit for data exfiltration.

| | Metadata-only (default) | Capture (opt-in, per host) |
|---|---|---|
| Payload touched | Never | Terminated + DLP-redacted on device |
| Hosts | All AI apps | Only enabled catalog AI hosts |
| Leaves device raw | No data | Never (redacted form only) |
| Good for | Inventory, attribution, baseline | Targeted audit of a specific host |

## Operational notes

- **Scope tightly.** Use the cascade to enable capture at the narrowest scope that
  meets your need — a device or app type, not Global — and widen only deliberately.
  See [Enabling capture](./enable-capture).
- **Verify with Resolve.** The Guard resolver shows exactly which scope turned
  capture on, so there are no surprises.
- **Tell your users.** Capture changes the privacy posture of a machine; pair it
  with whatever notice your policies require.

## Next

- [Enabling capture](./enable-capture) — the step-by-step to turn it on safely.
- [How it works](../how-it-works) — where DLP sits in the architecture.


--------------------------------------------------------------------------------
# What Gets Installed
URL: https://waxell.ai/docs/endpoints/reference/what-gets-installed
Description: Reference — every component the Waxell desktop endpoint agent installs on macOS and Windows, what each one does, and where its files and logs live.
--------------------------------------------------------------------------------

# What Gets Installed

This page enumerates every component the Waxell desktop endpoint agent lays down on macOS and Windows, what each one does, and where its files, state, and logs live. Use it to audit a managed deployment, scope a security review, or clean up a host.

**Capture is OFF by default**
Nothing on the endpoint terminates TLS or captures request payloads until an admin explicitly enables capture for a specific host. Out of the box the agent observes **metadata flows only** (process → destination host, timing, byte counts). See [Turning capture on](../mdm/windows#turning-capture-on) for the enable path.


---

## macOS

The macOS agent ships as a single signed, notarized application bundle — **Waxell Setup.app** — distributed as a `.dmg`. The app embeds two network system extensions and a set of helper binaries; the helpers run as per-user LaunchAgents installed by the Setup UI at first run. All binaries are signed under Apple Developer Team ID **2RWC9968XY** with the hardened runtime.

### Components

| Component | Bundle / binary | Type | What it does |
|---|---|---|---|
| **Waxell Setup** | `dev.waxell.setup` | App (GUI) | The AppKit setup UI. Enrolls the device against the tenant CA, installs the system extensions and helper LaunchAgents, and surfaces status. Installs to `/Applications`. |
| **FlowMonitor extension** | `dev.waxell.setup.flowmonitor` | Network system extension (content filter) | Metadata flow observation. Sees every connection's process, destination host (SNI / QUIC), and timing — without terminating TLS. This is the always-on, default capability. |
| **ProxyMonitor extension** | `dev.waxell.setup.proxymonitor` | Network system extension (transparent proxy) | TLS-termination payload capture (L0). For admin-enabled AI hosts only, terminates TLS with a tenant-CA-chained leaf, DLP-redacts on-device, and uploads. **Off until a host is enabled.** |
| **WaxellPolicyDaemon** | `dev.waxell.setup.policydaemon` | Helper + LaunchAgent | Long-polls the control plane and applies guard / capture policy to the local extensions. |
| **WaxellFlowUploader** | `dev.waxell.setup.flowuploader` | Helper + LaunchAgent | Batches observed metadata flows and uploads them to the control plane. |
| **WaxellCoworkWatcher** | `dev.waxell.setup.coworkwatcher` | Helper + LaunchAgent | Watches Claude Cowork activity and emits the cowork audit event stream. |
| **wax-hook** | `dev.waxell.setup.hook` | Helper (no LaunchAgent) | Synchronous Claude Code hook, invoked per tool call. No daemon — the Setup UI wires it into `~/.claude/settings.json`; the binary sits in the bundle awaiting invocation. |

**NOTE**
The Setup app is ~3 MB. The Python `wax` CLI is no longer bundled — all endpoint operations run as native Swift Mach-O helpers. Power users who want the dev CLI install it separately via `pipx install waxell-observe`.


### File locations (macOS)

- **App bundle:** `/Applications/Waxell Setup.app`
- **Embedded helpers:** `/Applications/Waxell Setup.app/Contents/Helpers/` — `WaxellPolicyDaemon`, `WaxellFlowUploader`, `WaxellCoworkWatcher`, `wax-hook`
- **LaunchAgent templates:** `/Applications/Waxell Setup.app/Contents/Resources/` — `policy-daemon.plist.template`, `flow-uploader.plist.template`, `cowork-watcher.plist.template` (substituted and installed into `~/Library/LaunchAgents/` at first run)
- **System extensions:** activated from inside the app bundle and registered with the OS; managed via `systemextensionsctl`
- **Claude Code hook wiring:** `~/.claude/settings.json`
- **Managed config (MDM):** delivered as a `.mobileconfig` profile (see `installers/macos/mdm/`)

---

## Windows

The Windows agent ships as a signed MSI (`Waxell-<ver>.msi`). The MSI installs the Setup GUI, several auto-start LocalSystem services, the synchronous hook binary, and — for L0 payload capture — the WinDivert kernel driver. Managed configuration is delivered out-of-band via `HKLM\SOFTWARE\Waxell` (typically pushed by Intune / MDM).

### Components

| Component | Service / binary | Type | What it does |
|---|---|---|---|
| **WaxellSetup** | `WaxellSetup.exe` | App (GUI) | The Avalonia setup UI: service status, bridge status, enrollment. Also exposes `--reset` to clear persistent CNG keys + Waxell certs. |
| **WaxellNetworkMonitor** | `WaxellNetworkMonitor` | LocalSystem service (auto-start) | ETW flow observation (DNS + Schannel + Kernel-Network), PID/time correlation, metadata flow upload, and user-mode WFP blocking of disallowed processes. |
| **WaxellCoworkWatcher** | `WaxellCoworkWatcher` | LocalSystem service (auto-start) | Claude Cowork audit — emits the cowork event stream (parity with the macOS helper). |
| **WaxellPolicyDaemon** | `WaxellPolicyDaemon` | LocalSystem service (auto-start) | Long-polls the control plane and writes effective guard / capture policy to disk for the other services to read. |
| **wax-hook** | `wax-hook.exe` | Binary (no service) | Synchronous Claude Code hook, invoked per tool call. |
| **WinDivert** | WinDivert `.sys` driver | Kernel driver | Packet interception for L0 payload capture and HTTP/3 SNI extraction. Loaded only when capture is enabled. |

All three services boot at device startup and pick up the managed config (`HKLM\SOFTWARE\Waxell`) the moment it's present.

### File locations (Windows)

- **Program files:** `%PROGRAMFILES%\Waxell\` — `Setup\` (WaxellSetup.exe, `waxell-diagnose.ps1`) and `Service\` (the services + WinDivert driver)
- **State + logs:** `%PROGRAMDATA%\Waxell\` (ACL'd `BUILTIN\Users: Modify` so the policy bridge can write without elevation)
  - `policy.json` — the local policy the services enforce (block lists, etc.)
  - `guard-effective.json` — the resolved effective guard policy from the cascade
  - `service.log` — teed service logs
  - `device-identity.json` — the per-device enrollment identity (CSR → tenant intermediate)
- **Managed config:** `HKLM\SOFTWARE\Waxell` — `ApiUrl`, `ApiKey`, `CaptureEnabled`, `InterceptHosts`, `InterceptProcesses` (ACL'd to SYSTEM + Administrators so ordinary users can't read the API key)
- **Tenant root CA:** imported into `LocalMachine\Root`
- **Diagnostic bundle:** `"C:\Program Files\Waxell\Setup\waxell-diagnose.ps1"` → `%TEMP%\waxell-diag-<datestamp>.zip`

**L0 payload capture on Windows is still being built**
Windows L0 (TLS-termination payload capture) is in progress. It depends on the WinDivert kernel driver, which is **blocked by HVCI / Memory Integrity** on Windows 11 — `WinDivertOpen` returns `ERROR_DRIVER_BLOCKED` (1275). On those hosts the service silently **downgrades to metadata-only**: every other capability (metadata flows, app discovery, hook governance, guard cascade, cowork audit) keeps working; L0 and HTTP/3 SNI extraction are unavailable until an attestation-signed driver ships. See the [Memory Integrity (HVCI) section](../mdm/windows#memory-integrity-hvci-considerations).


---

## See also

- [Deploy to a Windows fleet via Intune](../mdm/windows) — the three MDM artifacts, managed config, and HVCI handling.
- [Turning capture on](../mdm/windows#turning-capture-on) — the default-OFF capture model and how to enable per-host.


--------------------------------------------------------------------------------
# Troubleshooting & Diagnostics
URL: https://waxell.ai/docs/endpoints/reference/troubleshooting
Description: Fixes for common Waxell desktop endpoint agent issues on macOS and Windows — system-extension approval, SmartScreen, Defender, browser sign-in, Memory Integrity (HVCI), and where to find logs.
--------------------------------------------------------------------------------

# Troubleshooting & Diagnostics

Fixes for the most common issues with the Waxell desktop endpoint agent. Find your platform below, then scan for the **symptom** that matches what you're seeing.

**Signing status**
The **macOS** app is notarized by Apple — Gatekeeper warnings shouldn't appear at all. The **Windows** MSI is **code-signed**, so current builds install without SmartScreen or "unknown publisher" warnings. The SmartScreen/Defender fixes below are kept for anyone still running an old pre-signing preview build — upgrading to the current MSI from `install.waxell.dev` is the real fix.


---

## macOS

### Gatekeeper / notarization warning

**Symptom — macOS says the app "can't be opened because it is from an unidentified developer," or won't launch.**

This shouldn't happen — the macOS build is notarized. If you do hit it (for example after copying the app between machines and stripping its quarantine attributes), **right-click the app → Open**, then confirm **Open** in the dialog. This only needs doing once.

### Network monitor stuck on "Approve in System Settings"

**Symptom — the app shows "Approve in System Settings" and the network monitor never activates.**

The system extension needs a one-time approval:

1. Open **System Settings → General → Login Items & Extensions**.
2. Find **Waxell** and enable it.
3. Click **Allow** on the content-filter dialog when macOS prompts you.
4. Make sure the app is installed in **`/Applications`** — the system extension won't load from a non-standard location (for example running it straight out of `~/Downloads`).

**TIP**
If the app isn't already in `/Applications`, drag it there, then relaunch and re-approve. macOS only loads the content-filter network extension for apps installed in the standard location.


### Sign-in browser never returns to the app

**Symptom — "Sign in with browser" opens your browser, you log in, but the app never flips to "Signed in."**

The custom `waxell://` URL handler that hands the browser session back to the app may not be registered. **Quit and reopen Waxell Setup** — it re-registers the handler on every launch — then try **Sign in with browser** again.

---

## Windows

**Preview build is unsigned**
The Windows installer is unsigned until our EV code-signing certificate finishes verifying. Use the bundled **`team-install.ps1`** script (which installs via `msiexec`) to avoid most warnings. The signed builds that follow won't show any of these.


### SmartScreen blocks the MSI

**Symptom — "This app isn't safe to install" or a blue SmartScreen screen when you open the MSI.**

Use the bundled **`team-install.ps1`** path, which runs the MSI via `msiexec` and doesn't trigger SmartScreen. If you double-clicked the `.msi` directly, click **More info → Run anyway**. Future signed builds won't show this.

### Service not running / Defender quarantine

**Symptom — after install, the `WaxellNetworkMonitor` service isn't running.**

The network monitor watches outbound traffic, which looks malware-shaped to Microsoft Defender heuristics and can get false-positive quarantined.

1. Open **Windows Security → Virus & threat protection → Protection history**.
2. If **`WaxellNetworkMonitor.exe`** was quarantined, **restore** it.
3. Add **`C:\Program Files\Waxell\`** to **Defender exclusions** (the `team-install.ps1` script normally does this for you).
4. **Re-run `team-install.ps1`.**

To confirm the service state from a terminal:

```powershell
sc query WaxellNetworkMonitor
# STATE: 4  RUNNING  when healthy
```

### Browser sign-in never returns to the app

**Symptom — "Sign in with browser" opens your browser, you log in, but Setup never flips to "Signed in."**

The custom `waxell://` URL handler may not be registered. **Close and reopen Waxell Setup** — it re-registers the handler on every launch — then try **Sign in with browser** again.

### PowerShell execution disabled

**Symptom — "PowerShell execution disabled" (or a script-execution policy error) when running `team-install.ps1`.**

Open **PowerShell as Administrator** and run:

```powershell
Set-ExecutionPolicy -Scope CurrentUser RemoteSigned -Force
```

Then run the script again. This should be rare — the script self-elevates with `-ExecutionPolicy Bypass` — but a locked-down policy can still block the initial launch.

### Memory Integrity (HVCI) is on

**Symptom — on the Captures tab you see "Memory Integrity is on — L0 unavailable," and payload capture / QUIC SNI extraction don't work.**

Windows 11 with **Memory Integrity** (a.k.a. **HVCI** — Hypervisor-protected Code Integrity) blocks the signed kernel driver that **L0 payload capture** and **HTTP/3 (QUIC) SNI extraction** depend on. The service silently downgrades to metadata-only.

**Everything else keeps working normally** — metadata flows, app discovery, hook governance, guard policy, cowork audit, and the in-app Captures and Activity tabs. The Captures tab surfaces the banner so an empty payload view isn't confusing.

If you need L0 capture on that endpoint, an admin can disable Memory Integrity (security trade-off) or wait for the attestation-signed driver. See the fleet guidance in the [Windows MDM deployment guide](../mdm/windows) for the full HVCI options.

---

## Diagnostics

### Windows — bundled diagnostic bundle

A diagnostic collector ships inside the install. Run it on a problem PC to gather everything support needs in one zip:

```powershell
"C:\Program Files\Waxell\Setup\waxell-diagnose.ps1"
```

It writes a bundle to **`%TEMP%\waxell-diag-&lt;datestamp&gt;.zip`**. The bundle includes the service state, configuration, and (in section 10b) the device's **HVCI / Memory Integrity** status, so you can tell at a glance whether an endpoint is downgraded to metadata-only.

### Where the logs live

| Platform | Log location |
|---|---|
| **Windows** | Service log: **`%PROGRAMDATA%\Waxell\service.log`** (i.e. `C:\ProgramData\Waxell\service.log`) |
| **macOS** | **Console.app**, or the app's **Troubleshoot** tab |

On Windows you can tail the service log directly:

```powershell
Get-Content 'C:\ProgramData\Waxell\service.log' -Tail 20
```

**Reporting a bug**
When you file an issue, attach the diagnostic bundle (`%TEMP%\waxell-diag-&lt;datestamp&gt;.zip` on Windows) or the service log, plus a short description of what happened and a screenshot if something in the UI looks wrong.


================================================================================
SECTION: PLATFORM
Billing, insights, and reseller administration
================================================================================

--------------------------------------------------------------------------------
# Insights — Executive Intelligence
URL: https://waxell.ai/docs/platform/insights
--------------------------------------------------------------------------------

# Insights

The Insights section provides executive-level visibility into your organization's AI agent activity without requiring technical depth.

## Pages

### Executive Dashboard
High-level KPIs: total runs, success rate, cost trend, governance compliance score. Designed for a weekly leadership check-in.

### Cost Intelligence
Cost attribution by agent, model, and team. Budget tracking with burn-rate projection. Model cost comparison to identify savings opportunities.

### Model Analytics
Which models your agents use, how they perform (latency, token efficiency), and where you're over-provisioning.

### Compliance Console
Governance policy coverage, blocked actions, framework alignment scores (OWASP LLM, NIST AI RMF, ISO 42001).

### Model Arena
Side-by-side model comparison on your own prompts. Run the same input through multiple models and compare quality, cost, and latency.

## Access control

Insights pages respect the `insights_only` product access level. Executive users see only Insights + personal settings — no agent configuration, governance editing, or billing management.


--------------------------------------------------------------------------------
# Billing & Subscription Management
URL: https://waxell.ai/docs/platform/billing
--------------------------------------------------------------------------------

# Billing

Waxell uses Stripe for subscription billing and usage-based overage charges.

## Plans

| Plan | Included | Overage |
|---|---|---|
| Free | 1,000 LLM calls/mo | Not available |
| Pro | 50,000 LLM calls/mo + 10 agents | Per-call pricing |
| Enterprise | Unlimited | Custom |

## Subscription management

Tenants manage their subscription via the Stripe Customer Portal. Plan upgrades and downgrades sync automatically to Waxell entitlements within seconds via the `subscription.updated` webhook.

## Usage billing

Usage overage is calculated on the 1st and 15th of each month. Line items include:
- LLM token consumption above plan limits
- CPU and memory usage for isolated-tier agents
- Storage for captured traces and documents

## Cost change notifications

When model prices change (via LiteLLM sync), affected tenants receive a daily digest email listing the models, old/new rates, and estimated impact on their usage patterns.

## Payment failures

Failed payments trigger an immediate email notification to the billing admin. The subscription enters `past_due` status. After 3 consecutive failures, the subscription is automatically canceled.


--------------------------------------------------------------------------------
# Partner & Reseller Program
URL: https://waxell.ai/docs/platform/reseller
--------------------------------------------------------------------------------

# Partner & Reseller Program

Waxell's partner program lets MSPs, dev shops, consultancies, and agencies run AI governance for every client from one portal — with commission on managed revenue, optional client discounts, and your brand on the invite.

## Becoming a partner

Apply at [waxell.dev/signup/partners](https://waxell.dev/signup/partners). The application takes about two minutes and is auto-approved: your partner workspace is provisioned immediately and your sign-in link arrives by email. A kickoff call is available after you're in — it's optional.

Your agreement starts with a 20% commission on managed revenue. Commission rate, client discount, and plan ceiling are managed by Waxell and can be tailored on the kickoff call.

## Managing client workspaces

Each client gets their own Waxell tenant with full schema isolation, their own users, agents, policies, and data.

From the partner portal (**Customers → Add customer**) you can create a client workspace in minutes:

- **With a client owner email** — the client receives a branded invite ("managed by *your brand*, powered by Waxell") and owns the workspace; you keep an admin seat.
- **Without an owner email** — you own the workspace yourself. Dev shops and platform teams use this for client projects, staging environments, and internal sandboxes.

Available plans for new client workspaces are bounded by your agreement's plan ceiling.

## The roll-up dashboard

The portal dashboard shows every managed workspace in one view:

- Client workspaces and active subscriptions
- MRR by plan and per-client
- Usage (agent executions, seats, devices) and 30-day activity per client
- Commission — estimated monthly, pending, and paid to date
- CSV export for your own reporting

Access to the portal requires an owner or admin role in your partner workspace.

## Partnership modes

Every partner agreement runs in one of three modes, matched to how you work:

- **Commission** (the default) — each client workspace pays Waxell directly and you earn commission on all managed revenue. Best for referral-style partnerships.
- **Wholesale** — you pay one consolidated monthly invoice covering all your managed workspaces at net pricing (list minus your partner discount), and you bill your clients yourself. Your margin is yours; no commission machinery involved.
- **Internal** — built for teams running multiple environments (staging, production, dev) under one organization. Same consolidated invoice as wholesale, and the portal speaks your language: workspaces are *Environments*, with quick labels for staging/production/dev/sandbox.

Your mode is set with Waxell — raise it on the kickoff call or with your account contact. In wholesale and internal modes, paid workspaces are added to a single subscription on your payment method, one line item per workspace, billed on the 1st of each month.

## Client discounts

Your agreement can include a default customer discount. In commission mode, every subscription created for your managed workspaces automatically carries that percentage off — applied directly in Stripe, no promo codes needed. In wholesale and internal modes, the same percentage defines your net price on the consolidated invoice.

## What partners cannot do

- See other partners' customers
- Access direct Waxell customers
- Exceed the plan ceiling on their agreement

## Commission payouts

Monthly payout snapshots are generated automatically on the 2nd of each month with a per-client revenue breakdown and the calculated commission. Payouts move `pending → approved → paid` as Waxell ops processes them; your payout history is visible in the portal.


================================================================================
SECTION: CLI
The wax command-line interface
================================================================================

--------------------------------------------------------------------------------
# CLI Reference
URL: https://waxell.ai/docs/reference/cli
Description: The complete guide to the wax CLI — authenticate, push agents, explore runs and traces, govern, and configure Waxell from your terminal.
--------------------------------------------------------------------------------

# Waxell CLI (`wax`)

The `wax` CLI is the primary developer interface to Waxell. It ships with the
`waxell` SDK and lets you authenticate, deploy agents, explore execution
runs and traces, manage governance, and configure everything about your
tenant from the terminal.

```bash
wax login          # authenticate
wax push           # deploy agents from waxell.yaml
wax runs           # browse execution runs
wax traces         # explore distributed traces
```

Run `wax --help` for the full command tree, or `wax <command> --help` for
options on any command.

## Installation

The `wax` CLI ships with the **[`waxell`](https://pypi.org/project/waxell/)**
package on PyPI. It requires **Python 3.10–3.14**.

### pipx (recommended for CLI use)

[`pipx`](https://pipx.pypa.io/) installs `waxell` into an isolated environment
and puts `wax` on your `PATH` without polluting your global or project Python.
This is the most reliable way to get the CLI.

```bash
pipx install waxell
```

If you don't have `pipx` yet:

```bash
python3 -m pip install --user pipx
python3 -m pipx ensurepath      # adds pipx's bin dir to your PATH
# open a new terminal so the PATH change takes effect
```

### pip (when you also want the Python library)

Use `pip` if you want to `import` the SDK in the same environment — ideally
inside a virtual environment:

```bash
python3 -m venv .venv && source .venv/bin/activate
pip install waxell
```

### Framework adapters (optional)

Install extras to bring in adapters for a specific agent framework:

```bash
pipx install "waxell[langgraph]"     # or: crewai, pydantic-ai, autogen, claude-sdk
pipx install "waxell[all]"           # every framework adapter
```

### Verify and upgrade

```bash
wax --version                  # confirm the CLI is on your PATH
wax --help                     # see the full command tree

pipx upgrade waxell            # if installed with pipx
pip install --upgrade waxell   # if installed with pip
```

### Troubleshooting installation

<details>
<summary><code>wax: command not found</code> after install</summary>

The package installed but its script directory isn't on your `PATH`.

- **pipx:** run `python3 -m pipx ensurepath`, then open a **new** terminal.
- **pip (`--user`):** the script lands in your user base bin dir. Find it with
  `python3 -m site --user-base` (the binary is in the `bin` subfolder on
  macOS/Linux, `Scripts` on Windows) and add that to your `PATH`.
- As a fallback, you can always invoke it as a module: `python3 -m waxell_sdk.cli --help`.

</details>

<details>
<summary><code>error: externally-managed-environment</code> (PEP 668)</summary>

Newer system/Homebrew Python installs block global `pip install`. This is
exactly the case `pipx` is built for — use `pipx install waxell` instead, or
install into a virtual environment (`python3 -m venv .venv && source
.venv/bin/activate && pip install waxell`). Avoid `pip install --break-system-packages`.

</details>

<details>
<summary>Wrong Python version / <code>requires-python</code> error</summary>

`waxell` needs Python **3.10–3.14**. Check with `python3 --version`. If your
default Python is too old or too new, point pipx at a supported interpreter:
`pipx install --python python3.12 waxell`.

</details>

<details>
<summary>Installed, but <code>wax --version</code> shows an old version</summary>

You likely have two copies (e.g. a pip install shadowing a pipx one). Run
`which -a wax` (macOS/Linux) or `where wax` (Windows) to see every `wax` on
your `PATH`, then remove the stale one — `pip uninstall waxell` and/or
`pipx uninstall waxell` — and reinstall via a single method.

</details>

<details>
<summary>SSL / corporate proxy errors during install</summary>

Behind a proxy, set `HTTPS_PROXY` (and `HTTP_PROXY`) before installing. If your
network does TLS interception, point pip at your corporate CA bundle with
`pip install --cert /path/to/ca-bundle.pem waxell` or the `PIP_CERT` env var.

</details>

Still stuck? `wax doctor` runs a full health check (auth, config, connectivity)
and repairs common breakage once the CLI is on your `PATH`.

## Authentication

### Interactive login

```bash
wax login                      # production (default → api.waxell.dev)
wax login --profile staging    # save credentials under a named profile
wax login --local              # local dev server (localhost:8001)
wax login --url https://...    # custom API URL
```

`wax login` prompts for an API key (get one from
**[app.waxell.dev/settings/api-keys](https://app.waxell.dev/settings/api-keys)**),
validates it against the server, and writes it to `~/.waxell/config` under the
chosen profile. Profiles never overwrite each other, so you can keep prod,
staging, and local credentials side by side.

### Configuration without a browser

If you already have an API key, scaffold the config directly — no login flow:

```bash
wax config init                       # interactive prompts
wax config init --no-input            # scaffold an empty profile, fill key later
wax config set api_key wax_sk_...     # set a value individually
wax whoami                            # verify you're authenticated
```

### Profiles

A profile is a named `[section]` in `~/.waxell/config` holding an API URL and
key. Select one for any command with `--profile` / `-p`:

```bash
wax -p staging whoami
wax -p staging runs
wax config profiles                   # list all configured profiles
```

**TIP**
By convention, `[default]` is reserved for developers; installers write a
`[connect]` profile plus per-tool sections. Pushing agents should always use an
explicit profile (`wax -p acme_local push`) rather than `--local`, so the
correct `domain_url` is registered.


### Global flags

These apply to every command (place them before the subcommand):

| Flag | Description |
|------|-------------|
| `--profile` / `-p <name>` | Config profile to use |
| `--local` | Use the local dev server (`localhost:8001`) |
| `--version` | Print the `wax` version and exit |

### Environment variables

| Variable | Description |
|----------|-------------|
| `WAX_API_KEY` / `WAXELL_API_KEY` | API key (overrides config file) |
| `WAX_API_URL` / `WAXELL_API_URL` | API URL (overrides config file) |

Configuration precedence: **`--profile` flag → `~/.waxell/config` →
environment variables**.

## Command catalog

`wax --help` groups commands into the same sections shown below. Run
`wax <command> --help` for full options on any command.

### Core

| Command | Description |
|---------|-------------|
| `wax login` | Authenticate with Waxell and save your API key |
| `wax whoami` | Show the current authenticated user and tenant |
| `wax doctor` | Run every health check and repair common breakage |
| `wax feedback` | File a bug report or feature request with the Waxell team |

### Observability

| Command | Description |
|---------|-------------|
| `wax agents` | Manage and monitor deployed agents |
| `wax runs` | Browse, inspect, and tail agent execution runs |
| `wax llm-calls` | LLM call browser and per-model analytics |
| `wax traces` | Trace explorer for distributed execution tracing |
| `wax sessions` | Session browser and analytics |
| `wax observe-users` | User analytics for agent usage |
| `wax errors` | Error analysis and diagnostics |
| `wax metrics` | Metrics dashboard and analytics |
| `wax costs` | Cost analytics and budget tracking |
| `wax lineage` | Walk the agent causality graph (runs + edges) |

### Governance

| Command | Description |
|---------|-------------|
| `wax policies` | Manage governance policies (`list`, `show`, `push`) |
| `wax kill-switches` | Manage kill switches (`list`, `show`) |
| `wax budgets` | Manage governance budgets (`list`, `show`) |
| `wax governance` | Governance events, compliance, and policy packs |

### Agent development

| Command | Description |
|---------|-------------|
| `wax push` | Auto-discover and deploy agents, tools, workflows, and skills |
| `wax push-tool` / `push-workflow` / `push-domain` / `push-skill` | Deploy a single standalone artifact |
| `wax signals` | Fire signals at agents and poll for results (`fire`, `show`) |
| `wax validate <agent>` | Validate an agent's registration and execution readiness |
| `wax context <agent>` | Check context-file coverage and quality for an agent |
| `wax test <scenarios>` | Run agent eval scenarios |
| `wax dev <path>` | Watch an agent directory and re-parse on save |
| `wax export <agent>` | Export an agent spec to JSON |
| `wax prompt` | Manage tenant-registered prompts |
| `wax mcp` | Manage the tenant's outbound MCP server registry |
| `wax claude-init` | Install the Waxell Runtime skill for Claude Code |

### Configuration

| Command | Description |
|---------|-------------|
| `wax config` | Manage `~/.waxell/config` (`init`, `show`, `set`, `profiles`) |
| `wax keys` | Manage API keys (`list`, `create`, `show`, `revoke`, `rotate`) |
| `wax secrets` | Manage secrets (`list`, `scan`, `check`) |
| `wax llm` | LLM configuration (`call`, `routing`, `models`) |
| `wax provider` | LLM provider instances, catalog, and capability groups |
| `wax domains` | Manage tenant domain configurations |
| `wax sub-user-auth` | Manage the per-tenant Ed25519 signing keypair |

### Organization & identity

| Command | Description |
|---------|-------------|
| `wax teams` / `users` / `roles` | Organization management |
| `wax identities` | Identity providers and SSO |
| `wax sub-tenants` | Manage sub-tenants |
| `wax end-users` / `end-user-groups` | Waxell end-user identity records |
| `wax billing` / `wax usage` | Billing and usage analytics |

### Integrations

| Command | Description |
|---------|-------------|
| `wax claude-code` | Claude Code observability & governance integration |
| `wax cowork` | Claude Cowork observability integration |
| `wax guard` | Inspect and edit the local-guard config |
| `wax network-monitor` | Network-monitor flow uploader (macOS) |
| `wax observe mcp-proxy` | Governance proxy for a third-party MCP server |

### Adapter platform

| Command | Description |
|---------|-------------|
| `wax adapter` | Inspect adapter capability manifests; bulk-import framework artifacts |
| `wax hub` | Inspect the canonical adapter event hub |
| `wax registry` | Browse and manage canonical registry artifacts |

## Deploying agents — `wax push`

`wax push` is the workhorse. It auto-discovers a `waxell.yaml` manifest,
packages your agents (plus tools, workflows, skills, and context files), and
deploys them to your tenant.

```bash
wax push                          # auto-discovers waxell.yaml in cwd
wax push ./projects/finance       # discover waxell.yaml under a path
wax push --dry-run                # validate and show what would push, no upload
```

| Flag | Description |
|------|-------------|
| `path` (positional, optional) | Directory or file to discover and push |
| `--profile <name>` | Config profile to push against |
| `--dry-run` | Show what would be pushed without uploading |
| `--bundle <name>` | Use a pre-approved dependency bundle (e.g. `data-science`, `web-scraping`, `ai-ml`) |
| `--include-source` / `--no-source` | Include source for isolated execution (default: auto-detect from deps) |

**TIP**
Always push with an explicit profile: `wax -p acme_local push`. Using `--local`
keeps the default profile (which has no `domain_url`), so domains register on
the wrong port and 404.


After pushing, activate or manage versions:

```bash
wax agents list                   # see all agents + execution metrics
wax agents info <name>            # signals and domain endpoints for an agent
wax agents activate <name>       # draft → active
wax agents pause <name>          # pause without deactivating
wax agents promote <name> --version <v>   # set an existing version as default
wax agents rollback <name> --version <v>  # restore a prior version as a new version
wax agents builds <name>          # container build status + history
wax agents schedule <name>        # show or manage cron schedules
```

The single-artifact variants (`push-tool`, `push-workflow`, `push-domain`,
`push-skill`) deploy one artifact without a full manifest.

## Exploring runs — `wax runs`

A **run** is one end-to-end execution of an agent workflow. Each run has a
numeric Run ID, an OTel `trace_id`, and a tree of spans (agent → LLM calls →
tool calls). Runs are listed newest-first and filtered by your plan's retention
window.

```bash
wax runs                                   # last 10 runs in the last 24h
wax runs list --hours 168                  # last week
wax runs list --agent outreach --status error
wax runs show 230 --spans                  # span tree for run 230
wax runs tail 230                          # follow a run's events live until it finishes
wax runs replay 230 --tail                 # re-fire a run with its original inputs
wax runs watch --agent my-agent            # tail new runs for one agent
wax runs stats --hours 168                 # aggregate totals + top agents
```

`wax runs list` flags:

| Flag | Description |
|------|-------------|
| `--agent <name>` | Filter to runs whose agent name contains NAME (case-insensitive) |
| `--status <success\|error\|running>` | Only show runs in this state |
| `--hours <N>` | Look back N hours (default 24). Try 168 for a week |
| `--limit <N>` | Max rows (default 10; server caps at 100) |
| `--format <table\|json>` | Pretty table or pipe-friendly JSON |

**NOTE**
`runs list` returns a **Run ID** column — pass that to `runs show`. `show` also
accepts an OTel `trace_id` or a `workflow_id`, whichever is easiest to paste.
Results are clamped to your tenant's retention window server-side.


## Firing signals — `wax signals`

Signals trigger agents and let you poll or stream the resulting run:

```bash
wax signals fire <name> --payload '{"key": "value"}'
wax signals fire <name> --payload-file ./input.json
wax signals show <run_id>                  # inspect a fired run
```

## Validating before you ship — `wax validate`

```bash
wax validate <agent_name>              # check registration + execution readiness
wax validate <agent_name> --verbose    # full diagnostic detail
wax context <agent_name>               # context-file coverage and quality
wax context <agent_name> --fix         # scaffold missing context stubs
```

## Eval harness — `wax test`

`wax test` runs scenario-based evals against an agent using the
Scenario / Turn / Expect DSL.

```bash
wax test ./evals                                  # run every scenario in a dir
wax test ./evals my_scenario                      # run named scenarios only
wax test ./evals --list                           # list scenarios + tags, don't run
wax test ./evals --tag smoke --exclude-tag slow   # filter by tag
wax test ./evals --bail                           # stop on first failure
wax test ./evals -p staging                       # run against a profile (uploads results)
```

| Flag | Description |
|------|-------------|
| `--tag <t>` / `--exclude-tag <t>` | Include/skip scenarios by tag (repeatable) |
| `--include-expensive` | Include scenarios tagged `expensive` (paid-API hits); off by default |
| `--bail` | Stop on the first scenario failure |
| `--list` | List discovered scenarios + tags and exit |
| `--report <path>` | Where to write the markdown report (default `/tmp/waxell-eval/<ts>/report.md`) |
| `--profile <name>` | `~/.waxell/config` profile to run against |
| `--trace-url-template <tpl>` | Format string for trace links in the report (`{trace_id}`) |
| `--upload` / `--no-upload` | POST results to `/v1/evals/runs/` (defaults ON when `--profile` is set) |
| `--triggered-by <user\|schedule\|ci>` | Run source, used by the eval dashboard for filtering |

## Configuration commands

```bash
wax config init                       # create ~/.waxell/config + first profile
wax config init --no-input            # scaffold without prompts
wax config show                       # print the active config (key masked)
wax config set api_key wax_sk_...     # set api_url | api_key | domain_url
wax config profiles                   # list configured profiles
```

API keys and secrets:

```bash
wax keys list                         # list API keys
wax keys create --name ci-key         # mint a new key
wax keys rotate <key_id>              # rotate a key
wax keys revoke <key_id>              # revoke a key

wax secrets list                      # list tenant secrets (values masked)
wax secrets scan ./my-agents          # scan source for hardcoded secrets
wax secrets check --agent <name>      # verify an agent's required secrets exist
```

## Common workflows

### Push and watch an agent

```bash
wax -p acme push ./my-agents          # push everything under a directory
wax runs watch --agent my-agent       # tail new runs as they arrive
wax traces                            # open the trace explorer
```

### Set up Claude Code observability

```bash
wax claude-code setup                         # install observability hooks
wax claude-code setup --governance            # + PreToolUse policy enforcement
wax claude-code setup --governance --mcp      # + MCP governance tools
wax claude-code status                        # current session state
```

### Diagnose a broken setup

```bash
wax doctor                            # runs every health check; repairs common breakage
wax whoami                            # confirm auth + tenant
```

## Next steps

- **[Installation Guide](/installation)** — get started with Waxell
- **[Runtime Quickstart](/runtime/quickstart)** — build your first agent
- **[waxell.yaml Reference](/runtime/waxell-yaml-reference)** — the manifest `wax push` reads
- **[Observe Overview](/observe/overview)** — observability for any framework


================================================================================
SECTION: GUIDES
Task-oriented walkthroughs
================================================================================

--------------------------------------------------------------------------------
# Architecture
URL: https://waxell.ai/docs/guides/architecture
Description: Deep dive into Waxell's data flow and security architecture.
--------------------------------------------------------------------------------

# Architecture

This document describes Waxell's data flow architecture, including tenant isolation, security features, and enterprise capabilities.

## Overview Diagram

```
┌─────────────────────────────────────────────────────────────────────────────────┐
│                                TENANT REQUEST FLOW                               │
└─────────────────────────────────────────────────────────────────────────────────┘

                              ┌──────────────────┐
                              │   Client/SDK     │
                              │  (API Request)   │
                              └────────┬─────────┘
                                       │
                    ┌──────────────────▼──────────────────┐
                    │           AWS WAF                    │
                    │   • Rate limiting (100 req/sec)      │
                    │   • SQL injection protection         │
                    │   • XSS filtering                    │
                    └──────────────────┬──────────────────┘
                                       │
                    ┌──────────────────▼──────────────────┐
                    │      Application Load Balancer       │
                    │   • TLS termination                  │
                    │   • Request routing                  │
                    └──────────────────┬──────────────────┘
                                       │
        ┌──────────────────────────────┼──────────────────────────────┐
        │                              │                              │
        ▼                              ▼                              ▼
┌───────────────┐            ┌───────────────┐            ┌───────────────┐
│  Controlplane │            │   Runtime     │            │   Celery      │
│   (Django)    │            │   Workers     │            │   Workers     │
└───────┬───────┘            └───────┬───────┘            └───────┬───────┘
        │                            │                            │
        └────────────────────────────┼────────────────────────────┘
                                     │
                    ┌────────────────┴────────────────┐
                    │                                  │
        ┌───────────▼───────────┐        ┌───────────▼───────────┐
        │   Tenant Middleware   │        │   Region Router       │
        │   • Schema selection  │        │   • DB routing        │
        │   • X-Tenant-Slug     │        │   • Cache routing     │
        └───────────┬───────────┘        └───────────┬───────────┘
                    │                                  │
                    └────────────────┬────────────────┘
                                     │
        ┌────────────────────────────┼────────────────────────────┐
        │                            │                            │
        ▼                            ▼                            ▼
┌───────────────┐            ┌───────────────┐            ┌───────────────┐
│  PostgreSQL   │            │    Redis      │            │      S3       │
│  (per-schema) │            │   (TLS+AUTH)  │            │  (encrypted)  │
│  • RDS SSL    │            │   • Celery    │            │  • per-region │
│  • Multi-AZ   │            │   • Cache     │            │  • Object Lock│
└───────────────┘            └───────────────┘            └───────────────┘
```

## Package Architecture

Waxell enforces strict boundaries between packages:

| Package | Responsibility | Dependencies |
|---------|---------------|--------------|
| **SDK** | Intent-only definitions | stdlib, pydantic |
| **Runtime** | Execution engine | SDK |
| **Controlplane** | Views, APIs, admin | SDK, Runtime |
| **Generation** | RAG, LLM synthesis | SDK |
| **Infra** | Production backends | All packages |

```
SDK ────────► Runtime
│                ▲
│                │
└──────► Generation

Controlplane ──► SDK
Controlplane ──► Runtime (APIs/queues only)

Infra orchestrates all packages
```

## Data Residency

Data residency allows enterprise tenants to select where their data is stored:

```
                              ┌─────────────────────┐
                              │  Tenant Request     │
                              │  tenant_id: abc123  │
                              └──────────┬──────────┘
                                         │
                                         ▼
                              ┌─────────────────────┐
                              │ TenantRegionRouter  │
                              │ get_tenant_region() │
                              └──────────┬──────────┘
                                         │
                    ┌────────────────────┴────────────────────┐
                    │                                         │
        ┌───────────▼───────────┐               ┌────────────▼────────────┐
        │  US East (Default)    │               │  EU West (+$500/mo)     │
        │  ─────────────────    │               │  ────────────────────   │
        │  • RDS: us-east-1     │               │  • RDS: eu-west-1       │
        │  • Redis: us-east-1   │               │  • Redis: eu-west-1     │
        │  • S3: waxell-us      │               │  • S3: waxell-eu        │
        │                       │               │                         │
        │  Compliance:          │               │  Compliance:            │
        │  • HIPAA, SOC2        │               │  • GDPR, SOC2           │
        └───────────────────────┘               └─────────────────────────┘
```

## Field-Level Encryption

Sensitive data is encrypted using tenant-specific keys:

```
WRITE PATH:
  Model.save() → EncryptedJSONField → TenantEncryptionService
                                              │
                      ┌───────────────────────┴───────────────────────┐
                      ▼                                               ▼
               AWS KMS (Get DEK)                          AES-256-GCM Encrypt
                      │                                               │
                      └───────────────────────┬───────────────────────┘
                                              ▼
                                   Ciphertext [key_id|nonce|data]

READ PATH:
  Model.query() → EncryptedJSONField → Parse key_id → KMS Decrypt DEK
                                                              │
                                                              ▼
                                                      Plaintext JSON
```

## Security Components

| Component | Purpose | Implementation |
|-----------|---------|----------------|
| TLS Everywhere | Encryption in transit | ALB, RDS, Redis |
| Field Encryption | Encryption at rest | AES-256-GCM + KMS |
| Schema Isolation | Tenant data separation | PostgreSQL schemas |
| Hash Chain Audit | Tamper-evident logging | SHA-256 chaining |
| WAF | Request filtering | AWS WAF |

## Next Steps

- **[Enterprise Guide](/guides/enterprise)** - Configure enterprise features
- **[Backends Reference](/runtime/backends)** - Configure production backends


--------------------------------------------------------------------------------
# Enterprise Guide
URL: https://waxell.ai/docs/guides/enterprise
Description: Configure enterprise security features including data residency, sub-tenants, and compliance.
--------------------------------------------------------------------------------

# Enterprise Guide

This guide covers configuring Waxell's enterprise security features.

## Prerequisites

- Enterprise plan subscription
- Admin access to your Waxell tenant
- (Optional) AWS account for BYOK encryption

## Data Residency

Data residency allows you to specify where your data is stored for compliance with regulations like GDPR.

### Available Regions

| Region | Code | Price | Compliance |
|--------|------|-------|------------|
| US East (Default) | `us-east-1` | Included | HIPAA, SOC2, FedRAMP |
| EU West | `eu-west-1` | $500/month | GDPR, SOC2, ISO27001 |
| APAC | `ap-southeast-1` | $750/month | SOC2, ISO27001 |

### Selecting a Region

1. Navigate to **Settings > Billing > Data Residency**
2. Review the available regions and their compliance certifications
3. Click **Select** on your desired region
4. Review the pricing and confirm

### Multi-Region Disaster Recovery

For additional resilience, enable Multi-Region DR:

- Data is replicated to both regions in real-time
- Automatic failover if primary region is unavailable
- RPO: < 1 minute, RTO: < 5 minutes
- Additional $1,000/month

## Sub-Tenants (B2B2C)

Sub-tenants allow you to create isolated environments for your customers.

### Creating a Sub-Tenant

```bash
curl -X POST https://api.waxell.dev/waxell/v1/sub-tenants/ \
  -H "X-Wax-Key: wax_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Acme Corp",
    "external_id": "customer-123",
    "contact_email": "admin@acme.com",
    "quotas": {
      "max_agents": 50,
      "max_tokens_per_day": 500000
    }
  }'
```

### Sub-Tenant Hierarchy

```
┌───────────────────────┐
│    Parent Tenant      │
│    (Enterprise)       │
│  max_agents: 1000     │
└───────────┬───────────┘
            │
┌───────────┼───────────┬───────────────┐
│           │           │               │
▼           ▼           ▼
Sub-A       Sub-B       Sub-C
agents:50   agents:100  agents:25
```

### Billing Configuration

Enable pass-through billing with markup:

- Set markup percentage (e.g., 20%)
- Usage tracked per sub-tenant
- Export usage reports for invoicing

## Compliance Configuration

### HIPAA Compliance ($500/month)

Enables:
- 7-year audit log retention
- Enhanced logging
- BAA (Business Associate Agreement)
- WORM-compliant audit export

### SOC2 Compliance ($300/month)

Enables:
- 1-year audit log retention
- Compliance reports
- Enhanced access controls

### Audit Log Export

Export audit logs for compliance:

```bash
# Via API
curl https://api.waxell.dev/waxell/v1/audit/export/ \
  -H "X-Wax-Key: wax_sk_..." \
  -o audit-logs.json.gz
```

Exports are:
- Daily, at 00:00 UTC
- Gzipped JSON format
- WORM-protected (immutable)

## Sub-User Identity

Track which of your end-users trigger agent executions.

### Creating Sub-User Tokens

```bash
curl -X POST https://api.waxell.dev/waxell/v1/identity/sub-users/token/ \
  -H "X-Wax-Key: wax_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "sub_user_id": "user-123",
    "email": "user@example.com",
    "roles": ["viewer", "agent_user"],
    "expires_in": 3600
  }'
```

### Using Sub-User Tokens

Include the token when triggering signals:

```bash
curl -X POST https://api.waxell.dev/api/v1/signals/lead_received \
  -H "X-Wax-Key: wax_sk_..." \
  -d '{
    "sub_user_token": "wax_su_eyJ...",
    "lead_id": "lead-456"
  }'
```

## Available Add-Ons

| Add-On | Price | Description |
|--------|-------|-------------|
| EU Data Residency | $500/mo | Store data in EU |
| APAC Data Residency | $750/mo | Store data in APAC |
| Multi-Region DR | $1,000/mo | Disaster recovery |
| HIPAA Compliance | $500/mo | Healthcare compliance |
| SOC2 Compliance | $300/mo | SOC2 certification |
| SAML SSO | $200/mo | Enterprise SSO |
| BYOK Encryption | $200/mo | Customer-managed keys |

## Next Steps

- **[API Reference](/reference/enterprise-api)** - Full API documentation
- **[Architecture](/guides/architecture)** - Technical deep dive


--------------------------------------------------------------------------------
# Best Practices
URL: https://waxell.ai/docs/guides/best-practices
Description: Patterns for production-ready Waxell agents — scoping, checkpointing, prompting, governance, testing, and what to monitor.
--------------------------------------------------------------------------------

# Best Practices

Recommended patterns for building production-ready Waxell agents. For the full
API surface, see [`@workflow`](/sdk/workflow-spec) and
[Execution Context](/runtime/execution-context).

## Agent design

### Keep agents focused

Each agent should have a single, clear responsibility:

```python
# Good: single responsibility
@agent(name="email_classifier", description="Classifies incoming emails")
class EmailClassifier:
    ...


@agent(name="email_responder", description="Drafts email responses")
class EmailResponder:
    ...
```

An agent that "classifies, responds, and archives" is three agents. Splitting
them gives you three independently versioned, independently governed, separately
attributable units — and lets one fail without taking the others down.

Delegate between them with `ctx.spawn("email_responder", inputs={...})` rather
than merging them.

### Reuse through tools and workflows

Extract shared behaviour into tools and workflows, and reference them by name:

```yaml
agents:
  - name: support_agent
    tools: [classify_text, lookup_order]
    workflows: [escalation]
```

## Workflow design

### Prefer small steps

The tool boundary is the checkpoint boundary, so step size is directly a
restart-cost decision:

```python
@workflow("process_order")
async def process_order(self, ctx, order: dict):
    # Each ctx.tool(...) is independently checkpointed.
    validated = await ctx.tool("validate", order=order)
    enriched = await ctx.tool("enrich", data=validated)
    return await ctx.tool("process", data=enriched)
```

Work done *between* tool calls isn't checkpointed. If repeating a computation
would hurt — an expensive query, a paid API call — it belongs in a `@tool`.

### Handle errors deliberately

```python
@workflow("resilient")
async def resilient(self, ctx, payload: dict):
    try:
        return await ctx.tool("risky_operation", payload=payload)
    except ValidationError:
        return await ctx.tool("handle_validation_error", payload=payload)
    except ExternalServiceError:
        return await ctx.tool("retry_with_backoff", payload=payload)
```

Catching changes your control flow, not the audit trail — the failure is still
recorded on the span and as an `ExecutionIncident`. And a **policy block is not
an exception**: a governed stop finishes the run in `BLOCKED` with a recorded
decision, so there's nothing to catch.

## Prompting

### Be explicit, and ask for JSON

```python
@tool
async def classify_intent(self, ctx, message: str) -> dict:
    return await ctx.llm.generate(
        prompt=(
            "Classify this support message by its primary topic as exactly one "
            "of: billing, technical, general. "
            'Reply with JSON: {"category": "<one of the three>"}.\n\n'
            f"{message}"
        ),
        output_format="json",
        task="classification",
    )
```

### Validate structured output

Ask for JSON and validate it — a model can always return something malformed,
and you want that caught at the boundary:

```python
from pydantic import BaseModel


class Analysis(BaseModel):
    sentiment: str
    confidence: float
    topics: list[str]


@tool
async def analyze(self, ctx, text: str) -> dict:
    raw = await ctx.llm.generate(
        prompt=f"Analyze and return JSON with sentiment, confidence, topics:\n\n{text}",
        output_format="json",
        task="analysis",
    )
    return Analysis.model_validate(raw).model_dump()
```

### Keep expensive mistakes out of the prompt

The most valuable design rule: **if being wrong is expensive, it doesn't belong
in the prompt.** A model should decide *which* invoice this is; it should not be
what decides whether two invoices are the same record. Put judgement in the
agent and arithmetic in a domain action. See
[where to put logic](/runtime/how-it-works#where-to-put-logic).

## Governance

### Bound cost, turns, and time

Declare limits on the agent:

```yaml
agents:
  - name: email_sender
    max_budget_usd: 2.00      # hard USD cap  (default 10.00)
    max_turns: 15             # agent turns   (default 25)
    timeout_seconds: 120      # wall time
```

Set these explicitly. The defaults are generous by design, and an agent that
loops will spend to them.

These caps cover one agent. Once you're running several, move the rules into the
policy engine instead: policies are scoped across agents, workflows, tools,
models, and teams, and an agent can't drop one by editing its own manifest. See
[Adding Governance](/tutorials/governance) for `policy()` + `wax policies push`,
and [Policy Categories](/observe/features/policy-categories) for the full set.

A run stopped by a policy finishes `BLOCKED` with an `ExecutionIncident`.

### Require approval for sensitive actions

Approval is a property of the **domain action**:

```yaml
domains:
  - name: accounts
    actions:
      - name: delete_account
        requires_approval: true
```

The run pauses with `PausedReason.awaiting_approval` and holds no compute until
someone decides, then resumes from its checkpoint. Attaching it to the action
means every agent that can reach it inherits the requirement — you can't forget
it on the fifth agent. See [Domains](/runtime/domains).

### Constrain what the agent can reach

Prefer domain actions over credentials. Giving an agent `document.read` is
auditable and revocable; giving it a database URL is neither.

Tighten the boundary further with
[Data Access](/observe/governance/data-access) and
[Network](/observe/governance/network) policies, which bound what any agent can
read or reach regardless of what its manifest declares.

## Testing

Test at three levels: individual tools, whole workflows, and governed end-to-end
runs.

```python
from waxell_infra.testing.harness import AgentTestHarness


async def test_classification_workflow():
    result = await AgentTestHarness.run_agent(
        ClassifierAgent,
        "classify",
        message="I can't log in",
    )

    assert result["category"] == "technical"
```

`run_agent` is an async classmethod taking `(agent_cls, workflow, **inputs)`. It
instantiates the real agent with full governance and sandboxing, so what passes
here is close to what runs in production. To control model output, use the fakes
in `waxell_infra.testing.fakes`.

## Monitoring

### What the platform records for you

Every run automatically produces a span tree, an `LlmCallRecord` per model call,
policy decisions, incidents, and usage events — see
[what a run leaves behind](/runtime/how-it-works#6-what-the-run-leaves-behind).
There's nothing to instrument. Use standard Python `logging` for anything extra;
it's captured in the run's logs.

### What to alert on

```yaml
telemetry:
  alerts:
    - p95_latency_ms > 5000
    - error_rate > 0.05
    - cost_per_run > 0.50
  sample_rate: 1.0
```

Beyond thresholds, the signal worth watching is **runs that complete with an
empty tool loop** — a green run that did nothing. The trace shows why: a policy
block, a missing tool, or a prompt problem.

## Next steps

- **[Production Guide](/tutorials/production)** — deploy with these practices
- **[Governance Tutorial](/tutorials/governance)** — limits, approvals, and the policy engine
- **[Policy Categories](/observe/features/policy-categories)** — all 49 categories
- **[How the Runtime Works](/runtime/how-it-works)** — the full path of a run


--------------------------------------------------------------------------------
# CLI Reference
URL: https://waxell.ai/docs/reference/cli
Description: The complete guide to the wax CLI — authenticate, push agents, explore runs and traces, govern, and configure Waxell from your terminal.
--------------------------------------------------------------------------------

# Waxell CLI (`wax`)

The `wax` CLI is the primary developer interface to Waxell. It ships with the
`waxell` SDK and lets you authenticate, deploy agents, explore execution
runs and traces, manage governance, and configure everything about your
tenant from the terminal.

```bash
wax login          # authenticate
wax push           # deploy agents from waxell.yaml
wax runs           # browse execution runs
wax traces         # explore distributed traces
```

Run `wax --help` for the full command tree, or `wax <command> --help` for
options on any command.

## Installation

The `wax` CLI ships with the **[`waxell`](https://pypi.org/project/waxell/)**
package on PyPI. It requires **Python 3.10–3.14**.

### pipx (recommended for CLI use)

[`pipx`](https://pipx.pypa.io/) installs `waxell` into an isolated environment
and puts `wax` on your `PATH` without polluting your global or project Python.
This is the most reliable way to get the CLI.

```bash
pipx install waxell
```

If you don't have `pipx` yet:

```bash
python3 -m pip install --user pipx
python3 -m pipx ensurepath      # adds pipx's bin dir to your PATH
# open a new terminal so the PATH change takes effect
```

### pip (when you also want the Python library)

Use `pip` if you want to `import` the SDK in the same environment — ideally
inside a virtual environment:

```bash
python3 -m venv .venv && source .venv/bin/activate
pip install waxell
```

### Framework adapters (optional)

Install extras to bring in adapters for a specific agent framework:

```bash
pipx install "waxell[langgraph]"     # or: crewai, pydantic-ai, autogen, claude-sdk
pipx install "waxell[all]"           # every framework adapter
```

### Verify and upgrade

```bash
wax --version                  # confirm the CLI is on your PATH
wax --help                     # see the full command tree

pipx upgrade waxell            # if installed with pipx
pip install --upgrade waxell   # if installed with pip
```

### Troubleshooting installation

<details>
<summary><code>wax: command not found</code> after install</summary>

The package installed but its script directory isn't on your `PATH`.

- **pipx:** run `python3 -m pipx ensurepath`, then open a **new** terminal.
- **pip (`--user`):** the script lands in your user base bin dir. Find it with
  `python3 -m site --user-base` (the binary is in the `bin` subfolder on
  macOS/Linux, `Scripts` on Windows) and add that to your `PATH`.
- As a fallback, you can always invoke it as a module: `python3 -m waxell_sdk.cli --help`.

</details>

<details>
<summary><code>error: externally-managed-environment</code> (PEP 668)</summary>

Newer system/Homebrew Python installs block global `pip install`. This is
exactly the case `pipx` is built for — use `pipx install waxell` instead, or
install into a virtual environment (`python3 -m venv .venv && source
.venv/bin/activate && pip install waxell`). Avoid `pip install --break-system-packages`.

</details>

<details>
<summary>Wrong Python version / <code>requires-python</code> error</summary>

`waxell` needs Python **3.10–3.14**. Check with `python3 --version`. If your
default Python is too old or too new, point pipx at a supported interpreter:
`pipx install --python python3.12 waxell`.

</details>

<details>
<summary>Installed, but <code>wax --version</code> shows an old version</summary>

You likely have two copies (e.g. a pip install shadowing a pipx one). Run
`which -a wax` (macOS/Linux) or `where wax` (Windows) to see every `wax` on
your `PATH`, then remove the stale one — `pip uninstall waxell` and/or
`pipx uninstall waxell` — and reinstall via a single method.

</details>

<details>
<summary>SSL / corporate proxy errors during install</summary>

Behind a proxy, set `HTTPS_PROXY` (and `HTTP_PROXY`) before installing. If your
network does TLS interception, point pip at your corporate CA bundle with
`pip install --cert /path/to/ca-bundle.pem waxell` or the `PIP_CERT` env var.

</details>

Still stuck? `wax doctor` runs a full health check (auth, config, connectivity)
and repairs common breakage once the CLI is on your `PATH`.

## Authentication

### Interactive login

```bash
wax login                      # production (default → api.waxell.dev)
wax login --profile staging    # save credentials under a named profile
wax login --local              # local dev server (localhost:8001)
wax login --url https://...    # custom API URL
```

`wax login` prompts for an API key (get one from
**[app.waxell.dev/settings/api-keys](https://app.waxell.dev/settings/api-keys)**),
validates it against the server, and writes it to `~/.waxell/config` under the
chosen profile. Profiles never overwrite each other, so you can keep prod,
staging, and local credentials side by side.

### Configuration without a browser

If you already have an API key, scaffold the config directly — no login flow:

```bash
wax config init                       # interactive prompts
wax config init --no-input            # scaffold an empty profile, fill key later
wax config set api_key wax_sk_...     # set a value individually
wax whoami                            # verify you're authenticated
```

### Profiles

A profile is a named `[section]` in `~/.waxell/config` holding an API URL and
key. Select one for any command with `--profile` / `-p`:

```bash
wax -p staging whoami
wax -p staging runs
wax config profiles                   # list all configured profiles
```

**TIP**
By convention, `[default]` is reserved for developers; installers write a
`[connect]` profile plus per-tool sections. Pushing agents should always use an
explicit profile (`wax -p acme_local push`) rather than `--local`, so the
correct `domain_url` is registered.


### Global flags

These apply to every command (place them before the subcommand):

| Flag | Description |
|------|-------------|
| `--profile` / `-p <name>` | Config profile to use |
| `--local` | Use the local dev server (`localhost:8001`) |
| `--version` | Print the `wax` version and exit |

### Environment variables

| Variable | Description |
|----------|-------------|
| `WAX_API_KEY` / `WAXELL_API_KEY` | API key (overrides config file) |
| `WAX_API_URL` / `WAXELL_API_URL` | API URL (overrides config file) |

Configuration precedence: **`--profile` flag → `~/.waxell/config` →
environment variables**.

## Command catalog

`wax --help` groups commands into the same sections shown below. Run
`wax <command> --help` for full options on any command.

### Core

| Command | Description |
|---------|-------------|
| `wax login` | Authenticate with Waxell and save your API key |
| `wax whoami` | Show the current authenticated user and tenant |
| `wax doctor` | Run every health check and repair common breakage |
| `wax feedback` | File a bug report or feature request with the Waxell team |

### Observability

| Command | Description |
|---------|-------------|
| `wax agents` | Manage and monitor deployed agents |
| `wax runs` | Browse, inspect, and tail agent execution runs |
| `wax llm-calls` | LLM call browser and per-model analytics |
| `wax traces` | Trace explorer for distributed execution tracing |
| `wax sessions` | Session browser and analytics |
| `wax observe-users` | User analytics for agent usage |
| `wax errors` | Error analysis and diagnostics |
| `wax metrics` | Metrics dashboard and analytics |
| `wax costs` | Cost analytics and budget tracking |
| `wax lineage` | Walk the agent causality graph (runs + edges) |

### Governance

| Command | Description |
|---------|-------------|
| `wax policies` | Manage governance policies (`list`, `show`, `push`) |
| `wax kill-switches` | Manage kill switches (`list`, `show`) |
| `wax budgets` | Manage governance budgets (`list`, `show`) |
| `wax governance` | Governance events, compliance, and policy packs |

### Agent development

| Command | Description |
|---------|-------------|
| `wax push` | Auto-discover and deploy agents, tools, workflows, and skills |
| `wax push-tool` / `push-workflow` / `push-domain` / `push-skill` | Deploy a single standalone artifact |
| `wax signals` | Fire signals at agents and poll for results (`fire`, `show`) |
| `wax validate <agent>` | Validate an agent's registration and execution readiness |
| `wax context <agent>` | Check context-file coverage and quality for an agent |
| `wax test <scenarios>` | Run agent eval scenarios |
| `wax dev <path>` | Watch an agent directory and re-parse on save |
| `wax export <agent>` | Export an agent spec to JSON |
| `wax prompt` | Manage tenant-registered prompts |
| `wax mcp` | Manage the tenant's outbound MCP server registry |
| `wax claude-init` | Install the Waxell Runtime skill for Claude Code |

### Configuration

| Command | Description |
|---------|-------------|
| `wax config` | Manage `~/.waxell/config` (`init`, `show`, `set`, `profiles`) |
| `wax keys` | Manage API keys (`list`, `create`, `show`, `revoke`, `rotate`) |
| `wax secrets` | Manage secrets (`list`, `scan`, `check`) |
| `wax llm` | LLM configuration (`call`, `routing`, `models`) |
| `wax provider` | LLM provider instances, catalog, and capability groups |
| `wax domains` | Manage tenant domain configurations |
| `wax sub-user-auth` | Manage the per-tenant Ed25519 signing keypair |

### Organization & identity

| Command | Description |
|---------|-------------|
| `wax teams` / `users` / `roles` | Organization management |
| `wax identities` | Identity providers and SSO |
| `wax sub-tenants` | Manage sub-tenants |
| `wax end-users` / `end-user-groups` | Waxell end-user identity records |
| `wax billing` / `wax usage` | Billing and usage analytics |

### Integrations

| Command | Description |
|---------|-------------|
| `wax claude-code` | Claude Code observability & governance integration |
| `wax cowork` | Claude Cowork observability integration |
| `wax guard` | Inspect and edit the local-guard config |
| `wax network-monitor` | Network-monitor flow uploader (macOS) |
| `wax observe mcp-proxy` | Governance proxy for a third-party MCP server |

### Adapter platform

| Command | Description |
|---------|-------------|
| `wax adapter` | Inspect adapter capability manifests; bulk-import framework artifacts |
| `wax hub` | Inspect the canonical adapter event hub |
| `wax registry` | Browse and manage canonical registry artifacts |

## Deploying agents — `wax push`

`wax push` is the workhorse. It auto-discovers a `waxell.yaml` manifest,
packages your agents (plus tools, workflows, skills, and context files), and
deploys them to your tenant.

```bash
wax push                          # auto-discovers waxell.yaml in cwd
wax push ./projects/finance       # discover waxell.yaml under a path
wax push --dry-run                # validate and show what would push, no upload
```

| Flag | Description |
|------|-------------|
| `path` (positional, optional) | Directory or file to discover and push |
| `--profile <name>` | Config profile to push against |
| `--dry-run` | Show what would be pushed without uploading |
| `--bundle <name>` | Use a pre-approved dependency bundle (e.g. `data-science`, `web-scraping`, `ai-ml`) |
| `--include-source` / `--no-source` | Include source for isolated execution (default: auto-detect from deps) |

**TIP**
Always push with an explicit profile: `wax -p acme_local push`. Using `--local`
keeps the default profile (which has no `domain_url`), so domains register on
the wrong port and 404.


After pushing, activate or manage versions:

```bash
wax agents list                   # see all agents + execution metrics
wax agents info <name>            # signals and domain endpoints for an agent
wax agents activate <name>       # draft → active
wax agents pause <name>          # pause without deactivating
wax agents promote <name> --version <v>   # set an existing version as default
wax agents rollback <name> --version <v>  # restore a prior version as a new version
wax agents builds <name>          # container build status + history
wax agents schedule <name>        # show or manage cron schedules
```

The single-artifact variants (`push-tool`, `push-workflow`, `push-domain`,
`push-skill`) deploy one artifact without a full manifest.

## Exploring runs — `wax runs`

A **run** is one end-to-end execution of an agent workflow. Each run has a
numeric Run ID, an OTel `trace_id`, and a tree of spans (agent → LLM calls →
tool calls). Runs are listed newest-first and filtered by your plan's retention
window.

```bash
wax runs                                   # last 10 runs in the last 24h
wax runs list --hours 168                  # last week
wax runs list --agent outreach --status error
wax runs show 230 --spans                  # span tree for run 230
wax runs tail 230                          # follow a run's events live until it finishes
wax runs replay 230 --tail                 # re-fire a run with its original inputs
wax runs watch --agent my-agent            # tail new runs for one agent
wax runs stats --hours 168                 # aggregate totals + top agents
```

`wax runs list` flags:

| Flag | Description |
|------|-------------|
| `--agent <name>` | Filter to runs whose agent name contains NAME (case-insensitive) |
| `--status <success\|error\|running>` | Only show runs in this state |
| `--hours <N>` | Look back N hours (default 24). Try 168 for a week |
| `--limit <N>` | Max rows (default 10; server caps at 100) |
| `--format <table\|json>` | Pretty table or pipe-friendly JSON |

**NOTE**
`runs list` returns a **Run ID** column — pass that to `runs show`. `show` also
accepts an OTel `trace_id` or a `workflow_id`, whichever is easiest to paste.
Results are clamped to your tenant's retention window server-side.


## Firing signals — `wax signals`

Signals trigger agents and let you poll or stream the resulting run:

```bash
wax signals fire <name> --payload '{"key": "value"}'
wax signals fire <name> --payload-file ./input.json
wax signals show <run_id>                  # inspect a fired run
```

## Validating before you ship — `wax validate`

```bash
wax validate <agent_name>              # check registration + execution readiness
wax validate <agent_name> --verbose    # full diagnostic detail
wax context <agent_name>               # context-file coverage and quality
wax context <agent_name> --fix         # scaffold missing context stubs
```

## Eval harness — `wax test`

`wax test` runs scenario-based evals against an agent using the
Scenario / Turn / Expect DSL.

```bash
wax test ./evals                                  # run every scenario in a dir
wax test ./evals my_scenario                      # run named scenarios only
wax test ./evals --list                           # list scenarios + tags, don't run
wax test ./evals --tag smoke --exclude-tag slow   # filter by tag
wax test ./evals --bail                           # stop on first failure
wax test ./evals -p staging                       # run against a profile (uploads results)
```

| Flag | Description |
|------|-------------|
| `--tag <t>` / `--exclude-tag <t>` | Include/skip scenarios by tag (repeatable) |
| `--include-expensive` | Include scenarios tagged `expensive` (paid-API hits); off by default |
| `--bail` | Stop on the first scenario failure |
| `--list` | List discovered scenarios + tags and exit |
| `--report <path>` | Where to write the markdown report (default `/tmp/waxell-eval/<ts>/report.md`) |
| `--profile <name>` | `~/.waxell/config` profile to run against |
| `--trace-url-template <tpl>` | Format string for trace links in the report (`{trace_id}`) |
| `--upload` / `--no-upload` | POST results to `/v1/evals/runs/` (defaults ON when `--profile` is set) |
| `--triggered-by <user\|schedule\|ci>` | Run source, used by the eval dashboard for filtering |

## Configuration commands

```bash
wax config init                       # create ~/.waxell/config + first profile
wax config init --no-input            # scaffold without prompts
wax config show                       # print the active config (key masked)
wax config set api_key wax_sk_...     # set api_url | api_key | domain_url
wax config profiles                   # list configured profiles
```

API keys and secrets:

```bash
wax keys list                         # list API keys
wax keys create --name ci-key         # mint a new key
wax keys rotate <key_id>              # rotate a key
wax keys revoke <key_id>              # revoke a key

wax secrets list                      # list tenant secrets (values masked)
wax secrets scan ./my-agents          # scan source for hardcoded secrets
wax secrets check --agent <name>      # verify an agent's required secrets exist
```

## Common workflows

### Push and watch an agent

```bash
wax -p acme push ./my-agents          # push everything under a directory
wax runs watch --agent my-agent       # tail new runs as they arrive
wax traces                            # open the trace explorer
```

### Set up Claude Code observability

```bash
wax claude-code setup                         # install observability hooks
wax claude-code setup --governance            # + PreToolUse policy enforcement
wax claude-code setup --governance --mcp      # + MCP governance tools
wax claude-code status                        # current session state
```

### Diagnose a broken setup

```bash
wax doctor                            # runs every health check; repairs common breakage
wax whoami                            # confirm auth + tenant
```

## Next steps

- **[Installation Guide](/installation)** — get started with Waxell
- **[Runtime Quickstart](/runtime/quickstart)** — build your first agent
- **[waxell.yaml Reference](/runtime/waxell-yaml-reference)** — the manifest `wax push` reads
- **[Observe Overview](/observe/overview)** — observability for any framework


--------------------------------------------------------------------------------
# Enterprise API
URL: https://waxell.ai/docs/reference/enterprise-api
Description: API reference for enterprise features including data residency, sub-tenants, and compliance.
--------------------------------------------------------------------------------

# Enterprise API Reference

This document covers the API endpoints for Waxell's enterprise features.

## Authentication

All enterprise APIs require authentication with your API key:

```bash
curl -H "X-Wax-Key: wax_sk_..." https://api.waxell.dev/waxell/v1/...
```

## Data Residency

### List Regions

```bash
GET /waxell/v1/regions/
```

Response:
```json
{
  "regions": [
    {
      "code": "us-east-1",
      "name": "US East",
      "price_cents": 0,
      "compliance": ["HIPAA", "SOC2", "FedRAMP"]
    },
    {
      "code": "eu-west-1",
      "name": "EU West",
      "price_cents": 50000,
      "compliance": ["GDPR", "SOC2", "ISO27001"]
    }
  ]
}
```

### Get Current Region

```bash
GET /waxell/v1/tenant/region/
```

### Change Region

```bash
POST /waxell/v1/tenant/region/
Content-Type: application/json

{
  "region_code": "eu-west-1"
}
```

## Sub-Tenants

### List Sub-Tenants

```bash
GET /waxell/v1/sub-tenants/
```

Response:
```json
{
  "sub_tenants": [
    {
      "id": "st_abc123",
      "name": "Acme Corp",
      "slug": "acme-corp",
      "status": "active",
      "created_at": "2024-01-15T10:00:00Z"
    }
  ]
}
```

### Create Sub-Tenant

```bash
POST /waxell/v1/sub-tenants/
Content-Type: application/json

{
  "name": "Acme Corp",
  "external_id": "customer-123",
  "contact_email": "admin@acme.com",
  "quotas": {
    "max_agents": 50,
    "max_tokens_per_day": 500000,
    "max_agent_runs_per_day": 1000
  }
}
```

### Update Sub-Tenant

```bash
PATCH /waxell/v1/sub-tenants/{sub_tenant_id}/
Content-Type: application/json

{
  "quotas": {
    "max_agents": 100
  }
}
```

### Suspend Sub-Tenant

```bash
POST /waxell/v1/sub-tenants/{sub_tenant_id}/suspend/
Content-Type: application/json

{
  "reason": "Payment overdue"
}
```

### Resume Sub-Tenant

```bash
POST /waxell/v1/sub-tenants/{sub_tenant_id}/resume/
```

## Sub-User Identity

### Create Token

```bash
POST /waxell/v1/identity/sub-users/token/
Content-Type: application/json

{
  "sub_user_id": "user-123",
  "email": "user@example.com",
  "display_name": "John Doe",
  "roles": ["viewer", "agent_user"],
  "permissions": ["agents:trigger"],
  "expires_in": 3600
}
```

Response:
```json
{
  "token": "wax_su_eyJ...",
  "session_id": "sess_abc123",
  "expires_at": "2024-01-15T13:00:00Z"
}
```

### Validate Token

```bash
POST /waxell/v1/identity/sub-users/validate/
Content-Type: application/json

{
  "token": "wax_su_eyJ..."
}
```

### Revoke Sessions

```bash
POST /waxell/v1/identity/sub-users/{sub_user_id}/revoke/
```

## Add-Ons

### List Available Add-Ons

```bash
GET /waxell/v1/add-ons/
```

### List Active Add-Ons

```bash
GET /waxell/v1/tenant/add-ons/
```

### Activate Add-On

```bash
POST /waxell/v1/tenant/add-ons/
Content-Type: application/json

{
  "add_on_code": "hipaa_compliance"
}
```

### Cancel Add-On

```bash
DELETE /waxell/v1/tenant/add-ons/{add_on_code}/
```

## Audit Logs

### List Audit Logs

```bash
GET /waxell/v1/audit/
```

Query Parameters:
- `start_date` - Filter by start date (ISO 8601)
- `end_date` - Filter by end date (ISO 8601)
- `actor_type` - Filter by actor type (user, agent, system)
- `action` - Filter by action
- `limit` - Number of results (default: 100)
- `offset` - Pagination offset

### Export Audit Logs

```bash
POST /waxell/v1/audit/export/
Content-Type: application/json

{
  "start_date": "2024-01-01T00:00:00Z",
  "end_date": "2024-01-31T23:59:59Z",
  "format": "json"
}
```

Response:
```json
{
  "export_id": "exp_abc123",
  "status": "processing",
  "download_url": null
}
```

### Get Export Status

```bash
GET /waxell/v1/audit/export/{export_id}/
```

## Error Responses

All errors follow this format:

```json
{
  "error": {
    "code": "quota_exceeded",
    "message": "Sub-tenant quota exceeds parent limit",
    "details": {
      "field": "max_agents",
      "requested": 100,
      "parent_limit": 50
    }
  }
}
```

Common error codes:
- `unauthorized` - Invalid or missing API key
- `forbidden` - Insufficient permissions
- `not_found` - Resource not found
- `quota_exceeded` - Quota limit reached
- `validation_error` - Invalid request data

## Next Steps

- **[Enterprise Guide](/guides/enterprise)** - Configuration walkthrough
- **[Architecture](/guides/architecture)** - Technical deep dive


================================================================================
SECTION: MIGRATING TO WAXELL
Move from other observability and agent platforms
================================================================================

--------------------------------------------------------------------------------
# LangChain vs Waxell
URL: https://waxell.ai/docs/migrate/langchain-vs-waxell
Description: Side-by-side comparison of LangChain alone, LangChain with Waxell Observe, and native Waxell for building AI agents.
--------------------------------------------------------------------------------

# LangChain vs Waxell

This page compares three approaches to building the same agent: LangChain alone, LangChain enhanced with Waxell Observe, and a fully native Waxell implementation. The use case is a **customer support classifier** that categorizes incoming tickets and generates responses.

---

## A) LangChain Alone

A standard LangChain agent using `ChatOpenAI` with a prompt template and tools. This works, but you are on your own for observability, cost tracking, and governance.

```python
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.tools import tool as langchain_tool

# Define a tool
@langchain_tool
def lookup_customer(customer_id: str) -> str:
    """Look up customer details by ID."""
    # In production, this would query your database
    return f"Customer {customer_id}: Premium tier, active since 2023"

@langchain_tool
def search_knowledge_base(query: str) -> str:
    """Search the support knowledge base."""
    return f"KB result for '{query}': Reset via Settings > Account > Password"

# Build the chain
llm = ChatOpenAI(model="gpt-4o", temperature=0)
prompt = ChatPromptTemplate.from_messages([
    ("system", (
        "You are a customer support classifier. "
        "Classify the ticket into: billing, technical, or general. "
        "Then generate a helpful response."
    )),
    ("human", "Customer {customer_id} says: {message}"),
])

chain = prompt | llm.bind_tools([lookup_customer, search_knowledge_base])

# Run it
result = chain.invoke({
    "customer_id": "C-1234",
    "message": "I can't reset my password",
})
print(result.content)
```

**What is missing:**

- No visibility into how many tokens were consumed or what they cost
- No audit trail of which agent ran, when, or what it decided
- No policy enforcement (any agent can run at any time with no limits)
- No budget controls (a runaway loop can burn through your API credits)
- No durable execution (if the process crashes mid-chain, you start over)

---

## B) LangChain + Waxell Observe

The same LangChain agent with **three extra lines** to add full observability. Your agent code stays exactly the same.

```python
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.tools import tool as langchain_tool
# highlight-next-line
from waxell_observe.integrations.langchain import WaxellLangChainHandler

@langchain_tool
def lookup_customer(customer_id: str) -> str:
    """Look up customer details by ID."""
    return f"Customer {customer_id}: Premium tier, active since 2023"

@langchain_tool
def search_knowledge_base(query: str) -> str:
    """Search the support knowledge base."""
    return f"KB result for '{query}': Reset via Settings > Account > Password"

llm = ChatOpenAI(model="gpt-4o", temperature=0)
prompt = ChatPromptTemplate.from_messages([
    ("system", (
        "You are a customer support classifier. "
        "Classify the ticket into: billing, technical, or general. "
        "Then generate a helpful response."
    )),
    ("human", "Customer {customer_id} says: {message}"),
])

chain = prompt | llm.bind_tools([lookup_customer, search_knowledge_base])

# highlight-start
# Three lines to add full observability
handler = WaxellLangChainHandler(agent_name="support-classifier")
result = chain.invoke(
    {"customer_id": "C-1234", "message": "I can't reset my password"},
    config={"callbacks": [handler]},
)
handler.flush_sync(result={"output": result.content})
# highlight-end
```

**What you now get -- with no changes to your agent logic:**

- Every LLM call tracked with model name, token counts, and cost estimates
- Chain and tool execution steps recorded automatically
- Pre-execution policy checks (budget limits, rate limiting, content filtering)
- Full run lifecycle visible in the Waxell dashboard
- Audit trail of inputs, outputs, and execution status

The `WaxellLangChainHandler` hooks into LangChain's callback system. It intercepts `on_llm_start`, `on_llm_end`, `on_chain_start`, `on_chain_end`, `on_tool_start`, and `on_tool_end` events automatically. You do not need to change any agent code.

---

## C) Waxell Native

The same customer support classifier built natively with the Waxell SDK. Here, observability, governance, and durability are built into the framework itself -- there is nothing to instrument.

```python
from waxell_runtime import agent, workflow, router, tool, WorkflowContext, RouterContext

@agent(
    name="support-classifier",
    description="Classifies customer support tickets and generates responses",
    signals=["support_ticket"],
    domains=["customers", "knowledge_base"],
)
class SupportClassifier:

    @tool
    async def lookup_customer(self, ctx: WorkflowContext, customer_id: str) -> dict:
        """Look up customer details by ID."""
        return await ctx.domain(
            "customers", "get_details", customer_id=customer_id
        )

    @tool
    async def search_knowledge_base(self, ctx: WorkflowContext, query: str) -> dict:
        """Search the support knowledge base."""
        return await ctx.domain(
            "knowledge_base", "search", query=query
        )

    @router("classify_and_respond", decision="classify_ticket")
    async def classify_and_respond(self, ctx: RouterContext) -> dict:
        """Classify the ticket and route to the appropriate handler."""
        # Add customer context for the LLM decision
        customer = await ctx.tool(
            "lookup_customer",
            customer_id=ctx.inputs.get("customer_id"),
        )
        ctx.add_signal("customer_tier", customer.get("tier", "standard"))

        return await ctx.route()

    @workflow("handle_billing")
    async def handle_billing(self, ctx: WorkflowContext, message: str) -> dict:
        """Handle billing-related tickets."""
        response = await ctx.llm.generate(
            prompt=f"Generate a billing support response for: {message}",
            output_format="json",
            task="billing_response",
        )
        return response

    @workflow("handle_technical")
    async def handle_technical(self, ctx: WorkflowContext, message: str) -> dict:
        """Handle technical support tickets."""
        kb_result = await ctx.tool("search_knowledge_base", query=message)
        response = await ctx.llm.generate(
            prompt=f"Using this KB article: {kb_result}\n\nRespond to: {message}",
            output_format="json",
            task="technical_response",
        )
        return response
```

**What you gain with native Waxell:**

- **Declarative agent definition**: The `@agent` decorator registers your agent with the control plane automatically
- **Durable workflows**: If the process crashes mid-workflow, execution resumes from the last checkpoint
- **LLM routing**: `ctx.llm.generate()` uses the configured LLM router with model selection, fallbacks, and rate limiting
- **Domain integration**: `ctx.domain()` calls route through governed, audited domain endpoints
- **Zero instrumentation**: Every LLM call, tool invocation, and workflow step is tracked automatically
- **Signal-driven execution**: Agents are triggered by signals (webhooks), enabling centralized orchestration

---

## Comparison Table

| Capability | LangChain | LangChain + Observe | Waxell Native |
|---|---|---|---|
| **Agent definition** | Imperative (chains, agents) | Unchanged | Declarative (`@agent`, `@workflow`, `@tool`) |
| **Observability** | Manual (LangSmith or custom) | Automatic via callback handler | Built-in, zero instrumentation |
| **LLM cost tracking** | Not included | Automatic for 20+ models | Built-in with tenant-level overrides |
| **Policy enforcement** | Not included | Pre-execution checks | Full lifecycle governance |
| **Budget limits** | Not included | Supported via policies | Built-in with tenant/agent scoping |
| **Durable workflows** | Not included | Not included | Checkpoint/resume with WorkflowEnvelope |
| **Approval workflows** | Not included | Not included | Built-in with pause/resume |
| **Multi-tenancy** | Not included | Tenant-scoped via control plane | Native tenant isolation |
| **Audit logging** | Not included | Run-level audit trail | Full execution trace with `agent_trace` |

---

## Which Approach Should You Choose?

**Start with Observe**
If you already have LangChain agents in production, start with **LangChain + Observe**. You get immediate value (visibility, cost tracking, policy enforcement) with zero changes to your agent code. You can always migrate to native Waxell later if you need durable workflows or full governance.


See the [Progressive Migration](./overview) guide for a phased approach to adopting Waxell.


--------------------------------------------------------------------------------
# CrewAI vs Waxell
URL: https://waxell.ai/docs/migrate/crewai-vs-waxell
Description: Side-by-side comparison of CrewAI alone, CrewAI with Waxell Observe, and native Waxell for building AI agents.
--------------------------------------------------------------------------------

# CrewAI vs Waxell

This page compares three approaches to building the same agent: CrewAI alone, CrewAI enhanced with Waxell Observe, and a fully native Waxell implementation. The use case is a **research agent** that searches for information and produces a summary report.

---

## A) CrewAI Alone

A standard CrewAI setup with an Agent, Task, and Crew. CrewAI provides a high-level abstraction for multi-agent orchestration, but leaves observability, cost tracking, and governance to you.

```python
from crewai import Agent, Task, Crew
from crewai_tools import SerperDevTool

# Define tools
search_tool = SerperDevTool()

# Define the research agent
researcher = Agent(
    role="Research Analyst",
    goal="Find comprehensive information about the given topic",
    backstory="You are an expert research analyst skilled at finding and synthesizing information.",
    tools=[search_tool],
    llm="gpt-4o",
    verbose=True,
)

# Define the summarizer agent
summarizer = Agent(
    role="Report Writer",
    goal="Produce a clear, well-structured summary report",
    backstory="You are a skilled technical writer who distills complex research into actionable summaries.",
    llm="gpt-4o",
    verbose=True,
)

# Define tasks
research_task = Task(
    description="Research the topic: {topic}. Find key facts, recent developments, and expert opinions.",
    expected_output="A detailed research brief with sources",
    agent=researcher,
)

summary_task = Task(
    description="Using the research brief, write a concise executive summary with key findings and recommendations.",
    expected_output="A structured executive summary in markdown",
    agent=summarizer,
)

# Build and run the crew
crew = Crew(
    agents=[researcher, summarizer],
    tasks=[research_task, summary_task],
    verbose=True,
)

result = crew.kickoff(inputs={"topic": "AI governance trends 2025"})
print(result)
```

**What is missing:**

- No centralized tracking of LLM calls across agents
- No cost visibility (how much did this crew run cost?)
- No policy enforcement (no budget limits, no content filtering)
- No audit trail beyond verbose console output
- No durability (a crash means re-running the entire crew from scratch)

---

## B) CrewAI + Waxell Observe

The same CrewAI workflow wrapped with `WaxellContext` for full observability. Your CrewAI code stays the same -- you just wrap the execution in a context manager.

```python
from crewai import Agent, Task, Crew
from crewai_tools import SerperDevTool
# highlight-next-line
from waxell_observe import WaxellContext

search_tool = SerperDevTool()

researcher = Agent(
    role="Research Analyst",
    goal="Find comprehensive information about the given topic",
    backstory="You are an expert research analyst skilled at finding and synthesizing information.",
    tools=[search_tool],
    llm="gpt-4o",
    verbose=True,
)

summarizer = Agent(
    role="Report Writer",
    goal="Produce a clear, well-structured summary report",
    backstory="You are a skilled technical writer who distills complex research into actionable summaries.",
    llm="gpt-4o",
    verbose=True,
)

research_task = Task(
    description="Research the topic: {topic}. Find key facts, recent developments, and expert opinions.",
    expected_output="A detailed research brief with sources",
    agent=researcher,
)

summary_task = Task(
    description="Using the research brief, write a concise executive summary with key findings and recommendations.",
    expected_output="A structured executive summary in markdown",
    agent=summarizer,
)

crew = Crew(
    agents=[researcher, summarizer],
    tasks=[research_task, summary_task],
    verbose=True,
)

# highlight-start
# Wrap execution with Waxell Observe
async with WaxellContext(agent_name="research-crew") as ctx:
    result = crew.kickoff(inputs={"topic": "AI governance trends 2025"})

    # Record LLM calls (CrewAI exposes token usage in its output)
    if hasattr(result, "token_usage"):
        ctx.record_llm_call(
            model="gpt-4o",
            tokens_in=result.token_usage.get("prompt_tokens", 0),
            tokens_out=result.token_usage.get("completion_tokens", 0),
            task="research_crew",
        )

    # Record execution steps
    ctx.record_step("research", output={"status": "complete"})
    ctx.record_step("summarize", output={"status": "complete"})

    ctx.set_result({"summary": str(result)})
# highlight-end
```

**Decorator alternative**
You can also use the `@waxell_agent` decorator for simpler CrewAI setups:

```python
from waxell_observe import waxell_agent

@waxell_agent(agent_name="research-crew")
async def run_research(topic: str, waxell_ctx=None) -> str:
    crew = Crew(agents=[researcher, summarizer], tasks=[research_task, summary_task])
    result = crew.kickoff(inputs={"topic": topic})

    if waxell_ctx and hasattr(result, "token_usage"):
        waxell_ctx.record_llm_call(
            model="gpt-4o",
            tokens_in=result.token_usage.get("prompt_tokens", 0),
            tokens_out=result.token_usage.get("completion_tokens", 0),
        )

    return str(result)
```


**What you now get -- with minimal changes to your CrewAI code:**

- Every crew execution tracked as a run with inputs, outputs, and status
- LLM call recording with cost estimates
- Step-by-step execution trail for debugging
- Pre-execution policy checks (budget enforcement, rate limiting)
- Full audit trail visible in the Waxell dashboard

---

## C) Waxell Native

The same research-and-summarize use case built natively with the Waxell SDK. The workflow, tools, and LLM calls are all governed and tracked by default.

```python
from waxell_runtime import agent, workflow, tool, WorkflowContext

@agent(
    name="research-agent",
    description="Researches topics and produces executive summaries",
    signals=["research_request"],
    domains=["search"],
)
class ResearchAgent:

    @tool
    async def web_search(self, ctx: WorkflowContext, query: str) -> dict:
        """Search the web for information on a topic."""
        return await ctx.domain("search", "web_search", query=query)

    @workflow("research_and_summarize")
    async def research_and_summarize(self, ctx: WorkflowContext, topic: str) -> dict:
        """Research a topic and produce an executive summary."""

        # Step 1: Search for information
        search_results = await ctx.tool("web_search", query=topic)
        ctx.log_step("research_complete", {"results_count": len(search_results)})

        # Step 2: Synthesize research into a brief
        research_brief = await ctx.llm.generate(
            prompt=(
                f"Synthesize these search results into a research brief:\n"
                f"Topic: {topic}\n"
                f"Results: {search_results}"
            ),
            output_format="text",
            task="research_synthesis",
        )

        # Step 3: Generate executive summary
        summary = await ctx.llm.generate(
            prompt=(
                f"Write a concise executive summary with key findings "
                f"and recommendations based on this research brief:\n\n"
                f"{research_brief}"
            ),
            output_format="json",
            task="executive_summary",
        )

        return summary
```

**What you gain with native Waxell:**

- **Single-agent simplicity**: No need to define separate "agents" for research and summarization -- workflows handle orchestration
- **Durable execution**: Each step is a checkpoint. If the process crashes after the search completes, it resumes at the synthesis step
- **LLM routing**: `ctx.llm.generate()` automatically selects models based on task type, applies rate limiting, and handles fallbacks
- **Domain abstraction**: `ctx.domain("search", "web_search", ...)` routes through a governed endpoint, not a hardcoded API call
- **Zero instrumentation**: Every LLM call, tool invocation, and step is tracked in the control plane automatically

---

## Comparison Table

| Capability | CrewAI | CrewAI + Observe | Waxell Native |
|---|---|---|---|
| **Agent definition** | Agent/Task/Crew classes | Unchanged | Declarative (`@agent`, `@workflow`, `@tool`) |
| **Multi-agent orchestration** | Built-in (Crew) | Unchanged | Workflows with steps |
| **Observability** | Verbose console output | Run tracking via WaxellContext | Built-in, zero instrumentation |
| **LLM cost tracking** | Not included | Manual recording with auto-estimation | Built-in with tenant-level overrides |
| **Policy enforcement** | Not included | Pre-execution checks | Full lifecycle governance |
| **Budget limits** | Not included | Supported via policies | Built-in with tenant/agent scoping |
| **Durable workflows** | Not included | Not included | Checkpoint/resume with WorkflowEnvelope |
| **Approval workflows** | Not included | Not included | Built-in with pause/resume |
| **Multi-tenancy** | Not included | Tenant-scoped via control plane | Native tenant isolation |
| **Audit logging** | Not included | Run-level audit trail | Full execution trace with `agent_trace` |

---

## Which Approach Should You Choose?

**Start with Observe**
If you already have CrewAI agents running, start with **CrewAI + Observe**. Wrap your `crew.kickoff()` calls with `WaxellContext` and you immediately get cost tracking, policy enforcement, and audit trails. Migration to native Waxell can happen later when you need durable workflows.


See the [Progressive Migration](./overview) guide for a phased approach to adopting Waxell.


--------------------------------------------------------------------------------
# Feature Comparison Matrix
URL: https://waxell.ai/docs/migrate/feature-matrix
Description: Comprehensive feature comparison across LangChain, CrewAI, custom Python agents, Waxell Observe, and Waxell Native.
--------------------------------------------------------------------------------

# Feature Comparison Matrix

A comprehensive comparison of what you get with each framework and integration approach. The columns progress from standalone frameworks on the left to full Waxell native on the right.

## Legend

| Symbol | Meaning |
|--------|---------|
| **Yes** | Fully supported out of the box |
| **Partial** | Partially supported or requires significant custom work |
| **No** | Not available |

---

## Core Agent Capabilities

| Feature | LangChain | LangChain + Observe | CrewAI | CrewAI + Observe | Custom Python | Custom + Observe | Waxell Native |
|---|---|---|---|---|---|---|---|
| Declarative agent definitions | No | No | Partial | Partial | No | No | **Yes** |
| Tool abstraction | Yes | Yes | Yes | Yes | Manual | Manual | **Yes** |
| Workflow orchestration | Partial | Partial | Yes | Yes | Manual | Manual | **Yes** |
| Multi-agent coordination | Partial | Partial | Yes | Yes | Manual | Manual | **Yes** |
| LLM model routing | Manual | Manual | Manual | Manual | Manual | Manual | **Yes** |

**Notes on partial support**
- **LangChain declarative**: LangChain Expression Language (LCEL) provides composable chains, but agent definitions remain imperative.
- **CrewAI declarative**: Agent/Task/Crew classes are semi-declarative, but orchestration logic is still code-level.
- **LangChain workflow orchestration**: LangGraph adds graph-based workflows, but lacks durability and governance.


---

## Observability

| Feature | LangChain | LangChain + Observe | CrewAI | CrewAI + Observe | Custom Python | Custom + Observe | Waxell Native |
|---|---|---|---|---|---|---|---|
| Execution run tracking | No | **Yes** | No | **Yes** | No | **Yes** | **Yes** |
| LLM call tracking | No | **Yes** | No | **Yes** | No | **Yes** | **Yes** |
| Automatic token counting | No | **Yes** | No | Partial | No | Partial | **Yes** |
| Step-by-step execution trail | No | **Yes** | No | **Yes** | No | **Yes** | **Yes** |
| Input/output capture | No | **Yes** | No | **Yes** | No | **Yes** | **Yes** |
| Dashboard UI | No | **Yes** | No | **Yes** | No | **Yes** | **Yes** |

**Notes on automatic token counting**
- **LangChain + Observe**: The `WaxellLangChainHandler` callback extracts token usage from LangChain's `LLMResult` automatically.
- **CrewAI + Observe / Custom + Observe**: Token counts must be passed to `ctx.record_llm_call()` manually or extracted from your LLM client's response.
- **Waxell Native**: The LLM router tracks all token usage automatically.


---

## Cost Management

| Feature | LangChain | LangChain + Observe | CrewAI | CrewAI + Observe | Custom Python | Custom + Observe | Waxell Native |
|---|---|---|---|---|---|---|---|
| LLM cost estimation | No | **Yes** | No | **Yes** | No | **Yes** | **Yes** |
| Per-model pricing (20+ models) | No | **Yes** | No | **Yes** | No | **Yes** | **Yes** |
| Tenant-level cost overrides | No | **Yes** | No | **Yes** | No | **Yes** | **Yes** |
| Budget enforcement | No | **Yes** | No | **Yes** | No | **Yes** | **Yes** |
| Cost-per-run visibility | No | **Yes** | No | **Yes** | No | **Yes** | **Yes** |

---

## Governance and Policy

| Feature | LangChain | LangChain + Observe | CrewAI | CrewAI + Observe | Custom Python | Custom + Observe | Waxell Native |
|---|---|---|---|---|---|---|---|
| Pre-execution policy checks | No | **Yes** | No | **Yes** | No | **Yes** | **Yes** |
| Budget limit policies | No | **Yes** | No | **Yes** | No | **Yes** | **Yes** |
| Rate limiting / throttling | No | **Yes** | No | **Yes** | No | **Yes** | **Yes** |
| Content filtering | No | Partial | No | Partial | No | Partial | **Yes** |
| Approval workflows | No | No | No | No | No | No | **Yes** |
| Dynamic policy management | No | No | No | No | No | No | **Yes** |
| Full governance lifecycle | No | No | No | No | No | No | **Yes** |

**Notes on content filtering**
- **+Observe**: Policy checks can block execution based on agent name, workflow, or budget. Content-level filtering (inspecting prompts/responses) requires custom policy rules in the control plane.
- **Waxell Native**: The `DynamicPolicyManager` evaluates policies at every governance hook point, including pre-execution, mid-workflow, and post-completion.


---

## Durability and Infrastructure

| Feature | LangChain | LangChain + Observe | CrewAI | CrewAI + Observe | Custom Python | Custom + Observe | Waxell Native |
|---|---|---|---|---|---|---|---|
| Durable workflows | No | No | No | No | No | No | **Yes** |
| Checkpoint / resume | No | No | No | No | No | No | **Yes** |
| Pause / resume (human-in-the-loop) | No | No | No | No | No | No | **Yes** |
| Multi-tenancy | No | Partial | No | Partial | No | Partial | **Yes** |
| Signal-driven execution (webhooks) | No | No | No | No | No | No | **Yes** |
| Production backends (Redis, Celery) | No | No | No | No | No | No | **Yes** |
| Audit trail with `agent_trace` | No | No | No | No | No | No | **Yes** |
| Generation layer (RAG, prompt versioning) | No | No | No | No | No | No | **Yes** |

**Notes on multi-tenancy**
- **+Observe**: Runs are scoped to a tenant via the control plane API key. Data isolation is at the API level.
- **Waxell Native**: Full tenant isolation at the database level, with per-tenant policies, model configurations, and billing.


---

## Summary

The progression from left to right represents increasing levels of governance and infrastructure:

1. **Standalone frameworks** (LangChain, CrewAI, custom) give you agent capabilities but no governance.
2. **+ Waxell Observe** adds observability, cost tracking, and basic policy enforcement with minimal code changes.
3. **Waxell Native** provides the full stack: declarative definitions, durable workflows, signal-driven execution, full governance lifecycle, and production infrastructure.

Each level delivers standalone value. See the [Progressive Migration](./overview) guide for how to move between them at your own pace.


--------------------------------------------------------------------------------
# Progressive Migration
URL: https://waxell.ai/docs/migrate/overview
Description: A phased approach to adopting Waxell. Each phase delivers standalone value -- you can stop at any phase.
--------------------------------------------------------------------------------

# Progressive Migration

Waxell is designed for incremental adoption. You do not need to rewrite your agents to get value. Each phase below builds on the previous one, and **every phase delivers standalone value** -- you can stop at any phase and still have a better system than where you started.

---

## The Migration Staircase

```
Phase 4: Full Native Waxell
          Durable workflows, domains, native governance, generation layer
          ──────────────────────────────────────────────────────────
Phase 3: Agent Builder
          AI-assisted conversion of existing agents (coming soon)
          ──────────────────────────────────────────────────────────
Phase 2: Signals
          Webhook-triggered execution, centralized orchestration
          ──────────────────────────────────────────────────────────
Phase 1: Observe
          Observability, cost tracking, policy enforcement
          ──────────────────────────────────────────────────────────
Your Existing Agents
LangChain, CrewAI, custom Python, anything else
```

Each level adds capabilities without removing what you already have. Your existing agents keep running throughout the process.

---

## Phase 1: Add Observability

**What it is**: Install `waxell-observe` and wrap your existing agents with a decorator, context manager, or LangChain callback handler. No agent code changes required.

**What you gain**:
- Execution run tracking with inputs, outputs, and status
- LLM call visibility with token counts and cost estimates
- Pre-execution policy enforcement (budget limits, rate limiting)
- A dashboard to see what your agents are doing

**Effort level**: Minutes. Install the package, add a few lines of configuration, wrap your agent entry points.

**Read more**: [Phase 1: Add Observability](./phase-1-observe)

---

## Phase 2: Add Signals

**What it is**: Move from ad-hoc agent triggering to signal-driven execution. Register webhook endpoints with Waxell and trigger agent runs via HTTP POST.

**What you gain**:
- Centralized orchestration of all your agents from one place
- Built-in retry logic and error handling for triggers
- Audit trail of what triggered each agent run and when
- Scheduling and deduplication via idempotency keys

**Effort level**: Hours. Register signal endpoints, update your trigger code to POST to Waxell instead of calling agents directly.

**Read more**: [Phase 2: Add Signals](./phase-2-signals)

---

## Phase 3: Agent Builder

**What it is**: An AI-assisted tool that analyzes your existing agent code and generates native Waxell equivalents. It maps your patterns to Waxell SDK constructs and lets you review the output before committing.

**What you gain**:
- Automated translation of existing agent logic to `@agent`, `@workflow`, `@tool` decorators
- Pattern recognition (your LLM calls become `@decision`, your orchestration becomes `@workflow`)
- Review-and-customize workflow so you stay in control

**Effort level**: Minimal. Run the tool, review the output, customize as needed.

**Coming Soon**
Agent Builder is currently in development. [Sign up for early access](https://waxell.ai) to be notified when it is available.


**Read more**: [Phase 3: Agent Builder](./phase-3-agent-builder)

---

## Phase 4: Go Fully Native

**What it is**: Build your agents using the Waxell SDK from the ground up. Define agents, workflows, tools, and decisions declaratively. Run on Waxell's durable runtime with full infrastructure support.

**What you gain**:
- Declarative agent definitions with `@agent`, `@workflow`, `@tool`, `@router`
- Durable workflows with checkpoint/resume via `WorkflowEnvelope`
- Full governance lifecycle with `DynamicPolicyManager`
- Production infrastructure: Redis backends, Celery task execution, Django ORM
- Generation layer for RAG, prompt versioning, and content synthesis
- Multi-tenancy with per-tenant policies, model configs, and billing

**Effort level**: Days to weeks depending on agent complexity. This is a full rewrite, but you have full control over the architecture.

**Read more**: [Phase 4: Go Fully Native](./phase-4-native)

---

## Decision Tree: Which Phase Should I Target?

Use this to decide where to start -- and where to stop.

### "I just want visibility into what my agents are doing."

Start and stop at **Phase 1**. Install `waxell-observe`, add a decorator or context manager to your agent entry points, and you are done. You get a dashboard, cost tracking, and policy enforcement with no agent rewrites.

### "I want centralized orchestration of my agents."

Go to **Phase 2**. After setting up observability in Phase 1, register signal endpoints to trigger your agents via webhooks. This gives you a single control point for all agent execution, with retry logic and audit trails.

### "I want durable workflows and full governance."

Target **Phase 4**. If your use case demands checkpoint/resume, approval workflows, or infrastructure-grade durability, the native Waxell SDK is the right choice. Consider using Agent Builder (Phase 3) to accelerate the migration.

### "I am not sure yet."

**Start with Phase 1**. It takes minutes, delivers immediate value, and gives you the data to make an informed decision about further migration. Once you can see what your agents are doing -- how much they cost, how often they fail, where they spend time -- the right next step usually becomes obvious.

---

## Comparison of Phases

| | Phase 1: Observe | Phase 2: Signals | Phase 3: Builder | Phase 4: Native |
|---|---|---|---|---|
| **Agent code changes** | None to minimal | Trigger code only | AI-assisted | Full rewrite |
| **Time to value** | Minutes | Hours | Minutes | Days to weeks |
| **Observability** | Yes | Yes | Yes | Yes |
| **Cost tracking** | Yes | Yes | Yes | Yes |
| **Policy enforcement** | Basic | Basic | Full | Full |
| **Centralized triggering** | No | Yes | Yes | Yes |
| **Durable workflows** | No | No | Yes | Yes |
| **Approval workflows** | No | No | Yes | Yes |
| **Multi-tenancy** | API-level | API-level | Native | Native |

---

## Next Steps

- [LangChain vs Waxell](./langchain-vs-waxell) -- Side-by-side comparison with code examples
- [CrewAI vs Waxell](./crewai-vs-waxell) -- Side-by-side comparison with code examples
- [Feature Comparison Matrix](./feature-matrix) -- Comprehensive feature table
- [Phase 1: Add Observability](./phase-1-observe) -- Get started in 5 minutes


--------------------------------------------------------------------------------
# Phase 1: Add Observability
URL: https://waxell.ai/docs/migrate/phase-1-observe
Description: Add observability, cost tracking, and policy enforcement to your existing AI agents with waxell-observe.
--------------------------------------------------------------------------------

# Phase 1: Add Observability

**You are here if**: you have existing AI agents (LangChain, CrewAI, custom Python, or anything else) and you want to add observability without rewriting them.

**What you will have after this phase**: execution traces, LLM call visibility, cost tracking, and basic policy enforcement -- all without changing your agent logic.

---

## What You Get

### Execution Run Tracking

Every agent invocation becomes a tracked **run** in the Waxell control plane. Each run records:

- Agent name and workflow
- Start time and duration
- Input parameters and output results
- Final status (success, error, or policy-blocked)

### LLM Call Visibility

See exactly what your agents are doing under the hood:

- Which models are being called and how often
- Token counts (prompt and completion) per call
- Prompt and response previews for debugging
- Cost estimates based on per-model pricing tables

### Cost Tracking

Automatic cost estimation for 20+ LLM models. The control plane aggregates costs by agent, workflow, tenant, and time period. Set budget policies to get alerts or block execution when spending exceeds thresholds.

### Policy Enforcement

Pre-execution policy checks run before your agent starts. Policies can:

- **Allow** the run to proceed
- **Block** the run with a `PolicyViolationError`
- **Warn** (allow but log a warning)
- **Throttle** (rate-limit executions)

---

## How to Add It

Choose the integration pattern that fits your codebase. All three patterns provide the same observability features.

### Pattern 1: Decorator (Simplest)

Best for standalone agent functions.

```python
from waxell_observe import waxell_agent

@waxell_agent(agent_name="my-agent")
async def run_my_agent(query: str, waxell_ctx=None) -> str:
    result = await my_llm_call(query)

    # Optional: record individual LLM calls for cost tracking
    if waxell_ctx:
        waxell_ctx.record_llm_call(
            model="gpt-4o",
            tokens_in=150,
            tokens_out=80,
        )

    return result
```

The decorator handles run lifecycle automatically: it starts a run on entry, captures inputs/outputs, checks policies, and completes the run on exit.

### Pattern 2: Context Manager (Fine-grained)

Best when you need explicit control over what gets recorded.

```python
from waxell_observe import WaxellContext

async with WaxellContext(agent_name="my-agent") as ctx:
    # Your existing agent code -- unchanged
    result = await my_agent.run(query)

    # Record LLM calls
    ctx.record_llm_call(model="gpt-4o", tokens_in=500, tokens_out=200)

    # Record execution steps
    ctx.record_step("classify", output={"category": "billing"})
    ctx.record_step("generate_response", output={"length": 250})

    # Set the final result
    ctx.set_result({"answer": result})
```

The context manager gives you access to `record_llm_call()`, `record_step()`, `set_result()`, and `check_policy()` for mid-execution policy checks.

### Pattern 3: LangChain Callback (Zero-change)

Best for LangChain agents. LLM calls are captured automatically via callback hooks.

```python
from waxell_observe.integrations.langchain import WaxellLangChainHandler

handler = WaxellLangChainHandler(agent_name="my-langchain-agent")

# Your existing chain -- unchanged
result = chain.invoke(
    {"question": "How do I reset my password?"},
    config={"callbacks": [handler]},
)

# Flush buffered telemetry when done
handler.flush_sync(result={"output": result.content})
```

The handler automatically intercepts LLM calls, chain executions, and tool usage. No manual `record_llm_call()` needed.

---

## What to Do After Setup

### 1. Review Your Dashboard

Log in to your Waxell control plane and explore the runs view. You should see:

- Recent agent executions with status and duration
- LLM calls with model, token, and cost breakdowns
- Execution steps showing the flow of each run

### 2. Set Up Cost Alerts

In the control plane, configure budget policies for your agents:

- **Daily token limits**: Block or warn when an agent exceeds a token budget
- **Per-run cost limits**: Prevent expensive individual runs
- **Aggregate spend alerts**: Get notified when total spend crosses a threshold

### 3. Configure Policies

Start with conservative policies and tune as you learn your agents' behavior:

- **Rate limiting**: Prevent runaway loops from overwhelming your LLM provider
- **Budget enforcement**: Set spending limits per agent and per tenant
- **Execution policies**: Control which agents can run and when

### 4. Explore Agent Behavior

With observability in place, you can now answer questions you could not before:

- Which agents are the most expensive to run?
- What is the average token usage per agent invocation?
- Are there agents that fail frequently? Why?
- How much does each LLM call cost relative to the value it produces?

---

## What You Do Not Get (Yet)

Phase 1 focuses on observability and basic governance. The following capabilities require further migration:

- **Durable workflows**: Checkpoint/resume requires native Waxell workflows (Phase 4)
- **Approval workflows**: Pause/resume with human-in-the-loop requires native Waxell (Phase 4)
- **Signal-driven execution**: Webhook triggering requires Phase 2
- **Centralized orchestration**: Coordinating multiple agents from one place requires Phase 2

---

## Next Steps

- [Observe Overview](/observe/overview) -- Detailed documentation for `waxell-observe`
- [Observe Quickstart](/observe/quickstart) -- Step-by-step setup guide
- [Phase 2: Add Signals](./phase-2-signals) -- Move to centralized orchestration
- [Progressive Migration Overview](./overview) -- See all migration phases


--------------------------------------------------------------------------------
# Phase 2: Add Signals
URL: https://waxell.ai/docs/migrate/phase-2-signals
Description: Move from ad-hoc agent triggering to webhook-driven execution with Waxell signals.
--------------------------------------------------------------------------------

# Phase 2: Add Signals

**You are here if**: you have completed Phase 1 (your agents are observed) and you want centralized control over when and how your agents are triggered.

**What you will have after this phase**: webhook-driven agent execution with retry logic, deduplication, audit trails, and a single control point for all your agents.

---

## What Are Signals?

A **signal** is a typed HTTP webhook that triggers an agent run through the Waxell control plane. Instead of calling your agent function directly, you POST a signal payload to Waxell, and the control plane routes it to the right agent with full governance applied.

```
Your App ──POST──> Waxell Control Plane ──triggers──> Your Agent
                          │
                          ├── Policy check
                          ├── Rate limiting
                          ├── Audit logging
                          └── Retry on failure
```

Signals decouple your application code from your agent execution. Your app does not need to know where the agent runs, how it is configured, or what policies apply -- it just emits a signal.

---

## How Signals Work

### 1. Define a Signal in Your Agent

If you are already using Waxell Observe, your agents are tracked but triggered directly. To add signal-driven execution, define the signal your agent listens for:

```python
# In your Waxell control plane configuration, register a signal:
# Signal name: "support_ticket"
# Payload schema: { customer_id: string, message: string }
# Routes to: support-classifier agent
```

### 2. Emit Signals from Your Application

Replace direct agent calls with signal emissions using the Waxell SDK client:

**Before (direct call):**

```python
# Your application calls the agent directly
result = await handle_support_ticket(customer_id="C-1234", message="I can't log in")
```

**After (signal-driven):**

```python
from waxell_sdk import WaxellClient

client = WaxellClient()

result = await client.emit_signal(
    signal_name="support_ticket",
    payload={
        "customer_id": "C-1234",
        "message": "I can't log in",
    },
    idempotency_key="ticket-5678",  # Prevents duplicate processing
)

if result.success:
    print(f"Signal emitted: {result.signal_id}")
else:
    print(f"Error: {result.error}")
```

### 3. Configure the Client

The `WaxellClient` reads configuration from multiple sources (highest priority first):

1. Explicit constructor arguments
2. Global configuration via `WaxellClient.configure()`
3. CLI config file (`~/.waxell/config`)
4. Environment variables (`WAX_API_URL`, `WAX_API_KEY`)

```python
# Option A: Environment variables (recommended for production)
# export WAX_API_URL="https://acme.waxell.dev"
# export WAX_API_KEY="wax_sk_..."
client = WaxellClient()

# Option B: Global configuration at app startup
WaxellClient.configure(
    api_url="https://acme.waxell.dev",
    api_key="wax_sk_...",
)
client = WaxellClient()

# Option C: Per-instance (overrides everything)
client = WaxellClient(
    api_url="https://acme.waxell.dev",
    api_key="wax_sk_...",
)
```

For synchronous contexts (e.g., Django views without async), use the sync variant:

```python
result = client.emit_signal_sync(
    signal_name="support_ticket",
    payload={"customer_id": "C-1234", "message": "I can't log in"},
)
```

---

## What You Gain

### Centralized Orchestration

All agent triggering flows through the Waxell control plane. You can see every signal that was emitted, which agent it triggered, and what happened.

### Retry Logic

If the agent fails, the control plane can retry the execution based on your configured retry policy. Your application does not need to implement retry logic.

### Deduplication

The `idempotency_key` parameter prevents duplicate processing. If the same signal is emitted twice (e.g., due to a network retry on the caller side), only one agent execution occurs.

### Audit Trail

Every signal emission is logged with:

- Who emitted it (API key / tenant)
- When it was emitted
- The full payload
- Which agent run it triggered
- The execution result

### Scheduling

Signals can be scheduled for future execution. Emit a signal with a delay or cron expression to trigger agents on a schedule, all managed through the control plane.

---

## Integration Pattern: Existing Observed Agents

If your agents are already wrapped with `@waxell_agent` or `WaxellContext` (from Phase 1), signals integrate cleanly. The control plane calls your observed agent, and both the signal audit trail and the observe telemetry are linked.

```python
from waxell_observe import waxell_agent

# This agent is both observed AND signal-triggered
@waxell_agent(agent_name="support-classifier")
async def handle_support_ticket(
    customer_id: str,
    message: str,
    waxell_ctx=None,
) -> str:
    result = await classify_and_respond(customer_id, message)

    if waxell_ctx:
        waxell_ctx.record_llm_call(
            model="gpt-4o",
            tokens_in=200,
            tokens_out=150,
            task="classify_ticket",
        )

    return result
```

The signal payload fields (`customer_id`, `message`) are passed as arguments to your agent function. The `waxell_ctx` is injected by the decorator as usual.

---

## What You Do Not Get (Yet)

Phase 2 adds centralized triggering but your agents still run as standard Python functions. The following require further migration:

- **Durable workflows**: Checkpoint/resume requires native Waxell workflows (Phase 4)
- **Approval workflows**: Pause/resume with human-in-the-loop requires native Waxell (Phase 4)
- **Domain abstraction**: Governed external system calls require native Waxell domains (Phase 4)
- **LLM routing**: Automatic model selection and fallbacks require native Waxell (Phase 4)

---

## Next Steps

- [Phase 3: Agent Builder](./phase-3-agent-builder) -- AI-assisted migration to native Waxell
- [Phase 4: Go Fully Native](./phase-4-native) -- Manual migration guide
- [Progressive Migration Overview](./overview) -- See all migration phases


--------------------------------------------------------------------------------
# Phase 3: Agent Builder
URL: https://waxell.ai/docs/migrate/phase-3-agent-builder
Description: AI-assisted migration tool that converts existing agents to native Waxell SDK definitions.
--------------------------------------------------------------------------------

# Phase 3: Agent Builder

**Coming Soon**
Agent Builder is currently in development. The feature described on this page is not yet available. Sign up below to be notified when it launches.


**You are here if**: you have existing agents running with Waxell Observe (Phase 1) and optionally signals (Phase 2), and you want to migrate to native Waxell without manually rewriting everything.

**What you will have after this phase**: your existing agent logic expressed as native Waxell SDK constructs (`@agent`, `@workflow`, `@tool`, `@router`), with durable workflows and full governance.

---

## What Agent Builder Will Do

Agent Builder is an AI-assisted tool that automates the conversion of existing agent code into native Waxell SDK definitions. It works in four steps:

### 1. Analyze Your Existing Code

Point Agent Builder at your agent source files. It reads your code and identifies:

- LLM calls (OpenAI, Anthropic, or other provider SDKs)
- Tool definitions and external API integrations
- Orchestration logic (sequencing, branching, loops)
- State management patterns
- Error handling and retry logic

### 2. Map Patterns to Waxell Constructs

Agent Builder maps what it finds to the corresponding Waxell SDK primitives:

| Your Code | Waxell Equivalent |
|---|---|
| LLM API calls | `@router` with decision or `ctx.llm.generate()` |
| Tool functions | `@tool` decorators |
| Orchestration logic | `@workflow` decorators |
| Agent class or entry point | `@agent` decorator |
| External API integrations | Domain calls via `ctx.domain()` |
| Manual observability code | Removed (built-in) |

### 3. Generate Native Waxell Code

Agent Builder produces complete, working Waxell agent definitions. The generated code follows Waxell best practices:

- Declarative `@agent` definitions with signals, domains, and capabilities
- Workflows that express orchestration as durable step sequences
- Tools with typed inputs extracted from your existing function signatures
- LLM calls routed through `ctx.llm.generate()` with task annotations

### 4. Review and Customize

Agent Builder does not commit anything automatically. It presents the generated code for your review in a side-by-side view:

- **Left**: Your original agent code
- **Right**: The generated Waxell equivalent
- **Annotations**: Explanations of each mapping decision

You can accept, modify, or reject each generated component individually.

---

## Example: What It Looks Like

Given a LangChain agent like this:

```python
from langchain_openai import ChatOpenAI
from langchain_core.tools import tool

@tool
def search_docs(query: str) -> str:
    """Search documentation."""
    return requests.get(f"https://api.docs.com/search?q={query}").text

llm = ChatOpenAI(model="gpt-4o")
agent = create_tool_calling_agent(llm, [search_docs], prompt)
executor = AgentExecutor(agent=agent, tools=[search_docs])

result = executor.invoke({"input": "How do I configure SSL?"})
```

Agent Builder would generate:

```python
from waxell_sdk import agent, workflow, tool, WorkflowContext

@agent(
    name="docs-assistant",
    description="Searches documentation and answers questions",
    signals=["docs_query"],
    domains=["documentation"],
)
class DocsAssistant:

    @tool
    async def search_docs(self, ctx: WorkflowContext, query: str) -> dict:
        """Search documentation."""
        return await ctx.domain("documentation", "search", query=query)

    @workflow("answer_question")
    async def answer_question(self, ctx: WorkflowContext, question: str) -> dict:
        """Search docs and generate an answer."""
        docs = await ctx.tool("search_docs", query=question)
        answer = await ctx.llm.generate(
            prompt=f"Using these docs: {docs}\n\nAnswer: {question}",
            output_format="text",
            task="docs_answer",
        )
        return {"answer": answer}
```

---

## Who Is It For?

Agent Builder is most useful when you have:

- **Multiple agents** that would be tedious to convert manually
- **Complex orchestration** with many steps, branches, or tool calls
- **LangChain or CrewAI agents** that follow common patterns Agent Builder recognizes
- **Teams** where not everyone is familiar with the Waxell SDK yet

If you have a single, simple agent, [manual migration](./phase-4-native) may be faster.

---

## Early Access

Agent Builder is under active development. To be notified when it launches:

**[Sign up for early access at waxell.ai](https://waxell.ai)**

---

## Next Steps

- [Phase 4: Go Fully Native](./phase-4-native) -- Manual migration guide (available now)
- [Phase 2: Add Signals](./phase-2-signals) -- Add centralized triggering while you wait
- [Progressive Migration Overview](./overview) -- See all migration phases


--------------------------------------------------------------------------------
# Phase 4: Go Fully Native
URL: https://waxell.ai/docs/migrate/phase-4-native
Description: Manual migration guide for converting existing agents to native Waxell SDK definitions with full governance and durable workflows.
--------------------------------------------------------------------------------

# Phase 4: Go Fully Native

**You are here if**: you want the full power of the Waxell platform -- durable workflows, declarative agent definitions, built-in governance, and production infrastructure. This is a manual migration guide for converting existing agents to native Waxell.

**What you will have after this phase**: agents defined with `@agent`, `@workflow`, `@tool`, and `@router` decorators running on Waxell's durable runtime with checkpoint/resume, full governance, and production backends.

---

## The Mapping

Every pattern in your existing agent code has a native Waxell equivalent:

| Your existing code | Waxell SDK equivalent |
|---|---|
| LLM API calls (`openai.chat.completions.create(...)`) | `await ctx.llm.generate(prompt=..., task=...)` |
| Orchestration logic (function sequences, if/else, loops) | `@workflow` decorator with `ctx.log_step()` |
| External API calls (`requests.get(...)`, SDK clients) | `await ctx.domain("service", "action", ...)` |
| Tool functions | `@tool` decorator |
| Agent class or entry point | `@agent` decorator |
| LLM-based classification / routing | `@router` decorator with decision spec |
| Manual observability (`print`, logging, custom dashboards) | Remove it -- built-in |
| Manual cost tracking | Remove it -- built-in |
| Manual retry logic | Remove it -- built-in via durable workflows |

---

## Step-by-Step: Converting a LangChain Agent

Here is a complete walkthrough converting a LangChain customer support agent to native Waxell.

### Before: LangChain

```python

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.tools import tool as langchain_tool
from langchain.agents import create_tool_calling_agent, AgentExecutor

@langchain_tool
def lookup_customer(customer_id: str) -> str:
    """Look up customer details by ID."""
    import requests
    resp = requests.get(
        f"https://api.myapp.com/customers/{customer_id}",
        headers={"Authorization": f"Bearer {os.environ['APP_API_KEY']}"},
    )
    return resp.json()

@langchain_tool
def search_kb(query: str) -> str:
    """Search the knowledge base for relevant articles."""
    import requests
    resp = requests.get(
        f"https://api.myapp.com/kb/search",
        params={"q": query},
        headers={"Authorization": f"Bearer {os.environ['APP_API_KEY']}"},
    )
    return resp.json()

@langchain_tool
def create_ticket(customer_id: str, category: str, summary: str) -> str:
    """Create a support ticket."""
    import requests
    resp = requests.post(
        "https://api.myapp.com/tickets",
        json={
            "customer_id": customer_id,
            "category": category,
            "summary": summary,
        },
        headers={"Authorization": f"Bearer {os.environ['APP_API_KEY']}"},
    )
    return resp.json()

llm = ChatOpenAI(model="gpt-4o", temperature=0)
prompt = ChatPromptTemplate.from_messages([
    ("system", (
        "You are a customer support agent. Look up the customer, "
        "search the knowledge base, classify the issue, and create a ticket. "
        "Respond with a helpful message."
    )),
    ("human", "Customer {customer_id}: {message}"),
    ("placeholder", "{agent_scratchpad}"),
])

tools = [lookup_customer, search_kb, create_ticket]
agent = create_tool_calling_agent(llm, tools, prompt)
executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

# Run it
result = executor.invoke({
    "customer_id": "C-1234",
    "message": "I was charged twice for my subscription",
})
print(result["output"])
```

### After: Waxell Native

```python
from waxell_sdk import agent, workflow, tool, router, WorkflowContext, RouterContext

@agent(
    name="support-agent",
    description="Handles customer support tickets with classification and response",
    signals=["support_ticket"],
    domains=["customers", "knowledge_base", "tickets"],
)
class SupportAgent:

    # ----------------------------------------------------------------
    # Tools: External integrations become domain calls
    # ----------------------------------------------------------------

    @tool
    async def lookup_customer(self, ctx: WorkflowContext, customer_id: str) -> dict:
        """Look up customer details by ID."""
        return await ctx.domain("customers", "get_details", customer_id=customer_id)

    @tool
    async def search_kb(self, ctx: WorkflowContext, query: str) -> dict:
        """Search the knowledge base for relevant articles."""
        return await ctx.domain("knowledge_base", "search", query=query)

    @tool
    async def create_ticket(
        self,
        ctx: WorkflowContext,
        customer_id: str,
        category: str,
        summary: str,
    ) -> dict:
        """Create a support ticket."""
        return await ctx.domain(
            "tickets", "create",
            customer_id=customer_id,
            category=category,
            summary=summary,
        )

    # ----------------------------------------------------------------
    # Router: LLM-based classification replaces AgentExecutor
    # ----------------------------------------------------------------

    @router("classify_and_handle", decision="classify_ticket")
    async def classify_and_handle(self, ctx: RouterContext) -> dict:
        """Classify the ticket and route to the appropriate handler."""
        # Gather context for the LLM decision
        customer = await ctx.tool(
            "lookup_customer",
            customer_id=ctx.inputs.get("customer_id"),
        )
        ctx.add_signal("customer_info", customer)

        kb_results = await ctx.tool(
            "search_kb",
            query=ctx.inputs.get("message"),
        )
        ctx.add_signal("kb_results", kb_results)

        return await ctx.route()

    # ----------------------------------------------------------------
    # Workflows: Each category gets a dedicated handler
    # ----------------------------------------------------------------

    @workflow("handle_billing")
    async def handle_billing(
        self,
        ctx: WorkflowContext,
        customer_id: str,
        message: str,
    ) -> dict:
        """Handle billing issues."""
        # Generate a response using the LLM router
        response = await ctx.llm.generate(
            prompt=(
                f"Generate a helpful billing support response.\n"
                f"Customer issue: {message}"
            ),
            output_format="json",
            task="billing_response",
        )

        # Create a ticket
        ticket = await ctx.tool(
            "create_ticket",
            customer_id=customer_id,
            category="billing",
            summary=message,
        )
        ctx.log_step("ticket_created", {"ticket_id": ticket.get("id")})

        return {"response": response, "ticket": ticket}

    @workflow("handle_technical")
    async def handle_technical(
        self,
        ctx: WorkflowContext,
        customer_id: str,
        message: str,
    ) -> dict:
        """Handle technical issues."""
        kb_result = await ctx.tool("search_kb", query=message)
        ctx.log_step("kb_searched", {"results_count": len(kb_result)})

        response = await ctx.llm.generate(
            prompt=(
                f"Using this knowledge base article: {kb_result}\n\n"
                f"Generate a helpful technical support response for: {message}"
            ),
            output_format="json",
            task="technical_response",
        )

        ticket = await ctx.tool(
            "create_ticket",
            customer_id=customer_id,
            category="technical",
            summary=message,
        )
        ctx.log_step("ticket_created", {"ticket_id": ticket.get("id")})

        return {"response": response, "ticket": ticket}
```

---

## What Changed and Why

### Direct API calls became domain calls

**Before**: `requests.get(f"https://api.myapp.com/customers/{customer_id}")` with hardcoded URLs and manual auth headers.

**After**: `await ctx.domain("customers", "get_details", customer_id=customer_id)` -- the control plane routes this to your configured domain endpoint with proper authentication, audit logging, and error handling.

### AgentExecutor became a router

**Before**: LangChain's `AgentExecutor` lets the LLM decide which tools to call in a loop. The classification happens implicitly as part of the agent's reasoning.

**After**: The `@router` decorator makes classification explicit. The LLM sees the available capabilities and selects one. The decision is logged, auditable, and governed by policies.

### Manual tool definitions became @tool

**Before**: `@langchain_tool` decorators with string return types and inline HTTP calls.

**After**: `@tool` decorators with typed parameters that delegate to domain calls. The tool is registered with the control plane and tracked in every execution.

### Orchestration logic became workflows

**Before**: The `AgentExecutor` handles all orchestration in a single opaque loop.

**After**: Each handling path (`handle_billing`, `handle_technical`) is an explicit workflow with named steps. Each step is a checkpoint -- if the process crashes after `kb_searched`, it resumes from there.

### Observability code was removed

**Before**: `verbose=True` for console output, plus any custom logging or metrics you added.

**After**: Nothing. Every LLM call, tool invocation, domain call, and workflow step is automatically tracked by the runtime. The control plane dashboard shows everything.

---

## What You Gain at Phase 4

### Durable Workflows with Checkpoint/Resume

Every workflow step is a durable checkpoint. If your process crashes, the `WorkflowEnvelope` resumes execution from the last completed step. No data is lost, no work is repeated.

### Full Governance Lifecycle

The `DynamicPolicyManager` evaluates policies at every hook point:

- **Pre-execution**: Can this agent run right now?
- **Pre-step**: Can this workflow step proceed?
- **Post-step**: Should execution continue based on this step's output?
- **Post-execution**: Record the final outcome for audit

### Infrastructure Package

Native Waxell agents run on production-grade infrastructure:

| Component | Default (local dev) | Production |
|---|---|---|
| State backend | `InMemoryBackend` | `DjangoRuntimeBackend` (PostgreSQL) |
| Token semaphore | `InMemoryTokenSemaphore` | `RedisTokenSemaphore` |
| Task execution | In-process | `CeleryTaskStatusProvider` |
| Governance hooks | None | `PolicyGovernanceHook` |

Switching from local dev to production happens automatically when the `waxell_infra` Django app loads -- no code changes needed.

### Generation Layer

For agents that produce content (emails, reports, summaries), the generation package provides:

- RAG (Retrieval-Augmented Generation) pipelines
- Prompt versioning and A/B testing
- Content synthesis with quality controls
- Model routing per task type

### Multi-Tenancy

Native agents support full tenant isolation:

- Per-tenant policies and model configurations
- Per-tenant billing and cost tracking
- Per-tenant audit trails
- Database-level data isolation

### Management Commands

Operate your agents via Django management commands:

```bash
# List all registered agents
python manage.py agent_list

# Run an agent directly
python manage.py agent_run support-agent

# Interactive agent shell
python manage.py agent_shell support-agent

# Run agent tests
python manage.py agent_test support-agent

# View execution trace
python manage.py agent_trace <execution_id>
```

---

## Migration Checklist

Use this checklist when converting an existing agent:

- [ ] Identify all LLM calls and map them to `ctx.llm.generate()` with appropriate `task` values
- [ ] Identify all external API calls and map them to domain calls via `ctx.domain()`
- [ ] Identify tool functions and convert them to `@tool` decorators
- [ ] Identify the main orchestration flow and convert it to a `@workflow` or `@router`
- [ ] Define the `@agent` decorator with name, description, signals, and domains
- [ ] Remove all manual observability code (logging wrappers, metrics, cost tracking)
- [ ] Remove all manual retry logic (the runtime handles this)
- [ ] Configure domain endpoints in the control plane
- [ ] Register signal types in the control plane
- [ ] Test with `python manage.py agent_test <agent_name>`
- [ ] Verify execution traces with `python manage.py agent_trace <execution_id>`

---

## Next Steps

- [SDK Overview](/sdk/overview) -- Full SDK documentation
- [Workflow Spec](/sdk/workflow-spec) -- Workflow definition reference
- [Runtime Overview](/runtime/overview) -- How the execution engine works
- [Execution Context](/runtime/execution-context) -- ExecutionContext API reference
- [Progressive Migration Overview](./overview) -- See all migration phases