> ## Documentation Index > Fetch the complete documentation index at: https://docs.bedrock.orinlabs.org/llms.txt > Use this file to discover all available pages before exploring further. # Tracing > Observability and debugging with traces and spans # Tracing Bedrock provides full **observability** into agent execution through traces and spans. Every agent run creates a detailed record of what happened. ## Traces and Spans * **Trace**: A complete record of an agent run * **Span**: A single operation within a trace (LLM call, tool execution, etc.) A **Trace** (one agent run) contains nested **Spans**: * **run\_agent** * **turn\_0** — `openai_api_call` (LLM), `list_tasks` (tool), `assistant_message` (text) * **turn\_1** — `anthropic_api_call` (LLM), `create_task` (tool), `sleep` (tool) * ...additional turns ## Span Types | Type | Description | | ---------- | ----------------------------------------- | | `text` | Text operations, agent messages, thinking | | `tool` | Tool invocations | | `llm` | LLM API calls (OpenAI, Anthropic) | | `audio` | Audio processing (for voice agents) | | `function` | Function executions | ## Listing Traces Get traces for an agent: ```bash theme={null} curl -X GET "https://api.bedrock.orinlabs.org/api/tracing/traces/list/?agent=AGENT_ID&limit=10" \ -H "Authorization: Bearer YOUR_API_KEY" ``` Response: ```json theme={null} { "results": [ { "id": "trace-uuid", "name": "run_agent", "agent": "agent-uuid", "started_at": "2024-01-15T10:30:00Z", "ended_at": "2024-01-15T10:32:15Z", "error": null, "metadata": {"agent_id": "agent-uuid"} } ], "pagination": { "total_count": 45, "limit": 10, "offset": 0, "has_more": true } } ``` ### Query Parameters | Parameter | Description | | --------------- | ------------------------------------------------ | | `agent` | Filter by agent UUID | | `name` | Filter by trace name (case-insensitive contains) | | `started_after` | Filter by start time | | `limit` | Results per page (default 50) | | `offset` | Pagination offset | | `sort` | Sort field (e.g., `-started_at`, `span_count`) | ## Getting a Trace with Spans ```bash theme={null} curl -X GET https://api.bedrock.orinlabs.org/api/tracing/traces/TRACE_ID/ \ -H "Authorization: Bearer YOUR_API_KEY" ``` Response: ```json theme={null} { "id": "trace-uuid", "name": "run_agent", "started_at": "2024-01-15T10:30:00Z", "ended_at": "2024-01-15T10:32:15Z", "spans": [ { "id": "span-uuid", "name": "anthropic_api_call", "span_type": "llm", "started_at": "2024-01-15T10:30:01Z", "ended_at": "2024-01-15T10:30:03Z", "input_text": "{\"model\": \"claude-sonnet-4\", ...}", "output_text": "tokens_in=1500, tokens_out=200", "metadata": { "model": "claude-sonnet-4", "provider": "anthropic", "llm_cost": { "input_tokens": 1500, "output_tokens": 200, "cached_tokens": 1200, "total_cost_usd": 0.0045, "model": "claude-sonnet-4" } } }, { "id": "span-uuid-2", "name": "list_tasks", "span_type": "tool", "input_text": "{\"args\": {\"limit\": 10}, \"reasoning\": \"Checking current tasks\"}", "output_text": "Found 3 tasks:\n1. Review proposal..." } ] } ``` ## Span Details Each span contains: | Field | Description | | ------------- | --------------------------------------- | | `name` | Operation name | | `span_type` | Type (text, tool, llm, audio, function) | | `parent` | Parent span UUID (for nesting) | | `started_at` | When the operation started | | `ended_at` | When it completed | | `input_text` | Input to the operation | | `output_text` | Output/result | | `error` | Error message if failed | | `metadata` | Additional structured data | ## LLM Cost Tracking LLM spans include cost metadata: ```json theme={null} { "llm_cost": { "input_tokens": 1500, "output_tokens": 200, "cached_tokens": 1200, "total_cost_usd": 0.0045, "model": "claude-sonnet-4", "provider": "anthropic" } } ``` ## Creating Custom Traces You can create traces programmatically for custom operations: ```bash theme={null} # Create a trace curl -X POST https://api.bedrock.orinlabs.org/api/tracing/traces/ \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "custom_operation", "agent": "AGENT_ID", "metadata": {"custom_field": "value"} }' # Create a span within the trace curl -X POST https://api.bedrock.orinlabs.org/api/tracing/spans/ \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "trace": "TRACE_ID", "name": "my_operation", "span_type": "function", "input_text": "Starting operation..." }' # End the span curl -X POST https://api.bedrock.orinlabs.org/api/tracing/spans/SPAN_ID/end/ \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "output_text": "Operation completed successfully" }' # End the trace curl -X POST https://api.bedrock.orinlabs.org/api/tracing/traces/TRACE_ID/end/ \ -H "Authorization: Bearer YOUR_API_KEY" ``` ## Usage Records When a trace ends, Bedrock automatically creates an `AgentUsage` record that aggregates all LLM costs from that trace. Query these via the agent usage endpoint: ```bash theme={null} curl -X GET "https://api.bedrock.orinlabs.org/api/cloud/agents/AGENT_ID/usage/" \ -H "Authorization: Bearer YOUR_API_KEY" ``` ## Flagging Spans for Review Mark spans that need attention (e.g., incorrect agent responses): ```bash theme={null} curl -X POST https://api.bedrock.orinlabs.org/api/tracing/spans/SPAN_ID/flag/ \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "correction": "The agent should have asked for clarification instead of assuming" }' ``` Flagged spans can be used for evaluation and fine-tuning. Remove a flag: ```bash theme={null} curl -X POST https://api.bedrock.orinlabs.org/api/tracing/spans/SPAN_ID/unflag/ \ -H "Authorization: Bearer YOUR_API_KEY" ``` ## Evals (Flagged Span Corrections) Flagged spans are stored as **eval test cases** that you can browse and curate via `/api/tracing/evals/`: | Method | Endpoint | Description | | -------- | ------------------------------- | --------------------------------------------- | | `GET` | `/api/tracing/evals/` | List eval cases (flagged spans + corrections) | | `POST` | `/api/tracing/evals/` | Create an eval case manually | | `GET` | `/api/tracing/evals/{eval_id}/` | Get eval case | | `PATCH` | `/api/tracing/evals/{eval_id}/` | Update correction text | | `DELETE` | `/api/tracing/evals/{eval_id}/` | Delete eval case | For running full scenario-based evaluations, see [Evaluations](/concepts/evals). ## Debugging with Traces Common debugging patterns: ### Find Failed Runs ```bash theme={null} curl -X GET "https://api.bedrock.orinlabs.org/api/tracing/traces/list/?agent=AGENT_ID" \ -H "Authorization: Bearer YOUR_API_KEY" ``` Look for traces with non-null `error` field. ### Analyze Tool Usage Filter spans by type to see which tools were called: ```bash theme={null} curl -X GET "https://api.bedrock.orinlabs.org/api/tracing/spans/?trace=TRACE_ID" \ -H "Authorization: Bearer YOUR_API_KEY" ``` ### Check Token Usage LLM spans contain detailed token counts in metadata for cost analysis. ## Best Practices Check the `error` field to find and fix issues. Use LLM span metadata to track spending. Use the flag endpoint to mark incorrect responses. Create traces for operations outside agent runs.