Tracing

Bedrock provides full observability into agent execution through traces and spans. Every agent run creates a detailed record of what happened.

Traces and Spans

Trace: A complete record of an agent run
Span: A single operation within a trace (LLM call, tool execution, etc.)

A Trace (one agent run) contains nested Spans:

run_agent
- turn_0 — openai_api_call (LLM), list_tasks (tool), assistant_message (text)
- turn_1 — anthropic_api_call (LLM), create_task (tool), sleep (tool)
- …additional turns

Span Types

Type	Description
`text`	Text operations, agent messages, thinking
`tool`	Tool invocations
`llm`	LLM API calls (OpenAI, Anthropic)
`audio`	Audio processing (for voice agents)
`function`	Function executions

Listing Traces

Get traces for an agent:

curl -X GET "https://api.bedrock.orinlabs.org/api/tracing/traces/list/?agent=AGENT_ID&limit=10" \
  -H "Authorization: Bearer YOUR_API_KEY"

Response:

{
  "results": [
    {
      "id": "trace-uuid",
      "name": "run_agent",
      "agent": "agent-uuid",
      "started_at": "2024-01-15T10:30:00Z",
      "ended_at": "2024-01-15T10:32:15Z",
      "error": null,
      "metadata": {"agent_id": "agent-uuid"}
    }
  ],
  "pagination": {
    "total_count": 45,
    "limit": 10,
    "offset": 0,
    "has_more": true
  }
}

Query Parameters

Parameter	Description
`agent`	Filter by agent UUID
`name`	Filter by trace name (case-insensitive contains)
`started_after`	Filter by start time
`limit`	Results per page (default 50)
`offset`	Pagination offset
`sort`	Sort field (e.g., `-started_at`, `span_count`)

Getting a Trace with Spans

curl -X GET https://api.bedrock.orinlabs.org/api/tracing/traces/TRACE_ID/ \
  -H "Authorization: Bearer YOUR_API_KEY"

Response:

{
  "id": "trace-uuid",
  "name": "run_agent",
  "started_at": "2024-01-15T10:30:00Z",
  "ended_at": "2024-01-15T10:32:15Z",
  "spans": [
    {
      "id": "span-uuid",
      "name": "anthropic_api_call",
      "span_type": "llm",
      "started_at": "2024-01-15T10:30:01Z",
      "ended_at": "2024-01-15T10:30:03Z",
      "input_text": "{\"model\": \"claude-sonnet-4\", ...}",
      "output_text": "tokens_in=1500, tokens_out=200",
      "metadata": {
        "model": "claude-sonnet-4",
        "provider": "anthropic",
        "llm_cost": {
          "input_tokens": 1500,
          "output_tokens": 200,
          "cached_tokens": 1200,
          "total_cost_usd": 0.0045,
          "model": "claude-sonnet-4"
        }
      }
    },
    {
      "id": "span-uuid-2",
      "name": "list_tasks",
      "span_type": "tool",
      "input_text": "{\"args\": {\"limit\": 10}, \"reasoning\": \"Checking current tasks\"}",
      "output_text": "Found 3 tasks:\n1. Review proposal..."
    }
  ]
}

Span Details

Each span contains:

Field	Description
`name`	Operation name
`span_type`	Type (text, tool, llm, audio, function)
`parent`	Parent span UUID (for nesting)
`started_at`	When the operation started
`ended_at`	When it completed
`input_text`	Input to the operation
`output_text`	Output/result
`error`	Error message if failed
`metadata`	Additional structured data

LLM Cost Tracking

LLM spans include cost metadata:

{
  "llm_cost": {
    "input_tokens": 1500,
    "output_tokens": 200,
    "cached_tokens": 1200,
    "total_cost_usd": 0.0045,
    "model": "claude-sonnet-4",
    "provider": "anthropic"
  }
}

Creating Custom Traces

You can create traces programmatically for custom operations:

# Create a trace
curl -X POST https://api.bedrock.orinlabs.org/api/tracing/traces/ \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "custom_operation",
    "agent": "AGENT_ID",
    "metadata": {"custom_field": "value"}
  }'

# Create a span within the trace
curl -X POST https://api.bedrock.orinlabs.org/api/tracing/spans/ \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "trace": "TRACE_ID",
    "name": "my_operation",
    "span_type": "function",
    "input_text": "Starting operation..."
  }'

# End the span
curl -X POST https://api.bedrock.orinlabs.org/api/tracing/spans/SPAN_ID/end/ \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "output_text": "Operation completed successfully"
  }'

# End the trace
curl -X POST https://api.bedrock.orinlabs.org/api/tracing/traces/TRACE_ID/end/ \
  -H "Authorization: Bearer YOUR_API_KEY"

Usage Records

When a trace ends, Bedrock automatically creates an AgentUsage record that aggregates all LLM costs from that trace. Query these via the agent usage endpoint:

curl -X GET "https://api.bedrock.orinlabs.org/api/cloud/agents/AGENT_ID/usage/" \
  -H "Authorization: Bearer YOUR_API_KEY"

Flagging Spans for Review

Mark spans that need attention (e.g., incorrect agent responses):

curl -X POST https://api.bedrock.orinlabs.org/api/tracing/spans/SPAN_ID/flag/ \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "correction": "The agent should have asked for clarification instead of assuming"
  }'

Flagged spans can be used for evaluation and fine-tuning. Remove a flag:

curl -X POST https://api.bedrock.orinlabs.org/api/tracing/spans/SPAN_ID/unflag/ \
  -H "Authorization: Bearer YOUR_API_KEY"

Evals (Flagged Span Corrections)

Flagged spans are stored as eval test cases that you can browse and curate via /api/tracing/evals/:

Method	Endpoint	Description
`GET`	`/api/tracing/evals/`	List eval cases (flagged spans + corrections)
`POST`	`/api/tracing/evals/`	Create an eval case manually
`GET`	`/api/tracing/evals/{eval_id}/`	Get eval case
`PATCH`	`/api/tracing/evals/{eval_id}/`	Update correction text
`DELETE`	`/api/tracing/evals/{eval_id}/`	Delete eval case

For running full scenario-based evaluations, see Evaluations.

Debugging with Traces

Common debugging patterns:

Find Failed Runs

curl -X GET "https://api.bedrock.orinlabs.org/api/tracing/traces/list/?agent=AGENT_ID" \
  -H "Authorization: Bearer YOUR_API_KEY"

Look for traces with non-null error field.

Analyze Tool Usage

Filter spans by type to see which tools were called:

curl -X GET "https://api.bedrock.orinlabs.org/api/tracing/spans/?trace=TRACE_ID" \
  -H "Authorization: Bearer YOUR_API_KEY"

Check Token Usage

LLM spans contain detailed token counts in metadata for cost analysis.

Best Practices

Review Failed Traces

Check the error field to find and fix issues.

Monitor Costs

Use LLM span metadata to track spending.

Flag Bad Outputs

Use the flag endpoint to mark incorrect responses.

Trace Custom Ops

Create traces for operations outside agent runs.

Get Started

Concepts

Tracing

Tracing

Traces and Spans

Span Types

Listing Traces

Query Parameters

Getting a Trace with Spans

Span Details

LLM Cost Tracking

Creating Custom Traces

Usage Records

Flagging Spans for Review

Evals (Flagged Span Corrections)

Debugging with Traces

Find Failed Runs

Analyze Tool Usage

Check Token Usage

Best Practices

Review Failed Traces

Monitor Costs

Flag Bad Outputs

Trace Custom Ops

​Tracing

​Traces and Spans

​Span Types

​Listing Traces

​Query Parameters

​Getting a Trace with Spans

​Span Details

​LLM Cost Tracking

​Creating Custom Traces

​Usage Records

​Flagging Spans for Review

​Evals (Flagged Span Corrections)

​Debugging with Traces

​Find Failed Runs

​Analyze Tool Usage

​Check Token Usage

​Best Practices

Review Failed Traces

Monitor Costs

Flag Bad Outputs

Trace Custom Ops

Tracing

Traces and Spans

Span Types

Listing Traces

Query Parameters

Getting a Trace with Spans

Span Details

LLM Cost Tracking

Creating Custom Traces

Usage Records

Flagging Spans for Review

Evals (Flagged Span Corrections)

Debugging with Traces

Find Failed Runs

Analyze Tool Usage

Check Token Usage

Best Practices