Skip to main content

​
Tracing

Bedrock provides full observability into agent execution through traces and spans. Every agent run creates a detailed record of what happened.

​
Traces and Spans

  • Trace: A complete record of an agent run
  • Span: A single operation within a trace (LLM call, tool execution, etc.)
Trace (agent run)
├── Span: run_agent
│   ├── Span: turn_0
│   │   ├── Span: openai_api_call (LLM)
│   │   ├── Span: list_tasks (tool)
│   │   └── Span: assistant_message (text)
│   ├── Span: turn_1
│   │   ├── Span: anthropic_api_call (LLM)
│   │   ├── Span: create_task (tool)
│   │   └── Span: sleep (tool)

​
Span Types

TypeDescription
textText operations, agent messages, thinking
toolTool invocations
llmLLM API calls (OpenAI, Anthropic)
audioAudio processing (for voice agents)
functionFunction executions

​
Listing Traces

Get traces for an agent: 
curl -X GET "https://api.bedrock.orinlabs.org/api/tracing/traces/list/?agent=AGENT_ID&limit=10" \
  -H "Authorization: Bearer YOUR_API_KEY"
Response: 
{
  "results": [
    {
      "id": "trace-uuid",
      "name": "run_agent",
      "agent": "agent-uuid",
      "started_at": "2024-01-15T10:30:00Z",
      "ended_at": "2024-01-15T10:32:15Z",
      "error": null,
      "metadata": {"agent_id": "agent-uuid"}
    }
  ],
  "pagination": {
    "total_count": 45,
    "limit": 10,
    "offset": 0,
    "has_more": true
  }
}

​
Query Parameters

ParameterDescription
agentFilter by agent UUID
product_idFilter by product UUID
nameFilter by trace name
started_afterFilter by start time
limitResults per page (default 50)
offsetPagination offset
sortSort field (e.g., -started_at)

​
Getting a Trace with Spans

curl -X GET https://api.bedrock.orinlabs.org/api/tracing/traces/TRACE_ID/ \
  -H "Authorization: Bearer YOUR_API_KEY"
Response: 
{
  "id": "trace-uuid",
  "name": "run_agent",
  "started_at": "2024-01-15T10:30:00Z",
  "ended_at": "2024-01-15T10:32:15Z",
  "spans": [
    {
      "id": "span-uuid",
      "name": "anthropic_api_call",
      "span_type": "llm",
      "started_at": "2024-01-15T10:30:01Z",
      "ended_at": "2024-01-15T10:30:03Z",
      "input_text": "{\"model\": \"claude-sonnet-4\", ...}",
      "output_text": "tokens_in=1500, tokens_out=200",
      "metadata": {
        "model": "claude-sonnet-4",
        "provider": "anthropic",
        "llm_cost": {
          "input_tokens": 1500,
          "output_tokens": 200,
          "cached_tokens": 1200,
          "total_cost_usd": 0.0045,
          "model": "claude-sonnet-4"
        }
      }
    },
    {
      "id": "span-uuid-2",
      "name": "list_tasks",
      "span_type": "tool",
      "input_text": "{\"args\": {\"limit\": 10}, \"reasoning\": \"Checking current tasks\"}",
      "output_text": "Found 3 tasks:\n1. Review proposal..."
    }
  ]
}

​
Span Details

Each span contains:
FieldDescription
nameOperation name
span_typeType (text, tool, llm, audio, function)
parentParent span UUID (for nesting)
started_atWhen the operation started
ended_atWhen it completed
input_textInput to the operation
output_textOutput/result
errorError message if failed
metadataAdditional structured data

​
LLM Cost Tracking

LLM spans include cost metadata: 
{
  "llm_cost": {
    "input_tokens": 1500,
    "output_tokens": 200,
    "cached_tokens": 1200,
    "total_cost_usd": 0.0045,
    "model": "claude-sonnet-4",
    "provider": "anthropic"
  }
}

​
Creating Custom Traces

You can create traces programmatically for custom operations: 
# Create a trace
curl -X POST https://api.bedrock.orinlabs.org/api/tracing/traces/ \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "custom_operation",
    "agent": "AGENT_ID",
    "metadata": {"custom_field": "value"}
  }'

# Create a span within the trace
curl -X POST https://api.bedrock.orinlabs.org/api/tracing/spans/ \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "trace": "TRACE_ID",
    "name": "my_operation",
    "span_type": "function",
    "input_text": "Starting operation..."
  }'

# End the span
curl -X POST https://api.bedrock.orinlabs.org/api/tracing/spans/SPAN_ID/end/ \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "output_text": "Operation completed successfully"
  }'

# End the trace
curl -X POST https://api.bedrock.orinlabs.org/api/tracing/traces/TRACE_ID/end/ \
  -H "Authorization: Bearer YOUR_API_KEY"

​
Usage Records

When a trace ends, Bedrock automatically creates an AgentUsage record that aggregates all LLM costs from that trace. Query these via the agent usage endpoint: 
curl -X GET "https://api.bedrock.orinlabs.org/api/cloud/agents/AGENT_ID/usage/" \
  -H "Authorization: Bearer YOUR_API_KEY"

​
Flagging Spans for Review

Mark spans that need attention (e.g., incorrect agent responses): 
curl -X POST https://api.bedrock.orinlabs.org/api/tracing/spans/SPAN_ID/flag/ \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "correction": "The agent should have asked for clarification instead of assuming"
  }'
Flagged spans can be used for evaluation and fine-tuning.

​
Debugging with Traces

Common debugging patterns:

​
Find Failed Runs

curl -X GET "https://api.bedrock.orinlabs.org/api/tracing/traces/list/?agent=AGENT_ID" \
  -H "Authorization: Bearer YOUR_API_KEY"
Look for traces with non-null error field.

​
Analyze Tool Usage

Filter spans by type to see which tools were called: 
curl -X GET "https://api.bedrock.orinlabs.org/api/tracing/spans/?trace=TRACE_ID" \
  -H "Authorization: Bearer YOUR_API_KEY"

​
Check Token Usage

LLM spans contain detailed token counts in metadata for cost analysis.

​
Best Practices

Review Failed Traces

Check the error field to find and fix issues.

Monitor Costs

Use LLM span metadata to track spending.

Flag Bad Outputs

Use the flag endpoint to mark incorrect responses.

Trace Custom Ops

Create traces for operations outside agent runs.