Skip to main content

​
Enabling tracing

OpenTelemetry support is enabled by default via the otel feature. Configure your OTLP endpoint using standard environment variables (for example OTEL_EXPORTER_OTLP_ENDPOINT) to export traces — no endpoint set means no OTel layer is installed. Before you set an endpoint, review Content recording and Security and Data Handling, because exported spans can include prompt, response, and tool content. AURA emits spans using the OpenInference semantic convention (llm.*, tool.*, input.*, output.*) rather than the gen_ai.* conventions. Any gen_ai.* attributes from underlying provider libraries (Rig.rs) are automatically translated to OpenInference equivalents at export time. This makes AURA traces natively compatible with Phoenix and other OpenInference-aware observability tools. Set PHOENIX_PROJECT_NAME to choose which Phoenix project receives your traces. It sets the resource attribute openinference.project.name, which is the only attribute Phoenix uses to route a trace to a project; Phoenix ignores service.name and OTEL_SERVICE_NAME for routing. When you leave PHOENIX_PROJECT_NAME unset, AURA sets openinference.project.name to the service name instead (OTEL_SERVICE_NAME, then aura), so your traces are grouped under a project of that name rather than one you chose. In a multi-replica deployment, set the same PHOENIX_PROJECT_NAME value on every replica or pod in an environment so their traces group into one Phoenix project; do not template it from a per-pod value such as the pod name or hostname, or each pod’s traces land in a separate project. See the CLI reference for the full OpenTelemetry environment-variable table.

​
Trace structure

Every request produces two traces:
  1. HTTP trace — covers the request/response lifecycle
  2. Agent trace — covers the LLM/tool execution
The agent trace is rooted at agent.stream with parent: None so Phoenix sees it as an independent trace root with all LLM I/O attributes.

​
HTTP trace (both modes)

​
Single-agent mode

​
Orchestration mode

RAG-enabled agents emit an additional vector.search (RETRIEVER) span under execute_tool, alongside mcp.tool_call, wherever the agent calls a vector-search tool.

​
Span attributes

When OTEL_RECORD_CONTENT=true, AURA also records the assembled system prompt on the agent entry-point spans (agent.stream, agent.prompt, agent.chat) and on the orchestration.planning and orchestration.worker spans. See Content recording.

​
Agent root (agent.stream)

user.id, session.id, metadata, input.value, output.value, aura.version, aura.mode. aura.version and aura.mode are stamped on the streaming root so you can filter traces by release and by serving path. service.version is set on the tracer resource.

​
Agent and LLM spans

AURA emits the following OpenInference attributes on the agent and LLM spans so Phoenix renders them:
  • llm.provider is recorded beside llm.system. Phoenix’s pricing table matches on provider and model together.
  • agent.name is recorded on the agent.turn span, translated from gen_ai.agent.name, whenever the agent, coordinator, or worker is built with a name. It is what stops Phoenix from showing “Unnamed Agent”.
  • llm.invocation_parameters comes from the effective LLM config.
  • llm.tools.{i}.tool.json_schema holds the MCP tool schemas each agent advertises, filtered per worker.
  • user.id comes from the request user field, and metadata comes from the request metadata map. See the web server reference for those request fields.

​
Orchestration spans

Per-turn prompt and completion token counts (llm.token_count.prompt and llm.token_count.completion) live on each agent.turn span, recorded by Rig. From those turn counts:
  • Phoenix rolls the turn counts up as the single cost anchor, so every aggregate (per worker, per phase, per trace) is computed from the turns.
  • AURA’s agent-level spans carry no token counts, so they price at zero and leave the agent.turn totals intact.
  • Per-worker LLM overrides price at their own rate because provider and model are stamped on each worker’s turn spans, through the existing [orchestration.worker.<name>.llm] config; no new config is needed.

​
Retriever spans

vector.search spans export as RETRIEVER kind and carry retrieval.documents.{i}.* attributes: the documents returned by a vector search. Their content follows the OTEL_RECORD_CONTENT gate like other recorded content.

​
Tool spans (execute_tool)

For HITL-gated tool calls, execute_tool carries a decision_id attribute whose value is the UUID of the approval decision that gated the call. This is the same decision_id that appears on the approval webhook payload, the SSE lifecycle events, and the approval ingress endpoint (see Human-in-the-Loop Approval Gates). The attribute is stamped before the approval outcome is known, so every gated call carries it regardless of outcome — approved, denied, timed_out, cancelled, or errored. It is stamped for gated calls only; ungated tool calls carry no decision_id attribute. decision_id sits on the execute_tool (TOOL) parent, not on the nested mcp.tool_call child, where the tool’s own status and errors land.

​
OpenInference span kinds

Each span’s kind follows from its span name. The agent-level spans now map to LLM so Phoenix renders their tools and invocation parameters, though they carry no token counts (see the cost note above). chat, agent.prompt, and agent.chat are non-streaming entry points; the trace diagrams above cover the streaming path, which is the one AURA’s HTTP/CLI backends use.
llm.token_count.total and llm.token_count.tool_completion are no longer emitted on any AURA-owned span. The per-turn counts llm.token_count.prompt and llm.token_count.completion on each agent.turn span replace them. If you have dashboards or alerts keyed on those two attributes on AURA spans, point them at llm.token_count.prompt and llm.token_count.completion on agent.turn before upgrading.

​
Span parenting

  • agent.stream is created with parent: None to break the link from the HTTP handler trace, making it an independent trace root in Phoenix.
  • The producer task is instrumented with agent.stream so Rig’s agent.turn becomes a direct child.
  • In orchestration mode, the orchestrator instruments its spawned task with the agent.stream span so all orchestration child spans nest under the trace root.
  • Tool execution propagates the current span into its background task so mcp.tool_call nests under Rig’s execute_tool.

​
Content recording

OTEL_RECORD_CONTENT governs only AURA-owned spans. When false (the default), it omits prompt and completion text, and tool arguments and results. When true, it includes them, along with the assembled system prompt (the preamble, which includes the orchestration instructions and the skills catalog). AURA truncates all of this content to OTEL_CONTENT_MAX_LENGTH (default 1000 bytes, rounded down to the nearest UTF-8 boundary). At export, the recorded system prompt appears as leading system entries in the input messages (llm.input_messages) in Phoenix and other OpenInference viewers. AURA translates the raw gen_ai.system_instructions key into those leading system entries rather than exporting it directly. Rig-owned spans (chat, chat_streaming, agent.turn, and execute_tool) record content through gen_ai.* attributes. The OpenInference exporter translates and exports these attributes regardless of OTEL_RECORD_CONTENT. Content is truncated to OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT (default 65,536 bytes, rounded down to the nearest UTF-8 boundary), not OTEL_CONTENT_MAX_LENGTH. Because of this, OTEL_RECORD_CONTENT=false is not a complete content-suppression control. See Security and Data Handling before you enable OTLP export. On worker spans, llm.prompt_template.template is static and always recorded. llm.prompt_template.variables carries request content, so it follows the OTEL_RECORD_CONTENT gate and the same truncation as other recorded content.

​
Error spans

Every error carries an exception span event with an exception.message attribute, which is the format Phoenix’s exception view expects.

​
Known limitations

Tool errors are only recorded on the mcp.tool_call child span, not on Rig’s execute_tool parent. This is intentional: mcp.tool_call is the canonical TOOL span for Phoenix.