LLM Observability Implementation Guide

Overview

Make customer-facing LLM workflows explainable in production. Connect each response to its request, model and configuration, retrieval and tools, cost estimate, failures, and latency.

User request
     │
     ▼
Application request span ──► retrieval ──► model call ──► tool calls ──► response
     │                            │              │             │
     └─ correlated logs ──────────┴──────────────┴─────────────┴──► Tsuga
                                      traces, logs, metrics

LLM Observability is trace-first. When enabled, validate in LLM Explorer and trace detail that the summary shows model/provider, total calls, tokens, and errors. Confirm feature-flag availability before committing it to the customer.

[!Warning] The OpenTelemetry GenAI semantic conventions are still evolving, and OpenTelemetry, OpenLLMetry, and OpenInference emit different variants in practice. Use the version emitted by the chosen instrumentation library, preserve source fields, and pin or review upgrades deliberately. Do not independently rename fields to chase newer convention drafts.

This path assumes the application already exports traces with stable resource identity. Establish the request span that parents every model call, the span metrics behind level 7’s monitors, and the service.name/service.version convention before starting this path. The investigations below also require structured logging and trace correlation.

Before starting

• Agree the application, customer journey, model providers, environments, and owners in scope.
• Define the data policy before enabling content capture. Prompts, completions, retrieved documents, tool arguments, and tool results can contain customer data, credentials, or regulated information.
• Set stable resource identity on traces, logs, and metrics: service.name, service.version, and deployment.environment.name. Add the platform-owned Kubernetes, host, or cloud context where applicable.
• Decide the intended model of each workflow: direct chat, retrieval-augmented generation, tool-using agent, asynchronous batch workflow, or a combination.
• Identify the provider/framework instrumentation in use and the format it emits: OpenTelemetry GenAI, OpenLLMetry, OpenInference, or a native framework integration. Add manual spans only for application-specific work the integration cannot observe.

References: OpenTelemetry GenAI semantic conventions.

Level 1 - Choose the instrumentation and normalize it deliberately

Confirm the runtime, package version, provider, and framework before choosing an integration. Prefer supported provider or framework instrumentation, such as OpenTelemetry GenAI, OpenLLMetry, or OpenInference, over hand-written model-call spans.

Compare a harmless sample trace with the fields Tsuga needs. The normalized model-call span must contain:

• Provider: gen_ai.provider.name.
• Model: gen_ai.request.model; also capture gen_ai.response.model when the provider returns the served model.
• Operation: gen_ai.operation.name.
• Token use: gen_ai.usage.input_tokens and gen_ai.usage.output_tokens.
• Permitted content only: gen_ai.system_instructions, gen_ai.input.messages, and gen_ai.output.messages, and only when the approved capture policy allows them.
• Errors: standard span status ERROR and error.type when an error is recorded.

Normalize older or alternative conventions in a controlled Collector path. Preserve source attributes, record the mapping, and validate the normalized trace before creating dashboards or monitors.

The current internal direction is to use the OpenTelemetry Collector GenAI normalizer for OpenLLMetry and OpenInference formats where needed. Treat that as a compatibility assessment, not a universal default: verify the exact Collector version, processor support, and resulting fields before rollout.

Exit criteria: one representative trace is recognized as an LLM workflow in Tsuga. Its source format and any mapping are documented. Original source attributes remain inspectable. The trace displays the provider, model, operation, and token information expected by the product.

References: OpenTelemetry GenAI semantic conventions for the fields each instrumentation can emit.

Level 2 - Trace the customer request and the model call

Start with one complete customer action, such as “submit support question” or “generate a product summary.” Its application request span must remain the parent of the model-provider client span. This connects the model invocation to the user-facing route, authentication decision, business workflow, and downstream dependencies.

The model call is a meaningful remote operation. Ensure its span has:

• Duration.
• Status and provider failure details.
• Retry behaviour.
• Its causal position in the customer request.

Do not create a second manual span when a provider or framework integration already creates an equivalent one. Validate the emitted span first, then add only missing application-level operations.

For streaming responses, when the product displays partial output:

• Measure time to first token and total completion time as separate user experiences.
• Keep cancellation, timeout, and retry outcomes visible.
• Check for expensive failed attempts hidden behind an apparently successful final answer.

Validate with one harmless test request:

1. The application request trace is visible in Tsuga.
2. The model call is a child of, or otherwise causally connected to, the request.
3. The trace shows the request route or named workflow, provider call, status, and duration.
4. A provider error, timeout, or cancellation produces an actionable failure signal.

Exit criteria: an investigator can open one user request and see the model call that produced the response, its duration, and its outcome.

Level 3 - Capture model identity, operation, and token usage

Use bounded GenAI fields emitted by the integration for operation, provider, requested and returned model, and token use. Common fields include gen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model, gen_ai.response.model, gen_ai.usage.input_tokens, and gen_ai.usage.output_tokens.

Use spans for individual evidence and metrics for aggregate latency, error rate, request count, tokens, and estimated cost. Limit metric dimensions to model, provider, operation, environment, and a fixed workflow name. Exclude user, conversation, prompt, response, request, and trace identifiers.

Record a business-level workflow name only when it is from a short, controlled vocabulary, for example support_answer, document_summary, or product_search. Do not use a customer-created assistant name, a free-form prompt template, or a generated session name as a metric attribute.

Validate with one representative model call:

1. The trace shows the provider, requested model, returned model when available, and operation.
2. The model call can be grouped by provider, model, operation, and controlled workflow.
3. Input and output token counts are visible for the call.
4. Token use, duration, request count, error rate, and estimated cost can be monitored by the approved bounded dimensions.
5. No user, conversation, prompt, response, request, or trace identifier is used as a metric attribute.

Exit criteria: each model call can be grouped by provider, model, operation, and controlled workflow. Token usage is visible for representative success and failure paths. All metric dimensions have bounded values.

References: OpenTelemetry GenAI attributes.

Level 4 - Add policy-safe prompt and response visibility

Model identity and token counts describe the operation. Content makes an answer explainable. Enable prompt, completion, system-instruction, and tool content only after the customer has agreed exactly what may be retained, for how long, and who may view it. Retain only the fields needed to investigate the customer journey.

Use the GenAI message and retrieval fields when the integration supports them, but default to one of these safer patterns:

• Capture no content and retain only model, token, status, and workflow metadata.
• Capture a redacted or truncated prompt and completion for selected non-sensitive environments or allowlisted workflows.
• Capture stable prompt-template or policy version identifiers, rather than raw prompts, for every environment.
• Hash or replace sensitive fields at the source before telemetry is emitted. Use Tsuga native Sensitive Data Scanner or Collector redaction as defense in depth.

Validate with both an approved benign example and a deliberately sensitive test fixture. Confirm that useful approved context remains available while the prohibited value is absent from logs, spans, and metric attributes.

Exit criteria: the customer has approved a documented capture policy. Every captured content field has a purpose and retention owner. Redaction is validated with a negative test.

References: OpenTelemetry GenAI attributes.

Level 5 - Instrument retrieval-augmented generation

Trace RAG retrieval between the request and model call. Record query preparation, the vector or search-store request, optional reranking, selected-document count, and the consuming model call. This separates retrieval evidence from model behavior.

Record an approved, bounded document identifier or source collection. Keep raw documents, queries, tenant IDs, and unique document IDs out of metric dimensions. Use spans or logs for individual evidence and metrics for aggregate retrieval latency, error rate, empty-result rate, and a bounded result-count distribution.

Validate an expected-answer case and an empty or poor-retrieval case:

1. The trace shows whether retrieval occurred and which downstream model call used its result.
2. Retrieval duration, errors, empty-result rate, and selected-document count can be monitored.
3. Empty, failed, and slow retrieval are distinguishable.
4. Document and query content follows the approved capture policy and is not used as a metric dimension.

Exit criteria: retrieval is visible as a causal part of the request. Empty, failed, and slow retrieval are distinguishable. Document or query content follows the approved capture policy.

References: OpenTelemetry GenAI attributes.

Level 6 - Trace agent decisions and tool execution

Model tool workflows as request, model decision, tool execution, optional follow-up model call, and response. Give duration-bearing, causally relevant tool calls spans. Record point-in-time decisions as structured logs or span events.

Record a tool name and bounded category. Capture arguments and results only under the approved policy, never as metric dimensions. Preserve trace propagation across queues, workers, and callbacks. Keep queue wait separate from user-facing operation duration.

Validate one successful tool workflow, a tool failure, and a tool timeout:

1. The trace connects the model decision, tool execution, any follow-up model call, and final response.
2. Tool duration, failure rate, timeout outcomes, and bounded tool category can be monitored.
3. A failure identifies the tool or dependency that failed, rather than only the final model response.
4. Tool arguments and results follow the approved capture policy and are not used as metric dimensions.

Exit criteria: every material tool execution is attributable to its model workflow. Tool failures are visible at their source. Asynchronous execution preserves the intended causal relationship.

References: OpenTelemetry GenAI attributes.

Level 7 - Operationalize quality, cost, and response

Start with request success and latency, model errors and duration, token use by approved model and workflow, retrieval empty-result or error rate, and tool failure rate. Use span-derived metrics when the trace represents the operation. Add custom metrics only for aggregate product state that traces cannot express.

Treat cost as an estimate unless it comes from a provider billing source. Show token volume by provider, model, and controlled workflow. Apply a documented price table outside the telemetry stream, review it when the provider or model changes, and label every estimate.

Create monitors only for conditions with a defined response: sustained model-provider errors, latency that breaks the user experience, a material increase in empty retrieval, tool failures that block customer actions, and unexpected token growth on an approved workflow. Pair each monitor with a named owner and a runbook that begins from the trace, logs, and relevant infrastructure view.

Validate the operational view:

1. Request success and latency, model errors and duration, token use, retrieval health, and tool failures are visible for the agreed workflow.
2. Token use can be broken down by approved provider, model, and controlled workflow without exposing content.
3. Each monitor has a defined response, named owner, runbook, and tested notification route.
4. The first-response path starts from the relevant trace, logs, and infrastructure view.

Exit criteria: the team can detect and investigate a degraded customer LLM workflow, distinguish model, retrieval, tool, and application failures, and explain token-volume changes without exposing prohibited content.

Troubleshooting path

Start with a known customer action. Verify the request trace and connected model span, including provider, model, token, status, and duration. Check retrieval and tool spans only for workflows that use them. Check capture policy and integration settings before changing redaction. Repair propagation across queues or workers in the application. Verify the underlying trace and bounded metric dimensions before changing a dashboard or monitor.

Completion criterion

An investigator opens one known customer request in Tsuga and reads, from the trace alone, which model and provider served it, its token use and outcome, which retrieval and tool calls influenced it, and where the workflow failed or spent its time. No prohibited content appears in spans, logs, or metric attributes.

Validate with the Tsuga CLI

# Confirm the customer request and its model call arrived as spans.
tsuga traces search \
  --query "context.service.name:<service> gen_ai.provider.name:<provider>" \
  --from -15m \
  --to now \
  --max-results 10

# Negative test: confirm a known sensitive token is absent from logs.
tsuga logs search \
  --query "context.service.name:<service> <known-sensitive-test-value>" \
  --from -15m \
  --to now \
  --max-results 10

The first command returns the model-call spans for the test request with their provider, model, operation, and token fields. If the provider filter returns nothing while spans exist, the instrumentation is emitting a different convention variant and level 1’s mapping is unfinished. The second command must return nothing. A result means the sensitive value reached Tsuga: suppress it at the source, then re-run before treating level 4 as complete.