APM Implementation Guide

Overview

Move from an uninstrumented service to actionable APM in Tsuga. Add traces, span metrics, release identity, log correlation, operational coverage, then sampling in that order.

Application SDK / auto-instrumentation
                 │ OTLP traces
                 ▼
      OpenTelemetry Collector ─────────────► Tsuga Traces
                 │
                 ├─ span_metrics ──────────► Tsuga Metrics - Service page, deployments, health context
                 │
Application logger + active trace context ─► Tsuga Logs Correlated with Traces

Collect and structure application logs before level 4, which assumes a working log route. Deploy the Collector that receives the traces through the applicable host or cluster collection route, because this path does not deploy one.

Before starting

Agree the services, environments, owners, and the first business-critical journey in scope. Instrument one service end to end before widening.
Decide the collection route: direct SDK export for a first validation, or a Collector when services need shared batching, enrichment, filtering, routing, or credentials. Level 2 requires a Collector, so a service that will need span metrics should start there.
Fix the identity convention before rollout: service.name, deployment.environment.name, and service.version from an immutable build identifier. Renaming these later splits history.
Keep the Tsuga OTLP endpoint and ingestion key in the platform secret store, never in a Collector file, container image, or checked-in values file.
Agree the span-attribute and metric-dimension policy before instrumenting. Use route, method, operation, and status code as dimensions. Do not use user IDs, request IDs, raw URLs, or payloads.
Confirm outbound HTTPS from the sender to the Tsuga intake endpoint.

References: OpenTelemetry language SDKs, resource semantic conventions, and Collector configuration.

Level 1 - Forward traces to Tsuga

Instrument the service with the language SDK, an agent, or auto-instrumentation, then export traces through OTLP. Direct SDK export is acceptable for a small first validation. Use a Collector when services need common batching, enrichment, filtering, routing, shared credentials, or trace-origin tracking.

Set stable resource identity from the start:

service.name identifies the service.
deployment.environment.name separates production from non-production traffic. In Tsuga, make sure you use the right API key with the correct env tag setup.

Exit criteria: a known request is searchable as a trace in Tsuga with the agreed service.name and environment, and its root span represents the entry operation rather than an internal function.

References:

Runtime guides: OpenTelemetry language SDKs for Java, Node.js, Python, .NET, Go, PHP, Ruby, Rust, and C++.

Level 2 - Generate metrics from traces

Raw traces answer “what happened to this request?” They do not populate Tsuga’s Service page. Add the Collector span_metrics connector so traces also produce request, error, and duration metrics. Use delta aggregation for span-derived RED metrics in dynamic workloads:

connectors:
  span_metrics:
    aggregation_temporality: AGGREGATION_TEMPORALITY_DELTA

With cumulative aggregation, an inactive pod can retain its final counter and cumulative_to_delta can emit repeated zeroes. This can keep a stale service version visible. Delta reports each interval’s work and stops when the pod stops serving.

Cumulative source:          120 → 120 → 120 → 120
After cumulative-to-delta:    0 →   0 →   0

Delta source while serving:    8 →  11 →   6
Pod stops serving:            no further datapoints

Span metrics populate Service page traffic, error rate, latency, operation breakdowns, and deployments. Observed service.version values support faulty-deployment detection and connect service health with monitors, SLOs, and alerts.

Keep span-metric dimensions intentionally small. Use route, method, and status code. User IDs, request IDs, raw URLs, and message IDs create costly high-cardinality time series.

Configure the connector for the deployment model

The Tsuga opentelemetry-kube-stack Helm chart, version 0.11.4 or later, uses delta aggregation by default. Set it explicitly when upgrading from an earlier chart or when you want the choice recorded in your values file:

agent:
  spanMetrics:
    aggregationTemporality: AGGREGATION_TEMPORALITY_DELTA

The Tsuga chart already declares and wires span_metrics. For a community chart, configure it as a traces exporter and metrics receiver. A declared but unwired connector does not run:

connectors:
  span_metrics:
    aggregation_temporality: AGGREGATION_TEMPORALITY_DELTA

service:
  pipelines:
    traces:
      exporters: [span_metrics, otlp_http/tsuga]
    metrics:
      receivers: [otlp, span_metrics]

For a standalone or local Collector, keep cumulative_to_delta in the metrics pipeline for application metrics that arrive cumulatively. It passes span metrics that are already delta through unchanged:

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  resource_detection:
    detectors: [env, system]
    override: false
  cumulative_to_delta: {}
  batch: {}

connectors:
  span_metrics:
    aggregation_temporality: AGGREGATION_TEMPORALITY_DELTA

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, resource_detection, batch]
      exporters: [span_metrics, otlp_http/tsuga]
    metrics:
      receivers: [otlp, span_metrics]
      processors: [memory_limiter, resource_detection, cumulative_to_delta, batch]
      exporters: [otlp_http/tsuga]

For a host-local receiver, bind OTLP to 127.0.0.1. Expose another interface only when senders run outside that network namespace, then secure the path. Keep the Tsuga OTLP endpoint and ingestion key in environment variables or the platform secret store, never in the Collector file.

Why delta is the default

Delta aggregation expires inactive pod and version series, avoids repeated zero-value ingestion during churn, and keeps Service and Deployments views focused on recent traffic. Choose cumulative only when recovering counts after a missed export matters more than prompt expiry of inactive series.

Cumulative aggregation remains useful when export reliability matters more than prompt expiry of inactive series. A later cumulative total can let downstream conversion recover counts from an interval that failed to export. A lost delta interval cannot be recovered. Use the configurable cumulative setting only for that recovery behavior:

agent:
  spanMetrics:
    aggregationTemporality: AGGREGATION_TEMPORALITY_CUMULATIVE

When span_metrics or cumulative_to_delta runs in a scaled Collector tier, route each generated metric stream consistently to the same instance. Both components keep state. Splitting a stream produces incorrect results.

Exit criteria: the Service page shows request rate, error rate, and latency for the instrumented service. The connector is wired from the traces pipeline into the metrics pipeline. A stopped workload’s series stop appearing instead of reporting zeroes.

Level 3 - Version services and releases

Set service.version as a resource attribute on every released workload. Use for instance the immutable commit SHA that built the deployment as the default value. A release tag or semantic version is also suitable only when it identifies exactly one immutable build. Do not use a mutable value such as a branch name, latest, or an environment name.

Set the same version on traces, metrics, and logs through deployment configuration such as OTEL_RESOURCE_ATTRIBUTES. With service.name and deployment.environment.name, it gives Tsuga the identity needed to compare releases and surface faulty versions.

Exit criteria: the Deployments tab lists the current and previous service.version for the service, and the same version value appears on its traces, span metrics, and logs.

Level 4 - Correlate logs and traces

Logs need the active span’s trace_id and span_id plus matching service attributes. Enable a bridge before framework initialization or add the fields in the JSON formatter. The Collector preserves context that arrives with the log.

Asynchronous workflows

Inject W3C context when publishing. Continue immediate work in the trace. For delayed or independent work, use a CONSUMER span linked to the producer, then validate the producer message, consumer span, and correlated log.

Exit criteria: a log written during a known request opens its trace, and the span shows that log. The same check passes for each in-scope background job or queue consumer.

Level 5 - Create monitors and route the response

Start with error-rate and latency monitors for the service and environment. Add a traffic floor, owner, priority, tags, data access, no-data behavior, and notification rule.

[!warning] Set service.version on every release A stable version makes alerts and SLO breaches attributable to a release.

Suggested first monitors:

Error rate above the agreed threshold for the agreed time window, with enough requests to make the percentage meaningful.
High p95 latency for an important endpoint, with a traffic floor to avoid paging on one slow request.
A no-data condition only when loss of telemetry or request traffic is itself actionable.

Select the coverage set

Cover the small set of services that support a business-critical journey. Record the accountable team, environment, key operation, dependencies, and user impact. Before paging on a service, confirm its stable identity, span metrics, version, and dependencies in the Services list and Service page.

Bonus: adaptive paging for a critical journey (upcoming)

Adaptive paging uses traces matching a customer-facing error to identify a likely service and route the alert to its destination. It can reduce duplicate pages during cascading failures. Scope it to one service, set a destination and fallback, test a known failure, and confirm Tsuga availability before committing to it.

Create SLOs for customer-visible outcomes

Create one SLO for the most important journey. Use an event-based SLO for a success ratio or a time-based SLO for a latency target. Scope it to the service, environment, and operation, then set its owner, target, rolling window, and error-budget alert. Add coverage only after the first SLO produces actionable data.

Exit criteria: the business-critical service has an owned error-rate monitor, a latency monitor with a traffic floor, and at most one SLO. A test notification reached the on-call route. The team knows the expected response.

Level 6 - Sampling position

[!warning] Default to full fidelity Retain all traces unless measured volume or cost requires reduction.

Place sampling after span_metrics so request and error counts still represent all traffic. SDK head sampling drops evidence before the Collector can process it. Use head sampling for the simplest reduction. Use tail sampling when retaining errors, slow requests, or critical workflows justifies a stateful, trace-ID-routed Collector layer.

Exit criteria: the sampling decision is recorded with its reason, and either no sampling is in place or the decision runs after span_metrics so request and error counts still represent all traffic.

Troubleshooting path

Validate in signal order and stop at the first failing boundary.

No traces. Check that the SDK or agent started before the framework, that the exporter targets the intended endpoint and protocol, and that the Collector reports no export or authentication errors. Search for the service in Tsuga by name before assuming the data is missing.
Traces arrive, the Service page is empty. This is a span_metrics topology problem, not an instrumentation problem. Confirm the connector is an exporter of the traces pipeline and a receiver of the metrics pipeline, and that it runs in the Collector tier you are inspecting.
A stale service version shows zero traffic. Check aggregation temporality. Cumulative counters converted to delta emit repeated zeroes for a workload that stopped serving.
Wrong or duplicated service identity. Fix service.name and deployment.environment.name at the deployment or runtime layer. Do not rename spans or log records after collection.
Logs do not link to traces. Check that the application wrote the active trace_id and span_id, and that logs and traces share the same service.name. A Collector cannot recreate context the application dropped. Check asynchronous boundaries separately.
A monitor never fires or fires constantly. Verify the underlying metric and its dimensions in Analytics first, then the traffic floor and window.

Completion criterion

The Service page shows request rate, error rate, and latency for a known service version, and a log from one of those requests opens its trace.

Validate with the Tsuga CLI

# Confirm the service emitted spans after instrumentation.
tsuga traces search \
  --query "context.service.name:<service>" \
  --from -15m \
  --to now \
  --max-results 10

# Confirm a log from the same request carries the trace context.
tsuga logs search \
  --query "context.service.name:<service> <unique-log-token>" \
  --from -15m \
  --to now \
  --max-results 10

The first command returns recent spans with the agreed service.name. An empty result means the export path failed, not that the Service page is misconfigured. The second returns the test log with a non-empty trace_id that matches a trace from the first command. To confirm that span metrics exist rather than only spans, use Generate Span Metrics and the Service page.