OpenTelemetry Collector Health Implementation Guide

Overview

Use this path after a Collector is forwarding production telemetry. It makes the Collector observable as part of that path: whether it is accepting work, building queue pressure, refusing data under memory pressure, or failing to export to Tsuga. It does not instrument the applications or infrastructure behind the Collector.

Start by exporting the Collector’s own metrics, then validate a known signal through the existing pipelines, and finally add a small set of platform-owned alerts. The Collector’s telemetry reader sends metrics directly to Tsuga; it does not pass through a normal service.pipelines exporter.

Applications and receivers ──> Collector pipelines ──> otlp_http/tsuga ──> Tsuga
                                  │
Collector runtime ──> service.telemetry.metrics ──> OTLP metrics reader ──> Tsuga
                                  │
                                  └── health_check ──> local platform health probe

Before starting

Use OpenTelemetry Collector version 0.83 or later. The service.telemetry.metrics.readers format below requires it; use the configuration syntax supported by an older deployed version instead of copying this block.
Preserve the existing Collector configuration and use the same TSUGA_OTLP_ENDPOINT and TSUGA_INGESTION_KEY environment variables already used by otlp_http/tsuga. Do not add the endpoint or key to version-controlled YAML.
Choose a Collector identity that is separate from application services. otel-collector is a suitable service.name; retain the deployment environment and platform resource attributes that distinguish its operating boundary.
Decide which team owns the Collector, its capacity, upgrades, and alert response. A queue alert without an owner only reports that evidence may already be missing.
Keep health_check, pprof, and zpages on a local or private interface. A health probe proves local process availability; self-metrics prove its telemetry path is functioning.

Level 1 - Export Collector self-metrics

Add this to the existing service: block. Do not create a second service: block. The reader cannot reuse otlp_http/tsuga: pipeline exporters only run for components wired into service.pipelines, while service.telemetry exports the Collector’s runtime metrics directly. Both blocks can safely reference the same secret-backed environment variables.

service:
  telemetry:
    resource:
      service.name: otel-collector
      deployment.environment.name: ${env:DEPLOYMENT_ENVIRONMENT_NAME}
    metrics:
      readers:
        - periodic:
            exporter:
              otlp:
                endpoint: ${env:TSUGA_OTLP_ENDPOINT}/v1/metrics
                headers:
                  Authorization: Bearer ${env:TSUGA_INGESTION_KEY}
                protocol: http/protobuf

The /v1/metrics suffix is required. A normal otlp_http pipeline exporter receives the base OTLP endpoint and expands signal paths itself; the telemetry reader sends metrics directly to its configured endpoint.

Keep memory_limiter first in every existing signal pipeline. Give the Collector a real host or container memory boundary so percentage-based memory limits have meaning. If the deployment supports it, set GOMEMLIMIT near the Collector’s hard memory limit to keep Go heap growth within the same boundary.

Validate the rendered configuration with the same binary that runs it, then restart or roll out through the deployment system. For example:

otelcol-contrib validate --config=<COLLECTOR_CONFIG_PATH>

Exit criteria: the configuration validates, the Collector starts without telemetry-reader or authentication errors, and recent otelcol_* metrics are visible in Tsuga under service.name = otel-collector.

References: Collector internal telemetry and Collector configuration.

Level 2 - Read pressure, rejection, and export failure

Start with the metrics that identify the first overloaded boundary:

Metric family	What it indicates	First response
`otelcol_exporter_queue_size` and `otelcol_exporter_queue_capacity`	Batches are waiting to leave the Collector. A sustained high ratio means export is slower than receive.	Check downstream availability, network errors, and export rate before increasing replicas or queue size.
`otelcol_exporter_enqueue_failed_*`	Telemetry could not enter the exporter queue.	Check queue capacity, persistent-storage health where used, and incoming volume.
`otelcol_exporter_send_failed_*`	Telemetry failed permanently after export retries.	Check authentication, endpoint reachability, response codes, and retry policy.
`otelcol_processor_refused_*`	The `memory_limiter` rejected telemetry under memory pressure.	Check the Collector memory limit, `GOMEMLIMIT`, receiver rate, and processor cost.
`otelcol_loadbalancer_backend_latency`	One backend behind a `load_balancing` exporter is slower than its peers.	Investigate that backend or rebalance traffic; do not scale every Collector blindly.

Interpret the families together. A growing queue without failures is early pressure. Rising enqueue failures, send failures, or refused telemetry means evidence is already being lost or rejected. More Collector replicas help only when Collector CPU or memory is the bottleneck; they do not fix a saturated Tsuga endpoint, network path, or downstream backend.

Send one known trace, metric, or log through an existing pipeline after each Collector change. Confirm the local health endpoint if enabled, inspect Collector logs for receiver and exporter errors, then verify that signal in Tsuga. A running process or a successful config validation alone is not evidence of successful export.

Exit criteria: the team can distinguish queue pressure, queue admission failure, permanent export failure, and memory-limiter refusal, and a known signal reaches Tsuga through the changed Collector.

Level 3 - Make Collector health operational

Create a platform-owned view with the Collector’s queue ratio, enqueue failures, send failures, refused telemetry, runtime resource use, and the platform’s health-probe status. Filter it by deployment environment and the Collector’s platform boundary so a gateway issue is not confused with a node agent issue.

Add alerts only for conditions that need a response:

A queue that remains near capacity for a sustained period, after accounting for normal short bursts.
Any sustained increase in otelcol_exporter_enqueue_failed_*, otelcol_exporter_send_failed_*, or otelcol_processor_refused_*.
Missing Collector self-metrics for an expected running deployment, paired with the platform health probe to distinguish an unavailable Collector from a broken telemetry export path.

Use the rate of change of failure counters rather than their lifetime total. Set an owner, priority, notification route, and a tested first response. Record the deployment-specific capacity change that is safe: replica count for a stateless gateway, resource limit for an agent, or queue and storage sizing for an outage window.

Exit criteria: the platform team can see Collector pressure and data-loss signals, receives a tested notification for a sustained failure, and has a documented first action for each alert.

Troubleshooting path

Validate in order and stop at the first failing boundary:

Validate the rendered configuration with the deployed Collector binary.
Confirm the process or workload is running and the health_check endpoint responds, if enabled.
Check Collector logs for telemetry-reader startup, authentication, endpoint, queue, or memory-limiter errors.
Confirm recent otelcol_* metrics in Tsuga with the expected Collector identity.
Send one known signal through the affected pipeline and verify it in Tsuga.

No Collector metrics arrive: check the Collector version, that the telemetry reader is under the active service: block, and that its endpoint ends in /v1/metrics. Do not point the reader at a pipeline exporter name.
Metrics arrive under application identity: set a distinct service.telemetry.resource.service.name and preserve the relevant environment and platform resource attributes.
Queue size rises without failures: find the slow downstream path before enlarging the queue. A larger queue delays loss but consumes memory or disk.
The queue is full or enqueue_failed rises: retry cannot recover batches that never entered the queue. Check queue sizing, storage permissions and capacity, and incoming volume.
send_failed rises: inspect the exporter response and network path. Verify the ingestion key through its secret-management boundary; do not paste it into logs, tickets, or commands.
refused rises: the memory limiter is protecting the Collector. Check the memory boundary and workload before raising the limit or reducing receiver load.

Completion criterion

Recent Collector self-metrics are visible in Tsuga with a distinct Collector identity, a known signal passes through the Collector to Tsuga, and the platform team has an owned response for sustained queue pressure, failed export, rejected telemetry, and missing Collector telemetry.