Collector to Collector: choose authentication for the hop you have

An OpenTelemetry Collector hop does two jobs: it accepts telemetry and it opens a new connection to the next destination. Treating those jobs as one security boundary creates confusion.

The useful questions are simpler:

1. What boundary does this connection cross? pod, namespace, cluster, region, …
2. Does the receiving pipeline need a trustworthy name for the sender?
3. Must that name survive the next Collector hop?

TLS, mTLS, basic authentication, bearer tokens, OIDC, and a service mesh answer different parts of those questions. None of these is an upgrade on the one before it. They answer different parts of those questions, and the right choice depends on the hop.

A Collector hop resets transport identity

A hop is one direct network connection between two components. Consider an agent Collector on each node, a gateway Collector in the cluster, and a backend beyond it.

The gateway terminates the inbound connection. Its exporter creates a new outbound request. The backend therefore authenticates the gateway, not the agent that first produced the telemetry.

That does not make per-sender attribution impossible. If identity is important, at the gateway, read an identity established by an authenticator, write it into a resource attribute, or arrange for a trusted component to inject a new outbound header. Do not expect the next Collector to infer it from the previous TLS connection.

This distinction also separates two security properties that are often conflated:

• Admission control answers: may this sender connect?
• Attribution answers: which authenticated sender produced this data?

You may need both.

Start with exposure and reachability

Before you choose a credential, decide who can reach the receiver. Since Collector v0.110.0, server components bind to localhost by default. That protects an unmodified receiver, but many Kubernetes examples set 0.0.0.0 so a service can reach it. The Collector security guidance recommends binding to a specific interface such as the pod IP instead.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: ${env:MY_POD_IP}:4317

In Kubernetes, add a NetworkPolicy that limits the source namespaces or pods allowed to reach OTLP ports.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: gateway-otlp-ingress
  namespace: observability
spec:
  podSelector:
    matchLabels:
      app: otel-gateway
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              otel-send: "true"
      ports:
        - protocol: TCP
          port: 4317
        - protocol: TCP
          port: 4318

NetworkPolicy only works when the cluster network implementation enforces it. Test from an unselected namespace and confirm the connection fails. A policy visible in the API but ignored by the CNI does not reduce exposure.

Network controls constrain the blast radius. They do not prove which permitted workload sent a request. Add transport security and authentication when the hop needs them.

Use TLS as the transport baseline

Use TLS for any Collector connection that leaves a node, crosses an untrusted network segment, or reaches an externally exposed endpoint. Validate the peer certificate. Do not set insecure_skip_verify (in Kubernetes for instance) to make a certificate-name mismatch disappear.

mTLS adds client-certificate verification. On an OTLP receiver, client_ca_file causes the server to require and verify a certificate signed by a trusted client CA.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: ${env:MY_POD_IP}:4317
        tls:
          cert_file: /certs/gateway.crt
          key_file: /certs/gateway.key
          client_ca_file: /certs/collector-ca.crt
          reload_interval: 1h
          client_ca_file_reload: true

The TLS settings reload the certificate on an interval. client_ca_file_reload reloads the client-CA file when it changes, and it defaults to false, so the line above is not redundant.

mTLS gives strong admission control. By itself, it does not give processors a portable, documented client-certificate attribute to read. The standard authentication context is populated by server authenticator extensions, not by the receiver TLS configuration. If you need a pipeline-visible sender name, use an authenticator that exposes one or establish the identity at a proxy and pass it to the Collector under a trust boundary you control.

The same point applies when a Kubernetes Gateway or load balancer terminates mTLS. That component can reject untrusted clients, but the Collector behind it only sees the connection the proxy makes. The Gateway API mTLS example is a good pattern for securely exposing an OTLP endpoint; it is not, on its own, a way to expose client certificate identity to a Collector processor.

Choose an authenticator when the pipeline needs identity

The Collector’s authentication context is the key distinction among the common options.

• Static bearer token: no useful sender identity on successful authentication. Token files are supported, but live reload is not documented. Use it for a single sender or a narrow compatibility case.
• Basic auth: exposes the authenticated username. Client username and password files are watched for changes. Use it for a small set of named senders you operate.
• OIDC receiver auth: exposes verified JWT claims. IdP-issued tokens and key discovery manage the lifecycle. Use it for multi-tenant or partner ingestion.
• mTLS at the Collector: provides certificate admission but no standard auth-context identity. Certificate and CA reload settings are available. Use it for strong connection admission control.

Basic auth deserves a more precise recommendation than “better than a bearer token.” Both are static-secret mechanisms and both require TLS. Basic auth becomes useful when you assign each sender a distinct username and need that identity in the pipeline. The basic-auth extension exposes the authenticated username on successful server authentication. Its client username_file and password_file options watch for changes, so a sending Collector can rotate credentials without restarting.

Note where that stops. The watching is documented for the client-side credential files, not for the server’s htpasswd.file. On the receiving gateway, adding or revoking a sender is not a documented live-reload path, so plan that as a config rollout.

That lets a gateway stamp an authenticated identity into telemetry:

processors:
  resource/sender_identity:
    attributes:
      - key: telemetry.sender
        from_context: auth.username
        action: upsert

The context key is auth.username, which is the attribute the extension documents on client.Info.Auth after successful server authentication. The basic-auth extension is beta, and the resource and attributes processors are beta for traces, metrics, and logs.

OIDC is the better fit when identity carries claims such as tenant, audience, or subject. The OIDC extension verifies a JWT and makes claims available to processors.

processors:
  resource/tenant:
    attributes:
      - key: tenant.id
        from_context: auth.claims.tenant_id
        action: upsert

The difference between auth.claims.tenant_id and metadata.tenant_id is a security boundary. metadata. reads a request header supplied by the client. auth. reads data the receiver’s authenticator established after verification. Do not base routing, tenancy, or billing on a sender-controlled header.

For a partner integration, pair an OIDC-capable receiver with an OAuth 2.0 or OIDC token issuer. The Collector’s OAuth2 client extension helps an outbound Collector obtain and refresh client-credentials tokens; it does not replace the receiver-side verification step.

Use OIDC when identity needs richer claims, delegation, tenant isolation, or partner-facing auth. Use basic auth when you need lightweight per-sender identity and do not already operate an identity provider.

A mesh changes the transport work, not the attribution work

If both Collectors run in a service mesh, the mesh can supply mTLS and service-to-service authorization. With Istio, configure authorization policy for the gateway OTLP ports and verify that the relevant workloads run in strict mTLS mode. Permissive mode accepts both mTLS and plaintext traffic during migration.

That can remove certificates and TLS settings from Collector configuration. It does not automatically make the mesh workload identity available to Collector processors. Treat mesh policy as connection admission unless you have tested a supported identity-propagation path for your proxy, protocol, and receiver. A local Compose lab is the cheapest place to find out what your receiver actually sees.

Preserve the identity you need, deliberately

Once a gateway receives data, it has three places an identity can matter:

1. In the telemetry itself, usually as a resource attribute such as telemetry.sender or a tenant attribute.
2. In routing decisions inside the gateway.
3. On the outbound request to the next system.

This is a different identity from the workload identity you should already be attaching at the agent, where the guidance is to enrich records before network hops make association ambiguous. That one describes what the workload is. This one records who the sender proved to be, and only the gateway can establish it.

For the first two, write the authenticated identity into the signal before the gateway forwards it. Make the ownership and trust contract clear: the gateway, not the original sender, owns that attribute. If senders can set the same attribute, overwrite it at the gateway rather than preserving their value.

For the third, have a trusted gateway or proxy inject the outbound header after it authenticates the inbound request. This mirrors the model used by multi-tenant backends: the component that verifies the caller writes the tenant value last. A header becomes dangerous when an untrusted sender gets the final write.

Batching needs attention in multi-sender gateways. The batch processor can separate batches by client.Metadata with metadata_keys, but receivers must enable include_metadata: true. Each distinct key combination creates a long-lived batcher and holds a pending batch, while metadata_cardinality_limit defaults to 1,000. The batch processor documentation calls out the memory cost and recommends validating the metadata values with an auth extension.

Do not use an arbitrary tenant header as a batch key and call that isolation. Authenticate it first, or derive it from authenticated identity.

Decision guide

• Application to a node-local agent: use localhost or a Unix socket. Identity is usually unnecessary.
• Agent to gateway in one cluster: use pod-IP binding, NetworkPolicy, and TLS. Add basic auth with one username per sender, or OIDC, when the gateway needs a sender name.
• Collector hop already covered by a mesh: use mesh mTLS and authorization policy. Add a tested proxy-to-Collector identity contract when needed.
• Cluster-to-cluster connection you operate: use private connectivity, TLS, and mTLS where PKI is practical. Add basic auth or OIDC if the gateway must name the source cluster.
• Partner or tenant ingestion: use TLS plus an issuer and receiver-side OIDC validation. Promote verified claims into controlled resource attributes.
• Large fleet with workload-level identity: use workload identity such as SPIFFE/SPIRE if you can operate it. Design explicit mapping or propagation for pipeline use.

For most gateways, the implementation order is straightforward:

1. Bind the receiver narrowly and prove the network policy blocks an unauthorized source.
2. Enable validated TLS wherever the hop leaves the node or trusted segment.
3. Use mTLS or mesh policy to control admission across cluster, account, or organizational boundaries.
4. Add basic auth with distinct credentials, or OIDC with verified claims, only where the pipeline needs an authenticated sender name.
5. Promote the identity into controlled telemetry attributes before forwarding it, and overwrite any sender-supplied value.
6. Configure metadata-aware batching only when request-level separation is required, then monitor its cardinality.

Recap

What each mechanism actually gives you, which is the distinction the rest of this post turns on:

The strongest design is not the one with the most authentication mechanisms. It is the one that gives every hop the admission control it needs and makes sender identity explicit wherever a downstream decision depends on it.