Beyond the defaults: extending Kubernetes enrichment with your own labels and annotations

The k8s_attributes processor ships with six attributes.

Turn on the Kubernetes attributes processor and your telemetry starts carrying six things:

k8s.namespace.name
k8s.pod.name
k8s.pod.uid
k8s.pod.start_time
k8s.deployment.name
k8s.node.name

That is a good default, and deliberately minimal for a processor that watches Kubernetes resources and keeps metadata cached in each Collector instance. It tells you where something ran, which is usually why the processor went in.

What it cannot tell you is anything about meaning. k8s.deployment.name says a Deployment called checkout produced this trace. It does not say that checkout belongs to the payments team, that it is tier-1, or that this project is ch-321. That information already exists in your cluster on labels and annotations, maintained by people who may not know it is one config block away from every span, log and metric you collect.

The syntax for pulling those attributes in is a dozen lines. The rest of this article is about where to put them so they take effect, and what they cost.

[!info] Version Everything below is pinned to Collector contrib v0.158.0, whose k8s_attributes documentation declares Semantic Conventions v1.42.0. Links to the live OpenTelemetry semantic-convention documentation may show newer convention text over time.

Everything in this article also assumes pod association already works. Association decides which pod a telemetry item belongs to; extraction decides what metadata to add once that association has been made.

Before reaching for labels: the six defaults are not the full built-in metadata set. extract.metadata can enable additional attributes such as k8s.container.name, container.id, container image metadata, ReplicaSet, StatefulSet, DaemonSet, Job, CronJob and cluster identity, as well as supported service.* attributes. Check the built-in metadata first; use label and annotation extraction when the information you need actually lives in your Kubernetes metadata.

Extending it: labels and annotations from anything

The extension is based on two config keys, extract.labels and extract.annotations, and they take the same four fields:

Field	What it does
`key`	the exact label or annotation key to read
`key_regex`	a pattern instead of an exact key; mutually exclusive with `key`
`from`	which object to read it from
`tag_name`	what to call the resulting attribute; optional, more on this below

from is the one worth a second look. It accepts pod, namespace, deployment, statefulset, daemonset, job and node, defaulting to pod (k8s_attributes README). That means you are not limited to what somebody remembered to put on the pod template.

processors:
  k8s_attributes:
    auth_type: serviceAccount
    extract:
      labels:
        # Project lives on the namespace, where it is set once
        # instead of on every workload that ships into it.
        - from: namespace
          key: project
          tag_name: deployment.project.name

        # Ownership on the pod, from the template.
        - from: pod
          key: app.kubernetes.io/component

        # Topology from the node, which no application can know.
        - from: node
          key: topology.kubernetes.io/zone
          tag_name: cloud.availability_zone

Put shared values at the highest scope. Project on the namespace is better than project on forty Deployments, because forty places to update is forty places to drift.

For a family of related keys, key_regex with a backreference is better than writing them out:

      labels:
        - from: pod
          key_regex: observability\.example\.com/(.*)
          tag_name: $1

Note the narrow prefix. key_regex: (.*) also works and I would not run it, more on that later.

The annotation path for standard attribute names

A second way in is cleaner for attributes whose names you already want to expose directly. OpenTelemetry defines an annotation prefix, and the processor will translate the whole family for you with one switch:

    extract:
      otel_annotations: true

With that on, pod annotations become the attributes they name. This configuration option is pod-scoped. It is not extract.annotations with a from: you can point elsewhere:

metadata:
  annotations:
    resource.opentelemetry.io/service.version: "2.4.1"
    resource.opentelemetry.io/deployment.environment.name: production

resource.opentelemetry.io/service.version becomes service.version. No extraction rule, no tag_name, no per-attribute config. The semantic conventions are explicit that annotations with the resource.opentelemetry.io/ prefix are intended to translate into the corresponding resource attributes.

The mechanism is not restricted to semantic-convention attribute names. For example, resource.opentelemetry.io/deployment.project.name can become deployment.project.name, but deployment.project.name is a custom attribute, not an OpenTelemetry semantic-convention attribute.

It defaults to false, so nobody gets it without manual configuration. Enable otel_annotations by default when you treat resource.opentelemetry.io/ as a governed API: workload owners are trusted to choose telemetry resource attributes, and you have policy or review around what can go under that prefix. If you require a centrally controlled attribute allowlist, explicit extract.labels / extract.annotations rules are the safer mechanism.

Let the processor name your attributes

tag_name is optional, which most examples do not mention. Leave it out and the processor derives the attribute name from the Kubernetes object, preserving both the key and where it came from:

extract:
  labels:
    - from: pod
      key: app.kubernetes.io/component

    - from: namespace
      key: team

    - from: node
      key: topology.kubernetes.io/zone

On v0.158.0, the exact generated name depends on the processor.k8sattributes.EmitV1K8sConventions feature gate.

Without the gate, the processor uses the older plural form:

k8s.pod.labels.app.kubernetes.io/component
k8s.namespace.labels.team
k8s.node.labels.topology.kubernetes.io/zone

With processor.k8sattributes.EmitV1K8sConventions enabled, it uses the newer semantic-convention form:

k8s.pod.label.app.kubernetes.io/component
k8s.namespace.label.team
k8s.node.label.topology.kubernetes.io/zone

The singular k8s.<resource>.label.<key> attributes are defined in the Kubernetes semantic conventions and marked Stable in the attribute registry.

That naming is useful because it preserves provenance. k8s.namespace.label.team=payments tells you not only that the team is payments, but that the value came from the team label on the namespace.

Compare that with:

- from: namespace
  key: team
  tag_name: organization.team

Now the same value arrives as organization.team=payments. That may be what you want, but you have made a translation: the Kubernetes label has become an organization-wide concept, and its Kubernetes provenance is no longer visible in the attribute name.

If the Kubernetes metadata itself is what you want to expose, let the processor name it:

- from: namespace
  key: team

If the Kubernetes metadata is just the source for an attribute with broader semantics, name the translation explicitly:

- from: namespace
  key: project
  tag_name: deployment.project.name

- from: node
  key: topology.kubernetes.io/zone
  tag_name: cloud.availability_zone

Here deployment.project.name is an organization-defined attribute, not an OpenTelemetry semantic-convention attribute. You are asserting that the namespace’s project label represents your shared deployment-project concept. In the second rule, you are translating the Kubernetes topology label into the cloud availability-zone semantic meaning.

That translation earns its place even when no standard OpenTelemetry attribute exists, because you may have standardized an attribute across environments. If Kubernetes, serverless workloads and VMs all report organization.team, hiding the Kubernetes-specific source can be the point.

So the rule is not never use tag_name. It is: omit tag_name when you want to preserve Kubernetes metadata as Kubernetes metadata; use it when you intentionally want to translate that metadata into a different, shared semantic meaning.

Why your rule sometimes does nothing

You add an extraction rule, the label exists, RBAC is correct, k8s_attributes is in the pipeline and the value you expected still does not appear.

Before debugging the YAML, check whether the attribute already had a value.

The processor follows a simple rule: enrichment fills gaps; it does not overwrite non-empty resource attributes.

In v0.158.0, the relevant code is:

func setResourceAttribute(attributes pcommon.Map, key, val string) {
    attr, found := attributes.Get(key)
    if !found || attr.AsString() == "" {
        attributes.PutStr(key, val)
    }
}

So for every attribute it tries to add:

If the attribute does not exist, the processor adds it.
If the attribute exists but is empty, the processor fills it.
If the attribute already has a non-empty value, the processor leaves it alone.

There is no overwrite, and no warning that your extraction rule lost.

A concrete collision

Suppose the Collector has these rules:

extract:
  labels:
    - from: pod
      key: app.kubernetes.io/name
      tag_name: service.name
    - from: namespace
      key: project
      tag_name: deployment.project.name

The pod label says:

app.kubernetes.io/name=checkout-from-label

and the namespace label says:

project=payment

Now compare two runs with the same Collector configuration.

Run 1: the application only sets service.name:

Application sends:
  service.name = checkout-from-sdk

Collector tries to add:
  service.name = checkout-from-label
  deployment.project.name = payment

Result:
  service.name = checkout-from-sdk
  deployment.project.name = payment

service.name was already populated, so the Collector left it alone. deployment.project.name was missing, so the Collector filled the gap.

Run 2: the application sets both:

Application sends:
  service.name = checkout-from-sdk
  deployment.project.name = payment-sdk

Collector tries to add:
  service.name = checkout-from-label
  deployment.project.name = payment

Result:
  service.name = checkout-from-sdk
  deployment.project.name = payment-sdk

Nothing in the Collector configuration changed. The only difference was what had already arrived in the resource attribute map.

If an extraction rule looks correct but its value does not appear, check whether that attribute was already non-empty before k8s_attributes ran.

Where can the existing value come from?

service.name is an easy attribute to collide on, but it is not the only one. service.version, service.instance.id, deployment.environment.name, custom deployment attributes, cloud attributes and Kubernetes attributes can all have another source.

The easiest way to reason about them is in pipeline order.

Before telemetry reaches k8s_attributes:

The application SDK can set resource attributes directly. OTEL_SERVICE_NAME provides service.name, while OTEL_RESOURCE_ATTRIBUTES can provide many others:
```
service.name=checkout
service.version=2.4.1
deployment.project.name=payment
service.instance.id=checkout-7d9f6b8c9f-x2k4m
```
SDK resource configuration and resource detectors can contribute attributes too.
The OpenTelemetry Operator can inject or derive resource attributes for auto-instrumented workloads. Depending on its configuration, the application might export attributes such as:
```
service.name=checkout
service.namespace=shop
service.instance.id=checkout-7d9f6b8c9f-x2k4m
```
The Operator has its own precedence rules for deciding which source wins before the telemetry is exported.

resource.opentelemetry.io/ pod annotations can become resource attributes before export when the Operator is responsible for translating them:

metadata:
  annotations:
    resource.opentelemetry.io/service.name: checkout
    resource.opentelemetry.io/service.version: "2.4.1"
    resource.opentelemetry.io/deployment.project.name: payment

By the time telemetry reaches the Collector, those may already be:

service.name=checkout
service.version=2.4.1
deployment.project.name=payment

At this point telemetry reaches the Collector with a resource attribute map that may already contain application identity, deployment information and infrastructure metadata. k8s_attributes enriches that map; it does not start with an empty one.

When k8s_attributes processes it:

otel_annotations: true can translate resource.opentelemetry.io/ pod annotations into candidate resource attributes:

metadata:
  annotations:
    resource.opentelemetry.io/service.version: "2.4.1"
    resource.opentelemetry.io/deployment.project.name: payment

which gives the processor candidates for:

service.version=2.4.1
deployment.project.name=payment

extract.labels and extract.annotations can produce the attributes you explicitly map from Kubernetes metadata:

extract:
  labels:
    - from: namespace
      key: project
      tag_name: deployment.project.name

    - from: namespace
      key: environment
      tag_name: deployment.environment.name

    - from: node
      key: topology.kubernetes.io/zone
      tag_name: cloud.availability_zone

    - from: pod
      key: team

giving candidates such as:

deployment.project.name=ch-321
deployment.environment.name=production
cloud.availability_zone=eu-west-1a
k8s.pod.label.team=payments

Processor-derived Kubernetes metadata can add configured attributes from the Kubernetes hierarchy:

k8s.namespace.name=shop
k8s.pod.name=checkout-7d9f6b8c9f-x2k4m
k8s.pod.uid=3f2...
k8s.deployment.name=checkout
k8s.node.name=worker-03

The important word in steps 4–6 is candidate. Kubernetes may tell the processor:

deployment.project.name=payment
cloud.availability_zone=eu-west-1a

but telemetry might have arrived with:

deployment.project.name=proj-payment
cloud.availability_zone=eu-west-1b

In that case, the existing values survive.

This is not one global precedence list

The six sources above operate at different stages, which is why they do not collapse into a precedence table.

The SDK and Operator can modify the resource before export. k8s_attributes operates after the Collector receives it. Even the same resource.opentelemetry.io/ annotation can be translated at different layers depending on your setup.

So the useful question is:

What was already in the resource attribute map when k8s_attributes tried to enrich it?

There are still precedence rules within individual mechanisms, and those have changed between versions. The semantic conventions define an order for deriving service identity from Kubernetes metadata. For service.name, that includes the resource annotation and application labels before falling back through workload hierarchy such as Deployment, ReplicaSet, StatefulSet, DaemonSet, CronJob, Job, Pod and Container. Implementations have not always followed those rules correctly.

Test those precedence rules on the versions you actually run. But for debugging your own extraction rule, start simpler: inspect what entered k8s_attributes. If the key was already populated, Kubernetes enrichment will not replace it.

That also gives you a useful ownership rule: use Collector enrichment primarily for attributes the application should not have to know. Infrastructure identity, topology, ownership labels and environment are natural Collector concerns. Service identity usually belongs closer to the application, set by the team that owns the service.

A tag_name: service.name rule in the Collector should therefore be a deliberate fallback, not another competing source of truth.

What to extract, and who signs off

Every attribute you extract is a resource attribute, which means it rides on every span, every log record and every metric data point from that workload. That is exactly why the processor is useful, and exactly why the allowlist deserves a decision rather than a reflex.

The processor accepts this:

- from: pod
  key_regex: (.*)

Do not, unless you control every label in the cluster. A catch-all rule ingests whatever Kubernetes, Helm and your controllers happen to put on a pod: rollout hashes, pod-template-hash, chart metadata, generated values, and occasionally something that should never have left the cluster at all. The README’s own guidance on the all-labels example is “use with caution, may extract many attributes”.

Keeping a value out of the pipeline is cheaper than redacting it downstream.

Also keep the Kubernetes distinction between labels and annotations intact. Labels are intended to identify and select objects; annotations are for non-identifying metadata. Do not turn metadata into a Kubernetes label solely because you want it in telemetry: extract.annotations exists for that case.

A workable review test, four questions per proposed attribute:

How many distinct values will this have? A team name may have twenty values. A commit SHA introduces a new resource value on every deploy and may become an expensive dimension depending on how your backend maps resource attributes.
Who owns the key? If a controller generates it, you do not control it, and it can change shape in a chart upgrade.
Would anyone query it? An attribute nobody filters or groups by is weight without value.
Is it safe in a telemetry backend? Different blast radius from a Kubernetes annotation.

What extending costs the Collector

The concrete cost is in the Collector.

Extracting labels or annotations from higher-level objects such as deployment, statefulset, daemonset and job is disabled by default, and the README says why: “Enabling extraction of these metadata comes with an extra memory consumption cost.” Each object type you add can require additional Kubernetes watches, caches and RBAC.

Deployment handling deserves one distinction in v0.158.0: the processor can derive k8s.deployment.name from ReplicaSet ownership/naming without requiring you to extract Deployment labels or annotations. Asking for metadata from the Deployment object, however, requires the additional lookup machinery and corresponding ReplicaSet/Deployment access. Do not assume that because the default k8s.deployment.name is present, Deployment metadata extraction is free.

At size, one knob is watch_sync_period, which defaults to 5m; the README suggests setting it to 0s on large clusters to stop periodic resyncs churning through every cached object.

Another lifecycle knob:

processors:
  k8s_attributes:
    pod_delete_grace_period: 120s

pod_delete_grace_period defaults to 120 seconds. Deleted pods remain available for lookup during that grace period, which gives delayed telemetry a chance to receive pod metadata even after Kubernetes has deleted the pod. Increasing it extends that safety window at the cost of retaining deleted-pod metadata longer.

Prove it took effect

The fastest read is a debug exporter, temporarily:

exporters:
  debug:
    verbosity: detailed

Look at the Resource attributes block and confirm the keys you expect, with the names you expect, and that nothing you configured is missing because something upstream already set it.

Then the processor’s own telemetry. otelcol.k8s.pod.association counts association evaluations, which is the thing that has to work before any enrichment happens at all. otelcol.k8s.watcher.pod_cache.size is the size of the pod cache specifically useful for watching pod-metadata memory, but do not read it as a total. Adding higher-level object extraction can start separate watchers that this gauge does not measure; the per-object watcher counters are what move there.

One trap produces a convincing false negative: a Collector that has just restarted may not enrich the first records through, because the informer cache is still filling. It has been reported upstream as labels missing from the first batch after a restart, on an older release.

One setting covers this:

processors:
  k8s_attributes:
    wait_for_metadata: true
    wait_for_metadata_timeout: 10s

It defaults to false. Turn it on and the processor is not ready until metadata is synced, so start-up blocks instead of a first batch going out unenriched. The tradeoff: if metadata cannot sync inside the timeout, the Collector fails to start rather than starting degraded. For most production setups that is the trade you want: a Collector that will not start is a page, whereas a Collector quietly emitting unattributed telemetry is a mystery later on.

Together, the two lifecycle settings protect opposite ends of a pod’s life: wait_for_metadata protects enrichment while caches are filling at Collector startup, while pod_delete_grace_period keeps recently deleted pods available for delayed telemetry.

If you leave wait_for_metadata off, do not judge your extraction rules in the first seconds after a rollout.

And if attributes are missing entirely rather than occasionally, the problem is upstream of everything in this article: association. That is a different subject with its own failure modes: gateways, proxies and meshes that hide the source pod’s address.

One container-specific trap is worth calling out even though it is outside custom extraction: in multi-container pods, container-level image enrichment needs enough incoming identity, such as container.id or k8s.container.name, to determine which container the telemetry belongs to. Pod association alone does not identify the container.

The table worth writing down

Extending the processor is a dozen lines of YAML. What makes it stick is deciding, once, who owns what.

Attribute	Owner	Mechanism
`service.name`, `service.version`	the application team	SDK env vars, or `resource.opentelemetry.io/` annotations
`deployment.project.name` (custom)	platform	namespace label → `extract.labels`
Cluster, namespace, node, workload identity	platform	`extract.metadata`
Topology and zone	platform	node labels, or a cloud resource detector
Team, tier, cost center	whoever maintains the labels	`extract.labels`, no `tag_name`

Start with one row: pick the label your on-call rota already thinks in, extract it without a tag_name, and see how much faster the next incident routes.