Kubernetes Implementation Guide

Overview

Build Kubernetes visibility in order: collection, inventory, dashboards, then alerts. Application traces alone do not make a cluster operationally visible.

Start with the supported Tsuga Kubernetes Helm chart, which deploys an agent DaemonSet and cluster receiver. If the customer owns the Collector resources directly, follow the Collector deployment guidance and name one owner.

This path collects cluster, node, and container telemetry. Application instrumentation must set service.name, export traces to the node-local Collector, and populate the Service page. Configure application log format, routing, and Grok parsing separately when the container stdout path in level 3 is insufficient.

Before starting

• Agree a stable cluster name. It becomes k8s.cluster.name, the key attribute for cluster filtering and attribution.
• Create a new ingestion key and store its credentials through the customer’s secret-management process. The chart needs the Tsuga OTLP endpoint and ingestion key. Never put either in a checked-in values file.
• Confirm that the OpenTelemetry Operator CRDs exist, or enable the chart’s Operator installation. The chart documentation also lists cert-manager as a prerequisite.
• Decide which namespaces and nodes are in scope, and confirm the Collector has the required Kubernetes RBAC and egress to Tsuga.

References: OpenTelemetry Collector Helm chart and Collector components for Kubernetes.

Deployment configurations

Use Option A for the supported, Operator-managed stack. Choose Option B when the customer owns the Collector lifecycle. Run one chart for each collection responsibility to avoid duplicate signals. Create tsuga-credentials through the secret manager with TSUGA_API_KEY and TSUGA_OTLP_ENDPOINT before deployment.

Option A - Tsuga Kubernetes stack (default)

Use the Tsuga Kubernetes chart when the customer wants the supported, Operator-managed stack. The chart creates the agent DaemonSet and the cluster receiver Deployment. Save this as tsuga-kube-stack-values.yaml and install it with the existing secret.

clusterName: "<CLUSTER_NAME>"

opentelemetry-operator:
  enabled: true

secret:
  create: false
  name: tsuga-credentials

References: OpenTelemetry Operator for Kubernetes.

Option B - Community Collector chart

Use the public OpenTelemetry Collector Helm chart when the customer wants to own the Collector releases directly. Deploy the two files below as separate releases: the agent on every node, and a single cluster receiver. The chart presets create the required Collector configuration, host mounts, and RBAC for their enabled components.

otel-agent-values.yaml

mode: daemonset

image:
  repository: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-k8s

command:
  name: otelcol-k8s

extraEnvsFrom:
  - secretRef:
      name: tsuga-credentials

presets:
  logsCollection:
    enabled: true
  hostMetrics:
    enabled: true
  kubeletMetrics:
    enabled: true
  kubernetesAttributes:
    enabled: true

config:
  processors:
    resource/cluster:
      attributes:
        - key: k8s.cluster.name
          value: "<CLUSTER_NAME>"
          action: upsert
    cumulative_to_delta: {}

  exporters:
    otlp_http/tsuga:
      endpoint: ${env:TSUGA_OTLP_ENDPOINT}
      compression: gzip
      headers:
        Authorization: Bearer ${env:TSUGA_API_KEY}
      retry_on_failure:
        enabled: true
      timeout: 30s

  service:
    pipelines:
      logs:
        receivers: [otlp]
        processors: [memory_limiter, k8s_attributes, resource/cluster, batch]
        exporters: [otlp_http/tsuga]
      metrics:
        receivers: [otlp, prometheus]
        processors: [memory_limiter, k8s_attributes, resource/cluster, cumulative_to_delta, batch]
        exporters: [otlp_http/tsuga]
      traces:
        receivers: [otlp]
        processors: [memory_limiter, k8s_attributes, resource/cluster, batch]
        exporters: [otlp_http/tsuga]

otel-cluster-values.yaml

mode: deployment
replicaCount: 1

image:
  repository: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-k8s

command:
  name: otelcol-k8s

extraEnvsFrom:
  - secretRef:
      name: tsuga-credentials

presets:
  clusterMetrics:
    enabled: true
  kubernetesEvents:
    enabled: true
  kubernetesAttributes:
    enabled: true

config:
  processors:
    resource/cluster:
      attributes:
        - key: k8s.cluster.name
          value: "<CLUSTER_NAME>"
          action: upsert
    cumulative_to_delta: {}

  exporters:
    otlp_http/tsuga:
      endpoint: ${env:TSUGA_OTLP_ENDPOINT}
      compression: gzip
      headers:
        Authorization: Bearer ${env:TSUGA_API_KEY}
      retry_on_failure:
        enabled: true
      timeout: 30s

  service:
    pipelines:
      logs:
        receivers: [otlp]
        processors: [memory_limiter, k8s_attributes, resource/cluster, batch]
        exporters: [otlp_http/tsuga]
      metrics:
        receivers: [otlp, prometheus]
        processors: [memory_limiter, k8s_attributes, resource/cluster, cumulative_to_delta, batch]
        exporters: [otlp_http/tsuga]
      traces:
        receivers: [otlp]
        processors: [memory_limiter, k8s_attributes, resource/cluster, batch]
        exporters: [otlp_http/tsuga]

Configure each workload to export to the Collector on its node through the chosen node-local address. localhost only works when the workload shares the Collector network namespace.

References: OpenTelemetry Collector Helm chart and Collector components for Kubernetes.

Level 1 - Node-level collection with the agent DaemonSet

Run an agent Collector as a DaemonSet so every eligible node has a local collection point. This is the foundation for node and pod visibility, and it lets workloads export OTLP to a Collector on their own node rather than depending on an application-facing central gateway.

The standard agent path should provide:

• OTLP input for instrumented applications.
• Host metrics, kubelet metrics, and Collector self-metrics.
• A k8s_attributes enrichment path so application signals gain pod, namespace, node, workload, and cluster context.

Expose receivers only to required senders. Keep memory_limiter first, Kubernetes enrichment before filtering or redaction, and batching near the end of each pipeline.

Exit criteria: one ready agent per intended node. An application can export a test trace, metric, and log to its node-local path. The resulting signals carry a stable cluster identity.

Validate with the Tsuga CLI

# Confirm that the agent exported a recent kubelet node CPU metric.
# Replace the Unix timestamps with a recent validation window.
tsuga aggregation scalar \
  --data '{
    "timeRange": {"from": <from-unix-seconds>, "to": <to-unix-seconds>},
    "dataSource": "metrics",
    "queries": [{
      "aggregate": {"type": "sum", "field": "k8s.node.cpu.time"},
      "filter": "context.k8s.cluster.name:<cluster>"
    }],
    "formula": "q1"
  }'

A result for k8s.node.cpu.time confirms that a kubelet-derived metric from the agent is arriving with the expected cluster identity. Use the test application trace, metric, and log from the exit criteria to validate the node-local OTLP receiver separately.

Level 2 - Cluster-level collection with the cluster receiver

Run the cluster receiver as a dedicated Deployment for API-derived metrics and Kubernetes events. Scale it only with leader election or sharding, since duplicate readers can duplicate cluster signals. Enable extra object watches only for a defined use case because they expand API-watch and RBAC scope.

Exit criteria: the cluster receiver is ready. Recent cluster, namespace, workload, pod, and node metric families are visible. Kubernetes events arrive through the same Tsuga export path.

Validate with the Tsuga CLI

# Confirm that recent cluster-derived metrics have made the cluster discoverable.
tsuga kubernetes clusters --search <cluster>

The command should return the cluster with node-readiness and pod-phase counts. No result points to the cluster receiver or its export path.

The default cluster receiver does not scrape Kubernetes API-server Prometheus metrics. When an explicit kube-apiserver scrape is configured, validate that additional source with:

# Confirm that a recent Kubernetes API-server request metric is arriving.
# Replace the Unix timestamps with a recent validation window.
tsuga aggregation scalar \
  --data '{
    "timeRange": {"from": <from-unix-seconds>, "to": <to-unix-seconds>},
    "dataSource": "metrics",
    "queries": [{
      "aggregate": {"type": "sum", "field": "apiserver_request_total"},
      "filter": "context.k8s.cluster.name:<cluster>"
    }],
    "formula": "q1"
  }'

A result confirms the API-server scrape is exporting request telemetry. If the metric name is unknown or the command returns no metric, inspect the metric catalog for the name emitted by the configured scrape before changing the query.

Level 3 - Container logs with Kubernetes context and trace correlation

Collect container stdout/stderr through the node agent’s file-based log path. Parse the container log wrapper, then structured JSON where used. Enrich the log records with Kubernetes context so that at minimum the cluster, namespace, pod, container, node, and owning workload can be used to investigate them.

Keep service.name consistent across logs, metrics, and traces. Promote trace and span IDs to top-level trace_id and span_id fields. In shared receivers, derive service identity from Kubernetes metadata or structured logs.

Exit criteria: a known container log can be found in Tsuga and filtered by cluster, namespace, pod, and service. A log emitted during a traced request links back to the trace.

Validate with the Tsuga CLI

# Confirm that a known container log arrived with Kubernetes and service identity.
tsuga logs search \
  --query "context.k8s.cluster.name:<cluster> context.k8s.namespace.name:<namespace> context.service.name:<service> <unique-log-token>" \
  --from -15m \
  --to now \
  --max-results 10

At least one result confirms that the new log path is exporting a container log with the expected context.

Level 4 - Kubernetes inventory: make the cluster explorable

Once the Kubernetes metrics and object events arrive with their identity attributes, validate the Kubernetes views in Tsuga. They should make clusters, namespaces, Deployments, StatefulSets, DaemonSets, and Pods discoverable, with pivots to logs, spans, and Analytics.

Inventory is a data-quality gate. Check Kubernetes metrics, object events, and k8s.cluster.name, namespace, pod, node, and workload identities before changing application SDKs.

Exit criteria: the target cluster and its agreed namespaces/workloads appear in the correct views. Pods show health/resource context and can pivot to scoped logs and spans. Internal namespaces are deliberately included or excluded.

Level 5 - Kubernetes dashboards: the default operating view

Deploy the agreed Kubernetes dashboard pack once the underlying data is proven. The Collector chart establishes telemetry and inventory. Dashboards are a separate operational artifact, so confirm what “default dashboards” means for the customer rather than assuming the chart provisions them.

Start with cluster and scoped views for nodes, namespaces, workloads, pods, resource pressure, readiness, restarts, and Collector health. Add agreed filters and validate every widget against its source signal.

Exit criteria: the agreed default Kubernetes dashboards are deployed, owned by the right team, and usable with the agreed filters. A user can pivot from a concerning chart point into logs or traces.

Level 6 - Monitors: alert on what requires action

Create monitors only after the data, dashboards, and ownership are trustworthy. Scope them to the correct Tsuga cluster and owning team, assign a priority, and connect them to notification rules. A monitor definition is not proof that alerts are being received, so test the notification route and a maintenance silence as part of handover.

Start with the conditions that have an immediate operating response:

• Node readiness and node resource pressure.
• Pod crash/restart, pending, or readiness degradation.
• Deployment, StatefulSet, and DaemonSet replica shortfall.
• Sustained CPU or memory saturation, using requests/limits as denominators where available.
• Collector health: export errors, queue pressure, drops, or loss of node-agent coverage.

Use thresholds for understood limits. Add anomaly or new-error-pattern monitors only when they support a clear response. Aggregate short-lived pod and event signals before paging.

Exit criteria: each selected monitor is active, owned, scoped, tagged, and routed. The team has tested a notification and knows the expected response. Dashboard and monitor coverage are reviewed as the cluster changes.

Troubleshooting path

Validate from sender to Tsuga: export acceptance, Collector export, then expected traces, metrics, and logs. For incomplete inventory, check the Kubernetes metric family and identity attributes, then object events, collection, and RBAC. Repair the missing collection path before changing application instrumentation.

Completion criterion

A known container log appears in Tsuga with the agreed cluster, namespace, pod, and service.name, and the same cluster is visible in the Kubernetes views with recent node and workload metrics.