Root Cause Analysis Implementation Guide

Overview

This path sets up a root-cause-analysis workflow that runs in the engineer’s existing local agent: Claude Code, Codex, Cursor, or another CLI-capable LLM agent. The agent reasons locally, where it can reach the codebase and approved internal context.

Tsuga provides the observable evidence. Git provides change history. The incident platform and documentation provide operational context.

The investigation remains human-led. The agent should gather evidence, test hypotheses, and produce a reviewable conclusion. It should not receive broad write access or make a production change as part of its default RCA workflow.

Local LLM agent
      │
      ├── Tsuga CLI or MCP ─── logs, traces, metrics, alerts, assets
      ├── GitHub CLI + repos ── code, PRs, releases, deployments
      ├── Incident platform ─── timeline, responders, customer impact
      └── Docs / prior RCAs ─── ownership, runbooks, known failure patterns
                                      │
                                      ▼
                        Evidence-backed RCA and next actions

When to use

Use this path when the user wants their own developers or responders to investigate an active incident or a past failure from a local coding agent.

The incident platform retains ownership of communication, severity, responders, timelines, and decisions. The agent accelerates evidence gathering and explanation.

Step 1 - Choose the local agent and scope its authority

Choose an agent that runs local commands and connects to MCP servers. Use Claude Code, Codex, or Cursor only when its model, billing, data handling, and installation model are approved. Browser-only chat cannot inspect local repositories or run repeatable investigation commands.

Before installing anything, agree:

• who can run the agent and which team is included in the pilot.
• which model provider and data-residency policy are approved.
• whether source code, telemetry, incident transcripts, and documentation may be included in the model context.
• the maximum access required for investigation.
• that investigation access is read-only by default.

Use OAuth (recommended) or give every investigator an individual credential. Avoid a shared Tsuga or GitHub token: individual identity makes access review, revocation, and audit practical.

Exit criteria: one approved local agent can run commands, and the pilot’s access policy explicitly separates read-only investigation from remediation.

Step 2 - Connect the agent to Tsuga through the CLI or MCP

Use the Tsuga CLI as the primary interface because it is explicit, testable outside the agent, and uses less model context. Use Tsuga MCP when the client’s tool discovery or interactive UI benefits the workflow.

Start individually operated agents with OAuth. The CLI uses browser login, and OAuth-capable MCP clients authorize in a browser on first connection. Requests use the individual’s Tsuga permissions and avoid a stored Operation key. See Connect your AI agent for the access-path comparison.

Use an Operation key when OAuth is unavailable or the integration is shared automation. Create one named key per agent runtime. For a first pilot, grant telemetry and asset read access only, scope it to the owning team where possible, and name it after the individual and machine.

CLI setup with OAuth

npm install -g @tsuga/cli
tsuga auth login
tsuga auth status

# Smoke test the connection with scoped, time-bounded reads
tsuga services list
tsuga logs search \
  --query 'context.service.name:<service>' \
  --from -15m \
  --to now \
  --max-results 10
tsuga traces search \
  --query 'context.service.name:<service>' \
  --from -15m \
  --to now \
  --max-results 10
tsuga monitors list -d '{"limit":50}'

Each search returns up to ten recent records for one service. Require a service and explicit time range in every query. Bounds protect model context and expose relevant evidence. See Tsuga CLI for installation, browser login, and credential precedence.

CLI setup with an Operation key

Use this route only when OAuth is unavailable or unsuitable for the agent runtime. Store the key in the platform secret store or environment, never in a repository file.

export TSUGA_OPERATION_API_KEY="<TSUGA_OPERATION_KEY>"
tsuga auth operation-key "$TSUGA_OPERATION_API_KEY"
tsuga auth status

The saved Operation key replaces any saved OAuth session. To avoid saving the key in the CLI configuration, retain the environment variable and omit tsuga auth operation-key.

The agent needs guidance on how to construct Tsuga queries and interpret metrics. Install the Tsuga investigation skills or plugin supported by the chosen agent, and keep them updated. For Claude Code, the current marketplace flow is:

claude plugin marketplace add tsuga-dev/agent-plugins
claude plugin install tsuga@tsuga

MCP setup with OAuth

Use the hosted streamable HTTP server at https://mcp.tsuga.com/v0/mcp. For Claude Code, register the server without an authorization header, then approve the browser sign-in. Open /mcp if Claude Code does not open the browser automatically.

claude mcp add --transport http tsuga https://mcp.tsuga.com/v0/mcp

For Codex, add the server to ~/.codex/config.toml, then start the OAuth flow:

[mcp_servers.tsuga]
url = "https://mcp.tsuga.com/v0/mcp"

codex mcp login tsuga

For Cursor and other supported clients, use the OAuth configuration in Tsuga MCP. It contains the current client-specific setup steps.

MCP setup with an Operation key

Use this route only for a client without OAuth support or for shared automation. Store the key as TSUGA_MCP_TOKEN in the environment or the agent’s secret store. Do not place it in a repository configuration file.

export TSUGA_MCP_TOKEN="<TSUGA_OPERATION_KEY>"
claude mcp add --transport http tsuga https://mcp.tsuga.com/v0/mcp \
  --header "Authorization: Bearer $TSUGA_MCP_TOKEN"

Claude Code copies the token into its registered header. Register the server again after rotating the key. For Codex, Cursor, and other supported clients, use the Operation-key configuration in Tsuga MCP.

Exit criteria: outside the agent and then inside it, list services and retrieve a scoped log or trace. The agent must not be given a write-capable key merely to pass this test.

Step 3 - Give the agent useful GitHub and code access

Telemetry shows what changed in production. Source code and Git history help explain why. Give the agent read access to repositories that build and deploy in-scope services, including pull requests, commits, releases, workflow runs, and deployment metadata.

On each pilot machine:

# Authenticate through the organization’s approved GitHub SSO flow
gh auth login
gh auth status

# Keep relevant repositories under a documented root
mkdir -p ~/repos/<organization>
cd ~/repos/<organization>
git clone [email protected]:<organization>/<repository>.git

Use a documented repository root. For each service, identify the code, deployment, infrastructure-as-code, and CI/CD workflow locations. In a monorepo, also record the service directory and deployment manifest.

The minimum useful GitHub permissions are read access to the repositories and actions needed for correlation. Do not grant repository administration, secrets access, merge permission, or deployment approval as part of the RCA setup.

Exit criteria: given a service.version or deployment revision observed in Tsuga, the agent can find the corresponding commit, pull request, release, and workflow history.

Step 4 - Add the incident platform as the source of incident context

Connect the agent to the approved incident system, such as incident.io, PagerDuty, FireHydrant, Opsgenie, Jira, or ChatOps. Supply the initial question, time window, severity, affected customers, responders, and mitigation status. Use recorded incident facts rather than inferring them from a telemetry spike.

Use one of these patterns, in order of preference:

1. Incident prompt or export: copy the incident summary, links, and absolute time range into the local agent. This is the most predictable pilot path.
2. Read-only API or MCP connection: expose only the incident fields, timeline, and linked resources needed for investigation.
3. Curated incident archive: transform resolved incident reports and threads into reviewed, local knowledge for recurring patterns.

Keep live incident communication under human control. The agent may draft a status update, but it should not change severity, add responders, resolve the incident, or post externally without explicit approval.

Use the incident context to form a bounded question before the first telemetry query:

Investigate the production checkout failure from 10:03 to 10:19 UTC.
Start with the frontend error-rate alert. Determine the affected operation,
the earliest error, the downstream propagation path, relevant changes, and
the strongest supported root-cause hypothesis. Cite evidence and open questions.

Exit criteria: the agent receives a reproducible incident window and impact statement before it begins searching telemetry.

Step 5 - Provide documentation, ownership, and previous incidents

RCA also needs the operational knowledge engineers normally supply from memory: ownership, deployment, dependencies, expected behavior, and prior failures.

Begin with approved, structured sources:

• service catalog and ownership map.
• runbooks and architecture documentation.
• deployment and rollback procedures.
• post-mortems or incident records.
• known customer or tenant context when it is allowed and relevant.

Start with approved, read-only sources. Broad live search adds noise, consumes tokens, and can expose irrelevant sensitive data. Prefer a small source or curated local bundle with reviewed service facts and incident summaries.

A useful incident summary records the verified diagnostic path: Saw → Check → Confirms. For example: “Saw checkout errors after version X. Checked the downstream quote-service traces and logs. Confirmed an uncaught exception introduced by the deployment.” This gives future investigations a pattern to test without treating the prior cause as the current answer.

Exit criteria: for every pilot service, the agent can find an owner, a runbook or architecture reference, and any approved relevant prior RCA.

Step 6 - Define the investigation contract and guardrails

Give the agent a clear task shape. It should start with the incident window and symptom, then narrow rather than sweep every source.

1. Restate the incident question, time range, service, and environment.
2. Find the onset and affected operation from monitor and metric evidence.
3. Inspect representative failing traces and follow the downstream path.
4. Pivot between traces and correlated logs for concrete failure detail.
5. Compare affected versions and changes against a pre-incident baseline.
6. Inspect code, deployment history, and the relevant runbook only after a lead exists.
7. Report observations, hypothesis, counter-evidence, confidence, and next checks.

Require the final output to include:

• incident impact and affected scope.
• a short causal chain, separating symptom from likely cause.
• links, queries, or commands supporting each material claim.
• uncertainty and rejected alternatives.
• recommended next actions, clearly marked as suggestions rather than actions already taken.

Pre-approve only read-only commands, such as tsuga searches, gh reads, git log, and rg. Require approval for deployments, repository writes, secret access, and production database changes.

Exit criteria: two investigators using the same prompt can reproduce the key evidence and understand which conclusions are proven versus suspected.

Step 7 - Validate with a known incident before going live

Choose a resolved incident from the previous six to twelve months. Give the agent only evidence available before the root cause and remediation were recorded. This prevents answer leakage during the test.

Validate that it can:

1. connect to Tsuga and GitHub without credential errors.
2. locate the original alert, symptom, and incident time range.
3. find the relevant metrics, traces, and logs.
4. correlate the observed version or infrastructure change to source history.
5. produce a conclusion consistent with the recorded RCA.
6. identify any missing access, telemetry, or documentation rather than filling the gap with a confident guess.

Measure time to a supported conclusion, irrelevant searches, token cost, and evidence quality. Address gaps in access, instrumentation, skills, or documentation when the replay fails.

Exit criteria: the agent provides a reviewable RCA consistent with the historical record and calls out uncertainty where the evidence is incomplete.

Final note: iterate on output quality

After every pilot or incident, review claim citations, the causal chain, and how uncertainty was handled. Compare the report with the human outcome and record the missing skill, access boundary, telemetry, or documentation behind each false lead.

Update the agent’s instructions, skills, and curated context for the recorded gap. Replay a known incident after each material change. Retain a change when it improves evidence quality, reduces irrelevant searches, or makes the conclusion easier to review.

Troubleshooting path

The agent cannot query Tsuga

Check the CLI outside the agent first: tsuga services list. If it fails, verify that you are logged in with OAuth or that the Operation API key is active, scoped to the required team, and authenticated on that machine. For MCP, confirm the server URL, the environment variable name, and that the agent was restarted after the secret was added.

The agent can see telemetry but cannot connect it to a change

Confirm that service.version is emitted as an immutable build or commit identifier and that deployment history maps that value to a release, workflow, or pull request. Check that the agent can read the relevant repository and that the right deployment repository is locally available.

The agent finds the symptom but not the failure detail

Check trace-to-log correlation: logs need the active trace_id and span_id, and traces and logs need matching service and environment identity. Then inspect asynchronous boundaries, queue consumers, worker threads, and scheduled jobs, where context propagation often breaks.

The agent makes unsupported claims

Require evidence for every material conclusion and make it label uncertainty. Reduce its access to the scoped incident, then require it to state the next query or check that would confirm the hypothesis. A plausible narrative is not an RCA.

Completion criterion

Replaying a known past incident, the agent reaches a conclusion consistent with the recorded RCA, cites the query, trace, log, or commit behind each material claim, and names what it could not determine while holding read-only access throughout.

Optional - Observe the investigator agents themselves

Agent telemetry can show investigation duration, model usage, token cost, and tool-call patterns. It can also include prompts and tool content. Choose content capture, metadata only, or no agent telemetry, and use a separate ingestion key from the investigation credential.