Dashboards, alerts, and debugging runbook

Overview

Phases 1-13, 18 and 19 emit the telemetry; this phase makes it usable. Provision hango dashboards into the Grafana LGTM stack, wire Tempo↔Loki↔Prometheus correlation so one click moves between a trace, its logs and its metrics, add alert rules for the failure modes the earlier phases named, codify Langfuse tag and metadata conventions, and write the end-to-end runbook that takes a failed run id to the exact span, log line and tool call that caused it. No new server features: this phase turns “instrumented” into “operable”.

Requirements

Functional

• Provisioned Grafana dashboards (JSON in the repo, mounted into lgtm):
  • hango-runs: runs by state and tenant, queue depth per lane, oldest queued run age, run claims won/lost and reclaimed runs, run duration p50/p95/p99, iterations histogram, cancellations, drain events.
  • hango-llm: token usage by model/type/purpose, cost by tenant/model/purpose, model-call latency p50/p95, time-to-first-chunk, cache-read ratio, provider error and retry rates, compaction rate.
  • hango-tools: tool call volume, duration per tool, error rate per tool, repeat-call blocks, result offloads, filesystem operations.
  • hango-api: request rate, latency and status by http.route (the label added in phase 1), auth failures by reason including pre-auth rejections, rate-limit and quota rejections, SSE subscribers, broadcaster topic count and lagged subscribers, plus an Audit panel group: hango.audit.events by action and by outcome, the denied rate (outcome="denied" over all events), and hango.audit.write_failures.
  • hango-channels: the two ways in and out that do not go through hango’s HTTP API. Telegram: updates by outcome, unknown senders, Bot API calls by method and status, Bot API latency, and hango.telegram.edit_lag p50/p95. Claude Code CLI: running processes, exits by outcome, bridge tool calls by tool and status, unauthorized bridge requests, and stream parse errors.
  • hango-runtime: Go runtime metrics (goroutines, heap, GC pause, threads) plus pgx pool saturation.
• Tempo data source tracesToLogsV2 configured with datasourceUid, tags (hango.tenant.id, hango.run.id), filterByTraceID: true, and the span time shifts; Loki derived field extracting trace_id into a Tempo link.
• Alert rules (Grafana unified alerting, provisioned as YAML):
  • run failure rate above a threshold over 15m;
  • provider 429/5xx spike;
  • queue depth above a threshold for 5m;
  • usage-record write failures non-zero;
  • unknown-model pricing events non-zero;
  • SSE lagged subscribers spiking;
  • /readyz failing or drain exceeding the timeout;
  • authentication-failure rate above a threshold over 5m, split by `reason`, with pre-auth rejections as a separate series (the brute-force signal);
  • `hango.run.oldest_queued_age` above a threshold for 5m (runs stranded in `queued`);
  • `hango.run_events.write_failures` non-zero;
  • audit denied-rate spike: `hango.audit.events{outcome="denied"}` as a share of all audit events above a threshold over 15m, which is the signal that someone is probing the API with credentials that no longer carry the access they used to, and `hango.audit.write_failures` non-zero (an audit gap is itself an incident);
  • Telegram Bot API failing: hango.telegram.api_calls with status other than ok above a threshold share over 10m, or no successful getUpdates call for 5m (the bot is deaf);
  • Claude Code CLI failing: hango.claudecli.exits with outcome = error or session_missing above a threshold over 30m, or hango.claudecli.stream_parse_errors non-zero (a CLI upgrade changed the stream format);
  • hango.claudecli.unauthorized non-zero (something other than the owner’s CLI process called the tool bridge).
• Langfuse conventions document: which tags, which langfuse.trace.metadata.* keys (tenant_id, agent_id, run_id, lane, subagent depth), how to find a run, how to compare prompt versions, and what is deliberately not sent (prompt bodies unless enabled).
• docs/learning/14-debugging-a-failed-run.md: the runbook, written as a literal walkthrough of a real failed run.

Non-functional

• Dashboards are provisioned from files, not hand-built in the UI, so they are reviewable and reproducible.
• Every panel query uses only metrics and label keys that earlier phases actually emit. The verification is a **simple label-key check**, not a regex extraction of PromQL: each dashboard JSON carries a `hango.metrics` array listing the metric constants it uses, and the test asserts every entry exists in `internal/telemetry/metrics_names.go` and that every label key referenced in a panel's legend or variable is one the emitting phase declares. Parsing arbitrary PromQL with a regex is not a contract worth maintaining.
• `metrics_names.go` already exists from phase 1 and every phase, including 18 and 19, registered through it, so this phase contains **no metric-name retrofit refactor**.
• Alert thresholds are config-documented defaults, not tuned guesses presented as truth.

Architecture

No Go interfaces and no tables. The deliverables are provisioning artifacts plus one verification test.

deploy/grafana/
  dashboards/hango-runs.json
  dashboards/hango-llm.json
  dashboards/hango-tools.json
  dashboards/hango-api.json
  dashboards/hango-channels.json
  dashboards/hango-runtime.json
  provisioning/dashboards.yaml        # file provider pointing at /var/lib/grafana/dashboards
  provisioning/datasources.yaml       # Tempo (tracesToLogsV2) + Loki (derived field) + Prometheus
  provisioning/alerting.yaml          # alert rules and a contact point placeholder

Instrumentation

• This phase adds no new spans; it adds the hango.build.version and deployment.environment.name resource attributes as dashboard variables, and a single synthetic hango.telemetry.selftest gauge emitted at startup so a dashboard can prove the pipeline is alive even when no traffic exists.
• A make obs-verify target sends one synthetic run through the system and asserts each dashboard’s key series is present in Prometheus and the trace is queryable in Tempo.

Related Code Files

Create

• deploy/grafana/dashboards/*.json (six dashboards).
• deploy/grafana/provisioning/dashboards.yaml, datasources.yaml, alerting.yaml.
• internal/telemetry/selftest.go — emits hango.telemetry.selftest at startup with version and environment attributes.
• internal/telemetry/dashboards_test.go — reads each dashboard’s declared hango.metrics list and its panel label keys and asserts they exist in metrics_names.go and in the declared label sets.
• docs/observability.md — data sources, dashboards, alerts, Langfuse conventions, what is and is not sent.
• docs/learning/14-debugging-a-failed-run.md.

Modify

• compose.yaml — mount deploy/grafana/dashboards and deploy/grafana/provisioning into the lgtm container (documented mount points /otel-lgtm/... and the Grafana provisioning path), and add the GF_* variables needed for provisioning.
• Makefile — obs-up, obs-verify, obs-reset.
• internal/telemetry/telemetry.go — call the self-test emitter.
• No metric-registration refactor: phases 1-13, 18 and 19 already register through `metrics_names.go`. This phase only adds any constant a dashboard needs that no phase emitted, which is a defect in that phase, not a refactor here.
• README.md, docs/system-architecture.md (Observability section becomes a pointer to docs/observability.md), docs/learning/README.md.

Implementation Steps

1. Audit `internal/telemetry/metrics_names.go` (built incrementally since phase 1) against what phases 1-13 actually emit, together with phases 18 and 19, and fix any phase that emitted a literal. This is the contract the dashboards are built against; it is verified, not created, here.
2. Write internal/telemetry/selftest.go emitting hango.telemetry.selftest = 1 at startup with service.version and deployment.environment.name.
3. Author deploy/grafana/provisioning/datasources.yaml: Prometheus, Tempo with jsonData.tracesToLogsV2 (datasourceUid: loki, tags: [{key: "hango.tenant.id"}, {key: "hango.run.id"}], filterByTraceID: true, spanStartTimeShift: -5m, spanEndTimeShift: 5m), and Loki with a derived field extracting trace_id and linking to Tempo. Note in a comment that lgtm provisions its own data sources, so these files override them by uid.
4. Author the six dashboards. Build each panel from the metric constants; include template variables for tenant_id, agent_id, model and environment; every latency panel uses histogram quantiles, every error panel uses a rate ratio.
5. Author deploy/grafana/provisioning/alerting.yaml with the fifteen rules, each carrying a summary that names the likely cause and links to the relevant runbook section. Phases 15 and 16 each add their own rules to this file as an explicit step (approvals-pending gauge; RLS denials), so the alert set stays current with the system.
6. Mount everything in compose.yaml and add the Makefile targets.
7. Write internal/telemetry/dashboards_test.go: read each dashboard’s declared hango.metrics list plus its panel label keys and fail if any is absent from metrics_names.go or from the declared label set. Simple, deterministic, and no PromQL parsing.
8. Write make obs-verify: start the stack, create a tenant/user/key/agent, run a scripted run that includes a tool call, a compaction and a subagent, point the Telegram bot at a fake Bot API server through HANGO_TELEGRAM_API_URL and deliver one update from a mapped user, then query Prometheus for the key series and Tempo for the trace id returned by GET /v1/runs/{id}, failing on any miss.
9. Write docs/observability.md: the data flow diagram (app → collector → lgtm/Langfuse), the attribute dictionary (hango.*, GenAI, langfuse.*), the dashboard catalogue, the alert catalogue with default thresholds, the Langfuse tag and metadata conventions, and the content-privacy policy (HANGO_TRACE_LLM_CONTENT, collector stripping).
10. Write docs/learning/14-debugging-a-failed-run.md as the runbook (outline below), using a real failed run produced deliberately (a tool that errors inside a subagent after a compaction) and capturing the exact queries and clicks.
11. Run the full gate plus make obs-verify.

Learning Doc Outline

docs/learning/14-debugging-a-failed-run.md must contain:

• Concept: observability as a debugging contract — what questions the system must be able to answer, and the attribute chain tenant → agent → session → run → turn → model call → tool call that answers them.
• The runbook, as a numbered walkthrough from a single failed run id:
  1. GET /v1/runs/{id} → read state, error, trace_id, iterations, usage.
  2. Tempo: paste the trace id → read the invoke_agent root, find the first span with an error status, note whether it is chat or execute_tool.
  3. Span attributes → the exact model, finish reason, tool name, tool call id, and whether the turn was compacted.
  4. Tempo → Loki via tracesToLogsV2 → the log lines for that span, filtered by hango.run.id.
  5. Langfuse: open the trace by metadata.run_id → inspect the generation input/output (when content tracing is enabled) and the nested subagent observations → identify the prompt that produced the bad tool call.
  6. GET /v1/runs/{id}/events replay → what the client actually saw.
  7. read_file /large_tool_results/{tool_call_id} and /conversation_history/{session_id}.md → the payloads the model saw but the trace does not carry.
  8. `GET /v1/audit-events?since=` for the run's tenant around the failure time, joined by `trace_id` — who changed the agent, the quota, a file or an approval just before the run broke, and whether anything was denied. This is the step that separates "the system failed" from "someone changed something".
  9. Metrics: is this one run or a pattern (error rate per tool, per model, per tenant)?
• Seven worked failure signatures with the first query for each: provider rate limit storm; context overflow after a failed compaction; tool timeout inside a subagent; cross-tenant 404 caused by a wrong API key; SSE client seeing nothing because the subscriber lagged; a family member’s Telegram message that got no reply (start from telegram.update by outcome, then the span link into the run); a Claude CLI run that failed after an upgrade (claudecli.process exit outcome, then the captured stderr in Loki).
• How to debug this layer itself: no data in Grafana (collector logs, endpoint, HANGO_OTEL_ENABLED), data in Tempo but not Langfuse (basic auth, ingestion-version header, the collector’s Langfuse exporter queue), metrics present but dashboards empty (metric-name drift, caught by dashboards_test.go).
• goclaw comparison: DB-backed span storage with periodic flush and a build-tag-gated OTLP exporter, i.e. a bespoke tracing system with no dashboards or correlation; the trade-off is self-contained operation versus an ecosystem of tools.
• deepagents comparison: observability comes from LangSmith, which gives trace trees and prompt inspection for free but no infrastructure metrics — the Langfuse leg of hango’s pipeline is the deliberate equivalent, with Grafana covering what LangSmith does not.
• Why hango diverges: standard OTel with collector fan-out, so LLM-level and infrastructure-level debugging use the right tool for each.
• Exercises: deliberately break a tool and follow the runbook end to end; disable the collector and observe which alert fires.
• Time budget: 10h implementation + 2h learning doc (the runbook is the learning doc).

Success Criteria

• go build ./... && go vet ./... && go test ./... pass, including dashboards_test.go.
• docker compose up provisions all six dashboards and the three data sources into Grafana with no manual steps.
• From a trace in Tempo, one click reaches the correlated Loki logs; from a Loki line, one click reaches the trace.
• make obs-verify passes: the synthetic run’s trace is queryable by the trace_id from the API, and every dashboard’s key series exists in Prometheus.
• All fifteen alert rules load without error and each has a summary naming a likely cause.
• The hango-api Audit panel group renders hango.audit.events split by action and outcome against a real run, and the runbook’s audit step resolves a deliberately denied mutation to the exact audit_events row and its trace.
• docs/observability.md documents every hango.* attribute and metric emitted by phases 1-13, 18 and 19, verified against metrics_names.go.
• docs/learning/14-debugging-a-failed-run.md walks a real failed run end to end with the actual queries used.
• A deliberately failed run can be diagnosed to the exact tool call in under five minutes following only the runbook (timed, recorded in the doc).

Risk Assessment

Rollback

Pure artifacts. Removing the provisioning mounts returns Grafana to lgtm’s defaults; the application keeps emitting telemetry unchanged, so nothing observable regresses.