Evaluation loop

Overview

Requirements

Functional

• A task set lives in evals/tasks/ as one JSON file per task: id, a short description, the agent spec to create (model, provider, tools, system_prompt, optional subagents), the user input, and a list of checks. The set ships with 8-12 tasks that cover the harness features built so far: plain answer, single tool call, tool error recovery, planning with write_todos, a file written through the virtual filesystem, a subagent delegation, a long conversation that triggers compaction, and memory read on a second session.
• Checks are deterministic and closed: final_contains (substring), final_matches (regex), tool_called (name, optional minimum count), tool_not_called, file_exists (virtual path), file_contains (path and substring), max_turns, max_cost_micro_cents, run_state (expected terminal state, default succeeded). Each check scores 0 or 1. A task’s score is the mean of its checks, and it passes only when every check scores 1.
• hango eval run --tasks evals/tasks --label <name> [--repeat N] [--only <id,...>] reads HANGO_EVAL_BASE_URL and HANGO_EVAL_API_KEY (an owner key in a tenant reserved for evals), then for each task and repetition: upserts the agent by slug eval-<task-id>, creates a session, creates a run, follows GET /v1/runs/{id}/events until the run is terminal, reads the final messages, the run’s trace_id, the files the checks name, and the run’s token and cost totals, and scores the checks.
• A run that ends in awaiting_approval fails its task with reason awaiting_approval; eval agents configure no interrupt_on rules beyond the defaults, so this only happens when a task writes to a reserved prefix.
• Results are written to evals/results/<label>.json (git-ignored): per task and repetition the score, each check’s outcome and reason, run id, trace id, turns, tokens, cost and duration, plus the git commit and the model of each task.
• When LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY are set, the command upserts a Langfuse dataset hango-eval with one item per task, links each run’s trace to the dataset run named <label>, and posts one score per check plus a task_score on the trace. Langfuse’s trace id is the OTel trace id, so the score lands on the trace the run already produced. With the keys unset, the command logs that Langfuse push is skipped and still writes the results file.
• hango eval compare <base.json> <candidate.json> [--max-regression 0] prints a table per task (pass rate, mean score, turns, tokens, cost, duration, each as base → candidate with the delta), then the totals, and exits non-zero when the candidate’s overall pass rate falls by more than --max-regression percentage points.
• GET /v1/usage accepts a run_id filter so the eval command can read one run’s token and cost totals. The (tenant_id, run_id) index on usage_records from phase 13 already serves it.

Non-functional

• No new package. The command lives in cmd/hango next to probe and migrate; it is an HTTP client of hango’s public API plus a small Langfuse HTTP client, so it exercises the same auth, tenancy, audit and telemetry paths a real user does. The package cap stays at 18 plus internal/tool/builtin.
• Standard library only for the client code: net/http, encoding/json, regexp. No Langfuse SDK.
• The command never prints the API key or the Langfuse secret, and the results file never contains prompt bodies beyond the final assistant text the checks read.
• Repeats exist because model output is nondeterministic. The compare output reports pass rate over repetitions, not a single pass or fail, so one lucky or unlucky sample does not decide a change.

Architecture

// cmd/hango/eval_task.go
type evalTask struct {
	ID          string          `json:"id"`
	Description string          `json:"description"`
	Agent       json.RawMessage `json:"agent"` // POST /v1/agents body, slug forced to eval-<id>
	Input       string          `json:"input"`
	Checks      []evalCheck     `json:"checks"`
}

type evalCheck struct {
	Kind  string `json:"kind"` // final_contains|final_matches|tool_called|tool_not_called|file_exists|file_contains|max_turns|max_cost_micro_cents|run_state
	Value string `json:"value,omitempty"`
	Path  string `json:"path,omitempty"`
	Min   int    `json:"min,omitempty"`
}

// cmd/hango/eval_score.go
type checkResult struct {
	Kind, Reason string
	Score        float64 // 0 or 1
}

// observed is everything a check may read about one finished run.
type observed struct {
	State, FinalText, TraceID string
	ToolCalls                 map[string]int
	Files                     map[string]string // path -> content, only the paths the checks name
	Turns                     int
	CostMicroCents, Tokens    int64
	Duration                  time.Duration
}

func score(c evalCheck, o observed) checkResult

evals/
  tasks/plain-answer.json
  tasks/tool-call.json
  tasks/tool-error-recovery.json
  tasks/plan-with-todos.json
  tasks/write-file.json
  tasks/delegate-to-subagent.json
  tasks/long-conversation-compaction.json
  tasks/memory-second-session.json
  results/                 # git-ignored, one <label>.json per eval run

hango eval run --label base
  for each task, for each repetition:
    upsert agent eval-<id> → POST session → POST run → follow SSE to terminal
    read messages, files, usage(run_id), trace_id → score checks
    Langfuse: dataset item, dataset run item (trace_id, run name = label), scores on trace_id
  write evals/results/base.json
hango eval compare evals/results/base.json evals/results/candidate.json

Instrumentation

• The eval command sends no telemetry of its own. Every run it creates is a normal hango run, so its invoke_agent trace, logs, metrics, audit rows and Langfuse generations already exist; the scores attach to that trace.
• The results file stores each run’s trace_id, so a failed check opens straight into Tempo and Langfuse with the phase-14 runbook.

Related Code Files

Create

• cmd/hango/eval.go — the eval run and eval compare subcommands, flag parsing, the task loop and the results file.
• cmd/hango/eval_task.go — task and check types, loading and validating evals/tasks/*.json.
• cmd/hango/eval_score.go, cmd/hango/eval_score_test.go — score for every check kind, table-tested against hand-built observed values.
• cmd/hango/eval_client.go — the hango API client: agent upsert, session, run, SSE follow, messages, files, usage.
• cmd/hango/eval_langfuse.go, cmd/hango/eval_langfuse_test.go — dataset, dataset run item and score calls against httptest.Server.
• cmd/hango/eval_compare.go, cmd/hango/eval_compare_test.go — the comparison table and the regression exit code.
• evals/tasks/*.json — the task set.
• docs/learning/17-evaluation-loop.md.

Modify

• internal/server/handlers_usage.go, internal/store/pg/usage.go — the run_id filter on GET /v1/usage.
• .gitignore — evals/results/.
• Makefile — eval and eval-compare targets.
• .env.example — HANGO_EVAL_BASE_URL, HANGO_EVAL_API_KEY placeholders.
• README.md, docs/learning/README.md, docs/system-architecture.md.

Implementation Steps

1. Write eval_task.go: load every *.json in the task directory, reject unknown check kinds, duplicate ids and a missing input, and sort tasks by id so runs are reproducible.
2. Write eval_score.go and its table test first: one case per check kind for both outcomes, plus the empty-checks task (invalid at load time, never scored).
3. Add the run_id filter to GET /v1/usage with an integration test that two runs in one session report separate totals.
4. Write eval_client.go: upsert the agent by slug, create the session and run, follow the SSE stream with Last-Event-ID resume until a terminal state or a per-task timeout (--timeout, default 5m), then read messages, the named files and usage. Count turns and tool calls from the persisted messages, not from the stream, so a resumed stream cannot double-count.
5. Write eval.go for eval run: loop tasks and repetitions sequentially, score, write the results file atomically (write to a temp file, then rename), and print a one-line summary per task.
6. Write eval_langfuse.go: upsert dataset hango-eval and its items, create the dataset run item linking the trace to the run name <label>, post one score per check and a task_score. Test the request shapes against httptest.Server. Skip with a log line when the keys are unset.
7. Write eval_compare.go and its test: align tasks by id, compute pass rate over repetitions, mean score, and medians for turns, tokens, cost and duration, print the table, and return the exit code from --max-regression.
8. Write the task set. Start every task from a real phase capability, keep checks deterministic, and give each task a cost ceiling so a runaway loop fails the check instead of draining the budget.
9. Run make eval LABEL=base against docker compose up, change one thing on purpose (for example remove the planning middleware from the stack or switch the model), run make eval LABEL=candidate, and record the eval compare output and the Langfuse dataset-run comparison in the learning doc.
10. Update the Makefile, .gitignore, .env.example, README and docs, and write the learning doc.

Learning Doc Outline

docs/learning/17-evaluation-loop.md must contain:

• Concept: tests prove mechanics, evals measure behaviour. Why a harness change (prompt, middleware order, compaction threshold, model) needs an eval run before and after, and why the task set is versioned with the code.
• Design choices: deterministic checks before model-graded ones, pass rate over repetitions instead of a single sample, cost and turn ceilings as checks, the eval command as an ordinary API client so it goes through auth, tenancy and audit like any user.
• Trade-offs: deterministic checks vs an LLM judge (cheap and stable vs able to grade open-ended answers); a small fixed set vs a large sampled one; offline results file vs Langfuse datasets (both are kept: the file is the CI gate, Langfuse is the place to read traces side by side).
• How Langfuse datasets, dataset runs and scores fit together, and why the OTel trace id is what lets a score land on the run’s existing trace.
• goclaw comparison: what goclaw uses to check agent behaviour, if anything, and where it would plug in.
• deepagents comparison: LangSmith datasets and evaluators as the LangChain answer, and how hango’s Langfuse path maps onto them.
• How to debug this layer: a failed check names its reason and trace id; open the trace in Langfuse to read the generations, then follow the phase-14 runbook. A task that flips between pass and fail across repetitions is a flaky task, not a regression, and the compare table shows it as a pass rate below 100%.
• Follow-ups: an LLM-judge check kind, running a small subset in CI against a recorded provider, and a larger task set built from real failed runs.
• Exercises: add a task for a bug you fixed in an earlier phase; break one middleware and watch which tasks drop; run --repeat 5 on one task and read its pass rate.
• Time budget: 8h implementation + 2h learning doc.

Success Criteria

• go build ./... && go vet ./... && go test ./... pass, and the package list is unchanged (no new package).
• eval_score_test.go covers every check kind for both outcomes.
• Loading a task with an unknown check kind, a duplicate id or no input fails with a message naming the file.
• GET /v1/usage?run_id= returns one run’s totals; an integration test with two runs in one session shows separate totals.
• make eval LABEL=base against docker compose up runs every task and writes evals/results/base.json with a score, a trace id and the cost for each task.
• With Langfuse keys set, the dataset hango-eval shows a dataset run named base, and each linked trace carries one score per check plus task_score (manual check, recorded in the learning doc).
• With Langfuse keys unset, hango eval run logs that the push is skipped and still writes the results file.
• hango eval compare prints per-task pass rate, score, turns, tokens, cost and duration deltas, and exits non-zero when the pass rate drops by more than --max-regression (test).
• A deliberate regression (a middleware removed or the model changed) shows up as a lower pass rate in eval compare, recorded in the learning doc.
• Neither stdout nor the results file contains the API key or the Langfuse secret (test over a run against httptest.Server).

Risk Assessment

Rollback

The eval command and the task set are additive and live outside the server path. Removing cmd/hango/eval*.go and evals/ returns the binary to phase 16 behaviour. The run_id filter on GET /v1/usage is a backward-compatible query parameter and can stay.