Usage, quotas, rate limits

Overview

Make consumption measurable and boundable. Record one usage_records row per model call (including cache tokens and the call purpose, so summarizer and subagent cost are separable), price it from a config-driven table into integer micro-cents, enforce per-tenant monthly quotas checked before a run and accounted post-hoc, and rate-limit per API key with an in-process token bucket plus a TTL sweeper. Feed the priced cost into Langfuse cost_details so cost is visible next to the trace that produced it.

Requirements

Functional

• Dependencies are `[9, 10, 12]`: the `subagent` purpose comes from phase 9, the `summarize` purpose and the compaction path from phase 10, and the second provider's models and capabilities from phase 12.
• usage_records: one row per provider call with tenant, agent, session, run, turn, provider, model, purpose (main|summarize|subagent), input/output/cache-read/reasoning tokens, cache-write split into 5m and 1h buckets, cost in micro-cents, and created_at.
• Pricing table from config: per provider+model input, output, cache-read, cache-write-5m and cache-write-1h prices per million tokens (Anthropic prices the two TTLs differently; collapsing them silently mis-prices every cached run), with an explicit unknown model behaviour (record tokens, cost zero, log once, increment a counter).
• Cost is bigint micro-cents; no floating point anywhere in the money path.
• tenant_quotas: monthly token and cost caps plus max_concurrent_runs. Checked at run creation: over quota returns 402 with the current usage; a run already started is never killed mid-turn by quota.
• Usage aggregation endpoints: GET /v1/usage?from=&to=&group_by=agent|model|purpose|day (tenant-scoped) and GET /v1/admin/usage?tenant_id= for the platform principal.
• Rate limiting: per API key x/time/rate limiter with configurable rate and burst, a sharded map with a TTL sweeper evicting idle entries, and 429 responses carrying Retry-After and X-RateLimit-* headers.
• Concurrency limit: a tenant may not exceed max_concurrent_runs active runs; excess run creation returns 429 with a clear reason distinct from the rate-limit 429.

Non-functional

• Usage recording is on the run’s critical path but must never fail a run: write failures are logged, counted and retried once, then dropped.
• Quota checks are a single indexed aggregate query over the current month, cached per tenant for HANGO_QUOTA_CACHE_TTL (default 10s).
• The in-process limiter is a documented single-replica limitation; the learning doc states exactly what breaks at two replicas and what replaces it.
• Cross-tenant isolation tests for usage_records and tenant_quotas.

Architecture

// internal/usage/record.go
type Purpose string // "main" | "summarize" | "subagent"

type Record struct {
	TenantID, AgentID, SessionID, RunID uuid.UUID
	Turn        int
	Provider    string
	Model       string
	Purpose     Purpose
	// Red team: F13 — cache writes are priced per TTL bucket.
	Input, Output, CacheRead, CacheWrite5m, CacheWrite1h, Reasoning int
	CostMicroCents int64
}

// internal/usage/pricing.go
type Price struct { // micro-cents per million tokens. Red team: F13
	InputPerMTok, OutputPerMTok, CacheReadPerMTok           int64
	CacheWrite5mPerMTok, CacheWrite1hPerMTok                int64
}

type Pricing struct{ table map[string]Price } // key "provider/model"

func (p *Pricing) Cost(r Record) (int64, bool) // false = unknown model

// internal/usage/quota.go
type Quota struct {
	MonthlyTokens, MonthlyCostMicroCents int64
	MaxConcurrentRuns                    int
}

type Checker struct{} // cached per-tenant month aggregate

func (c *Checker) Allow(ctx context.Context) (Decision, error) // called at run creation

type Decision struct {
	Allowed bool
	Reason  string // "monthly_tokens" | "monthly_cost" | "max_concurrent_runs"
	Used, Limit int64
}

// internal/server/ratelimit.go
type Limiter struct{} // sharded map[keyID]*entry{limiter, lastSeen}; TTL sweeper via safego

func (l *Limiter) Allow(keyID uuid.UUID) (ok bool, retryAfter time.Duration, remaining int)

-- migrations/00009_usage_and_quotas.sql
create table usage_records (
  id uuid primary key,
  tenant_id uuid not null references tenants(id) on delete cascade,
  agent_id uuid not null references agents(id) on delete cascade,
  session_id uuid not null references sessions(id) on delete cascade,
  run_id uuid not null references runs(id) on delete cascade,
  turn int not null default 0,
  provider text not null,
  model text not null,
  purpose text not null check (purpose in ('main','summarize','subagent')),
  input_tokens int not null default 0,
  output_tokens int not null default 0,
  cache_read_tokens int not null default 0,
  cache_write_5m_tokens int not null default 0, -- Red team: F13
  cache_write_1h_tokens int not null default 0, -- Red team: F13
  reasoning_tokens int not null default 0,
  cost_micro_cents bigint not null default 0,
  created_at timestamptz not null default now()
);
create index usage_records_tenant_created_idx on usage_records (tenant_id, created_at desc);
create index usage_records_tenant_run_idx on usage_records (tenant_id, run_id);

create table tenant_quotas (
  tenant_id uuid primary key references tenants(id) on delete cascade,
  monthly_tokens bigint not null default 0,          -- 0 = unlimited
  monthly_cost_micro_cents bigint not null default 0, -- 0 = unlimited
  max_concurrent_runs int not null default 4,
  rate_per_minute int not null default 60,
  burst int not null default 20,
  updated_at timestamptz not null default now()
);

Instrumentation

• The chat span gains hango.cost.micro_cents and hango.usage.purpose; langfuse.observation.cost_details is set from the priced record (explicit JSON, since Langfuse’s inference of semconv cache-token keys is unverified) and langfuse.observation.usage_details is reconciled with the same numbers.
• The run’s root invoke_agent span gains totals: hango.cost.micro_cents, hango.usage.input_tokens, hango.usage.output_tokens, and langfuse.trace.metadata.cost_micro_cents.
• Spans: usage.record (INTERNAL, per write) and quota.check (INTERNAL, with hango.quota.reason, hango.quota.used, hango.quota.limit, hango.quota.cache_hit).
• Metrics: hango.usage.tokens counter (attributes gen_ai.request.model, gen_ai.token.type, purpose), hango.usage.cost counter in micro-cents (attributes model, purpose), hango.quota.rejections counter (attribute reason), hango.ratelimit.rejections counter, hango.ratelimit.tracked_keys gauge, hango.usage.record_failures counter, hango.pricing.unknown_model counter (attribute model).
• Log fields: quota and rate-limit rejections log at warn with tenant id, reason, used and limit — the two events users will ask about most.

Related Code Files

Create

• internal/usage/record.go, internal/usage/pricing.go, internal/usage/pricing_test.go, internal/usage/quota.go, internal/usage/quota_test.go, internal/usage/recorder.go (the write path with retry and metrics).
• internal/server/ratelimit.go, internal/server/ratelimit_test.go (using testing/synctest for deterministic bucket timing).
• internal/store/pg/usage.go, internal/store/pg/quota.go, internal/store/pg/usage_test.go.
• internal/server/handlers_usage.go, internal/server/handlers_quotas.go (platform principal sets quotas).
• migrations/00009_usage_and_quotas.sql.
• docs/learning/13-usage-quotas-rate-limits.md.

Modify

• internal/agent/loop.go — after each model call, build a usage.Record with the turn and purpose and hand it to the recorder; add the cost attributes to the span.
• internal/agent/mwsummarize.go, internal/agent/mwsubagent.go — tag their provider calls with the right Purpose via context.
• internal/server/handlers_runs.go — quota and concurrency check before insert; 402 and 429 paths.
• internal/server/auth_middleware.go — apply the per-key limiter after authentication, with per-tenant rate/burst from tenant_quotas.
• internal/app/app.go — build pricing from config, the limiter with its sweeper, the quota checker.
• internal/config/config.go — HANGO_PRICING (JSON table), HANGO_QUOTA_CACHE_TTL, HANGO_RATELIMIT_DEFAULT_RPM, HANGO_RATELIMIT_DEFAULT_BURST, HANGO_RATELIMIT_ENTRY_TTL.
• internal/store/pg/tenancy_test.go — add usage_records, tenant_quotas.
• docs/system-architecture.md, docs/learning/README.md.

Implementation Steps

1. Write migrations/00009_usage_and_quotas.sql and the scoped usage and quota stores, including the month-aggregate query (sum(input+output) tokens, sum(cost)), the per-run aggregate, and the group-by variants for the reporting endpoint.
2. Write internal/usage/pricing.go: parse HANGO_PRICING as {"anthropic/claude-x": {"input": 300, "output": 1500, "cache_read": 30, "cache_write_5m": 375, "cache_write_1h": 600}} in micro-cents per million tokens ; Cost computes with integer math only (tokens * pricePerMTok / 1_000_000, rounding half-up); unknown models return false.
3. Write internal/usage/recorder.go: Record(ctx, r) prices the record, writes it, sets the span attributes and metrics; on write failure it logs, increments hango.usage.record_failures, retries once, then drops without failing the run.
4. Thread Purpose through context so the summarizer and subagent child loops record their own purpose without the loop needing to know who called it.
5. Write internal/usage/quota.go: Checker caching the month aggregate per tenant for the TTL, plus an active-run count query for concurrency; Allow returns a Decision naming the binding limit.
6. Wire the check into run creation: Decision.Allowed == false returns 402 for quota reasons and 429 for concurrency, both with a JSON body carrying reason, used and limit.
7. Write internal/server/ratelimit.go: N shards keyed by hashed key id, each a mutex-guarded map[uuid.UUID]*entry; Allow uses limiter.Reserve() to derive Retry-After; a safego sweeper evicts entries idle beyond the TTL and updates hango.ratelimit.tracked_keys. Per-tenant rate and burst are read from tenant_quotas with a short cache.
8. Apply the limiter in the auth middleware after principal resolution (so limits are per key, not per IP) and set X-RateLimit-Limit, X-RateLimit-Remaining and Retry-After headers. This is in addition to, not a replacement for, the phase-2 pre-authentication limiter, which protects the unauthenticated path.
9. Write the usage reporting handlers and the platform quota-setting handler. The quota-setting handler calls SetAuditAction(c, "quota.update", "tenant_quota", tenantID) and records the old and new limits in detail, so a tenant that suddenly stops being rate-limited has a named cause.
10. Add langfuse.observation.cost_details and the run-level cost attributes.
11. Tests: pricing integer math including rounding and cache tokens; unknown model records tokens with zero cost and increments the counter; per-run usage aggregate matches the sum of records (integration); quota rejection at run creation with the right status and reason; a started run is not killed when it crosses the quota mid-run (test asserts completion); concurrency limit; rate limiter under testing/synctest asserting burst, refill and Retry-After; sweeper eviction; usage_records and tenant_quotas cross-tenant cases.
12. Update docs and write the learning doc.

Learning Doc Outline

docs/learning/13-usage-quotas-rate-limits.md must contain:

• Concept: why LLM quotas are inherently post-hoc (token counts exist only after the call) and what that means for enforcement design.
• Design choices: one record per model call with a purpose tag, integer micro-cents, config-driven pricing, pre-run check with post-hoc accounting, per-key token bucket with TTL sweep, distinct 402 and 429 semantics.
• Trade-offs: reserve-then-reconcile (goclaw’s shape) vs check-then-record (hango’s); in-process limiter vs Postgres counter vs Redis; per-key vs per-tenant vs per-IP limiting; caching the quota aggregate vs querying every run.
• The money-path rule: never floats, always integers, and why bigint micro-cents in both the DB and Go is the safe combination.
• Single-replica limitation, stated precisely: two replicas double the effective rate limit and can both admit a run at the quota boundary; the replacement is a Postgres counter with INSERT ... ON CONFLICT or a shared store.
• goclaw comparison: internal/usage pricing catalog with UsageReserveRequest → UsageReservationResult → UsageReconcileRequest, i.e. reserve-then-reconcile with a pre-flight token estimate, requiring a token counter on the critical path.
• deepagents comparison: no usage, cost or quota concept at all — this layer is purely a server concern, which is why it is not in the harness research.
• How to debug this layer: hango.usage.cost and hango.usage.tokens split by purpose answers “why is this tenant expensive” (often summarizer or subagent calls, which is exactly why purpose is a dimension); quota.check spans with used/limit explain a 402 to a user in one screenshot; hango.ratelimit.rejections versus hango.quota.rejections distinguishes the two 429-ish failures; in Langfuse, cost_details per observation attributes spend to the exact generation.
• Why cache-write pricing needs TTL buckets, and what a single collapsed field does to a monthly invoice reconciliation.
• Exercises: set a tiny quota and hit it; set a 1-rpm limit and read the headers; price an unknown model and watch the counter.
• Time budget: 9h implementation + 2h learning doc.

Success Criteria

• go build ./... && go vet ./... && go test ./... pass.
• Pricing tests cover integer rounding, unknown models, and the two cache-write TTL buckets priced at their own rates.
• GET /v1/usage?group_by=purpose for a run with a summarization and a subagent call shows three distinct purposes (integration test).
• Over-quota run creation returns 402 with reason, used and limit; a run already in flight completes normally (tests).
• Concurrency limit returns 429 with a reason distinct from the rate-limit 429 (test).
• Rate limiter test under testing/synctest asserts burst, refill rate and Retry-After; the sweeper evicts idle keys.
• usage_records and tenant_quotas are in the cross-tenant suite.
• Setting a quota writes an audit_events row with action quota.update carrying the previous and new limits in detail.
• In Grafana, hango.usage.cost and hango.usage.tokens series exist split by model and purpose; in Langfuse, the generation shows cost_details matching the recorded micro-cents.

Risk Assessment

Rollback

goose down one step drops usage_records and tenant_quotas. Disabling the quota checker and limiter in internal/app (both are single constructor calls) returns the server to phase 12 behaviour; usage recording is additive and can be left on independently of enforcement.