Long-term memory and skills

Overview

Two features, one mechanism: both are conventions over the phase-8 filesystem. There is no routing layer — /memories/ and /skills/ are reserved prefixes in the one Postgres-backed backend, enforced by internal/fs/path.go, because hango has exactly one backend and a router with a single destination buys nothing. Memory is always-on context: AGENTS.md and other configured sources are loaded once per run and contributed as prompt blocks after the cache breakpoint. Skills use progressive disclosure: only name, description and path enter the prompt; the body is read on demand with read_file.

Requirements

Functional

• `/memories/` and `/skills/` are reserved prefixes in the single `agent_files`-backed backend (namespace `{tenant_id, agent_id}`), validated in `internal/fs/path.go`. No `CompositeBackend`, no route table, no path rewriting.
• `MemoryMiddleware` implements exactly two hooks: `RunHook` (`BeforeRun` loads each configured source, default `/memories/AGENTS.md`, strips HTML comments, stores the content in its typed private state) and `PromptContributor` (`SystemBlocks` returns the loaded content as blocks). It does **not** implement `WrapModelCall`. Because system blocks are assembled in chain order at model-call time and Memory sits after `PromptCache` in the stack, its blocks land after the breakpoint index by construction.
• Missing memory files are not an error: they are skipped and logged once.
• The agent edits memory with the ordinary write_file/edit_file tools; no dedicated memory tool exists.
• SkillsMiddleware: BeforeRun scans configured sources for SKILL.md files, parses YAML frontmatter (name ≤64 chars, description ≤1024, optional allowed_tools), and injects only an index of name, description and path into the system prompt. Later sources override same-named skills.
• A skill's `allowed_tools` can only ever **narrow**: the effective set is `allowed_tools ∩ agent allowlist`. A skill file — which is agent-writable content and therefore attacker-influenceable — must never be able to grant a tool the agent is not already allowed to call. Names outside the intersection are dropped with a warning and an `agent.skill_tools_narrowed` span event.
• Writes to `/memories/` and `/skills/` follow the phase-8 rule: API-only for `owner`/`admin` until phase 15, then a default HITL interrupt rule.
• Invalid SKILL.md frontmatter is skipped with a warning naming the path and the reason; one bad skill never fails a run.
• Skills metadata and memory content never propagate from a subagent back to the parent.
• Memory and skills sources are configured per agent in agents.config (memory: [...], skills: [...]) with tenant-level defaults from config.

Non-functional

• Memory must sit after PromptCache in the stack so its contributed blocks land after the breakpoint index, or every turn pays full input cost. The order test asserts the breakpoint index, not a slice order.
• BeforeRun loading is bounded: a total byte cap across sources (HANGO_MEMORY_MAX_BYTES, default 64KiB) and a skill count cap (HANGO_SKILLS_MAX, default 100).
• Cross-tenant isolation test for memory and skill paths (tenant B cannot read tenant A’s /memories/AGENTS.md).

Architecture

// Red team: F14 — no CompositeBackend. Reserved prefixes are enforced in internal/fs/path.go
// (fs.PrefixMemories, fs.PrefixSkills, fs.IsReserved), over the single pg backend.

// internal/agent/mwmemory.go
type MemoryMiddleware struct {
	Backend fs.Backend
	Sources []string // default {"/memories/AGENTS.md"}
	MaxBytes int
} // RunHook (load once) + PromptContributor (blocks land after the cache breakpoint index)
// Red team: F6 — deliberately NOT a ModelWrapper.

// internal/agent/mwskills.go
type Skill struct {
	Name         string   `yaml:"name"`
	Description  string   `yaml:"description"`
	AllowedTools []string `yaml:"allowed_tools"`
	Path         string   `yaml:"-"`
}

type SkillsMiddleware struct {
	Backend fs.Backend
	Sources []string // default {"/skills/"}
	Max     int
} // RunHook (index scan) + PromptContributor (index only, never bodies)

No new tables: memory and skills are rows in agent_files under reserved prefixes.

Instrumentation

• Spans: agent.memory_load and agent.skills_index as children of agent.before_run, with hango.memory.sources, hango.memory.bytes, hango.skills.count, hango.skills.invalid.
• Span event agent.memory_missing per configured source that does not exist, and agent.skill_invalid per rejected SKILL.md with the path and reason.
• Attributes on the chat span: hango.memory.bytes and hango.skills.count so prompt growth is attributable; langfuse.observation.metadata.memory_bytes and ...skills_count for Langfuse filtering.
• Metrics: hango.memory.load.duration histogram, hango.memory.bytes histogram, hango.skills.indexed gauge, hango.skills.invalid counter, hango.cache.read_ratio re-checked here because memory is the most likely cache-invalidation culprit.
• Log fields: one info line per run summarizing loaded memory sources, total bytes and indexed skill count.

Related Code Files

Create

• internal/agent/mwmemory.go, internal/agent/mwmemory_test.go.
• internal/agent/mwskills.go, internal/agent/mwskills_test.go.
• internal/agent/frontmatter.go — minimal YAML frontmatter parser for the small fixed schema (no YAML dependency), plus its test.
• docs/learning/11-memory-and-skills.md.

Modify

• internal/agent/mwstack.go — fill the Skills slot (first, so its tools and index are available to everything) and the Memory slot (after PromptCache); update the pinned order test.
• internal/agent/subagent.go, internal/agent/childbuilder.go — subagents load their own memory and skills sources and never inherit the parent’s indexes.
• internal/fs/path.go — reserved-prefix constants and IsReserved already exist from phase 8; this phase only adds the tests that memory and skills paths validate correctly.
• internal/config/config.go — HANGO_MEMORY_DEFAULT_SOURCES, HANGO_MEMORY_MAX_BYTES, HANGO_SKILLS_DEFAULT_SOURCES, HANGO_SKILLS_MAX.
• internal/server/handlers_agents.go — validate memory and skills config entries are absolute paths under the reserved prefixes.
• docs/system-architecture.md, docs/learning/README.md.

Implementation Steps

1. Confirm the reserved prefixes behave: add table cases to `internal/fs/path_test.go` for `/memories/AGENTS.md`, `/skills/x/SKILL.md`, a near-miss (`/memorie/x`) and a traversal attempt into a reserved prefix. No new filesystem code is written in this phase.
2. Write internal/agent/frontmatter.go: parse a leading --- block of key: value lines plus simple [a, b] lists into the fixed Skill schema; reject unknown keys with a warning rather than an error; enforce the length caps.
3. Write internal/agent/mwmemory.go: BeforeRun reads each source (skipping missing files), strips <!-- ... --> comments, enforces the total byte cap by truncating with a marker, and stores the result in typed private state. SystemBlocks returns the content as blocks with a short header naming each source path so the agent knows where to edit. No WrapModelCall.
4. Write internal/agent/mwskills.go: BeforeRun globs {source}/**/SKILL.md, parses frontmatter, applies last-source-wins override by name, caps the count, intersects each skill’s allowed_tools with the agent allowlist, and stores the index. SystemBlocks renders only name — description (path) lines plus one sentence telling the model to read_file the path for the full instructions.
5. Place the middlewares in DefaultStack: Skills first, Memory after PromptCache. Add a comment at the Memory slot stating the cache-invalidation constraint, and extend the order test to assert Memory’s blocks land at a higher index than the cache breakpoint.
6. Update the child builder so subagents load their own sources and start with empty memory/skills private state.
7. Add the instrumentation from the section above.
8. Tests: memory content appears in the assembled system blocks at an index after the cache breakpoint (index assertion); missing source is skipped with an event; byte cap truncates with a marker; an owner can write /memories/AGENTS.md through the files API and the next run loads the new content, while an agent tool write is refused (integration test) ; skills index contains only name/description/path and never a body; a skill declaring allowed_tools outside the agent allowlist has them dropped and cannot call them ; an invalid SKILL.md is skipped while valid ones still load; last-source-wins override; cross-tenant test that tenant B cannot read tenant A’s /memories/AGENTS.md.
9. Update docs and write the learning doc.

Learning Doc Outline

docs/learning/11-memory-and-skills.md must contain:

• Concept: memory and skills as conventions over a filesystem rather than new subsystems; progressive disclosure as a context-budget technique.
• Design choices: /memories/ and /skills/ as reserved prefixes validated in path.go, memory as always-on context, skills as index-only injection, subagents loading their own sources.
• Learning note on the rejected option: deepagents routes these prefixes through a `CompositeBackend` because it genuinely has several backend kinds (state, filesystem, store). hango has one Postgres backend, so a router would be an indirection with a single destination — the prefix rule alone carries the same meaning. If a second backend kind ever lands (a real-disk or object-store backend), the router is the right move then, and this is the note that says so.
• Ordering constraint: memory’s blocks must land after the cache breakpoint index; what the cost looks like when this is violated, and why the assertion is on the index.
• Skill files are agent-writable content: why `allowed_tools` may only narrow, never grant.
• Trade-offs: always-on memory vs a retrieval tool vs embeddings; file-based skills vs registered plugins; per-agent vs per-tenant memory namespace.
• goclaw comparison: internal/memory with pgvector, channelmemory, a consolidation worker doing episodic/semantic “dreaming”, and a knowledge graph — all product features hango deliberately omits; goclaw’s skills are slash-commands, not progressive-disclosure documents.
• deepagents comparison: MemoryMiddleware loading sources at before_agent and appending them with an Anthropic cache_control breakpoint; memory edited through normal write_file on the /memories/ route of its CompositeBackend; SkillsMiddleware with SKILL.md frontmatter (name ≤64, description ≤1024, allowed_tools), index-only prompt injection, last-source-wins, and skills_metadata never inherited by subagents.
• Why hango copies this almost exactly, and where it tightens it (reserved-prefix validation, byte and count caps, invalid-skill tolerance).
• How to debug this layer: agent.memory_load and agent.skills_index spans show what was loaded and how long it took; agent.memory_missing and agent.skill_invalid events explain “why doesn’t the agent know about my skill”; hango.memory.bytes on the chat span explains prompt growth; a collapsed hango.cache.read_ratio after this phase points directly at memory ordering.
• Exercises: write memory through the files API and observe it in the next run’s prompt; add a skill and watch the index appear; break the frontmatter and read the warning.
• Time budget: 7h implementation + 2h learning doc.

Success Criteria

• go build ./... && go vet ./... && go test ./... pass.
• path_test.go covers the reserved prefixes, a near-miss prefix and a traversal attempt into a reserved prefix.
• Memory blocks appear in the assembled system blocks at an index strictly greater than the cache-breakpoint index, and reflect a file written through the files API in a previous run (integration test).
• A skill declaring allowed_tools: [dangerous_tool] outside the agent allowlist cannot cause that tool to be callable; the narrowing event is recorded.
• A missing memory source produces a span event and no error.
• The skills index contains name, description and path only; no skill body appears in any prompt (test).
• An invalid SKILL.md is skipped with a warning while the remaining skills still load (test).
• Tenant B cannot read tenant A’s /memories/AGENTS.md (test).
• In Grafana, agent.memory_load and agent.skills_index spans appear under agent.before_run, and hango.memory.bytes/hango.skills.indexed series exist; hango.cache.read_ratio has not collapsed compared with phase 10.

Risk Assessment

Rollback

No migration to revert. Removing the Memory and Skills slots from DefaultStack restores phase 10 behaviour; files under /memories/ and /skills/ remain in agent_files and stay readable.