Telegram media, pairing and approvals

Overview

Phase 18 gave the family a text-only bot in private chats. This phase brings hango’s Telegram channel close to what goclaw and OpenClaw offer. Family members send photos, documents and voice notes. The agent sends files back. New people link their Telegram account with a pairing code instead of a config edit and restart. A tool call that needs approval (phase 15) shows up as buttons in the chat.

The lesson is that a channel adds most of its work at the edges. Media turns into model input blocks and filesystem files. Pairing turns an untrusted platform identity into a hango user. Buttons need callback data that cannot be forged into another person’s approval. The harness core changes in one place: the provider message model gains an image block.

The phase has two review stops. 20a covers media: photos, documents, albums, voice notes, files out and inbound debounce. 20b covers access and control: pairing codes, approval buttons and status reactions. The phase file lists which steps and criteria belong to each stop.

Group chats are deferred. For now each person talks to the bot in a private chat, and group messages stay ignored as in phase 18. The goclaw-style group design is kept at the end of this file so a later phase can pick it up.

It runs after phase 16, so its tables are created with row-level security from the start, and before phase 17, so evals can include an image task.

Requirements

Functional

20a: media

• The bot serves one tenant, HANGO_TELEGRAM_TENANT_ID. Every lookup by Telegram id runs inside that tenant’s scope, so the bot never needs the phase-16 system pool, whose five callers stay five. Phase-18 static mappings must belong to this tenant, checked at startup.
• Photos: the largest size under HANGO_TELEGRAM_MEDIA_MAX_MB (default 20) is downloaded with getFile and written to the virtual filesystem at /uploads/{session_id}/{telegram file unique id}.{ext}. The run input gets an image block that references that path. A WrapModelCall middleware loads referenced images and inlines them as base64 for the provider call, so persisted messages hold paths, not bytes.
• Provider model: provider.Block gains Image *ImageBlock (MediaType, Path, and Data filled only for the provider call). The Anthropic adapter maps it to an image content block, and the OpenAI-compatible adapter maps it to an image_url data URI. A model whose registry entry has no vision support gets a text placeholder ("[image: /uploads/…]") instead, and the reply says the agent cannot see images.
• Documents: saved to /uploads/{session_id}/ the same way. The run input says the file name, size and path, so the agent reads text files with read_file. PDFs and other binary files are saved and named but not parsed in this phase.
• Albums: updates sharing a media_group_id are collected for 500 ms and become one run with every photo and the album caption.
• Inbound debounce: text messages from one chat arriving within 300 ms of each other are joined into one run, as OpenClaw does, so a long message that Telegram split in two becomes one question.
• Voice notes and audio: downloaded, then transcribed through an OpenAI-compatible /audio/transcriptions endpoint (HANGO_STT_BASE_URL, HANGO_STT_MODEL, credentials from the phase-12 registry entry it names). The transcript becomes the run input, prefixed with “(voice)”, and is sent back as a short quoted line so the sender sees what hango heard. Telegram voice notes are OGG/Opus, which the endpoint accepts as is, so no conversion step is needed.
• Vietnamese transcription: HANGO_STT_LANGUAGE (ISO 639-1, empty means auto-detect) is sent as the language form field. The family deployment sets vi, because a language hint avoids short Vietnamese clips being detected as another language. The Whisper family supports Vietnamese: OpenAI whisper-1, gpt-4o-transcribe and gpt-4o-mini-transcribe, and Groq whisper-large-v3 and whisper-large-v3-turbo, so either provider works. The transcript is normalized to Unicode NFC before use, so diacritics compare and render the same way everywhere (the quoted echo and the stored message). Transcription writes a usage_records row with purpose transcription, priced per second from the phase-13 pricing table.
• Files out: a builtin tool attach_file(path) marks a virtual-filesystem file as an attachment of the current run and emits a run event attachment. The Telegram stream sends images with sendPhoto and everything else with sendDocument after the text reply. The web chat later reads the same event. Attachments over Telegram’s 50 MB limit are refused with a tool error the model can explain.
• Stickers, video notes, locations and polls still get “I can’t read that yet.” and create no run.

20b: access and control

• Pairing codes, as goclaw and OpenClaw do it: an unknown sender gets an 8-character code (valid 1 hour, single use) instead of the phase-18 fixed reply. An owner or admin approves it with POST /v1/telegram/pairings/{code} (body: user_id, agent) or by sending /pair <code> <user email> <agent slug> from their own linked Telegram account. Approval writes a telegram_links row. Links take effect without a restart. DELETE /v1/telegram/links/{telegram_user_id} and /unlink <telegram id> remove one. Phase-18 static mappings still work and win over links, so the owner cannot be locked out by a bad link.
• Audit: two new actions, telegram.link and telegram.unlink, added to the closed action set from phase 2. Pairing requests from unknown senders are counted and logged, not audited, because they are not tenant actions.
• Approval buttons: when a run created from Telegram enters awaiting_approval, the bot sends the pending tool call (tool name and arguments, truncated to 1000 characters) to the run creator’s private chat with inline buttons Approve and Reject. Callback data is ap:<approval id>:<decision>:<hmac>, where the HMAC (truncated SHA-256 over the approval id, decision and Telegram user id, keyed by HANGO_TELEGRAM_CALLBACK_KEY) prevents a forged or forwarded button from deciding someone else’s approval. The callback goes through the phase-15 decision path with the pressing user’s scope, writes the same approval.decide audit row with detail.channel = "telegram", answers the callback query, and edits the message to show the decision and who made it. A second press gets “Already decided.” Edit and respond decisions stay on the HTTP API.
• Status reactions: the user’s message gets a reaction when the run starts (👀), succeeds (👍) or fails (😢), as goclaw’s StatusReactionController does. A reaction failure is logged and ignored.
• Error replies follow one rule: one short line per failed run with the run id, never the error text, which may contain tool output.

Non-functional

• No new package. The work lives in internal/telegram, with the image block in internal/provider and its adapters, the transcription client in internal/provider/openai, and attach_file in internal/tool/builtin.
• Migration 00012_telegram.sql creates telegram_links (tenant_id, telegram_user_id, user_id, agent_id, created_by, created_at) and telegram_pairing_codes (tenant_id, code_hash, telegram_user_id, expires_at, used_at), both with FORCE ROW LEVEL SECURITY and the phase-16 tenant policy in the same migration. Pairing codes are stored hashed.
• Downloads use the Bot API file URL, which contains the token, so the same error stripping as phase 18 applies. Media bytes are never logged or put in span attributes.
• Uploaded files count toward the phase-8 filesystem limits, and an upload that would exceed them is refused with a reply.

Architecture

// internal/provider/types.go (added)
type ImageBlock struct {
	MediaType string `json:"media_type"`     // image/jpeg, image/png, image/webp, image/gif
	Path      string `json:"path"`           // virtual-filesystem path, persisted
	Data      []byte `json:"data,omitempty"` // filled per call by the inline-images middleware, never persisted
}

// internal/telegram/links.go
type LinkStore interface {
	Resolve(ctx context.Context, telegramUserID int64) (Link, error) // runs in HANGO_TELEGRAM_TENANT_ID scope
	Approve(ctx context.Context, code string, userID, agentID uuid.UUID) (Link, error)
	Delete(ctx context.Context, telegramUserID int64) error
}

update ──▶ debounce / album window ──▶ resolve sender (static map, then telegram_links)
   unknown private sender ──▶ pairing code reply
   group or channel ──▶ ignored, as in phase 18
   media ──▶ getFile ──▶ /uploads/{session}/… ──▶ image block or file note or transcript
   run.Create (sender scope, session) ──▶ stream (phase 18) ──▶ attachments ──▶ sendPhoto/sendDocument
   awaiting_approval ──▶ inline buttons ──▶ callback_query ──▶ HMAC check ──▶ phase-15 decision

Instrumentation

• telegram.update gains hango.telegram.media_kind (photo|album|document|voice|none) The outcome set grows by pairing_code and callback.
• Spans: telegram.download (file size, media kind), stt.transcribe (model, audio seconds, as a GenAI-style client span so Langfuse shows it as a generation), telegram.send_attachment.
• Metrics: hango.telegram.media counter (attribute kind), hango.telegram.download_bytes histogram, hango.telegram.pairings counter (attribute outcome = issued|approved|expired|invalid), hango.telegram.callbacks counter (attribute outcome = decided|duplicate|forged|expired), hango.stt.duration histogram.
• Phase 14’s hango-channels dashboard gains panels for these, and one alert: hango.telegram.callbacks{outcome="forged"} non-zero.

Related Code Files

Create

• migrations/00012_telegram.sql.
• internal/telegram/media.go, media_test.go — download, size cap, path naming, album window, debounce.
• internal/telegram/links.go, links_test.go, and internal/store/pg/telegram_links.go — link store, pairing codes, scope checks.
• internal/telegram/approvals.go, approvals_test.go — buttons, callback HMAC, decision through phase 15.
• internal/telegram/reactions.go.
• internal/agent/inline_images.go, inline_images_test.go — the WrapModelCall middleware that fills ImageBlock.Data.
• internal/provider/openai/transcribe.go, transcribe_test.go.
• internal/tool/builtin/attach_file.go, attach_file_test.go.
• docs/learning/20-telegram-media-pairing-approvals.md.

Modify

• internal/provider/types.go, internal/provider/anthropic/convert.go, internal/provider/openai/convert.go — the image block and its mapping, plus a vision flag on registry entries.
• internal/server/handlers_telegram.go (new routes in the existing package) — pairing approval and link deletion, owner or admin only, with the new audit actions.
• internal/store/audit.go — telegram.link and telegram.unlink in the closed set.
• internal/run/events.go — the attachment event kind.
• internal/telegram/bot.go, internal/telegram/stream.go — routing for media, callbacks and attachments.
• internal/usage/pricing.go — per-second transcription prices.
• deploy/grafana/dashboards/hango-channels.json, deploy/grafana/provisioning/alerting.yaml — the new panels and the forged-callback alert.
• internal/config/config.go, internal/app/app.go, internal/telemetry/metrics_names.go, .env.example, README.md, docs/learning/README.md, docs/system-architecture.md.

Implementation Steps

Review stop 20a covers steps 1 to 6, and review stop 20b covers steps 7 to 11.

1. Add ImageBlock to the provider model and both adapters, with round-trip conversion tests and the non-vision placeholder.

2. Write the inline_images middleware: load each referenced path through the phase-8 backend in the run’s scope, fill Data, and leave persisted messages untouched (test that the stored message has no bytes).

3. Write media.go: getFile download with the size cap, file naming under /uploads/{session_id}/, the 500 ms album window and the 300 ms text debounce, tested with a fake Bot API and a fake clock.

4. Write transcribe.go against an httptest server, the stt.transcribe span, and the usage row; wire voice notes to it.

5. Write attach_file, the attachment run event, and sendPhoto/sendDocument in the stream.

6. Send a photo, an album, a text file and a voice note from a real phone and record the traces in the learning doc. Stop for review 20a.

7. Write 00012_telegram.sql with its RLS policies, and the link store, with an isolation test (a link in tenant A is invisible from tenant B) and an intra-tenant visibility test.

8. Write pairing: code issue, hashed storage, expiry, single use, the HTTP approval route and /pair, with the new audit actions. Test expiry, reuse and a member trying to approve (403).

9. Write approval buttons: message on awaiting_approval, HMAC callback data, decision through phase 15, message edit, duplicate press. Test a callback with a wrong HMAC, a callback pressed by a different Telegram user, and a callback for an approval that already expired.

10. Add status reactions, the metrics, dashboard panels and the alert.

11. Pair one family member’s account with a code, approve a tool call from the phone, and record it in the learning doc. Stop for review 20b.

12. Update docs and finish the learning doc.

Learning Doc Outline

docs/learning/20-telegram-media-pairing-approvals.md must contain:

• Concept: a channel’s work is at its edges. Media becomes model input and files, identity becomes a scoped hango user, and a button press becomes an audited decision.
• Design choices: images persisted as filesystem paths and inlined per call; one tenant per bot so no lookup needs the system pool; pairing codes approved by an owner; private chats only, with group chats deferred; HMAC-signed callback data.
• Trade-offs: inline image bytes vs paths (simple vs keeping the messages table small and the history replayable); transcribing in hango vs sending audio to the model (any model works vs one call).
• goclaw comparison: internal/channels/telegram (media.go and its STT manager, album_aggregator.go, commands_pairing.go, reactions.go, send_media_group.go), channel instances in the database. goclaw’s group chat code is compared in the deferred design at the end of the phase file.
• OpenClaw comparison: dmPolicy pairing, inbound debounce windows, and native approval buttons (approval-native.ts, approval-callback-data.ts).
• How to debug this layer: telegram.update by media_kind and outcome, telegram.download and stt.transcribe spans for slow replies, hango.telegram.callbacks by outcome when a button “does nothing”, and the telegram.link audit rows when someone says the bot forgot them.
• Vietnamese voice notes: why the language hint matters for short clips, which models support Vietnamese, and why the transcript is NFC-normalized.
• Follow-ups: group chats (design kept in the phase file), webhook mode, PDF parsing, the web chat, multi-bot.
• Exercises: forward someone else’s approval button to your own chat and press it; send a 30 MB photo; send three messages within 200 ms; send a Vietnamese voice note with HANGO_STT_LANGUAGE empty and then set to vi.
• Time budget: 23h implementation + 3h learning doc.

Success Criteria

Review stop 20a:

• go build ./... && go vet ./... && go test ./... pass, and the package list is unchanged from phase 19.
• A photo becomes an image block whose stored message holds a path and no bytes, and both adapters send it (tests).
• An album becomes one run with every photo, and three texts within 300 ms become one run (tests with a fake clock).
• A voice note becomes a transcript run with a transcription usage row and an stt.transcribe span, and HANGO_STT_LANGUAGE is sent as the language field (test).
• A recorded Vietnamese voice-note fixture (internal/provider/openai/testdata/voice-vi.ogg, a short sentence recorded by the owner) is transcribed through the real endpoint with diacritics preserved, checked against its expected words after NFC normalization (integration test, gated by HANGO_STT_IT=1).
• attach_file delivers a file from the virtual filesystem to the chat after the text reply (test).
• Media over the size cap is refused with a reply, and no media byte or token appears in logs or spans (tests).

Review stop 20b:

• telegram_links and telegram_pairing_codes have RLS policies, and the isolation and visibility tests pass.
• A pairing code works once, expires after an hour, and only an owner or admin can approve it; approvals and unlinks write telegram.link and telegram.unlink audit rows (tests).
• An approval button decides the approval through the phase-15 path with an approval.decide audit row; a forged, foreign or repeated press decides nothing (tests).
• The phase-16 system pool still has exactly five callers (the existing test passes).
• A real pairing, a Vietnamese voice note and an approval from a phone are recorded in the learning doc.

Risk Assessment

Rollback

Each feature is off unless configured: no HANGO_STT_BASE_URL means voice notes get the “can’t read that” reply, no HANGO_TELEGRAM_CALLBACK_KEY means approvals stay on the HTTP API. Rolling back the migration drops the link tables, and phase-18 static mappings keep working. The image block is additive, so reverting it only removes image input.

Deferred: group chats

The owner deferred group chats on 2026-10-14: for now each person talks to the bot in a private chat. When groups come back, they follow goclaw’s design below, and the open question is whether group runs belong to one group principal or to each sender. This section is not part of phase 20’s effort or success criteria.

The design follows goclaw (internal/channels/telegram/handlers.go, topic_config.go, wake_words.go, commands_writers.go, internal/channels/history.go), adapted to hango’s tenancy:

• Group pairing: the first time the bot is mentioned in an unknown group, it replies with an 8-character group pairing code, as goclaw’s default group_policy = "pairing" does. An owner or admin approves it with the same POST /v1/telegram/pairings/{code} route (body: agent) or /pair <code> <agent slug>. Approval writes a telegram_groups row and a telegram.link audit row with detail.kind = "group". Messages in an unapproved group are dropped without being recorded, as goclaw does. /unlink sent in the group by an owner or admin removes it.
• Group identity: approval creates one group principal, a users row with kind = 'telegram_group', role member, the chat title as display name and an email under the reserved groups.hango.invalid domain. Every group run runs as this principal, which matches goclaw’s shared scope group:telegram:<chat id>. Usage, sessions and audit rows of group runs therefore belong to the group, not to whoever spoke. The admin API refuses to issue an API key to a group principal, so nobody can log in as it. The Telegram user id and name of the sender go into the run input metadata (telegram_sender_id, telegram_sender_name) and into the run span, so a trace still shows who asked.
• Sessions: one shared session per group, and one per forum topic, keyed telegram:group:<chat id>:<unix seconds> and telegram:group:<chat id>:topic:<topic id>:<unix seconds>, as goclaw’s BuildSessionKey and BuildGroupTopicSessionKey do. /new in a group starts a new one and is allowed only for the group’s writers (below).
• Group policy, checked before the mention gate: open (any member of an approved group may wake the agent), allowlist (only Telegram ids in allow_from), or disabled. The default is open, as in goclaw. Being in an approved group is enough to talk to the agent there, because the owner chose to approve that group.
• Mention gate: require_mention (default true) and mention_mode. In strict mode the agent answers only an explicit @bot mention, a reply to one of its messages, a command, or a wake word. In yield mode it answers every message unless another bot or user is explicitly mentioned, and it never answers another bot’s message that does not mention it, which prevents two bots replying to each other forever. The bot’s own mention is stripped from the input.
• Wake words: an agent’s config may list telegram.wake_words (for example its name, “Bơ”). A wake word in the text or a caption counts as a mention. Matching is case-insensitive on NFC-normalized text at word boundaries, so Vietnamese words with diacritics match as typed.
• Group history: messages that do not wake the agent are recorded in telegram_group_pending (sender name, text or caption, a “[photo]” or “[file: name]” note for media, Telegram message id), keyed by chat and topic and capped at history_limit. The cap is configurable: HANGO_TELEGRAM_GROUP_HISTORY_LIMIT sets the default (50 when unset, goclaw uses 200, and 0 turns history off), and each group or topic can override it through its config. When the agent is woken, the pending entries are put in front of the input as a “Messages since your last reply” block, and then deleted. Media in history is noted, not downloaded, which differs from goclaw’s lazy download and keeps unaddressed photos out of the model.
• Sender label: each input in a group starts with [From: <first name> (@username)], as goclaw annotates it, so the agent knows who is talking in a shared session.
• Writers: each group has a writers list of Telegram ids, which may run /new in the group. Owners and admins manage it with /addwriter and /removewriter sent as a reply to the person’s message, and /writers lists it. The approver is a writer from the start.
• Per-group and per-topic config: the group row holds policy, require_mention, mention_mode, allow_from, history_limit, enabled, system_prompt and tools, and a topics JSON map with the same fields per forum topic. Config resolves in goclaw’s order: environment defaults (HANGO_TELEGRAM_GROUP_*, the equivalent of goclaw’s "*" group), then the group, then the topic. A topic’s system_prompt is appended to the group’s, and tools can only narrow the agent’s tool list, never widen it. PATCH /v1/telegram/groups/{chat id} (owner or admin) changes it and writes a telegram.group_update audit row.
• One tenant per bot still holds: every group lookup runs in the HANGO_TELEGRAM_TENANT_ID scope, so the system pool keeps its five callers.

Estimated effort when picked up: about 9h, plus migration additions (telegram_groups, telegram_group_pending, users.kind, a kind column on pairing codes) and a telegram.group_update audit action.