# Telegram Operations, Observability, Testing, and Release Telegram release readiness means the target bot, BFF, Mini App, bindings, scheduled delivery, commands, webhooks, admin channels, and provider calls agree in the deployed environment. Source tests and infrastructure declarations are necessary but not proof that Telegram is receiving or delivering traffic. ```mermaid flowchart TD Source[Source contracts migrations and deployment settings] --> Tests[Unit contract BFF integration and browser tests] Tests --> Deploy[Deploy compatible BFF stores Mini App and scheduled cycle] Deploy --> Register[Read back bot identity webhook commands and menu] Register --> Canary[Isolated test member chat channel and payment cohort] Canary --> Exercise[Commands callbacks rooms grounding reminders voice Mini App and linking] Exercise --> Provider[Inspect Bot API application results message ids and receipts] Provider --> Observe{Dashboards state and user-visible outcomes agree?} Observe -- no --> Contain[Stop affected cycle effect channel or token] Contain --> Reconcile[Reconcile update claims deliveries reminders bindings and payments] Reconcile --> Tests Observe -- yes --> Promote[Promote by bot environment tenant and capability] Promote --> Maintain[Parity probes incident drills token rotation and restore] Maintain --> Source ``` The release loop includes Telegram readback and receipts. A healthy BFF, a registered route, or generated response JSON cannot substitute for provider and member evidence. ## Runtime topology Maintain one declared owner and deployment for each role: - BFF HTTPS webhook route and secret verification; - durable update claim/dedupe, binding/user state, reminders, and delivery records; - command/menu registration and bot identity discovery; - scheduled reminder/outbound delivery cycle with fencing; - Bot API outbound egress and credentials; - Telegram Mini App origin and BFF CORS/session configuration; - web Login Widget/profile link path; - admin editorial channel registry and publishing task; - optional long-poll bot mode, mutually exclusive with webhook consumption; - grounding, rooms, STT/TTS, payment/upgrade, and other provider adapters. ## Configuration and secrets Exact names come from deployment manifests and configuration schemas; the operational categories are: - bot token and expected bot username/identity; - webhook public URL, route, secret token, allowed updates, and registration policy; - Mini App public origin, allowed CORS origin, session issuer/audience/signing keys and max initData/session age; - database/store URLs and migration revision; - command/menu registration enablement and Mini App base URL; - scheduled cycle enablement, cadence, batch, lease/fence, retries, and stop controls; - provider endpoints/credentials/limits for Sophia, rooms, STT, TTS, upgrade, payment, files, and editorial publication; - channel registry ids/chat ids/environment/tenant allowlists; - rate, message/media, retention, quiet-hour/frequency, and alert thresholds. Secrets remain server/task-only and are redacted from Terraform/Nx logs, process output, Bot API error logs, browser bundles, health endpoints, and test artifacts. Rotate bot/webhook/session/provider secrets with overlapping verification where the protocol permits and verify old credentials stop. ## Webhook registration and parity Operations can describe the bot username and current Telegram webhook via Bot API, then compare: - expected versus actual URL; - secret-token posture and allowed update types; - pending update count and last error time/message; - certificate/IP/max-connections/drop-pending configuration as applicable; - environment bot identity and Mini App/menu/command registration. Registration is explicit and idempotent. Deployment parity tests fail when code depends on a setting, route, scheduled task, store, or bridge absent from the target manifest. The helper script at `infra/hetzner/scripts/telegram-webhook.sh` is one operational entry point; it does not replace provider readback. Never use `drop_pending_updates` casually: it is a destructive choice that can discard member commands/payments. Document and approve the incident-specific reason. ## Command and menu discoverability After deployment, call registration and inspect every Bot API result. Verify in private and group clients that: - the private list matches the canonical registered command list; - personal commands are absent from the group list and blocked at runtime; - the menu opens the deployed Mini App or intentional commands fallback; - `/command@botusername`, `/start` payloads, and help work; - old commands/payloads have safe compatibility/fallback. Discoverability is monitored/deployment-gated because a bot can have a healthy webhook and be unusable from the client UI. ## Inbound update reliability Track received, verified, rejected-secret, malformed, unsupported, claimed, duplicate, in-progress/stale-claim, handled, effect-failed, delivered, and terminal-failed updates by bounded update kind/reason. Monitor claim age, attempts, processing latency, provider calls, and webhook response latency. Webhook response timing must respect Telegram retry behavior. Long work is bounded or moved behind durable state; returning early cannot lose the effect, and waiting cannot cause uncontrolled duplicates. Concurrency tests cover two workers claiming the same update and stale claim recovery. ## Outbound delivery reliability For each Bot API call record method, bounded content/media classification, chat/binding identity reference, correlation/idempotency, attempt, latency, HTTP/application result, Telegram message id when accepted, retry-after, and classified failure. Do not log the token or unnecessary body/content. Classify: - accepted; - partial multi-call delivery; - rate-limited/retryable transient; - invalid request/content/button/media; - chat not found/forbidden/blocked; - auth/token/configuration failure; - uncertain timeout after send; - policy/binding/stopped suppression before provider call. Retries are bounded, jittered, fenced, and aware of `retry_after`. Permanent or blocked outcomes transition member delivery state and do not churn. Uncertain send outcomes require reconciliation/product-safe behavior rather than blind duplication. ## Scheduled cycle health Monitor due backlog count/age, claims/leases, scan cadence, selected/delivered/ suppressed/retried/permanent/blocked counts, processing duration, stale claims, and next cycle. Alert when no eligible task is running, two unfenced cycles can run, backlog breaches promise/SLO, stop/quiet-hours is violated, or transient failure is marked terminal. The BFF `promise-kept`, `delivery-stop`, reminder-adapter, and deployment-parity tests pin these concerns. A local scheduler test is not enough—deployment must declare and run the cycle with its stores and credentials. ## Mini App operations Check public origin/TLS, route/static assets, Telegram WebApp CSP/frame/network requirements, CORS allowlist, BFF verification/session endpoints, bot menu URL, theme/readability, protected request success, and real effect wiring. Observe initData verification failures by safe reason, session issue/expiry/ scope/revocation, CORS rejection, BFF API status/latency, client errors, and per-surface journey outcomes without logging raw initData/session or sensitive content. The Mini App Playwright suite runs across supported viewport/theme behavior; the Docs Center walkthrough explicitly records local-only versus durable actions. ## Editorial and payment operations Editorial channel health covers bot admin rights, configured channel readback, approval/schedule backlog, send/edit/delete results, corrections/takedowns, rate limits, and audit. A provider message id is required delivery evidence. If payments are deployed, monitor invoice creation, pre-checkout deadline and rejection reason, successful-payment dedupe, billing/entitlement application, pending reconciliation, receipt, refund/reversal, and fraud/security signals. Never alert with full payment payloads or personal financial data. ## Dashboards and alerts At minimum expose: - webhook traffic, secret/malformed rejection, latency, errors, update lag and duplicate/claim state; - command/update kind and outcome, group-scope denials, rate/abuse, crisis and grounding abstention at privacy-safe aggregation; - Bot API attempts/latency/results/retry-after/blocked and partial delivery; - reminder due/backlog/age/delivery/snooze/done/stop/quiet-hour behavior; - binding/link/unlink and Mini App verification/session failures; - provider health for Sophia/rooms/STT/TTS/upgrade/payment/files; - command/menu/webhook registration parity and scheduled-task presence; - editorial publish/correction/takedown and payment reconciliation where used. Alerts link to a runbook, owner, affected capability, safe diagnostic query, and expected user posture. Keep dimensions bounded; raw chat/user ids do not belong in metric labels. ## Automated verification map | Lane | Coverage | | ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Shared unit/contract | Bot/update handler, command/group scope, deep links, keyboards/limits, rendering, delivery planning/provider responses, rooms, reminders, STT/TTS, security, inline, publishing, upgrade/payments, registry/dispatcher/boundary. | | BFF service/integration | Webhook secret/route, grounding and effect adapters, durable binding/user/update stores, dedupe/claim concurrency, Mini App surface/CORS/security, scheduled delivery/stop/promise, deployment parity. | | Bot app | Long-poll/thin composition, discoverability, delivery, crypto upgrade handler—without claiming it is deployed. | | Mini App | Surface data/router/view, WebApp/theme/client, protected API behavior, Playwright surface journeys and responsive/theme/accessibility. | | Web | Login widget/profile panel components, binding BFF, profile link/unlink and webhook-delivery E2E. | | Admin | Telegram channel registry loader/actions/page and authorized Playwright flow. | | Cross-channel | Messaging registry/policy/tier/consent/quiet/frequency/crisis, assistant bridge/client, payment bridge, editorial/social adapters. | | Walkthrough/release | Telegram Mini App route matrix, assistant-delivery journey/result, deployed smoke/readback, commands/menu/webhook/task/config parity. | Use targeted Nx/package commands while iterating and run every affected lane. Live/provider verification uses isolated test bots/chats/channels and credentials with explicit cleanup; do not send test/crisis/payment content to production members. ## Release sequence 1. Validate contracts, types, migrations, configuration schema, and generated artifacts. 2. Run targeted unit, integration, browser, accessibility, security, and deployment-parity tests. 3. Deploy compatible BFF/store changes with new effects disabled where needed. 4. Verify health, migrations, stores, provider egress, bot identity, webhook readback, and scheduled cycle. 5. Register commands/menu and verify private/group discoverability. 6. Enable a test tenant/bot cohort; exercise text, callback, room, grounder, reminder, stop/resume, voice, Mini App/auth/link, and configured editorial/ payment paths. 7. Inspect Telegram application responses, durable receipts/state, dashboards, alerts, and audit—not only UI copy. 8. Promote by declared cohort with abort/rollback/forward-fix criteria. 9. Record deployment revision, settings/secrets versions, Bot API readback, test evidence, limitations, and owner. ## Incident playbooks ### Webhook silent or wrong bot Verify DNS/TLS/route, Telegram webhook readback, bot identity/token environment, secret mismatch, last Telegram error, pending updates, BFF health/logs, update store, and egress. Do not register a different environment bot onto the same URL without resolving identity. ### Replies composed but not received Trace update → handler response → delivery call plan → Bot API request/body → application result/message id. Check long-text split, invalid markup/button, blocked state, rate limit, token, chat id, media download/upload, and partial multi-call outcome. ### Duplicate effects or reminders Disable/fence affected consumer/cycle, preserve update/reminder/delivery state, inspect unique keys/claims/leases and external message ids, reconcile effects, repair idempotency before replay, and notify affected members when needed. ### Mini App cannot authenticate or call BFF Verify menu/origin, Telegram runtime/initData age and bot token environment, server clock, signature algorithm, session issuer/audience/scope/key, binding, CORS exact origin, BFF route and primary-credential requirement. Never collect raw initData in a ticket/chat. ### Bot blocked, token leaked, or provider compromised Contain delivery/provider path, rotate token/webhook/session/provider secrets, re-register webhook/commands/menu, invalidate sessions, inspect safe audit and delivery/binding impact, reconcile scheduled/payment/editorial state, and document residual data/provider copies. ## Rollback and recovery Rollback must preserve compatibility with new update, binding, reminder, session, delivery, and payment records. Disable new commands/surfaces/effects before reverting an incompatible reader. Provider registrations (webhook, commands, menu, channel posts) and already-sent messages are external effects; database rollback does not reverse them. Restore verification includes binding uniqueness/status, update dedupe, reminder due/delivery/action state, stop/quiet/timezone preferences, Mini App session revocation posture, editorial/payment reconciliation, and a controlled Bot API delivery. Re-register provider configuration only after the restored runtime is safe to consume updates. The chronological [Telegram buildout/audit plan](../../TELEGRAM_SURFACE_PLAN_2026-08-05.md) is required reading because it records failures that package tests alone did not expose: unreachable effects, missing deployment wiring, command privacy, Telegram protocol limits, send-loop promises, local time, and provider replies.