diff --git a/apps/server/docs/ai-context/README.md b/apps/server/docs/ai-context/README.md index 2a7e290c7..7bf727bd7 100644 --- a/apps/server/docs/ai-context/README.md +++ b/apps/server/docs/ai-context/README.md @@ -33,6 +33,10 @@ - 全量 metric 目录（按域分组：HTTP / Auth / Engagement / Revenue / GenAI / Email / Rate limit / Runtime），含名字、类型、Labels、落点 - `metrics-ownership.md` - 指标分层规则：什么走 Grafana / 什么走 PostHog / 什么是 Postgres truth；含 7 题判定 Checklist、PostHog 事件命名约定、当前指标归属总表、PostHog 接入路线图 +- `product-analytics-instrumentation.md` + - 面向社区 / 产品问题的埋点补充方案：上手激活、Provider 配置、TTS 音色、语音输入、反馈、看板和异常播报 +- `product-analytics-dashboard-setup.md` + - 产品分析看板落地说明：PostHog insights、Grafana 产品事件面板、告警表达式；不含 Discord / QQ 同步和日报 / 周报脚本 - `auth-and-oidc.md` - 认证与 OIDC Provider 架构、登录流程、trusted clients、踩坑记录 - `email-auth-resend.md` @@ -55,6 +59,8 @@ - 70.2K 历史漏账的取证 SQL + Loki query 模板、处理决策框架、修补后的监控建议 - `verifications/admin-user-balance-ban.md` - Admin role 鉴权 / 封禁热路径闸 / 改余额：role adminGuard、resolveRequestAuth+userinfo 封禁、setFlux 的真实 PGlite/Hono 执行证据（含 flux-grants 集成测试走 role），及 better-auth admin 端点本身待端到端实测 +- `verifications/product-analytics-smoke.md` + - 产品分析埋点上线冒烟清单：PostHog journey events、Postgres TTS metadata、Grafana Product Analytics row、Prometheus label 安全边界 ## 快速结论 diff --git a/apps/server/docs/ai-context/product-analytics-dashboard-setup.md b/apps/server/docs/ai-context/product-analytics-dashboard-setup.md new file mode 100644 index 000000000..d46773787 --- /dev/null +++ b/apps/server/docs/ai-context/product-analytics-dashboard-setup.md @@ -0,0 +1,305 @@ +# Product Analytics Dashboard Setup + +This document turns `product-analytics-instrumentation.md` into dashboard setup steps. It intentionally excludes Discord / QQ ingestion and daily / weekly report automation. + +After setting up the dashboards, run `verifications/product-analytics-smoke.md` against the deployed environment. + +## Scope + +In scope: + +- PostHog insights for frontend product journeys. +- Grafana panels for server-side product event health. +- Alert rules that can be configured directly in PostHog / Grafana. + +Out of scope for this pass: + +- Discord / QQ bot or spreadsheet synchronization. +- Daily / weekly report generation scripts. + +## Destination Rules + +| Question | Destination | Reason | +|---|---|---| +| Can a user start chatting? | PostHog | Frontend journey and distinct user funnels | +| Where does provider setup fail? | PostHog | Provider config events are frontend PostHog events | +| Which voice is selected or previewed? | PostHog / Postgres metadata | `voice_id` is high-cardinality and must not be a Prometheus label | +| Is server TTS healthy right now? | Grafana | Server product events and OTel metrics are Prometheus-safe | +| Are users submitting feedback? | PostHog | App feedback is frontend product analytics | + +## PostHog Dashboard + +Create a dashboard named `AIRI Activation And Feedback`. + +Live dashboard created on 2026-06-30: + +- Project: `Project AIRI (Web)` (`90721`) +- URL: `https://us.posthog.com/project/90721/dashboard/1779029` +- Current cards: + - Text card: `AIRI product analytics runbook` + - Funnel: `Chat activation funnel` + - Trend: `Provider config failures` + - Trend: `TTS voice selection and preview` + - Trend: `Top selected TTS voices` + - Trend: `Voice input friction` + - Trend: `Feedback and bug reports` + +### Insight 1: Chat Activation Funnel + +Type: Funnel + +Steps: + +1. `chat_activation_started` +2. `chat_activation_succeeded` + +Breakdowns: + +- `provider_mode` +- `surface` + +Filters: + +- Date range: last 7 days +- Exclude internal users if the project has an internal user cohort. + +Watch for: + +- Official provider conversion lower than custom provider conversion. +- Large drop after `chat_activation_started`. + +### Insight 2: Chat Activation Failures + +Type: Trends + +Events: + +- `chat_activation_failed` + +Breakdowns: + +- `failure_stage` +- `error_code` +- `provider_mode` + +Display: + +- Stacked bar or line chart. + +Watch for: + +- `failure_stage = provider_config` +- `failure_stage = model_list` +- `failure_stage = llm_response` + +### Insight 3: Provider Configuration Health + +Type: Funnel + +Steps: + +1. `provider_config_started` +2. `provider_config_succeeded` + +Breakdowns: + +- `provider_mode` +- `step` + +Companion trend: + +- Event: `provider_config_failed` +- Breakdown: `error_code` + +Watch for: + +- Official provider failures greater than zero for more than 15 minutes. +- `step = manual_chat_ping` failures after auto validation succeeds. + +### Insight 4: Model List Health + +Type: Trends + +Events: + +- `model_list_loaded` +- `model_list_failed` + +Breakdowns: + +- `provider_id` +- `provider_mode` + +Watch for: + +- `model_list_failed` spikes for one provider. +- High failure rate after a release. + +### Insight 5: TTS Voice Selection + +Type: Trends + +Events: + +- `voice_selected` +- `voice_preview_played` +- `voice_pack_bound` + +Breakdowns: + +- `voice_type` +- `tts_provider_id` +- `source` + +Do not use: + +- Prometheus labels for `voice_id` or `voice_pack_id`. + +Use PostHog or SQL when grouping by: + +- `voice_id` +- `voice_pack_id` + +### Insight 6: Voice Input Friction + +Type: Funnel + +Steps: + +1. `voice_input_started` +2. `stt_succeeded` + +Companion trends: + +- `microphone_permission_denied` +- `audio_device_unavailable` +- `voice_input_cancelled` +- `stt_failed` + +Breakdowns: + +- `stt_provider_id` +- `error_code` +- `surface` + +### Insight 7: Feedback And Bug Reports + +Type: Trends + +Events: + +- `feedback_submitted` +- `bug_report_submitted` + +Breakdowns: + +- `category` +- `severity` +- `entrypoint` +- `surface` + +Watch for: + +- `severity = blocker` spikes. +- `entrypoint = about_update_error` after desktop releases. + +## Grafana Dashboard + +Source of truth: + +- `apps/server/otel/grafana/dashboards/build.ts` +- Generated JSON: `apps/server/otel/grafana/dashboards/airi-server-overview-cloud.json` + +The `Product Analytics` row includes: + +- `Product Events (range)` +- `Product Failure %` +- `TTS Success %` +- `TTS Failed / Blocked (range)` +- `Top Product Actions (range)` +- `Product Event Rate` +- `TTS Event Rate by Source` + +Live import status: + +- Imported on 2026-06-30. +- Live URL: `https://projairi.grafana.net/d/ad8qbp5/airi-server-overview` +- Dashboard: `AIRI Server Overview - Product Analytics` (`ad8qbp5`) +- The live dashboard now shows the full `Product Analytics` row: + - `Product Events (range)` + - `Product Failure %` + - `TTS Success %` + - `TTS Failed / Blocked (range)` + - `Top Product Actions (range)` + - `Product Event Rate` + - `TTS Event Rate by Source` +- The generated JSON remains the source of truth for the Product Analytics / TTS panel set. + +Permission notes from the import retry: + +- The first import attempt with title `AIRI Server Overview` and UID `rbr55dn` showed duplicate title / UID warnings because it targets the existing dashboard. +- A second import attempt with a new title / UID (`AIRI Server Overview - Product Analytics Test`, `airi-product-analytics-test`) removed the duplicate warnings, but still did not import. +- API confirmation returned `403 Access denied`: `You'll need additional permissions to perform this action. Permissions needed: any of dashboards:create, dashboards:write`. +- The logged-in Grafana user `1260907335@qq.com` has org role `Viewer`; API metadata for `/d/rbr55dn/airi-server-overview` reports `canSave=false`, `canEdit=false`, `canAdmin=false`. +- After permissions were updated, the generated dashboard was imported from Microsoft Edge. Grafana assigned the imported dashboard UID `ad8qbp5` instead of overwriting the earlier `rbr55dn` dashboard, so the imported dashboard was renamed to `AIRI Server Overview - Product Analytics` to avoid ambiguity. + +Regenerate after dashboard changes: + +```bash +node node_modules/tsx/dist/cli.mjs apps/server/otel/grafana/dashboards/build.ts +``` + +## Alert Setup + +### PostHog Alerts + +Configure these as insight subscriptions or monitor-style alerts. + +| Alert | Insight | Trigger | +|---|---|---| +| Activation drop | Chat Activation Funnel | `chat_activation_succeeded / chat_activation_started` drops by 15% vs previous 24h | +| Provider config regression | Provider Configuration Health | Official provider `provider_config_failed` is greater than 0 for 15 minutes | +| Voice input spike | Voice Input Friction | `stt_failed / voice_input_started` exceeds 20% over 1h | +| Feedback spike | Feedback And Bug Reports | `bug_report_submitted` doubles vs previous 24h | + +### Grafana Alerts + +Use these PromQL expressions from the server dashboard context. + +TTS success below 95% over 15 minutes: + +```promql +100 * sum(increase(airi_product_events_total{feature="tts", action="speech_succeeded", status="succeeded"}[15m])) +/ +clamp_min(sum(increase(airi_product_events_total{feature="tts", action="speech_requested", status="started"}[15m])), 1) +< 95 +``` + +TTS blocked spike over 15 minutes: + +```promql +sum(increase(airi_product_events_total{feature="tts", action="speech_blocked", status="blocked"}[15m])) > 10 +``` + +TTS failed spike over 15 minutes: + +```promql +sum(increase(airi_product_events_total{feature="tts", action="speech_failed", status="failed"}[15m])) > 5 +``` + +Product failure ratio above 10% over 15 minutes: + +```promql +100 * sum(increase(airi_product_events_total{feature!="", action!="", status="failed"}[15m])) +/ +clamp_min(sum(increase(airi_product_events_total{feature!="", action!=""}[15m])), 1) +> 10 +``` + +## Verification Checklist + +- PostHog can show `chat_activation_started -> chat_activation_succeeded` by `provider_mode`. +- PostHog can show `voice_selected` by `voice_type` and `tts_provider_id`. +- PostHog can show `feedback_submitted` and `bug_report_submitted`. +- Grafana dashboard JSON contains `TTS Success %`, `TTS Failed / Blocked (range)`, and `TTS Event Rate by Source`. +- Grafana product analytics panels use only bounded labels: `feature`, `action`, `status`, `source`. diff --git a/apps/server/docs/ai-context/product-analytics-instrumentation.md b/apps/server/docs/ai-context/product-analytics-instrumentation.md new file mode 100644 index 000000000..93c736972 --- /dev/null +++ b/apps/server/docs/ai-context/product-analytics-instrumentation.md @@ -0,0 +1,891 @@ +# Product Analytics Instrumentation Plan（产品埋点与数据播报方案） + +这份文档把 AIRI 当前埋点现状、社区侧最关心的问题、缺口、事件 schema、看板和异常播报整理到一处。它补充 [`metrics-ownership.md`](./metrics-ownership.md)：后者定义指标归属和命名规则，本文定义“为了回答产品/社区问题，接下来要补什么”。 + +## TL;DR + +- 现在已经能看 activation、Provider 配置次数、国家来源、付费漏斗、LLM 请求和服务端 TTS 请求健康。 +- 现在还不能可靠回答“用户最常用哪个 TTS 音色 / Voice Pack”，因为前端和服务端事件都缺稳定的 `voice_id` / `voice_type` / `voice_pack_id`。 +- 最优先补的不是更多点击，而是“能不能开始聊天”“卡在哪个配置步骤”“语音 / TTS 为什么失败”。 +- 官方 Provider 要和自配置 Provider 分开看，核心判断是官方路径是否提高 activation、降低配置失败和缩短首次聊天时间。 +- 自配置用户可能更高价值，不能只看失败率；需要同时看 retention、paid conversion 和 feedback rate。 +- PostHog 负责用户路径、漏斗、留存和分群；Grafana 负责服务端健康和异常；Postgres / SQL 负责高基数字段聚合，例如 voice / voice pack。 +- Prometheus 不要加 `voice_id`、`voice_pack_id`、用户自定义模型名等高基数字段。 +- 日报先做“指标 + 异常提醒”，周报再加入社区侧解释和下周行动建议。 +- Discord / QQ 反馈要有轻量标签，不能完全靠埋点替代社区观察。 + +## 背景 + +社区反馈里最常见的劝退点集中在： + +- 性能问题 +- 模型 / Provider 配置复杂 +- 配置失败或模型列表加载失败 +- Bug 多，用户不知道卡在哪 +- 语音输入体验不稳定 + +产品方向是降低上手门槛，让 AIRI 更开箱即用；官方 Provider 已经上线，但仍有一部分用户喜欢自配置模型和音色。社区侧同时关心海外用户、付费用户、二次开发用户、角色聊天用户，以及愿意反馈问题的用户。 + +因此埋点目标不是单纯统计点击，而是回答： + +1. 新用户有没有正常开始聊天？ +2. 用户卡在 Provider / 模型配置的哪一步？ +3. 官方 Provider 是否真的降低了上手门槛？ +4. 自配置用户和官方 Provider 用户在留存、付费、反馈上有什么差异？ +5. TTS 音色和提供商到底怎么被选择、试听和实际使用？ +6. Bug、性能、语音输入失败分别造成了多少流失？ + +## 现状审计 + +### PostHog + +线上 `Project AIRI (Web)` 已有核心 dashboard：`My App Dashboard`。 + +已覆盖： + +- first message 趋势 +- Provider 配置次数 +- 国家来源 +- Signup activation funnel +- 7-day activation retention +- Paywall conversion +- LLM request volume / error rate +- rage-click friction + +近 7 天仍活跃的主要产品事件包括： + +- `app_loaded` +- `first_message_sent` +- `first_model_selected` +- `model_switched` +- `message_send_started` +- `llm_request_started` +- `llm_first_token` +- `message_round` +- `chat_session_started` +- `chat_session_selected` +- `chat_message_deleted` +- `chat_messages_cleared` +- `pricing_page_viewed` +- `plan_selected` +- `checkout_started` +- `provider_card_clicked` +- `stt_started` +- `stt_succeeded` +- `stt_failed` +- `tts_stop_clicked` +- `character_switched` +- `character_deleted` + +PostHog 当前缺口： + +- 没有 `voice_id`、`voice_pack_id`、`voice_type` 等 TTS 音色维度。 +- 近 30 天匹配 `tts` / `speech` / `voice` 的客户端事件里，`voice`、`voice_id`、`voiceId`、`voice_pack_id`、`voice_type` 均未出现有效值。 +- `model` / `model_id` 仍有自由文本风险，需要收敛白名单和脱敏策略。 +- PostHog 漏斗主要覆盖前端旅程，服务端 TTS 真实请求没有同步进入 PostHog。 + +### Grafana / Prometheus + +线上 `AIRI Server Overview` 已覆盖： + +- DAU / WAU / MAU +- Active sessions +- HTTP / WS health +- Product Analytics：event volume、failure rate、top product actions、event rate +- LLM Gateway：request rate by model、latency、provider failure +- TTS Characters/s by Model +- Stripe / revenue + +Grafana 当前能回答： + +- TTS 请求量、成功量、失败量、余额挡住量。 +- TTS 字符消耗按 model 聚合，例如 `stepfun/stepaudio-2.5-tts`、`volcengine/seed-tts-2.0`、`alibaba/cosyvoice-v1`。 +- 服务端 `product_events` 的 Prometheus label 当前包括 `feature`、`action`、`status`、`source`，再加部分 metric 的 `model` / `provider`。 + +Grafana 当前不能回答： + +- 哪个 TTS 音色使用最多。 +- 哪个 Voice Pack 使用最多。 +- 官方默认音色与用户自选音色的转化差异。 +- 试听音色后是否真的绑定 / 使用。 + +## 用户分群 + +第一版不要求用户手动选 persona，先从行为推断。 + +| Segment | 初始推断规则 | 用途 | +|---|---|---| +| `new_user` | 账号创建后 7 天内或首次 `app_loaded` 后 7 天内 | 新手引导、激活漏斗 | +| `official_provider_user` | 关键事件里 `provider_mode = official` | 衡量开箱即用效果 | +| `custom_provider_user` | 关键事件里 `provider_mode = custom` | 衡量自配置门槛和失败率 | +| `role_chat_user` | 导入 / 切换角色、绑定音色、频繁角色聊天 | 角色陪伴体验 | +| `developer_user` | 使用插件、API、Devtools、二开相关入口 | 二次开发群体 | +| `paid_user` | Stripe / Postgres 付费状态为真 | 付费转化与保留 | +| `feedback_user` | 提交 bug report / feedback | 高价值社区用户 | + +公共字段建议： + +| Field | Values | Notes | +|---|---|---| +| `surface` | `web` / `desktop` / `mobile` | 所有关键前端事件必带 | +| `provider_mode` | `official` / `custom` / `unknown` | 官方开箱即用 vs 用户自配置 | +| `provider_id` | 白名单 ID | 不传 raw URL / raw key / 用户输入 | +| `model_id` | 白名单或归一化后的 ID | 自定义模型用 `is_custom_model = true` | +| `is_custom_model` | boolean | 避免把任意文本当 group-by | +| `is_paid_user` | boolean | 从服务端或 PostHog person profile 派生 | +| `setup_completed` | boolean | 是否完成基础配置 | +| `region` | PostHog geo | 不由客户端手动传 | + +## P0 埋点 + +### Chat Activation + +目标：社区侧判断“能正常开始聊天”。 + +| Event | Owner | Truth | When | +|---|---|---|---| +| `chat_activation_started` | frontend | PostHog | 用户进入首次聊天路径或点击发送第一条消息前 | +| `chat_activation_succeeded` | frontend | PostHog | 首次消息完成并看到 assistant response | +| `chat_activation_failed` | frontend | PostHog | 首次消息未完成，包含配置、网络、鉴权、余额、模型等失败 | + +字段： + +| Field | Required | Notes | +|---|---|---| +| `provider_mode` | yes | `official` / `custom` | +| `provider_id` | yes | 归一化 ID | +| `model_id` | yes | 归一化 ID | +| `surface` | yes | web / desktop / mobile | +| `time_to_first_message_ms` | success only | 从 app start 或 onboarding complete 到首次成功 | +| `error_code` | failed only | 稳定错误码 | +| `failure_stage` | failed only | `provider_config` / `model_list` / `message_send` / `llm_response` / `tts` | + +推荐看板： + +- 新用户 `chat_activation_started -> chat_activation_succeeded` 漏斗。 +- 按 `provider_mode` 拆分 activation conversion。 +- `chat_activation_failed` 按 `failure_stage` / `provider_id` 排名。 + +异常提醒： + +- 新用户 activation conversion 24h 环比下降超过 15%。 +- `provider_mode = official` 的 activation failure 上升，优先排查官方 Provider。 + +### Provider And Model Configuration + +目标：定位配置复杂和失败劝退。 + +| Event | Owner | Truth | When | +|---|---|---|---| +| `provider_config_started` | frontend | PostHog | 打开 Provider 配置表单 | +| `provider_config_succeeded` | frontend | PostHog | 配置保存并通过最小校验 | +| `provider_config_failed` | frontend | PostHog | 保存、校验、鉴权或连接测试失败 | +| `model_list_loaded` | frontend | PostHog | 模型列表加载成功 | +| `model_list_failed` | frontend | PostHog | 模型列表加载失败 | + +字段： + +| Field | Required | Notes | +|---|---|---| +| `provider_id` | yes | 归一化 ID | +| `provider_mode` | yes | 官方 Provider 也要记录 | +| `step` | yes | `open_form` / `save` / `validate` / `load_models` / `select_model` | +| `duration_ms` | no | 保存或加载耗时 | +| `error_code` | failed only | 不能写 raw error message | +| `http_status` | failed only | 有 HTTP 边界时记录 | + +推荐看板： + +- Provider 配置成功率。 +- 模型列表加载成功率。 +- Top failing providers。 +- 官方 Provider vs 自配置 Provider 配置耗时和失败率。 + +异常提醒： + +- `model_list_failed` 任一 Provider 1h 内增长超过 2 倍。 +- 官方 Provider `provider_config_failed` 非零连续 15 分钟。 + +### TTS Voice And Provider + +目标：回答“用户选择了哪种音色和提供商”。 + +| Event | Owner | Truth | When | +|---|---|---|---| +| `tts_provider_selected` | frontend | PostHog | 用户选择或切换 TTS Provider | +| `voice_selected` | frontend | PostHog | 用户选择音色，包括官方默认落地时的 baseline | +| `voice_preview_played` | frontend | PostHog | 用户试听音色 | +| `voice_pack_bound` | frontend | PostHog | 用户把 Voice Pack 绑定到角色 | +| `speech_requested` | server | Postgres/Grafana | REST / WS TTS 请求开始；沿用现有 `feature = tts` action | +| `speech_succeeded` | server | Postgres/Grafana | REST / WS TTS 交付成功；沿用现有 `feature = tts` action | +| `speech_failed` | server | Postgres/Grafana | TTS 上游、配置、路由失败；沿用现有 `feature = tts` action | +| `speech_blocked` | server | Postgres/Grafana | Flux 不足等业务阻断；沿用现有 `feature = tts` action | + +字段： + +| Field | Required | Notes | +|---|---|---| +| `tts_provider_id` | yes | 归一化 ID | +| `tts_model_id` | yes | 归一化 ID | +| `voice_id` | yes | 官方 catalog ID；自定义音色见数据卫生 | +| `voice_type` | yes | `official_default` / `official_selected` / `custom_configured` / `voice_pack` | +| `voice_pack_id` | no | Voice Pack 绑定和服务端使用时记录 | +| `source` | yes | `settings` / `onboarding` / `chat_auto_tts` / `manual_preview` | +| `trigger` | server TTS | `auto` / `manual` | +| `input_chars` | server TTS | 字数 | +| `duration_ms` | server TTS | 端到端耗时 | +| `error_code` | failed only | 稳定错误码 | + +推荐看板： + +- Top voices by `voice_selected`。 +- Top voices by server `feature = tts` + `action = speech_succeeded`。 +- Voice preview to selection conversion。 +- Official default vs official selected vs custom configured。 +- TTS failure / blocked rate by provider, model, voice type。 + +异常提醒： + +- 某个官方音色的 server `speech_failed` 持续上升。 +- `voice_preview_played -> voice_selected` 转化下降。 +- server `speech_blocked` 突增，提示 Flux 或默认策略可能影响体验。 + +### Voice Input + +目标：语音输入是明确社区痛点，需要把权限、设备和 Provider 失败拆开。 + +当前已有： + +- `stt_started` +- `stt_succeeded` +- `stt_failed` + +补充： + +| Event | Owner | Truth | When | +|---|---|---|---| +| `voice_input_started` | frontend | PostHog | 用户开始语音输入 | +| `microphone_permission_requested` | frontend | PostHog | 首次或重新请求麦克风权限 | +| `microphone_permission_denied` | frontend | PostHog | 权限被拒绝 | +| `audio_device_unavailable` | frontend | PostHog | 无设备、设备被占用、采样失败 | +| `voice_input_cancelled` | frontend | PostHog | 用户主动取消或超时 | + +字段： + +| Field | Required | Notes | +|---|---|---| +| `stt_provider_id` | yes | 归一化 ID | +| `surface` | yes | web / desktop / mobile | +| `duration_ms` | no | 用户按住或录音时长 | +| `error_code` | failed only | `permission_denied` / `device_unavailable` / `provider_error` / `timeout` | + +推荐看板： + +- Voice input start -> STT success funnel。 +- Permission denied rate by browser / surface。 +- STT failure rate by Provider。 + +### Feedback And Bug Reports + +目标：社区侧认为“愿意反馈问题”是高价值行为，应进入核心指标。 + +| Event | Owner | Truth | When | +|---|---|---|---| +| `bug_report_opened` | frontend | PostHog | 打开 bug report dialog | +| `bug_report_submitted` | frontend/server | PostHog + Postgres optional | 成功提交 | +| `feedback_submitted` | frontend/server | PostHog + Postgres optional | 非 bug 的反馈 | +| `community_feedback_tagged` | manual/job | Postgres optional | Discord / QQ 反馈被人工归类 | + +字段： + +| Field | Required | Notes | +|---|---|---| +| `source` | yes | `app` / `discord` / `qq` / `github` / `email` / `other` | +| `category` | no | 见下方社区反馈标签 | +| `severity` | yes | `blocker` / `major` / `minor` / `suggestion` | +| `user_type` | yes | `new_user` / `paid_user` / `overseas_user` / `developer_user` / `role_chat_user` / `unknown` | +| `entrypoint` | yes | `about_update_error` / `community_manual_tag` 等低基数入口 | +| `surface` | in-app | web / desktop / mobile | +| `provider_mode` | no | 可从最近一次配置状态补齐 | +| `description_length_bucket` | bug report | `empty` / `short` / `medium` / `long`，不要上传正文 | +| `include_triage_context` | bug report | 是否附带页面上下文 | +| `screenshot_attached` | bug report | 是否附带截图或录屏 | + +推荐看板： + +- Feedback users count。 +- Bug report trend by category。 +- 配置失败后 24h 内是否反馈。 + +社区反馈标签建议： + +| Category | 适用反馈 | +|---|---| +| `performance` | 卡顿、慢、首 token 慢、内存 / CPU 异常 | +| `provider_config` | Provider 配置复杂、保存失败、Key / Endpoint 不知道怎么填 | +| `model_list` | 模型列表加载失败、模型不可选、模型名不符合预期 | +| `crash` | 崩溃、白屏、不可恢复异常 | +| `ui_ux` | UI 操作不顺、按钮找不到、信息表达不清 | +| `voice_input` | 麦克风权限、录音、STT、语音输入失败 | +| `tts` | TTS 音色、试听、Voice Pack、音色绑定问题 | +| `payment` | Flux、余额不足、付费、checkout、扣费疑问 | +| `chat_activation` | 新手无法开始聊天、首次发送失败、开箱即用失败 | +| `live2d` | Live2D / 模型显示 / 角色舞台问题 | +| `desktop_window` | 桌宠窗口、置顶、显示器、多屏问题 | +| `mobile` | iOS / Android / 移动端体验 | +| `unknown` | 信息不足，待社区负责人二次归类 | + +社区侧现在可以这样做： + +1. Discord / QQ 反馈先人工记录，不等 App 内入口完善。 +2. 每条反馈只需要填：日期、来源、用户类型、分类、严重程度、原文链接、简短摘要、是否已转 issue。 +3. 原文和截图留在 Discord / QQ / issue，不进 PostHog；PostHog 只放标签和计数。 +4. 每周把 `category` + `severity` 聚合进周报，用来解释为什么某个指标变差。 + +## P1 埋点 + +### Onboarding + +当前 `onboarding_step_completed` 不足以定位用户卡点。 + +补充： + +- `onboarding_started` +- `onboarding_step_viewed` +- `onboarding_step_completed` +- `onboarding_skipped` +- `onboarding_failed` + +字段： + +- `step` +- `time_spent_ms` +- `provider_mode` +- `provider_id` +- `error_code` + +看板： + +- Onboarding step drop-off。 +- 官方 Provider onboarding conversion。 +- Skip 后是否仍完成 `chat_activation_succeeded`。 + +### Chat Reliability + +补充： + +- `message_send_failed` +- `assistant_response_failed` +- `generation_cancelled` +- `generation_retried` +- `response_regenerated` + +字段： + +- `model_id` +- `provider_id` +- `provider_mode` +- `has_voice` +- `latency_ms` +- `error_code` +- `surface` + +看板： + +- Message send success rate。 +- Retry / cancel rate。 +- Failure by provider and model。 + +### Character And Role Usage + +补充： + +- `character_created` +- `character_imported` +- `character_edited` +- `character_switched` +- `character_deleted` +- `display_model_changed` +- `voice_pack_bound` + +字段： + +- `character_type`: `built_in` / `imported` / `custom` +- `has_voice` +- `voice_type` +- `surface` + +看板： + +- Character adoption。 +- 角色用户与普通聊天用户 retention / payment 差异。 + +### Payment And Flux Friction + +当前已有 `pricing_page_viewed`、`plan_selected`、`checkout_started`，服务端有 payment truth。补充： + +- `flux_low_warning_shown` +- `flux_topup_clicked` +- `checkout_failed` +- `payment_completed_imported` + +字段： + +- `surface` +- `balance_state` +- `plan_id` +- `currency` +- `error_code` + +注意： + +- `payment_completed` 真相仍在 Postgres / Stripe webhook。 +- PostHog 只用于漏斗展示，优先用 Stripe connector 或离线导入。 + +## 事件复用关系 + +新增埋点时先复用现有事件，不要把相同事实拆成多个名字。 + +| Existing / New | Relationship | Notes | +|---|---|---| +| `first_message_sent` | 保留历史指标 | 继续用于老 dashboard；新激活口径用 `chat_activation_succeeded` | +| `chat_activation_started` / `chat_activation_succeeded` / `chat_activation_failed` | 新核心 activation 口径 | 用来回答“用户能不能正常开始聊天” | +| `provider_card_clicked` | 保留入口点击 | 不等于配置成功；成功 / 失败看 `provider_config_succeeded` / `provider_config_failed` | +| `first_model_selected` / `model_switched` | 保留模型选择行为 | 配置链路和模型列表健康看 `model_list_loaded` / `model_list_failed` | +| `stt_started` / `stt_succeeded` / `stt_failed` | 保留 STT Provider 结果 | 权限和设备问题用新增 `microphone_*` / `audio_device_unavailable` 拆开 | +| `tts_stop_clicked` | 保留用户停止行为 | 不代表音色选择；音色选择用 `voice_selected` | +| `speech_requested` / `speech_succeeded` / `speech_failed` / `speech_blocked` | 服务端 TTS truth | 继续沿用 `feature = tts`，只补 metadata 字段 | +| `pricing_page_viewed` / `plan_selected` / `checkout_started` | 保留付费漏斗前段 | 真正 payment completed 仍以 Stripe / Postgres 为准 | + +## 数据卫生 + +### 不要把自由文本直接作为分析维度 + +PostHog 线上已经能看到 `model` / `model_id` 存在自由文本风险。后续新增字段必须遵循： + +- Provider、model、voice 使用稳定 ID。 +- 自定义值不要直接 group-by。 +- 自定义模型传： + - `provider_id = custom` + - `model_family = custom` + - `is_custom_model = true` + - `custom_model_hash` 可选，必须单向 hash，不能还原原文。 +- 自定义 voice 传： + - `voice_type = custom_configured` + - `voice_id = custom` + - `custom_voice_hash` 可选，必须单向 hash。 +- 错误字段传稳定 `error_code`，不要传 raw error message。 + +### 字段基数约束 + +| Field | Cardinality | Rule | +|---|---|---| +| `provider_id` | low | 白名单 | +| `provider_mode` | low | enum | +| `model_id` | medium | 官方 catalog 或归一化 ID | +| `voice_id` | medium | 官方 catalog 或 `custom` | +| `voice_pack_id` | medium | 只进 PostHog / Postgres，不进 Prometheus label | +| `error_code` | low | enum | +| `source` | low | enum | +| `surface` | low | enum | + +Prometheus label 不放 `user_id`、`session_id`、`voice_pack_id`、自定义模型名、自定义音色名。 + +### 不要做什么 + +- 不要把用户输入、聊天正文、角色 prompt、raw API key、raw Endpoint、raw model name、raw voice name 发到 PostHog / Grafana。 +- 不要把 raw error message 作为分析字段；统一映射成稳定 `error_code`。 +- 不要在 Prometheus label 里加入 `voice_id`、`voice_pack_id`、`session_id`、`user_id`、自定义模型名、自定义音色名。 +- 不要为了看一个 funnel 同时新增两个语义相同的事件；优先查上面的事件复用关系。 +- 不要在请求主链路里同步等待 PostHog 发送完成；失败不能影响用户聊天 / TTS。 +- 不要只看点击量判断用户意图；至少结合 success / failed / blocked 和社区反馈标签。 + +## 看板建议 + +### PostHog + +新建或扩展 dashboard：`Onboarding and Activation` + +- 新用户 activation funnel： + - `app_loaded` + - `chat_activation_started` + - `provider_config_succeeded` + - `model_list_loaded` + - `chat_activation_succeeded` +- Official vs custom Provider activation split。 +- Provider configuration failure ranking。 +- Onboarding step drop-off。 + +新建 dashboard：`Voice and TTS Adoption` + +- Top voices by selected users。 +- Top voices by successful TTS requests。 +- Voice preview -> selection conversion。 +- Official default vs selected vs custom configured。 +- Voice input start -> STT success funnel。 + +扩展 `My App Dashboard` + +- 保留现有 activation、paywall、LLM、rage-click。 +- 增加 `chat_activation_succeeded`、`provider_config_failed`、`stt_failed`、`voice_selected` 摘要卡。 + +### Grafana + +扩展 `AIRI Server Overview` + +- TTS request / success / failed / blocked by source。 +- TTS character total by model over range。 +- Product action failure rate by feature/action。 + +不要在 Prometheus 增加 `voice_id` label。若要看音色排行： + +- Postgres `product_events.metadata.voice_id` 做 SQL / admin API 聚合。 +- 或离线导入 PostHog，用 PostHog group-by 展示。 + +## 播报、分析和可视化方案 + +### 分层原则 + +播报要分三层，不要把所有指标塞进一个 dashboard。 + +| Layer | Frequency | Audience | Goal | Tool | +|---|---|---|---|---| +| 实时异常 | 5m - 1h | 工程 / on-call / 社区负责人 | 发现“今天是不是坏了” | Grafana alert + PostHog insight alert | +| 日报 | daily | 社区 / 产品 / 工程 | 看上手、聊天、语音、付费是否正常 | PostHog + Grafana + 少量 SQL | +| 周报 | weekly | 产品 / 战略讨论 | 看趋势、用户意图、投入方向 | PostHog cohort/funnel + SQL 聚合 | + +工具分工： + +| Tool | 用途 | 不适合做什么 | +|---|---|---| +| PostHog | 用户路径、漏斗、留存、分群、前端行为 | 服务端真实扣费、低延迟 on-call | +| Grafana | 服务端健康、TTS/LLM 请求、错误、余额阻断、告警 | 高基数用户行为和音色排行 | +| Postgres / SQL | 付费事实、`product_events.metadata`、voice / voice pack 聚合 | 实时看板和复杂前端路径 | +| Community tags | Discord / QQ 反馈归类 | 自动替代埋点 | + +### 日报模板 + +日报回答“今天是否健康，有没有需要马上处理的问题”。 + +```md +# AIRI 数据日报 YYYY-MM-DD + +## 核心状态 + +- DAU: () +- New users: +- Chat activation: +- Official provider activation: +- Custom provider activation: +- Paid conversion proxy: pricing -> plan -> checkout = + +## 上手和配置 + +- Top provider config failures: + 1. / / + 2. / / +- Model list failure rate: +- New-user first-message median time: + +## 聊天与性能 + +- Message round success: +- LLM first-token p95: +- LLM / chat failures by provider: + 1. / / + +## 语音与 TTS + +- STT success rate: +- Microphone permission denied: +- TTS success / failed / blocked: // +- Top voices selected: + 1. / + 2. / +- Top voices used: + 1. / + 2. / + +## 反馈与异常 + +- Bug reports: +- Feedback submitted: +- Discord / QQ 高关键词: + - : +- 异常提醒: + - : vs , 建议动作 +``` + +第一版日报可以先不追求自动生成完整解释，只要自动填指标，并把异常规则命中的项放到“异常提醒”即可。 + +### 周报模板 + +周报回答“用户意图有什么变化，战略上要改什么”。 + +```md +# AIRI 数据周报 YYYY-WW + +## 本周结论 + +1. <最重要趋势，例如 official provider 激活率上升> +2. <最大风险，例如 custom provider 配置失败仍高> +3. <建议动作，例如下周优先修 model list failed> + +## 新手上手 + +- New users: +- Activation funnel: + - app_loaded -> chat_activation_started: + - chat_activation_started -> provider_config_succeeded: + - provider_config_succeeded -> chat_activation_succeeded: +- Official vs custom: + - official activation: + - custom activation: + - conclusion: + +## 用户意图 + +- Role chat users: +- Developer users: +- Voice users: +- Paid users: +- Feedback users: +- 海外用户占比: + +## 语音和音色 + +- Top selected voices: +- Top used voices: +- Default voice adoption: +- Custom configured voice adoption: +- Voice preview -> selected conversion: +- TTS blocked / failed trend: + +## 配置和 Bug + +- Top failing providers: +- Top failing error codes: +- Rage-click pages / surfaces: +- Discord / QQ feedback categories: + - performance: + - config: + - bug: + - voice_input: + +## 下周建议 + +- Product: +- Engineering: +- Community: +``` + +周报需要人工写“结论”和“建议动作”。指标只负责提示方向，社区反馈负责解释为什么。 + +### 可视化布局 + +#### Executive Overview + +给产品 / 社区快速看： + +- Activation conversion +- Official vs custom activation +- Provider config failures +- STT success rate +- TTS success / blocked +- Top voices selected / used +- Bug reports / feedback +- Paywall funnel + +#### Onboarding And Activation + +给负责上手体验的人看： + +- Funnel：`app_loaded -> chat_activation_started -> provider_config_succeeded -> model_list_loaded -> chat_activation_succeeded` +- Breakdown：`provider_mode`、`surface`、`region` +- Table：Top `provider_config_failed` by `provider_id` / `error_code` +- Timeseries：`time_to_first_message_ms` p50 / p95 + +#### Voice And TTS Adoption + +给语音和角色体验看： + +- Bar：Top voices by selected users +- Bar：Top voices by successful TTS requests +- Funnel：`voice_preview_played -> voice_selected -> speech_succeeded` +- Timeseries：`speech_succeeded` / `speech_failed` / `speech_blocked` +- Breakdown：`voice_type` = official default / official selected / custom configured / voice pack + +#### Reliability And Friction + +给工程和社区排障看： + +- Grafana：5xx、LLM latency、provider failure、TTS blocked +- PostHog：rage-click trend、failed frontend events +- Table：Top error_code by surface / provider +- Community tags：Discord / QQ 反馈分类趋势 + +### 分析方法 + +#### Official provider 是否降低门槛 + +看： + +- `chat_activation_succeeded / chat_activation_started` by `provider_mode` +- `time_to_first_message_ms` by `provider_mode` +- `provider_config_failed` by `provider_mode` +- D7 retention by `provider_mode` + +如果 official 激活率高、耗时短、失败率低，说明开箱即用策略有效。若 official 使用率高但失败率也高，优先修官方 Provider 稳定性。 + +#### 自配置用户是不是更高价值 + +看： + +- `custom_provider_user` 的 D7 / D30 retention +- `custom_provider_user` 的 feedback rate +- `custom_provider_user` 的 paid conversion +- `custom_provider_user` 的 config failure rate + +如果自配置用户付费和反馈更高，但失败率也高，可以把高级配置保留，但需要更好的错误提示和导入模板。 + +#### 哪些 Bug 最劝退 + +看： + +- `chat_activation_failed` by `failure_stage` +- `provider_config_failed` by `error_code` +- `model_list_failed` by `provider_id` +- `$rageclick` by page / surface +- Discord / QQ `category = bug` 的高频词 + +日报只报异常；周报把异常和社区反馈合并成“优先修复建议”。 + +#### TTS 音色策略 + +看： + +- `voice_selected` users by `voice_id` +- `speech_succeeded` count by `voice_id` +- `voice_preview_played -> voice_selected` conversion by `voice_id` +- `speech_failed / speech_requested` by `voice_id` +- retention / paid conversion by `voice_type` + +用法： + +- 选择多但使用少：可能试听不错，实际聊天不合适。 +- 使用多但失败高：优先修该音色或 Provider。 +- 默认音色使用高但切换少：默认可能足够好，也可能用户没发现入口，需要结合 `voice_preview_played` 看。 +- 自定义音色用户留存高：说明高阶用户重视声音个性化。 + +### 自动化路线 + +第一阶段：半自动日报。 + +- Grafana 提供服务端健康和 TTS/LLM 指标。 +- PostHog 提供 activation、provider、voice、STT、feedback 指标。 +- SQL 提供 voice / voice pack 聚合。 +- 由脚本生成 Markdown，发到 Discord / QQ / 飞书其中一个固定频道。 + +第二阶段：异常驱动播报。 + +- 每小时检查 activation、provider config、model list、STT、TTS blocked、bug report。 +- 只有超过阈值才发提醒。 +- 提醒里必须带“建议查看哪个 dashboard / query”。 + +第三阶段：周报带人工结论。 + +- 自动填指标和 Top lists。 +- 社区负责人补充 Discord / QQ 反馈解释。 +- 产品 / 工程共同确认下周行动。 + +## 异常播报 + +第一版日报 / 周报走“指标 + 异常提醒”。日报不要做复杂归因；周报允许加入人工结论。 + +### 每日必看指标 + +建议内容： + +- DAU / WAU / MAU。 +- New users。 +- `chat_activation_succeeded` 转化率。 +- Official vs custom Provider activation conversion。 +- Top Provider config failures。 +- STT success rate / failure rate。 +- TTS success / failed / blocked。 +- Top voices selected / used。 +- Bug reports / feedback count。 +- Paywall funnel：pricing -> plan -> checkout。 + +### 异常规则 + +| Alert | Trigger | Action | +|---|---|---| +| Activation drop | 24h `chat_activation_succeeded / chat_activation_started` 环比下降 15% | 查 Provider config / model_list / LLM failures | +| Official Provider regression | 官方 Provider `provider_config_failed` 连续 15 分钟非零或 24h 明显上升 | 优先排官方配置 | +| Model list failure spike | 任一 Provider `model_list_failed` 1h 翻倍 | 查 Provider API / auth / CORS | +| STT failure spike | `stt_failed / stt_started` 超过阈值 | 查权限、设备、Provider | +| TTS blocked spike | `feature = tts` + `action = speech_blocked` 1h 翻倍 | 查 Flux、默认音色成本、余额提示 | +| Bug report spike | `bug_report_submitted` 24h 翻倍 | 社区同步归类 | +| Rage-click spike | `$rageclick` 7d trend 异常 | 结合页面和 session replay | + +## 实施顺序 + +### 分阶段落地 + +| Phase | Scope | Owner | 产出 | +|---|---|---|---| +| Phase 0 | 字段归一化、事件复用确认、敏感字段拦截 | frontend / server | 公共 helper、事件字典、测试用例 | +| Phase 1 | Chat activation、Provider / model 配置、TTS voice 字段 | frontend / server | 能回答“能不能开始聊天”和“哪个音色常用” | +| Phase 2 | Voice input、feedback、community tags | frontend / community | 能定位语音输入和社区反馈高频问题 | +| Phase 3 | PostHog / Grafana dashboard、半自动日报、异常提醒 | data / server / community | 每日播报和异常告警可用 | +| Phase 4 | 周报、cohort、retention、付费 / 反馈关联分析 | product / community / data | 支持战略复盘和下周优先级 | + +### 任务顺序 + +1. 先补字段归一化 helpers，尤其是 Provider / model / voice。 +2. 补 Chat Activation 三个事件。 +3. 补 Provider / model 配置成功失败事件。 +4. 补 TTS voice 选择、试听、Voice Pack 绑定事件。 +5. 服务端 `product_events.metadata` 补 TTS `voice_id`、`voice_type`、`voice_pack_id`。 +6. 补语音输入权限 / 设备事件。 +7. 扩 PostHog dashboard。 +8. 扩 Grafana dashboard，但不把 voice 放进 Prometheus label。 +9. 建半自动日报 Markdown：PostHog + Grafana + SQL 聚合，先发固定频道。 +10. 建异常检查：activation、provider config、model list、STT、TTS blocked、bug report。 +11. 建周报模板：自动填指标，社区负责人补充 Discord / QQ 反馈解释和下周建议。 + +### 当前接入状态（2026-06-30） + +已接入代码： + +- Chat activation：`chat_activation_started`、`chat_activation_succeeded`、`chat_activation_failed`。 +- Model list：`model_list_loaded`、`model_list_failed`。 +- Provider config：`provider_config_started`、`provider_config_succeeded`、`provider_config_failed`。 +- TTS voice：`tts_provider_selected`、`voice_selected`、`voice_preview_played`、`voice_pack_bound`。 +- TTS 服务端 metadata：REST / WS TTS `product_events.metadata` 已补 `voice_id`、`voice_type`、`voice_pack_id`。 +- Voice input：`voice_input_started`、`microphone_permission_requested`、`microphone_permission_denied`、`audio_device_unavailable`、`voice_input_cancelled`。 +- STT：保留 `stt_started`、`stt_succeeded`、`stt_failed`，并将失败码收敛到稳定枚举，避免上报 raw error。 +- Feedback：`feedback_submitted` / `bug_report_submitted` 的低基数字段与 analytics API 已定义；产品内反馈提交入口与服务端收件流程拆到单独 PR。 +- Grafana Dashboard：`Product Analytics` 行已补 TTS success、TTS failed / blocked、TTS event rate by source 面板，并保留 voice drilldown 在 Postgres metadata / PostHog，不进入 Prometheus labels。 +- Dashboard setup 文档：`product-analytics-dashboard-setup.md` 已补 PostHog insights、Grafana panels、PostHog / Grafana alert 配置建议。 +- 上线冒烟文档：`verifications/product-analytics-smoke.md` 已补 PostHog、Postgres、Grafana 三层验证步骤。 + +待接入或待产品确认： + +- Discord / QQ 社区标签的数据入口，例如人工表格、bot 或 issue 同步。 +- PostHog Dashboard 需要在 PostHog 账号里按 setup 文档创建 insights / alerts。 +- Grafana Dashboard JSON 已更新，仍需部署 / import 到线上 Grafana。 +- 日报 / 周报自动拉取脚本与提醒频道。 + +## 验证清单 + +- 新用户完成一次官方 Provider 聊天后，PostHog 能看到 `chat_activation_succeeded`。 +- 自配置 Provider 失败时，PostHog 能按 `provider_id` + `error_code` 聚合。 +- 选择官方默认音色后，PostHog 能看到 `voice_selected` 且 `voice_type = official_default`。 +- 试听音色后，PostHog 能看到 `voice_preview_played`。 +- REST 和 WS TTS 成功后，Postgres `product_events.metadata` 能看到 `voice_id`。 +- Grafana 继续只按低基数字段聚合，不新增高基数 voice label。 +- 日报能输出：activation conversion、Top failing providers、STT success rate、TTS success/blocked、Top voices、feedback count。 +- 异常提醒命中时能带上：当前值、基线值、影响范围、建议查看的 dashboard / query。 +- 周报能输出：本周结论、用户意图变化、Discord / QQ 反馈分类、下周 Product / Engineering / Community 行动建议。 diff --git a/apps/server/docs/ai-context/verifications/product-analytics-smoke.md b/apps/server/docs/ai-context/verifications/product-analytics-smoke.md new file mode 100644 index 000000000..57c99d632 --- /dev/null +++ b/apps/server/docs/ai-context/verifications/product-analytics-smoke.md @@ -0,0 +1,275 @@ +# Verification: Product Analytics Smoke Test + +Status: **code-level instrumentation verified; live PostHog dashboard created; Grafana dashboard imported; alert setup pending** +Owner: Community / Product Analytics +Last updated: 2026-06-30 +Related: +- [`product-analytics-instrumentation.md`](../product-analytics-instrumentation.md) +- [`product-analytics-dashboard-setup.md`](../product-analytics-dashboard-setup.md) +- [`airi-server-overview-cloud.json`](../../../otel/grafana/dashboards/airi-server-overview-cloud.json) + +## 用户路径 + +- **场景**：验证新增埋点能回答“用户是否能正常开始聊天”“Provider 配置卡在哪里”“哪个 TTS 音色被选择 / 实际播放”“语音输入卡在哪里”“用户是否提交反馈”。 +- **预期**：PostHog 能看到前端 journey events；Postgres `product_events` 能看到服务端 TTS metadata；Grafana 能看到低基数 server-side product health。 +- **当前状态**：代码与 dashboard JSON 已验证；线上 PostHog dashboard 已创建；线上 Grafana `AIRI Server Overview - Product Analytics` (`ad8qbp5`) 已导入完整 Product Analytics row；alert 仍需人工配置。 + +## 已经由代码验证 + +| Area | Evidence | +|---|---| +| Frontend analytics API | `packages/stage-ui/src/composables/use-analytics.test.ts` 覆盖 activation、model list、provider config、voice selection、voice input、feedback event API | +| Chat activation hooks | `packages/core-agent/src/runtime/chat-orchestrator-runtime.test.ts` 覆盖 activation started / succeeded / failed hook | +| Voice input failures | `packages/stage-ui/src/composables/audio/audio-device.test.ts` 与 `packages/stage-ui/src/stores/modules/hearing.analytics.test.ts` 覆盖 permission / device / cancel / STT failed | +| Server TTS metadata | `apps/server/src/routes/openai/v1/route.test.ts` 与 `apps/server/src/routes/audio-speech-ws/route.test.ts` 覆盖 REST / WS TTS `voice_id`、`voice_type`、`voice_pack_id` metadata | +| Grafana product row | `apps/server/otel/grafana/dashboards/build.test.ts` 覆盖 Product Analytics panels、layout references、PromQL 不包含 high-cardinality voice / user fields | + +## Live Smoke Checklist + +Run this after deploying a build with the instrumentation changes. The PostHog dashboard and Grafana dashboard shell are already created, but they still need live event traffic from the deployed build. + +### 1. PostHog: chat activation + +Action: + +1. Use a fresh or test account. +2. Start with an official provider. +3. Send the first chat message and wait for the assistant response. + +Expected PostHog events: + +```text +chat_activation_started +chat_activation_succeeded +``` + +Required properties: + +```text +provider_mode = official +provider_id = +model_id = +surface = web | mobile | electron +``` + +Fail if: + +- `chat_activation_started` appears but `chat_activation_succeeded` never appears for a successful chat. +- `provider_mode` is missing or always `unknown`. +- `surface` is missing. + +### 2. PostHog: provider config failure + +Action: + +1. Configure a custom provider with an invalid key or invalid endpoint. +2. Trigger settings validation or manual chat ping. + +Expected PostHog events: + +```text +provider_config_started +provider_config_failed +``` + +Required properties: + +```text +provider_mode = custom +provider_id = +step = settings_auto_validate | manual_chat_ping +error_code = +``` + +Fail if: + +- Raw error text, API key fragments, endpoint secrets, or stack traces appear in event properties. +- `provider_config_failed` has no `error_code`. + +### 3. PostHog: TTS voice selection + +Action: + +1. Open speech settings. +2. Select an official TTS provider. +3. Select or keep an official voice. +4. Play voice preview once. + +Expected PostHog events: + +```text +tts_provider_selected +voice_selected +voice_preview_played +``` + +Required properties: + +```text +tts_provider_id = +tts_model_id = +voice_id = +voice_type = official_default | official_selected | custom_configured | voice_pack | unknown +source = settings | manual_preview +``` + +Fail if: + +- `voice_selected` is missing, because this blocks “哪个 TTS 音色比较多”的核心问题。 +- Official default voice is indistinguishable from custom configured voice. + +### 4. PostHog: voice input friction + +Action: + +1. Start voice input. +2. Test one failure path: deny microphone permission, use a browser/device with no microphone, or cancel input. + +Expected PostHog events: + +```text +voice_input_started +microphone_permission_requested +microphone_permission_denied +audio_device_unavailable +voice_input_cancelled +stt_failed +``` + +Only the events that match the exercised path need to appear. + +Fail if: + +- Permission denied or device unavailable is only visible as a generic `stt_failed`. +- `error_code` contains raw browser error text. + +### 5. Postgres: server-side TTS metadata + +Action: + +1. Trigger one REST TTS request. +2. Trigger one chat/WS TTS request if the deployed environment supports it. + +Query: + +```sql +SELECT + created_at, + user_id, + source, + provider, + model, + action, + status, + metadata->>'voice_id' AS voice_id, + metadata->>'voice_type' AS voice_type, + metadata->>'voice_pack_id' AS voice_pack_id +FROM product_events +WHERE feature = 'tts' + AND created_at >= now() - interval '1 hour' +ORDER BY created_at DESC +LIMIT 50; +``` + +Expected: + +- `speech_requested` and `speech_succeeded` rows exist for successful TTS. +- `voice_id` is present when the request provided a selected voice. +- `voice_type` distinguishes official default / selected / custom / voice pack where available. + +Fail if: + +- TTS succeeds but no `product_events` row is written. +- Metadata contains raw prompts, message text, API keys, or request bodies. + +### 7. Grafana: Product Analytics row + +Action: + +1. Open `https://projairi.grafana.net/d/ad8qbp5/airi-server-overview`. +2. Open the `Product Analytics` row. +3. Use a 1h time range after running the smoke actions above. + +Expected panels: + +```text +Product Events (range) +Product Failure % +TTS Success % +TTS Failed / Blocked (range) +Top Product Actions (range) +Product Event Rate +TTS Event Rate by Source +``` + +PromQL sanity: + +```promql +sum(increase(airi_product_events_total{feature="tts"}[1h])) +``` + +Expected: + +- Query returns a non-zero value after TTS smoke actions. +- Legends only use bounded labels: `feature`, `action`, `status`, `source`. + +Fail if: + +- Prometheus labels contain `voice_id`, `voice_pack_id`, `user_id`, `session_id`, or `request_id`. +- Grafana shows product panels but Postgres has no matching TTS rows. + +## Quick Analysis Queries + +Top selected voices should come from PostHog `voice_selected` for frontend intent. Server-side playback can be cross-checked from Postgres: + +```sql +SELECT + metadata->>'voice_id' AS voice_id, + metadata->>'voice_type' AS voice_type, + provider, + model, + COUNT(*) AS play_count, + COUNT(DISTINCT user_id) AS distinct_users +FROM product_events +WHERE feature = 'tts' + AND action = 'speech_succeeded' + AND created_at >= now() - interval '7 days' +GROUP BY 1, 2, 3, 4 +ORDER BY play_count DESC +LIMIT 20; +``` + +Server-side TTS blocked / failed ranking: + +```sql +SELECT + action, + status, + source, + reason, + COUNT(*) AS event_count, + COUNT(DISTINCT user_id) AS distinct_users +FROM product_events +WHERE feature = 'tts' + AND action IN ('speech_failed', 'speech_blocked') + AND created_at >= now() - interval '24 hours' +GROUP BY 1, 2, 3, 4 +ORDER BY event_count DESC; +``` + +## Exit Criteria + +| Item | Pass condition | +|---|---| +| Activation | PostHog funnel shows `chat_activation_started -> chat_activation_succeeded` by `provider_mode` | +| Provider config | Failed custom config emits `provider_config_failed` with bounded `error_code` | +| TTS voice | PostHog can rank `voice_selected` by `voice_id`; Postgres can rank actual `speech_succeeded` by metadata voice | +| Voice input | Permission / device / cancel paths are distinguishable | +| Feedback | Feedback event API exists with bounded fields; product feedback UI/server submission is split into a separate PR | +| Grafana | Product Analytics row renders and uses only bounded Prometheus labels | + +## Known Pending Work + +- PostHog dashboard and alerts still need to be created inside the PostHog account. +- Updated Grafana JSON still needs to be imported or deployed to the production Grafana workspace. +- Discord / QQ ingestion and daily / weekly automation scripts are intentionally excluded from this pass. diff --git a/apps/server/otel/grafana/dashboards/airi-server-overview-cloud.json b/apps/server/otel/grafana/dashboards/airi-server-overview-cloud.json index 723ce42ed..0806a459f 100644 --- a/apps/server/otel/grafana/dashboards/airi-server-overview-cloud.json +++ b/apps/server/otel/grafana/dashboards/airi-server-overview-cloud.json @@ -1274,6 +1274,294 @@ } } }, + "panel-99": { + "kind": "Panel", + "spec": { + "data": { + "kind": "QueryGroup", + "spec": { + "queries": [ + { + "kind": "PanelQuery", + "spec": { + "hidden": false, + "query": { + "datasource": { + "name": "grafanacloud-projairi-prom" + }, + "group": "prometheus", + "kind": "DataQuery", + "spec": { + "expr": "100 * sum(increase(airi_product_events_total{service_name=~\"$service\", deployment_environment=~\"$env\", feature!=\"\", action!=\"\", feature=\"tts\", action=\"speech_succeeded\", status=\"succeeded\"}[$__range])) / clamp_min(sum(increase(airi_product_events_total{service_name=~\"$service\", deployment_environment=~\"$env\", feature!=\"\", action!=\"\", feature=\"tts\", action=\"speech_requested\", status=\"started\"}[$__range])), 1)", + "legendFormat": "success %" + }, + "version": "v0" + }, + "refId": "A" + } + } + ], + "queryOptions": {}, + "transformations": [] + } + }, + "description": "Server-side TTS successes divided by TTS requests over the dashboard range. Includes REST and WS TTS product events. Drops here mean users are asking for speech but not receiving audio; inspect failed/blocked panels next.", + "id": 99, + "links": [], + "title": "TTS Success %", + "vizConfig": { + "group": "gauge", + "kind": "VizConfig", + "spec": { + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "red", + "value": 0 + }, + { + "color": "yellow", + "value": 90 + }, + { + "color": "green", + "value": 98 + } + ] + }, + "unit": "percent", + "decimals": 2, + "noValue": "0", + "min": 0, + "max": 100 + }, + "overrides": [] + }, + "options": { + "minVizHeight": 75, + "minVizWidth": 75, + "orientation": "auto", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showThresholdLabels": false, + "showThresholdMarkers": true, + "sizing": "auto" + } + }, + "version": "13.0.0-23630096546" + } + } + }, + "panel-100": { + "kind": "Panel", + "spec": { + "data": { + "kind": "QueryGroup", + "spec": { + "queries": [ + { + "kind": "PanelQuery", + "spec": { + "hidden": false, + "query": { + "datasource": { + "name": "grafanacloud-projairi-prom" + }, + "group": "prometheus", + "kind": "DataQuery", + "spec": { + "expr": "topk(12, sum by (action, status, source) (increase(airi_product_events_total{service_name=~\"$service\", deployment_environment=~\"$env\", feature!=\"\", action!=\"\", feature=\"tts\", action=~\"speech_failed|speech_blocked\"}[$__range])))", + "legendFormat": "{{action}} · {{status}} · {{source}}", + "instant": true, + "range": false + }, + "version": "v0" + }, + "refId": "A" + } + } + ], + "queryOptions": {}, + "transformations": [] + } + }, + "description": "TTS user-impacting failures over the dashboard range, split by action/status/source. `speech_failed` usually means upstream/runtime failure; `speech_blocked` usually means balance/preflight blocked. Keep voice/model drilldown in Postgres metadata, not Prometheus labels.", + "id": 100, + "links": [], + "title": "TTS Failed / Blocked (range)", + "vizConfig": { + "group": "bargauge", + "kind": "VizConfig", + "spec": { + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + }, + "unit": "short", + "noValue": "0" + }, + "overrides": [] + }, + "options": { + "displayMode": "gradient", + "maxVizHeight": 300, + "minVizHeight": 12, + "minVizWidth": 8, + "namePlacement": "auto", + "orientation": "horizontal", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showUnfilled": true, + "sizing": "auto", + "valueMode": "color" + } + }, + "version": "13.0.0-23630096546" + } + } + }, + "panel-101": { + "kind": "Panel", + "spec": { + "data": { + "kind": "QueryGroup", + "spec": { + "queries": [ + { + "kind": "PanelQuery", + "spec": { + "hidden": false, + "query": { + "datasource": { + "name": "grafanacloud-projairi-prom" + }, + "group": "prometheus", + "kind": "DataQuery", + "spec": { + "expr": "sum by (source, action, status) (rate(airi_product_events_total{service_name=~\"$service\", deployment_environment=~\"$env\", feature!=\"\", action!=\"\", feature=\"tts\"}[$__rate_interval]))", + "legendFormat": "{{source}} · {{action}} · {{status}}" + }, + "version": "v0" + }, + "refId": "A" + } + } + ], + "queryOptions": {}, + "transformations": [] + } + }, + "description": "TTS product event rate by source and action. Use this to distinguish chat auto-TTS, manual previews/settings tests, and API audio.speech traffic when speech health changes.", + "id": 101, + "links": [], + "title": "TTS Event Rate by Source", + "vizConfig": { + "group": "timeseries", + "kind": "VizConfig", + "spec": { + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 15, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "smooth", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + }, + "unit": "eps" + }, + "overrides": [] + }, + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [ + "lastNotNull", + "max" + ], + "displayMode": "table", + "placement": "right", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + } + }, + "version": "13.0.0-23630096546" + } + } + }, "panel-16": { "kind": "Panel", "spec": { @@ -4217,14 +4505,40 @@ "spec": { "element": { "kind": "ElementReference", - "name": "panel-97" + "name": "panel-99" }, - "height": 9, - "width": 12, + "height": 5, + "width": 6, "x": 12, "y": 0 } }, + { + "kind": "GridLayoutItem", + "spec": { + "element": { + "kind": "ElementReference", + "name": "panel-100" + }, + "height": 5, + "width": 6, + "x": 18, + "y": 0 + } + }, + { + "kind": "GridLayoutItem", + "spec": { + "element": { + "kind": "ElementReference", + "name": "panel-97" + }, + "height": 8, + "width": 12, + "x": 0, + "y": 5 + } + }, { "kind": "GridLayoutItem", "spec": { @@ -4234,9 +4548,22 @@ }, "height": 4, "width": 12, - "x": 0, + "x": 12, "y": 5 } + }, + { + "kind": "GridLayoutItem", + "spec": { + "element": { + "kind": "ElementReference", + "name": "panel-101" + }, + "height": 4, + "width": 12, + "x": 12, + "y": 9 + } } ] } diff --git a/apps/server/otel/grafana/dashboards/build.test.ts b/apps/server/otel/grafana/dashboards/build.test.ts new file mode 100644 index 000000000..4073c0cfb --- /dev/null +++ b/apps/server/otel/grafana/dashboards/build.test.ts @@ -0,0 +1,91 @@ +import { describe, expect, it } from 'vitest' + +import { checkDashboardLayoutReferences, dashboard } from './build' + +/** + * Narrows unknown dashboard nodes into indexable records for assertions. + */ +function asRecord(value: unknown, label: string): Record { + if (!value || typeof value !== 'object') + throw new TypeError(`${label} is not an object`) + + return value as Record +} + +/** + * Reads a generated panel title from the dashboard object. + */ +function panelTitle(panelName: string): string { + const panel = asRecord(dashboard.elements[panelName], panelName) + const spec = asRecord(panel.spec, `${panelName}.spec`) + if (typeof spec.title !== 'string') + throw new TypeError(`${panelName}.spec.title is not a string`) + + return spec.title +} + +/** + * Collects PromQL expression strings from nested Grafana panel objects. + */ +function collectQueryExpressions(value: unknown, expressions: string[] = []): string[] { + if (!value || typeof value !== 'object') + return expressions + + const record = value as Record + if (typeof record.expr === 'string') + expressions.push(record.expr) + + for (const nestedValue of Object.values(record)) + collectQueryExpressions(nestedValue, expressions) + + return expressions +} + +describe('grafana dashboard builder', () => { + /** + * @example + * const result = checkDashboardLayoutReferences(dashboard) + * expect(result.orphanRefs).toEqual([]) + */ + it('keeps every generated panel connected to the row layout', () => { + const result = checkDashboardLayoutReferences(dashboard) + + expect(result.orphanRefs).toEqual([]) + expect(result.unusedElems).toEqual([]) + }) + + /** + * @example + * expect(panelTitle('panel-99')).toBe('TTS Success %') + */ + it('keeps the product analytics row focused on server-side TTS health', () => { + expect(panelTitle('panel-95')).toBe('Product Events (range)') + expect(panelTitle('panel-96')).toBe('Product Failure %') + expect(panelTitle('panel-99')).toBe('TTS Success %') + expect(panelTitle('panel-100')).toBe('TTS Failed / Blocked (range)') + expect(panelTitle('panel-101')).toBe('TTS Event Rate by Source') + }) + + /** + * @example + * const rendered = JSON.stringify(dashboard.elements['panel-101']) + * expect(rendered).not.toContain('voice_id') + */ + it('keeps high-cardinality voice fields out of Prometheus queries', () => { + const productPanelExpressions = collectQueryExpressions([ + dashboard.elements['panel-95'], + dashboard.elements['panel-96'], + dashboard.elements['panel-97'], + dashboard.elements['panel-98'], + dashboard.elements['panel-99'], + dashboard.elements['panel-100'], + dashboard.elements['panel-101'], + ]).join('\n') + + expect(productPanelExpressions).not.toContain('voice_id') + expect(productPanelExpressions).not.toContain('voice_pack_id') + expect(productPanelExpressions).not.toContain('user_id') + expect(productPanelExpressions).not.toContain('session_id') + expect(productPanelExpressions).not.toContain('request_id') + }) +}) diff --git a/apps/server/otel/grafana/dashboards/build.ts b/apps/server/otel/grafana/dashboards/build.ts index 1de48db6f..d39252e91 100644 --- a/apps/server/otel/grafana/dashboards/build.ts +++ b/apps/server/otel/grafana/dashboards/build.ts @@ -29,8 +29,8 @@ import { writeFileSync } from 'node:fs' import { dirname, join } from 'node:path' -import { exit } from 'node:process' -import { fileURLToPath } from 'node:url' +import { argv, exit } from 'node:process' +import { fileURLToPath, pathToFileURL } from 'node:url' const PROM = { name: 'grafanacloud-projairi-prom' } const LOKI = { name: 'grafanacloud-projairi-logs' } @@ -527,7 +527,7 @@ elements['panel-92'] = timeseriesPanel( { unit: 'short', fillOpacity: 30 }, ) -// --- Product Analytics — event-volume view only --------------------------- +// --- Product Analytics — event volume + server-side TTS health ------------- // Prometheus deliberately does not carry user_id. These panels answer // "which product actions are happening and failing"; DB-side product_events // queries answer "how many distinct users used each feature". @@ -577,6 +577,42 @@ elements['panel-98'] = timeseriesPanel( { unit: 'eps', fillOpacity: 15 }, ) +elements['panel-99'] = gaugePanel( + 99, + 'TTS Success %', + 'Server-side TTS successes divided by TTS requests over the dashboard range. Includes REST and WS TTS product events. Drops here mean users are asking for speech but not receiving audio; inspect failed/blocked panels next.', + [query( + `100 * sum(increase(airi_product_events_total{${PRODUCT_EVENT_FILTER}, feature="tts", action="speech_succeeded", status="succeeded"}[$__range])) / clamp_min(sum(increase(airi_product_events_total{${PRODUCT_EVENT_FILTER}, feature="tts", action="speech_requested", status="started"}[$__range])), 1)`, + 'success %', + )], + { steps: [{ color: 'red', value: 0 }, { color: 'yellow', value: 90 }, { color: 'green', value: 98 }], max: 100, decimals: 2, noValue: '0' }, +) + +elements['panel-100'] = barGaugePanel( + 100, + 'TTS Failed / Blocked (range)', + 'TTS user-impacting failures over the dashboard range, split by action/status/source. `speech_failed` usually means upstream/runtime failure; `speech_blocked` usually means balance/preflight blocked. Keep voice/model drilldown in Postgres metadata, not Prometheus labels.', + [query( + `topk(12, sum by (action, status, source) (increase(airi_product_events_total{${PRODUCT_EVENT_FILTER}, feature="tts", action=~"speech_failed|speech_blocked"}[$__range])))`, + '{{action}} · {{status}} · {{source}}', + 'A', + PROM, + { instant: true }, + )], + { unit: 'short', noValue: '0' }, +) + +elements['panel-101'] = timeseriesPanel( + 101, + 'TTS Event Rate by Source', + 'TTS product event rate by source and action. Use this to distinguish chat auto-TTS, manual previews/settings tests, and API audio.speech traffic when speech health changes.', + [query( + `sum by (source, action, status) (rate(airi_product_events_total{${PRODUCT_EVENT_FILTER}, feature="tts"}[$__rate_interval]))`, + '{{source}} · {{action}} · {{status}}', + )], + { unit: 'eps', fillOpacity: 15 }, +) + // --- Row 2: HTTP — traffic ranking, error trend, latency trend ------------- elements['panel-16'] = barGaugePanel( 16, @@ -908,14 +944,17 @@ const rows = [ item('panel-81', 8, 0, 8, 4), item('panel-82', 16, 0, 8, 4), ]), - // Row 3: Product Analytics — Prom-safe event volume and failure trend. + // Row 3: Product Analytics — Prom-safe event volume and server TTS health. // Distinct-user analytics stay in Postgres `product_events`; this row // intentionally never uses user_id/session/request labels. row('Product Analytics', [ item('panel-95', 0, 0, 6, 5), item('panel-96', 6, 0, 6, 5), - item('panel-97', 12, 0, 12, 9), - item('panel-98', 0, 5, 12, 4), + item('panel-99', 12, 0, 6, 5), + item('panel-100', 18, 0, 6, 5), + item('panel-97', 0, 5, 12, 8), + item('panel-98', 12, 5, 12, 4), + item('panel-101', 12, 9, 12, 4), ]), // Row 3: HTTP — full-width error breakdown on top, then traffic ranking + // latency trend side by side. @@ -1039,7 +1078,7 @@ const variables = [ * 1. Service Health — signup/sessions/WS counts, req-rate, 5xx, status-code * heatmap, live WS trend: "is anything broken right now?" * 2. User Engagement — rolling DAU/WAU/MAU from user.last_seen_at - * 3. Product Analytics — Prom-safe product event volume + failure trend + * 3. Product Analytics — Prom-safe product event volume + server TTS health * 4. HTTP — error breakdown by route, request ranking, latency by route * 5. LLM Gateway — per-model request rate + latency (TTFB + end-to-end) * 6. Provider Upstreams — per-provider rate/latency/failure + TTS chars @@ -1056,7 +1095,7 @@ const variables = [ * Variables source from `target_info` (always present, no business-metric * dependency) so the dashboard never goes blank when an app metric is renamed. */ -const dashboard = { +export const dashboard = { annotations: [ { kind: 'AnnotationQuery', @@ -1097,28 +1136,65 @@ const dashboard = { variables, } -const here = dirname(fileURLToPath(import.meta.url)) -const outPath = join(here, 'airi-server-overview-cloud.json') -writeFileSync(outPath, `${JSON.stringify(dashboard, null, 2)}\n`) -console.info(`wrote ${outPath}`) +export interface DashboardLayoutCheckResult { + orphanRefs: string[] + unusedElems: string[] +} -// Cross-check elements ↔ layout references -const elementNames = new Set(Object.keys(dashboard.elements)) -const refs = new Set() -function walk(o: unknown): void { - if (!o || typeof o !== 'object') +/** + * Validates that every dashboard layout reference points to a defined element. + * + * Use when: + * - Regenerating the Grafana JSON from this dashboard builder. + * - Testing that row changes did not orphan panels or leave panels unused. + * + * Expects: + * - A Grafana dashboard object shaped like {@link dashboard}. + * + * Returns: + * - Orphan layout references and unused element names. + */ +export function checkDashboardLayoutReferences(targetDashboard: typeof dashboard): DashboardLayoutCheckResult { + const elementNames = new Set(Object.keys(targetDashboard.elements)) + const refs = new Set() + collectElementReferences(targetDashboard.layout, refs) + return { + orphanRefs: [...refs].filter(r => !elementNames.has(r)), + unusedElems: [...elementNames].filter(e => !refs.has(e)), + } +} + +/** + * Recursively collects Grafana row element references from the layout tree. + */ +function collectElementReferences(node: unknown, refs: Set): void { + if (!node || typeof node !== 'object') return - const node = o as { kind?: unknown, name?: unknown } - if (node.kind === 'ElementReference' && typeof node.name === 'string') - refs.add(node.name) - for (const v of Object.values(o)) walk(v) + const layoutNode = node as { kind?: unknown, name?: unknown } + if (layoutNode.kind === 'ElementReference' && typeof layoutNode.name === 'string') + refs.add(layoutNode.name) + for (const value of Object.values(node)) collectElementReferences(value, refs) } -walk(dashboard.layout) -const orphanRefs = [...refs].filter(r => !elementNames.has(r)) -const unusedElems = [...elementNames].filter(e => !refs.has(e)) -console.info(`panels defined: ${elementNames.size}, referenced: ${refs.size}, orphans: ${orphanRefs.length}, unused: ${unusedElems.length}`) -if (orphanRefs.length || unusedElems.length) { - console.error('orphans:', orphanRefs) - console.error('unused:', unusedElems) - exit(1) + +/** + * Writes the generated dashboard JSON and fails the CLI on layout drift. + */ +function writeDashboard(): void { + const here = dirname(fileURLToPath(import.meta.url)) + const outPath = join(here, 'airi-server-overview-cloud.json') + writeFileSync(outPath, `${JSON.stringify(dashboard, null, 2)}\n`) + console.info(`wrote ${outPath}`) + + const { orphanRefs, unusedElems } = checkDashboardLayoutReferences(dashboard) + const definedCount = Object.keys(dashboard.elements).length + const referencedCount = definedCount - unusedElems.length + orphanRefs.length + console.info(`panels defined: ${definedCount}, referenced: ${referencedCount}, orphans: ${orphanRefs.length}, unused: ${unusedElems.length}`) + if (orphanRefs.length || unusedElems.length) { + console.error('orphans:', orphanRefs) + console.error('unused:', unusedElems) + exit(1) + } } + +if (import.meta.url === pathToFileURL(argv[1] ?? '').href) + writeDashboard() diff --git a/apps/server/src/app.ts b/apps/server/src/app.ts index d7fac2ac4..3e9f80362 100644 --- a/apps/server/src/app.ts +++ b/apps/server/src/app.ts @@ -4,6 +4,7 @@ import type { AuthInstance } from './libs/auth' import type { Database } from './libs/db' import type { Env } from './libs/env' import type { OtelInstance } from './otel' +import type { StreamingTtsVoiceType } from './routes/audio-speech-ws/session' import type { ConfigKVService } from './services/adapters/config-kv' import type { AdminFluxGrantsService } from './services/domain/admin/flux-grants' import type { AdminRouterConfigService } from './services/domain/admin/router-config' @@ -213,6 +214,7 @@ export async function buildApp(deps: AppDeps) { return audioSpeechWsSetup(session.user.id, { trigger: c.req.query('tts_trigger') === 'auto' ? 'auto' : 'manual', source: parseTtsSource(c.req.query('tts_source'), 'audio.speech.ws'), + voiceType: parseTtsVoiceType(c.req.query('tts_voice_type')), }) })) @@ -458,6 +460,23 @@ function parseTtsSource( } } +/** + * Normalizes the client-provided streaming TTS voice bucket for product events. + */ +function parseTtsVoiceType( + value: string | undefined, +): StreamingTtsVoiceType { + switch (value) { + case 'official_default': + case 'official_selected': + case 'custom_configured': + case 'voice_pack': + return value + default: + return 'unknown' + } +} + export type AppType = Awaited>['app'] export async function createApp() { diff --git a/apps/server/src/routes/audio-speech-ws/route.test.ts b/apps/server/src/routes/audio-speech-ws/route.test.ts index 44235d7f6..855467c26 100644 --- a/apps/server/src/routes/audio-speech-ws/route.test.ts +++ b/apps/server/src/routes/audio-speech-ws/route.test.ts @@ -230,7 +230,7 @@ describe('audio-speech-ws route', () => { const deps = makeFakeDeps({ upstreamURL: upstream.url, fluxBalance: 100 }) const handlers = createAudioSpeechWsHandlers(deps as any) - const events = handlers('user-123') + const events = handlers('user-123', { voiceType: 'official_selected' }) const client = makeMockClientWs() await driveClientSession(events, client, [ @@ -279,6 +279,17 @@ describe('audio-speech-ws route', () => { status: 200, fluxConsumed: 1, }) + expect(deps.productEventService.track).toHaveBeenCalledWith(expect.objectContaining({ + userId: 'user-123', + feature: 'tts', + action: 'speech_succeeded', + status: 'succeeded', + model: 'volcengine/seed-tts-2.0', + metadata: expect.objectContaining({ + voice_id: 'mock', + voice_type: 'official_selected', + }), + })) }) it('refuses the session with insufficient_flux when the user is broke', async () => { diff --git a/apps/server/src/routes/audio-speech-ws/session.ts b/apps/server/src/routes/audio-speech-ws/session.ts index 393998159..83f45b43d 100644 --- a/apps/server/src/routes/audio-speech-ws/session.ts +++ b/apps/server/src/routes/audio-speech-ws/session.ts @@ -52,10 +52,12 @@ export interface AudioSpeechSessionState { export type StreamingTtsTrigger = 'auto' | 'manual' export type StreamingTtsSource = 'audio.speech.ws' | 'chat_auto_tts' | 'manual_preview' | 'settings_test' +export type StreamingTtsVoiceType = 'official_default' | 'official_selected' | 'custom_configured' | 'voice_pack' | 'unknown' export interface AudioSpeechSessionAnalytics { trigger?: StreamingTtsTrigger source?: StreamingTtsSource + voiceType?: StreamingTtsVoiceType } /** @@ -93,6 +95,7 @@ export function createSessionState( let billed = false let totalInputChars = 0 let modelLabel = STREAM_MODEL_LABEL_FALLBACK + let voiceLabel: string | undefined /** * Frames the client sent before the upstream finished dialing. Buffered to * avoid silently dropping the `start` frame; flushed in arrival order once @@ -114,6 +117,7 @@ export function createSessionState( model: modelLabel, metadata: { trigger: analytics.trigger, + ...streamingVoiceMetadata(voiceLabel, analytics.voiceType), }, }) @@ -227,6 +231,7 @@ export function createSessionState( metadata: { duration_ms: Date.now() - startedAt, trigger: analytics.trigger, + ...streamingVoiceMetadata(voiceLabel, analytics.voiceType), }, }) try { @@ -361,6 +366,9 @@ export function createSessionState( const model = (parsed as Record).model if (typeof model === 'string' && model.length > 0) modelLabel = model + const voice = (parsed as Record).voice + if (typeof voice === 'string' && voice.length > 0) + voiceLabel = voice } } catch { @@ -433,6 +441,7 @@ export function createSessionState( duration_ms: durationMs, flux_consumed: fluxConsumed, trigger: analytics.trigger, + ...streamingVoiceMetadata(voiceLabel, analytics.voiceType), }, }) @@ -470,6 +479,7 @@ export function createSessionState( close_code: code, duration_ms: Date.now() - startedAt, trigger: analytics.trigger, + ...streamingVoiceMetadata(voiceLabel, analytics.voiceType), }, }) if (clientWs) { @@ -503,6 +513,7 @@ export function createSessionState( close_code: code, duration_ms: Date.now() - startedAt, trigger: analytics.trigger, + ...streamingVoiceMetadata(voiceLabel, analytics.voiceType), }, }) if (clientWs) { @@ -531,6 +542,7 @@ function normalizeAnalytics(input: AudioSpeechSessionAnalytics): Required { + return { + ...(voiceId ? { voice_id: voiceId } : {}), + voice_type: voiceType, + } +} + function isPaymentRequiredError(err: unknown): boolean { if (err instanceof ApiError) return err.statusCode === 402 diff --git a/apps/server/src/routes/openai/v1/route.test.ts b/apps/server/src/routes/openai/v1/route.test.ts index ff115c2e3..24e5b732a 100644 --- a/apps/server/src/routes/openai/v1/route.test.ts +++ b/apps/server/src/routes/openai/v1/route.test.ts @@ -849,6 +849,87 @@ describe('v1CompletionsRoutes', () => { })) }) + /** + * @example + * POST /api/v1/audio/speech { "voice": "alloy", "extra_body": { "voice_pack": { "pack_id": "vp-premium" } } } + */ + it('records TTS voice and Voice Pack metadata in product events', async () => { + globalThis.fetch = vi.fn(async () => new Response(new Uint8Array([1]), { + status: 200, + headers: { 'Content-Type': 'audio/mpeg' }, + })) + + const productEventService = createMockProductEventService() + const voicePackService = createMockVoicePackService({ + findById: vi.fn(async () => ({ + id: 'vp-premium', + name: 'Premium', + description: null, + provider: 'azure', + model: 'microsoft/v1', + voiceId: 'alloy', + ttsModelId: 'tts-1', + params: {}, + costMultiplier: 2, + enabled: true, + createdAt: new Date(), + updatedAt: new Date(), + })), + }) + const app = createTestApp( + createMockFluxService(), + createMockConfigKV(), + undefined, + undefined, + undefined, + undefined, + createMockLlmTracing(), + productEventService, + voicePackService, + ) + + await app.fetch( + new Request('http://localhost/api/v1/audio/speech', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ + model: 'auto', + input: 'hello', + voice: 'alloy', + extra_body: { + voice_pack: { + pack_id: 'vp-premium', + cost_multiplier: 2, + }, + airi_analytics: { + source: 'manual_preview', + voice_type: 'voice_pack', + }, + }, + }), + }), + { user: testUser } as any, + ) + + expect(productEventService.track).toHaveBeenCalledWith(expect.objectContaining({ + action: 'speech_succeeded', + source: 'manual_preview', + metadata: expect.objectContaining({ + voice_id: 'alloy', + voice_type: 'voice_pack', + voice_pack_id: 'vp-premium', + }), + })) + expect(productEventService.track).toHaveBeenCalledWith(expect.objectContaining({ + action: 'speech_requested', + metadata: expect.objectContaining({ + voice_id: 'alloy', + voice_type: 'voice_pack', + voice_pack_id: 'vp-premium', + }), + })) + }) + it('should not charge when routeTts upstream returns error', async () => { const llmRouter = createMockLlmRouter({ routeTts: vi.fn(async () => new Response('{"error":"service down"}', { diff --git a/apps/server/src/services/domain/openai-speech/index.ts b/apps/server/src/services/domain/openai-speech/index.ts index 6e3c7cdf4..139eed93c 100644 --- a/apps/server/src/services/domain/openai-speech/index.ts +++ b/apps/server/src/services/domain/openai-speech/index.ts @@ -63,10 +63,12 @@ export interface OpenAiSpeechRequest { } type TtsTrigger = 'auto' | 'manual' +type TtsVoiceType = 'official_default' | 'official_selected' | 'custom_configured' | 'voice_pack' | 'unknown' interface TtsAnalyticsContext { trigger: TtsTrigger source: 'audio.speech' | 'chat_auto_tts' | 'manual_preview' | 'settings_test' + voiceType: TtsVoiceType } /** @@ -100,6 +102,11 @@ export function createOpenAiSpeechService(deps: OpenAiSpeechServiceDeps) { voice: typeof input.body.voice === 'string' ? input.body.voice : undefined, voicePackService: deps.voicePackService, }) + const voiceMetadata = ttsVoiceMetadata({ + voice: typeof input.body.voice === 'string' ? input.body.voice : undefined, + voicePackId: voicePackRequest.voicePackId, + voiceType: analytics.voiceType, + }) const billingUnits = Math.ceil(inputText.length * voicePackRequest.costMultiplier) logger.withFields({ @@ -120,6 +127,7 @@ export function createOpenAiSpeechService(deps: OpenAiSpeechServiceDeps) { metadata: { input_chars: inputText.length, trigger: analytics.trigger, + ...voiceMetadata, }, }) @@ -144,6 +152,7 @@ export function createOpenAiSpeechService(deps: OpenAiSpeechServiceDeps) { billing_units: billingUnits, balance_state: 'insufficient', trigger: analytics.trigger, + ...voiceMetadata, }, }) logger.withError(err).withFields({ @@ -219,6 +228,7 @@ export function createOpenAiSpeechService(deps: OpenAiSpeechServiceDeps) { http_status: failure.status, duration_ms: Date.now() - startedAt, trigger: analytics.trigger, + ...voiceMetadata, }, }) throw err @@ -245,6 +255,7 @@ export function createOpenAiSpeechService(deps: OpenAiSpeechServiceDeps) { http_status: response.status, duration_ms: durationMs, trigger: analytics.trigger, + ...voiceMetadata, }, }) logger.withFields({ requestId, userId: input.userId, model: requestModel, status: response.status, durationMs }) @@ -297,6 +308,7 @@ export function createOpenAiSpeechService(deps: OpenAiSpeechServiceDeps) { duration_ms: durationMs, flux_consumed: fluxConsumed, trigger: analytics.trigger, + ...voiceMetadata, }, }) deps.requestLogService.logRequest({ @@ -354,8 +366,40 @@ function ttsAnalyticsContext(body: Record): TtsAnalyticsContext || rawSource === 'settings_test' ? rawSource : 'audio.speech' + const voiceType = normalizeVoiceType(analytics?.voice_type) - return { trigger, source } + return { trigger, source, voiceType } +} + +/** + * Normalizes client-provided TTS voice type into bounded analytics values. + */ +function normalizeVoiceType(value: unknown): TtsVoiceType { + switch (value) { + case 'official_default': + case 'official_selected': + case 'custom_configured': + case 'voice_pack': + return value + default: + return 'unknown' + } +} + +/** + * Builds reusable low-cardinality voice metadata for every TTS product event. + */ +function ttsVoiceMetadata(input: { + voice?: string + voicePackId?: string + voiceType: TtsVoiceType +}): Record { + const voiceType = input.voicePackId ? 'voice_pack' : input.voiceType + return { + ...(input.voice ? { voice_id: input.voice } : {}), + voice_type: voiceType, + ...(input.voicePackId ? { voice_pack_id: input.voicePackId } : {}), + } } async function voicePackRequestOptions( @@ -365,12 +409,15 @@ async function voicePackRequestOptions( voice?: string voicePackService: VoicePackService }, -): Promise<{ extraOptions: Record | undefined, costMultiplier: number }> { +): Promise<{ extraOptions: Record | undefined, costMultiplier: number, voicePackId?: string }> { const extraBody = asRecord(body.extra_body) const voicePackOptions = asRecord(extraBody?.voice_pack) const pitch = readOptionalNumber(voicePackOptions, 'pitch') const volume = readOptionalNumber(voicePackOptions, 'volume') const costMultiplier = await resolveVoicePackCostMultiplier(voicePackOptions, context) + const voicePackId = typeof voicePackOptions?.pack_id === 'string' && voicePackOptions.pack_id.trim() + ? voicePackOptions.pack_id + : undefined const extraOptions: Record = {} if (pitch != null) extraOptions.pitch = pitch @@ -380,6 +427,7 @@ async function voicePackRequestOptions( return { extraOptions: Object.keys(extraOptions).length > 0 ? extraOptions : undefined, costMultiplier, + voicePackId, } } diff --git a/packages/core-agent/src/runtime/chat-orchestrator-runtime.test.ts b/packages/core-agent/src/runtime/chat-orchestrator-runtime.test.ts index 7c12d4ab6..1f2d90d9d 100644 --- a/packages/core-agent/src/runtime/chat-orchestrator-runtime.test.ts +++ b/packages/core-agent/src/runtime/chat-orchestrator-runtime.test.ts @@ -35,6 +35,9 @@ function createHarness() { const assistantTurns: unknown[] = [] const stateChanges: unknown[] = [] const telemetry = { + chatActivationStarted: [] as unknown[], + chatActivationSucceeded: [] as unknown[], + chatActivationFailed: [] as unknown[], messageSendStarted: [] as unknown[], llmRequestStarted: [] as unknown[], llmFirstToken: [] as unknown[], @@ -89,6 +92,9 @@ function createHarness() { onUserTurnReady: event => userTurns.push(event), onAssistantTurnReady: event => assistantTurns.push(event), onStateChange: state => stateChanges.push(state), + onChatActivationStarted: event => telemetry.chatActivationStarted.push(event), + onChatActivationSucceeded: event => telemetry.chatActivationSucceeded.push(event), + onChatActivationFailed: event => telemetry.chatActivationFailed.push(event), onMessageSendStarted: event => telemetry.messageSendStarted.push(event), onLlmRequestStarted: event => telemetry.llmRequestStarted.push(event), onLlmFirstToken: event => telemetry.llmFirstToken.push(event), @@ -323,6 +329,46 @@ describe('createChatOrchestratorRuntime', () => { hasVoice: true, model: 'gpt-test', }]) + expect(harness.telemetry.chatActivationStarted).toEqual([{ + model: 'gpt-test', + provider: 'mock-provider', + source: 'voice', + }]) + expect(harness.telemetry.chatActivationSucceeded).toEqual([{ + durationMs: 360, + model: 'gpt-test', + provider: 'mock-provider', + source: 'voice', + }]) + expect(harness.telemetry.chatActivationFailed).toEqual([]) + }) + + /** + * @example + * await expect(runtime.ingest('hello', { model, chatProvider })).rejects.toThrow('provider rejected') + */ + it('emits chat activation failure telemetry without raw provider messages', async () => { + const harness = createHarness() + harness.stream.mockRejectedValueOnce(new Error('provider rejected with sensitive details')) + + await expect(harness.runtime.ingest('hello', { + model: 'gpt-test', + chatProvider: provider, + })).rejects.toThrow('provider rejected') + + expect(harness.telemetry.chatActivationStarted).toEqual([{ + model: 'gpt-test', + provider: 'mock-provider', + source: 'text', + }]) + expect(harness.telemetry.chatActivationSucceeded).toEqual([]) + expect(harness.telemetry.chatActivationFailed).toEqual([{ + errorCode: 'llm_response_failed', + failureStage: 'llm_response', + model: 'gpt-test', + provider: 'mock-provider', + source: 'text', + }]) }) /** diff --git a/packages/core-agent/src/runtime/chat-orchestrator-runtime.ts b/packages/core-agent/src/runtime/chat-orchestrator-runtime.ts index ed164338f..239106db3 100644 --- a/packages/core-agent/src/runtime/chat-orchestrator-runtime.ts +++ b/packages/core-agent/src/runtime/chat-orchestrator-runtime.ts @@ -190,6 +190,27 @@ export interface ChatOrchestratorRuntimeDeps { onSendSettled?: (event: { sessionId: string }) => void /** Called when a send starts and the first assistant placeholder is created. */ onTrackFirstMessage?: () => void + /** Called when a user starts a chat activation attempt. */ + onChatActivationStarted?: (event: { + source: 'text' | 'voice' + model: string + provider: string + }) => void + /** Called after one user-to-assistant message round completes successfully. */ + onChatActivationSucceeded?: (event: { + source: 'text' | 'voice' + model: string + provider: string + durationMs: number + }) => void + /** Called after a chat activation attempt fails before assistant completion. */ + onChatActivationFailed?: (event: { + source: 'text' | 'voice' + model: string + provider: string + failureStage: 'llm_response' + errorCode: 'llm_response_failed' + }) => void /** Called when a user message send begins. */ onMessageSendStarted?: (event: { source: 'text' | 'voice' @@ -401,9 +422,16 @@ export function createChatOrchestratorRuntime(deps: ChatOrchestratorRuntimeDeps) id: createId(), } patchForegroundStream(sessionId, buildingMessage) + const sendSource = options.input ? 'voice' : 'text' + const activeProvider = deps.getActiveProvider?.() ?? '' deps.onTrackFirstMessage?.() + deps.onChatActivationStarted?.({ + source: sendSource, + model: options.model, + provider: activeProvider, + }) deps.onMessageSendStarted?.({ - source: options.input ? 'voice' : 'text', + source: sendSource, model: options.model, }) const roundStartedAt = monotonicNow() @@ -710,14 +738,28 @@ export function createChatOrchestratorRuntime(deps: ChatOrchestratorRuntimeDeps) }) resetForegroundStream(sessionId) + const durationMs = Math.round(monotonicNow() - roundStartedAt) deps.onMessageRound?.({ - durationMs: Math.round(monotonicNow() - roundStartedAt), + durationMs, hasVoice: !!options.input, model: options.model, }) + deps.onChatActivationSucceeded?.({ + durationMs, + source: sendSource, + model: options.model, + provider: activeProvider, + }) } catch (error) { console.error('Error sending message:', error) + deps.onChatActivationFailed?.({ + source: sendSource, + model: options.model, + provider: activeProvider, + failureStage: 'llm_response', + errorCode: 'llm_response_failed', + }) throw error } finally { diff --git a/packages/stage-pages/src/pages/settings/modules/speech.vue b/packages/stage-pages/src/pages/settings/modules/speech.vue index ef179f9b9..da9142e1e 100644 --- a/packages/stage-pages/src/pages/settings/modules/speech.vue +++ b/packages/stage-pages/src/pages/settings/modules/speech.vue @@ -1,4 +1,5 @@