diff --git a/apps/server/docs/ai-context/metrics-ownership.md b/apps/server/docs/ai-context/metrics-ownership.md index 9ef5f0424..776701aee 100644 --- a/apps/server/docs/ai-context/metrics-ownership.md +++ b/apps/server/docs/ai-context/metrics-ownership.md @@ -50,6 +50,14 @@ | LLM token / cost | **Grafana**（短期） | 后续若引入 Langfuse 则迁过去 | | 用户行为漏斗各步骤 | **PostHog**（必须） | 第一步通常是前端事件，Grafana 拿不到 | +### Better Auth session table 与活跃用户 + +`user_active_sessions`（COUNT(\*)）和 `user_distinct_active`（COUNT(DISTINCT user_id)）共享同一张 `session` 表： + +- **Better Auth 每次 sign-in / 每次 OIDC access-token 颁发都新建一条 session row，从不主动 GC 过期 row**——因为 `oauth_access_token.session_id` FK 指向 session（`apps/server/src/libs/auth.ts:513` 注释） +- 实战观察：~80K `user_active_sessions` 对应实际只有几百 distinct user。比例 5+ 就该考虑加 session GC cron 或缩短 Better Auth `expiresIn` +- 永远展示 `user_distinct_active` 给非工程师看（PM、运营）；`user_active_sessions` 留给工程师 debug + ### Dashboard 标注规则 两边都展示的指标，**必须**在 Grafana panel description 和 PostHog insight description 里： @@ -95,7 +103,7 @@ | WS | `ws_connections_active` / `ws_messages_*_total` | Grafana | | | LLM | `gen_ai_client_operation_count_total` / `gen_ai_client_first_token_duration_seconds` | Grafana | | | Billing | `airi_billing_flux_unbilled_total` | Grafana | **告警必须**：`increase(airi_billing_flux_unbilled_total[5m]) > 0` | -| Auth | `user_active_sessions` | Postgres → Grafana 派生 | 集群级 gauge，用 `avg()` 不要 `sum()` | +| Auth | `user_active_sessions` / `user_distinct_active` | Postgres → Grafana 派生 | 集群级 gauge，用 `avg()` 不要 `sum()`。两个一起看：`user_active_sessions` = `COUNT(*)`（session row 数，会膨胀）, `user_distinct_active` = `COUNT(DISTINCT user_id)`（真实活跃用户数）| | Stripe | `airi_stripe_revenue_minor_unit_total` / `stripe_events_total` | Postgres → 两边展示 | Grafana 是系统侧 webhook 计数 | | Runtime | `v8js_memory_*` / `nodejs_eventloop_delay_*` | Grafana | per `service_instance_id` | | Rate-limit | `airi_rate_limit_blocked_total` | Grafana | in-memory per replica | @@ -217,6 +225,30 @@ PostHog UI 配 cohort： 不能只用 source connector：它是 data warehouse 层，**不生成 person event，做不了漏斗**。 +## 5xx Triage 路径 + +Dashboard 上 follow 这条 panel 链可以从"出事了"一路 drill 到"哪个 trace 是真凶"： + +1. **panel-4 `5xx Rate %`**（Row 1）— 数字 / gauge 颜色变红，说明出事 +2. **panel-9 `Top Routes by 5xx`**（Row 2 donut）— "现在哪些 route 在失败" +3. **panel-44 `5xx Rate by Route`**（Row 5.5 timeseries）— "什么时候开始的、是单点还是普遍" +4. **panel-91 `5xx Error Logs`**（Row 8 上半）— 实际错误消息，里面有 `trace_id` field 可点 → Tempo 看完整 trace 回放 + +### Tempo / Loki derived fields 配置（一次性） + +panel-91 的 `trace_id` 字段必须配 Grafana Cloud Loki datasource 的 **Derived fields** 才能跳 Tempo。这不在 dashboard JSON 范围内，是 datasource 级配置： + +- **Grafana Cloud** → Connections → Data sources → 选 `grafanacloud-projairi-logs`（Loki）→ Derived fields +- 添加： + - **Name**: `trace_id` + - **Type**: Regex in label or value + - **Regex**: `"trace_id":"([a-f0-9]+)"`（匹配我们 logger 的 JSON 输出） + - **URL**: 留空 + - **Internal link**: ✓，datasource 选 `grafanacloud-projairi-traces`（Tempo） +- 同样手法可加 `req` (request id) → 配 internal link 回 Loki 自身，按 requestId filter + +配置完之后日志面板里 `trace_id` 会变成蓝色可点，直接跳 Tempo waterfall。这一步配置只做一次，新加 panel 自动享有。 + ## Grafana Alert SOP Alert rules **不放在** `apps/server/otel/grafana/dashboards/build.ts` 里——Grafana Cloud 用 Unified Alerting，rule 在 Grafana UI 或 alerting API 管理，跟 dashboard JSON 解耦。这一节维护我们应该配的 alert rule，新加 rule 时同步更新这里。 diff --git a/apps/server/docs/ai-context/verifications/flux-unbilled-exploit-fix.md b/apps/server/docs/ai-context/verifications/flux-unbilled-exploit-fix.md new file mode 100644 index 000000000..e082a2650 --- /dev/null +++ b/apps/server/docs/ai-context/verifications/flux-unbilled-exploit-fix.md @@ -0,0 +1,127 @@ +# Verification: Flux Unbilled Exploit Fix + +Status: **patched in commit `7267b0d6b`** (2026-05-15) for the chat-completion path. TTS flux-meter adaptation followed up in the same PR as this verification doc (see "Remaining gaps → Gap 1"). +Last attempted: 2026-05-15 +Owner: rbxin2003@gmail.com + +## 用户路径 + +- **场景**：用户 `0 < balance < fallbackRate`，开 N 个并发 LLM completion 请求 +- **预期（修补后）**：第一个 request 之后 pre-flight 拒绝；最多触发一次 partial debit（`charged < requested`），所有剩余 request 收 402 +- **实际（修补前）**：N 个 request 全部走到 stream end，每个 catch `consumeFluxForLLM` 失败回滚 → balance 不变 → 用户拿到 N 次免费 LLM 响应 + +## Before / After 行为 + +### Before（修补前漏洞链） + +1. Pre-flight 仅 `if (flux.flux <= 0)`（`apps/server/src/routes/openai/v1/index.ts`，旧版） +2. 用户余额 1 flux，但 `FLUX_PER_REQUEST` (fallback) 通常远大于 1 +3. 并发 N 请求全部通过 pre-flight，上游 LLM 全部完成（响应已 stream 出去） +4. `consumeFluxForLLM` 走 `debitFlux` (`apps/server/src/services/billing/billing-service.ts:107`)，旧逻辑： + ```ts + if (balanceBefore < input.amount) { + throw createPaymentRequiredError('Insufficient flux') + } + ``` + 整个 DB tx 回滚 → balance 维持 1 flux +5. catch 路径上报 `airi_billing_flux_unbilled_total{reason='debit_failed'}` += full amount +6. 用户重复同样的脚本，每次都拿免费响应 + +这是 Grafana panel-43 (Flux Unbilled) 累积到 **70.2K** 的根因。 + +### After（commit `7267b0d6b`） + +**1. Pre-flight 阈值改为 fallbackRate**（`apps/server/src/routes/openai/v1/index.ts:138-142`）： + +```ts +const fallbackRate = await configKV.getOrThrow('FLUX_PER_REQUEST') +const fluxPer1kTokens = await configKV.get('FLUX_PER_1K_TOKENS') + +const flux = await fluxService.getFlux(user.id) +if (flux.flux < fallbackRate) { + throw createPaymentRequiredError('Insufficient flux') +} +``` + +并发 N 请求时，pre-flight 直接 402 拒绝，不会进入上游 LLM 调用。 + +**2. Partial-debit 语义**（`apps/server/src/services/billing/billing-service.ts:107-130`）： + +```ts +if (balanceBefore <= 0) { + metrics?.fluxInsufficientBalance.add(1) + throw createPaymentRequiredError('Insufficient flux') +} + +const chargedAmount = Math.min(input.amount, balanceBefore) +const balanceAfter = balanceBefore - chargedAmount +const isPartial = chargedAmount < input.amount +``` + +- `balance > 0` 但不够 → drain 到 0，ledger 记 `amount = charged`，metadata 带 `requestedAmount + unbilled` +- `balance <= 0` 才 throw（catch path 上报 `reason='debit_failed'`） +- Partial drain 上报 `reason='partial_debit_drained'`，跟真实 DB 错误区分 + +**3. Idempotency 反映原始 charged**（`billing-service.ts:71-94`）： + +替换 request 的同 requestId 重放复用历史 `existing.amount` 作为 `charged`，避免重试时双扣 unbilled counter。 + +## Evidence + +| Item | Reference | +|---|---| +| Fix commit | `7267b0d6b feat(server/billing): partial-debit semantics to prevent unpaid usage exploit` | +| Pre-flight gate | `apps/server/src/routes/openai/v1/index.ts:138-142` | +| Partial-debit logic | `apps/server/src/services/billing/billing-service.ts:107-148` | +| Streaming path unbilled metric | `apps/server/src/routes/openai/v1/index.ts:299-313` (label `reason='partial_debit_drained'`, `stage='streaming'`) | +| Non-streaming path unbilled metric | `apps/server/src/routes/openai/v1/index.ts:374-388` (label `stage='non_streaming'`) | +| Regression tests added | `apps/server/src/routes/openai/v1/route.test.ts` (+97 lines), `apps/server/src/services/billing/tests/billing-service.test.ts` (+89 lines) | + +测试覆盖的两个核心 case： + +- `'rejects pre-flight when balance is below FLUX_PER_REQUEST (Issue: unpaid-usage-exploit)'` — 验证 pre-flight 在 partial-balance 用户上 reject，upstream 未被调用 +- `'non-streaming completion drains partial balance and logs charged (Issue: unpaid-usage-exploit)'` — 验证 partial-debit drain 到 zero + metric 上报 + +## Remaining gaps + +### Gap 1 — TTS flux-meter partial-debit 适配 ✅ 已修复 + +`apps/server/src/services/billing/flux-meter.ts:135-228` 的 `accumulate()` 现在解构 `{ charged, requested }`，partial drain 时： + +- 上报 `airi_billing_flux_unbilled_total{source='tts_meter', reason='partial_debit_drained', meter, gen_ai.request.model?}` +- `INCRBY` `(requested - charged) * unitsPerFlux` 回 Redis debt counter +- `AccumulateResult` 增加 `unbilledFlux` 字段供调用方观测 +- 加 invariant 校验 `charged > requested` / 非整数 / 负数 → throw 而不是静默 under-restore + +测试覆盖：`flux-meter.test.ts:217-260`，case 名带 `Issue: unpaid-usage-exploit follow-up`。 + +**残余风险（已知，留 follow-up）**：settlement (LUA `runScript`) 和 `INCRBY` restore 之间非原子，并发请求理论上可能在窗口内读到 mid-state。窗口很小（一个 DB tx），实际命中很难触发；长期修复需要 Redis lock 或 unbilled 单独 bucket。代码里 `flux-meter.ts` 有 `// REVIEW:` 标记。 + +### Gap 2 — 修补前的 70.2K 历史漏账未核销 + +panel-43 显示的 70.2K 是 counter 累积值（`increase($__range)`），代表修补前漏出去的总量。修补**不会让 panel 自动归零**，只会让新的增量趋近 0（除真实 DB 错误）。 + +**建议**： +- 把 dashboard 时间窗调到 commit `7267b0d6b` 部署后（2026-05-15 之后）观察增量斜率 +- 加 Grafana alert：`increase(airi_billing_flux_unbilled_total[5m]) > 0` → PagerDuty +- 如果业务需要核销历史 70.2K，从 `flux_transaction` ledger 反查 `metadata->>'reason' = 'debit_failed'` 的记录，配合 Loki 错误日志定位涉事 userId + +## What's verified + +- ✓ 代码层：pre-flight gate + partial-debit semantics 确实改了，逻辑正确 +- ✓ 测试层：两个核心 regression test 覆盖 exploit 场景，case 名带 `Issue: unpaid-usage-exploit` 标识 + +## What's pending live verification + +- ⊘ 生产 Grafana 上 panel-43 在 `2026-05-15` commit 部署后的斜率趋近 0 +- ⊘ 没跑 `pnpm -F @proj-airi/server exec vitest run` 实际确认新测试 pass（建议在 push 之前跑一次） +- ⊘ TTS partial-debit 适配的 live verification（代码已修补 + 单测 14/14 pass，需要在生产观察 `airi_billing_flux_unbilled_total{source='tts_meter'}` 出现合理流量后再 close） + +## Recommended follow-ups + +按优先级： + +1. **P0 — 加 Grafana alert** `increase(airi_billing_flux_unbilled_total[5m]) > 0` → 通知 +2. **P1 — 历史 70.2K 漏账处理**：查 ledger + Loki 决定核销还是补账 +3. ~~**P1 — 修 TTS flux-meter 适配 partial-debit 新语义**~~ ✅ 已修复，见 Gap 1 +4. **P2 — Dashboard 改造**（panel-43 时间窗注解 + 区分 `reason` label 的 stack 图，把 `partial_debit_drained` 跟 `debit_failed` 分色展示） diff --git a/apps/server/docs/ai-context/verifications/flux-unbilled-reconciliation.md b/apps/server/docs/ai-context/verifications/flux-unbilled-reconciliation.md new file mode 100644 index 000000000..e5fdc1e82 --- /dev/null +++ b/apps/server/docs/ai-context/verifications/flux-unbilled-reconciliation.md @@ -0,0 +1,195 @@ +# Verification: Flux Unbilled Historical Reconciliation + +Status: **investigation framework — data gathering pending** +Owner: rbxin2003@gmail.com +Last updated: 2026-05-15 +Related: [`flux-unbilled-exploit-fix.md`](./flux-unbilled-exploit-fix.md), [Grafana panel-43](../../../otel/grafana/dashboards/airi-server-overview-cloud.json) + +## 用户路径 + +- **场景**：commit `7267b0d6b` 之前累积了 ~70.2K Flux 的 unpaid usage（panel-43 `airi_billing_flux_unbilled_total` 显示值）。需要决定核销、补账、还是不处理 +- **预期**：跑下面的 SQL + Loki query → 区分 partial-drain（用户已部分付款）vs debit-failed（DB 错误，真零付款）→ 按 user 聚合 → 给出 reconciliation 决策 +- **当前状态**：没有 prod DB 访问权限的工程师跑下面的 query。下方 SQL/queries 是**待执行的模板**，不是已采集的数据 + +## 两类漏账的区分 + +修补前 `airi_billing_flux_unbilled_total` 是单一 counter，没区分 reason。修补后（`7267b0d6b`）按 `reason` 拆成两个 label： + +| reason label | 触发条件 | Ledger 是否有记录 | 用户实际付款比例 | +|---|---|---|---| +| `partial_debit_drained` | `0 < balance < amount`，drain 到 0 | ✓ 有（`amount = charged`，metadata 带 `unbilled`） | 部分付款（drain 数额） | +| `debit_failed` | `balance <= 0` 或 DB tx 抛错 | ✗ 无（tx 回滚） | 零付款 | + +**70.2K 全部发生在 5/15 之前**，那时所有失败都走 catch path → 全部记为 `reason='debit_failed'`，**全部无 ledger row** → 用户实际付款为 0。 + +但实际不全是漏洞：少部分是真正的 DB 错误（DB outage / 唯一索引冲突）。绝大部分是 exploit。 + +## 取证 SQL（待执行） + +> 在 Railway Postgres console 或本地 `psql $DATABASE_URL` 跑。如果 query 太重，先 `EXPLAIN ANALYZE` 看 cost；`flux_transaction` 有 `flux_tx_user_id_idx` 和 `flux_tx_created_at_idx` 索引可以走 + +### 1. 按 type 分类的 ledger 写入分布 + +`reason='debit_failed'` 没 ledger row，所以这条 query **拿不到**修补前的漏洞数据——它只能 sanity check 修补后的新 row： + +```sql +SELECT + type, + COUNT(*) AS row_count, + SUM(amount) AS total_amount, + MIN(created_at) AS first_seen, + MAX(created_at) AS last_seen +FROM flux_transaction +WHERE created_at >= '2026-04-15' -- 4 周窗口 +GROUP BY type +ORDER BY total_amount DESC; +``` + +### 2. Partial-drain ledger rows（修补后） + +`commit 7267b0d6b` 之后才会有这种 row。修补前漏出去的 70K 在这里**看不到**： + +```sql +SELECT + user_id, + COUNT(*) AS partial_debit_count, + SUM(amount) AS total_charged, + SUM((metadata->>'unbilled')::bigint) AS total_unbilled, + SUM((metadata->>'requestedAmount')::bigint) AS total_requested, + MIN(created_at) AS first_partial, + MAX(created_at) AS last_partial +FROM flux_transaction +WHERE type = 'debit' + AND metadata ? 'unbilled' + AND (metadata->>'unbilled')::bigint > 0 + AND created_at >= '2026-05-15' -- 修补后窗口 +GROUP BY user_id +ORDER BY total_unbilled DESC +LIMIT 50; +``` + +### 3. 流量最高的用户（用来定位 exploit 嫌疑） + +修补前的漏账主要靠这条 + Loki 日志交叉定位涉事 user： + +```sql +SELECT + user_id, + COUNT(*) AS debit_count, + SUM(amount) AS total_debited, + SUM(balance_after - balance_before) AS net_balance_change, + MIN(created_at) AS first_debit, + MAX(created_at) AS last_debit +FROM flux_transaction +WHERE type = 'debit' + AND created_at BETWEEN '2026-05-01' AND '2026-05-15' -- 修补前 2 周 +GROUP BY user_id +HAVING COUNT(*) > 100 +ORDER BY debit_count DESC +LIMIT 20; +``` + +异常用户特征：`debit_count` 极高 + `net_balance_change` 接近 0 即"balance 一直被推到底但没归零"。这种是 exploit 的核心 signature——攻击者维持 balance 卡在 `0 < x < fallbackRate` 区间反复触发免费请求。注意：`net_balance_change` 在 ledger 模型下应该等于 `-SUM(amount)`；如果两者接近 0 但 `SUM(amount)` 很大，说明 balance 被人工补回去过（信用 / 充值 / promo），需要进一步交叉检查。 + +### 4. 当前 user_flux 余额 vs ledger 一致性 sanity + +```sql +WITH ledger_balance AS ( + SELECT + user_id, + SUM(CASE WHEN type = 'credit' OR type = 'initial' OR type = 'promo' THEN amount + WHEN type = 'debit' THEN -amount + ELSE 0 + END) AS computed_balance + FROM flux_transaction + GROUP BY user_id +) +SELECT + uf.user_id, + uf.flux AS recorded_balance, + lb.computed_balance AS ledger_sum, + uf.flux - lb.computed_balance AS drift +FROM user_flux uf +LEFT JOIN ledger_balance lb USING (user_id) +WHERE ABS(uf.flux - COALESCE(lb.computed_balance, 0)) > 0 +ORDER BY ABS(uf.flux - COALESCE(lb.computed_balance, 0)) DESC +LIMIT 50; +``` + +正常情况下 drift 应该是 0——任何 drift 都说明 ledger 和 user_flux 表脱钩了，是 P0 事件。 + +## 取证 Loki（待执行） + +Grafana → Explore → Loki datasource。这是**修补前漏账数据的唯一来源**（无 ledger row）： + +### 漏账 error log 全量 + +```logql +{service_name="server"} |= "Failed to debit flux after streaming — unpaid usage" +| json +| line_format "{{.userId}} | req={{.requestId}} | flux={{.fluxConsumed}} | {{.error}}" +``` + +### 按 userId 聚合 unbilled 数量 + +```logql +sum by (userId) ( + count_over_time({service_name="server"} |= "Failed to debit flux after streaming" | json [30d]) +) +``` + +### 时间分布（找爆发时段） + +```logql +sum ( + rate({service_name="server"} |= "Failed to debit flux after streaming" | json [5m]) +) +``` + +修补前若有 sustained > 0 → exploit；若是窄峰 → 真实 DB outage。 + +## 处理决策框架 + +跑完上面 query + Loki 后，按下面决策树走： + +``` +┌──────────────────────────────────────────┐ +│ 单用户漏账 fluxConsumed > 1000? │ +├──────────────────────────────────────────┤ +│ YES → exploit 嫌疑 │ +│ ├─ 多 IP / 短时间高频 → confirmed │ +│ │ │ 不补账（用户已知道是漏洞） │ +│ │ │ Ban user 或 require email │ +│ │ │ verification + 强制 reauth │ +│ │ │ 已修补 → 单纯历史损失 │ +│ │ └─ 不需要在 Postgres 写新 ledger │ +│ └─ 单 IP / 时间分散 → 可能正常 power user │ +│ │ 主动联系用户，问明情况 │ +│ └─ 视情况决定是否赠送 flux 补偿 │ +│ │ +│ NO (用户总漏账 < 1000 flux) → 真异常 │ +│ │ 多半是 DB outage / 单次错误 │ +│ │ 不值得逐个追账 │ +│ └─ 整体核销 + 跑 sanity SQL 4 验证 │ +│ user_flux ≡ ledger 仍然一致 │ +└──────────────────────────────────────────┘ +``` + +**关键判断**：修补前的漏账**不在 ledger 里**（debit_failed 不写 row），所以**不需要在 DB 做任何"核销"操作**——余额是干净的，损失只是"曾经免费送出去的 LLM token 成本"。 + +唯一需要写 DB 操作的场景：sanity SQL #4 跑出非零 drift。那是另一个 bug（ledger ↔ user_flux 脱钩），跟漏账无关。 + +## 修补后的监控建议（持续） + +1. 加 Grafana alert：`increase(airi_billing_flux_unbilled_total{reason!="partial_debit_drained"}[5m]) > 0` → 立即 page（partial drain 是合理路径，不 page） +2. 加每周自动 cron job 跑 sanity SQL #4，drift > 0 → 报警（注意：项目里**不允许**新加后台 worker / cron，所以这个 job 应该走外部 ops 工具，比如 Railway scheduled command 或 GitHub Action） + +## What's verified / What's pending + +| Item | Status | +|---|---| +| 漏洞已堵（commit `7267b0d6b`） | ✓ 已确认（见 `flux-unbilled-exploit-fix.md`） | +| 70.2K 历史漏账的 user 分布 | ⊘ 待跑 Loki query | +| user_flux ↔ ledger drift 是否存在 | ⊘ 待跑 SQL #4 | +| Exploit 涉事 user 是否已 ban / re-auth | ⊘ 等数据出来后决定 | +| `airi_billing_flux_unbilled_total{reason!="partial_debit_drained"}` Grafana alert | ⊘ 待配（见 `metrics-ownership.md` Alert SOP） | diff --git a/apps/server/otel/grafana/dashboards/airi-server-overview-cloud.json b/apps/server/otel/grafana/dashboards/airi-server-overview-cloud.json index a2b91b0a5..8e209ecca 100644 --- a/apps/server/otel/grafana/dashboards/airi-server-overview-cloud.json +++ b/apps/server/otel/grafana/dashboards/airi-server-overview-cloud.json @@ -41,8 +41,8 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "expr": "avg(user_active_sessions{service_name=~\"$service\", deployment_environment=~\"$env\"})", - "legendFormat": "sessions" + "expr": "avg(user_distinct_active{service_name=~\"$service\", deployment_environment=~\"$env\"})", + "legendFormat": "users" }, "version": "v0" }, @@ -54,7 +54,7 @@ "transformations": [] } }, - "description": "Currently active sessions in Postgres (Better Auth `session.expires_at > now()`). Cluster-wide gauge — every replica polls the same DB on a 10s cache. We aggregate with `avg()` (not `sum()`, which would multiply by replica count; not `max()`, which biases high when one replica's cache is fresher than another's after a logout).", + "description": "COUNT(DISTINCT user_id) over the Better Auth `session` table where `expires_at > now()`. This is the *real* active-user count. The historical \"Active Users\" panel queried `user.active_sessions` (COUNT(*) on the same table), which counts session **rows** not users — Better Auth creates a new row per sign-in and per OIDC access-token issuance and never GCs expired rows, so the row count drifts up and we have seen it report ~80K on a deployment with hundreds of actual users. Compare with `panel-15` (Active Sessions) to spot session-row inflation; ratio > ~5 means it's time for a session GC cron. Cluster-wide gauge — `avg()`, not `sum()`.", "id": 1, "links": [], "title": "Active Users", @@ -106,7 +106,7 @@ } } }, - "panel-2": { + "panel-15": { "kind": "Panel", "spec": { "data": { @@ -124,8 +124,8 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "expr": "sum(ws_connections_active{service_name=~\"$service\", deployment_environment=~\"$env\"})", - "legendFormat": "connections" + "expr": "avg(user_active_sessions{service_name=~\"$service\", deployment_environment=~\"$env\"})", + "legendFormat": "sessions" }, "version": "v0" }, @@ -137,10 +137,10 @@ "transformations": [] } }, - "description": "Live registry size from chat-ws (ObservableGauge, scraped each export interval).", - "id": 2, + "description": "COUNT(*) over the Better Auth `session` table where `expires_at > now()`. Counts session **rows**, not users — see `panel-1` for the de-duplicated user count. Useful as a denominator to spot row inflation: divide by panel-1 to get rows-per-user, watch for sustained growth.", + "id": 15, "links": [], - "title": "WS Connections", + "title": "Active Sessions", "vizConfig": { "group": "stat", "kind": "VizConfig", @@ -156,6 +156,10 @@ { "color": "green", "value": 0 + }, + { + "color": "yellow", + "value": 5000 } ] }, @@ -216,7 +220,7 @@ "transformations": [] } }, - "description": "5-minute average inbound HTTP request rate. /health (Railway probe) is excluded at the @hono/otel middleware level so this reflects real user traffic.", + "description": "5-minute average inbound HTTP request rate. /health (Railway probe) is excluded at the @hono/otel middleware level so this reflects real user traffic. **Fixed 5m window — intentionally does not follow the dashboard time picker** (see row-level note). For trends, see panel-14 (HTTP Request Rate by Route).", "id": 3, "links": [], "title": "Req/s (5m)", @@ -304,7 +308,7 @@ "transformations": [] } }, - "description": "5xx responses ÷ all responses over the last 5m. Spikes correlate with deploys, upstream outages, or DB problems. >1% warns, >5% pages.", + "description": "5xx responses ÷ all responses over the last 5m. **Fixed 5m window — intentionally does not follow the dashboard time picker**: this is an on-call glance (\"is the service failing right now\"). For range-aware triage use panel-9 donut and panel-44 timeseries. Spikes correlate with deploys, upstream outages, or DB problems. >1% warns, >5% pages.", "id": 4, "links": [], "title": "5xx Rate %", @@ -392,7 +396,7 @@ "transformations": [] } }, - "description": "5-minute average LLM gateway request rate (chat + tts).", + "description": "5-minute average LLM gateway request rate (chat + tts). **Fixed 5m window — see row-level note.** For trends and per-model breakdown see panel-11.", "id": 5, "links": [], "title": "LLM Req/s (5m)", @@ -472,7 +476,7 @@ "transformations": [] } }, - "description": "Email failures ÷ total attempts over the last 5m. >5% means Resend / DNS / suppression-list problems blocking auth flows.", + "description": "Email failures ÷ total attempts over the last 5m. **Fixed 5m window — see row-level note.** >5% means Resend / DNS / suppression-list problems blocking auth flows.", "id": 6, "links": [], "title": "Email Failure %", @@ -548,7 +552,7 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "expr": "topk(8, sum by (gen_ai_request_model) (increase(gen_ai_client_operation_count_total{service_name=~\"$service\", deployment_environment=~\"$env\", gen_ai_request_model!=\"\"}[5m])))", + "expr": "topk(8, sum by (gen_ai_request_model) (increase(gen_ai_client_operation_count_total{service_name=~\"$service\", deployment_environment=~\"$env\", gen_ai_request_model!=\"\"}[$__range])))", "legendFormat": "{{gen_ai_request_model}}" }, "version": "v0" @@ -561,10 +565,10 @@ "transformations": [] } }, - "description": "Share of LLM gateway calls by model. Quickly shows which model is doing the heavy lifting.", + "description": "Share of LLM gateway calls by model, summed over the dashboard time range. Follows the time picker — pick 1h to see the last hour's model mix, pick 7d to see this week's.", "id": 8, "links": [], - "title": "LLM Models (last 5m)", + "title": "LLM Models (range)", "vizConfig": { "group": "piechart", "kind": "VizConfig", @@ -639,7 +643,7 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "expr": "topk(10, sum by (http_route) (increase(http_server_request_duration_seconds_count{service_name=~\"$service\", deployment_environment=~\"$env\", http_request_method!=\"OPTIONS\", http_route!=\"\"}[5m])))", + "expr": "topk(10, sum by (http_route) (increase(http_server_request_duration_seconds_count{service_name=~\"$service\", deployment_environment=~\"$env\", http_request_method!=\"OPTIONS\", http_route!=\"\", http_response_status_code=~\"5..\"}[$__range])))", "legendFormat": "{{http_route}}" }, "version": "v0" @@ -652,10 +656,10 @@ "transformations": [] } }, - "description": "Top 10 Hono-matched routes by request count over the last 5 minutes. Answers \"which endpoint is being hit, and how much\" — replaces the previous HTTP status-code donut whose 2xx slice dominated everything else. Cardinality is bounded because `http_route` is the matched route pattern, not the concrete URL.", + "description": "Top 10 Hono-matched routes by 5xx response count over the dashboard time range. Follows the time picker — pick 1h for \"what's failing right now\", pick 24h for \"what failed most today\". The overall 5xx% gauge (panel-4) is a fixed-5m snapshot for on-call glance; this donut respects the time picker for triage.", "id": 9, "links": [], - "title": "Top Routes by Requests (last 5m)", + "title": "Top Routes by 5xx (range)", "vizConfig": { "group": "piechart", "kind": "VizConfig", @@ -673,7 +677,7 @@ } }, "unit": "short", - "noValue": "no traffic" + "noValue": "no 5xx" }, "overrides": [] }, @@ -1947,6 +1951,122 @@ } } }, + "panel-44": { + "kind": "Panel", + "spec": { + "data": { + "kind": "QueryGroup", + "spec": { + "queries": [ + { + "kind": "PanelQuery", + "spec": { + "hidden": false, + "query": { + "datasource": { + "name": "grafanacloud-projairi-prom" + }, + "group": "prometheus", + "kind": "DataQuery", + "spec": { + "expr": "topk(10, sum by (http_route) (rate(http_server_request_duration_seconds_count{service_name=~\"$service\", deployment_environment=~\"$env\", http_request_method!=\"OPTIONS\", http_route!=\"\", http_response_status_code=~\"5..\"}[$__rate_interval])))", + "legendFormat": "{{http_route}}" + }, + "version": "v0" + }, + "refId": "A" + } + } + ], + "queryOptions": {}, + "transformations": [] + } + }, + "description": "5xx response rate split by route. Use this to confirm whether a 5xx spike in `panel-4` is concentrated on one endpoint (e.g. a broken deploy of /api/v1/openai/*) or scattered (e.g. DB outage taking down everything). Drill into the Logs row (`panel-91`) for the matching error bodies + trace ids.", + "id": 44, + "links": [], + "title": "5xx Rate by Route (top 10)", + "vizConfig": { + "group": "timeseries", + "kind": "VizConfig", + "spec": { + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "smooth", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + }, + "unit": "reqps" + }, + "overrides": [] + }, + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [ + "lastNotNull", + "max" + ], + "displayMode": "table", + "placement": "right", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + } + }, + "version": "13.0.0-23630096546" + } + } + }, "panel-30": { "kind": "Panel", "spec": { @@ -2239,7 +2359,7 @@ "transformations": [] } }, - "description": "PostgreSQL query duration P95 from PgInstrumentation. Spikes correlate with index misses, connection exhaustion, or backend lock contention.", + "description": "PostgreSQL query duration P95 from PgInstrumentation. **Fixed 5m window — does not follow the dashboard time picker** (same posture as the Service Health row stats: this is a \"right now\" glance). Spikes correlate with index misses, connection exhaustion, or backend lock contention.", "id": 50, "links": [], "title": "DB Query P95 (5m)", @@ -2708,6 +2828,71 @@ "version": "13.0.0-23630096546" } } + }, + "panel-91": { + "kind": "Panel", + "spec": { + "data": { + "kind": "QueryGroup", + "spec": { + "queries": [ + { + "kind": "PanelQuery", + "spec": { + "hidden": false, + "query": { + "datasource": { + "name": "grafanacloud-projairi-logs" + }, + "group": "loki", + "kind": "DataQuery", + "spec": { + "expr": "{service_name=~\"$service\", deployment_environment=~\"$env\"} | json | level=~\"warn|error\"", + "legendFormat": "" + }, + "version": "v0" + }, + "refId": "A" + } + } + ], + "queryOptions": {}, + "transformations": [] + } + }, + "description": "Server-side error logs (level=warn|error) from Loki. Loki derived fields turn `trace_id` and `req` into clickable links — `trace_id` jumps to Tempo for full request playback (spans + child calls + DB queries), `req` filters this panel to a single request id. Use this together with panel-9 (which route) and panel-44 (when).", + "id": 91, + "links": [], + "title": "5xx Error Logs", + "vizConfig": { + "group": "logs", + "kind": "VizConfig", + "spec": { + "fieldConfig": { + "defaults": {}, + "overrides": [] + }, + "options": { + "dedupStrategy": "none", + "enableInfiniteScrolling": false, + "enableLogDetails": true, + "prettifyLogMessage": false, + "showCommonLabels": false, + "showControls": false, + "showFieldSelector": false, + "showLabels": true, + "showLevel": true, + "showLogAttributes": true, + "showTime": true, + "sortOrder": "Descending", + "timestampResolution": "ms", + "unwrappedColumns": false, + "wrapLogMessage": true + } + }, + "version": "13.0.0-23630096546" + } + } } }, "layout": { @@ -2740,7 +2925,7 @@ "spec": { "element": { "kind": "ElementReference", - "name": "panel-2" + "name": "panel-15" }, "height": 4, "width": 4, @@ -3045,6 +3230,33 @@ "title": "Errors / Quality" } }, + { + "kind": "RowsLayoutRow", + "spec": { + "collapse": false, + "layout": { + "kind": "GridLayout", + "spec": { + "items": [ + { + "kind": "GridLayoutItem", + "spec": { + "element": { + "kind": "ElementReference", + "name": "panel-44" + }, + "height": 7, + "width": 24, + "x": 0, + "y": 0 + } + } + ] + } + }, + "title": "5xx Triage" + } + }, { "kind": "RowsLayoutRow", "spec": { @@ -3177,13 +3389,26 @@ "spec": { "element": { "kind": "ElementReference", - "name": "panel-90" + "name": "panel-91" }, - "height": 12, + "height": 10, "width": 24, "x": 0, "y": 0 } + }, + { + "kind": "GridLayoutItem", + "spec": { + "element": { + "kind": "ElementReference", + "name": "panel-90" + }, + "height": 10, + "width": 24, + "x": 0, + "y": 10 + } } ] } diff --git a/apps/server/otel/grafana/dashboards/build.ts b/apps/server/otel/grafana/dashboards/build.ts index 278d122e9..4ffa435cf 100644 --- a/apps/server/otel/grafana/dashboards/build.ts +++ b/apps/server/otel/grafana/dashboards/build.ts @@ -355,28 +355,50 @@ function row(title: string, items: ReturnType[], { collapse = false // between defined panel ids and layout references. const elements: Record = {} -// Row 1: Service Health — answers "is anything broken right now?" +// Row 1: Service Health — answers "is anything broken **right now**?" +// +// Time-window policy for this row: all rate / ratio queries use a fixed +// `[5m]` window and DO NOT follow the dashboard time picker. Reason: +// these panels are designed for on-call glance ("is the service healthy +// at this instant"), and we want the number to be stable across whatever +// time range the viewer happened to pick. If we used `$__rate_interval`, +// the same panel would show different numbers depending on whether the +// time picker is set to "last 1 hour" vs "last 7 days", which is +// confusing for an at-a-glance health board. +// +// To see trends over the time-picker range, use the Row 3 timeseries +// (HTTP / LLM / WS by-* panels) which DO follow the time picker. +// Panels titled "(range)" (Distribution donuts, Business stats) also +// follow the time picker by design. +// // Mix of stats (absolute counts) and gauges (bounded ratios with thresholds). elements['panel-1'] = statPanel( 1, 'Active Users', - 'Currently active sessions in Postgres (Better Auth `session.expires_at > now()`). Cluster-wide gauge — every replica polls the same DB on a 10s cache. We aggregate with `avg()` (not `sum()`, which would multiply by replica count; not `max()`, which biases high when one replica\'s cache is fresher than another\'s after a logout).', - [query(`avg(user_active_sessions{${SERVICE_FILTER}})`, 'sessions')], + 'COUNT(DISTINCT user_id) over the Better Auth `session` table where `expires_at > now()`. This is the *real* active-user count. The historical "Active Users" panel queried `user.active_sessions` (COUNT(*) on the same table), which counts session **rows** not users — Better Auth creates a new row per sign-in and per OIDC access-token issuance and never GCs expired rows, so the row count drifts up and we have seen it report ~80K on a deployment with hundreds of actual users. Compare with `panel-15` (Active Sessions) to spot session-row inflation; ratio > ~5 means it\'s time for a session GC cron. Cluster-wide gauge — `avg()`, not `sum()`.', + [query(`avg(user_distinct_active{${SERVICE_FILTER}})`, 'users')], { unit: 'short', steps: [{ color: 'green', value: 0 }, { color: 'yellow', value: 1000 }] }, ) -elements['panel-2'] = statPanel( - 2, - 'WS Connections', - 'Live registry size from chat-ws (ObservableGauge, scraped each export interval).', - [query(`sum(ws_connections_active{${SERVICE_FILTER}})`, 'connections')], - { unit: 'short' }, +elements['panel-15'] = statPanel( + 15, + 'Active Sessions', + 'COUNT(*) over the Better Auth `session` table where `expires_at > now()`. Counts session **rows**, not users — see `panel-1` for the de-duplicated user count. Useful as a denominator to spot row inflation: divide by panel-1 to get rows-per-user, watch for sustained growth.', + [query(`avg(user_active_sessions{${SERVICE_FILTER}})`, 'sessions')], + { unit: 'short', steps: [{ color: 'green', value: 0 }, { color: 'yellow', value: 5000 }] }, ) +// WS Connections stat was removed — its sparkline duplicated the +// timeseries in Row 3 (`panel-13`), which already shows the live +// connection count over time with the same `sum(ws_connections_active)` +// query. Keeping both meant the same number rendered twice on first +// look. The Row 3 timeseries wins because it lets you actually read off +// a value at a specific timestamp instead of squinting at the sparkline. + elements['panel-3'] = statPanel( 3, 'Req/s (5m)', - '5-minute average inbound HTTP request rate. /health (Railway probe) is excluded at the @hono/otel middleware level so this reflects real user traffic.', + '5-minute average inbound HTTP request rate. /health (Railway probe) is excluded at the @hono/otel middleware level so this reflects real user traffic. **Fixed 5m window — intentionally does not follow the dashboard time picker** (see row-level note). For trends, see panel-14 (HTTP Request Rate by Route).', [query(`sum(rate(http_server_request_duration_seconds_count{${SERVICE_FILTER}, http_request_method!="OPTIONS"}[5m]))`, 'req/s')], { unit: 'reqps', steps: [{ color: 'green', value: 0 }, { color: 'yellow', value: 100 }, { color: 'red', value: 500 }], decimals: 2 }, ) @@ -384,7 +406,7 @@ elements['panel-3'] = statPanel( elements['panel-4'] = gaugePanel( 4, '5xx Rate %', - '5xx responses ÷ all responses over the last 5m. Spikes correlate with deploys, upstream outages, or DB problems. >1% warns, >5% pages.', + '5xx responses ÷ all responses over the last 5m. **Fixed 5m window — intentionally does not follow the dashboard time picker**: this is an on-call glance ("is the service failing right now"). For range-aware triage use panel-9 donut and panel-44 timeseries. Spikes correlate with deploys, upstream outages, or DB problems. >1% warns, >5% pages.', [query( `100 * sum(rate(http_server_request_duration_seconds_count{${SERVICE_FILTER}, http_request_method!="OPTIONS", http_response_status_code=~"5.."}[5m])) / clamp_min(sum(rate(http_server_request_duration_seconds_count{${SERVICE_FILTER}, http_request_method!="OPTIONS"}[5m])), 1)`, 'fail %', @@ -395,7 +417,7 @@ elements['panel-4'] = gaugePanel( elements['panel-5'] = statPanel( 5, 'LLM Req/s (5m)', - '5-minute average LLM gateway request rate (chat + tts).', + '5-minute average LLM gateway request rate (chat + tts). **Fixed 5m window — see row-level note.** For trends and per-model breakdown see panel-11.', [query(`sum(rate(gen_ai_client_operation_count_total{${SERVICE_FILTER}}[5m]))`, 'req/s')], { unit: 'reqps', decimals: 2 }, ) @@ -403,7 +425,7 @@ elements['panel-5'] = statPanel( elements['panel-6'] = gaugePanel( 6, 'Email Failure %', - 'Email failures ÷ total attempts over the last 5m. >5% means Resend / DNS / suppression-list problems blocking auth flows.', + 'Email failures ÷ total attempts over the last 5m. **Fixed 5m window — see row-level note.** >5% means Resend / DNS / suppression-list problems blocking auth flows.', [query( `100 * sum(rate(airi_email_failures_total{${SERVICE_FILTER}}[5m])) / clamp_min(sum(rate(airi_email_send_total{${SERVICE_FILTER}}[5m])) + sum(rate(airi_email_failures_total{${SERVICE_FILTER}}[5m])), 1)`, 'fail %', @@ -422,22 +444,30 @@ elements['panel-6'] = gaugePanel( // surfaces 4xx/5xx independently. elements['panel-8'] = piePanel( 8, - 'LLM Models (last 5m)', - 'Share of LLM gateway calls by model. Quickly shows which model is doing the heavy lifting.', + 'LLM Models (range)', + 'Share of LLM gateway calls by model, summed over the dashboard time range. Follows the time picker — pick 1h to see the last hour\'s model mix, pick 7d to see this week\'s.', [query( - `topk(8, sum by (gen_ai_request_model) (increase(gen_ai_client_operation_count_total{${SERVICE_FILTER}, gen_ai_request_model!=""}[5m])))`, + `topk(8, sum by (gen_ai_request_model) (increase(gen_ai_client_operation_count_total{${SERVICE_FILTER}, gen_ai_request_model!=""}[$__range])))`, '{{gen_ai_request_model}}', )], ) +// "Top Routes by Requests" lives as a timeseries in `panel-14` (Top +// Endpoints row) — keeping a donut here too would just be a frozen +// snapshot of the timeseries. Instead, this slot answers the higher- +// value question "which routes are producing the 5xx right now?" so +// the dashboard surfaces *failing* endpoints, not just busy ones. +// Pair with panel-44 (5xx Rate by Route timeseries) for the same data +// over time. elements['panel-9'] = piePanel( 9, - 'Top Routes by Requests (last 5m)', - 'Top 10 Hono-matched routes by request count over the last 5 minutes. Answers "which endpoint is being hit, and how much" — replaces the previous HTTP status-code donut whose 2xx slice dominated everything else. Cardinality is bounded because `http_route` is the matched route pattern, not the concrete URL.', + 'Top Routes by 5xx (range)', + 'Top 10 Hono-matched routes by 5xx response count over the dashboard time range. Follows the time picker — pick 1h for "what\'s failing right now", pick 24h for "what failed most today". The overall 5xx% gauge (panel-4) is a fixed-5m snapshot for on-call glance; this donut respects the time picker for triage.', [query( - `topk(10, sum by (http_route) (increase(http_server_request_duration_seconds_count{${SERVICE_FILTER}, http_request_method!="OPTIONS", http_route!=""}[5m])))`, + `topk(10, sum by (http_route) (increase(http_server_request_duration_seconds_count{${SERVICE_FILTER}, http_request_method!="OPTIONS", http_route!="", http_response_status_code=~"5.."}[$__range])))`, '{{http_route}}', )], + { noValue: 'no 5xx' }, ) // Row 3: Traffic Trends — same data as Row 2, but answering "how is it changing" @@ -564,6 +594,22 @@ elements['panel-42'] = timeseriesPanel( { unit: 'ops' }, ) +// 5xx by route over time — complements panel-9 (donut: which routes +// are failing right now) and panel-40 (4xx/5xx by status code: what +// kind of error). This is the "when did /foo start blowing up" view. +// topk(10) keeps the legend readable when one bad deploy lights up +// the whole API surface. +elements['panel-44'] = timeseriesPanel( + 44, + '5xx Rate by Route (top 10)', + '5xx response rate split by route. Use this to confirm whether a 5xx spike in `panel-4` is concentrated on one endpoint (e.g. a broken deploy of /api/v1/openai/*) or scattered (e.g. DB outage taking down everything). Drill into the Logs row (`panel-91`) for the matching error bodies + trace ids.', + [query( + `topk(10, sum by (http_route) (rate(http_server_request_duration_seconds_count{${SERVICE_FILTER}, http_request_method!="OPTIONS", http_route!="", http_response_status_code=~"5.."}[$__rate_interval])))`, + '{{http_route}}', + )], + { unit: 'reqps' }, +) + // Row 6: Business — money flow elements['panel-30'] = statPanel( 30, @@ -602,7 +648,7 @@ elements['panel-32'] = piePanel( elements['panel-50'] = statPanel( 50, 'DB Query P95 (5m)', - 'PostgreSQL query duration P95 from PgInstrumentation. Spikes correlate with index misses, connection exhaustion, or backend lock contention.', + 'PostgreSQL query duration P95 from PgInstrumentation. **Fixed 5m window — does not follow the dashboard time picker** (same posture as the Service Health row stats: this is a "right now" glance). Spikes correlate with index misses, connection exhaustion, or backend lock contention.', [query( `histogram_quantile(0.95, sum by (le) (rate(db_client_operation_duration_seconds_bucket{${SERVICE_FILTER}}[5m])))`, 'p95', @@ -651,25 +697,43 @@ elements['panel-90'] = logsPanel( `{${SERVICE_FILTER}} |= \`\``, ) +// 5xx-only log stream — paired with the 5xx by-route timeseries and +// donut so on-call goes panel-4 (something is wrong) → panel-9 +// (where) → panel-44 (when) → panel-91 (actual error message + trace +// id, click trace_id → Tempo for full request playback). Filters at +// the Loki query level so Grafana doesn't ship the entire log +// firehose to the browser just to client-side filter. +elements['panel-91'] = logsPanel( + 91, + '5xx Error Logs', + 'Server-side error logs (level=warn|error) from Loki. Loki derived fields turn `trace_id` and `req` into clickable links — `trace_id` jumps to Tempo for full request playback (spans + child calls + DB queries), `req` filters this panel to a single request id. Use this together with panel-9 (which route) and panel-44 (when).', + `{${SERVICE_FILTER}} | json | level=~"warn|error"`, +) + // --------------------------------------------------------------------------- // Layout // --------------------------------------------------------------------------- const rows = [ - // Row 1: 6 stats/gauges × 4 wide × 4 high (full width) + // Row 1: 6 stats/gauges, each 4 wide (4×6=24). Active Users (panel-1) + // and Active Sessions (panel-15) sit side-by-side so on-call can spot + // session-row inflation at a glance (panel-15 climbs while panel-1 + // stays flat → Better Auth row leak, not real user growth). WS + // Connections stat is gone (duplicated by Row 3 timeseries panel-13). row('Service Health', [ item('panel-1', 0, 0, 4, 4), - item('panel-2', 4, 0, 4, 4), + item('panel-15', 4, 0, 4, 4), item('panel-3', 8, 0, 4, 4), item('panel-4', 12, 0, 4, 4), item('panel-5', 16, 0, 4, 4), item('panel-6', 20, 0, 4, 4), ]), - // Row 2: 2 donuts × 12 wide × 7 high — current-state distribution - // Dropped HTTP-methods donut (low-cardinality, redundant with Row 3 by-method - // timeseries) and HTTP-status donut (2xx dominated, 4xx/5xx already broken - // out in Row 5). Replaced status donut with Top Routes which answers a - // higher-information question with the same visual budget. + // Row 2: 2 donuts × 12 wide × 7 high — current-state distribution. + // Left donut: LLM models (where load is going). Right donut: 5xx by + // route (where failures are concentrated). The previous "Top Routes + // by Requests" donut was replaced because its timeseries form in + // Row 3.5 carries the same data with time context; 5xx-by-route is + // the higher-value glance. row('Distribution (now)', [ item('panel-8', 0, 0, 12, 7), item('panel-9', 12, 0, 12, 7), @@ -694,7 +758,7 @@ const rows = [ item('panel-20', 0, 0, 12, 8), item('panel-21', 12, 0, 12, 8), ]), - // Row 5: 1 stacked area + 2 stats + 1 timeseries × 7 high + // Row 5: 1 stacked area + 2 stats + 1 timeseries × 7 high. // Stream Interruptions and ⚠ Flux Unbilled sit next to the 4xx/5xx trend // so revenue-leak signal (which doesn't show up in 5xx) gets the same // glance-weight as transport-layer errors. @@ -704,6 +768,13 @@ const rows = [ item('panel-43', 14, 0, 4, 7), item('panel-42', 18, 0, 6, 7), ]), + // Row 5.5: 5xx by-route trend full width. Triage path: panel-4 + // (something wrong) → panel-9 donut (which route now) → here (when + // it started + per-route rates over time) → panel-91 (actual error + // log lines + clickable trace_id for full request replay in Tempo). + row('5xx Triage', [ + item('panel-44', 0, 0, 24, 7), + ]), // Row 6: 1 stat + 1 gauge + 1 donut × 8 wide × 7 high row('Business', [ item('panel-30', 0, 0, 8, 7), @@ -719,9 +790,12 @@ const rows = [ item('panel-52', 12, 0, 6, 6), item('panel-53', 18, 0, 6, 6), ], { collapse: true }), - // Row 8: full-width logs + // Row 8: full-width logs. Two panels stacked: errors-only on top + // (default focus for triage) and the full firehose below (manual + // filter when you need broader context). row('Logs', [ - item('panel-90', 0, 0, 24, 12), + item('panel-91', 0, 0, 24, 10), + item('panel-90', 0, 10, 24, 10), ]), ] diff --git a/apps/server/src/app.ts b/apps/server/src/app.ts index e06978a82..7cc43c07a 100644 --- a/apps/server/src/app.ts +++ b/apps/server/src/app.ts @@ -42,6 +42,7 @@ import { resolveRequestAuth } from './libs/request-auth' import { sessionMiddleware } from './middlewares/auth' import { emitOtelLog, initOtel } from './otel' import { registerActiveSessionsGauge } from './otel/gauges/active-sessions' +import { registerDistinctActiveUsersGauge } from './otel/gauges/distinct-active-users' import { createAdminFluxGrantsRoutes } from './routes/admin/flux-grants' import { createAuthRoutes } from './routes/auth' import { createCharacterRoutes } from './routes/characters' @@ -527,11 +528,19 @@ export async function createApp() { userDeletionService, posthog, }) - // Register the cluster-wide ObservableGauge for active sessions. Each - // replica polls the same DB (cached 10s, in-flight coalesced) and the - // dashboard aggregates with avg(), not sum(). See observability-conventions.md. - if (resolved.otel) + // Register the cluster-wide ObservableGauges for sessions / users. Each + // replica polls the same DB (cached 10s, in-flight coalesced); dashboards + // aggregate with avg(), not sum(). See observability-conventions.md. + // + // Both gauges share the same `session` table: `user.active_sessions` is + // `COUNT(*)` (row inflation prone), `user.distinct_active` is + // `COUNT(DISTINCT user_id)` (real active-user count). Comparing the two + // surfaces session-row leakage from missing GC + per-OIDC-token row + // creation. + if (resolved.otel) { registerActiveSessionsGauge(resolved.otel.auth.activeSessions, resolved.db, resolved.otel.observability.metricReadErrors) + registerDistinctActiveUsersGauge(resolved.otel.auth.distinctActiveUsers, resolved.db, resolved.otel.observability.metricReadErrors) + } const { app, injectWebSocket } = await buildApp({ auth: resolved.auth, diff --git a/apps/server/src/otel/gauges/distinct-active-users.ts b/apps/server/src/otel/gauges/distinct-active-users.ts new file mode 100644 index 000000000..a6f4e7269 --- /dev/null +++ b/apps/server/src/otel/gauges/distinct-active-users.ts @@ -0,0 +1,94 @@ +import type { AuthMetrics, ObservabilityMetrics } from '..' +import type { Database } from '../../libs/db' + +import { useLogger } from '@guiiai/logg' +import { countDistinct, gt } from 'drizzle-orm' + +import { session as sessionTable } from '../../schemas/accounts' + +/** + * Wire the `user.distinct_active` ObservableGauge to a Postgres + * `COUNT(DISTINCT user_id)` over the Better Auth session table. + * + * Use when: + * - Assembling DI in `createApp()`, exactly once per process. + * + * Why this exists alongside `registerActiveSessionsGauge`: + * - `user.active_sessions` is `COUNT(*)` — counts session **rows**. Better + * Auth creates a new row per sign-in and per OIDC access-token issuance + * (the `oauth_access_token` table has a FK to `session.id`) and never GCs + * expired rows, so the row-count drifts up over time independently of + * the real user base. We've seen this metric show ~80K on a small + * deployment where the real distinct-user count is ~hundreds. + * - `user.distinct_active` is `COUNT(DISTINCT user_id)` — the actual + * active-user gauge. Pair with `user.active_sessions` to spot session + * inflation: if rows / users ratio climbs past ~5 it's probably time + * to add a session-GC cron or shorten Better Auth's `expiresIn`. + * + * Multi-replica note: + * - Cluster-wide gauge — every replica reads the same DB and reports the + * same value. Dashboards MUST aggregate with `avg()`, NOT `sum()`. See + * observability-conventions.md. + * + * Concurrency: + * - Same in-flight promise lock pattern as `registerActiveSessionsGauge`, + * so concurrent OTel collection cycles fold into one DB query. + * + * Failure mode: + * - DB error → increment `airi.observability.read_errors{metric}` and skip + * `result.observe(...)`. Prometheus staleness exposes the outage instead + * of pinning a stale cached value forever. + */ +export function registerDistinctActiveUsersGauge( + gauge: AuthMetrics['distinctActiveUsers'], + db: Database, + metricReadErrors: ObservabilityMetrics['metricReadErrors'], +) { + const log = useLogger('distinct-active-users-gauge').useGlobalConfig() + const CACHE_TTL_MS = 10_000 + + let cachedAt = 0 + let cachedCount = 0 + let refreshInFlight: Promise | null = null + + async function refresh(): Promise { + try { + // Use the app clock (`new Date()`) for the same reason as + // `registerActiveSessionsGauge`: agree with Better Auth's own + // session-validity check, which uses `new Date()` rather than + // `NOW()`. + const rows = await db + .select({ count: countDistinct(sessionTable.userId) }) + .from(sessionTable) + .where(gt(sessionTable.expiresAt, new Date())) + cachedCount = Number(rows[0]?.count ?? 0) + cachedAt = Date.now() + return true + } + catch (err) { + log.withError(err).warn('Failed to read distinct active users for gauge') + metricReadErrors.add(1, { metric: 'user.distinct_active' }) + return false + } + } + + gauge.addCallback(async (result) => { + const now = Date.now() + + if (cachedAt !== 0 && now - cachedAt < CACHE_TTL_MS) { + result.observe(cachedCount) + return + } + + if (!refreshInFlight) { + refreshInFlight = refresh().finally(() => { + refreshInFlight = null + }) + } + const ok = await refreshInFlight + + if (ok) { + result.observe(cachedCount) + } + }) +} diff --git a/apps/server/src/otel/index.ts b/apps/server/src/otel/index.ts index 9ef6f35a5..5245db9f0 100644 --- a/apps/server/src/otel/index.ts +++ b/apps/server/src/otel/index.ts @@ -44,6 +44,7 @@ import { METRIC_STRIPE_PAYMENT_FAILED, METRIC_STRIPE_SUBSCRIPTION_EVENT, METRIC_USER_ACTIVE_SESSIONS, + METRIC_USER_DISTINCT_ACTIVE, METRIC_USER_LOGIN, METRIC_USER_REGISTERED, METRIC_WS_CONNECTIONS_ACTIVE, @@ -77,6 +78,21 @@ export interface AuthMetrics { * "Multi-Replica Considerations". */ activeSessions: ObservableGauge + /** + * Pull-based gauge for distinct users with ≥1 non-expired session. + * + * Use when: + * - Querying real "active users" — not session rows. Better Auth creates a + * new `session` row per sign-in and per OIDC token refresh, and never + * GCs expired rows, so {@link AuthMetrics.activeSessions} drifts up + * over time even when the actual user base is small. + * + * Expects: + * - Backed by `SELECT COUNT(DISTINCT user_id) FROM session WHERE expires_at > now()`. + * Same cluster-wide truth as `activeSessions`; dashboards MUST aggregate + * with `avg()`, not `sum()` — see observability-conventions.md. + */ + distinctActiveUsers: ObservableGauge } export interface EngagementMetrics { @@ -225,7 +241,10 @@ export function initOtel(env: Env): OtelInstance | null { description: 'Number of user sign-ins', }), activeSessions: meter.createObservableGauge(METRIC_USER_ACTIVE_SESSIONS, { - description: 'Active user sessions sourced from Postgres (cluster-wide; dashboard must use max(), not sum())', + description: 'Active user sessions sourced from Postgres (cluster-wide; dashboard must use avg(), not sum())', + }), + distinctActiveUsers: meter.createObservableGauge(METRIC_USER_DISTINCT_ACTIVE, { + description: 'Distinct users with ≥1 non-expired session — true active-user count, immune to per-row session inflation (cluster-wide; dashboard must use avg(), not sum())', }), } diff --git a/apps/server/src/utils/observability.ts b/apps/server/src/utils/observability.ts index d324c3333..fa1bade59 100644 --- a/apps/server/src/utils/observability.ts +++ b/apps/server/src/utils/observability.ts @@ -37,6 +37,11 @@ export const METRIC_AUTH_FAILURES = 'auth.failures' export const METRIC_USER_REGISTERED = 'user.registered' export const METRIC_USER_LOGIN = 'user.login' export const METRIC_USER_ACTIVE_SESSIONS = 'user.active_sessions' +// Distinct users with at least one non-expired session row. Pair with +// USER_ACTIVE_SESSIONS to detect "session row inflation" (Better Auth +// creates a new row per sign-in / per OIDC token refresh and never GCs) +// vs real user growth. +export const METRIC_USER_DISTINCT_ACTIVE = 'user.distinct_active' // Engagement (AIRI custom) export const METRIC_CHAT_MESSAGES = 'chat.messages' diff --git a/docs/ai/context/verification-automation.md b/docs/ai/context/verification-automation.md new file mode 100644 index 000000000..0579c8092 --- /dev/null +++ b/docs/ai/context/verification-automation.md @@ -0,0 +1,220 @@ +# Verification 自动化方案 + +设计稿，未实施。落到这里是为了让 verification 流程从「人工跑命令贴输出」走向「机器跑断言贴 evidence」，同时保留 AGENTS.md 里 Iron Law 的语义。 + +## TL;DR + +1. **原因**：现有 5 份 verification 文档结构清晰，但执行步骤需要人工跑命令、人工贴输出、人工记录「最后验证」日期。一旦超过 30 天，AGENTS.md 规定默认 unverified，没有机制能识别这种过期。 +2. **猜想**：verification 文档继续作为 single source of truth，每份文档关联一份可执行 artifact，artifact 跑通就是 evidence，跑通时间就是「最后验证」。 +3. **决策**：分三层实施，集成测试覆盖 in-repo 可重现路径，live verifier 覆盖只能在已部署环境验证的路径，CI 守护过期时间。 + +## 背景 + +`apps/server/docs/ai-context/verifications/` 下 5 份文档，结构基本统一： + +- `场景 / 用户路径`：写明用户敲 X，预期得到 Y +- `命令 / 步骤`：手工敲的 curl、SQL、UI 操作 +- `预期 / 实际输出`：贴 response body、log 节选、screenshot 路径 +- `Evidence`：commit SHA、行号引用、测试文件路径 +- `Status` 与 `最后验证`：人工维护 + +其中 3 份文档（`flux-unbilled-exploit-fix`、`flux-unbilled-reconciliation`、`admin-flux-grants`）引用了已落库的 vitest 单测，剩下 2 份（`email-auth`、`account-deletion`）以手工 curl + 真实 Resend / 真实数据库为主。 + +## 拆解现状 + +把 5 份文档里的步骤按「证据来源」拆开，能看到三类： + +1. **纯代码路径**，例如 partial-debit 的数值逻辑、ledger 行写入。这类已经被 vitest 单测覆盖，证据来源是 `expect()` 断言。 +2. **跨外部边界的用户路径**，例如「N 个并发 LLM completion 触发 pre-flight 拒绝 + ledger 写入 + metric 上报」。这类需要 pg、redis、Hono app、Prometheus `/metrics` 端点同时在场，目前没有自动化覆盖。 +3. **依赖部署环境的路径**，例如 Resend 真实投递、Stripe webhook 回调、Grafana panel 斜率、Better Auth 跨域 OIDC handoff。这类无论在 PR CI 还是本地都无法完整跑通，必须在 staging 或 prod 上验证。 + +第 1 类已经自动化，第 2、3 类是空缺。 + +## 提出猜想 + +verification 文档的「用户路径」描述天然适合作为测试用例标题。如果给每份文档加一份配套 artifact，artifact 类型按上面三类分发： + +- 纯代码路径，归到 `*.test.ts`，已经这样做 +- 跨边界的用户路径，归到 `*.integration.test.ts`，testcontainers 起依赖 +- 依赖部署环境的路径，归到 `*.verifier.ts`，针对 staging URL 跑，post-deploy 触发 + +每份文档头部加一段 frontmatter，机器读取后能回答三个问题： + +1. 这份文档对应的 feature 是什么 +2. 自动化 artifact 在哪里 +3. 上次自动化跑通是什么时候 + +## 分节解答 + +### 一、frontmatter schema + +```yaml +--- +feature: flux-unbilled-exploit-fix +owner: rbxin2003@gmail.com +automated_by: + - kind: unit + path: apps/server/src/services/billing/tests/billing-service.test.ts + cases: + - 'rejects pre-flight when balance is below FLUX_PER_REQUEST' + - 'non-streaming completion drains partial balance and logs charged' + - kind: integration + path: apps/server/tests/verifications/flux-unbilled.integration.test.ts + - kind: live + path: apps/server/tests/verifications/flux-unbilled.verifier.ts + schedule: post-deploy +last_verified: + unit: 2026-05-15 + integration: 2026-05-15 + live: 2026-05-14 +expires_after_days: 30 +--- +``` + +字段语义钉死： + +- `feature`：文档 slug，与文件名同名 +- `automated_by[].kind`：`unit` / `integration` / `live`，三选一 +- `automated_by[].path`：可执行文件路径，CI 跑通后能写回 `last_verified` +- `last_verified.`：YYYY-MM-DD，由 CI 自动写回，人不手动改 +- `expires_after_days`：默认 30，与 AGENTS.md 一致 + +### 二、集成测试 harness + +放在每个 app 下的 `tests/verifications/` 目录，例如 `apps/server/tests/verifications/`。harness 提供： + +1. testcontainers 起 Postgres 16 + Redis 7，注入与 `.env.example` 同 schema 的环境变量 +2. `createApp()` 直接 mount，不走真实端口，调用 `app.request(...)` +3. 三种断言入口： + - HTTP 响应，按现有 `app.test.ts` 范式 + - DB 状态，通过 drizzle 查 `flux_transaction` / `user_flux` + - Metric 状态，scrape `/metrics` 文本，匹配 `airi_billing_flux_unbilled_total{...} ` + +最小测试骨架： + +```ts +describe('verification: flux-unbilled-exploit-fix', () => { + let ctx: VerificationContext + + beforeAll(async () => { + ctx = await startVerificationContext() + }) + + afterAll(async () => { + await ctx.stop() + }) + + it('concurrent partial-balance requests yield one partial debit and N-1 pre-flight 402', async () => { + await ctx.seedUser({ id: 'u1', balance: 5 }) + await ctx.setConfig({ FLUX_PER_REQUEST: 100 }) + + const responses = await Promise.all( + Array.from({ length: 5 }, () => ctx.app.request('/api/v1/openai/...')), + ) + + expect(responses.filter(r => r.status === 402)).toHaveLength(5) + const ledger = await ctx.db.query.fluxTransaction.findMany({ where: ... }) + expect(ledger).toHaveLength(0) + + const metrics = await ctx.scrapeMetrics() + expect(metrics).toMatchMetric('airi_billing_flux_unbilled_total', { + labels: { reason: 'partial_debit_drained' }, + delta: 0, + }) + }) +}) +``` + +`MatchMetric` 与 `scrapeMetrics` 这两个 helper 放在 `packages/server-runtime` 或 `apps/server/src/testing/`，由集成测试和 live verifier 共用。 + +### 三、live verifier + +针对 staging / prod。形态选 vitest 也可以，选独立 CLI 也可以，差别在「是否需要被 CI 用 `--include` pattern 隔离」。建议直接沿用 vitest，给文件后缀 `.verifier.ts`，配 `vitest.config.ts` 的 `include` / `exclude` 把它们与 unit / integration 隔离。 + +live verifier 的断言对象不再是「mount 的 Hono app」，是「真实 URL」： + +```ts +describe('live verifier: flux-unbilled-exploit-fix', () => { + it('panel-43 slope is below alert threshold over the last 5 minutes', async () => { + const slope = await prometheusQuery( + 'increase(airi_billing_flux_unbilled_total[5m])', + { url: process.env.PROM_URL! }, + ) + expect(slope).toBeLessThan(0.5) + }) +}) +``` + +需要凭据的项目（Prometheus、Resend、Stripe）通过 env 注入，与 `secrets-management` 规则一致，不写进文件。 + +### 四、CI 编排 + +三条 GitHub Actions workflow： + +1. **`verification-unit.yml`**：PR 触发，跑全部 `*.test.ts`。现状已有，作为 baseline。 +2. **`verification-integration.yml`**：PR 触发，跑全部 `*.integration.test.ts`。预计单跑 60 至 180 秒（testcontainers 启动），用 matrix 拆分到多个 worker。仅在改动触及 `apps/server/**` 或 `packages/server-*/**` 时跑，其他改动 skip。 +3. **`verification-live.yml`**：post-deploy 触发（Railway deploy hook → GitHub repository_dispatch），针对 staging URL 跑全部 `*.verifier.ts`。跑通后自动 PR 一份更新 `last_verified.live` 的提交，或者直接 commit 回 main（按团队偏好选）。 + +第 2 类必要的 secret：testcontainers 自身不需要 secret，只需要 docker daemon，GitHub Actions runner 默认带。第 3 类需要 `PROM_URL`、`PROM_TOKEN`、`STRIPE_TEST_KEY`、`RESEND_API_KEY` 等，放到 GitHub Actions secrets。 + +### 五、过期守护 + +新增 `scripts/verification-doctor.ts`，在 `verification-unit.yml` 末尾跑： + +```ts +// 遍历所有 verification 文档 +// 读 frontmatter.last_verified +// 与 frontmatter.expires_after_days 比较 +// 超期 -> stderr 报告 + exit 1 +``` + +CI 失败时输出形如： + +``` +✗ flux-unbilled-reconciliation: last_verified.integration = 2025-12-01 (expired 165 days) +✗ email-auth: last_verified.live = (none) +``` + +主分支跑过期检查也跑，跑失败不阻塞 main，只发到 Slack / Lark 通知频道，避免老文档过期把全员卡住。 + +## 回指前文 + +回到 TL;DR 的三条决策： + +1. 「集成测试覆盖 in-repo 可重现路径」对应第二节，testcontainers + drizzle + metric scrape 是这一层的最小工具集。 +2. 「live verifier 覆盖只能在已部署环境验证的路径」对应第三节，针对真实 URL 跑 Prometheus query、Stripe test mode、Resend dashboard API。 +3. 「CI 守护过期时间」对应第五节，frontmatter 的 `last_verified` 由 CI 写回，doctor 脚本扫超期。 + +三层加起来，verification 文档从「人工 claim」变成「机器 claim + 人工 narrative」。 + +## 影响面 + +| 维度 | 影响 | +|---|---| +| 单测时间 | 不变 | +| PR CI 时间 | 新增 60 至 180 秒（取决于 testcontainers 并发 + matrix 拆分） | +| 本地开发 | 默认 `pnpm exec vitest run` 不跑 integration，要显式跑 `pnpm verify:integration` | +| docker 依赖 | 本地跑 integration 需要 docker daemon，已有 `docker-compose.otel.yml` 范式 | +| Secret 管理 | live verifier 需要 4 至 6 个 staging secret，放 GitHub Actions secrets | +| 文档维护 | verification 文档新增 frontmatter，原有 markdown 正文不变 | +| AGENTS.md | 加一段「如何写 verification artifact」，引用本文 | + +## 可观测性 / eval + +实施后用三个指标判断方案有效： + +1. **集成测试覆盖率**：5 份文档里有几份对应有 `*.integration.test.ts`，目标 100% +2. **live verifier 触发频率**：post-deploy 一次必跑，跑失败的次数与生产 incident 的相关性 +3. **doctor 报告超期数**：每周扫一次，超期数应当趋近 0 + +第 3 个指标如果长期不为 0，说明 verification 流程仍需要人工介入太多，要回头看 frontmatter 设计是否合适。 + +## 收束 + +这份方案保留 verification 文档的人工 narrative（root cause、why、tradeoff），把可执行部分挪到代码，把过期检测交给 CI。实施分三步： + +1. 先做 frontmatter schema 与 doctor 脚本，零代码改动，立即能识别已有 5 份文档的过期状态。 +2. 再做 `flux-unbilled-exploit-fix` 的集成测试样板，跑通一个 case 形成模板。 +3. 最后逐份补齐 integration 与 live verifier。 + +如果某一份文档（例如 `email-auth`）的 live 验证依赖 Resend 真实投递，确认收件状态需要轮询 Resend `/emails` API，这部分实现成本较高，可以推到第三步的尾巴上单独立项。