feat(server/otel): restructure observability metrics and add active sessions gauge

- Moved RateLimitMetrics import path to a more centralized location.
- Introduced a new file for active sessions gauge to track user sessions in the database.
- Updated index.ts to include new metrics and ensure proper initialization of observability metrics.
- Modified various routes and services to utilize the new observability structure.
- Added smoke tests for HTTP and WebSocket metrics to ensure proper metric registration and functionality.
- Enhanced error handling for metrics reading failures to improve observability.

apps/server/docs/ai-context/observability-conventions.md

@@ -76,21 +76,13 @@ Redis 相关优先复用 instrumentation 自动产生的标准属性，不要重
 
 ## Metric Name 策略
 
-当前 `apps/server` 仍保留以下 metric name：
+`apps/server` 的 LLM gateway metric 现在全部用标准 `gen_ai.client.*` semconv 名 + AIRI `airi.billing.*` 计费名。旧的 `llm.request.*` / `llm.tokens.*` / `flux.consumed` 字面名都已经迁移完，请**不要在新代码或 reviewer 建议里复活**它们 —— 代码里 const 命名（如 `METRIC_FLUX_CONSUMED`）保留是历史 identifier，对应的字面值已经是 `airi.billing.flux.consumed`，以字面值为准。
 
-- `llm.request.duration`
-- `llm.request.count`
-- `llm.tokens.prompt`
-- `llm.tokens.completion`
-- `flux.consumed`
+新增或重命名 metric 时遵守：
 
-这是有意为之，不是遗漏。
-
-原因：
-
-- metric name 改动比 attribute 改动更容易破坏现有 Prometheus 查询、Grafana 面板和告警。
-- 目前更高价值的是先统一 metric attributes，使查询维度稳定。
-- 如需迁移 metric name，应该走兼容迁移方案，而不是在普通功能改动里直接重命名。
+- metric name 改动比 attribute 改动更容易破坏现有 Prometheus 查询、Grafana 面板和告警。**先确认 dashboard / alerts 是否在跑这条 series**，再决定是否重命名。
+- 重命名一定要走兼容迁移：先双发新旧两条 series，留出窗口给消费方切换，再删旧的；不要在普通功能改动里直接重命名。
+- 完整 metric 清单（含 Prometheus 系列名）维护在 [`observability-metrics.md`](./observability-metrics.md)。新增任何 metric 都要同步更新那份文档。
 
 ## Grafana / Prometheus 查询策略
 
@@ -162,13 +154,13 @@ span name 目前允许保留业务可读格式，例如：
 
 - [packages/server-shared/src/observability.ts](/packages/server-shared/src/observability.ts)
 - [apps/server/src/routes/v1completions.ts](/apps/server/src/routes/v1completions.ts)
-- [apps/server/src/libs/otel.ts](/apps/server/src/libs/otel.ts)
+- [apps/server/src/otel/index.ts](/apps/server/src/otel/index.ts)
 - [services/telegram-bot/src/llm/actions.ts](/services/telegram-bot/src/llm/actions.ts)
 - [services/telegram-bot/src/bots/telegram/agent/actions/read-message.ts](/services/telegram-bot/src/bots/telegram/agent/actions/read-message.ts)
 
 ## SemconvStability 迁移说明
 
-`@opentelemetry/instrumentation-http` 0.215+ 默认 OLD semconv（`http.server.duration` in ms），不是 STABLE 名。AIRI 在 [apps/server/instrumentation.mjs](/apps/server/instrumentation.mjs) 顶部强制 `OTEL_SEMCONV_STABILITY_OPT_IN=http`（仅 STABLE）。
+`@opentelemetry/instrumentation-http` 0.215+ 默认 OLD semconv（`http.server.duration` in ms），不是 STABLE 名。AIRI 在 [apps/server/instrumentation.ts](/apps/server/instrumentation.ts) 顶部强制 `OTEL_SEMCONV_STABILITY_OPT_IN=http`（仅 STABLE）。
 
 | Semconv 模式 | 发哪些 series | 我们用 |
 |---|---|---|
@@ -185,20 +177,80 @@ span name 目前允许保留业务可读格式，例如：
 
 **何时切回 `dup`**：将来如果有别的 service 主动 scrape 本 server 的 OLD-name 系列，临时切几周完成迁移即可。
 
+## Multi-Replica 注意事项
+
+服务跑在 Railway 上有 ≥2 个副本（见 [workers-and-runtime.md](./workers-and-runtime.md)），所有 metric 设计必须显式考虑跨副本聚合。
+
+### `service.instance.id` 必须设
+
+[apps/server/instrumentation.ts](/apps/server/instrumentation.ts) 在 resource 上注入 `service.instance.id`，按优先级取 `RAILWAY_REPLICA_ID` → `SERVER_INSTANCE_ID` → `randomUUID()`（带 warn 日志，提示 ops 系列会随重启 churn）。`HOSTNAME` 曾经在 fallback 链里但 Railway 没文档化它是否 per-replica 唯一，所以踢出去了；需要跨重启稳定时显式设 `SERVER_INSTANCE_ID`。
+
+**没设的后果**：两个副本的所有 metric series label tuple 完全一致（`service_name` + `deployment_environment` 一样），Prometheus 收到时按规则丢一条 / collapse 系列，结果一个副本完全消失。
+
+加新 metric 时不用做任何事——只要从 `meter` 创建出来，instance id 自动随 resource 一起带上。
+
+### 按 instrument 类型的副本安全表
+
+| 类型 | 副本安全？ | 聚合方式 | 备注 |
+|---|---|---|---|
+| `Counter` | ✅ | `sum(rate(x[5m]))` | 每副本本地累加，`rate()` 自动处理重启 |
+| `Histogram` | ✅ | `histogram_quantile(0.95, sum by (le, ...) (rate(x_bucket[5m])))` | 每副本本地 bucket，`sum by (le)` 合并 |
+| `ObservableGauge`（**per-replica 状态**，如 `ws.connections.active`） | ✅ | `sum(x)` | callback 读本地 registry，所有副本求和 = 集群总量 |
+| `ObservableGauge`（**cluster-wide 状态**，如 `user.active_sessions`） | ⚠️ | `max(x)` 或 `avg(x)` | 所有副本读同一份外部状态（DB），sum 会乘以副本数 |
+| `UpDownCounter` | ⚠️ | 看场景 | 必须保证 `+1` 和 `-1` 在**同一副本**触发；否则单副本永久 +N 另一副本永久 -N |
+
+### `UpDownCounter` 红线
+
+只在以下情况用：
+- `+1` 和对应的 `-1` 都在**同一请求生命周期**或**同一进程的局部状态机**里发生（典型：`http.server.active_requests` —— 请求开始 +1，结束 -1，必在同一副本）
+- 不依赖任何外部 TTL / GC / 异步过期
+
+如果存在「TTL 自然过期」「跨实例资源转移」「依赖 webhook 异步触发 -1」之类的情况，**不要用 UpDownCounter**。改用：
+- `ObservableGauge` 从权威存储（DB / Redis）按 callback 读真实值，dashboard 用 `max()` / `avg()` 聚合
+- 或者只保留对应的 `Counter`（"created" + "deleted"），让 dashboard 自己算差值
+
+历史教训：`user.active_sessions` 最早是 UpDownCounter，登录 +1 / 登出 -1。但 Better Auth 的 session TTL 过期不会调 delete hook，counter 单实例就漂；多副本登录在 A、登出在 B 直接撕裂。改成 `ObservableGauge` 后由 [apps/server/src/app.ts](/apps/server/src/app.ts) 的 `registerActiveSessionsGauge` 通过 `SELECT COUNT(*) FROM session WHERE expires_at > NOW()` 在 scrape 时按需查 DB，带 10s 内存缓存避免 hammer。
+
+### Dashboard 查询模板
+
+加新 panel 时按这个清单核对：
+
+| 数据语义 | PromQL 模板 |
+|---|---|
+| 业务事件速率（Counter） | `sum(rate(x_total{...}[$__rate_interval]))` |
+| 按 label 切分速率 | `sum by (<label>) (rate(x_total{...}[$__rate_interval]))` |
+| 时延分位（Histogram） | `histogram_quantile(0.95, sum by (le, <label>) (rate(x_bucket{...}[$__rate_interval])))` |
+| 集群总量（per-replica gauge） | `sum(x{...})` |
+| 集群唯一值（cluster-wide gauge） | `max(x{...})` 或 `avg(x{...})` |
+| 按副本拆分调试 | `<agg> by (service_instance_id) (x{...})` |
+| 错误率 | `100 * sum(rate(x_total{...,status_code=~"5.."}[5m])) / clamp_min(sum(rate(x_total{...}[5m])), 1)` |
+
+红线：**任何 cumulative counter 都不能直接 `sum()` 不 wrap rate/increase**。Counter 在副本重启时归零，没有 rate() 包裹 Prometheus 会跳变；用 `increase($__range)` 看「时间窗口内总量」，用 `rate([interval])` 看「当前速率」。
+
+### 「按副本拆分」何时加
+
+默认 panel 都聚合到集群层面。但以下场景应该加 `by (service_instance_id)` 拆分图：
+
+- 进程级资源（heap、event loop、DB pool）——一个副本泄漏 / pin CPU 别的副本掩盖不掉
+- WS 连接 ——可以看出来是不是单个副本不均衡
+- 自定义的 ObservableGauge 排查
+
+Dashboard 当前 Infrastructure 行已经是 by instance 的（Heap、Event Loop、DB Pool）。
+
 ## Counter priming 注意事项
 
 OTel SDK 的 Counter / UpDownCounter 在第一次 `.add()` 之前**完全不出现在 Prometheus 抓取里**。Histogram 同理（要等第一次 `.record()`）。
 
 后果：低流量 metric 在 dashboard 上看起来像「埋点丢了」，告警里 `absent()` 也无法工作。
 
-[apps/server/src/libs/otel.ts](/apps/server/src/libs/otel.ts) 的 `primeCounter` 在 SDK 启动后给每个 Counter 调一次 `.add(0)`，把 series 注册出来；`0` 不影响 rate / sum 计算。
+[apps/server/src/otel/index.ts](/apps/server/src/otel/index.ts) 的 `primeCounter` 在 SDK 启动后给每个 Counter 调一次 `.add(0)`，把 series 注册出来；`0` 不影响 rate / sum 计算。
 
 加新 Counter 时**记得加进 prime 列表**，否则未触发的指标在 Grafana 里就是空的。
 
-验证脚本：[apps/server/src/scripts/otel-smoke.mjs](/apps/server/src/scripts/otel-smoke.mjs)
+验证脚本：[apps/server/src/scripts/otel/smoke.ts](/apps/server/src/scripts/otel/smoke.ts)
 
 ```sh
-pnpm -F @proj-airi/server exec node --import tsx ./src/scripts/otel-smoke.mjs
+pnpm -F @proj-airi/server exec node --import tsx ./src/scripts/otel/smoke.ts
 ```
 
 打印 SDK 启动后立即可见的所有 instrument 名字。

apps/server/docs/ai-context/observability-metrics.md

@@ -14,7 +14,7 @@ OTel SDK 在导出到 Prometheus 时做两件事：
    - `http_server_request_duration_seconds_bucket`（含 `le` label）
    - `http_server_request_duration_seconds_count`
    - `http_server_request_duration_seconds_sum`
-4. UpDownCounter 不加 `_total`：`ws.connections.active` → `ws_connections_active`
+4. UpDownCounter / ObservableGauge 不加 `_total`：`ws.connections.active` → `ws_connections_active`、`user.active_sessions` → `user_active_sessions`
 5. 带单位的 instrument 在 SDK 导出时把单位插进名字：`airi.stripe.revenue`（unit `minor_unit`）→ `airi_stripe_revenue_minor_unit_total`
 
 > 查询面板若拼名字时不确定后缀，先用 `{__name__=~"airi_billing_flux.*"}` 之类正则探一下。
@@ -23,10 +23,14 @@ OTel SDK 在导出到 Prometheus 时做两件事：
 
 | Metric | 类型 | Unit | 来源 | 关键 attributes |
 |---|---|---|---|---|
-| `http.server.request.duration` | Histogram | s | `instrumentation-http`（STABLE semconv） | `http.request.method`、`http.route`、`http.response.status_code` |
-| `http.server.active_requests` | UpDownCounter | — | [middlewares/otel.ts](../../src/middlewares/otel.ts) `otelMiddleware` | `http.request.method`、`http.route` |
+| `http.server.request.duration` | Histogram | s | [`@hono/otel`](https://www.npmjs.com/package/@hono/otel) `httpInstrumentationMiddleware` in [app.ts](../../src/app.ts) | `http.request.method`、`http.route`、`http.response.status_code` |
+| `http.server.active_requests` | UpDownCounter | — | 同上 | `http.request.method` |
 
-> **STABLE-only**：[instrumentation.mjs:25](../../instrumentation.mjs) 把 `OTEL_SEMCONV_STABILITY_OPT_IN=http` 提前注入。OLD 系列（`http.server.duration` in ms）不再发射。详见 [`observability-conventions.md` 的 SemconvStability 章节](./observability-conventions.md#semconvstability-迁移说明)。
+> **入站走 @hono/otel，出站走 auto HttpInstrumentation**：auto instrumentation 在 Node http 层抓数据时 Hono 还没匹配路由，`http.route` label 永远为空。`@hono/otel` 在 Hono middleware 链里跑，能拿到匹配后的路由 pattern（`/api/v1/users/:id` 而非具体 URL），所以入站 metric 由它产生。auto HttpInstrumentation 在 [instrumentation.ts](../../instrumentation.ts) 里通过 `ignoreIncomingRequestHook: () => true` 仅保留**出站**（LLM gateway、Stripe、Resend），那部分还是要它来跟踪。
+>
+> **STABLE-only**：[instrumentation.ts](../../instrumentation.ts) 把 `OTEL_SEMCONV_STABILITY_OPT_IN=http` 提前注入。OLD 系列（`http.server.duration` in ms）不再发射。详见 [`observability-conventions.md` 的 SemconvStability 章节](./observability-conventions.md#semconvstability-迁移说明)。
+>
+> `/health` 路径在 [app.ts](../../src/app.ts) 的 @hono/otel 包装层被显式 skip，Railway 健康检查不进 metric。
 
 ## Auth & Users
 
@@ -38,7 +42,11 @@ OTel SDK 在导出到 Prometheus 时做两件事：
 | `auth.failures` | Counter | `after` hook，`ctx.context.returned` 含 `error` | `auth.method` |
 | `user.registered` | Counter | `databaseHooks.user.create.after` | — |
 | `user.login` | Counter | `databaseHooks.session.create.after` | — |
-| `user.active_sessions` | UpDownCounter | session create / delete | — |
+| `user.active_sessions` | ObservableGauge | [app.ts](../../src/app.ts) `registerActiveSessionsGauge`，scrape 时查 `SELECT COUNT(*) FROM session WHERE expires_at > NOW()`（10s 内存缓存） | — |
+
+> **`user.active_sessions` 是 cluster-wide gauge，dashboard 必须用 `max()` / `avg()`，不能用 `sum()`**。所有副本读同一份 DB 报同一个值，sum 会乘以副本数。详见 [observability-conventions.md 的 Multi-Replica 章节](./observability-conventions.md#multi-replica-注意事项)。
+>
+> 历史：之前是 UpDownCounter（+1 on login, -1 on logout），但 Better Auth session TTL 过期不会调 delete hook，counter 单实例就漂；多副本下登录在 A、登出在 B 会直接撕裂正负数。所以改成 DB-backed gauge。
 
 ## Engagement
 
@@ -48,7 +56,7 @@ OTel SDK 在导出到 Prometheus 时做两件事：
 | `character.created` | Counter | [services/characters.ts](../../src/services/characters.ts) | — |
 | `character.deleted` | Counter | 同上 | — |
 | `character.engagement` | Counter | 同上（like/bookmark） | `action`（`like` / `unlike` / `bookmark` / `unbookmark`） |
-| `ws.connections.active` | UpDownCounter | [routes/chat-ws/index.ts](../../src/routes/chat-ws/index.ts) | — |
+| `ws.connections.active` | ObservableGauge | [routes/chat-ws/index.ts](../../src/routes/chat-ws/index.ts) `addCallback` walks `userConnections` Map | — |
 | `ws.messages.sent` | Counter | 同上 | — |
 | `ws.messages.received` | Counter | [services/chats.ts](../../src/services/chats.ts) | — |
 
@@ -73,12 +81,12 @@ OTel SDK 在导出到 Prometheus 时做两件事：
 |---|---|---|---|
 | `airi.billing.flux.consumed` | Counter | [routes/openai/v1/index.ts](../../src/routes/openai/v1/index.ts) `recordMetrics`（chat / tts） | `gen_ai.request.model`、`gen_ai.operation.name`/`airi.gen_ai.operation.kind`、`http.response.status_code` |
 | `airi.billing.flux.credited` | Counter | [services/billing/billing-service.ts](../../src/services/billing/billing-service.ts) 三条入账路径 | `source`（`stripe.checkout`/`stripe.invoice`/`promo`/`admin_grant`/...）、`type`（`credit`/`promo`） |
-| `airi.billing.flux.unbilled` | Counter | `routes/openai/v1/index.ts` 流式 debit 失败 catch | `gen_ai.request.model`、`reason`（`debit_failed`）、`stage`（`streaming`） |
+| `airi.billing.flux.unbilled` | Counter | [routes/openai/v1/index.ts](../../src/routes/openai/v1/index.ts) streaming 路径里 `consumeFluxForLLM` 失败的 catch | `gen_ai.request.model`、`reason`（`debit_failed`）、`stage`（`streaming`） |
 | `flux.insufficient_balance` | Counter | [services/billing/billing-service.ts](../../src/services/billing/billing-service.ts) `debitFlux` | — |
 | `airi.billing.tts.chars` | Counter | [services/billing/flux-meter.ts](../../src/services/billing/flux-meter.ts) `accumulate` | `meter`（`tts`）、`model` |
 | `airi.billing.tts.preflight_rejections` | Counter | `flux-meter.ts` `assertCanAfford` | `meter`、`reason`（`insufficient_balance`） |
 
-> **`airi.billing.flux.unbilled` 是 P0 告警金线**：任何持续 > 0 都意味着真实收入泄漏，应当 page。语义上等于"流式响应已经发给用户但 DB debit 失败的 Flux 量"。
+> **`airi.billing.flux.unbilled` 是 P0 告警金线**：流式响应已经发给用户（HTTP 200，token 已经流出），但 post-stream debit 抛错——response 路径不会因此 5xx，DB latency 也只在 catch 那一瞬间显著。HTTP / DB 告警**覆盖不到**这条静默 revenue leak。推荐 alert：`increase(airi_billing_flux_unbilled_total[5m]) > 0` 持续 > 0 立刻 page。
 
 ## GenAI
 
@@ -122,36 +130,37 @@ OTel SDK 在导出到 Prometheus 时做两件事：
 
 ## 已落地的 dashboard 行映射
 
-[airi-server-overview-cloud.json](../../otel/grafana/dashboards/airi-server-overview-cloud.json)，从上到下：
+[airi-server-overview-cloud.json](../../otel/grafana/dashboards/airi-server-overview-cloud.json) 由 [`build.ts`](../../otel/grafana/dashboards/build.ts) 生成（**直接改 JSON 会在下次 regenerate 时被覆盖；改 build.ts**），跑 `pnpm -F @proj-airi/server otel:dashboards` 重新生成。从上到下：
 
-| Row | 关键 metric |
-|---|---|
-| HTTP Overview | `http.server.request.duration`（rate / P95 / by route / 5xx 率） |
-| Auth & Users | `auth.attempts` / `auth.failures` / `user.{login,registered,active_sessions}` + 失败率 |
-| Engagement | `ws.connections.active` / `ws.messages.{sent,received}` / `chat.messages` / `character.{created,deleted,engagement}` |
-| Business Metrics | `airi.billing.flux.consumed` / `flux.insufficient_balance` / `gen_ai.client.token.usage.*` / `stripe.checkout.completed` / `airi.billing.flux.credited` |
-| Stripe Detail | `stripe.{events,subscription.event,payment.failed}` / checkout funnel / `airi.stripe.revenue` |
-| Node.js Runtime | runtime instrumentation 那一批 |
-| LLM Gateway | `gen_ai.client.{operation.count,operation.duration,token.usage.*,first_token.duration}` / `airi.billing.flux.consumed` / `airi.billing.flux.unbilled` / `airi.gen_ai.stream.interrupted` / `airi.billing.tts.chars` |
-| Reliability | `airi.email.{send,failures}` 失败率 / `airi.rate_limit.blocked` / `airi.billing.tts.preflight_rejections` |
-| Application Logs | Loki，不是 Prometheus |
+| Row | viz | 关键 metric |
+|---|---|---|
+| Service Health | stat / gauge | `user.active_sessions`（`max()`）、`ws.connections.active`（`sum()`）、`http.server.request.duration_count`（req/s + 5xx%）、`gen_ai.client.operation.count`、`airi.email.{send,failures}` 失败率 |
+| Distribution (now) | donut | HTTP methods / LLM models / HTTP status codes — `increase([5m])` |
+| Traffic Trends | timeseries | 同 distribution 的数据 over time |
+| Latency | timeseries | `http.server.request.duration_bucket`（P95 by route）、`gen_ai.client.first_token.duration_bucket`（P95 by model） |
+| Errors / Quality | mix | 4xx/5xx stacked area、`airi.gen_ai.stream.interrupted`、`airi.rate_limit.blocked` |
+| Business | stat / gauge / donut | `airi.stripe.revenue`（by currency）、checkout conversion %、`stripe.events` 分布 |
+| Infrastructure (collapsed, **by `service_instance_id`**) | timeseries | `db_client_operation_duration` P95（cluster）、`db_client_connection_count`、`v8js_memory_heap_used_bytes` %、`nodejs_eventloop_delay_p99_seconds` |
+| Logs | logs | Loki，不是 Prometheus |
+
+> **Multi-replica 聚合方式**：所有 panel 在 `build.ts` 里都已经按 `observability-conventions.md` 的副本安全表选择了正确的 aggregator（Counter 用 `sum(rate)`、cluster-wide gauge 用 `max()`、per-process gauge 用 `sum()`、infra 排查面板用 `by (service_instance_id)`）。加新 panel 时按那张表对照一遍。
 
 ## 验证 metric 是否已注册
 
-[`src/scripts/otel-smoke.mjs`](../../src/scripts/otel-smoke.mjs) 跑一遍：
+[`src/scripts/otel/smoke.ts`](../../src/scripts/otel/smoke.ts) 跑一遍：
 
 ```sh
-pnpm -F @proj-airi/server exec node --import tsx ./src/scripts/otel-smoke.mjs
+pnpm -F @proj-airi/server exec node --import tsx ./src/scripts/otel/smoke.ts
 ```
 
-会打印 SDK 启动时立即 export 的所有 instrument 名字。**Counter 通过 `.add(0)` priming**（[libs/otel.ts](../../src/libs/otel.ts) `primeCounter`）后会出现在这里 —— Histogram 不会，要等真实 `.record()` 才出现。
+会打印 SDK 启动时立即 export 的所有 instrument 名字。**Counter 通过 `.add(0)` priming**（[otel/index.ts](../../src/otel/index.ts) `primeCounter`）后会出现在这里 —— Histogram 不会，要等真实 `.record()` 才出现。
 
 ## 加新 metric 时的 checklist
 
 1. 决定命名空间：能映射到 OTel semconv 就用标准名，否则放 `airi.*`（不要造新顶级前缀）
 2. 在 [utils/observability.ts](../../src/utils/observability.ts) 加常量
-3. 在 [libs/otel.ts](../../src/libs/otel.ts) 的对应 metric group 接口（`HttpMetrics`/`AuthMetrics`/...）加字段，并在 `initOtel` 里 `meter.create*` 创建
+3. 在 [otel/index.ts](../../src/otel/index.ts) 的对应 metric group 接口（`HttpMetrics`/`AuthMetrics`/...）加字段，并在 `initOtel` 里 `meter.create*` 创建
 4. **如果是 Counter，在 `primeCounter` 调用列表里加一行** —— 否则低流量时 panel 看起来"没数据"
 5. 在 callsite 通过 DI 拿到 metrics 对象后调 `.add()` / `.record()`
-6. 跑 `pnpm -F @proj-airi/server exec node --import tsx ./src/scripts/otel-smoke.mjs` 确认注册
+6. 跑 `pnpm -F @proj-airi/server exec node --import tsx ./src/scripts/otel/smoke.ts` 确认注册
 7. 更新本文档对应章节

apps/server/docs/ai-context/workers-and-runtime.md

@@ -87,7 +87,7 @@
 
 ## OpenTelemetry
 
-初始化在 `src/libs/otel.ts`。
+初始化在 `instrumentation.ts`（NodeSDK lifecycle）+ `src/otel/index.ts`（metric handles）+ `src/otel/gauges/*.ts`（DB-backed ObservableGauge callbacks，例如 `gauges/active-sessions.ts`）。
 
 启用条件：
 

apps/server/instrumentation.ts

@@ -1,7 +1,7 @@
 /**
  * OpenTelemetry preload — single entry point for SDK setup.
  *
- * Loaded via `tsx --import ./instrumentation.mjs`, runs BEFORE any application
+ * Loaded via `tsx --import ./instrumentation.ts`, runs BEFORE any application
  * module is evaluated. By starting NodeSDK here:
  *   - require-in-the-middle hooks for http / pg / ioredis install before app
  *     code does `require('pg')` etc. (fixes the original commit-9451cd7c race).
@@ -24,6 +24,8 @@
 
 import process, { env, exit } from 'node:process'
 
+import { randomUUID } from 'node:crypto'
+
 import { diag, DiagConsoleLogger, DiagLogLevel } from '@opentelemetry/api'
 import { OTLPLogExporter } from '@opentelemetry/exporter-logs-otlp-proto'
 import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-proto'
@@ -65,7 +67,7 @@ else {
     diag.setLogger(new DiagConsoleLogger(), DiagLogLevel.DEBUG)
 
   // OTEL_EXPORTER_OTLP_HEADERS format: "key=value,key2=value2"
-  const headers = {}
+  const headers: Record<string, string> = {}
   for (const pair of (env.OTEL_EXPORTER_OTLP_HEADERS ?? '').split(',')) {
     const idx = pair.indexOf('=')
     if (idx > 0)
@@ -80,10 +82,35 @@ else {
     ? samplingRatioRaw
     : 1
 
+  // service.instance.id MUST be unique per replica. Without it, two replicas
+  // emit the same (service_name, deployment_environment) label tuple — when
+  // an OTel collector / Prometheus receives both, it can either drop one
+  // sample as a "duplicate timestamp" or collapse the series outright,
+  // making per-replica `sum()` aggregates undercount.
+  //
+  // Source preference, strongest → weakest:
+  //  1. RAILWAY_REPLICA_ID — Railway-managed, guaranteed unique per replica.
+  //  2. SERVER_INSTANCE_ID — operator-supplied override.
+  //  3. randomUUID() — per-process fallback. Logged as a warning so ops
+  //     know we're relying on a value that doesn't survive restarts (i.e.
+  //     metric series cardinality climbs every deploy until staleness
+  //     evicts old instance ids).
+  //
+  // HOSTNAME was previously used as a fallback but Railway's HOSTNAME
+  // semantics aren't documented as per-replica unique, so we no longer
+  // trust it. If you need to pin instance id to something stable across
+  // restarts, set SERVER_INSTANCE_ID explicitly.
+  let instanceId = env.RAILWAY_REPLICA_ID || env.SERVER_INSTANCE_ID
+  if (!instanceId) {
+    instanceId = randomUUID()
+    console.warn(`[otel-preload] No RAILWAY_REPLICA_ID or SERVER_INSTANCE_ID set — falling back to randomUUID() ${instanceId}. Multi-replica metric series will churn on every restart.`)
+  }
+
   const resource = resourceFromAttributes({
     [ATTR_SERVICE_NAME]: serviceName,
     [ATTR_SERVICE_VERSION]: env.npm_package_version || '0.0.0',
     'service.namespace': serviceNamespace,
+    'service.instance.id': instanceId,
     'deployment.environment': env.NODE_ENV || 'development',
   })
 

apps/server/otel/grafana/dashboards/airi-server-overview-cloud.json

@@ -41,7 +41,7 @@
                     "group": "prometheus",
                     "kind": "DataQuery",
                     "spec": {
-                      "expr": "sum(user_active_sessions{service_name=~\"$service\", deployment_environment=~\"$env\"})",
+                      "expr": "avg(user_active_sessions{service_name=~\"$service\", deployment_environment=~\"$env\"})",
                       "legendFormat": "sessions"
                     },
                     "version": "v0"
@@ -54,7 +54,7 @@
             "transformations": []
           }
         },
-        "description": "Currently authenticated user sessions across all instances. Stale sessions expire by Better Auth TTL.",
+        "description": "Currently active sessions in Postgres (Better Auth `session.expires_at > now()`). Cluster-wide gauge — every replica polls the same DB on a 10s cache. We aggregate with `avg()` (not `sum()`, which would multiply by replica count; not `max()`, which biases high when one replica's cache is fresher than another's after a logout).",
         "id": 1,
         "links": [],
         "title": "Active Users",
@@ -1606,6 +1606,90 @@
         }
       }
     },
+    "panel-43": {
+      "kind": "Panel",
+      "spec": {
+        "data": {
+          "kind": "QueryGroup",
+          "spec": {
+            "queries": [
+              {
+                "kind": "PanelQuery",
+                "spec": {
+                  "hidden": false,
+                  "query": {
+                    "datasource": {
+                      "name": "grafanacloud-projairi-prom"
+                    },
+                    "group": "prometheus",
+                    "kind": "DataQuery",
+                    "spec": {
+                      "expr": "sum(increase(airi_billing_flux_unbilled_total{service_name=~\"$service\", deployment_environment=~\"$env\"}[$__range]))",
+                      "legendFormat": "flux"
+                    },
+                    "version": "v0"
+                  },
+                  "refId": "A"
+                }
+              }
+            ],
+            "queryOptions": {},
+            "transformations": []
+          }
+        },
+        "description": "Flux value owed by users but never debited (post-stream debit failed AFTER the LLM response was already sent). Real revenue leak — DB latency and HTTP 5xx alerts do NOT cover this, because the response was 2xx and the catch path is silent. Any sustained >0 should page on-call.",
+        "id": 43,
+        "links": [],
+        "title": "⚠ Flux Unbilled (range)",
+        "vizConfig": {
+          "group": "stat",
+          "kind": "VizConfig",
+          "spec": {
+            "fieldConfig": {
+              "defaults": {
+                "color": {
+                  "mode": "thresholds"
+                },
+                "thresholds": {
+                  "mode": "absolute",
+                  "steps": [
+                    {
+                      "color": "green",
+                      "value": 0
+                    },
+                    {
+                      "color": "red",
+                      "value": 1
+                    }
+                  ]
+                },
+                "unit": "short",
+                "noValue": "0"
+              },
+              "overrides": []
+            },
+            "options": {
+              "colorMode": "value",
+              "graphMode": "none",
+              "justifyMode": "auto",
+              "orientation": "auto",
+              "percentChangeColorMode": "standard",
+              "reduceOptions": {
+                "calcs": [
+                  "lastNotNull"
+                ],
+                "fields": "",
+                "values": false
+              },
+              "showPercentChange": false,
+              "textMode": "auto",
+              "wideLayout": true
+            }
+          },
+          "version": "13.0.0-23630096546"
+        }
+      }
+    },
     "panel-42": {
       "kind": "Panel",
       "spec": {
@@ -1637,7 +1721,7 @@
             "transformations": []
           }
         },
-        "description": "Requests blocked by the in-memory rate limiter, by route + key type. Sustained activity here is either an attack or a misconfigured client.",
+        "description": "Requests blocked by the in-memory rate limiter, by route + key type. NOTE: limiter is in-memory per replica (`apps/server/src/middlewares/rate-limit.ts`), so the configured limit applies independently on each pod — effective cluster-wide allowance is roughly `limit × replica_count`. The values here are absolute blocks summed across replicas, not a percentage of capacity. Sustained activity = attack, misconfigured client, or limit-too-low for current traffic.",
         "id": 42,
         "links": [],
         "title": "Rate-Limit Blocks",
@@ -2089,8 +2173,8 @@
                     "group": "prometheus",
                     "kind": "DataQuery",
                     "spec": {
-                      "expr": "sum(db_client_connection_count{service_name=~\"$service\", deployment_environment=~\"$env\"})",
-                      "legendFormat": "open"
+                      "expr": "sum by (service_instance_id) (db_client_connection_count{service_name=~\"$service\", deployment_environment=~\"$env\"})",
+                      "legendFormat": "{{service_instance_id}}"
                     },
                     "version": "v0"
                   },
@@ -2102,18 +2186,52 @@
             "transformations": []
           }
         },
-        "description": "Open PostgreSQL connections across all instances. Compare to env DB_POOL_MAX × instance count.",
+        "description": "Open PostgreSQL connections, broken down per replica (`service_instance_id`). Each instance has its own pool sized by env `DB_POOL_MAX`. One instance with a permanently-high count = pool leak on that pod.",
         "id": 51,
         "links": [],
-        "title": "DB Pool Connections",
+        "title": "DB Pool Connections by Instance",
         "vizConfig": {
-          "group": "stat",
+          "group": "timeseries",
           "kind": "VizConfig",
           "spec": {
             "fieldConfig": {
               "defaults": {
                 "color": {
-                  "mode": "thresholds"
+                  "mode": "palette-classic"
+                },
+                "custom": {
+                  "axisBorderShow": false,
+                  "axisCenteredZero": false,
+                  "axisColorMode": "text",
+                  "axisLabel": "",
+                  "axisPlacement": "auto",
+                  "barAlignment": 0,
+                  "barWidthFactor": 0.6,
+                  "drawStyle": "line",
+                  "fillOpacity": 20,
+                  "gradientMode": "none",
+                  "hideFrom": {
+                    "legend": false,
+                    "tooltip": false,
+                    "viz": false
+                  },
+                  "insertNulls": false,
+                  "lineInterpolation": "smooth",
+                  "lineWidth": 1,
+                  "pointSize": 5,
+                  "scaleDistribution": {
+                    "type": "linear"
+                  },
+                  "showPoints": "auto",
+                  "showValues": false,
+                  "spanNulls": false,
+                  "stacking": {
+                    "group": "A",
+                    "mode": "none"
+                  },
+                  "thresholdsStyle": {
+                    "mode": "off"
+                  }
                 },
                 "thresholds": {
                   "mode": "absolute",
@@ -2129,21 +2247,24 @@
               "overrides": []
             },
             "options": {
-              "colorMode": "value",
-              "graphMode": "area",
-              "justifyMode": "auto",
-              "orientation": "auto",
-              "percentChangeColorMode": "standard",
-              "reduceOptions": {
-                "calcs": [
-                  "lastNotNull"
-                ],
-                "fields": "",
-                "values": false
+              "annotations": {
+                "clustering": -1,
+                "multiLane": false
               },
-              "showPercentChange": false,
-              "textMode": "auto",
-              "wideLayout": true
+              "legend": {
+                "calcs": [
+                  "lastNotNull",
+                  "max"
+                ],
+                "displayMode": "table",
+                "placement": "right",
+                "showLegend": true
+              },
+              "tooltip": {
+                "hideZeros": false,
+                "mode": "multi",
+                "sort": "desc"
+              }
             }
           },
           "version": "13.0.0-23630096546"
@@ -2168,8 +2289,8 @@
                     "group": "prometheus",
                     "kind": "DataQuery",
                     "spec": {
-                      "expr": "100 * sum(v8js_memory_heap_used_bytes{service_name=~\"$service\", deployment_environment=~\"$env\"}) / clamp_min(sum(v8js_memory_heap_limit_bytes{service_name=~\"$service\", deployment_environment=~\"$env\"}), 1)",
-                      "legendFormat": "used %"
+                      "expr": "100 * sum by (service_instance_id) (v8js_memory_heap_used_bytes{service_name=~\"$service\", deployment_environment=~\"$env\"}) / clamp_min(sum by (service_instance_id) (v8js_memory_heap_limit_bytes{service_name=~\"$service\", deployment_environment=~\"$env\"}), 1)",
+                      "legendFormat": "{{service_instance_id}}"
                     },
                     "version": "v0"
                   },
@@ -2181,18 +2302,52 @@
             "transformations": []
           }
         },
-        "description": "V8 heap used ÷ heap limit. Sustained >85% = consider raising memory or hunting a leak.",
+        "description": "V8 heap used ÷ heap limit, per replica (`service_instance_id`). A single replica trending up while others stay flat = leak on that pod. Cluster-wide average masks that — show by instance.",
         "id": 52,
         "links": [],
-        "title": "Heap Used %",
+        "title": "Heap Used % by Instance",
         "vizConfig": {
-          "group": "gauge",
+          "group": "timeseries",
           "kind": "VizConfig",
           "spec": {
             "fieldConfig": {
               "defaults": {
                 "color": {
-                  "mode": "thresholds"
+                  "mode": "palette-classic"
+                },
+                "custom": {
+                  "axisBorderShow": false,
+                  "axisCenteredZero": false,
+                  "axisColorMode": "text",
+                  "axisLabel": "",
+                  "axisPlacement": "auto",
+                  "barAlignment": 0,
+                  "barWidthFactor": 0.6,
+                  "drawStyle": "line",
+                  "fillOpacity": 20,
+                  "gradientMode": "none",
+                  "hideFrom": {
+                    "legend": false,
+                    "tooltip": false,
+                    "viz": false
+                  },
+                  "insertNulls": false,
+                  "lineInterpolation": "smooth",
+                  "lineWidth": 1,
+                  "pointSize": 5,
+                  "scaleDistribution": {
+                    "type": "linear"
+                  },
+                  "showPoints": "auto",
+                  "showValues": false,
+                  "spanNulls": false,
+                  "stacking": {
+                    "group": "A",
+                    "mode": "none"
+                  },
+                  "thresholdsStyle": {
+                    "mode": "off"
+                  }
                 },
                 "thresholds": {
                   "mode": "absolute",
@@ -2200,38 +2355,32 @@
                     {
                       "color": "green",
                       "value": 0
-                    },
-                    {
-                      "color": "yellow",
-                      "value": 70
-                    },
-                    {
-                      "color": "red",
-                      "value": 90
                     }
                   ]
                 },
-                "unit": "percent",
-                "decimals": 1,
-                "min": 0,
-                "max": 100
+                "unit": "percent"
               },
               "overrides": []
             },
             "options": {
-              "minVizHeight": 75,
-              "minVizWidth": 75,
-              "orientation": "auto",
-              "reduceOptions": {
-                "calcs": [
-                  "lastNotNull"
-                ],
-                "fields": "",
-                "values": false
+              "annotations": {
+                "clustering": -1,
+                "multiLane": false
               },
-              "showThresholdLabels": false,
-              "showThresholdMarkers": true,
-              "sizing": "auto"
+              "legend": {
+                "calcs": [
+                  "lastNotNull",
+                  "max"
+                ],
+                "displayMode": "table",
+                "placement": "right",
+                "showLegend": true
+              },
+              "tooltip": {
+                "hideZeros": false,
+                "mode": "multi",
+                "sort": "desc"
+              }
             }
           },
           "version": "13.0.0-23630096546"
@@ -2256,8 +2405,8 @@
                     "group": "prometheus",
                     "kind": "DataQuery",
                     "spec": {
-                      "expr": "max(nodejs_eventloop_delay_p99_seconds{service_name=~\"$service\", deployment_environment=~\"$env\"})",
-                      "legendFormat": "p99"
+                      "expr": "max by (service_instance_id) (nodejs_eventloop_delay_p99_seconds{service_name=~\"$service\", deployment_environment=~\"$env\"})",
+                      "legendFormat": "{{service_instance_id}}"
                     },
                     "version": "v0"
                   },
@@ -2269,18 +2418,52 @@
             "transformations": []
           }
         },
-        "description": "P99 event-loop delay. >50ms means CPU-bound work (sync JSON parsing, CPU-heavy regex) is blocking the loop.",
+        "description": "P99 event-loop delay per replica. One replica climbing while others stay flat = CPU-bound work pinning that pod. >50ms sustained is bad anywhere.",
         "id": 53,
         "links": [],
-        "title": "Event Loop Delay P99",
+        "title": "Event Loop Delay P99 by Instance",
         "vizConfig": {
-          "group": "stat",
+          "group": "timeseries",
           "kind": "VizConfig",
           "spec": {
             "fieldConfig": {
               "defaults": {
                 "color": {
-                  "mode": "thresholds"
+                  "mode": "palette-classic"
+                },
+                "custom": {
+                  "axisBorderShow": false,
+                  "axisCenteredZero": false,
+                  "axisColorMode": "text",
+                  "axisLabel": "",
+                  "axisPlacement": "auto",
+                  "barAlignment": 0,
+                  "barWidthFactor": 0.6,
+                  "drawStyle": "line",
+                  "fillOpacity": 20,
+                  "gradientMode": "none",
+                  "hideFrom": {
+                    "legend": false,
+                    "tooltip": false,
+                    "viz": false
+                  },
+                  "insertNulls": false,
+                  "lineInterpolation": "smooth",
+                  "lineWidth": 1,
+                  "pointSize": 5,
+                  "scaleDistribution": {
+                    "type": "linear"
+                  },
+                  "showPoints": "auto",
+                  "showValues": false,
+                  "spanNulls": false,
+                  "stacking": {
+                    "group": "A",
+                    "mode": "none"
+                  },
+                  "thresholdsStyle": {
+                    "mode": "off"
+                  }
                 },
                 "thresholds": {
                   "mode": "absolute",
@@ -2288,38 +2471,32 @@
                     {
                       "color": "green",
                       "value": 0
-                    },
-                    {
-                      "color": "yellow",
-                      "value": 0.05
-                    },
-                    {
-                      "color": "red",
-                      "value": 0.2
                     }
                   ]
                 },
-                "unit": "s",
-                "decimals": 3
+                "unit": "s"
               },
               "overrides": []
             },
             "options": {
-              "colorMode": "value",
-              "graphMode": "area",
-              "justifyMode": "auto",
-              "orientation": "auto",
-              "percentChangeColorMode": "standard",
-              "reduceOptions": {
-                "calcs": [
-                  "lastNotNull"
-                ],
-                "fields": "",
-                "values": false
+              "annotations": {
+                "clustering": -1,
+                "multiLane": false
               },
-              "showPercentChange": false,
-              "textMode": "auto",
-              "wideLayout": true
+              "legend": {
+                "calcs": [
+                  "lastNotNull",
+                  "max"
+                ],
+                "displayMode": "table",
+                "placement": "right",
+                "showLegend": true
+              },
+              "tooltip": {
+                "hideZeros": false,
+                "mode": "multi",
+                "sort": "desc"
+              }
             }
           },
           "version": "13.0.0-23630096546"
@@ -2663,11 +2840,24 @@
                         "name": "panel-41"
                       },
                       "height": 7,
-                      "width": 6,
+                      "width": 4,
                       "x": 10,
                       "y": 0
                     }
                   },
+                  {
+                    "kind": "GridLayoutItem",
+                    "spec": {
+                      "element": {
+                        "kind": "ElementReference",
+                        "name": "panel-43"
+                      },
+                      "height": 7,
+                      "width": 4,
+                      "x": 14,
+                      "y": 0
+                    }
+                  },
                   {
                     "kind": "GridLayoutItem",
                     "spec": {
@@ -2676,8 +2866,8 @@
                         "name": "panel-42"
                       },
                       "height": 7,
-                      "width": 8,
-                      "x": 16,
+                      "width": 6,
+                      "x": 18,
                       "y": 0
                     }
                   }
@@ -2755,7 +2945,7 @@
                         "kind": "ElementReference",
                         "name": "panel-50"
                       },
-                      "height": 4,
+                      "height": 6,
                       "width": 6,
                       "x": 0,
                       "y": 0
@@ -2768,7 +2958,7 @@
                         "kind": "ElementReference",
                         "name": "panel-51"
                       },
-                      "height": 4,
+                      "height": 6,
                       "width": 6,
                       "x": 6,
                       "y": 0
@@ -2781,7 +2971,7 @@
                         "kind": "ElementReference",
                         "name": "panel-52"
                       },
-                      "height": 4,
+                      "height": 6,
                       "width": 6,
                       "x": 12,
                       "y": 0
@@ -2794,7 +2984,7 @@
                         "kind": "ElementReference",
                         "name": "panel-53"
                       },
-                      "height": 4,
+                      "height": 6,
                       "width": 6,
                       "x": 18,
                       "y": 0

apps/server/otel/grafana/dashboards/build.ts

@@ -1,7 +1,8 @@
 /**
  * Dashboard generator for `airi-server-overview-cloud.json`.
  *
- * Run: `node apps/server/otel/grafana/dashboards/build.mjs`
+ * Run: `pnpm -F @proj-airi/server otel:dashboards`
+ *  (or directly: `pnpm exec tsx apps/server/otel/grafana/dashboards/build.ts`)
  *
  * Why a generator instead of hand-edited JSON: the dashboard's Grafana v2
  * schema is verbose (~50 lines per panel). Rebuilding the file by hand every
@@ -35,7 +36,14 @@ const SCHEMA_VERSION = '13.0.0-23630096546'
 // the variable name only appears once.
 const SERVICE_FILTER = 'service_name=~"$service", deployment_environment=~"$env"'
 
-function query(expr, legend, refId = 'A', datasource = PROM) {
+// Build-script local types. Kept loose — Grafana owns the schema, and we
+// validate the rendered JSON by re-importing it into Grafana, not by typing.
+type DataSource = typeof PROM | typeof LOKI
+interface ThresholdStep { color: string, value: number }
+type PanelQuery = ReturnType<typeof query>
+type LegendCalc = 'lastNotNull' | 'max' | 'min' | 'mean' | 'sum'
+
+function query(expr: string, legend: string, refId = 'A', datasource: DataSource = PROM) {
   return {
     kind: 'PanelQuery',
     spec: {
@@ -52,14 +60,52 @@ function query(expr, legend, refId = 'A', datasource = PROM) {
   }
 }
 
-function thresholds(steps) {
+function thresholds(steps: ThresholdStep[]) {
   return { mode: 'absolute', steps }
 }
 
+interface DefaultsBlockOpts {
+  unit: string
+  steps: ThresholdStep[]
+  decimals?: number
+  noValue?: string
+  min?: number
+  max?: number
+}
+
+interface StatPanelOpts {
+  unit?: string
+  steps?: ThresholdStep[]
+  decimals?: number
+  noValue?: string
+  graphMode?: 'area' | 'none'
+}
+
+interface GaugePanelOpts {
+  unit?: string
+  steps: ThresholdStep[]
+  decimals?: number
+  min?: number
+  max?: number
+  noValue?: string
+}
+
+interface PiePanelOpts {
+  unit?: string
+  noValue?: string
+}
+
+interface TimeseriesPanelOpts {
+  unit?: string
+  stack?: boolean
+  fillOpacity?: number
+  legendCalcs?: LegendCalc[]
+}
+
 // `noValue` shows a friendly placeholder instead of "No data" red text when
 // the env genuinely has zero traffic (e.g. dev, fresh deploy). Empty-string
 // fields are omitted from the JSON to keep diffs tidy.
-function defaultsBlock({ unit, steps, decimals, noValue, min, max }) {
+function defaultsBlock({ unit, steps, decimals, noValue, min, max }: DefaultsBlockOpts) {
   return {
     color: { mode: 'thresholds' },
     thresholds: thresholds(steps),
@@ -71,7 +117,7 @@ function defaultsBlock({ unit, steps, decimals, noValue, min, max }) {
   }
 }
 
-function statPanel(id, title, description, queries, opts = {}) {
+function statPanel(id: number, title: string, description: string, queries: PanelQuery[], opts: StatPanelOpts = {}) {
   const { unit = 'short', steps = [{ color: 'green', value: 0 }], decimals, noValue, graphMode = 'area' } = opts
   return {
     kind: 'Panel',
@@ -107,7 +153,7 @@ function statPanel(id, title, description, queries, opts = {}) {
 // Bounded ratio with traffic-light thresholds. Use for percent or capacity
 // metrics; the radial fill instantly conveys "OK / warn / critical" without
 // reading the number.
-function gaugePanel(id, title, description, queries, opts = {}) {
+function gaugePanel(id: number, title: string, description: string, queries: PanelQuery[], opts: GaugePanelOpts) {
   const { unit = 'percent', steps, decimals = 1, min = 0, max = 100, noValue } = opts
   return {
     kind: 'Panel',
@@ -141,7 +187,7 @@ function gaugePanel(id, title, description, queries, opts = {}) {
 // Donut for distribution-at-a-glance. Each query result becomes a slice;
 // percentages render automatically. Use over stacked-area when the question
 // is "what's the current breakdown" rather than "how is it changing".
-function piePanel(id, title, description, queries, opts = {}) {
+function piePanel(id: number, title: string, description: string, queries: PanelQuery[], opts: PiePanelOpts = {}) {
   const { unit = 'short', noValue = 'no traffic' } = opts
   return {
     kind: 'Panel',
@@ -184,7 +230,7 @@ function piePanel(id, title, description, queries, opts = {}) {
   }
 }
 
-function timeseriesPanel(id, title, description, queries, opts = {}) {
+function timeseriesPanel(id: number, title: string, description: string, queries: PanelQuery[], opts: TimeseriesPanelOpts = {}) {
   const { unit = 'short', stack = false, fillOpacity = 20, legendCalcs = ['lastNotNull', 'max'] } = opts
   return {
     kind: 'Panel',
@@ -244,7 +290,7 @@ function timeseriesPanel(id, title, description, queries, opts = {}) {
   }
 }
 
-function logsPanel(id, title, description, expr) {
+function logsPanel(id: number, title: string, description: string, expr: string) {
   return {
     kind: 'Panel',
     spec: {
@@ -282,11 +328,11 @@ function logsPanel(id, title, description, expr) {
   }
 }
 
-function item(name, x, y, width, height) {
+function item(name: string, x: number, y: number, width: number, height: number) {
   return { kind: 'GridLayoutItem', spec: { element: { kind: 'ElementReference', name }, height, width, x, y } }
 }
 
-function row(title, items, { collapse = false } = {}) {
+function row(title: string, items: ReturnType<typeof item>[], { collapse = false }: { collapse?: boolean } = {}) {
   return {
     kind: 'RowsLayoutRow',
     spec: {
@@ -301,15 +347,21 @@ function row(title, items, { collapse = false } = {}) {
 // Panels
 // ---------------------------------------------------------------------------
 
-const elements = {}
+// Grafana v2 element entries are opaque to us — each helper returns a Panel
+// shape with deeply-nested fieldConfig/options that we don't statically type
+// (Grafana owns that schema, and any drift would surface at dashboard import
+// time, not compile time). Treat `elements` as a string-keyed bag of
+// `unknown`-shaped panel JSON; the cross-check below catches mismatches
+// between defined panel ids and layout references.
+const elements: Record<string, unknown> = {}
 
 // Row 1: Service Health — answers "is anything broken right now?"
 // Mix of stats (absolute counts) and gauges (bounded ratios with thresholds).
 elements['panel-1'] = statPanel(
   1,
   'Active Users',
-  'Currently authenticated user sessions across all instances. Stale sessions expire by Better Auth TTL.',
-  [query(`sum(user_active_sessions{${SERVICE_FILTER}})`, 'sessions')],
+  'Currently active sessions in Postgres (Better Auth `session.expires_at > now()`). Cluster-wide gauge — every replica polls the same DB on a 10s cache. We aggregate with `avg()` (not `sum()`, which would multiply by replica count; not `max()`, which biases high when one replica\'s cache is fresher than another\'s after a logout).',
+  [query(`avg(user_active_sessions{${SERVICE_FILTER}})`, 'sessions')],
   { unit: 'short', steps: [{ color: 'green', value: 0 }, { color: 'yellow', value: 1000 }] },
 )
 
@@ -473,10 +525,21 @@ elements['panel-41'] = statPanel(
   { unit: 'short', steps: [{ color: 'green', value: 0 }, { color: 'yellow', value: 1 }, { color: 'red', value: 10 }], noValue: '0', graphMode: 'none' },
 )
 
+elements['panel-43'] = statPanel(
+  43,
+  '⚠ Flux Unbilled (range)',
+  'Flux value owed by users but never debited (post-stream debit failed AFTER the LLM response was already sent). Real revenue leak — DB latency and HTTP 5xx alerts do NOT cover this, because the response was 2xx and the catch path is silent. Any sustained >0 should page on-call.',
+  [query(
+    `sum(increase(airi_billing_flux_unbilled_total{${SERVICE_FILTER}}[$__range]))`,
+    'flux',
+  )],
+  { unit: 'short', steps: [{ color: 'green', value: 0 }, { color: 'red', value: 1 }], noValue: '0', graphMode: 'none' },
+)
+
 elements['panel-42'] = timeseriesPanel(
   42,
   'Rate-Limit Blocks',
-  'Requests blocked by the in-memory rate limiter, by route + key type. Sustained activity here is either an attack or a misconfigured client.',
+  'Requests blocked by the in-memory rate limiter, by route + key type. NOTE: limiter is in-memory per replica (`apps/server/src/middlewares/rate-limit.ts`), so the configured limit applies independently on each pod — effective cluster-wide allowance is roughly `limit × replica_count`. The values here are absolute blocks summed across replicas, not a percentage of capacity. Sustained activity = attack, misconfigured client, or limit-too-low for current traffic.',
   [query(
     `sum by (route, key_type) (rate(airi_rate_limit_blocked_total{${SERVICE_FILTER}}[$__rate_interval]))`,
     '{{route}} ({{key_type}})',
@@ -530,31 +593,37 @@ elements['panel-50'] = statPanel(
   { unit: 's', steps: [{ color: 'green', value: 0 }, { color: 'yellow', value: 0.05 }, { color: 'red', value: 0.5 }], decimals: 3 },
 )
 
-elements['panel-51'] = statPanel(
+elements['panel-51'] = timeseriesPanel(
   51,
-  'DB Pool Connections',
-  'Open PostgreSQL connections across all instances. Compare to env DB_POOL_MAX × instance count.',
-  [query(`sum(db_client_connection_count{${SERVICE_FILTER}})`, 'open')],
+  'DB Pool Connections by Instance',
+  'Open PostgreSQL connections, broken down per replica (`service_instance_id`). Each instance has its own pool sized by env `DB_POOL_MAX`. One instance with a permanently-high count = pool leak on that pod.',
+  [query(
+    `sum by (service_instance_id) (db_client_connection_count{${SERVICE_FILTER}})`,
+    '{{service_instance_id}}',
+  )],
   { unit: 'short' },
 )
 
-elements['panel-52'] = gaugePanel(
+elements['panel-52'] = timeseriesPanel(
   52,
-  'Heap Used %',
-  'V8 heap used ÷ heap limit. Sustained >85% = consider raising memory or hunting a leak.',
+  'Heap Used % by Instance',
+  'V8 heap used ÷ heap limit, per replica (`service_instance_id`). A single replica trending up while others stay flat = leak on that pod. Cluster-wide average masks that — show by instance.',
   [query(
-    `100 * sum(v8js_memory_heap_used_bytes{${SERVICE_FILTER}}) / clamp_min(sum(v8js_memory_heap_limit_bytes{${SERVICE_FILTER}}), 1)`,
-    'used %',
+    `100 * sum by (service_instance_id) (v8js_memory_heap_used_bytes{${SERVICE_FILTER}}) / clamp_min(sum by (service_instance_id) (v8js_memory_heap_limit_bytes{${SERVICE_FILTER}}), 1)`,
+    '{{service_instance_id}}',
   )],
-  { steps: [{ color: 'green', value: 0 }, { color: 'yellow', value: 70 }, { color: 'red', value: 90 }], decimals: 1 },
+  { unit: 'percent' },
 )
 
-elements['panel-53'] = statPanel(
+elements['panel-53'] = timeseriesPanel(
   53,
-  'Event Loop Delay P99',
-  'P99 event-loop delay. >50ms means CPU-bound work (sync JSON parsing, CPU-heavy regex) is blocking the loop.',
-  [query(`max(nodejs_eventloop_delay_p99_seconds{${SERVICE_FILTER}})`, 'p99')],
-  { unit: 's', steps: [{ color: 'green', value: 0 }, { color: 'yellow', value: 0.05 }, { color: 'red', value: 0.2 }], decimals: 3 },
+  'Event Loop Delay P99 by Instance',
+  'P99 event-loop delay per replica. One replica climbing while others stay flat = CPU-bound work pinning that pod. >50ms sustained is bad anywhere.',
+  [query(
+    `max by (service_instance_id) (nodejs_eventloop_delay_p99_seconds{${SERVICE_FILTER}})`,
+    '{{service_instance_id}}',
+  )],
+  { unit: 's' },
 )
 
 // Row 8: Logs
@@ -596,11 +665,15 @@ const rows = [
     item('panel-20', 0, 0, 12, 8),
     item('panel-21', 12, 0, 12, 8),
   ]),
-  // Row 5: 1 stacked area + 1 stat + 1 timeseries × 8 high
+  // Row 5: 1 stacked area + 2 stats + 1 timeseries × 7 high
+  // Stream Interruptions and ⚠ Flux Unbilled sit next to the 4xx/5xx trend
+  // so revenue-leak signal (which doesn't show up in 5xx) gets the same
+  // glance-weight as transport-layer errors.
   row('Errors / Quality', [
     item('panel-40', 0, 0, 10, 7),
-    item('panel-41', 10, 0, 6, 7),
-    item('panel-42', 16, 0, 8, 7),
+    item('panel-41', 10, 0, 4, 7),
+    item('panel-43', 14, 0, 4, 7),
+    item('panel-42', 18, 0, 6, 7),
   ]),
   // Row 6: 1 stat + 1 gauge + 1 donut × 8 wide × 7 high
   row('Business', [
@@ -608,12 +681,14 @@ const rows = [
     item('panel-31', 8, 0, 8, 7),
     item('panel-32', 16, 0, 8, 7),
   ]),
-  // Row 7: 4 panels × 6 wide × 4 high (collapsed by default — only relevant when troubleshooting)
+  // Row 7: 1 stat + 3 by-instance timeseries × 6 wide × 6 high (collapsed by
+  // default — only relevant when triaging. By-instance breakdowns catch
+  // single-replica issues that cluster aggregates would average away.)
   row('Infrastructure', [
-    item('panel-50', 0, 0, 6, 4),
-    item('panel-51', 6, 0, 6, 4),
-    item('panel-52', 12, 0, 6, 4),
-    item('panel-53', 18, 0, 6, 4),
+    item('panel-50', 0, 0, 6, 6),
+    item('panel-51', 6, 0, 6, 6),
+    item('panel-52', 12, 0, 6, 6),
+    item('panel-53', 18, 0, 6, 6),
   ], { collapse: true }),
   // Row 8: full-width logs
   row('Logs', [
@@ -752,12 +827,13 @@ console.info(`wrote ${outPath}`)
 
 // Cross-check elements ↔ layout references
 const elementNames = new Set(Object.keys(dashboard.elements))
-const refs = new Set()
-function walk(o) {
+const refs = new Set<string>()
+function walk(o: unknown): void {
   if (!o || typeof o !== 'object')
     return
-  if (o.kind === 'ElementReference' && o.name)
-    refs.add(o.name)
+  const node = o as { kind?: unknown, name?: unknown }
+  if (node.kind === 'ElementReference' && typeof node.name === 'string')
+    refs.add(node.name)
   for (const v of Object.values(o)) walk(v)
 }
 walk(dashboard.layout)

apps/server/package.json

@@ -6,13 +6,14 @@
   "scripts": {
     "apply:env": "dotenvx run -f .env.local --overload --ignore=MISSING_ENV_FILE",
     "auth:generate": "pnpm run apply:env -- better-auth generate --config src/scripts/auth.ts --output src/schemas/accounts.ts -y",
-    "dev": "pnpm run apply:env -- tsx --import ./instrumentation.mjs --watch src/bin/run.ts api",
-    "start": "pnpm run apply:env -- tsx --import ./instrumentation.mjs src/bin/run.ts api",
-    "server": "pnpm run apply:env -- tsx --import ./instrumentation.mjs src/bin/run.ts",
+    "dev": "pnpm run apply:env -- tsx --import ./instrumentation.ts --watch src/bin/run.ts api",
+    "start": "pnpm run apply:env -- tsx --import ./instrumentation.ts src/bin/run.ts api",
+    "server": "pnpm run apply:env -- tsx --import ./instrumentation.ts src/bin/run.ts",
     "build": "tsc -b",
     "typecheck": "tsc --noEmit",
     "db:generate": "drizzle-kit generate",
-    "db:push": "pnpm run apply:env -- drizzle-kit push"
+    "db:push": "pnpm run apply:env -- drizzle-kit push",
+    "otel:dashboards": "tsx otel/grafana/dashboards/build.ts"
   },
   "dependencies": {
     "@better-auth/drizzle-adapter": "^1.6.5",

apps/server/src/app.ts

@@ -3,7 +3,7 @@ import type Redis from 'ioredis'
 import type { AuthInstance } from './libs/auth'
 import type { Database } from './libs/db'
 import type { Env } from './libs/env'
-import type { OtelInstance } from './libs/otel'
+import type { OtelInstance } from './otel'
 import type { AdminFluxGrantsService } from './services/admin-flux-grants'
 import type { BillingService } from './services/billing/billing-service'
 import type { FluxMeter } from './services/billing/flux-meter'
@@ -36,10 +36,11 @@ import { createAuth, getTrustedClientSeedSummaries, seedTrustedClients } from '.
 import { createDrizzle, migrateDatabase } from './libs/db'
 import { parsedEnv } from './libs/env'
 import { initializeExternalDependency } from './libs/external-dependency'
-import { emitOtelLog, initOtel } from './libs/otel'
 import { createRedis } from './libs/redis'
 import { resolveRequestAuth } from './libs/request-auth'
 import { sessionMiddleware } from './middlewares/auth'
+import { emitOtelLog, initOtel } from './otel'
+import { registerActiveSessionsGauge } from './otel/gauges/active-sessions'
 import { createAdminFluxGrantsRoutes } from './routes/admin/flux-grants'
 import { createAuthRoutes } from './routes/auth'
 import { createCharacterRoutes } from './routes/characters'
@@ -227,7 +228,7 @@ export async function buildApp(deps: AppDeps) {
     /**
      * V1 routes for official provider.
      */
-    .route('/api/v1/openai', createV1CompletionsRoutes(deps.fluxService, deps.billingService, deps.configKV, deps.requestLogService, deps.ttsMeter, deps.redis, deps.env, deps.otel?.genAi, deps.otel?.rateLimit))
+    .route('/api/v1/openai', createV1CompletionsRoutes(deps.fluxService, deps.billingService, deps.configKV, deps.requestLogService, deps.ttsMeter, deps.redis, deps.env, deps.otel?.genAi, deps.otel?.revenue, deps.otel?.rateLimit))
 
     /**
      * Flux routes.
@@ -272,7 +273,7 @@ export async function createApp() {
   })
 
   // NOTICE: OTel SDK lifecycle (start/shutdown) is owned entirely by
-  // instrumentation.mjs (preload). This factory only consumes the global
+  // instrumentation.ts (preload). This factory only consumes the global
   // MeterProvider that the preload set up, builds metric handles, and primes
   // counters. No `lifecycle.onStop(shutdown)` here — preload registers SIGTERM
   // / SIGINT to flush exporters on its own.
@@ -484,6 +485,12 @@ export async function createApp() {
     otel,
     userDeletionService,
   })
+  // Register the cluster-wide ObservableGauge for active sessions. Each
+  // replica polls the same DB (cached 10s, in-flight coalesced) and the
+  // dashboard aggregates with avg(), not sum(). See observability-conventions.md.
+  if (resolved.otel)
+    registerActiveSessionsGauge(resolved.otel.auth.activeSessions, resolved.db, resolved.otel.observability.metricReadErrors)
+
   const { app, injectWebSocket } = await buildApp({
     auth: resolved.auth,
     db: resolved.db,

apps/server/src/libs/auth.ts

@@ -1,8 +1,8 @@
+import type { AuthMetrics } from '../otel'
 import type { EmailService } from '../services/email'
 import type { UserDeletionService } from '../services/user-deletion'
 import type { Database } from './db'
 import type { Env } from './env'
-import type { AuthMetrics } from './otel'
 
 import { Buffer } from 'node:buffer'
 
@@ -644,12 +644,6 @@ export function createAuth(
         create: {
           after: async () => {
             metrics?.userLogin.add(1)
-            metrics?.activeSessions.add(1)
-          },
-        },
-        delete: {
-          after: async () => {
-            metrics?.activeSessions.add(-1)
           },
         },
       },

apps/server/src/libs/db.ts

@@ -16,7 +16,7 @@ export type Database = ReturnType<typeof createDrizzle>['db']
 type DrizzleEnv = Pick<Env, 'DATABASE_URL' | 'DB_POOL_MAX' | 'DB_POOL_IDLE_TIMEOUT_MS' | 'DB_POOL_CONNECTION_TIMEOUT_MS' | 'DB_POOL_KEEPALIVE_INITIAL_DELAY_MS'>
 
 // NOTICE: pg is imported statically here. The OTEL instrumentation hooks are
-// registered via --import ./instrumentation.mjs (preload) which runs before
+// registered via --import ./instrumentation.ts (preload) which runs before
 // tsx loads application modules, allowing require-in-the-middle to patch pg.
 export function createDrizzle(env: DrizzleEnv) {
   const pool = new pg.Pool({

apps/server/src/middlewares/rate-limit.ts

@@ -1,6 +1,6 @@
 import type { Context } from 'hono'
 
-import type { RateLimitMetrics } from '../libs/otel'
+import type { RateLimitMetrics } from '../otel'
 import type { HonoEnv } from '../types/hono'
 
 import { getConnInfo } from '@hono/node-server/conninfo'

apps/server/src/otel/gauges/active-sessions.ts

@@ -0,0 +1,102 @@
+import type { AuthMetrics, ObservabilityMetrics } from '..'
+import type { Database } from '../../libs/db'
+
+import { useLogger } from '@guiiai/logg'
+import { count, gt } from 'drizzle-orm'
+
+import { session as sessionTable } from '../../schemas/accounts'
+
+/**
+ * Wire the `user.active_sessions` ObservableGauge to a Postgres `COUNT(*)`
+ * over the Better Auth session table.
+ *
+ * Use when:
+ * - Assembling DI in `createApp()`, exactly once per process.
+ *
+ * Expects:
+ * - `gauge` is the ObservableGauge handle created in `initOtel`.
+ * - `db` is the migrated Drizzle handle.
+ * - `metricReadErrors` is the shared counter used to track failures inside
+ *   metric callbacks — increments are labelled with the originating metric
+ *   name so on-call can spot which gauge is degraded.
+ *
+ * Multi-replica note:
+ * - This is a cluster-wide gauge — every replica reads the same DB and
+ *   reports the same value. Dashboards MUST aggregate with `avg()`, NOT
+ *   `sum()`. See observability-conventions.md.
+ *
+ * Concurrency:
+ * - Multiple OTel collection cycles can race (forced flushes, multiple
+ *   readers). The in-flight promise lock keeps at most one DB query in
+ *   flight per process; all other concurrent callbacks await the same
+ *   result instead of stampeding the DB.
+ *
+ * Failure mode:
+ * - On DB error we increment `airi.observability.read_errors{metric}` and
+ *   intentionally DO NOT call `result.observe(...)`. Letting the gauge
+ *   skip an export cycle lets Prometheus staleness handle "DB is broken"
+ *   correctly — an absence-based alert will fire after ~5 minutes. The
+ *   previous version silently observed the stale cached value forever,
+ *   which masked permanent DB failures.
+ */
+export function registerActiveSessionsGauge(
+  gauge: AuthMetrics['activeSessions'],
+  db: Database,
+  metricReadErrors: ObservabilityMetrics['metricReadErrors'],
+) {
+  const log = useLogger('active-sessions-gauge').useGlobalConfig()
+  const CACHE_TTL_MS = 10_000
+
+  let cachedAt = 0
+  let cachedCount = 0
+  // Single shared promise representing "a refresh is in progress". All
+  // callbacks that arrive during a refresh attach to this and observe the
+  // same outcome. Reset to null when the refresh resolves.
+  let refreshInFlight: Promise<boolean> | null = null
+
+  async function refresh(): Promise<boolean> {
+    try {
+      // Use the app clock (`new Date()`) rather than DB clock (`NOW()`) so
+      // we agree with Better Auth's own session validity check, which uses
+      // `new Date()` in its session lookup (`better-auth/dist/session.mjs`
+      // and `dist/internal-adapter.mjs`). A DB/app clock skew would
+      // otherwise let this gauge disagree with auth-layer reality.
+      const rows = await db
+        .select({ count: count() })
+        .from(sessionTable)
+        .where(gt(sessionTable.expiresAt, new Date()))
+      cachedCount = Number(rows[0]?.count ?? 0)
+      cachedAt = Date.now()
+      return true
+    }
+    catch (err) {
+      log.withError(err).warn('Failed to read active sessions for gauge')
+      metricReadErrors.add(1, { metric: 'user.active_sessions' })
+      return false
+    }
+  }
+
+  gauge.addCallback(async (result) => {
+    const now = Date.now()
+
+    // Cache fresh — serve last good value without touching the DB.
+    if (cachedAt !== 0 && now - cachedAt < CACHE_TTL_MS) {
+      result.observe(cachedCount)
+      return
+    }
+
+    // Coalesce concurrent refreshes onto one in-flight promise.
+    if (!refreshInFlight) {
+      refreshInFlight = refresh().finally(() => {
+        refreshInFlight = null
+      })
+    }
+    const ok = await refreshInFlight
+
+    if (ok) {
+      result.observe(cachedCount)
+    }
+    // else: deliberately do nothing — let Prometheus staleness expose the
+    // outage instead of masking it with a stale cached number.
+  })
+}

apps/server/src/otel/index.ts

@@ -1,13 +1,13 @@
-import type { Counter, Histogram, ObservableGauge, UpDownCounter } from '@opentelemetry/api'
+import type { Counter, Histogram, ObservableGauge } from '@opentelemetry/api'
 // NOTICE:
 // HTTP server metrics (request duration, active requests) are emitted by
 // `@hono/otel`'s `httpInstrumentationMiddleware` registered in `app.ts`. It
 // records the standard semconv names with the matched Hono route pattern,
 // so we don't create those handles here. We keep the auto HttpInstrumentation
 // for OUTBOUND requests only (LLM gateway, Stripe, Resend) — see
-// `instrumentation.mjs`.
+// `instrumentation.ts`.
 
-import type { Env } from './env'
+import type { Env } from '../libs/env'
 
 import { useLogger } from '@guiiai/logg'
 import { metrics, trace } from '@opentelemetry/api'
@@ -18,7 +18,9 @@ import {
   METRIC_AIRI_EMAIL_FAILURES,
   METRIC_AIRI_EMAIL_SEND,
   METRIC_AIRI_FLUX_CREDITED,
+  METRIC_AIRI_FLUX_UNBILLED,
   METRIC_AIRI_GEN_AI_STREAM_INTERRUPTED,
+  METRIC_AIRI_OBSERVABILITY_READ_ERRORS,
   METRIC_AIRI_RATE_LIMIT_BLOCKED,
   METRIC_AIRI_STRIPE_REVENUE,
   METRIC_AIRI_TTS_CHARS,
@@ -56,7 +58,25 @@ export interface AuthMetrics {
   failures: Counter
   userRegistered: Counter
   userLogin: Counter
-  activeSessions: UpDownCounter
+  /**
+   * Cluster-wide active session count, sourced from Postgres (Better Auth
+   * `session` table where `expires_at > NOW()`).
+   *
+   * Why ObservableGauge instead of UpDownCounter:
+   * - UpDownCounter drifts: TTL expiration never fires a -1, and multi-
+   *   replica deploys split +1 / -1 across instances (signin on A, signout
+   *   on B). The previous implementation went unboundedly positive.
+   * - Reading from the source-of-truth DB at scrape time makes the metric
+   *   self-correcting.
+   *
+   * Multi-replica note:
+   * - Every replica reads the same DB and reports the same value, so the
+   *   dashboard MUST aggregate with `max()` (or `avg()`), NOT `sum()`.
+   *   Using sum() would multiply the real count by the replica count.
+   * - See `apps/server/docs/ai-context/observability-conventions.md`,
+   *   "Multi-Replica Considerations".
+   */
+  activeSessions: ObservableGauge
 }
 
 export interface EngagementMetrics {
@@ -95,6 +115,22 @@ export interface RevenueMetrics {
   stripeRevenue: Counter
   fluxInsufficientBalance: Counter
   fluxCredited: Counter
+  /**
+   * Streaming-only: Flux consumed by a request whose post-stream debit failed.
+   *
+   * Use when:
+   * - Tracking real revenue leak in the LLM streaming proxy.
+   *
+   * Why this needs its own metric:
+   * - The streaming response is already sent (HTTP 200, tokens delivered) by
+   *   the time the catch around `billingService.consumeFluxForLLM` runs.
+   *   DB latency / HTTP 5xx alerts do NOT fire on this path — the failure is
+   *   silent at the transport layer. This counter is the only signal that
+   *   ties Flux value owed to a failed debit.
+   * - Recommended alert: `increase(airi_billing_flux_unbilled_total[5m]) > 0`
+   *   pages on-call immediately on any sustained leak.
+   */
+  fluxUnbilled: Counter
   ttsChars: Counter
   ttsPreflightRejections: Counter
 }
@@ -119,6 +155,18 @@ export interface RateLimitMetrics {
   blocked: Counter
 }
 
+export interface ObservabilityMetrics {
+  /**
+   * Counts failures inside metric-pipeline callbacks (e.g. a DB-backed
+   * ObservableGauge that couldn't read from Postgres). Use for self-monitoring
+   * — when this is rising, treat the affected gauge's reported value as
+   * potentially stale.
+   *
+   * Labels: `metric` (the failing gauge's logical name).
+   */
+  metricReadErrors: Counter
+}
+
 export interface OtelInstance {
   auth: AuthMetrics
   engagement: EngagementMetrics
@@ -126,6 +174,7 @@ export interface OtelInstance {
   genAi: GenAiMetrics
   email: EmailMetrics
   rateLimit: RateLimitMetrics
+  observability: ObservabilityMetrics
 }
 
 /**
@@ -136,8 +185,8 @@ export interface OtelInstance {
  *   disabled (no OTLP endpoint), so callers can skip wiring `metrics?.…`.
  *
  * Expects:
- * - `instrumentation.mjs` has already started NodeSDK (loaded via
- *   `tsx --import ./instrumentation.mjs`). This function does NOT start the
+ * - `instrumentation.ts` has already started NodeSDK (loaded via
+ *   `tsx --import ./instrumentation.ts`). This function does NOT start the
  *   SDK — it only consumes the global MeterProvider that the preload set up.
  *   Calling it before the preload runs would yield NoopMeter for everything.
  *
@@ -167,8 +216,8 @@ export function initOtel(env: Env): OtelInstance | null {
     userLogin: meter.createCounter(METRIC_USER_LOGIN, {
       description: 'Number of user sign-ins',
     }),
-    activeSessions: meter.createUpDownCounter(METRIC_USER_ACTIVE_SESSIONS, {
-      description: 'Number of active user sessions',
+    activeSessions: meter.createObservableGauge(METRIC_USER_ACTIVE_SESSIONS, {
+      description: 'Active user sessions sourced from Postgres (cluster-wide; dashboard must use max(), not sum())',
     }),
   }
 
@@ -224,6 +273,9 @@ export function initOtel(env: Env): OtelInstance | null {
     fluxCredited: meter.createCounter(METRIC_AIRI_FLUX_CREDITED, {
       description: 'Total flux credited to user balances, by source',
     }),
+    fluxUnbilled: meter.createCounter(METRIC_AIRI_FLUX_UNBILLED, {
+      description: 'Flux owed but unbilled (post-stream debit failed). Real revenue leak.',
+    }),
     ttsChars: meter.createCounter(METRIC_AIRI_TTS_CHARS, {
       description: 'TTS input characters processed (billing base unit)',
     }),
@@ -278,6 +330,12 @@ export function initOtel(env: Env): OtelInstance | null {
     }),
   }
 
+  const observability: ObservabilityMetrics = {
+    metricReadErrors: meter.createCounter(METRIC_AIRI_OBSERVABILITY_READ_ERRORS, {
+      description: 'Failures reading metric values inside gauge callbacks',
+    }),
+  }
+
   // NOTICE:
   // OTel SDK only emits a Counter time series after .add() runs the first time.
   // Without this priming step, low-traffic counters (auth_failures_total,
@@ -306,6 +364,7 @@ export function initOtel(env: Env): OtelInstance | null {
     revenue.stripeRevenue,
     revenue.fluxInsufficientBalance,
     revenue.fluxCredited,
+    revenue.fluxUnbilled,
     revenue.ttsChars,
     revenue.ttsPreflightRejections,
     genAi.operationCount,
@@ -316,10 +375,11 @@ export function initOtel(env: Env): OtelInstance | null {
     email.send,
     email.failures,
     rateLimit.blocked,
+    observability.metricReadErrors,
   ]
   for (const counter of counters) counter.add(0)
 
-  return { auth, engagement, revenue, genAi, email, rateLimit }
+  return { auth, engagement, revenue, genAi, email, rateLimit, observability }
 }
 
 const severityMap: Record<string, SeverityNumber> = {

apps/server/src/routes/auth/index.ts

@@ -1,7 +1,7 @@
 import type { AuthInstance } from '../../libs/auth'
 import type { Database } from '../../libs/db'
 import type { Env } from '../../libs/env'
-import type { RateLimitMetrics } from '../../libs/otel'
+import type { RateLimitMetrics } from '../../otel'
 import type { ConfigKVService } from '../../services/config-kv'
 import type { HonoEnv } from '../../types/hono'
 

apps/server/src/routes/chat-ws/index.ts

@@ -1,7 +1,7 @@
 import type Redis from 'ioredis'
 
 import type { HonoWsInvocableEventContext } from '../../libs/eventa-hono-adapter'
-import type { EngagementMetrics } from '../../libs/otel'
+import type { EngagementMetrics } from '../../otel'
 import type { ChatService } from '../../services/chats'
 
 import { useLogger } from '@guiiai/logg'

apps/server/src/routes/openai/v1/index.ts

@@ -2,7 +2,7 @@ import type { Context } from 'hono'
 import type Redis from 'ioredis'
 
 import type { Env } from '../../../libs/env'
-import type { GenAiMetrics, RateLimitMetrics } from '../../../libs/otel'
+import type { GenAiMetrics, RateLimitMetrics, RevenueMetrics } from '../../../otel'
 import type { UsageInfo } from '../../../services/billing/billing'
 import type { BillingService } from '../../../services/billing/billing-service'
 import type { FluxMeter } from '../../../services/billing/flux-meter'
@@ -90,6 +90,7 @@ export function createV1CompletionsRoutes(
   redis: Redis,
   env: Env,
   genAi?: GenAiMetrics | null,
+  revenue?: RevenueMetrics | null,
   rateLimitMetrics?: RateLimitMetrics | null,
 ) {
   const logger = useLogger('v1-completions').useGlobalConfig()
@@ -284,9 +285,15 @@ export function createV1CompletionsRoutes(
               actualCharged = fluxConsumed
             }
             catch (err) {
-              // Debit-after-stream is a single Postgres transaction — failure
-              // means DB itself is unhealthy, which is its own incident. Logged
-              // at error level so it surfaces in alerts; not a separate metric.
+              // Real revenue leak: streaming response already sent (HTTP 200,
+              // tokens delivered), so this catch produces no 5xx and no DB
+              // latency spike on the request path. Without a dedicated counter,
+              // the failure is silent. Page on any sustained `increase()`.
+              revenue?.fluxUnbilled.add(fluxConsumed, {
+                [GEN_AI_ATTR_REQUEST_MODEL]: requestModel,
+                reason: 'debit_failed',
+                stage: 'streaming',
+              })
               logger.withError(err).withFields({ userId: user.id, fluxConsumed, requestId }).error('Failed to debit flux after streaming — unpaid usage')
             }
 

apps/server/src/routes/stripe/index.ts

@@ -1,7 +1,7 @@
 import type Redis from 'ioredis'
 
 import type { Env } from '../../libs/env'
-import type { RateLimitMetrics, RevenueMetrics } from '../../libs/otel'
+import type { RateLimitMetrics, RevenueMetrics } from '../../otel'
 import type { BillingService } from '../../services/billing/billing-service'
 import type { ConfigKVService } from '../../services/config-kv'
 import type { FluxService } from '../../services/flux'

apps/server/src/scripts/otel/http-smoke.ts

@@ -6,12 +6,12 @@
  *   - Auto `HttpInstrumentation` is disabled for incoming, so it does NOT
  *     double-record the same histogram.
  *
- * STANDALONE simulation — does NOT use `--import ./instrumentation.mjs`. It
+ * STANDALONE simulation — does NOT use `--import ./instrumentation.ts`. It
  * mirrors the preload's NodeSDK setup but swaps the OTLP exporter for an
  * InMemoryMetricExporter so the smoke can read back what was recorded.
  *
  * Usage:
- *   pnpm -F @proj-airi/server exec node --import tsx ./src/scripts/otel-http-smoke.mjs
+ *   pnpm -F @proj-airi/server exec node --import tsx ./src/scripts/otel/http-smoke.ts
  */
 import { env, exit } from 'node:process'
 
@@ -54,9 +54,13 @@ app.get('/health-test', c => c.text('ok'))
 
 // `serve` returns the http.Server synchronously but binding is async — wait
 // for the listen callback to capture the port (port: 0 = auto-assigned).
-let server
-const { port } = await new Promise((resolve) => {
-  server = serve({ fetch: app.fetch, port: 0, hostname: '127.0.0.1' }, info => resolve(info))
+const server = serve({ fetch: app.fetch, port: 0, hostname: '127.0.0.1' })
+const port = await new Promise<number>((resolve) => {
+  server.once('listening', () => {
+    const addr = server.address()
+    if (addr && typeof addr === 'object')
+      resolve(addr.port)
+  })
 })
 console.info(`[smoke] hono server listening on 127.0.0.1:${port}`)
 

apps/server/src/scripts/otel/smoke.ts

@@ -11,12 +11,12 @@
  * Histograms (gen_ai.client.first_token.duration, airi.email.duration, ...)
  * are intentionally NOT in the output — they only register on first .record().
  *
- * NOTE: Run WITHOUT `--import ./instrumentation.mjs`. The preload would start
+ * NOTE: Run WITHOUT `--import ./instrumentation.ts`. The preload would start
  * a real NodeSDK with OTLP exporter and override the InMemoryMetricExporter
  * this smoke installs as the global MeterProvider.
  *
  * Usage:
- *   pnpm -F @proj-airi/server exec node --import tsx ./src/scripts/otel-smoke.mjs
+ *   pnpm -F @proj-airi/server exec node --import tsx ./src/scripts/otel/smoke.ts
  */
 import { env, exit } from 'node:process'
 
@@ -43,8 +43,8 @@ env.DEFAULT_CHAT_MODEL ??= 'test'
 env.DEFAULT_TTS_MODEL ??= 'test'
 env.OTEL_EXPORTER_OTLP_ENDPOINT ??= 'http://localhost:4318'
 
-const { initOtel } = await import('../../libs/otel.ts')
-const { parseEnv } = await import('../../libs/env.ts')
+const { initOtel } = await import('../../otel/index')
+const { parseEnv } = await import('../../libs/env')
 
 const parsed = parseEnv(env)
 const inst = initOtel(parsed)

apps/server/src/scripts/otel/ws-smoke.ts

@@ -10,10 +10,10 @@
  * deployment/lifecycle issue, not a code bug.
  *
  * Usage:
- *   pnpm -F @proj-airi/server exec node --import tsx ./src/scripts/otel-ws-smoke.mjs
+ *   pnpm -F @proj-airi/server exec node --import tsx ./src/scripts/otel/ws-smoke.ts
  *
  * With OTel diagnostic logs (verbose, includes export cycles):
- *   OTEL_DEBUG=true pnpm -F @proj-airi/server exec node --import tsx ./src/scripts/otel-ws-smoke.mjs
+ *   OTEL_DEBUG=true pnpm -F @proj-airi/server exec node --import tsx ./src/scripts/otel/ws-smoke.ts
  */
 import { env, exit } from 'node:process'
 import { setTimeout as sleep } from 'node:timers/promises'
@@ -94,36 +94,45 @@ app.get('/ws', upgradeWebSocket((c) => {
   }
 }))
 
-let server
-const { port } = await new Promise((resolve) => {
-  server = serve({ fetch: app.fetch, port: 0, hostname: '127.0.0.1' }, info => resolve(info))
+const server = serve({ fetch: app.fetch, port: 0, hostname: '127.0.0.1' })
+const port = await new Promise<number>((resolve) => {
+  server.once('listening', () => {
+    const addr = server.address()
+    if (addr && typeof addr === 'object')
+      resolve(addr.port)
+  })
 })
 injectWebSocket(server)
 console.info(`[ws-smoke] listening on 127.0.0.1:${port}\n`)
 
-async function readGaugeNow() {
+async function readGaugeNow(): Promise<number | null> {
   await reader.forceFlush()
   const all = exporter.getMetrics()
   const last = all.at(-1)
   for (const sm of last?.scopeMetrics ?? []) {
     for (const m of sm.metrics) {
-      if (m.descriptor.name === 'ws.connections.active')
-        return m.dataPoints.at(-1)?.value ?? null
+      if (m.descriptor.name === 'ws.connections.active') {
+        // ObservableGauge always carries `value: number` (Histogram /
+        // ExponentialHistogram are different DataPointType). The generic
+        // `value` union is what the SDK type exposes; narrow it explicitly.
+        const value = m.dataPoints.at(-1)?.value
+        return typeof value === 'number' ? value : null
+      }
     }
   }
   return null
 }
 
-const results = []
-function assert(label, expected, actual) {
+const results: boolean[] = []
+function assert(label: string, expected: number, actual: number | null) {
   const ok = actual === expected
   console.info(`[ws-smoke] ${ok ? '✅' : '❌'} ${label}: expected=${expected}, observed=${actual}\n`)
   results.push(ok)
 }
 
-async function openClient(user) {
+async function openClient(user: string): Promise<WebSocket> {
   const ws = new WebSocket(`ws://127.0.0.1:${port}/ws?user=${encodeURIComponent(user)}`)
-  await new Promise((res, rej) => {
+  await new Promise<void>((res, rej) => {
     ws.addEventListener('open', () => res(), { once: true })
     ws.addEventListener('error', e => rej(e), { once: true })
   })
@@ -132,7 +141,7 @@ async function openClient(user) {
   return ws
 }
 
-async function closeClient(ws) {
+async function closeClient(ws: WebSocket) {
   ws.close()
   await sleep(150)
 }

apps/server/src/services/billing/billing-service.ts

@@ -1,7 +1,7 @@
 import type Redis from 'ioredis'
 
 import type { Database } from '../../libs/db'
-import type { RevenueMetrics } from '../../libs/otel'
+import type { RevenueMetrics } from '../../otel'
 import type { ConfigKVService } from '../config-kv'
 
 import { useLogger } from '@guiiai/logg'

apps/server/src/services/billing/flux-meter.ts

@@ -1,6 +1,6 @@
 import type Redis from 'ioredis'
 
-import type { RevenueMetrics } from '../../libs/otel'
+import type { RevenueMetrics } from '../../otel'
 import type { BillingService } from './billing-service'
 
 import { useLogger } from '@guiiai/logg'

apps/server/src/services/characters.ts

@@ -1,5 +1,5 @@
 import type { Database } from '../libs/db'
-import type { EngagementMetrics } from '../libs/otel'
+import type { EngagementMetrics } from '../otel'
 
 import { useLogger } from '@guiiai/logg'
 import { and, eq, isNull, or, sql } from 'drizzle-orm'

apps/server/src/services/chats.ts

@@ -1,7 +1,7 @@
 import type { MessageRole, WireMessage } from '@proj-airi/server-sdk-shared'
 
 import type { Database } from '../libs/db'
-import type { EngagementMetrics } from '../libs/otel'
+import type { EngagementMetrics } from '../otel'
 
 import { useLogger } from '@guiiai/logg'
 import { and, eq, gt, inArray, isNull, sql } from 'drizzle-orm'

apps/server/src/services/email.ts

@@ -1,6 +1,6 @@
 import type { Logger } from '@guiiai/logg'
 
-import type { EmailMetrics } from '../libs/otel'
+import type { EmailMetrics } from '../otel'
 
 import { useLogger } from '@guiiai/logg'
 import { errorMessageFrom } from '@moeru/std'

apps/server/src/utils/observability.ts

@@ -64,9 +64,16 @@ export const METRIC_FLUX_CONSUMED = 'airi.billing.flux.consumed'
 
 // AIRI billing — credit/debit visibility beyond raw consumption
 export const METRIC_AIRI_FLUX_CREDITED = 'airi.billing.flux.credited'
+// Streaming-only: token already streamed to user but post-stream debit failed.
+// Real revenue leak — every >0 sample should page. NOT covered by DB latency /
+// HTTP 5xx alerts because the response was 2xx and the catch path is silent.
+export const METRIC_AIRI_FLUX_UNBILLED = 'airi.billing.flux.unbilled'
 export const METRIC_AIRI_TTS_CHARS = 'airi.billing.tts.chars'
 export const METRIC_AIRI_TTS_PREFLIGHT_REJECTIONS = 'airi.billing.tts.preflight_rejections'
 
+// AIRI observability — self-monitoring for the metric pipeline
+export const METRIC_AIRI_OBSERVABILITY_READ_ERRORS = 'airi.observability.read_errors'
+
 // AIRI revenue — actual money in (smallest currency unit, e.g. cents)
 export const METRIC_AIRI_STRIPE_REVENUE = 'airi.stripe.revenue'
 

apps/server/tsconfig.json

@@ -21,6 +21,7 @@
   },
   "include": [
     "src/**/*.ts",
-    "src/**/*.d.ts"
+    "src/**/*.d.ts",
+    "instrumentation.ts"
   ]
 }