Files

T

RainbowBird a3a401d79d feat(server): new metrics for auth

- Added a new ObservableGauge for distinct active users to track real active user count, mitigating session row inflation issues.
- Updated the Grafana dashboard to reflect changes, including the removal of redundant WS Connections panel and the addition of new metrics for active sessions and distinct users.
- Improved documentation for verification automation processes, outlining a structured approach to automate verification steps and maintain evidence of tests.

2026-05-15 18:54:05 +08:00

6.8 KiB

Raw Blame History

Verification: Flux Unbilled Exploit Fix

Status: patched in commit 7267b0d6b (2026-05-15) for the chat-completion path. TTS flux-meter adaptation followed up in the same PR as this verification doc (see "Remaining gaps → Gap 1"). Last attempted: 2026-05-15 Owner: rbxin2003@gmail.com

用户路径

场景：用户 0 < balance < fallbackRate，开 N 个并发 LLM completion 请求
预期（修补后）：第一个 request 之后 pre-flight 拒绝；最多触发一次 partial debit（charged < requested），所有剩余 request 收 402
实际（修补前）：N 个 request 全部走到 stream end，每个 catch consumeFluxForLLM 失败回滚 → balance 不变 → 用户拿到 N 次免费 LLM 响应

Before / After 行为

Before（修补前漏洞链）

Pre-flight 仅 if (flux.flux <= 0)（apps/server/src/routes/openai/v1/index.ts，旧版）
用户余额 1 flux，但 FLUX_PER_REQUEST (fallback) 通常远大于 1
并发 N 请求全部通过 pre-flight，上游 LLM 全部完成（响应已 stream 出去）
consumeFluxForLLM 走 debitFlux (apps/server/src/services/billing/billing-service.ts:107)，旧逻辑：
```
if (balanceBefore < input.amount) {
  throw createPaymentRequiredError('Insufficient flux')
}
```
整个 DB tx 回滚 → balance 维持 1 flux
catch 路径上报 airi_billing_flux_unbilled_total{reason='debit_failed'} += full amount
用户重复同样的脚本，每次都拿免费响应

这是 Grafana panel-43 (Flux Unbilled) 累积到 70.2K 的根因。

After（commit `7267b0d6b`）

1. Pre-flight 阈值改为 fallbackRate（apps/server/src/routes/openai/v1/index.ts:138-142）：

const fallbackRate = await configKV.getOrThrow('FLUX_PER_REQUEST')
const fluxPer1kTokens = await configKV.get('FLUX_PER_1K_TOKENS')

const flux = await fluxService.getFlux(user.id)
if (flux.flux < fallbackRate) {
  throw createPaymentRequiredError('Insufficient flux')
}

并发 N 请求时，pre-flight 直接 402 拒绝，不会进入上游 LLM 调用。

2. Partial-debit 语义（apps/server/src/services/billing/billing-service.ts:107-130）：

if (balanceBefore <= 0) {
  metrics?.fluxInsufficientBalance.add(1)
  throw createPaymentRequiredError('Insufficient flux')
}

const chargedAmount = Math.min(input.amount, balanceBefore)
const balanceAfter = balanceBefore - chargedAmount
const isPartial = chargedAmount < input.amount

balance > 0 但不够 → drain 到 0，ledger 记 amount = charged，metadata 带 requestedAmount + unbilled
balance <= 0 才 throw（catch path 上报 reason='debit_failed'）
Partial drain 上报 reason='partial_debit_drained'，跟真实 DB 错误区分

3. Idempotency 反映原始 charged（billing-service.ts:71-94）：

替换 request 的同 requestId 重放复用历史 existing.amount 作为 charged，避免重试时双扣 unbilled counter。

Evidence

Item	Reference
Fix commit	`7267b0d6b feat(server/billing): partial-debit semantics to prevent unpaid usage exploit`
Pre-flight gate	`apps/server/src/routes/openai/v1/index.ts:138-142`
Partial-debit logic	`apps/server/src/services/billing/billing-service.ts:107-148`
Streaming path unbilled metric	`apps/server/src/routes/openai/v1/index.ts:299-313` (label `reason='partial_debit_drained'`, `stage='streaming'`)
Non-streaming path unbilled metric	`apps/server/src/routes/openai/v1/index.ts:374-388` (label `stage='non_streaming'`)
Regression tests added	`apps/server/src/routes/openai/v1/route.test.ts` (+97 lines), `apps/server/src/services/billing/tests/billing-service.test.ts` (+89 lines)

测试覆盖的两个核心 case：

'rejects pre-flight when balance is below FLUX_PER_REQUEST (Issue: unpaid-usage-exploit)' — 验证 pre-flight 在 partial-balance 用户上 reject，upstream 未被调用
'non-streaming completion drains partial balance and logs charged (Issue: unpaid-usage-exploit)' — 验证 partial-debit drain 到 zero + metric 上报

Remaining gaps

Gap 1 — TTS flux-meter partial-debit 适配 ✅ 已修复

apps/server/src/services/billing/flux-meter.ts:135-228 的 accumulate() 现在解构 { charged, requested }，partial drain 时：

上报 airi_billing_flux_unbilled_total{source='tts_meter', reason='partial_debit_drained', meter, gen_ai.request.model?}
INCRBY (requested - charged) * unitsPerFlux 回 Redis debt counter
AccumulateResult 增加 unbilledFlux 字段供调用方观测
加 invariant 校验 charged > requested / 非整数 / 负数 → throw 而不是静默 under-restore

测试覆盖：flux-meter.test.ts:217-260，case 名带 Issue: unpaid-usage-exploit follow-up。

残余风险（已知，留 follow-up）：settlement (LUA runScript) 和 INCRBY restore 之间非原子，并发请求理论上可能在窗口内读到 mid-state。窗口很小（一个 DB tx），实际命中很难触发；长期修复需要 Redis lock 或 unbilled 单独 bucket。代码里 flux-meter.ts 有 // REVIEW: 标记。

Gap 2 — 修补前的 70.2K 历史漏账未核销

panel-43 显示的 70.2K 是 counter 累积值（increase($__range)），代表修补前漏出去的总量。修补不会让 panel 自动归零，只会让新的增量趋近 0（除真实 DB 错误）。

建议：

把 dashboard 时间窗调到 commit 7267b0d6b 部署后（2026-05-15 之后）观察增量斜率
加 Grafana alert：increase(airi_billing_flux_unbilled_total[5m]) > 0 → PagerDuty
如果业务需要核销历史 70.2K，从 flux_transaction ledger 反查 metadata->>'reason' = 'debit_failed' 的记录，配合 Loki 错误日志定位涉事 userId

What's verified

✓ 代码层：pre-flight gate + partial-debit semantics 确实改了，逻辑正确
✓ 测试层：两个核心 regression test 覆盖 exploit 场景，case 名带 Issue: unpaid-usage-exploit 标识

What's pending live verification

⊘ 生产 Grafana 上 panel-43 在 2026-05-15 commit 部署后的斜率趋近 0
⊘ 没跑 pnpm -F @proj-airi/server exec vitest run 实际确认新测试 pass（建议在 push 之前跑一次）
⊘ TTS partial-debit 适配的 live verification（代码已修补 + 单测 14/14 pass，需要在生产观察 airi_billing_flux_unbilled_total{source='tts_meter'} 出现合理流量后再 close）

Recommended follow-ups

按优先级：

P0 — 加 Grafana alert increase(airi_billing_flux_unbilled_total[5m]) > 0 → 通知
P1 — 历史 70.2K 漏账处理：查 ledger + Loki 决定核销还是补账
~~P1 — 修 TTS flux-meter 适配 partial-debit 新语义~~ ✅ 已修复，见 Gap 1
P2 — Dashboard 改造（panel-43 时间窗注解 + 区分 reason label 的 stack 图，把 partial_debit_drained 跟 debit_failed 分色展示）

6.8 KiB Raw Blame History Unescape Escape