LLM/TTS router replacing knoway — verification

Verification artifacts for the in-process router. Scope tracked against docs/plans/2026-05-15-001-feat-llm-tts-router-replacing-knoway-plan.md.

Coverage status

User path	Code wired	Has fresh evidence
chat completions happy (router → OpenRouter)	✅	✅ commit `3a88f4225`, 2026-05-15
chat completions fallback (key/upstream exhaustion)	✅	❌ unit-test only, needs real-wire run
TTS speech (Azure) via `routeTts`	✅	⏳ pending (was knoway-fetch until 2026-05-15)
TTS speech (dashscope-cosyvoice) via `routeTts`	✅	⏳ pending
TTS speech (Volcengine) via `routeTts`	✅	⏳ pending
`/audio/voices` from adapter catalog (no upstream)	✅	⏳ pending (sanity curl)
`/livez`	✅	✅ commit `cfad87757`, 2026-05-15
`/readyz`	✅	✅ commit `cfad87757`, 2026-05-15

The TTS paths and /audio/voices were missing from the prior revision of this doc (which claimed shipped across U1-U9 while the route handlers were still hitting GATEWAY_BASE_URL). The router-side wiring landed 2026-05-15; the table above tracks the real evidence backlog so the doc stops asserting completion ahead of measurement.

E2E: chat completion through router service

Scenario: operator seeds LLM_ROUTER_CONFIG with one OpenRouter LLM upstream, then invokes the router directly to call OpenRouter for a chat completion. Validates envelope decrypt → configKV load → key rotation → upstream fetch → response parse on the real wire path.

Command:

cd apps/server
pnpm exec dotenvx run --env-file=.env.local -- \
  tsx scripts/seed-router-config.ts \
    --openrouter-key '<OPENROUTER_KEY>' \
    --openrouter-model openai/gpt-4o-mini \
    --default-chat-model chat-default
pnpm exec dotenvx run --env-file=.env.local -- \
  tsx scripts/e2e-llm-router.ts

Expected output: status 200, JSON body with choices[0].message.content populated, and E2E PASS — router service successfully called OpenRouter and returned a usable response.

Actual output (commit 3a88f4225, 2026-05-15):

→ calling router.route() with model=chat-default
  fetch → POST https://openrouter.ai/api/v1/chat/completions
  auth   = Bearer sk-or-v1-bb1a38505a7309...
  body   = {"messages":[{"role":"user","content":"Say \"hello world\" in exactly 3 words, no period."}],"max_tokens":20,"model":"openai/gpt-4o-mini"}
← status 200 (2958ms)

Assistant response:
  model:  openai/gpt-4o-mini
  text:   "hello world!"
  tokens: prompt=21 completion=3

E2E PASS — router service successfully called OpenRouter and returned a usable response.

Environment: commit 3a88f4225, local dev (Node 26, pnpm 10, Postgres
- Redis via local services), Hono 4.11.3, .env.local with a generated 32-byte base64 LLM_ROUTER_MASTER_KEY.
Last verified: 2026-05-15.

Liveness probe

Scenario: GET /livez returns 200 with {status: "live"} even when external dependencies are degraded. K8s-style flat naming; legacy /health and nested /healthz/live removed in this revision.
Command: curl -i http://localhost:3000/livez
Expected output: HTTP 200, body {"status":"live"}.
Actual output (commit cfad87757 + uncommitted route rename, 2026-05-15):
```
HTTP 200
{"status":"live"}
```
Cross-check: curl http://localhost:3000/health → HTTP 404 (legacy endpoint removed); curl http://localhost:3000/healthz/live → HTTP 404 (nested form removed).
Last verified: 2026-05-15.

Readiness probe

Scenario: GET /readyz returns 200 when Postgres + Redis both respond; 503 otherwise. Gateway-internal key health does NOT block readiness (R14).
Command: curl -i http://localhost:3000/readyz
Expected output: HTTP 200, body {"status":"ready","checks":{"db":"ok","redis":"ok"}}.
Actual output (commit cfad87757 + uncommitted route rename, 2026-05-15):
```
HTTP 200
{"status":"ready","checks":{"db":"ok","redis":"ok"}}
```
Last verified: 2026-05-15.

Test suite

Scenario: full unit-test suite for router-touched modules.
Command: pnpm -F @proj-airi/server exec vitest run
Expected output: green run; 91+ tests covering envelope-crypto, env, config-kv, llm-router/{router,key-rotator,config-loader,error-mapping}, tts-adapters, routes/openai/v1.
Actual output (commit 3a88f4225, 2026-05-15): 91 tests across 8 files green; pnpm -F @proj-airi/server typecheck exits 0.
Last verified: 2026-05-15.

Known limitations / follow-up

U9 admin HTTP endpoint: bootstrap currently goes through the scripts/seed-router-config.ts CLI. The plan's full HTTP admin endpoint with ETag + audit log + HMAC publish is deferred; tracked in plan U9.
GATEWAY_BASE_URL: still required in env schema. Resolved 2026-05-15: env entry removed, all routes go through llmRouter.route / routeTts / listTtsVoices. The LLM_ROUTER_MASTER_KEY env var is now required (no graceful skip).
Grafana dashboard JSON updates: the new airi.gen_ai.gateway.* counters … do not yet have panels for them. Partially resolved 2026-05-16: otel/grafana/dashboards/build.ts generates three router rows (Health / Trends / Admin Plane) covering the 7 gateway counters that have live producers: fallback_count, upstream_errors, key_exhausted, same_status_exhaustion, decrypt_failures, config_reload, and subscriber_state (producer added in the same PR — app.ts now emits connected / error / reconnecting from the configkv:invalidate subscriber). The remaining two counters (config_write, config_invalid_hmac) intentionally have no panels because their producer is the Plan U9 admin HTTP endpoint that has not shipped; they will rejoin Rows 6.5 / 6.7 alongside the U9 PR. Alert rules (key.exhausted > 0, fallback ratio > 30%, single-key error ratio > 80%) are still configured through Grafana UI, not build.ts — IaC-ifying them is a separate follow-up.
knoway compose retention: keep /Users/luoling8192/Git/proj-airi/airi-railway/knoway/
- the corresponding container entry in airi-railway/docker-compose.yml for at least 14 days without a P1+ incident before removing per plan R18.

6.3 KiB Raw Blame History