**Goal**: Wire Pub/Sub-driven invalidation of the LLM router config in-memory cache (KTD-4). Add per-instance `config.reload` OTel counter. Add `/healthz/live` (always 200) and `/healthz/ready` (Postgres + Redis ping) endpoints. **Gateway key health does NOT affect readiness** (R14).
**Goal**: Wire Pub/Sub-driven invalidation of the LLM router config in-memory cache (KTD-4). Add per-instance `config.reload` OTel counter. Add `/livez` (always 200) and `/readyz` (Postgres + Redis ping) endpoints. **Gateway key health does NOT affect readiness** (R14).
**Requirements**: R13, R14, R16, R16a (acknowledged as Outstanding Question — not actively delivered, see "Resolve before merging" below), KTD-4, KTD-12.
- On `configKV.set('LLM_ROUTER_CONFIG', value)`: publish to channel.
- In `createLlmRouterService` init: subscribe via separate `ioredis` instance (Redis Pub/Sub requires dedicated subscriber connection per ioredis docs). On message matching `key === 'LLM_ROUTER_CONFIG'`: call `config-loader.invalidate()` and increment `gateway.configReload.add(1, {service_instance_id, source: 'pubsub'})`.
- TTL fallback: in-memory cache has TTL = 5s. On TTL expiry next request reloads from configKV (Postgres+Redis source-of-truth chain) and increments counter with `source: 'ttl'`.
-`/healthz/live`: route returns `200 {status: 'live'}` always. No DB / Redis touch. Excluded from `httpInstrumentationMiddleware` (`apps/server/src/app.ts:126-128` pattern).
-`/healthz/ready`: route pings Postgres (`SELECT 1`) + Redis (`PING`). Returns 200 if both ok; 503 otherwise. **Does not check gateway key health** (R14 — single key flap can't take instance out of pool).
-Existing`/health` endpoint at `apps/server/src/app.ts:187` stays (keep external monitors stable); declared deprecated in docs in U8.
-`/livez`: route returns `200 {status: 'live'}` always. No DB / Redis touch. Excluded from `httpInstrumentationMiddleware` (`apps/server/src/app.ts:126-128` pattern).
-`/readyz`: route pings Postgres (`SELECT 1`) + Redis (`PING`). Returns 200 if both ok; 503 otherwise. **Does not check gateway key health** (R14 — single key flap can't take instance out of pool).
-Legacy`/health` endpoint removed post-implementation in favor of K8s-style `/livez` + `/readyz` (single source of truth, no overlap).
**Patterns to follow**:
- ioredis Pub/Sub: dedicated subscriber connection (search for existing pubsub usage in `apps/server` — `redis-boundaries-and-pubsub.md` references this)
@@ -593,17 +593,17 @@ apps/server/
- (1) Config-loader subscribes on init; on Pub/Sub message for `LLM_ROUTER_CONFIG`: cache cleared, next read fetches fresh.
- (2) Pub/Sub message for unrelated key: no invalidation, no counter increment.
- (3) TTL expiry path: cache populated → 5s elapse (mock clock or vitest fake timers) → next read fetches fresh + counter incremented with `source: 'ttl'`.
- (4) `GET /healthz/live` returns 200 + `{status: 'live'}` even when Redis is down (Redis client error mocked).
- (5) `GET /healthz/ready` returns 200 when both Postgres + Redis ping ok.
- (6) `GET /healthz/ready` returns 503 when Postgres down (mock pool query throws).
- (7) `GET /healthz/ready` returns 503 when Redis down (mock ping throws).
- (8) `GET /healthz/ready` returns 200 even with `LLM_ROUTER_CONFIG` missing (gateway state does not block readiness per R14).
- (9) `httpInstrumentationMiddleware` does NOT instrument `/healthz/*` requests (assert OTel http span count after probe = 0).
- (4) `GET /livez` returns 200 + `{status: 'live'}` even when Redis is down (Redis client error mocked).
- (5) `GET /readyz` returns 200 when both Postgres + Redis ping ok.
- (6) `GET /readyz` returns 503 when Postgres down (mock pool query throws).
- (7) `GET /readyz` returns 503 when Redis down (mock ping throws).
- (8) `GET /readyz` returns 200 even with `LLM_ROUTER_CONFIG` missing (gateway state does not block readiness per R14).
- (9) `httpInstrumentationMiddleware` does NOT instrument `/livez` or `/readyz` requests (assert OTel http span count after probe = 0).
**Verification**:
-`pnpm -F @proj-airi/server typecheck` passes
-`pnpm exec vitest run apps/server/src/app.test.ts` green
- Manual: `curl /healthz/live` → 200; `curl /healthz/ready` → 200 with Postgres + Redis up
- Manual: `curl /livez` → 200; `curl /readyz` → 200 with Postgres + Redis up
**Resolve before merging**:
- **R16a admin permission model** is an explicit Outstanding Question in origin; if it's not resolved before this unit ships, the admin set-config endpoint stays behind existing flat-admin-role auth. Note as known limitation in PR description: "Admin endpoint for `set LLM_ROUTER_CONFIG` uses existing flat admin role; role-scoping is follow-up work".
@@ -675,7 +675,7 @@ apps/server/
-`apps/server/src/libs/env.test.ts` (MODIFY)
-`apps/server/scripts/verify-router-config.ts` (NEW — operator script referenced in U1 Migration step; decrypts current `LLM_ROUTER_CONFIG` against `LLM_ROUTER_MASTER_KEY` to validate boot-time correctness)
-`apps/server/src/scripts/otel/llm-router-smoke.ts` (NEW — new smoke fixture that produces traces tagged with `airi.gen_ai.gateway.*` attrs; **note**: the previously-referenced `apps/server/src/scripts/otel/smoke.ts` does not exist — the actual existing file is `ws-smoke.ts` for WebSocket smoke; gateway-specific smoke is new work)
- Grafana dashboard JSON: add 3 panels (key exhausted count time series, fallback depth distribution, upstream errors by status code) + 3 alert rules (P0 key.exhausted > 0 in 5min, P1 fallback ratio > 30% in 15min, P2 single key > 80% errors in 30min). Use existing `build.ts` to assemble. Thresholds are placeholders — refine post-launch.
- DI wiring: extend `AppDeps` interface (`apps/server/src/app.ts:70-88`) with `llmRouter` field. Register via `injeca.provide('services:llmRouter', { dependsOn: ['services:configKV', 'libs:redis', 'otel'], build: ... })`. Thread into `createV1CompletionsRoutes` factory.
-`GATEWAY_BASE_URL` removal: delete from env schema; verify no remaining consumers via grep — current consumers per existing brainstorm context: `apps/server/src/libs/env.ts`, `apps/server/src/libs/env.test.ts`, `apps/server/src/routes/openai/v1/index.ts`, `apps/server/src/routes/openai/v1/route.test.ts`, `apps/server/src/scripts/otel/smoke.ts`. Update each.
- Verification doc (`apps/server/docs/ai-context/verifications/llm-router.md`): follow AGENTS.md template — for each user path (chat completions happy / chat completions fallback / TTS speech happy / voices listing / healthz live / healthz ready), include scenario / command / expected output / actual output (curl response snippets) / environment (commit SHA + deploy env) / last verified date.
- Verification doc (`apps/server/docs/ai-context/verifications/llm-router.md`): follow AGENTS.md template — for each user path (chat completions happy / chat completions fallback / TTS speech happy / voices listing / livez / readyz), include scenario / command / expected output / actual output (curl response snippets) / environment (commit SHA + deploy env) / last verified date.
- knoway compose: do not delete yet. Document retention criteria in transport-and-routes.md and PR description: "knoway compose stays until: 14 days post-deploy without P1+ incidents OR 1 peak-traffic event without P1+ incidents. Reset on any P1."
**Patterns to follow**:
@@ -772,8 +772,8 @@ Verification doc lives at `apps/server/docs/ai-context/verifications/llm-router.
4.**TTS speech (cosyvoice)**: same against cosyvoice model.
5.**TTS speech (Volcengine)**: same against Volcengine model.
6.**Voices listing**: `curl GET /api/v1/openai/audio/voices?model=azure-tts` returns voice catalog JSON.
'5-minute average inbound HTTP request rate. /health (Railway probe) is excluded at the @hono/otel middleware level so this reflects real user traffic. **Fixed 5m window — intentionally does not follow the dashboard time picker** (see row-level note). For trends, see panel-14 (HTTP Request Rate by Route).',
'5-minute average inbound HTTP request rate. /livez and /readyz (K8s probes) are excluded at the @hono/otel middleware level so this reflects real user traffic.',
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
RainbowBird