147d077aa7
## Problem The WebGPU inference pipeline had several structural issues: 1. **No unified protocol** — Kokoro TTS, Whisper ASR, and Background Removal workers each used their own ad-hoc message formats. Adding a new model meant reinventing worker communication from scratch. 2. **Infrastructure existed but was disconnected** — `GPUResourceCoordinator`, `LoadQueue`, `InferenceWorkerManager`, and `protocol.ts` were all implemented but had zero consumers. The adapters duplicated the same lifecycle/timeout/mutex patterns independently. 3. **Performance gaps** — Kokoro only offered fp32 on WebGPU (no fp16), Whisper warm-up compiled shaders for 187.5s of dummy audio, audio transfer went through unnecessary WAV blob encode/decode, and `listVoices` reloaded the model every time. 4. **Silent failures** — Whisper worker's `generate()` had no try-catch; errors were swallowed and the main thread waited until timeout. 5. **No graceful degradation** — Whisper and Background Removal workers hardcoded `device: 'webgpu'` with no WASM fallback. 6. **No observability** — Only Kokoro had performance tracing. No adapter reported status to `useInferenceStatus`. No cache management UI existed. 7. **Dead code accumulation** — Old `KokoroWorkerManager` (232 lines), legacy Whisper message types, and scattered duplicate constants. ## Changes ### Phase 0 — Critical Performance & Bugs - Add `fp16-webgpu` dtype for Kokoro TTS (~2x inference speed on supported GPUs) - Fix Whisper warm-up tensor from `[1, 128, 3000]` → `[1, 128, 1]` (minimal shader compilation) - Fix Whisper worker silent error bug (add try-catch to `generate()` and `load()`) ### Phase 1 — Data Transfer & Caching - Switch Kokoro audio to Float32Array transferable (skip WAV blob encode in worker, lightweight WAV encode on main thread) - Cache `listVoices` results (skip redundant model reload when adapter state is `ready`) - Normalize progress reporting to 0-100 across all adapters, differentiate `warmup` phase ### Phase 2 — Protocol Unification & Infrastructure - Migrate all 3 workers + 3 adapters to unified `protocol.ts` message types (`load-model`, `run-inference`, `model-ready`, `inference-result`, `progress`, `error`) - Wire `GPUResourceCoordinator` into all adapters (VRAM allocation tracking, LRU ordering, memory pressure events) - Wire `LoadQueue` into all adapters (priority-based sequential model loading: TTS=10 > ASR=5 > BG_REMOVAL=1) - Add `coordinator.ts` global singleton for GPU coordinator + load queue - Add WebGPU detection + WASM fallback in Whisper and Background Removal workers ### Phase 3 — Error Recovery & Observability - Add restart logic with exponential backoff to Whisper adapter (matching Kokoro's existing pattern) - Integrate `classifyError()` (OOM / DEVICE_LOST / TIMEOUT classification) in Whisper adapter - Extend `defaultPerfTracer` to Whisper `transcribe()` and Background Removal `processImage()` - Wire `useInferenceStatus` into all 3 adapters (downloading → ready → terminated lifecycle) ### Phase 4 — Tests - Add unit tests for `AsyncMutex` (4 tests), `LoadQueue` (4 tests), `GPUResourceCoordinator` (7 tests) — all 15 passing ### Phase 5 — Cleanup & Features - Delete old `KokoroWorkerManager` (232 lines, zero consumers) - Delete orphaned `libs/workers/types.ts` (old Whisper message types) - Clean up `workers/kokoro/types.ts` (remove legacy message types, keep domain types) - Create centralized `constants.ts` (MODEL_IDS, MODEL_NAMES, TIMEOUTS, MAX_RESTARTS) - Remove hardcoded WebGPU check from background-removal devtools pages (worker auto-detects) - Add `useModelPreload` composable for generic idle-time preloading - Add `useInferencePreload` composable that reads provider config and preloads configured local models - Wire preloading into both `stage-web` and `stage-tamagotchi` App.vue (Kokoro TTS preloads 3s after init) - Add `ModelCacheManager.vue` settings component (cache size display, per-model status, clear cache) - Document GPU Device isolation architecture in protocol.ts ## After This PR - All inference workers speak the same protocol → adding a new model adapter is straightforward - GPU memory is tracked across all models with automatic pressure warnings at 80%/95% of VRAM budget - Models load sequentially via priority queue → no bandwidth/VRAM contention - Workers auto-detect WebGPU and fall back to WASM → works on browsers without WebGPU - Kokoro TTS preloads during idle time → "instant" first use for configured users - All adapters auto-restart on worker crashes (max 3 attempts, exponential backoff) - 15 unit tests cover core infrastructure (mutex, queue, coordinator) - Zero dead code remains in the inference pipeline ## Test Plan - [x] `pnpm exec vitest run packages/stage-ui/src/libs/inference/` — 15 tests pass - [x] `pnpm -F @proj-airi/stage-ui exec tsc --noEmit` — no TypeScript errors - [x] `pnpm lint:fix` — no lint errors in changed files - [ ] Manual: verify Kokoro TTS works with fp16-webgpu on a supported browser - [ ] Manual: verify Whisper ASR loads and transcribes correctly - [ ] Manual: verify Background Removal works in devtools page - [ ] Manual: verify preloading triggers in console (`[Preload] Loading kokoro-...`) --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
196 lines
6.9 KiB
JSON
196 lines
6.9 KiB
JSON
{
|
|
"name": "@proj-airi/stage-ui",
|
|
"type": "module",
|
|
"version": "0.9.0",
|
|
"private": true,
|
|
"description": "Shared core for stage",
|
|
"author": {
|
|
"name": "Moeru AI Project AIRI Team",
|
|
"email": "airi@moeru.ai",
|
|
"url": "https://github.com/moeru-ai"
|
|
},
|
|
"license": "MIT",
|
|
"repository": {
|
|
"type": "git",
|
|
"url": "https://github.com/moeru-ai/airi.git",
|
|
"directory": "packages/stage-ui"
|
|
},
|
|
"exports": {
|
|
".": "./src/components/index.ts",
|
|
"./components/scenarios/settings/model-settings": "./src/components/scenarios/settings/model-settings/index.ts",
|
|
"./components/scenes": "./src/components/scenes/index.ts",
|
|
"./components/*": "./src/components/*.ts",
|
|
"./components": "./src/components/index.ts",
|
|
"./composables/*": "./src/composables/*.ts",
|
|
"./composables": "./src/composables/index.ts",
|
|
"./constants/*": "./src/constants/*.ts",
|
|
"./constants": "./src/constants/index.ts",
|
|
"./libs/inference/adapters/*": "./src/libs/inference/adapters/*.ts",
|
|
"./libs/inference": "./src/libs/inference/index.ts",
|
|
"./libs/*": "./src/libs/*.ts",
|
|
"./libs": "./src/libs/index.ts",
|
|
"./stores/providers/aliyun": "./src/stores/providers/aliyun/index.ts",
|
|
"./stores/analytics": "./src/stores/analytics/index.ts",
|
|
"./stores/analytics/posthog": "./src/stores/analytics/posthog.ts",
|
|
"./stores/analytics/privacy-policy": "./src/stores/analytics/privacy-policy.ts",
|
|
"./stores/character": "./src/stores/character/index.ts",
|
|
"./stores/settings/analytics": "./src/stores/settings/analytics.ts",
|
|
"./stores/settings": "./src/stores/settings/index.ts",
|
|
"./stores/modules/vision": "./src/stores/modules/vision/index.ts",
|
|
"./stores/*": "./src/stores/*.ts",
|
|
"./stores": "./src/stores/index.ts",
|
|
"./workers/vad": "./src/workers/vad/index.ts",
|
|
"./workers/*": "./src/workers/*.ts",
|
|
"./workers": "./src/workers/index.ts",
|
|
"./utils": "./src/utils/index.ts",
|
|
"./utils/tts": "./src/utils/tts.ts",
|
|
"./types/*": "./src/types/*.ts",
|
|
"./types": "./src/types/index.ts"
|
|
},
|
|
"scripts": {
|
|
"typecheck": "vue-tsc --noEmit",
|
|
"story:dev": "histoire dev",
|
|
"story:build": "histoire build",
|
|
"story:build-base": "histoire build --base /ui",
|
|
"story:preview": "histoire preview",
|
|
"build": "echo \"No build step required\"",
|
|
"test": "vitest",
|
|
"test:run": "vitest run"
|
|
},
|
|
"dependencies": {
|
|
"@date-fns/utc": "^2.1.1",
|
|
"@fontsource-variable/nunito": "catalog:",
|
|
"@formkit/auto-animate": "^0.9.0",
|
|
"@huggingface/transformers": "^3.8.1",
|
|
"@moeru/eventa": "catalog:",
|
|
"@opentelemetry/api": "^1.9.0",
|
|
"@opentelemetry/core": "^2.6.0",
|
|
"@opentelemetry/sdk-trace-base": "^2.6.0",
|
|
"@proj-airi/audio": "workspace:^",
|
|
"@proj-airi/ccc": "workspace:^",
|
|
"@proj-airi/chromatic": "^1.0.2",
|
|
"@proj-airi/core-character": "workspace:^",
|
|
"@proj-airi/drizzle-duckdb-wasm": "catalog:",
|
|
"@proj-airi/font-chillroundm": "workspace:^",
|
|
"@proj-airi/font-cjkfonts-allseto": "workspace:^",
|
|
"@proj-airi/font-xiaolai": "workspace:^",
|
|
"@proj-airi/i18n": "workspace:^",
|
|
"@proj-airi/model-driver-lipsync": "workspace:^",
|
|
"@proj-airi/pipelines-audio": "workspace:^",
|
|
"@proj-airi/server-sdk": "workspace:^",
|
|
"@proj-airi/stage-shared": "workspace:^",
|
|
"@proj-airi/stage-ui-live2d": "workspace:^",
|
|
"@proj-airi/stage-ui-three": "workspace:^",
|
|
"@proj-airi/ui": "workspace:^",
|
|
"@ricky0123/vad-web": "^0.0.30",
|
|
"@shikijs/rehype": "^4.0.2",
|
|
"@shopify/draggable": "catalog:",
|
|
"@vueuse/core": "catalog:",
|
|
"@vueuse/motion": "^3.0.3",
|
|
"@vueuse/shared": "^14.2.1",
|
|
"@xsai-ext/providers": "catalog:",
|
|
"@xsai-transformers/embed": "^0.1.0",
|
|
"@xsai-transformers/shared": "^0.1.0",
|
|
"@xsai/embed": "catalog:",
|
|
"@xsai/generate-speech": "catalog:",
|
|
"@xsai/generate-text": "catalog:",
|
|
"@xsai/generate-transcription": "catalog:",
|
|
"@xsai/model": "catalog:",
|
|
"@xsai/shared": "catalog:",
|
|
"@xsai/shared-chat": "catalog:",
|
|
"@xsai/stream-text": "catalog:",
|
|
"@xsai/stream-transcription": "catalog:",
|
|
"@xsai/tool": "catalog:",
|
|
"@xsai/utils-chat": "catalog:",
|
|
"animejs": "^4.3.6",
|
|
"better-auth": "catalog:",
|
|
"culori": "^4.0.2",
|
|
"d3": "catalog:",
|
|
"date-fns": "^4.1.0",
|
|
"dompurify": "^3.3.3",
|
|
"embla-carousel-autoplay": "9.0.0-rc01",
|
|
"embla-carousel-vue": "catalog:",
|
|
"es-toolkit": "catalog:",
|
|
"gpuu": "^1.0.6",
|
|
"hono": "catalog:",
|
|
"html2canvas": "^1.4.1",
|
|
"idb-keyval": "catalog:",
|
|
"kokoro-js": "^1.2.1",
|
|
"localforage": "^1.10.0",
|
|
"mediabunny": "^1.40.1",
|
|
"nanoid": "^5.1.7",
|
|
"node-vibrant": "^4.0.4",
|
|
"ofetch": "^1.5.1",
|
|
"pinia": "catalog:",
|
|
"posthog-js": "catalog:",
|
|
"rehype-katex": "^7.0.1",
|
|
"rehype-parse": "^9.0.1",
|
|
"rehype-stringify": "^10.0.1",
|
|
"reka-ui": "^2.9.5",
|
|
"remark-math": "^6.0.0",
|
|
"remark-parse": "^11.0.0",
|
|
"remark-rehype": "^11.1.2",
|
|
"shiki": "^4.0.2",
|
|
"splitpanes": "catalog:",
|
|
"unified": "^11.0.5",
|
|
"unist-builder": "^4.0.0",
|
|
"unist-util-visit": "^5.1.0",
|
|
"unspeech": "catalog:xsai",
|
|
"unstorage": "catalog:",
|
|
"uuid": "^13.0.0",
|
|
"valibot": "catalog:",
|
|
"vaul-vue": "^0.4.1",
|
|
"vue-i18n": "^11.3.2",
|
|
"vue-router": "^5.0.4",
|
|
"vue-sonner": "^2.0.9",
|
|
"web-haptics": "catalog:",
|
|
"xast-util-to-xml": "^4.0.0",
|
|
"xastscript": "^4.0.0",
|
|
"xsschema": "catalog:",
|
|
"zod": "^4.3.6"
|
|
},
|
|
"devDependencies": {
|
|
"@fontsource-variable/dm-sans": "^5.2.8",
|
|
"@fontsource-variable/jura": "^5.2.7",
|
|
"@fontsource-variable/quicksand": "^5.2.10",
|
|
"@fontsource-variable/urbanist": "^5.2.7",
|
|
"@fontsource/dm-mono": "^5.2.7",
|
|
"@fontsource/dm-serif-display": "^5.2.8",
|
|
"@fontsource/gugi": "^5.2.7",
|
|
"@fontsource/kiwi-maru": "^5.2.8",
|
|
"@fontsource/m-plus-rounded-1c": "^5.2.10",
|
|
"@histoire/plugin-vue": "catalog:",
|
|
"@iconify-json/carbon": "^1.2.20",
|
|
"@iconify-json/eos-icons": "^1.2.4",
|
|
"@iconify-json/lucide": "^1.2.102",
|
|
"@iconify-json/mingcute": "^1.2.7",
|
|
"@iconify-json/simple-icons": "^1.2.77",
|
|
"@iconify-json/solar": "^1.2.5",
|
|
"@iconify-json/svg-spinners": "^1.2.4",
|
|
"@iconify-json/vscode-icons": "^1.2.45",
|
|
"@moeru/std": "catalog:",
|
|
"@pinia/testing": "catalog:",
|
|
"@proj-airi/lobe-icons": "^1.0.18",
|
|
"@proj-airi/stream-kit": "workspace:*",
|
|
"@proj-airi/vite-plugin-warpdrive": "workspace:*",
|
|
"@types/audioworklet": "catalog:",
|
|
"@types/culori": "^4.0.1",
|
|
"@types/d3": "catalog:",
|
|
"@types/hast": "catalog:",
|
|
"@types/splitpanes": "catalog:",
|
|
"@types/unist": "catalog:",
|
|
"@unocss/reset": "^66.6.8",
|
|
"@vitejs/plugin-vue": "^6.0.5",
|
|
"clustr": "^1.0.2",
|
|
"csstype": "^3.2.3",
|
|
"histoire": "catalog:",
|
|
"is-network-error": "catalog:",
|
|
"unocss": "^66.6.8",
|
|
"unplugin-info": "catalog:",
|
|
"unplugin-yaml": "^4.1.0",
|
|
"vite": "^6.4.2",
|
|
"vue": "catalog:",
|
|
"vue-tsc": "^3.2.6"
|
|
}
|
|
}
|