## Problem
The WebGPU inference pipeline had several structural issues:
1. **No unified protocol** — Kokoro TTS, Whisper ASR, and Background
Removal workers each used their own ad-hoc message formats. Adding a new
model meant reinventing worker communication from scratch.
2. **Infrastructure existed but was disconnected** —
`GPUResourceCoordinator`, `LoadQueue`, `InferenceWorkerManager`, and
`protocol.ts` were all implemented but had zero consumers. The adapters
duplicated the same lifecycle/timeout/mutex patterns independently.
3. **Performance gaps** — Kokoro only offered fp32 on WebGPU (no fp16),
Whisper warm-up compiled shaders for 187.5s of dummy audio, audio
transfer went through unnecessary WAV blob encode/decode, and
`listVoices` reloaded the model every time.
4. **Silent failures** — Whisper worker's `generate()` had no try-catch;
errors were swallowed and the main thread waited until timeout.
5. **No graceful degradation** — Whisper and Background Removal workers
hardcoded `device: 'webgpu'` with no WASM fallback.
6. **No observability** — Only Kokoro had performance tracing. No
adapter reported status to `useInferenceStatus`. No cache management UI
existed.
7. **Dead code accumulation** — Old `KokoroWorkerManager` (232 lines),
legacy Whisper message types, and scattered duplicate constants.
## Changes
### Phase 0 — Critical Performance & Bugs
- Add `fp16-webgpu` dtype for Kokoro TTS (~2x inference speed on
supported GPUs)
- Fix Whisper warm-up tensor from `[1, 128, 3000]` → `[1, 128, 1]`
(minimal shader compilation)
- Fix Whisper worker silent error bug (add try-catch to `generate()` and
`load()`)
### Phase 1 — Data Transfer & Caching
- Switch Kokoro audio to Float32Array transferable (skip WAV blob encode
in worker, lightweight WAV encode on main thread)
- Cache `listVoices` results (skip redundant model reload when adapter
state is `ready`)
- Normalize progress reporting to 0-100 across all adapters,
differentiate `warmup` phase
### Phase 2 — Protocol Unification & Infrastructure
- Migrate all 3 workers + 3 adapters to unified `protocol.ts` message
types (`load-model`, `run-inference`, `model-ready`, `inference-result`,
`progress`, `error`)
- Wire `GPUResourceCoordinator` into all adapters (VRAM allocation
tracking, LRU ordering, memory pressure events)
- Wire `LoadQueue` into all adapters (priority-based sequential model
loading: TTS=10 > ASR=5 > BG_REMOVAL=1)
- Add `coordinator.ts` global singleton for GPU coordinator + load queue
- Add WebGPU detection + WASM fallback in Whisper and Background Removal
workers
### Phase 3 — Error Recovery & Observability
- Add restart logic with exponential backoff to Whisper adapter
(matching Kokoro's existing pattern)
- Integrate `classifyError()` (OOM / DEVICE_LOST / TIMEOUT
classification) in Whisper adapter
- Extend `defaultPerfTracer` to Whisper `transcribe()` and Background
Removal `processImage()`
- Wire `useInferenceStatus` into all 3 adapters (downloading → ready →
terminated lifecycle)
### Phase 4 — Tests
- Add unit tests for `AsyncMutex` (4 tests), `LoadQueue` (4 tests),
`GPUResourceCoordinator` (7 tests) — all 15 passing
### Phase 5 — Cleanup & Features
- Delete old `KokoroWorkerManager` (232 lines, zero consumers)
- Delete orphaned `libs/workers/types.ts` (old Whisper message types)
- Clean up `workers/kokoro/types.ts` (remove legacy message types, keep
domain types)
- Create centralized `constants.ts` (MODEL_IDS, MODEL_NAMES, TIMEOUTS,
MAX_RESTARTS)
- Remove hardcoded WebGPU check from background-removal devtools pages
(worker auto-detects)
- Add `useModelPreload` composable for generic idle-time preloading
- Add `useInferencePreload` composable that reads provider config and
preloads configured local models
- Wire preloading into both `stage-web` and `stage-tamagotchi` App.vue
(Kokoro TTS preloads 3s after init)
- Add `ModelCacheManager.vue` settings component (cache size display,
per-model status, clear cache)
- Document GPU Device isolation architecture in protocol.ts
## After This PR
- All inference workers speak the same protocol → adding a new model
adapter is straightforward
- GPU memory is tracked across all models with automatic pressure
warnings at 80%/95% of VRAM budget
- Models load sequentially via priority queue → no bandwidth/VRAM
contention
- Workers auto-detect WebGPU and fall back to WASM → works on browsers
without WebGPU
- Kokoro TTS preloads during idle time → "instant" first use for
configured users
- All adapters auto-restart on worker crashes (max 3 attempts,
exponential backoff)
- 15 unit tests cover core infrastructure (mutex, queue, coordinator)
- Zero dead code remains in the inference pipeline
## Test Plan
- [x] `pnpm exec vitest run packages/stage-ui/src/libs/inference/` — 15
tests pass
- [x] `pnpm -F @proj-airi/stage-ui exec tsc --noEmit` — no TypeScript
errors
- [x] `pnpm lint:fix` — no lint errors in changed files
- [ ] Manual: verify Kokoro TTS works with fp16-webgpu on a supported
browser
- [ ] Manual: verify Whisper ASR loads and transcribes correctly
- [ ] Manual: verify Background Removal works in devtools page
- [ ] Manual: verify preloading triggers in console (`[Preload] Loading
kokoro-...`)
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
* feat(minecraft): contextflow integration
Add AiriBridge to cognitive engine to enable communication with Airi system. Expose `notifyAiri` and `updateAiriContext` functions in JS planner runtime globals for sending notifications and context updates from agent code.
* refactor(minecraft): remove redundant destinations field from airi-bridge
Remove hardcoded `destinations: ['minecraft']` from spark:command, spark:notify, and context:update events as the destination is already implied by the bridge context.
* feat(minecraft): complete AIRI context flow integration
Wire up bidirectional AIRI ↔ Minecraft communication via dedicated
signal channels instead of reusing signal:chat_message.
- Add airi_command and airi_context to PerceptionSignalType
- Route spark:command intents: action/plan/proposal/reroute emit
signal:airi_command; context intent emits signal:airi_context
- Add inbound context:update listener so AIRI can push context to
the Minecraft agent
- airi_context signals are injected into conversation history directly
without triggering a full cognitive cycle
- airi_command events reset the no-action budget (like player chat)
- enterContext/exitContext auto-emit context:update back to AIRI for
task lifecycle visibility
- Add sendEmit() helper for queued/working/done progress reporting
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(minecraft): report bot startup state to AIRI
Send initial context update to AIRI when brain comes online, including
bot username, server address, position, health, game mode, and list of
other online players. Tagged with 'startup' and bot username for
filtering.
* feat(minecraft): manage bot config through airi
* feat(stage-ui): add minecraft service shell
* fix(stage-ui): use applied minecraft config for chat context
* fix(minecraft): fall back on invalid env port
* fix(minecraft): support quoted env values
* fix(minecraft): ignore empty env overrides
* fix(stage-ui): keep minecraft draft during apply
* fix(stage-ui): simplify minecraft heartbeat label
* feat(stage-ui): simplify minecraft settings shell
* fix(minecraft): use registry heartbeat for stage liveness
* chore(minecraft): remove annoying detector decision logging
* fix(minecraft): limit context flow to bot pushes
* fix(minecraft): use workspace server sdk
* fix(stage-ui): replay registry snapshots for late listeners
* chore(minecraft): add todo for explicit stage deregistration
* feat: improve context flow observability
* fix(stage-ui): remove redundant header and callout from settings and context flow pages
* feat(stage-ui): enhance context flow prompt projection UI with readiness indicator
* docs(stage-ui): clarify context flow prompt projection terminology
* fix(stage-ui): add test for clone-safe context snapshots with nested fields
* todo
* fix(minecraft): avoid blocking startup on airi connection
* Remove LLM timeouts from the Minecraft brain
* Remove context boundary system from Minecraft brain
* feat(minecraft): use isolated-vm for JS planner sandbox
* refactor(minecraft): harden planner sandbox defaults
* Remove LLM attempt timeout guard
* [autofix.ci] apply automated fixes
* Fix server runtime shutdown signal handling
* Remove misleading Minecraft integration toggle
* contextflow destination to stage-*
* chore(deps): updated
* Add settings layout route for web devtools
* chore: default to expand context panels
* fix(server-*): should export more types, and for module:announced should ignore self
* refactor(minecraft): better context orchestration, handle module announce, context sync, and more
* feat(stage-ui): not spark command ready
* [autofix.ci] apply automated fixes
* clean up
* refactor: use :class arrays per AGENTS.md styling guidelines
* cleanup bloated tests
* rm more tests
* use errorMessageFrom
* hardcode to reduce bloat
* update setup instructions
* timeout for brain
* remove more bloat
* fix
---------
Co-authored-by-agent: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by-agent: Codex
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Neko Ayaka <neko@ayaka.moe>
- Added package.json for @proj-airi/server-protocol with necessary configurations.
- Implemented chat event types including WireMessage, SendMessagesRequest, and PullMessagesRequest.
- Defined WebSocket event types and structures for better integration with AIRI components.
- Updated server-runtime and server-sdk to utilize the new server-protocol package.
- Refactored imports across various packages to replace server-shared types with server-protocol types.
- Enhanced type definitions and added TypeScript configurations for better development experience.
Richard Pinedo