leafyy
aa69ee62b5
## Description
The VRM and MMD renderers had the same two boundary problems:
- Both imported the Stage audio store from the `stage-ui` source tree.
- Both kept copies of the wLipSync profile and model-neutral vowel
policy.
This PR solves both problems as one architecture change.
```text
Stage owns AudioContext and the current audio source
-> ThreeScene / MMDScene receive both as inputs
-> model-driver-lipsync owns the profile and shared vowel policy
-> VRM maps AEIOU to aa / ee / ih / oh / ou
-> MMD maps AEIOU to vowelA / vowelE / vowelI / vowelO / vowelU
```
The shared driver owns viseme projection, winner and runner selection,
silence handling, and smoothing. It has no Vue, Three.js, VRM, or MMD
dependency. Each renderer keeps only its model-specific mapping.
The renderer packages no longer import the Stage audio store or depend
directly on `wlipsync`. Stage passes the owning `AudioContext` down with
the current audio source. Cleanup disconnects only the wLipSync node
owned by that renderer.
## Current solution
The package-boundary review identified two related findings:
- Finding 3: The VRM and MMD renderers imported `stage-ui` audio state
through source paths.
- Finding 4: The same renderers duplicated the wLipSync profile and
model-neutral vowel policy.
Both findings came from one missing boundary between Stage, the lip-sync
driver, and the renderer adapters. This PR fixes that boundary with one
ownership change.
| Layer | Responsibility |
| --- | --- |
| `stage-ui` | Owns the `AudioContext` and the current audio source. It
passes both values to each renderer. |
| `stage-ui-three` and `stage-ui-mmd` | Connect renderer components to
the shared driver. They do not import the Stage audio store. |
| `model-driver-lipsync/runtime/wlipsync` | Exposes `createWLipSyncNode`
as the public runtime entry. |
| `model-driver-lipsync/shared/wlipsync` | Owns the profile, public
types, and model-neutral vowel policy. |
Runtime flow:
1. `Stage.vue` passes `audioContext` and `currentAudioSource` to
`ThreeScene` or `MMDScene`.
2. The renderer composable watches the context and creates one wLipSync
node for that context.
3. The source watcher connects the current audio source to that node.
4. Each render frame sends the wLipSync values to
`createWLipSyncVowelDriver`.
5. The renderer adapter applies the returned AEIOU weights to its model
controls.
6. Cleanup disconnects the exact source and node pair that the renderer
owns.
The shared policy keeps the current renderer behavior:
- It maps the raw `S` viseme to `I`.
- It selects the two strongest vowels.
- It uses the existing volume and silence thresholds.
- It keeps the 160 ms silence window.
- It keeps the existing attack, release, cap, and output scale values.
The renderer-specific mappings stay in their renderer packages:
- VRM maps AEIOU to `aa`, `ee`, `ih`, `oh`, and `ou`.
- MMD maps AEIOU to `vowelA`, `vowelE`, `vowelI`, `vowelO`, and
`vowelU`.
This solution removes both duplicate profile files. It also removes the
direct `wlipsync` dependencies from both renderer packages.
## Linked Issues
None.
## Additional Context
There is no intended visual change. This refactor preserves the existing
vowel weights and renderer mappings.
Verification:
- `sem diff --staged --no-cosmetics -v --file-exts .ts .tsx`
- `sem diff --staged --no-cosmetics -v --file-exts .vue`
- `sem impact --entity-id <useVRMLipSync> --json --dependents`
- `sem impact --entity-id <useMMDLipSync> --json --dependents`
- `pnpm -F @proj-airi/model-driver-lipsync test` — 3 tests passed
- `pnpm -F @proj-airi/stage-ui-three exec vitest run` — 18 tests passed
- `pnpm -F @proj-airi/stage-ui-mmd exec vitest run` — 23 tests passed
- Type checks passed for `model-driver-lipsync`, `stage-ui-three`,
`stage-ui-mmd`, and `stage-ui`
- `pnpm install --frozen-lockfile --ignore-scripts`
- `pnpm lint` — 0 errors and 7 existing warnings outside this change
- `git diff --check`
The full workspace `pnpm typecheck` was also run. In the isolated
worktree it stopped in unchanged `electron-vueuse` and
`plugin-sdk-tamagotchi` because their generated dependency outputs were
not present. All affected package type checks above passed.
Project AIRI
Re-creating Neuro-sama, a soul container of AI waifu / virtual characters to bring them into our world.
[Join Discord Server] [Try it] [简体中文] [日本語] [Русский] [Tiếng Việt] [Français] [한국어]
Heavily inspired by Neuro-sama
Tip
On Windows, you can also install AIRI with winget:
winget install MoeruAI.AIRIOr install AIRI with Scoop:
scoop bucket add airi https://github.com/moeru-ai/airi scoop install airi/airiOn macOS, you can install AIRI with Homebrew Cask without adding a custom tap:
brew install --cask airi
Warning
Attention: We do not have any officially minted cryptocurrency or token associated with this project. Please check the information and proceed with caution.
Note
We've got a whole dedicated organization @proj-airi for all the sub-projects born from Project AIRI. Check it out!
RAG, memory system, embedded database, icons, Live2D utilities, and more!
Tip
We have a translation project on Crowdin. If you find any inaccurate translations, feel free to contribute improvements there.
Have you dreamed about having a cyber living being (cyber waifu, digital pet) or digital companion that could play with and talk to you?
With the power of modern large language models like ChatGPT and famous Claude, asking a virtual being to roleplay and chat with us is already easy enough for everyone. Platforms like Character.ai (a.k.a. c.ai) and JanitorAI as well as local playgrounds like SillyTavern are already good-enough solutions for a chat based or visual adventure game like experience.
But, what about the abilities to play games? And see what you are coding at? Chatting while playing games, watching videos, and is capable of doing many other things.
Perhaps you know Neuro-sama already. She is currently the best virtual streamer capable of playing games, chatting, and interacting with you and the participants. Some also call this kind of being "digital human." Sadly, as it's not open sourced, you cannot interact with her after her live streams go offline.
Therefore, this project, AIRI, offers another possibility here: let you own your digital life, cyber living, easily, anywhere, anytime.
Recent DevLogs & Stories
- DevLog @ 2026.03.23: Mobile performance and game engine exploration - March 23, 2026
- DevLog @ 2026.03.14: VRM stage lifecycle, cache, and observability - March 14, 2026
- DevLog @ 2026.02.16: Dome Keeper data collection and training pipeline - February 16, 2026
- DevLog @ 2026.01.01: AIRI Pocket and FlowChat memory experiments - January 1, 2026
- DevLog @ 2025.10.20: Electron migration, new models, and Moeru AI updates - October 20, 2025
- DevLog @ 2025.08.26: Pure vision progress for airi-factorio - August 26, 2025
- DevLog @ 2025.08.05: AIRI v0.7 release highlights - August 5, 2025
- DevLog @ 2025.08.01: Streaming text animation and grapheme clusters - August 1, 2025
- DevLog @ 2025.07.18: Rethinking airi-factorio with Factorio Learning Environment - July 18, 2025
- DreamLog 0x1: The backstory of Project AIRI - June 16, 2025
- More on the documentation site
What's So Special About This Project?
Unlike the other AI driven VTuber open source projects, アイリ was built with support of many Web technologies such as WebGPU, WebAudio, Web Workers, WebAssembly, WebSocket, etc. from the first day.
Tip
Worrying about the performance drop since we are using Web related technologies?
Don't worry, while the Web browser version is meant to give an insight into how much we can push and do inside browsers and webviews, we will never fully rely on this. The desktop version of AIRI is capable of using native NVIDIA CUDA and Apple Metal by default (thanks to HuggingFace & the beloved candle project), without any complex dependency management. Considering the tradeoff, it is partially powered by Web technologies for graphics, layouts, animations, and the WIP plugin system for everyone to integrate things.
This means that アイリ is capable of running on modern browsers and devices and even on mobile devices (already done with PWA support). This brings a lot of possibilities for us (the developers) to build and extend the power of アイリ VTuber to the next level, while still leaving the flexibility for users to enable features that require TCP connections or other non-Web technologies such as connecting to a Discord voice channel or playing Minecraft and Factorio with friends.
Note
We are still in the early stage of development where we are seeking out talented developers to join us and help us to make アイリ a reality.
It's ok if you are not familiar with Vue.js, TypeScript, and devtools required for this project, you can join us as an artist, designer, or even help us to launch our first live stream.
Even if you are a big fan of React, Svelte or even Solid, we welcome you. You can open a sub-directory to add features that you want to see in アイリ, or would like to experiment with.
Fields (and related projects) that we are looking for:
- Live2D modeller
- VRM modeller
- VRChat avatar designer
- Computer Vision
- Reinforcement Learning
- Speech Recognition
- Speech Synthesis
- ONNX Runtime
- Transformers.js
- vLLM
- WebGPU
- Three.js
- WebXR (checkout the another project we have under the @moeru-ai organization)
If you are interested, why not introduce yourself here? Would like to join part of us to build AIRI?
Current Progress & Roadmap
Capable of
- Brain
- Play Minecraft
- Play Factorio (WIP, but PoC and demo available)
- Play Kerbal Space Program (announcement TBD)
- Co-play Helldivers 2 (WIP)
- Chat in Telegram
- Chat in Discord
- Memory
- Pure in-browser database support (DuckDB WASM |
pglite) - Memory Alaya (WIP)
- Pure in-browser database support (DuckDB WASM |
- Pure in-browser local (WebGPU) inference
- Ears
- Audio input from browser
- Audio input from Discord
- Client side speech recognition
- Client side talking detection
- Mouth
- Multi-provider voice synthesis, including ElevenLabs, Microsoft/Azure Speech, OpenAI-compatible TTS, Alibaba Cloud Model Studio, and local Kokoro TTS
- Body
- VRM support
- Control VRM model
- VRM model animations
- Auto blink
- Auto look at
- Idle eye movement
- Live2D support
- Control Live2D model
- Live2D model animations
- Auto blink
- Auto look at
- Idle eye movement
- VRM support
Development
For detailed instructions to develop this project, follow CONTRIBUTING.md
Note
By default,
pnpm devwill start the development server for the Stage Web (browser version). If you would like to try developing the desktop version, please make sure you read CONTRIBUTING.md to setup the environment correctly.
pnpm i
pnpm dev
Stage Web (Browser Version at airi.moeru.ai)
pnpm dev
Stage Tamagotchi (Desktop Version)
pnpm dev:tamagotchi
A Nix package for Tamagotchi is included. To run airi with Nix, first make sure to enable flakes, then run:
nix run github:moeru-ai/airi
NixOS
Electron requires shared libraries that aren't in standard paths on NixOS. Use the FHS shell defined in flake.nix:
nix develop .#fhs
pnpm dev:tamagotchi
Stage Pocket (Mobile Version)
Start the development server for the capacitor:
pnpm dev:pocket:ios --target "<DEVICE_ID_OR_SIMULATOR_NAME>"
# Or
CAPACITOR_DEVICE_ID_IOS="<DEVICE_ID_OR_SIMULATOR_NAME>" pnpm dev:pocket:ios
Quote the target: simulator names such as iPhone 16 Pro contain spaces.
You can see the list of available devices and simulators by running pnpm -F @proj-airi/stage-pocket exec cap run ios --list. @capacitor/cli is declared in apps/stage-pocket, so cap does not resolve from the repository root.
If you need to connect server channel on pocket in wireless mode, you need to start tamagotchi as root:
sudo pnpm dev:tamagotchi
Then enable secure websocket in tamagotchi settings/connections.
Documentation Site
pnpm dev:docs
Publish
Run bumpp to update the monorepo version:
npx bumpp --no-commit --no-tag
Support of LLM API Providers (powered by xsai)
- AIHubMix (recommended)
- OpenRouter
- vLLM
- SGLang
- Ollama
- 302.AI (sponsored)
- OpenAI
- Anthropic Claude
- AWS Claude (PR welcome)
- DeepSeek
- Qwen
- Google Gemini
- xAI
- Groq
- Mistral
- Cloudflare Workers AI
- Together.ai
- Fireworks.ai
- Novita
- Zhipu
- SiliconFlow
- Stepfun
- Baichuan
- Minimax
- Moonshot AI
- ModelScope
- Player2
- Tencent Cloud
- Sparks (PR welcome)
- Volcano Engine (PR welcome)
- Xiaomi Mimo
Sub-projects Born from This Project
- Awesome AI VTuber: A curated list of AI VTubers and related projects
unspeech: Universal endpoint proxy server for/audio/transcriptionsand/audio/speech, like LiteLLM but for any ASR and TTShfup: tools to help on deploying, bundling to HuggingFace Spacesxsai-transformers: Experimental 🤗 Transformers.js provider for xsAI.- WebAI: Realtime Voice Chat: Full example of implementing ChatGPT's realtime voice from scratch with VAD + STT + LLM + TTS.
@proj-airi/drizzle-duckdb-wasm: Drizzle ORM driver for DuckDB WASM@proj-airi/duckdb-wasm: Easy to use wrapper for@duckdb/duckdb-wasm- AIRI Factorio: Allow AIRI to play Factorio.
- AIRI DomeKeeper: Allow AIRI to play DomeKeeper.
- Factorio RCON API: RESTful API wrapper for Factorio headless server console
autorio: Factorio automation librarytstl-plugin-reload-factorio-mod: Reload Factorio mod when developing- Velin: Use Vue SFC and Markdown to write easy to manage stateful prompts for LLM
demodel: Easily boost the speed of pulling your models and datasets from various of inference runtimes.inventory: Centralized model catalog and default provider configurations backend service- MCP Launcher: Easy to use MCP builder & launcher for all possible MCP servers, just like Ollama for models!
- 🥺 SAD: Documentation and notes for self-host and browser running LLMs.
flowchart LR
subgraph Apps[Stage applications]
Web[stage-web]
Desktop[stage-tamagotchi]
Pocket[stage-pocket\nexperimental]
end
subgraph Shared[Shared product packages]
StageUI[stage-ui]
Domain[core-agent\ncore-character]
Audio[pipelines-audio]
Renderers[stage-ui-live2d\nstage-ui-three and renderers]
SDK[server-sdk\nserver-shared]
end
subgraph Channel[Desktop server channel]
Runtime[server-runtime]
end
subgraph Integrations[Source-configured integrations]
Discord[discord-bot]
Minecraft[minecraft-bot]
Other[Telegram, Factorio, MCP, and others]
end
subgraph Hosted[Hosted backend]
Edge[Caddy]
API[api-server]
Auth[auth-server]
Database[(PostgreSQL)]
Cache[(Redis)]
end
Web --> StageUI
Desktop --> StageUI
Pocket --> StageUI
StageUI --> Domain
StageUI --> Audio
StageUI --> Renderers
StageUI --> SDK
Desktop --> Runtime
SDK <-->|server channel| Runtime
Discord --> SDK
Minecraft --> SDK
Other -. optional .-> SDK
Edge --> API
Edge --> Auth
API --> Database
API --> Cache
Auth --> Database
Auth --> Cache
Similar Projects
Open sourced ones
- kimjammer/Neuro: A recreation of Neuro-Sama originally created in 7 days.: very well completed implementation.
- SugarcaneDefender/z-waif: Great at gaming, autonomous, and prompt engineering
- semperai/amica: Great at VRM, WebXR
- elizaOS/eliza: Great examples and software engineering on how to integrate agent into various of systems and APIs
- ardha27/AI-Waifu-Vtuber: Great about Twitch API integrations
- InsanityLabs/AIVTuber: Nice UI and UX
- IRedDragonICY/vixevia
- t41372/Open-LLM-VTuber
- PeterH0323/Streamer-Sales
Non-open-sourced ones
- https://clips.twitch.tv/WanderingCaringDeerDxCat-Qt55xtiGDSoNmDDr https://www.youtube.com/watch?v=8Giv5mupJNE
- https://clips.twitch.tv/TriangularAthleticBunnySoonerLater-SXpBk1dFso21VcWD
- https://www.youtube.com/@NOWA_Mirai
Project Status
Acknowledgements
- Reka UI: for designing the documentation site, the new landing page is based on this, as well as implementing a massive amount of UI components. (shadcn-vue is using Reka UI as the headless, do checkout!)
- pixiv/ChatVRM
- josephrocca/ChatVRM-js: A JS conversion/adaptation of parts of the ChatVRM (TypeScript) code for standalone use in OpenCharacters and elsewhere
- Design of UI and style was inspired by Cookard, UNBEATABLE, and Sensei! I like you so much!, and artworks of Ayame by Mercedes Bazan with Wish by Mercedes Bazan
- mallorbc/whisper_mic
xsai: Implemented a decent amount of packages to interact with LLMs and models, like Vercel AI SDK but way small.
Supporters
Thank you for supporting Project AIRI through OpenCollective, Patreon, and Ko-fi.
Special Thanks
Special thanks to all contributors for their contributions to Project AIRI ❤️