アイリ VTuber

[Try it] [简体中文文档]

Heavily inspired by Neuro-sama

Current progress

Capable of

• Brain
  • Play Minecraft
  • Play Factorio (WIP, but PoC and demo available)
  • Chat in Telegram
  • Chat in Discord
• Ears
  • Audio input from browser
  • Audio input from Discord
  • Client side speech recognition
  • Client side talking detection
• Mouth
  • ElevenLabs voice synthesis
• Body
  • VRM support
    • Control VRM model
  • VRM model animations
    • Auto blink
    • Auto look at
    • Idle eye movement
  • Live2D support
    • Control Live2D model
  • Live2D model animations
    • Auto blink
    • Auto look at
    • Idle eye movement

Development

pnpm i

pnpm dev

Sub-projects born from this project

• unspeech: Universal endpoint proxy server for /audio/transcriptions and /audio/speech, like LiteLLM but for any ASR and TTS
• hfup: tools to help on deploying, bundling to HuggingFace Spaces
• @proj-airi/drizzle-duckdb-wasm: Drizzle ORM driver for DuckDB WASM
• @proj-airi/duckdb-wasm: Easy to use wrapper for @duckdb/duckdb-wasm
• @proj-airi/lobe-icons: Iconify JSON bundle for amazing AI & LLM icons from lobe-icons, support Tailwind and UnoCSS
• @proj-airi/elevenlabs: TypeScript definitions for ElevenLabs API
• Airi Factorio: Allow Airi to play Factorio
• Factorio RCON API: RESTful API wrapper for Factorio headless server console
• autorio: Factorio automation library
• `tstl-plugin-reload-factorio-mod: Reload Factorio mod when developing
• 🥺 SAD: Documentation and notes for self-host and browser running LLMs

%%{ init: { 'flowchart': { 'curve': 'catmullRom' } } }%%

flowchart TD

  CORE("Core")
  UNSPEECH["unspeech"]
  DB1["@proj-airi/drizzle-duckdb-wasm"]
  DB2["[WIP] Memory Alaya"]
  DB0["@proj-airi/duckdb-wasm"]
  ICONS["@proj-airi/lobe-icons"]
  UI("UI")
  Stage("Stage")
  ELEVENLABS{{"@proj-airi/elevenlabs"}}
  F_AGENT("Factorio Agent")
  F_API["Factorio RCON API"]
  F_MOD1["autorio"]
  SVRT["@proj-airi/server-runtime"]
  MC_AGENT("Minecraft Agent")
  XSAI["xsai"]
  
  subgraph airi-vtuber
    DB0 --> DB1 --> DB2 --> CORE
    ICONS --> UI --> Stage --> CORE
    CORE --> ELEVENLABS
    CORE --> SVRT
  end
  ELEVENLABS --> |Speaking|UNSPEECH
  SVRT --> |Playing Factorio|F_AGENT

  subgraph Airi-Factorio
    F_AGENT --> F_API -..- factorio-server
    subgraph factorio-server-wrapper
      subgraph factorio-server
        F_MOD1
      end
    end
  end

  SVRT --> |Playing Minecraft|MC_AGENT -..- minecraft-server

  XSAI --> CORE
  XSAI --> F_AGENT
  XSAI --> MC_AGENT

%%{ init: { 'flowchart': { 'curve': 'catmullRom' } } }%%

flowchart TD
  subgraph deploy&bundle
    direction LR
    HFUP["hfup"]
    HF[/"HuggingFace Spaces"\]
    HFUP -...- UI -...-> HF
    HFUP -...- whisper-webgpu -...-> HF
    HFUP -...- moonshine-web -...-> HF
  end

Models used

• onnx-community/whisper-large-v3-turbo · Hugging Face

Similar Projects

• SugarcaneDefender/z-waif: Great at gaming, autonomous, and prompt engineering
• semperai/amica: Great at VRM, WebXR
• elizaOS/eliza: Great examples and software engineering on how to integrate agent into various of systems and APIs
• ardha27/AI-Waifu-Vtuber: Great about Twitch API integrations
• InsanityLabs/AIVTuber: Nice UI and UX
• IRedDragonICY/vixevia
• t41372/Open-LLM-VTuber
• PeterH0323/Streamer-Sales

Acknowledgements

• pixiv/ChatVRM
• josephrocca/ChatVRM-js: A JS conversion/adaptation of parts of the ChatVRM (TypeScript) code for standalone use in OpenCharacters and elsewhere
• Design of UI and style was inspired by Cookard, UNBEATABLE, and Sensei! I like you so much!, and artworks of Ayame by Mercedes Bazan with Wish by Mercedes Bazan
• mallorbc/whisper_mic
• xsai: Implemented a decent amount of packages to interact with LLMs and models, like Vercel AI SDK but a lot more smaller.