From aa69ee62b5b538e157b03e3b08364b572461fd45 Mon Sep 17 00:00:00 2001 From: leafyy Date: Mon, 31 Aug 2026 14:58:43 +0800 Subject: [PATCH] refactor(lipsync): share renderer vowel policy (#2374) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Description The VRM and MMD renderers had the same two boundary problems: - Both imported the Stage audio store from the `stage-ui` source tree. - Both kept copies of the wLipSync profile and model-neutral vowel policy. This PR solves both problems as one architecture change. ```text Stage owns AudioContext and the current audio source -> ThreeScene / MMDScene receive both as inputs -> model-driver-lipsync owns the profile and shared vowel policy -> VRM maps AEIOU to aa / ee / ih / oh / ou -> MMD maps AEIOU to vowelA / vowelE / vowelI / vowelO / vowelU ``` The shared driver owns viseme projection, winner and runner selection, silence handling, and smoothing. It has no Vue, Three.js, VRM, or MMD dependency. Each renderer keeps only its model-specific mapping. The renderer packages no longer import the Stage audio store or depend directly on `wlipsync`. Stage passes the owning `AudioContext` down with the current audio source. Cleanup disconnects only the wLipSync node owned by that renderer. ## Current solution The package-boundary review identified two related findings: - Finding 3: The VRM and MMD renderers imported `stage-ui` audio state through source paths. - Finding 4: The same renderers duplicated the wLipSync profile and model-neutral vowel policy. Both findings came from one missing boundary between Stage, the lip-sync driver, and the renderer adapters. This PR fixes that boundary with one ownership change. | Layer | Responsibility | | --- | --- | | `stage-ui` | Owns the `AudioContext` and the current audio source. It passes both values to each renderer. | | `stage-ui-three` and `stage-ui-mmd` | Connect renderer components to the shared driver. They do not import the Stage audio store. | | `model-driver-lipsync/runtime/wlipsync` | Exposes `createWLipSyncNode` as the public runtime entry. | | `model-driver-lipsync/shared/wlipsync` | Owns the profile, public types, and model-neutral vowel policy. | Runtime flow: 1. `Stage.vue` passes `audioContext` and `currentAudioSource` to `ThreeScene` or `MMDScene`. 2. The renderer composable watches the context and creates one wLipSync node for that context. 3. The source watcher connects the current audio source to that node. 4. Each render frame sends the wLipSync values to `createWLipSyncVowelDriver`. 5. The renderer adapter applies the returned AEIOU weights to its model controls. 6. Cleanup disconnects the exact source and node pair that the renderer owns. The shared policy keeps the current renderer behavior: - It maps the raw `S` viseme to `I`. - It selects the two strongest vowels. - It uses the existing volume and silence thresholds. - It keeps the 160 ms silence window. - It keeps the existing attack, release, cap, and output scale values. The renderer-specific mappings stay in their renderer packages: - VRM maps AEIOU to `aa`, `ee`, `ih`, `oh`, and `ou`. - MMD maps AEIOU to `vowelA`, `vowelE`, `vowelI`, `vowelO`, and `vowelU`. This solution removes both duplicate profile files. It also removes the direct `wlipsync` dependencies from both renderer packages. ## Linked Issues None. ## Additional Context There is no intended visual change. This refactor preserves the existing vowel weights and renderer mappings. Verification: - `sem diff --staged --no-cosmetics -v --file-exts .ts .tsx` - `sem diff --staged --no-cosmetics -v --file-exts .vue` - `sem impact --entity-id --json --dependents` - `sem impact --entity-id --json --dependents` - `pnpm -F @proj-airi/model-driver-lipsync test` — 3 tests passed - `pnpm -F @proj-airi/stage-ui-three exec vitest run` — 18 tests passed - `pnpm -F @proj-airi/stage-ui-mmd exec vitest run` — 23 tests passed - Type checks passed for `model-driver-lipsync`, `stage-ui-three`, `stage-ui-mmd`, and `stage-ui` - `pnpm install --frozen-lockfile --ignore-scripts` - `pnpm lint` — 0 errors and 7 existing warnings outside this change - `git diff --check` The full workspace `pnpm typecheck` was also run. In the isolated worktree it stopped in unchanged `electron-vueuse` and `plugin-sdk-tamagotchi` because their generated dependency outputs were not present. All affected package type checks above passed. --- packages/model-driver-lipsync/README.md | 49 +++++ packages/model-driver-lipsync/package.json | 8 + .../src/runtime/wlipsync/index.ts | 1 + .../src/shared/wlipsync/index.ts | 3 +- .../src/shared/wlipsync/vowel-driver.test.ts | 49 +++++ .../src/shared/wlipsync/vowel-driver.ts | 130 ++++++++++++ .../model-driver-lipsync/vitest.config.ts | 7 + packages/stage-ui-mmd/README.md | 1 + packages/stage-ui-mmd/package.json | 4 +- .../src/assets/lip-sync-profile.json | 1 - .../src/components/scenes/MMD.vue | 14 +- .../src/composables/mmd/lip-sync.ts | 186 +++++++----------- packages/stage-ui-three/README.md | 6 + packages/stage-ui-three/package.json | 4 +- .../src/assets/lip-sync-profile.json | 1 - .../src/components/Model/VRMModel.vue | 5 +- .../src/components/ThreeScene.vue | 3 + .../src/composables/vrm/lip-sync.ts | 182 +++++++---------- .../stage-ui/src/components/scenes/Stage.vue | 2 + pnpm-lock.yaml | 16 +- vitest.config.ts | 1 + 21 files changed, 428 insertions(+), 245 deletions(-) create mode 100644 packages/model-driver-lipsync/README.md create mode 100644 packages/model-driver-lipsync/src/runtime/wlipsync/index.ts create mode 100644 packages/model-driver-lipsync/src/shared/wlipsync/vowel-driver.test.ts create mode 100644 packages/model-driver-lipsync/src/shared/wlipsync/vowel-driver.ts create mode 100644 packages/model-driver-lipsync/vitest.config.ts delete mode 100644 packages/stage-ui-mmd/src/assets/lip-sync-profile.json delete mode 100644 packages/stage-ui-three/src/assets/lip-sync-profile.json diff --git a/packages/model-driver-lipsync/README.md b/packages/model-driver-lipsync/README.md new file mode 100644 index 000000000..2d6c89a1a --- /dev/null +++ b/packages/model-driver-lipsync/README.md @@ -0,0 +1,49 @@ +# `@proj-airi/model-driver-lipsync` + +Shared lip-sync profiles and model-neutral mouth-driving policies for AIRI. + +## What It Does + +- Exposes the shared wLipSync profile. +- Converts raw AEIOUS frames into stable AEIOU weights. +- Owns winner selection, silence detection, and weight smoothing. +- Provides the existing Live2D lip-sync driver. +- Exposes the browser-only wLipSync node factory through a separate runtime entry. + +The package does not write weights to VRM expressions or MMD morphs. Each renderer owns that mapping. + +## Exports + +- `@proj-airi/model-driver-lipsync`: the Live2D driver. +- `@proj-airi/model-driver-lipsync/shared/wlipsync`: the profile, types, and pure vowel driver. +- `@proj-airi/model-driver-lipsync/runtime/wlipsync`: the browser-only audio-node factory. + +The shared entry has no Web Audio side effects. Node-based tools and tests can import it safely. + +## How To Use It + +```ts +import { createWLipSyncNode } from '@proj-airi/model-driver-lipsync/runtime/wlipsync' +import { + createWLipSyncVowelDriver, + wlipsyncProfile, +} from '@proj-airi/model-driver-lipsync/shared/wlipsync' + +const node = await createWLipSyncNode(audioContext, wlipsyncProfile) +const driver = createWLipSyncVowelDriver() +const weights = driver.update(node, deltaSeconds) +``` + +The caller owns the `AudioContext`, the source node, and the source lifecycle. + +## When To Use It + +- Use the shared entry when a renderer needs standard AEIOU weights. +- Use the runtime entry when browser code creates a wLipSync audio node. +- Keep renderer-specific expression and morph mappings in the renderer package. + +## When Not To Use It + +- Do not import the runtime entry from Node-only code. +- Do not add Vue, Three.js, VRM, or MMD dependencies to this package. +- Do not move renderer-specific model writes into the shared driver. diff --git a/packages/model-driver-lipsync/package.json b/packages/model-driver-lipsync/package.json index dc4e64226..7346858b9 100644 --- a/packages/model-driver-lipsync/package.json +++ b/packages/model-driver-lipsync/package.json @@ -6,10 +6,18 @@ "exports": { ".": "./src/index.ts", "./live2d": "./src/live2d/index.ts", + "./runtime/wlipsync": "./src/runtime/wlipsync/index.ts", "./shared/wlipsync": "./src/shared/wlipsync/index.ts", "./shared/wlipsync/profile.json": "./src/shared/wlipsync/profile.json" }, + "scripts": { + "test": "vitest run", + "typecheck": "tsc --noEmit" + }, "dependencies": { "wlipsync": "catalog:" + }, + "devDependencies": { + "vitest": "catalog:vitest" } } diff --git a/packages/model-driver-lipsync/src/runtime/wlipsync/index.ts b/packages/model-driver-lipsync/src/runtime/wlipsync/index.ts new file mode 100644 index 000000000..a3d4fca84 --- /dev/null +++ b/packages/model-driver-lipsync/src/runtime/wlipsync/index.ts @@ -0,0 +1 @@ +export { createWLipSyncNode } from 'wlipsync' diff --git a/packages/model-driver-lipsync/src/shared/wlipsync/index.ts b/packages/model-driver-lipsync/src/shared/wlipsync/index.ts index 003d78958..233851a79 100644 --- a/packages/model-driver-lipsync/src/shared/wlipsync/index.ts +++ b/packages/model-driver-lipsync/src/shared/wlipsync/index.ts @@ -1,2 +1,3 @@ export { default as wlipsyncProfile } from './profile.json' with { type: 'json' } -export type { Profile } from 'wlipsync' +export * from './vowel-driver' +export type { Profile, WLipSyncAudioNode } from 'wlipsync' diff --git a/packages/model-driver-lipsync/src/shared/wlipsync/vowel-driver.test.ts b/packages/model-driver-lipsync/src/shared/wlipsync/vowel-driver.test.ts new file mode 100644 index 000000000..0e6b02f43 --- /dev/null +++ b/packages/model-driver-lipsync/src/shared/wlipsync/vowel-driver.test.ts @@ -0,0 +1,49 @@ +import { describe, expect, it } from 'vitest' + +import { createWLipSyncVowelDriver } from './index' + +describe('wLipSync vowel driver', () => { + it('maps S to I and returns only the two strongest vowels', () => { + const driver = createWLipSyncVowelDriver(() => 1000) + + const weights = driver.update({ + volume: 1, + weights: { + A: 1, + E: 0.8, + I: 0.1, + O: 0.6, + S: 0.9, + U: 0.4, + }, + }, 1) + + expect(weights.A).toBeCloseTo(0.49) + expect(weights.I).toBeCloseTo(0.245) + expect(weights.E).toBe(0) + expect(weights.O).toBe(0) + expect(weights.U).toBe(0) + }) + + it('uses the release rate when the audio becomes silent', () => { + let now = 1000 + const driver = createWLipSyncVowelDriver(() => now) + const activeWeights = driver.update({ volume: 1, weights: { A: 1 } }, 0.016) + + now += 16 + const silentWeights = driver.update({ volume: 0, weights: {} }, 0.016) + + expect(silentWeights.A).toBeGreaterThan(0) + expect(silentWeights.A).toBeLessThan(activeWeights.A) + }) + + it('resets all smoothing state', () => { + const driver = createWLipSyncVowelDriver(() => 1000) + driver.update({ volume: 1, weights: { A: 1 } }, 1) + + driver.reset() + const weights = driver.update({ volume: 0, weights: {} }, 1) + + expect(weights).toEqual({ A: 0, E: 0, I: 0, O: 0, U: 0 }) + }) +}) diff --git a/packages/model-driver-lipsync/src/shared/wlipsync/vowel-driver.ts b/packages/model-driver-lipsync/src/shared/wlipsync/vowel-driver.ts new file mode 100644 index 000000000..7ded821d5 --- /dev/null +++ b/packages/model-driver-lipsync/src/shared/wlipsync/vowel-driver.ts @@ -0,0 +1,130 @@ +const RAW_VISEMES = ['A', 'E', 'I', 'O', 'U', 'S'] as const + +/** The vowels that model renderers can apply to their mouth controls. */ +export const WLIP_SYNC_VOWELS = ['A', 'E', 'I', 'O', 'U'] as const + +/** The wLipSync values that the shared vowel driver reads for one frame. */ +export interface WLipSyncFrame { + /** The normalized volume from the wLipSync audio node. */ + volume?: number + /** The raw AEIOUS viseme weights from the wLipSync audio node. */ + weights: Readonly>> +} + +/** A vowel weight that a model renderer can apply to its mouth controls. */ +export type WLipSyncVowel = typeof WLIP_SYNC_VOWELS[number] + +/** + * Converts wLipSync frames into stable vowel weights for model renderers. + * + * The driver owns the winner selection, silence policy, and smoothing state. + * It does not know how a renderer stores or applies mouth controls. + */ +export interface WLipSyncVowelDriver { + /** Clears the smoothing and silence state. */ + reset: () => void + /** + * Reads one wLipSync frame and returns weights for the five vowels. + * + * @param frame - The current values from the wLipSync audio node. + * @param deltaSeconds - The elapsed render time. The default is 0.016 seconds. + */ + update: (frame: WLipSyncFrame, deltaSeconds?: number) => WLipSyncVowelWeights +} + +/** The normalized vowel weights that a renderer maps to model controls. */ +export type WLipSyncVowelWeights = Record + +type WLipSyncRawViseme = typeof RAW_VISEMES[number] + +const RAW_TO_VOWEL: Record = { + A: 'A', + E: 'E', + I: 'I', + O: 'O', + S: 'I', + U: 'U', +} + +/** + * Creates the shared vowel driver used by the VRM and MMD renderers. + * + * @param now - Returns the current monotonic time in milliseconds. + * + * @example + * const driver = createWLipSyncVowelDriver() + * driver.update({ volume: 1, weights: { A: 0.8, I: 0.2 } }, 1) + * // => { A: 0.49, E: 0, I: 0.078..., O: 0, U: 0 } + */ +export function createWLipSyncVowelDriver( + now: () => number = () => performance.now(), +): WLipSyncVowelDriver { + const smoothWeights = emptyVowelWeights() + let lastActiveAt = 0 + + function reset() { + for (const vowel of WLIP_SYNC_VOWELS) + smoothWeights[vowel] = 0 + lastActiveAt = 0 + } + + function update(frame: WLipSyncFrame, deltaSeconds = 0.016): WLipSyncVowelWeights { + const amplitude = Math.min((frame.volume ?? 0) * 0.9, 1) ** 0.7 + const projected = emptyVowelWeights() + + for (const rawViseme of RAW_VISEMES) { + const vowel = RAW_TO_VOWEL[rawViseme] + const rawWeight = frame.weights[rawViseme] ?? 0 + projected[vowel] = Math.max(projected[vowel], rawWeight * amplitude) + } + + let winner: WLipSyncVowel = 'I' + let runner: WLipSyncVowel = 'E' + let winnerWeight = -Infinity + let runnerWeight = -Infinity + + for (const vowel of WLIP_SYNC_VOWELS) { + const weight = projected[vowel] + if (weight > winnerWeight) { + runnerWeight = winnerWeight + runner = winner + winnerWeight = weight + winner = vowel + } + else if (weight > runnerWeight) { + runnerWeight = weight + runner = vowel + } + } + + const currentTime = now() + let silent = amplitude < 0.04 || winnerWeight < 0.05 + if (!silent) + lastActiveAt = currentTime + if (currentTime - lastActiveAt > 160) + silent = true + + const targetWeights = emptyVowelWeights() + if (!silent) { + targetWeights[winner] = Math.min(0.7, winnerWeight) + targetWeights[runner] = Math.min(0.35, runnerWeight * 0.6) + } + + const result = emptyVowelWeights() + for (const vowel of WLIP_SYNC_VOWELS) { + const from = smoothWeights[vowel] + const to = targetWeights[vowel] + const rate = 1 - Math.exp(-(to > from ? 50 : 30) * deltaSeconds) + smoothWeights[vowel] = from + (to - from) * rate + result[vowel] = (smoothWeights[vowel] <= 0.01 ? 0 : smoothWeights[vowel]) * 0.7 + } + + return result + } + + return { reset, update } +} + +function emptyVowelWeights(): WLipSyncVowelWeights { + return { A: 0, E: 0, I: 0, O: 0, U: 0 } +} diff --git a/packages/model-driver-lipsync/vitest.config.ts b/packages/model-driver-lipsync/vitest.config.ts new file mode 100644 index 000000000..647f39364 --- /dev/null +++ b/packages/model-driver-lipsync/vitest.config.ts @@ -0,0 +1,7 @@ +import { defineConfig } from 'vitest/config' + +export default defineConfig({ + test: { + include: ['src/**/*.test.ts'], + }, +}) diff --git a/packages/stage-ui-mmd/README.md b/packages/stage-ui-mmd/README.md index 05a34f353..3e7b96a00 100644 --- a/packages/stage-ui-mmd/README.md +++ b/packages/stage-ui-mmd/README.md @@ -45,6 +45,7 @@ import { MMDScene, useMMD } from '@proj-airi/stage-ui-mmd' ```vue () const clock = new Clock() let rafHandle = 0 -// Lip-sync owns Vue lifecycle hooks, so it must be created during setup. It -// is fed the live audio source and applied to whichever morphs are mounted. -const audioRef = shallowRef(props.currentAudioSource) -watch(() => props.currentAudioSource, v => audioRef.value = v) -const lipSync = useMMDLipSync(audioRef) +// Lip-sync owns Vue lifecycle hooks, so it must be created during setup. +const lipSync = useMMDLipSync( + toRef(props, 'audioContext'), + toRef(props, 'currentAudioSource'), +) const blink = useMMDBlink() let gaze: ReturnType | undefined diff --git a/packages/stage-ui-mmd/src/composables/mmd/lip-sync.ts b/packages/stage-ui-mmd/src/composables/mmd/lip-sync.ts index 54e4d128f..4f5111bbc 100644 --- a/packages/stage-ui-mmd/src/composables/mmd/lip-sync.ts +++ b/packages/stage-ui-mmd/src/composables/mmd/lip-sync.ts @@ -1,147 +1,97 @@ +import type { Profile, WLipSyncAudioNode, WLipSyncVowel } from '@proj-airi/model-driver-lipsync/shared/wlipsync' import type { Ref } from 'vue' -import type { Profile } from 'wlipsync' import type { VowelSlot } from '../../constants/morphs' import type { MorphController } from './morph' -import { useAsyncState } from '@vueuse/core' -import { onUnmounted, watch } from 'vue' -import { createWLipSyncNode } from 'wlipsync' +import { createWLipSyncNode } from '@proj-airi/model-driver-lipsync/runtime/wlipsync' +import { + createWLipSyncVowelDriver, + WLIP_SYNC_VOWELS, + wlipsyncProfile, +} from '@proj-airi/model-driver-lipsync/shared/wlipsync' +import { shallowRef, watch } from 'vue' -import profile from '../../assets/lip-sync-profile.json' with { type: 'json' } - -// NOTICE: -// Cross-package source import (no package.json dependency) to reach the shared -// AudioContext. Root cause: the audio context is owned by `@proj-airi/stage-ui`, -// which in turn depends on this package, so adding it as a dependency would -// create a cycle. The VRM renderer reaches the same store the same way. -// Source: packages/stage-ui-three/src/composables/vrm/lip-sync.ts. -// Removal condition: the AudioContext provider moves to a dependency-free -// shared package both renderers can import. -import { useAudioContext } from '../../../../stage-ui/src/stores/audio' - -/** wLipSync emits 6 visemes; we fold the sibilant `S` into `I`. */ -const RAW_KEYS = ['A', 'E', 'I', 'O', 'U', 'S'] as const -type LipKey = 'A' | 'E' | 'I' | 'O' | 'U' -const LIP_KEYS: LipKey[] = ['A', 'E', 'I', 'O', 'U'] - -/** Maps wLipSync visemes onto the MMD vowel morph slots (あいうえお). */ -const LIP_TO_SLOT: Record = { +const MMD_SLOT_BY_VOWEL: Record = { A: 'vowelA', E: 'vowelE', I: 'vowelI', O: 'vowelO', U: 'vowelU', } -const RAW_TO_LIP: Record = { - A: 'A', - E: 'E', - I: 'I', - O: 'O', - U: 'U', - S: 'I', -} - -const ATTACK = 50 // approach speed toward the next mouth shape -const RELEASE = 30 // decay speed when a shape ends -const CAP = 0.7 // max morph weight, mirrors the VRM tuning -const SILENCE_VOL = 0.04 -const SILENCE_GAIN = 0.05 -const IDLE_MS = 160 /** - * Audio-driven mouth animation for MMD models. + * Applies shared wLipSync vowel weights to MMD mouth morphs. * - * Reuses the exact wLipSync profile and winner/runner blending strategy as - * the VRM renderer (only the two strongest visemes are mixed, so the wide - * "A" shape does not dominate), but writes the result to MMD vowel morphs - * through a {@link MorphController} instead of VRM expressions. - * - * Returns an `update(delta)` to call once per frame, after the animation - * helper has run, so lip-sync wins over any VMD mouth keyframes. + * The caller owns the AudioContext and audio source lifecycle. This composable + * owns only the connection between that source and its wLipSync node. + * Call `update` after the MMD animation helper so lip-sync wins over VMD morphs. */ -export function useMMDLipSync(audioNode: Ref) { - const { audioContext } = useAudioContext() - const { state: lipSyncNode, isReady } = useAsyncState(createWLipSyncNode(audioContext, profile as Profile), undefined) +export function useMMDLipSync( + audioContext: Readonly>, + audioSource: Readonly>, +) { + const lipSyncNode = shallowRef() + const vowelDriver = createWLipSyncVowelDriver() - const smoothState: Record = { A: 0, E: 0, I: 0, O: 0, U: 0 } - let lastActiveAt = 0 - - watch([isReady, audioNode], ([ready, newAudioNode], [, oldAudioNode]) => { - if (oldAudioNode && oldAudioNode !== newAudioNode) { - try { - oldAudioNode.disconnect() - } - catch {} - } - if (!ready || !newAudioNode || !lipSyncNode.value) + watch(audioContext, (context, _, onCleanup) => { + lipSyncNode.value = undefined + vowelDriver.reset() + if (!context) return - try { - newAudioNode.connect(lipSyncNode.value) - } - catch {} + + let active = true + let createdNode: undefined | WLipSyncAudioNode + onCleanup(() => { + active = false + createdNode?.disconnect() + }) + + void createWLipSyncNode(context, wlipsyncProfile as Profile) + .then((node) => { + createdNode = node + if (!active) { + node.disconnect() + return + } + lipSyncNode.value = node + }) + .catch((error) => { + if (active) + console.error('[stage-ui-mmd] Failed to create the MMD lip-sync node.', error) + }) }, { immediate: true }) - onUnmounted(() => audioNode.value?.disconnect()) + watch([lipSyncNode, audioSource], ([node, source], _, onCleanup) => { + if (!node || !source) + return + + try { + source.connect(node) + } + catch (error) { + console.error('[stage-ui-mmd] Failed to connect the MMD lip-sync node.', error) + return + } + + onCleanup(() => { + try { + source.disconnect(node) + } + catch { + // The source can end before Vue runs this watcher cleanup. + } + }) + }, { immediate: true }) function update(morphs: MorphController | undefined, delta = 0.016) { const node = lipSyncNode.value if (!morphs || !node) return - const vol = node.volume ?? 0 - const amp = Math.min(vol * 0.9, 1) ** 0.7 - - // Project the 6 raw visemes down to 5 vowels, scaled by amplitude. - const projected: Record = { A: 0, E: 0, I: 0, O: 0, U: 0 } - for (const raw of RAW_KEYS) { - const lip = RAW_TO_LIP[raw] - const rawVal = node.weights[raw] ?? 0 - projected[lip] = Math.max(projected[lip], rawVal * amp) - } - - // Only blend the two strongest vowels. Mixing all five biases toward the - // wide "A" shape because it has the largest deformation. - let winner: LipKey = 'I' - let runner: LipKey = 'E' - let winnerVal = -Infinity - let runnerVal = -Infinity - for (const key of LIP_KEYS) { - const val = projected[key] - if (val > winnerVal) { - runnerVal = winnerVal - runner = winner - winnerVal = val - winner = key - } - else if (val > runnerVal) { - runnerVal = val - runner = key - } - } - - // Treat low energy / brief gaps as silence so the mouth fully closes. - const now = performance.now() - let silent = amp < SILENCE_VOL || winnerVal < SILENCE_GAIN - if (!silent) - lastActiveAt = now - if (now - lastActiveAt > IDLE_MS) - silent = true - - const target: Record = { A: 0, E: 0, I: 0, O: 0, U: 0 } - if (!silent) { - target[winner] = Math.min(CAP, winnerVal) - target[runner] = Math.min(CAP * 0.5, runnerVal * 0.6) - } - - for (const key of LIP_KEYS) { - const from = smoothState[key] - const to = target[key] - const rate = 1 - Math.exp(-(to > from ? ATTACK : RELEASE) * delta) - smoothState[key] = from + (to - from) * rate - const weight = (smoothState[key] <= 0.01 ? 0 : smoothState[key]) * 0.7 - morphs.set(LIP_TO_SLOT[key], weight) - } + const weights = vowelDriver.update(node, delta) + for (const vowel of WLIP_SYNC_VOWELS) + morphs.set(MMD_SLOT_BY_VOWEL[vowel], weights[vowel]) } return { update } diff --git a/packages/stage-ui-three/README.md b/packages/stage-ui-three/README.md index 4a282ce46..b532a6e07 100644 --- a/packages/stage-ui-three/README.md +++ b/packages/stage-ui-three/README.md @@ -32,6 +32,12 @@ VRM runtime management is explicit in this package. This keeps ordinary remounts and HMR from immediately forcing deep VRM disposal, while still making model switches deterministic. +## Lip-Sync Boundary + +`ThreeScene` receives the `AudioContext` and current audio source from its caller. The package does not import the Stage audio store. + +The shared driver produces AEIOU weights. The VRM adapter maps these weights to `aa`, `ee`, `ih`, `oh`, and `ou` expressions. + ## Scene Lifecycle `ThreeScene` coordinates two independent async readiness signals before the scene becomes interactive: diff --git a/packages/stage-ui-three/package.json b/packages/stage-ui-three/package.json index 3bf2a841a..336c65d6e 100644 --- a/packages/stage-ui-three/package.json +++ b/packages/stage-ui-three/package.json @@ -29,6 +29,7 @@ "@pixiv/three-vrm": "catalog:", "@pixiv/three-vrm-animation": "catalog:", "@pixiv/three-vrm-core": "catalog:", + "@proj-airi/model-driver-lipsync": "workspace:*", "@proj-airi/stage-shared": "workspace:*", "@proj-airi/ui": "workspace:*", "@tresjs/cientos": "catalog:", @@ -40,8 +41,7 @@ "pinia": "catalog:", "postprocessing": "catalog:", "three": "catalog:", - "vue": "catalog:", - "wlipsync": "catalog:" + "vue": "catalog:" }, "devDependencies": { "@types/culori": "catalog:", diff --git a/packages/stage-ui-three/src/assets/lip-sync-profile.json b/packages/stage-ui-three/src/assets/lip-sync-profile.json deleted file mode 100644 index 879765368..000000000 --- a/packages/stage-ui-three/src/assets/lip-sync-profile.json +++ /dev/null @@ -1 +0,0 @@ -{ "jsonPath": "/home/steamvr/projects/wLipSync/www/profile.json", "mfccNum": 12, "mfccDataCount": 12, "melFilterBankChannels": 30, "targetSampleRate": 16000, "sampleCount": 1024, "useStandardization": false, "compareMethod": 2, "mfccs": [{ "name": "A", "mfccCalibrationDataList": [{ "array": [94.40318298339844, 0.32245922088623049, -65.5116195678711, -29.537851333618165, 4.888294219970703, 14.523965835571289, -32.6411247253418, 1.6505765914916993, -9.960077285766602, -5.7025322914123539, -11.886154174804688, -24.35236358642578] }, { "array": [94.40318298339844, 0.32245922088623049, -65.5116195678711, -29.537851333618165, 4.888294219970703, 14.523965835571289, -32.6411247253418, 1.6505765914916993, -9.960077285766602, -5.7025322914123539, -11.886154174804688, -24.35236358642578] }, { "array": [102.16287231445313, -3.3587560653686525, -65.58428192138672, -25.24440574645996, 3.224522590637207, 12.005892753601075, -29.293079376220704, 0.6378564834594727, -10.817683219909668, -1.3263540267944337, -14.543159484863282, -24.169780731201173] }, { "array": [99.35592651367188, -3.681424140930176, -65.20439910888672, -23.45950698852539, 6.205645561218262, 14.96288013458252, -29.882709503173829, 0.6733551025390625, -7.077619552612305, -3.5570802688598635, -14.427347183227539, -23.340003967285158] }, { "array": [99.35592651367188, -3.681424140930176, -65.20439910888672, -23.45950698852539, 6.205645561218262, 14.96288013458252, -29.882709503173829, 0.6733551025390625, -7.077619552612305, -3.5570802688598635, -14.427347183227539, -23.340003967285158] }, { "array": [104.24951171875, -2.8328847885131838, -66.59016418457031, -22.962886810302736, 5.519782066345215, 16.50394058227539, -32.338768005371097, 5.820473670959473, -10.59586238861084, -2.7398462295532228, -12.5281400680542, -24.459365844726564] }, { "array": [104.24951171875, -2.8328847885131838, -66.59016418457031, -22.962886810302736, 5.519782066345215, 16.50394058227539, -32.338768005371097, 5.820473670959473, -10.59586238861084, -2.7398462295532228, -12.5281400680542, -24.459365844726564] }, { "array": [95.58644104003906, -5.775191307067871, -61.220008850097659, -24.658382415771486, 4.4112701416015629, 13.673284530639649, -25.223039627075197, -2.0546646118164064, -6.887641906738281, -5.683987617492676, -11.20918083190918, -23.215322494506837] }, { "array": [98.24864196777344, -3.8367862701416017, -62.34006118774414, -24.563793182373048, 4.608433723449707, 16.228965759277345, -28.992279052734376, 2.1237001419067385, -9.07174015045166, -4.581008434295654, -10.662440299987793, -26.19581413269043] }, { "array": [102.7921142578125, -4.580304145812988, -62.531837463378909, -26.292770385742189, 7.911410331726074, 17.136384963989259, -31.118263244628908, 5.196089744567871, -10.010396957397461, -0.8527965545654297, -12.346561431884766, -23.580944061279298] }, { "array": [102.12345123291016, -1.5254135131835938, -62.21220397949219, -26.728734970092775, 10.62057876586914, 16.918357849121095, -28.815664291381837, 3.6714258193969728, -9.673786163330079, -0.7385025024414063, -9.717185020446778, -27.09702491760254] }, { "array": [102.12345123291016, -1.5254135131835938, -62.21220397949219, -26.728734970092775, 10.62057876586914, 16.918357849121095, -28.815664291381837, 3.6714258193969728, -9.673786163330079, -0.7385025024414063, -9.717185020446778, -27.09702491760254] }] }, { "name": "I", "mfccCalibrationDataList": [{ "array": [14.441835403442383, 52.400115966796878, 57.377838134765628, -33.70046615600586, -12.751934051513672, -15.709930419921875, -41.381065368652347, -12.190519332885743, -2.863154411315918, -8.727733612060547, 2.6656012535095217, 1.4855976104736329] }, { "array": [14.441835403442383, 52.400115966796878, 57.377838134765628, -33.70046615600586, -12.751934051513672, -15.709930419921875, -41.381065368652347, -12.190519332885743, -2.863154411315918, -8.727733612060547, 2.6656012535095217, 1.4855976104736329] }, { "array": [15.294279098510743, 50.07628631591797, 57.262847900390628, -31.748844146728517, -13.642471313476563, -13.48408031463623, -41.535011291503909, -16.863862991333009, -1.739903450012207, -9.32723331451416, 8.31618881225586, -1.779850959777832] }, { "array": [15.685098648071289, 49.68647766113281, 58.447113037109378, -32.513519287109378, -16.664287567138673, -13.78364372253418, -40.48309326171875, -16.04582405090332, -3.5356569290161135, -8.654275894165039, 10.645575523376465, -2.556441307067871] }, { "array": [15.685098648071289, 49.68647766113281, 58.447113037109378, -32.513519287109378, -16.664287567138673, -13.78364372253418, -40.48309326171875, -16.04582405090332, -3.5356569290161135, -8.654275894165039, 10.645575523376465, -2.556441307067871] }, { "array": [15.685098648071289, 49.68647766113281, 58.447113037109378, -32.513519287109378, -16.664287567138673, -13.78364372253418, -40.48309326171875, -16.04582405090332, -3.5356569290161135, -8.654275894165039, 10.645575523376465, -2.556441307067871] }, { "array": [15.225826263427735, 43.872196197509769, 52.844512939453128, -32.51786804199219, -17.806241989135743, -10.609650611877442, -40.13084411621094, -11.58648681640625, -5.082568168640137, -14.396997451782227, 6.896979331970215, -0.785430908203125] }, { "array": [15.571629524230957, 48.634883880615237, 59.5339469909668, -34.59955596923828, -21.959871292114259, -10.298498153686524, -39.11286926269531, -11.998537063598633, -8.433327674865723, -10.80599594116211, 8.789299011230469, -1.884697437286377] }, { "array": [15.571629524230957, 48.634883880615237, 59.5339469909668, -34.59955596923828, -21.959871292114259, -10.298498153686524, -39.11286926269531, -11.998537063598633, -8.433327674865723, -10.80599594116211, 8.789299011230469, -1.884697437286377] }, { "array": [15.571629524230957, 48.634883880615237, 59.5339469909668, -34.59955596923828, -21.959871292114259, -10.298498153686524, -39.11286926269531, -11.998537063598633, -8.433327674865723, -10.80599594116211, 8.789299011230469, -1.884697437286377] }, { "array": [19.23290252685547, 47.433998107910159, 54.90937423706055, -33.7783203125, -13.836353302001954, -5.141571044921875, -39.34584045410156, -13.409493446350098, -4.945652008056641, -12.960502624511719, 12.210061073303223, 0.5807018280029297] }, { "array": [19.124774932861329, 46.46723937988281, 53.41281509399414, -34.65093994140625, -18.181049346923829, -7.733134746551514, -45.67931365966797, -10.64135456085205, -2.624391555786133, -15.708955764770508, 7.3649187088012699, -5.627689361572266] }] }, { "name": "U", "mfccCalibrationDataList": [{ "array": [83.38372802734375, 42.39790725708008, 27.812450408935548, 10.696150779724121, -13.612553596496582, -32.487091064453128, -35.2574348449707, -6.425739288330078, -4.214997291564941, -6.896385669708252, -3.49631404876709, 4.997060775756836] }, { "array": [103.65653228759766, 38.661563873291019, 30.985050201416017, 17.432451248168947, -14.383820533752442, -39.810001373291019, -39.63761901855469, 1.2333955764770508, -4.217883110046387, -3.005303382873535, -6.272947311401367, 4.751875877380371] }, { "array": [103.44242095947266, 45.238216400146487, 27.622669219970704, 18.682138442993165, -16.854982376098634, -39.85029602050781, -34.15940856933594, -2.7482595443725588, -3.7410573959350588, -1.0625238418579102, -4.215768814086914, 6.514510154724121] }, { "array": [103.44242095947266, 45.238216400146487, 27.622669219970704, 18.682138442993165, -16.854982376098634, -39.85029602050781, -34.15940856933594, -2.7482595443725588, -3.7410573959350588, -1.0625238418579102, -4.215768814086914, 6.514510154724121] }, { "array": [99.99739837646485, 39.48998260498047, 26.057384490966798, 23.26814079284668, -16.25522804260254, -38.15496063232422, -35.70051574707031, -1.2821111679077149, -2.3941946029663088, -0.15543842315673829, -4.757769584655762, 2.198577880859375] }, { "array": [99.99739837646485, 39.48998260498047, 26.057384490966798, 23.26814079284668, -16.25522804260254, -38.15496063232422, -35.70051574707031, -1.2821111679077149, -2.3941946029663088, -0.15543842315673829, -4.757769584655762, 2.198577880859375] }, { "array": [107.09538269042969, 37.704010009765628, 17.795482635498048, 21.882326126098634, -14.739266395568848, -36.407527923583987, -37.95854949951172, -1.4393510818481446, -1.9593324661254883, -0.7294750213623047, -7.93386173248291, 3.9560585021972658] }, { "array": [103.5069351196289, 35.08988952636719, 23.21630859375, 23.947580337524415, -14.157055854797364, -38.546836853027347, -39.75208282470703, 0.516876220703125, -2.715259552001953, -4.0768208503723148, -4.716378211975098, 4.662134170532227] }, { "array": [103.5069351196289, 35.08988952636719, 23.21630859375, 23.947580337524415, -14.157055854797364, -38.546836853027347, -39.75208282470703, 0.516876220703125, -2.715259552001953, -4.0768208503723148, -4.716378211975098, 4.662134170532227] }, { "array": [103.5069351196289, 35.08988952636719, 23.21630859375, 23.947580337524415, -14.157055854797364, -38.546836853027347, -39.75208282470703, 0.516876220703125, -2.715259552001953, -4.0768208503723148, -4.716378211975098, 4.662134170532227] }, { "array": [98.43344116210938, 35.42580032348633, 29.2958984375, 24.73729133605957, -15.485936164855957, -44.676483154296878, -39.978858947753909, 0.5548343658447266, -2.2034664154052736, -3.485844612121582, -7.421210289001465, 6.30616569519043] }, { "array": [94.0390625, 36.81925582885742, 24.73573875427246, 22.579418182373048, -14.354126930236817, -37.92849349975586, -44.69046401977539, 0.7474861145019531, -3.3195743560791017, -3.9850082397460939, -5.991059303283691, 6.134122848510742] }] }, { "name": "E", "mfccCalibrationDataList": [{ "array": [60.52040481567383, 14.444153785705567, 50.91899108886719, 6.730878829956055, -58.12107467651367, -16.403745651245118, -25.244909286499025, 5.399906158447266, -7.63681697845459, -2.4964828491210939, 7.271292209625244, 1.7322711944580079] }, { "array": [59.327247619628909, 14.82846450805664, 51.402244567871097, 5.413976669311523, -55.603240966796878, -15.348665237426758, -25.923606872558595, 3.0006580352783205, -4.183259963989258, -3.4587841033935549, 7.941498756408691, 3.4499120712280275] }, { "array": [59.327247619628909, 14.82846450805664, 51.402244567871097, 5.413976669311523, -55.603240966796878, -15.348665237426758, -25.923606872558595, 3.0006580352783205, -4.183259963989258, -3.4587841033935549, 7.941498756408691, 3.4499120712280275] }, { "array": [64.61061096191406, 7.8438310623168949, 54.753726959228519, 11.154451370239258, -62.99680709838867, -10.397377967834473, -36.124359130859378, 12.57413387298584, -6.086113452911377, -3.032306671142578, 10.453157424926758, -0.00012826919555664063] }, { "array": [59.8906135559082, 13.646936416625977, 53.14240646362305, 11.346290588378907, -60.17724609375, -15.942718505859375, -29.547088623046876, 8.241331100463868, -6.8904523849487309, -3.6554131507873537, 14.714229583740235, -2.811859607696533] }, { "array": [53.77520751953125, 6.071747779846191, 47.870723724365237, 5.943275451660156, -51.11442184448242, -16.625276565551759, -24.842336654663087, 4.076478004455566, -8.835965156555176, -4.306196689605713, 13.907751083374024, -0.6555676460266113] }, { "array": [53.77520751953125, 6.071747779846191, 47.870723724365237, 5.943275451660156, -51.11442184448242, -16.625276565551759, -24.842336654663087, 4.076478004455566, -8.835965156555176, -4.306196689605713, 13.907751083374024, -0.6555676460266113] }, { "array": [53.77520751953125, 6.071747779846191, 47.870723724365237, 5.943275451660156, -51.11442184448242, -16.625276565551759, -24.842336654663087, 4.076478004455566, -8.835965156555176, -4.306196689605713, 13.907751083374024, -0.6555676460266113] }, { "array": [62.81553649902344, 6.203365325927734, 48.45057678222656, 8.571174621582032, -53.907508850097659, -16.376169204711915, -25.989578247070314, 5.736949920654297, -8.150140762329102, -5.895424842834473, 13.745902061462403, -4.22935676574707] }, { "array": [53.43303680419922, 7.019550323486328, 43.32084655761719, 7.639513969421387, -49.81471633911133, -18.708377838134767, -21.690540313720704, 0.34458446502685549, -8.689970970153809, -3.96992826461792, 11.29841423034668, -4.165286540985107] }, { "array": [56.29218673706055, 7.115049362182617, 47.741546630859378, 4.102975845336914, -50.46143341064453, -18.235626220703126, -22.557659149169923, 6.925202369689941, -13.170380592346192, -1.0300326347351075, 8.813325881958008, -3.2347850799560549] }, { "array": [56.29218673706055, 7.115049362182617, 47.741546630859378, 4.102975845336914, -50.46143341064453, -18.235626220703126, -22.557659149169923, 6.925202369689941, -13.170380592346192, -1.0300326347351075, 8.813325881958008, -3.2347850799560549] }] }, { "name": "O", "mfccCalibrationDataList": [{ "array": [108.12348937988281, 57.48288345336914, 0.23154354095458985, -31.144771575927736, -38.093109130859378, -18.33026885986328, -11.250101089477539, -0.7636222839355469, -1.371236801147461, -9.181392669677735, -1.6202507019042969, -7.501105308532715] }, { "array": [108.12348937988281, 57.48288345336914, 0.23154354095458985, -31.144771575927736, -38.093109130859378, -18.33026885986328, -11.250101089477539, -0.7636222839355469, -1.371236801147461, -9.181392669677735, -1.6202507019042969, -7.501105308532715] }, { "array": [120.48326873779297, 61.383270263671878, -2.6785545349121095, -32.3900146484375, -40.94635772705078, -12.681024551391602, -10.979912757873536, -0.7160100936889648, -2.9078426361083986, -12.300739288330079, 1.8719825744628907, -6.2853875160217289] }, { "array": [109.58128356933594, 58.46321105957031, 0.1479501724243164, -31.080230712890626, -34.73053741455078, -17.423336029052736, -10.221624374389649, 0.3863086700439453, -1.2579708099365235, -7.037452220916748, -2.894855499267578, -7.0971550941467289] }, { "array": [109.58128356933594, 58.46321105957031, 0.1479501724243164, -31.080230712890626, -34.73053741455078, -17.423336029052736, -10.221624374389649, 0.3863086700439453, -1.2579708099365235, -7.037452220916748, -2.894855499267578, -7.0971550941467289] }, { "array": [116.19681549072266, 60.3573112487793, 0.07350921630859375, -32.384727478027347, -34.47290802001953, -16.373615264892579, -14.695085525512696, 0.3102083206176758, 0.28844451904296877, -8.948881149291993, -3.104994773864746, -9.900266647338868] }, { "array": [116.19681549072266, 60.3573112487793, 0.07350921630859375, -32.384727478027347, -34.47290802001953, -16.373615264892579, -14.695085525512696, 0.3102083206176758, 0.28844451904296877, -8.948881149291993, -3.104994773864746, -9.900266647338868] }, { "array": [98.63448333740235, 48.661781311035159, 2.394181251525879, -28.785797119140626, -31.548860549926759, -18.37759017944336, -14.998208999633789, -1.8050260543823243, -2.018402099609375, -4.584748268127441, -5.160560607910156, -7.968695163726807] }, { "array": [124.25032043457031, 59.27610397338867, 2.7454710006713869, -36.72577667236328, -38.65552520751953, -3.116687774658203, -24.24558448791504, 0.5085678100585938, 2.3633852005004885, -10.51361083984375, 1.7447805404663087, -13.22685432434082] }, { "array": [87.89213562011719, 45.08750534057617, 4.292821884155273, -26.482845306396486, -30.386096954345704, -20.410654067993165, -11.817208290100098, -3.1270408630371095, -1.1370172500610352, -6.159217357635498, -3.454045295715332, -5.7265400886535648] }, { "array": [87.89213562011719, 45.08750534057617, 4.292821884155273, -26.482845306396486, -30.386096954345704, -20.410654067993165, -11.817208290100098, -3.1270408630371095, -1.1370172500610352, -6.159217357635498, -3.454045295715332, -5.7265400886535648] }, { "array": [87.89213562011719, 45.08750534057617, 4.292821884155273, -26.482845306396486, -30.386096954345704, -20.410654067993165, -11.817208290100098, -3.1270408630371095, -1.1370172500610352, -6.159217357635498, -3.454045295715332, -5.7265400886535648] }] }, { "name": "S", "mfccCalibrationDataList": [{ "array": [-94.29214477539063, 9.299236297607422, -21.6169376373291, 4.123956203460693, 1.3645498752593995, -2.339733839035034, 12.92388916015625, -7.490560531616211, 10.520170211791993, -5.4832611083984379, 2.1621110439300539, 2.6961774826049806] }, { "array": [-94.29214477539063, 9.299236297607422, -21.6169376373291, 4.123956203460693, 1.3645498752593995, -2.339733839035034, 12.92388916015625, -7.490560531616211, 10.520170211791993, -5.4832611083984379, 2.1621110439300539, 2.6961774826049806] }, { "array": [-92.13811492919922, 8.712703704833985, -17.194181442260743, 4.387561798095703, 2.288078784942627, -9.562786102294922, 9.854814529418946, -7.843216896057129, 8.969221115112305, -7.9670305252075199, 0.38271141052246096, 2.731431007385254] }, { "array": [-92.13811492919922, 8.712703704833985, -17.194181442260743, 4.387561798095703, 2.288078784942627, -9.562786102294922, 9.854814529418946, -7.843216896057129, 8.969221115112305, -7.9670305252075199, 0.38271141052246096, 2.731431007385254] }, { "array": [-90.87933349609375, 12.622742652893067, -12.25172233581543, 6.775156497955322, 6.892632007598877, -4.708589553833008, 10.558273315429688, -10.194192886352539, 10.907587051391602, 0.7259445190429688, 6.631969451904297, 1.2803831100463868] }, { "array": [-90.87933349609375, 12.622742652893067, -12.25172233581543, 6.775156497955322, 6.892632007598877, -4.708589553833008, 10.558273315429688, -10.194192886352539, 10.907587051391602, 0.7259445190429688, 6.631969451904297, 1.2803831100463868] }, { "array": [-94.86066436767578, 18.40726089477539, -5.981902599334717, 7.446142196655273, 11.998884201049805, 1.1316719055175782, 5.726372718811035, -9.411809921264649, 9.966836929321289, 0.6692547798156738, 3.0949947834014894, -0.5439543724060059] }, { "array": [-94.86066436767578, 18.40726089477539, -5.981902599334717, 7.446142196655273, 11.998884201049805, 1.1316719055175782, 5.726372718811035, -9.411809921264649, 9.966836929321289, 0.6692547798156738, 3.0949947834014894, -0.5439543724060059] }, { "array": [-94.86066436767578, 18.40726089477539, -5.981902599334717, 7.446142196655273, 11.998884201049805, 1.1316719055175782, 5.726372718811035, -9.411809921264649, 9.966836929321289, 0.6692547798156738, 3.0949947834014894, -0.5439543724060059] }, { "array": [-100.9681396484375, 15.002283096313477, -4.994745254516602, 0.22259855270385743, -0.15606117248535157, -1.8661277294158936, 1.5652005672454835, -13.30648422241211, 12.554527282714844, -2.990779399871826, 3.5510034561157228, 5.119507312774658] }, { "array": [-100.9681396484375, 15.002283096313477, -4.994745254516602, 0.22259855270385743, -0.15606117248535157, -1.8661277294158936, 1.5652005672454835, -13.30648422241211, 12.554527282714844, -2.990779399871826, 3.5510034561157228, 5.119507312774658] }, { "array": [-101.28047943115235, 14.962509155273438, -10.988410949707032, 3.6384878158569338, -1.4698257446289063, -4.758091449737549, -1.3547701835632325, -12.941855430603028, 3.3519961833953859, -5.5131611824035648, 9.386914253234864, 3.8310816287994386] }] }, { "name": "A", "mfccCalibrationDataList": [{ "array": [4.20286750793457, -73.493896484375, -24.746726989746095, -41.51460266113281, 36.48657989501953, -18.2531795501709, -42.99116516113281, 25.612823486328126, -18.336681365966798, -15.366691589355469, -4.867555618286133, -8.545194625854493] }, { "array": [6.645953178405762, -70.99688720703125, -23.49920082092285, -40.70307922363281, 35.09113311767578, -19.63579750061035, -41.851219177246097, 26.548370361328126, -20.361244201660158, -15.091900825500489, -5.332237243652344, -7.199653625488281] }, { "array": [6.645953178405762, -70.99688720703125, -23.49920082092285, -40.70307922363281, 35.09113311767578, -19.63579750061035, -41.851219177246097, 26.548370361328126, -20.361244201660158, -15.091900825500489, -5.332237243652344, -7.199653625488281] }, { "array": [3.4860363006591799, -73.30689239501953, -21.244325637817384, -38.725765228271487, 41.22803497314453, -15.975458145141602, -42.09079360961914, 27.367385864257814, -19.52243995666504, -16.397735595703126, -3.7795209884643556, -4.958502292633057] }, { "array": [0.09921550750732422, -70.22235107421875, -20.888980865478517, -33.53620910644531, 48.01523208618164, -13.365336418151856, -42.88488006591797, 26.451934814453126, -13.952343940734864, -16.65743064880371, -1.894343376159668, -1.8812918663024903] }, { "array": [0.09921550750732422, -70.22235107421875, -20.888980865478517, -33.53620910644531, 48.01523208618164, -13.365336418151856, -42.88488006591797, 26.451934814453126, -13.952343940734864, -16.65743064880371, -1.894343376159668, -1.8812918663024903] }, { "array": [0.09921550750732422, -70.22235107421875, -20.888980865478517, -33.53620910644531, 48.01523208618164, -13.365336418151856, -42.88488006591797, 26.451934814453126, -13.952343940734864, -16.65743064880371, -1.894343376159668, -1.8812918663024903] }, { "array": [-4.117016792297363, -72.00682830810547, -19.490331649780275, -34.48163986206055, 51.383995056152347, -9.989368438720704, -41.8690185546875, 23.824867248535158, -11.14547061920166, -14.500547409057618, -2.504335403442383, 1.0616645812988282] }, { "array": [-3.0772790908813478, -77.95940399169922, -21.169992446899415, -43.19057846069336, 47.594181060791019, -14.239681243896485, -43.05297088623047, 23.796558380126954, -16.033828735351564, -14.308491706848145, -4.90071964263916, -0.6597251892089844] }, { "array": [-3.0772790908813478, -77.95940399169922, -21.169992446899415, -43.19057846069336, 47.594181060791019, -14.239681243896485, -43.05297088623047, 23.796558380126954, -16.033828735351564, -14.308491706848145, -4.90071964263916, -0.6597251892089844] }, { "array": [0.27674388885498049, -74.16876220703125, -19.241043090820314, -43.69765853881836, 45.94886779785156, -15.777923583984375, -40.226318359375, 25.209468841552736, -19.91909408569336, -14.123311042785645, -6.749327659606934, -2.186051368713379] }, { "array": [1.313084602355957, -70.17343139648438, -20.149150848388673, -40.1507568359375, 43.281288146972659, -17.598236083984376, -39.989742279052737, 19.475574493408204, -18.73434066772461, -15.377893447875977, -3.6761083602905275, -2.3720903396606447] }] }, { "name": "I", "mfccCalibrationDataList": [{ "array": [-43.23860549926758, 15.659211158752442, 48.96119689941406, -84.8009033203125, -2.832998275756836, -25.101383209228517, -19.283388137817384, -1.3310365676879883, -3.9067935943603517, -5.5547566413879398, -24.206161499023439, 19.690078735351564] }, { "array": [-43.23860549926758, 15.659211158752442, 48.96119689941406, -84.8009033203125, -2.832998275756836, -25.101383209228517, -19.283388137817384, -1.3310365676879883, -3.9067935943603517, -5.5547566413879398, -24.206161499023439, 19.690078735351564] }, { "array": [-43.23860549926758, 15.659211158752442, 48.96119689941406, -84.8009033203125, -2.832998275756836, -25.101383209228517, -19.283388137817384, -1.3310365676879883, -3.9067935943603517, -5.5547566413879398, -24.206161499023439, 19.690078735351564] }, { "array": [-35.28649139404297, 11.94938850402832, 46.56689453125, -80.19161987304688, -5.332358360290527, -16.747013092041017, -17.014617919921876, -6.106320381164551, -3.5759763717651369, -11.297148704528809, -19.73563575744629, 19.95285415649414] }, { "array": [-35.28649139404297, 11.94938850402832, 46.56689453125, -80.19161987304688, -5.332358360290527, -16.747013092041017, -17.014617919921876, -6.106320381164551, -3.5759763717651369, -11.297148704528809, -19.73563575744629, 19.95285415649414] }, { "array": [-33.85117721557617, 12.748700141906739, 48.141944885253909, -75.80270385742188, -0.9702749252319336, -14.077031135559082, -15.911153793334961, -4.433165073394775, -4.00740909576416, -9.756240844726563, -20.910476684570314, 18.42697525024414] }, { "array": [-29.72378921508789, 13.039468765258789, 47.7448616027832, -74.11089324951172, 0.8275318145751953, -16.012189865112306, -17.36796760559082, -1.0376081466674805, -4.8292741775512699, -6.667880058288574, -23.82168960571289, 16.032718658447267] }, { "array": [-29.259418487548829, 11.963765144348145, 41.54622268676758, -75.67298889160156, -2.0329294204711916, -20.586360931396486, -21.125045776367189, 0.5504961013793945, -5.419882774353027, -6.314876556396484, -25.0130615234375, 14.262750625610352] }, { "array": [-29.259418487548829, 11.963765144348145, 41.54622268676758, -75.67298889160156, -2.0329294204711916, -20.586360931396486, -21.125045776367189, 0.5504961013793945, -5.419882774353027, -6.314876556396484, -25.0130615234375, 14.262750625610352] }, { "array": [-28.383831024169923, 13.103409767150879, 39.94292449951172, -81.08953857421875, -4.134577751159668, -21.590072631835939, -23.217021942138673, -0.3798789978027344, -5.0724334716796879, -5.94474983215332, -26.63843536376953, 16.777332305908204] }, { "array": [-30.048954010009767, 16.332015991210939, 44.49327850341797, -81.91828155517578, -4.171995162963867, -19.618621826171876, -20.534595489501954, -0.9673957824707031, -3.2188777923583986, -6.572293758392334, -26.59181785583496, 18.48187255859375] }, { "array": [-30.987979888916017, 14.425168991088868, 48.951114654541019, -82.33119201660156, -4.302616119384766, -17.642169952392579, -19.921981811523439, -1.7414522171020508, -1.7319145202636719, -6.870545387268066, -22.85688591003418, 19.475980758666993] }] }, { "name": "U", "mfccCalibrationDataList": [{ "array": [35.53300476074219, -11.371288299560547, 16.699298858642579, -39.32943344116211, 2.8931827545166017, -39.35669708251953, -22.81580924987793, -8.255973815917969, -7.3601884841918949, 3.866161346435547, -36.18340301513672, 0.44779300689697268] }, { "array": [34.92810821533203, -12.66100025177002, 19.551185607910158, -38.589942932128909, 1.5609407424926758, -42.781375885009769, -22.405025482177736, -4.380008220672607, -9.27183723449707, 5.952349662780762, -38.181243896484378, 1.4162702560424805] }, { "array": [34.508079528808597, -10.0599946975708, 18.88445281982422, -34.42189407348633, -0.8881950378417969, -42.52379608154297, -20.37147331237793, -0.6865062713623047, -8.89057731628418, 6.381434440612793, -34.816715240478519, 3.530543327331543] }, { "array": [34.508079528808597, -10.0599946975708, 18.88445281982422, -34.42189407348633, -0.8881950378417969, -42.52379608154297, -20.37147331237793, -0.6865062713623047, -8.89057731628418, 6.381434440612793, -34.816715240478519, 3.530543327331543] }, { "array": [37.634185791015628, -14.223724365234375, 21.262420654296876, -39.50825119018555, 0.31142520904541018, -43.201324462890628, -19.136680603027345, -0.7461652755737305, -8.253379821777344, 9.12716293334961, -33.483253479003909, 9.340311050415039] }, { "array": [38.28554916381836, -15.814170837402344, 19.503570556640626, -40.19694900512695, -0.7106151580810547, -45.373783111572269, -20.631587982177736, -2.2332963943481447, -6.609874725341797, 8.43967342376709, -33.560943603515628, 8.549976348876954] }, { "array": [38.28554916381836, -15.814170837402344, 19.503570556640626, -40.19694900512695, -0.7106151580810547, -45.373783111572269, -20.631587982177736, -2.2332963943481447, -6.609874725341797, 8.43967342376709, -33.560943603515628, 8.549976348876954] }, { "array": [36.15192413330078, -19.43944549560547, 19.22710609436035, -40.229862213134769, -1.7119722366333008, -44.58899688720703, -22.39651870727539, -4.6873579025268559, -5.184035301208496, 5.601207733154297, -35.313777923583987, 6.856324195861816] }, { "array": [33.697601318359378, -19.53946304321289, 18.151920318603517, -38.29360580444336, -1.9536991119384766, -39.185218811035159, -21.181903839111329, -8.092260360717774, -6.441320419311523, 3.8472461700439455, -32.4590950012207, 8.79825210571289] }, { "array": [31.768585205078126, -17.137840270996095, 15.726847648620606, -36.897605895996097, -3.8221397399902345, -35.492652893066409, -19.920631408691408, -10.421395301818848, -7.471479415893555, 1.7216854095458985, -25.726573944091798, 12.090950965881348] }, { "array": [31.768585205078126, -17.137840270996095, 15.726847648620606, -36.897605895996097, -3.8221397399902345, -35.492652893066409, -19.920631408691408, -10.421395301818848, -7.471479415893555, 1.7216854095458985, -25.726573944091798, 12.090950965881348] }, { "array": [32.6759033203125, -15.928326606750489, 16.0853271484375, -34.126686096191409, -6.689325332641602, -33.79350662231445, -19.242847442626954, -10.59890079498291, -10.992877006530762, -2.9957642555236818, -23.329811096191408, 14.231583595275879] }] }, { "name": "U", "mfccCalibrationDataList": [{ "array": [50.45172882080078, -7.723395347595215, 32.294891357421878, -11.293773651123047, -22.777332305908204, -36.17817687988281, -17.044910430908204, -3.7672786712646486, -14.233147621154786, -17.250513076782228, -19.240345001220704, -6.971443176269531] }, { "array": [53.59954071044922, -9.874463081359864, 35.237457275390628, -11.929043769836426, -23.915904998779298, -38.07780838012695, -15.71041202545166, -6.737283706665039, -12.06786060333252, -14.436643600463868, -20.401880264282228, -7.594654560089111] }, { "array": [57.36424255371094, -14.57245922088623, 35.10681915283203, -13.044787406921387, -24.196090698242189, -36.5896110534668, -16.161855697631837, -10.228910446166993, -10.43470287322998, -10.997936248779297, -20.92641830444336, -7.631929397583008] }, { "array": [57.65166473388672, -16.92430305480957, 33.022151947021487, -12.667181968688965, -21.69562530517578, -32.90369415283203, -14.275350570678711, -10.332601547241211, -10.544659614562989, -6.801647186279297, -22.47809410095215, -8.94782543182373] }, { "array": [49.60702133178711, -9.94367504119873, 25.04156494140625, -9.98995304107666, -21.814006805419923, -28.99759292602539, -16.792327880859376, -9.5930814743042, -11.523345947265625, -7.568509578704834, -20.688356399536134, -10.5545015335083] }, { "array": [53.433990478515628, -10.947863578796387, 30.426055908203126, -10.506410598754883, -22.997278213500978, -31.270254135131837, -16.90880584716797, -9.606002807617188, -10.785472869873047, -9.04238510131836, -22.82281494140625, -12.42250919342041] }, { "array": [57.11727523803711, -14.77857780456543, 33.16322326660156, -13.437588691711426, -25.450626373291017, -33.84945297241211, -17.6593017578125, -11.152002334594727, -12.203851699829102, -9.72322940826416, -26.653217315673829, -12.098143577575684] }, { "array": [58.65357208251953, -15.040183067321778, 32.92494583129883, -12.598053932189942, -24.03311538696289, -33.4146728515625, -15.565327644348145, -11.081777572631836, -10.47522258758545, -9.327695846557618, -28.43020248413086, -11.224303245544434] }, { "array": [53.888484954833987, -13.592279434204102, 31.6711368560791, -9.440587043762207, -23.316177368164064, -35.3663444519043, -19.26239776611328, -13.472862243652344, -11.312352180480957, -7.2335309982299809, -25.466888427734376, -8.924440383911133] }, { "array": [54.30769348144531, -13.315286636352539, 26.960655212402345, -11.682543754577637, -23.105655670166017, -32.507049560546878, -17.883333206176759, -10.961587905883789, -11.377249717712403, -7.610130786895752, -24.951541900634767, -9.38066577911377] }, { "array": [57.15829849243164, -13.771659851074219, 31.29568099975586, -9.493865966796875, -23.708837509155275, -33.85169982910156, -19.99985122680664, -12.555244445800782, -14.447962760925293, -7.608822822570801, -23.899951934814454, -6.755941867828369] }, { "array": [47.781307220458987, -14.279045104980469, 23.228328704833986, -14.190330505371094, -23.399112701416017, -34.95072555541992, -21.406070709228517, -10.15461254119873, -14.686234474182129, -9.932022094726563, -22.141719818115236, -8.354757308959961] }] }, { "name": "E", "mfccCalibrationDataList": [{ "array": [7.131360054016113, -36.049346923828128, 45.31159210205078, -45.00178527832031, -15.53347110748291, 1.3071203231811524, 2.57974910736084, -1.8762083053588868, -25.02313995361328, -22.275257110595704, -16.383546829223634, -17.868574142456056] }, { "array": [7.131360054016113, -36.049346923828128, 45.31159210205078, -45.00178527832031, -15.53347110748291, 1.3071203231811524, 2.57974910736084, -1.8762083053588868, -25.02313995361328, -22.275257110595704, -16.383546829223634, -17.868574142456056] }, { "array": [11.067873001098633, -36.16781234741211, 45.893943786621097, -43.60368347167969, -15.017866134643555, -1.2796411514282227, 2.090773582458496, -2.00726318359375, -22.139572143554689, -22.111957550048829, -16.975831985473634, -17.187711715698243] }, { "array": [9.000146865844727, -36.58183288574219, 46.39983367919922, -45.5433349609375, -15.245804786682129, -2.223395347595215, 4.547385215759277, -1.7552833557128907, -24.691539764404298, -22.836109161376954, -15.247169494628907, -15.090204238891602] }, { "array": [6.99749755859375, -37.912235260009769, 46.3832893371582, -47.186683654785159, -13.492773056030274, -2.4662675857543947, 5.109303951263428, -1.4368247985839844, -25.965919494628908, -23.327587127685548, -14.712173461914063, -15.369802474975586] }, { "array": [8.53531551361084, -38.04619598388672, 47.71860885620117, -47.188323974609378, -12.696022987365723, 0.9119815826416016, 5.812184810638428, 1.9683570861816407, -28.441104888916017, -21.203857421875, -16.284610748291017, -15.28965950012207] }, { "array": [8.53531551361084, -38.04619598388672, 47.71860885620117, -47.188323974609378, -12.696022987365723, 0.9119815826416016, 5.812184810638428, 1.9683570861816407, -28.441104888916017, -21.203857421875, -16.284610748291017, -15.28965950012207] }, { "array": [10.728363037109375, -37.86867141723633, 48.31739044189453, -47.47754669189453, -12.409613609313965, -0.4410533905029297, 5.663397789001465, 3.449146270751953, -27.80557632446289, -20.967662811279298, -15.19067096710205, -14.877350807189942] }, { "array": [10.065123558044434, -36.660072326660159, 44.76460266113281, -47.56492233276367, -12.322249412536621, -2.3014774322509767, 3.445328712463379, 3.1899805068969728, -25.734312057495118, -21.715232849121095, -13.482653617858887, -17.041053771972658] }, { "array": [8.782588958740235, -36.92534255981445, 44.4034309387207, -46.98899841308594, -8.84085464477539, -3.2711610794067385, 2.8354501724243166, 3.1592531204223635, -24.939672470092775, -21.54747772216797, -15.242938041687012, -17.53165054321289] }, { "array": [6.508709907531738, -36.80110168457031, 42.792266845703128, -46.29795837402344, -6.919498443603516, -3.788252830505371, 2.5329980850219728, 4.23353385925293, -24.539836883544923, -21.791210174560548, -16.75529670715332, -18.159324645996095] }, { "array": [6.508709907531738, -36.80110168457031, 42.792266845703128, -46.29795837402344, -6.919498443603516, -3.788252830505371, 2.5329980850219728, 4.23353385925293, -24.539836883544923, -21.791210174560548, -16.75529670715332, -18.159324645996095] }] }, { "name": "O", "mfccCalibrationDataList": [{ "array": [53.443660736083987, -12.53360652923584, -26.871780395507814, -71.98885345458985, -5.861575126647949, -1.7120800018310547, -32.90825653076172, 23.514209747314454, -10.125606536865235, -11.977684020996094, 9.883563041687012, -5.261895179748535] }, { "array": [55.69048309326172, -10.316678047180176, -26.289718627929689, -74.05561828613281, -4.348355293273926, -1.1102313995361329, -32.498268127441409, 22.016807556152345, -11.227481842041016, -11.753466606140137, 10.583023071289063, -3.583785057067871] }, { "array": [55.69048309326172, -10.316678047180176, -26.289718627929689, -74.05561828613281, -4.348355293273926, -1.1102313995361329, -32.498268127441409, 22.016807556152345, -11.227481842041016, -11.753466606140137, 10.583023071289063, -3.583785057067871] }, { "array": [56.333030700683597, -9.761429786682129, -24.525028228759767, -75.67504119873047, -2.9411144256591799, -1.0509262084960938, -32.58983612060547, 20.397789001464845, -13.52730941772461, -10.221673965454102, 8.806441307067871, -6.1672868728637699] }, { "array": [58.425559997558597, -13.048576354980469, -26.326568603515626, -77.33265686035156, 0.8971290588378906, -0.13758087158203126, -34.79779052734375, 22.524978637695314, -15.138383865356446, -9.027335166931153, 10.94324779510498, -6.867808818817139] }, { "array": [57.16611099243164, -17.380069732666017, -26.70465087890625, -76.53448486328125, 3.203751564025879, 1.6217775344848633, -36.67759323120117, 24.14405059814453, -12.522785186767579, -8.60572338104248, 13.969680786132813, -4.909186840057373] }, { "array": [55.17543029785156, -22.491680145263673, -25.568838119506837, -74.10460662841797, 2.1666202545166017, 4.1397705078125, -37.21670913696289, 21.997055053710939, -11.680967330932618, -8.396781921386719, 13.454421997070313, -3.452665328979492] }, { "array": [55.18680953979492, -23.176528930664064, -23.64011573791504, -69.64533233642578, 1.9893865585327149, 7.129931449890137, -35.871803283691409, 19.410160064697267, -13.193324089050293, -8.643393516540528, 10.173726081848145, -3.2846717834472658] }, { "array": [55.18680953979492, -23.176528930664064, -23.64011573791504, -69.64533233642578, 1.9893865585327149, 7.129931449890137, -35.871803283691409, 19.410160064697267, -13.193324089050293, -8.643393516540528, 10.173726081848145, -3.2846717834472658] }, { "array": [52.92599105834961, -22.969005584716798, -25.62136459350586, -65.79485321044922, 0.01132965087890625, 5.457650184631348, -33.96955871582031, 16.389381408691408, -14.026248931884766, -7.287093162536621, 11.159339904785157, -3.7541093826293947] }, { "array": [51.816688537597659, -22.891870498657228, -29.805912017822267, -68.66901397705078, -1.9484624862670899, 1.6389532089233399, -33.593971252441409, 17.095460891723634, -13.046170234680176, -8.923750877380371, 13.052698135375977, -5.068996429443359] }, { "array": [54.392459869384769, -23.71658706665039, -31.424976348876954, -75.520263671875, -4.738470077514648, -0.12287521362304688, -36.41456604003906, 18.125713348388673, -14.02833366394043, -11.63118839263916, 12.234237670898438, -4.546117782592773] }] }] } diff --git a/packages/stage-ui-three/src/components/Model/VRMModel.vue b/packages/stage-ui-three/src/components/Model/VRMModel.vue index 6e9890aaf..458d0c98a 100644 --- a/packages/stage-ui-three/src/components/Model/VRMModel.vue +++ b/packages/stage-ui-three/src/components/Model/VRMModel.vue @@ -108,6 +108,8 @@ import { * - modelRotationY: The rotation of the model (y-axis) */ const props = withDefaults(defineProps<{ + /** The context that owns `currentAudioSource`. */ + audioContext?: AudioContext currentAudioSource?: AudioBufferSourceNode cursorPosition?: { x: number, y: number } lastCommittedModelSrc?: string @@ -148,6 +150,7 @@ const emit = defineEmits<{ }>() const { + audioContext, currentAudioSource, lastCommittedModelSrc, modelSrc, @@ -195,7 +198,7 @@ type UpdatableMaterial = Material & { const blink = useBlink() const idleEyeSaccades = useIdleEyeSaccades() const vrmEmote = ref>() -const vrmLipSync = useVRMLipSync(currentAudioSource) +const vrmLipSync = useVRMLipSync(audioContext, currentAudioSource) // For sky box update const nprProgramVersion = ref(0) diff --git a/packages/stage-ui-three/src/components/ThreeScene.vue b/packages/stage-ui-three/src/components/ThreeScene.vue index e1585552d..abda4e113 100644 --- a/packages/stage-ui-three/src/components/ThreeScene.vue +++ b/packages/stage-ui-three/src/components/ThreeScene.vue @@ -52,6 +52,8 @@ import { SkyBox } from './Environment' import { VRMModel } from './Model' const props = withDefaults(defineProps<{ + /** The context that owns `currentAudioSource`. */ + audioContext?: AudioContext currentAudioSource?: AudioBufferSourceNode cursorPosition?: { x: number, y: number } modelSrc?: string @@ -843,6 +845,7 @@ defineExpose({ = { + A: 'aa', + E: 'ee', + I: 'ih', + O: 'oh', + U: 'ou', +} -import { useAudioContext } from '../../../../stage-ui/src/stores/audio' +/** + * Applies shared wLipSync vowel weights to a VRM expression manager. + * + * The caller owns the AudioContext and audio source lifecycle. This composable + * owns only the connection between that source and its wLipSync node. + */ +export function useVRMLipSync( + audioContext: Readonly>, + audioSource: Readonly>, +) { + const lipSyncNode = shallowRef() + const vowelDriver = createWLipSyncVowelDriver() -export function useVRMLipSync(audioNode: Ref) { - const { audioContext } = useAudioContext() - const { state: lipSyncNode, isReady } = useAsyncState(createWLipSyncNode(audioContext, profile as Profile), undefined) - - // https://github.com/mrxz/wLipSync/blob/c3bc4b321dc7e1ca333d75f7aa1e9e746cbbb23a/example/index.js#L50-L66 - const RAW_KEYS = ['A', 'E', 'I', 'O', 'U', 'S'] as const - type LipKey = 'A' | 'E' | 'I' | 'O' | 'U' - const LIP_KEYS: LipKey[] = ['A', 'E', 'I', 'O', 'U'] - const BLENDSHAPE_MAP: Record = { - A: 'aa', - E: 'ee', - I: 'ih', - O: 'oh', - U: 'ou', - } - const RAW_TO_LIP: Record = { - A: 'A', - E: 'E', - I: 'I', - O: 'O', - U: 'U', - S: 'I', - } - - const smoothState: Record = { A: 0, E: 0, I: 0, O: 0, U: 0 } - const ATTACK = 50 // the speed moving to the next mouth shape animation - const RELEASE = 30 // the speed ending the current mouth shape animation - const CAP = 0.7 - const SILENCE_VOL = 0.04 - const SILENCE_GAIN = 0.05 - const IDLE_MS = 160 - let lastActiveAt = 0 - - watch([isReady, audioNode], ([ready, newAudioNode], [, oldAudioNode]) => { - if (oldAudioNode && oldAudioNode !== newAudioNode) { - try { - oldAudioNode.disconnect() - } - catch {} - } - if (!ready || !newAudioNode || !lipSyncNode.value) + watch(audioContext, (context, _, onCleanup) => { + lipSyncNode.value = undefined + vowelDriver.reset() + if (!context) return + + let active = true + let createdNode: undefined | WLipSyncAudioNode + onCleanup(() => { + active = false + createdNode?.disconnect() + }) + + void createWLipSyncNode(context, wlipsyncProfile as Profile) + .then((node) => { + createdNode = node + if (!active) { + node.disconnect() + return + } + lipSyncNode.value = node + }) + .catch((error) => { + if (active) + console.error('[stage-ui-three] Failed to create the VRM lip-sync node.', error) + }) + }, { immediate: true }) + + watch([lipSyncNode, audioSource], ([node, source], _, onCleanup) => { + if (!node || !source) + return + try { - newAudioNode.connect(lipSyncNode.value) + source.connect(node) } - catch {} + catch (error) { + console.error('[stage-ui-three] Failed to connect the VRM lip-sync node.', error) + return + } + + onCleanup(() => { + try { + source.disconnect(node) + } + catch { + // The source can end before Vue runs this watcher cleanup. + } + }) }, { immediate: true }) - onUnmounted(() => audioNode.value?.disconnect()) function update(vrm?: VRMCore, delta = 0.016) { const node = lipSyncNode.value if (!vrm?.expressionManager || !node) return - const vol = node.volume ?? 0 - const amp = Math.min(vol * 0.9, 1) ** 0.7 - - // Remapping wLipSync output AEIOUS to AEIOU - const projected: Record = { A: 0, E: 0, I: 0, O: 0, U: 0 } - for (const raw of RAW_KEYS) { - const lip = RAW_TO_LIP[raw] - const rawVal = node.weights[raw] ?? 0 - projected[lip] = Math.max(projected[lip], rawVal * amp) - } - - // winner + runner - // Original code: all AEIOU mouth shape blended together. Because the A mouth shape has the largest deformation, mixing A-E-I-O-U based on their raw weights causes the combined result to be biased heavily toward A in most cases. - // Improved code: Only the 2 mouth shapes with the largest weights will be blended. - let winner: LipKey = 'I' - let runner: LipKey = 'E' - let winnerVal = -Infinity - let runnerVal = -Infinity - for (const key of LIP_KEYS) { - const val = projected[key] - if (val > winnerVal) { - runnerVal = winnerVal - runner = winner - winnerVal = val - winner = key - } - else if (val > runnerVal) { - runnerVal = val - runner = key - } - } - - // Detect pause or keep silence - const now = performance.now() - let silent = amp < SILENCE_VOL || winnerVal < SILENCE_GAIN - if (!silent) - lastActiveAt = now - if (now - lastActiveAt > IDLE_MS) - silent = true - - // winner + runner weights - const target: Record = { A: 0, E: 0, I: 0, O: 0, U: 0 } - if (!silent) { - target[winner] = Math.min(CAP, winnerVal) - target[runner] = Math.min(CAP * 0.5, runnerVal * 0.6) - } - - // smoothness and expression generation - for (const key of LIP_KEYS) { - const from = smoothState[key] - const to = target[key] - // lerp - const rate = 1 - Math.exp(-(to > from ? ATTACK : RELEASE) * delta) - smoothState[key] = from + (to - from) * rate - const weight = (smoothState[key] <= 0.01 ? 0 : smoothState[key]) * 0.7 - vrm.expressionManager.setValue(BLENDSHAPE_MAP[key], weight) - } + const weights = vowelDriver.update(node, delta) + for (const vowel of WLIP_SYNC_VOWELS) + vrm.expressionManager.setValue(VRM_EXPRESSION_BY_VOWEL[vowel], weights[vowel]) } return { update } diff --git a/packages/stage-ui/src/components/scenes/Stage.vue b/packages/stage-ui/src/components/scenes/Stage.vue index 8a5715d4c..f4e79be68 100644 --- a/packages/stage-ui/src/components/scenes/Stage.vue +++ b/packages/stage-ui/src/components/scenes/Stage.vue @@ -1114,6 +1114,7 @@ defineExpose({ :paused="paused" :show-axes="stageViewControlsEnabled" :enable-orbit-controls="props.enableOrbitControls" + :audio-context="audioContext" :current-audio-source="currentAudioSource" @error="console.error" @vrm-interact="onVRMInteract" @@ -1157,6 +1158,7 @@ defineExpose({ :paused="paused" :cursor-position="cursorPosition" :enable-orbit-controls="props.enableOrbitControls" + :audio-context="audioContext" :current-audio-source="currentAudioSource" @error="console.error" /> diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 8c0f9c8f5..b1be3ac1c 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -3784,6 +3784,10 @@ importers: wlipsync: specifier: 'catalog:' version: 1.3.1 + devDependencies: + vitest: + specifier: catalog:vitest + version: 4.1.11(@opentelemetry/api@1.9.1)(@types/node@26.2.0)(@vitest/browser-playwright@4.1.11)(@vitest/coverage-v8@4.1.11)(jsdom@30.0.1(@noble/hashes@2.3.0)(canvas@3.2.3))(vite@8.2.2(@types/node@26.2.0)(esbuild@0.28.2)(jiti@2.7.0)(less@4.9.0(supports-color@10.2.2))(terser@5.50.0)(tsx@4.23.12)(yaml@2.9.0)) packages/model-driver-magic-live2d: dependencies: @@ -4884,6 +4888,9 @@ importers: '@moeru/three-mmd-physics-ammo': specifier: 'catalog:' version: 0.1.0-beta.7(@moeru/three-mmd@0.1.0-beta.7(@types/three@0.185.4)(three@0.185.1))(@types/three@0.185.4)(three@0.185.1) + '@proj-airi/model-driver-lipsync': + specifier: workspace:^ + version: link:../model-driver-lipsync '@proj-airi/stage-shared': specifier: workspace:^ version: link:../stage-shared @@ -4920,9 +4927,6 @@ importers: vue: specifier: 'catalog:' version: 3.5.41(typescript@6.0.3) - wlipsync: - specifier: 'catalog:' - version: 1.3.1 devDependencies: '@types/culori': specifier: 'catalog:' @@ -5049,6 +5053,9 @@ importers: '@pixiv/three-vrm-core': specifier: 'catalog:' version: 3.5.5(@types/three@0.185.4)(three@0.185.1) + '@proj-airi/model-driver-lipsync': + specifier: workspace:* + version: link:../model-driver-lipsync '@proj-airi/stage-shared': specifier: workspace:* version: link:../stage-shared @@ -5085,9 +5092,6 @@ importers: vue: specifier: 'catalog:' version: 3.5.41(typescript@6.0.3) - wlipsync: - specifier: 'catalog:' - version: 1.3.1 devDependencies: '@types/culori': specifier: 'catalog:' diff --git a/vitest.config.ts b/vitest.config.ts index 313f9ace8..7f6c8c43d 100644 --- a/vitest.config.ts +++ b/vitest.config.ts @@ -14,6 +14,7 @@ export default defineConfig({ 'packages/input-gamepad', 'packages/input-gamepad-vueuse', 'packages/input-playstation-dualsense-5', + 'packages/model-driver-lipsync', 'packages/better-ws', 'packages/plugin-sdk', 'packages/plugin-sdk-tamagotchi',