feat(server,stage-ui): bidirectional streaming TTS + audio path refactor
Why:
- Add a real bidirectional streaming TTS path: raw LLM tokens are
forwarded to the upstream model (Volcengine v3 via the unspeech ws
bridge) without client-side segmentation, so the model owns sentence
splitting and audio chunks play as they arrive.
- Move audio endpoints out of /api/v1/openai/. `/audio/voices`,
`/audio/models`, `/audio/voices/streaming` are not real OpenAI public
APIs, and the streaming TTS surface has nothing to do with OpenAI —
keeping them under /openai/ mislabelled the contract.
- Introduce `capabilities.speech.transport` on ProviderDefinition so
future streaming providers (ElevenLabs / Cartesia / OpenAI Realtime)
opt in without touching Stage.vue or the session factory.
- Unify Stage.vue's TTS path through a single StageTtsSession so the
chat-orchestrator hooks no longer branch on provider id.
What:
- apps/server: new ws proxy /api/v1/audio/speech/ws bridges client ↔
unspeech with auth, pre-flight flux check, billing from upstream
session.finished.usage, OTel spans.
- apps/server: audio routes moved from /api/v1/openai/audio/* to
/api/v1/audio/* (hard cutover; 404 sentinel tests added).
- apps/server: new /api/v1/audio/voices/streaming proxy reads voices
from unspeech /api/voices?provider=volcengine.
- apps/server: new STREAMING_TTS_UPSTREAM configKV entry +
scripts/seed-streaming-tts.ts.
- stage-ui: new libs/speech/streaming-pipeline.ts opens one ws per LLM
intent (appendText / finish / cancel + onSentence / onError / onDone).
- stage-ui: new libs/speech/tts-session.ts — StageTtsSession interface
with segmenter and streaming adapters; factory dispatches by
capabilities.speech.transport instead of hard-coded provider id.
- stage-ui: providerOfficialSpeechStreaming with capabilities.speech =
{ transport: 'bidirectional-ws' }; settings page with model/voice
picker + ws-based preview.
- stage-ui: Stage.vue chat hooks collapsed to a single currentSession;
hot-swap watcher cancels mid-session on provider/voice/model change;
unmount cancels and drains playback.
Tests:
- 9 streaming-pipeline tests (happy path / buffered / error / cancel /
truncation)
- 11 tts-session tests (factory branch coverage + adapter contracts)
- 4 audio-speech-ws route tests (forwarding / billing / pre-flight /
config-missing)
- 3 legacy-path 404 sentinels in v1 route tests
- Verification doc updated to reflect automated coverage.
This commit is contained in:
@@ -83,19 +83,22 @@ once it lands.
|
||||
|
||||
```bash
|
||||
cd apps/server
|
||||
LLM_ROUTER_MASTER_KEY="$LLM_ROUTER_MASTER_KEY" \
|
||||
STREAMING_TTS_UPSTREAM_URL="ws://airi-unspeech.railway.internal:5933/v1/audio/speech/stream" \
|
||||
VOLCENGINE_TTS_API_KEY="$VOLCENGINE_TTS_API_KEY" \
|
||||
pnpm exec dotenvx run --env-file=.env.local -- \
|
||||
tsx scripts/seed-router-config.ts \
|
||||
--streaming-tts-key "$VOLCENGINE_TTS_API_KEY"
|
||||
tsx scripts/seed-streaming-tts.ts
|
||||
```
|
||||
|
||||
> Note: `seed-router-config.ts` does not yet implement
|
||||
> `--streaming-tts-key`. Until then, write the configKV entry directly
|
||||
> from a one-off node REPL using `configKV.set('STREAMING_TTS_UPSTREAM',
|
||||
> { baseURL: 'ws://localhost:5933/v1/audio/speech/stream', keys: [{ id:
|
||||
> 'volcengine-prod-1', ciphertext: envelopeCrypto.encryptKey(<plaintext>,
|
||||
> { modelName: 'streaming-tts', keyEntryId: 'volcengine-prod-1' }) }] })`.
|
||||
The script reads `LLM_ROUTER_MASTER_KEY` and `REDIS_URL` from
|
||||
`.env.local`, envelope-encrypts the Volcengine key under AAD
|
||||
`{ modelName: 'streaming-tts', keyEntryId: 'volcengine-prod-1' }`, and
|
||||
writes the `STREAMING_TTS_UPSTREAM` configKV entry. Use `--dry-run` to
|
||||
preview the ciphertext length without committing.
|
||||
|
||||
To point at a different unspeech instance later, just re-run the script
|
||||
with a different `STREAMING_TTS_UPSTREAM_URL`. To rotate the upstream
|
||||
key, re-run with `--key-id volcengine-prod-N` (the audio-speech-ws
|
||||
route always reads `keys[0]`, so a write replaces the active key).
|
||||
|
||||
### Scenario L1: streaming session happy path
|
||||
|
||||
|
||||
Reference in New Issue
Block a user