- Remove stage-tamagotchi, stage-pocket, server, and engine workspaces with their workflows, addons, and docs
- Switch the toolchain from pnpm to bun, replace lockfiles with bun.lock
- Rewrite remaining product references to Moeka

2026-09-14 16:40:49 +07:00

3.6 KiB

Raw Permalink Blame History

Feasibility Summary

This document records the validated state of the Moeka-specific macOS desktop orchestration v1 in services/computer-use-mcp.

Bottom Line

The current direction is feasible and materially stronger than the earlier pure-vision-only path. The validated architecture is now:

Moeka keeps the control plane
computer-use-mcp keeps trace, audit, screenshot persistence, policy, and the MCP surface
the primary execution backend is local macOS window automation
Moeka desktop handles human approval through native dialogs
terminal commands run in a controlled background shell runner instead of a Terminal tab script

That makes the feature an orchestration layer, not just a mouse-clicking demo.

What Was Verified

1. The service still fits Moeka's MCP attachment model

Moeka continues to use the existing stdio MCP bridge through mcp.json. No transport rewrite was required.

2. The service now exposes a tool-first desktop orchestration surface

Validated surface in this checkout:

desktop observation tools
deterministic app open/focus tools
background terminal execution tools
primitive UI interaction tools
approval / trace / audit helpers

This is a better fit for Moeka than leading with pure screenshot-driven action selection.

3. Terminal execution is now first-class and auditable

Validated by tests:

commands run in a local background shell process
non-zero exit codes are returned without throwing away stderr/stdout
cwd is sticky across calls unless explicitly overridden
reset clears terminal state

This gives Moeka a deterministic execution path for many developer workflows without relying on Terminal.app scripting.

4. Native approval now sits in the Moeka desktop layer

Validated by implementation shape:

MCP still returns approval_required
Moeka renderer intercepts computer_use pending actions
Electron main shows a native approval dialog
Moeka automatically follows up with approve/reject tool calls
session-scoped approval reuse is limited to terminal and app open/focus actions

That keeps approval as a user action, not a model action.

5. The old remote Linux path still compiles and tests

The previous linux-x11 backend remains available as a legacy experimental path. It is not the primary v1 story anymore, but it was intentionally kept compiling so existing remote smoke tooling still works.

Current Boundary

Main v1 story:

executor: macos-local
apps explicitly supported for open/focus by default:
- Terminal
- Cursor
- Google Chrome
safety boundary:
- native approval dialogs
- denyApps
- trace / audit
- screenshot persistence
- operation budgets

Explicit non-goals of this pass:

PTY/TUI terminal automation
deep accessibility tree grounding
strict app-level UI sandboxing
remote sandbox hosting for other users
Windows / Wayland / multi-monitor support

Commands Verified In This Checkout

bun run --filter @proj-airi/computer-use-mcp typecheck
bun run --filter @proj-airi/computer-use-mcp test
bun run --filter @proj-airi/stage-ui typecheck
bun run --filter @proj-airi/stage-web typecheck

Practical Interpretation

The feature is now credible as:

a macOS desktop orchestration layer for Moeka
a way to connect chat, MCP, terminal execution, and UI observation into one task flow
a safer incremental path than trying to solve generic pure-vision computer use first

It should not be pitched as:

a general desktop sandbox platform
a no-approval autonomous agent
a production-grade app-isolated desktop security boundary

3.6 KiB Raw Permalink Blame History