feat(mcp-computer-use): lay down the foundation of computer use (#1380)

2026-04-11 00:54:01 +08:00
parent 933ef01b10
commit aa7428ace2
137 changed files with 31272 additions and 1 deletions
@@ -0,0 +1,108 @@
+# Feasibility Summary
+
+This document records the validated state of the AIRI-specific macOS desktop orchestration v1 in `services/computer-use-mcp`.
+
+## Bottom Line
+
+The current direction is feasible and materially stronger than the earlier pure-vision-only path.
+The validated architecture is now:
+
+- AIRI keeps the control plane
+- `computer-use-mcp` keeps trace, audit, screenshot persistence, policy, and the MCP surface
+- the primary execution backend is local macOS window automation
+- AIRI desktop handles human approval through native dialogs
+- terminal commands run in a controlled background shell runner instead of a Terminal tab script
+
+That makes the feature an orchestration layer, not just a mouse-clicking demo.
+
+## What Was Verified
+
+### 1. The service still fits AIRI's MCP attachment model
+
+AIRI continues to use the existing stdio MCP bridge through `mcp.json`.
+No transport rewrite was required.
+
+### 2. The service now exposes a tool-first desktop orchestration surface
+
+Validated surface in this checkout:
+
+- desktop observation tools
+- deterministic app open/focus tools
+- background terminal execution tools
+- primitive UI interaction tools
+- approval / trace / audit helpers
+
+This is a better fit for AIRI than leading with pure screenshot-driven action selection.
+
+### 3. Terminal execution is now first-class and auditable
+
+Validated by tests:
+
+- commands run in a local background shell process
+- non-zero exit codes are returned without throwing away stderr/stdout
+- cwd is sticky across calls unless explicitly overridden
+- reset clears terminal state
+
+This gives AIRI a deterministic execution path for many developer workflows without relying on Terminal.app scripting.
+
+### 4. Native approval now sits in the AIRI desktop layer
+
+Validated by implementation shape:
+
+- MCP still returns `approval_required`
+- AIRI renderer intercepts `computer_use` pending actions
+- Electron main shows a native approval dialog
+- AIRI automatically follows up with approve/reject tool calls
+- session-scoped approval reuse is limited to terminal and app open/focus actions
+
+That keeps approval as a user action, not a model action.
+
+### 5. The old remote Linux path still compiles and tests
+
+The previous `linux-x11` backend remains available as a legacy experimental path.
+It is not the primary v1 story anymore, but it was intentionally kept compiling so existing remote smoke tooling still works.
+
+## Current Boundary
+
+Main v1 story:
+
+- executor: `macos-local`
+- apps explicitly supported for open/focus by default:
+  - `Terminal`
+  - `Cursor`
+  - `Google Chrome`
+- safety boundary:
+  - native approval dialogs
+  - `denyApps`
+  - trace / audit
+  - screenshot persistence
+  - operation budgets
+
+Explicit non-goals of this pass:
+
+- PTY/TUI terminal automation
+- deep accessibility tree grounding
+- strict app-level UI sandboxing
+- remote sandbox hosting for other users
+- Windows / Wayland / multi-monitor support
+
+## Commands Verified In This Checkout
+
+- `pnpm -F @proj-airi/computer-use-mcp typecheck`
+- `pnpm -F @proj-airi/computer-use-mcp test`
+- `pnpm -F @proj-airi/stage-ui typecheck`
+- `pnpm -F @proj-airi/stage-tamagotchi typecheck`
+
+## Practical Interpretation
+
+The feature is now credible as:
+
+- a macOS desktop orchestration layer for AIRI
+- a way to connect chat, MCP, terminal execution, and UI observation into one task flow
+- a safer incremental path than trying to solve generic pure-vision computer use first
+
+It should not be pitched as:
+
+- a general desktop sandbox platform
+- a no-approval autonomous agent
+- a production-grade app-isolated desktop security boundary