Desktop Lane Status

Updated: 2026-04-14

This note is a factual status memo for the current desktop lane work around PR #1649. It is intentionally narrow: only current state, actual blockers, and what should happen now vs later.

What is already true

The desktop lane direction is stable:
- macOS only
- Chrome-first
- visual + semantic tree + OS input
- overlay is a visualization layer, not a second system cursor
The following baselines already exist in code:
- /Users/liuziheng/airi/services/computer-use-mcp/src/executors/macos-local.ts
  - saves the real cursor position and restores it with CGWarpMouseCursorPosition(...)
- /Users/liuziheng/airi/apps/stage-tamagotchi/src/main/windows/shared/window.ts
  - makeWindowPassThrough() uses ignore-mouse-events + non-focusable overlay behavior
- /Users/liuziheng/airi/services/computer-use-mcp/src/browser-dom/cdp-bridge.ts
  - 5-second heartbeat with teardown after 3 consecutive failures
The Chrome extension bridge and iframe offset work are no longer hypothetical:
- PR #1649 already contains a real extension-side WebSocket client bridge
- PR #1649 already contains frame offset propagation for iframe DOM candidates

What is actually still blocking

These are the remaining real issues, ordered by severity.

1. Extension unknown actions still return `ok: true`

File:
- /Users/liuziheng/airi-pr1649/services/computer-use-mcp/chrome-extension/background.js
Current behavior:
- unsupported actions fall into result = { error: ... }
- but the response still returns { ok: true, result }
Why this matters:
- upper layers can interpret unsupported DOM actions as successful bridge execution
- that can suppress OS-input fallback even though nothing actually happened
This is still a real unresolved review blocker.

2. Browser-dom click routing still ignores non-default click semantics

File:
- /Users/liuziheng/airi-pr1649/services/computer-use-mcp/src/browser-action-router.ts
- called from /Users/liuziheng/airi-pr1649/services/computer-use-mcp/src/server/register-desktop-grounding.ts
Current behavior:
- chrome_dom candidates route to browser-dom if selector + bridge are available
- routing does not currently incorporate button / clickCount
Why this matters:
- right-click or double-click can still be routed to a DOM path that only performs a standard primary click
This is not as severe as the first issue, but it is still a real correctness gap.

3. Overlay lifecycle / RPC readiness is not fully closed yet

Files currently being worked on:
- /Users/liuziheng/airi-pr1649/apps/stage-tamagotchi/src/main/windows/desktop-overlay/rpc/contracts.ts
- /Users/liuziheng/airi-pr1649/apps/stage-tamagotchi/src/main/windows/desktop-overlay/rpc/index.electron.ts
- /Users/liuziheng/airi-pr1649/apps/stage-tamagotchi/src/renderer/pages/desktop-overlay-polling.ts
- /Users/liuziheng/airi-pr1649/apps/stage-tamagotchi/src/renderer/pages/desktop-overlay-polling.test.ts
Current state:
- there is already a preload-order mitigation in desktop-overlay/index.ts
- there is already a per-call timeout in desktop-overlay-polling.ts
- there is now work-in-progress code for an explicit readiness contract
Why this is not yet "done":
- the readiness flow is still uncommitted work
- the live window context still needs one narrow verification pass
This is not proven broken today, but it is the most likely remaining runtime risk on the overlay path.

What is not a current blocker

These items are real ideas or cleanup work, but they are not the thing that should block the line right now.

Eager overlay init cleanliness in apps/stage-tamagotchi/src/main/index.ts
Refactoring nested browser-dom routing logic for readability
Turning macos-local.ts into instant-warp-only fallback with zero motion trace
Rewriting overlay visuals, ghost pointer polish, or extra renderer debug UI

How to interpret m13v's comments

m13v's comments were useful because they matched the real platform constraints, but they should be split correctly:

Already aligned with current code:
- save → act → restore cursor pattern
- overlay should not intercept user input
- heartbeat teardown for crashed CDP sessions
Still useful as future refinement:
- reducing native motion trace so UI owns more of the visible pointer animation
- deeper runtime discipline around session lifecycle

In short: m13v gave good runtime advice. That does not mean every suggestion is a current blocker.

What should happen now

Fix the extension unknown-action response contract so unsupported actions return ok: false.
Restrict browser-dom click routing to left single-click only; force OS-input for right-click or multi-click.
Finish or explicitly shelve the overlay readiness contract work:
- if kept, validate it in a live overlay window context before merging
- if not finished now, do not half-merge it

What should happen later

Only after the above is clean:

Optional follow-up:
- fix(stage-tamagotchi): validate desktop overlay lifecycle and RPC readiness in live window context
Optional follow-up:
- refactor(computer-use-mcp): evaluate instant-warp-only macOS fallback against ghost-pointer UX
Optional follow-up:
- strengthen iframe anchor matching when sibling iframes are highly similar

Bottom line

The desktop lane is not blocked by direction. It is blocked by a small number of correctness issues and one still-open overlay lifecycle validation step.

Do not reopen architecture. Do not mix in polish. Do not keep piling unrelated changes onto the same PR.

5.6 KiB Raw Blame History