Commit Graph
596 Commits
Author SHA1 Message Date
Zhou fanqi f6b53a37fa feat(minecraft): accurate perception, autonomic reflexes, and brain reliability (#1915)
## Summary

This PR upgrades the Minecraft bot's cognitive stack so it perceives
danger
accurately, reacts below the LLM brain with fast reflexes, and recovers
from
common brain failure modes. All changes are confined to
`services/minecraft/**`.

The desktop-side integration (spoken in-game chat, the online-gated
relay tool,
master binding in the UI) is submitted separately to keep this PR
focused; the
two are coupled only at runtime via AIRI context lanes, not at compile
time.

## What's included

**Perception — accurate damage attribution**
- `fall-tracker`: classify fall damage from physics-tick vertical
velocity.
- `attacker-tracker`: record the *real* attacker from the `entityHurt`
source,
  fixing "a mob shot me but I chased the player standing next to me".
- `low-health`: emit a danger signal at health ≤ 6 when no ready food is
held.
- `damage-taken`: resolve the source via a lava → drown → fire → fall →
  attacker chain and surface the attacker's name to the brain.

**Reflex — autonomic behaviors (run on the FSM tick, below the brain)**
- `defend`: auto-engage a hostile mob that attacks the bot/master and
hold the
  fight instead of thrashing (suppress damage wakes while attacking).
- `escape-hazard`: climb out of lava, or surface when drowning.
- `auto-eat`: eat ready food at health ≤ 6 (skipped during combat).
- A `reflexEngaged` autonomy flag suppresses auto-follow and auto-eat
while a
  defend/escape reflex owns the body.

**Conscious / skills — master recognition and reliability**
- Bind the "主人" role to the owner's in-game username so the bot
recognizes its
master in-world; enforce master-only command authority, never attack the
master, and react to a light hit with a short protest. Resolve players
by
  username, never the literal "player"/"主人".
- `js-planner`: strip prose-mixed LLM replies that caused `X is not
defined` and
give-up spirals; add a `botCall` sandbox bridge and the `whereName`
query
  chain method.
- Brain-prompt discipline: *saying is not doing*, *`recipePlan` is recon
not
crafting*, *eat only when food < 18* (regen is time-based), *defer equip
until
  a queued craft lands*.
- Skills: collect drops that spawn shortly after a kill (no more 0
loot);
`goToBed` skips occupied beds; detach auto-follow while
mining/collecting so
  digging is not interrupted.
- Platform: load perception rules on Windows (`fileURLToPath`); widen
the AIRI
  client read timeout to stop connection flapping.

The master username is injected via `generateBrainSystemPrompt(..., {
masterUsername })`
from config — it is **not** hardcoded.

## How tested

```bash
pnpm -F @proj-airi/minecraft-bot exec vitest run   # 220 tests; +56 new, all green
pnpm -F @proj-airi/minecraft-bot typecheck
pnpm exec moeru-lint services/minecraft            # 0 problems on changed files
```

- Each feature ships with unit tests (trackers, reflex behaviors,
planner
  salvage, skills, prompt, context service).
- Manual in-game testing: combat vs. pillagers/zombies, lava/water
escape,
low-health eating, crafting, master recognition with multiple players
present.

### Pre-existing baseline (not introduced by this PR)

Measured on `main` (`cb5c0783e`) before any change here:
- `tsc` reports 12 errors, all from a `vec3` 0.1.x/0.2.0
dependency-dedup
mismatch (`angleTo`) and `unknown[]` casts in `map-renderer.ts` /
`mcdata.ts` —
  files this PR does not touch. This PR adds **0** new type errors.
- 7 unit tests already fail on `main` (4 in `map-renderer.test.ts`, 2
timing-sensitive `brain.test.ts`, 1 timing-sensitive `rules.test.ts`).
This
  PR leaves those exactly as-is and does not touch those modules.

## Known limitations / follow-ups

- The `defend` reflex does not yet reliably engage ranged pillagers at
distance;
  the brain's `attack` tool currently covers that case.
- `craftRecipe` does not auto-unpack blocks (e.g. diamond_block →
diamond) or
  auto-gather missing intermediates for complex recipes.
- `inferDamageSource` orders environmental causes before
`recentAttacker`, which
  can mis-attribute an in-water-while-shot hit; low impact, deferred.
2026-06-05 11:51:18 +08:00
RainbowBird 7841df28f8 build(deps): migrate workspace dependencies to catalogs
Move direct external dependency versions into pnpm catalogs and update workspace packages to reference catalog entries.

Signed-off-by: RainbowBird <git@luoling.moe>

Commit-Message-Assisted-by: Codex
2026-06-03 17:12:13 +08:00
刘梓恒 d86c502f04 feat(computer-use-mcp): add browser repair suggestions (#1920)
## Summary

Adds browser DOM repair suggestions inside `services/computer-use-mcp`.

Known browser DOM action failures now get structured, advisory-only
repair suggestions for selector misses, hidden elements, timeouts,
detached frames, and stale elements. The registration path uses the
suggestions for `browser_dom_click` and `browser_dom_wait_for_element`
thrown-error responses.

## Boundary

- Only touches `services/computer-use-mcp`.
- Does not touch app, renderer, overlay, extension assets, desktop v3
files, or root project governance files.
- Does not import preview changes that delete current upstream
Chrome/desktop files.
- Does not include local AI config, screenshots, tokens, account
identifiers, or raw environment dumps.

## Validation

Sanitized evidence is committed in
`services/computer-use-mcp/validation/browser-repair-contract.md`.

Ran:

- `pnpm install --ignore-scripts --frozen-lockfile`
- `pnpm -F @proj-airi/computer-use-mcp exec vitest run
src/browser-dom/browser-repair-contract.test.ts
src/server/register-tools-pty-approval.test.ts --config
./vitest.config.ts`
- `pnpm exec moeru-lint --fix
services/computer-use-mcp/validation/browser-repair-contract.md
services/computer-use-mcp/src/browser-dom/browser-repair-contract.ts
services/computer-use-mcp/src/browser-dom/browser-repair-contract.test.ts
services/computer-use-mcp/src/server/register-tools.ts
services/computer-use-mcp/src/server/register-tools-pty-approval.test.ts`
- `git diff --check`

`pnpm -F @proj-airi/computer-use-mcp typecheck` is currently blocked by
existing baseline errors in `chrome-session-manager` and
`desktop-grounding`, outside this patch. The validation note lists the
error classes without local paths or raw logs.
2026-06-02 10:50:34 +08:00
刘梓恒 7021572cb8 feat(computer-use-mcp): add tool lane hygiene advisory (#1919)
## Summary

Adds advisory-only tool lane hygiene inside `services/computer-use-mcp`.

This tracks the most recent non-exempt tool lane and appends a
non-blocking advisory when a registered tool is called from a different
active lane. The change is intentionally scoped to computer-use MCP
server registration and run state.

## Boundary

- Only touches `services/computer-use-mcp`.
- Does not touch app, renderer, overlay, extension, or root project
governance files.
- Does not include local AI config, local worktree artifacts,
screenshots, tokens, account identifiers, or raw environment dumps.

## Validation

Sanitized evidence is committed in
`services/computer-use-mcp/validation/tool-lane-hygiene.md`.

Ran:

- `pnpm install --ignore-scripts --frozen-lockfile`
- `pnpm -F @proj-airi/computer-use-mcp exec vitest run
src/server/tool-lane-hygiene.test.ts --config ./vitest.config.ts`
- `pnpm -F @proj-airi/computer-use-mcp exec vitest run
src/server/tool-lane-hygiene.test.ts
src/server/register-tools-coordinate-contract.test.ts
src/server/register-tools-pty-approval.test.ts --config
./vitest.config.ts`
- `pnpm exec moeru-lint --fix
services/computer-use-mcp/validation/tool-lane-hygiene.md
services/computer-use-mcp/src/server/tool-lane-hygiene.ts
services/computer-use-mcp/src/server/tool-lane-hygiene.test.ts
services/computer-use-mcp/src/server/register-tools.ts
services/computer-use-mcp/src/state.ts`
- `git diff --check`

`pnpm -F @proj-airi/computer-use-mcp typecheck` is currently blocked by
existing baseline errors in `chrome-session-manager` and
`desktop-grounding`, outside this patch. The validation note lists the
error classes without local paths or raw logs.
2026-06-01 17:15:58 +08:00
0xSelenicDoveandautofix-ci[bot] b84d59438c fix(ci): upgrade playwright to ^1.60.0 to resolve install hang on node 24 (#1896)
### Description
Upgrades Playwright from `^1.59.1` to `^1.60.0` across the monorepo
workspace.

### Why is this needed?
In clean CI environments using Node.js 24 (the current LTS), Playwright
versions older than `1.60.0` hang indefinitely during the zip extraction
phase of `playwright install chromium`.

This is a known compatibility issue with Playwright's older extraction
utility (`yauzl`) running on Node 24+. Playwright resolved this bug in
version `1.60.0`. Upgrading ensures the CI test pipeline runs smoothly
without hanging.

### Proposed Changes
- Upgraded `"playwright"` to `^1.60.0` in:
  - Root `package.json`
  - `packages/vishot-runner-browser/package.json`
  - `packages/vishot-runner-electron/package.json`
  - `services/twitter-services/package.json`
- Regenerated `pnpm-lock.yaml` via `pnpm install`.

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-05-29 19:07:58 +08:00
3828d06322 fix(computer-use-mcp): bound waitForElement frame timeouts (#1856)
## Summary\n- Pass the remaining waitForElement budget into each
frame-level CU_ACTION send so unresponsive frames cannot consume the
fixed 8s sendMessage timeout.\n- Use the remaining deadline for each
poll and stop polling immediately when the budget is exhausted.\n-
Reduce the bridge-side waitForElement grace from the legacy 9.5s buffer
to a small transport grace.\n- Add a regression test covering the
hanging-extension case.\n\n## Validation\n- pnpm -C
services/computer-use-mcp exec vitest run --config ./vitest.config.ts
src/browser-dom/extension-bridge.test.ts

---------

Co-authored-by: Neko <neko@ayaka.moe>
Co-authored-by: 刘梓恒 <160735726+3361559784@users.noreply.github.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-05-22 15:29:52 +08:00
luyua9 3a6f7a7628 fix(telegram-bot): skip empty structured messages (#1857) 2026-05-21 03:06:52 +08:00
duyua9and刘梓恒 5ab8b0e33e fix(computer-use-mcp): bound terminal output capture (#1802)
## Summary
- cap local `terminal_exec` stdout/stderr capture at a fixed per-stream
limit
- report whether each stream was truncated, along with the original
captured length
- add runner coverage for commands that emit large stdout and stderr
payloads

## Why
The local shell runner currently appends stdout/stderr without a
boundary before returning `TerminalCommandResult`. Large command output
can grow the MCP response and stored terminal state far beyond what is
useful for the agent.

This keeps command execution semantics the same while bounding the
returned text and making truncation explicit to callers.

## Tests
- `pnpm -F @proj-airi/computer-use-mcp test --
src/terminal/runner.test.ts`
- `pnpm -F @proj-airi/computer-use-mcp typecheck`
- `git diff --check origin/main...HEAD`

Co-authored-by: 刘梓恒 <160735726+3361559784@users.noreply.github.com>
2026-05-13 12:47:57 +08:00
刘梓恒 746e486628 feat(computer-use-mcp): add background desktop scheduler (#1805) 2026-05-13 12:16:39 +08:00
刘梓恒 aec5486c51 test(computer-use-mcp): add desktop v3 smoke coverage (#1780) 2026-05-13 12:08:38 +08:00
刘梓恒 fdb8309559 chore(computer-use-mcp): added docs defining plast-mem plugin (#1779) 2026-05-13 11:59:06 +08:00
刘梓恒 5182a4a492 test(computer-use-mcp): define planning orchestration contract (#1778) 2026-05-13 11:58:35 +08:00
刘梓恒 4fcf494822 chore(computer-use-mcp): added docs defining mimic baseline training boundary (#1777) 2026-05-13 11:57:18 +08:00
刘梓恒 f81e116022 feat(computer-use-mcp): add terminal screen heuristics (#1738) 2026-05-13 11:56:09 +08:00
刘梓恒 cad10ca090 chore(computer-use-mcp): added architecture docs outlining airi cli chafa (#1735) 2026-05-13 11:55:33 +08:00
Neko Ayaka 6d61699d10 release: v0.10.2 2026-05-07 19:36:33 +08:00
Neko Ayaka 6dd370433a release: v0.10.1 2026-05-05 14:10:55 +08:00
刘梓恒 8239e290bf fix(stage-tamagotchi): add macOS cursor restore discipline (#1754) 2026-05-04 06:38:05 +08:00
Neko Ayaka ca722f69b7 release: v0.10.0 2026-05-03 23:17:03 +08:00
刘梓恒 5bd0b19e84 fix(stage-tamagotchi): harden overlay isolation and iframe coordinates (#1751)
---------

Co-authored-by-agent: Antigravity <antigravity@gemini.com>
2026-04-29 22:52:00 +08:00
刘梓恒 9d0719e6b5 fix(computer-use-mcp): add macOS display geometry contract (#1750)
---------

Co-authored-by-agent: Antigravity <antigravity@gemini.com>
2026-04-29 22:51:19 +08:00
刘梓恒 40531b0022 fix(computer-use-mcp): harden desktop runtime cleanup (#1746)
---------

Co-authored-by-agent: Antigravity <antigravity@gemini.com>
2026-04-29 22:48:38 +08:00
刘梓恒 b30ab8e07e test(computer-use-mcp): align transcript projection baseline (#1742)
Co-authored-by-agent: Antigravity <antigravity@gemini.com>
2026-04-29 22:48:08 +08:00
刘梓恒 51bff3ed73 chore(agent): add computer-use-mcp agent governance (#1737)
Co-authored-by-agent: Antigravity <antigravity@gemini.com>
2026-04-27 15:31:02 +08:00
刘梓恒 f92e670dd9 feat(computer-use-mcp): add transcript truth source and safe projection (#1734)
Co-authored-by-agent: Antigravity <antigravity@gemini.com>
2026-04-27 15:25:26 +08:00
刘梓恒 e145ce81c1 feat(computer-use-mcp): add read-only DOM tools parity to extension bridge (#1733)
Co-authored-by-agent: Antigravity <antigravity@gemini.com>
2026-04-27 15:24:27 +08:00
刘梓恒 2e9020a06f fix(computer-use-mcp): route target clicks through action executor (#1727)
---------

Co-authored-by-agent: Antigravity <antigravity@gemini.com>
2026-04-26 05:53:15 +08:00
刘梓恒 d7402ade9f feat(stage-tamagotchi,stage-ui,computer-use-mcp): intro agent-owned session and ghost pointer phases (#1649) 2026-04-25 04:14:33 +08:00
刘梓恒 255ce71cba feat(stage-tamagotchi,computer-use-mcp): implement browser-native DOM action routing (#1648)
---------

Co-authored-by-agent: Antigravity <antigravity@gemini.com>
Co-authored-by-agent: Codex <267193182+codex@users.noreply.github.com>
2026-04-25 00:08:22 +08:00
刘梓恒 632aa1ad99 [1/3] feat(desktop): add desktop observation and overlay baseline (#1647)
---------

Co-authored-by-agent: Antigravity <antigravity@gemini.com>
Co-authored-by-agent: Codex <267193182+codex@users.noreply.github.com>
2026-04-24 23:43:40 +08:00
Neko Ayaka a406ffb67c feat(stage-tamagotchi): widget now supports customizable ui from plugin/extension 2026-04-21 02:23:34 +08:00
Neko Ayaka 2ab21879dd chore(deps): bump dependencies 2026-04-18 19:18:17 +08:00
Neko Ayaka b9f88ef84f style: lint 2026-04-15 14:44:59 +08:00
Neko Ayaka f286464e03 chore(deps): bump dependencies 2026-04-15 14:44:59 +08:00
d86314f5a3 fix(minecraft): multiple core crashes and logic bugs in Minecraft bots (#1375)
## Description

本次修复解决了 Minecraft 机器人运行中的多个核心崩溃与逻辑问题,让 AI 能稳定执行挖掘、跟随、资源收集等任务:

1.  **修复 `expectMoved()` 验证逻辑**:
    - 当动作结果中不存在 `movedDistance` 字段时,默认使用 0 而非抛出错误
    - 新增非移动动作白名单(挖掘、放置、交互等),对这类动作直接返回成功,不再强制要求移动距离
- 解决了「挖掘动作被误判失败」的核心问题(报错:`Expectation failed: expectMoved() requires
last action result with movedDistance telemetry`)

2.  **优化 `breakBlockAt()` 挖掘逻辑**:
    - 修正脚下方块位置计算,提升挖掘精准度
    - 补充详细调试日志,便于排查挖掘相关问题

3.  **修复 `InventoryQueryChain.count()` 空值问题**:
    - 增加对 `name` 参数的空值检查,避免 `undefined` 调用 `toLowerCase()` 导致崩溃
- 解决了 `Cannot read properties of undefined (reading 'toLowerCase')` 报错

4.  **补全查询链缺失方法**:
    - 为 `EntityQueryChain` 新增 `whereName()` 方法,支持按实体名称过滤
    - 为 `BlockQueryChain` 新增 `where()` 方法,支持自定义谓词过滤
- 解决了 `whereName is not a function` / `where is not a function` 等类型错误

5.  **修复 `JavaScriptPlanner.runAction()` 异步竞态问题**:
- 在异步操作后增加 `this.activeRun` 空值检查,避免 `activeRun` 为 `null` 时访问 `executed`
导致崩溃
    - 解决了 `Cannot read properties of null (reading 'executed')` 报错

## Linked Issues

Fixes #1352

## Additional Context
- 所有修改均在 Minecraft 服务目录下,不影响其他模块
- 测试验证:机器人可正常执行挖掘、跟随、资源收集等任务,不再出现上述崩溃
- 原项目 Mineflayer 版本即将弃用,但本修复可让当前实验版机器人稳定可用,也为后续原生 Mod 版本提供参考

---------

Co-authored-by: Rin <shinohara-rin@users.noreply.github.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-04-13 18:21:38 +08:00
Neko Ayaka b18158bce2 chore(computer-use-mcp): remove all hardcoded file path 2026-04-11 05:13:55 +08:00
Neko Ayaka feb2055ed2 style(computer-use-mcp): house keeping, lint fixes, type missing, and many more 2026-04-11 04:59:13 +08:00
Neko Ayaka 95eac8e41c chore(deps): bump dependencies 2026-04-11 04:00:49 +08:00
刘梓恒 aa7428ace2 feat(mcp-computer-use): lay down the foundation of computer use (#1380) 2026-04-11 00:54:01 +08:00
Neko Ayaka dd284f0f27 release: v0.9.0 2026-04-10 14:48:30 +08:00
Neko Ayaka a0caaf3214 release: v0.9.0-rc.1 2026-04-10 03:40:59 +08:00
Neko Ayaka dc7132bb91 release: v0.9.0-beta.7 2026-04-08 04:17:03 +08:00
Neko Ayaka 7f7f50b462 release: v0.9.0-beta.6 2026-04-07 04:58:46 +08:00
Richard Pinedo 99be852fe4 feat(stage-tamagotchi,stage-ui,server-sdk,server-runtime): zero-trust websocket authentication (#1582) 2026-04-07 04:05:35 +08:00
Neko Ayaka 389388820f release: v0.9.0-beta.5 2026-04-06 02:10:21 +08:00
Neko Ayaka 476ef2018f style: many typescript v6 pending errors 2026-04-05 02:07:09 +08:00
fdeb9561bc feat(minecraft,stage-*): airi integration, isolated-vm plus misc updates (#1371)
* feat(minecraft): contextflow integration

Add AiriBridge to cognitive engine to enable communication with Airi system. Expose `notifyAiri` and `updateAiriContext` functions in JS planner runtime globals for sending notifications and context updates from agent code.

* refactor(minecraft): remove redundant destinations field from airi-bridge

Remove hardcoded `destinations: ['minecraft']` from spark:command, spark:notify, and context:update events as the destination is already implied by the bridge context.

* feat(minecraft): complete AIRI context flow integration

Wire up bidirectional AIRI ↔ Minecraft communication via dedicated
signal channels instead of reusing signal:chat_message.

- Add airi_command and airi_context to PerceptionSignalType
- Route spark:command intents: action/plan/proposal/reroute emit
  signal:airi_command; context intent emits signal:airi_context
- Add inbound context:update listener so AIRI can push context to
  the Minecraft agent
- airi_context signals are injected into conversation history directly
  without triggering a full cognitive cycle
- airi_command events reset the no-action budget (like player chat)
- enterContext/exitContext auto-emit context:update back to AIRI for
  task lifecycle visibility
- Add sendEmit() helper for queued/working/done progress reporting

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(minecraft): report bot startup state to AIRI

Send initial context update to AIRI when brain comes online, including
bot username, server address, position, health, game mode, and list of
other online players. Tagged with 'startup' and bot username for
filtering.

* feat(minecraft): manage bot config through airi

* feat(stage-ui): add minecraft service shell

* fix(stage-ui): use applied minecraft config for chat context

* fix(minecraft): fall back on invalid env port

* fix(minecraft): support quoted env values

* fix(minecraft): ignore empty env overrides

* fix(stage-ui): keep minecraft draft during apply

* fix(stage-ui): simplify minecraft heartbeat label

* feat(stage-ui): simplify minecraft settings shell

* fix(minecraft): use registry heartbeat for stage liveness

* chore(minecraft): remove annoying detector decision logging

* fix(minecraft): limit context flow to bot pushes

* fix(minecraft): use workspace server sdk

* fix(stage-ui): replay registry snapshots for late listeners

* chore(minecraft): add todo for explicit stage deregistration

* feat: improve context flow observability

* fix(stage-ui): remove redundant header and callout from settings and context flow pages

* feat(stage-ui): enhance context flow prompt projection UI with readiness indicator

* docs(stage-ui): clarify context flow prompt projection terminology

* fix(stage-ui): add test for clone-safe context snapshots with nested fields

* todo

* fix(minecraft): avoid blocking startup on airi connection

* Remove LLM timeouts from the Minecraft brain

* Remove context boundary system from Minecraft brain

* feat(minecraft): use isolated-vm for JS planner sandbox

* refactor(minecraft): harden planner sandbox defaults

* Remove LLM attempt timeout guard

* [autofix.ci] apply automated fixes

* Fix server runtime shutdown signal handling

* Remove misleading Minecraft integration toggle

* contextflow destination to stage-*

* chore(deps): updated

* Add settings layout route for web devtools

* chore: default to expand context panels

* fix(server-*): should export more types, and for module:announced should ignore self

* refactor(minecraft): better context orchestration, handle module announce, context sync, and more

* feat(stage-ui): not spark command ready

* [autofix.ci] apply automated fixes

* clean up

* refactor: use :class arrays per AGENTS.md styling guidelines

* cleanup bloated tests

* rm more tests

* use errorMessageFrom

* hardcode to reduce bloat

* update setup instructions

* timeout for brain

* remove more bloat

* fix

---------

Co-authored-by-agent: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by-agent: Codex
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Neko Ayaka <neko@ayaka.moe>
2026-04-01 23:33:59 +08:00
LemonNeko 60112eaa85 release: v0.9.0-beta.4 2026-04-01 19:19:22 +08:00
Makito e9be0cf98c release: v0.9.0-beta.3 and ios build 8 2026-04-01 15:35:07 +09:00
Neko Ayaka 497524a22f release: v0.9.0-beta.2 2026-03-31 17:33:53 +08:00