AI-Controlled Real Devices: XPhoneUse Implementation and Pitfalls
XPhoneUse packs an AI Agent into a real phone. It mirrors your device just like Scrcpy, but adds a Skill layer that lets Claude or Codex control the actual hardware. The workflow is dead simple: connect the phone, let the Agent install the Skill link, and start issuing commands. In our test, DeepSeek V4.1 paired with OMP handled language learning apps (Duolingo Japanese lessons), checked streaks, and sent messages to specific WeChat contacts.
Core mechanism: The Agent “sees” your phone through the ADB channel and the accessibility tree. If that tree fails, the Agent goes blind. In one case from the article, WeChat’s interface returned empty tree nodes, so the Agent wrote its own OCR tool based on Swift Vision to read coordinates from screenshots instead of relying on tree nodes.
Safety rules: The Agent follows an “unknown means no action” principle. It won’t tap blindly when it can’t see the screen. For sensitive actions—like sending red packets on WeChat or opening Duolingo treasure chests—it only executes explicit commands or closes the flow, never improvising. When faced with an audio quiz it can’t process (no hearing model), it admits defeat rather than guessing, protecting the user’s streak.
Troubleshooting and fixes: A bottom-level coordinate bug caused misplaced taps—the screen is 1080×2340, but normalized values like 472×1024 overflowed. The Agent switched to passing device pixels directly via `input tap` and updated the Skill docs to flag the issue. It also patched a silent failure caused by an argparse naming conflict in a shell subcommand.
Risks: The biggest danger is the lack of rollback. On a real device, sending the wrong message or accidentally confirming logout from WeChat Web is irreversible. The Agent relies entirely on a single-step “see–tap–see” verification loop to keep every action accurate. This tech shines for automated testing and unattended mobile operations, but users must stay sharp about the social or financial fallout from a mistaken command.
Source · Appinn: Read original article →