XPhoneUse: Let AI Agents Directly Control Real Phones

AI Summary · From a Serial Entrepreneur’s Perspective (This content is synthesized by AI; all opinions belong to the original author. Reading this is sufficient—skipping the original article is fine.)

XPhoneUse lets agents like Claude, Codex, and others control a real Android phone directly via ADB mirroring and Skill packages. It can handle tasks ranging from Duolingo lessons to sending WeChat messages. The article walks through a classic failure mode: when the accessibility tree breaks, the agent goes “blind.” It shows how the agent wrote its own OCR tool on the fly, established a “don’t touch what I can’t see” safety rule, and patched a coordinate-mapping bug at the system level. For builders, this is core infrastructure for automated mobile operations and AI interaction products—but the lack of an undo mechanism makes accidental taps genuinely risky.

• Mobile automation via ADB + Skill packages…
• Non-standard UIs (like WeChat) require a custom OCR channel or a switch to visual models
• Establish a “verify every action” SOP to prevent accidental sending of sensitive messages
• The toolchain can be productized as a “mobile AI ops assistant”…
• Watch out for the device-coordinate normalization bug; hardcoded pixel values are more stable

AI-Controlled Real-Device Automation: From Scrcpy to the XPhoneUse Workflow

The playbook has shifted. You no longer need expensive emulators or purpose-built AI phones—just an idle Android device and a USB cable. XPhoneUse gives agents like Claude, Codex, and DeepSeek actual “eyes and hands” through ADB mirroring and Skill packages. Setup is lean: install the client, plug in the phone, point the agent at the Skill package, and start issuing commands. The agent handled Duolingo quizzes and WeChat messages end-to-end, even writing an OCR tool on the fly when the accessibility tree went dark. This is foundational infrastructure for unattended mobile operations and automated testing. But real devices have no undo button: one stray tap or a misread confirmation dialog is irreversible.

After-action report: when the agent went blind

The case study comes from a developer running DeepSeek V4.1 Flash. The mission set included grinding Duolingo, checking streaks, sending WeChat messages, and sitting through a Japanese lesson. For the first forty minutes, the agent read the Duolingo UI cleanly through the accessibility tree—down to ID-based answer validation. Everything broke when WeChat opened: the tree returned empty and the screen rendered as `[disabled]`. Without a visual model, that’s total blindness. The agent didn’t throw an error; it simply reported being unable to confirm what was on screen.

It then wrote a 30-line Swift script on a Mac, tapped the Vision Framework, and turned screenshots into “text + coordinates.” Slow, but it restored operability. Early on it made a textbook mistake: it clicked the first row of the chat list before OCR was ready and landed on a system conversation that popped up the “log out of WeChat Web” confirmation dialog. It hit back without confirming, which set new rules for every session after that: don’t interact with anything you can’t see, verify the result after each action, and invalidate coordinates whenever the screen scrolls. It even wrote a fresh Skill to close that blind-spot loop.

Boundary controls and low-level bug fixes

The agent showed critical rejection discipline. Before sending a WeChat message, it read the contact name at the top to confirm the recipient. For the Duolingo reward popup, it clicked “Close” on both the treasure chest and the gift options—those belong to the user and fell outside the task scope. When a listening exercise came up, it skipped rather than guessed randomly, protecting the user’s streak. It also patched two low-level bugs: a silent failure caused by a shell subcommand parameter name collision, and a coordinate normalization error where the real screen size of 1080×2340 was treated as 472×1024, pushing coordinates off-screen. It switched to absolute pixel values via `input tap` and documented the whole mess so the next person wouldn’t repeat it.

Outcome and risk warnings

By the end, the agent finished a Japanese lesson, nailed all six questions, and ran the whole thing in 4 minutes 11 seconds—pushing the user’s streak from 1659 to 1660 days. It then reset the device: locked the screen, disabled always-on, and deleted the temporary screenshots. Real-device control is fundamentally different from sandboxed code: code errors roll back; phone taps don’t. The biggest risk is the absence of any rollback mechanism. The agent’s “don’t touch what I don’t understand” rule is the main defense against misclicked payments, accidental logouts, or sensitive-message leaks. For builders, this toolchain can be wrapped into a “mobile AI ops assistant” for automated testing or unattended tasks on specific apps—but you must restrict the agent’s actionable area and block access to payment flows, account deletion, and other high-risk actions. Note: iPhones aren’t supported; this works only on Android. The tool lowers the barrier to mobile AI automation while simultaneously amplifying the cost of mistakes. Safety SOPs matter more than raw speed.

Original post · Appinn: Read the original →