Someone has posted a plugin on X called iPhone Use. It lets Codex directly control the real iPhone you hold in your hand — not a simulator, but the actual physical device connected via USB.

In the demo video, Codex opens WeChat to browse Moments, summarizes group chats, adds a roll of toilet paper to the cart on Taobao, orders takeout on Meituan, and posts on Xiaohongshu. The phone screen is displayed in real time on the computer's sidebar throughout the entire process.

Here's a screenshot of an actual conversation, where the task was to use iPhone Use to open Taobao, select toilet paper, and add it to the cart. The connection dropped halfway through, and Codex recovered automatically, finally verifying the product and quantity in the cart:


This plugin was developed by Zhong Erxin, and the code is hosted on GitHub. The working principle is not complicated: on the iPhone side, it uses Appium-maintained WebDriverAgent (WDA) as the execution service, while the computer runs a local MCP server that communicates with the phone via USB. Through this toolchain, Codex can read the UI tree, capture screenshots, perform taps, swipes, and text input.

A key advantage: it reads the actual UI structure, not just relies on screenshots. When asked how this differs from iPhone screen mirroring, the developer answered: "It can read the UI tree instead of only depending on screen captures, which makes it faster and more stable." The downside of screenshot-only approaches is obvious: they are easily disturbed by loading states, pop-ups, and guide overlays. The UI tree directly provides control positions and attributes, eliminating a layer of guesswork.

It supports a wide range of operations, as shown in the table below:

| Feature | Example |

| --- | --- |

| App navigation | Open target app, navigate to search, detail, and settings pages |

| Page reading | Get control tree, text, positions, and screenshots |

| Taps and gestures | Tap by control or coordinate, swipe, drag |

| Chinese input | Full Unicode support, split long text into chunks |

| Continuous tasks | Combine multiple steps into a single batch |

| List collection | Bounded scrolling, deduplication, return coverage range |

| Failure recovery | Relocate controls via screenshot when they cannot be found |

| Real-time screen | Display phone screen and tap effects in the sidebar |

It also has the ability to scroll backward to a specified date. When asked if it can only read currently loaded messages when summarizing WeChat group chats, the developer confirmed it can scroll, match date labels, and fall back to screenshot analysis when necessary to reach the target date.

The installation barrier is relatively high: it requires macOS + Codex desktop or CLI, a full Xcode installation, an Apple ID capable of signing apps, a real iPhone connected via USB (the phone must trust the Mac and have developer mode enabled), plus Python 3.9+ and Node.js 20.19+. You need to install and launch WDA Runner for the first time, but the developer provides a prompt that lets Codex complete most of the installation process automatically.

After setup, the plugin provides 17 tools including diagnostics, configuration, readiness checks, observation, lookup, app launch, tapping, swiping, input, waiting, batch operations, scroll search, and list collection. When the UI behaves abnormally, the tool first returns a screenshot for the model to determine the next step, instead of blindly repeating swipes or actions.

There are several notable technical design choices:

- Local USB channel: no data is written to disk, not even preview streams.

- Reuses HTTP connections, WDA sessions, and existing builds to reduce redundant work.

- Clear failure semantics: multi-process shared operation lock; timed out actions are marked as indeterminate, and the actual state is checked before deciding whether to retry.

- Screenshot fallback: when element positioning fails, screenshot coordinates are converted to iPhone point coordinates and passed to the tap tool.

- Input and submission are separated; the tool does not submit text by default. Passwords, verification codes, and Face ID are left for the user to complete.

One netizen commented: "The note in README that it checks the actual state first after a timeout makes perfect sense for takeout scenarios: check the order status before deciding whether to retry, so a single disconnection doesn't result in two orders." The developer replied: "Hahaha yes, you really understand Agent Harness."

Another detail: the names of contacts in Moments in the demo video are all blurred. When asked how the blurring was done, the developer said after recording the video, it was sent to Claude Code, which wrote a motion-based script to do the blurring automatically — it doesn't naively process every frame.

There are still limitations. It doesn't work on Windows; the developer notes "It probably doesn't work on Windows yet." Regarding speed, some users have feedback "it feels a bit slow", and the developer did not dispute this.

Project repository: https://github.com/zhongerxin/iPhone-use (currently private, access permission or source code package is required). Licensed under MIT, it depends on the Appium ecosystem and WebDriverAgent. If you have a real iPhone and an Xcode environment, it's worth trying out — it's time to let AI do more than just type on a computer.