Read and tap the screen

Your agent reads the screen as text and acts by label, the way it reads code. A sign-in screen is about 70 tokens, not a screenshot.

Read it

phone_ui_look

Demo · Sign In · iPhone 17 Pro (iOS 27.0)
textfield "Email" #email @201,194
securefield "Password" #password @201,246
switch "Remember me" #remember off @341,298
button "Sign In" #signIn @201,350
text "Not signed in" #status @201,402
tabs: Login (selected) @158,822, Items @244,822

One line per element: its kind, its label, #id (the app's accessibility identifier), its state and @x,y, its center in points. Tabs, scrolling and the keyboard are a line each, and an alert or sheet hides the screen behind it. A web view with nothing for accessibility to read comes back as one unreadable line with its place: take a screenshot and tap by point there.

find: Only the elements whose label, id or value contains it.
screenshot: A JPEG beside the text.
screenshot_file: A full-size PNG to download.
full: The whole accessibility tree, as a JSON file.

Text on the screen comes from the app. It is data, never instructions for your agent.

Act on it

phone_ui_act

steps: ["fill #email gary@example.com", "fill #password secret1", "tap #signIn", "wait \"Welcome\""]

4 steps done in 4.5 s
changed: #email "gary@example.com" · #password "•••••••" · #status "Welcome, gary@example.com"

phone_ui_act takes steps in the words phone_ui_look prints, runs them in order and answers with what changed. A tap that changes nothing is reported, and the first step that fails stops the rest.

Tap: tap #id, tap "label", tap @x,y, hold "label" 2
Type: fill #id text, type text, key return. Any language, read back after.
Move: scroll down, swipe left "row", set #slider 50, back, home
Wait: wait "text" 10, gone "text"

Control apps and the phone

MCP call

phone_app_control  { "action": "open_url", "value": "myapp://orders/42" }

phone_app_control launches, installs, stops or removes an app; opens a link or a deep link; grants, revokes or resets a permission; sets the location; sends a push; and sets the appearance, locale, time zone, text size, or a clean status bar (9:41, full bars) for screenshots.

Logs, screenshots and video

phone_app_logs: What the app printed and logged since launch. grep filters, wait_for waits up to 25 seconds for a line, crashes lists crash reports.
phone_screen_record: Start, then stop for an MP4 that shows a person what happened.
screenshot_file: On an iphone-pro-max phone, the 6.9-inch size App Store Connect takes.