Can AI Agents Control a Real Phone With Real Apps?
Capable, not yet competent
Get started with AgentEnv Framework by following our docs.
Frontier agents are increasingly capable of completing complex computer use tasks. But does that translate to phones and apps people use every day? Most phone-use agent environments rely on virtual devices and simulated applications. These controlled environments are valuable for training because they can be reset, run at scale, and allow for database interactions needed for task initialization and verification. But they don't fully capture what it means to operate a real phone. Physical devices have their own constraints, while production third-party apps connect to live services, maintain persistent state, and change based on time, location, and account history.
At Scale, we set out to explore whether agents could complete real-world phone tasks correctly and efficiently. To do this, we built a fleet of physical devices running real third-party applications, allowing us to explore what it takes for agents to successfully operate in realistic mobile environments.
An agent using a phone
Each turn, the agent sees the screen and sends one action: a tap, a swipe, a key, or text.
Moving beyond simulators introduced challenges at every stage. Real phones and apps are harder to make reproducible, agents struggle with interactions humans take for granted, and evaluating success requires examining not just the outcome but the actions taken to reach it. Addressing these challenges required infrastructure to manage physical devices and application state, planning and feedback systems to improve agent execution, and a vision-based judge to fairly grade performance. We discovered that frontier models have the capability to control these real apps, but actually completing realistic tasks in an efficient manner is still beyond their ability.
1. Real apps don't have a reset button
Each phone in our fleet was tied to a persona: a simulated user with a phone number, email, and accounts across real third-party apps. These personas were configured on each device and logged into production applications, recreating the environment an agent would encounter when completing tasks on a person's phone.
Running on real phones and third-party apps introduces challenges that controlled environments largely avoid.
State resets. Unlike controlled app clones, third-party applications don't give us direct control over their backend state. Actions like searching for products, adding items to carts, and modifying wishlists persist between runs. To address this, we introduced a janitor agent that reverses the solver agent's actions after each task, clearing histories, emptying carts, and removing wishlists wherever possible. The app is then closed and the phone returns to its home screen. Some actions, such as placing orders or sending messages, cannot easily be reversed, so safeguards prevent these actions unless explicitly authorized.
Scheduling. Physical devices cannot be parallelized like virtual machines, and some tasks require particular personas. We built a queueing system to ensure that each phone serves only one agent at a time and that tasks are assigned to compatible devices as they become available.
Location and time. Real applications also respond to conditions outside the agent's control. Certain features, inventory, and availability can change depending on where and when a task is performed. To improve repeatability, we pinned devices to specific GPS locations for affected tasks and restricted when those tasks could be assigned to phones.
With these systems in place, we could run agents against real applications while maintaining reasonably consistent tasking environments. But giving agents access to real phones revealed another challenge: they often struggled with even basic interactions.
One task on the device fleet
Every run claims a free device, is judged on what the screen showed, and leaves the device clean for the next task.
2. Where phone agents get stuck
Once agents could navigate real phones, we found that they struggled with many interactions humans take for granted. These failures generally fell into four categories: app navigation, precise movement, long-horizon memory, and repeated loops. Even frontier models struggled with completing these seemingly simple tasks.
App navigation. Agents often struggled to find features that human users navigate intuitively. Filters, search controls, and booking options are frequently buried behind multiple screens, and agents would overlook them even when they offered the most direct path to completing a task. As a result, agents sometimes failed to complete tasks or took unnecessarily long routes.
Precise movement. Agents struggled with interactions requiring fine control. When adjusting price sliders or selecting calendar dates, they frequently overshot their targets and wasted actions correcting themselves. Swipes were particularly challenging because agents needed to select appropriate start and end coordinates, distance, and speed to produce the intended movement.
Long-horizon memory. Tasks involving several requirements exposed another weakness. Agents sometimes forgot which steps they had already completed and attempted them again. They also struggled to retain and compare information collected across multiple screens, such as comparing nutritional information between grocery products.
Repeated loops. When an approach failed, agents often repeated the same unsuccessful actions rather than exploring another path. Some became stuck in long sequences of repeated attempts, even after being instructed to change their approach.
Where phone agents get stuck
Four failure patterns we saw again and again.
3. Planning and feedback improve agent execution
These failures made it difficult to consistently generate trajectories where the agent efficiently completed the task. We experimented with several forms of agent assistance and found two approaches particularly effective: providing an expert-informed navigation plan before execution and targeted feedback after unsuccessful attempts.
The planner agent was given detailed knowledge of the application's navigation paths and generated a step-by-step plan for the solver to follow. With this guidance, the solver could locate filters and other app features without spending unnecessary actions exploring the interface. The plan also helped address long-horizon memory failures by giving the agent a reference for tracking completed steps and determining what to do next.
A planner turns the task into checkpoints
One coffee-delivery run (62 actions): the plan's sub-goals, and the action where the agent completed each.
We also introduced a feedback loop between attempts. After each evaluated run, a feedback agent analyzed what went wrong and provided targeted advice for the next attempt. Unlike the initial plan, feedback addressed mistakes specific to the previous trajectory. We found that giving agents two or three attempts with feedback substantially increased their chances of properly completing the task.
The feedback loop
The judge's finding on attempt 1 becomes part of attempt 2's instructions.
These improvements helped us generate more successful and efficient trajectories without changing the underlying model. But identifying which trajectories actually met our standards required a separate solution.
4. Why task completion isn't enough
In a synthetic app environment, we can check task success by verifying the agent's behavior against the app database's end state. For real applications, we needed to rely on the screenshots, tool calls, and narration within the trajectory. We found that the agent's narration was unreliable and that the last few screenshots of the end state were often insufficient for judging success. To reliably grade a task, we needed to validate each rubric criterion against associated screenshot-action pairs from the full trajectory.
We started with a rubric generated alongside each task, where every criterion was a requirement of the task. Some of those requirements can be verified from the last screen, and others only by looking at what the agent did along the way. For a task that asks for the cheapest flight on specific dates, the final booking screen confirms the dates, but it says nothing about whether the agent compared other flights first, because that happened several screens earlier. A judge that only looked at the end state would pass that run whether or not the agent ever checked the price.
The agent's narration was even less reliable. An agent might report a completed grocery order when one item was missing and another had been added at double the quantity, and about a third of runs that claimed success had in fact failed at least one criterion on screen. We repeatedly encountered cases where agents' descriptions of their actions did not match what actually happened on screen.
What the agent said vs. what the screen showed
Roughly a third of runs that claimed success had failed at least one criterion on screen.
We therefore developed a vision-based judge that examines screenshots and tool calls throughout the entire trajectory. We settled on a budget of up to 90 frames (the median run was about 50 actions), selected and tagged per rubric criterion.
But satisfying the explicit asks doesn't always ensure that the agent completed the task in an acceptable manner. We also needed to account for execution quality: unnecessary actions, repeated attempts, and inefficient navigation. We introduced an efficiency judge to evaluate these behaviors alongside the task rubric. A trajectory only passed if every rubric criterion was marked complete, the agent didn't go off track, and it remained within our thresholds for unnecessary or repeated actions.
Finally, we validated the evaluation system against human reviewers. Reviewers graded and edited thousands of runs and rejected half with documented reasons. When we re-graded a sample using the production judge, it agreed with reviewers on 94% of the runs they had approved. When the judge accepted runs that reviewers had rejected, the problem was often an incomplete rubric rather than the judge itself. In these cases, improving the evaluation meant refining our definition of success rather than simply changing the judging model.
5. What real-world phone use requires
So, can AI agents control real phones and real apps? Yes, but giving an agent access to a device is only the beginning. Agents still struggle with basic UI interactions, lose track of requirements, and struggle navigating unfamiliar application interfaces. Even when an agent reaches the correct outcome, it often fails to do so reliably and efficiently.
Our experience testing these agents in the real world reveals how much of the challenge lies outside the underlying model. We needed to build infrastructure and agentic tooling to make evaluation-ready environments on real devices, provide agents with guidance to navigate those environments effectively, and develop new methodology for grading their performance.
Simulated environments remain valuable for scalable training, while real devices and live applications let us collect and evaluate behavior in the conditions agents will encounter outside controlled settings. Building the infrastructure to make those interactions reproducible, effective, and verifiable is an important step toward more capable real-world phone agents.
Try AgentEnv Framework
It's open source, and one install away. Its hello task runs on your machine with no Docker, model or cloud account.
pip install agentenv-frameworkThen follow a cookbook to build a complete task, from an empty folder to a graded run.
Acknowledgements
Written by Sarah Chitty, Kai Hackney, and Judah Engel.
To put your own agents on a real Android or iOS phone, try the Android and iOS Mobile RL Env plugin (Mobilerun) for AgentEnv Framework.