GPA: Learning GUI Process Automation from Demonstrations
GUI Process Automation (GPA) is a lightweight but general vision-based Robotic Process Automation (RPA), which enables fast and stable process replay with only a single demo. Addressing the fragility of traditional RPA and the non-deterministic risks of current vision language model-based GUI agents, GPA introduces three core benefits: (1) Robustness via Sequential Monte Carlo-based localization to handle rescaling and detection uncertainty; (2) Deterministic and Reliability safeguarded by readiness calibration; and (3) Privacy through fast, fully local execution. This approach delivers the adaptability, robustness, and security required for enterprise workflows. It can also be used as an MCP/CLI tool by other agents with coding capabilities so that the agent only reasons and orchestrates while GPA handles the GUI execution. We conducted a pilot experiment to compare GPA with Gemini 3 Pro (with CUA tools) and found that GPA achieves higher success rate with 10 times faster execution speed in finishing long-horizon GUI tasks.
69cf17dde02fdd68b91b7a01
Zirui Zhao
claimed_verified
2026-04-03T09:08:02.449Z
65abd99ea92a64ef5bb59325
/avatars/1809479790584c5be6a5d370042155a9.svg
Zirui Zhao
ryanzzr
user
ryanzzr
69cf17dde02fdd68b91b7a02
Jun Hao Liew
claimed_verified
2026-04-03T09:08:00.363Z
6305bb149d2531fabd198638
/avatars/0ec3c55d445264d43c0430f9edf88bf8.svg
Jun Hao Liew
junhao910323
user
junhao910323
69cf17dde02fdd68b91b7a03
Yan Yang
69cf17dde02fdd68b91b7a04
Wenzhuo Yang
69cf17dde02fdd68b91b7a05
Ziyang Luo
69cf17dde02fdd68b91b7a06
Doyen Sahoo
69cf17dde02fdd68b91b7a07
Silvio Savarese
69cf17dde02fdd68b91b7a08
Junnan Li
GPA: Learning GUI Process Automation from Demonstrations
GUI Process Automation (GPA) is a lightweight but general vision-based Robotic Process Automation (RPA), which enables fast and stable process replay with only a single demo. Addressing the fragility of traditional RPA and the non-deterministic risks of current vision language model-based GUI agents, GPA introduces three core benefits: (1) Robustness via Sequential Monte Carlo-based localization to handle rescaling and detection uncertainty; (2) Deterministic and Reliability safeguarded by readiness calibration; and (3) Privacy through fast, fully local execution. This approach delivers the adaptability, robustness, and security required for enterprise workflows. It can also be used as an MCP/CLI tool by other agents with coding capabilities so that the agent only reasons and orchestrates while GPA handles the GUI execution. We conducted a pilot experiment to compare GPA with Gemini 3 Pro (with CUA tools) and found that GPA achieves higher success rate with 10 times faster execution speed in finishing long-horizon GUI tasks.
GUI Process Automation (GPA) offers robust, deterministic, and privacy-preserving vision-based robotic process automation with faster execution than current vision-language model approaches.
GUI automation Robotic Process Automation vision-based RPA Sequential Monte Carlo localization readiness calibration fully local execution GUI agents CUA tools long-horizon tasks