← Back to Concept Contrast

Concept Contrast Fable

Computer use and GUI agents

Conclusion first:One is about "how to operate things on the screen," and the other is about "what goal to pursue to complete a whole sequence of screen operations." Mistaking how to operate for the goal itself makes it easy to build a system that can click buttons but does not know what to do after clicking.

One is about "how to operate things on the screen," and the other is about "what goal to pursue to complete a whole sequence of screen operations." Mistaking how to operate for the goal itself makes it easy to build a system that can click buttons but does not know what to do after clicking.

Fable

A hand that can click a mouse and a hand with a goal

Old Qian's money house still used a row of old counters, with seals, drawers, and account pages arranged according to old rules. The clever new apprentice managed two seemingly similar matters: one was seeing what was on the counter and copying it by pointing, pressing, moving, and writing; the other was knowing which counters a task had to pass through from beginning to end. The new apprentice found this troublesome and merged the two ledgers into one, thinking that both were related to success or failure anyway.

At first no one objected. The apprentice could copy an address slip into the ledger, so everyone assumed he could complete the entire relocation errand alone. When the shopkeeper saw fewer pages, he even praised them for saving effort. But when peak season arrived, the old accounts suddenly could not be explained clearly: whether the previous stall had not finished or the next stall had not held the line, no one could answer.

They first added more seals and slogans and required everyone to write in more detail. As a result, the booklet grew thicker and disputes increased. When something went wrong, everyone could find a sentence to excuse themselves, but no one knew which end to save first.

The old steward called everyone into the courtyard and had them retrace the actual course of events. Whatever concerned the immediate action went into the first book; whatever concerned subsequent responsibility went into the second. The two books referenced each other, but were no longer mixed onto the same page.

He stopped after writing the first page; the escort rosters, monthly settlements and receipts that followed were left untouched. Once separated, many old conclusions changed: some work looked busy but never actually blocked trouble; some arrangements seemed unremarkable in peacetime yet preserved the whole in disaster.

To prevent newcomers from mixing things up again, the old steward had them each handle a small case. Only when the two ledgers could each answer their own questions and also connect with each other at the boundary would the small case pass. If someone still wrote the two matters on one page, they had to return to the scene and look again at who acted first and who was ultimately responsible; when it was unclear, it was better to ask once more than let a muddled account stay overnight; because on the day something truly went wrong, one missing line of distinction would make an extra team work in vain.

The shopkeeper later gave him a task roadmap: first update the general ledger, then notify the escort office, and finally collect the old receipts; any misstep meant going back to the previous step. Later, when newcomers entered the trade, the old steward had them remember just one line: being able to operate the object in front of you only counts as skill with your hands; being able to safeguard the whole course of a task is what makes you trustworthy.

Computer Use

The ability of a model to directly operate computer interfaces, including understanding screen content, locating UI elements, mouse clicks, keyboard input, scrolling and other basic operations. It solves "how to interact with a graphical interface" and is not responsible for deciding the business goals and order of operations.

GUI agents

An intelligent system that, on top of computer-use capability, adds goal planning, step orchestration and completion-condition verification. It breaks a business goal into multiple GUI operation steps, executes them in order and checks the completion status of each step, autonomously adjusting strategy when exceptions occur.

Story Mapping

  • The model stops after completing the save: computer use solved operational capability, but not goal orchestration capability.
  • After an address change, you also need to update the logistics system and file compliance tickets: real business goals require coordination across multiple systems.
  • The GUI agent framework adds business process templates: on top of computer use, it layers goal decomposition and step orchestration.
  • A layered diagram where the lower layer handles operations and the upper layer handles goals: each layer does its own job, with capabilities stacked but responsibilities different.
  • Repetitive data entry scenarios do not need a GUI agent: the complexity of the scenario goal determines whether an orchestration layer is needed.

In the project

When evaluating GUI automation needs, first draw an operation flow chart and label each step with two tags: operational complexity (whether buttons are clear, whether windows are standard) and goal complexity (whether it crosses systems, whether it depends on prior step results, whether there are branching decisions). Scenarios with high operational complexity but low goal complexity should prioritize computer use modules; high goal complexity, even if operations are simple, requires a GUI agent framework to manage step orchestration and exception handling.

京ICP备2026014272号