← Back to Concept Contrast

Concept Contrast Fable

Agents that operate general interfaces and agents that operate web pages

Conclusion first:GUI agents target arbitrary graphical interfaces — desktop software, backend systems, even non-browser windows. Web agents mainly target browsers and web pages. The two may appear to both click buttons, but their operating objects, observability and failure modes differ greatly.

GUI agents target arbitrary graphical interfaces — desktop software, backend systems, even non-browser windows. Web agents mainly target browsers and web pages. The two may appear to both click buttons, but their operating objects, observability and failure modes differ greatly.

Fable

Two runners appeared in one government office

The county office had two runners, both sent out on errands. Old Jia could enter any shop — no matter how out-of-the-way the storefront, how high the counter, or whether the drawer was on the left or the right, he pushed the door open, read the sign, felt the counter, and opened the drawer. Old Yi only recognized the official notice boards along the street; he knew the format of every notice, the position of the signature, and the style of the seal, and could recognize them even on different paper.

Once, all the shops in town changed their signs—wooden plaques became bronze, and counters moved two steps forward. Old Jia went in, looked around, and found the ledger shelf in the time it took to drink a cup of tea, then got his business done as usual. Old Yi encountered three newly installed glass doors on the street, did not know how to push them, and stood frozen outside. That day everyone said Old Jia was the capable one.

On another day, the county magistrate wanted to see all the notices from every cloth shop in the city over the past three months. Old Jia visited every shop one by one, taking five days, and sometimes missed notices because shopkeepers had rolled them up in a corner. Old Yi pulled out the archive index of the notice board and listed the complete inventory in the time it takes two incense sticks to burn. The magistrate said Old Yi was the real treasure.

What truly exposed the difference was the assignment of disbursing silver. Old Jia entered the storeroom to check goods and saw the account book on the table, the silver in the chest, and the seal in the drawer, all handled properly; another assignment required only filling out a form in the official document grid according to the format. Old Jia instead could not find the blank where he should write, because that form was tucked behind a folded page, and he took the folded page for another case register and stood there frozen.

The secretary began assigning tasks by type. Anything that required entering a room to inspect the counter, feel the drawers, and identify the shop's furnishings was given to Old Jia; anything that required searching documents along the notice-board catalog, filling out forms according to the grid, and checking the signature format was given to Old Yi. But no one could say who should be assigned a task that required both entering the warehouse to count goods and copying the results onto the street notice board—each of them could only handle half.

Once a government office changed its new process: first copy data from the warehouse ledgers, then fill in the supplier receiving form at the city gate notice board. Old Jia could complete the first half, Old Yi could complete the second half, but the paper format changed twice in between, and the notations for weight and box count didn't match, and no one noticed. It wasn't until the supplier sent a return order that they discovered the manual bridging step had been missed.

The advisor later made a table dividing assignments into three categories by object of operation: those touching only the shop counter, those touching only public notices and documents, and those requiring a bridge between the two. The third category needs specially designed handoff logic; you cannot expect two runners to coordinate automatically. Treating both runners as the same type of person and assigning work the same way wastes time on the wrong division of labor.

GUI agents

Controls any graphical interface by identifying screen elements and simulating mouse and keyboard operations, not limited to browsers. Relies on pixels or the accessibility tree to perceive interface state.

Web agents

Primarily operates web pages within a browser, using the DOM tree, HTML structure, and web interaction patterns to navigate, fill forms, and extract information.

Story Mapping

  • Old Jia can enter any shop: a GUI agent that faces any graphical interface, relying on vision and element perception
  • Old B only recognizes the notice board: web agents, dependent on web page structure and DOM information
  • The shop can still do business after changing its sign: pixel-level adaptation for GUI agents
  • Notice archive listed in two incense sticks' time: efficient retrieval by web agents using structural information
  • Old Jia cannot find the pop-up window: the blind spot of GUI agents for web-specific interaction patterns
  • Data leaked twice in transit: workflows spanning desktop and web need dedicated bridging logic

Down to automation selection

When the operating target is a web page, prioritize a web agent—parsing the DOM is faster, more accurate, and more cost-effective than pixel recognition. Only when the operating target is desktop software, a bank terminal, a backend management system, or requires cross-window interaction should you use a GUI agent. Complex cross-browser and desktop workflows cannot be forced through by a single agent; you need to design dedicated handoff steps and validate intermediate data. When a vendor says it can operate any interface, ask clearly whether it relies on DOM or screenshot recognition—this directly affects accuracy, speed, and maintenance cost.

京ICP备2026014272号