Concept Contrast Fable

Observability and LLM observability: one is monitoring for the whole building, the other is a camera in the elevator

Conclusion first:Many teams adopt an LLM and assume their existing application monitoring already covers it. In reality, observability is a panoramic view spanning infrastructure, applications, and business, while LLM observability focuses only on signals unique to large models—token consumption, hallucination trends, safety filter hits, retrieval quality. Using general monitoring to watch an LLM is like using a thermometer to measure server temperature.

Many teams adopt an LLM and assume their existing application monitoring already covers it. In reality, observability is a panoramic view spanning infrastructure, applications, and business, while LLM observability focuses only on signals unique to large models—token consumption, hallucination trends, safety filter hits, retrieval quality. Using general monitoring to watch an LLM is like using a thermometer to measure server temperature.

Fable

The mall's central control room and the ice cream stand's thermostat

A new commercial complex opened downtown. The central control room had an entire monitoring wall showing foot traffic density, air-conditioning temperature, elevator operating status, and fire alarms on each floor. The mall manager felt this wall was already enough to see the entire mall's operating condition.

A handmade ice cream stall newly moved into the first floor. The owner's ingredients were extremely temperature-sensitive: cream began to separate above four degrees, and jam darkened above six degrees. In the first two weeks after opening, the ice cream's texture received many negative reviews. The manager looked at the master control wall: the mall's overall temperature was twenty-two degrees year-round, and the air conditioning system had no alarm records. He thought the problem was in the recipe.

The stall owner bought a small temperature controller and placed it next to the refrigerated cabinet. A few days later, he discovered the cause: the mall's central air conditioning automatically reduced power during low-traffic periods, and the ice cream stall happened to be in a dead corner of the air outlet. Every day from 2 p.m. to 4 p.m., the actual temperature of the refrigerated cabinet quietly climbed to seven degrees, then dropped back after the central air conditioning returned to full power. This four-hour window was completely invisible in the floor average temperature on the main control wall.

The stall owner took the thermostat data to the mall and negotiated an adjustment plan: not changing the mall's entire temperature-control strategy, but adding a small independent cooling module and a dedicated temperature probe near the ice cream stall. The change was small, but it hit the problem.

The mall later wrote this lesson into its leasing handbook: standard surveillance covers public areas and infrastructure; if a specific business type depends on environment-sensitive equipment, it needs its own dedicated monitoring. The central control wall was not wrong, but the granularity it was designed for could not manage the ice cream stand's cream.

Later, when a milk tea shop entered, it brought four probes directly: one to measure the temperature of the tea base, one for the pearl-cooking pot, one for the takeout packing area, and one for how long takeout riders lingered. The control room only took data from the first three onto the big screen; the fourth was the milk tea shop's own business logic, for the store manager to view alone.

The mall operations director once said at a sharing session: The master control wall lets you know the mall is alive; dedicated probes let you know whether this business can survive. If you insist on cramming everything onto the same wall, either the wall becomes too heavy or the signal becomes too sparse.

Observability

A monitoring, logging, tracing, and alerting system covering the full stack of the system, answering whether the system is running normally; it is a general capability.

LLM observability

Observability specifically for large model calls, tracking LLM-specific signals such as token usage, hallucination rate, safety filtering, and retrieval relevance.

Story Mapping

The master control wall monitors foot traffic, air conditioning, and elevators: general observability covering infrastructure and the application layer
A dedicated thermostat for the ice cream stand: LLM observability, tracking signals unique to large models
The dead corner at the air outlet quietly warms up in the afternoon: the granularity of general monitoring cannot capture LLM-layer problems
The mall manual requires specific business types to configure dedicated monitoring: LLM scenarios need a dedicated layer independent of general monitoring

Applying it to an Agent project

General observability continuously tracks metrics like latency, error rate, and resource utilization. After an Agent connects to an LLM, set up a separate LLM observability layer to track at least the token consumption of each call, whether the output conforms to the expected format, whether the safety module intercepted content, and whether the retrieval step returned empty results. Keep the two layers of data on separate dashboards; do not expect one Grafana panel to cover everything.

Observability LLM observability