← Back to concept explainers

Concept Fable

Data connector

Conclusion first:A Data Connector loads external files, databases, SaaS content, web pages, or APIs into an AI application's indexing and retrieval pipeline. It typically handles authentication, incremental sync, format conversion, metadata, permission labels, error recovery, and data lineage, and is the entry point that enables enterprise RAG and knowledge base Q&A to run continuously and reliably.

Data Connector · Data ingestion integration

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

The waterwheel sends every village's ledger into the same mill

The three mills in the valley each recorded grain prices, inventory, and credit separately, but the chamber of commerce wanted to view them together every day. When people were sent temporarily to copy the ledgers, units, times, and missing fields were always inconsistent, and a late page could cause the general ledger to misjudge.

The bridge repairer installed an account-taking port on each mill: first authenticate the visitor, then take incremental accounts in the agreed format, preserving source, time, and weight units. When there is a bad account or a broken bridge, the entrance clearly returns an error and does not write guesses into the general ledger.

The guild also found that some warehouse ledgers were only allowed to be viewed by their own workshop. So the ledger-fetching gateway carried permission tags along, synced only newly added or changed records, and left a trace of who read them and when.

A rainstorm damaged the waterwheel, and the bridge repairer continued from the last successful position instead of copying all three mills from the beginning. Retries, rate limiting, and handling of duplicate records were all written into the rules.

A connector is therefore not a one-time migration script, but the boundary responsible for authentication, transformation, synchronization, and traceability when external systems enter AI retrieval and workflows.

Concept Anchor

This story is about: data connectors

A Data Connector loads external files, databases, SaaS content, web pages, or APIs into an AI application's indexing and retrieval pipeline. It typically handles authentication, incremental sync, format conversion, metadata, permission labels, error recovery, and data lineage, and is the entry point that enables enterprise RAG and knowledge base Q&A to run continuously and reliably.

What matters is that it is not just a term, but a structural problem that repeatedly appears in real AI / Agent systems. Understanding it lets you judge when to add a model, when to change a process, and when to add governance.

Story Mapping

  • Village ledgers: files, databases, SaaS, web pages, and business APIs
  • Manual data transfer: manual upload or one-time import
  • Unified ledger requirement: forcing source systems to be rebuilt, with high real-world cost
  • Water wheel: data connector / loader
  • Extract, Transform, Filter: ETL / ingestion pipeline
  • Permissions, versions, and source labels: metadata, ACL, provenance
  • Refetch only what changed: incremental sync
  • Final insight: Data Connector determines whether an AI system can continuously obtain fresh, compliant, traceable data

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

loaderingestion pipelineETLLlamaHubconnectorRAG
京ICP备2026014272号