Why AI Models Should Not Own the System Model

AI models are powerful interpreters. They are not reliable authorities for the current state of a software system.

This is a separation-of-responsibilities problem, not a judgment about model quality.

Interpretation is not authority

An AI model can:

• summarize source code;
• infer likely dependencies;
• propose ownership or risk classifications;
• identify missing evidence;
• suggest commands and verification plans.

But a shared system model must also answer:

• Which source produced this fact?
• At which revision and time was it observed?
• Is it verified, inferred, stale, conflicting, or unknown?
• Which contract makes the fact machine-readable?
• What invalidates it?

A generated sentence does not answer those questions by itself.

The safe boundary

The agent participates in the system without becoming its root authority.

Why ownership creates failure modes

If each agent owns its own system model:

• two agents can act on incompatible architectures;
• prompt changes silently change system truth;
• model upgrades become schema migrations without governance;
• hallucinated relationships become operational inputs;
• the reasoning path cannot be reproduced by CI or another agent.

This does not mean every fact must be extracted without AI. Inferred facts can be valuable when they are labelled, scoped, linked to observations, and kept separate from verified evidence.

The design principle is simple:

AI may propose meaning. The system must govern truth.