Evaluating Workspace Intelligence Systems

Workspace Intelligence systems should be evaluated by the decisions they can support, not by the number of files indexed or tokens supplied to a model.

Evaluation dimensions

Coverage

Measure which system entities and relationships are represented:

repositories, projects, services, and runtimes;
dependencies, contracts, ownership, and policy;
commands, artifacts, evidence, and release gates;
change, impact, and verification relationships.

Coverage should include explicit unknowns. A system that omits unsupported relationships can look more complete than one that reports its gaps.

Evidence quality

Sample important facts and verify that provenance, scope, revision, confidence, and freshness are available. Separate observed, verified, inferred, stale, conflicting, and unknown states.

Freshness and invalidation

Change a source and measure:

whether affected facts become stale;
whether unrelated evidence remains valid;
how quickly the model converges;
whether consumers can identify the new model revision.

Explainability

Ask the system to justify an impact or verification recommendation. The answer should trace through typed relationships and evidence, not only provide fluent prose.

Interoperability

Use at least two different consumers. Confirm that they resolve the same entity identities and evidence states from versioned contracts while receiving appropriately scoped views.

Operational usefulness

Evaluate real tasks:

onboarding an existing project;
preparing an agent context;
identifying affected projects after a change;
selecting verification commands;
explaining a blocker;
producing release evidence.

The output should reduce reconstruction effort without hiding uncertainty.

Failure tests

A credible evaluation includes adversarial conditions:

malformed or missing manifests;
conflicting runtime signals;
moved repositories and renamed projects;
stale caches and partial scans;
unsupported consumer versions;
incomplete ownership and policy data;
changes during model generation.

A minimal scorecard

Dimension	Weak signal	Strong signal
Coverage	file count	typed system entities and explicit gaps
Trust	generated summary	evidence, confidence, scope, freshness
Change	full blind rebuild	targeted invalidation with revision tracking
Explanation	prose only	traceable entities, edges, and evidence
Integration	vendor-specific prompt	versioned consumer contract and adapters
Outcome	more context	safer decisions and reproducible verification

This framework keeps the category focused on system understanding rather than equating intelligence with retrieval volume or model fluency.

The scorecard is designed as a contract-and-scenario evaluation rather than a single synthetic benchmark. Begin by selecting a representative workspace that contains multiple projects, at least two runtimes, an explicit contract, and a repeatable verification command. Record a baseline model and then run a fixed sequence of interventions: edit a dependency, rename a project, invalidate a test result, introduce a conflicting runtime signal, and remove an ownership declaration.

For every intervention, capture the machine-readable model revision, changed entities, invalidated evidence, impact output, verification recommendation, and explanation trace. Two independent consumers should then read the same versioned artifacts. Agreement is measured on entity identity and evidence state, not on identical prose.

The evaluation should report both correctness and abstention. A correct unknown is preferable to a confident relationship that cannot be traced. Results should therefore include false assertions, missed relationships, correctly reported gaps, time to convergence, and the amount of unaffected evidence reused after each change. This makes the procedure reproducible while remaining useful across repositories with different languages and layouts.

Limitations

This framework does not define a universal numeric score. Weighting ownership, release safety, graph coverage, or freshness depends on the system and the decision being supported. It also cannot prove that a model captures tacit organizational knowledge that was never represented by an observable source.

Runtime behavior, production topology, and human approval boundaries may require evidence outside the workspace. A strong result therefore establishes fitness for the tested scope and revision; it must not be interpreted as proof of complete understanding. Comparative evaluations should publish their workspace fixture, supported relationship types, unavailable evidence, and consumer versions so readers can interpret results without assuming broader coverage.