The eval framework.

What is an operator eval?

An Operator Evaluation measures how a person or team actually operates AI systems across relevant tasks, workflows, models, and operating conditions. It is not simply a test of AI knowledge. It examines observable operating behavior and performance.

What is measured?

MO§ES™ works from content-free or content-minimized telemetry — token counts and structural signals, not prompt text. The core data surface is four signals:

Level 2 deployments add: timestamps, sessions, model/tool/agent identifiers, acceptance signals, retries, errors, and richer provider fields where available. No prompt content is required for any core measurement.

Five canonical derived metrics

I = Input, O = Output, R = Cache Read, W = Cache Write. All metrics are computed from canonical telemetry. No prompt content is required.

How are benchmarks created?

A performative benchmark compares how operators actually perform under observed or defined operating conditions — rather than relying only on self-reported proficiency, certification, or generic knowledge tests.

Six benchmark types

Percentile bands

Benchmarks should be contextual. A high-performing operator in one workflow may not be the strongest in another. "Best operator" should require the same qualifier as "best model." Best at what?

How are bespoke evals constructed?

MO§ES™ does not force every company into the same AI proficiency model. Enterprises define the workflows, roles, models, behaviors, risks, and operational questions that matter to them. MO§ES™ turns those into measurable, repeatable evals.

Enterprises define:

Then construct repeatable operator evals around those conditions. Configurations are saveable as JSON and shareable across CLI, TUI, and MCP.

How are results interpreted?

The system never collapses measurement, diagnosis, intervention, and validation into a single claim. Every output carries an evidence label that declares its epistemic status.

Operators are analyzed as a population.

Individual operators are measured in the context of the cohort, not in isolation. The cohort has distribution, shape, clusters, and movement.

Extreme performers are not noise.

In operator fields, performance forms a broad distribution with clear high-performing tails. Outliers are not discarded — they are analyzed. The top 1% and top 0.1% may reveal operating patterns that the rest of the cohort has not yet developed.

However, not every metric creates the same ranking. Top operators can differ by task, workflow, model, operating style, efficiency, leverage, and output behavior. The highest-volume AI users are not necessarily the highest-performing operators.

Every measurement carries its history.

• Source telemetry window and operator identifiers
• Evidence label (PROVEN, MEASURED, OBSERVED, DERIVED, HYPOTHESIS)
• Decision-use label (DEVELOPMENTAL, ASSOCIATION)
• Synthetic-data flag (pilot runs on synthetic data by default)
• Validation status (VALIDATION REQUIRED for unvalidated claims)
• Intervention target and follow-up window (for intervention records)

Evaluate operation without surveillance.

The system can work from telemetry and structural signals rather than requiring full prompt-content inspection. This is not a claim of absolute privacy — it is a design principle.

What the system cannot do.

• MO§ES™ cannot directly prove business productivity, ROI, employee quality, or causal outcomes unless those are specifically being measured and validated through experimental design.
• Outcome joins are labeled ASSOCIATION — never CAUSATION. A correlation between an intervention and a business metric is not proof that the intervention caused the business change.
• Diagnoses are hypotheses. They carry evidence, alternatives, and HYPOTHESIS status. They are never presented as established fact.
• Not all derived metrics are independently validated performance measures. Some are structural signals whose relationship to actual performance is still being tested.
• The system does not measure prompt content by default. Some bespoke evals may require enriched data (Level 2), but content-free telemetry is the default.
• Reference population comparison provides structural context, not a universal ranking of "better" or "worse" operators.
• Longitudinal measurement requires multiple windows. Single-window measurements are snapshots, not trajectories.

Canonical concepts, defined.

Each concept has a dedicated definition page with the formula, interpretation, how MO§ES™ uses it, and governance caveats.

The system is designed to be wrong.

If an intervention does not produce a measurable change in the target metric, the system says so. If a diagnosis cannot be distinguished from an alternative, the system says so. If an outcome join cannot be validated, the system says so.

This is a formal research framework, not established science. It carries papers, experiments, datasets, patents, and falsifiability work. It is presented with appropriate epistemic humility.