How to evaluate AI operators in enterprise teams.
A step-by-step guide to measuring how your team operates AI systems — from defining workflows to re-measuring after interventions. Seven steps, grounded in content-free telemetry, governed by DEVELOPMENTAL labels, and designed to be falsifiable at every stage.
The evaluation cycle.
Enterprise AI operator evaluation is a seven-step cycle: define workflows, collect telemetry, compute metrics, benchmark, diagnose, intervene, and re-measure. The cycle is iterative — each pass refines the evaluation and produces new hypotheses. No step produces a final verdict. Every output carries an evidence label and a decision-use label.
Define workflows, roles, and operating conditions.
Before any measurement begins, the enterprise defines what is being evaluated. This is not a technical step — it is an organizational one. The evaluation is only as good as the workflows it measures.
- Roles — who is being evaluated and in what capacity
- Workflows — what tasks and processes are in scope
- Models — which AI providers and models are used (ChatGPT, Claude, Codex, Copilot, Cursor)
- Systems — what tools and platforms are involved (21 MCP tools: 16 read + 5 write)
- Operating conditions — what constraints, environments, and contexts apply
- Desired behaviors — what good operation looks like for your workflows
- Constraints — what boundaries operators should stay within
- Operational goals — what the evaluation is trying to improve
- Risk boundaries — what the evaluation must not enable
- Performance questions — what you want to learn
Configurations are saveable as JSON and shareable across CLI, TUI, and MCP. This makes the definition step reproducible — the same evaluation can be run again with the same parameters, or modified for a different cohort.
Collect canonical token telemetry.
Once workflows are defined, the system collects telemetry from the AI providers in scope. The core data surface is four content-free signals — no prompt text required.
In the synthetic demo cohort, telemetry is collected from 5 AI providers across 50 operators, producing 1,668 observations over a 30-day window. Level 2 deployments add timestamps, sessions, model identifiers, acceptance signals, retries, errors, and richer provider fields. No prompt content is required for any core measurement.
Privacy boundary: cohort-level reporting is the default. Individual identity requires separate authorization. No adverse employment actions are permitted in pilot.
Compute the five canonical derived metrics.
From the four telemetry signals, the system computes five canonical derived metrics. Each captures a different aspect of the operator's token economy.
| Metric | Formula | What it captures |
|---|---|---|
| Leverage | (R + W) / I | Context reuse and construction efficiency relative to new input |
| Yield | O / (I + O + R + W) | Productive output share of total token flow |
| Token SNR | O / (I + O + R) | Signal quality — output relative to input and reused context |
| Log Leverage | log(1 + L) | Compressed leverage scale for ranking |
| Construction | W / R | Ratio of new context built to context reused |
I = Input, O = Output, R = Cache Read, W = Cache Write. All metrics are DERIVED — structural signals whose relationship to performance is still being tested. Labeled DEVELOPMENTAL.
The system also computes the AI Operator Development Index — a 0–100 composite score combining Leverage (30%), Yield (30%), Token SNR (20%), and Construction (20%). The index is a summary for ranking and tracking, not a substitute for reading the individual metrics.
Benchmark operators against the cohort.
Metrics in isolation are just numbers. Benchmarking places each operator in context — relative to peers in the same cohort, under the same workflow and model conditions.
Choose from 13 benchmark classes based on the evaluation question. Internal cohort, external field, longitudinal, intervention, role-specific, and more.
Position each operator in percentile bands: median, top 25%, top 10%, top 5%, top 1%, top 0.1%. Bands normalize across cohort sizes.
Every benchmark result carries provenance: benchmark class, reference cohort, percentile band, evidence label, and decision-use label.
Benchmarks should be contextual. A high-performing operator in one workflow may not be the strongest in another. "Best operator" should require the same qualifier as "best model." Best at what? The system never collapses benchmark position into a universal ranking.
Diagnose patterns and generate hypotheses.
Benchmarking reveals where operators stand. Diagnosis asks why. The system analyzes patterns across the cohort and generates hypotheses — never presented as established fact.
- Usage-operation divergence — where usage rank does not match evaluation rank
- Cluster archetypes — natural groupings by operating behavior
- Band stability — whether operators stay in their tier or move
- Concentration — whether advanced capability is organizational or isolated
- Movement — acceleration, stall, or reversion over time
- Hypotheses with evidence and alternatives
- HYPOTHESIS label — never presented as fact
- DEVELOPMENTAL label — routes workflows, not people
- Provenance on every diagnostic claim
- Validation status — VALIDATION REQUIRED for unvalidated claims
The system evaluates operators against 15 eval families — sets of evaluation criteria tailored to different workflow types. This ensures that diagnosis is contextual: an operator's patterns are interpreted against the expectations of their workflow, not against a universal standard.
Intervene with declared target metrics.
When diagnosis identifies a pattern worth addressing, the system designs an intervention. Every intervention declares its target metric and follow-up window before it begins.
In the synthetic demo cohort, 12 interventions were tested — from context window training to MCP tool access to workflow restructuring. Each targeted a specific metric: Leverage, Yield, Token SNR, Construction, Composite Score, or Divergence. The system reports both target and non-target metric deltas, so side effects are visible.
ASSOCIATION — never CAUSATION. A correlation between an intervention and a business metric is not proof that the intervention caused the business change. Outcome joins carry ASSOCIATION labels and governance metadata.
Re-measure and close the cycle.
After the follow-up window closes, the system re-evaluates the operator and compares pre- and post-intervention values. The cycle then repeats — new measurements, new benchmarks, new diagnoses, new interventions.
Did the intervention produce a measurable change in the metric it was designed to improve? If not, the system says so.
Did the intervention affect metrics it was not designed to touch? Side effects — positive or negative — are reported.
Based on the evidence, the intervention is retained, adjusted, or discarded. The decision is documented with provenance.
The cycle is iterative. Each pass produces new data, new hypotheses, and new interventions. Over multiple 30-day windows, the system tracks how operators move — whether they accelerate, stall, or revert. Longitudinal measurement requires multiple windows. Single-window measurements are snapshots, not trajectories.
Every step carries its labels.
Governance is not a final step — it is embedded in every stage of the cycle. The system enforces evidence labels and decision-use labels in code, not just in policy.
All results route workflows, not people. No adverse employment actions — hiring, firing, promotion, compensation, or performance review.
All diagnoses carry evidence, alternatives, and HYPOTHESIS status. Never presented as established fact.
All outcome joins are labeled ASSOCIATION — never CAUSATION. Internal and external deltas are kept in separate fields.
The system is designed to be wrong. If an intervention does not produce a measurable change in the target metric, the system says so. If a diagnosis cannot be distinguished from an alternative, the system says so. If an outcome join cannot be validated, the system says so. This is a formal research framework, not established science.
Go deeper.
The full eval framework — five questions, canonical telemetry, benchmarks, governance.
The four content-free signals that power all metrics.
The 12 interventions tested in the synthetic demo cohort.
Evidence labels, decision-use constraints, and privacy boundaries.
12 questions about AI operator evaluation, answered.
Results from the synthetic demo cohort of 50 operators.