# CLARK Criterion: a research agenda for AI workflow decisions
**27 October 2026 · prospective research plan accompanying CLARK Frontier 0.6.1**

**CLARK Frontier** is the workbench. **CLARK Criterion** is its mathematical evaluation engine. **Evidence before action.** is the product tagline. **Boldly go. Carefully extrapolate.** remains the brand personality.

## The recommendation

Make Criterion an engine that decides **which evidence to acquire next, and when that evidence supports choosing an AI workflow**. A workflow includes the exact model version, prompt, retrieval, tools, reasoning budget and escalation policy. Frontier lets a user declare the decision, compare these configurations and inspect what supports the result.

The strongest first research question is:

> Can checked partial proofs and paired workflow trials reduce the total cost of selecting a reliable AI workflow, while preserving a stated error guarantee and abstaining when the evidence is insufficient?

This is a candidate contribution, not an established novelty claim. A focused search through primary papers available on 27 October 2026 found close prior work. Generic model routing, LLM judges, adaptive evaluation, human auditing, anytime stopping and constrained selection are already developed research areas. A new name or their combination does not establish a new mathematical result.

The current release implements the approved naming and documents this research direction. It does **not** implement the proposed outer AI evaluation runner, claim a new cost-optimal algorithm, or report actual provider-model experiments. The existing synthetic studies and their limitations remain intact.

## What could make the work meaningfully new

### 1. Evidence that serves several decisions

An independent checker could accept a witness that restricts several workflow outcomes at once. For example, a validated resource impossibility witness could settle failure for multiple proposed schedules under the same declared simulator contract. A shared source contradiction could establish that several recommendations lack a required citation, if the outcome definition explicitly tests that property.

The evidence is a sound set of possible joint losses, not merely a confidence score from an LLM. One witness might settle a pairwise comparison without settling either workflow's absolute failure rate; the risk gate still needs its own evidence.

The research problem is when these shared, outcome-dependent restrictions change the **cost of identification**, including proof search and checking costs. Ordinary shared observations are already studied in feedback-graph pure exploration [7]. The new work must demonstrate a distinction and benefit beyond that setting.

### 2. Learn where to spend evaluation effort

Use AI to predict which unresolved comparison, verifier attempt or audit could change the decision. It can also propose candidate workflows and search for structured witnesses. Keep those learning roles explicit and separately testable.

For each acquisition opportunity, the evaluator could choose a bounded proof attempt, an executable workflow trial, a cheap judge query, an authoritative reference measurement, or further simulation. It would account for unsuccessful proof attempts, tokens, tool execution, review time and orchestration. It should sometimes skip the judge when a direct check is cheaper.

The resulting allocation rule begins as a heuristic. Calling it optimal requires a new proof and assumptions. Recent work already studies cost-aware AI scoring and human escalation [14], and LLM-judge best-arm identification [4,15].

### 3. Select feasible workflows, with honest unresolved states

The output should identify a workflow that meets a declared failure ceiling and is sufficiently good relative to eligible alternatives—or return unresolved. Deployment eligibility, such as local-only execution, should filter the candidate set before comparison.

Measure unsupported-action rate and successful completion separately. Otherwise an assistant that always abstains could appear safest while being useless. Evidence must name its workload, model and tool versions, reference evaluator, assumptions and date. A model or prompt update changes the evaluated object; old evidence should not automatically carry over.

This decision contract is useful product design. Its ingredients overlap constrained best-arm identification and sequential constrained confidence sets [6,8]. The new contribution needs either a sharper acquisition/computational result or a reproducible empirical finding.

## Primary research that sets the bar

The researchers below are relevant because their work directly bears on this problem, not because their involvement or endorsement is implied. Recent preprints are identified as such; this is a focused review, not a complete priority search.

| Work and research group | Already established | Consequence for CLARK |
|---|---|---|
| **RouteLLM**, Ong et al., including Joseph Gonzalez and Ion Stoica; 2024/revised 2025 [1] | Learned cost-quality routing between models using preference data. | A model router alone is weak novelty. |
| **SELECT-LLM**, Durmazkeser et al., including Torsten Hoefler and Nezihe Merve Gürel; canonical revision September 2026 [2] | Selects informative queries to identify good models with fewer annotations. | Compare against active model selection, not just exhaustive evaluation. |
| **Scaling Up Active Testing to LLMs**, Berrada et al., including Yarin Gal and Tom Rainforth; NeurIPS 2025 [3] | Cheap frozen surrogate models can guide economical evaluation. | Charge acquisition overhead and include a strong cheap-surrogate baseline. |
| **Best Arm Identification with LLM Judges and Limited Human Audits**, Ao et al., including David Simchi-Levi; January 2026 preprint [4] | Biased judges, propensity-corrected human audits, anytime confidence sequences and adaptive allocation. | The basic judge/audit/sequential combination is already close prior art. |
| **Optimal Multi-Fidelity Best-Arm Identification**, Poiani et al., including Rémy Degenne and Emilie Kaufmann; 2024/revised 2025 [5] | Instance-dependent cost bounds and asymptotically optimal multi-fidelity allocation. | Multiple evidence prices and fidelities alone do not establish novelty. |
| **Constrained Best Arm Identification with Tests for Feasibility**, Cai and Kandasamy; AAAI 2026 [6] | Separately allocates performance and feasibility tests. | A hard risk constraint alone is insufficient. |
| **Pure Exploration with Feedback Graphs**, Russo, Song and Pacchiano; 2025 [7] | Shared feedback, identification bounds and an asymptotically optimal allocation method. | Compare shared certificates against known side-observation methods. |
| **Sequential Confidence Sets for Coverage-Constrained Conformal Model Selection**, Li and Zhu; September 2026 preprint [8] | Certified/possible feasibility and anytime constrained selection. | Abstention and feasibility projection are necessary but occupied territory. |
| **Agent Evaluation Reliability**, Hardy et al., including Reuel, Kochenderfer and Koyejo; September 2026 preprint [9] | Distinguishes claims about fixed model–scaffold systems from claims about underlying models. | Evaluate complete frozen workflows and state what the result means. |
| **AI Agents That Matter**, Kapoor et al., including Arvind Narayanan; 2024 [10] | Cost, application suitability, proper holdouts and reproducibility matter alongside accuracy. | Publish total costs, traces and realistic outcomes. |
| **Test-Time Compute Scaling**, Snell, Lee, Xu and Kumar; 2024 [11] | Adaptive inference effort can beat fixed computation allocation. | A stronger-model comparison must control reasoning budget. |
| **Active Statistical Inference**, Zrnic and Candès; 2024 [12], and **Prediction-Powered E-Values**, Csillag, Struchiner and Goedert; 2025 [13] | ML-assisted evidence acquisition and valid sequential inference with prediction-powered factors. | Statistical correction is a foundation to build on, not a new invention. |

Two particularly close late-2026 sources require attention: SCALE [14] already studies selective AI scoring and sequential human escalation with cost optimality; Jedra, Al Marjani and Benomar's October 2026 OpenReview listing [15] is titled *Near Cost-Optimal Best-Arm Identification with LLM Judges*. The latter's listing was verified but its full text was unavailable in this review. A full-text comparison is required before claiming priority.

LLMRouterBench [16] reports that several complex routers do not reliably outperform simple baselines. That reinforces the need to compare against the strongest static allocation and publish a null result if it wins. SELECT-LLM's duplicate arXiv entry 2605.24981 was withdrawn as a duplicate; use canonical 2510.09418 [2].

## A precise mathematical target

Freeze a finite eligible workflow set. For a fresh case, let the workflow losses form a bounded vector. A sound checker returns a set containing that true vector. Its information may determine exact labels, ranges or pairwise contrasts.

For unresolved quantities, allow cheap predictions followed by independently randomized audits with logged positive probabilities. Predictions can be wrong. The correction must remain valid under those errors and under the actual adaptive sampling scheme. Human labels need an explicit adjudication or noise model; “human” does not automatically mean ground truth.

A basic validity construction is available from established prediction-powered e-process methods [13]. Choose a valid base factor before seeing the current case. After cheap information arrives, use a surrogate factor plus an inverse-propensity correction on audited cases, with a positivity constraint. Exact certificates can replace factors they determine. A union bound across declared targets and regimes can support simultaneous decision bounds.

That construction is useful but is **not the proposed main novelty**. It also cannot be dropped into the current Bernoulli beta-mixture implementation: corrected scores/factors need a separate inference primitive, numerical tests and proof review.

The ambitious theorem target is a tractable allocation method for a structured class of shared certificates that approaches an instance-dependent information-per-cost bound while preserving constrained identification validity. A generic information-theoretic lower bound is already classical. The hard work is representing outcome-dependent joint constraints, choosing useful proof budgets, handling random costs, and proving computational and statistical performance relative to the closest methods.

Start with a unique eligible optimum separated from risk and comparison boundaries. Near ties, set-valued near-optimal decisions and exact boundary cases require their own treatment. Do not promise finite stopping at a binding feasibility boundary or efficiency for arbitrary proof systems.

The companion technical note, `AI_MATHEMATICAL_RESEARCH_NOTES.md`, states a modest validity proposition and an explicitly unproved cost-allocation target. Neither is a newly established main theorem of the released manuscript.

## The first experiment to run

**Study:** choose a source-grounded planning assistant under reliability and cost constraints.

Keep the first task inside the current planning environment, where an executable outcome evaluator already exists. Give each agent a planning question, frozen source packet and residual-work snapshot. It must identify applicable source versions, compile a typed contract or request missing evidence, propose an allowed intervention, invoke Criterion and return a supported recommendation or an appropriate unresolved state.

Use four frozen AI workflows: two eligible model backends crossed with two source-processing approaches—a reasonable raw-source retrieval baseline and the source-linked typed-contract pipeline. Retain the deterministic planner as a non-LLM reference. A reviewed gold-contract condition is a privileged diagnostic, clearly labeled, not an equal-information competitor.

An initial feasibility design is 200 distinct case families: 40 development, 40 calibration and 120 final holdout, with three stochastic attempts per workflow. This is a planning example, not a power calculation or an operational validation claim. It implies 2,400 bounded AI episodes before extra baselines, potentially many more model calls. First use a small development pilot to measure costs and variability; then freeze the final design and budget.

Keep all revisions and variants of a source family in one split. Include superseded rules, future-effective amendments, contradictory sources, missing information, infeasible requests and cases where abstention is correct. Ground truth must be independently reviewed rather than secretly generated by the tested system.

**The outer sampling unit is a new case.** Repeated attempts are nested within cases. Inner simulator worlds answer questions about one plan; they are not independent observations of the model's ability across tasks. Pair cases and exogenous environments across workflows. Identical provider seeds do not establish common random numbers for LLM generation. Treat unresolved inner decisions as unresolved, or derive a valid nested uncertainty procedure.

Primary endpoint: total evaluation cost required to make a valid constrained workflow selection. Secondary outcomes: false selection, unsupported-action rate, supported completion, abstention, unresolved-at-budget probability, deployment execution cost and latency. Report intervals and the empirical precision of rare-event estimates. The proposed 120-case holdout alone cannot substantiate a strong rare-failure claim.

Hold candidate workflows and outcome checks fixed while comparing acquisition policies: full/uniform evaluation; strong static allocation; sequential confidence-bound selection; active testing/SELECT-LLM where their assumptions fit; audited-judge best-arm selection; and the proposed certificate-aware procedure. Use common precomputed traces to control comparisons initially, and then a smaller prospective execution study to measure real cost and latency. Charge trace generation to methods according to what they actually require; do not give expensive responses away for free.

Ablate exact screening, shared witnesses, pairing, audited correction and adaptive acquisition individually. Include deliberately biased judges, expensive or useless proof search, no certifiable cases, rare adverse events, near ties and no feasible candidate. The proposed benefit should disappear where its mechanism has no useful information. There is no paper-worthy speedup claim if the strongest static method matches it.

## The proposed AI architecture

Use an existing execution harness such as **Inspect** [17] for provider adapters, local models, tools, sandboxes and traces. Build a thin development adapter around the current Python engine before adding production integration.

| Layer | Proposed responsibility | Current status |
|---|---|---|
| Frontier | Declare task, freeze candidate workflows, review sources, inspect decisions and costs | Workbench implemented; real-workflow evaluation screens remain future |
| AI execution runner | Execute bounded model/tool episodes and record traces and usage | To build; no provider calls added in 0.6.1 |
| Criterion | Check typed evidence, maintain validity, choose the next evidence action and decide/abstain | Current synthetic plan engine exists; outer evaluation and new audit inference remain future |
| Ask Bolt / Optimize development | Consume or invoke the bounded experiment through an adapter | Existing development interface; production disconnected |

The current mission tools are `get_snapshot`, `compile_contract`, `evaluate_plans`, `search_interventions`, `explain_evidence` and `replay_cohorts`. Protocol compatibility with an external harness requires an integration test. The current source compiler is deterministic; it is not evidence of successful LLM source interpretation.

Each run should preserve a workflow manifest, episode trace, outcome record, cost record and decision record. Pin model identifiers, prompts, source snapshots, tools, evaluator version and code revision. For local models include weight/tokenizer hashes, quantization and serving configuration. Provider/gateway receipts improve provenance but do not prove proprietary model weights.

Measure provider charges and cache usage, local GPU/device time, tool time, proof-search costs and review effort. Separate the cost of obtaining evidence from the cost of deploying the selected workflow. Unknown costs remain unknown rather than zero. For enclave use, first restrict the menu to approved local candidates and run the same harness inside that environment.

A learned controller that chooses between retrieval, stronger models, further reasoning and human escalation is a promising second study. It needs independent episode-level guarantees and comparisons against routing, cascades and budget-matched inference. A failed-feed troubleshooting task is a later second domain once an executable incident environment and independent adjudication exist; existing scheduling proofs do not certify diagnosis.

## Publication standard and sequence

1. Freeze the task, workflow and outcome contracts; implement trace and cost records.
2. Build the development runner and independent verifiers; run a small real-model feasibility pilot.
3. Implement the separate audited-evidence inference primitive and review its proof and numerics.
4. Preregister candidate workflows, baselines, error budgets, cost definitions, splits and stopping rules.
5. Run untouched confirmatory cases, publish failures and null results, and release reproducible permitted traces.
6. Claim a theory contribution only after the allocation theorem and priority comparison hold up.

A suitable working title is **“Certificate-Aware Active Evaluation of AI Workflows under Reliability Constraints.”** It describes a research target, not a completed second paper. The existing manuscript remains **“CLARK Criterion: Proof-Assisted Paired Monte Carlo for Risk-Constrained Planning.”**

A credible systems/evaluation paper could come from a rigorous, useful real-model benchmark and demonstrated cost benefit. A strong theory paper additionally needs a nontrivial result beyond composition. Acceptance, priority and a major breakthrough cannot be promised from the present evidence.

## Primary references

1. Ong et al. *RouteLLM: Learning to Route LLMs with Preference Data*. 2024; revised 2025. https://arxiv.org/abs/2406.18665
2. Durmazkeser et al. *Large Language Model Selection with Limited Annotations*. 2025; revised 28 September 2026. https://arxiv.org/abs/2510.09418
3. Berrada et al. *Scaling Up Active Testing to Large Language Models*. NeurIPS 2025. https://arxiv.org/abs/2508.09093
4. Ao et al. *Best Arm Identification with LLM Judges and Limited Human Audits* (full-text title; abstract record truncates the final word). 2026 preprint. https://arxiv.org/html/2601.21471v1
5. Poiani et al. *Optimal Multi-Fidelity Best-Arm Identification*. 2024; revised 2025. https://arxiv.org/abs/2406.03033
6. Cai and Kandasamy. *Constrained Best Arm Identification with Tests for Feasibility*. AAAI 2026. https://arxiv.org/abs/2511.09808
7. Russo, Song and Pacchiano. *Pure Exploration with Feedback Graphs*. 2025. https://arxiv.org/abs/2503.07824
8. Li and Zhu. *Sequential Confidence Sets for Coverage-Constrained Conformal Model Selection*. 2026 preprint. https://arxiv.org/abs/2609.28522
9. Hardy, Azam, Reuel, Kochenderfer and Koyejo. *Agent Evaluation Reliability: More Tasks Won't (Always) Fix An Agent Leaderboard*. 2026 preprint. https://arxiv.org/abs/2610.00651
10. Kapoor et al. *AI Agents That Matter*. 2024. https://arxiv.org/abs/2407.01502
11. Snell, Lee, Xu and Kumar. *Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters*. 2024. https://arxiv.org/abs/2408.03314
12. Zrnic and Candès. *Active Statistical Inference*. 2024. https://arxiv.org/abs/2403.03208
13. Csillag, Struchiner and Goedert. *Prediction-Powered E-Values*. 2025. https://arxiv.org/abs/2502.04294
14. Ham, Zhao, Jasin and Yang. *Human–AI-Powered Hypothesis Testing: Cost-Aware Selective AI Scoring and Sequential Human Escalation*. 2026 preprint. https://arxiv.org/html/2609.28859v1
15. Jedra, Al Marjani and Benomar. *Near Cost-Optimal Best-Arm Identification with LLM Judges*. OpenReview listing, 3 October 2026; full text not inspected in this review. https://openreview.net/forum?id=8Iz82S2UDW
16. Li et al. *LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routing*. 2026 preprint. https://arxiv.org/abs/2601.07206
17. UK AI Security Institute / Meridian Labs. *Inspect* official documentation, accessed 27 October 2026. https://inspect.aisi.org.uk/