A decision workbench for AI
VIBES ARENOTEVIDENCE.
Evidence before action.
Compare AI workflows. Inspect uncertainty. Know when the evidence supports your next move.
Boldly go. Carefully extrapolate.
CLARK Criterion / AI workflow evaluation
Choose the workflow.
Know the evidence.
Model · prompt · tools · budget
The whole system is the candidate.
The bundled example uses deterministic synthetic workflows.
Evidence before action.
Ready to test the evidence.
Import complete workflow traces to inspect outcomes, costs and what remains unresolved.
Loss and uncertainty
Observed archive loss Audit boundsThe complete workflow comparison will appear here.
No replay audit work yet.
Observed results describe the imported cases. No statistical decision has been issued.
| Frozen workflow | Loss / bounds | Completion | Unsupported action | Provider charges | Eligibility |
|---|---|---|---|---|---|
| No trace bank loaded. | |||||
Inspect sources, workflow versions and file integrity
The dataset, manifests and response hashes remain inspectable.
Check the decision independently with OCaml
Evaluate evidence in this tab, then export a verification request. The native OCaml core recomputes conservative bounds and the decision using integer interval arithmetic. Exporting does not run the checker.
Build the checker from the research package and run it locally:
clark-criterion verify CLARK_Criterion_Verification_Request.json --checker verification/ocaml/_build/default/bin/criterion_check.exe --out verification.json
The first checker covers one archive regime without imported certificates. It checks numerical replay, not label truth, provider identity or future performance. A mismatch or unsupported request does not receive a successful verification.
Inspect the decision record
No statistical decision record yet.
Mission / intelligence assurance
One question.
A defensible next move.
Synthetic context. A model comparison needs a frozen question.
Choose the complete workflow
Which model earns the decision?
Compare frozen outputs on the same futures. Model names alone are not evidence.
Decision evidence
A model must earn its place.
Run the same declared experiment across the local candidate workflows or import frozen model responses.
The cost–risk frontier
Empirical stressed risk Worst-case upper boundActual comparison results will map this frontier.
Tournament costs come from an evaluator-owned pricebook frozen before proposals. Resource caps and a common budget apply to every candidate. Units are synthetic, not dollars. Points estimate worst-case risk over the declared mixture ball; their vertical intervals are joint anytime bounds. Empirical points are descriptive; eligibility and recommendations use the joint anytime bounds. A dotted line joins the estimated Pareto frontier. Simulator-conditional floating-point bounds are not formally validated numerical enclosures.
| Workflow | Nominal risk | Worst-case risk bounds | Relative cost | Status |
|---|---|---|---|---|
| No results collected. | ||||
Inspect provenance, prices and resource limits
The model version, prompt, tools, context hash and measured simulation work will remain inspectable.
The included ridge workflow is fitted locally. Imported LLM outputs are frozen responses, not live agents. This prototype does not establish operational validity or evaluate superintelligence.
Ask Bolt / experiment desk
Turn knowledge into a testable question.
A source-backed wiki proposes the contract. Mathematics tests its consequences.
Deterministic local drafting from the declared inputs. No LLM or Ask Bolt service is connected.
BOLT Wiki snapshot
Local exampleSource passages, rule interpretations, revisions and review status travel with the experiment.
Review or edit the local source snapshot
Edits are explicit local assertions. Check authority, units and the effective date before approving an interpretation.
Contract review
Knowledge needs a boundary.
The example includes a proposed source amendment. It is not silently promoted into a new operational fact.
Inspect the proposed experiment and source snapshot
Question → proposed experiment → reviewed contract → fresh simulation.
Optimize / intervention desk
Find the smallest useful change.
Predeclare the menu, test on fresh futures, and keep uncertainty visible.
Finite resource search
Compare adding zero, one or two units of the mentor and gate resources. The result concerns this declared menu.
A cheaper plan that remains possible prevents a minimum-cost claim.
Challenge the conditional laws
Attendance, retries and resource availability change within regimes. Each context gets fresh evidence; mixture stress alone cannot cover these changes.
Resource evidence
Test the intervention menu.
Minimum-cost advice is withheld while an unclassified cheaper option could still meet the requirement.
| Intervention | Relative cost | Nominal risk | Worst-case bounds | Acceptance |
|---|---|---|---|---|
| No intervention search yet. | ||||
Inspect the lab's exact scope
Synthetic sequential work; no deployed Optimize data or provider calls. A changed conditional law requires fresh evidence.
Outcome ledger / historical replay
Did reality agree?
Keep the forecast cutoff, censoring and measured outcomes together.
Reality check
The cutoff is part of the evidence.
Only records available at the cutoff belong to a historical prediction. Censored completions remain censored.
Inspect the complete replay report
No replay audited yet.
Local outcome ledger
Evidence stays in this tab until exported. Nothing is written to production.
Mathematics before mythology. The interface explores uncertain futures; the current evidence remains synthetic and conditional on its simulator.
CLARK Criterion / paired decision lab
Inspect the complete experiment.
Computed on your device.
No upload. No API key.
Current evidence
A decision needs evidence.
A recommendation must meet both the risk limit and regret tolerance.
Risk evidence, as it accumulates
Upper worst-case risk · limit − −A target is a declared completion threshold and day.
| Plan | Nominal estimate | Worst-case risk bounds | Constrained regret | Relative cost |
|---|---|---|---|---|
| Run the live evaluation to compare the plans. | ||||
Loss = target-failure probability + cost weight × declared relative cost. A plan must meet the risk ceiling before it can be recommended. Costs are illustrative inputs, not quoted prices. Estimates are descriptive under adaptive stopping; inference uses the anytime bounds.
Where the next samples go
What to do next
Start with balanced pilot observations across the four regimes.
The model is fitted before evaluation; its training worlds are counted.
Inspect a capacity impossibility proof
Run the evaluation to search for an exact resource-cut witness.
Inspect a sampled disagreement
No sampled disagreement yet.
Bring an AI proposal.
Freeze a model's priorities before the test. Compare its consequences on the same futures.
What this example does and does not establish
The confidence family covers the frozen plans and within-regime simulator laws. A 95% simulation statement does not establish real-world model adequacy. No language model or superintelligent system was tested; the built-in learned candidate is a fitted ridge priority model.
One mathematical question. Many pathways.
What is holding readiness back?
Sport pilot training
Aircraft, CFIs, solo practice, weather, repeated lessons and examiner access. The MOSAIC-aware roadmap separates training readiness from legal eligibility.
Clinical education
Placement capacity, preceptors and assessments. The present workday proxy needs a proper curriculum and competency adapter before it can predict medical graduation.
Aircrew qualification
Aircraft, instructors, simulators and ordered events. Compare the effect of one more shared resource across the whole cohort.
Detailed forecasts and recorded model studiesOpen research tools ↗
The working demonstration
A forecast you can inspect.
No data upload or telemetry.
Graduation, as a distribution.
Cohort completion curve
Mean Scenario P10–P90The shaded range represents variation across simulated futures, not a calibrated confidence interval.
Individual graduation outlook
Dates include shared resource competition and repeated events.
| Pilot | Target | P10 | Median | P90 | On target |
|---|---|---|---|---|---|
| Run the forecast to inspect each pilot. | |||||
“Beyond horizon” preserves right-censoring. It does not invent a graduation date.
Where time is lost
Mean counts per simulated future. Weather uses training days; other rows use events or pilot opportunities. Several constraints can affect one day.
What changes the outlook?
Compare one resource change at a time on the same simulated futures.
Choose with evidence
Which model earns the job?
Compare held-out completion-date predictions on the same cases. Accuracy, uncertainty, latency, and cost belong in the decision together.
Bring the results, keep the provenance.
Import a held-out model benchmark or raw predictions. This panel does not invent scores for commercial LLMs or infer quality from model size.
Model identifiers, case counts, evaluation split, and measurement units remain visible.
| Model | MAE, days ↓ | RMSE, days ↓ | Interval coverage | Latency, ms | Cases |
|---|
A model that predicts completion dates is different from a language model that proposes a schedule. Evaluate each complete workflow against its own held-out decision contract.
Policy decision evidenceFrozen research run
Frozen research run; changing live forecast controls does not recompute these certificates.
| Method | Robust interval width ↓ | Possible frontier size | Qualified versus FIFO |
|---|
Each contrast is policy A minus policy B. Lower loss favors A. A possible frontier includes policies that the recorded evidence has not ruled out; it is not a declaration that those policies are equivalent.
| A − B | Loss metric | Estimate | Nominal interval | Mixture-robust interval | Upper-envelope TV radius |
|---|
Inspect a mixture fragility witness
The TV radius tests the zero-loss improvement boundary, separately from the declared FIFO noninferiority margins. Mixture sensitivity changes the weights of declared disruption regimes while freezing their conditional simulator laws. These certificates do not account for every simulator error, prove real-world safety, or measure a superintelligent system.
CLARK Criterion / the mathematical engine
From a plausible answer
to a measurable decision.
CLARK Criterion: Proof-Assisted Paired Monte Carlo for Risk-Constrained Planning
CLARK Criterion is the mathematical evaluation engine inside the CLARK Frontier workbench. It samples disruption regimes adaptively, fits a frozen priority model, and can qualify a risk-acceptable plan or explicitly withhold a recommendation. The paper separates proved claims, synthetic experiments, and research still to be done.
The contribution is a specified composition of resource-infeasibility tests, paired anytime inference and constrained selection over a frozen menu. No general Monte Carlo speedup, optimal sampling rule or real-world calibration is established.
Inspect the mathematical contract
A shared integer target. Both screening and rollout label a miss when fewer than the same required number of learners finish. The target is fixed before evaluation.
Capacity impossibility. If every eligible target-sized group needs more weighted work than a sampled future supplies, no compliant scheduler can meet that target in that future.
∑j=1…m v(j)(y) > ∑r yrCr
Risk-constrained regret. Joint paired confidence sequences first qualify acceptable risk, then bound extra risk-plus-cost loss against every potentially acceptable plan, including permitted mixture shifts. The live evaluation can stop or return unresolved.
Evidence boundary. Proofs replace exact binary labels; they do not invent dates. Adaptive allocation has not established a general advantage over strong static allocation. Calculations use floating-point arithmetic, not formally validated interval arithmetic. No actual LLM or superintelligence has been tested.
The companion AI workflows paper describes the new evaluation system and its measured evidence. Inspect the provenance of each study; synthetic fixtures, recorded model traces and live runs are distinct.
From demonstration to deployment
Start with one cohort.
Use a sanitized course snapshot to inspect assumptions, identify bottlenecks, and replay decisions. An Optimize integration can then replace the example inputs with an approved syllabus and resource adapter.
Discuss an Optimize pilotMap the operational contract. Pin the syllabus, available records, resource rules, and decision cutoff.
Validate against completed cohorts. Measure date error, interval coverage, and the consequences of a wrong recommendation.
Evaluate the deployed workflow. Compare models and policies with their actual latency, cost, and constraints.