A · Foundation model
Direct model call without external tools; a minimum-complexity baseline.
Research protocol · Manifold Grace · 4 November 2026
Reliable agentic systems require an evaluation approach that measures factual outcomes and the procedures used to produce them. This proposed protocol compares evidence synthesis, causal reasoning, uncertainty, robustness, safety, autonomy, resource consumption and generalisation across agent architectures. It is a research specification rather than an implemented benchmark or certification.
Scientific status: Proposed experiment. No collected benchmark results are claimed. A high score on a specialised task, large-scale concurrency or autonomous execution does not demonstrate artificial superintelligence. Keep performance, generality, autonomy and deployment risk as distinct constructs.
| Dimension | Measures | Illustrative task | Release condition |
|---|---|---|---|
| Factual & evidential quality | Claim precision/recall, primary-source coverage, citation validity, temporal accuracy, disagreement retention | Evidence diagrams and institution records | No unsupported high-impact claim accepted |
| Causal reasoning | Causal graph correctness, confounding recognition, intervention checks, counterfactual coherence | Policy intervention and synthetic controls | No causal claim from correlation alone |
| Stochastic prediction | Brier score, calibration error, intervals, scenario coverage, model drift | Forecasts and event outcomes | Disclose calibration failures and distribution shift |
| Robustness & recovery | Perturbation sensitivity, outage recovery, retry/idempotency, dependency consistency | Missing sources, malformed documents, failed tool calls | No silent data loss or circular retry |
| Safety & authorisation | Unauthorised actions, approval-gate adherence, prompt-injection resilience, escalation quality | Read-only vs write tasks, untrusted materials | No unapproved external writes |
| Autonomy & controllability | Steps completed without intervention, human review count, abort reliability, reversibility | Long-horizon multi-stage casework | Escalation when authority or evidence is insufficient |
| Efficiency & economics | Total tokens, model/API calls, compute, monetary cost, time-to-result, useful work per budget | Single / parallel / hybrid topology comparisons | Cost and latency reported alongside quality |
| Cross-domain generalisation | Held-out domains, new task types, transfer gap, expert baselines | Law, science, historical archives, engineering | No ASI conclusion based on one specialised benchmark |
Direct model call without external tools; a minimum-complexity baseline.
Same foundation model with retrieval and authorised specialist tools.
Same base models and data access, parallel or hierarchical task coordination.
Configuration C plus independently specified provenance, verification and correction gates.
Each arm receives the same held-out case materials and comparable permissions. Use repeated trials, pre-registered metrics, blinded adjudication where possible, and a separately evaluated independent checker. Different outcomes should not be attributed to verification if a condition secretly received more data, a stronger base model or more test-time compute.
Preselect resource tiers (for example, one-quarter, one-half, one and two times an agreed reference budget). These are relative study conditions, not observed results. Track tokens, reasoning calls, retrieval requests, CPU/GPU time where available, cost, concurrency, peak load, wall-clock latency, retry count and energy estimates only when instrumented. Report quality–cost–latency trade-offs and the Pareto frontier rather than declaring that more agents are always better.
Each test item should retain its original source, literal and inferred claims, relevant date and definition, gold-standard evidence or qualified uncertainty, domain, acceptable interpretations and disagreement records. The automated judge must be evaluated against human reviewers, including inter-rater agreement and false-positive/failure analyses. Two model instances that consult the same erroneous source do not constitute independent corroboration.
Retain failed forecasts and superseded assessments with their original timestamps. Label unsupported claims as unsupported rather than false when the available evidence is incomplete. Preserve disagreement when credible evidence is inconclusive.
Define thresholds before running tests and adapt them by domain risk, rather than declaring universal pass scores after results are seen. Safety-critical tasks may require zero observed unauthorised writes in a predefined test suite, independently verified permissions, documented incident procedures and expert sign-off; even passing a finite test suite cannot prove zero future risk.
Use fail-closed controls for sensitive actions, explicit approval gates, audit trails and appropriate review rights. In legal, medical, financial or industrial contexts the system should provide bounded decision support, not unreviewable final authority. Keep safety approval separate from research-performance scores.
Study ID / version:
Question, hypothesis and preregistration:
Task set, sampling and excluded data:
Source dates and ground-truth process:
Architecture and model/tool versions:
Allowed permissions and safety boundaries:
Compute, latency and monetary budgets:
Human / automated assessor identities and independence:
Primary and secondary outcome measures:
Confidence intervals and calibration:
Per-domain findings and negative results:
Errors, reversals and intervention records:
Reproducibility artefacts and limitations:
Publication and change-log date:
If condition D outperforms matched baselines across held-out tasks, the evidence would support the narrower claim that this evaluation architecture improves the measured aspects of reliability or efficiency under stated conditions. It would not by itself prove human-level general intelligence, ASI, independent scientific discovery, safe real-world deployment or original authorship of every constituent technical method.
Related studies: Verifiable Advanced Intelligence · Agentic Architecture Patterns · AI Research Lab · Original feature-request register.
New research synthesis dated 4 November 2026. Earlier feature proposals remain preserved as separate dated records. No third-party figures reproduced.
A dated research note proposes evaluating claim-to-proof correspondence, correction propagation, independent review, verification costs and the distinction between mathematical correctness and scientific significance. View the research review log.
Logged 5 November 2026 · Exploratory · Follow-up investigation pending.
A course reading note compares cybernetic feedback and oscillation with modern agent retries, calibration, bounded correction and independent evaluation. The analogy requires technical and historical qualification. Read the dated course-reading crosswalk.
5 November 2026 · Source-based reading links and later research questions; no existing content replaced.