Manifold Grace

Research protocol · Manifold Grace · 4 November 2026

Beyond Answer Accuracy: Evaluating Verifiable Advanced AI

Reliable agentic systems require an evaluation approach that measures factual outcomes and the procedures used to produce them. This proposed protocol compares evidence synthesis, causal reasoning, uncertainty, robustness, safety, autonomy, resource consumption and generalisation across agent architectures. It is a research specification rather than an implemented benchmark or certification.

Scientific status: Proposed experiment. No collected benchmark results are claimed. A high score on a specialised task, large-scale concurrency or autonomous execution does not demonstrate artificial superintelligence. Keep performance, generality, autonomy and deployment risk as distinct constructs.

Eight evaluation dimensions

DimensionMeasuresIllustrative taskRelease condition
Factual & evidential qualityClaim precision/recall, primary-source coverage, citation validity, temporal accuracy, disagreement retentionEvidence diagrams and institution recordsNo unsupported high-impact claim accepted
Causal reasoningCausal graph correctness, confounding recognition, intervention checks, counterfactual coherencePolicy intervention and synthetic controlsNo causal claim from correlation alone
Stochastic predictionBrier score, calibration error, intervals, scenario coverage, model driftForecasts and event outcomesDisclose calibration failures and distribution shift
Robustness & recoveryPerturbation sensitivity, outage recovery, retry/idempotency, dependency consistencyMissing sources, malformed documents, failed tool callsNo silent data loss or circular retry
Safety & authorisationUnauthorised actions, approval-gate adherence, prompt-injection resilience, escalation qualityRead-only vs write tasks, untrusted materialsNo unapproved external writes
Autonomy & controllabilitySteps completed without intervention, human review count, abort reliability, reversibilityLong-horizon multi-stage caseworkEscalation when authority or evidence is insufficient
Efficiency & economicsTotal tokens, model/API calls, compute, monetary cost, time-to-result, useful work per budgetSingle / parallel / hybrid topology comparisonsCost and latency reported alongside quality
Cross-domain generalisationHeld-out domains, new task types, transfer gap, expert baselinesLaw, science, historical archives, engineeringNo ASI conclusion based on one specialised benchmark

Four matched experimental conditions

A · Foundation model

Direct model call without external tools; a minimum-complexity baseline.

B · Tool-enabled agent

Same foundation model with retrieval and authorised specialist tools.

C · Coordinated agents

Same base models and data access, parallel or hierarchical task coordination.

D · Coordinated agents + assurance

Configuration C plus independently specified provenance, verification and correction gates.

Each arm receives the same held-out case materials and comparable permissions. Use repeated trials, pre-registered metrics, blinded adjudication where possible, and a separately evaluated independent checker. Different outcomes should not be attributed to verification if a condition secretly received more data, a stronger base model or more test-time compute.

Performance as compute and cost change

Equal compute
Same total model/tool budget
Equal elapsed time
Same wall-clock deadline
Performance curves
Accuracy, risk and costs across budgets
Original conceptual illustration: resource-matched comparisons answer a different question from time-matched parallel-agent comparisons. Neither should be hidden in one composite leaderboard.

Preselect resource tiers (for example, one-quarter, one-half, one and two times an agreed reference budget). These are relative study conditions, not observed results. Track tokens, reasoning calls, retrieval requests, CPU/GPU time where available, cost, concurrency, peak load, wall-clock latency, retry count and energy estimates only when instrumented. Report quality–cost–latency trade-offs and the Pareto frontier rather than declaring that more agents are always better.

Robustness, safety and failure-injection suite

  1. Conflicting diagrams that use different definitions or silently omit the reference date.
  2. Source corruption, broken references, scanned pages, and incomplete provenance.
  3. Prompt injection embedded in retrieved documents or tool outputs; verify it is treated as data, not authority.
  4. Concurrent agents producing contradictory decisions from stale shared state.
  5. Duplicate webhook events, delayed responses, network partitions, tool timeouts and retry storms.
  6. Domain transfer to unseen legal, scientific, institutional and historical examples.
  7. Policy amendments between source publication and execution, requiring as-of-date reasoning.
  8. Requests exceeding delegated permissions, requiring human approval or refusal to act.
  9. Changes in model versions, evaluation rubrics and available context windows.

Independent adjudication and epistemic integrity

Each test item should retain its original source, literal and inferred claims, relevant date and definition, gold-standard evidence or qualified uncertainty, domain, acceptable interpretations and disagreement records. The automated judge must be evaluated against human reviewers, including inter-rater agreement and false-positive/failure analyses. Two model instances that consult the same erroneous source do not constitute independent corroboration.

Retain failed forecasts and superseded assessments with their original timestamps. Label unsupported claims as unsupported rather than false when the available evidence is incomplete. Preserve disagreement when credible evidence is inconclusive.

Acceptance thresholds and governance

Define thresholds before running tests and adapt them by domain risk, rather than declaring universal pass scores after results are seen. Safety-critical tasks may require zero observed unauthorised writes in a predefined test suite, independently verified permissions, documented incident procedures and expert sign-off; even passing a finite test suite cannot prove zero future risk.

Use fail-closed controls for sensitive actions, explicit approval gates, audit trails and appropriate review rights. In legal, medical, financial or industrial contexts the system should provide bounded decision support, not unreviewable final authority. Keep safety approval separate from research-performance scores.

Reproducible reporting template

Study ID / version:
Question, hypothesis and preregistration:
Task set, sampling and excluded data:
Source dates and ground-truth process:
Architecture and model/tool versions:
Allowed permissions and safety boundaries:
Compute, latency and monetary budgets:
Human / automated assessor identities and independence:
Primary and secondary outcome measures:
Confidence intervals and calibration:
Per-domain findings and negative results:
Errors, reversals and intervention records:
Reproducibility artefacts and limitations:
Publication and change-log date:

What a positive result would — and would not — establish

If condition D outperforms matched baselines across held-out tasks, the evidence would support the narrower claim that this evaluation architecture improves the measured aspects of reliability or efficiency under stated conditions. It would not by itself prove human-level general intelligence, ASI, independent scientific discovery, safe real-world deployment or original authorship of every constituent technical method.

Research foundations

Related studies: Verifiable Advanced Intelligence · Agentic Architecture Patterns · AI Research Lab · Original feature-request register.

New research synthesis dated 4 November 2026. Earlier feature proposals remain preserved as separate dated records. No third-party figures reproduced.

Case study reserved for investigation: AI-generated mathematics

A dated research note proposes evaluating claim-to-proof correspondence, correction propagation, independent review, verification costs and the distinction between mathematical correctness and scientific significance. View the research review log.

Logged 5 November 2026 · Exploratory · Follow-up investigation pending.