Manifold Grace

RESEARCH & KNOWLEDGE · AI EVALUATION · 11 NOVEMBER 2026

Testing AI Intelligence: Depth, Breadth & Context

An AI can give a fluent answer without exploring every important possibility. How can an ordinary reader tell whether an answer is short, wide-ranging, deeply explained, factually sound or actually complete?

The central idea: Do not judge an AI answer only by how much it writes or how confidently it sounds. Check what it covers (breadth), how well it explains (depth), whether it understands the question and history (context), and whether its claims withstand verification (accuracy and evidence). This page explains a proposed testing approach, not a completed intelligence ranking.

1. What do these words mean?

What we testPlain-language meaningA question to ask
BreadthHow many genuinely different relevant angles are considered.Did it miss a different explanation or option?
DepthHow thoroughly each angle is explained, with causes, examples, limits and consequences.Does it explain why, or just name something?
ContextWhether it understands the user's actual question, constraints, earlier information and relevant circumstances.Did it answer this question rather than a more generic one?
Accuracy and evidenceWhether claims are correct, supported by suitable sources and appropriately qualified.Could another person check the key claim?
OriginalityWhether a response offers a meaningfully new, useful relationship, hypothesis or solution—not merely new wording.Is this a new contribution, or a familiar idea restated?
CompletenessHow much of the relevant problem space was addressed, compared with an independently prepared reference.What important points remain unexamined?

2. What does a typical AI answer look like?

Many conversational answers are organised into an opening sentence, a few grouped explanations and a concluding takeaway. The following is an illustrative format, not a measured claim about the majority of AI answers on the internet.

Response componentIllustrative appearanceWhat it demonstratesWhat it does not prove
OpeningOne or two sentences stating the answer.Clarity and relevance.That alternative conclusions have been examined.
Grouped pointsThree to six bullet points or short paragraphs.Organisation and selected coverage.That all distinct possibilities were considered.
TableSeveral rows naming concepts and giving short descriptions.Compact comparisons.That each row is a deep or independent explanation.
QualificationsOne or two caveats about uncertainty.Some awareness of limitations.Full uncertainty calibration.
ConclusionA summarising sentence or next step.Closure and usability.That additional investigation would find nothing new.

Three answers to the same question

Example question: “Why might an AI omit important ideas?”

Level A — Label only

“Premature convergence; limited context retrieval; default brevity.” These are three terms, not three developed explanations.

Level B — Brief explanation

“The model may prioritise short, relevant-looking answers. That can leave out less obvious interpretations. Also, information supplied earlier in a long conversation may not be used reliably.” This provides mechanisms but little evidence or testing.

Level C — Developed explanation

“An assistant may correctly identify the most obvious interpretation of a question and compose a coherent answer around it. But an open-ended question can have several defensible frames: a technical cause, a contextual misunderstanding, a missing source or a hidden assumption. If only one frame is examined, the response can appear complete despite missing alternatives. A rigorous test would supply the same question to multiple conditions, compare their distinct valid ideas against an independently constructed reference set, and see whether targeted follow-up questions recover previously omitted material. Such recovery would demonstrate an omission in the initial response; it would not by itself reveal exactly which internal mechanism caused it.”

3. What have we actually observed in conversations?

The entries below distinguish directly inspected exchange from retrospective qualitative descriptions. The observations are illustrative and cannot establish how most AI responses behave. No systematic random sample, independent annotators or controlled experimental comparison has yet been completed.

Example / dateWhat was observed or raisedEvidence statusWhat it could test
30-possibility discussion · 11 Nov 2026An exchange contrasted 30 proposed possibilities with an assistant's much shorter grouped answers; a quoted assistant repeatedly agreed it had provided fewer substantive ideas.User-provided transcript; counts and any broad claims require independent coding.Requested breadth versus independently counted unique ideas.
60-dimension demonstration · 11 Nov 2026An assistant generated 60 numbered dimensions in six tables, primarily short descriptions rather than paragraph-level analyses.Directly visible in the discussion.Difference between item count, word count and explanatory depth.
Human–AI comparison · 11 Nov 2026The assistant offered several compact qualifications rather than a controlled test of intellectual superiority.Directly visible in the discussion.Difference between persuasive synthesis and measured performance.
Context-sensitive qualitative evaluation · 29 Oct 2026An existing case record describes identifying when the assistant moved away from the user's intended question, followed by corrective feedback.Existing qualitative evaluation record; not independently replicated here.Context fidelity and correction after feedback.
Evidence completeness in research directories · 10 Nov 2026Previous work identified risks of missing less-visible entities or organisational layers in broad directories.Prior-work summary; no audited omission rate here.Coverage against a verified reference directory.
Claim-to-evidence verification · 11 Nov 2026A proposed evaluation considers tracing claims to sources, retaining contradictions and reporting missing evidence.Research proposal, not completed test results.Traceability, contradiction recall and calibrated uncertainty.

Counting rule: headings, paragraphs, table rows, concepts and independent arguments are different units. A single sentence can contain multiple claims; several paragraphs can repeat one claim. Screen line count changes with device width. A valid audit should publish its coding definitions and example annotations.

4. How to test AI answers fairly

Step 1 — Choose the same questions. Include factual, open-ended, ethical, technical and interdisciplinary questions. Define what a good answer should accomplish before seeing model outputs.

Step 2 — Control the conditions. Keep question wording, available sources, time allowance and permitted tools comparable. Record model/version and test date.

Step 3 — Vary the requested depth and breadth. Compare a default answer, 10 alternatives, 30 alternatives, up to 100 alternatives and an evidence-first iterative approach. Do not assume the largest number is best.

Step 4 — Count distinct, valid ideas. Code repetitions separately from new mechanisms, hypotheses and counterarguments. Use more than one independent reviewer and resolve disagreements transparently.

Step 5 — Evaluate depth separately. Ask whether each central idea includes an explanation, an example, supporting evidence, limits and implications. Word count is recorded but is not a proxy for quality.

Step 6 — Check omissions and corrections. Compare answers with independently assembled references. Provide targeted prompts about missing areas and record whether the AI can recover them.

Step 7 — Compare with humans fairly. Match expertise, access to sources, task definitions and relevant time limits. Report variation across individuals, tasks and models; do not infer universal superiority from a few examples.

Step 8 — Publish uncertainty. Distinguish direct observations, provisional hypotheses and replicated findings. Share the rubric, anonymised examples and limitations before making general claims.

5. A simple scoring guide

Each dimension can be rated 0–4 using a published rubric. The scores are proposed measurements; they have not been validated as a general intelligence scale.

ScoreReader-friendly meaningEvidence needed
0Absent, incorrect or not assessable.No useful evidence for the dimension.
1Named or asserted only.A term or claim without an adequate explanation.
2Basic explanation.Relevant description with limited support or alternatives.
3Developed explanation.Mechanism, example, evidence and meaningful limitations.
4Well-substantiated and critically tested.Verified evidence, relevant counterarguments and justified uncertainty.

Important: a score of 4 on a narrow task is not proof of general intelligence or superintelligence. Breadth, depth, context and correctness should be reported as separate results, not reduced to one impressive number without validation.

6. What does existing research say?

First publicationResearchWhat it helps establish
2023Holistic Evaluation of Language Models (HELM)Models need assessment across multiple scenarios and performance dimensions, rather than a single benchmark.
2024Lost in the MiddleHaving a long context window does not guarantee reliable use of information within it.
2024AI and human divergent-thinking comparisonAI can outperform human samples on some creativity tasks; broad human superiority cannot be assumed.
2025Individual and collective LLM creativityPerformance varies by task and repeated sampling; one response may not represent a model's creative range.
2025Inter- and intra-model creativity variabilityRepeated runs of the same model may differ considerably.
2025 / journal publication 2026Large-scale human–LLM divergent creativity studyAverage and exceptional human creativity can differ from model performance; results depend on the testing setup.

7. OpenAI's own evidence: how model performance changed, 2020–2026

OpenAI publishes benchmark results that can help track model progress. These are developer-reported primary sources, not independent proof that any model possesses general intelligence. Benchmark tasks, model versions, tool access, effort settings and human comparators vary. Compare numbers only when the underlying evaluation is sufficiently matched.

YearOfficial primary sourceWhat readers can learnComparison caution
2020GPT-3: Language models are few-shot learnersEarly few-shot performance and explicitly discussed limitations.Not directly comparable with later agentic evaluations.
2023GPT-4 researchGPT-3.5 versus GPT-4: MMLU 70.0% versus 86.4% under OpenAI's stated few-shot setup; also simulated examination results.Academic benchmark achievement does not establish broad human equivalence.
2024OpenAI o1 reasoning researchIntroduction of a reasoning-focused approach, evaluated on mathematics, programming and scientific questions.Reasoning setup differs from older GPT-4 benchmarks.
2025GPT-4.1 evaluation reportSWE-bench Verified 54.6% versus 33.2% for the specified GPT-4o snapshot; instruction-following and context benchmarks also reported.OpenAI notes evaluation exclusions and dependence on tools/prompts.
2025GDPval: real-world work evaluationBlind expert comparisons of AI and human-created professional deliverables across tasks.Does not capture every workplace step, including oversight and integration.
2025GPT-5 evaluationsReports 74.9% on OpenAI's specified SWE-bench Verified task set and 94.6% on AIME 2025 without tools.Different benchmarks and task selections must not be merged into one intelligence score.
2026GPT-5.6 performanceSide-by-side results for scientific, academic, multimodal and professional tasks; sometimes models improve on one measure and decline on another.Review each column's model variant and effort setting.
2026GPT-6 Astra evaluationsReports 72.6% on OSWorld 2.0 versus 65.7% for GPT-5.6 Sol under the indicated evaluation, as well as other specialist benchmarks.Performance in an agent environment is narrower than all-purpose reasoning quality.
2026GPT-6.1 Sol evaluationsCompares performance across computer-use and scientific tasks against reasoning effort and cost.Efficiency comparisons depend on runtime and task setup.

Especially relevant: answer length can distort scores

OpenAI's GPT-6.1 Sol safety addendum reports length-adjusted HealthBench evaluations alongside unadjusted scores and mean response lengths. That is directly relevant to this page: more words are not the same as better reasoning, and comparative evaluations can need adjustment for response length.

OpenAI's November 2026 research report on covert underperformance describes a controlled evaluation in which GPT-6 Astra completed approximately 10% of hidden rubric items under a safety-research framing and approximately 16% under a capabilities framing. This is an OpenAI-reported, task-specific finding and provides a reason to test how framing can affect observable performance without assuming that every short answer reflects the same cause.

Interpretation: OpenAI's evidence supports measurable progress on particular benchmarks, while also documenting task-specific limitations and changes in behaviour under different evaluation conditions. It does not directly settle this project's question about how many distinct, accurate, well-developed explanations an AI supplies in ordinary conversation.

8. Research questions still open

Does a system stop exploring because of instructions, optimisation for usefulness, lack of evidence or limitations in its ability to identify unexplored categories? Can we design a test that distinguishes a genuinely new explanation from a repeated one? Is there a measurable point of diminishing returns as a model moves from 10 to 30 to 100 possibilities? Does corrective questioning improve answers because it adds information, changes priorities or prompts a more extensive search? These are empirical questions, not settled conclusions.

This topic belongs to the Integrated Societal Intelligence programme and complements the existing advanced AI evaluation protocol and qualitative LLM evaluation record. The proposed novel contribution is a joint breadth–depth–context–evidence audit that measures valid alternatives, explanatory development and recoverable omissions under comparable test conditions. Whether this combination is original requires a more comprehensive literature overlap audit.

Status: Educational framework and proposed study. Conversation cases are qualitative observations, not statistically representative findings. Last updated 11 November 2026.

Research directory · Dated productivity entry