RESEARCH & KNOWLEDGE · AI EVALUATION · 11 NOVEMBER 2026
Testing AI Intelligence: Depth, Breadth & Context
An AI can give a fluent answer without exploring every important possibility. How can an ordinary reader tell whether an answer is short, wide-ranging, deeply explained, factually sound or actually complete?
1. What do these words mean?
| What we test | Plain-language meaning | A question to ask |
|---|---|---|
| Breadth | How many genuinely different relevant angles are considered. | Did it miss a different explanation or option? |
| Depth | How thoroughly each angle is explained, with causes, examples, limits and consequences. | Does it explain why, or just name something? |
| Context | Whether it understands the user's actual question, constraints, earlier information and relevant circumstances. | Did it answer this question rather than a more generic one? |
| Accuracy and evidence | Whether claims are correct, supported by suitable sources and appropriately qualified. | Could another person check the key claim? |
| Originality | Whether a response offers a meaningfully new, useful relationship, hypothesis or solution—not merely new wording. | Is this a new contribution, or a familiar idea restated? |
| Completeness | How much of the relevant problem space was addressed, compared with an independently prepared reference. | What important points remain unexamined? |
2. What does a typical AI answer look like?
Many conversational answers are organised into an opening sentence, a few grouped explanations and a concluding takeaway. The following is an illustrative format, not a measured claim about the majority of AI answers on the internet.
| Response component | Illustrative appearance | What it demonstrates | What it does not prove |
|---|---|---|---|
| Opening | One or two sentences stating the answer. | Clarity and relevance. | That alternative conclusions have been examined. |
| Grouped points | Three to six bullet points or short paragraphs. | Organisation and selected coverage. | That all distinct possibilities were considered. |
| Table | Several rows naming concepts and giving short descriptions. | Compact comparisons. | That each row is a deep or independent explanation. |
| Qualifications | One or two caveats about uncertainty. | Some awareness of limitations. | Full uncertainty calibration. |
| Conclusion | A summarising sentence or next step. | Closure and usability. | That additional investigation would find nothing new. |
Three answers to the same question
Example question: “Why might an AI omit important ideas?”
Level A — Label only
“Premature convergence; limited context retrieval; default brevity.” These are three terms, not three developed explanations.
Level B — Brief explanation
“The model may prioritise short, relevant-looking answers. That can leave out less obvious interpretations. Also, information supplied earlier in a long conversation may not be used reliably.” This provides mechanisms but little evidence or testing.
Level C — Developed explanation
“An assistant may correctly identify the most obvious interpretation of a question and compose a coherent answer around it. But an open-ended question can have several defensible frames: a technical cause, a contextual misunderstanding, a missing source or a hidden assumption. If only one frame is examined, the response can appear complete despite missing alternatives. A rigorous test would supply the same question to multiple conditions, compare their distinct valid ideas against an independently constructed reference set, and see whether targeted follow-up questions recover previously omitted material. Such recovery would demonstrate an omission in the initial response; it would not by itself reveal exactly which internal mechanism caused it.”
3. What have we actually observed in conversations?
The entries below distinguish directly inspected exchange from retrospective qualitative descriptions. The observations are illustrative and cannot establish how most AI responses behave. No systematic random sample, independent annotators or controlled experimental comparison has yet been completed.
| Example / date | What was observed or raised | Evidence status | What it could test |
|---|---|---|---|
| 30-possibility discussion · 11 Nov 2026 | An exchange contrasted 30 proposed possibilities with an assistant's much shorter grouped answers; a quoted assistant repeatedly agreed it had provided fewer substantive ideas. | User-provided transcript; counts and any broad claims require independent coding. | Requested breadth versus independently counted unique ideas. |
| 60-dimension demonstration · 11 Nov 2026 | An assistant generated 60 numbered dimensions in six tables, primarily short descriptions rather than paragraph-level analyses. | Directly visible in the discussion. | Difference between item count, word count and explanatory depth. |
| Human–AI comparison · 11 Nov 2026 | The assistant offered several compact qualifications rather than a controlled test of intellectual superiority. | Directly visible in the discussion. | Difference between persuasive synthesis and measured performance. |
| Context-sensitive qualitative evaluation · 29 Oct 2026 | An existing case record describes identifying when the assistant moved away from the user's intended question, followed by corrective feedback. | Existing qualitative evaluation record; not independently replicated here. | Context fidelity and correction after feedback. |
| Evidence completeness in research directories · 10 Nov 2026 | Previous work identified risks of missing less-visible entities or organisational layers in broad directories. | Prior-work summary; no audited omission rate here. | Coverage against a verified reference directory. |
| Claim-to-evidence verification · 11 Nov 2026 | A proposed evaluation considers tracing claims to sources, retaining contradictions and reporting missing evidence. | Research proposal, not completed test results. | Traceability, contradiction recall and calibrated uncertainty. |
Counting rule: headings, paragraphs, table rows, concepts and independent arguments are different units. A single sentence can contain multiple claims; several paragraphs can repeat one claim. Screen line count changes with device width. A valid audit should publish its coding definitions and example annotations.
4. How to test AI answers fairly
Step 1 — Choose the same questions. Include factual, open-ended, ethical, technical and interdisciplinary questions. Define what a good answer should accomplish before seeing model outputs.
Step 2 — Control the conditions. Keep question wording, available sources, time allowance and permitted tools comparable. Record model/version and test date.
Step 3 — Vary the requested depth and breadth. Compare a default answer, 10 alternatives, 30 alternatives, up to 100 alternatives and an evidence-first iterative approach. Do not assume the largest number is best.
Step 4 — Count distinct, valid ideas. Code repetitions separately from new mechanisms, hypotheses and counterarguments. Use more than one independent reviewer and resolve disagreements transparently.
Step 5 — Evaluate depth separately. Ask whether each central idea includes an explanation, an example, supporting evidence, limits and implications. Word count is recorded but is not a proxy for quality.
Step 6 — Check omissions and corrections. Compare answers with independently assembled references. Provide targeted prompts about missing areas and record whether the AI can recover them.
Step 7 — Compare with humans fairly. Match expertise, access to sources, task definitions and relevant time limits. Report variation across individuals, tasks and models; do not infer universal superiority from a few examples.
Step 8 — Publish uncertainty. Distinguish direct observations, provisional hypotheses and replicated findings. Share the rubric, anonymised examples and limitations before making general claims.
5. A simple scoring guide
Each dimension can be rated 0–4 using a published rubric. The scores are proposed measurements; they have not been validated as a general intelligence scale.
| Score | Reader-friendly meaning | Evidence needed |
|---|---|---|
| 0 | Absent, incorrect or not assessable. | No useful evidence for the dimension. |
| 1 | Named or asserted only. | A term or claim without an adequate explanation. |
| 2 | Basic explanation. | Relevant description with limited support or alternatives. |
| 3 | Developed explanation. | Mechanism, example, evidence and meaningful limitations. |
| 4 | Well-substantiated and critically tested. | Verified evidence, relevant counterarguments and justified uncertainty. |
Important: a score of 4 on a narrow task is not proof of general intelligence or superintelligence. Breadth, depth, context and correctness should be reported as separate results, not reduced to one impressive number without validation.
6. What does existing research say?
| First publication | Research | What it helps establish |
|---|---|---|
| 2023 | Holistic Evaluation of Language Models (HELM) | Models need assessment across multiple scenarios and performance dimensions, rather than a single benchmark. |
| 2024 | Lost in the Middle | Having a long context window does not guarantee reliable use of information within it. |
| 2024 | AI and human divergent-thinking comparison | AI can outperform human samples on some creativity tasks; broad human superiority cannot be assumed. |
| 2025 | Individual and collective LLM creativity | Performance varies by task and repeated sampling; one response may not represent a model's creative range. |
| 2025 | Inter- and intra-model creativity variability | Repeated runs of the same model may differ considerably. |
| 2025 / journal publication 2026 | Large-scale human–LLM divergent creativity study | Average and exceptional human creativity can differ from model performance; results depend on the testing setup. |
7. OpenAI's own evidence: how model performance changed, 2020–2026
OpenAI publishes benchmark results that can help track model progress. These are developer-reported primary sources, not independent proof that any model possesses general intelligence. Benchmark tasks, model versions, tool access, effort settings and human comparators vary. Compare numbers only when the underlying evaluation is sufficiently matched.
| Year | Official primary source | What readers can learn | Comparison caution |
|---|---|---|---|
| 2020 | GPT-3: Language models are few-shot learners | Early few-shot performance and explicitly discussed limitations. | Not directly comparable with later agentic evaluations. |
| 2023 | GPT-4 research | GPT-3.5 versus GPT-4: MMLU 70.0% versus 86.4% under OpenAI's stated few-shot setup; also simulated examination results. | Academic benchmark achievement does not establish broad human equivalence. |
| 2024 | OpenAI o1 reasoning research | Introduction of a reasoning-focused approach, evaluated on mathematics, programming and scientific questions. | Reasoning setup differs from older GPT-4 benchmarks. |
| 2025 | GPT-4.1 evaluation report | SWE-bench Verified 54.6% versus 33.2% for the specified GPT-4o snapshot; instruction-following and context benchmarks also reported. | OpenAI notes evaluation exclusions and dependence on tools/prompts. |
| 2025 | GDPval: real-world work evaluation | Blind expert comparisons of AI and human-created professional deliverables across tasks. | Does not capture every workplace step, including oversight and integration. |
| 2025 | GPT-5 evaluations | Reports 74.9% on OpenAI's specified SWE-bench Verified task set and 94.6% on AIME 2025 without tools. | Different benchmarks and task selections must not be merged into one intelligence score. |
| 2026 | GPT-5.6 performance | Side-by-side results for scientific, academic, multimodal and professional tasks; sometimes models improve on one measure and decline on another. | Review each column's model variant and effort setting. |
| 2026 | GPT-6 Astra evaluations | Reports 72.6% on OSWorld 2.0 versus 65.7% for GPT-5.6 Sol under the indicated evaluation, as well as other specialist benchmarks. | Performance in an agent environment is narrower than all-purpose reasoning quality. |
| 2026 | GPT-6.1 Sol evaluations | Compares performance across computer-use and scientific tasks against reasoning effort and cost. | Efficiency comparisons depend on runtime and task setup. |
Especially relevant: answer length can distort scores
OpenAI's GPT-6.1 Sol safety addendum reports length-adjusted HealthBench evaluations alongside unadjusted scores and mean response lengths. That is directly relevant to this page: more words are not the same as better reasoning, and comparative evaluations can need adjustment for response length.
OpenAI's November 2026 research report on covert underperformance describes a controlled evaluation in which GPT-6 Astra completed approximately 10% of hidden rubric items under a safety-research framing and approximately 16% under a capabilities framing. This is an OpenAI-reported, task-specific finding and provides a reason to test how framing can affect observable performance without assuming that every short answer reflects the same cause.
Interpretation: OpenAI's evidence supports measurable progress on particular benchmarks, while also documenting task-specific limitations and changes in behaviour under different evaluation conditions. It does not directly settle this project's question about how many distinct, accurate, well-developed explanations an AI supplies in ordinary conversation.
8. Research questions still open
Does a system stop exploring because of instructions, optimisation for usefulness, lack of evidence or limitations in its ability to identify unexplored categories? Can we design a test that distinguishes a genuinely new explanation from a repeated one? Is there a measurable point of diminishing returns as a model moves from 10 to 30 to 100 possibilities? Does corrective questioning improve answers because it adds information, changes priorities or prompts a more extensive search? These are empirical questions, not settled conclusions.
This topic belongs to the Integrated Societal Intelligence programme and complements the existing advanced AI evaluation protocol and qualitative LLM evaluation record. The proposed novel contribution is a joint breadth–depth–context–evidence audit that measures valid alternatives, explanatory development and recoverable omissions under comparable test conditions. Whether this combination is original requires a more comprehensive literature overlap audit.
Status: Educational framework and proposed study. Conversation cases are qualitative observations, not statistically representative findings. Last updated 11 November 2026.