1. Discovery & feasibility
Define the problem, users, intended benefit, constraints and alternatives. Decide whether AI is warranted and what would justify stopping.
Evidence: Problem statement, scope, stakeholder map and feasibility decision.
Open each project, change inputs, run a small experiment, inspect the exact algorithm and trace the architecture from data to output. Designed for understanding how AI systems are assembled, tested and secured.
● Ten browser-side experiments available · No login or external execution
Educational demonstration only. No real model training, cloud instances, data uploads or remote terminal connections occur on this page. Never enter confidential or identifying records.
RELATED INTERDISCIPLINARY RESEARCH · 3 NOVEMBER 2026
Evidence Verification Across Sectors — research and applications →
Source provenance, historical records, claim verification and AI evaluation for education, RPL, teaching, healthcare, legal practice, accounting, audit and public administration. Each field retains its own standards and professional judgments.
Separate specialist study: criminal justice evidence evaluation →
Assess research contribution, human capability and the protection of people through separate evidence.
Use the lifecycle to connect purpose, requirements, design, implementation, testing and ongoing responsibility. These stages repeat as evidence changes; security, ethics, accessibility and QA run throughout.
Define the problem, users, intended benefit, constraints and alternatives. Decide whether AI is warranted and what would justify stopping.
Evidence: Problem statement, scope, stakeholder map and feasibility decision.
Specify functional behaviour, data needs, privacy, accessibility, performance and error limits. Agree measurable acceptance criteria and prohibited uses before building.
Evidence: Versioned requirements and traceable acceptance tests.
Map data flow, interfaces, trust boundaries, permissions and failure modes. Choose appropriate components and plan safe fallback, monitoring and rollback.
Evidence: Architecture diagram, threat model, data contracts and design review.
Build small increments, version changes and review dependencies. Keep data handling, computation and rendering separate; verify component interactions.
Evidence: Reviewed source, reproducible configuration and integration evidence.
Check implementation against requirements and suitability for the intended use. Test normal, edge, invalid and failure cases; investigate subgroup impacts and document limitations.
Evidence: Test results, reproducible failures, fixes, regression reports and QA findings.
Make an evidence-based release decision. Validate the deployment environment, access controls, recovery and support arrangements; begin with a bounded pilot.
Evidence: Release record, unresolved risks, operational runbook and rollback checks.
Monitor outcomes and drift, handle incidents, maintain dependencies and reassess impacts. Revalidate changes to code, models, data, prompts or context.
Evidence: Monitoring history, incident records and change approvals with retest evidence.
Plan replacement or withdrawal, hand over responsibilities, retire resources and apply justified retention/deletion. Preserve authorised evidence and lessons.
Evidence: Handover, retirement verification, archive decisions and final report.
Where this lab is now: the page contains runnable prototypes and a verification toolkit. A full live-page QA review remains to be completed. Trained-model evaluation, authenticated storage and production deployment are future extensions, not completed milestones.
AI work adds iterative data preparation, model evaluation, drift and impact review to the software lifecycle. Gantt plans schedule activities; review gates require evidence.
Suggested interview answers · select a question to expand · swipe through topics.
Follow an input through processing, results and inspection. These transparent baselines demonstrate the workflow; they do not use pretrained language or vision models.
Run a baseline, change one input, then compare the outputs. Build an evidence record before extending a prototype with a trained model.
Records remain in this page session. Export before refreshing or leaving. No server storage or model connection is added; image files themselves are not included.
Run any of the ten projects to capture its latest result.
Explain the hypothesis, baseline, single changed variable, observed differences and limitations. Exported records include the project, inputs, output, timestamp and a fingerprint of the executable demo scripts. These records support reproducibility; they do not prove model accuracy, fairness or production security.
The next stage is a versioned evaluation dataset with expected outcomes, then a model-backed comparison under the same tests. Production use would need authenticated private storage, audit trails and independently evaluated models.
Use synthetic fixtures to test the same algorithms as the ten experiments. Introduce a controlled fault in the test harness, observe the failures, then remove it and rerun. Live project algorithms remain unchanged by fault selection.
Ready: 20 checks across ten projects. No external systems are targeted.
Use the supplied synthetic cases, your own non-sensitive examples and authorised local tests. Challenge empty inputs, duplicates, thresholds, irrelevant retrieval, ambiguous categories and constant series. Define an expected result before running. Keep scope local: these exercises do not scan or attack other websites.
Controlled local faults, synthetic data and explicit expected outcomes.
Reproduce the failing case, identify its cause, change one control, then rerun the failure case and the full regression suite. The Apply fix button removes the simulated harness fault; it does not patch production code. A real fix requires a reviewed source change, regression evidence and a deployment/rollback plan.
Reproduce → isolate → correct → regress → document remaining limits.
Separate input handling, validated computation and safe output rendering. In production, add authenticated identities, server-side authorisation, isolated workloads, bounded resources, secret management, dependency review, safe logs and monitoring. Browser-only fixtures verify algorithm behaviour; they cannot verify backend isolation or access controls that are not present.
Define trust boundaries and verify controls where they actually run.
Define affected people, intended use, harms and review routes. Test subgroup outcomes and abstention where applicable, examine evidence quality and document uncertainty. A passing selection-rate test proves arithmetic on a fixture, not social fairness. Twenty passing cases provide bounded regression evidence, not assurance for unseen data.
Separate observed behaviour from broader claims about fairness and safety.
Use NIST AI RMF to structure Govern, Map, Measure and Manage activities. Use OWASP ASVS to select and verify relevant application-security requirements. Record the version, requirement, applicability, evidence, owner and unresolved gaps. These learning exercises are not certification or a completed standards assessment.
Map controls to requirements and collect evidence; do not assume compliance.
Freeze a labelled evaluation set; compare the baseline with one real classifier using identical cases. Record model/version, licence, data permissions, latency, errors, calibration and subgroup limitations. Then evaluate private storage and access controls before adding durable records. No trained model or persistent service is connected here yet.
Evaluation dataset → model comparison → private records → operational review.
Progress through these stages after the break–fix exercises. Each is planned work, with a concrete result to review before moving on.
Start here: expand the evaluation cases. More passing fixtures strengthen coverage; unseen-data performance and production assurance still require separate evidence.
Once a release is justified, move through controlled operation and continuing review. These are future stages; the current laboratory remains a browser demonstration.
Continuing cycle: observe → investigate → improve → revalidate. Higher stakes or wider deployment require stronger evidence and renewed review.
Tab through eight illustrative plans. Bars show parallel work; diamonds mark review gates. Timings are relative planning examples, not delivery estimates or evidence that work has been completed.
Scaling changes every illustrative duration and gate together. It does not recalculate resources, dependencies or a critical path.
Blue: planned activity · gold diamond: review gate · W: relative week. Gate outcomes may require rework, restriction or stopping.
Replace illustrative durations with estimates based on access, skills, resources and uncertainty. Identify dependencies, owners, acceptance criteria, review time and rework allowances. Recheck the plan when data access, model behaviour, requirements or deployment conditions change. A gate is a decision supported by evidence, not automatic permission to proceed.
AI projects can combine many methods, domains and deployment settings. Their milestones depend on the intended use, data, architecture and consequences of failure; the families below overlap rather than form a fixed list of 1,000 distinct types.
Swipe across project families; expand a card for milestones, architecture, challenges and review gates.
Milestones: Elicit rules → resolve conflicts → validate exceptions.
Architecture: Rule engine, knowledge base and traceable decisions.
Challenge: Incomplete knowledge, conflicting rules and maintenance drift.
Review gate: Review rule coverage, conflict handling and explanations with domain experts.
Incomplete knowledge, conflicting rules and maintenance drift.
Milestones: Define labels → split data → train → calibrate → evaluate.
Architecture: Versioned features, training pipeline and inference service.
Challenge: Label quality, leakage, imbalance and distribution shift.
Review gate: Evaluate held-out performance, relevant subgroup errors and abstention.
Label quality, leakage, imbalance and distribution shift.
Milestones: Define normality → choose distance → inspect clusters → validate alerts.
Architecture: Feature pipeline, statistical or learned baseline and review queue.
Challenge: No definitive labels; unstable clusters, false alarms and changing baselines.
Review gate: Test stability and usefulness with reviewers; do not equate unusual with harmful.
No definitive labels; unstable clusters, false alarms and changing baselines.
Milestones: Evaluate model → constrain context → test outputs → monitor use.
Architecture: Model endpoint or local inference, prompt templates and output controls.
Challenge: Unsupported claims, injection, privacy, misuse and evaluation ambiguity.
Review gate: Assess factual support, harmful outputs, abstention and human-review routes.
Unsupported claims, injection, privacy, misuse and evaluation ambiguity.
Milestones: Authorise sources → index → retrieve → generate → verify citations.
Architecture: Permission-aware retrieval, model inference and citation provenance.
Challenge: Stale sources, retrieval misses, cross-user leakage and unsupported citations.
Review gate: Test retrieval and generation separately, including access-denied and no-answer cases.
Stale sources, retrieval misses, cross-user leakage and unsupported citations.
Milestones: Collect representative images → annotate → train → validate conditions.
Architecture: Image decoding, preprocessing, model inference and output interpretation.
Challenge: Lighting, occlusion, demographic coverage, image privacy and compute costs.
Review gate: Test representative environments and shift; include uncertain or unsupported images.
Lighting, occlusion, demographic coverage, image privacy and compute costs.
Milestones: Capture audio → segment → transcribe/classify → validate speakers.
Architecture: Audio pipeline, speech model and timing or transcript outputs.
Challenge: Accent variation, noise, diarisation errors, consent and latency.
Review gate: Evaluate task-appropriate errors across speaking conditions and user groups.
Accent variation, noise, diarisation errors, consent and latency.
Milestones: Define objective → test relevance → pilot exposure → monitor feedback.
Architecture: Candidate retrieval, ranking model, experimentation and feedback logs.
Challenge: Feedback loops, exposure inequality, popularity bias and objective gaming.
Review gate: Balance relevance with user welfare and exposure; review online and offline evidence.
Feedback loops, exposure inequality, popularity bias and objective gaming.
Milestones: Order data → backtest → model seasonality → quantify uncertainty.
Architecture: Time-aware feature pipeline, rolling evaluation and forecast monitoring.
Challenge: Temporal leakage, regime change and overconfident long-range predictions.
Review gate: Use time-based validation and uncertainty checks; compare simple baselines.
Temporal leakage, regime change and overconfident long-range predictions.
Milestones: Define rewards → simulate → constrain actions → evaluate policy.
Architecture: Environment, agent policy, reward signals and action boundary.
Challenge: Reward gaming, unsafe exploration and simulator-to-world mismatch.
Review gate: Test constrained policies in safe environments before consequential real-world actions.
Reward gaming, unsafe exploration and simulator-to-world mismatch.
Milestones: Scope tools → enforce permissions → test plans → supervise actions.
Architecture: Planner, tool gateway, bounded memory, audit trail and approval controls.
Challenge: Excessive permissions, indirect injection, cascading errors and irreversible actions.
Review gate: Verify least privilege, stop conditions, action review and recovery under failure.
Excessive permissions, indirect injection, cascading errors and irreversible actions.
Milestones: Profile hardware → optimise → test device → update safely.
Architecture: On-device inference, constrained memory/power and signed update path.
Challenge: Latency, thermal limits, offline operation and physical safety constraints.
Review gate: Measure on target hardware and validate fallback, updates and failure behaviour.
Latency, thermal limits, offline operation and physical safety constraints.
Milestones: Agree participation → distribute updates → aggregate → evaluate leakage.
Architecture: Participant clients, aggregation coordinator and privacy controls.
Challenge: Non-identical data, poisoned updates, privacy leakage and participant dropout.
Review gate: Test aggregation resilience and privacy assumptions; decentralisation alone is not privacy.
Non-identical data, poisoned updates, privacy leakage and participant dropout.
Milestones: Specify assumptions → calibrate → validate → assess sensitivity.
Architecture: Model equations or learned components, provenance and reproducible analysis.
Challenge: Confounding, identifiability, uncertain parameters and invalid extrapolation.
Review gate: Check assumptions and independent evidence; predictive fit alone does not establish causality.
Confounding, identifiability, uncertain parameters and invalid extrapolation.
Milestones: Align modalities → integrate components → test interfaces → evaluate system.
Architecture: Combined text, image, audio, rules or retrieval components.
Challenge: Cross-modal errors, missing inputs, inconsistent meanings and complex dependencies.
Review gate: Test each component and interactions, including conflicting or unavailable modalities.
Cross-modal errors, missing inputs, inconsistent meanings and complex dependencies.
Revisit these considerations at discovery, evaluation, release, operation and closure. Evidence requirements grow with the consequences of failure.
Define the problem, affected people, intended benefit, alternatives and reasons to use AI. Prefer the least complex approach that meets the requirement; record out-of-scope uses and stop conditions.
Define the problem, affected people, intended benefit, alternatives and reasons to use AI. Prefer the least complex approach that meets the requirement; record out-of-scope uses and stop conditions.
Identify the owner, operator, reviewer, data steward and incident decision-maker. Specify how affected users can question outcomes and who can suspend, correct or retire the system.
Identify the owner, operator, reviewer, data steward and incident decision-maker. Specify how affected users can question outcomes and who can suspend, correct or retire the system.
Establish permission, origin, quality, representativeness and retention. Track source transformations and access boundaries. Assess whether labels, training material or derived outputs expose private or restricted information.
Establish permission, origin, quality, representativeness and retention. Track source transformations and access boundaries. Assess whether labels, training material or derived outputs expose private or restricted information.
Choose metrics from the task and harms, not convenience. Define acceptance criteria before testing; include leakage checks, edge cases, relevant subgroup analysis, uncertainty and independent validation.
Choose metrics from the task and harms, not convenience. Define acceptance criteria before testing; include leakage checks, edge cases, relevant subgroup analysis, uncertainty and independent validation.
Document trust boundaries, interfaces, versioned contracts, dependencies and failure modes. Test component interactions, graceful degradation and rollback rather than relying on a model score alone.
Document trust boundaries, interfaces, versioned contracts, dependencies and failure modes. Test component interactions, graceful degradation and rollback rather than relying on a model score alone.
Use threat modelling, least privilege, server-side authorisation where needed, input/resource limits, secret management and dependency controls. Test only authorised environments; keep a record of unresolved vulnerabilities and mitigations.
Use threat modelling, least privilege, server-side authorisation where needed, input/resource limits, secret management and dependency controls. Test only authorised environments; keep a record of unresolved vulnerabilities and mitigations.
Test whether people understand outputs, limitations and review routes. Consider accessibility, automation bias, workload, meaningful oversight and whether users can recover from mistakes.
Test whether people understand outputs, limitations and review routes. Consider accessibility, automation bias, workload, meaningful oversight and whether users can recover from mistakes.
Measure relevant compute, latency, storage, hardware and operational effort. Compare resource demand with evidenced benefit, including costs of maintenance, retraining and human review.
Measure relevant compute, latency, storage, hardware and operational effort. Compare resource demand with evidenced benefit, including costs of maintenance, retraining and human review.
Version models, data, prompts and configuration. Define monitoring, incident response, recovery and rollback. Revalidate meaningful changes and context shifts instead of treating initial approval as permanent.
Version models, data, prompts and configuration. Define monitoring, incident response, recovery and rollback. Revalidate meaningful changes and context shifts instead of treating initial approval as permanent.
Determine which requirements apply to the actual use, jurisdiction and sector. Map selected standard requirements to evidence and owners; keep unresolved gaps visible. Recheck applicability when purpose or deployment changes.
Determine which requirements apply to the actual use, jurisdiction and sector. Map selected standard requirements to evidence and owners; keep unresolved gaps visible. Recheck applicability when purpose or deployment changes.
Separate measured results, assumptions and hypotheses. Report methods, limitations, exclusions and uncertainty. Protect private material while preserving reproducibility and source attribution.
Separate measured results, assumptions and hypotheses. Report methods, limitations, exclusions and uncertainty. Protect private material while preserving reproducibility and source attribution.
Specify ownership after handover, record retention, deletion, appeal support and incident follow-up. End active inference deliberately; archive authorised evidence and verify that credentials, integrations and resources are retired.
Specify ownership after handover, record retention, deletion, appeal support and incident follow-up. End active inference deliberately; archive authorised evidence and verify that credentials, integrations and resources are retired.
Use the appropriate evidence: a research prototype, public recommender, clinical model and autonomous device require different acceptance criteria. A shared lifecycle provides structure; task-specific validation determines whether a particular claim is justified.
Expand each question to reveal a short answer, an architectural sequence and an invitation to explore the relevant technical or ethical trade-off. These are learning prompts, not universal solutions.
Each project now has a full, editable Python script in this repository, separate from the small browser demonstration. The scripts run with the Python standard library; they use synthetic data and provide real source-code walkthroughs for studying algorithms.
Launch a GitHub Codespace for this repository ↗ · Read run instructions
GitHub Codespaces is an external, account-based hosted development environment; it may be subject to access requirements and usage charges. This is not a private multi-user HPC cluster, nor does this public repository provide regulated-data storage.
Do not upload restricted datasets or API secrets to a public repository. A research production service needs separately managed identity, private storage, compute isolation and project-level permissions.
This page intentionally runs transparent, deterministic JavaScript algorithms so you can inspect every transformation. Deploying real notebooks, pretrained models, protected research data or GPU jobs requires a separately authenticated backend, secure provider connections and resource controls. A public GitHub Pages website cannot safely store API keys or execute private server-side jobs.
A possible evaluation use case for the laboratory's existing QA methods is a synthetic criminal-case document set. Test whether a system preserves the source and date of each assertion, flags inconsistent dates without inventing explanations, identifies absent records, and distinguishes a document summary from independently corroborated evidence. Reviewers should test false positives, omissions, wrongly merged people, hallucinated citations, version changes and access restrictions.
Suggested output: a claim-to-source matrix, contradiction and uncertainty log, reproducible test cases, and a record of human reviewer decisions. No actual criminal-case records are required for this proposed test. The laboratory does not claim to provide a validated legal investigation system, legal advice or autonomous evidential findings.
Evidence retrieval framework · Related AI feature proposals · Cross-disciplinary research context.
Use this review after the project and testing sections. QA also belongs throughout development: this is a focused check that the page works, communicates accurately and fulfils its learning purpose.
Complete one journey first, then broaden coverage. Scroll through the steps and expand only the detail you need.
Open one project, keep its example inputs and run it. Record the project, inputs and expected result; inspect the actual output.
Check for: A valid output and a clearly stated baseline expectation.
In Record, Compare & Explain, label the run Baseline, add your expectation and observation, then save the latest run.
Check for: A saved baseline with inputs, output and context.
Return to the same project and change one parameter or one data element. Predict the effect before running again; label and save the changed run.
Check for: A comparison in which the changed variable is identifiable.
Select the two saved runs of the same project. Inspect changed inputs and outputs; explain an unexpected difference or an unchanged result.
Check for: A reasoned observation, with uncertainties recorded.
Run the verification suite without a fault. Inject each controlled fault, inspect the failures, then select Apply fix & rerun. These fixes remove simulated harness faults rather than patching the source.
Check for: A recorded baseline, detected failure and successful regression rerun.
Export experiment records, verification evidence and the research report. Open each downloaded file and check that it contains the intended run, outputs and notes.
Check for: Readable exports containing the expected evidence.
Open each Gantt tab, change the timeline scale, inspect review gates and the schedule table, then download and open a chart.
Check for: Consistent chart/table timing and a usable SVG download.
In the QA realm, enter device/browser, set each reviewed check’s status and add expected-versus-actual evidence. Leave unreviewed checks as Not tested.
Check for: A truthful checklist with open issues and a provisional acceptance decision.
Repeat the project journey for all ten experiments, including invalid inputs and abstention cases. Check file handling for the document and image projects.
Check for: Coverage across every project, including error paths.
Test phone and desktop layouts, keyboard controls, focus, zoom, labels, horizontal scrolling and reduced-motion behaviour. Check that text stays readable and controls remain reachable.
Check for: Device-specific findings and accessible routes through the page.
Prioritise failures by impact. Record an owner and corrective action; reproduce the issue, fix it, then rerun the affected journey and regression checks.
Check for: A documented fix with retest evidence and remaining limitations.
Review whether the page meets its learning purpose. Once material defects are resolved, evaluate one trained model against a baseline on held-out examples.
Check for: A justified acceptance decision and a bounded next-stage plan.
Keep experiment and QA exports before leaving: current records are held only in the page session.
All checks start as Not tested. Statuses below are reviewer-entered observations, not automated verification or certification. This checklist does not execute security scans.
0 of 12 checks reviewed.
Keep failed and blocked checks visible. Fix defects, rerun affected tests and regression checks, then reassess acceptance. The checklist remains in this page session; export before leaving.
The next thing is a new discovery and scoping cycle. A completed live system moves into operations; a completed research project leaves a documented contribution and questions for future work.
Software changes follow review → automated checks → preview → acceptance → release → monitoring → correction. Data/model changes also require appropriate evaluation, versioning and impact review. Repeat the necessary loop without losing the project’s exit criteria. Pipeline quality gates are planned work here, not an established repository workflow.
A study may end in a report, an audit in findings and remedial actions, a model experiment in a reproducible evaluation, and a product in a supported release. Each needs a clear outcome, evidence, limitations, an owner for ongoing duties and an explicit closure decision. Some projects stop before release when evidence does not justify proceeding.
Current transition: finish live QA and establish repeatable quality gates, document acceptance and handover, then choose the next bounded model-backed experiment.
Twelve verification functions and four controlled evaluation conditions explore independent assessment of coordinated AI systems, with explicit attention to reliability, cost and unfamiliar tasks. Read the proposed research framework.
Additive research note, 4 November 2026. Conceptual and testable; not a validated ASI capability.
For a bounded next research task, see the research delivery roadmap and its 15 fictional claim-relation examples. The pack is an initial educational fixture for testing contradictions, temporal versions, evidence gaps and uncertainty, not a validated model benchmark.
Added 5 November 2026 · Existing lab content preserved.
For a concise successor handover that links the existing experiments, decisions, validation gates, cross-disciplinary connections and lifecycle safety risks without duplicating this milestone table, see Research Transfer, Scope & Completion. The original results and their limitations remain recorded below.
Research in brief. A sequence of offline Python/FastAPI experiments progressively separates software correctness, evidence-relation classification, structured reasoning and natural-language extraction. The most useful discovery is an architectural one: passing tests or matching familiar labels does not establish accurate interpretation of unfamiliar documents. Each result below is qualified by its test conditions; no real-world model reliability, autonomous intelligence or institutional adoption is claimed.
Execution priorities · Evaluation protocol · Initial synthetic examples · Original feature proposals
| Phase | Pinpointed observation | Lesson and architecture response |
|---|---|---|
| v0.1–0.3 · Foundation | Offline FastAPI scaffold and synthetic batch; v0.3 baseline abstained on 15/15 cases (1/15 matching editorial label). 16 tests passed. | Reproducibility and cautious abstention come before model complexity; report coverage alongside agreement. |
| v0.4 · Familiar cases | Evidence-aware lexical rules matched 14/15 in-sample labels; 26 tests passed. | Good developer-fixture performance is not evidence of transfer. Keep the earlier baseline and record remaining disagreements. |
| v0.5 · New wording | Frozen v0.4 rules matched 5/30 new editorial labels, abstaining on 28/30 (baseline 4/30). | Keyword dependence is brittle under paraphrase. Avoid tuning on the held-out material while still calling it unseen. |
| v0.6 · Typed propositions | Structured claim schema and controlled logic/API contract; 45 tests passed. Earlier results frozen. | Separate correctly representing a claim from reasoning over the representation. API correctness is not document understanding. |
| v0.7 · Extraction pipeline | Restricted sentence templates: 35/40 statements parsed; 4/20 pairs abstained; 89 tests passed. | Make extraction failures visible; do not manufacture semantic certainty from unsupported language. Contract agreement was in-sample. |
| v0.8 · Challenge cases | Frozen parser on 24 authored pairs: 36/42 annotated statements matched all typed fields; 12/18 adjudicated pair relations matched end-to-end; 6/6 ambiguous pairs abstained; 95 tests passed. | Paraphrases caused the six end-to-end misses; the reasoner matched 18/18 on hand-labelled typed inputs. Prioritise extraction evaluation and source-grounded provenance, not more agents. |
| v0.9 · Review readiness | 12 fresh fictional pairs prepared with two blank reviewer forms, agreement/adjudication tooling and 101 passing regression tests; 0/12 independently labelled so far. | Separate evaluation governance from model scoring. A reproducible annotation workflow is necessary, but it does not generate independent ground truth or establish improved accuracy. |
| v1.0 · Consistency and metamorphic checks | Frozen v0.7 extractor: 100/100 transformations met their stated behavioural invariants on 20 already-exposed fictional contract pairs; 104 automated tests passed. | Separate implementation invariants (symmetry, normalisation, safe refusals and human-review gating) from semantic accuracy. This is regression evidence, not new blind validation or evidence of general-language understanding. |
| v1.1 · Audit and adaptation gates | Read-only local audit checked five preserved SHA-256 baseline components, 100/100 exposed metamorphic invariants and 109 passing automated tests; independent reviewer labels remain 0/12. | File integrity and consistent implementation are necessary but insufficient: independent semantic evidence and production security require separate gates. Publish explicit pass/blocked criteria, not a single inflated readiness score. |
Procedure: from the separate offline Python v1.1 package, run python -m mg_lab.repro_audit_v11 --output RESULTS_REPRO_AUDIT_v11.json, followed by python -m pytest -q. The checker verifies five frozen component hashes, replays the v1.0 fixture invariants, checks independent reviewer-form completion and refuses to infer production release readiness. Recorded outcome: five frozen files preserved, 100/100 invariants passed, 109 tests passed, reviewer relation labels 0/12; independent accuracy and deployment gates remain blocked.
Architecture by goal: For repeatable research, preserve hashes, source versions, synthetic fixtures and observable traces. For reliable evidence interpretation, add blinded independent annotation and separate field extraction from typed reasoning. For audit-sensitive use, require source provenance, historical as-of logic, error disclosures, human sign-off and measured false certainty. For lower latency, compare resource usage at equivalent quality; simply adding agents is not proof of improvement. For a deployed API, require independently verified auth, rate limits, access control, logging, secret handling and rollback testing.
Failure response: a hash mismatch blocks claims of frozen-baseline reproduction; failed invariants trigger a software correction and repeat test; missing independent labels block new accuracy claims; absent security testing blocks external deployment. No synthetic or editorial label establishes real source truth. See the evaluation protocol and current execution handoff. The Python v1.1 archive is a separate local research deliverable, not a public hosted service.
Procedure: retain the v0.7 parser and original fixture; run python -m pytest -q and python -m mg_lab.metamorphic_v10 in the separate local Python v1.0 package. For each of the 20 exposed pairs, test swapping claim order, uppercasing, harmless outer whitespace, an unsupported report prefix and a malformed date prefix. Preserve observed and expected relations, an explicit abstention for unsupported syntax, and the mandatory human-review flag. Record 100 checks and their failure details; the observed implementation result was 100/100.
Architectural lesson: invariants can check code paths and gate conditions, but successful transformations of developer-exposed examples cannot measure extraction on unfamiliar evidence. If invariants break, correct canonicalisation, temporal parsing or approval control flow before expanding architecture. If they pass, proceed to genuinely independent, blind annotation before comparing extraction models. Source-truth verification requires additional provenance checks.
Re-use: the v1.0 local deliverable includes the executable module, archived v0.3–v0.9 materials, tests, machine-readable results and a milestone handoff. It has not been deployed or uploaded as a public source-code package.
Run local regression checks with python -m pytest -q; keep fixtures, component hashes, environment, expected labels, predicted labels and abstentions together. v0.3–0.5 compare authored synthetic relation labels. v0.6 checks typed contracts. v0.7 contract labels were constructed alongside the parser, so a match is not independent language-understanding accuracy. v0.8 was written after the parser freeze but was still assistant-authored rather than blind expert-labelled. Its 12/12 answered-only matches must be reported alongside 12/18 end-to-end coverage; do not disguise abstentions as correct relation predictions. The 18/18 typed comparison is a gold-input contract check, not a source-truth test.
Keep: versioned data and control paths, typed claims, source/as-of fields, conservative abstention, explicit human review, frozen baselines and reproducible trace outputs. Improve: paraphrase and entity normalisation, multiple propositions, negation, observation versus publication time, source reliability and independent evaluation labels. Avoid: automatic approval, mixing real confidential records into public fixtures, treating simulated scores as measured capability, and equating agreement with reference labels to factual truth. The optimum design depends on intended outcomes: an audit tool may privilege provenance and abstention, whereas a discovery assistant may also require measured coverage and calibrated uncertainty.
1 · Prepare: freeze versioned fictional text, hash inputs and withhold predictions/reference labels. 2 · Annotate: two actual independent reviewers label subject, predicate, value, scope, observation versus publication time, evidence status and relation; uncertain fields remain unknown. 3 · Reconcile: compute agreement on both-labelled cases and field-level completeness; preserve individual reviews and retain adjudication notes. 4 · Evaluate: lock adjudicated labels before scoring frozen extraction and reasoning components, reporting all-case coverage, answered-only agreement, abstention and unsafe assertions separately. 5 · Adapt: choose architecture changes based on demonstrated failure type, comparing outcomes against reliability, privacy and cost constraints. The prepared v0.9 local Python package contains twelve fictional unlabelled cases, reviewer CSV templates, and an agreement checker; reviewers have not submitted any annotations and no independent performance score is available.
Before altering the extractor, freeze the current code and evaluation definitions; create a fresh dataset whose labels and predictions are concealed from at least two independent reviewers; record field-level disagreements and an adjudication log. Test extraction and relation reasoning separately and end-to-end, reporting all-case agreement, answered-only agreement, abstentions, error types and costs. If independent annotation is unavailable, describe findings as exploratory engineering evidence only. Publish a concise results table plus a linked technical appendix, without changing historical findings.
Shareable fictional reviewer packet: 12 claim pairs, no labels (JSON) · Reviewer A blank CSV · Reviewer B blank CSV · Review protocol and coordinator safeguards.
Researcher sequence: distribute the same scenario-tag-free case text to each reviewer separately; obtain independent field and relation labels; privately retain signed-off originals, then calculate agreement and adjudicate disagreements before revealing any system outputs. Record unanswered items rather than manufacture consensus. This publication contains no model predictions, reference labels or completed reviewer answers.
Outcome-based adaptation: if disagreements cluster around temporal wording, change time representation and annotation rules; if they cluster around negation, test extraction logic; if scope or evidence is unspecified, prefer recorded abstention and review over confident resolution. Neither agreement between reviewers nor successful implementation establishes factual truth without external source verification.
Status: reviewer materials published; 0/12 independently reviewed so far. The set remains assistant-authored and requires external reviewer participation and independently sourced cases for stronger future inference.
Use this five-step researcher handoff with the milestone table above. Each change needs a failure mechanism, a reproducible comparison and a stated evidence gate; a larger model or more agents is not automatically progress.
Download the structured decision trace (JSON, five decisions and five failure responses). It links v0.3–v1.1 observations to design rationale and next tests; it does not replace the original code, artefacts or labels.
Observed: v0.4 matched 14/15 authored development labels, then the frozen rules matched 5/30 newly worded labels in v0.5, with 28 abstentions. Interpretation: lexical matching did not transfer reliably to changed phrasing under those authored synthetic conditions. Design response: keep claim extraction separate from relation comparison; log field-level errors. Next falsification: compare both extractors on newly and independently annotated material, including answer coverage and unsupported claims. Blocked inference: no general-world accuracy improvement is demonstrated.
Observed: v0.8 matched 18/18 editorial relation labels with supplied typed claims versus 12/18 end-to-end, with six misses associated with unsupported paraphrases. Interpretation: input representation, not only downstream logic, constrains the pipeline. Design response: diagnose extraction, comparison and evidence provenance independently; never treat gold structured inputs as an actual document extractor. Next falsification: independently label fields and relationships, then compare conditional and all-case performance at fixed abstention policy.
Evidence integrity: prioritise source/date/scope provenance, preserved records, human adjudication and abstention. Higher coverage: test an alternative extractor, count unsupported assertions and compare against the same frozen cases. Lower operating cost: benchmark runtime and resource use at equal measured error/coverage, not raw speed alone. Production use: require permissions, authentication, logging, privacy and security tests independent of synthetic NLP scores. If reference labels are missing, stop before an accuracy claim. If file hashes change, stop before a reproducibility claim. If deployment controls are untested, do not advertise production readiness.
Status: this is an evidence-to-decision synthesis of already documented local work, not a new algorithm benchmark or an independently validated finding. No independent annotations are yet recorded.
Research status: local synthetic implementations and recorded tests, not a deployed or independently certified service. Full code bundles and detailed v0.7/v0.8 reports were created in local development artefacts; this website summary does not imply those files have been uploaded to the public repository. Added 5 November 2026 as an additive milestone record.