Manifold Grace
Project explanations with grey annotated pseudocode · Model evaluation · Python API example
RESEARCH / INTERACTIVE LEARNING ENVIRONMENT

AI Research Laboratory

Open each project, change inputs, run a small experiment, inspect the exact algorithm and trace the architecture from data to output. Designed for understanding how AI systems are assembled, tested and secured.

● Ten browser-side experiments available · No login or external execution

Educational demonstration only. No real model training, cloud instances, data uploads or remote terminal connections occur on this page. Never enter confidential or identifying records.

SDLC · SOFTWARE DEVELOPMENT LIFE CYCLE

Lifecycle considerations before you build

Use the lifecycle to connect purpose, requirements, design, implementation, testing and ongoing responsibility. These stages repeat as evidence changes; security, ethics, accessibility and QA run throughout.

1. Discovery & feasibility

Define the problem, users, intended benefit, constraints and alternatives. Decide whether AI is warranted and what would justify stopping.

Evidence: Problem statement, scope, stakeholder map and feasibility decision.

Project considerations →

2. Requirements & acceptance

Specify functional behaviour, data needs, privacy, accessibility, performance and error limits. Agree measurable acceptance criteria and prohibited uses before building.

Evidence: Versioned requirements and traceable acceptance tests.

Acceptance review →

3. Architecture & design

Map data flow, interfaces, trust boundaries, permissions and failure modes. Choose appropriate components and plan safe fallback, monitoring and rollback.

Evidence: Architecture diagram, threat model, data contracts and design review.

Architecture discussions →

4. Implementation & integration

Build small increments, version changes and review dependencies. Keep data handling, computation and rendering separate; verify component interactions.

Evidence: Reviewed source, reproducible configuration and integration evidence.

Runnable projects →

5. Verification & validation

Check implementation against requirements and suitability for the intended use. Test normal, edge, invalid and failure cases; investigate subgroup impacts and document limitations.

Evidence: Test results, reproducible failures, fixes, regression reports and QA findings.

Break, Fix & Verify →

6. Release & deployment

Make an evidence-based release decision. Validate the deployment environment, access controls, recovery and support arrangements; begin with a bounded pilot.

Evidence: Release record, unresolved risks, operational runbook and rollback checks.

Readiness roadmap →

7. Operation & change

Monitor outcomes and drift, handle incidents, maintain dependencies and reassess impacts. Revalidate changes to code, models, data, prompts or context.

Evidence: Monitoring history, incident records and change approvals with retest evidence.

Operational stages →

8. Retirement & closure

Plan replacement or withdrawal, hand over responsibilities, retire resources and apply justified retention/deletion. Preserve authorised evidence and lessons.

Evidence: Handover, retirement verification, archive decisions and final report.

Project closure →

Where this lab is now: the page contains runnable prototypes and a verification toolkit. A full live-page QA review remains to be completed. Trained-model evaluation, authenticated storage and production deployment are future extensions, not completed milestones.

AI work adds iterative data preparation, model evaluation, drift and impact review to the software lifecycle. Gantt plans schedule activities; review gates require evidence.

INTERACTIVE PROJECT WORKBENCH

Testing steps

Discuss this project

Suggested interview answers · select a question to expand · swipe through topics.

PROJECTS 06–10 · END-TO-END BROWSER DEMOS

Five further AI project patterns

Follow an input through processing, results and inspection. These transparent baselines demonstrate the workflow; they do not use pretrained language or vision models.

NEXT STEP · REPRODUCIBLE EXPERIMENTS

Record, compare & explain

Run a baseline, change one input, then compare the outputs. Build an evidence record before extending a prototype with a trained model.

  1. Run baseline
  2. Record expectation
  3. Save result
  4. Change one input
  5. Run again
  6. Compare results
  7. Explain differences
  8. Export evidence

Records remain in this page session. Export before refreshing or leaving. No server storage or model connection is added; image files themselves are not included.

Run any of the ten projects to capture its latest result.

Save two runs of the same project to inspect their differences.
What this adds to an interview answer

Explain the hypothesis, baseline, single changed variable, observed differences and limitations. Exported records include the project, inputs, output, timestamp and a fingerprint of the executable demo scripts. These records support reproducibility; they do not prove model accuracy, fairness or production security.

The next stage is a versioned evaluation dataset with expected outcomes, then a model-backed comparison under the same tests. Production use would need authenticated private storage, audit trails and independently evaluated models.

ENGINEERING & ETHICS · BREAK, FIX & VERIFY

Test the system, challenge the claim

Use synthetic fixtures to test the same algorithms as the ten experiments. Introduce a controlled fault in the test harness, observe the failures, then remove it and rerun. Live project algorithms remain unchanged by fault selection.

  1. Define requirement
  2. Predict failure
  3. Run baseline
  4. Inject fault
  5. Inspect evidence
  6. Apply fix
  7. Run regression
  8. Review residual risk

Ready: 20 checks across ten projects. No external systems are targeted.

How to break it ethically

Use the supplied synthetic cases, your own non-sensitive examples and authorised local tests. Challenge empty inputs, duplicates, thresholds, irrelevant retrieval, ambiguous categories and constant series. Define an expected result before running. Keep scope local: these exercises do not scan or attack other websites.

Controlled local faults, synthetic data and explicit expected outcomes.

How to fix and verify

Reproduce the failing case, identify its cause, change one control, then rerun the failure case and the full regression suite. The Apply fix button removes the simulated harness fault; it does not patch production code. A real fix requires a reviewed source change, regression evidence and a deployment/rollback plan.

Reproduce → isolate → correct → regress → document remaining limits.

Architecture & cybersecurity

Separate input handling, validated computation and safe output rendering. In production, add authenticated identities, server-side authorisation, isolated workloads, bounded resources, secret management, dependency review, safe logs and monitoring. Browser-only fixtures verify algorithm behaviour; they cannot verify backend isolation or access controls that are not present.

Define trust boundaries and verify controls where they actually run.

Ethics & evidence

Define affected people, intended use, harms and review routes. Test subgroup outcomes and abstention where applicable, examine evidence quality and document uncertainty. A passing selection-rate test proves arithmetic on a fixture, not social fairness. Twenty passing cases provide bounded regression evidence, not assurance for unseen data.

Separate observed behaviour from broader claims about fairness and safety.

Standards & review gates

Use NIST AI RMF to structure Govern, Map, Measure and Manage activities. Use OWASP ASVS to select and verify relevant application-security requirements. Record the version, requirement, applicability, evidence, owner and unresolved gaps. These learning exercises are not certification or a completed standards assessment.

Map controls to requirements and collect evidence; do not assume compliance.

Next model-backed extension

Freeze a labelled evaluation set; compare the baseline with one real classifier using identical cases. Record model/version, licence, data permissions, latency, errors, calibration and subgroup limitations. Then evaluate private storage and access controls before adding durable records. No trained model or persistent service is connected here yet.

Evaluation dataset → model comparison → private records → operational review.

NEXT STEPS · FROM DEMONSTRATION TO ASSURANCE

Build stronger evidence

Progress through these stages after the break–fix exercises. Each is planned work, with a concrete result to review before moving on.

  1. Expand evaluation cases

    Build a versioned set of normal, edge, invalid and adversarial examples. Record expected outcomes before running, include subgroup cases where relevant, and keep a separate held-out evaluation set.

    Evidence: dataset provenance, permissions, expected results, coverage gaps and repeatable test reports.

  2. Visualise model behaviour

    Add ranking charts, confusion matrices, threshold curves, group comparisons and anomaly plots. Show sample sizes, uncertainty and assumptions so a chart supports interpretation rather than implying certainty.

    Evidence: plots linked to saved run inputs and outputs, with concise explanations of observed trade-offs.

  3. Compare trained models

    Start with one text classifier. Run the transparent baseline and a trained model on the same evaluation cases; record model version, licence, latency, errors, calibration and relevant subgroup performance.

    Evidence: reproducible baseline-versus-model comparison, model documentation and a reasoned decision about when to abstain or request human review.

  4. Secure persistent records

    Add authenticated private storage and project-level permissions. Verify server-side authorisation, tenant separation, secret handling, retention and deletion before using sensitive research data.

    Evidence: trust-boundary diagram, threat model, access-control tests and a reviewed retention policy.

  5. Review deployment readiness

    Review requirements, ethical impacts and applicable security controls; document unresolved risks and obtain independent review. Then test monitoring, incident handling and rollback in a controlled deployment.

    Evidence: requirements-to-test mapping, research report, operational runbook and a justified release or no-release decision.

Start here: expand the evaluation cases. More passing fixtures strengthen coverage; unseen-data performance and production assurance still require separate evidence.

AFTER READINESS · OPERATE, REVIEW & RESEARCH

What comes after?

Once a release is justified, move through controlled operation and continuing review. These are future stages; the current laboratory remains a browser demonstration.

  1. Run controlled pilots

    Begin with a limited, authorised user group and low-consequence tasks. Set measurable success criteria, stop conditions, human review and a fallback before widening use.

    Evidence: pilot protocol, consent where applicable, user feedback and a documented decision to continue, revise or stop.

  2. Monitor changing behaviour

    Track quality, latency, failures, data drift and subgroup outcomes against the approved baseline. Set alert thresholds and owners, minimise logged personal data and review whether the original purpose still holds.

    Evidence: monitoring history, drift investigations, service objectives and documented responses.

  3. Practise incident recovery

    Run authorised tabletop exercises and controlled recovery drills. Test escalation, rollback, restoration and communication; verify that failures can be contained without continuing harmful decisions.

    Evidence: drill results, incident timelines, recovery checks and regression tests for fixes.

  4. Reassess societal impacts

    Review accessibility, exclusion, contestability, privacy and unintended uses with affected people and independent reviewers. Compare realised benefits and harms with the original assumptions, including reasons to restrict or retire the system.

    Evidence: updated impact assessment, stakeholder findings, appeal outcomes and a reasoned continuation or retirement decision.

  5. Reproduce and contribute

    Package methods, permitted data, versions and limitations for independent replication. Compare findings with prior research, share only authorised material and turn unresolved questions into the next hypothesis.

    Evidence: reproducible research package, external review, documented contribution and a new testable research question.

Continuing cycle: observe → investigate → improve → revalidate. Higher stakes or wider deployment require stronger evidence and renewed review.

PROJECT PLANNING · INTERACTIVE GANTT PREVIEWS

Different projects, different plans

Tab through eight illustrative plans. Bars show parallel work; diamonds mark review gates. Timings are relative planning examples, not delivery estimates or evidence that work has been completed.

Scaling changes every illustrative duration and gate together. It does not recalculate resources, dependencies or a critical path.

Blue: planned activity · gold diamond: review gate · W: relative week. Gate outcomes may require rework, restriction or stopping.

Read the schedule as a table

Adapt a plan before using it

Replace illustrative durations with estimates based on access, skills, resources and uncertainty. Identify dependencies, owners, acceptance criteria, review time and rework allowances. Recheck the plan when data access, model behaviour, requirements or deployment conditions change. A gate is a decision supported by evidence, not automatic permission to proceed.

COMPLETE THE LIFECYCLE · ADAPT TO THE PROJECT

Key considerations through project end

AI projects can combine many methods, domains and deployment settings. Their milestones depend on the intended use, data, architecture and consequences of failure; the families below overlap rather than form a fixed list of 1,000 distinct types.

How project shapes differ

Swipe across project families; expand a card for milestones, architecture, challenges and review gates.

Rules & expert systems

Milestones: Elicit rules → resolve conflicts → validate exceptions.

Architecture: Rule engine, knowledge base and traceable decisions.

Challenge: Incomplete knowledge, conflicting rules and maintenance drift.

Review gate: Review rule coverage, conflict handling and explanations with domain experts.

Incomplete knowledge, conflicting rules and maintenance drift.

Supervised prediction

Milestones: Define labels → split data → train → calibrate → evaluate.

Architecture: Versioned features, training pipeline and inference service.

Challenge: Label quality, leakage, imbalance and distribution shift.

Review gate: Evaluate held-out performance, relevant subgroup errors and abstention.

Label quality, leakage, imbalance and distribution shift.

Clustering & anomaly detection

Milestones: Define normality → choose distance → inspect clusters → validate alerts.

Architecture: Feature pipeline, statistical or learned baseline and review queue.

Challenge: No definitive labels; unstable clusters, false alarms and changing baselines.

Review gate: Test stability and usefulness with reviewers; do not equate unusual with harmful.

No definitive labels; unstable clusters, false alarms and changing baselines.

Generative language models

Milestones: Evaluate model → constrain context → test outputs → monitor use.

Architecture: Model endpoint or local inference, prompt templates and output controls.

Challenge: Unsupported claims, injection, privacy, misuse and evaluation ambiguity.

Review gate: Assess factual support, harmful outputs, abstention and human-review routes.

Unsupported claims, injection, privacy, misuse and evaluation ambiguity.

Retrieval-augmented generation

Milestones: Authorise sources → index → retrieve → generate → verify citations.

Architecture: Permission-aware retrieval, model inference and citation provenance.

Challenge: Stale sources, retrieval misses, cross-user leakage and unsupported citations.

Review gate: Test retrieval and generation separately, including access-denied and no-answer cases.

Stale sources, retrieval misses, cross-user leakage and unsupported citations.

Computer vision

Milestones: Collect representative images → annotate → train → validate conditions.

Architecture: Image decoding, preprocessing, model inference and output interpretation.

Challenge: Lighting, occlusion, demographic coverage, image privacy and compute costs.

Review gate: Test representative environments and shift; include uncertain or unsupported images.

Lighting, occlusion, demographic coverage, image privacy and compute costs.

Speech & audio

Milestones: Capture audio → segment → transcribe/classify → validate speakers.

Architecture: Audio pipeline, speech model and timing or transcript outputs.

Challenge: Accent variation, noise, diarisation errors, consent and latency.

Review gate: Evaluate task-appropriate errors across speaking conditions and user groups.

Accent variation, noise, diarisation errors, consent and latency.

Recommenders & ranking

Milestones: Define objective → test relevance → pilot exposure → monitor feedback.

Architecture: Candidate retrieval, ranking model, experimentation and feedback logs.

Challenge: Feedback loops, exposure inequality, popularity bias and objective gaming.

Review gate: Balance relevance with user welfare and exposure; review online and offline evidence.

Feedback loops, exposure inequality, popularity bias and objective gaming.

Forecasting & time series

Milestones: Order data → backtest → model seasonality → quantify uncertainty.

Architecture: Time-aware feature pipeline, rolling evaluation and forecast monitoring.

Challenge: Temporal leakage, regime change and overconfident long-range predictions.

Review gate: Use time-based validation and uncertainty checks; compare simple baselines.

Temporal leakage, regime change and overconfident long-range predictions.

Reinforcement learning

Milestones: Define rewards → simulate → constrain actions → evaluate policy.

Architecture: Environment, agent policy, reward signals and action boundary.

Challenge: Reward gaming, unsafe exploration and simulator-to-world mismatch.

Review gate: Test constrained policies in safe environments before consequential real-world actions.

Reward gaming, unsafe exploration and simulator-to-world mismatch.

Agents & autonomous tools

Milestones: Scope tools → enforce permissions → test plans → supervise actions.

Architecture: Planner, tool gateway, bounded memory, audit trail and approval controls.

Challenge: Excessive permissions, indirect injection, cascading errors and irreversible actions.

Review gate: Verify least privilege, stop conditions, action review and recovery under failure.

Excessive permissions, indirect injection, cascading errors and irreversible actions.

Edge & embedded AI

Milestones: Profile hardware → optimise → test device → update safely.

Architecture: On-device inference, constrained memory/power and signed update path.

Challenge: Latency, thermal limits, offline operation and physical safety constraints.

Review gate: Measure on target hardware and validate fallback, updates and failure behaviour.

Latency, thermal limits, offline operation and physical safety constraints.

Federated & privacy-preserving AI

Milestones: Agree participation → distribute updates → aggregate → evaluate leakage.

Architecture: Participant clients, aggregation coordinator and privacy controls.

Challenge: Non-identical data, poisoned updates, privacy leakage and participant dropout.

Review gate: Test aggregation resilience and privacy assumptions; decentralisation alone is not privacy.

Non-identical data, poisoned updates, privacy leakage and participant dropout.

Scientific & causal modelling

Milestones: Specify assumptions → calibrate → validate → assess sensitivity.

Architecture: Model equations or learned components, provenance and reproducible analysis.

Challenge: Confounding, identifiability, uncertain parameters and invalid extrapolation.

Review gate: Check assumptions and independent evidence; predictive fit alone does not establish causality.

Confounding, identifiability, uncertain parameters and invalid extrapolation.

Multimodal & hybrid systems

Milestones: Align modalities → integrate components → test interfaces → evaluate system.

Architecture: Combined text, image, audio, rules or retrieval components.

Challenge: Cross-modal errors, missing inputs, inconsistent meanings and complex dependencies.

Review gate: Test each component and interactions, including conflicting or unavailable modalities.

Cross-modal errors, missing inputs, inconsistent meanings and complex dependencies.

Questions to carry throughout

Revisit these considerations at discovery, evaluation, release, operation and closure. Evidence requirements grow with the consequences of failure.

Purpose & proportionality

Define the problem, affected people, intended benefit, alternatives and reasons to use AI. Prefer the least complex approach that meets the requirement; record out-of-scope uses and stop conditions.

Define the problem, affected people, intended benefit, alternatives and reasons to use AI. Prefer the least complex approach that meets the requirement; record out-of-scope uses and stop conditions.

Stakeholders & accountability

Identify the owner, operator, reviewer, data steward and incident decision-maker. Specify how affected users can question outcomes and who can suspend, correct or retire the system.

Identify the owner, operator, reviewer, data steward and incident decision-maker. Specify how affected users can question outcomes and who can suspend, correct or retire the system.

Data rights & provenance

Establish permission, origin, quality, representativeness and retention. Track source transformations and access boundaries. Assess whether labels, training material or derived outputs expose private or restricted information.

Establish permission, origin, quality, representativeness and retention. Track source transformations and access boundaries. Assess whether labels, training material or derived outputs expose private or restricted information.

Evaluation & uncertainty

Choose metrics from the task and harms, not convenience. Define acceptance criteria before testing; include leakage checks, edge cases, relevant subgroup analysis, uncertainty and independent validation.

Choose metrics from the task and harms, not convenience. Define acceptance criteria before testing; include leakage checks, edge cases, relevant subgroup analysis, uncertainty and independent validation.

Architecture & dependencies

Document trust boundaries, interfaces, versioned contracts, dependencies and failure modes. Test component interactions, graceful degradation and rollback rather than relying on a model score alone.

Document trust boundaries, interfaces, versioned contracts, dependencies and failure modes. Test component interactions, graceful degradation and rollback rather than relying on a model score alone.

Security & abuse resistance

Use threat modelling, least privilege, server-side authorisation where needed, input/resource limits, secret management and dependency controls. Test only authorised environments; keep a record of unresolved vulnerabilities and mitigations.

Use threat modelling, least privilege, server-side authorisation where needed, input/resource limits, secret management and dependency controls. Test only authorised environments; keep a record of unresolved vulnerabilities and mitigations.

Human factors & accessibility

Test whether people understand outputs, limitations and review routes. Consider accessibility, automation bias, workload, meaningful oversight and whether users can recover from mistakes.

Test whether people understand outputs, limitations and review routes. Consider accessibility, automation bias, workload, meaningful oversight and whether users can recover from mistakes.

Cost & environmental impact

Measure relevant compute, latency, storage, hardware and operational effort. Compare resource demand with evidenced benefit, including costs of maintenance, retraining and human review.

Measure relevant compute, latency, storage, hardware and operational effort. Compare resource demand with evidenced benefit, including costs of maintenance, retraining and human review.

Change & operational resilience

Version models, data, prompts and configuration. Define monitoring, incident response, recovery and rollback. Revalidate meaningful changes and context shifts instead of treating initial approval as permanent.

Version models, data, prompts and configuration. Define monitoring, incident response, recovery and rollback. Revalidate meaningful changes and context shifts instead of treating initial approval as permanent.

Governance & applicability

Determine which requirements apply to the actual use, jurisdiction and sector. Map selected standard requirements to evidence and owners; keep unresolved gaps visible. Recheck applicability when purpose or deployment changes.

Determine which requirements apply to the actual use, jurisdiction and sector. Map selected standard requirements to evidence and owners; keep unresolved gaps visible. Recheck applicability when purpose or deployment changes.

Communication & research claims

Separate measured results, assumptions and hypotheses. Report methods, limitations, exclusions and uncertainty. Protect private material while preserving reproducibility and source attribution.

Separate measured results, assumptions and hypotheses. Report methods, limitations, exclusions and uncertainty. Protect private material while preserving reproducibility and source attribution.

Closure & residual obligations

Specify ownership after handover, record retention, deletion, appeal support and incident follow-up. End active inference deliberately; archive authorised evidence and verify that credentials, integrations and resources are retired.

Specify ownership after handover, record retention, deletion, appeal support and incident follow-up. End active inference deliberately; archive authorised evidence and verify that credentials, integrations and resources are retired.

Finish responsibly

  1. Complete acceptance review

    Reconcile objectives against evidence, document residual risks and record the decision: accept, restrict, rework or stop. Include independent review where the stakes require it.

    Evidence: acceptance record, unmet requirements, limitations and accountable decision owner.

  2. Transfer operational ownership

    Hand over versioned artifacts, runbooks, monitoring, support routes, permissions and training. Identify responsibilities for incidents, changes and ongoing stakeholder concerns.

    Evidence: handover record, responsible owners and verified access/support arrangements.

  3. Retire or replace

    Plan withdrawal, notify relevant users, stop inference and scheduled jobs, revoke unnecessary credentials and retire resources. Validate replacement behaviour and maintain appropriate routes for unresolved decisions.

    Evidence: retirement checklist, resource checks, migration tests and communication record.

  4. Retain authorised evidence

    Keep only records justified by research, contractual or applicable retention requirements. Apply agreed deletion to unnecessary data and document limitations of what can be reproduced after deletion.

    Evidence: archive index, retention decisions and deletion verification.

  5. Close and learn

    Record lessons, unresolved research questions, benefits and harms. Close outstanding actions or assign continuing owners. Feed findings into future work without losing source provenance.

    Evidence: final report, lessons register and owned follow-up actions.

Use the appropriate evidence: a research prototype, public recommender, clinical model and autonomous device require different acceptance criteria. A shared lifecycle provides structure; task-specific validation determines whether a particular claim is justified.

RESEARCH INTERVIEW & ETHICS WORKSHOP

Advanced engineering questions & ethical dilemmas

Expand each question to reveal a short answer, an architectural sequence and an invitation to explore the relevant technical or ethical trade-off. These are learning prompts, not universal solutions.

CLOUD DEVELOPMENT · OPEN SOURCE CODE

Open the working Python projects remotely

Each project now has a full, editable Python script in this repository, separate from the small browser demonstration. The scripts run with the Python standard library; they use synthetic data and provide real source-code walkthroughs for studying algorithms.

Launch a GitHub Codespace for this repository ↗ · Read run instructions

How remote execution fits the architecture
Website→GitHub identity→Codespaces development VM→Python scripts→Terminal results

GitHub Codespaces is an external, account-based hosted development environment; it may be subject to access requirements and usage charges. This is not a private multi-user HPC cluster, nor does this public repository provide regulated-data storage.

Do not upload restricted datasets or API secrets to a public repository. A research production service needs separately managed identity, private storage, compute isolation and project-level permissions.

From a teaching demo to a real doctoral lab

This page intentionally runs transparent, deterministic JavaScript algorithms so you can inspect every transformation. Deploying real notebooks, pretrained models, protected research data or GPU jobs requires a separately authenticated backend, secure provider connections and resource controls. A public GitHub Pages website cannot safely store API keys or execute private server-side jobs.

DISTINCT QA REALM · VERIFY BEFORE EXPANDING

Quality assurance & validation

Use this review after the project and testing sections. QA also belongs throughout development: this is a focused check that the page works, communicates accurately and fulfils its learning purpose.

What to do next

Complete one journey first, then broaden coverage. Scroll through the steps and expand only the detail you need.

1. Run a baseline

Open one project, keep its example inputs and run it. Record the project, inputs and expected result; inspect the actual output.

Check for: A valid output and a clearly stated baseline expectation.

2. Save baseline evidence

In Record, Compare & Explain, label the run Baseline, add your expectation and observation, then save the latest run.

Check for: A saved baseline with inputs, output and context.

3. Change one input

Return to the same project and change one parameter or one data element. Predict the effect before running again; label and save the changed run.

Check for: A comparison in which the changed variable is identifiable.

4. Compare the results

Select the two saved runs of the same project. Inspect changed inputs and outputs; explain an unexpected difference or an unchanged result.

Check for: A reasoned observation, with uncertainties recorded.

5. Break and repair

Run the verification suite without a fault. Inject each controlled fault, inspect the failures, then select Apply fix & rerun. These fixes remove simulated harness faults rather than patching the source.

Check for: A recorded baseline, detected failure and successful regression rerun.

6. Check downloads

Export experiment records, verification evidence and the research report. Open each downloaded file and check that it contains the intended run, outputs and notes.

Check for: Readable exports containing the expected evidence.

7. Review project plans

Open each Gantt tab, change the timeline scale, inspect review gates and the schedule table, then download and open a chart.

Check for: Consistent chart/table timing and a usable SVG download.

8. Record QA findings

In the QA realm, enter device/browser, set each reviewed check’s status and add expected-versus-actual evidence. Leave unreviewed checks as Not tested.

Check for: A truthful checklist with open issues and a provisional acceptance decision.

9. Repeat across projects

Repeat the project journey for all ten experiments, including invalid inputs and abstention cases. Check file handling for the document and image projects.

Check for: Coverage across every project, including error paths.

10. Review access and layout

Test phone and desktop layouts, keyboard controls, focus, zoom, labels, horizontal scrolling and reduced-motion behaviour. Check that text stays readable and controls remain reachable.

Check for: Device-specific findings and accessible routes through the page.

11. Fix and recheck

Prioritise failures by impact. Record an owner and corrective action; reproduce the issue, fix it, then rerun the affected journey and regression checks.

Check for: A documented fix with retest evidence and remaining limitations.

12. Decide the next build

Review whether the page meets its learning purpose. Once material defects are resolved, evaluate one trained model against a baseline on held-out examples.

Check for: A justified acceptance decision and a bounded next-stage plan.

Keep experiment and QA exports before leaving: current records are held only in the page session.

All checks start as Not tested. Statuses below are reviewer-entered observations, not automated verification or certification. This checklist does not execute security scans.

0 of 12 checks reviewed.

Functional testing

Usability & accessibility

Content validation

Regression testing

Acceptance testing

Security & ethical review

Keep failed and blocked checks visible. Fix defects, rerun affected tests and regression checks, then reassess acceptance. The checklist remains in this page session; export before leaving.

PROJECT → CLOSURE → NEXT WORK

Finish this project. Begin the next.

The next thing is a new discovery and scoping cycle. A completed live system moves into operations; a completed research project leaves a documented contribution and questions for future work.

1. Confirm the brief

Agree the purpose, affected people, requirements, constraints and acceptance criteria. Adapt evidence requirements to the project type and consequences of error.

2. Verify the work

Review implementation, data/model assumptions, security, ethics and integration. Complete functional, accessibility and acceptance checks; record unresolved gaps.

3. Fix and regress

Reproduce defects, make reviewed changes and rerun relevant tests. Repeat until acceptance criteria are met or a decision is made to restrict, redirect or stop.

4. Release or conclude

For a deployed product, verify the release and rollback path. For a research study, validate findings and publish or archive authorised outputs. A no-release or negative-result decision can also close a project.

5. Hand over ownership

Transfer artifacts, evidence, limitations, support duties and outstanding actions. A live service continues under operational ownership; it does not need to be retired merely because development closes.

6. Close the project

Record acceptance or termination, final results, lessons, resource commitments, retention decisions and accountable owners. Archive authorised evidence and close or transfer remaining actions.

7. Select the next work

Review unmet needs, lessons, risks and opportunities. Choose the next project by expected value, feasibility and available capacity, rather than expanding the previous project indefinitely.

8. Begin new discovery

Create a fresh brief, hypothesis, owner, scope and success criteria. Reuse justified components and evidence, but revalidate assumptions when purpose, users, data or stakes change.

CI/CD and model loops within the sequence

Software changes follow review → automated checks → preview → acceptance → release → monitoring → correction. Data/model changes also require appropriate evaluation, versioning and impact review. Repeat the necessary loop without losing the project’s exit criteria. Pipeline quality gates are planned work here, not an established repository workflow.

Different project shapes, shared closure

A study may end in a report, an audit in findings and remedial actions, a model experiment in a reproducible evaluation, and a product in a supported release. Each needs a clear outcome, evidence, limitations, an owner for ongoing duties and an explicit closure decision. Some projects stop before release when evidence does not justify proceeding.

Current transition: finish live QA and establish repeatable quality gates, document acceptance and handover, then choose the next bounded model-backed experiment.

Evaluation research extension

Twelve verification functions and four controlled evaluation conditions explore independent assessment of coordinated AI systems, with explicit attention to reliability, cost and unfamiliar tasks. Read the proposed research framework.

Additive research note, 4 November 2026. Conceptual and testable; not a validated ASI capability.

Synthetic evidence-assessment starter pack

For a bounded next research task, see the research delivery roadmap and its 15 fictional claim-relation examples. The pack is an initial educational fixture for testing contradictions, temporal versions, evidence gaps and uncertainty, not a validated model benchmark.

Added 5 November 2026 · Existing lab content preserved.

Research transfer and declared completion perimeter

For a concise successor handover that links the existing experiments, decisions, validation gates, cross-disciplinary connections and lifecycle safety risks without duplicating this milestone table, see Research Transfer, Scope & Completion. The original results and their limitations remain recorded below.

Verification lab: milestones, findings & design lessons

Research in brief. A sequence of offline Python/FastAPI experiments progressively separates software correctness, evidence-relation classification, structured reasoning and natural-language extraction. The most useful discovery is an architectural one: passing tests or matching familiar labels does not establish accurate interpretation of unfamiliar documents. Each result below is qualified by its test conditions; no real-world model reliability, autonomous intelligence or institutional adoption is claimed.

Execution priorities · Evaluation protocol · Initial synthetic examples · Original feature proposals

Observed local development experiments, 5 November 2026. Reference labels are synthetic/editorial, not independently expert-adjudicated.
PhasePinpointed observationLesson and architecture response
v0.1–0.3 · FoundationOffline FastAPI scaffold and synthetic batch; v0.3 baseline abstained on 15/15 cases (1/15 matching editorial label). 16 tests passed.Reproducibility and cautious abstention come before model complexity; report coverage alongside agreement.
v0.4 · Familiar casesEvidence-aware lexical rules matched 14/15 in-sample labels; 26 tests passed.Good developer-fixture performance is not evidence of transfer. Keep the earlier baseline and record remaining disagreements.
v0.5 · New wordingFrozen v0.4 rules matched 5/30 new editorial labels, abstaining on 28/30 (baseline 4/30).Keyword dependence is brittle under paraphrase. Avoid tuning on the held-out material while still calling it unseen.
v0.6 · Typed propositionsStructured claim schema and controlled logic/API contract; 45 tests passed. Earlier results frozen.Separate correctly representing a claim from reasoning over the representation. API correctness is not document understanding.
v0.7 · Extraction pipelineRestricted sentence templates: 35/40 statements parsed; 4/20 pairs abstained; 89 tests passed.Make extraction failures visible; do not manufacture semantic certainty from unsupported language. Contract agreement was in-sample.
v0.8 · Challenge casesFrozen parser on 24 authored pairs: 36/42 annotated statements matched all typed fields; 12/18 adjudicated pair relations matched end-to-end; 6/6 ambiguous pairs abstained; 95 tests passed.Paraphrases caused the six end-to-end misses; the reasoner matched 18/18 on hand-labelled typed inputs. Prioritise extraction evaluation and source-grounded provenance, not more agents.
v0.9 · Review readiness12 fresh fictional pairs prepared with two blank reviewer forms, agreement/adjudication tooling and 101 passing regression tests; 0/12 independently labelled so far.Separate evaluation governance from model scoring. A reproducible annotation workflow is necessary, but it does not generate independent ground truth or establish improved accuracy.
v1.0 · Consistency and metamorphic checksFrozen v0.7 extractor: 100/100 transformations met their stated behavioural invariants on 20 already-exposed fictional contract pairs; 104 automated tests passed.Separate implementation invariants (symmetry, normalisation, safe refusals and human-review gating) from semantic accuracy. This is regression evidence, not new blind validation or evidence of general-language understanding.
v1.1 · Audit and adaptation gatesRead-only local audit checked five preserved SHA-256 baseline components, 100/100 exposed metamorphic invariants and 109 passing automated tests; independent reviewer labels remain 0/12.File integrity and consistent implementation are necessary but insufficient: independent semantic evidence and production security require separate gates. Publish explicit pass/blocked criteria, not a single inflated readiness score.
v1.1: reproduce the audit and adapt architecture to desired outcomes

Procedure: from the separate offline Python v1.1 package, run python -m mg_lab.repro_audit_v11 --output RESULTS_REPRO_AUDIT_v11.json, followed by python -m pytest -q. The checker verifies five frozen component hashes, replays the v1.0 fixture invariants, checks independent reviewer-form completion and refuses to infer production release readiness. Recorded outcome: five frozen files preserved, 100/100 invariants passed, 109 tests passed, reviewer relation labels 0/12; independent accuracy and deployment gates remain blocked.

Architecture by goal: For repeatable research, preserve hashes, source versions, synthetic fixtures and observable traces. For reliable evidence interpretation, add blinded independent annotation and separate field extraction from typed reasoning. For audit-sensitive use, require source provenance, historical as-of logic, error disclosures, human sign-off and measured false certainty. For lower latency, compare resource usage at equivalent quality; simply adding agents is not proof of improvement. For a deployed API, require independently verified auth, rate limits, access control, logging, secret handling and rollback testing.

Failure response: a hash mismatch blocks claims of frozen-baseline reproduction; failed invariants trigger a software correction and repeat test; missing independent labels block new accuracy claims; absent security testing blocks external deployment. No synthetic or editorial label establishes real source truth. See the evaluation protocol and current execution handoff. The Python v1.1 archive is a separate local research deliverable, not a public hosted service.

v1.0: reproducibility steps and research-design handoff

Procedure: retain the v0.7 parser and original fixture; run python -m pytest -q and python -m mg_lab.metamorphic_v10 in the separate local Python v1.0 package. For each of the 20 exposed pairs, test swapping claim order, uppercasing, harmless outer whitespace, an unsupported report prefix and a malformed date prefix. Preserve observed and expected relations, an explicit abstention for unsupported syntax, and the mandatory human-review flag. Record 100 checks and their failure details; the observed implementation result was 100/100.

Architectural lesson: invariants can check code paths and gate conditions, but successful transformations of developer-exposed examples cannot measure extraction on unfamiliar evidence. If invariants break, correct canonicalisation, temporal parsing or approval control flow before expanding architecture. If they pass, proceed to genuinely independent, blind annotation before comparing extraction models. Source-truth verification requires additional provenance checks.

Re-use: the v1.0 local deliverable includes the executable module, archived v0.3–v0.9 materials, tests, machine-readable results and a milestone handoff. It has not been deployed or uploaded as a public source-code package.

Methods and metric interpretation

Run local regression checks with python -m pytest -q; keep fixtures, component hashes, environment, expected labels, predicted labels and abstentions together. v0.3–0.5 compare authored synthetic relation labels. v0.6 checks typed contracts. v0.7 contract labels were constructed alongside the parser, so a match is not independent language-understanding accuracy. v0.8 was written after the parser freeze but was still assistant-authored rather than blind expert-labelled. Its 12/12 answered-only matches must be reported alongside 12/18 end-to-end coverage; do not disguise abstentions as correct relation predictions. The 18/18 typed comparison is a gold-input contract check, not a source-truth test.

What to correct, preserve or adapt in future architectures

Keep: versioned data and control paths, typed claims, source/as-of fields, conservative abstention, explicit human review, frozen baselines and reproducible trace outputs. Improve: paraphrase and entity normalisation, multiple propositions, negation, observation versus publication time, source reliability and independent evaluation labels. Avoid: automatic approval, mixing real confidential records into public fixtures, treating simulated scores as measured capability, and equating agreement with reference labels to factual truth. The optimum design depends on intended outcomes: an audit tool may privilege provenance and abstention, whereas a discovery assistant may also require measured coverage and calibrated uncertainty.

v0.9 reviewer handoff: repeatable process and decision gates

1 · Prepare: freeze versioned fictional text, hash inputs and withhold predictions/reference labels. 2 · Annotate: two actual independent reviewers label subject, predicate, value, scope, observation versus publication time, evidence status and relation; uncertain fields remain unknown. 3 · Reconcile: compute agreement on both-labelled cases and field-level completeness; preserve individual reviews and retain adjudication notes. 4 · Evaluate: lock adjudicated labels before scoring frozen extraction and reasoning components, reporting all-case coverage, answered-only agreement, abstention and unsafe assertions separately. 5 · Adapt: choose architecture changes based on demonstrated failure type, comparing outcomes against reliability, privacy and cost constraints. The prepared v0.9 local Python package contains twelve fictional unlabelled cases, reviewer CSV templates, and an agreement checker; reviewers have not submitted any annotations and no independent performance score is available.

Next publication gate: independent review and experiment

Before altering the extractor, freeze the current code and evaluation definitions; create a fresh dataset whose labels and predictions are concealed from at least two independent reviewers; record field-level disagreements and an adjudication log. Test extraction and relation reasoning separately and end-to-end, reporting all-case agreement, answered-only agreement, abstentions, error types and costs. If independent annotation is unavailable, describe findings as exploratory engineering evidence only. Publish a concise results table plus a linked technical appendix, without changing historical findings.

Reviewer materials: obtain separate labels before scoring (available now)

Shareable fictional reviewer packet: 12 claim pairs, no labels (JSON) · Reviewer A blank CSV · Reviewer B blank CSV · Review protocol and coordinator safeguards.

Researcher sequence: distribute the same scenario-tag-free case text to each reviewer separately; obtain independent field and relation labels; privately retain signed-off originals, then calculate agreement and adjudicate disagreements before revealing any system outputs. Record unanswered items rather than manufacture consensus. This publication contains no model predictions, reference labels or completed reviewer answers.

Outcome-based adaptation: if disagreements cluster around temporal wording, change time representation and annotation rules; if they cluster around negation, test extraction logic; if scope or evidence is unspecified, prefer recorded abstention and review over confident resolution. Neither agreement between reviewers nor successful implementation establishes factual truth without external source verification.

Status: reviewer materials published; 0/12 independently reviewed so far. The set remains assistant-authored and requires external reviewer participation and independently sourced cases for stronger future inference.

From a discovery to a justified design change

Use this five-step researcher handoff with the milestone table above. Each change needs a failure mechanism, a reproducible comparison and a stated evidence gate; a larger model or more agents is not automatically progress.

1. Observation 2. Failure/uncertainty 3. Architecture choice 4. Falsifiable test 5. Evidence gate

Download the structured decision trace (JSON, five decisions and five failure responses). It links v0.3–v1.1 observations to design rationale and next tests; it does not replace the original code, artefacts or labels.

Example: familiar-case success → paraphrase failure → modular extraction

Observed: v0.4 matched 14/15 authored development labels, then the frozen rules matched 5/30 newly worded labels in v0.5, with 28 abstentions. Interpretation: lexical matching did not transfer reliably to changed phrasing under those authored synthetic conditions. Design response: keep claim extraction separate from relation comparison; log field-level errors. Next falsification: compare both extractors on newly and independently annotated material, including answer coverage and unsupported claims. Blocked inference: no general-world accuracy improvement is demonstrated.

Example: typed reasoning works → end-to-end coverage still falls short

Observed: v0.8 matched 18/18 editorial relation labels with supplied typed claims versus 12/18 end-to-end, with six misses associated with unsupported paraphrases. Interpretation: input representation, not only downstream logic, constrains the pipeline. Design response: diagnose extraction, comparison and evidence provenance independently; never treat gold structured inputs as an actual document extractor. Next falsification: independently label fields and relationships, then compare conditional and all-case performance at fixed abstention policy.

Decision by desired outcome, including when to stop

Evidence integrity: prioritise source/date/scope provenance, preserved records, human adjudication and abstention. Higher coverage: test an alternative extractor, count unsupported assertions and compare against the same frozen cases. Lower operating cost: benchmark runtime and resource use at equal measured error/coverage, not raw speed alone. Production use: require permissions, authentication, logging, privacy and security tests independent of synthetic NLP scores. If reference labels are missing, stop before an accuracy claim. If file hashes change, stop before a reproducibility claim. If deployment controls are untested, do not advertise production readiness.

Status: this is an evidence-to-decision synthesis of already documented local work, not a new algorithm benchmark or an independently validated finding. No independent annotations are yet recorded.

Research status: local synthetic implementations and recorded tests, not a deployed or independently certified service. Full code bundles and detailed v0.7/v0.8 reports were created in local development artefacts; this website summary does not imply those files have been uploaded to the public repository. Added 5 November 2026 as an additive milestone record.