Manifold Grace

INDEPENDENT RESEARCH ยท PRACTICAL EXPLANATION

How to audit AI systems for bias, reliability and explainability

An AI audit needs a defined system, intended use, evidence and evaluation criteria. This guide describes a research-oriented workflow for identifying risks and reporting limitations; it is not a certification or a claim that any specific system has been audited.

Scope: Informational research guide. Adapt the method to the relevant domain; follow applicable professional, legal and safeguarding requirements.

1. Define the audit scope

Record system version, intended users, permitted tasks, deployment setting, data sources, known constraints and decisions the system may influence. Specify what the audit does not cover.

2. Choose measures appropriate to the task

Depending on context, measure factual accuracy, false-positive and false-negative rates, calibration, latency, consistency, human override and error severity. No single metric establishes overall fairness.

3. Build representative test sets

Test common, uncommon and difficult cases; document sampling limitations. Where lawful and ethical, evaluate relevant subgroup outcomes while avoiding inappropriate inference about protected characteristics.

4. Check robustness and explainability

Vary phrasing, data quality and contextual inputs; compare outputs under meaningful changes. Explainability requires usable evidence about a system's behaviour, not merely confident-sounding explanations.

5. Record harms and human oversight

Document incidents, foreseeable misuse, escalation thresholds, responsibilities, challenge routes and mechanisms for correction. Distinguish research observations from organisational decisions.

6. Publish auditable findings

Provide method, test dates, model version, denominator, uncertainty, failures, limitations, mitigation actions and a decision log. A model card or risk register can support traceability.

Worked reasoning example

Example: For an information-retrieval assistant, compare answer accuracy on a documented set of routine and edge-case prompts, track unsupported citations and corrections, then report the conditions under which performance degrades.

Related Manifold Grace research

Standards and supporting references

Published 9 November 2026. This page describes a methodology; it is not a record of a completed external audit or a guarantee of search placement.