Manifold Grace

Foreseeable Harm Catalogue

Documented AI failures, reported incidents, and earlier automation cases. Start with the most recent record and expand any entry for its sources, lessons and potential consequences.

Ministerial accountability proposal

Scope, dates and evidence standards

Coverage snapshot: 11 November 2026. This is a selected catalogue from the 1956 Dartmouth workshop onward. Some cases use the date of a ruling, report, public disclosure or regulatory update, as indicated by their context. Empty years do not establish an absence of incidents. Historical precursors, allegations, studies and adjudicated findings are labelled. The lessons and future consequences are analysis.

To assess foreseeability, identify the warnings and evidence available before a decision, who had authority and budget to act, which controls were feasible, and which outcomes were recorded. Later findings can inform this inquiry. Date new evidence and link it from the original entry so prior wording and citations remain traceable.

Further records: OECD AI Incidents Monitor · AI Incident Database

2026

Specialist funding crowded out

· Foreseeable funding risk

Context and sources

Giving interdisciplinary proposals priority over specialist proposals can weaken the expertise that cross-disciplinary projects depend on, especially where both draw from the same budget. Warning signs include shrinking specialist calls, falling real-terms awards and loss of specialist posts or facilities. UKRI’s 2023 cross-council pilot documents support for interdisciplinary research. Whether such funding displaces specialist work depends on the allocation and award data.

UKRI cross-council funding pilot · Research on data quality in high-stakes AI

Lessons for prevention

Research funding evidence sources

Track awards and budgets by discipline over time, maintain routes for specialist work, and require funded AI projects to involve relevant specialists in data design, annotation and testing.

Consequences if unresolved

Loss of specialist capacity can make rare cases harder to recognise, leave training data without important distinctions and allow models to pass broad evaluations while failing on consequential exceptions.

Grok imagery investigation

· Regulatory investigation

Context and sources

Ofcom opened its inquiry in January after reports of sexualised images of real people, including possible images involving children. Its October update is the catalogue date.

Source for this entry

Lessons for prevention

Test image safeguards and response procedures before release. Monitor abuse across both generation and sharing.

Consequences if unresolved

Non-consensual imagery can spread faster than victims can secure removal.

Third-party agent impact

· Provider investigation

Context and sources

OpenAI published an account of reported effects on third parties involving misaligned models, with a September update investigating further claims.

Source for this entry

Lessons for prevention

Limit delegated permissions and test whether the system respects stop and approval boundaries.

Consequences if unresolved

People outside the deployment can bear costs from unintended automated activity.

2025

Fabricated government report citations

· Reported incident

Context and sources

Deloitte agreed to a partial refund on an Australian government report after errors including a fabricated quotation attributed to a court judgment were identified.

Source for this entry

Lessons for prevention

Require delivery teams to check every cited source and quotation against the original.

Consequences if unresolved

False authority can inform policy and impose repeat review and procurement costs.

Agent deleted project data

· Reported incident

Context and sources

An incident ledger records a Replit coding agent deleting project database data during a code freeze. The record describes a developer’s experience.

Source for this entry

Lessons for prevention

Separate environments, restrict destructive actions and test restoration.

Consequences if unresolved

Automated changes can destroy valuable work faster than a team can intervene.

False court authorities

· Judicial review of filings

Context and sources

The English High Court addressed false citations in the Ayinde and Al-Haroun proceedings. The Judiciary later reviewed safeguards for AI use in preparing court documents.

Source for this entry

Lessons for prevention

Verify every cited authority against an authoritative text before filing, with named professional responsibility.

Consequences if unresolved

False authorities can mislead courts, delay cases and increase costs.

Invented summer reading list

· Publisher acknowledgement

Context and sources

A third-party supplement carried a summer reading list containing invented books. The Chicago Sun-Times published a response explaining the origin of the material.

Source for this entry

Lessons for prevention

Apply editorial verification to syndicated material as well as staff-produced work.

Consequences if unresolved

Fabricated recommendations waste readers’ time and undermine trust in genuine coverage.

2024

Cloned election robocalls

· Regulatory enforcement

Context and sources

The FCC imposed a penalty over illegal robocalls involving an AI-generated imitation of President Biden before the New Hampshire primary.

Source for this entry

Lessons for prevention

Provide rapid ways to authenticate public election messages and flag impersonation.

Consequences if unresolved

Voters can receive misleading instructions too late for a correction to reach them.

Deepfake finance meeting

· Reported fraud

Context and sources

Arup confirmed it was the victim of a Hong Kong fraud involving a deepfake video conference. The Financial Times reported losses of about US$25 million.

Source for this entry

Lessons for prevention

Verify unusual payment requests through established independent channels and require a second approver.

Consequences if unresolved

Convincing impersonation can make an unauthorised transfer appear routine.

Biometric staff attendance

· Regulatory enforcement

Context and sources

The UK ICO took enforcement action against Serco Leisure and associated trusts over biometric employee attendance monitoring.

Source for this entry

Lessons for prevention

Assess necessity, proportionality and less intrusive alternatives before collecting biometrics.

Consequences if unresolved

Workers may have little meaningful choice over collection of persistent identifiers.

Chatbot bereavement fare

· Tribunal ruling

Context and sources

The British Columbia Civil Resolution Tribunal held Air Canada liable after its website chatbot gave incorrect information about obtaining a bereavement fare.

Source for this entry

Lessons for prevention

Bind customer guidance to current policy and provide a clear remedy when the guidance is wrong.

Consequences if unresolved

Customers may spend money on terms they cannot actually obtain.

2023

Retail facial misidentification

· Regulatory complaint

Context and sources

The US FTC alleged that Rite Aid used facial recognition without adequate safeguards and that false alerts led to some customers being followed or confronted.

Source for this entry

Lessons for prevention

Measure false matches under actual conditions and require independent verification before intervention.

Consequences if unresolved

False identification can cause public humiliation and discriminatory treatment.

Automated applicant age exclusion

· Regulatory settlement

Context and sources

The EEOC announced a settlement over allegations that iTutorGroup’s application software automatically rejected older applicants based on age thresholds.

Source for this entry

Lessons for prevention

Audit screening rules against relevant requirements and test boundary cases before using them.

Consequences if unresolved

Eligible applicants can be excluded before any human reviewer sees them.

Robodebt income averaging

· Official inquiry · automated predecessor

Context and sources

Australia’s Royal Commission examined a debt scheme that used averaged annual income to infer fortnightly earnings. The 2023 report is the anchor for the earlier scheme.

Source for this entry

Lessons for prevention

Require legally sufficient evidence for individual debts and check whether recipients can realistically challenge them.

Consequences if unresolved

False debts can cause financial distress at scale while shifting the burden onto claimants.

2022

Galactica demo withdrawn

· Public demonstration

Context and sources

Meta’s public research model demo was withdrawn after users found it could generate false scientific material and offensive responses.

Source for this entry

Lessons for prevention

Test plausible falsehoods and make it easy to verify whether a cited study exists.

Consequences if unresolved

Scientific-looking fabrication can contaminate research and teaching material.

2021

Dutch childcare risk profiling

· Regulatory finding · automated predecessor

Context and sources

The Dutch data protection authority fined the tax administration over discriminatory and unlawful processing of nationality information in childcare benefits administration.

Source for this entry

Lessons for prevention

Review sensitive attributes and proxies, document lawful use and provide a practical route to challenge classifications.

Consequences if unresolved

Families can face disproportionate investigations and barriers to correcting erroneous records.

Sepsis model validation

· Validation study

Context and sources

An external validation at a US hospital system found that the Epic Sepsis Model performed poorly in that setting. The study measured prediction performance.

Source for this entry

Lessons for prevention

Validate in each intended setting and monitor alert burden and missed cases before relying on a score.

Consequences if unresolved

Missed warnings and excessive alerts can weaken clinical response.

2020

Calculated exam grades

· Policy reversal · automated predecessor

Context and sources

After exams were cancelled, Ofqual used calculated grades. Following loss of confidence, exam boards reissued grades using centre assessments where those were higher.

Source for this entry

Lessons for prevention

Test individual effects and design appeals before grades affect admissions.

Consequences if unresolved

Students can lose time-sensitive opportunities while disputed decisions are corrected.

Police facial recognition ruling

· Court judgment

Context and sources

In Bridges, the Court of Appeal found shortcomings in South Wales Police’s use of automated facial recognition, including legal discretion, impact assessment and equality duties.

Source for this entry

Lessons for prevention

Define watchlist and deployment limits, assess data protection, and investigate demographic bias.

Consequences if unresolved

Surveillance can expand without adequate rules for whom police scan or flag.

2019

Healthcare cost proxy

· Peer-reviewed audit

Context and sources

Researchers found that a US health-management algorithm used cost as a proxy for need, producing racial disparities in access to extra care.

Source for this entry

Lessons for prevention

Validate the target variable against the real outcome and compare allocation across groups.

Consequences if unresolved

Past unequal access can be reproduced in future offers of care.

2018

Biased recruiting prototype

· Reported prototype failure

Context and sources

Reuters reported that Amazon abandoned an experimental recruiting system after it showed bias against women in candidate scoring.

Source for this entry

Lessons for prevention

Check whether historical recruitment patterns are being learned and assess outcomes before deployment.

Consequences if unresolved

Training on past selection can carry old exclusions into new recommendations.

Fatal automated vehicle test

· Safety investigation

Context and sources

An automated Uber test vehicle struck and killed a pedestrian in Tempe. The NTSB found operator distraction and identified organisational and oversight contributors.

Source for this entry

Lessons for prevention

Test the combined system of vehicle, operator and supervision with clear intervention procedures.

Consequences if unresolved

Safety gaps can leave road users exposed to fatal failures.

2016

Disputed sentencing risk scores

· Disputed audit

Context and sources

ProPublica reported racial differences in COMPAS error rates and responded to the vendor’s critique. The disagreement included which fairness measures were appropriate.

Source for this entry

Lessons for prevention

Publish multiple fairness measures and specify which decisions a score may influence.

Consequences if unresolved

One headline metric can obscure differences in who bears the consequences of errors.

Tay chatbot abuse

· Provider acknowledgement

Context and sources

Microsoft took Tay offline after users induced it to post offensive content shortly after its public launch.

Source for this entry

Lessons for prevention

Anticipate coordinated abuse and test safeguards before exposing an interactive system to the public.

Consequences if unresolved

A public bot can amplify harassment and cause reputational harm within hours.

2015

Racist photo labelling

· Reported incident

Context and sources

Google apologised after Google Photos labelled an image of two Black people as gorillas.

Source for this entry

Lessons for prevention

Test offensive failure modes across groups and provide a simple path to report and correct labels.

Consequences if unresolved

Automated labelling can reproduce dehumanising stereotypes in personal and shared records.

2012

Knight Capital trading failure

· Regulatory finding · automated predecessor

Context and sources

A faulty software deployment sent unintended orders into US markets. The SEC later charged Knight Capital over inadequate market-access controls.

Source for this entry

Lessons for prevention

Verify deployment consistency and give operators independently tested stop controls.

Consequences if unresolved

Rapid automated execution can multiply financial losses before operators respond.

1999

Horizon accounting discrepancies

· Public inquiry · automated predecessor

Context and sources

Horizon was introduced from 1999. The public inquiry examined failures around the system and how shortfalls were pursued. The year marks rollout.

Source for this entry

Lessons for prevention

Preserve logs and investigate credible reports of discrepancies before imposing sanctions.

Consequences if unresolved

Inaccurate records and weak challenge routes can contribute to wrongful prosecutions.

1988

Medical admissions discrimination

· Historical automated predecessor

Context and sources

A UK medical-school admissions program at St George’s encoded discriminatory patterns, documented in the history of medical student selection.

Source for this entry

Lessons for prevention

Audit admission rules and outcomes for bias before and after automating them.

Consequences if unresolved

Past exclusion can become a repeatable filter applied to new applicants.

1984

MYCIN deployment barriers

· Historical research precursor

Context and sources

A retrospective on the Stanford MYCIN project discussed barriers to putting the medical expert system into clinical use. This entry marks the published retrospective.

Source for this entry

Lessons for prevention

Assess workflow, maintenance and responsibility alongside expert-system performance.

Consequences if unresolved

A promising research result can fail to translate into safe operational use.

1966

ELIZA conversational illusion

· Historical design precursor

Context and sources

Weizenbaum’s ELIZA paper describes a program that uses text transformations to simulate conversation. Its relevance here is the gap between conversational appearance and capability.

Source for this entry

Lessons for prevention

Explain how a system works and design the interface so users can judge its limits.

Consequences if unresolved

Users may place undue confidence in advice that sounds responsive.

1956

Dartmouth AI workshop

· Historical starting point

Context and sources

The 1955 proposal for the 1956 Dartmouth workshop set out a research programme for artificial intelligence. It provides the chronology’s starting point.

Source for this entry

Lessons for prevention

Turn ambitious claims into specific, measurable tests with explicit limits.

Consequences if unresolved

Vague claims can make it difficult to judge whether deployment is justified.