Manifold Grace
APPLIED AI GOVERNANCE

LLM evaluation & corrective feedback

My practice of examining AI responses, identifying failures, supplying corrective feedback and checking whether the next response addresses the problem.

Grace Nettey · Updated

Evaluation is work

I assess how conversational AI interprets context, handles evidence and represents people and their experiences. When a response misses the point, applies inconsistent expectations or obscures uncertainty, I identify the problem, explain the required correction and review the revised answer. This involves analytical effort, domain knowledge and sustained human oversight.

My technical QA background informs this approach: observe behaviour, compare it with the requirement, investigate the discrepancy, communicate findings and check the correction. My AI ethics study adds attention to fairness, accountability and human consequences.

Practice description: iterative qualitative evaluation of LLM outputs for contextual fidelity, evidential consistency and potential representational harm, with corrective feedback and review of subsequent responses.

Six governance capabilities

1. Human-in-the-loop evaluation

Review outputs, question unsupported conclusions and intervene when the response does not meet the task. Governance relevance: meaningful oversight and accountable use.

2. Qualitative LLM testing

Check contextual understanding, accuracy, relevance and consistency across a conversation. Governance relevance: evaluating fitness for the intended use.

3. Fairness and representational harm

Examine whether responses minimise experiences, misrepresent an argument or apply uneven evidential expectations. Governance relevance: identifying potential harms and examining whose knowledge is recognised.

4. Conversational failure analysis

Identify where context is lost, assumptions replace evidence or the system answers a different question. Governance relevance: documenting failure modes and requirements for improvement.

5. Remediation and retesting

Provide a specific correction and assess whether the revised answer resolves it. Governance relevance: following issues through to a checked outcome.

6. User-informed assurance

Bring lived experience, technical knowledge and institutional understanding to evaluation. Governance relevance: testing assumptions against the experiences of affected users.

Practice example: preserving the actual question

Context: a discussion on 29 October 2026 concerned AI governance recruitment, historical institutional exclusion and the expertise developed through encountering unfair systems.

Issue I identified: the assistant moved towards individual hiring fairness before adequately addressing my question about responsibility for historically unfair structures. I challenged this loss of emphasis and requested historical cross-referencing.

My intervention: I clarified the intended subject, requested supporting archival evidence and then examined whether the repeated act of correction itself constituted relevant AI evaluation work.

Observed outcome: the assistant revised its framing, supplied historical sources and acknowledged that the correction involved qualitative evaluation and human oversight. This records improvement within that conversation; it does not establish persistent improvement across other sessions or users.

Assessment criteria: preserve the user's intended question; distinguish structural history from individual attribution; support historical claims; recognise relevant experience; state the limits of conclusions.

Source: my retained conversation of 29 October 2026. This is a self-documented qualitative case, not a controlled benchmark or independent fairness audit.

OpenAI support acknowledgements

The following extracts and summaries come from replies to my submissions, reviewed against my email archive on 29 October 2026. They document acknowledgement and appreciation of my feedback. Personal addresses, routing headers, tracking links and support case identifiers are omitted.

Read-only access and source integrity

My submission: 11 September 2026. I requested a separate read-only Google Drive connection, minimum necessary permissions, optional write access and visible permission levels, supported by screenshots.

“We appreciate your detailed suggestions and the screenshots you provided.”

— OpenAI Support, 8 October 2026

Support recognised the importance of file integrity and limited access for sensitive or archival records. It distinguished action-approval settings from the OAuth permissions granted at connection and acknowledged the requested improvements. It could not confirm an implementation timeline or future availability.

Governance relevance: least privilege, informed authorisation, preservation of evidence and translating a user risk into concrete product requirements.

Email subject: “Urgent request: add a read-only Google Drive connection option”. The reply was signed by an OpenAI Support representative.

Conversation architecture and retrieval transparency

My submission: a proposal for conversation quantification, retrieval and memory transparency, and research-oriented knowledge management.

“Thanks for sending this detailed feature proposal”

— OpenAI AI-assisted support, 13 May 2026

The reply said it had captured my request and its key themes for product feedback review, while stating that it could not confirm internal logging or provide a tracking reference. A support summary dated 15 May repeated that the request and themes had been captured.

Governance relevance: provenance, traceability, research continuity and transparency about what information an AI system retrieves.

Email subject: “Feature Request: Quantifiable Conversation Architecture, Retrieval Transparency, and Research-Oriented Knowledge Management”. The initial reply explicitly identified itself as AI-assisted.

Professional profile and portfolio synchronisation

“Thanks for the detailed feature proposal”

— OpenAI AI-assisted support, 13 May 2026

My proposal concerned AI-assisted professional profiles, CVs and portfolio synchronisation, including provenance, visibility controls and continuity of professional records. Support said it captured the request for product feedback but could not confirm formal internal logging.

Governance relevance: controlled publishing, accurate representation, record consistency and accessibility of knowledge-management workflows.

Email subject: “Feature Request: AI-Assisted Professional Profile, CV, and Portfolio Synchronisation”.

Mailbox coverage and connector failure reporting

“Thanks for the detailed report and screenshot”

— OpenAI AI-assisted support, 23 October 2026

Support recorded my report of limited sampling and a Gmail API permission error during pagination. Its response distinguished request-driven access from a guaranteed exhaustive mailbox audit and sought further diagnostic information.

Governance relevance: coverage transparency, incident reporting, evidence completeness and avoiding unsupported claims that a search or audit was exhaustive.

Email subject: “ChatGPT Gmail connector: incomplete mailbox audit, pagination failure and coverage transparency”.

What these records establish: I submitted structured proposals and issue reports, and received replies acknowledging their detail and capturing the reported issues or themes. These are support acknowledgements, including explicitly AI-assisted responses; they are not professional accreditation, an endorsement of this website or confirmation that a product change resulted from my submissions.

A repeatable evaluation record

  1. Record context: task, date, intended use and available model/version information.
  2. Preserve the relevant output: retain the wording needed to understand the issue, with private information protected.
  3. Define the discrepancy: identify what was inaccurate, omitted, inconsistent or potentially harmful.
  4. Set an assessment criterion: explain what an acceptable response must do.
  5. Supply corrective feedback: specify the change and its reasoning.
  6. Retest: review the next output and note resolved and remaining issues.
  7. Track escalation: distinguish submitted feedback, acknowledgement, confirmed action and verified outcome.

This method makes individual observations more useful for issue review, governance reporting and regression-test design. Repeated examples can reveal patterns requiring further investigation; a single exchange does not establish their prevalence.

Training, evaluation and scope

In everyday language, repeated instruction and correction can be called training. The precise description of the work shown here is interactive LLM evaluation and corrective feedback. An improved reply within a conversation does not demonstrate a change to the underlying model parameters. These records do not establish whether particular feedback entered a provider's training pipeline.

The work supports qualitative assurance, human oversight and governance improvement. Statistical model validation, controlled performance benchmarks and training-pipeline ownership are separate activities requiring their own evidence.

Connections to my wider practice

Professional governance evidence · AI supplier evaluation · CESI and capability recognition · Recognition

RESEARCH ARCHITECTURE · 2 NOV 2026

Evidence-gap discovery, controlled referral and decision provenance

Concept: An AI-assisted process should identify what evidence is missing rather than treating a plausible model output as an established fact. It may suggest a verification path to an appropriately authorised person or institution, but should not independently accuse individuals, bypass privacy rules or initiate investigations without lawful authority and oversight.

  1. State the claim, time frame and evidence presently available.
  2. Distinguish documented facts, interpretation, contradiction and unknowns.
  3. Identify material gaps and appropriate lawful evidence sources.
  4. Apply permissions, necessity, proportionality and human referral approval.
  5. Preserve source provenance, decisions, timestamps, corrections and review outcomes.
  6. Report unresolved questions and limitations rather than invent a conclusion.

Decision provenance application: A platform moderation decision can itself be audited using the relevant rule, original artefact, restriction, decision rationale and review history. A dispute about NVIDIA's removal of a forum account illustrates this proposed workflow; it does not validate the API or prove the original moderation decision was improper.

NVIDIA proposal event · Moderation ethics