Human-in-the-Loop Review

1. Why Human-in-the-Loop Review

AI assessment without traceable human oversight is just a chatbot with a dashboard. Consultancies need to prove their review methodology. This feature makes the human review traceable, defensible, and auditable.

The Problem

When a consultancy delivers an IPA gateway review report, they need to demonstrate four things:

1. Every finding was reviewed by a qualified professional
2. Challenges were considered and either incorporated or explained
3. Professional judgement was applied — the AI did not simply produce a report that was rubber-stamped
4. The review process was systematic — not ad hoc

Without Programme Insights, a consultancy's review workbook is an Excel spreadsheet with criteria, RAG ratings, and short comments. No traceability of how ratings were determined. No evidence of systematic review methodology. Typically produced by 2–3 consultants over 2–3 weeks.

With human-in-the-loop review, the same workbook becomes a structured assessment with full evidence citations, a complete audit trail showing who reviewed what, when, and why, and traceable challenges with resolutions. Produced in 2–3 days with 1–2 reviewers using AI as the first-pass assessor.

The Escalation Model

Programme Insights follows an escalation-based model: the AI handles the assessment autonomously, and human reviewers focus on the exceptions — the findings that need professional judgement.

The Stanford Enterprise AI Playbook (March 2026), studying 51 enterprise deployments, found that escalation-based models deliver significantly higher productivity gains than approval-only models where humans must sign off every output.

In practice, the AI handles the 80%+ of criteria where the evidence is clear. The human reviewer focuses on the 15–20% that need professional judgement. The audit trail proves the methodology was applied.

2. The Six Interaction Types

Every finding in your assessment results is interactive. You can query it, challenge it, add context, accept it, dismiss it, or override it. Each action triggers a different system response and is recorded in the audit trail.

The Action Strip

Each criterion row in the assessment view has a persistent action strip on the right side:

• Query, Challenge, Add Context are always visible as buttons
• More dropdown contains Dismiss (Force Override only appears after a rejected challenge)
• Accept is a standalone checkmark toggle — most frequent action, needs lowest friction

3. The Review Interface

The review interface uses a side drawer pattern — the criteria list stays visible on the left while the interaction content opens on the right. You never lose context of what you are reviewing.

Why a Side Drawer

• Not a modal — you need to see the finding you are interacting with. A modal obscures context.
• Not inline expansion — interaction content (AI explanations, re-assessment results, audit history) can be lengthy. Expanding inline would push other findings out of view.
• Not a floating chat — interactions are anchored to specific findings, not free-form. This is structured professional review, not conversation.

Layout Overview

Drawer Content

When you click on any criterion, the drawer opens with three tabs:

Opening and Closing

• Click any criterion row to open the drawer with that finding's detail
• Click a different criterion to switch the drawer content without closing it
• Press Esc or click outside the drawer to close it
• The drawer slides in from the right (300ms ease-out transition)

4. Challenging a Finding

When you disagree with a finding, challenge it. The system re-assesses with your input and either changes the rating or explains why the original stands. Both outcomes are recorded.

Step-by-Step Walkthrough

1. You are scanning the criteria browser. You see GR-3.1: Robust cost estimation methodology rated RED. Based on your experience, you believe this should be AMBER.
2. Click the Challenge button on the GR-3.1 row. The side drawer slides open with the challenge form pre-focused.
3. Select your proposed rating from the dropdown: AMBER. Write your justification in the text area (minimum 20 characters — forces substantive reasoning).
4. Click Submit Challenge. The button changes to a spinner: "Re-assessing with your input... (15–30s)". The system runs a targeted re-assessment of that single criterion, injecting your challenge text as additional context.
5. Result appears. One of two outcomes:
   • Rating changed: Banner shows "Rating updated: RED → AMBER. See explanation below." The new justification appears with your context highlighted.
   • Rating unchanged: Banner shows "Rating remains RED after re-assessment. See explanation below." The AI explains why the original rating holds despite your input.

When the System Disagrees

This is the most important UX moment. The system does not simply say "no". It:

1. Acknowledges your point — never dismisses it
2. Explains the specific evidence that supports the original rating
3. Distinguishes scope — often your concern is valid but applies to a different criterion
4. Offers alternatives — "Your concern about independent verification may be better captured under criterion GR-3.4. Would you like to review that criterion?"

What Happens Under the Hood

The re-assessment does not re-run the full assessment pipeline (that would take 2–5 minutes). It runs a targeted 3-step process:

5. Force Override

When the AI disagrees with your challenge but your professional judgement takes precedence, Force Override is your final authority. It is the safety valve that ensures human expertise always wins.

When It Becomes Available

Force Override is not available as a first action. It only appears after a challenge has been rejected by the system. This sequence is deliberate:

This means every override has context: you challenged, the system explained why it disagreed, and you still chose to override. The full chain is preserved.

How to Override

1. After your challenge is rejected, the Force Override button appears below the system's response in the drawer.
2. Click Force Override. A form appears with a dropdown for your chosen rating and a justification text area.
3. Write your justification — minimum 50 characters. This must explain your professional reasoning. One-word justifications are not accepted.
4. Click Confirm Override. The rating changes immediately to your specified value.

What Changes

• The criterion's RAG badge updates to your chosen rating
• A Human Override badge appears on the finding with a distinct visual marker
• Category and overall ratings recalculate to reflect the change
• The full chain is recorded in the audit trail: AI rating → your challenge → AI rejection reasoning → your override with justification

Rules and Limits

6. The Audit Trail

The audit trail is not a compliance checkbox. For consultancies delivering IPA gateway reviews, the audit trail IS the professional work product. It proves systematic review methodology was applied.

What Gets Recorded

Every interaction records the following data:

The Audit Timeline

In the Audit tab of the side drawer, interactions appear in reverse chronological order. Here is an example for criterion GR-3.1:

Why It Matters

The interaction audit trail provides something consultancies have never had before: perfect traceability of the review process. When a client or IPA assessor asks "how did you arrive at this rating?", the answer is not a verbal explanation — it is a timestamped record of every query, challenge, resolution, and sign-off.

7. Review Progress and Analytics

Track your review progress at a glance. Visual indicators on every criterion show its interaction state, and a progress bar per category tells you how much of the assessment has been reviewed.

Progress Bar

A review progress bar sits at the top of each category in the criteria browser:

The progress bar gives you a sense of completion and helps you track where you left off between sessions.

Visual Indicators Per Criterion

Each criterion row in the browser shows its current interaction state with an icon:

Keyboard Shortcuts

When a criterion is focused in the browser, you can work through your review without touching the mouse:

Batch Review

For categories where most criteria are straightforward, batch review saves time:

1. Toggle Review Mode at the top of the criteria browser. Checkboxes appear on each criterion row.
2. Select multiple criteria by checking boxes, or use Select All in Category.
3. Click Accept All Selected to stamp all as reviewed in a single action, with bulk audit entries created.

You can also use sequential review mode: arrow keys or the "Next" button in the drawer advances to the next criterion without closing it. Work through an entire category without mouse interaction: review, accept (Enter), next (arrow), review, challenge (C), type, submit, next.

8. How It Flows Into Reports

Your review interactions do not disappear into a database. They appear in three distinct places in your PDF reports, turning AI assessment into a documented professional review.

1. Inline Reviewer Notes on Findings

Each finding that has been reviewed shows a Reviewer Notes section beneath the assessment justification in the Full Assessment report:

2. Review Summary Section

A new section appears between the Executive Summary and Detailed Findings in all report types:

3. Full Interaction Log (Premium Reports)

For assurance-grade reports, an appendix lists every interaction with full text. This is the work product that consultancies attach to their deliverables — the equivalent of a traditional review workbook, but with perfect traceability.

The appendix includes:

• Every interaction, chronologically ordered, per criterion
• Full text of reviewer comments and AI responses
• Rating change history with before/after values
• Override justifications with the full challenge-rejection-override chain
• Dismiss reasons for excluded criteria

9. Getting Started

A practical checklist to work through your first review. Start with the category that matters most and work outward.

1. Open your assessment results. Navigate to the assessment you want to review from your dashboard. The criteria browser shows all findings grouped by category.
2. Start with the category that matters most. If you are preparing for an IPA gateway, start with the category most likely to draw scrutiny. The progress bar at the top of each category shows 0% reviewed — this is your starting point.
3. Work through criteria: accept the clear ones, query the uncertain ones. For findings where the AI's assessment matches your view, hit Enter to accept and move on. For findings where you want to understand the reasoning, press Q to query.
4. Challenge findings you disagree with — provide evidence. When you see a rating that does not match your professional judgement, press C to challenge. Select your proposed rating and write your justification. The more specific your reasoning, the better the re-assessment.
5. Use Force Override only when your professional judgement demands it. If the system rejects your challenge and you still disagree, override it. Write a clear justification — this appears in the audit trail and reports.
6. Check the progress bar — aim for 100% reviewed. A fully reviewed assessment produces the strongest reports. The progress bar tracks your completion per category.
7. Generate your report — the audit trail appears automatically. When you generate a PDF report, all reviewer notes, the review summary, and the interaction log (premium reports) are included without any extra steps.

Quick Reference: Action Summary