How the Assessment Pipeline Works

Introduction

Why structured assessment matters — and why "AI read your document" is not the same as evidence-based assessment with guardrails.

Most AI document tools work like this: they feed your document into a language model, ask it to evaluate against some criteria, and return whatever the model says. No verification. No structure. No way to know if the answer is grounded in evidence or fabricated from training data.

That approach fails for serious assessment work. When a programme director needs to know whether their business case addresses all five HMT Green Book dimensions, or whether an IPA gateway submission covers the required delivery confidence criteria, "the AI thinks it looks fine" is not an acceptable answer.

What Makes This Different

Programme Insights uses a multi-step assessment pipeline that separates evidence retrieval from judgement, applies deterministic guardrails over AI scoring, and optionally runs adversarial verification on borderline results. Every rating traces back to specific passages in your documents.

This guide explains each step of that pipeline: what it does, why it makes assessment more accurate, and what goes wrong without it. It is written for technical evaluators who need to understand the mechanism, not just the marketing.

Pipeline Overview

Three layers work together: orchestration for reliability, standard assessment for every criterion, and extended verification for high-stakes results.

Architecture at a Glance

Layer	Steps	Purpose
Orchestration	10 steps	Run management, crash recovery, progress streaming, criteria loading
Standard Tier	2 steps per criterion	Evidence retrieval + scoring with deterministic guardrails
Extended Tier	4 additional steps	Corrective RAG, citation verification, self-critique, adversarial debate

Three LLM Tiers

Model	Used For	Why This Model
GPT-4o	Scoring, summary, debate	Strongest reasoning for judgement tasks. Temperature 0.0 for reproducibility
GPT-4.1-mini	CRAG, critique, citation verification	Fast, cost-effective for classification and verification tasks
text-embedding-3-small	Vector embeddings	High-quality semantic search at low latency

Orchestration Layer

The infrastructure that ensures assessments are reliable, resumable, and observable — before a single criterion is evaluated.

An assessment run against a complex framework can take several minutes. Documents may be large, criteria sets can include 50+ items, and network or API failures happen. The orchestration layer handles all of this so that the assessment logic never has to.

1. Duplicate Run Prevention A 10-minute window prevents the same assessment from running twice. If a user accidentally double-clicks, the second request is rejected. This prevents wasted compute and conflicting results.
2. Stale Run Resumption If a previous run crashed mid-assessment, it is detected and resumed rather than restarted. Completed criteria are kept; only the remaining criteria are re-assessed. This turns a 15-minute failure into a 2-minute recovery.
3. Create Assessment Run Record An atomic version number is assigned. Every result in this run links back to this record, creating a complete audit trail. You can compare run v3 against run v1 and see exactly what changed.
4. Initialise Progress Tracker Server-Sent Events (SSE) stream real-time progress to the UI. Users see each criterion being assessed as it happens, not a spinner followed by a data dump.
5. Load Criteria The system routes to the correct criteria source: a pre-built module pack (IPA, Green Book, NEC), user-defined criteria, or a custom framework. It builds a search configuration with semantic weight 0.7, keyword weight 0.3, and a relevance threshold of 0.35.

After Assessment Completes

Steps 7-8 handle post-assessment work. The Summary Generator creates a Delivery Confidence Assessment (DCA) rating and an Executive Summary by aggregating individual criterion scores. The Finalise Run step marks the run as complete, records timing metrics, and triggers any downstream notifications.

Steps 9-10 provide structured error handling: per-criterion timeouts (300s standard, 480s extended), graceful degradation when individual criteria fail, and comprehensive error logging for debugging.

Standard Tier: Retrieval

Step 1 of every criterion assessment. Finding the right evidence is harder than judging it — and most AI tools get this wrong.

The Problem with Single-Query Search

If you ask an LLM "does this document address stakeholder engagement?", it searches for those exact words. But the evidence might be described as "community consultation", "public participation", or "interested party management". A single query misses evidence that exists but uses different language.

How Programme Insights Retrieves Evidence

1. Query Expansion

For each criterion, the system generates 2-3 query variants that capture different ways the same concept might be expressed. A criterion about "risk management" might expand to queries about "risk register", "mitigation strategies", and "threat assessment".

2. Hybrid Search

Each query variant runs against two search systems simultaneously:

Search Type	Technology	What It Catches	What It Misses
Semantic	pgvector embeddings	Conceptually similar text, paraphrases, domain synonyms	Exact terms, clause numbers, acronyms
Keyword	Full-text search	Precise terms, regulatory references, specific numbers	Rephrased concepts, alternative terminology

3. Reciprocal Rank Fusion (RRF)

Results from all query variants and both search types are combined using RRF. This algorithm merges ranked lists by giving higher scores to chunks that appear in multiple result sets. A chunk that ranks #3 in semantic search and #5 in keyword search gets a combined score higher than a chunk that ranks #1 in only one system.

4. Output: 15 Ranked Chunks

The retrieval step produces 15 ranked evidence chunks, each with a relevance score, source document reference, and page location. These chunks — not the full documents — are what the scoring step evaluates.

Standard Tier: Scoring

Step 2 of every criterion assessment. The AI scores the evidence — but deterministic rules have the final word.

Structured Output at Temperature 0.0

GPT-4o evaluates the 15 retrieved evidence chunks against the criterion using structured output mode — the model returns a defined JSON schema, not free text. Temperature is set to 0.0, meaning the same evidence produces the same score every time. No creative interpretation. No hallucinated confidence.

Dual Rating System

For each criterion, the model produces two independent ratings:

Rating	What It Measures	Scale
Coverage	Does the evidence address the criterion? How thoroughly?	RED / AMBER / GREEN
Delivery Confidence	Is the approach credible? Are the plans realistic and achievable?	RED / AMBER / GREEN

Coverage answers "is it there?" Delivery confidence answers "is it good?" A business case might thoroughly cover financial projections (GREEN coverage) but with unrealistic assumptions (RED delivery confidence). These are different problems requiring different responses.

Analyst Personas

The scoring prompt instructs GPT-4o to adopt the role of a specialist assessor appropriate to the framework being used. For IPA gateway assessments, it acts as an experienced IPA reviewer. For Green Book appraisals, it acts as a Treasury-trained economist. This persona framing improves the relevance and rigour of the assessment without changing the deterministic guardrails that override the AI's output.

Deterministic Guardrails: Scoring Rules

The system does not trust the AI. It verifies. These rules are hard-coded — no prompt engineering can override them.

After GPT-4o produces its structured score, the result passes through a set of deterministic rules. These are not AI suggestions. They are if/then logic that runs in application code. The AI cannot talk its way past them.

Sub-Question Coverage Calculation

Each criterion can be broken into sub-questions. The system scores each sub-question individually, then calculates a coverage percentage:

Sub-Question Result	Score Assigned	Example
Yes — fully addressed	100%	Risk register exists with mitigations and owners
Partial — mentioned but incomplete	70%	Risks listed but no mitigation strategies
No — not addressed	0%	No mention of risk management

RAG Rating Thresholds

The sub-question scores are averaged into a coverage percentage, which maps to a RAG rating:

Coverage %	Rating	Meaning
70% and above	GREEN	Criterion is well addressed. Evidence is comprehensive.
40% – 69%	AMBER	Partially addressed. Gaps exist that need attention.
Below 40%	RED	Not adequately addressed. Significant work required.

Deterministic Guardrails: Override Rules

Four rules that can override the AI's rating. They catch the cases where LLMs are systematically wrong.

LLMs have known failure modes in assessment work: they are overconfident about vague text, they miss the difference between aspirational language and concrete plans, and they sometimes rate highly when retrieval found very little evidence. These four guardrails catch those failures.

Worked Example

Extended Tier: Corrective RAG

When the retrieval step fails to find enough evidence, CRAG diagnoses why and tries again with better queries.

The Problem

Even with query expansion and hybrid search, retrieval can fail. The document might use highly specialised terminology, the relevant section might be embedded in an appendix, or the criterion might span multiple disconnected sections. The standard retrieval step does not know whether it succeeded — it returns 15 chunks regardless of quality.

How CRAG Works

CRAG (Corrective Retrieval-Augmented Generation) adds a quality check after retrieval. GPT-4.1-mini classifies the retrieval result into one of three categories:

Classification	What It Means	What Happens Next
Correct	Retrieved chunks are relevant and sufficient	Proceed to scoring with current evidence
Ambiguous	Some relevant chunks, but gaps exist	Decompose criterion into sub-queries, re-retrieve for gaps
Incorrect	Retrieved chunks are mostly irrelevant	Generate entirely new queries, re-retrieve from scratch

Sub-Query Decomposition

When classified as ambiguous, CRAG breaks the criterion into more specific sub-queries. For example, a criterion about "project governance arrangements" might decompose into:

• Governance structure and roles Search for: board composition, reporting lines, decision authority, RACI matrix
• Meeting and review cadence Search for: board meetings, gateway reviews, stage-gate process, review schedule
• Escalation and assurance Search for: escalation procedures, independent assurance, audit arrangements

Each sub-query runs through the same hybrid search pipeline. The results are merged with the original retrieval using RRF, producing a richer evidence set for scoring.

Extended Tier: Citation Verification

Every claim the system makes must trace back to your document. This step ensures it does.

The Hallucination Problem

Stanford research found that LLMs hallucinate in 17-33% of legal document analysis tasks. The model generates text that sounds authoritative but does not appear in the source material. In assessment work, this means a criterion might be rated GREEN based on evidence the AI invented rather than evidence your document contains.

This is not a theoretical risk. It happens routinely with off-the-shelf AI tools applied to document assessment.

How Citation Verification Works

After scoring, GPT-4.1-mini performs a verbatim check. For each claim in the assessment output, it:

1. Extracts the claim Identifies each factual assertion in the scoring output — "the document includes a cost-benefit analysis covering a 25-year appraisal period"
2. Locates the source chunk Matches the claim to the specific retrieved chunk it should be grounded in
3. Verifies the text Checks whether the claim is substantiated by the actual text of the source chunk — not a paraphrase of training data
4. Flags or confirms Claims that cannot be verified are flagged. If enough claims fail verification, the criterion score is adjusted downward

The Audit Trail

Every assessment result includes a citation chain: criterion → rating → supporting evidence → source document → page number. A reviewer can follow any rating back to the exact text that supports it. If the citation verification step flagged any issues, those flags are visible in the results.

Extended Tier: Self-Critique

When the system is uncertain, it challenges its own reasoning before committing to a score.

When Self-Critique Activates

Self-critique triggers when the scoring model's confidence falls below 70%. This is not a random threshold — it corresponds to the boundary where LLM scoring reliability drops measurably. Above 70%, the model's structured output is generally consistent and well-grounded. Below 70%, the model is uncertain, and its rating becomes less reliable.

The Three-Question Probe

When triggered, GPT-4.1-mini asks the scoring output three diagnostic questions:

What Happens After

If the self-critique identifies issues, the scoring step is re-run with the critique findings included as additional context. The model sees its own weaknesses and can correct for them. If the re-scored result still has low confidence, the criterion is flagged for human review in the final report.

Extended Tier: Adversarial Debate

AMBER results get stress-tested by specialist personas arguing opposing positions. The boundary between AMBER and GREEN (or AMBER and RED) is where assessments matter most.

Why AMBER Is the Danger Zone

GREEN and RED ratings are usually unambiguous. The evidence clearly addresses the criterion, or it clearly does not. AMBER is where the difficult judgement calls live: is this "partially addressed" or "nearly complete"? Is this a genuine gap or a terminology mismatch?

These boundary decisions are exactly where human assessors disagree with each other, and where AI assessors are least reliable. The debate step forces the system to stress-test its AMBER ratings before finalising them.

How the Debate Works

GPT-4o runs a multi-persona adversarial debate. The personas depend on the framework:

Framework	Debate Panel	Perspectives
IPA Gateway	3 IPA specialists	Delivery confidence, commercial viability, governance & assurance
General frameworks	2 generalists	One argues for upgrade (GREEN), one argues for downgrade (RED)

Debate Structure

1. Position Statements Each persona states their position on the criterion rating, citing specific evidence from the retrieved chunks. No general opinions — every argument must reference text from the document.
2. Cross-Examination Each persona challenges the others' arguments. "You cite paragraph 4.3, but it only describes the current state, not the planned approach. That is not evidence of a delivery plan."
3. Judgement A synthesiser weighs the arguments and produces a final recommendation: stay AMBER, upgrade to GREEN, or downgrade to RED. The reasoning is included in the assessment output.

Summary Generation & DCA

Individual criterion scores roll up into a Delivery Confidence Assessment and an Executive Summary that tells the story behind the numbers.

Dimension Grouping

Criteria are not treated as a flat list. They are grouped into dimensions — the higher-level categories that a framework uses. For IPA assessments, these might be: Strategic Case, Economic Case, Commercial Case, Financial Case, and Management Case. For custom frameworks, the grouping comes from the module pack or user-defined structure.

Each dimension gets its own summary score, derived from its constituent criteria. This reveals patterns: a business case might score GREEN on Strategic and Economic cases but RED on Management Case, signalling a specific weakness rather than overall failure.

Showstopper Detection

Some criteria are flagged as showstoppers in the module pack. These are non-negotiable requirements where a RED rating means the assessment cannot receive an overall GREEN regardless of how well other criteria score. Examples include:

• Absence of a benefits realisation plan in a gateway review
• No evidence of an approved budget in a business case
• Missing safety case in nuclear or high-hazard assessments

DCA Rating

The overall Delivery Confidence Assessment combines dimension scores, showstopper flags, and a holistic judgement. It follows a rules + holistic model:

Incremental Assessment

When your documents change, the system re-assesses only what is affected — not everything from scratch.

The Evidence Lineage Problem

Documents evolve between assessment runs. A section gets rewritten, new appendices are added, figures are updated. A full re-assessment every time is wasteful and makes it impossible to track what actually changed. You need to know: "this criterion moved from AMBER to GREEN because the stakeholder engagement section was expanded", not just "it is GREEN now".

How Incremental Assessment Works

Evidence Links Table

The system maintains an evidence_links table that records which document chunks contributed to which criterion scores. When a document is re-uploaded, the system identifies which chunks changed by comparing embeddings and content hashes.

Carry-Forward Check

At the start of each criterion assessment in a new run, the system checks: has any evidence linked to this criterion changed since the last run? If not, the previous score carries forward without re-assessment. This is step 6 of the orchestration layer — the carry-forward check runs before the agentic pipeline.

Targeted Re-Assessment

Only criteria with changed evidence go through the full pipeline (retrieval, scoring, guardrails, and optionally the extended tier). The result is a new run that includes both carried-forward scores and freshly assessed scores, all under a single version number.

Scenario	Full Re-Assessment	Incremental
50-criterion framework, 3 sections changed	~25 minutes, 50 LLM calls	~4 minutes, 8 LLM calls
Version comparison	Compare two full runs manually	System highlights exactly what changed and why
Audit trail	Two independent snapshots	Linked lineage showing score progression per criterion

How This Compares

Not all AI document assessment is created equal. Here is how Programme Insights compares to the alternatives.

Capability	Programme Insights	ChatGPT / Generic AI	Manual Assessment	Checkbox Tools
Evidence retrieval	Hybrid search + query expansion + RRF	Single-pass context window	Human reading	Keyword matching
Scoring method	Structured output + deterministic guardrails	Free-text opinion	Expert judgement	Binary pass/fail
Hallucination control	Citation verification + source tracing	None	N/A (human reads source)	None
Borderline handling	Adversarial debate with specialist personas	None	Panel discussion (if available)	None
Reproducibility	Temp 0.0 + fixed guardrails = same input, same output	Variable (temperature, prompt drift)	Low (assessor variability)	High (but shallow)
Audit trail	Criterion → evidence → source → page	None	Written report (if created)	Checkbox record
Incremental updates	Evidence lineage, targeted re-assessment	Full re-run required	Full re-read required	Full re-check required
Time for 50 criteria	8-15 minutes	Not feasible (context limits)	3-5 days	1-2 hours (shallow)
Framework awareness	IPA, Green Book, NEC, nuclear, custom	General knowledge only	Depends on assessor	Template-dependent
Cost per assessment	Minutes of compute	Subscription + time	Consultant day rates	Licence + time