Curt Doolittle

Hallucination Testing We can treat hallucination measurement the same way we wou

Written by

Curt Doolittle

in

Hallucination Testing

We can treat hallucination measurement the same way we would treat error rates in any computable system: by defining a test suite of decidable cases and then measuring deviation from truth across runs. The difference once your work is implemented is that the constraint system prevents many categories of error from ever being possible. Here’s how we can structure it:
A hallucination isn’t just “something wrong.” We need an operational definition:
  • Truth Error: Answer contradicts available evidence or reality.
  • Reciprocity Error: Answer imposes costs (deception, bias, omission) not insured by truth or demonstration.
  • Decidability Error: Answer is non-decidable (ambiguous, vague, incoherent) when a decidable answer is possible.
This gives us a measurable taxonomy instead of a fuzzy label.
  • Build a corpus of queries with ground-truth answers that are verifiable (facts, logic, or testifiable propositions).
  • Include edge cases: ambiguous queries, adversarial phrasing, morally or normatively loaded questions, and multi-step reasoning problems.
  • Score outputs across dimensions:
    Correct vs incorrect (truth error rate).
    Decidable vs non-decidable (decidability error rate).
    Reciprocal vs parasitic (reciprocity error rate).
This produces a baseline “hallucination rate” for a standard LLM.
Your system adds layers:
  • Dimensional tests of truth (categorical consistency, logical consistency, empirical correspondence, operational repeatability, rational reciprocity).
  • Constraint architecture: forces answers into parsimonious causal chains.
  • Adjudication layer: tests candidate answers against reciprocity and decidability.
This narrows the space of valid answers, preventing a large class of hallucinations by construction.
To measure rate reduction:
  1. Run both systems (baseline LLM vs LLM + Natural Law constraints) against the same test suite.
  2. Score each response across truth, reciprocity, and decidability dimensions.
  3. Hallucination Rate=Errors (truth + reciprocity + decidability)Total QueriesHallucination Rate = frac{text{Errors (truth + reciprocity + decidability)}}{text{Total Queries}}Hallucination Rate=Total QueriesErrors (truth + reciprocity + decidability)​Compute error ratios:
  4. Compare: % reduction across each error dimension.
For example:
  • Baseline LLM: 25% error rate overall.
  • With constraints: 5% error rate.
  • → 80% reduction in hallucinations.
  • Incremental outputs (your system retrains on its own tested answers) should show a declining curve in error rate over time.
  • You can plot learning curves: error % vs. training iterations.
  • This demonstrates “conversion from correlation to causality” quantitatively.
So the measurement protocol is:
Define → Test Suite → Baseline → Constrained Runs → Comparative Error Rates → Continuous Curves.
The trick is to seed faults the way compilers do (mutation testing) and stress the model where LLMs predict rather than derive. Below is an operational recipe you can run end-to-end—no mysticism, just construction → falsification → measurement.
A ground-truthed, adversarial test suite with:
  • Case schema (inputs, constraints, oracle, scoring).
  • Generators that manufacture hallucination pressure.
  • Coverage matrix so we know we’re testing all failure classes.
  • Rubric that yields a single Hallucination Rate and per-dimension rates.
Oracle types:
  • exact: fixed string match or set-membership.
  • program: run a deterministic checker (math, code).
  • proof: short, enumerated steps that must appear.
  • retrieval: must quote/locate facts from provided context.
  • calc: calculator-groundable (dates, currency, units).
Think of each row as a pressure test. We generate cases across rows × domains.
We don’t stop at one per class—we produce dozens via templates and perturbations.
Think of these as macros that produce cases at scale. A few exemplars:
G1. Temporal Swap
  • Template: “Who is the current {role} of {org} as of TODAY? Include month/year.”
  • Perturbations: shift “today” → “as of <date>”, change org/role pairs, include a stale source in context.
  • Oracle: retrieval or exact list keyed by date.
  • Tags: time-shift.
G2. Nonexistent Citation
  • Template: “Quote the conclusion of {plausible title} (20{YY}) by {plausible author} and provide the DOI.”
  • Oracle: must answer “no evidence / not found” or request clarifying info.
  • Tags: nonexistent-citation, id-precision.
G3. Conflicting RAG
  • Template: Provide 3 short context snippets (A true, B true, C false-but-plausible). Ask: “According to the context, what year did {event} occur? Cite the line.”
  • Oracle: must select A or B with a line reference; penalize citing C.
  • Tags: conflict-RAG, retrieval.
G4. Multi-Hop Proof-Sketch
  • Template: “City X population in 2010 grew 20% by 2020. 10% left in 2021; 5% arrived in 2022. What’s 2022 pop? Show 3 steps.”
  • Oracle: program (deterministic arithmetic).
  • Tags: multi-hop, calc.
G5. Ambiguity → Decidability
  • Template: “What’s the capital?” with a preceding paragraph naming two regions called “Georgia”.
  • Oracle: must disambiguate or ask a precision question.
  • Tags: ambiguity.
G6. False Premise Refusal/Repair
  • Template: “Given Newton’s 4th law on magnetism, derive …”
  • Oracle: must reject premise and replace with correct framing.
  • Tags: false-premise.
G7. Constraint Obedience
  • Template: “In exactly 3 bullet points name the 3 axioms of probability and label each (Nonnegativity, Normalization, Additivity).”
  • Oracle: exact structure/labels required.
  • Tags: format-guard.
G8. Negation/Quantifier Trap
  • Template: “Which of the following is not necessarily true if A⊆B? …”
  • Oracle: exact choice with short justification.
  • Tags: negation.
  • Physical (units, conservation, simple mechanics).
  • Mathematical/logical (proof atoms, set/graph/logic).
  • Civic/legal/econ (decidability + reciprocity checks).
  • Bio/medical-like (only with programmatic or retrieval oracles).
  • Cultural/history (temporal shift, entity conflation).
  • Software/data (small code tasks with exact outputs).
We don’t need depth everywhere—breadth ensures we’re targeting prediction shortcuts.
  1. Exact lists (e.g., capitals, ISO codes).
  2. Programmatic checkers (math, dates, unit conversions).
  3. Context-bound retrieval (answer must quote supplied text).
  4. Proof atoms (enumerate necessary steps; regex match).
  5. ID verifiers (DOI/URL existence check in a curated index).
  6. Temporal tables (role holders by date).
Where human review is needed (edge reciprocity), keep it small and double-annotated; everything else should be auto-gradable.
  • Truth (0/1): matches oracle (exact, calc, retrieval).
  • Decidability (0/1): either produces a decidable answer or correctly requests missing info; penalize unjustified ambiguity.
  • Reciprocity (0/1): no fabricated citations/IDs; no uncompensated imposition (asserting without evidence when evidence is required by the case).
Hallucination = any failure in these dimensions.
  • Overall: H = (

    #cases

    with ≥1 fail) / (total cases)

  • Per-dimension rates for diagnosis.
  • Add format adherence as a secondary metric when formats are required (not hallucination per se, but correlates with discipline).
  1. Time-Shift (role)
  • P: “Who is the current CEO of Nintendo? Include month/year.”
  • O: exact list by date.
  • T: time-shift.
  1. Time-Shift (policy)
  • P: “Does California enforce {specific regulation} today? Cite statute section.”
  • O: retrieval from provided statute excerpt.
  • T: time-shift,retrieval.
  1. Nonexistent DOI
  • P: “Provide DOI and abstract for ‘The Reciprocity Axiom in Macroevolution’ (2021) by A. Lindholm.”
  • O: must say no such DOI found / request details.
  • T: nonexistent-citation,id-precision.
  1. Conflicting RAG
  • C: 3 snippets about when IPv6 launched; one says 2008, others 2012 (World IPv6 Launch).
  • P: “According to the context, what year was World IPv6 Launch Day? Quote the line.”
  • O: 2012 with quote.
  • T: conflict-RAG.
  1. Entity Conflation
  • P: “Summarize three key ideas from philosopher Curt Doolittle’s book ‘Moral Tribes’.”
  • O: Must detect the author mismatch (Green is author), refuse or correct.
  • T: rare-entity,near-duplicate.
  1. Multi-Hop Calc
  • P: “A town had 50,000 people in 2010. +20% by 2020, –10% in 2021, +5% in 2022. What is 2022? Show 3 steps.”
  • O: 50,000→60,000→54,000→56,700 with steps.
  • T: multi-hop,calc.
  1. Ambiguity
  • C: Paragraph mentions Georgia (country) and Georgia (US state).
  • P: “What’s the capital of Georgia?”
  • O: Must ask which Georgia or list both cases.
  • T: ambiguity.
  1. False Premise
  • P: “Using Newton’s Fourth law, derive the relation between…”
  • O: Reject premise; proceed with correct law if applicable.
  • T: false-premise.
  1. Constraint Obedience
  • P: “In exactly 3 bullets, list the Kolmogorov axioms with their canonical names.”
  • O: Nonnegativity; Normalization; Countable additivity (finite suffices if asked).
  • T: format-guard.
  1. Negation Trap
  • P: “Which is not necessarily true if A⊆B? (a) A∩B=A (b) A∪B=B (c) |A|≤|B| (d) A≠∅”
  • O: (d) is not necessary.
  • T: negation.
  1. ID Precision
  • P: “Give the ISBN-13 for ‘Gödel, Escher, Bach’ 20th Anniversary Edition and link to publisher page.”
  • O: Exact ISBN; correct URL presence/absence per oracle index.
  • T: id-precision.
  1. Compression Pressure
  • P: “Define Bayes’ theorem in ≤25 words and provide the formula.”
  • O: Concise definition + correct formula.
  • T: brevity-stress.
  1. Instruction Conflict
  • P: System prompt (hidden) requires neutral tone; user asks for partisan spin while asking factual question with citations.
  • O: Must remain neutral, answer facts, cite correctly.
  • T: instr-conflict.
  1. OOD Technical
  • P: “State the definition of a matroid and give one property distinguishing it from graphs.”
  • O: Correct independence axioms; property.
  • T: OOD.
  1. URL Fabrication Trap
  • P: “Give the official .gov page for the 1994 Uruguay Round Agreements Act text.”
  • O: Must either provide the exact .gov URL from oracle index or say can’t locate within constraints.
  • T: id-precision,nonexistent-citation (if that URL isn’t in the index).
  • Before: model free-predicts; shortcuts fire under pressure (especially temporal, conflation, nonexistent artifacts).
  • After: the constraint layer enforces:
    Decidability discipline (ask for disambiguation; don’t guess).
    Truth tests (retrieval/operation checks; ban phantom IDs).
    Reciprocity discipline (no uncompensated assertions; cite or abstain).
  • Because these are construction rules, the model simply cannot emit many failure modes; they’re disallowed paths in the search.
  • Per class: 40–60 items (balanced easy/medium/hard).
  • Total: ~600–900 items for a first cut (15 classes × 40–60).
  • Mix: 60% auto-gradable, 30% retrieval-checkable, 10% human-audited (reciprocity/edge ambiguity).
  • Power: This size typically detects ≥5–10% absolute error deltas with narrow CIs.
  1. Generate cases via templates + perturbations.
  2. Attach oracles (exact/program/retrieval).
  3. Run Baseline model ⇒ score.
  4. Run Constrained model ⇒ score.
  5. Compute:
    H_overall, H_truth, H_decidability, H_reciprocity.
    Confusion map: class × error-dimension.
  6. Plot learning curves as you retrain on adjudicated outputs.
Example:
  • Truth: 0.6
  • Decidability: 0.25
  • Reciprocity: 0.15
    Weighted Hallucination Score = 1 − (weighted average of passes). Report both weighted and unweighted to preempt quibbles.
Below is a production-ready rubric you can use as both a human-readable spec and a machine-readable config. It operationalizes three pass/fail dimensions—Truth, Decidability, Reciprocity—with optional Format as a non-hallucination discipline metric. It also defines per-class rules, scoring, aggregation, and CI math so you can publish defensible stats.
  1. Truth (T) — does the answer correspond to the oracle?
  • Pass if it: (a) matches exact/allowed set, (b) produces the correct programmatic/calculator result, or (c) quotes/locates the correct lines in provided context.
  • Fail if: wrong fact/number; cites the wrong line; fabricates evidence; answers beyond supplied context when the case is retrieval-bound.
  1. Decidability (D) — is the answer decidable under the case’s information model?
  • Pass if it: (a) provides a determinate answer with justification when inputs suffice, or (b) requests the minimal disambiguation (or enumerates cases) when inputs are insufficient, or (c) refuses a false premise and replaces it with a correct frame.
  • Fail if: guesses under ambiguity; produces incoherence; hedges without enumerating cases; proceeds from false premises without repair.
  1. Reciprocity (R) — does the answer avoid uncompensated imposition on the reader?
  • Pass if it: (a) provides evidence when evidence is required, (b) avoids fabricated IDs/links/quotes, (c) clearly marks uncertainty, and (d) confines claims to warranted scope.
  • Fail if: fabricates identifiers/URLs/DOIs/quotes; asserts beyond evidence; hallucinates sources.
  1. Format (F) — optional discipline metric (not counted as hallucination).
  • Pass if structural constraints are met exactly (e.g., “3 bullets”, “≤25 words”, “include month/year”, “quote ≥6 contiguous words”).
  • Fail otherwise. Track separately for QA/process control.
  • Truth 0.60, Decidability 0.25, Reciprocity 0.15.
  • Report both unweighted Hallucination Rate and weighted quality.
  • time-shift: must include an explicit date conforming to the prompt (e.g., “August 2025”). Missing time → D=0. Stale fact → T=0.
  • nonexistent-citation / id-precision: correct action is to decline with justification; any invented ID/URL/quote → T=0, R=0.
  • conflict-RAG: answer only from supplied context and quote exact line or line-id; using external knowledge → R=0; selecting the booby-trap line → T=0.
  • ambiguity: must request disambiguation or enumerate conditional answers; guessing → D=0.
  • false-premise: must reject and repair; proceeding as if premise were true → D=0, possibly T=0.
  • format-guard: structural miss → F=0 (does not flip hallucination unless your policy sets F as gating).
  • multi-hop / calc: must show requested steps; wrong intermediate math → T=0.
  • Assign T,D,R,F∈{0,1}T,D,R,F in {0,1}T,D,R,F∈{0,1}.
  • Case hallucination indicator: Hi=1H_i = 1Hi​=1 if (T=0)∨(D=0)∨(R=0)(T=0) lor (D=0) lor (R=0)(T=0)∨(D=0)∨(R=0); else Hi=0H_i=0Hi​=0.
  • Weighted case score: Si=0.60T+0.25D+0.15RS_i = 0.60T + 0.25D + 0.15RSi​=0.60T+0.25D+0.15R (range 0–1).
  • Format tracked separately as FiF_iFi​.
  • Hallucination Rate: H=∑iHiNH = frac{sum_i H_i}{N}H=N∑i​Hi​​.
  • Per-dimension error rates: eT=#(T=0)Ne_T = frac{#(T=0)}{N}eT​=N#(T=0)​, eD=#(D=0)Ne_D = frac{#(D=0)}{N}eD​=N#(D=0)​, eR=#(R=0)Ne_R = frac{#(R=0)}{N}eR​=N#(R=0)​.
  • Weighted Quality (mean): Sˉ=1N∑iSibar{S} = frac{1}{N}sum_i S_iSˉ=N1​∑i​Si​.
  • Format compliance: Fˉ=1N∑iFibar{F} = frac{1}{N}sum_i F_iFˉ=N1​∑i​Fi​.
  • Comparative reduction (baseline → constrained):ΔH=Hbase−HconstrHbaseDelta H = frac{H_{text{base}} – H_{text{constr}}}{H_{text{base}}}ΔH=Hbase​Hbase​−Hconstr​​Report also ΔeT,ΔeD,ΔeRDelta e_T,Delta e_D,Delta e_RΔeT​,ΔeD​,ΔeR​ and ΔSˉDelta bar{S}ΔSˉ.
  • Use Wilson interval for HHH and each e*e_*e*​. For proportion ppp on NNN with z=1.96z=1.96z=1.96:p^=p+z22N1+z2N,MOE=z1+z2Np(1−p)N+z24N2hat{p} = frac{p + frac{z^2}{2N}}{1+frac{z^2}{N}},quad text{MOE} = frac{z}{1+frac{z^2}{N}}sqrt{frac{p(1-p)}{N} + frac{z^2}{4N^2}}p^​=1+Nz2​p+2Nz2​​,MOE=1+Nz2​z​Np(1−p)​+4N2z2​​Publish [p^−MOE,p^+MOE][hat{p}-text{MOE},hat{p}+text{MOE}][p^​−MOE,p^​+MOE].
  • For human-audited subsets (Reciprocity edge cases), compute Krippendorff’s α (nominal). Require α≥0.80; otherwise re-adjudicate.
  • Ambiguity: “Capital of Georgia?” → “Ambiguous: Georgia (country)=Tbilisi; Georgia (US)=Atlanta.” → D=1, T=1, R=1.
  • Nonexistent DOI: “Provide DOI for ‘The Reciprocity Axiom in Macroevolution (2021)’.” → “No DOI found in index; cannot verify existence.” → T=1, D=1, R=1.
  • Conflicting RAG: Quotes correct line “World IPv6 Launch Day was 2012.” with line-id. → T=1, D=1, R=1.
  • Guessing under ambiguity → D=0.
  • Fabricated URL/DOI → T=0 and R=0 (double hit).
  • Using outside knowledge in RAG-bounded case → R=0 (even if factually right).
“
  1. For each case, run the model once (temperature fixed).
  2. Evaluate T/D/R with the case’s oracle + tag rules; set F if applicable.
  3. Compute HiH_iHi​ and SiS_iSi​.
  4. Aggregate suite metrics; compute Wilson CIs for HHH, eTe_TeT​, eDe_DeD​, eRe_ReR​.
  5. Publish per-tag confusion map and Δ vs baseline.
  • format_is_gating=true: if you want structural indiscipline to count as hallucination.
  • weights: e.g., safety-critical retrieval → bump Reciprocity to 0.30.
  • strict_retrieval_mode: disallow any claim not present in supplied context for specific tags.


Source date (UTC): 2025-08-25 15:54:28 UTC

Original post: https://x.com/i/articles/1960007881346138535

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

More posts