# Limitations This project is a small, deliberately bounded learning and interview artefact. The limitations below are as important as the findings; read them before drawing any conclusion from this repository. ## Sample size and scope - **Ten tasks is a diagnostic sample, not a statistically representative one.** Counts and percentages drawn from ten records are described as *concentration*, *recurring observation*, or *diagnostic signal* throughout the app and docs: never as a statistically significant trend or a production-quality rate. - **The same ten tasks are reused for V1 and V2.** This lets every rubric dimension be re-checked on directly comparable inputs, which is useful for spotting regressions and improvements, but it is not an independent validation set. A generation system configuration can look good on a reused set and still fail differently on new tasks. - **Ordinary, low-risk domains only.** The task set deliberately avoids regulated or safety-sensitive domains (insurance, credit, healthcare, gambling, legal). Findings here say nothing about locale-conditioned generation in higher-stakes domains. - **V1's frozen output was originally generated against a structured breakdown of each task (a brief, a facts list, a constraints list), not literally by typing the free-text `user_prompt` now shown as the model's input.** That structured breakdown has since been removed from this repository entirely (it didn't correspond to any real product or evaluation mechanism; see `docs/provenance.md`), but the already-generated V1 outputs were not regenerated to match its removal. Both forms carried identical informational content, and the output itself was never edited, regenerated, or cherry-picked either time. V2 generation, once it exists, is performed directly against `user_prompt`, so this specific gap is V1-only. ## The human evaluator assessment - **Not expert judgement or gold-standard data.** It is explicitly labelled *"Human evaluator assessment: created by the MVP author for demonstration"* and represents the shape of a structured human evaluation record, not a validated ground truth. - **Single reviewer, single pass.** There is no second reviewer, no adjudication, and no inter-annotator agreement measurement (no Cohen's kappa, no Fleiss' kappa); this project deliberately does not simulate an evaluator workforce. - **Assistant-drafted, human-approved.** Candidate annotations were drafted with assistant help; only annotations Giles has explicitly reviewed and approved are described as finished human evaluator assessments (see [provenance.md](provenance.md) for exactly which records and when). ## The provisional LLM-as-a-judge - **Provisional by design, not a validated automated evaluator.** Ten examples cannot calibrate or validate an LLM-as-a-judge. Known biases (self-preference, verbosity bias, position bias, limited consistency) are documented in Zheng et al., *Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena* () and are not ruled out here. - **Blind, where practical, to version and to the human evaluator's assessment.** But "where practical" is a real caveat: a careful reader could sometimes infer V1 vs V2 from writing style even though the prompt never labels it, and a single independent pass cannot fully rule out systematic blind spots. - **A real, disclosed instance of judge/human divergence on `accuracy`/`addition` split into two findings.** The judge flagged `accuracy`/`addition` far more often than the human evaluator did on V1, and on inspection most of the LLM's flags were legitimate. Finding 2 (in `data/findings.json`) treats this as a genuine V1 generation defect and its system-instruction recommendation is now built into V2; Finding 3 treats the human evaluator's original miss rate on the same defect as a separate, evaluation-side calibration gap and recommends evaluator training, which has not been actioned. See [methodology.md](methodology.md) for the alternative-explanations check behind both findings and the optional recommendations a production system might consider for the evaluator-training gap. - **The judge's blindness to `retrieved_context` has a real cost, not just a benefit.** Keeping the judge blind to V2's retrieved locale profile and glossary (see the independence measures above) is what lets it stay blind to version, but it also means the judge cannot distinguish a legitimate, retrieval-authorised terminology substitution from an invented one, and would flag it as a fabrication. This is disclosed as a structural cost of the blindness design, not something to correct after the fact if it occurs: "correcting" a provisional judge's output would itself be exactly the kind of manufactured convenience this project is built to avoid. - **View 5's V1-vs-V2 comparison groups annotations by dimension and subtype only, not by the specific underlying claim.** This is a real limitation, not just a simplification: two different claims of the same type can be shown as one "persisting" issue even when the original was genuinely fixed and an unrelated new one took its place, or as one "fixed" issue when a different new claim of the same type replaced it. The dimension/subtype grouping alone cannot disambiguate this; only the annotation text itself (visible in the same drilldown) can. - **This project's evaluator coverage ratio is inverted from a typical production system.** Every V1 output here gets a full human evaluator assessment and a full LLM-as-a-judge assessment (100%/100%), because ten tasks is small enough to hand-label completely. A real system would typically run the cheap, fast judge at or near full coverage and reserve human review for a smaller, deliberately chosen sample (a fixed audit rate, or one targeted at low-confidence or disagreement cases) used to calibrate and validate the judge, not the reverse. Nothing about the 100%/100% ratio here should be read as a template for how a production system should allocate review effort. ## The V2 recommendation and comparison - **Lightweight retrieval, not sophisticated RAG.** V2's context construction is plain, unfiltered inclusion of a small CSV glossary's contents: it demonstrates the mechanics of retrieval-augmented context, not a production-grade retrieval system. - **A rudimentary regression check, not production regression testing.** The paired V1/V2 comparison re-evaluates every rubric dimension on the same ten tasks and surfaces new errors, which is useful, but it is not an independent validation set, a held-out test set, or longitudinal monitoring. - **No claim of success is assumed.** The comparison view reports whatever the stored evidence shows, including the possibility of a null or negative result, and avoids celebratory language regardless of outcome. - **Findings 1 and 2's system-change recommendations were bundled into a single V2, not tested as separate versions.** Each recommendation is independently evidenced and the two touch unrelated mechanisms (locale/glossary retrieval vs. an instruction-following constraint), so there was no reason to expect them to interact; bundling them was a deliberate efficiency trade-off, not an oversight. The cost of that choice is real, though: if V2 shows an improvement, this build cannot cleanly attribute how much of it came from which fix, and a rigorous before/after test would isolate them as separate versions. One concrete side effect of bundling surfaced during generation: removing Finding 2's unsupported claims from T06, T08 and T09 dropped those outputs below their required word count, since the fabricated content had been padding them out. That shortfall is disclosed in each record's `output.generation_notes` rather than papered over, and is itself a small demonstration of why isolating one change at a time is the more rigorous default. ## Generation process - **The model under test and the LLM-as-a-judge are not from the same evaluation-independence chain as a production setup would require**, since this MVP is authored end-to-end in a short build window. The generation model (Claude Sonnet 5) and the human-evaluator-assessment drafting both happened within the same build session; the LLM-as-a-judge model (GPT-5.6 Sol) was deliberately kept separate and run independently to reduce, though not eliminate, that conflict of interest. See [provenance.md](provenance.md). - **Outputs were generated once and frozen.** They were not regenerated, cherry-picked, or edited after the fact to manufacture a convenient error or improvement. If an early exploratory pass had produced no useful variation, the task set itself would have been revised before baselining, not the outputs. ## What this repository does not claim at all - Professional evaluator, localisation-specialist, ML-engineer, or production-platform experience. - Any real company's data, workflow, internal tooling, or product resemblance. - Multilingual expertise or translation-quality evaluation. - A production evaluation platform, Label Studio functionality, or real evaluator-workforce operations.