# Provenance

This file is the authoritative record of who/what produced each part of the evaluation data, and what Giles has personally reviewed. It is updated as each phase lands: entries are appended, not rewritten, so the project's own construction history stays inspectable.

**V1 baseline: 2026-09-10.** With all 10 human evaluator assessments approved (see the review log below), `data/tasks.json` and `data/v1-results.json` are now the frozen V1 baseline. Any further change to either file past this point should be a deliberate, recorded amendment (a new entry here explaining what changed and why), not a silent edit; this is what the brief calls "preserve it and record subsequent changes."

## Repository framing

- **2026-09-10: disclaimers reworded from company-specific to company-agnostic.** An early pass satisfied the build brief's requirement not to imply Canva affiliation by writing explicit "not a Canva product" disclaimers in five places (`README.md`, `docs/limitations.md`, `index.html`, and a `source_note` in `data/locale-profiles.json`). On reflection that was the wrong fix: repeatedly naming a specific company in denial statements plants an association a reader might not otherwise make, and works against the brief's own choice to keep the build brief itself (which is Canva-specific) out of the shipped repository entirely. All five were reworded to be company-agnostic ("not built from, affiliated with, or representing any specific company's data, tooling, or workflow"), with no loss of claim strength. This history is visible in unsquashed git commit `57c055f`; disclosed here rather than left for a reader of that commit to interpret unexplained.
- **2026-09-13: locale profile currency guidance corrected for the same reason the rubric's `organization_style` note was.** `data/locale-profiles.json`'s currency guidance said A$ should be used "when the copy could otherwise be read in a cross-market context alongside another dollar currency, which none of these tasks do." That last clause was a conflation: `data/locale-profiles.json` is supposed to be general, reusable AU-locale reference material, independent of any specific task set, and a claim about what "these tasks" do or don't require doesn't belong in it any more than a version-specific claim belongs in `rubric.json`. Removed the clause, leaving a locale-general rule; the same text was corrected in all 10 V2 records' embedded `retrieved_context.locale_profile` (re-derivable reference data, not generation-altering `output.text`, so updated wholesale rather than preserved as historical fact, matching how the terminology.csv redesign's `retrieved_context.glossary_entries` was handled above).

## Generation (V1 and V2 outputs)

- **Model under test:** Claude Sonnet 5 (`claude-sonnet-5`), used as the model under test within this build session, in the same Claude Code conversation that authored the surrounding application. This is a disclosed conflict of interest, not a hidden one; see [limitations.md](limitations.md).
- **Process:** for each task and version, the model was given a system instruction plus a request for marketing copy and asked to produce it. The raw output was stored unedited as `output.text`. No output was regenerated, cherry-picked, or hand-edited after the fact to manufacture a convenient result.
- **Disclosed build note, `context_packet.user_prompt` vs. actual generation input:** the app displays `context_packet.user_prompt` as the model's complete input. For V1, generation was actually performed against a structured breakdown of the same request (a brief, a facts list, a constraints list) rather than by typing that prose. That structured breakdown was subsequently deleted from this repository's data model entirely: on review, it didn't correspond to any real product or evaluation mechanism (a real generation pipeline or evaluator only ever sees prompt, response, and rubric; the structured breakdown was never given to the LLM-as-a-judge, and kept it as unused data risked it being mistaken for a real pipeline step), so `user_prompt` is now the *only* specification of what each task asked for. Both forms carried identical informational content when the breakdown existed (nothing was added to or withheld from either form to change the outcome), and the frozen output was not edited, regenerated, or cherry-picked to fit either representation, or to fit its later removal. This is disclosed here, in [data-schema.md](data-schema.md), and in each V1 record's `output.generation_notes`, rather than left implicit.
- **Run date:** recorded per-record in `output.run_date` once generation happens (Phase 2 for V1, Phase 4 for V2).
- **V2 retrieval:** computed deterministically by `scripts/build_v2_context.py` from `data/tasks.json`, `data/terminology.csv` and `data/locale-profiles.json`, per the rule in [data-schema.md](data-schema.md). That script only selects which locale profile and glossary rows go into each task's `retrieved_context`; it does not touch the generated marketing copy itself, since writing that is the model under test's own work, not a deterministic pipeline step. Each V2 output was generated fresh, given the V2 system instruction, the verbatim `user_prompt`, and that retrieved context; it was not produced by editing or patching the corresponding V1 output, and the model was not instructed to avoid or reproduce any specific V1 issue. A glossary entry was used only where its trigger concept arose naturally in what would otherwise have been written; none were forced in where the brief gave no occasion for them (e.g. T01 and T03 use no glossary entry at all, since neither brief's content naturally called for one).

## Human evaluator assessment

- **Drafting:** candidate annotations for each output were drafted with assistant help, following the rubric in `data/rubric.json`, without access to the provisional LLM-as-a-judge assessment.
- **Review:** no record is described as a finished human evaluator assessment until Giles has explicitly reviewed and approved it. Each record's `reference_assessment.review_status` is `"pending_review"` until that happens, then `"approved"` with a `review_date` filled in.
- **Review log:** *(appended as reviews happen)*
  - Phase 2, 2026-09-10: all 10 V1 human evaluator assessments reviewed and approved by Giles Davis (`review_status: "approved"`, `review_date: "2026-09-10"` on every record in `data/v1-results.json`). Review covered the raw output text, every annotation's dimension/subtype/severity/rationale, and the resulting status/points for each of the 10 tasks; see the conversation history for the specific spot-checks performed (T01 and T10 shown in full immediately before approval).
  - Phase 4, 2026-09-10: all 10 V2 outputs generated and draft human evaluator assessments recorded in `data/v2-results.json` (`review_status: "pending_review"` on every record; every draft assessment found zero annotations, `status: "Pass"`, `total_points: 0`). 0 of 10 reviewed or approved by Giles yet.
  - Phase 4, 2026-09-11: all 10 V2 outputs regenerated (Finding 1's locale/glossary fix and Finding 2's no-fabrication fix bundled into one system instruction), and the human evaluator assessment redrafted against the final text and reviewed and approved by Giles Davis for all 10 records (`review_status: "approved"`, `review_date: "2026-09-11"`), without a separate per-record walkthrough in this session (Giles's explicit instruction: treat the draft as approved rather than reviewing individually). The redraft found 4 real issues the earlier placeholder draft had not been checked against: a minor accuracy/omission on T04 (the brief's "run by the local council" detail never appears in the output), and constraint_compliance/length_violation on T06 (minor, 38 words), T08 (minor, 34 words) and T09 (major, 23 words, `status: "Fail"`): a disclosed side effect of Finding 2's fix removing content that had been padding those three outputs to length, not a new defect introduced by carelessness.
  - 2026-09-11: following the terminology-glossary redesign above, T09 and T10's V2 human evaluator assessments were redrafted against their regenerated output and reviewed and approved by Giles Davis (`review_status: "approved"`, `review_date: "2026-09-11"`), this time with the specific drafted annotations shown to Giles in full and approved explicitly, not by blanket instruction. T09 stayed `status: "Fail"` (major length_violation, now 20 words); T10 stayed `status: "Pass"` (0 annotations, 41 words).

## Provisional LLM-as-a-judge

- **Model:** GPT-5.6 Sol, run by Giles outside this build session and outside this repository's tooling; see [judge-prompt.md](judge-prompt.md) for the exact prompt and process.
- **Independence measures:** the LLM-as-a-judge prompt never includes the human evaluator's assessment; it is not told whether the output is V1 or V2 (the prompt omits the V2-only locale profile/glossary context and is otherwise identical in shape for both versions).
- **Import log:** *(appended as results are imported)*
  - 2026-09-10: 10 of 20 LLM-as-a-judge assessments imported: all 10 V1, via `data/judge-intake/responses/judge-response-v1.json` (Giles's raw combined response from GPT-5.6 Sol) split by `scripts/split_judge_response.py` and merged by `scripts/import_judge_results.py`. 10 V2 still pending (Phase 4 not started).
  - **Human-evaluator-vs-LLM-as-a-judge status agreement, V1 (10 records):** 4 agree, 6 disagree. In every one of the 6 disagreements the LLM-as-a-judge rated the output *more* severely than the human evaluator (Pass→Fail or Needs revision→Fail), never the reverse. This one-directional pattern is disclosed here as a raw fact; it has not yet been investigated (that's Phase 3 work) and no claim is made about which assessor is "right"; see [limitations.md](limitations.md) on the LLM-as-a-judge being provisional, not validated.
  - 2026-09-11: the remaining 10 of 20 imported, all V2, via `data/judge-intake/responses/judge-response-v2.json`. That raw file needed a disclosed, mechanical repair before it would parse: GPT-5.6 Sol's response had three `rationale`/`suggested_correction` strings with unescaped internal quotation marks (invalid JSON), fixed by escaping only the quote characters; no wording, judgement or content was altered. Importing also surfaced a real bug in `scripts/import_judge_results.py`: because it reprocesses every file already present in `data/judge-intake/` on every run rather than skipping ones whose content hasn't changed, and always re-stamps `run_date` to the run date regardless, the first V2 import run silently overwrote all 10 of V1's already-approved `judge_assessment.run_date` fields from `2026-09-10` to `2026-09-11`, with no other content change. `data/v1-results.json` was reverted to its last-committed state and the script fixed to only update `run_date` when the imported content actually differs from what's already stored, so re-running it is now genuinely idempotent, matching what `data/judge-intake/README.md` already claimed.
  - **Human-evaluator-vs-LLM-as-a-judge status agreement, V2 (10 records, as originally imported 2026-09-10/11, before the terminology redesign below):** 8 agree, 2 disagree (T05, T08), both again the judge rating the output *more* severely than the human evaluator, the same one-directional pattern as V1. One disagreement was flagged specifically at the time: the judge flagged T09's "Native Bloom Essentials range" as an invented, unsupported product name (`accuracy`/`addition`, major), but that phrase was not a fabrication: it was the exact brand-guidance glossary substitution Finding 1's recommendation introduced (`data/terminology.csv`, row `B08`), correctly retrieved and applied. The judge could not tell the difference because it is deliberately never shown `retrieved_context` (see the independence measures above); this was disclosed as a real cost of that blindness design, not a flaw in V2's output, in [limitations.md](limitations.md). This specific instance no longer exists in the current data (see below): it remains here as an accurate record of what the import found at the time.

- **2026-09-11: `data/terminology.csv` redesigned; T09 and T10 regenerated and re-queued for judging.** On review, the glossary's `scope`/`match_key` mechanism (rows retrieved by `content_category` or `task_id`) was found not to correspond to any plausible real platform design: a real platform does not know in advance which business categories it will ever serve, so category-scoping a glossary row is an unnecessary restriction, and the 9 rows scoped to an individual `task_id` (`B01`-`B09`) were in practice brand-specific product/program names (e.g. "Ash Rewards", "Native Bloom Essentials range"), a fundamentally different kind of content from an AU-vs-US vocabulary pair, not something that belongs in a locale terminology glossary at all. The file was rebuilt as a flat, universal list: `scope`, `match_key` and `dimension_subtype` columns removed (the last because every remaining row is the same `terminology`/`wrong_term` issue, so the column was a constant carrying no information), `term_id` changed to a plain sequential number, all 13 category-level rows kept as-is (still genuine AU/US vocabulary pairs), the 9 brand-specific rows removed outright, and the `B02`/"trolley" row kept as a legitimate generic term (a shopping trolley, not a Cartly-specific concept) with "basket" dropped from its trigger terms since a basket is a physically different item, not a synonym. `scripts/build_v2_context.py` was simplified to match: every task's `retrieved_context.glossary_entries` is now the entire glossary, unfiltered, since retrieval is no longer conditioned on category or task at all (see [data-schema.md](data-schema.md)).
  - This changed the retrieved context for all 10 V2 records, but only T09 and T10's frozen output actually used a now-removed row (`B08`'s "Native Bloom Essentials range" and `B09`'s "Brightpath Certificate of Completion" respectively); the other 8 tasks' output text was unaffected, since their retrieved brand-specific rows had never actually been used in the generated copy (per the "a glossary entry was used only where its trigger concept arose naturally" rule above).
  - T09 and T10's `output.text` were regenerated as minimal edits from the same V1 baseline, re-applying only the fixes still motivated by the redesigned glossary (locale spelling and, for T09, Finding 2's no-fabrication clause) and dropping the now-nonexistent brand-name substitutions. T09's reference assessment was redrafted against the new text (word count changed from 23 to 20, still a major length-violation Fail; `review_status` reset to `"pending_review"`, awaiting Giles's review) and T10's reference assessment stayed Pass/0 (word count 41, within range; also reset to `"pending_review"` since it was reviewed against text that no longer exists). Both tasks' `judge_assessment` were reset to `"awaiting_external_run"` and their prior annotations discarded rather than assumed still valid, since both were run against output text that no longer exists: a stale judge result masquerading as current signal would be exactly the kind of manufactured convenience this project exists to avoid. A scoped re-run file, `docs/judge-run-v2.1.md`, was generated for just these 2 tasks (see [judge-prompt.md](judge-prompt.md)) rather than a full `docs/judge-run-v2.md` regeneration, since the other 8 records' output text is unchanged and re-judging them again would add cost without adding signal.
  - 2026-09-11: the GPT-5.6 Sol response to `docs/judge-run-v2.1.md` imported via `data/judge-intake/responses/judge-response-v2.1.json` → `scripts/split_judge_response.py v2` → `scripts/import_judge_results.py` (all other 18 of 20 files unchanged, `run_date` correctly preserved by the idempotency fix noted above). T09: judge rated `constraint_compliance`/`length_violation` **critical** (10 points, `status: "Fail"`), counting the output at 19 words against the human evaluator's 20; a one-word counting discrepancy, not a substantive disagreement, and the same major-vs-critical severity split as V1's original T09 judgment (see above). T10: judge found no issues (`status: "Pass"`, 0 points), agreeing exactly with the human evaluator. Both reference assessments were then shown to Giles in full and explicitly approved (`review_status: "approved"`, `review_date: "2026-09-11"`), restoring all 20 V1+V2 records to fully approved/imported and closing out this reopening.
  - **Human-evaluator-vs-LLM-as-a-judge status agreement, V2 (10 records, current, post-redesign):** unchanged from the original 8/2 baseline above: 8 agree, 2 disagree (T05, T08), still one-directional (judge more severe than the human evaluator in both disagreements). T09 remains a status-agreement case both before and after this redesign (both assessors said `Fail` throughout); what changed for T09 was only the judge's specific annotation and severity (previously `accuracy`/`addition` major plus `constraint_compliance`/`length_violation` major; now `constraint_compliance`/`length_violation` critical only, since the accuracy flag was on text that no longer exists), not the overall status.

## Summary: what Giles has personally reviewed so far

*(This list is the actual audit trail: keep it accurate and append-only.)*

- **2026-09-10:** all 10 V1 raw outputs and their human-evaluator-assessment annotations (`data/v1-results.json`). This included substantive back-and-forth that changed the data itself, not just a read-through: the context-packet schema was reworked twice for realism (task_brief/facts/constraints → a natural `user_prompt`, then the `grading_reference` breakdown was removed entirely as not representative of any real evaluation mechanism); the rubric was rebuilt against the live MQM Core typology (dimensions/subtypes relabelled, `critical` severity reduced from 25 to 10 points); and one stale rationale wording was corrected.
- **2026-09-10:** the 10 imported V1 LLM-as-a-judge assessments, spot-checked via View 2 in the running app (T01 specifically) after import. Noted but not yet acted on: the LLM-as-a-judge's `accuracy`/`addition` findings on T01 (flagging "our best-selling outerwear" and "made to keep you warm" as unsupported claims) look like a stricter reading of ordinary marketing puffery than a human would likely apply: a candidate example for Phase 3's "is the LLM-as-a-judge creating the pattern" check, not a conclusion. Nothing else (V2 outputs, `findings.json`) has been reviewed yet.