# Distilling judgment, not structure: a controlled experiment

> Two ways to distill expert writing into a reusable skill: extract the structural template (what to write) or the judgment logic (why it is written that way). A full experiment report: K=10 independently distilled skill pairs, dual judges, skill-level statistics — and three methodological corrections that each nearly invalidated the result: effective N≈1, silent metric degeneration, and an adversarial purity audit. Verdict: method-level support for H1, not yet established.

- Canonical (HTML): https://kaguc.com/blog/distill-judgment-experiment/
- Date: 2026-07-29


*Also in [中文](/blog/distill-judgment-experiment-zh/). Part 12 of the Engineering LLM Applications series.*

## The hypothesis: can distilled judgment beat a distilled template?

Hand expert exemplars to a model and ask it to induce a reusable writing skill (a prompt), and there are two routes:

- **Path A (structural template)**: extract "how many parts, what goes into each, in what order, in what format";
- **Path B (judgment logic)**: contrastive rationale extraction — show the model strong and weak versions of the same task and force it to articulate *why the strong one is better, what it assumes about the reader, how a novice gets it wrong*; aggregate the output into judgment rules + do/don'ts + a reader model.

**H1: the Path-B skill beats the Path-A skill in pairwise preference judging.** Everything is inference-only — no training, no fine-tuning, LLM APIs all the way down. Most prompt-engineering practice defaults to Path A; if H1 holds, there are many places where the recipe should change. This article is a full experiment report, including three methodological corrections that each came close to voiding the conclusion.

## Design: control everything except one variable

All data comes from public benchmarks. Exemplars and strong/weak pairs are drawn from ASAP-AES 2.0 (English argumentative essays, human holistic scores 1–6): exemplars are score-6 essays; strong/weak pairs contrast score 5–6 against score 1–2 whole essays. A further 120 human-scored essays (20 per band) serve as a judge-credibility anchor, and the ablation uses arXivEdits same-sentence before/after expert rewrites.

The control discipline: A and B share the same exemplar input, the same generation model (DeepSeek-v4-pro, T=0.7), the same tasks, the same length constraint (roughly 250–400 words, no padding), and the same judges — the only variable is the distilled skill. Three bias controls on the judging side: position swapping (a win counts only if both presentation orders agree, otherwise it is a tie), identity masking (judges see only "Essay One / Essay Two"), and length neutrality (the rubric explicitly ignores length and penalizes verbosity). Dual judges: Claude-Sonnet-4.6 (primary) + Qwen3.7-max (cross-check).

The two distilled prompts are distinguishable at a glance. skill_A is position-and-proportion instructions throughout; a typical entry:

> Introduction (1 paragraph) … A clear thesis statement that takes a definite position. … Relative length: **3–5 sentences.**

Every entry in skill_B carries a "for whom, and why":

> When you acknowledge a counterargument, respond with a genuine rebuttal – not dismissal – **because your reader expects fairness and intellectual seriousness.**

## Statistical lesson 1: effective N ≈ 1

The first round distilled a single pair (skill_A, skill_B), generated on 30 tasks, judged pairwise, and ran a binomial test treating the 30 pairings as independent samples: p ≈ 7×10⁻⁹. A beautiful number built on a broken inference: those 30 pairings share one and the same skill pair — only the tasks vary. What the test measures is "how stable this one skill pair is across 30 tasks". H1, however, is a claim about *methods*, and the independent unit for that claim is the skill — of which there was exactly one pair. **Effective N ≈ 1.**

The redesign: first expand the exemplar pool from 12 to ≥24 essays, then use distinct random seeds to draw 6-exemplar subsets (Path B draws matching strong/weak-pair subsets), distilling **K=10 independent skill pairs**. Each pair generates and is judged on a shared task set (15 pairings per pair entered the statistics as executed), yielding one preference score w_k per pair; the test then runs at the skill level — Wilcoxon signed-rank on whether the K values of w_k are systematically above 0.5, with confidence intervals from a cluster bootstrap that resamples by skill. At the redesign's budgeted scale this comes to 1200+ LLM calls across generation, judging, and distillation — roughly 5–6× the first round. Statistical honesty costs money.

```mermaid
flowchart LR
    P[Exemplar pool ≥24] -->|seed k draws 6 exemplars + pair subset| D[Distill K=10 pairs<br/>skill_A_k and skill_B_k]
    D --> G[Shared tasks ×2 generations per pair<br/>same model · T=0.7 · 250–400 words]
    G --> J[Dual-judge pairwise eval<br/>position swap · masking · length-neutral]
    J --> W[One preference score w_k<br/>per pair per judge]
    W --> S[Skill-level Wilcoxon<br/>+ cluster bootstrap CI]
```

## Statistical lesson 2: metrics degenerate silently

A measurement check before the redesign saved the entire test: **across 55 decided pairings under two judges and two rounds, A had won zero times** — all the variation lay between "B wins" and "tie". Define w_k = B/(B+A) as intuition suggests, and with A ≈ 0 almost every w_k collapses to exactly 1.0 — K samples with zero variance, and the Wilcoxon test degenerates into measuring nothing. The fix is a tie-inclusive preference score:

```text
w_k = (B wins + 0.5 × ties) / T
```

Ties pull w_k toward 0.5 and restore real variance: on the original single pair this measures ≈0.97 under Claude and ≈0.73 under Qwen — the effect-size gap between the two judges is preserved and becomes testable.

## Results: direction rock-stable, effect size judge-dependent

| Judge | w_k mean | median | std | 95% CI (cluster bootstrap) | w_k>0.5 | Wilcoxon two-tailed p | old "per-pairing" p (overestimate, do not cite) |
|---|---|---|---|---|---|---|---|
| Claude (primary) | **0.830** | 0.833 | 0.143 | [0.747, 0.913] | **10/10** | **0.0020** | 7.95e-24 |
| Qwen (cross) | 0.617 | 0.617 | 0.086 | [0.567, 0.667] | 9/10 | 0.0039 | 2.06e-06 |

Four notes that must be read alongside the table:

1. The old per-pairing p values are flagged *in the statistics script's own output*: "**overestimate: K×T pairings are non-independent; shown for contrast only, do not cite**". Landing a known-misleading number on disk together with its warning is more reliable than clarifying it verbally afterwards.
2. **p=0.0020 is a floor, not an effect size.** When all K=10 pairs point the same way, the exact two-tailed Wilcoxon p bottoms out at 2/2¹⁰ ≈ 0.0020 — equivalent to a sign test. It encodes only "10 pairs, one direction", not *by how much*: w_k values of all-0.51 and all-0.99 yield the identical p, and p<0.001 is unreachable before K≥17. Effect-size evidence lives in the CIs.
3. **Qwen's "significance" carries less weight**: tie-dominated, swap-consistency mostly 0.2–0.5, CI lower bound 0.567 hugging 0.5. But a decided-only sensitivity analysis (ties removed) still shows 9/10 above 0.5 with zero net losses — the significance is not manufactured by tie-weighting.
4. **A is not winless**: A won individual pairings in 4 pairs under Claude and 8 under Qwen — but B suffered a net loss in no pair under either judge.

## Ablation: does the judgment signal survive same-sentence pairs?

B's contrast source is cross-topic whole-essay pairs — its wins could be contaminated by topic/content differences rather than pure "judgment". Swapping the contrast source for arXivEdits same-sentence rewrite pairs (near-zero topic difference) yields B_sent, run over the same 10-pair protocol:

| Judge | B_sent w_k mean | 95% CI | pairs >0.5 | vs A (Wilcoxon p) | B vs B_sent |
|---|---|---|---|---|---|
| Claude | **0.720** | [0.663, 0.773] | 10/10 | **0.0020** (significant) | +0.110, p=0.186 (n.s.) |
| Qwen | 0.567 | [0.493, 0.640] | 6/10 | 0.109 (not significant) | +0.050, p=0.281 (n.s.) |

The reading stays conservative: under Claude, B_sent still significantly beats A and is statistically indistinguishable from B, suggesting the advantage stems from judgment rather than topic. But Qwen does not reproduce the ablation, and a genre confound remains — arXivEdits are scientific-paper sentences while the generation task is argumentative essays, so "same-sentence" and "genre" changed simultaneously; the clean single-factor control has not been run.

## Judgment purity: an adversarial audit revises the self-assessment down

One question could still overturn the whole narrative: are skill_B's entries genuine judgment rules, or structural descriptions in new clothing? LLM self-assessment cannot be trusted — the pilot's sharpness self-check reported 100%. Instead, an 11-agent adversarial audit (executed by Claude Opus 4.8: classify every entry as judgment/structure, cross-compare against the paired skill_A for re-skinning, and specifically attack "is the *because* a tautology?") spot-checked 5 pairs:

| Pair | Initial self-ratio | After adversarial review | Overturned entries (re-skinning suspects) |
|---|---|---|---|
| 1 | 87% | **80%** | context-in-opening ≈ A's Intro recipe |
| 2 | 82% | **73%** | "vivid example" ≈ show-don't-tell tautology |
| 3 | 73% | **67%** | ordering by persuasiveness ≈ structural organization |
| 4 | 82% | **64%** | rebuttal / closing ≈ A's counterargument / conclusion sections |
| 5 | 100% | **90%** | point ordering = an organization rule |

After adversarial review the mean drops to **0.746**, yet all 5/5 pairs remain judgment-dominated and substantively distinct from skill_A (which is position/format instructions throughout, with zero reasoning). Verdict: "**pass (with conditions)**" — 0.75 is an upper bound that includes redundant clusters, and pair 4 is the weakest link (0.64). The overturned entries share one pattern: a structural action glued to a near-vacuous *because*. Note that this remains automated AI review, not independent human verification. (Adversarial review as an engineering practice is the subject of Part 13, [Multi-agent adversarial review as an engineering practice](/blog/adversarial-review/).)

## Seven honest limitations

1. **No human anchor collected (the most fatal).** A 30-pair blind-evaluation list has been exported (the machine ruling on the same pairings: B=20 / A=3 / tie=7), but human scores are not yet in. Until then, every piece of preference evidence comes from LLM judges, and a systematic LLM-judge preference cannot be ruled out.
2. **Judge–distiller family overlap.** The primary judge, Claude-Sonnet, shares a model family with the Claude Opus that performed skill distillation and the purity audit; every "strong" result happens to come from that same-family judge — a self-preference risk.
3. **p bottomed out; effective N still optimistic.** p=0.0020 encodes only "10/10 same direction"; and the 10 subsets, drawn from a finite pool, necessarily overlap — "effective N=10" is not strictly i.i.d. and remains an overestimate.
4. **Effect size is judge-sensitive.** Claude 0.83, Qwen 0.62: "B does not lose to A" is solid; "how much better B is" is not settled.
5. **Judge range collapse.** QWK=0.456 (Spearman 0.813), and the confusion matrix shows the judge never awards a 5 or 6 — discrimination at the high end is unverified, precisely the region where B vs A ("both essays are decent") is being compared. The systematic treatment of judge validity is Part 9, [Validity engineering for LLM-as-judge](/blog/judge-validity/).
6. **Single-judge ablation + genre confound.** The same-sentence ablation is significant only under Claude, and it mixes in the scientific-sentence → argumentative-essay genre variable.
7. **Mechanism and external validity.** The secondary signal — rhetorical move coverage, A 0.89 vs B 0.86 (a weak negative conclusion: no evidence that B wins by stacking moves) — comes from the pilot, was not re-measured in the main experiment, and carries no statistical test; and only one generation model and one genre were tested.

The conclusion therefore freezes as: **under the LLM-as-judge lens, "learn judgment > learn structure" receives preliminary support that is method-level, directionally highly consistent, and purity-checked (with conditions); H1 is a well-evidenced candidate, not an established result.** Establishing it requires human evaluation, a third-party judge, and a clean same-sentence same-genre ablation.

## When not to do this

1. **If a deterministic metric exists, skip the pairwise judges.** This heavyweight statistical apparatus is only warranted where quality can *only* be measured by preference comparison; wherever an assertion can be written, one line of assertion beats 800 judging calls.
2. **Path B presupposes strong/weak pairs.** Contrastive distillation extracts rationale by confronting strong and weak versions; in domains with no natural quality banding (such as human scores) and no credible synthetic degradation, Path B has nothing to contrast.
3. **If the budget cannot afford K pairs, do not claim a method-level conclusion.** Single-pair results support only "B beat A on this pair"; treating the task count as the sample size was this experiment's earliest mistake — and the easiest one to make.
4. **Structural templates are not useless.** Path A matches B on rhetorical move coverage (0.89 vs 0.86); when the task's bottleneck is format compliance rather than quality judgment, Path A is cheaper and steadier.