Distilling judgment, not structure: a controlled experiment

Also in 中文. Part 12 of the Engineering LLM Applications series.

The hypothesis: can distilled judgment beat a distilled template?

Hand expert exemplars to a model and ask it to induce a reusable writing skill (a prompt), and there are two routes:

• Path A (structural template): extract “how many parts, what goes into each, in what order, in what format”;
• Path B (judgment logic): contrastive rationale extraction — show the model strong and weak versions of the same task and force it to articulate why the strong one is better, what it assumes about the reader, how a novice gets it wrong; aggregate the output into judgment rules + do/don’ts + a reader model.

H1: the Path-B skill beats the Path-A skill in pairwise preference judging. Everything is inference-only — no training, no fine-tuning, LLM APIs all the way down. Most prompt-engineering practice defaults to Path A; if H1 holds, there are many places where the recipe should change. This article is a full experiment report, including three methodological corrections that each came close to voiding the conclusion.

Design: control everything except one variable

All data comes from public benchmarks. Exemplars and strong/weak pairs are drawn from ASAP-AES 2.0 (English argumentative essays, human holistic scores 1–6): exemplars are score-6 essays; strong/weak pairs contrast score 5–6 against score 1–2 whole essays. A further 120 human-scored essays (20 per band) serve as a judge-credibility anchor, and the ablation uses arXivEdits same-sentence before/after expert rewrites.

The control discipline: A and B share the same exemplar input, the same generation model (DeepSeek-v4-pro, T=0.7), the same tasks, the same length constraint (roughly 250–400 words, no padding), and the same judges — the only variable is the distilled skill. Three bias controls on the judging side: position swapping (a win counts only if both presentation orders agree, otherwise it is a tie), identity masking (judges see only “Essay One / Essay Two”), and length neutrality (the rubric explicitly ignores length and penalizes verbosity). Dual judges: Claude-Sonnet-4.6 (primary) + Qwen3.7-max (cross-check).

The two distilled prompts are distinguishable at a glance. skill_A is position-and-proportion instructions throughout; a typical entry:

Introduction (1 paragraph) … A clear thesis statement that takes a definite position. … Relative length: 3–5 sentences.

Every entry in skill_B carries a “for whom, and why”:

When you acknowledge a counterargument, respond with a genuine rebuttal – not dismissal – because your reader expects fairness and intellectual seriousness.

Statistical lesson 1: effective N ≈ 1

The first round distilled a single pair (skill_A, skill_B), generated on 30 tasks, judged pairwise, and ran a binomial test treating the 30 pairings as independent samples: p ≈ 7×10⁻⁹. A beautiful number built on a broken inference: those 30 pairings share one and the same skill pair — only the tasks vary. What the test measures is “how stable this one skill pair is across 30 tasks”. H1, however, is a claim about methods, and the independent unit for that claim is the skill — of which there was exactly one pair. Effective N ≈ 1.

The redesign: first expand the exemplar pool from 12 to ≥24 essays, then use distinct random seeds to draw 6-exemplar subsets (Path B draws matching strong/weak-pair subsets), distilling K=10 independent skill pairs. Each pair generates and is judged on a shared task set (15 pairings per pair entered the statistics as executed), yielding one preference score w_k per pair; the test then runs at the skill level — Wilcoxon signed-rank on whether the K values of w_k are systematically above 0.5, with confidence intervals from a cluster bootstrap that resamples by skill. At the redesign’s budgeted scale this comes to 1200+ LLM calls across generation, judging, and distillation — roughly 5–6× the first round. Statistical honesty costs money.

Statistical lesson 2: metrics degenerate silently

A measurement check before the redesign saved the entire test: across 55 decided pairings under two judges and two rounds, A had won zero times — all the variation lay between “B wins” and “tie”. Define w_k = B/(B+A) as intuition suggests, and with A ≈ 0 almost every w_k collapses to exactly 1.0 — K samples with zero variance, and the Wilcoxon test degenerates into measuring nothing. The fix is a tie-inclusive preference score:

w_k = (B wins + 0.5 × ties) / T

Ties pull w_k toward 0.5 and restore real variance: on the original single pair this measures ≈0.97 under Claude and ≈0.73 under Qwen — the effect-size gap between the two judges is preserved and becomes testable.

Results: direction rock-stable, effect size judge-dependent

Four notes that must be read alongside the table:

1. The old per-pairing p values are flagged in the statistics script’s own output: “overestimate: K×T pairings are non-independent; shown for contrast only, do not cite”. Landing a known-misleading number on disk together with its warning is more reliable than clarifying it verbally afterwards.
2. p=0.0020 is a floor, not an effect size. When all K=10 pairs point the same way, the exact two-tailed Wilcoxon p bottoms out at 2/2¹⁰ ≈ 0.0020 — equivalent to a sign test. It encodes only “10 pairs, one direction”, not by how much: w_k values of all-0.51 and all-0.99 yield the identical p, and p<0.001 is unreachable before K≥17. Effect-size evidence lives in the CIs.
3. Qwen’s “significance” carries less weight: tie-dominated, swap-consistency mostly 0.2–0.5, CI lower bound 0.567 hugging 0.5. But a decided-only sensitivity analysis (ties removed) still shows 9/10 above 0.5 with zero net losses — the significance is not manufactured by tie-weighting.
4. A is not winless: A won individual pairings in 4 pairs under Claude and 8 under Qwen — but B suffered a net loss in no pair under either judge.

Ablation: does the judgment signal survive same-sentence pairs?

B’s contrast source is cross-topic whole-essay pairs — its wins could be contaminated by topic/content differences rather than pure “judgment”. Swapping the contrast source for arXivEdits same-sentence rewrite pairs (near-zero topic difference) yields B_sent, run over the same 10-pair protocol:

The reading stays conservative: under Claude, B_sent still significantly beats A and is statistically indistinguishable from B, suggesting the advantage stems from judgment rather than topic. But Qwen does not reproduce the ablation, and a genre confound remains — arXivEdits are scientific-paper sentences while the generation task is argumentative essays, so “same-sentence” and “genre” changed simultaneously; the clean single-factor control has not been run.

Judgment purity: an adversarial audit revises the self-assessment down

One question could still overturn the whole narrative: are skill_B’s entries genuine judgment rules, or structural descriptions in new clothing? LLM self-assessment cannot be trusted — the pilot’s sharpness self-check reported 100%. Instead, an 11-agent adversarial audit (executed by Claude Opus 4.8: classify every entry as judgment/structure, cross-compare against the paired skill_A for re-skinning, and specifically attack “is the because a tautology?”) spot-checked 5 pairs:

After adversarial review the mean drops to 0.746, yet all 5/5 pairs remain judgment-dominated and substantively distinct from skill_A (which is position/format instructions throughout, with zero reasoning). Verdict: “pass (with conditions)” — 0.75 is an upper bound that includes redundant clusters, and pair 4 is the weakest link (0.64). The overturned entries share one pattern: a structural action glued to a near-vacuous because. Note that this remains automated AI review, not independent human verification. (Adversarial review as an engineering practice is the subject of Part 13, Multi-agent adversarial review as an engineering practice.)

Seven honest limitations

1. No human anchor collected (the most fatal). A 30-pair blind-evaluation list has been exported (the machine ruling on the same pairings: B=20 / A=3 / tie=7), but human scores are not yet in. Until then, every piece of preference evidence comes from LLM judges, and a systematic LLM-judge preference cannot be ruled out.
2. Judge–distiller family overlap. The primary judge, Claude-Sonnet, shares a model family with the Claude Opus that performed skill distillation and the purity audit; every “strong” result happens to come from that same-family judge — a self-preference risk.
3. p bottomed out; effective N still optimistic. p=0.0020 encodes only “10/10 same direction”; and the 10 subsets, drawn from a finite pool, necessarily overlap — “effective N=10” is not strictly i.i.d. and remains an overestimate.
4. Effect size is judge-sensitive. Claude 0.83, Qwen 0.62: “B does not lose to A” is solid; “how much better B is” is not settled.
5. Judge range collapse. QWK=0.456 (Spearman 0.813), and the confusion matrix shows the judge never awards a 5 or 6 — discrimination at the high end is unverified, precisely the region where B vs A (“both essays are decent”) is being compared. The systematic treatment of judge validity is Part 9, Validity engineering for LLM-as-judge.
6. Single-judge ablation + genre confound. The same-sentence ablation is significant only under Claude, and it mixes in the scientific-sentence → argumentative-essay genre variable.
7. Mechanism and external validity. The secondary signal — rhetorical move coverage, A 0.89 vs B 0.86 (a weak negative conclusion: no evidence that B wins by stacking moves) — comes from the pilot, was not re-measured in the main experiment, and carries no statistical test; and only one generation model and one genre were tested.

The conclusion therefore freezes as: under the LLM-as-judge lens, “learn judgment > learn structure” receives preliminary support that is method-level, directionally highly consistent, and purity-checked (with conditions); H1 is a well-evidenced candidate, not an established result. Establishing it requires human evaluation, a third-party judge, and a clean same-sentence same-genre ablation.

When not to do this

1. If a deterministic metric exists, skip the pairwise judges. This heavyweight statistical apparatus is only warranted where quality can only be measured by preference comparison; wherever an assertion can be written, one line of assertion beats 800 judging calls.
2. Path B presupposes strong/weak pairs. Contrastive distillation extracts rationale by confronting strong and weak versions; in domains with no natural quality banding (such as human scores) and no credible synthetic degradation, Path B has nothing to contrast.
3. If the budget cannot afford K pairs, do not claim a method-level conclusion. Single-pair results support only “B beat A on this pair”; treating the task count as the sample size was this experiment’s earliest mistake — and the easiest one to make.
4. Structural templates are not useless. Path A matches B on rhetorical move coverage (0.89 vs 0.86); when the task’s bottleneck is format compliance rather than quality judgment, Path A is cheaper and steadier.