Reasoning–Answer Entailment Under GRPO and DPO Fine-Tuning

Chain-of-thought (CoT) prompting can improve the problem-solving ability of large language models (LLMs), but a generated rationale does not always reflect the process by which a model reaches its answer, which limits the value of CoT for oversight and safety monitoring. We compare two post-training methods, Group Relative Policy Optimization (GRPO) and Direct Preference Optimization (DPO), on one measurable component of this problem: reasoning–answer entailment, whether the reasoning a model states supports the answer it gives. Entailment is a necessary condition for faithful reasoning, not a certificate of it, and we scope our claims accordingly.

Across four scales of the Qwen2.5-Instruct family (1.5B–14B) on GSM8K, we evaluate eight fine-tuned models on accuracy, reliability, and entailment metrics computed from persisted per-example generations, report bootstrap confidence intervals and paired significance tests, and assess the robustness of LLM-as-a-judge scoring with two independent judges. GRPO yields significantly higher entailment at every scale (margins of +.14 to +.31, all p < 10⁻⁷) and sweeps all metrics at 7B, while DPO matches GRPO’s greedy and pass@1 accuracy at 14B at lower training cost (self-consistency still favors GRPO).

The methods fail differently. GRPO produces internally consistent rationales even when its answers are wrong; DPO’s stated reasoning fails to entail a majority of its correct answers at every scale, consistent with the DPO recipe optimizing answer style rather than the reasoning–answer link. A forensic analysis attributes the mid-scale (3B) collapse of both methods largely to output-format failure rather than lost reasoning ability, and a headline 14B judge-score gap does not survive re-evaluation with a second judge.