Does your LLM's reasoning actually drive its answers? The data says: probably not.
Chain-of-thought (CoT) prompting produces reasoning traces that look coherent, but are they causally connected to the model's final answer? Or are they post-hoc rationalizations of a decision made elsewhere in the model? This distinction matters enormously for AI safety and interpretability. If CoT is just a narrative layer, then 'reasoning-based' safety interventions are built on sand. This project investigates CoT faithfulness using causal intervention: does perturbing the reasoning trace change the model's output? And critically: does fine-tuning on high-quality CoT help or hurt?
Faithfulness is operationalized as the causal impact of the reasoning trace on the final prediction. Traces are perturbed and the change in output is measured.
NPO fine-tunes the model to prefer outputs where the CoT and the answer are causally linked, penalizing disconnected reasoning.
Models were fine-tuned on curated high-quality CoT datasets and evaluated for faithfulness, isolating whether data quality or training objective is the bottleneck.
Systematic causal intervention on CoT content reveals the degree to which the model's prediction is determined by the reasoning vs. the question alone.
Fine-tuning on high-quality CoT reduces faithfulness by 56–75%, a counterintuitive but robust finding
The root cause is data curation, not the training objective: models learn to produce impressive reasoning without using it
NPO partially recovers faithfulness but is insufficient alone
Findings suggest CoT-based safety interventions need structural changes, not just better data