Alexander Koran
All projects

December 2024

Improving the Out-of-Distribution Performance of Distilled Chain-of-Thought Language Models

Testing whether small distilled language models generalize beyond their training distributions, and whether Fishr regularization can improve that generalization.

View poster Read report

Research poster presenting out-of-distribution evaluations and Fishr regularization experiments for distilled chain-of-thought models

Project overview

In this collaborative COMP 767 project with Thomas Lai, we studied whether chain-of-thought reasoning distilled into smaller language models survives distribution shift. We evaluated a 220M-parameter T5 model across symbolic reasoning, mathematics, and natural language inference, finding substantial performance drops on out-of-distribution datasets. We then applied Fishr regularization to the coin-flip experiments and observed improved OOD performance in several settings, alongside a trade-off in some in-distribution results. My work focused on the mathematics OOD experiments, support for the Fishr experiments, the poster design and content, and the final report.