Abstract
Linear models of image transformations provide a way to study how pretrained visual features respond to changes in an image. Enforcing exact group composition can worsen their predictions, but the resulting loss may reflect how the model is fitted rather than an inherent incompatibility between the features and the group laws. We investigate this distinction using orthogonal group models in fixed reduced spaces of frozen OpenCLIP, DINOv2, and SigLIP features. In spatial experiments, local power-repair procedures preserve more predictive accuracy than enforcing consistent rotations or constructing our full exact reference. We then refit the exact families through a shared orthogonal change of basis, preserving their representation type and every group law. Across spatial evaluation settings, this removes 29.3–42.8% of the exact reference’s excess prediction error relative to independently fitted maps. Independent orthogonal maps fitted to the same prediction objective remain more accurate, leaving a residual gap between the tested procedures. Controlled color experiments further show that prediction fitting outperforms matrix matching under the same optimization budget. Better exact predictions improve action identification over the reference, while the tested category-recognition task shows no additional benefit. Overall, the observed prediction penalty of exact group models depends materially on how they are fitted and cannot be attributed to the group laws alone.