$\cos^2(\alpha-\beta)+2ab\sin(\alpha-\beta)=\cos^2(\alpha-\beta)+2\cos(\theta-\alpha)\sin(\theta-\beta)\sin(\alpha-\beta)$
$\qquad\qquad\qquad\qquad\qquad\quad=\cos^2(\alpha-\beta)+[\sin(2\theta-(\alpha+\beta)^2+\sin(\alpha-\beta)]\sin(\alpha-\beta)$
$\qquad\qquad\qquad\qquad\qquad\quad=1+\sin\big(2\theta-(\alpha+\beta)\big)\sin(\alpha-\beta)$
$\qquad\qquad\qquad\qquad\qquad\quad=1+\sin\{(\theta-\alpha)+(\theta-\beta)\}\sin\{(\theta-\beta)-(\theta-\alpha)\}$
$\qquad\qquad\qquad\qquad\qquad\quad=1+\sin^2(\theta-\beta)-\sin^2(\theta-\alpha)$
$\qquad\qquad\qquad\qquad\qquad\quad=\cos^2(\theta-\alpha)+\sin^2(\theta-\beta)$
$\qquad\qquad\qquad\qquad\qquad\quad=a^2+b^2$
Hence (c) is the correct answer.