Given :
$\cot^{-1}(\sqrt{\cos \alpha})-\tan^{-1}(\sqrt{\cos \alpha})=x$
$\Rightarrow\:\tan^{-1}\big(\large\frac{1}{\sqrt{\cos \alpha}}\big)$$-\tan^{-1}\big(\sqrt{\cos\alpha})=x$
$\Rightarrow\:\tan^{-1}\large\frac{1/\sqrt{\cos\alpha}-\sqrt{\cos\alpha}}{1+1/\sqrt{\cos\alpha}.\sqrt{\cos\alpha}}$$=x$
$\Rightarrow\:\large\frac{1-\cos\alpha }{2\sqrt{\cos\alpha}}$$=\tan x$
$\Rightarrow\:\sin x=\large\frac{1-\cos \alpha}{1+\cos\alpha}=\frac{2sin^2\alpha/2}{2cos^2\alpha/2}$
$\Rightarrow\:\sin x=\tan^2\large\frac{\alpha}{2}$
Hence (a) is the correct answer.