Given :
$\cos^{-1}x-\cos^{-1}y/2=\alpha$
$\Rightarrow \cos^{-1}\big[\large\frac{xy}{2}+$$\sqrt{1-x^2}\sqrt{1-\large\frac{y^2}{4}}\big]=\alpha$
$\large\frac{xy}{2}$$+\sqrt{1-x^2}\sqrt{1-\large\frac{y^2}{4}}$$=\cos\alpha$
$2\sqrt{1-x^2}\sqrt{1-\large\frac{y^2}{4}}$$=2\cos \alpha-xy$
Step 2:
On squaring both sides we get,
$4(1-x^2)[\large\frac{4-y^2}{4}\big]$$=4\cos^2\alpha+x^2y^2-4xy \cos \alpha$
$\Rightarrow 4-y^2-4x^2+x^2y^2=4\cos^2\alpha+x^2y^2-4xy\cos\alpha$
$\Rightarrow 4x^2+y^2-4xy\cos\alpha=4-4\cos^2\alpha$
$\Rightarrow 4x^2-4xy\cos\alpha+y^2=4(1-\cos^2\alpha)$
$\Rightarrow 4\sin^2\alpha$
Hence (b) is the correct answer.