Step 1:
Let $I=\int\tan^{-1}\sqrt x dx.$
Put $x=t^2$,then on differentiating with respect to $t$,we get $dx=2t.dt$
$I=\int \tan^{-1}t.2t \;dt.$
$\;\;=2\int t\tan^{-1}t.dt.$
Clearly this is of the form $\int u dv.$
Hence this can be solved by the method of integration by parts.
$\int udv=uv-\int vdu.$
Let $u=\tan^{-1}t$
On differentiating we get,
$du=\large\frac{1}{1+t^2}$$dt$
Let $dv=t dt$,on integrating we get,
$v=\large\frac{t^2}{2}$
Now substituting for $u,v,du$ and $dv$,we get
$2\int t.\tan^{-1}t dt=2[\large\frac{t^2}{2}$$\tan^{-1}(t)-\int\large\frac{t^2}{2}.\frac{1}{1+t^2}$$dt$]
Step 2:
Consider $\int\large \frac{t^2}{1+t^2}$$dt$
Add and subtract 1 to the numerator
$\;\;=\int \large\frac{t^2+1-1}{1+t^2}$$dt$.
On splitting the terms as,
$\int\large\frac{t^2+1}{1+t^2}$$dt$$-\int\large\frac{1}{1+t^2}$$dt$
$\;\;=\int dt-\int \large\frac{1}{1+t^2}$$dt$
Step 3:
Now integrating this we get,
$t-\tan^{-1}(t)+c$
Hence $I=2\begin{bmatrix}\large\frac{t^2}{2}\normalsize\tan^{-1}(t)-\large\frac{1}{2}\normalsize(t-\tan^{-1}(t)\end{bmatrix}+c.$
$\qquad\;\;=t^2\tan^{-1}t-t+\tan^{-1}t+c$
Combining the like terms,
$\;\;=\tan^{-1}t(t^2+1)-t+c.$
Substituting for $t$ we get
$I=\tan^{-1}\sqrt x(x+1)-\sqrt x+c.$
Hence the correct option is A.