Step 1
$y=x(x-3)^2$
differentiating w.r.t $x$ we get,
( apply product rule)
$ y'=x[2(x-3)]+(x-3)^2.1$
$y'=2x(x-3)+(x-3)^2$
$=2x^2-6x+x^2-6x+9$
$ = 3x^2-12x+9$
Step 2
If it is a decreasing function, then $y'<0$
$ \Rightarrow 3x^2-12x+9 <0$
$ \Rightarrow 3(x^2-4x+3) < 0$
$ \Rightarrow 3(x-3)(x-1)<0$
This clearly implies the function decreases when
$ 1 < x < 3$
Hence A is the correct answer.