Gravitational Potential energy at a distance r from center where $r \leq R$ is
$ E(r) =\large\frac{GM}{R^3} r$
$ \qquad =\large\frac{Gr}{R^3}\bigg(\frac{4}{3} $$ \pi R^3 \rho\bigg)$
$\qquad =\large\frac{4 G r \pi \rho}{3}$
Hence a is the correct answer.