Given a binary operation $\ast$ in Q defined by $a \ast b=a^2+b^2$:

$\textbf {Step 1: Checking if the operation is Commutative}$:

For an operation $\ast$ to be commutative $a\ast b = b \ast a$.

$\Rightarrow a \ast b = a^2 + b^2 = b^2 + a^2 = b \ast a$.

Since $a\ast b = b \ast a$ the operation $\ast$ is commutative.

$\textbf {Step 2: Checking if the operation is Associative}$:

For an operation $\ast$ on $A$ is associative $a\ast ( b \ast c) = (a \ast b) \ast c\; \forall \; a, b, c \in A$

$\Rightarrow a\ast ( b \ast c) = a \ast (b^2 + c^2) = a^2 + (b^2+c^2)^2$

$\Rightarrow (a \ast b) \ast c = (a^2 + b^2) \ast c = (a^2 + b^2)^2 + c^2$

Clearly, $a\ast ( b \ast c) \neq (a \ast b) \ast c$. Hence the operation $\ast$ is not associative unless $a=b=c=1$.