Given a binary operation $\ast$ on set $N$ defined as $a \ast a = a^2 + b^2$:
$\textbf {Step 1: Checking if the operation is Commutative}$:
An operation $\ast$ on $A$ is commutative if $a\ast b = b \ast a\; \forall \; a, b \in A$
If $\ast$ is commuattive then $a \ast b= a^2 + b^2$ must be $= b \ast a = b^2 + a^2$ which is true.
Hence the $\ast$ operation is commutative.
$\textbf {Step 2: Checking if the operation is Associative}$:
An operation $\ast$ on $A$ is associative if $a\ast ( b \ast c) = (a \ast b) \ast c\; \forall \; a, b, c \in A$
Therefore, $a\ast ( b \ast c) = a \ast (b^2+c^2) = (a^2+(b^2+c^2)^2) = a^2+b^4+c^4+2b^2c^2$
Similarly, $(a \ast b) \ast c = (a^2+b^2) \ast c = (a^2+b^2)^2+c^2 = a^4+b^4+2a^2b^2+c^c$
We see that clearly $a\ast ( b \ast c) \neq (a \ast b) \ast c$ and hence $\ast$ is not associative.