$\textbf{Step 1}$:

Let $(a\cos \theta.b \sin \theta)$ be one of the vertices of the rectangle.

Hence the length=$2a \cos \theta$ and the width=$2b \sin \theta$

$\Rightarrow$ Area of the rectangle $A\; = 2a\cos \theta \times 2 b \sin \theta$ $=2ab.2 \sin \theta \cos \theta$

$\Rightarrow A =2ab\sin 2 \theta$

$\Rightarrow A'=2ab.2\cos 2 \theta$

$\textbf{Step 2}$:

For maximun or minimum values of A, $\rightarrow A'=0$

$\Rightarrow 4ab\cos 2 \theta=0$

$\Rightarrow \cos 2 \theta=\large \frac{\pi}{2}$ $\rightarrow \theta =\frac{\pi}{4}$

$\Rightarrow A =2ab \sin2\frac{\pi}{4}$

$\Rightarrow A=2ab \sin (\large\frac{\pi}{2})$$=2ab$

Now $A''=-8ab \sin 2 \theta \rightarrow \;$Area is maximum when $A=2ab$