$\textbf{Step 1}$:
Let $(a\cos \theta.b \sin \theta)$ be one of the vertices of the rectangle.
Hence the length=$2a \cos \theta$ and the width=$2b \sin \theta$
$\Rightarrow$ Area of the rectangle $A\; = 2a\cos \theta \times 2 b \sin \theta$ $=2ab.2 \sin \theta \cos \theta$
$\Rightarrow A =2ab\sin 2 \theta$
$\Rightarrow A'=2ab.2\cos 2 \theta$
$\textbf{Step 2}$:
For maximun or minimum values of A, $\rightarrow A'=0$
$\Rightarrow 4ab\cos 2 \theta=0$
$\Rightarrow \cos 2 \theta=\large \frac{\pi}{2}$ $\rightarrow \theta =\frac{\pi}{4}$
$\Rightarrow A =2ab \sin2\frac{\pi}{4}$
$\Rightarrow A=2ab \sin (\large\frac{\pi}{2})$$=2ab$
Now $A''=-8ab \sin 2 \theta \rightarrow \;$Area is maximum when $A=2ab$