Step 1:

Perimeter of the window is $P=3x+2y=12$

$2y=\large\frac{12-3x}{2}$

Area of the window =area of the rectangle+area of the triangle

$A=xy+\large\frac{\sqrt 3}{4}$$x^2$

$\;\;=x(\large\frac{12-3x}{2})+\frac{\sqrt 3}{4}$$x^2$

$\;\;=\large\frac{12x}{2}-\frac{3x^2}{2}+\frac{\sqrt 3}{4}$$x^2$

$A=6x-\large\frac{3}{2}$$x^2+\large\frac{\sqrt 3}{4}$$x^2$

Step 2:

$\large\frac{dA}{dx}$$=6-\large\frac{6x}{2}+\frac{\sqrt 3\times 2x}{4}$

$\quad=6-3x+\large\frac{\sqrt 3}{2}$$x$

For the area to be maximum or minimum

$\large\frac{dA}{dx}$$=0$

$x(\large\frac{\sqrt 3}{2}$$-3)+6=0$

$\Rightarrow x(3-\large\frac{\sqrt 3}{2})=$$6$

$x=\large\frac{6\times 2}{6-\sqrt 3}=\frac{12}{6-\sqrt 3}$

$x=\large\frac{12(6+\sqrt 3)}{3}$

$x=4(6+\sqrt 3)$

Step 3:

Diff again w.r.t $x$ we get

$\large\frac{d^2A}{dx^2}=$$-3+\large\frac{\sqrt 3}{2}$$< 0$

Hence the dimension of the rectangle produce maximum area when

$x=4(6+\sqrt 3)$ and $y=12-\large\frac{3(6+\sqrt 3)}{2}$

$y=\large\frac{24-18-3\sqrt 3}{2}$

$\;\;=6-3\sqrt 3$

$y=3(2-\sqrt 3)$

Hence $x=4(6+\sqrt 3)$ and $y=3(2-\sqrt 3)$