Your goal is to squeeze two non-overlapping quadrilaterals within a unit circle (i.e., a circle with radius 1). The quadrilaterals can share common edges, parts of edges, or vertices, but their interiors may not overlap. Their vertices may also lie on the circle’s circumference.
What is the greatest combined area these quadrilaterals can have?
When it comes to fitting shapes inside a circle, the best guess for largest possible area will be a regular polygon. So let's see what configurations we can come up with for two smushed together quadrilaterals. If we have two quadrilaterals that don't share any edges whatsoever, then we're not trying very hard since we could just expand one or the other quadrilateral until they do have a nonempty intersection and in turn get a larger total area. Similarly, if we have quadrilaterals that only intersect in a portion of an edge, then we can enlarge one or the other of the quadrilaterals until the entire edge is shared and get a larger total area. If the quadrilaterals only intersect in a single vertex then either one or the other can be orthogonally rotated such that the resulting quadrilaterals are all
OK, so now that we've covered that we should share entire edges, let's cover two cases: If we have two quadrilaterals that share two sides, then at most there could only be $4$ vertices along the circle. This configuration would have a maximum area obtained by a square inscribed in the circle, which has an area of $2.$ If instead, we have two quadrilaterals that share exactly one side, then at most there could be $6$ vertices along the circle. The area of a regular hexagon inscribed in a unit circle is $\frac{3\sqrt{3}}{2} = 2.59807...$ which is the largest total area of two non-overlapping quadrilaterals within a unit circle.
But what if we wanted to stick to only four vertices along the circle and define $Z$ as the following sum: the area of the (convex) quadrilateral formed by all four of these points plus the area of the largest triangle (by area) formed by any three of these points. Given that the four distinct points can be anywhere on the unit circle, what is the greatest possible value of $Z$?
Let's assume that the four vertices are numbered in say clockwise order, $z_1$, $z_2$, $z_3$, and $z_4$. Let the measure of the angle between the radii connecting $z_i$ and $z_{i+1}$ be $\alpha_i,$ for $i = 1, 2,$ and $3.$ For completeness, let the measure of the angle between the radii connecting $z_4$ and $z_1$ is $\alpha_4 = 2\pi - \alpha_1 - \alpha_2 - \alpha_3.$ In this case, the area of the quadrilateral is given by $$A_\square = \frac{1}{2} \sum_{i=1}^4 \sin \alpha_i.$$ Similarly, the area of the triangle formed by $z_1,$ $z_2$ and $z_3$ is $$A_\triangle = \frac{1}{2} \sin \alpha_1 + \frac{1}{2} \sin \alpha_2 - \frac{1}{2} \sin (\alpha_1 + \alpha_2).$$ Extending this to all of the triangles gives us the formula for \begin{align*}Z = Z(\alpha_1, \alpha_2, \alpha_3, \alpha_4) &= \frac{1}{2} \sum_{i=1}^4 \sin \alpha_i + \frac{1}{2} \max \left\{ \sin \alpha_1 + \sin \alpha_2 - \sin ( \alpha_1 + \alpha_2 ) ,\right.\\ &\quad\quad\quad \sin \alpha_2 + \sin \alpha_3 - \sin ( \alpha_2 + \alpha_3 ),\\ &\quad\quad\quad \sin \alpha_3 + \sin \alpha_4 - \sin ( \alpha_3 + \alpha_4 ),\\ &\quad\quad\quad \left.\sin \alpha_4 + \sin \alpha_1 - \sin ( \alpha_4 + \alpha_1 ) \right\}.\end{align*} Then all we need to do is optimize subject to $\alpha_1 + \alpha_2 + \alpha_3 + \alpha_4 = 2\pi.$ Without loss of generality, let's assume that the triangle between $z_1,$ $z_2$ and $z_3$ is (one of) the largest of the triangles, so that in some region of the optimal values of $\alpha_1, \alpha_2, \alpha_3, \alpha_4$ we have $$\tilde{Z} = \sin \alpha_1 + \sin \alpha_2 + \frac{1}{2} \sin \alpha_3 + \frac{1}{2} \sin \alpha_4 - \frac{1}{2} \sin ( \alpha_1 + \alpha_2 ).$$ In this case, we would have expect from the method of Lagrange multipliers that there is some $\lambda \in \mathbb{R}$ such that the following system of equations holds \begin{align*} \cos \alpha_1 - \frac{1}{2} \cos (\alpha_1 + \alpha_2) + \lambda &= 0 \\ \cos \alpha_2 - \frac{1}{2} \cos (\alpha_1 + \alpha_2) + \lambda &= 0\\ \frac{1}{2} \cos \alpha_3 + \lambda &= 0 \\ \frac{1}{2} \cos \alpha_4 + \lambda &= 0\end{align*}
From the last two equations, we can conclude that $\cos \alpha_3 = \cos \alpha_4,$ so since $\alpha_i \in [0,\pi],$ for $i = 1, 2, 3, 4$ we conclude that $\alpha_3 = \alpha_4.$ Similarly, from the first two equations, we can conclude that $\cos \alpha_1 = \cos \alpha_2,$ and that $\alpha_1 = \alpha_2.$ Therefore, we can reduce the Lagrange multiplier equations above, by setting $\eta = \alpha_1 = \alpha_2$ and $\eta = \alpha_3 = \alpha_4.$ In this case we have the following equations \begin{align*} \cos \zeta - \frac{1}{2} \cos 2\zeta + \lambda &= 0 \\ \frac{1}{2} \cos \eta + \lambda &= 0 \\ \zeta + \eta &= \pi\end{align*} Further reducing to a single variable we see that from the last equation $\eta = \pi - \zeta,$ and the second equation reveals $$\lambda = -\frac{1}{2} \cos \eta = - \frac{1}{2} \cos (\pi - \zeta) = \frac{1}{2} \cos \zeta,$$ so we a single equation $$ \frac{3}{2} \cos \zeta - \frac{1}{2} \cos 2\zeta = \frac{3}{2} \cos \zeta - \cos^2 \zeta + \frac{1}{2} = 0.$$ Solving this quadratic we get $\cos \zeta = \frac{3 - \sqrt{17}}{4},$ since the other root would be greater than $1,$ which in turn leads to $$\zeta = \cos^{-1} \frac{3 - \sqrt{17}}{4} \approx 1.85539928817...$$ From here, since $$\sin \zeta = \sqrt{ 1 - \cos^2 \zeta } = \sqrt{ \frac{3 \sqrt{17} - 5}{8} }$$ we get an optimal $Z$ of \begin{align*}\hat{Z} = Z( \zeta, \zeta, \pi - \zeta, \pi - \zeta ) &= 2\sin \zeta + \sin (\pi - \zeta) - \frac{1}{2} \sin 2\zeta \\&= 3 \sin \zeta - \sin \zeta \cos \zeta \\ &= (3 - \cos \zeta) \sin \zeta \\&= \left( 3 - \frac{3 - \sqrt{17}}{4} \right) \sqrt{ \frac{3 \sqrt{17} - 5}{8} } \\&= \frac{\sqrt{214 + 102 \sqrt{17}} }{8} \approx 3.14880129428....\end{align*}
