D.48 About Lagrangian multipliers

This note will derive the Lagrangian multi­pliers for an example problem. Only calculus will be used. The example problem will be to find a stationary point of a function $f$ of four variables if there are two constraints. Different numbers of variables and constraints would work out in similar ways as this example.

The four variables that example function $f$ depends on will be denoted by $x_1$, $x_2$, $x_3$, and $x_4$. The two constraints will be taken to be equations of the form $g(x_1,x_2,x_3,x_4)$ $\vphantom0\raisebox{1.5pt}{$=$}$ 0 and $h(x_1,x_2,x_3,x_4)$ $\vphantom0\raisebox{1.5pt}{$=$}$ 0, for suitable functions $g$ and $h$. Constraints can always be brought in such a form by taking everything in the constraint’s equation to the left-hand side of the equals sign.

So the example problem is:

$$\begin{eqnarray*} \mbox{stationarize:} && f(x_1,x_2,x_3,x_4) \\ \mbox{subject to:} && g(x_1,x_2,x_3,x_4)=0,\ h(x_1,x_2,x_3,x_4)=0 \end{eqnarray*}$$

Stationarize means to find locations where the function has a minimum or a maximum, or any other point where it does not change under small changes of the variables $x_1,x_2,x_3,x_4$ as long as these satisfy the constraints.

The first thing to note is that rather than considering $f$ to be a function of $x_1,x_2,x_3,x_4$, you can consider it instead to be to be a function of $g$ and $h$ and only two additional variables from $x_1,x_2,x_3,x_4$, say $x_3$ and $x_4$:

$$f(x_1,x_2,x_3,x_4)=\tilde f(g,h,x_3,x_4)$$

The reason you can do that is that you should in principle be able to reconstruct the two missing variables $x_1$ and $x_2$ given $g$, $h$, $x_3$, and $x_4$.

As a result, any small change in the function $f$, regardless of constraints, can be written using the expression for a total differen­tial as:

$${\rm d}f = \frac{\partial \tilde f}{\partial g}{\rm d}g ... ...m d}x_3 + \frac{\partial \tilde f}{\partial x_4}{\rm d}x_4.$$

At the desired stationary point, acceptable changes in variables are those that keep $g$ and $h$ constant at zero; they have ${\rm d}{g}$ $\vphantom0\raisebox{1.5pt}{$=$}$ 0 and ${\rm d}{h}$ $\vphantom0\raisebox{1.5pt}{$=$}$ 0. So for $f$ to be stationary under all acceptable changes of variables, you must have that the final two terms are zero for any changes in variables. This means that the partial derivatives in the final two terms must be zero since the changes ${{\rm d}}x_3$ and ${{\rm d}}x_4$ can be arbitrary.

For changes in variables that do go out of bounds, the change in $f$ will not be zero; that change will be given by the first two terms in the right-hand side. So, the erroneous changes in $f$ due to going out of bounds are these first two terms, and if we subtract them, we get zero net change for any arbitrary change in variables:

$${\rm d}f - \frac{\partial \tilde f}{\partial g}{\rm d}g ... ...ac{\partial \tilde f}{\partial h}{\rm d}h = 0 \mbox{ always.}$$

In other words, if we “penalize” the change in $f$ for going out of bounds by amounts ${\rm d}{g}$ and ${\rm d}{h}$ at the rate above, any change in variables will produce a penalized change of zero, whether it stays within bounds or not.

The two derivatives at the stationary point in the expression above are the Lagrangian multi­pliers or penalty factors, call them $\epsilon_1$ $\vphantom0\raisebox{1.5pt}{$=$}$ $\partial\tilde{f}$$\raisebox{.5pt}{$/$}$$\partial{g}$ and $\epsilon_2$ $\vphantom0\raisebox{1.5pt}{$=$}$ $\partial\tilde{f}$$\raisebox{.5pt}{$/$}$$\partial{h}$. In those terms

$${\rm d}f - \epsilon_1 {\rm d}g - \epsilon_2{\rm d}h = 0$$

for whatever is the change in the variables $g,h,x_3,x_4$, and that means for whatever is the change in the original variables $x_1,x_2,x_3,x_4$. Therefore, the change in the penalized function

$$f - \epsilon_1 g - \epsilon_2 h$$

is zero whatever is the change in the variables $x_1,x_2,x_3,x_4$.

In practical appli­cation, explicitly computing the Lagrangian multi­pliers $\epsilon_1$ and $\epsilon_2$ as the derivatives of function $\tilde{f}$ is not needed. You get four equations by putting the derivatives of the penalized $f$ with respect to $x_1$ through $x_4$ equal to zero, and the two constraints provide two more equations. Six equations is enough to find the six unknowns $x_1$ through $x_4$, $\epsilon_1$ and $\epsilon_2$.