Week 2: Closure

Week 2: Closure#

Continue working on the the preparatory exercises and the in-class exercises that you have not yet completed.

Key Concepts#

  • Vector functions of multiple variables

  • Directional derivatives

  • Differentiability

  • The Jacobian matrix (or just the Jacobian)

  • The gradient vector

  • The chain rule

  • The Hessian matrix (or just the Hessian)

If there are still concepts you are unsure about, you should reread the relevant chapters in the textbook or revisit the exercises of the week.

Extra Exercises#

We do not expect you to complete more exercises than those from from the week’s program. The following additional exercises are purely an optional offer for those who want extra practice and challenge.

1: Discontinuity of Heaviside’s step function#

This exercise revisits the topic from week 1: Continuity (or rather, the lack of continuity).

Question a#

Plot Heaviside’s step function given here in Python.

Hint

See the Python demo from week 1.

Question b#

Indicate at which points the function is discontinuous.

Answer

The function is only discontinuous at \(0\).

Question c#

Can you prove that the function is discontinuous at \(x_0=0\)?

Hint

Discontinuity at a point intuitively means that the function “jumps” at that point. However, this is not a definition we can work with. For example, there are many functions that seemingly “jump” everywhere but are still continuous. Thomae’s function is an example of a function that appears to “jump” on the entire real line but is actually continuous at all irrational numbers (and discontinuous at all rational numbers!).

Hint

To prove discontinuity, we must use the precise definition of this property. Find the \(\epsilon-\delta\) technique in the textbook.

Hint

A function \(f\) is discontinuous at \(x_0\) if we can find an \(\epsilon > 0\) such that no \(\delta > 0\) can make \(|x-x_0| < \delta \Rightarrow |f(x)-f(x_0)| < \epsilon\) true.

Written with quantifiers: \(f\) is discontinuous at \(x_0\) if and only if \(\exists \epsilon >0 \forall \delta >0 \exists x : |x-x_0| < \delta \text{ and } |f(x)-f(x_0)| \ge \epsilon\).

Hint

The function is discontinuous at \(0\), so we must consider \(x_0 = 0\). Remember that the Heaviside function satisfies \(f(0) = 1\).

Hint

Choose \(\epsilon = \frac{1}{2}\) (the exact value does not matter as long as it is \(<1\)).

Answer

Let \(\delta > 0\) be an arbitrarily small number, e.g. \(\delta = 10^{-100}\) (remember that \(\delta\) can be arbitrarily small, as long as it is greater than zero). Choose \(x = -\delta/2\). Then \(|x-x_0| = |-\delta/2 - 0| = \delta/2 < \delta\) and \(|f(x)-f(x_0)| = |0 - 1| = 1 \ge 1/2 = \epsilon\).

This shows that the Heaviside function is discontinuous at \(x_0 = 0\). Alternatively, we can think of this as saying that no \(\delta > 0\) can make \(|x-0| < \delta \Rightarrow |f(x)-1| < 1/2\) true.

Above, one could also choose \(x = -\delta/3\) (why would that be possible?). However, the argument falls apart if we choose, for example, \(x = \delta/2\).

2: Orthogonality between a Parametrized level curve and the Gradient#

We consider a function \(f: \mathbb{R}^2 \to \mathbb{R}\) given by

\[\begin{equation*} f(x,y) = x^2 - 4x + y^2. \end{equation*}\]

A level curve of \(f\) is given by \(f(x,y) = c\) for a constant \(c>-4\).

Question a#

Show that the gradient of \(f\) is given by

\[\begin{equation*} \nabla f(x,y) = (2x - 4, 2y). \end{equation*}\]

Answer

Since

\[\begin{equation*} \frac{\partial f}{\partial x}= 2x-4 \quad \text{and} \quad \frac{\partial f}{\partial y}= 2y, \end{equation*}\]

we get

\[\begin{equation*} \nabla f(x,y)= (2x-4, 2y). \end{equation*}\]

Question b#

Consider the parametrized curve \(\pmb{r}(t)\) given by:

\[\begin{equation*} \pmb{r}(t) = (t+2, \sqrt{c - 4 - t^2}), \quad t \in [-\sqrt{c+4}, \sqrt{c+4}]. \end{equation*}\]

Verify that this curve is located on the level curve \(f(x,y) = c\).

Hint

Substitute \(x=t+2\) and \(y=\sqrt{c+4-t^2}\) into the equation of the level curve.

Answer

Write down the equation of the level curve in the form

\[\begin{equation*} (x-2)^2+y^2 = c+4. \end{equation*}\]

With \(x=t+2\), we have \((x-2)^2= t^2\) and \(y^2=c+4-t^2\). Hence:

\[\begin{equation*} (x-2)^2+y^2 = t^2 + \bigl(c+4-t^2\bigr)= c+4. \end{equation*}\]

Since \(f(x,y)= (x-2)^2+y^2-4\), we get

\[\begin{equation*} f(x,y)= (c+4)-4 = c. \end{equation*}\]

Hense, \(\operatorname{im}(\pmb{r})\) is a subset of the level curve \(f(x,y)=c\).

Question c#

Calculate the tangent vector \(\pmb{r}'(t)\) to the curve, and investigate whether it is orthogonal to the gradient \(\nabla f(x,y)\) at every point \(\pmb{r}(t)\), \(t \in \Bigl]-\sqrt{c+4},\sqrt{c+4}\Bigr[\).

Hint

Differentiate \(\pmb{r}(t)\) with respect to \(t\) and then calculate the inner product (which in this context is the dot product) of the tangent vector and the gradient evaluated at the point \(\pmb{r}(t)\).

Answer

Differentiation of

\[\begin{equation*} \pmb{r}(t)= \Bigl(t+2,\; \sqrt{c+4-t^2}\Bigr) \end{equation*}\]

gives

\[\begin{equation*} \pmb{r}'(t)= \left(1,\; -\frac{t}{\sqrt{c+4-t^2}}\right). \end{equation*}\]

The gradient is

\[\begin{equation*} \nabla f(x,y)= (2x-4, 2y) = \Bigl(2(t+2)-4,\, 2\sqrt{c+4-t^2}\Bigr)= \bigl(2t,\, 2\sqrt{c+4-t^2}\bigr). \end{equation*}\]

Calculate the dot product:

\[\begin{equation*} \pmb{r}'(t)\cdot \nabla f(\pmb{r}(t)) = 1\cdot (2t) + \left(-\frac{t}{\sqrt{c+4-t^2}}\right) \cdot \Bigl(2\sqrt{c+4-t^2}\Bigr)= 2t - 2t = 0. \end{equation*}\]

Since the dot product is zero for all \(t\), the tangent vector is orthogonal to the gradient at all points on the parametrized part of the level curve.

Question d#

Can you adjust the parametrization used above to parametrize the entire level curve \(f(x,y)=c\)?

Hint

Consider using an “angle-based” parametrization of the circle by use of sine and cosine.

3: Directional Derivative and the Gradient#

Consider the function

\[\begin{equation*} g(x,y)= x^2+y^2. \end{equation*}\]

Question a#

Show that the directional derivative of \(g\) in the direction of a unit vector \(\pmb{v} \in \mathbb{R}^2\) is given by

\[\begin{equation*} \nabla_{\pmb{v}} g(x,y)= \langle \pmb{v}, (2x, 2y) \rangle. \end{equation*}\]

Question b#

Compute the directional derivative of \(g\) at the point \((2,1)\) in the following directions:

  • \(\pmb{e}_1 = (1,0)\) (first coordinate direction).

  • \(\pmb{e}_2 = (0,1)\) (second coordinate direction).

  • \(\pmb{v}_1 = \left(\frac{1}{\sqrt{2}}, \frac{1}{\sqrt{2}}\right)\) (diagonal direction).

  • \(\pmb{v}_2 = \left(\frac{2}{\sqrt{5}}, \frac{1}{\sqrt{5}}\right)\) (another unit direction).

  • The opposite direction of the above directions.

  • In the direction of the gradient \(\frac{\nabla g(2,1)}{||\nabla g(2,1)||}\).

  • In the opposite gradient direction \(-\frac{\nabla g(2,1)}{||\nabla g(2,1)||}\).

Hint

Calculate \(\nabla g(2,1)\) and its norm.

Answer

The gradient at \((2,1)\) is:

\[\begin{equation*} \nabla g(2,1) = (4,2). \end{equation*}\]

The norm of the gradient is:

\[\begin{equation*} ||\nabla g(2,1)|| = \sqrt{4^2 + 2^2} = \sqrt{20} = 2\sqrt{5}. \end{equation*}\]

For \(\pmb{v}_1\):

\[\begin{equation*} \nabla_{\pmb{v}_1} g(2,1) = (4,2) \cdot \left(\frac{1}{\sqrt{2}}, \frac{1}{\sqrt{2}}\right) = \frac{4}{\sqrt{2}} + \frac{2}{\sqrt{2}} = 3\sqrt{2}. \end{equation*}\]

For \(\pmb{v}_2\):

\[\begin{equation*} \nabla_{\pmb{v}_2} g(2,1) = (4,2) \cdot \left(\frac{2}{\sqrt{5}}, \frac{1}{\sqrt{5}}\right) = \frac{8}{\sqrt{5}} + \frac{2}{\sqrt{5}} = 2\sqrt{5}. \end{equation*}\]

In the direction of the gradient:

\[\begin{equation*} \nabla_{\text{grad}} g(2,1) = ||\nabla g(2,1)|| = 2\sqrt{5}. \end{equation*}\]

In the opposite gradient direction:

\[\begin{equation*} \nabla_{-\text{grad}} g(2,1) = -||\nabla g(2,1)|| = -2\sqrt{5}. \end{equation*}\]

Question c#

The directional derivative is by definition the rate of change of the function in a given direction. Which direction gives the largest and the smallest directional derivatives, respectively?

Hint

Compare the calculated values and note which one corresponds to the norm of the gradient.

Answer

The largest directional derivative is

\[\begin{equation*} 2\sqrt{5}, \end{equation*}\]

which corresponds to \(||\nabla g(2,1)||\). This maximum value is achieved when we move in the direction of the gradient, which confirms that the gradient “points in the direction of the steepest ascent”. Similarly, the smallest (i.e., most negative) directional derivative is \(-2\sqrt{5}\), obtained in the opposite direction.