{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3eb2db55",
   "metadata": {},
   "source": [
    "# Week 2: Differentiability"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd8c37f8",
   "metadata": {},
   "source": [
    "## Key Terms"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6144ef39",
   "metadata": {},
   "source": [
    "* Vector functions of multiple variables\n",
    "* Directional derivative\n",
    "* Differentiability\n",
    "* The Jacobian matrix, the gradient vector\n",
    "* The chain rule\n",
    "* The Hessian matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d94c0b9d",
   "metadata": {},
   "source": [
    "## Preparation and Syllabus"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa4d8d01",
   "metadata": {},
   "source": [
    "* Long Day: The rest of <a href=\"../_assets/notesvol2.pdf#sec:part-deriv-second\">Chapter 3</a>  \n",
    "* Short Day: [Theme Exercise 1](/tema_dir/theme1.md)\n",
    "* Python demo for [week 2](/demos/demo02_differentiability.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc4da125",
   "metadata": {},
   "source": [
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ddc4903",
   "metadata": {},
   "source": [
    "## Exercises -- Long Day"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "488b8064",
   "metadata": {},
   "source": [
    "### 1: Level Curves and Directional Derivative for Scalar Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51c581da",
   "metadata": {},
   "source": [
    "A function $f:\\mathbb{R}^2\\rightarrow\\mathbb{R}$ is given by the expression \n",
    "\\begin{equation*}\n",
    "    f(x,y)=x^2+y^2.\n",
    "\\end{equation*}\n",
    "\n",
    "Another function $g:\\mathbb{R}^2\\rightarrow\\mathbb{R}$ is given by the expression \n",
    "\\begin{equation*}\n",
    "    g(x,y)=x^2-4x+y^2.\n",
    "\\end{equation*}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65000aff",
   "metadata": {},
   "source": [
    "#### Question a"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5f32d1d",
   "metadata": {},
   "source": [
    "Decribe the level curves given by $f(x,y)=c$ for the values $c\\in\\{1,2,3,4,5\\}$."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8c611e0",
   "metadata": {},
   "source": [
    "#### Question b"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcf4ca19",
   "metadata": {},
   "source": [
    "Determine the gradient of $f$ at the point $(1,1)$ and determine the directional derivative of $f$ at the point $(1,1)$ in the direction that is given by the unit direction vector $\\pmb{e}=(1,0)$."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b381ac4",
   "metadata": {},
   "source": [
    "#### Question c"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30c6f7dc",
   "metadata": {},
   "source": [
    "Describe the level curves given by $g(x,y)=c$ for the values $c \\in\\{-3,-2,-1,0,1\\}$."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5d202bc",
   "metadata": {},
   "source": [
    "#### Question d"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b96ce58",
   "metadata": {},
   "source": [
    "Compute the gradient of $g$ at the point $(1,2)$ and compute the directional derivative of $g$ at the point $(1,2)$ in the direction towards the origin $(0,0)$."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21a20e0e",
   "metadata": {},
   "source": [
    "### 2: Jacobian Matrices for Different Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85a36b71",
   "metadata": {},
   "source": [
    "We define functions below of the form $\\pmb{f}: \\mathbb{R}^n \\to \\mathbb{R}^k$, where $n$ and $k$ can be read from the functional expression.\n",
    "\n",
    "#### Question a"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f887c4d",
   "metadata": {},
   "source": [
    "1. Let ${f}(x_1, x_2, x_3) = x_1^2x_2 + 2x_3$. Compute the Jacobian matrix $J_{f}(\\pmb{x})$ and evaluate it at the point $\\pmb{x} = (1, -1, 3)$. Confirm that the Jacobian matrix of a scalar function of multiple variables only has one row.\n",
    "1. Let $\\pmb{f}(x) = (3x, x^2, \\sin(2x))$. Compute the Jacobian matrix $J_{\\pmb{f}}(x)$ and evaluate it at the point $x = 2$. Confirm that the Jacobian matrix of a vector function of one variable only has one column.\n",
    "1. Let $\\pmb{f}(x_1, x_2) = (x_1^2, -3x_2, 12x_1)$. Compute the Jacobian matrix $J_{\\pmb{f}}(\\pmb{x})$ and evaluate it at the point $\\pmb{x} = (2, 0)$.\n",
    "1. Let $\\pmb{f}(x_1, x_2, x_3) = (x_2 \\sin(x_3), 3x_1x_2 \\ln(x_3))$. Compute the Jacobian matrix $J_{\\pmb{f}}(\\pmb{x})$ and evaluate it at the point $\\pmb{x} = (-1, 3, 2)$.\n",
    "1. Let $\\pmb{f}(x_1, x_2, x_3) = (x_1 e^{x_2}, 3x_2 \\sin(x_2), -x_1^2 \\ln(x_2 + x_3))$. Compute the Jacobian matrix $J_{\\pmb{f}}(\\pmb{x})$ and evaluate it at the point $\\pmb{x} = (1, 0, 1)$.\n",
    "\n",
    "<!-- engelsk\n",
    "1. Let $\\pmb{f}(x_1, x_2, x_3) = x_1^2x_2 + 2x_2$. Compute $J_{\\pmb{f}}(\\pmb{x})$ and evaluate it at $\\pmb{x} = (1, -1, 3)$. Confirm that the Jacobian of a function of several variables has only one row.\n",
    "1. Let $\\pmb{f}(x) = (3x, x^2, \\sin(2x))$. Compute $J_{\\pmb{f}}(\\pmb{x})$ and evaluate it at $\\pmb{x} = 2$. Confirm that the Jacobian of a vector valued function variables has only one column.\n",
    "1. Let $\\pmb{f}(x_1, x_2) = (x_1^2, -3x_2, 12x_1)$. Compute $J_{\\pmb{f}}(\\pmb{x})$ and evaluate it at $\\pmb{x} = (2, 0)$.\n",
    "1. Let $\\pmb{f}(x_1, x_2, x_3) = (x_2 \\sin(x_3), 3xx_2 \\ln(x_3))$. Compute $J_{\\pmb{f}}(\\pmb{x})$ and evaluate it at $\\pmb{x} = (-1, 3, 2)$.\n",
    "1. Let $\\pmb{f}(x_1, x_2, x_3) = (x_1 e^x_2, 3x_2 \\sin(x_2), -x_1^2 \\ln(x_2 + x_3))$. Compute $J_{\\pmb{f}}(\\pmb{x})$ and evaluate it at $\\pmb{x} = (1, 0, 1)$.\n",
    "-->"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5550f9e9",
   "metadata": {},
   "source": [
    "#### Question b"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fa6db83",
   "metadata": {},
   "source": [
    "All functions from the previous question are differentiable. How can one argue for this? For which of the functions can we determine the Hessian matrix? Compute the Hessian matrix of the functions for which it is defined."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5164828f",
   "metadata": {},
   "source": [
    "#### Question c"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f579311",
   "metadata": {},
   "source": [
    "Let $\\pmb{v} = (1,1,1)$. Normalise the vector $\\pmb{v}$ and denote the result by $\\pmb{e}$. Check that $||\\pmb{e}||=1$. Compute the directional derivative of the scalar function ${f}(x_1, x_2, x_3) = x_1^2x_2 + 2x_3$ at the point $\\pmb{x} = (1, -1, 3)$ in the direction along $\\pmb{v}$. Then compute $J_f(\\pmb{x}) \\pmb{e}$. Compare with the directional derivative. Are they equal? If so, is that a coincidence?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f277551",
   "metadata": {},
   "source": [
    "### 3: Description of Sets in the Plane"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58a60d8d",
   "metadata": {},
   "source": [
    "Draw in each of the four below cases a sketch of the given set $\\,A\\,$, its interior $\\,A^{\\circ}\\,$, its boundary $\\,\\partial A\\,$ and its closure $\\,\\bar{A}\\,$. Investigate further whether $\\,A\\,$ is open, closed or neither. Finally, state whether $\\,A\\,$ is bounded or not.\n",
    "\n",
    "1. $\\{(x,y)\\,\\vert\\, xy\\neq 0\\}$  \n",
    "1. $\\{(x,y)\\,\\vert\\, 0<x<1\\,\\,\\,\\mathrm{and}\\,\\,\\,1\\leq y\\leq 3\\}$  \n",
    "1. $\\{(x,y)\\,\\vert\\, y\\geq x^2 \\,\\,\\,\\mathrm{and}\\,\\,\\,y<2 \\}$  \n",
    "1. $\\{(x,y)\\,\\vert\\, x^2+y^2-2x+6y\\leq 15 \\}$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "674ac27c",
   "metadata": {},
   "source": [
    "### 4: All Linear Maps from $\\mathbb{R}^n$ to $\\mathbb{R}$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "469d675c",
   "metadata": {},
   "source": [
    "Let $L: \\mathbb{R}^n \\to \\mathbb{R}$ be a (arbitrary) linear map. Let $e = \\pmb{e}_1, \\pmb{e}_2, \\dots, \\pmb{e}_n$ be the standard basis of $\\mathbb{R}^n$, and let $\\beta$ be the standard basis of $\\mathbb{R}$. Remember the standard basis from [Mathematics 1a](https://math1a.compute.dtu.dk/_assets/enotesvol1.pdf#prop:standardbasisFn).\n",
    "Note that since the dimension of $\\mathbb{R}$ (over $\\mathbb{R}$) is 1, the standard basis of $\\mathbb{R}$ is just the number $1$. \n",
    "\n",
    "Show that a column vector $\\pmb{c} \\in \\mathbb{R}^n$ exists such that \n",
    "\\begin{equation*}\n",
    "   L(\\pmb{x}) = \\pmb{c}^T \\pmb{x} = \\langle \\pmb{x}, \\pmb{c} \\rangle  ,\n",
    "\\end{equation*}\n",
    "where $\\langle \\cdot, \\cdot \\rangle$ denote the usual inner product on $\\mathbb{R}^n$. (The column vector is uniquely given, but it is not a part of this exercise to argue for that)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b8e0776",
   "metadata": {},
   "source": [
    "### 5: Linear(?) Vector Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b740cff",
   "metadata": {},
   "source": [
    "We consider the following two functions: \n",
    "\n",
    "1. $f: \\mathbb{R}^{2 \\times 2} \\to \\mathbb{R}^{2 \\times 2}, f(X) = C X B$, where $C = \\mathrm{diag}(2,1) \\in \\mathbb{R}^{2 \\times 2}$ and $B = \\begin{bmatrix} 1 & 1 \\\\ 0 & 1 \\end{bmatrix}$.\n",
    "1. $g: \\mathbb{R}^n \\to \\mathbb{R}, g(\\pmb{x}) = \\pmb{x}^T A \\pmb{x}$, where $A$ is an $n \\times n$ matrix (and not the zero matrix)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbe15588",
   "metadata": {},
   "source": [
    "Determine for each function whether it is a linear map. If the map is linear, then find the [mapping matrix](https://math1a.compute.dtu.dk/_assets/enotesvol1.pdf#lem:matrix_repr) with respect to, respectively:\n",
    "\n",
    "1. the standard basis $E=\\begin{bmatrix} 1 & 0 \\\\ 0 & 0 \\end{bmatrix}, \\begin{bmatrix} 0 & 1 \\\\ 0 & 0 \\end{bmatrix}, \\begin{bmatrix} 0 & 0 \\\\ 1 & 0 \\end{bmatrix}, \\begin{bmatrix} 0 & 0 \\\\ 0 & 1 \\end{bmatrix}$ in $\\mathbb{R}^{2 \\times 2}$. Remember [this example from Math1a](https://math1a.compute.dtu.dk/_assets/enotesvol1.pdf#ex:matricesEij).\n",
    "1. the standard basis $e$ in $\\mathbb{R}^n$. Remember [this result from Math1a](https://mat1a.compute.dtu.dk/_assets/enotesvol1.pdf#prop:standardbasisFn)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "565651d3",
   "metadata": {},
   "source": [
    "### 6: The Chain Rule for an Altitude Function"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1dd9551",
   "metadata": {},
   "source": [
    "We consider a real function of two real variables given by the expression\n",
    "\\begin{equation*}\n",
    "    f(x,y)=\\ln(9-x^2-y^2).\n",
    "\\end{equation*}\n",
    "\n",
    "#### Question a"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4303cc0",
   "metadata": {},
   "source": [
    "Determine the domain of $f$, and characterise it using terms such as open, closed, bounded, unbounded."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcffd4f9",
   "metadata": {},
   "source": [
    "We not consider a parametrized curve $\\pmb{r}$ in the $\\,(x,y)$ plane given by\n",
    "\\begin{equation*}\n",
    "    \\pmb{r}(u)=(u,u^3)\\,,\\,u\\in \\left[-1.2\\,,\\,1.2\\right].\n",
    "\\end{equation*}\n",
    "\n",
    "#### Question b"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c65ecd66",
   "metadata": {},
   "source": [
    "Which curve are we dealing with here (you are familiar with its *equation*!)?\n",
    "\n",
    "We now consider the composite function \n",
    "\\begin{equation*}\n",
    "    h(u)=f(\\pmb{r}(u)).\n",
    "\\end{equation*}\n",
    "\n",
    "#### Question c"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ecb4eb8",
   "metadata": {},
   "source": [
    "Why is it reasonable to call $h$ an *altitude function*?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8bca373",
   "metadata": {},
   "source": [
    "#### Question d"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb2c0644",
   "metadata": {},
   "source": [
    "Determine $h'(1)\\,$ using two different methods: \n",
    "1. Determine a functional expression for $h(u)$, and differentiate it as usual. \n",
    "1. Use <a href=\"../_assets/notesvol2.pdf#cor:chain-rule-simple-Rn-R\">the chain rule</a> in Section 3.7. \n",
    "\n",
    "\n",
    "\n",
    "### 7: Partial Derivatives but not Differentiable"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d06c5320",
   "metadata": {},
   "source": [
    "The function $f:\\mathbb{R}^2 \\to \\mathbb{R}$ where \n",
    "\\begin{equation*}\n",
    "    f(x_1,x_2)=x_1^2-4x_1+x_2^2\n",
    "\\end{equation*}\n",
    "is given.\n",
    "\n",
    "#### Question a"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41b288e5",
   "metadata": {},
   "source": [
    "Let $\\pmb{x}_0 = (x_1,x_2) \\in \\mathbb{R}^2$ be an arbitrary point. Justify that $f$ is differentiable at $\\pmb{x}_0$, and compute the gradient of $f$ at $\\pmb{x}_0$.\n",
    "\n",
    "> *Hard version*: Solve the task direction from <a href=\"../_assets/notesvol2.pdf#def:diff-scalar-fun-several-var\">the definition</a> of differentiability in Section 3.6. \n",
    "\n",
    "> *Soft version*: Use the result in <a href=\"../_assets/notesvol2.pdf#thm:partial_deriv-implies-diff\">this theorem</a>."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9534b29",
   "metadata": {},
   "source": [
    "#### Question b"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f98751be",
   "metadata": {},
   "source": [
    "To conclude differentiability based on the partial derivatives, according to <a href=\"../_assets/notesvol2.pdf#thm:partial_deriv-implies-diff\">this theorem</a>, it is required that the partial derivatives are continuous. Why is it not enough that the partial derivatives exist? We will be investigating this question via concrete example. But first we generalize a (from highschool) well-known statement about a function of one variable: If it is differentiable at a point, then it is also continuous at that point."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8457fdc",
   "metadata": {},
   "source": [
    "Show that if a function of two variables is differentiable at a point $\\pmb{x}_0$, then it is also continuous at that point. \n",
    "\n",
    "\n",
    "And now for the example. We consider the function\n",
    "\n",
    "\\begin{equation*}\n",
    "f(x_1,x_2) =  \n",
    "\\begin{cases}\n",
    "    \\frac{x_1^2x_2}{x_1^4+x_2^2}, & \\text{for } (x_1,x_2) \\neq (0,0) \\\\\n",
    "    0, & \\text{for } (x_1,x_2)=(0,0).\n",
    "\\end{cases}\n",
    "\\end{equation*}\n",
    "\n",
    "\n",
    "#### Question c\n",
    "\n",
    "Show that the partial derivatives of $f$ exist at $(0,0)$, but that $f$ is not differentiable at this point."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c31cd0a",
   "metadata": {},
   "source": [
    "### 8: The Generalized Chain Rule"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5b8aa40",
   "metadata": {},
   "source": [
    "In this exercise we will be using the theorem: <a href=\"../_assets/notesvol2.pdf#thm:diff_gen_chain_rule\">Generalized chain rule</a>.\n",
    "\n",
    "Given functions:\n",
    "1. $\\pmb{g} : \\mathbb{R}^3 \\to \\mathbb{R}^2$ defined by $\\pmb{g}(x_1, x_2, x_3) = (g_1(x_1, x_2, x_3), g_2(x_1, x_2, x_3))$, where:\n",
    "    \\begin{align*}\n",
    "        g_1(x_1, x_2, x_3) &= x_1^2 + x_2^2 + x_3^2, \\\\\n",
    "        g_2(x_1, x_2, x_3) &= e^{x_1 + x_2} \\, \\cos(x_3).\n",
    "    \\end{align*}\n",
    "1. $f : \\mathbb{R}^2 \\to \\mathbb{R}$ defined by $f(y_1, y_2) = y_1 \\, \\sin(y_2)$.\n",
    "1. The composition of these functions: $h = f \\circ \\pmb{g}$.  \n",
    "\n",
    "We will in this exercise compute the Jacobian matrix of $h$ (with respect to the variables $x_1, x_2,$ and $x_3$) using the generalized chain rule. You may carry out the computations in SymPy."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a26bd66",
   "metadata": {},
   "source": [
    "#### Question a"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb05f7fb",
   "metadata": {},
   "source": [
    "Find the functional expression of $h$ as well as the domain and co-domain. Compute the gradient of $h$."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02d32da9",
   "metadata": {},
   "source": [
    "#### Question b"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dfb28c5d",
   "metadata": {},
   "source": [
    "Compute the Jacobian matrix of $\\pmb{g}$. Compute the Jacobian matrix of $f$. What is the connection between the gradient and the Jacobian matrix of $f$?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7df5f65d",
   "metadata": {},
   "source": [
    "#### Question c"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "740648de",
   "metadata": {},
   "source": [
    "Now use the chain rule and the Jacobian matrices from the previous question to find the Jacobian matrix of $h$. Compare with the answer to [Question a](exercise:den-generaliserede-kæderegel:spm-a)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "240d0b45",
   "metadata": {},
   "source": [
    "### 9: Gradient Vector Field and Hessian Matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63382b32",
   "metadata": {},
   "source": [
    "#### Question a"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4516e636",
   "metadata": {},
   "source": [
    "The gradient vector of $f(x_1, x_2) = x_1^2 \\sin(x_2)$ is $\\nabla f(\\pmb{x})=(2x_1 \\sin(x_2),x_1^2 \\cos(x_2))$. The gradient vector can thus be considered as a map $\\nabla f : \\mathrm{dom}(f) \\to \\mathbb{R}^2$. Write down the map as a function (where you state $\\mathrm{dom}(f)$) and plot it as a vector field."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8178b1cc",
   "metadata": {},
   "source": [
    "#### Question b"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "241c9c31",
   "metadata": {},
   "source": [
    "Now compute the Jacobian matrix of $\\nabla f : \\mathbb{R}^2 \\to \\mathbb{R}^2$ at the point $(x_1,x_2)$."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4f6b8c1",
   "metadata": {},
   "source": [
    "#### Question c"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b1c1941",
   "metadata": {},
   "source": [
    "Compute the Hessian matrix of $f : \\mathbb{R}^2 \\to \\mathbb{R}$ at the point $(x_1,x_2)$, and compare with the answer to the previous question."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06e44502",
   "metadata": {},
   "source": [
    "___\n",
    "\n",
    "## Theme Exercise -- Short Day\n",
    "\n",
    "Today we will work through [Theme Exercise 1](/tema_dir/theme1.md)."
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "formats": "md:myst"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}