Dual spaces and the Fenchel conjugate

Posted on:

\newcommand{\round}[1]{\left( #1 \right)} \newcommand{\curly}[1]{\left\lbrace #1 \right\rbrace} \newcommand{\squarebrack}[1]{\left\lbrack #1 \right\rbrack} \newcommand{\sumi}[2]{\sum\limits_{i=#1}^{#2}} \newcommand{\sumj}[2]{\sum\limits_{j=#1}^{#2}} \newcommand{\sumk}[2]{\sum\limits_{k=#1}^{#2}} \newcommand{\sump}[2]{\sum\limits_{p=#1}^{#2}} \newcommand{\suml}[2]{\sum\limits_{l=#1}^{#2}} \newcommand{\sumn}[2]{\sum\limits_{n=#1}^{#2}} \newcommand{\summ}[2]{\sum\limits_{m=#1}^{#2}} \newcommand{\sumt}[2]{\sum\limits_{t=#1}^{#2}} \newcommand{\Sum}{\sum_{i = 1}^{n}} \newcommand{\Sumi}[1]{\sum\limits_{i = 1}^{#1}} \newcommand{\Sumt}[1]{\sum\limits_{t = 1}^{#1}} \newcommand{\abs}[1]{\left\lvert #1 \right\rvert} \newcommand{\norm}[2]{\left\lVert#2\right\rVert_{#1}} \newcommand{\esqnorm}[1]{\left\lVert#1\right\rVert_2^2} \newcommand{\enorm}[1]{\left\lVert#1\right\rVert_2} \newcommand{\infnorm}[1]{\left\lVert#1\right\rVert_\infty} \newcommand{\opnorm}[1]{\left\lVert#1\right\rVert_\text{op}} \newcommand{\normF}[1]{\left\lVert#1\right\rVert_{\text{F}}} \newcommand{\inner}[1]{\left\langle#1\right\rangle} \newcommand{\ceil}[1]{\left\lceil#1\right\rceil} \newcommand{\floor}[1]{\left\lfloor#1\right\rfloor} \newcommand{\zero}{\mathbf{0}} \newcommand{\one}{\mathbf{1}} \newcommand{\avec}{\mathbf{a}} \newcommand{\bvec}{\mathbf{b}} \newcommand{\cvec}{\mathbf{c}} \newcommand{\dvec}{\mathbf{d}} \newcommand{\e}{\mathbf{e}} \newcommand{\f}{\mathbf{f}} \newcommand{\g}{\mathbf{g}} \newcommand{\h}{\mathbf{h}} \newcommand{\ivec}{\mathbf{i}} \newcommand{\jvec}{\mathbf{j}} \newcommand{\kvec}{\mathbf{k}} \newcommand{\lvec}{\mathbf{l}} \newcommand{\m}{\mathbf{m}} \newcommand{\n}{\mathbf{n}} \newcommand{\ovec}{\mathbf{o}} \newcommand{\p}{\mathbf{p}} \newcommand{\q}{\mathbf{q}} \newcommand{\rvec}{\mathbf{r}} \newcommand{\s}{\mathbf{s}} \newcommand{\tvec}{\mathbf{t}} \newcommand{\uvec}{\mathbf{u}} \newcommand{\vvec}{\mathbf{v}} \newcommand{\w}{\mathbf{w}} \newcommand{\x}{\mathbf{x}} \newcommand{\y}{\mathbf{y}} \newcommand{\z}{\mathbf{z}} \newcommand{\A}{\mathbf{A}} \newcommand{\B}{\mathbf{B}} \newcommand{\C}{\mathbf{C}} \newcommand{\D}{\mathbf{D}} \newcommand{\Emat}{\mathbf{E}} \newcommand{\F}{\mathbf{F}} \newcommand{\G}{\mathbf{G}} \newcommand{\Hmat}{\mathbf{H}} \newcommand{\I}{\mathbf{I}} \newcommand{\J}{\mathbf{J}} \newcommand{\K}{\mathbf{K}} \newcommand{\Lmat}{\mathbf{L}} \newcommand{\M}{\mathbf{M}} \newcommand{\N}{\mathbf{N}} \newcommand{\Omat}{\mathbf{O}} \newcommand{\Pmat}{\mathbf{P}} \newcommand{\Q}{\mathbf{Q}} \newcommand{\Rmat}{\mathbf{R}} \newcommand{\Smat}{\mathbf{S}} \newcommand{\T}{\mathbf{T}} \newcommand{\U}{\mathbf{U}} \newcommand{\V}{\mathbf{V}} \newcommand{\W}{\mathbf{W}} \newcommand{\X}{\mathbf{X}} \newcommand{\Y}{\mathbf{Y}} \newcommand{\Z}{\mathbf{Z}} \newcommand{\SIGMA}{\mathbf{\Sigma}} \newcommand{\LAMBDA}{\mathbf{\Lambda}} \newcommand{\Acal}{\mathcal{A}} \newcommand{\Bcal}{\mathcal{B}} \newcommand{\Ccal}{\mathcal{C}} \newcommand{\Dcal}{\mathcal{D}} \newcommand{\Ecal}{\mathcal{E}} \newcommand{\Fcal}{\mathcal{F}} \newcommand{\Gcal}{\mathcal{G}} \newcommand{\Hcal}{\mathcal{H}} \newcommand{\Ical}{\mathcal{I}} \newcommand{\Jcal}{\mathcal{J}} \newcommand{\Kcal}{\mathcal{K}} \newcommand{\Lcal}{\mathcal{L}} \newcommand{\Mcal}{\mathcal{M}} \newcommand{\Ncal}{\mathcal{N}} \newcommand{\Ocal}{\mathcal{O}} \newcommand{\Pcal}{\mathcal{P}} \newcommand{\Qcal}{\mathcal{Q}} \newcommand{\Rcal}{\mathcal{R}} \newcommand{\Scal}{\mathcal{S}} \newcommand{\Tcal}{\mathcal{T}} \newcommand{\Ucal}{\mathcal{U}} \newcommand{\Vcal}{\mathcal{V}} \newcommand{\Wcal}{\mathcal{W}} \newcommand{\Xcal}{\mathcal{X}} \newcommand{\Ycal}{\mathcal{Y}} \newcommand{\Zcal}{\mathcal{Z}} \newcommand{\alphavec}{\boldsymbol{\alpha}} \newcommand{\betavec}{\boldsymbol{\beta}} \newcommand{\gammavec}{\boldsymbol{\gamma}} \newcommand{\deltavec}{\boldsymbol{\delta}} \newcommand{\epsvec}{\boldsymbol{\epsilon}} \newcommand{\etavec}{\boldsymbol{\eta}} \newcommand{\nuvec}{\boldsymbol{\nu}} \newcommand{\tauvec}{\boldsymbol{\tau}} \newcommand{\rhovec}{\boldsymbol{\rho}} \newcommand{\lmbda}{\boldsymbol{\lambda}} \newcommand{\muvec}{\boldsymbol{\mu}} \newcommand{\thetavec}{\boldsymbol{\theta}} \newcommand{\BigO}[1]{\mathcal{O}\round{#1}} \newcommand{\BigOmega}[1]{\Omega\round{#1}} \newcommand{\R}{\mathbb{R}} \newcommand{\Rd}[1]{\mathbb{R}^{#1}} \newcommand{\Natural}{\mathbb{N}} \newcommand{\Complex}{\mathbb{C}} \newcommand{\Integer}{\mathbb{Z}} \newcommand{\Rational}{\mathbb{Q}} \newcommand{\Field}{\mathbb{F}} \newcommand{\E}[1]{\mathbb{E}\squarebrack{#1}} \newcommand{\Exp}[2]{\mathbb{E}_{#1}\squarebrack{#2}} \newcommand{\Prob}[1]{P\curly{#1}} \newcommand{\Var}[1]{\mathrm{Var}\squarebrack{#1}} \newcommand{\inv}[1]{\frac{1}{#1}} \newcommand{\indicator}[2]{\mathbbm{1}_{#1}\squarebrack{#2}} \newcommand{\Tr}[1]{\text{Tr}\squarebrack{#1}} \newcommand{\BOX}[1]{\fbox{\parbox{\linewidth}{\centering#1}}} \newcommand{\textequal}[1]{\stackrel{#1}{=}} \newcommand{\textleq}[1]{\stackrel{#1}{\leq}} \newcommand{\textgeq}[1]{\stackrel{#1}{\geq}} \newcommand{\dd}[2]{\frac{d #1}{d #2}} \newcommand{\ddn}[3]{\frac{d^{#1} #2}{d #3^{#1}}} \newcommand{\dodo}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\dodon}[3]{\frac{\partial^{#1} #2}{\partial {#3}^{#1}}}Dual spaces lie at the core of linear algebra and allow us to reason formally about duality in mathematics. Duality appears naturally in measure theory, functional analysis, and mathematical optimization. In this post, I explore dual spaces and their interpretation in linear algebra, motivated by the so-called convex conjugate, or Fenchel conjugate, in mathematical optimization.

For the sake of abstractness and general applicability, we will consider that we are in a general field F\Field which for example can be R\R, C\Complex etc. Our space of vectors, is the vector space V\Vcal, which in general can be subspace of Fd\Field^d for some d∈Nd\in\Natural for example.

Dual spaces

Dual space - The dual space Vβˆ—\Vcal^{*} of the vector space V\Vcal, is the set of all linear transformations ff from V\Vcal to F\Field denoted as Vβˆ—=L(V,F)\Vcal^{*} = \mathscr{L}(\Vcal,\Field).

Linear functionals - An element f:Vβ†’Ff:\Vcal\to\Field of Vβˆ—\Vcal^{*} is called a linear functional, which takes in a vector x∈Vx\in\Vcal, and outputs an element in F\Field.

Note that if f,g∈Vβˆ—f,g\in\Vcal^{*}, then Ξ±f+Ξ²g\alpha f + \beta g is also in Vβˆ—\Vcal^{*} for Ξ±,β∈F\alpha,\beta\in\Field and would be defined as (Ξ±f+Ξ²g)(x)=Ξ±f(x)+Ξ²g(x)(\alpha f + \beta g)(x) = \alpha f(x) + \beta g(x) for all x∈Vx\in\Vcal. Let us look at a few examples :

  1. If V=Pn\Vcal = \Pcal_n is the space of all univariate polynomial over the field R\R with degree n∈Nn\in\Natural, then ff defined as f(p)=p(1)+2pβ€²(0)f(p) = p(1)+2p’(0), is a linear functional contained in Vβˆ—=L(Pn,R)\Vcal^{*} = \mathscr{L}(\Pcal_n,\R).
  2. If V=RmΓ—n\Vcal = \Rd{m\times n} be the space of all real matrices of size mΓ—nm\times n, then ff defined as f(M)=Tr[M]f(\M) = \Tr{\M}, is a linear functional contained in Vβˆ—=L(RmΓ—n,R)\Vcal^{*} = \mathscr{L}(\Rd{m\times n},\R).
  3. If V=C([βˆ’Ο€,Ο€])\Vcal = \Ccal\round{\squarebrack{-\pi,\pi}} is the space of all real valued continuous functions on [βˆ’Ο€,Ο€]\squarebrack{-\pi,\pi}, then ff defined as f(g)=βˆ«βˆ’Ο€Ο€g(x)cos⁑(2x)dxf(g) = \int_{-\pi}^{\pi} g(x)\cos(2x)dx, is a linear functional contained in Vβˆ—=L([βˆ’Ο€,Ο€],R)\Vcal^{*} = \mathscr{L}(\squarebrack{-\pi,\pi},\R). This function essentially outputs the 2nd Fourier coefficient of its input function.

Dual basis

Let V\Vcal be a finite dimensional vector space, and Ξ²={vi}i=1n\beta = \curly{v_i}_{i=1}^n be a basis of V\Vcal. Therefore, to determine the action of any linear functional f∈Vβˆ—f\in\Vcal^{*}, it is enough to know its action on the basis vectors {vi}i=1n\curly{v_i}_{i=1}^n, i.e., it is enough to know what {f(vi)∈F}i=1n\curly{f(v_i)\in\Field}_{i=1}^n are.

Dual basis - If V\Vcal is finite dimensional and Ξ²={vi}i=1n\beta = \curly{v_i}_{i=1}^n is a basis of V\Vcal, then Ξ²βˆ—={fi}i=1n\beta^{*} = \curly{f_i}_{i=1}^n, the set of functionals defined by fi(vj)=Ξ΄ijΒ βˆ€Β i,j∈[n]f_i(v_j) = \delta_{ij}\ \forall\ i,j\in\squarebrack{n}, is the corresponding basis of Vβˆ—\Vcal^{*} also known as a dual basis of V\Vcal.

To verify that Ξ²βˆ—\beta^{*} is indeed a basis of Vβˆ—\Vcal^{*}, we just need to check the following

  1. Ξ²βˆ—\beta^{*} is a linearly independent set of linear functionals, i.e., if βˆ‘i=1naifi=f0\sumi{1}{n}a_if_i = f_0 for some set {aj∈F}j=1n\curly{a_j\in\Field}_{j=1}^n, then aj=0Β βˆ€Β j∈[n]a_j=0\ \forall\ j\in\squarebrack{n}, where f0f_0 is the 00 functional. To verify this, we just need to check this condition for all the basis elements of V\Vcal, v∈βv \in\beta, which gives us βˆ‘i=1naifi(vj)=0Β βˆ€Β j∈[n]\sumi{1}{n}a_if_i(v_j) = 0\ \forall\ j \in\squarebrack{n}, and implies that aj=0Β βˆ€Β j∈[n]a_j = 0\ \forall\ j \in\squarebrack{n} establishing the linear independence of the set Ξ²βˆ—\beta^{*}.
  2. Ξ²βˆ—\beta^{*} spans Vβˆ—\Vcal^{*}, i.e., for any f∈Vβˆ—f\in\Vcal^{*}, we have some set of elements {aj∈F}j=1n\curly{a_j\in\Field}_{j=1}^n such that f(v)=βˆ‘i=1naifi(v)Β βˆ€Β v∈Vf(v) = \sumi{1}{n}a_if_i(v)\ \forall\ v\in\Vcal. To check this, we evaluate ff on basis vectors vj∈βv_j\in\beta for j∈[n]j\in\squarebrack{n}, which gives us f(vj)=βˆ‘i=1naifi(vj)=βˆ‘i=1naiΞ΄ij=ajΒ βˆ€Β j∈[n]f(v_j) = \sumi{1}{n}a_if_i(v_j) = \sumi{1}{n}a_i\delta_{ij} = a_j\ \forall\ j\in\squarebrack{n}, or, aj=f(vj)Β βˆ€Β j∈[n]a_j = f(v_j)\ \forall\ j \in\squarebrack{n}. Therefore, if ff is in the span of Ξ²βˆ—\beta^{*}, then it must be the case that the coefficients satisfy aj=f(vj)Β βˆ€Β j∈[n]a_j=f(v_j)\ \forall\ j \in \squarebrack{n}. Now, if g=βˆ‘i=1nf(vi)fig = \sumi{1}{n}f(v_i)f_i, then to show that g=fg=f, it is enough to show that they agree on the basis vectors {vj}\curly{v_j} of V\Vcal, that is f(vi)=g(vi)f(v_i)=g(v_i). This is simply true by the construction of {fi}i=1n\curly{f_i}_{i=1}^n, therefore Ξ²βˆ—\beta^{*} spans Vβˆ—\Vcal^{*}.

This establishes that Ξ²βˆ—={fi}i=1n\beta^{*} = \curly{f_i}_{i=1}^n is indeed a basis of Vβˆ—\Vcal^{*}, and any f∈Vβˆ—f\in\Vcal^{*} can be decomposed into the basis of Vβˆ—\Vcal^{*} as

f=βˆ‘i=1nf(vi)fi\begin{equation} f = \sumi{1}{n}f(v_i)f_i \end{equation}

where Ξ²={vi}\beta = \curly{v_i} is the basis of V\Vcal. The existence of dual basis Ξ²βˆ—\beta^{*} and the relation with Ξ²\beta also shows that V\Vcal and Vβˆ—\Vcal^{*} are isomorphic.

Transpose

The existence of the dual basis Ξ²βˆ—\beta^{*} helps us define the concept of transpose in linear algebra which is extensively used in application.

Transpose - Let T:Vβ†’WT:\Vcal\to\Wcal be a linear transformation from the vector space V\Vcal to the vector space W\Wcal, then T⊀:Wβˆ—β†’Vβˆ—T^\top:\Wcal^{*}\to\Vcal^{*} is a linear transformation from Wβˆ—\Wcal^{*} to Vβˆ—\Vcal^{*} that takes in a linear functional g∈Wβˆ—g\in\Wcal^{*}, and outputs another linear functional T⊀(g)∈Vβˆ—T^\top(g)\in\Vcal^{*} defined as T⊀(g)(x)=g(T(x))Β βˆ€Β x∈VT^\top(g)(x) = g\round{T(x)}\ \forall\ x\in\Vcal.

xβ†’TT(x)β†’g∈Wβˆ—Β g(T(x))≑xβ†’T⊀(g)∈Vβˆ—g(T(x))Vβ†’TWβ†’g∈Wβˆ—Β FVβ†’T⊀(g)∈Vβˆ—F \begin{align} x &\xrightarrow{T} T(x) &\xrightarrow{g\in\Wcal^{*}}&\ g(T(x)) &\qquad\equiv\qquad& x &\xrightarrow{T^\top(g)\in\Vcal^*}& g(T(x))\\ \Vcal &\xrightarrow{T}\Wcal &\xrightarrow{g\in\Wcal^*}&\ \Field &\qquad\qquad& \Vcal &\xrightarrow{T^\top(g)\in\Vcal^*}& \Field \end{align}

Theorem - Let T:Vβ†’WT:\Vcal\to\Wcal be a linear transformation from vector space V\Vcal to vector space W\Wcal. Let Ξ²={vi}i=1n\beta = \curly{v_i}_{i=1}^n and Ξ²βˆ—={fi}i=1n\beta^{*} = \curly{f_i}_{i=1}^n be the basis of V\Vcal and Vβˆ—\Vcal^{*} respectively, and Ξ³={wj}j=1m\gamma = \curly{w_j}_{j=1}^m and Ξ³βˆ—={gj}j=1m\gamma^{*} = \curly{g_j}_{j=1}^m be the basis of W\Wcal and Wβˆ—\Wcal^{*} respectively. Let A=[T]Ξ²Ξ³A = \squarebrack{T}_{\beta}^\gamma be the matrix which transforms a vector in V\Vcal to a vector in W\Wcal. Then [T⊀]Ξ³βˆ—Ξ²βˆ—=A⊀\squarebrack{T^\top}_{\gamma^{*}}^{\beta^{*}}=A^\top.

The matrix of [T⊀]Ξ³βˆ—Ξ²βˆ—\squarebrack{T^\top}_{\gamma^{*}}^{\beta^{*}} takes in an element of Wβˆ—\Wcal^{*} represented in its basis Ξ³βˆ—\gamma^{*}, and outputs an element in Vβˆ—\Vcal^{*} represented in its basis Ξ²βˆ—\beta^{*}. Therefore, it is sufficient to represent every element in {T⊀(gj)}j=1m\curly{T^\top(g_j)}_{j=1}^m in the basis Ξ²βˆ—\beta^{*} of Vβˆ—\Vcal^{*}. Since any vector ff in Vβˆ—\Vcal^{*} can be expressed as a linear combination of the basis vectors Ξ²βˆ—\beta^{*} as we proved earlier, we have

T⊀(gj)=βˆ‘i=1nT⊀(gj)(vi)fi=βˆ‘i=1ngj(T(vi))fi \begin{align} T^\top(g_j) &= \sumi{1}{n}T^\top(g_j)(v_i)f_i\\ &= \sumi{1}{n}g_j(T(v_i))f_i \end{align}

Therefore, the (i,j)(i,j)-th element of the matrix [T⊀]Ξ³βˆ—Ξ²βˆ—\squarebrack{T^\top}_{\gamma^{*}}^{\beta^{*}} is gj(T(vi))g_j(T(v_i)). Now, the elements of the matrix [T]Ξ²Ξ³\squarebrack{T}_{\beta}^{\gamma} are the representations of the basis vectors Ξ²\beta of V\Vcal, in the basis vectors Ξ³\gamma of W\Wcal. Therefore we have,

T(vi)=βˆ‘k=1mAkiwk⟹gj(T(vi))=gj(βˆ‘k=1mAkiwk)=βˆ‘k=1mAkigj(wk)(∡gΒ is a linear functional)=Aji(∡gj(wk)=Ξ΄jk) \begin{align} T(v_i) &= \sumk{1}{m}A_{ki}w_k\\ \implies g_j(T(v_i)) &= g_j\round{\sumk{1}{m}A_{ki}w_k}\\ &= \sumk{1}{m}A_{ki}g_j(w_k)\qquad(\because g \text{ is a linear functional})\\ &= A_{ji}\qquad(\because g_j(w_k)=\delta_{jk}) \end{align}

Which finally gives us

T⊀(gj)=βˆ‘i=1nAjifi \begin{align} T^\top(g_j) &= \sumi{1}{n}A_{ji}f_i \end{align}

and shows us that the (i,j)(i,j)-th element of the matrix [T⊀]Ξ³βˆ—Ξ²βˆ—\squarebrack{T^\top}_{\gamma^{*}}^{\beta^{*}} is nothing but the (j,i)(j,i)-th element of AA, which implies that the matrix [T⊀]Ξ³βˆ—Ξ²βˆ—=A⊀\squarebrack{T^\top}_{\gamma^{*}}^{\beta^{*}} = A^\top.

Double Dual

If V\Vcal is a vector space, then its dual space Vβˆ—\Vcal^{*} is also a vector space. We can again define the set of all linear transformations from Vβˆ—\Vcal^{*} to F\Field as its dual space Vβˆ—βˆ—=L(Vβˆ—,F)\Vcal^{**} = \mathscr{L}(\Vcal^{*},\Field), which makes it the double dual space of V\Vcal. Unlike Vβˆ—\Vcal^{*} where we needed to define basis, there is an elegant way to go from V\Vcal to Vβˆ—βˆ—\Vcal^{**} via an injective map Ξ¨\Psi. If x∈Vx\in\Vcal, and x^=Ξ¨(x)∈Vβˆ—βˆ—\hat{x} = \Psi(x)\in\Vcal^{**} acts on a linear functional f∈Vβˆ—f\in\Vcal^{*} and evaluates ff at xx returning an element in the field F\Field, i.e.,

fβ†’x^∈Vβˆ—βˆ—f(x)Vβˆ—β†’x^∈Vβˆ—βˆ—F \begin{align} f & \xrightarrow{\hat{x}\in\Vcal^{**}} & f(x)\\ \Vcal^* & \xrightarrow{\hat{x}\in\Vcal^{**}} & \Field \end{align}

Theorem - If V\Vcal is a vector space of finite dimension, then Ξ¨:Vβ†’Vβˆ—βˆ—\Psi : \Vcal\to\Vcal^{**} is an isomorphism. First, we show that Ξ¨\Psi is linear. If x,y∈Vx,y\in\Vcal, f∈Vβˆ—f\in\Vcal^{*}, and c∈Fc\in\Field, then

Ψ(x+cy)(f)=f(x+cy)=f(x)+cf(y)=(Ψ(x)+cΨ(y))(f) \begin{align} \Psi(x+cy)(f) &= f(x+cy)\\ &= f(x)+cf(y)\\ &= (\Psi(x) + c\Psi(y))(f) \end{align}

Next, we show that Ξ¨\Psi is one to one. Suppose x^=0\hat{x} = 0, the zero functional, then x=0x = 0. Let Ξ²={vi}i=1n\beta = \curly{v_i}_{i=1}^n, then x=βˆ‘i=1naivix = \sumi{1}{n}a_iv_i for some {ai∈F}\curly{a_i\in\Field}. Therefore, if Ξ¨(x)=x^=0\Psi(x) = \hat{x}=0, then

Ξ¨(x)=Ξ¨(βˆ‘i=1naivi)=βˆ‘i=1naiΞ¨(vi)⟹Ψ(x)(f)=βˆ‘i=1naiΞ¨(vi)(f)βˆ€Β f∈Vβˆ—=βˆ‘i=1naif(vi) \begin{align} \Psi(x) &= \Psi\round{\sumi{1}{n}a_iv_i}\\ &= \sumi{1}{n}a_i\Psi(v_i)\\ \implies \Psi(x)(f) &= \sumi{1}{n}a_i\Psi(v_i)(f)\qquad \forall\ f \in\Vcal^*\\ &= \sumi{1}{n}a_if(v_i) \end{align}

For f∈{fi}i=1nf\in\curly{f_i}_{i=1}^n, we get ai=0Β βˆ€Β i∈[n]a_i = 0\ \forall\ i \in\squarebrack{n}. This implies that x=βˆ‘i=1naivi=0x = \sumi{1}{n}a_iv_i = 0. Since dim(V)=dim(Vβˆ—βˆ—)\mathrm{dim}(\Vcal) = \mathrm{dim}(\Vcal^{**}), we have that Ξ¨\Psi is one to one and therefore an isomorphism. The isomorphism between V\Vcal and Vβˆ—βˆ—\Vcal^{**}, Ξ¨\Psi, allows us to map each element in V\Vcal to a unique element in Vβˆ—βˆ—\Vcal^{**}.

The infinite sequence of dual spaces

We can continue thinking about the dual of the parent space and there always will exist an isomorphism between alternate dual spaces. Formally, Let us denote V(i+1)βˆ—=(V(i)βˆ—)βˆ—\Vcal^{(i+1)*} = \round{\Vcal^{(i)*}}^{*} for all i∈Ni\in\Natural where V(0)βˆ—=V\Vcal^{(0)*} = \Vcal is a finite dimensional vector space. Let Ξ¨i\Psi_i be the isomorphism between V(iβˆ’1)βˆ—\Vcal^{(i-1)*} and V(i+1)βˆ—\Vcal^{(i+1)*}. Then, in general define x^(i+1)=Ξ¨i(x^(iβˆ’1))\hat{x}^{(i+1)} = \Psi_{i}\round{\hat{x}^{(i-1)}} for x^(iβˆ’1)∈V(iβˆ’1)βˆ—\hat{x}^{(i-1)}\in\Vcal^{(i-1)*} when ii is odd, and f^(i+1)=Ξ¨i(f^(iβˆ’1))\hat{f}^{(i+1)} = \Psi_{i}\round{\hat{f}^{(i-1)}} for f^(iβˆ’1)∈V(iβˆ’1)βˆ—\hat{f}^{(i-1)}\in\Vcal^{(i-1)*} when ii is an even natural number, with x^(0)=x∈V\hat{x}^{(0)} = x\in\Vcal, and f^(1)=f∈Vβˆ—\hat{f}^{(1)} = f\in\Vcal^{*}. Then by the property of these isomorphisms we have

x^(i+1)(f^(i))=Ξ¨i(x^(iβˆ’1))(f^(i))=f^(i)(x^(iβˆ’1))=…=x^(2)(f^(1))=x^(2)(f) \hat{x}^{(i+1)}\round{\hat{f}^{(i)}} = \Psi_i\round{\hat{x}^{(i-1)}}\round{\hat{f}^{(i)}} = \hat{f}^{(i)}\round{\hat{x}^{(i-1)}} = \ldots = \hat{x}^{(2)}\round{\hat{f}^{(1)}} = \hat{x}^{(2)}(f)

when ii is odd, and

f^(i+1)(x^(i))=Ξ¨i(f^(iβˆ’1))(x^(i))=x^(i)(f^(iβˆ’1))=…=f(1)(x)=f(x) \hat{f}^{(i+1)}\round{\hat{x}^{(i)}} = \Psi_{i}\round{\hat{f}^{(i-1)}}\round{\hat{x}^{(i)}} = \hat{x}^{(i)}\round{\hat{f}^{(i-1)}} = \ldots = f^{(1)}(x) = f(x)

when ii is an even natural number. However, note that x^(2)(f)\hat{x}^{(2)}(f) and f(x)f(x) are the same quantities in F\Field, the difference lies in the order of application of these linear functionals, which makes them the dual representation of each other.

Application to Fenchel conjugate

With this formalization, one can reason about the mathematical object, Fenchel conjugate in an elegant way. Let us first define what a Fenchel conjugate of a function is.

Fenchel conjugate - Let Vβˆ—=L(V,F)\Vcal^{*} = \mathscr{L}(\Vcal,\Field) be the dual space of V\Vcal, and βŸ¨Β β‹…Β ,Β β‹…Β βŸ©:Vβˆ—Γ—Vβ†’F\inner{\ \cdot\ ,\ \cdot\ } : \Vcal^{*} \times \Vcal \to \Field denote the dual pairing between the two vector spaces. For a function f:Vβ†’Fβˆͺ{∞F}f : \Vcal\to \Field\cup\curly{\infty_{\Field}}, its convex conjugate fβˆ—:Vβˆ—β†’Fβˆͺ{∞F}f^{*} : \Vcal^{*}\to \Field\cup\curly{\infty_{\Field}} is defined as

fβˆ—(g)=supx∈V{⟨g,xβŸ©βˆ’f(x)}βˆ€Β g∈Vβˆ—\begin{equation} f^{*}(g) = \sup\limits_{\x\in\Vcal}\curly{\inner{g,\x} - f(\x)} \qquad \forall\ g\in\Vcal^{*} \end{equation}

For a differentiable function f:Vβ†’Ff:\Vcal\to\Field, we have from the Taylor’s expansion that βˆ‡f(x)\nabla f(\x) is a linear functional for all x∈V\x\in\Vcal, and therefore lies in the dual space Vβˆ—\Vcal^{*}. If fβˆ—:Vβˆ—β†’Ff^{*}:\Vcal^{*}\to\Field is the convex conjugate of ff, we have that βˆ‡fβˆ—(g)\nabla f^{*}(g) is an element in Vβˆ—βˆ—\Vcal^{**} for all g∈Vβˆ—g\in\Vcal^{*}. Since there exists a natural isomorphism Ξ¨:Vβ†’Vβˆ—βˆ—\Psi : \Vcal\to\Vcal^{**} between V\Vcal and Vβˆ—βˆ—\Vcal^{**} which is the evaluation map, this in particular gives us Ξ¨(x)=x^=βˆ‡fβˆ—(βˆ‡f(x))∈Vβˆ—βˆ—\Psi(\x) = \hat{\x} = \nabla f^{*}\round{\nabla f(\x)}\in\Vcal^{**}.