Linear Algebra, Inner product spaces

Recall that in your study of vectors, we looked at an operation known as the dot product, and that if we have two vectors in {ℝ}^{n}, we simply multiply the components together and sum them up. With the dot product, it becomes possible to introduce important new ideas like length and angle. The length of a vector 𝐚, is just ‖ 𝐚 ‖ = \sqrt{𝐚 ⋅ 𝐚}. The angle between two vectors, 𝐚 and 𝐛, is related to the dot product by

cos ⁡ θ = \frac{𝐚 ⋅ 𝐛}{‖ 𝐚 ‖ ⋅ ‖ 𝐛 ‖}

It turns out that only a few properties of the dot product are necessary to define similar ideas in vector spaces other than {ℝ}^{n}, such as the spaces of m × n matrices, or polynomials. The more general operation that will take the place of the dot product in these other spaces is called the "inner product".

The inner product

Say we have two vectors:

𝐚 = ( \begin{matrix}2 \\ 1 \\ 4\end{matrix} ) , 𝐛 = ( \begin{matrix}6 \\ 3 \\ 0\end{matrix} )

If we want to take their dot product, we would work as follows

𝐚 ⋅ 𝐛 = {a}_{1} {b}_{1} + {a}_{2} {b}_{2} + {a}_{3} {b}_{3} = ( 2 ) ( 6 ) + ( 1 ) ( 3 ) + ( 4 ) ( 0 ) = 15

Because in this case multiplication is commutative, we then have 𝐚 ⋅ 𝐛 = 𝐛 ⋅ 𝐚.

But then, we observe

𝐯 ⋅ ( α 𝐚 + β 𝐛 ) = α 𝐯 ⋅ 𝐚 + β 𝐯 ⋅ 𝐛

much like the regular algebraic equality v ( a A + b B ) = a v A + b v B.
For regular dot products this is true since, for {ℝ}^{3}, for example, one can expand both sides out to obtain

\begin{matrix}( α {v}_{1} {a}_{1} + β {v}_{1} {b}_{1} ) + ( α {v}_{2} {a}_{2} + β {v}_{2} {b}_{2} ) + ( α {v}_{3} {a}_{3} + β {v}_{3} {b}_{3} ) = \\ ( α {v}_{1} {a}_{1} + α {v}_{2} {a}_{2} + α {v}_{3} {a}_{3} ) + ( β {v}_{1} {b}_{1} + β {v}_{2} {b}_{2} + β {v}_{3} {b}_{3} )\end{matrix}

Finally, we can notice that 𝐯 ⋅ 𝐯 is always positive or greater than zero - checking this for {ℝ}^{3} gives this as

𝐯 ⋅ 𝐯 = {v}_{1}^{2} + {v}_{2}^{2} + {v}_{3}^{2}

which can never be less than zero since a real number squared is positive. Note 𝐯 ⋅ 𝐯 = 0 if and only if 𝐯 = 0.

In generalizing this sort of behaviour, we want to keep these three behaviours. We can then move on to a definition of a generalization of the dot product, which we call the inner product. An inner product of two vectors in some vector space V, written ⟨ 𝐱 , 𝐲 ⟩ is a function V × V → ℝ, which obeys the properties

• ⟨ 𝐱 , 𝐲 ⟩ = ⟨ 𝐲 , 𝐱 ⟩
• ⟨ 𝐯 , α 𝐚 + β 𝐛 ⟩ = α ⟨ 𝐯 , 𝐚 ⟩ + β ⟨ 𝐯 , 𝐛 ⟩
• ⟨ 𝐚 , 𝐚 ⟩ ≥ 0 with equality if and only if 𝐚 = 0.

The vector space V and some inner product together are known as an inner product space.

The dot product in {ℂ}^{n}

Given two vectors 𝐚 = {a}_{1} {\overset{→}{e}}_{1} + {a}_{2} {\overset{→}{e}}_{2} + … + {a}_{n} {\overset{→}{e}}_{n} ∈ {ℂ}^{n} and 𝐛 = {b}_{1} {\overset{→}{e}}_{1} + {b}_{2} {\overset{→}{e}}_{2} + … + {b}_{n} {\overset{→}{e}}_{n} ∈ {ℂ}^{n}, the dot product generalized to complex numbers is:

𝐚 ⋅ 𝐛 = {∑}_{i = 1}^{n} {a}_{i}^{∗} {b}_{i} = {a}_{1}^{∗} {b}_{1} + {a}_{2}^{∗} {b}_{2} + … + {a}_{n}^{∗} {b}_{n}

where {z}^{∗} for an arbitrary complex number z = c + d i is the complex conjugate: {z}^{∗} = c − d i.

The dot product is "conjugate commutative": 𝐚 ⋅ 𝐛 = ( 𝐛 ⋅ 𝐚 {)}^{∗}. One immediate consequence of the definition of the dot product is that the dot product of a vector with itself is always a non-negative real number: 𝐚 ⋅ 𝐚 ≥ 0.

𝐚 ⋅ 𝐚 = 0 if and only if 𝐚 = \overset{→}{0}

The Cauchy-Schwarz Inequality for {ℂ}^{n}

Cauchy-Schwarz Inequality

Given two vectors 𝐚 , 𝐛 ∈ {ℂ}^{n}, it is the case that | 𝐚 ⋅ 𝐛 | ≤ | 𝐚 | | 𝐛 |

In {ℝ}^{n}, the Cauchy-Schwarz inequality can be proven from the triangle inequality. Here, the Cauchy-Schwarz inequality will be proven algebraically.

To make the proof more intuitive, the algebraic proof for 𝐚 , 𝐛 ∈ {ℝ}^{n} will be given first.

Proof for 𝐚 , 𝐛 ∈ {ℝ}^{n}

| 𝐚 ⋅ 𝐛 | ≤ | 𝐚 | | 𝐛 | follows from
| 𝐚 ⋅ 𝐛 {|}^{2} ≤ | 𝐚 {|}^{2} | 𝐛 {|}^{2} which is equivalent to

{( {∑}_{i = 1}^{n} {a}_{i} {b}_{i} )}^{2} ≤ ( {∑}_{i = 1}^{n} {a}_{i}^{2} ) ( {∑}_{j = 1}^{n} {b}_{j}^{2} )

expanding both sides gives:

{∑}_{i = 1}^{n} {∑}_{j = 1}^{n} {a}_{i} {b}_{i} {a}_{j} {b}_{j} ≤ {∑}_{i = 1}^{n} {∑}_{j = 1}^{n} {a}_{i}^{2} {b}_{j}^{2}

\ ⟺ \ {∑}_{i = 1}^{n} {∑}_{j = 1}^{n} ( {a}_{i} {b}_{j} ) ( {a}_{j} {b}_{i} ) ≤ {∑}_{i = 1}^{n} {∑}_{j = 1}^{n} ( {a}_{i} {b}_{j} {)}^{2}

"Folding" the double sums along the diagonal, and cancelling out the diagonal terms which are equivalent on both sides, gives:

\ ⟺ \ {∑}_{i = 2}^{n} {∑}_{j = 1}^{i − 1} 2 ( {a}_{i} {b}_{j} ) ( {a}_{j} {b}_{i} ) ≤ {∑}_{i = 2}^{n} {∑}_{j = 1}^{i − 1} ( ( {a}_{i} {b}_{j} {)}^{2} + ( {a}_{j} {b}_{i} {)}^{2} )

\ ⟺ \ 0 ≤ {∑}_{i = 2}^{n} {∑}_{j = 1}^{i − 1} ( {a}_{i} {b}_{j} − {a}_{j} {b}_{i} {)}^{2}

The above inequality is clearly true, therefore the Cauchy-Schwarz inequality holds for 𝐚 , 𝐛 ∈ {ℝ}^{n}.

Proof for 𝐚 , 𝐛 ∈ {ℂ}^{n}

Note that | 𝐚 ⋅ 𝐛 | ≤ | 𝐚 | | 𝐛 | follows from
| 𝐚 ⋅ 𝐛 {|}^{2} ≤ | 𝐚 {|}^{2} | 𝐛 {|}^{2} which is equivalent to
( 𝐚 ⋅ 𝐛 {)}^{∗} ( 𝐚 ⋅ 𝐛 ) ≤ ( 𝐚 ⋅ 𝐚 ) ( 𝐛 ⋅ 𝐛 ). Expanding both sides yields:

{( {∑}_{i = 1}^{n} {a}_{i}^{∗} {b}_{i} )}^{∗} ( {∑}_{i = 1}^{n} {a}_{i}^{∗} {b}_{i} ) ≤ ( {∑}_{i = 1}^{n} | {a}_{i} {|}^{2} ) ( {∑}_{i = 1}^{n} | {b}_{i} {|}^{2} )

\ ⟺ \ {∑}_{i = 1}^{n} {∑}_{j = 1}^{n} ( {a}_{i}^{∗} {b}_{i} {)}^{∗} ( {a}_{j}^{∗} {b}_{j} ) ≤ {∑}_{i = 1}^{n} {∑}_{j = 1}^{n} | {a}_{i} {|}^{2} | {b}_{j} {|}^{2}

\ ⟺ \ {∑}_{i = 1}^{n} {∑}_{j = 1}^{n} ( {a}_{j} {b}_{i} {)}^{∗} ( {a}_{i} {b}_{j} ) ≤ {∑}_{i = 1}^{n} {∑}_{j = 1}^{n} | {a}_{i} {b}_{j} {|}^{2}

"Folding" the double sums along the diagonal, and cancelling out the diagonal terms which are equivalent on both sides, gives:

\ ⟺ \ {∑}_{i = 2}^{n} {∑}_{j = 1}^{i − 1} ( ( {a}_{j} {b}_{i} {)}^{∗} ( {a}_{i} {b}_{j} ) + ( {a}_{i} {b}_{j} {)}^{∗} ( {a}_{j} {b}_{i} ) ) ≤ {∑}_{i = 2}^{n} {∑}_{j = 1}^{i − 1} ( | {a}_{i} {b}_{j} {|}^{2} + | {a}_{j} {b}_{i} {|}^{2} )

\ ⟺ \ {∑}_{i = 2}^{n} {∑}_{j = 1}^{i − 1} 2 ℜ ( ( {a}_{i} {b}_{j} {)}^{∗} ( {a}_{j} {b}_{i} ) ) ≤ {∑}_{i = 2}^{n} {∑}_{j = 1}^{i − 1} ( | {a}_{i} {b}_{j} {|}^{2} + | {a}_{j} {b}_{i} {|}^{2} )

Given complex numbers z and w, it can be proven that {}2 ℜ ( {z}^{∗} w ) ≤ | z {|}^{2} + | w {|}^{2} (this is similar to {}2 x y ≤ {x}^{2} + {y}^{2} for real numbers). The above inequality holds, and therefore the Cauchy-Schwarz inequality holds for complex numbers.

Where this page came from

This page was imported from Wikibooks. From “Linear Algebra” on Wikibooks, by its contributors, under CC BY-SA 4.0. Changed here: set as a page; navigation and edit links left out; each image under its own licence, credited in its caption.

Nobody has written it yet — it is the source material at a new address, which is why search engines are asked to skip it and why no one earns from it. It is up for grabs: take it on, and it is yours to rewrite and to earn from.