When you have two subspaces of a space V are complimentary when they don’t overlap but in combination they cover the whole space
V = X + Y (V is the *direct sum* of X and Y) while X ∩ Y = 0
Every vector v in V will have a corresponding x and y vector in each subspace of the two subspaces X Y. These vectors are unique
If there where two sets of vectors x_1 y_1 and x_2 y_2 then
v = x_1 + y_1
v = x_2 + y_2 \ y_2 - y_1 = x_1 + x_2
But since X ∩ Y = 0 this is impossible. So there must be only one x,y pair
The basis vectors of the two subspaces B_x B_y together span all of V. In matrix form we can just smash the columns together
B_y = [ B_x | B_y ]
Because each vector v in V has a unique x and y vectors in the X and Y subspaces we need a simple way to get an x and y given a v. This is what a projection matrix does.
Note that is v lies wholly in subspace y then the x component is [0]. All basis vectors of Y for instance will have no component in subspace X. Conversely, all vectors in x projectored onto the space X just give you back x
So we just write a projection matrix P as if it exists and solve
PXB_v = PX[ B_X | B_Y ] = [ PXB_X | PXB_Y ] = [ B_X | 0 ]
PX= [ B_X | 0 ]BV-1 \ PX= \ B_V \ [ \ I_r 0\ 0 0\ ]\ BV-1 \
Note:
- P^2 = P .. you project twice onto a subspace you get the same thing are projecting once
- PY = I - PX /.. v = x + y = Pv + y .. therefore y = v - Pv = (I - P)v
- PX spans all of X .. b/c obviously it needs to be able to project onto the whole subspace from V
In fact you can go further and say any matrix that is idempotent (ie. A^2 = A) is a projector with complementary subspaces R(A) and N(A.)
The subspaces X Y can be chosen arbitrarily as long as they satify the condition of spanning all of V and having not overlap. Interesting pairs to consider are orthogonal complementary pairs. You choose one basis M and then the other is the space orthogonal to it - M⊥
In other words all vectors in M⊥ are orthogonal to all vectors in M.
For instance.. If V = R^2 M can be a line and M⊥ can be another line. (Note that M doesn’t strictly need to be a subspace..)
Or if V = R^3 M⊥ can be a line and M can be a plane - or vice versa.
dim( M⊥ ) = n - dim( M )
[M⊥]⊥ = M
Note that if A is a nonsingular matrix then that means R(A) and R(A)⊥ will span all of R^n
A non-obvious decomposition emerges
We construct an x which is orthogonal to the range of A. So for all values of Ay (where y ∈ Rn) the x is orthogonal (here the book uses bracket notation which I translate to matrix products
x ∈ R(A)⊥
we define the x such that <Ay|x> = 0 -> [Ay]Tx = 0 \ yTATx = 0 (distribution of the transposition operator)\ <y|ATx> = 0 \ This means that .. ATx = 0 .. b/c the only way this product is zero is if *Ax* is zero \ so.. \ x ∈ N(A^T)
This is a very strange an unintuitive conclusion
R(A)⊥ = N(A^T)
N(A)⊥ = R(A^T)
In combination with the previous orthogonal complementary subspaces this means that R(A) and R(A)⊥ span all of Rn and therefore
Rn = R(A) and R(A)⊥{} = R(A) and N(A^T)
Rm = N(A) and N(A)⊥{} = N(A) and R(A^T)\
This final set of equations is truely bizarre as the columns of matrix A along with the nullspace of the rows of A cover all of Rn
If we choose to express ranges and nullspaces with orthonormal bases then we can concatenate them to make an orthonormal basis for all of R^n
BR^{N} = U = [ BR(A) | BN(A^T) ]
BR^{N} = V = [ BR(A^T) | BN(A) ]
Now notice that if we write UTAV in block matrix form then the nullspace-bases blocks will zero out A sub-blocks. Only the upper left block remains.
\begin{equation}
UTAV
\=
\begin{bmatrix}
BR(A)
BN(A^T)
\end{bmatrix}
\begin{bmatrix}
A1,1 & A1,2 \
A2,1 & A2,2
\end{bmatrix}
\begin{bmatrix}
BR(A^T) & BN(A)
\end{bmatrix}
\=
\begin{bmatrix}
C & 0 \
0 & 0
\end{bmatrix}
= R
\end{equation}
We can then flip the equation to make a factorization. This is done trivially b/c orthonormal matrices are inverted by transposition - A-1 = A^T.
\begin{equation}
A = URVT
\=
\begin{bmatrix}
BR(A) & BN(A^T)
\end{bmatrix}
\begin{bmatrix}
C & 0
0 & 0
\end{bmatrix}
\begin{bmatrix}
BR(A^T) \
BN(A)
\end{bmatrix}
\end{equation}
The original orthonormal bases we chose for the range and nullspace were not unique, hence the factorization is not unique either.