• The rowspace and nullspace are orthogonal (the angle between them is 90 degrees). Same for the columnspace and the left nullspace.

  • Orthogonal - in N-dimensional space, the angle between vectors is 90 degrees.

  • Test for orthogonality - two vectors are orthogonal if the dot product (\(x^Ty\)) is zero.

  • Shows the connection between the Pythagorean theorem and orthogonality.

    • Pythagorean theorem: \(\lvert x\rvert^2 + \lvert y\rvert^2 = \lvert x+y\rvert^2\)
    • Squared length of vector \(x\): \(x^Tx\)
    • When vectors are orthogonal (sub into Pythagorean):
\[x^Tx + y^Ty = (x+y)^T(x+y)\] \[x^Tx + y^Ty = x^Tx + y^Ty + x^Ty + y^Tx\] \[0 = 2x^Ty\] \[0 = x^Ty\]

  • Zero vector is orthogonal to all vectors.

  • Subspace \(S\) is orthogonal to subspace \(T\) when every vector in \(S\) is orthogonal to every vector in \(T\).

  • Rowspace is orthogonal to the nullspace. Why?

    • \(Ax = 0\) defines the nullspace
    • Alternatively, you can think of it as:
\[\begin{bmatrix} \begin{array}{c} \text{row 1 of A} \\ \text{row 2 of A} \\ ... \\ \text{row m of A} \\ \end{array} \end{bmatrix} * \begin{bmatrix} \begin{array}{r} x_1 \\ ... \\ x_n \\ \end{array} \end{bmatrix} = \begin{bmatrix} 0 \\ 0 \\ ... \\ 0 \end{bmatrix}\]
  • Each row of \(A\) is orthogonal to \(x\) because that row multiplied by \(x\) equals 0.
    • You also have to show that \(x\) is orthogonal to every linear combination of the rows of \(A\).
    • If \(c_1\text{row}_1^Tx = 0\) and \(c_2\text{row}_2^Tx = 0\) then use distributive property to show that:
\[(c_1\text{row}_1 + c_2\text{row}_2)^Tx = 0_{ }\]
  • The rowspace and nullspace carve \(R^n\) into two orthogonal subspaces. The columnspace and left nullspace do the same for \(R^m\). They are orthogonal complements (the complements contain all the vectors in the space they carve up).
    • The nullspace contains all vectors perpendicular to the row space.
  • Up next: solve \(Ax=b\) when there is no solutions.
  • Consider \(A^TA\) (where \(A\) is \(m \times n\)):
    • It’s square \(n \times n\)
    • It’s symmetric: \((A^TA)^T = A^TA^{TT} = A^TA\)

  • The “good” equation used for solving \(Ax=b\) when there is no solution is achieved by multiplying both sides by \(A^T\) to get \(A^TAx=A^Tb\).

  • The nullspace of \(A^TA\) equals the nullspace of \(A\).
  • The rank of \(A^TA\) equals the rank of \(A\).
  • \(A^TA\) is invertible exactly if \(A\) has independent columns.