SB

On Outer Products

Sun, 28 May 2023 14:31:00 +0000

Everyone that has taken linear algebra is quite comfortable with the idea of the standard inner product between two vectors $\mathbf{x}, \mathbf{y} \in \mathbb{R}^n$.

\[\mathbf{x}^\top\mathbf{y} = \begin{bmatrix} x_1 & x_2 & \cdots & x_n\end{bmatrix}\begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_n\end{bmatrix} = \sum_{i=0}^nx_i\cdot y_i \in \mathbb{R}\]

This is the dot product we use in high school physics and mathematics. This has a nice geometric interpretation as shown in the following figure, where we assume that vector $\mathbf{x}$ is of unit length, i.e. $\mathbf{x}^\top\mathbf{x} = 1$.

We could this of the number $\mathbf{x}^\top\mathbf{y}$ as a measure of the amount of information shared by the two vectors $\mathbf{x}$ and $\mathbf{y}$. When $\mathbf{x}^\top\mathbf{y}=0$ then the two vectors convey mutually exclusive information.

An innocent little switching of transpose operation in the inner product $\mathbf{x}^\top\mathbf{y}$ results in a completely different operation that produces an $n \times n$ matrix!

\[\mathbf{x}\mathbf{y}^\top = \begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_n\end{bmatrix}\begin{bmatrix} y_1 & y_2 & \cdots & y_n \end{bmatrix} = \begin{bmatrix} x_1y_1 & x_1y_2 & \cdots & x_1y_n \\ x_2y_1 & x_2y_2 & \cdots & x_2y_n \\ \vdots & \vdots & \ddots & \vdots \\ x_ny_1 & x_ny_2 & \cdots & x_ny_n \end{bmatrix} \in \mathbb{R}^{n \times n} \tag{1}\]

This matrix can be seen as an operator $\mathcal{O}$ that maps $\mathcal{O}: \mathbb{R}^n\mapsto \mathbb{R}^n.$ This is the outer product. When the vectors are from different spaces, $\mathbf{x}\in\mathbb{R}^n$ and $\mathbf{y}\in\mathbb{R}^m$, then $\mathbf{x}\mathbf{y}^\top\in\mathbb{R}^{n\times m}$, and $\mathcal{O}: \mathbb{R}^m\mapsto \mathbb{R}^n.$

Outer products product rank-1 matrices, and any rank-1 matrix can be represented as an outer product between two vectors.

What is the geometry associated with the outer product?

To understand the geometry of the outer product, we will assume that we are only dealing with unit length vectors, $\Vert \mathbf{u} \Vert = \Vert \mathbf{v} \Vert = 1$. This is because the outer product between two vectors $\mathbf{u}, \mathbf{v}$ of arbitrary lengths is a scalar multiple of the outer product between the unit vectors along $\mathbf{u}, \mathbf{v}$,

\[\mathbf{u}\mathbf{v}^\top = \Vert\mathbf{u}\Vert\cdot\Vert\mathbf{v}\Vert\cdot\left(\frac{\mathbf{u}}{\Vert \mathbf{u} \Vert}\right)\left(\frac{\mathbf{v}}{\Vert \mathbf{v} \Vert}\right)^\top\]

The outer product of a vector $\mathbf{u}$ $\left( \Vert \mathbf{u} \Vert = 1 \right)$ with itself is given by,

\[\mathbf{u}\mathbf{u}^\top=\begin{bmatrix} u_1^2 & u_1 u_2 & \cdots & u_1u_n \\ u_2u_1 & u_2^2 & \cdots & u_2u_n\\ \vdots & \vdots & \ddots & \vdots\\ u_nu_1 & u_n^2 & \cdots & u_n^2\end{bmatrix}\]

This is the orthogonal projection matrix onto the space spanned by $\mathbf{u}$, as shown in the following figure. The outer product appears in places involving orthogonal projections.

What if we wanted to do an orthogonal projection onto a subspace $\mathcal{S}$ spanned by a set of vectors $\left\{ \mathbf{x}_1, \mathbf{x}_2, \ldots \mathbf{x}_k \right\}$ that also form an orthonormal basis for $\mathcal{S}$?

We simply form a matrix $\mathbf{X} = \begin{bmatrix} \mathbf{x}_1 & \mathbf{x}_2 & \cdots & \mathbf{x}_k \end{bmatrix}$ and the orthogonal projection matrix onto $\mathcal{S}$ is given by,

\[\mathbf{P}_{\mathcal{S}}=\mathbf{X}\mathbf{X}^\top=\sum_{i=1}^k \mathbf{x}_i\mathbf{x}_i^\top \tag{2}\]

When the basis $\left\{ \mathbf{x}_1, \mathbf{x}_2, \ldots \mathbf{x}_k \right\}$ of $\mathcal{S}$ is not orthonormal, the orthogonal projection matrix onto $\mathcal{S}$ is given by,

\[\mathbf{P}_{\mathcal{S}} = \mathbf{X}\left(\mathbf{X}^\top\mathbf{X}\right)^{-1}\mathbf{X}^\top \tag{3}\]

This does appear to have the outer product form but there is the strange inverse matrix in the middle. It can be shown that the product of three matrices $\mathbf{A}\in\mathbb{R}^{n_1\times n_2}, \mathbf{B}\in\mathbb{R}^{n_2\times n_3}, \mathbf{C}\in\mathbb{R}^{n_3\times n_4}$ can be expressed as the following,

\[\mathbf{ABC} = \sum_{i=1}^{n_2}\sum_{j=1}^{n_3} b_{ij}\cdot \mathbf{a}_i\tilde{\mathbf{c}}_j^\top \tag{4}\]

where,

$\mathbf{a}_i$ are the columns of $\mathbf{A}$.
$\tilde{\mathbf{c}}_j^\top$ are the rows of $\mathbf{C}$, and
$b_{ij}$ is the element in the $i^{th}$ row and $j^{th}$ column of $\mathbf{B}$.

The product of these three matrices is the weighted linear combination of the outer product of the columns $\mathbf{A}$ and the rows of $\mathbf{C}$.

Let $\mathbf{B}=\left(\mathbf{X}^\top\mathbf{X}\right)^{-1}$, then we have

\[\mathbf{P}_{\mathcal{S}}=\mathbf{X}\left(\mathbf{X}^\top\mathbf{X}\right)^{-1}\mathbf{X}^\top=\sum_{i=1}^{k}\sum_{j=1}^{k} b_{ij} \mathbf{x}_i\mathbf{x}_j^\top \tag{5}\]

We again see that outer products come into the picture when dealing with orthogonal projection. Here, all possible $k^2$ outer products are considered when the basis is not orthonormal. Equation (2) is a special case of Equation (5); when the basis is orthonormal the matrix $\mathbf{X}^\top\mathbf{X}=\mathbf{I}_k$, which leaves only the outer products of the form $\mathbf{x}_i\mathbf{x}_i^\top$.

We can think of the outer product $\mathbf{u}\mathbf{v}^\top$ as a generalization of the orthogonal projection that we get with $\mathbf{v}\mathbf{v}^\top$. I found a beautiful answer on StackOverflow about this here.

Consider three vectors, $\mathbf{u}, \mathbf{v}, \mathbf{x} \in \mathbb{R}^n$, where $\Vert \mathbf{u} \Vert = \Vert \mathbf{v} \Vert = 1$, then

\[\left(\mathbf{u}\mathbf{v}^\top\right)\mathbf{x} = \left(\mathbf{v}^\top \mathbf{x}\right)\mathbf{u}\]

The outer product $\mathbf{u}\mathbf{v}^\top$ is a matrix that help compute the component of $\mathbf{x}$ along the $\mathbf{v}$ direction taken along the $\mathbf{u}$ direction. When $\mathbf{u} = \mathbf{v}$, then this corresponds to the orthogonal projection operation, i.e. the component of $\mathbf{x}$ along $\mathbf{u}$, taken along $\mathbf{u}$. This is demonstrated in the following figure.

The process of computing the components along one set of vectors and taking those components along another set of vectors is done by linear transformation represented by a matrix $\mathbf{A}$.

Consider any matrix $\mathbf{A} \in \mathbb{R}^{n \times m}$ of rank $k$, which can be thought of as a linear transformation from $\mathbb{R}^m$ to $\mathbb{R}^n$. We can express this matrix using the SVD as the following,

\[\mathbf{A}=\mathbf{U}\mathbf{\Sigma}\mathbf{V}^\top, \quad \mathbf{U}\in\mathbb{R}^{n \times k}, \,\, \mathbf{V} \in \mathbb{R}^{m \times k}, \mathbf{\Sigma} \in \mathbb{R}^{k \times k}\] \[\mathbf{A}=\sum_{i=1}^k\sigma_i\mathbf{u}_i\mathbf{v }_i^\top\]

The columns of $\mathbf{V}$ form an orthonormal basis for the row space of $\mathbf{A}$ and the columns of $\mathbf{U}$ form an orthonormal basis of the column space of $\mathbf{A}$. We could view the linear transformation performed by $\mathbf{A}$ on a vector $\mathbf{x}\in \mathbb{R}^m$ as the following,

Find the components of the vector $\mathbf{x}$ onto the right singular vectors $\left\{\mathbf{v}_i\right\}_{i=1}^m$.
Scale the individual components by the singular values $\left\{\sigma_i\right\}_{i=1}^k$ .
Compute the components along the left singular vectors $\left\{\mathbf{u}_i\right\}_{i=1}^k$ using the scaling factor $\left\{\sigma_i\mathbf{v}_i^\top\mathbf{x}\right\}_{i=1}^k$.
Add these components to compute $\mathbf{Ax}$.

In fact, we can think of the linear transformation performed by an matrix $\mathbf{A}$ in a similar way when we can decompose the matrix as a product of two matrics, such as the LU decomposition,

\[\mathbf{A}=\mathbf{L}\mathbf{U}=\hat{\mathbf{L}}\mathbf{D}\hat{\mathbf{U}} = \sum_i d_i \mathbf{l}_i \tilde{\mathbf{u}}_i^\top\]

$\mathbf{l}_i$ are the columns of $\hat{\mathbf{L}}$ that form a basis for the column space of $\mathbf{A}$, with unit length.
$\tilde{\mathbf{u}}_i^\top$ are the rows of $\hat{\mathbf{U}}$ that form a basis for the row space of $\mathbf{A}$, with unit length.
$\mathbf{D}$ is a diagonal matrix with the scaling factors.

The outer product is an fundamental operation in linear transformation. It allows us to represent and understand linear transformations as a superposition of simple transformations represented by rank-1 matrices.

Understanding matrix inverses - II: Tall matrices

Sun, 12 Sep 2021 14:31:00 +0000

$\newcommand{\mf}{\mathbf} \newcommand{\mb}{\mathbb} \newcommand{\mc}{\mathcal}$ For full (column or row) rank non-square matrices $\mf{A} \in \mb{R}^{n \times m}$, we can still have inverses but there are some peculiarities:

We can only have left or right inverses, as is described below, and
The left and right inverses are not unique.

We will talk about the inverses of tall matrices in this post, i.e. matrices where $n > m$. Tall full rank matrices only have left inverses:

\[n > m, \,\, rank\left(\mf{A}\right) = m \implies \exists \mf{B} \in \mb{R}^{m \times n} \,\, s.t. \,\, \mf{B} \mf{A} = \mf{I}_{m}\]

This means that, $\mf{B}\mf{A}\mf{x} = \mf{B}\mf{y} \implies \mf{x} = \mf{B}\mf{y}$.

What $\mf{x}$ represents depends on whether or not $\mf{y}$ is in the column space of $\mf{A}$.

$\mf{y} \in \mc{C}\left( \mf{A} \right) \implies $ $\mf{x} = \mf{B}\mf{y}$ is the representation of $\mf{y}$ in the column basis of $\mf{A}$.
$\mf{y} \notin \mc{C}\left( \mf{A} \right) \implies $ $\mf{x} = \mf{Bb}$ is the representation of some vector $\hat{\mf{y}}$ $=$ $\mf{A}\mf{x} = \mf{ABb}$, which in the column basis of $\mf{A}$.

When $\mf{x}$ is substituted back into the original equation, we get

\[\begin{cases} \mf{y} \in \mc{C}\left(\mf{A}\right) &\implies \hat{\mf{y}} = \mf{A}\left(\mf{B}\mf{y}\right) = \mf{y} \\ \mf{y} \notin \mc{C}\left(\mf{A}\right) &\implies \hat{\mf{y}} = \mf{A}\left(\mf{B}\mf{y}\right) \neq \mf{y} \end{cases}\]

For a given left inverse $\mf{B}$, adding a matrix $\mf{C}$ whose rows are orthogonal to the columns of $\mf{A}$ will result in another left inverse of $\mf{A}$.

\[\left( \mf{B} + \mf{C} \right) \mf{A} = \mf{I}_m, \, s.t. \, \mf{C}\mf{A} = \mf{0}\]

The rows of $\mf{C}$ will be vectors from $\mc{N}\left(\mf{A}^\top\right)$ - the left nullspace of $\mf{A}$. Thus, it is clear that there are infinitely many left inverses for $\mf{A}$.

What do all these left inverse matrices do?

A left inverse allows us to find the representation of $\hat{\mf{y}}$ – a component of $\mf{y}$ in $\mc{C}\left(\mf{A}\right)$ – in the column basis of $\mf{A}$.

These components are oblique projections onto $\mc{C}\left(\mf{A}\right)$ along a complementary subspace to $\mc{C}\left(\mf{A}\right)$ in $\mb{R}^n$. The matrix $\mf{A}\left( \mf{B} + \mf{C}\right)$ is this oblique projection matrix.

\[\hat{\mf{y}} = \mf{A}\left(\mf{B} + \mf{C}\right) \mf{y}\]

In the absence of a mathetical proof of this, let’s convince ourselves that this is true through a simple example. Let $\mf{A} = \begin{bmatrix} 1 \\ 1\end{bmatrix}$. The following are the various possible left inverses of $\mf{A}$,

\[\mf{B} + \mf{C} = \begin{bmatrix} \mf{\tilde{b}}_1^\top \end{bmatrix} + \begin{bmatrix} \mf{\tilde{c}}_1^\top \end{bmatrix} = \frac{1}{2}\begin{bmatrix} 1 & 1 \end{bmatrix} + \begin{bmatrix} \alpha & -\alpha \end{bmatrix}\]

The following figures depicts this particular example,

In this figure, we have assumed $\alpha = -1 \implies \mf{\tilde{c}}_1 = \begin{bmatrix} -1 \\ 1\end{bmatrix}$, and $\mf{y} = \begin{bmatrix} 1 \\ 2\end{bmatrix}$. Thus,

\[\begin{split} \hat{\mf{y}} &= \mf{A} \left( \mf{B} + \mf{C}\right) \mf{y} \\ &= \begin{bmatrix} 1 \\ 1\end{bmatrix} \begin{bmatrix} -\frac{1}{2} & \frac{3}{2}\end{bmatrix} \begin{bmatrix} 1 \\ 2\end{bmatrix} \\ &= \begin{bmatrix} \frac{5}{2} \\ \frac{5}{2} \end{bmatrix} \end{split}\]

$\hat{\mf{y}}$ is the oblique projection of $\mf{y}$ onto $\mc{C}\left(\mf{A}\right)$ along the subspace parallel to the dashed black line joining $\mf{y}$ and $\hat{\mf{y}}$ in the above figure. This is the subspace spanned by the vector $\mf{y} - \hat{\mf{y}} = \begin{bmatrix} \frac{3}{2} \\ \frac{1}{2} \end{bmatrix}$. It is important to that $\mf{\tilde{b}}_1 + \mf{\tilde{c}}_1$ is orthogonal to $\mf{y} - \hat{\mf{y}}$.

\[\left( \mf{\tilde{b}}_1 + \mf{\tilde{c}}_1\right)^\top\left( \mf{y} - \hat{\mf{y}} \right) = 0\]

The following examples demonstrates the same for a different value of $\alpha = 2.5$.

It is left as an exercise to work out the different components and verfiy that $\left( \mf{\tilde{b}}_1 + \mf{\tilde{c}}_1\right) \perp \left(\mf{y} - \hat{\mf{y}}\right)$.

For any choice of $\alpha$ we can get a oblique project matrix onto $\mc{C}\left(\mf{A}\right)$ along a subspace orthogonal to that of $\mc{C}\left(\mf{B}^\top + \mf{C}^\top\right)$, i.e. the row space of the left inverse.

A special left inverse.

Among the infintely many oblique projectors, the orthogonal projector is a speical one. The left inverse of $\mf{A}$ that will allow us to do orthogonal projection is the Moore-Penrose Inverse or the Pseudoinverse, expressed as $\mf{A}^\dagger$.

\[\mf{A}^\dagger = \left(\mf{A}^\top\mf{A}\right)^{-1}\mf{A}^\top\]

The matrix $\mf{B} = \begin{bmatrix} \mf{\tilde{b}}_1^\top \end{bmatrix} = \frac{1}{2}\begin{bmatrix} 1 & 1 \end{bmatrix}$ shown in the above figures is the pseudoinverse of $\mf{A}$.

The mapping performed by the psuedoinverse $\mathbf{A}^\dagger$ is shown in the above figure.

How do know that this is in fact a orthogonal projector?

$\mc{C}\left(\left(\mf{A}^\dagger\right)^\top\right) = \mc{C}\left(\mf{A}\right)$, which means that the matrix $\mf{A}\mf{A}^\dagger$ is the projection matrix onto $\mc{C}\left( \mf{A} \right)$ along the subspace orthogonal to $\mc{C}\left(\mf{A}\right)$, which is $\mc{N}\left(\mf{A}^\top\right)$.

I hope this short presentaion helps shed some light on the geometry operation behind left inverses. The next part of this series will focus on the right inverse.

Understanding matrix inverses - Part I: Square matrices

Thu, 09 Sep 2021 14:31:00 +0000

Matrices are a representation of linear maps between finite-dimensional vector spaces. Consider a linear map $\mathcal{L}$ from $\mathbb{R}^m$ to $\mathbb{R}^n$.

\[\mathcal{L}: \mathbb{R}^m \mapsto \mathbb{R}^n\]

This linear map can be represented using a real matrix $\mathbf{A} \in \mathbb{R}^{n \times m}$, such that

\[\mathbf{y} = \mathcal{L}\left( \mathbf{x} \right) = \mathbf{A}\mathbf{x}, \quad \mathbf{x} \in \mathbb{R}^m, \, \mathbf{y} \in \mathbb{R}^n\]

When the domain and range spaces are the same, then $\mathbf{A}$ is a square matrix.

The matrix $\mathbf{A}$ has a unique inverse $\mathbf{A}^{-1}$, if and only if the $\mathbf{A}$ is full-rank, i.e. the columns of the matrix form a basis for $\mathbb{R}^n$. In this case, $\mathbf{A}^{-1}\mathbf{A} = \mathbf{A}\mathbf{A}^{-1} = \mathbf{I}_n$, where $\mathbf{I}_n$ is the $n \times n$ identity matrix.

Matrix inverses allow us to change basis

Consider the equation, $\mathbf{A}\mathbf{x} = \mathbf{y}$. This equation says that the vector $\mathbf{y}$ is a linear combination of the columns of $\mathbf{A}$, with the weight for each column given by the elements of $\mathbf{x}$.

\[\mathbf{y} = \mathbf{A}\mathbf{x} = \begin{bmatrix} \mathbf{a}_1 & \mathbf{a}_2 & \cdots & \mathbf{a}_n\end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_n\end{bmatrix} = \sum_{i=1}^n x_i\mathbf{a}_i\]

Here, $\mathbf{x}$ is the representation of the vector $\mathbf{y}$ in the basis of $\mathbb{R}^n$ formed by the columns of $\mathbf{A}$ (we will refer to this basis as the column basis). Given $\mathbf{A}$ and $\mathbf{y}$, we can solve for $\mathbf{x}$ using the inverse of $\mathbf{A}$,

\[\mathbf{x} = \mathbf{A}^{-1}\mathbf{y}\]

Thus, we see that $\mathbf{A}^{-1}$ is the matrix that allows us to change the representation of a vector to the column basis.

Structure of a matrix inverse Let us express $\mathbf{A}^{-1}$ as a column of rows.

\[\mathbf{A} = \begin{bmatrix} \mathbf{a}_1 & \mathbf{a}_2 & \ldots & \mathbf{a}_n \end{bmatrix} \longrightarrow \mathbf{A}^{-1} = \begin{bmatrix} \mathbf{\tilde{b}}_1^\top \\ \mathbf{\tilde{b}}_2^\top \\ \vdots \\ \mathbf{\tilde{b}}_n^\top\end{bmatrix}\]

Let $\mathbf{A} = \begin{bmatrix} 1 & 1\\ 0 & 1 \end{bmatrix} = \begin{bmatrix} \mathbf{a}_1 & \mathbf{a}_2 \end{bmatrix}$, then $\mathbf{A}^{-1} = \begin{bmatrix} 1 & -1 \\ 0 & 1\end{bmatrix} = \begin{bmatrix} \mathbf{\tilde{b}_1^\top} \\ \mathbf{\tilde{b}_2^\top}\end{bmatrix}$. Here,

\[\mathbf{\tilde{b}}_1^\top \mathbf{a}_1 = \mathbf{\tilde{b}}_2^\top \mathbf{a}_2 = 1 \quad \text{and} \quad \mathbf{\tilde{b}}_1^\top \mathbf{a}_2 = \mathbf{\tilde{b}}_2^\top \mathbf{a}_1 = 0\]

The components of $\mathbf{y}$ in column basis is given by, $\mathbf{x} = \begin{bmatrix} x_1 \\ x_2 \end{bmatrix} = \mathbf{A}^{-1}\mathbf{y}$.

\[x_1 = \mathbf{\tilde{b}}_1^\top \mathbf{y} \quad \quad x_2 = \mathbf{\tilde{b}}_2^\top \mathbf{y}\]

Here, $\mathbf{y}_1 = x_1 \mathbf{y}$ is the oblique projection of $\mathbf{y}$ onto the subspace spanned by $\mathbf{a}_1$, along the subspace spanned by $\mathbf{a}_2$. The scaling factor $x_1$ for this oblique projection is obtained by taking the inner product of $\mathbf{y}$ with $\mathbf{\tilde{b}}_1$.

Similarly, $\mathbf{y}_2 = x_2 \mathbf{a}_2$ is the oblique projection of $\mathbf{y}$ onto the subspace spanned by $\mathbf{a}_2$ along the subspace spanned by $\mathbf{a}_1$. $x_2$ is obtained by taking the inner product of $\mathbf{y}$ with $\mathbf{\tilde{b}}_2$.

This can be generalized to a $n \times n$ square matrix $\mathbf{A}$.

The oblique projection of $\mathbf{y}$ onto $span\left( \{ \mathbf{a}_i \}\right)$ along $span\left( \{ \mathbf{a}_j \}_{j=1, j \neq i}^{n}\right)$ is obtained by computing the scaling factor $x_i$ as the following,

\[x_i = \mathbf{\tilde{b}}_i^\top \mathbf{y}\] \[\mathbf{y}_i = x_i \mathbf{a}_i = \left( \mathbf{\tilde{b}}_i^\top \mathbf{y} \right) \mathbf{a}_i = \left( \mathbf{a}_i \mathbf{\tilde{b}}_i^\top \right) \mathbf{y} = \mathbf{P}_i \mathbf{y}\]

Note: $span\left( \{ \mathbf{a}_j \}_{j=1, j \neq i}^{n}\right)$ is the subspace spanned by the columns of $\mathbf{A}$ except $\mathbf{a}_i$.

$\mathbf{P}_i = \mathbf{a}_i \mathbf{\tilde{b}}_i^\top$ is the projection matrix that does oblique projections onto $span\left( \{ \mathbf{a}_i \}\right)$ along $span\left( \{ \mathbf{a}_j \}_{j=1, j \neq i}^{n}\right)$.

\[\mathbf{x} = \begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_n \end{bmatrix} = \mathbf{A}^{-1} \mathbf{y} = \begin{bmatrix} \mathbf{\tilde{b}}_1^\top \\ \mathbf{\tilde{b}}_2^\top \\ \vdots \\ \mathbf{\tilde{b}}_n^\top \end{bmatrix} \mathbf{y} = \begin{bmatrix} \mathbf{\tilde{b}}_1^\top \mathbf{y} \\ \mathbf{\tilde{b}}_2^\top \mathbf{y} \\ \vdots \\ \mathbf{\tilde{b}}_n^\top \mathbf{y} \end{bmatrix}\]

The matrix $\mathbf{P}_{1:r} = \sum_{i=1}^{r} \mathbf{P}_i = \sum_{i=1}^{r} \mathbf{a}_i \mathbf{\tilde{b}}_i^\top$ is the oblique projection matrix onto $span\left( \{ \mathbf{a}_i \}_{i=1}^{r}\right)$ along $span\left( \{ \mathbf{a}_i \}_{i=r+1}^{n}\right)$.

Taking the inner product with the rows $\left\{ \mathbf{\tilde{b}}_i^\top \right\}_{i=1}^r$ will give us the scaling factors required to obtain the oblique projection along the subspace that is orthogonal to $span\left( \left\{ \mathbf{\tilde{b}}_i \right\}_{i=1}^r \right)$, which is $span\left( \left\{ \mathbf{a}_i \right\}_{i=r+1}^n\right)$. This fact will come in handy in understanding the geometric operations performed by left inverses.

In the next two parts of the blog on matrix inverses, we will go into the details of left and right inverses of full rank rectangular matrices.

Representation of Numbers in Computers - I: Whole numbers and Integers

Sat, 28 Mar 2020 18:30:00 +0000

I have long wanted to write something about the different binary number representations used in a computer, along their corresponding arithmetic operations. I would have loved to have stumbled upon something like this during my undergraduate years. So, I hope that this will be useful for others who might be looking for a short tutorial-like material on numerical data types used in a computer, in particular integers, fixed-point, and floating-point numbers. In this two part blog-post, I first write about the representation of whole numbers and integers, along with the details of how to add and multiply them.

Whole Numbers

The binary representation of whole numbers is something all of us are familiar with. In the decimal system, the number two hundred and thirty five is represented as the following,

\[235 = 2 \times 10^2 + 3 \times 10^1 + 5 \times 10^0\]

In the decimal system, the base is 10 and 10 unique (decimal) digits $0-9$ are used for representing all numbers.

In the binary system, this number has the following representation, where the base is 2 and the digits 0 and 1 are used.

\[\begin{split} 235 =& 1 \times 2^7 + 1 \times 2^6 + 1 \times 2^5 + 0 \times 2^4 + \\ & 1 \times 2^3 + 0 \times 2^2 + 1 \times 2^1 + 1 \times 2^0 \\ =& \left[ 11101011 \right]_2 \end{split}\]

(Note: $\left[\cdot\right]_2$ indicates that the number in the square brackets is a binary number. We will use this representation only when there is ambiquity regarding the base of a number.)

A $n$ bit binary number $B$ is written as $B = \left[ b_{n-1}b_{n-2} \ldots b_1b_0 \right]_2$, where $b_i \in [0, 1], \, 0 \leq i \leq n-1$. Let $d(\cdot)$ be the function that coverts a binary number $B$ to its decimal equivalent, $d(B) = \sum_{i=0}^{n-1} b_i \times 2^i$.

Prove: The maximum whole number that can be represented by a $n$-bit number is $2^n - 1$.

Arithmetic operations on binary numbers are carried out exactly the way we perform decimal arithmetic. We will only consider the addition and multiplication operations here, as the set of whole numbers are not closed under the subtraction and division operations.

Binary Addition can be carried out with the knowledge of the following truth table. Addition of two $n$ bit binary numbers $A$ and $B$ can result in a binary number $C = A + B$ with at most $n+1$ bits.

A	B	C = A + B	C = A x B
0	0	00₂	00₂
0	1	01₂	01₂
1	0	01₂	00₂
1	1	10₂	01₂

Here is an example demonstrating the procedure for adding two binary numbers,

\[\left[ 0110 \right]_2 + \left[ 1101 \right]_2 = \begin{matrix} \begin{array}{r} & 0 & 1 & 1 & 0\\ + & 1 & 1 & 0 & 1\\ \hline 1 & 0 & 0 & 1 & 1 \end{array} \end{matrix} = \left[ 10011 \right]_2\] \[\left[ 6 \right]_{10} + \left[ 13 \right]_{10} = \left[ 19 \right]_{10}\]

Binary Multiplication can be carried out with the knowledge of the above truth table. Multipliation of two $n$ bit binary numbers $A$ and $B$ can result in a binary number $C = A \times B$ with at most $2n$ binary bits.

\[\left[ 0110 \right]_2 \times \left[ 1101 \right]_2 = \begin{matrix} \begin{array}{r} & & & 0 & 1 & 1 & 0\\ & & \times & 1 & 1 & 0 & 1\\ \hline & & & 0 & 1 & 1 & 0\\ & & 0 & 0 & 0 & 0 & 0\\ & 0 & 1 & 1 & 0 & 0 & 0\\ 0 & 1 & 1 & 0 & 0 & 0 & 0\\ \hline 1 & 0 & 0 & 1 & 1 & 1 & 0\\ \end{array} \end{matrix} = \left[ 1001110 \right]_2\] \[\left[ 6 \right]_{10} + \left[ 13 \right]_{10} = \left[ 78 \right]_{10}\]

Numbers represented using a finite number of bits behave like modulo-$n$ numbers

Numbers represented in computer use a finite number of bits. For example, a 64 bit microprocessor uses 64 bits to represent data, which means that with a single word (64 bits) we can only represent upto a $2^{64}$ different numbers or data types. If we were representing whole numbers then, the maximum number that can be represented is $2^{64} - 1$. Arthimetic operations performed with 64 bit numbers are also restrcited to be 64 bit numbers. This results in these numbers behaving differently under arthimetic operations, compared to numbers with no such restriction on the number of bit. For example, 3-bit numbers under the operation of addition and multiplication will behave like modulo-$2^{3}$ numbers; overflows are ignored and the lower three bits are considered the results of the arthimethic operations.

In general, whole numbers can be represented as discrete, equi-distant points on a number line (infinitely long) as shown in the following figure,

While, the whole numbers represented with a finite number of bits can be represented on a circle (number-wheel) as the following figure,

Integers

When representing negative numbers, on paper, one could simply use the ‘-‘ symbol, e.g. -51 in binary can be written as -110011. Since 0s and 1s are the only available symbols when representing things on a computer, we need a different approach to represent negative numbers: 2’s complement representation. Let’s assume that we working with 3 bit numbers; we can represent upto 8 different numbers. In the 2’s complement representation, the binary representation of positive numbers and zero remain unchanged. However, when we have a negative number $N$, we first take the binary representation of $\vert N \vert$, invert the bits, and add 1 to the inverted numbers.

\[-2 \xrightarrow[\text{binary}]{\text{2 in}} 010 \xrightarrow[\text{bits}]{\text{Invert}} 101 \xrightarrow[\text{1}]{\text{Add}} 110\]

In the 2’s complement representation, the most significant bit represents the sign bit, and it is 1 for negative numbers. As the following table indicates.

Binary No.	Whole number	Integer
000₂	0	0
001₂	1	1
010₂	2	2
011₂	3	3
100₂	4	-4
101₂	5	-3
110₂	6	-2
111₂	7	-1

We can convert a 2’s complement number to the corresponding decimal number using the following formula,

\[N = -b_{n-1}2^{n-1} + \sum_{i=0}^{n-2} b_i2^i\]

The followiing image shows another way to look at 2’s complement numbers.

Prove: In the 2's complement representation using $n$-bits, (a) the highest positive integer that can be represented is $2^{n-1} - 1$; and (b) the lowest negative integer that can be represented is $-2^{n-1}$.

Binary Addition. The truth table for addition remains the same, and we can add two numbers like we did with whole numbers. This handles the addition of both negative and positive numbers, but we need to be careful when interpreting the results because of the sign bit. Addition of numbers corresponds to clockwise movement along the number-wheel, and the 2’s complement representation is such that addition by positive numbers can be seen as clockwise movements, and that of negative numbers as anti-clockwise movement.

However, we need to keep in mind the following conditions, which indicate an overflow:

Addition of two positive numbers cannot result in a negative number, e.g. $3 + 2 = [011]_2 + [010]_2 = [101]_2$.
Addition of two negative numbers cannot result in a positive number, e.g. $-3 + -4 = [101]_2 + [100]_2 = [001]_2$. Note that the addition of negative numbers in 2’s complement number will result a four bit number. Here we simply ignore the fourth bit.

When adding a positive and negative 2’s complement number, we never run into trouble.

It should be noted that overflow here does not mean the presence of a carryover bit in the $(n+1)^{th}$ position, but rather when the boundary on the number-wheel between extreme positive and negative numbers is crossed by the addition or multiplication operation.

Binary Multiplication. We can multiply 2’s complement numbers like we do with binary whole numbers. The products come out with the appropriate signs provided there is no overflow. Consider wo binary signed numbers $B_1$ and $B_2$. An overflow occurs when: the product $\vert B_1 \times B_2$ is greater than $2^{n-1}$ when the $\text{sign}\left(B_1 \times B_2\right)$ is $-1$, or when the product $\vert B_1 \times B_2$ is greater than $2^{n-1} - 1$ when the $\text{sign}\left(B_1 \times B_2\right)$ is $+1$. There is no overflow when one of the numbers is $0$. You can verify this writing down the product table for 2 bit numbers.

The second part of this blog-post will deal with fixed and floating point numbers.

Understanding the Monty Hall Problem

Mon, 08 Jul 2019 03:31:00 +0000

s I had come across the Monty Hall problem a year back, and my answer that could come up with the standard one - “I would not switch my choice”. In case you do not know what that means, let me start by describing the Monty Hall problem.

The Monty Hall problem is a puzzle in probability. Assume that you are in a gameshow, where there are three doors (A, B, C) - one of the doors conceals a car, which you can win if you choose this door correctly; the other two doors have things that are not of any value. You are first required to choose a door. Following this the host checks the remaining doors, and opens of the on that does not contain the car. You are now given a chance to either stick with your original choice, or switch your choice to the remaining door. What would you be the better option? Stick with your original choice, or switch to the other door.

We will first look at the standard wrong solution, and then look at the correct solution.

Standard Solution: The probability of our first choice (e.g. A) being correct is $\frac{1}{3}$. Then, when the host opens a door that does not contain the car (e.g. B). We are now left with two doors (A and C), and the probability of the car being behind either one of these doors is $\frac{1}{2}$. Thus, it would make no difference whether we switch or stick to our original choice.

Lets look at this systematically and see if the above solution is correct. Because there are only three doors, we could easily enumerate all possiblities and see if it gives us an inight into this problem and the solution.

Car	Chosen	Revealed	Remaining	Switch?
A	A	B/C	C/B	No
A	B	C	A	Yes
A	C	B	A	Yes
B	A	C	B	Yes
B	B	A/C	C/A	No
B	C	B	A	Yes
C	A	B	C	Yes
C	B	A	C	No
C	C	B	A	Yes

Looking at the last column, we see that, irrespective of the door containing the car and the our initial choice, swtiching would help us win $\frac{2}{3}$rd of the times. Our standard answer does not line-up with systematic analysis.

Let $A$ be the event of our first choice of door containing the car. Then, $P(A) = \frac{1}{3}$. After we select a door, the host chooses a door to reveal; the door revealed depends on our original choice as the host knows which door contains the car. Let $B$ represent the event that the door that remains contains the car. The probability of $B$ depends on $A$. If $A$ is true then $P(B \, \lvert \, A) = 0$; the door with the car has been selected so $B$ cannot be true. However, if $A$ is false, then $P(B \, \lvert \, A^c) = 1$; this is because the host will only reveal the door without the car. Using the theorem of total probability, we have

\[P(B) = P(B \, \lvert \, A)P(A) + P(B \, \lvert \, A^c)P(A^c) = 0 \cdot \frac{1}{3} + 1 \cdot \frac{2}{3} = \frac{2}{3}\]

This number agrees with the results in the table above. So, where did we go wrong in the standard solution?

The problem with the standard solution is that we ignore the fact that the host has knowledge of the door containing the car. This is implied in the statement “… We are now left with two doors (A and C), and the probability of the car being behind either one of these doors is $\frac{1}{2}$ …”. But this is false. The door that the host leave unopened is actually more likely to have the car as he will only open the door without the car. To see this more cearly, lets see what happens when the host randomly chooses a door to open and that this door does not contain the car. Then, we have the following,

\[P(B \, \lvert \, A) = 0 \quad \text{and} \quad P(B \, \lvert \, A^c) = \frac{1}{2}\]

Thus,

\[P(B) = P(B \, \lvert \, A)P(A) + P(B \, \lvert \, A^c)P(A^c) = 0 \cdot \frac{1}{3} + \frac{1}{2} \cdot \frac{2}{3} = \frac{1}{3}\]

Here, we see that both $P(A)$ and $P(B)$ are equal and thus there would be no benefit in switching our choice.

After I spent some time thinking about this simply problem and its solution, I realize that the problem with my original approach to the problem was really just lazy thinking.

There are lots of such problems involving conditional probability that put you understanding on the nuances of probability theory to the test. I hope to write about these various problems in my later posts.

Using Sinc Pulses to Characterize LTI Systems

Tue, 25 Jun 2019 15:31:00 +0000

I was involved in a discussion, a few months back, about a linear time-invariant (LTI) filter that someone was trying to characterize. Their plan was to use a sinc pulse as an input, and use the Fourier spectrum of the filter’s output as a direct readout of the filter’s frequency response.

At first, it sounded like a strange idea but after some thought, it sounded reasonable. I had never thought about characterizing a filter using a sinc pulse before. I think my confusion was due to the fact that we often use impulse, step, and sinusoisal signals to characterize LTI systems, as these are easier to generate (or simulate) using analog hardware. I am not aware of an analog circuit that would generate an approximate sinc pulse.

Consider a continuous-time LIT system $H$, with impulse response $h(t)$ and frequency response $H(j\omega)$, the input-output relationship is given by,

\[y(t) = H\left\{x(t)\right\} = h(t) * x(t) = \int_{-\infty}^{\infty} x(\tau)h(t - \tau)d\tau \quad \text{and} \quad Y(j\omega) = H(j\omega)X(j\omega)\]

where, $x(t)$ is the input and $y(t)$ is the output of the system $H$.

For an input sinc pulse $x(t) = \frac{\omega_c}{\pi}\text{sinc} \left(\omega_c t\right) = \frac{\omega_c}{\pi}\frac{\sin \left( \omega_c t \right)}{\omega_c t}$, the Fourier spectrum of the output is given by,

\[Y(j\omega) = H(j\omega)X(j\omega) = \begin{cases} H(j\omega) & \left| \omega \right| \leq \omega_c \\ 0 & \left| \omega \right| > \omega_c \\ \end{cases}\]

Here, the sinc function acts as an approximate impulse function; increasing $\omega_c$ provides a better and better approximation of the impulse fucntion.

\[\int_{-\infty}^{\infty} \frac{\omega_c}{\pi}\text{sinc} \left(\omega_c t\right)dt = 1 \quad \text{and} \quad \lim_{\omega_c \to \infty} \frac{\omega_c}{\pi}\text{sinc} \left(\omega_c t\right) = \delta(t)\]

Simultaneously, the brickwall frequency response tends towards a contant function $\lim_{\omega_c \to \infty}X(j\omega) = 1$.

Thus, if $H$ is a bandlimited filter $\left( H\left( j\omega \right) = 0, \forall \vert \omega \vert > \omega_H\right)$, then $Y\left(j\omega\right) = H\left(j\omega\right)$ when $\omega_c \geq \omega_H$.

This means that the output of the system $H$ to two sinc inputs $x_1(t) = \frac{\omega_{c1}}{\pi}\text{sinc} \left(\omega_{c1} t\right)$ and $x_2(t) = \frac{\omega_{c2}}{\pi}\text{sinc} \left(\omega_{c2} t\right)$ will be the same, as long as $\omega_{c1} \geq \omega_H$ and $\omega_{c2} \geq \omega_H$. This implies that $x_1(t) - x_2(t)$ is in the nullspace or the kernel of the system $H$. The nullspace of this system is the eigenspacce spanned by the set of eigenvectors $\left\{e^{j\omega t}\right\}_{\forall \lvert \omega \rvert \geq \omega_H}$; it can be easily verified that $x_1(t) - x_2(t)$ is in $\text{span}(\left\{e^{j\omega t}\right\}_{\forall \lvert \omega \rvert \geq \omega_H})$.

For systems that are practically bandlimited limited, sufficiently narrow and tall sinc pulse is as good an impulse function for the purpose of characterizing the system.

Least Square Estimation of AR Models and Whitening - Part I

Sat, 15 Sep 2018 10:55:06 +0000

Consider an $p$-order stationary autoregressive model driven by white noise.

\[x_n = \sum_{k=1}^{p} a_kx_{n-k} + \epsilon_n\]

where, $\epsilon_n$ is Gaussian white noise with zero mean and variance $\sigma_{\epsilon}^2$.

Let us assume that we have $N+$ sample of this AR process and we are interested in estimating the parameter $a_k$. We can arrange the data into a set of $M=N+1-p$ linear equations,

\[\begin{bmatrix} x_{0} & x_{1} & \cdots & x_{p-1} \\ x_{1} & x_{2} & \cdots & x_{p} \\ x_{2} & x_{3} & \cdots & x_{p+1} \\ % x_{p+1} & x_{p} & \cdots & x_{2} \\ \vdots & \vdots & \ddots & \vdots \\ x_{N-p} & x_{N-p+1} & \cdots & x_{N-1} \end{bmatrix} \begin{bmatrix} a_p \\ a_{p-1} \\ \vdots \\ a_1 \end{bmatrix} = \begin{bmatrix} x_{p}\\ x_{p+1}\\ x_{p+2}\\ \vdots\\ x_{N} \end{bmatrix}\] \[\begin{bmatrix} \mathbf{x}_{N-p, M} & \mathbf{x}_{N-p+1, M} & \cdots & \mathbf{x}_{N-1, M} \end{bmatrix} \mathbf{a} = \mathbf{x}_{N, M}\] \[\mathbf{X}_{N, M}\mathbf{a} = \mathbf{x}_{N, M}\]

where, $\mathbf{x}_{k, M}$ is a column vector whose elements are the past $M$ of $x_n$ starting from the instant $k$; $x_{k-M+1}$, $x_{k}$ are the first and last elements of the vector, respectively. $\mathbf{X}_{N, M}$ consists of the columns $\mathbf{x}_{N-p, M}, \,\mathbf{x}_{N-p+1, M}, \, \ldots \, , \mathbf{x}_{N-p, M}$.

The least-squares estimate of $\mathbf{a}$ is given by,

\[\hat{\mathbf{a}} = \left(\mathbf{X}_{N, M}^T\mathbf{X}_{N, M}\right)^{-1}\mathbf{X}_{N, M}^T\mathbf{x}_{N, M}\]

Post-multiplying $\mathbf{x}_{N, M}$ by the pseudo-inverse will provide the least square estimate of $\mathbf{a}$.

A simple example: Let us start with the simplest possible example of a AR process where $p=1$.

\[x_n = a_1 x_{n-1} + \epsilon_n\]

Lets assume that we have $N$ samples of $x_n$, we can then estimate the parameter $a_1$ using the following,

\[\hat{a}_1 = \frac{\mathbf{x}_{N-1, M}^T\mathbf{x}_{N, M}}{\mathbf{x}_{N, M}^T\mathbf{x}_{N, M}}\]

Running estimate of a AR process of order 1

The following figure shows the result from an estimation procedure for a autoregressive process of order 1. The code used for generate this plot can be found here.

Running estimate of a AR process of order ‘p’

The following figure shows the result from an estimation procedure for a autoregressive process of order 3. The code used for generate this plot can be found here.

Whitening using estimated AR parametes $\left(\hat{a}\right)$

Once $\hat{a}$ is obtained, then the signal $y_n$ can be whitened by passing it through the following moving average filter.

\[w_n = y_n - \sum_{k=1}^{p} \hat{a}_ky_{n-k}\]

$w_n$ would be the out of this moving average filter, and $w_n$ will be a white noise. The input to this moving average filter the measured time series $y_n$.

Input-Output Relationship of a Linear Time-Varying System

Mon, 20 Aug 2018 13:00:00 +0000

Most of us that have studied signal processing or control theory would be familiar with the input-output relationship of linear time-invariant (LTI) systems. If $H$ is an LTI system with impulse response $h(t)$, then the output of the system for any arbitrary input is given by convolution integral,

\[y(t) = \int_{-\infty}^{\infty} h(\tau) x(t - \tau) d\tau \,, \quad t \in \mathbb{R}\]

The same idea can be applied to a discrete-time system with impulse response $h[n]$. The output in this case is given by the convolution sum,

\[y[n] = \sum_{k=-\infty}^{\infty}h[k]x[n - k] \,, \quad n \in \mathbb{Z}\]

The fact that input-output (IO) relationships for LIT systems can be determined through convolution, also leads to popular transform-based representation, analysis and design of LIT systems.

Linear-time varying (LTV) systems on the other hand cannot be understood through the convolution integral or the sum as shown above. The behavior of a LTV system depends explicity on time, and its IO behavior changes with time. If $H$ is a LTV system, then if $H\left\{ x(t) \right\} = y(t)$, then $H\left\{x(t - t_0)\right\} \neq y(t - t_0)$, i.e. time-shifted inputs do not produce time-shifted outputs.

A LTV system can be viewed as a system with impulse response that is time dependent, i.e. the output of the system to an impulse input $\delta(t - s)$ depends on both $t$ and $s$.

\[\phi(t, s) = H\left\{ \delta(t - s) \right\}, \quad t, s \in \mathbb{R}\]

where, $t$ is time, and $s$ is the time instant at which the impulse is applied. In the case of an LTI system, the response to $\delta(t - s)$ is $h(t - s)$; the output only depends on a single argument $t - s$.

In LTI systems, $h(t)$ acts as a weighting function that determines how the past, present and future input values are combined to produce the present output; this is the operation performed by the covolution operation. $\phi(t, s)$ also plays a similar role. The output at a given time $t$ for an LTV system, will weight the past, present and future values according to the function $\phi(t, s)$. In order to find out the output of an LTV system for an arbitrary input $x(t)$, lets start with the following,

\[x(t) = \int_{-\infty}^{\infty} x(\tau) \delta(t - \tau) d\tau\]

The output of the system $H$ to the input $\delta(t - \tau)$ is given by $h(\tau, t)$. Thus, $y(t)$ can be obtained as the following,

\[y(t) = H\left\{ \int_{-\infty}^{\infty} x(\tau) \delta(t - \tau) d\tau \right\} = \int_{-\infty}^{\infty} x(\tau) H\left\{ \delta(t - \tau) \right\}d\tau\] \[y(t) = \int_{-\infty}^{\infty} x(\tau) h(\tau, t) d\tau\]

In the above equation the input at time $\tau$ is weighted by the factor $h(\tau, t)$ to determine the output of the system at time $t$.

\[\phi(\tau, t) \longrightarrow \begin{cases} \text{Weight for past input} & \tau < t \\ \text{Weight for the present input} & \tau = t \\ \text{Weight for the future input} & \tau > t \\ \end{cases}\]

In the case of a causal LTV system, we have $h(\tau, t) = 0$ for $\tau > t$.

In the case of a discrete-time LTV system, we could express the IO relatioship similarly,

\[y[n] = \sum_{k=-\infty}^{\infty} x[k] h[k, n]\]

where, $h[k, n]$ is the output of the LTV system to an impulse $\delta[n - k]$.

Car	Chosen	Revealed	Remaining	Switch?
A	A	B/C	C/B	No
A	B	C	A	Yes
A	C	B	A	Yes
B	A	C	B	Yes
B	B	A/C	C/A	No
B	C	B	A	Yes
C	A	B	C	Yes
C	B	A	C	No
C	C	B	A	Yes

Car	Chosen	Revealed	Remaining	Switch?
A	A	B/C	C/B	No
A	B	C	A	Yes
A	C	B	A	Yes
B	A	C	B	Yes
B	B	A/C	C/A	No
B	C	B	A	Yes
C	A	B	C	Yes
C	B	A	C	No
C	C	B	A	Yes

SB