Hacking Math I

Matrices

This topic:

1. Matrices
2. Matrix-vector multiplication
3. Linear systems of equations
4. Matrix-matrix multiplication

Reading:

• I2ALA Chapter 6 (Matrices)
• I2ALA Chapter 7 (Matrix Examples)
• I2ALA Chapter 8 (Linear Equations)
• I2ALA Chapter 10 (Matrix Multiplication)
• CTM Chapter 4 (The Matrix)

I. Matrices

Matrices

A matrix $\mathbf A$ is a rectangular array of numbers, of size $m \times n$ as follows:

$\mathbf A = \begin{bmatrix} A_{1,1} & A_{1,2} & A_{1,3} & \dots & A_{1,n} \\ A_{2,1} & A_{2,2} & A_{2,3} & \dots & A_{2,n} \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ A_{m,1} & A_{m,2} & A_{m,3} & \dots & A_{m,n} \end{bmatrix}$

Where the numbers $A_{ij}$ are called the elements of the matrix. We describe matrices as wide if $n > m$ and tall if $n < m$. They are square iff $n = m$.

NOTE: naming convention for scalars vs. vectors vs. matrices.

Slides courtesy of Boyd & Vandenberghe vmls-book.stanford.edu

Implementing a Matrix using Python Lists

How might you do this?

Method 1: Just using a single list and put elements in some predetermined order

$$\begin{bmatrix} A_{1,1} & A_{1,2} & A_{1,3} & \dots & A_{1,n} & A_{2,1} & A_{2,2} & A_{2,3} & \dots & A_{2,n} & \ddots & A_{m,1} & A_{m,2} & A_{m,3} & \dots & A_{m,n} \end{bmatrix}$$

Your matrix algebra functions will need to access the data properly.

Exercise: if you store a matrix this way, what is code to extract element $A_{i,j}$?

Method 2: Nested lists

There are two different ways to interpret this, what are they?

Exercise: implement a zero matrix and identity Matrix using Python

Identity Matrix

$I_3 = \begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & 0 \\ 0 & 0 & 1\\ \end{bmatrix}$

Here we can write $\textit{I}\textit{B} = \textit{B}\textit{I} = \textit{B}$ or $\textit{I}\textit{I} = \textit{I}$

Again, it is important to reiterate that matrices are not in general commutative with respect to multiplication. That is to say that the left and right products of matrices are, in general different.

$AB \neq BA$

Scalar Multiplication

Scalar multiplication of a matrix $\textit{A}$ and a scalar α is defined to be a new matrix $\textit{B}$, written $\textit{B} = \alpha\ \textit{A}$ or $\textit{B} = \textit{A} \alpha$, whose components are given by $b_{ij} = \alpha a_{ij}$.

Matrix Addition

Addition of two $m \times n$ -dimensional matrices $\textit{A}$ and $\textit{B}$ is defined as a new matrix $\textit{C}$, written $\textit{C} = \textit{A} + \textit{B}$, whose components $c_{ij}$ are given by addition of each component of the two matrices, $c_{ij} = a_{ij}+b_{ij}$.

Matrix Equality

Two matrices are equal when they share the same dimensions and all elements are equal. I.e.: $a_{ij}=b_{ij}$ for all $i \in I$ and $j \in J$.

Exercise: make functions to implement these with your matrix.

Matrix Transpose

The transpose of a matrix $\textit{A}$ is formed by interchanging the rows and columns of $\textit{A}$. That is

$(A^T)_{ij} = A_{ji}$

Example 1:

$\textit{A} = \begin{bmatrix} 1 & 2 \\ 0 & 1 \\ \end{bmatrix}$

$\textit{A}^{T} = ?$

Matrix Transpose

The transpose of a matrix $\textit{A}$ is formed by interchanging the rows and columns of $\textit{A}$. That is

$(A^T)_{ij} = A_{ji}$

Example 1:

$\textit{A} = \begin{bmatrix} 1 & 2 \\ 0 & 1 \\ \end{bmatrix}$

$\textit{A}^{T} = \begin{bmatrix} 1 & 0 \\ 2 & 1 \\ \end{bmatrix}$

Matrix Transpose

The transpose of a matrix $\textit{A}$ is formed by interchanging the rows and columns of $\textit{A}$. That is

$(A^T)_{ij} = A_{ji}$

Example 2:

$\textit{B} = \begin{bmatrix} 1 & 2 \\ 0 & -3 \\ 3 & 1 \\ \end{bmatrix}$

$\textit{B}^{T} = ?$

Matrix Transpose

The transpose of a matrix $\mathbf A$ is formed by interchanging the rows and columns of $\mathbf A$. That is

$(\mathbf A)^T_{ij} = A_{ji}$

Example 2:

$\mathbf B = \begin{bmatrix} 1 & 2 \\ 0 & -3 \\ 3 & 1 \\ \end{bmatrix}$

$\mathbf{B}^{T} = \begin{bmatrix} 1 & 0 & 3 \\ 2 & -3 & 1 \\ \end{bmatrix}$

BONUS:

Show that $(\mathbf{A}\mathbf{B})^{T} = \mathbf{B}^{T}\mathbf{A}^{T}$.

Hint: the $ij$th element on both sides is $\sum_{k}A_{jk}A_{ki}$

Exercise: make a function to transpose a Matrix using Python

II. Matrix-vector multiplication

Matrix-Vector Multiplication

Two perspectives:

1. Linear combination of columns

2. Dot product of vector with rows of matrix

$\begin{bmatrix} 2 & -6 \\ -1 & 4\\ \end{bmatrix} \begin{bmatrix} 2 \\ -1 \\ \end{bmatrix} = ?$

Test both ways out.

Matrix-Vector Multiplication examples

Lab: Matrix-vector multiplication

We just defined two different procedures for computing the matrix-vector product. Let us write them in Python.

Suppose we defined vectors as lists, and a matrix as a list of vectors.

1. Write code to compute the matrix vector product assuming $\mathbf A$ is a list of rows
2. Write code to compute the matrix vector product assuming $\mathbf A$ is a list of columns

III. Linear Systems of Equations

Matrix equations $\mathbf A \mathbf x = \mathbf b$

Can view a matrix as list of rows or list of columns

Hence can view this system two different ways.

Perspective 1: Dot products ("System of Linear equations")

$$ \mathbf a^{(1)} \cdot \mathbf x = b_1 \\ \mathbf a^{(2)} \cdot \mathbf x = b_2 \\ \vdots \\ \mathbf a^{(m)} \cdot \mathbf x = b_m \\ $$

where $\mathbf a^{(i)}$ are rows of $\mathbf A$

Perspective 2: Linear combination of columns

$$ x_1 \mathbf a_1 + x_2 \mathbf a_2 + ... + x_n \mathbf a_n = \mathbf b $$

where $\mathbf a_1$ are columns of $\mathbf A$.

Linear system examples

IV. Matrix-matrix Multiplication

Matrix Multiplication

Multiplication of an $m \times n$ -dimensional matrices $\textit{A}$ and a $n \times k$ matrix $\textit{B}$ is defined as a new matrix $\textit{C}$, written $\textit{C} = \textit{A}\textit{B}$, whose elements $C_{ij}$ are

$$ C_{i,j} = \sum_{l=1}^n A_{i,l}B_{l,j} $$

This can be memorized as row by column multiplication, where the value of each cell in the result is achieved by multiplying each element in a given row $i$ of the left matrix with its corresponding element in the column $j$ of the right matrix and adding the result of each operation together. This sum is the value of the new the new component $c_{ij}$.

Note that the product of matrices and vectors is a special case, under the assumption that the vector is oriented correctly and is of correct dimension (same rules as a matrix). In this case, we simply treat the vector as a $n \times 1$ or $1 \times n$ matrix.

Also note that $\textit{B}\textit{A} \neq \textit{A}\textit{B}$ in general.

Example 1:

$\textit{C} = \textit{A}\textit{B} = \begin{bmatrix} 1 & 2 \\ 0 & 1 \\ \end{bmatrix} \begin{bmatrix} 2 & 0 \\ 1 & 4 \\ \end{bmatrix} = \begin{bmatrix} 4 & 8 \\ 1 & 4 \\ \end{bmatrix}$

Example 2:

$\begin{bmatrix} 1 & 2 \\ 0 & -3 \\ 3 & 1 \\ \end{bmatrix} \begin{bmatrix} 2 & 6 & -3 \\ 1 & 4 & 0 \\ \end{bmatrix} = ?$

Example 2:

$\begin{bmatrix} 1 & 2 \\ 0 & -3 \\ 3 & 1 \\ \end{bmatrix} \begin{bmatrix} 2 & 6 & -3 \\ 1 & 4 & 0 \\ \end{bmatrix} = \begin{bmatrix} 4 & 14 & -3\\ -3 & -12 & 0 \\ 7 & 22 & -9\\ \end{bmatrix}$

Example 3:

$\begin{bmatrix} 2 & 6 & -3 \\ 1 & 4 & 0 \\ \end{bmatrix} \begin{bmatrix} 1 & 2 \\ 0 & -3 \\ 3 & 1 \\ \end{bmatrix} = ?$

Example 3:

$\begin{bmatrix} 2 & 6 & -3 \\ 1 & 4 & 0 \\ \end{bmatrix} \begin{bmatrix} 1 & 2 \\ 0 & -3 \\ 3 & 1 \\ \end{bmatrix} = \begin{bmatrix} -7 & -17\\ 1 & -10 \\ \end{bmatrix}$

Example 4:

$\begin{bmatrix} 2 & -6 \\ -1 & 4\\ \end{bmatrix} \begin{bmatrix} 1 \\ 0 \\ \end{bmatrix} = ?$

Example 4:

$\begin{bmatrix} 2 & -6 \\ -1 & 4\\ \end{bmatrix} \begin{bmatrix} 1 \\ 0 \\ \end{bmatrix} = \begin{bmatrix} 2\\ -1\\ \end{bmatrix}$

QUIZ:

Can you take the product of $\begin{bmatrix} 2 & -6 \\ -1 & 4\\ \end{bmatrix}$ and $\begin{bmatrix} 12 & 46 \\ \end{bmatrix}$ ?

If question seems vague, list all possible ways to address this question.

Matrix multiplication examples

Properties of Matrices

For three matrices $\mathbf{A}$, $\mathbf{B}$, and $\mathbf{C}$ we have the following properties

1. Commutative Law of Addition: $\mathbf{A} + \mathbf{B} = \mathbf{B} + \mathbf{A}$

2. Associative Law of Addition: $(\mathbf{A} + \mathbf{B}) + \mathbf{C} = \mathbf{A} + (\mathbf{B} + \mathbf{C})$

3. Associative Law of Multiplication: $\mathbf{A}(\mathbf{B}\mathbf{C}) = (\mathbf{A}\mathbf{B})\mathbf{C}$

4. Distributive Law: $\mathbf{A}(\mathbf{B} + \mathbf{C}) = \mathbf{A}\mathbf{B} + \mathbf{A}\mathbf{C}$

5. Identity: There is the matrix equivalent of one. We define a matrix $\mathbf{I_n}$ of dimension $n \times n$ such that the elements of $\mathbf{I_n}$ are all zero, except the diagonal elements $i=j$; where $I_{i,i} = 1$

6. Zero: We define a matrix $\mathbf 0$ of $m \times n$ dimension as the matrix where all components $(\mathbf 0)_{i,j}$ are 0

Prove these are true using the properties of scalar algebra

Lab: Matrix-matrix multiplication

There are yet more ways to programmatically implement matrix multiplication

Let us focus on just the two which are direct extensions of the preview matrix-vector multiplication methods

Use your dense vector functions to perform vector matrix multiplication - both by columns and by rows