Vector Notation

In machine learning, vector notation is often used to describe various definitions and algorithm in a more concise way.

Use the following Latex, we can specify row vectors (which are just numpy array):

# row vectors are simply numpy arrays
import numpy as np

a = np.array([1, 3, -5])
b = np.array([4, -2, -1])

Note that the arrow vector notation above is the default style in Latex.

Boldface style for the vector notation is used more often in Machine learning, as follows:

We will use the boldface style for vectors in this course. You can change the default \vec style by using $\renewcommand{\vec}[1]{\mathbf{#1}}$ command. Or just use \mathbf for vectors.

Linear Algebra and Column Vector

You need to know the basic linear algebra to understand various vector/matrix calculations, such as dot product, transpose, etc.

In machine learning, vectors are often represented as column vectors instead of row vectors. A column vector's transpose is a row vector and vice versa.

The dot product can be written as a matrix multiplication:

# row vector/1D array has shape (n,)
# column vector/2D numpy array has shape (n, 1)
 
# a and b are 1D array
a = np.array([1, 3, -5])
b = np.array([4, -2, -1])
print(a, b, a.shape, b.shape)

print('*' * 50)
# reshape a and b to be column vectors
a = a.reshape(3, 1)
b = b.reshape(3, 1)
print(a)
print(b)
print(a.shape, b.shape)

print('*' * 50)
# dot product
np.dot(a.T, b)

Notation Notes

We use the following notations:

• m is the number of instances in the training dataset, e.g., m = 16512 for the housing dataset after splitting
• $\mathbf x^{(i)}$ is a vector of all the feature values of the $i^{th}$ instance in the dataset and $y^{i}$ is the corresponding target value

Recall the first two rows of data:

• first row of training data: $452600 = \theta_0 + \theta_1 * 322 + \theta_2 * 8.3252 + \theta_3 * 41.0$
• second row of training data: $358500 = \theta_0 + \theta_1 * 2401 + \theta_2 * 8.3014 + \theta_3 * 21.0$

The column vector representations of $\theta$ is:

The column vector representations of the first row of feature values $\mathbf x^{(1)}$ is (note 1 is added for the error term $\theta_0$):

The target value is $y^1 = 452600$

so the vectorized notation for the first row is:

Given the followings:

Now, we can put all rows together in vectorized form: