Neural Network forward propagation and cost

In the previous tutorial, we initialized our model parameters. In this part, we'll compute forward propagation and cost function. Here are the mathematical expression formulas of forward propagation algorithm for one example x(i) from our previous tutorial part:

$$\begin{gathered} z^{\left[1\right]\left(i\right)}=W^{\left[1\right]}{x}^{\left(i\right)}+b^{\left[1\right]} \\ {a}^{\left[1\right]\left(i\right)}=\tan h\left(z^{\left[1\right]\left(i\right)}\right) \\ {z}^{\left[2\right]\left(i\right)}=W^{\left[2\right]}{a}^{\left[1\right]\left(i\right)}+b^{\left[2\right]} \\ \hat{y}=a^{\left[2\right]\left(i\right)}=\sigma \left(z^{\left[2\right]\left(i\right)}\right) \end{gathered}$$

So at first, we'll retrieve each parameter from the dictionary "parameters," and then we'll compute Z[1], A[1], Z[2], and A[2] (the vector of all your predictions on all the examples in the training set). Then we'll store values into the cache, which will be used as an input to the backpropagation function.

Code for our forward propagation function:

Arguments:

X - input data of size (input_layer, number of examples)
parameters - python dictionary containing your parameters (output of initialization function)

Return:

A2 - The sigmoid output of the second activation
cache - a dictionary containing "Z1", "A1", "Z2" and "A2"

def forward_propagation(X, parameters):
    # Retrieve each parameter from the dictionary "parameters"
    W1 = parameters["W1"]
    b1 = parameters["b1"]
    W2 = parameters["W2"]
    b2 = parameters["b2"]

    # Implementing Forward Propagation to calculate A2 probabilities
    Z1 = np.dot(W1, X) + b1
    A1 = np.tanh(Z1)
    Z2 = np.dot(W2, A1) + b2
    A2 = sigmoid(Z2)

    # Values needed in the backpropagation are stored in "cache"
    cache = {"Z1": Z1,
             "A1": A1,
             "Z2": Z2,
             "A2": A2}
    
    return A2, cache

Computing Neural network cost:

Now that we have computed A[2] (in the Python variable A2), which contains a[2](i) for every example, we can compute the cost function, which looks like this:

$J=-\frac{1}{m}\sum _{i=1}^m\left(y^{\left(i\right)}\log \left(a^{\left[2\right]\left(i\right)}\right)-(1-y^{\left(i\right)})\log (1-a^{\left[2\right]\left(i\right)})\right)$

Code for our cost function:

Arguments:

A2 - The sigmoid output of the second activation, of shape (1, number of examples);
Y - "true" labels vector of shape (1, number of examples);
parameters - python dictionary containing parameters W1, b1, W2, and b2.

Return:

cost - cross-entropy cost.

def compute_cost(A2, Y, parameters):
    # number of example
    m = Y.shape[1]

    # Compute the cross-entropy cost
    logprobs = np.multiply(np.log(A2),Y) +  np.multiply(np.log(1-A2), (1-Y))
    cost = -1/m*np.sum(logprobs)

    # makes sure cost is the dimension we expect, E.g., turns [[51]] into 51 
    cost = np.squeeze(cost)
    
    return cost

Full tutorial code:

import os
import cv2
import numpy as np
import matplotlib.pyplot as plt
import scipy

ROWS = 64
COLS = 64
CHANNELS = 3

#TRAIN_DIR = 'Train_data/'
#TEST_DIR = 'Test_data/'

#train_images = [TRAIN_DIR+i for i in os.listdir(TRAIN_DIR)]
#test_images =  [TEST_DIR+i for i in os.listdir(TEST_DIR)]

def read_image(file_path):
    img = cv2.imread(file_path, cv2.IMREAD_COLOR)
    return cv2.resize(img, (ROWS, COLS), interpolation=cv2.INTER_CUBIC)

def prepare_data(images):
    m = len(images)
    X = np.zeros((m, ROWS, COLS, CHANNELS), dtype=np.uint8)
    y = np.zeros((1, m))
    for i, image_file in enumerate(images):
        X[i,:] = read_image(image_file)
        if 'dog' in image_file.lower():
            y[0, i] = 1
        elif 'cat' in image_file.lower():
            y[0, i] = 0
    return X, y

def sigmoid(z):
    s = 1/(1+np.exp(-z))
    return s
'''
train_set_x, train_set_y = prepare_data(train_images)
test_set_x, test_set_y = prepare_data(test_images)

train_set_x_flatten = train_set_x.reshape(train_set_x.shape[0], ROWS*COLS*CHANNELS).T
test_set_x_flatten = test_set_x.reshape(test_set_x.shape[0], -1).T

train_set_x = train_set_x_flatten/255
test_set_x = test_set_x_flatten/255
'''
#train_set_x_flatten shape: (12288, 6002)
#train_set_y shape: (1, 6002)

def initialize_parameters(input_layer, hidden_layer, output_layer):
    # initialize 1st layer output and input with random values
    W1 = np.random.randn(hidden_layer, input_layer) * 0.01
    # initialize 1st layer output bias
    b1 = np.zeros((hidden_layer, 1))
    # initialize 2nd layer output and input with random values
    W2 = np.random.randn(output_layer, hidden_layer) * 0.01
    # initialize 2nd layer output bias
    b2 = np.zeros((output_layer,1))
    
    parameters = {"W1": W1,
                  "b1": b1,
                  "W2": W2,
                  "b2": b2}
    
    return parameters

def forward_propagation(X, parameters):
    # Retrieve each parameter from the dictionary "parameters"
    W1 = parameters["W1"]
    b1 = parameters["b1"]
    W2 = parameters["W2"]
    b2 = parameters["b2"]

    # Implementing Forward Propagation to calculate A2 probabilities
    Z1 = np.dot(W1, X) + b1
    A1 =  np.tanh(Z1)
    Z2 = np.dot(W2, A1) + b2
    A2 = sigmoid(Z2)

    # Values needed in the backpropagation are stored in "cache"
    cache = {"Z1": Z1,
             "A1": A1,
             "Z2": Z2,
             "A2": A2}
    
    return A2, cache

def compute_cost(A2, Y, parameters):
    # number of example
    m = Y.shape[1]

    # Compute the cross-entropy cost
    logprobs = np.multiply(np.log(A2),Y) +  np.multiply(np.log(1-A2), (1-Y))
    cost = -1/m*np.sum(logprobs)

    # makes sure cost is in dimension we expect, E.g., turns [[51]] into 51 
    cost = np.squeeze(cost)
    
    return cost