# Getting Started With Neural Networks

```
from pathlib import Path
DATA_DIR = Path("/kaggle/input")
if (DATA_DIR / "ucfai-core-fa19-nns").exists():
DATA_DIR /= "ucfai-core-fa19-nns"
elif DATA_DIR.exists():
# no-op to keep the proper data path for Kaggle
pass
else:
# You'll need to download the data from Kaggle and place it in the `data/`
# directory beside this notebook.
# The data should be here: https://kaggle.com/c/ucfai-core-fa19-nns/data
DATA_DIR = Path("data")
```

## Before we get started

We need to import some libraries. PyTorch is imported as just `torch`

, otherwise we’ve seen everything else before.

```
import numpy as np
import pandas as pd
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import Dataset, DataLoader
from torch import optim
import time
```

## Tensors

Tensors live at the heart of `PyTorch`

.

You can think of tensors as an $N$-dimensional data container similar to the containers that exist in `numpy`

.

Below we have some *magical* tensor stuff going on to show you how to make some tensors using the built-in tensor generating functions.

```
# create a tensor
new_tensor = torch.Tensor([[1, 2], [3, 4]])
# create a 2 x 3 tensor with random values
empty_tensor = torch.Tensor(2, 3)
# create a 2 x 3 tensor with random values between -1and 1
uniform_tensor = torch.Tensor(2, 3).uniform_(-1, 1)
# create a 2 x 3 tensor with random values from a uniform distribution on the interval [0, 1)
rand_tensor = torch.rand(2, 3)
# create a 2 x 3 tensor of zeros
zero_tensor = torch.zeros(2, 3)
```

To see what’s inside of the tensor, put the name of the tensor into a code block below and run it.

These notebook environments are meant to be easy for you to debug your code, so this will not work if you are writing a python script and running it in a command line.

```
new_tensor
```

```
tensor([[1., 2.],
[3., 4.]])
```

You can replace elements in tensors with indexing. It works a lot like arrays you will see in many programming languages.

```
new_tensor[0, 0] = 5
new_tensor
```

```
tensor([[5., 2.],
[3., 4.]])
```

How the tensor is put together is going to be important, so there are some built-in commands in torch that allow you to find out some information about the tensor you are working with.

```
# type of a tensor
print(new_tensor.type())
# shape of a tensor
print(new_tensor.shape)
print(new_tensor.size())
# dimension of a tensor
print(new_tensor.dim())
```

```
torch.FloatTensor
torch.Size([2, 2])
torch.Size([2, 2])
2
```

## Coming from Numpy

Much of your data manipulation will be done in either `pandas`

or `numpy`

.
To feed that manipulated data into a `Tensor`

for use in `torch`

, you will have to use the `.from_numpy`

command.

```
np_ndarray = np.random.randn(2,2)
np_ndarray
```

```
array([[-1.78410271, -0.98539235],
[-0.06980782, -1.69461514]])
```

```
# NumPy ndarray to PyTorch tensor
to_tensor = torch.from_numpy(np_ndarray)
to_tensor
```

```
tensor([[-1.7841, -0.9854],
[-0.0698, -1.6946]], dtype=torch.float64)
```

## Checking for CUDA (NVIDIA GPUs only)

CUDA will speed up the training of your Neural Network greatly.

Your notebook should already have CUDA enabled, but the following command can be used to check for it.

TL;DR: **CUDA rocks for NNs**

```
torch.cuda.is_available()
```

```
False
```

## Defining Networks

In the example below, we are going to make a simple example to show how you will go about building a Neural Network using a randomly generated dataset. This will be a simple network with one hidden layer.

First, we need to set some placeholder variables to define how we want the network to be set up.

```
n_in, n_h, n_out, batch_size = 10, 5, 1, 10
```

Next, we are going to generate our lovely randomised dataset.

We are not expecting any insights to come from this network as the data is generated randomly.

```
x = torch.randn(batch_size, n_in)
y = torch.tensor([[1.0], [0.0], [0.0], [1.0], [1.0], [1.0], [0.0], [0.0], [1.0], [1.0]])
```

Next, we are going to define what our model looks like. The `Linear()`

part applies a linear transformation to the
incoming data, with `Sigmoid()`

being the activation function that we use for that layer.

So, for this network, we have two fully connected layers with a sigmoid as the activation function. This looks a lot like the network we saw in the slide deck with one input layer, one hidden layer, and one output layer.

```
model = nn.Sequential(
nn.Linear(n_in, n_h),
nn.Sigmoid(),
nn.Linear(n_h, n_out),
nn.Sigmoid()
)
```

Next, let’s define our **loss function**.

For this example, we are going to use **Mean Squared Error**, but there are many different loss functions we could use.

```
criterion = nn.MSELoss()
```

`optimizer`

is how the network will be training.

We are going to be using a standard gradient descent method in this example.

We will have a learning rate of 0.01, which is pretty standard too. You are going to want to keep this learning rate pretty low, as high learning rates cause problems in training.

```
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
```

Now, let’s train!

To train, we combine all the different parts that we defined into one for loop.

```
for epoch in range(50):
# Forward Propagation
y_pred = model(x)
# Compute and print loss
loss = criterion(y_pred, y)
print('epoch: ', epoch,' loss: ', loss.item())
# Zero the gradients
optimizer.zero_grad()
# perform a backward pass (backpropagation)
loss.backward()
# Update the parameters
optimizer.step()
```

```
epoch: 0 loss: 0.24477167427539825
epoch: 1 loss: 0.24474850296974182
epoch: 2 loss: 0.24472537636756897
epoch: 3 loss: 0.24470224976539612
epoch: 4 loss: 0.24467918276786804
epoch: 5 loss: 0.24465613067150116
epoch: 6 loss: 0.24463315308094025
epoch: 7 loss: 0.24461016058921814
epoch: 8 loss: 0.2445872277021408
epoch: 9 loss: 0.24456433951854706
epoch: 10 loss: 0.2445414811372757
epoch: 11 loss: 0.24451863765716553
epoch: 12 loss: 0.24449582397937775
epoch: 13 loss: 0.24447302520275116
epoch: 14 loss: 0.24445028603076935
epoch: 15 loss: 0.24442759156227112
epoch: 16 loss: 0.2444048821926117
epoch: 17 loss: 0.24438223242759705
epoch: 18 loss: 0.2443595826625824
epoch: 19 loss: 0.24433700740337372
epoch: 20 loss: 0.24431447684764862
epoch: 21 loss: 0.24429190158843994
epoch: 22 loss: 0.2442694455385208
epoch: 23 loss: 0.2442469298839569
epoch: 24 loss: 0.24422451853752136
epoch: 25 loss: 0.24420210719108582
epoch: 26 loss: 0.24417972564697266
epoch: 27 loss: 0.2441573590040207
epoch: 28 loss: 0.2441350668668747
epoch: 29 loss: 0.2441127598285675
epoch: 30 loss: 0.2440905123949051
epoch: 31 loss: 0.24406829476356506
epoch: 32 loss: 0.24404609203338623
epoch: 33 loss: 0.24402391910552979
epoch: 34 loss: 0.24400177597999573
epoch: 35 loss: 0.24397964775562286
epoch: 36 loss: 0.24395756423473358
epoch: 37 loss: 0.24393554031848907
epoch: 38 loss: 0.24391348659992218
epoch: 39 loss: 0.24389147758483887
epoch: 40 loss: 0.24386951327323914
epoch: 41 loss: 0.2438475638628006
epoch: 42 loss: 0.24382564425468445
epoch: 43 loss: 0.2438037395477295
epoch: 44 loss: 0.24378187954425812
epoch: 45 loss: 0.24376006424427032
epoch: 46 loss: 0.24373821914196014
epoch: 47 loss: 0.24371647834777832
epoch: 48 loss: 0.2436947077512741
epoch: 49 loss: 0.24367299675941467
```

In this example, we printed out the loss each time we completed an epoch.

This ran very quickly, but with more complex models, those outputs are going to be very important for checking on how your network is doing during the training process which could take hours if not days!

More likely than not, you’re going to see that this network is not converging, which is to be expected with random data.

In our next example, we’re going to be building a proper model with an awesome dataset.

## Diabetes dataset

- What causes someone to have diabetes?

With this dataset, we are going to see if some basic medical data about a person can help us predict if someone is diabetic or not using *magical* neural networks.

First though, let’s get that dataset and see what’s inside.

```
dataset = pd.read_csv(DATA_DIR / "train.csv", header=None).values
pd.DataFrame(dataset).head()
```

## What are we looking at?

This is a fairly small dataset that includes some basic information about an individual’s health.

Using this information, we should be able to make a model that will allow us to determine if a person has diabetes or not.

The last column, `Outcome`

, is a single digit that tells us if an individual has diabetes.

We need to clean up the data a bit, so let’s get rid of the first row with the labels on them.

```
dataset = np.delete(dataset, 0, 0)
pd.DataFrame(dataset).head()
```

Alright, now let’s break up our data into test and train set. Once we have those sets, we’ll need to set them to be tensors. This bit of code below does just that!

```
# split into x and y sets
X = dataset[:,:-1].astype(np.float32)
Y = dataset[:,-1].astype(np.float32)
# Needed to make PyTorch happy
Y = np.expand_dims(Y, axis = 1)
# Test-Train split
from sklearn.model_selection import train_test_split
split = train_test_split(X, Y, test_size=0.1)
xTrain, xTest, yTrain, yTest = split
# Here we're defining what component we'll use to train this model
# We want to use the GPU if available, if not we use the CPU
# If your device is not cuda, check the GPU option in the Kaggle Kernel
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
device
```

### PyTorch Dataset

Our next step is to create PyTorch **Datasets** for our training and validation sets. `torch.utils.data.Dataset`

is an abstract class that represents a dataset and has several handy attributes we’ll utilize from here on out.

To create one, we simply need to create a class which inherits from PyTorch’s `Dataset`

class and override the constructor, as well as the `__len__()`

and `__getitem__()`

methods.

```
class PyTorch_Dataset(Dataset):
def __init__(self, data, outputs):
self.data = data
self.outputs = outputs
def __len__(self):
'Returns the total number of samples in this dataset'
return len(self.data)
def __getitem__(self, index):
'Returns a row of data and its output'
x = self.data[index]
y = self.outputs[index]
return x, y
```

With the class written, we can now create our training and validation datasets by passing the corresponding data to our class

```
train_dataset = PyTorch_Dataset(xTrain, yTrain)
val_dataset = PyTorch_Dataset(xTest, yTest)
datasets = {'Train': train_dataset, 'Validation': val_dataset}
```

### PyTorch Dataloaders

It’s quite inefficient to load an entire dataset onto your RAM at once, so PyTorch uses `DataLoaders`

to load up batches of data on the fly. We pass a batch size of 16, so in each iteration the loaders will load 16 rows of data and return them to us.

For the most part, Neural Networks are trained on **batches** of data so these `DataLoaders`

greatly simplify the process of loading and feeding data to our network. The rank 2 tensor returned by the `DataLoader`

is of size (16, 8).

```
dataloaders = {
x: DataLoader(datasets[x], batch_size=16, shuffle=True, num_workers=4)
for x in ['Train', 'Validation']
}
```

### PyTorch Model

We need to define how we want the neural network to be structured, so let’s set those hyper-parameters and create our model.

```
inputSize = 8 # how many classes of input
hiddenSize = 15 # Number of units in the middle
numClasses = 1 # Only has two classes
numEpochs = 20 # How many training cycles
learningRate = 0.01 # Learning rate
class NeuralNet(nn.Module):
def __init__(self, input_size, hidden_size, num_classes):
super(NeuralNet, self).__init__()
self.fc1 = nn.Linear(input_size, hidden_size)
self.fc2 = nn.Linear(hidden_size, num_classes)
def forward(self, x):
x = F.relu(self.fc1(x))
return torch.sigmoid(self.fc2(x))
```

### PyTorch Training

Now we create an instance of this `NeuralNet()`

class and define the `criterion`

and `optimizer`

we’ll use to train our model.
In our case we’ll use **Binary Cross Entropy Loss**, a commonly used loss function for binary classification problems.

For the `optimizer`

we’ll use **Adam**, an easy to apply but powerful optimizer which is an extension of the popular **Stochastic Gradient Descent** method. We need to pass it all of the trainable parameters with `model.parameters()`

and the **learning rate** we’ll use.

```
# Creating our model
model = NeuralNet(inputSize, hiddenSize, numClasses)
criterion = nn.BCELoss()
optimizer = optim.Adam(model.parameters(), lr = learningRate)
model.to(device)
print(model)
```

At this point we’re finally ready to train our model! In PyTorch we have to write our own training loops before getting to actually train the model. This can seem daunting at first, so let’s break up each stage of the training process.

The bulk of the function is handled by a nested `for`

loop, the outer looping through each epoch and the inner looping through all of the batches of images in our dataset. Each epoch has a **training** and **validation** phase, where batches are served from their respective loaders. Both phases begin by feeding a batch of inputs into the model, which implicity calls the forward() function on the input. Then we calculate the loss of the outputs against the true labels of the batch.

If we’re in training mode, here is where we perform back-propagation and adjust our weights. To do this, we first zero the gradients, then perform backpropagation by calling `.backward()`

on the loss variable. Finally, we call `optimizer.step()`

to adjust the weights of the model in accordance with the calculated gradients.

The remaining portion of one epoch is the same for both training and validation phases, and simply involves calculating and tracking the accuracy achieved in both phases. A nifty addition to this training loop is that it tracks the highest validation accuracy and only saves weights which beat that accuracy, ensuring that the best performing weights are returned from the function.

```
def run_epoch(model, dataloaders, device, phase):
running_loss = 0.0
running_corrects = 0
if phase == 'Train':
model.train()
else:
model.eval()
# Looping through batches
for i, (inputs, labels) in enumerate(dataloaders[phase]):
# ensures we're doing this calculation on our GPU if possible
inputs = inputs.to(device)
labels = labels.to(device)
# Zero parameter gradients
optimizer.zero_grad()
# Calculate gradients only if we're in the training phase
with torch.set_grad_enabled(phase == 'Train'):
# This calls the forward() function on a batch of inputs
outputs = model(inputs)
# Calculate the loss of the batch
loss = criterion(outputs, labels)
# Adjust weights through backpropagation if we're in training phase
if phase == 'Train':
loss.backward()
optimizer.step()
# Get binary predictions
preds = torch.round(outputs)
# Document statistics for the batch
running_loss += loss.item() * inputs.size(0)
running_corrects += torch.sum(preds == labels)
# Calculate epoch statistics
epoch_loss = running_loss / datasets[phase].__len__()
epoch_acc = running_corrects.double() / datasets[phase].__len__()
return epoch_loss, epoch_acc
```

```
def train(model, criterion, optimizer, num_epochs, dataloaders, device):
start = time.time()
best_model_wts = model.state_dict()
best_acc = 0.0
print('| Epoch\t | Train Loss\t| Train Acc\t| Valid Loss\t| Valid Acc\t|')
print('-' * 73)
# Iterate through epochs
for epoch in range(num_epochs):
# Training phase
train_loss, train_acc = run_epoch(model, dataloaders, device, 'Train')
# Validation phase
val_loss, val_acc = run_epoch(model, dataloaders, device, 'Validation')
# Print statistics after the validation phase
print("| {}\t | {:.4f}\t| {:.4f}\t| {:.4f}\t| {:.4f}\t|".format(epoch + 1, train_loss, train_acc, val_loss, val_acc))
# Copy and save the model's weights if it has the best accuracy thus far
if val_acc > best_acc:
best_acc = val_acc
best_model_wts = model.state_dict()
total_time = time.time() - start
print('-' * 74)
print('Training complete in {:.0f}m {:.0f}s'.format(total_time // 60, total_time % 60))
print('Best validation accuracy: {:.4f}'.format(best_acc))
# load best model weights and return them
model.load_state_dict(best_model_wts)
return model
```

Now, let’s train the model!

```
model = train(model, criterion, optimizer, numEpochs, dataloaders, device)
```

```
# Function which generates predictions, given a set of inputs
def test(model, inputs, device):
model.eval()
inputs = torch.tensor(inputs).to(device)
outputs = model(inputs).cpu().detach().numpy()
preds = np.where(outputs > 0.5, 1, 0)
return preds
```

```
preds = test(model, xTest, device)
```

Now that our model has made some predictions, let’s find the Mathew’s Correlation Coefficient:

```
# import functions for matthews and confusion matrix
from sklearn.metrics import confusion_matrix, matthews_corrcoef
matthews_corrcoef(preds, yTest)
```

Let’s check the confusion matrix

```
confusion_matrix(preds, yTest)
```

Ehhhhhh, that’s not bad…

There’s probably a bunch of things we could do to improve accuracy.

**Why don’t we have you give it a shot?**

## Make this model better!

There is no right or wrong way to optimise this model. Use your understanding of Neural Networks as a launching point. You can use the previous code-cells to save some time.

There are many aspects to this model that can be changed to increase accuracy, like:

- Make the NN deeper (increase number of layers)
- Change the learning rate
- Add more hidden units
- train for more epochs
- and a whole bunch more!

```
# TODO, make a better model!
### BEGIN SOLUTION
inputSize = 8 # how many classes of input
hiddenSize = 15 # Number of units in the middle
numClasses = 1 # Only has two classes
numEpochs = 69 # How many training cycles
learningRate = .01 # Learning rate
class NeuralNet(nn.Module):
def __init__(self, input_size, hidden_size, num_classes):
super(NeuralNet, self).__init__()
self.fc1 = nn.Linear(input_size, hidden_size)
self.fc2 = nn.Linear(hidden_size, hidden_size)
self.fc3 = nn.Linear(hidden_size, num_classes)
def forward(self, x):
x = F.relu(self.fc1(x))
x = F.relu(self.fc2(x))
return torch.sigmoid(self.fc3(x))
model = train(model, criterion, optimizer, numEpochs, dataloaders, device)
predictions = test(model, xTest, device)
### END SOLUTION
```

```
# Run this to generate the submission file for the competition!
### Make sure to name your model variable "model" ###
# load in test data:
test_data = pd.read_csv(DATA_DIR / "test.csv", header=None).values
# remove row with column labels:
test_data = np.delete(test_data, 0, 0)
# convert to float32 values
X = test_data.astype(np.float32)
# get indicies for each entry in test data
indicies = [i for i in range(len(X))]
# generate predictions
preds = test(model, X, device)
# create our pandas dataframe for our submission file. Squeeze removes dimensions of 1 in a numpy matrix Ex: (161, 1) -> (161,)
preds = pd.DataFrame({'Id': indicies, 'Class': np.squeeze(preds)})
# save submission csv
preds.to_csv('submission.csv', header=['Id', 'Class'], index=False)
```