# Introduction to Neural Network Theory

Neural networks are modelled after biological neural networks and attempt to allow computers to learn similarly like a human – **reinforcement learning**. The applicability of neural networks include pattern recognition, signal processing, anomaly detection and etc… Neural networks attempt to solve problems that would normally be easy for humans but hard for computers (such as recognising a picture of a person).

## The Perceptron – simplest neural network

A perceptron consists of one or more inputs, a processor, and a single output. A perceptron follows the “feed-forward” mode, where inputs are sent into the neuron and are processed, resulting in an output. A perceptron process follows 4 main steps:

- Receive inputs
- Weight inputs
- Sum inputs
- Generate output

Each input that is sent into the neuron must first be weighted (multiply by some value, often between -1 and 1). When creating a perceptron, we’ll typically begin by assigning random weights. The output of a perceptron is generated by passing that sum through an activation function. For example, in a simply binary output, the activation function is what tells the perceptron whether to “fire” or not. There are many activation functions to choose from. For example, we can make the activation function the sign of the sum. Therefore, if the sum is a positive number, the output is 1, if negative, the output is -1. In addition, we also need to consider **Bias**. We need to add an additional input known as bias input (not shown in graph) to avoid any issues such as what if the sum activation function receives 0 as the sum?

## How to train the perceptron

- Provide the perceptron with inputs for which there is a known output
- Ask the perceptron to guess an output
- Compute the error
- Adjust all the weights according to the error
- Repeat Step 1 and repeat the whole process until we reach an error we are satisfied with

This is how a **single **perceptron works. In order to create a neural network, you have to link many perceptrons together in layers, which could look as follows:

You will have an input layer and an output layer. Any layers in between are known as hidden layers because you don’t directly “see” anything but the input and output. In the picture above, we have two hidden layers. **Deep Learning **is just a neural network with many hidden layers! For example, Microsoft’s state of the art vision recognition uses 152 layers.

## TensorFlow – Deep Learning

TensorFlow was developed by Google and has quickly become the most popular deep learning library. It can run on either CPU or GPU although typically, deep neural networks run much faster on GPU.

**GPU vs CPU Performance**

A CPU consists of a few cores optimised for **sequential** serial processing while a GPU has a massively parallel architecture consisting of thousands of smaller, more efficient cores designed for handling multiple tasks **simultaneously**.

The basic idea of TensorFlow is the ability to create data flows graphs. These graphs have nodes and edges just like the neural networks picture above. The data passed along from layer of nodes to layer of nodes is known as a Tensor. There are two ways to use TensorFlow: 1) **Customisable Graph Session** 2) **Scikit-Learn type interface with Contrib.Learn**.

### TensorFlow Basics

```
import tensorflow as tf
```

** Simple Constants **

```
hello = tf.constant('Hello World')
```

```
type(hello) # Tensor object
```

```
x = tf.constant(100)
```

```
type(x)
```

** Running Sessions **

A TensorFlow Session is a class for running TensorFlow operations. A Session object encapsulates the environment in which Operation objects are executed, and Tensor objects are evaluated.

```
sess = tf.Session()
```

```
sess.run(hello)
```

```
sess.run(x)
```

```
type(sess.run(x))
```

** Operations **

```
x = tf.constant(2)
y = tf.constant(3)
```

```
with tf.Session() as sess:
print('Operations with Constants')
print('Additions: ', sess.run(x+y))
print('Subtractions: ',sess.run(x-y))
print('Mulptiplication: ',sess.run(x*y))
print('Division: ',sess.run(x/y))
```

** Placeholder **

You may not always have the constants right away, and you may be waiting for a constant to appear after a cycle of operations. **tf.placeholder** inserts a placeholder for a tensor that will always be fed. Its value must be fed using the feed_dict optional argument to Session.run().

```
x = tf.placeholder(tf.int32)
y =tf.placeholder(tf.int32)
```

```
x
```

```
add = tf.add(x,y)
sub = tf.subtract(x,y)
mul = tf.multiply(x,y)
```

```
with tf.Session() as sess:
print('Operations with Placeholders')
print('Addition',sess.run(add,feed_dict={x:20,y:40}))
```

** Matrix Multiplication **

```
import numpy as np
```

```
a = np.array([[5.0,5.0]]) # 1x2
```

```
b = np.array([[2.0],[2.0]]) # 2x1
```

```
mat1 = tf.constant(a)
```

```
mat2 = tf.constant(b)
```

```
matrix_multi = tf.matmul(mat1,mat2)
```

```
with tf.Session() as sess:
result = sess.run(matrix_multi)
print(result)
```