Deep Feed-Forward Neural Networks

---

Overview

• Definition: A deep feed-forward neural network is an artificial neural network consisting of an input layer, multiple hidden layers, and an output layer, where each neuron in a layer is connected to every neuron in the subsequent layer.

• Purpose: To learn a mapping function $y = f^*(x)$ from input $x$ to output $y$, typically for regression or classification tasks.

• Data flows forward only: No connections within a layer or feedback to previous layers (unlike recurrent networks).

---

Structure

1. Layers

• Input Layer: Receives raw input features (e.g., pixel values for images).

• Hidden Layers: One or more layers where data is transformed linearly and non-linearly. Each neuron computes a weighted sum of inputs and applies an activation function (e.g., ReLU, sigmoid, tanh).

• Output Layer: Provides final predictions. Output neurons correspond to regression values or classification probabilities.

2. Connections

• Weights: Connections between neurons are associated with weights, which determine the strength of influence of one neuron on the next. Bias terms shift activation thresholds.

---

Mathematical Operation

For an input $x$:

1. Activation calculation at layer $l$:

   $$z^{(l)} = W^{(l)} a^{(l-1)} + b^{(l)}$$

   Where $W$ are weights, $b$ are biases, and $a^{(l-1)}$ is the output from the previous layer.

2. Apply activation function $\phi$:

   $$a^{(l)} = \phi(z^{(l)})$$

3. The process repeats through all layers until the output.

---

Training the Network

• Feedforward: Input data passes layer-by-layer to produce output predictions.

• Backpropagation: Network computes error by comparing predicted vs. true output. Then, it propagates the error backward from output to input, updating weights using gradient descent.

• Epochs and Loss: Training data is iteratively presented; weights are updated to minimize loss (e.g., mean squared error for regression, cross-entropy for classification).

---

Activation Functions

• Sigmoid: For outputs in (0, 1).

• Tanh: For outputs in (−1, 1).

• ReLU (Rectified Linear Unit): For faster training and better performance in deep networks.

---

Applications

• Pattern recognition: Image, speech, or handwriting classification.

• Regression: Predicting continuous values (e.g., house price prediction).

• Time-series prediction: Stock market, sales forecasting.

• Feature learning: Transformation of raw data to useful representations in deep learning.

---

Key Points

• Deep feed-forward networks: Data flows only forward.

• They approximate complex, nonlinear functions by stacking layers and using nonlinear activations.

• Training uses feedforward pass, error calculation, and backpropagation.

• The choice of network depth, layer width, and activation functions impacts accuracy and efficiency.

---