Optimization for Training Deep Models

Optimization is a core aspect of deep learning, enabling neural networks to learn from data by minimizing a loss function with respect to model parameters.

---

1. Introduction

• Training deep neural networks is posed as an optimization problem, typically minimizing the empirical risk (average training error).

• The goal: Find parameter values (weights and biases) that minimize the chosen loss function and thus improve model accuracy.

---

2. Challenges in Deep Model Optimization

• Non-convexity: Deep networks have highly non-convex loss landscapes with many local minima and saddle points.

• Vanishing & Exploding Gradients: Gradients may become too small or too large, especially in deep networks, hindering effective learning.

• Ill-conditioning: Sharp or flat regions can slow down training or cause instability.

• Generalization vs. Overfitting: Optimization should balance fitting training data and generalizing to new data.

---

3. Basic Optimization Algorithms

Gradient Descent (GD)

• Updates parameters in the direction of the negative gradient of loss.

• Can be performed as:

  • Batch GD: Uses all data for each update.

  • Stochastic Gradient Descent (SGD): Uses one data sample per update for faster, noisier steps.

  • Mini-batch GD: Uses a subset of samples, balancing speed and stability.

Momentum

• Accelerates updates in consistent directions and dampens oscillations.

• Update rule includes a velocity term:

  $$v_{t+1} = \gamma v_t + \alpha \nabla L(\theta)$$ $$\theta_{t+1} = \theta_t - v_{t+1}$$

Adaptive Algorithms (Popular in Deep Learning)

• AdaGrad: Adapts learning rate for each parameter based on historical gradients.

• RMSProp: Resolves AdaGrad’s decaying learning rate via exponential average.

• Adam: Combines momentum and adaptive learning rate; widely effective, robust choice for deep networks.

---

4. Advanced Optimization Methods

• Second-order methods (L-BFGS, Conjugate Gradient):

  • Use curvature information (the Hessian or its approximations); rarely used for deep nets due to computational cost but useful for small models.

• Regularization and Batch Normalization:

  • Help stabilize optimization and improve generalization.

• Learning Rate Scheduling:

  • Decays or adapts the learning rate during training for better convergence.

---

5. Modern Model-Specific Techniques

• Pruning: Remove unnecessary weights for efficiency.

• Quantization: Reduce precision for speed and deployment.

• Knowledge Distillation: Train smaller models to mimic larger ones for deployment.

---

6. Optimization in Practice

• Parameter Initialization: Good initialization (like He/Xavier) avoids poor local minima.

• Batch Size: Affects convergence speed, stability, and generalization.

• Early Stopping: Prevents overfitting by stopping training when validation loss stops improving.

---

7. Schematic Overview

---

8. Conclusion

• Selecting and tuning the optimization algorithms and techniques is critical for training deep learning models efficiently and effectively.

• Combining optimizers with regularization, initialization, and normalization helps address deep learning’s unique challenges.