Home
Deep Learning For Vision

Heuristics for Avoiding Bad Local Minima

1. Understanding the Problem: Bad Local Minima in Deep Learning

  • Local minima are points in the loss landscape where the loss function value is lower than neighboring regions but are not the absolute lowest (global minimum).

  • In deep neural networks, the highly non-convex nature of loss functions can create many local minima and saddle points.

  • Bad local minima are suboptimal minima leading to poor generalization or high training error if the optimization gets trapped.

  • However, in deep networks, evidence suggests local minima are often not a serious problem compared to saddle points.AL3502.pdf

2. Heuristics to Avoid Bad Local Minima

2.1 Initialization Strategies

  • Careful weight initialization (e.g., Xavier/Glorot, He initialization) helps start optimization in favorable regions of the loss surface.

  • Prevents saturation of activation functions and maintains gradient flow, avoiding poor initial points that lead to bad minima.

2.2 Learning Rate Scheduling

  • Adaptive learning rates: Using schedules (step decay, cyclical learning rates, warm restarts) allows the optimizer to escape shallow local minima by increasing or decreasing step sizes dynamically.

  • Large initial rates enable exploration; smaller rates later refine convergence.

2.3 Stochastic Optimization

  • Stochastic Gradient Descent (SGD) and its variants inject noise due to mini-batch sampling.

  • Noise helps jump out of local minima or saddle points by not converging too quickly.

  • Momentum-based optimizers (Adam, RMSProp) smooth updates and help overcome bad minima traps.

2.4 Architectural Choices

  • Skip connections / Residual Networks (ResNet) allow gradient flow through identity mappings, reducing the likelihood of getting stuck in poor minima.

  • Increasing network depth while maintaining gradient norms helps avoid certain bad minima.

2.5 Regularization Techniques

  • Dropout randomly silences neurons during training, adding noise and preventing overfitting to local minima.

  • Weight decay / L2 regularization discourages complex models that could overfit to suboptimal minima.

2.6 Batch Normalization

  • Normalizes activations between layers to reduce internal covariate shift.

  • Creates smoother, better-conditioned optimization landscapes, reducing poor local minima chances.

3. Practical Tips

  • Use early stopping to avoid training too long in poor minima.

  • Experiment with multiple random starts—train models from different initial weights and pick the best.

  • Visualize loss landscape or track gradient norms to detect trapping behavior.

4. Summary Table of Heuristics

HeuristicDescriptionEffect
Weight InitializationXavier, He initializersStart in “good” regions of parameter space
Learning Rate SchedulingStep decay, cyclic learning ratesPrevent premature convergence to bad minima
Stochastic OptimizationSGD noise and momentumEscape shallow minima and saddle points
Architectural DesignResNets, skip connectionsHelp gradients bypass poor regions
RegularizationDropout, weight decayGuide optimization towards generalizable minima
Batch NormalizationNormalize activationsBetter conditioned and smoother loss surfaces

5. Conclusion

Avoiding bad local minima in deep learning combines theoretical understanding of loss landscapes with empirical heuristics like smart initialization, stochastic training, adaptive rates, and appropriate architectural choices. These heuristics collectively improve optimization robustness and model performance.