Comprehensive survey on the effectiveness of sharpness aware minimization and its progressive variants

As advancements push for larger and more complex Artificial Intelligence (AI) models to improve performance, preventing the occurrence of overfitting when training overparameterized Deep Neural Networks (DNNs) remains a challenge. Despite the presence of various regularization techniques aimed at mitigating this issue, poor generalization remains a concern, especially when handling diverse and limited data. This paper explores one of the latest and most promising strategies to address this challenge, Sharpness Aware Minimization (SAM), which concurrently minimizes loss value and sharpness-related loss. While this method exhibits substantial effectiveness, it comes with a notable trade-off in increased training time and is founded on several approximations. Consequently, several variants of SAM have emerged to alleviate these limitations and bolster model performance. This survey paper examines the significant advancements achieved by SAM, delves into its constraints, and categorizes recent progressive variants that further enhance current State-of-the-Art results.