A Bifurcation Theory for the Equilibria of Modern Hopfield Networks
Vincenzo Schimmenti (MPIPKS Dresden)
Modern Hopfield networks provide a unifying framework for associative memory, transformer attention, diffusion-based generative models, and biological attractor dynamics, linking these systems through a common energy-based dynamics in which states are updated toward weighted combinations of stored patterns. Across these settings, network dynamics is determined by the organization of the energy landscape and the bifurcations of its fixed points. Despite their central role, a general theory of these bifurcations has remained unavailable beyond specific architectures and idealized pattern ensembles. Here we develop a framework for studying the stability and bifurcations of Modern Hopfield network equilibria directly from the correlation geometry of their stored patterns. We derive general stability criteria in the convex-dual formulation and use them to characterize how correlations organize successive localization transitions. For random, block-correlated, and hierarchical pattern ensembles, we identify regimes of staged symmetry breaking and conditions under which intermediate localization levels are skipped through discontinuous transitions. Our results establish a direct connection between memory correlations and the organization of attractors, providing a common framework for studying hierarchical representations in associative-memory and related energy-based dynamics.
Seminar of the Series MLP@P (Machine Learning Physics @ Plateau), joint with LISN and IPhT.
More info at: https://vale1925.wixsite.com/mlpatp