BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//wp-events-plugin.com//6.4.7.2//EN
TZID:Europe/Paris
X-WR-TIMEZONE:Europe/Paris
BEGIN:VEVENT
UID:0-919@lptms.universite-paris-saclay.fr
DTSTART;TZID=Europe/Paris:20231205T110000
DTEND;TZID=Europe/Paris:20231205T120000
DTSTAMP:20231201T105443Z
URL:http://www.lptms.universite-paris-saclay.fr/seminars/seminaire-du-lptm
 s-antonio-sclocchi-epfl-2/
SUMMARY:Séminaire du LPTMS : Antonio Sclocchi (EPFL Lausanne) - Salle des 
 séminaires du FAST et du LPTMS\, bâtiment Pascal n°530 - 5 Déc 23 11:0
 0
DESCRIPTION:On the different regimes of Stochastic Gradient DescentAntonio 
 Sclocchi (EPFL Lausanne)The success of modern deep learning relies on the 
 way neural networks are trained\, which consists in the optimization of a 
 high-dimensional loss landscape. This is done with the Stochastic Gradient
  Descent (SGD) algorithm\, where the loss gradients are estimated on a sma
 ll batch of the data at each time step. The choice of the batch size and t
 he step size (or learning rate) is observed to be important to have good p
 erformances in real applications\, but it is poorly theoretically understo
 od and relies heavily on expensive grid-search procedures. In this work\, 
 we clarify how the batch size and the learning rate affect the training dy
 namics of neural networks\, leading to a phase diagram with three distinct
  dynamical regimes: (i) a noise-dominated phase\, where SGD is described b
 y a stochastic process\, (ii) a large-first-step dominated phase\, and (ii
 i) a phase where it is equivalent to simple Gradient Descent (GD). We obta
 in these results in a teacher-student perceptron model and show empiricall
 y that our predictions still apply to deep networks on benchmark tasks\, l
 ike image classification. Our results lead to new predictions on how the s
 ize of the training dataset and the hardness of the task affect the traini
 ng dynamics\, and open the way to understanding its relationship with neur
 al network performances.------ This seminar will be on zoom also: https://
 universite-paris-saclay-fr.zoom.us/j/93668397465?pwd=YWJUMkQ1dUdaV1lXWmFPe
 WZIT3R5QT09 Meeting ID: 936 6839 7465 Passcode: 870463
CATEGORIES:seminars
LOCATION:Salle des séminaires du FAST et du LPTMS\, bâtiment Pascal n°53
 0\, rue André Riviere\, Orsay\, 91405\, France
X-APPLE-STRUCTURED-LOCATION;VALUE=URI;X-ADDRESS=rue André Riviere\, Orsay\
 , 91405\, France;X-APPLE-RADIUS=100;X-TITLE=Salle des séminaires du FAST 
 et du LPTMS\, bâtiment Pascal n°530:geo:0,0
END:VEVENT
BEGIN:VTIMEZONE
TZID:Europe/Paris
X-LIC-LOCATION:Europe/Paris
BEGIN:STANDARD
DTSTART:20231029T020000
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
TZNAME:CET
END:STANDARD
END:VTIMEZONE
END:VCALENDAR