29 November - Seminar by Sebastian Goldt at Area Science Park

[Quantitative Life Sciences]

Tomorrow, Wed. 29 November 2024,  Sebastian Goldt (SISSA) will give a 
seminar titled "The Gaussian world is not enough -- how training data 
shapes neural representations"

Location: Conference Hall S39, building C1, Località Padriciano 99, 
34149 Trieste

Time: 11.30am

Abstract: What do neural networks learn from their data ? We discuss 
this question in two learning

paradigms: supervised classification with feed-forward networks, and 
masked language
modelling with transformers. First, we give analytical and experimental 
evidence for a
“distributional simplicity bias”, whereby neural networks learn 
increasingly complex
distributions of their inputs. We then show that neural networks learn 
from the higher-order
cumulants (HOCs) more efficiently than lazy methods, and show how HOCs 
shape the learnt
features. We finally characterise the distributions that are learnt by 
single- and multi-layer
transformers, and show a similar distributional simplicity bias for 
masked language modelling.


-- 
Erica Sarnataro
Group Secretary
Quantitative Life Sciences
The Abdus Salam International Centre for Theoretical Physics (ICTP)
Trieste,  Italy
Tel. +39-040-2240623
www.ictp.it/research/qls.aspx
e-mail: qls at ictp.it