QLS zoom seminar - K. Krishamurti (Princeton University) - 9 March at 15.00 CET

[QLS]

Dear All,

on Tuesday, 9 March2021 at 15:00 CETKamesh Krishnamurti(Princeton 
University), will give a seminartitled:

"*Theory of gating in recurrent neural networks**"*

Abstract: Recurrent neural networks (RNNs) are powerful dynamical 
models, widely used in machine learning (ML) for processing sequential 
data, and in neuroscience, to understand the emergent properties of 
networks of real neurons. Prior theoretical work in understanding the 
properties of RNNs has focused on networks with additive interactions. 
However, gating – i.e. multiplicative – interactions are ubiquitous in 
real neurons, and gating is also the central feature of the 
best-performing RNNs in ML. Here, we study the consequences of gating 
for the dynamical behavior of RNNs. We show that gating leads to slow 
modes and a novel, marginally-stable state. The network in 
this marginally-stable state can function as a robust integrator, and 
unlike previous approaches, gating permits this function without 
parameter fine-tuning or special symmetries. We study the 
long-time behavior of the gated network using its Lyapunov spectrum, and 
provide a novel relation between the maximum Lyapunov exponent and the 
relaxation time of the dynamics. Gating is also shown to give rise to a 
novel, discontinuous transition to chaos, where the proliferation of 
critical points (topological complexity) is decoupled from the 
appearance of chaotic dynamics (dynamical complexity), in contrast to a 
seminal result for additive RNNs. The rich dynamical behavior is 
summarized in a phase diagram indicating critical surfaces and regions 
of marginal stability – thus, providing a map for principled parameter 
choices to ML practitioners. Finally, we develop a field theory 
for gradients that arise in training, by combining the adjoint formalism 
from control theory with the dynamical mean-field theory. This paves the 
way for the use of powerful field theoretic techniques to study training 
and gradients in large RNNs.

http://indico.ictp.it/event/9600/ <http://indico.ictp.it/event/9600/>


Zoom Meeting ID to attend the online seminar: 475-819-702

Join Zoom Meeting:https://zoom.us/j/475819702 <https://zoom.us/j/475819702>


If you haven't registered for previous QLS webinars, please contact 
qls at ictp.it <mailto:qls at ictp.it>to obtain the PASSWORD for this zoom 
meeting.


Kindest regards

Barbara Valassi for QLS