Applied Maths Seminar: Ngampruetikorn -- Stochastic resetting accelerates reinforcement learning beyond random search

Wave Ngampruetikorn (School of Physics) will be giving a seminar on Wednesday 14th
October at 12pm in Carslaw 451.  After this we will go to lunch as usual.  (Free for
students.)  

Title: Stochastic resetting accelerates reinforcement learning beyond random search 

Abstract: Stochastic resetting---randomly returning a process to a fixed reference
state---has emerged as an effective mechanism for optimizing first-passage properties.
For a diffusive searcher in free space, it makes the infinite mean first-passage time
finite and minimizable.  Existing theory has so far treated processes that search but do
not learn.  Here we investigate how stochastic resetting interacts with reinforcement
learning, in which the underlying dynamics adapt through experience.  In tabular
environments, we find that resetting can speed up learning even when it does not reduce
diffusive search time.  Our study reveals a distinct mechanism through which resetting
speeds the propagation of reward information.  We also show that deterministic resetting
accelerates learning more than the stochastic protocol, but only over a narrower range
of reset rates.  In a continuous-state task with neural-network-based value
approximation, resetting speeds up learning when exploration is hard and rewards are
sparse.  Our results establish stochastic resetting as a simple, tunable mechanism for
accelerating learning by shaping how experience accumulates, extending a canonical
statistical-mechanics phenomenon to adaptive systems.