Learning strategies associated with distinct neural signatures: Caltech

The process of learning requires the sophisticated ability to constantly update our expectations of future rewards so we may make accurate predictions about those rewards in the face of a changing environment. Although exactly how the brain orchestrates this process remains unclear, a new study by researchers at the California Institute of Technology (Caltech) suggests that a combination of two distinct learning strategies guides our behavior.

The two learning systems and the neural correlate of their respective prediction error. The model-free system (green) learns the expected rewards of stimuli using a reward prediction error; the model-based system (red) learns a model of the environment using a state prediction error. The systems are combined using an exponential weighting function (blue line) to reach a decision about the available stimuli. [Credit: Jan Gläscher/Caltech]

A paper about the work will appear in the May 27 issue of the journal Neuron.

One accepted learning strategy, called model-free learning, relies on trial-and-error comparisons between the reward we expect in a given situation and the reward we actually get. The result of this comparison is the generation of a "reward prediction error," which corresponds to that difference. For example, a reward prediction error might correspond to the difference between the projected monetary return on a financial investment and our real earnings.

In the second mechanism, called model-based learning, the brain generates a cognitive map of the environment that describes the relationship between different situations. "Model-based learning is associated with the generation of a 'state prediction error,' which represents the brain's level of surprise in a new situation given its current estimate of the environment," says Jan Gläscher, a postdoctoral scholar at Caltech and the lead author of the study.

"Think about a situation in which you always take the same route when driving home after work, but on a particular day the usual way is blocked due to construction work," Gläscher says. "A model-free learning system would be helplessly lost; it is only concerned with taking actions that in the past were rewarding, so if those actions are no longer available it wouldn't be able to decide where to go next. But a model-based system would be able to query its cognitive map and figure out an efficient detour using an alternative route."

"Although the simpler model-free learning mechanism has been well studied and its basic learning mechanism-which is driven by reward prediction errors-is relatively well understood, the mechanisms underlying the more sophisticated model-based learning system, with its rich adaptability and flexibility, are less well understood" says John P. O'Doherty, professor of psychology at Caltech and the Thomas N. Mitchell Professor of Cognitive Neuroscience at Trinity College in Dublin, Ireland.