In a paper published in Nature Neuroscience last week, University of Pittsburgh researchers described how reward signals in the brain are modulated by uncertainty. Dopamine signals are intertwined with reward learning; they teach the brain which signals or actions predict the best rewards. New results from the Stauffer lab at the Pitt School of Medicine show that dopamine cues also reflect the certainty surrounding reward predictions.
In short, dopamine signals can teach the brain about the likelihood of receiving a reward.
The authors of the study were three graduate students Catherine (Kati) Rotenhofer, Aydin Alikaya, and Tao Hong, as well as associate professor of neuroscience Dr. William Stauffer.
Rotenhofer (KR) and Stauffer (WS) shared their views on the key messages their work reveals about the inner workings of the brain.
In short, what is the background to this study?
KR: We studied ambiguity – a complex environmental factor that makes it difficult for humans and animals to know what to predict, and this project was a cool detour that emerged organically from our preliminary data. We found something interesting that we didn't expect, and we saw it through.
WS: Dopamine neurons are crucial for reward learning. Dopamine neurons are activated by rewards that are better than predicted and suppressed by rewards that are worse than predicted. This activity model resembles "reward prediction errors", the differences between the received and predicted rewards.
Reward prediction errors are critical for animal learning and machine learning. However, in the classical theories of animal learning and machine learning, "predicted rewards" are simply the average value of past results. While these forecasts are useful, it would be much more useful to predict averages, as well as more complex statistics that reflect uncertainty. Therefore, we would like to know whether the dopamine learning signals reflect these more complex statistics and whether they can be used to train the brain in real-world stimuli.
What are the main conclusions of your work?
VS: The main conclusion is that rare rewards enhance dopamine responses compared to rewards of the same size, which are delivered with greater frequency. This means that the predictive neuronal signal reflects the uncertainty surrounding the predictions, not just the predicted values. This also means that one of the main reward learning systems in the brain can assess uncertainty and potentially teach downstream brain structures about this uncertainty.
What conclusions made you personally excited?
KR: This was the first neural dataset I ever collected, so just to have such clear results showing that dopamine neurons are more complex than we thought in the past was really amazing. It is not every day that you can add new information to a long-accepted dogma in the literature.
WS: Computers are used as an analogy to understand the brain and its functions. However, many neuroscientists believe that computing is more than an analogy. At the most basic level, the brain is an information processing system. I study dopamine neurons because we can "see" the brain doing mathematical calculations. There are several other neural systems where we have such direct evidence of the algorithmic nature of neural responses. It's just fascinating, and these results point to a new aspect of this algorithm. Namely, that reward learning systems adapt to uncertainty.
What recommendations do you have for future research as a result of this study?
KR: I'm very excited to get back to our originally conceived project, which is how the brain deals with ambiguous choices. This will integrate what we now know about how dopamine neurons code information about complex reward conditions with what decision-makers believe about ambiguous choices, and how they prefer to make decisions in these contexts.
VS: The reason we did the research is that I'm interested in understanding how beliefs about probability apply to choices under ambiguity. With ambiguity, economic decisions are made without knowing the probability of the outcome. Therefore, decision-makers are forced to apply their ideas about the probability of choice. We did this research as a first step towards understanding how the value and reward of probability distributions are encoded in the brain, and in what form these beliefs can take. With these results in hand, we will now return to the study of choice! However, I am confident that these results have far-reaching implications for biological and artificial intelligence-based learning systems.