Reinforcement Learning / Bayes tutorial

Want to use learning models to look at your data, but don't know where to start?

Together with Jill O'Reilly, Hanneke developed this reinforcement learning / Bayes tutorial, to help beginning modellers to start using computational models in order to better understand behaviour. The original tutorial is in Matlab, an adapted version for python by Renée is now also available.

The tutorial focuses on learning to implement and fit very basic learning models from two classes of models: one reinforcement learning model, and one normative Bayesian learning model, and show you how you can start to use these models understand your subjects' behaviour.

We first developed this tutorial for the Donders Institute summerschool on Computational Approaches to Decision Making, August 10-14 in Nijmegen, the Netherlands, and by popular demand decided to put it online. We hope that it will help you to take your first steps in the wonderful world of computationals models. It’s now been used by >10.000 unique users and in (under)graduate programmes around the world!

Enjoy :D

Our favourite model fitting and stats toolboxes

If this tutorial has wet your appetite and you want to go little further and are looking help to navigate the complexities of model fitting, and don't want to write your own code, have a look these excellent toolboxes - note that this list is by in no way exhaustive, and new good stuff comes out every day…

  • CBM is a toolbox for hierarchical Bayesian inference for concurrent model fitting and random effects model comparison by Payam Piray. Here is the paper describing the toolbox.

  • HDDM is a python toolbox for hierarchical Bayesian parameter estimation of the Drift Diffusion Model, which can also be integrated with RL models. Here is the paper introducing this toolbox

  • JASP is a great free, open-source statistics toolbox that allows you to do Bayesian (and frequentist) statistics - You can finally accept your null hypothesis!

  • Jean Daunizeau wrote a number of excellent toolboxes, for which you can find the code here 

    • Group-level (random-effect) Bayesian model selection

    • Reviewing DCM for fMRI results (SPM8 compatible)

    • and... a virtual Rubik's cube!

Getting started with DCM for fMRI data

Hanneke often gets questions about where and how to get started using dynamic causal modelling (DCM), so here is a short summary of what DCM is, and some pointers as to where to go next. Note that this section doesn’t get updated very much these days, but we keep it here as we hope it’s still helpful!

The aim DCM is to infer the causal architecture of coupled or distributed dynamical systems. It is a Bayesian model comparison procedure that rests on comparing models of how data were generated. DCM was developed for estimating coupling among brain regions and how that coupling is influenced by experimental changes (e.g., time, context or drug). The basic idea is to construct reasonably realistic models of interacting (cortical) regions or nodes. These models are then supplemented with a forward model of how the hidden states of each node (e.g., neuronal activity) map to measured responses (e.g. BOLD resonse). This enables the best model and its parameters (i.e., effective connectivity) to be identified from observed data. Bayesian model comparison is used to select the best model in terms of its evidence (inference on model-space), which can then be characterised in terms of its parameters (inference on parameter-space). DCM allows us to test hypotheses about how nodes communicate; e.g., whether activity in a given neuronal population modulates the coupling between other populations, in a task-specific fashion. (edited from DCM Scholarpedia page)

Where to learn more? 

If you want start using DCM, I would recommend to attend one of the SPM courses, for example

Literature

Over the years I have compiled a short (and certainly non-exhaustive) reading list that I always present at the end of my lectures, and thought it would be nice to share with you here. When I have time, I will complete the links below. Until then, pubmed is your friend ;-). 

To get you started 

Some extensions over the years

  • Nonlinear DCM for fMRI: Test whether connectivity between two regions is mediated by another region. (Stephan et al. 2008)

  • Embedding computational models in DCMs: (den Ouden et al. 2010, J Neurosci.)

  • DCM as a summary statistic: clustering and classification: Classify patients, or even find new sub-categories (Brodersen et al. 2011, Neuroimage)

  • Integrating tractography and DCM: Prior variance is a good way to embed other forms of information, test validity (Stephan et al. 2009, NeuroImage)

  • Stochastic / spectral DCM: Model resting state studies / background fluctuations (Li et al. 2011 Neuroimage, Daunizeau et al. Physica D 2009)

  • Integrating behavioural and neural data in DCM (2014) Daunizeau et al. PLoS Comput Biol.

And if you like your validation studies, here are some for you

  • reliability (reproducibilty) of parameter estimates (Schuyler et al. 2010) and model selection (Rowe et al. 2010)

  • face validity: simulations and empirical studies (Stephan et al. 2007, 2008)

  • construct validity: comparison with structural equation modelling (Penny et al. 2004)

  • predictive validity:

    • infer correct site of seizure origin (David et al. 2008)

    • infer primary recipient of vagal nerve stimulation (Reyt et al. 2010)

    • predict sensory stimulation (Brodersen et al. 2010)

    • predict presence/absence of remote lesion (Brodersen et al. 2011)