What Can Brain Decoding Tell Us About Consciousness?

Presenter: Colin Clifford, University of Sydney


  1. Hey Colin, I have a very basic terminological question: why call all this “decoding”, rather than just ” pattern classification”? A while back (probably at ASSC 10 in Oxford) the philosopher Thomas Metzinger raised this question. Back then, I thought the issue was rather unimportant and almost too pedantic to be considered by an eminent philosopher. But I’m a scientist and I suppose I can ask trivial terminological things. And this is a philosophy conference after all. Meanwhile, the term “decoding” also really seems to have caught on, much to my dismay. My point is, there are two things new in this kind of SVM / multivoxel analysis here. One is, we analyze the patterns of fMRI voxel activity. Second, we classify behavioral response / stimulus / psychological state based on fMRI activity, rather than test for difference in fMRI activity based on behavioral response / stimulus / psychological state. The latter is somewhat trivial and unimportant: if we know P(A|B), it’s easy to work out P(B|A). In any case, once we know brain activity reflects a psychological process, it doesn’t really matter much which way we draw our interference. So the novelty really is because it’s multivoxel. But really, we are analyzing multi-variate data based on blood oxygenation. Certainly, nobody really think the pattern of fMRI activity is a “code” used by the brain? Of course, the terms decoding and encoding come from the people who study neurons and what they represent. In those cases though, we actually write down the encoding model mathematically, and we actually believe that such codes are decoded somewhere downstream in the brain. For BOLD signal though, it is oxygenation in the blood, that changes indirectly due to neural activity. So it’s rather like a bi-product, rather than a code used by anybody.

    Certainly, we don’t say we decode weather, even though we may use multi-variate data to classify and predict weather?

    Is it too trivial a point to care? Thing is, people are saying this to the media and people are really coming to believe that we’re cracking the brain code with fMRI, as if we’re achieving something really novel conceptually, perhaps even more than single-cell studies (oh because they’re only studying p values and correlations, they can’t *predict” what the monkeys see, blah blah). Would it not be nice to use the term more precisely?

  2. Hi Hakwan – thanks for getting the ball rolling with the comments. The only thing I would say in defence of the “decoding” tag is that it beats “mind reading”! But seriously, I agree with much of what you say and make some similar points in my talk, for example that essential novelty in “brain decoding” is that it is *multivariate* pattern analysis rather than the prediction framework. I also say that “mind reading” is a misleading term as it tells us nothing about the “language of the brain” because the voxels upon which it is based are part of the analysis process extrinsic to the brain. Similarly, it tells us nothing about the neural code by which information is transmitted between neurons. So why did I use the term “brain decoding” in the title of my talk? Simply because it was in the title of symposium – i.e. blame Richard Brown! Personally, I prefer “multivariate pattern analysis” for much that same reasons as you outline.

  3. Here I am! I was attending to some minor snafus…

    I will take part of the blame here since I did indeed name the session 🙂

    But ultimately I blame you guys (scientists) and the ASSC!

    Kamitani, Y., & Tong, F. (2005). Decoding the visual and subjective contents of the
    human brain. Nature Neuroscience, 8, 679-685

    Kamitani, Y, & Tong, F. (2006). Decoding seen and attended motion directions from
    activity in the human visual cortex. Current Biology, 16, 1096-1102

    J. Haynes (2006) Decoding mental states from brain activity in humans. Nature Reviews Neuroscience 7, 523-534

    In 2007 Rees gave the Francis Crick Prize Lecture to the Royal Society talk called “Decoding Consciousness”

    Still, though there is something right about calling it ‘decoding’. Granted we are not getting at the actual mechanisms by which the brain encodes information and so in that sense we are not decoding anything at this point (a more likely candidate for that title, in my view, would be the work of people like Wolfe Singer). But we are learning how to “read out” the encoded information in the brain…isn’t this a kind of decoding? Sure it is indirect but so what?

  4. good defense, richard.

    about “reading out” though, that goes back to the issue of p(brain activity|percept) vs p(percept|brain activity), which is neither new nor crucial in multivoxel analysis.

    so if that count as decoding, when the ERP people show that you can predict which of their hands subjects are gonna use to press a button (by looking at the LRP, lateralized readiness potential), that’s also decoding. conceptually nothing is new here, apart from the resolution.

    if resolution is the thing then, why doesn’t single-cell people say they’re doing decoding? (sometimes they do say that when they are actually working out the encoding and decoding models, but that’s not usually the case that that’s specific).

    i fear the answer is partly that their field, unlike fMRI, is more mature and less “spin-driven” (and i’m not talking about quantum mechanics here). research there depends more on scientific quality rather than ability to impress the media. i hope i’m wrong. but if i ain’t, it’s a little sad isn’t it…

  5. Thank you for all of your work on this—the slides were very clear and decoding is an interesting new area. (Great Australia=brain, Sydney=vision image!) A couple of thoughts:

    I haven’t looked at the Schurger paper, and I generally find Aaron to be a responsible thinker, but I wonder why one would think that the issue is consciousness rather than binocular vs. monocular vision? That is, yes one image is visible and one isn’t, but the latter is not visible binocularly but is monocularly. Thus, this seems the more significant difference when mapping the images to different areas of the brain.

    Also, doesn’t it seem a bit early to say that the decoding is even the neural signature of consciousness when it only investigates the location of particular events, rather than the pattern of firing? It seems as though the neural signature is going to be more complicated than thick-grained timing for fine-grained locations.

  6. Also, I thought that “decoding” referred to the connection between the coded formula and the brain pattern. That is, the brain pattern is coded into a formula and the formula is decoded into a brain pattern. Is that correct?

  7. Pattern classification seems to be superior to decoding. However, I disagree that decoding is a better term than mind-reading. All we are looking at are BOLD levels. This is not the code that the brain communicates in. However, using machine learning and pattern classification, we can “read someone’s mind” in certain circumstances. Regardless, what we are doing is pattern classification of BOLD levels to predict, from brain data, what the person is experiencing. Thus, we should call it that.
    Hakwan, we have obviously seen that researchers using fMRI to ‘read minds’ get a lot of media attention, and most of the time it is unwarranted. Remember “Your Brain on Politics” from the NY Times? What a piece of bullshit. However, we shouldn’t let this make us recoil from pattern classification. It’s a big step and important step to go from trying to get a mapping of 1 to 1 brain region to process(like amygdala=anger) to actually mapping out neural networks in the brain temporally and spatially.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s