Popping the hood on synaesthesia – what’s going on in there?

-->Synaesthesia – a “mixing of the senses” – was a popular scientific topic in the late 19^th century, but fell out of favour during the mid-20^th century, mainly due to the influence of behaviorism, which held that subjective experience was not a suitable subject for serious science. The start of this century has seen resurgence in interest in the topic, partly fuelled by the hope that neuroimaging would provide objective measures of what is happening in the brains of people during synaesthetic experiences.

We (Erik O’Hanlon, Fiona Newell and myself) have recently published our own neuroimaging study of synaesthesia, combining structural and functional analyses. Some of what follows is pulled from that paper, which contains references to the many studies cited below. Many of the ideas below are also discussed in a chapter I wrote for the new Oxford Handbook of Synaesthesia: Synaesthesia and cortical connectivity – a neurodevelopmental perspective.

The term synaesthesia refers both to the experience of some kind of cross-activation from one sense to another (but see more below) and to the condition of being prone to such experiences. It can be caused acutely in some people by drugs like LSD or psilocybin, which can famously induce visual experiences in response to music. It can also arise, very rarely, due to brain injuries, which leave one part of the brain without its normal innervation, causing invasion of neighbouring nerve fibres and a rewiring of the source of activation of a region, while the “meaning” of its activation remains the same.

Both of those types differ in many aspects from what is known as developmental synaesthesia. This is a heritable condition in which particular stimuli generate specific and consistent additional sensory percepts or associations in another sensory modality or processing stream. Easy for me to say, I know – that is such a mouthful because it has to encompass many different forms. These include seeing letters or words in colour or associating them with colours, seeing colours in response to sounds (typically words or music), tasting words, feeling tastes as tactile sensation, associating numbers or calendar units with spatial locations and many others. It is surprisingly common, with between 1 and 4% of the population estimated to have the condition.

Though originally defined as a cross-sensory phenomenon, many cases involve more conceptual inducing stimuli (“inducers”) or resultant percepts (“concurrents”). Synaesthesia may thus be better thought of as the association of additional attributes into what some psychologists call the “schema” of the inducing object. Thus, the schema of the letter “A” would incorporate not only its particular shapes and sounds, but also the fact that it is, say, olive-green. Middle C may smell of oranges, Wednesday may be located behind a person’s head and the word “shed” may taste of boiled cabbage – these kinds of associations are idiosyncratic but highly stable in individual synaesthetes.

The mechanism driving these additional percepts or associations is unknown, though most researchers agree it is likely related to functions in the cerebral cortex. This is the part of the brain where specialised areas emerge that are dedicated to processing the kinds of stimuli that often induce synaesthetic experiences or associations – such as letters, words, musical notes, numbers, calendar units. These are specialised categories, each with many different members, which are learned through experience. As a child has repeated exposures to stimuli such as letters, a particular part of the visual cortex becomes specialised for processing them, showing more and more selective responses for letters with greater experience. That region not only becomes more responsive to letters, it becomes less responsive to other stimuli. Also, learning has to sharpen the representations of each letter, so that all the various forms of the letter “A” are recognised as such, while simultaneously being distinguished from “D”, “R”, and other visually similar graphemes. In addition, the shapes for “A” have to be linked to the various sounds that it can make, in different contexts.

In contrast to the inducing stimuli, the concurrent percepts associated with the synaesthetic experience tend to be much simpler: colours, tastes, textures, spatial locations. These perceptual primitives are also typically processed by specialised circuits or areas of the cortex, but ones that mature much earlier and that develop in a manner that is not so strictly driven by experience. For example, a particular area of the visual cortex, called V4, is selectively involved in processing colour: this area is strongly activated by coloured stimuli; if it is stimulated with an electrode, patches of colour may be seen in the visual field; and, finally, damage to V4 can lead to complete colour blindness.

So, a simple model for what is happening in synaesthesia is that activation of one cortical area by an inducing stimulus (say, letters), aberrantly and consistently causes co-activation of another cortical area (say, the colour area), leading to an additional colour percept or association. Neuroimaging seems like the perfect way to test this hypothesis – if true, we should be able to see additional areas “lighting up” in functional magnetic resonance imaging (fMRI) scans when synaesthetes are exposed to stimuli that induce a synaesthetic experience.

By now, about a dozen functional neuroimaging experiments have been performed to try to define the neural correlates of synaesthetic experiences. Most of these have studied subjects with grapheme-colour or sound-colour synaesthesia and many have looked specifically for activation of V4 or other visual areas in response to the presentation of the “inducer” – either aurally presented sounds or visually presented achromatic graphemes. These have indeed provided some insights into the neural basis of synaesthesia but their findings are surprisingly variable.

Some of them have reported exactly the expected observation – extra activation of regions such as V4. However, it is not at all clear that such an effect can be taken as a ground truth, as other studies have not observed this but have seen activation or functional connectivity differences in other visual areas or in other brain regions, such as parietal cortex. Still others have observed no additional activation correlating with the synaesthetic experience at all. One early positron emission tomography (PET) study even found, in addition to some areas of extra activation in coloured-hearing synaesthetes, greater cortical deactivation in other areas in response to spoken words that induced a synaesthetic experience of colour.

What is going on in the brain of synaesthetes during a synaesthetic experience thus remains very much an open question. Phenotypic heterogeneity may explain some of the variation in these results – perhaps all of the results are “right” and mechanisms differ across synaesthetes in different studies. Even if that is the case, a simple model of excess cross-activation between highly restricted cortical areas seems too minimal to accommodate all these findings. Rather, these findings suggest that differences in connectivity may be quite extensive in the brains of synaesthetes, a hypothesis which is supported by structural neuroimaging studies. 

These studies have been performed to try and identify anatomical correlates of the condition of synaesthesia (as opposed to the fMRI experiments which are looking at the experience of synaesthesia). They aimed to test the hypothesis that cortical modularity breaks down in people with synaesthesia due to the presence of additional anatomical connections between normally segregated cortical areas. (The alternative type of model proposes altered neurochemistry, leading to disinhibition of normally existing connections).

Here, the findings are somewhat more consistent, at least on a general level. Several studies have now identified structural differences in the brains of synaesthetes compared to controls. In almost all cases, synaesthetes showed greater volumes of areas of grey or white matter or greater "fractional anisotropy" within certain white matter tracts than controls. Some of these differences are in the general region of visual areas thought to be involved in the synaesthetic experience but others are more widespread, in parietal or even frontal regions. A recent study analysed global connectivity patterns in the brains of synaesthetes, using networks derived from correlations in cortical thickness. The global network topology was significantly different between synaesthetes and controls, with synaesthetes showing increased clustering, suggesting global hyperconnectivity. The differences driving these effects were widespread and not confined to areas hypothesised to be involved in the grapheme-colour synaesthetic experience itself. Widespread functional connectivity differences have also been observed in a study using resting-state fMRI.

There is thus a strong general trend: the brains of groups of synaesthetes do show structural differences to those of groups of controls, these are concentrated in occipital and temporal regions but extend also to parietal and frontal lobes, and they almost always involve increases in the measured parameters in synaesthetes. Though the exact locations of such differences vary between studies, the fact that they all agree in the direction of the effects strongly argues that they represent a real, generalizable finding.

If only we knew what it meant. It could mean that the primary cause of synaesthesia is really a structural difference in the brain. However, the imaging parameters measured (like volume of some cluster of grey matter or fractional anisotropy of a white matter tract) are really quite crude and influenced by many variables at a cellular neuroanatomical level. What has not yet emerged is tractography evidence showing an example of connections that are clearly not present in non-synaesthetes. It is thus not obvious how the observed structural differences can explain the synaesthetic experience. It could just as well be that structural differences are secondary and arise due to a lifetime of altered activity patterns in the neural circuits involved. Or the structural differences might be entirely unrelated to the experience of synaesthesia and reflect instead some broader phenotypes associated with the condition.

With this as background, we designed a neuroimaging study aimed at probing the functional involvement in the synaesthetic experience of areas with structural differences. What we found surprised us.

We compared a group of 13 synaesthetes with a group of 11 controls (decent sample sizes for this field, but more on that below). First we looked for average structural differences between the members of these two groups. Using a method called voxel-based morphometry, we identified multiple clusters of increased volume of either grey or white matter in the synaesthetes compared to controls. We also used diffusion-weighted imaging to look at the structural parameters of nerve fibres and found multiple regions of increased fractional anisotropy in synaesthetes compared to controls. Similar to previous studies, these structural differences were concentrated in but not exclusive to the back of the brain (occipital and temporal lobes) and were all increases in synaesthetes.

So far, so good – these results generally replicate and extend previous findings. We then used fMRI to investigate how the areas showing a structural difference responded to stimuli that induce a synaesthetic experience. All the synaesthetes in the study had grapheme-colour synaesthesia – they attribute colours to letters of the alphabet. We showed them images of letters or of non-meaningful characters, as a contrast, and examined responses in nine areas of increased grey matter volume. Four of those areas showed a differential response to this contrast, in synaesthetes but not in controls (a “group by condition interaction”).

When we looked more closely at the responses in these areas we found something really surprising. Two of them showed a clear difference in response to letters, but this was driven by a very strong reduction in activity in synaesthetes. Not only was the BOLD (blood oxygen level-dependent) signal lower than in controls, it was lower than baseline in those voxels. There is good evidence that negative BOLD signals of this type reflect cortical deactivations – a suppression of neuronal activity in that region. None of the areas showed a greater response to letters in synaesthetes.

We also performed an unbiased, whole-brain analysis with the same contrast, again expecting to find regions with an increased selective response to letters in synaesthetes. We found fourteen areas showing a group by condition interaction, but none of these were driven by increased activation to letters in synaesthetes. Three of them were driven by negative BOLD responses in synaesthetes (these did not overlap the areas with grey matter volume differences).

What does this all mean?

My first thought, and I hope it is yours too, is: “possibly nothing”. After all, these are unexpected results from exploratory analyses. While they are corrected for multiple tests, they still could represent a false positive observation – a statistical blip that occurred in that experiment, with that sample, that does not represent a generalizable finding. This is a problem that dogs the fMRI literature and there is only one solution to it – replication, replication, replication! Because our study was designed as at least a conceptual replication and extension of previous findings, we did not include a separate replication sample. (It was honestly also partly because the field does not demand it). If we were designing a similar study today, I would certainly aim for a larger sample and an independent replication sample (and would hope that funding agencies would begin to apply these standards more rigorously).

Actually though, this finding is not completely novel – cortical deactivations were previously reported in response to synaesthesia-inducing stimuli in a PET study, some in the same areas we observe. Whether they have occurred in other fMRI studies is a little hard to know – experimental designs focusing on specific regions or looking specifically for positive differences may have missed these kinds of effects.

The idea that cortical deactivations might be involved in synaesthetic experiences is also neither unprecedented nor outlandish. Here’s what we say in the paper:

“One possible, though speculative, explanation for these observations relates to the fact that the synaesthetic percept or association is internally generated and often reported as being “in the mind’s eye”. A number of studies have shown that generation of an internal sensory representation induces deactivation of regions which might compete for attention or provide conflicting information. For example, visual imagery induces negative BOLD in auditory cortex, verbal memory induces deactivation across auditory and visual cortices and imagery of visual motion induces deactivation of early visual cortices (V1-3). Amedi and colleagues found a strong correlation across subjects between the deactivation of auditory cortex during visual mental imagery and their score on the vividness of visual imagery questionnaire (VVIQ). We have previously reported that synaesthetes tend to score higher on this imagery measure. This is not to suggest that the synaesthetic percepts arise from the same processes as mental imagery per se – there is evidence from functional imaging that this is not the case. But it is possible that the vividness of a mental image and of a synaesthetic percept both rely on deactivation of other areas. 

Such a conclusion is supported by findings from a transcranial direct current stimulation (tDCS) study.

[This technique basically hooks up a 9-volt battery to electrodes on your scalp, and applies small zaps of current in particular patterns. It can be applied to affect particular regions and to either activate them or inhibit them. Activating motor cortex can cause muscle movements while activating visual cortex can cause perception of winking lights or “phosphenes” in the visual field].

Terhune and colleagues found that synaesthetes showed enhanced cortical excitability of primary visual cortex, with a 3-fold lower phosphene detection threshold in response to activation by tCDS. [This finding is consistent with a previous study from our own group using electroencephalography, which found that the amplitude of early visual evoked potentials was larger in synaesthetes compared to controls, even in response to very simple visual stimuli that did not induce a synaesthetic experience].

They tested whether this hyperexcitability of primary cortex could be either a contributing source to the generation of the synaesthetic percept, or, alternatively, a competing signal, which would interfere with the conscious perception of the synaesthetic percept. They show strong evidence that the latter is the case – stimulation or inhibition of primary visual cortical activity diminished or enhanced, respectively, the synaesthetic experience, based on both self-reports and behavioural interference measures. It thus seems plausible that the cortical deactivations we observe in response to stimuli that induce the synaesthetic experience could be an important part of that response, possibly involved in reducing the signals of competing percepts and allowing the internally generated synaesthetic percept to reach conscious awareness.”

Future studies will hopefully tell whether these kinds of cortical deactivations really are an important component of the synaesthetic experience. For now, our findings add to a quite varied set of neuroimaging findings, which have yet to definitively nail down the neural correlates of the synaesthetic experience. Perhaps expecting a single mechanism is a mistake – if the condition is really heterogeneous we may need some other means (like genetics perhaps) to segregate subjects and elucidate the neural underpinnings of this fascinating condition.