Brain implants to restore sight, like Neuralink’s Blindsight, face a fundamental problem − more pixels don’t ensure better vision

Elon Musk recently pronounced that the next Neuralink project will be a “Blindsight” cortical implant to restore vision: “Resolution will be low at first, like early Nintendo graphics, but ultimately may exceed normal human vision.”

Unfortunately, this claim rests on the fallacy that neurons in the brain are like pixels on a screen. It’s not surprising that engineers often assume that “more pixels equals better vision.” After all, that is how monitors and phone screens work.

In our newly published research, we created a computational model of human vision to simulate what sort of vision an extremely high-resolution cortical implant might provide. A movie of a cat with a resolution of 45,000 pixels is sharp and clear. A movie generated using a simplified version of a model of 45,000 cortical electrodes, each of which stimulates a single neuron, still has a recognizable cat but most of the details of the scene are lost.

The reason why the movie generated by electrodes is so blurry is because neurons in the human visual cortex do not represent tiny dots or pixels. Instead, each neuron has a particular receptive field, which is the location and pattern a visual stimulus must have in order to make that neuron fire. Electrically stimulating a single neuron produces a blob whose appearance is determined by that neuron’s receptive field. The tiniest electrode – one that stimulates a single neuron – will produce a blob that is roughly the size of your pinkie’s width held at arm’s length.

Consider what happens when you look at a single star in the night sky. Each point in space is represented by many thousands of neurons with overlapping receptive fields. A tiny spot of light, such as a star, results in a complex pattern of firing across all these neurons.

To generate the visual experience of seeing a single star with cortical stimulation, you would need to reproduce a pattern of neural responses that is similar to the pattern that would be produced by natural vision.

In order to do this, you would obviously need thousands of electrodes. But you would also need to replicate the correct pattern of neuronal responses, which requires knowing every neuron’s receptive field. Our simulations show that knowing the location of each neuron’s receptive field in space is not enough – if you don’t also know the orientation and size of each receptive field, then the star becomes a fuzzy mess.

So, even a single star – a single, bright pixel – generates an immensely complex neural response in the visual cortex. Imagine the even more complex pattern of cortical stimulation needed to accurately reproduce natural vision.

Some scientists have suggested that by stimulating exactly the right combination of electrodes, it would be possible to produce natural vision. Unfortunately, no one has yet suggested a sensible way to determine the receptive field of each individual neuron in a specific blind patient. Without that information, there is no way to see the stars. Vision from cortical implants will remain grainy and imperfect, regardless of the number of electrodes.

Sight restoration is not simply an engineering problem. Predicting what kind of vision a device will provide requires knowing how the technology interfaces with the complexities of the human brain.

How we created our virtual patients

In our work as computationalneuroscientists, we develop simulations that predict the perceptual experience of patients seeking to restore their sight.

We have previously created a model to predict the perceptual experience of patients with a retinal implant. To create a virtual patient to predict what cortical implant patients would see, we simulated the neurophysiological architecture of the area of the brain involved in the first stage of visual processing. Our model approximates how receptive fields increase in size from central to peripheral vision and the fact that every neuron has a unique receptive field.

Our model successfully predicted data describing the perceptual experience of participants across a wide range of studies on cortical stimulation in people. Having confirmed that our model could predict existing data, we used it to make predictions about the quality of vision that possible future cortical implants might produce.

Models like ours are an example of virtual prototyping, which involves using computer systems to improve product design. These models can facilitate new technology development and evaluate device performance. Our study shows they can also offer more realistic expectations about what kind of vision bionic eyes might provide.

First do no harm

In our almost 20 years researching bionic eyes, we’ve seen the complexity of the human brain defeat company after company. Patients pay the cost when these devices fail, left stranded with orphaned technologies in their eye or brain.

The Food and Drug Administration could mandate that sight recovery tech companies must develop failure plans that minimize harm to patients when technologies stop working. Possibilities include requiring companies implanting neuroelectronic devices into patients to participate in technology escrow agreements and carry insurance to ensure continuing medical care and technology support if they go bankrupt.

If cortical implants can achieve anything close to the resolution of our simulations, that would still be an accomplishment worth celebrating. Grainy and imperfect vision would be life-changing for many thousands of people who currently suffer from incurable blindness. But this is a moment for cautious rather than blind optimism.

Ione Fine receives funding from NIH National Eye Institute Grants R01EY014645

Geoffrey Boynton receives funding from grant EY R01 EY014645 from the National Institute of Health