Brain Calligraphy

A paradigm shift for brain-based communication

This article originally appeared on Psychology Today

The promises of modern neurotechnology are plentiful, but perhaps none are quite as enticing as the prospect of using brain-computer interfaces for high-speed communication. Efficiently turning thoughts into words could help people with neurological disorders interact more seamlessly with the world around them, and in a more distant future, enable anyone to reach new levels of symbiosis with our increasingly omnipresent digital realities.

Unfortunately, even the most advanced neural implants available today, coupled with state-of-the-art algorithms, cannot reach anywhere near the interaction bandwidth achieved by a healthy pair of hands. A striking example of this shortcoming is text input. An average adult typing on a keyboard produces about 40 words per minute (exact values are hard to come by and depend on a very large number of factors, but this is a fair estimate). On a phone and with two thumbs, people are slightly slower, with an average of 36 words per minute (Palin et al., 2019). Interestingly, younger age groups (i.e. digital natives who have used smartphones from toddlerhood) achieve close to 40 words per minute, comparable to keyboard users.

Far from these speeds, the best invasive brain-computer interfaces to date only achieve about 8 words per minute. There’s clearly a long way to go. While there are multiple reasons for this stark difference, one of them is how typing happens with a brain-computer interface. Typically, a user will move a cursor on a screen (a mind-controlled computer mouse if you will) and use it to click on a virtual keyboard. This is similar to the painfully inefficient experience of entering text on a TV with a remote control: a single letter at a time. One can easily see why this leads to low typing speeds. To be clear, even a slow interface can be beneficial, especially for people with severe motor disabilities who have a hard time communicating or interacting with a digital device. Still, improving communication bandwidth is a major goal for brain-computer interfaces, with far-reaching clinical implications.

A new approach, described in a recently released pre-print article from Prof. Krishna Shenoy’s lab at Stanford, proposes to dramatically improve the experience of brain-controlled writing (Willett et al., 2020). The fundamental idea is simple: What if instead of controlling a cursor to click on virtual keys, one simply imagined writing letters with a pen?

A brain microelectrode array.

To achieve this, the study participant, who suffered a spinal cord injury, had two arrays of electrodes inserted in the area of his brain responsible for moving his right hand. Each of these arrays can be thought of as a “bed of nails” with about a hundred individual shafts (i.e. “nails") penetrating the brain (see picture). The activity of a handful of neurons close to the tip of each of these “nails" can be measured, allowing researchers to eavesdrop on the complex “neural symphony" of this small region of the brain. By listening in on the neural activity while the participant imagined writing letters with a pen for several hours, the authors were able to train a machine-learning algorithm to predict what letter the participant was trying to write based on its “neural signature.” Once the algorithm was trained, it could print letters on the screen almost as soon as the participant imagined writing them.

Using this new approach, the study participant was able to input text at a rate of just under 20 words per minute, about double what was previously possible with this type of brain implant. Importantly, this impressive gain in speed came without any changes to the implant. It was simply the result of a smarter strategy and better algorithms. One of the reasons this new technique performed better is the significantly reduced reliance on precision. Clicking an arbitrarily small virtual key requires accuracy. The smaller the key, the more precise (and slow) the pointing needs to be. But make the keys larger and the distance between them grows, negating much of the gains with increased travel times. Instead, when writing a letter with an (imagined) pen, the entire trajectory matters, not just the endpoint, relaxing the constraints on precision and speed.

While impressive, these results still fall short of average typing speeds attained with keyboards and smartphones, and about 5% of characters were misidentified by the software. This highlights the long road these systems still have to go before they can match physical interfaces.

Nevertheless, that “simple” changes in strategy can improve performance from 8 to 20 words per minute is a very promising sign. It hints at the possibility of using new paradigms to overcome the limited bandwidth of modern brain-computer user interfaces. Given that handwriting is typically limited to about 20 to 30 words per minute (Hardcastle et al. 1991, Burger et al. 2011), it’s likely that future gains in speed will require even more creative solutions, such as using stenography instead of longhand, or abstract interaction schemes that bypass the “physical" constraints of a virtual keyboard or pen.

Palin, Kseniia, Anna Maria Feit, Sunjun Kim, Per Ola Kristensson, and Antti Oulasvirta. "How do people type on mobile devices? Observations from a study with 37,000 volunteers." In Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services, pp. 1-12. 2019. Willett, Francis R., Donald T. Avansino, Leigh R. Hochberg, Jaimie M. Henderson, and Krishna V. Shenoy. "High-performance brain-to-text communication via imagined handwriting." bioRxiv (2020) Hardcastle, R. A., and C. J. Matthews. "Speed of writing." Journal of the Forensic Science Society 31, no. 1 (1991): 21-29 Burger, Donné Kelly, and Annie McCluskey. "Australian norms for handwriting speed in healthy adults aged 60–99 years." Australian occupational therapy journal 58, no. 5 (2011): 355-363.

If you enjoyed this story, consider subscribing to my website (you can use this link). That way, you'll automagically be notified every time a new story is online, right in your mailbox! I know, technology, right?

←A neuroscientist’s thoughts on Kernel's announcement