FOR IMMEDIATE RELEASE
Researchers at the Weizmann Institute of Science report that a new artificial intelligence model can recreate images a person has viewed using only fMRI brain-scan data, with greater accuracy in color and composition than earlier systems.
Rehovot, Israel, October 2024 — Scientists at the Weizmann Institute of Science in Israel have developed an AI model called Brain-IT that reconstructs images from functional magnetic resonance imaging (fMRI) scans of the brain, according to a paper submitted to the International Conference on Learning Representations. The research team said the system is designed to decode what a person saw based on brain activity alone, and it produces reconstructions that more closely match the original images in detail and color than previous approaches, Popular Mechanics reported.
Michal Irani, who leads the Weizmann Institute lab that developed the model, said existing brain-to-image translation models can preserve the general meaning of an image but often get basic features wrong. “There exist nowadays models that translate brain activity into images, and they can even produce impressive reconstructions that preserve the semantic meaning of the image reasonably well,” Irani said in a press statement reported by Popular Mechanics. “However, they tend to make mistakes in basic features such as composition and color. The new model we developed outperforms them in reconstructing both the content of the image and its details.”
According to Popular Mechanics, the research relied on the Natural Scenes Dataset, which includes fMRI recordings from eight individuals who viewed roughly 73,000 images in total while their brains were scanned. The publication reported that this volume of data is still insufficient to train an AI model robustly on its own, so the Weizmann team built a method that lets the system work in both directions: decoding brain scans into images and also predicting what a brain scan would look like for a given image.
Irani described how this back-and-forth process expands the available training data. “We realized that by translating back and forth – from a random image that had never been viewed in an fMRI machine, to a predicted brain scan, and then back to the image we started with – the models would effectively build themselves a massive dataset,” Irani said in a press statement cited by Popular Mechanics.
Popular Mechanics traced the history of this research area, noting that scientists at Purdue University built a model in 2017 that could identify viewed objects but produced only blurry reconstructed images. By 2022, researchers at Osaka University used the Stable Diffusion model to generate clearer images, though that approach depended on text captions. The new Brain-IT model instead uses what the researchers call a Brain Interaction Transformer, which maps brain activity across roughly 40,000 voxels, or three-dimensional pixels, recording how each one reacts to visual input such as color, location, and whether the subject is viewing a face or an object like a sandwich, according to the report.
Popular Mechanics stated that the model identified 128 brain regions tied to image processing, including some that had not previously been documented by neurologists. The publication said this capability could eventually assist people with severe paralysis in communicating and could give researchers new tools to study how the brain processes visual information by comparing predicted and actual fMRI responses.
Irani also addressed the limits of the technology and what might come next, according to Popular Mechanics. “What remains especially challenging is decoding video … dozens of images change every second, while an fMRI scan takes about two seconds,” Irani said in a press statement. “If we overcome all these obstacles, it’s possible that in the future we may even be able to read dreams.”
About
This release was prepared by the communications office of the organization that publishes this site, which issues formal releases on developments in science and technology reported in the public record. The office compiles and distributes accounts of such findings for public reference and does not conduct independent research into the subjects it covers.
