ΒιΆΉΣ³»­΄«Γ½ researchers have developed a device for artificial intelligence that mimics the retina of the eye.

The development could lead to advanced AI that can instantly recognize what it sees, like automatic descriptions of pictures taken by a camera or phone. The technology also has applications in self-driving vehicles and robotics.

The device, which is detailed in a new study in the journal ACS Nano, also outperforms the eye in the number of wavelengths it can see, from ultraviolet to visible light and on to the infrared spectrum.

Its uniqueness also comes from its ability to integrate three different operations into one. Current intelligent imaging technology, like what’s used in self-driving vehicles, requires separate sensing, memorization and processing of data.

By combining the three steps, the ΒιΆΉΣ³»­΄«Γ½-designed device is many times faster than current technology, the researchers say. The technology is also very small, with hundreds of the devices fitting on a one-inch-wide chip.

device for AI that mimics retina of human eye
The technology is very small, with hundreds of the devices fitting on a one-inch-wide chip.

β€œIt will change the way artificial intelligence is realized today,” says study principal investigator Tania Roy, an assistant professor in ΒιΆΉΣ³»­΄«Γ½β€™s Department of Materials Science and Engineering and NanoScience Technology Center. β€œToday, everything is discrete components and running on conventional hardware. And here, we have the capacity to do in-sensor computing using a single device on one small platform.”

The technology expands upon previous work by the research team that created brain-like devices that can enable AI to work in remote regions and space.

β€œWe had devices, which behaved like the synapses of the human brain, but still, we were not feeding them the image directly,” Roy says. β€œNow, by adding image sensing ability to them, we have synapse-like devices that act like β€˜smart pixels’ in a camera by sensing, processing and recognizing images simultaneously.”

For self-driving vehicles, the versality of the device will allow for safer driving in a range of conditions, including at night, says Molla Manjurul Islam ’17MS, the study’s lead author and a doctoral student in ΒιΆΉΣ³»­΄«Γ½β€™s Department of Physics.

β€œIf you are in your autonomous vehicle at night and the imaging system of the car operates only at a particular wavelength, say the visible wavelength, it will not see what is in front of it,” Islam says. β€œBut in our case, with our device, it can actually see in the entire condition.”

β€œThere is no reported device like this, which can operate simultaneously in ultraviolet range and visible wavelength as well as infrared wavelength, so this is the most unique selling point for this device,” he says.

Molla Manjurul Islam, the study’s lead author and a doctoral student in ΒιΆΉΣ³»­΄«Γ½β€™s Department of Physics, examines the retina-like devices on a chip.

Key to the technology is the engineering of nanoscale surfaces made of molybdenum disulfide and platinum ditelluride to allow for multi-wavelength sensing and memory. This work was performed in close collaboration with YeonWoong Jung, an assistant professor with joint appointments in ΒιΆΉΣ³»­΄«Γ½β€™s NanoScience Technology Center and Department of Materials Science and Engineering, part of ΒιΆΉΣ³»­΄«Γ½β€™s College of Engineering and Computer Science.

The researchers tested the device’s accuracy by having it sense and recognize a mixed wavelength image β€” an ultraviolet number β€œ3” and an infrared part that is the mirror image of the digit that were placed together to form an β€œ8.” They demonstrated that the technology could discern the patterns and identify it both as a β€œ3” in ultraviolet and an β€œ8” in infrared.

β€œWe got 70 to 80% accuracy, which means they have very good chances that they can be realized in hardware,” says study co-author Adithi Krishnaprasad ’18MS, a doctoral student in ΒιΆΉΣ³»­΄«Γ½β€™s Department of Electrical and Computer Engineering.

The researchers say the technology could become available for use in the next five to 10 years.

Study co-authors also included Durjoy Dev ’21, a graduate of ΒιΆΉΣ³»­΄«Γ½β€™s doctoral program in electrical engineering; Ricardo Martinez-Martinez ’19MS, a student in ΒιΆΉΣ³»­΄«Γ½β€™s doctoral program in optics and photonics; Victor Okonkwo, a ΒιΆΉΣ³»­΄«Γ½ undergraduate student studying biomedical sciences and mechanical engineering; Benjamin Wu with Stony Brook University; Sang Sub Han, a postdoctoral associate in the Jung Research Group at ΒιΆΉΣ³»­΄«Γ½; Tae-Sung Bae and Hee-Suk Chung with the Korea Basic Science Institute; and Jimmy Touma, a research scientist at the U.S. Air Force Research Laboratory.

The work was funded by the U.S. Air Force Research Laboratory through the Air Force Office of Scientific Research, and the U.S. National Science Foundation through its CAREER program.

Roy joined ΒιΆΉΣ³»­΄«Γ½ in 2016 and is a part of the NanoScience Technology Center with a joint appointment in the Department of Materials Science and Engineering, the Department of Electrical and Computer Engineering and the Department of Physics. Her National Science Foundation CAREER award focuses on the development of devices for artificial intelligence applications. Roy was a postdoctoral scholar at the University of California, Berkeley prior to joining ΒιΆΉΣ³»­΄«Γ½. She received her doctorate in electrical engineering from Vanderbilt University.