Launching an AI and Photonics Initiative at Duke

AI and photonics have a long history with one another. As the nascent field of machine learning took hold in areas outside of niche programming applications in the 1990s, one of its first major successes was in image recognition.

At Duke, this growing revolution was led by Larry Carin and Leslie Collins—both new faculty members in electrical and computer engineering at the time—who began teaching computers how to spot buried landmines. But it didn’t take long for those early image-processing strategies to move from their military roots to the medical realm.

Researchers at the University of Pennsylvania soon added Carin’s algorithms to software they were developing to help doctors classify cancerous cells. While the software already worked reasonably well, the new algorithms made it more accurate and more consistent. The enhanced toolkit also reduced the time physicians had to spend labeling cell samples to train the system because the algorithm automatically selected the best examples.

Jump ahead a quarter of a century, and AI algorithms built to distinguish and classify images—and even to generate original images based on descriptions—are now commonplace. They’ve moved well beyond military and health care applications, too; researchers across Duke are using AI for light-based research ranging from tracking the movements of live organisms to helping autonomous vehicles navigate to shaping electromagnetic waves to carry information.

This widespread growth between the two disciplines is why the Fitzpatrick Institute for Photonics (FIP) is launching a new AI and Photonics Initiative that intends to draw together researchers and increase cross-campus collaboration in these tactics.

“The marriage between AI and photonics represents a powerful and rapidly evolving interdisciplinary field leading to many important applications ranging from medical imaging, robotics and advanced manufacturing to quantum cryptography, computer vision and augmented reality,” said Tuan Vo-Dinh, the R. Eugene and Susie E. Goodson Distinguished Professor of Biomedical Engineering at Duke and director of FIP. “At our institute, we are very excited to be at the interface of these emerging cross-disciplinary areas integrating photonics and AI.”

Co-leading the new initiative are two Duke Engineering faculty members who are at the forefront of this scientific intersection: Roarke Horstmeyer and Jessilyn Dunn, both assistant professors of biomedical engineering.

Horstmeyer uses machine learning methods and deep learning networks to synthesize data taken from imaging devices such as microscopes. The abilities of AI help construct new nonlinear algorithms to assemble overlapping views from different cameras into a single, cohesive picture in real time. The results allow him to develop platforms that stitch dozens of cameras together to create movies of a microscope platform’s entire field of view with resolutions down to the cellular level.

Horstmeyer’s collaborators are using these AI tools to classify cells or detect cancer within slides containing cytopathology material during surgeries. The approach speeds their reading of these slides by automatically segmenting and highlighting important image areas for review and even by offering initial diagnoses.

“AI is expanding the speed of imaging and analyzing pathology slides at the cellular scale beyond human capabilities,” Horstmeyer said.

Dunn’s research in the field mirrors Horstmeyer’s. She is a leading expert in exploring how wearable technologies like smart watches can gather more accurate bioinformation and then translate that data into actionable knowledge and predictions about a person’s health. (Read a related feature article in Broadband magazine.)

In some instances, Dunn uses AI to label data collected by sensors so that other algorithms can quickly make use of it. Beyond these recognition tasks, Dunn also uses AI to take that input data and connect it to a particular outcome, like automatically detecting different stages of sleep or certain types of heart arrhythmia.

“We’re also using AI to help combine information about a person’s various biosignals like heart rate and sleep and predict markers of prediabetes or respiratory infections like Covid, the flu, RSV and others,” Dunn said.

While Horstmeyer’s and Dunn’s specialties lie primarily within the health care industry, there are also many examples of researchers across Duke using AI and photonics in other fields, such as Duke’s trailblazing Metamaterials Group.

Metamaterials are synthetic materials composed of many individual engineered features, which together produce properties not found in nature through their structure rather than their chemistry. In many of the labs at Duke, metamaterials are built to interact with certain wavelengths of light such as the terahertz, infrared or microwave regimes.

Depending on the goal, metamaterials can be built to steer specific wavelengths of light around an object, essentially creating a 2D cloaking device. Metamaterials can also act like flat lenses, shaping and focusing light to better transport data stored in WiFi signals or wirelessly charge a battery.

Designing the size, shape and configuration of the individual elements within a metamaterial, however, is not a straightforward task. Several electrical and computer engineering researchers at Duke are enlisting the help of AI to quickly tackle the job for them rather than trying to write new traditional design software.

AI and machine learning have been an area of great interest among FIP faculty. For example:

• Natasha Litchinitser is using AI to design small silicon chips for routing optical interconnects.
• Willie Padilla is designing sustainable types of thermal energy harvesters and lighting.
• David R. Smith, one of the original inventors of metamaterials, is creating new types of imaging sensors that work with microwaves.
• Guillermo Sapiro works extensively on theory and applications in computer vision, computer graphics, medical imaging, image analysis and machine learning.
• Sina Farsiu and Joseph Izatt have integrated machine learning and optics to design a multi-aperture optical system capable of sampling non-confocal light while simultaneously performing confocal imaging.
• Tuan Vo-Dinh and his team use machine learning methods to rapidly analyze biomedical markers’ signals for early cancer detection and environmental monitoring of toxic pollutants.
• David E. Carlson is focused on novel machine learning and artificial intelligence techniques for ecological and mental health applications. 
• Timothy Dunn develops machine learning techniques in computational neuroscience and traumatic brain injury studies.

“AI is a hot topic today in simulating human responses and fully automating platforms that mimic what humans would do,” Horstmeyer said. “But most of what’s happening with AI and photonics at Duke has more to do with advanced design and data analytics, synthesizing many types of data in a complex way using similar tools.”

With such a wide range of research topics and a large number of faculty in the space, Dunn and Horstmeyer say their first steps with the new initiative will be to build a community, making connections among physics, chemistry, biology, medicine, engineering and more. Even if a researcher isn’t currently using AI but is interested in exploring potential applications, they want to bring them into the group and help get them started.

The duo also plans to create a series of lectures specifically targeting AI in photonics to expand the group’s knowledge base and bring in new ideas for researchers to consider. And, eventually, they want to apply for a large training grant or collaborative grant to cement the AI in Photonics Initiative as an active entity for years to come.