Xiang Cheng: Using Geometry to Push AI’s Boundaries

Xiang Cheng has joined the faculty of Duke University’s Department of Electrical and Computer Engineering, beginning his position as an assistant professor on August 1, 2024. Having a strong interest in machine learning and AI ever since his days as an undergraduate, Cheng focuses on innovative mathematical techniques for understanding and designing deep learning models, specifically on large language models (LLMs) and diffusion models.

“The kinds of tools that are used for AI have changed quite drastically even just from when I started my PhD at UC-Berkeley until now,” said Cheng, who joins Duke from a postdoctoral position in the laboratory of Professor Suvrit Sra at the Massachusetts Institute of Technology. “But throughout all that change, I think certain approaches based on geometry or randomness have always played an important role, which is why I’m particularly interested in those aspects.”

Consider, for instance, the generation of data using a diffusion process, where a large amount of randomness is injected to the dataset, and new data is generated by gradually removing this randomness in a controlled manner using a neural network.

In the case of images, that noise injection can be thought of as overlaying a pattern like that of static on an old television set. But this static does not have to be completely random. The noise that is introduced can have geometric constraints and symmetries. For example, correlating all the pixels that are physically close in the image.

“If those correlations match the underlying structure of the image, it can create an advantage for training the model to work even better,” Cheng said. “I’m working to understand a wide variety of geometries and what kinds of models fit them best.”

Geometric structures can also extend beyond the common visual sense and into the architectures of the algorithms themselves. If you think of the internals of LLMs or any deep learning model as a multi-step dynamical process, it quickly becomes evident that they have their own geometric structures as well. The way these algorithms are constructed introduces correlations that can be exploited to train them more quickly and effectively.

“There are geometric structures in lots of different fields, too, such as molecular chemistry and other physical phenomena,” Cheng said. “Understanding the math behind these kinds of architectures is relevant to many different applications.”

Cheng joins a quickly expanding set of AI experts at Duke, where he sees himself finding a wide range of faculty to collaborate with. The two who immediately come to his mind are Professor Larry Carin, whose AI work is especially focused on medicine and security, and Professor Vahid Tarokh, who pursues new mathematical formulations and approaches to get the most out of datasets. He is also looking forward to Duke Engineering’s proximity and close ties to the Duke University School of Medicine so that he can explore potential applications within molecular biology.

On the teaching side of the equation, Cheng will initially launch two classes in the coming academic year; an undergraduate class focused on the mathematics of machine learning and a graduate course on diffusion models, which focuses on the mathematics of stochastic differential equations.

Besides the opportunities for collaboration and talented students to mentor, Cheng says it was the people and atmosphere at Duke Engineering that drew him to the school.

“The people at any school are the most important piece, and everybody at Duke was extremely welcoming and collaborative,” Cheng said. “I was also impressed by the ease at which faculty can collaborate across departments, whether it be in chemistry or biology, people at Duke do that quite often.”