Online International Collaboration Demonstrates Reliability of Quantum Simulations for Materials Design

Scientists from Duke University teamed up with colleagues from over 30 universities and institutes to investigate to what extent quantum simulations of material properties agree when they are performed by different researchers and with different software. Thanks to an online collaboration, they successfully demonstrated that the most recent generations of codes agree well, in contrast to earlier generations. Their study appears in this week's edition of Science.

Reproducibility does not come easily

It's a corner stone of science: independent yet identical experiments should produce identical results. Only in this way can science identify ‘laws’, which lead to new insight and new technologies. However, several recent studies have pointed out that such reproducibility does not always come spontaneously. In scientific areas as diverse as psychology research and genetic research, cases were identified where repeating previous experiments led to very different results. Even predictions by computer codes require caution, since the way in which theoretical models are implemented may affect simulation results. This is a reason for concern in any field of research that critically depends on computer simulations. For the study and design of materials, for instance, there are several independent software packages available based on quantum physics. They are moreover being used increasingly often in automated procedures with limited human supervision. It is therefore essential to know to what extent predicted materials properties depend on the code that has been used.

Online collaboration brings experts together

Despite the need for reliable property predictions of materials, the reproducibility of quantum simulations had not been investigated systematically before. This is mainly because there is no single person sufficiently skilled in all existing codes. Scientists from Duke University therefore joined forces with more than 60 colleagues, bringing together the know-how of over 30 prominent institutions. The researchers investigated 40 different methods to describe the influence of pressure in 71 different crystals. Due to the highly international composition of the team, discussions and collaboration were mainly conducted via online tools – similarly to the way people collaborate to write Wikipedia. The team can now demonstrate that, although a few of the older methods clearly yield deviating results, predictions by recent codes are entirely equivalent. They moreover define a quality criterion that allows the verification of future software developments against their extensive database. New test data are continuously added to a publicly available website. The researchers involved hope that their work will contribute to higher standards for materials property simulations, and that it will facilitate the development of improved simulation codes and methods.

Led by Kurt Lejaeghere and Stefaan Cottenier of the Center for Molecular Modeling at Ghent University in Belgium, the collaboration investigated 40 different methods to describe the so-called equation of state in 71 different crystals.

“If you want to predict a material’s properties, you need to know what its structure is at given conditions. And to predicts its structure, you need its equation of state,” said Volker Blum, associate professor of mechanical engineering and materials science and of chemistry at Duke, whose code called FHI-aims proved to be one of the most accurate. However, there is more work to be done.

“The equation of state goes a long way to make sure the foundation is sound, but many quantities require additional numerical approximations,” said Blum. “These would be good to test in the future in a similarly broad way. The present study is, in a sense, a trailblazer for such work.”

###

"Reproducibility in density-functional theory calculations of solids." Kurt Lejaeghere et. al. Science 351 (2016), DOI: 10.1126/science.aad3000