Published: Benchmarking autonomous scattering experiments illustrated on TAS

I am happy to announce my first article publication as a postdoc at the Jülich Centre for Neutron Science (JCNS).

My colleagues and I propose a benchmarking procedure that captures essential components when it comes to measuring performance in autonomous scattering experiments. The procedure is designed as a cost-benefit analysis and illustrated on the setting of three-axes spectrometry (TAS).

We are curious of the comments and feedback from the community and open for a critical discussion on our ideas.

Abstract. With the advancement of artificial intelligence and machine learning methods, autonomous approaches are recognized to have great potential for performing more efficient scattering experiments. In our view, it is crucial for such approaches to provide thorough evidence about respective performance improvements in order to increase acceptance within a scientific community. Therefore, we propose a benchmarking procedure designed as a cost-benefit analysis that is applicable to any scattering method sequentially collecting data during an experiment. For a given approach, the performance assessment is based on how much benefit, given a certain cost budget, it is able to acquire in predefined test cases. Different approaches thus get a chance for comparison and can make their advantages explicit and visible. Key components of the procedure, i.e., cost measures, benefit measures, and test cases, are made precise for the setting of three-axes spectrometry (TAS) as an illustration. Finally, we discuss neglected aspects and possible extensions for the TAS setting and comment on the procedure's applicability to other scattering methods. A Python implementation of the procedure to simplify its utilization by interested researchers from the field is also provided.