Speaker
Description
Procurements of high-performance computers usually involve benchmarking. Benchmarking is the element in a procurement where the fitness of a computer for its intended research uses is assessed by actual tests. Designing and defining these tests to be representative, fair and robust is not trivial and requires specific skills and is helped by experience. The 'return on investment' of good benchmarking can be very substantial. There is also a growing in interest in benchmarking for high-performance computing outside of procurements, particularly from cloud vendors, for assessing and demonstrating the (relative) performance of available systems.
I will talk about the developing benchmarking effort in the UK Pathway to Exascale programme and some similar nascent efforts. A key consideration is how we can make the development and maintenance of benchmarks a continuous and 'living' project and maintain skills in the community, rather following the historic pattern of discrete efforts. Inclusiveness, transparency and openness is central to our efforts. This promotes community building and supports our aim of a living benchmark suite.
