How do we open access to compute?
Opening access to compute
The UK has become increasingly aware of the benefits of widen and facilitating access to high performance computing (HPC) beyond the traditional physical science domains. The interest in “democratising” access to HPC was initially sparked by the realisation that social, governmental, economic and business challenges could benefit from its application to new domains like natural language processing, forecasting and digital twinning, in addition to more generic simulation and modelling and data analysis usage. The advent of “big data” with data volumes growing at unprecedented rate has been a catalyst for change over the past decade, while the harnessing of technology to support COVID-19 research fostered a new spirit of inter-disciplinary collaboration to deliver on shared goals, with HPC, data analytics and AI/ML playing a central role in the UK’s very successful pandemic programme of work.
We are now at a tipping point where interest in large scale compute (LSC) and HPC is high but where availability, accessibility and usability outside traditional groups remain low. One of the organisations working with UK industry and government to ameliorate this is the Science and Technology Facilities Council’s Hartree Centre, based at Daresbury Laboratory. The centre’s mission is to help UK businesses and organisations, of any size, to explore and adopt HPC, data science and AI technologies to enhance productivity and to drive smarter innovation and economic growth. Based on the Hartree Centre’s experiences over the six years from 2016-2022, when I was its Director, here are a few of my observations and take-aways.
Democratising HPC and AI
One of the watchwords of the UK’s globally-renowned eScience programme 25 years ago was “ramps not steps”, referring to the necessity of enabling users and prospective users to move as seamlessly as possible from their current practices in data science to the adoption of new methods, technologies and tools, to facilitate uptake. This is an approach which we should be looking to replicate in democratising access to HPC and AI/ML.
For too long, large HPC systems have been very difficult to use. There has been the assumption that users will start with very small systems then migrate to using larger and more complex systems as their needs and experience grows, obviating the need to develop more user-friendly approaches to providing access to inexperienced users. This is not conducive to supporting organisations to realise the benefits of scaling up their technology usage more quickly. Rather, three priorities have been identified to open access from the start.
The first of these is the provision of training in relevant technologies, particularly for mid-career staff, who often have essential domain or industry expertise but little or no HPC or AI/ML expertise, to enable them to contribute to or lead inter-disciplinary teams. We need to achieve a better balance between technology push and market pull, when designing new applications and understanding infrastructure needs for the UK.
The second priority is the design of tools and applications – everything from “front-ends” (potentially cloud-like) to large systems to design of specific applications for new sectors. This is an area where much closer partnerships with the vendor and tool development community, including the UK’s rich ecosystem of start-ups will yield dividends and bring economic benefit to the UK.
The third priority is to provide local support to SMEs who would benefit from supported access to compute as end-users of novel technologies. Many SMEs have aspirations to make more effective use of compute but are both resource and time poor. Small demonstrator projects with expert support can help them to understand what expertise, infrastructure and tools they should put in place to reap benefits. This is particularly effective if the support is local to the SME to maximise the benefits of close collaboration, and can help them work though questions and concerns e.g. about using proprietary data on shared machines or about IP and patent considerations.
Forging ahead
The UK is currently one of the few major countries without a 10-year plan for large scale compute and with no globally competitive HPC or AI infrastructure having yet been announced. However, with the publication of the UK strategy imminent, the UK is poised to benefit from second-mover advantage. We can learn from and build on the experiences (both successes and failures) of our international competitors though engaging with them in collaborative and mutually beneficial partnerships. Successful democratisation will result in technology development by the UK for the wider benefit of the UK, utilising skills and expertise across society as a whole. The sector has made a promising start by coming together to engage in productive discussions about the next steps and we are well placed to respond to the demands of the next few years.
techUK – Supercharging UK Tech and Innovation
The opportunities of innovation are endless. Automation, IoT, AI, Edge, Quantum, Drones and High Performance Computing all have the power to transform the UK. techUK members lead the development of these technologies. Together we are working with Government and other stakeholders to address tech innovation priorities and build an innovation ecosystem that will benefit people, society, economy and the planet - and supercharge the UK as a global leader in tech and innovation.
For more information, or to get in touch, please visit our Innovation Hub and click ‘contact us’.
Get our tech and innovation insights straight to your inbox
Sign-up to get the latest updates and opportunities from our Technology and Innovation and AI programmes.
Keen to learn more? Get in touch:
Laura Foster
Laura is techUK’s Head of Programme for Technology and Innovation.
Rory Daniels
Rory joined techUK in June 2023 after three years in the Civil Service on its Fast Stream leadership development programme.
Elis Thomas
Elis joined techUK in December 2023 as a Programme Manager for Tech and Innovation, focusing on AI, Semiconductors and Digital ID.