A vision for UK-wide data sharing: Designing the National Data Library
In December 2024, the Wellcome Trust and the Economic and Social Research Council (ESRC) put out a call for technical visions for a National Data Library (NDL). The goal was to create the blueprint for a data-sharing architecture that would connect thousands of disconnected databases to improve vital research and public services across the UK.
In today's digital landscape, data supply chains have emerged as critical infrastructure for effective decision-making. Just as physical supply chains connect raw materials to finished products, data supply chains transform raw data into actionable insights that power services and research. Building end-to-end visibility across these data supply chains is crucial to eliminate information asymmetry, where some stakeholders possess more complete information than others. This asymmetry creates inefficiencies, reduces trust, and limits collaboration potential. By establishing transparent data flows with clear provenance, governance, and accessibility, organisations can ensure that all participants operate with consistent information, leading to more equitable outcomes and enhanced innovation. An effective NDL would address these challenges by creating standardised pathways for data to move securely between trusted parties while maintaining visibility throughout the entire journey.
Snowflake co-authored one of five successful white papers that shared a vision for the future of the UK’s public services data sharing. In this article, we look at some of the challenges in creating a distributed architecture for research and explore our recommended approach to creating an NDL, along with examples of other projects where we’ve helped organisations share data at scale.
Big ambitions require overcoming big challenges
The creation of an NDL would have a profound impact on the UK’s ability to deliver public services, but there is a paradox at the heart of this problem; any proposed solution needs to enable maximal data sharing while also ensuring complete control over who can access data.
To address this problem, the architecture behind an NDL must be based on an open standard for how data is stored, ensuring both data security and accessibility. This is vital for increasing public trust and reducing complexity. There must also be robust mechanisms for how data is shared, supported by strong governance to connect trusted parties to this data.
The most important consideration is that the data library should prioritise the ecosystem itself above any individual user or contributor. In doing so, it can be built in a way that benefits every party.
The right approach will move public services away from siloed data that provides a partial view of populations to a world of unified, on-demand data. Data that’s accessible to multiple providers — all with standardised workflows and approaches to citizen engagement.
The principles behind our architecture
By applying secure cloud collaboration, template-driven infrastructure architectures, and proven, scalable technologies, we proposed a blueprint for an economically viable ecosystem that shares valuable data applications and insights.
At the heart of this ecosystem are nine key principles. Think of them like a manifesto designed to increase public trust, reproducibility and sustainability, while simultaneously reducing complexity, costs and risk.
-
Services are customer-driven, not centrally determined.
-
We prefer analytical code to be open-source rather than reinvented.
-
We should minimise data duplication wherever possible.
-
Personal data should remain as close to the source as possible to provide security and ease of privacy management.
-
Code moves to the data rather than moving data.
-
Personal identifiers should be the last resort when linking datasets.
-
Simple access to data is essential, as friction and complexity reduce usage and value.
-
We trust domain experts to develop their own data assets.
-
Design must be iterative and responsive to changing needs.
Three approaches at the heart of our NDL
These nine principles are supported by three core approaches to data use that underpin our proposal.
-
Secure data sharing
Snowflake allows for secure data sharing, by ensuring that no actual data is copied or transferred. Instead, the organisation you wish to share data with can query and process the original data as if it’s within its own environment.
We enable this through a virtual data warehouse that separates where data is stored from who queries it. Through this approach, you can share data directly with multiple organisations while ensuring sensitive data never leaves your control. This is key to both ensuring compliance and building trust.
-
Listings
Part of the problem with large-scale data sharing is that not everyone knows what data exists to support research. Snowflake allows users to create a shared area that lists all available data. It provides full control of who has access to view these listings, and additional external sources can easily be discovered on the Snowflake Marketplace. As with secure data sharing, these listings allow you to bring the organisations you partner with to your data, rather than the other way round, so you can maintain complete integrity and control.
-
Data clean rooms
For the most sensitive data, that categorically cannot be viewed outside of an organisation, data clean rooms provide an environment for connecting and linking data sets to show aggregated results, without allowing individuals to view the underlying data. To ensure data privacy, clean rooms deploy differential privacy technology that adds “noise” to those results so you can run computation directly on top of encrypted data.
Our experience with data mesh architectures
Our proposal for the NDL incorporates a ‘data mesh’ or ‘data fabric’ architecture; an approach to enhancing self-service data use which many leading compute, storage, and security platforms are now supporting.
Our vision is based on the LATTICE architecture developed at the Francis Crick Institute. ‘The Crick’, as it’s colloquially known, is Europe’s largest discovery life sciences research institute and uses Snowflake’s AI Data Cloud to power its exploratory research.
A data mesh architecture enables the perfect balance between open sharing and consistent security, and our past experience shows us this works in practice as well as in theory.
Most recently, working with Roche Diagnostics, the Swiss healthcare multinational, we helped to create a dynamic, decentralised data mesh architecture that enables Roche to make more efficient use of data, reduce costs, and ensure governance.
Using a combination of Snowflake and Immuta, Roche eliminated a significant role-based access burden and reduced required access groups by 94%. It has also increased the production of data models by up to 20% using Snowpark, and accelerated time to action by up to 300%.
A dedication to collaboration and data sharing
Roche isn’t the only example of our recent work in this field. We have vast experience bringing data sharing ambitions to life, and regularly collaborate with other organisations to simplify data sharing and uncover new opportunities.
In 2024, we collaborated with the California Department of Technology and California State Water Resources Control Board to implement the data connectors that support The California Open Data Portal. This portal makes data publicly available in a safe and secure way.
We’ve also helped the Met Office enhance its weather data delivery, which improves decision-making across industries like healthcare and aviation. The Met Office uses Snowflake Marketplace to make data more accessible, supporting critical offerings like emergency services and public transit. The accessibility of Snowflake Marketplace means a larger userbase can access the Met Office’s data, helping it reach new sectors and provide more detail to its data users.
To learn more about our vision for an NDL, read our full paper, along with those from the other five organisations who took part.
techUK - Seizing the AI Opportunity
The UK is a global leader in AI innovation, development and adoption.
The economic growth and productivity gain that AI can unlock is vast, but to fully harness this transformative opportunity, immediate action is required. Our aim is to ensure the UK seizes the opportunities presented by AI technology and continues to be a world leader in AI development.
Get involved: techUK runs a busy calendar of activities including events, reports, and insights to demonstrate some of the most significant AI opportunities for the UK. Our AI Hub is where you will find details of all upcoming activity. We also send a monthly AI newsletter which you can subscribe to here.
Upcoming AI events
Latest news and insights
Subscribe to our AI newsletter
AI updates
Sign-up to our monthly newsletter to get the latest updates and opportunities from our AI and Data Analytics Programme straight to your inbox.
Contact the team
Usman Ikhlaq
Usman Ikhlaq
Usman joined techUK in January 2024 as Programme Manager for Artificial Intelligence.
He leads techUK’s AI Adoption programme, supporting members of all sizes and sectors in adopting AI at scale. His work involves identifying barriers to adoption, exploring solutions, and helping to unlock AI’s transformative potential, particularly its benefits for people, the economy, society, and the planet. He is also committed to advancing the UK’s AI sector and ensuring the UK remains a global leader in AI by working closely with techUK members, the UK Government, regulators, and devolved and local authorities.
Since joining techUK, Usman has delivered a regular drumbeat of activity to engage members and advance techUK's AI programme. This has included two campaign weeks, the creation of the AI Adoption Hub (now the AI Hub), the AI Leader's Event Series, the Putting AI into Action webinar series and the Industrial AI sprint campaign.
Before joining techUK, Usman worked as a policy, regulatory and government/public affairs professional in the advertising sector. He has also worked in sales, marketing, and FinTech.
Usman holds an MSc from the London School of Economics and Political Science (LSE), a GDL and LLB from BPP Law School, and a BA from Queen Mary University of London.
When he isn’t working, Usman enjoys spending time with his family and friends. He also has a keen interest in running, reading and travelling.
Sue Daley OBE
Sue Daley OBE
Sue leads techUK's Technology and Innovation work.
This includes work programmes on cloud, data protection, data analytics, AI, digital ethics, Digital Identity and Internet of Things as well as emerging and transformative technologies and innovation policy.
In 2025, Sue was honoured with an Order of the British Empire (OBE) for services to the Technology Industry in the New Year Honours List.
She has been recognised as one of the most influential people in UK tech by Computer Weekly's UKtech50 Longlist and in 2021 was inducted into the Computer Weekly Most Influential Women in UK Tech Hall of Fame.
A key influencer in driving forward the data agenda in the UK, Sue was co-chair of the UK government's National Data Strategy Forum until July 2024. As well as being recognised in the UK's Big Data 100 and the Global Top 100 Data Visionaries for 2020 Sue has also been shortlisted for the Milton Keynes Women Leaders Awards and was a judge for the Loebner Prize in AI. In addition to being a regular industry speaker on issues including AI ethics, data protection and cyber security, Sue was recently a judge for the UK Tech 50 and is a regular judge of the annual UK Cloud Awards.
Prior to joining techUK in January 2015 Sue was responsible for Symantec's Government Relations in the UK and Ireland. She has spoken at events including the UK-China Internet Forum in Beijing, UN IGF and European RSA on issues ranging from data usage and privacy, cloud computing and online child safety. Before joining Symantec, Sue was senior policy advisor at the Confederation of British Industry (CBI). Sue has an BA degree on History and American Studies from Leeds University and a Masters Degree on International Relations and Diplomacy from the University of Birmingham. Sue is a keen sportswoman and in 2016 achieved a lifelong ambition to swim the English Channel.
- Email:
- [email protected]
- Phone:
- 020 7331 2055
- Twitter:
- @ChannelSwimSue,@ChannelSwimSue
Tess Buckley
Tess Buckley
A digital ethicist and musician, Tess holds a MA in AI and Philosophy, specialising in ableism in biotechnologies. Their professional journey includes working as an AI Ethics Analyst with a dataset on corporate digital responsibility, followed by supporting the development of a specialised model for sustainability disclosure requests. Currently at techUK as programme manager in digital ethics and AI safety, Tess focuses on demystifying and operationalising ethics through assurance mechanisms and standards. Their primary research interests encompass AI music systems, AI fluency, and technology created by and for differently abled individuals. Their overarching goal is to apply philosophical principles to make emerging technologies both explainable and ethical.
Outside of work Tess enjoys kickboxing, ballet, crochet and jazz music.
- Email:
- [email protected]
- Website:
- tessbuckley.me
- LinkedIn:
- https://www.linkedin.com/in/tesssbuckley/
Laura Foster
Laura Foster
Laura is techUK’s Associate Director for Technology and Innovation.
She supports the application and expansion of emerging technologies, including Quantum Computing, High-Performance Computing, AR/VR/XR and Edge technologies, across the UK. As part of this, she works alongside techUK members and UK Government to champion long-term and sustainable innovation policy that will ensure the UK is a pioneer in science and technology
Before joining techUK, Laura worked internationally as a conference researcher and producer covering enterprise adoption of emerging technologies. This included being part of the strategic team at London Tech Week.
Laura has a degree in History (BA Hons) from Durham University, focussing on regional social history. Outside of work she loves reading, travelling and supporting rugby team St. Helens, where she is from.
- Email:
- [email protected]
- LinkedIn:
- www.linkedin.com/in/lauraalicefoster
Nimmi Patel
Nimmi Patel
Nimmi Patel is the Head of Skills, Talent and Diversity at techUK. She works on all things skills, education, and future of work policy, focusing on upskilling and retraining. Nimmi is also an Advisory Board member of Digital Futures at Work Research Centre (digit). The Centre research aims to increase understanding of how digital technologies are changing work and the implications for employers, workers, job seekers and governments.
Prior to joining the techUK team, she worked for the UK Labour Party and New Zealand Labour Party, and holds an MA in Strategic Communications at King’s College London and BA in Politics, Philosophy and Economics from the University of Manchester. She is currently taking part in the 2024-25 University of Bath Institute for Policy Research Policy Fellowship Programme.
- Email:
- [email protected]
- Phone:
- 07805744520
- Twitter:
- @nimmiptl
- Website:
- www.techuk.org
- LinkedIn:
- https://www.linkedin.com/in/nimmi-patel1/
Audre Verseckaite
Audre Verseckaite
Audre joined techUK in July 2023 as a Policy Manager for Data. Previously, she was a Policy Advisor in the Civil Service, where she worked on the Digital Markets, Competition and Consumers Bill at the Department for Science, Innovation and Technology, and at HM Treasury on designing COVID-19 support schemes and delivering the Financial Services and Markets Bill. Before that, Audre worked at a public relations consultancy, advising public and private sector clients on their communications, public relations, and government affairs strategy.
Prior to this, Audre completed an MSc in Public Policy at the Korea Development Institute and a Bachelor's in International Relations and History from SOAS, University of London. Outside of work, she enjoys spending time outdoors, learning about new cultures through travel and food, and going on adventures.
Edward Emerson
Edward Emerson
Edward leads the Digital Economy programme at techUK, which includes our work on online safety, fraud, and regulation for growth initiatives.
He has prior experience working for the Department for Digital, Culture, Media and Sport and has previously worked for a number of public affairs consultancies specialising in research and strategy, working with leading clients in the technology and financial services sectors.
- Email:
- [email protected]
- LinkedIn:
- https://www.linkedin.com/in/edward-emerson-009189183
Heather Cover-Kus
Heather Cover-Kus
Heather is Head of Central Government Programme at techUK, working to represent the supplier community of tech products and services to Central Government.
Prior to joining techUK in April 2022, Heather worked in the Economic Policy and Small States Section at the Commonwealth Secretariat. She led the organisation’s FinTech programme and worked to create an enabling environment for developing countries to take advantage of the socio-economic benefits of FinTech.
Before moving to the UK, Heather worked at the Office of the Prime Minister of The Bahamas and the Central Bank of The Bahamas.
Heather holds a Graduate Diploma in Law from BPP, a Masters in Public Administration (MPA) from LSE, and a BA in Economics and Sociology from Macalester College.
- Email:
- [email protected]
- LinkedIn:
- https://www.linkedin.com/in/heather-cover-kus-ba636538
Visit our AI Hub - the home of all our AI content:
Authors
Fawad Qureshi
Fawad is a strategic technology leader with more than two decades of international industry project experience involving consulting on enterprise data warehouses, big data analytics, and cloud solution architectures. He has worked across all stages of the technology life cycle starting from engineering, professional services, pre-sales, and business development. In his current role he works as Global Industry Field CTO, helping customers in multiple industries across the globe. He is passionate about sustainability and helps his clients in achieving their sustainability goals through the use of data and analytics.