Intelligent Automation to ease the Data Cleansing burden

Techmodal 

In common with much of the public sector, the technology estate across the Ministry of Defence (MOD) is very broad, with a wide range of legacy systems either MOD-managed or supplier-managed.  The various data sets held in these systems are accessed, updated and exploited by multiple teams across MOD and supplier organisations.  This inevitably leads to the need for Data Cleansing activities to ensure that the quality of each data set is as high as possible.  These activities are usually very labour intensive and time-consuming.

There is significant potential to use Intelligent Automation to assist with this Data Cleansing effort.  Using a combination of Machine Learning and Robotic Process Automation (RPA) we can identify poor quality data, make recommendations for the correction and action those corrections on the source system.

Focusing on specific high-impact data quality issues where the criteria for what constitutes good quality data are clear, step one is to create an application for the user to triage each issue identified.  This triage (and any record of historic data corrections) builds an evidence set against which a recommendation engine can be trained.  Step two is for the application to start suggesting to the user the most appropriate action for each case.  The user then either agrees or adjusts the action, providing the necessary feedback loop to improve the accuracy of the recommendations.  As confidence builds, the time spent manually triaging each case reduces, allowing the user, as the subject matter expert, to focus on the more complex cases.

While the recommendation engine represents one aspect of Intelligent Automation, and can save time and manual effort, it does not tackle the manual action to correct the data in the source system.  Ideally the corrective actions needed would be automated through APIs but given the nature of disparate legacy systems, API access is often not available and not feasible to introduce.  In this case RPA can be employed to make the changes, accepting an input from the data cleansing application of the actions required and executing those actions on the source system.

There are several challenges in achieving this, such as identification of suitable subject areas, establishing a training and test data set for the machine learning algorithms, and agreeing access protocols for the RPA agent to the source systems.  Additionally, while IA can help to ease the manual burden and improve data quality, it does this in collaboration with subject matter experts not instead of them.  However, in specific cases, confidence in the automated recommended actions may be high enough to allow the application to run autonomously.

Will Intelligent Automation solve our all data quality issues across a fragmented and often aged IT estate?  Of course not, but it can ease the manual burden, accelerate improvements and allow the experts to focus on the high-value high-knowledge activities.

You can read all insights from techUK's Intelligent Automation Week here

Laura Foster

Head of Technology and Innovation, techUK

Laura is techUK’s Head of Programme for Technology and Innovation.

She supports the application and expansion of emerging technologies, including Quantum Computing, High-Performance Computing, AR/VR/XR and Edge technologies, across the UK. As part of this, she works alongside techUK members and UK Government to champion long-term and sustainable innovation policy that will ensure the UK is a pioneer in science and technology

Before joining techUK, Laura worked internationally as a conference researcher and producer covering enterprise adoption of emerging technologies. This included being part of the strategic team at London Tech Week.

Laura has a degree in History (BA Hons) from Durham University, focussing on regional social history. Outside of work she loves reading, travelling and supporting rugby team St. Helens, where she is from.

Email:

[email protected]

LinkedIn:

www.linkedin.com/in/lauraalicefoster

Read lessmore

Chris Hazell

Programme Manager - Cloud, Tech and Innovation, techUK

Chris is the Programme Manager for Cloud, Tech and Innovation

Email:

[email protected]

Read lessmore

Sue Daley

Director, Technology and Innovation

Sue leads techUK's Technology and Innovation work.

This includes work programmes on cloud, data protection, data analytics, AI, digital ethics, Digital Identity and Internet of Things as well as emerging and transformative technologies and innovation policy. She has been recognised as one of the most influential people in UK tech by Computer Weekly's UKtech50 Longlist and in 2021 was inducted into the Computer Weekly Most Influential Women in UK Tech Hall of Fame. A key influencer in driving forward the data agenda in the UK Sue is co-chair of the UK government's National Data Strategy Forum. As well as being recognised in the UK's Big Data 100 and the Global Top 100 Data Visionaries for 2020 Sue has also been shortlisted for the Milton Keynes Women Leaders Awards and was a judge for the Loebner Prize in AI. In addition to being a regular industry speaker on issues including AI ethics, data protection and cyber security, Sue was recently a judge for the UK Tech 50 and is a regular judge of the annual UK Cloud Awards.

Prior to joining techUK in January 2015 Sue was responsible for Symantec's Government Relations in the UK and Ireland. She has spoken at events including the UK-China Internet Forum in Beijing, UN IGF and European RSA on issues ranging from data usage and privacy, cloud computing and online child safety. Before joining Symantec, Sue was senior policy advisor at the Confederation of British Industry (CBI). Sue has an BA degree on History and American Studies from Leeds University and a Masters Degree on International Relations and Diplomacy from the University of Birmingham. Sue is a keen sportswoman and in 2016 achieved a lifelong ambition to swim the English Channel.

Email:

[email protected]

Phone:

020 7331 2055

Twitter:

@ChannelSwimSue,@ChannelSwimSue

Read lessmore