Intelligent Automation to ease the Data Cleansing burden
In common with much of the public sector, the technology estate across the Ministry of Defence (MOD) is very broad, with a wide range of legacy systems either MOD-managed or supplier-managed. The various data sets held in these systems are accessed, updated and exploited by multiple teams across MOD and supplier organisations. This inevitably leads to the need for Data Cleansing activities to ensure that the quality of each data set is as high as possible. These activities are usually very labour intensive and time-consuming.
There is significant potential to use Intelligent Automation to assist with this Data Cleansing effort. Using a combination of Machine Learning and Robotic Process Automation (RPA) we can identify poor quality data, make recommendations for the correction and action those corrections on the source system.
Focusing on specific high-impact data quality issues where the criteria for what constitutes good quality data are clear, step one is to create an application for the user to triage each issue identified. This triage (and any record of historic data corrections) builds an evidence set against which a recommendation engine can be trained. Step two is for the application to start suggesting to the user the most appropriate action for each case. The user then either agrees or adjusts the action, providing the necessary feedback loop to improve the accuracy of the recommendations. As confidence builds, the time spent manually triaging each case reduces, allowing the user, as the subject matter expert, to focus on the more complex cases.
While the recommendation engine represents one aspect of Intelligent Automation, and can save time and manual effort, it does not tackle the manual action to correct the data in the source system. Ideally the corrective actions needed would be automated through APIs but given the nature of disparate legacy systems, API access is often not available and not feasible to introduce. In this case RPA can be employed to make the changes, accepting an input from the data cleansing application of the actions required and executing those actions on the source system.
There are several challenges in achieving this, such as identification of suitable subject areas, establishing a training and test data set for the machine learning algorithms, and agreeing access protocols for the RPA agent to the source systems. Additionally, while IA can help to ease the manual burden and improve data quality, it does this in collaboration with subject matter experts not instead of them. However, in specific cases, confidence in the automated recommended actions may be high enough to allow the application to run autonomously.
Will Intelligent Automation solve our all data quality issues across a fragmented and often aged IT estate? Of course not, but it can ease the manual burden, accelerate improvements and allow the experts to focus on the high-value high-knowledge activities.
Laura is techUK’s Programme Manager for Technology and Innovation.
She supports the application and expansion of emerging technologies across business, including Geospatial Data, Quantum Computing, AR/VR/XR and Edge technologies.
Before joining techUK, Laura worked internationally in London, Singapore and across the United States as a conference researcher and producer covering enterprise adoption of emerging technologies. This included being part of the strategic team at London Tech Week.
Laura has a degree in History (BA Hons) from Durham University, focussing on regional social history. Outside of work she loves reading, travelling and supporting rugby team St. Helens, where she is from.
Sue leads techUK's Technology and Innovation work.
This includes work programmes on cloud, data protection, data analytics, AI, Digital Identity and Internet of Things as well as emerging and transformative technologies and innovation policy. She has been recognised as one of the most influential women in UK tech by Computer Weekly and as a key influencer in driving forward the Big Data agenda in the UK Big Data 100. Sue has also been shortlisted for the Milton Keynes Women Leaders Awards and was a judge for the Loebner Prize in AI. In addition to being a regular industry speaker on issues including AI ethics, data protection and cyber security, Sue was recently a judge for the UK Tech 50 and is a regular judge of the annual UK Cloud Awards.
Prior to joining techUK in January 2015 Sue was responsible for Symantec's Government Relations in the UK and Ireland. She has spoken at events including the UK-China Internet Forum in Beijing, UN IGF and European RSA on issues ranging from data usage and privacy, cloud computing and online child safety. Before joining Symantec, Sue was senior policy advisor at the Confederation of British Industry (CBI). Sue has an BA degree on History and American Studies from Leeds University and a Masters Degree on International Relations and Diplomacy from the University of Birmingham. Sue is a keen sportswoman and in 2016 achieved a lifelong ambition to swim the English Channel.
Zoe is a Programme Assistant, supporting techUK's work across Policy, Technology and Innovation.
The team makes the tech case to government and policymakers in Westminster, Whitehall, Brussels and across the UK on the most pressing issues affecting this sector and supports the Technology and Innovation team in the application and expansion of emerging technologies across business, including Geospatial Data, Quantum Computing, AR/VR/XR and Edge technologies.
Before joining techUK, Zoe worked as a Business Development and Membership Coordinator at London First and prior to that Zoe worked in Partnerships at a number of Forex and CFD brokerage firms including Think Markets, ETX Capital and Central Markets.
Zoe has a degree (BA Hons) from the University of Westminster and in her spare time, Zoe enjoys travelling, painting, keeping fit and socialising with friends.