29 Apr 2021

Intelligent Automation to ease the Data Cleansing burden

Intelligent Automation can ease the manual burden, accelerate improvements and allow the experts to focus on the high-value high-knowledge activities writes Ewen Malloch, Head of Technology at Techmodal for techUK's Intelligent Automation Week #AutomationUK

Techmodal 

twitter graphic.jpg 5

In common with much of the public sector, the technology estate across the Ministry of Defence (MOD) is very broad, with a wide range of legacy systems either MOD-managed or supplier-managed.  The various data sets held in these systems are accessed, updated and exploited by multiple teams across MOD and supplier organisations.  This inevitably leads to the need for Data Cleansing activities to ensure that the quality of each data set is as high as possible.  These activities are usually very labour intensive and time-consuming.

There is significant potential to use Intelligent Automation to assist with this Data Cleansing effort.  Using a combination of Machine Learning and Robotic Process Automation (RPA) we can identify poor quality data, make recommendations for the correction and action those corrections on the source system.

Focusing on specific high-impact data quality issues where the criteria for what constitutes good quality data are clear, step one is to create an application for the user to triage each issue identified.  This triage (and any record of historic data corrections) builds an evidence set against which a recommendation engine can be trained.  Step two is for the application to start suggesting to the user the most appropriate action for each case.  The user then either agrees or adjusts the action, providing the necessary feedback loop to improve the accuracy of the recommendations.  As confidence builds, the time spent manually triaging each case reduces, allowing the user, as the subject matter expert, to focus on the more complex cases.

While the recommendation engine represents one aspect of Intelligent Automation, and can save time and manual effort, it does not tackle the manual action to correct the data in the source system.  Ideally the corrective actions needed would be automated through APIs but given the nature of disparate legacy systems, API access is often not available and not feasible to introduce.  In this case RPA can be employed to make the changes, accepting an input from the data cleansing application of the actions required and executing those actions on the source system.

There are several challenges in achieving this, such as identification of suitable subject areas, establishing a training and test data set for the machine learning algorithms, and agreeing access protocols for the RPA agent to the source systems.  Additionally, while IA can help to ease the manual burden and improve data quality, it does this in collaboration with subject matter experts not instead of them.  However, in specific cases, confidence in the automated recommended actions may be high enough to allow the application to run autonomously.

Will Intelligent Automation solve our all data quality issues across a fragmented and often aged IT estate?  Of course not, but it can ease the manual burden, accelerate improvements and allow the experts to focus on the high-value high-knowledge activities.

You can read all insights from techUK's Intelligent Automation Week here

Rory Daniels

Rory Daniels

Programme Manager, Emerging Technologies

Rory joined techUK in June 2023 after three years in the Civil Service on its Fast Stream leadership development programme.

During this time, Rory worked on the Government's response to Covid-19 (NHS Test & Trace), school funding strategy (Department for Education) and international climate and nature policy (Cabinet Office). He also tackled the social care crisis whilst on secondment to techUK's Health and Social Care programme in 2022.

Before this, Rory worked in the House of Commons and House of Lords alongside completing degrees in Political Economy and Global Politics.

Today, he is techUK's Programme Manager for Emerging Technologies, covering dozens of technologies including metaverse, drones, future materials, robotics, blockchain, space technologies, nanotechnology, gaming tech and Web3.0.

Email:
[email protected]
LinkedIn:
https://www.linkedin.com/in/rorydaniels28/

Read lessmore

Laura Foster

Laura Foster

Head of Technology and Innovation, techUK

Laura is techUK’s Head of Programme for Technology and Innovation.

She supports the application and expansion of emerging technologies, including Quantum Computing, High-Performance Computing, AR/VR/XR and Edge technologies, across the UK. As part of this, she works alongside techUK members and UK Government to champion long-term and sustainable innovation policy that will ensure the UK is a pioneer in science and technology

Before joining techUK, Laura worked internationally as a conference researcher and producer covering enterprise adoption of emerging technologies. This included being part of the strategic team at London Tech Week.

Laura has a degree in History (BA Hons) from Durham University, focussing on regional social history. Outside of work she loves reading, travelling and supporting rugby team St. Helens, where she is from.

Email:
[email protected]
LinkedIn:
www.linkedin.com/in/lauraalicefoster

Read lessmore

 

Related topics