Data analysis in a cyber security platform

  • techUK techUK
    Thursday28May 2020
    Opinions

    Guest Blog: Ryan Thornley, Senior Engineer, explores how CACI ensures that the UK is more resilient to cyber-attacks by bundling and analysing data for an automated...

CACI are working with HMG on a prototype cyber security platform to improve capability in the area of automated network defence. The platform ingests messages from disparate sources, including mainstream cyber security sensors and several bespoke sensors that are unique to our customer, and analyse the resulting data so a corresponding action can be automated.

Before you can respond to a cyber threat you must first detect that threat in real time, with attackers aiming to subvert detection by any means necessary, sometimes ‘real time’ can mean data spread across days, months or years, depending on the value of the target.

Our ingest pipeline was designed with these factors in mind, developing a scalable solution using message queues and data stored in elastic search which allows us to receive a large amount of fine grained data from sensors. The data received into the system is characteristically small and noisy. The natural background noise on these networks can make it difficult to decipher a legitimate cyber-attack and sensors often only report small snippets of information. The first step is to normalise the data, extracting common features from the sensors data such as devices involved, files, URLs, timings and severity.

Once all data is normalised into a common model, we seek to understand more about it by passing it through an enrichment process. In an ideal world you would perform a high level of enrichment on every message, but this is computationally expensive – especially if it requires a 3rd party service such as a DNS lookup. We aim to perform a basic level of enrichment on every message, for example, by using internal databases we can geocode external IPs to their country. An asset library of all known devices within the system is a valuable resource, adding in device information, physical location and operational status. We can optionally use 3rd party enrichment or even query back into the network using tools like Osquery on an on-demand basis. This allows us to make decisions on how to enhance the dataset while balancing the load on the system and network.

Once the ingest process has finished enrichment, we have a large pool of data to analyse. To reduce the burden on the Cyber Analyst we make use of ML techniques. Using Recommendation Engines, we can look at the previous actions performed by the user for similar messages and make a suggestion on the correct response. If the confidence of the recommendation is high, then a response can be automated by the system, for example blocking an attacker’s access to the network.

Our UI is a key part of the application and allows for Cyber Analysts to browse the data within the system. They follow threads through the data, spotting patterns that could indicate the presence of an attacker. It’s important the UI can enable the user’s workflow of pivoting on the data and following these threads. We provide tools to group the data together to provide context. The system has built-in tasks to replicate these groups and present it to the user, should a similar pattern of events happen in the future. The sequence of events can be as important as the body of the events themselves, and our system’s UI accounts for this by enabling a timeline view.

Ultimately the automated data analysis and ML applied in this project means that the caseload of the Cyber Analyst is reduced, responses to cyber threats can be made at all hours of the day with significant levels of trust and can result in the fast, automatic removal of a malicious entity from an unmonitored network.

To read more from techUK's Data Analytics week, visit our landing page by visiting here!

  • Katherine Mayes

    Katherine Mayes

    Programme Manager | Cloud, Data, Analytics and AI
    T 020 7331 2019

Share this

FROM SOCIAL MEDIA

A thank you letter from @NHSX and @tara_donnelly1 to all the industry partners that have helped sustain and protect… https://t.co/1eoIZWCHaU
techUK is delighted to share an early insider's glimpse of @educationgovuk ambitious new programme to provide lapto… https://t.co/zrZezoVKUw
.@MicrosoftUK has committed to help 25 million people who have lost their job due to #COVID19 to reskill for roles… https://t.co/OMbuxDTGcS
ICYMI @MeVitae discussed how to ensure the hiring process is free of prejudice & latent bias with new technologies… https://t.co/PkINRjxvBK
Fewer frameworks and more ministerial muscle – how government can make good on its SME agenda - techUK's Head of Pu… https://t.co/YwATVtAIGT
We have an exciting line-up of speakers who will be sharing their insights on customer innovation and Open Banking… https://t.co/IUpE3DKMCd
Last chance to register for our Monday event on how AI can help us adapt to climate change. Awesome panel with our… https://t.co/gmPF7VyvE9
Become a Member
×

Become a techUK Member

By becoming a techUK member we will help you grow through:

Click here to learn more...