Overhauling technology by changing working practices

At CACI IIG, we are constantly pushing ourselves to improve working practices and facilitate collaboration to achieve significant operational improvements and commercial benefits for customers. Largescale IT projects often present us with several technological and cultural hurdles to reach success.

The secret to this is remove barriers and work together more closely than ever before.

We have worked in partnership with Met Office for over four years to develop this capability. Our strong relationship resulted in being shortlisted for the Most Successful Cultural Transformation at the DevOps Industry Awards for our work on Space Weather (SWX). This blog will explore the challenges and successes of the project.

The Challenges:

• Technology stack - Models had complex dependencies and on-premise environment hampers techniques.

• Data requirements - Often using non-operational datasets. Several input and output formats to support.

• Compute requirements - Execution time varies between seconds and hours.

• Toolsets - Disparity caused ambiguity and friction

• Culture - The teams were close and familiar but siloed

• Resourcing - Fully utilised teams need quick return on investment.

• Skillset Inconsistency - Use of different tools, techniques and languages.

The Target

With the weaknesses identified, we created the following aims:

• Reduce elapsed time to production

• Increase value of models - identifying additional opportunities.

• Improve the confidence in outputs.

• Improve autonomy of releases

 

… and focused on three core objectives to meet these aims:

• Improve the cross-visibility of priorities and dependencies.

• Breakdown silos with a ‘one-team’ ethos.

• Remove complexity from integration activities.

 

The Approach

The teams took a multi-staged approach.

Tightening the feedback loop

Inefficiencies in communication were a major source of friction, causing delays and rework.

•            Separate backlogs - priorities were contentious. 

•            Lack of understanding between teams - key information surfaced late. 

•            Difficulty sharing code - models were treated as black boxes.

Focuses on cultural and process changes resolved these issues.  Scientists started to attend stand-ups; defined a joint “Definition of Ready” to ensure that everyone is aware of vital information blocking progress, and wireframing activities were performed much earlier in the development cycle. The teams could do more in parallel while ensuring models delivered.

Toolset alignment

The next stage looked at reducing the number of handovers, many of these were caused by opacity of each team’s toolsets.

• Two project backlogs caused duplication 

• Disparate source control mechanisms - changes to the model were not always shared with the development team quickly nor did they feel empowered to make changes 

• Documentation was not consistent

They moved to a shared project board, a standardised source control system and a set of rules as to what documentation belonged where, with a strategy for cross-referencing.

The result was a process with fewer handovers, eliminating re-planning and re-working, and streamlining the development process.  Curating a greater understanding of how the work could be split into smaller parts, allowing for a tighter iterative approach.

Shared build pipelines

Despite the process improvements made so far, there were still invalid assumptions, unexpected complexity, and last-minute scope creep. Technology needed to step in.

Empowered by Met Office’s adoption of AWS as a platform, the development team looked to move away from the traditional on-premises architecture for executing model code. Using AWS native technologies simplified the architecture in terms of complexity and supportability.  Model inputs and outputs stored and archived made it easier to share with the wider team, using them to monitor the scientific accuracy of the model over time.

Resulting in an integrated build and test pipeline that enables continuous integration and deployment of the model and operational wrap.

Success Outcomes

Thanks to the effort of CACI and Met Office teams, the silos of the scientists and technologists have been broken down, leading to significant efficiency improvements:

  • Model time turnaround
    • 5x reduction in time taken for a model to be fully deployed
  • Availability and reliability
    • Reduction in production issues, due to consistent infrastructure and continuous quality assurance
  • Satisfaction and morale
    • Scientists and technologists can work closer
    • Updated infrastructures reduced challenges
  • Satisfaction of senior stakeholders
    • SWX  is used as an exemplar case study for breaking down silos

Summary

The Met Office strategy is to focus everything they do on delivering greater benefit and impact to users, and to achieve this through exceptional science, technology and operations. 

In partnership with CACI, Met Office’s SWX programme has comprehensively demonstrated that collaborating in a joined-up way can bring benefits including better visibility; exploitation of advances in science; and technology optimisation.  As a result, more value is delivered to the end users, faster.

To read more from techUK's Data Analytics week, visit our landing page by visiting here!

  • Katherine Mayes

    Katherine Mayes

    Programme Manager | Cloud, Data, Analytics and AI
    T 020 7331 2019

Share this

FROM SOCIAL MEDIA

Our first virtual #cyberinnovation den has come to a close. A big thank you to our sponsors @RaytheonIntel &… https://t.co/6Pd46quphz
techUK are delighted to announce that @surevine have won our second annual Cyber Exchange Pitching Competition as p… https://t.co/1rPBMVEZLC
We're now hearing from the winner of last year's #cyberinnovation SME pitching competition. Haydn Brooks of… https://t.co/Nra0LQ2jul
Our final pitch of the day comes from Jeremy Martin of 27k1 who is pitching their 270001 compliance solution. Thank… https://t.co/dIUiZR2Y4f
Our penultimate pitch of the day comes from @Zerto which aims to recover business applications with minimal data loss. #CyberInnovation
Guy Lloyd, Commercial Director at CySure argues that CySure's VOSO can help SMEs make sense of cyber security. #CyberInnovation
.@AngokaCyber headed by Yuri Andersson, are now pitching their secure solution for network to network communications. #CyberInnovation
Fabian Eberle from @KeylessTech makes the case for their Zero-Trust passwordless authentication system which is now… https://t.co/IbftPd6dZ0
.@Ckizilari from @MortenIct_UK makes the case for their MFA system. #CyberInnovation
Become a Member
×

Become a techUK Member

By becoming a techUK member we will help you grow through:

Click here to learn more...