At CACI IIG, we are constantly pushing ourselves to improve working practices and facilitate collaboration to achieve significant operational improvements and commercial benefits for customers. Largescale IT projects often present us with several technological and cultural hurdles to reach success.
The secret to this is remove barriers and work together more closely than ever before.
We have worked in partnership with Met Office for over four years to develop this capability. Our strong relationship resulted in being shortlisted for the Most Successful Cultural Transformation at the DevOps Industry Awards for our work on Space Weather (SWX). This blog will explore the challenges and successes of the project.
• Technology stack - Models had complex dependencies and on-premise environment hampers techniques.
• Data requirements - Often using non-operational datasets. Several input and output formats to support.
• Compute requirements - Execution time varies between seconds and hours.
• Toolsets - Disparity caused ambiguity and friction
• Culture - The teams were close and familiar but siloed
• Resourcing - Fully utilised teams need quick return on investment.
• Skillset Inconsistency - Use of different tools, techniques and languages.
With the weaknesses identified, we created the following aims:
• Reduce elapsed time to production
• Increase value of models - identifying additional opportunities.
• Improve the confidence in outputs.
• Improve autonomy of releases
… and focused on three core objectives to meet these aims:
• Improve the cross-visibility of priorities and dependencies.
• Breakdown silos with a ‘one-team’ ethos.
• Remove complexity from integration activities.
The teams took a multi-staged approach.
Tightening the feedback loop
Inefficiencies in communication were a major source of friction, causing delays and rework.
• Separate backlogs - priorities were contentious.
• Lack of understanding between teams - key information surfaced late.
• Difficulty sharing code - models were treated as black boxes.
Focuses on cultural and process changes resolved these issues. Scientists started to attend stand-ups; defined a joint “Definition of Ready” to ensure that everyone is aware of vital information blocking progress, and wireframing activities were performed much earlier in the development cycle. The teams could do more in parallel while ensuring models delivered.
The next stage looked at reducing the number of handovers, many of these were caused by opacity of each team’s toolsets.
• Two project backlogs caused duplication
• Disparate source control mechanisms - changes to the model were not always shared with the development team quickly nor did they feel empowered to make changes
• Documentation was not consistent
They moved to a shared project board, a standardised source control system and a set of rules as to what documentation belonged where, with a strategy for cross-referencing.
The result was a process with fewer handovers, eliminating re-planning and re-working, and streamlining the development process. Curating a greater understanding of how the work could be split into smaller parts, allowing for a tighter iterative approach.
Shared build pipelines
Despite the process improvements made so far, there were still invalid assumptions, unexpected complexity, and last-minute scope creep. Technology needed to step in.
Empowered by Met Office’s adoption of AWS as a platform, the development team looked to move away from the traditional on-premises architecture for executing model code. Using AWS native technologies simplified the architecture in terms of complexity and supportability. Model inputs and outputs stored and archived made it easier to share with the wider team, using them to monitor the scientific accuracy of the model over time.
Resulting in an integrated build and test pipeline that enables continuous integration and deployment of the model and operational wrap.
Thanks to the effort of CACI and Met Office teams, the silos of the scientists and technologists have been broken down, leading to significant efficiency improvements:
- Model time turnaround
- 5x reduction in time taken for a model to be fully deployed
- Availability and reliability
- Reduction in production issues, due to consistent infrastructure and continuous quality assurance
- Satisfaction and morale
- Scientists and technologists can work closer
- Updated infrastructures reduced challenges
- Satisfaction of senior stakeholders
- SWX is used as an exemplar case study for breaking down silos
The Met Office strategy is to focus everything they do on delivering greater benefit and impact to users, and to achieve this through exceptional science, technology and operations.
In partnership with CACI, Met Office’s SWX programme has comprehensively demonstrated that collaborating in a joined-up way can bring benefits including better visibility; exploitation of advances in science; and technology optimisation. As a result, more value is delivered to the end users, faster.