07 May 2021

The other AI Skills Gap

Guest blog: Rui Silva, Principal Engineer at Critical Software as part of techUK's AI Week #AIWeek2021

AI has become an integral part of our lives. Companies from all sectors have embraced the trend of evermore intelligent systems to enable them to better understand and serve their customers.

From its early days being used in business applications as a powerful tool for running projections, AI has now evolved into something much more that can significantly expand human and business potential. By drawing meaning from huge swathes of data through the use of sophisticated algorithms and learning models, it has raised the competitive bar for businesses, who find themselves in need of hiring highly-specialised professionals to take advantage of its potential. Even if certain aspects of AI have started to become commoditised by Big Tech companies, stitching things together still requires a fair number of specialised skills.

It is no wonder then that universities around the world cannot train data and computer scientists fast enough to satisfy demand. And there is also a less obvious AI skillset gap which is the focus of this article.

How is AI integrated into businesses?

In most cases, AI is leveraged to make sense of data relevant to a given business. This could be data about customer behaviours, usage data, consumption data, weather data, or pretty much any type of structured or unstructured data sets that we can imagine.

Data has to be cleaned up and organised to ensure that it represents faithfully the reality it intends to describe. Data engineers must wade through vast amounts of data with various degrees of correlation to establish meaningful connections. They must also clean the data, removing or correcting input errors, ensuring values can be correlated and that equivalent representations of reality are used everywhere.

Once data has been coerced into a mesh of information points, data scientist and other AI specialists work to extract knowledge and insight from it. Frequently, this takes the form of very complex algorithms and data queries that are almost unreadable or even seem to make little sense to the untrained eye.

A focus on results

AI specialists focus on getting results from the data sets they are tasked to work on. Their objective is to create systems that can provide businesses with valuable insights. These systems are extremely complex, relying on deep data analysis algorithms, sometimes taking an inordinate amount of time to run. It is not uncommon that systems like these have to run for days to produce results.

One reason that may contribute to this is the fact that a data scientist or AI specialist is not too concerned about the computing performance of the models and algorithms they produce as long as they lead to meaningful results (in much the same way as a researcher investigates a domain looking for insights with little concern for the practicality of their findings). AI is still, in fact, strongly coupled with academic research, which is not surprising considering how much it is still evolving.

But this poses a problem of scale. As data grows, the impact of these unoptimised algorithms and models will be more keenly felt, to the point of becoming a real problem to the organisations that have deployed them.

Automation

Another problem is that AI solutions often result from arduous effort and are organically grown out of separate pieces of software components that need to be stitched together following highly customised recipes. Build and deployment automation becomes extremely difficult to achieve, especially if the build process changes every time.

The field of MLOps attempts to apply DevOps principles to the AI and Machine Learning (ML) fields. The movement is still very new and only recently has it seen its first true implementations. For the most part, these rely on specific AI and ML incarnations to be able to automate the build and deployment workflow. Seeing how much the AI field itself is still subject to rapid flow and change, they may not be applicable to more custom solutions.

The AI engineering skills gap

In conclusion, on top of the more prominent lack of specialised AI professionals, we're looking at a less immediately visible but no less important lack of traditional software engineering skills within the AI field to ensure high-performant, reliable, safe and easily-deployable software products.

A mix of AI specialisation with traditional software engineering practices is required to successfully bridge this gap. After all, current AI systems are mostly implemented within software, so designing and building them with established software engineering and computer science principles and practices seems logical.

Expert software services companies such as Critical Software, with a strong tradition in software engineering and a significant investment in the AI field, are uniquely positioned to eliminate process silos and bring AI to a level of practical and efficient implementation that it has not seen yet.

Rui Silva, Principal Engineer at Critical Software

You can read all insights from techUK's AI Week here