One of the hot topics in the developer world at the moment is building microservice-based cloud native applications and the pervasive nature of this approach in today’s development ecosystem. However, a significant number of developers have adopted a different approach.
Many of the developers I have spoken to recently have had a couple of things in common; firstly, they all come from organisations that own or process huge volumes of data on a daily basis. Secondly, all need to create a process involving both analytical and transactional workloads with that data using a variety of different technologies.
These analytical workflows are varied in nature. Many have found that being able to run SQL, for example, isn’t enough to solve the functional and non-functional requirements of queries in an adequate manner.
Consequently, a number of developers have been looking at non-traditional, bespoke data management solutions, usually running in a private cloud and often on specialist hardware. Given there is such a wide variety of well-known cloud native and on-premise platforms out there, in both the SQL and NoSQL spaces, it is interesting to see the number of organisations feeling the need to do this.
This is notable as while it may seem like the most cost-effective option to begin with, it is likely to turn out to be far less economic in the long-term. Besides, there is considerable risk inherent in this approach.
For these organisations, all will be well if the developers building the solutions remain with them. However, if they decide to leave – and let’s face it the competition for developers couldn’t be stronger – then, their employer has to find a way to retain them or face a knowledge vacuum surrounding the platform.
The other issue is a matter of functionality. Once the organisation wants to do something extra with the platform they will need to set up a data pipeline and replicate it in another data store, reconstructing it along the way. Before they know it, they have the same data replicated in four or five different structures. Suddenly, what started out as a cost-effective platform developed for a particular purpose has become both expensive and complex.
Interestingly, concerns about complexity are increasingly evident. When the number of discrete elements to maintain, monitor and manage increases, the complexity of doing so can scale in a non-linear manner. People are recognising that this can be a significant barrier to supporting and enhancing systems that have been deployed in a highly granular architecture and are looking for alternatives; this is one aspect deterring developers from going cloud native.
Additionally, if you consider the lower cost storage options on AWS, or any of the other cloud platforms, they are not fast mechanisms. If a business is data-intensive, then it does beg the question as to whether cloud native is the right route, as the data replication inherent in satisfying transactional and analytical workloads in a cloud native environment becomes increasingly expensive.
Of course, there are positives, especially if a developer makes use of all available services offered by the cloud provider. This can significantly reduce the time to build and deploy a solution – and ensures it is highly scalable. However, this does tie the organisation to a particular cloud provider; if a solution is built as a native application, for example in GCP, it can be difficult to refactor to move to another cloud environment.
It’s not surprising that many developers are building their own cloud native applications. However, it is surprising that so many, from such a technically inquisitive and inventive group, are implementing their own data platforms. They are forced to do so because of the shortcomings of cloud native architectures that scale well for relatively simple use cases that can easily be decomposed into small, functional units, but aren’t as good of a fit for more technically challenging requirements. This all means that new market developments which are addressing these challenges in different ways are certainly moving in the right direction.