How to manage data in hybrid cloud environments

This article was originally published on DevOps.com.

We’ve long-since established that multi-generational, multi-cloud environments are the new norm. But how does one go about managing data in those environments? In this post, we’ll take a look at the ramifications of this hybrid cloud reality and best practices for managing your data for increased scalability and heightened availability.

It’s all cloud

Whether it’s Amazon Web Services (AWS), Google Cloud, or Microsoft Azure (or some combination thereof), cloud is all the rage, with a hybrid approach gaining momentum, with 81% of cloud users working with two or more providers (according to a recent Gartner study). Not only does this strategy take companies out of the business of hosting their own applications, it leads to benefits like avoiding vendor lock-in, reduced costs, and optimized performance.

Benefits to mixing cloud and on-prem environments

Mixing cloud and on-prem environments means you can pick and choose where your workloads and data live — it’s really an optimization. For example, if your application is CPU intensive, you can run it on the cloud provider that gives you the most CPU for your dollar. Similarly, if your data workload requires significant storage, you can go to the cloud provider that offers the best bang for your buck on storage. Businesses should look at their application and database and then use the cloud provider that offers the best of class for that workload. Mixed cloud also gives you the ability to always have a low-cost secondary site recovery plan in place — you’re always ready to spin up on another cloud provider if you need to.

Managing data: the challenges of hybrid cloud

The benefits of multi-cloud are many, but when it comes to managing your data, things can get tricky. Not only do you have multiple cloud providers to balance, you also have to consider data stored on-premises and how that data is accessed from the cloud.

You also have to consider data replication between clouds. If you’re not doing a full mirror of your data between each cloud, it’s critical you remain acutely aware of what data lives where. You’ll want to consider creating extensive documentation and diagrams of what data lives in which cloud. If you are replicating data between clouds, you also have to justify the additional operating and engineering cost to accomplish your goal.

How to manage data in hybrid cloud environments

Fortunately, there are several libraries providing abstractions for cloud services, such as object stores. Leveraging abstractions within your applications allows you to move more freely between cloud providers.

Companies should strongly consider storing their data in a consistent manner within each cloud, leveraging the same technologies to reduce the sprawl, maintenance, and expertise required to operate those technologies. For example, use a relational database that all the cloud vendors provide — for data updated and changed frequently, it should live in the scope of the local cloud provider to reduce latency and network costs, because traffic within their infrastructure is generally free. If you were to write to a database outside the cloud provider, it’d cost more and be considerably slower. The leaders in the cloud space (mentioned earlier) offer a fairly secure environment for storing sensitive data, as long as you’re following good practices and using your own source of encryption. However, it all comes down to your individual environment. For some organizations, there will be workloads that should remain on-premises (such as highly sensitive data for which regulations require on-premises storage, for instance).

If data has to be stored encrypted on-premises, many cloud providers offer a secure means of connecting your cloud environments to your on-premises environment for access to sensitive information, such as providing secure VPN connectivity to virtual private clouds.

Multi-cloud might not be for everyone; you have to weigh the benefits with the costs, especially as those factors relate to your individual organization. And while there are pitfalls, the practices and tooling to effectively manage your data in hybrid cloud environments are all out there. Because so many of us are choosing to run our workloads on multiple clouds, collectively we’ve overcome many of the aforementioned challenges and can take advantage of all the benefits hybrid cloud provides.