As Artificial Intelligence (AI) & Machine Learning (ML) gains increasing importance in creating value for the Enterprise, it has become imperative for decision-makers to understand the deployment of ML Models at scale. Thankfully, Enterprises can now adopt a few new architectures to operationalize ML. Amongst the enabling options, we will discuss Docker, Kubernetes(K8S) and Micro-Services architecture that can help solve the heterogeneity and infrastructure challenges.

The task of deploying any model is to seamlessly integrate into Production Environment, where it takes an input and returns a desired output i.e. usually a prediction in case of ML.

DevOps Pipeline Workflow to Deploy AI Models with Docker and Kubernetes - SH1

While the End Point remains the same, deployment of ML Models presents unique considerations:

  • Being written in different programming languages (Python, R, etc) with multitude of dependencies (libraries, versions)
  • Require high compute power, relying on GPU’s
  • Lacking Portability (different Operating Systems)
  • Challenges of Scaling to meet increases in performance and application demand in production

If you want to expose your model as an endpoint, containers provide a great way of separating the model application code from model serving infrastructure.

Containerization technologies, such as Docker, can solve incompatibility and portability challenges introduced by the multitude of tools.

Kubernetes (K8S) works alongside a Containerization tool, like Docker to automate Scaling and Management of Models.

DevOps Pipeline Workflow to Deploy AI Models with Docker and Kubernetes - SH2

Think of this post as more of a ‘Precursor to ML on Kubernetes’ for demonstrating capability and allowing stakeholders to get up-and-running quickly and familiarizing with the basic concepts and patterns. It is not a comprehensive guide to Kubernetes, Docker or ML.

This post is designed to be read in conjunction with A Step-by-Step Guide for Deploying AI Models using Docker and Kubernetes on GCP, that contains the Python modules, Docker configuration files and Kubernetes instructions for demonstrating how a simple Python ML model can be turned into a production-grade RESTful model-scoring (or prediction) API service, using Docker and Kubernetes – both locally and with Google Cloud Platform (GCP).

Brief about Docker:

Docker is a tool designed to make it easier to create, deploy, and run applications by using containers. Containers allow a developer to package up an application with all of the parts it needs, such as libraries and other dependencies, and deploy it as one package. With Docker you can separate your applications from your infrastructure and treat your infrastructure like a managed application.

Because Docker containers encapsulate everything an application needs to run (and only those things), they allow applications to be shuttled easily between environments. Any host with the Docker runtime installed—be it a developer’s laptop or a public cloud instance—can run a Docker container.

Some Major Advantages of using Docker:

  • Docker enables more efficient use of system resources
  • Docker enables faster software delivery cycles
  • Docker enables application portability
  • Docker shines for microservices architecture
  • Docker is Open Source
Pipeline Workflow to Deploy AI Models with Docker and Kubernetes - SH4

Docker Containers are a good way to bundle and run your applications.

In a production environment, you need to manage the containers that run the applications and ensure that there is no downtime. For example, if a container goes down, another container needs to start. Wouldn’t it be easier if this behavior was handled by a system? That’s how Kubernetes comes to the rescue!

Brief about Kubernetes (K8S):

Kubernetes provides you with a framework to run distributed systems resiliently. It takes care of scaling and fail-over for your application, provides deployment patterns, and more. For example, Kubernetes can easily manage a canary deployment for your system.

DevOps Pipeline Workflow to Deploy AI Models with Docker and Kubernetes SH5

Kubernetes provides you with:

  • Service discovery and Load Balancing
  • Storage Orchestration
  • Automated rollouts and rollbacks
  • Automatic bin packing
  • Self-healing

Why use Kubernetes?

  • It can be run anywhere. Be it On Premise (bare metal, Open Stack) or Public Clouds (GCP, Azure, AWS)
  • Kubernetes avoids Vendor lock-in as much as by not using vendor-specific API’s or services

With this as a brief Introduction to Deployment, Containerization, Docker and Kubernetes, refer to the next blog in the Series: A Step-by-Step Guide for Deploying AI Models using Docker and Kubernetes on GCP  to understand the steps in developing a CI (Continuous Integration) / CD (Continuous Delivery) Pipeline Workflow with Docker & Kubernetes.

DevOps Pipeline Workflow to Deploy AI Models with Docker and Kubernetes SH3

At StrataHive, Our Core Capability is in Artificial Intelligence-based Software Component Development and Integration with other Systems/Application. We assist our Clients in adopting new Architectures and to operationalize their AI and ML Initiatives.

Know more about StrataHive’s Solution Offerings at this link.

Thanks for reading through this post. Any suggestions for further improving this would be cheerfully solicited.

Leave a Reply