Welcome to this Step-by-Step Guide on getting started with building deep learning model, serve it as REST API with Flask and deploy it using Docker and Kubernetes on Google Cloud Platform (GCP). Throughout this post, the focus will be on steps to successfully deploy the AI Model.

A Step-by-Step Guide for Deploying AI Models using Docker and Kubernetes on GCP

This is a guide for anyone out there who has heard about Docker and Kubernetes but hasn’t tried it out yet. To know more about DevOps Pipeline Workflow to Deploy AI Models with Docker and Kubernetes, read the related post  at An Overview of Pipeline Workflow to Deploy AI Models with Docker and Kubernetes

As this guide uses Docker and Kubernetes from GCP, you do not need get into the hassle of installing Docker and Kubernetes on your system.

Step 0: Need Google Account for GCP

A Step-by-Step Guide for Deploying AI Models using Docker and Kubernetes on GCP - SH2

Step 0.1: Creation of Account on Google

To get started we need a Google account. Once our google account is ready, we need to setup GCP.

Once our GCP is ready, we need to create a virtual machine on GCP to run out project.

Step 0.2: Creation of VM Instance on GCP

To create a VM instance you need to navigate to Compute Engine from the navigation bar on the top left corner of your screen and then select VM instances.

Then choose “Create”. Next, we will choose the operating system, the computational power, disk size and a few more specifications.

A Step-by-Step Guide for Deploying AI Models using Docker and Kubernetes on GCP - SH3.1
A Step-by-Step Guide for Deploying AI Models using Docker and Kubernetes on GCP- SH4

And now click on “Create”.

Step 0.3: Accessing the VM Instance using SSH Terminal.

Let’s get started with setting up the VM Instance for deploying using Docker. Click on ‘SSH’ to open SSH terminal.

A Step-by-Step Guide for Deploying AI Models using Docker and Kubernetes on GCP - SH5

Step 1: Docker Set Up on VM Instance

Step 1.1:

Firstly, we will remove the existing Docker from our VM Instance to do a clean installation of Docker. Note: if you choose OS other than CentOS 7, these commands may vary.


sudo yum remove docker docker-client docker-client-latest docker-common docker-latest docker-latest-logrotate docker-logrotate docker-selinux docker-engine-selinux docker-engine

Step 1.2:

Install Docker


sudo yum install -y yum-utils device-mapper-persistent-data lvm2

sudo yum-config-manager –add-repo https://download.docker.com/linux/centos/docker-ce.repo

sudo yum install docker-ce

Step 1.3:

Start Docker & Check if Docker is running properly. Once Docker installation is completed, we will start Docker

Check if docker is running properly


sudo systemctl start docker

sudo docker run hello-world

If you get the output as the one below, you are ready to move to next step:

Hello from Docker!
This message shows that your installation appears to be working correctly.To generate this message, Docker took the following steps: 1. The Docker client contacted the Docker daemon. 2. The Docker daemon pulled the “hello-world” image from the Docker Hub. (amd64) 3. The Docker daemon created a new container from that image which runs the executable that produces the output you are currently reading. 4. The Docker daemon streamed that output to the Docker client, which sent it to your terminal.

Step 1.4: Including the Flask API for Deep Learning Model on this VM Docker Instance

First, create a new directory called “docker_project”

Make sure to change directory and move into that folder


mkdir docker_project

cd docker_project

Step 1.4.1: Create app.py

Now, we will create app.py which will contain our deep learning model served as REST API on Flask.

Make sure, when calling app.run() specify the host as ‘’ like app.run(host=’’).  It makes our app available on localhost as well as on external IP.



vim app.py

Copy paste the  below code into app.py

from keras.applications import ResNet50
from keras.preprocessing.image import img_to_array
from keras.applications import imagenet_utils
from PIL import Image
import numpy as np
import flask
import io
import tensorflow as tf

# initialize our Flask application and the Keras model
app = flask.Flask(__name__)
model = None

def load_model():
	# load the pre-trained Keras model (here we are using a model
	# pre-trained on ImageNet and provided by Keras, but you can
	# substitute in your own networks just as easily)
	global model
	model = ResNet50(weights="imagenet")
	global graph
	graph = tf.get_default_graph()

def prepare_image(image, target):
	# if the image mode is not RGB, convert it
	if image.mode != "RGB":
		image = image.convert("RGB")

	# resize the input image and preprocess it
	image = image.resize(target)
	image = img_to_array(image)
	image = np.expand_dims(image, axis=0)
	image = imagenet_utils.preprocess_input(image)

	# return the processed image
	return image

@app.route("/predict", methods=["POST"])
def predict():
	# initialize the data dictionary that will be returned from the
	# view
	data = {"success": False}

	# ensure an image was properly uploaded to our endpoint
	if flask.request.method == "POST":
		if flask.request.files.get("image"):
			# read the image in PIL format
			image = flask.request.files["image"].read()
			image = Image.open(io.BytesIO(image))

			# preprocess the image and prepare it for classification
			image = prepare_image(image, target=(224, 224))

			# classify the input image and then initialize the list
			# of predictions to return to the client
			with graph.as_default():
				preds = model.predict(image)
				results = imagenet_utils.decode_predictions(preds)
				data["predictions"] = []

				# loop over the results and add them to the list of
				# returned predictions
				for (imagenetID, label, prob) in results[0]:
					r = {"label": label, "probability": float(prob)}

				# indicate that the request was a success
				data["success"] = True

	# return the data dictionary as a JSON response
	return flask.jsonify(data)

# if this is the main thread of execution first load the model and
# then start the server
if __name__ == "__main__":
	print(("* Loading Keras model and Flask starting server..."
		"please wait until server has fully started"))

Once you have copied the above code, you exit edit mode by hitting the “Esc” key. You then save and close the file by typing “:x”

Step 1.4.2: Create requirements.txt

To install dependencies required for our app, we need to create a file called requirements.txt.

 This file will contain the packages our code needs to run, e.g., keras, flask, and others. This way no matter where we ship our Docker container, the underlying server will be able to install our code’s required dependencies.



vim requirements.txt

Copy the following into requirements.txt and save and close as you did before for app.py file


Step 1.4.3: Create Dockerfile

A Dockerfile is a text file that contains a list of commands that the Docker daemon calls while creating an image. The Dockerfile contains all the information that Docker needs to know to run the app — a base Docker image to run from, location of your project code, any dependencies it has, and what commands to run at start-up. It is a simple way to automate the image creation process. The best part is that the commands you write in a Dockerfile are almost identical to their equivalent Linux commands. This means you don’t really have to learn new syntax to create your own Dockerfile.

Copy the following into Dockerfile. Save and close as you did before (using ‘Esc’ followed by :x”)

Docker reads this file to build and run the project.


vim Dockerfile


FROM python:3.6.10
COPY . /app

RUN pip install -r requirements.txt


CMD [“python”, “app.py”]

Step 2: Build New Docker Image

Once your Dockerfile is ready.  you can go ahead build your docker container. To build our Docker container, run the following command.

This instructs Docker to build a container for the code located in our current working director “docker_project”. With this command, python and the packages listed in requitements.txt are installed in our Docker Container.


sudo docker build -t docker_project:latest .

Step 3: Run Tests

Step 3.1: Once building the Docker Image is built, we will run this docker image and check if it works.

Check the status of your container


sudo docker run -p 5000:5000 docker_project

sudo docker ps -a

Step 3.2 : Testing out model.

  • With our model running, it’s time to test it. Select “View gcloud command” from SSH Options
A Step-by-Step Guide for Deploying AI Models using Docker and Kubernetes on GCP - SH6
  • Select Run in CLOUD SHELL.
  • Execute the first command as it is.
A Step-by-Step Guide for Deploying AI Models using Docker and Kubernetes on GCP - SH7

And run the following command:

curl -X POST -F image=@tiger.jpg ‘http://localhost:5000/predict’

Make sure tiger.jpg is in your current directory

You should see result like this:

A Step-by-Step Guide for Deploying AI Models using Docker and Kubernetes on GCP - SH8

We can see that our model correctly classified image of a tiger, as a TIGER. Bravo! You have successfully run a trained a deep learning model with Keras, served it with Flask, and wrapped it with Docker.  The hard part is over.

But, we are not yet done here. You will ONLY be able to access this docker image through the same VM machine.

What if you want to run the prediction from our deep learning model from anywhere in the world? This is where Kubernetes comes into play.

 Now, let’s deploy this container with Kubernetes.

Initiate Deployment on KUBERNETES

Step 4: To get started with Kubernetes, we/you need Docker Hub Account.

The reason we do this is that we won’t physically move our container to our Kubernetes cluster. Instead, we will instruct Kubernetes to install our container from a centrally hosted server, i.e., Docker Hub.

Now, we need to login to Docker Account from the same SSH

Step 4.1: Docker Hub Login

If you see message as : “Login Succeeded”, you are successfully logged into your Docker account, you can move to next step.

Step 4.2: Tagging Docker Container

Now, we need to tag our container before we upload it. For this we locate the image id .


sudo docker login


sudo docker images

Output will look like this:

A Step-by-Step Guide for Deploying AI Models using Docker and Kubernetes on GCP - SH9

Step 4.2: Tagging Docker Container (contd..)

Now, let’s tag our container . NOTE: Change the highlighted text as per your image ID , your Docker Hub ID and project name


sudo docker tag 79bf849d84c7 <Docker Hub ID>/docker_project

Step 4.3: Once our docker container is tagged, we can push our container to Docker Hub.

If you navigate to your Docker Hub Account, you should see the docker_project app in your repository


sudo docker push <Docker Hub ID>/docker_project

Step 5: Create Kubernetes Cluster for Deployment

Step 5.1: Navigate to Kubernetes Engine in GCP

A Step-by-Step Guide for Deploying AI Models using Docker and Kubernetes on GCP - SH10

Step 5.2: Create a new Kubernetes Cluster

A Step-by-Step Guide for Deploying AI Models using Docker and Kubernetes on GCP - SH11

After that click “Create”.

To connect to this cluster, click on “Connect”.

A Step-by-Step Guide for Deploying AI Models using Docker and Kubernetes on GCP - SH12

Click on “Run in Cloud Shell”. It will bring up a new console for Kubernetes Cluster.

A Step-by-Step Guide for Deploying AI Models using Docker and Kubernetes on GCP - SH13

Step 5.3: Now, We run our Docker container in Kubernetes

Let’s create a deployment of our app.


kubectl create deployment docker-project –image=<Docker Hub ID>/docker_project

Step 6: Run Kubernetes Service

Once the deployment is created, we can run our app. We’ll specify with –port that we want to run our app on port 5000


kubectl run docker-project –image=<Docker Hub ID>/docker_project –port 5000

Step 7: Expose the Deployment

We need to expose out pod on port 80 to outside world.


kubectl expose deployment docker-project –type=LoadBalancer –port 80 –target-port 5000

A Step-by-Step Guide for Deploying AI Models using Docker and Kubernetes on GCP - SH14

Open terminal on any PC and run the following command:


curl -X POST -F image=@tiger.jpg ‘http://<External IP of your VM instance>/predict’

By executing, the above steps you have now successfully build a deep learning model, served it as REST API with Flask and deployed it using Docker and Kubernetes on Google Cloud Platform (GCP).

At StrataHive, Our Core Capability is in Artificial Intelligence-based Software Component Development and Integration with other Systems/Application. We assist our Clients in adopting new Architectures and to operationalize their AI and ML Initiatives.

Know more about StrataHive’s Solution Offerings at this link.

Thanks for reading through this post. Any suggestions for further improving this would be cheerfully solicited.

Leave a Reply