Kubernetes cluster on Azure, AWS, GCP with Terraform and Seldon

Major cloud providers provide managed Kubernetes cluster offering and cli tool to create it easily. However, lots of companies are using are using IaC tool terraform for automating infrastructure provisioning. In this article, let’s use terraform to create Kubernetes cluster on different clouds, and quickly test a machine learning model deployment using Seldon

Install Terraform CLI

Install Terraform

On Ubuntu, it can be as simple as

sudo snap install terraform

Update: latest snap is not stable, follow article below to install terraform

Install Terraform

Install kubectl

Install and Set Up kubectl

Install cloud CLI (command line interface) and authenticate

Azure

Install the Azure CLI

Sign in with the Azure CLI

Azure cloud shell has terraform installed.

GCP

Installing Google Cloud SDK | Cloud SDK Documentation

gcloud auth login | Cloud SDK Documentation | Google Cloud

AWS

Installing, updating, and uninstalling the AWS CLI

Configuring the AWS CLI

AWS CloudShell is a browser-based shell that makes it easy to securely manage, explore, and interact with your AWS resources. CloudShell is pre-authenticated with your console credentials. Common development and operations tools are pre-installed, so no local installation or configuration is required.

AWS CloudShell — The Cloudly Engineer

verify

aws sts get-caller-identity

It should print out your signed-in user without needing to run aws configure.

Install terraform

AWS Cloud shell does not have terraform installed. Install by following Hashicorp documentation

Install Terraform | Terraform — HashiCorp Learn

sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/AmazonLinux/hashicorp.repo
sudo yum -y install terraform

Create Kubernetes cluster

Overall steps

1. Define VPC/virtual network/firewall rules
2. Define kubernetes cluster (master node/control plane, cluster name, location, VPC/subnet)
3. Define node pool (worker node/data plane, specification, node count)

AKS

Provision an AKS Cluster (Azure)

difference: service principal

Create a Kubernetes cluster with Azure AKS using Terraform

GKE

Provision a GKE Cluster (Google Cloud)

EKS

Provision an EKS Cluster (AWS)

Model deployment

Install s2i

How To Install and Use Source-To-Image (S2I) Toolkit on Linux | ComputingForGeeks

Install seldon-core

seldon-core

sklearn_iris model deployment

SeldonIO/seldon-core

Run following command to train model

python3 train_iris.py

Now run

s2i build -E environment . seldonio/seldon-core-s2i-python3:0.18 seldonio/sklearn-iris:0.1

Local docker test

docker run --name "iris_predictor" -d -p 5000:5000 seldonio/sklearn-iris:0.1
curl  -s http://localhost:5000/predict -H "Content-Type: application/json" -d '{"data":{"ndarray":[[5.964,4.006,2.081,1.031]]}}'

Setup seldon-core operator and ambassador

# helm issue https://github.com/datawire/ambassador-chart/issues/90

kubectl apply -f https://www.getambassador.io/yaml/ambassador/ambassador-crds.yaml
kubectl apply -f https://www.getambassador.io/yaml/ambassador/ambassador-rbac.yaml
kubectl config set-context $(kubectl config current-context) --namespace=default
helm install seldon-core seldon-core-operator --repo https://storage.googleapis.com/seldon-charts --set ambassador.enabled=true --set usageMetrics.enabled=true --namespace seldon

Seldon Core Setup - seldon-core documentation

Deploy model

kubectl create -f sklearn_iris_deployment.yaml
kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=seldon-deployment-example \
                                 -o jsonpath='{.items[0].metadata.name}')

Test model

# through seldon deployment pod
kubectl port-forward svc/sklearn-iris-example-sklearn-iris-example  8000:8000
curl  http://localhost:8000/api/v1.0/predictions -H "Content-Type: application/json" -d '{"data":{"ndarray":[[5.964,4.006,2.081,1.031]]}}'
# through ambassador
kubectl port-forward $(kubectl get pods  -l app.kubernetes.io/managed-by=getambassador.io -o jsonpath='{.items[0].metadata.name}') 8003:8080
curl  -s http://localhost:8003/seldon/default/seldon-deployment-example/api/v1.0/predictions -H "Content-Type: application/json" -d '{"data":{"ndarray":[[5.964,4.006,2.081,1.031]]}}'

Result

{"data":{"names":["t:0","t:1","t:2"],"ndarray":[[0.9548873249364059,0.04505474761562512,5.
7927447968953825e-05]]},"meta":{}}

Access ambassador diagnostic UI

kubectl port-forward --address 0.0.0.0  svc/ambassador-admin 8877
http://<ip address>:8877/ambassador/v0/diag/