Skip to main content

Do you even Kubernetes ? - in private cloud

Kubernetes (“koo-burr-NET-eez”) /κυβερνήτης/ - Can be used as noun or verb.
Noun
"helmsman" or "pilot" or "Orchestrator".
We use Kubernetes to achieve resiliency for our application.
Verb
Perform the act of doing Kubernetes. When done using TKG it is easy but can be super hard if the right tool is not used.
Do you even Kubernetes?

If I were to survey about how many people in IT industry (regardless of role) knows or at least heard about Kubernetes I would be very surprised if the percentage came out any less than at least 80%.

I am curious though,
  • How many people have actually deployed on Kubernetes?
  • How many people have created a Kubernetes cluster?
  • How?

The answer could go either way of "Yeah, it's easy" OR "Dude!! it's hard".

This is because, in my opinion, it all depends on choosing the right toolset that are fit for purpose.


In this post I will create a Kubernetes cluster and deploy a microservice application End-To-End, the easy way (really, if you follow along you will have your own cluster provisioned and app running in just under 20 mins).


If you would like to jump straight into the codes then check out my GitHub repo containing all the codes I used to create the below:

  • 2 Kubernetes cluster (1 for app, 1 for CICD tool Jenkins) in a fully private environment
  • Provisioned private container registry
  • Deploy 3 micro-services on the application cluster
  • Deploy Jenkins on a separate cluster and deploying using automated CICD pipeline. 

Github: https://github.com/alinahid477/VMW/tree/main/calcgithub


Before I start let me state 1 thing: 

"There is no way I will ever (ever, ever, ever) create a Kubernetes cluster from scratch" (unless someone pays me a billion dollars).  

Why:

  • First of all, no one does it. 
  • Second, it's really not worth the effort. 
  • Third, I tried and failed to create from scratch and then discovered tools and services for managing Kubernetes. Trust me, you should too. 

WHY should you Kubernetes?

According to IDC's prediction: "By 2023, 80% of Workloads Will Shift to or Be Created with Containers/Microservices". 

Why? Well, because of the bellow few of many simple and obvious reasons:

  • Consistent Environment: Regardless of whichever environment the containerised application are deployed to eg> DEV, UAT, PROD etc it runs and behaves the same way.
  • Run anywhere: Does not matter whichever host OS eg> Windows, Linux etc OR whichever cloud eg> public or private cloud, a containerised application just works and runs, giving it much needed portability in todays world.
  • Isolation: As containers virtualize resources at the OS level, a containerised application runs logically isolated from other applications. This leads to efficient usage of resources and significant reduction of running cost. 

I hope you are convinced why you must containerise whether your role is business focused or development or delivery  focused. 
Now you need a container orchestration engine so that you can achieve:
  • High availability
  • Fault tolerance
  • Resiliency
  • Streamlined and simplified operation
Kubernetes is the most popular container orchestration engine available in the market. 
Here're some of the benefits that resonated well for me:
  • Open source & Community: Kubernetes is open source and it is backed by a massive community and support and contribution from organization like Google, VMware.
  • Extensibility: Kubernetes is very extensible. Hence, there are literally thousands of plugins available. On top you can build your own. 
  • Multi-cloud flexibility: Kubernetes makes it easy to run application on Hybrid and/or Multi-cloud. This allows to put the right workloads on the right cloud and to help avoid vendor lock-in. 
  • Future: With the adoption rate growing exponentially as companies embark on digital transformation journey and backed by data from IDC's research, it is safe to say that Kubernetes is here to stay.

Backstory to my usecase:

I had a monolith calculator application running on 4 VMs behind a L4 load balancer. The application is used for proprietary calculations hence running on private cloud.

The problem:

Like most tech issues this one also has 3 aspects to it.

The technology: 
Few days ago it became super popular, a huge number of people started using the sum functionality of the application. The underlying infrastructure did not scale as the demand grew and the spike started to crash the application frequently.
This failure also started to chew up huge amount manual effort to bring it back up.

The people: 
Initially the application's giant code base was being managed by a large team of developers. So, coordinating the maintenance and development of features and functionalities of the application was a challenge.  

The process: 
Although the application had a CICD pipeline using Jenkins it was failing randomly during build times due to either lack of memory on the VM or space shortage.

Solution evaluation:

Option#1-Virtical scaling: 
This might work but will have a huge cost implication. And the question still remains that to what extent the resources should be vertically increased? So neither a sustainable not cost effective solution. 

Option#2-Horizontal scaling: 
This will work. With the growth of the application it is possible to add more VMs to support the spike. But this is not a cost effective solution and will waste resources. On top it will be a deployment nightmare with the current cumbersome deployment process.

Option#3-Containerize and Kubernetes: 
This will work nicely. Because:
  • Through Kubernetes it is possible to maintain a desired state and scale the app through pods and when needed VMs. This efficiently make use of allocated resources, thus cost effective. 
  • Through containerisation deployment process will become portable and installation of dependencies on VMs can be avoided. Thus operation will become smoother and simpler. 
  • Although, it does add some initial effort to modernize the application (eg> make it containerised) and setting up Kubernetes cluster but this is well justified effort by scoring the benefits it brings to the table.
Because only the sum feature of the application was identified as the most used feature I also decided to make that as its own service so that functionality can scale with demand independently to other features of the application. This will also simplify the operation of development and deployment.

WHAT to Kubernetes?

What is What in Kubernetes:

I don't want to sound like a broken record by explaining what is what in Kubernetes. Here's is diagram to visualize just in case:
Anatomy of Kubernetes
More details with good explanation here: https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/ 

Choosing the right toolset:

Private Cloud (existing):
My private cloud is on VMware vSphere 7 (ESXi on bare metal as hyperscaler and vSAN for storage virtualization) with NSX-T for network virtualization. 

TKG for creating Kubernetes cluster:
Fortunately, vSphere 7 also comes with the popular Kubernetes cluster management tool, TKG (Tanzu Kubernetes Grid). The TKG on vSphere 7 acts like a managed service for Kubernetes cluster on vSphere data centre making it super easy to operationalize and develop on. 

Ok, I realized I have said many products names. Below diagram should help visualizing my private cloud:

VMware SDDC

  • The most impressive thing is on vSphere 7 (when Tanzu is enabled) the Kubernetes is treated as first class citizen; 
  • Meaning I do NOT need to go through the hassle for creating VMs and adding Kubernetes on it. I would only need to tell TKG to create a cluster and it will provision the underlying VMs and Networking for me. 
  • More to it, impressively, the networking, compute and storage are provisioned adhering to policy setup by the IT Governance team. 
  • AND TKG comes as an embedded service in vSphere 7.
So, TKG is the right choice here. (Why would I bother with other third party when TKG is embedded and Kubernetes is available as a service).

Harbor for Container Registry:
As I will be containerizing my applications I will need a container hosting aka container registry (so Kubernetes can pull the the container on demand). 
When Tanzu product suit is enabled on vSphere 7 it also comes with Harbor container registry. With simple wizard like setup process I will be able to easily provision it on my private cloud and control my container registry to keep it private by not only applying the right user access but also through network security. So this is also a no brainer.


Once the Tanzu is enabled on vSphere 7, below diagram shows how the private cloud looks like from Operation and DevOps perspective:

vSphere 7 with Tanzu

Jenkins for CICD pipeline:
TBH, I am not a big fan of Jenkins. But since this is a private cloud and everything is restricted in the private cloud (meaning very limited interaction with internet egress and zero ingress) then Jenkins is a natural fit. Jenkins is popular and had been out there for a long time. 
Now that I have Kubernetes capabilities in my vSphere I can super charge Jenkins on Kubernetes by deploying and configuring Jenkins on and for Kubernetes. 
So Jenkins will run on a Kubernetes pod and whenever a job (CI and/or CD) comes in Jenkins will spin the agent to process the job in a new pod. 
Thus it becomes highly scalable CICD solution.

Below diagram shows how Jenkins would look like in kubernetes:

Jenkins on Kubernetes


NGINX-Ingress Controller:
I will de-structure the monolith application into 3 pieces: 
  • Calculator interface (which is original application written in ReactJS but with 2 features decoupled.).
  • Sum service (Decoupled sum functionality now written in Java Spring Boot)
  • Substract service (Decoupled sum functionality now written in Java Spring Boot)
All these pieces (micro services) will need to be exposed so that users can access it, as well as one component can call the other. The simplest way to expose these 3 applications is by creating 3 services in Kubernetes. Here's a diagram of what it would look like in Kubernetes:

Application's Kubernetes cluster using LB only



But that would also mean creating 3 load balancers (which is an overhead in this case) and also hard coding the IP or the URL of the Load Balancers in the code; not a good practice. 
This is a very common dilemma in the Kubernetes world. But there's always a solution in Kubernetes. The solution is to use an Ingress-Controller. I will use  NGINX as the ingress controller. 
With ingress controller in place, the 3 microservices services will be routed through ingress controller and I will only need one Layer 4 Load Balancer to expose the ingress controller. 
It's like a L7 LB sitting behind a L4 LB.
User requests will come through the LB and pass through ingress controller to the applications' services to serve and it is handled through the ingress controller. 
Below diagram shows how it looks like after ingress controller is applied:

Apps in Kubernetes using ingress controller



The below diagram is my high level solution design
                                     HLSD (High level solution diagram - 1000 feet view)





HOW to Kubernetes?

Now that our planning is done of what to do with our applications and pipeline and what components to provision using the power of Kubernetes, let's get applying.

This section is pretty much the tutorial about:
  • Create Kubernetes cluster using TKG
  • Deploy Jenkins on Kubernetes (there are some stuffs you may want to record).
  • Deploy microservice application on Kubernetes. 

Create Kubernetes cluster using TKG

With TKG, creating kubernetes cluster is as simple as defining yaml file and deploying it using kubectl.

Here's a sample yaml:

apiVersion: run.tanzu.vmware.com/v1alpha1 #TKG API endpoint 
kind: TanzuKubernetesCluster #required parameter 
metadata: 
  name: calc-k8-cluster #cluster name, user defined 
  namespace: calc #supervisor namespace 
spec: 
  distribution: 
    version: v1.18 #resolved kubernetes version 
  topology: 
    controlPlane: 
      count: 1 #number of control plane nodes 
      class: best-effort-small #vmclass for control plane nodes 
      storageClass: pacific-gold-storage-policy #storageclass for control plane
    workers: 
      count: 4 #number of worker nodes 
      class: best-effort-small #vmclass for worker nodes 
      storageClass: pacific-gold-storage-policy #storageclass for worker nodes

What I am doing here are:

Please see code and the step by step guide here: https://github.com/alinahid477/VMW/tree/main/calcgithub/calc-devops 
Follow step 1 - 3.

Deploy Jenkins on Kubernetes:

I will deploy Jenkins in its separate cluster so that I can use it to deploy on other clusters. Jenkins is just going to be a CICD tool for this purpose. 

The step by step is documented here: https://github.com/alinahid477/VMW/tree/main/jenkins


Some important things to pay attention to are:
  • Persistent volume for Jenkins and volume mount (See Step#2)
  • Service account, so that Jenkins can spin pods for running agent (See Step#3)
  • Volume mount and Docker Sock (See Step#4)
  • Kubernetes token for Continuous Deployment using Jenkins (See Step#5)

Deploy microservice application on Kubernetes

Before we can start deploying workload/apps on Kubernetes cluster we need to perform some one-off tasks for configuring kubernetes to prepare the kubernetes cluster for our applications.

Follow step 4,5,6.

Few important things to look out for are:
  • Give Jenkins (our CICD tool) permissions to deploy to our cluster (as Jenkins is running on a different cluster.) through the usage of K8's Service Account. (See Step#4)
  • Integrating private container registry to kubernetes (so that kubernetes knows how to pull the images for the apps when spinning up pods). (See Step#5)
  • POD template defined in the Jenkins file. Pay special attention to service account used, volumes mounted and harbor integration. (Explained Step#5, Section: "Configmap for private registry with self signed ssl")


How long did it take?

Approx 20 mins (including Jenkins).


What's next:

  • Use TKG to deploy to public cloud
  • Use Tanzu Mission Control to bring all the k8 cluster into one single management place.

Now, the question to you is:



 

References:



Labels:

Comments

  1. Shahriar18 May, 2021

    Why not use Tekton for CI/CD, which is more native to k8s and more importantly to save you a separate cluster for CI/CD. More specifically I would use Tekton automate my builds and ArgoCD to automate the deployment of manifests into Kubernetes.

    ReplyDelete

Post a Comment

Popular posts from this blog

The story of a Hack Job

"So, you have hacked it" -- Few days ago one of the guys at work passed me this comment on a random discussion about something I built. I paused for a moment and pondered: Do I reply defending how that's not a hack. OR Do I just not bother I picked the second option for 2 reasons: It was late. It probably isn't worth defending the "hack vs" topic as the comment passed was out of context. So I chose the next best action and replied "Yep, sure did and it is working great.". I felt like Batman in the moment. In this post I will rant about the knowledge gap around hacking and then describe about one of the components of my home automation project (really, this is the main reason for this post) and use that as an example how hacking is cool and does not always mean bad. But first lets align on my definition of hacking: People use this term in good and bad, both ways. For example: "He/she did a hack job" -- Yeah, that probably
Post a Comment
Read more

Smart wifi controlled irrigation system using Sonoff and Home Assistant on Raspberry Pi - Part 1

If you have a backyard just for the sake of having one or it came with the house and you hate watering your garden or lawn/backyard then you have come to the right place. I genuinely believe that it is a waste of my valuable time. I would rather watch bachelorette on TV than go outside, turn on tap, hold garden hose in hand to water. Too much work!! Luckily, we have things like sprinkler system, soaker etc which makes things a bit easy. But you still have to get off that comfy couch and turn on tap (then turn off if there's no tap timer in place). ** Skip to the youtube video part if reading is not your thing   When I first moved into my house at first it was exciting to get a backyard (decent size), but soon that turned on annoyance when it came down maintaining it, specially the watering part. I laid bunch sprinklers and soaker through out the yard and bought tap timer but I still needed to routinely turn on the tap timer. Eventually few days ago I had enough of this rub
Post a Comment
Read more

Exception Handling With Exception Policy

This is how I would think of an application at the very basic level: Now this works great. But one thing that is missing in this picture is Exception Handling . In many cases we pay very less attention to it and take it as "we'll cross that bridge when it'll come to that". We can get away with this as in many application as exceptions does not stop it from being in the state "is the application working" as long as we code it carefully and at the very least handling the exceptions in code blocks. This works. But we end up having try catch and if else everywhere and often with messy or no direction to what type of exception is to be handled where and how. Nonetheless, when it comes down an enhancement that depends upon different types exceptions, we will end up writing/modifying code every where, resulting in even messier code. I'm sure no one wants that. Even, in scenarios, a custom handler is not the answer either. Cause this way we will s
Post a Comment
Read more