Kubernetes (“koo-burr-NET-eez”) /κυβερνήτης/ - Can be used as noun or verb.
Noun"helmsman" or "pilot" or "Orchestrator".
We use Kubernetes to achieve resiliency for our application.
Verb
Perform the act of doing Kubernetes. When done using TKG it is easy but can be super hard if the right tool is not used.
Do you even Kubernetes?
If I were to survey about how many people in IT industry (regardless of role) knows or at least heard about Kubernetes I would be very surprised if the percentage came out any less than at least 80%.
I am curious though,
- How many people have actually deployed on Kubernetes?
- How many people have created a Kubernetes cluster?
- How?
The answer could go either way of "Yeah, it's easy" OR "Dude!! it's hard".
This is because, in my opinion, it all depends on choosing the right toolset that are fit for purpose.
In this post I will create a Kubernetes cluster and deploy a microservice application End-To-End, the easy way (really, if you follow along you will have your own cluster provisioned and app running in just under 20 mins).
If you would like to jump straight into the codes then check out my GitHub repo containing all the codes I used to create the below:
- 2 Kubernetes cluster (1 for app, 1 for CICD tool Jenkins) in a fully private environment
- Provisioned private container registry
- Deploy 3 micro-services on the application cluster
- Deploy Jenkins on a separate cluster and deploying using automated CICD pipeline.
Github: https://github.com/alinahid477/VMW/tree/main/calcgithub
Before I start let me state 1 thing:
"There is no way I will ever (ever, ever, ever) create a Kubernetes cluster from scratch" (unless someone pays me a billion dollars).
Why:
- First of all, no one does it.
- Second, it's really not worth the effort.
- Third, I tried and failed to create from scratch and then discovered tools and services for managing Kubernetes. Trust me, you should too.
WHY should you Kubernetes?
According to IDC's prediction: "By 2023, 80% of Workloads Will Shift to or Be Created with Containers/Microservices".
Why? Well, because of the bellow few of many simple and obvious reasons:
- Consistent Environment: Regardless of whichever environment the containerised application are deployed to eg> DEV, UAT, PROD etc it runs and behaves the same way.
- Run anywhere: Does not matter whichever host OS eg> Windows, Linux etc OR whichever cloud eg> public or private cloud, a containerised application just works and runs, giving it much needed portability in todays world.
- Isolation: As containers virtualize resources at the OS level, a containerised application runs logically isolated from other applications. This leads to efficient usage of resources and significant reduction of running cost.
I hope you are convinced why you must containerise whether your role is business focused or development or delivery focused.
Now you need a container orchestration engine so that you can achieve:
- High availability
- Fault tolerance
- Resiliency
- Streamlined and simplified operation
Here're some of the benefits that resonated well for me:
- Open source & Community: Kubernetes is open source and it is backed by a massive community and support and contribution from organization like Google, VMware.
- Extensibility: Kubernetes is very extensible. Hence, there are literally thousands of plugins available. On top you can build your own.
- Multi-cloud flexibility: Kubernetes makes it easy to run application on Hybrid and/or Multi-cloud. This allows to put the right workloads on the right cloud and to help avoid vendor lock-in.
- Future: With the adoption rate growing exponentially as companies embark on digital transformation journey and backed by data from IDC's research, it is safe to say that Kubernetes is here to stay.
Backstory to my usecase:
I had a monolith calculator application running on 4 VMs behind a L4 load balancer. The application is used for proprietary calculations hence running on private cloud.
The problem:
Like most tech issues this one also has 3 aspects to it.
The technology:
Few days ago it became super popular, a huge number of people started using the sum functionality of the application. The underlying infrastructure did not scale as the demand grew and the spike started to crash the application frequently.
This failure also started to chew up huge amount manual effort to bring it back up.
The people:
Initially the application's giant code base was being managed by a large team of developers. So, coordinating the maintenance and development of features and functionalities of the application was a challenge.
The process:
Although the application had a CICD pipeline using Jenkins it was failing randomly during build times due to either lack of memory on the VM or space shortage.
Solution evaluation:
Option#1-Virtical scaling:
This might work but will have a huge cost implication. And the question still remains that to what extent the resources should be vertically increased? So neither a sustainable not cost effective solution.
Option#2-Horizontal scaling:
This will work. With the growth of the application it is possible to add more VMs to support the spike. But this is not a cost effective solution and will waste resources. On top it will be a deployment nightmare with the current cumbersome deployment process.
Option#3-Containerize and Kubernetes:
This will work nicely. Because:
- Through Kubernetes it is possible to maintain a desired state and scale the app through pods and when needed VMs. This efficiently make use of allocated resources, thus cost effective.
- Through containerisation deployment process will become portable and installation of dependencies on VMs can be avoided. Thus operation will become smoother and simpler.
- Although, it does add some initial effort to modernize the application (eg> make it containerised) and setting up Kubernetes cluster but this is well justified effort by scoring the benefits it brings to the table.
WHAT to Kubernetes?
What is What in Kubernetes:
I don't want to sound like a broken record by explaining what is what in Kubernetes. Here's is diagram to visualize just in case:
Anatomy of Kubernetes
More details with good explanation here: https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/
Choosing the right toolset:
Private Cloud (existing):
My private cloud is on VMware vSphere 7 (ESXi on bare metal as hyperscaler and vSAN for storage virtualization) with NSX-T for network virtualization.
TKG for creating Kubernetes cluster:
Fortunately, vSphere 7 also comes with the popular Kubernetes cluster management tool, TKG (Tanzu Kubernetes Grid). The TKG on vSphere 7 acts like a managed service for Kubernetes cluster on vSphere data centre making it super easy to operationalize and develop on.
Ok, I realized I have said many products names. Below diagram should help visualizing my private cloud:
VMware SDDC
- The most impressive thing is on vSphere 7 (when Tanzu is enabled) the Kubernetes is treated as first class citizen;
- Meaning I do NOT need to go through the hassle for creating VMs and adding Kubernetes on it. I would only need to tell TKG to create a cluster and it will provision the underlying VMs and Networking for me.
- More to it, impressively, the networking, compute and storage are provisioned adhering to policy setup by the IT Governance team.
- AND TKG comes as an embedded service in vSphere 7.
So, TKG is the right choice here. (Why would I bother with other third party when TKG is embedded and Kubernetes is available as a service).
Harbor for Container Registry:
As I will be containerizing my applications I will need a container hosting aka container registry (so Kubernetes can pull the the container on demand).
When Tanzu product suit is enabled on vSphere 7 it also comes with Harbor container registry. With simple wizard like setup process I will be able to easily provision it on my private cloud and control my container registry to keep it private by not only applying the right user access but also through network security. So this is also a no brainer.
Once the Tanzu is enabled on vSphere 7, below diagram shows how the private cloud looks like from Operation and DevOps perspective:
vSphere 7 with Tanzu
Jenkins for CICD pipeline:
TBH, I am not a big fan of Jenkins. But since this is a private cloud and everything is restricted in the private cloud (meaning very limited interaction with internet egress and zero ingress) then Jenkins is a natural fit. Jenkins is popular and had been out there for a long time.
Now that I have Kubernetes capabilities in my vSphere I can super charge Jenkins on Kubernetes by deploying and configuring Jenkins on and for Kubernetes.
So Jenkins will run on a Kubernetes pod and whenever a job (CI and/or CD) comes in Jenkins will spin the agent to process the job in a new pod.
Thus it becomes highly scalable CICD solution.
Below diagram shows how Jenkins would look like in kubernetes:
Jenkins on Kubernetes
I will de-structure the monolith application into 3 pieces:
- Calculator interface (which is original application written in ReactJS but with 2 features decoupled.).
- Sum service (Decoupled sum functionality now written in Java Spring Boot)
- Substract service (Decoupled sum functionality now written in Java Spring Boot)
All these pieces (micro services) will need to be exposed so that users can access it, as well as one component can call the other. The simplest way to expose these 3 applications is by creating 3 services in Kubernetes. Here's a diagram of what it would look like in Kubernetes:
Application's Kubernetes cluster using LB only
But that would also mean creating 3 load balancers (which is an overhead in this case) and also hard coding the IP or the URL of the Load Balancers in the code; not a good practice.
This is a very common dilemma in the Kubernetes world. But there's always a solution in Kubernetes. The solution is to use an Ingress-Controller. I will use NGINX as the ingress controller.
With ingress controller in place, the 3 microservices services will be routed through ingress controller and I will only need one Layer 4 Load Balancer to expose the ingress controller.
It's like a L7 LB sitting behind a L4 LB.
User requests will come through the LB and pass through ingress controller to the applications' services to serve and it is handled through the ingress controller.
Below diagram shows how it looks like after ingress controller is applied:
Apps in Kubernetes using ingress controller
The below diagram is my high level solution design
HLSD (High level solution diagram - 1000 feet view)
HOW to Kubernetes?
Now that our planning is done of what to do with our applications and pipeline and what components to provision using the power of Kubernetes, let's get applying.
This section is pretty much the tutorial about:
- Create Kubernetes cluster using TKG
- Deploy Jenkins on Kubernetes (there are some stuffs you may want to record).
- Deploy microservice application on Kubernetes.
Create Kubernetes cluster using TKG
With TKG, creating kubernetes cluster is as simple as defining yaml file and deploying it using kubectl.
Here's a sample yaml:
apiVersion: run.tanzu.vmware.com/v1alpha1 #TKG API endpoint
kind: TanzuKubernetesCluster #required parameter
metadata:
name: calc-k8-cluster #cluster name, user defined
namespace: calc #supervisor namespace
spec:
distribution:
version: v1.18 #resolved kubernetes version
topology:
controlPlane:
count: 1 #number of control plane nodes
class: best-effort-small #vmclass for control plane nodes
storageClass: pacific-gold-storage-policy #storageclass for control plane
workers:
count: 4 #number of worker nodes
class: best-effort-small #vmclass for worker nodes
storageClass: pacific-gold-storage-policy #storageclass for worker nodes
What I am doing here are:
- Creating a cluster name "calc-k8-cluster" in the vSphere Namespace called "calc"
- The cluster will be created using Kubernetes version 1.18
- 1 VM will be created and assigned/allocated by TKG for control plane node
- 4 VMs will will be created and assigned/allocated by TKG for worker nodes
- best-effort-small machine type will be used in both cases master and workers. See more details about machine types here: https://docs.vmware.com/en/VMware-vSphere/7.0/vmware-vsphere-with-tanzu/GUID-7351EEFF-4EF0-468F-A19B-6CEA40983D3D.html
Please see code and the step by step guide here: https://github.com/alinahid477/VMW/tree/main/calcgithub/calc-devops
Follow step 1 - 3.
Deploy Jenkins on Kubernetes:
I will deploy Jenkins in its separate cluster so that I can use it to deploy on other clusters. Jenkins is just going to be a CICD tool for this purpose.
The step by step is documented here: https://github.com/alinahid477/VMW/tree/main/jenkins
Some important things to pay attention to are:
Deploy microservice application on Kubernetes
Before we can start deploying workload/apps on Kubernetes cluster we need to perform some one-off tasks for configuring kubernetes to prepare the kubernetes cluster for our applications.
The steps are described here: https://github.com/alinahid477/VMW/tree/main/calcgithub/calc-devops
Follow step 4,5,6.
Few important things to look out for are:
- Give Jenkins (our CICD tool) permissions to deploy to our cluster (as Jenkins is running on a different cluster.) through the usage of K8's Service Account. (See Step#4)
- Integrating private container registry to kubernetes (so that kubernetes knows how to pull the images for the apps when spinning up pods). (See Step#5)
- POD template defined in the Jenkins file. Pay special attention to service account used, volumes mounted and harbor integration. (Explained Step#5, Section: "Configmap for private registry with self signed ssl")
Chekout the code in my Github: https://github.com/alinahid477/VMW/tree/main/calcgithub
How long did it take?
Approx 20 mins (including Jenkins).
What's next:
- Use TKG to deploy to public cloud
- Use Tanzu Mission Control to bring all the k8 cluster into one single management place.
Now, the question to you is:
Why not use Tekton for CI/CD, which is more native to k8s and more importantly to save you a separate cluster for CI/CD. More specifically I would use Tekton automate my builds and ArgoCD to automate the deployment of manifests into Kubernetes.
ReplyDelete