Skip to main content

Jenkins on k8s - can it be this easy?

 As developers or devops we have had a somewhat love and hate relationship with Jenkins like "love oss based ci/cd that can be hosted on any environment with ranges of community plugins for pretty much anything" BUT "hate messy UI, lack of documentations, difficult to configure" etc etc.

But this post isn't about pros and cons of Jenkins, rather it is about how you can get Jenkins on your k8s super quick and easy (using Merlin).


Git Repo: https://github.com/alinahid477/jenkinsonk8s

Table of contents:

Why Jenkins

Jenkins remains a popular choice when it comes to CICD solution with a massive community of users and contributors (despite the fact there are new cool kids in block like Tekton etc). The way I see it (because of our love and hate relationship with it) "Jenkins is not CICD tool that you want it's the CICD too that you need" (get it? It's a quote from The Dark Night 😅). 

Here're few survey results I found in the internet re-iterating the Jenkins' slice in the CI/CD pie:


Why Merlin for Jenkin

Jenkins dominated as CI/CD for VM based workload, I am sure there're still Jenkins instances running on VMs and that's cool. But when it comes down to CI CD for container based workloads we start thinking about other tools like Tekton or Circle etc. I wonder if the reason for it are: 
  • "Jenkins needs to run on VMs and in the new world it is all k8s cluster (no more VMs)"
OR
  • "Making Jenkins work on k8s is hard, too many instructions, too many components to configure etc"

I get it, 
  • I read the similar blog posts and watched youtube videos and yes it felt like there're a fair bit of things to do. 
  • I also missed (probably couldn't find the right post) an end-to-end instructions on how to get Jenkins up and running on k8s and get it to CI and CD containers to k8s.
  • I also had somewhat reluctant attitude to learn another yaml/xml/json pipeline (Jenkin, AZ, CloudBuild, GitLab they all have different ways of pipeline definition -- Seriously, how many pipeline definition a guy/girl needs to learn to do simple CI CD? CI/CD is supposed to make life easy, not difficult).
I simply wanted to get a CI/CD pipeline going for my workload.( preferably something that I am familiar with and possibly have used before or have most of my pipelines already defined, like Jenkins).

This is where, I think, "Merlin for Jenkins" can help. Think of it like a installer for Jenkins on k8s. 


What is Merlin for Jenkins

In short, "Merlin for Jenkins" is a Docker based installer that lets you deploy a configured Jenkins on your k8s cluster. (optionally, along with a sample pipeline that you can customise or clone from for your own pipeline.)

When you hear the word "Merlin" the first thing that pops into head is "the mythic Merlin the Wizard". Well, "Merlin for Jenkins" is also a wizard but in software context. It is a combination of few things:
  • A bootstrapped docker container (based on debian distro) for deploying Jenkins on k8s equipped with:
    • kubectl
    • openssh (for ssh tunnel, if needed for tunneling through a bastion host in the case of private k8s cluster where k8s api server is not exposed)
    • mounted localfiles from the running directory
      • yaml files for Jenkins' k8s deployment
      • sh scripts executing commands (user input, display, kubectl etc) 
  • The docker container has wizard UI (not GUI) that walks you through the installation/deployment process of Jenkins on k8s (including a sample pipeline)
  • The wizard's automation (written in bash script) under the hood takes care of 
    • All the necessary preconfigs of the k8s cluster for Jenkins deployment (eg: PVC claim, PSP etc)
    • Deployment of Jenkins and exposing it through k8s's SVC for accessing
    • Configuring Jenkins with necessary plugins (eg: Kubernetes Cloud, GIT, Secret Manager etc)
    • Configuring k8s cluster (eg: k8s SA, Secret, Token etc) for running it "the k8s way" (Jenkins on Controller Pod and each job is run in Executor Pod).
    • (Optional) Creating a Sample-Pipeline definition based on the user's choice to get the user started quickly on pipeline definition for running job on executor pods. User can choose one of the below 2 options:
      • Traditional pipeline requiring Dockerfile in the source code repo.
      • Modern Build Service (TBS) based pipeline that auto detects source code and created OCI compliant image. Read more about Tanzu Build Service here.
    • (Optional) Configuring Jenkins secrets for connecting Jenkins to Git repo and Container Registry for the sample pipeline. 




How Merlin for Jenkins work

Before I start to the tech details, here are some FYI
  • It is not something new that you need to learn
  • It is simply a convenient utility tool that under-the-hood does the stuffs on k8s that you are familiar with and it is open source (meaning you have the source codes to look at and change if needed).
  • It is an end-to-end for running Jenkins on k8s that covers 
    • deployment
    • configure
    • sample code for pipeline ready for re-use
  • All data are handled in local context and for k8s secrets. No data is stored in a database and nothing goes out beyond the context of your local environment, target k8s cluster and Jenkins deployed on it.

Ok, hopefully the above clarifies few things. 
Below diagram shows "Merlin for Jenkins" high level architecture.

What you need before hand:

  • Docker-ce or Docker-ee on your machine
  • A k8s cluster, that you have access to, where you would like to deploy Jenkins to

How to use "Merlin for Jenkins" to deploy Jenkins on k8s:

  • Clone the git repo: git clone https://github.com/alinahid477/jenkinsonk8s.git
  • Provide details for accessing remote k8s
    • Environment variables as per instructions in the README.md 
    • Optionally supply a kubeconfig file named "config" (in the case of other k8s cluster that Tanzu K8S Grid)
    • Optionally supply a private key file named "id_rsa" for bastion host (in the case of ssh tunnel through a bastion host if the k8s api server is not public and only accessible through jump host or bastion host)
  • CD into the directory where the start.sh/bat file is located and Run it using ./Start.sh/bat as per instruction in README.md
    • The start script will:
      • Build docker container based on Dockerfile located in the current directory (hence you CD into the directory)
      • The docker container is based on Debian distro with several utility tools (eg: jq, kubectl, openssl etc) installed in it.
      • Mount current directory and run the docker container, thus within the docker container it has access to the necessary files (eg: .env file, yaml files etc)
    • The bash scripts in "Merlin for Jenkins" will automatically do the following:
      • Present series of questions for user input (Wizard UI)
      • Display information (Console UI) of its actions
      • Connect to the remote or local k8s cluster either directly or via ssh tunnel (as per your input via .env file for Tanzu k8s Grid cluster on vSphere or kubeconfig file for any other k8s cluster running anywhere and id_rsa file in .ssh directory if needed for bastion host)
      • Deploy Jenkins on k8s (as per the yaml files in kubernetes directory)
      • Configure Jenkins plugins for k8s cloud
      • Configure Jenkins plugins for pipeline
      • Create a Jenkins sample-pipeline (if you instruct it to) that will contain sample pipeline script which you can modify or clone after deployment finishes.
      • When deployment is complete it will give you bash/shell access to remote k8s cluster where you can perform kubectl commands if needed or simply exit. (kubectl is bootstarpped in the docker container)
  • Follow the wizard prompts 
    • Give input that it asks for
    • See what it is doing (it will display all the actions that it will perform, good for learning too)
  • Optionally you can also view the k8s yaml files and modify if needed that will get deployed on your cluster by the scripts (.sh files)

That's it. 

"Merlin for Jenkins" will 
  • Deploy Jenkins on the target k8s cluster
  • Configure Jenkins for running on k8s 
  • Optionally, create a sample pipeline script for defining job to run on k8s  

How Jenkins on k8s work

Jenkins (the jenkins controller) has been available in container form for sometime now. For running it on k8s few things are configured in a slightly different way. 

Here's a diagram to visualise how things are working inside in a Jenkins on k8s:


As you can see in the diagram, the below things are at play that's making Jenkins on k8s work.
  • Jenkins runs on controller pod(s) and uses PVC to persist data (eg: stored creds, configs etc).
  • k8s service (of type LB in this case) exposes the Jenkins controller / Jenkins UI on port 80 and 8080 (this is defined in kubernetes/jenkins/service.yaml file). 
  • "Kubernetes Cloud" plugin enables Jenkins to create pods for running Jenkins jobs. These pods are called executor pods.
  • Pipeline definition written in script (groovy script) contains the pod definition along with the containers it will use to run the tasks (eg: mvn build/test, kubectl deploy etc). (You can also use "kubernetes" plugin to have pre-defined pod template definition using UI. But I did not like this option because it restricts things and UI is fiddly).
  • When a job is submitted, aka a pipeline is triggered, Jenkins controller creates an executor pod as per definition (described above). Check out the 'sample-pipeline-job' pipeline definition script, that "Merlin for Jenkins" wizard will create for reference/sample code.
  • Jenkins on k8s can run parallel jobs. Each job creates a new executor pod. Hence, if you're submitting 10 jobs (10 things to CI / CD) 10 executor pods will be created, each tasked with running 1 job.
  • The executor pods are created dynamically (using Service Account with PSP) and when the job is completed (successful, failed or aborted) the pods are deleted right away. (You can configure how long the pod will live after no jobs have run on it using idleMinutes 5 in the executor pod definition. Read documentation here to learn more. Best practice is to delete it right away, default config. Only use idleMinutes to debug.)
  • The executor pods and jenkins controller pods will communicate using 50000 port over k8s svc (can also be created using internal service of type ClusterIP. This is in my TODO list for the "Merlin for Jenkins". Right now 50000 port is also configured on the external LB itself).
  • This utility tool, for its sample pipeline, creates a user called "Jenkins-Robot" with role "cluster-admin" for authentication in the same cluster (that jenkins is deployed on) for example purpose only. It saves the auth token for user "Jenkins-Robot" as "Secret Text" type in Jenkins controller and uses it in the pipeline definition script. The pipeline stage authenticates into target k8s cluster (in this context it is the same cluster) and executes kubectl apply or kubectl patch command. Checkout the code for the sample-pipeline job once "Merlin for Jenkins" finishes the deployment and configuration. You can leverage similar or clone as per the sample-pipeline for your own pipeline definition.

One of the sample pipeline jobs (using Tanzu Build Service, because I do not like Dockerfile in my repository nor I like to think about build and push stuff in my pipeline during dev phase) that gets created by "Merlin for Jenkins" (should you instruct the wizard to do so) called "sample-java-pipeline-tbs" looks like below:




This job definition
  • Make use of 3 plugins:
    • Credentials plugin: The credentials for git repo, k8s, container registry etc are stored using this plugin. This plugin is leveraged in pretty much all the stages except CI-Test stage.
    • GIT plugin: This plugin is used to checkout source code from GIT repository at the "CI-GIT" stage.
    • Kubernetes CLI: This plugin is used to authenticate to k8s cluster leveraging secrets stored using Credentials plugin at the "CD-Deploy" stage.
  • Make use of 2 containers:
    • Maven: Based on the source image "maven" this container has mvn tool installed in it. Thus when within the context of this container the pipeline can run mvn commands. This container is used at "CI - Compile & Test" stage.
    • KP: This is a custom image with has KPack cli installed in it. Thus when within the context of this container and authenticated to a k8s cluster I can execute kp cli command. This container is used at "CD - Make & Push container" stage. Within this container: 
      • The pipeline will use Kubernetes CLI plugin to authenticate into a remote k8s cluster where Tanzu Builder Service is installed/deployed & running.
      • The Tanzu Build Service builder is configured with 
        • A set of languages it needs to detect and compile
        • Base images that the container image will be sourced on 
        • container registry details where it needs to push the image to 
      • Using KP cli the pipeline will instructs Build Service to perform OCI image based on sourcecode it checked out.  

So, it is not really rocket since, is it? Simple enough.

Anticipated FAQs (for fun):

What about JenkinsX ?
Yes, JenkinsX is cool. Use it if you like. Nothing to do with this post.

Jenkins is so old school and complicated.
Mmmm, No, not really, at least not for the next several years in my opinion. If you think it is old use something else, no worries.

Tekton performs much better in k8s?
Really? Awesome. Good for Tekton. Maybe I will create a Merlin for Tekton if needs be in future.

I use AZ DevOps, I am all about Cloud Tech.
Sure. I get it. I am not really advocating for Jenkins here. This post is simply about using a wizard how it is a simplified process of get Jenkins on k8s up and running super fast.


Labels:

Popular posts from this blog

The story of a Hack Job

"So, you have hacked it" -- Few days ago one of the guys at work passed me this comment on a random discussion about something I built. I paused for a moment and pondered: Do I reply defending how that's not a hack. OR Do I just not bother I picked the second option for 2 reasons: It was late. It probably isn't worth defending the "hack vs" topic as the comment passed was out of context. So I chose the next best action and replied "Yep, sure did and it is working great.". I felt like Batman in the moment. In this post I will rant about the knowledge gap around hacking and then describe about one of the components of my home automation project (really, this is the main reason for this post) and use that as an example how hacking is cool and does not always mean bad. But first lets align on my definition of hacking: People use this term in good and bad, both ways. For example: "He/she did a hack job" -- Yeah, that probably
Post a Comment
Read more

Smart wifi controlled irrigation system using Sonoff and Home Assistant on Raspberry Pi - Part 1

If you have a backyard just for the sake of having one or it came with the house and you hate watering your garden or lawn/backyard then you have come to the right place. I genuinely believe that it is a waste of my valuable time. I would rather watch bachelorette on TV than go outside, turn on tap, hold garden hose in hand to water. Too much work!! Luckily, we have things like sprinkler system, soaker etc which makes things a bit easy. But you still have to get off that comfy couch and turn on tap (then turn off if there's no tap timer in place). ** Skip to the youtube video part if reading is not your thing   When I first moved into my house at first it was exciting to get a backyard (decent size), but soon that turned on annoyance when it came down maintaining it, specially the watering part. I laid bunch sprinklers and soaker through out the yard and bought tap timer but I still needed to routinely turn on the tap timer. Eventually few days ago I had enough of this rub
Post a Comment
Read more

Exception Handling With Exception Policy

This is how I would think of an application at the very basic level: Now this works great. But one thing that is missing in this picture is Exception Handling . In many cases we pay very less attention to it and take it as "we'll cross that bridge when it'll come to that". We can get away with this as in many application as exceptions does not stop it from being in the state "is the application working" as long as we code it carefully and at the very least handling the exceptions in code blocks. This works. But we end up having try catch and if else everywhere and often with messy or no direction to what type of exception is to be handled where and how. Nonetheless, when it comes down an enhancement that depends upon different types exceptions, we will end up writing/modifying code every where, resulting in even messier code. I'm sure no one wants that. Even, in scenarios, a custom handler is not the answer either. Cause this way we will s
Post a Comment
Read more