AIOps Agents - PoC to Prod - OpenClaw to Kagenti

AI Agents are here and most of us are now used to agents assisting us with several types of tasks. As a result: 

• New kind of usecases are emerging to improve efficiency, reduce cost, reduce effort on repeating tasks etc 
  
  
• New types software engineering paradigm and architectural pattern are coming up
  
  
• New category terminologies has also surfaced and still getting introduced.

One of such new types of uses cases (that also leverages the other new types I listed above) is "Agents for Ops". 

In this post I will explain my journey to creating agents for ops. (I build 2: 1 for K8s and 1 for VMs fleet; but to keep this post short I will focus on the k8s-namespace-monitoring-agent). 

It is a description of path I took to build an agent that monitors applications (multiple) deployed on K8s cluster and remediates issues that impacts the applications' functionality. But it does so based on a knowledge base given to it, nothing more and nothing less and that's the purpose -- the agent is like an intern performing level 1 SRE tasks.   

The Demo:

OpenClaw:

Kagenti:

The usecase:

My usecase is simple:

• I have 2 sample applications (a nginx based web server, a micro-services app that stores data in db and processes information from external services) running in my OpenShift k8s cluster. These workloads emits logs messages (error, info, warning etc). To simulate breaking effects I have added "chaos monkey" in there that randomly break things.
  
  
• I have a knowledge base where I will have some details about log messages that applications may throw. It captures information such as the log message, what it means, what action to take to remediate it.
  
  
• The idea here is that a system powered by AI should be able to understand this information (the KB) and cross match it against the actual log messages emitting from the applications and interpret its criticality, whether remediation is needed or what remediation is needed and perform the remediation actions to resolve the issues. It should also let the human know its findings, actions taken by it and what advise what action is remaining for user (all based on the KB) and validation in a human friendly way.   

This calls for an "Ops Agent". I, non-creatively, called mine K8s-namespace-monitor-agent. If I interpret the usecases in "capabilities of the agent" it translates to something like this:

• The agent is going to do a series of tasks in the below order:
• reading the logs and understand which logs to read from the KB.
• cross matching it against KB, 
• interpret the meaning for human communication
• interpret the remediation plain text information into actionable items 
• execute the actionable items the way it makes sense
• also, optimise the action executions such a way that if a remediation action to fix one issue is also the fix for other issues it should understand it and optimise the actions accordingly. 
  
  
• As a guardrail the agent should not deviate from the knowledge base. This means that I get to control the scope of the agent (eg: namespaces to monitor, logs (error, warn, info) to interpret/tackle, actions to take etc) via the knowledge base.
  
  
• In short it boils down to
• contextualise information
• orchestrate workflow
• execute actions using tools 

The guiding principle:

The LLM: This is the brain of the operation. But as you can see from the use case description the brain does not need to be very intelligent. It just needs to be intelligent enough. We also need to keep the cost of harnessing super low so that it does not require additional budget to run.

The PoC to production: LLM to make decision is nondeterministic in nature. But there are ways to reach tolerable reliability when harnessing LLM. So before I spend a lot of time behind developing the agent I need to prove the hypothesis that this can be achieved with tolerable reliability. Then understand what would it take to deploy it to production. More on this below.

The Tool Selection:

The LLM: To keep the inferencing cost to minimum I went with small LLM as opposed to the front tier models such as Claude, GPT, Gemini etc. I chose Qwen3 4b and run it on OpenShift AI platform using MaaS (model as service). I made this choice based on the below selections:

• Tool calling capability: Quen3-4b model can make tool calls. I tested this with great success rate.
  
  
• The contextualisation: The use case is specific about processing KB and not a broad set K8s operations. The small LLM was able to do it with almost no errors.
  
  
• Since this small LLM does not require a huge GPU capacity I was able to self host it on RTX 3060 in lab in the PoC phase then moved to a more server grade GPU like L40 as well as T4. This will nicely keep the cost low and the LLM won't even take up entire GPU capacity. 
  
  
• OpenShift AI's platform comes with various GPU accelerator; so provisioning MaaS using the qwen3-4b was easy.

There are a bit more work went into this to come to the above realisation which deserves it's own post. I will link that post here shorly.

The PoC: The next challenge was to decide a quick way of PoCing it. So naturally I looked for "ready-to-go" "off the shelf" agent solution to keep the development effort to bare minimum. The tool choice here was to use OpenClaw because of the below decisions:

• I did some digging and turns out I could easily deploy openclaw in sandbox mode on an OpenShift cluster (more on this below) and there are ways to restrict it to my usecases only. The advantage here (when deployed to a OpenShift cluster in sandbox mode) is that it is an isolated instance and only has access to stuffs that I give it to.
  
  
• I can create agent skills for my usecases with low effort and get OpenClaw agent to execute tasks based on skill. This is a low effort way to test the hypothesis.
  
  
• There's a sandbox environment ready to go available (at no cost) from Red Hat at Red Hat Developer Sanbox portal. 

See below for more details on this PoC.

The production deployment: After the PoC had proven to successful (meaning, OpenClaw was able to execute as per skill and find and remediate issues within the boundaries of the KB) it was time to move to "how to implement in production" environment. I could have taken the path to deploy OpenClaw in my "production" environment but there were a lot of concerns specially around security and governance. OpenClaw wasn't the right. But Kagenti was. The decisions were:

• Kagenti is integrated with SPIFFE/SPIRE (on OpenShift's Zero Trust Workload Identity Manager) and agent and tools deployed in it are auth protected. This alone solves a lot of security concerns and completely removes the need to have credentials baked into agents and tools. It also solves the "confused deputy" issue that I was worried about in the case of OpenClaw PoC.
  
  
• Kagenti is K8s native. Meaning it uses K8s native components (such as ZTWIM, cert-manager, Istio, Kuadrant or OpenShift connectivity link, K8s's Reference Grant, Gateway etc) to achieve a complete set of  components and assets that make it easier to manage AI agents and tools and integrate their fine-grained authorization into modern cloud-native environments.
  
  
• The Gateway components (MCP Gateway, AI Gateway etc) were a big win. Because it makes it simple for the agent (the agent development) to keep track of endpoint (only gateway endpoint) and make all the required tool calls and AI access. It also add to governance and security maintained that platform level rather than baked into source codes.
  
  
• The usage of the gateways also means I could switch the underlying tools or inference endpoints without needing to change agent and vice versa. I can also use prefixes, filters etc at the platform level to enable agents/LLM with multiple tools and switching LLM endpoints. This is good for CI and CD both. 
  
  
• It supports A2A protocol. This was needed for future scenario where I would deploy orchestration agent and wire the sub agents to perform fleet management (k8s, vms, different orgs etc).
  
  
• It support MCP protocol for tool/server integration. This was required.

So far in my search for Kagenti came out to be most fitting (if not the only one) platform for this AI Ops Agentic usecase.

The implementation:  

It is a 3 part process.

Part 1 - The MaaS:

I used the OpenShift AI's MaaS (model as service) tool to self host Qwen3-4B model. Under the hood it uses vLLM via KServe to expose an endpoint to inference against the LLM. It has its own gateway (backed by istio) called maas-common-gateway which is the default gateway. I used the default. This was easy enough process as long as OpenShift AI 3.0 is available on the cluster (which was a bit complex and deserves its own post; I will post it later).

Qwen3-4b-MaaS-Github

Part 2 - PoC using OpenClaw:

This was an easy enough process. I never deployed OpenClaw on a machine (laptop, workstation etc). I first tested it in a docker container then I deployed it in K8s cluster. 

• I followed the OpenClaw docker install documentation to create my own image. The setup.sh script worked with some tinkering (eg: adding oc cli, python binary, running it as node user etc etc in the Dockerfile). But I was able to generate a image and run it. Later on I tagged the image accordingly to pushed to my quay.io registry.
  
  
• Then I created k8s deployments for it. One thing I needed to tweak there using init-containers is to supply the custom skills. Rather than going into too much details here's the Github location of the yamls for my OpenClaw's K8s deployment.  The README.md explains what does what.
  
  
• In the Github Repo you will a skills-repo folder. This is simple pod that gets the skills from git repo and serves it as tar file over http endpoint. And the OpenClaw deployment's init-containers grabs it from there and unpacks it in its relevant directory (workspace/skills) which OpenClaw then discovers as a skill (custom skill) it has. This way I can develop the skills and update it and OpenClaw gets it via Git repo. 
  
  
• Then I created the skill (eg: k8s-namespace-monitor). To be honest, I did not completely handcode this from scratch, rather, I vibe coded it using Claude sonnet (this is where the power of frontier model came in place). The skill along with instructions to the LLM in the skill.md file has a few bash scripts and py scripts that do some of the heavy lifting tasks like getting the logs, ranking the logs, executing the fix, verifying the fix etc and the LLM instructs OpenClaw (the agent) when and what to execute to complete its task based on the skill.md. This way I am minimising the contextual information the model needs to process (eg: it does not need to know general k8s knowledge, it only needs to know my pods or deployments that may throw errors and the specific error logs. The scripts captures it and gives to LLM). 
  
  
• I configured the OpenClaw to communicate (send messages and receive messages) via Telegram channel (This is restricted to private DM only and bound to my userid).
  
  
• I also added an OpenClaw cronjob to run the skill periodically. 

Here's the Git hub repository. 

The result:

• This proved the hypothesis for building an k8s SRE agent. It took very minimum effort. 
• The OpenClaw agent was monitoring my namespaces (and the apps deployed in it) by examining the logs and if there were issues it would cross match with the KB and proceed to remediate it. This is all by harnessing a small LLM from the MaaS inference endpoint.
• One critical thing the OpenClaw failed to do here is spawing of subagents in K8s environment. This mean scaling agent won't be possible. I read the doc that this capability exists; perhaps I could not get it right.
• Another weird thing it did, as I observed, because I could not exactly control its behavior (skill only gave it direction on what to do) the OpenClaw time to time would halt and ask permission to proceed to next step even though its instruction were to run end to end. But I could easily imagine how it would translate to when I would want to simply scale its monitoring scope. The underlying script doing the heavy lifting would also become complex in that case. It may, very well, become the same effort as writing my own agent.   

Here're a screen shots from the telegram of one of its run:
   

And the before and after of my apps looked like below:

I did noticed some hickups here and there, like the agent would start, send me first few messages from the skill execution but then would get wait for a response (could be anything) from human to nudge it. 

So, yes, although the reliability score wasn't great it proved the theory of a k8s monitor and remediation usecase using agent. May be harnessing a larger LLM would result better --> I have not tried it and do not want to try it because that defeats the guiding principle #1 "using "

Part 3 - Productionise on Kagenti:

The PoC only proved that an agent can solve the usecase. But it has issues and very few ways to resolve it using OpenClaw that makes it kind of unqualified for to productionise it. Below are a few of those:

• Credentials are embedded into the agent and/or skills. There is no way to make it credential less. This is a huge security hole.
  
  
• Scaling (agent scaling) is a big issue. We cannot keep loading skills after skills into it. We can deploy multiple OpenClaw instances but that creates orchestration problem because sub-agents are OpenClaw internal only. 
  
  
• Usage of direct CLI tool instead of MCP looses complete control on governance and maintenance of the tool. I could use MCP instead, but the skill would become complex and scripts will need to be tailored. This is an anti-pattern.
  
  
• The reliability wasn't great as highlighted before. Yes, it works but it needs nudge from human.
  
  
• Everything is tightly coupled (MaaS endpoint, Tools etc) and contained within the agent.
  

Kagenti (now known as Rossoctl) is a set of platform primitives for agent security, resilience, and efficiency that platform owners can build on. It, by design, solves all of the issues above. I describe the how part below. 

Understanding Kagenti:

Below is the overview diagram of Kagenti:

image source: https://github.com/rossoctl/rossoctl

** I did not use Shipwright. I deployed the Agents and Tools as containers/pods via my pre-existing pipeline. I, personally, do not like connecting my githab with many different tools. 

Below diagram shows how Kagenti and a few of its ecosystem components works behind the scene to provide networking and security when agents or tools are deployed on the platform:

Kagenti Operator:

After a Agent or Tool pod is deployed in a Kagenti enabled K8s/OCP cluster I needed to create a CRD: AgentRuntime (type tool, type agent) for it to bring it under Kagenti. Kagenti sees this CRD and does the following things:

• creates SPIRE registration entries for new workloads
• registers OAuth clients, creates credential Secrets
• Injects sidecars these 2 sidecars: 
• spiffe helper:
• fetches SVIDs from SPIRE
• authbridge (envoy proxy):
• for outgoing request it does the below:
• Intercepts outbound call
• Does token exchange using svid with keycloak
• Injects bearer token into the request
• For incoming request it does the below:
• Validates inbound JWT
• check audience matches own SPIFFE
• forward to app if valid otherwise return 401 

Authorino:

Kuadrant ( aka Red Hat Connectivity Link) is also deployed as past of Kagenti deployment. 

• I created a Kuadrant Auth Policy for the MCP Gateway to make the gateway accept connection that has valid JWT token. 
• Kuadrant Authorino does the leg work behind the scene to check the validity of JWT with Keycloak. Authorization is also possible, but I did not implement it here.

Kuadrant MCP Gateway Controller:

• For registering the MCP Tool (K8s MCP Server) I deployed the a CRD: MCPServerRegistration. 
• The Kuadrant MCP Gateway controller reads the CRD and tells the mcp-gateway (Gateway API) about the how to reach (forward tool calls and read tools) to MCP Server.

Here's how I implemented the agents and tools and deployed on Kagenti. 

Skill to LangGraph ReAct: 

First, I converted the skills written for OpenClaw into a LangGraph ReAct code base. It took a few iterations but I was able to convert it into LangGraph ReAct based agent. It wasn't a simple like for like conversion it was improved by a lot because here I could write my own logic in code. I even observed better result from the LLM because here I could control how to do certain things such as: 

• where to delay, 
• where to async, 
• multiple system prompts for different tasks 
• user prompts straight from KB.
• etc

Here's a mermaid graph of it:

The tool to interact with K8s:
I used the official light weight Kubernetes MCP Server. I am discounting direct CLI usage for production for now because it is very had to put governance around it. I had to do a few minor tweaks to it to make it suitable for my usecase fitting into Kagenti. For example: The MCP Server by default uses multi-cluster mode. I had to force it to single cluster. And this was because the auth token the agent was passing (as result of workload identity implementation) the Kubernetes MCP Server will try to use it as user (extracted from Bearer token) as its default behavior. So even thought the MCP Server pod is running as service account it will completely ignore it and try to use the workload identity as user and fail. Then there's a bit more I had to do to secure it. I described it below. Read on.

Here's the Github for this container.

Using the Gateway and Broker:

From the architecture pattern perspective using Gateways (MCP Gateway, AI Gateway etc) achieves: 

• It decouples among the agents, tools, LLM inference. 
• It shifts the routing responsibility to the platform. 
• It adds central control and governance such as 
• tool filtering (important to control what tools should be available to agent)
• tool prefix (important to avoid agent's tool confusion when dealing with multiple tools)
• rate limiting, 
• circuit breaking, 
• retry counter, 
• authentication and autorization for specific tools, 
• endpoints etc. 

This is good. Kagenti leverages the K8s native Gateway implemented by Istio for this. 

The internal mechanics:

Here's a highlevel diagram how the agent works inside the cluster:

Some important highlights:

• All 3 components, the Agent, the MCP Gateway and the MCP Server, all have authentication (and to some extend authorization) baked in. 
• The auth mechanism between the components/workload are passwordless (svid + short lived JWT) via workload identity.
• Workload identity is really important, specially for agentic system, to avoid "confused deputy problem".  

Below diagram shows how everything fits together:

Four-Layer Defense in Depth:

The K8s MCP Server is a very powerful tool. It can do a lot of things (CRUD operation + some) on a K8s cluster. Which also means it can do a lot of damage as well if in wrong hands or wrong LLM tool call based on wrong decisions. So it is super important that we apply proper defense mechanism around it. Below diagram shows the 4 layer defense in depth. Layer 1,2,3 is Kagenti provided using K8s native tools and Layer 4 is pure K8s. The diagram shows MCP Tool POD but this is also true for other workloads; eg: agent pods. This is why in previous diagram I showed malicious workloads getting blocked from accessing the Kagenti workloads (Gateways, Agents, Tools) even from within the cluster. 

That's it. 

• Agent is secured and can get input via A2A protocol. ✓
• MCP tool/server is secured. ✓
• All decoupled. ✓
• Security and governance implemented at the platform level via Gateways. ✓
• Scaling by more agents or by more tools can happen independently. ✓

Here's the output from this agent run:

(it is more precise than that of from OpenClaw)

Here's the Git hub repository. 

What's next:

• Convert the KB into RAG to accommodate large KB.
• Introduce memory to the agent for frequent and adhoc runs to further reduce LLM tokens. The current frequency is hardcoded to 1hr.  
• Make the K8s Namespace Monitor agent and VMs Monitor agent as sub agent orchestrated by another parent agent --> This is will make it a close loop agentic system for ops to manage both VMs and K8s fleet. Kagenti has all the foundational component for it. 

Conclusion:

So, yes, agree with Jensen Huang that every organisation should have an OpenClaw strategy. It certainly helped me to accelerated the PoC path. Then Kagenti took over the production part. Deploying Kagenti and deploying on Kagenti did consume some effort (more that my likings), but it is still alpha. I can see where its headed and its awesome. 

Perhaps, in future off the shelf harnessing agent will ship with security and governance, proper segregation between platform vs workload etc design paradigms or will have integrations with platform like Kagenti to "push to prod".  Until then, hopefully this Github may be able to help.

Happy Agenting... 😀😊😎💪🤖🤖