AI agents are rapidly transforming software engineering—introducing new architectural patterns, terminology, and operational workflows designed to automate repetitive tasks.
One of the most compelling emerging use cases is Agents for Ops.
In this post, I share my journey building a Kubernetes namespace monitoring agent. Think of it as a digital intern performing Level 1 SRE tasks: it monitors K8s workloads and automatically remediates failures strictly within the guardrails of a provided knowledge base—nothing more, nothing less.
I built 2 agents: 1 for K8s namespaces per cluster and 1 for VMs fleet; but to keep this post short I will focus on the k8s-namespace-monitoring-agent.
The Demo:
OpenClaw:
Kagenti:
The usecase:
My usecase is simple:
- The Environment: Two workloads (a web server and a database-backed microservice) running on OpenShift, with a "Chaos Monkey" injecting random failures to simulate real life application failures.
- The Knowledge Base: A mapping of known log patterns to their root causes and remediation actions.
- The Agent's Job: Cross-reference runtime K8s error logs against the KB, interpret the failure, and automatically execute optimized fixes.
To handle this, I built the K8s-namespace-monitor-agent. Its core capabilities and workflow break down into:
- Parse & Match: Reads application logs and cross-references them against the Knowledge Base (KB).
- Translate & Optimize: Converts KB text into executable fix commands, optimizing execution to avoid redundant actions.
- Human Reporting: Summarizes issue context and actions in plain English for operators.
- KB Guardrails: Operates strictly within the scope (namespaces, log types, actions) defined by the KB.
In short: Contextualise information → Orchestrate workflow → Execute via tools
The guiding principle:
The LLM Strategy: "Smart enough" over frontier AI. The agent needs reliable tool-calling and pattern matching, not massive reasoning models. Self-hosting a smaller model keeps inference costs near zero.
PoC-to-Production: Prove reliability early. Because LLMs are non-deterministic, validate tolerable reliability in a quick sandbox PoC and the required LLM size and model before investing heavy engineering effort into production infrastructure.
The Tool Selection:
The LLM:
To keep the inferencing cost to minimum I went with small LLM as opposed to the front tier models such as Claude, GPT, Gemini etc. I chose Qwen3 4b and ran it on OpenShift AI platform using MaaS (model as service). I made this choice based because of the below reasons:- Tool calling capability: Quen3-4b model can make tool calls. I tested this with great success rate.
- The contextualisation: The use case is specific about processing KB and not a broad set K8s operations. The small LLM was able to do it with success threshold (why spend $$$ when I can do it for almost free).
- Token cost & Minimum GPU requirement: Since this small LLM does not require a huge GPU capacity I was able to self host it on RTX 3060 in lab in the PoC phase then moved to a more server grade GPU like L40 as well as T4. This will nicely keep the cost low and it will still leave lots of VRAM space.
- LLM inference: OpenShift AI's platform comes with various GPU accelerator and prebuilt wiring to serve on vLLM; so provisioning MaaS using the qwen3-4b was easy.
There are a bit more work went into this to come to the above realisation which deserves it's own post. I will link that post here shorly.
The PoC:
The next challenge was to decide a quick way of PoCing it. So naturally I looked for "ready-to-go" "off the shelf" agent solution to keep the development effort to bare minimum. The tool choice here was OpenClaw because of the below reasons:
- OpenClaw on K8s in Sandbox mode: OpenClaw can be deployed on K8s cluster in sandbox mode (more on this below) and there are ways to restrict it to my usecases only. The main advantage here is that it is an isolated instance and only has access to stuffs that I give it to.
- Plug and play agent skills: I can create agent skills for my usecases with low effort and get OpenClaw agent to execute tasks based on skill. This is a low effort way to test the hypothesis.
- Ready to go Sandbox: There's a sandbox environment ready to go available (at no cost) from Red Hat at Red Hat Developer Sanbox portal.
While OpenClaw validated the PoC, enterprise production required strict security and platform governance. Kagenti was the right fit for several key reasons:
- Zero Trust Security: Native SPIFFE/SPIRE integration eliminates hardcoded credentials and solves the "confused deputy" vulnerability.
- Kubernetes-Native: Uses standard K8s primitives (Istio, cert-manager, Gateway API etc) for agent security, resilience, efficiency and unified management.
- Centralized Gateways: AI and MCP Gateways abstract LLM inference endpoint and tool calls into single platform endpoints, offloading governance from application code to platform.
- CI/CD Flexibility: Decouples the agent logic from the backend — swap LLM endpoints or tools without refactoring agent code and CI and CD them independently via K8s native way.
- Standardized Protocols: Built-in support for MCP (tooling) and A2A (Agent-to-Agent protocols for future multi-agent fleet orchestration).
The implementation:
It is a 3 part process.
Part 1 - The MaaS:
I used the OpenShift AI's MaaS (model as service) tool to self host Qwen3-4B model. Under the hood it uses vLLM via KServe to expose an endpoint to inference against the LLM. It has its own gateway (backed by istio) called maas-common-gateway which is the default gateway. I used the default. This was easy enough process as long as OpenShift AI 3.0 is available on the cluster (which was a bit complex and deserves its own post; I will post it later).
Source code: Qwen3-4b-MaaS
Part 2 - PoC using OpenClaw:
This was an easy enough process. I never deployed OpenClaw on a machine (laptop, workstation etc). I first tested it in a docker container then I deployed it in K8s cluster.
- Custom Containerization: I packaged OpenClaw with required tooling (oc CLI, Python runtime, non-root permissions; here's the Dockerfile) and pushed it to Quay.io following OpenClaw docker install documentation.
- "Vibe-Coded" Skills: Used Claude Sonnet to write the k8s-namespace-monitor skill. Offloaded heavy tasks (log parsing, ranking, execution) to helper Bash/Python scripts to keep the LLM’s context window lean and focused. Skills source codes: k8s-namespace-monitor.
- Dynamic Skill Provisioning: Implemented a HTTP server pod so K8s init-containers can pull and unpack custom skills on startup—decoupling agent code from skill updates. Here's the Github location of the yamls for my OpenClaw's K8s deployment. The README.md explains what does what.
- Interface & Scheduling: Integrated a private Telegram bot for interactive control and set up an OpenClaw Cronjob for periodic, automated monitoring runs.
Here's the Git hub repository.
Here're a screen shots from the telegram of one of its run:
And the before and after of my apps looked like below:
PoC Results & Key Takeaways:
- Hypothesis Validated: Proved that a small LLM (via MaaS) can reliably monitor K8s workloads and perform KB-driven remediations with minimal effort.
- Scalability Bottlenecks: I could not solve spawning subagents within K8s, limiting horizontal scaling. There was a known issue in the version in used. May be this is resolved in the latest release.
- Occasional Execution Friction: I observed that the agent occasionally halted for human approval, failing to run strictly end-to-end despite explicit prompt instructions.
- Script Overhead: This approach may hit a max threshold to scale and increasing the monitoring scope will make the underlying helper scripts increasingly complex—approaching the same engineering effort as building a custom agent.
So, yes, although the reliability score wasn't great it proved the theory of a k8s monitor and remediation usecase using agent. May be harnessing a larger LLM would result better --> I have not tried it.
Why OpenClaw Isn't Production-Ready (and How Kagenti (now known as Rossoctl) Fixes It):
- Credential Vulnerability: OpenClaw (at my approach and observation was that) requires hardcoding credentials into agents or skills. Kagenti enables credential-less, zero-trust identity.
- Scalability Bottlenecks: Overloading a single instance with skills isn't scalable, and multi-instance OpenClaw creates orchestration headaches. Kagenti manages sub-agent architectures natively.
- Governance & Control: Relying on direct CLI tools bypasses governance. Kagenti enforces MCP-based tool abstraction at the platform layer. The only thing here I needed to do was mapping the CLI capability to the MCP tool.
- Reliability Friction: The reliability wasn't great as highlighted before. Yes, it works but it needs nudge from human.
- Tight Coupling: Everything is tightly coupled (MaaS endpoint, Tools etc) and contained within the agent making maintenance and updates fragile.
Part 3 - Productionise on Kagenti:
Kagenti (now known as Rossoctl) is a set of platform primitives for agent security, resilience, and efficiency that platform owners can build on. It, by design, solves all of the issues above. I describe the how part below.
Understanding Kagenti:
Below is the overview diagram of Kagenti:
image source: https://github.com/rossoctl/rossoctl
** I did not use Shipwright. I deployed the Agents and Tools as containers/pods via my pre-existing pipeline. I, personally, do not like connecting my githab with many different tools.
Below diagram shows how Kagenti and a few of its ecosystem components works behind the scene to provide networking and security when agents or tools are deployed on the platform:
Kagenti Operator:
After a Agent or Tool pod is deployed in a Kagenti enabled K8s/OCP cluster I needed to create a CRD: AgentRuntime (type tool, type agent) for it to bring it under Kagenti. Kagenti sees this CRD and does the following things:
- creates SPIRE registration entries for new workloads
- registers OAuth clients, creates credential Secrets
- Injects sidecars these 2 sidecars:
- spiffe helper:
- fetches SVIDs from SPIRE
- authbridge (envoy proxy):
- for outgoing request it does the below:
- Intercepts outbound call
- Does token exchange using svid with keycloak
- Injects bearer token into the request
- For incoming request it does the below:
- Validates inbound JWT
- check audience matches own SPIFFE
- forward to app if valid otherwise return 401
Authorino:
Kuadrant ( aka Red Hat Connectivity Link) is also deployed as past of Kagenti deployment.
- I created a Kuadrant Auth Policy for the MCP Gateway to make the gateway accept connection that has valid JWT token.
- Kuadrant Authorino does the leg work behind the scene to check the validity of JWT with Keycloak. Authorization is also possible, but I did not implement it here.
Kuadrant MCP Gateway Controller:
- For registering the MCP Tool (K8s MCP Server) I deployed the a CRD: MCPServerRegistration.
- The Kuadrant MCP Gateway controller reads the CRD and tells the mcp-gateway (Gateway API) about the how to reach (forward tool calls and read tools) to MCP Server.
Here's how I implemented the agents and tools and deployed on Kagenti.
Skill.md to LangGraph ReAct:
First, I converted the skills written for OpenClaw into a LangGraph ReAct code base. It took a few iterations but I was able to convert it into LangGraph ReAct based agent. It wasn't a simple like for like conversion it was improved by a lot because here I could write my own logic in code. I even observed better result from the LLM because here I could control how to do certain things such as:
- Async tool calls, delay the execution etc
- tailored system prompts for different tasks.
- LLM user prompts straight from KB.
- By defining precise tasks and prompts, we give the small LLM clear execution guardrails —significantly boosting its success rate despite its inherent non-determinism.
- It resulted (as I observed) in significant improvement on matching logs to KB instructions and LLM executing precise tool calls for remediation. This improved the reliability of the agent significantly.
- Now, I am convinced of the benefits of custom agents built with LangGraph (or similar framework) over enabling off the shelf agents with skills.
Here's a mermaid graph of the Agent:
The tool to interact with K8s:
- Governance First: Need to replace CLI tools with the official lightweight Kubernetes MCP Server to enable platform-level governance such as:
- Filtering: to improve security posture.
- Prefixing: to future proof agent scalability (expanding to more tool servers).
- Gateway: to abstract tool servers (not tools) from agents.
- Auth Alignment: The MCP Server by default uses multi-cluster mode. I forced single-cluster mode so that ZTWIM's workload identity bearer tokens wouldn't override the pod's Service Account authentication and cause auth failures. This is something to be aware of when deploying an MCP tool in Kagenti system.
- Hardened Integration: Applied additional security configurations utilising Kagenti's ecosystem tools to ensure seamless and secure operation. This was very important implementation for productionising MCP servers. I described it below.
Here's the Github for this container.
Using the Gateway and Broker:
From the architecture pattern perspective using Gateways (MCP Gateway, AI Gateway etc) achieves:
- It decouples among the agents, tools, LLM inference.
- It shifts the routing responsibility to the platform.
- It adds central control and governance such as
- tool filtering (important to control what tools should be available to agent)
- tool prefix (important to avoid agent's tool confusion when dealing with multiple tools)
- rate limiting,
- circuit breaking,
- retry counter,
- authentication and autorization for specific tools,
- endpoints etc.
This is good. Kagenti leverages the K8s native Gateway implemented by Istio for this.
The internal mechanics:
Here's a highlevel diagram how the agent works inside the cluster:
Some important highlights:
- All 3 components, the Agent, the MCP Gateway and the MCP Server, all have authentication (and to some extend authorization) baked in.
- The auth mechanism between the components/workload are passwordless (svid + short lived JWT) via workload identity.
- Workload identity is really important, specially for agentic system, to avoid "confused deputy problem".
Below diagram shows how everything fits together:
Four-Layer Defense in Depth:
Given the K8s MCP Server's powerful CRUD capabilities, strict security guardrails are essential to prevent cluster damage from malicious traffic or bad LLM tool calls.
- Layers 1–3 (Kagenti): Leverages K8s-native tools to enforce zero-trust traffic isolation, preventing unauthorized inter-pod access across agents, gateways, and tools.
- Layer 4 (Native K8s): Applies core Kubernetes security controls (RBAC, namespace isolation, and NetworkPolicies).
This multi-layered approach protects the cluster from both compromised workloads and non-deterministic agent decisions.
- Agent is secured and can get input via A2A protocol. ✓
- MCP tool/server is secured. ✓
- All decoupled. ✓
- Security and governance implemented at the platform level via Gateways. ✓
- Scaling by more agents or by more tools can happen independently. ✓
Here's the output from this agent run:
(it is more precise than that of from OpenClaw)
What's next:
- Convert the KB into RAG to accommodate large KB.
- Introduce memory to the agent for frequent and adhoc runs to further reduce LLM tokens. The current frequency is hardcoded to 1hr.
- Make the K8s Namespace Monitor agent and VMs Monitor agent as sub agent orchestrated by another parent agent --> This is will make it a close loop agentic system for ops to manage both VMs and K8s fleet. Kagenti has all the foundational component for it.
Conclusion:
OpenClaw for Rapid PoCs: As Jensen Huang suggested, every organization should have an OpenClaw strategy—in my case, it was essential for quickly proving the concept.
Kagenti for Production: Kagenti delivers the critical security, governance, and platform separation needed for enterprise deployment. It is in Alpha and a huge amount of effort in on going. I am super impressed by what Kagenti's brings to the table for production grade agent and tools platform.
Perhaps, in future off the shelf harnessing agent will ship with security and governance, proper segregation between platform vs workload etc design paradigms or will have integrations with platform like Kagenti to "push to prod".
Until then, hopefully this Github may be able to help.
Happy Agenting... 😀😊😎💪🤖🤖
Comments
Post a Comment