Maximise GPU utilisation and self serve GPU as a Service

GPUs are a critical yet expensive commodity for AI workloads. Justifying their high cost requires maximizing utilization.

To keep things in focus, we are not discussing token consumption or tokeneconomics here; This is about GPU optimisation (at the infra and platform).

In this post I will describe an approach to building an optimized GPU-as-a-Service using the below key components:

1. GPU Sharing: Increases workload density and optimising GPU usage.
2. Model As A Service (MaaS): Standardizes and centralizes model deployment and consumption.
3. Kueue: Controls prioritisation, duration and placements of GPU workloads.
4. Self serve: Enables different roles to self serve to utilise GPUs available within the organisation for usecases.

GPU Sharing + MaaS + Kueue + Self-Serve = GPU-as-a-Service

Solution overview:

The big picture:

The above goal translates to below high level overview:

Demonstration of GPU-as-a-Service in action:

Let's dive into the bits now.

GPU Sharing:

GPUs were designed for massive parallel execution of single applications rather than native multi-tenant operating system multitasking. Orchestration tools like Ray only provide logical isolation. A process assigned a fraction of a GPU will still attempt to consume 100% of the physical compute and VRAM. 

Most GPUs these days far outpace the compute and memory demands of lightweight LLMs or low-batch workloads. ML frameworks, vLLM inferencing runtime etc operate under the assumption of exclusive device ownership. Without sharing, up to 90% of the physical silicon sits idle powered on but processing nothing, leading to massive infrastructure bloat and poor cost efficiency.

Initializing an AI workload (on NVIDIA card) creates an isolated software encapsulation called a CUDA context. Standard GPU drivers prevent work from different CUDA contexts from executing concurrently on the same compute engine, locking the hardware to a single process regardless of how little compute capacity it actually uses.

Hence, unlike CPUs, special/different sharing mechanism exist for GPUs. 

The reasons for GPU Sharing is pretty simple.

More efficient use of GPUs + Fewer GPUs per workload = Lower Cost

There are 3 ways (so far) to do GPU Sharing:

MIG (Multi-Instance GPU): 

Hardware-level partitioning available on Ampere+ GPUs (A100, H100). Each partition gets dedicated SMs, memory, and L2 cache — behaving like an independent smaller GPU with fully isolated, deterministic performance. Max 7 instances. Think of it as the opposite of vCPU: MIG trades density for isolation.

Time Slicing: 

Software-level GPU sharing that works on any NVIDIA GPU. Workloads take turns using the full GPU via rapid context-switching — no isolation, shared everything. The GPU equivalent of overcommitted vCPUs: trades isolation for density. Higher concurrency than MIG, but noisy-neighbour risk. In this post this is what I implemented. This sharing technic is available across all GPUs and can be applied on top of MIGs as well.

MPS (Multi-Process Service):

CUDA-level feature (Volta+ GPUs) that lets multiple workloads run on the GPU truly concurrently — not taking turns like time-slicing. A single MPS daemon funnels all CUDA requests so the GPU executes kernels from different processes simultaneously. Better utilisation than time-slicing, but still no memory isolation.

Each options has advantages and tradeoffs and their fit for purpose is uscases basis.

Model As A Service (MaaS):

MaaS is a capability in Red Hat OpenShift AI (RHOAI 3.4+) that turns the OpenShift cluster into an OpenAI-compatible LLM serving platform. 

Under the hood the the LLM is served by vLLM engine, Auth + Rate Limit done by Red Hat Connectivity Link, Kuebernetes Gateway API implemented by Istio, Kserve looking after all the deployed components from LLMInferenceService CRD. 

From a GPU optimisation perspective, MaaS is the workload consolidation layer — instead of 5 teams each claiming a dedicated GPU to run their own copy of the same model, MaaS lets one shared instance serve all of them through a governed API. This directly increases GPU utilisation: fewer duplicate models means fewer GPUs needed, and rate limiting prevents any single consumer from monopolising GPU-backed inference capacity. Combined with GPU Sharing (section 1), MaaS can run multiple different models on a single GPU, turning one expensive card into a shared multi-model serving platform.

It also ticks the boxes of Self Serving capability and governance at the platform level.

Kueue

Kueue, in simple term, is a Kubernetes-native job queueing system. A Red Hat Build of Kueue operator is available in OpenShift via Operator Hub / Eco System. 

When we combine Kueue with GPU Sharing and MaaS it does wonders — it becomes the decision layer that answers "who gets the GPU right now?" by managing a ClusterQueue with a quota correlating to available capacity that is set and designed by cluster admin / at the org level when exposing the GPU(s) as a Service.

Unlike traditional k8s workloads where horizontal scale is convenient (CPU resources are cheap), AI workloads compete for scarce, expensive GPU slots. Without a queuing layer, a training or inferencing job submitted at the wrong time either stays Pending forever or causes OOM kill to itself or worst others — Kueue automates this entirely.

For GPU utilisation, Kueue ensures GPUs are never idle when there is work waiting. When the Job expires or a higher-priority workload arrives, Kueue (via preemption) frees the GPU instantly. This on-demand activation pattern means GPUs utilization is optimised that actually relates to business needs — eliminating the common enterprise problem of "reserved but idle" GPUs.

For self-service, Kueue enables a platform model where users submit GPU consumption requests without needing cluster-admin access or knowledge of GPU availability. Users simply specify what they need (model, batch job, priority, duration) — Kueue handles admission, queuing, and preemption behind the scenes based on the predefined quota and priority rules. This shifts GPU access from "adhoc day2 operation/engineering" to true on-demand self-service.

Here's how I configured Kueue for this usecase:

This is, at a high level, how Kueue is working:

Self Serve UI

I implemented the Self Serve UI using Red Hat Developer Hub (RHDH) — a platform for building fit-for-purpose internal UIs that has built-in form rendering, input processing, and output actions (YAML, PRs, API calls) out of the box that are K8s native. It runs on OpenShift via operator model. This reduces the tech debt of building a bespoke portal.

In the context of GPU-as-a-Service, it is the user-facing layer. Platform team creates a Scaffolder template that presents a simple form that end user uses to consume GPUs — behind the scenes it renders a Kueue Job YAML. The user does not need to know Kubernetes, Kueue, or GPU availability. They fill in a form, click submit, and their GPU workload request is processed. This removes the human and adhoc day2 bottleneck to platform engineers to action GPU consumption requests. Authorised users can request GPU access on-demand, governed by predefined Kueue configuration.

This approach also helps at scale — across multiple clusters. The RHDH interact with Kueue directly or to a single cluster. Instead, a GitOps process loose couple it and places Job to the appropriate spoke clusters. This means a single RHDH UI on the hub can serve GPU requests across an entire fleet of GPU clusters. This reduces further complexity with multi-cluster scenario. The end users requests through the same form targetting different cluster, and the GitOps pipeline places it on the right cluster. This is multi-cluster GPU-as-a-Service via self-serve — portal, across multi-cluster with governance implemented by design.

The technical bits:

Ok. Now that we covered "who is doing what" lets get into their technical details.

• Wiring Time Slicing to NVIDIA GPU Operator:
  Details written here README.md along with configmap and gpu-operator CR.
  
  
• Wiring Model-as-a-Service:
  Details written here README.md along with necessary yamls
  
  
• Configuring Kueue:
  Details written here README.md along with necessary yamls
  
  
• Self Serve UI (using Red Hat Developer Hub):
  Details written here README.md along with necessary yamls
  
  

Conclusion:

It is clear that these technologies at infra and platform leyer exposes capabilities to increase GPU usage but workloads/usecases/application needs to be there to take advantage of it. To be fair, AI workload (LLM referencing, Agents, ML training, Synthetic Data Generation etc) and GPU-as-a-Service goes hand in hand and one is useless without the other. The GPU-as-a-Service capability will certainly help with hardware ROI but finding the balance is the key to achieving the overall ROI (not just hardware ROI but also AI measurably returning value for the business).

That's it. Happy innovating...