Planning a POC? Review the ARO POC Guide for OpenShift AI for scenario planning, infrastructure sizing, and success criteria before starting installation.

Installing Red Hat OpenShift AI 3.x on Azure Red Hat OpenShift AI

Overview

Red Hat OpenShift AI provides a comprehensive platform for developing, training, and deploying AI/ML models on Azure Red Hat OpenShift (ARO). This guide covers deploying OpenShift AI Self-Managed (version 3.4+) on ARO clusters, including configuration for GPU workloads, storage options, and deployment patterns for both connected and egress-restricted environments.

Why this matters: This guide covers OpenShift AI Self-Managed deployment patterns optimized for Azure infrastructure, including:

• GPU Support - NVIDIA GPU Operator configuration for Azure NC/ND-series VMs
• Storage Integration - Azure Disk, Azure Files, OpenShift Data Foundation (ODF), and object storage for pipeline artifacts
• Egress-Restricted Deployments - Leveraging ARO egress lockdown for private clusters
• Model Serving - KServe and ModelMesh for production inference workloads
• Data Science Pipelines - Kubeflow Pipelines integration with Azure Blob Storage or S3-compatible backends

🗺️ Quick Start Navigator

New to OpenShift AI? Choose your deployment path based on your needs:

📋 Deployment Paths Overview

*Estimated Azure costs based on pay-as-you-go pricing

🎯 Quick Links by Scenario

I want to…

• Just try OpenShift AI → Start with Path A: Minimal
• Deploy models to production → Go to Path B: Standard
• Train large ML models with GPUs → Jump to Path C: Full
• Use OpenShift Data Foundation storage → See Part 3: Optional Enhancements
• Deploy in air-gapped/private cluster → Check Part 4: Advanced Topics
• Troubleshoot an issue → Visit Part 5: Troubleshooting

📖 Guide Structure

This guide is organized into 5 parts:

1. Part 1: Foundation - Prerequisites and base OpenShift AI installation (required for all paths)
2. Part 2: Deployment Paths - Choose Path A, B, or C based on your needs
3. Part 3: Optional Enhancements - ODF storage, GPU advanced features, TrustyAI
4. Part 4: Advanced Topics - Egress-restricted deployments, upgrades, uninstall
5. Part 5: Reference & Troubleshooting - Complete matrices, validation, troubleshooting

Part 1: Foundation (Required for All Paths)

This section covers prerequisites and initial setup required regardless of which deployment path you choose.

Prerequisites

Before installing OpenShift AI on ARO, ensure the following requirements are met:

Cluster Requirements

• ARO cluster with supported OpenShift version for OpenShift AI
• Worker node capacity for AI workloads:
  • Minimum 3 worker nodes
  • Recommended: 16+ vCPUs, 64+ GB RAM per node for AI workloads
  • For GPU workloads: NC-series or ND-series VMs (see GPU Support section)
  • High-memory instances (D-series v5 with 8:1 memory-to-vCPU ratio): Cost-effective alternative to GPUs for large model inference and training that can take advantage of increased memory without GPU acceleration
• Cluster admin access to install operators and configure components

Storage Requirements

• Persistent storage configured:
  • Azure Disk CSI driver (default on ARO, supports RWO volumes)
  • For RWX (ReadWriteMany) volumes, choose one of:
    • Azure Files CSI driver (default on ARO)
    • OpenShift Data Foundation (ODF) in Internal mode
• Object storage for Data Science Pipelines (required for pipeline artifacts and data):
  • Azure Blob Storage (recommended for ARO)
  • Azure Data Lake Storage Gen2 (better for large-scale data science workloads)
  • Self-hosted MinIO
  • External S3-compatible providers with proxy solutions
  • Requires: endpoint URL, access key, secret key, container/bucket name

Operator Dependencies

OpenShift AI requires the following operators to be installed manually via OperatorHub before enabling certain features:

• Red Hat OpenShift Serverless Operator - Required for KServe model serving
• Red Hat OpenShift Service Mesh Operator - Required for KServe model serving
• Red Hat OpenShift Pipelines Operator - Required for Data Science Pipelines

Important: The operators themselves must be installed manually via OperatorHub. OpenShift AI then automatically provisions the necessary configuration resources (ServiceMeshControlPlane, KNativeServing) when you enable KServe or Data Science Pipelines in your DataScienceCluster.

Component-to-Operator Dependency Matrix

This table shows which operators are required for each OpenShift AI component:

Operator Installation Summary

Note: ARO clusters may have governance policies that require manual InstallPlan approval even when installPlanApproval: Automatic is set. All operator installation scripts in Step 3 include automatic InstallPlan detection and approval to handle this.

Minimal vs Full Installation

Minimal Installation (Dashboard + Workbenches only):

• Required: OpenShift AI Operator, Service Mesh Operator (auto-installed)
• Components: Dashboard, Workbenches

Standard Installation (Model Serving + Pipelines):

• Required: OpenShift AI, Service Mesh (auto), Serverless, Pipelines, S3-compatible storage
• Components: Dashboard, Workbenches, Data Science Pipelines, KServe, Model Registry

Full Installation (All Features + Training):

• Required: All Standard operators + Cert-Manager, Kueue, JobSet, S3 storage
• Recommended: Leader Worker Set, Custom Metrics Autoscaler, MariaDB (for TrustyAI)
• Optional (GPU): Node Feature Discovery, NVIDIA GPU Operator
• Components: All Standard + Training Operator, Ray, Kueue, TrustyAI

References:

• Installing the Single-Model Serving Platform - Service Mesh and Serverless configuration
• Working with Data Science Pipelines - Pipelines operator requirement

Foundation: Deployment Options

Before starting installation, understand your deployment environment:

Connected Deployment (Standard)

Standard ARO clusters with internet egress can install OpenShift AI directly from Red Hat OperatorHub. This is the simplest deployment method and supports automatic operator updates.

Use Cases:

• Development and testing environments
• Production workloads with internet access
• Azure commercial cloud deployments

Egress-Restricted Deployment

ARO supports egress lockdown for private clusters with zero public IPs. OpenShift AI is compatible with this configuration.

For detailed egress lockdown configuration: See ARO Egress Lockdown Overview

Note: Fully air-gapped ARO is not supported. For air-gapped OpenShift AI, use self-managed OpenShift Container Platform on Azure VMs. See Operating OpenShift in air-gapped environments

Foundation: Installation Steps

Complete Steps 1-2 before choosing your deployment path.

Configuration Steps

1. Verify Cluster Resources

Before installing operators, verify cluster has sufficient resources:

2. Install OpenShift AI Operator

Install the OpenShift AI Self-Managed operator from OperatorHub:

Important Configuration Notes:

• channel: stable-3.x - Uses the Self-Managed 3.x channel (latest compatible 3.x version)
  • DO NOT use channel: stable - This is for older versions, always use stable-3.x or version-specific channels
  • For OpenShift AI 3.3.x specifically, use channel: stable-3.3
  • For latest 3.x (currently 3.4.x), use channel: stable-3.x
• installPlanApproval: Manual - Prevents automatic upgrades, giving you control over version updates
• startingCSV - Optional field to pin to specific operator version
  • Omit this field to get the latest version in the channel
  • Or specify version like rhods-operator.3.3.4 for version pinning
  • If the specified version doesn’t exist in the catalog, installation will fail with “constraints not satisfiable”

3. Install Required Dependent Operators

📚 This is the comprehensive reference section for all 11 operators. If you’re following a deployment path, install only the operators your path requires and use this section for detailed instructions.

Path-based installation guide:

• Path A (Minimal): No additional operators needed — skip to Part 2: Path A
• Path B (Standard): Install operators 3.1, 3.2, 3.4, 3.5, 3.6 — see Part 2: Path B for your streamlined guide
• Path C (Full): Install all operators (3.1-3.11) — see Part 2: Path C for your streamlined guide

Operator Quick Reference:

3.1. Install Red Hat OpenShift Serverless Operator

🔵 Required for: Path B, Path C (KServe model serving)

Note: Do NOT create a KnativeServing resource manually - OpenShift AI creates it automatically when you enable KServe in the DataScienceCluster.

3.2. Install Red Hat OpenShift Pipelines Operator

🔵 Required for: Path B, Path C (Data Science Pipelines)

3.3. Verify Red Hat OpenShift Service Mesh Operator

⚙️ Auto-installed: All paths (KServe model serving)

Note: Service Mesh v3 is automatically installed when you install OpenShift AI operator. Do NOT create a ServiceMeshControlPlane manually - OpenShift AI creates it automatically.

Troubleshooting: If Service Mesh CSV shows “Failed” status with error “OwnNamespace InstallModeType not supported”:

3.4. Install Red Hat Cert-Manager Operator

🔵 Required for: Path B, Path C (certificate management for Training Operator, Kueue, JobSet)

Note: Cert-Manager is a foundational operator required by multiple OpenShift AI components. It automatically creates certificates for secure communication between components.

3.5. Install Kueue Operator

🔵 Required for: Path B, Path C (job queuing and resource management for Training Operator)

Important: Kueue OperatorGroup must have empty spec: {} (AllNamespaces mode). Using targetNamespaces causes installation failure with “OwnNamespace InstallModeType not supported”.

3.6. Install JobSet Operator

🔵 Required for: Path B, Path C (distributed training job management)

3.7. Install Leader Worker Set Operator (Optional)

🟣 Optional for: Path C (advanced distributed inference workloads)

3.8. Install Custom Metrics Autoscaler (KEDA) (Optional)

🟣 Optional for: Path C (event-driven autoscaling for advanced workloads)

3.9. Install MariaDB Operator (Optional)

🟣 Optional for: Path C (TrustyAI database mode - required only if using TrustyAI with database persistence)

Note: MariaDB Operator is from certified-operators catalog, not redhat-operators.

3.10. Install Node Feature Discovery Operator

🔴 Required for: Path C with GPU (GPU hardware detection - mandatory for GPU support)

Important: Do NOT specify a hardcoded image version in the NFD instance - let the operator manage the image version to avoid compatibility issues.

3.11. Install NVIDIA GPU Operator

🔴 Required for: Path C with GPU (NVIDIA GPU drivers and device plugin - mandatory for GPU support)

⚠️ IMPORTANT: Do NOT create a ClusterPolicy until GPU nodes exist in the cluster. Create the operator subscription now, but wait to create ClusterPolicy in the GPU Support section after GPU nodes are provisioned.

Note: ClusterPolicy creation is documented in the GPU Support section. Creating ClusterPolicy before GPU nodes exist causes operator errors.

3.12. Verify All Installed Operators

Run this comprehensive verification script to check all operators:

Part 2: Deployment Paths

Foundation complete? ✅ Now choose your deployment path based on your requirements.

Each path builds on the previous one. You can start with Path A and upgrade to B or C later.

Path A: Minimal Deployment (Dashboard + Workbenches)

⏱️ Time: 30 minutes | 💰 Cost: ~$276/month | 👥 Best for: Learning, development, POCs

What You Get

• ✅ OpenShift AI Dashboard
• ✅ Jupyter Notebooks (Workbenches)
• ✅ Model Registry
• ✅ Basic data science development environment

Prerequisites

• ✅ Foundation Steps 1-2 completed
• ✅ 2-3 worker nodes (minimum: Standard_D8s_v3)
• ✅ Default storage class configured

Installation Steps

Step 1: Verify Operators

Path A only needs 2 operators (already installed):

• ✅ OpenShift AI Operator (Step 2)
• ✅ Service Mesh v3 (auto-installed with OpenShift AI)

Step 2: Configure DataScienceCluster (Minimal)

Create a minimal DataScienceCluster with just Dashboard and Workbenches:

Step 3: Validate Installation

✅ Path A Complete!

What’s Next?

• Create your first Jupyter notebook workbench
• Explore pre-built notebook images
• Ready for production ML? → Upgrade to Path B

Path B: Standard Deployment (+ Model Serving + Pipelines)

⏱️ Time: 60 minutes | 💰 Cost: ~$1,686/month | 👥 Best for: Production ML workflows

What You Get

• ✅ Everything in Path A, plus:
• ✅ KServe Model Serving - Deploy models to production endpoints
• ✅ Data Science Pipelines - Kubeflow Pipelines for ML workflows
• ✅ Training Operator - Distributed training (PyTorch, TensorFlow)
• ✅ Production-ready ML platform

Prerequisites

• ✅ Path A completed OR Foundation Steps 1-2
• ✅ 3-4 worker nodes (recommended: Standard_D16s_v3)
• ✅ S3-compatible object storage (Azure Blob, MinIO, or AWS S3)

Installation Steps

Step 1: Install Additional Operators

Path B requires 6 operators. Install these from Step 3 :

Follow the detailed installation instructions in Step 3 for each operator, then return here.

Step 2: Configure Object Storage

Data Science Pipelines requires S3-compatible object storage. Choose one option from Step 5 :

• Option A: Azure Blob Storage + MinIO gateway
• Option B: Self-hosted MinIO
• Option C: External S3 (AWS S3 or compatible)

Step 3: Configure DataScienceCluster (Standard)

Enable model serving and pipelines:

Step 4: Validate Installation

✅ Path B Complete!

What’s Next?

• Deploy your first model with KServe
• Create a Data Science Pipeline
• Need GPU training? → Upgrade to Path C

Path C: Full Deployment (+ GPU + Advanced Features)

⏱️ Time: 2 hours | 💰 Cost: ~$2,454/month | 👥 Best for: Large-scale ML platform with GPU training

What You Get

• ✅ Everything in Path B, plus:
• ✅ GPU-accelerated training - NVIDIA T4, V100, or A100 GPUs
• ✅ Distributed training - Multi-GPU training jobs
• ✅ Ray framework - Distributed compute for large workloads
• ✅ Advanced autoscaling - KEDA for event-driven scaling
• ✅ Complete ML platform - All OpenShift AI features

Prerequisites

• ✅ Path B completed OR Foundation Steps 1-2 + Path B operators
• ✅ 5+ worker nodes (CPU + GPU workers)
• ✅ Azure GPU quota - Request quota for NC/ND-series VMs
• ✅ GPU workers - NC8as_T4_v3 or better (see GPU Support section)

Installation Steps

Step 1: Create GPU Worker Nodes

Create GPU-enabled MachineSets. See GPU Support section for detailed instructions:

Step 2: Install GPU Operators

Install NFD and GPU Operator from Step 3 :

Step 3: Install Optional Operators (if needed)

Step 4: Verify GPU Setup

Step 5: Configure DataScienceCluster (Full)

Enable all components including Ray and distributed training:

Step 6: Validate Full Installation

✅ Path C Complete!

What’s Next?

• Deploy GPU-accelerated training jobs
• Use Ray for distributed compute
• Configure GPU time-slicing → Part 3: Optional Enhancements
• Set up Multi-Instance GPU (MIG) for A100s

Part 2 Summary

You’ve completed one of the deployment paths! Here’s what each path enabled:

Continue to:

• Part 3: Optional Enhancements - ODF storage, GPU advanced features
• Part 5: Validation & Troubleshooting - Verify your installation

4. Configure DataScienceCluster

Note: If you followed a Path above, you’ve already configured DataScienceCluster. This section provides the reference configuration.

Create a DataScienceCluster resource to enable OpenShift AI components:

Component Management States:

• Managed - Operator installs and manages the component
• Removed - Operator actively removes the component if present

Customization Strategy:

1. Start with minimal components (dashboard, workbenches, kserve)
2. Add components incrementally based on workload requirements
3. Removed state reduces resource consumption and attack surface

5. Configure Object Storage for Data Science Pipelines

Data Science Pipelines require object storage for storing pipeline artifacts, datasets, and models. Configure one of the following options:

Option A: Azure Blob Storage (Recommended for ARO)

Step 1: Create Azure Storage Account and Container

Step 2: Create OpenShift Secret

⚠️ Note: Azure Blob Storage does not natively support S3 API. For S3-compatible access required by some Data Science Pipeline components, consider:

• Using Azure Data Lake Storage Gen2 with S3-compatible proxy
• Deploying MinIO as S3-compatible layer (see Option B)
• Using external S3-compatible providers (see Option C)

Reference: Azure Blob Storage Documentation

Option B: Azure Data Lake Storage Gen2 (for Data Science Workloads)

Azure Data Lake Storage Gen2 is built on Blob Storage but adds hierarchical namespace, making it ideal for data science and analytics workloads.

Advantages for Data Science:

• Optimized for big data analytics (Spark, Synapse integration)
• Folder-like semantics for organizing datasets
• Better performance for large-scale data processing
• Native integration with Azure ML and analytics services

Option C: Self-Hosted MinIO (S3-Compatible)

Step 1: Deploy MinIO on OpenShift

Step 2: Create Bucket and Credentials

⚠️ Production Considerations for MinIO:

• Use distributed mode (4+ nodes) for high availability
• Configure TLS certificates properly
• Use strong credentials (not default minio/minio123)
• Implement backup and disaster recovery
• Consider managed alternatives for production

Option D: External S3-Compatible Providers

For AWS S3, MinIO Cloud, or other S3-compatible providers:

Common S3-Compatible Endpoints:

• AWS S3: https://s3.<region>.amazonaws.com
• MinIO Cloud: Provided by your MinIO Cloud account
• Other providers: Check provider documentation

6. Deploy OpenShift Data Foundation (Optional, for Advanced Storage)

OpenShift Data Foundation (ODF) provides unified storage for block, file, and object storage on ARO. Unlike basic CSI drivers, ODF is particularly valuable for OpenShift AI because it delivers all three storage types from a single platform, optimized for AI/ML workloads.

Why ODF Matters for OpenShift AI

Different OpenShift AI components require different storage types. ODF provides superior performance and native compatibility for many AI/ML scenarios:

ODF Storage Classes for OpenShift AI

ODF creates three storage classes, each serving specific AI/ML use cases:

1. ocs-storagecluster-ceph-rbd (Block Storage - RWO)

• Use Cases: Individual Jupyter notebooks, single-pod training jobs, GPU workload storage
• Performance: 10,000-100,000 IOPS, 500-2000 MB/s throughput
• Advantages over Azure Disk: Faster provisioning (no Azure API calls), snapshots for ML experiments

2. ocs-storagecluster-cephfs (File Storage - RWX)

• Use Cases: Shared datasets (ImageNet, COCO), model registries, distributed training checkpoints, team collaboration
• Performance: 1-2 GB/s per client for sequential reads, 5,000-50,000 IOPS
• Advantages over Azure Files: 5-10x faster for large files, better concurrency, data stays within cluster, optimized for container workloads

3. openshift-storage.noobaa.io (Object Storage - S3-compatible)

• Use Cases: Data Science Pipelines artifacts, MLflow model registry, dataset staging, immutable model versioning
• Performance: S3-compatible API with low latency (in-cluster)
• Advantages over Azure Blob: Native S3 API (no S3Proxy needed), Kubernetes-native, multi-cloud federation

When to Use ODF vs. Azure-Native Storage

✅ Use ODF when:

• Multiple AI teams sharing infrastructure
• Need RWX storage for shared datasets > 100GB
• Running Data Science Pipelines (requires S3-compatible storage)
• Distributed training with multi-GPU checkpoint sharing
• High-performance file I/O for large datasets (TB-scale)
• Data stays within cluster (no egress to external storage)
• Prefer Kubernetes-native storage management
• Need both RWX file storage AND S3-compatible object storage

✅ Use Azure-native storage when:

• Single-user or small team (< 5 data scientists)
• Mostly RWO workloads (individual notebooks only)
• Small RWX needs < 100GB (Azure Files sufficient)
• Minimal operational overhead (no storage cluster to manage)
• Leveraging Azure-native services (Azure ML, Synapse)

Prerequisites

Before deploying ODF:

• 3 additional worker nodes (recommended: one per Azure availability zone)
• Minimum: 16 vCPUs, 64 GB RAM per ODF node
• Azure Premium Managed Disks (managed-csi storage class)
• Dedicated nodes for storage isolation (best practice)

Step 1: Provision ODF Nodes

Note: ARO uses MachineSets, not az aro machinepool (which doesn’t exist).

Step 2: Install ODF Operators

Step 3: Create Storage Cluster

Expected Storage Classes:

• ocs-storagecluster-ceph-rbd - Block storage (RWO) - Individual workbenches, GPU training jobs
• ocs-storagecluster-cephfs - File storage (RWX) - Shared datasets, model registries, distributed training
• openshift-storage.noobaa.io - Object storage (S3-compatible) - Data Science Pipelines, MLflow artifacts

ODF Integration with OpenShift AI Components

1. Shared Dataset Example (CephFS - RWX)

Benefits: 10+ data scientists access same dataset, 5-10x faster than Azure Files for large files, data remains in-cluster.

2. Data Science Pipelines with ODF NooBaa (S3-compatible)

Benefits: Native S3 API (no MinIO deployment needed), in-cluster storage (lower latency), integrated with ODF monitoring.

3. Model Registry for KServe (CephFS - RWX)

Benefits: Multiple model server replicas access same PVC, fast model loading from CephFS, model updates visible to all pods immediately.

4. Distributed Training Checkpoints (CephFS - RWX)

Benefits: All training pods (master + 4 workers) write/read checkpoints to shared storage, fast recovery on pod failure, CephFS optimized for parallel I/O.

ODF Performance for AI Workloads

Read Performance (Sequential - Large Files):

• ODF CephFS: 1-2 GB/s per client - Loading ImageNet, model files
• Azure Files Premium: 100-400 MB/s per share - Shared file access
• Azure Disk Premium: 200-900 MB/s - Single-node training

Write Performance (Checkpointing):

• ODF RBD (block): 500-2000 MB/s - Single-GPU training checkpoints
• ODF CephFS: 500-1500 MB/s - Multi-GPU distributed training
• Azure Disk Premium: 250-900 MB/s - Single-node training

IOPS (Random Access):

• ODF RBD: 10,000-100,000 - Database-like workloads, metadata
• ODF CephFS: 5,000-50,000 - Parallel file access
• Azure Disk Premium: 5,000-20,000 - General workloads

ODF Operational Considerations

Ongoing Maintenance:

Capacity Planning:

• Expect 30-40% storage overhead for Ceph metadata and 3x replication
• Plan for 20-30% growth per quarter for AI workloads (datasets accumulate)
• Reserve 15-20% free space for Ceph rebalancing operations
• Example: 3 nodes × 512GB disks = 1,536GB raw → ~512GB usable after replication

High Availability:

• Data replicated 3x across availability zones
• Self-healing: Automatic recovery from node failures
• Rolling updates: No downtime for ODF component upgrades
• Disaster recovery: Snapshots and backups for datasets/models

Scaling ODF:

ODF Deployment Architecture for OpenShift AI

Recommended Setup:

Why Dedicated ODF Nodes:

• Isolation - Storage performance not impacted by AI workload spikes
• Taints - Prevents AI pods from scheduling on storage nodes
• Predictability - Guaranteed resources for storage operations
• High Availability - One node per Azure availability zone ensures data resilience

Reference: Configure ARO with OpenShift Data Foundation

7. Verify Installation

8. Upgrade OpenShift AI

Self-managed deployments require manual upgrade approval:

⚠️ Important: New components added in upgrades are NOT automatically enabled in the DataScienceCluster. Manually update the DataScienceCluster CR to enable new components.

9. Uninstall OpenShift AI

Complete removal procedure:

GPU Support

OpenShift AI workloads benefit significantly from GPU acceleration. ARO supports NVIDIA GPUs via Azure NC-series and ND-series VMs.

Supported Azure GPU VM Sizes

Important Notes:

• Azure quota is per-core - request quota in multiples matching VM size
• NC24ads_A100_v4 and above require Generation 2 VM images
• GPU provisioning takes 10-15 minutes

Reference: Use GPU workloads with ARO

Create GPU MachineSet

Important: ARO uses OpenShift MachineSets (not Azure CLI commands) to create worker nodes. The az aro machinepool create command does not exist.

This procedure creates a GPU-enabled MachineSet based on an existing worker MachineSet:

Alternative: Manual YAML Creation

If you prefer to create the MachineSet manually, use this template:

Important Notes:

• Replace <cluster-name>, <region>, and <zone> with your values
• For A100 VMs (NC24ads_A100_v4+), you must use Generation 2 VM images
  • Check image SKU: az vm image list --offer aro4 --publisher azureopenshift -o table
  • Use SKU with -v2 suffix (e.g., v410-v2 instead of aro_410)
• Verify Azure GPU quota before creating MachineSet (see Common Pitfalls section)

Reference: Use GPU workloads with ARO

Install Node Feature Discovery (NFD)

NFD detects hardware features (including GPUs) and labels nodes accordingly:

Install NVIDIA GPU Operator

Verify GPU Setup

GPU Time-Slicing Configuration (T4, V100, L4)

GPU time-slicing allows multiple workloads to share a single GPU by time-multiplexing. Useful for development and inference workloads.

⚠️ Time-Slicing Limitations:

• Does not isolate GPU memory between workloads
• Performance degrades with concurrent workloads
• Best for inference, development, not training
• A100 GPUs should use MIG instead

Multi-Instance GPU (MIG) for A100

MIG partitions a single A100 GPU into multiple isolated instances with dedicated memory and compute resources.

MIG Profiles for A100 80GB:

• 1g.10gb: 7 instances (1/7 compute, 10GB memory each)
• 2g.20gb: 3 instances (2/7 compute, 20GB memory each)
• 3g.40gb: 2 instances (3/7 compute, 40GB memory each)
• 7g.80gb: 1 instance (full GPU)

Reference: NVIDIA MIG User Guide

Validation

Pre-Deployment Validation

Before installing OpenShift AI, verify prerequisites:

Post-Deployment Validation

After installing OpenShift AI, verify all components:

Data Science Pipeline Validation

Model Serving Validation

Common Pitfalls

1. Operator Dependency Not Installed

Problem: KServe or Data Science Pipelines fail to initialize because required operators (Serverless, Service Mesh, Pipelines) are not installed.

Example/Impact:

• DataScienceCluster shows KServe component in error state
• Dashboard shows “KServe requires OpenShift Serverless” warning
• InferenceService creation fails with validation errors

Solution:

Prevention:

• Install all prerequisite operators before enabling KServe/Pipelines in DataScienceCluster
• Use installPlanApproval: Automatic for dependency operators to ensure timely updates
• Monitor operator health in openshift-operators namespace

Reference: Installing the Single-Model Serving Platform

2. Storage Class Not Configured for Notebooks

Problem: Data Science workbenches (Jupyter notebooks) fail to start because default storage class is missing or incorrect.

Example/Impact:

• PVC remains in Pending state
• Notebook pod shows FailedScheduling event: “persistentvolumeclaim not found”
• Dashboard shows “PVC provisioning failed” error

Solution:

Prevention:

• Validate storage class before OpenShift AI installation
• For shared datasets requiring RWX access:
  • Deploy ODF for internal RWX storage (recommended for production)
  • Use Azure Files CSI driver (default on ARO, suitable for small workloads)
• Document required storage class in project onboarding

3. GPU Driver Version Incompatibility

Problem: GPU workloads fail to start or crash due to driver/CUDA version mismatches.

Example/Impact:

• Pod logs show CUDA driver version is insufficient for CUDA runtime version
• GPU not detected by containers despite nvidia.com/gpu resource request
• nvidia-smi command fails inside containers

Solution:

Prevention:

• Pin GPU operator version in production
• Test GPU workloads in development cluster before production deployment
• Maintain inventory of driver and CUDA version compatibility matrix
• Use NVIDIA container images from nvcr.io with known compatible CUDA versions

Reference: NVIDIA GPU Operator on OpenShift

4. Insufficient GPU Quota

Problem: GPU MachineSet creation fails due to Azure regional quota limits.

Example/Impact:

• Machine creation fails with “QuotaExceeded”, “SkuNotAvailable”, or “OperationNotAllowed” error
• Error message: “Current Limit: 0, Current Usage: 0, Additional Required: 8”
• GPU nodes stuck in Pending or Failed state
• MachineSet shows machines that won’t provision

Solution:

Prevention:

• Request GPU quota early in project planning phase
• Start with smaller GPU VM sizes for development (NC4as_T4_v3)
• Use time-slicing to maximize GPU utilization before scaling
• Monitor quota usage in Azure portal

5. ODF Storage Cluster Not Ready

Problem: OpenShift Data Foundation storage cluster fails to reach healthy state, blocking RWX storage provisioning.

Example/Impact:

• ODF pods in CrashLoopBackOff or Pending state
• ocs-storagecluster-cephfs storage class not created
• Notebooks requiring RWX volumes fail to start

Solution:

Prevention:

• Provision ODF on dedicated nodes with sufficient resources (16 vCPU, 64 GB RAM minimum)
• Deploy one ODF node per Azure availability zone for high availability
• Use Azure Premium Managed Disks (managed-csi storage class)
• Monitor ODF health proactively: oc get storagecluster -n openshift-storage

6. Azure Blob Storage Not S3-Compatible

Problem: Data Science Pipelines fail to connect to Azure Blob Storage due to missing S3 API compatibility.

Example/Impact:

• Pipeline runs fail with “S3 connection error”
• Artifact storage shows “Access Denied” or “Invalid endpoint”
• Pipeline server logs show “boto3.exceptions.NoCredentialsError”

Solution:

Option A: Use Azure Data Lake Storage Gen2 + S3-compatible proxy

Option B: Deploy MinIO as S3-compatible layer

Option C: Use external S3-compatible service

Prevention:

• Plan object storage strategy early in deployment
• For ARO, prefer MinIO (self-hosted S3-compatible) or external S3 providers
• Azure Data Lake Storage Gen2 is better for data science workloads than plain Blob Storage
• Document storage architecture in deployment guide

Reference: Azure Blob Storage and S3 API Compatibility

7. KServe Ingress Certificate Issues

Problem: Model serving endpoints fail with SSL/TLS certificate errors.

Example/Impact:

• InferenceService shows Ready=False with certificate error
• Inference requests fail with “SSL certificate verification failed”
• curl to model endpoint returns certificate errors

Solution:

Prevention:

• Configure certificate strategy in DataScienceCluster from initial deployment
• Use OpenshiftDefaultIngress for production to leverage ARO’s managed certificates
• Test model serving endpoints with SSL verification enabled
• Document certificate renewal procedures

8. Network Policy Blocking Model Traffic

Problem: Multi-tenant environments with NetworkPolicies block traffic between data science projects and model serving endpoints.

Example/Impact:

• Applications cannot reach InferenceService endpoints
• Curl from pod to model URL times out
• Service mesh shows connection refused errors

Solution:

Prevention:

• Design NetworkPolicy strategy before multi-tenant deployment
• Create template NetworkPolicies for data science projects
• Test model serving connectivity in isolated namespace
• Document required NetworkPolicy rules in project onboarding

Related Sections

• Cluster Sizing - Worker node sizing for AI workloads, GPU machine pool configuration
• Storage Configuration - Azure Disk, Azure Files, ODF deployment, object storage
• Networking - Service mesh configuration, ingress routes for model endpoints, egress lockdown
• IAM Configuration - Managed identities for workload identity, Azure permissions for storage
• Monitoring - GPU metrics collection, model serving observability
• Security/Compliance - Pod security standards, network policies for multi-tenant AI projects

Additional Resources

Official Documentation

• Red Hat OpenShift AI Self-Managed 3.4 Documentation
• Installing and Uninstalling OpenShift AI Self-Managed
• Installing and Managing OpenShift AI Components

GPU and Hardware Acceleration

• Use GPU workloads with ARO
• ARO with NVIDIA GPU Workloads - Red Hat Cloud Experts
• NVIDIA GPU Operator on OpenShift
• Node Feature Discovery Operator
• NVIDIA Multi-Instance GPU (MIG) User Guide
• GPU Time-Slicing Configuration

Storage

• Configure ARO with OpenShift Data Foundation
• Deploying ODF using Microsoft Azure
• Azure Disk CSI Driver
• Azure Files CSI Driver
• Azure Blob Storage Documentation
• Azure Data Lake Storage Gen2
• MinIO Object Storage

ARO-Specific

• ARO Egress Lockdown Overview
• Create Private ARO Cluster
• ARO Documentation
• Azure Red Hat OpenShift - Red Hat Cloud Experts

Disconnected Deployments

• How to operate OpenShift in air-gapped environments
• Simplify OpenShift installation in air-gapped environments
• Deploy GPU Operators in disconnected environment

Community Resources

• Red Hat MOBB Guides - Validated field content for OpenShift
• OpenShift AI GitHub
• KServe Documentation