AI Cluster Pre-flight Check#

By Xperf Inc. — part of the ClusterReady automatic cluster and rack validation platform.

AI-powered GPU cluster pre-flight validation skill. Run 26 GPU cluster node readiness checks locally on a bare-metal node. Auto-detects GPU vendor (NVIDIA / AMD) and network type (InfiniBand / RoCE / Ethernet). Validates GPU detection, PCIe topology, RDMA/InfiniBand networking, Docker, IOMMU, NUMA, firewall, BIOS, and more.

Example Prompts#

• "Run a pre-flight check on this node"
• "Is my GPU node ready for training?"
• "Check cluster node health"
• "Validate this node before training"
• "Are my GPUs and network configured correctly?"
• "Run all health checks on this bare-metal node"
• "Check if GPUDirect and RDMA are working"
• "Is IOMMU in passthrough mode?"

When to Use#

• Before running GPU workloads — ensure the node is properly configured
• After provisioning — validate new bare-metal nodes are cluster-ready
• Periodic health monitoring — catch hardware or config drift
• Troubleshooting — run specific checks to diagnose issues

How to Run#

Run all checks:

bash

bash {baseDir}/preflight.sh

Run specific checks only:

bash

PREFLIGHT_CHECKS=1.3,1.4,1.5 bash {baseDir}/preflight.sh

Run in strict mode (no skippable failures):

bash

PREFLIGHT_STRICT=true bash {baseDir}/preflight.sh

Enable cross-node checks:

bash

PREFLIGHT_PEER_IPS=10.0.1.11,10.0.1.12 bash {baseDir}/preflight.sh

Environment Variables#

VariableRequiredDefaultDescriptionPREFLIGHT_NODE_IDNohostnameNode identifier in outputPREFLIGHT_CHECKSNoallComma-separated check subset (e.g. 1.2,1.3,1.5)PREFLIGHT_STRICTNofalseTreat skippable failures as errorsPREFLIGHT_PEER_IPSNo—Comma-separated peer IPs for cross-node checks (1.1, 1.26, mesh ping)MOUNT_POINTNoauto-detectShared filesystem path for check 1.10SWITCH_HOSTNo—Switch hostname for check 1.25SWITCH_CLI_CMDNo—Direct switch CLI command for check 1.25SWITCH_USERNoadminSwitch SSH userSWITCH_SHOW_CMDNoshow interface statusSwitch show command

Output#

JSON to stdout, diagnostics to stderr. Pipe through jq for pretty-printing:

bash

bash {baseDir}/preflight.sh 2>/dev/null | jq .

Exit Codes#

• 0 — All checks passed (or skipped)
• 1 — One or more checks failed
• 2 — Configuration error

Check Catalog#

IDNameScopeDescription1.1SSH ConnectivityCross-nodeTests SSH to peer nodes1.2Docker ServiceLocalVerifies Docker daemon is running1.3GPU DetectionLocalDetects NVIDIA or AMD GPUs1.4GPU Count and ModelLocalReports GPU count and model1.5GPU Driver VersionLocalReports GPU driver version1.6OS and KernelLocalReports OS and kernel version1.7Network TypeLocalDetects InfiniBand, RoCE, or Ethernet1.8IB/NIC PortsLocalChecks network port status1.9GPU Container ToolkitLocalTests GPU access inside Docker1.10Shared FilesystemLocalValidates shared mount read/write1.11GPUDirect Kernel ModuleLocalChecks RDMA peer memory modules1.12IOMMU PassthroughLocalValidates IOMMU in passthrough mode1.13NUMA BalancingLocalReports NUMA balancing setting1.14Firewall DisabledLocalChecks firewall is inactive1.15PCIe Link Speed/WidthLocalValidates PCIe link negotiation1.16PCIe ACS DisabledLocalChecks ACS on PCI bridges1.17GPU-NIC PCIe AffinityLocalReports GPU/NIC topology1.18NIC FirmwareLocalReports NIC firmware version1.19BIOS SettingsLocalValidates BIOS/IPMI settings1.20Link QualityLocalChecks error counters1.21Transceiver HealthLocalChecks optical module health1.22MTU ConfigurationLocalValidates jumbo frames (MTU >= 9000)1.23Fabric TopologyLocalDiscovers IB fabric topology1.24RoCE QoS/NIC ConfigLocalReports QoS and GID configuration1.25Switch QoSCross-nodeValidates switch configuration1.26Network RoutingCross-nodeTests ping to peer nodesmeshL3 Mesh PingCross-nodeTests connectivity to all peers

Interpreting Results#

• pass — Check succeeded
• fail — Check failed (node may not be ready)
• skip — Check failed with a known non-critical signature (e.g. missing optional tool). In strict mode, these become failures.

Checks auto-detect GPU vendor (NVIDIA vs AMD) and network type (InfiniBand vs RoCE vs Ethernet), running the appropriate commands for each.

Support#

• Website: xperf.ai
• Platform: clusterready.xperf.ai
• Contact: techsupport@xperf.ai