CCE Cluster Upgrade Planner#

Overview#

Plan and assess CCE (Cloud Container Engine) Kubernetes cluster version upgrades using hcloud CLI. Covers upgrade path validation, 76-item pre-check checklist, addon compatibility, version-specific breaking changes, deprecated APIs, upgrade window estimation, and execution preview with two-step confirmation.

Architecture: hcloud CLI → CCE OpenAPI → Cluster Info / Upgrade Paths / Upgrade Workflow / Addon Info / Node Pool Info

Nodes

Standard workflow:

text

1. Collect cluster current state (version, nodes, addons, node pools)
2. Query upgrade paths via ListClusterUpgradePaths
3. Run pre-check via ShowClusterUpgradeInfo (or CreateUpgradeWorkFlow)
4. Evaluate addon compatibility for each installed addon
5. Estimate upgrade window based on node count, addon count, batch strategy
6. Generate upgrade plan with specific hcloud CLI commands (execution preview)
7. Provide rollback strategy and post-upgrade verification checklist

Prerequisites#

Prerequisite check: hcloud (KooCLI) >= 7.2.2 required Run hcloud version to verify, and hcloud configure list to check profile exists.

bash

hcloud version
hcloud configure list

Security Constraints#

Dangerous Operation Confirmation Mechanism#

This skill strictly enforces a two-step confirmation mechanism for upgrade execution preview.

All upgrade execution previews require explicit user confirmation before any changes are made. The process:

Step 1: Preview — Show upgrade commands, target version, affected components, and risk warnings

Step 2: Confirm & Execute — Only after user explicitly confirms

Operations Requiring Confirmation

OperationRisk LevelDescriptionUpgradeCluster🔴 CriticalUpgrades Kubernetes control plane, irreversible once startedUpgradeNodePool🟠 HighUpgrades node pool Kubernetes version, nodes become temporarily unschedulableCreateUpgradeWorkFlow🟠 HighCreates upgrade workflow with pre-check, cluster upgrade, and post-check phases

Credential Security#

• Never expose AK/SK values in conversation, commands, or output
• Never ask user to input AK/SK directly in conversation
• Only use hcloud configure list to check credential status (presence only, not values)
• Prefer profile mode or environment variables over explicit AK/SK parameters

Command Format Standard#

CCE upgrade follows standard hcloud format:

bash

hcloud CCE <Operation> --param=value --cli-region=<region> --cli-output=json

Upgrade-specific Parameter Rules#

1. Cluster ID required: all upgrade operations need --cluster_id
2. Target version required: upgrade operations need --spec.clusterUpgradeAction.targetVersion=v1.XX
3. Addon upgrade uses array format: --spec.clusterUpgradeAction.addons.1.addonTemplateName=<name> --spec.clusterUpgradeAction.addons.1.version=<ver>
4. Node pool priority uses key-value format: --spec.clusterUpgradeAction.nodePoolOrder.key1=value1
5. Node selector uses nested format: --spec.clusterUpgradeAction.nodeOrder.key1.1.nodeSelector.key=<label-key> --spec.clusterUpgradeAction.nodeOrder.key1.1.nodeSelector.operator=In --spec.clusterUpgradeAction.nodeOrder.key1.1.nodeSelector.value.1=<val>
6. Upgrade strategy: --spec.clusterUpgradeAction.strategy.type=inPlaceRollingUpdate (only in-place supported)
7. Batch size: --spec.clusterUpgradeAction.strategy.inPlaceRollingUpdate.userDefinedStep=<1-40> (default 20, recommended)
8. Batch scope: --spec.clusterUpgradeAction.strategy.inPlaceRollingUpdate.scope=Cluster or NodePool

⚠️ Critical: Before constructing any upgrade command, always run hcloud CCE <Operation> --help to verify exact parameter names. CCE upgrade APIs have hundreds of parameters; the help output is the authoritative source.

Scenario Routing#

User IntentReference DocumentFull upgrade assessment (7-step workflow)references/upgrade-workflow.mdPre-upgrade checklist (76 items)references/pre-upgrade-checklist.mdAddon compatibility matrix & upgrade orderreferences/addon-compatibility.mdK8s version upgrade path rulesreferences/k8s-version-matrix.mdUpgrade window time estimationreferences/upgrade-window-estimation.mdVersion-specific breaking changesreferences/pre-upgrade-checklist.mdDeprecated API migrationreferences/pre-upgrade-checklist.mdRisk constraints & rollbackreferences/risk-rules.mdOutput schemareferences/output-schema.md

Core Commands#

Step 1: Collect Cluster Current State#

bash

# Get cluster details (version, node count, addons)
hcloud CCE ShowCluster --cluster_id=<cluster-id> --detail=true --cli-region=<region> --cli-output=json

# List installed addons
hcloud CCE ListAddonInstances --cluster_id=<cluster-id> --cli-region=<region> --cli-output=json

# List node pools
hcloud CCE ListNodePools --cluster_id=<cluster-id> --cli-region=<region> --cli-output=json

# List nodes
hcloud CCE ListNodes --cluster_id=<cluster-id> --cli-region=<region> --cli-output=json

Step 2: Query Upgrade Paths#

bash

# Get all available upgrade paths for the cluster
hcloud CCE ListClusterUpgradePaths --cli-region=<region> --cli-output=json

Step 3: Pre-Upgrade Check#

bash

# Get upgrade info (includes pre-check status)
hcloud CCE ShowClusterUpgradeInfo --cluster_id=<cluster-id> --cli-region=<region> --cli-output=json

# Create upgrade workflow (triggers pre-check automatically)
hcloud CCE CreateUpgradeWorkFlow \
  --cluster_id=<cluster-id> \
  --apiVersion=v3 \
  --kind=WorkFlowTask \
  --spec.targetVersion=<target-version> \
  --spec.clusterVersion=<current-version> \
  --cli-region=<region> \
  --cli-output=json

Step 4: Addon Compatibility Check#

bash

# Get addon details (check version compatibility)
hcloud CCE ListAddonInstances --cluster_id=<cluster-id> --cli-region=<region> --cli-output=json

# List addon template versions (check available versions for target K8s)
hcloud CCE ListAddonTemplates --cluster_id=<cluster-id> --addon_template_name=<addon-name> --cli-region=<region> --cli-output=json

Step 5: Upgrade Execution Preview#

bash

# Preview cluster upgrade (without confirm=true, no execution)
hcloud CCE UpgradeCluster \
  --cluster_id=<cluster-id> \
  --metadata.apiVersion=v3 \
  --metadata.kind=UpgradeTask \
  --spec.clusterUpgradeAction.targetVersion=<target-version> \
  --spec.clusterUpgradeAction.strategy.type=inPlaceRollingUpdate \
  --spec.clusterUpgradeAction.strategy.inPlaceRollingUpdate.userDefinedStep=20 \
  --spec.clusterUpgradeAction.strategy.inPlaceRollingUpdate.scope=Cluster \
  --cli-region=<region> \
  --cli-output=json

With addons (specify addon upgrade during cluster upgrade):

bash

hcloud CCE UpgradeCluster \
  --cluster_id=<cluster-id> \
  --metadata.apiVersion=v3 \
  --metadata.kind=UpgradeTask \
  --spec.clusterUpgradeAction.targetVersion=<target-version> \
  --spec.clusterUpgradeAction.strategy.type=inPlaceRollingUpdate \
  --spec.clusterUpgradeAction.strategy.inPlaceRollingUpdate.userDefinedStep=20 \
  --spec.clusterUpgradeAction.addons.1.addonTemplateName=coredns \
  --spec.clusterUpgradeAction.addons.1.operation=patch \
  --spec.clusterUpgradeAction.addons.1.version=<target-addon-version> \
  --cli-region=<region> \
  --cli-output=json

With node pool priority (control upgrade order):

bash

hcloud CCE UpgradeCluster \
  --cluster_id=<cluster-id> \
  --metadata.apiVersion=v3 \
  --metadata.kind=UpgradeTask \
  --spec.clusterUpgradeAction.targetVersion=<target-version> \
  --spec.clusterUpgradeAction.nodePoolOrder.<nodepool-id>=<priority> \
  --cli-region=<region> \
  --cli-output=json

Step 6: Monitor Upgrade Progress#

bash

# Check upgrade workflow status
hcloud CCE ShowUpgradeWorkFlow \
  --cluster_id=<cluster-id> \
  --upgrade_workflow_id=<workflow-id> \
  --cli-region=<region> \
  --cli-output=json

# List upgrade tasks
hcloud CCE ListUpgradeClusterTasks --cluster_id=<cluster-id> --cli-region=<region> --cli-output=json

# Pause upgrade (if issues found)
hcloud CCE PauseUpgradeClusterTask --cluster_id=<cluster-id> --cli-region=<region> --cli-output=json

# Continue paused upgrade
hcloud CCE ContinueUpgradeClusterTask --cluster_id=<cluster-id> --cli-region=<region> --cli-output=json

# Retry failed upgrade
hcloud CCE RetryUpgradeClusterTask --cluster_id=<cluster-id> --cli-region=<region> --cli-output=json

Step 7: Cancel Upgrade Workflow#

bash

# Cancel upgrade workflow (status must not be Running/Success/Cancel)
hcloud CCE UpgradeWorkFlowUpdate \
  --cluster_id=<cluster-id> \
  --upgrade_workflow_id=<workflow-id> \
  --status.phase=Cancel \
  --cli-region=<region> \
  --cli-output=json

Upgrade Window Estimation Formula#

See references/upgrade-window-estimation.md for detailed formula and examples.

Quick estimation:

text

T_total = T_control_plane + T_node_batch + T_addon + T_verify + T_buffer

T_control_plane = 10-30 minutes
T_node_batch = N_nodes / batch_size * (5-15 min/node)
T_addon = count(addons) * (5-15 min/addon)
T_verify = 30 minutes
T_buffer = 20% * (T_control_plane + T_node_batch + T_addon)

Example: 3-node-pool cluster (10 nodes, 5 addons) upgrading v1.23→v1.25:

• Control plane: 15 min
• Node batches: 10/4 = 3 batches (1+4+5 nodes), ~9 min each = 9 min total (first batch 1 node at 5 min, second 4 nodes at 8 min, third 5 nodes at 10 min)
• Addon upgrade: 5 * 10 min = 50 min
• Verification: 30 min
• Buffer (20%): ~19 min
• Total: ~123 min (2h 3min)

Upgrade Methods#

MethodAdvantagesConstraintsRecommendedIn-place upgradeBusiness pods not interrupted during control plane upgrade; nodes upgraded in batches; addons auto-upgradedNodes temporarily unschedulable during upgrade; Docker→Containerd runtime switch needed for v1.27+Most scenarios (default)MigrationClean slate; no compatibility risk accumulation; skip multiple intermediate upgradesFull workload redeployment; requires double resources; longer downtimeCross-version jumps (e.g. v1.15→v1.28); incompatible runtime

Key Constraints#

• No skip-version upgrades: must follow upgrade path step by step (e.g., v1.21→v1.23→v1.25→v1.27→v1.28)
• Patch version first: upgrade to latest patch before major version upgrade
• Control plane before nodes: upgrade control plane first, then node pools
• Addons after cluster: upgrade addons after cluster reaches target version
• v1.28+ control node IP change: upgrading to v1.28+ creates new control nodes, IP addresses change
• Autoscaling paused: during control plane upgrade, autoscaling is paused; resumes after control plane done, but shrink waits until full completion

Parameter Reference#

CCE Upgrade API Parameters#

ParameterCommandRequiredDescriptionConstraints--cluster_idAll upgrade operationsYesCCE cluster IDMust reference existing cluster--spec.clusterUpgradeAction.targetVersionUpgradeCluster, CreateUpgradeWorkFlowYesTarget Kubernetes versionMust follow upgrade path rules (no skip-version)--spec.clusterUpgradeAction.strategy.typeUpgradeClusterYesUpgrade strategy typeOnly inPlaceRollingUpdate supported--spec.clusterUpgradeAction.strategy.inPlaceRollingUpdate.userDefinedStepUpgradeClusterNoBatch size per upgrade step1-40, default 20--spec.clusterUpgradeAction.strategy.inPlaceRollingUpdate.scopeUpgradeClusterNoBatch upgrade scopeCluster or NodePool``--spec.clusterUpgradeAction.addons.N.addonTemplateNameUpgradeClusterNoAddon template nameMust match installed addon name--spec.clusterUpgradeAction.addons.N.versionUpgradeClusterNoTarget addon versionMust be compatible with target K8s version--spec.clusterUpgradeAction.nodePoolOrder.keyUpgradeClusterNoNode pool upgrade prioritykey=nodepool-id, value=priority number--metadata.apiVersionUpgradeClusterYesAPI versionv3``--metadata.kindUpgradeClusterYesResource kindUpgradeTask``--cli-regionAll hcloud commandsRequiredRegion IDConfig value or env variable--cli-outputAll hcloud commandsRecommendedOutput formatjson recommended

Upgrade Workflow Parameters#

ParameterCommandRequiredDescriptionConstraints--spec.targetVersionCreateUpgradeWorkFlowYesTarget version for pre-checkMust follow upgrade path--spec.clusterVersionCreateUpgradeWorkFlowYesCurrent cluster versionMust match actual cluster version--upgrade_workflow_idShowUpgradeWorkFlow, UpgradeWorkFlowUpdateYesWorkflow task IDObtained from CreateUpgradeWorkFlow response

Output Format#

Upgrade Assessment Report#

json

{
  "cluster_id": "abc123",
  "current_version": "v1.23",
  "target_version": "v1.25",
  "upgrade_paths": ["v1.23 → v1.24 → v1.25"],
  "pre_check_status": "passed|failed|warning",
  "pre_check_items": { "total": 76, "passed": 72, "failed": 4 },
  "addon_compatibility": [
    { "addon": "coredns", "current": "1.23", "compatible": true }
  ],
  "estimated_window_minutes": 123,
  "risk_level": "low|medium|high",
  "recommended_strategy": "inPlaceRollingUpdate",
  "execution_commands": ["hcloud CCE UpgradeCluster ..."]
}

Upgrade Execution Preview#

json

{
  "cluster_id": "abc123",
  "step": "cluster_upgrade_preview",
  "target_version": "v1.25",
  "strategy": "inPlaceRollingUpdate",
  "batch_size": 20,
  "addons_upgrade": ["coredns → 1.25"],
  "estimated_time": "2h 3min",
  "confirmation_required": true,
  "warning": "Control plane upgrade is irreversible once started"
}

Verification#

Pre-Upgrade Verification#

bash

hcloud CCE ShowClusterUpgradeInfo --cluster_id=<cluster-id> --cli-region=<region> --cli-output=json

• Verify preCheckStatus is passed
• Check all 76 items in preCheckItems

Post-Upgrade Verification#

bash

# Check cluster version
hcloud CCE ShowCluster --cluster_id=<cluster-id> --detail=true --cli-region=<region> --cli-output=json

# Check addon status
hcloud CCE ListAddonInstances --cluster_id=<cluster-id> --cli-region=<region> --cli-output=json

# Check node status
kubectl get nodes

# Check workload health
kubectl get pods -A

Notes#

• No skip-version upgrades: must follow upgrade path step by step
• In-place upgrade is the only supported strategy: inPlaceRollingUpdate only
• v1.28+ control node IP change: upgrading to v1.28+ creates new control nodes with different IP addresses
• Autoscaling paused during upgrade: cluster autoscaling pauses during control plane upgrade
• Addon upgrade must follow cluster upgrade: addons upgrade after cluster reaches target version
• Pre-check is mandatory: never skip ShowClusterUpgradeInfo or CreateUpgradeWorkFlow pre-check

Best Practices#

1. Always run pre-check first — use CreateUpgradeWorkFlow or ShowClusterUpgradeInfo to validate readiness
2. Upgrade patch version first — ensure current patch is latest before major version upgrade
3. Business low-traffic window — schedule upgrades during low-traffic periods
4. Keep ≥2 nodes per node pool for redundancy during upgrade
5. Monitor upgrade progress — check ShowUpgradeWorkFlow status after each phase
6. Verify after each phase — run post-upgrade verification after control plane and node upgrades
7. Rollback plan — use backup data and PauseUpgradeClusterTask if issues arise

References#

DocumentDescriptionupgrade-workflow.mdFull 7-step upgrade workflow detailpre-upgrade-checklist.md76-item pre-check checklist and version-specific breaking changesaddon-compatibility.mdAddon compatibility matrix, DaemonSet plugins, upgrade orderk8s-version-matrix.mdOfficial CCE upgrade path table and patch version rulesupgrade-window-estimation.mdUpgrade window estimation formula, batch strategy, examplesrisk-rules.mdRisk constraints, rollback strategies, guardrailsoutput-schema.mdAssessment report and execution preview JSON schema