Huawei Cloud CCE-CCI Bursting Deployer#

Overview#

This skill configures CCE workloads to burst into CCI 2.0 serverless capacity for elastic scaling. It automates the full workflow: precheck cluster readiness including NodeCheck for physical-node headroom, ensure VPCEP dependencies for image pulling, install or update the virtual-kubelet addon, discover tenant-owned smoke images, deploy a smoke workload, verify the virtual node and pod status, and diagnose addon failures.

Architecture: python3 scripts/huawei-cloud.py → CCE/VPCEP/VPC API → Precheck → NodeCheck → VPCEP creation → Addon install → Image discovery → Smoke deployment → Verification → Diagnosis

Key Principle: Preview-first. Read-only checks (precheck, NodeCheck, verify, discover images, diagnose addon) run immediately, but VPCEP creation, addon installation, node pool changes, and smoke workload deployment require explicit user approval with confirm=true before execution.

Related Skills:

• huawei-cloud-cce-cluster-management - Cluster lifecycle, addon listing, kubeconfig retrieval
• huawei-cloud-cce-pod-failure-diagnoser - Pod failure diagnosis when bursting pods fail to reach Running
• huawei-cloud-cce-network-failure-diagnoser - Network diagnosis for VPCEP connectivity issues

Prerequisites#

1. Python Requirements (MANDATORY)#

• Python 3 installed (version >= 3.8)
• Run python3 --version to verify installation
• huawei-cloud.py script available in the scripts directory
• Required packages: huaweicloudsdkcce, huaweicloudsdkcore, huaweicloudsdkvpc, huaweicloudsdkvpcep, kubernetes

2. Credential Configuration#

• Valid Huawei Cloud credentials (AK/SK mode)

• Security Rules:

  • Never expose AK/SK values in code, conversation, or commands
  • Never use echo commands to check credential environment variables
  • Use environment variables: HUAWEI_AK, HUAWEI_SK, HUAWEI_PROJECT_ID
  • Alternative env vars: HUAWEICLOUD_SDK_AK, HUAWEICLOUD_SDK_SK, HW_ACCESS_KEY, HW_SECRET_KEY
  • Prefer IAM users over root account for cloud operations
  • Never persist AK/SK in skill files, debug files, reports, or shell history

Configuration Method (Environment Variables Only):

bash

export HUAWEI_AK=<your-ak>
export HUAWEI_SK=<your-sk>
export HUAWEI_PROJECT_ID=<your-project-id>

3. IAM Permission Requirements#

API ActionPermissionPurposecce:cluster:getGet cluster detailsRead cluster network spec (VPC, subnet, ENI)cce:addon:listList addonsCheck virtual-kubelet installation statecce:addon:createInstall addonInstall virtual-kubelet addoncce:addon:updateUpdate addonConfigure virtual-kubelet bursting parametersvpcep:endpoint:createCreate VPCEPCreate SWR/OBS interface endpointsvpcep:endpoint:listList VPCEPCheck existing VPCEP endpointsvpcep:service:listList VPCEP servicesDiscover public service detailsvpc:subnet:listList subnetsValidate subnet IDs in cluster VPCvpc:routetable:listList route tablesFind route table IDs for OBS gateway VPCEP

Permission Failure Handling: When any command fails due to IAM permission errors, verify the permissions listed above, guide the user to create custom IAM policies, and pause execution until permissions are confirmed.

Core Commands#

All tools are invoked through python3 scripts/huawei-cloud.py with key=value parameters.

1. Precheck (Read-Only)#

bash

python3 scripts/huawei-cloud.py huawei_precheck_cce_cci_bursting region=cn-north-4 cluster_id=<cluster-id> vpcep_subnet_id=<vpc-subnet-id>

Inspects cluster networking, resolves subnet roles, checks addon state, and reports any blocking issues. The cluster must be a Turbo/ENI cluster for CCI bursting.

2. Ensure VPCEP Dependencies#

bash

# Preview (no mutation)
python3 scripts/huawei-cloud.py huawei_ensure_cce_cci_vpcep region=cn-north-4 cluster_id=<cluster-id> vpcep_subnet_id=<vpc-subnet-id>

# Apply after user approval
python3 scripts/huawei-cloud.py huawei_ensure_cce_cci_vpcep region=cn-north-4 cluster_id=<cluster-id> vpcep_subnet_id=<vpc-subnet-id> confirm=true

Creates or reuses SWR, SWR-API, and OBS-compatible VPCEP interface endpoints in the cluster VPC. If OBS information is missing, pass obs_endpoint_service_name obtained from the Huawei Cloud service ticket (do not guess).

3. Setup CCI Bursting#

bash

# Preview
python3 scripts/huawei-cloud.py huawei_setup_cce_cci_bursting region=cn-north-4 cluster_id=<cluster-id> vpcep_subnet_id=<vpc-subnet-id>

# Apply after user approval
python3 scripts/huawei-cloud.py huawei_setup_cce_cci_bursting region=cn-north-4 cluster_id=<cluster-id> vpcep_subnet_id=<vpc-subnet-id> confirm=true

Ensures VPCEP dependencies, installs virtual-kubelet if absent, and configures CCI network parameters. Idempotent: updates existing addon configuration without uninstalling.

4. Verify Bursting Readiness (Read-Only)#

bash

python3 scripts/huawei-cloud.py huawei_verify_cce_cci_bursting region=cn-north-4 cluster_id=<cluster-id>

Checks addon state, virtual node readiness, and optional workload pod status. Returns ready=true when the virtual node is Ready and all workload pods are Running on it.

5. Deploy Smoke Workload#

bash

# Preview
python3 scripts/huawei-cloud.py huawei_deploy_cce_cci_smoke_workload region=cn-north-4 cluster_id=<cluster-id> replicas=2

# Apply after user approval
python3 scripts/huawei-cloud.py huawei_deploy_cce_cci_smoke_workload region=cn-north-4 cluster_id=<cluster-id> replicas=2 confirm=true

Creates or patches a small Deployment forced onto CCI capacity using the bursting.cci.io/burst-to-cci: enforce label. Uses a regional SWR image by default.

6. Verify Smoke Workload (Read-Only)#

bash

python3 scripts/huawei-cloud.py huawei_verify_cce_cci_bursting region=cn-north-4 cluster_id=<cluster-id> namespace=cci2-burst-lab workload_name=cci2-burst-demo

Confirms test pods reach Running on the virtual node ( bursting-node or virtual-kubelet).

7. Check Node Capacity (Read-Only)#

bash

python3 scripts/huawei-cloud.py huawei_check_cce_cci_node_capacity region=cn-north-4 cluster_id=<cluster-id>

Inspects physical-node addon headroom. The precheck includes NodeCheck automatically, warning when schedulable capacity is below the 2C/4GiB conservative baseline. Preview node pool expansion when the baseline is not met.

8. List Node Pools (Read-Only)#

bash

python3 scripts/huawei-cloud.py huawei_list_cce_nodepools region=cn-north-4 cluster_id=<cluster-id>

Lists CCE node pools in the cluster. Used after NodeCheck warns about insufficient headroom to identify existing pools for resize.

9. Resize Node Pool#

bash

# Preview
python3 scripts/huawei-cloud.py huawei_resize_cce_nodepool region=cn-north-4 cluster_id=<cluster-id> nodepool_id=<nodepool-id> node_count=3

# Apply after user approval
python3 scripts/huawei-cloud.py huawei_resize_cce_nodepool region=cn-north-4 cluster_id=<cluster-id> nodepool_id=<nodepool-id> node_count=3 confirm=true

Resizes an existing node pool to increase physical-node capacity. Apply only after explicit user approval with confirm=true.

10. Create Node Pool#

bash

# Preview
python3 scripts/huawei-cloud.py huawei_create_cce_nodepool region=cn-north-4 cluster_id=<cluster-id> nodepool_name=<name> flavor=<flavor> availability_zone=<az> root_volume_size=40 root_volume_type=SSD

# Apply after user approval
python3 scripts/huawei-cloud.py huawei_create_cce_nodepool region=cn-north-4 cluster_id=<cluster-id> nodepool_name=<name> flavor=<flavor> availability_zone=<az> root_volume_size=40 root_volume_type=SSD confirm=true

Creates a new node pool. Prefer SSH keypair authentication. Apply only after explicit user approval with confirm=true.

11. Discover Smoke Images (Read-Only)#

bash

python3 scripts/huawei-cloud.py huawei_discover_cce_cci_smoke_images region=cn-north-4

Discover tenant-owned SWR basic images through namespace, repository, and tag queries. The smoke deployment automatically selects a discovered tenant-owned image when image is omitted.

12. Diagnose Bursting Addon (Read-Only)#

bash

python3 scripts/huawei-cloud.py huawei_diagnose_cce_cci_bursting_addon region=cn-north-4 cluster_id=<cluster-id>

Returns read-only addon diagnostics when verification fails or the virtual node does not become Ready. Inspect the returned diagnostics to identify region mismatches, missing project IDs, or configuration issues.

Parameter Reference#

Common Parameters#

ParameterRequired/OptionalDescriptionDefaultregionRequiredHuawei Cloud region IDHUAWEI_AK env regioncluster_idRequiredCCE cluster IDN/AakOptionalAccess Key (overrides env var)HUAWEI_AK envskOptionalSecret Key (overrides env var)HUAWEI_SK envproject_idOptionalProject ID (overrides env var)Auto-resolved via IAM

Precheck Parameters#

ParameterRequired/OptionalDescriptionNotesvpcep_subnet_idOptionalVPC subnet ID for VPCEP placementDefaults to cluster host subnet

VPCEP Parameters#

ParameterRequired/OptionalDescriptionNotesvpcep_subnet_idOptionalVPC subnet ID for VPCEPDefaults from precheckobs_endpoint_service_nameOptionalExact OBS VPCEP service nameObtain from Huawei Cloud service ticket; do not guessroute_table_idsOptionalRoute table IDs for OBS gatewayAuto-resolved if omitted; comma-separatedconfirmRequired for mutationApprove VPCEP creationtrue to apply, omit to preview

Setup Parameters#

ParameterRequired/OptionalDescriptionNotesvpcep_subnet_idOptionalVPC subnet ID for VPCEPDefaults from precheckcci_subnet_idOptionalNeutron subnet ID for addonDefaults from spec.eni_network``obs_endpoint_service_nameOptionalOBS VPCEP service nameRequired if precheck reports missing OBSroute_table_idsOptionalRoute table IDs for OBS gatewayAuto-resolved if omittedaddon_versionOptionalvirtual-kubelet addon versionDefaults to 1.5.82 or existing versionconfirmRequired for mutationApprove setuptrue to apply, omit to preview

Smoke Workload Parameters#

ParameterRequired/OptionalDescriptionDefaultnamespaceOptionalSmoke namespacecci2-burst-lab``workload_nameOptionalSmoke Deployment namecci2-burst-demo``imageOptionalContainer imageRegional SWR nginx imagereplicasOptionalPod replica count2``confirmRequired for mutationApprove deploymenttrue to apply, omit to preview

Verify Parameters#

ParameterRequired/OptionalDescriptionNotesnamespaceOptionalWorkload namespaceFilter pods and deploymentsworkload_nameOptionalWorkload nameFilter pods by app label

Subnet Role Reference#

ParameterID TypeUsed Bycci_subnet_idNeutron subnet UUIDvirtual-kubelet addon networkID, subnet_id, subnets[].subnetID``vpcep_subnet_idVPC subnet UUIDVPCEP interface endpoint placement

These are different ID namespaces. Never swap them. For a Turbo/ENI cluster, huawei_precheck_cce_cci_bursting resolves cci_subnet_id from spec.eni_network. Pass vpcep_subnet_id explicitly when a dedicated endpoint subnet is preferred.

Output Format#

All tools return JSON with the following structure:

FieldDescriptionsuccessBoolean: true if operation succeeded, false otherwiseactionAction name that was executedregionHuawei Cloud regioncluster_idCCE cluster ID

Precheck Output:

FieldDescriptionnetworkCluster network context (VPC, subnets, ENI)subnet_rolesResolved cci_subnet_id and vpcep_subnet_id``virtual_kubeletExisting addon info or null``issuesList of blocking issues (empty if ready)

Verify Output:

FieldDescriptionreadyBoolean: addon installed, virtual node Ready, workload pods Runningaddonvirtual-kubelet addon detailsvirtual_nodesList of virtual nodes in the clusterworkload.phase_distributionPod phase counts (Running, Pending, etc.)workload.node_distributionPod node assignment countswarning_eventsRecent warning events for the workload

Verification#

Step-by-step Verification Checklist#

1. Verify AK/SK credentials are configured via environment variables
2. Run huawei_precheck_cce_cci_bursting and confirm issues is empty
3. Verify the cluster is Turbo/ENI type ( container_network_mode = eni)
4. Run huawei_ensure_cce_cci_vpcep preview and confirm VPCEP plan is correct
5. After user approval, apply VPCEP with confirm=true
6. Run huawei_setup_cce_cci_bursting preview, then apply with confirm=true
7. Run huawei_verify_cce_cci_bursting and confirm ready=true
8. Deploy smoke workload with preview, then confirm=true
9. Run final verification with namespace and workload_name parameters
10. Confirm all pods are Running on bursting-node or virtual-kubelet

Best Practices#

1. Always precheck first: Run huawei_precheck_cce_cci_bursting before any mutation to identify blocking issues
2. Preview before apply: Always run mutation actions without confirm=true first, review the plan, then re-run with confirm=true after explicit user approval
3. Never swap subnet IDs: cci_subnet_id (Neutron UUID) and vpcep_subnet_id (VPC UUID) are different ID namespaces; swapping them causes addon failure
4. Prefer tenant-owned SWR images: Docker Hub images and public namespace images timeout in CCI capacity; prefer a tenant-owned regional SWR image via huawei_discover_cce_cci_smoke_images; public namespace images are fallback-only
5. Obtain OBS service name from service ticket: Never guess the obs_endpoint_service_name from a similar regional public service
6. Verify after each change: Run huawei_verify_cce_cci_bursting after each applied change to confirm progress
7. Setup is idempotent: huawei_setup_cce_cci_bursting updates existing addon configuration without uninstalling; safe to re-run
8. Reuse existing VPCEPs: The tool reuses accepted VPCEPs in the cluster VPC; no duplicate creation
9. Do not delete resources automatically: Never auto-delete VPCEPs, namespaces, workloads, addons, or addon ReplicaSets
10. Diagnose addon failures: When verification fails, run huawei_diagnose_cce_cci_bursting_addon for read-only diagnostics; never patch the internal bursting-status ConfigMap automatically
11. Node headroom is a warning: The 2C/4GiB NodeCheck is a conservative small-cluster warning, not a platform hard limit; size production resources with the official addon formula
12. Explicit project ID when needed: Pass project_id explicitly when IAM auto-resolution is unavailable

Reference Documents#

DocumentDescriptionWorkflowAction sequence, subnet roles, and command examplesRisk RulesPreview-first constraints, billing scope, and safe defaultsTroubleshootingSymptom-cause-action table for common bursting failures

Notes#

• Preview-first by design — VPCEP creation, addon installation, node pool changes, and workload deployment return a preview without confirm=true; apply only after explicit user approval
• Idempotent setup — huawei_setup_cce_cci_bursting may update the existing virtual-kubelet addon configuration but never uninstalls it; it resolves and writes the regional project ID
• Turbo/ENI required — CCE to CCI bursting requires a Turbo cluster with ENI container network mode
• No credential persistence — AK/SK exists only during API calls; never written to disk, logs, or reports
• Cross-skill escalation — If bursting pods show CrashLoopBackOff or ImagePullBackOff, hand off to huawei-cloud-cce-pod-failure-diagnoser; if VPCEP connectivity fails, hand off to huawei-cloud-cce-network-failure-diagnoser
• Tenant-owned smoke images — Prefer a tenant-owned SWR image discovered via huawei_discover_cce_cci_smoke_images; public namespace images are fallback-only because CCI pulling through VPCEP may fail
• NodeCheck warning baseline — The 2C/4GiB value is a conservative small-cluster warning; size production addon resources with the official formula

Common Pitfalls#

PitfallSymptomQuick FixSwapped subnet IDsvirtual-kubelet restarts, virtual node never ReadyRun precheck; use cci_subnet_id from spec.eni_network (Neutron UUID), not the VPC subnet UUIDMissing SWR VPCEPsCCI pod ImagePullBackOff or image pull timeoutRun huawei_ensure_cce_cci_vpcep to create SWR endpointsPublic namespace smoke imageCCI pod image pull timeout through VPCEPUse huawei_discover_cce_cci_smoke_images to find a tenant-owned SWR imageMissing bursting labelWorkload pods stay Pending on real nodesAdd bursting.cci.io/burst-to-cci: enforce label to pod templateGuessed OBS service nameOBS gateway VPCEP creation failsObtain exact obs_endpoint_service_name from Huawei Cloud service ticketInsufficient physical-node headroombursting-cceaddon-* Pods stay PendingRun huawei_check_cce_cci_node_capacity; preview node pool resize or createAddon region mismatchLogs report northchina or southchinaRun huawei_diagnose_cce_cci_bursting_addon; do not hard-code region mappingMissing project IDAddon reports IAM denied or project ID missingPass project_id explicitly when auto-resolution unavailableMultiple active ReplicaSetsbursting-cceaddon-* has several ReplicaSetsInspect Deployment rollout state; do not auto-delete ReplicaSetsNon-ENI clusterPrecheck reports CCI bursting not supportedUse a Turbo/ENI cluster; overlay_l2 clusters cannot burst to CCINon-ENI clusterPrecheck reports CCI bursting not supportedUse a Turbo/ENI cluster; overlay_l2 clusters cannot burst to CCIInsufficient physical-node headroombursting-cceaddon-* Pods stay PendingRun huawei_check_cce_cci_node_capacity; preview node pool resize or createAddon region mismatchLogs report northchina or southchinaRun huawei_diagnose_cce_cci_bursting_addon; do not hard-code region mappingMissing project IDAddon reports IAM denied or project ID missingPass project_id explicitly when IAM auto-resolution unavailableMultiple active ReplicaSetsbursting-cceaddon-* has several ReplicaSetsInspect Deployment rollout state; do not auto-delete ReplicaSets