Huawei Cloud CCE/UCS Workload Manager#

Overview#

This skill provides workload lifecycle management capabilities for Huawei Cloud CCE and UCS-managed Kubernetes clusters using hcloud CLI for kubeconfig acquisition and kubectl for Kubernetes resource operations.

Architecture: hcloud CLI → kubeconfig YAML → kubectl --kubeconfig → k8s resources (Deployment/StatefulSet/DaemonSet/Job/CronJob/Service/Ingress/ConfigMap/Secret/PVC)

Related Skills:

• cce-cluster-management - CCE cluster infrastructure creation, scaling, and deletion
• ucs-cluster-onboarding-manager - UCS cluster onboarding, lifecycle, and fleet grouping
• ucs-policy-governor - UCS policy governance, compliance, and audit management

Capabilities:

• Obtain kubeconfig for CCE clusters (direct cluster access)
• Obtain federation kubeconfig for UCS fleet (multi-cluster fleet operations)
• Manage Deployment/StatefulSet/DaemonSet lifecycle (create, query, scale, update, delete)
• Manage Job/CronJob lifecycle (create, query, suspend, resume, delete)
• Configure HPA autoscaling for Deployments and StatefulSets
• Manage Service/Ingress networking, ConfigMap/Secret configuration, PVC storage
• Observe Pod status, logs, and events
• Manage namespaces
• Install and configure kubectl

Typical Use Cases:

• "Get kubeconfig for my CCE cluster"
• "Deploy an application to CCE"
• "Scale my Deployment to 5 replicas"
• "Roll back a Deployment update"
• "Create a StatefulSet for my database"
• "Deploy a logging agent as DaemonSet"
• "Create a CronJob for scheduled tasks"
• "Set up HPA for auto-scaling"
• "Check Pod logs for debugging"
• "Create a Service and Ingress for my app"
• "Manage ConfigMaps and Secrets"
• "Create a PersistentVolumeClaim"
• "List namespaces in my cluster"
• "Install kubectl on my machine"

Prerequisites#

1. hcloud CLI Requirements (MANDATORY)#

• hcloud CLI installed (version >= 7.2.2)
• Run hcloud version to verify installation
• First-time usage: printf "y\n" | hcloud version to accept privacy statement

2. kubectl Requirements (MANDATORY)#

• kubectl installed (version compatible with cluster Kubernetes version)
• See Task: kubectl Setup for installation guidance
• Run kubectl version --client to verify installation

3. Credential Configuration#

• Valid Huawei Cloud credentials (AK/SK mode)

• Security Rules:

  • 🚫 Never expose AK/SK values in code, conversation, or commands
  • 🚫 Never use echo $HUAWEI_CLOUD_AK or echo $HUAWEI_CLOUD_SK to check credentials
  • ✅ Use environment variables: HUAWEI_CLOUD_AK, HUAWEI_CLOUD_SK, HUAWEI_CLOUD_REGION
  • ✅ Prefer IAM users over root account for cloud operations
  • ✅ Enable MFA for sensitive operations

Configuration Method (Environment Variables Only):

bash

export HUAWEI_CLOUD_AK=<your-ak>
export HUAWEI_CLOUD_SK=<your-sk>
export HUAWEI_CLOUD_REGION=cn-north-4

⚠️ Important Security Notes:

• Never commit credentials to version control
• Use IAM users with minimal required permissions
• Enable MFA for sensitive operations
• Rotate AK/SK regularly

4. IAM Permission Requirements#

API ActionPermissionPurposecce:cluster:getGet clusterView cluster detailscce:cluster:createCertCreate certificateObtain CCE cluster kubeconfigucs:kubeconfig:createCreate kubeconfigObtain UCS cluster kubeconfigucs:federationKubeconfig:getGet federationDownload UCS federation kubeconfig

Two-Layer Permission Model:

1. Huawei Cloud IAM: Controls kubeconfig acquisition via hcloud CLI (who can get a kubeconfig for CCE or UCS fleet)
2. Kubernetes RBAC: Controls kubectl operations after kubeconfig is obtained (what the user can do in the cluster or fleet)

Permission Failure Handling:

1. When hcloud commands fail due to IAM permission errors, verify the IAM permissions listed above
2. When kubectl commands fail due to RBAC permission errors, the cluster administrator must configure appropriate RBAC roles
3. Guide the user to create custom policies in the IAM console for Huawei Cloud permissions
4. Guide the user to create ClusterRole/Role bindings for Kubernetes RBAC permissions
5. Pause execution and wait for user confirmation that permissions have been granted

Core Commands#

1. Kubeconfig Acquisition#

See Task: Kubeconfig Acquisition for detailed workflows.

bash

# CCE cluster kubeconfig (duration in days, 1-1827)
hcloud CCE CreateKubernetesClusterCert --cluster_id=<cce-cluster-id> --duration=1 --cli-region=cn-north-4

# UCS federation kubeconfig (duration in days, 1-1825)
# Used for multi-cluster fleet operations via UCS
hcloud UCS DownloadFederationKubeconfig --clustergroupid=<fleet-id> --duration=1 --cli-region=cn-north-4

Save kubeconfig and verify:

bash

# Save CCE kubeconfig to file
hcloud CCE CreateKubernetesClusterCert --cluster_id=<cce-cluster-id> --duration=1 --cli-region=cn-north-4 > ~/.kube/cce-kubeconfig.yaml

# Verify connection
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml cluster-info

Key Parameter Differences:

• CCE uses --cluster_id (with underscore) for cluster ID
• UCS uses --clusterid (no underscore) for cluster ID
• CCE CreateKubernetesClusterCert --duration is in days (1-1827)
• UCS DownloadFederationKubeconfig --duration is in days (1-1825)

2. Workload Management (Deployment/StatefulSet/DaemonSet)#

See Task: Deployment Management and Task: StatefulSet/DaemonSet Management for detailed workflows.

All commands use kubectl --kubeconfig=<kubeconfig-file> -n <namespace> pattern.

bash

# Create Deployment from YAML file
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml apply -f deployment.yaml -n production

# Create Deployment inline
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml create deployment my-app --image=myapp:v1 --replicas=3 -n production

# Query Deployment status
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get deployments -n production
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml describe deployment my-app -n production

# Scale Deployment
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml scale deployment my-app --replicas=5 -n production

# Update Deployment (rolling update)
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml set image deployment/my-app my-app=myapp:v2 -n production
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml rollout status deployment/my-app -n production

# Rollback Deployment
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml rollout undo deployment/my-app -n production

# Delete Deployment
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml delete deployment my-app -n production

# StatefulSet operations (same pattern)
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml apply -f statefulset.yaml -n production
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml scale statefulset my-db --replicas=3 -n production

# DaemonSet operations (same pattern)
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml apply -f daemonset.yaml -n production
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml rollout status daemonset/log-agent -n production

3. Job/CronJob Management#

bash

# Create Job from YAML
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml apply -f job.yaml -n production

# Create Job inline
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml create job data-migration --image=migrator:v1 -n production

# Query Job status
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get jobs -n production
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml describe job data-migration -n production

# Delete Job
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml delete job data-migration -n production

# Create CronJob from YAML
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml apply -f cronjob.yaml -n production

# Create CronJob inline
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml create cronjob nightly-backup --image=backup:v1 --schedule="0 2 * * *" -n production

# Suspend CronJob
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml patch cronjob nightly-backup --type merge --patch-file=suspend.json -n production

# Resume CronJob
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml patch cronjob nightly-backup --type merge --patch-file=resume.json -n production

# Delete CronJob
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml delete cronjob nightly-backup -n production

4. Observability + Config#

bash

# Pod status
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get pods -n production
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get pods -o wide -n production

# Pod logs
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml logs my-app-pod -n production
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml logs my-app-pod --tail=100 -n production
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml logs my-app-pod -f -n production

# Pod events
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml describe pod my-app-pod -n production

# Namespace management
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get namespaces
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml create namespace staging
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml delete namespace staging

# Service
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml apply -f service.yaml -n production
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get services -n production

# Ingress
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml apply -f ingress.yaml -n production
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get ingress -n production

# ConfigMap
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml create configmap app-config --from-literal=key1=value1 -n production
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get configmaps -n production

# Secret
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml create secret generic db-secret --from-literal=password=s3cret -n production
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get secrets -n production

# PVC
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml apply -f pvc.yaml -n production
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get pvc -n production

# HPA autoscaling
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml autoscale deployment my-app --min=2 --max=10 --cpu=80% -n production
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get hpa -n production

5. kubectl Installation & Configuration#

See Task: kubectl Setup for detailed installation and configuration guidance.

Parameter Reference#

hcloud Parameters (Kubeconfig Acquisition)#

ParameterCommandRequiredDescriptionConstraints--cluster_idCCE CreateCertYesCCE cluster IDMust reference existing CCE cluster--clustergroupidUCS DownloadFederationYesFleet group IDMust reference existing fleet group--durationCCE CreateCertYes*Certificate validity in days1-1827 days; at least one of --duration/--expire_at required--expire_atCCE CreateCertYes*Certificate expiry timestampISO format; mutually exclusive with --duration--durationUCS DownloadFederationYesToken validity in days (1-1825)Integer (1-1825)--project_idCCE CreateCertRequired (auto-filled)Project IDAuto-filled from hcloud config if not specified--cli-regionAll hcloud commandsRequiredHuawei Cloud region IDConfig value or HUAWEI_CLOUD_REGION

kubectl Flags#

FlagDescriptionExample--kubeconfigPath to kubeconfig file--kubeconfig=~/.kube/cce-kubeconfig.yaml``-nNamespace for the operation-n production``-oOutput format (wide/yaml/json)-o wide, -o yaml, -o json``-fYAML/JSON file path-f deployment.yaml``--replicasNumber of replicas for scaling--replicas=5``--imageContainer image for create/set--image=myapp:v1``--tailNumber of log lines to show--tail=100``-f (logs)Follow log output (stream)logs -f``--min/--maxHPA min/max replicas--min=2 --max=10``--cpu-percentHPA CPU target percentage--cpu-percent=80

Output Format#

CCE CreateKubernetesClusterCert (Kubeconfig YAML)#

Returns a standard Kubernetes kubeconfig YAML document containing:

yaml

apiVersion: v1
kind: Config
clusters:
- cluster:
    certificate-authority-data: <base64-encoded-ca>
    server: https://<cluster-endpoint>:5443
  name: internalCluster
- cluster:
    server: https://<cluster-eip>:5443
    insecure-skip-tls-verify: true
  name: externalCluster
- cluster:
    certificate-authority-data: <base64-encoded-ca>
    server: https://<cluster-eip>:5443
  name: externalClusterTLSVerify
contexts:
- context:
    cluster: internalCluster
    user: user
  name: internal
- context:
    cluster: externalCluster
    user: user
  name: external
- context:
    cluster: externalClusterTLSVerify
    user: user
  name: externalTLSVerify
current-context: external

UCS DownloadFederationKubeconfig (Federation Kubeconfig YAML)#

Returns a federation kubeconfig with two contexts for multi-cluster fleet operations:

yaml

apiVersion: v1
kind: Config
clusters:
- cluster:
    certificate-authority-data: <base64-encoded-ca>
    server: https://<fleet-name>.fleet.ucs.<region>.myhuaweicloud.com:5443
  name: federation
- cluster:
    certificate-authority-data: <base64-encoded-ca>
    server: https://<fleet-name>.fleet.ucs.<region>.myhuaweicloud.com:5443/apis/cluster.karmada.io/v1alpha1/clusters/*/proxy
  name: karmada-aggregated-apiserver
contexts:
- context:
    cluster: federation
    user: user
  name: federation
- context:
    cluster: karmada-aggregated-apiserver
    user: user
  name: karmada-aggregated-apiserver
current-context: federation

Federation vs Karmada context:

• federation: Operates on fleet-level resources (propagated workloads, fleet policies)
• karmada-aggregated-apiserver: Proxy access to individual member clusters via /clusters/<cluster-name>/proxy

Network prerequisite: UCS federation kubeconfig uses <fleet-name>.fleet.ucs.<region>.myhuaweicloud.com as the API server domain. This domain resolves via UCS VPC Endpoint (VPCEP). If DNS resolution fails, ensure your network can reach the UCS VPCEP (e.g., via VPC peering, VPN, or direct cloud network access).

text

### kubectl Output Formats

| `-o` flag        | Description                        | Use Case                                    |
| ---------------- | ---------------------------------- | ------------------------------------------- |
| (default)        | Tabular human-readable output      | Quick status check                          |
| `-o wide`        | Extended tabular with extra columns | Detailed Pod/IP info                       |
| `-o yaml`        | YAML format                        | Export/edit resource specs                  |
| `-o json`        | JSON format                        | Scripting/automation                        |
| `-o name`        | Resource name only                 | Quick list of names                         |

## Common Region IDs

| Region Name                    | Region ID        |
| ------------------------------ | ---------------- |
| North China - Beijing 4        | `cn-north-4`     |
| North China - Beijing 1        | `cn-north-1`     |
| East China - Shanghai 1        | `cn-east-3`      |
| East China - Shanghai 2        | `cn-east-2`      |
| South China - Guangzhou        | `cn-south-1`     |
| South China - Shenzhen         | `cn-south-4`     |
| Southwest China - Guiyang 1    | `cn-southwest-2` |
| Asia Pacific - Bangkok         | `ap-southeast-2` |
| Asia Pacific - Singapore       | `ap-southeast-1` |
| Asia Pacific - Hong Kong       | `ap-southeast-3` |
| Europe - Paris                 | `eu-west-0`      |

## Verification

### Kubeconfig Acquisition Verification

```bash
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml cluster-info
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get nodes

Workload Deployment Verification#

bash

kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get deployments -n production
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml rollout status deployment/my-app -n production
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get pods -o wide -n production

HPA Verification#

bash

kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get hpa -n production

Best Practices#

1. Namespace-First: Always specify -n <namespace> explicitly; never rely on default namespace
2. Kubeconfig Security: Store kubeconfig files with restricted file permissions (chmod 600); never expose them in public repositories or CI logs
3. Short-Duration Certificates: Use minimum --duration needed for CCE kubeconfig; prefer 1 day for interactive sessions
4. RBAC Alignment: Ensure Huawei Cloud IAM permissions and Kubernetes RBAC roles are aligned; IAM grants kubeconfig access, RBAC grants kubectl operations
5. YAML-Based Deployment: Prefer kubectl apply -f over inline kubectl create for reproducibility and auditability
6. Rollout Monitoring: Always check rollout status after updates; use rollout undo for emergency rollback
7. Namespace Isolation: Use separate namespaces for production, staging, and development workloads
8. HPA Baseline: Set HPA --min to your steady-state replica count; set --max based on resource budget

Reference Documents#

DocumentDescriptionTask: Kubeconfig AcquisitionKubeconfig acquisition workflowsTask: kubectl Setupkubectl installation and configurationTask: Deployment ManagementDeployment lifecycle workflowsTask: StatefulSet/DaemonSet ManagementStatefulSet and DaemonSet workflows

Notes#

• kubectl is the primary tool — all workload operations use kubectl --kubeconfig=<file> after kubeconfig acquisition via hcloud
• kubeconfig is a secret — treat it like a credential; never share or expose publicly
• RBAC governs kubectl access — even with a valid kubeconfig, Kubernetes RBAC controls what operations are permitted
• cce-cluster-management handles infrastructure — cluster creation, deletion, and node management belong to the cce-cluster-management skill
• Two-layer permission model — Huawei Cloud IAM controls kubeconfig acquisition, Kubernetes RBAC controls kubectl operations
• UCS is for fleet operations — UCS kubeconfig is only for federation (DownloadFederationKubeconfig), not single cluster (CreateClusterKubeconfig is out of scope for this skill)
• UCS federation kubeconfig provides two contexts — federation (fleet-level) and karmada-aggregated-apiserver (proxy to member clusters)
• UCS federation requires network access — fleet API server domain ( <fleet>.fleet.ucs.<region>.myhuaweicloud.com) requires VPCEP access

Common Pitfalls#

PitfallSymptomQuick Fixkubectl not installedCommand not foundInstall kubectl (see references/task-kubectl-setup.md)Wrong cluster_id404 or kubeconfig for wrong clusterVerify cluster ID with hcloud CCE ListClustersKubeconfig expiredAuthentication failuresRe-acquire kubeconfig with CreateKubernetesClusterCertRBAC insufficientForbidden errors in kubectlConfigure appropriate ClusterRole/Role bindingsMissing namespace flagResources in wrong namespaceAlways specify -n <namespace> explicitlyCCE vs UCS param confusionParameter not recognizedCCE: --cluster_id, UCS: --clustergroupid for federationDuration unit confusionCertificate expires immediatelyCCE: days (1-1827), UCS federation: days (1-1825)UCS federation DNS unreachableno such host on federation APIEnsure VPCEP/network access to UCS fleet domainInline create vs YAML applyHard to reproduce/auditPrefer kubectl apply -f <yaml> for productionRollout without status checkUnknown deployment stateAlways run rollout status after updatesKubeconfig file permissionsSecurity warning or access deniedSet file permissions to 600 ( chmod 600)Wrong StorageClass in PVCPVC stuck PendingUse csi-disk (not cce-standard); run kubectl get sc to verifyMetrics API not availabletop pods/nodes failsInstall metrics-server addon via CCE consolePowerShell JSON patch escapingpatch -p fails with JSON errorsUse --patch-file instead of inline -p JSONHPA --cpu-percent deprecatedWarning flag deprecatedUse --cpu=80% instead of --cpu-percent=80