Huawei Cloud CCE Pressure Test#

Trigger keywords: pressure test, 压测, load test, 负载测试, stress test, 压力测试, performance test, 性能测试, k6 test, k6 压测, ELB traffic test, ELB 流量测试, end-to-end pressure test, 全链路压测, elasticity evaluation, 弹性评估, traffic generation, 流量生成

⚠️ Execution Method (Must Read): This skill executes queries via the local Python dispatcher script. Using hcloud, openstack, or other CLI tools or direct API calls is prohibited.

• The dispatcher script is located at scripts/huawei-cloud.py within the skill directory
• All scripts and environment check scripts are inside the skill package. You must use skill action=exec to execute them. Do not run them directly in a shell.
• Do not attempt hcloud, openstack, curl IAM, or any other CLI/API methods. This skill does not depend on those tools.
• All paths are relative to the skill directory, which is the directory where this SKILL.md is located.

Overview#

This skill runs controlled workload pressure tests on Huawei Cloud CCE. It builds a complete traffic path from a k6 client through ELB to nginx-ingress to workload pods, supports short-connection, keepalive, and ramp traffic models, collects ELB metrics and AOM observability data, evaluates elasticity phases, and generates bilingual Markdown and HTML reports with performance curves.

Architecture: Python dispatcher ( scripts/huawei-cloud.py) → Huawei Cloud Python SDK + Kubernetes client → ELB/CCE/AOM/APM APIs → Route preparation → Client generation → Traffic run → Observability collection → Report generation

Key Principle: Preview-first. Read-only checks may run immediately, but Service/Ingress changes, traffic generation, and workload scaling require explicit user approval with confirm=true before execution.

Related Skills:

SkillPurposehuawei-cloud-cce-autoscaling-diagnoserDiagnose autoscaling failures when HPA or CA does not respond during pressure testshuawei-cloud-cce-observability-context-builderBuild observability context when AOM/APM evidence is neededhuawei-cloud-cce-pod-failure-diagnoserDiagnose Pod runtime failures during traffic runshuawei-cloud-cce-node-failure-diagnoserDiagnose node-level failures under loadhuawei-cloud-cce-workload-failure-diagnoserDiagnose workload rollout failures after scalinghuawei-cloud-cce-auto-remediation-runnerExecute remediation actions when issues are found during testshuawei-cloud-cce-cluster-managementCluster lifecycle, nodepool management, kubeconfig retrieval

Capabilities:

1. End-to-end traffic path: k6 client → ELB → nginx-ingress → workload Service → Pod
2. Short-connection ( short), keepalive ( keepalive), and staged ramp ( ramp) traffic models
3. Isolated Java sample deployment for lab environments
4. ELB creation when no reusable ingress-controller ELB exists
5. Route preparation: Service and Ingress creation/patching with preview-first
6. k6 client generation with ConfigMap and Job
7. Traffic run with replica and Pod count monitoring
8. APM Java agent injection for distributed tracing during tests
9. Observability collection: ELB metrics, Pod CPU/memory, AOM logs
10. Bilingual Markdown and HTML report generation with SVG performance curves
11. Elasticity evaluation: baseline vs scaled phase comparison

Prerequisites#

1. Python Requirements (MANDATORY)#

• Python >= 3.6 installed
• Required packages: huaweicloudsdkcore, huaweicloudsdkcce, huaweicloudsdkvpc, huaweicloudsdkvpcep, huaweicloudsdkaom, huaweicloudsdkelb, kubernetes, matplotlib
• Verify: python3 --version
• Install packages: pip3 install huaweicloudsdkcore huaweicloudsdkcce huaweicloudsdkvpc huaweicloudsdkvpcep huaweicloudsdkaom huaweicloudsdkelb kubernetes matplotlib

2. Credential Configuration#

• Valid Huawei Cloud credentials (AK/SK mode)

• Security Rules:

  • 🚫 Never expose AK/SK values in code, conversation, or commands
  • 🚫 Never use echo $HUAWEI_AK or echo $HUAWEI_SK to check credentials
  • 🚫 Never write credentials to files, logs, or responses
  • ✅ Use environment variables: HUAWEI_AK, HUAWEI_SK, HUAWEI_REGION
  • ✅ Credentials exist only in the current request call stack and are released after each invocation
  • ✅ Prefer IAM users over root account for cloud operations

Configuration Method (Environment Variables Only):

bash

export HUAWEI_AK=<your-ak>
export HUAWEI_SK=<your-sk>
export HUAWEI_REGION=cn-north-4

Additional Variables:

VariableRequiredDescriptionHUAWEI_AKYesHuawei Cloud Access KeyHUAWEI_SKYesHuawei Cloud Secret KeyHUAWEI_REGIONNoDefault region (overrides region param if set)HUAWEI_PROJECT_IDNoProject ID (auto-obtained via IAM API when not set)

3. IAM Permission Requirements#

API ActionServicePurposeCCE cluster readCCECluster info, workload inspection, Pod metricsCCE workload read/writeCCE (kubeconfig)Deploy Java sample, create Service/Ingress, scale replicasCCE addon readCCECheck ingress controller and metrics addonELB create/list/readELBCreate load balancer for test traffic pathVPC subnet listVPCFind candidate subnets for ELB VIPVPCEP endpoint listVPCEPCheck existing VPCEP endpointsAOM metrics readAOMCollect Pod CPU/memory and custom metricsAOM instances list/resolveAOMFind AOM instance for the clusterAPM master address readAPMGet APM master address for Java agent injectionLTS log queryLTSCorrelate application errors with traffic curves

Permission Failure Handling:

1. When any action fails due to IAM permission errors, display the required permission list
2. Guide the user to create custom policies in the IAM console for Huawei Cloud permissions
3. Pause execution and wait for user confirmation that permissions have been granted
4. Retry the failed action

Core Commands#

All actions are invoked via the dispatcher script:

bash

python3 scripts/huawei-cloud.py <action> region=<region> cluster_id=<cluster_id> [key=value ...]

1. Deploy Java Sample (Optional)#

bash

# Preview (no mutation)
python3 scripts/huawei-cloud.py huawei_deploy_cce_pressure_test_java_sample \
  region=cn-north-4 cluster_id=<cluster_id>

# Apply after user approval
python3 scripts/huawei-cloud.py huawei_deploy_cce_pressure_test_java_sample \
  region=cn-north-4 cluster_id=<cluster_id> confirm=true

Creates an isolated Namespace, ConfigMap (small Java HTTP server), and Deployment with readiness/liveness probes. Exposes /healthz, /api/hello, and /api/work on port 8080.

2. Create ELB (When No Reusable ELB Exists)#

bash

# Preview (no mutation)
python3 scripts/huawei-cloud.py huawei_create_elb \
  region=cn-north-4 name=<elb-name> vip_subnet_cidr_id=<subnet-network-id>

# Apply after user approval
python3 scripts/huawei-cloud.py huawei_create_elb \
  region=cn-north-4 name=<elb-name> vip_subnet_cidr_id=<subnet-network-id> confirm=true

Creates a chargeable ELB. Review billable fields before applying. Does not silently create an EIP.

3. Prepare Route#

bash

# Preview (no mutation)
python3 scripts/huawei-cloud.py huawei_prepare_cce_pressure_test_route \
  region=cn-north-4 cluster_id=<cluster_id> namespace=<namespace> workload_name=<deployment-name> target_port=8080

# Apply after user approval
python3 scripts/huawei-cloud.py huawei_prepare_cce_pressure_test_route \
  region=cn-north-4 cluster_id=<cluster_id> namespace=<namespace> workload_name=<deployment-name> target_port=8080 confirm=true

Creates or patches a ClusterIP Service and Ingress pointing to the workload. Discover an existing nginx ingress controller LoadBalancer address. Does not silently create a chargeable ELB.

4. Inject APM Java Agent (Optional)#

bash

python3 scripts/huawei-cloud.py huawei_inject_cce_apm_javaagent \
  region=cn-north-4 cluster_id=<cluster_id> namespace=<namespace> \
  workload_name=<deployment-name> app_name=<apm-app-name> \
  business=<apm-business> env_name=<apm-env>

Injects the Huawei Cloud APM Java agent into the workload Deployment for distributed tracing during the pressure test.

5. Generate k6 Client#

bash

python3 scripts/huawei-cloud.py huawei_generate_cce_pressure_test_client \
  target_url=http://<elb-address>/<path> namespace=pressure-test \
  model=keepalive vus=10 duration_seconds=60

Generates the k6 ConfigMap and Job manifest. Use a regional SWR mirror for the k6 image when public image pulls are unavailable.

Supported Traffic Models:

ModelPurposeshortDisable connection reuse; observe new-connection pressurekeepaliveReuse connections for a steady HTTP baselinerampIncrease traffic in stages then scale down

6. Run Pressure Test#

bash

# Preview (no mutation)
python3 scripts/huawei-cloud.py huawei_run_cce_pressure_test \
  region=cn-north-4 cluster_id=<cluster_id> target_url=http://<elb-address>/<path>

# Apply after user approval
python3 scripts/huawei-cloud.py huawei_run_cce_pressure_test \
  region=cn-north-4 cluster_id=<cluster_id> target_url=http://<elb-address>/<path> confirm=true

Ensures the client namespace, creates the ConfigMap and Job, waits for completion by default, records replica/Pod count samples, and extracts the k6 summary from logs.

7. Collect Observability#

bash

python3 scripts/huawei-cloud.py huawei_collect_cce_pressure_test_observability \
  region=cn-north-4 cluster_id=<cluster_id> namespace=<namespace> \
  workload_name=<deployment-name> elb_id=<elb-id>

Collects ELB QPS, active connections, layer-7 response time, 2xx ratio, Pod CPU/memory metrics, and AOM evidence. Pass label_selector to scope Pod metrics to the workload. Pod CPU and memory are collected from node metrics when aom_instance_id is not provided.

8. Generate Report#

bash

python3 scripts/huawei-cloud.py huawei_generate_cce_pressure_test_report \
  result_path=<path-to-run-json>

Generates Markdown, HTML, and SVG curve artifacts from the run JSON and optional observability JSON.

9. Scale Workload (Elasticity Evaluation)#

bash

# Preview (no mutation)
python3 scripts/huawei-cloud.py huawei_scale_cce_workload \
  region=cn-north-4 cluster_id=<cluster_id> workload_type=Deployment \
  name=<deployment-name> namespace=<namespace> replicas=5

# Apply after user approval
python3 scripts/huawei-cloud.py huawei_scale_cce_workload \
  region=cn-north-4 cluster_id=<cluster_id> workload_type=Deployment \
  name=<deployment-name> namespace=<namespace> replicas=5 confirm=true

For elasticity evaluation: run a baseline phase, preview scaling, apply only after approval with confirm=true, then run a second phase and compare reports.

10. Supporting Actions (Read-Only)#

ActionRequired ParamsDescriptionhuawei_list_cce_hpas``region, cluster_idList HPA specs and conditionshuawei_get_cce_pods``region, cluster_idList Pod phase, resources, node assignmenthuawei_get_cce_services``region, cluster_idList Service types and cluster IPshuawei_get_cce_ingresses``region, cluster_idList Ingress rules and LoadBalancer IPshuawei_get_cce_pod_metrics_topN``region, cluster_idPod resource metric rankinghuawei_get_elb_metrics``region, elb_idELB QPS, connections, latency, success ratehuawei_get_elb_backend_status``region, elb_idBackend member health statushuawei_list_vpc_subnets``regionList VPC subnets for ELB VIP placementhuawei_list_aom_instances``regionList AOM instanceshuawei_resolve_cce_aom_instance``region, cluster_idResolve AOM instance for a CCE clusterhuawei_get_apm_master_address``regionGet APM master address for agent injectionhuawei_get_aom_metrics``region, aom_instance_id, queryAOM/Prometheus metric querieshuawei_query_aom_logs``region, cluster_idQuery AOM LTS logshuawei_get_cce_logconfigs``region, cluster_idList CCE log collection configshuawei_create_cce_logconfig``region, cluster_id, logconfig_name, source_type, log_group_id, log_stream_idCreate CCE log collection confighuawei_get_application_logconfigs``region, cluster_id, app_nameGet application log configshuawei_query_application_logs``region, cluster_id, app_nameQuery application logshuawei_analyze_application_logs``region, cluster_id, app_nameAnalyze application logs for errorshuawei_generate_monitor_dashboard``region, cluster_idGenerate AOM monitor dashboard URL

Parameter Reference#

Common Parameters#

ParameterRequiredDescriptionDefaultregionYesHuawei Cloud region-cluster_idYes (most actions)CCE cluster ID-namespaceAction-dependentKubernetes namespace-workload_nameAction-dependentDeployment name-akNoAccess Key (overrides env var)HUAWEI_AK envskNoSecret Key (overrides env var)HUAWEI_SK envproject_idNoProject ID (overrides env var)Auto-resolved

Deploy Java Sample Parameters#

ParameterRequiredDescriptionNotesconfirmRequired for mutationApprove deploymenttrue to apply, omit to preview

ELB Creation Parameters#

ParameterRequiredDescriptionNotesnameYesELB name-vip_subnet_cidr_idYesVPC subnet network ID for ELB VIPObtain from huawei_list_vpc_subnets``availability_zone_listNoAZ listComma-separatedl4_flavor_idNoLayer-4 flavor IDOptional performance specl7_flavor_idNoLayer-7 flavor IDOptional performance specconfirmRequired for mutationApprove ELB creationtrue to apply, omit to preview

Route Preparation Parameters#

ParameterRequiredDescriptionNotestarget_portNoContainer target portDefaults to workload's first container porthostNoIngress host rulePass same value as host_header to clientselector_jsonNoPod selector for non-Deployment workloadsJSON stringconfirmRequired for mutationApprove route creationtrue to apply, omit to preview

Client Generation Parameters#

ParameterRequiredDescriptionDefaulttarget_urlYesFull URL to test-namespaceNok6 client namespacepressure-test``modelNoTraffic model ( short, keepalive, ramp)keepalive``vusNoNumber of virtual users10``duration_secondsNoTest duration60``host_headerNoHost header for Ingress host rulesMatches host from route preparation

Traffic Run Parameters#

ParameterRequiredDescriptionNotestarget_urlYesFull URL to testMust match client targetconfirmRequired for mutationApprove traffic runtrue to apply, omit to preview

Observability Collection Parameters#

ParameterRequiredDescriptionNoteselb_idNoELB ID for traffic/connection/latency metricsCritical for full observabilitylabel_selectorNoPod label selectorScope Pod metrics to workloadaom_instance_idNoAOM instance IDAuto-resolved if omitted

APM Injection Parameters#

ParameterRequiredDescriptionNotesapp_nameYesAPM application name-businessYesAPM business name-env_nameYesAPM environment name-

Report Generation Parameters#

ParameterRequiredDescriptionNotesresult_pathYesPath to run JSON fileCombine with observability JSON if available

Common Region IDs#

Region NameRegion IDNorth China - Beijing 4cn-north-4North China - Beijing 1cn-north-1East China - Shanghai 1cn-east-3East China - Shanghai 2cn-east-2South China - Guangzhoucn-south-1South China - Shenzhencn-south-4Southwest China - Guiyang 1cn-southwest-2Asia Pacific - Bangkokap-southeast-2Asia Pacific - Singaporeap-southeast-1Asia Pacific - Hong Kongap-southeast-3Europe - Pariseu-west-0

Output Format#

All actions return JSON with the following common structure:

FieldDescriptionsuccessBoolean: true if operation succeeded, false otherwiseactionAction name that was executedregionHuawei Cloud region

Run JSON ( huawei_run_cce_pressure_test when output_dir is provided):

FieldDescriptionjobJob creation, completion, failure, and timeout statesamplesWorkload replica and Pod count samples collected during the runmetric_seriesStandardized replica curves from the runk6_summaryRequests, RPS, success rate, latency percentiles, bytes, and max VUs parsed from k6 logsdata_gapsMissing evidence that should be called out in the report

Observability JSON ( huawei_collect_cce_pressure_test_observability):

FieldDescriptioninventoryPods, Services, and Ingresses in the workload namespacepod_metricsNode-backed Pod CPU and memory metricselb_metricsELB QPS, connections, latency, and response metrics when elb_id is providedelb_backend_statusBackend member and health-monitor evidencemetric_seriesStandardized curves consumed by the report action

Report Artifacts ( huawei_generate_cce_pressure_test_report):

ArtifactPurpose*-report.mdPortable Markdown report*-report.htmlStyled HTML report with risk-colored recommendations and data-gap tables*-curves.svgEmbedded curve chart for traffic, latency, resources, and replicas

See Output Schema for the full JSON response schema.

Verification#

Step-by-step Verification Checklist#

1. Verify AK/SK credentials are configured via environment variables
2. Inspect existing Services, Ingresses, HPA, ingress controller, and ELB context
3. Confirm cluster, workload, namespace, target port, test window, traffic model, VUs, and output directory
4. Deploy Java sample (preview → confirm) if needed for an isolated lab
5. Create ELB (preview → confirm) if no reusable ingress-controller ELB exists
6. Prepare route (preview → confirm) — Service and Ingress
7. Inject APM Java agent (optional) for distributed tracing
8. Generate k6 client manifest — review ConfigMap and Job
9. Run pressure test (preview → confirm)
10. Collect observability with elb_id for full evidence
11. Generate report and review Markdown/HTML/SVG artifacts
12. For elasticity evaluation: run baseline → scale workload → run second phase → compare reports

Best Practices#

1. Start with low traffic: Always begin with low VUs and short duration to validate the traffic path before ramping up
2. Preview before apply: Always run mutation actions without confirm=true first, review the plan, then re-run with confirm=true after explicit user approval
3. Use regional SWR images for k6: Docker Hub images may timeout; mirror k6 to a regional SWR namespace
4. Pass elb_id for observability: ELB metrics (QPS, connections, latency, success rate) are critical evidence; always pass elb_id when collecting observability
5. Run at least two phases for elasticity: Baseline + scaled phase comparison is the standard methodology for evaluating autoscaling behavior
6. Confirm test target before traffic: Never send traffic to production paths without an approved test window
7. Stop raising traffic when indicators degrade: Halt when success rate drops, latency rises sharply, or resource waterlines exceed agreed limits
8. Do not delete resources automatically: Never auto-delete ELBs, namespaces, workloads, Jobs, or ConfigMaps

Reference Documents#

DocumentDescriptionWorkflowStaged execution, action parameters, and command examplesRisk RulesPreview-first constraints, mutation boundaries, and operational limitsOutput SchemaJSON response schema and report artifact structure

Notes#

• Preview-first by design — Service/Ingress changes, traffic generation, and workload scaling return a preview without confirm=true; apply only after explicit user approval
• Standard traffic path — Pod → Service → nginx-ingress → ELB; the route preparation action discovers existing ingress controllers and does not silently create chargeable resources
• ELB is chargeable — creating a new ELB incurs hourly billing; always preview and confirm before creation
• No credential persistence — AK/SK exists only during API calls; never written to disk, logs, or reports
• Cross-skill escalation — If autoscaling does not respond during tests, hand off to huawei-cloud-cce-autoscaling-diagnoser; if Pods fail, hand off to huawei-cloud-cce-pod-failure-diagnoser

Common Pitfalls#

PitfallSymptomQuick FixPublic Docker Hub k6 imageImage pull timeout in CCEMirror k6 image to regional SWR namespaceMissing ELB in traffic pathk6 client cannot reach workloadCreate ELB or verify ingress-controller LoadBalancer IPIngress host mismatchk6 requests rejected with 404Pass host during route prep and host_header to clientNo observability without elb_idMissing traffic/latency curves in reportAlways pass elb_id to observability collectionSkipping baseline phaseCannot evaluate elasticity improvementRun baseline phase before scalingWrong vip_subnet_cidr_idELB creation failsUse network ID from huawei_list_vpc_subnets, not subnet UUIDProduction traffic without approvalUnintended load on productionConfirm test target and namespace before traffic generation