Huawei Cloud CCE Network Failure Diagnoser#

⚠️ Execution Method (Must Read): This skill executes diagnosis via local Python scripts using a dispatcher pattern. Using hcloud, openstack, or other CLI tools or direct API calls is prohibited.

• The dispatcher script is scripts/huawei-cloud.py, invoked as python3 scripts/huawei-cloud.py <action> <key=value params>
• All scripts and environment check scripts are inside the skill package. You must use skill action=exec to execute them. Do not run them directly in a shell.
• For action details and parameters, refer to references/workflow.md, references/risk-rules.md, and references/output-schema.md
• Do not attempt hcloud, openstack, curl IAM, or any other CLI/API methods. This skill does not depend on those tools.
• All paths are relative to the skill directory, which is the directory where this SKILL.md is located.

Overview#

This skill diagnoses CCE (Cloud Container Engine) network failures by performing a layered, read-only diagnosis across the full network stack — from node infrastructure, DNS, Service/EndpointSlice, NetworkPolicy, Ingress to cloud-side ELB/EIP/NAT/VPC security policies. It produces a complete Markdown diagnosis report that must include the investigation process, evidence, conclusions, confidence levels, and verification criteria.

Use this skill when:

1. CCE Service connectivity is broken (Service unreachable, intermittent, or flapping)
2. DNS/CoreDNS resolution failures (NXDOMAIN, timeout)
3. Ingress 502/504 errors or ELB backend health issues
4. NetworkPolicy blocking traffic between Pods
5. VPC/Subnet/Security Group/ACL configuration affecting cluster networking
6. EIP/NAT gateway affecting external access from the cluster

This skill does NOT handle:

1. Creating, modifying, or deleting any resources
2. Binding/unbinding EIP or modifying security groups/ACLs/ELB listeners
3. Scaling workloads or restarting components
4. Pod-level or Node-level root causes (cross-reference to huawei-cloud-cce-pod-failure-diagnoser, huawei-cloud-cce-node-failure-diagnoser, huawei-cloud-cce-workload-failure-diagnoser)

---

Prerequisites#

You must run the environment check script first to complete environment validation and dependency installation in one step:

• Linux / macOS: skill action=exec: bash skill://scripts/check_env.sh
• Windows: skill action=exec: powershell -ExecutionPolicy Bypass -File skill://scripts/check_env.ps1

Windows note: Do not use && to chain commands (PowerShell 5.x does not support it); use semicolons if you need to change directories first.

The script will check in order: Python >= 3.6 → install dependencies → validate SDK → validate credentials → validate service availability. If the environment check fails, fix the issues before proceeding.

Environment Variables:

VariableRequiredDescriptionHW_ACCESS_KEYYesHuawei Cloud AK (Access Key)HW_SECRET_KEYYesHuawei Cloud SK (Secret Key)HW_REGION_NAMENoDefault cn-north-4HW_PROJECT_IDNoProject ID (automatically obtained via IAM API when not set)HW_SECURITY_TOKENNoRequired when using temporary AK/SKHW_CCE_CLUSTER_IDYesCCE cluster ID for diagnosis targetKUBECONFIGNoKubernetes config; auto-obtained from CCE API if not set

Security Constraints:

1. Never persist AK/SK/Token/Certificate to filesystem
2. AK/SK exists only in the current call stack; released after call ends
3. Only non-sensitive project IDs may be cached in process memory (never written to disk)
4. All temporary certificate files must be deleted immediately after use
5. Never leak AK/SK in logs, responses, or error messages
6. Never send credentials to any third-party server

Do not output the values of environment variables.

---

IAM Permission Requirements#

API ActionPermissionPurposecce:cluster:getGet clusterView cluster detailscce:cluster:listList clustersList CCE clusterscce:node:listList nodesList cluster nodesvpc:vpc:listList VPCsQuery VPC detailsvpc:subnet:listList subnetsQuery subnet detailselb:loadbalancer:listList ELBsQuery ELB detailselb:listener:listList listenersQuery ELB listenersaom:*:getRead AOMQuery AOM metrics and alarms

Permission Failure Handling:

1. When any command fails due to permission errors, display required permission list
2. Guide the user to create a custom policy in the IAM console
3. Pause execution and wait for user confirmation

---

Core Tools#

All actions are invoked via the Python dispatcher script:

text

python3 scripts/huawei-cloud.py <action> region=<region> cluster_id=<cluster_id> namespace=<namespace> [other_params...]

Execution via skill:

• Linux / macOS: skill action=exec: skill://.venv/bin/python3 skill://scripts/huawei-cloud.py <action> <params>
• Windows: skill action=exec: skill://.venv/Scripts/python3.exe skill://scripts/huawei-cloud.py <action> <params>

Primary Diagnosis Action#

ActionDescriptionhuawei_network_failure_diagnoseOne-shot diagnosis: collects K8s and cloud-side read-only snapshots, returns structured findings + report_markdown

Kubernetes Evidence Actions#

ActionDescriptionhuawei_get_cce_servicesList Services in a namespacehuawei_get_cce_ingressesList Ingresses in a namespacehuawei_get_cce_podsList Pods in a namespacehuawei_get_kubernetes_nodesList cluster Nodeshuawei_get_cce_eventsList cluster Eventshuawei_get_pod_logsRetrieve Pod container logs

Cloud Network Evidence Actions#

ActionDescriptionhuawei_get_elb_backend_statusRead ELB pool/member/health monitor/load balancer statushuawei_get_elb_metricsRetrieve ELB monitoring metricshuawei_list_elbList ELB load balancershuawei_list_elb_listenersList ELB listenershuawei_list_eipList EIP addresseshuawei_get_eip_metricsRetrieve EIP monitoring metricshuawei_list_natList NAT gatewayshuawei_get_nat_gateway_metricsRetrieve NAT gateway metricshuawei_list_security_groupsList VPC security groupshuawei_list_vpc_aclsList VPC ACLs

Legacy Compatibility Actions#

ActionDescriptionhuawei_network_diagnoseLegacy comprehensive network diagnosishuawei_network_diagnose_by_alarmDiagnosis triggered by alarm correlationhuawei_network_verify_pod_schedulingVerify Pod scheduling constraints (read-only)

---

Parameter Reference#

Required Parameters#

ParameterDescriptionregionHuawei Cloud region, e.g., cn-north-4``cluster_idCCE cluster IDnamespaceKubernetes namespace

Optional Parameters (provide as many as possible for accurate diagnosis)#

ParameterDescriptionfailure_symptomSymptom description: domain_unresolvable, in_cluster_service_unreachable, service_intermittent, external_access_failed, ingress_502_504``target_kindResource type: Pod, Service, Ingress, etc.target_nameResource nameservice_nameTarget Service nameingress_nameTarget Ingress namesource_podSource Pod name or labeldestination_podDestination Pod name or labeldomainDomain name for DNS diagnosiselb_idELB load balancer ID

---

Output Format#

huawei_network_failure_diagnose returns structured JSON with an embedded report_markdown:

json

{
  "success": true,
  "action": "huawei_network_failure_diagnose",
  "region": "cn-north-4",
  "cluster_id": "cluster-id",
  "namespace": "default",
  "conclusion": "high signal conclusion",
  "confidence": "High",
  "pipeline_pruned": false,
  "findings": [
    {
      "stage": "Stage 3: East-West Routing and Policy Layer",
      "type": "NetworkPolicyBlocked",
      "title": "NetworkPolicy selects target Pod but does not allow source Pod labels or target port",
      "confidence": 1.0,
      "severity": "critical",
      "evidence": [],
      "recommendation": [],
      "prune": false
    }
  ],
  "top_causes": [],
  "snapshot": {
    "inputs": {},
    "nodes": [],
    "pods": [],
    "services": [],
    "ingresses": [],
    "endpoint_slices": [],
    "network_policies": [],
    "events": [],
    "logs": {},
    "cloud": {
      "elb_ids": [],
      "elbs": {},
      "eips": {},
      "nat": {},
      "security_groups": {},
      "vpc_acls": {}
    }
  },
  "report_markdown": "# CCE Network Failure Automated Diagnosis Report\n..."
}

Markdown Report Sections#

The report_markdown must contain the following headings:

1. Diagnosis Overview — target, symptom, conclusion, confidence, collection time, pruned stages
2. Investigation Process — per-stage status (checked, abnormal, pruned/skipped)
3. Link Topology — DNS path, east-west path, or north-south path based on failure type
4. Key Object Snapshot — Service, EndpointSlice, Backend Pods, Ingress, NetworkPolicy, Cloud ELB
5. Evidence Matrix — stage, type, confidence, evidence summary
6. Diagnosis Conclusion — top root causes (max 3), each backed by evidence
7. Recommended Actions and Verification Criteria — read-only verification steps or change suggestions to hand off to huawei-cloud-cce-auto-remediation-runner

Finding Types#

Common type values in findings:

TypeDescriptionNodeUnhealthyNode Ready=False or Ready=UnknownNodePressureMemory/Disk/PID/Network pressure on nodePodDNSConfigMissingPod dnsPolicy=None with no dnsConfigKubeDnsNoEndpointkube-dns EndpointSlice has 0 ready endpointsCoreDNSRestartingCoreDNS pods showing OOMKilled/LivenessProbe failuresCoreDNSNxDomainCoreDNS logs showing NXDOMAIN responsesCoreDNSUpstreamTimeoutCoreDNS logs showing upstream i/o timeoutNetworkPolicyBlockedNetworkPolicy blocks source Pod traffic (confidence 100%)ServiceNoReadyEndpointService has 0 ready endpoints in EndpointSliceServiceSelectorMismatchService selector matches no PodsReadinessFlappingBackend Pod readiness probe flappingBackendOverloadedApplication logs show OOM/connection pool exhaustedLoadBalancerProvisioningFailedLoadBalancer Ingress status empty with CCM errorsELBBackendUnhealthyELB member unhealthy while K8s backend Pod is ReadyIngressUpstreamErrorIngress controller logs show 502/504

---

Verification#

1. Run the environment check script to confirm dependencies and credentials are available
2. Execute huawei_network_failure_diagnose with a known-healthy cluster and verify the report structure
3. Cross-reference findings with Huawei Cloud console data (ELB health, security groups, VPC ACLs)
4. Verify pipeline_pruned flag is set correctly when node-level issues prune upper layers
5. Confirm that confidence and severity values are present in all findings

---

Best Practices#

1. Always provide failure_symptom to direct the diagnosis pipeline to the relevant stage (DNS, east-west, or north-south)
2. Provide as many optional parameters as possible ( service_name, ingress_name, source_pod, destination_pod, domain) for more precise diagnosis
3. Start with huawei_network_failure_diagnose for one-shot comprehensive diagnosis; use individual actions only for targeted follow-up queries
4. When evidence is insufficient, state "evidence insufficient" explicitly — never present guesses as conclusions
5. For north-south (external access) issues, always supplement with huawei_get_elb_backend_status and huawei_list_security_groups to check cloud-side configuration
6. When node-level issues are found, note that upper-layer diagnosis may be pruned; cross-reference with huawei-cloud-cce-node-failure-diagnoser

---

Reference Documents#

• Diagnosis workflow, reuse priorities, and layered pipeline: references/workflow.md
• Risk rules and action boundaries: references/risk-rules.md
• Output schema and finding type reference: references/output-schema.md

---

Notes#

1. This skill is strictly read-only; it never modifies Service, Ingress, NetworkPolicy, CoreDNS ConfigMap, security groups, ACLs, ELB listeners/backends, EIP bindings, or NAT rules
2. Never execute kubectl exec, packet capture, stress testing, or active traffic injection unless the user explicitly requests and acknowledges the risk
3. huawei_network_verify_pod_scheduling is for verification only; it does not replace scaling actions
4. Any network change suggestion must describe impact scope, rollback method, and verification criteria, and be handed off to huawei-cloud-cce-auto-remediation-runner for preview
5. Do not output the values of environment variables such as HW_ACCESS_KEY, HW_SECRET_KEY, HW_SECURITY_TOKEN
6. All scripts must be executed via skill action=exec; do not run them directly in a shell

---

Common Pitfalls#

1. Missing cluster_id: The cluster_id parameter is required for all CCE actions. If the user only provides a cluster name, query huawei_list_cce_clusters first to resolve the ID
2. Wrong failure_symptom: Using a wrong symptom category (e.g., ingress_502_504 for an in-cluster issue) may misdirect the pipeline. Always confirm the symptom type with the user
3. Ignoring node-level root cause: If nodes are NotReady, upper-layer diagnosis may be pruned. Do not skip the node-layer check even when the symptom appears to be Service/DNS-level
4. Confusing K8s-side and cloud-side: ELB backend unhealthy does not always mean the K8s Pod is unhealthy — check both huawei_get_elb_backend_status and huawei_get_cce_pods together
5. Over-interpreting insufficient evidence: When EndpointSlice has 0 ready endpoints, it could be selector mismatch, readiness flapping, or Pod crash. Do not jump to conclusions without checking Pod events and logs
6. Not checking NetworkPolicy for east-west issues: NetworkPolicy blocking has 100% confidence when confirmed, but is easily overlooked. Always check NetworkPolicy in the target namespace