Huawei Cloud CCE Node Failure Diagnoser#

⚠️ Execution Method (Must Read): This skill executes queries via the local Python dispatcher script. Using hcloud, openstack, or other CLI tools or direct API calls is prohibited.

• The dispatcher script is located at scripts/huawei-cloud.py within the skill directory
• All scripts and environment check scripts are inside the skill package. You must use skill action=exec to execute them. Do not run them directly in a shell.
• Do not attempt hcloud, openstack, curl IAM, or any other CLI/API methods. This skill does not depend on those tools.
• All paths are relative to the skill directory, which is the directory where this SKILL.md is located.

Overview#

This skill diagnoses CCE/Kubernetes node failures and produces a complete Markdown diagnosis report. It uses the local Python dispatcher ( scripts/huawei-cloud.py) to call the Huawei Cloud Python SDK and Kubernetes client APIs, collecting node status, kube-node-lease evidence, events, pod symptoms, and AOM metrics.

The skill covers: node NotReady, disk/memory/CPU pressure, network abnormalities, kubelet/CRI failures, NPD events, and workload impact on the affected node.

Related Skills#

SkillPurposehuawei-cloud-cce-pod-failure-diagnoserPod-level failure diagnosishuawei-cloud-cce-network-failure-diagnoserNetwork failure diagnosishuawei-cloud-cce-storage-failure-diagnoserStorage failure diagnosishuawei-cloud-cce-auto-remediation-runnerExecute remediation actions (cordon, drain, reboot)huawei-cloud-cce-metric-analyzerMetric trend analysishuawei-cloud-cce-observability-context-builderObservability context enrichment

Capabilities#

1. One-shot node failure diagnosis with structured evidence and Markdown report ( huawei_node_failure_diagnose)
2. Kubernetes node status and conditions collection ( huawei_get_kubernetes_nodes)
3. Node/Pod event timeline retrieval ( huawei_get_cce_events)
4. Pod phase/reason/state aggregation per node ( huawei_get_cce_pods)
5. Node and Pod AOM metric queries ( huawei_get_cce_node_metrics, huawei_get_cce_pod_metrics_topN)
6. Node inspection items (status, resource, vulnerability) ( huawei_node_status_inspection, huawei_node_resource_inspection, huawei_node_vul_inspection)
7. Security group and HSS vulnerability correlation ( huawei_list_security_groups, huawei_hss_list_hosts, huawei_hss_list_host_vuls_all)

Typical Use Cases#

• Diagnose a CCE node that transitioned to NotReady state
• Investigate node disk or memory pressure conditions
• Analyze kubelet or container runtime failures on a node
• Check node network connectivity issues (CNI sandbox failures)
• Assess pod impact when a node becomes unhealthy
• Review NPD events and node-level security findings

---

Prerequisites#

Python Dependencies#

The dispatcher script requires Python >= 3.6 and the following packages:

• huaweicloudsdkcore
• huaweicloudsdkcce
• huaweicloudsdkaom
• huaweicloudsdkhss
• huaweicloudsdkvpc
• huaweicloudsdkecs
• huaweicloudsdkces
• huaweicloudsdkevs
• huaweicloudsdkeip
• huaweicloudsdkelb
• huaweicloudsdkiam
• kubernetes

Credential Configuration#

VariableRequiredDescriptionHUAWEI_AKYesHuawei Cloud Access KeyHUAWEI_SKYesHuawei Cloud Secret KeyHUAWEI_REGIONNoDefault region (overrides region param if set)HUAWEI_PROJECT_IDNoProject ID (auto-obtained via IAM API when not set)HUAWEI_SECURITY_TOKENNoRequired when using temporary AK/SK

🚫 Never expose or log AK/SK values. Credentials exist only in the current request call stack and are released after each invocation. Do not write credentials to files, logs, or responses.

✅ Use environment variables HUAWEI_AK / HUAWEI_SK for authentication. The dispatcher reads them automatically.

IAM Permissions#

PermissionServiceRequired ForCCE cluster/node readCCEhuawei_list_cce_nodes, huawei_get_cce_nodes, huawei_node_diagnoseKubernetes API readCCE (kubeconfig)huawei_get_kubernetes_nodes, huawei_node_failure_diagnoseAOM metrics readAOMhuawei_get_cce_node_metrics, huawei_get_cce_pod_metrics_topNCES alarm readCEShuawei_get_cce_eventsHSS host/vul readHSShuawei_hss_list_hosts, huawei_hss_list_host_vuls_allVPC/SG readVPChuawei_list_security_groups, huawei_list_vpc_acls

---

Core Tools#

All actions are invoked via the dispatcher script:

bash

python3 scripts/huawei-cloud.py <action> region=<region> cluster_id=<cluster_id> [key=value ...]

Primary Diagnosis Action#

bash

python3 scripts/huawei-cloud.py huawei_node_failure_diagnose \
  region=cn-north-4 cluster_id=<cluster_id> \
  node_name=<node_name> lease_timeout_seconds=40 \
  event_limit=500 hours=1 include_metrics=true

Returns structured evidence + report_markdown (complete Markdown diagnosis report).

Evidence Collection Actions#

ActionRequired ParamsDescriptionhuawei_get_kubernetes_nodes``region, cluster_idQuery v1.Node Ready/conditions statushuawei_get_cce_events``region, cluster_idRetrieve Kubernetes eventshuawei_get_cce_pods``region, cluster_idList Pod phase/reason/lastStatehuawei_get_cce_node_metrics``region, cluster_id, node_ipQuery node CPU/memory/disk metricshuawei_get_cce_node_metrics_topN``region, cluster_idTop-N node metricshuawei_get_cce_pod_metrics_topN``region, cluster_idTop-N pod metrics (supports node_ip filter)

Inspection Actions#

ActionRequired ParamsDescriptionhuawei_node_status_inspection``region, cluster_idNode status health inspectionhuawei_node_resource_inspection``region, cluster_idNode resource utilization inspectionhuawei_node_vul_inspection``region, cluster_idNode vulnerability inspection

Security Correlation Actions#

ActionRequired ParamsDescriptionhuawei_list_security_groups``regionList VPC security groupshuawei_list_vpc_acls``regionList VPC network ACLshuawei_hss_list_hosts``regionList HSS host security statushuawei_hss_list_host_vuls_all``region, host_idList all vulnerabilities for a host

---

Parameter Reference#

huawei_node_failure_diagnose#

ParameterRequiredDefaultDescriptionregionYes-Huawei Cloud region (e.g., cn-north-4)cluster_idYes-CCE cluster IDnode_nameNo*-Target node name (one of node_name or node_ip required)node_ipNo*-Target node internal IP (one of node_name or node_ip required)lease_timeout_secondsNo40Kube-node-lease stale threshold in secondsevent_limitNo500Maximum events to retrievehoursNo1Metric lookback window in hoursinclude_metricsNotrueWhether to include AOM metrics

*At least one of node_name or node_ip must be provided. If both are omitted, the action returns an error.

Common Parameters#

ParameterRequiredDescriptionregionYesHuawei Cloud regioncluster_idYes (most actions)CCE cluster IDnode_ipRequired for huawei_get_cce_node_metricsNode internal IPtop_nNoNumber of top results (default 10)hoursNoMetric lookback hours (default 1)

---

Output Format#

The primary action huawei_node_failure_diagnose returns structured evidence and a Markdown report. See references/output-schema.md for the full JSON response schema.

Key output fields:

FieldDescriptionsuccessWhether the diagnosis completed successfullynodeNode name, IP, Ready status, conditionsleasekube-node-lease renew time, stale status, delay secondslivenessControl plane liveness case (A/B/C/D) and conclusionroot_categoryRoot cause category (ControlPlaneDisconnected, MemoryPressure, DiskPressure, Network, Kubelet, NotReady, Healthy)confidenceConfidence level (High/Medium/Low)evidenceList of evidence items with category, severity, signal, source, detailpod_summaryPod phase counts and symptomatic pod listhealth_itemsNode health check items with statusreport_markdownComplete Markdown diagnosis report (use as final output)

When report_markdown is present, use it as the final report body. You may add clarifications the user requests, but do not discard evidence tables.

---

Verification#

1. Run the dispatcher with a known cluster and node to confirm connectivity: bash

   python3 scripts/huawei-cloud.py huawei_get_kubernetes_nodes region=cn-north-4 cluster_id=<cluster_id>
   
   

2. Execute huawei_node_failure_diagnose on a healthy node; expect root_category=Healthy and confidence=High

3. Verify report_markdown contains all required sections (see references/output-schema.md)

4. Compare node conditions in the output with the CCE console

---

Best Practices#

1. Always call huawei_node_failure_diagnose first; use manual fallback actions only if the primary action fails
2. When Ready=Unknown and lease is stale, conclude "control plane disconnected from node" rather than prematurely attributing to kubelet or network alone
3. When pressure conditions are Unknown with NodeStatusUnknown reason, label them "indeterminate" — do not mark as "normal"
4. Correlate Event signals with Pod symptoms before forming conclusions; evidence strength determines confidence level
5. Include security group and HSS checks only when network or vulnerability hypotheses are strong
6. Do not single-point metric peaks as root cause; validate with trend data

---

Reference Documents#

DocumentDescriptionreferences/workflow.mdDiagnosis triage flow, evidence rules, and fallback workflowreferences/output-schema.mdOutput JSON schema and required Markdown report sectionsreferences/risk-rules.mdRisk boundary rules: allowed read actions, prohibited write actionsHuawei Cloud Python SDK DocumentationSDK referenceHuawei Cloud API ExplorerAPI interactive explorer

---

Notes#

1. This skill is read-only diagnosis only — it does not cordon, uncordon, drain, reboot, or modify vulnerability status
2. When remediation actions are needed, hand off to huawei-cloud-cce-auto-remediation-runner and require user confirmation
3. Never expose or log AK/SK or environment variable values
4. All actions are executed via python3 scripts/huawei-cloud.py <action>; do not use hcloud CLI or direct API calls
5. If the primary action huawei_node_failure_diagnose fails, follow the manual fallback workflow in references/workflow.md

---

Common Pitfalls#

PitfallCorrect ApproachConcluding "kubelet failure" when Ready=Unknown + lease staleConclude "control plane disconnected from node (network or kubelet/CRI heartbeat interrupted, requires node-side verification)"Marking Unknown pressure conditions as "normal"Label as "indeterminate — no independent evidence available"Using a single metric spike as root causeValidate with trend data over time; use hours parameter for lookbackSkipping event/pod correlationAlways cross-reference Event signals with Pod symptoms before forming conclusionsExecuting cordon/drain/reboot directlyThis skill does not perform write actions; hand off to huawei-cloud-cce-auto-remediation-runnerIgnoring CNI sandbox failures in Pod eventsFailedCreatePodSandBox + CNI error patterns are strong network abnormality evidenceNot checking kube-node-lease when node is NotReadyLease staleness is critical evidence for control plane connectivity