CCE Change Impact Analyzer#

⚠️ Execution Method (Must Read): This skill executes diagnosis via local Python scripts using the scripts/huawei-cloud.py dispatcher. Using hcloud, kubectl, or other CLI tools or direct API calls is prohibited.

• All actions are dispatched through scripts/huawei-cloud.py with --action <action_name> and --params <json_params>
• All scripts and environment check scripts are inside the skill package. You must use skill action=exec to execute them; do not run them directly in a shell
• For action names and parameters, see the Core Tools section below
• Do not attempt hcloud, kubectl, curl IAM, or other CLI/API methods. This skill does not depend on these tools
• All paths are relative to the skill directory, which is the directory where this SKILL.md resides

Overview#

This skill turns "what changed before the incident" into provable causal attribution. It ingests audit logs, K8s historical events, AOM active+history alarms, and current resource topology snapshots; filters noise; maps core changes to blast radius; scores risk by sensitivity, topology scope, security boundary span, temporal proximity to fault, and event/alarm correlation; then outputs a complete Markdown report with investigation steps, core change timeline, evidence matrix, blast radius, Top N risk alerts, conclusion, and data gaps.

This skill is applicable to the following scenarios:

1. Incidents where recent workload releases, config updates, or network/security policy changes may be the cause
2. CoreDNS, kube-proxy, or cluster plugin configuration changes causing business-wide failures
3. Node taint, cordon/drain, node pool resize, or cluster upgrade triggering Pod Pending, Evicted, or NotReady
4. NetworkPolicy/RBAC changes causing connection timeouts, 403 errors, DNS anomalies, or cross-namespace access failures
5. Service/Ingress/Gateway route changes causing traffic routing failures
6. Correlating audit trail changes with observed failures and alarm timelines

This skill does NOT handle the following:

1. Executing any remediation actions (rollback, scale, delete, drain, reboot, modify NetworkPolicy/RBAC/Security Group/VPC ACL)
2. Making causal conclusions from object updates alone without temporal or response-signal correlation
3. Creating, modifying, or deleting CCE resources
4. Guessing or fabricating diagnosis results without evidence

---

Prerequisites#

Before using, you must run the environment check script to complete environment validation and dependency installation in one step:

• Linux / macOS: skill action=exec: bash skill://scripts/check_env.sh
• Windows: skill action=exec: powershell -ExecutionPolicy Bypass -File skill://scripts/check_env.ps1

Windows Note: Do not use && to chain commands (PowerShell 5.x does not support it). Use semicolons if you need to change directories first.

The script will check in sequence: Python >= 3.6 → install dependencies → validate SDK → validate credentials → validate service availability. If the environment check fails, fix the issues before continuing with other actions.

Environment Variables:

VariableRequiredDescriptionHW_ACCESS_KEYYesHuawei Cloud AKHW_SECRET_KEYYesHuawei Cloud SKHW_REGION_NAMENoDefault cn-north-4HW_PROJECT_IDNoProject ID (automatically obtained via IAM API when not set)HW_SECURITY_TOKENNoRequired when using temporary AK/SKHW_CLUSTER_IDNoDefault CCE cluster ID (can also be passed per action)

Security Constraints:

1. Never persist credentials (AK/SK/Token/Certificate) to the filesystem
2. AK/SK exist only within the current request call stack; released after use
3. Only non-sensitive project IDs are cached in process memory (never written to disk)
4. All temporary certificate files must be deleted immediately after use
5. Never expose AK/SK in logs, responses, or error messages

Do not output the values of the above environment variables.

---

IAM Permission Requirements#

API ActionPermissionPurposecce:cluster:getGet clusterView cluster detailscce:cluster:listList clustersList CCE clusterscce:node:listList nodesList cluster nodescce:nodepool:listList node poolsList node poolsaom:*:getRead AOMQuery AOM metrics and alarmsaom:alarmRule:listList alarm rulesQuery alarm rulesaom:event:listList eventsQuery AOM alarm events

Permission Failure Handling:

1. When any command fails due to permission errors, display required permission list
2. Guide the user to create a custom policy in the IAM console
3. Pause execution and wait for user confirmation

---

Core Tools#

All actions are dispatched through scripts/huawei-cloud.py using skill action=exec.

Primary Change Impact Analysis#

ActionRequired ParametersDescriptionhuawei_change_impact_analyzeregion, cluster_idPrimary comprehensive action: orchestrates audit log ingestion, K8s event correlation, AOM alarm correlation, resource snapshot collection, noise filtering, blast radius modeling, and risk scoring into a unified change impact report with Top N risk alerts

Audit and Event Collection#

ActionRequired ParametersDescriptionhuawei_query_cce_audit_logsregion, cluster_idQuery CCE Kubernetes audit logs for create/update/patch/delete operations with actor, verb, resource, namespace, name, requestURI, statusCodehuawei_query_k8s_events_from_ltsregion, cluster_idQuery historical K8s Events from LTS (overcomes the K8s API short event window)huawei_get_cce_eventsregion, cluster_idList current Kubernetes Events when LTS is unavailable

Alarm Correlation#

ActionRequired ParametersDescriptionhuawei_analyze_aom_alarmsregion, cluster_idAnalyze AOM active + history alarm patterns and correlation across resources

Domain Drill-Down (Read-Only)#

ActionRequired ParametersDescriptionhuawei_workload_rollout_diagnoseregion, cluster_id, namespace, kind, nameDrill down when changes point to Deployment/StatefulSet/DaemonSet rollout failures (cross-skill: huawei-cloud-cce-workload-failure-diagnoser)huawei_network_failure_diagnoseregion, cluster_idDrill down when changes point to Service/Ingress/NetworkPolicy/ELB connectivity failures (cross-skill: huawei-cloud-cce-network-failure-diagnoser)huawei_node_failure_diagnoseregion, cluster_idDrill down when changes point to Node taint, NotReady, scheduling, or resource pressure (cross-skill: huawei-cloud-cce-node-failure-diagnoser)

Current Topology Snapshots#

ActionRequired ParametersDescriptionhuawei_get_cce_podsregion, cluster_idList current Pod status for blast radius modelinghuawei_get_cce_deploymentsregion, cluster_idList current Deployment statushuawei_get_cce_servicesregion, cluster_idList current Service selector/ports for impact mappinghuawei_get_cce_ingressesregion, cluster_idList current Ingress rules/backends for impact mappinghuawei_get_kubernetes_nodesregion, cluster_idList current Node status for taint/impact mappinghuawei_list_cce_configmapsregion, cluster_idList current ConfigMap objects (identify CoreDNS, kube-proxy, business configs)huawei_list_cce_secretsregion, cluster_idList current Secret objectshuawei_list_cce_nodepoolsregion, cluster_idList current NodePool status for infrastructure change context

Cloud Network Snapshots#

ActionRequired ParametersDescriptionhuawei_list_security_groupsregionList current Security Group rules for cloud network contexthuawei_list_vpc_aclsregionList current VPC ACL rules for cloud network context

---

Parameter Reference#

Common Parameters:

ParameterRequiredDescriptionregionYesHuawei Cloud region, e.g., cn-north-4cluster_idYesCCE cluster ID

Optional Parameters (passed via --params JSON):

ParameterDescriptionhoursAnalysis window in hours (default 1)start_timeAnalysis window start (YYYY-MM-DD HH:MM:SS), alternative to hoursend_timeAnalysis window end (YYYY-MM-DD HH:MM:SS), alternative to hoursnamespaceNarrow scope to a namespace, but do not exclude kube-system/CoreDNS global changestarget_nameTarget object name for scope narrowingworkload_nameWorkload name for scope narrowingapp_nameApplication name for scope narrowingfault_timeIncident time point for temporal proximity scoringincident_timeAlternative to fault_timelog_group_idAudit log group ID (manual fallback when auto-discovery fails)log_stream_idAudit log stream ID (manual fallback when auto-discovery fails)include_auditEnable/disable audit log collection (default true)include_k8s_eventsEnable/disable K8s event collection (default true)include_aomEnable/disable AOM alarm collection (default true)include_snapshotsEnable/disable resource snapshot collection (default true)top_nNumber of top risk alerts in report (default 3)output_filePath to write the Markdown report fileakOverride AK (uses HW_ACCESS_KEY by default)skOverride SK (uses HW_SECRET_KEY by default)project_idOverride project ID (auto-obtained via IAM when not set)

---

Output Format#

Primary: huawei_change_impact_analyze#

Returns structured JSON with embedded report_markdown. See references/output-schema.md for full schema.

json

{
  "success": true,
  "analysis_trace_id": "CIA-yyyymmddHHMMSS-xxxxxxxx",
  "analysis_window": {
    "start_time": "YYYY-MM-DD HH:MM:SS",
    "end_time": "YYYY-MM-DD HH:MM:SS",
    "hours": 1
  },
  "scope": {
    "region": "cn-north-4",
    "cluster_id": "cluster-id",
    "namespace": "optional",
    "target_name": "optional"
  },
  "summary": {
    "core_change_count": 3,
    "top_risk_count": 3,
    "data_sources": {
      "CCE Audit Logs": "success",
      "K8s Historical Events": "success",
      "AOM Alarms": "success",
      "Current Resource Snapshots": "success"
    }
  },
  "top_changes": [
    {
      "time": "YYYY-MM-DD HH:MM:SS",
      "verb": "patch",
      "resource": "configmaps",
      "namespace": "kube-system",
      "name": "coredns",
      "object_key": "kube-system/coredns",
      "category": "global_config_change",
      "title": "Cluster core configuration change",
      "actor": "user or serviceAccount",
      "semantic_fields": ["data", "Corefile"],
      "blast_radius": "cluster-wide",
      "impacted_entities": {
        "pods": [],
        "services": ["kube-system/kube-dns"],
        "ingresses": [],
        "nodes": ["node-a"]
      },
      "risk_score": 96,
      "risk_level": "Critical",
      "confidence": "high",
      "risk_reasons": [],
      "evidence": []
    }
  ],
  "changes": [],
  "report_markdown": "# CCE Change Impact Analysis Report\n...",
  "report_file": "/optional/path/report.md",
  "capture_metadata": {}
}

---

Verification#

1. Run the environment check script to confirm dependencies and credentials are available
2. Use huawei_change_impact_analyze on a known stable cluster to verify it returns success: true with zero or low-confidence core changes
3. Use huawei_change_impact_analyze on a cluster with known recent changes to verify Top N risk alerts are accurately identified
4. Verify that noise filtering correctly excludes HPA replica-only updates, controller status writes, Lease/Token/status subresource writes
5. Verify that CoreDNS/kube-proxy/kube-system changes are always included regardless of namespace scope
6. Verify that blast radius mapping correctly traces Service selector → Pod → Ingress → Node propagation
7. Confirm that low-confidence conclusions are clearly labeled with data gaps

---

Best Practices#

1. Always start with huawei_change_impact_analyze for comprehensive change correlation; drill down into domain diagnoser actions only when specific evidence requires deeper analysis
2. Find changes first, then map impact, then align with alarms/events/fault time — do not conclude root cause from object updates alone
3. CoreDNS, kube-proxy, network plugin, and Ingress controller config changes in kube-system must always be included in business fault analysis regardless of target namespace scope
4. Deployment HPA-only replicas adjustments are noise; image, startup args, probe, resource spec, environment variable, and ConfigMap/Secret reference changes are core changes
5. NetworkPolicy/RBAC changes must be correlated with connection timeouts, 403, DNS anomalies, and cross-namespace access failures
6. Node taint, cordon/drain, node pool resize, and cluster upgrade changes must be correlated with Pod Pending, Evicted, NotReady, and node events
7. All remediation actions must be output as recommendations only and handed off to huawei-cloud-cce-auto-remediation-runner
8. Clearly label low-confidence conclusions with required supplementary data; never present speculation as fact

---

Reference Documents#

• Four-stage pipeline and risk scoring rules: references/workflow.md
• Reusable capabilities, gaps, and suggested atomic actions: references/capability-map.md
• Output field specification and Markdown template: references/output-schema.md
• Read-only boundaries and remediation handoff rules: references/risk-rules.md
• Huawei Cloud CCE Documentation
• Huawei Cloud Python SDK Documentation

---

Notes#

1. This skill is read-only analysis and report generation only; no modification of workloads, rollback, ConfigMap/Secret changes, Security Group/ACL/NetworkPolicy/RBAC adjustments, or node cordon/drain/reboot operations
2. Do not output the values of HW_ACCESS_KEY, HW_SECRET_KEY, HW_SECURITY_TOKEN, or other environment variables
3. All scripts must be executed via skill action=exec; do not run them directly in a shell
4. Any remediation action must be handed off to huawei-cloud-cce-auto-remediation-runner; this skill never executes remediation
5. The environment check script must be run before any analysis action
6. When using temporary AK/SK, HW_SECURITY_TOKEN must be set
7. Cross-skill references: remediation → huawei-cloud-cce-auto-remediation-runner; comprehensive root cause → huawei-cloud-cce-root-cause-analyzer; workload diagnosis → huawei-cloud-cce-workload-failure-diagnoser; network diagnosis → huawei-cloud-cce-network-failure-diagnoser; node diagnosis → huawei-cloud-cce-node-failure-diagnoser

---

Common Pitfalls#

1. Concluding root cause from object updates alone — Always require temporal proximity, event/alarm response, and topology impact evidence; an object update without correlation is insufficient evidence
2. Excluding kube-system changes when scoped to a namespace — CoreDNS, kube-proxy, and cluster plugin changes are global even when the target namespace is different; always include them
3. Treating HPA replica updates as core changes — HPA-only replicas modifications are noise; only image, probe, resource, env, config reference changes are core
4. Not correlating NetworkPolicy/RBAC with connectivity symptoms — NetworkPolicy/RBAC changes must be cross-referenced with connection timeout, 403, DNS anomaly, and cross-namespace access failure events
5. Attempting remediation actions from this skill — All changes must be handed off to huawei-cloud-cce-auto-remediation-runner; this skill only outputs recommendations
6. Failing to label low-confidence conclusions — When evidence is insufficient, write "insufficient evidence" explicitly with data gaps; never present guesses as conclusions
7. Ignoring controller and platform noise — Lease, Token, status subresource, Node status patch, scheduler binding, and CCE platform-managed RBAC updates must all be filtered out; they are control-plane closed-loop operations, not user changes
8. Not building a fault timeline — Establish user-perceived fault time, alarm trigger time, Kubernetes event time, and change time before scoring risk