Kafka Instance Capacity Assessment#

Based on the incident information provided by the user, this skill queries CloudMonitor metrics and instance metadata to determine whether a Kafka instance has reached a capacity bottleneck, and provides upgrade recommendations accordingly.

Architecture: Kafka Instance (v2/v3 Series) + CloudMonitor (CMS) + Aliyun CLI

---

Pre-check: Aliyun CLI >= 3.3.3 required#

Run aliyun version to verify >= 3.3.3. If not installed or version too low, run curl -fsSL https://aliyuncli.alicdn.com/setup.sh | bash to install/update, or see references/cli-installation-guide.md for installation instructions.

Pre-check: Aliyun CLI plugin update required

[MUST] run aliyun configure set --auto-plugin-install true to enable automatic plugin installation. [MUST] run aliyun plugin update to ensure that any existing plugins are always up-to-date.

---

Authentication#

Pre-check: Alibaba Cloud Credentials Required

Security Rules:

• NEVER read, echo, or print AK/SK values (e.g., echo $ALIBABA_CLOUD_ACCESS_KEY_ID is FORBIDDEN)
• NEVER ask the user to input AK/SK directly in the conversation or command line
• NEVER use aliyun configure set with literal credential values
• ONLY use aliyun configure list to check credential status

bash

aliyun configure list

Check the output for a valid profile (AK, STS, or OAuth identity).

If no valid profile exists, STOP here.

1. Obtain credentials from Alibaba Cloud Console
2. Configure credentials outside of this session (via aliyun configure in terminal or environment variables in shell profile)
3. Return and re-run after aliyun configure list shows a valid profile

---

Observability (MUST follow for every aliyun command)#

Upon loading this skill, generate a random session ID (32-char lowercase hex string) once for the entire session. Use it as {session-id} below.

Rule: Every aliyun CLI command that calls a cloud API MUST include the --user-agent flag. Local utility commands (e.g. configure, plugin, version) do not support this flag and should be excluded.

text

--user-agent AlibabaCloud-Agent-Skills/alibabacloud-kafka-capacity-assessment/{session-id}

Example (assuming session-id is a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6):

bash

aliyun ecs describe-instances --biz-region-id cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-kafka-capacity-assessment/a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6

Do not skip, alter the format, or omit --user-agent on any aliyun API command invocation.

Script / Terraform execution: When running Python SDK scripts or Terraform commands or bash scripts, inject the session-id via inline environment variable so the code can read it at runtime:

bash

# Python SDK script
SKILL_SESSION_ID={session-id} python3 scripts/deploy.py

# Terraform
SKILL_SESSION_ID={session-id} terraform apply

Scripts and Terraform configs should read SKILL_SESSION_ID from the environment (default to empty string if absent). See references/how-to-implement-by-common-sdk.md for SDK patterns.

---

RAM Policy#

This skill requires the following RAM permissions:

APIPermission ActionDescriptionGetInstanceList``alikafka:GetInstanceListRetrieve the Kafka instance list and instance metadataDescribeMetricList``cms:DescribeMetricListQuery CloudMonitor metric data

For the full permission list, see references/ram-policies.md.

[MUST] Permission Failure Handling: When any command or API call fails due to permission errors at any point during execution, follow this process:

1. Read references/ram-policies.md to get the full list of permissions required by this SKILL
2. Use ram-permission-diagnose skill to guide the user through requesting the necessary permissions
3. Pause and wait until the user confirms that the required permissions have been granted

---

Parameter Confirmation#

IMPORTANT: Parameter Confirmation — Before executing any command or API call, ALL user-customizable parameters (e.g., RegionId, instance names, CIDR blocks, passwords, domain names, resource specifications, etc.) MUST be confirmed with the user. Do NOT assume or use default values without explicit user approval.

ParameterRequired/OptionalDescriptionDefaultInstance RegionRequiredRegion ID where the instance resides, e.g. cn-hangzhouNoneInstance IDRequiredKafka instance ID, e.g. alikafka_post-cn-xxxxxNoneIncident Time / Time RangeOptionalMinute-level precision is preferredLast 1 hour from current timeSymptomsOptionalConsumer lag / producer failure / connection anomaly, etc.None (full assessment if omitted)

---

Core Workflow (SOP)#

Hard Rules#

1. This skill performs read-only operations only. If a bottleneck is identified, inform the user which metrics need to be scaled up via the console. Calling any write OpenAPI or executing CLI commands to upgrade the instance on behalf of the user is strictly prohibited.
2. v2 and v3 instances use different CloudMonitor metrics. Always query according to the specifications below. Using v2 MetricNames on a v3 instance, or vice versa, is strictly prohibited.

Step 1: Collect Incident Context#

Gather parameters from the user (see the Parameter Confirmation section above). For any required parameter not yet provided, prompt for it individually.

Key symptoms to watch for:

• Message backlog / increasing consumer lag
• Producer send failures / reduced send throughput
• Reduced consumer consumption rate
• Clients unable to connect to the broker
• Messages deleted before the committed retention period expires
• Instance throughput throttled

Step 2: Identify Instance Series, Edition, and Specification Family#

bash

aliyun alikafka get-instance-list --biz-region-id <RegionId> --region <RegionId> \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-kafka-capacity-assessment/{session-id}

In Aliyun CLI Plugin Mode, the region must be passed via both --biz-region-id and --region.

Extract the following key fields from the response:

• Series: Identifies the instance series ( v2 or v3)
• v2 series: Refer to Kafka v2 Instance Specification Reference Section 1 to determine the edition and specification family
• v3 series: Refer to Kafka v3 Instance Specification Reference Section 1 to determine the edition (v3 has no specification family distinction)

If the user has provided an instance ID, filter by it:

bash

aliyun alikafka get-instance-list --biz-region-id <RegionId> --region <RegionId> --instance-id <InstanceId> \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-kafka-capacity-assessment/{session-id}

Filter by series:

bash

aliyun alikafka get-instance-list --biz-region-id <RegionId> --region <RegionId> --series v2 \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-kafka-capacity-assessment/{session-id}

Step 3: Infer Metrics Likely Approaching Their Limits#

Based on the symptoms described by the user and the capacity bottleneck guidance in the knowledge base, infer the likely bottleneck:

SymptomProbable Bottleneck MetricMessages deleted before retention period expiresDisk space at capacityMessage backlogNetwork throughput throttled, or produce/consume request rate throttledProducer send throughput degradedProduce traffic throttled or produce request rate throttledConsumer consumption rate degradedConsume traffic throttled or consume request rate throttledClients unable to connect to brokerConnection count at limit (necessary but not sufficient — verify with metrics)

Step 4: Query CloudMonitor Metric Data#

4.1 Determine Query Parameters

Based on the Series value confirmed in Step 2, determine the Namespace and MetricName:

• Namespace: Fixed value — acs_kafka
• v2 instance MetricNames: See Kafka v2 Instance Specification and Capacity Policy Section 5
• v3 instance MetricNames: See Kafka v3 Instance Specification, Elastic Strategy, and Capacity Policy Section 6

4.2 Construct the Query Command

bash

aliyun cms describe-metric-list \
  --namespace acs_kafka \
  --metric-name <MetricName> \
  --period 60 \
  --start-time "<start time YYYY-MM-DD HH:mm:ss>" \
  --end-time "<end time YYYY-MM-DD HH:mm:ss>" \
  --dimensions '[{"instanceId":"<InstanceId>"}]' \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-kafka-capacity-assessment/{session-id}

Parameter notes:

• --period: Aggregation interval in seconds; valid values: 15 / 60 / 900 / 3600. Recommended: 60
• --start-time / --end-time: Time range; the interval must not exceed 31 days; the range is left-open, right-closed. If the user does not specify a time range, default to the last 1 hour ( --end-time = current time, --start-time = current time - 1 hour), and note this in the diagnostic report
• --dimensions: JSON string specifying the instance ID

4.3 Query the Applicable Metrics by Instance Series

Metrics for v2 instances (select based on the inferred bottleneck from Step 3):

MetricNameDescriptionBottleneck ThresholdInstanceMessageInputRatioV2Produce traffic as a percentage of instance specification limitApproaching 100% indicates a bottleneckInstanceMessageOutputRatioV2Consume traffic as a percentage of instance specification limitApproaching 100% indicates a bottleneckPartitionInstanceRatioV2Partition count as a percentage of instance specification limitApproaching 100% indicates a bottleneckinstance_disk_capacityInstance disk utilization**> 80%** indicates near-full; Kafka dynamic cleanup beginsInstanceMaxConnectionMaximum connections per node (public + private network)Compare against formula-calculated limitInstanceMaxInternetConnectionMaximum connections per node (public network only)Compare against formula-calculated limitinstance_reqs_inputProduce request rate (requests/sec)Compare against formula-calculated limitinstance_reqs_outputConsume request rate (requests/sec)Compare against formula-calculated limit

Metrics for v3 instances (select based on the inferred bottleneck from Step 3):

MetricNameDescriptionBottleneck ThresholdInstanceMessageInputRatioV3Produce traffic as a percentage of the elastic ceiling> 100% means the elastic ceiling is exceeded and throttling is occurring; (0%, 100%] means reserved capacity is exceeded, incurring elastic overage chargesInstanceMessageOutputRatioV3Consume traffic as a percentage of the elastic ceilingSame as aboveInstanceMaxConnectionRatioV3Connection utilization per node (public + private network)Approaching 100% indicates a bottleneckInstanceMaxInternetConnectionRatioV3Connection utilization per node (public network only)Approaching 100% indicates a bottleneckInstanceThrottleTimeP99InputV3Produce throttle duration**> 0 indicates active throttling**InstanceThrottleTimeP99OutputV3Consume throttle duration**> 0 indicates active throttling**

Step 5: Analyze Findings and Generate Diagnostic Report#

5.1 Analysis

Combine the following inputs for a comprehensive assessment:

1. Instance metadata (Step 2) → obtain the specification limits for each metric
2. CloudMonitor data (Step 4) → obtain the actual usage of each metric during the incident window
3. When investigating a historical incident, prioritize ratio-based metrics (e.g., InstanceMessageInputRatioV2, instance_disk_capacity, InstanceMessageInputRatioV3) or anomaly-based metrics (e.g., InstanceThrottleTimeP99InputV3). Fall back to absolute-value capacity metrics only if the preferred metrics return no data.

v2 instance capacity limit calculation:

See Kafka v2 Instance Specification and Capacity Policy Section 3

v3 instance capacity limit calculation:

See Kafka v3 Instance Specification, Elastic Strategy, and Capacity Policy Sections 2–4

5.2 Generate Diagnostic Report

Output the following structured report to the user:

markdown

## Kafka Instance Capacity Assessment Report

### Basic Information
- Instance ID:
- Region:
- Instance Series: v2 / v3
- Edition:
- Specification:
- Incident Time:

### Incident Summary
- User-Reported Symptoms:
- Anomalous Metrics:

### Monitoring Data Analysis
| Metric Name | Observed Value | Specification Limit | Utilization / Threshold Exceeded | Conclusion |
|:---|:---|:---|:---|:---|
| ... | ... | ... | ... | ... |

### Diagnostic Conclusion
- Root Cause:
- Metrics at or Exceeding Limit:

### Upgrade Recommendations
- Metrics to scale up and recommended target values:
- How to proceed: Navigate to Alibaba Cloud Console > Cloud Message Queue for Apache Kafka > Instance Details > Upgrade

---

Success Verification#

See references/verification-method.md for verification steps.

---

Cleanup#

This skill performs read-only query operations only and does not create or modify any resources. No cleanup is required.

---

Command Tables#

For a complete list of CLI commands used by this skill, see references/related-commands.md.

---

Best Practices#

1. Prefer ratio/percentage metrics for bottleneck detection to avoid inconsistencies caused by historical data recorded under a different specification tier
2. For v3 instances, distinguish between "reserved capacity" and "elastic ceiling": exceeding reserved capacity only incurs elastic overage charges; throttling occurs only when the elastic ceiling is exceeded
3. CloudMonitor data has an approximate 1-minute propagation delay; account for this time offset when querying
4. For v2 instances, when disk utilization exceeds 80%, Kafka begins dynamic log cleanup, which may cause messages to be deleted before the configured retention period
5. Applicable only to standard instances listed in official Alibaba Cloud documentation; not applicable to non-standard instances or self-managed Kafka clusters
6. Calling any write-operation API is prohibited; only read-only queries are permitted
7. v2 and v3 instances use different CloudMonitor MetricNames; mixing them is strictly prohibited

---

Reference Links#

DocumentDescriptionKnowledge Base Notes and Freshness AdvisoryLast-updated timestamp, data staleness risks, and links to official documentationKafka v2 Instance Specification and Capacity Policyv2 instance specification tables, partition limits, connection count / request rate formulas, disk cleanup policy, CloudMonitor metrics, and bottleneck troubleshooting guideKafka v3 Instance Specification, Elastic Strategy, and Capacity Policyv3 instance elastic strategy, elastic ceiling calculation, connection count / request rate formulas, CloudMonitor metrics, and bottleneck troubleshooting guideCLI Installation GuideComplete guide for installing and configuring Aliyun CLIRAM PoliciesFull list of RAM permissions required by this skillRelated CommandsAll CLI commands used by this skillVerification MethodSteps for verifying capacity assessment resultsAcceptance CriteriaAcceptance criteria and test scenarios for this skill