OpenSearch Vector Search Expert#

GitHub: norrishuang/opensearch-vector-search-skill — Issues, PRs, and new reference contributions are welcome!

Safety Notes#

• Pricing script ( scripts/get_opensearch_pricing.py): Makes outbound HTTPS requests to the AWS Pricing API ( pricing.us-east-1.amazonaws.com). Requires boto3 and valid AWS credentials. The script is read-only (fetches public pricing data) and does not modify any AWS resources. Only run it when the user explicitly requests cost estimation.
• Reference examples: Code snippets in references/ contain example API calls to localhost:9200 (standard OpenSearch endpoint). These are documentation examples only — do NOT execute them automatically. Present them to the user as configuration references.
• Cluster analyzer ( scripts/analyze_cluster.py): Connects to a user-provided OpenSearch cluster and performs read-only analysis. It NEVER creates, modifies, or deletes any indices or data. Only run it when the user explicitly provides cluster credentials (URL + username/password).
• Vector benchmark skill ( opensearch-vector-benchmark/SKILL.md): VectorDBBench creates, loads, force-merges, and may drop benchmark indices. Use only with a dedicated benchmark target and explicit user confirmation.

Knowledge Base Structure#

Read the corresponding reference file based on the question type:

Question TypeReference FileKeywordsVector search, k-NN, HNSW, disk modereferences/vector-search.mdvector, knn, hnsw, warmup, disk mode, on_disk1-bit SQ, 32x on-disk tuning and costreferences/quantization-1bit-32x.md1-bit, scalar quantization, SQ, 32x, on_disk, oversample, costOther quantization techniquesreferences/quantization-techniques.mdquantization, binary, byte, fp16, product quantizationCost optimization, instance sizing, memory calcreferences/cost-optimization.mdcost, pricing, instance, memory calculation, cluster sizing, budgetCluster tuning, JVM, thread poolsreferences/cluster-tuning.mdJVM, heap, thread pool, node role, shard allocationPerformance benchmarks, dataset sizingreferences/performance-benchmarks.mdbenchmark, QPS, latency, recall, dataset sizeIndexing strategies, mappingreferences/indexing-strategies.mdindex, mapping, shard, replica, lifecycleQuery optimizationreferences/query-optimization.mdquery, filter, aggregation, cache, paginationOptimized instances (OR1/OR2/OM2/OI2)references/optimized-instances.mdoptimized, OR1, OR2, OM2, OI2, S3 durability, indexing throughputLive cluster analysisscripts/analyze_cluster.pyanalyze cluster, connect, diagnose, review config, health checkExecute a vector benchmarkopensearch-vector-benchmark/SKILL.mdVectorDBBench, benchmark, QPS, latency, recall, indexing throughput

Core Workflows#

1. Answering Vector Search Configuration Questions#

1. Read references/vector-search.md
2. Recommend in-memory mode or disk mode based on user scenario (latency requirements, data scale, QPS)
3. Provide specific mapping JSON configuration
4. Recommend FAISS engine + cosine similarity + 7/8 series instances

2. Capacity Planning & Instance Sizing (Most Common Scenario)#

After user provides vector count and dimensions:

1. Read references/cost-optimization.md for memory calculation formulas and examples

2. Calculate using the standard HNSW memory formula (source: AWS official blog): text

   Unquantized (float32):
     Memory = 1.1 × (4 × d + 8 × m) × num_vectors × (replicas + 1) bytes
   
   Quantized (FAISS engine, compressed vectors in memory):
     FP16 (2x):    Memory = 1.1 × (2 × d + 8 × m) × num_vectors × (replicas + 1)
     Byte (4x):    Memory = 1.1 × (1 × d + 8 × m) × num_vectors × (replicas + 1)
     Binary 4-bit: Memory = 1.1 × (d/2 + 8 × m) × num_vectors × (replicas + 1)
     Binary 2-bit: Memory = 1.1 × (d/4 + 8 × m) × num_vectors × (replicas + 1)
     Scalar 1-bit (32x): Memory = 1.1 × (d/8 + 8 × m) × num_vectors × (replicas + 1)
   
   Where: d=vector dimensions, m=HNSW connections (default 16), num_vectors=total vector count
   
   

3. Apply OpenSearch node memory allocation rules: text

   JVM Heap = min(node_memory × 50%, 32GB)
   Remaining memory = node_memory - JVM Heap
   KNN available memory = remaining × 75%  (with knn.memory.circuit_breaker.limit=70%, ~35% of node memory)
   
   

4. Select instance type, ensuring total cluster KNN available memory > vector index memory requirement

5. Run pricing script for real-time pricing (see below)

3. Cost Estimation (with Real-Time Pricing)#

When user needs cost estimation:

1. Complete capacity planning above

2. Run pricing script for real-time prices: bash

   python3 scripts/get_opensearch_pricing.py --region <region> --instance-type <type>
   
   

3. Calculate monthly cost: text

   Instance cost = unit_price × node_count × (1 + replica_count)
   EBS cost = capacity(GB) × $0.08 + additional IOPS charges
   Total cost = Instance cost + EBS cost
   
   

4. Compare cost differences across quantization options

For 1-bit or 32x on-disk questions, read references/quantization-1bit-32x.md before estimating capacity or cost. Treat its 100M benchmark as a versioned reference point, not a portable guarantee. Refresh current instance and EBS prices before quoting a present-day total.

4. Live Cluster Analysis (When User Provides Cluster Credentials)#

When the user provides an OpenSearch cluster URL and credentials, use the cluster analyzer to connect and review their vector search configuration. This is read-only — never modify the cluster.

Prerequisites: User must explicitly provide:

• Cluster URL (e.g., https://my-cluster.us-east-1.es.amazonaws.com)
• Username and password (basic auth), OR --no-auth for clusters without authentication

Workflow:

1. Ask for credentials if not provided: URL, username, password

2. Run cluster overview to get health, nodes, and k-NN index list: bash

   python3 scripts/analyze_cluster.py --url <url> -u <user> -p <pass> --action cluster-overview -f pretty
   
   

3. Analyze specific index if user specifies one, or pick the most important k-NN index: bash

   python3 scripts/analyze_cluster.py --url <url> -u <user> -p <pass> --action index-detail --index <index_name> -f pretty
   
   

4. Analyze shard distribution for the target index: bash

   python3 scripts/analyze_cluster.py --url <url> -u <user> -p <pass> --action shard-analysis --index <index_name> -f pretty
   
   

5. Run all analyses at once (for a comprehensive report): bash

   python3 scripts/analyze_cluster.py --url <url> -u <user> -p <pass> --action all --index <index_name> -f pretty
   
   

6. Interpret the JSON output and present findings to the user:

   • Cluster health status and node resource utilization
   • Vector field configurations (engine, dimensions, HNSW params, quantization)
   • Memory estimates vs actual cluster capacity
   • Auto-generated recommendations (from the script)

7. Provide actionable advice based on findings:

   • Suggest better engine/quantization if needed (provide example mapping JSON)
   • Suggest instance resizing if memory is over/under-provisioned
   • Suggest shard rebalancing if distribution is uneven
   • NEVER execute write operations — only provide example configurations for the user to apply

Cluster Analyzer Script Reference:

text

Usage:
  python3 scripts/analyze_cluster.py --url <url> -u <user> -p <pass> [options]

Actions:
  --action cluster-overview   Cluster health, nodes, k-NN stats, and all k-NN index summary (default)
  --action index-detail       Deep dive into a specific index's vector config + memory estimates
  --action shard-analysis     Shard distribution and sizing for a specific index
  --action all                Run all analyses

Options:
  --index <name>     Target a specific index (required for index-detail and shard-analysis)
  --no-auth          Connect without authentication
  --verify-ssl       Verify SSL certificates (default: skip)
  --format pretty    Human-readable JSON output

Output: JSON with these top-level keys:
  - cluster_overview: health, version, nodes (memory/CPU/JVM), knn_stats
  - knn_indices: list of all k-NN enabled indices with vector field summaries
  - index_detail/index_details: vector field configs, memory estimates, search stats
  - shard_analysis/shard_analyses: shard distribution across nodes
  - recommendations: auto-generated optimization suggestions with severity levels

Safety constraints for live cluster analysis:

• The script is strictly read-only (uses only GET/CAT APIs)
• NEVER create, update, or delete indices on the user's cluster
• NEVER change cluster settings or mappings
• Only provide example JSON configurations for the user to review and apply themselves
• If the user asks to apply changes, provide the exact API calls/JSON but let the user execute them

Pricing Script Usage#

bash

# Query all instance prices for a region
python3 scripts/get_opensearch_pricing.py --region us-east-1

# Query specific instance type (no .search suffix needed)
python3 scripts/get_opensearch_pricing.py --region us-east-1 --instance-type r7g.xlarge

# JSON format output (for calculations)
python3 scripts/get_opensearch_pricing.py --region us-east-1 --instance-type r7g.xlarge --format json

Output fields: instance_type, vcpu, memory_gib, price_per_hour_usd, price_per_month_usd, network

Recommended Defaults#

Always recommend these defaults unless user has specific requirements:

• Engine: FAISS

• Similarity: cosine

• Instance family (Gen 7+ only, never recommend older generations):

  • Vector search (k-NN): r7g/r8g/r8gd (memory-optimized, lowest search latency; r8g Graviton4 ~30% faster than r7g)
  • Indexing-heavy + vector: OR2 (optimized, S3 durability, good memory-to-price ratio)
  • Indexing-heavy (no vector): OM2 (highest indexing throughput, 15% faster than OR1)
  • Large dataset with NVMe: OI2 (storage-optimized, no EBS needed)
  • Do NOT recommend: r6g, r5, m5, c5, i3, or any older instance families

• HNSW parameters: ef_construction=512, m=16

• Quantization preference: Byte (4x) for conservative production use; 1-bit SQ (32x) for aggressive cost optimization on OpenSearch 3.6+, preferably 3.7+

• Disk mode threshold: Consider when data > 50M vectors and 100-200ms latency is acceptable

Instance Selection Decision Tree#

text

Is this primarily a vector search (k-NN) workload?
├─ YES → r7g/r8g/r8gd (best search latency, standard EBS; prefer r8g for Graviton4)
│        └─ Need S3 durability? → OR2 (accept 10s refresh interval tradeoff)
├─ Mixed (logs + vectors) → OR2 for log nodes, r7g/r8g for vector nodes
└─ NO (logs/observability/analytics)
   ├─ Write-heavy → OM2 (highest ingest throughput)
   ├─ Balanced → OR2 (good all-around with S3 durability)
   └─ Need NVMe IOPS → OI2

Response Template#

Organize cost/sizing answers in this structure:

1. Requirements confirmation: Vector count, dimensions, QPS, latency requirements
2. Memory calculation: Raw size → quantized size → required KNN memory
3. Cluster configuration: Instance type × count, shards, replicas
4. Cost estimation: Instance cost + EBS cost = monthly total
5. Optimization suggestions: Quantization comparison, Reserved Instance discounts