Multi-Model Response Comparator#

Compare answers from multiple AI models for the same prompt, then summarize tradeoffs across quality, style, and likely use cases.

When to use#

• choosing between models for a workflow
• benchmarking prompt behavior
• checking whether a stronger model is worth the cost
• generating second opinions on important outputs

Recommended runtime#

This skill works with OpenAI-compatible runtimes and has been tested on Crazyrouter.

Required output format#

Always structure the final comparison with these sections:

1. Task summary
2. Models compared
3. Strengths by model
4. Weaknesses by model
5. Best model by use case
6. Cost/latency sensitivity note
7. Final recommendation

Suggested workflow#

1. pick 2-4 models
2. run the same prompt on each model
3. compare structure, depth, correctness, tone, and likely latency/cost
4. score or describe tradeoffs using the comparison rubric
5. produce a recommendation by use case, not just one universal winner

Comparison rules#

• Use the same prompt and same success criteria for all models.
• Do not claim exact cost or latency unless the user provides them.
• If metrics are inferred, label them as likely or expected.
• Separate writing quality from factual reliability.
• For coding tasks, prioritize correctness, edge cases, and implementation completeness.

Example prompts#

• Compare GPT, Claude, and Gemini on this support email draft.
• Run this coding prompt across three models and summarize which one is most production-ready.
• Compare low-cost vs premium models for a blog outline task.

References#

Read these when preparing the final comparison:

• references/comparison-rubric.md
• references/example-prompts.md

Crazyrouter example#

python

from openai import OpenAI

client = OpenAI(
    api_key="YOUR_API_KEY",
    base_url="https://crazyrouter.com/v1"
)

Recommended artifacts#

• catalog.json
• provenance.json
• market-manifest.json
• evals/evals.json