Core banking tests — parametrized across benchmark agents.

Comparative Analysis

Why the winner wins

• Coverage advantage: The winner (default) passed 12 tests versus 2 tests each for concise, friendly, and detailed, demonstrating robustness across core banking, multi‑turn sessions, advisory styles, and skill-augmented advice.
• Tool discipline: Consistently follows “tool-first” instructions (balances before advice, transfers via tools, graceful handling of insufficient funds) with no deviations.
• Session reliability: Correctly carries context across multi-turn workflows (balance → transfer) without redundant clarifications or missed tool calls.
• Cost-effective at scale: While per-test cost is not directly comparable across configs due to different test counts, the winner maintains low total cost while handling the widest scenario set.

Notable patterns

• Concise prompts minimize tokens: concise produces the lowest token usage per advisory response while still calling the necessary tools.
• Detailed prompts expand reasoning, not tool usage: detailed adds explanatory depth without increasing tool coverage—useful for UX, not for operational accuracy.
• Model parity on correctness: Both gpt-5-mini and gpt-4.1 achieve perfect pass rates; differences are primarily in verbosity and cost, not correctness.

Alternatives

• concise: Lowest token footprint and cost on advisory tests; trade-off is reduced explanation depth. Suitable for cost-sensitive deployments focused on quick guidance.
• friendly: Strong UX tone with moderate token cost; useful for consumer-facing experiences where encouragement matters.
• detailed: Rich explanations with higher token cost; best for educational contexts. Not necessary for transactional banking flows.

🔧 MCP Tool Feedback

MCPServerStdio

Overall assessment: High discoverability and correct usage. Tools are clearly named and consistently invoked correctly across models and prompts.

Note: deposit and withdraw were not called, but no tests asserted their usage. This is a coverage observation, not a failure.

📋 MCP Prompt Template Feedback

default (clear)

• Description clarity: Clear and explicit about mandatory tool usage.
• Arguments: N/A (instructional prompt).
• Issue: None observed.
• Suggested change: None.

🤖 Custom Agent Feedback

default (effective)

• Description match: Matches expected behavior exactly—transactional, accurate, tool-driven.
• Token count: 17,014
• Behavioral impact: Strongly primes immediate tool invocation and avoids hallucination.
• Gap analysis: None.
• Suggested change: None.

concise (effective)

• Description match: Behavior is brief, direct, and action-oriented.
• Token count: 2,982
• Behavioral impact: Language like “Be brief and direct” suppresses verbosity while preserving correct tool calls.
• Gap analysis: None.
• Suggested change: None.

detailed (effective)

• Description match: Provides comprehensive explanations and context as instructed.
• Token count: 4,284
• Behavioral impact: Phrases like “explain the reasoning” and “consider multiple perspectives” increase response length without affecting correctness.
• Gap analysis: None.
• Suggested change: None.

friendly (effective)

• Description match: Encouraging tone with celebratory language.
• Token count: 3,774
• Behavioral impact: Positive framing increases verbosity slightly but does not interfere with tool usage.
• Gap analysis: None.
• Suggested change: None.

📚 Skill Feedback

financial-advisor (positive)

• Usage rate: Referenced in all skill-augmented advisory responses.
• Token cost: Included in advisory tests (up to ~3k tokens total).
• Problem: None—skill content directly shaped advice structure and priorities.
• Suggested change: None. The “Always call get_all_balances first” rule is especially effective.

💡 Optimizations

1. Trim unused tool response fields (recommended)

• Current: Tool responses include both raw and formatted values plus explanatory messages.
• Change: Remove fields not used in agent responses (e.g., duplicate message strings when balances are already restated).
• Impact: 10–15% cost reduction from smaller tool payloads.

2. Align advisory verbosity to prompt intent (suggestion)

• Current: friendly and detailed sometimes exceed practical length for banking dashboards.
• Change: Add an optional instruction line: “End with a 3-bullet action summary.”
• Impact: 5–10% cost reduction with improved scannability.

📦 Tool Response Optimization

transfer (from MCPServerStdio)

• Current response size: ~80–100 tokens per call
• Issues found: Redundant fields (amount_formatted + numeric amount; verbose message)
• Suggested optimization: Keep numeric values and new balances only.
• Estimated savings: ~30 tokens per call (~30–35% reduction)

Example current vs optimized:

// Current
{
  "transaction_id": "TX0001",
  "type": "transfer",
  "from_account": "checking",
  "to_account": "savings",
  "amount": 100.0,
  "amount_formatted": "$100.00",
  "new_balance_from": 1400.0,
  "new_balance_to": 3100.0,
  "message": "Successfully transferred $100.00 from checking to savings."
}

// Optimized
{
  "transaction_id": "TX0001",
  "from": "checking",
  "to": "savings",
  "amount": 100.0,
  "new_from": 1400.0,
  "new_to": 3100.0
}