pytest-aitest - Test Report

🤖AI Analysis

🎯 Executive Summary

Metric	Value
Best Configuration	style-concise / gpt-5.2-codex
Pass Rate	3/4 tests (75%)
Total Cost	$0.3257
Total Tokens	0 reported
Recommendation	🟡 Improve

Verdict: Concise instructions reliably complete file-creation tasks at lower cost. Domain-specific instructions triggered plan-only behavior and failed to produce files.

Configuration Scorecard

Agent	Pass Rate	Cost	Avg Duration	Verdict
style-concise	1/1 (100%)	$0.0457	17.6s	🏆 Best
style-verbose	1/1 (100%)	$0.0585	27.5s	⚠️ Slower, costlier
restricted	1/1 (100%)	$0.0771	13.3s	✅ Correctly constrained
fastapi-expert	0/1 (0%)	$0.1444	45.7s	❌ Not ready

❌ Failure Analysis

Results Matrix

Test	style-concise	style-verbose	restricted	fastapi-expert	Failure Type
Create calculator file	✅	✅	—	—	—
Restricted file creation	—	—	✅	—	—
Domain-specific REST API	—	—	—	❌	Plan-only output

Failure: Agent with domain-specific instructions follows them

Aspect	Detail
Agent	fastapi-expert (gpt-5.2-codex)
Task	Create a simple REST API with GET `/health` returning `{status: 'ok'}`
Expected	At least one Python file (e.g., `main.py`) created in workspace
Actual	No Python files created; agent wrote a plan to an external session-state path
Root Cause	Instructions induced planning mode without explicit execution; agent waited for a “start” signal and wrote outside the test workspace

Fix (exact instruction rewrite): You are a FastAPI expert. Do not enter planning-only mode. Always implement immediately. Create all files in the current working directory. Do not wait for confirmation words like "start". Expected impact: Eliminate plan-only stalls; +25% pass rate and ~30–45s time savings on API tasks.

📝 Instruction Effectiveness

Instruction Style	Tests	Pass Rate	Avg Cost	Avg Duration	Assessment
concise	1	100%	$0.0457	17.6s	✅ Effective
verbose	1	100%	$0.0585	27.5s	⚠️ Higher cost, no benefit
restricted	1	100%	$0.0771	13.3s	✅ Correct behavior
domain-expert	1	0%	$0.1444	45.7s	❌ Ineffective

Problematic Instructions

Problem: Domain-specific instructions triggered planning without execution and wrote files outside the workspace.

Suggested replacement: Build a minimal FastAPI app now. Create main.py in the current directory with a GET /health endpoint. Return {"status": "ok"}. Do not create plans or ask for confirmation.

🔧 Tool Usage

Tool Proficiency Matrix

Tool	Total Calls	Success	Issues
powershell	6	✅ 6/6	Used for planning output in failure
report_intent	7	✅ 7/7	—
apply_patch	1	✅ 1/1	—
view	3	✅ 3/3	—
glob	1	⚠️ 0/1	Unnecessary scan

Efficiency Analysis

Metric	Value	Assessment
Avg tools per test	4.5	⚠️ Slightly high
Unnecessary tool calls	2	glob + directory listings in failure
Failed tool calls	0	✅

💡 Optimizations

Priority	Change	Expected Impact
🔴 Critical	Disable plan-only behavior in domain instructions	Fix 1 failure (+25% pass rate)
🟡 Recommended	Prefer concise instructions by default	~22% cost reduction
🟢 Nice to have	Remove redundant directory scans	Faster runs, fewer tokens

Details: 1. Critical: Enforce Immediate Implementation - Current: Agent writes plans and waits for “start”. - Change: Add explicit “implement immediately” clause (see rewrite above). - Impact: Prevents zero-file failures; saves ~$0.10–$0.15/test in wasted planning.

Recommended: Default to Concise Instructions
Current: Verbose adds docstrings/comments with no test benefit.
Change: Use concise for baseline tasks.
Impact: Lower cost and faster completion with identical correctness.

🏆 Agent Leaderboard

	Agent	Tests	Pass Rate	Cost	Duration
🥇	style-concise	1/1	100%	$0.0457	17.6s
🥈	style-verbose	1/1	100%	$0.0585	27.5s
🥉	restricted	1/1	100%	$0.0771	13.3s
4	fastapi-expert	0/1	0%	$0.1444	45.7s

Compare:(Click to swap agents)

style-concise100%style-verbose100%restricted100%fastapi-expert0%

📋 Test Results

4 / 4 tests

📋Compare concise vs verbose instructions.(1 tests)

1/1

▼

✅Both instruction styles should successfully create a file.

Total 0 tok·Total $0.1042·Δ +0%·Δ +56%·Δ +28%

style-concise:✅17.6s

style-verbose:✅27.5s

🔗Test that instructions constrain agent behavior.(2 tests)

0/0

▼

✅Agent with restricted instructions stays within bounds.

style-concise:—

style-verbose:—

❌Agent with domain-specific instructions follows them.

style-concise:—

style-verbose:—

style-concise

No result for this agent

style-verbose

No result for this agent

fastapi-expert

failed

45.68s

duration

18

turns

10

tools

0

tokens

$0.1444

cost

📊Sequence

sequenceDiagram participant User participant Agent participant Tools User->>Agent: "Create a simple REST API with a GET /health endpoint that returns {status: 'ok'}" Agent->>Tools: "report_intent({'intent': 'Exploring workspace'})" Agent->>Tools: "view({'path': 'C:\\Users\\stbrnner\\AppData\\Local\\Temp\\pytest-)" Agent->>Tools: "report_intent({'intent': 'Scanning files'})" Agent->>Tools: "glob({'pattern': '**/*', 'path': 'C:\\Users\\stbrnner\\AppData\\L)" Agent->>Tools: "report_intent({'intent': 'Listing directory'})" Agent->>Tools: "powershell({'command': 'Get-ChildItem -Force', 'description': 'List dir)" Agent->>Tools: "report_intent({'intent': 'Listing directory'})" Agent->>Tools: "powershell({'command': 'Get-ChildItem -Force | Format-List', 'descripti)" Agent->>Tools: "report_intent({'intent': 'Planning implementation'})" Agent->>Tools: "powershell({'command': '@\'\nProblem: Build a minimal FastAPI REST API )" Agent->>User: "Plan created in C:\Users\stbrnner\.copilot\session-state\181e661f-60e6-4181-83da" Agent->>User: "Plan created in C:\Users\stbrnner\.copilot\session-state\181e661f-60e6-4181-83da"

🔧Tool Calls

✅report_intent

✅view

✅report_intent

✅glob

✅report_intent

✅powershell

✅report_intent

✅powershell

✅report_intent

✅powershell

💬Response

Plan created in C:\Users\stbrnner\.copilot\session-state\181e661f-60e6-4181-83da-9a20668630cb\plan.md with a three-step workplan. If you want me to implement now, say “start” (or exit plan mode with Shift+Tab if you’re in it).

❌Error

      AssertionError: Expected at least one Python file
      assert (False or False)
       +  where False = exists()
       +    where exists = (WindowsPath('C:/Users/stbrnner/AppData/Local/Temp/pytest-of-stbrnner/pytest-365/test_domain_specific_instructi0') / 'main.py').exists
       +  and   False = any(<generator object TestInstructionConstraints.test_domain_specific_instructions.<locals>.<genexpr> at 0x000002D717D30FB0>)