CLI Options

Recommended: pyproject.toml

Configure once, run simply:

[tool.pytest.ini_options]
addopts = """
--aitest-summary-model=azure/gpt-5.2-chat
--aitest-html=aitest-reports/report.html
"""

Then just pytest tests/ — reports are generated automatically.

pytest Options

Option	Description	Required
--aitest-summary-model=MODEL	Model for AI insights	Yes (for HTML/MD reports)
--aitest-html=PATH	Generate HTML report	No
--aitest-md=PATH	Generate Markdown report	No
--aitest-json=PATH	Custom JSON path	No (default: aitest-reports/results.json)
--aitest-min-pass-rate=N	Fail if overall pass rate below N%	No
--aitest-iterations=N	Run each test N times and aggregate results	No (default: 1)
--aitest-analysis-prompt=PATH	Custom analysis prompt file for AI insights	No
--aitest-summary-compact	Omit full conversation turns for passed tests in AI analysis (reduces tokens)	No
--aitest-print-analysis-prompt	Print resolved analysis prompt source/path at runtime	No
--llm-model=MODEL	Model for llm_assert semantic assertions (default: openai/gpt-5-mini)	No
--llm-vision-model=MODEL	Vision model for llm_assert_image assertions (defaults to --llm-model)	No

Note

JSON is always generated after every test run, even without --aitest-summary-model. HTML and Markdown reports require a summary model for AI-powered analysis. JSON output contains raw test data that can be used later to regenerate reports via pytest-aitest-report.

CLI Examples

# Run tests with HTML report
pytest tests/ \
    --aitest-summary-model=azure/gpt-5.2-chat \
    --aitest-html=report.html

# Run tests with Markdown report
pytest tests/ \
    --aitest-summary-model=azure/gpt-5.2-chat \
    --aitest-md=report.md

# With JSON output
pytest tests/ \
    --aitest-summary-model=azure/gpt-5.2-chat \
    --aitest-html=report.html \
    --aitest-json=results.json

# Run each test 3 times for statistical confidence
pytest tests/ \
    --aitest-summary-model=azure/gpt-5.2-chat \
    --aitest-html=report.html \
    --aitest-iterations=3

# Reduce summary token usage for large suites
pytest tests/ \
    --aitest-summary-model=azure/gpt-5.2-chat \
    --aitest-html=report.html \
    --aitest-summary-compact

# Debug which analysis prompt is used (CLI file / hook / built-in)
pytest tests/ \
    --aitest-summary-model=azure/gpt-5.2-chat \
    --aitest-html=report.html \
    --aitest-print-analysis-prompt

pytest-aitest-report CLI

Regenerate reports from saved JSON without re-running tests.

pytest-aitest-report <json-file> [options]

Option	Description	Required
--html PATH	Generate HTML report	At least one of --html or --md
--md PATH	Generate Markdown report	At least one of --html or --md
--summary	Generate AI-powered summary	No
--summary-model MODEL	Model for AI insights	Required with --summary
--analysis-prompt PATH	Custom analysis prompt file for AI insights	No
--compact	Omit full conversation turns for passed tests (reduces tokens)	No
--print-analysis-prompt	Print resolved analysis prompt source/path before summary generation	No

--summary-model can also be set via AITEST_SUMMARY_MODEL env var or [tool.pytest-aitest-report] in pyproject.toml.

Examples

# Regenerate HTML from existing JSON (uses insights already in JSON)
pytest-aitest-report results.json --html report.html

# Generate Markdown report
pytest-aitest-report results.json --md report.md

# Generate both formats
pytest-aitest-report results.json --html report.html --md report.md

# Generate with fresh AI analysis
pytest-aitest-report results.json \
    --html report.html \
    --summary \
    --summary-model azure/gpt-5.2-chat

# Debug which prompt source is used (file vs built-in)
pytest-aitest-report results.json \
    --html report.html \
    --summary \
    --summary-model azure/gpt-5.2-chat \
    --print-analysis-prompt

Environment Variables

LLM authentication is handled by Pydantic AI:

Provider	Variable
Azure OpenAI	AZURE_API_BASE + az login
OpenAI	OPENAI_API_KEY
Anthropic	ANTHROPIC_API_KEY
Google	GEMINI_API_KEY

Azure OpenAI Setup

# Set endpoint
export AZURE_API_BASE=https://your-resource.openai.azure.com/

# Authenticate (no API key needed!)
az login

OpenAI Setup

export OPENAI_API_KEY=sk-xxx