API reference¶

Signatures and type documentation below are generated from the current source. Most tests start with CopilotEval and the copilot_eval pytest fixture. Use the configuration guide for settings and the result reference for interpreting returned evidence.

Session configuration and result¶

CopilotEval(name: str = 'copilot', model: str | None = None, reasoning_effort: CopilotReasoningEffort | None = None, client_mode: Literal['copilot-cli', 'empty'] = 'copilot-cli', image_detail: Literal['auto', 'low', 'high'] | None = None, audit_requests: bool = False, instructions: str | None = None, system_message_mode: Literal['append', 'replace'] = 'append', working_directory: str | None = None, allowed_tools: list[str] | None = None, excluded_tools: list[str] | None = None, max_turns: int = 25, timeout_s: float = 300.0, max_tool_calls: int | None = None, auto_confirm: bool = True, mcp_servers: dict[str, CopilotMCPServerConfig] = dict(), custom_agents: list[CopilotCustomAgentConfig] = list(), skill_directories: list[str] = list(), disabled_skills: list[str] = list(), extra_config: dict[str, Any] = dict(), active_agent: str | None = None, hooks: CopilotSessionHooks = _empty_session_hooks(), persona: CopilotPersona = _default_persona()) `dataclass` ¶

Configuration for a GitHub Copilot agent test.

Maps to the Copilot SDK's SessionConfig. User-facing field names are kept intuitive (e.g. instructions), while build_session_config() maps them to the SDK's actual system_message TypedDict.

The SDK's SessionConfig has no maxTurns field. The runner enforces a hard wall-clock limit via timeout_s; max_turns is advisory for the top-level session (it is not hard-enforced mid-run) and is used when configuring subagent turn caps.

Example

Minimal¶

CopilotEval()

With instructions and model¶

CopilotEval( name="security-reviewer", model="claude-sonnet-4", instructions="Review code for security vulnerabilities.", )

With custom tools and references¶

CopilotEval( name="file-creator", instructions="Create files as requested.", working_directory="/tmp/workspace", allowed_tools=["create_file", "read_file"], )

`build_session_config() -> dict[str, Any]` ¶

Build a SessionConfig dict for the Copilot SDK.

Returns a dict compatible with CopilotClient.create_session(). Only includes non-None/non-default fields to avoid overriding SDK defaults.

SDK field mapping (Python snake_case TypedDict keys): instructions → system_message: {mode, content} allowed_tools → available_tools excluded_tools → excluded_tools reasoning_effort → reasoning_effort working_directory → working_directory mcp_servers → mcp_servers custom_agents → custom_agents skill_directories → skill_directories disabled_skills → disabled_skills

Note: max_turns is NOT part of SessionConfig. The runner enforces timeout_s as the hard limit; max_turns is advisory (not hard-enforced mid-run) and used only to cap subagent turns.

`create_subagent(*, name: str, model: str | None, reasoning_effort: CopilotReasoningEffort | None, instructions: str, timeout_s: float, max_turns: int, allowed_tools: list[str] | None, mcp_servers: dict[str, CopilotMCPServerConfig]) -> CopilotEval` ¶

Create the exact nested eval configuration for a custom agent.

`from_copilot_config(path: str | Path = '.', **overrides: Any) -> 'CopilotEval'` `classmethod` ¶

Load a CopilotEval from a directory containing Copilot config files.

Looks for the following files under path:

.github/copilot-instructions.md → instructions
.github/agents/*.agent.md → custom_agents

Point path at any directory — your production project, a dedicated test fixture project, or a shared team config repo. Any keyword argument overrides the loaded value.

Parameters:

Name	Type	Description	Default
`path`	`str \| Path`	Root directory to load config from. Defaults to the current working directory.	`'.'`
`**overrides`	`Any`	Override any `CopilotEval` field after loading, e.g. `model="claude-opus-4.5"`.	`{}`

Returns:

Type	Description
`'CopilotEval'`	A `CopilotEval` initialised from the discovered config files.

Example::

# Load from the current project (production config as baseline)
baseline = CopilotEval.from_copilot_config()

# A/B test: same config, one instruction changed
treatment = CopilotEval.from_copilot_config(
    instructions="Always add type hints.",
)

# Load from a dedicated test-fixture project
agent = CopilotEval.from_copilot_config("tests/fixtures/strict-agent")

# Load from a shared team agent library
agent = CopilotEval.from_copilot_config("/shared/team/copilot-config")

`from_plugin(path: str | Path, *, model: str = '', persona: CopilotPersona | None = None, instructions: str = '', working_directory: str = '', name: str = '', **overrides: Any) -> 'CopilotEval'` `classmethod` ¶

Create a CopilotEval from a plugin directory.

Loads the plugin's agents, skills, MCP servers, and instructions, then constructs a CopilotEval with all components wired together.

Uses :func:~pytest_skill_engineering.core.plugin.load_plugin to discover the plugin structure. The persona is auto-detected from the plugin path (ClaudeCodePersona for .claude/ paths, VSCodePersona otherwise) unless explicitly overridden.

Parameters:

Name	Type	Description	Default
`path`	`str \| Path`	Path to the plugin directory (may contain `plugin.json`, or be a `.github/` / `.claude/` project config directory).	required
`model`	`str`	Model override for the eval.	`''`
`persona`	`CopilotPersona \| None`	IDE persona override. Auto-detected when `None`.	`None`
`instructions`	`str`	Additional instructions to append to the plugin's discovered instructions.	`''`
`working_directory`	`str`	Override the working directory.	`''`
`name`	`str`	Override the eval name (defaults to plugin metadata name).	`''`
`**overrides`	`Any`	Override any additional `CopilotEval` fields such as `max_turns` or `timeout_s`.	`{}`

Returns:

Type	Description
`'CopilotEval'`	A `CopilotEval` initialised from the plugin.

Example::

agent = CopilotEval.from_plugin("my-plugin/")

agent = CopilotEval.from_plugin(
    ".claude/",
    model="claude-sonnet-4",
    instructions="Focus on security reviews.",
)

`from_claude_config(path: str | Path = '.', *, model: str = '', persona: CopilotPersona | None = None, instructions: str = '', working_directory: str = '', name: str = 'claude-code-eval', **overrides: Any) -> 'CopilotEval'` `classmethod` ¶

Create a CopilotEval from a Claude Code project directory.

Scans for:

CLAUDE.md (project root) and .claude/CLAUDE.md → instructions
.claude/agents/*.md → custom agents
.claude/skills/ → skill directories (subdirs with SKILL.md)
.mcp.json → MCP server configs

Parameters:

Name	Type	Description	Default
`path`	`str \| Path`	Root of the Claude Code project (default: current directory).	`'.'`
`model`	`str`	Model override for the eval.	`''`
`persona`	`CopilotPersona \| None`	IDE persona override. Defaults to `ClaudeCodePersona`.	`None`
`instructions`	`str`	Additional instructions to append to the discovered `CLAUDE.md` content.	`''`
`working_directory`	`str`	Override the working directory.	`''`
`name`	`str`	Override the eval name.	`'claude-code-eval'`
`**overrides`	`Any`	Override any additional `CopilotEval` fields such as `max_turns` or `timeout_s`.	`{}`

Returns:

Type	Description
`'CopilotEval'`	A `CopilotEval` initialised from the discovered config files.

Example::

# Load from the current project
agent = CopilotEval.from_claude_config()

# Load from a specific directory with model override
agent = CopilotEval.from_claude_config(
    "tests/fixtures/claude-project",
    model="claude-sonnet-4",
)

CopilotResult(turns: list[Turn] = list(), success: bool = True, error: str | None = None, duration_ms: float = 0.0, usage: list[UsageInfo] = list(), reasoning_traces: list[str] = list(), subagent_invocations: list[SubagentInvocation] = list(), permission_requested: bool = False, permissions: list[dict[str, Any]] = list(), model_used: str | None = None, total_premium_requests: float = 0.0, raw_events: list[Any] = list(), capture_errors: list[str] = list(), request_audit: list[RequestAudit] = list(), stop_reason: StopReason | None = None, tool_calls_admitted: int = 0, skill_discovery: SkillDiscovery | None = None, configuration: dict[str, Any] | None = None, agent: CopilotResultAgent | None = None) `dataclass` ¶

Result of running a prompt against GitHub Copilot.

Captures the full event stream from the SDK, including tool calls, reasoning traces, subagent routing, permissions, and token usage.

Example

result = await copilot_eval(agent, "Create hello.py") assert result.success assert result.tool_was_called("create_file") assert "hello" in result.final_response.lower()

`evidence_complete: bool` `property` ¶

Whether every observed call has a complete, correlated record.

`execution_results: list[CopilotResult]` `property` ¶

This session and framework-owned descendants, each included once.

`all_usage: list[UsageInfo]` `property` ¶

Captured usage across this session and its descendants.

`final_response: str | None` `property` ¶

Get the last assistant response.

`all_responses: list[str]` `property` ¶

Get all assistant responses.

`all_tool_calls: list[ToolCall]` `property` ¶

Get all tool calls across all turns.

`tool_names_called: set[str]` `property` ¶

Get set of all tool names that were called.

`total_input_tokens: int | None` `property` ¶

Total input tokens across all model turns.

`total_output_tokens: int | None` `property` ¶

Total output tokens across all model turns.

`total_tokens: int | None` `property` ¶

Total tokens (input + output) across all model turns.

`token_usage: dict[str, int]` `property` ¶

Token usage dict compatible with pytest-skill-engineering's EvalResult.

Keys use short names (prompt, completion, total) to match the format pytest-skill-engineering reads in its collector and generator.

`working_directory: Path` `property` ¶

Working directory where the agent operated.

Resolved from agent.working_directory when set; falls back to the current working directory.

`tool_was_called(name: str) -> bool` ¶

Check if a specific tool was called.

`tool_call_count(name: str) -> int` ¶

Count how many times a specific tool was called.

`tool_calls_for(name: str) -> list[ToolCall]` ¶

Get all calls to a specific tool.

`tool_was_called_with(name: str, **expected_args: Any) -> bool` ¶

Check if a tool was called with specific argument values.

Returns True if at least one call to the named tool has all the specified argument key-value pairs.

Parameters:

Name	Type	Description	Default
`name`	`str`	Tool name to check.	required
`**expected_args`	`Any`	Expected argument key-value pairs.	`{}`

Example::

assert result.tool_was_called_with("size_vm", region="westeurope", cores=8)
assert result.tool_was_called_with("get_balance", account="checking")

`file(path: str) -> str` ¶

Read the content of a file relative to the working directory.

Parameters:

Name	Type	Description	Default
`path`	`str`	Relative file path (e.g. `"main.py"` or `"src/utils.py"`).	required

Returns:

Type	Description
`str`	File content as a string.

Raises:

Type	Description
`FileNotFoundError`	If the file does not exist.

`file_exists(path: str) -> bool` ¶

Check whether a file exists in the working directory.

Parameters:

Name	Type	Description	Default
`path`	`str`	Relative file path.	required

Returns:

Type	Description
`bool`	`True` if the file exists, `False` otherwise.

`files_matching(pattern: str = '**/*') -> list[Path]` ¶

Find files matching a glob pattern in the working directory.

Parameters:

Name	Type	Description	Default
`pattern`	`str`	Glob pattern relative to the working directory. Defaults to `"*/"` (all files recursively).	`'*/'`

Returns:

Type	Description
`list[Path]`	Sorted list of matching `Path` objects (files only, no
`list[Path]`	directories).

Example::

# All Python files created by the agent
py_files = result.files_matching("**/*.py")
assert py_files, "No Python files were created"

# Top-level test files
test_files = result.files_matching("test_*.py")

Shared result records¶

These types describe captured execution. They are not independent task verdicts.

EvalResult(turns: list[Turn], success: bool, error: str | None = None, duration_ms: float = 0.0, token_usage: dict[str, int] = dict(), cost_usd: float = 0.0, _messages: list[Any] = list(), session_context_count: int = 0, assertions: list[Assertion] = list(), available_tools: list[ToolInfo] = list(), skill_info: SkillInfo | None = None, skill_discovery: SkillDiscovery | None = None, effective_system_prompt: str | None = None, mcp_prompts: list[MCPPrompt] = list(), prompt_name: str | None = None, custom_agent_info: CustomAgentInfo | None = None, premium_requests: float = 0.0, instruction_files: list[InstructionFileInfo] = list(), configuration: dict[str, Any] | None = None, capture_errors: list[str] = list(), evidence_complete: bool | None = None, request_audit: list[RequestAudit] = list(), stop_reason: StopReason | None = None, usage: list[UsageInfo] = list(), tool_calls_admitted: int = 0, model_used: str | None = None, reasoning_traces: list[str] = list(), permission_requested: bool = False, permissions: list[dict[str, Any]] = list(), subagent_invocations: list[SubagentInvocation] = list(), comparison_role: Literal['baseline', 'treatment'] | None = None, clarification_stats: ClarificationStats | None = None) `dataclass` ¶

Result of running an agent with rich inspection capabilities.

Example

result = await copilot_eval(agent, "Hello!") assert result.success assert "hello" in result.final_response.lower() assert result.tool_was_called("read_file")

`execution_results: list[EvalResult]` `property` ¶

This session and separately captured framework-owned descendants.

`messages: list[Any]` `property` ¶

Get full conversation messages for session continuity.

This is a copy of captured messages for inspection. The public copilot_eval fixture does not accept a message-history argument or reuse conversation state between calls.

`is_session_continuation: bool` `property` ¶

Check if this result is part of a multi-turn session.

Returns True if prior messages were passed via the messages parameter.

`final_response: str` `property` ¶

Get the last assistant response.

`all_responses: list[str]` `property` ¶

Get all assistant responses.

`all_tool_calls: list[ToolCall]` `property` ¶

Get all tool calls across all turns.

`tool_names_called: set[str]` `property` ¶

Get set of all tool names that were called.

`asked_for_clarification: bool` `property` ¶

Check if the agent asked for clarification instead of acting.

Returns True if clarification detection was enabled AND the agent asked at least one clarifying question.

Example

result = await copilot_eval(agent, "Check my balance") assert not result.asked_for_clarification

`clarification_count: int` `property` ¶

Number of times the agent asked for clarification.

`tool_context: str` `property` ¶

Summarise tool calls and their results as plain text.

This is a readable evidence summary for investigation. It is not an independent correctness check and does not replace the full trace.

`tool_was_called(name: str) -> bool` ¶

Check if a specific tool was called.

`tool_was_called_from_server(server_name: str, tool_name: str) -> bool` ¶

Check if a specific tool from a named MCP server was called.

Useful for plugin testing where tools may be namespaced by server. Checks both the plain tool name and the server_tool prefixed form.

Parameters:

Name	Type	Description	Default
`server_name`	`str`	Name of the MCP server (e.g., `"filesystem"`).	required
`tool_name`	`str`	Name of the tool (e.g., `"read_file"`).	required

Returns:

Type	Description
`bool`	True if either `tool_name` or `server_name_tool_name` was called.

Example::

assert result.tool_was_called_from_server("banking", "get_balance")

`tool_call_count(name: str) -> int` ¶

Count how many times a specific tool was called.

`tool_calls_for(name: str) -> list[ToolCall]` ¶

Get all calls to a specific tool.

`tool_call_arg(tool_name: str, arg_name: str) -> Any` ¶

Get argument value from the first call to a tool.

Parameters:

Name	Type	Description	Default
`tool_name`	`str`	Name of the tool	required
`arg_name`	`str`	Name of the argument	required

Returns:

Type	Description
`Any`	Argument value or None if not found

`tool_was_called_with(name: str, **expected_args: Any) -> bool` ¶

Check if a tool was called with specific argument values.

Returns True if at least one call to the named tool has all the specified argument key-value pairs.

Parameters:

Name	Type	Description	Default
`name`	`str`	Tool name to check.	required
`**expected_args`	`Any`	Expected argument key-value pairs.	`{}`

Example::

assert result.tool_was_called_with("size_vm", region="westeurope", cores=8)
assert result.tool_was_called_with("get_balance", account="checking")

`tool_images_for(name: str) -> list[ImageContent]` ¶

Get all images returned by a specific tool.

Parameters:

Name	Type	Description	Default
`name`	`str`	Name of the tool (e.g., "screenshot")	required

Returns:

Type	Description
`list[ImageContent]`	List of ImageContent objects from tool calls that returned images.

Example

screenshots = result.tool_images_for("screenshot") assert len(screenshots) > 0 assert screenshots[-1].media_type == "image/png"

`Turn(role: str, content: str, tool_calls: list[ToolCall] = list())` `dataclass` ¶

A single conversational turn.

`text: str` `property` ¶

Get the text content of this turn.

`ToolCall(name: str, arguments: dict[str, Any], result: str | None = None, error: str | None = None, duration_ms: float | None = None, image_content: bytes | None = None, image_media_type: str | None = None, additional_images: list[ImageContent] = list(), call_id: str | None = None, completion_received: bool | None = None, success: bool | None = None)` `dataclass` ¶

A tool call made by the agent.

`evidence_complete: bool` `property` ¶

Whether completion, outcome, and returned content or error were captured.

`ClarificationStats(count: int = 0, turn_indices: list[int] = list(), examples: list[str] = list())` `dataclass` ¶

Statistics about clarification requests detected during execution.

Tracks when the agent asks for user input instead of executing the task. Only populated when clarification_detection is enabled on the agent.

Example

result = await copilot_eval(agent, "Check my balance") if result.clarification_stats: print(f"Eval asked {result.clarification_stats.count} question(s)")

`ToolInfo(name: str, description: str, input_schema: dict[str, Any], server_name: str)` `dataclass` ¶

Recorded metadata about an MCP tool.

Captures the description and schema exposed to the model for inspection.

`SkillInfo(name: str, description: str, instruction_content: str, reference_names: list[str] = list())` `dataclass` ¶

Recorded metadata about a skill.

Captures the skill's system prompt content and references for inspection.

`SubagentInvocation(invocation_id: str, name: str, status: str, duration_ms: float | None = None, result: EvalResult | None = None)` `dataclass` ¶

A subagent invocation observed during agent execution.

Tracks when an orchestrator agent dispatches work to a named sub-agent, along with the final status and duration of that invocation.

Example

result = await copilot_eval(agent, "Build and test the project") assert any(s.name == "coder" for s in result.subagent_invocations) assert all(s.status == "completed" for s in result.subagent_invocations)

Skill test cases and grading export¶

Load skill test cases to parametrize ordinary pytest tests. Free-text expectations are descriptions that your test must translate into checks. export_grading formats supplied booleans and evidence; it does not run a judge.

`load_skill_evals(skill_path: Path | str) -> list[SkillEvalCase]` ¶

Load eval cases from a skill-creator compatible skill directory.

Looks for evals/evals.json in the skill directory and parses it into SkillEvalCase objects for use with pytest parametrize.

Parameters:

Name	Type	Description	Default
`skill_path`	`Path \| str`	Path to skill directory containing evals/evals.json, or direct path to evals.json	required

Returns:

Type	Description
`list[SkillEvalCase]`	List of eval cases

Raises:

Type	Description
`FileNotFoundError`	If evals/evals.json doesn't exist
`ValueError`	If the JSON format is invalid

`SkillEvalCase(id: int, name: str, prompt: str, expected_output: str | None = None, expectations: tuple[str, ...] = (), files: tuple[str, ...] = ())` `dataclass` ¶

A single eval case from skill-creator's evals/evals.json.

Attributes:

Name	Type	Description
`id`	`int`	Numeric identifier from the eval file
`name`	`str`	Human-readable name derived from the prompt
`prompt`	`str`	The user prompt to send to the agent
`expected_output`	`str \| None`	Description of expected result (for documentation)
`expectations`	`tuple[str, ...]`	Descriptions to translate into explicit output checks
`files`	`tuple[str, ...]`	Paths to files referenced by this eval case

`export_grading(result: EvalResult | CopilotResult, expectations: Sequence[str], expectation_results: Sequence[bool], evidence: Sequence[str] | None = None) -> dict[str, Any]` ¶

Export eval result as skill-creator compatible grading.json.

Parameters:

Name	Type	Description	Default
`result`	`EvalResult \| CopilotResult`	The eval result from a test run (EvalResult or CopilotResult)	required
`expectations`	`Sequence[str]`	The assertion texts that were checked	required
`expectation_results`	`Sequence[bool]`	Whether each expectation passed	required
`evidence`	`Sequence[str] \| None`	Optional evidence strings for each expectation	`None`

Returns:

Type	Description
`dict[str, Any]`	Dict matching skill-creator's grading.json schema

Raises:

Type	Description
`ValueError`	If expectations and results have different lengths

Native execution evidence¶

SuiteReport contains captured executions and their ordinary pytest outcomes. generate_json saves it; load_suite_report reads schema-4.0 JSON without starting Copilot. These APIs do not interpret or render the evidence.

`TestReport(name: str, outcome: str, duration_ms: float, eval_result: EvalResult | None = None, error: str | None = None, assertions: list[dict[str, Any]] = list(), docstring: str | None = None, class_docstring: str | None = None, agent_id: str = '', eval_name: str = '', model: str | None = None, system_prompt_name: str | None = None, skill_name: str | None = None, iteration: int | None = None, properties: list[tuple[str, Any]] = list())` `dataclass` ¶

Native evidence for one captured test execution.

Attributes:

Name	Type	Description
`name`	`str`	Full test node ID (e.g., "test_balance[gpt-4o-PROMPT_V1]")
`outcome`	`str`	Test outcome - "passed", "failed", or "skipped"
`duration_ms`	`float`	Test duration in milliseconds
`eval_result`	`EvalResult \| None`	Optional EvalResult from copilot_eval
`error`	`str \| None`	Error message if test failed
`assertions`	`list[dict[str, Any]]`	List of assertion results
`docstring`	`str \| None`	Test function's docstring (first line) for human-readable description
`class_docstring`	`str \| None`	Test class docstring (first line) for human-readable group name
`agent_id`	`str`	Identity supplied by the execution wrapper
`eval_name`	`str`	Name identifying the eval configuration
`model`	`str \| None`	LLM model name (without provider prefix)
`system_prompt_name`	`str \| None`	Label for the system prompt variant
`skill_name`	`str \| None`	Name of the skill used

`short_name: str` `property` ¶

Extract just the test function name from the full node ID.

'tests/test_foo.py::TestClass::test_bar[param]' -> 'test_bar[param]'

`display_name: str` `property` ¶

Human-readable test name: docstring if available, else short test name.

`tokens_used: int` `property` ¶

Get total tokens used from eval_result if present.

`tool_calls: list[str]` `property` ¶

Get tool call names from eval_result if present.

`SuiteReport(name: str, timestamp: str, duration_ms: float, tests: list[TestReport] = list(), passed: int = 0, failed: int = 0, skipped: int = 0, suite_docstring: str | None = None, models_without_pricing: list[str] = list())` `dataclass` ¶

Native evidence for captured executions in a pytest session.

Counts reflect recorded pytest outcomes, not framework-created judgments.

`total_tokens: int` `property` ¶

Sum of all tokens used.

`total_cost_usd: float` `property` ¶

Sum of all costs in USD.

`token_stats: dict[str, int]` `property` ¶

Get min/max/avg token usage.

`test_files: list[str]` `property` ¶

Unique test file paths.

`generate_json(report: SuiteReport, output_path: str | Path) -> None` ¶

Atomically save recorded evidence and pytest outcomes as JSON.

Property values must be JSON-compatible. Unsupported values are errors, not opportunities to replace evidence with string representations.

`load_suite_report(path: str | Path) -> SuiteReport` ¶

Load current-schema native evidence; never infer results or grade answers.

Skill files¶

`Skill(path: Path, metadata: SkillMetadata, content: str, references: dict[str, str] = dict(), scripts: dict[str, str] = dict(), assets: tuple[str, ...] = ())` `dataclass` ¶

An Eval Skill loaded from a SKILL.md file.

Skills provide domain knowledge to agents by: 1. Prepending instructions to the system prompt 2. Optionally providing reference documents via virtual tools

Example

skill = Skill.from_path(Path("skills/my-skill")) agent = CopilotEval(skill_directories=[str(skill.path)])

`name: str` `property` ¶

Skill name from metadata.

`description: str` `property` ¶

Skill description from metadata.

`has_references: bool` `property` ¶

Whether this skill has reference documents.

`has_scripts: bool` `property` ¶

Whether this skill has executable scripts.

`has_assets: bool` `property` ¶

Whether this skill has asset files.

`assets_dir: Path | None` `property` ¶

Path to assets directory, if it exists.

`from_path(path: Path | str) -> Skill` `classmethod` ¶

Load a skill from a directory containing SKILL.md.

Parameters:

Name	Type	Description	Default
`path`	`Path \| str`	Path to skill directory or SKILL.md file	required

Returns:

Type	Description
`Skill`	Loaded Skill instance. Reference keys are slash-separated paths
`Skill`	relative to references/, including any nested subdirectories.

Raises:

Type	Description
`SkillError`	If skill cannot be loaded or is invalid

`SkillMetadata(name: str, description: str, version: str | None = None, license: str | None = None, tags: tuple[str, ...] = (), compatibility: str | None = None, metadata_entries: tuple[tuple[str, str], ...] = (), allowed_tools: tuple[str, ...] = ())` `dataclass` ¶

Metadata from SKILL.md frontmatter.

Required fields per agentskills.io spec: - name: lowercase letters and hyphens only, 1-64 chars - description: what the skill does, max 1024 chars

Optional fields: - version: semantic version string - license: SPDX license identifier - tags: list of categorization tags - compatibility: environment requirements, max 500 chars - metadata: arbitrary key-value pairs (stored as tuple of tuples for frozen compat) - allowed_tools: tool names the skill is designed to work with

`metadata_dict: dict[str, str]` `property` ¶

Return metadata entries as a dict for convenient access.

`__post_init__() -> None` ¶

Validate metadata per agentskills.io spec.

`load_skill(path: Path | str) -> Skill` ¶

Load a skill from a path.

Convenience function wrapping Skill.from_path(). This is the runner's local skill validation, without SDK startup or model execution.

Parameters:

Name	Type	Description	Default
`path`	`Path \| str`	Path to skill directory or SKILL.md file	required

Returns:

Type	Description
`Skill`	Loaded Skill instance

Custom agent files¶

`load_custom_agent(path: Path | str, *, overrides: dict[str, Any] | None = None) -> dict[str, Any]` ¶

Load a .agent.md or .md file into a custom agent dict.

Parses YAML frontmatter with PyYAML for structured metadata (description, tools, handoffs, maturity, etc.) and uses the markdown body as the agent's prompt.

Parameters:

Name	Type	Description	Default
`path`	`Path \| str`	Path to the `.agent.md` or `.md` file.	required
`overrides`	`dict[str, Any] \| None`	Additional fields to merge into the result. Use this to set `tools`, `mcp_servers`, or `infer` fields that aren't in the agent file itself.	`None`

Returns:

Type	Description
`dict[str, Any]`	Dict with keys: - `name` (str): Declared name, or derived from filename when absent. - `prompt` (str): Markdown body after frontmatter. - `description` (str): From frontmatter, empty if absent. - `metadata` (dict): Full parsed frontmatter dict.
`dict[str, Any]`	Compatible with :attr:`CopilotEval.custom_agents`.

Raises:

Type	Description
`FileNotFoundError`	If the file does not exist.
`ValueError`	If the file has no content after frontmatter stripping.

`load_custom_agents(directory: Path | str, *, include: set[str] | None = None, exclude: set[str] | None = None, overrides: dict[str, dict[str, Any]] | None = None) -> list[dict[str, Any]]` ¶

Load all .agent.md files from a directory.

Parameters:

Name	Type	Description	Default
`directory`	`Path \| str`	Path to directory containing `.agent.md` files.	required
`include`	`set[str] \| None`	If set, only load agents with these names. Names are derived from filenames (e.g. `task-researcher.agent.md` → `task-researcher`).	`None`
`exclude`	`set[str] \| None`	Eval names to skip.	`None`
`overrides`	`dict[str, dict[str, Any]] \| None`	Per-agent override dicts keyed by agent name. Merged into each matching agent's config.	`None`

Returns:

Type	Description
`list[dict[str, Any]]`	List of custom agent dicts, sorted by name.

Raises:

Type	Description
`FileNotFoundError`	If the directory does not exist.

Plugin records¶

`Plugin(metadata: PluginMetadata, path: Path, agents: list[dict[str, Any]] = list(), skills: list[Skill] = list(), mcp_servers: dict[str, dict[str, Any]] = dict(), hooks: list[HookDefinition] = list(), instructions: str = '', extensions: list[Path] = list())` `dataclass` ¶

A loaded plugin with all its components resolved.

Created by :func:load_plugin. Contains everything discovered from the plugin directory: agents, skills, MCP servers, hooks, instructions, and extensions.

`PluginMetadata(name: str, version: str = '', description: str = '', author: str = '')` `dataclass` ¶

Plugin manifest metadata.

Extracted from plugin.json or inferred from the directory name.

`HookDefinition(event: str, command: str, pattern: str = '')` `dataclass` ¶

A lifecycle hook from a plugin's hooks configuration.

Hooks allow plugins to execute shell commands at specific lifecycle events.

Example hooks.json::

[
    {
        "event": "tool.execution_complete",
        "command": "npm run lint",
        "pattern": "*.ts"
    }
]

Plugin loading¶

`load_plugin(path: str | Path) -> Plugin` ¶

Load a plugin from a directory containing plugin.json.

Supports both GitHub Copilot CLI and Claude Code plugin formats. Discovers and loads: custom agents, skills, MCP servers, hooks, instructions, and extensions.

For .github/ directories, discovers agents from agents/ and instructions from copilot-instructions.md.

For .claude/ directories, discovers agents from agents/ and instructions from CLAUDE.md.

Parameters:

Name	Type	Description	Default
`path`	`str \| Path`	Path to the plugin directory (must contain plugin.json, or be a `.github/` / `.claude/` project config directory)	required

Returns:

Type	Description
`Plugin`	Plugin with all components resolved

Raises:

Type	Description
`FileNotFoundError`	If plugin.json doesn't exist (for non-project dirs)
`ValueError`	If plugin.json is invalid

API reference¶

Session configuration and result¶

Minimal¶

With instructions and model¶

With custom tools and references¶

build_session_config() -> dict[str, Any] ¶

create_subagent(*, name: str, model: str | None, reasoning_effort: CopilotReasoningEffort | None, instructions: str, timeout_s: float, max_turns: int, allowed_tools: list[str] | None, mcp_servers: dict[str, CopilotMCPServerConfig]) -> CopilotEval ¶

from_copilot_config(path: str | Path = '.', **overrides: Any) -> 'CopilotEval' classmethod ¶

from_plugin(path: str | Path, *, model: str = '', persona: CopilotPersona | None = None, instructions: str = '', working_directory: str = '', name: str = '', **overrides: Any) -> 'CopilotEval' classmethod ¶

from_claude_config(path: str | Path = '.', *, model: str = '', persona: CopilotPersona | None = None, instructions: str = '', working_directory: str = '', name: str = 'claude-code-eval', **overrides: Any) -> 'CopilotEval' classmethod ¶

evidence_complete: bool property ¶

execution_results: list[CopilotResult] property ¶

all_usage: list[UsageInfo] property ¶

final_response: str | None property ¶

all_responses: list[str] property ¶

all_tool_calls: list[ToolCall] property ¶

tool_names_called: set[str] property ¶

total_input_tokens: int | None property ¶

total_output_tokens: int | None property ¶

total_tokens: int | None property ¶

token_usage: dict[str, int] property ¶

working_directory: Path property ¶

tool_was_called(name: str) -> bool ¶

tool_call_count(name: str) -> int ¶

tool_calls_for(name: str) -> list[ToolCall] ¶

tool_was_called_with(name: str, **expected_args: Any) -> bool ¶

file(path: str) -> str ¶

file_exists(path: str) -> bool ¶

files_matching(pattern: str = '**/*') -> list[Path] ¶

Shared result records¶

execution_results: list[EvalResult] property ¶

messages: list[Any] property ¶

is_session_continuation: bool property ¶

final_response: str property ¶

all_responses: list[str] property ¶

all_tool_calls: list[ToolCall] property ¶

tool_names_called: set[str] property ¶

asked_for_clarification: bool property ¶

clarification_count: int property ¶

tool_context: str property ¶

tool_was_called(name: str) -> bool ¶

tool_was_called_from_server(server_name: str, tool_name: str) -> bool ¶

tool_call_count(name: str) -> int ¶

tool_calls_for(name: str) -> list[ToolCall] ¶

tool_call_arg(tool_name: str, arg_name: str) -> Any ¶

tool_was_called_with(name: str, **expected_args: Any) -> bool ¶

tool_images_for(name: str) -> list[ImageContent] ¶

Turn(role: str, content: str, tool_calls: list[ToolCall] = list()) dataclass ¶

text: str property ¶

evidence_complete: bool property ¶

ClarificationStats(count: int = 0, turn_indices: list[int] = list(), examples: list[str] = list()) dataclass ¶

ToolInfo(name: str, description: str, input_schema: dict[str, Any], server_name: str) dataclass ¶

SkillInfo(name: str, description: str, instruction_content: str, reference_names: list[str] = list()) dataclass ¶

SubagentInvocation(invocation_id: str, name: str, status: str, duration_ms: float | None = None, result: EvalResult | None = None) dataclass ¶

Skill test cases and grading export¶

load_skill_evals(skill_path: Path | str) -> list[SkillEvalCase] ¶

SkillEvalCase(id: int, name: str, prompt: str, expected_output: str | None = None, expectations: tuple[str, ...] = (), files: tuple[str, ...] = ()) dataclass ¶

export_grading(result: EvalResult | CopilotResult, expectations: Sequence[str], expectation_results: Sequence[bool], evidence: Sequence[str] | None = None) -> dict[str, Any] ¶

Native execution evidence¶

short_name: str property ¶

display_name: str property ¶

tokens_used: int property ¶

tool_calls: list[str] property ¶

SuiteReport(name: str, timestamp: str, duration_ms: float, tests: list[TestReport] = list(), passed: int = 0, failed: int = 0, skipped: int = 0, suite_docstring: str | None = None, models_without_pricing: list[str] = list()) dataclass ¶

total_tokens: int property ¶

total_cost_usd: float property ¶

token_stats: dict[str, int] property ¶

test_files: list[str] property ¶

generate_json(report: SuiteReport, output_path: str | Path) -> None ¶

load_suite_report(path: str | Path) -> SuiteReport ¶

Skill files¶

Skill(path: Path, metadata: SkillMetadata, content: str, references: dict[str, str] = dict(), scripts: dict[str, str] = dict(), assets: tuple[str, ...] = ()) dataclass ¶

name: str property ¶

description: str property ¶

has_references: bool property ¶

has_scripts: bool property ¶

has_assets: bool property ¶

assets_dir: Path | None property ¶

from_path(path: Path | str) -> Skill classmethod ¶

SkillMetadata(name: str, description: str, version: str | None = None, license: str | None = None, tags: tuple[str, ...] = (), compatibility: str | None = None, metadata_entries: tuple[tuple[str, str], ...] = (), allowed_tools: tuple[str, ...] = ()) dataclass ¶

`build_session_config() -> dict[str, Any]` ¶

`create_subagent(*, name: str, model: str | None, reasoning_effort: CopilotReasoningEffort | None, instructions: str, timeout_s: float, max_turns: int, allowed_tools: list[str] | None, mcp_servers: dict[str, CopilotMCPServerConfig]) -> CopilotEval` ¶

`from_copilot_config(path: str | Path = '.', **overrides: Any) -> 'CopilotEval'` `classmethod` ¶

`from_plugin(path: str | Path, *, model: str = '', persona: CopilotPersona | None = None, instructions: str = '', working_directory: str = '', name: str = '', **overrides: Any) -> 'CopilotEval'` `classmethod` ¶

`from_claude_config(path: str | Path = '.', *, model: str = '', persona: CopilotPersona | None = None, instructions: str = '', working_directory: str = '', name: str = 'claude-code-eval', **overrides: Any) -> 'CopilotEval'` `classmethod` ¶

`evidence_complete: bool` `property` ¶

`execution_results: list[CopilotResult]` `property` ¶

`all_usage: list[UsageInfo]` `property` ¶

`final_response: str | None` `property` ¶

`all_responses: list[str]` `property` ¶

`all_tool_calls: list[ToolCall]` `property` ¶

`tool_names_called: set[str]` `property` ¶

`total_input_tokens: int | None` `property` ¶

`total_output_tokens: int | None` `property` ¶

`total_tokens: int | None` `property` ¶

`token_usage: dict[str, int]` `property` ¶

`working_directory: Path` `property` ¶

`tool_was_called(name: str) -> bool` ¶

`tool_call_count(name: str) -> int` ¶

`tool_calls_for(name: str) -> list[ToolCall]` ¶

`tool_was_called_with(name: str, **expected_args: Any) -> bool` ¶

`file(path: str) -> str` ¶

`file_exists(path: str) -> bool` ¶

`files_matching(pattern: str = '**/*') -> list[Path]` ¶

`execution_results: list[EvalResult]` `property` ¶

`messages: list[Any]` `property` ¶

`is_session_continuation: bool` `property` ¶

`final_response: str` `property` ¶

`all_responses: list[str]` `property` ¶

`all_tool_calls: list[ToolCall]` `property` ¶

`tool_names_called: set[str]` `property` ¶

`asked_for_clarification: bool` `property` ¶

`clarification_count: int` `property` ¶

`tool_context: str` `property` ¶

`tool_was_called(name: str) -> bool` ¶

`tool_was_called_from_server(server_name: str, tool_name: str) -> bool` ¶

`tool_call_count(name: str) -> int` ¶

`tool_calls_for(name: str) -> list[ToolCall]` ¶

`tool_call_arg(tool_name: str, arg_name: str) -> Any` ¶

`tool_was_called_with(name: str, **expected_args: Any) -> bool` ¶

`tool_images_for(name: str) -> list[ImageContent]` ¶

`Turn(role: str, content: str, tool_calls: list[ToolCall] = list())` `dataclass` ¶

`text: str` `property` ¶

`evidence_complete: bool` `property` ¶

`ClarificationStats(count: int = 0, turn_indices: list[int] = list(), examples: list[str] = list())` `dataclass` ¶

`ToolInfo(name: str, description: str, input_schema: dict[str, Any], server_name: str)` `dataclass` ¶

`SkillInfo(name: str, description: str, instruction_content: str, reference_names: list[str] = list())` `dataclass` ¶

`SubagentInvocation(invocation_id: str, name: str, status: str, duration_ms: float | None = None, result: EvalResult | None = None)` `dataclass` ¶

`load_skill_evals(skill_path: Path | str) -> list[SkillEvalCase]` ¶

`SkillEvalCase(id: int, name: str, prompt: str, expected_output: str | None = None, expectations: tuple[str, ...] = (), files: tuple[str, ...] = ())` `dataclass` ¶

`export_grading(result: EvalResult | CopilotResult, expectations: Sequence[str], expectation_results: Sequence[bool], evidence: Sequence[str] | None = None) -> dict[str, Any]` ¶

`short_name: str` `property` ¶

`display_name: str` `property` ¶

`tokens_used: int` `property` ¶

`tool_calls: list[str]` `property` ¶

`SuiteReport(name: str, timestamp: str, duration_ms: float, tests: list[TestReport] = list(), passed: int = 0, failed: int = 0, skipped: int = 0, suite_docstring: str | None = None, models_without_pricing: list[str] = list())` `dataclass` ¶

`total_tokens: int` `property` ¶

`total_cost_usd: float` `property` ¶

`token_stats: dict[str, int]` `property` ¶

`test_files: list[str]` `property` ¶

`generate_json(report: SuiteReport, output_path: str | Path) -> None` ¶

`load_suite_report(path: str | Path) -> SuiteReport` ¶

`Skill(path: Path, metadata: SkillMetadata, content: str, references: dict[str, str] = dict(), scripts: dict[str, str] = dict(), assets: tuple[str, ...] = ())` `dataclass` ¶

`name: str` `property` ¶

`description: str` `property` ¶

`has_references: bool` `property` ¶

`has_scripts: bool` `property` ¶

`has_assets: bool` `property` ¶

`assets_dir: Path | None` `property` ¶

`from_path(path: Path | str) -> Skill` `classmethod` ¶

`SkillMetadata(name: str, description: str, version: str | None = None, license: str | None = None, tags: tuple[str, ...] = (), compatibility: str | None = None, metadata_entries: tuple[tuple[str, str], ...] = (), allowed_tools: tuple[str, ...] = ())` `dataclass` ¶

`metadata_dict: dict[str, str]` `property` ¶

`__post_init__() -> None` ¶

`load_skill(path: Path | str) -> Skill` ¶

`load_custom_agent(path: Path | str, *, overrides: dict[str, Any] | None = None) -> dict[str, Any]` ¶

`load_custom_agents(directory: Path | str, *, include: set[str] | None = None, exclude: set[str] | None = None, overrides: dict[str, dict[str, Any]] | None = None) -> list[dict[str, Any]]` ¶

`Plugin(metadata: PluginMetadata, path: Path, agents: list[dict[str, Any]] = list(), skills: list[Skill] = list(), mcp_servers: dict[str, dict[str, Any]] = dict(), hooks: list[HookDefinition] = list(), instructions: str = '', extensions: list[Path] = list())` `dataclass` ¶

`PluginMetadata(name: str, version: str = '', description: str = '', author: str = '')` `dataclass` ¶

`HookDefinition(event: str, command: str, pattern: str = '')` `dataclass` ¶

`load_plugin(path: str | Path) -> Plugin` ¶