pytest-skill-engineering¶
Test whether GitHub Copilot can complete tasks with your tools and instructions.
pytest-skill-engineering runs real Copilot sessions from pytest and saves their execution as JSON. Use it for MCP servers, CLI tools, skills, system prompts, plugins, and custom agents.
Each test has three parts:
- Configure a session with
CopilotEval. - Send a user task through the
copilot_evalfixture. - Check the actual output with ordinary assertions.
The framework manages sessions, limits, usage, and captured evidence. You supply the application fixtures and success criteria. A completed session or a tool call does not establish that the task succeeded; check files, application state, or operation output.
Start here¶
Getting started walks through installation, the bundled Todo MCP test, and the saved result. You need Python 3.11+, uv, and GitHub Copilot access. Live tests may consume premium requests.
Find the right guide¶
| Goal | Start here |
|---|---|
| Run a first test | Getting started |
| Test your own interface | Task guides |
| Compare two configurations | Comparisons |
| Investigate a failure | Inspect evidence |
| Look up a setting or result field | Reference |
| Understand execution and verification | Explanation |
| Upgrade from 0.6.x | 1.0 migration |
Learn from recorded experiments¶
The companion-skill study and Excel skill study show how fixed output checks can test whether added guidance helps. They are historical observations, not setup requirements or promises about future runs.