Test Report
Direct Links
The embedded report above is the latest CI-generated run. The static example showcases a mix of passed, failed, skipped, xfailed, xpassed, and error results.
About the Test Report
This report is automatically generated during CI/CD runs and includes:
- Test Results: Pass/fail status for all tests
- Coverage Data: Per-test coverage information showing which files and lines each test exercises
- LLM Annotations (optional): Scenario, why needed, key assertions, confidence score, and token usage when LLMs are enabled
- Execution Time: Duration for each test
- Error Messages: Full error details for failed tests
- Source Coverage: Per-file covered/missed/percentage summary (similar to pytest-cov)
The report uses the same pytest-llm-report plugin that this documentation describes.
Report Features
Interactive Filtering
- Search tests by name
- Filter by test status (passed, failed, skipped, etc.)
- Failures Only: Quickly filter to show only failed/error tests
- Toggle visibility of passed tests
Test Organization
- Table of Contents: Quick navigation to Source Coverage and Test Details
- Group by File: Tests are grouped by their source file for better overview
Coverage Details
For each test, you can expand to see:
- Which source files were executed during the test
- Specific line ranges covered in each file
- Total line count per file
PDF Report
The PDF report mirrors the HTML view but is optimized for printing, with all test details expanded to show LLM annotations and line coverage context.
Report Metadata
The report header shows:
- Total test count and breakdown by status
- Overall test suite duration
- Total coverage percentage (when available)
- Run metadata (timestamp, Python version, configuration)
- Versioning Info:
- Plugin version (with Git SHA and dirty status)
- Analyzed Repository version (from
pyproject.toml+ Git SHA and dirty status)
Source Coverage Summary
At the bottom of the report you'll find a table of source files with:
- Statements, missed, and covered counts
- Coverage percentage per file
- Covered and missed line ranges for quick inspection