compiler/docs/en/TEST_COVERAGE_ANALYZER.md

Test Coverage Checklist

bin/analyze-test-coverage.php generates a coverage checklist from the source of PHPT and compiler PHPUnit fixtures. It is a static test-intent analysis tool and does not replace test execution.

Usage

# terminal summary
php bin/analyze-test-coverage.php

# reviewable full matrix
php bin/analyze-test-coverage.php \
  --format=markdown \
  --output=build/test-coverage.md

# for CI or other tools to read
php bin/analyze-test-coverage.php \
  --format=json \
  --output=build/test-coverage.json \
  --strict

By default it scans tests/compiler, phpunit/src, and phpunit/code. One or more PHPT files or directories can also be passed at the end of the command; --no-phpunit analyzes only PHPT, and --php-versions=8.4,8.5 sets the PHP version columns of the matrix.

--strict returns a non-zero status when there are unexpected source-parse failures or unresolvable PHPUnit fixture references. Samples in negative data providers that are intentionally unparseable by php-parser are recorded separately under expected_parser_diagnostics and are not disguised as tool failures.

Three categories of coverage evidence

Each applicable PHP version × feature row records:

• positive_compile: valid PHPT, or positive PHPUnit compile fixtures;
• runtime_semantics: valid PHPT containing EXPECT, EXPECTF, or EXPECTREGEX;
• negative_diagnostic: PHPT expecting diagnostics, or PHPUnit tests/data providers that explicitly expect failure.

XFAIL and unconditional SKIPIF are not counted on any evidence axis. PHP version ranges are inferred from test titles, PHP_VERSION_ID conditions in SKIPIF, and version strings in PHPUnit data rows.

Denominator

The report only gives ratios with an explicit denominator:

• AST node coverage denominator: the concrete AST node kinds provided by the currently installed nikic/php-parser; Expr_Error, used for error recovery, is not counted.
• Feature-axis coverage denominator: the number of rows in the feature catalog with introduced <= target PHP version. Each of the positive-compile, runtime-semantics, and negative-diagnostic axes is computed independently.

The tool does not combine the three axes of different meanings into a single "overall project coverage". The full JSON preserves the feature catalog, per-item evidence sources, matrix, AST node occurrence counts, parse issues, and exclusion reasons, for further inspection by CI.

Classification boundaries

AST nodes are extracted automatically by the parser. Semantic features that cannot be distinguished by nodes alone (such as DNF occurrence positions, property hook variants, exit(message: ...)) are supplemented by the explicit feature catalog in the analyzer. When adding a new language feature, register its introduction version and detection rule at the same time to keep the version matrix's denominator explicit.