From 37faeda17c015c64125801358df5323ad935d943 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 16 Aug 2026 09:11:30 +0000 Subject: [PATCH 01/11] Make the test runners quieter for non-interactive consumers Adds two opt-in environment variables to the runner scripts, both unset by default so existing output is unchanged: * NO_COLOR (https://no-color.org/) stops the runners from forcing ANSI color codes on. parallel-lint and PHPUnit forced them unconditionally, which meant escape sequences in every captured log. * WP_CLI_TEST_QUIET switches the reporters to their most compact form: PHP_CodeSniffer to one line per violation with no progress ticker, PHPStan to one line per error with no progress bar and no result table, and Behat to omitting step definition snippets. Also documents narrowing a Behat run to a single scenario, --stop-on-failure and composer behat-rerun. Refs https://github.com/wp-cli/wp-cli/issues/6161 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01VcGmbu6CYzGGhP3jXuWDkJ --- .readme-partials/USING.md | 33 +++++++++++++++++++++++++++++++++ README.md | 33 +++++++++++++++++++++++++++++++++ bin/run-behat-tests | 13 +++++++++++++ bin/run-linter-tests | 10 +++++++++- bin/run-php-unit-tests | 14 ++++++++++++-- bin/run-phpcs-tests | 12 +++++++++++- bin/run-phpstan-tests | 19 +++++++++++++++++-- 7 files changed, 128 insertions(+), 6 deletions(-) diff --git a/.readme-partials/USING.md b/.readme-partials/USING.md index 3b2528f76..32d94e3b2 100644 --- a/.readme-partials/USING.md +++ b/.readme-partials/USING.md @@ -104,6 +104,39 @@ composer behat -- features/cli-info.feature Prepending with the double dash is needed because the arguments would otherwise be sent to Composer itself, not the tool that Composer executes. +The same mechanism works for narrowing a run down further, or for bailing out early: +```bash +# A single scenario, identified by the line it starts on. +composer behat -- features/cli-info.feature:12 + +# Every scenario carrying a given tag. +composer behat -- --tags=@require-wp-5.0 + +# Stop at the first failing scenario instead of running the whole suite. +composer behat -- --stop-on-failure + +# Re-run only the scenarios that failed the last time. +composer behat-rerun +``` + +### Controlling the amount of output + +Two environment variables make the test tools less chatty. Both are unset by default, which leaves the output exactly as it has always been. + + - `NO_COLOR` (the [no-color.org](https://no-color.org/) convention) stops the runners from forcing ANSI color codes on, and leaves the decision to each tool's own terminal detection. Set this when capturing output to a file or a pipe, where the escape sequences are noise. + - `WP_CLI_TEST_QUIET` switches the reporters to their most compact form: PHP_CodeSniffer reports one `file:line:col` line per violation with no progress ticker, PHPStan reports one `file:line:message` line per error with no progress bar and no result table, and Behat stops printing step definition snippets for undefined steps. + +```bash +NO_COLOR=1 WP_CLI_TEST_QUIET=1 composer phpstan +``` + +This is worth setting permanently in environments that read the output back rather than display it, such as an AI coding agent's shell: + +```bash +export NO_COLOR=1 +export WP_CLI_TEST_QUIET=1 +``` + ### Controlling the test environment #### WordPress Version diff --git a/README.md b/README.md index 6eeff87ab..6a9b6171f 100644 --- a/README.md +++ b/README.md @@ -115,6 +115,39 @@ composer behat -- features/cli-info.feature Prepending with the double dash is needed because the arguments would otherwise be sent to Composer itself, not the tool that Composer executes. +The same mechanism works for narrowing a run down further, or for bailing out early: +```bash +# A single scenario, identified by the line it starts on. +composer behat -- features/cli-info.feature:12 + +# Every scenario carrying a given tag. +composer behat -- --tags=@require-wp-5.0 + +# Stop at the first failing scenario instead of running the whole suite. +composer behat -- --stop-on-failure + +# Re-run only the scenarios that failed the last time. +composer behat-rerun +``` + +### Controlling the amount of output + +Two environment variables make the test tools less chatty. Both are unset by default, which leaves the output exactly as it has always been. + + - `NO_COLOR` (the [no-color.org](https://no-color.org/) convention) stops the runners from forcing ANSI color codes on, and leaves the decision to each tool's own terminal detection. Set this when capturing output to a file or a pipe, where the escape sequences are noise. + - `WP_CLI_TEST_QUIET` switches the reporters to their most compact form: PHP_CodeSniffer reports one `file:line:col` line per violation with no progress ticker, PHPStan reports one `file:line:message` line per error with no progress bar and no result table, and Behat stops printing step definition snippets for undefined steps. + +```bash +NO_COLOR=1 WP_CLI_TEST_QUIET=1 composer phpstan +``` + +This is worth setting permanently in environments that read the output back rather than display it, such as an AI coding agent's shell: + +```bash +export NO_COLOR=1 +export WP_CLI_TEST_QUIET=1 +``` + ### Controlling the test environment #### WordPress Version diff --git a/bin/run-behat-tests b/bin/run-behat-tests index b276c7cbb..6d959bafa 100755 --- a/bin/run-behat-tests +++ b/bin/run-behat-tests @@ -141,6 +141,19 @@ if [[ "${WP_CLI_TEST_COVERAGE}" == "true" ]] && vendor/bin/behat --help 2>/dev/n BEHAT_EXTRA_ARGS+=('--xdebug') fi +# Honor the NO_COLOR convention (https://no-color.org/). +if [ -n "${NO_COLOR}" ]; then + BEHAT_EXTRA_ARGS+=('--no-colors') +fi + +# Drop the step definition snippets that Behat prints for undefined steps. They +# are long, and they are only actionable when you are writing new step +# definitions in this package. Useful for CI logs and for AI coding agents, +# which pay for every token of the report they read back. +if [ -n "${WP_CLI_TEST_QUIET}" ]; then + BEHAT_EXTRA_ARGS+=('--no-snippets') +fi + # Run the functional tests. FORMAT_ARGS=(--format progress) for arg in "$@"; do diff --git a/bin/run-linter-tests b/bin/run-linter-tests index add8ec336..d2c374ca7 100755 --- a/bin/run-linter-tests +++ b/bin/run-linter-tests @@ -1,3 +1,11 @@ #!/bin/sh -vendor/bin/parallel-lint -j 10 --colors --exclude vendor . "$@" +# Honor the NO_COLOR convention (https://no-color.org/) by leaving the color +# decision to the tool's own TTY detection instead of forcing it. +COLOR_ARGS="--colors" +if [ -n "${NO_COLOR}" ]; then + COLOR_ARGS="" +fi + +# shellcheck disable=SC2086 # Intentional word splitting of the optional arguments. +vendor/bin/parallel-lint -j 10 $COLOR_ARGS --exclude vendor . "$@" diff --git a/bin/run-php-unit-tests b/bin/run-php-unit-tests index 6182d0746..3d89befab 100755 --- a/bin/run-php-unit-tests +++ b/bin/run-php-unit-tests @@ -10,9 +10,19 @@ then EXTRA_ARGS="--display-warnings --fail-on-warning --display-notices --fail-on-notice --display-deprecations --fail-on-deprecation" fi + # Honor the NO_COLOR convention (https://no-color.org/). This has to be an + # explicit "never" rather than an omitted flag, because a project's + # phpunit.xml may well turn colors on by itself. + COLOR_ARGS="--color=always" + if [ -n "${NO_COLOR}" ]; then + COLOR_ARGS="--color=never" + fi + if [ -f "./vendor/wp-cli/wp-cli-tests/tests/bootstrap.php" ]; then - vendor/bin/phpunit --color=always "$@" $EXTRA_ARGS --bootstrap ./vendor/wp-cli/wp-cli-tests/tests/bootstrap.php + # shellcheck disable=SC2086 # Intentional word splitting of the optional arguments. + vendor/bin/phpunit $COLOR_ARGS "$@" $EXTRA_ARGS --bootstrap ./vendor/wp-cli/wp-cli-tests/tests/bootstrap.php else - vendor/bin/phpunit --color=always "$@" $EXTRA_ARGS + # shellcheck disable=SC2086 # Intentional word splitting of the optional arguments. + vendor/bin/phpunit $COLOR_ARGS "$@" $EXTRA_ARGS fi fi diff --git a/bin/run-phpcs-tests b/bin/run-phpcs-tests index 82d96b4e5..cff48a9bf 100755 --- a/bin/run-phpcs-tests +++ b/bin/run-phpcs-tests @@ -3,5 +3,15 @@ # Run the code style check only if a configuration file exists. if [ -f ".phpcs.xml" ] || [ -f "phpcs.xml" ] || [ -f ".phpcs.xml.dist" ] || [ -f "phpcs.xml.dist" ] then - vendor/bin/phpcs "$@" + EXTRA_ARGS="" + + # Compact, one-line-per-violation output without the progress ticker. + # Useful for CI logs and for AI coding agents, which pay for every token of + # the report they read back. + if [ -n "${WP_CLI_TEST_QUIET}" ]; then + EXTRA_ARGS="-q --report=emacs" + fi + + # shellcheck disable=SC2086 # Intentional word splitting of the optional arguments. + vendor/bin/phpcs $EXTRA_ARGS "$@" fi diff --git a/bin/run-phpstan-tests b/bin/run-phpstan-tests index 21f276900..235242ae8 100755 --- a/bin/run-phpstan-tests +++ b/bin/run-phpstan-tests @@ -1,7 +1,22 @@ #!/bin/sh -# Run the code style check only if a configuration file exists. +# Run the static analysis only if a configuration file exists. if [ -f "phpstan.dist.neon" ] || [ -f "phpstan.neon.dist" ] || [ -f "phpstan.neon" ] then - vendor/bin/phpstan --memory-limit=2048M analyse "$@" + EXTRA_ARGS="" + + # Honor the NO_COLOR convention (https://no-color.org/). + if [ -n "${NO_COLOR}" ]; then + EXTRA_ARGS="--no-ansi" + fi + + # Replace the redrawing progress bar and the box-drawing result table with + # plain `file:line:message` lines. Useful for CI logs and for AI coding + # agents, which pay for every token of the report they read back. + if [ -n "${WP_CLI_TEST_QUIET}" ]; then + EXTRA_ARGS="${EXTRA_ARGS} --no-progress --error-format=raw" + fi + + # shellcheck disable=SC2086 # Intentional word splitting of the optional arguments. + vendor/bin/phpstan --memory-limit=2048M analyse $EXTRA_ARGS "$@" fi From f38e2e6226115aadba2abf8d54b1ded5dd2b2a89 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 16 Aug 2026 09:11:30 +0000 Subject: [PATCH 02/11] Cache the WP_VERSION lookups performed by the Behat runner Every `composer behat` invocation resolved WP_VERSION over the network: one request to api.wordpress.org for `latest`, and a second one to the wp-versions artifact when the version has no patch number. That cost applies equally to a full suite run and to re-running one scenario for the fifth time while iterating on a fix. The answers now go into a cache in the system temp directory with a configurable lifetime, defaulting to a day. Two side effects worth noting: * A run without connectivity falls back to the last known answer rather than continuing with an empty WP_VERSION, which silently disabled the filtering of version-specific tags. * When there is nothing to fall back to, that case is now reported instead of being silent. Refs https://github.com/wp-cli/wp-cli/issues/6161 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01VcGmbu6CYzGGhP3jXuWDkJ --- .readme-partials/USING.md | 6 ++++ README.md | 6 ++++ bin/run-behat-tests | 71 +++++++++++++++++++++++++++++++++++++-- 3 files changed, 81 insertions(+), 2 deletions(-) diff --git a/.readme-partials/USING.md b/.readme-partials/USING.md index 32d94e3b2..dfa697125 100644 --- a/.readme-partials/USING.md +++ b/.readme-partials/USING.md @@ -152,6 +152,12 @@ Here's how to run your tests against the latest trunk version of WordPress: WP_VERSION=trunk composer behat ``` +Resolving `latest`, or a `X.Y` version without a patch number, needs a network +request. The answer is cached in the system temp directory for a day, so that +repeated runs do not repeat the lookup, and so that a run without connectivity +falls back to the last known answer. `WP_CLI_TEST_WP_VERSION_CACHE_TTL` sets the +lifetime of that cache in seconds; `0` looks the version up every time. + #### WordPress Archive Instead of downloading WordPress from WordPress.org, you can run the tests against an arbitrary diff --git a/README.md b/README.md index 6a9b6171f..022bf2f7d 100644 --- a/README.md +++ b/README.md @@ -163,6 +163,12 @@ Here's how to run your tests against the latest trunk version of WordPress: WP_VERSION=trunk composer behat ``` +Resolving `latest`, or a `X.Y` version without a patch number, needs a network +request. The answer is cached in the system temp directory for a day, so that +repeated runs do not repeat the lookup, and so that a run without connectivity +falls back to the last known answer. `WP_CLI_TEST_WP_VERSION_CACHE_TTL` sets the +lifetime of that cache in seconds; `0` looks the version up every time. + #### WordPress Archive Instead of downloading WordPress from WordPress.org, you can run the tests against an arbitrary diff --git a/bin/run-behat-tests b/bin/run-behat-tests index 6d959bafa..068a283bd 100755 --- a/bin/run-behat-tests +++ b/bin/run-behat-tests @@ -97,9 +97,64 @@ if [ -n "${WP_CLI_TEST_CORE_ZIP-}" ] && [ -z "${WP_VERSION-}" ]; then export WP_VERSION=trunk fi +# Resolving WP_VERSION costs up to two network round trips on every invocation. +# Cache the answers, so that re-running a single scenario while iterating does +# not repeat them, and so that a run without connectivity can fall back to the +# last known answer instead of ending up with no version at all. +# +# Set WP_CLI_TEST_WP_VERSION_CACHE_TTL to 0 to always refetch. +WP_VERSION_CACHE_DIR="${TMPDIR:-/tmp}/wp-cli-test-wp-version-cache" +WP_VERSION_CACHE_TTL="${WP_CLI_TEST_WP_VERSION_CACHE_TTL:-86400}" + +# Print a cache entry if it exists and is younger than the given number of +# seconds. A negative TTL accepts an entry of any age. +read_version_cache() { + local cache_file="${WP_VERSION_CACHE_DIR}/$1" + local ttl="$2" + local age + + [ -s "${cache_file}" ] || return 1 + + if [ "${ttl}" -ge 0 ]; then + # PHP rather than `find -newermt`, which is not portable across + # GNU and BSD userlands. The Behat runner needs PHP anyway. + age=$(php -r 'echo time() - filemtime( $argv[1] );' "${cache_file}" 2>/dev/null) + case ${age} in + ''|*[!0-9]*) return 1;; + esac + [ "${age}" -lt "${ttl}" ] || return 1 + fi + + cat "${cache_file}" +} + +write_version_cache() { + mkdir -p "${WP_VERSION_CACHE_DIR}" 2>/dev/null || return 0 + printf '%s' "$2" > "${WP_VERSION_CACHE_DIR}/$1" 2>/dev/null || true +} + # Turn WP_VERSION into an actual number to make sure our tags work correctly. if [ "${WP_VERSION-latest}" = "latest" ]; then - export WP_VERSION=$(curl -s https://api.wordpress.org/core/version-check/1.7/ | jq -r ".offers[0].current") + LATEST_WP_VERSION=$(read_version_cache latest "${WP_VERSION_CACHE_TTL}") + + if [ -z "${LATEST_WP_VERSION}" ]; then + LATEST_WP_VERSION=$(curl -s https://api.wordpress.org/core/version-check/1.7/ | jq -r ".offers[0].current // empty") + + if [ -n "${LATEST_WP_VERSION}" ]; then + write_version_cache latest "${LATEST_WP_VERSION}" + else + # Prefer a stale answer over no answer. + LATEST_WP_VERSION=$(read_version_cache latest -1) + + if [ -n "${LATEST_WP_VERSION}" ]; then + echo "Warning: Could not reach api.wordpress.org, falling back to the cached latest WordPress version ${LATEST_WP_VERSION}." + else + echo "Warning: Could not determine the latest WordPress version. Version-specific tags will not be filtered." + fi + fi + fi + + export WP_VERSION="${LATEST_WP_VERSION}" fi # Normalize WP_VERSION=X.Y.0 to X.Y (WordPress uses X.Y for the initial release, not X.Y.0). @@ -107,7 +162,19 @@ fi if [[ "${WP_VERSION}" =~ ^([0-9]+\.[0-9]+)\.0$ ]]; then export WP_VERSION="${BASH_REMATCH[1]}" elif [[ "${WP_VERSION}" =~ ^[0-9]+\.[0-9]+$ ]]; then - WP_VERSIONS_JSON=$(curl -s https://raw.githubusercontent.com/wp-cli/wp-cli-tests/artifacts/wp-versions.json) + WP_VERSIONS_JSON=$(read_version_cache wp-versions.json "${WP_VERSION_CACHE_TTL}") + + if [ -z "${WP_VERSIONS_JSON}" ]; then + WP_VERSIONS_JSON=$(curl -s https://raw.githubusercontent.com/wp-cli/wp-cli-tests/artifacts/wp-versions.json) + + # Only cache a well-formed response; a proxy error page is not one. + if echo "${WP_VERSIONS_JSON}" | jq -e 'type == "object"' > /dev/null 2>&1; then + write_version_cache wp-versions.json "${WP_VERSIONS_JSON}" + else + WP_VERSIONS_JSON=$(read_version_cache wp-versions.json -1) + fi + fi + if [ -n "${WP_VERSIONS_JSON}" ]; then RESOLVED_VERSION=$(echo "${WP_VERSIONS_JSON}" | jq -r --arg prefix "${WP_VERSION}." 'keys | map(select(startswith($prefix))) | sort_by(split(".") | map(tonumber)) | last // empty') if [ -n "${RESOLVED_VERSION}" ]; then From 1d6afbc6399e99f88b35236f1bd33c8a8791df4f Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 16 Aug 2026 09:27:40 +0000 Subject: [PATCH 03/11] Add a Gherkin linter to the test suite The feature files are linted on every pull request, but the check lives entirely in the reusable CI workflow and its ruleset lives in wp-cli/.github, so contributors cannot run it locally at all. A green `composer test` is therefore not a green build, and the way to find out is to push. Moves the check to where the other suites are: `.gherkin-lintrc` ships with this package as the shared default, a project can override it by committing its own, and `composer lint-gherkin` runs it. CI can then call the same script rather than reimplementing the invocation. Uses gherkin-lint-plus, pinned, and overridable through WP_CLI_TEST_GHERKIN_LINT_VERSION. Being a Node package, it is invoked through npx and skips with a message where npx is absent, rather than failing a suite that is otherwise entirely PHP. The linter colors its report unconditionally and offers no plain output format, so NO_COLOR strips the escape sequences from its output. Refs https://github.com/wp-cli/wp-cli/issues/6161 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01VcGmbu6CYzGGhP3jXuWDkJ --- .gherkin-lintrc | 32 +++++++++++++++ .readme-partials/USING.md | 15 +++++++ README.md | 15 +++++++ bin/run-gherkin-lint-tests | 82 ++++++++++++++++++++++++++++++++++++++ composer.json | 3 ++ 5 files changed, 147 insertions(+) create mode 100644 .gherkin-lintrc create mode 100755 bin/run-gherkin-lint-tests diff --git a/.gherkin-lintrc b/.gherkin-lintrc new file mode 100644 index 000000000..a0e5d631d --- /dev/null +++ b/.gherkin-lintrc @@ -0,0 +1,32 @@ +{ + "file-name": [ + "on", + { + "style": "kebab-case" + } + ], + "indentation": [ + "on", + { + "Feature": 0, + "Background": 2, + "Scenario": 2, + "Examples": 4, + "Step": 4, + "given": 4, + "example": 6, + "and": 4 + } + ], + "no-dupe-feature-names": "on", + "no-dupe-scenario-names": "off", + "no-empty-file": "on", + "no-files-without-scenarios": "on", + "no-multiple-empty-lines": "off", + "no-partially-commented-tag-lines": "on", + "no-trailing-spaces": "off", + "no-unnamed-features": "on", + "no-unnamed-scenarios": "on", + "no-scenario-outlines-without-examples": "on", + "use-and": "on" +} diff --git a/.readme-partials/USING.md b/.readme-partials/USING.md index dfa697125..3a0d29ffa 100644 --- a/.readme-partials/USING.md +++ b/.readme-partials/USING.md @@ -90,6 +90,7 @@ You can use the following commands to control the tests: * `composer prepare-tests` - Set up the database that is needed for running the functional tests. This is only needed once. * `composer test` - Run all test suites. * `composer lint` - Run only the linting test suite. +* `composer lint-gherkin` - Run only the Gherkin linter over the feature files. * `composer phpcs` - Run only the code sniffer test suite. * `composer phpcbf` - Run only the code sniffer cleanup. * `composer phpunit` - Run only the unit test suite. @@ -119,6 +120,18 @@ composer behat -- --stop-on-failure composer behat-rerun ``` +### Linting the feature files + +`composer lint-gherkin` checks `features/` with +[gherkin-lint-plus](https://www.npmjs.com/package/gherkin-lint-plus), against the +`.gherkin-lintrc` ruleset shipped with this package. A project that needs +different rules can override it by committing its own `.gherkin-lintrc`. + +The linter is a Node package, so it is run through `npx` and needs Node.js 20 or +later. Where `npx` is not available the check reports that it is skipping, rather +than failing a suite that is otherwise entirely PHP. Pin a different release of +the linter with `WP_CLI_TEST_GHERKIN_LINT_VERSION`. + ### Controlling the amount of output Two environment variables make the test tools less chatty. Both are unset by default, which leaves the output exactly as it has always been. @@ -126,6 +139,8 @@ Two environment variables make the test tools less chatty. Both are unset by def - `NO_COLOR` (the [no-color.org](https://no-color.org/) convention) stops the runners from forcing ANSI color codes on, and leaves the decision to each tool's own terminal detection. Set this when capturing output to a file or a pipe, where the escape sequences are noise. - `WP_CLI_TEST_QUIET` switches the reporters to their most compact form: PHP_CodeSniffer reports one `file:line:col` line per violation with no progress ticker, PHPStan reports one `file:line:message` line per error with no progress bar and no result table, and Behat stops printing step definition snippets for undefined steps. +`NO_COLOR` also covers the Gherkin linter, which colors its report unconditionally and has no plain output format of its own. + ```bash NO_COLOR=1 WP_CLI_TEST_QUIET=1 composer phpstan ``` diff --git a/README.md b/README.md index 022bf2f7d..199c89200 100644 --- a/README.md +++ b/README.md @@ -101,6 +101,7 @@ You can use the following commands to control the tests: * `composer prepare-tests` - Set up the database that is needed for running the functional tests. This is only needed once. * `composer test` - Run all test suites. * `composer lint` - Run only the linting test suite. +* `composer lint-gherkin` - Run only the Gherkin linter over the feature files. * `composer phpcs` - Run only the code sniffer test suite. * `composer phpcbf` - Run only the code sniffer cleanup. * `composer phpunit` - Run only the unit test suite. @@ -130,6 +131,18 @@ composer behat -- --stop-on-failure composer behat-rerun ``` +### Linting the feature files + +`composer lint-gherkin` checks `features/` with +[gherkin-lint-plus](https://www.npmjs.com/package/gherkin-lint-plus), against the +`.gherkin-lintrc` ruleset shipped with this package. A project that needs +different rules can override it by committing its own `.gherkin-lintrc`. + +The linter is a Node package, so it is run through `npx` and needs Node.js 20 or +later. Where `npx` is not available the check reports that it is skipping, rather +than failing a suite that is otherwise entirely PHP. Pin a different release of +the linter with `WP_CLI_TEST_GHERKIN_LINT_VERSION`. + ### Controlling the amount of output Two environment variables make the test tools less chatty. Both are unset by default, which leaves the output exactly as it has always been. @@ -137,6 +150,8 @@ Two environment variables make the test tools less chatty. Both are unset by def - `NO_COLOR` (the [no-color.org](https://no-color.org/) convention) stops the runners from forcing ANSI color codes on, and leaves the decision to each tool's own terminal detection. Set this when capturing output to a file or a pipe, where the escape sequences are noise. - `WP_CLI_TEST_QUIET` switches the reporters to their most compact form: PHP_CodeSniffer reports one `file:line:col` line per violation with no progress ticker, PHPStan reports one `file:line:message` line per error with no progress bar and no result table, and Behat stops printing step definition snippets for undefined steps. +`NO_COLOR` also covers the Gherkin linter, which colors its report unconditionally and has no plain output format of its own. + ```bash NO_COLOR=1 WP_CLI_TEST_QUIET=1 composer phpstan ``` diff --git a/bin/run-gherkin-lint-tests b/bin/run-gherkin-lint-tests new file mode 100755 index 000000000..78fc82bd7 --- /dev/null +++ b/bin/run-gherkin-lint-tests @@ -0,0 +1,82 @@ +#!/bin/sh + +# Run the Gherkin linter only if there are feature files to lint. +if [ ! -d "features" ] +then + exit 0; +fi + +# The linter is a Node package, which a PHP project cannot assume is present. +# Skip rather than fail, the same way the Behat runner skips a package without a +# behat.yml. The check still runs in CI, where Node is always available. +if ! command -v npx > /dev/null 2>&1 +then + echo 'Did not detect the "npx" command, skipping the Gherkin linting.' + echo "It is part of Node.js 20 or later, see https://nodejs.org/ for installation instructions." + exit 0; +fi + +# To retrieve the WP-CLI tests package root folder, we start with this scripts +# location. +SOURCE="$0" + +# Resolve $SOURCE until the file is no longer a symlink. +while [ -h "$SOURCE" ]; do + DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )" + SOURCE="$( readlink "$SOURCE" )" + # If $SOURCE was a relative symlink, we need to resolve it relative to the + # path where the symlink file was located. + case $SOURCE in + /*) ;; + *) SOURCE="$DIR/$SOURCE";; + esac +done + +# Fetch the root folder of the WP-CLI tests package. +WP_CLI_TESTS_ROOT="$( cd -P "$( dirname "$SOURCE" )/.." && pwd )" + +# A project can override the shared ruleset by committing its own .gherkin-lintrc. +CONFIG_FILE=".gherkin-lintrc" +if [ ! -f "${CONFIG_FILE}" ] +then + CONFIG_FILE="${WP_CLI_TESTS_ROOT}/.gherkin-lintrc" +fi + +# Pinned so that a new release of the linter cannot turn a green build red +# without a commit in this repository. +GHERKIN_LINT_VERSION="${WP_CLI_TEST_GHERKIN_LINT_VERSION:-1.0.2}" + +# Without an explicit path the linter walks the whole working directory, vendor +# included. The feature files are the target. +if [ $# -eq 0 ] +then + set -- features +fi + +run_linter() { + npx --yes "gherkin-lint-plus@${GHERKIN_LINT_VERSION}" --config "${CONFIG_FILE}" "$@" +} + +if [ -n "${NO_COLOR}" ] +then + # The linter colors its report unconditionally: it honors neither NO_COLOR + # nor the absence of a terminal, and "stylish" is its only output format. + # It writes the report to STDERR, so strip the escape sequences from that + # stream while leaving STDOUT and the exit code alone. + STDERR_FILE=$( mktemp ) + + run_linter "$@" 2> "${STDERR_FILE}" + STATUS=$? + + if [ -s "${STDERR_FILE}" ] + then + ESC=$( printf '\033' ) + sed "s/${ESC}\[[0-9;]*m//g" "${STDERR_FILE}" >&2 + fi + + rm -f "${STDERR_FILE}" + + exit ${STATUS}; +fi + +run_linter "$@" diff --git a/composer.json b/composer.json index c32507878..c0d596827 100644 --- a/composer.json +++ b/composer.json @@ -79,6 +79,7 @@ "bin/install-package-tests", "bin/rerun-behat-tests", "bin/run-behat-tests", + "bin/run-gherkin-lint-tests", "bin/run-linter-tests", "bin/run-php-unit-tests", "bin/run-phpcs-tests", @@ -89,6 +90,7 @@ "behat": "run-behat-tests", "behat-rerun": "rerun-behat-tests", "lint": "run-linter-tests", + "lint-gherkin": "run-gherkin-lint-tests", "phpcs": "run-phpcs-tests", "phpcbf": "run-phpcbf-cleanup", "phpstan": "run-phpstan-tests", @@ -96,6 +98,7 @@ "prepare-tests": "install-package-tests", "test": [ "@lint", + "@lint-gherkin", "@phpcs", "@phpstan", "@phpunit", From 203f0caaf6bcf477ad49a78ee857f21fa4dc876e Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 16 Aug 2026 09:44:01 +0000 Subject: [PATCH 04/11] Move the Gherkin linter pin into package.json Drops the WP_CLI_TEST_GHERKIN_LINT_VERSION override, which was configuration nobody asked for, and puts the pinned version somewhere a dependency bot can see it. A version string inside a shell script is invisible to Dependabot; a devDependency in package.json is not. The package.json exists only to hold that pin: it is private, has no scripts, and nothing runs `npm install` against it. The runner reads the version out of it and fails loudly if it is missing, rather than quietly falling through to whatever the latest release happens to be. Note that picking these updates up needs an npm entry in the dependabot.yml that wp-cli/.github syncs out. Refs https://github.com/wp-cli/wp-cli/issues/6161 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01VcGmbu6CYzGGhP3jXuWDkJ --- .readme-partials/USING.md | 4 ++-- README.md | 4 ++-- bin/run-gherkin-lint-tests | 14 +++++++++++--- package.json | 9 +++++++++ 4 files changed, 24 insertions(+), 7 deletions(-) create mode 100644 package.json diff --git a/.readme-partials/USING.md b/.readme-partials/USING.md index 3a0d29ffa..91f6f67b6 100644 --- a/.readme-partials/USING.md +++ b/.readme-partials/USING.md @@ -129,8 +129,8 @@ different rules can override it by committing its own `.gherkin-lintrc`. The linter is a Node package, so it is run through `npx` and needs Node.js 20 or later. Where `npx` is not available the check reports that it is skipping, rather -than failing a suite that is otherwise entirely PHP. Pin a different release of -the linter with `WP_CLI_TEST_GHERKIN_LINT_VERSION`. +than failing a suite that is otherwise entirely PHP. Its version is pinned in +this package's `package.json`, which exists only to hold that pin. ### Controlling the amount of output diff --git a/README.md b/README.md index 199c89200..d871c2163 100644 --- a/README.md +++ b/README.md @@ -140,8 +140,8 @@ different rules can override it by committing its own `.gherkin-lintrc`. The linter is a Node package, so it is run through `npx` and needs Node.js 20 or later. Where `npx` is not available the check reports that it is skipping, rather -than failing a suite that is otherwise entirely PHP. Pin a different release of -the linter with `WP_CLI_TEST_GHERKIN_LINT_VERSION`. +than failing a suite that is otherwise entirely PHP. Its version is pinned in +this package's `package.json`, which exists only to hold that pin. ### Controlling the amount of output diff --git a/bin/run-gherkin-lint-tests b/bin/run-gherkin-lint-tests index 78fc82bd7..794eb5e38 100755 --- a/bin/run-gherkin-lint-tests +++ b/bin/run-gherkin-lint-tests @@ -42,9 +42,17 @@ then CONFIG_FILE="${WP_CLI_TESTS_ROOT}/.gherkin-lintrc" fi -# Pinned so that a new release of the linter cannot turn a green build red -# without a commit in this repository. -GHERKIN_LINT_VERSION="${WP_CLI_TEST_GHERKIN_LINT_VERSION:-1.0.2}" +# The linter is pinned so that one of its releases cannot turn a green build red +# without a commit here. The pin lives in package.json rather than in this file, +# because a version string inside a shell script is invisible to Dependabot. +PACKAGE_JSON="${WP_CLI_TESTS_ROOT}/package.json" +GHERKIN_LINT_VERSION=$( php -r '$package = json_decode( (string) file_get_contents( $argv[1] ), true ); echo isset( $package["devDependencies"]["gherkin-lint-plus"] ) ? $package["devDependencies"]["gherkin-lint-plus"] : "";' "${PACKAGE_JSON}" 2> /dev/null ) + +if [ -z "${GHERKIN_LINT_VERSION}" ] +then + echo "Could not read the pinned gherkin-lint-plus version from ${PACKAGE_JSON}." + exit 1; +fi # Without an explicit path the linter walks the whole working directory, vendor # included. The feature files are the target. diff --git a/package.json b/package.json new file mode 100644 index 000000000..c069a68bc --- /dev/null +++ b/package.json @@ -0,0 +1,9 @@ +{ + "name": "wp-cli-tests", + "description": "Pins the Node tooling used by the WP-CLI testing framework. Not published to npm.", + "private": true, + "license": "MIT", + "devDependencies": { + "gherkin-lint-plus": "1.0.2" + } +} From e844d3533a86e0a00763a74222cff08a9106ded0 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 16 Aug 2026 13:27:25 +0000 Subject: [PATCH 05/11] Address review feedback on the Behat runner Two points from review: * The wp-versions artifact already marks the current release with a "latest" status, so the separate request to api.wordpress.org was redundant. Both the "latest" and the X.Y resolution now come out of that one file, which means one cached artifact and at most one network request per run instead of two. * Behat's step definition snippets are not only printed when writing new step definitions; they are also how a typo in an existing step surfaces. That makes them a diagnostic rather than noise, and they only appear when something is already wrong, so suppressing them under WP_CLI_TEST_QUIET saved nothing in the passing case and cost information in the failing one. Dropped, which leaves WP_CLI_TEST_QUIET with no effect on Behat. Also adds lint-gherkin to the setup instructions, which listed the scripts a consuming package should wire up but not the new one. Refs https://github.com/wp-cli/wp-cli/issues/6161 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01VcGmbu6CYzGGhP3jXuWDkJ --- .readme-partials/USING.md | 15 +++-- README.md | 15 +++-- bin/run-behat-tests | 123 ++++++++++++++++++-------------------- 3 files changed, 76 insertions(+), 77 deletions(-) diff --git a/.readme-partials/USING.md b/.readme-partials/USING.md index 91f6f67b6..bd384b401 100644 --- a/.readme-partials/USING.md +++ b/.readme-partials/USING.md @@ -11,12 +11,14 @@ To make use of the WP-CLI testing framework, you need to complete the following "behat": "run-behat-tests", "behat-rerun": "rerun-behat-tests", "lint": "run-linter-tests", + "lint-gherkin": "run-gherkin-lint-tests", "phpcs": "run-phpcs-tests", "phpcbf": "run-phpcbf-cleanup", "phpunit": "run-php-unit-tests", "prepare-tests": "install-package-tests", "test": [ "@lint", + "@lint-gherkin", "@phpcs", "@phpunit", "@behat" @@ -137,7 +139,7 @@ this package's `package.json`, which exists only to hold that pin. Two environment variables make the test tools less chatty. Both are unset by default, which leaves the output exactly as it has always been. - `NO_COLOR` (the [no-color.org](https://no-color.org/) convention) stops the runners from forcing ANSI color codes on, and leaves the decision to each tool's own terminal detection. Set this when capturing output to a file or a pipe, where the escape sequences are noise. - - `WP_CLI_TEST_QUIET` switches the reporters to their most compact form: PHP_CodeSniffer reports one `file:line:col` line per violation with no progress ticker, PHPStan reports one `file:line:message` line per error with no progress bar and no result table, and Behat stops printing step definition snippets for undefined steps. + - `WP_CLI_TEST_QUIET` switches the reporters to their most compact form: PHP_CodeSniffer reports one `file:line:col` line per violation with no progress ticker, PHPStan reports one `file:line:message` line per error with no progress bar and no result table. Behat's own output is already minimal, so it is unaffected. `NO_COLOR` also covers the Gherkin linter, which colors its report unconditionally and has no plain output format of its own. @@ -167,11 +169,12 @@ Here's how to run your tests against the latest trunk version of WordPress: WP_VERSION=trunk composer behat ``` -Resolving `latest`, or a `X.Y` version without a patch number, needs a network -request. The answer is cached in the system temp directory for a day, so that -repeated runs do not repeat the lookup, and so that a run without connectivity -falls back to the last known answer. `WP_CLI_TEST_WP_VERSION_CACHE_TTL` sets the -lifetime of that cache in seconds; `0` looks the version up every time. +Resolving `latest`, or a `X.Y` version without a patch number, needs the +WordPress versions data, which is fetched once and cached in the system temp +directory for a day. Repeated runs do not repeat the request, and a run without +connectivity falls back to the last known copy. +`WP_CLI_TEST_WP_VERSION_CACHE_TTL` sets the lifetime of that cache in seconds; +`0` fetches it every time. #### WordPress Archive diff --git a/README.md b/README.md index d871c2163..be54b714d 100644 --- a/README.md +++ b/README.md @@ -22,12 +22,14 @@ To make use of the WP-CLI testing framework, you need to complete the following "behat": "run-behat-tests", "behat-rerun": "rerun-behat-tests", "lint": "run-linter-tests", + "lint-gherkin": "run-gherkin-lint-tests", "phpcs": "run-phpcs-tests", "phpcbf": "run-phpcbf-cleanup", "phpunit": "run-php-unit-tests", "prepare-tests": "install-package-tests", "test": [ "@lint", + "@lint-gherkin", "@phpcs", "@phpunit", "@behat" @@ -148,7 +150,7 @@ this package's `package.json`, which exists only to hold that pin. Two environment variables make the test tools less chatty. Both are unset by default, which leaves the output exactly as it has always been. - `NO_COLOR` (the [no-color.org](https://no-color.org/) convention) stops the runners from forcing ANSI color codes on, and leaves the decision to each tool's own terminal detection. Set this when capturing output to a file or a pipe, where the escape sequences are noise. - - `WP_CLI_TEST_QUIET` switches the reporters to their most compact form: PHP_CodeSniffer reports one `file:line:col` line per violation with no progress ticker, PHPStan reports one `file:line:message` line per error with no progress bar and no result table, and Behat stops printing step definition snippets for undefined steps. + - `WP_CLI_TEST_QUIET` switches the reporters to their most compact form: PHP_CodeSniffer reports one `file:line:col` line per violation with no progress ticker, PHPStan reports one `file:line:message` line per error with no progress bar and no result table. Behat's own output is already minimal, so it is unaffected. `NO_COLOR` also covers the Gherkin linter, which colors its report unconditionally and has no plain output format of its own. @@ -178,11 +180,12 @@ Here's how to run your tests against the latest trunk version of WordPress: WP_VERSION=trunk composer behat ``` -Resolving `latest`, or a `X.Y` version without a patch number, needs a network -request. The answer is cached in the system temp directory for a day, so that -repeated runs do not repeat the lookup, and so that a run without connectivity -falls back to the last known answer. `WP_CLI_TEST_WP_VERSION_CACHE_TTL` sets the -lifetime of that cache in seconds; `0` looks the version up every time. +Resolving `latest`, or a `X.Y` version without a patch number, needs the +WordPress versions data, which is fetched once and cached in the system temp +directory for a day. Repeated runs do not repeat the request, and a run without +connectivity falls back to the last known copy. +`WP_CLI_TEST_WP_VERSION_CACHE_TTL` sets the lifetime of that cache in seconds; +`0` fetches it every time. #### WordPress Archive diff --git a/bin/run-behat-tests b/bin/run-behat-tests index 068a283bd..ccf133106 100755 --- a/bin/run-behat-tests +++ b/bin/run-behat-tests @@ -97,89 +97,90 @@ if [ -n "${WP_CLI_TEST_CORE_ZIP-}" ] && [ -z "${WP_VERSION-}" ]; then export WP_VERSION=trunk fi -# Resolving WP_VERSION costs up to two network round trips on every invocation. -# Cache the answers, so that re-running a single scenario while iterating does -# not repeat them, and so that a run without connectivity can fall back to the -# last known answer instead of ending up with no version at all. +# Everything WP_VERSION resolution needs is in one file: the wp-versions artifact +# maps every WordPress release to its status, with the current one marked +# "latest". Cache it, so that re-running a single scenario while iterating does +# not refetch it every time, and so that a run without connectivity can fall back +# to the last known answer instead of ending up with no version at all. # # Set WP_CLI_TEST_WP_VERSION_CACHE_TTL to 0 to always refetch. -WP_VERSION_CACHE_DIR="${TMPDIR:-/tmp}/wp-cli-test-wp-version-cache" -WP_VERSION_CACHE_TTL="${WP_CLI_TEST_WP_VERSION_CACHE_TTL:-86400}" - -# Print a cache entry if it exists and is younger than the given number of -# seconds. A negative TTL accepts an entry of any age. -read_version_cache() { - local cache_file="${WP_VERSION_CACHE_DIR}/$1" - local ttl="$2" +WP_VERSIONS_URL="https://raw.githubusercontent.com/wp-cli/wp-cli-tests/artifacts/wp-versions.json" +WP_VERSIONS_CACHE_FILE="${TMPDIR:-/tmp}/wp-cli-test-wp-version-cache/wp-versions.json" +WP_VERSIONS_CACHE_TTL="${WP_CLI_TEST_WP_VERSION_CACHE_TTL:-86400}" + +# Print the cached versions file if it is younger than the given number of +# seconds. A negative TTL accepts it at any age. +read_versions_cache() { + local ttl="$1" local age - [ -s "${cache_file}" ] || return 1 + [ -s "${WP_VERSIONS_CACHE_FILE}" ] || return 1 if [ "${ttl}" -ge 0 ]; then # PHP rather than `find -newermt`, which is not portable across # GNU and BSD userlands. The Behat runner needs PHP anyway. - age=$(php -r 'echo time() - filemtime( $argv[1] );' "${cache_file}" 2>/dev/null) + age=$(php -r 'echo time() - filemtime( $argv[1] );' "${WP_VERSIONS_CACHE_FILE}" 2>/dev/null) case ${age} in ''|*[!0-9]*) return 1;; esac [ "${age}" -lt "${ttl}" ] || return 1 fi - cat "${cache_file}" + cat "${WP_VERSIONS_CACHE_FILE}" } -write_version_cache() { - mkdir -p "${WP_VERSION_CACHE_DIR}" 2>/dev/null || return 0 - printf '%s' "$2" > "${WP_VERSION_CACHE_DIR}/$1" 2>/dev/null || true -} - -# Turn WP_VERSION into an actual number to make sure our tags work correctly. -if [ "${WP_VERSION-latest}" = "latest" ]; then - LATEST_WP_VERSION=$(read_version_cache latest "${WP_VERSION_CACHE_TTL}") +# Print the WordPress versions data, from the cache where possible. Warnings go +# to STDERR so that they cannot end up inside the returned JSON. +get_wp_versions() { + local json - if [ -z "${LATEST_WP_VERSION}" ]; then - LATEST_WP_VERSION=$(curl -s https://api.wordpress.org/core/version-check/1.7/ | jq -r ".offers[0].current // empty") + json=$( read_versions_cache "${WP_VERSIONS_CACHE_TTL}" ) + if [ -n "${json}" ]; then + printf '%s' "${json}" + return 0 + fi - if [ -n "${LATEST_WP_VERSION}" ]; then - write_version_cache latest "${LATEST_WP_VERSION}" - else - # Prefer a stale answer over no answer. - LATEST_WP_VERSION=$(read_version_cache latest -1) + json=$( curl -s "${WP_VERSIONS_URL}" ) - if [ -n "${LATEST_WP_VERSION}" ]; then - echo "Warning: Could not reach api.wordpress.org, falling back to the cached latest WordPress version ${LATEST_WP_VERSION}." - else - echo "Warning: Could not determine the latest WordPress version. Version-specific tags will not be filtered." - fi - fi + # Only cache a well-formed response; an error page is not one. + if echo "${json}" | jq -e 'type == "object" and length > 0' > /dev/null 2>&1; then + mkdir -p "$( dirname "${WP_VERSIONS_CACHE_FILE}" )" 2>/dev/null \ + && printf '%s' "${json}" > "${WP_VERSIONS_CACHE_FILE}" 2>/dev/null || true + printf '%s' "${json}" + return 0 fi - export WP_VERSION="${LATEST_WP_VERSION}" -fi + # Prefer a stale answer over no answer. + json=$( read_versions_cache -1 ) + if [ -n "${json}" ]; then + echo "Warning: Could not fetch the WordPress versions data, falling back to the cached copy." >&2 + printf '%s' "${json}" + return 0 + fi -# Normalize WP_VERSION=X.Y.0 to X.Y (WordPress uses X.Y for the initial release, not X.Y.0). -# If WP_VERSION=X.Y (major.minor only), resolve to the latest available patch release. -if [[ "${WP_VERSION}" =~ ^([0-9]+\.[0-9]+)\.0$ ]]; then - export WP_VERSION="${BASH_REMATCH[1]}" -elif [[ "${WP_VERSION}" =~ ^[0-9]+\.[0-9]+$ ]]; then - WP_VERSIONS_JSON=$(read_version_cache wp-versions.json "${WP_VERSION_CACHE_TTL}") + return 1 +} - if [ -z "${WP_VERSIONS_JSON}" ]; then - WP_VERSIONS_JSON=$(curl -s https://raw.githubusercontent.com/wp-cli/wp-cli-tests/artifacts/wp-versions.json) +# Turn WP_VERSION into an actual number to make sure our tags work correctly. +if [ "${WP_VERSION-latest}" = "latest" ]; then + WP_VERSION=$( get_wp_versions | jq -r 'to_entries | map( select( .value == "latest" ) ) | last | .key // empty' ) - # Only cache a well-formed response; a proxy error page is not one. - if echo "${WP_VERSIONS_JSON}" | jq -e 'type == "object"' > /dev/null 2>&1; then - write_version_cache wp-versions.json "${WP_VERSIONS_JSON}" - else - WP_VERSIONS_JSON=$(read_version_cache wp-versions.json -1) - fi + if [ -z "${WP_VERSION}" ]; then + echo "Warning: Could not determine the latest WordPress version. Version-specific tags will not be filtered." fi - if [ -n "${WP_VERSIONS_JSON}" ]; then - RESOLVED_VERSION=$(echo "${WP_VERSIONS_JSON}" | jq -r --arg prefix "${WP_VERSION}." 'keys | map(select(startswith($prefix))) | sort_by(split(".") | map(tonumber)) | last // empty') - if [ -n "${RESOLVED_VERSION}" ]; then - export WP_VERSION="${RESOLVED_VERSION}" - fi + export WP_VERSION +# Normalize WP_VERSION=X.Y.0 to X.Y (WordPress uses X.Y for the initial release, +# not X.Y.0). This asks for that specific release, so it must not fall through to +# the patch resolution below. +elif [[ "${WP_VERSION}" =~ ^([0-9]+\.[0-9]+)\.0$ ]]; then + export WP_VERSION="${BASH_REMATCH[1]}" +# If WP_VERSION=X.Y (major.minor only), resolve to the latest available patch release. +elif [[ "${WP_VERSION}" =~ ^[0-9]+\.[0-9]+$ ]]; then + RESOLVED_VERSION=$( get_wp_versions | jq -r --arg prefix "${WP_VERSION}." 'keys | map( select( startswith( $prefix ) ) ) | sort_by( split(".") | map( tonumber ) ) | last // empty' ) + + if [ -n "${RESOLVED_VERSION}" ]; then + export WP_VERSION="${RESOLVED_VERSION}" fi fi @@ -213,14 +214,6 @@ if [ -n "${NO_COLOR}" ]; then BEHAT_EXTRA_ARGS+=('--no-colors') fi -# Drop the step definition snippets that Behat prints for undefined steps. They -# are long, and they are only actionable when you are writing new step -# definitions in this package. Useful for CI logs and for AI coding agents, -# which pay for every token of the report they read back. -if [ -n "${WP_CLI_TEST_QUIET}" ]; then - BEHAT_EXTRA_ARGS+=('--no-snippets') -fi - # Run the functional tests. FORMAT_ARGS=(--format progress) for arg in "$@"; do From 68fe99ff93f2005afd50ab5720888f7f415aee19 Mon Sep 17 00:00:00 2001 From: Pascal Birchler Date: Wed, 26 Aug 2026 14:05:29 +0000 Subject: [PATCH 06/11] Harden the WP_VERSION cache against typos, hangs and concurrent runs Three points from review, all on the caching added to the Behat runner: * A non-numeric WP_CLI_TEST_WP_VERSION_CACHE_TTL made `[ "${ttl}" -ge 0 ]` fail rather than evaluate, which skipped the age check entirely and left the cached copy valid forever. A typo therefore turned the cache into one that never refreshes. Validated up front now, with a warning and the default TTL. * The fetch was unbounded, so an unreachable or unresponsive host held up the whole run instead of falling back to the cached copy. Bounded with --connect-timeout and --max-time, after which the existing stale-cache path takes over. * The cache was written by truncating the target in place, so a concurrent runner could read it between the truncation and the end of the write and hand partial JSON to jq. It now goes to a temporary file in the same directory and is moved into place, which is atomic. The move keeps the file readable to others, as the shared cache directory needs and the old redirect did. Refs https://github.com/wp-cli/wp-cli/issues/6161 Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01W7iMRSfmFda1bzp87UcYoM --- bin/run-behat-tests | 41 ++++++++++++++++++++++++++++++++++++++--- 1 file changed, 38 insertions(+), 3 deletions(-) diff --git a/bin/run-behat-tests b/bin/run-behat-tests index ccf133106..18a6b726d 100755 --- a/bin/run-behat-tests +++ b/bin/run-behat-tests @@ -108,6 +108,14 @@ WP_VERSIONS_URL="https://raw.githubusercontent.com/wp-cli/wp-cli-tests/artifacts WP_VERSIONS_CACHE_FILE="${TMPDIR:-/tmp}/wp-cli-test-wp-version-cache/wp-versions.json" WP_VERSIONS_CACHE_TTL="${WP_CLI_TEST_WP_VERSION_CACHE_TTL:-86400}" +# A typo must not turn into a cache that never expires: the age comparison below +# errors out on a non-numeric TTL, which would then accept the cached copy at +# any age. +if ! is_numeric "${WP_VERSIONS_CACHE_TTL}"; then + echo "Warning: WP_CLI_TEST_WP_VERSION_CACHE_TTL is not a number of seconds, falling back to 86400." + WP_VERSIONS_CACHE_TTL=86400 +fi + # Print the cached versions file if it is younger than the given number of # seconds. A negative TTL accepts it at any age. read_versions_cache() { @@ -129,6 +137,32 @@ read_versions_cache() { cat "${WP_VERSIONS_CACHE_FILE}" } +# Store the versions data for the next run. The write goes through a temporary +# file in the same directory, so that a concurrent runner reading the cache sees +# either the previous copy or the new one, never a partial write. Caching is +# best effort throughout: a temp directory that cannot be written to is not a +# reason to fail the run. +write_versions_cache() { + local dir + local tmp + + dir=$( dirname "${WP_VERSIONS_CACHE_FILE}" ) + mkdir -p "${dir}" 2>/dev/null || return 0 + + tmp=$( mktemp "${dir}/wp-versions.XXXXXX" 2>/dev/null ) || return 0 + + if printf '%s' "$1" > "${tmp}" 2>/dev/null; then + # mktemp creates the file private to its owner; the cache directory is + # shared, and the data in it is public. + chmod 644 "${tmp}" 2>/dev/null + mv -f "${tmp}" "${WP_VERSIONS_CACHE_FILE}" 2>/dev/null || rm -f "${tmp}" + else + rm -f "${tmp}" + fi + + return 0 +} + # Print the WordPress versions data, from the cache where possible. Warnings go # to STDERR so that they cannot end up inside the returned JSON. get_wp_versions() { @@ -140,12 +174,13 @@ get_wp_versions() { return 0 fi - json=$( curl -s "${WP_VERSIONS_URL}" ) + # Bounded, so that an unreachable or unresponsive host falls back to the + # cached copy instead of holding up the run indefinitely. + json=$( curl -s --connect-timeout 10 --max-time 30 "${WP_VERSIONS_URL}" ) # Only cache a well-formed response; an error page is not one. if echo "${json}" | jq -e 'type == "object" and length > 0' > /dev/null 2>&1; then - mkdir -p "$( dirname "${WP_VERSIONS_CACHE_FILE}" )" 2>/dev/null \ - && printf '%s' "${json}" > "${WP_VERSIONS_CACHE_FILE}" 2>/dev/null || true + write_versions_cache "${json}" printf '%s' "${json}" return 0 fi From debfc3573218c93c5e68c645e07e398012866459 Mon Sep 17 00:00:00 2001 From: Pascal Birchler Date: Wed, 26 Aug 2026 15:56:54 +0000 Subject: [PATCH 07/11] Validate the cached WordPress versions data before using it The fetch path checked that a response was a non-empty object before caching it, but the read path only checked that the file was not empty. An unparsable cache was therefore handed straight to jq, which meant a run inside the TTL made no request at all, leaked a jq parse error, and continued with an empty WP_VERSION -- silently dropping the @require-wp-* filtering that the cache was added to protect. Every subsequent run repeated it until the entry aged out. Both paths now share one check, so an unusable cache counts as a miss and the network gets a chance to replace it. The TTL validation moves into get_wp_versions along with it, so that a typo is only reported when a lookup is actually going to happen rather than on every run, and its warning joins the other one on STDERR. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01GjJi6F76YZBt7cRrvXS6s6 --- bin/run-behat-tests | 32 ++++++++++++++++++++++---------- 1 file changed, 22 insertions(+), 10 deletions(-) diff --git a/bin/run-behat-tests b/bin/run-behat-tests index 18a6b726d..6a9d3d4e0 100755 --- a/bin/run-behat-tests +++ b/bin/run-behat-tests @@ -108,13 +108,11 @@ WP_VERSIONS_URL="https://raw.githubusercontent.com/wp-cli/wp-cli-tests/artifacts WP_VERSIONS_CACHE_FILE="${TMPDIR:-/tmp}/wp-cli-test-wp-version-cache/wp-versions.json" WP_VERSIONS_CACHE_TTL="${WP_CLI_TEST_WP_VERSION_CACHE_TTL:-86400}" -# A typo must not turn into a cache that never expires: the age comparison below -# errors out on a non-numeric TTL, which would then accept the cached copy at -# any age. -if ! is_numeric "${WP_VERSIONS_CACHE_TTL}"; then - echo "Warning: WP_CLI_TEST_WP_VERSION_CACHE_TTL is not a number of seconds, falling back to 86400." - WP_VERSIONS_CACHE_TTL=86400 -fi +# The versions data is a non-empty object mapping every release to its status. +# An error page, or a copy that was truncated on its way to disk, is not one. +is_valid_versions_json() { + jq -e 'type == "object" and length > 0' > /dev/null 2>&1 +} # Print the cached versions file if it is younger than the given number of # seconds. A negative TTL accepts it at any age. @@ -134,6 +132,11 @@ read_versions_cache() { [ "${age}" -lt "${ttl}" ] || return 1 fi + # Held to the same standard as a fetched copy, so that an unusable cache + # counts as a miss and the network gets a chance to replace it, instead of + # being served unchecked for the rest of its lifetime. + is_valid_versions_json < "${WP_VERSIONS_CACHE_FILE}" || return 1 + cat "${WP_VERSIONS_CACHE_FILE}" } @@ -167,8 +170,17 @@ write_versions_cache() { # to STDERR so that they cannot end up inside the returned JSON. get_wp_versions() { local json + local ttl="${WP_VERSIONS_CACHE_TTL}" + + # A typo must not turn into a cache that never expires: the age comparison + # errors out on a non-numeric TTL, which would then accept the cached copy + # at any age. + if ! is_numeric "${ttl}"; then + echo "Warning: WP_CLI_TEST_WP_VERSION_CACHE_TTL is not a number of seconds, falling back to 86400." >&2 + ttl=86400 + fi - json=$( read_versions_cache "${WP_VERSIONS_CACHE_TTL}" ) + json=$( read_versions_cache "${ttl}" ) if [ -n "${json}" ]; then printf '%s' "${json}" return 0 @@ -179,7 +191,7 @@ get_wp_versions() { json=$( curl -s --connect-timeout 10 --max-time 30 "${WP_VERSIONS_URL}" ) # Only cache a well-formed response; an error page is not one. - if echo "${json}" | jq -e 'type == "object" and length > 0' > /dev/null 2>&1; then + if printf '%s' "${json}" | is_valid_versions_json; then write_versions_cache "${json}" printf '%s' "${json}" return 0 @@ -201,7 +213,7 @@ if [ "${WP_VERSION-latest}" = "latest" ]; then WP_VERSION=$( get_wp_versions | jq -r 'to_entries | map( select( .value == "latest" ) ) | last | .key // empty' ) if [ -z "${WP_VERSION}" ]; then - echo "Warning: Could not determine the latest WordPress version. Version-specific tags will not be filtered." + echo "Warning: Could not determine the latest WordPress version. Version-specific tags will not be filtered." >&2 fi export WP_VERSION From a444f0d24bfcc8ef5a3b49e68739426f26d959c7 Mon Sep 17 00:00:00 2001 From: Pascal Birchler Date: Wed, 26 Aug 2026 15:57:33 +0000 Subject: [PATCH 08/11] Turn color off explicitly in the linter runner under NO_COLOR Dropping --colors left php-parallel-lint at its default autodetection, which only asks whether STDOUT is a terminal -- php-console-color has no NO_COLOR handling of its own. So NO_COLOR=1 composer lint still printed escape sequences interactively, while the other three runners disabled color outright. --no-colors has been available since v1.3, so this is the same one-flag treatment the PHPUnit, PHPStan and Behat runners already get. The README described the old behavior, which contradicted the no-color.org convention it cites; it now says what the runners do. The two bullets also pick up the list marker used everywhere else in the file. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01GjJi6F76YZBt7cRrvXS6s6 --- .readme-partials/USING.md | 4 ++-- README.md | 4 ++-- bin/run-linter-tests | 8 +++++--- 3 files changed, 9 insertions(+), 7 deletions(-) diff --git a/.readme-partials/USING.md b/.readme-partials/USING.md index 930ff8025..228a9440f 100644 --- a/.readme-partials/USING.md +++ b/.readme-partials/USING.md @@ -208,8 +208,8 @@ this package's `package.json`, which exists only to hold that pin. Two environment variables make the test tools less chatty. Both are unset by default, which leaves the output exactly as it has always been. - - `NO_COLOR` (the [no-color.org](https://no-color.org/) convention) stops the runners from forcing ANSI color codes on, and leaves the decision to each tool's own terminal detection. Set this when capturing output to a file or a pipe, where the escape sequences are noise. - - `WP_CLI_TEST_QUIET` switches the reporters to their most compact form: PHP_CodeSniffer reports one `file:line:col` line per violation with no progress ticker, PHPStan reports one `file:line:message` line per error with no progress bar and no result table. Behat's own output is already minimal, so it is unaffected. +* `NO_COLOR` (the [no-color.org](https://no-color.org/) convention) turns off the ANSI color codes in the output of every runner. Set this when capturing output to a file or a pipe, where the escape sequences are noise. +* `WP_CLI_TEST_QUIET` switches the reporters to their most compact form: PHP_CodeSniffer reports one `file:line:col` line per violation with no progress ticker, PHPStan reports one `file:line:message` line per error with no progress bar and no result table. Behat's own output is already minimal, so it is unaffected. `NO_COLOR` also covers the Gherkin linter, which colors its report unconditionally and has no plain output format of its own. diff --git a/README.md b/README.md index 62b8cbccd..4efaf1a65 100644 --- a/README.md +++ b/README.md @@ -219,8 +219,8 @@ this package's `package.json`, which exists only to hold that pin. Two environment variables make the test tools less chatty. Both are unset by default, which leaves the output exactly as it has always been. - - `NO_COLOR` (the [no-color.org](https://no-color.org/) convention) stops the runners from forcing ANSI color codes on, and leaves the decision to each tool's own terminal detection. Set this when capturing output to a file or a pipe, where the escape sequences are noise. - - `WP_CLI_TEST_QUIET` switches the reporters to their most compact form: PHP_CodeSniffer reports one `file:line:col` line per violation with no progress ticker, PHPStan reports one `file:line:message` line per error with no progress bar and no result table. Behat's own output is already minimal, so it is unaffected. +* `NO_COLOR` (the [no-color.org](https://no-color.org/) convention) turns off the ANSI color codes in the output of every runner. Set this when capturing output to a file or a pipe, where the escape sequences are noise. +* `WP_CLI_TEST_QUIET` switches the reporters to their most compact form: PHP_CodeSniffer reports one `file:line:col` line per violation with no progress ticker, PHPStan reports one `file:line:message` line per error with no progress bar and no result table. Behat's own output is already minimal, so it is unaffected. `NO_COLOR` also covers the Gherkin linter, which colors its report unconditionally and has no plain output format of its own. diff --git a/bin/run-linter-tests b/bin/run-linter-tests index d2c374ca7..e43cacfc3 100755 --- a/bin/run-linter-tests +++ b/bin/run-linter-tests @@ -1,10 +1,12 @@ #!/bin/sh -# Honor the NO_COLOR convention (https://no-color.org/) by leaving the color -# decision to the tool's own TTY detection instead of forcing it. +# Honor the NO_COLOR convention (https://no-color.org/). This has to turn the +# color off explicitly rather than omit the flag, because the tool's own +# detection only asks whether STDOUT is a terminal and knows nothing about +# NO_COLOR, so an interactive run would still be colored. COLOR_ARGS="--colors" if [ -n "${NO_COLOR}" ]; then - COLOR_ARGS="" + COLOR_ARGS="--no-colors" fi # shellcheck disable=SC2086 # Intentional word splitting of the optional arguments. From ccd58c5f41945eaa7fb666dec3c17a2b3d9c1841 Mon Sep 17 00:00:00 2001 From: Pascal Birchler Date: Wed, 26 Aug 2026 15:58:15 +0000 Subject: [PATCH 09/11] Let an explicit --report win over the quiet one in the PHPCS runner WP_CLI_TEST_QUIET is meant to be exported once and left set, so it has to stay out of the way of a report asked for on the command line. It did not: PHPCS gives no way for a later --report to replace an earlier one, so composer phpcs -- --report=summary printed both reports on 3.x and only the injected emacs one on 4.x. The compact report is now skipped when the caller names a report of their own. -q stays either way, since a later -v does override it. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01GjJi6F76YZBt7cRrvXS6s6 --- bin/run-phpcs-tests | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/bin/run-phpcs-tests b/bin/run-phpcs-tests index cff48a9bf..26a88ab75 100755 --- a/bin/run-phpcs-tests +++ b/bin/run-phpcs-tests @@ -9,7 +9,16 @@ then # Useful for CI logs and for AI coding agents, which pay for every token of # the report they read back. if [ -n "${WP_CLI_TEST_QUIET}" ]; then - EXTRA_ARGS="-q --report=emacs" + EXTRA_ARGS="-q" + + # A report asked for on the command line wins. PHPCS does not let a + # later --report replace an earlier one -- 3.x prints both reports and + # 4.x keeps only the first it saw -- so the compact one has to stay out + # of the way rather than rely on ordering. + case " $* " in + *" --report="*|*" --report-"*) ;; + *) EXTRA_ARGS="${EXTRA_ARGS} --report=emacs" ;; + esac fi # shellcheck disable=SC2086 # Intentional word splitting of the optional arguments. From cdb727515f74f4eee96975ae0c6ddd3a6ba00197 Mon Sep 17 00:00:00 2001 From: Pascal Birchler Date: Wed, 26 Aug 2026 15:59:02 +0000 Subject: [PATCH 10/11] Clean up the Gherkin runner's scratch file on interrupt The temporary file holding the linter's STDERR was only removed on the normal path, so a Ctrl-C during a lint left it behind. Use the same trap the PHPStan runner already uses for its own scratch directory. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01GjJi6F76YZBt7cRrvXS6s6 --- bin/run-gherkin-lint-tests | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/bin/run-gherkin-lint-tests b/bin/run-gherkin-lint-tests index 794eb5e38..89c82b15c 100755 --- a/bin/run-gherkin-lint-tests +++ b/bin/run-gherkin-lint-tests @@ -72,6 +72,7 @@ then # It writes the report to STDERR, so strip the escape sequences from that # stream while leaving STDOUT and the exit code alone. STDERR_FILE=$( mktemp ) + trap 'rm -f "${STDERR_FILE}"' EXIT HUP INT TERM run_linter "$@" 2> "${STDERR_FILE}" STATUS=$? @@ -82,8 +83,6 @@ then sed "s/${ESC}\[[0-9;]*m//g" "${STDERR_FILE}" >&2 fi - rm -f "${STDERR_FILE}" - exit ${STATUS}; fi From a88c6b2be71a8c8e9ca0af769e4057183f566f56 Mon Sep 17 00:00:00 2001 From: Pascal Birchler Date: Wed, 26 Aug 2026 17:58:52 +0000 Subject: [PATCH 11/11] Address review feedback on the quiet mode docs and the Gherkin runner Two findings from the review of the merge commit: WP_CLI_TEST_QUIET was described as switching "the reporters" to their most compact form, which overpromised. It reaches the analysis of the PHP files; the checks over the PHP blocks extracted from feature files keep their own reports, because both rewrite their findings back onto the feature file a block came from and a compact report is not what those rewrites are written against. The README now says so. A signal handler resumes where it left off rather than ending the script, so after an interrupt the Gherkin runner went on to read its report back out of the file the handler had just removed. The cleanup on EXIT stays; HUP, INT and TERM now clean up and stop, with the 128+n status a shell reports for a death by signal. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01GjJi6F76YZBt7cRrvXS6s6 --- .readme-partials/USING.md | 2 +- README.md | 2 +- bin/run-gherkin-lint-tests | 9 ++++++++- 3 files changed, 10 insertions(+), 3 deletions(-) diff --git a/.readme-partials/USING.md b/.readme-partials/USING.md index 0439ccbfd..94a96d637 100644 --- a/.readme-partials/USING.md +++ b/.readme-partials/USING.md @@ -261,7 +261,7 @@ this package's `package.json`, which exists only to hold that pin. Two environment variables make the test tools less chatty. Both are unset by default, which leaves the output exactly as it has always been. * `NO_COLOR` (the [no-color.org](https://no-color.org/) convention) turns off the ANSI color codes in the output of every runner. Set this when capturing output to a file or a pipe, where the escape sequences are noise. -* `WP_CLI_TEST_QUIET` switches the reporters to their most compact form: PHP_CodeSniffer reports one `file:line:col` line per violation with no progress ticker, PHPStan reports one `file:line:message` line per error with no progress bar and no result table. Behat's own output is already minimal, so it is unaffected. +* `WP_CLI_TEST_QUIET` switches the reporters to their most compact form: PHP_CodeSniffer reports one `file:line:col` line per violation with no progress ticker, PHPStan reports one `file:line:message` line per error with no progress bar and no result table. This covers the analysis of the PHP files themselves; the checks over the PHP blocks embedded in feature files keep their own reports, which are rewritten to point back at the feature file a block came from. Behat's own output is already minimal, so it is unaffected. `NO_COLOR` also covers the Gherkin linter, which colors its report unconditionally and has no plain output format of its own. diff --git a/README.md b/README.md index 4efaf1a65..b78b790be 100644 --- a/README.md +++ b/README.md @@ -220,7 +220,7 @@ this package's `package.json`, which exists only to hold that pin. Two environment variables make the test tools less chatty. Both are unset by default, which leaves the output exactly as it has always been. * `NO_COLOR` (the [no-color.org](https://no-color.org/) convention) turns off the ANSI color codes in the output of every runner. Set this when capturing output to a file or a pipe, where the escape sequences are noise. -* `WP_CLI_TEST_QUIET` switches the reporters to their most compact form: PHP_CodeSniffer reports one `file:line:col` line per violation with no progress ticker, PHPStan reports one `file:line:message` line per error with no progress bar and no result table. Behat's own output is already minimal, so it is unaffected. +* `WP_CLI_TEST_QUIET` switches the reporters to their most compact form: PHP_CodeSniffer reports one `file:line:col` line per violation with no progress ticker, PHPStan reports one `file:line:message` line per error with no progress bar and no result table. This covers the analysis of the PHP files themselves; the checks over the PHP blocks embedded in feature files keep their own reports, which are rewritten to point back at the feature file a block came from. Behat's own output is already minimal, so it is unaffected. `NO_COLOR` also covers the Gherkin linter, which colors its report unconditionally and has no plain output format of its own. diff --git a/bin/run-gherkin-lint-tests b/bin/run-gherkin-lint-tests index 89c82b15c..aa8339695 100755 --- a/bin/run-gherkin-lint-tests +++ b/bin/run-gherkin-lint-tests @@ -72,7 +72,14 @@ then # It writes the report to STDERR, so strip the escape sequences from that # stream while leaving STDOUT and the exit code alone. STDERR_FILE=$( mktemp ) - trap 'rm -f "${STDERR_FILE}"' EXIT HUP INT TERM + trap 'rm -f "${STDERR_FILE}"' EXIT + + # A signal handler resumes where it left off, so stopping here is what keeps + # the report below from being read back out of the file the handler has just + # removed. 128+n is the status a shell reports for a death by signal n. + trap 'rm -f "${STDERR_FILE}"; exit 129' HUP + trap 'rm -f "${STDERR_FILE}"; exit 130' INT + trap 'rm -f "${STDERR_FILE}"; exit 143' TERM run_linter "$@" 2> "${STDERR_FILE}" STATUS=$?