feat(error-handler): add ErrorHandlerInterceptor plugin

.github/workflows/split-publish.yml

@@ -29,6 +29,7 @@ on:  # yamllint disable-line rule:truthy
       - 'convention-[0-9]*'
       - 'data-[0-9]*'
       - 'facade-[0-9]*'
+      - 'error-handler-[0-9]*'
       - 'filter-[0-9]*'
       - 'inline-[0-9]*'
       - 'lifecycle-[0-9]*'

composer.json

@@ -45,6 +45,7 @@
         "testo/codecov": "^0.2.1",
         "testo/convention": "^0.1.5",
         "testo/data": "^0.1.9",
+        "testo/error-handler": "^0.1",
         "testo/filter": "^0.1.7",
         "testo/inline": "^0.1.9",
         "testo/lifecycle": "^0.1.6",
@@ -101,6 +102,7 @@
             "Tests\\Convention\\": "plugin/convention/tests/",
             "Tests\\Data\\": "plugin/data/tests/",
             "Tests\\Facade\\": "plugin/facade/tests/",
+            "Tests\\ErrorHandler\\": "plugin/error-handler/tests/",
             "Tests\\Filter\\": "plugin/filter/tests/",
             "Tests\\Lifecycle\\": "plugin/lifecycle/tests/",
             "Tests\\Repeat\\": "plugin/repeat/tests/",
@@ -126,6 +128,7 @@
                     "testo/convention": "0.1.x-dev",
                     "testo/data": "0.1.x-dev",
                     "testo/facade": "0.1.x-dev",
+                    "testo/error-handler": "0.1.x-dev",
                     "testo/filter": "0.1.x-dev",
                     "testo/inline": "0.1.x-dev",
                     "testo/lifecycle": "0.1.x-dev",

docs/spec/skills-regrouping.md

@@ -0,0 +1,82 @@
+# Skills: group by direction, not by plugin
+
+## Problem
+
+`skills/` holds 14 skills; nine of them describe exactly one plugin (`testo-flaky-tests` is
+Retry + Repeat, `testo-coverage` is Codecov, `testo-inline-tests` is Inline, and so on). The agent
+picks a skill by *what it is doing*, not by *which package is installed*, so a per-plugin split
+makes it guess the package first. Every extra skill also adds a permanently loaded description to
+the agent's context, and every new plugin threatens to add one more.
+
+`testo-test-doubles` is the shape to copy: one skill for one direction ("isolate a collaborator"),
+a short `SKILL.md` that routes, and one file per approach under `references/` (Double, Mockery,
+hand-written). The plugin is a detail inside the direction.
+
+## Target layout
+
+| Direction | Skill | `SKILL.md` holds | `references/` |
+|---|---|---|---|
+| Run the suite | `testo-run-tests` | as is | |
+| Configure `testo.php` | `testo-configure` | as is, plus a table of optional plugins pointing at the skill that covers each | |
+| Write a test | `testo-write-tests` | `#[Test]`, `Assert`, `Expect`, skip/cancel, lifecycle hooks, groups | `data-driven.md` (DataSet, DataProvider, DataZip, DataCross), `inline-tests.md` (`#[TestInline]`) |
+| Isolate a collaborator | `testo-test-doubles` | as is | as is |
+| Keep tests reliable and clean | `testo-test-hygiene` (new) | the symptom table below, routing | `flaky.md` (Retry, Repeat), `php-errors.md` (error-handler plugin, `#[ExpectErrorHandlerChange]`), `leaks.md` (`Expect::leaks` / `notLeaks`, process-global state) |
+| Test async code | `testo-async` | as is (already two approaches in one skill) | |
+| Measure performance | `testo-benchmarks` | as is | |
+| Measure test quality | `testo-test-quality` (merge) | routing between the three | `coverage.md` (CodecovPlugin, levels, reports, `#[Covers]`), `increase-coverage.md` + its `scripts/`, `mutation-testing.md` + its `scripts/` |
+| Migrate from PHPUnit | `testo-migrate-from-phpunit` | as is | as is |
+| Extend Testo | `testo-plugin-author` | as is | as is |
+
+Fourteen skills become ten. Skills removed: `testo-data-driven`, `testo-inline-tests`,
+`testo-flaky-tests`, `testo-coverage`, `testo-increase-coverage`, `testo-mutation-testing`,
+`testo-php-errors` (folded into hygiene). `testo-async` and `testo-benchmarks` stay separate: each
+is a direction of its own, and merging them anywhere loses the trigger words.
+
+## Rules for a direction skill
+
+- **`SKILL.md` routes; `references/` explains.** The top file answers "which approach, and why"
+  in one table and states what every branch needs (fetch `llms.txt`, run from the project root).
+  Everything only one branch needs goes into that branch's reference file.
+- **Description carries every trigger the merged skills had.** Merging `testo-flaky-tests` into
+  hygiene must keep "flaky", "intermittent failure", "retry", "rerun" in the description, or the
+  agent stops reaching the material. One trigger per branch; drop synonyms.
+- **Quote the description** in the frontmatter whenever it contains `#[Attr]`: YAML cuts an
+  unquoted value at ` #`.
+- **A reference file is self-contained for its branch.** The agent loads one reference, not all
+  of them; a reference may not depend on a sibling being read first.
+- **Symptom-first routing.** The user names a symptom ("test passes but a warning scrolls by",
+  "green locally, red in CI"), not a plugin. The routing table maps symptoms to references.
+
+## `testo-test-hygiene` routing table (draft)
+
+| Symptom | Reference |
+|---|---|
+| Fails sometimes, passes on rerun; "verify the fix sticks" | `flaky.md` |
+| A PHP warning, notice or deprecation appears during a test and nothing fails; deprecations must fail the build; tested code installs its own error handler | `php-errors.md` |
+| Memory grows across tests; an object must not survive the test; a static or global keeps state | `leaks.md` |
+
+## `testo-test-quality` routing table (draft)
+
+| Intent | Reference |
+|---|---|
+| Turn coverage on, pick a level, produce a report, scope with `#[Covers]` | `coverage.md` |
+| Raise the number: find the least-covered files and write tests for them | `increase-coverage.md` |
+| Check whether the tests would notice a bug: Infection setup, MSI, killing mutants | `mutation-testing.md` |
+
+## Order of work
+
+1. `testo-test-hygiene`: create it from `testo-flaky-tests` and the standalone
+   `testo-php-errors` skill; write `leaks.md` from `Expect::leaks` in `plugin/assert`. Delete the
+   two source skills.
+2. `testo-write-tests`: move the data-driven and inline material into `references/`; keep a
+   one-line pointer per reference in the body where the topic would otherwise appear. Delete the
+   two source skills.
+3. `testo-test-quality`: merge the coverage trio; move each skill's `scripts/` under the merged
+   skill and fix the relative paths the references cite.
+4. `testo-configure`: add the optional-plugin table.
+5. `skills/README.md`: rewrite the table to the ten skills. `AGENTS.md` names
+   `skills/README.md` as the index, so nothing else needs to change.
+
+Each step is one commit and leaves every skill in the table loadable on its own. After each step
+load every touched skill once (`composer install` re-syncs `.agents/skills` through `llm/skills`)
+to confirm its frontmatter still parses and its `references/` paths resolve.

plugin/error-handler/composer.json

@@ -0,0 +1,39 @@
+{
+    "name": "testo/error-handler",
+    "description": "Error handler interceptor plugin for the Testo testing framework.",
+    "license": "BSD-3-Clause",
+    "type": "library",
+    "keywords": [
+        "testo",
+        "error-handler",
+        "test"
+    ],
+    "authors": [
+        {
+            "name": "Aleksei Gagarin (roxblnfk)",
+            "homepage": "https://github.com/roxblnfk"
+        }
+    ],
+    "funding": [
+        {
+            "type": "boosty",
+            "url": "https://boosty.to/roxblnfk"
+        }
+    ],
+    "require": {
+        "php": ">=8.2",
+        "testo/testo": "0.10.34 - 1"
+    },
+    "autoload": {
+        "psr-4": {
+            "Testo\\ErrorHandler\\": "src/"
+        }
+    },
+    "minimum-stability": "dev",
+    "prefer-stable": true,
+    "extra": {
+        "branch-alias": {
+            "dev-1.x": "1.x-dev"
+        }
+    }
+}

plugin/error-handler/src/CapturedError.php

@@ -0,0 +1,47 @@
+<?php
+
+declare(strict_types=1);
+
+namespace Testo\ErrorHandler;
+
+/**
+ * A single PHP error captured during test execution.
+ *
+ * @api
+ */
+final readonly class CapturedError implements \Stringable
+{
+    /**
+     * @param bool $handled The handler installed before the test took the error (returned true), so
+     *        PHP would not have printed it.
+     */
+    public function __construct(
+        public int $severity,
+        public string $message,
+        public string $file,
+        public int $line,
+        public bool $handled = false,
+    ) {}
+
+    /**
+     * The error as PHP itself prints it.
+     *
+     * @return non-empty-string
+     */
+    #[\Override]
+    public function __toString(): string
+    {
+        return \sprintf('%s: %s in %s on line %d', self::label($this->severity), $this->message, $this->file, $this->line);
+    }
+
+    private static function label(int $severity): string
+    {
+        return match ($severity) {
+            \E_WARNING, \E_USER_WARNING, \E_CORE_WARNING, \E_COMPILE_WARNING => 'Warning',
+            \E_NOTICE, \E_USER_NOTICE => 'Notice',
+            \E_DEPRECATED, \E_USER_DEPRECATED => 'Deprecated',
+            \E_RECOVERABLE_ERROR => 'Recoverable fatal error',
+            default => 'Error',
+        };
+    }
+}

plugin/error-handler/src/CapturedErrors.php

@@ -0,0 +1,28 @@
+<?php
+
+declare(strict_types=1);
+
+namespace Testo\ErrorHandler;
+
+/**
+ * Collection of PHP errors accumulated during test execution.
+ *
+ * Stored as a {@see \Testo\Core\Context\TestResult} attribute under the key {@see CapturedErrors::class}.
+ * Renderers that wish to display collected errors should retrieve it from the result.
+ *
+ * @api
+ */
+final readonly class CapturedErrors
+{
+    /**
+     * @param list<CapturedError> $errors
+     */
+    public function __construct(
+        public array $errors,
+    ) {}
+
+    public function isEmpty(): bool
+    {
+        return $this->errors === [];
+    }
+}

plugin/error-handler/src/ErrorHandlerPlugin.php

@@ -0,0 +1,37 @@
+<?php
+
+declare(strict_types=1);
+
+namespace Testo\ErrorHandler;
+
+use Internal\Container\Container;
+use Testo\Common\PluginConfigurator;
+use Testo\ErrorHandler\Internal\ErrorHandlerInterceptor;
+use Testo\Pipeline\InterceptorCollector;
+
+/**
+ * Plugin that captures PHP errors raised during test execution.
+ *
+ * By default errors are collected and stored as a {@see CapturedErrors} attribute
+ * on the {@see \Testo\Core\Context\TestResult}, but the test still passes. Pass
+ * {@see $failOnError}: true to make any captured error fail the test instead.
+ *
+ * @api
+ */
+final readonly class ErrorHandlerPlugin implements PluginConfigurator
+{
+    /**
+     * @param bool $failOnError When true, a captured PHP error fails the test. When false
+     *        (default) errors are collected but the test result is unchanged.
+     */
+    public function __construct(
+        private bool $failOnError = false,
+    ) {}
+
+    #[\Override]
+    public function configure(Container $container): void
+    {
+        $container->get(InterceptorCollector::class)
+            ->addInterceptor(new ErrorHandlerInterceptor($this->failOnError));
+    }
+}

plugin/error-handler/src/Exception/ErrorHandlerUnchanged.php

@@ -0,0 +1,20 @@
+<?php
+
+declare(strict_types=1);
+
+namespace Testo\ErrorHandler\Exception;
+
+use Testo\ErrorHandler\ExpectErrorHandlerChange;
+
+/**
+ * A test declares {@see ExpectErrorHandlerChange} but left the error-handler stack as it found it.
+ *
+ * @api
+ */
+final class ErrorHandlerUnchanged extends \LogicException
+{
+    public function __construct()
+    {
+        parent::__construct('The test declares #[ExpectErrorHandlerChange] but left the error-handler stack unchanged.');
+    }
+}

plugin/error-handler/src/ExpectErrorHandlerChange.php

@@ -0,0 +1,26 @@
+<?php
+
+declare(strict_types=1);
+
+namespace Testo\ErrorHandler;
+
+use Testo\Core\Value\Status;
+
+/**
+ * Declares that a test intentionally changes the PHP error-handler stack and leaves the change in
+ * place: it installs a handler it does not remove, or removes one it did not install.
+ *
+ * The attribute is a two-way contract checked at the end of the run:
+ *
+ * - a test marked with it that changes the stack stays {@see Status::Passed} (without it such a test
+ *   is reported as {@see Status::Risky}, as its handler would shadow the ones of the tests after it);
+ * - a test marked with it that leaves the stack untouched is {@see Status::Failed} with
+ *   {@see Exception\ErrorHandlerUnchanged}, because the declaration no longer holds.
+ *
+ * The stack is restored to its pre-test state afterwards in either case. On a class the attribute
+ * applies to every test of the class.
+ *
+ * @api
+ */
+#[\Attribute(\Attribute::TARGET_CLASS | \Attribute::TARGET_METHOD | \Attribute::TARGET_FUNCTION)]
+final readonly class ExpectErrorHandlerChange {}