Formalize real-PLC acceptance testing with Gherkin scenarios

[gijzelaerr]

Summary

Formalize volunteer testing against real Siemens PLCs as a repeatable acceptance-test program based on Cucumber/Gherkin scenarios, with results recorded as structured GitHub issues.

We already have real-hardware pytest coverage in tests/test_client_e2e.py (legacy S7) and tests/test_s7_e2e.py (S7CommPlus). The missing part is a stable, documented procedure and a queryable record of exactly which source revision was tested, on which PLC/firmware/configuration, by whom, when, and with what result.

This complements the protocol conformance suite from #620: conformance tests validate packets without hardware; this suite validates user-visible behaviour against real PLCs. It should also replace one-off reporting such as #793 with a reusable format.

Goals

• Give volunteers one safe command and a small set of understandable scenarios.
• Reuse the existing pytest e2e implementation instead of maintaining a parallel test suite.
• Cover both legacy S7 and S7CommPlus, while allowing protocol-specific scenarios.
• Preserve enough environment and PLC metadata to reproduce failures.
• Make tested combinations and gaps discoverable through GitHub search.
• Produce useful diagnostics when a scenario fails without publishing PLC credentials, certificates, IP addresses, or proprietary programs.

Proposed implementation

1. Gherkin is the specification; pytest remains the runner

Add version-controlled .feature files under tests/features/real_plc/. Use pytest-bdd as the preferred Cucumber-compatible adapter because the project already uses pytest fixtures, markers, command-line options, JUnit output, and test helpers. Validate that choice with a small spike before converting all e2e coverage.

Step definitions should call shared helpers extracted from the current e2e tests. They must not duplicate protocol logic or invoke one test from another. Keep lower-level diagnostic tests in pytest where Gherkin would add no clarity.

Suggested tags:

• @smoke: read-only, safe to run on any correctly prepared test PLC
• @write: modifies only the dedicated scratch DB and always restores original bytes
• @legacy_s7: snap7.Client
• @s7commplus: sync S7CommPlus client
• @async: async client parity
• @tls, @symbolic, @routing, @logo: capability-specific scenarios
• @administrative: potentially disruptive operations; never part of the default run

2. Initial scenarios

Feature: Connect to and identify a real PLC

  Background:
    Given I am using a dedicated test PLC
    And the test configuration contains no secrets in reportable fields

  @smoke
  Scenario Outline: Establish and close a session
    Given the client uses the "<protocol>" protocol path
    When I connect to the configured PLC
    Then the client reports that it is connected
    And the negotiated protocol and security mode are recorded
    And the PLC order code, CPU family, firmware and CPU state are recorded when available
    When I disconnect
    Then the client reports that it is disconnected

    Examples:
      | protocol   |
      | legacy_s7 |
      | s7commplus |

Feature: Read a known test data block

  @smoke
  Scenario: Read the canonical fixture
    Given the read-only test DB has the documented canonical layout
    When I read the complete fixture DB
    Then INT, REAL, BYTE, WORD, DWORD, DINT, CHAR and BOOL values match the fixture
    And individual reads return the same values as the complete-block read

  @smoke
  Scenario: Read multiple values in one request
    Given the read-only test DB has the documented canonical layout
    When I read values of different sizes in one multi-variable request
    Then every value and result code matches the fixture

Feature: Write safely to a scratch data block

  @write
  Scenario Outline: Round-trip a value and restore the PLC
    Given writing has been explicitly enabled
    And I have a dedicated scratch DB
    And the original bytes at the target address have been saved
    When I write a valid <type> value
    Then reading the address returns the written value
    And the original bytes are restored even if the scenario fails

    Examples:
      | type  |
      | INT   |
      | REAL  |
      | BYTE  |
      | WORD  |
      | DWORD |
      | DINT  |
      | CHAR  |
      | BOOL  |

Feature: Recover from normal connection lifecycle events

  @smoke
  Scenario: Reconnect after a clean disconnect
    Given I connected to and disconnected from the PLC
    When I reconnect with the same configuration
    Then a known read succeeds

  @smoke
  Scenario: Repeated operations do not corrupt the session
    Given I am connected to the PLC
    When I read the canonical fixture repeatedly
    Then every read succeeds with the expected value
    And disconnect completes cleanly

Protocol-specific follow-ups should cover S7CommPlus authentication/TLS negotiation, symbolic browse/read/write, optimized versus non-optimized DB behaviour, legacy I/Q/M access, routing, LOGO, and sync/async parity. Those scenarios should be skipped with a machine-readable reason when the PLC does not advertise or is not configured for the capability; unsupported is not the same as failed.

3. Test tiers and safety

The default command runs only @smoke scenarios and is read-only. Write scenarios require both an explicit flag (for example --allow-plc-write) and a configured scratch DB. Before each write, save the affected bytes; restore them in fixture teardown/finally logic and verify restoration.

PLC start/stop, clock changes, block upload/download, protection changes, and other operational actions belong in @administrative. They require a separate explicit opt-in and a dedicated non-production PLC. They must never run as part of the normal volunteer procedure.

Provide a canonical TIA Portal fixture project or precise setup document for the existing 37-byte DB layout. Document optimized-block requirements, PUT/GET requirements, rack/slot defaults, and least-privilege/security prerequisites. Never ask testers to weaken a PLC that is in service.

4. Reproducible report

Each run should emit:

• a human-readable terminal summary;
• JUnit XML for scenario-level pass/fail/skip results;
• a small JSON report with a versioned schema.

The JSON report should include:

• test-suite schema version, UTC start/end time, and overall result;
• exact git commit SHA and whether the worktree was dirty (or exact installed package version and source);
• Python version, OS, architecture, and relevant optional dependencies;
• tester's GitHub handle (supplied explicitly or taken from the report issue author);
• PLC family/model/order code, firmware, rack/slot, protocol path/version, TLS/security mode, and relevant TIA configuration;
• scenario IDs/tags and pass/fail/skip status;
• sanitized exception, traceback and logs for failures.

Do not record PLC IPs, passwords, private keys, certificate private material, plant names, or other network/site identifiers. Packet captures should be optional, manually reviewed, sanitized, and attached only when needed.

5. GitHub-native tracking

Add an issue form such as .github/ISSUE_TEMPLATE/plc-test-result.yml for one issue per tester + PLC configuration + source revision + run. The form should collect the report fields above and require the generated summary/JSON attachment. The GitHub issue author and creation timestamp provide an auditable "who" and "when"; the explicit UTC run time covers delayed submissions.

Use a small, stable label set rather than labels for every firmware/Python/OS combination:

• hardware-test
• test-result: pass, test-result: fail, or test-result: partial
• protocol: legacy-s7 or protocol: s7commplus

Keep high-cardinality data (exact PLC model, firmware, Python version, OS, commit SHA) in the structured report so it can be parsed and searched without label sprawl.

Add a saved issue-search link to the documentation for each useful view, for example open failures, S7CommPlus runs, and results for a release/commit. Use this issue as the implementation tracker; create/link individual result issues rather than accumulating free-form comments here. GitHub Discussions can coordinate volunteers, but test evidence belongs in issues so it remains structured and queryable. Do not require GitHub Projects, since Projects are currently disabled for this repository.

Optionally add a GitHub Action triggered for hardware-test issues to validate the JSON schema and post a normalized summary. It must not treat self-reported hardware results as a required PR check. A later action can generate a compatibility matrix from valid result issues; the issues remain the source of truth.

Coverage strategy

Track coverage across these dimensions:

1. Client/protocol: legacy S7, S7CommPlus sync, S7CommPlus async, LOGO, routing where available.
2. PLC family/model: S7-200, S7-300, S7-400, S7-1200, S7-1500, LOGO, with exact order code.
3. Firmware and security mode: firmware version, PUT/GET, optimized blocks, access protection, TLS/authentication mode.
4. Host: Python 3.10-3.14, Windows/Linux/macOS, architecture.
5. Source: release version plus immutable commit SHA.

Prioritize PLC model/firmware/protocol coverage over exhaustively multiplying host OS and Python versions; CI already covers the host matrix against emulated/local tests. For a release candidate, define a small required hardware baseline and record gaps explicitly. A result becomes stale when the tested code, scenario schema, relevant protocol implementation, or PLC firmware/configuration changes—not merely because time passed.

Deliverables / acceptance criteria

• Decide/prototype the Cucumber adapter; prefer pytest-bdd unless the spike identifies a concrete blocker.
• Add versioned Gherkin feature files for connection, fixture reads, multi-read, safe write/read/restore, and reconnect lifecycle.
• Refactor the existing e2e tests so Gherkin steps and diagnostic pytest tests share fixtures/helpers without duplicated protocol logic.
• Add a read-only default command and explicit write/admin opt-ins.
• Guarantee and test scratch-data restoration on success, assertion failure, exception, and interruption where process cleanup is possible.
• Add canonical PLC fixture setup and volunteer runbook documentation.
• Generate JUnit XML plus schema-versioned JSON containing source, environment, PLC metadata, and scenario results.
• Add the plc-test-result GitHub issue form and the small stable label set.
• Document saved searches and the policy for pass/fail/partial, skip reasons, reruns, superseded results, and stale results.
• Validate the procedure with at least one legacy S7 PLC and one S7CommPlus-capable S7-1200/1500, filing result issues for both.
• Keep real-PLC tests opt-in and out of normal hosted CI; do not add an internet-exposed or untrusted self-hosted PLC runner as part of this issue.

Open questions

• Should the canonical fixture be distributed as a small TIA Portal project, a source export, or both?
• Which exact PLC/firmware combinations form the minimum release-candidate baseline?
• Should a generated compatibility matrix live in the documentation, a release checklist issue, or both?