Skip to content

Grammar can't express key-press, pixel/color, process-control, or checked-state assertions #17

Description

@queso

Summary

FlowSpec's step/assertion grammar (src/types.ts: actions visit | click | fill | select | wait_for, assertions url | visible | matches | not_visible) can't express several real, common bug classes. Hit this today while using FlowSpec as the QA driver for an A(i)-Team mission — two real bugs escaped the whole test pipeline and were only caught by a human-style manual browser walk, and neither could be turned into a regression spec afterward because the grammar has no way to say what actually needs asserting.

Gaps found

  1. No key-press step. There's click, but no way to simulate pressing a specific key (Enter, Escape, Tab, etc.) on a focused element. Blocked writing specs for: submit-on-Enter in a text input, abandon-edit-on-Escape. Any keyboard-only interaction is currently unspecifiable.

  2. No color/pixel (visual) assertion. visible/not_visible/matches all operate on DOM presence/text — none can assert on computed style or rendered appearance. This mattered concretely: a checkbox rendered browser-default blue/gray pixels (native UA widget styling) despite every CSS computed-style check reporting a clean design-system-compliant black-and-white result. getComputedStyle doesn't see UA widget rendering; only an actual pixel/screenshot-level check does. No way to pin that as a regression spec today.

  3. No process-control step. Can't stop/start/restart a dependent process (e.g. the backing API) mid-flow. This blocked specifying an "app recovers correctly after the API comes back up" scenario — a real state-management bug (hasLoaded never reset after failure) was only reproducible by killing and restarting the API server by hand.

  4. No checked (or general form-control-state) predicate. Toggling a checkbox can be driven with click, but there's no assertion for its resulting checked state — only text-based assertions exist, so toggle correctness can't actually be verified, only clicked.

Impact

Because these can't be expressed, the right call when a fix lands for a bug in one of these classes is to not graduate a spec for it (writing a spec that can't actually fail is worse than an acknowledged gap) — which means FlowSpec currently can't build a regression net against exactly the bug classes that are hardest to catch by other means (CSS-computed-style checks + unit tests both miss all four of the above).

Possibly useful additions

  • press: <key> step (or press: <key> as a modifier alongside fill)
  • A computed_style or pixel_match assertion (even a coarse one — e.g. assert a named CSS custom property or computed color value at a selector)
  • A run: <command> / stop_process / start_process step pair, or at minimum a documented pattern for driving process lifecycle from a flow
  • A checked (and maybe generic state: <attr>=<value>) assertion for form controls

Happy to share the full evidence bundle (screenshots + repro steps) from the mission that surfaced this if useful — everything above was found by an agent (Frankie, from the A(i)-Team plugin) doing a first-time-user browser walk against a React todo app, not synthetically constructed.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions