Skip to content

Document developer vs independent test conventions - #674

Open
yihui wants to merge 1 commit into
mainfrom
doc-test-categories
Open

yihui wants to merge 1 commit into
mainfrom
doc-test-categories

Conversation

@yihui

@yihui yihui commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator

Adds a Test Categories section to the write-tests skill documenting the test-developer-* vs test-independent-* file-naming convention, which was previously implicit.

  • test-independent-* — validate outputs against an independent source of truth: a separate reference implementation (e.g. gsDesign::gsSurv(), gsDesign::toInteger()), a closed-form/hand-computed value, or a mathematical identity. Should not rely on gsDesign2 internals.
  • test-developer-* — exercise the package's own implementation behavior: default arguments, S3 class/attribute plumbing (e.g. uninteger_is_from), internal helpers (gsDesign2:::), consistency between exported and internal variants (e.g. gs_power_ahr() vs gs_power_ahr_()), and "does this feature survive this transformation?" checks.

Motivated by the review on #673, where a to_integer() harm-bound test was initially placed in test-independent-* but belonged in test-developer-*. Documenting the rule so future tests land in the right file.

Docs-only; no code or test changes.

🤖 Generated with Claude Code

Add a "Test Categories" section to the write-tests skill explaining the
test-developer-* vs test-independent-* file-naming convention:

- test-independent-* validate outputs against an independent source of truth
  (a separate reference implementation, closed-form value, or identity).
- test-developer-* exercise the package's own implementation behavior
  (defaults, S3 attributes, internal helpers, feature-survives-transform).

This convention was previously implicit; documenting it so new tests land in
the right file.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@jdblischak jdblischak left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yihui thanks for opening this Issue. I agree that these testing conventions should be clarified and documented, since I am not 100% sure how this is supposed to work.

Here is a what I think I know. As part of qualifying analytical software to be used in regulatory submissions, there can be a "double programming" step where an independent developer writes validation tests. In other words, the independent tester writes their own code to implement a particular method, and then tests to confirm that the package software returns the same result. These tests are in the test-independent-*.R files.

Here is the part I don't know. Can we as the developers of the software ever write "independent" tests? What if we write a test that uses a mathematical identity? Should these go in "independent", or should by definition any test written by a developer go in test-developer-*.R (since we can't be independent of ourselves, and any misunderstanding we have when writing a function will be carried over into our testing of the method, even if we use an orthogonal computation technique).

@yihui

yihui commented Sep 14, 2026

Copy link
Copy Markdown
Collaborator Author

@jdblischak Thanks for sharing your knowledge about these tests! I guess @LittleBeannie knows more than us and can help clarify.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants