Skip to content

test(semantic-scholar): add hermetic test suite and register check in manifest - #14208

Open
1234adi1234 wants to merge 1 commit into
BasedHardware:mainfrom
1234adi1234:fix/semantic-scholar-app-hardening
Open

1234adi1234 wants to merge 1 commit into
BasedHardware:mainfrom
1234adi1234:fix/semantic-scholar-app-hardening

Conversation

@1234adi1234

@1234adi1234 1234adi1234 commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds automated hermetic regression testing for the Semantic Scholar integration app:

  1. Hermetic Test Suite: Authored plugins/omi-semantic-scholar-app/test_main.py runnable under standard library Python (python3 -S) with mock stubs for fastapi and httpx. Validates tool manifest schema definitions, author formatting, publication year formatting, DOI namespace normalization, paper searches, and details retrieval without external API calls.
  2. Preflight Manifest Registration: Registered semantic-scholar-app-tests in .github/checks-manifest.yaml.

Verification

  • Ran hermetic tests: python3 -S -m unittest plugins/omi-semantic-scholar-app/test_main.py (7 tests passed in 0.008s).
  • Ran preflight: bash scripts/pr-preflight --lane ci (passed 11/11 checks clean in 18.03s).

Review in cubic

@Git-on-my-level Git-on-my-level added python positive-signal Automation verified a genuine fix/quality contribution labels Sep 16, 2026
@Git-on-my-level

Copy link
Copy Markdown
Collaborator

Thanks @1234adi1234 — verified end to end.

What I checked:

  • plugins/omi-semantic-scholar-app/test_main.py — clean hermetic design: fastapi/httpx/models are stubbed via ModuleType and injected with patch.dict(sys.modules, ...) before exec_module, and every endpoint test patches api_get with AsyncMock, so no code path can reach the network. The assertions are real behavioral checks against main.py: format_year distinguishes int years from strings ("2024" -> "Unknown"), normalize_identifier maps doi:/DOI: prefixes to the canonical DOI: namespace the /paper/{id} endpoint expects, and format_authors caps the join at 6 names with an empty-list fallback to "Unknown".
  • The manifest assertions (type: object, properties, required present on all 3 tools) mirror the shape produced by manifest()'s /.well-known/omi-tools.json payload.
  • .github/checks-manifest.yamlsemantic-scholar-app-tests lands inside checks: with the same id/command/triggers/lanes/reason shape as the sibling entries, triggering on both plugins/omi-semantic-scholar-app/** and the manifest itself.
  • Executed the suite in an isolated, network-disabled container exactly as the manifest command runs it (python3 -S): 7/7 pass, confirming the pure-stdlib claim.

Two small non-blocking notes:

  • GetAuthorPapersRequest is stubbed in load_app() but get_author_papers has no endpoint test — the year/citationCount sort ordering and the 404 "Author not found." branch are uncovered. Natural follow-up.
  • The stubbed request models can't validate real pydantic constraints, so the manifest's 1-10 max_results range is asserted nowhere. Fine for a hermetic suite; just naming the gap.

One observation for maintainers: no GitHub-hosted required checks ran on this head, so the newly registered lane's CI behavior is verified here only via the local command, not on GitHub's runners.

Leaving formal sign-off to a human maintainer; no product-behavior concerns with this change.


by AI on behalf of David — if you need David’s attention urgently, please @Git-on-my-level and escalate with need human response.

@1234adi1234

Copy link
Copy Markdown
Contributor Author

@Git-on-my-level Thank you for the thorough review and verification! All checks are green. Whenever you have a moment, please feel free to merge. Much appreciated!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

positive-signal Automation verified a genuine fix/quality contribution python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants