Skip to content

Decouple test suite from compiler internals for pure blackbox testing #27666

Description

@sbc100

Most tests in test/test_core.py and test/test_browser.py test emcc as a black box via CLI invocations. However, parts of test/test_other.py, test/test_sanity.py, and the test harness (test/common.py) directly import and depend on Python compiler internals (tools/*).

If we want to rewrite the compiler (or components like the linker) in another language, or run the test suite against alternative implementations, we need to eliminate these internal dependencies and make the suite purely blackbox.


Breakdown & Estimated Effort

Area Current Coupling Proposed Fix Effort
1. Global State Mutation Tests for -print-* flags in test_other.py mutate options.lto and settings.MEMORY64 to predict paths via cache.get_lib_dir(). Assert on CLI output paths directly without altering Python singletons. Small (~0.5 day)
2. Binary Type Checks test_other.py calls building.is_wasm(), is_wasm_dylib(), is_ar(). Inspect magic bytes (b'\0asm') or use llvm-readobj / llvm-ar. Small (~1 day)
3. Direct Unit Tests test_other.py / test_browser.py unit-test Python functions directly (binary_encode, wasm_sourcemap, response_file, closure_compiler). Test via CLI flags or isolate into a dedicated Python unit test directory. Small (~1–2 days)
4. Wasm Inspection ~15 tests in test_core.py and test_other.py use tools.webassembly.Module to check section names/features. Replace with standard CLI tools (llvm-readobj, llvm-objdump -h, wasm-dis). Medium (~2–3 days)
5. Cache & Ports Introspection Tests query cache.get_sysroot_dir() or ports.Ports.get_dir() to find files in the cache. Expose paths via CLI (e.g. emcc --print-sysroot) and test ports via --use-port. Medium (~2 days)
6. Harness Settings Filtering common.py imports COMPILE_TIME_SETTINGS to filter -s flags during compile-only (-c) runs. Make compiler ignore/warn on link settings during -c, or query capabilities via CLI. Medium (~2–3 days)
7. Sanity Suite Partitioning test_sanity.py (~1,000 lines) tests the Python toolchain driver (config.py, cache locking, version checks). Separate Python driver tests from the language-agnostic conformance test suite. Medium (~1–2 days)

Total Estimated Effort: ~2 to 3 weeks


Next Steps

  1. Clean up quick wins in test_other.py (remove options/settings mutation, replace building.is_* helpers).
  2. Replace tools.webassembly calls with standard binary tools (llvm-readobj).
  3. Decouple harness dependencies (COMPILE_TIME_SETTINGS) and separate test_sanity.py.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions