refactor: extract validation block and remove dead complex-data guard in transformers - #194
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
Repository policy requires an ## [Unreleased] CHANGELOG.md entry for PRs touching philanthropy/, and the updated complex-data behavior wording in transformer docstrings should be corrected to match runtime behavior.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR refactors philanthropy.preprocessing._transformers by extracting the repeated validate_data() try/except (with object-dtype fallback) into a shared _validate_X helper, and removes the redundant np.iscomplexobj complex-data guard that was effectively dead under the current validation behavior.
Changes:
- Added
_validate_X()to centralize thevalidate_data(..., dtype=None)+ object-cast retry logic. - Replaced duplicated inline validation blocks in
CRMCleanerandFiscalYearTransformerwith calls to_validate_X(). - Removed the trailing
np.iscomplexobj(...)checks and theirValueError("Complex data not supported")raises.
File summaries
| File | Description |
|---|---|
philanthropy/preprocessing/_transformers.py |
DRYs up validation logic via _validate_X and removes the redundant complex-object guard in both transformers. |
Review details
Suppressed comments (3)
philanthropy/preprocessing/_transformers.py:198
- This docstring claims complex data always raises, but complex values can survive validation when they arrive inside object-dtype columns (and are handled downstream). Update the wording to reflect that the ValueError is for inputs that fail validation (e.g., complex-dtype arrays).
If ``X`` contains complex data.
philanthropy/preprocessing/_transformers.py:306
- This docstring says transform raises whenever X contains complex data, but in practice the ValueError comes from sklearn validation for complex-dtype inputs; complex values inside object-dtype columns can flow through and get coerced. Updating the wording avoids overstating the contract.
If ``X`` contains complex data.
philanthropy/preprocessing/_transformers.py:277
- The docstring currently states that any complex data in X raises, but the validation logic only reliably raises for complex-dtype inputs; complex values inside object-typed columns may pass and be coerced later. Consider describing the error condition as 'fails validation' to match runtime behavior.
If ``fiscal_year_start`` is invalid or ``X`` contains complex data.
- Files reviewed: 1/1 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| @@ -161,19 +172,7 @@ def fit(self: _SelfC, X: Any, y: Any = None) -> _SelfC: | |||
| If ``fiscal_year_start`` is invalid or ``X`` contains complex data. | |||
|
Read the diff rather than merging on green, since this one touches validation shared by two transformers. It holds up. The reachability finding is right, and it is right for two independent reasons rather than one. A genuine complex dtype never reaches Checked that the deletion is not removing behaviour the suite was relying on: all four entry points still have a complex-rejection test ( Two small things, neither worth another round trip:
Thanks. Four copies became one and the dead line that made the coverage report lie is gone. |
Cut 0.7.1 so the notebooks stop depending on unreleased API. Notebooks 02 and 03 import datasets.make_donor_panel, which the published 0.7.0 wheel does not contain, so both fall into a try/except that installs from git+main and cannot recover in-process: the failed import leaves the stale module in sys.modules, so the re-import after a successful install raises the same error. The Colab badge therefore ran an unreleased snapshot rather than the archived release the paper points at. Version number: plan.md said 0.7.1, while the source tree already described this content as 0.8.0 in three _share_of_wallet.py strings and one deprecations heading. The block is minor-shaped, so 0.7.1 is a deliberate departure from the project's own semver policy, taken because cutting 0.8.0 would make three standing removal promises come due today, including the FiscalYearGroupedSplitter drop_repeat_donors default flip. Those stay queued. The rename shipping here is name-only: same values, same position, old spelling still reachable under a DeprecationWarning until 0.9.0. Also collapses the duplicate CHANGELOG headings the union merge driver left in the unreleased block (four Added, two each of Fixed, Changed and Deprecated) and moves the #194 refactor bullet from Fixed to Changed. RELEASING step 1 exists to catch exactly that; merge=union is line-based, not section-aware, so git will never flag it. Local gate: 1977 passed, 4 skipped, 97.99% total coverage against the 92% floor, riskcov 98% against 93, python -m build plus twine check --strict both PASSED on the 0.7.1 wheel and sdist.
This PR addresses #155.
While looking into the repeated validation blocks in
_transformers.py, I realized why the complex-data guard seemed unreachable.validate_data()returns anobjectdtype array when it falls back to mixed types (e.g. containing a mix of dates, strings, and complex numbers). Because of this,np.iscomplexobj()evaluates toFalse(since the overall array dtype isobject, notcomplex), meaning the guard was silently bypassed and acting as pure dead code.I've refactored the file by:
validate_datatry-except block into a single_validate_Xhelper function to DRY things up.iscomplexobjcheck entirely. Complex numbers now bypass it naturally and are handled downstream by_coerce_currency_to_float(which coerces them toNaNand emits a warning, exactly as expected by the existing test suite).All tests pass locally.
Closes #155