Skip to content

refactor: extract validation block and remove dead complex-data guard in transformers - #194

Merged
shivamlalakiya merged 3 commits into
PhilanthroPy-Project:mainfrom
shubhrai23:fix-155-refactor-transformers
Sep 8, 2026
Merged

refactor: extract validation block and remove dead complex-data guard in transformers#194
shivamlalakiya merged 3 commits into
PhilanthroPy-Project:mainfrom
shubhrai23:fix-155-refactor-transformers

Conversation

@shubhrai23

Copy link
Copy Markdown
Contributor

This PR addresses #155.

While looking into the repeated validation blocks in _transformers.py, I realized why the complex-data guard seemed unreachable. validate_data() returns an object dtype array when it falls back to mixed types (e.g. containing a mix of dates, strings, and complex numbers). Because of this, np.iscomplexobj() evaluates to False (since the overall array dtype is object, not complex), meaning the guard was silently bypassed and acting as pure dead code.

I've refactored the file by:

  1. Extracting the duplicated validate_data try-except block into a single _validate_X helper function to DRY things up.
  2. Removing the dead iscomplexobj check entirely. Complex numbers now bypass it naturally and are handled downstream by _coerce_currency_to_float (which coerces them to NaN and emits a warning, exactly as expected by the existing test suite).

All tests pass locally.

Closes #155

Copilot AI lite review requested due to automatic review settings September 7, 2026 03:12

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Repository policy requires an ## [Unreleased] CHANGELOG.md entry for PRs touching philanthropy/, and the updated complex-data behavior wording in transformer docstrings should be corrected to match runtime behavior.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR refactors philanthropy.preprocessing._transformers by extracting the repeated validate_data() try/except (with object-dtype fallback) into a shared _validate_X helper, and removes the redundant np.iscomplexobj complex-data guard that was effectively dead under the current validation behavior.

Changes:

  • Added _validate_X() to centralize the validate_data(..., dtype=None) + object-cast retry logic.
  • Replaced duplicated inline validation blocks in CRMCleaner and FiscalYearTransformer with calls to _validate_X().
  • Removed the trailing np.iscomplexobj(...) checks and their ValueError("Complex data not supported") raises.
File summaries
File Description
philanthropy/preprocessing/_transformers.py DRYs up validation logic via _validate_X and removes the redundant complex-object guard in both transformers.
Review details

Suppressed comments (3)

philanthropy/preprocessing/_transformers.py:198

  • This docstring claims complex data always raises, but complex values can survive validation when they arrive inside object-dtype columns (and are handled downstream). Update the wording to reflect that the ValueError is for inputs that fail validation (e.g., complex-dtype arrays).
            If ``X`` contains complex data.

philanthropy/preprocessing/_transformers.py:306

  • This docstring says transform raises whenever X contains complex data, but in practice the ValueError comes from sklearn validation for complex-dtype inputs; complex values inside object-dtype columns can flow through and get coerced. Updating the wording avoids overstating the contract.
            If ``X`` contains complex data.

philanthropy/preprocessing/_transformers.py:277

  • The docstring currently states that any complex data in X raises, but the validation logic only reliably raises for complex-dtype inputs; complex values inside object-typed columns may pass and be coerced later. Consider describing the error condition as 'fails validation' to match runtime behavior.
            If ``fiscal_year_start`` is invalid or ``X`` contains complex data.
  • Files reviewed: 1/1 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@@ -161,19 +172,7 @@ def fit(self: _SelfC, X: Any, y: Any = None) -> _SelfC:
If ``fiscal_year_start`` is invalid or ``X`` contains complex data.
@shivamlalakiya

Copy link
Copy Markdown
Contributor

Read the diff rather than merging on green, since this one touches validation shared by two transformers. It holds up.

The reachability finding is right, and it is right for two independent reasons rather than one. A genuine complex dtype never reaches np.iscomplexobj because scikit-learn's check_array calls _ensure_no_complex_data first and raises with exactly the phrase the except clause re-raises on. And an object-dtype array holding Python complex cells has dtype.kind == "O", so np.iscomplexobj returns False on it too. Both paths miss the guard, which is why nothing in the suite ever executed those four lines.

Checked that the deletion is not removing behaviour the suite was relying on: all four entry points still have a complex-rejection test (test_complex_column_is_re_raised_not_retried covers both fit methods, test_crm_transform_re_raises_complex_column and test_fiscal_year_transform_re_raises_complex_column cover both transform methods), and they pass on scikit-learn's own message. check_complex_data in test_sklearn_compliance.py is unaffected for the same reason. grep -c iscomplexobj is now 0, which clears the done-when condition in #155.

Two small things, neither worth another round trip:

Thanks. Four copies became one and the dead line that made the coverage report lie is gone.

@shivamlalakiya
shivamlalakiya merged commit 3ef3fb9 into PhilanthroPy-Project:main Sep 8, 2026
15 checks passed
shivamlalakiya added a commit that referenced this pull request Sep 8, 2026
Cut 0.7.1 so the notebooks stop depending on unreleased API. Notebooks 02
and 03 import datasets.make_donor_panel, which the published 0.7.0 wheel
does not contain, so both fall into a try/except that installs from
git+main and cannot recover in-process: the failed import leaves the stale
module in sys.modules, so the re-import after a successful install raises
the same error. The Colab badge therefore ran an unreleased snapshot
rather than the archived release the paper points at.

Version number: plan.md said 0.7.1, while the source tree already
described this content as 0.8.0 in three _share_of_wallet.py strings and
one deprecations heading. The block is minor-shaped, so 0.7.1 is a
deliberate departure from the project's own semver policy, taken because
cutting 0.8.0 would make three standing removal promises come due today,
including the FiscalYearGroupedSplitter drop_repeat_donors default flip.
Those stay queued. The rename shipping here is name-only: same values,
same position, old spelling still reachable under a DeprecationWarning
until 0.9.0.

Also collapses the duplicate CHANGELOG headings the union merge driver
left in the unreleased block (four Added, two each of Fixed, Changed and
Deprecated) and moves the #194 refactor bullet from Fixed to Changed.
RELEASING step 1 exists to catch exactly that; merge=union is line-based,
not section-aware, so git will never flag it.

Local gate: 1977 passed, 4 skipped, 97.99% total coverage against the 92%
floor, riskcov 98% against 93, python -m build plus twine check --strict
both PASSED on the 0.7.1 wheel and sdist.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

refactor: _transformers.py repeats one validation block four times, and its complex-data guard looks unreachable

3 participants