detect_fraud crashes on two-claim batches and scores unscaled mixed features: fraud flagging is numerically unsound

[kilodesodiq-arch]

Problem

detect_fraud is numerically unsound and crashes on a small, legitimate batch size. In app/ai-service/services/fraud_detection.py:

n_neighbors = min(20, max(2, len(claims) // 2))
lof = LocalOutlierFactor(n_neighbors=n_neighbors, contamination="auto")
lof.fit_predict(X_noise)

scikit-learn's LocalOutlierFactor requires n_neighbors < n_samples. For len(claims) == 2, n_neighbors is 2 and fit raises ValueError (Expected n_neighbors <= n_samples_fit, but n_neighbors = 2, n_samples_fit = 2). Only len(claims) == 1 is special-cased earlier, so a two-claim batch — a realistic batch for a small campaign — crashes the endpoint.

Separately, the feature matrix mixes incomparable scales:

return np.column_stack([
    ip_enc.transform(ips),      # label-encoded integer category codes
    hash_enc.transform(hashes),
    loc_enc.transform(locs),
    amounts,                    # raw float token amounts (up to 1e38)
]).astype(float)

LabelEncoder assigns arbitrary integers to distinct IPs/hashes/locations, so Euclidean distance in the LOF neighbourhood is dominated by whichever column has the largest code range and by raw amount magnitude, not by any meaningful similarity. Finally, _OUTLIER_THRESHOLD = -1.5 compares the raw negative_outlier_factor_ to a constant that has no calibration against this data, so is_flagged can flip with the batch composition.

Consequence: the fraud-risk signal that downstream flows use to flag claims either throws a 500 (two-claim batch) or produces scores driven by arbitrary category codes and magnitude rather than by anomalous behaviour. Flagging can be all-or-nothing depending on how many claims happen to be in the batch, which is a correctness problem for a disbursement-gating signal.

Root cause

The scorer was built around LOF defaults without a per-batch neighbour guard, without feature normalization/encoding design, and with a hardcoded raw-score threshold.

Why this is architecturally hard

1. This is a scoring-pipeline design, not a parameter tweak. The fix must choose a feature representation (one-hot / target encoding / distance-appropriate encoding for IP-hash-location, and normalization or a per-feature distance metric for amount), not just clamp n_neighbors.
2. Small-batch behaviour must be defined. For 2–3 claims, LOF is either undefined or degenerate; the pipeline must specify a fallback (e.g. a rule-based or threshold-on-features path) and return a defined score instead of crashing.
3. The threshold must be calibrated and versioned. A defensible is_flagged needs a documented calibration (percentile, contamination, or an explicit rule) and a model_version so that score meaning does not drift silently; closed issue Add a model_version field to all AI responses #272 already requires a model_version field on AI responses, which this must populate.
4. It is a trust boundary with no regression oracle. There is no golden set of known-fraudulent claims, so tests must at minimum pin: batch sizes 1, 2, 3 do not crash, and a constructed outlier is flagged while a homogeneous batch is not.

Proposed design

Guard n_neighbors to < n_samples (and define a fallback for tiny batches), encode categoricals with a distance-meaningful scheme and normalize/normalize-or-scale amount, and replace the raw -1.5 threshold with a calibrated, documented decision rule. Surface the chosen model_version in the response.

Acceptance criteria

Service

• detect_fraud returns a result (never raises) for batch sizes 1, 2, and 3.
• A constructed outlier in a batch is flagged, and a homogeneous batch is not, independent of the magnitude of the amount column.
• The response includes a model_version consistent with the scoring pipeline version.

Tests

• Tests pin the no-crash behaviour for sizes 1/2/3 and the outlier-flagging behaviour.

Out of scope

Proof-of-life liveness policy and OCR degradation handling are separate issues.

Getting started

Files: app/ai-service/services/fraud_detection.py, app/ai-service/schemas/fraud.py, app/ai-service/tests/.

cd app/ai-service
pytest

Good first files to read: services/fraud_detection.py (_vectorize/detect_fraud) and schemas/fraud.py for the response shape.

[Opulencechuks]

I am confident that I can implement this issue having worked on similar issues in the past.

[Opulencechuks]

I am confident that I can implement this issue having worked on similar issues in the past.

[amberly-d]

can i handle this?

[grantfox-oss]

🦊 GrantFox — @amberly-d has been assigned to this issue as part of the Third Campaign campaign!

Next steps:

1. Open a Pull Request referencing this issue (e.g., Closes #432)
2. Your PR will be reviewed by the ChainForgee maintainers

Good luck! Track your progress on GrantFox.