Skip to content

Add a UniqueSequenceProposer for unique/ID columns — no current proposer guarantees non-duplicate values #134

Description

@yhong123

Problem

No proposer currently generates new, unique, non-memorized values for a genuine unique-ID (or other unique) column, so the evaluator's recommender has no sound candidate to recommend for that column type.

Every generator in the ChoiceProposer family (ZipfChoiceProposer, UniformChoiceProposer, WeightedChoiceProposerproposers/choice.py) samples with replacement from at most MAXIMUM_CHOICES (500) distinct observed values, via dist_gen.choice_direct/zipf_choice_direct/weighted_choice in providers.py, which call random.choice/random.choices under the hood. Once the generated row count exceeds the number of distinct sampled values, duplicates are guaranteed — which would violate a real PRIMARY KEY/UNIQUE constraint on a genuine ID column.

The only other fallback, dist_gen.constant (providers.py:403), is worse: it returns the exact same value every time, so it duplicates as early as the second generated row.

In short: for a column that must contain unique values, every currently-registered proposer (see everything_factory() in proposers/init.py) will eventually produce a constraint violation once enough rows are generated. The evaluator's recommendation for such a column is therefore never actually usable at scale — it's picking "least-bad among broken options," not a correct one.

Proposed solution

Add a new UniqueSequenceProposer (+ matching ProposerFactory) that is guaranteed not to repeat values, registered alongside the existing factories in everything_factory() (proposers/init.py:43-60).

Acceptance criteria

  • New proposer(s) never produce a duplicate value across a full data-generation run for a column previously flagged as unique (verify against a dense serial PK column, e.g. Pagila actor_id).
  • Registered in everything_factory() so it appears as a candidate in propose/propose all.
  • Wins the evaluator's ranking for genuinely unique columns (see datafaker/evaluators/) without tripping weak_recommendation_warning (in proposal_ranking.py).
  • Existing uniqueness/resample penalty logic in the evaluator (currently penalizing ChoiceProposer and numeric-column candidates for exactly this failure mode) should no longer need to penalize this new proposer, since it isn't a resampler.

Context

Found while building an evaluator/recommender module (datafaker/evaluators/) that scores and ranks candidate generators for a column. The evaluator currently applies a uniqueness/resample penalty (1 - real_uniqueness × copy_fraction) to flag exactly this problem, but a penalty on bad options isn't a substitute for having a correct one to recommend.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions