Skip to content

A measured HLL entry overrides the HLL certificate and applies at any distinct count #1392

Open

Description

@zzylol

Example

A distinct-user count over a source that declares 5,000,000 distinct user_id values per evaluation, with a relative error target of 0.02 at δ = 0.01. A calibration table (a MeasuredSketchAccuracy, #1372) holds one HLL entry. The numbers below come from the unit test a_measured_entry_supplies_a_guarantee_where_theory_states_no_failure_probability, not from a run on real data; the sample size is my assumption, since the table does not record it:

{ "sketch": {"Hll": {"precision": 13}}, "statistic": "Cardinality",
  "accuracy_metric": "Relative", "trials": 1000,
  "error_at": [ { "failure_probability": 0.01, "error": 0.02 } ] }

Suppose those 1,000 trials each fed the sketch about 10,000 distinct values. By the code, the planner then treats HLL precision 13 as a certified (0.02, 0.01) readout at 5,000,000 distinct values as well. Two things go wrong:

Wanted: the entry is used only where it was measured, and the certificate is preferred where one exists.

Why it happens

Proposal

Add max_distinct_measured: Option<u64> to MeasuredAccuracyEntry: the largest distinct count per evaluation the calibration fed the sketch (None for sketches whose error does not depend on it, such as KLL). Then either, or both:

  1. Use the entry only when the source's declared max_distinct_per_evaluation is known and at most max_distinct_measured; with no declared bound, an entry that has max_distinct_measured is not used.
  2. Prefer the certificate whenever one exists for the edge, and consult the measured entry only when the theoretical failure probability is unknown.

AccuracyEvidence::measured_guarantee would take the declared bound as an argument (it already receives the sketch parameters and statistic), so the lookup in MeasuredSketchAccuracy stays the only place that applies the rule.

Scope and acceptance

  • In scope: the field, its validation (when present, it must be positive), the lookup rule, and the table JSON (deny_unknown_fields already refuses stale tables).
  • Regression tests that fail today: an HLL entry measured at 10,000 values is not returned for a source declaring 5,000,000 or declaring nothing; it is returned for a declared bound of 9,000; and the certificate wins where both apply.
  • Out of scope: measuring entries per cardinality bucket (a later extension of the same field).

Related

Activity

  1. added
    enhancementNew feature or request
    accuracyAccuracy requirements, guarantees, and propagation
    on Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

accuracyAccuracy requirements, guarantees, and propagationenhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions