Quantile Threshold Calculation + Filtering - #252

Draft

ajude2s wants to merge 24 commits into

quantile_calculation

Commits on Nov 18, 2025

refactor: remove ununsed notebook
mali-git
committed
63ad1e4
docs: add tutorial for generating embeddings
mali-git
committed
23dfa98
chore: improve figure
mali-git
committed
32ea27d
refactor: remove ununse notebook
mali-git
committed
cbfa05d

Commits on Nov 19, 2025

chore: add notebooks
mali-git
committed
bbedaee

Commits on Jan 20, 2026

feat: Implement paired threshold filtering for JSONL files
ajude2s
committed
2009cad
feat: Enhance threshold filter pipeline for paired JSONL processing
ajude2s
committed
7cc62aa

Commits on Jan 22, 2026

feat: Remove threshold filter pipeline configuration file
ajude2s
committed
503af94
Merge branch 'jsonl_filtering' into quantile_calculation
ajude2s
committed
15aa33f

Commits on Jan 23, 2026

refactor: move datatrove pipelines into data_pipelines and centralize execution helpers
AbasKhan
committed
c2b055c

Commits on Jan 24, 2026

feat: Add quantile data and tests for quantile pipeline functionality
AbasKhan
committed
afaa85d

Commits on Jan 26, 2026

Commits on Jan 27, 2026

Commits on Jan 28, 2026

feat: add average threshold filtering pipeline and associated utilities
ajude2s
committed
c22c786