Skip to content

[SYSTEMDS-XXXX] Add coresetDT builtin for coreset selection using a decision tree#2540

Open
Kenny-Dean wants to merge 1 commit into
apache:mainfrom
Kenny-Dean:coresetDT-builtin
Open

[SYSTEMDS-XXXX] Add coresetDT builtin for coreset selection using a decision tree#2540
Kenny-Dean wants to merge 1 commit into
apache:mainfrom
Kenny-Dean:coresetDT-builtin

Conversation

@Kenny-Dean

Copy link
Copy Markdown

The method is adapted from the datamap-driven coreset approach of Hadar et al.,
"Datamap-Driven Tabular Coreset Selection for Classifier Training"
(https://www.vldb.org/pvldb/vol18/p876-razmadze.pdf).

This patch includes:

  • The coresetDT builtin, that selects a smaller training coreset via a depth-bound tree for classification datasets. Informative regions are kept whole and redundant ones are sampled down to a target fraction.
  • builtin-reference.md documentation.
  • A test on the wine dataset.

The implementation works as intended, datasets are compressible down to a fraction of the full dataset while keeping the full datasets accuracy, however, parameter tweaking of frac, psi, tau, samp_ratio and max tree depth is possible, which allows someone to shift the focus from model hyperparameter tuning to coreset optimization.

The current default parameters are results from some sweeps over the local Titanic and Wine dataset as well as the Adult and a subsampled version on Covertype dataset, which all are medium in row size and manageablein coloum size.
Deviations from the default parameters of the paper include the psi (1 -> 0.9), tau (5 -> 10), datamap model (GBDT -> 1DT)(variation explained in paper), whereas samp_ratio stayed the same. This produces good results.

Some statistics:

Dataset config (psi / frac) Realised frac (coreset % of Full dataset) dtree: full -> core (delta) logreg: full -> core (delta)
Wine 0.90 / 0.15 0.14 97.97 -> (−2.43) 98.44 -> (−0.24)
Adult 0.80 / 0.40 0.40 83.63 -> (−1.32) 83.18 -> (−0.50)
Titanic 0.80 / 0.60 0.60 76.42 -> (−0.81) 78.05 -> (−0.81)
(subsampled) Covertype 0.80 / 0.60 0.60 75.76 -> (−3.00) 71.24 -> (−1.19)

Compressability and the floor are different for each dataset, so its advised to first run the function with verbose=True to get some insights over the region statistics and the dataset- (and parameter-) specific coreset floor.

Future work might include the implementation of the opt_per algorithm that is described in the same paper. It returns the minimal coreset given an accuracy drop you are willing to accept, but the ability to perform parameter sweeps for the best parameters is still possible.

PS: Since this Student Project was given to me without a Jira Issue present the title is only a placeholder for now.

…ecision tree

This patch includes coresetDT, a builtin that selects a smaller training coreset from a decision-tree datamap, with a test on the wine dataset.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

1 participant