Feat/verify and decontaminate

CHANGELOG.md

@@ -10,6 +10,8 @@ endorsed by Adaption Labs.
 
 ## [Unreleased]
 
+## [0.2.0] - 2026-07-17
+
 ### Added
 
 - `verify` command (`adaption-kit verify`) to prove rows are correct before you
@@ -26,9 +28,39 @@ endorsed by Adaption Labs.
   the cleaned rows.
 - Optional `verify` extra (`pip install -e ".[verify]"`) that adds sympy for the
   symbolic math equivalence check.
-- `doctor` command to check your environment and configuration (in progress).
-- `suggest` command to recommend recipes and brand controls for your domain
-  (in progress).
+- `doctor` command (`adaption-kit doctor`) for an offline environment healthcheck
+  of Python, the SDK, your env vars, and the host.
+- `suggest` command (`adaption-kit suggest`) that reads your file and recommends
+  the column mapping to use.
+
+### Fixed
+
+- `decontaminate`: a benchmark prompt shorter than the n-gram size that appeared
+  verbatim *inside* a longer training row was silently not flagged. Short
+  benchmark texts are now indexed at their own length and matched against
+  same-size windows, so embedded contamination is caught.
+- `verify --kind math`: whole-number answers now require exact equality. The old
+  relative tolerance accepted off-by-one errors once the answer magnitude reached
+  ~1e6.
+- `verify --kind code`: the pass/fail tally is read from a unique sentinel and the
+  last match, so a candidate that prints a `PASSED n` line can no longer spoof its
+  own result. Empty test entries no longer count as passes.
+- `decontaminate`: an n-gram size below 1 now fails loudly instead of flagging and
+  deleting every row.
+
+### Security
+
+- `verify --kind code` executes dataset code in a subprocess, which is NOT a
+  security sandbox — it runs with your full user privileges. The misleading "never
+  your machine" wording was corrected, and the CLI now prints a warning before it
+  runs. Only verify code from datasets you trust.
+
+### Tests
+
+- Test coverage for the SDK-backed helpers (`run.py`), the CLI router (`cli.py`),
+  and the publish helper (`publish.py`), using the `client=` injection hooks so no
+  network or real SDK is required.
+- Regression tests for each of the correctness fixes above.
 
 ## [0.1.0] - 2026-06-07
 
@@ -50,5 +82,6 @@ endorsed by Adaption Labs.
 - Templates for dataset schemas, dataset and model cards, a cover, and Kaggle
   metadata.
 
-[Unreleased]: https://github.com/A1VARA5/adaption-devkit/compare/v0.1.0...HEAD
+[Unreleased]: https://github.com/A1VARA5/adaption-devkit/compare/v0.2.0...HEAD
+[0.2.0]: https://github.com/A1VARA5/adaption-devkit/compare/v0.1.0...v0.2.0
 [0.1.0]: https://github.com/A1VARA5/adaption-devkit/releases/tag/v0.1.0

MAP.md

@@ -24,7 +24,7 @@ tooling.
 | `cookbook/` | Runnable notebooks that walk the full lifecycle. | You learn best by running real code. |
 | `templates/` | Dataset schemas, dataset and model cards, a cover, Kaggle metadata. | You are preparing a release. |
 | `graphics/` | The diagrams embedded in the README, as Mermaid in Markdown. | You want the source of a diagram. |
-| `pyproject.toml` | Package metadata and optional extras (`sdk`, `notebooks`). | You are installing or packaging. |
+| `pyproject.toml` | Package metadata and optional extras (`sdk`, `verify`, `all`, ...). | You are installing or packaging. |
 | `LICENSE` | Apache-2.0. | You need the license text. |
 
 ### The CLI commands
@@ -33,14 +33,17 @@ tooling.
 
 | Command | What it does |
 |---------|--------------|
+| `doctor` | Offline healthcheck of Python, the SDK, your env vars, and the host. |
+| `suggest` | Read your file and recommend the column mapping to use. |
 | `lint` | Preflight a dataset before a run. Catches duplicate prompts, encoding issues, and empty anchors before you spend credits. |
+| `verify` | Prove math answers and code rows are correct before you adapt them. |
+| `decontaminate` | Drop training rows that overlap a benchmark test set by an n-gram. |
+| `convert` | Convert a dataset between CSV, JSONL, and Parquet (BOM-safe). |
 | `estimate` | Quote credits and time for a run without starting one. |
 | `run` | Start an adaptation run, estimate first, optionally wait and print `improvement_percent`. |
 | `publish` | Publish helper that packages a release for Hugging Face and Kaggle, because the platform publish endpoint returns 501. |
 | `card` | Generate a dataset card, a model card, or Kaggle metadata. |
 | `cover` | Render a cover image for your release. |
-| `doctor` | Coming soon: check your environment and configuration. |
-| `suggest` | Coming soon: suggest recipes and controls for your domain. |
 
 ### The guides
 

README.md

@@ -89,8 +89,8 @@ pip install -e ".[sdk]"
 # the symbolic math check used by 'adaption-kit verify --kind math'
 pip install -e ".[verify]"
 
-# everything for the cookbook notebooks
-pip install -e ".[notebooks]"
+# everything (SDK, HF, Kaggle, Playwright cover, Parquet, sympy) for the cookbook
+pip install -e ".[all]"
 ```
 
 If an extra is not installed, the matching command tells you what to add. The

adaption_kit/__init__.py

@@ -9,7 +9,7 @@
 
 from __future__ import annotations
 
-__version__ = "0.1.0"
+__version__ = "0.2.0"
 __author__ = "Aivaras Navardauskas (MANIFESTA)"
 __license__ = "Apache-2.0"
 

adaption_kit/cli.py

@@ -70,6 +70,14 @@ def _cmd_lint(args: argparse.Namespace) -> int:
 def _cmd_verify(args: argparse.Namespace) -> int:
     from .verify import FAIL, verify_dataset
 
+    if args.kind == "code":
+        print(
+            "warning: 'verify --kind code' runs the dataset's code in a subprocess "
+            "with your full user privileges. It contains crashes and hangs but is "
+            "NOT a security sandbox. Only run it on data you trust.",
+            file=sys.stderr,
+        )
+
     report = verify_dataset(
         args.path,
         kind=args.kind,
@@ -129,9 +137,9 @@ def _cmd_estimate(args: argparse.Namespace) -> int:
             completion=args.completion,
             context=_split_csv(args.context),
             chat=args.chat,
-            deduplication=_tri(args.deduplication),
-            prompt_rephrase=_tri(args.prompt_rephrase),
-            reasoning_traces=_tri(args.reasoning_traces),
+            deduplication=args.deduplication,
+            prompt_rephrase=args.prompt_rephrase,
+            reasoning_traces=args.reasoning_traces,
         )
     except SdkNotInstalled as exc:
         print(str(exc), file=sys.stderr)
@@ -157,9 +165,9 @@ def _cmd_run(args: argparse.Namespace) -> int:
         completion=args.completion,
         context=_split_csv(args.context),
         chat=args.chat,
-        deduplication=_tri(args.deduplication),
-        prompt_rephrase=_tri(args.prompt_rephrase),
-        reasoning_traces=_tri(args.reasoning_traces),
+        deduplication=args.deduplication,
+        prompt_rephrase=args.prompt_rephrase,
+        reasoning_traces=args.reasoning_traces,
         idempotency_key=args.idempotency_key,
     )
     if args.pilot:
@@ -290,11 +298,6 @@ def _cmd_cover(args: argparse.Namespace) -> int:
 # ---------------------------------------------------------------------------
 
 
-def _tri(value: Optional[bool]) -> Optional[bool]:
-    """Pass through tri-state recipe flags (None = backend default)."""
-    return value
-
-
 def _write_or_print(text: str, out: Optional[Path], default_name: str) -> None:
     if out is None:
         sys.stdout.write(text)

adaption_kit/decontaminate.py

@@ -48,20 +48,41 @@ def ngrams(text: str, n: int = DEFAULT_N) -> set:
 
 
 class Decontaminator:
-    """Holds the benchmark n-gram set; flags any text that shares one."""
+    """Holds the benchmark n-gram set; flags any text that shares one.
+
+    Benchmark texts shorter than ``n`` tokens are indexed at their own length
+    (``min(n, len)``) rather than as one whole-string shingle, so a short
+    benchmark question embedded verbatim in a *longer* training row is still
+    caught. Matching compares training windows of the same sizes present in the
+    benchmark.
+    """
 
     def __init__(self, n: int = DEFAULT_N) -> None:
         self.n = n
         self.bench: set = set()
+        self._sizes: set = set()
 
     def add_text(self, text: str) -> None:
-        self.bench |= ngrams(text, self.n)
+        toks = _normalize(text).split()
+        if not toks:
+            return
+        k = min(self.n, len(toks))
+        self._sizes.add(k)
+        self.bench |= {" ".join(toks[i : i + k]) for i in range(len(toks) - k + 1)}
 
     def is_contaminated(self, text: str) -> bool:
-        g = ngrams(text, self.n)
-        if not g:
+        if not self.bench:
             return False
-        return not g.isdisjoint(self.bench)
+        toks = _normalize(text).split()
+        if not toks:
+            return False
+        for k in self._sizes:
+            if len(toks) < k:
+                continue
+            windows = {" ".join(toks[i : i + k]) for i in range(len(toks) - k + 1)}
+            if not windows.isdisjoint(self.bench):
+                return True
+        return False
 
 
 @dataclass
@@ -155,6 +176,9 @@ def decontaminate(
     """
     p = Path(path)
     report = DecontamReport(path=str(p), n=n)
+    if n < 1:
+        report.add(FAIL, "n-gram size must be >= 1 (got " + str(n) + ")")
+        return report
     if not p.exists():
         report.add(FAIL, "file does not exist")
         return report

adaption_kit/verify.py

@@ -6,8 +6,10 @@
   - math: the final answer the worked solution reaches must be equivalent to a gold
     answer (normalized string, then numeric, then symbolic via sympy if installed).
   - code: the solution must run and pass the unit tests shipped with it. Each
-    candidate runs in a fresh, short-lived subprocess with a hard timeout, so a bad
-    row can only ever crash its own child, never your machine or the build.
+    candidate runs in a fresh, short-lived subprocess with a hard timeout, which
+    contains crashes and hangs -- but it is NOT a security sandbox. The row's code
+    runs with your full user privileges, so only verify code from datasets you
+    trust. The CLI prints a warning before it runs.
 
 Adapting unverified rows is the most common way beginners burn credits on multiple
 tries: you pay to polish data that was wrong to begin with. Filter first, then run.
@@ -153,6 +155,10 @@ def answers_equivalent(gold: Optional[str], pred: Optional[str], timeout: float
         return True
     fg, fp = _try_float(ng), _try_float(npd)
     if fg is not None and fp is not None:
+        # Whole-number answers must match exactly; the relative tolerance below
+        # would otherwise accept off-by-one errors once |answer| >= ~1e6.
+        if float(fg).is_integer() and float(fp).is_integer():
+            return fg == fp
         return abs(fg - fp) <= 1e-6 * max(1.0, abs(fg), abs(fp))
     if not _HAVE_SYMPY:
         return False
@@ -187,16 +193,16 @@ def _coerce_tests(tests: Any) -> List[str]:
 def code_passes(output: str, tests: Any, setup: str = "") -> bool:
     """True only if the solution runs and passes EVERY assert-style test."""
     code = extract_code(output)
-    tests = _coerce_tests(tests)
+    tests = [t for t in _coerce_tests(tests) if t.strip()]
     if len(code) < 5 or not tests:
         return False
     harness = setup + "\n" + code + "\n\n_p = 0\n"
     for t in tests:
         harness += (
-            "try:\n" + textwrap.indent(t.strip() or "pass", "    ")
+            "try:\n" + textwrap.indent(t.strip(), "    ")
             + "\n    _p += 1\nexcept Exception:\n    pass\n"
         )
-    harness += "print('PASSED', _p)\n"
+    harness += "print('__ADK_PASSED__', _p)\n"
     with tempfile.TemporaryDirectory() as d:
         fp = os.path.join(d, "cand.py")
         with open(fp, "w", encoding="utf-8") as f:
@@ -208,8 +214,8 @@ def code_passes(output: str, tests: Any, setup: str = "") -> bool:
             )
         except Exception:
             return False
-    m = re.search(r"PASSED (\d+)", r.stdout or "")
-    return bool(m) and int(m.group(1)) == len(tests)
+    matches = re.findall(r"__ADK_PASSED__ (\d+)", r.stdout or "")
+    return bool(matches) and int(matches[-1]) == len(tests)
 
 
 # --------------------------------------------------------------------------- report

guides/quickstart.md

@@ -10,15 +10,15 @@ This guide takes you from nothing to your first Adaptive Data run, with a real
 
 ## 0. What you need
 
-- Python 3.9 or newer.
+- Python 3.10 or newer.
 - An Adaption API key.
 - A small data file to start with (`.csv`, `.json`, `.jsonl`, or `.parquet`).
 
 ## 1. Install
 
 ```bash
 pip install adaption          # the official SDK
-pip install adaption-devkit   # this community toolkit (provides the adaption-kit CLI)
+pip install adaption-kit      # this community toolkit (provides the adaption-kit CLI)
 ```
 
 If you are working from a checkout of this repo instead:

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "adaption-kit"
-version = "0.1.0"
+version = "0.2.0"
 description = "Community, unofficial open source toolkit for starting fast with Adaption Adaptive Data and AutoScientist."
 readme = "adaption_kit/README.md"
 requires-python = ">=3.10"