Skip to content

forgelm ingest writes rows without an id, so forgelm purge --row-id cannot erase them without a separate preparation step #1053

Open

Description

@cemililik

Summary

forgelm ingest writes {"text": "<chunk>"} rows with no id field, and forgelm purge --row-id matches only a row's id or row_id (it deliberately refuses line numbers). A corpus produced by ForgeLM's own ingestion therefore cannot be purged row by row until the operator adds ids with a script of their own, which the GDPR guide and manual now provide as a Python snippet.

Evidence

  • forgelm/ingestion.py writes each chunk with dumps_strict({"text": payload}, ...).
  • forgelm/cli/subcommands/_purge.py locates a row by id / row_id (_ROW_ID_KEYS); a corpus without them yields NoMatchingRow.
  • docs/guides/gdpr_erasure.md and docs/reference/purge_subcommand.md (and their Turkish mirrors) say ForgeLM provides no id-population helper and that a forgelm audit --add-row-ids flag is "on the Phase 28 backlog". No Phase 28 and no such backlog item exist in the roadmap; this issue is the tracking item. docs/usermanuals/{en,tr}/compliance/gdpr.md carries a standalone script.

Why it matters

An erasure request is the case where the operator is under a deadline (one month under GDPR Art. 12(3)). The documented path is: find the line with forgelm reverse-pii, then read that row's id and pass it to forgelm purge. For ForgeLM-ingested corpora that path needs a separate preparatory step that is easy to miss until the first request arrives. Re-assigning ids after training also breaks the link between the audit log and the corpus, so the ids have to exist from the start.

Acceptance criteria

  • forgelm ingest can write a stable id per chunk (an opt-in flag first; the default can follow once the data loaders are confirmed to ignore the extra key). The id is deterministic for the same source file and chunk position so a re-ingest does not orphan earlier audit events.
  • The id carries no personal data in clear: not the raw source file name (file names often hold a person's name) and not the chunk text. Ids are copied into datasets, splits and downstream artefacts.
  • forgelm purge --row-id <id> --corpus <ingest output> erases such a row without any helper script; a test covers ingest -> purge end to end.
  • The GDPR guide, the purge reference and the user manual (EN and TR) replace the snippet with the flag; the "ForgeLM does not provide a helper" sentences and the "Phase 28 backlog" pointer are removed. Whether corpora that ForgeLM did not ingest also need a helper (the documented audit --add-row-ids idea) is decided in the pull request; if it is wanted, it becomes its own issue.
  • Data loading and the audit are unaffected by the extra key (existing ingestion and data tests stay green).

Context

Found while correcting the GDPR erasure documentation for #101. Related: #102 (memorisation warning), #554 (CR-A10-036: --row-id matches on str() of the JSON value, so the id type matters), #555 (module sweep: purge / reverse-pii).

Activity

  1. added
    enhancementNew feature or request
    severity: mediumReal defect with limited reach or a workaround
    wave: 3Roadmap wave 3: medium severity, module sweeps
    on Oct 22, 2026
  2. added
    needs-triageNot triaged yet: every new issue starts with it; only a maintainer removes it
    on Oct 22, 2026
  3. cemililik commented on Oct 22, 2026

    @cemililik
    CollaboratorAuthor

    Triage (2026-10-22): accepted, scope extended. Medium · Wave 3 · parent #870 (Wave 3 · Module sweep).

    Verified on development at 735a02f: forgelm/ingestion.py writes only {"text": ...} and _purge.py matches id / row_id only.

    Two additions to the description:

    • docs/reference/purge_subcommand.md (EN/TR) carries the same "no helper" sentence as the guide. Both pages say forgelm audit --add-row-ids is "on the Phase 28 backlog", but no Phase 28 or such item exists in the roadmap. This issue is now the tracking item, and the criteria cover removing that pointer.
    • A criterion that the id carries no personal data in clear. Ingest file names often hold a person's name, and the id travels into datasets and artefacts.

    For the implementer: write the id as a JSON string. #554 (CR-A10-036) records that --row-id compares str() of the stored value, so null, bool and float ids misbehave.

    Not blocked. Neither Phase 16 S9 (#895, purge and erasure) nor S10 (#896, data layer) owns ingest row ids.

  4. removed
    needs-triageNot triaged yet: every new issue starts with it; only a maintainer removes it
    on Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: ingestionCode area: ingestionenhancementNew feature or requestseverity: mediumReal defect with limited reach or a workaroundwave: 3Roadmap wave 3: medium severity, module sweeps

    Type

    No type

    Projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions