formtransform

One library to transform surveys between the standards of the CDL survey ecosystem — XLSForm (Kobo Toolbox), LimeSurvey TSV, and DDI Codebook — built on a canonical survey type registry (registry/) that defines exactly what is supported and how the standards map onto each other.

What this repo ships

Four things other projects consume — all derived from the one registry:

Artifact	Where	Consumed by
@correlaid/formtransform — TypeScript library + formtransform CLI	npm package (github:CorrelAid/formtransform)	formtransform-app, qwacback, direct CLI use
schematron-worker image — Java NATS service validating DDI against the XSDs + CDL rules, baked in	ghcr.io/correlaid/schematron-worker:<version>, built from workers/schematron-worker/	qwacback (runs the image, does not build it)
DDI validation assets — DDI 2.5 XSDs + the codegen-written ddi_custom_rules.sch	ddi-validation/{xsd,schematron}/	qwacback, synced via its .registry-version pin
cdl-survey-types skill — self-contained Agent Skill (SKILL.md + references/), generated from the registry; portable to any agent runtime that reads the format	skills/cdl-survey-types/	formulaid's generating-xlsforms skill, which owns the workflow and includes this as the type reference

Everything else here (docs/ spec site, tests) is in-repo only; the fixtures and the question-type screenshots also ship as release assets (below).

Installation

As a library

Each release carries a prebuilt package. Installing it runs no build step:

npm install https://github.com/CorrelAid/formtransform/releases/download/v0.1.7/correlaid-formtransform-0.1.7.tgz

Installing from git also works, but builds dist/ on install through the prepare script, which needs TypeScript and install scripts enabled:

npm install github:CorrelAid/formtransform#v0.1.7

Releases also attach cdl-survey-types-<version>.tar.gz, the generated skills/cdl-survey-types/ sub-skill, and formtransform-fixtures-<version>.tar.gz, the example fixtures and blessed snapshots (registry/entities/, tests/fixtures/surveys/, same paths) for golden tests, and formtransform-screenshots-<version>.tar.gz, each registry type rendered in LimeSurvey and in Enketo (the engine behind KoboToolbox web forms), unpacking to screenshots/<slug>/{limesurvey,xlsform}.png with a screenshots/manifest.json (size, engine and the version that rendered each; see screenshots/README.md). Assets never change after publishing, so their checksums can be pinned: each release also attaches SHA256SUMS and lists the checksums in its notes.

As a CLI tool

npx github:CorrelAid/formtransform --help

Quick Start

TypeScript/JavaScript

One function per direction. An XLSForm source is the .xlsx bytes or its loaded sheets ({ surveyData, choicesData, settingsData }); a LimeSurvey source is the structure TSV text:

import {
  xlsformToLstsv,
  xlsformToDdi,
  lstsvToDdi,
  lstsvToXlsform,
  ddiToXlsform,
} from '@correlaid/formtransform';

const bytes = await file.arrayBuffer();
const tsv = await xlsformToLstsv(bytes);
const xml = xlsformToDdi(bytes);
const fromLs = lstsvToDdi(tsv);
const { survey, choices, settings } = lstsvToXlsform(tsv);
const back = ddiToXlsform(xml); // a codebook, or a <var>/<varGrp> fragment

Each rejects input outside its subset with a ConversionError unless you pass skipValidation: true (see Supported XLSForm Subset).

The building blocks behind these (the converter class, type mapper, TSV serializer, expression transpiler) are exported from @correlaid/formtransform/internals, with no stability promise. XLSFormToTSVConverter, XLSFormParser, buildDdiXml and lstsvToDdiXml still work from the main entry point but are deprecated and will be removed from it in the next minor release.

A DDI codebook and the data file it describes come from the same variable list, so the CSV headers match the XML <var name=""> elements one-to-one. For the whole path from a Kobo or LimeSurvey export, see RESPONSE_DATA.md:

import {
  xlsformToDdi,
  buildDataCsv,
  extractVariables,
  choicesByListFromRows,
} from '@correlaid/formtransform';

const xml = xlsformToDdi({ surveyData: survey, choicesData: choices, settingsData: settings }, { submissions });
const csv = buildDataCsv(
  extractVariables(survey, choicesByListFromRows(choices)),
  submissions,
);

Command Line

# Convert XLSForm to LimeSurvey TSV
formtransform xlsform2lstsv survey.xlsx -o survey.tsv

# Convert to DDI Codebook
formtransform xlsform2ddi survey.xlsx -o codebook.xml

# ...plus the response-data CSV (flat CSV, `;` or `,`, or a Kobo submissions
# JSON array). Writes data.csv beside codebook.xml and sets <caseQnty>.
formtransform xlsform2ddi survey.xlsx -o codebook.xml --data responses.csv

# Same for LimeSurvey: structure TSV + response export (question-code headings)
formtransform lstsv2ddi survey.tsv -o codebook.xml --data responses.csv

# Convert from LimeSurvey TSV back to XLSForm (emitted as JSON sheets)
formtransform lstsv2xlsform survey.tsv -o recovered.json

# Convert a DDI codebook (or a <var>/<varGrp> fragment) back to XLSForm (JSON sheets)
formtransform ddi2xlsform codebook.xml -o form.json

Asking the library what exists

Two generated catalogues are part of the public API, for consumers that render or generate surveys rather than convert them:

import { QUESTION_TYPES, APPEARANCES, TYPE_MAPPINGS } from '@correlaid/formtransform';

QUESTION_TYPES.select_one.label;          // "Select One"
QUESTION_TYPES.select_one.labels.de;      // "Einfachauswahl"
QUESTION_TYPES.select_one.useWhen;        // when to reach for this type
QUESTION_TYPES.select_one_other.base;     // "select_one" — a variant of it
QUESTION_TYPES.select_one_other.presentation; // { withOther: true, withLongList: false }
QUESTION_TYPES.grid.bases;                // composites span several types
QUESTION_TYPES.select_one.constraints;    // name/choice-code limits
APPEARANCES.label.carriesData;            // false — a matrix header stores no answer
TYPE_MAPPINGS.select_one.limeSurveyType;  // "L" — how it converts

QUESTION_TYPES is keyed by registry slug and answers what a row can be (labels in English and German, guidance, variant → base and presentation, authoring constraints, metadata rows); TYPE_MAPPINGS answers how it converts. Both are generated from registry/ — never hardcode a type list or a label in a consumer.

The Registry is the Source of Truth

registry/ is the single source every other artifact in this repo derives from. It is hand-authored (JSON-LD + vocabulary CSVs + per-entity fixtures); nothing else here defines what a question type is, how it maps between standards, or what is allowed. Every downstream artifact — the TypeScript library and its CLI, the generated Schematron rules and the worker image that ships them, the Claude Code skill, the spec site, the test fixtures — is either generated from the registry by codegen or reads registry-derived data at runtime.

Concretely, this means:

• A behaviour change starts in registry/, never downstream. Editing generated files by hand is pointless — the next uv run codegen overwrites them.
• codegen is the only writer of those artifacts, and it validates the registry first, so an invalid registry cannot produce artifacts at all.
• Downstream repos inherit the registry transitively. formtransform-app and qwacback depend on @correlaid/formtransform and on the worker image; formulaid depends on the generated skill. None of them carry their own type list — adding a question type here is what makes it exist for all of them.

Supported Standards

Three standards, each owned by a different tool and built for a different job. They disagree on basic terms — what a question is, what its parts are — so the library routes every transformation through the canonical registry, which records how each maps onto it.

Standard	Role	Type system
XLSForm (Kobo/ODK)	Authoring — surveys are written here	type strings (select_one, integer)
LimeSurvey TSV	Deployment — recreate the survey in LimeSurvey	type codes (L, M, F, N)
DDI Codebook 2.5	Documentation — describe the resulting dataset	interval class + response domains (category, multiple)

Supported directions: five, one per module under src/pipelines/ — xlsform2lstsv (deploy the survey), xlsform2ddi (document the dataset), lstsv2ddi, lstsv2xlsform and ddi2xlsform (the reverse paths). All are lossy for some types: nested groups flatten in LimeSurvey, choice codes over 5 chars truncate, select_multiple becomes N binary variables, and the LimeSurvey reverse paths cannot recover a select's authored list_name. A CDL codebook holds the whole form: XLSForm → DDI → XLSForm gives it back, and its codebook again.

DDI goes back to XLSForm: a CDL codebook carries the whole form. Skip logic, validation and required are each a readable <universe> sentence (a simple numeric range also <valrng>), plus the exact expression in a typed note such as <notes type="cdl:relevant" subject="xlsform-xpath"> (convention:logicMapping). Groups, order, hints, defaults, appearances and parameters, list names, note rows and metadata rows, settings and language names are in standard DDI where it has a place and typed notes where not (convention:ddiFields). ddiToXlsform reads it back; any other DDI converts as far as its standard elements go, with a warning per missing field (src/pipelines/ddi2xlsform/README.md).

<concept> is optional (#173). The codebook has one only where the form says what a question or group measures, in an optional concept column (concept::<lang> per language, like label); a select_*_from_file keeps its concept/@vocab. A form without the column gets no concept: the label is the question, not its concept.

A short label on a question group (#181). An optional label_short column (label_short::<lang> per language, like label) gives a group a short printed name, such as a scale's name (ATTARI-12) or a question number, that a reader can show as a title above the group's text. It goes in the varGrp's <labl>, before its <txt>: on a begin_group row, and on a select_multiple or semi-open question, which are varGrps too. CDL keeps labl off var, so on any other row the cell is kept as one of the row's columns (cdl:column). Two settings name the whole form the same way: form_subtitle is the study's <subTitl> and form_title_short its <altTitl> (plain, or one per language like form_title).

Errors and warnings

Every error the library throws is a ConversionError with a stable code (type-unregistered, choice-list-empty, xpath-syntax, …; see DiagnosticCode), a human message, and subject, the question it concerns. Branch on code, not on the message text.

xlsformToLstsv and xlsformToDdi check the whole form against the subset first (validateSubset, see below). If anything is outside it, they throw one ConversionError with code xlsform-outside-subset, and details holds every finding, so a form with several problems is fixed in one pass.

Warnings (a truncated name, an ignored appearance, a comparison with a value that isn't one of the choices, a constraint that can't be converted) go to an onWarning callback. This includes the subset check's warnings, each passed once. The default prints them to the console:

const warnings: Diagnostic[] = [];
await xlsformToLstsv(bytes, { onWarning: (w) => warnings.push(w) });

validateSubset returns the same Diagnostic objects without converting.

Supported XLSForm Subset

Not everything XLSForm allows is registered (supported). The library strictly validates against this subset, rejecting:

• Unregistered types (no LimeSurvey equivalent — image, audio, geopoint, etc.)
• Unregistered appearances (warn + ignore)
• Out-of-subset names — field names and answer codes must match ^[a-zA-Z0-9]+$ (no underscores, except the <question>_other companion), names at most 20 characters, codes at most 5. FieldSanitizer turns free text into conforming names: it transliterates (ä→ae, ß→ss), drops other diacritics and deletes the rest of the non-alphanumerics
• Deep nesting — max 3 levels (group/group/question)
• Duplicate or missing answer codes — each choice needs a code, unique within its list (an empty choice label is only a warning)
• Unresolvable answer options — a select_one/select_multiple needs a list name with rows on the choices sheet; a select_*_from_file needs a registered vocabulary (e.g. iso_3166_1.csv) or a CSV passed as fileChoices (the CLI reads CSVs beside the form)
• Unmappable languages — language tags are BCP 47 (de, fr-BE, zh-Hans), in label::<tag> or label::Name (<tag>) columns and default_language. LimeSurvey has a fixed code list: a regional tag it lacks becomes its language with a warning (fr-BE → fr), and a language it lacks entirely (eo), or two tags on one code, is an error. DDI keeps tags as written and carries every language the form has: the base language (default_language, else the first) untagged and first, each other one as an xml:lang sibling. Only the form's own texts go in, never a translation; a language that lacks a text gets no element
• Reserved words — relevance, validation, text, etc. (LimeSurvey internals)
• Dangling references — every ${name} in relevant or constraint must name a row of the survey sheet. A compared literal that the question can never take (${q} = 'Sonstiges' when the code is sonst, selected(${m}, 'x'), ${age} = 'old') is only a warning: the form converts, but the condition is never true

A select_*_from_file list keeps each code that fits LimeSurvey's answer codes (DE). Any other code (a 12-digit regional key, 059700040040) gets a 5-character code derived from the code itself, so it stays the same when the file is reordered or rows are added or removed (convention:externalCodeList, limesurveyCodes). The mapping beside the TSV (see --sanitize below) records each derived code, and lstsv2ddi --data writes the file's own codes back. appearance: minimal or autocomplete makes the question a LimeSurvey dropdown instead of a radio list. LimeSurvey's dropdown has no search: its cssclass cdlvocab-<vocabulary> records where the options came from (the way back reads it), and it is where a LimeSurvey theme can hook in a searchable dropdown. formtransform ships no such theme.

An exclusive answer in a select_multiple ("Keine Angabe", "Nichts davon") is marked with an exclusive column (yes) on the choices sheet, not a count-selected() constraint. LimeSurvey enforces it through exclude_all_others; DDI has no field for it.

count-selected() counts the ticked answers of a select_multiple. In a relevant (count-selected(${k04}) > 1) it becomes count() over the question's subquestions in LimeSurvey. A constraint on the question itself that bounds it (count-selected(.) >= 1 and count-selected(.) <= 3) becomes LimeSurvey's min_answers / max_answers; the reverse paths read those back as the same constraint.

A choice_filter that keeps the options ticked in an earlier select_multiple (selected(${k04}, name)) becomes LimeSurvey's array_filter, which needs the option codes to equal the filtering question's answer codes. When a choices column holds the matching codes instead (selected(${k04}, kurz)), the filtered question's LimeSurvey answer codes are that column's values, and a choice-filter-recoded warning says so: LimeSurvey then stores those codes for it. Any other choice_filter is dropped with a choice-filter-dropped warning, and LimeSurvey shows every option.

The or_other shorthand writes no text for its "other" answer; the survey tool supplies one. LimeSurvey shows its own Other: in the survey language (Sonstiges:), ODK and Kobo an untranslated "Other" / "Specify other.". The DDI records LimeSurvey's wording in each of the form's languages, and the DDI check warns (other-shorthand). To record your own text on every platform, write an other choice and a <question>_other text question instead.

To keep such names, convert with --sanitize (xlsformToLstsvWithMapping(form, { sanitize: true })). Names and codes outside the limits are then sanitized instead of refused (k23_ars → k23ars, a long name truncated and made unique, abwesend → abwes, a negative code -98 → n98), and every change is recorded in a mapping, survey.mapping.json beside the TSV (convention:lstsvMapping; the CLI writes it beside -o whenever the TSV renames anything). lstsv2ddi (with --data) and lstsv2xlsform read it, from beside the TSV or --mapping <file>, and give back the XLSForm's names and codes in the codebook, the data and the form. One XLSForm then serves Kobo and LimeSurvey without renaming it for either. validate --sanitize checks a form for this.

The name and code limits are LimeSurvey's. For DDI, check with validateSubset(survey, choices, { target: 'ddi' }) (CLI: validate --target ddi; xlsform2ddi does it by default): same rules without those limits, since DDI keeps names as authored. xlsformToDdi runs this check itself (skipValidation turns it off).

LimeSurvey's reverse-subset check (lstsv2xlsform) is narrower — no arrays, no ranking, no numeric/date expressions.

Development

# Install dependencies
npm install
uv sync

# Run tests
npm test           # TypeScript tests
uv run pytest      # Python tests
npm run test:live  # Docker integration tests

# Generate artifacts from registry
uv run codegen

# Bless snapshots after registry changes
npm run bless

Releasing

1. Bump version in package.json in a PR and merge it.
2. Publish a GitHub release tagged v<version> on that commit.

Publishing the tag builds the schematron-worker image (worker-image.yml). Publishing the release attaches the package tarball, the skill and fixture archives and SHA256SUMS (release-assets.yml), then opens the consumer bump PRs (below). That job fails if the tag and package.json disagree, and it never replaces an asset that already exists: consumers pin the checksums, so a fix ships as a new version.

Consumers pick up a release in one step

Three repos pin a formtransform release. After the assets are attached, bump-consumers.yml opens one PR in each, moving every pin to the new version; the consumer reviews and merges it, and its CI runs on the PR:

Consumer	Pins the PR updates
qwacback	.registry-version; docker-compose.yml schematron-worker image tag (the job waits until worker-image.yml has pushed it); ddi-emitter/package.json tarball URL + package-lock.json (via npm install --package-lock-only); then runs its scripts/check-registry-version.sh
formulaid	package.json tarball URL + bun.lock; scripts/build_skill.sh FORMTRANSFORM_VERSION and FORMTRANSFORM_SKILL_SHA256 (cdl-survey-types-<version>.tar.gz)
formtransform-app	package.json tarball URL + bun.lock; tests/formtransform-fixtures.json (version, url, sha256 of formtransform-fixtures-<version>.tar.gz)

qwac pins nothing of formtransform's (it talks to qwacback), so it gets no PR.

Checksums come from the release's SHA256SUMS; lockfiles are written by the consumer's own package manager. The pins are listed in one place, scripts/bump-consumers.mjs, which fails rather than skip a pin it can't find. When a consumer moves or adds a pin, update the list there (and tests/ts/unit/bumpConsumers.test.ts).

Setup: the workflow needs a repository secret CONSUMER_BUMP_TOKEN, a fine-grained token (or a GitHub App's) with Contents and Pull requests read/write on the three repos; GITHUB_TOKEN can't push to other repos. Without it the job only prints the manual command. Re-run it for a released tag from the Actions tab (Bump consumers, workflow_dispatch); an open PR for the same version is updated, not duplicated.

By hand, in a clean clone of a consumer:

node scripts/bump-consumers.mjs qwacback 0.8.1 ../qwacback   # or formulaid, formtransform-app

It rewrites the files and updates the lockfile (needs npm or bun), and never commits.

Documentation

• Architecture — Technical architecture and internal structure
• Response data — Kobo/LimeSurvey export → DDI codebook + data CSV (CLI and browser), and reading the codebook back in Python
• survey2ddi handover — plan for retiring survey2ddi in favour of this library (lives in survey2ddi)
• Question-bank alignment: tracked in qwac (#9, #10, #12, #13) and qwacback (#3 converter, #4 validation worker)
• Website content (umfragen.civic-data.de) that advertises these tools and feeds agent-readable XLSForm docs: tracked in cdl-wp-eins #26–#31
• Claude Code Integration — Claude-specific features and skills
• Pipeline Documentation — Transformation pipeline details
• Test Documentation — Test structure and running tests

AI Disclosure

See AI_DISCLOSURE.md.

License

MIT