Feat/databox v2 support

.claude/rules/api-client.md

@@ -0,0 +1,49 @@
+---
+paths:
+  - src/lib/api-client.ts
+  - src/base-command.ts
+---
+
+# API Client Contract
+
+## BaseCommand
+
+`BaseCommand<T>` provides:
+- `apiClient` — built in `init()` from config or flag overrides, so a missing key or unreadable config fails (exit 1) before `run()` and before any confirmation prompt.
+- `requireNumericId(value, name)` — validates string is all digits, errors with exit 2.
+- `requireUuid(value, name)` — validates a UUID, errors with exit 2. `init()` applies it to `--idempotency-key` before `run()`.
+- `requireMetricId(value, name)` — metric keys are opaque strings, so it rejects only what would rewrite the path: empty, blank, `.` and `..` (exit 2). Pair it with `encodeURIComponent`.
+- `accountHeaders` getter — returns `{'x-account-id': id}` if `--account-id` set, else `{}`. `init()` validates `--account-id` as numeric first.
+- `outputFormat` getter — `'table' | 'json' | 'csv'`; `--json` resolves to `'json'`.
+- `color` getter — false under `--no-color` or a non-empty `NO_COLOR`. The CLI's own output has no colour; any colour added must check it.
+- `catch()` — renders `ApiRequestError` via `describeApiError` (exit 1) and `ApiConnectionError` (exit 2) before oclif's handler prints them.
+- Base flags: `--output table|json|csv` (default `table`), `--json` (shorthand, exclusive with `--output`), `--verbose`, `--no-color`, `--api-key` (hidden), `--api-url` (hidden), `--account-id` (hidden).
+
+`auth login` extends `Command` directly (not `BaseCommand`) because it works without an existing API key.
+
+## ApiClient
+
+- Wraps native `fetch()` — no external HTTP dependencies.
+- Methods: `get<T>`, `post<T>`, `patch<T>`, `put<T>`, `delete<T>` — all generic.
+- Unwraps V2 envelope: returns `response.data`, not the full `{data, requestId, status}`.
+- Auth: `x-api-key` header on every request. Content-Type set only when body is present.
+- Errors: parses `errors[]` from the error envelope, throws `ApiRequestError(message, status, errors, requestId)`.
+  `code`, `field` and `type` are taken from the first error. A non-JSON or `null` body falls back to `API error: <status> <statusText>`.
+- Transport failures — `fetch()` rejecting, a timeout, or the body stream failing mid-read — throw `ApiConnectionError` (exit 2).
+- Redirects are refused (`redirect: 'error'`, in `ask-genie.ts` too): fetch strips only `Authorization` and `Cookie` on a
+  cross-origin redirect, so a followed one would resend `x-api-key`. The refusal is an `ApiConnectionError` (exit 2) with its own message.
+- `--verbose`: the `trace` option receives lines built only by `describeRequest(method, url)` and
+  `describeResponse(status, ms, requestId)`. Neither takes the headers, so the key is never in scope;
+  the header line is the literal `Headers: x-api-key: <redacted>`. Trace goes to stderr.
+
+## Rules
+
+- Never add external HTTP dependencies (axios, got, node-fetch) — use native `fetch()`.
+- API error detail (code, message, field, request ID) is printed by `BaseCommand.catch` in the cli.md §8 layout. Commands must not catch API errors themselves — that loses the detail and the exit code.
+  - The one sanctioned exception: a command may override `catch()` to append a hint to one specific API error code.
+    It re-renders that error with `describeApiError(error)` plus the hint line, passes it to `super.catch` as a
+    `CLIError` with exit 1, and delegates every other error to `super.catch` unchanged. No try/catch goes around the
+    API call itself. See `src/commands/user/invite.ts` (`duplicate_record` → a pointer to `user update`).
+- Never pass headers, or anything holding the key, into a trace or log line. Build such lines from explicit printable values.
+- Never bypass `ApiClient` for API calls unless the protocol requires it (SSE streaming).
+- The `apiKey` property must not appear in any `this.log()`, `console.log()`, or error output.

.claude/rules/commands.md

@@ -0,0 +1,124 @@
+---
+paths:
+  - src/commands/**
+  - src/lib/output.ts
+---
+
+# Command Conventions
+
+## Structure
+
+Every command extends `BaseCommand<T>` (exception: `auth login` extends `Command` directly).
+Static members are ordered alphabetically: `args`, `description`, `examples`, `flags`.
+
+```typescript
+// Good
+export default class DatasetGet extends BaseCommand<typeof DatasetGet> {
+  static args = { ... }
+  static description = 'Get details of a specific dataset'
+  static examples = [ ... ]
+  static flags = { ... }
+  async run(): Promise<void> { ... }
+}
+
+// Bad — wrong order, missing examples
+export default class DatasetGet extends BaseCommand<typeof DatasetGet> {
+  static description = '...'
+  static flags = { ... }
+  static args = { ... }
+  async run(): Promise<void> { ... }
+}
+```
+
+## Flags and arguments
+
+- Flag names: kebab-case (`page-size`, `data-source-id`). Body/query params: camelCase (`pageSize`, `dataSourceId`).
+- Use `Flags.string()`, `Flags.integer()`, `Flags.boolean()` — match the data type.
+- `required: true` on mandatory flags, `options: [...]` for enums, `exclusive: [...]` for mutual exclusion.
+- Shared flags from `src/lib/flags.ts`:
+  - `...paginationFlags` (`--page`, `--page-size` max 100, `--all`) on list endpoints; `...dataPaginationFlags` (`--page-size` max 1000) on the row-data endpoints (`dataset data`, `metric drilldown`).
+  - `...sortFlags(options)` — pass the sort fields the service validates; `sortFlags()` leaves `--sort-by` free.
+  - `...idempotencyFlags` on exactly the routes ingestion-api marks `[IdempotencyFilter]`, sending `{...this.accountHeaders, ...idempotencyHeaders(this.flags)}`.
+- Boolean flags: `default: false`.
+- Examples use `<%= config.bin %>` template, never hardcoded `databox`. At least 2 examples per command.
+- Args use `Args.string({ required: true })` — even numeric IDs are accepted as strings and validated later.
+
+Never drop `--flag ""` silently: the user typed it deliberately. Where the API validates the field, send the empty
+value (guard with `!== undefined`, never truthiness) and let the API reject it with a message; where the API would
+accept it silently, reject it locally with exit 2. A required name flag rejects blank locally unless the API does
+(the create commands check it anyway, to fail before the round trip). `runCommand` refuses an empty-string flag
+value, so the regression test for `--flag ""` belongs in `test/e2e/`.
+
+```typescript
+// Good — the API validates timezone, so `--timezone ""` reaches it and is rejected with a message
+if (flags.timezone !== undefined) body.timezone = flags.timezone
+// Good — the API would accept a blank integration key, so the CLI refuses it
+if (flags['integration-key'] !== undefined && flags['integration-key'].trim() === '') {
+  this.error('--integration-key cannot be empty.', {exit: 2})
+}
+
+// Bad — `--timezone ""` is dropped and the data source silently gets the default zone
+if (flags.timezone) body.timezone = flags.timezone
+```
+
+## Output by command type
+
+| Type | Output | Functions |
+|---|---|---|
+| List | Table + pagination | `formatOutput(data, columns, this.outputFormat)` + `showPagination(pagination, this.outputFormat)` |
+| Get / Create / Update | Single record | `formatSingle(data, this.outputFormat)` |
+| Delete / Purge / Clear | Confirmation message | `this.log('Resource ID action.')` |
+| Set (permissions, timezone) | Confirmation message or single record | `this.log()` or `formatSingle()` |
+
+Always pass `this.outputFormat`, never `this.flags.json`: it also covers `--output json|csv`.
+
+`--json` returns what the endpoint returned:
+
+- A pure `{items}` or `{items, pagination}` list unwraps to a bare array of the items, passed through whole.
+- A response whose siblings of `items` carry data returns the whole object: `dataset schema` `{items, primaryKey}`,
+  `dataset data` `{items, pagination, schema, lastUpdatedAt}`, `dataset preview-modification` `{items, pagination, schema}`.
+  Branch on `this.outputFormat === 'json'` and print it with `formatSingle(response, this.outputFormat)`.
+- A mutation that returns the resource (`set-timezone`, `set-sync-frequency`, `set-verification`) prints it with
+  `formatSingle` under json and csv, and keeps its confirmation line in table mode.
+- Lines that only make sense beside a table (a primary key, a "rows matched" count) are printed in table mode only,
+  never in CSV, so the stream stays parseable.
+
+List commands fetch through `fetchPaginated`, which implements `--all` and reports a short result on stderr:
+
+```typescript
+const response = await fetchPaginated(this.flags, query, pageQuery =>
+  this.apiClient.get<ListResponse>('/v2/resources', pageQuery, this.accountHeaders), warning => this.warn(warning))
+```
+
+## Destructive operations
+
+Delete, purge, and clear commands require:
+1. `--force` flag with `default: false`
+2. `confirm()` from `../../lib/prompt.js` when not forced
+3. `this.log('Aborted.')` when user declines
+4. Success message: `"Resource ID past-tense."` (e.g., `"Dataset 123 deleted."`)
+
+Off a terminal, `confirm()` writes its question to stderr and reads the answer from the first line
+of stdin: `y`/`yes` proceeds, anything else is declined (`Aborted.`, exit 0), and an empty first line
+or a stdin that ends with nothing throws exit 2 ("Refusing to prompt: … Pass --force to confirm.").
+Commands need no handling of their own.
+
+## Error codes
+
+- `this.error(msg, {exit: 1})` — general errors (missing auth, API failures)
+- `this.error(msg, {exit: 2})` — input validation errors (`requireNumericId`)
+
+## API calls
+
+- Always pass `this.accountHeaders` as the last argument to `apiClient.get/post/patch/put/delete`.
+- No try/catch around API calls — errors propagate to `BaseCommand.catch`, which prints their detail and sets the exit code.
+- No direct `fetch()` calls (exception: `ask-genie.ts` for SSE streaming).
+
+## Update commands
+
+Update commands with optional flags must guard against empty bodies:
+```typescript
+if (Object.keys(body).length === 0) {
+  this.error('Provide at least one field to update (--name or --title).', {exit: 1})
+}
+```

.claude/rules/e2e-testing.md

@@ -0,0 +1,111 @@
+# E2E Testing Conventions
+
+End-to-end suites live in `test/e2e/` and spawn the **built** CLI against a **real** API.
+They are separate from the mocked unit suite in `test/commands/`. Full usage:
+`test/e2e/README.md`.
+
+## Boundaries
+
+| | Unit (`test/commands/*.test.ts`) | E2E (`test/e2e/*.e2e.ts`) |
+|---|---|---|
+| Runs | `runCommand` in-process | `node bin/run.js` as a child process |
+| API | `global.fetch` mocked | Real, over the network |
+| Asserts | stdout string contents | exit code + stdout/stderr |
+| Command | `npm test` | `npm run test:e2e` |
+
+The `.e2e.ts` suffix is what keeps them apart — `npm test`'s glob is `test/**/*.test.ts`.
+Never rename an e2e file to `.test.ts`, and never add `test/e2e` to `.mocharc.yml`.
+
+## Rules
+
+- **Everything goes through the CLI.** No suite makes a direct HTTP call. Setup,
+  assertions and teardown all use `cli()`. The e2e layer has no API client of its own,
+  so it cannot drift from the one under test.
+- **One file per command group**, named for the group: `test/e2e/<group>.e2e.ts`.
+- **Name every created resource** with `e2eName(label)` and register it on a
+  `ResourceTracker`; tear it down in the suite's `after()`.
+- **Always pass `--force`** to destructive commands. Child stdin is `'ignore'`, so an
+  unforced `confirm()` exits 2 at once ("Refusing to prompt: stdin is not a terminal") and
+  the command does nothing.
+- **Match error text with `errorText(result)`**, never `result.stderr` directly — the
+  CLI hard-wraps messages, so a phrase can be split across lines with padding.
+- **Never add mocha `--retries` or `--parallel`.** Mocha retries re-run the whole test,
+  creating resources twice; parallel suites collide on shared organization state. Retry belongs
+  in `retryRead` (poll a read until the API's cache catches up) and `cliWithRetry` (re-run
+  a command whose failure matches a transient environment fault). `cliWithRetry` is safe
+  for assertions too — a genuine failure does not match a transient pattern, and a matched
+  one is still returned once attempts run out. Shared dev environments do fail in bursts,
+  including spurious 401s on a valid key.
+- **No API key in the repository** — this repo is public; keys come from
+  `DATABOX_E2E_API_KEY` only.
+- **Never mutate a resource the suite did not create without `withRestore()`.** It
+  records the undo on disk before the change, so an interrupted run can be repaired
+  with `npm run test:e2e:cleanup`. A bare `finally` does not survive Ctrl-C.
+- **A command rename or removal sweeps `test/e2e/` too.** `npm test` never loads e2e files,
+  so a stale `cli(['old-topic', 'cmd'])` stays green until the live suite runs.
+  `test/e2e-commands.test.ts` checks every e2e argv against `src/commands/`; a deliberate
+  negative test goes on its allowlist.
+
+## Classifying a failure
+
+A failing e2e test means one of three things. Say which, in the test:
+
+1. **A CLI defect** — **fix the command.** A skipped test is green and CI cannot tell it
+   from a passing one, so parking a known-broken command as a skip makes the suite lie.
+   Only if the fix is genuinely deferred, use `it.skip` with `[BROKEN: <what>]` in the
+   title plus a comment giving the API's real contract, the observed symptom, and the
+   source file that fixes it — and keep that list short.
+2. **An environment outage** — `this.skip()` at runtime via `serviceUnavailable(result)`,
+   which recognises the API's own 5xx/service-down messages. Never hard-code an outage
+   as expected behaviour.
+3. **A capability the organization lacks** — `this.skip()` with a logged reason: it manages
+   no accounts, no Advanced Security add-on, no databoards, no connections.
+
+A skip must always print or carry its reason. A silent skip is worse than a failure.
+
+## Verifying a suspected CLI defect
+
+Before marking anything `[BROKEN]`, confirm it against the raw endpoint with `curl`,
+so the report distinguishes a CLI bug from an API one. Two defect families found so far,
+both invisible to the unit suite:
+
+- **Envelope drift** — a command reads `response.items` where the API returns a bare
+  array, or hands `{items: […]}` straight to `formatOutput`. The unit mock defines the
+  shape, so it always agrees with itself. Only a real response settles it.
+- **Request-body drift** — a command sends a field name the API does not accept
+  (`interval` vs `syncInterval`, `status` vs `isVerified`, `tags` vs `synonyms`). These
+  commands could never succeed, and every one of them had a passing unit test.
+
+**Any command that sends a request body needs both**: an e2e test, and a unit test
+asserting the body via `lastBody(method, path)` from `test/helpers.ts`. The unit
+assertion is the cheap guard that runs on every `npm test`; the e2e test is what proves
+the field names are the ones the API actually wants.
+
+```typescript
+it('sends syncInterval, not interval', async () => {
+  await runCommand(['dataset', 'set-sync-frequency', '123', '--interval', '60'], {root: process.cwd()})
+  expect(lastBody('PUT', '/v2/datasets/123/sync-frequency')).to.deep.equal({syncInterval: 60})
+})
+```
+
+Note that `@oclif/test`'s `runCommand` refuses an **empty-string** flag value even though
+the real binary accepts one, so assertions about clearing a nullable field belong in the
+e2e suite.
+
+## The API is the source of truth
+
+`ingestion-api` defines the contract; the CLI follows it. Concretely:
+
+- A command exposes **every field** of its request contract in
+  `IngestionApi.Core/Contracts/Request/V2/`, and every query parameter its controller
+  action declares.
+- `--json` returns **what the endpoint returned**. Do not hand-pick a subset into a new
+  object, and do not reshape values. List commands unwrap `response.items` to a bare
+  array — that is the one established convention — but the item objects pass through whole.
+- Optional string fields are guarded with `!== undefined`, never truthiness, so an empty
+  string can clear a nullable field.
+- Response interfaces mirror the response contract, including inherited members (a
+  `…Detail` type extends its `…ListItem`).
+
+Re-derive the mapping from the API source rather than from memory or from the existing
+CLI code, and confirm anything surprising against a live endpoint with `curl`.

.claude/rules/security.md

@@ -0,0 +1,44 @@
+---
+paths:
+  - src/**
+---
+
+# Security Rules
+
+## API key handling
+
+- `apiClient.apiKey` must never appear in `this.log()`, `console.log()`, or error messages.
+- The only legitimate direct access to `apiKey` is in `ask-genie.ts` for the SSE `x-api-key` header.
+- Hidden flags (`--api-key`, `--api-url`, `--account-id`) must remain `hidden: true` — they do not appear in help output.
+
+## Config file
+
+- Config lives at `~/.config/databox-cli/config.json` with `apiKey` and optional `apiUrl`.
+- Never log config file contents. Never weaken file permissions.
+
+## URL interpolation
+
+- User-provided IDs are interpolated into API paths: `/v2/datasets/${args.datasetId}`.
+- Numeric IDs validate with `requireNumericId()`, UUIDs (ingestion IDs) with `requireUuid()` — both safe.
+- Metric keys are opaque strings: `requireMetricId()` plus `encodeURIComponent`. Encoding alone is not enough — it leaves `.`/`..` intact and `new URL()` resolves them, so `metric delete ..` would send `DELETE /v2/`.
+- Every command that interpolates user input into a URL path validates it first.
+
+```typescript
+// Safe — validated
+this.requireNumericId(args.datasetId, 'Dataset ID')
+const response = await this.apiClient.get(`/v2/datasets/${args.datasetId}`, ...)
+
+// Risk — unvalidated
+const response = await this.apiClient.get(`/v2/connections/${args.connectionId}`, ...)
+```
+
+## Input handling
+
+- `JSON.parse()` on user-provided flag values (--schema, --records, --data) must be wrapped in try/catch with a user-friendly error message. A bare `JSON.parse` leaks a raw `SyntaxError`.
+- `fs.readFileSync` on user-provided paths (--file flag) should check file existence first.
+- No unbounded stdin reads without size limits in new commands.
+
+## Error messages
+
+- Do not include request headers, full URLs, or API keys in error output.
+- `ApiRequestError` messages are displayed to the user — they come from the server and are acceptable.