fix(modifiers): read a stored field with no value as null at the end of the read

docs/3.table-modifiers.md

@@ -198,6 +198,8 @@ const german = product.localize("de");
 console.log(german.name); // "Premium Headphones" (fallback)
 ```
 
+A field without a `fallbackLocale` reads `null` in a locale that has no text. A field the row does not store at all reads `undefined`. An empty text `""` or `null` assigned in a locale is stored as no text, so saving a row read with no text stores nothing and the fallback still answers.
+
 #### Wildcard key
 
 The wildcard key `"*"` reads or writes all translations at once. List the localized fields with it, and `localize("*", fields)` types each listed field as its values by locale, a `LocalizedRecords<T, K>`: a field declared as `string` reads as a `Record<string, string>`. Calling `localize()` on that view with one locale gives back the row type. The view is not the row type, so pass the row itself to `insert()` or `update()`. Write all translations with `Object.assign`, which calls the field's setter. `undefined` or `null` clears every translation. Any other value that is not a record, such as a single text, throws a `TypeError` and leaves the stored translations as they were:
@@ -368,7 +370,7 @@ The generic parameters of `OneWayModifier<LockedType, Args, Meta, Options>` are:
 
 ### `TwoWayModifier`
 
-A `TwoWayModifier` extends `OneWayModifier` with an `unlock` method, for a transformation that can be reversed, such as encryption:
+A `TwoWayModifier` extends `OneWayModifier` with an `unlock` method, for a transformation that can be reversed, such as encryption. A read never passes `undefined` or `null` to `unlock`: a field that stores `null` reads `null`, and a field the row stores reads `null` when its modifiers find no value in it:
 
 ```ts docs-check
 import {
@@ -422,11 +424,11 @@ const Base64Encoded = MakePropertyDecorator(
 );
 ```
 
-Base64 is an encoding, not encryption. The example declares its `LockedType` as `unknown`, so `unlockrequest()` can return the stored value's proxy as it is. Database expressions therefore see the encoded value, and only instance reads decode it. `unlock` reads a stored value that is not a string as `undefined`.
+Base64 is an encoding, not encryption. The example declares its `LockedType` as `unknown`, so `unlockrequest()` can return the stored value's proxy as it is. Database expressions therefore see the encoded value, and only instance reads decode it. `unlock` reads a stored value that is not a string as `undefined`, so the field reads `null`: a field the row stores reads `null` when its modifiers find no value.
 
 ### `ContainerModifier`
 
-A modifier that stores several values under keys, as `LocalizationModifier` does, extends `ContainerModifier`. Its `lock`, `unlock`, and `unlockrequest` take the key as their argument, a `string` or `undefined`. `lock` and `unlock` treat the wildcard key `"*"` as the whole record, and `lock` throws a `TypeError` for a value under `"*"` that is not a record. `undefined` is no key: `lock` stores nothing under it, and `unlock` and `unlockrequest` find no value, so a `LocalizationModifier` answers its `fallbackLocale`, or `null` when that locale has no value either:
+A modifier that stores several values under keys, as `LocalizationModifier` does, extends `ContainerModifier`. Its `lock`, `unlock`, and `unlockrequest` take the key as their argument, a `string` or `undefined`. `lock` and `unlock` treat the wildcard key `"*"` as the whole record, and `lock` throws a `TypeError` for a value under `"*"` that is not a record. `undefined` is no key: `lock` stores nothing under it, and `unlock` and `unlockrequest` find no value, so a `LocalizationModifier` answers its `fallbackLocale`, and the field reads `null` when that locale has no value either. The modifier chain passes no `null` to a modifier stacked after it:
 
 ```typescript
 import { ContainerModifier } from "@antelopejs/interface-database-decorators";

skills/database-decorators-interface/SKILL.md

@@ -133,8 +133,9 @@ class Chef extends Table.with(
 - In plain data, a plain object given for a localized field is its texts by locale: `fromPlainData` and `insert` store every locale, whatever `localize()` call follows, and `update` replaces the stored texts with that record. A single text still waits for a locale. A field with another modifier inside its locales, such as `@Localized() @Encrypted()`, does not read a record this way.
 - `instance.localize("*", ["motto"])` reads and writes every locale at once, typed `Record<string, string>` for each listed field (`LocalizedRecords<T, K>`); without a field list the field types stay unchanged.
 - Under `localize("*")` a localized field takes the record of texts by locale, and `undefined` or `null` clears every locale. A single text throws a `TypeError`, whether it is assigned or held from `fromPlainData` when `localize("*")` runs.
-- A localized value `""` is stored as no value: reads and queries in that locale get the `fallbackLocale` value.
+- A localized value `""` or `null` is stored as no value: reads and queries in that locale get the `fallbackLocale` value. A row read with no text answers `null`, and saving that answer again stores nothing.
 - A `ContainerModifier` key of `undefined` is no key: writes store nothing under it, and reads and queries get no value. A localized field reads the `fallbackLocale` value, or `null` when there is none.
+- A modified field the row stores reads `null` when its modifiers find no value, such as `localize("de")` with no German text and no `fallbackLocale`. A field the row does not store reads `undefined`. A field that stores `null` reads `null`. A custom `unlock` never receives `undefined` or `null`.
 - `@Relation({ to: () => Target })` is declarative metadata only: no foreign-key enforcement, consumed by introspection tooling via `RelationStaticMetadata`.
 - `Model.validate` (and `{ validate: true }` on insert/update) only checks fields whose `@Field` type is an
   io-ts-style codec with `.decode`; plain string field tokens are skipped. Update validates with

src/modifiers/common.ts

@@ -64,6 +64,9 @@ export type ModifierEvent = (
   ...args: unknown[]
 ) => void;
 
+/** The stored values no modifier unlocks: a field holding one reads `null`. */
+const NO_VALUE = new Set<unknown>([undefined, null]);
+
 interface ModifiedField {
   id: string;
   modifier: Modifier;
@@ -136,6 +139,12 @@ export class ModifiersDynamicMetadata {
     );
   }
 
+  /**
+   * Reads a field through its modifiers. A field the row stores reads `null`
+   * when its modifiers find no value in it, as under a locale with no text,
+   * or when it stores `null`: no modifier unlocks `null`. A field the row
+   * does not store, or cannot read yet, reads `undefined`.
+   */
   get(field: PropertyKey, internal: InternalRecord) {
     if (Object.hasOwn(this.floating, field)) {
       return floatingByKey(this)[field];
@@ -144,9 +153,10 @@ export class ModifiersDynamicMetadata {
       return;
     }
     const list = fieldsByKey(this.staticMeta)[field];
-    let value = dataByKey(internal)[field];
+    const stored = dataByKey(internal)[field];
+    let value = stored;
     for (const { id, modifier, metaRef } of list) {
-      if (isTwoWay(modifier) && value !== undefined) {
+      if (isTwoWay(modifier) && !NO_VALUE.has(value)) {
         if (unlockArgsOf(this, field)[id]) {
           value = withRef(internal, id, modifier, metaRef, (modifier) =>
             modifier.unlock(value, ...unlockArgsOf(this, field)[id]),
@@ -158,7 +168,7 @@ export class ModifiersDynamicMetadata {
         }
       }
     }
-    return value;
+    return value === undefined && stored !== undefined ? null : value;
   }
 
   canSet(field: PropertyKey) {
@@ -190,11 +200,10 @@ export class ModifiersDynamicMetadata {
       const { id, modifier, metaRef } = list[i];
       prevlist[i] = { id, modifier, metaRef, prev };
       if (isTwoWay(modifier)) {
-        if (unlockArgsOf(this, field)[id]) {
-          prev = modifier.unlock(prev, ...unlockArgsOf(this, field)[id]);
-        } else {
-          prev = undefined;
-        }
+        prev =
+          unlockArgsOf(this, field)[id] && !NO_VALUE.has(prev)
+            ? modifier.unlock(prev, ...unlockArgsOf(this, field)[id])
+            : undefined;
       }
     }
     prevlist.push({
@@ -507,7 +516,7 @@ export function testValue<T extends { constructor: any }, K extends keyof T>(
         modifier.test(val, value, ...args),
       );
     }
-    if (isTwoWay(modifier) && val !== undefined) {
+    if (isTwoWay(modifier) && !NO_VALUE.has(val)) {
       return withRef(internal, id, modifier, metaRef, (modifier) =>
         modifier.unlock(val, ...args),
       );

src/modifiers/encryption.ts

@@ -152,15 +152,7 @@ export class EncryptionModifier extends TwoWayModifier<
 
   readonly autounlock = true;
 
-  /**
-   * Decrypts a stored value. `null` is no value and stays `null`, as a
-   * localized field read with no locale gives it.
-   */
-  public override unlock(locked: LockedType | null) {
-    if (locked === null) {
-      return null;
-    }
-    const [locked_value, iv_str, authTag] = locked;
+  public override unlock([locked_value, iv_str, authTag]: LockedType) {
     const decipher = createDecipheriv(
       this.options.algorithm || "aes-256-gcm",
       this.options.secretKey,

src/modifiers/localization.ts

@@ -39,12 +39,15 @@ export type LocalizedRecords<This, K extends keyof This> = Omit<
     localize(locale: string, fields?: ReadonlyArray<keyof This>): This;
   };
 
-/** A text left empty in one language counts as no text in that language. */
-const EMPTY_TEXT = "";
+/**
+ * The values that count as no text in one language: an empty text, and the
+ * `null` a read with no text answers.
+ */
+const NO_TEXT = new Set<unknown>([undefined, null, ""]);
 
-function withoutEmptyTexts<V>(values: Record<string, V>): Record<string, V> {
+function withoutMissingTexts<V>(values: Record<string, V>): Record<string, V> {
   return Object.fromEntries(
-    Object.entries(values).filter(([, value]) => value !== EMPTY_TEXT),
+    Object.entries(values).filter(([, value]) => !NO_TEXT.has(value)),
   );
 }
 
@@ -89,8 +92,9 @@ export class LocalizationModifier extends ContainerModifier<
   }
 
   /**
-   * Stores an empty text as no text, so that the fallback locale answers for
-   * it in queries too (`unlockrequest`).
+   * Stores an empty text or `null` as no text, so that the fallback locale
+   * answers for it in queries too (`unlockrequest`). A read with no text
+   * answers `null`, so saving that answer again stores nothing.
    */
   public override lock(
     // oxlint-disable-next-line anti-slop/no-unsafe-dictionary-type -- the container modifier leaves the values by locale untyped
@@ -99,26 +103,22 @@ export class LocalizationModifier extends ContainerModifier<
     value: unknown,
     key: string | undefined,
   ) {
-    return withoutEmptyTexts(super.lock(locked_value, value, key));
+    return withoutMissingTexts(super.lock(locked_value, value, key));
   }
 
-  /**
-   * The value under `key`, else under the fallback locale. With no `key`, a
-   * field without a fallback value reads `null`, as an unset field does.
-   */
+  /** The value under `key`, else under the fallback locale. */
   public override unlock(
     // oxlint-disable-next-line anti-slop/no-unsafe-dictionary-type -- the container modifier leaves the values by locale untyped
     locked_value: Record<string, unknown>,
     key: string | undefined,
   ) {
     const unlocked = super.unlock(locked_value, key);
-    if (unlocked !== undefined && unlocked !== EMPTY_TEXT) {
+    if (!NO_TEXT.has(unlocked)) {
       return unlocked;
     }
-    const fallback = this.options.fallbackLocale
-      ? super.unlock(locked_value, this.options.fallbackLocale)
-      : undefined;
-    return key === undefined ? (fallback ?? null) : fallback;
+    if (this.options.fallbackLocale) {
+      return super.unlock(locked_value, this.options.fallbackLocale);
+    }
   }
 
   /**

src/tests/end_of_read_null.test.ts

@@ -0,0 +1,130 @@
+import { expect } from "chai";
+import { MakePropertyDecorator } from "@antelopejs/interface-core/decorators";
+import { Field, Table } from "../table";
+import {
+  attachModifier,
+  fromDatabase,
+  fromPlainData,
+  lock,
+  MixinSymbol,
+  toDatabase,
+  toPlainData,
+  TwoWayModifier,
+  unlock,
+} from "../modifiers/common";
+import { LocalizationModifier, Localized } from "../modifiers/localization";
+
+class WrapModifier extends TwoWayModifier<[text: string]> {
+  readonly autolock = true;
+  readonly autounlock = true;
+
+  public override lock(_locked: [text: string] | undefined, value: string) {
+    return [value] satisfies [text: string];
+  }
+
+  public override unlock([text]: [text: string]) {
+    return text;
+  }
+
+  [MixinSymbol] = class {};
+}
+
+const Wrapped = MakePropertyDecorator((target, propertyKey) => {
+  attachModifier(target.constructor, WrapModifier, propertyKey, {});
+});
+
+describe("Modified fields - a read that finds no value", () => {
+  class Page extends Table.with(LocalizationModifier, WrapModifier) {
+    @Localized() @Wrapped() @Field("string") declare title?: string;
+    @Localized() @Field("string") declare summary?: string;
+  }
+
+  function pageInEnglish() {
+    const stored = {
+      _id: "page-1",
+      _internal: {
+        meta: {},
+        data: { title: { en: ["Hello"] }, summary: { en: "Greeting" } },
+      },
+    };
+    return fromDatabase(stored, Page).localize("en");
+  }
+
+  it("never passes null to a two-way modifier stacked after @Localized()", () => {
+    const page = pageInEnglish();
+    unlock(page, LocalizationModifier, undefined, undefined);
+
+    expect(page.title).to.equal(null);
+    expect(JSON.parse(JSON.stringify(page))).to.deep.include({ title: null });
+  });
+
+  it("reads null for a locale with no value and no fallback", () => {
+    const page = pageInEnglish();
+    const english = { title: page.title, summary: page.summary };
+    page.localize("de");
+
+    expect(english).to.deep.equal({ title: "Hello", summary: "Greeting" });
+    expect(page.summary).to.equal(null);
+    expect(page.title).to.equal(null);
+  });
+
+  it("reads undefined for a field the row does not store", () => {
+    const page = new Page().localize("en");
+
+    expect(page.summary).to.equal(undefined);
+  });
+
+  it("takes a new value on a stacked field that stores nothing or null", () => {
+    const fresh = new Page().localize("en");
+    const cleared = fromDatabase(
+      { _id: "page-3", _internal: { meta: {}, data: { title: null } } },
+      Page,
+    ).localize("en");
+
+    fresh.title = "Hello";
+    cleared.title = "Hello";
+
+    expect(fresh.title).to.equal("Hello");
+    expect(cleared.title).to.equal("Hello");
+    expect(toDatabase(cleared)._internal?.data.title).to.deep.equal({
+      en: ["Hello"],
+    });
+  });
+
+  describe("written back", () => {
+    class Note extends Table.with(LocalizationModifier) {
+      @Localized({ fallbackLocale: "fr" })
+      @Field("string")
+      declare title?: string;
+    }
+
+    it("keeps the fallback after a read answer is saved again", () => {
+      const stored = {
+        _id: "note-1",
+        _internal: { meta: {}, data: { title: {} } },
+      };
+      const read = fromDatabase(stored, Note).localize("en");
+      const resent = fromPlainData(toPlainData(read), Note).localize("en");
+      lock(resent, LocalizationModifier, undefined, "en");
+      const saved = fromDatabase(toDatabase(resent), Note).localize("fr");
+      saved.title = "Salut";
+
+      expect(read.title).to.equal(null);
+      expect(saved.localize("en").title).to.equal("Salut");
+    });
+
+    it("stores no text when lock writes back a read with no text", () => {
+      const stored = {
+        _id: "page-2",
+        _internal: { meta: {}, data: { summary: { fr: "Salut" } } },
+      };
+      const page = fromDatabase(stored, Page);
+      unlock(page, LocalizationModifier, ["summary"], "en");
+      lock(page, LocalizationModifier, ["summary"], "en");
+
+      expect(toDatabase(page)._internal?.data.summary).to.deep.equal({
+        fr: "Salut",
+      });
+    });
+  });
+});

src/tests/localized_plain_data.test.ts

@@ -70,14 +70,15 @@ describe("Localized fields - plain data with values by locale", () => {
     expect(getInternal(read).data.displayName).to.deep.equal({ en: "Members" });
   });
 
-  it("stores a null value under the locale it is read with", async () => {
+  it("stores a null value as no text under the locale it is read with", async () => {
     const read = await stored(
       RoleModel.fromPlainData({ displayName: wording, tooltip: null }).localize(
         "en",
       ),
     );
 
-    expect(getInternal(read).data.tooltip).to.deep.equal({ en: null });
+    expect(getInternal(read).data.tooltip).to.deep.equal({});
+    expect(read.localize("en").tooltip).to.equal(null);
   });
 
   it("stores no record for a field encrypted inside its locales", async () => {

src/tests/modifiers/encryption.test.ts

@@ -1,6 +1,11 @@
 import { expect } from "chai";
 import { Field, Table } from "../../table";
 import { Encrypted, EncryptionModifier } from "../../modifiers/encryption";
+import {
+  attachModifier,
+  fromDatabase,
+  testValue,
+} from "../../modifiers/common";
 
 describe("Modifiers - encryption", () => {
   it("creates encryption modifier", async () => {
@@ -153,4 +158,43 @@ describe("Modifiers - encryption", () => {
     expect(instance.payload).to.deep.equal(original);
     expect(instance.payload.date).to.be.instanceOf(Date);
   });
+  it("unlocks a stored value typed from unlock's parameter, which excludes null", () => {
+    class IvLength extends EncryptionModifier {
+      public override unlock(
+        stored: Parameters<EncryptionModifier["unlock"]>[0],
+      ) {
+        return Buffer.from(stored[1], "base64").length;
+      }
+    }
+    class Note extends Table.with(IvLength) {
+      @Field("string") declare body: number | string;
+    }
+    attachModifier(Note, IvLength, "body", {
+      secretKey: "12345678901234567890123456789012",
+    });
+
+    const note = new Note();
+    note.body = "hello";
+
+    expect(note.body).to.equal(16);
+  });
+  it("reads a stored null as null, without decrypting it", () => {
+    class Vault extends Table.with(EncryptionModifier) {
+      @Encrypted({ secretKey: "12345678901234567890123456789012" })
+      @Field("string")
+      declare secret?: string | null;
+    }
+    const stored = {
+      _id: "vault-1",
+      _internal: { meta: {}, data: { secret: null } },
+    };
+
+    const vault = fromDatabase(stored, Vault);
+
+    expect(vault.secret).to.equal(null);
+    expect(testValue(vault, "secret", null)).to.equal(true);
+    expect(JSON.parse(JSON.stringify(vault))).to.deep.include({
+      secret: null,
+    });
+  });
 });