feat(merge)!: texts two sources spell differently are written once

docs/api.md

@@ -549,7 +549,7 @@ are not re-exported at all.
 | Grabber | `grab`, `defineSiteConfig`, `defineStreamSiteConfig`, `defineXmltvSite`, `defineM3uSite`, `defineXtreamSite`, `defineSchedulesDirectSite`, `schedulesDirectAccount`, `defineCommandSite`, `defineTvGrabCommandSite`, `resolveChannels`, `resolveSites`, `channelsFromM3u`, `channelsFromChannelsXml`, `guideUrlsFromM3u`, `xtreamChannelExtras`, `xtreamProgrammeExtras`, `schedulesDirectChannelExtras`, `schedulesDirectProgrammeExtras`, `SCHEDULES_DIRECT_CHANNEL_ID`, `channelElement`, `siteHttp`, `kyWith`, `sitePacing`, `retryAfterMs`, `isUnchanged`, `UnchangedError` |
 | Site state | `SiteStateHandle`, `StateKey`, `TrackedMap`, `channelsMaxAgeMs`, `DEFAULT_CHANNELS_MAX_AGE_DAYS` — see [what a site remembers](#what-a-site-remembers-between-runs) |
 | Selection | `channelSelection`, `unmatched` — what `channels` means once `derived` is taken into account, and what it asked for that nothing produced |
-| Merge | `mergeProgrammes`, `mergeProgrammeLists`, `mergeInto`, `backfillInto`, `resolveMatch`, `normalizeTitle`, `titlesMatch`, `DEFAULT_MATCH`, `generateGuide`, `writeGuide`, `defaultChannelInfo` |
+| Merge | `mergeProgrammes`, `mergeProgrammeLists`, `mergeInto`, `backfillInto`, `resolveMatch`, `normalizeTitle`, `titlesMatch`, `DEFAULT_MATCH`, `resolveUnion`, `ResolvedUnion`, `DEFAULT_UNION`, `generateGuide`, `writeGuide`, `defaultChannelInfo` |
 
 Plus the types for all of the above (`EpgConfig`, `SiteConfig`,
 `GrabberChannel`, `XmltvProgramme`, `XmltvChannel`, `GuideServer`, …), and
@@ -991,6 +991,7 @@ a `Map` that records which keys were touched so nothing untouched is rewritten.
 
 `generateGuide`, `writeGuide`, `mergeProgrammes`, `mergeProgrammeLists`,
 `mergeInto`, `backfillInto`, `resolveMatch`, `normalizeTitle`, `titlesMatch`,
+`resolveUnion`, `ResolvedUnion`, `DEFAULT_UNION`,
 `DEFAULT_MATCH`, `DEFAULT_FILL_STOP_MS`*, `mergeChannels`*,
 `defaultChannelInfo`, `channelSelection`, `unmatched`.
 
@@ -1003,15 +1004,24 @@ or when the name says so, `.gz` / `.br` / `.zst`. See
 costs on a real one. `generateGuide` yields the document itself, so a caller
 piping it somewhere compresses it however they like.
 
-`mergeProgrammeLists(lists, strategy, match?, fillStopMs?)` combines per-site
-lists in priority order; `match` is a `ProgrammeMatch` object, a predicate of
+`mergeProgrammeLists(lists, strategy, match?, fillStopMs?, union?)` combines
+per-site lists in priority order; `match` is a `ProgrammeMatch` object, a predicate of
 your own, or `{ windowMs, matches }` — a predicate with the reach it needs. A
 bare predicate is taken to reach any distance (`windowMs: Infinity`), so it is
 asked about every pair; giving it a window is what keeps a merge linear. See
 [what counts as the same
 broadcast](./configuration.md#what-counts-as-the-same-broadcast). `fillStopMs`
 is for the `backfill` strategy: how long a programme with no `stop` is taken to
-cover, `DEFAULT_FILL_STOP_MS` (six hours) unless given.
+cover, `DEFAULT_FILL_STOP_MS` (six hours) unless given. `union` is
+`merge.union`, applied to each programme of the result — see [texts that say
+the same thing](./configuration.md#texts-that-say-the-same-thing); the pieces
+below it leave every text in, and
+`resolveUnion(union, resolveMatch(match).normalizeTitle).apply(programme)` is
+what applies it to one programme, its titles compared as the match compares
+them. A loaded config carries its `merge.union` already settled, as a
+`ResolvedUnion` — whose own fields are the union it was settled from, so a
+copy of it is that union — which `resolveUnion` takes as it is for the same
+match and settles again for another.
 `mergeInto(target, incoming, resolveMatch(match))` is the piece underneath
 `mergeProgrammeLists`:
 one source's list folded into a sorted accumulator, which is also how the guide

docs/configuration.md

@@ -1632,7 +1632,7 @@ questions: **who grabs** the channel, and **how what was grabbed is combined**.
   - `first-wins` — one `<channel>`, programmes only from the first covering site. The others are still grabbed, so switching to a strategy that uses them costs no refetch
   - `keep-all` — no deduplication
 - `programmeStrategy` (for `merge-programmes`)
-  - `merge` (default) — programmes describing the same broadcast become one element; language-tagged fields (`title`, `desc`, `category`, …) are unioned by `(lang, value)` — grab the same channel from a Slovak and an English source and get both languages in one programme
+  - `merge` (default) — programmes describing the same broadcast become one element; language-tagged fields (`title`, `desc`, `category`, …) are unioned within each language — grab the same channel from a Slovak and an English source and get both languages in one programme, and two spellings of one Slovak title as one; see [texts that say the same thing](#texts-that-say-the-same-thing)
   - `concat` — keep all programmes sorted by start
   - `backfill` — the first covering site contributes everything it has, and a lower-priority one only what falls in a hole it left; nothing is combined — see [filling the gaps](#filling-the-gaps)
 
@@ -1729,6 +1729,47 @@ custom matcher stops being quick — the window is what keeps the walk short.
 merge: { match: (a, b) => … }   // correct, and linear in nothing
 ```
 
+### Texts that say the same thing
+
+Two programmes that are one broadcast become one element, and their titles,
+descriptions and categories come together within each language. Two sources
+rarely write them the same to the letter — `DJ Mixes premiere` and
+`Dj Mixes Premiere`, `Film` and `film`, a teaser cut off with `…` beside the
+whole of it — so the guide takes such texts for one and writes the
+higher-priority source's:
+
+| field | default | the same when |
+|---|---|---|
+| `title`, `subTitle` | `'match'` | the match's `normalizeTitle` makes them the same — by default case, accents and spacing aside, so a title rule that loosens what is one broadcast loosens what is one title too |
+| `desc` | `'contained'` | one contains the other as whole words, but for case, spacing and a trailing `…` — a teaser cut off with `…` may end part way through one; the one containing it stands where the other stood, so a fuller description from a lower-priority source replaces the teaser it completes, while `Krimi` beside `Krimiseriál o …` stays |
+| `category`, `keyword`, `country` | `'caseless'` | they are the same but for case, spacing, hyphens and underscores — as a [profile](#output-profiles) finds a category's genre, so `Sci-Fi` and `Sci Fi` are one |
+
+`merge.union` sets any of them to another mode: `'exact'` for the texts as
+the sources gave them, as every field had them before, any of the others,
+`{ contained: fn }` to take a source's own suffix off a description before it
+is compared as `'contained'` compares, or a function of your own, under which
+two texts are one when it answers the same for both. `union: 'exact'` sets
+every field exact at once.
+
+```ts
+merge: {
+  union: {
+    // A source that signs off every description with its own note.
+    desc: { contained: (value) => value.replace(/\s*\(skryté titulky\)$/, '') },
+    // Two sources that number seasons differently: `Priatelia I.` and `Priatelia`.
+    title: (value) => value.replace(/\s+[IVX]+\.?$/, '').toLowerCase(),
+  },
+}
+```
+
+It applies as each programme is written, whatever sources it came from, and
+not as two are merged: matching decides what is one broadcast by its titles,
+so it sees every one of them, and a later source's spelling still matches the
+spelling a texts rule will leave out. Different languages are never compared
+— a missing language and an empty one are the same — and nothing a programme
+carries is rewritten: a text is kept or left out whole. A field or a mode it
+does not know is refused as the config is loaded, before anything is grabbed.
+
 ### Cleaning up the output
 
 Two defects arrive from almost every source, and the guide is in a position to

src/config.ts

@@ -13,6 +13,8 @@ import {
   type StageDefaults,
 } from './core/answers.js';
 import { resolveTimeZone } from './core/days.js';
+import { resolveMatch } from './merge/programme.js';
+import { resolveUnion } from './merge/union.js';
 import type { CompressionFormat, CompressionOptions } from './core/output.js';
 import type { MissingAllowance } from './grabber/missing.js';
 import type { EpgServeConfig } from './serve/config.js';
@@ -342,7 +344,22 @@ export async function resolveConfigSource(
 ): Promise<ResolvedConfig> {
   const config = typeof source === 'function' ? await source(...supplied) : source;
 
-  return { ...config, timeZone: resolveTimeZone(config.timeZone) };
+  // `merge.union` here as the zone is, rather than where the merge reads it: a
+  // misspelt field found by `epg build` would be found after the whole grab.
+  // Settled with the config's own match, and carried settled for the merge.
+  const merge =
+    config.merge === undefined || config.merge === null
+      ? undefined
+      : {
+          ...config.merge,
+          union: resolveUnion(config.merge.union, resolveMatch(config.merge.match).normalizeTitle),
+        };
+
+  return {
+    ...config,
+    ...(merge === undefined ? {} : { merge }),
+    timeZone: resolveTimeZone(config.timeZone),
+  };
 }
 
 /** Identity helper for type inference in `epg.config.ts` files. */

src/main.ts

@@ -308,6 +308,9 @@ export {
   normalizeTitle,
   resolveMatch,
   titlesMatch,
+  DEFAULT_UNION,
+  ResolvedUnion,
+  resolveUnion,
   generateGuide,
   writeGuide,
   defaultChannelInfo,

src/merge/channel.ts

@@ -1,5 +1,5 @@
 import type { XmltvChannel } from '../xmltv/types.js';
-import { textKey, unionBy, urlKey } from './programme.js';
+import { textKey, unionBy, urlKey } from './text.js';
 
 /**
  * Merge two `<channel>` descriptions of the same channel from different

src/merge/guide.ts

@@ -27,6 +27,7 @@ import {
   resolveMatch,
   type MergeWatch,
 } from './programme.js';
+import { resolveUnion } from './union.js';
 import type { ChannelSource, RegistryEntry } from './registry.js';
 import { channelSelection, configured, unmatched, unmatchedMessage } from './select.js';
 import type { BuildGuideOptions, FillGapsContext, FillGapsOptions, GuideContext } from './types.js';
@@ -483,6 +484,28 @@ export async function* generateGuide(options: BuildGuideOptions): AsyncGenerator
 
   const listStrategy = channelStrategy === 'merge-programmes' ? programmeStrategy : 'concat';
   const match = resolveMatch(options.merge?.match);
+  // Which texts of a programme say the same thing, applied as each is written.
+  // A loaded config's is settled already — refused, if it had to be, before
+  // anything was grabbed — and taken as it is; a caller's own is settled here.
+  const textUnion = resolveUnion(options.merge?.union, match.normalizeTitle);
+  /**
+   * A programme with its texts as the union leaves them, worked out once: one
+   * handed to a transform as `next` is finished itself straight after, and the
+   * union is asked of it once rather than twice. By identity, so one merged
+   * again in the meantime — held across midnight, and joined by the next day —
+   * is a programme of its own and asked anew.
+   */
+  const unioned = new WeakMap<XmltvProgramme, XmltvProgramme>();
+  const written = (programme: XmltvProgramme): XmltvProgramme => {
+    let result = unioned.get(programme);
+
+    if (result === undefined) {
+      result = textUnion.apply(programme);
+      unioned.set(programme, result);
+    }
+
+    return result;
+  };
   const {
     fillStop = true,
     clipOverlaps = true,
@@ -496,10 +519,10 @@ export async function* generateGuide(options: BuildGuideOptions): AsyncGenerator
       ? undefined
       : ((typeof fillStop === 'object' ? fillStop.maxMs : undefined) ?? DEFAULT_FILL_STOP_MS);
 
-  // Checked rather than trusted, which nothing else in `merge` needs to be: a
-  // `blockMs` of zero or a NaN makes the loop that lays blocks out never reach
-  // the end of a gap, inside a generator a file is being written from. A wedged
-  // merge is worse than a rejected option.
+  // Checked rather than trusted, as `union` is: a `blockMs` of zero or a NaN
+  // makes the loop that lays blocks out never reach the end of a gap, inside a
+  // generator a file is being written from. A wedged merge is worse than a
+  // rejected option.
   const gaps = fillGaps === false ? undefined : resolveFillGaps(fillGaps === true ? {} : fillGaps);
 
   // Two entries share an `xmltvId` under `keep-all`, so each would fill the
@@ -543,7 +566,7 @@ export async function* generateGuide(options: BuildGuideOptions): AsyncGenerator
     next: XmltvProgramme | undefined,
     xmltvId: string,
   ): XmltvProgramme | undefined => {
-    let result = programme;
+    let result = written(programme);
 
     if (next !== undefined) {
       const start = programme.start.getTime();
@@ -579,7 +602,10 @@ export async function* generateGuide(options: BuildGuideOptions): AsyncGenerator
         xmltvId,
         timeZone,
         ...mergeSays,
-        ...(next === undefined ? {} : { next }),
+        // With its texts as the union leaves them, as the guide will write
+        // them — unless it is the last of a day, held for the next one, which
+        // may yet merge more into it.
+        ...(next === undefined ? {} : { next: written(next) }),
       }) ?? undefined
     );
   };

src/merge/main.ts

@@ -5,10 +5,11 @@ export {
   mergeInto,
   mergeProgrammes,
   mergeProgrammeLists,
-  normalizeTitle,
   resolveMatch,
   titlesMatch,
 } from './programme.js';
+export { normalizeTitle } from './text.js';
+export { DEFAULT_UNION, ResolvedUnion, resolveUnion } from './union.js';
 export type { BackfillOptions, ResolvedMatch } from './programme.js';
 export { mergeChannels } from './channel.js';
 export { generateGuide, writeGuide, defaultChannelInfo } from './guide.js';