feat(grabber): a site remembers a programme's details between runs

docs/api.md

@@ -687,6 +687,12 @@ leaves it free to keep a group however it likes, an append-log replayed on read
 included, so long as `readState` answers with what the last `writeState` was
 given.
 
+The package keeps four for a site: `channels`, a list it fetched; `validators`,
+what each url last answered with; `state`, the site's own bag; and
+`remembered`, what a context's
+[`remember`](./site-config.md#a-programmes-details-asked-once) kept, each
+value with when it is forgotten. `StateKey` names them.
+
 `FsCacheDriver` puts each group in `<dir>/<site>/<key>.json`, beside the
 channel directories of the same site; a prune only ever looks at the day files,
 so state outlives every listing a site has. `SqliteCacheDriver` keeps a row per

docs/site-config.md

@@ -305,6 +305,7 @@ What a `parseDay` call is given:
 | `programme` | `(start, title, options?) => ProgrammeBuilder` | A [builder bound to this channel-day](#building-programmes). |
 | `http` | `KyInstance` | The site's client — the very instance `request` was handed. See [a parse that needs another request](#a-parse-that-needs-another-request). |
 | `paced` | `<T>(task) => Promise<T>` | Run a request through the site's queue, so its `concurrency`, `rateLimit` and backoff apply to it. |
+| `remember` | `(key, { until, miss? }, task) => Promise<T>` | What an earlier run kept under `key`, or `task`'s answer, kept until `until` — see [a programme's details, asked once](#a-programmes-details-asked-once). |
 | `state` | `Map<string, unknown>` | What this site [remembers between runs](#remembering-something-between-runs). |
 | `signal` | `AbortSignal \| undefined` | Already applied to `http`; here for work that does not go through it. |
 | `log`, `warn` | `(message, fields?) => void` | Somewhere to [say something](#saying-something), under the site's name. |
@@ -424,6 +425,68 @@ Nothing else in the package reads it, and it is stored [beside the
 listings](./api.md#what-a-site-remembers-between-runs) — `state.json` in the
 site's cache directory.
 
+### A programme's details, asked once
+
+A source that wants a request per programme for its details makes a nightly
+refetch expensive: today is fetched again every run, and with it every detail
+call for programmes already asked about. `remember` asks once and keeps the
+answer for the runs after:
+
+```ts
+async parseDay({ payload, http, paced, remember, programme }) {
+  return Promise.all(payload.items.map(async (item) => {
+    const details = await remember(
+      `detail:${item.id}|${item.start}`,
+      {
+        // Asked about while its day can be fetched again, and no longer.
+        until: Date.parse(item.end) + 86_400_000,
+        // "This one has no details" is an answer, and worth keeping.
+        miss: (error) => error instanceof HTTPError && error.response.status === 404,
+      },
+      async () => {
+        const full = await paced(({ signal }) =>
+          http.get(`detail/${item.id}`, { signal }).json<Detail>());
+
+        return { desc: full.description, cast: full.cast };   // only what the parse uses
+      },
+    );
+
+    const p = programme(new Date(item.start), item.title);
+    return details ? p.desc(details.desc) : p;
+  }));
+}
+```
+
+It hands back what an earlier run kept under the key, or the task's answer,
+kept until `until` — a `Date` or epoch milliseconds, always given, since how
+long something is worth keeping is the site's to say. Only an answer is kept: a
+failure `miss` matches is kept as `null`, and any other is thrown and kept by
+nobody, so the next run asks again. A kept "none" is an answer only to an ask
+with a `miss` of its own; one without asks again, and its type has no `null`
+in it.
+
+Two asks of one key while the first is still on its way are one task — a
+broadcast past midnight is in two days' listings — and each reads what that
+task came to with its own `miss`. A parse waiting on another's lets its slot go
+meanwhile, as it does for `paced`. A request cannot let its queue slot go, so
+one asks for itself instead.
+
+Each caller is handed a copy, so changing what it was given changes nothing
+kept. Like `state`, a value must be plain; a `Date` or a `Map` is refused
+rather than coming back as a string or `{}` on the next run. An `until` already
+past keeps the value for this run only, and says so once — usually an `until`
+in seconds where milliseconds were meant.
+
+It is kept in a group of its own beside `state`, written whole when anything in
+it changes, and what has passed its `until` is dropped at the end of every
+run. So keep only what the parse uses, and pick keys that change when what they
+are about does — a programme moved to another start is worth asking about
+again. `--refresh` reads nothing an earlier run kept, as it reads no cached
+listings, and still keeps what it is answered. Every context has it: a
+request, a parse, and a pass over a whole document. A mark that lasts — "this
+channel has no details, ask again in a week" — is a value remembered for a
+week.
+
 ### A parse that needs another request
 
 Plenty of sources put the schedule in one response and the description of each
@@ -772,8 +835,8 @@ channel-day, its `grabbedAt` and `programmeCount`, and absent where it holds
 nothing — since a stale channel-day is usually there and merely old enough to
 ask about again.
 
-And every mode gets `timeZone`, `dayZone` and [`at`](#times-of-day), as a parse
-does. Under `'days'` and `'both'` there is no one day to assume, so `at` takes
+And every mode gets `timeZone`, `dayZone`, [`at`](#times-of-day) and
+[`remember`](#a-programmes-details-asked-once), as a parse does. Under `'days'` and `'both'` there is no one day to assume, so `at` takes
 the day as well — from the first day's midnight to the end of the last is the
 stretch a range API wants, on the clock of the site's days, where `from` and
 `to` are only UTC midnights:

src/cli/try.ts

@@ -36,6 +36,7 @@ import { kyWith, resolveChannels, siteHttp } from '../grabber/channels.js';
 import { parseContext, requestContext, streamContext } from '../grabber/context.js';
 import { planRequests } from '../grabber/planner.js';
 import { resolveSite } from '../grabber/site.js';
+import { Memory } from '../grabber/remember.js';
 import { SiteStateHandle } from '../grabber/state.js';
 import type { AnySiteConfig, GrabberChannel, ParsedProgramme } from '../grabber/types.js';
 import { ProgrammeBuilder } from '../xmltv/builder.js';
@@ -405,6 +406,10 @@ export async function tryChannelDay(
       task({ ...(options.signal ? { signal: options.signal } : {}) }),
     timeZone: config.timeZone,
     dayZone: resolved.dayZone,
+    // Over a store that keeps nothing, as `state` is: remembered for as long
+    // as this command, so a parse asking twice sees the first answer, and by
+    // no run after it.
+    memory: new Memory(state, site.site, now),
   };
 
   const began = Date.now();

src/core/plain.ts

@@ -0,0 +1,71 @@
+/**
+ * Whether a value comes back from JSON as it went in.
+ *
+ * What the cache keeps is JSON, so anything handed to it to keep — a channel's
+ * `data`, a remembered value — has to be a value JSON gives back unchanged, or a
+ * run that wrote it and the run that reads it would see different things, and a
+ * site would find out only on the second.
+ */
+
+/**
+ * The first thing in a value that is not plain, named with where it is — or
+ * nothing, when all of it is.
+ *
+ * Plain is what JSON gives back as it was given: strings, finite numbers,
+ * booleans, `null`, arrays and plain objects. A property holding `undefined` is
+ * fine, since leaving it out is all JSON does with it and reading it gives the
+ * same. Anything else comes back as something else — a `Date` a string, a `Map`
+ * an empty object, an instance of a class without its methods, `NaN` a `null` —
+ * or not at all, as a function does.
+ */
+export function unplain(
+  value: unknown,
+  at: string,
+  ancestors = new Set<object>(),
+): string | undefined {
+  if (value === null || typeof value === 'string' || typeof value === 'boolean') {
+    return undefined;
+  }
+
+  if (typeof value === 'number') {
+    return Number.isFinite(value) ? undefined : `${String(value)} at ${at}`;
+  }
+
+  if (typeof value !== 'object') {
+    return `${typeof value === 'undefined' ? 'undefined' : `a ${typeof value}`} at ${at}`;
+  }
+
+  // The objects above this one, not every one seen: the same object twice in
+  // a list is fine — JSON writes it twice — and only one inside itself is not.
+  if (ancestors.has(value)) {
+    return `a circular reference at ${at}`;
+  }
+
+  const prototype: unknown = Object.getPrototypeOf(value);
+
+  if (!Array.isArray(value) && prototype !== Object.prototype && prototype !== null) {
+    const name = (value as { constructor?: { name?: string } }).constructor?.name;
+
+    return `${name ? `an instance of ${name}` : 'an object of a class'} at ${at}`;
+  }
+
+  ancestors.add(value);
+
+  try {
+    for (const [key, item] of Object.entries(value)) {
+      const where = Array.isArray(value) ? `${at}[${key}]` : `${at}.${key}`;
+      // `undefined` as a property is left out, which reads the same; in an
+      // array it becomes `null`, which does not.
+      const found =
+        item === undefined && !Array.isArray(value) ? undefined : unplain(item, where, ancestors);
+
+      if (found !== undefined) {
+        return found;
+      }
+    }
+
+    return undefined;
+  } finally {
+    ancestors.delete(value);
+  }
+}

src/grabber/channels.ts

@@ -8,6 +8,7 @@ import type { ChannelCoverage } from '../merge/types.js';
 import type { CacheStore } from '../cache/types.js';
 import { localTimeZone } from '../core/days.js';
 import { errorMessage, unanswered } from '../core/error.js';
+import { unplain } from '../core/plain.js';
 import type { Emit, Says } from '../core/events.js';
 import type { SiteSession } from './session.js';
 import type {
@@ -186,65 +187,6 @@ export interface ResolveChannelsOptions {
   paced?: <T>(fetch: () => Promise<T>) => Promise<T>;
 }
 
-/**
- * The first thing in a channel's `data` that is not a plain value, named with
- * where it is — or nothing, when all of it is.
- *
- * Plain is what JSON gives back as it was given: strings, finite numbers,
- * booleans, `null`, arrays and plain objects. A property holding `undefined` is
- * fine, since leaving it out is all JSON does with it and reading it gives the
- * same. Anything else comes back as something else — a `Date` a string, a `Map`
- * an empty object, an instance of a class without its methods, `NaN` a `null` —
- * or not at all, as a function does.
- */
-function unplain(value: unknown, at: string, ancestors = new Set<object>()): string | undefined {
-  if (value === null || typeof value === 'string' || typeof value === 'boolean') {
-    return undefined;
-  }
-
-  if (typeof value === 'number') {
-    return Number.isFinite(value) ? undefined : `${String(value)} at ${at}`;
-  }
-
-  if (typeof value !== 'object') {
-    return `${typeof value === 'undefined' ? 'undefined' : `a ${typeof value}`} at ${at}`;
-  }
-
-  // The objects above this one, not every one seen: the same object twice in
-  // a list is fine — JSON writes it twice — and only one inside itself is not.
-  if (ancestors.has(value)) {
-    return `a circular reference at ${at}`;
-  }
-
-  const prototype: unknown = Object.getPrototypeOf(value);
-
-  if (!Array.isArray(value) && prototype !== Object.prototype && prototype !== null) {
-    const name = (value as { constructor?: { name?: string } }).constructor?.name;
-
-    return `${name ? `an instance of ${name}` : 'an object of a class'} at ${at}`;
-  }
-
-  ancestors.add(value);
-
-  try {
-    for (const [key, item] of Object.entries(value)) {
-      const where = Array.isArray(value) ? `${at}[${key}]` : `${at}.${key}`;
-      // `undefined` as a property is left out, which reads the same; in an
-      // array it becomes `null`, which does not.
-      const found =
-        item === undefined && !Array.isArray(value) ? undefined : unplain(item, where, ancestors);
-
-      if (found !== undefined) {
-        return found;
-      }
-    }
-
-    return undefined;
-  } finally {
-    ancestors.delete(value);
-  }
-}
-
 /**
  * Refuse a channel list whose `data` holds anything but plain values.
  *

src/grabber/context.ts

@@ -15,6 +15,7 @@
 import { dayToDate, wallClock } from '../core/days.js';
 import type { Says } from '../core/events.js';
 import { ProgrammeBuilder } from '../xmltv/builder.js';
+import type { Away, Memory } from './remember.js';
 import type { Request } from './planner.js';
 import type { ResolvedBatching } from './planner.js';
 import type {
@@ -24,6 +25,8 @@ import type {
   GrabberChannel,
   PacedRequest,
   ParseContext,
+  Remember,
+  RememberOptions,
   RequestContextFor,
   SiteState,
   StreamContext,
@@ -42,6 +45,16 @@ export interface ContextDeps {
   timeZone: string;
   /** The zone the site's days are on: the run's, or the one its `dayZone` names. */
   dayZone: string;
+  /** What the site remembers between runs, for a context's `remember`. */
+  memory: Memory;
+  /**
+   * How this context waits on something another one started — an ask of
+   * `remember`'s — without holding a slot meanwhile: a parse lends its
+   * `localWork` slot, as `paced` does, and a stream holds none. Absent where
+   * the context holds one it cannot lend, a request its site's queue slot, so
+   * it asks for itself rather than wait.
+   */
+  away?: Away;
 }
 
 /** What every request context carries, whichever shape the rest of it takes. */
@@ -59,6 +72,7 @@ export function baseContext(request: Request, deps: ContextDeps): BaseRequestCon
     ...(deps.signal ? { signal: deps.signal } : {}),
     timeZone: deps.timeZone,
     dayZone: deps.dayZone,
+    remember: remembering(deps),
   };
 }
 
@@ -76,6 +90,20 @@ function clock(deps: ContextDeps, assumed?: string): (time: string, day?: string
   };
 }
 
+/**
+ * A context's `remember`: the site's memory, and how this context waits on an
+ * ask another one started.
+ *
+ * The same on every context, whatever days it covers, since how long a value
+ * is worth keeping is the site's to say — a programme's details for a while
+ * after it ends, a mark that a channel has none for a week. The options are
+ * the memory's to check, which knows whose site it is and says so.
+ */
+function remembering(deps: ContextDeps): Remember {
+  return ((key: string, options: RememberOptions, task: () => unknown) =>
+    deps.memory.remember(key, options, task, deps.away)) as Remember;
+}
+
 /**
  * The days a request covers, as a context that batches them says them.
  *
@@ -179,6 +207,7 @@ export function parseContext<TRaw>(
     timeZone: deps.timeZone,
     dayZone: deps.dayZone,
     at: clock(deps, day),
+    remember: remembering(deps),
     programme: (start, title, options) =>
       new ProgrammeBuilder({
         channel: channel.xmltvId,