diff --git a/README.md b/README.md index 2751e26..528a238 100644 --- a/README.md +++ b/README.md @@ -187,6 +187,7 @@ into place, so a reader never sees half a guide. | `--channels ` | `build`/`grab`/`merge`/`serve`/`filter`: [keep only these channels](./docs/configuration.md#keeping-only-some-channels), and fetch nothing for the rest — ids, or a playlist, `*.channels.xml`, guide or plain list naming them; repeatable | | `--allow-missing ` | exit 0 with up to this much of the guide missing — a number of channel-days, or a share like `5%`; [a site that answered nothing is never forgiven](./docs/configuration.md#allowing-some-of-the-guide-to-be-missing) | | `--dry-run` | `grab`/`build` only: [say what a run would fetch](./docs/configuration.md#what-a-run-would-do), and stop — `--format json` for a machine-readable one | +| `--explain ` | [list on stderr](./docs/configuration.md#what-a-run-said-explained), once the run is done, what it said of one thing and what that most likely means: `unmatched` for `build`/`merge` — the programmes that nearly merged, and what would join them; `empty`, `requests` or `held` for `build`/`grab`. Repeatable | | `--threads[=n]` | `build`/`grab`/`serve`: run the sites in [worker threads](./docs/configuration.md#sites-in-threads-of-their-own), one per core or at most `n`; `--no-threads` overrides a config that asks for them | | `--before ` | `cache prune` only: remove days before `YYYY-MM-DD`, default today | | `--keys` | `cache state` with a group only: the keys it holds, not what it holds | diff --git a/docs/api.md b/docs/api.md index f1dca24..ea2467f 100644 --- a/docs/api.md +++ b/docs/api.md @@ -116,15 +116,37 @@ Every event carries a `type`, the fields that type is about, a `level` than being chosen at the call site — `EVENT_KINDS` is the whole table, and it is the whole answer to "what does the default verbosity show?". +A reporter may also say which kinds it does anything with, as a `wants` +property, so a run need not make an event nobody will read. What it makes that +often is made only where a reporter wants it: the `entry:*` of each channel-day +but a failure, `request:started` and `request:done`, a site's `site:note`, +`pacing:rateLimit`, and `merge:unmatched`, one per near miss and thousands of +them where two sources run on two grids. Measured at `info`, for a site asking +once per channel-day: 7% of a grab's time, and 4% in a thread. A reporter +without `wants` wants every kind, as the one above does. The ones this package +ships want what their level shows — plus their failures, and what the live line +counts — `emitterReporter` what its emitter has listeners for, and +`combineReporters` what any of its reporters wants. It is asked of each kind +once, the first time a run asks — after the run's first wait, so what listens +from as the run is started is asked about — and the run holds to that set — a site in a thread +included, which neither makes nor sends what the set leaves out. It is never +trusted to filter: a reporter is still handed whatever is made. + +```ts +const reporter = Object.assign((event: EpgEvent) => metrics.increment(event.type), { + wants: (type: EpgEvent['type']) => type !== 'merge:unmatched', +}); +``` + | group | types | |---|---| | the run | `run:cancelled`, `grab:done` (with every request of the run, by label — which a request a parse did not wait for can take past what its sites' `site:done` add up to) | -| a site | `site:started` (with `dayZone` for a site on [another calendar](./configuration.md#which-day-is-today), whose days `days` then counts), `site:done` (its counts plus what it cost: `requests`, `ms`, `requestMs`, `requestsBy`), `site:failed`, `site:note`, `site:warning` (the last two carrying whatever [fields the site attached](./site-config.md#saying-something)) | +| a site | `site:started` (with `dayZone` for a site on [another calendar](./configuration.md#which-day-is-today), whose days `days` then counts), `site:done` (its counts plus what it cost: `requests`, `ms`, `requestMs`, `requestsBy`, and `msBy`, where its `ms` went), `site:failed`, `site:note`, `site:warning` (the last two carrying whatever [fields the site attached](./site-config.md#saying-something)) | | one channel-day | `entry:cached`, `entry:fetched`, `entry:appended`, `entry:unchanged`, `entry:failed` | | one request | `request:started`, `request:done` (with `ms`), `request:failed` | | a whole-document source | `stream:gaps`, `stream:ignored` | -| pacing | `pacing:held`, `pacing:slowed`, `pacing:recovered`, `pacing:rateLimit` | -| merging, tidying | `merge:channel`, `merge:summary` (what the sources made of each other), `merge:done`, `merge:note`, `merge:warning`, `prune:done` | +| pacing | `pacing:held` (with `retryAfterMs`, where the source said how long, and `lengthened`, where it made a running hold longer), `pacing:slowed`, `pacing:recovered`, `pacing:rateLimit` | +| merging, tidying | `merge:channel`, `merge:unmatched` (each programme that nearly merged, at `debug`), `merge:summary` (what the sources made of each other), `merge:done`, `merge:note`, `merge:warning`, `prune:done` | | [serving](./configuration.md#serving-the-guide) | `serve:started` (with the `url`), `serve:response` (`method`, `path`, `status`, `ms`), `serve:reloaded`, `serve:disconnected`, `serve:failed`, `serve:grabFailed` (a scheduled grab that threw outright, rather than one of its sites failing), `serve:stopped` | ### What a run cost, and what the merge made of it @@ -160,18 +182,57 @@ waiting on something of its own: its `rateLimit`, or the parsing and writing that follow each response, which share [`localConcurrency`](./configuration.md#how-caching-works) with every other site. That is the measurement to take before tuning either. +`msBy` takes it. It is `ms` again, by what the site was doing — at each moment +the first of these that held, so the parts never overlap and add up to `ms`: + +```json +"msBy": { "source": 939, "held": 0, "rateLimit": 0, "working": 1204, "queued": 168100, "other": 169 } +``` + +`held` is the time the source had the site's requests held after a `429` or +`503`, with any of them outstanding — waiting its turn, or out: under a hold, +that is one ky retries once the source's `Retry-After` is up, or one sent +before it that the hold holds up as much — so a site asking one request at a +time is seen to be held, not waiting on the source. `source` is the time at +least one of its requests was out to the source otherwise — once however many +were, so it is `requestMs` only for a site that sends one at a time — and a +streaming site's pass reading its document, its body arriving and split as it +does, where nothing else of the site's holds. +`rateLimit` is the time its own `rateLimit` held the next one back. `working` is its own work on this machine under way — a `parseDay`, +a write to the cache, a check of what is cached — and `queued`, that work +waiting for a `localConcurrency` slot with none of it under way, which other +sites' work holds. `other` is the rest: its own code between them, reading what +it remembered, its turn's beginning and end. The site above took 170 seconds +over two requests answered in under one, and spent 168 of them waiting for a +slot. A site in a thread measures its own in memory the run's thread reads, so +one whose thread dies has said where its time went until then — its waits for a +slot on the run's thread, writing its cache, among `queued`. A cancel drops +what was waiting, so a cancelled site's last moments are `other`. + +It is measured in every run, for a few microseconds a channel-day: measured on +7,000, a few percent of a grab whose requests and parses take no time at all, +and nothing worth the name beside a source that takes any. + `requestsBy` says the same by what each request was for, with what failed them — so "5291 requests" says that 3,242 of them were detail calls answered `404`: ```json "requestsBy": { - "request": { "requests": 49, "requestMs": 12034, "failed": {} }, - "detail": { "requests": 5242, "requestMs": 573118, "failed": { "404": 3242, "timeout": 3 } } + "request": { "requests": 49, "requestMs": 12034, "waitedMs": 2100, "failed": {} }, + "detail": { "requests": 5242, "requestMs": 573118, "waitedMs": 44021, "failed": { "404": 3242, "timeout": 3 } } } ``` `requestMs` is the label's share of the site's `requestMs`, the time spent waiting on the source — not of `ms`, which is the site's wall clock. +`waitedMs` is how long its requests sat in the site's queue before each was +sent: for the source to stop holding them, for the site's `rateLimit`, or for +one of its `concurrency` slots. Both are each request's own time added up, so +four in flight together for a second each are four seconds. A parse's `paced` +calls go ahead of the planned requests, so where both wait it is the planned +ones whose `waitedMs` grows — and a planned request is asked for only once the +response before it has been parsed and written, so that wait is in `msBy`, +not here. A label is the site's to name: [`requestLabel`](./site-config.md#reference) for its own requests — a request site's planned ones, or, since a stream has @@ -237,8 +298,10 @@ Merge: 4821 programmes combined, 37 nearly, 112 containers dropped sport.sk: 7m apart — "Správy" / "Správy TV" ``` -`combined` is programmes a later source joined to one already there. `nearMiss` -is programmes that joined nothing with something close by — the quiet failure +`combined` is programmes one source joined to another's already there — not to +its own, held over from the day before and said again. `nearMiss` is +programmes that joined nothing while another source's, within half an hour, was +left unjoined too — the quiet failure that matters, because a source published on a different grid merges with nothing and the guide then carries every programme twice. A gap larger than `merge.match.startToleranceMs` means they failed on time; smaller means they @@ -247,6 +310,27 @@ a guide of thousands the count is the alarm and the examples only have to say whether it is one channel or all of them. `containers` and `clipped` are what `dropContainers` and `clipOverlaps` took away. +Each of those near misses is a `merge:unmatched` of its own, at `debug`: the +programme that joined nothing, the unmatched one of another source's nearest +it — from a source before it in the merge, or one held over from the day before +whatever its place — each as `{ site, title, start }` — the start as the guide writes it, to find +them there by — `shiftMs`, how much later the one starts than the other, below +nought where it starts earlier, and `why`. `title` means they started +near enough and their titles differ, which `merge.match.normalizeTitle` decides; +`start`, that the titles agree and the starts are too far apart, which +`startToleranceMs` decides; `both`, both; `runs`, that the titles agree and the +starts are near enough but further apart than the shorter of the two lasts — +two programmes in a row, or one within the other, as a segment is within its +show — which no tolerance changes. Under a `match` of the +config's own, `matcher` is any pair it said no to, its reasons its own — none +of the shipped rules is asked on its behalf — and `start`, for one with a +`windowMs`, a pair that starts beyond it, which it is never asked about. Only another source's programme is a near +miss — never a source's own, held from the day before. `epg merge --explain +unmatched` and `epg build --explain unmatched` list them all on stderr once the +guide is written, grouped, with what would join them — see +[what a run said, explained](./configuration.md#what-a-run-said-explained); +for each as JSON, `--reporter json --log-level debug` has them as events. + Two things about the shape are deliberate, and both are the difference between a structured sink and a string one. **A failed request is one event, not one per channel-day it covered** — `request:failed` carries `entries`, so a site that is @@ -320,7 +404,15 @@ events.on('site:done', (event) => metrics.timing(event.site, event.ms)); ``` Each concern subscribes to the one it is about, instead of a `switch` in a -reporter growing a case per concern. That is also where a supervisor's +reporter growing a case per concern — and a run makes no event for a kind +nothing has subscribed to, where making it costs anything, since an emitter +says through its `listenerCount` which kinds have listeners — once a run first +asks, so a listener added as `grab(…)` is called is heard, and one added part +way through a run hears from the next. An emitter that +hears more than its listeners do — an `emit` of its own that passes every event +on, or a listener for any kind — says what it hears instead: +`emitterReporter(bridge, { wants: () => true })` — which a config takes as its +`events` as well. That is also where a supervisor's readiness ping belongs: this package sends none, because a library writing to an IPC channel nobody asked it to is a surprise rather than a feature. @@ -344,6 +436,18 @@ Both run, and `events` is honoured by every entry point that reads a config — export default defineConfig({ sites, output, events }); // the emitter itself ``` +`events` may be a reporter of your own as well, which is how a config gives an +emitter that hears more than its listeners do — an `emit` of its own that +passes every event on — and so says so: + +```ts +export default defineConfig({ + sites, + output, + events: emitterReporter(bridge, { wants: () => true }), +}); +``` + ### Naming one in a config `cache.driver` takes a name or a factory, and a reporter works the same way, so @@ -397,9 +501,11 @@ said so. `errorChain(error)` is the same walk, exported for a caller doing its own rendering. `emitter(options)` is the other end: it turns the `reporter` a caller passed -into the `Emit` the internals take, and answers a silent one when nobody is -listening. Only needed to drive `grab` or `generateGuide` directly and have -their events reach the same reporter `build` would have given them. +into the `Emit` the internals take — the function, and `hears`, the set of kinds +its `wants` said yes to, asked once, the first time the run asks — and answers a silent one +when nobody is listening. Only needed to drive `grab` or `generateGuide` +directly and have their events reach the same reporter `build` would have given +them, or to hand `sitePacing` an `emit`. ## Streaming a guide @@ -1229,6 +1335,12 @@ them. A loaded config carries its `merge.union` already settled, as a `ResolvedUnion` — whose own fields are the union it was settled from, so a copy of it is that union — which `resolveUnion` takes as it is for the same match and settles again for another. +`resolveMatch(match)` settles a match into what a merge asks of it: +`windowMs`, how far apart two programmes can be and still be one broadcast; +`normalizeTitle`; and `apart(a, b)`, what keeps two programmes from being one — +a `NearMissWhy`, the `why` of `merge:unmatched` — or nothing, where they are +one. One answer to both questions, so the two cannot disagree; it was +`matches(a, b)`, a yes or no, before near misses said why. `mergeInto(target, incoming, resolveMatch(match))` is the piece underneath `mergeProgrammeLists`: one source's list folded into a sorted accumulator, which is also how the guide diff --git a/docs/configuration.md b/docs/configuration.md index 6471b19..8088987 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -229,6 +229,7 @@ epg build -o /home/hts/.hts/tvheadend/epggrab/xmltv.sock # write into a socket | `--refresh` | refetch every day in the window, ignoring what is cached — the days still land in the cache for the run after | | `--threads[=n]` | `build`/`grab`/`serve`: run the sites in [worker threads](#sites-in-threads-of-their-own), one per core or at most `n`; `--no-threads` runs them all in one, whatever the config says | | `--dry-run` | `grab`/`build` only: say what a run would fetch, and stop — see [what a run would do](#what-a-run-would-do) | +| `--explain ` | list on stderr, once the run is done, what it said of one thing, and what that most likely means — `unmatched` for `build`/`merge`, the programmes that nearly merged; `empty`, `requests` or `held` for `build`/`grab`. Repeatable — see [what a run said, explained](#what-a-run-said-explained) | | `--allow-missing ` | exit 0 with up to this much of the guide missing: a number of channel-days, or a share like `5%` | | `--extensions ` | `build`/`merge`/`serve`/`filter`: keep only these [provider extensions](#provider-extensions), comma-separated — `--extensions lcn,uniqueID` | | `--no-extensions` | `build`/`merge`/`serve`/`filter`: leave every one out, for a guide that validates against the DTD | @@ -1413,6 +1414,108 @@ counts for one config. What the report does not apply is `cover`: a channel a config would make more requests than someone expected. `build --dry-run` reports the same and writes no guide. +### What a run said, explained + +A run counts what it has no room to list: `(5 empty)`, `5291 requests`, a +merge's `36 nearly`. `--explain` lists one of them in full, on stderr, once the +run is done, and says what the list most likely means — and what to change for +it. It can be repeated, and each is written once, in the order asked for: + +```sh +epg build --explain empty --explain requests --explain held +``` + +| Explains | For | What it lists, and reads | +| --- | --- | --- | +| `unmatched` | `build`, `merge` | every programme that nearly merged with another source's and did not, grouped by channel and source; a tolerance that would bring a group within reach, or a title rule that would join it | +| `empty` | `build`, `grab` | the channel-days a source answered with no programmes; why, from what the rest of the run answered | +| `requests` | `build`, `grab` | each site's requests by what they were for; where its time went, and what its failures mean | +| `held` | `build`, `grab` | the sites a source told to slow down; what each status means, how each hold ended, and the pace that earned them | + +**`unmatched`** groups the near misses by channel, by the two sources and by +what kept them apart, then lists each one. Where most of a group agree on a +shift, it says by how much one source runs behind, and the `startToleranceMs` +that would reach them all — beside the config's own, where it uses the shipped +match — which joins those of them that each last longer than they are apart, a +shorter pair being two programmes in a row as far as the merge can tell; or +the `windowMs` that would let a matcher of the config's own be asked about +them. Where one source's titles go on past the other's after the same +separator, or begin with more before it, it says a `normalizeTitle` that cuts +there would join them. See [what counts as the same broadcast](#what-counts-as-the-same-broadcast): + +``` +5 programmes nearly merged, on 1 channel: + ct1.cz, b.tv against a.tv: + 3 × starts too far apart: b.tv starts 7 minutes later on all 3 — a startToleranceMs of 420000 would bring them within reach (it is 300000), and join those whose shorter lasts longer than they are apart + 2 × titles differ: b.tv's titles go on past a.tv's after ": " on both, e.g. "DVTV: Jakub Kříž" / "DVTV" — a normalizeTitle that cuts them there would join them + +Each of them: + ct1.cz: 420s apart, starts too far apart — "Události" (b.tv, 06:07) / "Události" (a.tv, 06:00) + … +``` + +**`empty`** reads each empty channel-day against the rest of the run: a source +empty on every channel-day it was asked about, whose `parseDay` may no longer +read what it sends; one with nothing after a day on any channel, which has not +published that far; a channel-day another source has programmes for, which +this one may know by another id; one every source asked about it is empty on; +and one this source alone is empty on, a gap in its listings. It claims nothing a +channel-day served from the cache, said unchanged by the source, or failed +could make untrue, since none of those says what it holds: + +``` +10 channel-days came back empty: + skylink.sk: nothing after 2026-10-29 on any channel — it has not published that far yet: 2026-10-30..2026-10-31 (2 days), on 3 channels + dvojka.sk (magio.tv): 2026-10-26 — nothing else says otherwise: off air, or a gap in magio.tv's listings + joj.sk (magio.tv): 2026-10-25..2026-10-27 (3 days) — skylink.sk has programmes then: magio.tv may know it by another id, or have stopped carrying it +``` + +**`requests`** tables each site's requests — how long they were out to the +source, `waiting`, and how long they sat in its queue first, `in queue` — then +says where each site's time went, from `site:done`'s +[`msBy`](./api.md#what-a-run-cost-and-what-the-merge-made-of-it), and, where +one part is most of a second or more, what would change it: + +``` +Requests, by site and what each was for: + site label requests waiting in queue failed + magio.tv request 2 939ms 0ms + skylink.sk request 49 7310ms 2100ms + skylink.sk detail 5242 577s 44s 3242 × 404 + +Where each site's time went: + magio.tv took 170s: 168s waiting for a local slot, 1204ms parsing and writing, 939ms on the source, 169ms on other things. + Mostly waiting for one of the run's 16 local slots, which other sites' work held: a higher localConcurrency, or --threads, lets it in sooner. + skylink.sk took 11m37s: 560s on the source, 90s held by the source, 22s under its own rateLimit, 15s parsing and writing, 10s on other things. + Mostly waiting on the source, one request at a time: a concurrency above 1 would overlap them, where the source allows it. + 3242 of its 5242 detail requests were answered 404: the source has nothing at those addresses — ids it no longer knows, or details of programmes it has dropped. +``` + +**`held`** is what makes a [rate limit](./site-config.md#rate-limits-and-backoff) +visible. It says how each hold ended — as long as the source asked, cut short +by `backoff.maxMs`, which makes another one likely, or the fallback where the +source said nothing — and the pace the site kept while it had requests out and +was not held, which is what earned the holds, beside the `rateLimit` it sets, +or that it sets none: + +``` +1 site told to slow down: + skylink.sk: held twice, for up to 60s at a time; concurrency down to 2, back up to 3 + 2 × 429: too many requests — the source's rate limit + As long as the source asked, once. + Cut short by backoff.maxMs once, where the source asked for up to 120s: let go sooner than it wanted, the site is likely to be held again. + It sent 5291 requests, 9.4 a second over the 560s it had any out unheld, with no rateLimit: a rateLimit below that pace would keep it from being held, at the cost of the time that takes. +``` + +Each is made of events the run says anyway — `merge:unmatched`, +`entry:fetched`, `site:done` and `pacing:held` among them — and keeps what it +is about whatever `--log-level` shows. What it reads into them it says as +likely, never as found: it has the run's events and the config, not the source. +It is for reading, so `--format json` is refused beside it; a program has the +same events as JSON through `--reporter json --log-level debug`. A cancelled +run explains nothing. Asking for all four costs a run about 2µs a channel-day, +measured on 7,000. + ### `--offset` `--offset` shifts the window only; "now" is unchanged, so staleness and the @@ -1746,6 +1849,15 @@ the tolerance is. Inside one source's own list none of this applies: two entries in a schedule are two broadcasts, and only entries naming the identical instant are pooled. +**To see which of these to change**, `epg merge --explain unmatched` lists +every programme that nearly merged and did not — both titles, both sites, how +far apart — and says what kept them apart: the titles, which `normalizeTitle` +answers; the starts, which `startToleranceMs` does; or the durations, which +say they are two programmes in a row, or one within the other, and are better +left so. Where a group of +them agree, it says the tolerance that would bring them within reach, or the +title rule that would join them. + **A title rule is the cheap fix.** Where two sources describe the same broadcast but one prefixes or suffixes its titles, `normalizeTitle` says so in a line and everything else — the tolerance, the duration guard — keeps working: diff --git a/docs/site-config.md b/docs/site-config.md index b00005d..50ff27b 100644 --- a/docs/site-config.md +++ b/docs/site-config.md @@ -1737,6 +1737,12 @@ capped by `maxMs`, `fallbackMs` when the header says nothing), and then resumes **with its tasks intact**: nothing is dropped, nothing is refetched, the channel days waiting their turn simply wait a little longer. Requests already in flight are left alone, and `ky`'s own per-request retry then does its usual job on top. +Each hold is a `pacing:held`, its `ms` what it was given and its `retryAfterMs` +what the source asked for — so a hold `maxMs` cut short says so. A response +told off while a hold is running, that asks for longer than is left, makes the +hold longer, and is a `pacing:held` too, marked `lengthened` — and +[`--explain held`](./configuration.md#what-a-run-said-explained) reads them +back with the pace that earned them. `adapt` also halves the site's `concurrency` on a slow-down and gives it back one clean response at a time, ten in a row per step — a retreat that is quicker diff --git a/src/cli/explain.ts b/src/cli/explain.ts new file mode 100644 index 0000000..28eed8f --- /dev/null +++ b/src/cli/explain.ts @@ -0,0 +1,1152 @@ +/** + * `--explain`: what a run said of one thing, kept as it says it and written on + * stderr once it is done — listed in full, where a run's own lines have room + * for a count, and read: what the list most likely means, and what to change + * for it. + * + * Each is made of events a run says anyway, and keeps only those it is about, + * whatever level the run's reporter shows: what it is asked for is all of + * them. What it reads into them it says as likely, never as found — it has + * the events and the config, not the source. A program has every event of + * its own, through `--reporter json`. + */ + +import type { Writable } from 'node:stream'; +import type { EpgConfig } from '../config.js'; +import { count } from '../core/count.js'; +import { addDays, diffDays } from '../core/days.js'; +import { + type EpgEvent, + type Reporter, + type RequestCounts, + type TimeSpent, +} from '../core/events.js'; +import { daySpan, failures, render, took, UNMATCHED_WHY } from '../core/reporters.js'; +import { writeLines } from '../core/streams.js'; +import { localSlots } from '../grabber/grab.js'; +import { heldFor } from '../grabber/pacing.js'; +import type { AnySiteConfig } from '../grabber/types.js'; +import { mergeStrategies } from '../merge/guide.js'; +import { resolveMatch } from '../merge/programme.js'; + +/** What `--explain` can be asked about. */ +export const EXPLAINABLE = ['unmatched', 'empty', 'requests', 'held'] as const; + +export type Explainable = (typeof EXPLAINABLE)[number]; + +/** What an explanation is told of a run: each kind of event it is about, and what it keeps of one. */ +type Hears = { [T in EpgEvent['type']]?: (event: Extract) => void }; + +/** One explanation, as a run keeps it. */ +interface Explanation { + /** What it keeps of each kind of event it is about — and it is told of no other. */ + hears: Hears; + /** What it kept, once the run is done: a heading, and what is under it. */ + lines(config: EpgConfig): Iterable; +} + +/** Each explanation: the commands whose run it is about, and a new one for a run. */ +const EXPLANATION: Record Explanation }> = + { + unmatched: { commands: ['build', 'merge'], start: unmatched }, + empty: { commands: ['build', 'grab'], start: empty }, + requests: { commands: ['build', 'grab'], start: requests }, + held: { commands: ['build', 'grab'], start: held }, + }; + +/** How many lines are written at a time: an explanation can be a guide's length. */ +const BATCH = 1_000; + +/** The commands whose run an explanation is about. */ +export function explainCommands(name: Explainable): readonly string[] { + return EXPLANATION[name].commands; +} + +/** + * The explanations `names` asks for, as one: the reporter that keeps what each + * is about, to join to the run's own — none for none — and what writes them to + * `out` once the run is done, in the order asked for, each once. Said where it + * can be: stderr closed by whatever read it — `| head` — fails no command + * whose guide is written. + */ +export function explanations(names: readonly Explainable[]): { + reporter: Reporter | undefined; + render(out: Writable, config: EpgConfig): Promise; +} { + const started = [...new Set(names)].map((name) => EXPLANATION[name].start()); + /** + * Each kind of event, and the explanations it is told to: found once per + * event, so the many no explanation is about — a request starting, a day + * fresh in the cache — cost a run one lookup each. + */ + const hearing = new Map void>>(); + + for (const { hears } of started) { + for (const [type, hear] of Object.entries(hears)) { + const told = hearing.get(type); + const one = hear as (event: EpgEvent) => void; + + if (told === undefined) { + hearing.set(type, [one]); + } else { + told.push(one); + } + } + } + + return { + reporter: + started.length === 0 + ? undefined + : Object.assign( + (event: EpgEvent) => { + const told = hearing.get(event.type); + + if (told !== undefined) { + for (const hear of told) { + hear(event); + } + } + }, + // The kinds it hears, and no other: a run need not make the rest. + { wants: (type: EpgEvent['type']) => hearing.has(type) }, + ), + async render(out, config) { + let batch: string[] = []; + /** + * What is batched, written — and whether it could be: a stderr closed by + * whatever read it has nowhere left to say the rest. Only the write is + * let go of so: what an explanation throws is a fault, and the run's to + * say. + */ + const written = async (): Promise => { + try { + await writeLines(out, ...batch); + } catch { + return false; + } + + batch = []; + + return true; + }; + + for (const [at, explanation] of started.entries()) { + // A blank line before each but the first, so each heading begins a + // block of its own. + if (at > 0) { + batch.push(''); + } + + for (const line of explanation.lines(config)) { + // At or past it: the blank line between two can bring it there. + if (batch.push(line) >= BATCH && !(await written())) { + return; + } + } + } + + if (batch.length > 0) { + await written(); + } + }, + }; +} + +/** The site of that name in the config, where there is one. */ +function siteIn(config: EpgConfig, name: string): AnySiteConfig | undefined { + return config.sites.find((site) => site.site === name); +} + +// ── unmatched ─────────────────────────────────────────────────────────────── + +type NearMiss = Extract; + +/** + * `unmatched`: every programme the merge nearly joined to another source's and + * did not — what `merge:summary` counts, and shows a few of — grouped by + * channel, by the two sources and by what kept them apart, with what would + * have joined a group, where its pairs agree on it: a source on a grid of its + * own, which `startToleranceMs` would reach, or titles with more to them, which + * a `normalizeTitle` would cut. Then each of them. + */ +function unmatched(): Explanation { + const said: NearMiss[] = []; + + return { + hears: { + 'merge:unmatched': (event) => { + said.push(event); + }, + }, + *lines(config) { + const matching = matchingStrategy(config); + + if (matching !== undefined) { + yield ( + `Near misses are looked for only under channelStrategy 'merge-programmes' and ` + + `programmeStrategy 'merge', and this merge is under ${matching}.` + ); + + return; + } + + if (said.length === 0) { + yield 'No programmes nearly merged.'; + + return; + } + + /** + * By channel, then the two sources, whichever of them came first: a + * source's programme held over from the day before is met by the other's + * first list as much as the other way round, and the two are one pair. + */ + const groups = Map.groupBy(said, (event) => + [event.channelId, ...[event.programme.site, event.nearby.site].sort()].join('\0'), + ); + + const channels = new Set(said.map((event) => event.channelId)); + + yield `${count(said.length, 'programme')} nearly merged, on ${count(channels.size, 'channel')}:`; + + for (const key of [...groups.keys()].sort()) { + const pairs = groups.get(key)!; + const [channelId, one, two] = key.split('\0') as [string, string, string]; + // Said of the one that more often joined nothing — the later of the + // two, mostly — so it reads as the source that runs behind or ahead. + const ones = pairs.filter((pair) => pair.programme.site === one).length; + const [site, other] = ones * 2 >= pairs.length ? [one, two] : [two, one]; + + yield ` ${channelId}, ${site} against ${other}:`; + + const byWhy = Map.groupBy(pairs, (event) => event.why); + + for (const [why, these] of [...byWhy].sort((a, b) => b[1].length - a[1].length)) { + yield ` ${these.length} × ${UNMATCHED_WHY[why]}${whyMore(why, these, config, site)}`; + } + } + + yield ''; + yield 'Each of them:'; + + for (const event of said) { + yield ` ${render(event) ?? ''}`; + } + }, + }; +} + +/** What a group of pairs kept apart for one reason says, after a colon — or nothing. */ +function whyMore( + why: NearMiss['why'], + pairs: readonly NearMiss[], + config: EpgConfig, + site: string, +): string { + const [first] = pairs as [NearMiss]; + const example = `, e.g. "${first.programme.title}" / "${first.nearby.title}"`; + + switch (why) { + case 'start': + return startsApart(pairs, config, site); + case 'title': + return titlesApart(pairs, resolveMatch(config.merge?.match).normalizeTitle) ?? example; + case 'runs': + return ': programmes in a row, or one within the other — rightly kept apart'; + case 'both': + case 'matcher': + return example; + } +} + +/** + * A group whose titles agree and whose starts are too far apart: by how much + * `site` runs behind the other, where most of the pairs agree on it, and the + * tolerance that would reach every one of them — said only where the config's + * match is the shipped one, whose tolerance it is. Each pair's shift as of + * `site`, whichever of the two its programme was. + */ +function startsApart(pairs: readonly NearMiss[], config: EpgConfig, site: string): string { + const shifts = new Map(); + let widest = 0; + + for (const pair of pairs) { + const shift = pair.programme.site === site ? pair.shiftMs : -pair.shiftMs; + const minutes = Math.round(shift / 60_000); + + shifts.set(minutes, (shifts.get(minutes) ?? 0) + 1); + widest = Math.max(widest, Math.abs(pair.shiftMs)); + } + + /** The shift most of them share, and how many do. */ + const [minutes, sharing] = [...shifts].sort((a, b) => b[1] - a[1])[0] ?? [0, 0]; + const said: string[] = []; + + // Most of them, by the same few minutes: a source running behind the other. + if (minutes !== 0 && sharing * 2 > pairs.length) { + said.push( + `${site} starts ${count(Math.abs(minutes), 'minute')} ` + + `${minutes > 0 ? 'later' : 'earlier'}${onHowMany(sharing, pairs.length)}`, + ); + } + + const match = config.merge?.match; + const reaching = Math.ceil(widest / 60_000) * 60_000; + const them = pairs.length === 1 ? 'it' : 'them'; + // Within reach is not joined: the shipped match still keeps apart two that + // start further apart than the shorter lasts, which these say nothing of. + const tolerance = (now: number): string => + `a startToleranceMs of ${reaching} would bring ${them} within reach (it is ${now}), ` + + (pairs.length === 1 + ? 'and join it if the shorter of the two lasts longer than they are apart' + : 'and join those whose shorter lasts longer than they are apart'); + + // The shipped match's tolerance; a matcher's own window, which only lets it + // be asked; and nothing of a bare function's, which has none. Each as the + // merge resolves it, so what is said to be is what it was. + const { windowMs } = resolveMatch(match); + + if (typeof match !== 'function') { + said.push( + match !== undefined && match !== null && 'matches' in match + ? `a windowMs of ${reaching} would let its match be asked about ${them} (it is ${windowMs})` + : tolerance(windowMs), + ); + } + + return said.length === 0 ? '' : `: ${said.join(' — ')}`; +} + +/** + * A group whose titles differ: where most of its pairs differ the same way — + * one source's titles going on past the other's after the same separator + * (`DVTV: Jakub Kříž` against `DVTV`), or beginning with more before it — + * which a `normalizeTitle` cutting there would join. Nothing, where they do not. + * Compared as the merge compares them, by `normalize`. + */ +function titlesApart( + pairs: readonly NearMiss[], + normalize: (title: string) => string, +): string | undefined { + const ways = new Map< + string, + { site: string; other: string; sep: string; after: boolean; n: number; e: NearMiss } + >(); + + for (const pair of pairs) { + const way = differs(pair, normalize); + + if (way !== undefined) { + const key = `${way.site}\0${way.sep}\0${String(way.after)}`; + const seen = ways.get(key); + + if (seen === undefined) { + ways.set(key, { ...way, n: 1, e: pair }); + } else { + seen.n++; + } + } + } + + const [most] = [...ways.values()].sort((a, b) => b.n - a.n); + + if (most === undefined || most.n * 2 <= pairs.length) { + return undefined; + } + + const on = onHowMany(most.n, pairs.length); + const example = `"${most.e.programme.title}" / "${most.e.nearby.title}"`; + + return most.after + ? `: ${most.site}'s titles go on past ${most.other}'s after "${most.sep}"${on}, e.g. ${example} — ` + + `a normalizeTitle that cuts them there would join them` + : `: ${most.site}'s titles begin with more, before "${most.sep}"${on}, e.g. ${example} — ` + + `a normalizeTitle that keeps what follows it would join them`; +} + +/** + * Of how many of a group's pairs something holds — ` on both`, ` on all 5`, + * ` on 4 of them` — or nothing, for a group of one: it is said of that one. + */ +function onHowMany(n: number, of: number): string { + if (of === 1) { + return ''; + } + + if (n < of) { + return ` on ${n} of them`; + } + + return of === 2 ? ' on both' : ` on all ${n}`; +} + +/** + * How one title of a pair is the other with more to it: whose, what separates + * the rest, and whether the rest comes after — or nothing, where neither + * holds the other whole. + */ +function differs( + pair: NearMiss, + normalize: (title: string) => string, +): { site: string; other: string; sep: string; after: boolean } | undefined { + const a = normalize(pair.programme.title); + const b = normalize(pair.nearby.title); + const [longer, shorter, site, other] = + a.length >= b.length + ? [a, b, pair.programme.site, pair.nearby.site] + : [b, a, pair.nearby.site, pair.programme.site]; + + if (shorter === '' || longer === shorter) { + return undefined; + } + + if (longer.startsWith(shorter)) { + const sep = /^[^\p{L}\p{N}]+/u.exec(longer.slice(shorter.length))?.[0]; + + return sep === undefined ? undefined : { site, other, sep, after: true }; + } + + if (longer.endsWith(shorter)) { + const sep = /[^\p{L}\p{N}]+$/u.exec(longer.slice(0, longer.length - shorter.length))?.[0]; + + return sep === undefined ? undefined : { site, other, sep, after: false }; + } + + return undefined; +} + +/** + * The strategy a merge says it with that looks for no near misses, or + * nothing, where it does: `merge-programmes` for its channels, and `merge` for + * their programmes. `backfill` matches across sources too, to leave out what + * another already has, but what it nearly matched it keeps as it is. + */ +function matchingStrategy(config: EpgConfig): string | undefined { + const { channelStrategy: channels, programmeStrategy: programmes } = mergeStrategies( + config.merge, + ); + + if (channels !== 'merge-programmes') { + return `channelStrategy: '${channels}'`; + } + + return programmes === 'merge' ? undefined : `programmeStrategy: '${programmes}'`; +} + +// ── empty ─────────────────────────────────────────────────────────────────── + +/** Why a channel-day came back empty, as far as the run's other answers say. */ +type EmptyWhy = 'elsewhere' | 'everywhere' | 'gap'; + +/** + * `empty`: the channel-days a source answered with no programmes — what a run + * counts as `(n empty)`, and names none of — and what the run's other answers + * make of them. A source empty on every channel-day it was asked about, whose + * parse may no longer read what it sends; one with nothing after a day on any + * channel, which has not published that far; a channel-day another source has + * programmes for, which this one may know by another id; one every source asked + * about it is empty on; and one this source alone is empty on, a gap in its + * listings. + * + * Read only from what the run saw. A channel-day served from the cache, said + * unchanged by the source, or failed says nothing of what it holds, so it is + * taken as one that may have programmes: nothing is claimed that it could make + * untrue. + */ +function empty(): Explanation { + /** Each channel's sources, each with the days it came back empty. */ + const channels = new Map>>(); + /** Each channel-day's sources that answered it with programmes. */ + const answered = new Map>(); + /** + * Each channel-day's sources whose programmes for it the run does not know: + * served from the cache, said unchanged by the source, or failed. + */ + const unknown = new Map>(); + /** Sources that failed whole — their channel list, or their turn — which may carry any channel. */ + const failed = new Set(); + /** Each source's last day with programmes, or not known to have none — and its days with any. */ + const lastWith = new Map(); + const daysWith = new Map>(); + + const add = (into: Map>, key: string, value: string): void => { + let set = into.get(key); + + if (set === undefined) { + into.set(key, (set = new Set())); + } + + set.add(value); + }; + const reached = (site: string, day: string): void => { + const last = lastWith.get(site); + + if (last === undefined || day > last) { + lastWith.set(site, day); + } + }; + const unsure = (site: string, channelId: string, day: string): void => { + add(unknown, `${channelId}\0${day}`, site); + reached(site, day); + }; + const had = (site: string, channelId: string, day: string): void => { + add(answered, `${channelId}\0${day}`, site); + add(daysWith, site, day); + reached(site, day); + }; + + return { + hears: { + 'entry:fetched': (event) => { + if (event.programmes > 0) { + had(event.site, event.channelId, event.day); + + return; + } + + let sites = channels.get(event.channelId); + + if (sites === undefined) { + channels.set(event.channelId, (sites = new Map())); + } + + add(sites, event.site, event.day); + }, + 'entry:appended': (event) => { + // Said a second time by a document not grouped by channel, and no + // longer empty — as a run counts it. + if (event.total > 0) { + channels.get(event.channelId)?.get(event.site)?.delete(event.day); + had(event.site, event.channelId, event.day); + } + }, + // What the run cannot see into: taken as days that may have programmes, + // so nothing is claimed of them that they could make untrue. + 'entry:cached': (event) => { + unsure(event.site, event.channelId, event.day); + }, + 'entry:unchanged': (event) => { + unsure(event.site, event.channelId, event.day); + }, + 'entry:failed': (event) => { + unsure(event.site, event.channelId, event.day); + }, + 'site:failed': (event) => { + failed.add(event.site); + }, + // Every channel-day it covered, which no `entry:failed` says one by one. + 'request:failed': (event) => { + for (const channelId of event.channels) { + for (const day of event.days) { + unsure(event.site, channelId, day); + } + } + }, + }, + *lines() { + /** Every empty channel-day, then each source's: its site-wide reason, if it has one, and each one's. */ + const empties = [...channels].flatMap(([channelId, sites]) => + [...sites].flatMap(([site, these]) => [...these].map((day) => ({ channelId, site, day }))), + ); + const bySite = Map.groupBy(empties, (one) => one.site); + const days = empties.length; + + if (days === 0) { + yield 'No channel-day came back empty.'; + + return; + } + + yield `${count(days, 'channel-day')} came back empty:`; + + /** What is left once a source's own reason has taken its share: channel, source, why, day. */ + const left: Array<{ + channelId: string; + site: string; + why: EmptyWhy; + day: string; + others: string[]; + }> = []; + + for (const site of [...bySite.keys()].sort()) { + const list = bySite.get(site)!; + const last = lastWith.get(site); + + // Nothing from it at all: not a channel's problem, but the source's. + if (last === undefined) { + const named = new Set(list.map((one) => one.channelId)); + + yield ( + ` ${site}: every channel-day it was asked about came back empty — ${count(list.length, 'channel-day')} on ${count(named.size, 'channel')}: ` + + `its parseDay may no longer read what the source sends, or the source has nothing` + ); + continue; + } + + const after = list.filter((one) => one.day > last); + + if (after.length > 0) { + const named = new Set(after.map((one) => one.channelId)); + + yield ( + ` ${site}: nothing after ${last} on any channel — it has not published that far yet: ` + + `${spans([...new Set(after.map((one) => one.day))].sort())}, on ${count(named.size, 'channel')}` + ); + } + + for (const { channelId, day } of list) { + if (day > last) { + continue; + } + + const key = `${channelId}\0${day}`; + const others = [...(answered.get(key) ?? [])].filter((other) => other !== site).sort(); + // The sources that answered this channel-day with none, read off what + // is kept of each empty day rather than kept a second time. + const blank = [...(channels.get(channelId) ?? [])] + .filter(([, days]) => days.has(day)) + .map(([other]) => other); + + if (others.length > 0) { + left.push({ channelId, site, why: 'elsewhere', day, others }); + } else if ( + blank.some((other) => other !== site) && + ![...(unknown.get(key) ?? [])].some((other) => other !== site) + ) { + // Every source that answered: one that failed whole and said + // nothing of the day may have had it. + const heard = new Set([ + ...(answered.get(key) ?? []), + ...blank, + ...(unknown.get(key) ?? []), + ]); + const silent = [...failed].filter((other) => !heard.has(other)).sort(); + + left.push({ channelId, site, why: 'everywhere', day, others: silent }); + } else { + left.push({ channelId, site, why: 'gap', day, others: [] }); + } + } + } + + // Each channel and source's days of one reason as one line, in runs. + const lines = Map.groupBy(left, (one) => `${one.channelId}\0${one.site}\0${one.why}`); + + for (const key of [...lines.keys()].sort()) { + const these = lines.get(key)!; + const { channelId, site, why } = these[0]!; + const daysOf = these.map((one) => one.day).sort(); + const others = [...new Set(these.flatMap((one) => one.others))].sort(); + + yield ` ${channelId} (${site}): ${spans(daysOf)} — ${emptyBecause(why, site, others, daysOf, daysWith)}`; + } + }, + }; +} + +/** Why a run of one source's empty channel-days came back so, in words. */ +function emptyBecause( + why: EmptyWhy, + site: string, + others: readonly string[], + days: readonly string[], + daysWith: ReadonlyMap>, +): string { + switch (why) { + case 'elsewhere': + return ( + `${others.join(', ')} ${others.length === 1 ? 'has' : 'have'} programmes then: ` + + `${site} may know it by another id, or have stopped carrying it` + ); + // Of those asked: a source that carries it, and was not asked about the + // day — fewer `days`, or a turn cut short — may have it. + case 'everywhere': + return others.length === 0 + ? 'every source asked about it came back empty then: off air, or not published yet' + : `every source asked about it that answered came back empty then, though ` + + `${others.join(', ')} failed and may have it: off air, or not published yet`; + case 'gap': + return days.every((day) => daysWith.get(site)?.has(day)) + ? `${site} has programmes on other channels then: a gap in its listings, or off air` + : `nothing else says otherwise: off air, or a gap in ${site}'s listings`; + } +} + +/** Days, in order, as the runs they make: `2026-10-24..2026-10-26 (3 days), 2026-10-28`. */ +function spans(days: readonly string[]): string { + const runs: string[] = []; + let first = days[0]!; + let last = first; + + for (const day of days.slice(1)) { + if (day !== addDays(last, 1)) { + runs.push(span(first, last)); + first = day; + } + + last = day; + } + + runs.push(span(first, last)); + + return runs.join(', '); +} + +/** One run of days: the day, or `first..last (n days)`. */ +function span(first: string, last: string): string { + return first === last ? first : daySpan(first, last, diffDays(last, first) + 1); +} + +// ── requests ──────────────────────────────────────────────────────────────── + +/** What each part of a site's time is, in words — see `TimeSpent`. */ +const SPENT_ON: Record = { + source: 'on the source', + held: 'held by the source', + rateLimit: 'under its own rateLimit', + working: 'parsing and writing', + queued: 'waiting for a local slot', + other: 'on other things', +}; + +/** + * What a request's failure most likely means, by what it failed with — and, + * of a status, whether the site's backoff holds it for one: `holds`. + */ +function failureMeans(failure: string, holds: readonly number[]): string { + // Held for it, and failed all the same. + if (holds.includes(Number(failure))) { + return `were told to slow down (${failure}) — see --explain held`; + } + + switch (failure) { + case '404': + case '410': + return `were answered ${failure}: the source has nothing at those addresses — ids it no longer knows, or details of programmes it has dropped`; + case '401': + case '403': + return `were refused (${failure}): credentials, or a block`; + case '429': + return 'were told to slow down (429), which its backoff does not hold for: backoff, or a rateLimit, would pace it'; + case 'timeout': + return "took longer than the site's timeout"; + case 'network': + return 'were answered by nothing: the source down, or out of reach'; + case 'error': + return "threw something that was not the source's answer: the site's own code"; + default: + return /^5\d\d$/.test(failure) + ? `failed at the source (${failure})` + : `were answered ${failure}: the source would not give what was asked`; + } +} + +/** + * `requests`: every site's requests by what each was for — how many, how long + * they were out to the source, how long they waited in its queue first, and + * what failed them, as a table — then where each site's time went, what it + * spent most of it on and what would change that, and what its failures most + * likely mean. + */ +function requests(): Explanation { + /** Each site's `site:done`. */ + const sites = new Map>(); + const pace = new Pace(); + + return { + hears: { + ...pace.hears, + 'site:done': (event) => { + sites.set(event.site, event); + }, + }, + *lines(config) { + const rows: string[][] = []; + const none: string[] = []; + const names = [...sites.keys()].sort(); + + for (const site of names) { + const labels = Object.entries(sites.get(site)!.requestsBy); + + if (labels.length === 0) { + none.push(site); + } + + for (const [label, counts] of labels) { + rows.push([ + site, + label, + String(counts.requests), + took(counts.requestMs), + took(counts.waitedMs), + failures(counts.failed), + ]); + } + } + + if (sites.size === 0) { + yield 'No site was run.'; + + return; + } + + if (rows.length === 0) { + yield 'No site sent a request.'; + } else { + yield 'Requests, by site and what each was for:'; + yield* table([['site', 'label', 'requests', 'waiting', 'in queue', 'failed'], ...rows]); + + if (none.length > 0) { + yield ` None from ${none.join(', ')}.`; + } + } + + yield ''; + yield "Where each site's time went:"; + + for (const site of names) { + yield* spentBy(sites.get(site)!, config, pace.of(site)); + } + }, + }; +} + +/** One site's time, part by part, what it spent most of it on, and its failures read. */ +function* spentBy( + done: Extract, + config: EpgConfig, + told: Paced | undefined, +): Generator { + const parts = (Object.entries(done.msBy) as Array<[keyof TimeSpent, number]>) + .filter(([, ms]) => ms > 0) + .sort((a, b) => b[1] - a[1]); + + yield ( + ` ${done.site} took ${took(done.ms)}` + + (parts.length === 0 + ? '.' + : `: ${parts.map(([part, ms]) => `${took(ms)} ${SPENT_ON[part]}`).join(', ')}.`) + ); + + const [most] = parts; + + // Only where one part is most of a time worth asking about. + if (most !== undefined && done.ms >= 1_000 && most[1] * 2 >= done.ms) { + yield ` ${mostly(most[0], done, config, told)}`; + } + + const holds = heldFor(siteIn(config, done.site)?.backoff); + + for (const [label, counts] of Object.entries(done.requestsBy)) { + yield* failuresOf(label, counts, holds); + } +} + +/** + * What a site spending most of its time on `part` most likely means — of one + * the source held, or whose concurrency it took down, what it was told rather + * than that it could ask for more. + */ +function mostly( + part: keyof TimeSpent, + done: Extract, + config: EpgConfig, + told: Paced | undefined, +): string { + const site = siteIn(config, done.site); + + switch (part) { + case 'source': { + const at = site?.concurrency ?? 1; + + // Nothing to overlap, nor to be told to slow down for. + if (done.requests === 1) { + return 'Mostly waiting on the source, for its one request: the source is the slow part.'; + } + + if (told?.lowest !== undefined && told.lowest < at) { + return ( + `Mostly waiting on the source, its concurrency of ${at} taken as low as ${told.lowest} ` + + `by the source's slow-downs — see --explain held.` + ); + } + + // Told to slow down, so not to be asked any faster. + if (told !== undefined && told.holds > 0) { + return ( + `Mostly waiting on the source, ${at === 1 ? 'one request' : `${at} requests`} at a time, ` + + `and told to slow down ${often(told.holds)}: it would take no more — see --explain held.` + ); + } + + return at === 1 + ? 'Mostly waiting on the source, one request at a time: a concurrency above 1 would overlap them, where the source allows it.' + : `Mostly waiting on the source, ${at} requests at a time: the source is the slow part.`; + } + case 'held': + return 'Mostly held by the source: it is asked faster than it allows — see --explain held.'; + case 'rateLimit': + return site?.rateLimit === undefined + ? 'Mostly under its own rateLimit: the source may allow more.' + : `Mostly under its own rateLimit of ${site.rateLimit.requests} per ${site.rateLimit.perMs}ms: the source may allow more.`; + case 'working': + return config.threads + ? 'Mostly its own parsing and writing: a parse, or a cache, that is slow.' + : 'Mostly its own parsing and writing: a parse, or a cache, that is slow — with --threads, only the sites that share its thread wait on it.'; + case 'queued': { + const slots = localSlots(config.localConcurrency); + + return config.threads + ? `Mostly waiting for a local slot — one of its thread's ${slots}, or one of the run's thread's ${slots}, where its cache is read and written — which other work held: a higher localConcurrency lets it in sooner.` + : `Mostly waiting for one of the run's ${slots} local slots, which other sites' work held: a higher localConcurrency, or --threads, lets it in sooner.`; + } + case 'other': + return 'Mostly none of these: its own code between requests, or reading what it remembered.'; + } +} + +/** What each way a label's requests failed most likely means — see {@link failureMeans}. */ +function* failuresOf( + label: string, + counts: RequestCounts, + holds: readonly number[], +): Generator { + for (const [failure, n] of Object.entries(counts.failed)) { + yield ` ${n} of its ${counts.requests} ${label} requests ${failureMeans(failure, holds)}.`; + } +} + +/** + * Rows as columns, each as wide as its widest: text to the left, the counts and + * times — the third to fifth — to the right, and the last as it is. + */ +function* table(rows: readonly string[][]): Generator { + const widths = rows[0]!.map((_, column) => Math.max(...rows.map((row) => row[column]!.length))); + const last = widths.length - 1; + + for (const row of rows) { + const cells = row.map((cell, column) => + column === last + ? cell + : column >= 2 + ? cell.padStart(widths[column]!) + : cell.padEnd(widths[column]!), + ); + + yield ` ${cells.join(' ')}`.trimEnd(); + } +} + +// ── held ──────────────────────────────────────────────────────────────────── + +/** What a site was told about its pace — see {@link Pace}. */ +interface Paced { + /** How many times it was told to hold — a hold begun, or made longer — by the status it was told with. */ + statuses: Record; + /** How many holds it had, how many times one was made longer, and the longest it was told to hold for. */ + holds: number; + lengthened: number; + longestMs: number; + /** Times the source said how long for: as long as it asked, and cut short by `backoff.maxMs`. */ + asked: number; + capped: number; + /** The longest a capped hold asked for. */ + askedMs: number; + /** How low its concurrency was taken, and where it was left — if it was. */ + lowest?: number; + last?: number; +} + +/** + * What each site was told about its pace — every hold and slow-down — kept by + * each explanation that reads it, its `hears` spread into that one's: one way + * of counting them, wherever they are counted. + */ +class Pace { + readonly #sites = new Map(); + + readonly hears: Hears = { + 'pacing:held': (event) => { + const one = this.#paced(event.site); + const status = String(event.status); + + one.statuses[status] = (one.statuses[status] ?? 0) + 1; + // One hold made longer is not two. + if (event.lengthened) { + one.lengthened += 1; + } else { + one.holds += 1; + } + + one.longestMs = Math.max(one.longestMs, event.ms); + + if (event.retryAfterMs === undefined) { + return; + } + + if (event.retryAfterMs > event.ms) { + one.capped += 1; + one.askedMs = Math.max(one.askedMs, event.retryAfterMs); + } else { + one.asked += 1; + } + }, + 'pacing:slowed': (event) => { + const one = this.#paced(event.site); + + one.lowest = Math.min(one.lowest ?? Infinity, event.concurrency); + one.last = event.concurrency; + }, + 'pacing:recovered': (event) => { + this.#paced(event.site).last = event.concurrency; + }, + }; + + /** Each site told anything, by name. */ + get sites(): ReadonlyMap { + return this.#sites; + } + + /** What `site` was told, if anything. */ + of(site: string): Paced | undefined { + return this.#sites.get(site); + } + + #paced(site: string): Paced { + let one = this.#sites.get(site); + + if (one === undefined) { + this.#sites.set( + site, + (one = { + statuses: {}, + holds: 0, + lengthened: 0, + longestMs: 0, + asked: 0, + capped: 0, + askedMs: 0, + }), + ); + } + + return one; + } +} + +/** What a status a source holds a site with means. */ +const HELD_WITH: Record = { + 429: "too many requests — the source's rate limit", + 503: 'unavailable — overloaded, or a limit it does not name', +}; + +/** + * `held`: the sites a source told to slow down — how often, with what, and for + * how long at most — and how far that took their concurrency; then what each + * status means, whether it was held as long as the source asked, and the pace + * that earned it beside the `rateLimit` that would have kept under it. + */ +function held(): Explanation { + const pace = new Pace(); + /** Each site's pace, kept for whichever is held — known only once it is done. */ + const done = new Map(); + + return { + hears: { + ...pace.hears, + 'site:done': (event) => { + done.set(event.site, { requests: event.requests, outMs: event.msBy.source }); + }, + }, + *lines(config) { + if (pace.sites.size === 0) { + yield 'No site was told to slow down.'; + + return; + } + + yield `${count(pace.sites.size, 'site')} told to slow down:`; + + for (const site of [...pace.sites.keys()].sort()) { + yield* heldBy(site, pace.of(site)!, done.get(site), config); + } + }, + }; +} + +/** How often: `once`, `twice`, `3 times`. */ +function often(n: number): string { + return n === 1 ? 'once' : n === 2 ? 'twice' : `${n} times`; +} + +/** + * How many requests a site sent, and how long it had any out to the source + * and was not held — the pace it was sending at, which earned the holds: the + * holds themselves, a wait for a local slot, or its own parsing, say nothing + * of it. + */ +interface Sent { + requests: number; + outMs: number; +} + +/** One site's holds, and what they most likely mean. */ +function* heldBy( + name: string, + paced: Paced, + done: Sent | undefined, + config: EpgConfig, +): Generator { + const { statuses, holds, lengthened, longestMs, asked, capped, askedMs, lowest, last } = paced; + /** Each time it was told to hold: a hold begun, or one made longer. */ + const times = holds + lengthened; + + yield ( + ` ${name}: held ${often(holds)}` + + (lengthened > 0 + ? `, made longer ${often(lengthened)} while it held — for up to ${took(longestMs)} each time it was told` + : holds === 1 + ? `, for ${took(longestMs)}` + : `, for up to ${took(longestMs)} at a time`) + + (lowest === undefined + ? '' + : `; concurrency down to ${lowest}` + + (last !== undefined && last > lowest ? `, back up to ${last}` : '')) + ); + + for (const [status, n] of Object.entries(statuses)) { + yield ` ${n} × ${status}: ${HELD_WITH[status] ?? 'a status its backoff.statuses lists'}`; + } + + const unsaid = times - asked - capped; + + if (asked > 0) { + yield ` As long as the source asked, ${asked === times ? 'each time' : often(asked)}.`; + } + + if (capped > 0) { + yield ( + ` Cut short by backoff.maxMs ${often(capped)}, where the source asked for up to ` + + `${took(askedMs)}: let go sooner than it wanted, the site is likely to be held again.` + ); + } + + if (unsaid > 0) { + yield ` For the fallback ${often(unsaid)} — backoff.fallbackMs, at most backoff.maxMs — the source saying nothing of how long.`; + } + + const site = siteIn(config, name); + + if (done !== undefined && done.outMs > 0) { + const pace = (done.requests * 1000) / done.outMs; + const rate = pace >= 10 ? pace.toFixed(0) : pace.toFixed(1); + + yield ( + ` It sent ${count(done.requests, 'request')}, ${rate} a second over the ${took(done.outMs)} it had any out unheld, ` + + (site?.rateLimit === undefined + ? 'with no rateLimit: a rateLimit below that pace would keep it from being held, at the cost of the time that takes.' + : `under a rateLimit of ${site.rateLimit.requests} per ${site.rateLimit.perMs}ms: the source allows less than that.`) + ); + } +} diff --git a/src/cli/run.ts b/src/cli/run.ts index 32f4e3c..65ea18e 100644 --- a/src/cli/run.ts +++ b/src/cli/run.ts @@ -36,10 +36,11 @@ import { type Emit, type EventLevel, } from '../core/events.js'; -import { FAILURE_MODES, reporterFor, REPORTER_NAMES } from '../core/reporters.js'; +import { combineReporters, FAILURE_MODES, reporterFor, REPORTER_NAMES } from '../core/reporters.js'; import { drain, writeFlushed, writeLines } from '../core/streams.js'; import { initGrabber } from './scaffold.js'; import { REPORT_FORMATS, type ReportFormat } from './format.js'; +import { EXPLAINABLE, explainCommands, explanations } from './explain.js'; import { OUTPUT_PROFILE_NAMES } from '../xmltv/profile.js'; import type { OutputProfileName } from '../xmltv/profile.js'; import { outputOptions } from '../xmltv/serialize.js'; @@ -82,6 +83,15 @@ Options: a number of channel-days, or a share like 5% --dry-run grab/build only: say what a run would fetch, and stop. --format json for a machine-readable one + --explain Once the run is done, list on stderr what it said of + one thing, and what that most likely means. Repeatable. + build/merge: unmatched — the programmes that nearly + merged with another source's, and what would join them. + build/grab: empty — the channel-days a source answered + with nothing, and why; requests — each site's requests + and where its time went; held — the sites a source told + to slow down, and the pace that earned it. Its events + as JSON: --reporter json --log-level debug --extensions build/merge/serve/filter: keep only these provider extensions, comma-separated (e.g. lcn,uniqueID) --no-extensions build/merge/serve/filter: leave every provider @@ -720,6 +730,9 @@ async function execute( 'grab-at': { type: 'string', transform: timeOfDay }, raw: { type: 'boolean' }, format: { type: 'string', choices: REPORT_FORMATS }, + // Repeatable, and each for the commands whose run it is about: what a + // run said of one thing, explained once it is done. + explain: { type: 'string', choices: EXPLAINABLE, multiple: true }, strict: { type: 'boolean' }, // `epg channels` only: the file naming the channels somebody wants a // guide for, and whether a mismatch should fail a CI step. @@ -781,6 +794,30 @@ async function execute( throw new UsageError(`Unknown command: ${command}`); } + // Before anything a command does: an option it would only pass over is a + // mistake to say rather than to swallow. + if (values.explain !== undefined) { + for (const name of values.explain) { + const commands = explainCommands(name); + + if (!commands.includes(command)) { + throw new UsageError(`--explain ${name} is for ${commands.join(', ')}, not ${command}`); + } + } + + if (values['dry-run']) { + throw new UsageError('--explain is for a run, not --dry-run'); + } + + // On stderr, beside what else a run says there: what reads it is a person. + // A program has every one of them as an event of its own. + if (values.format === 'json') { + throw new UsageError( + '--explain is for reading — for its events as JSON: --reporter json --log-level debug', + ); + } + } + // Before the config is even looked for: a guide named on the command line is // the whole of what this needs, and refusing to read one because there is no // project in the working directory would be absurd. Without a name it is the @@ -936,8 +973,12 @@ async function execute( // disjoint: `merge:done` is the command's, everything inside a run is not. const { reporter: said } = reportingTo(chosen ? { reporter: chosen } : {}, config.events); const emit = said ? stamped(said) : silent; + + // What `--explain` asked for, kept as the run says it. + const { reporter: collector, render: renderExplanations } = explanations(values.explain ?? []); + const reporter = collector ? combineReporters(chosen, collector) : chosen; const runOptions = { - ...(chosen ? { reporter: chosen } : {}), + ...(reporter ? { reporter } : {}), ...(values.offset !== undefined ? { offset: values.offset } : {}), ...(signal ? { signal } : {}), // The file this loaded, which is what a site's thread imports to find its @@ -1003,17 +1044,26 @@ async function execute( } emit({ type: 'merge:done', output: String(config.output) }); + await renderExplanations(stderr, config); return exitCode(summary, config.allowMissing); } case 'grab': { const summary = await runGrab(config, runOptions); - return signal?.aborted ? EXIT_CANCELLED : exitCode(summary, config.allowMissing); + if (signal?.aborted) { + return EXIT_CANCELLED; + } + + await renderExplanations(stderr, config); + + return exitCode(summary, config.allowMissing); } case 'merge': { await runMerge(config, runOptions); + emit({ type: 'merge:done', output: String(config.output) }); + await renderExplanations(stderr, config); return 0; } diff --git a/src/config.ts b/src/config.ts index 140a174..00c41ce 100644 --- a/src/config.ts +++ b/src/config.ts @@ -21,6 +21,7 @@ import type { EpgServeConfig } from './serve/config.js'; import type { AnySiteConfig } from './grabber/types.js'; import type { DerivedChannels, GuideMeta, MergeOptions } from './merge/types.js'; import type { SerializeOptions } from './xmltv/serialize.js'; +import type { Reporter } from './core/events.js'; import type { EpgEventSink, ReporterFactory, ReporterName } from './core/reporters.js'; /** @@ -266,12 +267,24 @@ export interface EpgConfig { * export default defineConfig({ sites, output, events }); * ``` * + * Or a reporter of your own, told as a run's is — what an emitter that + * hears more than its listeners do is given as, since nothing else here can + * say so of it: + * + * ```ts + * export default defineConfig({ + * sites, + * output, + * events: emitterReporter(bridge, { wants: () => true }), + * }); + * ``` + * * Honoured wherever a run is started from this config — `epg serve` and * `epg grab`, and `build`, `runGrab`, `runMerge` and `createGuideHandler` * just the same. A library caller's own `reporter` is kept beside it rather * than replaced. */ - events?: EpgEventSink; + events?: EpgEventSink | Reporter; } /** Builds the config from whatever answered the questions. */ diff --git a/src/core/events.ts b/src/core/events.ts index 21a626e..6843f34 100644 --- a/src/core/events.ts +++ b/src/core/events.ts @@ -167,7 +167,77 @@ export function noCounts(): GrabCounts { /** A label's requests at nought, before any is counted — see {@link RequestCounts}. */ export function noRequests(): RequestCounts { - return { requests: 0, requestMs: 0, failed: {} }; + return { requests: 0, requestMs: 0, waitedMs: 0, failed: {} }; +} + +/** + * Where a site's `ms` went — `site:done`'s `msBy`. At each moment the site was + * in the first of these that held, so the parts never overlap and add up to + * `ms`: a site that took 170 seconds over two requests answered in under one + * says here what it was doing with the rest. + */ +export interface TimeSpent { + /** + * At least one of its requests out, waiting on the source, and none held — + * once however many were: the wall clock they covered, where `requestMs` + * adds up each one's own, and is the same only for a site that sends one at + * a time. A streaming site's pass reading its document is the source's too, + * where nothing else of the site's holds: its body arriving long after its + * headers did, and split as it does — the two are one, and not told apart. + */ + source: number; + /** + * Its requests held, the source having said slow down — a `429` or `503` — + * with one of them outstanding: waiting its turn, or out, which under a + * hold is one ky retries once the source's `Retry-After` is up, or one sent + * before it that the hold holds up as much. First of them all, so a site + * asking one at a time is seen to be held, and not waiting on the source. + */ + held: number; + /** None out, and waiting on its own `rateLimit` to let the next one go. */ + rateLimit: number; + /** Its own work on this machine under way: a `parseDay`, a write to the cache, a check of what is cached. */ + working: number; + /** None of that under way, and some of it waiting for one of the run's `localConcurrency` slots. */ + queued: number; + /** None of these: its own code between them, reading what it remembered, its turn's beginning and end. */ + other: number; +} + +/** The parts of {@link TimeSpent} a run measures — `other` is what is left of `ms`. */ +export type MeasuredTime = Exclude; + +/** Every part of {@link TimeSpent} a run measures — which a moment is given to is `SiteClock`'s to say. */ +export const MEASURED_TIME = [ + 'source', + 'held', + 'rateLimit', + 'working', + 'queued', +] as const satisfies readonly MeasuredTime[]; + +/** + * What kept two programmes from being one — `merge:unmatched`'s `why`. + * + * Under the shipped match: `title` — they started near enough, and their + * titles differ, which `normalizeTitle` decides; `start` — their titles agree, + * and they start too far apart, which `startToleranceMs` decides; `both`; and + * `runs` — their titles agree and they start near enough, but further apart + * than the shorter of them lasts: two programmes in a row, or one within the + * other, as a segment is within its show. Under the config's own + * `match`: `matcher` — it said no, for reasons of its own — or, for one with a + * `windowMs`, `start`: they start beyond it, where it is never asked. + */ +export type NearMissWhy = 'title' | 'start' | 'both' | 'runs' | 'matcher'; + +/** One side of a `merge:unmatched` pair. */ +export interface UnmatchedProgramme { + /** The site it came from. */ + site: string; + /** Its title — the first it has. */ + title: string; + /** Its start, as the guide writes it: `20261024200500 +0200`. */ + start: string; } /** @@ -193,9 +263,19 @@ export interface RequestCounts { /** * How long they spent waiting on the source, pacing not included: the * label's share of `site:done`'s `requestMs`, and nothing like its `ms`, - * which is the site's wall clock. + * which is the site's wall clock. Each request's own time, added up — four + * in flight together for a second each are four seconds here, and one + * second of `msBy.source`. */ requestMs: number; + /** + * How long they waited in the site's queue before each was sent: for the + * source to stop holding them, for its `rateLimit`, or for one of its + * `concurrency` slots — added up as `requestMs` is. A parse's `paced` calls go + * ahead of the planned requests, so where both are waiting it is the planned + * ones whose `waitedMs` grows. + */ + waitedMs: number; /** * The ones that failed, by what failed them: the status the source answered * with (`'404'`), `'timeout'`, `'network'` where nothing answered, or @@ -286,6 +366,10 @@ export type EpgEventInput = * `requestsBy` is `requests` and `requestMs` again, by what each request was * for, with what failed them — see {@link RequestCounts}. "5291 requests" * does not say that 3,242 of them were detail calls answered `404`. + * + * `msBy` is `ms` again, by what the site was doing — see {@link TimeSpent}. + * A site whose `ms` is far above what its requests explain was waiting on + * something else, and this says which. */ | ({ type: 'site:done'; @@ -294,6 +378,7 @@ export type EpgEventInput = ms: number; requestMs: number; requestsBy: Record; + msBy: TimeSpent; } & GrabCounts) /** The site could not be read or run at all, so none of it was grabbed. */ | { type: 'site:failed'; site: string; error: unknown } @@ -356,13 +441,53 @@ export type EpgEventInput = /** Emissions nobody asked about. Diagnostic only; a count is plenty. */ | { type: 'stream:ignored'; site: string; count: number } // ── pacing ─────────────────────────────────────────────────────────────── - /** Told to slow down, and holding every request for `ms`. */ - | { type: 'pacing:held'; site: string; status: number; ms: number } + /** + * Told to slow down, and holding every request for `ms`. `retryAfterMs` is + * how long the source's `Retry-After` asked for, before `backoff.maxMs` had + * its say — absent where it sent none, and `ms` is then `backoff.fallbackMs`, + * or `backoff.maxMs` where that is the less. `lengthened` where it is not a + * hold begun but one already holding, made to end later by another response + * that asked for longer. + */ + | { + type: 'pacing:held'; + site: string; + status: number; + ms: number; + retryAfterMs?: number; + lengthened?: true; + } | { type: 'pacing:slowed'; site: string; concurrency: number } | { type: 'pacing:recovered'; site: string; concurrency: number } | { type: 'pacing:rateLimit'; site: string; waiting: boolean } // ── merging, and tidying up ────────────────────────────────────────────── | { type: 'merge:channel'; channelId: string } + /** + * Two programmes from different sources that nearly merged, and did not: + * one that joined nothing, and the nearest unmatched one already there, + * within half an hour, from another source — one before it in the merge, or + * one whose programme was held over from the day before, whatever its place. + * Each of what `merge:summary` counts as a + * near miss — at `debug`, since a guide whose sources run on two grids has + * thousands. `--explain unmatched`, on `epg merge` or `epg build`, lists + * them all, and what would join them. + */ + | { + type: 'merge:unmatched'; + channelId: string; + /** + * How much later `programme` starts than `nearby` — below nought where it + * starts earlier — so which of two sources runs behind is read off it, + * not worked out again from the starts as written. + */ + shiftMs: number; + /** What kept them apart — see {@link NearMissWhy}. */ + why: NearMissWhy; + /** The one that joined nothing. */ + programme: UnmatchedProgramme; + /** The unmatched one already there, nearest it. */ + nearby: UnmatchedProgramme; + } /** * What the merge made of the sources, once the whole guide is out. * @@ -379,9 +504,15 @@ export type EpgEventInput = */ | { type: 'merge:summary'; - /** Programmes a later source joined to one already there. */ + /** + * Programmes one source joined to another's already there — not to its + * own, held over from the day before and said again. + */ combined: number; - /** Programmes that joined nothing, with something close by. */ + /** + * Programmes that joined nothing, with another source's within half an + * hour left unjoined too — each one a `merge:unmatched`. + */ nearMiss: number; /** A few of them, in full — a count alone says nothing about why. */ examples: { @@ -492,6 +623,7 @@ export const EVENT_KINDS = { 'pacing:recovered': { level: 'info', phase: 'grab' }, 'pacing:rateLimit': { level: 'debug', phase: 'grab' }, 'merge:channel': { level: 'debug', phase: 'merge' }, + 'merge:unmatched': { level: 'debug', phase: 'merge' }, 'merge:summary': { level: 'info', phase: 'merge' }, 'merge:done': { level: 'info', phase: 'merge' }, 'merge:note': { level: 'info', phase: 'merge' }, @@ -506,6 +638,12 @@ export const EVENT_KINDS = { 'serve:stopped': { level: 'info', phase: 'serve' }, } as const satisfies Record; +/** Every kind of event, as {@link EVENT_KINDS} lists them. */ +export const EVENT_TYPES = Object.keys(EVENT_KINDS) as EpgEventType[]; + +/** Every kind, as a set — what a reporter that did not say wants. */ +const EVERY_KIND: ReadonlySet = new Set(EVENT_TYPES); + /** * What a reporter is given: what happened, and how loudly. * @@ -538,11 +676,47 @@ export type EpgEventMap = { * on a sink, and both commands end by draining what they queued rather than * awaiting each line — see `streams.ts`. A reporter that wants to do something * slow should queue it and be drained by whatever owns it. + * + * `wants` says which kinds of event it does anything with — what its level + * shows, or what it counts — so that a run need not make an event nobody will + * read: what it makes often, an `entry:fetched` a channel-day or a + * `merge:unmatched` a programme, is made only where a reporter wants it. + * Without one, a reporter wants every kind, as a sink of one's own is told + * everything. Asked of each kind once, the first time a run asks — see + * {@link stamped} — and never trusted to filter: a reporter is still handed + * whatever is made. */ -export type Reporter = (event: EpgEvent) => void; +export type Reporter = ((event: EpgEvent) => void) & { + wants?: (type: EpgEventType) => boolean; +}; + +/** + * What the run says, before it is stamped — and `hears`, the kinds of event + * its reporters want, as they said it the first time the run asked: every + * kind, where they did not say. What a run makes often, it makes only where `hears` has + * it. One set, whether the run is here or a site's thread. Made by + * {@link emitter}. + */ +export type Emit = ((event: EpgEventInput) => void) & { + readonly hears: ReadonlySet; +}; -/** What the run says, before it is stamped. */ -export type Emit = (event: EpgEventInput) => void; +/** + * An {@link Emit}: what it says events with, and `hears` — the set itself, or + * what answers it as it is asked, for one taken the first time it is, or that + * is another's. The one way one is made, so a set asked for later is asked + * for the same way wherever it is. + */ +export function emitOf( + say: (event: EpgEventInput) => void, + hears: ReadonlySet | (() => ReadonlySet), +): Emit { + return Object.defineProperty( + say, + 'hears', + typeof hears === 'function' ? { get: hears } : { value: hears }, + ) as Emit; +} /** * Stamp what an emitter said, and pass it on. @@ -553,7 +727,20 @@ export type Emit = (event: EpgEventInput) => void; * the type came from. */ export function stamped(reporter: Reporter): Emit { - return (event) => reporter({ ...event, ...EVENT_KINDS[event.type] } as EpgEvent); + let hears: ReadonlySet | undefined; + + return emitOf( + (event) => reporter({ ...event, ...EVENT_KINDS[event.type] } as EpgEvent), + // What it wants, asked of each kind once, the first time the run asks — + // never before its first wait, so a listener added as it is started is + // asked about too — and held to from then on. Asked of the reporter, as + // its own: a `wants` that reads `this` reads it. + () => + (hears ??= + reporter.wants === undefined + ? EVERY_KIND + : new Set(EVENT_TYPES.filter((type) => reporter.wants!(type)))), + ); } /** How often the loop is sampled — see {@link watchEventLoop}. */ @@ -614,7 +801,7 @@ export async function watchEventLoop(): Promise<{ } /** A run nobody is listening to. */ -export const silent: Emit = () => {}; +export const silent: Emit = emitOf(() => {}, new Set()); /** What a caller says when asked where the events go. */ export interface ReportedOptions { @@ -816,6 +1003,20 @@ export function keepingThrows(options: T): KeptThrows } }; + // Asked as the reporter is told, and what it throws kept as what that + // throws is: a reporter that cannot say what it wants is given everything. + if (reporter.wants !== undefined) { + keepingReporter.wants = (type) => { + try { + return reporter.wants!(type); + } catch (error) { + keep(error, { type } as EpgEvent); + + return true; + } + }; + } + keeping.add(keepingReporter); return { @@ -882,31 +1083,53 @@ function isThenable(value: unknown): value is PromiseLike { } /** - * The options a run reports through, with the config's own emitter added. + * The options a run reports through, with the config's own emitter — or + * reporter — added. * * `reporter` is how a run talks to whoever started it, and a library caller * passes its own. `events` is what the *program* asked to be told, and it is * in the config because that is where a program says it — so every way of * starting a run honours it, not only the command that happens to read the - * config file. + * config file. A reporter there is joined as it is: one that says what it + * wants, which an emitter given alone could not. */ export function reportingTo( options: T, - events: EpgEventSink | undefined, + events: EpgEventSink | Reporter | undefined, ): T { if (events === undefined) { return options; } - return { ...options, reporter: combineReporters(options.reporter, emitterReporter(events)) }; + return { + ...options, + reporter: combineReporters( + options.reporter, + typeof events === 'function' ? events : emitterReporter(events), + ), + }; } /** * What {@link emitterReporter} needs of an emitter — Node's own fits, and so - * does anything with an `emit` taking a name and a value. + * does anything with an `emit` taking a name and a value. One that can say how + * many listen for a kind is asked, so a run need not make what none hears. */ export interface EpgEventSink { emit(type: EpgEvent['type'], event: EpgEvent): unknown; + listenerCount?(type: EpgEvent['type']): number; +} + +/** How {@link emitterReporter} says what its emitter hears. */ +export interface EmitterReporterOptions { + /** + * The kinds the emitter hears, asked of each once a run asks — for one + * whose listeners are not all that hear: an `emit` of its own that passes + * every event on, or a listener for any kind. By default, the kinds something + * listens for, where the emitter can say how many do, and otherwise every + * kind. + */ + wants?: (type: EpgEventType) => boolean; } /** @@ -927,12 +1150,31 @@ export interface EpgEventSink { * * Typing the emitter with {@link EpgEventMap} is what makes each listener's * argument the event it asked for. Nothing is filtered: an emitter with no - * listener for a type is the cheapest sink there is. + * listener for a type is the cheapest sink there is — and, asked by its + * `listenerCount`, one a run need not make that event for at all. One that + * hears more than its listeners do says so with `options.wants`: + * + * ```ts + * emitterReporter(bridge, { wants: () => true }); + * ``` */ -export function emitterReporter(events: EpgEventSink): Reporter { - return (event) => { +export function emitterReporter( + events: EpgEventSink, + options: EmitterReporterOptions = {}, +): Reporter { + const reporter: Reporter = (event) => { events.emit(event.type, event); }; + + // Asked once a run first asks: a listener added part way through one hears + // the next. + if (options.wants !== undefined) { + reporter.wants = (type) => options.wants!(type); + } else if (events.listenerCount !== undefined) { + reporter.wants = (type) => events.listenerCount!(type) > 0; + } + + return reporter; } /** @@ -1002,7 +1244,22 @@ export function combineReporters(...reporters: (Reporter | undefined)[]): Report }); }; - return combined; + // Wanted by any of them — every kind by one that did not say, and none by + // none at all. Asked only until one wants it: the rest are given it then + // whatever they would say. One that throws is thrown through, to what keeps + // a run's throws — which takes it for one that wants it, as one is given + // everything — so it is asked no further either. + return Object.assign(combined, { + wants: (type: EpgEventType) => { + for (const reporter of listening) { + if (reporter.wants?.(type) ?? true) { + return true; + } + } + + return false; + }, + }); } /** diff --git a/src/core/reporters.ts b/src/core/reporters.ts index fa9961c..cdd5e93 100644 --- a/src/core/reporters.ts +++ b/src/core/reporters.ts @@ -20,12 +20,15 @@ import type { Writable } from 'node:stream'; import { inspect } from 'node:util'; import { atLevel, + EVENT_KINDS, noCounts, type EpgEvent, type EventLevel, type GrabCounts, + type NearMissWhy, type RequestCounts, type Reporter, + type UnmatchedProgramme, } from './events.js'; import { errorChain, errorMessage, MAX_CAUSE_DEPTH } from './error.js'; import { queueLine } from './streams.js'; @@ -69,7 +72,7 @@ export type ReporterFactory = (runtime: ReporterRuntime) => Reporter; * seconds and minutes above it, because `2543871ms` is a number nobody * subtracts in their head. */ -function took(ms: number): string { +export function took(ms: number): string { if (ms < 10_000) { return `${String(ms)}ms`; } @@ -105,12 +108,6 @@ function cost(event: { */ function byLabel(requestsBy: Record): string { const labels = Object.entries(requestsBy); - // " and " between failures, so the commas are left to separate labels. - // `failed` read as `requestsBy` was, where an event from elsewhere may lack it. - const failures = (failed: Record | undefined): string => - Object.entries(failed ?? {}) - .map(([failure, n]) => `${n} × ${failure}`) - .join(' and '); if (labels.length === 1) { const failed = failures(labels[0]![1].failed); @@ -131,6 +128,17 @@ function byLabel(requestsBy: Record): string { return ` (${each.join(', ')})`; } +/** + * A label's failed requests by what failed them — `3242 × 404 and 3 × timeout`, + * or nothing. " and " between them, so commas are left to separate labels; + * `failed` read as `requestsBy` is, where an event from elsewhere may lack it. + */ +export function failures(failed: Record | undefined): string { + return Object.entries(failed ?? {}) + .map(([failure, n]) => `${n} × ${failure}`) + .join(' and '); +} + /** What a run or a site came to, in the one form both say it. */ function counts({ fetched, empty, fromCache, unchanged, failed, sitesFailed }: GrabCounts): string { // A day that came back with nothing is not a failure — a channel with no @@ -159,13 +167,16 @@ function counts({ fetched, empty, fromCache, unchanged, failed, sitesFailed }: G function describeSpan(channels: string[], days: string[]): string { const channelPart = channels.length === 1 ? channels[0]! : count(channels.length, 'channel'); const dayPart = - days.length === 1 - ? days[0]! - : `${days[0]}..${days[days.length - 1]} (${count(days.length, 'day')})`; + days.length === 1 ? days[0]! : daySpan(days[0]!, days[days.length - 1]!, days.length); return `${channelPart} ${dayPart}`; } +/** A run of days as every line that names one says it: `2026-10-24..2026-10-26 (3 days)`. */ +export function daySpan(first: string, last: string, days: number): string { + return `${first}..${last} (${count(days, 'day')})`; +} + /** `[site] ` unless a reporter was told not to. */ function at(site: string, prefix: boolean): string { return prefix ? `[${site}] ` : ''; @@ -247,7 +258,9 @@ export function render(event: EpgEvent, prefix = true): string | undefined { `answered in ${event.ms}ms` ); case 'pacing:held': - return `${at(event.site, prefix)}HTTP ${event.status}: holding requests for ${event.ms}ms`; + return event.lengthened + ? `${at(event.site, prefix)}HTTP ${event.status}: holding requests longer, for ${event.ms}ms from now` + : `${at(event.site, prefix)}HTTP ${event.status}: holding requests for ${event.ms}ms`; case 'pacing:slowed': return `${at(event.site, prefix)}concurrency down to ${event.concurrency}`; case 'pacing:recovered': @@ -263,6 +276,13 @@ export function render(event: EpgEvent, prefix = true): string | undefined { // No site on either, so no prefix: a merge transform is the config's own // code, and there is nobody to attribute it to. return `${event.message}${said(event.data)}`; + case 'merge:unmatched': + // Which two, how far apart, and what kept them so — with whose each is, + // and when it starts by its own clock, to find them by in the guide. + return ( + `${event.channelId}: ${took(Math.abs(event.shiftMs))} apart, ${UNMATCHED_WHY[event.why]} — ` + + `${side(event.programme)} / ${side(event.nearby)}` + ); case 'merge:summary': { // Only what there is: a guide whose sources all line up says one number, // and a run that dropped nothing does not report two noughts every night. @@ -320,6 +340,43 @@ export function render(event: EpgEvent, prefix = true): string | undefined { } } +/** What kept two programmes apart, in words — see `merge:unmatched`, and `--explain unmatched`. */ +export const UNMATCHED_WHY = { + title: 'titles differ', + start: 'starts too far apart', + both: 'titles and starts differ', + runs: 'they start further apart than the shorter lasts', + matcher: "the config's match said no", +} as const satisfies Record; + +/** One side of a near miss: its title, its site, and when it starts by its own clock. */ +function side(programme: UnmatchedProgramme): string { + return `"${programme.title}" (${programme.site}, ${clockOf(programme.start)})`; +} + +/** + * When a start written as the guide writes it is, by its own clock: `20:05` + * of `20261024200500 +0200` — or as much of it as the source said, which may + * be the hour (`20:00`), the day (`2026-10-24`), the month (`2026-10`) or the + * year alone (`2026`). + */ +function clockOf(start: string): string { + const [digits = ''] = start.split(' '); + + if (digits.length >= 10) { + return `${digits.slice(8, 10)}:${digits.length >= 12 ? digits.slice(10, 12) : '00'}`; + } + + if (digits.length >= 8) { + return `${digits.slice(0, 4)}-${digits.slice(4, 6)}-${digits.slice(6, 8)}`; + } + + return digits.length >= 6 ? `${digits.slice(0, 4)}-${digits.slice(4, 6)}` : digits.slice(0, 4); +} + +/** The kinds of event that are a failure — see {@link isFailure}. */ +const FAILURES = new Set(['entry:failed', 'request:failed', 'site:failed']); + /** The three events that are a failure, and what one reads as. */ export type FailureEvent = Extract< EpgEvent, @@ -328,9 +385,7 @@ export type FailureEvent = Extract< /** Whether this is one of those, so a reporter can hold it back. */ export function isFailure(event: EpgEvent): event is FailureEvent { - return ( - event.type === 'entry:failed' || event.type === 'request:failed' || event.type === 'site:failed' - ); + return FAILURES.has(event.type); } /** @@ -442,14 +497,21 @@ export function loggerReporter(options: LoggerReporterOptions): Reporter { const { logger, level = 'info', prefix = true } = options; const cause = atLevel('debug', level); - return (event) => { - if (isFailure(event)) { - logger.error(renderFailure(event, prefix, cause)); + // Its failures whatever the level, and the rest by it. + const shows = (type: EpgEvent['type'], at: EventLevel): boolean => + FAILURES.has(type) || atLevel(at, level); + const reporter: Reporter = (event) => { + // Nothing it says it does not want, whether or not it was asked first — + // by the event's own level, so one of a kind this version does not know, + // read back from a later one's log, is told by it as it always was. + if (!shows(event.type, event.level)) { return; } - if (!atLevel(event.level, level)) { + if (isFailure(event)) { + logger.error(renderFailure(event, prefix, cause)); + return; } @@ -477,10 +539,14 @@ export function loggerReporter(options: LoggerReporterOptions): Reporter { } } }; + + reporter.wants = (type) => shows(type, EVENT_KINDS[type].level); + + return reporter; } export { combineReporters, emitterReporter } from './events.js'; -export type { EpgEventSink } from './events.js'; +export type { EmitterReporterOptions, EpgEventSink } from './events.js'; export interface TextReporterOptions { /** Where progress goes. */ @@ -508,6 +574,16 @@ export interface TextReporterOptions { /** How many failures a block shows before counting the rest. */ export const DEFAULT_FAILURE_CAP = 20; +/** + * What a text reporter at `level` writes or keeps of an event of `type` at + * `at`: its failures whatever the level, what ends a run — the block flushed, + * or dropped — and the rest by it. + */ +function textShows(level: EventLevel): (type: EpgEvent['type'], at: EventLevel) => boolean { + return (type, at) => + FAILURES.has(type) || type === 'run:cancelled' || type === 'grab:done' || atLevel(at, level); +} + /** * Lines of text, which is what a person reads and what a CI log keeps. * @@ -547,7 +623,15 @@ export function textReporter(options: TextReporterOptions): Reporter { dropped = 0; }; - return (event) => { + const shows = textShows(level); + + const reporter: Reporter = (event) => { + // Nothing it says it does not want, whether or not it was asked first — + // by the event's own level: see `loggerReporter`. + if (!shows(event.type, event.level)) { + return; + } + if (isFailure(event)) { const line = renderFailure(event, prefix, cause); @@ -588,6 +672,10 @@ export function textReporter(options: TextReporterOptions): Reporter { flush(); } }; + + reporter.wants = (type) => shows(type, EVENT_KINDS[type].level); + + return reporter; } /** An error as JSON keeps it, `cause` chain and all. */ @@ -635,7 +723,9 @@ export interface JsonReporterOptions { export function jsonReporter(options: JsonReporterOptions): Reporter { const { stream, level = 'debug', pretty = false } = options; - return (event) => { + const reporter: Reporter = (event) => { + // Nothing it says it does not want, whether or not it was asked first — + // by the event's own level: see `loggerReporter`. if (!atLevel(event.level, level)) { return; } @@ -654,6 +744,10 @@ export function jsonReporter(options: JsonReporterOptions): Reporter { queueLine(stream, line); }; + + reporter.wants = (type) => atLevel(EVENT_KINDS[type].level, level); + + return reporter; } /** @@ -665,16 +759,30 @@ export function jsonReporter(options: JsonReporterOptions): Reporter { */ const ENDS = new Set(['grab:done', 'run:cancelled', 'merge:done', 'prune:done']); +/** What the live line counts — see {@link progressReporter}. */ +const COUNTED = new Set([ + 'site:started', + 'site:done', + 'entry:fetched', + 'entry:cached', + 'entry:unchanged', + 'entry:failed', + 'site:failed', + 'request:failed', + 'merge:channel', +]); + +/** + * What the live line draws again rather than lets the lines write. Only kinds + * it wants, shown at `info` or counted: nothing else reaches it — a `debug` + * kind it neither shows nor counts is turned away first, redrawn for nothing. + */ const SUMMARIZED = new Set([ 'site:started', 'site:done', 'entry:cached', 'entry:fetched', - 'entry:appended', 'entry:unchanged', - 'request:started', - 'request:done', - 'stream:ignored', 'merge:channel', ]); @@ -784,7 +892,19 @@ export function progressReporter(options: ProgressReporterOptions): Reporter { shown = next; }; - return (event) => { + // What the lines underneath show, what the live line counts, and what ends + // a half of the run: the rest it would only draw again, as it was. + const linesShow = textShows(level); + const shows = (type: EpgEvent['type'], at: EventLevel): boolean => + linesShow(type, at) || COUNTED.has(type) || ENDS.has(type); + + const reporter: Reporter = (event) => { + // Nothing it says it does not want, whether or not it was asked first — + // by the event's own level: see `loggerReporter`. + if (!shows(event.type, event.level)) { + return; + } + switch (event.type) { case 'site:started': sites.add(event.site); @@ -847,6 +967,10 @@ export function progressReporter(options: ProgressReporterOptions): Reporter { lines(event); paint(); }; + + reporter.wants = (type) => shows(type, EVENT_KINDS[type].level); + + return reporter; } /** diff --git a/src/grabber/clock.ts b/src/grabber/clock.ts new file mode 100644 index 0000000..c524086 --- /dev/null +++ b/src/grabber/clock.ts @@ -0,0 +1,434 @@ +/** + * Where a site's wall clock goes — `site:done`'s `msBy`, measured as it runs. + * + * At any moment a site is in the first of these that holds: held, the source + * having said slow down, with a request of its outstanding — waiting its turn, + * or out, which under a hold is a request ky is retrying once the source's + * `Retry-After` is up, or one sent before it that the hold holds up as much; + * a request of its out to the source; one waiting on its own `rateLimit`; its + * local work under way; its local work waiting for a slot; a streaming site's + * pass reading its document, given to the source too — its body arriving and + * split as it does, which is the one request's, though its headers came long + * since. A hold with nothing outstanding holds nothing up: the site's own work + * goes on, and is what it is doing. + * Each change to what holds gives the time since the last change to whichever + * held until it, so the parts never overlap — and what none of them covers is + * `other`, the rest of `ms`. + * + * Measured only while the site is on the clock that `ms` is: its turn, and a + * request it asked for outside one — its channel list, settled before the + * grab. A hold that outlives the list it was answered to, waiting for the turn, + * is time `ms` does not count either. + */ + +import { performance } from 'node:perf_hooks'; +import { MEASURED_TIME, type MeasuredTime, type TimeSpent } from '../core/events.js'; + +/** Told of each stretch of time as it is given to a part — a thread's shared memory. */ +export type TimeSink = (part: MeasuredTime, ms: number) => void; + +/** The measured parts at nought. */ +export function noTime(): Record { + return { source: 0, held: 0, rateLimit: 0, working: 0, queued: 0 }; +} + +/** + * What a moment is given to: a part — or `elsewhere`, local work waiting on the + * run's thread with nothing else of the site's under way, given to `working` + * until the answer says how much of it was a wait for a slot there. + */ +type Holding = MeasuredTime | 'elsewhere'; + +/** A stretch of time, `performance.now()` to `performance.now()`. */ +type Stretch = [from: number, to: number]; + +export class SiteClock { + readonly #spent = noTime(); + readonly #sink: TimeSink | undefined; + /** + * What the time since {@link #since} is being given to — nothing, where none + * of the parts holds, or the site is off the clock. + */ + #part: Holding | undefined; + /** When it began to be. */ + #since = 0; + /** Whether the site's turn is under way. */ + #turn = false; + /** Requests asked for outside a turn and not yet settled: what puts the site on the clock then. */ + #asked = 0; + /** Requests waiting in the site's queue, and out to the source. */ + #waiting = 0; + #out = 0; + #held = false; + #rateLimited = false; + /** Its local work under way, and waiting for a slot. */ + #working = 0; + #queued = 0; + /** + * Of its work under way, the calls waiting on the run's thread — a thread's + * cache call — by when each went. + */ + readonly #calls: number[] = []; + /** + * The stretches given to `working` while those calls were all the site was + * doing, and the stretches the calls back from there ran in, each as + * `from, to` — kept until no call still out could change what they say. + */ + #alone: Stretch[] = []; + #ran: Stretch[] = []; + /** Its passes reading their documents — see {@link reading}. */ + #reading = 0; + /** Whether the run was cancelled: what is waiting then never begins. */ + #cancelled = false; + + constructor(sink?: TimeSink) { + this.#sink = sink; + } + + /** + * What each part has been given so far, in milliseconds — up to now, so the + * stretch under way is given out first, and a thread's shared memory told of + * it as of any other. + */ + spent(): Record { + this.#close(); + + return { ...this.#spent }; + } + + /** The site's turn begins, or ends. */ + turn(on: boolean): void { + this.#turn = on; + this.#moved(); + } + + /** + * A request asked for, waiting in the site's queue until it is {@link sent}. + * Outside the site's turn it is also what puts the site on the clock — and + * this answers whether it did, so that it is {@link settled} once it is, and + * a request in a turn costs nothing more. + */ + ask(): boolean { + // Dropped by the cancel the moment it is asked, and never sent. + if (this.#cancelled) { + return false; + } + + this.#waiting++; + + if (this.#turn) { + this.#moved(); + + return false; + } + + this.#asked++; + this.#moved(); + + return true; + } + + /** A request {@link ask} counted outside a turn, settled — answered, or dropped before it was sent. */ + settled(): void { + this.#asked--; + this.#moved(); + } + + /** A request that was waiting, sent. */ + sent(): void { + this.#waiting--; + this.#out++; + this.#moved(); + } + + /** A request that was out, answered. */ + answered(): void { + this.#out--; + this.#moved(); + } + + /** The source holding the site's requests, or letting them go. */ + held(on: boolean): void { + this.#held = on; + this.#moved(); + } + + /** + * A streaming site's pass beginning to read its document, or done with it: + * the source's, wherever nothing else holds. + */ + reading(on: boolean): void { + this.#reading += on ? 1 : -1; + this.#moved(); + } + + /** The site's `rateLimit` holding the next request back, or letting it go. */ + rateLimited(on: boolean): void { + this.#rateLimited = on; + this.#moved(); + } + + /** Local work waiting for a slot. */ + queued(): void { + if (this.#cancelled) { + return; + } + + this.#queued++; + this.#moved(); + } + + /** Local work that was waiting for a slot, under way. */ + started(): void { + this.#queued--; + this.#working++; + this.#moved(); + } + + /** Local work under way, done. */ + finished(): void { + this.#working--; + this.#moved(); + } + + /** + * Local work under way that has gone to wait on the run's thread — a + * thread's cache call — until it is {@link back}: answering when it went, + * for that. + */ + elsewhere(): number { + const since = performance.now(); + + this.#calls.push(since); + this.#moved(); + + return since; + } + + /** + * Local work back from the run's thread, gone `since`, which says the first + * `queuedMs` of its call was a wait for a slot there and the rest its own + * work there. Of what `working` was given while work waiting there was all + * the site was doing, a moment no call ran in was every call there waiting + * for a slot, and is given to `queued` — two calls there at once are not one's + * wait during the other's work. A moment the site's other work was under way + * here stays that work's. + */ + back(since: number, queuedMs: number): void { + const at = this.#calls.indexOf(since); + + // Only a call still out: one back twice is back once. + if (at === -1) { + return; + } + + const now = this.#close(); + const ran = Math.min(since + queuedMs, now); + + if (ran < now) { + this.#ran.push([ran, now]); + } + + this.#calls.splice(at, 1); + this.#settle(); + this.#moved(); + } + + /** + * The run cancelled. What waits for a slot of the site's queue, or of the + * run's local work, is dropped by the same cancel and never begins, nor does + * anything asked after it: none of it waits any more. Nor does a hold, or a + * rate limit, hold anything up. + */ + cancelled(): void { + this.#cancelled = true; + this.#waiting = 0; + this.#queued = 0; + this.#held = false; + this.#rateLimited = false; + this.#moved(); + } + + /** + * After a change, what holds now — and where that is not what held before, + * the time since given to what did. Read only then: while a request is out, + * a parse beginning or ending changes nothing, and asks the clock nothing. + */ + #moved(): void { + const part = this.#turn || this.#asked > 0 ? this.#holding() : undefined; + + if (part !== this.#part) { + this.#close(); + this.#part = part; + } + } + + /** + * What no call still out on the run's thread can change, settled: the time + * before the earliest of them went, when every call there was one now back + * and has said when it ran. A moment of it no call ran in is given to + * `queued`. + */ + #settle(): void { + const upTo = this.#calls.length === 0 ? Infinity : Math.min(...this.#calls); + // In the order they began, so that a sweep along them counts a moment two + // ran in at once only once. + const ran = this.#ran.length > 1 ? this.#ran.toSorted((a, b) => a[0] - b[0]) : this.#ran; + const alone: Stretch[] = []; + let waited = 0; + + for (const [from, to] of this.#alone) { + if (from >= upTo) { + alone.push([from, to]); + continue; + } + + const until = Math.min(to, upTo); + + waited += until - from - this.#covered(from, until, ran); + + if (to > upTo) { + alone.push([upTo, to]); + } + } + + this.#alone = alone; + this.#ran = this.#ran.filter(([, to]) => to > upTo); + + if (waited > 0) { + this.#spent.working -= waited; + this.#spent.queued += waited; + this.#sink?.('working', -waited); + this.#sink?.('queued', waited); + } + } + + /** How much of `from` to `to` the stretches `ran`, in the order they began, cover. */ + #covered(from: number, to: number, ran: readonly Stretch[]): number { + let ms = 0; + let reached = from; + + for (const [start, end] of ran) { + const since = Math.max(start, reached); + const until = Math.min(end, to); + + if (until > since) { + ms += until - since; + reached = until; + } + } + + return ms; + } + + /** + * The stretch since the last change given to what held through it, and a new + * one begun — answering when. + */ + #close(): number { + const now = performance.now(); + const ms = now - this.#since; + + // Nothing to give for a stretch of no time — two changes in one instant. + if (this.#part !== undefined && ms > 0) { + const part = this.#part === 'elsewhere' ? 'working' : this.#part; + + if (this.#part === 'elsewhere') { + const last = this.#alone.at(-1); + + // Split only by a call back while another is still out. + if (last?.[1] === this.#since) { + last[1] = now; + } else { + this.#alone.push([this.#since, now]); + } + } + + this.#spent[part] += ms; + this.#sink?.(part, ms); + } + + this.#since = now; + + return now; + } + + /** The first part that holds, or none. */ + #holding(): Holding | undefined { + if (this.#held && (this.#waiting > 0 || this.#out > 0)) { + return 'held'; + } + + if (this.#out > 0) { + return 'source'; + } + + if (this.#waiting > 0 && this.#rateLimited) { + return 'rateLimit'; + } + + // Some of its work under way here, not only waiting on the run's thread — + // read off two counts, each call there taken for one of the work counted + // here. A parse that runs on without its slot, given up for a `paced` it + // never awaited, is not counted: while it writes there, another parse's + // work reads as a wait. + if (this.#working > this.#calls.length) { + return 'working'; + } + + if (this.#calls.length > 0) { + return 'elsewhere'; + } + + if (this.#queued > 0) { + return 'queued'; + } + + // Below a wait for a slot: a pass held back while its writes wait for one + // is waiting for it too. + return this.#reading > 0 ? 'source' : undefined; + } +} + +/** + * `ms` as `site:done` says it, with `measured` given out of it whole and what + * is left as `other` — `ms` taken up to what was measured where a clock read on + * another thread, or rounded, comes to a little more. + */ +export function timeSpent( + ms: number, + measured: Record, +): { ms: number; msBy: TimeSpent } { + const parts = noTime(); + let total = 0; + + for (const part of MEASURED_TIME) { + parts[part] = Math.round(measured[part]); + total += parts[part]; + } + + const whole = Math.max(Math.round(ms), total); + + return { ms: whole, msBy: { ...parts, other: whole - total } }; +} + +/** Two measures of a site's time, part by part. */ +export function addedTime( + a: Record, + b: Record, +): Record { + const sum = noTime(); + + for (const part of MEASURED_TIME) { + sum[part] = a[part] + b[part]; + } + + return sum; +} + +/** The measured parts of `msBy`, without `other` — what two of them are added from. */ +export function measuredOf(msBy: TimeSpent): Record { + const { other: _other, ...measured } = msBy; + + return measured; +} diff --git a/src/grabber/grab.ts b/src/grabber/grab.ts index 08bdeae..9993fbb 100644 --- a/src/grabber/grab.ts +++ b/src/grabber/grab.ts @@ -23,6 +23,7 @@ import { type RequestSink, type SiteSession, } from './session.js'; +import { noTime } from './clock.js'; import { SiteRun, type Run } from './site-run.js'; import { siteName } from './site.js'; import { SiteNotRunError, SiteThreads, threadCount } from './thread.js'; @@ -36,6 +37,14 @@ import type { AnySiteConfig, GrabOptions, GrabSummary } from './types.js'; */ const DEFAULT_LOCAL_CONCURRENCY = 16; +/** + * How many slots the run's local work has: `localConcurrency`, at least one — + * what a run is given, and what `--explain` says it had. + */ +export function localSlots(localConcurrency: number | undefined): number { + return Math.max(1, localConcurrency ?? DEFAULT_LOCAL_CONCURRENCY); +} + export async function grab(configs: AnySiteConfig[], options: GrabOptions): Promise { return grabWith(configs, options, new Map()); } @@ -130,7 +139,7 @@ async function grabbing( * throttled by them, nor take a request's slot. */ const localWork = new PQueue({ - concurrency: Math.max(1, options.localConcurrency ?? DEFAULT_LOCAL_CONCURRENCY), + concurrency: localSlots(options.localConcurrency), }); // Watched for the whole run, because the number it produces is the one that @@ -180,6 +189,10 @@ async function grabbing( enqueue, tally, requested, + // Nothing to keep of a site's time as it goes: each site's is read off its + // own clock as its turn ends — and a run's is no sum of its sites', which + // run side by side. + spent: undefined, }; // Said first, and the way a site says it when its list fails in its own turn, @@ -204,6 +217,7 @@ async function grabbing( // What its list cost — the `404` it was answered with included. session?.requestsBy ?? {}, { ...noCounts(), sitesFailed: 1 }, + session?.clock.spent() ?? noTime(), ), ); }; diff --git a/src/grabber/pacing.ts b/src/grabber/pacing.ts index 21f1536..28beef8 100644 --- a/src/grabber/pacing.ts +++ b/src/grabber/pacing.ts @@ -11,6 +11,7 @@ import type { KyInstance } from 'ky'; import PQueue from 'p-queue'; import { silent, type Emit } from '../core/events.js'; import { siteHttp } from './channels.js'; +import type { SiteClock } from './clock.js'; import type { AnySiteConfig, SiteBackoff } from './types.js'; const DEFAULT_BACKOFF: Required = { @@ -22,6 +23,11 @@ const DEFAULT_BACKOFF: Required = { adapt: true, }; +/** The statuses a site's `backoff` holds it for — none, where it is off. */ +export function heldFor(backoff: false | SiteBackoff | undefined): readonly number[] { + return backoff === false ? [] : (backoff?.statuses ?? DEFAULT_BACKOFF.statuses); +} + /** * Clean responses in a row before the concurrency lost to a slow-down is given * back, one at a time — additive increase after a multiplicative decrease, so @@ -104,8 +110,20 @@ export function sitePacing( signal?: AbortSignal; /** Say what the queue did — see `core/events.ts`. */ emit?: Emit; + /** Told when the site is held, or waits on its `rateLimit` — see `msBy`. */ + clock?: SiteClock; } = {}, ): SitePacing { + // Said here rather than by the queue's own listener mid-grab, where a throw + // ends the process: an `emit` from a JavaScript caller, or a cast, may be a + // plain function, which says nothing of what is wanted. Asked by `in`, so a + // run's set is not taken before the run asks for it. + if (options.emit !== undefined && !('hears' in options.emit)) { + throw new TypeError( + `sitePacing for ${config.site} was given an emit with no hears: make it with emitter({ reporter })`, + ); + } + const emit = options.emit ?? silent; const queue = requestQueue(config); const backoff = config.backoff === false ? undefined : { ...DEFAULT_BACKOFF, ...config.backoff }; @@ -120,6 +138,7 @@ export function sitePacing( const release = (): void => { timer = undefined; holdingUntil = 0; + options.clock?.held(false); queue.start(); }; @@ -131,8 +150,9 @@ export function sitePacing( * alone — it is a response, not a mistake — and nothing is dropped: the * queue keeps its tasks and hands them out again on release. */ - const hold = (ms: number, status: number): void => { - const until = Date.now() + ms; + const hold = (ms: number, status: number, asked: number | undefined): void => { + const now = Date.now(); + const until = now + ms; // A longer hold already covers this one; two 429s in flight together must // not shorten the wait to the second one's. @@ -140,10 +160,21 @@ export function sitePacing( return; } + // Said, so that one hold made longer is not read as two. + const lengthened = now < holdingUntil; + holdingUntil = until; clearTimeout(timer); queue.pause(); - emit({ type: 'pacing:held', site: config.site, status, ms: Math.round(ms) }); + options.clock?.held(true); + emit({ + type: 'pacing:held', + site: config.site, + status, + ms: Math.round(ms), + ...(asked === undefined ? {} : { retryAfterMs: Math.round(asked) }), + ...(lengthened ? { lengthened: true } : {}), + }); // Deliberately not unref'd: a run in the middle of a hold has to stay // alive, or the process would exit with the guide half grabbed. Save once // the site is let go of: a hold then is a request its parse did not wait @@ -190,12 +221,13 @@ export function sitePacing( }); } + // What the source asked for, said beside what it is given. + const asked = retryAfterMs(response.headers.get('retry-after')); + hold( - Math.min( - retryAfterMs(response.headers.get('retry-after')) ?? backoff.fallbackMs, - backoff.maxMs, - ), + Math.min(asked ?? backoff.fallbackMs, backoff.maxMs), response.status, + asked, ); return; @@ -223,12 +255,21 @@ export function sitePacing( // Only ever fires for a site that is paced, and it is the one thing a // `rateLimit` gives no other sign of. `addEventListener` rather than `on`: // p-queue 10 is an `EventTarget`. - queue.addEventListener('rateLimit', () => - emit({ type: 'pacing:rateLimit', site: config.site, waiting: true }), - ); - queue.addEventListener('rateLimitCleared', () => - emit({ type: 'pacing:rateLimit', site: config.site, waiting: false }), - ); + queue.addEventListener('rateLimit', () => { + options.clock?.rateLimited(true); + + // Twice a window, for a site paced tight. + if (emit.hears.has('pacing:rateLimit')) { + emit({ type: 'pacing:rateLimit', site: config.site, waiting: true }); + } + }); + queue.addEventListener('rateLimitCleared', () => { + options.clock?.rateLimited(false); + + if (emit.hears.has('pacing:rateLimit')) { + emit({ type: 'pacing:rateLimit', site: config.site, waiting: false }); + } + }); // A cancelled run must not sit out the rest of a hold — the timer is not // unref'd, so it would hold the process open — and must not resume either. @@ -242,10 +283,17 @@ export function sitePacing( clearTimeout(timer); timer = undefined; holdingUntil = 0; + options.clock?.cancelled(); }; options.signal?.addEventListener('abort', cancelled, { once: true }); + // Cancelled before the site was even set up — a thread told so as it read + // its config — which no listener added now will hear. + if (options.signal?.aborted) { + cancelled(); + } + return { queue, http, diff --git a/src/grabber/session.ts b/src/grabber/session.ts index eb07931..b689885 100644 --- a/src/grabber/session.ts +++ b/src/grabber/session.ts @@ -16,12 +16,16 @@ import type { CacheStore } from '../cache/types.js'; import { causes, failureOf } from '../core/error.js'; import { + emitOf, noRequests, + silent, type Emit, type EpgEventInput, type GrabCounts, + type MeasuredTime, type RequestCounts, } from '../core/events.js'; +import { SiteClock, timeSpent, type TimeSink } from './clock.js'; import { sitePacing, type SitePacing } from './pacing.js'; import { isUnchanged } from './revalidate.js'; import { SiteStateHandle } from './state.js'; @@ -55,6 +59,12 @@ export interface SiteSession { * where its requests were all it did, as a list settled before the grab's. */ readonly spanMs: number; + /** + * Where the site's time goes, its requests' share of it measured here, its + * pacing's by {@link pacing}, and the rest by whoever runs its turn — see + * `site:done`'s `msBy`. + */ + readonly clock: SiteClock; /** * One request through the queue, counted and timed under `label` — handed * the queue's own signal, which is the run's. @@ -73,9 +83,10 @@ export interface SiteSession { ): Promise; /** * Be part of `run` from here: its `requested` told what has been counted so - * far, and of every request after, and its `emit` what the session says — for - * a session opened before the run it reports to, as one settling a channel - * list is. + * far, and of every request after, and its `emit` what the session says — + * for a session opened before the run it reports to, as one settling a + * channel list is. Its time is read off its {@link clock} by whoever says + * the site is done. */ joinRun(run: { requested: RequestSink; emit: Emit }): void; } @@ -99,6 +110,7 @@ export function addRequests( total.requests += done.requests; total.requestMs += done.requestMs; + total.waitedMs += done.waitedMs; for (const [failure, n] of Object.entries(done.failed)) { total.failed[failure] = (total.failed[failure] ?? 0) + n; @@ -128,14 +140,20 @@ export function totalOf(counted: Iterable): { requests: number; r return { requests, requestMs }; } -/** A site's `site:done`: its requests added up from `requestsBy`, beside its channel-days. */ +/** + * A site's `site:done`: its requests added up from `requestsBy`, beside its + * channel-days, and its wall clock `ms` with what of it was `measured` — see + * `timeSpent`. + */ export function siteDone( site: string, - ms: number, + wall: number, requestsBy: Record, counts: GrabCounts, + measured: Record, ): Extract { const { requests, requestMs } = totalOf(Object.values(requestsBy)); + const { ms, msBy } = timeSpent(wall, measured); // The counts and nothing else, whatever else `counts` holds — an event. const { fetched, empty, fromCache, unchanged, failed, sitesFailed } = counts; @@ -146,6 +164,7 @@ export function siteDone( ms, requestMs, requestsBy, + msBy, fetched, empty, fromCache, @@ -173,9 +192,9 @@ export function mergedRequests( /** What `by` holds, as an event carries it: copies, which go on saying what they said. */ export function requestsObject(by: Map): Record { return Object.fromEntries( - [...by].map(([label, { requests, requestMs, failed }]) => [ + [...by].map(([label, { requests, requestMs, waitedMs, failed }]) => [ label, - { requests, requestMs, failed: { ...failed } }, + { requests, requestMs, waitedMs, failed: { ...failed } }, ]), ); } @@ -225,13 +244,21 @@ export function openSession( emit?: Emit; /** The run's, told of each request as it is done — see {@link SiteSession.joinRun}. */ counted?: RequestSink; + /** Told of the site's time as it is spent: a thread's shared memory — see `Run.spent`. */ + spent?: TimeSink | undefined; } = {}, ): SiteSession { /** Whoever is told what the session says: the run's, once it has joined one. */ let say = options.emit; + const clock = new SiteClock(options.spent); const pacing = sitePacing(config, { ...(options.signal ? { signal: options.signal } : {}), - emit: (event) => say?.(event), + // To whoever is told now, and wanting what they want. + emit: emitOf( + (event) => say?.(event), + () => (say ?? silent).hears, + ), + clock, }); /** By label, in the order each was first sent — which is the order they are reported in. */ const by = new Map(); @@ -267,22 +294,31 @@ export function openSession( get spanMs() { return asked === undefined || answered === undefined ? 0 : answered - asked; }, + clock, joinRun(run) { tellAll(run.requested, by); sink = run.requested; say = run.emit; }, send: (task, { label, ...sending }) => { - asked ??= Date.now(); + const queued = Date.now(); + // Waiting its turn — and outside a turn, what puts the site on the clock + // until it is settled. + const counted = clock.ask(); + + asked ??= queued; // The run's signal on every task, as the run's own queues have it: a // waiting request is dropped on a cancel rather than started. - return pacing.queue.add( + const sent = pacing.queue.add( async (slot) => { - // Sent now — and so listed in the order each label was first sent. - count(label, { requests: 1, requestMs: 0, failed: {} }); - const began = Date.now(); + + // Sent now — and so listed in the order each label was first sent, + // with how long it waited for its turn. + count(label, { requests: 1, requestMs: 0, waitedMs: began - queued, failed: {} }); + clock.sent(); + let failure: string | undefined; try { @@ -292,16 +328,27 @@ export function openSession( throw error; } finally { + clock.answered(); answered = Date.now(); count(label, { requests: 0, requestMs: answered - began, + waitedMs: 0, failed: failure === undefined ? {} : { [failure]: 1 }, }); } }, { ...sending, ...(options.signal ? { signal: options.signal } : {}) }, ); + if (counted) { + // Settled however it went — answered, or dropped by a cancel before + // its turn — and no longer keeping the site on the clock. + const settled = (): void => clock.settled(); + + void sent.then(settled, settled); + } + + return sent; }, }; } diff --git a/src/grabber/shared-counts.ts b/src/grabber/shared-counts.ts index e4bbdf0..b10db26 100644 --- a/src/grabber/shared-counts.ts +++ b/src/grabber/shared-counts.ts @@ -3,34 +3,38 @@ * thread that dies has lost none of what it counted, and nothing of the * counting crosses as a message but each request count's name, once. * - * When its thread began on it takes the first slot, and its channel-day counts - * — `GrabCounts` — the ones after: always the same slots, which need no name, - * and which the run's thread sees as soon as they are written — so a thread - * that dies leaves no doubt whether the site had begun, or had counted itself - * failed. After them, a label is given two slots - * side by side when it is first counted, for the - * two counts every label has — its `requests`, then its `requestMs` — and a - * kind of failure one of its own when it first fails that way: those are the - * names said. The site's thread alone adds to them, and the run's reads them - * when the site is done, or its thread gone. Counts it has no slot left for — - * a site of more labels and failures than {@link SLOTS} holds — are handed - * back for the thread to say another way. + * When its thread began on it takes the first slot, its channel-day counts + * — `GrabCounts` — the ones after, and where its time went — `msBy`, in + * microseconds — the ones after those: always the same slots, which need no + * name, and which the + * run's thread sees as soon as they are written — so a thread that dies leaves + * no doubt whether the site had begun, or had counted itself failed. After + * them, a label is given three slots side by side when it is first counted, + * for the three counts every label has — its `requests`, its `requestMs`, then + * its `waitedMs` — and a kind of failure one of its own when it first fails + * that way: those are the names said. The site's thread alone adds to them, + * and the run's reads them when the site is done, or its thread gone. Counts + * it has no slot left for — a site of more labels and failures than + * {@link SLOTS} holds — are handed back for the thread to say another way. + * + * Its time is added as everything else is, with `Atomics`, though that makes a + * `BigInt` of every stretch its clock gives out: about 65ns a stretch, against + * 2ns for a float added without — measured, some 1.4ms of a grab of 7,000 + * channel-days one request at a time, which is none worth one rule less. */ -import { noRequests, type GrabCounts, type RequestCounts } from '../core/events.js'; +import { + MEASURED_TIME, + noRequests, + type GrabCounts, + type MeasuredTime, + type RequestCounts, +} from '../core/events.js'; import { addRequests } from './session.js'; /** How many counts a site's memory holds: far more than any site names. */ export const SLOTS = 1024; -/** Where in a site's memory each thing is that is always there, before any label's slots. */ -const Slot = { - /** When the site's thread began on it — nought, until it has. */ - BEGAN: 0, - /** Where the channel-day counts start, in the order of {@link TALLIED}. */ - TALLY: 1, -} as const; - /** * The channel-day counts, in the order of their slots from {@link Slot.TALLY}: * every one of `GrabCounts`, which `count` and `tally` do not compile without. @@ -44,9 +48,33 @@ const TALLIED = [ 'sitesFailed', ] as const satisfies ReadonlyArray; +/** Where in a site's memory each thing is that is always there, before any label's slots. */ +const Slot = { + /** When the site's thread began on it — nought, until it has. */ + BEGAN: 0, + /** Where the channel-day counts start, in the order of {@link TALLIED}. */ + TALLY: 1, + /** Where its time starts, in the order of `MEASURED_TIME`, in microseconds. */ + SPENT: 1 + TALLIED.length, +} as const; + /** - * What a slot counts: a label — its `requests` there, and its `requestMs` in - * the slot after — or, with `failure`, how many of its requests failed so. + * Each channel-day count by its slot's place after {@link Slot.TALLY}, and each + * part of a site's time by its place after {@link Slot.SPENT} — found once + * rather than searched for at every count. `Map`s, so a name that is none — + * `constructor`, which every object has — finds nothing. + */ +const TALLY_AT: ReadonlyMap = new Map( + TALLIED.map((field, at) => [field, at]), +); +const SPENT_AT: ReadonlyMap = new Map( + MEASURED_TIME.map((part, at) => [part, at]), +); + +/** + * What a slot counts: a label — its `requests` there, its `requestMs` in the + * slot after and its `waitedMs` in the one after that — or, with `failure`, + * how many of its requests failed so. */ export interface SlotName { label: string; @@ -64,7 +92,7 @@ export class CountWriter { readonly #view: BigInt64Array; readonly #name: (index: number, slot: SlotName) => void; /** - * Where each label's two counts are — nowhere, once there was no room — and + * Where each label's three counts are — nowhere, once there was no room — and * each of its kinds of failure: looked up by the label's own string, so a * slot already named costs a request nothing to find. */ @@ -72,8 +100,8 @@ export class CountWriter { string, { at: number | undefined; failed: Map } >(); - /** The first slot nothing counts in yet: when it began, and the channel-day counts, have theirs. */ - #free = Slot.TALLY + TALLIED.length; + /** The first slot nothing counts in yet: when it began, its channel-day counts and its time have theirs. */ + #free = Slot.SPENT + MEASURED_TIME.length; constructor(buffer: SharedArrayBuffer, name: (index: number, slot: SlotName) => void) { this.#view = new BigInt64Array(buffer); @@ -87,16 +115,28 @@ export class CountWriter { /** Add `delta` to a channel-day count — less, as an empty day that then is not. */ count(field: keyof GrabCounts, delta: number): void { - const at = TALLIED.indexOf(field); + const at = TALLY_AT.get(field); - // Said, rather than added to the slot before the first: when it began. - if (at === -1) { + // Said, rather than added to whatever slot `NaN` comes to: when it began. + if (at === undefined) { throw new TypeError(`No channel-day count is called ${String(field)}`); } Atomics.add(this.#view, Slot.TALLY + at, BigInt(delta)); } + /** Add `ms` to a part of the site's time — see `SiteClock`. */ + spend(part: MeasuredTime, ms: number): void { + const at = SPENT_AT.get(part); + + // As for a count: said, rather than added to when it began. + if (at === undefined) { + throw new TypeError(`No part of a site's time is called ${String(part)}`); + } + + Atomics.add(this.#view, Slot.SPENT + at, BigInt(Math.round(ms * 1000))); + } + /** * Add what `done` says of `label` — and answer what there was no slot for, * where anything, for the caller to say another way. @@ -108,7 +148,7 @@ export class CountWriter { let slots = this.#slots.get(label); if (slots === undefined) { - slots = { at: this.#take(2), failed: new Map() }; + slots = { at: this.#take(3), failed: new Map() }; this.#slots.set(label, slots); if (slots.at !== undefined) { @@ -117,10 +157,16 @@ export class CountWriter { } if (slots.at === undefined) { - left = { requests: done.requests, requestMs: done.requestMs, failed: {} }; + left = { + requests: done.requests, + requestMs: done.requestMs, + waitedMs: done.waitedMs, + failed: {}, + }; } else { Atomics.add(this.#view, slots.at, BigInt(done.requests)); Atomics.add(this.#view, slots.at + 1, BigInt(done.requestMs)); + Atomics.add(this.#view, slots.at + 2, BigInt(done.waitedMs)); } for (const failure in done.failed) { @@ -205,6 +251,16 @@ export class CountReader { ) as Record<(typeof TALLIED)[number], number>; } + /** Where the site's time went, as far as its thread said, in milliseconds. */ + spent(): Record { + return Object.fromEntries( + MEASURED_TIME.map((part, at) => [ + part, + Number(Atomics.load(this.#view, Slot.SPENT + at)) / 1000, + ]), + ) as Record; + } + /** * What the site counted, by label, in the order each was first counted. A * slot whose name has not been heard is not read: there is no saying what @@ -220,8 +276,13 @@ export class CountReader { by, slot.label, slot.failure === undefined - ? { requests: at(index), requestMs: at(index + 1), failed: {} } - : { requests: 0, requestMs: 0, failed: { [slot.failure]: at(index) } }, + ? { + requests: at(index), + requestMs: at(index + 1), + waitedMs: at(index + 2), + failed: {}, + } + : { requests: 0, requestMs: 0, waitedMs: 0, failed: { [slot.failure]: at(index) } }, ); }); diff --git a/src/grabber/site-run.ts b/src/grabber/site-run.ts index 57d057d..dc7cfd0 100644 --- a/src/grabber/site-run.ts +++ b/src/grabber/site-run.ts @@ -28,6 +28,7 @@ import { noCounts, type Emit, type GrabCounts } from '../core/events.js'; import { ProgrammeBuilder } from '../xmltv/builder.js'; import type { XmltvProgramme } from '../xmltv/types.js'; import { resolveChannels } from './channels.js'; +import type { SiteClock, TimeSink } from './clock.js'; import { parseContext, requestContext, streamContext, type ContextDeps } from './context.js'; import { LocalSlot } from './local-slot.js'; import { planRequests, type Pair, type Request } from './planner.js'; @@ -98,6 +99,14 @@ export interface Run { * has still said what it asked. */ requested: RequestSink; + /** + * Told of the site's time as it is spent — see `site:done`'s `msBy`. A site + * in a thread's into the memory both threads see, so one whose thread dies + * has said where its time went. Said, every run, if only as `undefined`: a + * site on this thread's is read off its own clock as its turn ends, and has + * nothing to keep as it goes — which costs its clock nothing. + */ + spent: TimeSink | undefined; } /** A parse may hand back either form; the cache only knows the object. */ @@ -215,8 +224,13 @@ export class SiteRun { this.#site = resolved.site; this.#revalidates = config.conditionalGet === true; this.#says = { - log: (message, data) => - run.emit({ type: 'site:note', site: resolved.site, message, ...(data ? { data } : {}) }), + // A site may log at every step, with fields of its own: nothing made of + // it where nobody reads it — nor, from a thread, sent. + log: (message, data) => { + if (run.emit.hears.has('site:note')) { + run.emit({ type: 'site:note', site: resolved.site, message, ...(data ? { data } : {}) }); + } + }, warn: (message, data) => run.emit({ type: 'site:warning', site: resolved.site, message, ...(data ? { data } : {}) }), }; @@ -237,6 +251,7 @@ export class SiteRun { ...(run.signal ? { signal: run.signal } : {}), emit: run.emit, counted: run.requested, + spent: run.spent, }); this.#http = this.#session.pacing.http; this.#state = this.#session.state; @@ -304,6 +319,7 @@ export class SiteRun { */ async run(): Promise { this.#began = Date.now(); + this.#session.clock.turn(true); let failed = false; try { @@ -320,10 +336,8 @@ export class SiteRun { this.#count('sitesFailed'); this.#run.emit({ type: 'site:failed', site: this.#site, error }); } finally { - this.#session.pacing.dispose(); - - // Beside `dispose`, and for the same reason: a site that threw, or was - // cancelled part way, has as much to hand back as one that finished — the + // A site that threw, or was cancelled part way, has as much to hand back + // as one that finished — the // channel list it fetched, and whatever its own code remembered. Only the // groups that changed are written. What the site remembered is pruned // first, so a site that no longer asks stops keeping what it kept. @@ -345,24 +359,39 @@ export class SiteRun { this.#count('sitesFailed'); this.#run.emit({ type: 'site:failed', site: this.#site, error }); } + } finally { + // Let go of only once the turn's last wait is over: until then a + // cancel is still the site's to hear — its pacing lets go of a hold, + // and its clock of what the cancel dropped — however the turn ended, + // and whatever saying so above threw. + this.#session.pacing.dispose(); } // For a failed site too, with what it did count — nought, where it // failed before it began — and beside the `site:failed` that says why. + // + // Wall clock, which is the one a nightly run is long by — and nothing + // like its `requestMs` once a site runs several requests at once. + const ms = Date.now() - this.#began + this.#settledMs; + + this.#session.clock.turn(false); this.#run.emit( siteDone( this.#site, - // Wall clock, which is the one a nightly run is long by — and - // nothing like its `requestMs` once a site runs several requests at - // once. - Date.now() - this.#began + this.#settledMs, + ms, this.#session.requestsBy, this.#counts, + this.#session.clock.spent(), ), ); } } + /** Where the site's time goes — see `site:done`'s `msBy`. */ + get clock(): SiteClock { + return this.#session.clock; + } + /** The work itself, so {@link run} is only about letting go of things. */ async #work(): Promise { // Both groups at once, and before anything asks the source: a site's own code @@ -466,12 +495,12 @@ export class SiteRun { * against thousands of round trips saved for a store that has to make them. */ async #collectStale(): Promise { - const { cache, emit, enqueue, localWork, now } = this.#run; + const { cache, emit, localWork, now } = this.#run; const site = this.#site; const checked = await Promise.all( this.#channels.map((channel) => - enqueue(localWork, async (): Promise => { + this.#local(localWork, async (): Promise => { const keys = this.#resolved.window.map((day) => ({ site, channelId: channel.xmltvId, @@ -498,7 +527,10 @@ export class SiteRun { } this.#count('fromCache'); - emit({ type: 'entry:cached', site, channelId: channel.xmltvId, day }); + + if (emit.hears.has('entry:cached')) { + emit({ type: 'entry:cached', site, channelId: channel.xmltvId, day }); + } return []; }); @@ -645,13 +677,15 @@ export class SiteRun { this.#count('empty'); } - emit({ - type: 'entry:fetched', - site, - channelId: channel.xmltvId, - day, - programmes: mine.length, - }); + if (emit.hears.has('entry:fetched')) { + emit({ + type: 'entry:fetched', + site, + channelId: channel.xmltvId, + day, + programmes: mine.length, + }); + } return; } @@ -670,14 +704,16 @@ export class SiteRun { this.#count('empty', -1); } - emit({ - type: 'entry:appended', - site, - channelId: channel.xmltvId, - day, - added: mine.length, - total: programmes.length, - }); + if (emit.hears.has('entry:appended')) { + emit({ + type: 'entry:appended', + site, + channelId: channel.xmltvId, + day, + added: mine.length, + total: programmes.length, + }); + } } /** @@ -698,8 +734,12 @@ export class SiteRun { day: string, payload: unknown, ): Promise { - const { enqueue, localWork, signal } = this.#run; - const slot = new LocalSlot(localWork, enqueue, 1); + const { localWork, signal } = this.#run; + const slot = new LocalSlot( + localWork, + (queue, task, options) => this.#local(queue, task, options), + 1, + ); return slot.hold(async () => { const parsed = await config.parseDay( @@ -728,15 +768,44 @@ export class SiteRun { day: string, programmes: ParsedProgramme[], ): Promise { - const { enqueue, localWork } = this.#run; - - return enqueue( - localWork, + return this.#local( + this.#run.localWork, ({ signal: taskSignal }) => this.#persist(channel, day, programmes, taskSignal), { priority: 1 }, ); } + /** + * The run's `enqueue`, for this site's local work — each of its checks of + * what is cached, its parses and their writes, and what a stream hands over + * — timed as it waits for a slot and as it is under way: see `msBy`. + * + * One a cancel drops before its turn never says it stopped waiting, which + * would cost every task a promise: the cancel tells the clock instead — see + * `SiteClock.cancelled`. + */ + #local( + queue: PQueue, + task: (options: { signal?: AbortSignal | undefined }) => Promise, + options?: { priority?: number }, + ): Promise { + this.clock.queued(); + + return this.#run.enqueue( + queue, + async (slot) => { + this.clock.started(); + + try { + return await task(slot); + } finally { + this.clock.finished(); + } + }, + options, + ); + } + /** * What a request covers, as the three events about it name it. * @@ -793,7 +862,9 @@ export class SiteRun { // One each rather than a count for the request: what a reader wants of a // kept channel-day is which one it was, and `site:done` already carries // how many there were. - emit({ type: 'entry:unchanged', site: this.#site, channelId: channel.xmltvId, day }); + if (emit.hears.has('entry:unchanged')) { + emit({ type: 'entry:unchanged', site: this.#site, channelId: channel.xmltvId, day }); + } } } @@ -844,8 +915,9 @@ export class SiteRun { } const asking = this.#revalidationFor(request); - const span = SiteRun.#span(request); let payload: unknown; + /** Its channels and days, as its events say them: made once, where one is wanted. */ + let span: { channels: string[]; days: string[] } | undefined; try { // The request queue's slot covers this and nothing else, so a parse below is @@ -863,7 +935,13 @@ export class SiteRun { // Timed from inside the task, so what is reported is the request rather // than the wait for a slot — a site paced to one request a second would // otherwise report every request as having taken a second. - this.#run.emit({ type: 'request:started', site: this.#site, ...span }); + if (this.#run.emit.hears.has('request:started')) { + this.#run.emit({ + type: 'request:started', + site: this.#site, + ...(span ??= SiteRun.#span(request)), + }); + } const began = Date.now(); @@ -876,12 +954,14 @@ export class SiteRun { ); return (this.#revalidates ? revalidation.run(asking, fetch) : fetch()).then((raw) => { - this.#run.emit({ - type: 'request:done', - site: this.#site, - ...span, - ms: Date.now() - began, - }); + if (this.#run.emit.hears.has('request:done')) { + this.#run.emit({ + type: 'request:done', + site: this.#site, + ...(span ??= SiteRun.#span(request)), + ms: Date.now() - began, + }); + } return raw; }); @@ -953,6 +1033,8 @@ export class SiteRun { const asking = this.#revalidationFor(request); let ignored = 0; let failure: unknown; + /** Its channels and days, as its events say them: made once, where one is wanted. */ + let span: { channels: string[]; days: string[] } | undefined; const write = (pair: Pair, programmes: ParsedProgramme[]): void => { writes.push( @@ -970,69 +1052,82 @@ export class SiteRun { // So a pass is paced the way a `parseDay` is, by sending its own fetches // through the queue one at a time, and a slot keeps meaning what it always // meant: one request to the source. - emit({ type: 'request:started', site: this.#site, ...SiteRun.#span(request) }); + if (emit.hears.has('request:started')) { + emit({ type: 'request:started', site: this.#site, ...(span ??= SiteRun.#span(request)) }); + } const began = Date.now(); + // Its document read, and split as it arrives: the source's, where its + // writes, or a fetch of its own, are not what the site is doing. + this.clock.reading(true); - await revalidation.run(asking, async () => { - for await (const emission of config.stream( - streamContext(request, { - ...this.#deps(signal), - // As a parse's is, and ahead of the planned ones for the same - // reason: a pass already under way is finished rather than joined. - paced: (task, options) => - this.#sending(task, { - priority: 1, - label: this.#pacedLabel(options, this.#resolved.requestLabel), - }), - // A pass holds no slot while it runs, so it waits as it is. - away: (wait) => wait(), - }), - )) { - // Between emissions, which is as often as this has anything to say: a - // cancelled run stops here rather than writing the rest of a document - // nobody is waiting for. - signal?.throwIfAborted(); - - const { channel, day } = emission; - const id = `${channel?.xmltvId}|${day}`; - const pair = owed.get(id); - - if (pair === undefined) { - if (this.#written.has(id) && emission.unchanged !== true) { - // Said again: added to what the earlier emission wrote, rather - // than put in its place. A document not grouped by channel. - write({ channel, day }, emission.programmes); + try { + await revalidation.run(asking, async () => { + for await (const emission of config.stream( + streamContext(request, { + ...this.#deps(signal), + // As a parse's is, and ahead of the planned ones for the same + // reason: a pass under way is finished rather than joined. + paced: (task, options) => + this.#sending(task, { + priority: 1, + label: this.#pacedLabel(options, this.#resolved.requestLabel), + }), + // A pass holds no slot while it runs, so it waits as it is. + away: (wait) => wait(), + }), + )) { + // Between emissions, which is as often as this has anything to + // say: a cancelled run stops here rather than writing the rest of + // a document nobody is waiting for. + signal?.throwIfAborted(); + + const { channel, day } = emission; + const id = `${channel?.xmltvId}|${day}`; + const pair = owed.get(id); + + if (pair === undefined) { + if (this.#written.has(id) && emission.unchanged !== true) { + // Said again: added to what the earlier emission wrote, + // rather than put in its place. A document not grouped by + // channel. + write({ channel, day }, emission.programmes); + } else { + // A channel-day nobody asked about — one already fresh in + // the cache, a channel outside the list, or an emission that + // makes no sense. Counted, and reported once at the end. + ignored++; + } + } else if (emission.unchanged === true) { + // Nothing to write: the pass says what is cached still stands. + // Held until the stream ends, since a pass that then fails has + // not vouched for anything. + owed.delete(id); + keeping.push(pair); } else { - // A channel-day nobody asked about — one already fresh in the - // cache, a channel outside the list, or an emission that makes no - // sense. Counted, and reported once at the end. - ignored++; + owed.delete(id); + write(pair, emission.programmes); } - } else if (emission.unchanged === true) { - // Nothing to write: the pass says what is cached still stands. Held - // until the stream ends, since a pass that then fails has not - // vouched for anything. - owed.delete(id); - keeping.push(pair); - } else { - owed.delete(id); - write(pair, emission.programmes); - } - // Backpressure, and the only thing holding the parser back: writing is - // queued rather than awaited, so the split runs on while entries land, - // but no further ahead than `localConcurrency` of them. - await localWork.onSizeLessThan(localWork.concurrency); - } - }); + // Backpressure, and the only thing holding the parser back: + // writing is queued rather than awaited, so the split runs on while + // entries land, but no further ahead than `localConcurrency` of + // them. + await localWork.onSizeLessThan(localWork.concurrency); + } + }); + } finally { + this.clock.reading(false); + } - emit({ - type: 'request:done', - site: this.#site, - ...SiteRun.#span(request), - ms: Date.now() - began, - }); + if (emit.hears.has('request:done')) { + emit({ + type: 'request:done', + site: this.#site, + ...(span ??= SiteRun.#span(request)), + ms: Date.now() - began, + }); + } } catch (error) { failure = error; } diff --git a/src/grabber/thread.ts b/src/grabber/thread.ts index d7c8416..d5dfd22 100644 --- a/src/grabber/thread.ts +++ b/src/grabber/thread.ts @@ -41,6 +41,7 @@ */ import { availableParallelism } from 'node:os'; +import { performance } from 'node:perf_hooks'; import { pathToFileURL } from 'node:url'; import { inspect } from 'node:util'; import { MessageChannel, Worker } from 'node:worker_threads'; @@ -51,14 +52,17 @@ import type { CacheStore, ChannelDayKey } from '../cache/types.js'; import { resolveConfigSource, type ConfigSource, type EpgConfig } from '../config.js'; import { errorMessage, MAX_CAUSE_DEPTH } from '../core/error.js'; import { + emitOf, noCounts, noRequests, type EpgEventInput, + type EpgEventType, type GrabCounts, type RequestCounts, } from '../core/events.js'; import type { XmltvProgramme } from '../xmltv/types.js'; import { assertPlainData } from './channels.js'; +import { addedTime, measuredOf, type SiteClock } from './clock.js'; import { addRequests, mergedRequests, @@ -93,6 +97,12 @@ export interface SiteThreadData { codec?: CodecName; /** The memory its request counts go in, which the run's thread reads — see `shared-counts.ts`. */ counts: SharedArrayBuffer; + /** + * The kinds of event the run's reporter wants, as the run took them when it + * began — its own `Emit.hears` — so that what nobody reads is neither made + * nor sent. + */ + hears: EpgEventType[]; } /** Either end of the conversation: a worker, its `parentPort`, or a channel's port. */ @@ -128,6 +138,16 @@ export class SiteNotRunError extends Error { /** The cache's methods, which is all the worker's store asks the main thread. */ type CacheOp = Exclude | 'writeEncoded'; +/** + * Whether the run's thread answers `op` through a slot of its `localWork` — + * every call but what a site remembers, which is small and read once a turn. + * The one rule both sides keep to: the run's thread, to say how long a call + * waited for its slot, and a site's, to tell its clock it is waiting there. + */ +function throughLocalWork(op: CacheOp): boolean { + return op !== 'getState' && op !== 'setState'; +} + type ToHost = | { kind: 'event'; event: EpgEventInput } | { kind: 'cache'; id: number; op: CacheOp; args: unknown[] } @@ -140,7 +160,12 @@ type ToHost = | { kind: 'failed'; error: unknown }; type ToWorker = - | { kind: 'answer'; id: number; value?: unknown; error?: unknown } + /** + * A cache call answered — with how long it waited for a slot of the run's + * `localWork` first, where it did, which is time the site's clock otherwise + * gives to its work: see `SiteClock.back`. + */ + | { kind: 'answer'; id: number; value?: unknown; error?: unknown; queuedMs?: number } | { kind: 'abort'; reason: unknown }; /** An error as it crosses: what the reporters read, and nothing a clone would lose. */ @@ -274,6 +299,7 @@ function handableOf(message: ToHost | ToWorker): ToHost | ToWorker | undefined { ? { kind: 'answer', id: message.id, + ...(message.queuedMs === undefined ? {} : { queuedMs: message.queuedMs }), error: plain( new TypeError("The run's cache answered with what no thread can be handed"), ), @@ -458,6 +484,7 @@ export class SiteThreads { ? { codec: run.cache.codecName } : {}), counts: reader.buffer, + hears: [...run.emit.hears], }; const { port1, port2 } = new MessageChannel(); let pooled: PooledWorker; @@ -757,10 +784,14 @@ export function hostSite(port: SitePort, run: Run, hosting: Hosting): Promise<() addRequests(by, label, done); } - const done = siteDone(site, Date.now() - began!, requestsObject(by), { - ...counted, - sitesFailed: 1, - }); + const done = siteDone( + site, + Date.now() - began!, + requestsObject(by), + { ...counted, sitesFailed: 1 }, + // Where its time went, as far as its thread said before it went. + reader.spent(), + ); // Once: a site that failed has counted it, and said why. if (counted.sitesFailed === 0) { @@ -845,21 +876,31 @@ export function hostSite(port: SitePort, run: Run, hosting: Hosting): Promise<() break; case 'cache': { const { id, op, args } = message; - const answering = - op === 'getState' || op === 'setState' - ? answer(cache, op, args) - : localWork.add(() => answer(cache, op, args), { - // A channel-day written — encoded already, for a store with - // a codec — or read to be added to, ahead of a sweep. + /** How long it waited for a slot of `localWork` — none, for what never asks for one. */ + let queuedMs: number | undefined; + const asked = performance.now(); + const answering = !throughLocalWork(op) + ? answer(cache, op, args) + : localWork.add( + () => { + queuedMs = performance.now() - asked; + + return answer(cache, op, args); + }, + { + // A channel-day written — encoded already, for a store + // with a codec — or read to be added to, ahead of a sweep. priority: op === 'write' || op === 'writeEncoded' || op === 'read' ? 1 : 0, - }); + }, + ); + const waited = (): { queuedMs?: number } => (queuedMs === undefined ? {} : { queuedMs }); void answering.then( (value) => { - send(port, { kind: 'answer', id, value }); + send(port, { kind: 'answer', id, value, ...waited() }); }, (error: unknown) => { - send(port, { kind: 'answer', id, error: plain(error) }); + send(port, { kind: 'answer', id, error: plain(error), ...waited() }); }, ); break; @@ -938,10 +979,19 @@ export class RemoteCache implements CacheStore { readonly #port: SitePort; /** The codec the run's store keeps its entries with, run here when there is one. */ readonly #codec: CodecName | undefined; + /** Each call not yet answered — and the clock told it went to the run's thread, where one was. */ readonly #waiting = new Map< number, - { resolve(value: unknown): void; reject(error: unknown): void } + { + resolve(value: unknown): void; + reject(error: unknown): void; + /** The clock told it went to the run's thread, and when. */ + clock: SiteClock | undefined; + since: number; + } >(); + /** The clock of the site this asks for — see {@link timeWith}. */ + #clock: SiteClock | undefined; /** What {@link close} is waiting on, told once no call is left waiting — a turn later. */ #drained: (() => void) | undefined; /** Whether {@link close} is done, after which nothing is answered. */ @@ -964,7 +1014,8 @@ export class RemoteCache implements CacheStore { // Nothing asked from here is answered either: refused, rather than sent. this.#closed = true; - for (const { reject } of this.#waiting.values()) { + for (const { reject, clock, since } of this.#waiting.values()) { + clock?.back(since, 0); reject(new Error('The run stopped answering its cache before this call was answered')); } @@ -980,6 +1031,9 @@ export class RemoteCache implements CacheStore { const waiting = this.#waiting.get(message.id); + // Back, with what of it was a wait for a slot on the run's thread. + waiting?.clock?.back(waiting.since, message.queuedMs ?? 0); + // Its caller told first, so one asking again as it goes on is still waited // for by a close that is waiting. if (message.error === undefined) { @@ -1008,21 +1062,35 @@ export class RemoteCache implements CacheStore { } const id = this.#next++; + // What the run's thread answers through a slot of its local work: the + // site's work waits on it there, where the clock is told. + const clock = throughLocalWork(op) ? this.#clock : undefined; + const since = clock?.elsewhere() ?? 0; return new Promise((resolve, reject) => { - this.#waiting.set(id, { resolve, reject }); + this.#waiting.set(id, { resolve, reject, clock, since }); try { send(this.#port, { kind: 'cache', id, op, args }, transfer); } catch (error) { // Never sent — what no thread can be handed, as what a site keeps may // hold — so never to be answered. + clock?.back(since, 0); this.#forget(id); throw error; } }); } + /** + * Tell `clock` how long each call waits for a slot on the run's thread: time + * the site's own work is said to have taken otherwise, where it is the run's + * `localConcurrency` that held it up. + */ + timeWith(clock: SiteClock): void { + this.#clock = clock; + } + getMeta(key: ChannelDayKey): ReturnType { return this.#ask('getMeta', [key]); } @@ -1190,7 +1258,6 @@ export async function workSite( // As the run's own queues are: the run's signal handed to every task — see `grab`. const enqueue: Enqueue = (queue, task, options = {}) => queue.add(task, { ...options, signal: stop.signal }); - const run: Run = { defaults: data.defaults, cache, @@ -1198,9 +1265,10 @@ export async function workSite( grabbedAt: data.grabbedAt, startDay: data.startDay, timeZone: data.timeZone, - emit: (event) => { + // The run's own set, so a site asks here what it would ask there. + emit: emitOf((event) => { send(port, { kind: 'event', event: plainEvent(event) }); - }, + }, new Set(data.hears)), signal: stop.signal, localWork, enqueue, @@ -1225,6 +1293,10 @@ export async function workSite( send(port, { kind: 'requested', label, done: left }); } }, + // Into the memory both threads see, as each stretch is spent — see `writer`. + spent: (part, ms) => { + writer.spend(part, ms); + }, }; const site = new SiteRun( @@ -1232,6 +1304,8 @@ export async function workSite( run, ); + cache.timeWith(site.clock); + // Into memory rather than said, so the run's thread knows it the moment it // is so: a thread that dies a moment later may have run some of the site, // and a message saying so may not arrive before word that it died. @@ -1266,6 +1340,7 @@ function withSettled( event.ms + settled.spanMs, mergedRequests(settled.requestsBy, event.requestsBy), event, + addedTime(measuredOf(event.msBy), settled.clock.spent()), ), }; } diff --git a/src/main.ts b/src/main.ts index 2662a8a..3f81791 100644 --- a/src/main.ts +++ b/src/main.ts @@ -36,6 +36,8 @@ export type { RunPhase, SaidFields, Says, + TimeSpent, + UnmatchedProgramme, } from './core/events.js'; export { DEFAULT_FAILURE_CAP, @@ -53,6 +55,7 @@ export { textReporter, } from './core/reporters.js'; export type { + EmitterReporterOptions, FailureEvent, EpgEventSink, FailureMode, @@ -335,5 +338,11 @@ export { writeGuide, defaultChannelInfo, } from './merge/main.js'; -export type { BackfillOptions, ChannelSelection, ResolvedMatch, Selectable } from './merge/main.js'; +export type { + BackfillOptions, + ChannelSelection, + NearMissWhy, + ResolvedMatch, + Selectable, +} from './merge/main.js'; export type * from './merge/types.js'; diff --git a/src/merge/guide.ts b/src/merge/guide.ts index 2accdcd..de491c2 100644 --- a/src/merge/guide.ts +++ b/src/merge/guide.ts @@ -11,11 +11,11 @@ import { import { errorMessage, GrabberError } from '../core/error.js'; import { writeOutput, type OutputOptions, type OutputTarget } from '../core/output.js'; import { emitter } from '../core/events.js'; -import type { Says } from '../core/events.js'; +import type { Says, UnmatchedProgramme } from '../core/events.js'; import { channelElement, covered, defaultChannelInfo, resolveSites } from '../grabber/channels.js'; import { siteDayZone } from '../grabber/site.js'; import type { GrabberChannel } from '../grabber/types.js'; -import { getXmltvOffset, writeXmltvStream, xmltvDate } from '../xmltv/main.js'; +import { formatXmltvDate, getXmltvOffset, writeXmltvStream, xmltvDate } from '../xmltv/main.js'; import { outputOptions } from '../xmltv/serialize.js'; import type { XmltvChannel, XmltvProgramme } from '../xmltv/types.js'; import { mergeChannels } from './channel.js'; @@ -23,14 +23,25 @@ import { derivedChannelElement, resolveDerived, shiftProgrammes } from './derive import { backfillInto, DEFAULT_FILL_STOP_MS, + firstSite, mergeInto, resolveMatch, + type MergeOrigin, type MergeWatch, + type Sites, } from './programme.js'; import { resolveUnion } from './union.js'; import type { ChannelSource, RegistryEntry } from './registry.js'; import { channelSelection, configured, unmatched, unmatchedMessage } from './select.js'; -import type { BuildGuideOptions, FillGapsContext, FillGapsOptions, GuideContext } from './types.js'; +import type { + BuildGuideOptions, + ChannelStrategy, + FillGapsContext, + FillGapsOptions, + GuideContext, + MergeOptions, + ProgrammeStrategy, +} from './types.js'; /** * How many channel-days are read from the cache ahead of the writer when @@ -285,12 +296,30 @@ function titleOf(programme: XmltvProgramme): string { return programme.title[0]?.value ?? ''; } +/** One side of a near miss, as `merge:unmatched` says it. */ +function nearMissSide(programme: XmltvProgramme, site: string): UnmatchedProgramme { + return { site, title: titleOf(programme), start: formatXmltvDate(programme.start) }; +} + +/** + * The strategies a merge goes by: the config's, and where it says none, the + * defaults — what a guide is built by, and what `--explain` says it was. + */ +export function mergeStrategies(merge: MergeOptions | undefined): { + channelStrategy: ChannelStrategy; + programmeStrategy: ProgrammeStrategy; +} { + return { + channelStrategy: merge?.channelStrategy ?? 'merge-programmes', + programmeStrategy: merge?.programmeStrategy ?? 'merge', + }; +} + export async function* generateGuide(options: BuildGuideOptions): AsyncGenerator { const now = options.now ?? new Date(); const timeZone = resolveTimeZone(options.timeZone); const days = [...dayRange(options.startDay ?? dayIn(now, timeZone), options.days ?? 7)]; - const channelStrategy = options.merge?.channelStrategy ?? 'merge-programmes'; - const programmeStrategy = options.merge?.programmeStrategy ?? 'merge'; + const { channelStrategy, programmeStrategy } = mergeStrategies(options.merge); const { cache } = options; const emit = emitter(options); @@ -855,7 +884,12 @@ export async function* generateGuide(options: BuildGuideOptions): AsyncGenerator }: { entry: RegistryEntry; day: string; - }): Promise<{ entry: RegistryEntry; day: string; lists: XmltvProgramme[][] }> => { + }): Promise<{ + entry: RegistryEntry; + day: string; + lists: XmltvProgramme[][]; + sites: string[]; + }> => { // A derived channel reads what it shifts. Its source's rows rather than its // source's *output*, so the lists stay separate and the merging below does // for the derived channel exactly what it does for the channel it shifts. @@ -871,14 +905,17 @@ export async function* generateGuide(options: BuildGuideOptions): AsyncGenerator ); const lists: XmltvProgramme[][] = []; + /** Whose each list is, in step with them. */ + const sites: string[] = []; - for (const mine of cached) { + for (const [index, mine] of cached.entries()) { if (mine !== undefined) { lists.push(clamped(offsetMs === 0 ? mine : shiftProgrammes(mine, offsetMs, entry.xmltvId))); + sites.push(read.sources[index]!.config.site); } } - return { entry, day, lists }; + return { entry, day, lists, sites }; }; async function* programmes(): AsyncGenerator { @@ -916,18 +953,23 @@ export async function* generateGuide(options: BuildGuideOptions): AsyncGenerator /** The offset to write a synthesized time in — the last real one seen. */ let offset = 0; /** - * What one channel's sources made of each other — see `merge:summary`. - * - * Built only for the lists after the first, which is where the question - * means anything: the first list of a day folds into yesterday's tail, and - * a programme joining *that* is one source's own broadcast continuing - * rather than two sources agreeing. + * Which site each programme of a channel came from, for a channel more + * than one site covers — the only kind with a near miss to say. Kept by + * the merge as it makes new programmes out of old ones. + */ + const origins = new WeakMap(); + /** + * What one channel's sources made of each other — see `merge:summary` — + * as `site`'s list folds in: the first of a day's too, which folds into + * yesterday's tail, where another source's programme may be held over. Its + * own, continuing, is no other source's: `origin` says whose each is. */ - const watching = (channelId: string): MergeWatch => ({ + const watching = (channelId: string, origin: MergeOrigin): MergeWatch => ({ + origin, combined: () => { combined += 1; }, - nearMiss: (gapMs, programme, candidate) => { + nearMiss: (gapMs, programme, candidate, why) => { nearMiss += 1; if (examples.length < MAX_NEAR_MISSES) { @@ -938,10 +980,25 @@ export async function* generateGuide(options: BuildGuideOptions): AsyncGenerator nearby: titleOf(candidate), }); } + + // Made only where a reporter wants it — one at `debug`, `--explain + // unmatched`, a sink that did not say — since each side's start is + // formatted as the guide writes it, for thousands of these where two + // sources run on two grids, and a run at `info` reads none of them. + if (emit.hears.has('merge:unmatched')) { + emit({ + type: 'merge:unmatched', + channelId, + shiftMs: programme.start.getTime() - candidate.start.getTime(), + why, + programme: nearMissSide(programme, origin.site), + nearby: nearMissSide(candidate, firstSite(origins.get(candidate)) ?? ''), + }); + } }, }); - for await (const { entry, day, lists } of readAhead( + for await (const { entry, day, lists, sites } of readAhead( plan, Math.max(1, options.readAhead ?? DEFAULT_READ_AHEAD), readChannelDay, @@ -953,8 +1010,21 @@ export async function* generateGuide(options: BuildGuideOptions): AsyncGenerator options.signal?.throwIfAborted(); if (listStrategy === 'merge') { + // Whose each programme is, for a channel more than one site covers — + // from the first list on, and on a day only one of them answered for, + // since tomorrow's lists meet what today's leave. Not for a channel + // derived from one: the same lists again, shifted, whose sources + // agreeing or nearly agreeing is said once, of the channel itself. + const several = entry.derivedFrom === undefined && entry.sources.length > 1; + for (const [index, list] of lists.entries()) { - mergeInto(pending, list, match, index > 0 ? watching(entry.xmltvId) : undefined); + mergeInto( + pending, + list, + match, + // Made only where it is kept: one source's channel pays nothing. + several ? watching(entry.xmltvId, { site: sites[index]!, sites: origins }) : undefined, + ); } } else if (listStrategy === 'backfill') { // Across the sites of *this day* first, into an array of its own, and diff --git a/src/merge/main.ts b/src/merge/main.ts index 339d877..5f90b4f 100644 --- a/src/merge/main.ts +++ b/src/merge/main.ts @@ -10,7 +10,7 @@ export { } from './programme.js'; export { normalizeTitle } from './text.js'; export { DEFAULT_UNION, ResolvedUnion, resolveUnion } from './union.js'; -export type { BackfillOptions, ResolvedMatch } from './programme.js'; +export type { BackfillOptions, NearMissWhy, ResolvedMatch } from './programme.js'; export { mergeChannels } from './channel.js'; export { generateGuide, writeGuide, defaultChannelInfo } from './guide.js'; export { channelSelection, unmatched } from './select.js'; diff --git a/src/merge/programme.ts b/src/merge/programme.ts index 8f9baf5..750f034 100644 --- a/src/merge/programme.ts +++ b/src/merge/programme.ts @@ -1,3 +1,4 @@ +import type { NearMissWhy } from '../core/events.js'; import type { XmltvCredits, XmltvPersonValue, XmltvProgramme } from '../xmltv/types.js'; import type { MergeUnion, @@ -230,7 +231,12 @@ function duration(programme: XmltvProgramme): number | undefined { */ export interface ResolvedMatch { windowMs: number; - matches: ProgrammeMatcher; + /** + * What keeps two programmes from being one broadcast — nothing, where they + * are. One answer to both questions a merge asks of a pair, whether they are + * one and why not, so the two cannot disagree: see {@link NearMissWhy}. + */ + apart(a: XmltvProgramme, b: XmltvProgramme): NearMissWhy | undefined; /** * How it compares titles — a match's own `normalizeTitle`, or the default * for a matcher of one's own, which says nothing about titles. What a @@ -239,6 +245,19 @@ export interface ResolvedMatch { normalizeTitle: (title: string) => string; } +/** What kept two programmes from being one — declared with the event that says it. */ +export type { NearMissWhy }; + +/** + * A matcher of one's own as a {@link ResolvedMatch.apart}: beyond its window, + * which it is never asked about, `start`; within it, what it says — and where + * it says no, `matcher`. Nothing of the shipped rules is asked on its behalf: + * what titles it compares, and how, is its own business. + */ +function ownMatcher(matches: ProgrammeMatcher, windowMs: number): ResolvedMatch['apart'] { + return (a, b) => (shiftOf(a, b) > windowMs ? 'start' : matches(a, b) ? undefined : 'matcher'); +} + /** * Settle {@link ProgrammeMatch} into something to match with. A matcher of * one's own passes through with no reach to speak of — only it knows what it @@ -251,11 +270,19 @@ export function resolveMatch( // Everything, because a function that never said how far it looks might // look anywhere — and the cost of that is in `WindowedMatcher`, which is // how a matcher says. - return { windowMs: Number.POSITIVE_INFINITY, matches: match, normalizeTitle }; + return { + windowMs: Number.POSITIVE_INFINITY, + apart: ownMatcher(match, Number.POSITIVE_INFINITY), + normalizeTitle, + }; } if (match !== undefined && match !== null && 'matches' in match) { - return { windowMs: match.windowMs, matches: match.matches, normalizeTitle }; + return { + windowMs: match.windowMs, + apart: ownMatcher(match.matches, match.windowMs), + normalizeTitle, + }; } const { @@ -270,34 +297,44 @@ export function resolveMatch( return { windowMs: startToleranceMs, normalizeTitle: normalize, - matches: (a, b) => { - const shift = Math.abs(a.start.getTime() - b.start.getTime()); - - if (shift > startToleranceMs) { - return false; - } - - // The same instant: two sources describing one broadcast, unless a - // config says even that has to be corroborated by the title. - if (shift === 0) { - return titles === 'always' ? titlesMatch(a, b, normalize) : true; - } - - if (titles !== 'never' && !titlesMatch(a, b, normalize)) { - return false; + apart: (a, b) => { + const shift = shiftOf(a, b); + // The titles, where they are compared: on an identical start only when a + // config says even that has to be corroborated, and never under `never`. + const compared = titles === 'always' || (titles !== 'never' && shift !== 0); + const titlesDiffer = compared && !titlesMatch(a, b, normalize); + const beyond = shift > startToleranceMs; + const why: NearMissWhy = titlesDiffer + ? beyond + ? 'both' + : 'title' + : beyond + ? 'start' + : 'runs'; + + // The titles agreeing and the starts near enough, one broadcast — save + // where they start further apart than the shorter lasts. Consecutive + // programmes sit exactly one duration apart, and a segment lies within + // the show it is part of, so anything that far out is the neighbour or + // the part rather than the same broadcast told twice: decidable only + // when both say where they end, and never of the same instant. + if ( + why === 'runs' && + (shift === 0 || shift < Math.min(duration(a) ?? Infinity, duration(b) ?? Infinity)) + ) { + return undefined; } - // What keeps a wide tolerance from swallowing the programme next door: - // consecutive programmes sit exactly one duration apart, so anything - // that far out is the neighbour rather than the same broadcast told - // twice. Only decidable when both sides say where they end. - const shorter = Math.min(duration(a) ?? Infinity, duration(b) ?? Infinity); - - return shift < shorter; + return why; }, }; } +/** How far apart two programmes start. */ +function shiftOf(a: XmltvProgramme, b: XmltvProgramme): number { + return Math.abs(a.start.getTime() - b.start.getTime()); +} + /** First index whose start is at or after `time`, in a list sorted by start. */ function lowerBound(programmes: XmltvProgramme[], time: number): number { let low = 0; @@ -317,42 +354,162 @@ function lowerBound(programmes: XmltvProgramme[], time: number): number { } /** - * Where in `target` `programme` belongs, and what it belongs with: the index of - * the programme it should be merged into, or `-1` to insert it. + * The nearest programme of another source's a programme did not join — kept + * by {@link matchOf} as it scans the window, and by {@link nearMissOf} beyond + * it. One for a whole list, set afresh for each programme: asking costs a + * merge nothing per programme. + */ +interface Refused { + index: number; + gapMs: number; + /** What kept them apart — asked of one beyond the window only once it is the nearest. */ + why: NearMissWhy | undefined; +} + +/** A near miss, as {@link MergeWatch.nearMiss} is told it. */ +interface NearMiss { + index: number; + gapMs: number; + candidate: XmltvProgramme; + why: NearMissWhy; +} + +/** + * Whether `candidate` is another source's than the list folding in — see + * {@link MergeOrigin}: the one rule for which of `target` the list may have + * nearly joined, and which it combined with. + */ +function others(origin: MergeOrigin | undefined, candidate: XmltvProgramme): boolean { + // This source's own, held from the day before, is the programme before this + // one, not one it nearly was, nor another source agreeing with it. + return origin === undefined || !ownedBy(origin.sites.get(candidate), origin.site); +} + +/** + * The index of the programme of `target` that `programme` should be merged + * into, or `-1` to insert it — and, where given `refused`, the nearest within + * the window that another source's said no to, and why. Each pair judged once. * * The nearest acceptable candidate wins. A source shifted by a minute against * two of another's programmes must join the one it is a minute from, not * whichever the scan reached first. */ -function findMatch( +function matchOf( target: XmltvProgramme[], programme: XmltvProgramme, match: ResolvedMatch, taken: ReadonlySet, + origin: MergeOrigin | undefined, + refused: Refused | undefined, ): number { const start = programme.start.getTime(); + const { windowMs } = match; let best = -1; let bestShift = Number.POSITIVE_INFINITY; - for (let index = lowerBound(target, start - match.windowMs); index < target.length; index++) { + for (let index = lowerBound(target, start - windowMs); index < target.length; index++) { const candidate = target[index]!; const shift = candidate.start.getTime() - start; - if (shift > match.windowMs) { + // Past the window — or, with a match in hand, past where anything could be + // nearer than it: the list is sorted, so all that follows is further. + if (shift > windowMs || (best !== -1 && shift >= bestShift)) { break; } const distance = Math.abs(shift); - if (distance < bestShift && !taken.has(index) && match.matches(candidate, programme)) { + // Taken already, or no nearer than a match in hand: nothing to ask. + if (taken.has(index) || (best !== -1 && distance >= bestShift)) { + continue; + } + + const why = match.apart(candidate, programme); + + if (why === undefined) { best = index; bestShift = distance; + } else if ( + refused !== undefined && + distance <= NEAR_MISS_MS && + distance < refused.gapMs && + others(origin, candidate) + ) { + refused.index = index; + refused.gapMs = distance; + refused.why = why; } } return best; } +/** + * The near miss of a programme that matched nothing: what {@link matchOf} kept + * of the window, or — where the window had nothing even nearly it — the + * nearest other source's programme within {@link NEAR_MISS_MS}, beyond it. + * Anything beyond the window is further than what is in it, so that wider + * scan is only for where the window had none, and why is asked of the + * nearest, once. + */ +function nearMissOf( + target: XmltvProgramme[], + programme: XmltvProgramme, + match: ResolvedMatch, + taken: ReadonlySet, + origin: MergeOrigin | undefined, + refused: Refused, +): NearMiss | undefined { + const start = programme.start.getTime(); + const { windowMs } = match; + + if (refused.index === -1 && NEAR_MISS_MS > windowMs) { + for (let index = lowerBound(target, start - NEAR_MISS_MS); index < target.length; index++) { + const candidate = target[index]!; + const shift = candidate.start.getTime() - start; + + if (shift > NEAR_MISS_MS) { + break; + } + + const distance = Math.abs(shift); + + if ( + distance > windowMs && + distance < refused.gapMs && + !taken.has(index) && + others(origin, candidate) + ) { + refused.index = index; + refused.gapMs = distance; + refused.why = undefined; + } + } + } + + if (refused.index === -1) { + return undefined; + } + + const candidate = target[refused.index]!; + + return { + index: refused.index, + gapMs: refused.gapMs, + candidate, + why: refused.why ?? match.apart(candidate, programme) ?? 'matcher', + }; +} + +/** {@link Refused} with nothing in it yet. */ +function refusedAfresh(refused: Refused): Refused { + refused.index = -1; + refused.gapMs = Number.POSITIVE_INFINITY; + refused.why = undefined; + + return refused; +} + /** * One source's list with its own duplicate rows pooled — programmes naming the * *identical* instant, and only those. @@ -396,6 +553,8 @@ export function mergeInto( match: ResolvedMatch, watch?: MergeWatch, ): XmltvProgramme[] { + const origin = watch?.origin; + // Every candidate is decided against `target` as it stands, and only then is // anything added to it. Otherwise a list's second programme could be paired // with its own first one the moment that landed in `target` — the tolerance @@ -404,35 +563,77 @@ export function mergeInto( const inserts: XmltvProgramme[] = []; const taken = new Set(); + const looking = watch?.nearMiss !== undefined; + const refused: Refused | undefined = looking + ? { index: -1, gapMs: Number.POSITIVE_INFINITY, why: undefined } + : undefined; + /** What matched nothing, with what it nearly did — said once the whole list is placed. */ + const missed: Array<[programme: XmltvProgramme, near: NearMiss]> = []; + for (const programme of pool(incoming)) { // Taken entries are passed over: one programme of `target` stands for one // broadcast, so it absorbs one programme of `incoming` and no more. - const index = findMatch(target, programme, match, taken); + const index = matchOf( + target, + programme, + match, + taken, + origin, + refused && refusedAfresh(refused), + ); if (index === -1) { inserts.push(programme); - if (watch !== undefined) { - const near = nearest(target, programme, taken); + const near = refused && nearMissOf(target, programme, match, taken, origin, refused); - if (near !== undefined) { - watch.nearMiss(near.gapMs, programme, near.candidate); - } + if (near !== undefined) { + missed.push([programme, near]); } } else { taken.add(index); merges.push([index, programme]); - watch?.combined(); + + // Two sources agreeing — not one source and its own programme, held + // over from the day before and said again today. + if (others(origin, target[index]!)) { + watch?.combined?.(); + } + } + } + + // Said once the whole list is placed: a programme a later one of it then + // took is no near miss, and the nearest still left in its place is. The + // left only ever narrow, so one still left was the nearest all along. + for (const [programme, near] of missed) { + let nearest: NearMiss | undefined = near; + + if (taken.has(near.index)) { + // Matches nothing still: the window is only narrower. + matchOf(target, programme, match, taken, origin, refusedAfresh(refused!)); + nearest = nearMissOf(target, programme, match, taken, origin, refused!); + } + + if (nearest !== undefined) { + watch!.nearMiss!(nearest.gapMs, programme, nearest.candidate, nearest.why); } } // Merging first, while the indices above still mean what they meant. A merge - // keeps the base's start, so it moves nothing. + // keeps the base's start, so it moves nothing — and its sites, with this one. for (const [index, programme] of merges) { - target[index] = mergeProgrammes(target[index]!, programme); + const base = target[index]!; + const merged = mergeProgrammes(base, programme); + + if (origin !== undefined) { + origin.sites.set(merged, withSite(origin.sites.get(base), origin.site)); + } + + target[index] = merged; } for (const programme of inserts) { + origin?.sites.set(programme, origin.site); // After anything starting at the same instant, so what could not be merged // still follows what it arrived behind. target.splice(lowerBound(target, programme.start.getTime() + 1), 0, programme); @@ -462,49 +663,72 @@ export const NEAR_MISS_MS = 30 * 60_000; * single-source channels pays nothing for it. */ export interface MergeWatch { - /** One of `incoming` joined a programme already in `target`. */ - combined(): void; /** - * One of `incoming` joined nothing, and something was close. + * Whose list is folded in, and whose each programme of `target` is — kept as + * the merge makes new programmes out of old ones, for saying which two + * nearly matched, and for not taking a source's own for another's. + */ + origin?: MergeOrigin; + /** + * One of `incoming` joined a programme already in `target` — another site's, + * where `origin` says whose: not its own, held over from the day before. + */ + combined?(): void; + /** + * One of `incoming` joined nothing, and `candidate` — the nearest of `target` + * within {@link NEAR_MISS_MS}, another site's where `origin` says whose — is + * left unjoined too, once the whole of `incoming` is placed. * - * `gapMs` is what the two starts are apart. Larger than the match tolerance - * means they failed on time; smaller means they failed on titles, which is - * the same news about a different cause. + * `gapMs` is what the two starts are apart, and `why` what kept them apart. */ - nearMiss(gapMs: number, programme: XmltvProgramme, candidate: XmltvProgramme): void; + nearMiss?( + gapMs: number, + programme: XmltvProgramme, + candidate: XmltvProgramme, + why: NearMissWhy, + ): void; } /** - * The unmatched programme nearest this one, where anything is near enough. - * - * Only asked after a match failed, and only when somebody is watching — a scan - * of its own, because {@link findMatch} deliberately never looks past the - * tolerance it was given. + * Which sites each programme of a merge came from, for saying which two nearly + * matched — kept by {@link mergeInto} for every programme it puts in `target`: + * one added, `site`'s; one merged, its base's and then each that joined it. */ -function nearest( - target: XmltvProgramme[], - programme: XmltvProgramme, - taken: ReadonlySet, -): { gapMs: number; candidate: XmltvProgramme } | undefined { - const start = programme.start.getTime(); - let found: { gapMs: number; candidate: XmltvProgramme } | undefined; +export interface MergeOrigin { + /** Whose list is folded in now. */ + site: string; + /** Every programme's sites, as far as it has been told — see {@link Sites}. */ + sites: WeakMap; +} - for (let index = lowerBound(target, start - NEAR_MISS_MS); index < target.length; index++) { - const candidate = target[index]!; - const shift = candidate.start.getTime() - start; +/** + * Whose a programme is: the one site it came from, or — once others joined it + * — each of them, its base's first. A string for the one, which is nearly + * every programme, rather than a list each. + */ +export type Sites = string | readonly string[]; - if (shift > NEAR_MISS_MS) { - break; - } +/** Whether `site` is one of a programme's `sites`. */ +export function ownedBy(sites: Sites | undefined, site: string): boolean { + return sites === site || (Array.isArray(sites) && sites.includes(site)); +} - const gapMs = Math.abs(shift); +/** The site a programme came from first — its base's, once others joined it. */ +export function firstSite(sites: Sites | undefined): string | undefined { + return typeof sites === 'string' ? sites : sites?.[0]; +} - if (!taken.has(index) && (found === undefined || gapMs < found.gapMs)) { - found = { gapMs, candidate }; - } +/** `sites`, with `site` among them. */ +function withSite(sites: Sites | undefined, site: string): Sites { + if (sites === undefined) { + return site; + } + + if (ownedBy(sites, site)) { + return sites; } - return found; + return typeof sites === 'string' ? [sites, site] : [...sites, site]; } /** @@ -587,7 +811,7 @@ export function backfillInto( // own first one the moment that lands, and a run of them filling one hole // would turn each other away. for (const [index, programme] of pooled.entries()) { - if (findMatch(target, programme, match, taken) !== -1) { + if (matchOf(target, programme, match, taken, undefined, undefined) !== -1) { continue; } diff --git a/test/build.test.ts b/test/build.test.ts index dcafd58..d953638 100644 --- a/test/build.test.ts +++ b/test/build.test.ts @@ -17,6 +17,7 @@ import { } from '../src/core/events.js'; import type { ChannelDayKey, StoredEntryMeta } from '../src/cache/main.js'; import { defineConfig, type EpgConfig } from '../src/config.js'; +import { emitterReporter } from '../src/core/reporters.js'; import { parseXmltvString } from '../src/xmltv/main.js'; import { defineSiteConfig, type SiteConfig } from '../src/grabber/types.js'; import type { XmltvProgramme } from '../src/xmltv/types.js'; @@ -380,7 +381,12 @@ describe('a site whose channel list fails', () => { (event) => event.type === 'site:done' && event.site === 'broken.example', ) as Extract; const counted = { - channels: { requests: 1, requestMs: expect.any(Number), failed: { 404: 1 } }, + channels: { + requests: 1, + requestMs: expect.any(Number), + waitedMs: expect.any(Number), + failed: { 404: 1 }, + }, }; expect(done.requests).toBe(1); @@ -1586,6 +1592,28 @@ describe('what a config asked to be told', () => { expect(heard).toEqual(['done 1']); }); + it('takes a reporter as the config’s events, for an emitter that hears more than its listeners', async () => { + const dir = await tempDir(); + const forwarded: string[] = []; + + // Every event on to somewhere else, with no listener for any of them. + class Bridge extends EventEmitter { + override emit(type: string | symbol, ...args: unknown[]): boolean { + forwarded.push(String(type)); + + return super.emit(type, ...args); + } + } + + await runGrab( + { ...config(dir), events: emitterReporter(new Bridge(), { wants: () => true }) }, + { now: NOW }, + ); + + expect(forwarded).toContain('entry:fetched'); + expect(forwarded).toContain('request:started'); + }); + it('reaches it once from a build, through both halves', async () => { const dir = await tempDir(); const heard = new Map(); diff --git a/test/cli.test.ts b/test/cli.test.ts index 580ac81..c9024c7 100644 --- a/test/cli.test.ts +++ b/test/cli.test.ts @@ -2213,6 +2213,229 @@ describe('flushing', () => { }); }); +describe('--explain', () => { + /** Two sites for one channel, whose one programme each starts seven minutes apart. */ + async function driftingConfig(dir: string): Promise { + const site = (name: string, start: string): string => `{ + site: '${name}', + channels: [{ xmltvId: 'one.example.tv', siteId: '1', name: 'One' }], + async request({ day }) { return { day }; }, + parseDay: ({ channel, day }) => [{ + channel: channel.xmltvId, + start: new Date(day + 'T${start}:00.000Z'), + title: [{ value: 'Show' }], + }], + }`; + + return configFile( + dir, + `export default { + sites: [${site('a.example.tv', '06:00')}, ${site('b.example.tv', '06:07')}], + days: 1, + output: ${JSON.stringify(join(dir, 'guide.xml'))}, + cache: { dir: ${JSON.stringify(join(dir, 'cache'))} }, + };`, + ); + } + + it('lists what nearly merged, on stderr once the guide is written', async () => { + const dir = await tempDir(); + const config = await driftingConfig(dir); + const { code, stdout, stderr } = await run([ + 'build', + '--config', + config, + '--explain', + 'unmatched', + ]); + + expect(code).toBe(0); + expect(stdout).toContain('Guide written to'); + expect(stderr).toContain( + '1 programme nearly merged, on 1 channel:\n' + + ' one.example.tv, b.example.tv against a.example.tv:\n' + + ' 1 × starts too far apart: b.example.tv starts 7 minutes later — ' + + 'a startToleranceMs of 420000 would bring it within reach (it is 300000)' + + ', and join it if the shorter of the two lasts longer than they are apart\n' + + '\n' + + 'Each of them:\n' + + ' one.example.tv: 420s apart, starts too far apart — ' + + '"Show" (b.example.tv, 06:07) / "Show" (a.example.tv, 06:00)\n', + ); + }); + + it('lists them for a merge as for a build', async () => { + const dir = await tempDir(); + const config = await driftingConfig(dir); + + await run(['grab', '--config', config]); + + const { code, stderr } = await run(['merge', '--config', config, '--explain', 'unmatched']); + + expect(code).toBe(0); + expect(stderr).toContain('1 programme nearly merged, on 1 channel:'); + expect(stderr).toContain('starts too far apart'); + }); + + it('says a merge that matches nothing across sources has none to look for', async () => { + const dir = await tempDir(); + const config = await driftingConfig(dir); + + await run(['grab', '--config', config]); + + // The same config, its programmes put side by side rather than matched. + const backfilling = config.replace('.mjs', '.backfill.mjs'); + + await writeFile( + backfilling, + `import base from ${JSON.stringify(config)};\n` + + `export default { ...base, merge: { programmeStrategy: 'backfill' } };\n`, + ); + + const { code, stderr } = await run([ + 'merge', + '--config', + backfilling, + '--explain', + 'unmatched', + ]); + + expect(code).toBe(0); + // Backfill matches across sources, to leave out what another already has — + // but keeps nothing of what it nearly matched. + expect(stderr).toContain( + "Near misses are looked for only under channelStrategy 'merge-programmes' and " + + "programmeStrategy 'merge', and this merge is under programmeStrategy: 'backfill'.", + ); + }); + + it('is refused as JSON, and points to the events that are', async () => { + const dir = await tempDir(); + const { code, stderr } = await run([ + 'build', + '--config', + await driftingConfig(dir), + '--explain', + 'unmatched', + '--format', + 'json', + ]); + + expect(code).toBe(2); + expect(stderr).toContain('--reporter json --log-level debug'); + }); + + it('says there were none, where none did', async () => { + const dir = await tempDir(); + const { code, stderr } = await run([ + 'build', + '--config', + await plainConfig(dir), + '--explain', + 'unmatched', + ]); + + expect(code).toBe(0); + expect(stderr).toContain('No programmes nearly merged.'); + }); + + it('is refused on a command whose run it is not about, before any of them returns', async () => { + const dir = await tempDir(); + const config = await plainConfig(dir); + const grab = await run(['grab', '--config', config, '--explain', 'unmatched']); + const merge = await run(['merge', '--config', config, '--explain', 'held']); + // Answered before a config is looked for, and refused before that. + const validate = await run(['validate', join(dir, 'guide.xml'), '--explain', 'unmatched']); + const dry = await run(['build', '--config', config, '--dry-run', '--explain', 'empty']); + + expect([grab.code, merge.code, validate.code, dry.code]).toEqual([2, 2, 2, 2]); + expect(grab.stderr).toContain('--explain unmatched is for build, merge, not grab'); + expect(merge.stderr).toContain('--explain held is for build, grab, not merge'); + expect(validate.stderr).toContain('--explain unmatched is for build, merge, not validate'); + expect(dry.stderr).toContain('--explain is for a run, not --dry-run'); + }); + + it('lists nothing for a run that was cancelled', async () => { + const dir = await tempDir(); + const controller = new AbortController(); + const config = await configFile( + dir, + `export default { + sites: [{ + site: 'example.tv', + channels: [{ xmltvId: 'one.example.tv', siteId: '1' }], + async request() { + globalThis.cancelTheRun(); + return {}; + }, + parseDay: () => [], + }], + days: 1, + output: ${JSON.stringify(join(dir, 'guide.xml'))}, + cache: { dir: ${JSON.stringify(join(dir, 'cache'))} }, + };`, + ); + + Object.assign(globalThis, { cancelTheRun: () => controller.abort(new Error('SIGINT')) }); + + const { code, stderr } = await run( + ['grab', '--config', config, '--quiet', '--explain', 'held'], + controller.signal, + ); + + expect(code).toBe(130); + expect(stderr).not.toContain('slow down'); + }); + + it('lists what a grab said of each thing asked for, once it is done', async () => { + const dir = await tempDir(); + // Two channels, the second of which the source has nothing on. + const config = await configFile( + dir, + `export default { + sites: [{ + site: 'example.tv', + channels: [ + { xmltvId: 'one.example.tv', siteId: '1', name: 'One' }, + { xmltvId: 'two.example.tv', siteId: '2', name: 'Two' }, + ], + async request({ day }) { return { day }; }, + parseDay: ({ channel, day }) => channel.siteId === '2' ? [] : [{ + channel: channel.xmltvId, + start: new Date(day + 'T06:00:00.000Z'), + title: [{ value: 'Show' }], + }], + }], + days: 2, + output: ${JSON.stringify(join(dir, 'guide.xml'))}, + cache: { dir: ${JSON.stringify(join(dir, 'cache'))} }, + };`, + ); + const { code, stderr } = await run([ + 'grab', + '--config', + config, + '--explain', + 'empty', + '--explain', + 'requests', + '--explain', + 'held', + '--quiet', + ]); + + expect(code).toBe(0); + // Its other channel has programmes those days: a gap in its listings. + expect(stderr).toMatch( + /^2 channel-days came back empty:\n {2}two\.example\.tv \(example\.tv\): \d{4}-\d\d-\d\d\.\.\d{4}-\d\d-\d\d \(2 days\) — example\.tv has programmes on other channels then: a gap in its listings, or off air\n\n/, + ); + // Its four requests, how long they took, and where its time went. + expect(stderr).toMatch(/\n {2}example\.tv {2}request +4 +\d+ms +\d+ms\n/); + expect(stderr).toMatch(/\nWhere each site's time went:\n {2}example\.tv took \d+ms: /); + expect(stderr).toMatch(/\n\nNo site was told to slow down\.\n$/); + }); +}); + describe('a reporter that throws', () => { it('says what the run failed of, and what its reporter threw beside it', async () => { const dir = await tempDir(); diff --git a/test/clock.test.ts b/test/clock.test.ts new file mode 100644 index 0000000..0b9c5df --- /dev/null +++ b/test/clock.test.ts @@ -0,0 +1,391 @@ +import { performance } from 'node:perf_hooks'; +import { afterEach, describe, expect, it, vi } from 'vitest'; +import { + addedTime, + measuredOf, + SiteClock, + timeSpent, + type TimeSink, +} from '../src/grabber/clock.js'; +import type { MeasuredTime } from '../src/core/events.js'; + +/** A clock read off `now`, which the test moves on by hand — telling `sink`, where there is one. */ +function clocked(sink?: TimeSink): { clock: SiteClock; at(ms: number): void } { + let now = 0; + + vi.spyOn(performance, 'now').mockImplementation(() => now); + + return { + clock: new SiteClock(sink), + at: (ms) => { + now = ms; + }, + }; +} + +afterEach(() => { + vi.restoreAllMocks(); +}); + +describe('SiteClock', () => { + it('gives each moment to the first part that holds, and no moment twice', () => { + const { clock, at } = clocked(); + + clock.turn(true); + clock.queued(); // 0–10 queued + at(10); + clock.queued(); + clock.started(); // 10–30 working, though one still waits for a slot + at(30); + clock.ask(); + clock.sent(); // 30–35 out to the source + at(35); + clock.ask(); // one more waiting behind it + clock.held(true); // 35–50 held: what is out is held up with it + at(40); + clock.answered(); // held still, with one waiting behind it + at(45); + clock.rateLimited(true); // held still + at(50); + clock.held(false); // 50–60 its own rate limit + at(60); + clock.rateLimited(false); // 60–70 working again + at(70); + clock.finished(); // 70–75 queued again + at(75); + clock.started(); // 75–77 working + at(77); + clock.finished(); // 77–80 none of them + at(80); + + expect(clock.spent()).toEqual({ source: 5, held: 15, rateLimit: 10, working: 32, queued: 15 }); + }); + + it('gives a hold the request out under it, as ky retrying a 429 holds one', () => { + const { clock, at } = clocked(); + + clock.turn(true); + clock.ask(); + clock.sent(); + at(5); + // Answered 429: held, and the same request out again once it is let go. + clock.held(true); + at(65); + clock.held(false); + at(70); + clock.answered(); + + expect(clock.spent()).toEqual({ source: 10, held: 60, rateLimit: 0, working: 0, queued: 0 }); + }); + + it('gives a hold, or its rate limit, only what a request waits behind it', () => { + const { clock, at } = clocked(); + + clock.turn(true); + clock.queued(); + clock.started(); + // Held with nothing waiting: its own work goes on, and is what it does. + clock.held(true); + clock.rateLimited(true); + at(10); + clock.ask(); // 10–15 held, a request waiting + at(15); + clock.held(false); // 15–20 its rate limit + at(20); + clock.sent(); // 20–25 out + at(25); + clock.answered(); // 25–30 working: the limit holds nothing up + at(30); + + expect(clock.spent()).toEqual({ source: 5, held: 5, rateLimit: 5, working: 15, queued: 0 }); + }); + + it('counts only its turn, and a request asked for outside one', () => { + const { clock, at } = clocked(); + + // Before its turn, and nothing asked: off the clock. + at(10); + // Its list, settled before the grab. + expect(clock.ask()).toBe(true); + clock.sent(); + at(30); + clock.answered(); + clock.settled(); + // A hold that outlived the list, waiting for the turn: not counted. + clock.held(true); + at(50); + clock.turn(true); + // In its turn, a request puts nothing more on the clock — and waits behind + // the hold. + expect(clock.ask()).toBe(false); + at(60); + clock.held(false); + clock.turn(false); + at(90); + + expect(clock.spent()).toEqual({ source: 20, held: 10, rateLimit: 0, working: 0, queued: 0 }); + }); + + it('reads the clock only when what holds changes', () => { + const { clock } = clocked(); + const reads = vi.mocked(performance.now); + + clock.turn(true); + clock.ask(); + clock.sent(); + // A parse beginning and ending while a request is out changes nothing. + clock.queued(); + clock.started(); + clock.finished(); + clock.ask(); + clock.sent(); + clock.answered(); + expect(reads).toHaveBeenCalledTimes(1); + + clock.answered(); + expect(reads).toHaveBeenCalledTimes(2); + }); + + it('gives a wait on the run’s thread to queued, out of what its work alone was given', () => { + const told: Array<[MeasuredTime, number]> = []; + const { clock, at } = clocked((part, ms) => told.push([part, ms])); + + clock.turn(true); + clock.queued(); + clock.started(); + at(10); + // A cache call, of which 6 of its 8 milliseconds waited for a slot there. + const call = clock.elsewhere(); + + at(18); + clock.back(call, 6); + expect(clock.spent()).toEqual({ source: 0, held: 0, rateLimit: 0, working: 12, queued: 6 }); + + // Never more than that work alone was given: here, a request was out. + const again = clock.elsewhere(); + + clock.ask(); + clock.sent(); + at(30); + clock.answered(); + clock.back(again, 12); + expect(clock.spent()).toEqual({ source: 12, held: 0, rateLimit: 0, working: 12, queued: 6 }); + expect(told).toEqual([ + ['working', 10], + ['working', 8], + ['working', -6], + ['queued', 6], + ['source', 12], + ]); + }); + + it('leaves a moment the site’s other work was under way to that work', () => { + const { clock, at } = clocked(); + + clock.turn(true); + clock.queued(); + clock.queued(); + clock.started(); + clock.started(); + // One parse's write waits on the run's thread while the other parses. + const call = clock.elsewhere(); + + at(20); + clock.finished(); + // The other done: from here, only the wait. + at(25); + clock.back(call, 25); + + expect(clock.spent()).toEqual({ source: 0, held: 0, rateLimit: 0, working: 20, queued: 5 }); + }); + + it('gives a call’s wait to queued, whenever the site’s other work was under way', () => { + const { clock, at } = clocked(); + + clock.turn(true); + clock.queued(); + clock.started(); + // The call waits 6 for a slot there with nothing else of the site's under + // way, and runs 4 there while another parse does here. + const call = clock.elsewhere(); + + at(6); + clock.queued(); + clock.started(); + at(10); + clock.back(call, 6); + + expect(clock.spent()).toEqual({ source: 0, held: 0, rateLimit: 0, working: 4, queued: 6 }); + }); + + it('takes a call back twice as back once, leaving the others out', () => { + const { clock, at } = clocked(); + + clock.turn(true); + clock.queued(); + clock.started(); + const call = clock.elsewhere(); + + at(2); + const other = clock.elsewhere(); + + at(10); + clock.back(call, 10); + clock.back(call, 10); + // Still waiting there, and all the site is doing: one wait, 0–20. + at(20); + clock.back(other, 18); + + expect(clock.spent()).toEqual({ source: 0, held: 0, rateLimit: 0, working: 0, queued: 20 }); + }); + + it('gives a pass reading its document to the source, where nothing else holds', () => { + const { clock, at } = clocked(); + + clock.turn(true); + clock.reading(true); // 0–10 the document arriving + at(10); + clock.queued(); + clock.started(); // 10–15 a write under way + at(15); + clock.finished(); + clock.queued(); // 15–20 a write waiting for a slot, the pass held back behind it + at(20); + clock.started(); + clock.finished(); // 20–30 reading again + at(30); + clock.reading(false); // 30–40 done: none of them + at(40); + + expect(clock.spent()).toEqual({ source: 20, held: 0, rateLimit: 0, working: 5, queued: 5 }); + }); + + it('settles a call’s wait once no call still out ran in it', () => { + const { clock, at } = clocked(); + + clock.turn(true); + clock.queued(); + clock.queued(); + clock.started(); + clock.started(); + + // A goes at 0, waits 8 for a slot there and runs 2; B's parse goes on here + // until it goes at 2, waits 2 and runs 8. Back first, A cannot yet say + // which of its wait was B's run. + const a = clock.elsewhere(); + + at(2); + const b = clock.elsewhere(); + + at(10); + clock.back(a, 8); + clock.finished(); + expect(clock.spent().queued).toBe(0); + + at(12); + clock.back(b, 2); + clock.finished(); + + // 0–2 B's parse, 2–4 both waiting, 4–12 B running and A's 8–10 with it. + expect(clock.spent()).toEqual({ source: 0, held: 0, rateLimit: 0, working: 10, queued: 2 }); + }); + + it('takes no call’s wait out of another’s work there, two being there at once', () => { + const { clock, at } = clocked(); + + clock.turn(true); + clock.queued(); + clock.queued(); + clock.started(); + clock.started(); + + // Both go to the run's thread at once: A waits 10 for a slot and runs 2; + // B waits 40 and runs 5. + const a = clock.elsewhere(); + const b = clock.elsewhere(); + + at(12); + clock.back(a, 10); + clock.finished(); + at(45); + clock.back(b, 40); + clock.finished(); + + // 0–10 both waiting, 10–12 A running, 12–40 B waiting, 40–45 B running. + expect(clock.spent()).toEqual({ source: 0, held: 0, rateLimit: 0, working: 7, queued: 38 }); + }); + + it('lets go of what a cancel dropped, and of anything asked after it', () => { + const { clock, at } = clocked(); + + clock.turn(true); + clock.queued(); + clock.ask(); + clock.held(true); + at(10); + clock.cancelled(); + // Asked once cancelled, and dropped at once: nothing waits — not behind a + // hold a response still in flight then brings. + clock.queued(); + clock.ask(); + clock.held(true); + at(30); + + expect(clock.spent()).toEqual({ source: 0, held: 10, rateLimit: 0, working: 0, queued: 0 }); + }); + + it('tells its sink of each stretch as it is given', () => { + const told: Array<[MeasuredTime, number]> = []; + const { clock, at } = clocked((part, ms) => told.push([part, ms])); + + clock.turn(true); + clock.ask(); + clock.sent(); + at(5); + clock.answered(); + clock.queued(); + clock.started(); + at(12); + clock.finished(); + + expect(told).toEqual([ + ['source', 5], + ['working', 7], + ]); + }); +}); + +describe('timeSpent', () => { + it('rounds each part, and leaves the rest of ms as other', () => { + expect( + timeSpent(100, { source: 10.4, held: 0, rateLimit: 0, working: 20.6, queued: 0.2 }), + ).toEqual({ + ms: 100, + msBy: { source: 10, held: 0, rateLimit: 0, working: 21, queued: 0, other: 69 }, + }); + }); + + it('takes ms up to what was measured, where that came to more', () => { + expect( + timeSpent(29.6, { source: 20.6, held: 0, rateLimit: 0, working: 10, queued: 0 }), + ).toEqual({ + ms: 31, + msBy: { source: 21, held: 0, rateLimit: 0, working: 10, queued: 0, other: 0 }, + }); + }); + + it('adds two measures part by part', () => { + expect( + addedTime( + { source: 1, held: 2, rateLimit: 3, working: 4, queued: 5 }, + { source: 10, held: 20, rateLimit: 30, working: 40, queued: 50 }, + ), + ).toEqual({ source: 11, held: 22, rateLimit: 33, working: 44, queued: 55 }); + }); + + it('is what two of them are added from, without other', () => { + expect( + measuredOf({ source: 1, held: 2, rateLimit: 3, working: 4, queued: 5, other: 6 }), + ).toEqual({ source: 1, held: 2, rateLimit: 3, working: 4, queued: 5 }); + }); +}); diff --git a/test/explain.test.ts b/test/explain.test.ts new file mode 100644 index 0000000..a42e11b --- /dev/null +++ b/test/explain.test.ts @@ -0,0 +1,1110 @@ +import { Writable } from 'node:stream'; +import { describe, expect, it } from 'vitest'; +import { explanations, type Explainable } from '../src/cli/explain.js'; +import type { EpgConfig } from '../src/config.js'; +import { stamped, type EpgEventInput, type TimeSpent } from '../src/core/events.js'; +import type { AnySiteConfig } from '../src/grabber/types.js'; + +/** What a run writes on stderr: every chunk, as it was written. */ +class Sink extends Writable { + readonly chunks: string[] = []; + + override _write(chunk: Buffer | string, _encoding: string, done: () => void): void { + this.chunks.push(String(chunk)); + done(); + } + + get text(): string { + return this.chunks.join(''); + } +} + +const CONFIG: EpgConfig = { sites: [], output: 'guide.xml' }; + +/** A config naming one site, with what an explanation reads of it. */ +function configWith( + site: Partial & { site: string }, + more: Partial = {}, +): EpgConfig { + return { + ...CONFIG, + sites: [ + { channels: [], request: async () => ({}), parseDay: () => [], ...site } as AnySiteConfig, + ], + ...more, + }; +} + +/** What the explanations `names` write, once a run has said `events`. */ +async function written( + names: Explainable[], + events: EpgEventInput[], + config: EpgConfig = CONFIG, +): Promise { + const { reporter, render } = explanations(names); + const emit = stamped(reporter!); + + for (const event of events) { + emit(event); + } + + const out = new Sink(); + + await render(out, config); + + return out.text; +} + +/** One channel-day, fetched with `programmes` programmes. */ +function fetched(site: string, channelId: string, day: string, programmes = 0): EpgEventInput { + return { type: 'entry:fetched', site, channelId, day, programmes }; +} + +/** One channel-day, fresh in the cache. */ +function cached(site: string, channelId: string, day: string): EpgEventInput { + return { type: 'entry:cached', site, channelId, day }; +} + +/** + * A site done in `ms`, with these requests — `[requests, requestMs, waitedMs, + * failed]` by label — and its time as `spent` says, the rest of it other. + */ +function done( + site: string, + ms: number, + requestsBy: Record?]> = {}, + spent: Partial> = {}, +): EpgEventInput { + const measured = { source: 0, held: 0, rateLimit: 0, working: 0, queued: 0, ...spent }; + const by = Object.entries(requestsBy).map( + ([label, [requests, requestMs, waitedMs, failed = {}]]) => [ + label, + { requests, requestMs, waitedMs, failed }, + ], + ); + + return { + type: 'site:done', + site, + requests: Object.values(requestsBy).reduce((sum, [requests]) => sum + requests, 0), + ms, + requestMs: 0, + requestsBy: Object.fromEntries(by), + msBy: { ...measured, other: ms - Object.values(measured).reduce((sum, part) => sum + part, 0) }, + fetched: 0, + empty: 0, + fromCache: 0, + unchanged: 0, + failed: 0, + sitesFailed: 0, + }; +} + +/** A near miss on `channelId`: the programme that joined nothing, and the one nearest it. */ +function nearMiss( + channelId: string, + why: Extract['why'], + programme: [site: string, title: string, start: string], + nearby: [site: string, title: string, start: string], +): EpgEventInput { + const at = (start: string): number => + Date.UTC( + +start.slice(0, 4), + +start.slice(4, 6) - 1, + +start.slice(6, 8), + +start.slice(8, 10), + +start.slice(10, 12), + ); + + return { + type: 'merge:unmatched', + channelId, + shiftMs: at(programme[2]) - at(nearby[2]), + why, + programme: { site: programme[0], title: programme[1], start: `${programme[2]}00 +0000` }, + nearby: { site: nearby[0], title: nearby[1], start: `${nearby[2]}00 +0000` }, + }; +} + +describe('explanations', () => { + it('hears nothing and writes nothing where none is asked for', async () => { + const { reporter, render } = explanations([]); + const out = new Sink(); + + await render(out, CONFIG); + + expect(reporter).toBeUndefined(); + expect(out.text).toBe(''); + }); + + it('writes each once, in the order asked for, a blank line between them', async () => { + expect(await written(['held', 'empty', 'held'], [])).toBe( + 'No site was told to slow down.\n\nNo channel-day came back empty.\n', + ); + }); + + it('writes a long one a thousand lines at a time', async () => { + const { reporter, render } = explanations(['unmatched']); + const emit = stamped(reporter!); + + for (let n = 0; n < 2_500; n++) { + emit( + nearMiss( + 'one.tv', + 'start', + ['b.tv', `Show ${n}`, '202610240607'], + ['a.tv', `Show ${n}`, '202610240600'], + ), + ); + } + + const out = new Sink(); + + await render(out, CONFIG); + + // Its heading, its one group, the blank line and the list's own heading, + // then 2,500 lines — in three writes, and the empty one each waits on. + const lines = out.chunks + .filter((chunk) => chunk !== '') + .map((chunk) => chunk.split('\n').length - 1); + + expect(lines).toEqual([1_000, 1_000, 505]); + expect(out.text.split('\n')[2_504]).toContain('"Show 2499"'); + }); + + it('says what an explanation throws, rather than ending it quietly', async () => { + const { reporter, render } = explanations(['requests']); + + // A site:done from no run of this version, with no requestsBy to read. + stamped(reporter!)({ type: 'site:done', site: 'a.tv', requests: 1, ms: 5 } as never); + + await expect(render(new Sink(), CONFIG)).rejects.toThrow(TypeError); + }); + + it('writes a batch the blank line between two filled, and every one after it', async () => { + const { reporter, render } = explanations(['empty', 'unmatched']); + const emit = stamped(reporter!); + + // A heading and 998 lines, so the blank line after them is the 1000th. + emit(fetched('a.tv', 'busy.tv', '2026-10-24', 3)); + + for (let n = 0; n < 998; n++) { + emit(fetched('a.tv', `c${String(n).padStart(3, '0')}.tv`, '2026-10-24')); + } + + for (let n = 0; n < 1_500; n++) { + emit( + nearMiss( + 'one.tv', + 'start', + ['b.tv', 'Show', '202610240607'], + ['a.tv', 'Show', '202610240600'], + ), + ); + } + + const out = new Sink(); + + await render(out, CONFIG); + + const lines = out.chunks + .filter((chunk) => chunk !== '') + .map((chunk) => chunk.split('\n').length - 1); + + expect(lines.reduce((sum, n) => sum + n, 0)).toBe(999 + 1 + 1_505); + expect(Math.max(...lines)).toBeLessThanOrEqual(1_001); + }); + + it('counts each hold once in each explanation that reads it', async () => { + const events: EpgEventInput[] = [ + { type: 'pacing:held', site: 'a.tv', status: 429, ms: 1_000 }, + { type: 'pacing:held', site: 'a.tv', status: 429, ms: 1_000 }, + done('a.tv', 10_000, { request: [40, 1, 0] }, { source: 9_000 }), + ]; + const text = await written(['requests', 'held'], events); + + expect(text).toContain('told to slow down twice'); + expect(text).toContain(' a.tv: held twice,'); + // Heard only where something reads it. + expect(explanations(['unmatched']).reporter!.wants!('pacing:held')).toBe(false); + expect(explanations(['requests']).reporter!.wants!('pacing:held')).toBe(true); + }); + + it('fails nothing for a stderr nothing reads any more', async () => { + const gone = new Writable({ + write(_chunk, _encoding, callback) { + callback(Object.assign(new Error('write EPIPE'), { code: 'EPIPE' })); + }, + }); + + gone.on('error', () => {}); + + const { reporter, render } = explanations(['empty']); + + stamped(reporter!)(fetched('a.tv', 'one.tv', '2026-10-24')); + + await expect(render(gone, CONFIG)).resolves.toBeUndefined(); + }); +}); + +describe('the unmatched explanation', () => { + it('groups them by channel, sources and why, says what would join them, then lists each', async () => { + expect( + await written( + ['unmatched'], + [ + nearMiss( + 'ct1.cz', + 'start', + ['b.tv', 'Zprávy', '202610250607'], + ['a.tv', 'Zprávy', '202610250600'], + ), + nearMiss( + 'ct1.cz', + 'start', + ['b.tv', 'Zprávy', '202610250807'], + ['a.tv', 'Zprávy', '202610250800'], + ), + nearMiss( + 'ct1.cz', + 'start', + ['b.tv', 'Počasí', '202610251410'], + ['a.tv', 'Počasí', '202610251400'], + ), + nearMiss( + 'dvtv.cz', + 'title', + ['b.tv', 'DVTV: Jakub Kříž', '202610252002'], + ['a.tv', 'DVTV', '202610252000'], + ), + nearMiss( + 'dvtv.cz', + 'title', + ['b.tv', 'DVTV: Petra Nová', '202610252101'], + ['a.tv', 'DVTV', '202610252100'], + ), + nearMiss( + 'dvtv.cz', + 'title', + ['b.tv', 'DVTV: Ivan Hořejší', '202610252201'], + ['a.tv', 'DVTV', '202610252200'], + ), + nearMiss( + 'dvtv.cz', + 'runs', + ['b.tv', 'Clip', '202610252301'], + ['a.tv', 'Clip', '202610252300'], + ), + ], + ), + ).toBe( + '7 programmes nearly merged, on 2 channels:\n' + + ' ct1.cz, b.tv against a.tv:\n' + + ' 3 × starts too far apart: b.tv starts 7 minutes later on 2 of them — ' + + 'a startToleranceMs of 600000 would bring them within reach (it is 300000), and join those whose shorter lasts longer than they are apart\n' + + ' dvtv.cz, b.tv against a.tv:\n' + + ' 3 × titles differ: b.tv\'s titles go on past a.tv\'s after ": " on all 3, ' + + 'e.g. "DVTV: Jakub Kříž" / "DVTV" — a normalizeTitle that cuts them there would join them\n' + + ' 1 × they start further apart than the shorter lasts: programmes in a row, or one within the other — rightly kept apart\n' + + '\n' + + 'Each of them:\n' + + ' ct1.cz: 420s apart, starts too far apart — "Zprávy" (b.tv, 06:07) / "Zprávy" (a.tv, 06:00)\n' + + ' ct1.cz: 420s apart, starts too far apart — "Zprávy" (b.tv, 08:07) / "Zprávy" (a.tv, 08:00)\n' + + ' ct1.cz: 10m00s apart, starts too far apart — "Počasí" (b.tv, 14:10) / "Počasí" (a.tv, 14:00)\n' + + ' dvtv.cz: 120s apart, titles differ — "DVTV: Jakub Kříž" (b.tv, 20:02) / "DVTV" (a.tv, 20:00)\n' + + ' dvtv.cz: 60s apart, titles differ — "DVTV: Petra Nová" (b.tv, 21:01) / "DVTV" (a.tv, 21:00)\n' + + ' dvtv.cz: 60s apart, titles differ — "DVTV: Ivan Hořejší" (b.tv, 22:01) / "DVTV" (a.tv, 22:00)\n' + + ' dvtv.cz: 60s apart, they start further apart than the shorter lasts — "Clip" (b.tv, 23:01) / "Clip" (a.tv, 23:00)\n', + ); + }); + + it('says which source runs behind, earlier as well as later, and on both of two', async () => { + const text = await written( + ['unmatched'], + [ + nearMiss( + 'one.tv', + 'start', + ['b.tv', 'News', '202610250553'], + ['a.tv', 'News', '202610250600'], + ), + nearMiss( + 'one.tv', + 'start', + ['b.tv', 'News', '202610250753'], + ['a.tv', 'News', '202610250800'], + ), + ], + ); + + expect(text).toContain('b.tv starts 7 minutes earlier on both —'); + expect(text).toContain('a startToleranceMs of 420000 would bring them within reach'); + }); + + it('takes two sources for one pair, whichever of them joined nothing', async () => { + const text = await written( + ['unmatched'], + [ + nearMiss( + 'one.tv', + 'start', + ['b.tv', 'News', '202610250607'], + ['a.tv', 'News', '202610250600'], + ), + nearMiss( + 'one.tv', + 'start', + ['b.tv', 'News', '202610250807'], + ['a.tv', 'News', '202610250800'], + ), + // B's held over from the day before, met by A's first list. + nearMiss( + 'one.tv', + 'start', + ['a.tv', 'News', '202610250553'], + ['b.tv', 'News', '202610250600'], + ), + ], + ); + + expect(text).toContain( + ' one.tv, b.tv against a.tv:\n 3 × starts too far apart: b.tv starts 7 minutes later on all 3 —', + ); + expect(text).not.toContain('a.tv against b.tv'); + }); + + it('says no shift where the pairs do not agree on one, and the tolerance a config set', async () => { + const text = await written( + ['unmatched'], + [ + nearMiss( + 'one.tv', + 'start', + ['b.tv', 'News', '202610250603'], + ['a.tv', 'News', '202610250600'], + ), + nearMiss( + 'one.tv', + 'start', + ['b.tv', 'News', '202610250809'], + ['a.tv', 'News', '202610250800'], + ), + ], + { ...CONFIG, merge: { match: { startToleranceMs: 120_000 } } }, + ); + + expect(text).toContain( + '2 × starts too far apart: a startToleranceMs of 540000 would bring them within reach (it is 120000), and join those whose shorter lasts longer than they are apart', + ); + }); + + it('rounds the tolerance it suggests up to the minute, so it reaches the widest', async () => { + const event = nearMiss( + 'one.tv', + 'start', + ['b.tv', 'News', '202610250607'], + ['a.tv', 'News', '202610250600'], + ); + + // Seven minutes and twenty seconds: eight reach it, seven do not. + const text = await written(['unmatched'], [{ ...event, shiftMs: 440_000 } as EpgEventInput]); + + expect(text).toContain('a startToleranceMs of 480000 would bring it within reach'); + }); + + it('suggests a window to a matcher that has one, and the tolerance a shipped match was given', async () => { + const pair = nearMiss( + 'one.tv', + 'start', + ['b.tv', 'News', '202610250607'], + ['a.tv', 'News', '202610250600'], + ); + const windowed = await written(['unmatched'], [pair], { + ...CONFIG, + merge: { match: { windowMs: 120_000, matches: () => false } }, + }); + const shipped = await written(['unmatched'], [pair], { + ...CONFIG, + merge: { match: { titles: 'always' } }, + }); + + expect(windowed).toContain( + '— a windowMs of 420000 would let its match be asked about it (it is 120000)\n', + ); + expect(windowed).not.toContain('startToleranceMs'); + expect(shipped).toContain( + 'a startToleranceMs of 420000 would bring it within reach (it is 300000)', + ); + }); + + it('suggests no tolerance where the config matches programmes its own way', async () => { + const text = await written( + ['unmatched'], + [ + nearMiss( + 'one.tv', + 'start', + ['b.tv', 'News', '202610250607'], + ['a.tv', 'News', '202610250600'], + ), + ], + { ...CONFIG, merge: { match: () => false } }, + ); + + // Of one pair, said of it — and no tolerance, which is the config's own. + expect(text).toContain(' 1 × starts too far apart: b.tv starts 7 minutes later\n'); + }); + + it('says where one source’s titles begin with more, whichever side', async () => { + const text = await written( + ['unmatched'], + [ + nearMiss( + 'films.tv', + 'title', + ['b.tv', 'Titanic', '202610252000'], + ['a.tv', 'Film - Titanic', '202610252001'], + ), + nearMiss( + 'films.tv', + 'title', + ['b.tv', 'Ghost', '202610252200'], + ['a.tv', 'Film - Ghost', '202610252201'], + ), + ], + ); + + expect(text).toContain( + 'a.tv\'s titles begin with more, before " - " on both, e.g. "Titanic" / "Film - Titanic" — ' + + 'a normalizeTitle that keeps what follows it would join them', + ); + }); + + it('compares titles as the merge does: by the config’s normalizeTitle, accents aside', async () => { + const accents = await written( + ['unmatched'], + [ + nearMiss( + 'one.tv', + 'title', + ['b.tv', 'Večerníček: Krtko', '202610251900'], + ['a.tv', 'Vecernicek', '202610251901'], + ), + ], + ); + const own = await written( + ['unmatched'], + [ + nearMiss( + 'two.tv', + 'title', + ['b.tv', 'Zprávy: Večer', '202610251900'], + ['a.tv', 'News', '202610251901'], + ), + ], + { + ...CONFIG, + merge: { + match: { normalizeTitle: (title) => title.replace(/^Zprávy/, 'News').toLowerCase() }, + }, + }, + ); + + expect(accents).toContain('b.tv\'s titles go on past a.tv\'s after ": "'); + expect(own).toContain('b.tv\'s titles go on past a.tv\'s after ": "'); + }); + + it('suggests no title rule that only half a group bears out, nor one from an empty title', async () => { + const text = await written( + ['unmatched'], + [ + nearMiss( + 'one.tv', + 'title', + ['b.tv', 'DVTV: Jakub Kříž', '202610252002'], + ['a.tv', 'DVTV', '202610252000'], + ), + nearMiss( + 'one.tv', + 'title', + ['b.tv', 'News', '202610252101'], + ['a.tv', 'Weather', '202610252100'], + ), + nearMiss( + 'two.tv', + 'title', + ['b.tv', '', '202610252001'], + ['a.tv', ': Show', '202610252000'], + ), + ], + ); + + expect(text).toContain(' 2 × titles differ, e.g. "DVTV: Jakub Kříž" / "DVTV"\n'); + expect(text).toContain(' 1 × titles differ, e.g. "" / ": Show"\n'); + expect(text).not.toContain('normalizeTitle'); + }); + + it('gives an example only, where titles differ otherwise, or the config said no', async () => { + const text = await written( + ['unmatched'], + [ + nearMiss( + 'one.tv', + 'title', + ['b.tv', 'News', '202610250601'], + ['a.tv', 'Weather', '202610250600'], + ), + nearMiss( + 'two.tv', + 'matcher', + ['b.tv', 'Film', '202610250601'], + ['a.tv', 'Film', '202610250600'], + ), + nearMiss( + 'three.tv', + 'both', + ['b.tv', 'Talk', '202610250620'], + ['a.tv', 'Chat', '202610250600'], + ), + ], + ); + + expect(text).toContain(' 1 × titles differ, e.g. "News" / "Weather"\n'); + expect(text).toContain(' 1 × the config\'s match said no, e.g. "Film" / "Film"\n'); + expect(text).toContain(' 1 × titles and starts differ, e.g. "Talk" / "Chat"\n'); + }); + + it('says a merge whose channels are kept apart has none to look for', async () => { + expect( + await written(['unmatched'], [], { ...CONFIG, merge: { channelStrategy: 'keep-all' } }), + ).toBe( + "Near misses are looked for only under channelStrategy 'merge-programmes' and " + + "programmeStrategy 'merge', and this merge is under channelStrategy: 'keep-all'.\n", + ); + }); + + it('says where none nearly merged', async () => { + expect(await written(['unmatched'], [])).toBe('No programmes nearly merged.\n'); + }); +}); + +describe('the empty explanation', () => { + it('reads each empty channel-day from what the run’s other answers say', async () => { + const days = ['2026-10-25', '2026-10-26', '2026-10-27', '2026-10-28', '2026-10-29']; + const events: EpgEventInput[] = []; + + // sky.sk has programmes up to the 28th, and nothing after it on any channel. + for (const channel of ['joj.sk', 'ta3.sk']) { + for (const day of days) { + events.push(fetched('sky.sk', channel, day, day < '2026-10-29' ? 20 : 0)); + } + } + + events.push( + // magio.tv has none for joj.sk on days sky.sk does. + fetched('magio.tv', 'joj.sk', '2026-10-25'), + fetched('magio.tv', 'joj.sk', '2026-10-26'), + // Nor for dvojka.sk on one day it has the rest of its channels for. + fetched('magio.tv', 'dvojka.sk', '2026-10-25', 30), + fetched('magio.tv', 'dvojka.sk', '2026-10-27'), + fetched('magio.tv', 'joj.sk', '2026-10-27', 12), + // Nor for one it and sky.sk carry, on a day both are empty. + fetched('magio.tv', 'markiza.sk', '2026-10-26'), + fetched('sky.sk', 'markiza.sk', '2026-10-26'), + // And nothing at all from a source whose parse has broken. + fetched('broken.tv', 'one.tv', '2026-10-25'), + fetched('broken.tv', 'two.tv', '2026-10-25'), + ); + + expect(await written(['empty'], events)).toBe( + '9 channel-days came back empty:\n' + + ' broken.tv: every channel-day it was asked about came back empty — 2 channel-days on 2 channels: ' + + 'its parseDay may no longer read what the source sends, or the source has nothing\n' + + ' sky.sk: nothing after 2026-10-28 on any channel — it has not published that far yet: ' + + '2026-10-29, on 2 channels\n' + + ' dvojka.sk (magio.tv): 2026-10-27 — magio.tv has programmes on other channels then: ' + + 'a gap in its listings, or off air\n' + + ' joj.sk (magio.tv): 2026-10-25..2026-10-26 (2 days) — sky.sk has programmes then: ' + + 'magio.tv may know it by another id, or have stopped carrying it\n' + + ' markiza.sk (magio.tv): 2026-10-26 — every source asked about it came back empty then: off air, or not published yet\n' + + ' markiza.sk (sky.sk): 2026-10-26 — every source asked about it came back empty then: off air, or not published yet\n', + ); + }); + + it('claims nothing a channel-day served from the cache could make untrue', async () => { + const text = await written( + ['empty'], + [ + fetched('a.tv', 'one.tv', '2026-10-25', 10), + // Cached for the 26th, so a.tv may well have published that far. + cached('a.tv', 'two.tv', '2026-10-26'), + fetched('a.tv', 'one.tv', '2026-10-26'), + // b.tv's 25th cached: whether it has programmes then is not known — + // so c.tv's being empty too is not every source's. + cached('b.tv', 'three.tv', '2026-10-25'), + fetched('c.tv', 'three.tv', '2026-10-25'), + fetched('c.tv', 'four.tv', '2026-10-25', 5), + fetched('a.tv', 'three.tv', '2026-10-25'), + ], + ); + + expect(text).not.toContain('published that far'); + expect(text).not.toContain('every source'); + expect(text).toContain( + " one.tv (a.tv): 2026-10-26 — nothing else says otherwise: off air, or a gap in a.tv's listings\n", + ); + expect(text).toContain( + ' three.tv (a.tv): 2026-10-25 — a.tv has programmes on other channels then: a gap in its listings, or off air\n', + ); + }); + + it('takes a day the source said unchanged, or that failed, as one it may have programmes on', async () => { + const text = await written( + ['empty'], + [ + // Unchanged everywhere but one day: not a parse that has broken. + { type: 'entry:unchanged', site: 'a.tv', channelId: 'one.tv', day: '2026-10-25' }, + { type: 'entry:unchanged', site: 'a.tv', channelId: 'one.tv', day: '2026-10-27' }, + fetched('a.tv', 'one.tv', '2026-10-26'), + // Empty at b.tv and g.tv, failed at c.tv and in a request of d.tv's: not + // every source's. + fetched('b.tv', 'two.tv', '2026-10-26'), + fetched('b.tv', 'three.tv', '2026-10-26', 4), + fetched('g.tv', 'two.tv', '2026-10-26'), + fetched('g.tv', 'three.tv', '2026-10-26', 4), + { + type: 'entry:failed', + site: 'c.tv', + channelId: 'two.tv', + day: '2026-10-26', + error: new Error('no'), + }, + fetched('e.tv', 'four.tv', '2026-10-26'), + fetched('e.tv', 'five.tv', '2026-10-26', 4), + fetched('f.tv', 'four.tv', '2026-10-26'), + fetched('f.tv', 'five.tv', '2026-10-26', 4), + { + type: 'request:failed', + site: 'd.tv', + channels: ['four.tv', 'six.tv'], + days: ['2026-10-25', '2026-10-26'], + entries: 4, + error: new Error('no'), + }, + ], + ); + + expect(text).not.toContain('every channel-day it was asked about'); + expect(text).not.toContain('every source'); + expect(text).toContain( + " one.tv (a.tv): 2026-10-26 — nothing else says otherwise: off air, or a gap in a.tv's listings\n", + ); + }); + + it('says every source asked that answered, where one failed whole and said nothing of the day', async () => { + const text = await written( + ['empty'], + [ + fetched('a.tv', 'one.tv', '2026-10-26'), + fetched('a.tv', 'two.tv', '2026-10-26', 3), + fetched('b.tv', 'one.tv', '2026-10-26'), + fetched('b.tv', 'two.tv', '2026-10-26', 3), + { type: 'site:failed', site: 'c.tv', error: new Error('no channel list') }, + // Failed too, though only once it had answered the day — empty. + fetched('d.tv', 'one.tv', '2026-10-26'), + fetched('d.tv', 'two.tv', '2026-10-26', 3), + { type: 'site:failed', site: 'd.tv', error: new Error('its thread went') }, + ], + ); + + expect(text).toContain( + ' one.tv (a.tv): 2026-10-26 — every source asked about it that answered came back empty then, ' + + 'though c.tv failed and may have it: off air, or not published yet\n', + ); + }); + + it('leaves out a day a second emission filled, as a run counts it', async () => { + const appended = (channelId: string, total: number): EpgEventInput => ({ + type: 'entry:appended', + site: 'a.tv', + channelId, + day: '2026-10-24', + added: total, + total, + }); + + const text = await written( + ['empty'], + [ + fetched('a.tv', 'one.tv', '2026-10-24'), + fetched('a.tv', 'two.tv', '2026-10-24'), + fetched('a.tv', 'three.tv', '2026-10-24'), + appended('one.tv', 2), + appended('two.tv', 0), + ], + ); + + expect(text).toMatch(/^2 channel-days came back empty:\n/); + expect(text).not.toContain('one.tv'); + expect(text).toContain(' three.tv (a.tv): 2026-10-24 —'); + expect(text).toContain(' two.tv (a.tv): 2026-10-24 —'); + }); + + it('says a source has programmes on other channels only where it has them every day of the run', async () => { + const text = await written( + ['empty'], + [ + fetched('a.tv', 'one.tv', '2026-10-26'), + fetched('a.tv', 'one.tv', '2026-10-27'), + fetched('a.tv', 'two.tv', '2026-10-27', 5), + ], + ); + + expect(text).toContain( + " one.tv (a.tv): 2026-10-26..2026-10-27 (2 days) — nothing else says otherwise: off air, or a gap in a.tv's listings\n", + ); + }); + + it('runs days on across the end of a month', async () => { + const text = await written( + ['empty'], + [ + fetched('a.tv', 'two.tv', '2026-11-01', 5), + fetched('a.tv', 'one.tv', '2026-10-31'), + fetched('a.tv', 'one.tv', '2026-11-01'), + ], + ); + + expect(text).toContain(' one.tv (a.tv): 2026-10-31..2026-11-01 (2 days) —'); + }); + + it('says where none came back empty', async () => { + expect(await written(['empty'], [fetched('a.tv', 'one.tv', '2026-10-24', 3)])).toBe( + 'No channel-day came back empty.\n', + ); + }); +}); + +describe('the requests explanation', () => { + it('tables each site’s requests, then says where its time went, and why', async () => { + expect( + await written( + ['requests'], + [ + done( + 'sky.sk', + 697_000, + { + request: [49, 7_310, 2_100], + detail: [5_242, 577_400, 44_000, { 404: 3_242, timeout: 3 }], + }, + { source: 560_000, held: 90_000, rateLimit: 22_000, working: 15_000 }, + ), + done('cached.tv', 412), + done( + 'magio.tv', + 170_412, + { request: [2, 939, 0] }, + { source: 939, working: 1_204, queued: 168_100 }, + ), + ], + configWith({ site: 'sky.sk', concurrency: 4 }), + ), + ).toBe( + 'Requests, by site and what each was for:\n' + + ' site label requests waiting in queue failed\n' + + ' magio.tv request 2 939ms 0ms\n' + + ' sky.sk request 49 7310ms 2100ms\n' + + ' sky.sk detail 5242 577s 44s 3242 × 404 and 3 × timeout\n' + + ' None from cached.tv.\n' + + '\n' + + "Where each site's time went:\n" + + ' cached.tv took 412ms: 412ms on other things.\n' + + ' magio.tv took 170s: 168s waiting for a local slot, 1204ms parsing and writing, ' + + '939ms on the source, 169ms on other things.\n' + + " Mostly waiting for one of the run's 16 local slots, which other sites' work held: " + + 'a higher localConcurrency, or --threads, lets it in sooner.\n' + + ' sky.sk took 11m37s: 560s on the source, 90s held by the source, 22s under its own rateLimit, ' + + '15s parsing and writing, 10s on other things.\n' + + ' Mostly waiting on the source, 4 requests at a time: the source is the slow part.\n' + + ' 3242 of its 5242 detail requests were answered 404: the source has nothing at those ' + + 'addresses — ids it no longer knows, or details of programmes it has dropped.\n' + + " 3 of its 5242 detail requests took longer than the site's timeout.\n", + ); + }); + + it('reads whichever part is most of a site’s time', async () => { + const mostly = async ( + spent: Partial>, + config: EpgConfig = CONFIG, + ): Promise => + (await written(['requests'], [done('a.tv', 10_000, { request: [40, 1, 0] }, spent)], config)) + .split('\n') + .find((line) => line.startsWith(' Mostly'))! + .trim(); + + expect(await mostly({ source: 9_000 })).toBe( + 'Mostly waiting on the source, one request at a time: a concurrency above 1 would overlap them, where the source allows it.', + ); + // Nothing to overlap. + expect( + ( + await written( + ['requests'], + [done('a.tv', 10_000, { request: [1, 1, 0] }, { source: 9_000 })], + ) + ) + .split('\n') + .find((line) => line.startsWith(' Mostly'))! + .trim(), + ).toBe('Mostly waiting on the source, for its one request: the source is the slow part.'); + expect(await mostly({ held: 9_000 })).toBe( + 'Mostly held by the source: it is asked faster than it allows — see --explain held.', + ); + expect(await mostly({ rateLimit: 9_000 })).toBe( + 'Mostly under its own rateLimit: the source may allow more.', + ); + expect( + await mostly( + { rateLimit: 9_000 }, + configWith({ site: 'a.tv', rateLimit: { requests: 2, perMs: 1000 } }), + ), + ).toBe('Mostly under its own rateLimit of 2 per 1000ms: the source may allow more.'); + expect(await mostly({ working: 9_000 })).toBe( + 'Mostly its own parsing and writing: a parse, or a cache, that is slow — with --threads, only the sites that share its thread wait on it.', + ); + expect(await mostly({ working: 9_000 }, { ...CONFIG, threads: true })).toBe( + 'Mostly its own parsing and writing: a parse, or a cache, that is slow.', + ); + expect(await mostly({ queued: 9_000 }, { ...CONFIG, threads: 2, localConcurrency: 4 })).toBe( + "Mostly waiting for a local slot — one of its thread's 4, or one of the run's thread's 4, where its cache is read and written — which other work held: a higher localConcurrency lets it in sooner.", + ); + expect(await mostly({})).toBe( + 'Mostly none of these: its own code between requests, or reading what it remembered.', + ); + }); + + it('says the pace a site kept on the source, where its slow-downs took it below its own', async () => { + const config = configWith({ site: 'a.tv', concurrency: 8 }); + const said = async (events: EpgEventInput[]): Promise => + (await written(['requests'], events, config)) + .split('\n') + .find((line) => line.startsWith(' Mostly'))! + .trim(); + const site = done('a.tv', 10_000, { request: [40, 1, 0] }, { source: 9_000 }); + + expect( + await said([ + { type: 'pacing:slowed', site: 'a.tv', concurrency: 1 }, + // Given some back, and slowed again — not as far. + { type: 'pacing:slowed', site: 'a.tv', concurrency: 2 }, + { type: 'pacing:slowed', site: 'b.tv', concurrency: 1 }, + site, + ]), + ).toBe( + "Mostly waiting on the source, its concurrency of 8 taken as low as 1 by the source's slow-downs — see --explain held.", + ); + expect(await said([site])).toBe( + 'Mostly waiting on the source, 8 requests at a time: the source is the slow part.', + ); + }); + + it('advises no more requests at once to a site the source held, one at a time', async () => { + const said = async (events: EpgEventInput[], config: EpgConfig = CONFIG): Promise => + (await written(['requests'], events, config)) + .split('\n') + .find((line) => line.startsWith(' Mostly'))! + .trim(); + const site = done('a.tv', 10_000, { request: [40, 1, 0] }, { source: 9_000 }); + + expect( + await said([ + { type: 'pacing:held', site: 'a.tv', status: 429, ms: 1_000 }, + { type: 'pacing:held', site: 'a.tv', status: 429, ms: 1_000 }, + site, + ]), + ).toBe( + 'Mostly waiting on the source, one request at a time, and told to slow down twice: ' + + 'it would take no more — see --explain held.', + ); + // Its local slots as the run takes them: at least one. + expect( + await said([done('a.tv', 10_000, { request: [1, 1, 0] }, { queued: 9_000 })], { + ...CONFIG, + localConcurrency: 0, + }), + ).toContain("one of the run's 1 local slots"); + }); + + it('reads nothing into a short site, or one no part is most of', async () => { + const text = await written( + ['requests'], + [ + done('quick.tv', 900, { request: [1, 1, 0] }, { source: 800 }), + done( + 'even.tv', + 10_000, + { request: [1, 1, 0] }, + { source: 4_000, working: 3_000, queued: 3_000 }, + ), + ], + ); + + expect(text).not.toContain('Mostly'); + }); + + it('says what each way a request failed most likely means', async () => { + const text = await written( + ['requests'], + [ + done('a.tv', 100, { + request: [9, 1, 0, { 403: 1, 410: 1, 429: 1, 502: 1, 418: 1, network: 1, error: 1 }], + }), + ], + ); + + expect(text).toContain( + ' 1 of its 9 request requests were refused (403): credentials, or a block.\n', + ); + expect(text).toContain( + ' 1 of its 9 request requests were answered 410: the source has nothing at those addresses', + ); + expect(text).toContain( + ' 1 of its 9 request requests were told to slow down (429) — see --explain held.\n', + ); + + // Its backoff off, it was held for neither: a 503 is the source failing. + const off = await written( + ['requests'], + [done('b.tv', 100, { request: [4, 1, 0, { 429: 1, 503: 1 }] })], + configWith({ site: 'b.tv', backoff: false }), + ); + + expect(off).toContain( + ' 1 of its 4 request requests were told to slow down (429), which its backoff does not ' + + 'hold for: backoff, or a rateLimit, would pace it.\n', + ); + expect(off).toContain(' 1 of its 4 request requests failed at the source (503).\n'); + expect(text).toContain(' 1 of its 9 request requests failed at the source (502).\n'); + expect(text).toContain( + ' 1 of its 9 request requests were answered 418: the source would not give what was asked.\n', + ); + expect(text).toContain( + ' 1 of its 9 request requests were answered by nothing: the source down, or out of reach.\n', + ); + expect(text).toContain( + " 1 of its 9 request requests threw something that was not the source's answer: the site's own code.\n", + ); + }); + + it('says where no site was run, or none sent a request', async () => { + expect(await written(['requests'], [])).toBe('No site was run.\n'); + expect(await written(['requests'], [done('a.tv', 5)])).toBe( + "No site sent a request.\n\nWhere each site's time went:\n a.tv took 5ms: 5ms on other things.\n", + ); + }); +}); + +describe('the held explanation', () => { + it('says how often each site was held, with what, how it was let go, and the pace that earned it', async () => { + expect( + await written( + ['held'], + [ + { type: 'pacing:slowed', site: 'b.tv', concurrency: 1 }, + { type: 'pacing:held', site: 'b.tv', status: 429, ms: 30_000, retryAfterMs: 30_000 }, + { type: 'pacing:recovered', site: 'b.tv', concurrency: 2 }, + { type: 'pacing:recovered', site: 'b.tv', concurrency: 3 }, + { type: 'pacing:slowed', site: 'b.tv', concurrency: 2 }, + { type: 'pacing:held', site: 'b.tv', status: 429, ms: 60_000, retryAfterMs: 120_000 }, + { type: 'pacing:held', site: 'b.tv', status: 503, ms: 5_000 }, + { type: 'pacing:recovered', site: 'b.tv', concurrency: 3 }, + // Its requests out for 560s of its 11m37s: the rest, other work. + done('b.tv', 697_000, { request: [5_291, 1, 0] }, { source: 560_000, queued: 120_000 }), + { type: 'pacing:held', site: 'a.tv', status: 418, ms: 5_000, retryAfterMs: 5_000 }, + { type: 'pacing:held', site: 'c.tv', status: 503, ms: 1_000 }, + { type: 'pacing:slowed', site: 'c.tv', concurrency: 1 }, + done('c.tv', 4_000, { request: [10, 1, 0] }, { source: 2_000 }), + ], + configWith({ site: 'c.tv', rateLimit: { requests: 5, perMs: 1000 } }), + ), + ).toBe( + '3 sites told to slow down:\n' + + ' a.tv: held once, for 5000ms\n' + + ' 1 × 418: a status its backoff.statuses lists\n' + + ' As long as the source asked, each time.\n' + + ' b.tv: held 3 times, for up to 60s at a time; concurrency down to 1, back up to 3\n' + + " 2 × 429: too many requests — the source's rate limit\n" + + ' 1 × 503: unavailable — overloaded, or a limit it does not name\n' + + ' As long as the source asked, once.\n' + + ' Cut short by backoff.maxMs once, where the source asked for up to 120s: ' + + 'let go sooner than it wanted, the site is likely to be held again.\n' + + ' For the fallback once — backoff.fallbackMs, at most backoff.maxMs — the source saying nothing of how long.\n' + + ' It sent 5291 requests, 9.4 a second over the 560s it had any out unheld, with no rateLimit: ' + + 'a rateLimit below that pace would keep it from being held, at the cost of the time that takes.\n' + + ' c.tv: held once, for 1000ms; concurrency down to 1\n' + + ' 1 × 503: unavailable — overloaded, or a limit it does not name\n' + + ' For the fallback once — backoff.fallbackMs, at most backoff.maxMs — the source saying nothing of how long.\n' + + ' It sent 10 requests, 5.0 a second over the 2000ms it had any out unheld, under a rateLimit of 5 per 1000ms: ' + + 'the source allows less than that.\n', + ); + }); + + it('counts a hold made longer as one hold, and each time it was told as one', async () => { + const text = await written( + ['held'], + [ + { type: 'pacing:held', site: 'a.tv', status: 429, ms: 5_000, retryAfterMs: 5_000 }, + // Two more answers while it held, asking longer: still one hold. + { + type: 'pacing:held', + site: 'a.tv', + status: 429, + ms: 20_000, + retryAfterMs: 20_000, + lengthened: true, + }, + { type: 'pacing:held', site: 'a.tv', status: 503, ms: 8_000, lengthened: true }, + { type: 'pacing:held', site: 'b.tv', status: 429, ms: 1_000, retryAfterMs: 1_000 }, + { + type: 'pacing:held', + site: 'b.tv', + status: 429, + ms: 2_000, + retryAfterMs: 2_000, + lengthened: true, + }, + { type: 'pacing:held', site: 'b.tv', status: 429, ms: 1_000, retryAfterMs: 1_000 }, + ], + ); + + expect(text).toBe( + '2 sites told to slow down:\n' + + ' a.tv: held once, made longer twice while it held — for up to 20s each time it was told\n' + + " 2 × 429: too many requests — the source's rate limit\n" + + ' 1 × 503: unavailable — overloaded, or a limit it does not name\n' + + ' As long as the source asked, twice.\n' + + ' For the fallback once — backoff.fallbackMs, at most backoff.maxMs — the source saying nothing of how long.\n' + + ' b.tv: held twice, made longer once while it held — for up to 2000ms each time it was told\n' + + " 3 × 429: too many requests — the source's rate limit\n" + + ' As long as the source asked, each time.\n', + ); + }); + + it('rounds a pace of ten or more a second to the whole', async () => { + const text = await written( + ['held'], + [ + { type: 'pacing:held', site: 'a.tv', status: 429, ms: 1_000 }, + { type: 'pacing:held', site: 'a.tv', status: 429, ms: 1_000 }, + { type: 'pacing:held', site: 'a.tv', status: 429, ms: 1_000 }, + done('a.tv', 1_000, { request: [124, 1, 0] }, { source: 1_000 }), + ], + ); + + expect(text).toContain(' For the fallback 3 times —'); + expect(text).toContain( + 'It sent 124 requests, 124 a second over the 1000ms it had any out unheld', + ); + }); +}); diff --git a/test/grabber.test.ts b/test/grabber.test.ts index c1d0ca4..79bfb68 100644 --- a/test/grabber.test.ts +++ b/test/grabber.test.ts @@ -1,3 +1,4 @@ +import { EventEmitter, getEventListeners } from 'node:events'; import { createServer } from 'node:http'; import type { IncomingMessage, ServerResponse } from 'node:http'; import type { AddressInfo } from 'node:net'; @@ -25,10 +26,18 @@ import { UnchangedError, } from '../src/grabber/main.js'; import { errorMessage } from '../src/core/error.js'; -import type { EpgEvent } from '../src/core/events.js'; +import { + EVENT_TYPES, + noCounts, + type EpgEvent, + type EpgEventInput, + type EpgEventMap, + type TimeSpent, +} from '../src/core/events.js'; import { MANY_LABELS, openSession } from '../src/grabber/session.js'; -import { SiteRun } from '../src/grabber/site-run.js'; -import { combineReporters, jsonReporter } from '../src/core/reporters.js'; +import PQueue from 'p-queue'; +import { SiteRun, type Run } from '../src/grabber/site-run.js'; +import { combineReporters, emitterReporter, jsonReporter } from '../src/core/reporters.js'; import { collect } from './reporting.js'; import type { ChannelsBatching, @@ -1380,6 +1389,87 @@ describe('grab', () => { expect(seen.filter((entry) => entry === undefined)).toHaveLength(1); }); + it('is saved by a site that still hears a cancel, so its pace lets go of a hold', async () => { + const run = new AbortController(); + const cache = new MemoryCache(); + let listening: boolean | undefined; + const save = cache.setState.bind(cache); + + // Asked, as the site saves what it remembered: is its pacing still + // listening for the run's cancel? + cache.setState = async (...args: Parameters) => { + listening = getEventListeners(run.signal, 'abort').some( + (listener) => listener.name === 'cancelled', + ); + + return save(...args); + }; + + await grab( + [ + makeConfig({ + async request({ state }) { + state.set('token', 'remembered'); + + return {}; + }, + }), + ], + { cache, now: NOW, signal: run.signal }, + ); + + expect(listening).toBe(true); + }); + + it('lets go of the run’s cancel, though saying its save failed throws', async () => { + const stop = new AbortController(); + const cache = new MemoryCache(); + + cache.setState = async () => { + throw new Error('the store is gone'); + }; + + const run: Run = { + defaults: { days: 1 }, + cache, + now: NOW, + grabbedAt: NOW.toISOString(), + startDay: TODAY, + timeZone: 'UTC', + // A reporter nothing keeps the throws of. + emit: Object.assign( + (event: EpgEventInput) => { + if (event.type === 'site:failed') { + throw new Error('the reporter is gone'); + } + }, + { hears: new Set(EVENT_TYPES) }, + ), + signal: stop.signal, + localWork: new PQueue({ concurrency: 1 }), + enqueue: (queue, task, options = {}) => + queue.add(task, { ...options, signal: stop.signal }), + tally: noCounts(), + requested: () => {}, + spent: undefined, + }; + const site = new SiteRun( + makeConfig({ + async request({ state }) { + state.set('token', 'remembered'); + + return {}; + }, + }), + run, + ); + + await expect(site.run()).rejects.toThrow('the reporter is gone'); + expect( + getEventListeners(stop.signal, 'abort').map((listener) => listener.name), + ).not.toContain('cancelled'); + }); + it('is the same bag its channel list was handed', async () => { const cache = new MemoryCache(); const seen: unknown[] = []; @@ -2801,6 +2891,27 @@ describe('a site that streams its whole window', () => { return { channel, day, programmes: [programme(`${day}T06:00:00.000Z`, channel.siteId)] }; } + it('gives the time its document takes to arrive to the source, not to nothing', async () => { + const report = collect(); + + await grab( + [ + streamSite(() => [], { + // A document arriving slowly, long after any headers did. + async *stream({ channels, days }) { + await new Promise((resolve) => setTimeout(resolve, 40)); + yield some(days[0]!, channels[0]!); + }, + }), + ], + { cache: new MemoryCache(), now: NOW, reporter: report.reporter }, + ); + + const [done] = report.of('site:done'); + + expect(done!.msBy.source).toBeGreaterThanOrEqual(30); + }); + it('writes every channel-day it yields, and counts them as fetched', async () => { const cache = new MemoryCache(); const summary = await grab( @@ -4438,6 +4549,92 @@ describe('what a run reports', () => { }); }); +describe('what a run makes of what its reporter does not want', () => { + it('asks once it is under way, so what listens from as it is started is heard', async () => { + const events = new EventEmitter(); + let fetched = 0; + const running = grab([makeConfig()], { + cache: new MemoryCache(), + now: NOW, + reporter: emitterReporter(events), + }); + + events.on('entry:fetched', () => { + fetched++; + }); + await running; + + expect(fetched).toBe(1); + }); + + it('makes none of it, of a site that requests or one that streams', async () => { + const unwanted = new Set([ + 'site:note', + 'request:started', + 'request:done', + 'entry:fetched', + 'entry:appended', + 'entry:cached', + 'pacing:rateLimit', + ]); + const said = makeConfig({ + // Two days, one request a window: its second waits on its rateLimit. + days: 2, + rateLimit: { requests: 1, perMs: 10 }, + // Its days from the cache, the second time. + staleness: { alwaysRefetchDays: 0 }, + parseDay({ day, log }) { + log('parsed it'); + + return [programme(`${day}T06:00:00.000Z`)]; + }, + }); + // A document not grouped by channel: its one channel-day said twice. + const streamed: StreamSiteConfig = { + site: 'stream.example', + channels: [channel('a')], + days: 1, + async *stream({ channels, days }) { + for (const at of ['06', '09']) { + yield { + channel: channels[0]!, + day: days[0]!, + programmes: [programme(`${days[0]}T${at}:00:00.000Z`, 'site-a')], + }; + } + }, + }; + /** The kinds two runs make, the second from the cache, for a reporter wanting what `wants` does. */ + const kinds = async (wants?: (type: EpgEvent['type']) => boolean): Promise> => { + const heard = new Set(); + const reporter = Object.assign( + (event: EpgEvent) => void heard.add(event.type), + wants === undefined ? {} : { wants }, + ); + const cache = new MemoryCache(); + + await grab([said, streamed], { cache, now: NOW, reporter }); + await grab([said], { cache, now: NOW, reporter }); + + return heard; + }; + + // Each of them made, where it is wanted. + const all = await kinds(); + + expect([...unwanted].filter((type) => !all.has(type))).toEqual([]); + + // And none of them where it is not — each left out on its own, so that the + // kind a run asks about and the kind it then makes cannot part unseen. + for (const type of unwanted) { + const some = await kinds((kind) => kind !== type); + + expect(some.has(type), type).toBe(false); + expect(some).toContain('site:done'); + } + }); +}); + describe('what a run says its requests were for', () => { /** An error as ky throws one for a status, shaped rather than made. */ const status = (code: number): Error => @@ -4488,9 +4685,24 @@ describe('what a run says its requests were for', () => { expect(done.requests).toBe(6); expect(done.requestsBy).toEqual({ - listings: { requests: 1, requestMs: expect.any(Number), failed: {} }, - detail: { requests: 4, requestMs: expect.any(Number), failed: { 404: 1, timeout: 1 } }, - paced: { requests: 1, requestMs: expect.any(Number), failed: {} }, + listings: { + requests: 1, + requestMs: expect.any(Number), + waitedMs: expect.any(Number), + failed: {}, + }, + detail: { + requests: 4, + requestMs: expect.any(Number), + waitedMs: expect.any(Number), + failed: { 404: 1, timeout: 1 }, + }, + paced: { + requests: 1, + requestMs: expect.any(Number), + waitedMs: expect.any(Number), + failed: {}, + }, }); // In the order each was first asked for. expect(Object.keys(done.requestsBy)).toEqual(['listings', 'detail', 'paced']); @@ -4532,7 +4744,12 @@ describe('what a run says its requests were for', () => { expect(Object.keys(by('listed.example'))).toEqual(['channels', 'request']); expect(by('stream.example')).toEqual({ - stream: { requests: 1, requestMs: expect.any(Number), failed: {} }, + stream: { + requests: 1, + requestMs: expect.any(Number), + waitedMs: expect.any(Number), + failed: {}, + }, }); }); @@ -4636,8 +4853,18 @@ describe('what a run says its requests were for', () => { await grab([streamed], { cache: new MemoryCache(), now: NOW, reporter: report.reporter }); expect(doneOf(report.events).requestsBy).toEqual({ - pages: { requests: 2, requestMs: expect.any(Number), failed: {} }, - episodes: { requests: 1, requestMs: expect.any(Number), failed: {} }, + pages: { + requests: 2, + requestMs: expect.any(Number), + waitedMs: expect.any(Number), + failed: {}, + }, + episodes: { + requests: 1, + requestMs: expect.any(Number), + waitedMs: expect.any(Number), + failed: {}, + }, }); }); @@ -4660,7 +4887,12 @@ describe('what a run says its requests were for', () => { expect(report.events.at(-1)).toMatchObject({ type: 'grab:done' }); expect((report.events.at(-1) as Extract).requestsBy).toEqual({ - request: { requests: 3, requestMs: expect.any(Number), failed: { 503: 2 } }, + request: { + requests: 3, + requestMs: expect.any(Number), + waitedMs: expect.any(Number), + failed: { 503: 2 }, + }, }); }); @@ -4793,6 +5025,7 @@ describe('what a run says its requests were for', () => { expect(doneOf(report.events).requestsBy.detail).toEqual({ requests: 1, requestMs: expect.any(Number), + waitedMs: expect.any(Number), failed: {}, }); }); @@ -4886,6 +5119,7 @@ describe('what a run says its requests were for', () => { requests: 0, ms: 0, requestMs: 0, + msBy: { source: 0, held: 0, rateLimit: 0, working: 0, queued: 0, other: 0 }, requestsBy: {}, fetched: 0, empty: 0, @@ -5165,8 +5399,18 @@ describe('what a run says its requests were for', () => { await grab([config], { cache: new MemoryCache(), now: NOW, reporter: report.reporter }); expect(doneOf(report.events).requestsBy).toEqual({ - constructor: { requests: 1, requestMs: expect.any(Number), failed: {} }, - toString: { requests: 1, requestMs: expect.any(Number), failed: { 404: 1 } }, + constructor: { + requests: 1, + requestMs: expect.any(Number), + waitedMs: expect.any(Number), + failed: {}, + }, + toString: { + requests: 1, + requestMs: expect.any(Number), + waitedMs: expect.any(Number), + failed: { 404: 1 }, + }, }); expect((report.events.at(-1) as Extract).requestsBy).toEqual( doneOf(report.events).requestsBy, @@ -5190,7 +5434,12 @@ describe('what a run says its requests were for', () => { await grab([config], { cache: new MemoryCache(), now: NOW, reporter: report.reporter }); expect(doneOf(report.events).requestsBy).toEqual({ - relácie: { requests: 3, requestMs: expect.any(Number), failed: {} }, + relácie: { + requests: 3, + requestMs: expect.any(Number), + waitedMs: expect.any(Number), + failed: {}, + }, }); }); @@ -5376,22 +5625,32 @@ describe('what a run says its requests were for', () => { expect(doneOf(report.events).requestsBy.detail).toEqual({ requests: 1, requestMs: expect.any(Number), + waitedMs: expect.any(Number), failed: { error: 1 }, }); }); it('says what a session settled before the run says as the run it joins says it', async () => { + const hears = new Set(EVENT_TYPES); const before: EpgEvent['type'][] = []; const after: EpgEvent['type'][] = []; // Paced, so its queue says when it is held as well. const session = openSession( makeConfig({ rateLimit: { requests: MANY_LABELS, perMs: 20 } }), new MemoryCache(), - { emit: (event) => before.push(event.type) }, + // Wanting nothing — what the run it joins wants is what it then makes. + { + emit: Object.assign((event: EpgEventInput) => void before.push(event.type), { + hears: new Set(), + }), + }, ); try { - session.joinRun({ requested: () => {}, emit: (event) => after.push(event.type) }); + session.joinRun({ + requested: () => {}, + emit: Object.assign((event: EpgEventInput) => void after.push(event.type), { hears }), + }); for (let id = 0; id <= MANY_LABELS; id++) { await session.send(async () => id, { label: `item-${id}` }); @@ -5424,7 +5683,12 @@ describe('what a run says its requests were for', () => { }); expect(doneOf(cancelled.events).requestsBy).toEqual({ - request: { requests: 1, requestMs: expect.any(Number), failed: {} }, + request: { + requests: 1, + requestMs: expect.any(Number), + waitedMs: expect.any(Number), + failed: {}, + }, }); const unchanged = collect(); @@ -5441,7 +5705,12 @@ describe('what a run says its requests were for', () => { ); expect(doneOf(unchanged.events).requestsBy).toEqual({ - request: { requests: 1, requestMs: expect.any(Number), failed: {} }, + request: { + requests: 1, + requestMs: expect.any(Number), + waitedMs: expect.any(Number), + failed: {}, + }, }); }); @@ -5523,6 +5792,136 @@ describe('what a run says its requests were for', () => { }); }); +describe('where a run says a site’s time went', () => { + const doneOf = (events: EpgEvent[], site = 'example.com') => + events.find((event) => event.type === 'site:done' && event.site === site) as Extract< + EpgEvent, + { type: 'site:done' } + >; + const whole = (msBy: TimeSpent): number => Object.values(msBy).reduce((sum, ms) => sum + ms, 0); + const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)); + + it('gives it to its requests and its own work, and adds up to ms', async () => { + const report = collect(); + const config = makeConfig({ + async request() { + await sleep(60); + + return {}; + }, + parseDay({ day }) { + // Thirty milliseconds of work of its own, on this thread. + const until = performance.now() + 30; + + while (performance.now() < until) { + // Working. + } + + return [programme(`${day}T06:00:00.000Z`)]; + }, + }); + + await grab([config], { cache: new MemoryCache(), now: NOW, reporter: report.reporter }); + + const { ms, msBy } = doneOf(report.events); + + expect(msBy.source).toBeGreaterThanOrEqual(50); + expect(msBy.working).toBeGreaterThanOrEqual(25); + expect(whole(msBy)).toBe(ms); + }); + + it('says its work waited for a slot another site held', async () => { + const report = collect(); + // The run's one slot, held a while by a parse waiting on no request. + const holding = makeConfig({ + site: 'holding.example', + async parseDay({ day }) { + await sleep(120); + + return [programme(`${day}T06:00:00.000Z`)]; + }, + }); + const waiting = makeConfig({ + site: 'waiting.example', + async request() { + await sleep(20); + + return {}; + }, + }); + + await grab([holding, waiting], { + cache: new MemoryCache(), + now: NOW, + localConcurrency: 1, + reporter: report.reporter, + }); + + const { ms, msBy } = doneOf(report.events, 'waiting.example'); + + expect(msBy.queued).toBeGreaterThanOrEqual(60); + expect(whole(msBy)).toBe(ms); + }); + + it('times a list settled before the grab only while its request is out', async () => { + const session = openSession(makeConfig(), new MemoryCache()); + + try { + await session.send( + async () => { + await sleep(30); + + return []; + }, + { label: 'channels' }, + ); + // Settled: a hold now, waiting for the turn, is no time of its. + session.clock.held(true); + await sleep(30); + + expect(session.clock.spent().source).toBeGreaterThanOrEqual(25); + expect(session.clock.spent().held).toBe(0); + } finally { + session.pacing.dispose(); + } + }); + + it('says how long each kind of request waited for its turn', async () => { + const report = collect(); + const config = makeConfig({ + channels: [channel('one.example'), channel('two.example')], + // The second planned request waits for the window. + rateLimit: { requests: 1, perMs: 80 }, + async parseDay({ day, paced }) { + // Three at once, one at a time: the second and third wait their turn. + await Promise.all( + ['a', 'b', 'c'].map((id) => + paced( + async () => { + await sleep(30); + + return id; + }, + { label: 'detail' }, + ), + ), + ); + + return [programme(`${day}T06:00:00.000Z`)]; + }, + }); + + await grab([config], { cache: new MemoryCache(), now: NOW, reporter: report.reporter }); + + const { request, detail } = doneOf(report.events).requestsBy; + + expect(request!.waitedMs).toBeGreaterThanOrEqual(30); + // Each channel-day's three: the second waits one, the third two — and + // the window, which every request is under. + expect(detail!.waitedMs).toBeGreaterThanOrEqual(150); + }); +}); + describe('what a site can say, and from where', () => { it('lets a fetched channel list speak, which it had no way to before', async () => { // `request` and `parseDay` always had `log`/`warn`; `channels` was handed diff --git a/test/merge.test.ts b/test/merge.test.ts index 223960b..ca187d8 100644 --- a/test/merge.test.ts +++ b/test/merge.test.ts @@ -2,7 +2,7 @@ import { existsSync } from 'node:fs'; import { mkdtemp, readFile, readdir } from 'node:fs/promises'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; -import { describe, expect, it } from 'vitest'; +import { describe, expect, it, vi } from 'vitest'; import type { CacheStore, ChannelDayKey } from '../src/cache/types.js'; import type { EpgEvent } from '../src/core/events.js'; import type { GrabberChannel, SiteConfig } from '../src/grabber/types.js'; @@ -753,7 +753,127 @@ describe('mergeProgrammeLists', () => { }); }); +describe('why two programmes did not match', () => { + const at = (start: string, title: string, stop?: string): XmltvProgramme => ({ + ...prog('X', start, title), + ...(stop === undefined ? {} : { stop: parseXmltvDate(stop) }), + }); + const why = (match: ReturnType, a: XmltvProgramme, b: XmltvProgramme) => + match.apart(a, b); + + it('says the titles, the starts, both, or the runs, by the rules it matched by', () => { + const match = resolveMatch(); + + // Nothing, of one broadcast: one answer to whether they are, and why not. + expect(why(match, at('2026-01-15T10:00:00Z', 'News'), at('2026-01-15T10:02:00Z', 'News'))).toBe( + undefined, + ); + + expect( + why(match, at('2026-01-15T10:00:00Z', 'News'), at('2026-01-15T10:02:00Z', 'Správy')), + ).toBe('title'); + expect(why(match, at('2026-01-15T10:00:00Z', 'Film'), at('2026-01-15T10:07:00Z', 'Film'))).toBe( + 'start', + ); + expect(why(match, at('2026-01-15T10:00:00Z', 'News'), at('2026-01-15T10:07:00Z', 'Film'))).toBe( + 'both', + ); + // Near enough and the same title, but further apart than the shorter + // lasts: two clips in a row, which no tolerance would join… + expect( + why( + match, + at('2026-01-15T10:00:00Z', 'Clip', '20260115100200 +0000'), + at('2026-01-15T10:03:00Z', 'Clip', '20260115100500 +0000'), + ), + ).toBe('runs'); + // …or a segment within the show it is part of. + expect( + why( + match, + at('2026-01-15T06:00:00Z', 'Morning', '20260115090000 +0000'), + at('2026-01-15T06:04:00Z', 'Morning', '20260115060600 +0000'), + ), + ).toBe('runs'); + }); + + it('counts the titles only where they are compared', () => { + // Never: the starts alone, whatever the titles say. + expect( + why( + resolveMatch({ titles: 'never' }), + at('2026-01-15T10:00:00Z', 'News'), + at('2026-01-15T10:07:00Z', 'Správy'), + ), + ).toBe('start'); + // Always: on an identical start too. + expect( + why( + resolveMatch({ titles: 'always' }), + at('2026-01-15T10:00:00Z', 'News'), + at('2026-01-15T10:00:00Z', 'Správy'), + ), + ).toBe('title'); + }); + + it('says a matcher of one’s own said no, whatever the shipped rules would say', () => { + // At the same instant, under the same title: its own reasons. + expect( + why( + resolveMatch(() => false), + at('2026-01-15T10:00:00Z', 'News'), + at('2026-01-15T10:00:00Z', 'News'), + ), + ).toBe('matcher'); + // Titles the shipped rules would call different are its business too: + // nothing of them is asked on its behalf. + expect( + why( + resolveMatch({ windowMs: 180_000, matches: () => false }), + at('2026-01-15T10:00:00Z', 'News'), + at('2026-01-15T10:02:00Z', 'Správy'), + ), + ).toBe('matcher'); + }); + + it('says a matcher with a window that two start beyond it, and never asks it', () => { + const asked = vi.fn(() => true); + + expect( + why( + resolveMatch({ windowMs: 60_000, matches: asked }), + at('2026-01-15T10:00:00Z', 'News'), + at('2026-01-15T10:02:00Z', 'News'), + ), + ).toBe('start'); + expect(asked).not.toHaveBeenCalled(); + }); +}); + describe('mergeProgrammeLists start tolerance', () => { + it('joins the nearest it matches, though a farther one comes after it', () => { + const merged = mergeProgrammeLists( + [ + [ + prog('X', '2026-01-15T10:00:00Z', 'News', 'sk'), + prog('X', '2026-01-15T10:04:00Z', 'News', 'sk'), + ], + // A minute from the first, three from the second: both within reach. + [ + prog('X', '2026-01-15T10:01:00Z', 'News', 'en', { + desc: [{ value: 'the second source' }], + }), + ], + ], + 'merge', + ); + + // Joined to the first — nearest — and the second left as it was. + expect(merged).toHaveLength(2); + expect(merged[0]!.desc).toEqual([{ value: 'the second source' }]); + expect(merged[1]!.desc).toBeUndefined(); + }); + it('merges a shifted start when the titles agree', () => { const merged = mergeProgrammeLists( [ @@ -1728,6 +1848,328 @@ describe.skipIf(!xmltvReady)('generateGuide', () => { expect(said[0]?.examples).toHaveLength(5); }); + it('says each near miss, whose each one is, and what kept them apart', async () => { + const said: Extract[] = []; + + await generate({ + sites: [siteA, siteB], + cache: createFakeCache({ + [`site-a.sk|X|${DAY}`]: [ + prog('X', '2026-01-15T10:00:00Z', 'Správy', 'sk'), + prog('X', '2026-01-15T12:00:00Z', 'Film', 'sk'), + prog('X', '2026-01-15T14:00:00Z', 'Šport', 'sk'), + ], + [`site-b.com|X|${DAY}`]: [ + // Near enough, under another name. + prog('X', '2026-01-15T10:02:00Z', 'News', 'en'), + // The same name, too far apart. + prog('X', '2026-01-15T12:07:00Z', 'Film', 'en'), + // Neither. + prog('X', '2026-01-15T14:07:00Z', 'Football', 'en'), + ], + }), + days: 1, + now: NOW, + reporter: (event) => { + if (event.type === 'merge:unmatched') { + said.push(event); + } + }, + }); + + expect(said.map(({ type: _type, level: _level, phase: _phase, ...rest }) => rest)).toEqual([ + { + channelId: 'X', + shiftMs: 120_000, + why: 'title', + programme: { site: 'site-b.com', title: 'News', start: '20260115100200 +0000' }, + nearby: { site: 'site-a.sk', title: 'Správy', start: '20260115100000 +0000' }, + }, + { + channelId: 'X', + shiftMs: 420_000, + why: 'start', + programme: { site: 'site-b.com', title: 'Film', start: '20260115120700 +0000' }, + nearby: { site: 'site-a.sk', title: 'Film', start: '20260115120000 +0000' }, + }, + { + channelId: 'X', + shiftMs: 420_000, + why: 'both', + programme: { site: 'site-b.com', title: 'Football', start: '20260115140700 +0000' }, + nearby: { site: 'site-a.sk', title: 'Šport', start: '20260115140000 +0000' }, + }, + ]); + // At `debug`: a guide on two grids says thousands of these. + expect(said[0]).toMatchObject({ level: 'debug', phase: 'merge' }); + }); + + it('says how much later a near miss starts than the one it nearly met — or earlier', async () => { + const said: Extract[] = []; + + await generate({ + sites: [siteA, siteB], + cache: createFakeCache({ + [`site-a.sk|X|${DAY}`]: [prog('X', '2026-01-15T10:00:00Z', 'Správy', 'sk')], + [`site-b.com|X|${DAY}`]: [prog('X', '2026-01-15T09:53:00Z', 'Správy', 'en')], + }), + days: 1, + now: NOW, + reporter: (event) => { + if (event.type === 'merge:unmatched') { + said.push(event); + } + }, + }); + + expect(said).toEqual([expect.objectContaining({ shiftMs: -420_000, why: 'start' })]); + }); + + it('takes for no near miss a programme a later one of the same list then joined', async () => { + const said: Extract[] = []; + const summary: Extract[] = []; + + await generate({ + sites: [siteA, siteB], + cache: createFakeCache({ + [`site-a.sk|X|${DAY}`]: [ + prog('X', '2026-01-15T19:45:00Z', 'Reklama', 'sk'), + prog('X', '2026-01-15T20:00:00Z', 'Film', 'sk'), + prog('X', '2026-01-15T23:00:00Z', 'Správy', 'sk'), + ], + // B's filler is nearest A's film, which B's own film then joins: the + // filler's near miss is A's advert, the nearest still left. + [`site-b.com|X|${DAY}`]: [ + prog('X', '2026-01-15T19:53:00Z', 'Filler', 'en'), + prog('X', '2026-01-15T20:00:00Z', 'Film', 'en'), + // And one whose only candidate was taken: none at all. + prog('X', '2026-01-15T22:53:00Z', 'Promo', 'en'), + prog('X', '2026-01-15T23:00:00Z', 'Správy', 'en'), + ], + }), + days: 1, + now: NOW, + reporter: (event) => { + if (event.type === 'merge:unmatched') { + said.push(event); + } else if (event.type === 'merge:summary') { + summary.push(event); + } + }, + }); + + expect(said.map((event) => [event.programme.title, event.nearby.title])).toEqual([ + ['Filler', 'Reklama'], + ]); + expect(summary[0]).toMatchObject({ combined: 2, nearMiss: 1 }); + }); + + it('says what the sources made of each other once, though a channel is derived from them', async () => { + const said: EpgEvent[] = []; + + await generate({ + sites: [siteA, siteB], + cache: createFakeCache({ + [`site-a.sk|X|${DAY}`]: [ + prog('X', '2026-01-15T10:00:00Z', 'Správy', 'sk'), + prog('X', '2026-01-15T12:00:00Z', 'Film', 'sk'), + ], + [`site-b.com|X|${DAY}`]: [ + prog('X', '2026-01-15T10:00:00Z', 'Správy', 'en'), + prog('X', '2026-01-15T12:07:00Z', 'Film', 'en'), + ], + }), + days: 1, + now: NOW, + derived: [{ xmltvId: 'X.plus1', from: 'X', offset: 60 }], + reporter: (event) => said.push(event), + }); + + const unmatched = said.filter((event) => event.type === 'merge:unmatched'); + + expect(unmatched).toEqual([expect.objectContaining({ channelId: 'X' })]); + expect(said.find((event) => event.type === 'merge:summary')).toMatchObject({ + combined: 1, + nearMiss: 1, + }); + }); + + it('makes no near miss for a reporter that does not want one, and counts it all the same', async () => { + const said: EpgEvent[] = []; + + await generate({ + sites: [siteA, siteB], + cache: createFakeCache({ + [`site-a.sk|X|${DAY}`]: [prog('X', '2026-01-15T10:00:00Z', 'Správy', 'sk')], + [`site-b.com|X|${DAY}`]: [prog('X', '2026-01-15T10:07:00Z', 'News', 'en')], + }), + days: 1, + now: NOW, + // Told everything it is handed, and wanting all but those. + reporter: Object.assign((event: EpgEvent) => said.push(event), { + wants: (type: EpgEvent['type']) => type !== 'merge:unmatched', + }), + }); + + expect(said.some((event) => event.type === 'merge:unmatched')).toBe(false); + expect(said.find((event) => event.type === 'merge:summary')).toMatchObject({ nearMiss: 1 }); + }); + + it('takes no source’s own programme of the day before for one it nearly was', async () => { + const NEXT_DAY = '2026-01-16'; + const said: Extract[] = []; + + await generate({ + sites: [siteA, siteB], + cache: createFakeCache({ + [`site-a.sk|X|${DAY}`]: [prog('X', '2026-01-15T12:00:00Z', 'Obed', 'sk')], + [`site-a.sk|X|${NEXT_DAY}`]: [prog('X', '2026-01-16T12:00:00Z', 'Obed', 'sk')], + // B's late programme, held into the next day, and B's next one after it. + [`site-b.com|X|${DAY}`]: [prog('X', '2026-01-15T23:50:00Z', 'Late', 'en')], + [`site-b.com|X|${NEXT_DAY}`]: [prog('X', '2026-01-16T00:05:00Z', 'Night', 'en')], + }), + days: 2, + now: NOW, + reporter: (event) => { + if (event.type === 'merge:unmatched') { + said.push(event); + } + }, + }); + + // Two programmes of B's in a row, not two sources that nearly agreed. + expect(said).toEqual([]); + }); + + it('takes no source’s own programme of the day before for one it nearly was, though another joined it', async () => { + const NEXT_DAY = '2026-01-16'; + const said: Extract[] = []; + + await generate({ + sites: [siteA, siteB], + cache: createFakeCache({ + // A's and B's late programme, one broadcast, held into the next day. + [`site-a.sk|X|${DAY}`]: [prog('X', '2026-01-15T23:50:00Z', 'Late', 'sk')], + [`site-a.sk|X|${NEXT_DAY}`]: [prog('X', '2026-01-16T12:00:00Z', 'Obed', 'sk')], + [`site-b.com|X|${DAY}`]: [prog('X', '2026-01-15T23:50:00Z', 'Late', 'en')], + [`site-b.com|X|${NEXT_DAY}`]: [prog('X', '2026-01-16T00:05:00Z', 'Night', 'en')], + }), + days: 2, + now: NOW, + reporter: (event) => { + if (event.type === 'merge:unmatched') { + said.push(event); + } + }, + }); + + // B's programme as much as A's: two of B's in a row. + expect(said).toEqual([]); + }); + + it('counts no source’s own programme of the day before, said again, as two that agreed', async () => { + const NEXT_DAY = '2026-01-16'; + const said: Extract[] = []; + + await generate({ + sites: [siteA, siteB], + cache: createFakeCache({ + [`site-a.sk|X|${DAY}`]: [prog('X', '2026-01-15T12:00:00Z', 'Obed', 'sk')], + [`site-a.sk|X|${NEXT_DAY}`]: [prog('X', '2026-01-16T12:00:00Z', 'Obed', 'sk')], + // B's late programme, and B saying it again with the next day's. + [`site-b.com|X|${DAY}`]: [prog('X', '2026-01-15T23:50:00Z', 'Late', 'en')], + [`site-b.com|X|${NEXT_DAY}`]: [ + prog('X', '2026-01-15T23:50:00Z', 'Late', 'en'), + prog('X', '2026-01-16T00:05:00Z', 'Night', 'en'), + ], + }), + days: 2, + now: NOW, + reporter: (event) => { + if (event.type === 'merge:summary') { + said.push(event); + } + }, + }); + + expect(said[0]).toMatchObject({ combined: 0 }); + }); + + it('watches a day’s first source too, against another’s programme held over from the day before', async () => { + const NEXT_DAY = '2026-01-16'; + const unmatched: Extract[] = []; + const summary: Extract[] = []; + + await generate({ + sites: [siteA, siteB], + cache: createFakeCache({ + // Only B answered for the first day, so its late programmes are held. + [`site-b.com|X|${DAY}`]: [ + prog('X', '2026-01-15T23:30:00Z', 'Late', 'en'), + prog('X', '2026-01-15T23:50:00Z', 'Later', 'en'), + ], + // A, first of the next day, meets one of them and nearly meets the other. + [`site-a.sk|X|${NEXT_DAY}`]: [ + prog('X', '2026-01-15T23:30:00Z', 'Late', 'sk'), + prog('X', '2026-01-15T23:57:00Z', 'Later', 'sk'), + ], + }), + days: 2, + now: NOW, + reporter: (event) => { + if (event.type === 'merge:unmatched') { + unmatched.push(event); + } else if (event.type === 'merge:summary') { + summary.push(event); + } + }, + }); + + expect(unmatched).toEqual([ + expect.objectContaining({ + why: 'start', + programme: expect.objectContaining({ site: 'site-a.sk', title: 'Later' }), + nearby: expect.objectContaining({ site: 'site-b.com', title: 'Later' }), + }), + ]); + expect(summary[0]).toMatchObject({ combined: 1, nearMiss: 1 }); + }); + + it('says which site a near miss is with, though two had merged it, or a third joined it', async () => { + const siteC = makeSite('site-c.cz', [{ xmltvId: 'X', siteId: 'c-x' }]); + const said: Extract[] = []; + + await generate({ + sites: [siteA, siteB, siteC], + cache: createFakeCache({ + // A's and B's 10:00 are one programme, made anew; B's 12:00 is B's alone. + [`site-a.sk|X|${DAY}`]: [prog('X', '2026-01-15T10:00:00Z', 'Správy', 'sk')], + [`site-b.com|X|${DAY}`]: [ + prog('X', '2026-01-15T10:00:00Z', 'Správy', 'en'), + prog('X', '2026-01-15T12:00:00Z', 'Film', 'en'), + ], + // And C nearly meets each. + [`site-c.cz|X|${DAY}`]: [ + prog('X', '2026-01-15T10:07:00Z', 'Správy', 'cs'), + prog('X', '2026-01-15T12:07:00Z', 'Film', 'cs'), + ], + }), + days: 1, + now: NOW, + reporter: (event) => { + if (event.type === 'merge:unmatched') { + said.push(event); + } + }, + }); + + expect(said.map((event) => [event.programme.site, event.nearby.site])).toEqual([ + // The merged one is its base's: the higher site's. + ['site-c.cz', 'site-a.sk'], + ['site-c.cz', 'site-b.com'], + ]); + }); + it('skips missing cache entries silently', async () => { const siteY = makeSite('site-a.sk', [ { xmltvId: 'X', siteId: 'a-x' }, diff --git a/test/pacing.test.ts b/test/pacing.test.ts index 2ba3182..e23bd95 100644 --- a/test/pacing.test.ts +++ b/test/pacing.test.ts @@ -3,9 +3,11 @@ import { createServer, type Server } from 'node:http'; import type { AddressInfo } from 'node:net'; import type { KyInstance } from 'ky'; import { describe, expect, it, vi } from 'vitest'; +import { CacheManager, MemoryCacheDriver } from '../src/cache/main.js'; +import { openSession } from '../src/grabber/session.js'; import { retryAfterMs, sitePacing } from '../src/grabber/pacing.js'; import type { AnySiteConfig } from '../src/grabber/types.js'; -import { emitter } from '../src/core/events.js'; +import { emitter, type Emit } from '../src/core/events.js'; import { collect } from './reporting.js'; /** A site config with just enough on it for the pacing to be built. */ @@ -240,12 +242,190 @@ describe('sitePacing', () => { // Ten minutes is what it asked for; a grab is not sitting still for that. expect(report.messages).toContain('[example.tv] HTTP 429: holding requests for 40ms'); + // Said beside what it was given, so a reader knows it was cut short. + expect(report.of('pacing:held')).toEqual([ + expect.objectContaining({ status: 429, ms: 40, retryAfterMs: 600_000 }), + ]); dispose(); } finally { server.close(); } }); + it('says a hold made longer, rather than one begun', async () => { + // The second asks for longer than is left of the first: one hold, longer. + const server = await scripted([ + { status: 429 }, + { status: 429, headers: { 'retry-after': '5' } }, + ]); + + try { + const report = collect(); + const { http, dispose } = sitePacing(site({ backoff: { fallbackMs: 2_000 } }), { + emit: emitter({ reporter: report.reporter }), + }); + + await hit(http, server.url); + await hit(http, server.url); + + const [begun, longer] = report.of('pacing:held'); + + expect(begun).not.toHaveProperty('lengthened'); + expect(longer).toMatchObject({ ms: 5_000, retryAfterMs: 5_000, lengthened: true }); + expect(report.messages).toContain( + '[example.tv] HTTP 429: holding requests longer, for 5000ms from now', + ); + dispose(); + } finally { + server.close(); + } + }); + + it('refuses an emit that says nothing of what it wants, before a grab could throw on it', () => { + // As a JavaScript caller would hand it: a function, and nothing more. + const plain = ((): void => {}) as unknown as Emit; + + expect(() => sitePacing(site(), { emit: plain })).toThrow( + /emit with no hears: make it with emitter/, + ); + }); + + it('says the source asked for nothing, where it sent no Retry-After', async () => { + const server = await scripted([{ status: 503 }]); + + try { + const report = collect(); + const { http, dispose } = sitePacing(site({ backoff: { fallbackMs: 30 } }), { + emit: emitter({ reporter: report.reporter }), + }); + + await hit(http, server.url); + + const [held] = report.of('pacing:held'); + + expect(held).toMatchObject({ status: 503, ms: 30 }); + expect(held).not.toHaveProperty('retryAfterMs'); + dispose(); + } finally { + server.close(); + } + }); + + it('tells the clock how long a request waited behind a hold, and no longer', async () => { + const server = await scripted([{ status: 429 }, { status: 200 }]); + const session = openSession( + site({ backoff: { fallbackMs: 40 } }), + new CacheManager({ driver: new MemoryCacheDriver() }), + ); + + try { + session.clock.turn(true); + await session.send(() => hit(session.pacing.http, server.url), { label: 'request' }); + // Held: this one waits behind it for the forty milliseconds. + await session.send(() => hit(session.pacing.http, server.url), { label: 'request' }); + const { held } = session.clock.spent(); + + // Nothing waiting, so nothing more held up — though the hold says nothing + // of that, and a timer that runs late on a busy machine is no failure. + await new Promise((resolve) => setTimeout(resolve, 30)); + + expect(held).toBeGreaterThanOrEqual(30); + expect(session.clock.spent().held).toBe(held); + } finally { + session.pacing.dispose(); + server.close(); + } + }); + + it('tells the clock the run was cancelled, so what it dropped waits no more', async () => { + const run = new AbortController(); + const session = openSession(site(), new CacheManager({ driver: new MemoryCacheDriver() }), { + signal: run.signal, + }); + + try { + session.clock.turn(true); + // Local work waiting for a slot, which the cancel drops. + session.clock.queued(); + run.abort(); + await new Promise((resolve) => setTimeout(resolve, 30)); + + expect(session.clock.spent().queued).toBeLessThan(5); + } finally { + session.pacing.dispose(); + } + }); + + it('tells the clock a request ky retries after a 429 was held, not out to the source', async () => { + const server = await scripted([{ status: 429 }, { status: 200 }]); + const session = openSession( + site({ backoff: { fallbackMs: 60 } }), + new CacheManager({ driver: new MemoryCacheDriver() }), + ); + + try { + session.clock.turn(true); + // One request, retried by ky itself: what a site asking one at a time does. + await session.send( + async () => { + const response = await session.pacing.http.get(server.url, { + retry: { limit: 1, delay: () => 60 }, + }); + + await response.body?.cancel(); + }, + { label: 'request' }, + ); + + expect(server.hits).toBe(2); + expect(session.clock.spent().held).toBeGreaterThanOrEqual(50); + } finally { + session.pacing.dispose(); + server.close(); + } + }); + + it('tells the clock of a run cancelled before the site was set up', async () => { + const run = new AbortController(); + + run.abort(); + + const session = openSession(site(), new CacheManager({ driver: new MemoryCacheDriver() }), { + signal: run.signal, + }); + + try { + session.clock.turn(true); + // Dropped the moment it is asked: nothing waits. + session.clock.queued(); + await new Promise((resolve) => setTimeout(resolve, 20)); + + expect(session.clock.spent().queued).toBe(0); + } finally { + session.pacing.dispose(); + } + }); + + it('tells the clock how long its own rate limit held a request back', async () => { + const session = openSession( + site({ rateLimit: { requests: 1, perMs: 60 } }), + new CacheManager({ driver: new MemoryCacheDriver() }), + ); + + try { + session.clock.turn(true); + // Two at once: the second waits for the window. + await Promise.all([ + session.send(async () => {}, { label: 'request' }), + session.send(async () => {}, { label: 'request' }), + ]); + + expect(session.clock.spent().rateLimit).toBeGreaterThanOrEqual(40); + } finally { + session.pacing.dispose(); + } + }); + it('halves concurrency on a slow-down and gives it back on clean responses', async () => { const server = await scripted([ { status: 429 }, diff --git a/test/reporters.test.ts b/test/reporters.test.ts index 6a6fcf4..70c35d0 100644 --- a/test/reporters.test.ts +++ b/test/reporters.test.ts @@ -1,6 +1,6 @@ import { EventEmitter } from 'node:events'; import { Writable } from 'node:stream'; -import { describe, expect, it } from 'vitest'; +import { describe, expect, it, vi } from 'vitest'; import { atLevel, EVENT_KINDS, @@ -167,6 +167,64 @@ describe('the event union', () => { describe('render', () => { it.each<[EpgEventInput, string]>([ + [ + { + type: 'merge:unmatched', + channelId: 'ct1.cz', + shiftMs: 300_000, + why: 'title', + programme: { site: 'b.tv', title: 'DVTV: Jakub Kříž', start: '20261024200500 +0200' }, + nearby: { site: 'a.tv', title: 'DVTV', start: '20261024200000 +0200' }, + }, + 'ct1.cz: 300s apart, titles differ — "DVTV: Jakub Kříž" (b.tv, 20:05) / "DVTV" (a.tv, 20:00)', + ], + // A start said to the hour, or to the day: as much of the clock as there is. + [ + { + type: 'merge:unmatched', + channelId: 'ct1.cz', + shiftMs: 180_000, + why: 'runs', + programme: { site: 'b.tv', title: 'Clip', start: '2026102420 +0200' }, + nearby: { site: 'a.tv', title: 'Clip', start: '20261024 +0200' }, + }, + 'ct1.cz: 180s apart, they start further apart than the shorter lasts — "Clip" (b.tv, 20:00) / "Clip" (a.tv, 2026-10-24)', + ], + [ + { + type: 'merge:unmatched', + channelId: 'ct1.cz', + shiftMs: 0, + why: 'matcher', + programme: { site: 'b.tv', title: 'News', start: '20261024200000 +0200' }, + nearby: { site: 'a.tv', title: 'News', start: '20261024200000 +0200' }, + }, + `ct1.cz: 0ms apart, the config's match said no — "News" (b.tv, 20:00) / "News" (a.tv, 20:00)`, + ], + [ + // Earlier rather than later: apart by as much either way. + { + type: 'merge:unmatched', + channelId: 'ct1.cz', + shiftMs: -420_000, + why: 'start', + programme: { site: 'b.tv', title: 'News', start: '20261024195300 +0200' }, + nearby: { site: 'a.tv', title: 'News', start: '20261024200000 +0200' }, + }, + 'ct1.cz: 420s apart, starts too far apart — "News" (b.tv, 19:53) / "News" (a.tv, 20:00)', + ], + [ + // A start said to the month, and to the year alone. + { + type: 'merge:unmatched', + channelId: 'ct1.cz', + shiftMs: 0, + why: 'matcher', + programme: { site: 'b.tv', title: 'Film', start: '202610 +0000' }, + nearby: { site: 'a.tv', title: 'Film', start: '2026 +0000' }, + }, + `ct1.cz: 0ms apart, the config's match said no — "Film" (b.tv, 2026-10) / "Film" (a.tv, 2026)`, + ], [ { type: 'entry:cached', site: 'a.tv', channelId: 'one', day: '2026-09-01' }, '[a.tv] one 2026-09-01: fresh in cache, skipping', @@ -194,6 +252,10 @@ describe('render', () => { { type: 'pacing:held', site: 'a.tv', status: 429, ms: 60 }, '[a.tv] HTTP 429: holding requests for 60ms', ], + [ + { type: 'pacing:held', site: 'a.tv', status: 429, ms: 90, lengthened: true }, + '[a.tv] HTTP 429: holding requests longer, for 90ms from now', + ], [ { type: 'request:started', site: 'a.tv', channels: ['one'], days: ['2026-09-01'] }, '[a.tv] one 2026-09-01: asking', @@ -209,6 +271,7 @@ describe('render', () => { requests: 96, ms: 42_000, requestMs: 28_500, + msBy: { source: 0, held: 0, rateLimit: 0, working: 0, queued: 0, other: 42_000 }, requestsBy: {}, fetched: 412, empty: 0, @@ -226,9 +289,15 @@ describe('render', () => { requests: 5291, ms: 697_000, requestMs: 585_000, + msBy: { source: 0, held: 0, rateLimit: 0, working: 0, queued: 0, other: 697_000 }, requestsBy: { - request: { requests: 49, requestMs: 12_000, failed: {} }, - detail: { requests: 5242, requestMs: 573_000, failed: { 404: 3242, timeout: 3 } }, + request: { requests: 49, requestMs: 12_000, waitedMs: 0, failed: {} }, + detail: { + requests: 5242, + requestMs: 573_000, + waitedMs: 0, + failed: { 404: 3242, timeout: 3 }, + }, }, fetched: 144, empty: 0, @@ -249,7 +318,8 @@ describe('render', () => { requests: 3, ms: 3000, requestMs: 2000, - requestsBy: { listings: { requests: 3, requestMs: 2000, failed: { 500: 1 } } }, + msBy: { source: 0, held: 0, rateLimit: 0, working: 0, queued: 0, other: 3000 }, + requestsBy: { listings: { requests: 3, requestMs: 2000, waitedMs: 0, failed: { 500: 1 } } }, fetched: 2, empty: 0, fromCache: 0, @@ -268,6 +338,7 @@ describe('render', () => { requests: 3, ms: 3000, requestMs: 2000, + msBy: { source: 0, held: 0, rateLimit: 0, working: 0, queued: 0, other: 3000 }, requestsBy: { request: { requests: 1, requestMs: 500 }, detail: { requests: 2, requestMs: 1500 }, @@ -305,6 +376,7 @@ describe('render', () => { requests: 1, ms: 2543, requestMs: 2100, + msBy: { source: 0, held: 0, rateLimit: 0, working: 0, queued: 0, other: 2543 }, requestsBy: {}, fetched: 0, empty: 0, @@ -721,6 +793,7 @@ describe('progressReporter', () => { ms: 0, requestMs: 0, requestsBy: {}, + msBy: { source: 0, held: 0, rateLimit: 0, working: 0, queued: 0, other: 0 }, fetched: 0, empty: 0, fromCache: 0, @@ -745,9 +818,17 @@ describe('progressReporter', () => { run(progressReporter({ stream: out }), [ started, - // Neither of these moves a number the line shows. + // None of these moves a number the line shows. { type: 'request:started', site: 'a.tv', channels: ['one'], days: ['2026-09-01'] }, { type: 'stream:ignored', site: 'a.tv', count: 2 }, + { + type: 'merge:unmatched', + channelId: 'one', + shiftMs: 420_000, + why: 'start', + programme: { site: 'b.tv', title: 'Show', start: '20260901060700 +0000' }, + nearby: { site: 'a.tv', title: 'Show', start: '20260901060000 +0000' }, + }, ]); expect(out.raw).toBe('a.tv · 0/6 channel-days'); @@ -925,6 +1006,31 @@ describe('emitterReporter and combineReporters', () => { expect(seen).toEqual(['started http://127.0.0.1:8080/guide.xml', 'reloaded']); }); + it('wants what its emitter has listeners for, as it has them, and everything of one that cannot say', () => { + const events = new EventEmitter(); + const report = emitterReporter(events); + + events.on('serve:started', () => {}); + + expect(report.wants!('serve:started')).toBe(true); + expect(report.wants!('merge:unmatched')).toBe(false); + + // One added part way through a run is heard from then on. + events.on('merge:unmatched', () => {}); + expect(report.wants!('merge:unmatched')).toBe(true); + + expect(emitterReporter({ emit: () => true }).wants).toBeUndefined(); + + // One that hears more than its listeners do says what it does. + const forwarding = emitterReporter(events, { wants: (type) => type !== 'serve:response' }); + + expect(forwarding.wants!('merge:unmatched')).toBe(true); + expect(forwarding.wants!('serve:response')).toBe(false); + expect(emitterReporter({ emit: () => true }, { wants: () => false }).wants!('site:done')).toBe( + false, + ); + }); + it('gives every event to each reporter, and skips the ones that are not there', () => { const said: string[] = []; const report = combineReporters( @@ -985,6 +1091,216 @@ describe('emitterReporter and combineReporters', () => { }); }); +describe('what a reporter wants', () => { + /** A terminal, as the live line asks for. */ + class Terminal extends Sink { + readonly isTTY = true; + readonly columns = 200; + } + + it('is what a shipped one shows at its level, and its failures whatever that is', () => { + const text = textReporter({ stream: new Sink(), level: 'info' }); + const quiet = textReporter({ stream: new Sink(), level: 'error' }); + const json = jsonReporter({ stream: new Sink(), level: 'info' }); + const logger = loggerReporter({ logger: { info() {}, warn() {}, error() {} }, level: 'error' }); + + expect(text.wants!('merge:summary')).toBe(true); + expect(text.wants!('merge:unmatched')).toBe(false); + // What ends a run, whatever it shows: the failures held for it are written then. + expect(quiet.wants!('entry:failed')).toBe(true); + expect(quiet.wants!('grab:done')).toBe(true); + expect(quiet.wants!('run:cancelled')).toBe(true); + expect(quiet.wants!('site:done')).toBe(false); + expect(json.wants!('merge:unmatched')).toBe(false); + expect(jsonReporter({ stream: new Sink() }).wants!('merge:unmatched')).toBe(true); + expect(logger.wants!('request:failed')).toBe(true); + expect(logger.wants!('merge:summary')).toBe(false); + }); + + it('is, of the live line, what it counts beside what its lines want', () => { + const live = progressReporter({ stream: new Terminal(), level: 'info' }); + + // Counted though it is never shown at `info`. + expect(live.wants!('entry:fetched')).toBe(true); + // Drawn again as it was, which is nothing. + expect(live.wants!('request:started')).toBe(false); + expect(live.wants!('merge:unmatched')).toBe(false); + expect(live.wants!('merge:done')).toBe(true); + // Off a terminal, the lines alone. + expect(progressReporter({ stream: new Sink(), level: 'info' }).wants!('entry:fetched')).toBe( + false, + ); + }); + + it('filters a kind this version does not know by its own level, rather than throwing', () => { + const out = new Sink(); + const log = (): void => {}; + // As a later version's log, read back, would hand it one. + const unknown = (level: EventLevel): EpgEvent => + ({ type: 'future:thing', site: 'a.tv', level, phase: 'grab' }) as unknown as EpgEvent; + const shipped = [ + textReporter({ stream: new Sink(), level: 'info' }), + jsonReporter({ stream: out, level: 'info' }), + loggerReporter({ logger: { info: log, warn: log, error: log }, level: 'info' }), + progressReporter({ stream: new Terminal(), level: 'info' }), + ]; + + for (const reporter of shipped) { + expect(() => reporter(unknown('debug'))).not.toThrow(); + expect(() => reporter(unknown('warn'))).not.toThrow(); + } + + expect(out.lines.map((line) => JSON.parse(line).level)).toEqual(['warn']); + }); + + it('does nothing with a kind it says it does not want, though it is given it', () => { + const sink = new Sink(); + const terminal = new Terminal(); + const told: string[] = []; + const log = (line: string): void => { + told.push(line); + }; + const shipped = [ + textReporter({ stream: sink, level: 'error' }), + jsonReporter({ stream: sink, level: 'warn' }), + loggerReporter({ logger: { info: log, warn: log, error: log, debug: log }, level: 'error' }), + progressReporter({ stream: terminal, level: 'info' }), + ]; + const writes = [vi.spyOn(sink, 'write'), vi.spyOn(terminal, 'write')]; + let unwanted = 0; + + for (const reporter of shipped) { + for (const [type, kind] of Object.entries(EVENT_KINDS) as Array< + [EpgEvent['type'], EventKind] + >) { + if (!reporter.wants!(type)) { + unwanted++; + // Nothing of it but its kind: nothing is to read any more of it. + reporter({ type, ...kind } as EpgEvent); + } + } + } + + expect(unwanted).toBeGreaterThan(0); + + for (const write of writes) { + expect(write).not.toHaveBeenCalled(); + } + + expect(told).toEqual([]); + }); + + it('is wanted by any of several, every kind by one that did not say, and none by none', () => { + const none: Reporter = Object.assign(() => {}, { wants: () => false }); + const some: Reporter = Object.assign(() => {}, { + wants: (type: EpgEvent['type']) => type === 'merge:unmatched', + }); + + expect(combineReporters(none, some).wants!('merge:unmatched')).toBe(true); + expect(combineReporters(none, some).wants!('merge:summary')).toBe(false); + expect(combineReporters(none, () => {}).wants!('merge:summary')).toBe(true); + expect(combineReporters().wants!('merge:summary')).toBe(false); + }); + + it('is asked of the reporter as its own, and what it throws is kept as a throw is', () => { + const kinds = new Set(['merge:summary']); + // A `wants` written as a method, reading `this`. + const own = Object.assign(() => {}, { + kinds, + wants(this: { kinds: Set }, type: EpgEvent['type']) { + return this.kinds.has(type); + }, + }) as Reporter; + const broken = new TypeError('cannot read properties of undefined'); + const throwing: Reporter = Object.assign(() => {}, { + wants: (): boolean => { + throw broken; + }, + }); + const keeping = keepingThrows({ reporter: throwing }); + + expect(stamped(own).hears.has('merge:summary')).toBe(true); + expect( + stamped(keepingThrows({ reporter: own }).options.reporter!).hears.has('merge:summary'), + ).toBe(true); + // Given everything, and the throw said once the run is done. + expect(stamped(keeping.options.reporter!).hears.has('merge:unmatched')).toBe(true); + expect(() => keeping.rethrow()).toThrow(ReporterError); + }); + + it('asks several only until one wants it, and throws one that throws through', () => { + const throwing: Reporter = Object.assign(() => {}, { + wants: (): boolean => { + throw new TypeError('no'); + }, + }); + const asked: string[] = []; + const wanting: Reporter = Object.assign(() => {}, { + wants: () => { + asked.push('wanting'); + + return true; + }, + }); + const not: Reporter = Object.assign(() => {}, { + wants: () => { + asked.push('not'); + + return false; + }, + }); + + expect(() => combineReporters(not, throwing, wanting).wants!('merge:unmatched')).toThrow( + TypeError, + ); + expect(asked).toEqual(['not']); + // What one that wants it says decides it: those after it are given it + // whatever they would say. + expect(combineReporters(wanting, throwing, not).wants!('merge:unmatched')).toBe(true); + expect(asked).toEqual(['not', 'wanting']); + + // Under what keeps a run's throws: given everything, and said once it is done. + const keeping = keepingThrows({ reporter: combineReporters(not, throwing) }); + + expect(keeping.options.reporter!.wants!('merge:unmatched')).toBe(true); + expect(() => keeping.rethrow()).toThrow(ReporterError); + }); + + it('is passed on by what stamps a reporter, or keeps what it throws', () => { + const some: Reporter = Object.assign(() => {}, { + wants: (type: EpgEvent['type']) => type === 'merge:unmatched', + }); + const kept = keepingThrows({ reporter: some }).options.reporter!; + + expect(stamped(some).hears.has('merge:unmatched')).toBe(true); + expect(stamped(some).hears.has('merge:summary')).toBe(false); + expect(kept.wants!('merge:summary')).toBe(false); + // One that did not say wants everything; a run nobody listens to, nothing. + expect(stamped(() => {}).hears.has('merge:summary')).toBe(true); + expect(silent.hears.has('merge:summary')).toBe(false); + }); + + it('is asked of each kind once, the first time the run asks, and held to that', () => { + const events = new EventEmitter(); + const report = emitterReporter(events); + const asked = vi.spyOn(report, 'wants'); + const emit = stamped(report); + + // Nothing asked yet: a listener added as the run is started is heard. + expect(asked).not.toHaveBeenCalled(); + events.on('entry:fetched', () => {}); + expect(emit.hears.has('entry:fetched')).toBe(true); + expect(asked).toHaveBeenCalledTimes(Object.keys(EVENT_KINDS).length); + asked.mockClear(); + + // A listener from part way through: heard from the next run on. + events.on('merge:unmatched', () => {}); + expect(emit.hears.has('merge:unmatched')).toBe(false); + expect(asked).not.toHaveBeenCalled(); + expect(stamped(report).hears.has('merge:unmatched')).toBe(true); + }); +}); + describe('keepingThrows and ReporterError', () => { const at = (type: 'site:done' | 'site:failed' | 'serve:reloaded') => event( @@ -999,6 +1315,7 @@ describe('keepingThrows and ReporterError', () => { ms: 0, requestMs: 0, requestsBy: {}, + msBy: { source: 0, held: 0, rateLimit: 0, working: 0, queued: 0, other: 0 }, fetched: 0, empty: 0, fromCache: 0, diff --git a/test/schedules-direct/site.test.ts b/test/schedules-direct/site.test.ts index 3d343b3..930858e 100644 --- a/test/schedules-direct/site.test.ts +++ b/test/schedules-direct/site.test.ts @@ -144,7 +144,12 @@ describe('defineSchedulesDirectSite', () => { const { requestsBy } = done; expect(Object.keys(requestsBy)).toEqual(['channels', 'md5', 'schedules', 'programs']); - expect(requestsBy.channels).toEqual({ requests: 1, requestMs: expect.any(Number), failed: {} }); + expect(requestsBy.channels).toEqual({ + requests: 1, + requestMs: expect.any(Number), + waitedMs: expect.any(Number), + failed: {}, + }); for (const [label, path] of [ ['md5', 'schedules/md5'], @@ -154,6 +159,7 @@ describe('defineSchedulesDirectSite', () => { expect(requestsBy[label]).toEqual({ requests: source.countOf(path), requestMs: expect.any(Number), + waitedMs: expect.any(Number), failed: {}, }); } diff --git a/test/shared-counts.test.ts b/test/shared-counts.test.ts index 27f8409..f256139 100644 --- a/test/shared-counts.test.ts +++ b/test/shared-counts.test.ts @@ -24,15 +24,20 @@ describe('shared counts', () => { it('reads back what was added, by label, in the order each was first counted', () => { const { writer, reader } = shared(); - writer.add('listings', { requests: 1, requestMs: 0, failed: {} }); - writer.add('detail', { requests: 1, requestMs: 0, failed: {} }); - writer.add('listings', { requests: 0, requestMs: 120, failed: {} }); - writer.add('detail', { requests: 0, requestMs: 30, failed: { 404: 1 } }); - writer.add('detail', { requests: 1, requestMs: 25, failed: { 404: 1, timeout: 1 } }); + writer.add('listings', { requests: 1, requestMs: 0, waitedMs: 40, failed: {} }); + writer.add('detail', { requests: 1, requestMs: 0, waitedMs: 3, failed: {} }); + writer.add('listings', { requests: 0, requestMs: 120, waitedMs: 0, failed: {} }); + writer.add('detail', { requests: 0, requestMs: 30, waitedMs: 0, failed: { 404: 1 } }); + writer.add('detail', { + requests: 1, + requestMs: 25, + waitedMs: 9, + failed: { 404: 1, timeout: 1 }, + }); expect([...reader.read()]).toEqual([ - ['listings', { requests: 1, requestMs: 120, failed: {} }], - ['detail', { requests: 2, requestMs: 55, failed: { 404: 2, timeout: 1 } }], + ['listings', { requests: 1, requestMs: 120, waitedMs: 40, failed: {} }], + ['detail', { requests: 2, requestMs: 55, waitedMs: 12, failed: { 404: 2, timeout: 1 } }], ]); }); @@ -40,54 +45,79 @@ describe('shared counts', () => { const { writer, reader, named } = shared(); for (let i = 0; i < 10_000; i++) { - writer.add('detail', { requests: 1, requestMs: 2, failed: i % 2 === 0 ? { 404: 1 } : {} }); + writer.add('detail', { + requests: 1, + requestMs: 2, + waitedMs: 0, + failed: i % 2 === 0 ? { 404: 1 } : {}, + }); } - // The label, with its two counts — and its one kind of failure. + // The label, with its three counts — and its one kind of failure. expect(named).toEqual([{ label: 'detail' }, { label: 'detail', failure: '404' }]); expect(reader.read().get('detail')).toEqual({ requests: 10_000, requestMs: 20_000, + waitedMs: 0, failed: { 404: 5_000 }, }); }); - it('names a label once for both its counts, before it has taken any time', () => { + it('names a label once for all three of its counts, before it has taken any time', () => { const { writer, reader, named } = shared(); - writer.add('detail', { requests: 1, requestMs: 0, failed: {} }); - writer.add('detail', { requests: 0, requestMs: 9, failed: {} }); + writer.add('detail', { requests: 1, requestMs: 0, waitedMs: 0, failed: {} }); + writer.add('detail', { requests: 0, requestMs: 9, waitedMs: 0, failed: {} }); expect(named).toEqual([{ label: 'detail' }]); - expect(reader.read().get('detail')).toEqual({ requests: 1, requestMs: 9, failed: {} }); + expect(reader.read().get('detail')).toEqual({ + requests: 1, + requestMs: 9, + waitedMs: 0, + failed: {}, + }); }); it('hands back what it has no slot left for, and keeps counting what it has', () => { const { writer, reader } = shared(); - // Every slot taken, two to a label after the seven always there — when it - // began, and the six of the channel-days — and the odd one by a failure. - const labels = Math.floor((SLOTS - 7) / 2); + // Every slot taken, three to a label after the twelve always there — when + // it began, the six of the channel-days and the five of its time — and the + // odd one by a failure. + const labels = Math.floor((SLOTS - 12) / 3); for (let i = 0; i < labels; i++) { - expect(writer.add(`label${i}`, { requests: 1, requestMs: 0, failed: {} })).toBeUndefined(); + expect( + writer.add(`label${i}`, { requests: 1, requestMs: 0, waitedMs: 0, failed: {} }), + ).toBeUndefined(); } - expect(writer.add('label1', { requests: 0, requestMs: 0, failed: { 500: 1 } })).toBeUndefined(); + expect( + writer.add('label1', { requests: 0, requestMs: 0, waitedMs: 0, failed: { 500: 1 } }), + ).toBeUndefined(); // A label already slotted still counts; a failure it has no slot for, and // a label of none, are handed back. - expect(writer.add('label0', { requests: 1, requestMs: 7, failed: { 503: 1 } })).toEqual({ + expect( + writer.add('label0', { requests: 1, requestMs: 7, waitedMs: 0, failed: { 503: 1 } }), + ).toEqual({ requests: 0, requestMs: 0, + waitedMs: 0, failed: { 503: 1 }, }); - expect(writer.add('more', { requests: 1, requestMs: 3, failed: {} })).toEqual({ + expect(writer.add('more', { requests: 1, requestMs: 3, waitedMs: 4, failed: {} })).toEqual({ requests: 1, requestMs: 3, + waitedMs: 4, + failed: {}, + }); + expect(reader.read().get('label0')).toEqual({ + requests: 2, + requestMs: 7, + waitedMs: 0, failed: {}, }); - expect(reader.read().get('label0')).toEqual({ requests: 2, requestMs: 7, failed: {} }); expect(reader.read().size).toBe(labels); }); @@ -99,7 +129,7 @@ describe('shared counts', () => { writer.count('fetched', 1); // A day that came back empty and then did not. writer.count('empty', -1); - writer.add('request', { requests: 1, requestMs: 0, failed: {} }); + writer.add('request', { requests: 1, requestMs: 0, waitedMs: 0, failed: {} }); expect(reader.tally()).toEqual({ fetched: 2, @@ -111,7 +141,40 @@ describe('shared counts', () => { }); // And none of a label's slots taken by them. expect(named).toEqual([{ label: 'request' }]); - expect(reader.read().get('request')).toEqual({ requests: 1, requestMs: 0, failed: {} }); + expect(reader.read().get('request')).toEqual({ + requests: 1, + requestMs: 0, + waitedMs: 0, + failed: {}, + }); + }); + + it('keeps where its time went in slots of their own, to the microsecond', () => { + const { writer, reader, named } = shared(); + + writer.spend('source', 1.5); + writer.spend('queued', 2); + writer.spend('source', 0.25); + // The slots either side of them: the last channel-day count, and the + // first label's. + writer.count('sitesFailed', 1); + writer.add('request', { requests: 1, requestMs: 0, waitedMs: 0, failed: {} }); + + expect(reader.spent()).toEqual({ + source: 1.75, + held: 0, + rateLimit: 0, + working: 0, + queued: 2, + }); + expect(reader.tally().sitesFailed).toBe(1); + expect(named).toEqual([{ label: 'request' }]); + expect(reader.read().get('request')).toEqual({ + requests: 1, + requestMs: 0, + waitedMs: 0, + failed: {}, + }); }); it('says when the site began, in a slot of its own', () => { @@ -140,10 +203,29 @@ describe('shared counts', () => { writer.begin(1_700_000_000_000); expect(() => writer.count('hours' as never, 1)).toThrow('No channel-day count is called hours'); + // Nor one every object has, which a lookup in a plain object would find. + expect(() => writer.count('constructor' as never, 1)).toThrow( + 'No channel-day count is called constructor', + ); // When it began, untouched. expect(reader.began()).toBe(1_700_000_000_000); }); + it('refuses a part of its time it does not know, rather than adding it to another', () => { + const { writer, reader } = shared(); + + writer.begin(1_700_000_000_000); + + expect(() => writer.spend('napping' as never, 1)).toThrow( + "No part of a site's time is called napping", + ); + expect(() => writer.spend('constructor' as never, 1)).toThrow( + "No part of a site's time is called constructor", + ); + expect(reader.began()).toBe(1_700_000_000_000); + expect(Object.values(reader.spent())).toEqual([0, 0, 0, 0, 0]); + }); + it('names a slot only once what was added is in it', () => { const counts = sharedCounts(); const reader = new CountReader(counts); @@ -155,11 +237,11 @@ describe('shared counts', () => { seen.push(reader.read().get('detail')); }); - writer.add('detail', { requests: 1, requestMs: 7, failed: { 404: 1 } }); + writer.add('detail', { requests: 1, requestMs: 7, waitedMs: 0, failed: { 404: 1 } }); expect(seen).toEqual([ - { requests: 1, requestMs: 7, failed: {} }, - { requests: 1, requestMs: 7, failed: { 404: 1 } }, + { requests: 1, requestMs: 7, waitedMs: 0, failed: {} }, + { requests: 1, requestMs: 7, waitedMs: 0, failed: { 404: 1 } }, ]); }); @@ -173,7 +255,7 @@ describe('shared counts', () => { }); try { - writer.add('detail', { requests: 1, requestMs: 2, failed: { 404: 1 } }); + writer.add('detail', { requests: 1, requestMs: 2, waitedMs: 0, failed: { 404: 1 } }); } finally { delete (Object.prototype as { polluted?: unknown }).polluted; } @@ -181,6 +263,7 @@ describe('shared counts', () => { expect(reader.read().get('detail')).toEqual({ requests: 1, requestMs: 2, + waitedMs: 0, failed: { 404: 1 }, }); }); @@ -189,7 +272,7 @@ describe('shared counts', () => { const counts = sharedCounts(); const writer = new CountWriter(counts, () => {}); - writer.add('detail', { requests: 1, requestMs: 5, failed: {} }); + writer.add('detail', { requests: 1, requestMs: 5, waitedMs: 0, failed: {} }); expect(new CountReader(counts).read().size).toBe(0); }); diff --git a/test/thread.test.ts b/test/thread.test.ts index 8dbccac..d4c0ed6 100644 --- a/test/thread.test.ts +++ b/test/thread.test.ts @@ -5,7 +5,7 @@ import PQueue from 'p-queue'; import { describe, expect, it, vi } from 'vitest'; import { CacheManager, MemoryCacheDriver } from '../src/cache/main.js'; import type { CacheStore } from '../src/cache/types.js'; -import type { EpgEventInput, RequestCounts } from '../src/core/events.js'; +import { EVENT_TYPES, type EpgEventInput, type RequestCounts } from '../src/core/events.js'; import { build } from '../src/build.js'; import { grab, StateKey } from '../src/grabber/main.js'; import type { Run } from '../src/grabber/site-run.js'; @@ -19,6 +19,7 @@ import { type SiteThreadData, } from '../src/grabber/thread.js'; import { addRequests, openSession, requestsObject } from '../src/grabber/session.js'; +import { SiteClock } from '../src/grabber/clock.js'; import { CountReader, CountWriter, sharedCounts } from '../src/grabber/shared-counts.js'; import { getXmltvOffset } from '../src/xmltv/main.js'; @@ -45,9 +46,12 @@ function hosting( grabbedAt: NOW.toISOString(), startDay: DAY, timeZone: 'UTC', - emit: (event) => { - events.push(event); - }, + emit: Object.assign( + (event: EpgEventInput) => { + events.push(event); + }, + { hears: new Set(EVENT_TYPES) }, + ), ...(signal ? { signal } : {}), localWork: new PQueue({ concurrency: 4 }), enqueue: (queue, task, options = {}) => @@ -56,6 +60,7 @@ function hosting( requested: (label, done) => { requested.push([label, done]); }, + spent: undefined, }; return { run, events, requested }; @@ -72,6 +77,7 @@ function started(site: string, extra: Partial = {}): SiteThreadD defaults: { days: 1 }, localConcurrency: 4, counts: sharedCounts(), + hears: EVENT_TYPES, ...extra, }; } @@ -158,6 +164,7 @@ function siteDoneOf(fetched: number): EpgEventInput { requests: 0, ms: 5, requestMs: 0, + msBy: { source: 0, held: 0, rateLimit: 0, working: 0, queued: 0, other: 5 }, requestsBy: {}, fetched, empty: 0, @@ -283,14 +290,17 @@ describe('a site in a thread', () => { const { run, events } = hosting(cache, controller.signal); const said = run.emit; - run.emit = (event) => { - said(event); + run.emit = Object.assign( + (event: EpgEventInput) => { + said(event); - // Once the request is waiting, which it does until it is told to stop. - if (event.type === 'request:started') { - controller.abort(new Error('run cancelled')); - } - }; + // Once the request is waiting, which it does until it is told to stop. + if (event.type === 'request:started') { + controller.abort(new Error('run cancelled')); + } + }, + { hears: said.hears }, + ); await converse(run, started('slow.example')); @@ -605,7 +615,12 @@ describe('a site in a thread', () => { await converse(run, started('offset.example'), (message) => kinds.push(message.kind)); expect(toldOf(requested)).toEqual({ - request: { requests: 1, requestMs: expect.any(Number), failed: {} }, + request: { + requests: 1, + requestMs: expect.any(Number), + waitedMs: expect.any(Number), + failed: {}, + }, }); // One name, for the label's two counts — and no message for the request // itself. @@ -649,7 +664,12 @@ describe('a site in a thread', () => { await converse(run, started('failing.example')); expect(toldOf(requested)).toEqual({ - request: { requests: 1, requestMs: expect.any(Number), failed: { network: 1 } }, + request: { + requests: 1, + requestMs: expect.any(Number), + waitedMs: expect.any(Number), + failed: { network: 1 }, + }, }); }); @@ -789,8 +809,11 @@ describe('a site in a thread', () => { // memory — the last of them counted just before the thread went. writer.begin(Date.now() - 50); writer.count('fetched', 1); - writer.add('request', { requests: 1, requestMs: 30, failed: {} }); - writer.add('request', { requests: 1, requestMs: 10, failed: { 404: 1 } }); + writer.add('request', { requests: 1, requestMs: 30, waitedMs: 6, failed: {} }); + writer.add('request', { requests: 1, requestMs: 10, waitedMs: 0, failed: { 404: 1 } }); + // And where its time went, as its clock said it. + writer.spend('source', 30.4); + writer.spend('working', 9.6); lose(new Error('the thread exited')); port.close(); @@ -805,19 +828,33 @@ describe('a site in a thread', () => { sitesFailed: 1, requests: 2, requestMs: 40, - requestsBy: { request: { requests: 2, requestMs: 40, failed: { 404: 1 } } }, + requestsBy: { request: { requests: 2, requestMs: 40, waitedMs: 6, failed: { 404: 1 } } }, + }); + + const { ms, msBy } = events[1] as Extract; + + // From when it began — and what of it the thread had measured, the rest + // of it other. + expect(ms).toBeGreaterThanOrEqual(50); + expect(msBy).toEqual({ + source: 30, + held: 0, + rateLimit: 0, + working: 10, + queued: 0, + other: ms - 40, }); - // From when it began. - expect((events[1] as { ms: number }).ms).toBeGreaterThanOrEqual(50); // Its channel-days at once, as its turn ended, and its label's place — // what it asked once the run hangs up. expect(run.tally).toMatchObject({ fetched: 1, sitesFailed: 1 }); - expect(toldOf(requested)).toEqual({ request: { requests: 0, requestMs: 0, failed: {} } }); + expect(toldOf(requested)).toEqual({ + request: { requests: 0, requestMs: 0, waitedMs: 0, failed: {} }, + }); hangUp(); expect(toldOf(requested)).toEqual({ - request: { requests: 2, requestMs: 40, failed: { 404: 1 } }, + request: { requests: 2, requestMs: 40, waitedMs: 6, failed: { 404: 1 } }, }); expect(run.tally).toMatchObject({ fetched: 1, sitesFailed: 1 }); } finally { @@ -867,7 +904,7 @@ describe('a site in a thread', () => { try { writer.begin(); - writer.add('request', { requests: 1, requestMs: 5, failed: {} }); + writer.add('request', { requests: 1, requestMs: 5, waitedMs: 0, failed: {} }); port.postMessage({ kind: 'done' }); const hangUp = await hosted; @@ -875,7 +912,9 @@ describe('a site in a thread', () => { hangUp(); hangUp(); - expect(toldOf(requested)).toEqual({ request: { requests: 1, requestMs: 5, failed: {} } }); + expect(toldOf(requested)).toEqual({ + request: { requests: 1, requestMs: 5, waitedMs: 0, failed: {} }, + }); } finally { close(); } @@ -992,16 +1031,18 @@ describe('a site in a thread', () => { port.postMessage({ kind: 'requested', label: 'more', - done: { requests: 1, requestMs: 4, failed: {} }, + done: { requests: 1, requestMs: 4, waitedMs: 0, failed: {} }, }); port.postMessage({ kind: 'done' }); const hangUp = await hosted; // As it was said, not once the run hangs up — and once. - expect(requested).toEqual([['more', { requests: 1, requestMs: 4, failed: {} }]]); + expect(requested).toEqual([['more', { requests: 1, requestMs: 4, waitedMs: 0, failed: {} }]]); hangUp(); - expect(toldOf(requested)).toEqual({ more: { requests: 1, requestMs: 4, failed: {} } }); + expect(toldOf(requested)).toEqual({ + more: { requests: 1, requestMs: 4, waitedMs: 0, failed: {} }, + }); } finally { close(); } @@ -1014,15 +1055,15 @@ describe('a site in a thread', () => { try { writer.begin(); - writer.add('request', { requests: 1, requestMs: 5, failed: {} }); + writer.add('request', { requests: 1, requestMs: 5, waitedMs: 0, failed: {} }); port.postMessage({ kind: 'done' }); const hangUp = await hosted; // A request a parse did not wait for, answered after its site was done — // and under a label, and a failure, first named then. - writer.add('detail', { requests: 1, requestMs: 0, failed: {} }); - writer.add('detail', { requests: 0, requestMs: 7, failed: { 503: 1 } }); + writer.add('detail', { requests: 1, requestMs: 0, waitedMs: 0, failed: {} }); + writer.add('detail', { requests: 0, requestMs: 7, waitedMs: 0, failed: { 503: 1 } }); await new Promise((resolve) => setImmediate(resolve)); // Each label in its place among the run's as it was first sent. @@ -1031,8 +1072,8 @@ describe('a site in a thread', () => { hangUp(); expect(toldOf(requested)).toEqual({ - request: { requests: 1, requestMs: 5, failed: {} }, - detail: { requests: 1, requestMs: 7, failed: { 503: 1 } }, + request: { requests: 1, requestMs: 5, waitedMs: 0, failed: {} }, + detail: { requests: 1, requestMs: 7, waitedMs: 0, failed: { 503: 1 } }, }); } finally { close(); @@ -1146,6 +1187,85 @@ describe('a site in a thread', () => { } }); + it('says how long a call waited for a slot, and nothing of one that never asks for one', async () => { + const cache = new CacheManager({ driver: new MemoryCacheDriver() }); + const { run } = hosting(cache); + + run.localWork = new PQueue({ concurrency: 1 }); + + let unblock = (): void => {}; + + void run.localWork.add(() => new Promise((resolve) => (unblock = resolve))); + + const { hosted, port, close } = hostOnly(run); + + try { + const answers = new Map(); + const both = new Promise((resolve) => { + port.on('message', (message: { kind: string; id: number; queuedMs?: number }) => { + if (message.kind === 'answer' && answers.set(message.id, message).size === 2) { + resolve(); + } + }); + }); + + port.postMessage({ kind: 'cache', id: 1, op: 'getMeta', args: [{}] }); + port.postMessage({ kind: 'cache', id: 2, op: 'getState', args: ['lost.example', 'one'] }); + await vi.waitFor(() => expect(run.localWork.size).toBe(1)); + await new Promise((resolve) => setTimeout(resolve, 40)); + unblock(); + await both; + + expect(answers.get(1)!.queuedMs).toBeGreaterThanOrEqual(35); + expect(answers.get(2)).not.toHaveProperty('queuedMs'); + + port.postMessage({ kind: 'done' }); + (await hosted)(); + } finally { + close(); + } + }); + + it('tells the site’s clock what a call waited for a slot on the run’s thread', async () => { + const { port1, port2 } = new MessageChannel(); + + // The run's side, answering after 40ms, 25 of them spent waiting for a slot. + port1.on('message', (message: { kind: string; id: number }) => { + if (message.kind === 'cache') { + setTimeout(() => { + port1.postMessage({ kind: 'answer', id: message.id, value: undefined, queuedMs: 25 }); + }, 40); + } + }); + + const cache = new RemoteCache(port2, undefined); + const clock = new SiteClock(); + + try { + cache.timeWith(clock); + clock.turn(true); + clock.queued(); + clock.started(); + await cache.getMeta({ site: 'a.tv', channelId: 'one', day: DAY }); + + // Its work's time, less the 25 it waited — which is now waiting for a slot. + expect(clock.spent().queued).toBeCloseTo(25, 1); + expect(clock.spent().working).toBeGreaterThanOrEqual(10); + + // And a state call, answered on the run's thread with no slot: nothing + // of it is waiting elsewhere. + const before = clock.spent(); + + await cache.getState('a.tv', 'one'); + expect(clock.spent().queued).toBe(before.queued); + + await cache.close(); + } finally { + port1.close(); + port2.close(); + } + }); + it('answers with an error whose cause no thread can be handed', async () => { const cache = new CacheManager({ driver: new MemoryCacheDriver() }); const { run } = hosting(cache); diff --git a/test/threads.test.ts b/test/threads.test.ts index 5a1de45..e81d2d4 100644 --- a/test/threads.test.ts +++ b/test/threads.test.ts @@ -30,6 +30,8 @@ async function grabbed( abortOn?: (event: EpgEventInput) => boolean; /** How long a site's state takes to save, as it does in a store across a network. */ savingMs?: number; + /** The kinds the run's reporter wants — every kind, where left out. */ + wants?: (type: EpgEventInput['type']) => boolean; } = {}, ) { const cache = new CacheManager({ driver: new MemoryCacheDriver() }); @@ -61,13 +63,16 @@ async function grabbed( configModule: CONFIG, ...(options.siteConcurrency === undefined ? {} : { siteConcurrency: options.siteConcurrency }), signal: stop.signal, - reporter: (event) => { - events.push(event); + reporter: Object.assign( + (event: EpgEventInput) => { + events.push(event); - if (options.abortOn?.(event)) { - stop.abort(new Error('run cancelled')); - } - }, + if (options.abortOn?.(event)) { + stop.abort(new Error('run cancelled')); + } + }, + options.wants === undefined ? {} : { wants: options.wants }, + ), }); return { cache, events, summary }; @@ -84,6 +89,24 @@ describe('sites in worker threads', () => { expect(getXmltvOffset(programme!.start)).toBe(120); }); + it('neither makes nor sends what the run’s reporter does not want', async () => { + const unwanted = new Set(['site:note', 'request:started', 'request:done', 'entry:fetched']); + const kinds = async (wants?: (type: string) => boolean): Promise => + (await grabbed(['offset.example'], wants === undefined ? {} : { wants })).events.map( + (event) => event.type, + ); + + // Each of them made, where it is wanted — the site logs as it parses. + const all = new Set(await kinds()); + + expect([...unwanted].filter((type) => !all.has(type))).toEqual([]); + + const some = await kinds((type) => !unwanted.has(type)); + + expect(some.filter((type) => unwanted.has(type))).toEqual([]); + expect(some).toContain('site:done'); + }); + it("counts each site's requests by kind over there, and sums them here", async () => { const { events } = await grabbed(['offset.example', 'failing.example']); @@ -92,12 +115,22 @@ describe('sites in worker threads', () => { ) as Extract; expect(failing.requestsBy).toEqual({ - request: { requests: 1, requestMs: expect.any(Number), failed: { network: 1 } }, + request: { + requests: 1, + requestMs: expect.any(Number), + waitedMs: expect.any(Number), + failed: { network: 1 }, + }, }); // Read from what a thread said as each site finished. expect(events.at(-1)).toMatchObject({ type: 'grab:done' }); expect((events.at(-1) as Extract).requestsBy).toEqual({ - request: { requests: 2, requestMs: expect.any(Number), failed: { network: 1 } }, + request: { + requests: 2, + requestMs: expect.any(Number), + waitedMs: expect.any(Number), + failed: { network: 1 }, + }, }); }); @@ -138,8 +171,18 @@ describe('sites in worker threads', () => { } const counted = { - channels: { requests: 1, requestMs: expect.any(Number), failed: {} }, - request: { requests: 1, requestMs: expect.any(Number), failed: {} }, + channels: { + requests: 1, + requestMs: expect.any(Number), + waitedMs: expect.any(Number), + failed: {}, + }, + request: { + requests: 1, + requestMs: expect.any(Number), + waitedMs: expect.any(Number), + failed: {}, + }, }; const done = events.find((event) => event.type === 'site:done') as Extract< EpgEventInput, @@ -147,9 +190,12 @@ describe('sites in worker threads', () => { >; expect(done).toMatchObject({ requests: 2 }); - // Its list's time its own too, as in a run on one thread. + // Its list's time its own too, as in a run on one thread — and where it + // went: the list out to the source here, beside what the thread measured. expect(done.ms).toBeGreaterThanOrEqual(50); expect(done.ms).toBeGreaterThanOrEqual(done.requestMs - 5); + expect(done.msBy.source).toBeGreaterThanOrEqual(50); + expect(Object.values(done.msBy).reduce((sum, ms) => sum + ms, 0)).toBe(done.ms); expect(done.requestsBy).toEqual(counted); expect(Object.keys(done.requestsBy)).toEqual(['channels', 'request']); expect((events.at(-1) as Extract).requestsBy).toEqual( diff --git a/tsconfig.json b/tsconfig.json index 505c648..1cc5774 100644 --- a/tsconfig.json +++ b/tsconfig.json @@ -3,7 +3,7 @@ "target": "ES2023", "module": "NodeNext", "moduleResolution": "NodeNext", - "lib": ["ES2023", "ESNext.Disposable"], + "lib": ["ES2023", "ES2024.Collection", "ESNext.Disposable"], "types": ["node"], "strict": true, "noUncheckedIndexedAccess": true,