diff --git a/README.md b/README.md index cb11115..bcc59c7 100644 --- a/README.md +++ b/README.md @@ -320,20 +320,25 @@ Scheduler, cron and container arrangements. ### Large libraries -TheIntroDB allows 1000 requests per UTC day with an API key, and 500 without one. -A library of tens of thousands of items cannot be scanned in a day, so the tool -is built to be left running rather than to be finished in one pass: +TheIntroDB allows 100000 requests per UTC day with an API key, and 5000 without +one. With a key that is more than a full pass over a large library needs, so a +run is bounded by pacing rather than by the allowance: the client sends at most +25 requests per 10 seconds, which puts 40,000 items at about four and a half +hours at that rate. Without a key the allowance is worked through over several +runs. + +What the tool is built around is that each item costs one request, ever: - **An item is scanned once.** Every lookup is recorded, and a recorded item is answered from the ledger afterwards without a request, however long ago it was scanned. Expiring that record is what would make a large library never finish, so nothing expires it. -- **A run uses the whole day's allowance.** It scans the items it has never seen - until the allowance is spent, then stops cleanly and logs how many are left. - That is not an error, and the nightly timer keeps its schedule. +- **A run spends the allowance on items it has never seen**, then stops cleanly + and logs how many are left. That is not an error, and the nightly timer keeps + its schedule. - **The next run continues.** Because every scan is recorded, it picks up exactly - where the last one stopped. A 40,000-item library at 1000 requests a day - converges in about 40 days, and each run writes the markers it has data for. + where the last one stopped, so an interrupted or allowance-stopped run costs + only the items it never reached. - **Re-scanning is deliberate.** Nothing is asked about twice on a timer. When a submission lands and you want fresh answers, name the items: @@ -371,7 +376,7 @@ Lookup order: `$PLEX_SYNC_CONFIG`, `./plex-sync.toml`, | `plex.device_name` | `plex-sync` | what Plex shows for this tool in its device list | | `plex.client_id` | stored in `state_dir` | the identity Plex keys this install's device entry on | | `theintrodb.api_key` | | optional TheIntroDB API key | -| `theintrodb.daily_budget` | `1000` | requests per UTC day | +| `theintrodb.daily_budget` | `100000` | requests per UTC day | | `sources.chapters` | `false` | use chapter names as a source | | `sources.detection` | `false` | run local fingerprint detection | | `segments.intro` | `true` | write intro markers | diff --git a/docs/scheduling.md b/docs/scheduling.md index f39b639..c41498d 100644 --- a/docs/scheduling.md +++ b/docs/scheduling.md @@ -21,9 +21,15 @@ so `journalctl`, the Docker log or a file all work as a record of what happened. ## Large libraries, and the daily allowance -TheIntroDB allows 1000 requests per UTC day with an API key and 500 without one, -so a library of tens of thousands of items is not scanned in one night. That is -expected, and the schedule is how it is meant to be worked through: +TheIntroDB allows 100000 requests per UTC day with an API key and 5000 without +one. With a key that covers a full pass over a large library — 40,000 items need +40,000 requests, and the client's default pacing limit of 25 requests per 10 +seconds puts that at about four and a half hours at that rate, longer if +responses are slow or `theintrodb.max_per_window` is set lower — so a nightly +schedule is enough. Without a key the allowance has to be worked through over +several nights. + +The rules that make either case finish: - An item that has been looked up is recorded, and a recorded item is answered from the ledger afterwards without a request. Nothing expires that record, so @@ -34,8 +40,10 @@ expected, and the schedule is how it is meant to be worked through: that reported an error every night would be a timer nobody reads. - The next run continues from the same place, because every scan is recorded. -A schedule therefore converges: a 40,000-item library at 1000 requests a day -gets through in about 40 days, writing the markers it has data for each night. +A schedule therefore converges: every item is asked about once, so a run that +stops — on the clock, on the allowance, or by being interrupted — costs only the +items it never reached, and the next one carries on from there. With a key, a +40,000-item library usually finishes in the first night. Raising `theintrodb.daily_budget` beyond the allowance does not help — the server refuses the request either way; the budget is what stops the client before the refusal. diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index bc6fe37..17302db 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -229,14 +229,15 @@ which puts them back. ## "budget" or "usage-limited" in a plan The day's allowance is spent. The run stops cleanly and resumes tomorrow. Without -an API key the allowance is 500 requests a day instead of 1000; a key on your +an API key the allowance is 5000 requests a day instead of 100000; a key on your TheIntroDB account raises it. ## Running out of allowance every day -Expected on a large library: a full pass needs more requests than a day allows. -Progress accumulates, because every answer including a 404 is cached, so each -day works on new content. +Expected without a key, or on a library big enough that one day's allowance does +not cover it: a full pass needs more requests than a day allows. Progress +accumulates, because every answer including a 404 is recorded, so each day works +on new content. ## Undo diff --git a/internal/config/config.go b/internal/config/config.go index bf0ac89..83f0994 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -211,7 +211,7 @@ func Default() *Config { TheIntroDB: TheIntroDB{ BaseURL: "https://api.theintrodb.org/v4", TimeoutS: 20, - DailyBudget: 1000, + DailyBudget: AccountDailyBudget, MaxPerWindow: 25, WindowS: 10, MissTTLDays: 14, @@ -272,19 +272,23 @@ func (t TheIntroDB) MinDelay() float64 { return t.WindowS / float64(t.MaxPerWindow) } -// AnonymousDailyBudget is the allowance without an API key: 500 per public IP -// per UTC day, against 1000 for an account. -const AnonymousDailyBudget = 500 +// AnonymousDailyBudget is the allowance without an API key: 5000 per public IP +// per UTC day, against 100000 for an account. +// +// The numbers are the API's own (UsageLimitMediaPublicIP and +// UsageLimitMediaAuthenticated, and the X-UsageLimit-Limit header it sends +// back), and they are higher than they were: an account used to get 1000 a day. +const AnonymousDailyBudget = 5000 // AccountDailyBudget is the allowance with an API key, and the highest any // caller can have. -const AccountDailyBudget = 1000 +const AccountDailyBudget = 100000 // EffectiveDailyBudget is the allowance that actually applies. // -// Budgeting for 1000 requests when only 500 are allowed means discovering the -// limit by being rate-limited for the rest of the day, so an anonymous client -// budgets for the anonymous allowance. A non-positive budget still means +// Budgeting for 100000 requests when only 5000 are allowed means discovering +// the limit by being rate-limited for the rest of the day, so an anonymous +// client budgets for the anonymous allowance. A non-positive budget still means // "uncounted". func (t TheIntroDB) EffectiveDailyBudget() int { if t.DailyBudget <= 0 { @@ -674,10 +678,10 @@ func (c *Config) Validate() error { if c.TheIntroDB.DailyBudget <= 0 { problems = append(problems, "theintrodb.daily_budget must be positive") } - // An account allowance of 1000 is the highest anyone can have, so a bigger - // number is not a bigger budget, it is a number that cannot be true. Left - // unchecked it would sit in the file looking like an allowance nobody will - // honour. + // An account allowance of 100000 is the highest anyone can have, so a + // bigger number is not a bigger budget, it is a number that cannot be + // true. Left unchecked it would sit in the file looking like an allowance + // nobody will honour. if c.TheIntroDB.DailyBudget > AccountDailyBudget { problems = append(problems, fmt.Sprintf( "theintrodb.daily_budget must not exceed the ceiling of %d requests a day", @@ -871,7 +875,7 @@ client_id = "" # The TheIntroDB API key is optional. With a key the daily allowance is higher # and your own pending submissions are included in what you get back. api_key = "" -daily_budget = 1000 +daily_budget = 100000 [sources] # Alternate marker sources. TheIntroDB always wins a segment type it answered; diff --git a/internal/config/config_test.go b/internal/config/config_test.go index 696ed94..34c8cc3 100644 --- a/internal/config/config_test.go +++ b/internal/config/config_test.go @@ -65,15 +65,23 @@ func TestPacingStaysUnderTheServerCeiling(t *testing.T) { } func TestAnonymousClientBudgetsForTheAnonymousAllowance(t *testing.T) { - anonymous := TheIntroDB{DailyBudget: 1000} + // A budget above the anonymous allowance is not a bigger budget without a + // key: the client would discover the real limit by being rate-limited. + anonymous := TheIntroDB{DailyBudget: AccountDailyBudget} if got := anonymous.EffectiveDailyBudget(); got != AnonymousDailyBudget { t.Errorf("without a key the budget is %d, want %d: budgeting for more means "+ "discovering the limit by being rate-limited", got, AnonymousDailyBudget) } - withKey := TheIntroDB{DailyBudget: 1000, APIKey: "abc"} - if got := withKey.EffectiveDailyBudget(); got != 1000 { - t.Errorf("with a key the budget is %d, want the configured 1000", got) + // A budget under the allowance is what the caller asked for, and stays. + modest := TheIntroDB{DailyBudget: 120} + if got := modest.EffectiveDailyBudget(); got != 120 { + t.Errorf("a budget below the allowance became %d, want 120", got) + } + + withKey := TheIntroDB{DailyBudget: AccountDailyBudget, APIKey: "key"} + if got := withKey.EffectiveDailyBudget(); got != AccountDailyBudget { + t.Errorf("with a key the budget is %d, want the configured %d", got, AccountDailyBudget) } // A non-positive budget means "do not count", and must stay that way. @@ -83,6 +91,40 @@ func TestAnonymousClientBudgetsForTheAnonymousAllowance(t *testing.T) { } } +// The allowances are the API's, not invented here, and when the API raises them +// these are the numbers that move. An account used to get 1000 requests a day; +// raising it without raising the ceiling below made the real allowance +// unconfigurable. +func TestTheAllowancesMatchTheAPI(t *testing.T) { + if AccountDailyBudget != 100000 { + t.Errorf("account allowance = %d, want the API's 100000 (UsageLimitMediaAuthenticated)", + AccountDailyBudget) + } + if AnonymousDailyBudget != 5000 { + t.Errorf("anonymous allowance = %d, want the API's 5000 (UsageLimitMediaPublicIP)", + AnonymousDailyBudget) + } +} + +// The ceiling has to accept the allowance the API actually grants, or a config +// naming the real allowance is refused as a number that cannot be true. +func TestTheBudgetCeilingAcceptsTheAccountAllowance(t *testing.T) { + cfg := Default() + if got := cfg.TheIntroDB.DailyBudget; got != AccountDailyBudget { + t.Errorf("the default budget is %d, want the account allowance %d", got, AccountDailyBudget) + } + + cfg.TheIntroDB.DailyBudget = 100000 // the API's usage limit for an account + if err := cfg.Validate(); err != nil { + t.Errorf("the account allowance must be configurable: %v", err) + } + + cfg.TheIntroDB.DailyBudget = 100001 + if err := cfg.Validate(); err == nil { + t.Error("a budget above the account allowance must be refused") + } +} + func TestLoadReadsFileEnvironmentAndDefaults(t *testing.T) { dir := t.TempDir() path := filepath.Join(dir, "config.toml") diff --git a/internal/ledger/ledger.go b/internal/ledger/ledger.go index 0bebcfc..7f7b12c 100644 --- a/internal/ledger/ledger.go +++ b/internal/ledger/ledger.go @@ -71,7 +71,7 @@ var schemaStatements = []string{ // // The cache answers "do we still trust this body"; this answers "have we // spent a request on this item yet". On a library of tens of thousands of - // items against a daily allowance of 500 or 1000, the second question is + // items against a daily allowance of 5000 or 100000, the second question is // the one that matters: a run must spend its requests on items it has never // seen, and must be able to stop and pick up where it left off tomorrow. // Expiring these rows is exactly what would make a large library never diff --git a/internal/tidb/client.go b/internal/tidb/client.go index d15a16b..c9925e7 100644 --- a/internal/tidb/client.go +++ b/internal/tidb/client.go @@ -9,7 +9,7 @@ // wrong: // // - A 429 is either the rate window (30 requests per 10 seconds) or the daily -// allowance (1000 per account, 500 per public IP, reset at UTC midnight). +// allowance (100000 per account, 5000 per public IP, reset at UTC midnight). // A rate-limit 429 carries only a Retry-After in seconds. A usage-limit 429 // carries X-UsageLimit-Reset in SECONDS UNTIL UTC MIDNIGHT, which can be up // to about 86400, plus a JSON body with a code of usage_limit_exceeded or @@ -229,7 +229,7 @@ type Client struct { // own default, and a non-positive DailyBudget means "uncounted". A nil ledger // disables caching and budget counting; a nil httpclient gets a default one. func NewClient(cfg config.TheIntroDB, led *ledger.Ledger, hc *httpclient.Client) *Client { - // Without a key the public allowance is 500 requests a day, not 1000. + // Without a key the public allowance is 5000 requests a day, not 100000. // Budget for the allowance that actually applies rather than discovering it // by being rate-limited for the rest of the day. cfg.DailyBudget = cfg.EffectiveDailyBudget() diff --git a/internal/tidb/client_test.go b/internal/tidb/client_test.go index 13542b3..e1e11c4 100644 --- a/internal/tidb/client_test.go +++ b/internal/tidb/client_test.go @@ -737,7 +737,7 @@ func TestUsageLimitWaitIsNotClampedToFiveMinutes(t *testing.T) { h := newHarness(t, func(call int, w http.ResponseWriter, r *http.Request) { w.Header().Set("X-UsageLimit-Reset", fmt.Sprint(reset)) - w.Header().Set("X-UsageLimit-Limit", "1000") + w.Header().Set("X-UsageLimit-Limit", "100000") w.Header().Set("X-UsageLimit-Remaining", "0") js(w, 429, `{"error":"daily limit reached","retry_after":"2 hours","code":"usage_limit_exceeded"}`) }, nil) diff --git a/internal/tidb/lookup.go b/internal/tidb/lookup.go index 6318e0b..bd28dbb 100644 --- a/internal/tidb/lookup.go +++ b/internal/tidb/lookup.go @@ -63,7 +63,7 @@ func (r LookupResult) OK() bool { return r.Status == 200 || r.Status == 404 } // The order is deliberate. First, the item's scan record: if it has been looked // up before, the answer stored then is returned and no request is made, however // long ago that was. A library of tens of thousands of items against a daily -// allowance of 500 or 1000 only ever finishes if each item is asked about once, +// allowance of 5000 or 100000 only ever finishes if each item is asked about once, // and an item that was scanned is what "finished" means; the way to ask again // is LookupForced, which is what a plan that names the item for a re-scan does. // diff --git a/internal/tui/settings.go b/internal/tui/settings.go index 1e0db8c..0ecb059 100644 --- a/internal/tui/settings.go +++ b/internal/tui/settings.go @@ -224,7 +224,7 @@ func settingsRows() []setting { }, { section: "TheIntroDB", key: "theintrodb.daily_budget", label: "daily budget", kind: settingInt, - help: "Requests per UTC day. The ceiling is 500 without a key and 1000 with one.", + help: "Requests per UTC day. The ceiling is 5000 without a key and 100000 with one.", get: intGet(func(c *config.Config) int { return c.TheIntroDB.DailyBudget }), set: intSet(func(c *config.Config, n int) { c.TheIntroDB.DailyBudget = n }), }, diff --git a/internal/tui/settings_test.go b/internal/tui/settings_test.go index 10ba66c..099a7a9 100644 --- a/internal/tui/settings_test.go +++ b/internal/tui/settings_test.go @@ -3,6 +3,7 @@ package tui import ( "os" "path/filepath" + "strconv" "strings" "testing" @@ -258,7 +259,11 @@ func TestARejectedValueIsNotApplied(t *testing.T) { for range m.buffer { press(m, "backspace") } - typeText(m, "99999") // above the server's ceiling + // One above the ceiling the API grants. Taken from the constant so this case + // cannot quietly become a valid value when the allowance is raised -- which + // is what happened when it was hardcoded to 99999 and the ceiling moved to + // 100000. + typeText(m, strconv.Itoa(config.AccountDailyBudget+1)) press(m, "enter") if !m.editing { diff --git a/internal/tui/view.go b/internal/tui/view.go index f20d6f4..2b84c3d 100644 --- a/internal/tui/view.go +++ b/internal/tui/view.go @@ -108,7 +108,7 @@ func (m *Model) statusScreen() string { b.WriteString(" Plex " + m.ok(m.readiness.PlexOK, m.readiness.PlexVersion, m.readiness.PlexError) + "\n") status := "present" if cfg.TheIntroDB.APIKey == "" { - status = "absent, 500 requests a day instead of 1000" + status = "absent, 5000 requests a day instead of 100000" } b.WriteString(" TheIntroDB " + m.ok(m.readiness.TIDBOK, status, m.readiness.TIDBError) + "\n") }