diff --git a/.github/workflows/inttest.yml b/.github/workflows/inttest.yml new file mode 100644 index 00000000..820aa447 --- /dev/null +++ b/.github/workflows/inttest.yml @@ -0,0 +1,33 @@ +name: integration +on: + push: + branches: + - main + - unstable + pull_request: + branches: + - main + - unstable +permissions: + contents: read +jobs: + integration: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 + with: + go-version-file: go.mod + - name: Install archive tools + run: | + sudo apt-get update + sudo apt-get install -y rar p7zip-full zip unzip + - name: Build unpackerr + run: go build -o unpackerr . + - name: Clone integration tests + run: git clone --depth 1 https://github.com/Unpackerr/unpackerr-inttest.git inttest + - name: Integration tests + working-directory: inttest + env: + UNPACKERR_BIN: ${{ github.workspace }}/unpackerr + run: go test -tags=integration -timeout 5m -count=1 -v ./test/... diff --git a/INTERNALS.md b/INTERNALS.md index 8866ac88..e225bca3 100644 --- a/INTERNALS.md +++ b/INTERNALS.md @@ -25,7 +25,7 @@ Two stacked feature series plus a follow-up mux swap. Closed duplicates (`#688`, | [#693](https://github.com/Unpackerr/unpackerr/pull/693) | OpenAPI | Embedded `pkg/unpackerr/openapi.json`. | | [#697](https://github.com/Unpackerr/unpackerr/pull/697) | Stdlib mux | Dropped `julienschmidt/httprouter`. Go `http.ServeMux` with `{section}` and `GET …/{$}` for the index. | | [#722](https://github.com/Unpackerr/unpackerr/pull/722) | PUT env overlay | Starr / folder / hook PUTs re-apply `UN_*` onto live so omitting an env-only slug cannot wipe it. File snapshot stays file-shaped. | -| v1.0.0 (September 2026) | Named instance maps | Sonarr, Radarr, Lidarr, Readarr, folders, webhooks, and cmdhooks are `map[string]*Config` keyed by a slug. Dual-read old `[[section]]` arrays as `"0"`, `"1"`, …. PUT writes the request body as the file document. Live is `clone(fileConfig)` + `ParseENV` (cnfg overlays existing map entries). Env-only slugs still appear on live after `{}`. A client that PUTs live overlay values persists them. | +| v1.0.0 (October 2026) | Named instance maps | Sonarr, Radarr, Lidarr, Readarr, folders, webhooks, and cmdhooks are `map[string]*Config` keyed by a slug. Dual-read old `[[section]]` arrays as `"0"`, `"1"`, …. PUT writes the request body as the file document. Live is `clone(fileConfig)` + `ParseENV` (cnfg overlays existing map entries). Env-only slugs still appear on live after `{}`. A client that PUTs live overlay values persists them. | The mux PR is routing only. Behavior below is from the API stacks unless noted. @@ -219,7 +219,7 @@ Env-only (no config path): skip write, still apply live. **New `ui_password` on PUT:** `!!cryptd!!…`, `webauth`, `noauth`, or `user:<64-char hex>` where the hex is the same PBKDF2 digest as login (`CryptPass.Set`, then bcrypt; mixed-case hex is stored lowercase). Plaintext `user:pass` is **400**. While live auth is local password, changing the hash or switching to header/noauth requires `uiCurrentKdf` (login `Valid()` on the current username). Header/noauth live mode does not. `uiCurrentKdf` is a PUT-only JSON field and is never written to TOML. A body that contains only `uiCurrentKdf` is **400** (empty section), so it cannot wipe `listen_addr` / keys / roles. Omitting `uiPassword` or sending the on-disk value unchanged keeps the live overlay, so `UN_WEBSERVER_UI_PASSWORD` is not replaced by the file hash. -**Starr PUT:** JSON object keyed by slug (letters, digits, `_`, `-`; same charset as roles). `name` is display only. Invalid URL/key on a **PUT-body** instance is **400** after `ParseENV` fills env secrets (so a Save that omits `apiKey` because `UN_*_API_KEY` is set still succeeds). Env-only overlay slugs that startup would skip (URL without key, or the reverse) are dropped from live, not 400 — they cannot block saving other instances. File commit is the PUT body. Live map is that body plus `ParseENV`, so a *complete* env-only extra instance survives `{}`. `path` merges into `paths` without dupes. Last poll `Queue` carries over when `url` + expanded `apiKey` match (`starrIdentity`). Work thread pool **grows** to `starrAppCount`. Changed in v1.0.0 (September 2026). +**Starr PUT:** JSON object keyed by slug (letters, digits, `_`, `-`; same charset as roles). `name` is display only. Invalid URL/key on a **PUT-body** instance is **400** after `ParseENV` fills env secrets (so a Save that omits `apiKey` because `UN_*_API_KEY` is set still succeeds). Env-only overlay slugs that startup would skip (URL without key, or the reverse) are dropped from live, not 400 — they cannot block saving other instances. File commit is the PUT body. Live map is that body plus `ParseENV`, so a *complete* env-only extra instance survives `{}`. `path` merges into `paths` without dupes. Last poll `Queue` carries over when `url` + expanded `apiKey` match (`starrIdentity`). Work thread pool **grows** to `starrAppCount`. Changed in v1.0.0 (October 2026). **Folders PUT:** wrapper `{ buffer, folder }`; inner `folder` is a slug map. Per-folder `interval` (default `0s` / off) starts one radovskyb poller for that path. Always `restartRequired: true`. Watcher is built once; rebuilding in-process was rejected (leak / dual poller). diff --git a/go.mod b/go.mod index 2c19ee83..f8bb8a9e 100644 --- a/go.mod +++ b/go.mod @@ -24,7 +24,7 @@ require ( golift.io/rotatorr v0.1.0 golift.io/starr v1.4.1 golift.io/version v0.0.2 - golift.io/xtractr v0.7.0 + golift.io/xtractr v0.7.1-0.20261001061402-3616e46c087b gopkg.in/yaml.v3 v3.0.1 ) diff --git a/go.sum b/go.sum index f6d17849..3fb70ca6 100644 --- a/go.sum +++ b/go.sum @@ -180,8 +180,8 @@ golift.io/udf v0.1.0 h1:arzc1C6Ko9MR2lvpZuZ1WXceGkUxsTOvxg3NIiX6iiA= golift.io/udf v0.1.0/go.mod h1:rw8k1sEHRzKZJCgBFfSPolPb0CpyS0apwoeOwwQQM6o= golift.io/version v0.0.2 h1:i0gXRuSDHKs4O0sVDUg4+vNIuOxYoXhaxspftu2FRTE= golift.io/version v0.0.2/go.mod h1:76aHNz8/Pm7CbuxIsDi97jABL5Zui3f2uZxDm4vB6hU= -golift.io/xtractr v0.7.0 h1:Mznk1RKI7gfLa0BeltA7mhUVLLLjwZCsRYH1mpYOgF0= -golift.io/xtractr v0.7.0/go.mod h1:my82SdhkWFM+6jGnWoxEhKxtcGUmeYsAzmiw2Ku90sE= +golift.io/xtractr v0.7.1-0.20261001061402-3616e46c087b h1:uI2CQFtkBE4Boh3iIm703IafdtlGtUAITAkWv1lOFmU= +golift.io/xtractr v0.7.1-0.20261001061402-3616e46c087b/go.mod h1:my82SdhkWFM+6jGnWoxEhKxtcGUmeYsAzmiw2Ku90sE= google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fwxc= google.golang.org/protobuf v1.36.12/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= diff --git a/pkg/configdef/definitions.yml b/pkg/configdef/definitions.yml index 2977f80a..353909f6 100644 --- a/pkg/configdef/definitions.yml +++ b/pkg/configdef/definitions.yml @@ -586,7 +586,7 @@ sections: ## The following sections can be repeated if you have more than one Sonarr, ## ## Radarr, Lidarr, Readarr, Folder, Webhook, and/or Command Hook. ## ## Identify each instance with a short key: [sonarr.uhd], [folder.tv]. ## - ## Changed in v1.0.0 (September 2026): map keys, not [[array]] list rows. ## + ## Changed in v1.0.0 (October 2026): map keys, not [[array]] list rows. ## ## You MUST uncomment the [header.0], url and api_key for any Starr app. ## ## Uncomment [sonarr.0], [radarr.0], [lidarr.0], and/or [readarr.0] plus the ## ## url and api_key if that app is in use. ## diff --git a/pkg/unpackerr/openapi.json b/pkg/unpackerr/openapi.json index 9c6ee758..04196eae 100644 --- a/pkg/unpackerr/openapi.json +++ b/pkg/unpackerr/openapi.json @@ -342,7 +342,7 @@ }, "StarrInstance": { "type": "object", - "description": "One Sonarr, Radarr, Lidarr, or Readarr instance. The map key is the slug, not this object. name is display only. Changed in v1.0.0 (September 2026).", + "description": "One Sonarr, Radarr, Lidarr, or Readarr instance. The map key is the slug, not this object. name is display only. Changed in v1.0.0 (October 2026).", "properties": { "name": {"type": "string"}, "url": {"type": "string"}, @@ -459,7 +459,7 @@ } }, "ConfigSection": { - "description": "Shape depends on section. Starr (sonarr/radarr/lidarr/readarr) and destination hooks are slug maps. Folders is {buffer, folder} with folder a slug map. hooks is {customIDs, titles}. General and webserver stay objects. anyOf (not oneOf): the object alternatives overlap (optional properties, catch-all object). Changed in v1.0.0 (September 2026).", + "description": "Shape depends on section. Starr (sonarr/radarr/lidarr/readarr) and destination hooks are slug maps. Folders is {buffer, folder} with folder a slug map. hooks is {customIDs, titles}. General and webserver stay objects. anyOf (not oneOf): the object alternatives overlap (optional properties, catch-all object). Changed in v1.0.0 (October 2026).", "anyOf": [ {"$ref": "#/components/schemas/StarrMap"}, {"$ref": "#/components/schemas/FoldersSection"}, @@ -997,7 +997,7 @@ "get": { "tags": ["config"], "summary": "Read the on-disk config section", - "description": "Requires config:{section}:read. Returns file-shaped values (filepath: prefixes kept). Starr, folders.folder, webhooks, and cmdhooks are objects keyed by slug (v1.0.0, September 2026); env-only slugs are omitted. Starr API keys are visible as stored. Unpackerr webserver apiKeys[].key is returned only to callers with *; PUT keeps a blank key of the same name. ui_password is !!cryptd!!, webauth, noauth, or filepath:; plaintext user:pass is blanked. PUT a new password as user: (same digest as login), not plaintext. Use GET /api/config/{section}/live for expanded running values. PUT this payload back to save.", + "description": "Requires config:{section}:read. Returns file-shaped values (filepath: prefixes kept). Starr, folders.folder, webhooks, and cmdhooks are objects keyed by slug (v1.0.0, October 2026); env-only slugs are omitted. Starr API keys are visible as stored. Unpackerr webserver apiKeys[].key is returned only to callers with *; PUT keeps a blank key of the same name. ui_password is !!cryptd!!, webauth, noauth, or filepath:; plaintext user:pass is blanked. PUT a new password as user: (same digest as login), not plaintext. Use GET /api/config/{section}/live for expanded running values. PUT this payload back to save.", "responses": { "200": { "description": "On-disk section payload", @@ -1011,7 +1011,7 @@ "put": { "tags": ["config"], "summary": "Replace one config section and rewrite the TOML file", - "description": "Requires config:{section}:write. Body is the file-shaped GET payload (not /live). Starr/hooks are slug maps; {} clears file instances (legacy [] still loads as keys 0, 1, …). Folders is {buffer, folder}; empty {} is 400, folder may be {}. PUT writes the request body as the file document; live is clone(file)+ParseENV so env-only slugs still appear after {}. Unknown JSON fields, extra values, empty general/webserver objects, and nil list entries return 400. Validation, the TOML write, and the live apply run on the main loop; a persist failure is 500 and leaves runtime unchanged. A filepath: string is expanded for the running process and kept as filepath: on disk. Adding or changing a filepath: is allowed; a missing secret file is 400. Replacing filepath: with a literal value is allowed. Empty apiKeys[].key keeps the existing key of the same name so a redacted GET can round-trip without *. webserver uiPassword on PUT is !!cryptd!!, webauth, noauth, filepath:, or user:<64-char PBKDF2 hex> (the same kdf as login); plaintext user:pass is 400. Changing the live password hash or auth type while local password auth is on requires uiCurrentKdf (KDF of the current username+password). General interval changes reset the running tickers. Folder lists, webserver listen_addr, urlbase, TLS, metrics, pprof, log settings, debug, quiet, parallel, and file modes return restartRequired: true; the daemon then re-execs itself once nothing is queued, extracting, or awaiting delete. In-flight extracts are never cancelled. Changed in v1.0.0 (September 2026).", + "description": "Requires config:{section}:write. Body is the file-shaped GET payload (not /live). Starr/hooks are slug maps; {} clears file instances (legacy [] still loads as keys 0, 1, …). Folders is {buffer, folder}; empty {} is 400, folder may be {}. PUT writes the request body as the file document; live is clone(file)+ParseENV so env-only slugs still appear after {}. Unknown JSON fields, extra values, empty general/webserver objects, and nil list entries return 400. Validation, the TOML write, and the live apply run on the main loop; a persist failure is 500 and leaves runtime unchanged. A filepath: string is expanded for the running process and kept as filepath: on disk. Adding or changing a filepath: is allowed; a missing secret file is 400. Replacing filepath: with a literal value is allowed. Empty apiKeys[].key keeps the existing key of the same name so a redacted GET can round-trip without *. webserver uiPassword on PUT is !!cryptd!!, webauth, noauth, filepath:, or user:<64-char PBKDF2 hex> (the same kdf as login); plaintext user:pass is 400. Changing the live password hash or auth type while local password auth is on requires uiCurrentKdf (KDF of the current username+password). General interval changes reset the running tickers. Folder lists, webserver listen_addr, urlbase, TLS, metrics, pprof, log settings, debug, quiet, parallel, and file modes return restartRequired: true; the daemon then re-execs itself once nothing is queued, extracting, or awaiting delete. In-flight extracts are never cancelled. Changed in v1.0.0 (October 2026).", "requestBody": { "required": true, "content": {"application/json": {"schema": {"$ref": "#/components/schemas/ConfigSectionPut"}}} diff --git a/pkg/unpackerr/start.go b/pkg/unpackerr/start.go index ecf9be2f..4c1beaab 100644 --- a/pkg/unpackerr/start.go +++ b/pkg/unpackerr/start.go @@ -27,7 +27,7 @@ import ( const ( defaultMaxRetries = 2 // two retries after the first try (3 attempts). defaultMaxFiles = 1000 // Starr cap. Folders default to 0 (unlimited). - defaultMaxRatio = 5.0 // Starr cap. Folders default to 0 (unlimited). + defaultMaxRatio = 7.5 // Starr cap. Folders default to 0 (unlimited). defaultSonarrMaxBytes = "20GB" defaultRadarrMaxBytes = "75GB" defaultLidarrMaxBytes = "4GB"