Skip to content

Release SILO Console 2.7.0: usage-worker, plain-HTTP and upload fixes - #87

Merged
Vonng merged 14 commits into
mainfrom
claude/console-2.7.1
Nov 9, 2026
Merged

Vonng merged 14 commits into
mainfrom
claude/console-2.7.1

Conversation

@Vonng

@Vonng Vonng commented Nov 9, 2026 •

Copy link
Copy Markdown
Member

Release SILO Console 2.7.0 with the fixes found in its final pre-release review. The earlier v2.7.0 tag (26ce922) only ever produced an unpublished Draft Release and candidate image; after this PR lands, the draft and tag are replaced and v2.7.0 is recut on this branch's last commit. The candidate image was never pulled outside release verification, and latest never pointed to it.

Fixes

  • Plain HTTP: "Calculate folder sizes" and the decryption-key copy now work when Console is opened over plain HTTP. The common embedded URL http://<ip>:9001 is not a secure context, so crypto.randomUUID and navigator.clipboard do not exist there.
  • Usage worker TLS: console usage-worker accepts --certs-dir. It trusts certs/CAs and Console's public.crt, and applies CONSOLE_MINIO_SERVER_TLS_SKIP_VERIFY to the SILO origin only. The worker neither creates the directory nor loads private.key. console server certificate loading is unchanged.
  • Scan budget: SCAN_TIMEOUT now counts listing time summed across slices, not time spent queued behind other buckets. The exhausted budget is recorded on every duration exit.
  • Worker shutdown: a scan interrupted by shutdown, including one stopped during a database write, goes back to the queue with its checkpoint. It is no longer shown as error with a 30 s backoff.
  • Bucket deletion: the worker handles s3:BucketRemoved and s3:BucketCreated. A deleted bucket's totals are forgotten, so a bucket re-created under the same name never shows them. If the deletion cannot be persisted at once, it is kept and retried.
  • Busy scanner: without the usage database, Refresh retries an HTTP 429 with backoff before marking a bucket as failed.
  • Multipart uploads: malformed or truncated multipart uploads return an error instead of a nil-part panic. Only an unwrapped io.EOF ends the loop, because the standard library reports a truncated body as a wrapped EOF.
  • Hardening:
    • A canceled bucket-usage authorization probe is never treated as authorized.
    • The share-link expiry no longer depends on Luxon's locale. This is not a user-facing bug today: Luxon formats as en-US unless a locale is set.
  • Release metadata: the fixes are recorded in the v2.7.0 CHANGELOG section, and the source docs no longer use draft or candidate wording. The version and embedded assets stay at 2.7.0.

Verification (local)

  • Go: build, vet and go test ./... pass. golangci-lint reports 0 issues. The CREDITS, replacement and deps-release structural checks pass.
  • PostgreSQL: the usage integration suite (-tags=integration) passes against PostgreSQL 18.
  • Frontend:
    • Unit tests: 792/792 passed in the first full run. A later run had one failure in ff-controls.unit.ts, a file this PR does not touch, which passed three times in isolation.
    • tsc, prettier, knip and the bundle budgets pass.
    • The embedded assets rebuild byte for byte.
  • Regression tests: each new test fails on the previous code. Exceptions: the budget-race test fails with high probability rather than deterministically, and the shutdown-during-write test injects the canceled error directly.
  • Review: an adversarial Codex review asked for changes in round 1 (7 items). After the fixes, round 2 returned APPROVE WITH NITS. The two remaining nits are P3 test-coverage suggestions.

Upgrade note

During a rolling upgrade, stop older usage workers before starting 2.7.0 workers. A mixed fleet does not accumulate the new listing-time budget consistently.

🤖 Generated with Claude Code

Vonng and others added 13 commits November 9, 2026 17:44
Embedded Console is usually opened at http://<ip>:9001, which is not a
secure context. There crypto.randomUUID and navigator.clipboard are
undefined, so "Calculate folder sizes" threw before sending its request
and the decryption-key copy button did nothing, both without feedback.

Use the shared makeid() task-ID helper, and copy the key with
copy-to-clipboard (already shipped through react-copy-to-clipboard),
keeping the field's single overlay button. Add a static guard against
secure-context-only APIs and exercise the real Inspect dialog's copy on
a plain-HTTP origin.

Signed-off-by: Feng Ruohang <rh@vonng.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DaysSelector formatted the expiry with Luxon toFormat and ShareFile
parsed it back with new Date(). toFormat falls back to en-US unless a
locale is specified, which Console does not do, so links work today.
A configured Luxon default locale such as ar-EG, ar-SA or fa, however,
would emit native digits that new Date() rejects, and the share link
would silently never be generated.

Produce the timestamp with Luxon's technical ISO output, which never
depends on the locale and is unchanged for every locale today, and
guard it with a test.

Signed-off-by: Feng Ruohang <rh@vonng.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Without the usage database, Refresh scans buckets two at a time through
prefix-usage, whose slots (two by default) are shared by every Console
session. When another tab, user or folder scan held a slot, the server
answered 429 and the bucket was immediately shown as "Update failed".

Treat 429 as busy: wait with jittered, capped backoff (about 30 s in
total), honour cancellation, and retry the same bucket before reporting
a failure. Other status mappings are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
uploadFiles only stopped on an unwrapped io.EOF from NextPart. Any
other error (a malformed part header, an empty or boundary-less body,
or a body truncated after a boundary, which the standard library
reports as a wrapped EOF) left the part nil and p.FormName() panicked
inside the request handler.

Return those errors to the caller. A wrapped EOF is deliberately not
treated as the closing boundary, so a truncated upload is not accepted
as complete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
console usage-worker built its scan client and change-stream observer
on minio-go's default transport, so it trusted only the system roots.
With a private CA in certs/CAs, console server connected to SILO while
the worker failed with "certificate signed by unknown authority", and
the worker ignored CONSOLE_MINIO_SERVER_TLS_SKIP_VERIFY.

Give the worker the server's --certs-dir flag. It resolves the
directory without creating it and loads only the trusted roots (system,
certs/CAs and public.crt, never private.key), so a read-only or
key-protected certs directory cannot stop it; console server's loading
is unchanged. Pass the SILO endpoint transport, including its
origin-scoped skip-verify policy, through a new usage.Config.Transport
field used by both the scanner and the observer.

Signed-off-by: Feng Ruohang <rh@vonng.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The worker bounded each bucket scan by the wall clock since the scan
started. Fair scheduling yields a large bucket after every slice and
puts it behind the others, so that clock also ran while the bucket
waited for other buckets' slices. Once the buckets' combined listing
time exceeded SCAN_TIMEOUT x CONCURRENT_SCANS they all ended
limited/duration, although each one alone finished well within the
budget. A limited checkpoint kept the expired start time, so automatic
retries ended limited again at once, and only a manual Refresh cleared
it, which requeued every bucket into the same race.

Record the listing time spent so far in the checkpoint
(listing_time_ns) and give each slice the remaining budget. A runaway
listing is still bounded; time spent queued is no longer charged.
Checkpoints written by earlier workers have no such field and resume
with a full budget, so buckets left limited by this bug recover on
their next retry; earlier workers ignore the new field.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
When the worker stopped, every scan in progress was recorded through
Fail as state=error/worker_shutdown with a retry backoff of at least
RETRY_INTERVAL (30 s), so a rolling restart showed those buckets as
"Update failed" and delayed their completion.

Add Store.Release, which returns the leased job to the queue with its
checkpoint and progress, ready immediately, without an error code or
an attempt, and use it for worker_shutdown. Lease loss is still
ignored. Also give the queue-wait budget test a wider timing margin.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
The worker subscribed only to object events. Force-deleting a bucket
removes its objects without per-object events and emits only
s3:BucketRemoved, and incremental reconciliation leaves existing rows
alone. A bucket re-created under the same name, possibly by another
owner, therefore showed the deleted bucket's totals until an object
event, a worker restart or the 24-hour full reconciliation.

Subscribe to s3:BucketCreated and s3:BucketRemoved. On removal, clear
the row's results and checkpoint, fence any scan in progress by
dropping its lease, and queue a rescan, which either retires the row
(MissingBucket) or measures the re-created bucket. Coalescing keeps a
deletion even when a later event for the same bucket arrives in the
same batch. Root ListenNotification permission already covers bucket
events, so the worker policy is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
authorizeBucketUsage treated a listing channel that closed without an
object as an authorized, empty bucket. minio-go also closes the channel
without an error when the context is canceled, and select may take
that branch before ctx.Done(), so a canceled request could pass the
check. It is safe today only because the later database calls share the
canceled context and fail.

Return ctx.Err() when the channel closes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
When the budget runs out, the listing channel closes as the scan context
expires and select may take either case. The closed-channel exit returned
limited/duration without saving the checkpoint, so the database kept the
listing time of the last progress write and an automatic retry listed
again instead of ending at once. Persist the checkpoint there as well, as
the other duration exits do, and cover the race with a busy lister.

Signed-off-by: Feng Ruohang <rh@vonng.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A shutdown that cancels a checkpoint, yield or completion write makes
RunSlice return the database error rather than worker_shutdown, so the
job skipped Release and stayed running until its lease expired (about
two minutes by default). Release whenever the worker's context is
canceled; the lease guard keeps it a no-op for a job already completed,
yielded or taken over. Move the settlement into settleScan to test it
deterministically.

Signed-off-by: Feng Ruohang <rh@vonng.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Received changes that failed to persist were restored into a map owned
by one observer lease and dropped when that observer exited. Reconnect
reconciliation re-queues every bucket but cannot tell that a bucket was
deleted and re-created, so a deletion lost to a transient database
failure left a limited row's old totals in place. Keep the pending
changes for the life of the worker and flush them on every observer
loop, with or without the lease.

Signed-off-by: Feng Ruohang <rh@vonng.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rebuild web-app/build with Node 24.22.1 for the plain-HTTP, share-link
expiry and busy-scanner fixes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Record the pre-release fixes in the v2.7.0 CHANGELOG section, join its
list, and describe the source as 2.7.0 without draft or candidate
wording. The v2.7.0 tag is recut on this commit: the earlier tag on
26ce922 only ever produced an unpublished draft. The embedded assets
already carry version 2.7.0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
@Vonng
Vonng force-pushed the claude/console-2.7.1 branch from 842734a to 2aae2c1 Compare November 9, 2026 11:25
@Vonng Vonng changed the title Release SILO Console 2.7.1: usage-worker, plain-HTTP and upload fixes Release SILO Console 2.7.0: usage-worker, plain-HTTP and upload fixes Nov 9, 2026
@Vonng
Vonng merged commit 2aae2c1 into main Nov 9, 2026
77 of 79 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant