Skip to content

fix: preserve large prefix usage totals - #285

Closed
houseme wants to merge 1 commit into
mainfrom
houseme/fix/prefix-summary-large-values
Closed

houseme wants to merge 1 commit into
mainfrom
houseme/fix/prefix-summary-large-values

Conversation

@houseme

@houseme houseme commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Pull Request

Description

A folder calculation could fail when the total exceeded JavaScript's safe integer limit, even when the object count was within the existing scan limit. For example, 2,048 objects of 4 TiB each exceed that limit. Accumulation now retains the fast number path for safe totals and switches to BigInt before precision is lost. Human-readable formatting and mixed number/BigInt sorting support large totals.

The scan remains bounded to 100 pages or 100,000 current objects; this change does not provide billion-object statistics. The separate near-real-time backend index work is tracked in rustfs/backlog#2990.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • Test improvements

Testing

  • Unit tests added/updated
  • Manual testing completed

At commit 8163153dc2bf74c5e7bd1eec67234bc783cd168d, Node 22.23.4 / pnpm 12.3.4:

corepack pnpm install --frozen-lockfile
corepack pnpm type-check
corepack pnpm lint
corepack pnpm format:check
corepack pnpm test:run
git diff --check

All passed; the full suite ran 891 tests. Coverage includes the safe-integer boundary, invalid object sizes, 2,048 large objects, formatting and ascending/descending mixed-type TanStack sorting. Independent review found no supported correctness issue.

In an isolated headless Chrome run, 100 pages of synthetic S3 XML with 1,000 objects per page were processed, the page displayed 488.3 PiB, and cancellation was checked separately with an intentionally delayed request. This is not a production load test or a billion-object measurement. The browser's sampled JS heap and timing are not guaranteed peak or production performance measurements.

Checklist

  • Code follows the project's style guidelines
  • Self-review completed
  • TypeScript types are properly defined
  • All commit messages are in English (Conventional Commits)
  • All existing tests pass
  • No new dependencies added, or they are justified

Related Issues

Related to rustfs/rustfs#9082 and rustfs/backlog#2990.

Screenshots (if applicable)

Same synthetic source and desktop viewport: the baseline fails after the second page; the fix completes 100 pages. The screenshots use test data only.

Before After
Before After

Mobile result.

Additional Notes

The displayed unit remains rounded to one decimal place; totals and sorting are kept precise. Large-scale near-real-time indexing requires separate bucket/prefix authorization, completeness, persistence and recovery validation before it can replace this bounded calculation.

Promote prefix totals beyond the safe integer range without losing bytes, and preserve correct display and sorting.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
@cxymds cxymds closed this Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants