Skip to content

fix(core): report in-memory file sizes in UTF-8 bytes - #1524

Open
Elon-Jiang wants to merge 2 commits into
VoltAgent:mainfrom
Elon-Jiang:fix/in-memory-file-byte-sizes
Open

Elon-Jiang wants to merge 2 commits into
VoltAgent:mainfrom
Elon-Jiang:fix/in-memory-file-byte-sizes

Conversation

@Elon-Jiang

@Elon-Jiang Elon-Jiang commented Nov 10, 2026 •

Copy link
Copy Markdown

PR Checklist

  • The commit message follows the contribution guidelines.

Bugs / Features

  • Related issue(s) linked — no existing issue found; discovered through the public WorkspaceFilesystem API.
  • Tests for the changes have been added.
  • Docs added / updated — no new API; restores the existing byte-size behavior displayed by filesystem tools and provided by the disk backend.
  • Changeset added for @voltagent/core.

What is the current behavior?

After await fs.write("/content.txt", "你好") on the default in-memory workspace filesystem, stat, lsInfo, and globInfo report size 2. The same public calls using NodeFilesystemBackend report 6. The memory backend uses JavaScript string length (UTF-16 code units), while filesystem output labels the value as bytes. Byte-limit consumers also receive the smaller value.

What is the new behavior?

Compute UTF-8 byte lengths through one shared helper at the three metadata sites. The example reports 6 across both backends. Use Buffer.byteLength when available to avoid allocating a full encoded byte array just to obtain its size; retain a TextEncoder fallback for runtimes without Buffer.

The regression runs real public write/readRaw/stat/lsInfo/globInfo operations against both memory and temporary on-disk files. Chinese, emoji with a newline and combining marks cover multibyte inputs; ASCII and empty files are controls. Before: 9 fail, 8 pass including two existing tests. After: all 18 pass, including an additional compatibility control with Buffer unavailable. The filesystem/search/skills test group passes 39 tests. No model, network service, or replacement implementation is used in these regressions.

Notes for reviewers

  • pnpm --filter @voltagent/core build: passed CJS, ESM and declaration builds (Node 24.13.0, pnpm 8.10.5, macOS).
  • Standalone probe against the actual built package: all five inputs report the expected byte sizes through all three interfaces and read back unchanged.
  • Biome on both changed TS files, Prettier on the changeset and git diff --check: passed.
  • pnpm --filter @voltagent/core run test --run --maxWorkers=1 --minWorkers=1: 1,416 passed, 2 skipped, 1 todo; 110 test files passed, no type errors.
  • Focused all-state searches, all 108 open PRs and their complete file lists were checked. The same-file mkdir fix in fix(core): preserve recursive mkdir parent directories #1452 changes directory bookkeeping, not size calculation.
  • Investigation, implementation, agent self-review and validation used OpenAI Codex.

Summary by cubic

Fixes the in-memory workspace filesystem to report file sizes in UTF-8 bytes, matching the disk backend for non-ASCII content.

  • Previously stat, lsInfo, and globInfo reported sizes in UTF-16 code units (e.g., "你好" showed 2 instead of 6).
  • Size computation now uses Buffer.byteLength when available, falling back to TextEncoder to stay portable outside Node.
  • Adds regression tests covering Chinese, emoji, combining marks, ASCII, and empty files across both backends.

Written for commit 250fe18. Summary will update on new commits.

View guided diff

Summary by CodeRabbit

  • Bug Fixes
    • In-memory workspace file sizes now report UTF-8 byte counts, matching disk-backed filesystems. File details, directory listings, and glob results show consistent sizes for non-ASCII content, including emoji and accented characters, as well as empty files and files containing newlines. This makes reported sizes consistent regardless of which workspace filesystem is in use.

Co-authored-by: OpenAI Codex <noreply@openai.com>
@changeset-bot

changeset-bot Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 250fe18

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
@voltagent/core Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 3 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/core/src/workspace/filesystem/backends/in-memory.ts">

<violation number="1" location="packages/core/src/workspace/filesystem/backends/in-memory.ts:84">
P3: These metadata paths allocate a full UTF-8 byte array for every file just to obtain its length, and `lsInfo`/`globInfo` repeat that allocation across all entries. Use a shared byte-length helper backed by `Buffer.byteLength(content, "utf8")` (or otherwise avoid materializing the encoded bytes) at all three sites.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic

}

const size = fd.content.join("\n").length;
const size = new TextEncoder().encode(fd.content.join("\n")).byteLength;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: These metadata paths allocate a full UTF-8 byte array for every file just to obtain its length, and lsInfo/globInfo repeat that allocation across all entries. Use a shared byte-length helper backed by Buffer.byteLength(content, "utf8") (or otherwise avoid materializing the encoded bytes) at all three sites.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/core/src/workspace/filesystem/backends/in-memory.ts, line 84:

<comment>These metadata paths allocate a full UTF-8 byte array for every file just to obtain its length, and `lsInfo`/`globInfo` repeat that allocation across all entries. Use a shared byte-length helper backed by `Buffer.byteLength(content, "utf8")` (or otherwise avoid materializing the encoded bytes) at all three sites.</comment>

<file context>
@@ -81,7 +81,7 @@ export class InMemoryFilesystemBackend implements FilesystemBackend {
       }
 
-      const size = fd.content.join("\n").length;
+      const size = new TextEncoder().encode(fd.content.join("\n")).byteLength;
       infos.push({
         path: filePath,
</file context>

@coderabbitai

coderabbitai Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

In-memory workspace filesystem metadata now reports file sizes as UTF-8 byte counts. Tests compare stat, lsInfo, and globInfo results with the disk backend across several content types and check behavior when Buffer is unavailable.

Changes: Filesystem byte-size reporting

Layer / File(s) Summary
UTF-8 byte counts across filesystem metadata
packages/core/src/workspace/filesystem/backends/in-memory.ts, packages/core/src/workspace/filesystem/index.spec.ts, .changeset/in-memory-file-byte-sizes.md
lsInfo, stat, and globInfo now report UTF-8 byte counts for in-memory files. Tests check sizes for ASCII, empty, Chinese, newline and emoji, and combining-mark content across memory and disk backends. They also check the fallback when Buffer is unavailable. A patch changeset records the update.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Suggested reviewers: omeraplak

Merge Risk: Minimal · up to 250fe

File metadata now reports UTF-8 byte sizes. No material merge risk was identified.

Pre-merge checks: Passed 5

Check name Status Explanation
Title check Passed The title clearly and concisely describes the main change: in-memory file sizes are reported in UTF-8 bytes.
Description check Passed The description covers the checklist, current and new behavior, tests, changeset, and reviewer notes. It explains why no issue or documentation update is included.
Docstring Coverage Passed No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 2 files.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 68eac7c4-50d9-4958-b7d7-03f7d766ecfe
Commits

Reviewing files that changed from the base of the PR and between 3f02ef6 and 250fe18.

Files selected for processing: 2

  • packages/core/src/workspace/filesystem/backends/in-memory.ts
  • packages/core/src/workspace/filesystem/index.spec.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 8 remain after this review.

Co-authored-by: OpenAI Codex <noreply@openai.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant