Repository navigation
fix(core): report in-memory file sizes in UTF-8 bytes - #1524
Elon-Jiang wants to merge 2 commits into
Conversation
Co-authored-by: OpenAI Codex <noreply@openai.com>
🦋 Changeset detectedLatest commit: 250fe18 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
There was a problem hiding this comment.
1 issue found across 3 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/core/src/workspace/filesystem/backends/in-memory.ts">
<violation number="1" location="packages/core/src/workspace/filesystem/backends/in-memory.ts:84">
P3: These metadata paths allocate a full UTF-8 byte array for every file just to obtain its length, and `lsInfo`/`globInfo` repeat that allocation across all entries. Use a shared byte-length helper backed by `Buffer.byteLength(content, "utf8")` (or otherwise avoid materializing the encoded bytes) at all three sites.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Re-trigger cubic
| } | ||
|
|
||
| const size = fd.content.join("\n").length; | ||
| const size = new TextEncoder().encode(fd.content.join("\n")).byteLength; |
There was a problem hiding this comment.
P3: These metadata paths allocate a full UTF-8 byte array for every file just to obtain its length, and lsInfo/globInfo repeat that allocation across all entries. Use a shared byte-length helper backed by Buffer.byteLength(content, "utf8") (or otherwise avoid materializing the encoded bytes) at all three sites.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/core/src/workspace/filesystem/backends/in-memory.ts, line 84:
<comment>These metadata paths allocate a full UTF-8 byte array for every file just to obtain its length, and `lsInfo`/`globInfo` repeat that allocation across all entries. Use a shared byte-length helper backed by `Buffer.byteLength(content, "utf8")` (or otherwise avoid materializing the encoded bytes) at all three sites.</comment>
<file context>
@@ -81,7 +81,7 @@ export class InMemoryFilesystemBackend implements FilesystemBackend {
}
- const size = fd.content.join("\n").length;
+ const size = new TextEncoder().encode(fd.content.join("\n")).byteLength;
infos.push({
path: filePath,
</file context>
|
No actionable comments were generated in the recent review. Walkthrough
Merge Risk:
|
Co-authored-by: OpenAI Codex <noreply@openai.com>
PR Checklist
Bugs / Features
@voltagent/core.What is the current behavior?
After
await fs.write("/content.txt", "你好")on the default in-memory workspace filesystem,stat,lsInfo, andglobInforeport size2. The same public calls using NodeFilesystemBackend report6. The memory backend uses JavaScript string length (UTF-16 code units), while filesystem output labels the value as bytes. Byte-limit consumers also receive the smaller value.What is the new behavior?
Compute UTF-8 byte lengths through one shared helper at the three metadata sites. The example reports
6across both backends. UseBuffer.byteLengthwhen available to avoid allocating a full encoded byte array just to obtain its size; retain a TextEncoder fallback for runtimes without Buffer.The regression runs real public write/readRaw/stat/lsInfo/globInfo operations against both memory and temporary on-disk files. Chinese, emoji with a newline and combining marks cover multibyte inputs; ASCII and empty files are controls. Before: 9 fail, 8 pass including two existing tests. After: all 18 pass, including an additional compatibility control with Buffer unavailable. The filesystem/search/skills test group passes 39 tests. No model, network service, or replacement implementation is used in these regressions.
Notes for reviewers
pnpm --filter @voltagent/core build: passed CJS, ESM and declaration builds (Node 24.13.0, pnpm 8.10.5, macOS).git diff --check: passed.pnpm --filter @voltagent/core run test --run --maxWorkers=1 --minWorkers=1: 1,416 passed, 2 skipped, 1 todo; 110 test files passed, no type errors.Summary by cubic
Fixes the in-memory workspace filesystem to report file sizes in UTF-8 bytes, matching the disk backend for non-ASCII content.
stat,lsInfo, andglobInforeported sizes in UTF-16 code units (e.g.,"你好"showed2instead of6).Buffer.byteLengthwhen available, falling back toTextEncoderto stay portable outside Node.Written for commit 250fe18. Summary will update on new commits.
Summary by CodeRabbit