diff --git a/docs/README.md b/docs/README.md index 21c953d23..0d38473ea 100644 --- a/docs/README.md +++ b/docs/README.md @@ -13,6 +13,10 @@ layered on the runtime's `EventTarget`, the GC contract (weak timers and `any()` links, listener-driven persistence), and the `DOMException` stand-in (name-patched `Error` reasons). +- [TextEncoder / TextDecoder and atob / btoa](text-encoding.md) — the WHATWG + encoding and base64 globals (`TextEncoder`, `TextDecoder`, `atob`, `btoa`), + the supported encodings with their label sets, streaming decode semantics, + and the lazy-global tier that runs their builtins only on first use. - [Error handling](error-handling.md) — global `error`/`unhandledrejection` events, `reportError`, catching Java exceptions in JS (`error.nativeException`), forwarding JS throws to Java callers (`interop.escapeException`), JS stacks on Java exceptions (`com.tns.JavaScriptStackTrace`), configuration flags, and crash-reporter integration. - [structuredClone](structured-clone.md) — the WHATWG `structuredClone(value, { transfer })` global: what clones, how graph identity and cycles are preserved, `ArrayBuffer` transfer, and the `DataCloneError`-named `Error` that stands in for `DOMException`. - [Implementing additional Chrome DevTools protocol Domains](extending-inspector.md) diff --git a/docs/ns-builtin-modules.md b/docs/ns-builtin-modules.md index c7e48c10e..ce4a5290d 100644 --- a/docs/ns-builtin-modules.md +++ b/docs/ns-builtin-modules.md @@ -59,6 +59,7 @@ Rules: |---|---| | `inspect(value[, options])` | Formats any value for human consumption: depth-limited, output-capped, cycle-safe, never invokes getters (except a guarded `error.stack` read and custom `toString` overrides, which are honored). `options.depth` (number) overrides the default depth of 2. Other option keys are reserved. | | `format(fmt, ...args)` | Node-style printf formatting: `%s`, `%d`, `%i`, `%f`, `%j`, `%o`, `%O`, `%%`. Extra arguments are appended space-separated, objects rendered via `inspect`. When `fmt` is not a string or contains no substitutions, all arguments are formatted and joined with spaces. `console.*` routes its arguments through this, so `console.log("%d apples", 3)` works. | +| `TextEncoder` / `TextDecoder` | The WHATWG encoding interfaces, **the very same class objects the globals of those names hold** (`require("ns:util").TextDecoder === globalThis.TextDecoder`). Reading either member is what materializes them, so requiring the module costs nothing extra. | ```js const { inspect, format } = require("ns:util"); @@ -78,6 +79,13 @@ format("%j", { ok: true }); // '{"ok":true}' format("100% sure", "extra"); // "100% sure extra" (no placeholder consumed) ``` +```js +const { TextEncoder, TextDecoder } = require("ns:util"); + +TextDecoder === globalThis.TextDecoder; // true +new TextDecoder().decode(new TextEncoder().encode("héllo")); // "héllo" +``` + **Stability caveat (verbatim from Node's contract):** the output of `inspect` (and therefore `format`'s object rendering) may change between runtime versions for readability; it is intended for humans and must not be parsed @@ -400,7 +408,7 @@ unmodified where a shim exists: | module | exports | notes | |---|---|---| -| `node:util` | `inspect`, `format` | Re-exports `ns:util`'s members unchanged (`nodeUtil.inspect === nsUtil.inspect`) from a **distinct, separately frozen module object**. Documented as partial. | +| `node:util` | `inspect`, `format`, `TextEncoder`, `TextDecoder` | Re-exports `ns:util`'s members unchanged (`nodeUtil.inspect === nsUtil.inspect`) from a **distinct, separately frozen module object**. `TextEncoder`/`TextDecoder` are the globals of those names, as they are in Node. Documented as partial. | | `node:url` | `fileURLToPath`, `pathToFileURL` | Node-strict converters between `file:` URLs and paths. Documented as partial — no `URL`/`URLSearchParams` re-exports (both are globals), no legacy `url.parse`/`format`/`resolve`. | | `node:module` | `createRequire` | Re-exports `ns:module`'s `createRequire` unchanged from a **distinct, separately frozen module object**. `createPumpingRequire` is deliberately absent: it has no Node counterpart, so code written against this shim keeps running on Node. `require.resolve`/`.cache`/`.main` are not implemented, and neither is any other `node:module` member (`Module`, `builtinModules`, `isBuiltin`, `register`, `syncBuiltinESMExports`). Documented as partial. | diff --git a/docs/text-encoding.md b/docs/text-encoding.md new file mode 100644 index 000000000..163a6032d --- /dev/null +++ b/docs/text-encoding.md @@ -0,0 +1,70 @@ +# TextEncoder / TextDecoder and atob / btoa + +Native, WHATWG-conformant `TextEncoder`, `TextDecoder` +([Encoding Standard](https://encoding.spec.whatwg.org)) and `atob` / `btoa` +([HTML Standard §8.3](https://html.spec.whatwg.org/multipage/webappapis.html#atob)) +globals, and the **lazy-global tier** they ride on. + +## Lazy globals + +These globals are registered on the global template as lazy data properties +(`LazyGlobals`, `test-app/runtime/src/main/cpp/LazyGlobals.cpp`): the builtin +behind a name is not compiled, run, or allocated until app code first reads it, +and V8 then replaces the property with a plain data property so later reads +cost nothing. Sibling names from one builtin (`TextEncoder` + `TextDecoder`) +share a single run per isolate. Workers get the same globals — the tier is +registered in every isolate's template. Assigning over one of these names +before its first read replaces the global, like any other writable global. + +The tier is the intended home for further web globals (`Blob`, `fetch`, +`crypto`, `DOMException`, …) with zero cost when unused; see +`test-app/runtime/src/main/cpp/js/README.md` for the rules a lazy builtin +lives by. + +The per-isolate exports cache behind the tier (`BuiltinLoader::GetExports`) is +shared with the `ns:`/`node:` module registry: `require("ns:util").TextDecoder` +and `require("node:util").TextDecoder` are the very class objects the globals +hold, whichever entry point is reached first +(see [ns-builtin-modules](ns-builtin-modules.md)). + +## TextEncoder / TextDecoder + +Node's split: `js/text-encoding.js` owns the WebIDL surface (brand checks via +private fields, enumerable prototype members, `Symbol.toStringTag`), +`TextEncoding.cpp` owns the bytes. + +- **Decoder encodings**: the `TextDecoder` constructor resolves utf-8, + utf-16le, utf-16be and windows-1252, each with its complete WHATWG label + set; an unknown label throws `RangeError`. (Precedent: Node without ICU + ships utf-8/utf-16le; utf-16be and windows-1252 are cheap, and windows-1252 + covers the `ascii`/`latin1`/`iso-8859-1` aliases web code actually uses.) + `TextEncoder` is UTF-8-only and takes no label, as the spec defines it. +- **Streaming**: full `decode(…, { stream: true })` support. Incomplete + sequences (split BOMs and split utf-16 code units included) carry across + calls in a 16-byte `Uint8Array` the builtin owns — no per-instance native + handle, no finalizer. +- **Replacement semantics**: WHATWG utf-8 state machine with one U+FFFD per + maximal invalid subpart; `fatal: true` throws `TypeError`; `ignoreBOM` + honored. +- `encode()` / `encodeInto()` with correct USV conversion and partial-write + boundaries (`encodeInto` never splits an encoded code point). +- **Fast paths**: pure-ASCII utf-8 and C1-free windows-1252 decode straight + through `String::NewFromOneByte`; results downgrade to one-byte strings when + possible. `encodeInto` registers a V8 Fast API overload + (`NATIVESCRIPT_ENABLE_FAST_API`, default on), live once a call site tiers + up. + +## atob / btoa + +WHATWG forgiving-base64 (`Base64.cpp`): whitespace stripping, padding rules, +alphabet validation. With no `DOMException` in the runtime yet, failures throw +the name-patched `Error` (`InvalidCharacterError`) stand-in the abort-signal +and performance builtins already use; a follow-up will introduce +`DOMException` and upgrade these. + +## Tests + +The shared suite (`test-app/app/src/main/assets/app/shared/TextEncoding`) +holds the conformance specs, feature-detecting so runtimes without these +globals report pending rather than failing; it was independently validated +against Node 24 (full ICU) as a reference. diff --git a/eslint.config.mjs b/eslint.config.mjs index ab05c8e6f..a1bd9d493 100644 --- a/eslint.config.mjs +++ b/eslint.config.mjs @@ -32,7 +32,7 @@ const capturedStatics = [ // Captured constructors. A destructure from `primordials` shadows the global, // so these only fire on the unguarded reference. -const restrictedGlobals = ['Date', 'FinalizationRegistry', 'Map', 'Number', 'Proxy', 'RangeError', 'Set', 'String', 'TypeError', 'WeakRef'].map((name) => ({ +const restrictedGlobals = ['Date', 'FinalizationRegistry', 'Map', 'Number', 'Proxy', 'RangeError', 'Set', 'String', 'TypeError', 'Uint8Array', 'Uint32Array', 'WeakRef'].map((name) => ({ name, message: `Destructure ${name} from primordials — builtins must not read intrinsics off globals user code can replace.`, })); diff --git a/test-app/app/src/main/assets/app/mainpage.js b/test-app/app/src/main/assets/app/mainpage.js index 4a15c9440..068a7a782 100644 --- a/test-app/app/src/main/assets/app/mainpage.js +++ b/test-app/app/src/main/assets/app/mainpage.js @@ -20,6 +20,7 @@ shared.runRuntimeTests(); shared.runWorkerTests(); shared.runPerformanceTests(); shared.runStructuredCloneTests(); +shared.runTextEncodingTests(); require("./tests/testWebAssembly"); require("./tests/testEventLoop"); require("./tests/testMultithreadedJavascript"); diff --git a/test-app/app/src/main/assets/app/shared b/test-app/app/src/main/assets/app/shared index 0baab7cce..364cba6f2 160000 --- a/test-app/app/src/main/assets/app/shared +++ b/test-app/app/src/main/assets/app/shared @@ -1 +1 @@ -Subproject commit 0baab7cceaca2bb5fdb7b697c08b19be1e46d925 +Subproject commit 364cba6f26f540a47e3c62a9029135218851f5a1 diff --git a/test-app/app/src/main/assets/app/tests/nsUtilEncodingOrderWorker.js b/test-app/app/src/main/assets/app/tests/nsUtilEncodingOrderWorker.js new file mode 100644 index 000000000..3a7e6c702 --- /dev/null +++ b/test-app/app/src/main/assets/app/tests/nsUtilEncodingOrderWorker.js @@ -0,0 +1,28 @@ +// A worker is a fresh isolate, which is what makes the access order testable: +// the parent realm has already materialized TextEncoder/TextDecoder by the +// time any spec runs. Nothing here may touch either name before the handler, +// or the requested order is lost. +onmessage = function (msg) { + var order = msg.data; + var results = { order: order }; + + if (order === "global-first") { + var globalEncoder = globalThis.TextEncoder; + var globalDecoder = globalThis.TextDecoder; + var nsUtil = require("ns:util"); + var nodeUtil = require("node:util"); + results.encoder = nsUtil.TextEncoder === globalEncoder && nodeUtil.TextEncoder === globalEncoder; + results.decoder = nsUtil.TextDecoder === globalDecoder && nodeUtil.TextDecoder === globalDecoder; + results.roundTrip = new nsUtil.TextDecoder().decode(new nodeUtil.TextEncoder().encode("ok")); + } else { + var util = require("ns:util"); + var node = require("node:util"); + var utilEncoder = util.TextEncoder; + var utilDecoder = node.TextDecoder; + results.encoder = globalThis.TextEncoder === utilEncoder && node.TextEncoder === utilEncoder; + results.decoder = globalThis.TextDecoder === utilDecoder && util.TextDecoder === utilDecoder; + results.roundTrip = new node.TextDecoder().decode(new util.TextEncoder().encode("ok")); + } + + postMessage(results); +}; diff --git a/test-app/app/src/main/assets/app/tests/testNsUtil.js b/test-app/app/src/main/assets/app/tests/testNsUtil.js index acf05c0ff..77689907c 100644 --- a/test-app/app/src/main/assets/app/tests/testNsUtil.js +++ b/test-app/app/src/main/assets/app/tests/testNsUtil.js @@ -12,6 +12,49 @@ describe("ns:util", function () { expect(require("ns:util")).toBe(util); }); + it("exposes the encoding interfaces the globals expose", function () { + expect(typeof util.TextEncoder).toBe("function"); + expect(typeof util.TextDecoder).toBe("function"); + // One run of the text-encoding builtin backs both entry points, so the + // classes are identical objects no matter which is reached first. + expect(util.TextEncoder).toBe(globalThis.TextEncoder); + expect(util.TextDecoder).toBe(globalThis.TextDecoder); + }); + + it("round trips text through the module's encoding interfaces", function () { + var bytes = new util.TextEncoder().encode("héllo"); + expect(bytes instanceof Uint8Array).toBe(true); + expect(bytes.length).toBe(6); + expect(new util.TextDecoder().decode(bytes)).toBe("héllo"); + }); + + it("keeps the classes identical in a fresh isolate, whichever is touched first", function (done) { + var orders = ["global-first", "util-first"]; + var replies = 0; + orders.forEach(function (order) { + var worker = new Worker("./nsUtilEncodingOrderWorker.js"); + worker.onmessage = function (msg) { + expect(msg.data).toEqual({ + order: order, + encoder: true, + decoder: true, + roundTrip: "ok", + }); + worker.terminate(); + replies++; + if (replies === orders.length) { + done(); + } + }; + worker.onerror = function (error) { + fail("worker (" + order + ") failed: " + error.message); + worker.terminate(); + done(); + }; + worker.postMessage(order); + }); + }); + it("throws for an unknown builtin", function () { expect(function () { require("ns:definitely-not-a-module"); @@ -158,6 +201,15 @@ describe("node:util", function () { expect(Object.isFrozen(nodeUtil)).toBe(true); expect(nodeUtil.inspect).toBe(util.inspect); expect(nodeUtil.format).toBe(util.format); + expect(nodeUtil.TextEncoder).toBe(util.TextEncoder); + expect(nodeUtil.TextDecoder).toBe(util.TextDecoder); + }); + + it("exposes Node's encoding interfaces, identical to the globals", function () { + expect(Object.keys(nodeUtil).sort()).toEqual(["TextDecoder", "TextEncoder", "format", "inspect"]); + expect(nodeUtil.TextEncoder).toBe(globalThis.TextEncoder); + expect(nodeUtil.TextDecoder).toBe(globalThis.TextDecoder); + expect(new nodeUtil.TextDecoder("utf-8").decode(new nodeUtil.TextEncoder().encode("ok"))).toBe("ok"); }); it("is a singleton per realm", function () { diff --git a/test-app/runtime/CMakeLists.txt b/test-app/runtime/CMakeLists.txt index 0d641a1b3..69536da1f 100644 --- a/test-app/runtime/CMakeLists.txt +++ b/test-app/runtime/CMakeLists.txt @@ -69,6 +69,7 @@ include_directories( set(RUNTIME_BUILTIN_JS_DIR ${PROJECT_SOURCE_DIR}/src/main/cpp/js) set(RUNTIME_BUILTIN_JS ${RUNTIME_BUILTIN_JS_DIR}/abort-signal.js + ${RUNTIME_BUILTIN_JS_DIR}/base64.js ${RUNTIME_BUILTIN_JS_DIR}/blob-url.js ${RUNTIME_BUILTIN_JS_DIR}/error-events.js ${RUNTIME_BUILTIN_JS_DIR}/events.js @@ -84,6 +85,7 @@ set(RUNTIME_BUILTIN_JS ${RUNTIME_BUILTIN_JS_DIR}/primordials.js ${RUNTIME_BUILTIN_JS_DIR}/require-factory.js ${RUNTIME_BUILTIN_JS_DIR}/structured-clone.js + ${RUNTIME_BUILTIN_JS_DIR}/text-encoding.js ${RUNTIME_BUILTIN_JS_DIR}/weak-ref.js ) set(RUNTIME_BUILTINS_GENERATED_DIR ${PROJECT_SOURCE_DIR}/src/main/cpp/generated) @@ -168,6 +170,7 @@ add_library( src/main/cpp/ArrayElementAccessor.cpp src/main/cpp/ArrayHelper.cpp src/main/cpp/AssetExtractor.cpp + src/main/cpp/Base64.cpp src/main/cpp/BuiltinLoader.cpp src/main/cpp/CallbackHandlers.cpp src/main/cpp/ConcurrentQueue.cpp @@ -189,6 +192,7 @@ add_library( src/main/cpp/JsArgConverter.cpp src/main/cpp/JsArgToArrayConverter.cpp src/main/cpp/JSONObjectHelper.cpp + src/main/cpp/LazyGlobals.cpp src/main/cpp/Logger.cpp src/main/cpp/ManualInstrumentation.cpp src/main/cpp/MetadataMethodInfo.cpp @@ -215,6 +219,7 @@ add_library( src/main/cpp/SimpleProfiler.cpp src/main/cpp/StructuredClone.cpp src/main/cpp/StructuredSerialization.cpp + src/main/cpp/TextEncoding.cpp src/main/cpp/Util.cpp src/main/cpp/V8GlobalHelpers.cpp src/main/cpp/V8StringConstants.cpp diff --git a/test-app/runtime/src/main/cpp/Base64.cpp b/test-app/runtime/src/main/cpp/Base64.cpp new file mode 100644 index 000000000..76a192e59 --- /dev/null +++ b/test-app/runtime/src/main/cpp/Base64.cpp @@ -0,0 +1,185 @@ +#include "Base64.h" + +#include + +#include "BuiltinLoader.h" +#include "Util.h" + +using namespace v8; + +namespace tns { + +namespace { + +constexpr char kAlphabet[] = + "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"; + +// 6-bit value per ASCII byte; 0xFF marks everything outside the alphabet. +constexpr uint8_t kInvalid = 0xFF; + +uint8_t SixBits(uint8_t c) { + if (c >= 'A' && c <= 'Z') { + return static_cast(c - 'A'); + } + if (c >= 'a' && c <= 'z') { + return static_cast(c - 'a' + 26); + } + if (c >= '0' && c <= '9') { + return static_cast(c - '0' + 52); + } + if (c == '+') { + return 62; + } + if (c == '/') { + return 63; + } + return kInvalid; +} + +bool IsAsciiWhitespace(uint8_t c) { + return c == '\t' || c == '\n' || c == '\f' || c == '\r' || c == ' '; +} + +// The string's code units as bytes. Fails when any unit is above U+00FF, +// which neither op can represent. +bool GetLatin1Bytes(Isolate* isolate, Local value, + std::vector* out) { + if (!value->IsString()) { + return false; + } + Local str = value.As(); + if (!str->ContainsOnlyOneByte()) { + return false; + } + const int length = str->Length(); + out->resize(static_cast(length)); + if (length > 0) { + str->WriteOneByteV2(isolate, 0, static_cast(length), out->data()); + } + return true; +} + +// btoa: base64-encode the input's code units. +void BtoaCallback(const FunctionCallbackInfo& info) { + Isolate* isolate = info.GetIsolate(); + std::vector input; + if (!GetLatin1Bytes(isolate, info[0], &input)) { + info.GetReturnValue().SetNull(); + return; + } + + std::vector out; + out.reserve((input.size() + 2) / 3 * 4); + size_t i = 0; + for (; i + 3 <= input.size(); i += 3) { + const uint32_t group = (static_cast(input[i]) << 16) | + (static_cast(input[i + 1]) << 8) | + input[i + 2]; + out.push_back(kAlphabet[(group >> 18) & 0x3F]); + out.push_back(kAlphabet[(group >> 12) & 0x3F]); + out.push_back(kAlphabet[(group >> 6) & 0x3F]); + out.push_back(kAlphabet[group & 0x3F]); + } + const size_t remaining = input.size() - i; + if (remaining == 1) { + const uint32_t group = static_cast(input[i]) << 16; + out.push_back(kAlphabet[(group >> 18) & 0x3F]); + out.push_back(kAlphabet[(group >> 12) & 0x3F]); + out.push_back('='); + out.push_back('='); + } else if (remaining == 2) { + const uint32_t group = (static_cast(input[i]) << 16) | + (static_cast(input[i + 1]) << 8); + out.push_back(kAlphabet[(group >> 18) & 0x3F]); + out.push_back(kAlphabet[(group >> 12) & 0x3F]); + out.push_back(kAlphabet[(group >> 6) & 0x3F]); + out.push_back('='); + } + + if (out.empty()) { + info.GetReturnValue().Set(v8::String::Empty(isolate)); + return; + } + Local result; + if (v8::String::NewFromOneByte(isolate, out.data(), NewStringType::kNormal, + static_cast(out.size())) + .ToLocal(&result)) { + info.GetReturnValue().Set(result); + } +} + +// atob: forgiving-base64 decode +// (https://infra.spec.whatwg.org/#forgiving-base64-decode). +void AtobCallback(const FunctionCallbackInfo& info) { + Isolate* isolate = info.GetIsolate(); + std::vector raw; + if (!GetLatin1Bytes(isolate, info[0], &raw)) { + info.GetReturnValue().SetNull(); + return; + } + + std::vector data; + data.reserve(raw.size()); + for (uint8_t c : raw) { + if (!IsAsciiWhitespace(c)) { + data.push_back(c); + } + } + + if (data.size() % 4 == 0) { + size_t strip = 0; + while (strip < 2 && !data.empty() && data.back() == '=') { + data.pop_back(); + strip++; + } + } + if (data.size() % 4 == 1) { + info.GetReturnValue().SetNull(); + return; + } + + std::vector out; + out.reserve(data.size() / 4 * 3 + 2); + uint32_t accumulator = 0; + uint32_t bits = 0; + for (uint8_t c : data) { + const uint8_t value = SixBits(c); + if (value == kInvalid) { + info.GetReturnValue().SetNull(); + return; + } + accumulator = (accumulator << 6) | value; + bits += 6; + if (bits >= 8) { + bits -= 8; + out.push_back(static_cast((accumulator >> bits) & 0xFF)); + } + } + + if (out.empty()) { + info.GetReturnValue().Set(v8::String::Empty(isolate)); + return; + } + Local result; + if (v8::String::NewFromOneByte(isolate, out.data(), NewStringType::kNormal, + static_cast(out.size())) + .ToLocal(&result)) { + info.GetReturnValue().Set(result); + } +} + +MaybeLocal CreateBinding(Local context) { + Isolate* isolate = v8::Isolate::GetCurrent(); + Local binding = Object::New(isolate); + tns::SetMethodNoSideEffect(context, binding, "btoa", BtoaCallback); + tns::SetMethodNoSideEffect(context, binding, "atob", AtobCallback); + return binding; +} + +} // namespace + +MaybeLocal Base64::GetExports(Local context) { + return BuiltinLoader::GetExports(context, BuiltinId::kBase64, CreateBinding); +} + +} // namespace tns diff --git a/test-app/runtime/src/main/cpp/Base64.h b/test-app/runtime/src/main/cpp/Base64.h new file mode 100644 index 000000000..df0378855 --- /dev/null +++ b/test-app/runtime/src/main/cpp/Base64.h @@ -0,0 +1,24 @@ +#ifndef BASE64_H_ +#define BASE64_H_ + +#include "v8.h" + +namespace tns { + +/* + * Native ops behind the base64 builtin (internal/base64.js): the WHATWG + * forgiving-base64 codec backing the atob / btoa globals. Both ops answer + * null instead of throwing, so the builtin owns the error shape. + */ +class Base64 { +public: + /* + * The builtin's exports, `{ atob, btoa }`, from the one run it gets per + * isolate. + */ + static v8::MaybeLocal GetExports(v8::Local context); +}; + +} // namespace tns + +#endif /* BASE64_H_ */ diff --git a/test-app/runtime/src/main/cpp/BuiltinLoader.cpp b/test-app/runtime/src/main/cpp/BuiltinLoader.cpp index 6abecf60d..eae4ddd6c 100644 --- a/test-app/runtime/src/main/cpp/BuiltinLoader.cpp +++ b/test-app/runtime/src/main/cpp/BuiltinLoader.cpp @@ -38,6 +38,15 @@ constexpr const char* kPrimordialsParamName = "primordials"; constexpr const char* kInternalsParamName = "internals"; constexpr size_t kParamCount = 6; +/* + * `module.exports` of every builtin that has run in this isolate, indexed by + * id. Per isolate because a builtin is a singleton per realm, so workers run + * their own copy of a file and export their own objects. + */ +struct BuiltinExportsState { + v8::Global exports[static_cast(BuiltinId::kCount)]; +}; + /* * This runtime's intrinsics snapshot, builtin require and shared internals * object. Per-runtime state rather than an isolate-keyed shared map, so @@ -258,4 +267,32 @@ MaybeLocal BuiltinLoader::RunBuiltin(Local context, BuiltinId id return CallBuiltin(context, id, binding, primordials, internals); } +MaybeLocal BuiltinLoader::GetExports(Local context, BuiltinId id, + BindingFactory bindingFactory) { + Isolate* isolate = v8::Isolate::GetCurrent(); + auto* state = RuntimeState::For(isolate); + if (state == nullptr) { + return MaybeLocal(); + } + + const unsigned index = static_cast(id); + if (!state->exports[index].IsEmpty()) { + return state->exports[index].Get(isolate); + } + + Local binding; + if (bindingFactory != nullptr && !bindingFactory(context).ToLocal(&binding)) { + return MaybeLocal(); + } + + Local result; + if (!RunBuiltin(context, id, binding).ToLocal(&result) || !result->IsObject()) { + return MaybeLocal(); + } + + Local exports = result.As(); + state->exports[index].Reset(isolate, exports); + return exports; +} + } // namespace tns diff --git a/test-app/runtime/src/main/cpp/BuiltinLoader.h b/test-app/runtime/src/main/cpp/BuiltinLoader.h index 7078dbc7b..5db89ac63 100644 --- a/test-app/runtime/src/main/cpp/BuiltinLoader.h +++ b/test-app/runtime/src/main/cpp/BuiltinLoader.h @@ -8,6 +8,13 @@ namespace tns { class BuiltinLoader { public: + /* + * Builds the bag of natives a builtin receives as its `binding` parameter. + * GetExports calls it only when the builtin actually runs, so a call site + * that hits the cache pays nothing for it. + */ + using BindingFactory = v8::MaybeLocal (*)(v8::Local); + /* * Compiles the builtin identified by id as a function body with the fixed * parameters `exports`, `require`, `module`, `binding` (Node's module @@ -32,6 +39,18 @@ class BuiltinLoader { static v8::MaybeLocal RunBuiltin( v8::Local context, BuiltinId id, v8::Local binding = v8::Local()); + + /* + * The builtin's `module.exports`, running it at most once per isolate. + * Every entry point that reaches the same file — the `ns:`/`node:` module + * registry, the lazy-global tier, another builtin's `require` — shares + * that one run, so a value a file exports is the same object through all + * of them. Empty when the builtin failed to run (an exception is pending) + * or exported a non-object. + */ + static v8::MaybeLocal GetExports(v8::Local context, + BuiltinId id, + BindingFactory binding); }; } // namespace tns diff --git a/test-app/runtime/src/main/cpp/LazyGlobals.cpp b/test-app/runtime/src/main/cpp/LazyGlobals.cpp new file mode 100644 index 000000000..5fbe09b48 --- /dev/null +++ b/test-app/runtime/src/main/cpp/LazyGlobals.cpp @@ -0,0 +1,72 @@ +#include "LazyGlobals.h" + +#include "ArgConverter.h" +#include "Base64.h" +#include "TextEncoding.h" + +using namespace v8; + +namespace tns { + +namespace { + +/* + * The builtin's `module.exports`, from the single run it gets per isolate + * (BuiltinLoader::GetExports). Two globals out of the same file therefore + * cost one run, and so does a module that exports the same interfaces. + */ +using ExportsAccessor = MaybeLocal (*)(Local); + +struct LazyGlobalEntry { + const char* name; + const char* exportName; // key of `name` in the builtin's module.exports + ExportsAccessor exports; +}; + +constexpr LazyGlobalEntry kLazyGlobals[] = { + {"TextEncoder", "TextEncoder", TextEncoding::GetExports}, + {"TextDecoder", "TextDecoder", TextEncoding::GetExports}, + {"atob", "atob", Base64::GetExports}, + {"btoa", "btoa", Base64::GetExports}, +}; + +void LazyGlobalGetter(Local property, + const PropertyCallbackInfo& info) { + Isolate* isolate = info.GetIsolate(); + const auto* entry = static_cast( + info.Data().As()->Value(v8::kExternalPointerTypeTagDefault)); + Local context = isolate->GetCurrentContext(); + + Local exports; + if (!entry->exports(context).ToLocal(&exports)) { + return; + } + Local value; + if (!exports->Get(context, + ArgConverter::ConvertToV8String(isolate, entry->exportName)) + .ToLocal(&value)) { + return; + } + info.GetReturnValue().Set(value); +} + +} // namespace + +void LazyGlobals::Init(Isolate* isolate, Local globalTemplate) { + for (const LazyGlobalEntry& entry : kLazyGlobals) { + Local data = + External::New(isolate, const_cast(&entry), + v8::kExternalPointerTypeTagDefault); + // SetLazyDataProperty, not a getter that rewrites the property itself: + // defining over an API accessor reads its current descriptor, which + // calls the getter again and recurses. V8 does the rewrite from the + // outside, and gives a setter-less accessor its + // ReconfigureToDataProperty setter, so an assignment landing before + // the first read replaces the global as well. + globalTemplate->SetLazyDataProperty( + ArgConverter::ConvertToV8String(isolate, entry.name), + LazyGlobalGetter, data, PropertyAttribute::DontEnum); + } +} + +} // namespace tns diff --git a/test-app/runtime/src/main/cpp/LazyGlobals.h b/test-app/runtime/src/main/cpp/LazyGlobals.h new file mode 100644 index 000000000..e93383761 --- /dev/null +++ b/test-app/runtime/src/main/cpp/LazyGlobals.h @@ -0,0 +1,33 @@ +#ifndef LAZYGLOBALS_H_ +#define LAZYGLOBALS_H_ + +#include "v8.h" + +namespace tns { + +/* + * Globals whose implementation is a runtime builtin that must not run until + * someone actually reaches for the name. Each entry is registered on the + * global template as a lazy data property; the first read runs the builtin + * through the per-isolate exports cache (BuiltinLoader::GetExports), so + * sibling names (TextEncoder and TextDecoder) share the run — as does a + * module exporting the same interfaces — and V8 then replaces the property + * with a plain data property so later reads cost nothing. + * + * A builtin behind this tier runs at an arbitrary point in the isolate's life + * rather than during init, so it may only consume `internals` keys published + * by eager builtins (see src/main/cpp/js/README.md). + */ +class LazyGlobals { +public: + /* + * Registers every lazy global. Must run before Context::New, on the same + * template the eager globals use. + */ + static void Init(v8::Isolate* isolate, + v8::Local globalTemplate); +}; + +} // namespace tns + +#endif /* LAZYGLOBALS_H_ */ diff --git a/test-app/runtime/src/main/cpp/NsBuiltinModules.cpp b/test-app/runtime/src/main/cpp/NsBuiltinModules.cpp index 4f88a23d8..60be4a93b 100644 --- a/test-app/runtime/src/main/cpp/NsBuiltinModules.cpp +++ b/test-app/runtime/src/main/cpp/NsBuiltinModules.cpp @@ -10,6 +10,7 @@ #include "NativeScriptAssert.h" #include "Runtime.h" #include "RuntimeState.h" +#include "TextEncoding.h" #include "TraceLog.h" #include "console/Console.h" #include "robin_hood.h" @@ -23,9 +24,19 @@ namespace { constexpr const char* kNsPrefix = "ns:"; constexpr const char* kNodePrefix = "node:"; +/* Defined below, each next to the natives it gathers. */ +MaybeLocal NsModuleBinding(Local context); +MaybeLocal NsRuntimeBinding(Local context); +MaybeLocal NsUtilBinding(Local context); + struct Registration { const char* specifier; BuiltinId builtin; + /* + * Natives the file receives as `binding`, null when it needs none (every + * `node:` shim, which reaches its `ns:` module through require instead). + */ + BuiltinLoader::BindingFactory binding; }; /* @@ -35,12 +46,12 @@ struct Registration { * never carries compatibility code. */ constexpr Registration kRegistry[] = { - {"ns:module", BuiltinId::kNsModule}, - {"ns:runtime", BuiltinId::kNsRuntime}, - {"ns:util", BuiltinId::kNsUtil}, - {"node:module", BuiltinId::kNodeModule}, - {"node:url", BuiltinId::kNodeUrl}, - {"node:util", BuiltinId::kNodeUtil}, + {"ns:module", BuiltinId::kNsModule, NsModuleBinding}, + {"ns:runtime", BuiltinId::kNsRuntime, NsRuntimeBinding}, + {"ns:util", BuiltinId::kNsUtil, NsUtilBinding}, + {"node:module", BuiltinId::kNodeModule, nullptr}, + {"node:url", BuiltinId::kNodeUrl, nullptr}, + {"node:util", BuiltinId::kNodeUtil, nullptr}, }; constexpr const char* kDebugKey = "debug"; @@ -124,12 +135,12 @@ bool HasPrefix(const std::string& specifier, const char* prefix) { /* * A builtin module is a singleton per realm, so every cache here is per - * runtime: workers get their own exports objects and their own synthetic - * modules. The ES module registry deliberately holds none of this. Touched - * only from its own runtime's thread. + * runtime: workers get their own synthetic modules. The exports objects the + * modules wrap live in the per-isolate builtin exports cache + * (BuiltinLoader::GetExports). The ES module registry deliberately holds none + * of this. Touched only from its own runtime's thread. */ struct RealmState { - robin_hood::unordered_map*> exports; robin_hood::unordered_map*> modules; // Specifiers currently being built, so a shim requiring back into the // module that is loading it fails instead of recursing. @@ -138,9 +149,6 @@ struct RealmState { bool formatUnavailable = false; ~RealmState() { - for (auto& entry : exports) { - delete entry.second; - } for (auto& entry : modules) { delete entry.second; } @@ -157,102 +165,74 @@ RealmState* GetRealm(Isolate* isolate) { return RuntimeState::For(isolate); } -MaybeLocal BuildBinding(Local context, BuiltinId builtin) { +MaybeLocal NsModuleBinding(Local context) { Isolate* isolate = v8::Isolate::GetCurrent(); Local binding = Object::New(isolate); - - switch (builtin) { - case BuiltinId::kNsModule: { - // The module loader's control surface (ModuleInternalCallbacks.cpp). - // The binding builder decides build-dependent membership; - // ns-module.js only shapes and freezes whatever arrives. - if (!BuildNsModuleBinding(context, binding)) { - return MaybeLocal(); - } - break; - } - case BuiltinId::kNsRuntime: { - Local setConfig, getConfig; - if (!v8::Function::New(context, SetConfigCallback).ToLocal(&setConfig) || - !v8::Function::New(context, GetConfigCallback).ToLocal(&getConfig) || - !binding->Set(context, ArgConverter::ConvertToV8String(isolate, "setConfig"), - setConfig) - .FromMaybe(false) || - !binding->Set(context, ArgConverter::ConvertToV8String(isolate, "getConfig"), - getConfig) - .FromMaybe(false)) { - return MaybeLocal(); - } - break; - } - case BuiltinId::kNsUtil: { - // The console formatter is built once per realm; ns:util - // re-exports that instance instead of creating a second one. - Local inspect = Console::getInspect(context); - if (inspect.IsEmpty()) { - return MaybeLocal(); - } - if (!binding->Set(context, ArgConverter::ConvertToV8String(isolate, "inspect"), inspect) - .FromMaybe(false)) { - return MaybeLocal(); - } - break; - } - default: - break; + // The module loader's control surface (ModuleInternalCallbacks.cpp). The + // binding builder decides build-dependent membership; ns-module.js only + // shapes and freezes whatever arrives. + if (!BuildNsModuleBinding(context, binding)) { + return MaybeLocal(); } - return binding; } -/* - * Runs a module's builtin and caches its exports. Always leaves an exception - * pending when it returns false. - */ -bool Instantiate(Local context, const Registration& requested) { +MaybeLocal NsRuntimeBinding(Local context) { Isolate* isolate = v8::Isolate::GetCurrent(); - RealmState* realmState = GetRealm(isolate); - if (realmState == nullptr) { - isolate->ThrowException(Exception::Error(ArgConverter::ConvertToV8String( - isolate, "Cannot load a builtin module: the runtime is shutting down"))); - return false; + Local binding = Object::New(isolate); + Local setConfig, getConfig; + if (!v8::Function::New(context, SetConfigCallback).ToLocal(&setConfig) || + !v8::Function::New(context, GetConfigCallback).ToLocal(&getConfig) || + !binding->Set(context, ArgConverter::ConvertToV8String(isolate, "setConfig"), setConfig) + .FromMaybe(false) || + !binding->Set(context, ArgConverter::ConvertToV8String(isolate, "getConfig"), getConfig) + .FromMaybe(false)) { + return MaybeLocal(); } - RealmState& realm = *realmState; + return binding; +} - /* - * A shim reaches its ns: module through the builtin require, so the graph - * is walked while a module is still being built; a cycle would otherwise - * recurse until the stack runs out. - */ - if (realm.inProgress.count(requested.specifier) > 0) { - isolate->ThrowException(Exception::Error(ArgConverter::ConvertToV8String( - isolate, "Circular require of built-in module: " + - std::string(requested.specifier)))); - return false; +/* + * TextEncoder / TextDecoder for ns:util, read straight out of the + * text-encoding builtin's per-isolate run, so the module's classes are the + * objects the globals of the same name expose. + */ +void TextEncodingClassGetter(Local property, const PropertyCallbackInfo& info) { + Local context = info.GetIsolate()->GetCurrentContext(); + Local exports; + Local value; + if (TextEncoding::GetExports(context).ToLocal(&exports) && + exports->Get(context, property).ToLocal(&value)) { + info.GetReturnValue().Set(value); } - realm.inProgress.emplace(requested.specifier); +} - TryCatch tc(isolate); - Local binding; - Local result; - bool built = BuildBinding(context, requested.builtin).ToLocal(&binding) && - BuiltinLoader::RunBuiltin(context, requested.builtin, binding).ToLocal(&result) && - result->IsObject(); - realm.inProgress.erase(requested.specifier); +MaybeLocal NsUtilBinding(Local context) { + Isolate* isolate = v8::Isolate::GetCurrent(); + Local binding = Object::New(isolate); - if (built) { - realm.exports[requested.specifier] = new Persistent(isolate, result.As()); - return true; + // The console formatter is built once per realm; ns:util re-exports that + // instance instead of creating a second one. + Local inspect = Console::getInspect(context); + if (inspect.IsEmpty()) { + return MaybeLocal(); + } + if (!binding->Set(context, ArgConverter::ConvertToV8String(isolate, "inspect"), inspect) + .FromMaybe(false)) { + return MaybeLocal(); } - if (tc.HasCaught()) { - tc.ReThrow(); - return false; + // Lazy so that requiring ns:util does not run the text-encoding builtin; + // ns-util.js keeps the read inside its own getters to preserve that. + for (const char* name : {"TextEncoder", "TextDecoder"}) { + if (binding->SetLazyDataProperty(context, ArgConverter::ConvertToV8String(isolate, name), + TextEncodingClassGetter) + .IsNothing()) { + return MaybeLocal(); + } } - isolate->ThrowException(Exception::Error(ArgConverter::ConvertToV8String( - isolate, "Failed to initialize built-in module '" + std::string(requested.specifier) + - "'"))); - return false; + + return binding; } /* @@ -344,17 +324,35 @@ MaybeLocal NsBuiltinModules::GetExports(Local context, return MaybeLocal(); } RealmState& realm = *realmState; - auto it = realm.exports.find(specifier); - if (it == realm.exports.end()) { - if (!Instantiate(context, *registration)) { - return MaybeLocal(); - } - it = realm.exports.find(specifier); - if (it == realm.exports.end()) { - return MaybeLocal(); - } + + /* + * A shim reaches its ns: module through the builtin require, so the graph + * is walked while a module is still being built; a cycle would otherwise + * recurse until the stack runs out. + */ + if (realm.inProgress.count(specifier) > 0) { + isolate->ThrowException(Exception::Error(ArgConverter::ConvertToV8String( + isolate, "Circular require of built-in module: " + specifier))); + return MaybeLocal(); } - return it->second->Get(isolate); + realm.inProgress.emplace(specifier); + + TryCatch tc(isolate); + Local exports; + bool built = BuiltinLoader::GetExports(context, registration->builtin, registration->binding) + .ToLocal(&exports); + realm.inProgress.erase(specifier); + + if (built) { + return exports; + } + if (tc.HasCaught()) { + tc.ReThrow(); + return MaybeLocal(); + } + isolate->ThrowException(Exception::Error(ArgConverter::ConvertToV8String( + isolate, "Failed to initialize built-in module '" + specifier + "'"))); + return MaybeLocal(); } MaybeLocal NsBuiltinModules::GetModule(Local context, diff --git a/test-app/runtime/src/main/cpp/Runtime.cpp b/test-app/runtime/src/main/cpp/Runtime.cpp index 3795bb08b..493f1e7d6 100644 --- a/test-app/runtime/src/main/cpp/Runtime.cpp +++ b/test-app/runtime/src/main/cpp/Runtime.cpp @@ -26,6 +26,7 @@ #include "IsolateTracked.h" #include "JType.h" #include "JsArgToArrayConverter.h" +#include "LazyGlobals.h" #include "ManualInstrumentation.h" #include "MetadataNode.h" #include "ModuleBinding.h" @@ -969,6 +970,11 @@ Isolate* Runtime::PrepareV8Runtime(const string& filesPath, CallbackHandlers::CreateGlobalCastFunctions(isolate, globalTemplate); + // Lazy web globals (TextEncoder/TextDecoder, atob/btoa): registered on the + // template so the builtins behind them run only on first use, in every + // isolate — workers included. + LazyGlobals::Init(isolate, globalTemplate); + Local context = Context::New(isolate, nullptr, globalTemplate); auto global = context->Global(); diff --git a/test-app/runtime/src/main/cpp/TextEncoding.cpp b/test-app/runtime/src/main/cpp/TextEncoding.cpp new file mode 100644 index 000000000..ff6479087 --- /dev/null +++ b/test-app/runtime/src/main/cpp/TextEncoding.cpp @@ -0,0 +1,737 @@ +#include "TextEncoding.h" + +#include +#include + +#include "v8-fast-api-calls.h" + +#include "ArgConverter.h" +#include "BuiltinLoader.h" +#include "Util.h" + +using namespace v8; + +namespace tns { + +namespace { + +// Encoding ids shared with text-encoding.js, which maps them back to the +// canonical names through its kEncodingNames array — keep the two in step. +enum Encoding : uint32_t { + kUtf8 = 0, + kUtf16le = 1, + kUtf16be = 2, + kWindows1252 = 3, +}; + +// Decode option bits, mirrored by kFlag* in text-encoding.js. +constexpr uint32_t kFlagFatal = 1; +constexpr uint32_t kFlagIgnoreBOM = 2; +constexpr uint32_t kFlagStream = 4; + +constexpr uint16_t kReplacementCharacter = 0xFFFD; + +struct EncodingLabel { + const char* label; + Encoding encoding; +}; + +// The complete label set of the four encodings the runtime supports +// (https://encoding.spec.whatwg.org/#names-and-labels), sorted by label so +// lookup is a binary search over a table with no runtime setup cost. +constexpr EncodingLabel kLabels[] = { + {"ansi_x3.4-1968", kWindows1252}, + {"ascii", kWindows1252}, + {"cp1252", kWindows1252}, + {"cp819", kWindows1252}, + {"csisolatin1", kWindows1252}, + {"csunicode", kUtf16le}, + {"ibm819", kWindows1252}, + {"iso-10646-ucs-2", kUtf16le}, + {"iso-8859-1", kWindows1252}, + {"iso-ir-100", kWindows1252}, + {"iso8859-1", kWindows1252}, + {"iso88591", kWindows1252}, + {"iso_8859-1", kWindows1252}, + {"iso_8859-1:1987", kWindows1252}, + {"l1", kWindows1252}, + {"latin1", kWindows1252}, + {"ucs-2", kUtf16le}, + {"unicode", kUtf16le}, + {"unicode-1-1-utf-8", kUtf8}, + {"unicode11utf8", kUtf8}, + {"unicode20utf8", kUtf8}, + {"unicodefeff", kUtf16le}, + {"unicodefffe", kUtf16be}, + {"us-ascii", kWindows1252}, + {"utf-16", kUtf16le}, + {"utf-16be", kUtf16be}, + {"utf-16le", kUtf16le}, + {"utf-8", kUtf8}, + {"utf8", kUtf8}, + {"windows-1252", kWindows1252}, + {"x-cp1252", kWindows1252}, + {"x-unicode20utf8", kUtf8}, +}; + +// windows-1252 index, pointers 0x80-0x9F. Everything outside that block is +// Latin-1 (identity), including the C1 controls this table maps to +// themselves. +constexpr uint16_t kWindows1252Index[32] = { + 0x20AC, 0x0081, 0x201A, 0x0192, 0x201E, 0x2026, 0x2020, 0x2021, + 0x02C6, 0x2030, 0x0160, 0x2039, 0x0152, 0x008D, 0x017D, 0x008F, + 0x0090, 0x2018, 0x2019, 0x201C, 0x201D, 0x2022, 0x2013, 0x2014, + 0x02DC, 0x2122, 0x0161, 0x203A, 0x0153, 0x009D, 0x017E, 0x0178}; + +bool IsAsciiWhitespace(char c) { + return c == '\t' || c == '\n' || c == '\f' || c == '\r' || c == ' '; +} + +int32_t LookupEncoding(const std::string& rawLabel) { + size_t begin = 0; + size_t end = rawLabel.size(); + while (begin < end && IsAsciiWhitespace(rawLabel[begin])) { + begin++; + } + while (end > begin && IsAsciiWhitespace(rawLabel[end - 1])) { + end--; + } + + std::string label = rawLabel.substr(begin, end - begin); + for (char& c : label) { + if (c >= 'A' && c <= 'Z') { + c = static_cast(c - 'A' + 'a'); + } + } + + size_t lo = 0; + size_t hi = sizeof(kLabels) / sizeof(kLabels[0]); + while (lo < hi) { + size_t mid = lo + (hi - lo) / 2; + int cmp = label.compare(kLabels[mid].label); + if (cmp == 0) { + return static_cast(kLabels[mid].encoding); + } + if (cmp < 0) { + hi = mid; + } else { + lo = mid + 1; + } + } + return -1; +} + +// Everything a decoder must remember between streaming calls. It lives in a +// Uint8Array the builtin allocates per TextDecoder, so the native side stays +// stateless and no instance needs a finalizer. +constexpr int kDecoderStateBytes = 10; // bytes Store writes +static_assert(kDecoderStateBytes <= TextEncoding::kDecoderStateSize, + "the builtin allocates too little decoder state"); + +struct DecoderState { + bool bomSeen = false; + uint8_t utf8PendingLength = 0; + uint8_t utf8Pending[3] = {0, 0, 0}; + bool hasLeadByte = false; + uint8_t leadByte = 0; + bool hasLeadSurrogate = false; + uint16_t leadSurrogate = 0; + + void Load(const uint8_t* raw) { + bomSeen = (raw[0] & 1) != 0; + utf8PendingLength = raw[1] > 3 ? 0 : raw[1]; + utf8Pending[0] = raw[2]; + utf8Pending[1] = raw[3]; + utf8Pending[2] = raw[4]; + hasLeadByte = raw[5] != 0; + leadByte = raw[6]; + hasLeadSurrogate = raw[7] != 0; + leadSurrogate = static_cast(raw[8] | (raw[9] << 8)); + } + + void Store(uint8_t* raw) const { + raw[0] = bomSeen ? 1 : 0; + raw[1] = utf8PendingLength; + raw[2] = utf8Pending[0]; + raw[3] = utf8Pending[1]; + raw[4] = utf8Pending[2]; + raw[5] = hasLeadByte ? 1 : 0; + raw[6] = leadByte; + raw[7] = hasLeadSurrogate ? 1 : 0; + raw[8] = static_cast(leadSurrogate & 0xFF); + raw[9] = static_cast(leadSurrogate >> 8); + } + + void Reset() { *this = DecoderState(); } +}; + +// Collects decoded UTF-16 code units and tracks whether they all fit in one +// byte, so the finished string can take V8's one-byte representation. +class Utf16Sink { +public: + void Append(uint16_t unit) { + units_.push_back(unit); + orAll_ |= unit; + } + + void AppendCodePoint(uint32_t codePoint) { + if (codePoint <= 0xFFFF) { + Append(static_cast(codePoint)); + return; + } + codePoint -= 0x10000; + Append(static_cast(0xD800 + (codePoint >> 10))); + Append(static_cast(0xDC00 + (codePoint & 0x3FF))); + } + + MaybeLocal Finish(Isolate* isolate) const { + if (units_.empty()) { + return v8::String::Empty(isolate); + } + if (orAll_ <= 0xFF) { + std::vector oneByte(units_.size()); + for (size_t i = 0; i < units_.size(); i++) { + oneByte[i] = static_cast(units_[i]); + } + return v8::String::NewFromOneByte(isolate, oneByte.data(), + NewStringType::kNormal, + static_cast(oneByte.size())); + } + return v8::String::NewFromTwoByte(isolate, units_.data(), + NewStringType::kNormal, + static_cast(units_.size())); + } + +private: + std::vector units_; + uint32_t orAll_ = 0; +}; + +// A code point leaving a utf-8 / utf-16 decoder, with the leading-BOM removal +// TextDecoder performs once per stream. +class BomFilter { +public: + BomFilter(Utf16Sink& sink, DecoderState& state, bool ignoreBOM) + : sink_(sink), state_(state), ignoreBOM_(ignoreBOM) {} + + void Emit(uint32_t codePoint) { + if (!state_.bomSeen) { + state_.bomSeen = true; + if (!ignoreBOM_ && codePoint == 0xFEFF) { + return; + } + } + sink_.AppendCodePoint(codePoint); + } + +private: + Utf16Sink& sink_; + DecoderState& state_; + const bool ignoreBOM_; +}; + +// WHATWG utf-8 decoder. Incomplete trailing sequences are kept as raw bytes +// and replayed at the head of the next call, so the boundary constraints of a +// split sequence are re-derived from its own lead byte rather than carried in +// the saved state. Returns false when fatal mode hits invalid input. +bool DecodeUtf8(const uint8_t* input, size_t inputLength, DecoderState& state, + bool stream, bool fatal, bool ignoreBOM, Utf16Sink& sink) { + const size_t pendingLength = state.utf8PendingLength; + const size_t total = pendingLength + inputLength; + auto byteAt = [&](size_t index) -> uint8_t { + return index < pendingLength ? state.utf8Pending[index] + : input[index - pendingLength]; + }; + state.utf8PendingLength = 0; + + BomFilter out(sink, state, ignoreBOM); + uint32_t codePoint = 0; + uint32_t bytesNeeded = 0; + uint32_t bytesSeen = 0; + uint32_t lowerBoundary = 0x80; + uint32_t upperBoundary = 0xBF; + size_t sequenceStart = 0; + size_t index = 0; + + while (index < total) { + const uint8_t byte = byteAt(index); + if (bytesNeeded == 0) { + sequenceStart = index; + index++; + if (byte <= 0x7F) { + out.Emit(byte); + } else if (byte >= 0xC2 && byte <= 0xDF) { + bytesNeeded = 1; + codePoint = byte & 0x1F; + } else if (byte >= 0xE0 && byte <= 0xEF) { + if (byte == 0xE0) { + lowerBoundary = 0xA0; + } else if (byte == 0xED) { + upperBoundary = 0x9F; + } + bytesNeeded = 2; + codePoint = byte & 0x0F; + } else if (byte >= 0xF0 && byte <= 0xF4) { + if (byte == 0xF0) { + lowerBoundary = 0x90; + } else if (byte == 0xF4) { + upperBoundary = 0x8F; + } + bytesNeeded = 3; + codePoint = byte & 0x07; + } else { + if (fatal) { + state.Reset(); + return false; + } + out.Emit(kReplacementCharacter); + } + continue; + } + + if (byte < lowerBoundary || byte > upperBoundary) { + // One replacement for the maximal subpart consumed so far; the + // offending byte is reprocessed as the start of a new sequence + // (index unchanged). + codePoint = 0; + bytesNeeded = 0; + bytesSeen = 0; + lowerBoundary = 0x80; + upperBoundary = 0xBF; + if (fatal) { + state.Reset(); + return false; + } + out.Emit(kReplacementCharacter); + continue; + } + + lowerBoundary = 0x80; + upperBoundary = 0xBF; + codePoint = (codePoint << 6) | (byte & 0x3F); + bytesSeen++; + index++; + if (bytesSeen != bytesNeeded) { + continue; + } + const uint32_t finished = codePoint; + codePoint = 0; + bytesNeeded = 0; + bytesSeen = 0; + out.Emit(finished); + } + + if (bytesNeeded == 0) { + return true; + } + if (!stream) { + if (fatal) { + state.Reset(); + return false; + } + out.Emit(kReplacementCharacter); + return true; + } + // At most three bytes, and each is read before the slot it overwrites. + const size_t carried = total - sequenceStart; + for (size_t i = 0; i < carried; i++) { + state.utf8Pending[i] = byteAt(sequenceStart + i); + } + state.utf8PendingLength = static_cast(carried); + return true; +} + +// WHATWG shared utf-16 decoder, both endiannesses. +bool DecodeUtf16(const uint8_t* input, size_t inputLength, bool bigEndian, + DecoderState& state, bool stream, bool fatal, bool ignoreBOM, + Utf16Sink& sink) { + BomFilter out(sink, state, ignoreBOM); + + auto process = [&](uint16_t unit) -> bool { + if (state.hasLeadSurrogate) { + const uint16_t lead = state.leadSurrogate; + state.hasLeadSurrogate = false; + if (unit >= 0xDC00 && unit <= 0xDFFF) { + out.Emit(0x10000u + (static_cast(lead - 0xD800) << 10) + + (unit - 0xDC00)); + return true; + } + if (fatal) { + return false; + } + // The unpaired lead is replaced and `unit` starts over below. + out.Emit(kReplacementCharacter); + } + if (unit >= 0xD800 && unit <= 0xDBFF) { + state.hasLeadSurrogate = true; + state.leadSurrogate = unit; + return true; + } + if (unit >= 0xDC00 && unit <= 0xDFFF) { + if (fatal) { + return false; + } + out.Emit(kReplacementCharacter); + return true; + } + out.Emit(unit); + return true; + }; + + for (size_t i = 0; i < inputLength; i++) { + const uint8_t byte = input[i]; + if (!state.hasLeadByte) { + state.hasLeadByte = true; + state.leadByte = byte; + continue; + } + const uint16_t unit = + bigEndian ? static_cast((state.leadByte << 8) | byte) + : static_cast((byte << 8) | state.leadByte); + state.hasLeadByte = false; + if (!process(unit)) { + state.Reset(); + return false; + } + } + + if (stream) { + return true; + } + if (state.hasLeadByte || state.hasLeadSurrogate) { + state.hasLeadByte = false; + state.hasLeadSurrogate = false; + if (fatal) { + state.Reset(); + return false; + } + out.Emit(kReplacementCharacter); + } + return true; +} + +void DecodeWindows1252(const uint8_t* input, size_t inputLength, + Utf16Sink& sink) { + for (size_t i = 0; i < inputLength; i++) { + const uint8_t byte = input[i]; + sink.Append(byte >= 0x80 && byte <= 0x9F ? kWindows1252Index[byte - 0x80] + : byte); + } +} + +bool AllBytesBelow(const uint8_t* input, size_t length, uint8_t limit) { + for (size_t i = 0; i < length; i++) { + if (input[i] >= limit) { + return false; + } + } + return true; +} + +bool NoC1Bytes(const uint8_t* input, size_t length) { + for (size_t i = 0; i < length; i++) { + if (input[i] >= 0x80 && input[i] <= 0x9F) { + return false; + } + } + return true; +} + +// Bytes of an ArrayBuffer, SharedArrayBuffer or any ArrayBufferView; a +// detached buffer reads as empty. Only the builtin calls in, so anything else +// is a programming error rather than a user-visible one. +bool GetByteSource(Local value, const uint8_t** data, size_t* length) { + *data = nullptr; + *length = 0; + if (value.IsEmpty() || value->IsUndefined()) { + return true; + } + if (value->IsArrayBufferView()) { + Local view = value.As(); + Local buffer = view->Buffer(); + void* base = buffer->Data(); + if (base == nullptr) { + return true; + } + *data = static_cast(base) + view->ByteOffset(); + *length = view->ByteLength(); + return true; + } + if (value->IsArrayBuffer()) { + Local buffer = value.As(); + if (buffer->Data() == nullptr) { + return true; + } + *data = static_cast(buffer->Data()); + *length = buffer->ByteLength(); + return true; + } + if (value->IsSharedArrayBuffer()) { + Local buffer = value.As(); + if (buffer->Data() == nullptr) { + return true; + } + *data = static_cast(buffer->Data()); + *length = buffer->ByteLength(); + return true; + } + return false; +} + +void LabelToEncodingCallback(const FunctionCallbackInfo& info) { + Isolate* isolate = info.GetIsolate(); + info.GetReturnValue().Set(LookupEncoding(Util::ToString(isolate, info[0]))); +} + +void DecodeCallback(const FunctionCallbackInfo& info) { + Isolate* isolate = info.GetIsolate(); + + const uint8_t* input = nullptr; + size_t inputLength = 0; + if (!GetByteSource(info[0], &input, &inputLength)) { + isolate->ThrowException(Exception::TypeError(ArgConverter::ConvertToV8String( + isolate, + "The \"input\" argument must be an ArrayBuffer, SharedArrayBuffer or " + "ArrayBufferView"))); + return; + } + + const uint32_t encoding = + static_cast(info[1].As()->Value()); + const uint32_t flags = + static_cast(info[2].As()->Value()); + const bool fatal = (flags & kFlagFatal) != 0; + const bool ignoreBOM = (flags & kFlagIgnoreBOM) != 0; + const bool stream = (flags & kFlagStream) != 0; + + Local stateArray = info[3].As(); + uint8_t* rawState = static_cast(stateArray->Buffer()->Data()) + + stateArray->ByteOffset(); + DecoderState state; + state.Load(rawState); + + // Byte-for-byte one-byte results skip the intermediate code-unit buffer. + if (input != nullptr && !stream && state.utf8PendingLength == 0 && + !state.hasLeadByte && !state.hasLeadSurrogate) { + const bool asciiUtf8 = + encoding == kUtf8 && AllBytesBelow(input, inputLength, 0x80); + const bool latin1Windows1252 = + encoding == kWindows1252 && NoC1Bytes(input, inputLength); + if (asciiUtf8 || latin1Windows1252) { + state.Reset(); + state.Store(rawState); + Local result; + if (v8::String::NewFromOneByte(isolate, input, NewStringType::kNormal, + static_cast(inputLength)) + .ToLocal(&result)) { + info.GetReturnValue().Set(result); + } + return; + } + } + + Utf16Sink sink; + bool ok = true; + switch (encoding) { + case kUtf8: + ok = DecodeUtf8(input, inputLength, state, stream, fatal, ignoreBOM, + sink); + break; + case kUtf16le: + case kUtf16be: + ok = DecodeUtf16(input, inputLength, encoding == kUtf16be, state, + stream, fatal, ignoreBOM, sink); + break; + default: + DecodeWindows1252(input, inputLength, sink); + break; + } + + // A non-streaming call is the end of a stream: the next one starts from a + // clean decoder, BOM tracking included. + if (!stream) { + state.Reset(); + } + state.Store(rawState); + if (!ok) { + isolate->ThrowException(Exception::TypeError(ArgConverter::ConvertToV8String( + isolate, "The encoded data was not valid"))); + return; + } + + Local result; + if (sink.Finish(isolate).ToLocal(&result)) { + info.GetReturnValue().Set(result); + } +} + +void EncodeUtf8Callback(const FunctionCallbackInfo& info) { + Isolate* isolate = info.GetIsolate(); + Local source = info[0].As(); + + const size_t length = source->Utf8LengthV2(isolate); + std::unique_ptr store = + ArrayBuffer::NewBackingStore(isolate, length); + if (length > 0) { + source->WriteUtf8V2(isolate, static_cast(store->Data()), length, + v8::String::WriteFlags::kReplaceInvalidUtf8); + } + Local buffer = ArrayBuffer::New(isolate, std::move(store)); + info.GetReturnValue().Set(Uint8Array::New(buffer, 0, length)); +} + +// encodeInto status codes, mirrored in text-encoding.js. +constexpr int32_t kEncodeIntoOk = 0; +constexpr int32_t kEncodeIntoBadDestination = 1; +// Fast path only: the view's buffer is still on the V8 heap and Buffer() would +// allocate to materialize it, which a fast callback must not do. The builtin +// retries through encodeIntoFallback, which always runs the slow callback. +constexpr int32_t kEncodeIntoRetrySlow = 2; + +// Writes as much of `source` as fits into `destination` without splitting an +// encoded code point, and reports {read, written} through `results` — the +// Uint32Array the binding owns, so the op returns only a status code and +// stays expressible as a fast call. A destination that is a Uint8Array but +// detached or empty is a zero-length write, not a failure. +int32_t EncodeIntoImpl(Isolate* isolate, Local sourceValue, + Local destinationValue, Local resultsValue) { + if (!destinationValue->IsUint8Array()) { + return kEncodeIntoBadDestination; + } + if (!sourceValue->IsString() || !resultsValue->IsUint32Array()) { + return kEncodeIntoOk; + } + + Local results = resultsValue.As(); + uint32_t* resultData = static_cast(results->Buffer()->Data()); + if (resultData == nullptr || results->Length() < 2) { + return kEncodeIntoOk; + } + resultData += results->ByteOffset() / sizeof(uint32_t); + resultData[0] = 0; + resultData[1] = 0; + + Local destination = destinationValue.As(); + void* base = destination->Buffer()->Data(); + const size_t capacity = destination->ByteLength(); + if (base == nullptr || capacity == 0) { + return kEncodeIntoOk; + } + + size_t read = 0; + const size_t written = sourceValue.As()->WriteUtf8V2( + isolate, static_cast(base) + destination->ByteOffset(), + capacity, v8::String::WriteFlags::kReplaceInvalidUtf8, &read); + resultData[0] = static_cast(read); + resultData[1] = static_cast(written); + return kEncodeIntoOk; +} + +void EncodeIntoCallback(const FunctionCallbackInfo& info) { + info.GetReturnValue().Set( + EncodeIntoImpl(info.GetIsolate(), info[0], info[1], info[2])); +} + +#if NATIVESCRIPT_ENABLE_FAST_API +// Fast-call overload of encodeInto, live once a call site tiers up. A fast +// callback must not allocate on the JS heap, which shapes all three inputs: +// the kSeqOneByteString parameter keeps cons and two-byte sources on the slow +// callback (WriteUtf8V2 flattens, which allocates) and the latin-1 units are +// encoded by hand; a view whose buffer is still on-heap is declined with +// kEncodeIntoRetrySlow rather than materialized. +int32_t FastEncodeInto(Local receiver, const FastOneByteString& source, + Local destinationValue, Local resultsValue, + // NOLINTNEXTLINE(runtime/references) + FastApiCallbackOptions& options) { + HandleScope scope(options.isolate); + if (!destinationValue->IsUint8Array()) { + return kEncodeIntoBadDestination; + } + if (!resultsValue->IsUint32Array()) { + return kEncodeIntoOk; + } + Local destination = destinationValue.As(); + Local results = resultsValue.As(); + if (!destination->HasBuffer() || !results->HasBuffer()) { + return kEncodeIntoRetrySlow; + } + + uint32_t* resultData = static_cast(results->Buffer()->Data()); + if (resultData == nullptr || results->Length() < 2) { + return kEncodeIntoOk; + } + resultData += results->ByteOffset() / sizeof(uint32_t); + resultData[0] = 0; + resultData[1] = 0; + + void* base = destination->Buffer()->Data(); + const size_t capacity = destination->ByteLength(); + if (base == nullptr || capacity == 0) { + return kEncodeIntoOk; + } + uint8_t* out = static_cast(base) + destination->ByteOffset(); + + size_t read = 0; + size_t written = 0; + for (; read < source.length; read++) { + const uint8_t unit = static_cast(source.data[read]); + if (unit < 0x80) { + if (written + 1 > capacity) { + break; + } + out[written++] = unit; + } else { + if (written + 2 > capacity) { + break; + } + out[written++] = 0xC0 | (unit >> 6); + out[written++] = 0x80 | (unit & 0x3F); + } + } + resultData[0] = static_cast(read); + resultData[1] = static_cast(written); + return kEncodeIntoOk; +} + +const CFunction kFastEncodeInto = CFunction::Make(FastEncodeInto); +#endif + +MaybeLocal CreateBinding(Local context) { + Isolate* isolate = v8::Isolate::GetCurrent(); + Local binding = Object::New(isolate); + + tns::SetMethodNoSideEffect(context, binding, "labelToEncoding", + LabelToEncodingCallback); + tns::SetMethod(context, binding, "decode", DecodeCallback); + tns::SetMethodNoSideEffect(context, binding, "encodeUtf8", + EncodeUtf8Callback); +#if NATIVESCRIPT_ENABLE_FAST_API + tns::SetFastMethod(context, binding, "encodeInto", EncodeIntoCallback, + &kFastEncodeInto); +#else + tns::SetMethod(context, binding, "encodeInto", EncodeIntoCallback); +#endif + // Same slow callback with no fast overload: where the fast path answers + // kEncodeIntoRetrySlow, the builtin finishes the call through this name. + tns::SetMethod(context, binding, "encodeIntoFallback", EncodeIntoCallback); + + // Native ArrayBuffers carry a real backing store from birth, so the fast + // path's HasBuffer test always passes for the results array. + Local resultsBuffer = ArrayBuffer::New(isolate, 2 * sizeof(uint32_t)); + if (!binding->Set(context, ArgConverter::ConvertToV8String(isolate, "encodeIntoResults"), + Uint32Array::New(resultsBuffer, 0, 2)) + .FromMaybe(false)) { + return MaybeLocal(); + } + + return binding; +} + +} // namespace + +MaybeLocal TextEncoding::GetExports(Local context) { + return BuiltinLoader::GetExports(context, BuiltinId::kTextEncoding, + CreateBinding); +} + +} // namespace tns diff --git a/test-app/runtime/src/main/cpp/TextEncoding.h b/test-app/runtime/src/main/cpp/TextEncoding.h new file mode 100644 index 000000000..d451cd16e --- /dev/null +++ b/test-app/runtime/src/main/cpp/TextEncoding.h @@ -0,0 +1,30 @@ +#ifndef TEXTENCODING_H_ +#define TEXTENCODING_H_ + +#include "v8.h" + +namespace tns { + +/* + * Native ops behind the text-encoding builtin (internal/text-encoding.js): + * the WHATWG label table, UTF-8 encoding and the decoders for the encodings + * the runtime supports (utf-8, utf-16le, utf-16be, windows-1252). The builtin + * owns the web-facing shapes; everything that touches bytes lives here. + */ +class TextEncoding { +public: + /* + * The builtin's exports, `{ TextEncoder, TextDecoder }`, from the one run + * it gets per isolate. The lazy globals and `ns:util` both hand out these + * objects, so require("ns:util").TextDecoder === globalThis.TextDecoder + * whichever is reached first. + */ + static v8::MaybeLocal GetExports(v8::Local context); + + // Bytes of decoder state the builtin must hand back on every decode call. + static constexpr int kDecoderStateSize = 16; +}; + +} // namespace tns + +#endif /* TEXTENCODING_H_ */ diff --git a/test-app/runtime/src/main/cpp/Util.h b/test-app/runtime/src/main/cpp/Util.h index b7b601db2..ee102fcd5 100644 --- a/test-app/runtime/src/main/cpp/Util.h +++ b/test-app/runtime/src/main/cpp/Util.h @@ -110,6 +110,13 @@ void SetMethod(v8::Isolate* isolate, const char* name, v8::FunctionCallback callback, v8::Local data = v8::Local()); +// Whether the runtime registers v8::CFunction fast-call overloads next to the +// slow callbacks. Android runs V8 with the optimizing tiers enabled, so a +// registered overload is live once a call site tiers up; embeds that build V8 +// jitless can define this to 0 to drop the overloads entirely. +#ifndef NATIVESCRIPT_ENABLE_FAST_API +#define NATIVESCRIPT_ENABLE_FAST_API 1 +#endif void SetFastMethod(v8::Isolate* isolate, v8::Local that, const char* name, diff --git a/test-app/runtime/src/main/cpp/js/README.md b/test-app/runtime/src/main/cpp/js/README.md index 4cc45e04d..4d3ef822e 100644 --- a/test-app/runtime/src/main/cpp/js/README.md +++ b/test-app/runtime/src/main/cpp/js/README.md @@ -66,12 +66,41 @@ module.exports = somethingTheCallSiteNeeds; - Destructure `binding` and `primordials` once, at the top of the file, so the file's dependencies are visible and greppable. +## Eager and lazy builtins + +Most builtins run during `Runtime::PrepareV8Runtime` and install their globals +themselves. A **lazy** builtin instead exports its interfaces and is run by +`LazyGlobals` (`src/main/cpp/LazyGlobals.cpp`), which registers each global it +backs as a lazy data property on the global template: the first read of the +name runs the file through the per-isolate exports cache +(`BuiltinLoader::GetExports`) so sibling names share one run, and V8 replaces +the property with a plain data property. That cache is the same one the +`ns:`/`node:` module registry uses, so a module re-exporting a lazy builtin's +interfaces (`ns:util`'s `TextEncoder`) hands out the objects the globals hold, +in either access order. Until then nothing of it exists — no compile, no run, +no allocation. `text-encoding.js` (`TextEncoder`/`TextDecoder`) and +`base64.js` (`atob`/`btoa`) are the current ones; new globals join by adding a +row to `kLazyGlobals`. + +The two extra rules a lazy builtin lives by: + +- **It runs at an arbitrary point in the isolate's life, not at init.** The + `internals` channel is therefore off limits: its producers publish during + their own init, and a consumer that reads a key it does not find fails at + first use instead of loudly at boot. Anything a lazy builtin needs from + another builtin has to come through `require` or its `binding`. +- **It must not install anything on `globalThis`.** The C++ tier owns + placement; a file that self-installs would have to run to do it, which is + the thing being avoided. + ## Rules -- Run at isolate init, before any user code: capture any global you rely on - (e.g. `globalThis.Event`) eagerly so later monkey-patching can't break you. - For intrinsics that is what `primordials` is; for everything else - (`URLSearchParams`, …) capture it into a file-level `const`. +- Eager builtins run at isolate init, before any user code: capture any global + you rely on (e.g. `globalThis.Event`) eagerly so later monkey-patching can't + break you. For intrinsics that is what `primordials` is; for everything else + (`URLSearchParams`, …) capture it into a file-level `const`. A lazy builtin + gets the same pristine `primordials`, but the live globals it would capture + are whatever user code left behind, so it should not reach for them at all. - No `import`/`export` — these are classic function bodies, not modules. - ESLint (`eslint.config.mjs` at the repo root, `npm run lint`) declares `exports`, `require`, `module`, `binding`, `primordials` and the reachable diff --git a/test-app/runtime/src/main/cpp/js/base64.js b/test-app/runtime/src/main/cpp/js/base64.js new file mode 100644 index 000000000..3fa075423 --- /dev/null +++ b/test-app/runtime/src/main/cpp/js/base64.js @@ -0,0 +1,46 @@ +"use strict"; +// atob / btoa (HTML Standard §8.3, base64 utility methods) over the WHATWG +// forgiving-base64 codec in Base64.cpp. +// +// This file exports the two functions instead of installing them; the C++ +// lazy-global tier (LazyGlobals) places them and is what runs this file, on +// the first read of either name. Nothing here may depend on a builtin that +// runs after it, so `internals` is off limits — see the README. +// +// Deliberate deviation from the spec: no DOMException in this runtime, so the +// failure is an Error with `name` patched to "InvalidCharacterError", the same +// stand-in abort-signal.js and performance.js use. The native ops answer null +// on failure rather than throwing, so that shape stays here. +const { Error, TypeError } = primordials; + +const { atob: decodeBase64, btoa: encodeBase64 } = binding; + +function invalidCharacterError() { + const e = new Error("Invalid character"); + e.name = "InvalidCharacterError"; + return e; +} + +function btoa(data) { + if (arguments.length < 1) { + throw new TypeError("btoa requires 1 argument"); + } + const result = encodeBase64(`${data}`); + if (result === null) { + throw invalidCharacterError(); + } + return result; +} + +function atob(data) { + if (arguments.length < 1) { + throw new TypeError("atob requires 1 argument"); + } + const result = decodeBase64(`${data}`); + if (result === null) { + throw invalidCharacterError(); + } + return result; +} + +module.exports = { atob, btoa }; diff --git a/test-app/runtime/src/main/cpp/js/node-util.js b/test-app/runtime/src/main/cpp/js/node-util.js index a89c2826e..fd965d47f 100644 --- a/test-app/runtime/src/main/cpp/js/node-util.js +++ b/test-app/runtime/src/main/cpp/js/node-util.js @@ -11,6 +11,18 @@ // surfaces can diverge without either one carrying the other's baggage. const { ObjectFreeze } = primordials; -const { inspect, format } = require("ns:util"); +const util = require("ns:util"); +const { inspect, format } = util; -module.exports = ObjectFreeze({ inspect, format }); +// Node exposes the two encoding interfaces on util; they stay lazy here for +// the same reason they are lazy on ns:util. +module.exports = ObjectFreeze({ + inspect, + format, + get TextEncoder() { + return util.TextEncoder; + }, + get TextDecoder() { + return util.TextDecoder; + }, +}); diff --git a/test-app/runtime/src/main/cpp/js/ns-util.js b/test-app/runtime/src/main/cpp/js/ns-util.js index a48fdf5b2..d7b466fd8 100644 --- a/test-app/runtime/src/main/cpp/js/ns-util.js +++ b/test-app/runtime/src/main/cpp/js/ns-util.js @@ -148,4 +148,16 @@ function format(...args) { return str; } -module.exports = ObjectFreeze({ inspect, format }); +// `binding.TextEncoder`/`.TextDecoder` are lazy: reading either one runs the +// text-encoding builtin, so the reads stay inside these getters instead of +// joining the destructuring at the top of the file. +module.exports = ObjectFreeze({ + inspect, + format, + get TextEncoder() { + return binding.TextEncoder; + }, + get TextDecoder() { + return binding.TextDecoder; + }, +}); diff --git a/test-app/runtime/src/main/cpp/js/primordials.js b/test-app/runtime/src/main/cpp/js/primordials.js index 7f60fb069..cb5c81a73 100644 --- a/test-app/runtime/src/main/cpp/js/primordials.js +++ b/test-app/runtime/src/main/cpp/js/primordials.js @@ -29,6 +29,8 @@ const intrinsics = { Set, String, TypeError, + Uint8Array, + Uint32Array, URL, WeakRef, diff --git a/test-app/runtime/src/main/cpp/js/text-encoding.js b/test-app/runtime/src/main/cpp/js/text-encoding.js new file mode 100644 index 000000000..0bf080119 --- /dev/null +++ b/test-app/runtime/src/main/cpp/js/text-encoding.js @@ -0,0 +1,184 @@ +"use strict"; +// TextEncoder / TextDecoder (WHATWG Encoding Standard, +// https://encoding.spec.whatwg.org). +// +// This file exports the two interfaces instead of installing them; the C++ +// lazy-global tier (LazyGlobals) places them and is what runs this file, on +// the first read of either name. Nothing here may depend on a builtin that +// runs after it, so `internals` is off limits — see the README. +// +// The supported encodings (utf-8, utf-16le, utf-16be, windows-1252) with +// their complete label sets, the decoders and the UTF-8 encoder all live in +// TextEncoding.cpp. Per-decoder streaming state is the Uint8Array this file +// allocates and the native decoder reads and rewrites, so a TextDecoder needs +// neither a native handle nor a finalizer. +const { + ObjectDefineProperty, + ObjectGetOwnPropertyDescriptor, + RangeError, + SymbolToStringTag, + TypeError, + Uint8Array, +} = primordials; + +const { + labelToEncoding, + decode, + encodeUtf8, + encodeInto, + encodeIntoFallback, + encodeIntoResults, +} = binding; + +// Indexed by the encoding ids labelToEncoding returns. +const kEncodingNames = ["utf-8", "utf-16le", "utf-16be", "windows-1252"]; + +// Mirror the kFlag* constants in TextEncoding.cpp. +const kFlagFatal = 1; +const kFlagIgnoreBOM = 2; +const kFlagStream = 4; + +// Mirrors TextEncoding::kDecoderStateSize. +const kDecoderStateSize = 16; + +// Mirror the kEncodeInto* status codes in TextEncoding.cpp. The op reports +// {read, written} through binding.encodeIntoResults rather than allocating a +// result object per call; it is synchronous, so that one native Uint32Array +// serves every encoder in the isolate. +const kEncodeIntoBadDestination = 1; +const kEncodeIntoRetrySlow = 2; + +// WebIDL dictionary conversion: undefined and null mean "all defaults", +// anything else must be an object. +function toDictionary(value, name) { + if (value === undefined || value === null) { + return undefined; + } + if (typeof value !== "object" && typeof value !== "function") { + throw new TypeError(`The "${name}" argument must be an object`); + } + return value; +} + +class TextEncoder { + #brand; + + static #check(receiver) { + if (!(#brand in receiver)) { + throw new TypeError("Illegal invocation"); + } + } + + get encoding() { + TextEncoder.#check(this); + return "utf-8"; + } + + encode(input = "") { + TextEncoder.#check(this); + return encodeUtf8(`${input}`); + } + + encodeInto(source, destination) { + TextEncoder.#check(this); + const text = `${source}`; + let code = encodeInto(text, destination, encodeIntoResults); + if (code === kEncodeIntoRetrySlow) { + code = encodeIntoFallback(text, destination, encodeIntoResults); + } + if (code === kEncodeIntoBadDestination) { + throw new TypeError( + 'The "destination" argument must be an instance of Uint8Array' + ); + } + return { read: encodeIntoResults[0], written: encodeIntoResults[1] }; + } +} + +class TextDecoder { + #encoding; + #fatal; + #ignoreBOM; + #flags; + #state; + + static #check(receiver) { + if (!(#encoding in receiver)) { + throw new TypeError("Illegal invocation"); + } + } + + constructor(label = "utf-8", options = undefined) { + const name = `${label}`; + const dictionary = toDictionary(options, "options"); + const encoding = labelToEncoding(name); + if (encoding < 0) { + throw new RangeError(`The encoding "${name}" is not supported`); + } + const fatal = dictionary !== undefined && !!dictionary.fatal; + const ignoreBOM = dictionary !== undefined && !!dictionary.ignoreBOM; + this.#encoding = encoding; + this.#fatal = fatal; + this.#ignoreBOM = ignoreBOM; + this.#flags = (fatal ? kFlagFatal : 0) | (ignoreBOM ? kFlagIgnoreBOM : 0); + this.#state = new Uint8Array(kDecoderStateSize); + } + + get encoding() { + TextDecoder.#check(this); + return kEncodingNames[this.#encoding]; + } + + get fatal() { + TextDecoder.#check(this); + return this.#fatal; + } + + get ignoreBOM() { + TextDecoder.#check(this); + return this.#ignoreBOM; + } + + decode(input = undefined, options = undefined) { + TextDecoder.#check(this); + const dictionary = toDictionary(options, "options"); + const stream = dictionary !== undefined && !!dictionary.stream; + return decode( + input, + this.#encoding, + stream ? this.#flags | kFlagStream : this.#flags, + this.#state + ); + } +} + +// WebIDL shape: interface members are enumerable prototype properties and the +// class string is a configurable, non-writable Symbol.toStringTag; class +// syntax alone yields non-enumerable members. +function finishInterface(ctor, tag, members) { + const proto = ctor.prototype; + ObjectDefineProperty(proto, SymbolToStringTag, { + value: tag, + writable: false, + enumerable: false, + configurable: true, + }); + for (let i = 0; i < members.length; i++) { + const desc = ObjectGetOwnPropertyDescriptor(proto, members[i]); + desc.enumerable = true; + ObjectDefineProperty(proto, members[i], desc); + } +} +finishInterface(TextEncoder, "TextEncoder", [ + "encoding", + "encode", + "encodeInto", +]); +finishInterface(TextDecoder, "TextDecoder", [ + "encoding", + "fatal", + "ignoreBOM", + "decode", +]); + +module.exports = { TextEncoder, TextDecoder };