diff --git a/CHANGELOG.md b/CHANGELOG.md index 0653d69..5c4a94a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -30,14 +30,28 @@ Route logging is on by default and costs roughly 4x the rest of a trivial dispatch, so this is worth setting on hot routes. -- New `benchmark/` suite covering the request path, with a layered breakdown so - a regression can be attributed rather than just observed. See - `benchmark/README.md`. +- New `benchmark/` suites, with layered breakdowns so a regression can be + attributed rather than just observed. See `benchmark/README.md`. ``` - dart run benchmark/bones_api_benchmark.dart + dart run benchmark/bones_api_benchmark.dart # request path + dart run benchmark/json_benchmark.dart # JSON request/response + dart run benchmark/db_benchmark.dart # DB entity path ``` + They record that query parsing is well cached (~300x cheaper than parsing) + and SQL generation is under a microsecond. JSON encoding already runs close + to a bare `dart:convert` encode, and request bodies use `dart:convert` + directly, so no JSON optimization came out of that suite. + +- `DBSQLMemoryAdapter` now answers a select by ID with a direct lookup in the + table `Map`, which is already keyed by ID, instead of scanning it. A miss + still falls through to the scan, so results are unchanged. + + `selectByID` was O(rows) and is now flat: 7.8us -> 7.1us at 10 rows, + 9.7us -> 7.2us at 50, and 25.3us -> 7.2us at 400. This mostly speeds up the + test suite and development, since the memory adapter is where those run. + ## 1.14.0 - New `DBSQLiteAdapter`: an embedded SQLite DB adapter, backed by the diff --git a/benchmark/README.md b/benchmark/README.md index 9ca244a..b31c644 100644 --- a/benchmark/README.md +++ b/benchmark/README.md @@ -1,12 +1,19 @@ # Benchmarks -In-process benchmarks for the request path: building an `APIRequest`, resolving -the module/route, dispatching through `APIRoot.call`, and serializing the -response payload. No socket and no HTTP client are involved, so what is -measured is the framework's own overhead. +In-process benchmarks for the framework's own overhead — no socket, no HTTP +client, and (for the DB suite) an in-memory adapter, so what is measured is +`bones_api` itself rather than I/O. + +| Suite | Covers | +|---|---| +| `bones_api_benchmark.dart` | The request path: `APIRequest`, routing, `APIRoot.call` | +| `json_benchmark.dart` | JSON request/response encoding and decoding | +| `db_benchmark.dart` | The DB entity path against `DBSQLMemoryAdapter` | ```bash dart run benchmark/bones_api_benchmark.dart +dart run benchmark/json_benchmark.dart +dart run benchmark/db_benchmark.dart ``` To compare a change, record a baseline on your machine first — throughput is @@ -55,3 +62,80 @@ Disable it per route when throughput matters more than the audit trail: ```dart routes.get('ping', handler, config: const APIRouteConfig(log: false)); ``` + +## Where the DB and JSON time goes + +Recorded once on one machine, as orders of magnitude rather than targets. + +**JSON** is in reasonable shape. `Json.encodeToSink` — the response path — runs +close to a bare `dart:convert` encode of the same value (~1.5us vs ~1.2us for a +small map), and is *faster* for larger payloads because it writes bytes to a +sink instead of building a `String`. Request bodies are parsed with +`dart:convert` directly, so there is no `bones_api` layer to remove there. + +**DB**, per `db_benchmark.dart` (50 rows): + +| | us/op | +|---|---| +| `ConditionParseCache.parseQuery` (cached) | 0.009 | +| `Entity.toJson` | 0.074 | +| `generateSelectSQL` | 0.81 | +| `EntityHandler.createFromMap` | 1.11 | +| `ConditionParser.parse` (shared parser) | 2.67 | +| `Transaction.executeBlock` (empty) | 2.40 | +| `repository.selectByID` | 7.2 | +| `repository.selectByQuery` | 20.4 | + +The two things a query is *assumed* to be expensive for are not: query parsing +is cached (~300x cheaper than parsing), and SQL generation is under a +microsecond. + +**Read `selectByQuery` with the row count in mind.** `DBSQLMemoryAdapter` +answers a non-ID condition by scanning the table `Map` and evaluating the +condition per row, so that number is mostly the scan, not framework overhead. +Use `--rows=N` to separate the two: + +```bash +dart run benchmark/db_benchmark.dart --rows=400 +``` + +| rows | `selectByQuery` | +|---|---| +| 10 | 10.9us | +| 50 | 20.4us | +| 400 | 105.9us | + +Linear: roughly 8.5us fixed plus ~0.24us per row. Only the fixed part is +framework cost shared with a real SQL adapter, where the database does the +filtering — so do not read 20us as "the cost of a query" in production. + +Of that fixed part, an empty `Transaction.executeBlock` is ~2.3us. That is the +floor under every DB operation, and the largest remaining fixed cost. + +### What the transaction floor is *not* + +Measured and ruled out, so this does not have to be repeated: + +| | us/op | +|---|---| +| `Transaction()` constructor | 0.068 | +| `Zone.current.fork()` | 0.032 | +| `asyncTry` (sync block, `onError` + `onFinally`) | 0.030 | +| `Completer()` | 0.011 | +| commit logging (`root=INFO` vs `OFF`) | ~0.23 | +| **`Transaction.executeBlock` (empty)** | **2.3** | + +A nested `executeBlock` adds only ~0.04us, since it short-circuits to the +enclosing transaction — that path is already optimal. + +None of the named pieces accounts for the total. What is left is the async +plumbing itself: the commit path threads through several `resolveMapped` hops, +completers and zone-field reads, and an `await` of an already-completed value +costs ~0.15us on its own. Cutting it means restructuring the synchronous path +so it stops allocating futures, which is a real refactor of core transaction +code rather than an incremental fix — worth doing deliberately, with a +profiler, not opportunistically. + +Note `ConditionParser` builds its PetitParser grammar lazily on first use +(~125us). `bones_api` holds it in a `static final`, so this is a one-off +startup cost — but constructing a `ConditionParser` per query would not be. diff --git a/benchmark/db_benchmark.dart b/benchmark/db_benchmark.dart new file mode 100644 index 0000000..dddab0c --- /dev/null +++ b/benchmark/db_benchmark.dart @@ -0,0 +1,254 @@ +import 'dart:io'; + +import 'package:bones_api/bones_api.dart'; + +import 'src/bench_runner.dart'; + +/// Benchmarks for the DB entity path, against `DBSQLMemoryAdapter`. +/// +/// ``` +/// dart run benchmark/db_benchmark.dart +/// ``` +/// +/// The memory adapter keeps I/O out of the picture, so most of what is left is +/// framework: condition parsing, SQL generation, and mapping rows to and from +/// entities. +/// +/// One exception, and it is easy to misread: the memory adapter answers a +/// non-ID condition by *scanning* the table and evaluating the condition per +/// row, which a real SQL adapter does not do. `selectByQuery` is therefore +/// linear in the row count — see `--rows=N` below and `README.md`. +Future main(List args) async { + var provider = _BenchProvider(); + await provider.ensureInitialized(); + + var adapter = await provider.adapter; + var repository = provider.userRepository; + + // `DBSQLMemoryAdapter` answers a non-ID condition with a full scan of the + // table `Map`, so the row count is a parameter of the result, not a detail. + // Vary it with `--rows=N` to separate per-row cost from fixed cost. + var rows = + int.tryParse( + args + .firstWhere((a) => a.startsWith('--rows='), orElse: () => '') + .split('=') + .last, + ) ?? + 50; + + for (var i = 1; i <= rows; ++i) { + await repository.store( + BenchUser('user$i', 'user$i@example.com', i, id: null), + ); + } + + print('-- table rows: $rows'); + + var runner = BenchRunner(); + + // ----------------------------------------------------------------------- + // Entity <-> Map, the row mapping every adapter performs. + // ----------------------------------------------------------------------- + + var entity = BenchUser('joe', 'joe@example.com', 42, id: 1); + var row = { + 'id': 1, + 'name': 'joe', + 'email': 'joe@example.com', + 'level': 42, + }; + + runner.run('Entity.toJson', () => entity.toJson()); + + await runner.runAsync( + 'EntityHandler.createFromMap', + () => benchUserEntityHandler.createFromMap(row), + ); + + // ----------------------------------------------------------------------- + // Query parsing (cached by `ConditionParseCache`) and SQL generation. + // ----------------------------------------------------------------------- + + // A `ConditionParser` builds its PetitParser grammar lazily on first use, + // which costs ~125us. `bones_api` holds it in a `static final`, so that is a + // one-off; re-creating one per query would not be. + var parser = ConditionParser(); + parser.parse(' email == ? '); // build the grammar outside the measurement. + + runner.run( + 'ConditionParser.parse (shared parser)', + () => parser.parse(' email == ? '), + ); + + var parseCache = ConditionParseCache(); + + runner.run( + 'ConditionParseCache.parseQuery (cached)', + () => parseCache.parseQuery(' email == ? '), + ); + + var transaction = Transaction.autoCommit(); + var condition = parseCache.parseQuery(' email == ? '); + + await runner.runAsync( + 'generateSelectSQL: email == ?', + () => adapter.generateSelectSQL( + transaction, + 'BenchUser', + 'bench_user', + condition, + parameters: {'email': 'user7@example.com'}, + ), + ); + + // ----------------------------------------------------------------------- + // Repository operations, end to end through the adapter. + // + // `selectByID` is a keyed lookup and so is flat in the row count. + // `selectByQuery` is not: it is roughly a fixed cost plus a per-row scan, + // and only the fixed part is shared with a real SQL adapter. + // ----------------------------------------------------------------------- + + await runner.runAsync( + 'Transaction.executeBlock (empty)', + () => Transaction.executeBlock((t) => 1), + ); + + await runner.runAsync( + 'repository.selectByID', + () => repository.selectByID(7), + ); + + await runner.runAsync( + 'repository.selectByQuery: email == ?', + () => repository.selectByQuery( + ' email == ? ', + parameters: {'email': 'user7@example.com'}, + ), + ); + + await runner.runAsync( + 'repository.selectAll (all rows)', + () => repository.select(ConditionANY()), + ); + + runner.report(baseline: _baseline); + + if (args.contains('--emit-baseline')) { + print('const _baseline = {'); + runner.asBaseline().forEach((k, v) { + print(" '$k': ${v.toStringAsFixed(0)},"); + }); + print('};'); + } + + provider.close(); + exit(0); +} + +/// See the note on `_baseline` in `bones_api_benchmark.dart`. +const _baseline = {}; + +final benchUserEntityHandler = GenericEntityHandler( + instantiatorDefault: BenchUser.empty, + instantiatorFromMap: BenchUser.fromMap, + type: BenchUser, + typeName: 'BenchUser', +); + +class BenchUser extends Entity { + int? id; + String name; + String email; + int level; + + BenchUser(this.name, this.email, this.level, {this.id}); + + BenchUser.empty() : this('', '', 0); + + static BenchUser fromMap(Map map) => BenchUser( + map.getAsString('name') ?? '', + map.getAsString('email') ?? '', + map.getAsInt('level') ?? 0, + id: map['id'] as int?, + ); + + @override + String get idFieldName => 'id'; + + @override + List get fieldsNames => const [ + 'id', + 'name', + 'email', + 'level', + ]; + + @override + V? getField(String key) => switch (key) { + 'id' => id as V?, + 'name' => name as V?, + 'email' => email as V?, + 'level' => level as V?, + _ => null, + }; + + @override + TypeInfo? getFieldType(String key) => switch (key) { + 'id' => TypeInfo.tInt, + 'name' => TypeInfo.tString, + 'email' => TypeInfo.tString, + 'level' => TypeInfo.tInt, + _ => null, + }; + + @override + void setField(String key, V? value) { + switch (key) { + case 'id': + id = value as int?; + case 'name': + name = (value as String?) ?? ''; + case 'email': + email = (value as String?) ?? ''; + case 'level': + level = (value as int?) ?? 0; + } + } + + @override + Map toJson() => { + 'id': id, + 'name': name, + 'email': email, + 'level': level, + }; + + @override + bool operator ==(Object other) => + identical(this, other) || (other is BenchUser && id == other.id); + + @override + int get hashCode => id.hashCode; +} + +class _BenchProvider extends DBSQLEntityRepositoryProvider { + late final DBSQLEntityRepository userRepository; + + @override + Map get adapterConfig => {'sql.memory': {}}; + + @override + FutureOr buildAdapter() => + DBSQLMemoryAdapter(parentRepositoryProvider: this); + + @override + List buildRepositories(DBSQLAdapter adapter) => [ + userRepository = DBSQLEntityRepository( + adapter, + 'bench_user', + benchUserEntityHandler, + ), + ]; +} diff --git a/benchmark/json_benchmark.dart b/benchmark/json_benchmark.dart new file mode 100644 index 0000000..eeda6e7 --- /dev/null +++ b/benchmark/json_benchmark.dart @@ -0,0 +1,135 @@ +import 'dart:convert' as dart_convert; +import 'dart:io'; + +import 'package:bones_api/bones_api.dart'; + +import 'src/bench_runner.dart'; + +/// Benchmarks for the JSON request/response path. +/// +/// ``` +/// dart run benchmark/json_benchmark.dart +/// ``` +/// +/// `Json.encodeToSink(..., toEncodable: ...)` is what `APIServer` uses to turn +/// a response payload into bytes, so it is measured here in exactly that shape, +/// next to a bare `dart:convert` encode of the same value for reference. +Future main(List args) async { + var runner = BenchRunner(); + + var small = _smallPayload(); + var list = List.generate(50, (i) => _smallPayload()); + var jsonSmall = dart_convert.json.encode(small); + var jsonList = dart_convert.json.encode(list); + + // ----------------------------------------------------------------------- + // Reference: `dart:convert` with no bones_api machinery. + // ----------------------------------------------------------------------- + + runner.run( + 'dart:convert encode: small map', + () => dart_convert.json.encode(small), + ); + + runner.run( + 'dart:convert encode: 50 maps', + () => dart_convert.json.encode(list), + ); + + // ----------------------------------------------------------------------- + // `Json.encode`, the default (cached) encoder path. + // ----------------------------------------------------------------------- + + runner.run('Json.encode: small map', () => Json.encode(small)); + runner.run('Json.encode: 50 maps', () => Json.encode(list)); + + // ----------------------------------------------------------------------- + // The response path: `encodeToSink` with a `toEncodable`. + // ----------------------------------------------------------------------- + + runner.run('Json.encodeToSink: small map [response path]', () { + var sink = _BytesSink(); + Json.encodeToSink( + small, + sink, + toEncodable: ReflectionFactory.toJsonEncodable, + ); + return sink.length; + }); + + runner.run('Json.encodeToSink: 50 maps [response path]', () { + var sink = _BytesSink(); + Json.encodeToSink( + list, + sink, + toEncodable: ReflectionFactory.toJsonEncodable, + ); + return sink.length; + }); + + // Same output, but through the default encoder (no `toEncodable`), to show + // the cost attributable purely to building a per-call encoder. + runner.run('Json.encodeToSink: small map [no toEncodable]', () { + var sink = _BytesSink(); + Json.encodeToSink(small, sink); + return sink.length; + }); + + // ----------------------------------------------------------------------- + // Decoding: the request payload path. + // + // `APIServer` parses a JSON request body with `dart:convert` directly + // (`bones_api_server.dart`, `_resolvePayloadFromString`), so that is what is + // measured; `Json.decode` is the entity-aware decoder used elsewhere. + // ----------------------------------------------------------------------- + + runner.run( + 'dart:convert decode: small map [request path]', + () => dart_convert.json.decode(jsonSmall), + ); + + runner.run( + 'dart:convert decode: 50 maps [request path]', + () => dart_convert.json.decode(jsonList), + ); + + runner.report(baseline: _baseline); + + if (args.contains('--emit-baseline')) { + print('const _baseline = {'); + runner.asBaseline().forEach((k, v) { + print(" '$k': ${v.toStringAsFixed(0)},"); + }); + print('};'); + } + + exit(0); +} + +Map _smallPayload() => { + 'id': 12345, + 'name': 'Joe Smith', + 'email': 'joe@example.com', + 'enabled': true, + 'score': 98.6, + 'tags': ['alpha', 'beta', 'gamma'], + 'address': { + 'street': '123 Main St', + 'city': 'Springfield', + 'state': 'NY', + 'zip': '12345', + }, +}; + +/// See the note on `_baseline` in `bones_api_benchmark.dart`. +const _baseline = {}; + +class _BytesSink implements Sink> { + int length = 0; + + @override + void add(List data) => length += data.length; + + @override + void close() {} +} diff --git a/lib/src/bones_api_entity_db_memory.dart b/lib/src/bones_api_entity_db_memory.dart index 0eab970..37df242 100644 --- a/lib/src/bones_api_entity_db_memory.dart +++ b/lib/src/bones_api_entity_db_memory.dart @@ -842,9 +842,37 @@ class DBSQLMemoryAdapter extends DBSQLAdapter final condition = sql.condition; - Iterable> itr; + Iterable>? itr; + + // The table is a `Map` keyed by ID, so a select by ID does not need to + // scan it. Without this, every `selectByID` was O(rows): measured 7.8us + // at 10 rows and 25.3us at 400. + // + // A miss falls through to the scan below, so a `ConditionID` whose value + // does not match a key exactly (a `String` '7' against an `int` 7, say) + // still resolves exactly as before — only slower, as it always was. + if (condition is ConditionID) { + var parametersByPlaceholder = sql.parametersByPlaceholder; + + var id = condition.resolveIDValue( + parameters: + sql.namedParameters ?? + (parametersByPlaceholder.isNotEmpty + ? parametersByPlaceholder + : sql.positionalParameters), + ); + + if (id != null) { + var entry = map[id]; + if (entry != null) { + itr = [entry]; + } + } + } - if (condition == null) { + if (itr != null) { + // Resolved by ID. + } else if (condition == null) { itr = map.values; } else if (tableScheme == null || (tableScheme.fieldsReferencedTablesLength == 0 &&