[jwrosewell]

Problem

The release draft #83 failed its performance gate on Ubuntu_ARM_Node_24 in build 43335666904, measuring 30,352 detections a second against an average of 33,900 and a floor of 30,510 taken from the nightly builds of main.

Every measurement below ran on one ubuntu-22.04-arm runner per run, with each tree built whole in its own folder, the same TAC Enterprise data file of 109,162,108 bytes, the same 20,000 User-Agent evidence file of 2,726,867 bytes, and the order of the arms rotating each round. Figures are detections a second, as the performance example prints them.

1. rc is slower than main by more than the spread of the rounds. In run 43344516953, main at 384bff2d measured a median of 31,842.86 (31,383.60 to 32,516.08) and rc at 20868944 a median of 28,302.73 (27,342.10 to 28,899.65), every round of rc under every round of main.
2. The largest step is #53. The same run, and the bisect 43347107318, measured points along rc with the log masking line that #85 took off the detection path taken off them too. Every round at Give each web integration element its own switch, put Sequence first, and give it element data #53 (28,086.38 to 29,065.50) was under every round of the commit before it (29,975.51 to 31,872.31), and the three commits after it were flat.
3. Swapping that one change alone moves the gap both ways. Run 43351290526, seven rounds. The log's git diff --stat for each patched arm lists the named file and nothing else apart from the cxx submodule, which the unpatched rc arm lists too.

Arm	Median	Lowest	Highest
main	34,151.47	31,757.67	35,107.60
main with #53's engine filter alone	32,057.57	30,148.30	32,743.53
rc	29,620.76	28,958.61	36,499.98
rc with #53's engine filter taken out	33,077.88	31,671.02	33,918.88

Adding the filter to main costs 6.1 per cent, and taking it out of rc gains 11.7 per cent, which puts rc inside main's range.

The change in question is where #53 made the device detection engine pass the native engine only the evidence that engine reads, in deviceDetectionOnPremise.js lines 617 to 624, and the IP Intelligence engine the same in ipIntelligenceOnPremise.js lines 472 to 477. That runs EvidenceKeyFilter.filterEvidence, which calls BasicListEvidenceKeyFilter.filterEvidenceKey for each evidence key, and that function puts every key in the engine's list, and the key being checked once for each of them, in lower case on every call. Evidence.add already makes that check for every element as each value is added, so since #53 a detection walks the list at least twice.

What the specification says

evidence.md line 10:

Keys MUST be case-insensitive and will usually be split into <prefix>.<field>.

advertize-accepted-evidence.md lines 10 to 17:

For the majority of Flow Elements, simply returning a list of Evidence values they can use would be sufficient. For example, the Device Detection Engine will accept query.user-agent, header.user-agent, query.sec-ch-ua, header.sec-ch-ua, etc.

Where listing the keys is impractical, a Flow Element can use a filter function that returns true if the supplied key is accepted by the Element and false otherwise.

The specification asks for a list of keys compared ignoring letter case, and leaves how the comparison is made to the implementation.

What changed

basicListEvidenceKeyFilter.js keeps the list's keys in lower case in a Set, made on the first check and made again whenever the list is replaced, grows, shrinks or has a key changed. A check then puts only the key being checked in lower case and looks it up.

• Which keys are kept is unchanged, including for a list changed after the filter was made, because list is public and a caller may change it. Knowing whether it changed costs one comparison of each key's reference per check, against two calls to toLowerCase for each key before.
• Give each web integration element its own switch, put Sequence first, and give it element data #53's filter stays where it is, so the native engines are still passed only the evidence they read.
• types/basicListEvidenceKeyFilter.d.ts gains the #private; line that tsc writes for the private fields. The green run regenerates it and finds no difference.

Tests

tests/evidenceKeyFilterCost.test.js gives each key in a list its own toLowerCase that counts its calls.

1. checking keys puts the list in lower case once, 200 checks against a list of 10 keys, asking for 10.
2. a detection does not put the list in lower case again, 25 detections through a pipeline whose element reads its evidence through filterEvidence as the engines do, asking for 10 and checking each detection kept header.user-agent alone.
3. Six tests that pass before and after the change, being letter case in either direction, a key not in the list, an empty list, a key added to the list after a check, a key changed in place, a replaced list, and filterEvidence keeping the listed keys with their values.

Runs

• Red 43357744161 on 1d851e52, which is rc with the new test file and a throwaway test workflow and no production change. 2 failed, 1,140 passed, 1,142 total. The two are tests 1 and 2 above, counting 2000 and 750 where 10 is asked.
• Green 43357757260 on 69d18342, which is this head with the same throwaway workflow. 1,142 passed, 1,142 total. ESLint reports 0 errors and 188 warnings on both runs, none of them in the two changed source files.
• Measurement 43355023653, nine rounds, on 76eb8e7a, which differs from this head only in one blank line in the test file and the place of the #private; line in the type declaration.

Arm	Median	Lowest	Highest
main	33,711.49	33,012.51	35,009.76
rc	30,437.05	29,438.36	38,095.65
this draft	34,869.04	33,835.78	35,511.90

The draft measures 3.4 per cent above main's median and inside main's range.

• Full build 43356814715, green. All 7 Setting 'PullRequestSha' lines read 818c435437b7fbdc1309ab4195eba0859075a99b, which is this head, Configure printed "Left out 2 macOS entries", and each of the 6 jobs ran two suites, being 1,355 passed with 2 skipped and 132 passed with 4 skipped.

macOS was left out of this draft's full build under the 9 November decision. The rc head is built with macOS before it merges into main.

What remains

1. #71 costs a little as well. Against its own parent it measured 3.0 per cent lower with the rounds apart in run 43347107318 and 5.0 per cent lower with them overlapping in run 43351290526, and applied alone to main it measured 2.9 per cent lower with the rounds overlapping. This draft more than makes that up, as the measurement above shows, and it does not change Read and store evidence keys without regard to letter case #71.
2. IP Intelligence takes the same path, and no IP Intelligence measurement was made, so its gain is not known.
3. filterEvidenceKey returns undefined rather than false for a key it does not keep, as it did before. The specification says false. This draft leaves that alone, so that cost is the only thing it changes.
4. The release draft Release candidate #83 needs its build and its performance gate run again once this merges.