Add codspeed benchmarking in a dedicated CI job

[sirosen]

Also add a new tox config to support running codspeed locally and in CI.

[codspeed-hq]

Congrats! CodSpeed is installed 🎉

🆕 18 new benchmarks were detected.

You will start to see performance impacts in the reports once the benchmarks are run from your default branch.

Detected benchmarks

• test_deeply_nested_object_encoding[10w-5d]: 952.8 ms
• test_deeply_nested_object_encoding[1w-1000d]: 9.3 ms
• test_deeply_nested_object_encoding[1w-1d]: 117.1 µs
• test_deeply_nested_object_encoding[1w-5d]: 154.3 µs
• test_deeply_nested_object_encoding[5w-1d]: 149.9 µs
• test_deeply_nested_object_encoding[5w-5d]: 33.3 ms
• test_typical_object_encoding[auth.get_consents]: 318.8 µs
• test_typical_object_encoding[flows.get_run_logs]: 547.7 µs
• test_typical_object_encoding[transfer.create_tunnel]: 329.6 µs
• test_deep_scope_parsing[10]: 881.8 µs
• test_deep_scope_parsing[100]: 7.8 ms
• test_deep_scope_parsing[1000]: 155.3 ms
• test_deep_scope_parsing[2000]: 478.5 ms
• test_deep_scope_parsing[3000]: 1 s
• test_deep_scope_parsing[4000]: 1.6 s
• test_deep_scope_parsing[5000]: 2.6 s
• test_wide_scope_parsing[10000]: 539.6 ms
• test_wide_scope_parsing[5000]: 270.4 ms

---

[derek-globus]

First blush with CodSpeed, I'd like to understand better how you expect this tool should fit into the sdk ecosystem.

[derek-globus]

Does this just generate an artifact that we can inspect to more easily identify where a performance impact occurred? Or will this do something more aggressive like blocking PRs that hurt performance against prior benchmarks?

[sirosen]

It doesn't generate profiles or traces, but it does some sophisticated emulation work to produce stable benchmark results, which it then tracks in a project history.

It can comment on PRs which have significant performance impacts. And it defaults to blocking PRs which show a regression but I have explicitly turned that off for us. Here's how I configured it for now:

So if we have a perf regression it will tell us.

---

For a sample of what the PR comment looks like, I was experimenting with this in slyp over the weekend a little:
sirosen/slyp#34

[derek-globus]

Okay, thanks; that's helpful

[sirosen]

FWIW, I'm also new to codspeed. I'd like us to explore it, as a harness for running and tracking benchmark history, and improving our org-wide discipline. If it doesn't work out, we can remove it.

[sirosen]

I fixed the CI failure from codspeed; I was previously wrong about the cause. The tox env needs passenv = CODSPEED_* so that codspeed can pass info via those environment variables.

[derek-globus]

I'm onboard to give it a try. If it becomes onerous, I may advocate that we rethink our usage, but I can see the benefit to catching performance impacts before they make it to source code.

[sirosen]

FWIW, I see the value more as giving us a better way of talking about performance impacts. If I want to refactor something to be 3x larger but 10x faster, I should be able to prove that to my reviewers with a new benchmark. And we want to encode the idea that "this code is longer and more complex than other solutions because it's faster" into our testing process.