wasmtime: add options for configuring between winch and cranelift

[leonm1]

Winch has significantly faster load times than cranelift, with negligible difference in runtime performance:

With this change, we will be ready to switch Envoy over to using Wasmtime with Winch by default.

[PiotrSikora]

Definitely +1 for adding winch, but:

1. This is added as a configuration option, which doesn't match how other engine flavors are exposed in this library (e.g. wamr-interp, wamr-jit, wamr-aot).
2. I'm not sure if the optimization levels have any meaningful impact, so they might be only adding unnecessary noise and maintenance burden. Do you have any benchmarks?

[leonm1]

Indeed I do have benchmarks of the different opt levels:

wasmtime with cranelift and kSpeedAndSize

-----------------------------------------------------------------------------------------
Benchmark                               Time             CPU   Iterations UserCounters...
-----------------------------------------------------------------------------------------
Bench_wasmtime.CreateVm             77228 ns        77224 ns         9121 PeakVmRSS_KiB=111.032k
Bench_wasmtime.LoadPlugin      4718466121 ns   4717185369 ns            1 PeakVmRSS_KiB=230.44k
Bench_wasmtime.StartPlugin          30380 ns        30367 ns        23079 CompiledSizeB=22.9294Mi LogsSizeB=0 PeakVmRSS_KiB=191.828k WasmMemoryB=6Mi WasmSizeB=5.6383Mi
Bench_wasmtime.PluginLifecycle     110718 ns       110698 ns         6157 CompiledSizeB=22.9294Mi LogsSizeB=0 PeakVmRSS_KiB=191.828k WasmMemoryB=4.3125Mi WasmSizeB=5.6383Mi
Bench_wasmtime.StreamLifecycle      77979 ns        77958 ns         8874 CompiledSizeB=22.9294Mi LogsSizeB=0 PeakVmRSS_KiB=190.2k WasmMemoryB=4.3125Mi WasmSizeB=5.6383Mi
Bench_wasmtime.WithValidJwt        466003 ns       466190 ns         1476 CompiledSizeB=22.9294Mi LogsSizeB=0 PeakVmRSS_KiB=192.38k WasmMemoryB=6Mi WasmSizeB=5.6383Mi
Bench_wasmtime.NoJwt                87939 ns        88037 ns         7787 CompiledSizeB=22.9294Mi LogsSizeB=39 PeakVmRSS_KiB=191.564k WasmMemoryB=6Mi WasmSizeB=5.6383Mi
Bench_wasmtime.InvalidJwt          107567 ns       107739 ns         6490 CompiledSizeB=22.9294Mi LogsSizeB=39 PeakVmRSS_KiB=191.788k WasmMemoryB=6Mi WasmSizeB=5.6383Mi
Bench_wasmtime.NotAllowedJwt       475983 ns       476204 ns         1457 CompiledSizeB=22.9294Mi LogsSizeB=23 PeakVmRSS_KiB=192.276k WasmMemoryB=6Mi WasmSizeB=5.6383Mi

wasmtime with cranelift and kSpeed

-----------------------------------------------------------------------------------------
Benchmark                               Time             CPU   Iterations UserCounters...
-----------------------------------------------------------------------------------------
Bench_wasmtime.CreateVm             77586 ns        77552 ns         9055 PeakVmRSS_KiB=101.844k
Bench_wasmtime.LoadPlugin      4657859594 ns   4656706213 ns            1 PeakVmRSS_KiB=227.38k
Bench_wasmtime.StartPlugin          29929 ns        29917 ns        23053 CompiledSizeB=22.9294Mi LogsSizeB=0 PeakVmRSS_KiB=192.92k WasmMemoryB=6Mi WasmSizeB=5.6383Mi
Bench_wasmtime.PluginLifecycle     117656 ns       117595 ns         6080 CompiledSizeB=22.9294Mi LogsSizeB=0 PeakVmRSS_KiB=188.876k WasmMemoryB=4.3125Mi WasmSizeB=5.6383Mi
Bench_wasmtime.StreamLifecycle      78716 ns        78682 ns         8882 CompiledSizeB=22.9294Mi LogsSizeB=0 PeakVmRSS_KiB=188.048k WasmMemoryB=4.3125Mi WasmSizeB=5.6383Mi
Bench_wasmtime.WithValidJwt        479298 ns       479359 ns         1376 CompiledSizeB=22.9294Mi LogsSizeB=0 PeakVmRSS_KiB=188.584k WasmMemoryB=6Mi WasmSizeB=5.6383Mi
Bench_wasmtime.NoJwt                89346 ns        89404 ns         7770 CompiledSizeB=22.9294Mi LogsSizeB=39 PeakVmRSS_KiB=189.972k WasmMemoryB=6Mi WasmSizeB=5.6383Mi
Bench_wasmtime.InvalidJwt          108663 ns       108781 ns         6413 CompiledSizeB=22.9294Mi LogsSizeB=39 PeakVmRSS_KiB=189.976k WasmMemoryB=6Mi WasmSizeB=5.6383Mi
Bench_wasmtime.NotAllowedJwt       473127 ns       473237 ns         1481 CompiledSizeB=22.9294Mi LogsSizeB=23 PeakVmRSS_KiB=190.444k WasmMemoryB=6Mi WasmSizeB=5.6383Mi

wasmtime with cranelift and kNone

-----------------------------------------------------------------------------------------
Benchmark                               Time             CPU   Iterations UserCounters...
-----------------------------------------------------------------------------------------
Bench_wasmtime.CreateVm             77361 ns        77319 ns         9061 PeakVmRSS_KiB=105.212k
Bench_wasmtime.LoadPlugin      3942787317 ns   3941735586 ns            1 PeakVmRSS_KiB=254.236k
Bench_wasmtime.StartPlugin          31743 ns        31722 ns        21817 CompiledSizeB=24.4998Mi LogsSizeB=0 PeakVmRSS_KiB=200.068k WasmMemoryB=6Mi WasmSizeB=5.6383Mi
Bench_wasmtime.PluginLifecycle     113513 ns       113479 ns         6155 CompiledSizeB=24.4998Mi LogsSizeB=0 PeakVmRSS_KiB=206.224k WasmMemoryB=4.3125Mi WasmSizeB=5.6383Mi
Bench_wasmtime.StreamLifecycle      78507 ns        78496 ns         8899 CompiledSizeB=24.4998Mi LogsSizeB=0 PeakVmRSS_KiB=196.52k WasmMemoryB=4.3125Mi WasmSizeB=5.6383Mi
Bench_wasmtime.WithValidJwt        490178 ns       490184 ns         1468 CompiledSizeB=24.4998Mi LogsSizeB=0 PeakVmRSS_KiB=199.284k WasmMemoryB=6Mi WasmSizeB=5.6383Mi
Bench_wasmtime.NoJwt                91660 ns        91713 ns         7686 CompiledSizeB=24.4998Mi LogsSizeB=39 PeakVmRSS_KiB=205.184k WasmMemoryB=6Mi WasmSizeB=5.6383Mi
Bench_wasmtime.InvalidJwt          111096 ns       111230 ns         6184 CompiledSizeB=24.4998Mi LogsSizeB=39 PeakVmRSS_KiB=204.948k WasmMemoryB=6Mi WasmSizeB=5.6383Mi
Bench_wasmtime.NotAllowedJwt       469886 ns       469985 ns         1471 CompiledSizeB=24.4998Mi LogsSizeB=23 PeakVmRSS_KiB=207.556k WasmMemoryB=6Mi WasmSizeB=5.6383Mi

wasmtime with winch

-----------------------------------------------------------------------------------------
Benchmark                               Time             CPU   Iterations UserCounters...
-----------------------------------------------------------------------------------------
Bench_wasmtime.CreateVm             77016 ns        77001 ns         9073 PeakVmRSS_KiB=118.216k
Bench_wasmtime.LoadPlugin       865018440 ns    864909132 ns            1 PeakVmRSS_KiB=280.46k
Bench_wasmtime.StartPlugin          39353 ns        39398 ns        17753 CompiledSizeB=28.8986Mi LogsSizeB=0 PeakVmRSS_KiB=227.944k WasmMemoryB=6Mi WasmSizeB=5.6383Mi
Bench_wasmtime.PluginLifecycle     122342 ns       122294 ns         5669 CompiledSizeB=28.8986Mi LogsSizeB=0 PeakVmRSS_KiB=228.008k WasmMemoryB=4.3125Mi WasmSizeB=5.6383Mi
Bench_wasmtime.StreamLifecycle      79106 ns        79083 ns         8853 CompiledSizeB=28.8986Mi LogsSizeB=0 PeakVmRSS_KiB=225.992k WasmMemoryB=4.3125Mi WasmSizeB=5.6383Mi
Bench_wasmtime.WithValidJwt        823562 ns       823579 ns          846 CompiledSizeB=28.8986Mi LogsSizeB=0 PeakVmRSS_KiB=228.1k WasmMemoryB=6Mi WasmSizeB=5.6383Mi
Bench_wasmtime.NoJwt                91138 ns        91133 ns         7744 CompiledSizeB=28.8986Mi LogsSizeB=39 PeakVmRSS_KiB=227.668k WasmMemoryB=6Mi WasmSizeB=5.6383Mi
Bench_wasmtime.InvalidJwt          115471 ns       115584 ns         6089 CompiledSizeB=28.8986Mi LogsSizeB=39 PeakVmRSS_KiB=227.892k WasmMemoryB=6Mi WasmSizeB=5.6383Mi
Bench_wasmtime.NotAllowedJwt       826810 ns       826564 ns          841 CompiledSizeB=28.8986Mi LogsSizeB=23 PeakVmRSS_KiB=228.06k WasmMemoryB=6Mi WasmSizeB=5.6383Mi

Notably, the CompiledSizeB is the primary difference between the opt levels. I did not see a difference between kSpeedAndSize and kSpeed, which indicates they might be effectively the same. For this particular wasm module, which was not significantly faster to load, but for smaller wasm modules, the difference was significant (as noted above). (I had modified the suite to only test precompiled modules, when removing that bit, the difference is quite extreme. The results above have been updated.)

Would you prefer that we default to the highest opt level for cranelift, and provide only winch as the lighter option?

[leonm1]

Re: winch as a different compilation target:

Winch is already included in the cranelift build, as it simply uses cranelift_codegen without any intermediate representation or optimization passes. It's also offered upstream as a runtime config option (though you can avoid compiling certain features if you don't want them).

With the addition of the proxy_wasm::WasmVm::serialize API, it is very appealing for us to be able to instantiate a Wasmtime WasmVm with cranelift for in-process AOT compilation, but use a WasmVm with Winch for its potentially significantly faster load time when performing a cold start.

[PiotrSikora]

Random though, but perhaps we should default (or maybe even hardcode?) to Winch for .wasm bytecode and use Cranelift only for precompiled code? The Cranelift compilation time is high enough that it doesn't make any sense to use it at runtime.

[PiotrSikora]

Thanks! LGTM