Use zlib-ng (fast!) rather than mainline stale zlib in binary releases

@gpshead

BPO	47193
Nosy	@gpshead, @pfmoore, @tjguk, @zware, @zooba, @corona10, @arhadthedev

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2022-04-01.19:19:24.173>
labels = ['3.11', 'OS-windows', 'performance']
title = 'Use zlib-ng rather than zlib in binary releases'
updated_at = <Date 2022-04-02.09:48:33.061>
user = 'https://github.com/gpshead'

bugs.python.org fields:

activity = <Date 2022-04-02.09:48:33.061>
actor = 'corona10'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Windows']
creation = <Date 2022-04-01.19:19:24.173>
creator = 'gregory.p.smith'
dependencies = []
files = []
hgrepos = []
issue_num = 47193
keywords = []
message_count = 1.0
messages = ['416508']
nosy_count = 7.0
nosy_names = ['gregory.p.smith', 'paul.moore', 'tim.golden', 'zach.ware', 'steve.dower', 'corona10', 'arhadthedev']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue47193'
versions = ['Python 3.11']

Linked PRs

Checking internally for local patches to zlib-ng, this change zlib-ng/zlib-ng@ce6789c appears to fix that problem. I guess it hasn't landed in a zlib-ng release yet. (I don't see it in the 2.0.x branch)

Lets make sure that lands in zlib-ng before using it in our releases I guess.

Otherwise: I don't see any Python code using zlib.error that'd care (internally and in a large corpus of third party OSS python code). It is mostly only ever caught to bail out or follows this pattern to try the alternate meaning of wbits for zlib vs gzip format: https://github.com/urllib3/urllib3/blob/main/src/urllib3/response.py#L102

I started taking a look at this and it seems like we can build it without having to worry about their build system by renaming the zconf.h.in and zconf-ng.h.in files to remove the .in, and setting ZLIB_COMPAT preprocessor in pythoncore.vcxproj (as well as referencing the files).

I don't recommend bypassing the build system as it is used to determine features of the compiler and what optimizations to enable. If you only use Visual Studio then it is best to use cmake to generate your Visual Studio projects.

I don't recommend bypassing the build system

Yeah, I figured this would be a bad idea, but we're also not going to run a separate build system every time. We'll run it once and keep the generated files in our source mirror. (This extra work is why I didn't bother setting it all up earlier, and just hacked things enough to make it build.)

@gpshead

Is this change intended to remain Windows only? I am interested in packaging zlib_ng for python. (It will ship zlib_ng and gzip_ng modules and be fully compatible in every way, at least that is the plan).
But this only makes sense if CPython is not going to switch to using zlib_ng by default any time soon. @gpshead if I understand you correctly, this change is merely for complete distributions that do not dynamically link other libraries at runtime?

There's nothing stopping you from making zlib-ng wrapping modules on PyPI if you wanted to (check to see if anyone already has first). It'd allow applications on any Python version you provide wheels for to choose to use the faster library without rebuilding anything themselves.

Linux distros decide what zlib implementation they use for their own libz so that is not in our control, we don't want to vendor more third party libraries there if we can help it. On macOS I assume we're using a platform zlib as well? But it'd be more reasonable to ship our own different one there as we have to do that for some other libraries such as OpenSSL already.

But on Windows, we bring and build it all, so I suggest using zlib-ng on Windows only for starters.

zlib-ng is an API drop in compatible replacement for zlib.
As is the zlib implementation you can find in the Chromium repository: https://chromium.googlesource.com/chromium/src/third_party/zlib/.
Both are much faster than upstream stock zlib. (and better maintained)

We switched to chromium's zlib internally at work. It offers a "compatibility mode" that proces bit-identical compressed output to zlib at the cost of some performance. That made it a lot easier to move a huge codebase across tons of different systems over to as there were some things making the incorrect assumption that compressed output is canonical based on the input.

If Python runs into that kind of issue during beta releases, I'd suggest we step back and look at the chromium zlib in compatibility mode instead of zlib-ng.

Honestly people designing new systems are better off using zstandard for all lossless compression purposes. But the reason zlib exists and is used is compatibility with existing systems that only accept that format as their lowest common denominator compressed data format.

There's nothing stopping you from making zlib-ng wrapping modules on PyPI if you wanted to (check to see if anyone already has first).

I checked, no-one yet... At least not on PyPI. As the python-isal maintainer, I also feel I can do this with rather less effort than someone who is not as experienced with python's zlibmodule. So I am going to try.

zlib-ng is an API drop in compatible replacement for zlib.
As is the zlib implementation you can find in the Chromium repository: https://chromium.googlesource.com/chromium/src/third_party/zlib/.
Both are much faster than upstream stock zlib. (and better maintained)

We switched to chromium's zlib internally at work. It offers a "compatibility mode" that proces bit-identical compressed output to zlib at the cost of some performance. That made it a lot easier to move a huge codebase across tons of different systems over to as there were some things making the incorrect assumption that compressed output is canonical based on the input.

If Python runs into that kind of issue during beta releases, I'd suggest we step back and look at the chromium zlib in compatibility mode instead of zlib-ng.

That is an interesting thing to consider. Now I have to consider which one of those to wrap. Since zlib-ng is also available in conda (in the conda-forge channel) I think that is the most obvious choice for redistribution.

Honestly people designing new systems are better off using zstandard for all lossless compression purposes.

Depends. Intel's ISA-L igzip application compresses faster and at a better compression ratio than zstd can achieve at those speeds. It decompresses slower though. However one complete compress-decompress cycle is faster on igzip than zstd. So for my work (biofinormatics) where a lot of files are run trough multiple programs, this is most optimal. Also it is backwards-compatible with gzip. So it is quite a big win.
For most other use cases though, I am inclined to agree that zstd is much better. Especially when that will get its own Assembly implementation (if that will happen).

Hi. The python-zlib-ng project is going well: https://github.com/pycompression/python-zlib-ng. First release should be soon.

A couple of observations I thought I'd share which have relevance to this issue:

zlib-ng is a charm to build on Windows if CMake is available. Very easy.
zlib-ng is not as drop-in as it name suggests: compression level 1 is vastly different, sacrificing compression ratio for speed. (Files can be 50% bigger than files compressed with zlib). In my opinion it is quite unacceptable for python's zlib module to have such a big change in behaviour between releases. A couple of percent is doable, but this is just not nice. Especially since the goal of compression is to have less data.
zlib-ng is volunteer maintained and therefore has not had a stable release in more than a year. Releases are spotty at best. This would be fine, if it were not for the following point:
There is a bug in the current 2.0.6 release where the wbits parameter is effectively ignored at compression level 1. It is set to be at least 13, but this creates an issue where data compressed with wbits=9 can't be decompressed with wbits=9. This bug has already been fixed for a while, but still not released.

Now these are not deal-brakers when you want a zlib alternative that can be used next to it to enhance performance, but I feel this is less desirable for CPython. A more regularly maintained, stable and predictable library is more appropriate.

If Python runs into that kind of issue during beta releases, I'd suggest we step back and look at the chromium zlib in compatibility mode instead of zlib-ng.

I think this is the way to go based on my experience so far. This does give people on windows the least surprise. Of course YMMV but I thought I would add my 2 cents and maybe save someone else some work in trying it out.

zlib-ng is not as drop-in as it name suggests: compression level 1 is vastly different, sacrificing compression ratio for speed. (Files can be 50% bigger than files compressed with zlib). In my opinion it is quite unacceptable for python's zlib module to have such a big change in behaviour between releases. A couple of percent is doable, but this is just not nice. Especially since the goal of compression is to have less data.

At level 1, our goal is to sacrifice compression ratio for speed and we do this using Intel's quick deflate algorithm. It is possible to disable this algorithm using -D WITH_NEW_STRATEGIES=OFF. This should also solve your 4th bullet point about the bug.

Has anybody done any compression/decompression performance benchmarking with Chromium's zlib? As far as I remember they weren't focused on improving compression, only decompression. Benchmark from the last release of zlib-ng can be found here, however it only compares against original mainline zlib.

At level 1, our goal is to sacrifice compression ratio for speed and we do this using Intel's quick deflate algorithm. It is possible to disable this algorithm using -D WITH_NEW_STRATEGIES=OFF. This should also solve your 4th bullet point about the bug.

Ah, that is equivalent to ISA-L's level 0. I see. Thank you for the hint. The problem at the 4th bullet point seems fixes on the latest develop branch though, so I gues I will just wait it out until there is a new release. I think those new strategies are a key feature of zlib-ng so I will leave them in for the python bindings.

It does sway my opinion for CPython though. If it can be made more "drop-in" by turning on some options it seems a good fit. Zlib-ng is so easy to build on windows. That can't be said for every C project out there.

As far as I remember they weren't focused on improving compression, only decompression.

That makes sense for a browser, right? It would be a bit weird if compression was a focus. I don't know if browsers regularly sent compressed packets? They do receive them quite a lot.

I thought I should chime in here, and address the point about spotty releases. That is largely my fault and has to do with some health issues I have been going through, combined with a lot of code refactors and a bad strategy for backporting changes to the stable branch that requires me to do a lot of manual work.

There is currently a pre-release for a new 2.0 release zlib-ng/zlib-ng#1393, and it mostly lacks CI fixes so tests work again (The current plan for that is to copy over the new Github Actions files, remove CI runs that are incompatible with 2.0 and hopefully there will not be a lot more to fix manually). That 2.0 release will be the last 2.0 release that we backport changes to, after that there will only be important fixes if needed.

The plan is to release a 2.1 beta shortly after the next 2.0 release is out.
Hopefully the zlib-ng core codebase is now stable enough (in terms of not needing more big refactors) that we can do a more rolling kind of releases for 2.1.x. If that pans out, we will have more regular releases, each with a smaller amount of changes and less chances for breakage between each release. That way there will also be little need for backporting.

I fully expect there to be bugs in the current 2.1 development branch due to the large amount of refactors. It has been working perfectly in several production systems for a long time now, but those do not exercise nearly all the various code paths that a full distribution adoption will see. The automated testing for 2.1 is quite extensive though, so I hope we did not miss anything truly important.

@Dead2

@Dead2 I hope your health issues are resolved soon. Take care!
Also thanks a lot for zlib-ng. It is quite an awesome library. I am especially awed with how easy it was to build and integrate for a library that contains various optimizations for various architectures.

Hopefully the zlib-ng core codebase is now stable enough (in terms of not needing more big refactors) that we can do a more rolling kind of releases for 2.1.x. If that pans out, we will have more regular releases, each with a smaller amount of changes and less chances for breakage between each release. That way there will also be little need for backporting.

If I may gave some unsollicitied advice: don't keep a stable branch with backports. It is an extra maintenance burden. The code is only ever going to be as good as its test suite. This means the branch with the most extensive testing is going to be the most suitable for production. Making regular releases from the development head while creating a new test for every bug that is reported is therefore a very sound strategy for stable releases where no regressions occur.

The automated testing for 2.1 is quite extensive though, so I hope we did not miss anything truly important.

I will run the python-zlib-ng test suite on the branch as well. That has 14000+ compatibility tests with zlib proper, so that also may catch it when something is amiss.

Issue with test_wbits should be fixed with the next stable release of zlib-ng.

Just wanted to let you know that we are about to release beta2 of zlib-ng 2.1 shortly (within a day or two unless a problem crops up). No significant bugs were found during beta1, but we did pull in a couple changes that warrant another beta.

Also, regarding level1 trading off more speed for lower compression, it can be disabled by setting the compiler define -DNO_QUICK_STRATEGY if you want the compression ratio to stay more in line with stock zlib.
We do not have a configure/CMake option for disabling only deflate_quick, and none is planned.

hello hello,

I came across this thread while investigating the horrible performance of pip install

pip install --no-deps ./torch-2.1.2+cu121-cp311-cp311-linux_x86_64.whl is about 22 seconds, including about 18 seconds of extraction.
uv pip install ... is 3 seconds for the same thing including 2 seconds of extraction. but after a bit of debugging, it is cheating and not doing anything. it's extract files to a re-usable location and copy files from there. not fair :(
uv cache clear && uv pip install ... in copy mode is about 9 seconds, including 6 seconds to extract and 3 seconds to copy.
unzip torch-2.1.2+cu121-cp311-cp311-linux_x86_64.whl is 28 seconds, including 25 seconds of cpu time.

turns out. the zlib library in python is pretty terrible! compared to what is available in rust and uv.
and torch is a horribly large package with 10000 files and 2GB compressed and 4 GB decompressed.

zipinfo ./torch-2.1.2+cu121-cp311-cp311-linux_x86_64.whl  | tail
...
10622 files, 3801018214 bytes uncompressed, 2198312632 bytes compressed:  42.2%

one run of the profiler on another large package to identify where the time is taken.
13 seconds in zlib.decompress + 3 or 4 seconds of I/O read and related.

I spent some time to see if I could improve the decompression.
after a lot of investigation, the main issue is the zlib decompression library in python being too slow. I had a look into it but it looks pretty hard to optimize.
I had a look at alternatives libraries, I found a zlib-ng package on pypi ready to use.

trying the zlib-ng package in place of zlib. I adjusted the zipfile code to call both libraries and to pass the expected size and to measure time.
zlib-ng gives a massive performance improvement.

# adding code to measure the time taken in zlib.decompress in zipfile.ZipExtFile.read()
STDLIB total decompress time 13.996230 seconds
ZLIB-NG total decompress time 9.289302 seconds

# EDIT: adding a run, replacing the zlib.so with a build of zlib-ng in compatibility mode.
zlib.so replaced with zlibng.so total decompress time 9.791196 seconds
ZLIB-NG total decompress time 9.364660 seconds

debug log with files of various size. I adjusted the run to extract files in one block to make the benchmark easier.

...
extracting /turbo/rmorotti/pyenvs/githubpip/lib/python3.11/site-packages/torch/jit/mobile/__init__.py
STDLIB optimized extraction for 0MB to 0MB bytes took 0.000020 seconds (total 0.179887 seconds)
ZLIB-NG optimized extraction for 0MB to 0MB bytes took 0.000015 seconds (total 0.141444 seconds)
extracting /turbo/rmorotti/pyenvs/githubpip/lib/python3.11/site-packages/torch/lib/libc10.so
STDLIB optimized extraction for 0MB to 0MB bytes took 0.002562 seconds (total 0.182448 seconds)
ZLIB-NG optimized extraction for 0MB to 0MB bytes took 0.001438 seconds (total 0.142882 seconds)
extracting /turbo/rmorotti/pyenvs/githubpip/lib/python3.11/site-packages/torch/lib/libc10_cuda.so
STDLIB optimized extraction for 0MB to 1MB bytes took 0.003397 seconds (total 0.185845 seconds)
ZLIB-NG optimized extraction for 0MB to 1MB bytes took 0.001956 seconds (total 0.144838 seconds)
extracting /turbo/rmorotti/pyenvs/githubpip/lib/python3.11/site-packages/torch/lib/libcaffe2_nvrtc.so
STDLIB optimized extraction for 0MB to 0MB bytes took 0.000054 seconds (total 0.185899 seconds)
ZLIB-NG optimized extraction for 0MB to 0MB bytes took 0.000026 seconds (total 0.144864 seconds)
extracting /turbo/rmorotti/pyenvs/githubpip/lib/python3.11/site-packages/torch/lib/libnvfuser_codegen.so
STDLIB optimized extraction for 2MB to 9MB bytes took 0.021658 seconds (total 0.207557 seconds)
ZLIB-NG optimized extraction for 2MB to 9MB bytes took 0.013258 seconds (total 0.158122 seconds)
extracting /turbo/rmorotti/pyenvs/githubpip/lib/python3.11/site-packages/torch/lib/libshm.so
STDLIB optimized extraction for 0MB to 0MB bytes took 0.000135 seconds (total 0.207692 seconds)
ZLIB-NG optimized extraction for 0MB to 0MB bytes took 0.000085 seconds (total 0.158207 seconds)
extracting /turbo/rmorotti/pyenvs/githubpip/lib/python3.11/site-packages/torch/lib/libtorch.so
STDLIB optimized extraction for 0MB to 0MB bytes took 0.000399 seconds (total 0.208091 seconds)
ZLIB-NG optimized extraction for 0MB to 0MB bytes took 0.000203 seconds (total 0.158410 seconds)
extracting /turbo/rmorotti/pyenvs/githubpip/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so
STDLIB optimized extraction for 125MB to 450MB bytes took 1.095988 seconds (total 1.304078 seconds)
ZLIB-NG optimized extraction for 125MB to 450MB bytes took 0.670563 seconds (total 0.828974 seconds)
extracting /turbo/rmorotti/pyenvs/githubpip/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so
STDLIB optimized extraction for 747MB to 1168MB bytes took 4.775989 seconds (total 6.080067 seconds)
ZLIB-NG optimized extraction for 747MB to 1168MB bytes took 3.188712 seconds (total 4.017685 seconds)
extracting /turbo/rmorotti/pyenvs/githubpip/lib/python3.11/site-packages/torch/lib/libtorch_cuda_linalg.so
STDLIB optimized extraction for 82MB to 128MB bytes took 0.535490 seconds (total 6.615557 seconds)
ZLIB-NG optimized extraction for 82MB to 128MB bytes took 0.365313 seconds (total 4.382998 seconds)
extracting /turbo/rmorotti/pyenvs/githubpip/lib/python3.11/site-packages/torch/lib/libtorch_global_deps.so
STDLIB optimized extraction for 0MB to 0MB bytes took 0.000057 seconds (total 6.615614 seconds)
ZLIB-NG optimized extraction for 0MB to 0MB bytes took 0.000032 seconds (total 4.383030 seconds)
extracting /turbo/rmorotti/pyenvs/githubpip/lib/python3.11/site-packages/torch/lib/libtorch_python.so
STDLIB optimized extraction for 6MB to 23MB bytes took 0.058344 seconds (total 6.673958 seconds)
ZLIB-NG optimized extraction for 6MB to 23MB bytes took 0.034208 seconds (total 4.417238 seconds)
extracting /turbo/rmorotti/pyenvs/githubpip/lib/python3.11/site-packages/torch/lib/libgomp-a34b3233.so.1
STDLIB optimized extraction for 0MB to 0MB bytes took 0.000551 seconds (total 6.674509 seconds)
ZLIB-NG optimized extraction for 0MB to 0MB bytes took 0.000299 seconds (total 4.417538 seconds)
extracting /turbo/rmorotti/pyenvs/githubpip/lib/python3.11/site-packages/torch/lib/libcudnn_adv_infer.so.8
STDLIB optimized extraction for 81MB to 119MB bytes took 0.538487 seconds (total 7.212996 seconds)
ZLIB-NG optimized extraction for 81MB to 119MB bytes took 0.339576 seconds (total 4.757113 seconds)
extracting /turbo/rmorotti/pyenvs/githubpip/lib/python3.11/site-packages/torch/lib/libcudnn_adv_train.so.8
STDLIB optimized extraction for 79MB to 110MB bytes took 0.514080 seconds (total 7.727076 seconds)
ZLIB-NG optimized extraction for 79MB to 110MB bytes took 0.327647 seconds (total 5.084760 seconds)
extracting /turbo/rmorotti/pyenvs/githubpip/lib/python3.11/site-packages/torch/lib/libcudnn_cnn_infer.so.8
STDLIB optimized extraction for 344MB to 604MB bytes took 2.296822 seconds (total 10.023898 seconds)
ZLIB-NG optimized extraction for 344MB to 604MB bytes took 1.502345 seconds (total 6.587105 seconds)
extracting /turbo/rmorotti/pyenvs/githubpip/lib/python3.11/site-packages/torch/lib/libcudnn_cnn_train.so.8
STDLIB optimized extraction for 84MB to 126MB bytes took 0.537381 seconds (total 10.561278 seconds)
ZLIB-NG optimized extraction for 84MB to 126MB bytes took 0.363187 seconds (total 6.950292 seconds)
extracting /turbo/rmorotti/pyenvs/githubpip/lib/python3.11/site-packages/torch/lib/libcudnn_ops_infer.so.8
STDLIB optimized extraction for 58MB to 86MB bytes took 0.377759 seconds (total 10.939038 seconds)
ZLIB-NG optimized extraction for 58MB to 86MB bytes took 0.255620 seconds (total 7.205913 seconds)
extracting /turbo/rmorotti/pyenvs/githubpip/lib/python3.11/site-packages/torch/lib/libcudnn_ops_train.so.8
STDLIB optimized extraction for 50MB to 67MB bytes took 0.315887 seconds (total 11.254924 seconds)
ZLIB-NG optimized extraction for 50MB to 67MB bytes took 0.216482 seconds (total 7.422394 seconds)
extracting /turbo/rmorotti/pyenvs/githubpip/lib/python3.11/site-packages/torch/lib/libcudnn.so.8
STDLIB optimized extraction for 0MB to 0MB bytes took 0.000406 seconds (total 11.255331 seconds)
ZLIB-NG optimized extraction for 0MB to 0MB bytes took 0.000237 seconds (total 7.422631 seconds)
extracting /turbo/rmorotti/pyenvs/githubpip/lib/python3.11/site-packages/torch/lib/libcublas.so.12
STDLIB optimized extraction for 67MB to 102MB bytes took 0.440602 seconds (total 11.695932 seconds)
ZLIB-NG optimized extraction for 67MB to 102MB bytes took 0.293151 seconds (total 7.715782 seconds)
extracting /turbo/rmorotti/pyenvs/githubpip/lib/python3.11/site-packages/torch/lib/libcublasLt.so.12
STDLIB optimized extraction for 324MB to 491MB bytes took 2.102826 seconds (total 13.798758 seconds)
ZLIB-NG optimized extraction for 324MB to 491MB bytes took 1.454031 seconds (total 9.169813 seconds)
extracting /turbo/rmorotti/pyenvs/githubpip/lib/python3.11/site-packages/torch/lib/libcudart-9335f6a2.so.12
STDLIB optimized extraction for 0MB to 0MB bytes took 0.001567 seconds (total 13.800325 seconds)
ZLIB-NG optimized extraction for 0MB to 0MB bytes took 0.000871 seconds (total 9.170684 seconds)
extracting /turbo/rmorotti/pyenvs/githubpip/lib/python3.11/site-packages/torch/lib/libnvToolsExt-847d78f2.so.1
STDLIB optimized extraction for 0MB to 0MB bytes took 0.000118 seconds (total 13.800443 seconds)
ZLIB-NG optimized extraction for 0MB to 0MB bytes took 0.000073 seconds (total 9.170757 seconds)
extracting /turbo/rmorotti/pyenvs/githubpip/lib/python3.11/site-packages/torch/lib/libnvrtc-b51b459d.so.12
STDLIB optimized extraction for 21MB to 54MB bytes took 0.159425 seconds (total 13.959867 seconds)
ZLIB-NG optimized extraction for 21MB to 54MB bytes took 0.096855 seconds (total 9.267612 seconds)
extracting /turbo/rmorotti/pyenvs/githubpip/lib/python3.11/site-packages/torch/lib/libnvrtc-builtins.so.12.1
STDLIB optimized extraction for 1MB to 6MB bytes took 0.009176 seconds (total 13.969043 seconds)
ZLIB-NG optimized extraction for 1MB to 6MB bytes took 0.004023 seconds (total 9.271635 seconds)
...

As the maintainer of the python bindings for zlib-ng, I would much appreciate if this vital pillar of the entire python packages ecosystem would not rest on my hands.

I created the package only because in my work we regularly handle DEFLATE compressed formats that are hundreds of gigabytes in size. ISA-L is much more interesting for our use case. Having said that, zlib-ng is much more of a 1 to 1 replacement, and I could copy a lot of code over from python-isal, so I figured I could just create the package to handle some corners that python-isal does not handle.

I am glad to see the packages used by (de)compression geeks, but this really is something tangentially related to my work, that I spent only little time on.

In a more practical sense, I think pip should have the least amound of dependencies possible. Sinze zlib is already built-in that makes it much more suitable for a package manager.

As 4GB packages go. Internet connection is probably going to take the majority of install time still...

@gpshead

I'm not entirely sure we want to use the Python bindings of zlib-ng, but directly the zlib-ng C lib. I don't know if there would be maintenance burden on your side though. AFAIU, the purpose is to vendor zlib-ng C code and build it ourselves for Windows (so there is no notion of package management here I think). Or did I misunderstand the original intent by @gpshead?

@morotti

Oh yes. I was replying to @morotti for usage of zlib-ng in pip, he explicitly stated

I had a look at alternatives libraries, I found a zlib-ng package on pypi ready to use.

So that is not the C code, but the package I maintain. I think this particular post should be split of for a separate discussion in another thread as it is indeed different from the topic of this thread.

We currently statically link zlib, so yes, we'd want to do the same for zlib-ng. Which probably does mean copying the sources and setting up our own MSBuild project to compile (assuming we can't just directly call into the C files, zlib-style, which would be ideal).

We don't want a CMake build step/dependency, I don't want to dynamically link zlib-ng (or zlib - the current dynamically linked one is just for Tcl/Tk because their build system is inflexible), and building a static lib separately isn't supported (compiler versions need to match basically perfectly for link-time optimisation to work).

As 4GB packages go. Internet connection is probably going to take the majority of install time still...

nope :p
I rewrote the pip download code to make it faster. it can download packages at 700 MB/s now :D
remember to upgrade your pip. you want pip 24.2 or newer

I'm having another look at improving pip and the bottleneck is the decompression being too slow. zlib is slow.
I saw your zlib-ng python package and I was merely doing some run to test things up. it works nicely and it's a lot faster.
this confirms why pip is so much slower than uv on extraction. uv/rust have a much better deflate library.

I believe zlib-ng (the C library) should be integrated into the python interpreter and replace zlib.

In a more practical sense, I think pip should have the least amound of dependencies possible. Sinze zlib is already built-in that makes it much more suitable for a package manager.

Pip won't use any dependencies that have native code components, as this prevents us from having a single wheel that works on all platforms. So the only way pip will use zlib-ng is if it's built into Python. So you have no need to worry that we'd depend on your package (although even if we could, we wouldn't do so against your wishes).

pythongh-91349: Expose the crc32 function from the lzma library.

Hello,

Coming back to the ticket to see if there's any interest.
I think there is agreement that zlib is slow and hardly maintained anymore. it's not going to get further performance improvements.

The most likely replacement is zlib-ng and libdeflate in my mind.

libdeflate is about 100% faster. it only supports compression and extraction of a full buffer. no streaming.
zlib-ng is about 30% faster and supports all the features from zlib. it can be made into a new module or it is has a compat mode to be compiled in place of zlib.

Having full buffer extraction would work fine for many use cases like pip. Each file is compressed separately in a wheel package. The larger files are a few MB (except for tensorflow and torch), they can easily be decompressed in one chunk. I actually made performance improvements in pip last year improve performance, including raising extraction buffers to 1MB, we could certainly raise to 16MB or more if there was a benefit to it, that would cover 99% of packages).

I've started doing some testing.. and it turns out there is another issue with the CRC. zlib provides both the compression routines and the crc32 routines. The crc is pitifully slow too! 🙄
We could potentially start rolling better crc functions. I've noticed the lzma library has a crc function that is faster, but it's not exposed. We could expose it.

Unless there is any objection:

I can make a PR to expose the lzma.crc32 function.
I can make a PR to support building libdeflate.
I can make a PR to optimize the zipfile extraction to go in one chunk for small buffers. although this seems to have no impact on zlib, it's equally slow whether you provide the expected decompressed size or not.
I see the windows builds are using xz-5.2.5. is it possible to upgrade to latest xz-5.6.4 ?

thoughts?

some numbers on performance. calculating crc on tensorflow.whl 454MB

# ubuntu 22, all versions are quite old
# zlib 1.2.11, lzma 5.2.5, deflate 1.10.12
for module in binascii zlib lzma deflate; do echo -n "$module  "; ./python -m timeit --setup "import pathlib,$module;data=pathlib.Path('tensorflow.whl').read_bytes()" "$module.crc32(data)"; done

binascii  1 loop, best of 5: 341 msec per loop
zlib  1 loop, best of 5: 341 msec per loop
lzma  1 loop, best of 5: 216 msec per loop
deflate  10 loops, best of 5: 32 msec per loop

# newer libraries: zlib 1.3.1, lzma 5.6.4
binascii  2 loops, best of 5: 178 msec per loop
zlib  2 loops, best of 5: 178 msec per loop
lzma  5 loops, best of 5: 52.1 msec per loop
deflate  10 loops, best of 5: 32 msec per loop

looking at the performance. all software are getting faster with new versions.
zlib 1.2.12 (2022) had a line - Speed up software CRC-32 computation by a factor of 1.5 to 3
though I don't expect any further improvements from zlib and it's still very slow compared to specialized code in deflate and zlib-ng. (They can run the CRC at 60GB/s on my windows desktop!)

https://www.zlib.net/ChangeLog.txt

Pip won't use any dependencies that have native code components, as this prevents us from having a single wheel that works on all platforms.

This is a policy that IMNSHO pip should revisit as it has been preventing useful things from being done. But this issue is not the forum for that. Take it up in Packaging spaces.

I recommend using crc hashing from zlib-ng. It is better optimized for more platforms and is better maintained. We have more improvements in development.

some benchmarks on my windows machine on python 3.12, to compare the different algorithms, decompressing all the files from torch.

the most remarkable in my mind is the crc algorithm in libdeflate/zlib-ng doing up to 80GB/s with turbo boost. It's insane!
zlib is barely getting in the 2 GB/s range on the crc. whole seconds to extract a large archive like torch.

zlib is zlib
zlib.decompress_guess is when not passing the buffer size. (there seems to be a weird thing where zlib gets faster when it runs multiple times).
zlib ng is the python wrapper around zlib-ng.
isal_zlib is the python wrapper around intel library
deflate is the python wrapper around libdeflate
copy is memory copy with copy.copy(bytebuffer)

filepaths = glob.glob('./torch-2.1.2+cu121-cp311-cp311-linux_x86_64/**/*', recursive=True)
filepaths = sorted(filter(lambda x: os.path.isfile(x) and 4096 < os.path.getsize(x) < 1024*1024*1024, filepaths), key=lambda x: os.path.getsize(x))

----------
1793 files TURBOBOOST OFF
SUMMARY  zlib.decompress_guess   2445 MB in 8.270 seconds (avg=421 MB/s) at min=187 25%=330 50%=391 75%=473 max=1319 MB/s
SUMMARY  zlib.decompress        2445 MB in 8.025 seconds (avg=485 MB/s) at min=212 25%=379 50%=449 75%=542 max=1587 MB/s
SUMMARY  zlib_ng.decompress     2445 MB in 5.693 seconds (avg=587 MB/s) at min=155 25%=387 50%=530 75%=731 max=2314 MB/s
SUMMARY  isal_zlib.decompress   2445 MB in 4.717 seconds (avg=666 MB/s) at min=72 25%=459 50%=602 75%=803 max=2533 MB/s
SUMMARY  deflate_decompress     2445 MB in 4.472 seconds (avg=908 MB/s) at min=283 25%=637 50%=811 75%=1084 max=4636 MB/s
SUMMARY  zlib.crc32             (crc) 2445 MB in 0.911 seconds (avg=2398 MB/s) at min=1304 25%=2259 50%=2412 75%=2540 max=2693 MB/s
SUMMARY  binascii.crc32         (crc) 2445 MB in 0.911 seconds (avg=2387 MB/s) at min=958 25%=2246 50%=2391 75%=2536 max=2693 MB/s
SUMMARY  zlib_ng.crc32          (crc) 2445 MB in 0.053 seconds (avg=19418 MB/s) at min=1397 25%=11005 50%=15669 75%=24808 max=59464 MB/s
SUMMARY  isal_zlib.crc32        (crc) 2445 MB in 0.052 seconds (avg=21315 MB/s) at min=738 25%=11957 50%=17843 75%=27750 max=59716 MB/s
SUMMARY  deflate.crc32          (crc) 2445 MB in 0.051 seconds (avg=16827 MB/s) at min=1185 25%=8768 50%=13118 75%=21764 max=59339 MB/s

----------
1793 files TURBOBOOST ON
SUMMARY  zlib.decompress_guess   2445 MB in 6.127 seconds (avg=569 MB/s) at min=266 25%=444 50%=530 75%=641 max=1815 MB/s
SUMMARY  zlib.decompress        2445 MB in 5.914 seconds (avg=658 MB/s) at min=273 25%=512 50%=611 75%=743 max=2225 MB/s
SUMMARY  zlib_ng.decompress     2445 MB in 4.231 seconds (avg=797 MB/s) at min=169 25%=518 50%=724 75%=991 max=4619 MB/s
SUMMARY  isal_zlib.decompress   2445 MB in 3.536 seconds (avg=895 MB/s) at min=91 25%=622 50%=823 75%=1058 max=3101 MB/s
SUMMARY  deflate_decompress     2445 MB in 3.305 seconds (avg=1234 MB/s) at min=329 25%=857 50%=1116 75%=1480 max=6518 MB/s
SUMMARY  zlib.crc32             (crc) 2445 MB in 0.676 seconds (avg=3262 MB/s) at min=2028 25%=3051 50%=3260 75%=3484 max=3701 MB/s
SUMMARY  binascii.crc32         (crc) 2445 MB in 0.676 seconds (avg=3236 MB/s) at min=997 25%=3006 50%=3244 75%=3484 max=3712 MB/s
SUMMARY  zlib_ng.crc32          (crc) 2445 MB in 0.049 seconds (avg=26632 MB/s) at min=1505 25%=15088 50%=21645 75%=34389 max=80622 MB/s
SUMMARY  isal_zlib.crc32        (crc) 2445 MB in 0.047 seconds (avg=28822 MB/s) at min=931 25%=15824 50%=24056 75%=38187 max=80017 MB/s
SUMMARY  deflate.crc32          (crc) 2445 MB in 0.047 seconds (avg=22872 MB/s) at min=1565 25%=11759 50%=17924 75%=29445 max=79317 MB/s

SUMMARY  copy                   2445 MB in 0.026 seconds (avg=39489 MB/s) at min=443 25%=16601 50%=26887 75%=44551 max=663361 MB/s

It certainly sounds like it's worth dropping in zlib-ng instead of mainline zlib for the builds where we vendor, provided the compatibility is good enough. That may mean tweaking some defaults, but that's the aspect that needs analysis. More benchmarks don't help us if the change would make files or behaviour incompatible.

Last time I checked I'm pretty sure our tests had very good coverage of edge cases here, which ought to mean if we drop it in and they all pass then we're good. Anyone up for doing it?

That may mean tweaking some defaults

If I may provide some more information. I don't think there is anything to tweak. zlib-ng can be built in compatibility mode and replace zlib in place. I've tried it by replacing the .so. It works fine.
The question may be what is the the strategy for the interpreter. Whether to substitute zlib with zlib-ng? or to ship a different module zlib-ng? (or both?)

there are two things to be aware of:

zlib-ng level 1 is compressing a lot worse AND a lot faster. A 100MB file that was compressed to 30 MB at 100MB/s, might only get compressed to 40 MB at 200MB/s (you can see benchmarks on the internet). it can be a bit of surprise the first time.
old versions of zlib-ng and deflate are NOT safe to use IMO.
I do recalling trying and benchmarking them as far as 5 years ago (I guess I am a bit of a compression nerd) and they have issues with crash and malformed output. You can see a bug right at the top of this thread that prevented adoption last year (it's resolved now). I'm concerned that people running on linux distributions -they are always a few years behind- are going to run into bugs.
If the interpreter will detect and build with zlib-ng in place of zlib (I think we should!), it should probably check the version and only do it since the most recent releases (current is 2.2.4).

FWIW re: my earlier link to chromium's zlib above as well as another potential well maintained option. There are also (too many) other zlib performance forks out there such as https://github.com/cloudflare/zlib (not usable, they understandably ditched 32-bit support).

When we get to the point of supporting Rust components with in CPython itself https://github.com/trifectatechfoundation/zlib-rs is a performant option (there's even a zlib API compatible C library wrapping it, but it'd still involve extra complication for now).

But zlib-ng being maintained as its own C library for direct use, is seeing regular releases, including handling security fixes, and sees use on Linux distros. That suggests that if we just used that already people would be happy with it. We should probably do the same on macOS. We could recommend Linux distributors link against it as well and maybe steer builds in that direction via a configure.ac check. If that changes and we need to start managing zlib-ng security patches to it ourselves rather than pulling upstream updates we could revisit its use.

Chromium's zlib (definitely on top of security) offers a byte for byte compatibility story if we care about that - I no longer think we should, it was just an interesting option that happened to make sense within my former employer. People with that sort of custom build and need are going to pick their own zlib anyways.

Canonical exact output for a given input and compression level was never an intended guarantee of zlib. We're already in a world where I'm sure some people are already linking their Python's against different zlib implementations that do not guarantee that. If it fell out during a beta period testing that people do wind up depending on canonical zlib output in widespread difficult to stop important ways, we could revisit the choice.

old versions of zlib-ng and deflate are NOT safe to use IMO.

This is actually one of the reasons other performance forks like chromium's and cloudflare's exist. vendored versions we control, for a configure.ac check we'd declare a minimum.

Canonical exact output for a given input and compression level was never an intended guarantee of zlib. We're already in a world where I'm sure some people are already linking their Python's against different zlib implementations that do not guarantee that. If it fell out during a beta period testing that people do wind up depending on canonical zlib output in widespread difficult to stop important ways, we could revisit the choice.

Yeah, I think this is a fine approach. Beta is the only way we're going to flush out this kind of issue.

zlib-ng level 1 is compressing a lot worse AND a lot faster.

This isn't our default setting though, right? If it is, I think that's a problem, but if it's non-default then not a problem.

Just as a FYI, as you probably know, many Linux distros have been shipping a zlib-ng package using the native api for a few years now, but the adoption into 3rd party software has been rather slow.

However, in the past year several Linux distros have wholesale switched out their zlib package and started shipping zlib-ng-compat packages that are zlib-ng compiled with the zlib-compat mode enabled. This means that from for example Fedora 40, python uses zlib-ng-compat.

Most applications do not notice the change, usually only tests requiring identical compressed bitstreams gets upset, and Nginx of course, since it expects zlib to always malloc a very specific amount of memory (it works, it just spams your log with warnings and ends up doing the malloc twice).
The amount of complaints/bugs (reaching us at least) has been surprisingly low, but there have been a few hard to replicate corner-cases that have had fixes implemented.

Most distros do disable both deflate_quick and deflate_medium for their zlib-ng-compat package, others only disabling deflate_quick. This is to avoid unexpected changes in compression sizes. Deflate_quick is super fast and used for level 1, but does compress a lot worse than some might expect, although if you did chose level 1, you probably wanted speed over size.
The distros zlib-ng packages (without the native api) however enables both. So factor this into whether you want to support zlib-ng's native api or not, although we would of course suggest that you do so. It is also possible to support both using macros (see how we do that in minigzip for example) so the packager can choose.

When it comes to portability, I think zlib-ng is probably better than all the forks. Just yesterday two of the developers digged out their aging Mac G4s to benchmark some new CRC32 VMX optimizations for the Power cpu. While portability and optimizations to very old cpus are not really a priority, we do try hard not to make incompatible changes. Our upcoming Chorba CRC32 optimizations are really exciting for the pre-PCLMULQDQ cpus. To use the Power benchmarks as an example, old CRC32 C code: 333ms, new C code: 208ms, new VMX code: 135ms.
We also officially only support C11-capable compilers and up, but we don't go out of our way to make it so, so one of our contributors has been working on submitting small changes that makes zlib-ng compile on OpenWatcom.

And once again I post more than I intended to. Hope you found it interesting at least 😉.

Fedora 40 reference:

[hansr@hk zlib-ng]$ ldd /usr/lib64/python3.12/lib-dynload/zlib.cpython-312-x86_64-linux-gnu.so
        libz.so.1 => /lib64/libz.so.1 (0x00007f49a4b42000)
-snip-

[hansr@hk zlib-ng]$ rpm -ql zlib-ng-compat
/usr/lib64/libz.so.1
/usr/lib64/libz.so.1.3.1.zlib-ng
-snip-

[hansr@hk zlib-ng]$ rpm -qa|grep zlib
zlib-ng-2.1.7-2.fc40.x86_64
zlib-ng-compat-2.1.7-2.fc40.x86_64
zlib-ng-compat-2.1.7-2.fc40.i686
zlib-ng-compat-devel-2.1.7-2.fc40.x86_64

zlib-ng level 1 is compressing a lot worse AND a lot faster.

This isn't our default setting though, right? If it is, I think that's a problem, but if it's non-default then not a problem.

correct. 1 is just what people explicitly use when they want "fastest" with zlib. I wouldn't pre-worry about definitions of 1 being different any more than output differing between implementations.

zlib-ng sounds like a good idea.

We switched to zlib-ng for PNGs in the last Pillow release because it's much faster (up to 4x at highest compression).

And for file sizes: zlib-ng has same file sizes for compression level 0 and bigger files for compression level 1, but if you care about file size, you'll use a higher compression which gives more or less the same file size (and zlib-ng is faster).

Benchmark graphs at python-pillow/Pillow#8500 (comment)

We had a couple of reports from people whose tests started failing, but that was because they were comparing bytes in images, instead of image content/pixels.

Okay, so let's do it.

Do we need to put the sources in the cpython-source-deps repository (where zlib currently lives), or do we prefer to vendor directly in our repo?

I'd love to use the external repo(s) used by builds as we do for zlib. (i think we also have such a repo/branch used by macos builds as well? though not for zlib yet)

pythongh-91349: Replace zlib with zlib-ng in Windows build

@nmoinvaz

zlib-ng level 1 is compressing a lot worse AND a lot faster.

This isn't our default setting though, right? If it is, I think that's a problem, but if it's non-default then not a problem.

correct. 1 is just what people explicitly use when they want "fastest" with zlib. I wouldn't pre-worry about definitions of 1 being
different any more than output differing between implementations.

To quote @nmoinvaz

At level 1, our goal is to sacrifice compression ratio for speed and we do this using Intel's quick deflate algorithm. It is possible to disable this algorithm using -D WITH_NEW_STRATEGIES=OFF. This should also solve your 4th bullet point about the bug.

I recommend turning this on. A certain level of compression is expected from zlib level 1. The point of compression is to have smaller files, so having suddenly 30% bigger files may break some people's expectations. Especially in network scenarios this is not desirable.

(i think we also have such a repo/branch used by macos builds as well? though not for zlib yet)

We don't; for macOS installer builds, we use the upstream source releases, occasionally with a patch as needed, for the third-party libs that aren't available from the OS or that we want a newer version. We currently use the macOS-supplied version of zlib, although we have supplied our own in the distant past. It sounds like it shouldn't be a big issue to use zlib-ng instead. I'll look into it and, if it looks good, I'll make a PR.

At level 1, our goal is to sacrifice compression ratio for speed and we do this using Intel's quick deflate algorithm. It is possible to disable this algorithm using -D WITH_NEW_STRATEGIES=OFF. This should also solve your 4th bullet point about the bug.

I recommend turning this on. A certain level of compression is expected from zlib level 1. The point of compression is to have smaller files, so having suddenly 30% bigger files may break some people's expectations. Especially in network scenarios this is not desirable.

I have mixed feeling about that, because the level 1 compression in zlib is still so slow -around the 50MB/s or less depending on CPU- that it is unusable for many use cases (and it's the main driver of people moving to zstd).
Anything like compressing a backup or a container images or a large file, is severely limited by the compression speed at level 1. It's slower than the disk can read and write. It's slower than the network (any gigabit ethernet or any wireless LAN).

Also, please bear in mind that most servers CPU have many cores but the cores are slower (the fastest servers CPUs now are half the speed of the average desktop CPU for single core performance); while on the user front, many laptops have dramatically increased the CPU count and added efficiency cores, but the cores barely break the 1 GHz mark anymore. Performance has got worse in the last few years. zlib is heavily affected as it's single threaded compression.

It would be nice to have a compress level 1 that's a lot faster!

For reference, I just got patches shipped to tarfile few months ago to make tarfile twice as fast. Even after that, the compression is still too slow at level 1 for trivial usage to compress files and transfer between machines. (I can't change the algo because spark can only understand zip or tar.gz) #121269 (comment)

pythongh-91349: add support for libdeflate library.

Anything like compressing a backup or a container images or a large file, is severely limited by the compression speed at level 1. It's slower than the disk can read and write. It's slower than the network (any gigabit ethernet or any wireless LAN).

In that case use python-isal. If a specialized use case warrants the use of extra fast zlib, that is much better.

In this case where python's zlib module backend is changed, I think the changes in behavior should be minimal.

It's slower than the network (any gigabit ethernet or any wireless LAN).

Not if your network is "the internet". Then it really matters whether you sent a 30MiB or a 40MiB file. Also if you need fast, but reasonable compression to save disk space, those differences are quite huge. Therefore I think the ultra-fast strategy in zlib-ng should be disabled as it is not infeasible that this strategy will lead to unexpected nasty surprises for some users on windows CPython.

For applications that do need the utmost speed and stuff python-isal and python-zlib-ng are available. Python-zlib-ng as a non-standard library component of course has the default zlib-ng behavior for level 1.

We at zlib-ng think that if you already chose level 1, then you probably cared more about the speed than the compression ratio.
Some might get surprised, but having this much faster level 1 (nearly 2x the speed of level 2) also means that people can keep using deflate/gzip compression instead of having to migrate (if at all possible) to LZO/LZ4 that would require rewrites and breaking compatibility.

Also, if you are interested in fast CRC, our current develop contains a much faster CRC implementation called Chorba, so make sure to test that as well.
Currently only a C-Chorba is implemented, but work is ongoing on SSE2 and others as well. Chorba is not used by cpus with [V]PCLMULQDQ capabilities, as those are already faster with implementations utilizing those instructions. But Chorba is great for portability since it does not require any specific instruction sets.

These are my benchmarks of C-Chorba vs the old braid implementation as of today on a 2011-era low-end AMD E-450 cpu.

crc32/generic_chorba/1                          26.9 ns         26.8 ns     26157499
crc32/generic_chorba/8                          50.1 ns         49.9 ns     14035938
crc32/generic_chorba/12                         80.6 ns         80.3 ns      8719313
crc32/generic_chorba/16                         94.7 ns         94.3 ns      7426048
crc32/generic_chorba/32                          153 ns          153 ns      4584211
crc32/generic_chorba/64                          280 ns          279 ns      2500184
crc32/generic_chorba/512                         730 ns          726 ns       967636
crc32/generic_chorba/4096                       3772 ns         3755 ns       186302
crc32/generic_chorba/32768                     30684 ns        30535 ns        22923
crc32/generic_chorba/262144                   224062 ns       222951 ns         3140
crc32/generic_chorba/4194304                 3478228 ns      3404615 ns          206

crc32/braid/1                                   18.0 ns         17.9 ns     39011495
crc32/braid/8                                   44.0 ns         43.8 ns     15986689
crc32/braid/12                                  59.1 ns         58.9 ns     11871923
crc32/braid/16                                  73.2 ns         72.9 ns      9602697
crc32/braid/32                                   132 ns          132 ns      5319085
crc32/braid/64                                   260 ns          259 ns      2707563
crc32/braid/512                                 1283 ns         1277 ns       548611
crc32/braid/4096                                9249 ns         9210 ns        75989
crc32/braid/32768                              73471 ns        73154 ns         9567
crc32/braid/262144                            586380 ns       583859 ns         1198
crc32/braid/4194304                          9649556 ns      9563683 ns           73

Alright, I recall I had to patch tornado many years ago to fix the compression level.
I am reviewing the source code and still found the old comment https://github.com/tornadoweb/tornado/blob/d5ac65c1f1453c2aeddd089d8e68c159645c13e1/tornado/web.py#L3247

That reminds me, there is a massive bug in the python standard libraries around gzip, python is using level 9 compress by default, which is extremely slow (around 1MB/s).
Anything that is python with the default settings will find itself massively bottlenecked by the compression.
The default level used by all the tools is level 6, which is a more reasonable tradeoff.

I should have sent the fix 9 years ago to fix the level, when I was fixing tornado, but then I forgot ^^
Is there any objection if I send a patch to change the default compression level to 6 to tarfile gzip and co?

pythongh-91349: Adjust default compression level to 6 (down from 9) i…

@ned-deily

@ned-deily Make sure you enable zlib compatibility mode and probably disable the gtest and gzip stuff and it really does just drop in now (I did a bit more work because I didn't want to be running CMake as part of our build, but the only change I made to zlibmodule was to add ZLIBNG_VERSION).

Compression level 1 is already labeled "best speed", which to me says we should use the option that provides the best speed. Provided it isn't making data bigger, it is doing exactly what it claims to, and would be inaccurate if we didn't let it run with the best speed.

I assume that switching to compression level 2 is going to be fairly close to the old compression level 1? And if you didn't actually mean "best speed", then selecting an option other than "best speed" seems like a reasonable way to express that.

Changing the default from 9 to something else is likely fine, and we may as well do it at the same time. Though I'd like to see a more concrete argument than "probably a better CPU/size tradeoff".

I don't see any size benchmarks above covering level 6, and I'm not even sure what kind of files would be a reasonable benchmark here - probably the type of content likely to be served from a Python web app? JSON? Wheels get packed once and extracted millions of times, so the size benefit easily outweighs the compression time.

I assume that switching to compression level 2 is going to be fairly close to the old compression level 1? And if you didn't actually mean "best speed", then selecting an option other than "best speed" seems like a reasonable way to express that.

If deflate_quick is enabled (WITH_NEW_STRATEGIES=ON), then level 2 uses the settings level 1 normally uses. So in other words, zlib-ng level 2 with deflate_quick is exactly the same as level 1 is without deflate_quick.
In both cases, this is not exactly the same as zlib level 1 due to changes needed for better optimization, but very close.
Ref: https://github.com/zlib-ng/zlib-ng/blob/fd0d263cedab1a136f40d65199987e3eaeecfcbd/deflate.c#L146

If you do decide disable deflate_quick, I'd still suggest you keep deflate_medium enabled. If so, instead of setting WITH_NEW_STRATEGIES=OFF, you'd need to define NO_QUICK_STRATEGY as a CFLAG.
Compression with deflate_medium is very close to the original levels, just faster.

pythongh-91349: Expose the crc32 function from the lzma library.

@zooba

@zooba quick benchmark on compression level.

trying on the json file https://pypi.org/pypi/numpy/json
python 3.14 beta on main branch. windows laptop, turbo boost off.

level above 6 have zero benefit, they are significantly slower and they don't compress better.
we can benchmark various types of data, the result will be similar. (there are plenty of zlib benchmark online if you want to see)

that new laptop is doing good, gzip used to be single digit MB/s in the higher compression levels!
it's so slow that I had to set web servers like tornado to compress on level=1 in multiple of my day jobs, otherwise they spend more time compressing response than running the app.

gzip level=0 2331kB->2331kB in 0.005 seconds 438.96 MB/s 100.01% size
gzip level=1 2331kB->523kB in 0.037 seconds 53.88 MB/s 22.48% size
gzip level=2 2331kB->513kB in 0.036 seconds 56.11 MB/s 22.02% size
gzip level=3 2331kB->508kB in 0.039 seconds 51.37 MB/s 21.81% size
gzip level=4 2331kB->477kB in 0.065 seconds 30.64 MB/s 20.50% size
gzip level=5 2331kB->468kB in 0.074 seconds 26.95 MB/s 20.11% size
gzip level=6 2331kB->459kB in 0.088 seconds 22.64 MB/s 19.71% size
gzip level=7 2331kB->456kB in 0.098 seconds 20.51 MB/s 19.60% size
gzip level=8 2331kB->453kB in 0.139 seconds 14.37 MB/s 19.44% size
gzip level=9 2331kB->453kB in 0.147 seconds 13.62 MB/s 19.44% size

import pathlib
import gzip
import time
import zlib

data = pathlib.Path('numpy.json').read_bytes()
size = len(data)
for level in range(0, 10):
    start = time.perf_counter()
    compressed = gzip.compress(data, level)
    end = time.perf_counter()
    elapsed = end - start
    print(f"level={level} {len(data)//1024}kB->{len(compressed)//1024}kB in {elapsed:.3f} seconds {len(data)//1048576/elapsed:.2f} MB/s {len(compressed)/len(data)*100.0:.2f}% size")

EDIT: adding one run on bz2. I see bz2 is set to level=9 too. I guess we can keep it like that. the default for bzip tools seems to be level 9.

bz2 level=1 2331kB->466kB in 0.312 seconds 6.41 MB/s 20.01% size
bz2 level=2 2331kB->451kB in 0.332 seconds 6.02 MB/s 19.38% size
bz2 level=3 2331kB->446kB in 0.338 seconds 5.91 MB/s 19.14% size
bz2 level=4 2331kB->442kB in 0.351 seconds 5.70 MB/s 18.99% size
bz2 level=5 2331kB->440kB in 0.358 seconds 5.58 MB/s 18.90% size
bz2 level=6 2331kB->439kB in 0.375 seconds 5.33 MB/s 18.84% size
bz2 level=7 2331kB->438kB in 0.371 seconds 5.39 MB/s 18.83% size
bz2 level=8 2331kB->437kB in 0.384 seconds 5.21 MB/s 18.75% size
bz2 level=9 2331kB->437kB in 0.394 seconds 5.08 MB/s 18.77% size

pythongh-91349: Adjust default compression level to 6 (down from 9) i…

i suggest not overthinking exactly 1 means for now, we can tune that if it causes problems during the beta period.

gh-91349: Replace zlib with zlib-ng in Windows build (GH-131438)

@morotti

@morotti Thanks for those! Yeah, moving to 6 looks about right based on that, though it's broader than the implementation in the Windows distribution, so I'll look for others to chime in on it.

Leaving this issue open for now until @ned-deily decides on the macOS build and we answer the default level question.

quick benchmark on the compression on numpy.json (see script few comments above)
this is not meant to be an exhaustive benchmark.

windows laptop, turbo boost off
main branch yesterday before the PR to use zlib-ng VS main branch today with zlib-ng

quick takeaways:
level 1 is nearly three times faster but it's compressing a lot worse, as expected. It's great, this will allow a lot of legacy apps that were stuck on zip/tar.gz and bottleneck by the slow compression/decompression speed to run more effectively.
level 2 is nearly twice as fast and it's compressing slightly better than ALL the zlib levels up to 8. It's awesome.

algorithm	library	level	size_decompressed	size_compressed	duration_s	speed_mb_s	ratio
gzip-0	gzip	0	2331	2331	0.005	422.09	1.00
gzip-1	gzip	1	2331	523	0.036	55.91	0.22
gzip-2	gzip	2	2331	513	0.036	54.98	0.22
gzip-3	gzip	3	2331	508	0.038	52.1	0.22
gzip-4	gzip	4	2331	477	0.066	30.44	0.21
gzip-5	gzip	5	2331	468	0.074	26.99	0.20
gzip-6	gzip	6	2331	459	0.088	22.83	0.20
gzip-7	gzip	7	2331	456	0.096	20.85	0.20
gzip-8	gzip	8	2331	453	0.138	14.5	0.19
gzip-9	gzip	9	2331	453	0.147	13.58	0.19
gzip(zlib-ng)-0	gzip(zlib-ng)	0	2331	2331	0.004	526.03	1.00
gzip(zlib-ng)-1	gzip(zlib-ng)	1	2331	687	0.014	144.37	0.29
gzip(zlib-ng)-2	gzip(zlib-ng)	2	2331	471	0.022	92.42	0.20
gzip(zlib-ng)-3	gzip(zlib-ng)	3	2331	456	0.034	59.18	0.20
gzip(zlib-ng)-4	gzip(zlib-ng)	4	2331	448	0.037	54.19	0.19
gzip(zlib-ng)-5	gzip(zlib-ng)	5	2331	442	0.041	48.19	0.19
gzip(zlib-ng)-6	gzip(zlib-ng)	6	2331	437	0.049	40.42	0.19
gzip(zlib-ng)-7	gzip(zlib-ng)	7	2331	432	0.049	41.14	0.19
gzip(zlib-ng)-8	gzip(zlib-ng)	8	2331	428	0.063	31.65	0.18
gzip(zlib-ng)-9	gzip(zlib-ng)	9	2331	476	0.079	25.4	0.20

I think we can all agree that zlib is obsolete. can we start using zlib-ng on the linux builds too?
please please please

FYI, I did some testing many years ago and determined that 128k sized buffers were most optimal for performance. zlib-ng/zlib-ng#868

hello,

zlib-ng started being used for the windows build a week ago and it's awesome :)
any plan to start using it for the linux build soon?

Contributions to the linux build as in configure.ac & Makefile.pre.in are welcome so that when a recent enough version of zlib-ng is found, it is preferred over vanilla zlib.

pythongh-91349: Expose the crc32 function from the lzma library

quick update,

I have attempted to adjust the linux build to detect and use zlib-ng, to no success.
the .configure script eludes me.
there might only be 10 people in the world who understand any of it.
any help would be greatly appreciated.
liblzma just had a new release a few days ago (xz-utils package on linux). https://github.com/tukaani-project/xz/releases
the latest liblzma release 5.8.x is using hardware CRC instructions, it is now as fast as zlib-ng and libdeflate.
I have submitted a PR to expose crc32 from lzma.
after it is merged, I shall do more PR to pick the optimal crc function between zlib/zlibng/liblzma depending on what is available.

It would be great to see what the stats are compared to zlib-ng. It has optimizations for different architectures and even instruction sets. So it will perform well on older hardware if that is necessary for your use case.

pythongh-91349: Adjust default compression level to 6 (down from 9) i…

pythongh-91349: Expose the crc32 function from the lzma library

gh-91349: Adjust default compression level to 6 (down from 9) in gzip…

pythongh-91349: Adjust default compression level to 6 (down from 9) i…

pythongh-91349: Expose the crc32 function from the lzma library

Just a quick update since the Python interpreter transitioned to zlib-ng on Windows, I've performed extensive benchmarks and analysis since we did the transition, you can read here, it's a massive improvement! ->
Faster Python 3.14: faster compression and fixing 28-year-old bugs

@morotti

Thanks @morotti for the write up. Very interesting. It also seems that my side project of python-zlib-ng is now redundant for windows 3.14 and higher. Less maintenance for me in the future. I hope that zlib-ng gets shipped by distro's soon.

gh-91349: Update `gzip` docs and docstrings for the default compressl…

[3.15] gh-91349: Update `gzip` docs and docstrings for the default co…

Navigation Menu

Repository navigation

Use zlib-ng (fast!) rather than mainline stale zlib in binary releases #91349

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Use zlib-ng (fast!) rather than mainline stale zlib in binary releases #91349

Description

Linked PRs

Activity

gpshead commented on Aug 1, 2022

nmoinvaz commented on Aug 1, 2022

zooba commented on Aug 1, 2022

rhpvorderman commented on Jan 20, 2023

gpshead commented on Jan 20, 2023

rhpvorderman commented on Jan 21, 2023

rhpvorderman commented on Jan 30, 2023

nmoinvaz commented on Jan 30, 2023

rhpvorderman commented on Jan 31, 2023

Dead2 commented on Feb 2, 2023

rhpvorderman commented on Feb 3, 2023

rhpvorderman commented on Feb 17, 2023

Dead2 commented on May 16, 2023

morotti commented on Mar 13, 2025

rhpvorderman commented on Mar 14, 2025

picnixz commented on Mar 14, 2025

rhpvorderman commented on Mar 14, 2025

zooba commented on Mar 14, 2025

morotti commented on Mar 14, 2025

pfmoore commented on Mar 14, 2025

morotti commented on Mar 18, 2025

gpshead commented on Mar 18, 2025

nmoinvaz commented on Mar 18, 2025

morotti commented on Mar 18, 2025

zooba commented on Mar 18, 2025

morotti commented on Mar 18, 2025

gpshead commented on Mar 18, 2025

gpshead commented on Mar 18, 2025

zooba commented on Mar 18, 2025

Dead2 commented on Mar 18, 2025

gpshead commented on Mar 18, 2025

hugovk commented on Mar 18, 2025

zooba commented on Mar 18, 2025

gpshead commented on Mar 18, 2025

rhpvorderman commented on Mar 19, 2025

ned-deily commented on Mar 19, 2025

morotti commented on Mar 19, 2025

rhpvorderman commented on Mar 19, 2025

Dead2 commented on Mar 19, 2025

morotti commented on Mar 19, 2025

zooba commented on Mar 19, 2025

zooba commented on Mar 19, 2025

Dead2 commented on Mar 19, 2025

morotti commented on Mar 19, 2025

gpshead commented on Mar 19, 2025

zooba commented on Mar 19, 2025

morotti commented on Mar 20, 2025

nmoinvaz commented on Mar 20, 2025

morotti commented on Mar 27, 2025

gpshead commented on Mar 27, 2025

morotti commented on Apr 17, 2025

nmoinvaz commented on Apr 17, 2025

morotti commented on May 21, 2026

rhpvorderman commented on Jul 21, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions