feat(cache): compress big sqlite cache entries with zstd - #5744
Conversation
Store response bodies larger than the compressThreshold (default 1 MiB) compressed with zstdCompressSync, decompressing them when read back. Adds a compressThreshold option and a compressed column (bumping the cache table to V4), so large cached responses occupy less database space.
ronag
left a comment
There was a problem hiding this comment.
1 MB is quite big? text and json is quite compressible also for smaller size?
|
|
||
| -- Data returned to the interceptor | ||
| body BUF NULL, | ||
| compressed INTEGER NOT NULL DEFAULT 0, |
There was a problem hiding this comment.
can be good to make this a string and use "zstd", would make it more forward compatible
There was a problem hiding this comment.
Done - the compressed column now stores the algorithm as a string ("zstd") instead of an integer, and get() only decompresses rows whose algorithm is zstd, so future algorithms can be added without breaking existing data.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #5744 +/- ##
========================================
Coverage 93.47% 93.47%
========================================
Files 110 110
Lines 38908 39017 +109
========================================
+ Hits 36368 36473 +105
- Misses 2540 2544 +4 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| if (body && size > this.#compressThreshold && hasZstd) { | ||
| storedBody = zstdCompressSync(body) | ||
| compressed = 1 | ||
| } |
There was a problem hiding this comment.
I would recommend checking the response mime type to find easily compressible responses, e.g. json, xml, plain text etc... and ignore the compressThreshold for those
There was a problem hiding this comment.
Done - responses with an easily compressible Content-Type (text/*, application/json, application/xml, application/javascript, ...) are now compressed regardless of the compressThreshold. I also guard against pathological inflation: compression only applies when the zstd output is actually smaller than the original body.
Address review feedback: store the compression algorithm as a string
('zstd') in the compressed column for forward compatibility, and compress
responses with easily compressible Content-Types (json, xml, plain text)
even when they are below compressThreshold. Only compress when zstd actually
shrinks the payload.
Store SQLite cache response bodies that are larger than the new
compressThresholdoption (default 1 MiB) compressed withzstdCompressSync, decompressing them when read back so cached responses still return the exact original body. This lets large cached responses occupy significantly less database space.The change:
compressThresholdoption toSqliteCacheStore.compressedcolumn to the cache table (bumpingcacheInterceptorV3tocacheInterceptorV4) so stored bodies know whether to decompress.node:sqlite.SqliteCacheStoreOptstypes.