Increase speed for create and unpack bundle operations - #26
Open
CCPCookies wants to merge 2 commits into
Open
Conversation
Now can support bundles split on uncompressed size. This allows for compression to be deferred and run asynchronously. This method is not appropriate for all data types. Split on compressed size remains the default. Memory operations that were causing speed degredation have been mitigated. Processing bottleneck is now md5 calculations and compression. Thread options have been added. Chunk processing is now conducted asynchronously. With deferred compression this is a considerable speed increase. Memory operations that were causing speed degredation have been mitigated. This is aided by caching more in RAM and the cache value is exposed to the user. Processing bottleneck is now md5 checksum and IO. Tests functionally remain the same, changes were made to make them work with the changes in internal interfaces. Further tests to cover new options will follow. Options added to CLI contain appropriate documentation through help interface. Auto generated c++ api covers all additions to the interface. Supporting bundle documentation is now out of date, an update will follow.
1. Added option to fail if patch size exceeds certain size 2. create-patch will now also produce a resourcegroup of new files added 3. apply-patch can now be set to skip new files to allow for them to be delivered in an alternate manner. Ie through bundling which works better if the new files added are large.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The focus of this change is to increase both creation and unpacking of bundles so it can be used for delivery of large datasets.
Especially focused on datasets which contain very large files.
Local speed test results running on large data set
create-bundle
before: ~38mins
after: ~2mins
unpack-bundle
before: ~8mins
after: ~2mins
create-bundle operation
Now can support bundles split on uncompressed size. This allows for compression to be deferred and run asynchronously. This method is not appropriate for all data types. Split on compressed size remains the default.
Memory operations that were causing speed degredation have been mitigated. Processing bottleneck is now md5 calculations and compression.
Thread options have been added. Chunk processing is now conducted asynchronously. With deferred compression this is a considerable speed increase.
unpack-bundle operation
Memory operations that were causing speed degredation have been mitigated. This is aided by caching more in RAM and the cache value is exposed to the user. Processing bottleneck is now md5 checksum and IO.
Tests
Tests functionally remain the same, changes were made to make them work with the changes in internal interfaces.
Further tests to cover new options will follow.
Documentation
Options added to CLI contain appropriate documentation through help interface. Auto generated c++ api covers all additions to the interface. Supporting bundle documentation is now out of date, an update will follow.