Optionally use Zopfli on the trace data #294
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jakirkham
commented
Nov 8, 2018
| " with gzip.GzipFile(fileobj=compressed, mode=\"wb\", compresslevel=compresslevel) as compressor:\n", | ||
| " compressor.write(data)\n", | ||
| " return compressed.getvalue()\n", | ||
| "def gzip_compress(data, compresslevel=6, use_zopfli=False):\n", |
There was a problem hiding this comment.
May want to toy with how these are exposed to the user.
jakirkham
force-pushed
the
use_zopfli
branch
2 times, most recently
from
49a751f to
17cd7eb
Compare
Adds the `zopfli` dependency so as to utilize Python bindings to Google's Zopfli library. It can achieve a slightly higher compression ratio than zlib while still storing the data in a format usable by zlib or gzip decompression algorithms. This can help us compress our trace data a little better before generating the HTML ROI Trace plot.
Expose the zlib compression level for the traces in the ROI trace plot as a parameter at the top of the cell.
Includes an option to use Zopfli to perform gzip compression. This can make a small though notable improvement in compression size of the data. Though can take a bit longer to run. As the data is still gzip compressed, it can still be decompressed with any gzip implementation like Pako. So no other changes are needed to handle the data.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Provides the option to use Zopfli to perform gzip compression on the trace data for the ROI Trace plots. By default this is disabled as Zopfli can be quite a bit slower than zlib and it does not provide that much more compression. However it can be useful to do a bit more compression on data that might be around for a while and/or doesn't quite fit some size constraints (e.g. an email attachment).
As the data is still gzip compressed, it is possible to perform gzip decompression as usual. Also performance of decompression is basically unchanged as well. Given this Pako, is still used as is to handle decompression in JavaScript.