Rechunk test files

[asteiker]

Using the test files generated from #20 , create re-chunked files to include as test suite (so we are able to control and test across native, re-packed, re-chunked, and re-packed / re-chunked). We need to determine what chunk sizes to use: Leverage existing recommendations (1-2 MB).

Update:

• Probably best to re-chunk on the “slimmed down” files (dependent on Create controlled ATL03 HDF5 test file #27)
• There may be issues if the chunk sizes vary across the ATL03 files- This is what Sean Harkins had determined when working with ATL08. We may need to create synthetic data with uniform chunk sizes for benchmarking
• Will utilize h5pack. The command should be straightforward but may take a long time to run.
• There is also the rechunker package which is not a simple command - this is what's used in pangeo forge.

Tasks

Preview Give feedback

1. Determine chunk size variability within ATL03 test files
2. Determine chunk sizes to employ (in single digit MB range)
3. Run h5repack using determined chunk sizes against subsetted ATL03 test files

[asteiker]

We may want to rechunk on the controlled ATL03 HDF5 test file instead (so this would then be dependent on #27

[asteiker]

@betolink I may close this out as OBE and create a new ticket to reflect the IS2 data production script being used to generate rechunked files. Does that make sense to you?

[asteiker]

Closed as #34 is now in work based on ICESat-2 data production script.