H5Coro Integration

[zachghiaccio]

We are currently looking to update the icepyx reader module to use H5Coro to efficiently read in queried ICESat-2 data. As is, the reader module is quite slow, especially if reading in data from the cloud, and H5Coro should make data reading much faster.

However, some work is needed for H5Coro to work with icepyx. H5Coro needs variable paths to be specified in a list format. It is also unclear how feasible it is to convert data loaded with H5Coro to Xarray. The easiest method is to read the data into GeoDataFrames, but converting these to Xarray is too memory-intensive.

A proposed temporary solution for the reader module is to convert local files into Xarray, and cloud-based files to GeoDataFrames. Moving forward, a more concrete solution will be needed for the H5Coro integration.

Current Tasks

• Make H5Coro reader routine that accepts icepyx variable lists (both default and custom user variable lists)
• Test H5Coro reader for multiple ICESat-2 products, given several spatial/temporal domain sizes
• Add check in reader module for local, cloud-sourced data
• Integrate H5Coro routine to icepyx reader module

[zachghiaccio]

When testing user-customized variable queries, I noticed that attempting to append default variables to order_vars may not always work.

region.order_vars.append(defaults=True)

This has been observed for the following products: ATL08 (lat/lon/time only), ATL10 (returns value error), ATL12 (lat/lon/time only). Higher-level products have not yet been tested.

Will focus on ATL06 for H5coro integration tests, to avoid this issue for now.

[zachghiaccio]

Script that reads icepyx variables using H5Coro works, and is decently fast. I still want to try and improve the speed of the code (maybe parallelizing it somehow), and I need to add the beams as a variable column. Might also add whether it's a "strong" or a "weak" beam.

One issue I ran into is that some of the variables (delta_time, segment_id, signal_selection_source) are of different size to the rest, which causes problems when loading the data. Not sure how to reconcile this issue.

[zachghiaccio]

Update on the varying size issue. I am attempting to use the icepyx query.order_vars.parse_var_list() function, though the output differs if you use query.order_vars.avail() as an input rather than, say, query.order_vars.wanted (if only defaults are appended).

Sample output for query.order_vars.parse_var_list(query.order_vars.avail()):

{'atlas_sdp_gps_epoch': ['ancillary_data/atlas_sdp_gps_epoch'], 'control': ['ancillary_data/control'], 'data_end_utc': ['ancillary_data/data_end_utc'], 'data_start_utc': ['ancillary_data/data_start_utc'], 'end_cycle': ['ancillary_data/end_cycle'], 'end_delta_time': ['ancillary_data/end_delta_time'], 'end_geoseg': ['ancillary_data/end_geoseg'], 'end_gpssow': ['ancillary_data/end_gpssow'], 'end_gpsweek': ['ancillary_data/end_gpsweek'], 'end_orbit': ['ancillary_data/end_orbit'], 'end_region': ['ancillary_data/end_region']}

(full output is too long, but there are many more variable paths.

Sample output for query.order_vars.parse_var_list(query.order_vars.wanted):

{'delta_time': ['delta_time'], 'latitude': ['latitude'], 'longitude': ['longitude'], 'h_li': ['h_li'], 'h_li_sigma': ['h_li_sigma'], 'atl06_quality_summary': ['atl06_quality_summary'], 'segment_id': ['segment_id'], 'sigma_geo_h': ['sigma_geo_h'], 'x_atc': ['x_atc'], 'y_atc': ['y_atc'], 'seg_azimuth': ['seg_azimuth'], 'sigma_geo_at': ['sigma_geo_at'], 'sigma_geo_xt': ['sigma_geo_xt'], 'dh_fit_dx': ['dh_fit_dx'], 'dh_fit_dx_sigma': ['dh_fit_dx_sigma'], 'h_mean': ['h_mean'], 'dh_fit_dy': ['dh_fit_dy'], 'h_rms_misfit': ['h_rms_misfit'], 'h_robust_sprd': ['h_robust_sprd'], 'n_fit_photons': ['n_fit_photons'], 'signal_selection_source': ['signal_selection_source'], 'snr_significance': ['snr_significance'], 'w_surface_window_final': ['w_surface_window_final'], 'bsnow_conf': ['bsnow_conf'], 'bsnow_h': ['bsnow_h'], 'cloud_flg_asr': ['cloud_flg_asr'], 'cloud_flg_atm': ['cloud_flg_atm'], 'r_eff': ['r_eff'], 'tide_ocean': ['tide_ocean']}

Variable paths are missing from the variable dict, and the secondary path variable is empty (not shown). Since I am working with the latter variables, I am unsure how to proceed. @JessicaS11 , do you know what might be happening?